Title: False Discovery Proportion control for aggregated Knockoffs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Knockoffs

Derandomization of Knockoffs

False Discoveries Proportion control

Controlled variable selection

Statistical inference

High-dimensional inference
Abstract: Controlled variable selection is an important analytical step in various scientific fields, such as brain imaging or genomics. In these high-dimensional data settings, considering too many variables leads to poor models and high costs, hence the need for statistical guarantees on false positives. Knockoffs are a popular statistical tool for conditional variable selection in high dimension. However, they control for the expected proportion of false discoveries (FDR) and not the actual proportion of false discoveries (FDP). We present a new method, KOPI, that controls the proportion of false discoveries for Knockoff-based inference. The proposed method also relies on a new type of aggregation to address the undesirable randomness associated with classical Knockoff inference. We demonstrate FDP control and substantial power gains over existing Knockoff-based methods in various simulation settings and achieve good sensitivity/specificity tradeoffs on brain imaging data.

False Discovery Proportion control for aggregated Knockoffs

Abstract

Controlled variable selection is an important analytical step in various scientific fields, such as brain imaging or genomics. In these high-dimensional data settings, considering too many variables leads to poor models and high costs, hence the need for statistical guarantees on false positives. Knockoffs are a popular statistical tool for conditional variable selection in high dimension. However, they control for the expected proportion of false discoveries (FDR) and not their actual proportion (FDP). We present a new method, KOPI, that controls the proportion of false discoveries for Knockoff-based inference. The proposed method also relies on a new type of aggregation to address the undesirable randomness associated with classical Knockoff inference. We demonstrate FDP control and substantial power gains over existing Knockoff-based methods in various simulation settings and achieve good sensitivity/specificity tradeoffs on brain imaging and genomic data.

1 Introduction

Statistically controlled variable selection arises in many different application fields, when the aim is to identify variables that are important for predicting an outcome of interest. For instance, in the context of brain imaging, practitioners are interested in finding which brain areas are relevant for predicting behavior or brain diseases. Such problems also appear in genomics, where practitioners wish to select genes associated with disease outcomes.

More precisely, we consider here conditional variable selection, meaning that we wish to select variables that are relevant to predict an outcome given the other variables. This type of inference is substantially more challenging than marginal inference, especially in high-dimensional settings, where the number of variables exceeds the number of samples. This is typically the case for brain mapping studies that comprise at most a few hundred subjects (hence, samples), while modern functional Magnetic Resonance Imaging (MRI) scans consist of more than 100k voxels . In the context of conditional inference, those are typically reduced to a few hundreds of brain regions, still possibly more than the number of samples.

Importantly, statistical guarantees are needed to ensure that the inference is reliable - i.e. that the proportion of false discoveries made by the variable selection procedure is controlled.

In the Knockoffs framework [1, 6], this problem is tackled by building noisy copies of the original variables. These copies are then compared to their original counterpart to perform variable selection.

The intuition underlying Knockoffs is that irrelevant variables do not get a larger weight than their Knockoff, while relevant variables do. Crucially, the Model-X Knockoffs procedure [6] controls the False Discovery Rate [2] which is the expected proportion of false discoveries.

A major caveat with this procedure is the random nature of the Knockoffs generation process: for two runs of the Knockoffs procedure on the same data, different Knockoffs will be built and subsequently different variables may be selected. This undesirable behavior hinders reproducibility. A second caveat is that False Discovery Rate (FDR) control does not imply False Discovery Proportion (FDP) control [12]. This leads to potentially unreliable inference: single runs of the method can produce a much higher proportion of False Discoveries than the chosen FDR level.

In this work, we propose a novel Knockoff-based inference procedure that addresses both concerns while offering power gains over existing methods, for no significant computation cost. The paper is organized as follows. After a refresher on Knockoff inference and aggregation, we consider the π statistic introduced in [17] to rank variables by relevance. Using the symmetry of knockoffs under the null hypothesis, we construct explicit upper bounds on the Joint Error Rate (JER; 4) of these statistics, leading to FDP control. We then use the calibration principle of [4] to obtain sharper bounds. Finally, we obtain a robust version of this method using harmonic mean aggregation of the π statistics across multiple Knockoffs draws. We demonstrate empirical power gains in various simulation settings and show the practical benefits of the proposed method for conditionally important region identification on fMRI and genomic datasets.

2 Related work

There has been much effort in the statistical community to achieve derandomized Knockoff-based inference. [19] introduced the idea of running Model-X Knockoffs [6] multiple times and computing for each the proportion of runs for which it was selected. [9] explore the idea of sampling multiple Knockoffs simultaneously. This induces a massive computational cost, which is prohibitive compared to methods that can support parallel computing. [17] introduced an aggregation method that relies on viewing Model-X Knockoffs as a Benjamini-Hochberg (BH) procedure [2] on so-called intermediate p -values . Such p -values can be computed on different Knockoff runs and aggregated using quantile aggregation [15] - then, BH is performed on the aggregated p -values to select variables. This approach relies on the heavy assumption that Knockoff statistics are i.i.d. under the null. Additionally, it is penalized by the conservativeness of the quantile aggregation scheme. Alternative aggregation schemes such as the harmonic mean [28] can be used but do not yield valid p -values.

[18] introduced an alternative aggregation procedure where Model-X Knockoffs are viewed as an e-BH procedure [25] on well-defined e-values [24]. Since the mean of two e-values remains an e-value, aggregation is done by averaging e-values across different Knockoffs draws. Then, e-BH is performed on the aggregated e-values to select variables. FDR control on aggregated Knockoffs is achieved without any additional assumption compared to Model-X Knockoffs. However, this method requires the difficult setting of a hyperparameter related to the chosen risk level, which highly impacts power in practice. Other recent developments in Knockoffs include the conditional calibration framework of Luo et al. [14] which aims at improving the power of Knockoffs-based methods.

There have been a few attempts at controlling other type 1 errors than the FDR using Knockoffs. [11] achieves k-FWER control and proposes that FDP control can be obtained by using a procedure that leverages joint k-FWER control. Recently, [13] introduced such a procedure to reach FDP control based on the k-FWER control introduced in [11]. In summary, the KOPI approach is the first one that aims at controlling the FDP of knockoffs-based inference for any aggregation scheme, leading to both accurate FDP control and increased sensitivity.

3 Refresher on Knockoffs

Notation. We denote vectors by bold lowercase letters. A vector x = { x 1 , . . . , x p } from which we removed the j th coordinate is denoted by x -j , i.e. x \ { x j } . Independence between two random vectors x and y is denoted by x ⊥ y . For two vectors x and ˜ x and a subset S of indices, ( x , ˜ x ) swap ( S ) denotes the vector obtained from ( x , ˜ x ) by swapping the entries x j and ˜ x j for each j ∈ S . Matrices are denoted by bold uppercase letters, the only exception being the vector of Knockoff statistics

that we denote by W as in [1, 6]. For any set S , | S | denotes the cardinality of S . For a vector z = ( z j ) 1 ≤ j ≤ p and S ⊂ J p K , we denote by z ( j : S ) (or z ( j ) when there is no ambiguity) the j th smallest value in the sub-vector ( z s ) s ∈ S . For an integer k , J k K denotes the set { 1 , . . . , k } . Equality in distribution is denoted by d = .

Problem setup. The input data are denoted by X ∈ R n × p , where n is the number of samples and p the number of variables. The outcome of interest is denoted by y ∈ R n . The goal is to select variables that are relevant with regards to the outcome conditionally on all others . Formally, we test simultaneously for all j ∈ J p K :

The output of a variable selection method is a rejection set ˆ S ⊂ J p K that estimates the true unknown support H 1 = { j : y ̸⊥ x j | x -j } . Its complement is the set of true null hypotheses H 0 = { j : y ⊥ x j | x -j } . Its cardinality |H 0 | is denoted by p 0 . To ensure reliable inference, our aim is to provide a statistical guarantee on the proportion of False Discoveries in ˆ S . The False Discovery Proportion (FDP) and the False Discovery Rate (FDR) [2] are defined as:

An α -level post-hoc FDP upper bound [10] is a function V that verifies:

Knockoffs. The Knockoff filter is a variable selection technique introduced by [1] and refined by [6] which controls the FDR. This procedure relies on building noisy copies of the original variables called Knockoff variables, that are designed to serve as controls for variable selection.

Definition 1 (Model-X Knockoffs, 6) . For the family of random variables x = ( x 1 , . . . , x p ) , Knockoffs are a new family of random variables ˜ x = (˜ x 1 , . . . , ˜ x p ) satisfying:

˜ x ⊥ y | x .

Once we have such variables at our disposal, we quantify their importance relative to the original ones. This is done by computing Knockoff statistics W = ( W 1 , . . . , W p ) that are defined as follows. Definition 2 (Knockoff Statistic, 6) . Aknockoff statistic W = ( W 1 , . . . , W p ) is a measure of feature importance that satisfies:

W depends only on X , ˜ X and y : W = g ( X , ˜ X , y ) .

Swapping column x j and its knockoff column ˜ x j switches the sign of W j :

The most commonly used Knockoff statistic is the Lasso-coefficient difference (LCD) [27]. This statistic is obtained by fitting a Lasso estimator [22] on [ X , ˜ X ] ∈ R n × 2 p , which yields ̂ β ∈ R 2 p . Then, the Knockoff statistic can be computed using ̂ β :

This coefficient summarizes the importance of the original j th variable relative to its own Knockoff: W j > 0 indicates that the original variable is more more important for fitting y than the Knockoff variable, meaning that the j th variable is likely relevant. Conversely, W j < 0 indicates that the j th variable is probably irrelevant. We thus wish to select variables corresponding to large and positive W j . Formally, the rejection set ˆ S can be written ˆ S = { j : W j > T q } , where T q is chosen to provably control the FDR at level q [6].

Aggregation schemes. Due to the randomness in the knockoff generation process, different variables may be selected for two different runs of the method, which is undesirable. To mitigate this, aggregation of multiple Knockoffs runs is needed. Ren and Barber [18] introduced an aggregation scheme which relies defining Knockoffs e -values.

Such e-values can be averaged across D draws and e-BH [25] is performed for variable selection. Alternatively, [17] defines the following π -statistic, that quantifies the evidence against a variable:

In [17] π statistics are treated as p -values and aggregated using quantile aggregation [15]. However, they can only be considered p -values under restrictive assumptions that are hard to check. In the next section, these statistics are used as a building block to reach FDP control. The KOPI framework does not require π statistics to be valid p -values.

4 Main contribution: FDP control for aggregated Knockoffs

4.1 Post hoc FDP control for π statistics

To obtain FDP control, we rely on Joint Error Rate control as introduced in [4]. For k max ∈ J p K , we define a threshold family of size k max as a vector t = ( t j ) j ∈ J k max K such that 0 ≤ t 1 ≤ · · · ≤ t k max ≤ 1 .

Definition 3 (Joint Error Rate, 4) . Denote by π ( j : H 0 ) the j th smallest value π j amongst all null hypotheses. The JER associated with t = ( t j ) j ∈ J k max K is:

The threshold family t is said to control the JER at level α iff JER( t ) ≤ α .

An α -level FDP upper bound can be derived from JER control via the following result:

Proposition 1 (FDP control via JER control 4) . If t is a threshold family of length k max that controls the JER at level α , then, V t ( S ) / | S | is an α -level FDP upper bound, with:

The proof of this result - originally included in [4] - can be found in appendix A.1 for selfcontainedness. In the remainder of this section, we show how to obtain JER control for π statistics.

4.2 Joint distribution of π statistics under the null

By Definition 3, JER ( t ) of a given threshold family only depends on the joint null distribution of the π statistics. As for earlier FDR control [1] or k-FWER control [11] results, the key idea to obtain JER control for π statistics is to prove that the relevant part of this distribution is in fact known, thanks to the properties of knockoff statistics. We use the same notation as in [11]. Letting Z j = | { k ∈ J p K : W k ≤ -W j } | and χ j = sign ( W j ) , the π statistics ( π j ) j = J p K are given by:

For a given W , let σ ( W ) be a permutation of J p K that sorts W by decreasing modulus: σ ( W ) = ( σ 1 , . . . , σ p ) such that | W σ 1 | ≥ | W σ 2 | · · · ≥ | W σ p | . We start by proving that the Z statistics can be expressed as a function of the vector of χ statistics:

Proof of Lemma 1. Since χ σ j = 1 , we have:

Lemma 1 implies that the distribution of order statistics of π | σ ( W ) is entirely determined by that of χ | σ ( W ) . To formalize this, we introduce π 0 statistics.

Definition 4 ( π 0 statistics) . Let χ 0 = ( χ 0 j ) 1 ≤ j ≤ p be a collection of p i.i.d. Rademacher random variables, that is, for all j , P ( χ 0 j = 1) = P ( χ 0 j = -1) = 1 / 2 . The associated π 0 statistics are defined for j ∈ J p K by

Theorem 1. Let t be a threshold family of length k max . Then, for π 0 = ( π 0 j ) j ∈ J p K as in (4) ,

Proof of Theorem 1. Let k ∈ J k max K . Since t k ≤ 1 , we have π ( k : H 0 ) < t k if and only if N k ≥ k , where

With the notation of Definition 4, we define the random variable

If H 0 = J p K , then Lemma 1 implies that conditional on σ ( W ) , N k and N 0 k have the same distribution. Indeed, the vectors ( W j ) j/χ j =1 and ( Z j ) j/χ j =1 have the same ordering, and conditional on σ ( W ) , ( χ j ) j ∈H 0 are jointly independent and uniformly distributed on {-1 , 1 } (Lemma 2.1 in 11; 1). Using the same argument as in the proof of Lemma 3.1 in Janson and Su [11], in the case where H 0 ⊊ J p K , false null χ j will insert -1 's into the process on the nulls, implying that N k is stochastically dominated by N 0 k . Noting that N 0 k ≥ k if and only if π 0 ( k ) < t k , we obtain that

Taking the expectation with respect to σ ( W ) yields the desired result.

Theorem 1 is related to Lemma 3.1 of Janson and Su [11] and Lemma 3.1 of Li et al. [13], that rely on the sign-flip property of Knockoff statistics under the null [1]. The interest of Theorem 1 is that the upper bound JER 0 ( t ) only depends on the π 0 statistics and the threshold family t , and not on the original data. Therefore, it can be estimated with arbitrary precision for any given t using Monte-Carlo simulation, as explained in the next section and described in Algorithm 1 in Supp. Mat.

4.3 Joint Error Rate control for π statistics via calibration

To approximate the JER upper bound derived in Theorem 1, we draw B Monte-Carlo samples using Algorithm 1. This yields a set of B vectors of π 0 statistics denoted by π 0 b ∈ R p for each b ∈ J B K . This allows us to evaluate the empirical JER , which estimates the upper bound of interest.

Definition 5 (Empirical JER) . For B vectors of π 0 statistics and a threshold family t , the empirical JER is defined as:

where for each b ∈ J B K , π 0 b (1) ≤ · · · ≤ π 0 b ( p ) .

Since ̂ JER 0 B ( t ) can be made arbitrarily close (by choosing B large enough) to ̂ JER 0 ( t ) for any given threshold family t , it remains to choose t such that ̂ JER 0 ( t ) ≤ α in order to ensure JER control. To this end, we consider a sorted set of candidate threshold families called a template :

Definition 6 (Template [4]) . A template is a component-wise non-decreasing function T : [0 , 1] ↦→ R p that maps a parameter λ ∈ [0 , 1] to a threshold family T ( λ ) ∈ R p .

This definition is naturally extended to the case of templates containing a finite number of threshold families. The template corresponding to B ′ threshold families is then denoted by ( T ( b ′ /B ′ )) b ′ ∈ J B ′ K .

Once a template is specified, the calibration procedure [4] can be performed; this consists in finding the least conservative threshold family t amongst the template that controls the empirical JER at level α . Formally, we consider the threshold family defined t B α = T ( λ B ( α )) , where

As observed by Blain et al. [3], optimal power is reached when the candidate families match the shape of the distribution of the null statistics. We define a template based on the distribution of the π 0 statistics appearing in Theorem 1. In practice, we draw B ′ samples from this distribution independently from the B Monte Carlo samples to avoid circularity biases. Since a template has to be component-wise non-decreasing, i.e. the set of candidate threshold families has to be sorted, we extract empirical quantiles from these B ′ sorted vectors. This yields a template T 0 composed of B ′ candidate curves that match quantiles of the distribution of π 0 statistics. The b ′ B ′ -quantile curve defines the threshold family T 0 ( b ′ /B ′ ) . We obtain the following result:

Theorem 2 (JER control for π -statistics) . Consider the threshold family defined by t B α = T 0 ( λ B ( α )) . Then, as B → + ∞ ,

The number B of Monte-Carlo samples in Theorem 2 can be chosen arbitrarily large to obtain JER control, leading to valid FDP bounds via Equation 3. This result is proved in Appendix A.2.

4.4 False Discovery Proportion control for aggregated Knockoffs

In the previous section we have seen how to reach FDP control via Knockoffs. As explained above, aggregation is needed to mitigate the randomness of the Knockoff generation process. Therefore, we aim to extend the previous result to the case of aggregated Knockoffs. Let us first define aggregation:

Definition 7. For D draws of Knockoffs, an aggregation procedure is a function f : R D ↦→ R that maps a vector of ( π d ) d ∈ J D K statistics to an aggregated statistic π .

In practice, since we have p variables, aggregation is performed for each variable, i.e.:

Then, inference is performed on the vector of aggregated statistics ( π 1 , . . . , π p ) .

For a fixed aggregation scheme f , we can naturally extend the calibration procedure of the preceding section. Instead of drawing a single B × p matrix of π 0 statistics containing π 0 b ∈ R p for each b ∈ J B K , we draw D such matrices. Given d ∈ J D K , each matrix contains π 0 ,d b ∈ R p for each J B K .

Then, for each b ∈ J B K , we perform aggregation: π 0 b = f ( ( π 0 ,d b ) d ∈ J D K ) . The JER in the aggregated case is defined as:

We obtain the aggregated template following the same procedure, i.e. drawing D templates and aggregating them. For each b ′ ∈ B ′ , the aggregated threshold family is written:

We can then write the empirical JER in the aggregated case as:

Calibration can be performed in the same way as in the non-aggregated case. Note that we perform calibration after aggregating; therefore, JER control is ensured directly on aggregated statistics and is not a result of aggregating JER controlling families. Importantly, this approach holds without additional assumptions on the aggregation scheme f . We consider the threshold family t B α = T ( λ B ( α )) , where

With T 0 a template composed of B ′ candidate curves that match quantiles of the distribution of π 0 statistics, we obtain the following result:

Theorem 3 (JER control for aggregated π -statistics) . Consider the threshold family defined by t B α = T 0 ( λ B ( α )) . Then, as B → + ∞ ,

Proof. The proof is identical to that of Theorem 2 using the empirical aggregated JER .

The calibrated aggregated threshold family yields valid FDP upper bounds via Proposition 1. The proposed KOPI (Knockoffs π ) method therefore achieves FDP control on aggregated Knockoffs.

5 Experiments

Methods considered. In our implementation of KOPI, we rely on the harmonic mean [28] as the aggregation scheme f . Additionally, we set k max = ⌊ p/ 50 ⌋ following the approach of [3]. We also consider both state-of-the-art Knockoffs aggregation schemes: AKO (Aggregation of Multiple Knockoffs, 17) and e-values based aggregation [18]. Additionally, we consider Vanilla Knockoffs, i.e. [6] and FDP control via Closed Testing [13]. In simulated data experiments, we generate Knockoffs assuming a Gaussian distribution for X , with all variables centered. For methods that support aggregation, we use D = 50 Knockoff draws.

5.1 Simulated data

Setup. At each simulation run, we generate Gaussian data X ∈ R n × p with a Toeplitz correlation matrix corresponding to a first-order auto-regressive model with parameter ρ , i.e. Σ i,j = ρ | i -j | .

Then, we draw the true support β ∗ ∈ { 0 , 1 } p . The number of non-null coefficients of β ∗ is controlled by the sparsity parameter s p , i.e. s p = ∥ β ∗ ∥ 0 /p . The target variable y is built using a linear model:

0.0

0.2

0.3

0.4

1 - coverage

0.25

0.50

0.75

0.00

0.25

0.50

0.75

1.00

Empirical power

1

2

3

Signal-to-noise ratio

500

1000

1500

n_samples

0.1

0.2

sparsity

KOPI

Closed Testing

e-values

AKO

Vanilla KO

with σ controlling the amplitude of the noise: σ = ∥ X β ∗ ∥ 2 / ( SNR ∥ ϵ ∥ 2 ) , SNR being the signal-tonoise ratio. We choose the central setting n = 500 , p = 500 , ρ = 0 . 5 , s p = 0 . 1 , SNR = 2 . For each parameter, we explore a range of possible values to benchmark the methods across varied settings.

To select variables using FDP upper bounds, we retain the largest possible set of variables S such that V ( S ) ≤ q | S | (Algorithm 4). For each of the N simulations and each method, we compute the empirical FDP and True Positive Proportion (TPP):

If the FDP is controlled at level α , |{ k ∈ J N K : ̂ FDP ( S k ) > q }| ∼ B ( N,α ) . Then, we can compute error bands on the α -level using std ( B ( N,α ) /N ) = √ α (1 -α ) /N . The second row of Fig. 1 represents the empirical power achieved by each method, which corresponds to the average of TPPs defined above for N runs i.e. Power = ∑ N k =1 ̂ TPP ( S k ) /N. Fig. 1 shows that across all different settings, KOPI retains FDP control. We can also see that FDR control does not imply FDP control, as Vanilla Knockoffs are consistently outside of FDP bound coverage intervals. However, the two existing aggregation schemes (AKO and e-values) that formally guarantee FDR control are generally conservative and achieve FDP control empirically. This is consistent with the findings of [18]. The Closed Testing procedure of [13] achieves FDP control as announced but suffers from a lack of power.

Interestingly, KOPI achieves FDP control while offering power gains compared to FDR-controlling Knockoffs aggregation methods. Yet FDP control is a much stronger guarantee than FDR control, as discussed previously. These gains are especially noticeable in challenging inference settings where most methods exhibit a clear decrease in power or even catastrophic behavior (i.e. zero power).

Moreover, Fig. 3 (in appendix) shows that when using q = 0 . 05 rather than q = 0 . 1 as in Fig. 1, the robustness of KOPI with regards to difficult inference settings is even more salient. More precisely, for q = 0 . 05 , AKO and Closed Testing are always powerless. E-values aggregation yields good power in easier settings such as ρ ≤ 0 . 6 , SNR ≥ 2 . 5 or n > 750 but exhibits catastrophic behavior in harder settings. Overall, apart from KOPI, only Vanilla Knockoffs exhibit non-zero power, but this method fails to control the FDP as it is intended to control FDR. KOPI preserves FDP control in all settings while yielding superior power compared to all other methods.

KOPI

CT

e-values

AKO

Vanilla KO

0.00

0.05

0.10

0.15

0.20

False Discovery Proportion

KOPI

CT

e-values

AKO

Vanilla KO

0.0

0.1

0.2

0.3

0.4

0.5

Empirical Power

5.2 Brain data application

The goal of human brain mapping is to associate cognitive tasks with relevant brain regions. This problem is tackled using functional Magnetic Resonance Imaging (fMRI), which consists in recording the blood oxygenation level dependent signal via an MRI scanner. The importance of conditional inference for this problem has been outlined in [26]. We use the Human Connectome Project (HCP900) dataset that contains brain images of healthy young adults performing different tasks while inside an MRI scanner. Details about this dataset and empirical results can be found in Appendix E.

While these results demonstrate the face validity of the approach, FDP control and power cannot be evaluated. Therefore, following [16], we consider an additional experiment that consists in using semi-simulated data. We consider a first fMRI dataset ( X 1 , y 1 ) on which we perform inference using a Lasso estimator; this yields β ∗ 1 ∈ R p that we will use as our ground truth. Then, we consider a separate fMRI dataset ( X 2 , y 2 ) for data generation. The point of using a separate dataset is to avoid circularity between the ground truth definition and the inference procedure. Concretely, we discard the original response vector y 2 for this dataset and build a simulated response y sim 2 using a linear model, with the same notation as previously (we set σ so that SNR = 4 ): y sim 2 = X 2 β ∗ 1 + σ ϵ .

Then, inference is performed using Knockoffs-based methods on ( X 2 , y sim 2 ) . Since we consider β ∗ 1 as the ground truth, the FDP and TPP can be computed for each method. As can be seen in Fig. 2, KOPI is the most powerful method among those that control the FDP.

5.3 Genomic data application

In addition to the brain data application, we compared KOPI to other Knockoffs-based methods on gene-expression data [5] containing 79 samples and 90 genes. KOPI yields a non trivial selection for all runs, with 3 genes selected in 100% of all 50 runs of the experiment. Across all runs, only 8 different genes are selected by KOPI. Vanilla Knockoffs select 24 different genes across all runs and no gene exceeds a selection frequency of 70% . All other methods are powerless in all runs. Details and results of this experiment can be found in Appendix D.

6 Discussion

In this paper, we have proposed a novel method that reaches FDP control on aggregated Knockoffs. It combines the benefits of aggregation, i.e. improving the stability of the inference, in addition to providing a probabilistic control of the FDP, rather than controlling only its expectation, the FDR.

Simulation results support that KOPI indeed controls the FDP. Furthermore, while FDP control is a stricter guarantee than FDR control, KOPI actually offers power gains compared to state-of-the-art aggregation-based Knockoffs methods. This sensitivity gain is a direct benefit from the JER approach and its adaptivity to arbitrary aggregation schemes. While the latter has been formulated and used so far in mass univariate settings [3], the present work presents a first use of this approach in the context of multiple regression. Moreover, KOPI does not require any assumption on the data at hand or on the law of Knockoff statistics under the null.

The computation time of the proposed approach is comparable to existing aggregation schemes for Knockoffs: sampling π statistics under the null using Algorithm 1 can be done once and for all for a given value of p . JER estimation via Algorithm 2 and calibration can be performed via binary search of complexity O ( log ( B ′ )) . Finding the rejection set ˆ S after performing calibration is done in linear time via [7]. In practice, the computation time is the same as for classical knockoff aggregation [19] and is in minutes for the brain imaging datasets considered. Avenues for future work include a theoretical analysis of the False Negative Proportion (FNP) [8] of KOPI and developing a step-down version of the method to further improve power.

We provide a Python package containing the code for KOPI available at https://github.com/ alexblnn/KOPI .

7 Acknowledgments and disclosure of funding

This project was funded by a UDOPIA PhD grant from Université Paris-Saclay and also supported by the FastBig ANR project (ANR-17-CE23-0011), the KARAIB AI chair (ANR-20-CHIA-0025-01), the H2020 Research Infrastructures Grant EBRAIN-Health 101058516 and the SansSouci ANR project (ANR-16-CE40-0019). The authors thank Binh Nguyen for his precious help on the code base and Samuel Davenport for useful discussions about this work.

References

Barber, R. F. and Candès, E. J. (2015). Controlling the false discovery rate via knockoffs. The Annals of Statistics , 43(5):2055-2085.

Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) , 57(1):289-300.

Blain, A., Thirion, B., and Neuvial, P. (2022). Notip: Non-parametric true discovery proportion control for brain imaging. NeuroImage , 260:119492.

Blanchard, G., Neuvial, P., Roquain, E., et al. (2020). Post hoc confidence bounds on false positives using reference families. Annals of Statistics , 48(3):1281-1303.

Bourgon, R., Gentleman, R., and Huber, W. (2010). Independent filtering increases detection power for high-throughput experiments. Proceedings of the National Academy of Sciences , 107(21):9546-9551.

Candès, E., Fan, Y., Janson, L., and Lv, J. (2018). Panning for gold:'model-x'knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80(3):551-577.

Enjalbert-Courrech, N. and Neuvial, P. (2022). Powerful and interpretable control of false discoveries in two-group differential expression studies. Bioinformatics , 38(23):5214-5221.

Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. Journal of the Royal Statistical Society Series B: Statistical Methodology , 64(3):499-517.

Gimenez, J. R. and Zou, J. (2019). Improving the stability of the knockoff procedure: Multiple simultaneous knockoffs and entropy maximization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2184-2192. PMLR.

Goeman, J. J. and Solari, A. (2011). Multiple testing for exploratory research. Statistical Science , 26(4):584-597.

Janson, L. and Su, W. (2016). Familywise error rate control via knockoffs. Electronic Journal of Statistics , 10(1):960-975.

Korn, E. L., Troendle, J. F., McShane, L. M., and Simon, R. (2004). Controlling the number of false discoveries: application to high-dimensional genomic data. Journal of Statistical Planning and Inference , 124(2):379-398.

Li, J., Maathuis, M. H., and Goeman, J. J. (2022). Simultaneous false discovery proportion bounds via knockoffs and closed testing. arXiv preprint arXiv:2212.12822 .

Luo, Y., Fithian, W., and Lei, L. (2022). Improving knockoffs with conditional calibration.

Meinshausen, N., Meier, L., and Bühlmann, P. (2009). P-values for high-dimensional regression. Journal of the American Statistical Association , 104(488):1671-1681.

Nguyen, B. T., Thirion, B., and Arlot, S. (2022). A Conditional Randomization Test for Sparse Logistic Regression in High-Dimension. In NeurIPS 2022 , volume 35 of Advances in Neural Information Processing Systems , New Orleans, United States.

Nguyen, T.-B., Chevalier, J.-A., Thirion, B., and Arlot, S. (2020). Aggregation of multiple knockoffs. In International Conference on Machine Learning , pages 7283-7293. PMLR.

Ren, Z. and Barber, R. F. (2022). Derandomized knockoffs: leveraging e-values for false discovery rate control. arXiv preprint arXiv:2205.15461 .

Ren, Z., Wei, Y., and Candès, E. (2021). Derandomizing knockoffs. Journal of the American Statistical Association , pages 1-11.

Stiglic, G. and Kokol, P. (2010). Stability of ranked gene lists in large microarray analysis studies. Journal of biomedicine and biotechnology , 2010.

Thirion, B., Varoquaux, G., Dohmatob, E., and Poline, J.-B. (2014). Which fMRI clustering gives good brain parcellations? Frontiers in Neuroscience , 8(167):13.

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288.

Van Essen, D. C., Ugurbil, K., Auerbach, E. J., Barch, D. M., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., Della Penna, S., Feinberg, D. A., Glasser, M. F., Harel, N., Heath, A. C., Larson-Prior, L. J., Marcus, D. S., Michalareas, G., Moeller, S., Oostenveld, R., Petersen, S. E., Prior, F. W., Schlaggar, B. L., Smith, S. M., Snyder, A. Z., Xu, J., and Yacoub, E. (2012). The Human Connectome Project: a data acquisition perspective. Neuroimage , 62(4):2222-2231.

Vovk, V. and Wang, R. (2021). E-values: Calibration, combination and applications. The Annals of Statistics , 49(3):1736-1754.

Wang, R. and Ramdas, A. (2022). False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(3):822-852.

Weichwald, S., Meyer, T., Özdenizci, O., Schölkopf, B., Ball, T., and Grosse-Wentrup, M. (2015). Causal interpretation rules for encoding and decoding models in neuroimaging. NeuroImage , 110:48-59.

Weinstein, A., Su, W. J., Bogdan, M., Barber, R. F., and Candès, E. J. (2020). A power analysis for knockoffs with the lasso coefficient-difference statistic. arXiv preprint arXiv:2007.15346 .

Wilson, D. J. (2019). The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences , 116(4):1195-1200.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method KOPI, based on the knockoff framework that controls the false discovery _proportion_ instead of the false discovery _rate_.  The key idea of the approach is to note that certain summaries of the knockoff statistics are exactly equal in distribution to certain functions of independent Rademacher random variables under the case of all nulls.  One can then estimate the FDP via Monte Carlo using draws of Rademachers, allowing for the setting of thresholds to obtain a desired FDP.  This approach is then straightforwardly extended to the case of aggregating multiple draws of the knockoffs.  The approach is shown to truly control the FDP while maintaining good power on some simulated datasets.\n\nSTRENGTHS:\n* The approach is clever and relatively straightforward.\n* The approach does not rely on any distributional assumptions beyond the ability to generate knockoffs with the usual exchangeability property.\n* The paper does a good job of introducing the knockoff framework and situating the present developments in that context.\n* The approach seems well powered and has attractive theoretical guarantees.\n\nWEAKNESSES:\n* One minor weakness of the paper is that there are a number of potential tweaks to the method that are not explored or presented.  While the authors seem to have found choices that appear to work well, they appear somewhat arbitrary without additional context.  For example,   the choice of using harmonic mean for aggregation seems somewhat arbitrary without comparing to other choices, particularly the obvious choices like arithmetic or geometric mean. Another example is the choice of template.  The theoretical results are quite general, but then only a particular template is considered.\n\nTypos:\n* Line 75: is a word missing here: \"knockoffs-based inference arbitrary aggregation schemes\"?\n* Line 76: \"and increases sensitivity\" --> \"and increased sensitivity\"\n* Line 92: \"complementary\" --> \"complement\"\n* Line 93: \"cardinal\" --> \"cardinality\"\n* Line 140: Proposition 2 is empty and presumably included by mistake.\n* Line 153: The equations following this line are assuming that $W_j$ is positive, and that should be stated.  E.g., if $W_j = -2$ and $W_k = 1$, then $1 = W_k \\le -W_j = 2$ but $W_k$ is obviously not smaller than $0$.\n* Line 154: This line is assuming that $W_1,\\ldots, W_P$ are ordered, which they are not according to the notation. Either there should be a \"without loss of generality, assume that $W$ is sorted such that $\\sigma(W)$ is the identity\" or all of the indices in the proof should really be $\\sigma_j$ and $\\sigma_k$ instead of $j$ and $k$.\n* Line 184: \"is arbitrarily close\" should be changed to something like \"can be made arbitrarily close by taking $B$ large enough\"\n* Line 191: \"consists in finding\" --> \"consists of finding\"\n* Line 280: \"face validity\" --> \"validity\"\n* Line 308: Saying that the method requires no assumption on the \"law of the Knock statistics under the null\" is ignoring that the Knockoffs much be exchangeable with the original data, no?\n\nQUESTIONS:\n* The proof of Theorem 1 essentially bounds $p_0$ by $p$ at one point.  Would it be possible to do the usual trick from FDR-type procedures of using a conservative estimate of $p_0$ in order to obtain a more powerful procedure?  This is not necessary for the present paper, but I'm curious how difficult such an approach would be.\n\nLIMITATIONS:\nI do not believe the present work has any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

comments: SUMMARY:
I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.

STRENGTHS:
I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.

WEAKNESSES:
I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.

QUESTIONS:
I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.

LIMITATIONS:
I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

comments: SUMMARY:
This paper studies an important topic: variable selection. Concretely, the authors proposed a novel method KOPI which can theoretically control false discovery proportion at a pre-specified level. Besides, the authors also conduct lots of experiments based on simulated data and real data to verify the effectiveness of their proposed procedure.

STRENGTHS:
1，	This paper is well-written, its notations and definitions are clear;

2，	The authors provide some necessary theoretical guarantees for their proposed algorithm.

WEAKNESSES:
1，	Simulation experiments and real-data experiments are conducted, However, in order to verify the effectiveness of KOPI, more experiments on other real-data datasets are needed;

2，	It will be better if the authors can provide an analysis of the false nondiscovery proportion (FNP), a dual quantity of FNP, or its expectation FNR., see [1] .

[1] Genovese C, Wasserman L. Operating characteristics and extensions of the false discovery rate procedure[J]. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2002, 64(3): 499-517.

QUESTIONS:
No

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
In this paper, the authors discuss controls for false discoveries in variable selection. While how to perform FDR control is known, authors opt to control FDP, which is a random quantity depending on a specific dataset. The main concept is an upperbound of FDP (proposition 1), namely JER. The monte carlo version of JER is applied to perform FDP control. The authors also propose an aggregated JER to reduce the stochasticity JER for variable selection. Experiments show good FDP control while maintaining good power.

STRENGTHS:
1. The paper is very well written. 
2. FDP (not just FDR) control is an important problem. This paper targets on an important issue in our community. 
3. The proposed method has clear theoretical support (theorem 2 and 3), although I have some questions.

WEAKNESSES:
It is not entirely clear to me the delta between this paper and an important related work, i.e, [4]. From Section 4, it seems that using JER to control FDP has already been proposed. Thus, I wonder how significant the contribution of this work is. 

For example, has Proposition 1 or a similar version already been proposed in [4]? I would appreciate a short paragraph comparing this work and [4].

QUESTIONS:
I am not sure Why JER, not the empirical JER is studied in Theorem 2? In practice, shouldn't one use empirical JER with monte carlo samples?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06