Title: DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Feature Selection

Deep Learning

Model-X Knockoff

FDR Control

Boosting Power
Abstract: Model-X knockoff has garnered significant attention among various feature selection methods due to its guarantees for controlling the false discovery rate (FDR). Since its introduction in parametric design, knockoff techniques have evolved to handle arbitrary data distributions using deep learning-based generative models. However, we have observed limitations in the current implementations of the deep Model-X knockoff framework. Notably, the "swap property" that knockoffs require often faces challenges at the sample level, resulting in diminished selection power. To address these issues, we develop "Deep Dependency Regularized Knockoff (DeepDRK)," a distribution-free deep learning method that effectively balances FDR and power. In DeepDRK, we introduce a novel formulation of the knockoff model as a learning problem under multi-source adversarial attacks. By employing an innovative perturbation technique, we achieve lower FDR and higher power. Our model outperforms existing benchmarks across synthetic, semi-synthetic, and real-world datasets, particularly when sample sizes are small and data distributions are non-Gaussian.

DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection

Abstract

Model-X knockoff has garnered significant attention among various feature selection methods due to its guarantees for controlling the false discovery rate (FDR). Since its introduction in parametric design, knockoff techniques have evolved to handle arbitrary data distributions using deep learning-based generative models. However, we have observed limitations in the current implementations of the deep Model-X knockoff framework. Notably, the 'swap property' that knockoffs require often faces challenges at the sample level, resulting in diminished selection power. To address these issues, we develop 'Deep Dependency Regularized Knockoff (DeepDRK),' a distribution-free deep learning method that effectively balances FDR and power. In DeepDRK, we introduce a novel formulation of the knockoff model as a learning problem under multi-source adversarial attacks. By employing an innovative perturbation technique, we achieve lower FDR and higher power. Our model outperforms existing benchmarks across synthetic, semi-synthetic, and real-world datasets, particularly when sample sizes are small and data distributions are non-Gaussian.

1 Introduction

Feature selection (FS) has garnered significant attention over the past few decades due to the rapidly increasing dimensionality of data, as well as the associated computational, storage, and noise challenges [22, 15]. Successfully identifying the true informative features among inputs can significantly enhance the performance of analysis frameworks and drive advancements in fields such as biology, neuroscience, medicine, economics, and social sciences [15, 55]. However, the task of accurately pinpointing informative features is often considered nearly impossible, particularly due to the limited availability of data relative to the increasing dimensionality [55, 11]. A practical approach to address this challenge is the development of algorithms designed to select features while maintaining controlled error rates. Targeting this goal, Model-X knockoffs, a novel framework, is proposed in [4, 11] to select relevant features while controlling the false discovery rate (FDR). In contrast to the classical setup, where assumptions on the correlations between input features and the response are imposed [6, 19], the Model-X knockoff framework only requires a linear relationship between the response and the features. With a strong finite-sample FDR guarantee, Model-X knockoff saw broad applications in domains such as biology, neuroscience, and medicine, where the sample size is limited [4, 11, 27, 47, 55, 38, 62].

There have been considerable developments of knockoffs since its debut. In scenarios where feature distributions are complex, various deep learning methods [27, 47, 55, 38, 62] have been proposed. However, we observe major limitations despite improved performance. First, the performance of existing methods varies across different data distributions. Second, the quality of selection declines

when the sample size is relatively small. Third, training the deep knockoff generation models can be challenging due to competing loss terms in the training objective. We elaborate on the drawbacks in Sections 2.2 and 3.2.

In this paper, we address these issues by proposing the Deep Dependency Regularized Knockoff (DeepDRK), a deep learning-based pipeline. We formulate the knockoff generation as an adversarial attack problem involving multiple sources. By optimizing a model against the adversarial environments, we achieve better 'swap property' [4] compared to the baseline models. DeepDRK is also equipped with a novel perturbation technique to reduce 'reconstructability' [54], which in turn controls FDR and boosts selection power. The experiments conducted on synthetic, semi-synthetic, and real datasets demonstrate that our pipeline outperforms existing methods across various scenarios.

2 Background and Related Works

2.1 Model-X Knockoffs for FDR control

The Model-X knockoffs framework consists of two main components. Given the explanatory variables X ' p X 1 , X 2 , . . . , X p q J P R p and the response variable Y ( Y continuous for regression and categorical for classification), the framework requires: 1. a knockoff ˜ X ' p ˜ X 1 , ˜ X 2 , . . . , ˜ X p q J that 'fakes' X ; 2. the knockoff statistics w j for j P r p s that assess the importance of each feature X j . The knockoff ˜ X is required to be independent of Y conditioning on X , and must satisfy the swap property:

Here swap p B q exchanges the positions of any variable X j , j P B , with its knockoff ˜ X j . The knockoff statistic w j pp X, ˜ X q , Y q (for j P r p s ) depends on the concatenated variable p X, ˜ X q and Y and must satisfy the flip-sign property:

The functions w j p¨q with j P r p s have many candidates, for example w j ' | β j | ´ | ˜ β j | , where β j and ˜ β j are the corresponding regression coefficients of X j and ˜ X j with the regression function Y ' ř p j ' 1 p X j β j ` ˜ X j ˜ β j q ` ϵ , where ϵ is independently drawn from the normal distribution.

When the two knockoff conditions (i.e., Eq. (1) and (2)) are met, one can select features by S ' t w j ě τ q u , where

To assess the feature selection quality, FDR is commonly used as an average Type I error of selected features [4], which is defined as follows. Let S Ă r p s be an arbitrary set of selected indices, and let β ˚ denote the underlying true regression coefficients. The FDR for the selection S is

The control of FDR is guaranteed by the following theorem from [11]:

Theorem 2.1. Given the knockoff that satisfies the swap property in Eq. (1) , the knockoff statistic that satisfies Eq. (2) , and S ' t w j ě τ q u , we have FDR ď q .

2.2 Related Works

Model-X knockoff is first studied under Gaussian design. Namely, the original variable X ' N p µ, Σ q with µ and Σ known. Since Gaussian design does not naturally generalize to complex data distributions, several methods are proposed to weaken the assumption. Among them, model-specific ones

DeepDRKPipeline

Training Stage

Multi-Swappers

Trainable

Loss

Matrix

Network

Function

Operation

S

CSL(X, X, (SW})

+

Post-training Stage

LDRL(X,X)

X→

Knockoff

Transformer

Dependency

Z一

(KT)

Regularized

Perturbation(DRP)

xDRP

such as AEknockoff [34], Hidden Markov Model (HMM) knockoff [51], and MASS [20] all propose parametric alternatives to Gaussian design. These methods can better learn the data distribution, while keeping the sampling process relatively simple. Nevertheless, they pose assumptions to the design distribution, which can be problematic if actual data does not coincide. To gain further flexibility, various deep-learning-based models are developed to generate knockoffs from distributions beyond parametric setup. DDLK [55] and sRMMD [38] utilize different metrics to measure the distances between the original and the knockoff covariates. They apply different regularization terms to impose the 'swap property'. He et al. [24] introduced a KnockoffScreen procedure to generate multiple knockoffs to improve the stability by minimizing the variance during knockoff construction. KnockoffGAN [27] and Deep Knockoff [47] take advantage of the deep learning structures to create likelihood-free generative models for the knockoff generation.

Despite the flexibility to learn the data distribution, deep-learning-based models suffer from a major drawback. Knockoff generations based on distribution-free sampling methods such as generative adversarial networks (GAN) [21, 3] tend to overfit, namely to learn the data X exactly. The reason is that the notion of swap property for continuous distributions is not well defined at the sample level. To satisfy the swap property, one needs to independently sample ˜ X j from the conditional law P X j p¨| X ´ j q , where X ´ j denotes the vector p X 1 , . . . , X j ´ 1 , X j ` 1 , . . . , X p q . At the sample level, each realization of X ´ j ' x i ´ j is almost surely different and only associates to one corresponding sample X j ' x i j , causing the conditional law to degenerate to sum of Diracs. As a result, minimizing the distance between p X, ˜ X q and p X, ˜ X q swap p B q will push ˜ X towards X and introduce high collinearity that makes the feature selection powerless, i.e., with high type II error. To tackle this issue, DDLK [55] suggests an entropic regularization. Yet it still lacks power and is computationally expensive.

2.3 Boost Power by reducing reconstructability

The issue of lacking power in the knockoff selection is solved in the Gaussian case [54]. Assuming the knowledge of both mean and covariance of X ' N p µ, Σ q , the swap property is easily satisfied by setting ˜ X j ' N p µ j , Σ jj q and Σ ij ' Var p X i , ˜ X j q , for i ‰ j , i, j P r p s . Barber & Cand` es [4] originally propose to minimize Var p X j , ˜ X j q for all j P r p s using semi-definite programming (SDP), to prevent ˜ X j to be highly correlated with X j . However, Spector & Janson [54] observed that the SDP knockoff still lacks feature selection power, as merely decorrelating X j and ˜ X j is not enough, and p X, ˜ X q can still be (almost) linearly dependent in various cases. This is referred to as high reconstructability 1 in their paper, which can be considered as a population counterpart of collinearity (see Appendix A for more details). To tackle the problem, [54] proposed to maximize the expected conditional variance E Var p X j | X ´ j , ˜ X q , which admits close-form solution whenever X is Gaussian.

3 Method

DeepDRK in Figure 1 provides a novel way to generate knockoff ˜ X while reducing the reconstructability (see Section 2.3) between the generated knockoff ˜ X and the input X for data with

complex distributions. The generated knockoff can then be used to perform FDR-controlled feature selection following the Model-X knockoff framework (see Section 2.1). Overall, DeepDRK contains two main components. It first trains a transformer-based deep learning model, referred to as Knockoff Transformer (KT), to obtain the swap property and reduce the reconstructability of the generated knockoff. This is achieved by incorporating adversarial attacks with multi-swappers. Secondly, a dependency regularized perturbation technique (DRP) is developed to further reduce the reconstructability for ˜ X post training. We will elaborate on these two components in the following two subsections. In this section, we slightly abuse the notation such that X and ˜ X also denote the corresponding data matrices.

3.1 Training with Knockoff Transformer and Swappers

The KT aims to generate knockoffs, denoted by ˜ X θ , which are parameterized by a transformer network with parameters θ . The loss for training the KT contains a swap loss (SL) L SL, which enforces the swap property, and a dependency regularization loss (DRL) L DRL, which controls the reconstructibility of the knockoff. K different neural network parameterized swappers, denoted by t S ω i u K i ' 1 , are used to test whether the generated knockoffs satisfy the swap property. Thus, the KT is trained adversarially according to the following objective,

Note that we use ˜ X θ and ˜ X interchangeably; however, the former emphasizes that the knockoff depends on the model weights θ . We discuss each loss function below. Details on the network architectures for KT and swappers are deferred to Appendix B.1. The training algorithm is in Appendix B.2.

3.1.1 Swap Loss

The swap loss is designed to enforce the swap property and is defined as follows:

where λ 1 and λ 2 are hyperparameters.

The first term in Eq. (6) uses the sliced Wasserstein distance (SWD, see Appendix C for definition) to measure the distance between a pair of joint distributions for p X, ˜ X θ q and p X, ˜ X θ q S ω i , where the swapper is parameterized by ω i . We sum over K SWDs computed for the distributions modified under different swappers to capture the effects of multiple swap attacks. We utilize SWD to compare distributions because it excels in handling complex data distributions and is computationally efficient [13, 28, 14].

Sudarshan et al. [55] introduced a single swap attack parameterized by a neural network. However, we observe that minimizing the worst case swap for all B Ă r p s , as suggested by [55], cannot guarantee the swap property of ˜ X θ . To address this limitation, we introduce a 'multi-swapper' setup that uses multiple swappers to enforce the swap property. And this leads to the introduction of the second and the third terms in the objective in Eq. (6).

The second term in Eq. (6), REx p X, ˜ X θ , t S ω i u K i ' 1 q evaluates the variance of the sliced Wasserstein distances SWD pp X, ˜ X θ q , p X, ˜ X θ q S ω q with K realizations of ω [30]. When REx p X, ˜ X θ , t S ω i u K i ' 1 q' 0 , the sliced Wasserstein distances are identical across all swappers. Therefore, complementary to the first term, minimizing this term improves the adherence of swap property for the generated ˜ X θ .

The third term in Eq. (6) is introduced to avoid mode collapse on the parameters ω i of different swappers and ensure each swapper characterizes a different adversarial environment:

where C ' tp i, j q| i, j P r K s , i ‰ j u , and sim p¨ , ¨q is the cosine similarity between the weights ω of a pair of different swappers. Without this regularization, all swappers could collapse to a single mode such that the multi-swapper scheme reduces to a single-swapper setup.

Overall, the swap loss L SL enforces the swap property via the novel multi-swapper design. Such design provides a more robust assurance of the swap property through multiple adversarial swap attacks, which is shown in the ablation studies in Appendices I.2 and I.3.

3.1.2 Dependency Regularization Loss

As discussed in Section 2.2, pursuing the swap property at the sample level often leads to severe overfitting of ˜ X θ , i.e., pushing ˜ X θ towards X , which results in high collinearity in feature selection. To address this, the DRL is introduced to reduce the reconstructability between X and ˜ X :

where λ 3 is a hyperparameter. The SWC term in Eq. (8) refers to the sliced Wasserstein correlation [33], which quantitatively measures the dependency between two random vectors in the same space. More specifically, let Z 1 and Z 2 be two p -dimensional random vectors. SWC p Z 1 , Z 2 q ' 0 indicates that Z 1 and Z 2 are independent, while SWC p Z 1 , Z 2 q ' 1 suggests a linear relationship between each other (see Appendix D for more details on SWC). In DeepDRK, we minimize SWC to reduce the reconstructability, a procedure similar to [54]. The intuition is as follows. If the joint distribution of X is known, then for each j P r p s , the knockoff ˜ X j should be sampled from P j p¨| X ´ j q , making X j and ˜ X j less dependent. In such case the swap property is ensured, and collinearity/reconstructability is reduced due to independence. As we do not have access to the joint law, we want the variables to be less dependent. Since collinearity exists with in X , merely decorrelate X j and ˜ X j is not enough. Thus, we minimize SWC to reduce the dependence between X and ˜ X . We refer readers to Appendix A for more discussions.

3.2 Dependency Regularization Perturbation

Empirically we observe a competition between L SL and L DRLin Eq. (5), which adds difficulty to the training procedure. Specifically, the L SL is dominating and the L DRL increases quickly after a short decreasing period. We are the first to observe this phenomenon in all deep-learning based knockoff generation models when one tries to gain power [47, 55, 38, 27]. We include the experimental evidence in Appendix E. We suggest the following explanation: minimizing the swap loss, which corresponds to FDR control, is the same as controlling Type I error. Similarly, minimizing the dependency loss is to control Type II error. With a fixed number of observations, it is well known that Type I error and Type II error can not decrease at the same time after reaching a certain threshold. In the framework of model-X knockoff, we aim to boost as much power as possible given the FDR is controlled at a certain level, a similar idea as the uniformly most powerful (UMP) test [12]. For this reason, we propose DRP as a post-training technique to further boost power.

DRP is a sample-level perturbation that further eliminates dependency between X and and the knockoff. More specifically, DRP perturbs the generated ˜ X θ with the row-permuted version of X , denoted as X rp. After applying DRP, the final knockoff ˜ X DRP θ,n becomes:

where α n is a preset perturbation weight, n is the sample size, and α n Ñ 0 when n Ñ 8 . In the following, we remove n or θ whenever the context is clear. ˜ X DRP has a smaller SWC with X , since X rp is independent of X . Despite the perturbation increases the swap loss, such impact is negligible when the sample size is large. More specifically, we present the following Lemma 3.1 and Proposition 3.2. Let ˆ P n and ˆ P B n denote the empirical joint distribution of the sample X and ˜ X before and after the swap: p X, ˜ X q ' ˆ P n , p X, ˜ X q swap p B q ' ˆ P B n . And P and P B denote their corresponding population distributions.

Lemma 3.1. Under mild conditions[41], the slice Wasserstein distance between the empirical distributions of p X, ˜ X q and p X, ˜ X q swap p B q and their corresponding population distributions is of scale O p n ´ 1 { 2 q , i.e., SWD p ˆ P n , ˆ P B n q ' SWD p P, P B q ` O p n ´ 1 { 2 q .

0.2

0.4

0.6

0.8

1.0

Power

1.00

0.88

0.95

0.86

0.94

0.82

0.65

0.78

0.44

0.70

0.99

0.35

0.70

0.17

0.60

1.00

0.88

0.94

0.78

0.93

0.99

0.30

0.66

0.59

Sample Size: 200

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

Datasets

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.74

0.29

0.25

0.40

0.26

0.11

0.11

0.08

0.12

0.09

0.44

0.07

0.09

0.05

0.09

0.72

0.29

0.24

0.31

0.26

0.79

0.13

0.27

0.04

0.32

0.2

0.4

0.6

0.8

1.0

Power

1.00

0.89

0.93

0.72

0.89

0.98

0.92

0.97

0.48

0.94

0.98

0.45

0.80

0.18

0.71

1.00

0.88

0.94

0.78

0.94

1.00

0.96

1.00

0.69

0.99

Sample Size: 2000

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

Datasets

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.63

0.11

0.05

0.14

0.05

0.07

0.08

0.04

0.09

0.05

0.11

0.03

0.04

0.03

0.05

0.64

0.08

0.04

0.16

0.05

0.70

0.12

0.11

0.09

0.12

Methods

DeepDRK

KnockoffGAN

sRMMD

DDLK

The proofs can be found in Appendix F. Lemma 3.1 suggests a general case where the swap loss enforced at the sample level differs from that of the population level from a term of scale n ´ 1 { 2 , which vanishes when n Ñ8 . Proposition 3.2 provides the rationale on the proposal of the perturbation technique in Eq. (9), such that the DRP term is negligible asymptotically. Besides the theoretical justification, we also empirically show in Appendix G that the perturbation is beneficial.

4 Experiment

We assess the performance of DeepDRK against several benchmark models under three experimental setups: 1. a fully synthetic setup where both the input variables and the response variable follow predefined distributions; 2. a semi-synthetic setup, in which the input variables are derived from realworld datasets and the response variable is generated based on known relationships with the inputs; and 3. feature selection (FS) using a real-world dataset. The experiments are designed to encompass a range of datasets with varying p { n ratios and distributions of X , aiming to provide a comprehensive evaluation of model performance. The benchmark models are Deep Knockoff [47] 2 , DDLK [55] 3 , KnockoffGAN [27] 4 and sRMMD [38] 5 , with links of the code implementation listed in the footnote. The implementation details for the training, feature selection, and data preparation are available in Appendix H. In the following, we describe the datasets and the associated experimental results. We also consider the ablation study to illustrate the benefits obtained by having the following proposed terms: REx, L swapper, the multi-swapper setup, and the dependency regularization perturbation. Empirically, these terms help to improve power and control the FDR. Due to space limitation, we defer details for the ablation studies to Appendix I.2 and I.3. DeepDRK is implemented in PyTorch [45] and is accessible at: https://github.com/nowonder2000/DeepDRK .

4.1 The Synthetic Experiments

To properly evaluate the performance, we follow a well-designed experimental setup by [55, 38] to generate different datasets specified by p X,Y q . Here X P R p is the collection dependent variables that follows pre-defined distributions. Y P R is the response variable that is modeled as Y ' N p X T β, 1 q . The underlying true β is a p -dimensional vector, where each entry is drawn independently from the distribution p 15 ¨ ? n ¨ Rademacher(0.5).

0.2

0.4

0.6

0.8

1.0

Power

1.00

1.00

0.79

0.79

0.98

0.96

1.00

1.00

0.91

0.97

Sample Size: 200

0.7

0.8

base

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.73

0.75

0.15

0.21

0.42

0.46

0.75

0.75

0.73

0.78

0.2

0.4

0.6

0.8

1.0

Power

1.00

1.00

0.95

0.92

0.98

0.96

1.00

1.00

1.00

1.00

Sample Size: 2000

Methods

DeepDRK

KnockoffGAN

sRMMD

DDLK

0.7

0.8

base

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.68

0.71

0.12

0.19

0.16

0.25

0.63

0.67

0.74

0.77

Compared to the previous works [55, 38], which consider p ? n as the scaling factor for the Rademacher distribution, we reduce the magnitude of β by a factor of 15. This is because we find that in the original setup, the β scale is too large such that the feature selection enjoys high powers and low FDRs for all models. To compare the performance of the knockoff generation methods on various data, we consider the following distributions for X :

Gaussian mixture : We consider a Gaussian mixture model X ' ř 3 k ' 1 π k N p µ k , Σ k q , where π is the proportion of the k -th Gaussian with p π 1 , π 2 , π 3 q ' p 0 . 4 , 0 . 2 , 0 . 4 q . µ k P R p denotes the mean of the k -th Gaussian with µ k ' 1 p ¨ 20 ¨ p k ´ 1 q , where 1 p is the p -dimensional vector that has universal value 1 for all entries. Σ k P R p ˆ p is the covariance matrices whose p i, j q -th entry taking the value ρ | i ´ j | k , where ρ k ' ρ k ´ 0 . 1 base and ρ base ' 0 . 6 . Besides this experiment, we further perform two additional tests. The first one focuses on a mixture of Gaussians data of various ρ base to study the feature selection performance with highly correlated features. Namely, we consider additional ρ base P t 0 . 7 , 0 . 8 u . The second one explores the effect of p π 1 , π 2 , π 3 q to the feature selection performance. In this case, we uniformly draw 10 sets of p π 1 , π 2 , π 3 q , and evaluate the FS performance of all the models considered in this paper. The values of the mixture weights are presented in Table 4 in Appendix H.3.1.

Copulas [49]: We further use copula to model complex correlations within X . To the best of our knowledge, this is a first attempt to consider complex distributions other than the Gaussian mixture model in the knockoff framework. Specifically, we consider two copula families: Clayton, Joe with the consistent copula parameter of 2 in both cases. For each family, we consider two candidates for the marginal distributions: a uniform distribution (using the identity conversion function) and an exponential distribution with a rate of 1. We implement the copulas according to PyCop 6 .

We consider the following p n, p q setups: p 200 , 100 q and p 2000 , 100 q . This is in contrast to existing works, which consider only the p 2000 , 100 q as the smallest sample size setup [55, 38, 47]. Our goal is to demonstrate the consistent performance of DeepDRK across various p { n ratios, particularly when the sample size is small.

Results : Figure 2 compare FDRs and powers for all models across the datasets with two different setups for β . Figure 2 shows that DeepDRK consistently controls the false discovery rate (FDR) compared to other benchmark models across various data distributions and p { n ratios, with the exception of a few cases where the FDR exceeds the 0 . 1 threshold in the small sample size ( n ' 200 ) scenarios. Other models, though being able to reach higher power, comes at a cost of sacrificing FDR, which contradicts to the UMP philosophy (see Section 3.2). We also evaluate the performance on a mixture of Gaussians with increasing ρ base, indicating a higher correlation among the input variables. Note that the mixture of Gaussians example in Figure 2 has ρ base ' 0 . 6 . The results for ρ base P t 0 . 7 , 0 . 8 u are presented in Figure 3. Compared to other baseline models, DeepDRK maintains the lowest FDRs while achieving competitive powers across all ρ base values, highlighting its robustness to correlations in X . Overall, the results demonstrate the ability of DeepDRK in consistently performing FS with controlled FDR compared to other models across a range of different datasets, p { n ratios, and feature correlations in X . In addition, we present common statistics for the FDR and power on the mixture of Gaussians experiment with 10 different sets of p π 1 , π 2 , π 3 q in Table 1. It is clear that DeepDRK improves the robustness in maintaining FDR across various configurations of the Gaussian mixture models compared to existing approaches.

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

Mean

DeepDRK

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

sRMMD

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

KnockoffGAN

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

DDLK

Datasets

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

0.0

0.2

0.4

0.6

0.8

1.0

Power

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Joe & Exponential

Copula: Clayton & Gamma

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

Power

Gaussian Mixture

0.0

0.2

0.4

0.6

0.8

Copula: Clayton & Exponential

0.0

0.2

0.4

0.6

0.8

Copula: Joe & Exponential

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

Copula: Clayton & Gamma

Sample Size: 200

Sample Size: 2000

Method

sRMMD

DeepDRK

KnockoffGAN

DDLK

Scale

5

10

15

20

To understand why DeepDRK outperforms other baseline models, we consider measuring the distribution of the knockoff statistics, i.e., w j , for both nonnull and null features of X . Fan et al. [17] and Cand` es et al. [11] pointed out that a good knockoff requires the corresponding knockoff statistics to concentrate symmetrically around zero for the null features and to maintain high positive values for the nonnulls. However, theoretical analysis on the goodness of FDR or power requires access to the true knockoff ˜ X [17] to compare the distribution of w j 's with the ground truth, which is infeasible for non-Gaussian data. Nevertheless, we can still examine the distribution of the knockoff statistics as a surrogate to analyze model performance in terms of false discovery rate (FDR) or power, given the necessary properties of the knockoff statistics mentioned earlier.

Figure 4 shows the means and standard deviations of the empirical distributions of the knockoff statistics w j for both null and nonnull variables across different datasets and models. Clearly, compared to other benchmarks, DeepDRK maintains high values of w j for the nonnulls and relatively symmetric values around zero for the nulls. All other models experience positive shifts to the null statistics to some extent. Positive shifts in the null statistics lead to a degeneracy in performance because the threshold selection rule for false discovery rate (FDR) is based on the negative values of w j 's (see Eq. (3)). This has two negative impacts, one on the FS threshold and the other on its subsequent FS process. First, according to Eq. (3), the shift causes the chosen threshold to approach zero, as there are fewer null statistics remaining on the negative side, and those that do remain have smaller amplitudes. Subsequently, a lowered threshold leads to an increase in false positives, a phenomenon that becomes more pronounced with positive shifts in the null statistics. As shown in Figure 4, the w j values calculated using DeepDRK are centered around zero for the nulls, while exhibiting large positive values for the nonnulls. This aligns with the results in Figure 2, which demonstrate that DeepDRK effectively achieves good FDR control and high power. Due to space limit, we defer the results for n ' 2000 and the comparison of w j statistics for the Gaussian correlation setup to Appendix I.3.

To further verify the performance consistency of the proposed method, we include a comparison on different β scales. Specifically, we consider 4 different sets of β scales, i.e., p 5 ¨ ? n , p 10 ¨ ? n , p 15 ¨ ? n and p 20 ¨ ? n , for the previously considered data distributions. The results are summarized in Figure 5. It is observed that in all cases considered with various β scale, the proposed DeepDRK successfully

maintains relatively low FDRs with a higher power, compared to baseline methods. This phenomenon is especially pronounced with a low sample size ( n ' 200 ).

In addition to the above results, we provide the measurement of the swap property in Appendix I.1. The evaluation of model runtime is also included in Appendix I.4.

4.2 The Semi-synthetic Experiments

Power FDR Weconsider a semi-synthetic study with design X drawn from two real-world datasets and use X to simulate response Y . The first dataset contains single-cell RNA sequencing (scRNA-seq) data from 10 ˆ Genomics 7 . Each entry in X P R n ˆ p represents the observed gene expression of p genes in n cells. We refer readers to [23] and [1] for more background. Following the same preprocessing in [23], we obtain the final dataset X with n ' 10000 and p ' 100 8 . The preprocessing of X and the synthesis of Y (denoted as 'Linear' and 'Tanh' cases) are deferred to Appendix H.3.2.

0.2

0.4

0.6

0.8

1.0

0.14

0.73

0.81

Synthetic Rule: Tanh

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.00

0.08

0.00

0.00

0.14

Methods

DeepDRK

KnockoffGAN

sRMMD

DDLK

0.2

0.4

0.6

0.8

1.0

Power

0.73

0.90

0.86

0.70

0.83

Synthetic Rule: Linear

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.06

0.10

0.07

0.12

0.09

Methods

DeepDRK

KnockoffGAN

sRMMD

DDLK

The second publicly available dataset 9 is from a real case study entitled 'Longitudinal Metabolomics of the Human Microbiome in Inflammatory Bowel Disease (IBD)' [35]. The study seeks to identify important metabolites of two representative diseases of the inflammatory bowel disease (IBD): ulcerative colitis (UC) and Crohn's disease (CD). Specifically, we use the C18 Reverse-Phase Negative Mode dataset that has 546 samples and 91 metabolites. To mitigate the effects of missing values, we preprocess the dataset following a common procedure to remove metabolites that have over 20% missing values, resulting in 80 metabolites. We normalize the data matrix entry-wise to have zero mean and unit variance after a log transform and an imputation via the k -nearest neighbor algorithm following the same procedure in [38]. Finally, we synthesize the response Y with the real dataset of X via Y ' N p X T β, 1 q , where the entries of β drawn independently from one of the following three distributions: Unif(0, 1), N p 0 , 1 q , and Rademacher p 0 . 5 q , in three separate experiments.

Results : Figure 6 and 7 compare the feature selection performance on the RNA data and the IBD data respectively. In Figure 6, we observe that all but DDLK are bounded by the nominal 0.1 FDR threshold in the 'Tanh' case. However, KnockoffGAN and sRMMD have almost zero power. The power for Deep Knockoff is also very low compared to that of DeepDRK. Although DDLK provides high power, the associated FDR is not controlled by the threshold. In the 'Linear' case, almost all models have well controlled FDR, among which DeepDRK provides the highest power. Similar observations can be found in Figure 7. For the IBD data generated under the aforementioned synthesis rules, it is clear that all models except DDLK achieve well-controlled FDR. Apart from DDLK, DeepDRK consistently demonstrates the highest power. These results further underscore the potential of DeepDRK for real-world data applications.

4.3 A Case Study

Besides (semi-)synthetic setups, we carry out a case study with real data for both design X and response Y , in order to qualitatively evaluate the selection performance of DeepDRK. In this subsection, we use the IBD dataset [35] with the empirical response. The response variable Y is categorical: Y equals 1 if a given sample is associated with UC/CD and 0 otherwise. The covariates X is identical to the second semi-synthetic setup considered in Section 4.2. To properly evaluate results with no ground truth available, we search for the evidence of the IBD-associated metabolites using the existing literature. Specifically, we use three sources: 1. metabolites that are explicitly

0.2

0.4

0.6

0.8

1.0

Power

0.99

1.00

0.98

0.98

1.00

Synthetic Rule: Linear & Normal

IBD

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.02

0.06

0.05

0.02

0.43

0.2

0.4

0.6

0.8

1.0

Power

0.54

0.68

0.47

0.49

0.92

Synthetic Rule: Linear & Rade.

IBD

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.03

0.07

0.05

0.02

0.57

0.2

0.4

0.6

0.8

1.0

Power

0.92

0.97

0.90

0.89

0.99

Synthetic Rule: Linear & Uniform

IBD

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.04

0.10

0.05

0.04

0.24

Methods

DeepDRK

KnockoffGAN

sRMMD

DDLK

documented to have associations with IBD, UC, or CD in the PubChem database 10 ; 2. metabolites that are reported in the existing peer-reviewed publications; 3. metabolites that are reported in pre-prints. We identify 47 metabolites that are reported to have association with IBD. All referenced metabolites are included in Table 5 in Appendix H.3.3.

Our DeepDRK model training and knockoff generation are the same as before (see Table 3 in Appendix H.1). Likewise, to generate knockoff for the benchmark models, we follow their default setups. During the FS step, however, we use 0.2 as the FDR threshold instead of 0.1, and apply a different algorithm-DeepPINK [37] that is included in the knockpy 11 library-to generate w j . The values are subsequently used to identify metabolites. We choose DeepPINK over the previously considered ridge regression due to the nonlinear relationships between metabolites X and responses Y in this case study.

We compare the FS results with the 47 literature-supported metabolites and report the number of selections in Table 2. A detailed list of selected features for each model can be found in Table 9 in Appendix I.5. From Table 2, it is clear that, compared to the benchmark models, DeepDRK identifies the largest number of referenced metabolites while effectively limiting the number of metabolites not reported in existing literature (see Table 5 in Appendix H.3.3). The sRMMD model achieves the lowest false discovery rate, but this comes at the cost of missing a significant number of documented metabolites. Since there is no ground truth available, the results here should be viewed and analyzed qualitatively.

5 Conclusion

In this paper, we introduce DeepDRK, a deep learning-based knockoff generation pipeline consisting of two steps. First, it trains a Knockoff Transformer with multiple swappers to achieve the swap property while reducing reconstructability. In the post-training stage, a dependency-regularized perturbation is applied to further enhance power with controlled FDR. DeepDRK effectively balances FDR and power, which compete with each other at the sample level. To the best of our knowledge, this relationship has not been previously reported in the literature. Empirically, DeepDRK demonstrates the ability to maintain both controlled false discovery rates (FDR) and high power across various data distributions and different p { n ratios. Additionally, we provide insights into the distribution of knockoff statistics, which elucidate the reasons behind DeepDRK's consistently strong performance. The numerical results indicate that DeepDRK outperforms existing deep learning-based benchmark models. Experiments with real and semi-synthetic data further highlight the potential of DeepDRK for feature selection tasks involving non-Gaussian data.

Acknowledgments

This research was partially supported by Alfred P. Sloan foundation and NSF # 1934757.

References

Agarwal, D., Wang, J., and Zhang, N. R. Data denoising and post-denoising corrections in single cell RNA sequencing. Statistical Science , 35(1):112 - 128, 2020.

Ananthakrishnan, A. N., Luo, C., Yajnik, V., Khalili, H., Garber, J. J., Stevens, B. W., Cleland, T., and Xavier, R. J. Gut microbiome function predicts response to anti-integrin biologic therapy in inflammatory bowel diseases. Cell host & microbe , 21(5):603-610, 2017.

Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In ICML , pp. 214-223, 2017.

Barber, R. F. and Cand` es, E. J. Controlling the false discovery rate via knockoffs. The Annals of Statistics , 43(5):2055-2085, 2015.

Bauset, C., Gisbert-Ferr´ andiz, L., and Cos´ ın-Roger, J. Metabolomics as a promising resource identifying potential biomarkers for inflammatory bowel disease. Journal of Clinical Medicine , 10(4):622, 2021.

Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) , 57(1):289-300, 1995.

Bin Masud, S., Jenkins, C., Hussey, E., Elkin-Frankston, S., Mach, P., Dhummakupt, E., and Aeron, S. Utilizing machine learning with knockoff filtering to extract significant metabolites in Crohn's disease with a publicly available untargeted metabolomics dataset. Plos one , 16(7): e0255240, 2021.

Blaker, P. A., Arenas-Hernandez, M., Smith, M. A., Shobowale-Bakre, E. A., Fairbanks, L., Irving, P. M., Sanderson, J. D., and Marinaki, A. M. Mechanism of allopurinol induced TPMT inhibition. Biochemical pharmacology , 86(4):539-547, 2013.

Bonneel, N., Rabin, J., Peyr´ e, G., and Pfister, H. Sliced and Radon Wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision , 51:22-45, 2015.

Bonnotte, N. Unidimensional and evolution methods for optimal transportation . PhD thesis, Universit´ e Paris Sud-Paris XI; Scuola normale superiore (Pise, Italie), 2013.

Cand` es, E., Fan, Y ., Janson, L., and Lv, J. Panning for gold: 'Model-X' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80(3):551-577, 2018.

Casella, G. and Berger, R. L. Statistical inference . Cengage Learning, 2021.

Deshpande, I., Zhang, Z., and Schwing, A. G. Generative modeling using the sliced Wasserstein distance. In CVPR , pp. 3483-3491, 2018.

Deshpande, I., Hu, Y.-T., Sun, R., Pyrros, A., Siddiqui, N., Koyejo, S., Zhao, Z., Forsyth, D., and Schwing, A. G. Max-sliced Wasserstein distance and its use for GANs. In CVPR , pp. 10648-10656, 2019.

Dhal, P. and Azad, C. A comprehensive survey on feature selection in the various fields of machine learning. Applied Intelligence , 52(4):4543-4581, 2022.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16 ˆ 16 words: Transformers for image recognition at scale. In ICLR , 2021.

Fan, Y., Gao, L., and Lv, J. ARK: Robust Knockoffs inference with coupling. arXiv preprint arXiv:2307.04400 , 2023.

Fretland, D., Widomski, D., Levin, S., and Gaginella, T. Colonic inflammation in the rabbit induced by phorbol-12-myristate-13-acetate. Inflammation , 14(2):143-150, 1990.

Gavrilov, Y., Benjamini, Y., and Sarkar, S. K. An adaptive step-down procedure with proven FDR control under independence. The Annals of Statistics , 37(2):619-629, 2009.

Gimenez, J. R., Ghorbani, A., and Zou, J. Knockoffs for the mass: new feature importance statistics with false discovery guarantees. In AISTATS , pp. 2125-2133, 2019.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Guyon, I. and Elisseeff, A. An introduction to variable and feature selection. Journal of machine learning research , 3(Mar):1157-1182, 2003.

Hansen, D., Manzo, B., and Regier, J. Normalizing flows for knockoff-free controlled feature selection. In Advances in Neural Information Processing Systems , volume 35, pp. 16125-16137, 2022.

He, Z., Liu, L., Wang, C., Le Guen, Y., Lee, J., Gogarten, S., Lu, F., Montgomery, S., Tang, H., Silverman, E. K., et al. Identification of putative causal loci in whole-genome sequencing data via knockoff statistics. Nature communications , 12(1):1-18, 2021.

Hristache, M., Juditsky, A., and Spokoiny, V. Direct estimation of the index coefficient in a single-index model. Annals of Statistics , 29(3):595-623, 2001.

Jang, E., Gu, S., and Poole, B. Categorical reparameterization with Gumbel-softmax. In ICLR , 2017.

Jordon, J., Yoon, J., and van der Schaar, M. KnockoffGAN: Generating knockoffs for feature selection using generative adversarial networks. In ICLR , 2018.

Kolouri, S., Nadjahi, K., Simsekli, U., Badeau, R., and Rohde, G. Generalized sliced Wasserstein distances. In NeurIPS , volume 32, 2019.

Koon, H. W. A novel orally active metabolite reverses Crohn's disease-associated intestinal fibrosis. Inflammatory Bowel Diseases , 28(Supplement 1):S61-S62, 2022.

Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A. Out-of-distribution generalization via risk extrapolation (REx). In ICML , pp. 5815-5826, 2021.

Lavelle, A. and Sokol, H. Gut microbiota-derived metabolites as key actors in inflammatory bowel disease. Nature reviews Gastroenterology & hepatology , 17(4):223-237, 2020.

Lee, T., Clavel, T., Smirnov, K., Schmidt, A., Lagkouvardos, I., Walker, A., Lucio, M., Michalke, B., Schmitt-Kopplin, P., Fedorak, R., et al. Oral versus intravenous iron replacement therapy distinctly alters the gut microbiota and metabolome in patients with IBD. Gut , 66(5):863-871, 2017.

Li, T., Yu, J., and Meng, C. Scalable model-free feature screening via sliced-Wasserstein dependency. Journal of Computational and Graphical Statistics , 32(4):1501-1511, 2023.

Liu, Y. and Zheng, C. Auto-encoding knockoff generator for FDR controlled variable selection. arXiv preprint arXiv:1809.10765 , 2018.

Lloyd-Price, J., Arze, C., Ananthakrishnan, A. N., Schirmer, M., Avila-Pacheco, J., Poon, T. W., Andrews, E., Ajami, N. J., Bonham, K. S., Brislawn, C. J., et al. Multi-omics of the gut microbial ecosystem in inflammatory bowel diseases. Nature , 569(7758):655-662, 2019.

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In ICLR , 2019.

Lu, Y., Fan, Y., Lv, J., and Stafford Noble, W. DeepPINK: reproducible feature selection in deep neural networks. In NeurIPS , volume 31, 2018.

Masud, S. B., Werenski, M., Murphy, J. M., and Aeron, S. Multivariate soft rank via entropyregularized optimal transport: Sample efficiency and generative modeling. Journal of Machine Learning Research , 24(160):1-65, 2023.

Mehta, R. S., Taylor, Z. L., Martin, L. J., Rosen, M. J., and Ramsey, L. B. SLCO1B1 *15 allele is associated with methotrexate-induced nausea in pediatric patients with inflammatory bowel disease. Clinical and translational science , 15(1):63-69, 2022.

Minderhoud, I. M., Oldenburg, B., Schipper, M. E. I., Ter Linde, J. J. M., and Samsom, M. Serotonin synthesis and uptake in symptomatic patients with Crohn's disease in remission. Clinical Gastroenterology and Hepatology , 5(6):714-720, 2007.

Nadjahi, K., Durmus, A., Simsekli, U., and Badeau, R. Asymptotic guarantees for learning generative models with the sliced-Wasserstein distance. In Advances in Neural Information Processing Systems , volume 32, 2019.

Narasimhan, R. L., Throm, A. A., Koshy, J. J., Saldanha, K. M. R., Chandranpillai, H., Lal, R. D., Kumravat, M., K. M., A. K., Batra, A., Zhong, F., et al. Inferring intestinal mucosal immune cell associated microbiome species and microbiota-derived metabolites in inflammatory bowel disease. bioRxiv , 2020.

Nies, T. G., Staudt, T., and Munk, A. Transport dependency: Optimal transport based dependency measures. arXiv preprint arXiv:2105.02073 , 2021.

Nuzzo, A., Saha, S., Berg, E., Jayawickreme, C., Tocker, J., and Brown, J. R. Expanding the drug discovery space with predicted metabolite-target interactions. Communications biology , 4 (1):1-11, 2021.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , volume 32, 2019.

Qin, X. Etiology of inflammatory bowel disease: a unified hypothesis. World journal of gastroenterology: WJG , 18(15):1708, 2012.

Romano, Y., Sesia, M., and Cand` es, E. Deep knockoffs. Journal of the American Statistical Association , 115(532):1861-1872, 2020.

Saber, S., Khalil, R. M., Abdo, W. S., Nassif, D., and El-Ahwany, E. Olmesartan ameliorates chemically-induced ulcerative colitis in rats via modulating NF κ B and Nrf-2/HO-1 signaling crosstalk. Toxicology and applied pharmacology , 364:120-132, 2019.

Schmidt, T. Coping with copulas. Copulas-From theory to application in finance , 3:1-34, 2007.

Scoville, E. A., Allaman, M. M., Brown, C. T., Motley, A. K., Horst, S. N., Williams, C. S., Koyama, T., Zhao, Z., Adams, D. W., Beaulieu, D. B., et al. Alterations in lipid, amino acid, and energy metabolism distinguish Crohn's disease from ulcerative colitis and control subjects by serum metabolomic profiling. Metabolomics , 14(1):1-12, 2018.

Sesia, M., Sabatti, C., and Cand` es, E. J. Gene hunting with hidden Markov model knockoffs. Biometrika , 106(1):1-18, 08 2018.

Soderholm, J. D., Oman, H., Blomquist, L., Veen, J., Lindmark, T., and Olaison, G. Reversible increase in tight junction permeability to macromolecules in rat ileal mucosa in vitro by sodium caprate, a constituent of milk fat. Digestive diseases and sciences , 43(7):1547-1552, 1998.

S¨ oderholm, J. D., Olaison, G., Peterson, K. H., Franzen, L. E., Lindmark, T., Wir´ en, M., Tagesson, C., and Sj¨ odahl, R. Augmented increase in tight junction permeability by luminal stimuli in the non-inflamed ileum of Crohn's disease. Gut , 50(3):307-313, 2002.

Spector, A. and Janson, L. Powerful knockoffs via minimizing reconstructability. The Annals of Statistics , 50(1):252-276, 2022.

Sudarshan, M., Tansey, W., and Ranganath, R. Deep direct likelihood knockoffs. In NeurIPS , volume 33, 2020.

Suhre, K., Shin, S.-Y., Petersen, A.-K., Mohney, R. P., Meredith, D., W¨ agele, B., Altmaier, E., Deloukas, P., Erdmann, J., Grundberg, E., et al. Human metabolic individuality in biomedical and pharmaceutical research. Nature , 477(7362):54-60, 2011.

Uchiyama, K., Odahara, S., Nakamura, M., Koido, S., Katahira, K., Shiraishi, H., Ohkusa, T., Fujise, K., and Tajiri, H. The fatty acid profile of the erythrocyte membrane in initial-onset inflammatory bowel disease patients. Digestive diseases and sciences , 58(5):1235-1243, 2013.

Uko, V., Thangada, S., and Radhakrishnan, K. Liver disorders in inflammatory bowel disease. Gastroenterology research and practice , 2012(1):642923, 2012.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In NeurIPS , volume 30, 2017.

Villani, C. Optimal transport: old and new , volume 338. Springer, 2009.

Wiesel, J. C. Measuring association with Wasserstein distances. Bernoulli , 28(4):2816-2832, 2022.

Zhu, Z., Fan, Y., Kong, Y., Lv, J., and Sun, F. DeepLINK: Deep learning inference using knockoffs with applications to genomics. Proceedings of the National Academy of Sciences , 118(36):e2104683118, 2021.

Appendix

A Reconstructability and Selection Power

The concept of reconstructability was introduced by [54] as a population-level counterpart to what is commonly referred to as collinearity in linear regression. Under a Gaussian design where X ' N p 0 , Σ q , reconstructability is high if Σ is not of full rank, implying that there exists some j such that X j is almost surely a linear combination of X ´ j . More generally, if there exists more than one representation of the response Y using the explanatory variable X , we qualitatively consider the reconstructability to be high. High collinearity often impairs statistical power, and similarly, high reconstructability diminishes feature selection power. To illustrate this, we state a linear version of Theorem 2.3 from [54], originally formulated for a more general single-index model [25].

Theorem A.1. Let Y ' X J β J ` X ´ J β ´ J ` ε , where J Ă r p s and ε is centered Gaussian noise. Equivalently, this implies Y K K X J | X J β J , X ´ J . Suppose there exists a β ˚ J such that X J β J ' X J β ˚ J almost surely. Then, denoting Y ˚ ' X J β ˚ J ` X ´ J β ´ J ` ε , we have

Furthermore, in the knockoff framework [4], let w ' w pp X, ˜ X q , y q and w ˚ ' w pp X, ˜ X q , y ˚ q . Then, for all j P J ,

Eq. (11) implies a 'no free lunch' situation for selection power when there is exact reconstructability.

To address the reconstructability issue, [54] proposed two methods in Gaussian design. The first is the minimal variance-based reconstructability (MVR) knockoff, in which the knockoff ˜ X is sampled to minimize the loss

Note that this is equivalent to maximizing E ' Var ´ X j | X ´ j , ˜ X ¯ı for all j P r p s . Another approach is the maximum entropy (ME) knockoff, where ˜ X is sampled to maximize

Under Gaussian design, both optimizations have closed-form solutions. Since X is Gaussian, p X, ˜ X q must also be jointly Gaussian to satisfy the swap property. To optimize, one first calculates the covariance matrix using the SDP method in [4], yielding a diagonal matrix S . Then, both MVR and ME reduce to an optimization on S :

Although both methods show high power for feature selection, neither MVR nor ME can be directly extended to arbitrary distributions due to the intractability of conditional variance and likelihood.

In DeepDRK, we consider regularizing with a sliced-Wasserstein-based dependency correlation, which can be considered a stronger dependency regularization than entropy. A post-training perturbation is also applied to further reduce collinearity. However, the theoretical understanding of how these affect the swap property and power remains an open question.

B DeepDRK Model Architecture and Training Algorithm

B.1 Model Architecture

DeepDRK's knockoff Transformer (KT) model is based on the popular Vision Transformer (ViT) [16]. The primary difference is that the input dimension is 1D, not 2D, for X . We do not use patches as input; instead, we treat all entries of X to account for correlations between each pair of entries in the knockoff ˜ X . This structure is similar to the original Transformer [59]. Nonetheless, we retain other components from ViT, such as patch embedding, PreNorm, and 1D-positional encoding [16]. Since knockoff generation requires a distribution, we feed X and a uniformly distributed random variable Z with the same dimension as X to inject randomness. Specifically for the DeepDRK model, we use 8 attention heads, 8 layers, and a hidden dimension of 512 in the ViT model.

The swapper module, first introduced in DDLK [55], produces the index subset B for the adversarial swap attack. Optimizing knockoff against these adversarial swaps enforces the swap property. Specifically, the swapper consists of a matrix with shape 2 ˆ p (i.e., trainable model weights), where p is the dimension of X . This matrix controls the Gumbel-softmax distribution [26] for all p entries. Each entry is represented by a binary Gumbel-softmax random variable (i.e., it can only take values of 0 or 1). To generate the subset B , we sample b j from the corresponding j -th Gumbel-softmax random variable, and B is defined as t j P r p s ; b j ' 1 u . During optimization, we maximize Eq. (5) with respect to the weights ω i of the swapper S ω i , so that the sampled indices, with which the swap is applied, lead to a higher SWD in the objective in Eq. (5). Minimizing this objective with respect to ˜ X θ requires the knockoff to counteract the adversarial swaps, thereby enforcing the swap property. Compared to DDLK, the proposed DeepDRK utilizes multiple independent swappers (i.e., K ' 2 ). And we set the temperature for the Gumbel-softmax to be 0.2.

B.2 Training Algorithm

In Algorithm 1, we provide pseudo code for training the Knockoff Transformer and the swappers (i.e., the first stage shown in Figure 1).

Algorithm 1 DeepDRK Training

C From Wasserstein to Sliced Wasserstein Distance

The Wasserstein distance has gained popularity in both mathematics and machine learning due to its ability to compare different types of distributions [60] and its differentiability [3]. Here, we provide its definition. Let X,Y be two R d random vectors following distributions P X , P Y with finite p -th moments. The Wassersteinp distance between P X and P Y is:

where Γ p P X , P Y q denotes the set of all joint distributions such that their marginals are P X and P Y . When d ' 1 , the Wasserstein distance between two one-dimensional distributions can be written as:

where F X and F Y are the cumulative distribution functions (CDFs) of P X and P Y , respectively. Moreover, if p ' 1 , the Wasserstein distance further simplifies to

From the above, it is evident that the 1D Wasserstein distance is straightforward to compute, leading to the development of the sliced Wasserstein distance (SWD) [9]. To leverage this computational advantage in 1D, one first projects both distributions uniformly onto a 1D direction and computes the Wassersteinp distance between the two projected distributions. SWD is then calculated by taking the expectation over the random direction. Specifically, let µ P S d ´ 1 denote a projection direction, and the push-forward distribution [60] µ 7 P X denotes the law of µ T X . When µ is uniformly distributed on the d -dimensional sphere, the p -sliced Wasserstein distance between P X and P Y is given by:

Combining Eq. (18) and Eq. (16) yields:

Despite faster computation, the convergence of SWD has been shown to be equivalent to the convergence of the Wasserstein distance under mild conditions [10]. In practice, the expectation over µ is approximated by a finite summation over a number of projection directions chosen uniformly from S d ´ 1 .

D Sliced Wasserstein Correlation

The idea of metricizing independence is recently advanced using the Wasserstein distance [61, 43]. Given a joint distribution p X,Y q ' Γ XY and its marginal distributions X ' P X , Y ' P Y , the Wasserstein Dependency (WD) between X and Y is defined by WD p X,Y q ' W p p Γ XY , P X b P Y q . A trivial observation is that WD p X,Y q ' 0 implies that X and Y are independent. Due to the high computational cost of Wasserstein distance, sliced Wasserstein dependency (SWDep) [33] is developed using sliced Wasserstein distance (see Appendix C for SWD details). The SW dependency (SWDep) between X and Y is defined as SW p p Γ XY , P X b P Y q , and a zero SWDep indicates independence. Since the dependency metric is not bounded from above, sliced Wasserstein correlation

1

5

10

15

20

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Normalized Value

Swap Property v.s. Dependency Regularization

DeepDRK

(Swap)

DeepDRK

(Depend.)

KnockoffGAN(Depend.)

sRMMD(Swap)

sRMMD(Depend.)

DDLK(Swap)

DDLK(Depend.)

(SWC) is introduced to normalize SWDep. More specifically, the SWC between X and Y is defined as

where Γ X X and Γ Y Y are the joint distributions of p X,X q and p Y, Y q respectively. It is shown that 0 ď SWC p p X,Y q ď 1 and SWC p p X,Y q ' 1 when X has a linear relationship with Y [33].

In terms of computing SWC, we follow [33] and consider both laws of X and Y to be sums of 2 n Diracs, i.e., both variables are empirical distributions with data I full ' tp x i , y i qu 2 n i ' 1 . Define I ' tp x i , y i qu n i ' 1 and ˜ I ' tp ˜ x i , ˜ y i qu n i ' 1 , where p ˜ x i , ˜ y i q ' p x n ` i , y n ` i q . Because data is i.i.d., I and ˜ I are independent. We further introduce the following notation:

Then the empirical SWC can be computed by:

E Competing Losses

In Figure 8, we present scaled L SL and L DRL curves for each model considered. For better visualization, these curves are normalized to values between 0 and 1. We observe the first 20 epochs, as L DRL flattens out in later epochs without further decrease. The competition between losses is evident: as L SL decreases, L DRL increases, indicating difficulty in maintaining low reconstructability.

F Proofs

F.1 Proof of Lemma 3.1

Assuming all conditions in [41] hold, we prove this by applying inequalities on SWD p ˆ P n , ˆ P B n q and SWD p P, P B q :

0.0

0.2

0.4

0.6

0.8

1.0

n

0.04

0.06

0.08

0.10

0.12

0.14

0.0

0.2

0.4

0.6

0.8

1.0

n

0.2

0.4

0.6

0.8

1.0

Power

Dataset

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

where the first inequality is a direct application of the triangle inequality, and the second equality follows from Theorem 5 in [41]. Similarly, we obtain

Combining Eq. (23) and Eq. (24), we have

F.2 Proof of Proposition 3.2

From Eq. (9), we know ˜ X DRP θ,n a.s. Ý Ý Ñ ˜ X θ as n Ñ 8 . Combining this with Eq. (25), we obtain SWD p P, P B q ' SWD p ˆ P n , ˆ P B n q ' SWD p ˆ P n, DRP , ˆ P B n, DRP q as n Ñ8 .

G Effect of α n in ˜ X DRP θ

As discussed in Section 3.2, obtaining the swap property while maintaining low reconstructability at the sample level is empirically challenging. To address this, dependency regularization perturbation (DRP) is introduced. In this section, we evaluate the effect of α n in ˜ X DRP θ in Eq. (9) on feature selection performance. Results are summarized in Figure 9 for five synthetic datasets with the sample size n ' 2000 .

When α n is decreased, we observe an increase in power. However, the FDR exhibits a bowl-shaped pattern, consistent with the findings of [54]: introducing the permuted X rp reduces reconstructability, thereby increasing power. However, an overly dominant X rp compromises the swap property, resulting in higher FDRs. Based on our hyperparameter search and the results in Figure 9, we recommend choosing α n within the range of 0.4 to 0.5.

H Implementation Details

H.1 Training

To fit models, we first split datasets of X into training and validation sets with an 8:2 ratio. The training sets are used for model optimization, and the validation sets are used for early stopping based on the validation loss, with a patience period of 6. Since knockoffs are not unique [11], testing sets are not required. To evaluate DeepDRK's performance, we compare it with four state-of-the-art (SOTA) deep-learning-based knockoff generation models, focusing on non-parametric data. Specifically, we

include Deep Knockoff [47] 12 , DDLK [55] 13 , KnockoffGAN [27] 14 , and sRMMD [38] 15 . We use the recommended hyperparameter settings for each model, with the only difference being the number of training epochs, set to 200 for consistency across models.

We follow the model configuration in Table 3 to optimize DeepDRK. The architecture of the swappers S ω is based on [55]. Both the swappers and ˜ X θ are trained using the AdamW optimizer [36]. During training, we alternately optimize ˜ X θ and the swappers S ω , updating weights θ three times for each update of weights ω . This training scheme is similar to GAN training [21], though without discriminators. We apply early stopping to prevent overfitting. A pseudocode of the optimization is provided in Appendix B.2. In experiments, we set α n ' 0 . 5 universally as the dependency regularization coefficient due to its consistent performance 16 . A discussion on the effect of α n is provided in Appendix G.

Once trained, models generate the knockoff ˜ X given X data. The generated ˜ X is then combined with X for feature selection. Experiments are conducted on a single NVIDIA V100 16GB GPU.

H.2 Feature Selection

Once ˜ X is obtained, feature selection is performed in three steps. We first concatenate X and ˜ X to form an n ˆ 2 p design matrix p X, ˜ X q . In the second step, we use Ridge regression to get the estimated regression coefficients t ˆ β j u 2 p j ' 1 from Y and p X, ˜ X q . Finally, we compute the knockoff statistics w j ' | ˆ β j | ´ | ˆ β j ` p | , for j ' 1 , 2 , . . . , p , and then select features using the threshold defined in Eq. (3). We set q ' 0 . 1 as the FDR threshold, following its common use in other knockoff-based feature selection studies [47, 38]. Each experiment is repeated 600 times, with results reported as the average power and FDR over these 600 repetitions.

H.3 Data Preparation

H.3.1 Weights for the Gaussian mixture models

Table 4 includes 10 sets of p π 1 , π 2 , π 3 q for the Gaussian mixture models.

H.3.2 Preparation of the RNA Data

We first normalize the raw data X to value range r 0 , 1 s and then standardize it to have zero mean and unit variance. Y is synthesized according to X . We consider two different ways of synthesizing Y . The first, denoted as 'Linear', is similar to the previous setup in the full synthetic case with Y ' N p X T β, 1 q and β ' p 12 . 5 ¨ ? N ¨ Rademacher(0.5). For the second, denoted as 'Tanh', the response Y is

generated following the expression:

where ϵ follows the standard normal distribution and the 20 covariates are sampled uniformly.

H.3.3 Metabolite Information for the IBD Study

Here we provide the implementation details for the experiments described in Section 4.3. In Table 5, we provide all 47 referenced metabolites based on our comprehensive literature review.

I Additional Results

In this section, we include all results that are deferred from the main paper.

I.1 Swap Property

Weuse different metrics to empirically evaluate the swap property on the generated knockoff ˜ X and the original data X according to Eq. (1). In this paper, three metrics are considered: mean discrepancy distance with linear kernel, or 'MMD(Linear)' for

short; sliced Wasserstein-1 distance (SWD 1 ); and sliced Wasserstein-2 distance (SWD 2 ). We measure the sample level distances between the vector that concatenates X and ˜ X (i.e., p X, ˜ X q ) and the vector after randomly swapping the entries (i.e., p X, ˜ X q swap p B q ). To avoid repetition, please refer to section 2.1 and Eq. (1) for the definition of notation. Empirically, it is time consuming to evaluate all subsets B of the index set r p s . As a result, we alternatively define a swap ratio r s P t 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 u . The swap ratio controls the amount of uniformly sampled indices (i.e., the cardinality | B | ' r s ¨ p ) in a subset of r p s . For any X and ˜ X from the same experiment, 5 different subsets B are formed according to 5 different swap ratios. We report the average values over the swap ratios to represent the empirical quantification of the swap property. Results can be found in Table 7.

Clearly, compared to other models, the proposed DeepDRK achieves the smallest values in almost every entry across the three metrics and the first 4 datasets (i.e., J+G, C+G, C+E and J+E in Table 7). This explains why DeepDRK has lower FDRs as DeepDRK maintains the swap property relatively better than the baseline models (see results in Figure 2). Similarly, we observe that KnockoffGAN also achieves relatively small values, which leads to well-controlled FDRs compared to other baseline models. Overall, this verifies the argument in Cand` es et al. [11] that the swap property is important in guaranteeing FDR during feature selection.

However, we observe a difference for Gaussian mixture data. The proposed DeepDRK achieves the best performance in FDR control and power (see results in Figure 2), yet its swap property measured under the metrics in Table 7 is not the lowest. Despite this counter-intuitive observation, we want to highlight that it does not conflict with the argument in Cand` es et al. [11]. Rather, it supports our statement that the low reconstructability and the swap property cannot be achieved at the sample level (e.g., the free lunch dilemma in practice). After all, the swap property is not the only factor that determines FDR and power during feature selection.

0.2

0.4

0.6

0.8

1.0

Power

0.82

0.65

0.78

0.44

0.70

0.97

0.69

0.74

0.26

0.58

0.97

0.88

0.82

0.54

0.17

1.00

0.99

0.99

0.99

0.99

0.95

0.99

0.99

0.99

0.99

Sample Size: 200

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

Datasets

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.11

0.11

0.08

0.12

0.09

0.40

0.12

0.10

0.11

0.08

0.37

0.33

0.10

0.18

0.08

0.75

0.64

0.53

0.72

0.62

0.21

0.63

0.52

0.72

0.59

0.2

0.4

0.6

0.8

1.0

Power

0.98

0.92

0.97

0.48

0.94

0.89

0.98

0.19

0.66

0.69

0.98

0.95

0.11

0.65

0.89

0.99

1.00

1.00

1.00

1.00

0.99

1.00

1.00

1.00

1.00

Sample Size: 2000

Methods

DeepDRK

DeepDRK

No

swapper

No

REx

K

=1

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

Datasets

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.07

0.08

0.04

0.09

0.05

0.06

0.14

0.06

0.11

0.07

0.06

0.10

0.04

0.11

0.08

0.29

0.77

0.79

0.62

0.29

0.64

0.79

Null

0.05

0.00

0.05

0.10

0.15

0.20

Mean

DeepDRK

Null

0.05

0.00

0.05

0.10

0.15

0.20

sRMMD

Null

0.05

0.00

0.05

0.10

0.15

0.20

Null

0.05

0.00

0.05

0.10

0.15

0.20

KnockoffGAN

Null

0.05

0.00

0.05

0.10

0.15

0.20

DDLK

Datasets

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

I.2 Ablation Studies

In this subsection, we perform ablation studies on different terms introduced in Section 3.1.1, to show the necessity of designing these terms during the optimization for knockoffs. We consider the fully synthetic setup described in Section 4.1, and consider n ' 200 and n ' 2000 . The distribution of β is p ¨ Rademacher(0.5). We test the following terms: 1. REx; 2. the number of swappers K ; 3.

L swapper; 4. the dependency regularized perturbation (denoted as DeepDRK ). Five synthetic datasets are considered as before and we report the values for FDR and Power under each setup. The results are presented in Figure 10.

15 ¨ ? N :

Figure 10 illustrates the clear drawbacks of using only a single swapper ( K ' 1 ) and the case without the loss term REx. In those two cases, we fail to control the FDR on all datasets. The REx term is essential, as it ensures that the adversarial attacks generated by different swappers are addressed simultaneously. L swapper is also an important term, as it promotes a diverse adversarial environments. Without this term we observe an increase in FDR compared to DeepDRK or DeepDRK : . The difference between DeepDRK and DeepDRK : is more subtle as DeepDRK : has already obtained high quality knockoffs. However, we empirically observe that adding the perturbation further increases the power for some datasets without sacrificing the FDR controllability. To interpret these observations, we follow a similar procedure in Section 4.1 to study the distribution of the knockoff statistics. Results are included in Section I.3.

Overall, we verify that all terms are necessary components to achieve higher powers and controlled FDRs through the ablation studies.

I.3 Analysis on the distribution of knockoff statistics

We present the additional results on the means and standard deviations of both null and nonnull knockoff statistics under various experimental setups.

Figure 11 shows the results on w j for n ' 2000 and it complements the results on Power and FDR in Figure 2. We notice that all models have concentrated null features around zero and relatively high values for the nonnull knockoff statistics for the case with 2000 samples. This corresponds to consistently accurate FS results across all models and datasets, as shown in Figure 2.

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

Mean

DeepDRK

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

sRMMD

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

KnockoffGAN

Null

0.1

0.0

0.1

0.2

0.3

0.4

0.5

DDLK

base

0.7

0.8

Null

0.05

0.00

0.05

0.10

0.15

0.20

Mean

DeepDRK

Null

0.05

0.00

0.05

0.10

0.15

0.20

sRMMD

Null

0.05

0.00

0.05

0.10

0.15

0.20

Null

0.05

0.00

0.05

0.10

0.15

0.20

KnockoffGAN

Null

0.05

0.00

0.05

0.10

0.15

0.20

DDLK

base

0.7

0.8

Figure 12 and Figure 13 show the results for the Gaussian mixture data with increased correlations ( ρ base ' 0 . 7 and 0 . 8 ) for n ' 200 and n ' 2000 respectively. The figures complement the FDRpower results in Figure 3. It is clear for both n ' 200 and n ' 2000 cases, all models experience an increase in FDR and a decrease in power. This phenomenon can be reflected by the positive shifts of the knockoff statistics for the null features in Figure 12 and Figure 13. However, DeepDRK significantly controls the shifts, achieving the best results with the lowest FDRs and comparable power as shown in Figure 3.

In addition, we also compare the knockoff statistics for the models considered in the ablation studies in I.2. The results for both n ' 200 and n ' 2000 are in Figure 14 and Figure 15. Although the null knockoff statistics for DeepDRK, DeepDRK : and 'No L swapper' models concentrate symmetrically around zero, the heights of the nonnull knockoff statistics for DeepDRK are the highest, resulting higher power. And because some nonnull knockoff statistics have small values for the DeepDRK : and 'No L swapper' cases, we also expect them to have higher FDRs (see Figure 10). On the other hand, compared to DeepDRK, the models of 'No REx' and ' K ' 1 ' experience clear positive shifts for the null knockoff statistics, leading to higher FDRs during FS (see Figure 10).

I.4 Training time

We consider evaluating and comparing model training runtime in Table 8 with the (2000, 100) setup, as it is common in the existing literature. Although DeepDRK is not the fastest among the compared models, the time cost-7.35 minutes-is still short, especially when the performance is taken into account.

Null

0.2

0.0

0.2

0.4

0.6

Mean

DeepDRK

Null

0.2

0.0

0.2

0.4

0.6

DeepDRK

Null

0.2

0.0

0.2

0.4

0.6

No

swapper

Null

0.2

0.0

0.2

0.4

0.6

No

REx

Null

0.2

0.0

0.2

0.4

0.6

K

=1

Datasets

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

Null

0.2

0.0

0.2

0.4

0.6

Mean

DeepDRK

Null

0.2

0.0

0.2

0.4

0.6

DeepDRK

Null

0.2

0.0

0.2

0.4

0.6

No

swapper

Null

0.2

0.0

0.2

0.4

0.6

No

REx

Null

0.2

0.0

0.2

0.4

0.6

K

=1

Datasets

Gaussian Mixture

Copula: Clayton & Exponential

Copula: Clayton & Gamma

Copula: Joe & Exponential

I.5 Additional Results for the IBD Study

Here we provide the supplementary information for the experimental results described in Section 4.3. In Table 9, we provide the list of identified metabolites by each of the considered models. This table provides additional information for Table 2 in the main paper which only includes metabolite counts due to limited space.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While '[Yes] ' is generally preferable to '[No] ', it is perfectly acceptable to answer '[No] ' provided a proper justification is given (e.g., 'error bars are not reported because it would be too computationally expensive' or 'we were unable to find the license for the dataset we used'). In general, answering '[No] ' or '[NA] ' is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist' ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly describe the scope and the objective of the paper in the abstract and the introduction sections (also in the related work section).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation is described in the footnote 15 on page 20.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to the method section and the appendix for the complete proof. We provide assumptions in the method section.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We run the experiments multiple times for reproducibility check.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open code access.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: They are included in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Results shown in the experiment section are repeated for 600 times.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are performed with a single NVIDIA V100 GPU. It is described in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: N/A

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper focuses purely on methodology. How to use this method, and to serve what purpose is completely dependent on people who use it (see the conclusion section).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: It is an unreleated question to this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The code is open-sourced on GitHub, with necessary licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Code is the only asset. Since it is associated with the method (see the method section), the answer is yes.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: It is unrelated.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: It is unrelated.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a deep-learning-based method for performing \"knockoff\"-based feature selection. The main contribution is the proposal of a new optimization problem, which incorporates a four part loss.  The method is evaluated on synthetic, semi-synthetic, and real-world data, showing comparable and, at times, improved performance over relevant baselines.\n\nSTRENGTHS:\n**Experiments.** At times, this paper shows that the proposed method improves the power for a fixed FDR.\n\nWEAKNESSES:\n**Motivation.** It's not clear why this problem is of interest. It is true that many papers have been published on this topic. But the authors motivate the problem not by describing why knockoffs are particularly important; they simply say that finding information features is \"mission impossible,\" which doesn't give the reader any idea of why the problem is relevant.\n\n**Organization.** To articulate a list of contributions, a paper must clearly answer two fundamental questions: (1) \"What is the problem under consideration?\" and (2) \"How does the proposed method address this problem?\" Unfortunately, it's not clear that this paper answers the first or second questions.  Here are some comments:\n1. The authors claim that their method reduces reconstructability.  However, it's not clear what \"reconstructability\" is, or how it would be formally defined. The idea of reconstructability only arises in the discussion of related work, and even here, the problem is not presented in a way that is distinct from the solution presented in [4].  \n2. Section 2.1 is confusing. More care should be taken w/r/t the introduction of the problem of generating a knockoff sample.  The authors adopt quite a bit of notation from [11].  However, whereas [11] presents the method with examples and clearly articulated goals, this paper's summary is so coarse/brief that any reader who has not read this past work will have little to no chance of understanding the problem of finding knockoffs. \n3. Deferring the definition of FDR to the appendix doesn't make sense to me. This quantity is of fundamental interest; it is measured throughout the experiments, and theorems are presented (in part from past work) which rely on this definition. The paper is incomplete without including this definition in the main text.\n4. Theorem 2.1 is not clearly stated. The \"aforementioned property\" is not inferable from context.  The authors should make it clear that (1) they are referring to Theorem 1 in [11] and (2) that the \"aforementioned property\" is the choice of threshold in (3).\n5. Readability would be improved if the authors included an example of the swap function, such as the example in [11] after Def. 2.\n6. The authors seem to use (X,Z) and [X,Z] freely and interchangeably. This should be clarified. [11] sticks with (X,Z) (rather than square brackets) and I would recommend that the authors do the same.\n7. Make it abundantly clear that whenever the authors say \"power,\" they mean low type II error. This may not be clear to readers coming from deep learning.\n\n**Main method.** A few comments on the main method.\n1. The results are presented in a confusing order. (4) is presented before the authors know what any of the pieces are. Readers must wait around a page to get the details, which will frustrate. Furthermore, it's not clear why (4) is a fundamentally \"good\" objective, or where it came from. As written, it feels like the authors pulled this problem out of a hat.  Why does posing this as a minimax problem (as opposed to, e.g., a constrained problem) make sense? Why do the authors choose these two loss terms (and not others)? And how should one choose all of the trade-off parameters?\n2. The authors seem to imply that [56] has a problem. It's not clear what this problem is; the \"is far from enough\" sentence doesn't give any quantitative evidence as far as I can tell. The result of this is that when the authors say \"To fix it, ...,\" it's unclear what problem they are fixing.\n3. The authors borrow a loss term from the risk extrapolation (REx) paper from domain generalization. The authors should justify this choice, as well as the other loss functions that they append onto their objective. As written, it feels as if the authors are somewhat arbitrarily adding on loss functions, and there isn't a clear principle driving the decision making. This gives the paper a largely heuristic feel. And while the authors claim to provide ablations in the appendix, it's not clear where they are. The authors say that the results are \"in the associated tables,\" but it's not clear what tables these are. A similar concern arises w/r/t Figure 9.\n4. The trade-off between L_SL and L_DRL isn't clearly presented. This content should be presented in the main text; based on the figure in Appendix G, it's not clear that the trade-off always holds. Furthermore, even if it did hold, the contribution that the authors are \"the first to observe this phenomenon\" seems to be trivially true: If no one has used this loss before, then of course no one has seen this trade-off.\n\n**Experiments.** Some comments on the experiments.  The experiments seem to show marginal improvements for the proposed method. While DeepDRK does at times achieve higher power for a fixed FDR, the results are often fairly marginal, e.g., in Figure 5.  Given the shortcomings in the presentation of the method, it's not clear that these results do enough to justify acceptance.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}

comments: SUMMARY:
This paper considers the construction of knockoff features 
in the variable selection framework of model-X knockoffs.
The authors propose a deep learning-based method for 
generating knockoff features. Extensive numerical simulation and real-data examples show 
that the proposed method has advantageous performance compared
with other state-of-art methods, especially when the covariate distribution 
is complex and when the sample size is relatively small.

STRENGTHS:
The paper considers one fundamental problem in the model-X knockoff 
pipeline --- the construction of valid and high-quality knockoffs. 
This is key to the applicability of the knockoffs method.


Substantial engineering efforts have been made in this work and 
the proposed method is demonstrated to have satisfactory performance 
in extensive numerical experiments. This is a significant contribution to the 
knockoffs literature.

WEAKNESSES:
The proposed method is based mostly on empirics and heuristics, 
with a theoretical understanding lacking.

QUESTIONS:
1. The introduction of the dependency regularized perturbation is 
for power boosting, and yet in the ablation study, this feature seems to help 
more with FDR control as opposed to power-boosting. Is there a reason?
2. On page 4, line 132, the definition of REx: I am confused since 
the LHS does not depend on $i$ but the RHS depends on $i$. 
3. On page 4, line 148: "correltaion" -> "correlation"
4. On page 5, line 153: I am confused by the statement that "the knockoff 
 $X_j$ should be independently sampled from $p_j(\cdot \mid X_{-j})$"
--- this alone is not enough to ensure the knockoff property.
5.  On page 5, line 173, what is $\alpha$?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work introduces DeepDRK, a new algorithm for improving FDR in the model-X knockoff framework. In this framework, a knockoff covariate is generated for each existing covariate where then knockoff covariates have to satisfy the swap property. Given a knockoff statistic that satisfies the flip-sign property, one can perform feature selection with guaranteed false discovery rate by training a regression/classification model on the concatenated vector of original and knockoff covariates and then using any arbitrary feature importance score. Therefore, the main challenge in this framework is generating the knockoffs such that they satisfy the strong swap property: swapping any arbitrary subset of covariates with their knockoffs should not change the distribution of the concatenated vector. \n\nGiven that most real-world data does not follow Gaussian or a mixture of Gaussian distribution, generating knockoffs that satisfy the swap property is a challenge. Existing work is focused on using deep generative models (e.g. GANs) to handle this challenge. The paper argues that this approach for continuous covariates will result in reconstructability: overfitting on the data will make the distribution of knockoffs too similar to the data distribution and therefore making the concatenated vector too multicollinear for any feature-importance method to have discovery power.\n\nThe DeepDRK framework seeks to tackle reconstructability by training the deep model in such a way that not only satisfies the swap property but also reduces reconstructability. This is achieved in two steps:\nFirstly, by having a 1- the adversarial swap loss where a group of models compete adversarially against the knock-off generator model 2- a dependence regularization loss to reduce reconstructability by directly reducing the sliced-Wasserstein correlation between the original and knockoff vectors.\nSecondly, by the post-training dependency regularization which interpolates between the generated knockoff data table and the row permuted original data table. By making the interpolation weaker or stronger, one gains control in how much they can improve how much swap condition holds and therefore how good the FDR is at the cost of loosing power and vice versa.\nThe paper then shows experimental evidence in synthetic and semi-synthetic scenarios (where the true covariate-response) relationship is known. The argument is that compared to existing algorithms that have better power in low-sample settings, DeepDRK's empirical FDR is close the the selected level more consistently and this is proven both by the FDR vs Power results and the original vs knockoff feature importance distributions.\n\nSTRENGTHS:\nAll in all, the main novelties in the paper are:\n- main one is the post-training Dependency Regularization Perturbation\n- using multi swappers and SWD criterion\n- Experimenting on synthetic distributions other than GMMs\n- Showing that unlike other deep learning based knockoff algorithms, DeepDRK has more consistent FDR preservation. The ablation study really helps with understanding the benefit of each of the algorithmic choices.\n\nWEAKNESSES:\n- It seems necessary to see 4.1 results for various beta scales used for generating the response variable to compare different methods in various levels of difficulties to make sure the value of 15 has not been selected to favor DeepDRK. I think it'd be interesting to have the results in a 2D FDR vs Power scatter plot for better comparison.\n- I'd be interested in seeing mean + confidence interval performance plots for Figure 2 and Figure 3 using various distribution parameter selections (e.g. for GMM distribution using various \\pi parameters in addition to to more \\rho_base, etc)\n- The process for choosing the value of alpha=0.5 needs to be clarified as \"it leads to consistent performance\" might suggest choice based on final results which would be a case of overfitting. The discussion related to Figure 8 seems to confirm overfitting.\n\nQUESTIONS:\nDiscussed above\n\nLIMITATIONS:\nThere needs to be a discussion regarding the choice of hyperparameters especially the alpha parameter as in real-world applications of knockoffs it's impossible to know the best performing value in advance. Given the results in Figure 8, there seems to be high sensitivity to the alpha value which would make the framework unreliable. At the very least, the authors need to show the superiority of their framework to existing methods in a large interval of alpha values.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-29"}