DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:14:38

Abstract: Model-X knockoff has garnered significant attention among various feature selection methods due to its guarantees for controlling the false discovery rate (FDR). Since its introduction in parametric design, knockoff techniques have evolved to handle arbitrary data distributions using deep learning-based generative models. However, we have observed limitations in the current implementations of the deep Model-X knockoff framework. Notably, the "swap property" that knockoffs require often faces challenges at the sample level, resulting in diminished selection power. To address these issues, we develop "Deep Dependency Regularized Knockoff (DeepDRK)," a distribution-free deep learning method that effectively balances FDR and power. In DeepDRK, we introduce a novel formulation of the knockoff model as a learning problem under multi-source adversarial attacks. By employing an innovative perturbation technique, we achieve lower FDR and higher power. Our model outperforms existing benchmarks across synthetic, semi-synthetic, and real-world datasets, particularly when sample sizes are small and data distributions are non-Gaussian.

Main paper: 
 introduction: 
 Feature selection (FS) has drawn tremendous attention over the past decades, due to rapidly increasing data dimension (Guyon & Elisseeff, 2003) . Still, perfectly locating informative features is deemed mission impossible (Sudarshan et al., 2020) . It is hence necessary to devise algorithms to select features with controlled error rates.Targeting this goal, Model-X knockoffs, a novel framework, is proposed in Barber & Candès (2015) ; Candes et al. (2018) to select relevant features while controlling the false discovery rate (FDR). In contrast to the classical setup, where assumptions on the correlations between input features and the response are imposed (Benjamini & Hochberg, 1995; Gavrilov et al., 2009) , the Model-X knockoff framework only requires a linear relationship between the response and the features. With a strong finite-sample FDR guarantee, Model-X knockoff saw broad applications in domains such as biology, neuroscience, and medicine, where the size of data is limited (Sudarshan et al., 2020) .There have been considerable developments of knockoffs since its debut. In scenarios where feature distributions are complex, various deep learning methods have been proposed. However, we observe major limitations despite improved performance. First, the performances of existing methods vary across different data distributions. Second, selection quality deteriorates when observation number is relatively small. Third, the training is often difficult, due to competing losses in the training objective. We elaborate on the drawbacks in sections 2.2 and 3.2.In this paper, we remedy the issues by proposing the Deep Dependency Regularized Knockoff (DeepDRK), a deep learning-based pipeline that utilizes the Vision Transformer (Dosovitskiy et al., 2020) as the backbone model to generate knockoffs for feature selection. DeepDRK is designed to achieve the so-called "swap property" (Barber & Candès, 2015) and to reduce "recontructabililty" (Spector & Janson, 2022) , which in turn controls FDR and boosts selection power. We propose a "multi-swapper" adversarial training procedure to enforce the swap property, while a sliced-Wasserstein-based (Li et al., 2023) dependency regularization together with a novel perturbation technique are introduced to reduce reconstructability. We then carry out experiments on real, synthetic, and semi-synthetic datasets to show that our pipeline achieves better performance in different scenarios comparing to existing ones. 
background and related works: 
 2.1 MODEL-X KNOCKOFFS FOR FDR CONTROL The Model-X knockoffs framework consists of two main components. Given the explanatory variables X " pX 1 , X 2 , . . . , X p q J P R p and the response variable Y (Y continuous for regression and categorical for classification), the framework requires: 1. a knockoff X " p X1 , X2 , . . . , Xp q J that "fakes" X; 2. the knockoff statistics W j for j P rps that assess the importance of each feature X j . The knockoff X is required to be independent of Y conditioning on X, and must satisfy the swap property: pX, Xq swappBq d "pX, Xq, @B Ă rps.(1)Here swappBq exchanges the positions of any variable X j , j P B, with its knockoff Xj . The knockoff statistic W j " w j prX, Xs, Y q for j P rps, must satisfy the flip-sign property:EQUATIONThe functions w j p¨q, j P rps have many candidates, for example w j " |β j | ´| βj |, where β j and βj are the corresponding regression coefficient of X j and Xj with the regression function Y " pX, Xq.When the two knockoff conditions (i.e. Eq. ( 1) and ( 2)) are met, one can select features by S " tW j ě τ q u, where τ q " min tą0 ! t : 1 `|tj : W j ď ´tu| maxp1, |tj : W j ě tu|q ď qEQUATIONTo assess the feature selection quality, FDR (see Appendix A for precise definition) is commonly used as an average Type I error of selected features (Barber & Candès, 2015) . The control of FDR is guaranteed by the following theorem from Candes et al. (2018) :Theorem 1 Given knockoff copy and knockoff statistic satisfying the aforementioned property, and S " tW j ě τ q u, we have FDR ď q.Model-X knockoff is first studied under Gaussian design. Namely, the original variable X " N pµ, Σq with µ and Σ known. Since Gaussian design does not naturally generalize to complex data distributions, a number of methods are proposed to weaken the assumption. Among them, model-specific ones such as AEknockoff (Liu & Zheng, 2018) Hidden Markov Model (HMM) knockoff (Sesia et al., 2017), and MASS (Gimenez et al., 2019) all propose parametric alternatives to Gaussian design. These methods are able to better learn the data distribution, while keeping the sampling process relatively simple. Nevertheless, they pose assumptions to the design distribution, which can be problematic if actual data does not coincide. To gain further flexibility, various deeplearning-based models are developed to generate knockoffs from distributions beyond parametric setup. DDLK (Sudarshan et al., 2020) and sRMMD (Masud et al., 2021) utilize different metrics to measure the distances between the original and the knockoff covariates. They apply different regularization terms to impose the "swap property". He et al. (2021) proposes a KnockoffScreen procedure to generate multiple knockoffs to improve the stability by minimizing the variance during knockoff construction. KnockoffGAN (Jordon et al., 2018) and Deep Knockoff (Romano et al., 2020) take advantage of the deep learning structures to create likelihood-free generative models for the knockoff generation.Despite flexibility to learn the data distribution, deep-learning-based models suffer from major drawbacks. Knockoff generations based on distribution-free sampling methods such as generative adversarial networks (GAN) (Goodfellow et al., 2020; Arjovsky et al., 2017) tend to overfit, namely to learn the data X exactly. The reason is that the notion of swap property for continuous distributions is ill-defined on sample level. To satisfy swap property, one needs to independently sample Xj from the conditional law P Xj p¨|X ´j q, where X ´j denotes the vector pX 1 , . . . , X j´1 , X j`1 , . . . , X p q. On sample level, each realization of X ´j " x i ´j is almost surely different and only associates to one corresponding sample X j " x i j , causing the conditional law to degenerate to sum of Diracs. As a result, any effort to minimize the distance between pX, Xq and pX, Xq swappBq will push X towards X and introduce high collinearity that makes the feature selection powerless. To tackle the issue, DDLK (Sudarshan et al., 2020) suggests an entropic regularization. Yet it still lacks power and is computationally expensive.The issue of lacking power in the knockoff selection is solved in the Gaussian case. Assuming the knowledge of both mean and covariance of X " N pµ, Σq, the swap property is easily satisfied by setting Xj " N pµ j , Σ jj q and Σ ij " VarpX i , Xj q, for i ‰ j, i, j P rps. Barber & Candès (2015) originally propose to minimize VarpX j , Xj q for all j P rps using semi-definite programming (SDP), to prevent Xj to be highly correlated with X j . However, Spector & Janson (2022) observed that the SDP knockoff still lacks feature selection power, as merely decorrelating X j and Xj is not enough, and pX, Xq can still be (almost) linearly dependent in various cases. This is referred to as high reconstructability in their paper, which can be considered as a population counterpart of collinearity (see Appendix B for more details). To tackle the problem, Spector & Janson (2022) proposed to maximize the expected conditional variance EVarpX j | X ´j , Xq, which admits close-form solution whenever X is Gaussian. 
method: 
 DeepDRK provides a novel way to generate knockoffs X while reducing the reconstructability (see Section 2.3) between the generated knockoff X and the input X for data with complex distributions. Note that in this section we slightly abuse the notation such that X and X denote the corresponding data matrices. The generated knockoff can then be used to perform FDR-controlled feature selection following the Model-X knockoff framework (see Section 2.1). Overall, DeepDRK is a pipeline of two components. It first trains a transformer-based deep learning model, denoted as Knockoff Transformer (KT), to obtain swap property as well as reduce recontructability for the generated knockoff, with the help of adversarial attacks via multi-swappers. Second, a dependency regularized perturbation technique (DRP) is developed to further reduce the reconstructability for X post training. In the following subsections, we discuss two components in detail in the following two subsections. The KT (i.e. Xθ ), shown in Fig. 1 , is trained to generate knockoffs by minimizing a swap loss (SL) L SL and a dependency regularization loss (DRL) L DRL to achieve swap property and boost power. Swappers (i.e. S ωi ), on the other hand, are trained to generate adversarial swaps to enforce swap property. The total training objective is defined asEQUATIONNote that we use Xθ and X interchangeably, whereas the former is to emphasize that the knockoff depends on the model weights θ for KT. We detail the design of KT and swappers in Appendix C.In the following subsections, we breakdown each loss function.The swap loss is designed to enforce the swap property and is defined as follow:L SL pX, Xθ , tS ωi u K i"1 q " 1 K K ÿ i"1 SWDprX, Xθ s, rX, Xθ s Sω i q (5)`λ1 ¨RExpX, Xθ , tS ωi u K i"1 q `λ2 ¨Lswapper ptS ωi u K i"1 q where λ 1 and λ 2 are hyperparameters.The first term in Eq. ( 5)-"SWD"-measures the distance between two corresponding distributions. In our case, they are the laws of the joint distributions rX, Xθ s and rX, Xθ s Sω i . Here SWDp¨, ¨q is the sliced-Wasserstein distance (see Appendix D), S ωi is the i-th swapper with model parameters ω, and K is the total number of swappers. The swapper, introduced in (Sudarshan et al., 2020) , is a neural network for generating adversarial swaps (i.e. attacks). Training against such attacks enforces the swap property (see Appendix C for a detailed description). We adopt the same idea and extend it to a "multi-swapper" setup that jointly optimizes multiple independent swappers to better achieve swap property. In comparison with the likelihood-based loss in (Sudarshan et al., 2020) , which requires a normalizing flow (NF) (Papamakarios et al., 2021) to fit, SWD is adopted to compare distributions, as it not only demonstrates high performance on complex structures such as multi-modal data, but also fast to implement since it is likelihood-free (Kolouri et al., 2019; Deshpande et al., 2019; 2018; Nguyen et al., 2022) .The second term in Eq. ( 5)-"REx"-stands for risk extrapolation (Krueger et al., 2021) and is defined here as RExpX, Xθ , tS ωi u K i"1 q " x Var Sω i pSWDprX, Xθ s, rX, Xθ s Sω i q for i P rKs, which is a variant of the invariant risk minimization (IRM) (Arjovsky et al., 2019) loss. This term is commonly used to add robustness against adversarial attacks in generative modeling, which in our case, guarantees that the generated knockoffs Xθ are robust to multiple swap attacks simultaneously. Since swappers S ωi may have the same effect (identical adversarial attacks) if they share the same weights ω, it is important to distinguish them by enforcing different pairs of weights via the third term in Eq. ( 5):EQUATIONwhere C " tpi, jq|i, j P rKs, i ‰ ju, and simp¨, ¨q is the cosine similarity function.In all, the swap loss L SL enforces the swap property via SWD and the novel multi-swapper design.Such design provides a better guarantee of the swap property through multiple adversarial swap attacks, which is shown in Appendix L.3 for details.As discussed in Section 2.2, pursuing the swap property on sample level often leads to severe overfitting of Xθ (i.e., pushing Xθ towards X), which results in high collinearity in feature selection. To remedy, the DRL is introduced to reduce the reconstructability between X and X:L DRL pX, Xθ q " λ 3 ¨SWCpX, Xθ q `λ4 ¨LED pX, Xθ q, (7)where λ 3 and λ 4 are hyperparameters. The SWC term in Eq. ( 7) refers to the sliced-Wasserstein correltaion (Li et al., 2023) , which quantitatively measures the dependency between two random vectors in the same space. More specifically, let Z 1 and Z 2 be two p-dimensional random vectors. SWCpZ 1 , Z 2 q " 0 indicates that Z 1 and Z 2 are independent, while SWCpZ 1 , Z 2 q " 1 suggests a linear relationship between each other (see Appendix E for more details of SWC). In DeepDRK, we minimize SWC to reduce the reconstructability, a procedure similar to Spector & Janson (2022) .The intuition is as follows. If the joint distribution of X is known, then for each j P rps, the knockoff Xj should be independently sampled from p j p¨|X ´j q. In such case the swap property is ensured, and collinearity is avoided due to independence. As we do not have access to the joint law, we want the variables to be less dependent. Since collinearity exists with in X, merely decorrelate X j and Xj is not enough. Thus we minimize SWC so that X and X are less dependent. We refer readers to Appendix B for further discussion. In addition to SWC, the entry-wise decorrelation term L ED pX, Xθ q " ř p j"1 y corr ´Xj , Xθ,j ¯, proposed in Romano et al. (2020) , could be optionally added to stabilize training. We consider this term for all the experiments considered in this paper, however, we recommend its removal for the general use of DeepDRK. The discussion on this is in Appendix F.Empirically we observe a competition between L SL and L DRL in Eq. ( 4), which adds difficulty to the training procedure. Specifically, the L SL is dominating and the L DRL increases quickly after a short decreasing period. The phenomenon is also observed in all deep-learning based knockoff generation models when one tries to gain power (Romano et al., 2020; Sudarshan et al., 2020; Masud et al., 2021; Jordon et al., 2018) . We inlcude the experimental evidence in Appendix G. We speculate the reasons as follows: minimizing the swap loss, which corresponds to FDR control, is the same as controlling Type I error. Similarly, minimizing the dependency loss is to control Type II error. With a fixed number of observations, it is well known that Type I error and Type II error can not decrease at the same time after reaching a certain threshold. In the framework of model-X knockoff, we aim to boost as much power as possible given the FDR is controlled at a certain level, a similar idea as the uniformly most powerful (UMP) test Casella & Berger (2021) . For this reason, we propose DRP as a post-training technique to further boost power.DRP is a sample-level perturbation that further eliminates dependency between X and and the knockoff. More specifically, DRP perturbs the generated Xθ with the row-permuted version of X, denoted as X rp . After applying DRP, the final knockoff XDRP θ becomes:EQUATION)where α is a preset perturbation weight. In this way XDRP has a smaller SWC with X, since X rp is independent of X. Apparently by the perturbation, swap loss increases. We show in Appendix J and L.2 the effect of the perturbation on the swap property, FDR, and power. 
experiment: 
 We compare the performance of our proposed DeepDRK with a set of benchmarking models from three different perspectives. Namely, we focus on: 1. the fully synthetic setup that both input variables and the response variable have pre-defined distributions; 2. the semi-synthetic setup where input variables are rooted in real-world datasets and the response variable is created according to some known relationships with the input; 3. FS with a real-world dataset. Overall, the experiments are designed to cover various datasets with different p{n ratios and different distributions of X, aiming to evaluate the model performance comprehensively. In the following, we first introduce the model and training setups for DeepDRK and the benchmarking models. Next we discuss the configuration of the feature selection. Finally, we describe the dataset setups, and the associated results are presented. Additionally, we consider the ablation study to illustrate the benefits obtained by having the following proposed terms: SWC, REx, L swapper , and the multi-swapper setup. Empirically, these terms improve power and enforce the swap property for a lower FDR. We also demonstrate the improved stability by introducing entry-wise decorrelation L ED . Due to space limitation, we defer details to Appendix L.3.To fit models, we first split datasets of X into two parts, with an 8:2 training and validation ratios.The training sets are then used for model optimization, and validation sets are used for stopping the model training with an early stop rule on the validation loss with the patient period equals 6. Since knockoffs are not unique (Candes et al., 2018) , there is no need of the testing sets. To evaluate the model performance of DeepDRK, we compare it with 4 state-of-the-art (SOTA) deep-learning-based knockoff generation models as we focus on non-parametric data in this paper. Namely, we consider Deep Knockoff (Romano et al., 2020) 2 , DDLK (Sudarshan et al., 2020) 3 , KnockoffGAN (Jordon et al., 2018) 4 and sRMMD (Masud et al., 2021)5 , with links of the code implementation listed in the footnote. We follow all the recommended hyperparameter settings for training and evaluation in their associated code repositories. The only difference is the total number of training epochs. To maintain consistency, we set this number to 200 for all models (including DeepDRK).We follow the model configuration (see Appendix H) to optimize DeepDRK. The architecture of the swappers S ω is adopted from Sudarshan et al. (2020) . Optimizers for training both swappers and Xθ are AdamW (Loshchilov & Hutter, 2017) . We interchangeably optimize for Xθ and swappers S ω during training (see Eq. ( 4)). For every three times of updates for weights θ, the weights ω are updated once. The training scheme is similar to the training of GAN (Goodfellow et al., 2020) , except the absence of discriminators. We also apply early stopping criteria to prevent overfitting. A pseudo code for the optimization can be found in Appendix I. To conduct experiments, we set α " 0.5 universally for the dependency regularization coefficient as it leads to consistent performance across all experiments. A discussion on the effect of α is detailed in Appendix J. Additionally, we consider the effect of model size to the FS performance in Appendix K.Once models are trained, we generate the knockoff X given the data of X. The generated X are then combined with X for feature selection. We run each experiment on a single NVIDIA V100 16GB GPU. DeepDRK is implemented by PyTorch (Paszke et al., 2019)6 .Once the knockoff X is obtained, feature selection is performed following three steps. First, concatenate rX, Xs to form a n ˆp2pq design matrix pX, Xq. Second, fit a regression model Y " pX, Xq to obtain estimation coefficients t βj u jPr2ps . We use Ridge regression as it generally results in higher power. Third, compute knockoff statistics W j " | βj | ´| βj`p | (for j " 1, 2, . . . , p), and select features according to Eq. (3). We consider q " 0.1 as the FDR threshold due to its wide application in the analysis by other knockoff-based feature selection papers (Romano et al., 2020; Masud et al., 2021) . Each experiment is repeat 600 times, and the following results with synthetic and semi-synthetic datasets are provided with the average power and FDR of the 600 repeats.To properly evaluate the performance, we follow a well-designed experimental setup by Sudarshan et al. (2020); Masud et al. (2021) to generate different sets of n-samples of pX, Y q. Here X P R p is the collection dependent variables that follows pre-defined distributions. Y P R is the response variable that is modeled as Y " N pX T β, 1q. The underlying true β is the p-dimensional coefficient which is set to follow p 15¨?n ¨Rademacher(0.5) or p 12.5¨?n ¨Rademacher(0.5) distribution. Compared to Sudarshan et al. (2020) and Masud et al. (2021) , which consider p ? n as the scalar for the Rademacher, we downscale the scale of β by the factors of 15 or 12.5. This is because we find that in the original setup, the β scale is too large such that the feature selection enjoys high powers and low FDRs for all models. To compare the performance of the knockoff generation methods on various data, we consider the following distributions for X:Mixture of Gaussians: Following Sudarshan et al. (2020) , we consider a Gaussian mixture model X "ř 3 k"1 π k N pµ k , Σ k q,where π is the proportion of the k-th Gaussian with pπ 1 , π 2 , π 3 q " p0.4, 0.2, 0.4q. µ k P R p denotes the mean of the k-th Gaussian with µ k " 1 p ¨20 ¨pk ´1q, where 1 p is the p-dimensional vector that has universal value 1 for all entries. Σ k P R pˆp is the covariance matrices whose pi, jq-th entry taking the value ρ |i´j| k, where pρ 1 , ρ 2 , ρ 3 q " p0.6, 0.4, 0.2q.Copulas (Schmidt, 2007) : we further use copula to model complex correlations within X. To our knowledge, this is a first attempt to consider complicated distributions other than the mixture Gaussian in the knockoff framework. Specifically, we consider two copula families: Clayton, Joe with the consistent copula parameter of 2 in both cases. For each family, two candidates of the marginal distributions are considered: uniform (i.e. identity conversion function) and the exponential distribution with rate equals 1. We implement copulas according to PyCop7 .For both cases above, we consider the following pn, pq setups in the form of tuples: (200, 100), (500, 100), and (2000, 100) . This is in contrast to existing works, which considers only the (2000, 100) case for small n value (Sudarshan et al., 2020; Masud et al., 2021; Romano et al., 2020) . Our goal is to demonstrate a consistent performance by the proposed DeepDRK on different p{n ratios, especially when it is small.Results: Figure 2 and 3 compare FDRs and Powers for all models across the previously introduced datasets with two different setups for β, respectively. For illustration purposes, we keep sample size 200 and 2000 cases and defer the sample size 500 case to Appendix L.1. It is observed in Figure 2 that DeepDRK is capable of controlling FDR consistently compared to other benchmarking models across different data distributions and different p{n ratios. Other models, though being able to reach higher power, comes at a cost of sacrificing FDR, which contradicts to the UMP philosophy (see Section 3.2). The results in Figure 3 share a similar pattern to that of Figure 2 , except for the number of FDR-inflated cases (i.e. above the 0.1 threshold) increases due to the lowered scale of β. Overall, the results demonstrate the ability of DeepDRK in consistently performing FS with controlled FDR compared to other models across a range of different datasets and p{n ratios. Additionally, we provide the measurement of the swap property and the average runtime for each model in Appendix L.2 and M, respectively. The first dataset contains single-cell RNA sequencing (scRNA-seq) data from 10x Genomics8 . Each entry in X P R nˆp represents the observed gene expression of gene p in cell n. We refer readers to Hansen et al. (2022) and Agarwal et al. (2020) for further background. Following the same preprocessing in Hansen et al. (2022) , we obtain the final dataset of X is of size 10000 and dimension 1009 . The preprocessing of X and the synthesis of Y are deferred in Appendix N.The et al. (2021) . Finally, we synthesize the response Y with the real dataset of X via Y " N pX T β, 1q, where β follows entry-wise Unif(0, 1), N p0, 1q, and Rademacher(0.5) distributions.Results: Figure 4 and 5 compare the feature selection performance on the RNA data and the IBD data respectively. In Figure 4 , we observe that all but DDLK are bounded by the nominal 0.1 FDR threshold in the "Tanh" case. However, KnockoffGAN and sRMMD have almost zero power. The power for Deep Knockoff is also very low compared to that of DeepDRK. Although DDLK provides high power, the associated FDR is undesirable. In the "Linear" case, almost all models have well controlled FDR, among which DeepDRK provides the highest power. Similar observations can be found in Figure 5 . For the IBD data under the aforementioned three different data synthesis rules, it is clear all models but DDLK achieve well-controlled FDR. The red horizontal bar indicates the threshold q during FS.Besides (semi-)synthetic setups, we follow (Sudarshan et al., 2020; Romano et al., 2020) In all, we identify 47 metabolites that are reported to have association with IBD.To evaluate model performance, we first consider the same setup (see Table 2 in Appendix H) to train the model and generate knockoffs using the DeepDRK pipeline. During the FS step, however, we use 0.2 as the FDR threshold instead of 0.1, and apply a different algorithm-DeepPINK (Lu et al., 2018) that is included in the knockpy12 library-to generate knockoff statistics W j . The values are subsequently used to identify metabolites. We choose Deep Pink-a neural-network-based model-over the previously considered ridge regression due to the nonlinear relationships between metabolites (i.e. X) and labels (i.e. Y ) in this case study. Likewise, to generate knockoff for the benchmarking models, we follow their default setup. During FS, same changes are applied as in DeepDRK.We compare the FS results with the 47 metabolites and report the number of selections in Table 1 .A detailed list of selected features for each model can be found in Table 14 in Appendix O. From Table 1 , we clearly observe that compared to the benchmarking models, DeepDRK identifies the most number of referenced metabolites. KnockoffGAN and sRMMD behave preferably in limiting the number of discoveries given the fewer number of identified undocumented metabolites. The performance of Deep Knockoff and DDLK is intermediate, meaning a relatively higher number of discovered metabolites that are referenced, compared to KnockoffGAN and sRMMD, and a relatively lower number of discovered undocumented metabolites. Nevertheless, since there is no ground truth available, the comparison is only qualitative. 
conclusion: 
 In this paper, we introduced DeepDRK, a deep learning-based knockoff generation pipeline that consists of two steps. It first trains a Knockoff Transformer with multi-swappers to obtain swap property as well as reduce reconstructability. Second, the dependency regularized perturbation is applied to further boost power after training. DeepDRK is shown to achieve both low FDR and high power across various data distributions, with different p{n ratios. Our results also suggest an outperformance of DeepDRK over all considered deep-learning-based benchmarking models.Experiments with real and semi-synthetic data further demonstrate the potential of DeepDRK in feature selection tasks with complex data structures. 
 
 Appendix: The definition of FDR is a follows. Let S Ă rps be any selected indices and β ˚be the underlying true regression coefficients. The FDR for selection S is (Hristache et al., 2001) .FDR " E « # ␣ j : β j "Theorem 2 Let Y " X J β J `X´J β ´J `ε, where J Ă rps, and ε is a centered Gaussian noise.Equivalently it means Y K K X J | X J β J , X ´J . Suppose there exists a β J such that X J β J " X J β J a.s., then denoting Y ˚" X J β J `X´J β ´J `ε, we have prX, Xs, Y q d " ´rX, Xs swappJq , Y ˚¯and ´rX, Xs, Y ˚¯d " ´rX, Xs swappJq , Y ¯. (10)Furthermore in the knockoff framework (Barber & Candès, 2015) , let W " wprX, Xs, yq and W ˚" wprX, Xs, y ˚q, then for all j P J, P pW j ą 0q `P `W j ą 0 ˘ď 1.Equation 11 implies a no free lunch situation for selection power when there is exact reconstructability.To fix the recontructability issue, Spector & Janson (2022) proposed two methods in Guassian design. The first is the minimal variance-based reconstructability (MVR) knockoff, in which knockoff X is sampled to minimize the lossEQUATION)Note that is equivalent to maximize E " Var ´Xj | X ´j , X¯ı for all j P rps. Another way to achieve it is the maximum entropy (ME) knockoff, where X is sampled from maximizingEQUATIONFortunately under Gaussian design, the above two optimizations have closed-form solutions. Since X is Guassian, pX, Xq must be joint Gaussian to satisfy the swap property. To optimize, one first calculates the covariance matrix using the SDP method in Barber & Candès (2015) , which gives a diagonal matrix S. Then both MVR and ME boils down to an optimization on S:EQUATION)Though it is shown that both methods obtain high power for feature selection, however, neither MVR nor ME could be directly extended to arbitrary distributions, due to the fact that the conditional variance and the likelihood is intractable.In DeepDRK, we consider regularizing with a sliced-Wasserstein-based dependency correlation, which can be deemed a stronger dependency regularization than entropy. A post-training perturbation is also applied to further reduce collinearity. However, theoretical understanding of how they affect swap property and power remains to be studied.DeepDRK's knockoff Transformer (KT) model is designed based on the popular Vision Transformer (ViT) (Dosovitskiy et al., 2020) . The difference being the dimension of the input is 1D, not 2D, for X. We do not consider patches as the input. Instead, we consider all entries of X to consider correlations of each pair of entries in the knockoff X. This is structurally similar to the original Transformer (Vaswani et al., 2017) . Nonetheless, we keep all the remaining components from ViT, such as patch embedding, PreNorm and 1D-positional encoding (Dosovitskiy et al., 2020) . Since knockoff generation essentially requires a distribution. To enable this, we consider feeding X and a uniformly distributed random variable Z of the same dimension as X, to encode randomness.The swapper module is first introduced in DDLK (Sudarshan et al., 2020) , to produce the index subset B for the adversarial swap attack. Optimizing knockoff against these adversarial swaps enforces the swap property. Specifically, the swapper consists of a matrix of shape 2 ˆp (i.e. trainable model weights), where p is the dimension of X. This matrix controls the Gumbel-softmax distribution (Jang et al., 2017) for all p entries. Each entry is characterized by a binary Gumbel-softmax random variable (e.g. can only take values of 0 or 1). To generate the subset B, we consider drawing samples of b j from the corresponding j-th Gumbel-softmax random variable, and the subset B is defined as tj P rps; b j " 1u. During optimization, we maximize Eq. ( 4) w.r.t. to the weights ω i of the swapper S ωi such that the sampled indices, with which the swap is applied, lead to a higher SWD in the objective (Eq. ( 4)). Minimizing this objective w.r.t. Xθ requires the knockoff to fight against the adversarial swaps. Therefore, the swap property is enforced. Compared to DDLK, the proposed DeepDRK utilizes multiple independent swappers.Wasserstein distance has become popular in both mathematics and machine learning due to its ability to compare different kinds of distributions (Villani et al., 2009) and almost everywhere differentiability (Arjovsky et al., 2017) . Here we provide its definition. Let X, Y be two R d random vectors following distributions P X , P Y with finite p-th moment. The Wasserstein-p distance between P X and P Y is:EQUATION)where ΓpP X , P Y q denotes the set of all joint distributions such that their marginals are P X and P Y . When d " 1 in particular, the Wasserstein distance between two one-dimensional distributions can be written as:EQUATIONwhere F X and F Y are the cumulative distribution functions (CDF) of P X and P Y respectively. Moreover, if p " 1 as well, the Wasserstein distance can be further simplified asEQUATION)From the above it is easy to notice that the 1d Wasserstein distance is easy to compute, which leads to the development of sliced-Wasserstein distance (SWD) (Bonneel et al., 2015) . To exploit the computational advantage on 1d, one first projects both distributions uniformly on a 1d direction and computes the Wasserstein-p distance between the two projected distributions. SWD is then calculated by taking the expectation of the random direction. More specifically, let µ P S d´1 denote a projection directions, the following push-forward distribution (Villani et al., 2009) µ 7 P X denotes the law of µ T X. Let µ be d dimensional spherical uniform, the p-sliced-Wasserstein distance between P X and P Y reads:SW p pP X , P Y q " ż µPS d´1W p pµ 7 P X , µ 7 P Y q dµ. (18)Combining ( 18) and ( 16) gives:EQUATION"EQUATIONIn spite of faster computations, the convergence of SWD is shown to be equivalent to the convergence of Wasserstein distance under mild condtions (Bonnotte, 2013) . In practice, the expectation on µ is approximated by a finite summation over a number of projection directions uniformly chosen from S d´1 .The idea of metricizing independence is recently advanced using the Wasserstein distance (Wiesel, 2022; Nies et al., 2021) . Given a joint distribution pX, Y q " Γ XY and its marginal distributions X " P X , Y " P Y , the Wasserstein Dependency (WD) between X and Y is defined by W DpX, Y q " W p pΓ XY , P X b P Y q. A trivial observation is that W DpX, Y q " 0 implies that X and Y are independent. Due to the high computational cost of Wasserstein distance, sliced-Wasserstein dependency (SWDep) (Li et al., 2023) is developed using sliced-Wasserstein distance (see Appendix D for SWD details). The SW dependency between X and Y is defined as SW p pΓ XY , P X b P Y q, and a 0 SW dependency still indicates independence. Since the dependency metric is not bounded from above, sliced-Wasserstein correlation (SWC) is introduced to normalize SW dependency. More specifically, the SWC between X and Y is defined asEQUATIONwhere Γ X X and Γ Y Y are the joint distributions of pX, Xq and pY, Y q respectively. It is shown that 0 ď SWC p pX, Y q ď 1 and SWC p pX, Y q " 1 when X has a linear relationship with Y (Li et al., 2023) .In terms of computing SWC, we follow Li et al. (2023) and consider both laws of X and Y to be sums of 2n Diracs, i.e. both variables are empirical distributions with data I full " tpx i , y i qu 2n i"1 . Define I " tpx i , y i qu n i"1 and Ĩ " tpx i , ỹi qu n i"1 , where px i , ỹi q " px n`i , y n`i q. Because data is i.i.d., I and Ĩ are independent. We further introduce the following notation:I xy " tpx i , y i qu n i"1 , Ĩxy " tpx i , y i qu n i"1 I xx " tpx i , x i qu n i"1 , Ĩxx " tpx i , x i qu n i"1 , I yy " tpy i , y i qu n i"1 , Ĩyy " tpỹ i , y i qu n i"1Then the empirical SWC can be computed by: { SWC p pX, Y q :" SWD p ´IIxy , IThe entry-wise decorrelation term between X j and Xθ,j in Eq. ( 7) isEQUATION)This term is adopted from Romano et al. (2020) for the purpose of stabilizing training in DeepDRK.We discover that the training of the knockoff transformer would occasionally experience instability issue, leaving the training loss to go infinity on some datasets. And adding this loss term prevents this situation. We conduct experiments to verify that the removal of this term does not significantly affect the performance of DeepDRK in feature selection. Results are deferred in the appendix L.3 for structural consistency. In DeepDRK, we include this term and the hyperparameter configuration for λ 4 can be found in Table 2 . However, we suggest removing this term initially for reducing the search space of hyperparameters.In Figure 6 , we include scaled L SL curves and L DRL curves for each considered model. For comparison purposes, the curves that describe the changes of the corresponding losses are shown after normalizing the values to be within 0 and 1. We consider the first 20 epochs as the DRL would flatten out in latter epochs without dropping. The competition can be clearly observed as when L SL drops, L DRL increases, indicating difficulty to maintain low reconstructability.In Table 2 , we include configuration details on the KT and swappers.In Algorithm 1, we provide pseudo code for training the Knockoff Transformer (i.e. the first stage shown in Figure 1 ).As discussed in section 3.2, empirically it is difficult to obtain swap property while maintaining low reconstructability at sample level. To leverage this issue, dependency regularization perturbation (DRP) is introduced. In this section, we evaluate the effect of α in XDRP θ (e.g. Eq. ( 8)) to the feature selection performance. Results are summarized in Figure 7 , concerning 5 synthetic datasets. 24).When α is decreased, we observe an increment in power. However, the FDR follows a bowelshaped pattern. This is consistent with the statement in Spector & Janson (2022) , as introducing the permuted X rp will decrease reconstructability for higher power. However, a dominating X rp breaks the swap property for higher FDRs. Based on our hyperparameter search and the results in Figure 7 , we suggest choosing α within the range between 0.4 and 0.5. Initialize the knockoff transformer g θ p¨q with random weights 4:Initialize swappers S ωi , i " 1, . . . , K with random weights 5:Initialize the AdamW optimizer opt θ with the learning rate α θ for g θ p¨q 6:Initialize the AdamW optimizer opt ωi with the learning rate α s for S ωi , i " 1, . . . , K 7:for t " 1 to T do Ź Training start 8:for l " 1 to |Dtrain| Bs do 9:Sample B s samples of X from D train : X l 10:Generate knockoff Xl " g θ pX l q 11:Calculate L SL pX l , Xl , tS ωi u K i"1 q and L DRL pX l , Xl q 12: θ Ð θ + opt θ pL SL pX l , Xl , tS ωi u K i"1 q `LDRL pX l , Xl qq Ź Optimize KT 13:if l mod γ " 0 then Ź Optimize swappers 14: end for 22: end procedureω i Ð ω i + opt ωi p´L SL pX l , Xl , tS ωi u K i"1 qq, i " 1, . . . ,In this section, we consider the effect of the model size of KT to the feature selection performance. Namely, we change the number of hidden dimensions and the number of layers in half and compare their performance to the default DeepDRK. Results for the hidden dimension case are in We consider a result is better when it has higher power and the FDR is controlled at the nominal 0.1, or a lower FDR when FDR is above the 0.1 threshold.to the change of model size as in many cases in Table 3 and Table 4 , the performance of the reduced model is consistent with DeepDRK. This is mostly due to the early stopping criterion that is applied during training, which effectively prevents overfitting. However, there are cases with "Copula: Joe & Exponential" and "Copula: Joe & Gamma" datasets, where the drop in performance can be observed with the reduced models. As a result, we recommand start with the default DeepDRK configuration when applying the framework to other datasets, and change the model size if there is burden in the computational hardware.In this section, we include all results that are deferred from the main paper.L We use different metrics to empirically evaluate the swap property on the generated knockoff X and the original data X (i.e. Eq. ( 1)). In this paper, three metrics are considered: mean discrepancy distance with linear kernel, or "MMD(Linear)" for short; sliced wasserstein 1 distance (SWD1); and sliced wasserstein 2 distance (SWD2). We measure the sample level distance with the three metrics between the joint vector that concatenates X and X (e.g. pX, Xq) and the joint vector after randomly swapping the entries (e.g. pX, Xq swappBq ). To avoid repetition, please refer to section 2.1 and Eq. ( 1) for the definition of notation. Empirically, it is time consuming to evaluate all subsets B of the index set rps. As a result, we alternatively define a swap ratio r s P t0.1, 0.3, 0.5, 0.7, 0.9u. The swap ratio controls the amount of uniformly sampled indices (i.e. the cardinality |B| " r s ¨p) in a subset of rps. For any X and X from the same experiment, 5 different subsets B are formed according to 5 different swap ratios. And we report average value over the swap ratio to represent the empirical measurement on the swap property. Results can be found in Table 5 .Clearly, compared to other models, the proposed DeepDRK achieves the smallest values in almost every entry across the three metrics and the first 4 datasets (i.e. J+G, C+G, C+E and J+E in Table 5 ). This explains why DeepDRK has lower FDRs as DeepDRK maintains the swap property relatively better than the benchmarking models (see results in Figure 2 and 3). Similarly, we observe that KnockoffGAN also achieves relatively small values, which leads to well-controlled FDRs compared to other benchmarking models. Overall, this verifies the argument in Candes et al. (2018) that the swap property is important in guaranteeing FDR during feature selection.Nevertheless, we also identify a discrepancy, which is the case with mixture of Gaussian distributions. The proposed DeepDRK achieves the best performance in FDR control and power (see results in Figure 2 and 3 ), yet its swap property measured on the three proposed metrics in  and SWD2 ). The evaluation considers all benchmarking models and all datasets in the synthetic dataset setup. For space consideration, we use abbreviations to indicate the name of the datasets. The full name can be found in Table 6 .In this section, we perform ablation study on different terms introduced in Section 3.1.1, to show the necessity of designing these terms during the optimization for knockoffs. We consider the fully synthetic setup described in Section 4.3, and consider 200 and 2000 different sample sizes. The distribution for β is p 15¨?N ¨Rademacher(0.5), which is the harder case. Overall, we consider the Besides the above terms, we further consider the L ED term in an ablation study. Results can be found in Table 11 . This term is primarily used for the purpose of stabilizing the training. We find that the model without this term would fail 2 times out of 10 trials with some datasets. As shown in Table 11 , there is only minimal change of performance when L ED is removed compared to DeepDRK that equips with this term since most values differ in only the third decimal point. Therefore, we recommend to remove this term first when applying DeepDRK on a dataset for the purpose of reducing the number of hyperparameters and improving the training speed. If the optimization fails on the given dataset, one can add this term back.We consider evaluating and comparing model training runtime in We first normalize the raw data X to value range r0, 1s and then standardize it to have zero mean and unit variance. Y is synthesized according to X. We consider two different ways of synthesizing Y . The first is identical to the previous setup in the full synthetic case with Y " N pX T β, 1q and β " p 12.5¨?N ¨Rademacher(0.5). For the second, the response Y is generated following the expression:EQUATIONwhere ϵ follows the standard normal distribution and the 20 covariates are sampled uniformly.Here we provide the supplementary information for the experiments described in Section 4.5. In Table 13 , we provide all the 47 referenced metabolites based on our comprehensive literature review.In Table 14 , we provide the list of identified metabolites by each of the considered models. This table corresponds The "DeepDRK" term is pronounced as "Deep Dark" and is inspired by the underground cave biome prevalent in the world of Minecraft, whereas the proposed feature selection model acts as the guiding beacon amidst the darkness of unknown.https://github.com/msesia/deepknockoffshttps://github.com/rajesh-lab/ddlkhttps://github.com/vanderschaarlab/mlforhealthlabpub/tree/main/alg/ knockoffganhttps://github.com/ShoaibBinMasud/soft-rank-energy-and-applicationsThe implementation of DeepDRK can be found in https://NODISCLOSUREFORPEERREVIEW.https://github.com/maximenc/pycop/https://kb.10xgenomics.com/hc/en-usThe data processing code is adopted from this repo: https://github.com/dereklhansen/ flowselect/tree/master/datahttps://www.metabolomicsworkbench.org/ under the project DOI: 10.21228/M82T15.https://pubchem.ncbi.nlm.nih.gov/https://amspector100.github.io/knockpy/ 


Decision: The paper introduces DeepDRK, a new algorithm for improving FDR (by to constructing knockoff features) in the model-X knockoff framework. The authors propose a deep learning-based method for generating knockoff features in which a knockoff covariate is generated for each existing covariate (and knockoff covariates have to satisfy the so-called swap property). The authors provide extensive numerical simulation and real-data examples to show that the proposed method has advantageous performance compared with other state-of-the-art methods, especially when the covariate distribution is complex and when the sample size is small.

All the reviewers agree that the paper considers an important question in the knockoff framework and gives an empirically satisfactory solution. Reviewers had some concerns about the presentation of the paper (e.g. additional background, motivation of the problem, etc) which have been resolved after the discussion period, and hence I would like to recommend the paper for acceptance. Also, I would like to suggest to the authors to revise the manuscript to improve the presentation according to the comments provided by the reviewers.

Reviews: review: paper_summary: paper_summary: The work introduces DeepDRK, a new algorithm for improving FDR in the model-X knockoff framework. In this framework, a knockoff covariate is generated for each existing covariate where then knockoff covariates have to satisfy the swap property. Given a knockoff statistic that satisfies the flip-sign property, one can perform feature selection with guaranteed false discovery rate by training a regression/classification model on the concatenated vector of original and knockoff covariates and then using any arbitrary feature importance score. Therefore, the main challenge in this framework is generating the knockoffs such that they satisfy the strong swap property: swapping any arbitrary subset of covariates with their knockoffs should not change the distribution of the concatenated vector. 

Given that most real-world data does not follow Gaussian or a mixture of Gaussian distribution, generating knockoffs that satisfy the swap property is a challenge. Existing work is focused on using deep generative models (e.g. GANs) to handle this challenge. The paper argues that this approach for continuous covariates will result in reconstructability: overfitting on the data will make the distribution of knockoffs too similar to the data distribution and therefore making the concatenated vector too multicollinear for any feature-importance method to have discovery power.

The DeepDRK framework seeks to tackle reconstructability by training the deep model in such a way that not only satisfies the swap property but also reduces reconstructability. This is achieved in two steps:
Firstly, by having a 1- the adversarial swap loss where a group of models compete adversarially against the knock-off generator model 2- a dependence regularization loss to reduce reconstructability by directly reducing the sliced-Wasserstein correlation between the original and knockoff vectors.
Secondly, by the post-training dependency regularization which interpolates between the generated knockoff data table and the row permuted original data table. By making the interpolation weaker or stronger, one gains control in how much they can improve how much swap condition holds and therefore how good the FDR is at the cost of loosing power and vice versa.
The paper then shows experimental evidence in synthetic and semi-synthetic scenarios (where the true covariate-response) relationship is known. The argument is that compared to existing algorithms that have better power in low-sample settings, DeepDRK's empirical FDR is close the the selected level more consistently and this is proven both by the FDR vs Power results and the original vs knockoff feature importance distributions.
strength_weakness: strength_weakness: - It seems necessary to see 4.1 results for various beta scales used for generating the response variable to compare different methods in various levels of difficulties to make sure the value of 15 has not been selected to favor DeepDRK. I think it'd be interesting to have the results in a 2D FDR vs Power scatter plot for better comparison.
- I'd be interested in seeing mean + confidence interval performance plots for Figure 2 and Figure 3 using various distribution parameter selections (e.g. for GMM distribution using various \pi parameters in addition to to more \rho_base, etc)
- The process for choosing the value of alpha=0.5 needs to be clarified as "it leads to consistent performance" might suggest choice based on final results which would be a case of overfitting. The discussion related to Figure 8 seems to confirm overfitting.
questions: questions: Discussed above
limitations: limitations: There needs to be a discussion regarding the choice of hyperparameters especially the alpha parameter as in real-world applications of knockoffs it's impossible to know the best performing value in advance. Given the results in Figure 8, there seems to be high sensitivity to the alpha value which would make the framework unreliable. At the very least, the authors need to show the superiority of their framework to existing methods in a large interval of alpha values.
paper_summary: paper_summary: This paper considers the construction of knockoff features 
in the variable selection framework of model-X knockoffs.
The authors propose a deep learning-based method for 
generating knockoff features. Extensive numerical simulation and real-data examples show 
that the proposed method has advantageous performance compared
with other state-of-art methods, especially when the covariate distribution 
is complex and when the sample size is relatively small.
strength_weakness: strength_weakness: The proposed method is based mostly on empirics and heuristics, 
with a theoretical understanding lacking.
questions: questions: 1. The introduction of the dependency regularized perturbation is 
for power boosting, and yet in the ablation study, this feature seems to help 
more with FDR control as opposed to power-boosting. Is there a reason?
2. On page 4, line 132, the definition of REx: I am confused since 
the LHS does not depend on $i$ but the RHS depends on $i$. 
3. On page 4, line 148: "correltaion" -> "correlation"
4. On page 5, line 153: I am confused by the statement that "the knockoff 
 $X_j$ should be independently sampled from $p_j(\cdot \mid X_{-j})$"
--- this alone is not enough to ensure the knockoff property.
5.  On page 5, line 173, what is $\alpha$?
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes a deep-learning-based method for performing "knockoff"-based feature selection. The main contribution is the proposal of a new optimization problem, which incorporates a four part loss.  The method is evaluated on synthetic, semi-synthetic, and real-world data, showing comparable and, at times, improved performance over relevant baselines.
strength_weakness: strength_weakness: **Motivation.** It's not clear why this problem is of interest. It is true that many papers have been published on this topic. But the authors motivate the problem not by describing why knockoffs are particularly important; they simply say that finding information features is "mission impossible," which doesn't give the reader any idea of why the problem is relevant.

**Organization.** To articulate a list of contributions, a paper must clearly answer two fundamental questions: (1) "What is the problem under consideration?" and (2) "How does the proposed method address this problem?" Unfortunately, it's not clear that this paper answers the first or second questions.  Here are some comments:
1. The authors claim that their method reduces reconstructability.  However, it's not clear what "reconstructability" is, or how it would be formally defined. The idea of reconstructability only arises in the discussion of related work, and even here, the problem is not presented in a way that is distinct from the solution presented in [4].  
2. Section 2.1 is confusing. More care should be taken w/r/t the introduction of the problem of generating a knockoff sample.  The authors adopt quite a bit of notation from [11].  However, whereas [11] presents the method with examples and clearly articulated goals, this paper's summary is so coarse/brief that any reader who has not read this past work will have little to no chance of understanding the problem of finding knockoffs. 
3. Deferring the definition of FDR to the appendix doesn't make sense to me. This quantity is of fundamental interest; it is measured throughout the experiments, and theorems are presented (in part from past work) which rely on this definition. The paper is incomplete without including this definition in the main text.
4. Theorem 2.1 is not clearly stated. The "aforementioned property" is not inferable from context.  The authors should make it clear that (1) they are referring to Theorem 1 in [11] and (2) that the "aforementioned property" is the choice of threshold in (3).
5. Readability would be improved if the authors included an example of the swap function, such as the example in [11] after Def. 2.
6. The authors seem to use (X,Z) and [X,Z] freely and interchangeably. This should be clarified. [11] sticks with (X,Z) (rather than square brackets) and I would recommend that the authors do the same.
7. Make it abundantly clear that whenever the authors say "power," they mean low type II error. This may not be clear to readers coming from deep learning.

**Main method.** A few comments on the main method.
1. The results are presented in a confusing order. (4) is presented before the authors know what any of the pieces are. Readers must wait around a page to get the details, which will frustrate. Furthermore, it's not clear why (4) is a fundamentally "good" objective, or where it came from. As written, it feels like the authors pulled this problem out of a hat.  Why does posing this as a minimax problem (as opposed to, e.g., a constrained problem) make sense? Why do the authors choose these two loss terms (and not others)? And how should one choose all of the trade-off parameters?
2. The authors seem to imply that [56] has a problem. It's not clear what this problem is; the "is far from enough" sentence doesn't give any quantitative evidence as far as I can tell. The result of this is that when the authors say "To fix it, ...," it's unclear what problem they are fixing.
3. The authors borrow a loss term from the risk extrapolation (REx) paper from domain generalization. The authors should justify this choice, as well as the other loss functions that they append onto their objective. As written, it feels as if the authors are somewhat arbitrarily adding on loss functions, and there isn't a clear principle driving the decision making. This gives the paper a largely heuristic feel. And while the authors claim to provide ablations in the appendix, it's not clear where they are. The authors say that the results are "in the associated tables," but it's not clear what tables these are. A similar concern arises w/r/t Figure 9.
4. The trade-off between L_SL and L_DRL isn't clearly presented. This content should be presented in the main text; based on the figure in Appendix G, it's not clear that the trade-off always holds. Furthermore, even if it did hold, the contribution that the authors are "the first to observe this phenomenon" seems to be trivially true: If no one has used this loss before, then of course no one has seen this trade-off.

**Experiments.** Some comments on the experiments.  The experiments seem to show marginal improvements for the proposed method. While DeepDRK does at times achieve higher power for a fixed FDR, the results are often fairly marginal, e.g., in Figure 5.  Given the shortcomings in the presentation of the method, it's not clear that these results do enough to justify acceptance.
questions: questions: See above.
limitations: limitations: N/A

Comments: title: Response
comment: I want to thank the authors for addressing all of my questions comprehensively. The empirical results now seem robust and reliable. I have changed my review scores accordingly.
I want to thank the authors for their response. I find my concerns addressed, and remain positive about this submission.
Dear reviewer,
Thanks a lot for your time and effort. As the discussion period will end soon, we would greatly appreciate your feedback on whether our rebuttal has adequately addressed your concerns. Please reach out if you have further comments. Thank again!
Dear reviewer,
Thanks a lot for your time and effort. As the discussion period will end soon, we would greatly appreciate your feedback on whether our rebuttal has adequately addressed your concerns. Please reach out if you have further comments. Thank again!
Dear reviewer,
Thanks a lot for your time and effort. As the discussion period will end soon, we would greatly appreciate your feedback on whether our rebuttal has adequately addressed your concerns. Please reach out if you have further comments. Thank again!
Thanks for your valuable comments. Due to page limit, we provide brief introductions to the standard knockoff definitions and the corresponding selection procedures, as the general motivation and idea have been widely known by the statistics community. We do understand that it could cause inconvenience to people unfamiliar with the subject, and hereby in the revised manuscript we have added more details. We have also reorganized the paper and fixed typos based on your advice. 

Feature selection, or variable selection, is a data-driven method to determine which of the $p$ features $(X\_1, \ldots, X\_p)$ are statistically related to the response $y$ in a linear model. Feature selection methods are important in a number of application areas, such as medical healthcare, economy, and political science [1]. Among various selection procedures, model-X knockoff emerges to be useful in providing an effective way to control the false discovery rate (FDR) of the selection result, regardless of the regression algorithm. To improve readability, we have already moved the definition of FDR to Section 2.1 from Appendix A and added examples to better illustrate swap property in the Appendix. 

The knockoff selection method proposed in most statistics literatures [1] has two limitations. First, the assumption that $X$ is Gaussian is often unrealistic, and its underlying distribution is usually unknown. Second, simply decorrelating $X\_j$ and $\tilde{X}\_j$ is not enough whenever $X\_j$ is almost surely equal to a linear combination of $X\_{-j}$ [2], which is stated in Appendix B. This is referred to as “reconstructability”, which is a population counterpart of collinearity, see Section 2.3 and Appendix B for more discussions. When there is reconstructability, the resulting knockoff adds collinearity into the regressor $(X,\tilde{X})$, deteriorating the regression and thus the selection result. Due to the two drawbacks, we propose DeepDRK, a framework that performs powerful knockoff-based feature selection for unknown $X$ distributions.

- [1] Candes, E., Fan, Y., Janson, L., & Lv, J. (2018). Panning for gold:‘model-X’knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3), 551-577.
- [2] Spector, A., & Janson, L. (2022). Powerful knockoffs via minimizing reconstructability. The Annals of Statistics, 50(1), 252-276.