DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection
Authors: Hongyu Shen
Yici Yan
Zhizhen Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Feature Selection
Deep Learning
Model-X Knockoff
FDR Control
Boosting Power
Primary area: generative_models
TL;DR: A pipeline that consists of a transformer-based model with novel regularizations to boost power while controlling the FDR during feature selection

Abstract: Model-X knockoff has garnered significant attention among various feature selection methods due to its guarantees for controlling the false discovery rate (FDR). Since its introduction in parametric design, knockoff techniques have evolved to handle arbitrary data distributions using deep learning-based generative models. However, we have observed limitations in the current implementations of the deep Model-X knockoff framework. Notably, the "swap property" that knockoffs require often faces challenges at the sample level, resulting in diminished selection power. To address these issues, we develop "Deep Dependency Regularized Knockoff (DeepDRK)," a distribution-free deep learning method that effectively balances FDR and power. In DeepDRK, we introduce a novel formulation of the knockoff model as a learning problem under multi-source adversarial attacks. By employing an innovative perturbation technique, we achieve lower FDR and higher power. Our model outperforms existing benchmarks across synthetic, semi-synthetic, and real-world datasets, particularly when sample sizes are small and data distributions are non-Gaussian.

DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection
Hongyu Shen' Yici Yan? Zhizhen Zhao'
Department of Electrical and Computer Engineering', University of Illinois at Urbana Champaign. Department of Statistics?,
University of Illinois at Urbana Champaign.
[hongyu2, yiciyan2, zhizhenz Millinoiii edu
Abstract
Model-X knockoff has garnered significant attention among various feature selec- tion methods due its guarantees for controlling the false discovery rate (FDR). Since its introduction in parametric design, knockoff techniques have evolved to handle arbitrary data distributions using deep learning- -aaiin generative models. However, we have observed limitations in the current implementations of the deep Model-X knockoff framework. Notably, the "swap property" that knockoffs require often faces challenges at the sample level, resulting in diminished selection power. To address these issues, we develop "Deep Dependency Regularized Knockoff (DeepDRK))" instribution-free deep learning method that effectively balances FDR and power. DeepDRK, we introduce novel formulation of the knockoff model as learning problem under multi-source adversarial attacks. By employing an innovative perturbation technique, we achieve lower FDR and higher power. Our model outperforms existing benchmarks across synthetic, semi-synthetic, and real-world datasets, particularly when sample sizes are small and data distributions are non- Non-Gaussian.
Introduction
Feature selection (FS) has garnered significant attention over the past few decades due to the rapidly increasing dimensionality of data, well the associated computational, storage, and noise challenges 15). Successfully identifying the true informative features among inputs can significantly enhance the performance of analysis frameworks and drive advancements fields such as biology, neuroscience, medicine, economics, and social sciences [15] [5). However, the task of accurately pinpointing informative features is often considered nearly impossible, particularly due to the limited availability of data relative to the increasing dimensionality [55] I1I A practical approach to address this challenge is the development of algorithms designed select features while maintaining controlled error rates. Targeting this goal, Model-X knockoffs, novel framework, is proposed
[1 to select relevant features while controlling the false discovery rate (FDR). In contrast to the classical setup, where assumptions on the correlations between input features and the response are imposed [6] 19). the Model-X knockoff framework only requires linear relationship between the response and the features. With strong finite-sample FDR guarantee, Model-X knockoff saw broad applications in domains such as biology, neuroscience, and medicine, where the sample size is limited 4 27
55 38 62].
There have been considerable developments of knockoffs since its debut. In scenarios where feature distributions are complex, various deep learning methods [77]47 55 38 62) have been proposed. However, we observe major limitations despite improved performance. First, the performance of existing methods varies across different data distributions. Second, the quality of selection declines
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

when the sample size is relatively small. Third, training the deep knockoff generation models can be challenging due to competing loss terms in the training objective. We elaborate on the drawbacks in Sections 2.2 and 3.2
In this paper, we address these issues by proposing the Deep Dependency Regularized Knockoff (DeepDRK), a deep learning- beannigg-based pipeline. We formulate the knockoff generation as an adver- sarial attack problem involving multiple sources. By optimizing model against the adversarial environments, we achieve better "swap property" 4 compared to the baseline models. DeepDRK is also equipped with novel perturbation technique reduce "reconstructtaility" [54], which in turn controls FDR and boosts selection power. The experiments conducted on synthetic, semi-synthetic, and real datasets demonstrate that our pipeline outperforms existing methods across various scenarios.
Background and Related Works
2.1 Model-X Knockoffs for FDR control
The Model-X knockoffs framework consists of two main components. Given the explanatory variables X = .
E IK? and the response variable Y (Y continuous for regression and categorical for classification), the framework requires: 1. knockoff X (X1, X2,..., Xp that "fakes" X; 2. the knockoff statistics Wy for j [p] that assess the importance of each feature Xj. The knockoff X is required to be independent of Y conditioning on X, and must satisfy the swap property:
(X, X swap BY A (x, X): BB [p].
(I)
Here BB exchanges the positions of any variable Xy. j E B, with its knockoff Xy. The knockoff statistic wyy((X X), Y) (for j & (p]) depends on the concatenated variable (X, X) and and must satisfy the flip-sign property:
Wy Y) if B Wi (X, (xax))))))) X) = -w (). ifj
The functions w,(-) with j E [p] have many candidates, for example wy = 1851-12 1851-1881- where 33 and 3, the corresponding regression coefficients of X, and X; with the regression function Y (X,3, + X,,,) + c, where is independently drawn from the normal distribution.
When the two knockoff conditions (i.c., Eq. and 2 are met, {ur= e Ty), where
select features by S
1+ 10 1+10: wy < - < <
Tq
min 150
max(1, (1 wy 3 f)
To assess the feature selection quality, FDR is commonly used as average Type error of selected features [4) which is defined as follows. Let S = [p] be an arbitrary set selected indices, and let 3* denote the underlying true regression coefficients. The FDR for the selection S is
# {
35 and j 8}
FDR
E
(4)
#
v
The control of FDR is guaranteed by the following theorem from
Theorem 2.1. Given the knockoff that satisfies the swap property in Eq. L the knockoff statistic that satisfies Eq. ② and = {wg 3 TyJ, we have FDR & q.
2.2 Related Works
Model-X knockoff's is first studied under Gaussian design. Namely, the original variable X ~ (1, Σ) with / and Σ known. Since Gaussian design does not naturally generalize to complex data distribu- tions, several methods are proposed to weaken the assumption. Among them, model-specific ones

---

DeepDRK Pipeline
Training Stage
Multi Swappers
Traleable
Feactios
Opeeratis alirealien
CaulX. Kp. (
AamuX Xz)
Post-training Stage
Knockoof Transformer
Dependency
(KT)
Perturbation (DRP)
Regulanized
SDRP
The illustration of the DeepDRK pipeline, which consists of two components: 1. the training stage that optimizes the knockoff Transformer and swappers by LsL. and LDrl: 2. the post-training stage that generates the knockoff X via dependency regularized perturbation.
Figure 1:
such AEknockoff [3), Hidden Markov Model (HMM) knockoff [51] and MASS (20] all propose parametric alternatives to Gaussian design. These methods can better learn the data distribution, while keeping the sampling process relatively simple. Nevertheless, they pose assumptions the design distribution, which can be problematic if actual data does not coincide. To gain further flexi- bility, various deep-learning-baseee
models are developed to generate knockoffs from distributions beyond parametric setup. DDLK [55] and sRMMD [38] utilize different metrics to measure the distances between the original and the knockoff covariates. They apply different regularization terms to impose the "swap property". He et al. (24) introduced a KnockoffScreen procedure to generate multiple knockoffs improve the stability by minimizing the variance during knockoff construction. KnockoffGAN (27 and Deep Knockoff (47] take advantage of the deep learning structures to create likelihood-free free generative models for the knockoff generation.
Despite the flexibility to learn the data distribution, deep-learning- -based models suffer from major drawback. Knockoff generations based on distribution-free sampling methods such generative adversarial networks (GAN) 21 B
tend to overfit, namely to learn the data X exactly. The reason that the notion of swap property for continuous distributions is not well defined at the sample level. To satisfy the swap property, one needs to independently sample X, from the conditional law Px, (-IX-j). where X-j denotes the vector (XI...., X3 x, X1+1,, Xp). At the sample level, each realization of X-j x²= , is almost surely different and only associates to one corresponding sample X, . causing the conditional law degenerate sum of Diracs. As result, minimizing the distance between (X, X) and (X, X swap(() will push X towards X and introduce high collinear- ity that makes the feature selection powerless, i.e., with high type II error. To tackle this issue, DDLK (55) suggests an entropic regularization. Yet it still lacks power and is computationally expensive.
2.3 Boost Power by reducing reconstructability
The issue of lacking power in the knockoff selection is solved in the Gaussian case the knowledge of both mean and covariance of ~ NN,, Σ), the swap property is easily satisfied
[4 Assuming
by setting X, ~ N(ug Eny) and Σi = Var(X,, Xz), for * j. i,j E [p]. Barber & Candès 4 originally propose to minimize Var(X,, X,) for all j E [p] using semi defindefi programming (SDP), prevent X, to be highly correlated with X,, However, Spector & Janson [54] observed that the SDP knockoff still lacks feature selection power, as merely decorrelating X3 and X, is not enough, and (X, X) can still be (almost) linearly dependent in various cases. This referred to as high reconstructabilityy in their paper, which can be considered collinearity (see Appendix for more details). To tackle the problem, (54) proposed to maximize the expected conditional variance Va,, X-j, X), which admits close-form solution whenever X is Gaussian.
population counterpart of
3 Method
DeepDRK in Figure provides a novel way to generate knockoff X while reducing the recon- structability (see Section 2.3) between the generated knockoff X and the input X for data with
'The "reconstructabilitty" describes how easy a variable X, can be constructed deterministically from X-; and X [54]
3

---

complex distributions. The generated knockoff can then be used to perform CDR-contrlleed fea- ture selection following the Model-X knockoff framework (see Section 2.1). Overall, DeepDRK contains two main components. It first trains waasformer-based deep learning model, referred to as Knockoff Transformme (KT)), to obtain the swap property and reduce the reconstructability of the generated knockoff. This is achieved by incorporating adversarial attacks with multi- swapees. Secondly, a dependency regularized perturbation technique (DRP) is developed to further reduce the reconstructability for X post training. We will elaborate on these two components in the following two subsections. this section, we slightly abuse the notation such that X and X also denote the corresponding data matrices.
3.1 Training with Knockoff Transformer and Swappers
The KT aims to generate knockoff's denoted by Xg. which are parameterized by transformer network with parameters 0. The loss for training the KT contains a swap loss (SL) LSL. which enforces the swap property, and dependency regularization loss (DRL) CDRL. which controls the reconstructibility of the knockoff. K different neural network parameterized swappers, denoted by [Sw,)] K are used to test whether the generated knockoffs satisfy the swap property. Thus, the KT is trained adversarially according to the following objective,
min max (X, 1 ( + LDRI (X, Xo)). WipowK
(5)
Note that we use Xg and X interchangeably; however, the former emphasizes that the knockoff depends on the model weights 0. We discuss each loss function below. Details on the network architectures for KT and swappers are deferred to Appendix B.1 The training algorithm in Appendix B.2
3.1.1 Swap Loss
The swap loss is designed to enforce the swap property and is defined as follows:
Lx, (o =
K Σ (WD((X,Xo))) Xo)
1
A REx(X, Xo, (S + d2 Luwapper ( (1Swi 1 1om
(6)
where d and 22 are hyperparameters.
The first term Eq. O6) uses the sliced Wasserstein distance (SWD, see Appendix [for definition) to measure the distance between a pair of joint distributions for (X,Xo) and (X, Xa)s., where the swapper is parameterized by w; We sum over K SWDs computed for the distributions modified under different swappers to capture the effects of multiple swap attacks. We utilize SWD to com- pare distributions because it excels in handling complex data distributions and is computationally efficient [13 28 14
Sudarshan et al. (55] introduced a single swap attack parameterized by a neural network. However, we observe that minimizing the worst case swap for all B [p], as suggested by (55), cannot guarantee the swap property of Xg. To address this limitation, we introduce a "multi-swapper" setup that uses multiple swappers to enforce the swap property. And this leads to the introduction of the second and the third terms in the objective in Eq. (6).
The second term in Eq. (6) REX(X, Xo, (SW,) evaluates the variance of the sliced Wasserstein
distances SWD((X, Xo), (X, Xo)ss. with K realizations of (30]. When REx(X, Xo, (Sw; }K 0, the sliced Wasserstein distances are identical across all swappers. Therefore, complementary to the first term, minimizing this term improves the adherence of swap property for the generated X
The third term in Eq. O is introduced to avoid mode collapse on the parameters w; of different swappers and ensure each swapper characterizes a different adversarial environment:
Cswapper (1S,, 3k
Σ (G.j)EC
sim Sw,, Sw,),
|

---

where C= ((i,)) [K] f), and sim(:.-) ) is the cosine similarity between the weights of a pair of different swappers. Without this regularization, all swappers could collapse single mode such that the multi-swapper scheme reduces to a single-swapper setup.
Overall, the swap loss LsL enforces the swap property via the novel multi-swapper design. Such design provides a more robust assurance of the swap property through multiple adversarial swap attacks, which is shown in the ablation studies in Appendices I.2 and I.3
3.1.2 Dependency Regularization Loss
pursuing the swap property at the sample level often leads to severe overfitting of Xo, i.e., pushing Xe towards X. which results in high collinearity in feature selection. To address this, the DRL is introduced to reduce the reconstructability between X and X:
As discussed in Section 2.2
LDRL (X, Xx) Aa (WC((XXX))
88)
where Ay is hyperparameter. The SWC term in Eq. (8) refers to the sliced Wasserstein correla- tion [33], which quantitatively measures the dependency between two random vectors in the same space. More specifically, let Z1 and Zg be two p-dimensional random vectors. SWC( Z1. Zz) = 0 indicates that Z1 and Zz are independent, while SWC( 11. Z2) = suggests a linear relationship between each other (see Appendix D for more details SWC). In DeepDRK,
minimize SWC to reduce the reconstructability. a procedure similar to (4 The intuition is as follows. If the joint distribution of is known, then for each j e [p]. the knockoff X, should be sampled from P)(-|X-)), making X3 and X3 less dependent. In such case the swap property is ensured, and collnearty/rcoonttuttbbbttt
reduced due to independence. As we do not have access to the joint law, we want the variables to be less dependent. Since collinearity exists with in X, merely decorrelate X, and X, is not enough. Thus, we minimize SWC to reduce the dependence between X and X. We refer readers to Appendix A for more discussions.
3.2 Dependency Regularization Perturbation
Empirically we observe a competition between LSt and CDRLiN in Eq. (5). which adds difficulty to the training procedure. Specifically, the LSL is dominating and the LDRL increases quickly after short decreasing period. We are the first to observe this phenomenon in all deep-learning based knockoff generation models when one tries to gain power [47] 55 38 27). We include the experimental evidence in Appendix E. We suggest the following explanation: minimizing the swap loss, which corresponds to FDR control, is the same controlling Type error. Similarly, minimizing the dependency loss is to control Type II error. With a fixed number of observations, it is well known that Type I error and Type II error can not decrease the same time after reaching a certain threshold. In the framework of model-X knockoff, we aim to boost as much power as possible given the FDR is controlled a certain level, similar idea as the uniformly most powerful (UMP) test 12 For this reason, we propose DRP as post-trainne post-training technique to further boost power.
DRP is a sample-level perturbation that further eliminates dependency between X and and the knockoff. More specifically, DRP perturbs the generated X, with the row-permuted version of X, denoted as Xpp. After applying DRP, the final knockoff X-DRP becomes:
DRP 88,
(1 a.) X8 an Xip-
(9)
where an is preset perturbation weight, n is the sample size, and a, - O when n - oc. In the following, we remove n or 0 whenever the context is clear. XDRP has a smaller SWC with X, since X,p is independent of X. Despite the perturbation increases the swap loss, such impact is negligible when the sample size is large. More specifically, we present the following Lemma 33. and Proposition
3.2 Let P., and PB denote the empirical joint distribution of the sample X and X before and after the swap: (X,X) P... (X, X) )awap(B) PB And P and pB denote their corresponding population distributions.
Lemma 3.1. Under mild conditions(4II. the slice Wasserstein distance between the empirical distributions of (X,X) and (X,X) X)ssap(B)) and their corresponding population distributions is of scale O(n 1/2),, i.e., SWD(P,,, p SWD( PB) On
5

---

Sample Size: 200
Sample Size: 2000
Power
FDR 2200222000
Clatton Rapotenfal
Caprix Ghann & Gensa
Copraa Joe tapeneeb3)
Gunano
Mattone
Copula: Chay tir Guznn Rapatenfol
Copata Joe & Conton
Dalasets
Datasets
Figure 2: Power and FDR for different knockoff models on the synthetic datasets with 3 Rademacher(0.5), The red horizontal line indicates the 0.1 FDR threshold.
P
15./N
Proposition 3.2. Let On & O in Eq. (X, (DRR) BB
and denote XX,XDRP)
PR,DRP
PB 7.DRP- Then SWD(P fo D.DRP) SWD(P, pB) when n DC.
The proofs can be found in Appendix Lemma suggests a general case where the swap loss enforced at the sample level differs from that the population level from a term scale 1/2 which vanishes when
Proposition 3.2 provides the rationale the proposal of the perturbation technique in Eq. (9). such that the DRP term is negligible asymptotically. Besides the theoretical justification, we also empirically show in Appendix G that the perturbation is beneficial.
Experiment
We assess the performance of DeepDRK against several benchmark models under three experimental setups: 1. fully synthetic setup where both the input variables and the response variable follow predefined distributions; 2. semi-synthetic setup, in which the input variables are derived from real- world datasets and the response variable is generated based on known relationships with the inputs; and 3. feature selection (FS) using a real-world dataset. The experiments are designed to encompass a range of datasets with varying // n ratios and distributions of X, aiming to provide a comprehensive evaluation of model performance, The benchmark models are Deep Knockoff 47 DDLK (55)
KnockoffGAN [2 and sRMMD 38 with links the code implementation listed in the footnote. The implementation details for the training. feature selection, and data preparation available in Appendix H In the following. we describe the datasets and the associated experimental results. We also consider the ablation study to illustrate the benefits obtained by having the following proposed terms: REx, Lawapper. the multi-swapper setup, and the dependency regularization perturbation. Empirically, these terms help improve power and control the FDR. Due to space limitation, we defer details for the ablation studies to Appendix 1.2 and I.3 DeepDRK is implemented in PyTorch 455 and is accessible at: httph:tt//// thub. http://oooo02//////g//// DeepDRK
Sample Size: 200
Sample Size: 2000
4.1 The Synthetic Experiments
Evedei
To properly evaluate the performance, we follow well-designed experimen- tal setup by [55 38 to generate differ- ent datasets specified by Y). Here X R² is the collection dependent variables that follows pre-defined dis- tributions. Y E is the response variable that is modeled N(XT 3, 1). The underlying true / a p-dimensional vector, where each entry is drawn independently from the distribution I5.0%
Plano
Paass
Figure 3: Power and FDR for different knockoff models on the mixture of Gaussian data different Pbase setups. The red horizontal line indicates the 0.1 FDR threshold. This figure is complementary to Figure 2 for including two addi- tional Gaussian mixture data with higher Phose values.
Y
Rademacher(O.5).

---

Method
FDR
Power
mean
std
median
5% quantile
95% quantile
meun
std
median
5% quantile
95% quantile
DDLK
0.772
0.025
0.781
0.726
0.791
0.971
0.035
0.983
0.911
0.995
Deep Knockoff
0.735
0.028
0.740
0.692
0.769
0,999
0.001
0.999
0.997
1.000
200
KnockoftGAN
0.390
0.110
0.396
0.230
0.550
0.971
0.038
0.986
0.904
0,997
<RMMD
0.720
0.047
0.741
0.640
0.752
0.998
0.002
0.998
0.994
1.000
DeepDRK
0.116
0.040
0.100
0.086
0.187
0.791
0.039
0.804
0.720
0.824
DDLK
0.725
0.050
0.734
0.667
0.778
1.000
0.000
1,000
1.000
1.000
Deep Knockotf
0.626
0.092
0:672
0,460
0.694
1.000
0.001
1,000
0.997
1,000
2000
KaockoftGAN
0.118
0.014
0.114
0.102
0.139
0.973
0.012
0.978
0.953
0.986
<RMMD
0.658
0.060
0.644
0.583
0.736
1,000
0.001
1,000
0.998
1,000
DeepDRK
0
0.018
0.059
0.110
0.973
0.011
0.978
0.956
Table I: Comparison of different methods on FDR and power acTOss different ((T) T2, 33) for the
components in the Gaussian mixture setup.
Compared the previous works 55 38 which consider J/ the scaling factor for the Rademacher distribution, we reduce the magnitude of / by a factor of 15. This is because we find that in the original setup, the 3 scale is too large such that the feature selection enjoys high powers and low FDRs for all models. To compare the performance of the knockoff generation methods on various data, we consider the following distributions for X:
Gaussian mixture: We consider Gaussian mixture model X ~ E is the proportion of the k-th Gaussian with (T), 22, T3) = 0.2,0.2.0.4 Hs E Ip denotes the mean of the k-th Gaussian with μı 1 20 (k 1), where 1, is the p-dimensional vector that has universal value for all entries. Σς RPxP is the covariance matrices whose (i,,))-th entry
_k-1 (ANN, Σx), where
taking the value P | where Ps Phase perform two additional tests. The first one focuses on a mixture of Gaussians data of various phase to study the feature selection performance with highly correlated features. Namely, additional Pbase E {0.7, 0.8). The second one explores the effect of ( [T1, T22 Ty) to the feature selection performance. In this case, uniformly draw 10 sets of (T1, T2, T3). and evaluate the FS performance of all the models considered in this paper. The values of the mixture weights are presented in Table in Appendix 1
k-0.1 and Phase 0.6. Besides this experiment, we further
consider
Copulas 49 We further use copula to model complex correlations within X. To the best of our knowledge, this is first attempt to consider complex distributions other than the Gaussian mixture model in the knockoff framework. Specifically, we consider two copula families: Clayton, Joe with the consistent copula parameter of 2 in both cases. For each family, we consider two candidates for the marginal distributions: a uniform distribution (using the identity conversion function) and an exponential distribution with a rate of 1. We implement the copulas according to PyCop
We consider the following (n. p) setups: (200, 100) and (2000, 100). This is in contrast to existing
works, which consider only the (2000, 100) as the smallest sample size setup (55 38 47 Our goal to demonstrate the consistent performance of DeepDRK across various p/ n/ ratios, particularly when the sample size is small.
Results: Figure 2 compare FDRs and powers for all models across the datasets with two different setups for B. Figure[Jshows that DeepDRK consistently controls the false discovery rate (FDR) compared to other benchmark models across various data distributions and p/n ratios, with the exception of few cases where the FDR exceeds the 0.1 threshold in the small sample size (n = 200) scenarios. Other models, though being able reach higher power, comes at a cost of sacrificing FDR, which contradicts to the UMP philosophy (see Section 3.2). We also evaluate the performance on mixture of Gaussians with increasing Phases indicating a higher correlation among the input variables. Note that the mixture of Gaussians example in Figure 2 has Phuse 0.6. The results for Pouse E {0.7, 0.8) are presented in Figure 3 Compared to other baseline models, DeepDRK maintains the lowest FDRs while achieving competitive powers across all Pbase values, highlighting its robustness to correlations in X. Overall, the results demonstrate the ability of DeepDRK in consistently performing FS with controlled FDR compared to other models across a range of different datasets, p/r ratios, and feature correlations in X. In addition, we present common statistics for the FDR and power on the mixture of Gaussians experiment with 10 different sets of (T1, T2, T3) in Table D It is clear that DeepDRK improves the robustness in maintaining FDR across various configurations of the Gaussian mixture models compared to existing approaches.
7

---

DeepORK
3RMMD
Doep Kneckeet
KmeckofGGNN
DOLK
Dalouts
Creade Corter Genee
Copala jos lanee
Nocroll
Nell
Norzell
Nall
Norral
Nell
Norreil
Nel
Nill
Figure 4: The knockoff statistics (wy) for different knockoff models on the synthetic datasets with P
3 15.VN Rademacher(O.5), Each bar the plot represents the mean of the null/nonnull knockoff statistics averaging on 600 experiments. The error bar indicates the standard deviation. The sample size is 200.
Sample Size: 200
Ceumas Saata
Capua, Chette & Exparesss
Copala Joe Expeeezia
Capua. Canzza
Copua Chaytee Cazaaa
loep Kanduut
LOLK
Sumple Siow: 2000
Soka
Capida Chigte Expexesia
Capuaa Chaptce Cazea
1.0
15
FDR
FDR
FDR
10R
F13
Figure 5: Scatter plots of Power against FDR for different datasets and models. The red vertical P P P line indicates the 0.1 FDR threshold. Different scales for (e.g.. 5- 10-77² 15 and P indicated by different marker styles. Different models are indicated by different colors.
20 are
colorrs.
To understand why DeepDRK outperforms other baseline models, we consider measuring the distribution of the knockoff statistics, i.e., wy for both nonnull and null features of X. Fan et al. [17] and Candès et al. [11] pointed out that a good knockoff requires the corresponding knockoff statistics concentrate symmetrically around zero for the null features and maintain high positive values for the nonnulls. However, theoretical analysis on the goodness of FDR or power requires access to the true knockoff X [17 to compare the distribution of wyys with the ground truth, which is infeasible for non- Gausian data. Nevertheless, we can still examine the distribution of the knockoff statistics as a surrogate to analyze model performance in terms of false discovery rate (FDR) or power, given the necessary properties of the knockoff statistics mentioned earlier.
Figure shows the means and standard deviations of the empirical distributions of the knockoff statistics wy for both null and nonnull variables across different datasets and models. Clearly, compared to other benchmarks, DeepDRK maintains high values of wy for the nonnulls and relatively symmetric values around zero for the nulls. All other models experience positive shifts the null statistics to some extent. Positive shifts in the null statistics lead to a degeneracy in performance because the threshold selection rule for false discovery rate (FDR) is based on the negative values of wy's 's (see Eq. (3)) This has two negative impacts, one on the FS threshold and the other on its subsequent FS process. First, according approach zero, as there are fewer null statistics remaining on the negative side, and those that do remain have smaller amplitudes. Subsequently, a lowered threshold leads an increase in false positives, phenomenon that becomes more pronounced with positive shifts in the null statistics. As shown Figure 4 the wy values calculated using DeepDRK are centered around zero for the nulls, while exhibiting large positive values for the nonnulls. This aligns with the results in Figure 2 which demonstrate that DeepDRK effectively achieves good FDR control and high power. Due to space limit, we defer the results for n 2000 and the comparison of w statistics for the Gaussian correlation setup to Appendix I.3
Eq. (3), the shift causes the chosen threshold to
To further verify the performance consistency of the proposed method, we include a comparison on different 3 scales. Specifically, we consider 4 different sets of 33 scales, i.e., u.mm To To-/m and 20 for the previously considered data distributions. The results are summarized in Figure It is observed that in all cases considered with various & scale, the proposed DeepDRK successiully
8

---

maintains relatively low FDRs with higher power, compared to baseline methods. This phenomenon especially pronounced with low sample size (n 200).
In addition to the above results, we provide the measurement of the swap property in Appendix The evaluation of model runtime is also included in Appendix I.4
4.2 The Semi-synthetic Experiments
We consider semi- symi-syythetic study with de- sign X drawn from two real-world datasets and use X to simulate response Y. The
Synthetic Rule: Tanh
Synthetic Rule: Linear
g S
FDR Power CERFEERRE ft
Power
first dataset contains single-cell RNA se- quencing (scRNA-seq) data from 10x Ge-
nomies | Each entry in X E RRpp rep-
resents the observed gene expression of p
genes in n cells. We refer readers to [23
10.4
and [ for more background. Following
0.17
RNAASqq
RNASSq
the same preprocessing in [23] we obtain the final dataset X with
10000 and The preprocessing of X and the synthesis of Y (denoted as "Linear" and "Tanh" cases) are deferred to Appendix H.3.2
Figure 6: Power and FDR for different knockoff models on the semi-synthetic RNA dataset. The red horizontal line indicates the 0.1 FDR threshold.
p = 100 8
The second publicly available datase is from real case study entitled "Longitudinal Metabolomics of the Human Microbiome in Inflammatory Bowel Disease (IBDy" [35]. The study seeks to identify important metabolites of two representative discases of the inflammatory bowel discase (IBD); ulcerative colitis (UC) and Crohn's disease (CD). Specifically, we use the C18 Reverse-Phase Negative Mode dataset that has 546 samples and 91 metabolites. To mitigate the effects of missing values, we preprocess the dataset following a common procedure to remove metabolites that have over 20% missing values, resulting in 80 metabolites. We normalize the data matrix entry-wise have mean and unit variance after a log transform and an imputation via the n-nearest neighbor algorithm following the same procedure in (8 Finally, we synthesize the response Y with the real dataset of X via Y ~ N((TT T 9, 1). where the entries of 3 drawn independently from one of the following three distributions: Unif(O. (0, 1), and Rademacher((.5) in three separate experiments.
Results: Figure 6 and 7 compare the feature selection performance on the RNA data and the IBD data respectively. In Figure 6 we observe that all but DDLK are bounded by the nominal 0.1 FDR threshold in the "Tanh" case. However, KnockoffGAN and sRMMD have almost zero power The power for Deep Knockoff is also very low compared to that of DeepDRK. Although DDLK provides high power, the associated FDR is not controlled by the threshold. In the "Linear" case, almost all models have well controlled FDR. among which DeepDRK provides the highest power. Similar observations can be found in Figure For the IBD data generated under the aforementioned synthesis rules, it is clear that all models except DDLK achieve well-controlled FDR. Apart from DDLK, DeepDRK consistently demonstrates the highest power. These results further underscore the potential of DeepDRK for real-world data applications.
4.3 A Case Study
setups. we carry out case study with real data for both design X and response Y. in order to qualitatively evaluate the selection performance of DeepDRK. In this subsection, we use the IBD dataset [35] with the empirical response. The response variable Y is categorical: equals if given sample is associated with UC/CD and O otherwise. The covariates X is identical to the second semi-synthetic setup considered in Section 4.2) To properly evaluate results with no ground truth available, we search for the evidence of the IBD-associated metabolites using the existing literature. Specifically, we use three sources: 1. metabolites that are explicitly
Besides
https://kb.. htttp:://bb.1xxgennca www.eeomiicc.ccm/////eeeeeeee www/cc/mm/nmeeeeeeee hnnuu
The data processing code is adopted from this repo: htttp:://gsttuub.com///eeeett Toovellet/rree//atter/aaaaa
/tttps://www.i wwwp://wwbboooccccoooooooo.cc.c. under the project DOL: 10.21228/M82T15.

---

Synthetic Rule: Lineer & Normal
Synthetic Rule: Linear & Rade.
Syathetic Rule: Linear & Uniform
1.
6.0
1.0
Poweer
Power
8
92
FDR 22052282
9.1
18D
1BD
Figure 7: Power and FDR for different knockoff models the semi-synthetic IBD dataset. The red horizontal line indicates the 0.1 FDR threshold.
Model
DeepDRK
Deep Knockoff
sRMMD
KnockofIGAN
DDLK
Referenced / Identified
19/23
15/20
5/5
12/14
17/25
Table 2: The number of Interatue-suupoorted supported metabolites among the identified metabolites vs. the number of identified metabolites.
documented to have associations with IBD, UC, or CD in the PubChem database
10 2. metabolites
that are reported in the existing peer-reviewed publications; 3. metabolites that are reported in pre-prints. We identify 47 metabolites that are reported to have association with IBD. All referenced metabolites are included in Table in Appendix H...3
Our DeepDRK model training and knockoff generation are the same as before (see Table in Appendix H.T. Likewise, generate knockoff for the benchmark models, follow their detault setups. During the FS step. however, we use 0.2 as the FDR threshold instead of 0.1, and apply different algorithm- DeephniNK [37] that included in the knockpy T1 library- generate wy. The values are subsequently used to identify metabolites. We choose DeepPINK over the previously considered ridge regression due to the nonlinear relationships between metabolites X and responses Y in this case study.
We compare the FS results with the 47 literaturessupporet metabolites and report the number of selections in Table 2| A detailed list of selected features for each model can be found in Table 9 Appendix I.5 From Table it is clear that, compared the benchmark models, DeepDRK identifies the largest number of referenced metabolites while effectively limiting the number of metabolites reported in existing literature (see Table in Appendix H.3.3). The sRMMD model achieves the lowest false discovery rate, but this comes at the cost of missing significant number of documented metabolites. Since there is no ground truth available, the results here should be viewed and analyzed qualitatively.
5 Conclusion
In this paper, we introduce DeepDRK, a deep learning-based knockoff generation pipeline consisting of two steps. First, it trains Knockoff Transformer with multiple swappers to achieve the swap property while reducing reconstructability. In the post-training stage. a deependnncyregullate perturbation is applied to further enhance power with controlled FDR. DeepDRK effectively balances FDR and power, which compete with each other at the sample level. To the best of our knowledge, this relationship has not been previously reported in the literature. Empirically. DeepDRK demonstrates the ability to maintain both controlled false discovery rates (FDR) and high power across various data distributions and different p/n ratios. Additionally, provide insights into the distribution of knockoff statistics, which elucidate the reasons behind DeepDRK's consistently strong performance. The numerical results indicate that DeepDRK outperforms existing deep learning- based benchmark models. Experiments with real and semi-synthee data further highlight the potential of DeepDRK for feature selection tasks involving non- non-Gaussian data.
"attpe://pubchhe.nncb nlm. nih. nov. https://amspecctrr1000 withub. htt.uub...///oooyy./...
10

---

Acknowledgments
This research was partially supported by Alfred Sloan foundation and NSF # 1934757.
References
[1] Agarwal, D., Wang, J., and Zhang, N. R. Data denoising and post-denoising conrections in single cell RNA sequencing. Statistical Science, 35(1):112 128, 2020.
[2] Ananthakrishnam, A. N., Luo, C., Yajnik, V., Khalili, H., Garber, J. J., Stevens, B. W., Cleland,
T., and Xavier, R. J. Gut microbiome function predicts response to anti inti-integrin biologic therapy in inflammatory bowel diseases. Cell host microbe, 2 21(5):603-610, 2017.
[3] Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In ICML, pp. 214-223, 2017.
[4] Barber, R. F. and Candès, E. J. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055-2885, 2015.
[5] Bauset, C., identifying potential biomarkers for inflammatory bowel disease. Journal of Clinical Medicine, 10(4):622, 2021.
L., and Cosín- Rosse. J. Metabolomics as a promising resource
[6] Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal the Royal stafisfical society: series B (Methodological), 57(1):289-300, 1995.
[7] Bin Masud, S., Jenkins, C., Hussey, E., Elkin-Frankston, S., Mach, P., Dhummakupt, E., and Aeron, S. Utilizing machine learning with knockoff filtering extract significant metabolites in Crohn's ' disease with publicly available untargeted metabolomics dataset. Plos one, 16(7): c0255240, 2021.
[8] Blaker, P. A., Arenas-Hernandez M.. Smith, M. A., Shobowale-Bakre, E. A., Fairbanks, L., Irving, P. M., Sanderson, J. D., and Marinaki, A. M. Mechanism of allopurinol induced TPMT inhibition. Biochemical pharmacology, 86(4):539 -4--539997779 2013.
[9] Bonneel, N., Rabin, J., Peyré, G., and Plister, H. Sliced and Radon Wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51:22-45, 2015.
[10] Bonnotte, N. Unidimensional and evolution methods for optimal transportation. PhD thesis, Université Paris Sud-Paris XI: Scuola normale superiore (Pise, Italie), 2013.
[11] Candès, E., Fan, Y., Janson, L., and Lv, J. Panning for gold: 'Model-X' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 80(3):551-577, 2018.
[12] Casella, G. and Berger, L. Statistical inference. Cengage Learning, 2021.
[13] Deshpande, I., Zhang. Z., and Schwing, G. Generative modeling using the sliced Wasserstein distance. In CVPR, pp. 3488-3499 2018.
[14] Deshpande, I., Hu, Y.-T., Sun, R., Pyrros, A., Siddiqui, N., Koyejo, S., Zhao, Z., Forsyth, D., and Schwing, A. G. Max-sliced Wasserstein distance and its use for GANs. In CVPR, pp- 10648-10656, 2019.
[15] Dhal, P. and Azad, C. A comprehensive survey on feature selection in the various fields of machine learning. Applied Intelligence, 42(4):4543-4581 2022.
[16] Dosovitskiy, A., Beyer, I.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
[17] Fan, Y., Gao, I.., and Lv, J. ARK: Robust Knockoffs inference with coupling. arXiv preprint arXiv:2307.04-000 2023.

---

[18] Fretland, D., Widomski, D., Levin, S., and Gaginella, T. Colonic inflammation in the rabbit induced by phorbol- phorbol-12-myristate manyyristate---acetttt Inflammation, 1402):43--10, 1990.
[19]
Gavrilov, Y., Benjamini, Y., and Sarkar, S. K. An adaptive step-down procedure with proven FDR control under independence. The Annals of Statistics, 37(2):619-629) 2009.
[20] Gimenez, J. R., Ghorbani, A., and Zou, J. Knockoffs for the mass: new feature importance statistics with false discovery guarantees. In AISTATS, pp. 2125-2133, 2019.
[21] Goodfellow, I., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 2020.
[22] Guyon, I. and Elisseeff, A. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar): 1 118, 2003.
[23] Hansen, D., Manzo, B., and Regier, J. Normalizing flows for knockoff-free controlled feature selection. In Advances in Neural Information Processing Systems, volume 35, pp. 16125-16137, 2022.
[24] He, Z., Liu, L., Wang, C.,, Le Guen, Y., Lee, J., Gogarten, S., Lu, F., Montgomery, S., Tang, H., Silverman, E. K., et al. Identification of putative causal loci in whole-genome sequencing data via knockoff statisties. Nature communications, 12(1):1-18, 2021.
[25] Hristache, M.. Juditsky, A., and Spokoiny, V. Direct estimation of the index coefficient in single-index model. Annals of Statistics, (9(3))595-623 2001.
[26] Jang. E., Gu, S., and Poole, B. Categorical reparameterizzttion with Gumbel-softmax. In ICLR, 2017.
[27] Jordon, J., Yoon, J., and van der Schaar, M. KnockoffGAN: Generating knockoffs for feature selection using generative adversarial networks. In ICLR, 2018.
[28] Kolouri, S., Nadjahi, K., Simsekli, U., Badeau, R., and Rohde, G. Generalized sliced Wasserstein distances. NeurIPS, volume 32, 2019.
[29] Koon, H. W. novel orally active metabolite reverses Crohn's disease-associated intestinal fibrosis. Inflammatory Bowel Diseases, 28(Supplement..))566--52. 2022.
[30] Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A. Out-of-distribution generalization via risk extrapolation (REx). In ICML, pp- 5815-5826, 2021.
[31] Lavelle, A. and Sokol, H. Gut microbiota-derriedd metabolites as key actors in inflammatory bowel disease. Nature reviews Gastroenterology & hepatology, 17(4):223 227-272,,,,, 2020.
[32] Lee, T., Clavel, T., Smirnov, K., Schmidt, A., Lagkouvardos, I., Walker, A., Lucio, M., Michalke, B., Schmitt- Schmitt-Kppplin, P., Fedorak, R., et al. Oral versus intravenous iron replacement therapy distinctly alters the gut microbiota and metabolome in patients with IBD, Gut, 66(5):863-87,, 2017.
[33] Li, T., Yu, J., and Meng, C. Scalable model-free feature screening via sliced- Wasserstein dependency. Journal of Computational and Graphical Statistics, 1501-1550- 2023.
[34] Liu, Y. and Zheng, C. Auto-encoding knockoff generator for FDR controlled variable selection. arXiv preprint arXiv: 1809.00775 2018
[35] Lloyd-Price, J., Arze, C., Ananthakrishman, N., Schirmer, M., Avila-Pacheco, Poon, T. W., Andrews, E., Ajami, N. J., Bonham, K. S., Brislawn, C. J., et al. Multi-omics of the gut microbial ecosystem in inflammatory bowel diseases. Nature, 569(7758))655 662, 2019.
[36] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In ICLR, 2019.
[37] Lu, Y., Fan, Y., Lv, J., and Stafford Noble, W. DeepPINK: reproducible feature selection in deep neural networks. In NeurIPS, volume 31, 2018.
12

---

[38] Masud, S. B., Werenski, M., Murphy, J. M., and Aeron, S. Multivariate soft rank via entropy- regularized optimal transport: Sample efficiency and generative modeling. Journal of Machine Learning Research, 24( (4(160)11665, 2023.
[39] Mehta, R. S., Taylor, Z. L.. Martin, L. J., Rosen, M. J., and Ramsey, L. B. SLCOIBI *15 allele associated with methotrexate-induiee disease. Clinical and translational science, 15(1):63-00 69, 2022.
nausea in pediatric patients with inflammatory bowel
[40] Minderhoud, I. M., Oldenburg, B., Schipper, M. E. I., Ter Linde, J. M., and Samsom, M. Serotonin synthesis and uptake in symptomatic patients with Crohn's disease in remission. Clinical Gastroenterology and Hepatology, 5(6):714-1) 1-0144--0,-00 2007.
[41] Nadjahi, K., Durmus, A., Simsekli, U., and Badeau, R. Asymptotic guarantees for learning generative models with the Saieed-Wasserstein distance. In Advances in Neural Information Processing Systems, volume 32, 2019.
[42] Narasimhan, R. L., Throm, A. A., Koshy, J. J., Saldanha, K. M. R., Chandranpillai, H., Lal, R. D., Kumravat, M., K. M., A. K., Batra, A., Zhong, F,, et al. Inferring intestinal mucosal immune cell associated microbiome species and microbiota-derived metabolites inflammatory bowel disease. bioRriv, 2020.
[43] Nies, T. G., Staudt, T., and Munk, A. Transport dependency: Optimal transport based depen- dency measures. arXiv preprint arXiv:2105. 02070 2021.
[44] Nuzzo, A., Saha, S., Berg, E., Jayawickreme, C., Tocker, J., and Brown, J. R. Expanding the drug discovery space with predicted taage interactions. Communications biology, 4 (1):1-11, 2021.
[45] Paszke, A., Gross, S., Massa, F. Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, volume 32, 2019.
[46] Qin, X. Etiology of inflammatory bowel disease: a unified hypothesis. World journal of gastroenterology: WJG, 18(15):1708, 2012.
[47] Romano, Y., Sesia, M., and Candès, E. Deep knockoffs. Journal of the American Statistical Association, (15552))1861-18
2020.
[48] Saber, S.. Khalil, R. M., Abdo, W. S., Nassif, D.,, and El-Ahwany, E. Olmesartan ameliorates chemically-indueed ulcerative colitis in rats via modulating NF&B and Nrf-2/HO-1 signaling crosstalk. Toxicology and applied pharmacology.
2019.
[49] Schmidt, T. Coping with copulas. Copulas-Frie From theory to application in finance, 3:1-34, 2007.
[50] Scoville, E. A., Allaman, M. M., Brown, C. T., Motley, A. K., Horst, N., Williams, C. S., Koyama, T., Zhao, Z., Adams, D. W., Beaulieu, D. B., et al. Alterations in lipid, amino acid, and energy metabolism distinguish Crohn's 's disease from ulcerative colitis and control subjects by serum metabolomic profiling. Metabolomics, 14(1):1-12, 2018.
[51] Sesia, M., Sabatti, C., and Candès, E. J. Gene hunting with hidden Markov model knockoffs. Biometrika, 106(1):1-18, 08 2018.
[52] Soderholm, J. D.. Oman, H., Blomquist, L., Veen, J., Lindmark, T., and Olaison, G. Reversible increase in tight junction permeability to macromolecules in rat ileal mucosa in vitro by sodium caprate, a constituent of milk fat. Digestive diseases and sciences, 43(7):1547-1552, 1998.
[53] Söderholm, J. D., Olaison, G., Peterson, K. H., Franzen, L. E., Lindmark, T., Wirén, M., Tagesson, C., and Sjödahl, R. Augmented increase in tight junction permeability by luminal stimuli in the non-inflamed ileum of Crohn's 's disease. Gut, 50(3):307-313,, 2002.
[54] Spector, A. and Janson, L. Powerful knockoffs via minimizing reconstructabilityy The Annals of Statistics, 50(1 (0(1):252-276, 2022.
[55] Sudarshan, M., Tansey, W., and Ranganath, R. Deep direct likelihood knockoffs. NeurlPS, volume 33, 2020.
13

---

[56] Suhre, K., Shin. S.-Y., Petersen, A.-K., Mohney, R. P., Meredith, D., Wägele, B., Altmaier, E., Deloukas, P.. Erdmann, J., Grundberg, E., al. Human metabolic individuality in biomedical and pharmaceutical research. Nature, 477(6362):54 60, 2011.
[57] Uchiyama, K., Odahara, S., Nakamura, M., Koido, S., Katahira, K., Shiraishi, H., Ohkusa, T., Fujise, K., and Tajiri, H. The fatty acid profile the erythrocyte membrane initial-onset inflammatory bowel disease patients. Digestive diseases and sciences, 58(5): 2013.
[58] Uko, V., Thangada, S., and Radhakrishnan, K. Liver disorders in inflammatory bowel disease. Gastroenterology research and practice, 2012(1):642923, 2012.
[59] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention all you need. In NeurIPS, volume 30, 2017.
[60] Villani, C. Optimal transport: old and new, volume 338. Springer, 2009.
[61] Wiesel, J. C. Measuring association with Wasserstein distances. Bernoulli, 28(4):2816-2832, 2022.
[62] Zhu, Z., Fan, Y., Kong, Y., Lv, J., and Sun, F. DeepLINK: Deep learning inference using knockoffs with applications to genomics. Proceedings of the National Academy of Sciences, 11806)):21046631118 2021.
14

---

Appendix
and A Reconstructability and Selection Power
The concept of reconstructability was introduced by [54] as population-level counterpart to what commonly referred to as collinearity linear regression. Under Gaussian design where X (00, Σ), reconstructability is high if E is not of full rank, implying that there exists some j such that Xj is almost surely linear combination of X-j. More generally, if there exists more than one representation of the response Y using the explanatory variable X, we qualitatively consider the reconstructability be high. High collinearity often impairs statistical power, and similarly, high reconstructability diminishes feature selection power. To illustrate this, we state linear version of Theorem 2.3 from [54], originally formulated for a more general single-index model 25
Theorem A.1. Let Y X53r + X-J3-J E, where J = [p] and e is centered Gaussian noise. Equivalently, this implies Y 1L X, 11BJ, Suppose there exists 88 such that Xy 55 almost surely. Then, denoting Y X5 X-J E, we have
((X,X),
((X, )..p)swap()) X*)
((x, d (X, ) J): Y)
(10)
Furthermore, in the frameeworkk ( let w((X, x)X)) and u vo((xX)), y*). Then, for all j e J,
P (w,, 0) IP (w} 0) E
(11)
Eq. (11) implies "no free lunch" situation for selection power when there is exact reconstructabilityy
To address the reconstructability issue, [54] proposed methods Gaussian design. The first is
the minimal variance-based reconstructability (MVR) knockoff, in which the knockoff X is sampled
to minimize the loss
LMVR
(12)
E Var (X, X X- x]]
Note that this is equivalent to maximizing E Var (x, xx-, for all j e [p]. Another approach is the maximum entropy (ME) knockoff, where X is sampled to maximize
LME = ʃʃ p(a; a) log(p(æ, (app(,,))) B)) di dr.
(13)
Under Gaussian design, both optimizations have closed-form solutions. Since is Gaussian, (X, X) must also be jointly Gaussian to satisfy the swap property. To optimize, one first calculates the covariance matrix using the SDP method in [4) yielding a diagonal matrix S. Then, both MVR and ME reduce to an optimization on S:
Σ j-1 X (Gs)
2p
and T.M(SS) log det (Gg
Σ log
(14)
Xg (Gs)
Although both methods show high power for feature selection, neither MVR nor ME can be directly extended to arbitrary distributions due the intractability of conditional variance and likelihood.
In DeepDRK, we consider regularizing with sliced-We Wasserstein- dependency correlation, which can be considered a stronger dependency regularization than entropy. A post-training pertur- bation also applied to further reduce collinearity. However, the theoretical understanding of how these affect the swap property and power remains open question.
15

---

B DeepDRK Model Architecture and Training Algorithm
B.1 Model Architecture
DeepDRK's knockoff Transformer (KT) model based on the popular Vision Transformer (ViT) 16] The primary difference that the input dimension is ID, not 2D, for X. We do not use patches as input: instead, we treat all entries of X to account for correlations between each pair of entries in the knockoff X. This structure is similar to the original Transformer [59]. Nonetheless, we retain other components from ViT, such as patch embedding, PreNorm, and ID-positional encoding [16]. Since knockoff generation requires a distribution, we feed X and uniformly distributed random variable Z with the same dimension as X to inject randomness. Specifically for the DeepDRK model, we use 8 attention heads, layers, and hidden dimension of 512 in the ViT model.
The swapper module, first introduced in DDLK [55, produces the index subset B for the adversarial swap attack. Optimizing knockoff against these adversarial swaps enforces the swap property. Specifically, the swapper consists of a matrix with shape 2 p (i.e., trainable model weights), where p is the dimension of X. This matrix controls the Gumbel-softtax distribution (26 for all p entries. Each entry is represented by binary Gumbel-softnax random variable (i.e., it can only take values of 0 or 1). To generate the subset B, sample by from the corresponding j-th Gumbel-softmax random variable, and B is defined as (j E [p] by = 1}. During optimization, we maximize Eq. (5) with respect to the weights Wi of the swapper S.. so that the sampled indices, with which the swap is applied, lead to a higher SWD in the objective in Eq. (5). Minimizing this objective with respect to Xa requires the knockoff to counteract the adversarial swaps, thereby enforcing the swap property. Compared to DDLK. the proposed DeepDRK utilizes multiple independent swappers (i.e., K 2). And we set the temperature for the Humbel-softmai to be 0.2.
B.2 Training Algorithm
In Algorithm| we provide pseudo code for training the Knockoff Transformer and the swappers (i.e., the first stage shown in Figure
Algorithm 1 DeepDRK Training
1: Input: Knockoff transformer No, denoted as go(-); swappers Sw: number of swappers K; learning rate a, for the swappers; learning rate ag for the knockoff transformer, early stop tolerance 77, number of epochs T'; batch size Bg; dataset D; swapper update frequency 3
2: Output: 0 for Xa
3: Split dataset ID into training set Dirain and validation set D.al
4: Initialize the knockoff transformer go(-) with random weights
5: Initialize swappers S.,, i = 1, .., K with random weights
6: Initialize the AdamW optimizer opt, with learning rate ag for ge(+)
7: Initialize the AdamW optimizer opt.., with learning rate a, for Sun, i 1,.... K
8: for 1 to T do
9: for = 1 to [Deall do B.
10:
Sample B, samples of from Dirain: X₁
11:
Generate knockoff X, = (X()
12:
Calculate Cst(Xi,, X,, (Sw, )===)) and LDRI (Xi, X1)
13:
0 - 0 + (St (Xi, (Swm 1(11) )+ + CDRL (X1, Xi))
/ mod 7 = 0 then
Wi wi opt Lst & (K 1,... K
16:
end if
end for
18:
Calculate the validation loss on Dvai: Csal CoL
19:
if Caal. + C meets early stop condition at tolerance 7) then
20:
break
21:
end if
end for
16

---

C From Wasserstein to Sliced Wasserstein Distance
The Wasserstein distance has gained popularity in both mathematics and machine learning due to its ability to compare different types of distributions [60] and its differentiability [3. Here, we provide its definition. Let X, be two I2 random vectors following distributions Px, Py with finite p-th moments. The Wasserstein-p distance between Px and Py is:
Wx(Px, Pr) = inf
(B(za))allz y
(15)
-ET(Px.PY)
where 1'(xx, Py) denotes the set of all joint distributions such that their marginals are Px and Py. When 1, the Wasserstein distance between two one-dimensional distributions can be written as:
(xx, Pr) (S. F (v) (v) (v)
Fx
(16)
= -1 Fy IL.
where Fx and Fy are the cumulative distribution functions (CDFs) of Px and Py, respectively. Morcover, if p the Wasserstein distance further simplifies to
Wi(Px,Py) f(Ex(O) (v)lde = |F - Fyllo
(17)
From the above, it is evident that the ID Wasserstein distance is straightforward to compute, leading to the development of the sliced Wasserstein distance (SWD) ( To leverage this computational advantage in ID, one first projects both distributions uniformly onto a ID direction and computes the Wasserstein-p distance between the two projected distributions, SWD is then calculated by taking the expectation over the random direction. Specifically, let fu E gd denote a projection direction, and the push-forwand distribution (60) bu Px denotes the law of / X. When i is uniformly distributed on the d-dimensional sphere, the p-sliced Wasserstein distance between Px and Py is given by:
dp.
(18)
JESA-1
Combining Eq. 18 and Eq. (16) yields:
SW SWa(Px,Pr) =
Susgen (I.
du
(19)
= -S JUESA-11 [F((()) -F)
Fo(u)ldo du when p 1.
(20)
Despite faster computation, the convergence of SWD has been shown be equivalent the conver- gence of the Wasserstein distance under mild conditions 10). In practice, the expectation over is approximated by a finite summation over a number of projection directions chosen uniformly from gd-1
D Sliced Wasserstein Correlation
The idea of metricizing independence is recently advanced using the Wasserstein distance 61 43 Given a joint distribution (X, Y) ~ Txx and its marginal distributions X ~ Px,Y Y
Py, the
Wasserstein Dependency (WD) between X and Y defined by WD(X,Y) (T xy, Px X ).). A trivial observation is that WD(X, Y) 0 implies that X and Y are independent. Due to the high computational cost of Wasserstein distance, sliced Wasserstein dependency (SWDep) 33 is developed using sliced Wasserstein distance (see Appendix C for SWD details). The SW dependency (SWDepp) between and Y is defined as SW, (T,(Txx- xY, Px X Py), and zero SWDep indicates independence. Since the dependency metric is not bounded from above, sliced Wasserstein correlation
17

---

Stap Propertly ve Deperdency Repulertztiom
MeapORR/Svvpp
DeepORK(Drpod)
Dvep Krock:diSrrpp
1.0
Doep kuccke
KondM00AND--poo spend.
0.8
DOL KOSwop)
0.2
10
Spach
Figure 8: The competing relationship between the swap property (CsL in solid curves) and dependency regularization (CDRL in dashed curves).
(SWC) is introduced to normalize SWDep. More specifically, the SWC between X and Y is defined
SW.ep,(X.Y)
XXY, = /SWDep, (X, X) (Y.))
SWD, & Py)
(21)
SWD,, (Txx, Px & (x))WD (I'ry, Py PP)
where X and Y are the joint distributions of (X, X) and (Y, Y) respectively. It shown that and (XY) = when has a linear relationship with Y [33]
0 & SWC,(X,Y)
In terms of computing SWC, we follow 33 and consider both laws of X and Y to be sums of 2n Diracs, i.e., both variables are empirical distributions with data Tulll yx)) Define I = yx))" and I {(8,, v.))= where (x, ý:) (Xn+i. Vn+z). Because data is i..dd I and I are independent. We further introduce the following notation:
Ixy = {(x;, Ixy (:))?
Ixx ((x,,x;) Ixx x;xx,
Iyy ((r..)) Iay ((5 y:))
Then the empirical SWC can be computed by:
SWD, (Tay' Izm))
(Xy, =
(22)
SWD, (raer177 SWD, (Irop lzgy)
E Competing Losses
In Figure 8
present scaled LsL and LDRL curves for each model considered. For better visualiza- tion, these curves are normalized to values between and We observe the first 20 epochs, as CDRL flattens out in later epochs without further decrease. The competition between losses is evident: as LSL. decreases, LDRL increases, indicating difficulty in maintaining low reconstructabilitt.
F Proofs
F1 Proof of Lemma 3.1
Assuming all conditions in hold, we prove this by applying inequalities on SWD(Pn, PB) and SWD(P,PB),
SWD(P,,. P8) <
+ SWD(PB pB)
=SWD(P, PD) + 1/23)
(23)
18

---

1.0
0.8
FDR
Power
Dalaset
Gaussian Miature
Claylon Exponentaal
Copula: Clayton Gamma
Cupail:: Jos & Expocential
Copulai Jon
Op
Figure 9: The effect of an for X in Eq. (8) on FDR and power. When an O, we consider the knockoff generated from the knockoff transformer without any dependency regularization perturbation. When an there is only perturbation Xp without the knockoff. The sample size 2000.
where the first incquality is direct application of the triangle inequality, and the second equality follows from Theorem 5 in [4]]. Similarly, we obtain
SWD(P, PB) < SWD(P, P.) SWD(P PT) + SWD(P
SWW PM) 0(2
(24)
Combining Eq.
23
and Eq. 24
have
SWD(P, PB)
SWD(P.. PB) OOnnn(n)
(25)
F.2 Proof Proposition 3.2
DDRP a.s. X as 00. Combining this with Eq. (25). we obtain 8,n SWD(P, P") = SWD (, P = SWD( DR.P n PRDRRP) as n + OC.
From Eq. (.. we know
G Effect of on in DRP
As discussed in Section 3.2, obtaining the swap property while maintaining low reconstructability at the sample level is empirically challenging. To address this, dependency regularization perturbation (DRP) is introduced. In this section, evaluate the effect of an in X in Eq. (9) feature selection performance. Results are summarized in Figure for five synthetic datasets with the sample size 2000.
When 0,, decreased, we observe an increase in power. However, the FDR exhibits bowl-shaped pattern, consistent with the findings of [54]: introducing the permuted Xpp reduces reconstructabilityy thereby increasing power. However, an overly dominant Xpp compromises the swap property, resulting in higher FDRs. Based our hyperparameter search and the results in Figure 9 we recommend choosing an within the range of 0.4 to 0.5.
H Implementation Details
H.1
Training
To fit models, we first split datasets of X into training and validation sets with an 8:2 ratio. The training sets are used model optimization, and validation sets are used for early stopping based on the validation loss, with patience period of 6. Since knockoffs are not unique [11] testing sets are not required. To evaluate DeepDRK's performance, we compare with four state-of-the-art (SOTA) deep-lcarning- based knockoff generation models, focusing non non-parametric data. Specifically,
19

---

Parameter
Value
Parameter
Value
Sw Learning Rate
1 x 10-
No Learning Rate
1 10-6
Dropout Rate
0.1
# of Epochs
200
Batch Size
64
A1
30.0
A2 Early Stop Tolerance
1.0
X
20.0
6
On
0.5
Table 3: Training configuration.
We use the recommended hyperparameter settings for each model, with the only difference being the number of training epochs, set 200 for consistency across models.
include Deep Knockoff 47 DDLK [5
KnockoffGAN [27 and sRMMD 38
We follow the model configuration in Table optimize DeepDRK. The architecture of the swappers Sw based on [55]. Both the swappers and X9 are trained using the Adam optimizer [36]. During training, we alternately optimize Xg and the swappers Su, updating weights three times for each update of weights This training scheme is similar to GAN training [21], though without discriminators. We apply early stopping to prevent overfitting. A pseudocode of the optimization
is provided in Appendix [8.2] In experiments, set an 0.5 universally the dependency
regularization coefficient due to its consistent performance
16
A discussion on the effect of os is
provided in Appendix G
Once trained, models generate the knockoff X given X data. The generated X is then combined with X for feature selection. Experiments are conducted on a single NVIDIA V100 16GB GPU.
H.2 Feature Selection
Once X is obtained, feature selection is performed in three steps. We first concatenate X and X to form 2p design matrix (X, X). In the second step,
Ridge regression to get the estimated regression coefficients { from Y and (X. Finally, compute the knockoff statistics wi |33| - 13/+pl. for j 1......, and then select features using the threshold defined as the FDR threshold, following its common use in other knockoff-baat based
Eq. 3 We set y feature selection studies 47 (38). Each experiment is repeated 600 times, with results reported the average power and FDR over these 600 repetitions.
H.3
Data Preparation
H.3.1
Weights for the Gaussian mixture models
Set No.
T3
Table includes 10 sets of ( n2, Ra) for the Gaussian mixture models.
1
0.562
0.384
0.054
2
0.430
0.168
0.402
3
0.317
0.324
0.359
H.3.2 Preparation of the RNA Data
4
0.316
0.388
0.296
5
0.439
0.488
0.073
We first normalize the raw data to value range [0, 1] and then standardize it to have zero mean and unit variance. y synthesized according to X. We consider two different ways of synthesizing Y. The first, denoted as "Linear", is similar to the previous setup in the full synthetic case with Rademacher(O..), For the second, denoted as "Tanh" the response Y is
0.314
0.041
0.645
7
0.656
0.282
0.062
8
0.200
0.300
0.500
9
0.500
0.300
0.200
10
0.333
0.333
0.333
Y ( and B 12.5-/N
Table 4: 10 sets of (T1, #2, R3) for the Gaussian mixture models.
energy and- anpppaaaanddappllicctiono
We present this value for its consistent performance; however, it may not be the globally optimal value.
Future work may improve design.
20

---

Reference Type
Metabolite
Source
Meatbolite
Source
PubChem
palmitate
985
taurocholate
CID: 6675
cholate
CID: 221493
p-tydroxyphenyllaettae
CID: 127
linoleate
CID: 5280450
deoxycholate
CID: 222528
launchenodeoxycchlla
CID: 3673316
12.13-diHOME
dodecanedioute
?
urachidonate
eicosatricnoute
eicosadienonte
docosapentaenotte
taurolithocholate
sulicylate
saccharin
1.2.3.4-tetrahyydee
oleate
urachidate
glycocholate
chenodsoxycholate
phenyllactate
33 3
glycolithocholute
5
Publications
urobilin
58
40
caproate
38 :2)
bydrocinnamate
myristate
18
adrenate
88 35
olmesartan
38 45
tetradecanedioste
bexadecanedioote
oxypurinol
porphobilinggen
caprate
52
53
undocanedionate
32 58
5
oleanate
alycochenodeoyyccllacc
50
sebacate
nervonic acid
lithocholate
hipha-muricholate
tauro-alphu-nee mww.hhhholateeaan-eeeeau muriihhha-aaeamiihaaaatae
Preprints
17-methylstearute taurodeoxycholate
+2
myristoleute
ketodeoxycholate
Table 5: IBD-associated metabolites that are supported by the literature. This table includes all 47 referenced metabolites for the IBD case study. Each metabolite is supported by one of the three PubChem, peer-reviewed publications or preprints. For PubChem case, we report PubChem reference ID (CID), and for the other two cases report the publication references.
sources:
generated following the expression:
k [m/4]
(2) ~N(11)
9k 9k
(3) (4) (5) 1)
fk 8k 9k
(26)
-
k X + 84
(3) Xak-2
+ipk
tanh
(x) XAR-1 + ( XAR
where follows the standard normal distribution and the 20 covariates sampled uniformly.
H.3.3 Metabolite Information for the IBD Study
Here provide the implementation details for the experiments described in Section 4.3 we provide all 47 referenced metabolites based on our comprehensive literature review.
In Table 5
I
Additional Results
In this section, we include all results that deferred from the main paper.
Abbreviation
Full Name
J+G
Copula: Joe & Gamma
C+G
Copula: Clayton & Gamma
I.1 Swap Property
C+R
Copula: Clayton & Exponential
J+E
Copula: Joe & Exponential
We use different metrics to empirically eval- uate the swap property on the generated knockoff X and the original data X accord- ing to Eq. (I). In this paper, three metrics are considered: mean discrepancy distance with linear kernel, "MMD(Lincar)" for
MG
Mixture of Gaussians
Table 6: The abbreviations the names for the datasets.
21

---

Dataset
MMD(Linear)
SWD,
SWD2
J+G
2.68
0.18
0.08
C+G
2.49
0.07
DDLK
C+E
0.15
0.05
J+E
2.29
0.15
0.06
MG
306.49
2.15
J+G
7.01
0.15
0.08
C+G
5.11
0.16
0.04
KnockofIGAN
C+E
0.52
0.06
0,01
J+E
1.24
0.08
0.02
MG
1.08
3.28
26.09
J+G
13.09
0.21
0.11
C+G
19.04
0.27
0.14
Deep Knockoff
C+E
6.65
0.18
0.07
J+E
6.78
0.19
0.13
MG
2770.00
8,98
196.54
J+G
0.47
0.13
0.06
C+G
0.71
0.14
0.05
DeepDRK (Ours)
C+E
0.23
0.09
0.02
J+E
0.14
0.10
0.04
MG
380
6.13
84.37
J+G
130.02
0.68
0.71
C+G
175.74
0.73
0.96
<RMMD
C+E
49.09
0.41
0.35
J+E
33.24
0.35
0.29
MG
3040
8.51
142.99
Table 7: Evaluation of the swap property. This table empirically measures the swap property by three different metries: MMD(Linear), SWD1, and SWD,, The evaluation considers all baseline models and all datasets in the synthetic dataset setup. For space consideration, we use abbreviations indicate the name of the datasets. The full name can be found in Table
short; sliced Wasserstein-1 distance (SWD)); and sliced Wasserstein-2 distance (SWD2). We mea- sure the sample level distances between the vector that concatenates X and X (i.e., (X,X)) and the vector after randomly swapping the entries (i.e., (X, x swa)(B)). To avoid repetition, please refer to section 2.1 and Eq. (I) for the definition of notation. Empirically, it is time consuming to evaluate all subsets B of the index set [p]. As result, we alternatively define swap ratio r. E (0.1, 0.7, 0.9). The swap ratio controls the amount uniformly sampled indices (i.e., the cardinality [B] = r. p) in a subset of [p]. For any and X from the same experiment, different subsets B are formed according to 5 different swap ratios. We report the average values over the swap ratios to represent the empirical quantification of the swap property. Results can be found in 7
Clearly, compared other models, the proposed DeepDRK achieves the smallest values almost every entry across the three metrics and the first 4 datasets J+G, C+G, C+E and J+E in Table 7 This explains why DeepDRK has lower FDRs as DeepDRK maintains swap property relatively better than the baseline models (see results in Figure 2). Similarly, we observe that KnockoffGAN also achieves relatively small values, which leads to well-controlled FDRs compared to other baseline models. Overall, this verifies the argument in Candès et al. (II) that the swap property is important in guaranteeing FDR during feature selection.
However, we observe difference for Gaussian mixture data. The proposed DeepDRK achieves the best performance in FDR control and power (see results Figure 2). yet its swap property measured counter-intuitive
under the metrics in Table is not the lowest. Despite this counter- intuitive observation, we want to highlight that it does not conflict with the argument Candès al. [[T]. Rather, supports our statement that the low reconstructability and the swap property cannot be achieved at the sample level (e.g.. the free lunch dilemma in practice). After all, the swap property the only factor that determines FDR and power during feature selection.
22

---

Sample Size: 200
Sample Size: 2000
Power
S. B
FDR SEECERRCR
Copuix: Clayton Expetenbull Copuia: Clayton
Oopuie: Expanantial
Copula: Clayton Expanetiial
Copuia: Expanentall
Goursan
Capular Joe
Gusseian
Clayteed
Copuie: jue
Figure 10: Power and FDR in the ablation studies. The red horizontal line indicates the 0.1 FDR threshold. DeepDRK' is the model with dependency regularized perturbation removed. K indicates the number of swappers.
DeepORK
Deep
KnocknBiGAN
Dalords Gucatan Zepaserill
Cateo Copell Jos Catrra
Knockoff statistics (wy,) for different knockoff models on the synthetic datasets. Each bar in the plot represents the mean of the null/nonnull knockoff statisties averaging 600 experiments. The error bar indicates the standard deviation. The sample size is 2000.
Figure 11:
1.2 Ablation Studies
In this subsection, we perform ablation studies on different terms introduced in Section 3.1.1 to show the necessity of designing these terms during the optimization for knockoffs. We consider the fully synthetic setup described in Section 4.1 and consider n = 200 and n 2000. The distribution of B 15-VN Rademacher(0.5), We test the following terms: 1. REx: 2. the number of swappers K; 3.
Ssapperr 4. the dependency regularized perturbation (denoted as DeepDRK'), Five synthetic datasets are considered as before and we report the values for FDR and Power under each setup. The results are presented Figure T0
Figure Iijilusttatess the clear drawbacks of using only a single swapper (K = 1) and the case without cases, fail control the FDR on all datasets. The REx term
the loss term REx. In those is essential, as it ensures that the adversarial attacks generated by different swappers are addressed simultaneously. Lswapper is also important term, promotes diverse adversarial environments. Without this term we observe an increase in FDR compared to DeepDRK or DeepDRK1. The difference between DeepDRK and DeepDRK more subtle as DeepDRK has already obtained high quality knockoffs. However, we empirically observe that adding the perturbation further increases the power for some datasets without sacrificing the FDR controllability. interpret these observations, we follow a similar procedure in Section to study the distribution of the knockoff statistics. Results are included Section
Overall, we verify that all terms are necessary components to achieve higher powers and controlled FDRs through the ablation studies.
I.3 Analysis on the distribution of knockoff statistics
We present the additional results on the means and standard deviations of both null and nonnull knockoff statistics under various experimental setups.
Figure shows the results on wy for n = 2000 and complements results on Power and FDR in Figure 2
We notice that all models have concentrated null features around zero and relatively high values for the nonnull knockoff statisties for the case with 2000 samples. This corresponds to consistently accurate FS results across all models and datasets, shown in Figure
23

---

DeepDRK
ARMMD
Deep Stockoff
XrockofrGAN
BOLX
Norrol
Netrel
N4B
Nexval
Noll
Nenwal
N:ll
Nenvvll
Nill
Figure 12: Knockoff statistics (wy) for different models with the increased correlation of the Gaussian mixture data. Each bar in the plot represents the mean of the null/nonnull knockoff statisties averaging on 600 experiments. The error bar indicates the standard deviation. The sample size is 200.
car
DKk
620
ARMMI
326
Deep Knorked
KroebofiCAN
0.20
DOLK
615
915
0.15
0.1
0.3
038
0.00
0.0
C.0
000
0.90
0.35
Nexwvll
Nenvall
Nool
Nenwall
Nxt
Nitvull
Nzll
Nall
Figure 13: Knockoff statistics (wy) for different models with the increased correlation of the mixture of Gaussians data. Each bar in the plot represents the mean the null/nonnull knockoff statistics averaging on 600 experiments. The error bar indicates the standard deviation. The sample size considered 2000.
Figure and Figure 13 show the results for the Gaussian mixture data with increased correlations (Pbase 0.7 and 0.8) for = 200 and n = 2000 respectively. The figures complement the FDR- 200 and 2000 cases, all models experience
power results in Figure It is clear for both an increase FDR and a decrease in power. This phenomenon can be reflected by the positive shifts of the knockoff statistics for the null features in Figure 12 and Figure [3] However, DeepDRK significantly controls the shifts, achieving the best results with the lowest FDRs and comparable power shown in Figure
In addition, we also compare the knockoff statisties for the models considered in the ablation studies in 1.2 The results for both 200 and n = 2000 are in Figure 14 and Figure 155 Although the null knockoff statistics for DeepDRK, DeepDRK' and "No Luwapper models concentrate symmetrically around zero, the heights of the nonnull knockoff statistics for DeepDRK are the highest, resulting higher power. And because some nonnull knockoff statistics have small values for the DeepDRK and "No Lswapper" cases, we also expect them to have higher FDRs (see Figure [O). On the other hand, compared DeepDRK, the models of "No REx" and "K 1" experience clear positive shifts for the null knockoff statistics, leading to higher FDRs during FS (see Figure 10
L.4 Training time
We consider evaluating and comparing model training runtime in Table with the (2000, 100) setup, as it common the existing literature. Although DeepDRK not the fastest among the compared models, the time cost-7.35 minutes-28 is still short, especially when the performance is taken into account.
DeepORK
DeepDRK
Vie
Copta: Carte & Zegrourtal
Creak Pewarrii
Noozzai
Null
Noszall
Nonzal
Null
Nonzall
Nanzul
Nell
Figure 14: Knockoff statistics (w,) for different models in ablation studies. Each bar in the plot represents the mean of the null/nonnull knockoff statistics averaging on 600 experiments. The error bar indicates the standard deviation. K refers to the number of swappers. The sample size is 200.

---

DeepBRK
05
DeepDRK
No Curos
No REX
60
69
Norril
Nall
Nornol
Nil
Norral
Nall
Norzil
No0
Noncal
Nill
Figure 15: Knockoff statistics (wy) for different models in ablation studies. Each bar in the plot represents the mean of the null/nonnull knockoff statistics averaging 600 experiments. K refers to the number of swappers. The sample size considered is 2000,
DeepDRK
Deep Knockoff
sRMMD
KnockoffGAN
DDLK
7.35 min
1.08 min
6.38 min
10.32 min
53.63 min
Table 8: The training time for the models with 2000 and 100. The batch size is 64 and the models are all trained with 100 epochs.
1.5 Additional Results for the IBD Study
Here we provide the supplementary information for the experimental results described in Section 4. In Table we provide the list of identified metabolites by each of the considered models. This table provides additional information for Table 2 in the main paper which only includes metabolite counts due to limited space.
25

---

Metabolite
DeepDRK Deep Knockoff
sRMMD
KnockoffGAN DDLK
12.13-sHHOME
9.10-dIHOME
hydrocinnamate
mandelate
3-hydroxyon S-hydroxyoctaneeee
caprate
indoleacetate
3-hy e-hydroxydeeanotte
dodecanoute
undecanedionate
myristoleate
myristate
dodecanedioate
pentadecanrate
hydroxyyyrisstae
palmitoleate
palmitate
letradecanediootte
DD-heptadeeenutee
2 hydrox heesadecancatee
alpha-lin lentlie
linoleate
stcarate
bexadecamediestee
I0-nonddocnout
17 --tmlbylsscarrte
eieosaeentaenotee
arachidonate
cicosatrienoate
cicosadienoate
cicosenoate
arachidate
phytanate
dncosahexaenoate
docosopentueneate
adrenate
[-docosenonte
cicosanedioate
oleanate
masilinate
lithocholate
chenodeoxycholate
deoxycholate
kelodeaxycholie
Apha-murichollar
cholate
elycolitbocbolaee
gyoccenndeoxyccolat
glycodeox ycholate
elycocholate
tourolithocholaee
murochenodeeoyccollatt
tnurodeoxycholate
taurochollate
salicylate
saccharin
uzelate
sebacate
carhuxyibuprofen
olmesartan
1.2.3.4 tetrahyddo beta carboline 1.3. dicurboxylate
d-hydroxystyrene
acelytyrosine
Alpha-CEHC
carnosol
oxypurinol
paimityylchanolamide
pheny Ilactatee
p-hy hhtydroxyphhaylaetta.e
purphobilinogen
urobilin
acid oxymetazoline
Table A list of literature- iiteratue-suppported metabolites out a total of 80 candidates. "*" indicates the important metabolites marked by the corresponding algorithms.
26

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable "[No] ", it is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" "we were unable find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly describe the scope and the objective of the paper in the abstract and the introduction sections (also in the related work section).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
27

---

Justification:
The limitation is described in the footnote 15 on page 20.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to the method section and the appendix for the complete proof. We provide assumptions the method section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We run the experiments multiple times for reproducibility check.
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide open code access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//tpss.cc/pubicc/ggg... guides/ CodeSubmiss iismmissiii issi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
29

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are included the experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Results shown in the experiment section are repeated for 600 times. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments performed with single NVIDIA V100 GPU. It is described in the experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
30

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///rrr..ccpooooooc/ccccc..a
Answer: [Yes]
Justification: N/A
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper focuses purely on methodology. How to use this method, and serve what purpose is completely dependent people who use it (see the conclusion section).
Guidelines:
The answer NA means that there no societal impact of the work performed.
authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: It is unreleated question to this paper.
31

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The code is open-sourced on GitHub, with necessary licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, faperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code is the only asset. Since it is associated with the method (see the method section), the answer yes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
32

---

Answer:
[NA]
Justification: is unrelated.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: is unrelated.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "4KyTjsLufs", "reviewer": "Reviewer_WTXG", "summary": "The work introduces DeepDRK, a new algorithm for improving FDR in the model-X knockoff framework. In this framework, a knockoff covariate is generated for each existing covariate where then knockoff covariates have to satisfy the swap property. Given a knockoff statistic that satisfies the flip-sign property, one can perform feature selection with guaranteed false discovery rate by training a regression/classification model on the concatenated vector of original and knockoff covariates and then using any arbitrary feature importance score. Therefore, the main challenge in this framework is generating the knockoffs such that they satisfy the strong swap property: swapping any arbitrary subset of covariates with their knockoffs should not change the distribution of the concatenated vector. \n\nGiven that most real-world data does not follow Gaussian or a mixture of Gaussian distribution, generating knockoffs that satisfy the swap property is a challenge. Existing work is focused on using deep generative models (e.g. GANs) to handle this challenge. The paper argues that this approach for continuous covariates will result in reconstructability: overfitting on the data will make the distribution of knockoffs too similar to the data distribution and therefore making the concatenated vector too multicollinear for any feature-importance method to have discovery power.\n\nThe DeepDRK framework seeks to tackle reconstructability by training the deep model in such a way that not only satisfies the swap property but also reduces reconstructability. This is achieved in two steps:\nFirstly, by having a 1- the adversarial swap loss where a group of models compete adversarially against the knock-off generator model 2- a dependence regularization loss to reduce reconstructability by directly reducing the sliced-Wasserstein correlation between the original and knockoff vectors.\nSecondly, by the post-training dependency regularization which interpolates between the generated knockoff data table and the row permuted original data table. By making the interpolation weaker or stronger, one gains control in how much they can improve how much swap condition holds and therefore how good the FDR is at the cost of loosing power and vice versa.\nThe paper then shows experimental evidence in synthetic and semi-synthetic scenarios (where the true covariate-response) relationship is known. The argument is that compared to existing algorithms that have better power in low-sample settings, DeepDRK's empirical FDR is close the the selected level more consistently and this is proven both by the FDR vs Power results and the original vs knockoff feature importance distributions.", "questions": "Discussed above", "limitations": "There needs to be a discussion regarding the choice of hyperparameters especially the alpha parameter as in real-world applications of knockoffs it's impossible to know the best performing value in advance. Given the results in Figure 8, there seems to be high sensitivity to the alpha value which would make the framework unreliable. At the very least, the authors need to show the superiority of their framework to existing methods in a large interval of alpha values.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "All in all, the main novelties in the paper are:\n- main one is the post-training Dependency Regularization Perturbation\n- using multi swappers and SWD criterion\n- Experimenting on synthetic distributions other than GMMs\n- Showing that unlike other deep learning based knockoff algorithms, DeepDRK has more consistent FDR preservation. The ablation study really helps with understanding the benefit of each of the algorithmic choices.", "weaknesses": "- It seems necessary to see 4.1 results for various beta scales used for generating the response variable to compare different methods in various levels of difficulties to make sure the value of 15 has not been selected to favor DeepDRK. I think it'd be interesting to have the results in a 2D FDR vs Power scatter plot for better comparison.\n- I'd be interested in seeing mean + confidence interval performance plots for Figure 2 and Figure 3 using various distribution parameter selections (e.g. for GMM distribution using various \\pi parameters in addition to to more \\rho_base, etc)\n- The process for choosing the value of alpha=0.5 needs to be clarified as \"it leads to consistent performance\" might suggest choice based on final results which would be a case of overfitting. The discussion related to Figure 8 seems to confirm overfitting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and advice. Please checkout our responses below.\n\nComment: It seems necessary to see 4.1 results for various beta scales used for generating the response variable to compare different methods in various levels of difficulties to make sure the value of 15 has not been selected to favor DeepDRK. I think it'd be interesting to have the results in a 2D FDR vs Power scatter plot for better comparison.\n- Response: Thank you for the suggestions. We have provided experimental results concerning 4 different sets of the scale value: 5, 10, 15, 20. Results are presented in Figure 1 of the attached PDF file in the Author Rebuttal section. From the figure, we can clearly observe that the proposed DeepDRK outperforms other methods for a controlled FDR with relatively higher power (e.g. points are closer to the top left region). In addition, this figure also conveys a message that the commonly used $\\frac{p}{\\sqrt{n}}$ (with n = 2000) setup in existing works is a relatively easy task where most methods can achieve controlled FDR with high power.\n\nComment: I'd be interested in seeing mean + confidence interval performance plots for Figure 2 and Figure 3 using various distribution parameter selections (e.g. for GMM distribution using various $\\pi$ parameters in addition to to more $\\rho_\\text{base}$, etc)\n- Response: Thank you for the comments. We have provided GMM distribution with 10 sets of different $\\pi$ parameters. Results are presented in Figure 2 and Table 1 of the attached PDF. This should complement Figure 2 and 3 of the main manuscript on the change of $\\rho_base$ for the GMM distributions. From Figure 2, we visually observe that the model performance does not vary significantly for all tested models on 3 out of 10 $\\pi$ setups. Among all the models, the proposed DeepDRK is the only one that can control the FDR given the nominal 0.1 level. In addition, we also provide a table that includes the mean, standard deviation and median, 5\\% and 95\\% quantiles for the 10 complete setups. This should reveal the robustness of the proposed model in this setup. \n\nComment: The process for choosing the value of alpha=0.5 needs to be clarified as \"it leads to consistent performance\" might suggest choice based on final results which would be a case of overfitting. The discussion related to Figure 8 seems to confirm overfitting.\n- Response: First of all, please allow us to make corrections on Eq. (8). We realized that Eq. (8) has a missing coefficient $1- \\alpha\\_n$ in front of  $\\tilde{X}\\_{\\theta}$ . Essentially we perform interpolation between $\\tilde{X}\\_\\theta$ and $X\\_\\text{rp}$ as the former seeks to control FDR for a satisfied swap property while the latter seeks to reduce collinearity for higher power. We look for a balance between the two. The corrected equation is $\\tilde{X}^{\\text{DRP}}\\_{\\theta, n} = (1 - \\alpha\\_n) \\cdot \\tilde{X}\\_\\theta + \\alpha\\_n \\cdot X\\_{\\text{rp}}$ and this will be reflected in the revised manuscript. In addition, Figure 8 was plotted according to an earlier version of Eq. (8), where we take $\\tilde{X}^{\\text{DRP}}\\_{\\theta, n} =  \\alpha\\_n \\cdot \\tilde{X}\\_\\theta + (1-\\alpha\\_n) \\cdot X\\_{\\text{rp}}$.  In other words, the $\\alpha\\_n$ presented in Figure 8 refers to the coefficient for $\\tilde{X}_\\theta$, whereas $\\alpha_n$ should refer to the coefficient for $X\\_\\text{rp}$ in the updated Eq. (8). We have corrected Figure 8 and present the new figure as Figure 3 in the attached PDF.\n\n    As for the consistency of the model on the choice of alpha, we want to first point out that the choice of knockoff is not unique [1][2] and there is no closed form solution to the knockoff variable in non-Gaussian setups. As a result, it is hard to find a value for $\\alpha_n$ from the data via procedures like cross-validation. We want to point out that the choice of $\\alpha_n$ is not a result of overfitting. According to Figure 8 (i.e. Figure 3 in the updated PDF in the Author Rebuttal section), we clearly observed that choices of $\\alpha_n$ around 0.5 results in low FDR with relatively high power across multiple datasets, suggesting its generalizability for a range of different data setups. \n\n- [1] Candes, E., Fan, Y., Janson, L., & Lv, J. (2018). Panning for gold:‘model-X’knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3), 551-577.\n- [2] Spector, A., & Janson, L. (2022). Powerful knockoffs via minimizing reconstructability. The Annals of Statistics, 50(1), 252-276.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wcc1YxIzIV", "reviewer": "Reviewer_Znex", "summary": "This paper considers the construction of knockoff features \nin the variable selection framework of model-X knockoffs.\nThe authors propose a deep learning-based method for \ngenerating knockoff features. Extensive numerical simulation and real-data examples show \nthat the proposed method has advantageous performance compared\nwith other state-of-art methods, especially when the covariate distribution \nis complex and when the sample size is relatively small.", "questions": "1. The introduction of the dependency regularized perturbation is \nfor power boosting, and yet in the ablation study, this feature seems to help \nmore with FDR control as opposed to power-boosting. Is there a reason?\n2. On page 4, line 132, the definition of REx: I am confused since \nthe LHS does not depend on $i$ but the RHS depends on $i$. \n3. On page 4, line 148: \"correltaion\" -> \"correlation\"\n4. On page 5, line 153: I am confused by the statement that \"the knockoff \n $X_j$ should be independently sampled from $p_j(\\cdot \\mid X_{-j})$\"\n--- this alone is not enough to ensure the knockoff property.\n5.  On page 5, line 173, what is $\\alpha$?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper considers one fundamental problem in the model-X knockoff \npipeline --- the construction of valid and high-quality knockoffs. \nThis is key to the applicability of the knockoffs method.\n\n\nSubstantial engineering efforts have been made in this work and \nthe proposed method is demonstrated to have satisfactory performance \nin extensive numerical experiments. This is a significant contribution to the \nknockoffs literature.", "weaknesses": "The proposed method is based mostly on empirics and heuristics, \nwith a theoretical understanding lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and advice. Please checkout our responses below.\n\nQuestion: The introduction of the dependency regularized perturbation is for power boosting, and yet in the ablation study, this feature seems to help more with FDR control as opposed to power-boosting. Is there a reason?\n- Response: Thank you for your question. In principle DRP should be considered as an approach to reduce collinearity in the concatenated design matrix at the cost of increasing the swap loss. It is true that the design of the DRP technique does not aim for FDR reduction. Nonetheless, looking at the knockoff statistics $W\\_j$ in Figure 4 and Figure 13, we clearly observe that DRP tends to make the null statistics symmetric around zero. Since the knockoff statistics is the variable that is used for feature selection, the symmetric observation will reduce the number of negative $W\\_j$’s to be selected, according to the selection criteria Eq. (3). This leads to a lower FDR. \n\n\nQuestion: On page 4, line 132, the definition of REx: I am confused since the LHS does not depend on i but the RHS depends on i\n- Response: Thank you for pointing this out. We will revise the equation to $\\text{REx}(X, \\tilde{X}\\_{\\theta}, \\\\{S\\_{\\omega\\_i}\\\\}_{i=1}^K)  = \\widehat{\\text{Var}}\\_{S\\_{\\omega}}(\\text{SWD}([X, \\tilde{X}\\_\\theta], [X, \\tilde{X}\\_\\theta]\\_{S\\_{\\omega_i}}); i \\in [K])$ to make the LHS and RHS consistent.\n\nQuestion: On page 4, line 148: \"correltaion\" -> \"correlation\"\n- Response: Thank you for pointing this out, we will make corrections in the revision.\n\nQuestion: On page 5, line 153: I am confused by the statement that \"the knockoff $\\tilde{X}\\_j$ should be independently sampled from $p(\\cdot \\vert X\\_{-j})$ --- this alone is not enough to ensure the knockoff property.\n- Response: Thanks for pointing out. Indeed the description is a bit hand-waving. To be precise, we want to sample $\\tilde{X}\\_j$ from $p\\_j(\\cdot | X\\_{-j})$ such that $X\\_j$ and $\\tilde{X}\\_j$ are as less dependent as possible. This ensures the swap property but reduces reconstructability. We will revise the description in the manuscript.\n\nQuestion: On page 5, line 173, what is $\\alpha$\n- Response: Thank you for pointing this out. This is a typo and it should be $\\alpha\\_n$. We will correct it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AVdgVSO2Pt", "reviewer": "Reviewer_FDZg", "summary": "This paper proposes a deep-learning-based method for performing \"knockoff\"-based feature selection. The main contribution is the proposal of a new optimization problem, which incorporates a four part loss.  The method is evaluated on synthetic, semi-synthetic, and real-world data, showing comparable and, at times, improved performance over relevant baselines.", "questions": "See above.", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "**Experiments.** At times, this paper shows that the proposed method improves the power for a fixed FDR.", "weaknesses": "**Motivation.** It's not clear why this problem is of interest. It is true that many papers have been published on this topic. But the authors motivate the problem not by describing why knockoffs are particularly important; they simply say that finding information features is \"mission impossible,\" which doesn't give the reader any idea of why the problem is relevant.\n\n**Organization.** To articulate a list of contributions, a paper must clearly answer two fundamental questions: (1) \"What is the problem under consideration?\" and (2) \"How does the proposed method address this problem?\" Unfortunately, it's not clear that this paper answers the first or second questions.  Here are some comments:\n1. The authors claim that their method reduces reconstructability.  However, it's not clear what \"reconstructability\" is, or how it would be formally defined. The idea of reconstructability only arises in the discussion of related work, and even here, the problem is not presented in a way that is distinct from the solution presented in [4].  \n2. Section 2.1 is confusing. More care should be taken w/r/t the introduction of the problem of generating a knockoff sample.  The authors adopt quite a bit of notation from [11].  However, whereas [11] presents the method with examples and clearly articulated goals, this paper's summary is so coarse/brief that any reader who has not read this past work will have little to no chance of understanding the problem of finding knockoffs. \n3. Deferring the definition of FDR to the appendix doesn't make sense to me. This quantity is of fundamental interest; it is measured throughout the experiments, and theorems are presented (in part from past work) which rely on this definition. The paper is incomplete without including this definition in the main text.\n4. Theorem 2.1 is not clearly stated. The \"aforementioned property\" is not inferable from context.  The authors should make it clear that (1) they are referring to Theorem 1 in [11] and (2) that the \"aforementioned property\" is the choice of threshold in (3).\n5. Readability would be improved if the authors included an example of the swap function, such as the example in [11] after Def. 2.\n6. The authors seem to use (X,Z) and [X,Z] freely and interchangeably. This should be clarified. [11] sticks with (X,Z) (rather than square brackets) and I would recommend that the authors do the same.\n7. Make it abundantly clear that whenever the authors say \"power,\" they mean low type II error. This may not be clear to readers coming from deep learning.\n\n**Main method.** A few comments on the main method.\n1. The results are presented in a confusing order. (4) is presented before the authors know what any of the pieces are. Readers must wait around a page to get the details, which will frustrate. Furthermore, it's not clear why (4) is a fundamentally \"good\" objective, or where it came from. As written, it feels like the authors pulled this problem out of a hat.  Why does posing this as a minimax problem (as opposed to, e.g., a constrained problem) make sense? Why do the authors choose these two loss terms (and not others)? And how should one choose all of the trade-off parameters?\n2. The authors seem to imply that [56] has a problem. It's not clear what this problem is; the \"is far from enough\" sentence doesn't give any quantitative evidence as far as I can tell. The result of this is that when the authors say \"To fix it, ...,\" it's unclear what problem they are fixing.\n3. The authors borrow a loss term from the risk extrapolation (REx) paper from domain generalization. The authors should justify this choice, as well as the other loss functions that they append onto their objective. As written, it feels as if the authors are somewhat arbitrarily adding on loss functions, and there isn't a clear principle driving the decision making. This gives the paper a largely heuristic feel. And while the authors claim to provide ablations in the appendix, it's not clear where they are. The authors say that the results are \"in the associated tables,\" but it's not clear what tables these are. A similar concern arises w/r/t Figure 9.\n4. The trade-off between L_SL and L_DRL isn't clearly presented. This content should be presented in the main text; based on the figure in Appendix G, it's not clear that the trade-off always holds. Furthermore, even if it did hold, the contribution that the authors are \"the first to observe this phenomenon\" seems to be trivially true: If no one has used this loss before, then of course no one has seen this trade-off.\n\n**Experiments.** Some comments on the experiments.  The experiments seem to show marginal improvements for the proposed method. While DeepDRK does at times achieve higher power for a fixed FDR, the results are often fairly marginal, e.g., in Figure 5.  Given the shortcomings in the presentation of the method, it's not clear that these results do enough to justify acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. Please refer to the following for our response.\n\nComment: it's not clear why (4) is fundamentally \"good\" … \n- Response: Thank you for the comments and questions. We will add more descriptions on the motivation for the two terms in the revised manuscript. Essentially, the two general terms are introduced to enforce swap property and to reduce collinearity, respectively. This form of losses are seen in most existing deep-learning-based knockoff methods [1][2][3][4]. Nonetheless, our proposed version, as empirically verified in all the experiments presented in the paper, suggests a good performance during the FDR-controlled feature selection. Adopting the min-max formulation is natural, as we want to minimize the total loss over the swapper that maximizes the distance between $(X, \\tilde{X})$ and $(X, \\tilde{X})_{\\text{swap}(B)}$.\n   For the hyper parameters, we chose based on the performance across different dataset experiments and identified that the configuration outlined in Table 2 in general provides good controlled FDR and high power. All results (with different datasets) presented in the paper relies on this single setup, suggesting the adaptability of this method across different application scenarios.\n- [1] Romano, Y., et al. . Deep knockoffs. ` Journal of the American Statistical Association, 115(532):1861–1872, 2020.\n- [2] Jordon, J., et al. : Generating knockoffs for feature selection using generative adversarial networks. In International Conference on Learning Representations, 2018.\n- [3] Sudarshan, M., et al.. Deep direct likelihood knockoffs. Advances in neural information processing systems, 33:5036–5046, 2020.\n- [4] Masud, S. B., et al.. Multivariate rank via entropic optimal transport: sample efficiency and generative modeling. arXiv preprint arXiv:2111.00043, 2021.\n\nComment: The authors seem to imply that [56] has a problem. It's not clear…\n- Response: Thanks for pointing out the ambiguity. [56] uses single-swapper during training, which results in insufficient FDR control and selection power (see Figure 2). The necessity of introducing the multi-swapper setup can be justified by the nonlinearity of learning the knockoff copies from X in the deep learning setup. Given that this optimization is nonlinear, there could be multiple local optima that the swap property could be violated. Introducing this multi-swapper scheme would enforce the model on learning a good knockoff that improves the feature selection performance. We have included the ablation study on the effectiveness of this multi-swapper setup. Please refer to Figure 9 for comparing between the DeepDRK model and the K=1 model. We can see that a single-swapper setup results in failure of FDR control. \n\nComment: The authors borrow a loss term from the risk extrapolation (REx) …\n- Response: The first SWD term in Eq. (5) is introduced to directly enforce the swap property outlined in Eq. (1). There could be multiple choices such as KL divergence, Wasserstein Distance, etc. We choose SWD over other measurements due to the runtime benefit brought by the SWD (see Table 1 for the runtime comparison to other methods). The rationale behind the introduction of REx resides in the lack of FDR controllability in the single-swapper setup. Empirically in the case with only one swapper, we observe that the null knockoff statistics are not symmetric around zero, therefore it is hard to control the FDR and (see Figure 9 and Figure 13 in Appendix K2 and K3 for empirical results). This indicates that the swap property is not well-enforced. Thus we propose to use a multi-swapper setup. Each swapper represents one adversarial environment, so we are optimizing under multiple adversarial attacks. In this case, the REx term is naturally introduced to fight against multiple adversarial attacks simultaneously. Results can also be found in Figure 9 and Figure 13. The last $L\\_\\text{swapper}$ term is to ensure different adversarial environments. We have provided explanations of these terms in the paragraphs below Eq. (5) on page 4. We have improved the writing for better clarity in the revised paper.\n\nComment: The trade-off between L_SL and L_DRL isn't clearly presented. This content…\n- Response: We empirically observe a competition between the two losses from the fact that when one loss decreases, the other increases, and it is hard to find a good set of hyper-parameters $[\\lambda_1 , \\lambda_2]$ to strike a balance. It is not a training problem because, as stated in Section 3.2, we observe such a phenomenon in all baseline knockoff generation algorithms of this paper. The baseline methods of course don’t define the losses in the exact same way as we did, but they all have corresponding loss terms that serve the same purposes. Thus the observation is well worth pointing out and the underlying reasoning is also stated in our paper: minimizing the swap loss, which corresponds to FDR control, is the same as controlling Type I error. Similarly, minimizing the dependency regularization loss is to control Type II error. With a fixed number of observations, it is well known that Type I error and Type II error can not decrease at the same time after reaching a certain threshold. This is why the two losses always compete with each other.\n\nComment: The experiments seem to show marginal…\n- Response: Only looking at the right column in Figure 5, the improvement of our method is not significant, because the linear synthetic rule is an easy task, on which all methods perform quite well. However when a tanh synthetic rule is applied, i.e. the left column of Figure 5, it is apparent that DeepDRK is the only method that controls FDR under the target level and at the same time enjoys high selection power. Other results such as Figure 2 and 3 also suggest that for complex distributions and low sample size, DeepDRK performs significantly better than baseline algorithms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We have completed more experiments, as suggested in your comments. Please refer to the PDF file for details.

Specifically, Figure 1 refers to additional experiments on the change of $\beta$ coefficient scales. We include three scales, 5, 10, and 20, in addition to the default 15. Namely, we consider $\frac{p}{5\cdot \sqrt{n}}$, $\frac{p}{10\cdot \sqrt{n}}$, $\frac{p}{15\cdot \sqrt{n}}$ and $\frac{p}{20\cdot \sqrt{n}}$ in total.

Figure 2 provides barplots for three different setups for the weights (i.e. $\pi_1$, $\pi_2$ and $\pi_3$) of the components in mixture Gaussian . They are: 1. [$\frac{1}{3}$, $\frac{1}{3}$, $\frac{1}{3}$]; 2. [0.2, 0.3, 0.5]; 3. [0.5, 0.3, 0.2]. The barplots compare the model performance on these three setups on FDR and Power. In addition, we further consider 7 other setups and calculate statistics on the FDR and power (e.g. mean, standard deviation, median and quantiles). The weights are uniformly sampled. The results can be found in Table1. Weights detail is presented below:
- [0.56215729, 0.38386235, 0.05398036]
- [0.42981112, 0.16832496, 0.40186392]
- [0.31650882, 0.32417065, 0.35932052]
- [0.31627876, 0.38820664, 0.2955146]
- [0.43860741, 0.48803919, 0.0733534]
- [0.31395573, 0.04100264, 0.64504163]
- [0.65634908, 0.28201858, 0.06163234]

We also provide the corrected Figure 8 (of the manuscript) in Figure 3 in the PDF file. Essentially, it flipped the X-axis horizontally and nothing else was changed.

Sincerely,

The Authors of the Paper

Meta-review: The paper introduces DeepDRK, a new algorithm for improving FDR (by to constructing knockoff features) in the model-X knockoff framework. The authors propose a deep learning-based method for generating knockoff features in which a knockoff covariate is generated for each existing covariate (and knockoff covariates have to satisfy the so-called swap property). The authors provide extensive numerical simulation and real-data examples to show that the proposed method has advantageous performance compared with other state-of-the-art methods, especially when the covariate distribution is complex and when the sample size is small.

All the reviewers agree that the paper considers an important question in the knockoff framework and gives an empirically satisfactory solution. Reviewers had some concerns about the presentation of the paper (e.g. additional background, motivation of the problem, etc) which have been resolved after the discussion period, and hence I would like to recommend the paper for acceptance. Also, I would like to suggest to the authors to revise the manuscript to improve the presentation according to the comments provided by the reviewers.
Decision: Accept (poster)