Normalizing Flows for Knockoff-free Controlled Feature Selection
Authors: Derek Hansen
Brian Manzo
Jeffrey Regier
Venue: NeurIPS 2022 Accept (2022)
Keywords: controlled feature selection
variable selection
knockoffs
normalizing flows
false discovery rate control
Bayesian methods

Abstract: Controlled feature selection aims to discover the features a response depends on while limiting the false discovery rate (FDR) to a predefined level. Recently, multiple deep-learning-based methods have been proposed to perform controlled feature selection through the Model-X knockoff framework. We demonstrate, however, that these methods often fail to control the FDR for two reasons. First, these methods often learn inaccurate models of features. Second, the "swap" property, which is required for knockoffs to be valid, is often not well enforced. We propose a new procedure called FlowSelect to perform controlled feature selection that does not suffer from either of these two problems. To more accurately model the features, FlowSelect uses normalizing flows, the state-of-the-art method for density estimation. Instead of enforcing the "swap" property, FlowSelect uses a novel MCMC-based procedure to calculate p-values for each feature directly. Asymptotically, FlowSelect computes valid p-values. Empirically, FlowSelect consistently controls the FDR on both synthetic and semi-synthetic benchmarks, whereas competing knockoff-based approaches do not. FlowSelect also demonstrates greater power on these benchmarks. Additionally, FlowSelect correctly infers the genetic variants associated with specific soybean traits from GWAS data.


Normalizing Flows for Knockoff-free Controlled
Feature Selection
Derek Hansen Department of Statistics University of Michigan dereklh@umich edu
Brian Manzo Department of Statistics University of Michigan bmanzo &umi ch. edu
Jeffrey Regier
Department of Statistics University of Michigan regierguml ch. edu
Abstract
Controlled feature selection aims to discover the features a response depends on while limiting the false discovery rate (FDR) to predefined level. Recently, multi- ple deep-learning-basee
methods have been proposed perform controlled feature selection through the Model-X knockoff framework. We demonstrate, however, that these methods often fail to control the FDR for two reasons. First, these meth- ods often learn inaccurate models of features. Second, the "swap" property, which is required for knockoffs to be valid, is often not well enforced. We propose a new procedure called FL.WW SELECT perform controlled feature selection that does not suffer from either these two problems. To more accurately model the fea- tures, FLOWSELECT uses normalizing flows, the state-of-the-an art method for density estimation. Instead of enforcing the "swap" property, FLOW SELECT uses a novel MCMC-based procedure to calculate p-values for each feature directly. Asymp- totically, FLOWSELER computes valid p-values. Empirically, FLOWSELECT consistently controls the FDR on both synthetic and semi-synthetic benchmarks, whereas competing knockoff- choseoff-based approaches do not. FLOWSELECT also demon- strates greater power on these benchmarks. Additionally, FLOWSELECT correctly infers the genetic variants associated with specific soybean traits from GWAS data.
1 Introduction
Researchers machine learning have made much progress in developing regression and classification models that predict response based on features. In many application areas, however, practitioners need to know which features drive variation in the response, and they need to do so way that limits the number of false discoveries. For example, in genome-we wiide association studies (GWAS), scientists must consider hundreds of thousands of genetic markers to identify variants associated with a particular trait disease. The cost of false discoveries (i.e., selecting variants that are not associated with the disease) is high, as costly follow-up experiment is often conducted for each selected variant. Another example where controlled feature selection matters is analyzing observational data about the effectiveness of educational interventions. In this case, researchers may want to select certain educational programs implement on a larger scale and require confidence that their selection does not include unacceptably many ineffective programs. As a result, researchers are interested in methods that model the dependence structure of the data while providing an upper bound on the false discovery rate (FDR).
Model-X knockoffs (Candès et al., 2018) is popular method for controlled variable selection, offering theoretical guarantees of FDR control and the flexibility to use arbitrary predictive models. However, even with knowledge of the underlying feature distribution, the Model-X knockoff's method is not feasible unless the feature distribution is either finite mixture of Gaussians (Gimenez et al., 2019) or has a known Markov structure (Bates al., 2020). Hence, a body of research explores the use of empirical approaches that use deep generative models to estimate the distribution of X and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

sample knockoff features (Jordon et al., 2019; Liu & Zheng, 2018; Romano et al., 2020; Sudarshan et al., 2020).
The ability of these methods to control the FDR is contingent on their ability to correctly model the distribution of the features. By itself, learning sufficiently expressive feature model can be challenging. However, the knockoff procedure requires learning knockoff distribution that satisfies the swap property, which is a much stronger requirement. Formally, let X RR be sample from the feature distribution and X E RD be sample from the knockoff distribution conditioned on X. The swap property stipulates that the joint distribution (X, X) TR22 must be invariant to swapping the positions of any subset of features S E (1....,D}:
(x,
(S) (X.X)sswa(S))
2 (X
(1)
Here, Swap(S) means exchanging the positions of Xj and X; for all j E S. For example, in the case D = 3 and S = (1,3), the joint distribution is (X, X) = (X1, X2, X3, X1, X2, Xs). and the swapped joint distribution is (X, X) )swap(s) = (X1, X3, X1, Xa). Note that, for
S= (1....... D), the swap property implies that X 2 X. See Candès et al. (2018) for more detailed description of the swap property.
Even if distribution were found satisfying the swap property. it may not provide enough power to make discoverics. For example, both properties are trivially satisfied by constructing exact copies of the features knockoff's, but the resulting procedure has no power.
In situations where valid knockoff distribution is available to sample from, knockoff's are computa- tionally appealing because they require only one sample from knockoff distribution to assess the relevance of all p features. However, in situations where the joint density of the features is unknown, we show that empirical approaches to knockoff generation (Jordon et al., 2019; Liu & Zheng, 2018; Romano et al., 2020; Sudarshan et al., 2020) fail to characterize a valid knockoff distribution and therefore do not control the FDR. We further show that even with a known covariate model, it is not straightforward to construct a valid knockoff distribution unless a specific model structure known.
We propose a new feature selection method called FLOWSELECT (Section 3), which does not suffer from these problems. FLOWSELECT uses normalizing flows learn the joint density of the covariates. Normalizing flows is a state-of-the-ar method for density estimation; asymptotically, it can approximate any distribution arbitrarily well (Papamakarios et al., 2021; Kobyzev al., 2020: Huang et al., 2018). Additionally, FLOWSELECT circumvents the need to sample a knockoff distribution by instead applying a fast variant of the conditional randomization test (CRT) introduced in Candès et al. (2018). Samples from the complete conditionals are drawn using MCMC, ensuring they are unbiased with respect to the learned data distribution.
Asymptotically, FELESE computes correct p-values to use for feature selection (Section 4). Our proof assumes the universal approximation property of normalizing flows and the convergence of MCMC samples to the Markov chain's 's stationary distribution. Under the same assumptions as the CRT, which includes a multiple-testing correction in Benjamini & Hochberg (1995), a selection threshold can be picked which controls the FDR at a pre-defined level. Empirically, on both synthetic (Gaussian) data and semi-synthetic data (real predictors and a synthetic response). FLOWSELECT controls the FDR where other deep-learning based knockoff methods do not. In cases in which competing methods do control the FDR, FLOWSELECT shows higher power (Section 5). Finally, in challenging real-world problem with soybean genome-wide association study (GWAS) data, FLOWWSE..EC successfully harnesses normalizing flows for modeling discrete and sequential GWAS data, and for selecting genetic variants the traits depend on (Section 5.4).
2 Background
FLOESE brings together four existing lines of research, which we briefly introduce below.
Normalizing flows Normalizing flows is a general framework for density estimation of a multi- dimensional distribution with arbitrary dependencies (Papamakarios et al., 2021). A normalizing flow starts with a simple probability distribution (e.g., Gaussian or uniform), which called the base distribution and denoted Z. and transforms samples from this base distribution through

---

series of invertible and differentiable transformations. denoted G, to define the joint distribution of X E IR"" Px. A normalizing flow with enough transformations can approximate any multivariate density, subject to regularity conditions detailed by Kobyzev et al. (2020). Compared to other density- estimation methods, normalizing flows are computationally efficient. Details about the specific normalizing flow architecture used in FLOWSELECT are provided in Appendix A.
Controlled feature selection Consider a response Y which depends on vector of features X RD. Depending on how the features are chosen, it is plausible that only subset of the features contains all relevant information about Y. Specifically, conditioned on the relevant features in X. Y is independent of the remaining features in X (i.c. the null features). The goal of the controlled feature selection procedure is to maximize the number of relevant features selected while limiting the number of null features selected to predefined level. If we denote the total number of selected features R. then we can decompose R into V, the number of relevant features selected, and S, the number of null features selected.
Conditional randomization test Controlled feature selection can be seen as a multiple hypothesis lesting problem where there are p null hypotheses, each of which says that feature X, is conditionally independent of the response given all the other features X-j. Explicitly, the test of the following hypothesis is conducted for each feature = fl...., D}:
Ho X3 1 X|X->
versus HI : X3 X
To test these hypotheses, one can use a conditional randomization test (CRT) (Candès et al., 2018). For each feature tested in conditional randomization test, test statistic T, (e.g., the LASSO coefficient or another measure of feature importance) is first computed on the data. Then, the null distribution of T3 is estimated by computing its value T, based on samples X, drawn from the conditional distribution of Xj given X-j- Finally, the p-value is calculated based the empirical CDF of the null test statistics, and features whose p-values fall below the threshold set by the Denjamini-Hochheee procedure (Benjamini & Hochberg, 1995) are selected. Though the CRT is introduced as a computationally inefficient alternative to knockoff's the CRT nonetheless has appeal because requires only knowledge of the feature distribution, which can be learned empirically by maximum likelihood.
Holdout randomization test The holdout randomization test (HRT) (Tansey al., 2021) is fast variant of the CRT; it uses a test statistic that requires fitting the model only once. Let 0 represent the parameters of the chosen model, and let T(X, Y, 0) be an importance statistic calculated from the model with input data. For example, T' could be the predictive likelihood P X or the predictive score R2. To use the HRT, first fit model parameters & based on the training data. Next, for each covariate j. calculate the test statistic T +- T(X yles, 0). Then, generate & null samples and compute T,, <- T(Xtest (j+jza)' Y 0). where X (j+ replaces the j-th covariate with the k-th generated null sample. Finally, calculate the p-value as in the CRT, based on the empirical CDF of the null test statistics.
3 Methodology
FLOWSELECT implements the CRT for arbitrary feature distributions by using a normalizing flow to fit the feature distribution and Markov chain Monte Carlo (MCMC) to sample from each complete conditional distribution. Performing controlled feature selection with FLOWSELECT consists of the three steps below.
Step 1: Model the predictors with a normalizing flow
Starting with the observed samples of the features XI,..., XN ~ Px, we fit the parameters of normalizing flow Go to maximize the log likelihood of the data with respect a base distribution pz:
3

---

i = arg max
log pe (Xi))
8
1=1
(G)(XX)
where pe po(X,) pz(Gg(X)) det
0x
The resulting density Pa is a fitted approximation to the true density Px. The specific normalizing flow architecture use in our first two experiments consists of single Gaussianization layer (Meng et al., 2020) followed by a masked autoregressive flow (MAF) (Papamakarios et al., 2017). The first layer can learn complex marginal distributions for each covariate, while the MAF learns the dependencies between them. More detail on normalizing flows and on this particular architecture can be found in Appendix A.
Step 2: Sample from the complete conditionals with MCMC
For each feature j. we aim to sample corresponding null features Xii. for all k {1,..., K} that are equal in distribution to (x,j|X,,->) but independent of Y, However, directly sampling from this conditional distribution is intractable. Instead, we implement as a stationary distribution. The samples drawn from MCMC are autocorrelated, but any statistic calculated over these samples will converge almost surely to the correct value. The choice of the MCMC proposal distribution qj is ficxible. Because each Markov chain is only one-dimensional, a Hasssig Gaussian random walk with the standard deviation set based the covariance can be expected to mix rapidly. Alternatively, information from Pý- such as higher-order derivatives, could be used to construct more efficient proposal. Algorithm details how to implement step 2.
MCMC algorithm that admits
Step 3: Test for significance with the HRT
As in the CRT, feature j has high evidence of being significant if, under the assumption that j is a null feature, the probability of realizing a test statistic greater than the observed Tj (X) is low. Formally, letting [X X-Jl be the observed feature matrix with the observed feature X, swapped out with the null feature Xj. we can write this as p-value a;:
aj PX,IX (I)(X) (I)())<T xI,X,,,,))))
However, the above p-value aj is not tractable. For each sample X. j.k drawn using MCMC, we calculate the corresponding feature statistic and compare it to the real feature statistic, leading
Algorithm Step 2 of the FLOWSELECT procedure for drawing K null features Xxy|X,,-s for feature j observation i.
Input: Feature matrix X E ENN observation index i, feature index j. number of samples K, fitted normalizing flow Pq. MCMC proposal Output: Null features Xij, for k: 1,...,K K
for 1.....K K. do
Propose: Xi.. ~ -4 (-1-11111-11111----
Paj.A + Po(Xi N. (. y xx. Li (1.
Sample: U. Bernoulli(ry, (rano
if U.... = then
Xij.k Xi,jk
else
X1,. k 815,k-1
end if
end for

---

approximated p- p-value dy:
à = K x11 Σ ([)(X) < ( X-1D).
(5)
To control the FDR, we use the observed p-values. Specifically, we set the threshold to s(>) _ maxy {āg âg S $77, and select all features j such that aj I s(7).
procedure to establish a threshold for the
The Benjamini- Hochberg correction only guarantees FDR control provided that the p-values have either positive or zero correlation. Thus, the FDR control of FLOWSELECT depends on these assumptions being met. A more conservative correction from Benjamini & Yekutieli (2001) allows for arbitrary dependencies in p-values, but it suffers from low power. The Benjamini- Hochhnnn--- correction widely used and empirically robust (Tansey et al., 2021), so we report results using Across synthetic and semi-synthetic benchmarks in Section 5, we also find that FLOWSELECT maintains empirical FDR control.
Provided that the Benjamini-Hochberg assumptions are met, the FDR will be controlled, but the power of the test depends on T being higher when j is significant feature. For example, if Y is expected to vary approximately linearly with respect to X, (X) could be the absolute estimated regression coefficient | for the linear model Y x + c. Another choice is the HRT feature statistic described earlier.
4 Asymptotic results
The ability of FLOWSELECT to control the FDR relies on its ability to produce estimated p-values that converge the correct p-values for the hypothesis test in Equation (2).
Theorem 1. Let X ENXD D be a random feature matrix, where each row Xi,, is independent and identically distributed; x E RNx be the observed feature matrix; and (j be the p-value as defined in Equation (4) with test statistic T₃(X), Suppose there exists a sequence functions (Gm) (rm) )a=1 and a base random variable Z satisfying the following conditions:
Each G continuously differentiable and invertible.
2. G + G pointwise for some map that is triangular, increasing, continuously differen- tiable, and satisfies G(X,,)) 2 Z.
For = 1, 1,,,,.,.. let X be the random feature matrix where each row is independent and has (00))1(2) Then, the p-value in Equation (5) calculated using K MCMC samples targeting X.) X. = Z..-j converges the correct p-value aj with probability
distribution X.
Here sketch the proof. A full proof can be found in Appendix B. First, by construction each G" defines a distribution X 2 (G)) that in turn implies a conditional distribution X" [X -.. We show these conditional distributions converge the true conditional distribution of X.. given X. I..-j. Consequently, the probability of observing higher test statistic under the approximated null distribution X D X written az", will converge to the probability under the true null distribution X., X,, x. i.e. aj. Next, the Cesaro average of K samples from an MCMC algorithm targeting X. (X,,, z., written Ky will converge to az with probability 1 as K + co. Combining these two convergences leads to the stated result.
Assuming the limiting p-values } satisfy the chosen Theorem specifies additional conditions that are sufficient for FDR control. These conditions are not strictly fewer than those required for empirical model-X knockoff-basse based methods to control FDR, but they may be easier to satisfy adequately in practice. For example, the condition that there exists sequence ( converging to the true mapping G is satisfied asymptotically by many flow architectures that are universal distribution approximators, including the Gaussianization Flows and Masked Autoregressive Flows used in our experiments (Huang et al., 2018; Meng al., 2020: Kobyzev et al., 2020). In practice, it is unlikely that exact mapping G will be learned, doing so
assumptions,
5

---

Ground Truth
DeepKnockotf
KnockoftGAN
MASS
DDLK
FlowSelect
40-
40-
40-
40-
40-
40-
30-
30
30-
30
10-
10-
10-
10
10-
10-
0-
C-
0-
20 00
20
30
40
Figure I: A density plot of the feature distribution with coordinate j on the x-axis and coordinate j 2 on the y-axis. The ground truth density is compared to the normalizing flow fitted within FLOWSELECT and the distribution each knockoff method (DeepKnockoff, KnockofIGAN, MASS, and DDLK). To have FDR control, each distribution should match the distribution of the features.
could require infinite training data, infinitely deep transformations, and exact nonconvex optimization. Nonetheless, normalizing flows work extremely well in practice; Theorem gives intuition for the good performance of FLOWSELECT that we observe empirically.
Experiments
5.1 Synthetic experiment with mixture of highly correlated Gaussians
We compare FLOWSELECT to the aforementioned knockoff methods with synthetic data drawn from mixture of three highly correlated Gaussian distributions with dimension D 100." For each knockoff method, we use the exact implementation described in their respective papers, and we utilize the code made publicly available by the authors (c.f. Appendix D.3 for further details). For further comparison, we also implement the MASS knockoff procedure from Gimenez et al. (2019) and the RANK knockoff procedure from Fan al. (2020). These methods estimate the unknown feature distribution using either mixture of Gaussians (MASS) or sparse precision matrix (RANK), and then sample the knockoffs directly as Candès et al. (2018).
To generate the data, we draw N 100, 000 highly correlated samples. For i = 1,..., , N, we sample
X; idd Σ m=1 m= (XPP((Xi Em),
(0, 20, 40). and covariance matrices Em. Each covariance Σn follows an AR(1) pattern such that (Em)i,) p where (0.982, 0.976, 0.970). The response Y; is linear in fa(X, for some function fi and coefficient vector / i.e., Y, 3 + 63. Each coefficient 3, equals 100 VN B,, where B, with probability 0.8, B₁ = 1 with probability 0.1, and Bj = -1 with probability 0.1. We consider two different schemes for the Sa that connect the features to the response. In our linear setting, fi is equal the identity function. In our nonlinear setting, fx(x) is set equal to sin(5x) for odd and fü (x) = c5sr for even
with mixing weights (0.371, 0.258, 0.371), mean vector
The experimental setting we have described so far is adapted from Sudarshan et al. (2020). However, we found that the N 2000 they used was too few observations for any of the methods to do well in a general non-linear setting. Moreover, in many situations where controlled feature selection is deployed, neighboring features will be highly correlated. To reflect this, also increased the base correlation between features within each mixture to create a more challenging example. We show results under the original settings of Sudarshan et al. (2020) in Appendix K.
For each model, we use 90% the data for training to generate null features and the remaining 10% for calculating the feature statistics. To define the feature statistics, we use the holdout randomization test (HRT) described at the end Section For the HRT, we employ different predictive models for each response type ("linear" and "nonlinear"). Specifically, for the linear response, use the predictive log-likelihood from the LASSO (Tibshirani, 1996), and for the nonlinear response, we use the predictive negative mean-squared error from random forest regressor (Breiman, 2001).
Software to reproduce experiments available http:///tthbbboomm///cehhnnnnnss

---

Hiatura-of-Gaussians Gaussians
scRNA-seg
1.00
1.00
0.75
0.75
Power and FDR
0.50
Power and FDR
0.50
0.25
0.25
0.00
0.00
1.00
1.00
0.75
Nov-iines
0.75-
0.50
0.25
0.00
0.00
DuupKnookel
MASS
RANK
Rloweelee
Deapknewke
MASS
RANK
DDLA
RowSeleel
Deapknockeon
MASS
RANK
FlowSeeeed
Deepkrecsell
MASS
RANK
DDLK
FlowSeleel
Deepinoodoot
MASS
RANK
FlowSeleel
DBLA
Desptnockat
MAS
DELA FlowBelect
Krboorati
Metric
FDR
Power
Metric
FDR
Power
Figure 2: Comparison of power and false discovery rate (FDR) control of FLOWSELECT to knockoff methods on the FDRs of 0.05, 0.1, and 0.25 (indicated by the dashed lines). Each point indicates the mean power and FDR across 20 replications and the error bars span one standard deviation either direction. the top row, the response depends linearly on the features, and the feature statistics are calculated using the HRT with the LASSO. the bottom row, the response depends non-linearly on the features, and the feature statistics are calculated using the HRT with random forest regression.
dataset (left) and the scRNA-seq dataset (right) at targeted
First, we look at how each procedure models the covariate distribution in Figure 1. In order to be valid knockoff's, the distribution of two knockoff features needs to be equal to that of the covariates. In this challenging example, each of the empirical knockoff methods fails to match the ground truth. In particular, DDLK and DeepKnockoff' are over-dispersed, while KnockoffGAN suffers from mode collapse. These findings for Knocknoc and KnockoffGAN are similar to those reported by Sudarshan et al. (2020), Other than MASS, which directly fits a mixture of Gaussians, FLOWSELECT the only method that matches the basic structure of the ground truth.
Figure shows that the empirical knockoff procedures fail to control the FDR for both linear and nonlinear responses. One explanation for this lack of FDR control is the inability of the deep- learning-based methods to accurately model a knockoff distribution (c.f., Figure 1). As a result, the assumptions for the knockoff procedure will hold, and FDR control is not guaranteed.
The effects misspecification are clearly visible in the case of RANK, which approximates the mixture-of -frasssinns data with multivariate Gaussian. However, even MASS, when given access to the correct data distribution, does not achieve across-the- -ho-b FDR control. This highlights the potential sensitivity of knockoff's parameter misfit even when the underlying distributional family of the features is known. This confirmed by the fact that, when provided with the true parameters, the oracle Model-X maintains FDR control, though with significantly less power than SELWSE fc.f. Appendix H).
5.2 Semi-synthetic experiment with scRNA-seq data
In this experiment, we use single-cell RNA sequencing (scRNA-seq) data from 10x Genomics (10x Genomies, 2017). Each variable Xn.g is the observed gene expression of gene g in cell n. These data provide an experimental setting that is both realistic and, because gene expressions are often highly correlated, challenging. More background information about scRNA-seq data can be found in Agarwal et al. (2020).
We normalize the gene expression measurements to lie in [0. 1], and we add small amount of Gaussian noise so that the data is not zero-inflated. As in the semi-synthetic experiment from Sudarshan et al. (2020), we pick the 100 most correlated genes to provide challenging, yet realistic example. We simulate responses that are both linear and nonlinear that FL.OWSELEE maintains FDR control across multiple FDR target levels, feature statistics, and generated responses. cases in which the knockoff methods control FDR successfully, FLOCST has higher power in discovering the features the response depends
features. Figure shows
7

---

An advantage of knockoffs over CRT-based methods like FLOWSELECT is that the predictive model only needs be evaluated once. Hence, while FLOWSELECT has faster runtime than DDLK for this experiment, it is slower than DeepKnockoff and KnockofIGAN, However, Figure 2 shows that these two models fail to reliably control FDR and have much less power than FLOW it not clear how additional computational resources could be leveraged to improve the performance of these competing methods. A full table of runtimes on the scRNA-seq dataset can be found in Appendix F.
The need to compute a different predictive model for each feature within the CRT is mitigated by using efficient feature statistics such as the HRT (Tansey et al., 2021) and the distilled CRT (Liu et al., 2020). These methods fit larger predictive model once, then evaluate either the residuals or test mean waeraan-surrdd-eeee -eror for each feature individually. Moreover, the ability scale to large feature dimensions D is more limited by fitting the feature distribution than computational burden, trait shared by both knockoff- and CRT-based methods.
FLOWSELECT provides asymptotic guarantees of FDR control assuming sufficient MCMC samples have been drawn for the p-values to converge. In this experiment, the consequence of terminating MCMC sampling before convergence is low power, rather than loss of FDR control (see Figure 7 in Appendix J). Even for small numbers of MCMC samples, the FDR stabilizes below the target rate, while the power steadily increases with the number of samples. Because the MCMC run is initialized the true features, we speculate that the sampled features will be highly correlated with the true features in the beginning of the run, making it harder to reject the null hypothesis that feature is unimportant.
5.3 Ablation Study
FLOWSELECT differs from the competing moockoff-baseel approaches in two ways: using normalizing flows with MCMC to model the feature distribution for sampling null features and using the CRT for feature selection. To illustrate the impact of each of these components separately, compare to the procedure used in Tansey et al. (2021), which uses mixture density networks (MDNs) to model the complete conditional distribution of each feature P(X,l X-j) -j) separately. They then sample null features from these learned distributions directly and use the HRT for feature selection. Since both FLOWSELECT and this procedure utilize the HRT, this allows to evaluate whether the performance improvement of PLOWSELECT over empirical knockoffs is solely due to use of the HRT.
We compare the ADN-based approach to FLOW SELECT on the mixture-of- -fausssian (Section 5.1) and scRNA-seq (Section 5.2) datasets. A plot of this comparison can be found in Appendix G. While the MDN-based approach was able to match the performance of SELECT on the scRNA- seq dataset, it failed to control FDR at any level on the Mixture-of-Gaassinn that MDNs are less flexible than normalizing flows. In aggregate, these results show that both the normalizing flows paired with MCMC and the use of the HRT for significance testing are key to the performance of FLOWSELECT
dataset, indicating
5.4 Real data experiment: soybean GWAS
Genome-wide association studies are way for scientists to identify genetic variants (single-nucleotide polymorphisms, or SNPs) that are associated with particular trait (phenotype). We tested FLO LECT on dataset from the SoyNAM project (Song et al., 2017), which is used to conduct GWAS for soybeans. Each feature X, takes on one of four discrete values, indicating whether particular SNP is homozygous in the non-reference allele, heterozygous, homozygous in the reference allele, or missing. A number of traits are included in the SoyNAM data; we considered oil content (percentage in the seed) as the phenotype of interest in analysis. There are 5,128 samples and 4,236 SNPs in total.
To estimate the joint density of the genotypes, we used a discrete flow (Tran et al., 2019). Modeling of genomic data is typically done with hidden Markov model (Xavier et al., 2016); however, such a model may fail to account for long range dependence between SNPs, which normalizing flow better suited to handle. Having a more flexible model of the genome enables FLOWSELECT to provide better FDR control for assessing genotype/phenotyne relationships. For the predictive model, we used feed-forward neural network with three hidden layers. Additional details of training and architecture presented in Appendix E.
8

---

1605024
3753923
N67777
3739190
1821002
12
13
16
Chromosome
Figure 3: Manhattan plot for oil content in soybean GWAS experiment (Turner, 2018). is the estimated p-value from the FLOWSE procedure, and the blue line indicates the rejection threshold for nominal FDR of 20%.
A graphical representation of our results is shown as Manhattan plot in Figure 3, which plots the negative logarithm the estimated v-vllues for each SNP. At a nominal FDR of 20%, we identified seven SNPs that are associated with oil content in soybeans. We cross-referenced our discoveries with other publications identify SNPs that have been previously shown be associated with oil content in soybeans. For example, FLOWSELEC SELECT identifies one SNP on the 18th chromosome, Gm18_1685024, which is also selected in Liu al. (2019). FLOWSELECT also selects SNP on the 5th chromosome, Gm05_374677n 37746779777 which is near two SNPs (Gm05 38473956 and 1mm05385063777 identified Cao et (2017) but which are not in the SoyNAM dataset. Sonah et al. (2014) identifies eight SNPs near the start of the 14th chromosome, and we select multiple SNPs in nearby region on the 14th chromosome (seen in the peak of dots on chromosome 14 in Figure 3). However, the dataset exact comparison. A list SELECT tests over 4000 features in 10 hours using single GPU. None of the empirical knockoff procedures (Sudarshan et al., 2020; Jordon et al., 2019; Romano et al., 2020) tested more than 387 features. This shows the potential for FLOW SELECT for high-dimensiona feature selection with FDR control in reasonable amount of time. Additional details about this experiment are available in Appendix E.
in Sonah et al. (2014) is much larger (+= 47, 000 SNPs), which prevents of all SNPs selected by our method is provided in Appendix E. For this experiment,
6 Discussion
FLOWSELECT enables scientists and other practitioners to discover features a response depends on while controlling false discovery rate, using an arbitrary predictive model; even large-scale nonlinear machine learning models can be utilized. By making fewer false discoveries for fixed sensitivity level, FLOW SELECT can reduce the cost of follow-up experiments by limiting the number irrelevant features considered. In contrast to the original model-X knockoff's method, FLOWSELEC does not require the feature distribution to known a priori, nor does it require the feature distribution to have a particular form (c.g., Gaussian). Neither of these conditions are often satisfied in practice.
One limitation shared by both the conditional randomization test (CRT) and knockoffs is low power in cases in which important features are highly correlated with other important features. To mitigate this limitation, the CRT can be applied to test the significance groups correlated features rather than individual features. Within the FLLOWSEL.CC" framework, this entails modifying the MCMC step to draw null samples of groups of features conditioned on the others. The group's p-value can then be calculated with the same holdout randomization test (HRT) statistic used for testing individual

---

features. Group feature selection has also been explored for knockoff's (Dai & Barber, 2016; Liu al., 2020).
Another limitation of PLOWSELECT stems from its reliance on normalizing flows. The flexibility of normalizing flows, though often heneficial, comes at cost: sufficient training examples needed to learn the feature distribution, limiting applicability in data-starved regimes, Fortunately, as we show in Appendix FLOWSELECT fares no worse than competing methods in low-data settings. In these regimes, FLOW SELECT could also use other density estimation techniques such as autoregressive models.
Furthermore, learning the feature distribution (potentially from limited data) is not the sole difficulty that the deep-leamnng-baase based knockoff methods face. To demonstrate that there additional sources of difficult for knockoff-basee methods, we gave DDLK, which typically fits the data distribution as part of its training procedure, access to the the exact joint densiry; neither the empirical FDR the power improved significantly (c.f. Appendix 1). This result points to a failure of DDLK to enforce the swap property, which a challenging task the number of swaps grows exponentially with the number of features. FLOWSELECYS on the other hand, achieves FDR control under a different set of conditions that often are simpler to satisfy adequately in practice.
Acknowledgments and Disclosure of Funding
Derek Hansen acknowledges support from the National Science Foundation Graduate Research Fellowship Program under grant no. 1256260. Any opinions, findings, and conclusions recommen- dations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
10x Genomics,
Our 1.3 million single cell dataset is ready to download.
2017.
Agarwal, D., Wang, J., and Zhang. N. Data denoising and post-denoising corrections in single cell rna sequencing. Statistical Science, 35 2020.
Bates, S., Candès, E., Janson, L., and Wang. W. Metropolized knockoff sampling. Journal of the
American Statistical Association, pp. 1-15, March 2020. ISSN 0162-1459, 1537-274X. doi: 10.1080/0162144 6214691199999 200222111922222.11111 1729163.
Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B. 57(1))289- 300, 1995. doi: 1111111/22117-66111199999 1bt..99....0001....o URL https://doi.or 2517-6161.0 1995. tb02031.x.
1111/j.
Benjamini, Y. and Yekutieli, D. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 16(5):1655-1188 August 2001. ISSN 0090-5364, 2168-8966. doi:
Bogachev, V. I., Kolesnikov, A. V., and Medvedev, K. V. Triangular transformations of mea- doi: 10.1070/
sures. Shornik: Mathematics, 196(3):309, April 2005. ISSN 1064-5616. 1022005v196m0//BBHHOG
Breiman, L. Random forests. Machine Learning. 45(1):5-32, 2001. ISSN 0885-6125. doi:
10.1023/A:00109334444333 URL http: doi org/10. 10. 10t..002////10 /133111093334444224
Candès, E., Fan, Y., Janson, L., and Lv, J. Panning for gold: 'model-X' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B, 80(3):551-577, 2018. ISSN 1467-9868, doi: 10. 10.1111 1/rssb. 12265.
Cao, Y., Li, S., Wang, Z., Chang, F., Kong, J., Gai, J., and Zhao, T. Identification of major quantitative trait loci for seed oil content in soybeans by combining linkage and genome-wide association mapping. Frontiers in Plant Science, 8, July 2017. doi: 10.3389/fps..217.01222 URI. https:// //doi. org/10. 10. 3389/f339300 hfpp///pl..././ 2017. 2017.01222. 01222.
10

---

Dai, R. and Barber, R. The knockoff filter for FDR control in group-sparse and multitask regression. In Proceedings The 33rd International Conference on Machine Learning, pp. 1851-1859. PMLR, June 2016.
Fan, Y., Demirkaya, E., Li, G., and Lv, J. RANK: Large- Scale Inference With Graphical Nonlinear Knockoffs. Journal of the American Statistical Association, 115(529)362-379, January 2020.
ISSN 0162-1459. doi: 10.1080/0162 (100001211599900 459.20188 8. 1546589.
Friedman, J., Hastie, T., and Tibshirani, R. Sparse inverse covariance estimation with the graphical lasso. Biostatisticx, 93))4322-44, July 2008. ISSN 1465-4644. doi: 10. N0.000//biosaaissies/k//.55
Gayoso, A., Lopez, R., Xing. G., Boyeau, P., Wu, K., Jayasuriya, M., Mehlman, E.,, Langevin, M., Liu, Y., Samaran, J., Misrachi, G., Nazaret, A., Clivio, O., Xu, C., Ashuach, T., Lotfollahi, M., Svensson, V., da Veiga Beltrame, E., Talavera-Lopez, C., Pachter, L., Theis, F. J., Streets, A., Jordan, M. I., Regier, J., and Yosef, N. sevi-tools: library for deep probabilistic analysis of single-cell omics data. bioRxiv, 2021, doi: 10. 1011/10022210. 20..,2..44183... URL https: //www 2002//0//9/2000000 04 28. 441833.
biorxiv. wrg. wwarg//ontent/ear////
Gimenez, J. R., Ghorbani, A., and Zou, J. Knockoff's for the mass: New feature importance statistics with false discovery guarantees. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, 16-18 Apr 2019. URL httpt:///pp/eeeeeee v89/ gimenez19a html.
.mlr. .pres//
Huang, C.-W., Krueger, D., Lacoste, A., and Courville, A. Neural autoregressive flows. In Interna- fional Conference on Machine Learning, July 2018.
Jordon, J., Yoon, J., and van der Schaar, M. KnockoffGAN: Generating knockoffs for feature selection using generative adversarial networks. In International Conference on Learning Representations, ttttt/oorummidByy22555353333 2019. URL https:/ httptttttt::/eeeeeettete /otp://iie//reeiee///eeemee
Kobyzev, I., Prince, S., and Brubaker, M. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Liu, M., Katsevich, E., Janson, L.,, and Ramdas, A. Fast and Powerful Conditional Randomization Testing via Distillation. arXiv:2006.03980 July 2020.
Liu, Y. and Zheng, C. Auto-Encodiig Knockoff Generator for FDR Controlled Variable Selection. arXiv: 1809. 10765, September 2018. 8.
Liu, Y., Wang, D., He, F., Wang. J., Joshi, T., and Xu, D. Phenotype prediction and genome-wide association study using deep convolutional neural network of soybean. Frontiers in Genetics, 10, 2019. ISSN 1664-8021. doi: 203.8//fgeeee201..000 org/ Eng/articles//i 2388/ffgers 2019 01091/ful1.
URL https: //w::///www.... frontiersin.
Meng, C., Song, Y., Song, J., and Ermon, S. Gaussianization flows. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 26-28 Aug 2020. URL httpp:/prroceedigg mlr
www.srreessiiseeee mwwgg00n...ccmm//m mmmess//000///og20... html.
Papamakarios, G., Pavlakou, T., and Murray, I. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, vol- ume 30. 2017. URL sttps://proceedingg www.siigg...u.ss.......... https/scc/appp//0///00ee.
6c122886222772206777700666888883 Paper pdf.
Papamakarios, G., Nalisnick, E., Rezende, D. J.. Mohamed, S., and Lakshminarayanan, B. Normaliz- ing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22 (57):1-64, 64, 2021.
Romano, Y., Sesia, M., and Candès, E. Deep knockoffs. Journal of the American Statistical Association, 115(532):1811111822 2020. doi: 1..10000661149990999.6666 //doi.org/10.eeoo//oooo/oo 100....0000000001.011110000 2019 21114499...211111117644 1660174.
URL https:
Smith, A. F. M. and Roberts, G. Bayesian Computation Via the Gibbs Sampler and Related Markow Chain Monte Carlo Methods. Jourmal of the Royal Statistical Society: Series B (Methodological), 55(1):3-23, 1993. ISSN 2517-6161. doi: 1111111/2517-66111199..00000

---

Sonah, H.. O'Donoughue, L., Cober, E.. Rajcan, I., and Belzile. F. Identification of loci governing eight agronomic traits using GBS-GWAS approach and validation by QTL mapping soya bean. Plant Biotechnology Journal, 13(2):1112211, September 2014. doi: 10.111 http:/pb...24444 URL attpps///oot org/10. 10. 1111/pbi. 12249.
Song. Q. Yan, L., Quigley, C., Jordan, B. D., Fickus, E., Schroeder, S., Song, B.-H., Charles An, Y.-Q., Hyten, D., Nelson, R., Rainey, K., Beavis, W. D., Specht, J., Diers, B., and Cregan, P. Genetic characterization of the soybean nested association mapping population. The Plant 10.0109. URL https: Genome, 10(2), 2017. doi: caaley.com/ooii. /a/////////// http:///0..0///paaaageoneeee0o0 10. 0109. //acsess... onl iararrrrrrr...... caarrray.wlly.....
Sudarshan, M., Tansey, W., and Ranganath, R. Deep direct likelihood knockoffs. In Advances in Neural Information Processing Systems, volume 33, 2020.
Tansey, W., Veitch, V., Zhang, H., Rabadan, R., and Blei, D. M. The holdout randomization test for feature selection in black box models. Journul of Compulational and Graphical Statistics, 2021. [In press; available on arXiv].
Tibshirani, R. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B, 58:267-288, 1996.
Tran, D., Vafa, K., Agrawal, K., Dinh, L.,, and Poole, B. Discrete flows:
Invertible generative models
of discrete data. In Advances in Neural Information Processing Systems, volume 32, 2019.
Turner, S. qqman: an R package for visualizing GWAS results using Q-Q and Manhattan plots. The Journal of Open Source Software, 2018. doi: 10.21105/joss.0077 00731.
Xavier, A., Muir, W., and Rainey, K. Impact imputation methods on the amount genetic variation captured by single-nucleotide polymorphism panel in soybeans. BMC Bioinformatics, 17(1), February 2016. doi: 10.1188//12859-01666 1859-016-089- URL https: //op:: org/ /10. 1186/ s12859-016 019-0899--6.
Xavier, A., Beavis, W., Specht, J., Diers, B., Mian, R., Howard, R., Graef, G., Nelson, R., Schapaugh, W., Wang, D., Shannon, G., McHale, L.., Cregan, P., Song, Q., Lopez, M., Muir, W., and Rainey,, K. SoyNAM: Soybean Nested Association Mapping Dataset, 2019. URI.httss://CRt R- project. org/packageeeee package=SoyNNM R package version 1.6.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Runtime discussion in section 5.2
and limitations of normalizing flows in section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethies review guidelines and ensured that your paper conforms
them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Theorem statement lists all assumptions.
(b) Did you include complete proofs of all theoretical results? [es] Yes; Appendix B contains the proof to Theorem
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Code included as supplement, and specific details were described in the Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Appendices D and E.
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] All figures comparing FDR and Power results (Figure 2 in main text, Figures 4-6, 8 in the Appendix) contain error bars indicating results across runs.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We specify hardware used in Appendix F.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix C
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A] Data are publicly available
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "xU2qzLAtyR", "reviewer": "Reviewer_vcjW", "summary": "This work considers the controlled feature selection problem, or selecting a small number of important features subject to FDR control. The proposed method, FlowSelect, builds on the existing literature of conditional randomization tests and holdout randomization tests, as well as normalizing flows from the deep generative modeling literature.\n\nIn simple terms, FlowSelect uses normalizing flows to learn a data distribution and MCMC to sample individual held-out features from their conditional distribution; it then passes these samples through a single model fit on the original data to generate samples from the null distribution, and it then compares these to the observed test statistic calculated using the observed data (with the rejection threshold set according to Benjamini-Hochberg).\n\nThe proposed approach is justified using an asymptotic argument: normalizing flows are theoretically capable of modeling any distribution given enough transforms, and MCMC should be able to sample from the learned distribution given enough samples. The empirical results are quite promising, FlowSelect provides better results than several competing knockoff methods across two types of synthetic tasks, as well as reasonable results on a real scRNA-seq dataset.", "questions": "- Would the authors be able to include additional results showing how FlowSelect (and the baselines) perform given a smaller dataset? I would be curious to see if the normalizing flows approach scales better or worse than the other methods, particularly those that use deep learning\n- Based on a quick Google search, it seems there are other papers that have used MCMC to sample from distributions parameterized by normalizing flows (e.g., https://arxiv.org/abs/2105.12603). Are the authors aware of these, and would it make sense to comment on the similarities/differences between this and some existing use cases of this trick?\n- As mentioned above, the asymptotic argument for why FlowSelect should work is somewhat optimistic and doesn't quite explain why we should expect this approach to work in the finite regime (finite transformations in the flow model, finite MCMC sampling). Can the authors comment on whether they might be able to refine their analysis, or perhaps include a discussion of how to verify if either part is working \"well enough\" ? (E.g., metrics for whether the normalizing flow fit the distribution well, established techniques for deciding if MCMC has been run for long enough.)\n- Could the authors add a note about why it's important to add Gaussian noise in the scRNA-seq experiment (what the issue is with zero-inflation)?\n- This isn't a weakness in FlowSelect specifically, but perhaps in all CRT-based methods that may be helpful to address: it seems like a CRT may fail to identify an important feature that is correlated with another important feature, because removing either from the model should have no impact on the test statistic. Could the authors explain this and why it is/isn't a problem, and if necessary add a note (perhaps in section 2)? ", "limitations": "A couple of the questions above get at limitations that might be helpful to discuss:\n- Ability to identify important features that are strongly correlated with one another (in the worst case, identical)\n- Limited applicability of theory to practical usage\n- Unclear scaling with dataset size\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The method combines known methods for feature selection under FDR control with two new techniques (at least in this subfield): normalizing flows and MCMC to sample from the learned distribution\n- The asymptotic argument makes clear why this approach should work, at least in scenarios with a large amount of data, a large flow model, and sufficient MCMC samples\n- The empirical results are quite promising\n- The ablation study showed that multiple methodological innovations are important for helping FlowSelect outperform the baselines (the normalizing flows + MCMC approach for handling held-out features, as well as the HRT)\n\n### Weaknesses\n\n- The theory is somewhat optimistic: the p-values converge only under two asymptotic results. It's natural to develop asymptotic theory first as this is often easier, but I wonder if there's any way the theory could be refined to more closely reflect how FlowSelect performs in practice, or at least help control whether either of the key steps is working well enough (more on this in the questions section below)\n\n\nNits:\n- This may not be a mistake, but I wasn't sure of the reason for the +1 in the summation and denominator of eq. 5\n- There's a typo on line 252, \"to have support in...\"\n- I may have missed this, but what exactly was the test statistic used in the experiments? The caption of figure 2 says the statistics were calculated using either Lasso or random forest; was it the model's predictive accuracy? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PPhtR0kcow2", "reviewer": "Reviewer_qGxr", "summary": "This paper proposes a new variable selection method called FLOWSELECT to perform controlled feature selection that does not suffer the problems existed in Knockoffs-based methods. Asymptotically,  the proposed method computes valid p-values. Empirically, FLOWSELECT consistently controls the FDR on both synthetic and semi-synthetic benchmarks, whereas competing knockoff-based approaches do not. ", "questions": "[Fan Y. et al] propose a modified model-X knockoffs method, called graphical nonlinear knockoffs (RANK), to accommodate the unknown covariate distribution. The related theoretical results verify the robustness of the RANK by showing that the false discovery rate (FDR) is asymptotically controlled at the target level and the power is asymptotically one with the estimated covariate distribution. \n\n+ Does the RANK overcome the first problem often occurred in the knockoff-based method emphasized by the authors? \n\n+ It would be appreciated if the author could discuss the difference between this method and the above method. \n\n+  Can rank method be added as a baseline in the numerical experiments？\n[Fan Y. et al] Large-Scale Inference With Graphical Nonlinear Knockoffs, JASA.", "limitations": "All my concerns are presented in “Weakness” and “Questions”. Moreover, I do not have any concern on negative social impact.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+  The paper is well-motivated and well-written.\n+  The proposed method has nice theoretical guarantees and sufficient empirical evaluations.\n\n\nWeakness:\n+ In fact, some work on knockoff has improved the two problems highlighted by the author that lead to the failure of traditional knockoffs-based methods, e.g., “Fan Y. et al. RANK: Large-Scale Inference With Graphical Nonlinear Knockoffs.”. However, the authors does not discuss this related literature.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OPU-lprWYGn", "reviewer": "Reviewer_ULQR", "summary": "The paper considers the problem of multiple hypothesis testing with the FDR control. The work is situated in the model-X setting and is motivated by the limitations of the existing model-X multiple testing problems, with an emphasis on the knockoff-based methods. The paper proposes using the normalizing flow technique to estimate the conditional distribution of features, i.e., $X_j \\mid X_{-j}$, which combine with CRT/HRT yields (approximately) correct p-values for conditional hypothesis testing. The p-values are subsequently fed to the BHq procedure, and a selection set is provided. The proposed method is evaluated with synthetic and semi-synthetic data, showing satisfying empirical results when compared to other methods that rely on inexact knockoff constructions.", "questions": "My major questions have been listed in the \"strength and weaknesses\" part. Some minor questions follow.\n\n1. Line 55: should it be \"In situations...\"?\n\n2. Line 157: is $\\hat{\\alpha}_j$ used as the p-value for both hypothesis $j$ and the ordered p-value?\n\n3. Figure 1: what do the x and y-axis represent?\n\n4. Line 212: what is $\\Sigma_j$; line 213: what does the index $j$ mean in the definition of $\\rho$.\n\n5. Figure 2: it might be helpful to enlarge the fonts.\n\n6. Figure 4: what do the column panels represent?\n\n", "limitations": "My comments on the limitations have been included in the \"strengths and weaknesses\" part. The authors have adequately addressed the potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The paper finds a nice connection between a state-of-art  conditional density estimation method and the model-X feature\nselection method.\n\n2. The proposed method performs well empirically.\n\nWeaknesses:\n1. I find the motivation and comparison of the proposed method unsuitable. The paper tries to see the proposed method as an alternative method for the knockoff-based method; but at its best (i.e., when the mapping G is exactly known), the proposed method does not guarantee the FDR control due to the dependence between the p-values. Further, the target distribution knockoffs (or inexact knockoff machines) tries to learn is different from what the proposed method tries to learn: the former is a knockoff distribution satisfying the ``swapping invariant'' property while the latter is simply the conditional distribution. I would suggest framing the proposed method as a way of implementing CRT/HRT when the feature distribution is not exactly known. \n\n2. Following the previous comment, I find the comparison in the simulations a bit unfair: methods w/ different guarantees;\none-bit p-values versus multi-bit p-values. For example, it might be more appropriate to consider using the existing deep method to learn $X_j \\mid X_{-j}$ and apply CRT/HRT.\n\n3. The presentation of this paper can be improved. \n\na) the notation is not consistent throughout the paper. For example, the dimension is represented as d in line 44, then D in line 45, and then p in line 103. \n\nb) The description of the existing method is a bit hard to follow. For example, it may help to briefly explain where the test data and training data come from; Theorem 1 involves the index n, and it would help to explain what is this n in the actual flow used (is it M?).\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Rq6Y9W64e7", "reviewer": "Reviewer_rjGq", "summary": "The authors present a method, called FlowSelect based on normalizing flows to select features in a controlled fashion, meaning that the false discovery rate (FDR) is limited. They select features by first approximating the data distribution via the normalizing flow, and then compute a p-value for each feature explaining the data with respect to some model, e.g. a Lasso regression model or a Random Forest. Estimating the p-values involves sampling from the marginal distribution of the flow, which is done via MCMC.\n\nTo make sure that the FDR can be controlled, the authors prove that their estimates of the p-values are asymptotically correct, i.e. that the estimates converge to the true value almost surely.\n\nIn their experiments, they apply their method to a synthetic dataset, i.e. correlated Gaussians, a semi-synthetic dataset based on gene data, and a real-world dataset where they try to discover genes relevant for the oil content in soy beans. Their method outperforms several baseline procedures based on knockoffs, etc., and they are able to find SNPs relevant for the oil content which have also been confirmed by other studies.", "questions": "* What is the computational cost of computing the marginals of the flow for high dimensional systems?\n* How do you choose the proposal for the MCMC procedure sampling from the marginals?\n* Could you apply the other baseline methods to the genomic soy bean dataset?", "limitations": "It is not clear how well FlowSelect can be scaled up to very high dimensional datasets such as genomic data with 10k+ features.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nIn general, the paper is well-structured and easy to read. I appreciate the background section, especially, because it guides the reader through the four lines of research the authors take ideas from and combine them to a new method.\n\nFlowSelect combines several known and established methods in a creative way. It is well grounded from a theoretical perspective as the authors prove that their estimator for the p-values converges to the true value almost surely.\n\nFlowSelect was put to test on three different datasets. It should to have high power on the synthetic and semi-synthetic dataset, higher or on par with all of the baselines, while having FDR as high or lower than the threshold, which is not the case for all the baselines. That the authors are also able to identify SNPs, which are relevant for the oil content in soy beans, is impressive.\n\n## Weaknesses\n\nAlthough the authors compare the SNPs they identify in the soy bean experiment to what has been found in other studies, they do not apply the baselines they used in the previous experiments to the same dataset, which makes their findings slightly less convincing.\n\nMoreover, they give little insight in the computational cost of their method compared to competing procedures. Computing the marginals from the flow is only possible through MCMC, which can be very inefficient if the proposal distribution does not represent the actual distribution well. Especially when dealing with genomic data with 10k+ features this could render the method very inefficient. Unfortunately, the authors give very little insight into how they chose these proposal distributions and ensured that they represent the true marginals well.\n\n## Conclusion\n\nI'm in favor of accepting the article, although I wish that the others could clarify the concerns I raised above.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept