KnockoffGAN: Generating Knockoffs for Feature Selection using Generative Adversarial Networks

1 INTRODUCTION . Feature selection is a pervasive problem . Often the goal is to discover features that are relevant to a particular outcome , either for the sake of discovery itself or to aid in prediction [ 16 ; 25 ] . When the focus is on discovery , feature selection methods typically focus on trying to control either the Family-Wise Error Rate ( FWER ) or the False Discovery Rate ( FDR ) . The FWER measures the probability of making a single false discovery ( a Type I error ) among the selected features ( i.e . selecting one which is not relevant ) , whereas the FDR measures the proportion of false discoveries made ( i.e . the proportion of selected features which are false ) . Controlling FWER , however , leads to reduced power ( i.e . selecting fewer relevant variables ) since it controls the probability of making any false discovery , whereas FDR tries to control the proportion of false discoveries . Controlling the FDR is important [ 5 ; 6 ; 3 ] . Often , data-driven feature selection will be used to select a set of candidate features for further investigation . When further investigation is expensive ( for example when further investigation would involve conducting new experiments and collecting more data ) , a method that can not control the FDR may result in a large amount of wasted resources , with no guarantee that anything meaningful will be discovered . On the other hand , being able to control the FDR at , say , 10 % ensures that at most , 10 % of the spent resources are wasted , and 90 % are in fact spent on discovering positive , useful results . It should be noted , however , that estimating the FDR of a method empirically is hard in practice , since we do not have access to the ground truth relevance . As such , a theoretical analysis of the method and its ( potential ) FDR-controlling properties must be carried out , which does not exist for many existing feature selection methods . [ 3 ] is the seminal paper on the knockoff framework , which is an innovative FDR-controlling feature selection method . Knockoffs are features that are generated to “ look like ” the real features but be conditionally independent of the label given the real features . Feature statistics ( such as the coefficients of a LASSO [ 32 ] ) are compared between the real features and their knockoffs and a selection is made when this difference is sufficiently large . Performing the selection in this way allows for an estimate of the FDR to be obtained and the selection threshold can be adjusted to control the FDR at the selected level . In the original paper , the relationship between the label and the features is constrained to be of a very specific form ; in [ 7 ] , they remove this constraint and instead provide a theoretical analysis that shifts the burden of knowledge onto knowing the underlying feature distribution . Unfortunately , while the theoretical results hold for any feature distribution , they rely on being able to generate valid knockoffs , for which [ 7 ] only provide a method for generating knockoffs when the distribution is a ( known ) multivariate Gaussian distribution . In this paper , we modify the Generative Adversarial Networks ( GAN ) [ 11 ] framework to address this problem , allowing us to generate knockoffs for any distribution ( and without any prior knowledge of it ) . GANs have been shown to be a powerful method for learning to generate complex distributions [ 24 ; 20 ; 2 ] . Our main contribution is in modifying the discriminator used in the GAN framework in such a way that the generator learns to generate knockoffs satisfying the necessary swap condition [ 7 ] which requires that when a feature and its knockoff are swapped , the joint distribution remains unchanged . In addition , we propose a method for maximizing the power of our model using Mutual Information Neural Estimation ( MINE ) [ 4 ] and investigate a regularization method to improve the stability of training . Our model consists of four networks : ( 1 ) a generator network that takes as input noise and the real features , and outputs a set of candidate knockoff features ; ( 2 ) a discriminator network taking as input “ swapped ” feature-knockoff features that attempts to determine which variables have been swapped ; ( 3 ) a Wasserstein GAN discriminator that we use as a regularization term ; and ( 4 ) a MINE network that estimates the mutual information between each feature-knockoff pair allowing us to maximize the power of the knockoff procedure . 2 RELATED WORKS . Feature selection is a well-studied problem with a wealth of related works ( [ 12 ; 31 ; 17 ; 22 ] provide a summary of a lot of existing literature ) ; however , most methods do not attempt to control the FDR . The most common feature selection method for FDR control is the Benjamini-Hochberg ( BHq ) procedure and its variants [ 5 ; 6 ] , which relies on obtaining valid marginal p-values for each selection . Knockoffs are an active area of research [ 9 ; 18 ; 10 ] . The notion of a knockoff was first introduced in [ 3 ] with the theory there requiring that the relationship between the features and the label be of a specific form . In [ 7 ] , they build on the knockoff framework , removing this requirement but instead shifting the requirement to one of knowing the distribution of the features . As noted in the introduction , the theory in [ 7 ] holds independent of the distribution of the features - relying only on being able to generate valid knockoffs ( which exist for any distribution of features ) . However , they only propose a method for generating knockoffs when the distribution of features is jointly Gaussian . While they do propose a method for generating approximate knockoffs in the non-Gaussian setting ( by simply approximating the features as Gaussian ) , the guarantees on FDR control do not hold for their approximate knockoffs . In [ 26 ] and [ 10 ] , they add to the class of constructible knockoffs , describing methods for constructing knockoffs for Markov Chains , Hidden Markov Models and Gaussian Mixture Models . Though once again , knowledge of the full distribution is still necessary for their construction . In this paper we use a framework motivated by GANs [ 11 ] to learn to generate knockoffs without any assumptions on the distribution of the features . To do this , we modify the discriminator so that rather than trying to determine whether a sample is real or fake , it attempts to identify which components have been “ swapped ” . In [ 38 ] , an unconventional discriminator is used that performs component-wise discrimination for the purpose of imputation . While the problem addressed in that paper is different to the one here , the key idea relies on a similar modification to the discriminator to be able to appropriately guide the generator . In order to maximize the power of our variable selection mechanism , it will be desirable that the feature-knockoff pairs are ” as independent as possible ” ( this is discussed in [ 7 ] ) . In order to achieve this we will investigate the use of a promising recent paper , MINE [ 4 ] . MINE proposes a neural architecture and training procedure capable of estimating the mutual information between two random variables . As the mutual information between two random variables is zero only when they are independent , we will use this as a measure of independence and attempt to minimize it during the training of our modified GAN . 3 BACKGROUND . In this section we introduce our notation and define knockoffs as in [ 7 ] . Let us denote the feature space by X and the label space by Y . Let the dimension of X be d. Suppose that X = ( X1 , ... , Xd ) and Y are random variables over X and Y . As in [ 7 ] , we will work with the notion of a null set . Definition 1 . A variable Xj is said to be ” null ” if and only if Y is independent of Xj conditional on { Xi : i 6= j } . We defineH0 to be the set of all null variables . Our goal will be to discover as many relevant features as possible while controlling the FDR . For a given ( potentially random ) selection procedure that selects Ŝ ⊂ { 1 , ... , d } , we define the FDR to be FDR = E [ |Ŝ ∩ H0| |Ŝ| ] . Note that this agrees with the usual notion of FDR ( i.e . when defined in terms of the Markov blanket ) under mild assumptions ( for a more thorough discussion see [ 7 ] ) . 3.1 KNOCKOFFS . Definition 2 . A knockoff [ 7 ] for X is a random variable X̃ ∈ X satisfying the following two properties : ( X , X̃ ) d. = ( X , X̃ ) swap ( S ) ( 1 ) X̃ ⊥⊥ Y |X ( 2 ) for all S ⊂ { 1 , ... , d } where ( · , · ) swap ( S ) denotes the vector obtained by swapping the ith component with the ( i+ d ) th component for each i ∈ S and d.= is equality in distribution . In order to use knockoffs for feature selection , we must define an appropriate feature statistic , Wj , that depends on X , X̃ and Y , i.e . Wj = wj ( ( X , X̃ ) , Y ) for some function wj . This function wj must satisfy the following flip-sign property : wj ( ( X , X̃ ) swap ( S ) , Y ) = { wj ( ( X , X̃ ) , Y ) if j /∈ S −wj ( ( X , X̃ ) , Y ) if j ∈ S. ( 3 ) One of the procedures used in [ 7 ] to construct these statistics is to perform LASSO , treating the augmented feature-knockoffs as the features on which to regress . This gives LASSO coefficients b1 , ... , b2d , and the statistic Wj is set to be the LASSO Coefficient Difference given by Wj = |bj | − |bj+d| . Note that the FDR control guarantees hold independently of the choice of statistic , but a poorly chosen statistic can significantly impact the power of the test . In particular , using the LASSO Coefficient Difference in non-linear settings can yield few discoveries . The focus of this paper , however , is on generating the knockoffs , not on the statistic used on top of the generated knockoffs and so in our synthetic experiments , we use a linear model for Y to be able to draw fair comparisons between our model and [ 7 ] . In the real world data experiment , we use a statistic based on Random Forests for both methods [ 37 ] . The following result from [ 7 ] depends only on having obtained knockoffs that satisfy Definition 2 and feature statistics satisfying ( 3 ) ( and in particular do not depend specifically on using LASSO to obtain the statistics ) . Theorem 1 . Let q ∈ [ 0 , 1 ] . Given test statistics , W1 , ... , Wd , satisfying ( 3 ) , let τ = min { t > 0 : 1 + | { j : Wj ≤ −t } | | { j : Wj ≥ t } | ≤ q } . Then the procedure selecting the variables Ŝ = { j : Wj ≥ τ } controls the FDR at level q , i.e . E [ |Ŝ ∩ H0| |Ŝ| ∨ 1 ] ≤ q .

Summary: This paper introduces a novel feature selection method by utilizing GAN to learn the distributions. The novelty of this paper is to incorporate two recent works, i.e. knockoff for feature selection and W-GAN for generative models. Compared to the latest knockoff work which requires a known multivariate Gaussian distribution for the feature distribution, the proposed work is able to generate knockoffs for any distribution and without any prior knowledge of it.