Auto-Encoding Knockoff Generator for FDR  Controlled Variable Selection

1 INTRODUCTION . In the past decades , the machine learning methods have achieved great advancement in improving prediction accuracy . However , prediction accuracy is not the sole quest of ’ big data ’ analysis . A universal aim across a lot of scientific disciplines is to identify the causes of certain outcome . For example , in new drug development , a personalized medicine strategy can be formed by identifying the genetic markers that is biologically linked to certain drug response or resistance . In electronic health record analysis , the goal is to identify ’ actionable ’ factors for the purpose of reducing cost and improving the quality of care . In epidemiology or sociology studies , it is of interest to identify protection or risk factors , especially those that can be changed through public policy or civil planning to improve public welfare . In these scientific endeavors , the need is to identify the important factors associated with a certain outcome , so that further confirmatory investigation ( e.g . randomized international studies ) could be conducted to expand knowledge or change future actions for a better outcome . For this purpose , we are interested in procedures that control the Type I error , which is the chance of a false discovery . With a large amount of factors to test , controlling the chance of any false discovery ( family-wise error ) is too stringent . Therefore the objective is usually relaxed to control the False Discovery Rate ( FDR ) ( Benjamini & Hochberg , 1995 ) . A recent breakthrough in the statistical theory , i.e . the Model-X framework ( Barber & Candès , 2015 ; Candès et al. , 2018 ) , provides a general solution . It can be incorporated with any machine learning models to select true signals associated with the outcome , with rigorous control for FDR . This line of research sheds light on expanding the horizon of supervised learning methods beyond prediction . However , the implementation of Model-X requires the generation of the so called ’ knockoffs ’ , which has limited existing methods . The goal of our paper is to fill in the gap by proposing a model-free method for generating knockoffs with relaxed assumptions for the data , leveraging the power of the recent development in deep generative models . Specifically , we propose a framework to generate knockoffs based on latent variable Z which captures the correlation structure of X . This provides a great generalization from the existing knockoff generation methods for hidden Markov model ( HMM ) and mixture Gaussian to ( but not limited to ) Hidden Markov Random Field . The contribution of this paper is it propose a deep learning framework to generate Model-X knockoffs through latent variable . We provide theoretical justification of the approach and also demonstrate the state of art variational auto encoder ( VAE ) can achieve promising results in the task of FDR controlled variable selection . This paper is among the first works for the new application of representative learning in generating model-X knockoffs . Our discussion and preliminary results for goodness-of-fit also shed lights on future improvement of VAE for generating knockoffs . 2 BACKGROUND . The model-X framework ( Candès et al. , 2018 ) is proposed for the following problem . There are N i.i.d . samples ( X ( i ) , Y ( i ) ) from a population , where the predictors are random variables X = ( X1 , . . . , Xp ) . The outcome Y only depends on a subset of predictors S ∈ { 1 , 2 , . . . , p } , i.e . given { Xj , j ∈ S } , Y is independent of { Xj , j /∈ S } . The smallest set S that satisfies this requirement is considered as the true predictors . The goal is to find a good estimator Ŝ for S with control of the FDR = E ( # { j : j∈Ŝ\S } # { j : j∈Ŝ } ∨1 ) . Barber & Candès ( 2015 ) first proposed the method for linear regression with Gaussian errors , where the design matrix X is assumed to be fixed . Candès et al . ( 2018 ) greatly generalized it to the Model-X framework . The traditional statistical models focusing on the conditional distribution Y |X1 , . . . , Xp . In contrast , Model-X makes no assumption for the conditional distribution , but assumeX has a known distribution for the purpose of generating knockoffs X̃ that satisfies Definition 2.1 . Definition 2.1 Model-X knockoffs for the family of random variables X = ( X1 , . . . , Xp ) are a new family of random variables X̃ = ( X̃1 , . . . , X̃p ) constructed with the following two properties : ( 1 ) for any subset S ⊂ { 1 , . . . , p } , ( X , X̃ ) swap ( S ) =d ( X , X̃ ) ; ( 2 ) X̃ ⊥⊥ Y |X if there is a response Y . ( 2 ) is guaranteed if X̃ is constructed without looking at Y .. The following proposition 1 provide an equivalent condition for checking Model-X knockoffs . Proposition 1 ( Candès 2018 ) The random variables X̃ = ( X̃1 , . . . , X̃p ) are Model-X knockoffs for X = ( X1 , . . . , Xp ) if and only if the joint distributions are equal after swapping Xj and X̃j , i.e . ( Xj , X−j , X̃j , X̃−j ) =d ( X̃j , X−j , Xj , X̃−j ) , for any j ∈ { 1 , . . . , p } , where X−j denote the other variables . And Y |= X̃|X . Suppose one can generate Model-X knockoffs , the features are selected through the model-specific feature statistics Wj = wj ( [ X , X̃ ] , y ) . The requirement for the feature statistic is that the distribution for the Wj ’ s from the null variables ( Xj 6∈ S ) need to be symmetric . This is formalized as the flip-sign property , Wj ( [ X , X̃ ] swap ( S ) , y ) = ( 1− 21 { j∈S } ) wj ( [ X , X̃ ] , y ) . According to Thm 3.4 in Candès et al . ( 2018 ) , the following procedure ( Knockoff ) can estimate the Ŝ controlling for modified FDR , i.e . E ( # { j : j∈Ŝ\S } # { j : j∈Ŝ } +q−1 ) ≤ q . First , compute the threshold τ = min { t > 0 : # { j : Wj≤−t } # { j : Wj≥t } ≤ q } , then the selected set is ( Knockoff ) : Ŝ = { j : Wj ≥ τ } . ( 1 ) A slightly more conservatively procedure ( Knockoff+ ) increase the negative number in numerator by 1 , and is able to control for the usual FDR . The threshold for Knockoff+ is τ+ = min { t > 0 : 1+ # { j : Wj≤−t } # { j : Wj≥t } ≤ q } , and the selected set is ( Knockoff+ ) : Ŝ = { j : Wj ≥ τ+ } . ( 2 ) Notice that the Knockoff approach controls for the modified FDR , which might have more power than Knockoff+ , but it does not guarantee the control for the usual FDR . In the experiments of this paper , we used the Knockoff+ . An example of the feature statistics is the signed max lambda statistics in L1 penalized regressions Barber & Candès ( 2015 ) : for the concatenated design matrix [ X , X̃ ] , define β̂ ( λ ) = argminb { Loss ( y , [ XX̃ ] b ) + λ‖b‖L1 } . Intuitively , the true predictors with strong signals would have entered the LASSO selection with large λ and Xj is entering the model with larger λ than its knockoff X̃j . The null variables will enter the model with smaller penalty . Thus the signed max lambda statistic is defined as Wj = ( Zj ∨ Z̃j ) sign ( Zj − Z̃j ) . Where Zj = sup { λ : β̂2j−1 ( λ ) 6= 0 } and Z̃j = sup { λ : β̂2j ( λ ) 6= 0 } . Various feature statistics has been proposed for common learning methods ( Candès et al. , 2018 ; Gimenez et al. , 2018 ) , however the current knockoff generation methods are still limited for real data from unknown distributions . Although Candès et al . ( 2018 ) provides a general algorithm for generating Model-X knockoffs , i.e . the Sequential Conditional Independent Pairs algorithm , it needs to sample from the conditional distribution of X̃j from f ( Xj |X−j , X̃1 : j−1 ) which becomes intractable with slightly complex distributional assumption for X . Candès et al . ( 2018 ) also proposes a second-order approach by matching first two moments of ( X , X̃ ) swap ( S ) and ( X , X̃ ) . The second-order approach generate the true knockoff when X is from Gaussian distribution which have no high-order moments . Sesia et al . ( 2018 ) proposes an algorithm to sample knockoffs when X is from Hidden Markov Model . Gimenez et al . ( 2018 ) proposes algorithm that applicable for X from simple Bayesian network models ( such as mixture Gaussian ) with distributional assumptions for certain conditional probabilities which need to be estimated and sampled from . 3 A MODEL-FREE KNOCKOFF GENERATOR WITH LATENT ENCODING VARIABLES .. 3.1 EXISTENCE OF DESIRED LATENT VARIABLE Z . Conceptually , the Model-X aims at controlling for the FDR of testing the null hypotheses Xj |= Y |X−j , where j = 1 , . . . , p , and X−j denotes the variables except for Xj . Thus , a necessary condition for Xj to be chosen by Model-X is that it contains signals not contained in the other variables , i.e . Xj can not be a determinant function of X−j . In the knockoff generating process , any component of Xj ’ s that is independent from X−j is easy to be generated . The difficulty and necessity to consider the complex conditional distributions come from the correlated component . Thus we formed a general framework based on the assumption that there exists some latent variable Z so that Xj ’ s are mutually independent conditional on Z . Such Z always exists for any X , one can take Z = X or Z = f ( X ) , such that f is an one-onone mapping between X and Z . However , a larger difference in X̃j and Xj leads to more powerful knockoffs . In our framework , to generate X̃ different from originalX , our framework requires there exists Z such that X|Z is mutually independent and X|Z is not degenerated , i.e . V ar ( X|Z ) > 0 at least for a subset of X with non-zero measure . In the following , we provide motivational examples followed by a lemma to prove such Z exists for general X . A general class of models satisfies this assumption is the Hidden Markov Random Field as illustrated by the undirected graphical models in Figure 1 . The latent variables are from a Markov Random Field and the observed random variables Xj ’ s are generated from Z ′s with independent errors . However , our method is not limited for Hidden Markov Random Field , Z could exist for the non Gibbs Random Field ( X is not from Markov Random Field ) . For example , f ( x1 , x2 ) = 21 { x1 + x2 ≤ 1 , x1 ≥ 0 and x2 ≥ 0 } . Consider the partition of the support { x1 + x2 ≤ 1 , x1 ≥ 0 and x2 ≥ 0 } into countable many squares . The desired Z can be defined as the indicator which squarer X belong to . X|Z is then the uniform distribution on that square , X1 and X2 are independent given Z . The desired latent variable Z always exists for non-degenerate categorical X . If X is from binomial ( for binary data ) or multinomial distribution ( for categorical data ) with positive probability mass , the desired latent variable Z always exists as a categorical variable with probabilities from the tensor decomposition of multi-way contingency array of probability forX Bhattacharya & Dunson ( 2012 ) , i.e. , p ( x ) = ∑ k πk ∏ j p ( xj |z = k ) . Then Xi|Z ’ s are generated from Bernoulli or multinomial with parameters as functions of Z and thus the conditional density p ( Xi|Z ) ’ s are not degenerate . For continuous or infinite discrete state X , De Finetti ’ s theorem states that if the distribution ( X1 , X2 , . . . , Xn ) is exchangeable then there exists a one dimensional non-degenerate latent variable Z that satisfies our requirement . Thus the desired Z exists for the multivariate normal distribution or trans-elliptical distribution ( i.e. , any distribution that can be transformed into multivariate normal distribution by element-wise transformation ) . The mixture of trans-elliptical distribution also has the desired latent Z as the combination of the single latent variable which indicate the cluster and the latent Z for each cluster . More generally , we can prove the desired Z exists with the following Lemma 3.1 . In the lemma , we consider X from a non-degenerate distribution with density f ( x ) with respect to a measure µ for a sigma field defined on Rp . The following lemma showed that one can decompose f into mixture of densities fk , where at least one fk has non-zero variance . Lemma 3.1 For any non-negative density function f : Rp → R whose support contains an interior point , there exist non-negative functions fk : Rp → R , k = 1 , 2 , · · · such that f ( x ) = ∑ k πkfk ( x ) with πk > 0 and fk satisfies the following : 1 ) fk ’ s can be factorized into functions of each coordinates of x , i.e. , fk ( x ) = ∏ j fkj ( xj ) ; 2 ) at least one component fk has non-zero variance , i.e . V ( fk ) = ∫ x2fk ( x ) µ ( dx ) − [ ∫ xfk ( x ) µ ( dx ) ] 2 > 0 . Proof : Since f ( x ) ’ s discontinuous points are at most countable , there exists an interior point of the support x∗ at which f ( x ) is continuous . Therefore there exists a cubic area B ( x∗ , ) = { x : |xj − x∗j | ≤ , j = 1 , · · · , p } such that for any x ∈ B ( x∗ , ) , f ( x ) ≥ f ( x∗ ) /2 > 0 and π = P ( x ∈ B ( x∗ , ) ) > 0 . So we have f ( x ) = f ( x ∗ ) 2 ∏ j I ( xj ∈ [ x∗j − , x∗j + ] ) + g ( x ) where g ( x ) is a non- negative function . This can be viewed as a mixture of a uniform distribution on the cubic area with probability f ( x ∗ ) π 2 and a distribution with density g ( x ) 1− f ( x ∗ ) π 2 with probability 1− f ( x ∗ ) π 2 . Notice that the uniform component is mutually independent . The remaining non-negative g ( x ) can be further decomposed into ∑∞ k=2 πkfk , where fk ’ s are the sequence of step functions to approximate the g−1 ( x ) . So now we have f ( x ) = f ( x ∗ ) 2 ∏ j I ( xj ∈ [ x∗j − , x∗j + ] ) + ∑∞ k=2 πkfk and the first term has positive variance , which finish the proof . Remark : The desired Z thus exist as the latent indicator of which component fk , X is from . Where X|Z and Z have the following densities ( X|Z = k ) ∼ fk ( x ) ∫ fk ( x ) µ ( dx ) and P ( Z = k ) = πk ∫ fk ( x ) µ ( dx ) . At least for the uniform component , X|Z is not degenerated . To sum up , the desired latent variable Z generally exists for non degenerate continuous and discrete X .

Summary: This manuscript tackles an important problem, namely, generating the knockoff procedure for FDR-controlled feature selection so that it can work on any data, rather than only for data generated from a Gaussian (as in the original work) or several specific other cases (mixtures of Gaussians or HMMs).  The basic idea is to exploit ideas from variational autoencoders to create a generic knockoff generation mechanism. Specifically, the main idea of the paper is to map the input covariates X into latent variable Z using a variational auto-encoder, generate the knockoffs \tilde{Z} in the latent space, and then map \tilde{Z} back to the input space to get the knockoffs \tilde{X}. The authors claim that their contribution in threefold: