Knockoff-Inspired Feature Selection via Generative Models

1 INTRODUCTION . The availability of digital sensors and low-priced storage , computation , and communication channels is becoming increasingly pervasive . As a result , emerging applications involving extremely large datasets are now feasible . Thus , we now often rely on data-centric inference , in which large amounts of data are concentrated and processed via machine learning ( ML ) algorithms , based on suitable data models , to extract relevant information from the observations . However , such approaches often suffer from high computational complexity as well as limits on available storage , communication , and processing resources . To alleviate such issues , dimensionality reduction ( Samet , 2006 ) is commonly applied at the beginning of the data processing pipeline . Dimensionality reduction also addresses multiple issues with the performance of learning and inference from high-dimensional data that are commonly referred to as the curse of dimensionality or Hughes Phenomenon ( Jain & Zongker , 1997 ; Pavlenko , 2003 ; Guyon & Elisseeff , 2003 ) . In the most demanding applications involving real-time data processing or complex sensing architectures ( e.g. , low-power sensing , streaming data analytics , particle physics , and genetics ) , it may not be feasible to perform the sensing and/or computation required by most existing approaches to dimensionality reduction during data acquisition . Thus , the practitioner is limited in these cases to feature selection ( FS ) ( Guyon & Elisseeff , 2003 ; Liu & Motoda , 2008 ) , i.e. , to selecting the subset of the data dimensions that are most relevant in information extraction . These dimensions are the only ones to be sensed to provide a simple form of dimensionality reduction . FS has a strong connection with the variable selection ( VS ) problem in regression analysis for statistics ( Hastie et al. , 2009 ) , where the goal is to find a subset of the predictive variables that capture statistical dependence to the response variable of interest . Informally , the goal of a VS algorithm is to identify as many of the predictive variables on which the response variable is truly dependent ( measured by its predictive power ) while minimizing the number of selected predictive variables that are actually unrelated ( measured by the false discovery rate ) ; such a strong distinction between features is seldom observed in ML , where the focus of FS is to preserve performance while maximizing dimensionality reduction . The recent development of VS via knockoffs ( Barber & Candès , 2015 ; 2019 ; Weinstein et al. , 2017 ; Barber et al. , 2019 ) has attracted significant attention in the statistics community due to its guaranteed control of the false discovery rate , as well as its applicability to arbitrary statistical models for the variables . This new framework relies on the creation of samples of knockoff variables that provide counterparts to the original variables on existing training samples . The knockoff variable design must meet certain requirements : first , knockoff variables must follow the same probability distribution as the original variables , but they must be independent from the labels to be predicted , conditioned on the original variables ; second , they should be pairwise uncorrelated with the original variables in order to provide sufficient predictive power ; finally , swapping any two corresponding entries of the original and knockoff sample vectors must not change the likelihoods of these vectors . The framework then relies on numerical scores to compare the statistical dependence of original and knockoff variables with the response variable , discarding those variables for which the scores of these two variable versions ( original and knockoff ) are indistinguishable . While the original framework focused on jointly Gaussian variable models and Lasso-based linear regression , it has since been extended by new knockoff variable constructions for more sophisticated data models , such as Markov models and Bayesian networks ( Sesia et al. , 2019 ; Gimenez et al. , 2018 ; Romano et al. , 2019 ; Fan et al. , 2019 ; Liu & Zheng , 2018 ; Jordon et al. , 2019 ) , as well as proposing additional variable statistics for other regression schemes ( Lu et al. , 2018 ) ; furthermore , technical conditions on the data models and variable statistics have been simplified while preserving the original performance guarantees ( Candès et al. , 2018 ) . In this paper , we propose a novel framework for FS in ML inspired by the knockoff framework for VS that is geared towards applications featuring large-scale , high-dimensional datasets . We leverage the flexibility of the framework by designing knockoff feature constructions and scores based on data models that are popular in ML and driven by large-scale datasets . More specifically , we leverage generative models that provide a data distribution characterization that is compatible with the knockoff VS framework . Examples of knockoffs generated according to methods from the literature and from our proposed method are illustrated in Figure 1 for multiple samples from several benchmark image datasets , where it is clearly shown that the properties required for knockoff variables are better met by our constructions than by those previously proposed . It is worth remarking that since the analytical framework for VS performance evaluation does not translate to FS , the proposed knockoff generation methods and scores are best described as heuristics . This paper ’ s contributions can be summarized as follows . First , we propose a generic algorithm for knockoff variable design that assumes the existence of a generative parameterized data model composed of two mapping functions : one from the data feature space to the space of probability distributions for a low-dimensional parameter , and another from the parameter space to the space of probability distributions for the data . Second , we present example implementations of this algorithm using two standard data-driven models in ML : variational autoencoders and Gaussian process latent variable models . Third , we propose a knockoff-based scoring scheme that is designed for FS for softmax classification . Fourth , we provide numerical examples that compare the performance of the proposed FS schemes against multiple baselines , including approaches based on previously proposed knockoff-based VS designs . Finally , we discuss open questions and directions for future work . 2 BACKGROUND . Feature Selection ( FS ) ( Guyon & Elisseeff , 2003 ; Liu & Motoda , 2008 ) aims to identify a subset Ω of the indices { 1 , . . . , N } for the entries of the original feature vector x = [ x1 , . . . , xN ] T ∈ RN so that the resulting vector xΩ ∈ R|Ω| best preserves the dataset structure relevant for the ML problem at hand . Three classes of FS approaches have been described in the literature ( Guyon & Elisseeff , 2003 ; Li et al. , 2017 ) : filter methods , wrapper methods , and embedded methods . Of particular interest to this paper are filter methods , which consider a set of J labeled data examples { xj , yj } Jj=1 , and compute a score for each dimension S ( n ) based on the values xj , n and yj for each data point . The features are then ranked by this score and those with highest scores compose the set Ω . Common scores include Fisher ’ s prediction score , the data-to-label correlation , the mutual information I ( xj , n ; yj ) , etc . Variable Selection ( VS ) in statistics is similar to FS ( Hastie et al. , 2009 ) . Consider a set of predictive variables { x1 , . . . , xN } ( e.g. , the entries of the vector x ) , and a response variable y ( e.g „ the label or parameter to be estimated ) , and assume that the data points and matching labels correspond to independent and identically distributed ( i.i.d . ) samples of an underlying joint probability distribution p ( x , y ) . The goal of VS is to determine the set of true predictors { xj } j∈Ω so that the response is statistically independent of the remaining predictors conditioned on the chosen predictors , i.e. , y ⊥ xΩC |xΩ ; the variables xΩC are often referred to as null variables . The selection mechanism should have high power ( e.g. , high probability of the true predictors Ω being selected in a set estimate Ω̂ ) while minimizing the false discovery rate ( FDR ) ( e.g. , the expected fraction E [ |Ω̂ \Ω|/|Ω| ] of null variables included within the selected variable set ) . Standard methods , e.g. , ( Benjamini & Hochberg , 1995 ) , require knowledge of the conditional distribution p ( x|y ) . A recent framework for VS based on the construction of knockoff variables ( Barber & Candès , 2015 ; 2019 ; Weinstein et al. , 2017 ; Barber et al. , 2019 ; Candès et al. , 2018 ) has been shown to control the FDR while being applicable to a wide variety of cases without requiring any knowledge of the conditional response distribution p ( y|x ) . This generality of the approach is given by its single prerequisite knowledge of the distribution of the data prior p ( x ) . The knockoff variables x̃ = { x̃1 , . . . , x̃N } are constructed for each data sample using this prior so that any swap of a variable with its knockoff does not affect the data distribution . More formally , denote by ( x , x̃ ) swap ( Γ ) the pair of vectors ( x , x̃ ) where the entries of each of the vectors corresponding to the indices Γ has been swapped with the other . The knockoff variables x̃ are designed so that ( i ) ( x , x̃ ) swap ( Γ ) ≡ ( x , x̃ ) , where ≡ denotes equality in distribution ( and in practice , over the sample distribution ) ; and ( ii ) x̃ is independent of y given x ( which is trivially met by ignoring y during the design of the knockoffs ) . After knockoff samples are created for the VS training dataset , the framework relies on the computation of variable statistics vn = vn ( ( X , X̃ ) , Y ) for each predictor variable that may depend on all the original variables , all the knockoff variables , and the label ; here X , X̃ and Y denote the set of samples of the original and knockoff variables and the corresponding labels , respectively . The variable statistics generator must have a flip-sign property : the sign of the variable must change when the corresponding entries of the vectors are swapped , i.e. , vn ( ( X , X̃ ) swap ( n ) , Y ) = −vn ( ( X , X̃ ) , Y ) for each n. Flip-sign variable statistics can be easily obtained by creating variable importance scores ( s , s̃ ) = ( { s1 , . . . , sN } , { s̃1 , . . . , s̃N } ) = t ( ( X , X̃ ) , Y ) , measuring the impor- tance of the corresponding variables for the set of sample labels Y , that obey the swap property t ( ( X , X̃ ) swap ( n ) , Y ) = ( s , s̃ ) swap ( n ) for each n ; we can then define vn = sn − s̃n . VS proceeds by finding a magnitude threshold τ over the values of v = [ v1 , . . . vN ] that yields a sufficiently small ratio q̂ = | { n : vn < −τ } |/| { n : vn > τ } | ( e.g. , the ratio between the number of indices with negative evidence and the number of indices with positive evidence ) , since q̂ gives a proxy for the target FDR q ( Barber & Candès , 2019 ; Candès et al. , 2018 ) . The promising recent development of model-X knockoffs ( Candès et al. , 2018 ) simplifies the original framework above by showing that knockoffs only require distribution invariance to swapping on a single variable basis . In other words , knockoffs can be constructed simply by sampling each knockoff variable according to the conditional likelihood of the original variable given the rest of the original variables and the previously obtained knockoff variables , p ( xn|x̃1 : n−1 , x−n ) ; therefore , each knockoff variable can be generated independently in a particular sequence . The resulting simplicity of the model-X knockoff framework provides an exciting opportunity for the use of modern data models that aim to learn the distribution of the data x in novel schemes for knockoff generation . An autoencoder is a neural network that aims to obtain a low-dimensional data representation by ( i ) setting both the input and output of the network to be the data x itself , so that the network learns to recreate the data based on any of the hidden layer representations ; and ( ii ) including a specific hidden layer of low dimensionality whose output activation z provides a compact representation from which the data can be recovered . The neural network is thus naturally divided into encoder and decoder portions . Autoencoders have been used in ML to provide compact data representations z that are then fed into modern ML algorithms . However , the mapping obtained by the neural network is deterministic , and does not provide information on the distribution of the data p ( x ) . Variational autoencoders ( VAEs ) inherit this name because they follow the structure mentioned above . However , the low-dimensional representation z for the data follows a prescribed probability distribution p ( z ) . Consequently , the encoder and decoder aim to learn the forward and backward data distributions p ( z|x ) and p ( x|z ) ( Doersch , 2016 ) . More specifically , we assume that there exists a data generating function x = f ( z ; θ ) : Z ×Θ→ X that provides a map between a parameter value z and the expected signal x conditioned on z , where θ denotes the various model parameters . Practical VAEs leverage the common Gaussian assumption for the conditional data and parameter distributions . Under such a probabilistic model , the VAE network aims to learn two separate functions in its two halves : the encoder learns an independent-entry multivariate Gaussian approximationQ ( z|x , θ ) = N ( z|µ̂z|x , θ , Σ̂z|x , θ ) for the probability distribution P ( z|x , θ ) . Thus , the low-dimensional representation learned by the encoder in its narrowest layer corresponds to the approximated mean µz|x ≈ E [ z|x , θ ] and the diagonal of the variance matrix Σ̂z|x , θ ≈ Var ( z|x , θ ) . The decoder learns the mapping x = f ( z ; θ ) = E [ x|z , θ ] ( Doersch , 2016 ) . Nonlinear manifold models also aim at learning mappings from the high-dimensional space containing the data of interest to a low-dimensional representation space , which is usually described in terms of providing a parametrization of the data . As with autoencoders , the large majority of approaches are deterministic , as they aim at preserving the local geometry of the data in the high dimensional space . Furthermore , almost all existing methods for manifold learning do not provide a bidirectional mapping so that new data samples can be obtained from arbitrary representations , lacking a generative model to obtain data from arbitrary points in the parameter space . In contrast , Gaussian process latent variable models ( GPLVMs ) ( Lawrence , 2003 ) have been proposed to provide a probabilistic nonlinear mapping between the parameter space and the data space , inspired by probabilistic principal component analysis and kernel methods , and scalable to large-scale datasets ( Hensman et al. , 2013 ) . As in principal component analysis , it is first assumed that each data sample x can be written as x = Wz + e , where W is the basis for a subspace approximation of the data , z is a low-dimensional parametrization vector , and e is the model error . It is further assumed that z ∼ N ( 0 , I ) , that e ∼ N ( 0 , βI ) , and that each row of W is an i.i.d . zero-mean Gaussian vector with covariance αI . Given a data set X = [ x1 , x2 , . . . ] , the corresponding parameter vectors Z can be obtained via maximum likelihood estimation with the conditional distribution that marginalizes W : p ( X|Z , θ ) = 1 ( 2π ) DN 2 |K|D2 exp ( −1 2 trace ( K−1XTX ) ) , Algorithm 1 Knockoff Construction from Arbitrary Probabilistic Generative Model ( KnCAPGM ) Input : probability distributions p ( x|z , θ ) , p ( z|x , θ ) , input features x Output : output knockoff features x̃ 1 : initialize knockoff x̃ = x 2 : for n = 1 , 2 , . . . , N do 3 : marginalize xn from the distribution p ( z|x , θ ) to obtain p ( z|x−n , θ ) 4 : obtain z∗ as a sample from p ( z|x̃−n , θ ) 5 : obtain x∗ as a sample from p ( x|z∗ , θ ) 6 : update knockoff vector x̃ : x̃n = x∗n 7 : end for 8 : return knockoff x̃ where K = αZTZ + β−1I . In order to introduce nonlinearity in the map between z and x , the matrix K is instead rebuilt using a kernel space representation ( e.g. , with a radial basis function kernel ) : Ki , j = κ ( zi , zj ) = α exp ( −γ2 ‖zi − zj‖ 2 2 ) + β−1δi , j , where i , j are indices of the training dataset X. GPLVM training obtains the parameters θ = { α , β , γ } and Z via an iterative method that alternatively uses maximum likelihood for θ and maximum a posteriori estimation for Z with the conditional likelihood of the data X given above , respectively ( Lawrence , 2003 ; 2005 ) .

Summary: This paper proposes a way of employing modern generative models like VAEs and GPLVMs within the recently popular “knockoff” framework for variable/feature selection. Roughly speaking, the main idea of knockoffs is to construct a duplicate (“knockoff”) of each regression variable which matches its distributional properties but is independent of the response variable when conditioning on the true input variable the knockoff is based on. Under certain assumptions, fitting a model on top of both original and knocked-off features and observing the difference between the fitted coefficients for the true and the knockoff features can then be used for variable selection with guaranteed false discovery rate. The authors of this paper suggest that many of the current methods for construction of knockoffs may not be appropriate for image, text, and other types of datasets commonly used in modern ML, and suggest a heuristic way of employing VAEs and GPLVMs for this purpose. The paper concludes with an empirical study which shows that their algorithm is competitive with existing feature selections methods in terms of post-selection accuracy of the fitted model.