Deep unsupervised feature selection

1 INTRODUCTION . Many domains involve high-dimensional observations X ∈ Rd , and it is often desired to select a small number of representative features a priori and observe only this subset . For natural spatiotemporal signals , compressive sensing provides a popular solution to this problem ( Candès et al. , 2006 ; Donoho et al. , 2006 ) . While prior knowledge of the structure of data is not available in the more general setting , a similar approach can be fruitful : unsupervised feature selection should select features that are informative in a general sense , and not just for a specific supervised learning task . As a motivating example , we may be restricted to measuring the expression levels of only a small number of genes , and then use these measurements in a variety of future prediction tasks , such as disease subtype prediction , cell type classification , and so on . More generally , big data problems are impacting virtually all research , and the ability to observe an informative subset of the original features offers important advantages . It yields knowledge about the structure of the data , reduces storage requirements , makes downstream computation more efficient , and alleviates the curse of dimensionality . By contrast with feature extraction methods , such as PCA ( Jolliffe , 2011 ) , feature selection also reduces the burden of measuring all features . The prior work on unsupervised feature selection is diverse , with methods designed to provide good clustering results ( Cai et al. , 2010 ; Dy & Brodley , 2004 ) to preserve the local structure of data ( He et al. , 2006 ; Zhao & Liu , 2007 ; Cai et al. , 2010 ) , and to eliminate redundancy ( Farahat et al. , 2011 ; Mitra et al. , 2002 ) . The abundance of existing approaches reflects a lack of consensus on the correct optimization objective for unsupervised feature selection . We argue that for settings where the selected features will be used in downstream prediction tasks , the optimal approach is to select a set of features that can accurately reconstruct all the remaining features . The contributions of this paper are the following : 1 . We propose the “ restricted autoencoder ” ( RAE ) framework for selecting features based on their reconstruction ability . The approach is justified by novel theoretical results , which show that the reconstruction ability of a set of features bounds its performance in downstream supervised learning tasks . 2 . We propose an approximate optimization algorithm for the RAE objective function , based on learning per-feature corruption rates . The algorithm trains a RAE by iteratively eliminating features that are not important for performing accurate reconstruction . 3 . We apply our approach to two biological datasets , where finding a small number of informative features is an important scientific problem . Experiments demonstrate that RAEs outperform nine baseline methods : they select features that achieve lower reconstruction error and perform better in downstream prediction tasks . 2 RELATED WORK . Early work on unsupervised feature selection was motivated by the observation that PCA ( Pearson , 1901 ; Hotelling , 1933 ) yields principal components ( PCs ) that depend on all input features . In two early studies , Jolliffe ( 1972 ) developed heuristics to discard features while roughly preserving the results of PCA , and Krzanowski ( 1987 ) proposed assessing the quality of a feature subset through a rotation of its PCs . McCabe ( 1984 ) derived several criteria for defining an optimal subset of features , taking inspiration from the numerous criteria that are optimized by PCA . Among the criteria proposed by McCabe ( 1984 ) , one represents the mean squared error ( MSE ) when the rejected features are reconstructed by the selected features using a linear function . The reconstruction loss was revisited in several more recent works , with each addressing the challenge of combinatorial optimization . Farahat et al . derive an efficient algorithm for greedy feature selection ( Farahat et al. , 2011 ) , while Masaeli et al . ( 2010 ) and Zhu et al . ( 2015 ) select features by sparsifying an auto-associative linear model . Several methods optimize a similar objective using per-feature leverage scores ( Boutsidis et al. , 2009a ; b ; 2014 ; Papailiopoulos et al. , 2014 ) . The prior work that considers reconstruction ability has focused primarily on reconstruction with a linear function , although one study proposed using multivariate regression trees ( Questier et al. , 2005 ) , and there have been nascent efforts to perform feature selection with autoencoders ( Chandra & Sharma , 2015 ; Han et al. , 2018 ) . The differences between our work and these studies are i ) we rigorously justify the reconstruction loss through its implications for downstream prediction tasks , ii ) we explain the importance of considering reconstruction with a nonlinear function , and iii ) we propose an algorithm that optimizes the objective function more effectively . Other approaches for unsupervised feature selection find clusters of similar features ( Mitra et al. , 2002 ; Lu et al. , 2007 ) , select non-redundant features with a greedy algorithm based on PCA ( Cui & Dy , 2008 ) , use spectral information to preserve the local structure of data ( He et al. , 2006 ; Zhao & Liu , 2007 ; Zhao et al. , 2010 ) , retain local discriminative information ( Yang et al. , 2011 ) , and attempt to preserve clustering structure ( Dy & Brodley , 2004 ; Boutsidis et al. , 2009a ; Cai et al. , 2010 ) . These approaches are designed with other aims , not to select features for use in supervised learning tasks . In Section 5 , we implement many of these methods and compare them with our approach using multiple datasets and downstream experiments . 3 FEATURE SELECTION BASED ON IMPUTATION ABILITY . 3.1 IMPUTATION LOSS . For a random variable X ∈ Rd , feature selection algorithms determine a set S ⊂ { 1 , 2 , . . . , d } of selected indices , and a setR ≡ { 1 , 2 , . . . , d } \S of rejected indices . We use the notationXS ∈ R|S| and XR ∈ R|R| to denote selected and rejected features , respectively . For notational convenience we assume that all random variables have zero mean . The goal of unsupervised feature selection is to select features XS that are most representative of the full observation vector X . An approach that has received some interest in prior work is to measure how well XS can reconstruct XR ( McCabe , 1984 ; Questier et al. , 2005 ; Farahat et al. , 2011 ; Masaeli et al. , 2010 ; Papailiopoulos et al. , 2014 ; Zhu et al. , 2015 ; Han et al. , 2018 ) . It is intuitive to consider reconstruction ability , because if the rejected features can be reconstructed perfectly , then no information is lost when selecting a subset of features . To make the motivation for this approach precise , we derive a rigorous justification that has not been presented in prior work . To quantify reconstruction ability , we define the imputation loss L ( S ) and the linear imputation loss Llinear ( S ) . Both quantify how much information XS contains about XR , and it is clear from their definitions that L ( S ) ≤ Llinear ( S ) . Definition 1 ( Imputation loss ) . The imputation loss L ( S ) and the linear imputation loss Llinear ( S ) quantify how well XS can reconstruct XR using an unrestricted function , and using a linear function , respectively . They are defined as : L ( S ) = min h E [ ||XR − h ( XS ) ||2 ] ( 1 ) Llinear ( S ) = min W E [ ||XR −WXS ||2 ] ( 2 ) Either one could serve as a feature selection criterion , so we aim to address the following questions : 1 . Why is it desirable to optimize for the reconstruction ability of selected features XS ? 2 . Is it preferable to measure reconstruction ability using L ( S ) or Llinear ( S ) ? 3.2 IMPLICATIONS OF IMPUTATION LOSS FOR DOWNSTREAM TASKS . In this section , we demonstrate through theoretical results that reconstruction ability is connected to performance in downstream supervised learning tasks . These results aim to characterize the usefulness of XS for predicting a target variable Y ∈ R. All proofs are in Appendix C. To facilitate this analysis , we assume that all learned models are optimal ; this assumption ignores practical problems such as overfitting and non-convex optimization , but makes it possible to shed light on the connection between reconstruction ability and predictive power . Our analysis focuses on the degradation in performance ( henceforth called performance loss ) when a model is fitted to XS instead of X . Intuitively , performance in the downstream task must suffer when information is discarded . The question is , by how much ? Consider a fixed partitioning of features into ( XS , XR ) . The first result addresses the situation when a linear model is used to predict Y given XS . Theorem 1 states that the performance loss has an exact expression , which can be related to Llinear ( S ) . For this result we define the notation Σs = Cov ( X S ) , Σr = Cov ( XR ) , Σsr = Cov ( XS , XR ) , and Σr|s = Σr − ΣsrΣ−1s Σsr . Theorem 1 ( Performance loss with linear model ) . Assume a prediction target Y such that ( b∗ , c∗ ) = arg min b , c E [ ( Y − bTXS − cTXR ) 2 ] . ( 3 ) Then , the performance loss for features XS is : min u E [ ( Y − uTXS ) 2 ] −min v E [ ( Y − vTX ) 2 ] = cT∗ Σr|sc∗ . ( 4 ) The performance loss is a quadratic form based on the matrix Σr|s , so it is dependent on the eigenstructure of the matrix . To ensure strong performance in downstream tasks , it is therefore desirable to select XS so that Σr|s has small eigenvalues . The result is notable because it can be shown that the linear imputation loss is equal to the sum of the eigenvalues , i.e. , Llinear ( S ) = Tr ( Σr|s ) . Selecting XS to minimize Llinear ( S ) therefore minimizes the performance loss with linear models . This perspective has not been presented in prior work , but lends support to several existing approaches ( Masaeli et al. , 2010 ; Farahat et al. , 2011 ; Papailiopoulos et al. , 2014 ; Zhu et al. , 2015 ) . The second result addresses the situation where a nonlinear model is used to predict Y . Nonlinear models ( e.g. , neural networks , gradient boosting machines ) are predominant in many machine learning applications , so this result is more relevant to contemporary practices . Theorem 2 states that the performance loss can be upper bounded using L ( S ) . The result requires a mild assumption about the Hölder continuity ( a generalization of Lipschitz contintuity ) of the conditional expectation function E [ Y | X = x ] that is not verifiable in practice . Nonetheless , it connects the performance of XS in prediction tasks with its ability to impute XR . Theorem 2 ( Performance loss with nonlinear model ) . Assume a prediction target Y such that the conditional expectation E [ Y | X = x ] is ( C , α ) -Hölder continuous with exponent 0 < α ≤ 1 , so that the following holds almost everywhere in the distribution of X : ∣∣ E [ Y | X = a ] − E [ Y | X = b ] ∣∣ ≤ C · ||a− b||α2 . ( 5 ) Then , the performance loss for features XS can be upper bounded : min f1 E [ ( Y − f1 ( XS ) ) 2 ] −min f2 E [ ( Y − f2 ( X ) ) 2 ] ≤ C2 · L ( S ) α . ( 6 ) The bound in Theorem 2 suggests that XS should be selected to minimize L ( S ) , because doing so reduces the upper bound on the performance loss . We appeal to the minimax principle to argue that the minimization of L ( S ) , by effectively minimizing the worst-case performance loss , provides the most conservative way to select a small number of features ( Von Neumann & Morgenstern , 1944 ) . While this result is easiest to prove for Hölder continuous tasks with the MSE loss function , the idea that good reconstruction ability leads to strong predictive power should apply more broadly . Our experiments in Section 5 confirm this empirically . To make these results concrete , we present a simulation experiment involving two feature sets from the single-cell RNA sequencing data described in Section 5 . The feature sets XSnonlinear and XSlinear were chosen so that L ( XSnonlinear ) ≤ L ( XSlinear ) , but Llinear ( XSnonlinear ) ≥ Llinear ( XSlinear ) . We simulated 250 response variables with linear dependencies on subsets of the original features ( see Section 5.4 ) , and trained linear and nonlinear ( neural network ) predictive models for each task . Figure 1 shows the relative performance of the two feature sets . The important observations from this experiment are that across a wide variety of tasks , nonlinear models perform better overall ( Figure 1A ) , XSnonlinear usually performs better when using nonlinear models ( Figure 1B ) , and XSlinear usually performs better when using linear models ( Figure 1C ) . These results provide empirical confirmation of both Theorem 1 and Theorem 2 , and summarize our approach to feature selection . To our knowledge , these results are novel in the space of unsupervised feature selection , despite the long history of the linear imputation loss ( McCabe , 1984 ) and recent interest in the nonlinear imputation loss ( Questier et al. , 2005 ; Han et al. , 2018 ) . While it is tangential to unsupervised feature selection , we also show in Appendix A that the results can be stated more generally for unsupervised feature extraction . Given these findings , and the predominance of nonlinear models in contemporary machine learning , we proceed with an approach to select XS by minimizing the imputation loss as follows : S∗ = arg min |S|=k { min f E [ ||XR − f ( XS ) ||2 ] } ( 7 )

Summary: Authors of this paper propose the restricted autoencoder (RAE) framework for selecting features that can accurately reconstruct the rest of features. Authors justify the proposed method via the proof that the reconstruction ability of a set of features bounds its performance in downstream supervised learning tasks. The algorithm that iteratively eliminates features using learned per-feature corruption rates is proposed.