Locally Linear Unsupervised Feature Selection

1 INTRODUCTION . Machine Learning faces statistical and computational challenges due to the increasing dimension of modern datasets . Dimensionality reduction aims at addressing such challenges through embedding the data in a lower dimensionality space , in an unsupervised ( Karlen et al. , 2008 ; Roweis & Saul , 2000 ; Tenenbaum et al. , 2000 ; Wang et al. , 2015 ) or supervised ( Gaudel & Sebag , 2010 ; Fukumizu et al. , 2004 ; Nazarpour & Adibi , 2015 ) way . The requirement for understandable Machine Learning ( Vellido et al. , 2012 ; Doshi-Velez & Kim , 2017 ) however makes it desirable to achieve interpretable dimensionality reduction . In order to do so , the simplest way is to select a subset of the initial features , i.e . to achieve feature selection ( FS ) , as opposed to generating compound new features from the initial ones , a.k.a . feature construction . For instance , determining the genes most important w.r.t . a given disease or the underlying generative model of the data can be viewed as the mother goal in bioinformatics ( Guyon et al. , 2002b ; Libbrecht & Noble , 2015 ) . In the supervised ML setting , features are assessed and selected based on their relevance to the prediction goal ( Guyon & Elisseeff , 2003 ; Sheikhpour et al. , 2017 ; Chen et al. , 2017 ) . Unsupervised learning , aimed at making sense of the data , however constitutes a primary and most important task of ML , as emphasized by LeCun ( 2017 ) , while supervised ML intervenes at a later stage of the data exploitation process . Unsupervised FS approaches ( He et al. , 2005 ; Zhao & Liu , 2007 ; Cai et al. , 2010 ; Li et al. , 2012 ; Zhu et al. , 2017 ) ( more in section 2 ) essentially rely on the assumption that the data samples are structured in clusters , and use the cluster partition in lieu of labels , making it possible to fall down on supervised FS , and select the features most amenable to characterize and separate the clusters . A main limitation of this methodology is that clusters are bound to rely on some metric defined from the initial features ( with the notable exception of Li et al . ( 2012 ) ) , although this metric can be arbitrarily corrupted based on irrelevant or random features . On the other hand , as far as one considers the unsupervised setting , a feature can hardly be considered irrelevant per se . The main contribution of the paper is to address both limitations : the proposed approach , called Locally Linear Unsupervised Feature Selection ( LLUFS ) jointly determines patterns in the data , and features relevant to characterize these patterns . LLUFS is a 2-step process ( Sec . 3 ) : In a first step , a compressed representation of the data is built using Auto-Encoders ( Vincent et al. , 2008 ; Feng et al. , 2014 ) . In a second step , viewing the initial dataset as a high-dimensional embedding of the compressed dataset , each feature is scored according to its contribution to the reconstruction error of the embedding , taking inspiration from Locally Linear Embedding ( Roweis & Saul , 2000 ; Saul & Roweis , 2003 ; Wang , 2012 ) . After describing the goals of experiments and the experimental setting used to validate the approach , extensively relying on the scikit-feature project ( Li et al. , 2017 ; SKf , 2018 ) ( Sec . 4 ) , the empirical validation is presented and discussed ( Sec . 5 ) , establishing the merits and discussing the weaknesses of the approach . The paper concludes with a discussion and some perspectives for further research . NOTATIONS X denotes the m × D data matrix . Row X [ i , : ] , also noted xi when no confusion is to fear , is the i-th sample ( xi in RD ) . Column X [ : , j ] is the j-th feature . For j in [ [ 1 , D ] ] , µj and σj respectively denote the mean and the standard deviation of the j-th feature on the dataset . 1 denote the m-dimensional constant vector [ 1 , ... , 1 ] t. Let S denote an m ×m similarity matrix ( Si , i > 0 ) , with ∆ the associated diagonal degree matrix ( ∆i = ∑m i=1 Si , j ) . The un-normalized ( resp . normalized ) Laplacian associated to S is defined as L = ∆ − S ( resp . L̃ = ∆−12 L∆−12 ) . Two particular similarities will be considered in the following ; the supervised similarity SUP , with SUPi , j = 0 iff xi and xj do not belong to the same class , and 1/|Ci| if they both belong to class Ci , and the unsupervised similarity RBF , with RBFi , j = exp { − 1δ ‖xi − xj‖ 2 2 } , and δ a hyper-parameter of the method . 2 FORMAL BACKGROUND . Supervised FS aims to select a subset of features such that it maximizes the eventual classifier accuracy . Supervised FS algorithms divide into filters , wrappers and embedded methods . Filter methods ( Yu & Liu , 2003 ; Senawi et al. , 2017 ) operate at the data pre-processing stage , and are agnostic to the classifier algorithm . Wrappers methods ( Kabir et al. , 2010 ; Rodrigues et al. , 2014 ) aim to determine the feature subset yielding a best accuracy when used within a specific classifier , through solving a black-box optimization problem . Embedded methods ( Guyon et al. , 2002a ; Zhang et al. , 2015 ) alternatively learn and use the learned hypothesis to prune/select the unpromising/promising features . Admittedly , wrapper and embedded approaches might produce a candidate feature set with moderate generality ( being linked to a particular classifier ) and moderate interpretability ( with the retained features being good as a gang ) . Since this paper focuses on unsupervised and interpretable FS , only filter methods will thus be considered in the following . An early supervised filter method based on the so-called Fisher score was introduced by Duda et al . ( 2000 ) , independently ranking features according to their correlation with the labels.1 A general limitation of such scores is that they achieve a myopic feature selection , with XOR problems − where all relevant features need to be taken into account simultaneously − as typical failure cases . A prominent unsupervised filter approach is based on spectral clustering : data clusters are first built using some metric or similarity ; thereafter supervised FS approaches are used with these clusters in lieu of classes ( Von Luxburg , 2007 ; Binkiewicz et al. , 2017 ) . He et al . ( 2005 ) introduce the Laplacian score1 , where each feature score measures how well this feature accounts for the sample similarity . Interestingly , while the Fisher score is a particular case of Laplacian score using the SUP similarity , the Laplacian score overcomes the myopic limitations of the Fisher score when using the RBF similarity . The Laplacian score is also remotely related to the MaxVariance FS method ( Kantardzic , 2003 ) , selecting features with large variance for the sake of their higher representative power . Also relying on spectral clustering is the SPEC approach ( Zhao & Liu , 2007 ) , proposing three scores respectively noted φ1 , φ2 and φ3 . SPEC relies on the core idea that relevant features be smooth w.r.t . the graph , i.e . slowly varying among samples close to each other . After the spectral clustering theory ( Shi & Malik , 1997 ; Ng , 2001 ) , considering eigenvectors ξ0 , ... , ξm−1 of the normalized Laplacian L̃ ( respectively associated with eigenvalues λ0 < λ1 < ... < λm−1 ) , smooth features are aligned with the first eigenvectors , hence the score φ1 : ∀j ∈ [ [ 1 , D ] ] , φ1j = X̂ [ : , j ] t L̃ X̂ [ : , j ] where X̂ [ : , j ] = ∆ 1 2X [ : , j ] / ∥∥∥∆ 12X [ : , j ] ∥∥∥ ( 1 ) 1For the sake of space limitations , formal definitions are reminded in Appendix 1 . Eigenvectors ξ0 , ... , ξm−1 of L̃ define soft cluster indicators , and eigenvalues λ0 < λ1 < ... < λm−1 measure the separability of the clusters . The smaller φ1j , the more efficient the j-th feature is to separate the clusters . As the first eigenvector ξ0 = ∆ 1 2 1 does not carry any information , with λ0 = 0 , one might rather consider the projection of the feature vector X [ : , j ] on the orthogonal space of ξ0 : ∀j ∈ [ [ 1 , D ] ] , φ2j = 1 1− 〈X̂ [ : , j ] , ξ0〉 X̂ [ : , j ] t L̃ X̂ [ : , j ] ( 2 ) Finally , in the case where the target number of clusters κ is known , only the top-κ eigenvectors are considered , and score φ3 is defined as : ∀j ∈ [ [ 1 , D ] ] , φ3j = κ−1∑ k=1 ( 2− λk ) 〈X̂ [ : , j ] , ξk〉2 ( 3 ) Features are ranked in ascending order for φ1 and φ2 , and in descending order for φ3 . The above three scores measure the overall capacity of a feature to separate clusters , which might prove inefficient in multi-classes/multi clusters settings : a feature most efficient to separate a pair of clusters might have a mediocre general score . The Multi-Cluster Feature Selection ( MCFS ) ( Cai et al. , 2010 ) addresses this limitation by defining a score per cluster . Formally , the capacity of X [ : , j ] to separate clusters is estimated through fitting the eigenvectors ( reminding that ξk is a soft indicator of the k-th cluster ) up to a regularization term . Letting A [ k , : ] denote a vector inRD , with β a regularization weight : ∀k ∈ [ [ 1 , κ ] ] , A [ k , : ] = min B∈RD ‖ξk −XB‖22 + β‖B‖1 ( 4 ) The L1 regularization term enforces the sparsity of A [ k , : ] , retaining only the features most relevant to this cluster . The overall MCFS score simply takes the maximum over all clusters of the absolute value of Ak , j : ∀j ∈ [ [ 1 , D ] ] , MCFSj = max k∈ [ 1 , κ ] |Ak , j | ( 5 ) A general limitation of the above scores is to rely on a similarity metric that can be arbitrarily corrupted by noisy features , potentially leading to irrelevant clusters and scores . This limitation is addressed by Li et al . ( 2012 ) , introducing the Nonnegative Discriminative Feature Selection ( NDFS ) approach . NDFS jointly optimizes the D × D feature importance matrix A together with a cluster indicator matrix ξ , with an L2 regularization : ξ∗ , A∗ = arg min ξ , A Tr ( ξtL̃ ξ ) + α ( ‖ξ −XA‖2F + β ‖A‖ 2 1 ) ( 6 ) subject to ξ orthogonal and semi-positive definite ( ξtξ = ID , ξ ≥ 0 ) , with α , β regularization weights . Following Yu & Shi ( 2003 ) , the first term is rewritten as : Tr ( ξtL̃ξ ) = 1 2 m∑ i , j=1 S [ i , j ] ∥∥∥∥∥ξ [ i , : ] √∆i − ξ [ j , : ] √∆j ∥∥∥∥∥ 2 2 ( 7 ) The minimization of Eq . ( 6 ) tends to enforce the intra-cluster similarity and the inter-cluster dissimilarity . The orthogonality and nonnegativity constraints on ξ further enforce that each sample belong in exactly one cluster .

Summary: The paper proposes the LLUFS method for feature selection. The idea is to first apply a dimensionality reduction method on the input data X to find a low-dimensional representation Z. Next, each point in Z is represented by a linear combination of its nearest neighbors by finding a matrix W which minimizes || Z  - WZ||. Finally, these weights are used to asses the distortion of every feature in X by considering the reconstruction loss in the original space.