Differentiable Unsupervised Feature Selection based on a Gated Laplacian

1 Introduction . Unsupervised and self-supervised learning studies have been significantly growing interest in the neural network research community . This was prompted by the impressive empirical results that unsupervised learning methods produce in analyzing large amounts of unlabeled data , such as in natural language processing . Many scientific domains , such as biology and physics , have seen the growth of computational and storage resources , alongside advancements in the simultaneous measurement of numerous features , making the analysis of large , high-dimensional datasets a vital research need . ∗Indicates equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In such datasets , discarding irrelevant ( e.g. , noisy , high frequency , or information-poor ) features may reveal clear underlying natural structures that are otherwise hidden in the high dimensional space . We refer to features which do not correspond to the “ main ” data structure as “ nuisance features ” 2 . While nuisance features are mildly harmful in the supervised regime , discarding such features is critical in the unsupervised regime and may determine the success of downstream analysis tasks ( e.g. , clustering or manifold learning ) . Some of the pitfalls caused by nuisance features could be mitigated using an appropriate unsupervised feature selection method . Unsupervised feature selection methods mostly focus on one of two main tasks : either clustering or manifold learning . Among studies that tackle the former task , [ 33 , 12 , 1 ] use autoencoders to identify features that are sufficient for reconstructing the data . Other clustering-dedicated unsupervised feature selection methods assess the relevance of each feature based on different statistical or geometric measures . For example entropy , variance , divergence , and mutual information are used in [ 2 , 31 , 9 , 3 , 30 , 35 ] to identify features that are informative for clustering the data . A popular tool for evaluating features is the graph Laplacian [ 26 , 4 ] . The Laplacian Score ( LS ) [ 13 ] evaluates the importance of each feature by its ability to preserve the local structure . The features that most preserve the manifold structure ( captured by the Laplacian ) are retained . Several studies , such as [ 38 , 27 , 39 ] , extend the LS based on different spectral properties of the Laplacian . While these methods are widely used in the feature selection community , they rely on the success of the Laplacian in capturing the “ main ” structure of the data . We argue that the Laplacian often fails to identify the informative features when computed on the entire dataset ( as is demonstrated in Section 3 ) . This may happen in the presence of a large number of nuisance features : when the variability of the nuisance features masks the variability associated with the structured features . Scenarios resembling this are prevalent in bioinformatics for example , where a large number of biomarkers are measured to characterize developmental and chronological biology processes such as cell differentiation or cell cycle . These processes may depend merely on a few biomarkers . In such situations , it is desirable to have an unsupervised method that can filter nuisance features before the computation of the Laplacian . Related problems , in which data spanning low dimensional subspace is dressed by perturbations in a higher ambient space , were studied in [ 17 , 5 , 10 , 28 ] . In this study , we propose a differentiable objective for unsupervised feature selection . Our proposed method utilizes stochastic input gates , trained to select features with high correlation with the leading eigenvectors of a graph Laplacian computed based on these features . This gating mechanism allows us to re-evaluate the Laplacian for different subsets of features and , thus , unmask the “ main ” data structure buried by the nuisance features . We demonstrate that the proposed approach can significantly improve cluster assignments compared with leading baselines using high-dimensional datasets from multiple domains ( image , text , and biological observations ) . 2 Preliminaries . Consider a data matrix , X ∈ Rn×d , with d dimensional observations x1 , ... , xn . We refer to the columns of X as features f1 , ... , fd , where f i ∈ Rn , and , assume that features are centered and normalized such that 1Tf i = 0 and ‖f i‖22 = 1 . We assume that the data has an inherent structure , determined by a subset of the variables S∗ and that other variables are nuisance features ( i.e. , noisy or information-poor ) . The structured part of the data is considered as either a path-connected manifold , a union of sub-manifolds , or a set of clusters . Our goal is to identify the subset of relevant features S∗ , which preserve the data structure and discard the remaining ones . 2.1 Graph Laplacian . Given n data points , a kernel matrixK is an n× n matrix whoseKi , j entry represents the similarity between xi and xj . A popular choice forK is the Gaussian kernel Ki , j = exp ( −‖xi − xj‖ 2 2σ2b ) , ( 1 ) 2In section 3 , we consider scenarios where low dimensional datasets that form manifold or cluster structures are augmented with information-poor nuisance features that are independent of the original structure and are , themselves , of no particular underlying structure . where σb is a user-defined bandwidth ( chosen , for example , based on the maximum value of the 1- nearest-neighbors of all points ) . The unnormalized graph Laplacian matrix is defined asLun =D−K , whereD is a diagonal matrix , whose elementsDi , i = ∑n j=1Ki , j correspond to the degrees of the points i = 1 , ... , n. The diffusion graph Laplacian is defined as Ldiff =D −1K , ( 2 ) and expresses the transition probabilities of a random walk to move between data points and induces a diffusion distance [ 8 ] . Graph Laplacian matrices are extremely useful in many unsupervised machine learning tasks . In particular , it is known that the eigenvectors corresponding to the small eigenvalues of the unnormalized Laplacian ( or the large eigenvalues of the random walk Laplacian ) are useful for embedding the data in lower dimensions ( see , for example , [ 32 ] ) . 2.2 Laplacian Score . Following the success of the graph Laplacian [ 4 ] , and [ 26 ] , the authors in [ 13 ] have presented an unsupervised measure for feature selection , termed Laplacian Score ( LS ) . The LS evaluates each feature based on its correlation with the leading eigenvectors of the graph Laplacian . At the core of the LS method , the score of feature f is determined by the quadratic form fTLunf , where Lun is the unnormalized graph Laplacian . Since fTLunf = n∑ i=1 λi〈ui , f〉2 , where Lun = ∑n i=1 λiuiu T i is the eigen-decomposition of Lun , the score is smaller when f has a larger projection on the subspace of the leading eigenvectors ( corresponding to the smallest eigenvalues ) of Lun . Such features can be thought of as “ informative ” , as they respect the graph structure . Eigenvalues of the Laplacian can be interpreted as frequencies , and eigenvectors corresponding to larger eigenvalues of Lun ( or smaller eigenvalues of Ldiff ) oscillate faster . Based on the assumption that the interesting underlying structure of the data ( e.g . clusters ) depends on the slowly varying features in the data , [ 13 ] proposed to select the features with the smallest scores . 3 Demonstration of the Importance of Unsupervised Feature Selection in High Dimensional Data with Nuisance Features . By taking a diffusion perspective , we first demonstrate the importance of feature selection to unsupervised learning when the data contains nuisance features . Then , a simple two cluster structure hidden in the ambient space is used to analyze how Gaussian nuisance dimensions affect clustering capabilities . Here , we model the effect of nuisance variables by concatenating variables drawn from a normal Gaussian distribution with the structured part of the data . A concrete example is detailed in the following subsection . 3.1 A Diffusion Perspective . Consider the structured 2-dimensional dataset , known as two-moons , shown in the top-left panel of Fig . 1 . We concatenate the structured data with k “ nuisance ” dimensions , where each such dimension is composed of i.i.d unif ( 0 , 1 ) entries . This means that our new data consists of 2 + k variables . Note that the k Gaussian variables do not carry any information about the two “ moon-shaped ” clusters in this dataset . As one may expect , when the number of nuisance dimensions is large , the amount of unstructured information ( manifested by the nuisance dimensions ) dominates the amount of structured information ( manifested by the two leading dimensions ) . Consequently , attempts to recover the main structure of the data ( say , using manifold learning or clustering ) are likely to fail . From a diffusion perspective , data is considered to be clusterable when the time it takes a random walk starting in one cluster to transition to a point outside the cluster is long . These exit times from different clusters are manifested by the leading eigenvalues of the Laplacian matrix : Ldiff = D−1K , for which the large eigenvalues ( and their corresponding eigenvectors ) are the ones that capture different aspects of the data ’ s structure ( see , for example , [ 24 ] ) . Each added nuisance dimension increases the distance between points which are nearest neighbors in the two dimensional structured subspace . In addition , the noise creates spurious similarities between points , regardless of the cluster they belong to . Overall , this shortens the cluster exit times . This phenomenon is characterized by the second largest eigenvalue , λ2 of Ldiff ( the largest eigenvalue λ1 = 1 carries no information as it corresponds to the constant eigenvector ψ1 ) , which decreases as the number of nuisance dimensions grows , see the top right panel of Fig . 1 . The fact that λ2 decreases implies that the diffusion distances [ 24 ] decrease as well which , in turn , means that the clusters become more connected . A similar view may be obtained by observing that the second smallest eigenvalue of the un-normalized graph Laplacian Lun =D −K , also known as Fiedler number or algebraic connectivity , grows with the number of nuisance features . The fact that the graph becomes less clusterable as more nuisance dimensions are added is also manifested by the eigenvector , ψ2 , corresponding to the second largest eigenvalue of Ldiff ( or the second smallest eigenvalue of Lun ) , which becomes less representative of the cluster structure ( bottom left panel of Fig . 1 ) . Altogether , this means that , in order for the data to be clusterable , the nuisance features ought to be removed . One may argue that principal component analysis can be used to retain the structure of the two-moons while attenuating the effect of the k nuisance variables . However , as shown in the bottom right panel of Fig . 1 , projecting the data onto the first two principal directions does not yield the desired result , since the directions of maximal variance do not capture the two-moons structure , unfortunately . In the next sections , we will describe our differentiable unsupervised feature selection approach , demonstrating that it does succeed to recognize the cluster patterns of the data in this case .

Summary: This paper proposes a novel unsupervised feature selection method to efficiently remove nuisance features. The model combines a graph Laplacian score with stochastic gates, which adds a re-evaluation of the graph Laplacian to improve the performance in downstream applications. Experiment results are provided to validate the theoretical claims.