Differentiable Unsupervised Feature Selection based on a Gated Laplacian

1 Introduction . Unsupervised and self-supervised learning studies have been significantly growing interest in the neural network research community . This was prompted by the impressive empirical results that unsupervised learning methods produce in analyzing large amounts of unlabeled data , such as in natural language processing . Many scientific domains , such as biology and physics , have seen the growth of computational and storage resources , alongside advancements in the simultaneous measurement of numerous features , making the analysis of large , high-dimensional datasets a vital research need . ∗Indicates equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In such datasets , discarding irrelevant ( e.g. , noisy , high frequency , or information-poor ) features may reveal clear underlying natural structures that are otherwise hidden in the high dimensional space . We refer to features which do not correspond to the “ main ” data structure as “ nuisance features ” 2 . While nuisance features are mildly harmful in the supervised regime , discarding such features is critical in the unsupervised regime and may determine the success of downstream analysis tasks ( e.g. , clustering or manifold learning ) . Some of the pitfalls caused by nuisance features could be mitigated using an appropriate unsupervised feature selection method . Unsupervised feature selection methods mostly focus on one of two main tasks : either clustering or manifold learning . Among studies that tackle the former task , [ 33 , 12 , 1 ] use autoencoders to identify features that are sufficient for reconstructing the data . Other clustering-dedicated unsupervised feature selection methods assess the relevance of each feature based on different statistical or geometric measures . For example entropy , variance , divergence , and mutual information are used in [ 2 , 31 , 9 , 3 , 30 , 35 ] to identify features that are informative for clustering the data . A popular tool for evaluating features is the graph Laplacian [ 26 , 4 ] . The Laplacian Score ( LS ) [ 13 ] evaluates the importance of each feature by its ability to preserve the local structure . The features that most preserve the manifold structure ( captured by the Laplacian ) are retained . Several studies , such as [ 38 , 27 , 39 ] , extend the LS based on different spectral properties of the Laplacian . While these methods are widely used in the feature selection community , they rely on the success of the Laplacian in capturing the “ main ” structure of the data . We argue that the Laplacian often fails to identify the informative features when computed on the entire dataset ( as is demonstrated in Section 3 ) . This may happen in the presence of a large number of nuisance features : when the variability of the nuisance features masks the variability associated with the structured features . Scenarios resembling this are prevalent in bioinformatics for example , where a large number of biomarkers are measured to characterize developmental and chronological biology processes such as cell differentiation or cell cycle . These processes may depend merely on a few biomarkers . In such situations , it is desirable to have an unsupervised method that can filter nuisance features before the computation of the Laplacian . Related problems , in which data spanning low dimensional subspace is dressed by perturbations in a higher ambient space , were studied in [ 17 , 5 , 10 , 28 ] . In this study , we propose a differentiable objective for unsupervised feature selection . Our proposed method utilizes stochastic input gates , trained to select features with high correlation with the leading eigenvectors of a graph Laplacian computed based on these features . This gating mechanism allows us to re-evaluate the Laplacian for different subsets of features and , thus , unmask the “ main ” data structure buried by the nuisance features . We demonstrate that the proposed approach can significantly improve cluster assignments compared with leading baselines using high-dimensional datasets from multiple domains ( image , text , and biological observations ) . 2 Preliminaries . Consider a data matrix , X ∈ Rn×d , with d dimensional observations x1 , ... , xn . We refer to the columns of X as features f1 , ... , fd , where f i ∈ Rn , and , assume that features are centered and normalized such that 1Tf i = 0 and ‖f i‖22 = 1 . We assume that the data has an inherent structure , determined by a subset of the variables S∗ and that other variables are nuisance features ( i.e. , noisy or information-poor ) . The structured part of the data is considered as either a path-connected manifold , a union of sub-manifolds , or a set of clusters . Our goal is to identify the subset of relevant features S∗ , which preserve the data structure and discard the remaining ones . 2.1 Graph Laplacian . Given n data points , a kernel matrixK is an n× n matrix whoseKi , j entry represents the similarity between xi and xj . A popular choice forK is the Gaussian kernel Ki , j = exp ( −‖xi − xj‖ 2 2σ2b ) , ( 1 ) 2In section 3 , we consider scenarios where low dimensional datasets that form manifold or cluster structures are augmented with information-poor nuisance features that are independent of the original structure and are , themselves , of no particular underlying structure . where σb is a user-defined bandwidth ( chosen , for example , based on the maximum value of the 1- nearest-neighbors of all points ) . The unnormalized graph Laplacian matrix is defined asLun =D−K , whereD is a diagonal matrix , whose elementsDi , i = ∑n j=1Ki , j correspond to the degrees of the points i = 1 , ... , n. The diffusion graph Laplacian is defined as Ldiff =D −1K , ( 2 ) and expresses the transition probabilities of a random walk to move between data points and induces a diffusion distance [ 8 ] . Graph Laplacian matrices are extremely useful in many unsupervised machine learning tasks . In particular , it is known that the eigenvectors corresponding to the small eigenvalues of the unnormalized Laplacian ( or the large eigenvalues of the random walk Laplacian ) are useful for embedding the data in lower dimensions ( see , for example , [ 32 ] ) . 2.2 Laplacian Score . Following the success of the graph Laplacian [ 4 ] , and [ 26 ] , the authors in [ 13 ] have presented an unsupervised measure for feature selection , termed Laplacian Score ( LS ) . The LS evaluates each feature based on its correlation with the leading eigenvectors of the graph Laplacian . At the core of the LS method , the score of feature f is determined by the quadratic form fTLunf , where Lun is the unnormalized graph Laplacian . Since fTLunf = n∑ i=1 λi〈ui , f〉2 , where Lun = ∑n i=1 λiuiu T i is the eigen-decomposition of Lun , the score is smaller when f has a larger projection on the subspace of the leading eigenvectors ( corresponding to the smallest eigenvalues ) of Lun . Such features can be thought of as “ informative ” , as they respect the graph structure . Eigenvalues of the Laplacian can be interpreted as frequencies , and eigenvectors corresponding to larger eigenvalues of Lun ( or smaller eigenvalues of Ldiff ) oscillate faster . Based on the assumption that the interesting underlying structure of the data ( e.g . clusters ) depends on the slowly varying features in the data , [ 13 ] proposed to select the features with the smallest scores . 3 Demonstration of the Importance of Unsupervised Feature Selection in High Dimensional Data with Nuisance Features . By taking a diffusion perspective , we first demonstrate the importance of feature selection to unsupervised learning when the data contains nuisance features . Then , a simple two cluster structure hidden in the ambient space is used to analyze how Gaussian nuisance dimensions affect clustering capabilities . Here , we model the effect of nuisance variables by concatenating variables drawn from a normal Gaussian distribution with the structured part of the data . A concrete example is detailed in the following subsection . 3.1 A Diffusion Perspective . Consider the structured 2-dimensional dataset , known as two-moons , shown in the top-left panel of Fig . 1 . We concatenate the structured data with k “ nuisance ” dimensions , where each such dimension is composed of i.i.d unif ( 0 , 1 ) entries . This means that our new data consists of 2 + k variables . Note that the k Gaussian variables do not carry any information about the two “ moon-shaped ” clusters in this dataset . As one may expect , when the number of nuisance dimensions is large , the amount of unstructured information ( manifested by the nuisance dimensions ) dominates the amount of structured information ( manifested by the two leading dimensions ) . Consequently , attempts to recover the main structure of the data ( say , using manifold learning or clustering ) are likely to fail . From a diffusion perspective , data is considered to be clusterable when the time it takes a random walk starting in one cluster to transition to a point outside the cluster is long . These exit times from different clusters are manifested by the leading eigenvalues of the Laplacian matrix : Ldiff = D−1K , for which the large eigenvalues ( and their corresponding eigenvectors ) are the ones that capture different aspects of the data ’ s structure ( see , for example , [ 24 ] ) . Each added nuisance dimension increases the distance between points which are nearest neighbors in the two dimensional structured subspace . In addition , the noise creates spurious similarities between points , regardless of the cluster they belong to . Overall , this shortens the cluster exit times . This phenomenon is characterized by the second largest eigenvalue , λ2 of Ldiff ( the largest eigenvalue λ1 = 1 carries no information as it corresponds to the constant eigenvector ψ1 ) , which decreases as the number of nuisance dimensions grows , see the top right panel of Fig . 1 . The fact that λ2 decreases implies that the diffusion distances [ 24 ] decrease as well which , in turn , means that the clusters become more connected . A similar view may be obtained by observing that the second smallest eigenvalue of the un-normalized graph Laplacian Lun =D −K , also known as Fiedler number or algebraic connectivity , grows with the number of nuisance features . The fact that the graph becomes less clusterable as more nuisance dimensions are added is also manifested by the eigenvector , ψ2 , corresponding to the second largest eigenvalue of Ldiff ( or the second smallest eigenvalue of Lun ) , which becomes less representative of the cluster structure ( bottom left panel of Fig . 1 ) . Altogether , this means that , in order for the data to be clusterable , the nuisance features ought to be removed . One may argue that principal component analysis can be used to retain the structure of the two-moons while attenuating the effect of the k nuisance variables . However , as shown in the bottom right panel of Fig . 1 , projecting the data onto the first two principal directions does not yield the desired result , since the directions of maximal variance do not capture the two-moons structure , unfortunately . In the next sections , we will describe our differentiable unsupervised feature selection approach , demonstrating that it does succeed to recognize the cluster patterns of the data in this case .

Summary: The paper proposes a feature selection mechanism that aims to preserve the structure of the neighborhood graph of the data. The mechanism is based on the use of stochastic gates that is trained via a regularizer, which balances (i) the goodness of fit of the selected features to the neighborhood graph Laplacian with (ii) the expected value of the number of features preserved. The numerical results show improvements in performance vs. the literature of varying levels and in most tested datasets.