Neural Manifold Clustering and Embedding

Given a union of non-linear manifolds , non-linear subspace clustering or manifold clustering aims to cluster data points based on manifold structures and also learn to parameterize each manifold as a linear subspace in a feature space . Deep neural networks have the potential to achieve this goal under highly non-linear settings given their large capacity and flexibility . We argue that achieving manifold clustering with neural networks requires two essential ingredients : a domain-specific constraint that ensures the identification of the manifolds , and a learning algorithm for embedding each manifold to a linear subspace in the feature space . This work shows that many constraints can be implemented by data augmentation . For subspace feature learning , Maximum Coding Rate Reduction ( MCR2 ) objective can be used . Putting them together yields Neural Manifold Clustering and Embedding ( NMCE ) , a novel method for general purpose manifold clustering , which significantly outperforms autoencoder-based deep subspace clustering and achieve state-of-the-art performance on several important benchmarks . Further , on more challenging natural image datasets , NMCE can also outperform other algorithms specifically designed for clustering . Qualitatively , we demonstrate that NMCE learns a meaningful and interpretable feature space . As the formulation of NMCE is closely related to several important Self-supervised learning ( SSL ) methods , we believe this work can help us build a deeper understanding on SSL representation learning . 1 INTRODUCTION . Here we investigate unsupervised representation learning , which aims to learn structures ( features ) from data without the use of any label . If the data lie in a linear subspace , the linear subspace can be extracted by Principal Component Analysis ( PCA ) ( Jolliffe , 1986 ) , one of the most basic forms of unsupervised learning . When the data occupy a union of several linear subspaces , subspace clustering ( SC ) ( Vidal et al. , 2016 ) is needed to cluster each data point to a subspace as well as estimating the parameters of each subspace . Here we are concerned with even more challenging scenarios , when data points come from a union of several non-linear low-dimensional manifolds . In such scenarios , the clustering problem can be formulated as follows ( Elhamifar & Vidal , 2011 ) : Task 1 . Manifold Clustering and Embedding : Given that the data points come from a union of lowdimensional manifolds , we shall segment the data points based on their corresponding manifolds , and obtain a low-dimensional embedding for each manifold . Various methods have been developed to solve this problem ( Abdolali & Gillis , 2021 ) , but it is still an open question how to use neural networks effectively in manifold clustering problems ( Haeffele et al. , 2020 ) . In this paper , we propose neural manifold clustering and embedding ( NMCE ) that follows three principles : 1 ) The clustering and representation should respect a domain-specific constraint , e.g . local neighborhoods , local linear interpolation or data augmentation invariances . 2 ) The embedding of a particular manifold shall not collapse . 3 ) The embedding of identified manifolds shall be linearized and separated , i.e . they occupy different linear subspaces . We achieve 1 ) using data augmentations , and achieve 2 ) and 3 ) with the subspace feature learning algorithm maximal coding rate reduction ( MCR2 ) objective ( Yu et al. , 2020 ) . This work make the following specific contributions : 1 . We combine data augmentation with MCR2 to yield a novel algorithm for general purpose manifold clustering and embedding ( NMCE ) . We also discuss connections between the algorithm and self-supervised contrastive learning . 2 . We demonstrate that NMCE achieves state-of-the-art performance on standard subspace clustering benchmarks , and can outperform the best clustering algorithms on more challenging high-dimensional image datasets like CIFAR-10 and CIFAR-20 . Further , empirical evaluation suggests that our algorithm also learns a meaningful feature space . 2 RELATED WORK . Manifold Learning . In classical manifold learning , the goal is to map the manifold-structured data points to a low-dimensional representation space such that the manifold structure is preserved . There are two key ingredients : 1 ) Choosing a geometric property from the original data space to be preserved . For example , the local euclidean neighborhood ( Belkin & Niyogi , 2003 ) , or linear interpolation by neighboring data points ( Roweis & Saul , 2000 ) . 2 ) The embedding should not collapse to a trivial solution . To avoid the trivial solution , the variance of the embedding space is typically constrained in spectral-based manifold learning methods . Manifold Clustering and Embedding . When the data should be modeled as a union of several manifolds , manifold clustering is needed in addition to manifold learning . When these manifolds are linear , subspace clustering algorithms ( Ma et al. , 2007 ; Elhamifar & Vidal , 2013 ; Vidal et al. , 2016 ) can be used . When they are non-linear , manifold clustering and embedding methods were proposed . They generally divide into 3 categories ( Abdolali & Gillis , 2021 ) : 1 . Locality preserving . 2 . Kernel based . 3 . Neural Network based . Locality preserving techniques implicitly make the assumption that the manifolds are smooth , and are sampled densely ( Souvenir & Pless , 2005 ; Elhamifar & Vidal , 2011 ; Chen et al. , 2018 ) . Additionally , smoothness assumption can be employed ( Gong et al. , 2012 ) . Our method generalizes those techniques by realizing them with geometrical constraints . The success of kernel based techniques depends strongly on the suitability of the underlying kernel , and generally requires a representation of the data in a space with higher dimension than the data space ( Patel & Vidal , 2014 ) . Deep subspace clustering methods , such as Ji et al . ( 2017 ) ; Zhang et al . ( 2019 ) ; Zhou et al . ( 2018 ) jointly perform linear subspace clustering and representation of the data , and has the potential to handle high dimensional data effectively . However , it has been shown that most performance gains obtained by those methods should be attributed to an ad-hoc post-processing step applied to the self-expression matrix . Using neural networks only provide a very marginal gain compared to clustering the raw data directly using linear SC ( Haeffele et al. , 2020 ) . Our work differs from those techniques mainly in two aspects : i ) While most of the previous methods were generative ( autoencoders , GANs ) , our loss function is defined in the latent embedding space and is best understood as a contrastive method . ii ) While previous methods use self-expression based SC to guide feature learning , ours uses MCR2 to learn the subspace features . Recently , some deep SC techniques also applied data augmentation ( Sun et al. , 2019 ; Abavisani et al. , 2020 ) . However , in those works , data augmentation played a complementary role of improving the performance . In our method , data augmentation plays a central role for enabling the identification of the clusters . Self-Supervised Representation Learning . Recently , self-supervised representation learning achieved tremendous success with deep neural networks . Similar to manifold clustering and embedding , there are also two essential ingredients : 1 ) Data augmentations are used to define the domain-specific invariance . 2 ) The latent representation should not collapse . The second requirement can be achieved either by contrastive learning ( Chen et al. , 2020 ) , momentum encoder ( He et al. , 2020 ; Grill et al. , 2020 ) or siamese network structure ( Chen & He , 2021 ) . More directly related to our work is Tao et al . ( 2021 ) , which proposed feature orthogonalization and decorrelation , alongside contrastive learning . Recently , variance regularization along was also successfully used to achieve principle 2 ) ( Zbontar et al. , 2021 ; Bardes et al. , 2021 ) , attaining state-of-the-art SSL representation performance . Part of our method , the total coding rate ( TCR ) objective achieves a similar effect , see discussion in Appendix A.6 . However , beyond self-supervised features , our algorithm additionally show strong clustering performance , and directly learns a meaningful latent space . The simultaneous manifold clustering and embedding in NMCE is also related to online deep clustering ( Caron et al. , 2018 ; 2020 ) method . Also notable is Deshmukh et al . ( 2021 ) , where the concept of population consistency is closely related to the constraint functional we discussed . Clustering with Data Augmentation Our method use data augmentation to ensure correct clustering of the training data . Although not explicitly pointed out , this is also the case for other clustering techniques ( Shen et al. , 2021 ; Tsai et al. , 2020 ; Li et al. , 2021 ) . Our understanding of data augmentation is also consistent to works that specifically study the success of data augmentations ( HaoChen et al. , 2021 ; von Kügelgen et al. , 2021 ) . 3 NEURAL MANIFOLD CLUSTERING AND EMBEDDING . 3.1 PROBLEM SETUP . Let ’ s assume data points xi ∈ Rd are sampled from a union ⋃n j=1 Xj of manifolds X1 , X2 , ... , Xn . 1 As stated in Task 1 , the goal of manifold clustering and embedding is to assign each data point to the corresponding manifold ( clustering ) , as well as learning a coordinate for each manifold ( manifold learning ) . To achieve this goal , we use neural network f , which learns to map a data point x to the feature embedding z ∈ Rdemb and the cluster assignment c ∈ [ 1 , n ] , i.e . z , c = f ( x ) . The cluster assignment shall be equal to the ground-truth manifold assignment,2 z should parameterize the coordinates of the corresponding manifold Xc . To make the feature space easier to work with , one can enforce the additional separability requirement that for any Xj , Xk and j 6= k , the feature vectors are perpendicular , Zj ⊥ Zk . Here Zj denotes the embedding feature vectors of data points in Xj , and we define perpendicular between two sets in the following fashion : If Z̃j ⊆ Zj , Z̃k ⊆ Zk such that ∀zj ∈ Z̃j , zk ∈ Z̃k , we have zj · zk 6= 0 , then either Z̃j or Z̃k has zero measure . In the following , we first argue that to make clustering possible with a neural network , one should define the additional geometric constraint that makes the manifold clusters identifiable . Second , we discuss how to implement the required geometric constraints and combine them with a recently proposed joint clustering and subspace learning algorithm MCR2 ( Yu et al. , 2020 ) to achieve neural manifold clustering and embeddng ( NMCE ) . 3.2 CLUSTERING ALWAYS INVOLVES IMPLICIT ASSUMPTIONS . Even the simplest clustering algorithms rely on implicit assumptions . For example , in k-means clustering , the implicit assumption is that the clusters in the original data space are continuous in terms of L2 distance . For Linear SC , the assumption is that data points are co-linear within each cluster . If a neural network is trained on example data to learn a cluster assignment function 1We do not consider topological issues here and assume that all of them are homeomorphic to Rdi for some di 2Up to a permutation since the training is unsupervised . c = f ( x ) , the resulting clustering will be arbitrary and not resemble the solution of k-means or linear SC clustering . This is because neural networks are flexible and no constraint is imposed on the clustering function to force the result to respect the geometry of the original data . One example of this is shown in Figure 1 left panel , where a deep clustering method outputs a rather arbitrary clustering for the double spiral data . To make the clusters learnable for a neural network , one needs to introduce constraints explicitly . In the example in Figure 1 , one can easily reason that , to be able to separate the two spirals from each other , the clustering function needs to ensure that all close neighbors of a point from the data distribution are assigned to the same cluster , essentially the same assumption implicitly made in locality-based manifold clustering algorithms ( Abdolali & Gillis , 2021 ) . We formalize the notion of constraints to a constraint functional D ( f ) ( for a specific data distribution ) that has the following property : All cluster assignment functions f that makes D attain its minimum value ( assumed to be 0 ) and also cluster data points to the ground truth number of clusters will correctly cluster all data points . For example , we can construct a functional D that takes value D = 0 for all clustering functions that satisfy the locality constraint on the dataset , and D > 0 otherwise . This notion of constraint function is completely general . For example , linear subspace clustering is recovered if the constraint function take value 0 if and only if the found clusters are linear subspace . In practice , one usually can not optimize the neural network clustering function f subject to D = 0 , since the correct solution would have to be found at initialization . A more practical way to use the constraint function is to use the relaxed objective with weighting λ : L ( f ) = Lclst ( f ) + λ ∗D ( f ) ( 1 ) where Lclst is some objective function that will force f to cluster the dataset . With a suitable λ , optimizing this objective leads to learning of the correct clusters , by the assumption above . To achieve manifold clustering , one just needs to find the appropriate constraint functional .

Summary: This work proposed a general manifold clustering algorithm called Neural Manifold Clustering and Embedding (NMCE), which utilize Maximum Coding Rate Reduction (MCR2 ) as the objective function and data augmentation to enforce constrains.    In the implementation stage, given that even the toy experiment is difficult to optimize with the full NMCE objective, a multistage training procedure is applied with the first stage actually trying to optimize the Total Coding Rate (TCR), which is another kind of self-supervised learning objective claimed in this paper.   On synthetic and real-world datasets COIL20, COIL100, CIFAR-10, CIFAR-20 and STL-10, NMCE achieved comparable and sometimes better results, compared to baseline methods or some alternative manifold clustering methods listed in the paper.