Generalized Clustering by Learning to Optimize Expected Normalized Cuts

We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples . Our clustering objective is based on optimizing normalized cuts , a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity . We define a differentiable loss function equivalent to the expected normalized cuts . Unlike much of the work in unsupervised deep learning , our trained model directly outputs final cluster assignments , rather than embeddings that need further processing to be usable . Our approach generalizes to unseen datasets across a wide variety of domains , including text , and image . Specifically , we achieve state-of-the-art results on popular unsupervised clustering benchmarks ( e.g. , MNIST , Reuters , CIFAR-10 , and CIFAR-100 ) , outperforming the strongest baselines by up to 10.9 % . Our generalization results are superior ( by up to 21.9 % ) to the recent top-performing clustering approach with the ability to generalize . 1 INTRODUCTION . Clustering unlabeled data is an important problem from both a scientific and practical perspective . As technology plays a larger role in daily life , the volume of available data has exploded . However , labeling this data remains very costly and often requires domain expertise . Therefore , unsupervised clustering methods are one of the few viable approaches to gain insight into the structure of these massive unlabeled datasets . One of the most popular clustering methods is spectral clustering ( Shi & Malik , 2000 ; Ng et al. , 2002 ; Von Luxburg , 2007 ) , which first embeds the similarity of each pair of data points in the Laplacian ’ s eigenspace and then uses k-means to generate clusters from it . Spectral clustering not only outperforms commonly used clustering methods , such as k-means ( Von Luxburg , 2007 ) , but also allows us to directly minimize the pairwise distance between data points and solve for the optimal node embeddings analytically . Moreover , it is shown that the eigenvector of the normalized Laplacian matrix can be used to find the approximate solution to the well known normalized cuts problem ( Ng et al. , 2002 ; Von Luxburg , 2007 ) . In this work , we introduce CNC , a framework for Clustering by learning to optimize expected Normalized Cuts . We show that by directly minimizing a continuous relaxation of the normalized cuts problem , CNC enables end-to-end learning approach that outperforms top-performing clustering approaches . We demonstrate that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet , which consequently results in better clustering accuracy . Let us motivate CNC through a simple example . In Figure 1 , we want to cluster 6 images from CIFAR-10 dataset into two clusters . The affinity graph for these data points is shown in Figure 1 ( a ) ( details of constructing such graph is discussed in Section 4.2 ) . In this example , it is obvious that the optimal clustering is the result of cutting the edge connecting the two triangles . Cutting this edge will result in the optimal value for the normalized cuts objective . In CNC , we define a new differentiable loss function equivalent to the expected normalized cuts objective . We train a deep learning model to minimize the proposed loss in an unsupervised manner without the need for any labeled datasets . Our trained model directly returns the probabilities of belonging to each cluster ( Figure 1 ( b ) ) . In this example , the optimal normalized cuts is 0.286 ( Equation 1 ) , and as we can see , the CNC loss also converges to this value ( Figure 1 ( c ) ) . We compare the performance of CNC to several learning-based clustering approaches ( SpectralNet ( Shaham et al. , 2018 ) , DEC ( Xie et al. , 2016 ) , DCN ( Yang et al. , 2017 ) , VaDE ( Jiang et al. , 2017 ) , DEPICT ( Ghasedi Dizaji et al. , 2017 ) , IMSAT ( Hu et al. , 2017 ) , and IIC ( Ji et al. , 2019 ) ) on four datasets : MNIST , Reuters , CIFAR10 , and CIFAR100 . Our results show up to 10.9 % improvement over the baselines . Moreover , generalizing spectral embeddings to unseen data points , a task commonly referred to as out-of-sample-extension ( OOSE ) , is a non-trivial task ( Bengio et al. , 2003 ; Belkin et al. , 2006 ; Mendoza Quispe et al. , 2016 ) . Our results confirm that CNC generalizes to unseen data . Our generalization results are superior ( by up to 21.9 % ) to SpectralNet ( Shaham et al. , 2018 ) , the recent top-performing clustering approach with the ability to generalize . 2 RELATED WORK . Recent deep learning approaches to clustering attempt to embed the input data into a form that is amenable to clustering by k-means or Gaussian Mixture Models . ( Yang et al. , 2017 ; Xie et al. , 2016 ) focused on learning representations for clustering . To find the clustering-friendly latent representations and to better cluster the data , DCN ( Yang et al. , 2017 ) proposed a joint dimensionality reduction ( DR ) and K-means clustering approach in which DR is accomplished via learning a deep neural network . DEC ( Xie et al. , 2016 ) simultaneously learns cluster assignment and the underlying feature representation by iteratively updating a target distribution to sharpen cluster associations . Several other approaches rely on a variational autoencoder that utilizes a Gaussian mixture prior ( Jiang et al. , 2017 ; Dilokthanakul et al. , 2016 ; Hu et al. , 2017 ; Ji et al. , 2019 ; Ben-Yosef & Weinshall , 2018 ) . These approaches are mainly based on data augmentation , where the network is trained to maximize the mutual information between inputs and predicted clusters , while regularizing the network so that the cluster assignment of the data points is consistent with the assignment of the augmented points . Different clustering objectives , such as self-balanced k-means and balanced min-cut , have also been exhaustively studied ( Liu et al. , 2017 ; Chen et al. , 2017 ; Chang et al. , 2014 ) . One of the most effective techniques is spectral clustering , which first generates node embeddings in the eigenspace of the graph Laplacian , and then applies k-means clustering to these vectors ( Shi & Malik , 2000 ; Ng et al. , 2002 ; Von Luxburg , 2007 ) . To address the fact that clusters with the lowest graph conductance tend to have few nodes ( Leskovec , 2009 ; Zhang & Rohe , 2018 ) , ( Zhang & Rohe , 2018 ) proposed regularized spectral clustering to encourage more balanced clusters . Generalizing clustering to unseen nodes and graphs is nontrivial ( Bengio et al. , 2003 ; Belkin et al. , 2006 ; Mendoza Quispe et al. , 2016 ) . A recent work , SpectralNet ( Shaham et al. , 2018 ) , takes a deep learning approach to spectral clustering that generalizes to unseen data points . This approach first learns embeddings of the similarity of each pair of data points in Laplacian ’ s eigenspace and then applies k-means to those embeddings to generate clusters . Unlike SpectralNet , we propose an end-to-end learning approach with a differentiable loss that directly minimizes the normalized cuts . We show that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet , which consequently results in better clustering accuracy . Our evaluation results show that CNC improves generalization accuracy on unseen data points by up to 21.9 % . 3 PRELIMINARIES . Since CNC objective is based on optimizing normalized cuts , in this section , we briefly overview the formal definition of this metric . 3.1 FORMAL DEFINITION OF NORMALIZED CUTS . Let G = ( V , E , W ) be a graph where V = { vi } and E = { e ( vi , vj ) |vi ∈ V , vj ∈ V } are the set of nodes and edges in the graph and wij ∈W is the edge weight of the e ( vi , vj ) . Let n be the number of nodes . A graph G can be clustered into g disjoint sets S1 , S2 , . . . Sg , where the union of the nodes in those sets are V ( ⋃g k=1 Sk = V ) , and each node belongs to only one set ( ⋂g k=1 Sk = ∅ ) , by simply removing edges connecting those sets . For example , in Figure 1 ( a ) , by removing one edge two disjoint clusters are formed . Normalized cuts ( Ncuts ) which is defined based on the graph conductance , has been studied by ( Shi & Malik , 2000 ; Zhang & Rohe , 2018 ) , and the cost of a cut that forms disjoint sets S1 , S2 , . . . Sg is computed as : Ncuts ( S1 , S2 , . . . Sg ) = g∑ k=1 cut ( Sk , S̄k ) vol ( Sk , V ) ( 1 ) Where S̄k represents the complement of Sk , i.e. , S̄k = ⋃ i6=k Si . cut ( Sk , S̄k ) is called cut and is the total weight of the edges that are removed fromG in order to form disjoint sets Sk and S̄k . vol ( Sk , V ) is the total edge weights ( wij ) , whose end points ( vi , or vj ) belong to Sk . The cut and vol are : cut ( Sk , S̄k ) = ∑ vi∈Sk , vj∈S̄k wij , vol ( Sk , V ) = ∑ vi∈Sk ∑ vj∈V wij ( 2 ) Note that in Equation 2 , Sk and S̄k are disjoint , i.e. , Sk ∩ S̄k = ∅ , while in vol , Sk ⊂ V . In running example ( Figure 1 ) , since the edge weights are one , cut ( S1 , S̄1 ) = cut ( S2 , S̄2 ) = 1 , and vol ( S1 , V ) = vol ( S2 , V ) = 2 + 2 + 3 = 7 . Thus the Ncuts ( S1 , S2 ) = 17 + 1 7 = 0.286 . In this example one can see that such clustering results in minimum value of the normalized cuts . CNC aims to find a cut that the normalized cuts ( Equation 1 ) is minimized . 4 CNC FRAMEWORK . Finding the cluster assignments that minimizes the normalized cuts is NP-complete and an approximation to the this problem is based on the eigenvectors of the normalized graph Laplacian which has been studied in ( Shi & Malik , 2000 ; Zhang & Rohe , 2018 ) . CNC , on the other hand , is a neural network framework for learning to cluster in the absence of labeled examples by directly minimizing the continuous relaxation of the normalized cuts . As shown in Algorithm 1 , end-to-end training of the CNC contains two steps , i.e , ( i ) data points embedding ( line 3 ) , and ( ii ) clustering ( lines 4-9 ) . In data points embedding , the goal is to learn embeddings that capture the affinity of the data points , while the clustering step uses those embeddings to learn the CNC model and outputs the cluster assignments . Next , we first focus on the clustering step and we introduce our new differentiable loss function to train CNC model . Later in Section 4.2 , we discuss the details of the embedding step .

Summary: The paper suggests a differentiable objective that can be used to train a network to output cluster probabilities for a given datapoint, given a fixed number of clusters and embeddings of the data points to be clustered. In particular, this objective can be seen as a relaxation of the normalized cut objective, where indicator variables in the original formulation are replaced with their expectations under the trained model. The authors experiment with a number of clustering datasets where the number of cluster is known beforehand (and where, for evaluation purposes, the ground truth is known), and find that their method generally improves over the clustering performance of SpectralNet (Shaham et al., 2018) in terms of accuracy and normalized mutual information, and that it finds solutions with lower normalized cut values.