Generalized Clustering by Learning to Optimize Expected Normalized Cuts

We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples . Our clustering objective is based on optimizing normalized cuts , a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity . We define a differentiable loss function equivalent to the expected normalized cuts . Unlike much of the work in unsupervised deep learning , our trained model directly outputs final cluster assignments , rather than embeddings that need further processing to be usable . Our approach generalizes to unseen datasets across a wide variety of domains , including text , and image . Specifically , we achieve state-of-the-art results on popular unsupervised clustering benchmarks ( e.g. , MNIST , Reuters , CIFAR-10 , and CIFAR-100 ) , outperforming the strongest baselines by up to 10.9 % . Our generalization results are superior ( by up to 21.9 % ) to the recent top-performing clustering approach with the ability to generalize . 1 INTRODUCTION . Clustering unlabeled data is an important problem from both a scientific and practical perspective . As technology plays a larger role in daily life , the volume of available data has exploded . However , labeling this data remains very costly and often requires domain expertise . Therefore , unsupervised clustering methods are one of the few viable approaches to gain insight into the structure of these massive unlabeled datasets . One of the most popular clustering methods is spectral clustering ( Shi & Malik , 2000 ; Ng et al. , 2002 ; Von Luxburg , 2007 ) , which first embeds the similarity of each pair of data points in the Laplacian ’ s eigenspace and then uses k-means to generate clusters from it . Spectral clustering not only outperforms commonly used clustering methods , such as k-means ( Von Luxburg , 2007 ) , but also allows us to directly minimize the pairwise distance between data points and solve for the optimal node embeddings analytically . Moreover , it is shown that the eigenvector of the normalized Laplacian matrix can be used to find the approximate solution to the well known normalized cuts problem ( Ng et al. , 2002 ; Von Luxburg , 2007 ) . In this work , we introduce CNC , a framework for Clustering by learning to optimize expected Normalized Cuts . We show that by directly minimizing a continuous relaxation of the normalized cuts problem , CNC enables end-to-end learning approach that outperforms top-performing clustering approaches . We demonstrate that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet , which consequently results in better clustering accuracy . Let us motivate CNC through a simple example . In Figure 1 , we want to cluster 6 images from CIFAR-10 dataset into two clusters . The affinity graph for these data points is shown in Figure 1 ( a ) ( details of constructing such graph is discussed in Section 4.2 ) . In this example , it is obvious that the optimal clustering is the result of cutting the edge connecting the two triangles . Cutting this edge will result in the optimal value for the normalized cuts objective . In CNC , we define a new differentiable loss function equivalent to the expected normalized cuts objective . We train a deep learning model to minimize the proposed loss in an unsupervised manner without the need for any labeled datasets . Our trained model directly returns the probabilities of belonging to each cluster ( Figure 1 ( b ) ) . In this example , the optimal normalized cuts is 0.286 ( Equation 1 ) , and as we can see , the CNC loss also converges to this value ( Figure 1 ( c ) ) . We compare the performance of CNC to several learning-based clustering approaches ( SpectralNet ( Shaham et al. , 2018 ) , DEC ( Xie et al. , 2016 ) , DCN ( Yang et al. , 2017 ) , VaDE ( Jiang et al. , 2017 ) , DEPICT ( Ghasedi Dizaji et al. , 2017 ) , IMSAT ( Hu et al. , 2017 ) , and IIC ( Ji et al. , 2019 ) ) on four datasets : MNIST , Reuters , CIFAR10 , and CIFAR100 . Our results show up to 10.9 % improvement over the baselines . Moreover , generalizing spectral embeddings to unseen data points , a task commonly referred to as out-of-sample-extension ( OOSE ) , is a non-trivial task ( Bengio et al. , 2003 ; Belkin et al. , 2006 ; Mendoza Quispe et al. , 2016 ) . Our results confirm that CNC generalizes to unseen data . Our generalization results are superior ( by up to 21.9 % ) to SpectralNet ( Shaham et al. , 2018 ) , the recent top-performing clustering approach with the ability to generalize . 2 RELATED WORK . Recent deep learning approaches to clustering attempt to embed the input data into a form that is amenable to clustering by k-means or Gaussian Mixture Models . ( Yang et al. , 2017 ; Xie et al. , 2016 ) focused on learning representations for clustering . To find the clustering-friendly latent representations and to better cluster the data , DCN ( Yang et al. , 2017 ) proposed a joint dimensionality reduction ( DR ) and K-means clustering approach in which DR is accomplished via learning a deep neural network . DEC ( Xie et al. , 2016 ) simultaneously learns cluster assignment and the underlying feature representation by iteratively updating a target distribution to sharpen cluster associations . Several other approaches rely on a variational autoencoder that utilizes a Gaussian mixture prior ( Jiang et al. , 2017 ; Dilokthanakul et al. , 2016 ; Hu et al. , 2017 ; Ji et al. , 2019 ; Ben-Yosef & Weinshall , 2018 ) . These approaches are mainly based on data augmentation , where the network is trained to maximize the mutual information between inputs and predicted clusters , while regularizing the network so that the cluster assignment of the data points is consistent with the assignment of the augmented points . Different clustering objectives , such as self-balanced k-means and balanced min-cut , have also been exhaustively studied ( Liu et al. , 2017 ; Chen et al. , 2017 ; Chang et al. , 2014 ) . One of the most effective techniques is spectral clustering , which first generates node embeddings in the eigenspace of the graph Laplacian , and then applies k-means clustering to these vectors ( Shi & Malik , 2000 ; Ng et al. , 2002 ; Von Luxburg , 2007 ) . To address the fact that clusters with the lowest graph conductance tend to have few nodes ( Leskovec , 2009 ; Zhang & Rohe , 2018 ) , ( Zhang & Rohe , 2018 ) proposed regularized spectral clustering to encourage more balanced clusters . Generalizing clustering to unseen nodes and graphs is nontrivial ( Bengio et al. , 2003 ; Belkin et al. , 2006 ; Mendoza Quispe et al. , 2016 ) . A recent work , SpectralNet ( Shaham et al. , 2018 ) , takes a deep learning approach to spectral clustering that generalizes to unseen data points . This approach first learns embeddings of the similarity of each pair of data points in Laplacian ’ s eigenspace and then applies k-means to those embeddings to generate clusters . Unlike SpectralNet , we propose an end-to-end learning approach with a differentiable loss that directly minimizes the normalized cuts . We show that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet , which consequently results in better clustering accuracy . Our evaluation results show that CNC improves generalization accuracy on unseen data points by up to 21.9 % . 3 PRELIMINARIES . Since CNC objective is based on optimizing normalized cuts , in this section , we briefly overview the formal definition of this metric . 3.1 FORMAL DEFINITION OF NORMALIZED CUTS . Let G = ( V , E , W ) be a graph where V = { vi } and E = { e ( vi , vj ) |vi ∈ V , vj ∈ V } are the set of nodes and edges in the graph and wij ∈W is the edge weight of the e ( vi , vj ) . Let n be the number of nodes . A graph G can be clustered into g disjoint sets S1 , S2 , . . . Sg , where the union of the nodes in those sets are V ( ⋃g k=1 Sk = V ) , and each node belongs to only one set ( ⋂g k=1 Sk = ∅ ) , by simply removing edges connecting those sets . For example , in Figure 1 ( a ) , by removing one edge two disjoint clusters are formed . Normalized cuts ( Ncuts ) which is defined based on the graph conductance , has been studied by ( Shi & Malik , 2000 ; Zhang & Rohe , 2018 ) , and the cost of a cut that forms disjoint sets S1 , S2 , . . . Sg is computed as : Ncuts ( S1 , S2 , . . . Sg ) = g∑ k=1 cut ( Sk , S̄k ) vol ( Sk , V ) ( 1 ) Where S̄k represents the complement of Sk , i.e. , S̄k = ⋃ i6=k Si . cut ( Sk , S̄k ) is called cut and is the total weight of the edges that are removed fromG in order to form disjoint sets Sk and S̄k . vol ( Sk , V ) is the total edge weights ( wij ) , whose end points ( vi , or vj ) belong to Sk . The cut and vol are : cut ( Sk , S̄k ) = ∑ vi∈Sk , vj∈S̄k wij , vol ( Sk , V ) = ∑ vi∈Sk ∑ vj∈V wij ( 2 ) Note that in Equation 2 , Sk and S̄k are disjoint , i.e. , Sk ∩ S̄k = ∅ , while in vol , Sk ⊂ V . In running example ( Figure 1 ) , since the edge weights are one , cut ( S1 , S̄1 ) = cut ( S2 , S̄2 ) = 1 , and vol ( S1 , V ) = vol ( S2 , V ) = 2 + 2 + 3 = 7 . Thus the Ncuts ( S1 , S2 ) = 17 + 1 7 = 0.286 . In this example one can see that such clustering results in minimum value of the normalized cuts . CNC aims to find a cut that the normalized cuts ( Equation 1 ) is minimized . 4 CNC FRAMEWORK . Finding the cluster assignments that minimizes the normalized cuts is NP-complete and an approximation to the this problem is based on the eigenvectors of the normalized graph Laplacian which has been studied in ( Shi & Malik , 2000 ; Zhang & Rohe , 2018 ) . CNC , on the other hand , is a neural network framework for learning to cluster in the absence of labeled examples by directly minimizing the continuous relaxation of the normalized cuts . As shown in Algorithm 1 , end-to-end training of the CNC contains two steps , i.e , ( i ) data points embedding ( line 3 ) , and ( ii ) clustering ( lines 4-9 ) . In data points embedding , the goal is to learn embeddings that capture the affinity of the data points , while the clustering step uses those embeddings to learn the CNC model and outputs the cluster assignments . Next , we first focus on the clustering step and we introduce our new differentiable loss function to train CNC model . Later in Section 4.2 , we discuss the details of the embedding step .

Summary: This paper presents an end-to-end approach for clustering. The proposed model is called CNC. It simultaneously learns a data embedding that preserve data affinity using Siamese networks, and clusters data in the embedding space. The model is trained by minimizing a differentiable loss function that is derived from normalized cuts. As such, the embedding phase renders the data point friendly to spectral clustering.