Consensus Clustering with Unsupervised Representation Learning

1 INTRODUCTION . Supervised learning algorithms have shown great progress recently , but generally require a lot of labeled data . However , in many domains ( e.g. , advertising , social platforms , etc . ) , most of the available data are not labeled and manually labeling it is a very labor , time , and cost intensive task ( Xiao et al. , 2015 ; Deshmukh , 2019 ; Mintz et al. , 2009 ; Blum & Mitchell , 1998 ) . On the other hand , clustering algorithms do not need labeled data to group similar data points into clusters . Some popular clustering algorithms include k-means , hierarchical clustering , DBSCAN ( Ester et al. , 1996 ) , spectral clustering , etc. , and the usefulness of each algorithm varies with the application . In this work , we deal with the clustering of images . Traditional clustering approaches focus on hand crafted features on which out of the box clustering algorithms are applied . However , hand crafted features may not be optimal , and are not scalable to large scale real word datasets ( Wu et al. , 2019 ) . Advancements in deep learning techniques have enabled end-to-end learning of rich representations for supervised learning . On the other hand , simultaneously learning the feature spaces while clustering leads to degenerate solutions , which until recently limited end to end implementations of clustering with representation learning approaches ( Caron et al. , 2018 ) . Recent deep clustering works take several approaches to address this issue such as alternating pseudo cluster assignments and pseudo supervised training , comparing the predictions with their own high confidence assignments ( Caron et al. , 2018 ; Asano et al. , 2019 ; Xie et al. , 2016 ; Wu et al. , 2019 ) , and maximizing mutual information between predictions of positive pairs ( Ji et al. , 2019 ) . Although these methods show impressive performance on challenging datasets , we believe taking advantage of rich ideas from ensemble learning for clustering with representation learning will enhance the performance of deep clustering methods . Ensemble learning methods train a variety of learners and build a meta learner by combining the predictions of individual learners ( Dietterich , 2000 ; Breiman , 1996 ; Freund et al. , 1996 ) . In practice they have been heavily used in supervised learning setting . Ensemble learning methods have also found their place in clustering i.e . knowledge reuse framework ( Strehl & Ghosh , 2002 ) where a consensus algorithm is applied on constituent cluster partitions to generate an updated partition that clusters the data better than any component partition individually . However , the knowledge reuse framework and much of the consensus clustering literature that followed ( Fern & Brodley , 2003 ; Fred & Jain , 2005 ; Topchy et al. , 2005 ) do not make use of the underlying features used to generate the ensemble . We propose the use of consensus clustering as a way to extend ensemble methods to unsupervised representation learning . In particular , we define a ’ disagreement ’ measure among the constituents of the ensemble . The key motivation for this is that the diversity of the ensemble drives the minimization of the disagreement measure in a synergistic way , thereby leading to better representations . We propose Consensus Clustering with Unsupervised Representation Learning ( ConCURL ) and following are our main contributions : 1 . A novel ensemble learning algorithm which learns representations by creating a consensus on multiple clustering outputs generated by applying random transformations on the embeddings . 2 . Our method outperforms the current state of the art clustering algorithms on popular computer vision datasets based on clustering metrics ( A.4 ) . 3 . Even though there is no labeled data available while learning representations , clustering may still be prone to be overfitting to the “ training data. ” As stated in Bubeck & Von Luxburg ( 2007 ) , in clustering , we generally assume that the finite data set has been sampled from some underlying space and the goal is to find the true approximate partition of the underlying space rather than the best partition in a given finite data set . Hence , to check generalizability of the method proposed we also evaluate our models on the “ test data ” - data that was not available during training/representation learning . Our method is more generalizable compared to state of the art methods ( i.e . it outperforms the other algorithms when evaluated on the test set ) . 2 RELATED WORK . Clustering is a ubiquitous task and it has been actively used in many different scientific and practical pursuits such as detecting genes from microarray data ( Frey & Dueck , 2007 ) , clustering faces ( Rodriguez & Laio , 2014 ) , and segmentation in medical imaging to support diagnosis ( Masulli & Schenone , 1999 ) . We refer interested readers to these excellent sources for a survey of these uses Jain et al . ( 1999 ) ; Liao ( 2005 ) ; Xu & Wunsch ( 2005 ) ; Nugent & Meila ( 2010 ) . Clustering with Deep Learning : In their influential work , Caron et al . ( 2018 ) show that it is possible to train deep convolutional neural networks with pseudo labels that are generated by a clustering algorithm ( DeepCluster ) . More precisely , in DeepCluster , previous versions of representations are used to assign pseudo labels to the data using an out of the box clustering algorithm such as k-means . These pseudo labels are used to improve the learned representation of the data by minimizing a supervised loss . Along the same lines , several more methods have been proposed . For example , Gaussian ATtention network for image clustering ( GATCluster ) ( Niu et al. , 2020 ) comprises four self-learning tasks with the constraints of transformation invariance , separability maximization , entropy analysis and attention mapping . Training is performed in two distinct steps , similar to Caron et al . ( 2018 ) where the first step is to compute pseudo targets for a large batch of data and the second step is to train the model in a supervised way using the pseudo targets . Both DeepCluster and GATCluster use k-means to generate pseudo labels which may not scale well . Wu et al . ( 2019 ) propose Deep Comprehensive Correlation Mining ( DCCM ) , where discriminative features are learned by taking advantage of the correlations of the data using pseudo-label supervision and triplet mutual information among features . However , DCCM may be susceptible to trivial solutions ( Niu et al. , 2020 ) . Invariant Information Clustering ( IIC ) ( Ji et al. , 2019 ) maximizes mutual information between the class assignments of two different views of the same image ( paired samples ) in order to learn representations that preserve what is common between the views while discarding instance specific details . Ji et al . ( 2019 ) argue that the presence of an entropy term in mutual information plays an important role in avoiding the degenerate solutions . However a large batch size is needed for the computation of mutual information in IIC , which may not be scalable for larger image sizes which are common in popular datasets ( Ji et al. , 2019 ; Niu et al. , 2020 ) . Huang et al . ( 2020 ) extend the traditional maximal margin clustering idea to the deep learning paradigm , by learning the most semantically plausible clustering through minimizing a proposed partition uncertainty index . Their algorithm PICA uses a stochastic version of the index , thereby facilitating mini-batch training . PICA fails to assign a sample-correct cluster when that sample either has high foreground or background similarity to samples in other clusters . SCAN ( Van Gansbeke et al. , 2020 ) uses the learnt representations from a pretext task to find semantically closest images to the given image using nearest neighbors . SCAN achieves state-of-the-art results on CIFAR-10 , CIFAR-100 , STL-10 , however using such priors from existing pretext tasks deviates from the main idea of our paper of end-to-end learning . We believe however that we can benefit by using SCAN on top of the features learnt using our algorithm to improve the clustering accuracy . Our proposed approach ConCURL is scalable to large datasets , does not suffer from trivial solutions and shows superior performance on a challenging set of image data sets . As shown in the experimental results , our proposed method also generalizes well to data points that were not available during training , when compared to the above approaches . Self-supervised Representation Learning : Self-supervised learning is a sub-field of unsupervised learning in which the main goal is to learn general purpose representations by exploiting userdefined sub-tasks such as the relationship between different views of the same data . Although self-supervised learning methods show impressive performance on a variety of problems , it is not clear whether learned representations are good for clustering . There are many different flavors of self supervised learning such as Instance Recognition tasks ( Wu et al. , 2018 ) , contrastive techniques ( Chen et al. , 2020 ) , etc . In instance recognition tasks , each image is considered as its own category so that the learnt embeddings are well separated . Zhuang et al . ( 2019 ) propose a novel Local Aggregation method based on non-parametric instance discrimination tasks . They use a robust clustering objective ( using multiple runs of k-means ) to move statistically similar data points closer in the representation space and dissimilar data points further away . However , instance recognition tasks usually require a large memory bank which is memory intensive . In contrastive learning ( Tian et al. , 2019 ; He et al. , 2020 ; Hénaff et al. , 2019 ; Hjelm et al. , 2018 ; Chen et al. , 2020 ) , representations are learned by maximizing agreement between different augmented views of the same data example ( known as positive pairs ) and minimizing agreement between augmented views of different examples ( known as negative pairs ) . SimCLR ( Chen et al. , 2020 ) achieves state-of-the-art-results without specialized architectures or a memory bank of negative pairs ( usually required by contrastive learning techniques ) . However , it still requires negative examples and , as it applies to instance classification , it has to compare every pair of images . These issues are addressed by Bootstrap your own latent ( BYOL ) ( Grill et al. , 2020 ) and Swapping Assignments between multiple Views ( SwAV ) ( Caron et al. , 2020 ) . BYOL does not require negative examples and SwAV does not need to compare every pair of images . For the main study in this paper we use BYOL as a representation learning block and adapt the soft clustering loss which was used in SwAV to learn prototypes , thereby addressing both the issues of negative samples and need for comparing every pair of images . Note that our proposed algorithm can use any representation learning block like SimCLR , BYOL , & SwAV and can also use other soft clustering loss formulations . Consensus Clustering : Strehl & Ghosh ( 2002 ) propose a knowledge reuse framework ; to build an unsupervised ensemble by using several distinct clusterings of the same data and assuming that the underlying features that are used to compute the clustering are not available and fixed . Fern & Brodley ( 2003 ) build on the cluster ensemble framework based on random projections . In this framework , Fern & Brodley ( 2003 ) show that a single run of clustering ( random projection + Expectation Maximization ) is highly unstable . They perform multiple runs of clustering and compute an aggregated similarity matrix which is used to cluster the data using an agglomerative clustering algorithm . Fred & Jain ( 2005 ) propose a voting approach to map the cluster assignments in the ensemble into a new similarity measure between clusterings . The co-association matrix thus formed can be used with any clustering algorithm to result in a new data partition . Weighted cluster ensembles ( Domeniconi & Al-Razgan , 2009 ) uses Locally Adaptive Clustering ( LAC ) ( Domeniconi et al. , 2004 ) as a base algorithm which assigns weights to each feature of the sample according to the local variance of data along each dimension . Authors use LAC with different hyperparameters and propose three different partitioning algorithms to generate consensus among LAC clusterings . Huang et al . ( 2017 ) argues that most ensemble based methods treat all base clusterings equally and a few which weigh the base clusterings give those weights globally . Authors propose an ensemble clustering approach based on cluster uncertainty ( calculated by entropic criterion ) and a local weighting strategy . There are also survey papers which have in depth analysis and details of various methods used in consensus clusteringVega-Pons & Ruiz-Shulcloper ( 2011 ) ; Ghosh & Acharya ( 2011 ) . It is not clear how any of these methods can be adapted when one needs to do representation learning along with clustering . It is also not clear if one can come up with end-to-end learning architecture with any of the above methods . In contrast , our proposed consensus clustering method ConCURL can easily be integrated into any deep learning architecture for clustering and trained end-to-end .

Summary: The authors propose a learning-based approach for image clustering. In particular, similarly to recent algorithms fro unsupervised representation learning, such a DeepCluster, they propose to iterate between clustering the images in the feature space of the network and updating the network weights to respect the clusters. Two main differences with respect to DeepCluster-like algorithms is that they target the task of clustering itself, and do not evaluate the generalizability of the leaned representations for other tasks, and that they propose to use cluster ensembles to improve training robustness. In particular, they generate 2 clusterings of the images at every iteration by applying different sets of data augmentations and feature transformations to the input images. The objective is then not only to respect these clusterings but also to enforce consistency between them over time, thus improving representation invariance to irrelevant image details. In an experimental evaluation on a set of standard image clustering benchmarks they outperform prior work in most scenarios.