Unsupervised Clustering using Pseudo-semi-supervised Learning

1 INTRODUCTION . Semi-supervised methods , which make use of large unlabelled data sets and a small labelled data set , have seen recent success , e.g. , ladder networks Rasmus et al . ( 2015 ) achieves 99 % accuracy in MNIST using only 100 labelled samples . These approaches leverage the unlabelled data to help the network learn an underlying representation , while the labelled data guides the network towards separating the classes . In this paper , we ask two questions : is it possible to create the small labelled data set required by semi-supervised methods purely using unsupervised techniques ? If so , can semi-supervised methods leverage this autonomously generated pseudo-labelled data set to deliver higher performance than state-of-the-art unsupervised approaches ? We answer both these questions in the affirmative . We first find that prior approaches for identifying pseudo-labels Caron et al . ( 2018 ) ; Chen ( 2018 ) ; Lee ( 2013 ) perform poorly because of their low accuracy ( Section 2 ) . To create a high accuracy pseudo-labelled data set autonomously , we use a combination of ensemble of deep networks with a custom graph clustering algorithm ( Section 4 ) . We first train an ensemble of deep networks in an unsupervised manner . Each network independently clusters the input . We then compare two input data points . If all of the networks agree that these two data points belong to the same cluster , we can be reasonably sure that these data points belong to the same class . In this way , we identify all input data pairs belonging to the same class with high precision in a completely unsupervised manner . In the next step , we use these high quality input pairs to generate a similarity graph , with the data points as nodes and edges between data points which are deemed to be similar by our ensemble . From this graph , we extract tight clusters of data points , which serve as pseudo-labels . Note that , in this step , we do not cluster the entire dataset , but only a small subset on which we can get high ∗Work done as a Research Fellow at Microsoft Research India precision . Extracting high quality clusters from this graph while ensuring that the extracted clusters correspond to different classes is challenging . We discuss our approach in Section 4.2.1 for solving this problem . In this way , our method extracts unambiguous samples belonging to each class , which serves as pseudo-labels for semi-supervised learning . For semi-supervised learning using the labels generated above , one could use ladder networks Rasmus et al . ( 2015 ) . However , we found that ladder networks is unsuitable for the initial unsupervised clustering step as it can degenerate to outputting constant values for all inputs in the absence of unsupervised loss . To enable unsupervised clustering , we augment ladder networks using information maximization Krause et al . ( 2010 ) to create the Ladder-IM , and with a dot product loss to create Ladder-Dot . We show in Section 5 that Ladder-IM and Ladder-Dot , by themselves , also provide improvements over previous state of the art . We use the same models for both the first unsupervised learning step as well as the subsequent pseudo-semi-supervised iterations . Finally , the approach of finding high quality clusters using an ensemble , and using them as labels to train a new ensemble of semi-supervised models , is iterated , yielding continued improvements . The large gains of our method mainly come from this iterative approach , which can in some cases , yield upto 17 % gains in accuracy over the base unsupervised models ( see section 5.4 ) . We name our pseudo-semi-supervised learning approach Kingdra1 . Kingdra is independent of the type of data set ; we show examples of its use on both image and text data sets in Section 5 . This is in contrast to some previous approaches using CNNs , e.g . Chang et al . ( 2017 ) , Caron et al . ( 2018 ) , which are specialized for image data sets . We perform unsupervised classification using Kingdra on several standard image ( MNIST , CIFAR10 , STL ) and text ( reuters , 20news ) datasets . On all these datasets , Kingdra is able to achieve higher clustering accuracy compared to current state-of-the-art deep unsupervised clustering techniques . For example , on the CIFAR10 and 20news datasets , Kingdra is able to achieve classification accuracy of 54.6 % and 43.9 % , respectively , delivering 8-12 % absolute gains over state of the art results Hu et al . ( 2017 ) ; Xie et al . ( 2016 ) . 2 PRIOR WORK ON GENERATING PSEUDO-LABELS . Several techniques have been proposed in the literature for generating pseudo-labels ( Caron et al . ( 2018 ) ; Chen ( 2018 ) ; Lee ( 2013 ) . In Lee ( 2013 ) , the output class with the highest softmax value ( Argmax ) is taken to be the pseudo-label . In Caron et al . ( 2018 ) , the authors perform K-means clustering on the feature vector and use the K-means clusters as pseudo-labels . Finally , authors in Chen ( 2018 ) treat the softmax output as confidence and only label those items whose confidence value is above a high threshold . Note that none of these techniques for identifying pseudo-labels have been applied in our context , i.e. , for unsupervised clustering using semi-supervised models . In this section , we evaluate if pseudo-labels created by these prior techniques can be leveraged by semi-supervised models to improve clustering accuracy . We start with a semi-supervised model based on Ladder networks ( Rasmus et al . ( 2015 ) ) called Ladder-IM ( see Section 4.1 for model details ) and train using only its unsupervised loss terms on MNIST and CIFAR10 datasets . We use each of the above three pseudo-labelling approaches on the trained model to provide an initial set of pseudo-labels to the datasets ( e.g. , using K-means clustering on the feature vector of the model as in Caron et al . ( 2018 ) , etc. ) . We call the accuracy of these pseudo-labels the initial pseudo-label accuracy . We then use these generated pseudo-labels along with the datasets to train the model again , 1Our system is named after a semi-pseudo Pokémon . now with a supervised loss term ( based on the pseudo-labels ) and the earlier unsupervised loss terms . We again run the pseudo-labelling approaches on the newly trained model to derive an updated set of pseudo-labels . We iterate this process of training and pseudo-labelling until the pseudo-label accuracy stabilizes . We call this the final clustering accuracy . The initial pseudo-label accuracy and the final clustering accuracy results for the three approaches are shown in Table 1 . First , consider MNIST . The unsupervised clustering accuracy of Ladder-IM is 95.4 % . Argmax simply assigns pseudo-labels based on the model ’ s output and since this doesn ’ t add any new information for subsequent iterations , the final accuracy remains at 95.4 % . On the other hand , the pseudo-labels identified by both the K-means and threshold approaches result in worse initial label accuracy ( 75.4 % and 88.6 % ) . When this low-accuracy pseudo-label is used as supervision to train the model further , it results in a low final clustering accuracy of 60.9 % and 91.6 % , respectively . CIFAR10 results are similar . Ladder-IM clustering accuracy is 49 % which remains the same under Argmax as before . Pseudo-label accuracy using the K-means approach is worse and results in pulling down the final accuracy to 44.8 % . Interestingly , threshold results in a slightly higher initial accuracy of 60.5 % but even this is not high enough to improve the final clustering accuracy for CIFAR10 . From these results , we arrive at the following two conclusions . First , if the initial pseudo-label accuracy is not high , using pseudo-labels as supervision can result in bringing down the final clustering accuracy . Thus , high accuracy of initial pseudo-labels is crucial for improving clustering accuracy . Second , current approaches for identifying pseudo-labels do not deliver high accuracy and hence are unable to help improve overall clustering accuracy . 3 RELATED WORK . Unsupervised clustering : Various unsupervised clustering methods have been proposed over the years . Ng et al . ( 2002 ) uses a spectral clustering based approach , while Elhamifar & Vidal ( 2009 ) uses a sparse subspace approach for unsupervised learning . Recently , several deep neural networks based methods have been proposed , which scale well to large datasets . The ability of deep neural networks to learn higher level representations make them a good choice for unsupervised learning . Coates & Ng ( 2012 ) and Caron et al . ( 2018 ) use convnets and k-means for clustering . Caron et al . ( 2018 ) for example , iterates over clustering the features obtained from a convnet , and training the classifier using these clusters as pseudo-labels . The authors do not report clustering perfomence and we observed that this method can easily degenerate . Chang et al . ( 2017 ) uses pair-wise dot-product based similarity to identify close input pairs , which provide a supervisory signal . These convnets based approaches however work on only image datasets . Xie et al . ( 2016 ) simultaneously learns feature representations and cluster assignments using deep neural networks , and works on both image and text datasets . Hu et al . ( 2017 ) uses regularization combined with mutual information loss for unsupervised learning and achieves state of the art results . The authors conduct experiments in two settings - Random Perturbation Training and Virtual Adversarial Training . Other works such as Hjelm et al . ( 2018 ) using mutual information , maximize the mutual information between the spatial features and the non-spatial features . Ji et al . ( 2019 ) maximizes the mutual information between the predicted label of the image and the predicted label of the augmented image . This method uses convolution networks and requires domain knowledge of the dataset . Self-supervised learning : Another form of unsupervised learning uses auxiliary learning tasks for which labels can be self generated to generate useful representations from data . Many methods use spatial information of image patches to generate self-supervised data . E.g . Pathak et al . ( 2016 ) predicts pixels in an image patch using surrounding patches , while Doersch et al . ( 2015 ) predicts the relative position of image patches . Sermanet et al . ( 2018 ) uses time as a self supervisory signal between videos taken from different view points . Temporal signal is also used to learn representations from single videos by predicting future frames , e.g . Denton et al . ( 2017 ) . Our method uses correlation between outputs of input points across an ensemble as a supervisory signal to generate self-supervised pseudo-labels . Semi-Supervised learning : Semi-supervised approaches use sparse labelling of datapoints . Szummer & Jaakkola ( 2002 ) propagates labels based on nearest neighbors . Weston et al . ( 2012 ) uses a deep version of label propagation . Lee ( 2013 ) adjusts labels probabilities , starting with trusting only true labels and gradually increases the weight of pseudo labels . Rasmus et al . ( 2015 ) employs a denoising auto encoder architecture and have shown impressive performance . Tarvainen & Valpola ( 2017 ) uses an averaged model over previous iterations as a teacher . Other than these , some semi-supervised learning methods like Xie et al . ( 2019 ) and Berthelot et al . ( 2019 ) use data augmentation and assume some domain knowledge of the dataset with some of the data augmentation specific to image datasets . Miyato et al . ( 2018 ) and Shinoda et al . ( 2017 ) uses virtual adversarial training combined with the classification loss to perform semi-supervised classification . However , we found that these methods do not work well if we jointly train them with unsupervised losses . Ladder networks does not require any domain-dependent augmentation , works for both image and text datasets , and can be easily jointly trained with supervised and unsupervised losses . Thus , we chose to work with Ladder networks in our experiments , though our approach is general enough to work with any semi-supervised method that accommodates training with unsupervised loss terms . Unsupervised ensemble learning : Unsupervised ensemble learning has been mostly limited to generating a set of clusterings and combining them into a final clustering . Huang et al . ( 2016 ) cast ensemble clustering into a binary linear programming problem . Wang et al . ( 2009 ) ; Fred & Jain ( 2005 ) use a pair wise co-occurrence approach to construct a co-association matrix and use it to measure similarity between data points . See Vega-Pons & Ruiz-Shulcloper ( 2011 ) for a survey of ensemble clustering algorithms . Note that to the best of our knowledge none of the ensemble clustering algorithms employ a semi-supervised step like ours , or make use of deep networks . 4 PROPOSED FRAMEWORK An overview of the Kingdra method is summarized in Figure 1 . Given an unlabelled dataset X = { x1 , . . . , xn } , we start with unsupervised training of an ensemble of models M = { M1 , . . . , Mm } . For the individual models , any unsupervised model can be used . However , we propose a novel Ladder- * model , in which we build on ladder networks Rasmus et al . ( 2015 ) and modify it to support clustering . Next , we use the agreement between the ensemble models on a pair of data points , as a measure of similarity between the data points . This pairwise data is used to construct a similarity graph , from which we extract k tight clusters of data points , which serve as pseudo-labels . Note that , in this step , we do not cluster the entire dataset , but only a small subset on which we can get high precision . This is important for improving the accuracy of our semi-supervised training , as discussed in section 2 . These pseudo-labels then serve as training data for semi-supervised training of a new ensemble of Ladder- * models . Finally , we perform multiple iterations of the above steps for continued improvement .

Summary: This paper proposes a method for unsupervised clustering. Similarly to others unsupervised learning (UL) papers like "Deep Clustering for Unsupervised Learning of Visual Features" by Caron et al., they propose an algorithm alternating between a labelling phase and a training phase. Though, it has interesting differences. For example, unlike the Caron et al. paper, not all the samples get assigned a labels but only the most confident ones. These samples are determined by the pruning of a graph whose edges are determined by the votes of an ensemble of clustering models. Then, these pseudo labels are used within a supervised loss which act as a regularizer for the retraining of the clustering models.