Unsupervised Clustering using Pseudo-semi-supervised Learning
Authors: Divam Gupta
Ramachandran Ramjee
Nipun Kwatra
Muthian Sivathanu
Venue:  (2020)
Keywords: Unsupervised Learning
Unsupervised Clustering
Deep Learning

Abstract: In this paper, we propose a framework that leverages semi-supervised models to improve unsupervised clustering performance. To leverage semi-supervised models, we first need to automatically generate labels, called pseudo-labels. We find that prior approaches for generating pseudo-labels hurt clustering performance because of their low accuracy. Instead, we use an ensemble of deep networks  to construct a similarity graph, from which we extract high accuracy pseudo-labels. The approach of finding high quality pseudo-labels using ensembles and training the semi-supervised model is iterated, yielding continued improvement. We show that our approach outperforms state of the art clustering results for multiple image and text datasets. For example, we achieve 54.6% accuracy for CIFAR-10 and 43.9% for 20news, outperforming state of the art by 8-12% in absolute terms.

Published as a conference paper at ICLR 2020
UNSUPERVISED CLUSTERING USING PSEUDO-SEMI- SUPERVISED LEARNING
Divam Gupta
Ramachandran Ramjee
Carnegie Mellon University di vamê cmu edu
Microsoft Research India ram ramjee@mi wtw.emirrosssseeeee crosoft...... com
Nipun Kwatra
Muthian Sivathanu Microsoft Research India muthian@microsot com
Microsoft Research India nipun. www.:kaaiiaiiooo....... com
ABSTRACT
In this paper, we propose a framework that leverages semi-supervised models improve unsupervised clustering performance. To leverage semi-supervised models, we first need to automatically generate labels, called pseudo-labels. We find that prior approaches for generating pseudo-labels hurt clustering performance because of their low accuracy. Instead, we use an ensemble of deep networks construct a similarity graph, from which we extract high accuracy pseudo- labels. The approach of finding high quality pseudo-labed using ensembles and training the superviee model iterated, yielding continued improvement. We show that our approach outperforms state of the art clustering results for multiple image and text datasets. For example, we achieve 54.6% accuracy for CIFAR- 10 and 43.9% for 20news, outperforming state of the art by 8-12% in absolute terms. Project details and code are available at https: / /i wwiiavviaaaaaaaaooaooaa com/ pseudo- semi www--eei---uprrrree-------aeeee-
INTRODUCTION
Semi-supervised methods, which make use of large unlabelled data sets and a small labelled data set, have seen recent success, e.g., ladder networks Rasmus MNIST using only 100 labelled samples. These approaches leverage the unlabelled data help the network learn an underlying representation, while the labelled data guides the network towards separating the classes.
Rasmusss et al. (2015) achieves 99% accuracy in
In this paper, we ask two questions: is it possible to create the small labelled data set required by semi-supervised methods purely using unsupervised techniques? ? If so, can semi-supervised methods leverage this autonomously generated pseudo-labelled data set to deliver higher performance than state-of-the-ee unsupervised approaches? We answer both these questions in the affirmative.
We first find that prior approaches for identifying pseudo-labels Caron et al. 20; Chen 2018);
Lee (2013) perform poorly because of their low accuracy (Section 2). To create a high accuracy pseudo-labelled data set autonomously, we combination of ensemble of deep networks with custom graph clustering algorithm (Section 4). We first train an ensemble of deep networks in an unsupervised manner. Each network independently clusters the input. We then compare two input data points. If all of the networks agree that these two data points belong to the same cluster, we can be reasonably sure that these data points belong to the same class. In this way, we identify all input data pairs belonging to the same class with high precision in completely unsupervised manner.
In the next step, we use these high quality input pairs generate similarity graph, with the data points as nodes and edges between data points which are deemed to be similar by our ensemble. From this graph, we extract tight clusters of data points, which serve as pseudo-labels. Note that, in this step, we do not cluster the entire dataset, but only small subset on which we can get high
Work done Research Fellow Microsoft Research India

---

Published as a conference paper at ICLR 2020
precision. Extracting high quality clusters from this graph while ensuring that the extracted clusters correspond to different classes is challenging. We discuss our approach in Section bection|4.2.11 for solving this problem. In this way, our method extracts unambiguous samples belonging to each class, which serves as pseudo-labels for semi-supervised learning.
For semi-supervised learning using the labels generated above, one could use ladder networks Rasmus et al.] (2013). However, we found that ladder networks is unsuitable for the initial unsupervised clustering step as it can degenerate outputting constant values for all inputs in the absence of unsupervised loss. To enable unsupervised clustering, we augment ladder networks using information maximization Krause et al. (2010) to create the Ladder-IM, and with a dot product loss to create Ladder-Dot, We show in Section S that Ladder-IM and Ladder-Dor Dote by themselves. also provide improvements over previous state of the art. We use the same models for both the first unsupervised learning step as well as the subsequent pseudo-semi-supervve
iterations.
Finally, the approach of finding high quality clusters using an ensemble, and using them as labels to train new ensemble of semi-supervised models, is iterated, yielding continued improvements. The large gains of our method mainly come from this iterative approach, which can in some cases, yield upto 17% gains in accuracy over the base unsupervised models (see section 5.4). We name our aseudo-semis semi-supervised learning approach Kingdra Kingdra is independent of the type of data set; show examples of its use both image and text data sets in Section 5 This is in contrast to some previous approaches using CNNs, e.g. Chang et al. (2017). Caron et al. 2018), which are specialized for image data sets.
We perform unsupervised classification using Kingdra on several standard image (MNIST, CIFARIO, STL) and text (reuters, 20news) datasets. On all these datasets, Kingdra is able to achieve higher clustering accuracy compared to current state-of-the-art deep unsupervised clustering techniques. For example, on the CIFARIO 10 and 20news datasets, Kingdra is able to achieve classification accuracy of 54.6% and 43.9% respectively, delivering 8- 12% absolute gains over state of the art results Hu al. (2017); Xie al. (2016).
2 PRIOR WORK ON GENERATING PSEUDO-LABELS
Model + Pscudo-label Method
MNIST
CIFARIO
Tabel Acc.(%)
Cluster Acc. (%)
Label Acc. (%)
Cluster Acc. (%)
Ladder-IM Argmaa (Lee
2013)
95.4
95.4
49.0
49.0
Ladder-IM. IM + K means (Cron el al, 2018))
75.4
60.9
45.3
44.8
Ladder-IM + Threshold Chen
2018
88.6
91.6
60.5
47.4
Table 1: Pseudo-label and clustering accuracy
Several techniques have been proposed in the literature for generating pseudo-labels (Caron et al. (2018): Chen (2018): Lee (2013). Lee (201)). the output class with the highest softmax value (Argmax) taken to be the pseudo-label. In Caron et al. (2018), the authors perform K-means clustering on the feature vector and use the K-means clusters as pseudo-labels, Finally, authors in Chen (2018) treat the softmax output as confidence and only label those items whose confidence value is above a high threshold. Note that none of these techniques for identifying pseudo-labels have been applied in our context, i.e., for unsupervised clustering using semi-supervised models.
In this section, we evaluate if lseudo-labels created by these prior techniques can be leveraged by semi-supervised models to improve clustering accuracy. We start with a ceiissuereiseed model based on Ladder networks (Rasmus et al. (2015)) called Ladder-IM (see Section 4.1 for model details) and train using only its unsupervised loss terms on MNIST and CIFARIO datasets. We use each of the above three pseudo-labelling approaches on the trained model to provide an initial set of pseudo-labed to the datasets (e.g., using K-means clustering on the feature vector of the model as in Caron et al. 2018), etc.). We call the accuracy of these lscudo-labels the initial pseudo-label accuracy. We then use these generated pseudo-labels along with the datasets to train the model again,
Our system named after semi-pseudo Pokémon.
2

---

Published as a conference paper at ICLR 2020
now with supervised loss term (based on the pseudo-labels) and the earlier unsupervised loss terms. We again run the beeudo-labelling approaches on the newly trained model to derive an updated set of pseudo-labels. We iterate this process of training and pseudo-labelling until the pseudo-label accuracy stabilizes. We call this the final clustering accuracy.
The initial pseudo-label accuracy and the final clustering accuracy results for the three approaches are shown in [able [ First, consider MNIST. The unsupervised clustering accuracy of Ladder-IM is 95.4%. Argmax simply assigns pseudo-labels based on the model's output and since this doesn't t add any new information for subsequent iterations, the final accuracy remains at 95.4%. On the other hand, the pseudo-labels identified by both the K-means and threshold approaches result in worse initial label accuracy (75,4) 45 and 88.6%). When this low-accuracy pscudo-label is used as supervision to train the model further, results a low final clustering accuracy of 60.9% and 91.6%, respectively. CIFAR10 results are similar. Ladder-IM clustering accuracy is 49% which remains the same under Argmax as before. Pseudo-label accuracy using the K-means approach worse and results in pulling down the final accuracy to 44.8%. Interestingly, threshold results in slightly higher initial accuracy of 60.5% but even this is not high enough to improve the final clustering accuracy for CIFAR10.
From these results, we arrive at the following two conclusions. First, if the initial pseudo-label accuracy is not high, using pseudo-labels supervision result in bringing down the final clustering accuracy. Thus, high accuracy of initial pseudo-labels is crucial for improving clustering accuracy. Second, current approaches for identifying pseudo-labels do not deliver high accuracy and hence are unable to help improve overall clustering accuracy.
RELATED WORK
Various unsupervised clustering methods have been proposed over the years. Ng et al. (002)) uses spectral clustering based approach, while Elhamifar & Vidal (2009) uses a sparse subspace approach for unsupervised learning. Recently, several deep neural networks based methods have been proposed, which scale well to large datasets. The ability of deep neural networks to learn higher level representations make them good choice for unsupervised learning. Coates& & Ng (0122) and
Unsupervised clustering:
al. (2018) use convnets and k-means for clustering. Caron et al. (2018) for example, iterates over clustering the features obtained from a convnet, and training the classifier using these clusters as pseudo-labels. The authors do not report clustering perfomence and we observed that this method can easily degenerate. [hang et al. (017)) uses pair-wise dot-product based similarity to identify close input pairs, which provide supervisory signal. These convnets based approaches however work on only image datasets. Xie et al. (2016) simultaneously learns feature representations and cluster assignments using deep neural networks, and works on both image and text datasets. Hu et al. (2017) uses regularization combined with mutual information loss for unsupervised learning and achieves state of the art results. The authors conduct experiments in two settings Random Perturbation Training and Virtual Adversarial Training. Other works such HHHeelm jet al. (2018) using mutual information, maximize the mutual information between the spatial features and the non-spatial features. Ji et al. (2019) maximizes the mutual information between the predicted label of the image and the predicted label of the augmented image. This method uses convolution networks and requires domain knowledge of the dataset.
Self-supervieed learning: Another form of unsupervised learning uses auxiliary learning tasks for which labels can be self generated generate useful representations from data. Many methods use spatial information of image patches to generate self suff-uueee data. E.g. Pathak et al. (2066)
predicts pixels in an image patch using surrounding patches, while Doersch et al. (2015) predicts the relative position of image patches. [Sermanet et al. (2018) uses time as self supervisory signal between videos taken from different view points. Temporal signal is also used to learn representations from single videos by predicting future frames, e.g. Denton et al, al. (2017). Our method uses correlation between outputs of input points across an ensemble as a supervisory signal to generate self-supervised pseudo-labels.
Semi-Supervised learning: cemi-supereised approaches use sparse labelling of datapoints. Szum mer & Jaakkola (2022) propagates labels based on nearest neighbors. /Weston et al. (2012) uses a deep
version of label propagation. Lee (2013) adjusts labels probabilities, starting with trusting only true labels and gradually increases the weight of pseudo labels. |Ramuus al. (2015) employs denoising
3

---

Published as a conference paper at ICLR 2020
auto encoder architecture and have shown impressive performance. Tarvainen & Valpola (2017) uses an averaged model over previous iterations as a teacher. Other than these, some semi-supervise learning methods like Xie et al. (2019) and Berthelot al. (2019) use data augmentation and assume some domain knowledge of the dataset with some of the data augmentation specific to image datasets. Miyato et al. (2018) and (Shinoda et al. (2017) uses virtual adversarial training combined with the classification loss to perform semi-supervise classification. However, we found that these methods do not work well if we jointly train them with unsupervised losses. Ladder networks does not require any domain-depended augmentation, works for both image and text datasets, and can be easily jointly trained with supervised and unsupervised losses. Thus, we chose to work with Ladder networks in our experiments, though our approach is general enough to work with any superviee method that accommodates training with unsupervised loss terms.
Unsupervised ensemble learning has been mostly limited to generating a set of clusterings and combining them into final clustering. HHunn et al. (2016) cast ensemble clustering into binary linear programming problem. Wang et al. (2009); Fred & Jain (2005) use a pair wise co-occurrence approach to construct a co-association matrix and use it to measure similarity between data points. See Vega-Pons & Ruiz-Shulcloper (201T) for survey of ensemble clustering algorithms. Note that to the best of our knowledge none of the ensemble clustering algorithms employ semi-supervised step like ours, or make use of deep networks
Unsupervised ensemble learning:
4 PROPOSED FRAMEWORK
Generated Clusters
Emsuuble of Models
Constructed Graph
3
6
5
5
7 7 7 7
M,
M,
6
2
6
6
4
Output
M,
M
6
2
5
4
O
9
3
4
2
1 9 5
6
2
/
8
8 9
/
2
5
0
6
6
6
7
O
1
6
3
6
3
7 8 2
O
Unlabeled Samples
Pscudo Labeled Samples
3
7 4
6
6
2
Inper
2 9
3
3
Pruned High Confidence Clasters
Uninbeted daRaSEt
Figure 1: Kingdra overview. In step 1, we train all the models using the unlabeled samples. In step 2 we construct a graph modeling pairwise agreement of the models. step 3, we get & high confidence clusters by pruning out data-points for which the models do not agree. In step 4 we take the high confidence clusters and generate pscudo labels. In step 5 we train the models using both unlabeled samples and pseudo labeled samples. We iterate step 2 to step 5 and final clusters are generated.
An overview of the Kingdra method is summarized in Figure I Given an unlabelled dataset X (ri,..., Jn we start with unsupervised training of an ensemble of models M (M,.... Mn ). For the individual models, any unsupervised model can be used. However, we propose a novel Ladder-* model, in which we build ladder networks Rasmus et al. (2013) and modify it to support clustering. Next, we use the agreement between the ensemble models on pair of data points, as measure of similarity between the data points. This pairwise data used construct similarity graph, from which we extract h tight clusters of data points, which serve as pseudo-labels. Note that, in this step, we do not cluster the entire dataset, but only a small subset on which we can get high precision. This is important for improving the accuracy of our semi-supervised training, as discussed in section 2 These pseudo-labels then serve as training data for semi-supervised training of new ensemble of Ladder-* models. Finally, we perform multiple iterations of the above steps for continued improvement.
4.1
BASE MODEL
The first step of our method is unsupervised training of an ensemble of models. Our framework allows using any unsupervised method for this step, and we have experimented with existing approaches, such IMSAT Hu al. (2017). The accuracy of this base model directly impacts the accuracy of
4

---

Published as a conference paper at ICLR 2020
our final model, and thus using accurate base model clearly helps. In that light, we have also developed a novel unsupervised model Ladder- ", which outperforms other unsupervised models in most data sets.
Ladder networks Rasmus et al. (2015) have shown great success in semi-supervised setting. However, to the best of our knowledge, the ladder architecture has not been used for unsupervised clustering. One reason perhaps is that ladder networks can degenerate to outputting constant values for all inputs in the absence of supervised loss term. To circumvent this degeneracy, we add an unsupervised loss to the regular ladder loss terms so that it directs the network to give similar outputs for similar inputs, but overall maximizes the diversity in outputs, so that dissimilar inputs are directed towards dissimilar outputs. We achieve this objective by incorporating one of two losses the IM loss Krause et al. 2010); Hu et al. (2017) or the dot product loss Chang et al. (2017). We call the two variants Ladder-IM and Ladder-Dot respectively.
IM loss: The IM loss or the information maximization loss is simply the mutual information between the input X and output Y of the classifier:
((X;)) = H(Y H(IX))
(1)
where H-) and H(I-)) are the entropy and conditional entropy, respectively. Maximizing the marginal entropy term HY),, encourages the network to assign disparate classes to the inputs, and thus encourages a uniform distribution over the output classes. On the other hand, minimizing the conditional entropy encourages unambiguous class assignment for a given input.
Dot product loss:
The dot product loss is defined to be
X,,, Y, if
which forces the network outputs for different inputs to be as orthogonal as possible. This has a similar effect to IM loss, encouraging the network to assign disparate classes to the inputs.
Among Ladder-IM and Ladder-Dot, we found Ladder-IM to perform better than Ladder-Do Dor in most cases. However, we did find that Ladder-Dot along with Kingdra iterations outperforms when the data set has a large imbalance in the number of samples per class. The reason for this is that the dot product loss is agnostic to the number of samples per class, while the marginal entropy term in the IM loss will drive the network towards overfitting a class with more samples, compared to class with less number of samples. A more detailed presentation of Ladder-IN IM and Ladder-DD Dot can be found in the appendix.
4.2 UNSUPERVISED ENSEMBLING
Kingdra exploits an ensemble of Ladder- models to further improve the performance of unsupervised learning. Note that, in supervised learning, ensembling is trivial as we can simply average the outputs of the individual models or do voting on them. On the other hand, in unsupervised learning, it is not trivial to do voting, as in the absence of training labels there is no stable class assignment for outputs across different models, and thus we do not have any mapping of class IDs of one model to another. To solve this we propose a simple approach, where we look pairs of data-points, rather than at individual samples. Two data-points are in the same cluster with high confidence if majority (or all) of the models the ensembles put them in same cluster. For example, given an input pair x, xr, (x: = Mx/1)) for enough models, we can say with high confidence that they belong to the same class. Using this pairwise approach, we propose a graph based method to find small sized, but high precision clusters.
4.2.1 GRAPH BASED MINI-CLUSTERING
We construct a graph G = (X, Eposs Enegy } with nodes where each input data-ppinn is represented as node. Here Epos and Eneg are two types of edges in the graph:
Strong Positive Edges: A strong positive edge is added between two data-points when large number of models agree their predicted class. (I, E Epos
n agree(z, xf) 2 tpos where tyoss is a chosen threshold, and n n_agreee(z,2)) = km E M, M, m(x)))
5

---

Published as a conference paper at ICLR 2020
Strong Negative edges: A strong negative edge is added between two data-points when a large number of models disagree on their predicted class. (a, 2/) E Eneg > n disagree(x; (z, .z) 2 Eneg- where Eneg is chosen threshold, and 21) {m m E M, m(x) # m[x()]-
strong positive edge between two data points, implies that most models believe they are in the same class, while strong negative edge between two data points implies that most models believe they should belong to different classes.
Algorithm 1 Get high precision clusters using ensembles
1: procedure GETCLUSTERSS(X, k)
2: G= Eoos: Emeg)
3:
for W E (1,2,....,A k) do
4: Imax ar x1) E Eposl fz (I,Tmaa)
5:
Imax)
Epos} U {Imeo)
6:
for I' E do
7:
Remove x' from G. if (maz) & Eneg end for end for
8:
9:
10: Return S = S2..... Ss}
11: end procedure
After building the graph, each clique of strong positive edges would be cluster, where within clique, data-points belong to the same class with high confidence. Since we add only high confidence edges to the graph, the number of cliques can be much larger than k. Hence we need to select k cliques where we would like to maximize the size of each clique, but also require that the cliques are diverse (in order not select two cliques with data-points belonging to the same class). Hence, within a clique, nodes should be connected by strong positive edges, while across cliques, nodes should be connected by strong negative edges, As finding cliques not solvable in polynomial time, we use a simple and efficient greedy approximation algorithm, as shown in Algorithm
Rather than finding cliques, we greedily find nodes with the highest number of strong positive edges (line 4). The intuition is that most of the neighbours of that node will also be connected with each other. In the case of Cifar- 10, we find that with a threshold of 90%, 81% of nodes are fully connected with each other. If the threshold is 100%, all nodes in cluster are connected with each other by transitivity. We take the node with highest number of strong positive edges, along with other nodes connected to it by strong positive edges and add them to cluster (line 5). We then remove all the nodes that do not have a strong negative edge to the chosen node (line 6-7). The intuition here is that these nodes are not diverse enough from the chosen cluster (since some models think that they belong to the same class as the currently chosen node), and thus should not be part of the next set of chosen clusters. By repeating the process k times, we get diverse clusters, approximately satisfying our requirement.
4.3 ITERATIVE ENSEMBLE TRAINING
Once the high precision clusters are identified, we treat these clustered points (points in set S) as lseudo-labels, and solve our unsupervised clustering problem using semi-supervised method. Although any semi-supervised method can be used, as described section [.1 we use the proposed Ladder= * method, which we found superior to ladder networks in our experiments.
Instead of training a single semi-supervised model, we train an ensemble of models, and again use them to find high quality clusters. This approach can be iterated, yielding continued improvements. We name this approach Kingdra. Algorithm 2 describes the complete Kingdra algorithm. First, the individual models are trained using only the unsupervised Ladder- * loss (lines 1-4). Then, for each of the iterations, we obtain high precision clusters (line 6), derive pseudo-labels from them (line 8), and then train the models with both the unsupervised and supervised losses (lines 9-1
We compute the labuds-labels using the mini-clusters as follows. For a model M, E M and clusters S, we need find appropriate mapping of the clusters the output classes of the model. In

---

Published as a conference paper at ICLR 2020
Algorithm 2 Kingdra: Iterative Ensemble Training
Input: : Dataset Models M, ham_cluuters &
Output Cluster Labels
1: for (1,2.....m) do
2: Initialize weights of M,
3: Update Mj by minimizing losg
4: end for
5: for it E .1,2....,, n_iterr) do
6: S GetCClsttrs(XX, k)
7:
for jE (1,
do
8:
Get pseudo labels for Mj Update M, by minimizing: lossLadder= + up
9:
10:
b. Use pseudo labels for los
end for end for
12:
Use averaging on the ensemble models to return final clusters
particular, for cluster Sr E S, we assign all data- in St the following label:
v mode(( a' S')).
That is, we map a cluster to the output class to which most data-points in the cluster are mapped. These pseudo-labels are then used for computing the supervised loss of Ladder- This iterative approach leads to a continuous improvement of clustering quality. We observe that the size of clusters returedd by Algorithm| iincreases after each iteration until they cover almost the entire input set. The clustering performance of the model also generally improves with each iteration until it saturates, as show in Section [5] We also note that cluster assignments become
stable with subsequent iterations, which also leads to decrease in variance across multiple runs. That is, the variance across multiple runs decreases if we run Kingdra for more iterations.
EXPERIMENTS
In this section we evaluate the performance of Kingdra on several popular datasets. For a fair comparison, we use the same data pre-processing and same model layer sizes prior work Hu et al.) (2017).
5.1 DATASETS
We evaluate Kingdra on three image datasets and two text datasets: MNIST a dataset of 70000 handwritten digits of 28-by-28 pixel size. Here, the raw pixel values are normalized to range 0-1 and flattened to vector of 784 dimensions. CIFAR10 is a dataset of 32-by-32 color images with 10 classes having 6000 examples each. STL is a dataset of 96-by-96 color images with 10 classes having 1300 examples each. For CIFARIO and STL raw pixels are not suited for our goal as the color information dominates, hence as mentioned in Hu et al. (017)), we use features extracted from Resnet-50 network pre-trained on the ImageNet dataset. Reuters is dataset containing English news stories with imbalanced data and four categories. We used the same pre-pr pre-processing as used by Hu et al. (2017); after removing the stop-wood words, tf-idf features were used. 20News is dataset containing newsgroup documents with 20 different newsgroups. Similar to Hu et al. (2017). we remove stop words and keep 2000 most frequent words, and used tf-idf features. All our experiments were performed using the same pre-process pre-processed data.
5.2 EVALUATION METRIC
We use standard unsupervised evaluation methodology and protocol to compare different methods. Following Xie al, (2016), set the number of clusters the same the number of ground truth
7

---

Published as a conference paper at ICLR 2020
Method
MNIST
STL
CIFARTO
Reuters
20news
--means
53.3 (0.1)
(0.2)
(0.4
53.7 (0.4)
14.0
AC
62.1 (0.0)
82.2 (0.0)
(0.0)
(0.9)
18.6 (0.0)
d-E+K-means
67.7 (3.0)
20.8 (1.9)
45.2
33.7 (0.2)
7.9 (0.1)
dVAE+K -eaas
65.2 (3.4)
60.8 (1.9)
44.2 (0.2)
(1.4)
12.2 (0.2)
DEC Xie et al. (2016)
84.3
78.1 (0.1)
46,9 (0.9)
67.3 (0.2)
30.8 (1.8)
DeepCluster Caron et all (018))
(1.7)
(3.9)
37,2
43.1 (4.3)
15,8 (1.2)
Deep RIM Hu et al.
58.5 (3.5)
92.5 (2.2)
40.3 (3.5)
62.3 (3.9)
25.1 (2.8)
IMSAT (RPT) Hu et al. 2017)
89.6 (5.4)
92.8 (2.5)
45.5 (2.9)
71.9 (6.5(
24.4 (4.7)
IMSAT (VAT) Hu al. 2017)
98.4 (0.4)
94.1 (0.4)
45.6 (0.8)
(4.9)
31.1 (1.9)
LADDER-IM IM (ours)
(2.8)
90.7 (1.8)
49.5 (2.9)
68.2 (2.8)
38.4 (2.5)
HADDER-IM-ensee
(ours)
95.1 (0.4)
91.5 (0.3)
51.5 (0.9)
69.0 (3.4)
40.5 (0.6)
LADDER-DOT (ours)
(7.2)
76. (4.7)
(1.0)
66.6 (4.8)
25.6 (1.3)
KINGDRA-LADDER DOT (ours)
98.0 (0.01)
93.5 (1.4)
54.3 (2.5)
71.9 (3.4)
28.4 (1.2)
KINGDRA-LADDER (ours)
98.5 (0.4)
95.1 (0.1)
54.6 (0.9)
70.5 (2.0)
43,9 (1.4)
Table 2: Comparison of clustering accuracy (%) five benchmark datasets. Averages and standard deviations are reported. The results for DEC, Deep RIM, and IMSAT are excerpted from Hu et al. (0117).
classes and evaluated unsupervised clustering accuracy as:
ΣN 1f4 = p(c))
N
ACC = max
(4)
N
where and o; are the ground truth cluster label and the cluster label assigned by the model respectively. We find the best one-to-one mapping of ground truth label and model generated clusters with p ranging over all one-to-one mappings.
5.3 COMPARED METHODS
We compare Kingdra against several clustering algorithms on our datasets. Specifically, we compare against traditional clustering algorithms such as K-Means and Agglomerative Clustering(AC). We also compare against representation learning baselines where we use models such as Deep Auto- encoders(dAE) Deep Variational Auto-encoders (dVAE). and then use K-Means on the learned representations. Finally, also compare our model with deep learning based clustering methods such as Deep RIM, DÉC, DeepCluster, and IMSAT. Deep RIM uses a multi-layer neural network with the RIM objective. DEC iteratively learns lower dimensional feature representation and optimizes a clustering objective. We also compare with two versions of IMSAT (MPT) and (MSA where data augmentation is used to impose invariance in the model outputs. For our results, we report the performance of Ladder-IM and Ladder-Dor individually, and finally Kingdra that includes an ensemble of Ladder- * networks, along with the supervied iterations. For fair comparison we used the same network architecture for all the neural network based models.
5.4 EXPERIMENTAL RESULTS
Accuracy results of prior approaches and ours is shown in Table As can be seen from the table, Ladder-IM by itself delivers good performance and Kingdra-Ladder-IM achieves higher clustering accuracy than state-of-te the-an deep unsupervised approaches such as DEC Xie al.] (2016) and IMSAT Hu et al. (217)) in all five data sets. Further, the gap between Kingdra and prior approaches is significant two data sets: Kingdra-Ladder- compared to 45.6% for IMSAT and 46.9% for DEC an 8% increase in absolute accuracy. Similarly, Kingdra-LLadder-Ie achieves an average accuracy 43.9% for 20news compared to 31.1% % for IMSAT and 30.8% for DEC an increase of over 12% in absolute accuracy. Note that while deep networks are state-of-the-ar for most data sets, linear approaches outperform deep approaches on 20news with linear RIM achieving 50.9% accuracy Hu et al. (2017). We also tried DeepCluster ChepClusterCaaoon Caron jet al. (2018) in our experimental setting, but observed the model to degenerate, assigning most of the samples to the same cluster. Additional analysis of DeepCluster is in the Appendix.
achieves an average accuracy of 54.6% for CIFARIO
8

---

Published as a conference paper at ICLR 2020
Clustering acccuracy iterations
Number of pseudo labels iterationn
13000
12:00
11:00
05
10:00
9000
STL10
8000
91
16
12
34
7000
80
CIRARID (Compicte)
50000
CIFARLD (Pseudo labels)
65:00
40c00
70
15:00
30200
23:00
20200
15:00
CIFAR10
30000
150
50000
13200
50000
45000
MNST (Canplens)
MNIST (Prsude-asben)
40000
MNIST
Figure 2: The left graph shows clustering and pseudo-label accuracy iterations for STL, CIFARIO. and the MNIST datasets. The right graph shows the number of pseudo-labels vs iterations.
An interesting aspect to note is that the use of an ensemble by itself only provides small gains of 1-2%, similar what expects from ensembles in supervised learning (e.g., compare Ladder-IM with Ladder-IM-enen MMndeebbMM-nss.bll).. The large gains mainly come from Kingdra using the ensemble to generate pseudo-labels, which then iterated. For example, Kingdra-Lauder-IM provides absolute gains of 4-6% in most data sets over the base model. Similarly. Lingdra-Ladder-i Dot provides absolute gains of 9% in MNIST and 17% in STL over the base Ladder- -ot model. Thus, our approach of generating pseudo-labels from ensembles is a powerful approach that delivers large gains in unsupervised learning.
Also note that Kingdra-Ladder-IM performs better than Kingdra-Ladder-e for most data sets except for the Reuters data set where the latter performs better (Reuters has a large class imbalance with the largest class representing 43% of the data).
Finally, note standard deviation of the various approaches shown in the Table. One can see that Kingdra in general results in lower standard deviation than many of the prior approaches even while delivering higher accuracy.
Figure shows the accuracy of pseudo-labels and Kingdra-Ladder-IM. as well number of pseudo- labels identified by the graph clustering algorithm vs the number of iterations for STL, CIFAR I0, and MNIST datasets. As iterations progress, the accuracy of pseudo labels decreases more pseudo- labels get added; however, this still helps improve the overall clustering accuracy. Note that, unlike pure semi-supervised approaches which use small of (randomly sampled) data points that match the input data distribution, our pseudo-labels do not completely match the input data distribution (since our selection algorithm is biased towards easy data points). This causes an increased gap between the accuracy of pscudo-labels, and that of overall clustering.
5.5 QUALITATIVE ANALYSIS
Figure 3 shows the similarity graph obtained after the first three iterations of Kingdra on the MNIST dataset.As the iteration progresses, one can see that there are fewer inter-cluster linkages indicating that the models are converging on the labels for these data points. Figure 4 shows randomly selected examples from our final clusters generated by Kingdra. One can see that the examples are highly accurate for MNIST, thus resulting high overall accuracy. However, for CIFAR10, there are several incorrectly labelled examples, including two clusters which do not have a clear mapping with any ground truth class, thereby resulting in much lower overall accuracy.

---

Published as a conference paper at ICLR 2020
Iteration
Iteration
Iteration 3
Figure 3: Similarity graph from sample of the input data points, obtained during three iterations of Kingdra for MNIST. The strong positive edges are shown by black lines and grey lines indicate that at least two models think they belong to the same class. The different colors show different true class labels.
2
8
8
8
8
8
8
horse
6
6
6
6
6
6
6
6
6
6
Ce
6
car
7
7
7
7
7
7
7
7
7
7
2 7
frog
9
9
5
9
9
9
dog
5 5
5
6 5
5
5
5 5 S S S 5
truck
/
/
ship
2
2
2 2 / 2 2 2 2 2 2 3 2 Q
plane
O
O O O 0 O
O 3
0 0
bird
3 3
3 3 3 3
3
3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4 4
4
unk
Figure 4: Examples of randomly selected images obtained from our final clusters for MNIST and CIFAR10 datasets. The images with incorrect class associations are identified by red boxes.
CONCLUSION
In this paper, we introduced Kingdra, novel sseudo-semi-superess learning approach for clustering. Kingdra outperforms current state-of-the-art unsupervised deep learning based approaches, with 8-12% gains in absolute accuracy for CIFARIO 20news datasets. As part Kingdra, we proposed clustering ladder networks, Ladder- IM and Ladder- Dot, that works well in both unsupervised and semi-supervised settings.
DISCUSSION
While Kingdra performs well in the datasets we studied, the similarity- graph clustering algorithm used has difficulty the number of classes increase. For example, for the datasets we evaluated, the fpos and fneg can be simply set to the number of models in the ensemble. However, as the number of classes increase, these thresholds may need some tuning. For CIFAR100, with 100 classes, our graph clustering algorithm is not able to identify 100 diverse classes effectively. We are looking at improving the clustering algorithm part of future work. We also evaluating adding diversity to the models the ensemble, either via changing the model structure, size and/or through changing the standard deviation of random noise used in ladder networks.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: holistic approach to semi- supe-sspee learning. arXiv preprint arXiv: 1905.02249 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Computer Vision-ECCV 2018, pp. 139-156. Springer, 2018.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 5880- 5888. IEEE, 2017.
Nicholas FY Chen. Pseudo-labels for supervised learning on dynamic vision sensor data, applied to object detection under ego-motion. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 644-653, 2018.
Adam Coates and Andrew Ng. Learning feature representations with k-means, In Neural networks: Tricks of the trade, pp. 561-580. Springer, 2012.
Emily L Denton al. Unsupervised learning of disentangled representations from video. In Advances in Neural Information Processing Systems, pp. 4414 4423, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.
Ehsan Elhamifar and René Vidal. Sparse subspace clustering. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 2790-2797. IEEE, 2009.
Ana LN Fred and Anil Jain, Combining multiple clusterings using evidence accumulation. IEEE Transactions Pattern Analysis & Machine Intelligence, (6):835-8500 2005.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint / 066706670, 2018.
Karan Grewal, Phil Bachman, Adam
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self augmented training. In ICML, 2017.
Dong Huang, Jianhuang Lai, and Chang-Doon -oog Wang. Ensemble clustering using factor graph. Pattern Recognition, 2016.
2016.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference Computer Vision, pp.
2019.
Andreas Krause, Pietro Perona, and Ryan G. Gomes. Discriminative clustering by regularized information maximization. A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp- 775-783. Curran Associates, Inc., 2010.
D. Lafferty, C. Williams, Shawe-Taylor, R. S. Zemel, and
Dong-Hyum Lee. Pseudo-label: The simple and efficient sumi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, pp. 2013.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 4188):1979-1993, 2018.
Andrew Ng. Michael I Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In Advances neural information processing systems, pp. 849-856. 2002.
11

---

Published as a conference paper at ICLR 2020
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536-2544, 2016.
encoders:
Mohammad Pezeshki, Linxi Fan, Philemon Brakel, Aaron Courville, and Yoshua Bengio. Decon- structing the ladder network architecture. International Conference on Machine Learning, pp- 2016.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances Neural Information Processing Systems, pp. 3546- 3554, 2015.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastiv networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
Saki Shinoda, Daniel Worrall, and Gabriel Brostow. Virtual adversarial ladder networks for semi-supervised learning. arXiv preprint arXiv:1711.07476, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15()):1929-198, 2014.
Martin Szummer and Tommi Jaakkola. Partially labeled classification with markov random walks. In Advances in neural information processing systems, pp. 945-952, 2002.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve supersuper deep learning results. In Advances in neural information processing systems, pp. -195-1204, 2017.
Sandro Vega-Pons and José Ruiz-Shulcloper. A survey of clustering ensemble algorithms. Interna- tional Journal of Pattern Recognition and Artificial Intelligence, 23(03):337-372, 2011.
Xi Wang, Chunyu Yang, and Jie Zhou. Clustering aggregation by probability accumulation. Pattern Recognition, 42(5):668-6)) 675, 2009.
Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi- supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639-655. Springer, 2012.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-487, 2016.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoe Le. Unsupervised data augmentation. arXiv preprint 1884v.1904.12848, 2019.
A APPENDIX
B Ladder-** #. LADDER NETWORKS FOR CLUSTERING
We now describe the Ladder-* * architecture for the individual models in the ensemble. We use the same model architecture for both unsupervised learning in the initial step, and the subsequent semi-supervised learning iterations, the only difference being that the semi-supervised models carry an extra supervision loss term. Our architecture augments ladder networks Rasmus al. (2015) with one of two losses an information maximization loss similar to the RIM method described in Krause et al. (2010): Hu et al. (2017), or a dot product loss Chang al. (2017). We call the two variants Ladder-IM and Ladder- Dot, respectively. We first briefly describe the RIM method and ladder networks, followed by our Ladder-IMM IM and Ladder-Dot methods.
12

---

Published as a conference paper at ICLR 2020
REGULARIZED INFORMATION MAXIMIZATION (RIM)
The Regularized Information Maximization (RIM) approach for unsupervised learning was introduced in Krause et al. (011)) and extended by Hu et al. (2017) for multi-dimensional setting. The RIM method minimizes the following objective for a classifir:
R(0) XTXXY)
(5)
where R()) is a regularization term, and / (X; Y) is the mutual information between the input X and output Y of the classifier. The mutual information can be written the difference between marginal entropy and conditional entropy Hu et al. (0017):
T(X;) HYY H(|I))
where (() and (.I.) are entropy and conditional entropy, respectively. Maximizing the marginal entropy term HY)), encourages the network to assign disparate classes to the inputs, and thus encourages a uniform distribution over the output classes. On the other hand, minimizing the conditional entropy encourages unambiguous class assignment for a given input. In the unsupervised setting, where other priors are not known, this loss makes intuitive sense.
For the regularization loss term R((0) above, many options have been proposed. Hu et al. (017), for example, propose Self-Augmented Training (SAT) loss, which imposes invariance the outputs of original and slightly perturbed input data. The authors experimented with random perturbation (IMSAT-RPT), and adversarial perturbation (IMSAT-VAT) where the perturbation is chosen to maximize the divergence between the two outputs on the current model.
LADDER NETWORKS
Ladder networks Rasmus et al. (2015) have shown impressive performance for semi-supervised classification. They employ a deep denoising auto encoder architecture, in which an additive noise is added to each hidden layer in the encoder, and the decoder learns a denoising function for each layer. The objective function is weighted sum of supervised cross entropy loss on the output of the noisy encoder, and a squared error of the unsupervised denoising loss for all layers. Unlike standard auto-encoders, ladder networks also add lateral skip connections from each layer of the noisy encoder to the corresponding decoder layer. The additive noise acts as a regularizer for the supervised loss, while the lateral connections in the denoising decoder layers enable the higher layer features focus on more abstract and task-specific features. See Pezeshki et al. (2016) for detailed analysis.
Borrowing the formalism in Pezeshki al. 2016), ladder network with L. encoder/decoder layers can be defined as:
3, 2, 2,
(1)
(L)
Ji Encoder 83);
2 .....
(1)
(T)
V/ Encoderatean (I, 8g),
1 (1)
(L)) Decoder( (1) ..... =(L.) +5),
where Oy and 6 are the parameters for the Encoder and Decoder, respectively. The variables (())) ={(a) and =((+) are the hidden layer outputs for the clean, noisy, and denoised versions layer ke respectively. x, yi, Ti are the input, clean output and the noisy output, respectively. The objective function consists of the reconstruction loss between clean and decoded intermediate features:
denoise -5nh DL x
denoise
|| =
and supervised cross entropy loss the output of the noisy encoder (which is used only in the semi-supervised setting):
sus -E 1og (x())
Ladder-IM & Ladder-Dot
We now describe our novel Ladder-IM and Ladder- models. The unsupervised denoising loss in Equation / along with the lateral connections architecture enables ladder networks to learn useful features from unsupervised data. However, in the absence of any supervised loss (Equation 8). ladder
13

---

Published as a conference paper at ICLR 2020
networks can degenerate to the trivial solution of constant output for each encoder layer, as the decoder can then simply memorize these constants to make the denoising loss zero. Having batch normalization layers helps to alleviate this problem, but the loss function still allows the trivial solution. On the other hand, the mutual information loss (Equation 6) in RIM methods, in particular the marginal entropy term H (Y), encourages the network to assign disparate classes to the inputs.
Combining ladder networks with information maximization can fix the above degeneracy problem, while simultaneously encouraging the ladder output towards uniform distribution. We use both the clean, and noisy outputs of the ladder network for computing the mutual information loss, i.e.
Ladder-IM:
loss MI I(X;Y) I(X;Y)
where Y = (....... UNN } is the set of clean outputs, and y = .........] is the set of noisy outputs from the ladder network.
Another way of thinking about the Ladder-IM approach is completely within the RIM framework. The unsupervised ladder loss lossdenoise, can be simply thought of as the regularization term R(0) in equation 5 To that effect, we also add another regularization loss term, which is the KL divergence between the clean and noisy outputs of the ladder network encoder, i.e.
ladder K L(plū x), p(yl))))
(10)
This regularization can be thought of as a generalization of the random perturbation loss proposed in Hu et al. (0117), where the authors impose invariance on the outputs of original and randomly per- turbed imputs. Our regularization based on adding noise to the hidden layers is similar to dropout Sni vastava al. (2014), and can be thought of as adding higher level feature noise, rather than just input noise.
Thus, in the unsupervised case, this would lead to the following minimization objective:
loss Ladderre-- desosee + a ladder.
+ B Mos
(11)
In this paper, we set a and |3 to one. Finally, in the semi-supervised case, we also add the supervised cross entropy term (Equation (). as done in the original ladder networks.
Ladder-Dot: We also try a dot product loss to fix the above degeneracy problem. The dot product loss is defined to be
x,) = Y,Y, i
(12)
which forces the network outputs for different inputs to be as orthogonal as possible. This has a similar effect to IM loss, encouraging the network to assign disparate classes to the inputs.
Among Ladder-IM and Ladder-Dot, we found Ladder-IM, IM to perform better than Ladder-De in most cases. However, did find that Ladder-Dot along with Kingdra iterations outperforms when the data set has a large imbalance in the number of samples per class. The reason for this is that the dot product loss is agnostic to the number of samples per class, while the marginal entropy term in the IM loss will drive the network towards overfitting a class with more samples, compared to class with less number of samples.
Overall, we found in our experiments that Ladder-IM showed superior performance to IMSAT-RPT and IMSAT- VAT(Hu et al. (2017) on most data sets. Moreover, in pure supervisee settings also, Ladder-IM outerfoomedd vanilla ladder networks in our preliminary analysis.
C EXPERIMENTAL RESULTS
C.I IMPACT OF NUMBER OF MODELS IN ENSEMBLE
We evaluated the accuracy of KINGDRA-LADDER IM as the number of models in the ensemble was varied. MNIST accuracy with 1, 2, 5, 10, and 15 models is 95.0, 96.2, 97.4, 98.5, and 98.5 respectively. This suggests that accuracy saturates after 10 models and we use 10 models for our ensemble for all our experiments.
14

---

Published as a conference paper at ICLR 2020
Clustering acccuracy number iterations
100
80
Accuracj
20
CFAR10
MNIST
Reuters
Iterations
Figure 5: Graph shows clustering accuracy vs iterations for DeepCluster. We see that there is no improvement accuracy after the first iteration.
C.2 COMPUTATION COST
We have efficient implementation of clustering, which takes 210s for largest n 70000. On server with four P100 GPUs, CLadder-IM takes 2mins, CLadder-IM with ensemble takes Smins and Kingdra with 10 iterations takes 80mins while IMSAT(RPT) takes 5mins.
C.3 ANALYSIS OF DEEPCLUSTER
Here give an analysis of DeepCluster Caron al. (2018), explaining the shortcomings. We observed that the clustering accuracy generally decreases with iterations. This is because the pseudo- labels generated could be bad, which results in worse accuracy in the next iteration. On the other hand, our approach only uses small number high-confidence samples for pseudo-labels.
D DETAILS OF THE DATASETS
MNIST: dataset of 70000 handwritten digits of 28-by-28 pixel size. The raw pixel values are normalized to range 0-1 and flattened to vector of 784 dimensions.
CIFAR10: A dataset of 32-by-32 color images with 10 classes having 6000 examples each. Similar tqHu et al. (2017), features extracted using 50-layer pre-trained deep residual networks.
STL: A dataset of 96-by-96 color images with 10 classes having 1300 examples each. We do not the 100000 unlabeled images provided in the dataset. Similar to Hu et al, (201)), features are extracted using 50-layer pre-trained deep residual networks.
Reuters: A dataset containing English news stories with four categories : corpo- rate//ndustrial, govermment/social, markets, and economies. We used the same pre- processing as used by Hu et al. (2017). After removing the stop-words, td-idf features were used.
20News: A dataset containing newsgroup documents with 20 different newsgroups. Similar to Hu et al. (2017) after removing stop words and keeping 2000 most frequent words, td-idf features were used.
15


Reviews:
[{"review_id": "SyeYFWMb9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method where they 1) use an ensemble of networks to cluster unlabeled data and assign pairs of data points a cluster label only if all networks agree that the pair belongs to a cluster 2) use the labeled pairs to create a similarity matrix and find a \"tight\" cluster or set of points that are all very similar to each other. The paper then uses the \"labelled\" points for semi-supervised learning with a proposed ensemble of models. \n\nThe paper's method of creating high precision labels using their multi-step clustering algorithm with information measures is quite interesting. The experiment results look promising. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgjIIMDtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for unsupervised clustering. Similarly to others unsupervised learning (UL) papers like \"Deep Clustering for Unsupervised Learning of Visual Features\" by Caron et al., they propose an algorithm alternating between a labelling phase and a training phase. Though, it has interesting differences. For example, unlike the Caron et al. paper, not all the samples get assigned a labels but only the most confident ones. These samples are determined by the pruning of a graph whose edges are determined by the votes of an ensemble of clustering models. Then, these pseudo labels are used within a supervised loss which act as a regularizer for the retraining of the clustering models.\n\nNovelties /contributions/good points:\n* Votes from the clustering models to create a graph\n* Using a graph to identify the most important samples for pseudo labelling\n* Modification of the ladder network to be used as clustering algorithm\n* Good amount of experiments and good results\n\nWeaknesses:\n* The whole experiment leading to Table 1 in page 2 is unclear for me. I have trouble understanding the experiment settings. Could you please rephrase it. About initial/ final clustering for example and the rest as well. The whole thing puzzles me whereas the experiments section at the end is much more clear.\n* Lack of motivation about why using the Ladder method rather than another one. Other recent methods have better results in semi-supervised learning.\n* Algorithm 1 seems quite ad-hoc. Do more principled algos exist to solve this problem ? You could write about it and at least explain why it would not be feasible here. The sentence \"The intuition is that most of the neighbours of that node will also be connected with each other\" is unmotivated: no empirical proof for this ?\n* Related work section is too light. It is an important section and should really not be hidden or neglected.\n* In the experiments, you could add the \"Deep Clustering for Unsupervised Learning of Visual Features\"  as baseline as well even if they use it for unsupervised learning as they do clustering as well.\n* In the experiments, you use the features extracted from ResNet-50 but what about finetuning this network rather than adding something on top or even better starting from scratch. Because here CIFAR-10 benefits greatly from the ImageNet features. I know that you should reproduce the settings from other papers but it might be good to go a bit beyond. Especially, if the settings of previous papers are a bit faulty. \n* Regarding, the impact of number of models in section D of the appendix, there is no saturation at 10 models. So how many models are necessary for saturation of the performance ?\n* Minor point: several times, you write \"psuedo\".\n\nConclusion: the algorithm is novel and represents a nice contribution. Though, there are a lot of weaknesses that could be solved. So, I am putting \"Weak accept\" for the moment but it could change towards a negative rating depending on the rebuttal.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1etG5Xntr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an unsupervised learning method of clustering using semi-supervised clustering as a bridge. The method first trains an ensemble of clustering models and use the edge-level majority vote to determine a graph, and then applies rule to get partial clustering signals to feed the final semi-supervised clustering. The scheme is in an iterative fashion to further enhance the quality. I find this paper interesting and somewhat novel, with the following comments.\n\n1. In algorithm 1, is it possible that too many nodes are removed so one cannot get k clusters in the end? Though finding cliques are time consuming, have the authors conducted experiments to see the difference between the real clique finding algorithm and the greedy one proposed?\n\n2. Does the ensemble clustering step have stability issue regarding the method used? If a different clustering method is used, will the graph constructed later change drastically?\n\n3. The writing. First line of section 3, figure 4 seems to point to figure 1. Section 2 seems to have format issue at the beginning. Section 5 could be merged with section 2.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors addressed the issues raised by the reviewers, so I suggest the acceptance of this paper.
Decision: Accept (Poster)