Consensus Clustering with Unsupervised Representation Learning
Authors: Jayanth Reddy Regatti
Aniket Anand Deshmukh
Eren Manavoglu
Urun Dogan
Venue:  (2021)
Keywords: Clustering
Ensemble Learning
Representation Learning

Abstract: Recent advances in deep clustering and unsupervised representation learning are based on the idea that different views of an input image (generated through data augmentation techniques) must either be closer in the representation space, or have a similar cluster assignment. In this work, we leverage this idea together with ensemble learning to perform clustering and representation learning. Ensemble learning is widely used in the supervised learning setting but has not yet been practical in deep clustering. Previous works on ensemble learning for clustering neither work on the feature space nor learn  features. We propose a novel ensemble learning algorithm dubbed Consensus Clustering with Unsupervised Representation Learning (ConCURL) which learns representations by creating a consensus on multiple clustering outputs. Specifically, we generate a cluster ensemble using random transformations on the embedding space, and define a consensus loss function that measures the disagreement among the constituents of the ensemble. Thus, diverse ensembles minimize this loss function in a synergistic way, which leads to better representations that work with all cluster ensemble constituents. Our proposed method ConCURL is easy to implement and integrate into any representation learning or deep clustering block. ConCURL outperforms all state of the art methods on various computer vision datasets. Specifically, we beat the closest state of the art method by 5.9 percent on the ImageNet-10 dataset, and by 18 percent on the ImageNet-Dogs dataset in terms of clustering accuracy. We further shed some light on the under-studied overfitting issue in clustering and show that our method does not overfit as much as existing methods, and thereby generalizes better for new data samples. 

Under review as a conference paper at ICLR 2021
CONSENSUS CLUSTERING WITH UNSUPERVISED REP- RESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in deep clustering and unsupervised representation learning are based on the idea that different views of an input image (generated through data augmentation techniques) must either be closer in the representation space, have a similar cluster assignment. In this work, we leverage this idea together with ensemble learning to perform clustering and representation learning. Ensem- ble learning is widely used in the supervised learning setting but has not yet been practical in deep clustering. Previous works ensemble learning for clustering neither work on the feature space nor learn features. We propose novel ensemble learning algorithm dubbed Consensus Clustering with Unsupervised Representa- tion Learning (ConCURL) which learns representations by creating a consensus on multiple clustering outputs. Specifically, we generate cluster ensemble us- ing random transformations on the embedding space, and define a consensus loss function that measures the disagreement among the constituents of the ensemble. Thus, diverse ensembles minimize this loss function in synergistic way, which leads to better representations that work with all cluster ensemble constituents. Our proposed method ConCURL is easy implement and integrate into any rep- resentation learning or deep clustering block. ConCURL outperforms all state the art methods various computer vision datasets. Specifically, we beat the closest state the art method by 5.9 percent on the ImageNet-10 dataset, and by 18 percent the ImageNet-Dogs dataset in terms of clustering accuracy. We further shed some light on the under-studied overfitting issue in clustering and show that our method does not overfit as much existing methods, and thereby generalizes better for new data samples.
INTRODUCTION
Supervised learning algorithms have shown great progress recently, but generally require lot of labeled data. However, in many domains (e.g., advertising, social platforms, etc.), most the avail- able data are not labeled and manually labeling it is a very labor, time, and cost intensive task (Xiao) cct al.] 2015 [eshmukkh 2019: Mintz et al. 2009: Blum & Mitchell 1998). On the other hand, clus- tering algorithms do not need labeled data to group similar data points into clusters. Some popular clustering algorithms include k-means, hierarchical clustering, DBSCAN (Ester et al., 1996), spec- tral clustering, etc., and the usefulness of each algorithm varies with the application. In this work, deal with the clustering of images. Traditional clustering approaches focus on hand crafted features on which out of the box clustering algorithms be optimal, and are not scalable to large scale real word datasets (Wu et al.) (211).. Advancements in deep learning techniques have enabled end-to-end learning of rich representations for supervised learning. On the other hand, simultancously learning the feature spaces while clustering leads to degenerate solutions, which until recently limited end to end implementations of clustering with representation learning approaches (Caron al.] 2018). Recent deep clustering works take several approaches to address this issue such as alternating pseudo cluster assignments and pseudo super- vised training, comparing the predictions with their own high confidence assignments (Caron al.)
applied. However, hand crafted features may
2018 Asano et al. 2019 Xie et al.) 2016: Wu et al. 2019), and maximizing mutual information between predictions of positive pairs Ji et al. 2019). Although these methods show impressive per- formance on challenging datasets, we believe taking advantage of rich ideas from ensemble learning for clustering with representation learning will enhance the performance of deep clustering methods.

---

Under review as a conference paper at ICLR 2021
Ensemble learning methods train variety of learners and build meta learner by combining the predictions of individual learners (Dietterich) 2000, Breiman 1996 Freund et al. 1996). In practice they have been heavily used in supervised learning setting. Ensemble learning methods have also found their place in clustering i.e. knowledge reuse framework (Strehl & Ghosh] 2002) where a consensus algorithm is applied clusters the data better than any component partition individually. However, the knowledge reuse framework and much of the consensus clustering literature that followed (Fern & Brodley [203] Fred & Jain 2005 TTopchy et al. (2005) do not make use of the underlying features used to generate the ensemble. We propose the use of consensus clustering as a way to extend ensemble methods to unsupervised representation learning. In particular, we define a 'disagreement" measure among the constituents of the ensemble. The key motivation for this is that the diversity of the ensemble drives the minimization of the disagreement measure in a synergistic way, thereby leading to bet- ter representations. We propose Consensus Clustering with Unsupervised Representation Learning (ConCURL ) and following are our main contributions:
constituent cluster partitions generate updated partition that
1. A novel ensemble learning algorithm which learns representations by creating consen- sus on multiple clustering outputs generated by applying random transformations on the embeddings.
2. Our method outperforms the current state of the art clustering algorithms on popular com- puter vision datasets based on clustering metrics (A.4).
3. Even though there is no labeled data available while learning representations, cluster- ing may still be prone to be overfitting to the "training data." As stated in Bubeck & Von Luxburg (007). in clustering, we generally assume that the finite data set has been sampled from some underlying space and the goal is to find the true approximate partition of the underlying space rather than the best partition in a given finite data set. Hence, to check generalizability of the method proposed we also evaluate our models on the "test data" data that was not available during training/represnnattinn learning. Our method is more generalizable compared to state of the art methods (i.e. it outperforms the other algorithms when evaluated on the test set).
RELATED WORK
Clustering is a ubiquitous task and it has been actively used in many different scientific and prac- tical pursuits such as detecting genes from microarray data (Frey & Ducck (007). clustering faces medical imaging to support diagnosis (Masulli & 1999). We refer interested readers to these excellent sources for survey of these uses Jain et al.
(Rodriguez & Laio. 2014), and segmentation
Schenone
1999
Liao 2005):
Xu & Wunsch (2005); Nugent & Meila (0100).
Clustering with Deep Learning: In their influential work.[Caron et al. (2018) show that it is possi- ble to train deep convolutional neural networks with pseudo labels that are generated by a clustering algorithm (DeepCluster). More precisely, in DeepCluster, previous versions of representations are used to assign pseudo labels to the data using an out of the box clustering algorithm such as k-means. These pseudo labels are used to improve the learned representation of the data by minimizing pervised loss. Along the same lines, several more methods have been proposed. For example, Gaussian ATtention network for image clustering (GATCluster) (Niu et al. 2020) comprises four self-learning tasks with the constraints of transformation invariance, separability maximization, en- tropy analysis and attention mapping- Training is performed in two distinct steps, similar Caron et al. (2018) where the first step is to compute pseudo targets for a large batch of data and the sec- ond step train the model in supervised way using the pseudo targets. Both DeepCluster and GATCluster use k-means to generate pseudo labels which may not scale well. Wu et al. (2019) pro- pose Deep Comprehensive Correlation Mining (DCCM), where discriminative features are learned by taking advantage of the correlations of the data using pseudo-label supervision and triplet mutual information among features. However, DCCM may be susceptible to trivial solutions (Niu et al.) 2020). Invariant Information Clustering (IIC) (Ji et al. 2019) maximizes mutual information be- tween the class assignments of two different views of the same image (paired samples) in order to learn representations that preserve what is common between the views while discarding instance specific details. Ji et al. (2019) argue that the presence of an entropy term in mutual information plays important role in avoiding the degenerate solutions. However large batch size is needed
2

---

Under review as a conference paper at ICLR 2021
for the computation of mutual information in IIC. which may not be scalable for larger image sizes which are common in popular datasets (Ji et al. 2019 Niu et al. 2020). Huang et al. (2020) ex- tend the traditional maximal margin clustering idea to the deep learning paradigm, by learning the most semantically plausible clustering through minimizing a proposed partition uncertainty index. Their algorithm PICA uses stochastic version of the index, thereby facilitating mini- batch train- ing. PICA fails to assign a sample-correct cluster when that sample either has high foreground or background similarity to samples in other clusters. SCAN (Van Gansbeke et al. 2020) uses the learnt representations from a pretext task to find semantically closest images to the given image us- ing nearest neighbors. SCAN achieves oftte-of-the--eee art results on CIFAR-10, CIFAR- 100, STL-10, however using such priors from existing pretext tasks deviates from the main idea of our paper of end-to-end learning. We believe however that we can benefit by using SCAN on top of the features learnt using our algorithm to improve the clustering accuracy. Our proposed approach ConCURL. is scalable to large datasets, does not suffer from trivial solutions and shows superior performance on a challenging set of image data sets. As shown in the experimental results, our proposed method also generalizes well to data points that were not available during training, when compared to the above approaches.
Self-supervised Representation Learning: Sepf-super learning is a sub-field of unsupervised learning in which the main goal is to learn general purpose representations by exploiting user- defined sub-tasks such as the relationship between different views of the same data. Although self-supervised learning methods show impressive performance on a variety of problems, it is not clear whether learned representations are good for clustering. There are many different flavors of self supervised learning such as Instance Recognition tasks (Wu et al. 2018), contrastive techniques (Chen et al. 2020), etc. In instance recognition tasks, each image is considered as its own cate- gory so that the learnt embeddings are well separated. [Zhuang et al. (2019) propose novel Local Aggregation method based on non-parametric instance discrimination tasks. They use a robust clus- tering objective (using multiple runs of k-means) to move statistically similar data points closer in the representation space and dissimilar data points further away. However, instance recognition tasks usually require a large memory bank which is memory intensive. In contrastive learning (Tian) et al. tations are learned by maximizing agreement between different augmented views of the same data example (known as positive pairs) and minimizing agreement between augmented views of different examples (known as negative pairs). SimCLR (Chen et al. 2020) achieves state-of-the-art-resll without specialized architectures or a memory bank of negative pairs (usually required by contrastive learning techniques). However, it still requires negative examples and, as it applies to instance clas- sification, it has to compare every pair of images. These issues are addressed by Bootstrap your own latent (B YOL) (Grill et al. 2020) and Swapping Assignments between multiple Views (SwAV) (Caron et al. 2020). BYOL-does not require negative examples and SwAV does not need to compare every pair of images. For the main study in this paper we use BYOL as a representation learning block and adapt the soft clustering loss which was used in SWAV to learn prototypes, thereby ad- dressing both the issues of negative samples and need for comparing every pair of images. Note that our proposed algorithm can use any representation learning block like SimCLR, BYOL, & SWAV and can also other soft clustering loss formulations.
2019 He et al., 2020; Hénaff al. 2019 Hjelm al. 2018 Chen et al. 2020), represen-
Consensus Clustering: /ttrhll & Ghosh (2002) propose a knowledge reuse framework; to build an unsupervised ensemble by using several distinct clusterings of the same data and assuming that the underlying features that are used to compute the clustering are not available and fixed. |een & Brod- [cy] build on the cluster ensemble framework based on random projections. In this framework, Fern & Brodley (2003) show that a single run of clustering (random projection + Expectation Max- imization) highly unstable. They perform multiple runs of clustering and compute an aggregated similarity matrix which is used to cluster the data using an agglomerative clustering algorithm. Fred & Jain (205) propose a voting approach to map the cluster assignments in the ensemble into new similarity measure between clusterings. The co-association matrix thus formed can be used with any clustering algorithm to result in a new data partition. Weighted cluster ensembles (Domeniconi & Ra-gan 2009) uses Locally Adaptive Clustering (LAC) (Domeniconi et al., 2004) as a base algorithm which assigns weights to each feature of the sample according to the local variance of data along each dimension. Authors use LAC with different hyperparameters and propose three different partitioning algorithms to generate consensus among LAC clusterings. HHuang et al.] (2017) argues that most ensemble based methods treat all base clusterings equally and few which weigh the base clusterings give those weights globally. Authors propose an ensemble clustering approach based
3

---

Under review as a conference paper at ICLR 2021
on cluster uncertainty (calculated by entropic criterion) and local weighting strategy. There also survey papers which have in depth analysis and details various methods used in consensus clustering VeerrinnVeea--osss & Ruiz-Shulelope these methods can be adapted when one needs to do representation learning along with clustering. It is also not clear if one come up with end-to-end learning architecture with any of the above methods. In contrast, our proposed consensus clustering method ConCURL can easily be integrated into any deep learning architecture for clustering and trained end-to-end.
(201T); Ghosh & Acharya
It is not clear how any of
3 PROPOSED METHOD
Given set of observations X' ) the goal learn representation cluster representa- tions C' (called as prototypes henceforth) and partition N observations into K disjoint clusters (the value of K is assumed be known). To check the generalization of the algorithm, the final goal partition previously unseen Nr observations Xy [= -+++ learnt representation f and prototypes C.
into K disjoint clusters given
3.1 ALGORITHM
We propose a novel consensus loss to achieve the objective. Along with the novel consensus cluster- ing loss, the loss function comprises of two more loss terms, namely representation learning loss, soft clustering loss. The three loss terms share a common backbone (such as ResNet) and have different loss specific layers after the representation layer of the backbone. Given an input batch X, C X' of size B, we use different data augmentations in the image space, such as random horizon- tal flip, etc. (more details in A.3) to generate two augmented views Xr A2 of the input batch. fū f are representations that are corresponding to the two views the input batch. The superscript is used to identify the view of the input image. It's usage will be made clear from context, and ignored otherwise. Our algorithm is mini-batch based, and end-to-end.
3.1.1 UNSUPERVISED REPRESENTATION LEARNING
The first loss term unsupervised representation learning loss based a pretext task such as BYO (Grill et al. 2020). SimCLR (Chen et al. 2020), etc. In this work, we use BYOL which tries minimize the distance between the online and target network embeddings of the two augmented views X3 K,,, We denote this loss by L1. Optimizing for loss L1 alone generally results in good representations but may not be suitable for a clustering task.
3.1.2 SOFT CLUSTERING
For the second loss term, we use soft clustering objective. In this work, we follow the framework presented in Caron et al. (2020), which is a centroid based technique and aims to maintain con- sistency between the clusterings of the augmented views X3 and X We store a set of randomly initialized prototypes C {cI, .Kr E Rd where K is the number of clusters, and d is the dimension of the prototypes.
We two layer multi-layer perceptron g to project the features f and f to lower dimensional space (of size d). This technique was shown to be useful improving the representations of the laver before the MLP (Chen al.) 2020) and thereafter used in Grill et al. (2020) and Caron et al. 2020 The output of this MLP (referred to as embeddings) is denoted using Z [ z
and 2 = {2] ..... 2 for view and view respectively.
Soft clustering approaches based on centroids/prototypes often requires one to compute a measure of similarity between the image embeddings and the prototypes (Xie al. 2016 Caron et al., 2020). We compute the probability of assigning a cluster to image i using the normalized vectors
z!
2?
| and e,
cj as
exp( ! ( 22))
exp( ( (z?,E,))
pl.
Pi,
(1)
E, exp( 1 (7) ,,,))
E, exp( (.
4

---

Under review as a conference paper at ICLR 2021
We concisely write p 1 and P Note that, we use p, to denote the predicted cluster assignment probabilities for image (when not referring to particular view), and shorthand p is used when i is clear from context.
Here, is temperature parameter. {p,
The idea predicting assignments p. and then comparing them with high-confidence estimates codes henceforth) of the predictions is not (Xie al. 2016 While Xie et al. (2016) uses pretrained features (from autoencoder) to compute the predicted assignments and the codes, doing this in end end unsupervised manner might lead degenerate solutions. Asano et al. (2199) avoids such degenerate solutions by enforcing an equi-partition constraint (the prototypes equally partition the data) during code computation. CCron al. (2020) follow the same formulation but compute the codes in an online manner for each mini- batch. The assignment codes are computed by solving the following optimization problem
(referred
Q' arg max Tr(Q" C² z²) + cQ(()) and Q² arg max Tr(Q" (² cH()), (2)
QEQ
QEQ
where Q (q),-.
q
E R x
Q is the transportation polytope defined by
@ = QQ R+ Q1B K 1x, Q0 - 1
1
1 is a vector of ones dimension K and H(Q)= = - Σn, Qú, log Qs.j- The above optimization is computed using a fast version the Sinkhorn Sinkhor-Knopp algorithm (Cuturi 2013) as described in Caron et al. 22200.
After computing the codes Q* and Q², we compute the loss using the probabilities P-j and the assigned codes qıy by comparing the probabilities of view with the assigned codes of view 2 and vice versa, given as
1 2B
1
2,,1
q log P,ys
L2,
y log Pijy
La L2,1 2
(3)
2B
1=
Oxppal
Neverk
L2
Lg11
Lg
L2
Figure 1: The resulting ensemble of clusterings (p', q', (f(mm)) m mm,121122 is represented in the green block. L2,1, L2. represent soft clustering loss as in eqn. (3), La.1 La.2 represent consensus loss in eqn.

---

Under review as a conference paper at ICLR 2021
3.1.3 CONSENSUS CLUSTERING
The third loss is the proposed consensus clustering loss. We generate a cluster ensemble by first performing transformations on the embeddings Zz, Z² and the prototypes C. At the beginning of the algorithm we randomly initialize M such transformations and them throughout training. Suppose using particular random transformation (a randomly generated matrix A), we get 2 = Az, č = Ac. We then compute the softmax probabilities Pij using normalized vectors 8/|lž|| and è/ Repeating this with the M transformations results in M predicted cluster assignment probabilities for each view. When the network untrained, the embeddings z are random and applying the random transformations followed by computing the predicted cluster assignments leads to diverse set of soft cluster assignments.
There can be many clusterings for any given latent space (either by randomization introduced in the space or different clustering algorithms). We propose to use consensus clustering so that we can find a latent space that maximizes performance effective, we need more diversity in the clusterings among each component of the ensemble at the start and ideally the diversity should decrease with training and the clustering performance should get better. Creating such diverse ensemble leads to better representations and better clusters.
many different clusterings. To make the method
To compute the consensus loss, once the probabilities P.j are computed, we compare the codes generated using ( of view with the P of view and vice versa, given as
B
M
K ΣΣΣ ag log Pij
B
L31
ΣΣX
2 L log
Pij
(l.))
L32
(2.m)
(4)
2BM
2BM
i=
12
Lg
Lai + Lg2
(5)
The architecture for computing the ensemble and the computation of consensus loss is shown in Fig. T.. The final loss that we sought to minimize is the combination of the losses L1, L2, La
Local = aL1 + BL + LLa.
(6)
where a, 3, are non-negative constants. L minimizes the distance between the embeddings of different views of the input batch, T22 maintains consistency between clusterings of two augmented views of the input batch, and 1.3 maintains consistency between clusterings of the randomly trans- formed embeddings. During inference, to compute clustering of the input images, we use the computed assignments [q:): and assign the cluster index as C; arg max; qik for the zh data- point. The summary of the algorithm is presented A.2
3.2 CHOICE FEATURE TRANSFORMATIONS
Table 1: Different ways to generate ensembles
Data Representation
Clustering algorithms
Different data preprocessing techniques
Multiple clustering algorithms (k-means, GMM, etc)
Subsets of features
Same algorithm with different parameters or initial- izations
Different transformations of the features
Combination of multiple clustering and different parameters initializations
Fred & Jain 2005) discuss different ways generate ensemble of clusterings which are tabulated in Table I In our proposed algorithm, we focus on choosing of data representation to generate cluster ensembles.
Using different clustering methods (or multiple runs from different initializations) showed success when the goal is to aggregate the cluster partitions of the ensemble into final partition. However,

---

Under review as a conference paper at ICLR 2021
since goal is to learn representations simultaneously using back proppprppggttion it is natural to use a clustering algorithm in which we can update its parameters via back-propagation. is in- fact possible to use clustering algorithms like IIC ( al. 2019) that enable parameter updates, however, if multiple such algorithms are used to generate the ensemble, the backward computation graph consumes enormous amount of memory. For large datasets (and large architectures), such techniques may not be practical. However, by using feature transformations, the amount of such memory overhead small.
By fixing a stable clustering algorithm, we can generate arbitrarily large ensembles by applying different transformations on the embeddings. Random projections were successfully used in Con- sensus Clustering previously (Fern & Brodley] (200)). By generating ensembles using random pro- jections, we have control over the amount of diversity we can induce into the framework, by varying the dimension of the random projection. In addition to Random Projections, we also used diago- nal transformations (Hsu et al. 2018) where different components of the representation vector are scaled differently. HHsu et al. (2018) illustrate that such scaling enables diverse set of clusterings which is helpful for their meta learning task.
4 EXPERIMENTS
We evaluated our algorithm and compared against existing work on nine popular image datasets with mix of high and medium resolution datasets: ImageNet-10, ImageNet-Dogs, STL10, CIFAR-I0, CIFAR100 10000000,00000 CUB, Caltech-101, AwA2 and Intel Image classification. The dataset summary is given in Table 6 (see A.I.. The resolution column shows the size to which resized the images in our algorithm. To the best of our knowledge some of these datasets (Intel, Caltech101, CUB, and Aw) are being used for systematically evaluating clustering results for the first time (Section A. In our comparison, we considered some recent state-of-the-arr methods that directly solve for clustering objective in an end-to-end fashion from random initialization, and do not use any prior information (nearest neighbors for example) derived through other pretext tasks. The metrics for evaluation are explained in A.4
4.1 RESULTS
In Table 2. show the best accuracy get for the proposed method ConCURL and compare it against the best accuracy achieved by state-of-the-art methods. For ImageNet-10 and ImageNet- Dogs, we trained using the train split and evaluated on the train split. For STL10, CIFARIO and CIFAR-100, similar to earlier approaches (Huang et al., 2020 Niu et al.) 2020) we used both train and test splits for training and evaluation. Note that PICA also 100k points in STL In all datasets ConCURL outperforms all state-of-ther art algorithms. In the ImageNet-10 10 dataset, ConCURL is better by 5.9% and 21% compared to PICA and GATCluster respectively in terms of clustering accuracy. ConCURL is better than PICA by more than 9% in terms of NMI and by more than 11% in terms of ARI in the ImageNet- 10 dataset. We beat the next- best DCCM method by 18% in ImageNet-Dogs. We used the popular Iman and Daveport Test to calculate the p-value for statistical significance test (Garcia & Herrera 2008). Our proposed method ConCURL was consistently ranked at the top for all three metrics ACC, ARI and NMI with p-value of 1.34 x 10 4.57 x 10 and 1.45 x 10 respectively. This shows that results presented in the paper are statistically significant.
unlabelled data split of
Table 2: Clustering evaluation metries
Datasets
ImageNes-10
10
ImageNer- Dogs
$T1.10
CIFAR10
CIPAR100-20
Methode Metrics
Acc
NMI
ARI
NMI
ARI
Acc
NMI
ARI
Acc
NMI
ARI
Acc
NMI
ARI
DDCM
0.100
0.608
0.555
0.383
0.321
0.182
0.422
0.376
0,262
B663
0.46
0.408
0.327
0.285
0.173
GATClnster
0.762
0.609
0.522
0.333
0.322
0.200
0.583
0.446
0.365
0.610
0.475
0.402
0.281
0,215
0.116
PICA
0.761
0.352
0.201
0,337
ConCURL
0.922
0,877
0.852
0.452
0.447
0.288
0.728
0.634
0.554
0.720
0.608
0.520
0.385
0.377
0.223
We performed additional experiments to compare ConCURL to PICA when trained only on the "train" split for STL-10. CIFAR-10 and CIFAR-100. From Table observe that ConCURL achieves better clustering on all metrics. From Table[and] we can observe that increasing the data for training improves the performance of the algorithm. In order to understand the generalization of
7

---

Under review as a conference paper at ICLR 2021
the algorithm, in Table 4 show results for the when the models trained only on "train" split and evaluated on "test data" that was not used during training. We can observe that we perform better when evaluated on the test data.
We also perform an ablation study on the affect of the losses L1 L2, L3 (see A.6 and observe that the using consensus loss L3 almost always improves accuracy. This shows the importance of con- sensus loss (Lz) and how ensemble learning through proposed consensus helps achieving better clusters. Even though proposed method outperforms all state-of-the-ar methods there are some as- sumptions that we make which are same as all state-of-the- that "K number of clusters" are known and second assumption that the data sample is equally distributed among "K" clusters. These assumptions are important to make the methodology work because the final layer in our case needs to know the number of clusters so that we can evaluate later for fair comparison with other methods. Also, the fast Sinkhorn- Knopp al algorithm used to solve Eq. 2 may not be optimal if the data samples are not equally distributed among "K" clusters.
compared. First assumption is
Table Clustering evaluation metrics: Models trained using only train split, and evaluation on train split
Datasets
STL10
CIFARIO
CIFART00-20
Methods's Metrics
NM1
ART
Acc
PICA
0.494
0.424
0.297
0.559
0.310
0.402
0,293
0.273
0.138
ConCURL
0.623
0.514
0.428
0.705
0.545
0.507
0.366
0.351
0.195
Table 4: Clustering evaluation metrics: Models trained using only train split, and evaluation on test split of the data
Datusets
ImageNet-10
ImageNer.Doog Dogs
CIPARIO0-20
Methods' Metries
Metriis
Acc
NMI
ART
Acc
NMI
ARI
Asc
NMI
ARI
Acc
NMI
ART
Acc
NMI
ARI
PICA.
0.291
ConCURD
0.864
0840
0.700
0.455
0.477
0.274
0.611
0.498
0.410
0.693
0.527
0.888
0.363
0.354
0.193
4.2 EFFECT OF NUMBER OF RANDOM TRANSFORMATION. AND EMBEDDING SIZE
In order to study the sensitivity of the algorithm to the random transformations, we performed an ablation study for STLIO trained on the train split (Table 5 Recall that M is number of transfor- mations used in algorithm T and M = 30, yield good results for the two types of transformations (random projections and diagonal transformations). We can also observe that there isn't 't a large dif- ference between results obtained using either of the transformations. The results fluctuate with a margin of + 0.06 and still outperform the other baselines in almost all cases. In the first column of the Table [ we used diagonal transformations, and varied the number of transformations. The sec- ond column of the Table contains results with fixed dimension of random projection (=64), and varies the number of transformations. The third column of the Table [5 contains results with fixed number of transformations (=100), and varies the dimension of projection (the original embedding size is 256).
Table 5: Ablations on the number of transformations and dimension of random projections. These results were obtained by training ConCURL on train split of STL10 for 1000 epochs.
Number Diagonal Transfommtiinss
Number Random Projections Metrics
Emmbedding Dimension Size
#Trane) Metrics
Ace
NMI
ARI
Acc
NMI
ARI
Adimm Metrics
Acc
NMI
ARI
0.567
0.373
0.520
0.468
0.503
30
0.334
0.481
0.338
30
0.586
0.515
0.386
128
0.487
0.488
0,310
50
0.582
0.525
0.397
0.562
0.512
0.377
256
0.525
0.502
0.338
100
0.530
0.489
0.344
0.333
0.503
0.343
512
0.584
0.499
0,375
PICA
0.494
0.424
0.297
PICA
0.494
0.424
0.297
PICA
0.494
0.424
0.297
8

---

Under review as a conference paper at ICLR 2021
0.3
64
NMI mean
0.8
NMI std
0.2
256
128
0.6
64
512
128
0.1
0.4
256
512
0.2
0.0
0
200
400
600
800
1000
200
400
600
800
1000
epochs
epochs
(a)
(b)
Figure 2:
NMI as a way to measure the consensus/diversity in the ensemble. The result are for STL10 trained using the train split.
4.3 CONSENSUS AMONG EACH COMPONENT OF THE ENSEMBLE
We also measure the diversity the ensemble every epoch to observe the affect of consensus as training progresses. For each component of the ensemble, we use the softmax probability estimates p and compute cluster assignments by taking an arg max on p of each image. If there are M components in the ensemble,
get M such cluster assignments. We then compute pairwise NMI (Normalized Mutual Information) (similar analysis to Fern & Brodley (2003) between every two components of the ensemble, and compute the average and standard deviation of the pairwise NMI across the M(M--1) pairs. We observe from Figure 2a that the pairwise NMI increases as training progresses and becomes closer to 1. At the beginning of training, the ensemble is very diverse (small NMI score with a larger standard deviation); and as training progresses, the diversity is reduced (large NMI score with smaller standard deviation).
2
CONCLUSION
In this work, we leverage the ideas in unsupervised representation learning along with ensemble learning to perform clustering. We propose novel ensemble learning algorithm which learns representation by creating a consensus on multiple clustering outputs. Our proposed method outper- forms all state of the art methods various computer vision datasets. We also present the issue of overfitting in clustering and show that our method generalizes well on new data samples that were not available during training. This work is of the first successful applications of ensemble learn- ing in the deep clustering domain. This idea could casily be extended to use other general purpose representation and soft clustering algorithms than the ones used in this paper. A possible extension can be to leverage the knowledge reuse framework of Strehl & Ghosh (2002) and use the clusterings output by the ensemble to compute better quality partition of the input data. We believe that en- semble learning algorithms could also be effective in increasing robustness in clustering and we are planning investigate this point further.
REFERENCES
Adjusted Rand Score. //tpa///eoe/////////aaaaaaaa/aaa
http://www...aaaaoo////////aaaaa html =adj Accessed: 2020-009-23.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. arXiv preprint 1911:55 2019.
Ella Bingham and Heikki Mannila. Random projection dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference Knowledge discovery and data mining, pp. 245-250,2000
PICA trained models obtained from htpp://giinaa.coo//yyymmssssscce

---

Under review as a conference paper at ICLR 2021
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the eleventh annual conference on Computational learning theory, pp. 92-100, 1998.
Leo Breiman. Stacked regressions. Machine learning, 24(1):49-64, 1996.
Sébastien Bubeck and Ulrike Von Luxburg. Overfitting of clustering and how to avoid it. 2007.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882 2020.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.0577 055009 2020.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-225-222212 2011.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pp. 2292-2300, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Aniket Anand Deshmukh. Kernel Methods for Learning with Limited Labeled Data. PhD thesis, 2019.
Thomas G Dietterich. Ensemble methods in machine learning. International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Carlotta Domeniconi and Muna Al-Razgan. Weighted cluster ensembles: Methods and analysis. ACM Transactions on Knowledge Discovery from Data (TKDD), 2(4):1-40, 2009.
Carlotta Domeniconi, Dimitris Papadopoulos, Dimitrios Gunopulos, and Sheng Ma. Subspace clus- tering of high dimensional data. In Proceedings of the 2004 SIAM international conference on data mining, pp. 517-521. SIAM, 2004.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, volume 96, pp. 226-231, 1996.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and patlern recognition workshop. pp. 178-178. IEEE, 2004.
Xiaoli Z. Fern and Carla Brodley, Random projection for high dimensional data clustering: clus- ter ensemble approach. In Proceedings of the 20th international conference machine learning (ICML-03), pp. 186-193, 2003.
Ana LN Fred and Anil K Jain. Combining multiple clusterings using evidence accumulation. IEEE transactions on pattern analysis and machine intelligence, 27(6):33--550 2005.
Yoav Freund, Robert E. Schapire, et al. Experiments with new boosting algorithm. Citeseer, 1996.
Brendan Frey and Delbert Dueck. Clustering by passing messages between data points. science, 315(5814) 2007.
10

---

Under review as a conference paper at ICLR 2021
Salvador Garcia and Francisco Herrera. An extension ontstaiistial comparisons of classifiers multiple data sets "for all pairwise comparisons. Journal of machine learning research, 9Dec): 2677-2694, 2008.
Joydeep Ghosh and Ayan Acharya. Cluster ensembles. Mining and Knowledge Discovery, 1(4):305-315, 2011.
Wiley Interdisciplinary Reviews: Data
BastiBBatienn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena
Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-97388 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding, arXiv preprint urXiv:/905.09271 2019.
Devon Hjelm, Alex Fedorov, Samuel Mavoii-Marchilddon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1iX::1888.006611 2018.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. arXiv preprint 18X0:01100 2018.
Dong Huang, Chang-Dong Wang, and Jian-Huang Lai. Locally weighted ensemble clustering. IEEE transactions cybernetics, 48(5):1460-15 1473, 2017.
Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8849-8858, 2020.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification 2(1):193-218 1985.
Anil K Jain, Narasimha Murty, and Patrick Flynn. Data clustering: a review. ACM computing surveys (CSUR), 31(3):264 323, 1999.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9865-9874, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1eXiv:1412.6988 2014.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
T Warren Liao. Clustering of time series data survey. Pattern recognition, 38(11)11857-1874 2005.
Francesco Masulli and Andrea Schenone. A fuzzy clustering based segmentation system as support to diagnosis in medical imaging. Artificial intelligence in medicine, 16(2):129-147, 1999.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003-1011, 2009.
11

---

Under review as a conference paper at ICLR 2021
Chuang Niu, Jun Zhang, Ge Wang, and Jimin Liang. Gatcluster: Self-supervised gaussian attention -atssian-attention network for image clustering. arXiv preprint MXX::2200..1863,, 2020.
Rebecca Nugent and Marina Meila. An overview of clustering applied to molecular biology. In Statistical methods in molecular biology, pp. 369-404. Springer, 2010.
E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski. Kornia: an open source differentiable computer vision library for pytorch. Winter Conference on Applications of Computer Vision, 2020. URL /ttp:////w.vvvo.or/////
910. 021 10129.9ppp.. pdf.
Alex Rodriguez and Alessandro Laio. Clustering by fast search and find of density peaks. Science, 344(6191:11492-44499 2014.
Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Mare Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3260-3269, 2017.
Alexander Strehl and Joydeep Ghosh. Cluster ensembles knowledge reuse framework for com- bining multiple partitions. Journal of machine learning research, 3(De)))583-617, 2002.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 10X,::1906,05849, 2019.
Alexander Topchy, Anil K Jain, and William Punch. Clustering ensembles: Models of consensus and weak partitions. IEEE transactions on pattern analysis and machine intelligence, 27(12): 1866-1881, 2005.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning classify images without labels. 2020.
Sandro Vega-Pons and José Ruiz-Shuleloper. A survey of clustering ensemble algorithms. Interna- tional Journal of Pattern Recognition and Artificial Intelligence, 23(03)3337-372
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200- 201- Dataset. Technical Report CNS-TR-201 1-00010000,,, California Institute of Technology, 2011.
Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8150-815 0159,,1,,2000 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-22 12265,,, 2018
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699, 2015.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference machine learning, pp. 4784478-84 487, 2016.
Rui Xu and Donald Wunsch. Survey of clustering algorithms. IEEE Transactions on neural net- works, 16(3):645-678, 2005.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6002-6012, 2019.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DATASET SUMMARY
The dataset summary is given in 6 For ImageNet-10 10 and ImageNet-Dogs subsets of Deng et al. 2009). We used the same classes as Chang et al. (2017) for evaluating on these two datasets.
Table Dataset Summary
Dataset
Classes
Train Data
Test Data
Resolution
ImageNet- (Deng et al.
2009)
10
13000
500
224x 224
Imagenet-Dogs (Deng et al.
2009)
15
19500
750
224x 224
STL-10 (Coates et al.
10
5000
8000
96x 96
CIFARIO (Krizhvssky
2009)
10
50000
10000
32x 32
CIFAR 100-20 ( (Krizhevsky et al.
2009)
20
50000
10000
32x 32
CUB (Wah et al. 2011
200
5994
5794
224x 224
Caltech- 101
(Fei-Fei
2004)
101
7020
1657
224 224
Intel
6
14034
3000
128 128
AwA2 AwA2( et al.
2018)
50
29865
7457
224 x 224
A.2 ALGORITHM PSEUDO CODE
The pseudo code for our algorithm is given in Algorithm D
Algorithm 1: Consensus Clustering algorithm (ConCURL.)
Input: Dataset X [x.))_1-> F.m..KK K, B, B, M,d
Output: Cluster e;
Randomly initialize network parameters K prototypes eni, M random projection matrices ay to dimension d and O:
while total spoch
do
for 21, 2....
Seloct samples as 2'. from X:
Make forward pass on two views of the input batch (X). xx ). and ohtain the features xl. z representation
Compute loss Li which
Compute probability of sample belonging to the nth cluster, Pt.y for both views separately, using normalized x, eq
Compute cndes of the current batch q using eq
Compute loss L2 using oq
for (1, ....,
II
2, 6- Compute random transformationsdd of
P..) Compute probability th sample belonging the cluster, (P..) (L.m). (2:mm) using normalized eg
13
end
Compute loss La using eq
15
Compute total loss using eq i
Update the parameters, and prototypes using gradients
16
end
17
18
end
19
Make forward pass on all the data and store the features;
foreach m; X' do
21
Compute probability of th sample belonging the 1 cluster, pt. using mormalizeed i;; eq
22
Compute codes q using 2]
arg max; quk
end
A.3 IMPLEMENTATION DETAILS
We use residual network (He et al. 2016) with 34 layers (current state of the art clustering results Huang et al. (2020) also use the same architecture). The MLP projection head consists of a hidden layer of size 2048, followed by batch normalization and ReLU layers, and output layer of size 256. We use the Adam optimizer (Kingma & Ba 2014) with learning rate of 0.0005. We imple- mented our algorithm using the Pytorch framework and trained our algorithm using V100 GPU, taking 8 hours to train ImageNet-10 (13,000 training size) with a batch size of 128 for 500 epochs using 1 GPU.
13

---

Under review as a conference paper at ICLR 2021
We explain the generation of multiple augmented views that are shown to be very effective in unsu- pervised learning (Chen et al. 2020). Note that these augmented views are different from the views of Multi-view image data set such as Schops et al. (2017). Indeed it is possible to more than two augmented views, but we limit to two for the sake of simplicity. [Caron et al. (220)) propose an augmentation technique (Multi-Crop Crop) to use more than two views. In this work, we the augmen- tations used in Chen et al. (2020); Grill et al. (2020). We first crop random patch of the image with scale ranging from 0.08 1.0, and resize the cropped patch to 224x224 (128x x 128 and 96 x66 in the case of smaller resolution datasets such as Intel and STL10 respectively). The resulting image was then flipped horizontally with a probability of 0.5. We then apply color transformations, start- ing by randomly changing the brightness, contrast, saturation and hue with a probability of 0.8. The image was then changed to gray-scale with a probability of 0.2. Then we applied a Gaussian Blur with kernel size (23 x3) and a sigma chosen uniformly randomly between 0.1 and 2.0. The prob- ability of applying the Gaussian Blur was 1.0 for view and 0.5 for view 2. During evaluation, we resized the image such that the smaller edge of the image is of size 256 (not required for STL, Intel, CIFARIO, (FARR00)20), and a center crop is performed with the resolution mentioned in Table The color transformations were computed using Kornia (Riba et al. 2020) which is differentiable computer vision library for Pytorch.
To compute the random transformations on the embeddingg z, we followed two techniques. We used random projections (Bingham & Mannila (00T) with output dimension d, and transformed the embeddings the new space with dimension d. We also used diagonal transformation (Hsu al.) 2018) where we multiply with randomly generated diagonal matrix of the same dimension as z. We initialized M random transformations at the beginning and remain fixed throughout the training.
We performed model selection on the hyperparameters of the random transformations on the em- bedding space such as the number of random transformations M (ranging from 10 to 100) and the dimensionality of the output space if using random projection (we used [32, 64, 128, 256, 512]). We evaluated the models based on the metrics mentioned in Section A.44oo the data used for training the representations. Note that we fixed the number of prototypes to be equal to the number of ground truth classes. It was shown however that over-clustering Ieads to better representations (Caron et al.) 2020 Ji et al. 2019 Asano et al. 2019) and we can extend our model to include an over-clustering block with a larger set of prototypes (Ji et al. 2019) and alternate the training procedure between the blocks.
A.4 EVALUATION METRICS
We evaluate our algorithm by computing traditional clustering metrics (Cluster Accuracy, Normal- ized Mutual Information, and Adjusted Rand Index). Note usual approach taken literature is to evaluate the cluster metrics on the train data. Here, we report results on both the train data, as well as test data separately.
that for measuring clustering metrics, the
Cluster Accuracy The clustering accuracy is computed by first computing cluster partition of the input data. Once the partitions are computed and cluster indices assigned to each input data point, the linear assignment map computed using Kuhn-Munkres (Hungarian) algorithm that reassigns the cluster indices to the true labels of the data. Clustering accuracy is then given by
ACC = Ei H c(x+))
N
where Verue (I;) is a true label of x; and c(x;) is the cluster assignment produced by an algorithm (after Hungarian mapping).
Normalized Mutual Information For two clusterings U, V. with each containing UU) [V] clusters respectively, and let (U,| be the number of samples in cluster U, of clustering U (similarly for V), Mutual Information (MI) is given by
U
U
NU, 0 ViL |U:
MI(U.V)
log
14

---

Under review as a conference paper at ICLR 2021
where N is the number of data points under consideration. Normalized Mutual Information is defined as
MI(U,V)
NMI(U.V) /MTCUU)MIVV
Adjusted Rand Index (Hubert & Arabie 1985 Skl Suppose R is the groundtruth clustering and S is a partition, the RI of S is given as follows. Let a be the number of pairs of elements that are in the same set in R as well as in S; be the number of pairs of elements that are in diferent sets in R, and different sets S. Then
RI a
(2)
RI - [[[
ART
max (TT) R[
A.5
MORE DATASETS RESULTS
In Table 7
we show evaluation metrics on other four datasets for our method on both train and test
split.
Table More datasets: Clustering evaluation metrics of ConCURL on train and test splits
Datasets
Intel
Caltech101
CUB
AwA2
Split Metrics
Acc
NMI
ARI
Acc
NMI
ARI
Acc
NMI
ARI
NMI
ARI
Train
0.910
0.801
0.800
0.339
0.651
0.223
0.127
0.452
0.033
0.539
0.681
0.448
Test
0.899
0.783
0.776
0.361
0.691
0.218
0.113
0.442
0.025
0.542
0.685
0.446
A.6 ABLATION STUDY ON THE LOSSES
In this subsection, we study the effect weights 3 and y on the final metrics.Resulle for the weight configuration corresponding
= 1, = 1,7 is what is shown in the main paper. For the case of (o = 1 = 0, 0), we computed the cluster accuracy, NMI, ARI by computing the embeddings of all the data output by the representation learning algorithm used for L1 (here Grill et al. (2020)). Then we computed a K-means clustering on the embeddings (the target projection layer embeddings in this case) to obtain a partition of the data, and follow the same procedure mentioned in A.4 For all the experiments in this section, we trained only on the train split of the datasets.
Table 8: Cluster metrics evaluation on data points that available during training
Datauts
Imspehes-10
Inraches Dog
STL
hac
Caleeh
CLB
Ace
330
333
360
452
8030
3305
0375
1100
10
2007
11907
105
610
112
100
6310
0781
1761
1: 1-5
300
081
1188
0.90
083
630
2005
0900
m.w
1881
0291
1157
300
0008
6016
0701
Table 9: Cluster metrics evaluation on data points that were not available during training
ImapaNo-10
GrageNn Dagge
CLB
MEnoao MEEKS
Arz
Ag5
330
100
362
3001
362
8030
1300
1160
1706
1111
1000
100
2007
1767
198
108
618
117
6.30
1007
0760
1:2
3361
1300
0790
108
001
039
081
0700
11
0084
116
699
m m1
1991
0318
111
3000
0108
6381
0788
1010
A.7 T-SNE PLOTS
In figure we show t-sne plot of the ImageNet-10 embeddings obtained from ConCURL trained model. One can clearly see the separation between various clusters with the exception of airline and airship clusters. Airline and airship clusters are mixed together on leftmost and righmost part of the t-sne plot.
15

---

Under review as a conference paper at ICLR 2021
king_penguin
Maltese
snow leopard
airliner
airship
container ship
ball
sports car
trailer_truck
orange
Figure 3: t-sne plot of ImageNet-10 10
16


Reviews:
[{"review_id": "fivm5N4X6p7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effect of combining ensemble learning approaches with deep clustering. The paper wants to show that ensemble learning methods, in particular consensus clustering, can improve the clustering accuracy when combined with general representation learning/clustering blocks. However, I am not sure that the results presented in the paper are enough to support the claims.\n\nThe paper's  pros are:\n(+) It Is the first to combine ensemble methods with deep clustering models. Although ensemble methods have been widely applied, studying consensus clustering in the current problem setting is novel.\n(+) It Is the first to be able to have an ensemble deep clustering algorithm that gains empirically over other state-of-the-art models, showing the ideas to be potentially effective.\n(+) The writing is in general clear and undestandable.\n\nThe paper's cons are:\n(-) The wording in the abstract is a bit confusing in the sense that after reading it one might think the algorithm does consensus clustering first and uses the clustering to learn better representations of the input data. Although this is clarified later in the main body.\n(-) The description of the main algorithm seems to be more intuitive than innovative. Some algorimic design choices are not very convincing. For example, the choice of using random projections on embedding to produce different clusterings, although an interesting idea, makes me wonder why it is necessarily a good way to introduce randomness into the whole framework. The authors can expand their discussion on this. I also remain dubious about why different representation instead of different clustering methods \nAlso, since the authors meant the idea to be applicable to general representation learning/deep clustering blocks, I'm not sure the current experimental data in the paper can lead to that conclusion.\n(-) Using the performance metrics provided by the authors, I find it a bit hard to conclude that the proposed algorithm has a significant advantage over state of the art methods, especially PICA. Also, the fluctuation in performance metrics caused by different parameter settings seems to be, in magnitude, at least comparable to the margin of ConCURL over the baselines.\n\nOverall, I think this paper contains  interesting seed ideas such as combining consensus clustering with representation learning and making use of the learned representation to generate multiple clusterings. These seed ideas could be good for this venue. However, the work is still premature and flawed by crude algorithm/experiment design.  The quality can be significantly improved if the authors can give a more general algorithmic framework (since the authors meant the ideas to be applicable to general representation learning/clustering algorithms), equipped with more thorough experimental investigation to support the applicability and superiority of the current approach.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JMIpdYqowG1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a learning-based approach for image clustering. In particular, similarly to recent algorithms fro unsupervised representation learning, such a DeepCluster, they propose to iterate between clustering the images in the feature space of the network and updating the network weights to respect the clusters. Two main differences with respect to DeepCluster-like algorithms is that they target the task of clustering itself, and do not evaluate the generalizability of the leaned representations for other tasks, and that they propose to use cluster ensembles to improve training robustness. In particular, they generate 2 clusterings of the images at every iteration by applying different sets of data augmentations and feature transformations to the input images. The objective is then not only to respect these clusterings but also to enforce consistency between them over time, thus improving representation invariance to irrelevant image details. In an experimental evaluation on a set of standard image clustering benchmarks they outperform prior work in most scenarios. \n\nThe idea is reasonable and the method seems sound, however a similar approach has been proposed in Zhuang et al., ICCV'19 (Local Aggregation for Unsupervised Learning of Visual Embeddings). In contrast to this work, the authors of Zhuang et al., used different runs of the clustering algorithm to obtain diverse clusterings, instead of transforming the images, but the overall approach is very similar. The authors seem to be not aware of that work.\n\nIn the current from it is not clear whether the proposed approach has any advantages over Zhuang et al. In the rebuttal the authors need to provide a discussion of their novelty with respect to that method as well as an experimental comparison on ImageNet.\n\n\nI appreciate the authors' efforts to fairly compare to Zhuang et al. in the rebuttal, and I do find the preliminary evidence sufficient to establish that their approach outperforms LA on clustering metrics. However, the authors seem to miss the the point that LA is not just another consensus clustering approach which they forgot to include into literature review since it does not report clustering metrics. The main contribution of their work is combining consensus clustering with representation learning, which is exactly what the authors of LA had done before. It does seems that the particular approach proposed in this paper results in a better clustering performance, so the submission contains a valid contribution, but the relationship between the two methods needs to be discussed in a lot more detail, and they have to be throughly compared experimentally. I encourage the authors to improve the manuscript in this direction and resubmit to a different venue.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6VdH-_LXZDq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The idea of combining learning representations and ensemble clustering is interesting and possibly promising. I think this paper in its present form is not ready to be published though.\n\n- Cited literature on consensus clustering is old and outdated (up to 2005) and does not capture the state of the art. More recent work exists that outperforms the ones cited in this paper. The authors should also consider comparing their method with the state of the art on consensus clustering which is not deep learning based.\n\n- The description of the algorithm is often too vague and is fragmented in mini-sections, and the flow that brings them together is unclear. The individual pieces are not novel and no novel technical challenges or contributions are presented. Several steps and settings are ad-hoc and not well-motivated. What does it mean exactly: \"Performing a forward pass on the two views results in feature vectors f1,f2.\"? How are f1 and f2 obtained? What's the motivation for using a multi-layer perceptron to further reduce f1 and f2? Why should we expect this to produce a good representation for clustering? How were the dimensionalities of the hidden and output layers chosen?\n\n- The notation in Eq. (2) is not clearly defined. What is a transportation polytope?\n\n- Writing and organization need major work. There are many vague statements that are not well-supported or motivated. \n\n- How does the set of observations X defined in 3.1 relate with the input batch X_b in 3.1.2?\n\n- The process of generating two views of the input images is not clearly defined. It seems that this process mainly consists in adding random noise to the images? This is quite different from the notion of different *views* of a data object (e.g. image) in the literature. Also, why 2 views and not 3 or more?\n\n- I don't find Figure 1 particularly useful. Part (a) does not help at all in understanding the approach.\n\n- What's the size of a typical ensemble being generated? The paper mentions M transformations. What is the typical M value. In the experiments M ranges from 10 to 100. How sensitive is the approach to the specific transformations being applied and to the number of transformations?\n\n- The relationship and distinction between the views and the ensemble need to be better clarified.\n\n- I find the explanation above Eq (4) unconvincing. How diverse are the probability estimations generated by each component of the ensemble? Have the authors performed any analysis on this?\n\n- The authors should discuss assumptions and limitations of the proposed methodology. What assumptions does the proposed methodology make? Under which scenarios and data distributions is the methodology expected to perform well? What are its limitations?\n\n- No insight or analysis on the results is provided. Often PICA outperforms the proposed method. I understand that PICA uses both training and test data, but since we are dealing with clustering and unsupervised data, I am not sure this is a real advantage for PICA.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model for learning using ensemble clustering. The reviewers found the general idea promising.
However, while promising, all reviewers noted that in its curent form the paper is not fit for publication. The reviewers pointed out missing references, issues with the abstract, lack of motivation for some of the algorithmic choices, limited novelty over clarity in the description of difference w.r.t. previous work.
Because of all these reasons, this paper does not meet the bar of acceptance. I recommend the authors take into account the feedback provided in the reviews and discussion and resubmit to another venue.
Decision: Reject