You Never Cluster Alone
Authors: Yuming Shen
Ziyi Shen
Menghan Wang
Jie Qin
Philip Torr
Ling Shao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Unsupervised Clustering
Unsupervised Learning
Deep Learning

Abstract: Recent advances in self-supervised learning with instance-level contrastive objectives facilitate unsupervised clustering. However, a standalone datum is not perceiving the context of the holistic cluster, and may undergo sub-optimal assignment. In this paper, we extend the mainstream contrastive learning paradigm to a cluster-level scheme, where all the data subjected to the same cluster contribute to a unified representation that encodes the context of each data group. Contrastive learning with this representation then rewards the assignment of each datum. To implement this vision, we propose twin-contrast clustering (TCC). We define a set of categorical variables as clustering assignment confidence, which links the instance-level learning track with the cluster-level one. On one hand, with the corresponding assignment variables being the weight, a weighted aggregation along the data points implements the set representation of a cluster. We further propose heuristic cluster augmentation equivalents to enable cluster-level contrastive learning. On the other hand, we derive the evidence lower-bound of the instance-level contrastive objective with the assignments. By reparametrizing the assignment variables, TCC is trained end-to-end, requiring no alternating steps. Extensive experiments show that TCC outperforms the state-of-the-art on benchmarked datasets.

You Never Cluster Alone
Yuming Shen*! Ziyi Shen?, Menghan Wang Jie Qin', Philip H.S. Torr', and Ling Shao
University of Oxford University College London eeBay
*Nanjing University of Aeronautics and Astronautics *Inception Institute of Artificial Intelligence ymcidence@@gmail com
Abstract
Recent advances self-supervised learning with instance-level contrastive ob- jectives facilitate unsupervised clustering. However, standalone datum is not perceiving the context of the holistic cluster, and may undergo sub-optimal assign- ment. In this paper, we extend the mainstream contrastive learning paradigm to cluster-level scheme, where all the data subjected to the same cluster contribute to unified representation that encodes the context of each data group. Contrastive learning with this representation then rewards the assignment of each datum. To im- plement this vision, propose twin-contrast clustering (TCC). We define set of categorical variables as clustering assignment confidence, which links the instance- level learning track with the cluster-level one. On one hand, with the corresponding assignment variables being the weight, a weighted aggregation along the data points implements the set representation of cluster. We further propose heuristic cluster augmentation equivalents to enable cluster-level contrastive learning. On the other hand, we derive the evidence lower-bound the instance-level contrastive objective with the assignments. By reparametrizing the assignment variables, TCC is trained end-to-end, requiring no alternating steps. Extensive experiments show that TCC outperforms the oftte-of-hheatt on challenging benchmarks.
Introduction
Ancestored by various similarity-based [59] and feature-based [4, 56] approaches, unsupervised deep clustering jointly optimizes data representations and cluster assignments [81]. A recent fashion in this domain takes inspiration from contrastive learning in computer vision [11, 12, 24], leveraging the effectiveness and simplicity of discriminative feature learning. This strategy is experimentally reasonable, as previous research has found that the learnt representations reveal data semantics and locality [34, 80]. Even simple migration of contrastive learning significantly improves clustering performance, of which examples include two-stage clustering pipeline [73] with contrastive pre- training and k- m-mns [56] and composition of an InfoNCE loss [62] and a clustering one [81 1]] in [89]. Compared with the deep generative counterparts [16, 36, 54, 86], contrastive clustering free from decoding and computationally practical, with guaranteed feature quality.
However, have been paying too much attention the representation expressiveness of a single data point? Intuitively, a standalone data point, regardless of its feature quality, cannot tell us much about how the cluster looks like. Fig. 1 illustrates a simple analogy using the TwoMoons dataset. Without any context for the crescents, it is difficult to assign data point to either of the two clusters based on its own representation, as the point can be inside one moon but still close to the other. Accordingly, observing more data reveals more about the holistic distributions of the clusters, e.g., the shapes of the moons Fig. 1, and thus heuristically benefits clustering. Although we can implicitly parametrize the context of the clusters by the model itself, e.g., using Gaussian mixture model (GMM) [4] or encoding this information by deep model parameters, explicitly representing the context yields the most common deep learning practice. This further opens the door for learning cluster-level representations with all corresponding data points. Namely, you never cluster alone.
"A part of this work done when the author with eBay.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Asiggmmnt w without Contest
Encoding
Latent Contect: The Uyper Meow"
Learning for Contest Awrareses
Contect: "The Lower Moon'
Eaceding
Figure 1: The motivation behind this work. (a): When assessing a standalone data point, cluster assignment can be challenging due to the non-linearity of the feature space and the lack of context information for the data distribution. (b): Our model learns this context by representing each cluster with latent features.
In this paper, motivated by the thought experiment above, we develop multi-granularity contrastive learning framework, which includes instance-level granularity and a cluster-level one. The former learning track conveys the conventional functionalities of contrastive learning, i.e,, learning compact data representations and preserving underlying semanties. We further introduce set of latent variables cluster assignments, and derive an evidence lower bound (ELBO) of instance-level InfoNCE [62]. As per the cluster-level granularity, we leverage these latent variables as weights to aggregate all the corresponding data representations as the set representation [87] of cluster. We can then apply contrastive losses to all clusters, thereafter rewarding/updatinn the cluster assignments. Abbreviated as twin-contrast clustering (TCC), our work delivers the following contributions:
We develop the novel TCC model, which, for the first time, shapes and leverages a unified representation of the cluster semantics in the context of contrastive clustering.
We define and implement the cluster-level augmentations in a batch-based training and stochastic labelling procedure, which enables on-the-fly contrastive learning on clusters.
We achieve significant performance gains against the state-of-the art methods five bench- mark datasets. Moreover, TCC can be trained from scratch, requiring no pre- pre-trained models auxiliary knowledge from other domains.
2 Preliminaries
2.1 Contrastive Learning
Contrastive learning, as the name suggests, aims to distinguish an instance from all others using embeddings, with the dot-product similarity typically being used as the measurement. Let X { be an N N-point dataset, with the i-th observation x; E Rd= An arbitrary transformation f: Rd + dm encodes each data point into din-immesioonal vector. With the index i being the identifier, the InfoNCE loss [62] discriminates x; from the others through softmax-lin like likelihood:
exp fxI((x;)/T)
(ilx:) = log En exp (v]
(1)
A temperature hyperparameter 7 controls the concentration level [26, 80]. V = (v: refers to the vocabulary of the dataset, which usually based on the data embeddings under different augmentations. As caching the entire V is not practical for large-scale training, existing works propose surrogates of Eq. (1), egg. replacing V with a memory bank [80], queuing it with momentum network [12, 24], or training with large batches [11].
2.2 Deep Set Representations
-nniuaattiioonvviaannt To learn the representations of sets, we need to consider permutation invariant transformations. Zaheer al. [87] showed that all permutation inntaiaatiionnnariaatt functions T(-) applied to a X generally fall into the following form:
T(X) = (>/// N
h(xx))
(2)

---

Resteserizion
Clancrin
Nari
XLL
Seat--ecce Mullihuatto
Seppi
lastuncee
Gembel22.1
(A)ps)
gv(2 xi)
qu pxa)
(2)xx)
go
(2/p0)))
(Na(f)
fo(xs)
(xa)
Datuset
Figure 2: The schematic of TCC. Here, we demonstrate 2-way clustering example with three images. For simplicity, only illustrate the instance-level learning module the last image, while it is applied to all images.
where g()) and h-)) are arbitrary continuous transformations. Note that the aggregation above can be executed on weighted basis, which is typically achieved by the attention mechanism between the set-level queries and instance-level keys/values [31, 49]. Our design on each cluster is partially inspired by this, as back propppaapaattonnn does not support hard instance-level assignment. Next, we define our cluster representation along with the clustering procedure, and describe how it is trained with contrastive learning.
Swin-Contrast Clustering
We consider K-way clustering problem, with K being the number of clusters. Let k {1. ... K} denote the entry of a cluster that x; may belong to, and the categorical variable n. (1), , indicate the cluster assignment probabilities of x,- Following common practice, we regard image clustering as our target for simplicity. Fig. 2 provides a schematic of TCC. The cluster-level contrast track reflects our motivation from Sec. 1, while the instance-level one learns the semantics of each image. We bridge these two tracks with the inference model n, (k(k) = (k(x)x,) so that both losses reward and update the assignment on each x,. In particular, qo (k) x;) is parametrized by softmax operation:
(op(M[
Ty (k) = (k))),
(3)
ΣF=1 exp(ut fo(x))
where sa(-) is convolutional neural network (CNN) [47, 48] built upon random data augmentations, producing dm -dmmensiooal features. We denote μ4 = { 1 T== as a set of trainable cluster prototypes,
where 0 refers the collection of all parameters. In Sec. 3.1, cluster features, and Sec. 3.2 we derive the ELBO of Eq. (I) with qo (k(x;)) the following, we will omit the index i for brevity when and π clearly correspond to single data point.
leverage qo(k) (x) to aggregate
3.1 Representing and Augmenting the Context for Cluster-Level Contrast
Cluster-Level Representation We implement Eq. (2) for each cluster using soft aggregation, where (k(k)x weighs the relevance of the data to the given cluster. Denoted as rk, the k-th cluster representation computed by:
ho(x: k) = π (k)) fx(x),
Σ 1=1 E/
/|> := (x(x;;
(4)
= To (X;
ho (x k))
where |l:1/2 refers to the L2-norm. We adopt L2 normalization here for two main purposes. First, the summation of T (kk) qo (k:)x along x is not self-normalized. Second, and more importantly, as shown in [11, 12, 24], L.2-nommaized features benefit contrastive learning.
The Anchored Cluster Semantics Intuitively, π (k) reflects the degree of relevance of datum to the k-th cluster. With being the aggregation weight, r; represents the information that related to the corresponding prototype pa- In other words, our design treats each μ as the semantic anchor that queries the in-coming batch to form a representation describing a certain latent topic.

---

Cluster-Level Augmentation Equivalents Contrastive learning is usually employed alongside random data augmentation [11. 24] to obtain positive candidates. Though defining a uniform augmentation scheme for sets is beyond the scope of this paper, the proposed model reflects cluster- level augmentation in design by the following heuristics:
(a) Augmentation elements. TCC implicitly inherits existing image augmentation tech- niques (such as cropping, color jittering, random flipping, and grayscale conversion) by implementing them using fo(-).
(b) Irrelevant minorities. We consider injecting small proportion of irrelevant data into cluster representation, while keeping the main semantics the cluster unchanged. Eq. (4) turns out to be an equivalent to this. As the softmax product iT (Tkk is always positive, those data that are not very related to the given cluster still contribute to the cluster's representation, which compiles the irrelevant. Meanwhile, these irrelevant data are not dominating the value of the cluster representation, because the small value of (ik) scales the feature magnitude during aggregation, which counts the minority.
(c) Subsetting. Empirically, subset of cluster holds the same semantics the original. Batch-based training samples data each step, which naturally creates subsets for each cluster.
We experimentally find the above augmentation equivalents are sufficient for the clustering task. On the other hand, since Eq. (4) is permutation permutation-invariant, reordering the sequence of data does not yield a valid augmentation.
Cluster-Level Contrastive Learning We define a simple contrastive objective that preserves the identity of each cluster against the rest. Having everything in batch, e.g., SimCLR-like framework [11], does not allow augmentation (c) to be fully utilized in the loss, since the two augmented counterparts Fy and f& from a batch may form part of the same subset of a cluster. Hence, we opt for the MoCo-like solution [24], employing an L-sized memory queue P = {pi) to cache negative samples and momentum network produce f. kX; P stores each cluster representation under different subsets, training with which preserves the temporal semantic consistency [45] of clusters. Our cluster-level objective minimizes the following negative log- likelihood (NLL):
1 K
exp(FIra/T)
= f = R& log (k([k))
log
exp((frx/T)) + ΣL exp(p/rw/Ti mod K + k) 1=1
(5)
where 1()) is an indicator function and mod is the modular operator. Since the cluster number K can be less than the queue size L, we exclude the features that represent the same cluster as k in the negative sample collection P by inserting the indicator function into the loss above.
3.2 Instance-Level Contrast with Cluster Assignments
The ELBO
We propose to reuse the inference model qo qo x) discussed above to compute the instance-level contrastive loss, that the clustering process can benefit from contrastive learning. Let us start from the following ELBO of log pe (ilx) in Eq. (1):
log po (ilx) 2 (R(x) [o |x, ] KL (ko
(6)
where KL(-) is the Kullback-eibler (KL) divergence. We derive this ELBO in Appendix A. The true distribution po (k(x) not available under the unsupervised setting. We follow [40, 69] and use fixed prior instead. In practice, we employ the uniform distribution, i.e., pa (k|x) == po(k) = K. Then, the KL term above can be reduced to a simple form KL (ge (k|x))llpe (klx)) = log K + (k(xx)))
Empirically, this encourages an evenly distributed cluster assignment across the dataset.
Regarding the expectation term in Eq. (6), back- propagation through the discrete entry k is not feasible. We resort to the Gumbel softmax trick [32, 57] as relaxation. Specifically, a latent variable each as a replacement. Each entry c(k) yields the reparametrization
(0, 1)K is assigned c(k) = Softmax, ( (log (k) + (k)))/), where e ~ Gumbel(0, 1) and À is another temperature hyperparameter. Hence, we obtain the surrogate (x [log pe (ilx, k)] E, [logpe [i]x, c)] and the gradients can be estimated with Monte Carlo.

---

Instance-Level Contrastive Learning In alignment with Eq. (5), log ( c) learns the repre- sentation of on a momentum contrast basis [24] by defining the following transformation:
(fo(x) NNo()))/ll) + NNo(c))l2
exp(êTe/r)
log p (z) c) = llog exp(éTe/7) Σj= exp(q)e/T)
where NNo(-) denotes single fully connected network. We accordingly use è indicate the repre- sentation of x processed by a momentum network f.(.) and NN2(.) under different augmentations
and Gumbel samplings [32, 57]. A J-sized memory queue Q = {q)} is also introduced to cache negative samples, updated by é. In this way, we obtain the instance-level loss:
L2 F, Reg [log po po (i)x; c:)] H(go(k)xx)) log K].
(8)
3.3
Training and Inference
TCC enables end-to- training from scratch. Our learning objective is a simple convex com- bination of Eq. (5) and i.e.
Algorithm 1: Training Algorithm of TCC
Input: Dataset X j Output: Network parameters 0. Initialize 0
ac + (1 - C2.
(9)
The hyperparameter a E (0, 1) controls the contributions of the two contrastive learning tracks. As discussed in Sec. 3.1, L is com- batch-based routine. For each data point x, we obtain only one sample from the Gumbel distribution at each step, since this is usually sufficient for long-term training [39, 40, 69]. One may also regard this stochasticity as an alternative to data augmen- tation. The overall training algorithm is shown in Alg. 1. Here, I(.) indicates an arbitrary stochastic gradient descent (SGD) optimizer.
repeat
Randomly select a mini-batch from for each x; in the batch do Randomly augment x; twice
puted following
Sample with Gumbel softmax end
C c (9) f I
Update the queues P with i and Q with è Update Ô with momentum moving average until convergence or reuching max iteration;
All trainable components are subscripted by while those marked with Ô are the network momentum counterparts to be updated with momentum moving average. Inference with TCC only requires disabling random data augmentation and then computing argmax, qo (k/x).
Complexity When sampling once for each x during training, the time complexity for Eq. (9) is JJ, while the memory complexity for the memory bank turns out to be the same. Here we omit the complexity introduced by the CNN backbone and dot-product computation, as orthogonal to the design. Compared with the recent mixture-of mixture-of-experr approach [73], which requires time and memory complexity of ((J)), TCC is trained a more efficient way.
3.4 Relations to Existing Works
MiCE [73] also proposes lower bound for the instance-level contrastive objective. However, it does not directly reparametrize the variational model (k [)/x) for lower-bound computation and inference, but instead employs K K-expert solution with EM. This design is less efficient than TCC since each data point needs to be processed by all experts. Moreover, MiCE [73] does not consider cluster-level discriminability. SCI. [28] follows a similar motivation to TCC in cluster-level discriminability, but implements this with an instance-to-se similarity, while our model learns a unified representation for each cluster. Furthermore, in SCL [28]. the clustering inference model is disentangled from the instance-level contrastive objective. In contrast, the inference model qe (k|x) of TCC contributes to instance-level discrimination (Eq. (6)).
We recently find CC [53] comes with a cluster-level contrastive loss as well. It utilizes the in-batch inference results [ (k), ... Tn (k)) to describe the k-th cluster. However, this procedure is not literally learning the cluster representation, since it is not permutation free. Re-ordering the batch may shift the semantics of the produced feature. We mitigate this issue with deep sets [87] and the empirical cluster-level augmentations for temporal consistency [45]. similar problem is witnessed in [66]. In addition, our instance-level discrimination model yields more general case than the one
5

---

of CC [53]. When removing the stochasticity and enforeing po c) := po(ilc) in our model, L2 reduces the one of [53]. We experimentally show that our design preserves more data semanties, and thus benefits clustering. Being not related to our main contribution, we provide more elaboration on this in Appendix B under the framework of variational information bottleneck [2].
4 Related Work
Deep Clustering In addition to the classic approaches [4, 13, 18, 20, 43, 56, 78, 91], the concept of simultaneous feature learning and clustering with deep models can be traced back to [81, 83J. The successors, including [9, 23, 61, 68, 77, 79, 85], have continuously improved the performance since. As conventional option for unsupervised learning, deep generative models are also widely adopted in clustering [10, 16, 35, 36, 42, 51, 58, 86, 90], usually backboned by VAE [39] and GAN [21]. However, generators are computationally expensive for end-to-end training, and often less effective than the discriminative models [15, 27, 33] in feature learning [11]. Recent research has considered contrastive learning in clustering [28, 53, 73, 74, 89]. We discuss the drawback of them and their
relations to TCC in Sec. 3.4.
Contrastive Learning Contrastive learning learns compact image representations in a self- supervised manner [11, 12, 24, 62, 72]. There various applications for this paradigm [34, 41, 82, 93]. We note that several contrastive learning approaches [7, 52] conceptually in- volve clustering procedure. Nevertheless, they based unified pre-training framework to benefit the downstream tasks, instead of delivering a specific clustering model.
Set Representations Our cluster-level representation (Eq. (4)) is realization of deep sets [17, 87]. Existing research this area mainly focuses on set-level tasks [19, 31, 37, 76, 84]. is also notable that, though we leverage cluster-level representation learning, TCC is still an instance-level clustering model, which is different from the set-level clustering models [49, 50, 63].
Experiments
5.1 Settings
We follow the recent works [3] and
Table 1: Dataset settings for our experiments.
report the performance of TCC in terms of clustering accuracy (ACC) [81], nor-
Dataset
Images
Clusters (K)
Input Size
malized mutual information (NMI) [70]
CIFAR-IO [444
60,000
10
32 32
and adjusted random index (ART) [30].
CIFAR-100 [44]
60,000
20
32 32
For fair comparison with existing works,
STL-10 [14]
13,000
10
96 x 96
we do not use any supervised pre-trained models. The experiments are conducted on five benchmark datasets, including CIFAR-I0/100 [44], ImageNet-10/Dog [9] and STL-10 [14].
ImageNet-10 0 [9]
13,000
10
96 x 96
ImageNet- Dog [9]
19,500
15
96 96
Note that ImageNeet
[9] is a subset of the original ImageNet dataset [67]. Since most ex- isting works have dre-defined cluster numbers, we adopt this practice and follow their training/test protocols [29, 33, 61, 73). Tab. I depicts the details of the settings.
5.2 Implementation Details
TCC is implemented with the deep learning toolbox TensorFlow [1]. We choose the MoCo-style random image augmentations [24] for fair comparison with the recent works [28, 73]. We further link our choice of augmentations with the cluster representation temporal consistency in Sec. 3.1. Specifically, each image is successively processed by random cropping, gray-scaling, color jittering, and horizontal flipping, followed by mean-std standardization. We refer to [24, 73] for more details. We employ ResNet-34 [25] as the default CNN backbone fe(.), which is also identical to [28, 73]. Appendix C gives full illustration of the CNN structure. The image size d, and cluster number K are fixed for each dataset, as shown in Tab. 1. The feature dimensionality produced by CNN d. 128. Following common practice [12, 24, 11]. we fix the contrastive temperature T while using slightly lower À = 0.8 for the Gumbel softmax trick [32, 57] to encourage concrete assignments. We implement fixed-length instance-level memory bank Q with a size of J = 12, 800 to match up with the smallest dataset in our experiments. The size of the cluster- memory bank is set to T. = 100 x K, varying from each dataset. We have a = 0.5 so that L, and La provide equal contributions to training. The choice of batch size is of importance to TCC in computing

---

Table 2: Unsupervised clustering performance comparison with existing methods (in percentage %).. We provide additional results Tiny ImageNet [46] and comparison with more contrastive baselines such as SWAV [7] in Appendix D.
Method
CIFAR-10
CIFAR- 100
STL-10
ImageNet-10
ImageNet-Doet
ACC
ARI
NMI
ACC
ART
NMI
ACC
ART
NMI
ACC
ART
NMI
ART
AC [22]
22.8
13.8
3.4
23.9
33.2
14.0
13.8
24.2
6.7
3.7
NMF
8.1
19.0
3,4
7.9
11.8
2.6
9.6
18.0
4.6
13.2
23.0
6.5
4,4
11.8
[E
23.9
31.4
16.9
10.0
16.5
4.8
25.0
30.3
16.1
21.0
31.7
15.2
10.4
18.5
DAE [75]
25.1
29.7
16.3
11.1
15.1
4.6
22.4
30.2
15.2
20.6
30.4
13.8
10.4
19.0
DOGAN [65]
26.5
31.5
17.6
12.0
15.1
4.5
21.0
29.8
13.9
22,5
34.6
15.7
12.1
17.4
DeCNN [88]
24.0
28.2
17.4
9.2
13.3
3.8
22.7
29.9
16.2
18.6
31.3
14.2
9.8
17.5
7.3
VAE [39]
24.5
29.1
16.7
10.8
15.2
4.0
20.0
28.2
14.6
19.3
33.4
16.8
10.7
17.9
JULE [85]
19.2
27.2
13.8
10.3
13.7
3.3
18.2
27.7
16.4
17.5
30.0
13.8
5,4
13.8
2.8
DEC [81]
25.7
30,1
16.1
13.6
18.5
5.0
27.6
35.9
18.6
28,2
38.1
20.3
12.2
19.5
DAC [9]
39.6
52.2
30.6
18.5
23.8
8.8
36.6
47.0
25.7
39.4
52.7
30.2
21,9
27.5
11.1
ADC [23]
32.5
DDC [8]
42.4
52.4
32,9
37.1
48.9
26.7
43.3
57.7
34.5
DCCM [79]
32.7
17.3
37.6
48.2
26.2
60.8
IC [33]
51.3
61.7
41.1
25.7
43.1
49.9
29.5
31.2
61.1
71.9
27.4
PICA [29]
56.1
64.5
46.7
29.6
32.2
15.9
78.2
85.0
73.3
33.6
32.4
17.9
[92]
26.2
DHOG [15]
$8.5
66.6
49.2
25.8
26.1
11.8
41.3
48.3
27.2
GATCluster [6]]
28.1
11.6
58.3
36.3
IDED [71]
71.4
81.5
66.3
42.6
42.5
26.4
64.3
75.6
57.5
89.8
95.4
90.1
54.6
59.1
41.3
CC [53]
79.0
26.6
76,4
72.6
85.2
MoCo buseline [73]
66.9
77.6
60.8
39.0
39.7
24.2
61.5
72.8
52.4
34.7
33.8
19.7
MICE
43.6
44.0
28.0
63.5
75.2
57.5
TCC
79.0
90.6
73.3
47.9
49.1
31.2
73.2
81.4
68.9
84,8
89.7
82.5
55.4
59.5
41.7
the cluster-level representations. We set it to 32 K. by default to ensure that sufficient images can be assigned to a cluster at each step. Training TCC only requires SGD w.r.t. 0 and momentum update w.r.t. We employ the Adam optimizer [38] with a default learning rate of 3 x 10-3, without learning rate scheduling. The momentum network is updated by 0 + 0.999Ø + 0.0016, where all modules subscripted by 0 are involved in this procedure. We train TCC for at least 1, 000 epochs a single NVIDIA V100 GPU.
5.3
Comparison with the State-of-the-Art
Baselines Both deep clustering and traditional models are compared, including a MoCO-based two-stage baseline intro- duced by [73]. Similar the recent works [29,33 33, 73, 79], we pick deep models that enable training from scratch and do not require supervised pre- pre-training parameters, for fair and reason- able comparison. For this reason, baselines such VaDE [36] and SPICE [60] are not included here. We also exclude the clustering refinement approaches [64] from our comparison as they are orthogonal to our design.
CIBAR-10
60
30
Results The clustering performance (in percentage %) is shown in Tab. 2. For those baselines that are not designed for clustering [3, 21, 391, report the results with k-means on the produced features. TCC outperforms existing works on most benchmarks. In particular, CIFAR-I0 [44], TCC outperforms the state-of-the-ar methods by large margins, e.g., 7% higher in ACC than the second best one (i.e., MiCE [73] with even stronger augmentations [11]). As closely-related work. MiCE [73] only considers instance-level representation learning. The performance gain of TCC over MiCE endorses our motivation to introduce cluster-level representations. We also observe that TCC underperforms [53] on STL-10 [14] due the exceptionally high performance it on this dataset. How- ever, TCC is still the runner-up on this dataset by significant margin, and is superior to [53] on the other four datasets. We argue that the performance deep clustering methods, as contrastive learning is originally designed for large-scale tasks. Fig. illustrates the t-SNE [55] scattering results of TCC CIFAR- 10 [44] and STL-10 [14].
30
30
Figure t-SNE visualization on
CIFAR-10 [44] and STL-10 [14].
larger datasets is of more importance when comparing contrastive

---

PVfecuane 100
100
Petheraan (80
Potomazas
ARE
NHH
NHIT
(4]
66)
(4)
Figure 4: Hyperparameter analysis results on CIFAR-10 [44].
5,4 Ablation Study
We conduct an ablation study to validate our motivation and design, with the following baselines.
ii Without L1. As key component of TCC, the cluster-level contrastive learning objective L reflects our main motivation. We first assess the model performance when removing this loss, which reduces TCC to simple instance-level contrastive clustering model.
(ii) Without L2. We can also remove L2 to see if Li alone still yields a valid baseline.
(iii) Multiple Sampling, As described in Sec. 3.3, we only consider single sample с each time to compute the lower bound of the instance-level loss (Eq. (8)). We also consider applying the Gumbel softmax trick multiple times for each image. In particular, we sample 10 groups of latent variables each time to compute the expectation term E, [log (ilxx, c)] of Lo. On each batch, we enqueue the mean of é w.r.. all 10 sampled for each image.
(iv) Without P. Since we usually have small cluster number K, computing the cluster-level InfoNCE loss does not necessarily require memory bank to cache the negative sample surrogates. In this baseline, we remove the cluster-level memory bank P, and use the remaining K cluster representations as negative samples when computing Eq. (5).
(v) Without Augmentation (a) for L1. We validate cluster-level augmentation strategies by removing image augmentations when computing Eq. (8). Note that this baseline does not influence the instance-level objective by rendering both augmented and original images fe(-).
(vi) Without Augmentation (b) for L1. This baseline requires hard assignments at each step that cluster-level aggregation only involves images that are assigned to the corresponding clusters. This modification does not affect L2-
(vii) Without Augmentation (c) for L1. The final baseline changes the training pipeline. Since we do not subset any clusters here, cluster representation aggregation (Eq. (4)) runs on the whole training set after each epoch. We apply alternating training procedure as follows. First, Le optimizes the model for a full epoch. Then, we descend L1 with aggregated cluster features and repeat.
Baseline Comparison Results We show the ablation study results in Tab. 3. Without L,. TCC performs similarly to the two-stage baseline with MoCo [24] and k-means [56] reported
Table 3: Ablation study results (in percentage %)..
CIFAR-I0
Baseline
NMI
ACC
ART
Without ć
68.9
78.7
57.9
(ii
Without Co
37.1
45.4
24.5
in [73]. which is slightly lower than
(ii)
Multiple Sampling
78.5
90.1
74.2
MiCE [73]. Interestingly, L1 does not provide instance-level discriminative information. Although it does still serve valid baseline, it does not perform very well (Baseline (ii)). Specifically, we experience strong degeneracy [6, 33] with this baseline, but it still produces better results than the traditional models. We also observe that having multiple samples with the Gumbel softmax trick [32, 57] for gradient estimation does make much difference from single-sample solution. Baseline (iv) also underperforms the original model. As discussed in previous sections, having memory bank for cluster representations provides a way to acquire more negative samples for contrastive learning, considering the fact that the cluster number is usually limited.
(iv)
Without P
72.0
82.9
68.8
(v)
Without Augmentation (a) for Cr
73.5
85.3
69.1
(vi)
Without Augmentation for Li
68.5
79.2
60.6
(Vit)
Without Augmentation for 69.4
80.0
62.7
TCC Full Model
79.0
90.6
73.3
We evaluate the hyperparameters most essential to our design, including the loss weight a, the temperature of the Gumbel softmax X, the cluster-level memory queue length L, and the batch size. The InfoNCE temperature and the instance-level memory queue length J are not included here since they are not relevant to our key motivation and have been employed and evaluated in the recent works [24, 28, 73]. The corresponding results are plotted in Fig. 4. Though C₁ plays an
Hyperparameters

---

200
limined)
50-
ACC (50
(N)
0.14
0,07
20
30
250
410
140
1,39
200
600
1.000
Ependas
Truniigg Epociko
[1]
Figure 5: (a) and (b): On-batch ACC comparison between TCC and MiCE [73] w.E. training epochs and training time respectively on CIFAR-10 [44]. (c): The values DEC loss [81] during training. Note that TCC is not trained with DEC, and we just record the figures for illustration.
essential role in the proposed model, having large values a does not improve the performance, as the key instance-level semantics are yet learnt by L2. Only reasonable proportion of C1, e.g., = 0.25 or 0.5, the overall learning objective improves the performance our model. Further, we find that TCC is not very sensitive to the Gumbel softmax temperature X, while moderate hardness of the softmax produces the best results. Empirically, a large batch size benefits TCC, since more data can be involved in the subset of each cluster. Hence the aggregated features each batch be more representative. Fig. 4 (d) endorses this intuition. However, training with extremely large batch sizes may lead to out-of-memory problems with large images. To enable training single device, we opt to have a fixed batch size of 32 x K all experiments.
5.5 More Results
Training Time We compare the training epochs (Fig. 5 (a)) and training time (Fig. 5 (a)) of TCC and the re-implemented version of MiCE [73] with the same op- timizer setting. As discussed Sec. 3.3, MiCE obtains higher time complexity during training than TCC. This is reflected in Fig. (b), though not linearly proportional. In addition, TCC requires less training steps than MiCE to reach the best-performing results.
Conventional Clustering Losses During training, we also cache and observe the DEC loss [81], but we are not optimizing the model with it. In Fig. 5 (c), we show that by minimizing (Eq. (9)), the traditional DEC loss [81] also decreases. This implicitly endorse design.
2000
Figure 6: Histograms of cluster assign- ments during training on CIFAR-10 [44].
Assumptions in Design One merit of constrastive learning is that one does not need to assume any empiri- cal prior distribution the feature space, which benefits TCC when learning the cluster-level representations. The only assumption we employ is that the true posterior po (k|x) should be uniform simplify the computation of the KL-divergence in Eq. (6). As previously discussed, this conventional relaxation [69] is intuitively valid since we generally expect evenly assigned clusters. It is illustrated in Fig. 6 that TCC achieves this during training by p minimizing KL (ge( ())x pe(K ()) = log K H(go(k)))),
Conclusion
Inspired by the recent success of self-supervised learning, this paper proposes granulirraullrrty contrastive clustering framework to exploit the holistic context of a cluster in an unsupervised manner. The proposed TCC simultaneously learns instance- and cluster-level representations by leveraging cluster assignment variables. Cluster-level augmentation equivalents are derived to enable on-the-fi -ly contrastive learning on clusters. Moreover, by reparametrizing the assignment variables, TCC can be trained end-to-end without auxiliary steps. Extensive experiments validate the superiority of TCC, which consistently outperforms competitors on the five benchmarks often by large margins, echoing major motivation, i.e., not clustering alone.

---

Acknowledgments
This work is supported by the UKRI grant: Turing AT Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1. We also acknowledge the philanthropic support of the donors to the University of Oxford's COVID-19 Research Response Fund: BRD00230. We would like to thank the Royal Academy of Engineering and FiveAI.
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeftrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv:1603.0444: 2016.
[2] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy, Deep variational information
bottleneck. In ICLR, 2017. 6
[3] Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle, et al. Greedy layer-wise training of deep
networks. In NeurlPS, 2007. 7
[4] Christopher Bishop, Pattern recognition and machine learning. springer, 2006. 1, 6
[5] Deng Cai, Xiaofci Hc, Xuanhui Wang, Hujun Bao, and Jiawei Han. Locality preserving nonnegative matrix
factorization. IJCAI, 2009. 7
[6] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mathijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018. 8
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020. 6,
[8] Jianlong Chang, Yiwen Guo, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep discriminative clustering analysis. In CVPR, 2019.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In ICCV, 2017. 6,7
[10] Shlomo E Chazan, Sharon Gannot, and Jacob Goldberger. Deep clustering based mixture of autoencoders. In MI.SP, 2019. 6
[1]] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2, 4, 6,7
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04223 2020. 2, 6
[13] Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17907790,779, 1995. 6
[14] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 201 6,
[15] Luke Nicholas Darlow and Amos Storkey. Dhog: Deep hierarchical object grouping. arXiv preprint
arXiv:2003. 08882, 2020, 6,7
[16] Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lec, Hugh Salimbeni, Kai Arulku- maran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders.
arXiv preprint arXiv: 1611. 161.00644, 2016. 1,6
[17] Harrison Edwards and Amos Storkey Towards a neural statistician. In ICML, 2017. 6
[18] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In SIGKDD, 1996.
[19] Ji Feng and Zhi-Hua Zhou Deep miml network. AAAI, 2017.
[20] Brendan Frey and Delbert Dueck. Clustering by passing messages between data points. science,
315(5814)9972-9766 2007. 6
[21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NewrIPS, 2014. 6.7
[22] K Chidananda Gowda and Krishna. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 10(2):105-112 1978. 7
[23] Philip Haeusser, Johannes Plapp. Vladimir Golkov, Elie Aljalbout, and Daniel Cremers. Associative deep clustering: Training a classification network with no labels. In German Conference on Pattern Recognition. 2018. 6.7
[24] Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised
visual representation learning. In CVPR, 2020. 1, 2, 3, 4, 5, 6,
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016. 6
[26] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint
2015. 2
10

---

[27] Weihua Hu, Takeru Miyato, Seiya Tokui, Eichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In ICML, 2017. 6
[28] Jiabo Huang and Shaogang Gong. Deep clustering by semantic contrastive learning. arXiv preprint
urXiv:2103.0223 2021. 5, 6, 8
[29] Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognitiin,
pages 8849-8858, 2020. 6.7
[30] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification. 2(1):133-118 1985.
[31] Maximilian llse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In
ICML, 2018. 3,
[32] Fric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax. In ICLRR
2017. 4.5.6, 6,
[33] Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image
classification and segmentation. In ICCV, 2019, 6, 7, 8
[34] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe LLe Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102 009.059 2021. 6,
[35] Zhuxi Jiang. Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep emhedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.0514 2016.
[36] Zhuxi Jiang. Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep emhedding:
An unsupervised and generative approach to clustering. In I/CAI, 2017. 1, 6, 7
[37] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Fslami, Dan Rosenbaum, Oriol Vinyals,
and Yee Whye Teh. Attentive neural processes. In ICIR, 2019. 6
[38] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICIRR 2015. 7
[39] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 5, 6.7
[40] Diederik P Kingma. Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi Semi-supervised
learning with deep generative models. In NeurlPS, 2014. 5.5
[41] Alexander Kolesnikov. Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation
learning. In CVPR, 2019. 6
[42] Andreas Kopf, Vincent Fortuin, Vignesh Ram Somnath, and Manfred Claassen. Mixture-of-experts variational autoencoder for clustering and generating from similarity- based representations. arXiv preprint
arXiv:1910.07763, 2019. 6
[43] Hans-Peter Kriegel, Peer Kröger, Jörg Sander, and Arthur Zimek. Density- based clustering. Wiley
Interdisciplimary Reviews: Data Mining and Knowledge Discovery, 1(3):231 (19,223-220,, 2011. 6
[44] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical
Report, University of Toronto, 2009. 6, 7, 9
[45] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017. 4, 5
[46] Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[47] Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995 1995. 3
[48] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998. 3
[49] Juho Lec, Yoonho Lec, Jungtack Kim, Adam Kosiorek, Scungjin Choi, and Yee whye Teh. Set transformer:
A framework for attcntion-based permutation-invarian neural networks. In ICML, 2019. 3, 6
[50] Juho Lee, Yoonho Lee, and Yee whye Teh. Deep amortized clustering. arXiv preprint arXiv: 19999.13433. 2019. 6
[51] Chongxuan Li, Max Welling, Jun Zhu, and Bo Zhang. Graphical generative adversarial networks. In NeurlPS, 2018.
[52] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint 3xXiv:2005.04966 2020. 6
[53] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering.
arXiv preprint arXiv:2009.006 099877 2020. 5, 6,7
[54] Zhihan Li. Youjian Zhao, Haowen Xu, Wenxiao Chen, Shangqing Xu. Yilin Li, and Dan Pei. Unsupervised clustering through gaussian mixture variational autnencoder with non-raparameterized variational inference
and std anncaling. In IJCNN, 2020. 1
[55] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):5579-2600 2008. 7
[56] James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Berkeley symposium on mathematical statistics and probability, 1967. 1, 6,
[57] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017. 4, 6,

---

[58] Sudipto Mukherjee, Himanshu Asnani, Eugene Lin. and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In AAAI, 2019.
[59] Andrew Ng. Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In NeurIPS, 2001. 1,6
[60]
Chuang Niu and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. arXiv preprint
2021.7 7
[61] Chuang Niu, Jun Zhang. Ge Wang, and Jimin Liang. Gateluster: Self-superrised gaussian- attention network for image clustering. In ECCV, 2020. 6,
[62] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXivv/807.0374. 2018. 1,2,
[63] Ari Pakman, Yueqi Wang, Catalin Mitelut, JinHyung Lee, and Liam Paninski. Neural clustering processes.
In ICML, 2020. 6
[64] Sungwon Park, Sungwon Han, Sundong Kim, Danu Kim, Sungkyu Park, Scunghoon Hong, and Mecyoung Cha. Improving unsupervised image clustering with robust learning. arXiv preprint arXiv:2012. 1110000...111.,.. 1500 2020. 7
[65]
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In JCLR, 2016. 7
[66]
Jayanth Reddy Regatti, Aniket Anand Deshmukh, Eren Manavoglu, and Urun Dogan. Consensus clustering with unsupervised representation learning. In IJCNN, 2021. 5
[67] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.
International Journal of Computer Vixion, 115(3):211-222, 2015. 6
[68] Guy Shiran and Daphna Weinshall. Multi-modal deep clustering: Unsupervised partitioning of images. In ICPR, 2021. 6.7
[69] Kihyuk Sohn, Honglak Iec, and Xinchen Yan. Leamingg structured output representation using deep conditional generative models. In NeurlPS, 2015. 4, 5, 9
[70] Alexander Strehl and Joydeep Ghosh. Cluster ensembles-a knowledge reuse framework for combining multiple partitions. Journal of machine learning research. 3(Dec):583 617. 2002.
[71] Yaling Tao, Kentaro Takagi, and Kouta Nakata. Clustering friendly representation learning via instance
discrimination and feature decorrelation. In ICLR, 2021. 7
[72] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 1eXiv:/906.000 1906.05499 2019. 6
[73] Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image clustering. In ICLR, 2021. 1, 5, 6, 7, 9
[74] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Procsmans, and Luc Van Gool. Scan: Learning classify images without labels. In ECCV, 2020. 6
[75] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research. 11())), 2010. 7
[76] Edward Wagstaff, Fabian Fuchs, Martin Engelcke, Ingmar Posner, and Michael A Oshorne. On the limitations of representing functions on sets. In ICML, 2019. 6
[77] Jingyu Wang. Zhenyu Ma, Feiping Nie, and Xuelong Li, Progressive self-supervised clustering with novel category discovery. IEEE Transactions Cybernetics, 2021. 6
[78] Joe H Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American statistical
association, 58(301):236 244, 1963. 6
[79] Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Honghin Zha. Deep comprehensive correlation mining for image clustering. In ICCV, 2019. 6, 7
[80] Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 1,2
[81] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In
ICML, 2016. 1, 6, 7, 9
[82] Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. arXiv preprint arXiv:2008.00 04340, 2020. 6
[83] Bo Yang. Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means-friendl spaces:
Simultancous deep learning and clustering. In ICML, 2017. 6
[84] Bo Yang, Sen Wang, Andrew Markham, and Niki Trigoni. Robust attentional aggregation of deep feature sets for multi-view 3d reconstruction. International Journal of Computer Vision, 128( 1))53733,, 2020. [85] Jianwei Yang. Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016. 6.7
[86] Linxiao Yang, Ngai- Man Cheung. Jiaying Li, and Jun Fang. Deep clustering by gaussian mixture variational autoencoders with graph embedding. In ICCV, 2019. 1,6
12

---

[87] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In NewrIPS, 2017. 2, 6
[88]
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In
CVPR. 2010. 7
[89]
Dejiao Zhang. Feng Nan, Xiaokai Wei, Shangwen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati. Andrew Arnold, and Bing Xiang, Supporting clustering with contrastive learning. In NAACL, 2021. 1, Dejian Zhang. Yifan Sun, Brian Friksson, and Laura Balzano. Deep unsupervised clustering using mixture of autoencoders. arXiv preprint arXiv:1712.07720 2017. Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: efficient data clustering method for very large databases. SIGMOD, 1996. 6
[90]
[91]
[92] Junjie Zhao, Donghuan Lu, Kai Ma, Yu Zhang, and Yefeng Zheng. Deep image clustering with category- style representation. In ECCV, 2020. 7
[93] Chengxu Zhuang, Alex Zhai, and Daniel Yamins, Local aggregation unsupervised learning of visual emheddings. In ICCV, 2019.
13


Reviews:
[{"review_id": "vU7CCrQdzr", "reviewer": "Reviewer_XKuz", "summary": "For image clustering, this paper extends the mainstream contrastive learning paradigm to a cluster-level scheme, and proposes twin-contrast clustering. Heuristic cluster augmentation equivalents are presented to enable cluster-level contrastive learning. Besides, this paper also derives the evidence lower-bound of the instance-level contrastive objective with the assignments. Experiments on several datasets demonstrate its superiority.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) The overall writing and presentation are good.\n\n(2) The idea for cluster-level augmentation and contrastive learning is interesting.\n\n(3) Experimental results are good.\n\nCons:\n\n(1) How to achieve the irrelevant minorities augmentation? Where does the small proportion of irrelevant data come from? Please give more explanation.\n\n(2) The backbone might be different from other baselines. Please analyze the difference and its influence.\n\n(3) Please analyze the influence of training epochs. I notice that at least 1,000 epochs are needed for training, which is much larger than other methods. \n\n(4) Compared with the improvement on CIFAR and ImageNet-Dog, the results on STL-10 and ImageNet-10 are not satisfying. The explanation in subsection 5.3 is so simple. Please give more analysis about this.\n\n(5) In experiments, the largest dataset only contains 20 clusters. Will the proposed method be effective on datasets with more categories? More experiments are suggested.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "URPl9f1IjO", "reviewer": "Reviewer_sHMT", "summary": "This paper proposed a new clustering method called Twin-Contrast Clustering(TCC), which extends the mainstream contrastive learning paradigm to a cluster-level scheme. TCC simultaneously learns instance- and cluster-level representations by leveraging cluster assignment variables. And TCC uses Gumbel Softmax to reparametrize the assignment variables, which makes TCC is able to be trained end-to-end without auxiliary steps.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tThis paper is well-written and easy to follow. Most of the details are fully illustrated and the ablation study is quite adequate.\n2.\tThe network structure is very cleverly designed. The authors give in-depth thoughts on how to calculate the clustering level representation and how to introduce clustering information into the instance level contrastive learning. Gumbel Softmax is also a reasonable approach to obtain a more discrete category representation.\n\nCons:\n1.\tIs TCC more limited on large-scale data with more categories? According to the description in the methods section, batch_size=32xK and queue_size L=100xK, so the complexity of the algorithm and the memory requirements increase significantly when there are more categories. It would be helpful to experiment with data such as TinyImageNet with 200 classes in CC or ImageNet-100 for the main conclusion of the paper.\n2.\tFrom the experimental results, cluster-level augmentation has a large impact on the final performance, but the explanation of cluster-level aug(b) is not sufficient. From the descriptions of Line113-115 and Line299-301, aug(b) plays the role of using the default soft assignment instead of hard assignment, so how does hard assignment work? Is only the maximum probability feature retained?\n3.\tThe paper lacks a comparison of TCC with the best unsupervised methods, especially DeepCluster-V2 and SwAV, which introduce the concept of clustering. From aug(b) and the larger $\\lambda$ used in Gumbel Softmax, TCC still needs a soft semantic representation to help learning, which plays a similar effect to the prototype used in SwAV, so SwAV is a worthwhile baseline for comparison.\n4.\tFinally a small question, given that most methods deal with balanced category data, and TCC introduces a 1/K posterior probability prior, does this mean that TCC does not perform as robust as methods like CC when dealing with category unbalanced data?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to Cons in Main Review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "QaH0_av-aZD", "reviewer": "Reviewer_VWHL", "summary": "The authors propose a method for end-to-end deep clustering. Authors utilizes NCE loss and proposes a method to produce cluster level loss. They motivate their approach and supply empirical evaluation.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a method for end-to-end deep clustering. Authors utilizes NCE loss and proposes a method to produce cluster level loss. They motivate their approach and supply empirical evaluation. The paper is very clearly written and the problem under is considerations very important topic. As the title of the paper and introduction hints clustering is depending on the representation of one single point but representation of the group of point. Unfortunately, authors didn’t consider appropriate state-of-the art methods as baselines. A small list is given below:\na)\tClustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation (available in ICLR webpage since October 2020)\nb)\tConsensus Clustering with Unsupervised Representation Learning (available in arxiv since October 2020)\nc)\tRepresentation Learning for Clustering via Building Consensus (available in arxiv since May 2021)\nI would like to be fair to authors and I will only consider some parts of c) in the next steps. However, I don’t see any reason why a) and b) are not considered as  baselines or discussed in their paper. Both papers are end-to-end and contains relevant results. The method given in a) is outperforming the proposed method on CIFAR-10, STL-10, ImageNet-10. On ImageNet-10 the difference is app. 10%. \nOn the other hand, the method b) has comparable results however the main motivation of the paper under review is very similar to it. Please note that b) doesn’t use NCE type loss but it builds on BYOL.\nAlthough the paper c) is a direct baseline to the paper under review (NCE is used, different types of clustering constraints are handled) and it improves state of the art, I will ignore the clustering results. However, c) also proposes a new method of evaluating deep clustering method and points out several drawbacks of the clustering methods e.g., sensitivity to hyper-parameter tuning. I would like to see similar analysis from authors in the author response.\n\nAll these points bring questions about the novelty of the method and performance of the method. I have some concerns; I would like to see significant amount of improvements.\n\n<Please see the discussion below. After the rebuttal and discussion with authors, I have modified my score>", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "Mb14z_UvQ4r", "reviewer": "Reviewer_YYDW", "summary": "This paper extends the mainstream contrastive learning paradigm to a cluster-level scheme. Specifically, the author propose twin-contrastive clustering (TCC),  which links the instance-level learning track with the cluster-level one. Extensive experiments show the advantages of TCC.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of this paper is very intuitive, and in fact, some relevant papers (as described in the paper) have been explored . But interestingly, the authors compile contrastive learning while partially borrow from deep set representations, realizing the two-level contrastive learning in a novel way.\nSome issues:\n(1) In 'Instance-Level Contrastive Learning' part, for the term e, why use f_\\theta(x) + NN_\\theta(c)? How about  using concatenate operation？\n(2) In 'Cluster-Level Augmentation Equivalents', why consider 'irrelevant minorities'? How to generate and use such augmentation?\n(3) In fact, SwAV [a] is also a kind of contrastive learning with both instance-level and cluster-level, so it should be added as a baseline.\n(4) Compared with CC, will the improvement of TCC performance benefit from model capacity increase? and in fact TCC has more trainable parameters.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "There are not ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel extension to contrastive learning, using both cluster level and instance level tasks in representation learning. The bulk of the reviewers found this method interesting and well-motivated, and from my perspective as well using this additional structure is a good direction in contrastive learning / representation learning. The authors felt their work should be viewed independently from SwAV, but the reviewers clearly saw the strong overlap between what the two models do. The reviewers added SwAV as a baseline, and their model performed favorably on their benchmark tasks. However, I encourage the authors to be generous on the connection between their work and SwAV in particular in the final draft, while the downstream applications are similar, I believe the motivations are very similar. Finally, the reviewers requested more benchmarks, such as TinyImagenet, and this satisfied the reviewers. Therefore I recommend acceptance as a poster.

A note that I do believe that [a] "Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation" should be included in the final draft. I do not believe that the significance of the submitted work should be judge by [a], as it is quite new and could have feasibly fallen under the radar with no ill intent. But I believe both papers are doing similar things and as such this paper could do the community a service by including results in [a] and providing this as additional context to their results. I strongly encourage the authors to do this.
Decision: Accept (Poster)