Exploring Non-Contrastive Representation Learning for Deep Clustering
Authors: Zhizhong Huang
Jie Chen
Junping Zhang
Hongming Shan
Venue: ICLR 2022 Submitted (2022)
Keywords: Image Clustering
Representation Learning
Self-supervised Learning

Abstract: Existing deep clustering methods rely on contrastive learning for representation learning, which require negative examples to form an embedding space where all instances are well-separated. However, the negative examples inevitably give rise to the class collision issue, compromising the representation learning for clustering. In this paper, we explore the non-contrastive representation learning for deep clustering, termed NCC, which is based on BYOL, a representative method without negative examples.  First, we propose a positive sampling strategy to align one augmented view of instance with the neighbors of another view so that we can avoid the class collision issue caused by the negative examples and hence improve the within-cluster compactness. Second, we propose a novel prototypical contrastive loss, ProtoCL, which can encourage prototypical alignment between two augmented views and prototypical uniformity, hence maximizing the inter-cluster distance.  Moreover, we formulate NCC in an Expectation-Maximization (EM) framework, in which E-step utilizes spherical k-means to estimate the pseudo-labels of instances and distribution of prototypes from the target network and M-step leverages the proposed losses to optimize the online network. As a result, NCC is able to form an embedding space where all clusters are well-separated and within-cluster examples are compact. Experimental results on several clustering benchmark datasets as well as ImageNet-1K demonstrate that the proposed NCC outperforms the state-of-the-art methods by a significant margin.

Under review as a conference paper at ICLR 2022
EXPLORING NON-CONTRASTIVE REPRESENTATION LEARNING FOR DEEP CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
Existing deep clustering methods rely on contrastive learning for representation learning, which requires negative examples form an embedding space where all instances are well-separated. However, the negative examples inevitably give rise to the class collision issue, compromising the representation learning for clus- tering. In this paper, explore non-contrastive representation learning for deep clustering, termed NCC, which is based on BYOL, a representative method with- negative examples. First, we propose align one augmented view of instance with the neighbors of another view in the embedding space, called positive sam- pling strategy, which avoids the class collision issue caused by the negative ex- amples and hence improves the within-cluster compactness. Second, we propose encourage alignment between augmented views of one prototype and uni- formity among all prototypes, named prototypical contrastive loss or ProtoCL. which can maximize the inter-cluster distance. Moreover, we formulate NCC in an Expectation- Anpectatoo-MMxiiizzaiin Maximization (EM) framework, in which E-step utilizes spherical k-means to estimate the pseudo-labels of instances and distribution of prototypes from target network and M-step leverages the proposed losses to optimize an on- line network. As result, NCC forms an embedding space where all clusters are well-separated and within-cluster examples are compact. Experimental results on several clustering benchmark datasets including ImageNet- 1K demonstrate that NCC outperforms the state-of-the-an afetthe-rrt methods by significant margin.
INTRODUCTION
Deep clustering is gaining considerable attention as it can learn representation of images and perform clustering in an end-to-end fashion. Remarkably, contrastive
sarming--bssed methods (Wang et al. 2021 Van Gansbeke et al.] 2020 Li et al.) 202 b Tao al., 2021 Tsai et al., 2021] Niu et al.
2021 have become the main thrust to advance the representation of images on several complex
benchmark datasets, significantly contributing to the clustering performance. In addition, some
contrastive learning methods such as MoCo (He et al. 2020) and SimCLR (Chen et al. 2020) usually require specially designed losses (Wang al.] 2021 Li et al. 2021a Tao et al.) [021] T sai al.
2021 or an extra pre-training stage for more discriminative representations (Van Gansbeke et al. 2020
Niu et al. 202I).
Although achieving promising clustering results, contrastive learning requires a large number of negative examples to achieve the instance-wise discrimination in embedding space where all instances are well-separated. The constructed negative pairs usually require large batch size (Chen al. 2020), memory queue (He al. 2020). or memory bank (Wu al., 2018), which not only bring extra computational cost but also give rise to class collision issue (Saunshi et al. 2019). Here, the class collision issue refers to that different instances from the same semantic class are regarded negative pairs, hurting the representation learning for clustering, A question naturally arises: negative examples necessary for deep clustering?
Another kind of self-supervised learning is the non-contrastive methods such as BYOL Grril al. 2020) and SimSiam (Chen He 2021). which use the representations of one augmented view to predict another view. Their success demonstrates that negative examples not the key to avoiding representation collapse. However, to the best of our knowledge, almost all recent successful liter- ature of deep clustering is built upon contrastive learning-based methods such as MoCo (He al.

---

Under review as a conference paper at ICLR 2022
2020) and SimCLR (Chen et al. 2020). There is general consensus that the negative examples helpful to stabilize the training of representation learning for deep clustering. As discussed in (Wang & Isola 2020), the typical contrastive loss be identified into two properties: 1) aligument term to improve the closeness of positive pairs; and 2) uniformity term to encourage instances to be uniformly distributed on unit hypersphere. In contrast, non-contrastive methods such BYOL only optimize the alignment term, leading to unstable training and suffering from the representation collapse-which may be worsen when adding extra losses.
To tackle the class collision issue, we explore the non-contrastive representation learning for deep clustering, termed non-contrastive clustering or NCC, which based on BYOL, representative method without negative examples. First, instead of negative sampling that is a double-edged sword, ie., causing class collision issue but improving the training stability, we propose to align one aug- mented view of the instance with the neighhors of another view in the embedding space, called positive sampling strategy, which can avoid the class collision issue and hence improve the within- cluster compactness. Second, for the clustering task, the different clusters are truly negative pairs for contrastive loss. To this end, we propose to encourage the alignment between two augmented views of prototypes and the uniformity among all prototypes. named prototypical contrastive loss or ProtoCL, which can maximize the inter-cluster distance. Moreover, we formulate our method into an EM framework, in which we iteratively perform E-step as estimating the pseudo-labels of instances and distribution of prototypes via spherical k-means based on the target network and M- step as optimizing the online network via the proposed losses. As a result, NCC is able to form an embedding space where all clusters are well-separated and within-clusee examples are compact. The contributions of this paper are summarized follows:
We explore the non-contrastive representation learning for deep clustering by proposing non- contrastive clustering or NCC, which is based on the Bootstrap Your Own Latent (BYOL), a representative method without negative examples.
We propose positive sampling strategy to augment instance alignment by taking into account neighboring positive examples in the embedding space, which can avoid the class collision issue and hence improve the within-cluster compactness.
We propose a novel prototypical contrastive loss or ProtoCI, which can align one augmented view of prototypes with another view and encourage the uniformity among all prototypes unit hypersphere, hence maximizing the inter-cluster distance.
We formulate our method into an EM framework, in which we can iteratively estimate the pseudo- labels and distribution of prototypes via spherical k-means based on the target network and opti- mize the online network via the proposed losses.
Extensively experimental results on several benchmark datasets as well as ImageNet-1K IK demon- strate that NCC outperforms the existing state-of-the. the-art methods by a significant margin.
2 RELATED WORK
Deep clustering can be significantly advanced by discriminative representations. Examples of tra- ditional deep clustering methods include: Xie et al. (2016); Yang et al. (2017) use autoencoders to simultaneously perform representation learning and clustering; Chang et al. (2017): Haeusser et al.) (2018); Wu et al.) 2019): Ji et al. (2019) learn pair-wise relationships between original and aug- mented instances. However, they often suffer from inferior performance on some complex datasets such as CIFAR-20. Inspired by the success of contrastive learning, recent studies turn to exploit the discriminative representations learned from contrastive learning assist the downstream clus- tering tasks (Van Gansbeke et al.] [2200] Niu et al., (02T) or simultancously optimize representation learning and clustering (Tao et al. 2021] Tsai al., 2021] Li et al.] 2021a] Shen al., (021). SCAN (Van Gansbeke et al. (0200) uses the model pre-trained by SimCLR to yield the confident pseudo-labels. IDFD (Tao et al. [021) proposes to perform both instance discrimination and fea- ture decorrelation. GCC (Zhong et al. 2021) and WCL (Zheng et al.) (02T) build graph to label the neighbor samples as pseudo-positive examples, however, they still suffer from the class colli- sion issue due to the contrastive loss involved and these posiiivv examples that may not be truly positive. All of them are built upon the contrastive learning framework, which means that they require large number of negative examples for training stability, inevitably giving rise to class
2

---

Under review as a conference paper at ICLR 2022
collision issue. Different from prior work, this paper explores the con-contrastive self-supervised methods, ie., BYOL, to achieve both representation learning and clustering. We note that Regatti al. (202T) Lee et al. (020)) have tried to build the clustering framework based BYOL, however, their methods do not consider improving within-cluster compactness and maximizing inter-cluster distance like ours. Therefore, to the best of our knowledge, this is the first successful attempt that introduces the non-contrastive representation learning into deep clustering that yields a substantial performance improvement over previous state-of-the-art methods. In Appendix [ we present re- lated work on self-supervised learning and difference from existing methods including CC (Li et al. 2021b), GCC (Zhong et al. 202T). WCL (Zheng et al. 202T), and PCL (Li et al. 2021a).
3 PRELIMINARY
The most successful self-supervised learning methods in recent years be roughly divided into contrastive (Chen et al.) 2020) He et al.) 2020) and non-contrastive (Grill et al. 2020 Chen & He 202T] Here, we briefly summarize their formulas and discuss their difference.
Contrastive learning. Contrastive learning methods perform instance-wise discrimination (Wu al., (2188) using the InfoNCE loss (Oord al., 2018). Formally, assume that we have one instance x, its augmented version x by using random data augmentation, and set of M negative examples drawn from the dataset, {2r 22 ..... 2x J. The contrastive learning aims to learn an embedding function f that maps æ onto a unit hypersphere, in which the InfoNCE loss can be defined as:
exp f(a)))
Loontr = - log
(1)
exp (f(x))
+ Ens exp (((=))(((7)/))
M 2 f(x))) log Σ. exp (f(m) ((T))/))
(2)
where the first and second terms in Eq. refer as instance alignment and instance uniformity, respectively. Here, we assume that the output of f(.) is Eg normalized. That is, the representation is on a unit hypersphere. The temperature 7 controls the concentration level of representations; please refer to (Wang & Liu 2021) for detailed behaviors of in the contrastive loss. Intuitively, the In- foNCE loss aims to pull together the positive pair (æz, ax+) from two different data augmentations of the same instance, and push away from M negative examples of other instances. As discussed in (Wang& & Isola, 2020), when M + co, the InfoNCE loss in Eq. (I) can be approximately de- coupled into two terms: aligument and uniformity, shown in Eq. ( Despite the alignment term closes the positive pair, the key to avoiding representation collapse is the uniformity term, which makes the negative examples uniformly distributed the hypersphere. Although beneficial, the negative examples inevitably lead to the class collision issue, hurting the representation learning for clustering.
Non-contrastive learning. Non-contrastive leamnin-baased methods have shown more promising results than contrastive learning for representation learning and downstream tasks (Ericsson al.) 2021]. Non-contrastive methods only optimize the aligmment term in Eq. (2) to match the repre- sentations between two augmented views. Without negative examples, they leverage an online and a target network for two views, and use predictor network to bridge the gap between these two views. They also stop the gradient of the target network avoid the representation collapse. In particular, if r = 0.5, the loss used in (Grill et al. 2020
Chen & He 2021) can be written as:
Lnon-contr 22 (f(x)) f((x) ||9 (f(=)) - f(x [=x))=*)] I|? 1 const,
where the predictor; f and f' are the online and target networks, respectively; the outputs of g(f(-) and f'(-) are fz-normllieed. However, as mentioned in (Fetterman & Albrecht 2020). the non-contrastive learning methods often suffer from unstable training and highly rely on the batch-statistics and hyper-parameter tuning to avoid representation collapse. Even though Grill et al. (2020); Richemond et al. (2020) have proposed to use some tricks such SyncBN (loffe & Szegedy 2015) and weight normalization (Qiao et al.) 2019) to alleviate this issue, the additional computation cost is significant. Without negative examples, the collapse issue could be worsen when adding additional clustering losses for clustering task; see Fig. Al for the analysis of applying PCL. (Li et al. 2021a) to the BYOL.
3

---

Under review as a conference paper at ICLR 2022
In nutshell, most of existing successful deep clustering methods are based contrastive learning for representation learning giiingg rise to class collision issue-while the non-contrastive learning, due to unstable training with additional losses, is not yet ready for deep clustering. To that end, explore the non-contrastive learning, i.e. BYOL, for deep clustering with positive sampling strat- egy and prototypical contrastive loss avoid the class collision issue, improve the within-cluster compactness, and maximize the inter-class distance.
NON-CONTRASTIVE CLUSTERING
Poullive
(
Optimization
Feature Extraction
Sampling
Prodictor
Data
Input
Network
M-step
Moving Average
Representations
Target Network
every
E-step
Estimate p(
p(kiz)
Target Network
Clustering
For epuch
Figure The overall framework the proposed NCC an EM framework.
Fig. presents the overall framework of the proposed NCC. Based on BYOL, NCC is comprised of three networks: an online, target, and predictor. In Sec. 4.1 we propose a positive sampling strategy augment instance alignment to improve the within-cluster compactness. In Sec. 4.2 prototypical contrastive loss is introduced to maximize the inter-cluster distance using the pseudo- labels from k-means clustering, which can encourage uniform representations. Finally, we formulate NCC into an EM framework to facilitate the understanding of training procedure in Sec. 4.3
4.1 POSITIVE SAMPLING STRATEGY
awayy
pull together
(A) Negative Sampling
(b) Positive Sampling
(c) Prototypical Contrustive Loss
Figure 2: Ilstrrtitonn of the proposed techniques compared to negative sampling, (a) Negative sam- pling in contrastive learning giving rise class collision issue. (b) The proposed positive sampling encouraging the aligmment between neighbors of one view with another one. (c) The proposed pro- totypical contrastive loss encouraging prototypical alignment and prototypical uniformity.
The negative examples essential for contrastive Icarning- baaning-based deep clustering to stabilize the training of representation learning, at the cost inevitable class collision issue (Saunshi et al.) 2019), shown in Fig. a This issue can hurt the representation learning for clustering as the instances from the same class/cluster -sould close to each other- could be treated as negative pairs and are pushed away during training, discouraging the within-cluster compactness.
To address the class collision issue, we resort to non-contrastive learning- based methods for rep- resentation learning, which longer need negative examples. Although cannot optimize the

---

Under review as a conference paper at ICLR 2022
uniformity term like contrastive loss, our idea is to optimize the opposite of the uniformity instead. That is, we aim to encourage the neighboring examples around one augmented view, sampled in the embedding space, to be aligned with another view, as shown in Fig. 2bb). Our motivation is that although we cannot guarantee the negative examples in contrastive loss are from different classes, certainly assume that the neighboring samples around one view in the embedding space are positive with respect to another view and belong to the same class. Therefore, we propose a positive sampling strategy augment the instance alignment in Eq. by taking into account the neighboring samples towards within-cluster compactness.
Specifically, we model the representation of one augmented view of an instance as Gaussian dis- tribution in the embedding space, which can be formulated as follows:
(4) where I represents the identity matrix and is positive hyperparameter controlling how many samples around one view can be treated as positive pairs with another view. However, the sampled examples from Eq. (4) cannot allow the enror to be backpropagated through the network update the network parameters. We employ the reparametrization trick (Kingma & Welling, (2014) to achieve the backpropagation. As result, the positive sampling strategy can be implemented as follows:
1 ~ M (f(z)),
v セ= f(x) + ge, ~ N' (0, I).
(5)
Therefore, we can augment the instance alignment in Eq. (3) by taking into account the neighboring samples to encourage the within-cluste compactness. With only sampling one example from the Gaussian distribution, the augmented instance alignment term can be formally defined as:
[g( f' 112 || (J(z)) + σε) - )a
Lang- ins
(6)
The benefits of the proposed positive sampling are summarized as follows.
Improved within-cluster compactness. The conventional instance alignment in Eq. ( only en- courages the representation of one augmented view to be close to another view. In the context of clustering, such compactness is instance wise and neutral for the clustering. Put differently, all in- stances are treated as cluster centers and the semantic class information cannot be captured at only instance level. In contrast, our augmented instance alignment in Eq. (6) encourages neighboring examples around one augmented same/different augmented examples of different instances within the same cluster- to be positive pairs with another view. This is helpful to improve within-cluster compactness.
different augmented examples of the same instance
Avoidable class collision issue. As we mentioned before, class collision issue induced by the negative examples indicates that we cannot guarantee that the negative examples are from different clusters. However, our positive sampling strategy can guarantee that the positive examples around one instance, sampled in the embedding space, are from the same cluster as the instance, getting rid of class collision issue. The difference from recent work (Zhong et al.) [021]] Zheng et al.) 2021 is discussed in Appendix A We note that our positive sampling strategy does not consider uniformity, which is solved by the proposed prototypical contrastive loss in Sec. 4.2
4.2 PROTOTYPICAL CONTRASTIVE LOSS
A good clustering is supposed to have distinct semantic prototypes/clusters. Assume that the dataset has K clusters, where K a predefined hyperparameter, naturally constructs a contrastive loss for these K prototypes as for one prototype, the remaining K 1 prototypes are definitely negative examples. Therefore, we propose prototypical contrastive loss or ProtoCL, which encourages the prototypical alignment between two augmented views and the prototypical uniformity, hence maximizing the inter-cluster distance.
Specifically, assume we have K prototypes from the online network, {ui M2...., Mx }. and an- ). our proposed ProtoCL, illustrated
other prototypes from the target network, in Fig. (c). is given follows:
K
exp(xT
Cpel=
logg exp(uTu//T) + Σj=roEk exp(uTTu//T)
K
k=l
K
-I Exe log Σ م Dir exp(HT/5/T), K k=l
(8)
K k=l
5

---

Under review as a conference paper at ICLR 2022
where the first and second terms in Eq. (8) refer to prototypical alignment and prototypical uni- formity, respectively. Here, the cluster centers b4d and u4 are computed within a mini-batch as fo = Zmen f(a)) TEmen ((]]a) and U = II
E Se (( and p(k[2) is the cluster as-
signment posterior probability. When K > [B], P(
obvious that the mini-bater cannot cover all clusters. To this end, we zero out the losses and logits of empty clusters for each iteration; see the pseudocode in Appendix D more details.
Clearly, our ProtoCL is quite similar to conventional contrastive loss in Eq. ① but for prototypes with non-contrastive representation learning framework. The prototypical alignment is to align the prototypes derived from the online network with the ones from the target network, which can sta- bilize the update of the prototypes. The prototypical uniformity is to encourage the prototypes to be uniformly distributed on unit hypersphere, which can maximize the inter-cluster distance. The difference from ProtoNCE in (Li et al. 2021a] a) is discussed in Appendix A
4.3 EM FRAMEWORK
We formulate our NCC into an EM framework to facilitate the understanding of the training proce- dure, detailed in Fig. 1 and derived in Appendix
E-step. This step aims estimate p(klar). We perform spherical k-means algorithm on the fea- tures extracted from the target network since the target network performs more stable and yields more consistent clusters, similar to BYOL and MoCo. Although we need an additional k-means clustering to obtain the cluster pseudo-labels p(k|zz) for every epochs, we found that even with a larger r, rather than every epoch r 1, our method can still produce consistent performance im- provement over the baseline methods. Therefore, our method will not introduce much computation cost and is robust to the cluster pseudo-labelss see detailed results in Fig. A2 The analysis of com- putational cost is discussed in Appendix [ Finally, with p(k) (z), we build the cluster centers within a mini-batch without additional memory like queue (He et al. 2020) or bank (Wu et al., 2018).
M-step. Combining the augmented instance alignment loss in Eq. (6) and the proposed ProtoCI. in Eq. yields our objective function for M-step as follows:
L Laug-in ins + Apel Spel,
where Apel controls the balance between two loss components. Therefore, there are only two addi- tional pyper-parameters compared to original BYOL, including: o in Laug-ins ins and the loss weight Apel: detailed results of these two hyper- pareee--rameterss in Figs. A3 and A4
EXPERIMENTS
Table I; Summary of the datasets.
We conducted experiments on six
Dataset
Split
Samples
Classes
Image Size
benchmark datasets, including CIFAR-
CIFAR 10
Train-Test
60,000
10
32x32
10
(Krizhevsky et al.] 2009). CIFAR-
CIFAR-20
Teat
60,000
32x
20
Krizhevsky et al. 2009). STL-
STL-10 10
TrainTTsst
13,000
10
96x96
10
CCoates et al. 2011 ImageNet-
ImageNet-10
Train
96x96
ImageNet Dogs
Train
19,500
15
96x96
10
(Chang et al. 2017). Net-
ImageNet-
Train
1,281,177
1,000
224x224
Dogs (Chang al. (207)). and ImageNet-
IK (Deng et al.] 2009), which are summarized in Table | We note that CIFAR-20 contains 20
superclasses of 100. This paper follows the experimental settings widely used in deep
clustering work (Chang et al. 2017: Wu et al., 2019 Ji et al.) 2019: Tsai et al. 2021 TToo et al., 2021]. including the image size, backbone and train-test split. We employ three common metrics to evaluate the clustering performance, including Normalized Mutual Information (NMI), Cluster Accuracy (ACC), and Adjusted Rand Index (ARI) for the first five datasets. Following (Li et al.) 2021a), we report Adjusted Mutual Information (AMI) to evaluate the clustering performance for ImageNet-1K: IK. The results are presented in percentage (%) and the higher the better clustering performance. For fair comparisons, we use ResNet-34 (He et al. 2016) the backbone to report the results in Table 2 Unless noted otherwise, we use ResNet-18 18 for the rest of experiments. We each experiment three times and report the mean and standard deviation
the final results. We provided detailed training settings in Appendix C We also provide the pseudocode of NCC for better understanding in Appendix D The source code will be publicly available upon acceptance.

---

Under review as a conference paper at ICLR 2022
5.1
MAIN RESULTS
Table Clustering results (%) of various methods on five benchmark datasets. The best and second
best results are shown in bold and underline, respectively.
Dataset Method
CIFAR-10 10
CIFAR-20
STL-10
ImageNet-10
10
ImageNet-
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
k-meuns
8.7
22.9
4.9
8.4
13.0
2.8
12.5
19.2
6.1
11.9
24.1
5.7
5.5
10.5
2.0
SC
10.3
24.7
8.5
9.0
13.6
2.2
9.8
15.9
4.8
15.1
27.4
7.6
3.8
11.1
1.3
AE
23.9
31.4
16.9
10.0
16.5
4.8
25.0
30.3
16.1
21.0
31.7
15.2
10.4
18.5
7.3
VAE
24.5
29.1
16.7
10.8
15.2
4.0
20.0
28.2
14.6
19.3
33.4
16.8
10.7
17.9
7.9
JULE
19.2
27.2
13.8
10.3
13.7
3.3
18.2
27.7
16.4
17.5
30,0
13.8
5.4
13.8
2.8
DEC
25.7
30.1
16.1
13.6
18.5
5.0
27.6
35.9
18.6
28.2
38.1
20.3
12.2
19.5
7.9
DAC
39.6
52.2
30.6
18.5
23.8
8.8
36.6
47.0
25.7
39.4
52.7
30.2
21.9
27.5
11.1
ICC
51.3
61.7
41.1
25.7
43.1
29.5
DCCM
49.6
62.3
40.8
28.5
32.7
17.3
37.6
48.2
26.2
60.8
71.0
55.5
32.1
38.3
18.2
PICA
56.1
64.5
46.7
29.6
32.2
15.9
78.2
85.0
73.3
33.6
32.4
17.9
CC
70.5
79.0
63.7
43,1
42.9
26.6
76.4
85,0
72.6
85.9
89.3
82.2
44.5
42.9
27.4
SCAN
79.7
88.3
77.2
48.6
50.7
33.3
80.9
69.8
64.6
GCC
76.4
85.6
72.8
47.2
47.2
30.5
68.4
78.8
63.1
84.2
90.1
82.2
49.0
52.6
36.2
MICE
73.7
83.5
69.8
43.6
44.0
28.0
63.5
75.2
57.5
42.3
43.9
28.6
IDFD
71.1
81.5
66.3
42.6
42.5
26.4
64.3
75.6
57.5
89.8
95.4
90.1
54.6
59.1
41.3
PCL
80.2
87.4
76.6
52.8
52.6
36.3
71.8
41.0
67.0
84.1
90.7
82.2
44.0
41.2
29.9
BYOL
81.7
89.4
79.0
55,9
56.9
39.3
71.3
82.5
65.7
86.6
93.9
87.2
63.5
69.4
54.8
10.1
1.0.6
10.11
1.0.3
1.188
1.0.2
100.9
1.0.5
11.33
1.0.2
10.1
1.0.2
1.2.2
1.3.0
12.9
NCC
88.6
94.3
88.4
60.6
61.4
45.1
75.8
86.7
73.7
89.6
95.6
90.6
69.2
74.5
62.7
(ours)
10.1
11.1
112.4
1.0.2
k-means [Looyd]]] 1982
(Zelnik-manor Perona 2005
(Rengio et 2007].
Kingma Welling 2014
JULE (Yang et al. 2016),
DEC et er
2016), DAC (Chaing al, 2017). IIC (Ji al (019), DCCM (Wa et 20119)
PICA (Huang et all 2020,
CC-LIC et 2021b, SCAN Gansboke et al. 2020). GCC (Zhong et al. 2021
MiCE al 2021) IDFD [Tao [202 PCL (Liet al. (222 (Gril et al. 2020
CC uses a large image size (224) for datasels.
SCAN needs additional pro-training stage while NCC is trained
end
only training all datasets.
Quantitative results We compared NCC with previous state-of-the-ar clustering methods in Ta- b NCC achieves significant performance improvement on all five benchmark datasets, demon- strating the superiority of NCC for deep clustering to capture the semantic class information. In- terestingly, directly using the representations learned by BYOL for k-means clustering outperforms previous work including the contrastive-bass ones (Li et al., 202Tb Tsai et al.) 202 Tao et al.) 202T], which suggests a great potential for non-contrastive representation learning for deep cluster- ing without suffering class collision issuc.
On the ImageNet-100 10, our NCC achieves competitive performance as compared to IDFD Tao et al, 2021) since this dataset is relatively small with only 13K images, which cannot arise discriminative differences for current state-of- ofate-of-the-art methods. On the ImageNet-Dogg a fine- fine-grained dataset con- taining different species of dogs from the ImageNet dataset, there almost 20% improvements over previous SOTA work. The contrastive-based methods cannot handle this kind of dataset due to class collision issue that pushes away the instances from the same class. Meanwhile, IDFD can deal with this problem to some degree thanks to the feature decorrelation along with the instance discrimination. Without the need of negative examples, BYOL can achieve significant improvement, although its performance is unstable. Our NCC, built upon BYO with a positive sampling strategy and prototypical contrastive loss, has shown its significant and stable performance against vanilla B BYOL and contrastive-base methods.
Table 3: Clustering results (%) on ImageNet- IK.
Table 3 further presents the results between our NCC and baseline methods including DeepCluster (Caron et al.) 2018), MoCo (He) et al. 2020), and PCL. (Li et al.) a) on ImageNet-1K dataset, showing that NCC achieves significantly higher AMI score.
Method
AMI
DeepCluster
28.1
Although we employed the fair conditions, some work has trained the network with different split (Van Gansbeke et al. 2020) for
MoCo
28.5
PCL
41.0
CIFAR-10 and CIFAR-20 or large image size (Li et al. 2021b) for ImageNet- 10 and ImageNet- -eggeegs.tD For the sake of fair comparisons with different image sizes and splits, we report these comparison results in Table Al In addition,
NCC (Ours)
52.5
also reported the clustering results ImageNet subsets like SCAN (Van Gansbeke et al. 2020
7

---

Under review as a conference paper at ICLR 2022
in Table A2 and Imny-ImageNet (L & Yang 2015 in Table A3
We also conducted additional experiments in Table A4 to demonstrate the ability of NCC handling the long-tati sangtailed datasets.
Qualitative results. Fig. 3 visualizes the learned representations by t-SNE (Van der Maaten & Hinton 2008) for four different training epochs throughout the training process. At the beginning. the iandal-initialized model cannot distinguish the instances from different semantic classes, where all instances are mixed together. As the training process goes, NCC gradually attracts the instances from the same cluster while pushing the clusters away from each other. Obviously, at the end of the training, NCC produces clear boundary between clusters and within-cluster compactness. Visualization for the outlier points produced by the model at 1000-th epoch is shown Fig. A5
e) Epoch Visualization of feature representations learned by NCC CIFAR-10 with t-SNE. Dif- ferent colors denote the different semantic classes. Zoom in for better view.
Epoch (NMI-6.98% Epoch (NMI-77.8%)
Epoch 1000 (NMI-85.5%)
Figure 3:
5.2 ABLATION STUDY
Here, we perform detailed ablation studies with both quantitative and qualitative comparisons to provide more insights into why NCC performs so well for deep clustering.
Ablation studies for different self-supervised learning frameworks, and positive sampling (PS) strategy, and prototypical contrastive loss for NCC. The best and second best results are shown in bold and underline, respectively.
Table 4:
CIFAR-10
CTFAR-20
Prototypical
Method
PS
NM1
ACC
ARI
NM1
ACC
ARI
Alignment Uniformity
v2
et al.
2020
76.9-002
84 84.911.3
72.4:10.5
49.2:0.1
48.0:002
32.1-10.0
SimSiam
78.8=00
86.5
74.9+1.3
46.6208
47.3==1.1
28.8:112
CC
Li
21216
66.1=0.3
74.6+00.
58.3400.4
46.4+0.3
45.0=00.1
29.5440.2
Pratet
74.3=0.4
83.410.5
69.6:11.
48.31002
49.12:0.2
32.240.4
PCL.
et al.
2a
77.6-0.1
85.5+0.1
73.4+00.
50.0+0.3
48.6-0.7
32,7404
BYOL
Grill
2020
79.4=177
87.8.1.77
76.61228
55.52006
$3.92:16
37.6:10.9
76.6-3.1
86.3-22.7
73.8++4.77
51.0=200
48.9=
33.3-2.9
PCL
74.4=2.3
65.3+90.9
71.4+11.4
49.7+0.7
46.9=0.7
27.8+11.
79.4-0..
87.9:10.5
76.41:111
57.02000
55.02:06
39.8:1111
83.4=1.2
40.7.1.0
NCC (Ours)
79.6=0.7
87.8:11.5
76.542.11
56.720.3
56.61114
39.721.1
85.3-02
92.1+001
84.4+0.
$7.2+0.3
57.3-00
41.7+055
05
91.6:10.4
83,5400
58.2.0.3
578:022
42.3:103
Quantitative ablation study. We report the quantitative results of ablation studies in Table BYOL outperforms MoCo v2 and SimSiam by large margin on both two datasets. The differ- ence between BYOL and MoCo v2 is that MoCo v2 uses memory queue to store the consistent negative examples while BYOL directly aligns two augmented views with a predictor network. Dif- ferent from the BYOL that employs momentum-updated network as the target network to yield the positive representations, SimSiam shares the weights of the target and online networks. There- fore, BYOL outperforms MoCo v2 by dealing with the class collision issue and SimSiam by the momentum-updated target network.
Compared to vanilla BYOL, simply using the positive sampling strategy can stable and further im- prove the performance, especially when the number of semantic classes increases for CIFAR-20. Although ProtoCL improves the baseline results by large margin, positive sampling boost the clustering performance. This because ProtoCL only considers inter-cluster distance, and cannot benefit within-cluster compactness. Therefore, the combination of the positive sampling
further
8

---

Under review as a conference paper at ICLR 2022
and ProtoCL achieves the best clustering results, where positive sampling strategy can improve the within-cluster compactness and ProtoCL encourages the prototypical alignment between two aug- mented views and maximizes the inter-cluster distance.
To further explore the effect of the proposed ProtoCL, we split this loss function into prototypical alignment and uniformity as shown in Eq. (). It is clear that the performance gain from the align- ment term is marginal while the gain from the uniformity term is significant. Note that for only alignment term, we compute the loss after predictor network instead of feature extractor, otherwise representation collapse will turn out. This indicates that prototypical uniformity is more important than prototypical alignment since BOL has already performed instance alignment at an augmented instance level. However, we note that the prototypical alignment term is essential to stabilize the training process, as demonstrated the results for CIFAR-20 with more clusters.
To demonstrate that B BOL is not robust to additional losses for deep clustering tasks, we integrate CC (Li et al. 202Tb) and PCL (Li et al. 2021a) into BYOL; the results in Table 4 show that both of them compromise clustering performance and become unstable. This is because CC contrasts the cluster probability not helpful for representation learning, and the representations of PCL would collapse without negative examples; see Fig. AT for detailed analysis. For fair comparisons of the self-supervised learning framework, we integrate our ProtoCL into CC by replacing the cluster head with our ProtoCL on the representations while keeping other official syper-parameters unchanged. Although class collision issue remains, the significant improvements over CC on both datasets sug- gest that class-level contrastive loss over representations of cluster centers is better than the one over cluster probabilities and ProtoCL can be generalized to other self-supervised learning frameworks.
Qualitative ablation study. Fig. visualizes the distribution of representations learned from MoCo v2, BOL, NCC w/o ProtoCL, and NCC. The representations from MoCo v2 are mixed up at the center due the class collision issue. The rest can reduce this phenomenon where NCC w/o ProtoCL produces more compact clusters than B BOL, and NCC further maintains distinct bor- ders between different clusters. In addition, we also visualize the training and clustering results of B BYO and NCC in Fig. A7 demonstrating that NCC can produce more uniform representations, more balanced clusters, and better clustering performance than BYOL.
a) MoCo (NMI-76.6%)
b) BYOL (NMI-78.4%) NCC wie ProteCL (NMI-78.8%) NCC (NMI-85.5%)
Figure 4: Visualization of feature representations learned by different representation learning frame- works and our proposed NCC on CIFAR-10 with t-SNE. Zoom in for better view.
Additional ablation studies. To explore the influences of different hyper-parameters in NCC,
we perform the following ablation studies: 1) performing k-means clustering for every r epochs
in Fig. A2 2) in positive sampling in Fig. A33 3) Xpel for ProtoCL in Fig. A4 4) predefined
number of clusters in Fig. A6; 5) projection dumension for self-supervised learning in Fig. A8 6) data augmentation in Fig. A9 for self-supervised learning; and 7) different ResNet architectures in
Table A5 All ablation study results verify that the performance gain of NCC does not come from backbone, projection dimension, or any other hyper-parameters. The results also suggest that NCC is robust to the choice of hyper-parameters.
6 CONCLUSION
We have explored the non-contrastive representation learning for deep clustering. The proposed positive sampling strategy and prototypical contrastive loss can lead to within-cluster compactness and well-separated clusters towards the goodness of clustering. The results suggest that the proposed NCC outperforms the ffte-off-the-art methods by significant margin. We hope our study will attract the community's 's attention to the non-contrastive representation learning methods for deep clustering, which do not suffer from class collision issue.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, pp. 153-160, 2007.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aaaal margin loss. In Advances in Neural Information Processing Systems, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. In Proceedings of the IEEE International Conference Computer Vision, pp. 5879-5887, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple Siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Boo Ching- Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in Neural Information Processing Systems, 2020.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statisties, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 248-24882 255.....122 leee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in Neural Information Processing Systems, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. International Conference Learning Representations, 2017.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5414-5423, 2021.
Abe Fetterman and Josh Albrecht. Understanding self-supervised and contrastive learning with bootstrap your own latent (BYOL). https:/huttilled-ai. htwp://miiiieeddiiiiiuu..hca contrastive-earring himl, 2020.
dean-Bastin Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Advances in Neural Information Processing Systems, 2020.
10

---

Under review as a conference paper at ICLR 2022
Philip Haeusser, Johannes Plapp, Vladimir Golkov, Elie Aljalbout, and Daniel Cremers. Associative deep clustering: Training a classification network with no labels. In German Conference on Pattern Recognition, pp. 18-32. Springer, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9773, 2020.
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. AdCo: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1074-1083, 2021.
Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8849-8858, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 446-456.. PMLR, 2015.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9865-9874, 2019.
Pramnay Khosla, Piotr Teterwak, Chen Wang, Aaron Samna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 2020.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7);3 2015.
Junsoo Lee, Hojoon Lee, Inkyu Shin, Jaekyoung Bae, In So Kweon, and Jaegul Choo. Learning representations by contrasting clusters while bootstrapping instances. 2020.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In International Conference on Learning Representations, 202
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng. Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In AAAI Conference on Artificial Intelligence (AAAI), 2021b.
Stuart Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory. 28(2): 129-137, 1982.
Chuang Niu, Hongming Shan, and Ge Wang. SPICE: Semantic pseudo-labeling for image cluster- ing. arXiv preprint arXiv:2103 1XXi:22133309322 2021.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1rXiv: 2018.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. 2019.

---

Under review as a conference paper at ICLR 2022
Jayanth Reddy Regatti, Aniket Anand Deshmukh, Eren Manavoglu, and Urun Dogan. Consensus clustering with unsupervised representation learning. In International Joint Conference on Neural Nenworks (IJCNN), pp. 1-9. IEEE, 2021.
Pierre Richemond, nan-BBstien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. BYOL works even without batch statistics. arXiv preprint arXiv:2010.1022 2020.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Con- ference Machine Learning, pp. 5628 5637. PMIRR, 2019.
Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip HS Torr, and Ling Shao. You never cluster alone. arXiv preprint arXiv:2106.01900) 2021.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Advances in Neural Information Processing Systems, 2020.
Yaling Tao, Kentaro Takagi, and Kouta Nakata. cuustrrin-ffiendlly representation learning via instance discrimination and feature decorrelation. International Conference on Learning Repre- sentations, 2021.
Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. MiCE:
Mixture of contrastive experts for unsupervised image clustering. In International Conference on Learning Representations, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine
Learning Research, 9(11), 2008.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. SCAN: Learning to classify images without labels. In European Conference on Com- puter Vision, pp. 268-285. Springer, 2020.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2495-2504, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- PMLR, 2020.
Xudong Wang, Ziwei Liu, and Stella X Yu. Unsupervised feature learning by cross-level instance- group discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12586-12595, 2021.
Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 8150-8159, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pp. 478 487. PMLR, 2016.
Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means --means-friendl spaces: Simultaneous deep learning and clustering. In International Conference on Machine Learning, pp. 3861-3870 PMLR, 2017.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5147-5156, 2016.
12

---

Under review as a conference paper at ICLR 2022
Lihi Zelnik-manor and Pietro Perona. Self-tuning spectral clustering. In Advances in Neural Infor- mation Processing Systems, 2005.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Con- ference Computer Vision, pp. 649-666. Springer, 2016.
Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nic, Zhouchen Lin, and Xian-Sheng Hua. Graph contrastive clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: Bilateral-brance network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 9719-9728, 2020.
13

---

Under review as a conference paper at ICLR 2022
APPENDICES
A ADDITIONAL RELATED WORK
Self-supervised learning. Previous self-supervised learning (SSL) methods for representation learning attempt to capture the data distribution using generative models (Donahue et al. 2017; Donahue Simonyan 2019) or learn the representations through some special designed pretext
tasks (Doersch et al. 2015 Noroozi & Favaro 2016, Zhang et al. 2016 Caron et al., 2018 In recent years, contrastive learning methods (Wu al. 2018 He et al. 2020 Chen et al.) 2020) have shown promising results for both representation learning and downstream tasks. For example, MoCo (He et al., 2020) uses a memory queue to store the consistent representations output by a moving-averaged encoder. However, the class collision issue remains unavoidable; L.e, the seman- tic similar instances are pushed away since they could be regarded as negative pairs (Saunshi et al. 2019). Some attempts have been made to address this issue (Khosla al. 2020 Hu et al. 2021 Chuang et al. 2020). On the contrary, the recent studies of SSL demonstrate that the negative exam- ples necessary, termed non-contrastive methods (Caron al., 2020 Grill al. 2020 Chen et al. 2020). In summary, SSL methods mainly focus on inducing transferable representations for the downstream tasks instead of grouping the data into different semantic classes in deep clustering.
Relation to CC. Although both Ppel and CC (Li et al. 2021b) are class-level contrastive loss, which perform contrastive learning the cluster level, they have the following difference.
The class-level contrastive loss in CC implements the contrastive loss on the cluster probabilities while ours on the representation of cluster centers. Implementing contrastive loss on the cluster probability in eL al. (021b) would lose the semantic information the learned representations, which is not helpful for representation learning, Specifically, given the E B, CC obtains the cluster assignments Pa [p(k))z(")), ..... [N)) from one view and P from another view, and then contrasts P& and P at the cluster level using the InfoNCE loss. contrast, 4pd implements the contrastive loss on the representation of the cluster centers within mini-batch using the pscudo-labels from k-means clustering. As a result, Cpel is able to sense the semantic information of the latent space and make the representations of clusters more discriminative and suitable for the clustering task.
The class-level contrastive loss in CC does encourage cluster uniformity while our ProtoCL does. CC still needs the instance-wise contrastive loss to encourage the instance uniformity, which inevitably introduces the class collision issue. We have also conducted experiments that integrate CC into BOL and reported the results in Table The performance of BYOL drops and the training becomes unstable. Under the same conditions, NCC achieves significant improvements over CC.
Relation to GCC and WCL. GCC [hhog et al. 2021 and WCL (Zheng et al. 202 built a graph to label the neighbor samples as pseudo-positive examples. Then, they enforce the two data augmentations of one example to be close its multiple pseudo-positivs examples using supervised contrastive loss. GCC adopted a moving- averaged memory bank for the graph-based pseudo-labeling while WCL built the graph within mini-batch. GCC and WCL mainly focus on how to effectively select positive examples from cari-batth/mmmmorr bank to alleviate the class collision issue. Here, we divide the class collision issue into the following two cases:
Negative class collision issue: negative examples may not be truly negative, which the case we discussed in the paper.
Positive class collision issue: positive examples may not be truly positive, which is a new case raised in GCC and WCL.
Consequently, they still suffer from the positive class collision issue as the selected pseudo-positive examples may not be truly positive. addition to this, they also suffer from the negative class collision issue since they still need negative examples for mnstance-wise contrastive learning.
We summarize the difference between our positive sampling strategy and theirs in the following four aspects.
14

---

Under review as a conference paper at ICLR 2022
GCC and WCL select the examples that exist in the dataset (mini batch/memory bank) while ours samples augmented examples from the latent space that may not exist in the dataset.
GCC and WCL select neighbor examples in graph as pseudo-positive examples that may be truly positive while ours samples examples around the instance that can be assumed to be positive the semantic space.
GCC and WCL still rely instance-win wisee contrastive loss that could lead class collision issue while ours can avoid class collision issue by using BYOL.
GCC and WCL require additional computational cost for graph construction while ours is rather cheap in sampling one augmented example.
Relation to PCL. Here, we summarize the difference between our NCC and PCL (Li et al. 202 lap terms of the losses and EM frameworks. First, we summarize the difference of our ProtoC] loss and ProtoNCE loss in PCL as follow.
Our NCC can avoid class collision issue while PCL cannot. NCC is based on BYOL that does not require negative examples for representation learning while PCL is based on instance-wisg contrastive loss that requires a number of negative examples for representation learning, inevitably leading to class collision issue.
The proposed ProtoCL in NCC is conceptually different from the ProtoNCE in PCL. ProtoCL is to maximize the inter-cluster distance to form a uniformly distributed space while ProtoNCE is to minimize the instance-to-cluster distance to improve the within-cluster compactness. The within-cluster compactness NCC is improved by the proposed positive sampling strategy.
Pure ProtoCL can work well for deep clustering while ProtoNCE requires another InfoNCE to form uniformly distributed space. This is direct result of the different designs of the losses. ProtoCL can maximize the inter-cluster distance form a uniformly distributed space while Pro- toNCE suffers from collapse without the help of another InfoNCE to form such a space.
Second, we summarize difference between our NCC and PCL in the EM framework. Formulating NCC into EM framework can offer more insights about NCC and make it easy understand. Although both in an EM framework, the M-step in PCL is significantly different from the one in our NCC. More specifically, the M-step PCL optimize the ProtoNCE, which is an instance- lo-prototypes contrastive loss to improve the within-cluster compactness while the M-step in our NCC is optimize the proposed ProtoCL, which prototypes-to-proteee to maximize the inter-cluster distance for better clustering performance. In addition, NCC also proposes a positive sampling strategy by sampling positive examples around each sample to improve within-cluster compactness. Finally, Table 3 demonstrates that NCC outperforms PCL by almost 10% AMI on the ImageNet-1K dataset.
contrastive loss
B EM FRAMEWORK
In this section, we first describe the von Mises-Fisher (vMF) distribution on the hypersphere, and then derive the ELBBO for our EM framework, followed by detailed E-step and M-step. Finally, we describe our proposed prototypical contrastive loss and provide proof for convergence analysis.
Von Mises-Fisher distribution. Since the features in current SSL methods are usually ba- normalized, it is more proper to employ the spherical distribution to describe the features. The von Mises-Fisher (vMF) distribution, often seen as the Gaussian distribution parameterized by μ E Rd the mean direction and K E R4 the concentration around μ. For the special case of 0, the vMF distribution represents uniform distribution the hypersphere. The PDF of vMF distribution for the random unit vector v E Rd is defined as:
hypersphere, is
k d/2-1
p μt, R) (a(K) exp(xxi"
Ca(k) (2 (2T))//2a/2-1(11))
(A1)
where d is the feature dimension, |/ = 1, Ca(k)) the normalizing constant, and I, denotes the modified Bessel function of the first kind at order The standard Gaussian distribution (0 can be approximately seen as the uniform vMF distribution if the z is & no-omaallde and is large for the high dimension data.
15

---

Under review as a conference paper at ICLR 2022
Derivation of ELBO. Given the dataset D {x(1)) n== with N observed data points that are related to a set of K cluster latent variables, ht E K = (1,2..... K}. the marginal likelihood can be written as:
1 N L(D;0) = N n=1
N
Σ
log p (m)
1
Σ log Σ p p(x("), 1=
k: 0),
(A2)
N
KEC
where A denotes the model parameters. Eq. A2 is usually maximized to train the neural network. However, is hard to directly optimize the log-like likelihood function. Using an inference model g(k)
like VAE (Kingma & Welling 2014) to approximate the distribution of K, especially > g(k)
1, we can we-write the log-likelthood function for one example as:
p(:; 0)
Σ
(og)
(A3)
kEX
Σ ((k))loop(e k;0) logp(k((;; 0))
(A4)
kEK
p(x, k; 0)
Σ (() log p(k|2 0)
Σ (l:) log
(A5)
g(k)
((k)
REX
kEK
= ELBO x; 0) + (L(g())ll(((]z; 0)),
(A6)
where p(a, k; 0) pk|ar; 0)p(2; 0) so we have logp(a;; 0) log p(a. k; 0) pog x;; 0) and the evidence lower bound (ELBO) is the lower bound of log-likelihood function since (L(((k) [pp( æ; 0)) 2 O. When (L)(()))ppp(a)a 0)) = O, the ELBO reaches its maximum value log p(z; 0), making (k) p(k| 0). By replacing (lk) with p (s) 0) and ignoring the constant value Σtek (kk)) log (K)), we are ready to maximize:
Σ p(kka 0) logp(æ, REK
(A7)
E-step. With the fixed 0, the iteration t. this step aims to estimate (++1(k) that makes (o+1(k) = p(k x 0₂) so that ELBO(q-11 x; 0,) = p(x; Oz). Here, we perform the spheri- cal k-means algorithm estimate p(k|a:; 0.). We extract features from the target network since the target network performs more stable and yields more consistent clusters, similar to BYOL and MoCo.
M-step. With the fixed suboptimal (k) = p(k xe; 0,) after E-step, we turn to optimize the 0 to maximize the ELBO:
N
1 arg max
ELBO (n),
(A8)
n=1
Using uniform prior for h as p(k) 1/K, we can obtain p(ær, k; 0) p(k)p(arkk; 0) pl 0)/K, K. By replacing p(xr, k; 0) in Eq. (77) and ignoring constant value, in this step, we should maximize:
Σ 1(x E () kek
(A9)
where p(k ac; 9) 1( k)). 1(.) is an indicator function using the hard labels estimated from E- step so that 1(x E k) = 1 if æ belongs to k-th cluster, otherwise 1(x E k) = O. Following (Li et al.) 202 if we assume that the distribution for each cluster is the vMF distribution with constant
as the temperature of softmax function, we can further obtain the follow:
cxp(μTv/T)
plak 0) ER exp(uTu/T)
(A10)
where v = fæ; 0), - 1/k, and be is the cluster center of k-th cluster. Combining Eqs. (A9) and (A10), we can achieve the maximum lig-likelihood estimation find the optimal 0* by mini- mizing the following negative log-likellng likelleiihood:
exp(1it you w(m) /T)
0"
arg min
log ΣK exp(mguln))/
(A11)
16

---

Under review as a conference paper at ICLR 2022
where y(t) is the pseudo-label for (n)) estimated by the k-means algorithm in E-step.
Directly optimizing Eq. (ATT) usually leads to improve the cluster compactness, which, however, will compromise the stability of BYOL since it does not consider the uniformity term. To this end, we propose a prototypical contrastive loss (ProtoCL) to maximize the log-likelihood at the cluster level by employing the clusters centers as the special instances, or prototypes from set of instances. The ProtoCL is defined as:
1 Σ k=1 K
exp(pT
ppl
log
(A12)
cxp(p + Ejm
Live exp(u)p//7)
where {1l .2...... pKk) and {Wi, ,l........ [ are K prototypes from target and online networks, respectively. Here, instead using the centroids computed from k-means, our cluster center μag and u is empirically estimated within mini-batch B as follows:
EXER (()) [1 EBEB f(x))2 and P/ (mE
(A13)
where p(k(ar) is estimated from E-step. When K > 18), it is obvious that the mini-batch cannot cover all clusters. To this end, we zero out the losses and logits of empty clusters for each iteration; see the pseudocode in Appendix Dimendix[ffor for more details.
Intuitively, ProtoCL can encourage the prototypical alignment between two augmented views and the prototypical uniformity, hence maximizing the inter-cluster distance.
E-step of the iteration I, we estimate (t+1 to make ELBO (1BBO(141,)))) 02) log p(x; 0,). At M-step after E-step, we obtain the optimized 0 with the fixed (k) so that ELBO (g x;; 0.+1) 2 ELBO(9+11: æ; 0t). Consequently, we obtain the following sequence:
Convergence analysis.
log p(2; 14+1)= 2 ELBO 0.41) 2 ELBO(Q++1 x; 0,) = log p(:; 0₂).
(A14)
Given log p(2; 0.t1)= 2 logp(<:; 0.), can guarantee the convergence of our NCC.
C EXPERIMENTAL SETUP
Datasets. We conducted experiments six benchmark datasets, including CIFAR- 10 (Krizhevsky et al. 2009). CIFAR-20 (Krizhevsky et al. (2009). STL.-10 (Coates al.] 2011). ImmgeNNe-110 (Chang et al., (017). ImageNet-Dogs (Chang al.] (017). ImageNet- 1K (Deng et al.) 2009). which are summarized in Table ! We note that CIFAR-20 contains
20 superclasses of CIFAR-100. STL-10 includes extra unlabeled images. ImageNet-10 and ImageNet-Dogs are the subset of ImageNet-IK IK, containing 10 and 15 classes, respectively. This paper follows the experimental settings widely used in deep clustering work (Chang al.) 2017 Wu et al. 2019 Ji et al. 2019; Tsai et al.) 2021 Tao et al. 2021). including the image size, backbone and train-test split. For image size, we have used 32 32 for CIFAR-IO and CIFAR-20, 96 96 for STL-10. ImageNet- 10 and ImageNet-Dovg Dogs, and 224 x 224 for ImageNet- K. For train-test split, we use the whole datasets including training and testing set for CIFAR-10 and CIFAR-20.
Backbone. We use ResNet-34 (He et al. 2016) as the backbone for fair comparisons to report the results in Table Unless noted otherwise, we use R8 for the rest of the experiments. Since the image sizes of CIFAR-10 and CIFAR-100 are relatively small, following (Chen et al. (020)) we replace the first convolution layer of kernel size 7 x and stride 2 with a x Conv of stride and remove the first max-pooling layer for all experiments CIFAR-10 and 100.
We employ three common metrics to evaluate the clustering performance for the for- mer five datasets, including Normalized Mutual Information (NMI), Cluster Accuracy (ACC), and Adjusted Rand Index (ARI). Following (Li et al. 2021a), we report Adjusted Mutual Informa- tion (AMT) to evaluate the clustering performance for ImageNet- IK. All the metrics are presented in percentage (%), and the higher the better clustering performance. We run each experiment three times and report the mean and standard deviation as the final results.
Metrics.
We train all methods with 1000 epochs, strictly following the literature (Tsai et al. 2021 Tao et al. 2021), and adopt the SGD optimizer and the cosine decay learning rate schedule
Optimization.
17

---

Under review as a conference paper at ICLR 2022
with 50 epochs for learning rate warmup. The base learning rate for MoCo v2, BYOL, and NCC were 0.05, scaled linearly with the batch size (LearningRate = 0.05 BatchSize/256). Note that the learning rates for predictor networks of BYOL and NCC are 10x the learning rate of feature extractor. It is relatively important to achieve satisfactory performance, as discussed in (Grill et al.) 2020 Chen & He 2 For other hyperparameters of NCC, the temperature т, Apel for prototyp- ical contrastive loss, and a for positive sampling were set as 0.5, 0.1, and 0.001, respectively. The mini-batch size was 512 for MoCo and 256 for the rest methods, trained on 4 NVIDIA V100 GPUs. In terms of the results of CC (Li et al. 2021b) and PCL (Li et al. 2021a) in Table Ei best to reproduce their results for fair comparisons. For CC, we used their official code. For PCL, under the fair conditions of MoCo, we set the loss weight of ProtoNCE to 0.01 and the number of clusters to {250, 500, 1000} following the suggestions of authors, which we found can achieve the best results. We integrated CC and PCL into BOL by adding their losses without changing other settings.
we tried our
Configurations of supervidd learning frameworks. We adopt the same data augmentations as SimCLR (Chen et al. (020), including ResizedCrop, ColorJitter, Grayscale, and HorizontalFlip. We have removed GaussianBlur since only used a small image size for all datasets. We also strictly follow the settings of BOL (Grill et al. 2020). Specifically, despite the standard ResNet backbones, the projection and predictor networks have the architectures of FC-BN-ReLU-FF, where the projection dimension and hidden size were 256 and 4096 for both two networks, respectively. In the context of the whole paper, we have included the backbone and projection network as the encoder network, or feature extractor, since the architectures except predictor are symmetric across views. For fair comparisons, we have also set the projection dimension of MoCo v2 as 256. We have used symmetric loss for all methods, i.e., swapping two data augmentations to compute twice loss. Given the target network fgr and online network fe. the momentum rule describes that the O' is updated as 0' = m0 + (1 m 0 with momentum hyperparameter m E [0, 1). We set the m to 0.996 for both BYOL and NCC same as (Grill et al. 2020) and 0.99 for MoCo v2. For MoCo v2, the queue size. temperature for InfoNCE loss, weight decay were 4096, 1.0, 1.0 x 10 -4 respectively. We have not employed SyncBN in NCC like BYOL. We note that SyncBN will introduce 1/3 additional computation costs. Instead, we adopt the shufflingBN in MoCo avoid the trivial solution of non- contrastive learning.
Configurations of ImageNet-IK. For ImageNet- I K, the weight decay and base learning rate were 10 and 0.4, respectively. We employ ResNet-50 as the backbone, train our method for 200 epochs without symetric loss reduce training time, under the same conditions of PCL (Li et al. 2021a). We include the GaussianBlur to the data augmentations since we use the original image size for training. During evaluation, images are resized to 256 x 256 and center cropped. Due to the large computation cost, we run NCC on ImageNet- IK only once.
Computational cost. Both BYOL and MoCo use momentum-updated network fg.. The com- putation cost between them lies on the fact that MoCo needs a large number of negative examples while BYO uses a large hidden size for the predictor network. Our NCC is built upon BOL,, the main additional computational cost is the k-means clustering procedure. We have implemented the k-means algorithm with k-means+++ (Arthur& & Vassilvitskii 2006) initialization using PyTorch to utilize the GPU and accelerate the clustering process. Taking CIFAR-10 and CIFAR-20 datasets for example, it takes only 5 seconds for k-means clustering, leading to only 5 hours of training time. Besides, suggested in the results of Fig. A2 NCC is robust to the different r, so that there is no need to perform k-means for every epoch. The computational cost of the ProtoCL. is also small since build the cluster centers within mini-batch, saying that NCC does not need additional memory to store the cluster centers. Consequently, considering the promising performance improvements, the additional computational cost is relatively affordable.
18

---

Under review as a conference paper at ICLR 2022
D PSEUDOCODE OF NCC
Algorithm Pseudocode NCC in a PyTorch-like style
for for Dp: online network, target network, predictor respectively:
number clusters, temperatuze for ProtoCL loss;
m: momentum, max_epochst the training epochs;
# a: sampling standard deviation of Gaussian distribution/
Apel: loss weight of protocl loss, Apel =0 during learning-rate warmup; performs K-means clustering for every epoch;
compute the centers from their representations and pseudo-labels; def compute_centers (z, y):
weight onehot (y) .I XxN;
weight normal (weight, p-1, dim-1) f1-noraaizaation centers normal ze wwiight.mm (2), p-?, dim-1) izaormaniizaiin return centers
12
13
compute ProtoCL from the cluster centers of two views def compute protocl loss (par Per y) zero_classes = counts (y) # find the empty clusters with indices; align two views, Kx1;
16
17
18
proto_alignment (Pr Ma) . (dim=1, keepdim=True) /
proto_uni formiiyy fill the logits empty clusters with -10; proto_uniformityy = proto_uni fomifootty..lll.. # retrive inter-cluster distance for proto uniformity;
mm.mm(p
inter-cluster distance;
21
22
zero_classes, dim-1)
23
proto_uniformity proto www.uuiitotttlli...ooooooooooo.m diigonadaanaall mask] # Kx (x-1)));
25
protocl_ooss proto_ arign_alignment Logsumexp (eat + proto_unt formityl, dim-1), dim-l)
26
[rotocl_oosrrrrccclassseee [zero_classs]] 0 # zero the loss of empty clusters; neglect the empty clusters;
protool loss protocl sum).sum() (K len (zero (enzooo__caasses) return protocl loss 30
29
compute both augmented instance alignment and ProtoCi; def forward_ loss (im_q, im_k, y)
33
q, fo (im_g), f/r (im_k) forward;
34
ins_loss = msc (g. a randh_like (q)), k) instance alignment;
04 compute_centers (q, y) compute the centers of two views; 35
μ4 computa_centers (k, y)
protocl_ _ossssss___ compute_protos) loss (Mar Ps. y)
38
return ins_ protoci Apdl
39
40
cop.ypypoo. (0) # initialize the target network with online netwerk;
for epoch in range epochs) start training;
42
E-step
43
if epoch $ pr=- == 0:
44
K-means based on target network for every epochs: N labels pseudo_labels K-means (loader, f(r
46
M-step
47
for indices, im in loader: load
x with samples; im_g, im_k aug(im) $ two randomly data augmentations; caardd_labelllsndddd
48
pseudo-labels in each batch; symetric loss same vanilla BYOL: swap inputs of two views; loss forward loss (im_g, im_k, y) forward (im_k, im y) loss /= 2
52
53
SGC upoat ing onl inline network and predictory loss backward
update (0, (p)
56
momentum update target network; 8'. co (1-m) +))
57
19

---

Under review as a conference paper at ICLR 2022
E
ADDITIONAL EXPERIMENTAL RESULTS
a) Normalized Mutual Information
b) Standard Deviation
0.8
0.6
STD
0.4
0.2
PCL+MoCo
PCL+MoCo
PCL+BYOL
PCL+BYOL
0.0
200
400
600
800
1000
200
400
600
800
1000
Epoch
Epoch
Figure A1: Visualizations of NMIs and STDs by applying PCL (Li al. 2021ap to MoCo and BYOL CIFAR-10. Compared to PCL+BYOL, PCL+MoCo performs more stable during cluster- ing with a more uniform distribution of representations. The decreasing STDs (standard deviation of fy-normalized features) also indicate that PCL+BYOL. suffers from the representation collapse. This is because PCL can only improve the within-eluster compactness. During training, the fixed prototypes of PCL will also be gradually collapsed without negative examples, making the repre- sentations for BYO collapse at the same time. These results validate our assumptions that B BYOL is not robust to additional clustering losses for clustering tasks, since there is negative example for BYOL to maintain uniform representations to avoid collapse. Our NCC can avoid the class col- lision issue and representation collapse by the proposed positive sampling strategy improve the within-clusee compactness and prototypical contrastive loss to maximize the inter-class distance.
CIFAR-10
b) CIFAR-20
100
70
60
Performance (%)
Performance (%)
80
50
70
40
60
30
NMI
NMI
ACC
20
ACC
ARI
ARI
40
10
BYOL
r=1 r=2
1=4 4
r= 8 == 16
BYOL
r=1
r=2
f=4
r=
== 16
Figure A2: The effect of the hyperparameter r. NCC performs K-means clustering for every epoch. Here study how different influences the clustering performance. The results demonstrate NCC is robust to large r and the cluster pseudo-labels, which means it is not necessary to perform clus- tering for every epoch so that the computation cost can be significantly reduced. In summary, we suggest that r can be set to [1, 8] by considering the datasets and computation resources.
20

---

Under review as a conference paper at ICLR 2022
a) CIFAR-10
b) CIFAR-20
100
70
60
Performance (%)
Performance (%)
80
50
70
Metric
30
Metric
60
NMI
NMI
20
50
ACC
ACC
ARI
10
40
10 10-3
5x 10-2 5 x0
10 10-3
5 10 5 10
Figure A3: The effect of the hyperparameter for positive sampling. Taking a look at ~ [0, 10-3), although introducing the positive sampling into BYOL causes slight drop CIFAR-10, the clus- tering performance becomes more stable as evidenced by the standard deviation. This is because the neighbors of one sample regarded positive examples. Besides, the performance for CIFAR-20 has increased over baseline B BOL with the standard deviation reduced. These results indicate that positive sampling can improve the stability of BYOL. However, when is too large, the performance becomes unstable and drops a lot. is not surprised since during positive sampling with large a, the instances from other clusters could be sampled and regarded as positive examples. Therefore, suggest setting a to a small value, saying (0, 10 10-3..
a) CIFAR-10
b) CIFAR-20
100
70
90
60
Performance (%)
Performance
70
40
60
30
NMI
NMI
50
ACC
20
ACC
ARI
ARI
10
0.01
0.05
0.1
0.5
0.05
Figure A4: The effect of the hyperparameter Apel for the ProtoCL. The results suggest that NCC robust to different loss weights of ProtoCL loss on CIFAR-20. However, the higher loss weight leads to instability more semantic classes than CIFAR-10 (100 versus 10). Anyway, we suggest that the loss weight for ProtoCL loss be 0.1] for different situations, which has demonstrated superior performance on both two datasets.
CIFAR-10. The possible reason is that CIFAR-20 is more diverse and has
21

---

Under review as a conference paper at ICLR 2022
airplane automobile bird
deer
dog
frog
ship
Figure AS: Visualizations for the outlier points produced by the model at 1000-th epoch on CIFAR-
10 with t-SNE.
CIFAR-10
b) CIFAR-20
70
60
Performance (96)
Performance (9))
70
50
60
30
Method
Method
NCC
20
NCC
30
BYOL
BYOL
10
20
10
20
Number clusters
Number clusters
Figure A6: The effect of the predefined number of clusters K. We reported NMIs following CCaron ctt al.) 2018). We note that the predefined K of NCC during the training NCC is the same as the K in k-means clustering process for evaluation. To further demonstrate the influences of over- clustering, we also reported the results of vanilla BYOL during k-means clustering. The results demonstrate that both BYOL and NCC have the same over-clustering behavior these two datasets; that is, opposite trends on these two datasets. Specifically, over-clustering leads to the performance drop for CIFAR-10, but it leads performance increase for CIFAR-20. However, our NCC can still produce large improvements over BYO with the same predefined number of clusters. We note that the opposite results are due to the significant difference between these two datasets. Although having the same number of samples, CIFAR-10 has 10 distinct classes while CIFAR-20 has, in fact, 100 classes but uses 20 super-classes instead. If the representations are well aligned within the same semantic clusters, the over-clustering would try to destroy the structures of the clusters and push the semantically similar examples away, which certainly compromises the clustering performance.
22

---

Under review as a conference paper at ICLR 2022
a) Normalized Mutual Information
b) Standard Deviation
-0
NMI
STD
0.4
Method
Method
BYOL
BYOL
NCC
0
200
400
600
800
1000
200
400
600
800
1000
Epoch
Epoch
c) Cluster Imbalance Ratio
d) Cluster Statistics
0.8
Method
Imbalanos Ratio
Cluster Count
0.6
Method
NCC
7500
BYOL
0.4
NCC
5000
2500
0
200
Epoch
Cluster Index
Figure A7: Visualization of training and cluster statisties for BYOL and NCC: a) normalized mutual in- formation (NMI)' between the clustering results and ground-truth labels; b) standard deviation (STD)² of fg-normalized features to evaluate the uniformity, where the higher indicates the more uniform rep- resentations; c) cluster imbalance ratio computed by Nmin/ max. where N the number of samples in each class; and d) cluster statistics, or the sorted number of samples in each cluster for the model 1000-th epoch on CIFAR-10. Taking look at NMIs and STDs during the training stage in (a) and (b), NCC produces higher NMIs with stable and higher STDs, while BYOL performs unstable with the STDs gradually decreased. The results indicate that NCC yields more uniform representation space with the samples well-clustered. On the other hand, the uniform representations of NCC can also avoid the collapse of k-means clustering at the same time. As shown in (c), the k-means clustering process of NCC produces more balanced clusters with higher cluster imbalance ratio. On the contrary, the clus- ters of BYOL. are highly imbalanced, which is consistent with the unstable NMIs and decreasing STDs. Moreover, visualize the fourth cluster statistic in (d), with the sorted number of samples. Unsur- prisingly, the samples for NCC are approximately and equally assigned to different clusters, compared almost long-tailed assignments of YOL.
NMI is normalized mutual information between the clustering and ground truth labels.
Here we visualize the distribution of representations fallowing (Chen He 2021). Given the latent vectors ~ (O 7), the standard
deviation E2 nononllized
dimension of discussed Appendix noromalizzed standard Gaussian distribution can be approximately seen as the uniform vMF distribution on the hypersphere. Therefore,
=/| 2|
[='] 1/d2,
higher standard deviation close to 1/d indicates more uniform representations.
Same coefficient to control the data imbolance for long-tailed datasets in Table
A4 higher value indicates more balanced clusters.
23

---

Under review as a conference paper at ICLR 2022
CIFAR-10
ACC CIFAR-10
CIFAR-10
100
100
100
Porformanco (N0
2oloommn Comprnie (%)
Performance (%)
70
70
70
50
90
90
NCIC
NCC
NCC
40
40
128
258
912
1024
2048
128
2.88
512
1024
2048
128
255
512
1024
2048
NM CIFAR-20
ACC
CIFAR-20
ARI for CIFAR-20
60
60
Purfmmarss M80
Perfommnoc
40
80
40
300
10
10
1.2
258
912
1004
2048
128
258
512
1024
2048
128
258
512
1024
2048
Figure A8: The effect of the different projection dimension. NCC achieves consistent and significant performance improvement over BYOL regardless of different projection dimension.
a)
100
100
100
Method
Method
Method
90
90
90
Parforannd (N)
Performance (%)
Performance (N)
50
50
90
40
40
Bassline
Graysuale
No
Basnino
No Groyccil
No
Crip
Bomalin
No Groyscato
No
NII for CIFAR-20
ACC CIFAR-29
ARI for CIFAR-20
Method BYOL
Melhod
Method
BYOL
BYOL
80
80
Performnc (80
50
Performance 196)
50
Performance (%6)
50
40
80
40
Boveinn
No Grayscato
No Color Jiter
Crop only
Bawiin
No Grayenale
No Onter Jibe
Crop only
Bessinn
No Grayscdo
No Ontor J0M
Croo orly
Figure A9: The effect of the different data augmentation. is not surprised to see the performance drops for both BYOL and NCC when removing some data augmentations. On the contrary, the clustering results suggest that NCC still performs more stable and is robust to data augmentations.
24

---

Under review as a conference paper at ICLR 2022
Table Al: Clustering results (%) for fair comparisons. We train NCC demonstrate its effective- ness for fair comparisons with the following settings: 1) we exclude test set from the whole dataset; and 2) we use an original image size (224) for ImageNet- 10 and ImageNet-Doge All results were trained with ResNet-34. There is no clear margin for CIFAR- 10 and CIFAR-20 datasets with dif- ferent splits, while significant improvements can be observed for ImageNet- 10 and ImageNet-Doge datasets. Considering that NCC has already achieved state-of-the- the-art performance against previous work in Table these results further demonstrate the superiority of NCC.
NMI
ACC
ARI
NMI
ACC
ARI
CIFAR-10
CIFAR-20
Baseline (train+test) Exclude test set
88.6+11.0
94.3+1.6
88.4=1.1
60.6+0.3
61.4
45.1=0.1
88.3+0.2
94.2=0.2
88.1
61.2#0.9
61.5+009
45.9=0.5
ImageNet-10
ImageNet-
Baseline (96)
89.6110.2
95.6.10.0
90.6
69.210.3
14.5
62.7.0.1
Large image size (224)
90.810.4
96.2.1.11
91.
73.710.2
77.510.1
67.5
Clustering results (%) on the subsets of ImageNet. We strictly follow the settings in [Van Gansheke et al., 2020p: we have adopted the same 50, 100, and 200 classes from ImageNet, clustered on the training set and tested on the validation set. We have used the same experimental settings as the other benchmarked datasets and trained NCC with ResNet-50 for 300 epochs. We note that SCAN has used the pre-trained model of MoCo trained on the full ImageNet for 800 epochs. The results are directly referred from their published paper including k-means with pre- trained MoCo', SCAN after the clustering step?. and SCAN after the self-labeling step?. With much fewer training epochs and training data, NCC still produces better performance with clear margin, demonstrating the superiority of NCC.
Table A2:
ImageNet
50 Classes
100 Classes
200 Classes
Method
NMI
ARI
NMI
ARI
NMI
ARI
k-means'
77.5
57.9
76.1
50.8
75.5
43.2
SCAN2
80.5
63.5
78.7
54.4
75.7
44.1
SCAN3
82.2
66.1
80.8
57.6
77.2
47.0
NCC (Ours)
82.8
69.1
83.5
63.5
80.6
53.8
Table A3:
Clustering results (%) on Tiny-ImageNet (Le & Yang 2015). Imag-Im is also
a subset of ImageNet, with total of 200 classes and 500 images for each class. We trained NCC with same settings in Table exceptt for ResNet-18 and the image size 96 96. Even though with smaller backbone and image size (CC used ResNet-34 and the image size 224 x 224), our NCC still produces promising improvements over previous state-of-the methods.
Dataset
Tiny- ImageNet
Method
NMI
ACC
ARI
k-means (Lloyd 1982)
6.5
2.5
0.5
SC
ZZennik-manor manor & Perona 2005
6.3
2.2
0.4
AE (Bengio et
2007
13.1
4.1
0.7
VAE (Kingma & Welling
2014
11.3
3.6
0.6
JULE
Yang et al.
2016
10.2
3.3
0.6
DEC (Xie et al. 2016)
11.5
3.7
0.7
DAC CCang et al. 2011
19.0
6.6
1.7
DCCM (Wu al. 2019
22.4
10.8
3.8
PICA Huang et al. 2020
27.7
9.8
4.0
CC
(L.
al. 202 hb)
34.0
14.0
7.1
GCC (Zhong et al.
2021)
34.7
13.8
7.5
NCC (Ours)
40.5
25.6
14.3
25

---

Under review as a conference paper at ICLR 2022
Table A4: Clustering results (%) on long-tailes datasets of different self-supervised learning frame- works and our proposed NCC. We built the long-tailed version of CIFAR-10 and CIFAR-20 termed 10-AR- and CIFAR-20-L33 using the codes of (Tang et al. 2020p, which follows (Zhou al, Cao et al. 2019). Specifically, they were built upon the training datasets under the control of data imbalance ratio /in /Naxx 0.1 for the data distribution, where is the number of samples in each class. The samples in the long-tailed datasets are almost all in the head of distributions. MoCo cannot handle this problem well due to the class collision issue, as a results, the samples the head will be pushed away and the ones in the tail will be mixed together. The rest three methods do not need the negative examples so that they outperform MoCo v2 by a large margin. By introducing the positive sampling and ProtoCI, can further boost and stabilize the performance of vanilla BYO..
2020
CIFAR-10-LT
CIFAR-20-LT
Methods
NMI
ACC
ARI
NMI
ACC
ARI
MoCo v2
46.7+0.1
33.4=0.3
27.7=000
31.2+0..
28.2+00.
16.1=0.3
BYOL
51.61110
41.3.10.4
30.810.4
41.910.4
34.6.10.5
22.
NCC
55.3+0.4
43.9-0.1
36.3+0.3
44.6+0.2
39.0+0.7
27.3-0.2
w/o ProtoCL
53.110.7
42.710.4
31.610.8
43.4
35.1
24.0=0.1
Table A5: Clustering results (%) on different ResNet architectures. With the deeper ResNet net- works, there are little performance gain and even drops for MoCo v2 and BYOL. On the contrary, NCC achieves significant improvement with small standard deviation for clustering, demonstrating its superior stability and performance against the baseline MoCo v2 and YOL.. The best and second best results are shown in bold and underline, respectively.
Backbone
Method
Dataset
NMI
ACC
ARI
Dataset
NMI
ACC
ARI
MoCo
76.9+0.2
84.9=0.3
72.4+0.5
49.2+1..1
48,0=0.2
32.1 32.1000.
ResNet-18
BYOL
79.4+1.7
87.8=1.7
76.62288
55,540.6
53,9=1.6
37.6=0.9
CIFAR-10
85.1
91.6=0.4
83,5+0.7
CIFAR-20
58.2.+0.3
$7,8=0.2
42.3=0.3
MoCo
79.4++0.0
86.9=0.0
75.6+00.
51.3++0.0
49.2-0.6
34.3+00.3
ResNet-34
BYOL
81.7110
89.4-0.6
79.0+
55,9110.3
56,9=188
39.3+0.2
Ours
88.6:1100
94.3=0.6
8..4+1.1
60.6:10.3
61.4-11
45.1+0.1
MoCo
79.9+0.11
87.1=0.1
76.0=0.1
53.3++0.0
52.9=0.1
36.7+0.1
ResNet-50
BYOL
80.0+0.4
88.1=0.9
76.3+18
54.1+-14
53.1=2.4
36,9+2.9
Ours
89.4100.5
94.7=0.4
89.1
63.2++100
61.3=1.2
47.1=14
MoCo
83.3+.0.1
90.4=0.1
81.6#0.1
38.7:t0.5
36.7=0.5
23.9=00.6
R8sNet-18
BYOL
87.2410.6
94.2=0.5
87.611.1
59.6:10.8
66.1=1.1
49.8
ImageNet- Negg Dogs
Ours
89.3+0.4
95.4=0.1
89.7
64.4111.3
69.6-2.2
56.1+2.3
ImageNet-11
MoCo
84.24-0.1
90.7=0.1
82.3+0.1
41.6+11.
39.0=1.5
26.8+117
ResNet-34
BYOL
86.64002
93.9=0.1
87.2+0.2
63.54222
69.4-3.0
54.8+29
89.61002
95.6=0.0
90.620.1
69.2400.3
74.5-0.1
62.7-0.1
MoCo
82.6:10.1
90.0=0.00
80.9+0.0
40.8+110
38.6=11
25.4+10.7
ResNet-50
BYOL
88.7+11.8
95.2-0.9
89.7+1..
66.2400.
71.9-12
58.1+12
Ours
91.5:10.22
96.5-001
92.5+0.2
72.0-102
76.3-0.1
65.7-0.1
26


Reviews:
[{"review_id": "z097g_s_g6", "reviewer": "Reviewer_Na7g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) The writing and organization are good, which makes the paper easy to read and follow.\n\n(2) The authors conduct extensive experiments, including the ImageNet to show the superiority. The results are good.\n\nCons:\n\n(1) My main concern lies in the novelty, which is very limited. This paper simply combines two existing unsupervised learning methods, BYOL and PCL, and then applies it to the clustering task. The class collision issue for negative samples contrastive learning methods is well addressed by BYOL instead of this paper. The difference between the proposed ProtoCL and PCL comes from the BYOL. The EM framework is also presented by the PCL. Besides, the neighbor examples based positive sampling strategy has been well investigated in both unsupervised feature learning and deep clustering areas [1,2]. Therefore, the only contribution is the combination and application to the clustering task, which is obviously insufficient.\n\n(2) Two important references [1,2] are missing. These two methods thoroughly study the positive sampling strategy for clustering and unsupervised feature learning to handle the issue of inaccurate negative samples. Therefore, the positive sampling strategy in NCC is not new. Please also compare the results with these methods.\n\n(3) Though the reported results are very high, the comparison might be unfair. Some existing contrastive learning based clustering methods, such as CC and GCC [1], can also adopted the framework of BYOL and PCL to improve the results. Therefore, the results might be unconvincing.\n\n[1] Graph Contrastive Clustering, ICCV 2021\n[2] Weakly Supervised Contrastive Learning, ICCV 2021", "summary_of_the_paper": "For deep clustering, this paper explores the non-contrastive representation learning based on BYOL to handle the issue of the class collision caused by inaccurate negative samples. ProtoCL is proposed to encourage prototypical alignment between two augmented views and prototypical uniformity, hence maximizing the inter-cluster distance. Experiments on various datasets demonstrate the superiority of the proposed method.", "summary_of_the_review": "This paper proposes the non-contrastive representation learning method for deep clustering based on BYOL. It is a combination of two existing methods. The novelty is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uDClw-f3wv", "reviewer": "Reviewer_nh13", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n\n### Good performance\n\nThe paper demonstrates a clear improvement over existing methods on various datasets. It is unfortunate that the method is not tested on the same settings as PCL in Table 2, but Table 3 does seem to show that there is a large enough gap between the two.\n\n### Well motivated\n\nThe motivation for the paper is really on the fact that negatives harm, and the paper shows how to effectively avoid this from happening. Have your clusters well spread, but don't ask for individual samples to be further away, simply ask them to be closer to each center. And by enforcing the centres to be spread, you naturally avoid asking two different samples close by to be further away, which may not be nice.\n\n## Weaknesses\n\n### Missing PCL in Table 2\n\nPCL is perhaps the exact antithesis of the proposed method. The method already shows to work better than PCL in Table 3, so why not include it in Table 2 to make it VERY clear that the proposed way is the way to go compared to PCL? This is personally a must-have experiment for me.\n\n### Ablation study\n\nThe ablation study can be improved by perhaps also showing how the ProtoNCE's formulation of negative losses harms performance. For example, what happens if you simply add that term to the NCC formulation? \n\nAlso, from the ablation study, it is slightly questionable whether the alignment loss is necessary. In fact, shouldn't the positive sampling also do the same thing as the alignment loss? \n\n### Presentation\n\nThe presentation can be improved. The paper is not too hard to follow but could be presented in a more top-down way. This might be a presentation style issue, and it does come to personal taste, so please do take this into account only as a suggestion.\n\nThe abstract does not read well. There are multiple novel terms introduced, \"a positive sampling strategy\" and \"prototypical contrastive loss\" that are not explained, which makes it hard to follow. For example,\n\n> First, we propose a positive sampling strategy to align one augmented view of instance with the neighbors of another view so that we can avoid the class collision issue caused by the negative examples and hence improve the within-cluster compactness.\n\nCan be simply written as\n\n> First, we relate between neighbouring instances in the learned latent space to improve within-cluster compactness leading to less collision of non-related classes.\n\nPrototypical contrastive loss also can be presented in a similar way, without necessarily introducing the term. In fact, I would personally argue that it is easier to understand the method the other way around. The main idea of the paper seems to be that negative samples are risky, therefore, let's cluster samples in the latent space to form prototypical clusters, and apply contrastive loss only on the cluster centres and not on the features themselves, unlike in the case of ProtoNCE where all samples are optimized. This however risks clusters becoming meaningless, hence the cluster grouping loss which maximizes the likelihood of each sample being from a nearby cluster with a Gaussian assumption.\n\nThis problem persists also in the introduction, as the reader is unaware of what the paper means by \"positive sampling\" until Section 3.2, and Prototypical contrastive loss until 3.3. This problem is exacerbated since it is unclear what 3.2 and 3.3 are trying to achieve until you reach 3.4, which provides the picture in which all of these fit in place.\n", "summary_of_the_paper": "The paper proposes that negative samples are risky, therefore, let's cluster samples in the latent space to form prototypical clusters, and apply contrastive loss only on the cluster centres and not on the features themselves, unlike in the case of ProtoNCE where all samples are optimized. This however risks clusters becoming meaningless, hence the cluster grouping loss which maximizes the likelihood of each sample being from a nearby cluster with a Gaussian assumption. The method is tested on multiple datasets outperforming all compared methods.", "summary_of_the_review": "The paper shows promising results, with a great motivation. However, there are some issues that can be easily fixed to greatly improve the quality of the paper. Specifically, ProtoNCE, or PCL, should be included in Table 2, and an additional baseline that empirically shows the negative loss harming performance would be great. Presentation could also be improved to be more clear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qsh4TYo-bAo", "reviewer": "Reviewer_bq8Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. strengths\n\n-the authors provide a sufficient literature review.\n-the paper is well organized.\n-detailed appendix, including ELBO, convergence analysis, experimental setup, etc.\n-abundant experimental results.\n-large performance improvement.\n\n2. weaknesses\n\n-the non-contrastive representation motivation maybe overstated. L_{aug-ins} is a non-contrastive loss, but L_{pcl} is a typical contrastive loss. From Tab.4, L_{pcl} may be more important than L_{aug-ins}.\n-the proposed L_{pcl} is not novel enough. This loss is actually class-level contrastive loss, which is proposed in the work (Contrastive clustering, AAAI 2021)\n-the EM-based framework is not novel enough. ProtoNCE firstly adopt this framework in representation clustering related area.\n-the reviewer would like to see more clustering results in some challenging datasets, such as Tiny-ImageNet, ImageNet-50/100/200 subsets (in SCAN, ECCV 2020).\n", "summary_of_the_paper": "This paper proposes a novel deep clustering method with non-contrastive representation motivation. The authors provide detailed experimental results to show the superior performance of the proposed method.", "summary_of_the_review": "Although the reviewer provides some negative feedback in Main Review, the reviewer still likes this paper. Therefore, I will give a positive rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nmn7UiISpH", "reviewer": "Reviewer_dith", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n* The comparison with the prior works and the motivation of the proposed method are well described and reasonable.\n* The efficacy of the proposed methods are well proven with the experimental results.\n* A lot of experimental results on hyperparameters search show that the proposed method is quite robust.\n\n\n### Weaknesses\n* Class-collision issue arose from k-means clustering\n\n    I think it's kind of a chicken and egg situation, cause if the space is not well uniformly distributed, the result of the k-means clustering would be quite degenerated which would result in a severe class collapse issue.\nHave you checked the k-means clustering results with the ground truth class labels?\n\n* Lack of analysis on the over-clustering results\n\n    Based on the results in Table 4 and Figure A5, it seems that the pre-defined value of the number of clusters, K, has the largest impact on the performance (more than 10% in terms of accuracy score). It is quite intuitive to think that the more the clusters are set, the better the classification performance (cause there are more exemplars to infer the class), but the results are the opposite of this idea for the CIFAR-10 dataset. Since it is the factor that impacts the performance most, I think more elaboration of this result is required.", "summary_of_the_paper": "This paper proposes a novel method of self-supervised representation learning.\nTo circumvent the class collision issue arose from building a large set of negative samples in contrastive SSL based methods, it is built from non-contrastive SSL methods, such as BYOL.\nThe goal is to handle the weaknesses of non-contrastive SSL methods, training instability and representation collapse.\nThe two proposed methods for alleviating the two factors are i) augmented positive sampling and ii) optimizing uniformity of representation space via prototypical cluster features computed from k-means clustering.\n", "summary_of_the_review": "I think this paper is well written in that the motivation, the description of methodology and the analysis of the results are all reasonable.\nThough I feel more details are needed for some factors that i described in the main review section, I think this work is good since the proposed method has pointed out the weaknesses of the previous methods and improved it experimentally.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received the initial scores with large variance. During the intensive discussion (Number of Forum replies is up to 60), the opinions reached the consensus. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the recommendation on this paper, *rejection*. Here are the comments that I summarized, which include my opinion and evidence.

**Research Problem and Motivation**

(1) It seems that the authors aimed to address the question that “are negative examples necessary for deep clustering?” This research problem has been proposed and addressed in BYOL and SimSiam (If I remembered correctly, some reviewer pointed this out). What the authors actually did is to add two more components, positive sampling strategy and prototypical contrastive loss, on the top of BYOL. In my eyes, it is like putting two patches on BYOL, where one of them does not work (I will explain later). 

(2) Moreover, the authors failed to clearly illustrate the drawback of BYOL. In the last sentence of the third paragraph in the Introduction part, the authors mentioned that “BYOL only optimize the alignment term, leading to unstable training and suffering from the representation collapse.” This sentence is too general, which lacks strong motivation. 

Therefore, the research problem addressed here is an incremental problem over BYOL, rather than brings new insights into the contrastive learning community.

**Philosophy**

Without a clear motivation, it is difficult to catch the philosophy of this paper, i.e., how their proposed components tackle BYOL’s drawbacks. Moreover, the relationship between two components is also unclear. 

**Novelty**

I believe Reviewer Na7g has a thorough analysis of the novelty of this paper. I will not go into details here. The difference does not mean novelty. 

**Technique**

Positive sampling strategy does not work. If we take a closer look at Table 4, the rows of BYOL and NCC with PS, there is no significant performance gain. The p-values of t-test on ACC results on CIFAR-10 and CIFAR-20 are 0.92 and 0.32, respectively. Actually, the prototypical contrastive loss is the key element to boost performance over BYOL.

**Misleading Title**

Based on the above point, the title is misleading. Although no negative data sample pairs are used in the training, the contractiveness on the cluster-level should also belong to the scope of contrastive learning.

**Experiments**

(1) In the Introduction part, the authors mentioned that SimSiam is in the same non-contrastive category with this paper. However, this paper is not included in the comparison.

(2) The competitive methods in Table 2 and 3 are not consistent. The authors even did not report the performance of BYOL on ImageNet-1K.

(3) Positive sampling strategy does not work. See the above Technique point.

(4) The authors only reported the running time on CIFAR-10 and CIFAR-20.

Therefore, the experimental results are not very convincing and solid to me.

**Presentation**

I believe the presentation also needs many efforts to smooth the logic. For example, “Even though Grill et al. (2020); Richemond et al. (2020) have proposed to use some tricks such as SyncBN (Ioffe & Szegedy, 2015) and weight normalization (Qiao et al., 2019) to alleviate this issue, the additional computation cost is significant.” Actually, Grill et al. (2020) is BYOL, where the authors added their components on. The computational cost of the proposed method should be heavier than BYOL. 

Based on the above points, this paper suffers from several severe issues, which makes it not self-standing.
Decision: Reject