Learning Representations by Contrasting Clusters While Bootstrapping Instances
Authors: Junsoo Lee
Hojoon Lee
Inkyu Shin
Jaekyoung Bae
In So Kweon
Jaegul Choo
Venue:  (2021)
Keywords: unsupervised
self-supervised
image clustering
visual representation learning

Abstract: Learning visual representations using large-scale unlabelled images is a holy grail for most of computer vision tasks. Recent contrastive learning methods have focused on encouraging the learned visual representations to be linearly separable among the individual items regardless of their semantic similarity; however, it could lead to a sub-optimal solution if a given downstream task is related to non-discriminative ones such as cluster analysis and information retrieval. In this work, we propose an advanced approach to consider the instance semantics in an unsupervised environment by both i) Contrasting batch-wise Cluster assignment features and ii) Bootstrapping an INstance representations without considering negatives simultaneously, referred to as C2BIN. Specifically, instances in a mini-batch are appropriately assigned to distinct clusters, each of which aims to capture apparent similarity among instances. Moreover, we introduce a pyramidal multi-heads technique, showing positive effects on the representations by capturing multi-scale semantics. Empirically, our method achieves comparable or better performance than both representation learning and clustering baselines on various benchmark datasets: CIFAR-10, CIFAR-100, and STL-10.

Under review as a conference paper at ICLR 2021
LEARNING REPRESENTATIONS BY CONTRASTING CLUSTERS WHILE BOOTSTRAPPING INSTANCES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning visual representations using large-scale unlabelled images is a holy grail for most of computer vision tasks. Recent contrastive learning methods have focused on encouraging the learned visual representations to be linearly separable among the individual items regardless of their semantic similarity; however, it could lead to sub-optimal solution if given downstream task is related to non-discriminative ones such as cluster analysis and information retrieval. In this work, we propose advanced approach to consider the instance semantics in unsupervised environment by both i) Contrasting batch-wise Cluster assignment features and ii) Bootstrapping INstance representations without considering negatives simultaneously, referred to as C2BIN. Specifically, instances in a mini-batch are appropriately assigned to distinct clusters, each of which aims capture apparent similarity among instances. Moreover, we introduce a multi-scale clustering technique, showing positive effects on the representations by capturing multi-scale semantics. Empirically. our method achieves comparable or better performance than both representation learning and clustering baselines on various benchmark datasets: CIFAR-10, CIFAR- 100, and STL-10.
INTRODUCTION
Learning to extract generalized representations from tigh-dimensional image is essential in solving various down-stream tasks in computer vision. Though supervised learning framework has shown be useful in learning discriminative representations for pre-training the model, expensive labeling cost makes it practically infeasible in large-scale dataset. Moreover, relying the human-annotated labels tends to cause several issues such as class imbalance (Cui et al., 2019), noisy labels (Lee) al. 2019), and biased datasets (Bahng et al. 2019). To address these issues, self-supervised visual representation learning. which does not require any given labels, has emerged as an alternative training framework, being actively studied to find proper training objective.
Recently, self-supervised approaches with contrastive learning (Wu et al. 2018; Chen et al. 2020aa He al., 2020) have rapidly rowed the performance gap with supervised pre-training in various vision tasks. The contrastive method aims learn invariant map- ping (Hadsell et al. 2006) and instance discrimination. Intuitively, two augmented views of the same instance mapped the same latent space while different instances are pushed away. However, aforementioned instance discrimination does not consider the se- mantic similarities of the representations (e.g., same class), even pushing away the relevant instances. This affects the learned rep- resentations to exhibit uniformly distributed characteristics, proven by the previous works [Wang & Isola 2020 Chen & Li (020).
Class Class
Figure 1: Though illustrated 2D representations are linearly sep- arable, irrelevant instances are clustered together.
We point out that this uniformly distributed characteristic over instances can be a fundamental limitation against improving the leamned representation quality. For instance, consider the represen- tations illustrated in Fig. [ It indicates a simple case where linearly separable representations do not always guarantee that they can

---

Under review as a conference paper at ICLR 2021
Detases
x
x
Centroid * Data Points
Pull Push
(a) Clusterwise Contrast (b) Instance-wise bootstrap (c) Final Representation
Figure 2: Visual illustration of how our method leads to both linearly separable and clusterable repre- sentations. While semantically unrelated samples are pushed apart with the cluster-wisg contrastive loss, the invariant mapping can be maintained by our instance-wise bootstrapping loss.
be properly clustered, which is not appropriate for non-diseriminative downstream tasks such as information retrieval, density estimation, and cluster analysis (Wu al. 2013). response, we start this work by asking: How can we learn the representations to be properly clustered even without the class labels?
In this work, we propose self-supervised training framework that makes the learned representations not only linearly separable but also properly clustered, as illustrated in Fig. 2 mitigate the uniformly distributed constraint while preserving the invariant mapping, we replace the instance discrimination with an instance alignment problem, pulling the augmented views from the same instance without pushing away the views from the different images. However, learning the invariant mapping without discrimination can easily fall into a trivial solution that maps all the individual instances to single point. To alleviate this shortcoming, adopt bootstrapping strategy from Grill et al. 2020), utilizing the Siamese network, and a momentum update strategy (He et al. 2020).
In parallel, to properly cluster the semantically related instances, tional cluster branch. This branch aims to group the relevant representations by softly assigning the instances to each cluster. Since each of cluster assignments needs be discriminative, contrastive loss to the assigned probability distribution over the clusters with simple entropy-based regularization. In the meantime, we constructed the cluster branch in multi-scale clustering starategy
motivated to design addi-
employ the
where each head deals with different number of clusters (Lin et al. 2017). Since there exists various granularity of semantic information in images, it helps the model effectively capture the diverse level of semantics as analyzed in Section 4.5
In summary, our contributions are threefold, as follows:
We propose novel self-supervised framework which contrasts the clusters while bootstrap- ping the instances that can attain both linearly separable and clusterable representations.
We present a novel cluster branch with multi-scale strategy which effectively captures the different levels of semantics in images.
Our method empirically achieves state-of-the-art results CIFAR-1O, CIFAR-100, and STL-10 representation learning benchmarks, for both classification and clustering tasks.
RELATED WORK
Our work is closely related to unsupervised visual representation learning and unsupervised image clustering literature. Although both have slightly different viewpoints of the problem, they essentially similar in terms of its goal to find good representations unlabelled datasets.
Instance-level discrimination utilizes an image index as supervision because it is unique signal in the unsupervised environment. NPID (Wu et al.) 2018) firstly attempts to convert the class- wise classification into the extreme of instance-wiss wise discrimination by using external memory banks. MoCo (He et al.) 2020) replaces the memory bank by introducing a momentum encoder that memorizes knowledge learned from the previous mini-hatch. SimCLR (Chen al. 2020a presents
2

---

Under review as a conference paper at ICLR 2021
that it is crucial for representation quality to combine data augmentations using pretext head after the encoder. Although recent studies show promising results on benchmark datasets, the instance- wise contrastive learning approach has critical limitation that it pushes away representations from different images even if the images have similar semantics, e.g., belonging to the same class.
Cluster-level bootstrapping is alternative paradigm that enhancing the initial bias of the networks can be useful in obtaining a discriminative power in visual representations, since convolutional neural networks work well capturing the local patterns (Caron et al., 2018 In the case of using pseudo-labels, K-means (Caron et al. 2018) or optimal transport (Asano et al. 2019 Caron et al. 2020) are commonly adopted for clustering. On the other hand, soft clustering methods have also been actively studied to allow flexible cluster boundaries (Ji et al. 2019 Huang et al., 20220). Recently, a 2-stage training paradigm has been proposed to construct the cluster structure initialized from the representations learned by instance discrimination [Gansbeke et al. 2020
METHOD
Our work is motivated by an observation from SupCLR (Khosla al. 2020), which additionally pulls the representations together from different instances by using groundtruth labels. However, directly applying this idea in an unsupervised environment with pseudo-labels is challenging, because small false-positive errors at the initial step can be gradually spread out, degrading the quality of final representations.
Instead, the main idea of our approach avoid pushing away those instances close enough to each other. To validate this idea, we conducted a toy experiment that a pulling force is only corresponding to two augmented views of the same image while not pushing the images within the same class by using the groundtruth label. We found that its classification accuracy increases over 5% on STL-10 datasets compared to that of SimCLR (Chen et al.) (2020a). Inspired by this experiment, we design model (i) not to push away relevant instances with our instance-alignment loss (Section 3.2) while (ii)
discriminating the representations in cluster-wise manner. (Section 3.3 3.4
pekk
Fo
Gø
pa
Cation
fetust
Go
Fo
data flow
loss flow
X
uudeeeloow
stop gradert
Figure 3: Overall architecture of proposed C2BIN.
3.1 PRELIMINARIES
As shown in Fig. we adopt stochastic data augmentation algorithms (Chen et al. 2020a He et al.
2020, Chen et al.
2020b
Caron et al. 2020) to generate two different augmented views x; and
J of the same image x;
NN) where is the number of unlabelled images.
Inspired by Luo et al. (2018) Grill et al. 2020h, C2BIN consists of instance predictor p*(.). (-), cluster predictors pa (.), and two Siamese networks called the runner E- and the follower Es (-). respectively. The runner Eg is rapidly updated to find the optimal parameters 0* over the search spaces, while the follower E, generates the target representations for the Eg. The Eg is composed of the two neural functions: encoder F.(.) and instance projector (.).). and vice versa for the follower
3

---

Under review as a conference paper at ICLR 2021
Eg. To bootstrap the instance-lvel alignment, Eg, Eg, and pa are used. Afterwards, Fg and pek are utilized to contrast the cluster-wiss features.
3.2 BOOTSTRAPPING LOss OF INSTANCE REPRESENTATIONS
obtain two augmented views z' (x: and ======== +x)x) where t' and t" are sampled from set of stochastic data augmentations 7 as mentioned above. Even though augmented views are distorted, they should contain similar semantics, and the learned representations should be closely aligned in the latent space. For training, we forward x" through the follower E to obtain target representations at an instance level; the runner Eg aims make the embedding vector of x' closer to them. That is, we first extract image representations r = Fe(z') E Rd, where d, the number of dimensions of our representations. Afterwards, we introduce a pretext-specific instance-wist projector (-) and then obtain pretext embedding vectors 2, (g)rr R1xda the
Given image x; X,
target pretext vectors 2a can be obtained using the same procedure by E Motivated from Grill et al.
2020 we calculate our alignment loss as the cosine distance as
Pa (z)) &
Latign
(1)
(Z.. Zu 2 E Rxxda and adopt the number of dimensions of projected features da in Chen et (2020a
1
where Pa
3.3 CONTRASTIVE Loss OF BATCH-WISE CLUSTER ASSIGNMENTS
Our high-level motivation of this branch is that image feature can be represented the combination of cluster features capturing local patterns. However, grouping similar images conflict with the instance-level invariant mapping: therefore, introduce additional branch which contains cluster predictor (c) after the encoder (.). The cluster predictor pe, is a linear function whose takes r; as an input and transform it a K-ddimensional output vector. Therefore, of (r.)(r.)) represents a degree of confidence for the i-th image representations r; to belong to the k-th cluster feature, i.e.,
= 1 R E GRRR RI
(2)
where o indicate cluster membership distribution of the given image x;, Since sample items for training, Zo E Ro is the set of memberships distribution of the given mini-batch. Now we define batch-wise cluster assignment vectors (BCAs) Ck as
C₁ Z
E RIXI
which indicates how much the k-th cluster is mapped by images in the mini-batch. Although will dynamically change as a new mini-batch is given, the same cluster features between differently augmented views from the same image should be similar while pushing away the others to capture diverse patterns. To this end, we simply utilize the contrastive loss between the BCAs as
beaa clust
Σ log
K
EE "xpjeple o//7):
explc /T
(4)
K
where indicates temperature value. The vectors c' and c" are outputs of pa. following the encoder F6 by taking x' and x" respectively.
Unfortunately, most of the clustering based methods suffers from falling into degenerate solution where the majority of items are allocated in a few clusters, especially in an unsupervised environment.
To mitigate this issue, we first compute the mass of assignment to k-th cluster as ΣN C₁ (i) where C (i), indicating each element of Cę Afterwards, we encourage ry to be stochastically activated for diverse cluster features as much as possible by maximizing an entropy of s. To this end, we formulate the cluster loss function as
Lotust Lbbaa
Hen).(s),
(5)
where indicates an entropy function as H(s) = ΣV 8; log s; and Aent is the weight value for the regularization term.
4

---

Under review as a conference paper at ICLR 2021
3.4 MULTI-SCALE CLUSTERING STRATEGY
The multi-scale clustering strategy has often been used in prior research (Vaswani et al. 2017: Asano ict al. 2019), leveraging the ensembling effect. Extending this strategy, we propose a multi-scale clustering strategy for our task. Although contrasting between the BCAs encourages capture various aspects of local patterns, the performance may be sensitive to the number clusters k. To address this issue, introduce set of the cluster branches that have different number of cluster assignments in each scale. To this end, we reformulate Letust as
model to
Letust Ef k E K.
(6)
In this work, various values of k; e.g., K= {32, 64, 128).
3.5
TOTAL ORJECTIVE
Finally, our total objective function is written
Ltotal Latign + Actust Celust-
(7)
The parameters of the follower Eg gradually reflect those of the runner via
Φ 26 + (1 7,
(8)
where y indicates a momentum factor.
EXPERIMENTS
This section presents the experimental evaluation of C2BIN the standard benchmark datasets including CIFAR-10, CIFAR-100, STL-10, and ImageNet, which are commonly adopted in both self-supervised representation learning and unsupervised image clustering literature.
In Sections 4.3 we compare C2BIN with several representation learning methods and unsuper- vised clustering methods to verify that model can yield both lincarly separable and clusterable representations. Afterwards, Section 4.4 studies the robustness C2BIN an iaas-iimbllaeeed setting. Lastly, Section 4.5 presents ablation study for in-depth analysis of model behaviour.
4.1 REPRESENTATION LEARNING TASKS ON UNIFIED SETUP
Experimental setup. Because previous studies have their experimental settings in terms of datasets and backbone architectures, we prepared for a unified experimental setup for the fair comparison follows. We first employ the ResNet-18 architecture as the backbone architecture, following Wu et al. (2018). We used 3 standard benchmark datasets; CIFAR-10, 10, CIFAR- 100, and STL-10 for this experiment. All baselines are trained by using the identical data augmentation techniques, as used in Chen et al. 20200| Further training details can be found in Appendix A.
Evaluation metries. We adopt three standard evaluation metries: linear evaluation protocol (LP) (Zhang et al., 2017), --nearest-neighbourr (kNN) classifier with & 5 and &o 200. For the LP. we follow the recipe Grill et (2020). where we report the best evaluation score over five differently initialized learning rates. For KNN, we follow the settings used in Wu et al. (2018) 8) and the implementation of Asano et al. (2019).
For our baseline methods, we choose SimCLR, MoCo v2, and BYOL, which can work as the state- of-the-art methods for the instance-wise contrastive learning, momentum-based contrastive learning, and instance-wist bootstrapping, respectively. As seen in Tab. I C2BIN consistently outperforms all baselines across all benchmark datasets In the case of CIFAR- 100, the kNN accuracy of C2BIN significantly improves compared the baselines while its LP scores consistently increase well. We conjecture the reason is becuase C2BIN is appropriate to learn a hierarchical structure a dataset such CIFAR- 100
5

---

Under review as a conference paper at ICLR 2021
gap+I.P
Ar. Method
CIFAR-I0
CIFAR-100
STL-10
DC (Caron al.
2018)
-1.767.6
NPID (Wu et al.
2018
/ 80.8
-/-/51.6
SimCLK Chen al.
2020a))
81.3 82.4/81.7
59.8/ 66.8
83.1 70./78.3/77
83.1/78.3/78.. 78.8
ResNet-18
MoCo (He et al. 2020)
78.1 /9.17 / 77.7
50.2/ 58.2
19.4/4..11 74.2
MoCo V2
Chen et
2020c
81.6 81.4/114// 83.9
61.1/6288/33
83.5 78.2 78.11
BYOL Grill et al.
2020)
80.8/814/77 79.7
67.0/6333/1.
80.4 / 77.5
C2BIN [Mean] (Ours)
81.5/ 84.9
61.5/7222/2.6
83.6 79.1 80.1
C2BIN [Best] (Ours)
82.3 85.9 85.3
62.1 73.9 73.8
84.0 79.9 / 80.8
Table 1: Comparison with unsupervised representation methods. Note on for the fair comparison, we did not used the average gradient trick that was utilized in BYO (Grill et al. 2020).
4.2 REPRESENTATION LEARNING TASKS ON LARGE SCALE BENCHMARK
Experimental setup. To compare our method with concurrent and state-of-the -hrr--rt works in large scale dataset, we evaluate our method in ImageNet with ResNet-50 architecture as a backbone model. For fair comparison, most of the results are taken from the experiments that models are trained for 200 epochs with a batch size of 256. All of the baselines are trained with identical data augmentation techniques introduced (Chen
2020a Further training details can be found in Appendix A.2
Method
Epochs
Batch Size
Top-1 accuracy
NPID WW et al.
2018)
200
256
56.5
MoCo
He et al.
2020)
200
256
60.6
SimCLR (Chen al. 2020a)
200
256
61.9
MoCo v2 Chen et al.
2020c)
200
256
67.5
BYOL Grill et
2020
200
256
64.3
C2BIN (Ours)
200
256
64.4
SimCLR (Chen al. 2020a)
400
4096
68.2
SwAV CCaron et al.
2020
400
4096
70.1
MoCo v2 (Chen et al.
2020c
800
256
71.1
Table 2: Linear classifier top-1 accuracy comparison with unsupervised representation methods on ImageNet. Methods are arranged in chronological order. Note on f: BYOL (Grill et al. 2020) does not report the result ImageNet with the identical experimental setup. Therefore, adopted the results of BOL from Zhan et al. [2020). the most widely used open-source library for self-supervised learning, experimented without an average gradient technique for a fair comparison.
Though C2BIN has shown competitive performance compared to the baselines, MoCo v2 CChen le al. 2020c outperforms C2BIN on the large-scale dataset, which contradicts the findings in Section
Since C2BIN utilizes batch-wise clustering techniques to learn the cluster structure, it brings instability to the training process when a large number of cluster size is required compared to the batch size. Still, our method slightly outperforms the state-of-the-a instance bootstrapping method,
BYOL Grill et al. 2020). This result implies that simultaneously learning the cluster structure is not counter-effective enhance the discriminative power of the instance wise representation learning method.
4.3 IMAGE CLUSTERING TASKS
This section compares our approach to the baselines in the unsupervised image clustering task.
Experimental setup. For fair comparison, we keep most of the implementation details identical (o et al.] (2019): Huang et al. (2020) except for excluding the use of the Sobel filter. We use the architecture similar ar to R4sNet-34, with the 2-layer ML. for both the instance projector G3(+) and

---

Under review as a conference paper at ICLR 2021
the predictor (.). (:). For the clustering branch, three cluster heads are used as K {10, 40, 160) for CIFAR-10 and 10L-10, and K {20, 40, 160) for CIFAR-100. Further training details can be found in Appendix B.1
Evaluation metrics. Three standard clustering performance metrics are used for evaluation: (a) Normalized Mutual Information (NMI) measures the normalized mutual dependence between the predicted labels and the ground-truth labels. (b) Accuracy (ACC) is measured by assigning dominant class labels to each cluster and take the average precision. (c) Adjusted Rand Index (ARI) measures how many samples are assigned properly to different clusters. All the evaluation metrics range between 0 and 1, where the higher score indicates better performance.
NMI/ ACC/ ARI
Method
CIFAR-10
CIFAR-100
STL-10
K-means
0.09 0.29 0.05
0.08/0.13/0m
0.13/0.19 0.06
DEC (Xie et al. 2016)
0.26 / 0.30 0.16
0.14/0.19/0.5
0.28 0.36 0.19
DCCM Wu et al. 2019)
0.50 0.6///62/// 0.41
0.29 /0.33/ 0.17
0.38/ /.48 0.26
IIC (Ji et al. 2019
-/0.62 --
-/0.26/-
-/0.61 /- 0.61 / 0.53
PICA (Huang et al.
2020
0.59 0.70 / 0.51
0.31 /0.34 0.17
C2BIN [Mean] (Ours)
0.62 / 0.72 / 0.53
0.36 0.35 / 0.20
0.62 0.73 0.55
C2BIN [Best] (Ours)
0.63 / 0.73/ / 0.55
0.38 / 0.38 / 0.22
0.64 0.75 0.57
Table 3:
Comparison with end-to-end unsupervised representation methods in the clustering bench- mark. The results of previous methods are taken from Huang al. (020). We append full comparison results in Appendix IT)
As shown in Table C2BIN outperforms the state-of-the clustering performance in all datasets by significant margin, showing its capability of grouping the semantically related instances to distinct clusters. Moreover, C2BIN is shown to be robust, given that the averaged performance over five random trials even surpasses the best results from the previous literature.
4.4 CLASS-IMBALANCER EXPERIMENTS
Unlike the standard benchmark datasets used, is often the case that the real-world image dataset severely imbalanced terms its underlying class distribution. Therefore, we conducted additional experiments in a class-ime imbalaeed environment, following the experimental design proposed in Cui
ct al. (2019).
a) CIFAR-
b) CIFA R-100
Decresse of the ancurayy
Decresse of the accurac
0
-3
-3
-6
-9
Ours
-9
Ours
SimCLR
SimCLR
-12
PICA
-12
PICA
OL
BY
1.0
0.7
0.5
0.3
0.1
1.0
0.7
0.5
0.3
0.1
Balanced Rate
Balanced Rate
Figure 4:
1 accuracy degradation when using the ResNet-18 architecture under a linear evaluation protocol in a inss-immalanced setting.
Fig. 4 demonstrates the classification accuracy degradation in an imbalanced setting. The balanced rate indicates the relative ratio of the largest the smallest classes. We find that the performance of PICA, the clustering based method, significantly decreases compared to other baselines as the class imbalance gets apparent. In the case of imbalanced CIFAR- SimCLR, which contrasts
7

---

Under review as a conference paper at ICLR 2021
all instances within the mini-batch, is shown to get degraded faster than BYOL, which does not consider the relationship between other instances. On the other hand, the accuracy degradation of C2BIN is shown to be minimal for both CIFAR-10 and CIFAR-100, possibly due to our alignment loss (Section
4.5 DISCUSSIONS
Qualitative analysis on the learned representations.
Epoch: 5
Epoch: 30
Epoch: 100
Ours
ARI: 0.16
ARI: 0.46
A RI: 0.55
SimCLR
ARI: 0.15
ARI: 0.39
ARI: 0.49
Figure 5: I-SNE 2-D embedding visualization of C2BIN and SimCLR.
For the test data items in STL-10 dataset, we embed their high- nigh-dimensional representations obtained by our method and SimCLR in a 2-D space using t-SNE van der Maaten Hinton (2008). As shown in Fig. 5] the representations learned from our model show clearer cluster structure than SimCLR(Chen)) et al. (2020a)), as training proceeds.
Query
Ours
SimCLR
PICA
alploon
Figure 6: Qualitative comparisons of the top-k retrieved images by C2BIN (Columns 2-6), SimCLR (Columms 7-11), and PICA (Columns 12-17) given a query image (Column 1) from the STL-10 test set where the & is set {1, 10, 50, 100).
To understand the characteristics of both instance-wise alignment and cluster-wise discrimination in straightforward manner, we conduct the image retrieval experiment. As shown in Fig. 6 our method outperforms two proposed baselines from both perspectives. Since SimCLR only focuses on instance-wise discrimination, it fails to retrieve with larger value of & with given query image (e.g., airplane). Likewise, PICA lacks the alignment capability in an instance wise manner, resulting in poor performance with a lower value of k in contrast to C2BIN, This is also corroborated by the quantitative results in Appendix (Fig. S).
Ablation study. To further verify whether our loss terms complementary to each other, perform an ablation study on STL-10 dataset. As we can observe in Table bbbb and [e), a simple integration of the clustering method into the instance-wise bootstrapping (Table can degrade the representation quality unless an appropriate level of granularity is provided. Similar to the results from Asano al, (209)), using simple multi-scale clustering branch with specific number of
8

---

Under review as a conference paper at ICLR 2021
clusters (Table and (e)) is more effective strategy than single-head method. Furthermore, proposed multi-scale clustering strategy (Table((()) peaks out the best performance since it allows the model to capture the diverse semantic information at different level. This result justifies our motivation to utilize a clustering strategy in a multi-scale manner.
Latign
clust
771
m2
mg
Eq.
Eq.
5
Eq. 6
multi-scale
K
(a)
80.4
76.5
77.5
(b)
k1
78.4
(-1.6)
73.5 (-3.0)
73.9 (-3.6)
(c)
K2
81.3 (+0.9)
76.3 (-0.2)
(-.05
(d)
k3
79.3 (-0.9)
(-2.))
(-2.0)
(e)
K4
82.2 (+1.8)
76.3 (-0.2)
76.4
(f)
k5
84.0 (+3.6)
79.9 (+3.4)
80.8 (+3.3)
Table Performance improvements due each of our components. my, m2, and my indicate the linear evaluation protocol (LP), kNN(k=5), and kNN(k=200), respectively. K denotes set of cluster sizes: k (32). 2/2 {128}, k {32, 32, 32}, k4 {128, 128, 128), and k; (32, 64, 128].
Visual analysis on multi-scale clustering strategy. We also show the visual analysis on the multi- scale clustering strategy. Each scale represents the different semantic information shown in Appendix (Figs. 10 and TT]. Combining this semantic difference in each scale prevents our model from binding specific number of cluster assignments.
CONCLUSIONS
In this paper, proposed novel approach to improve the existing representation learning with unsupervised image clustering. By integrating the advantages of both literature, we present an advanced self-supervised framework that simultaneously learns cluster features well as image representations by contrasting clusters while bootstrapping instances. Moreover, in order to capture diverse semantic information, we suggest multi-scale clustering strategy. We also conduct ablation studies to validate complementary effects of our proposed loss functions.

---

Under review as a conference paper at ICLR 2021
REFERENCES
YM Asano, C Rupprecht, and Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations, 2019.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. arXiv preprint (00i::v990002066, 2019.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Deng Cai, Xiaofei He, Xuanhui Wang, Hujun Bao, and Jiawei Han. Locality preserving nonnegative matrix factorization. In IJCAI, volume 9, pp. 1010-1015, 2009.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng. Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In Proceedings of the IEEE international conference on computer vision, pp.
5879-5887, 2017.
Jianlong Chang, Yiwen Guo, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep discriminative clustering analysis. Proceedings of the IEEE international conference on computer vision, 2019.
Ting Chen and Lala Li. Intriguing properties of contrastive losses. arXiv preprint arXiv:2011.02803 2020.
Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.051: 050000 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020c.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. effective number of samples. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268-9277, 2019.
loss based on
Wouter Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. Proceedings of the European Conference on Computer Vision, 2020.
K Chidananda Gowda and G Krishna. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 1052) 1978.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint 2XXv:20006.07333. 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 066), volume 2, pp. 1735-1742. IEEE, 2006.
Philip Haeusser, Johannes Plapp. Vladimir Golkov, Elie Aljalbout, and Daniel Cremers. Associative deep clustering: Training a classification network with no labels. In German Conference on Pattern Recognition, pp. 18- 32. Springer, 2018.
10

---

Under review as a conference paper at ICLR 2021
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern 2020.
Recognition, pp.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9865-9874 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 13662, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114,22 2013.
arXiv preprint
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, pp. 3763-3772, 2019.
Tsung-Yr Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 2117-2125, 2017.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep con- volutional generative adversarial networks. International Conference on Learning Representations. 2016.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 11 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, (1(2), 2010.
Tongzhou Wang and Phillip Isola. Understanding contrastive represennatiin learning through align- ment and uniformity on the hypersphere. arXiv preprint arXiv:2005. 2020.
Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8150-8159, 2019.
Weili Wu, Hui Xiong, and Shashi Shekhar. Clustering and information retrieval, volume 11. Springer Science & Business Media, 2013.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference machine learning, pp. 478 487, 2016.

---

Under review as a conference paper at ICLR 2021
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5147-5156, 2016.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint Ttti::770.038880 2017.
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition, pp- IEEE, 2010.
Zelnik-Manor, Lihi, and Pietro Perona. Self-tuning spectral clustering. In Advances in neural information processing systems, pp. 1601-1608, 2005.
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6688-6697, 2020.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
12

---

Under review as a conference paper at ICLR 2021
A REPRESENTATION LEARNING EXPERIMENTS
A.1 IMPLEMENTATION DETAILS FOR UNIFIED SETTING
Hyperparameter
Value
Epoch
500
Optimizer
LARS YYou et
2017
Learning rate
Constant(U:2)
Weight decay
le-6
Weight momentum
0.9
Batch size
256
Cluster weight (Celust)
2.0
Entropy weight (Centropy)
1.0
Architecture
ResNet-18
Representation dim
512
Instance projector G
MLP(512, 512) ReLU
Instance predictor Po
MLP(512, 512) ReLU
Momentum factor (7)
0.990
Temperature (T)
0.5
Cluster head (K1)
(CIFAR-I0, CIFAR-I00, STL-10)
Cluster head (K2)
64
Cluster head (Kş)
128
Table 5: Hyperparameters of backbone models used in the experiment of Section 4.1
Hyperparameter
Value
Epoch
300
Optimizer
LARS (You et al. 2017 Constant( 0.2, 0.3, 0.4, 0.5))
Learning rate
Weight decay
1e-6
Weight momentum
0.9
Batch size Architecture
256
linear without batch-norm layer
Table Hyperparameters of the linear evaluation protocol used the experiment of Section 4.1
13

---

Under review as a conference paper at ICLR 2021
A.2
IMPLEMENTATION DETAILS FOR THE BARGE-SCALLE SETTING
Hyperparameter
Value
Epoch
200
Optimizer
SGD
Learning rate
0.03
Learning rate schedule
multiplying 0.1 at 120, 160 epoch.
Weight decay
le-6
Weight momentum
0.9
Batch size
256
Cluster weight (Catust)
1.0
Entropy weight (Centropy)
0.05
Architecture
ResNet-50
Representation dim
2048
Instance projector G
MLP(2048, 4096) ReLU
Instance predictor P
MLP(2048, 4096) ReLU
Momentum factor (7)
0.990
Temperature (T)
0.2
Cluster head (K1)
512
Cluster head (K2)
1024
Cluster head (Kg)
2048
Table 7: Hyperparameters of backbone models used in the experiment of Section 4.2
Hyperparameter
Value
Epoch
200
Optimizer
SGD
Learning rate
30.0
Learning rate schedule
multiplying 0.1 60 and 80 epoch
Weight decay
le-6
Weight momentum
0.9
Batch size Architecture
256
linear without batch-norm layer
Table 8: Hyperparameters of the linear evaluation protocol used in the experiment of Section 4.2
14

---

Under review as a conference paper at ICLR 2021
A.3 IMPACT STUDY FOR CHOICE OF K
Although the effectiveness of the multi-scale clustering technique is briefly described in Section 4.5 this section studies performance changes according the choice of the set K.
Calign
etustt
LP (%)
Eq.
Eq.
Eq.
multi-
(32, 64, 128)
84.0
(b)
{8]
75.28 (-8.72)
(16)
79.06 (-4.94)
(d)
(512)
80.01 (-3.9)
(e)
{8, 888,8)
76.08 (-7.92)
(f)
{16, 16, 16)
80.08 (-3.92)
(g)
512, 512]
80.50 (-3.5)
(h)
16, 32}
83.50 (-0.5)
())
(16,32,64 64)
83.91 (-0.09)
Table 9: An impact study about the choice of K on the STL-10 dataset. LP indicates an linear evaluation protocol described in Section 4.1
Table 9 shows how C2BIN's performance is damaged for the linear evaluation protocol (LP) on the STL-10 I0 dataset when the combination of K is changed. In Table g
(a) is the best score reported in the main paper and can be used as a pivot for comparison. For the rest of them, similar to Section 4.5
we divide experiments into three groups. First, (b)-(d) are matched with the case of attaching single and arbitrary selected cluster size. Unfortunately, this case does not help to improve performances and even dramatically degenerates the representation quality. We guess that attaching a single cluster head after the backbone network makes its representation quality sensitive according to the head size. The second group, (c)-(g), corresponds with the case of multiple but single-scale cluster heads. Although seems slight improvement compared to the sufficiently complementary in our setting. We think the effect of the multi-branch clustering seems small because each cluster head can capture similar patterns with others. Lastly, (h)-(i) is mapped to the case of our multiple and multi-scale clustering strategy, showing robust performance in regard to the combination of K if each element of K is assigned in different scales. We guess that the effect of the multi-task learning is maximized because an identical representation vector should be informative enough to satisfy the following clusters, which are from abstracted to detailed.
case, is difficult to be
15

---

Under review as a conference paper at ICLR 2021
B UNSUPERVISED CLUSTERING EXPERIMENTS
B.1 IMPLEMENTATION DETAILS
Hyperparameter
Value
Epoch
300
Optimizer
Adam (Kingma & Ba 2015)
Learning rate
Cosine annealing (3c-4, 0)
Weight decay
No weight decay
Batch size
256
Cluster weight (Cotust)
1.0
Entropy weight (Centropy) Architecture
1.0
ResNet-34
Representation dim Instance projector G
512
MLP(512, 512) ReLU
Instance predictor P
MLP(512, 512) ReLU
Momentum factor (?)
0.995
Temperature (r)
1.0
Cluster head (K1)
10 (CIFAR-1O, STL-10), 20 (CIFAR-100)
Cluster head (K2)
40
Cluster head (Kç)
160
Table 10: Hyperparameters used in unsupervised clustering experiments Section 4.3
B.2
CLUSTERING QUALITY COMPARISON
NMI/ACC/ARI
Fwk
Method
CIFAR-10
CIFAR-100
STL-10
K-means
0.29/ 0.05
0.08 008/0.13/00 0.00
0.13/0.19/0.06
SC
(Zelnik-Maane Manor et al.
2005
0.10/ /0.09
0.09/0.14/0.00
0.10/0.16/0.05
AC (Gowda & Krishna
1978)
0.11/0.23 10/23/00 0000
0.10/0.14/0.3
0.24/0.33/02.
NMF (Cai et al. 2009)
0.08 / 0.19 /0.0
0.08 10.12/
0.10/0.18 0.05
AE [Bengio
2007)
0.24 0.34 /
0.10 0.05
0 0.16
DAE Vincent el al. 201
0.25 0.30 6
0.00
0.2/0.30/0.15
DCGAN (Radlord
2016)
0.27 0.32
0.12/0.15/0.
0.21
DeCNN
(Zeiler et al.
2010)
0.24 / 0.28 0..7
0.09 /0.13//00044
0.33 0.16 0.16
End-to-En End
VAE
Kingma Welling
(2013)
0.25 / 0.29 0.17
0.11/
0.20 0.28 0.15
JULE
Yang et al.
2016)
0.19 0.27 /0.4
0.10 /.100.14/00
0.18/0228 0.16
DEC
et
2016)
0.26 / 0.16
/.14/0.19/0..
0.28 0.19
DAC
Chang al. [2017
0.19 0.244..00009
/..7
ADC
Hacusser cl al. 2018
0.33 /-
-
0
DDC
Chang et al. 2019)
0.42 / 0.52 0.33
-1-/-
0.37 / 0.49 0.27
DCCM [Wu et al. 2019
0.50 / 0.62 0.41
0.29 /0.33 0.17
0.38 / 0.48 / 0.26
IIC (Ji et
2019
0
/
-/ /-
PICA Avg]
(Huang et al.
2020)
0.56 / 0.65 / 0.47
0.30 0.0/ 0.16
0.59 0.69 / 0.50
PICA [Best]
(Fuang
2020
0.59 / 0.70 00.1
0.31 / 0.34//44/// 0.17
0.61 0.71 0.53
C2BIN [Avg] (Ours)
0.62 / 0.72 0.53
0.36 0.35 0.20
0.62 0.73 0.55
C2BIN [Best] (Ours)
0.63 / 0.73 0.55
0.38 / 0.38 / 0.22
0.64 0.75 0.57
2-step
SCAN
(aassbeke et al. 2020
0.80 / 0.88 / 0.77
0.49 / 0.51 / 0.33 0.70 0.81 / 0.65
Table 11: Full comparison with unsupervised representation models for clustering benchmark datasets.
The results of previous methods taken from Ji et al. (2019): Huang et al. (2020); Gansbeke et al. (2020).
16

---

Under review as a conference paper at ICLR 2021
B.3
QUALITATIVE EXAMPLES IMAGE RETRIEVAL TASKS
Query
Ours
SimCLR
PICA
dog
ho0se
shp
doer
brokk
altpaan morkey
Figure 7: Top-k retrieved images by C2BIN (Columns 2-6), SimCLR (Columns 7-11), and PICA (Columns 12-17) given the query image (Column 1) from the STL-10 test set where is set as 2, 10, 50, 100).
0.9
Average precision
0.8
0.7
Ours
SimCLR
0.6
PICA
1
2
4
8
16
32
64
128
256
Number of retrieved items
Figure 8: Image retrieval performance on STL-10 datasets.
17

---

Under review as a conference paper at ICLR 2021
B.4 VISUALIZATION CLUSTERS
Figure 9: This figure shows a random samples of STL-10 test set images associated to the selected clusters from & 10 cluster-branch. This visualization uses the experiment settings from unsu- pervised clustering experiment in Section 4.3 The border color enclosing each image indicates its ground-truth class.
18

---

Under review as a conference paper at ICLR 2021
Figure 10: This figure shows a random samples STL-10 test set images associated to the selected clusters from k = 40 cluster-branch. This visualization uses the experiment settings identical to Figure The border color enclosing each image indicates its ground-truth class.
19

---

Under review as a conference paper at ICLR 2021
Figure 11: This figure shows random samples of STL-10 test images associated the selected clusters from & = 160 cluster-branch This visualization uses the experiment settings identical to Figure 9 The border color enclosing each image indicates its ground-truth class.
20


Reviews:
[{"review_id": "tlCRe3eNSp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\\\nThis paper applies batch-wise cluster assignment with bootstrapping to learn unsupervised representation. This paper claims resulting representations are better suited to non-discriminative tasks where clustering is important.\n-----\n+Strengths\\\n+This paper motivates a good direction over current unsupervised representation learning. Considering clustering performance in addition to discriminative power is a fair research question in my opinion.\n+The proposed idea is interesting and seems reasonable.\n+Evaluating representation learning on non-discriminative tasks is a good idea.\n-----\n-Concerns\\\n-A critical concern is the experiment setup, particularly the choice of ResNet18 as backbone and only evluating on CIFAR/STL. These datasets are quite small and are not used as the primary performance benchmark for modern unsupervised image representation learning work. This paper claims improvement in certain aspects over SimCLR, MoCo v2, and BYOL which all experiment on much larger capacity models and much larger datasets. Many insights in these prior work tie heavily to scaling to larger dataset and model capacity. This makes it difficult to compare this work.\n-----\nRecommendation\\\nThe experiment setup in this paper deviates significantly from recent work of similar nature. Therefore I am not convinced by the findings presented and I recommend to reject this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UdCemKywXzu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n1- The paper presents a good solution for an important problem in self-supervised learning and contrastive learning. Proposed methods in the literature do not take the cluster structure of items into consideration. This paper proposes a hybrid loss function that aims to preserve the cluster structure (Equation 7)\n\n2- A wide range of experiments are conducted to evaluate the proposed method. C2BIN shows a significantly better performance using the kNN classifier (particularly on CIFAR100). The good performance is also evident in clustering experiments. Thus, as the method promises the cluster structure is better preserved.\n\nCons:\n\n1- The motivation part of the paper is not precise. The last sentence of the second paragraph of Section 1 states:\n\n“However, since aforementioned instance discrimination does not consider the semantic similarities of the representations (e.g., same class), it results the learned representations to be uniformly distributed (Wang & Isola, 2020).” \n\nIt is true that semantic similarities are not considered in self-supervised and contrastive learning settings. However, this is a part of the problem as class labels do not exist. Moreover, it is not clear to me why it would lead to a uniform distribution of representations. I agree that cluster structure might be lost.\n\n2- The notation used in Section 3 is confusing, I mention some possible misuses of notation or typos:\n\nSection 3.1, N is defined as the number of unlabeled images. Later in Section 3.3, second paragraph, N is used as the mini-batch size. \n \nSection 3.2 Equation (1), $P^a (z_a)$ is not properly defined. It refers to SimCLR and as I checked in the paper the same notation is not defined. Moreover, the paper should be written self-contained. Meaning that main formulation should be mentioned in the main body of the paper.\n\nFigure 3, two siamese networks, $E_\\theta$ and $E_\\phi$ are not depicted in the Figure.\n\n\nEquation 4, denominator, $i \\neq k$, should it be $i \\neq j$?\n\nAfter Equation 4, it is stated that: The vectors c’ and c’’ are obtained from x’_i and x’’_ i , respectively, using the encoder $F_\\theta$. This is not precise. The output of F would be r and it goes through $P^{c,k}$ to obtain z. Then c is computed using equation 3. \n\nComments and questions:\n\nTable 3 shows that the choice of set K is quite important. If we are not provided with proper K and we have no access to labels, can you recommend any strategy in this case? Do you think using a fixed K with various elements is good for any dataset?\n\nIt would be nice to sort the methods of table 7 chronologically.\n\nIt is not clear how Figure 9, 10 and 11 are produced. Are the grouped images random samples and nearest neighbors?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2VF6Lo0EA7d", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Overview]\n\nIn this paper, the authors augment the instance-level self-supervised learning with cluster-aware learning mechanism during the training procedure. Specifically, for each training batch, the authors project the instances into a clustering space and then utilize a cluster-aware contrastive loss to push the augmented samples from the same instance to belong to the same cluster, otherwise for different instances. To ensure the clustering not to collapse into a single or a few cluster to find the trivial solutions, the authors further add a penalization item keep the entropy of clustering assignment be uniform to some extent. The experimental results demonstrate that the proposed method can improve the representation learning performance over SoTA methods on several datasets, while also outperforms the previous methods on clustering task. Further ablation studies show that the loss is effective to ensure the learned representation more discriminative and clusterable.\n\n[Strength]\n\n1. The intuition behind the proposed method is intuitive and straightforward. It has been shown in previous work like [a] that combining them together can boost the self-supervised learning performance significantly. This paper further demonstrates the promise of this direction.\n\n[a] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. Caron et al.\n\n2. The authors performed experiments to show that the proposed method achieves better performance on both representation learning task and clustering task, on various image datasets, such as CIFAR-10, CIFAR-100 and STL-10. \n\n3. The ablation studies showed that the proposed clustering loss indeed helps to learn a better representation compared with the baseline model with a substantial margin, which demonstrates the effectiveness of the proposed method.\n\n[Weakness]\n\n1. The paper has a poor literature review of previous works. In the related work, both instance-level representation learning and deep clustering methods are not fully covered and compared. More importantly, the authors missed a very relevant and recent paper as pointed above [a]. The idea behind this paper is very similar to the above one.\n\n2.  In this paper, the authors merely presented the results on relatively small datasets. Though it is a bit harsh to always request experiments on the large-scale dataset, such as ImageNet, proving the efficiency seems necessary especially when it is known that keep training on a large-scale dataset for a long time may dismiss the gap.\n\n3. In table 3, it seems that only with multi-scale clustering loss, the performance will be improved across all metrics. This indicates that the proposed algorithm is a bit sensitive to the hyperparameter settings. Even with Eq.(1) + Eq.(5), the performance drops in some scenarios, which seems counterintuitive. All of these results demonstrate that the proposed method is still a bit mysterious and vulnerable. \n\n4. The notations in the paper is hard to interpret and a bit abuse. The formula of Eq.(4) is also a bit confusing. First, what does k stands for? Second, why the denominator excludes the case of i=k if it is a regular contrastive loss.\n\n[Summary]\n\nOverall, I think this paper is a good trial of combining instance-level contrastive loss and deep clustering philosophy into a single learning regime, which I think is a promising direction to explore. However, as I pointed above, the novelty of the paper should be better explained. Also, according to the ablation study, the performance seems vulnerable to the choice of hyperparameters, such as cluster numbers. This increases the uncertainty about the effectiveness of the proposed method. Furthermore, the proposed method is not demonstrated on large-scale dataset such as ImageNet, which is supposed to be a routine setting on self-supervised learning community. I would recommend the authors could answer my above questions raised above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The idea of combining instance-level contrastive loss and deep clustering is a promising direction in recent unsupervised/self-supervised visual representation learning studies. However, authors did a poor literature review and did not cite and compare with quite a few recent popular work exploring the similar direction. The proposed methodology is not particular novel and the experimental results are also not convincing. Overall, the paper explored a promising research direction, but the paper quality is clearly below acceptance bar. 
Decision: Reject