Prototypical Contrastive Learning of Unsupervised Representations
Authors: Junnan Li
Pan Zhou
Caiming Xiong
Steven Hoi
Venue: ICLR 2021 Poster (2021)
Keywords: self-supervised learning
unsupervised learning
representation learning
contrastive learning

Abstract: This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that bridges contrastive learning with clustering. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL.

Published as a conference paper at ICLR 2021
PROTOTYPICAL CONTRASTIVE LEARNING OF UNSUPERVISED REPRESENTATIONS
Junnan Li, Pan Zhou, Caiming Xiong, Steven C.H. Hoi Salesforce Research (junnan. 11, pzhou, exiong, shoi) @sa 1 lss force.com
ABSTRACT
This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that bridges contrastive learning with clustering. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it encodes semantic structures discovered by clustering into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihodd estimation of the network parameters in an pepectationnMaximizzaeee
framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-ar instance- wise contrastive learning methods multiple benehmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at attps: http::://gtthbbs com www.oo//waa.a.......ooooooo.eeee
INTRODUCTION
Unsupervised visual representation learning aims to learn image representations from pixels them- selves without relying on semantic annotations, and recent advances are largely driven by instance
discrimination tasks (Wu et al. 2018 Ye al., 2019 He et al. 2020; Misra & van der Maaten 2020; [ijelm et al.) 2019 Oord et al. 2018 Tian et al. 2019). These methods usually consist of two key components: image transformation and contrastive loss. Image transformation aims to
generate multiple embeddings that represent the same image, by data augmentation (Ye et al. 2019 Bachman et al., 2019; Chen et al.] 2020a),, patch perturbation (Misra & van der Maaten 2020), or using momentum features (He et al. 2020). The contrastive loss, in the form of a noise contrastive estimator (Gutmann & Hyvärinen (010), aims to bring closer samples from the same instance and separate samples from different instances. Essentially, instance-wise contrastive learning leads to an embedding space where all instances are well-separated, and each instance is locally smooth (i.e. input with perturbations have similar representations).
Despite their improved performance, instance discrimination methods share common weakness: the representation is not encouraged to encode the semantic structure of data. This problem arises because Instance-wise contrastive learning treats two samples as negative pair as long as they are from different instances, regardless of their semantic similarity. This is magnified by the fact that thousands of negative samples are generated to form the contrastive loss, leading to many negative pairs that share similar semantics but are undesirably pushed apart in the embedding space.
In this paper, we propose prototypical contrastive learning (PCL), new framework for unsupervised representation learning that implicitly encodes the semantic structure of data into the embedding space. Figure [ shows an illustration of PCL. A prototype is defined as "a representative embedding for a group of semantically similar instances". We assign several prototypes of different granularity to each instance, and construct contrastive loss which enforces the embedding of a sample to be more similar to its corresponding prototypes compared to other prototypes. In practice, we can find prototypes by performing clustering on the embeddings.
We formulate prototypical contrastive learning as an Expectation- Maximization (EM) algorithm, where the goal is to find the parameters of Deep Neural Network (DNN) that best describes the data
Dnpectatioo-Maxxiizzztiino

---

Published as a conference paper at ICLR 2021
Fine tne-ggrined prototypes (o.g. horse with man]
Coarse-grained prototypes (e.g. horse)
Instance-wise Contrastive Learning
Prototypical Contrastive Learning
Figure 1: Illustration of Prototypical Contrastive Learnnng Each instance is assigned multiple prototypes with different granularity. PCL learns an embedding space which encodes semantic structure of data.
distribution, by iteratively approximating and maximizing the log-likelihood function. Specifically, we introduce prototypes as additional latent variables, and estimate their probability in the E-step by performing k-means clustering. In the M-step, we update the network parameters by minimizing our proposed contrastive loss, namely ProtoNCE. We show that minimizing ProtoNCE is equivalent to maximizing the estimated oogllikelhhood, under the assumption that the data distribution around each prototype is isotropic Gaussian. Under the EM framework, the widely used instance discrimination task can be explained as a special case of prototypical contrastive learning, where the prototype for each instance its augmented feature, and the Gaussian distribution around each prototype has the same fixed variance. The contributions of this paper can be summarized as follows:
We propose prototypical contrastive learning, novel framework for unsupervised representation learning that bridges contrastive learning and clustering. The learned representation is encouraged to capture the hierarchical semantic structure of the dataset.
We give theoretical framework that formulates PCL as an based algorithm. The iterative steps of clustering and representation learning can be interpreted as upproximating and maximizing the log-libe likelihood function. The previous methods based on instance discrimination form special case in the proposed EM framework.
Maximizaaiinnn (EM)
We propose ProtoNCE, new contrastive loss which improves the widely used InfoNCE by dynam- ically estimating the concentration for the feature distribution around each prototype. ProtoNCE also includes an InfoNCE term in which the instance embeddings can interpreted as instance- based prototypes. We provide explanations for PCL from an information theory perspective, by showing that the learned prototypes contain more information about the image classes.
PCL outperforms instance-wist contrastive learning on multiple benchmarks with substantial improvements in low-resource transfer learning. PCL also leads to better clustering results.
RELATED WORK
Our work is closely related to two main branches of studies in unsuperviied//seffsspprviies learning: instance- wise contrastive learning and deep unsupervised clustering.
Instance-wise contrastive learning (Wu et al. 2018 Ye et al. 2019 He et al. 2020 Misra &
van der Maaten 2020 Zhuang et al. 2019 Hjelm et al. 2019; Oord et al.) 2018 Tian et al. 2019
Chen al.] 2020a) aims to learn an embedding space where samples (e.g. crops) from the same instance (e.g. an image) are pulled closer and samples from different instances are pushed apart. To construct the contrastive loss, positive instance features and negative instance features are generated for each sample. Different contrastive learning methods vary in their strategy generate instance features. The memory bank approach (Wu et al. 2018) stores the features of samples calculated in the previous step. The end-to-end approach Ye et al.) 2019 Tian et al.] 2019 Chen al.) 2020a) generates instance features using all samples within the current mini-batch. The momentum encoder approach (He et al., 2020) encodes samples on-the-fly by momentun-updated encoder, and maintains a queue of instance features.
Despite their improved performance, the existing methods based instance-wis contrastive learning have the following two major limitations, which can be addressed by the proposed PCL framework.
2

---

Published as a conference paper at ICLR 2021
The task of instance discrimination could be solved by exploiting low-level image differences, thus the learned embeddings do not necessarily capture high-level semantics. This is supported by the fact that the accuracy of instance classification often rapidly rises to a high level (>90% within 10 epochs) and further training gives limited informative signals. recent study also shows that better performance of instance discrimination could worsen the performance downstream tasks (Tschannen et al. 2020).
A sufficiently large number of negative instances need be sampled, which inevitably yields negative pairs that share similar semantic meaning and should be closer in the embedding space. However, they are undesirably pushed apart by the contrastive loss. Such problem is defined as class collision in (Saunshi et al. (019) and is shown to hurt representation learning. Essentially, instance discrimination learns an embedding space that only preserves the local smoothness around each instance but largely ignores the global semantic structure of the dataset.
Deep unsupervised clustering. Clustering based methods have been proposed for deep unsupervised learning. Xie et al. (2016); Yang et al. (016)): Liao et al. (2016); Yang et al. (017)) Chang et al.
(2017) Di al. [ Gansbeke et al. (2020) jointly learn image embeddings and cluster assignments, but they have not shown the ability to learn transferable representations from a large scale of images. Closer to our work, DeepCluster (Caron et al. 2018) performs iterative clustering and unsupervised representation learning, which is further improved by Zhan et al. (2020) with online clustering. However, our method is conceptually different from DeepCluster. In DeepCluster, the cluster assignments are considered as pseudo-labels and classification objective is optimized, which results two weaknesses: (1) the high-dimensional features from the penultimate layer of CenvNet are not optimal for clustering and need to be PCA-reduced; (2) an additional linear classification layer is frequently re- re-initialized which interferes with representation learning. In our method, representation learning happens directly in wiw-dimensional embedding space, by optimizing contrastive loss on the prototypes (cluster centroids). Concurrent to our work, SwAV (Caron et al. 2020) also brings together clustering objective with contrastive learning.
(
Self-supervised pretext tasks. Another line of self-supervised learning methods focus on training DNNs to solve pretext tasks, which usually involve hiding certain information about the input and training the network to recover those missing information. Examples include image inpainting (Pathak
al.
2016). colorization (Zhang et al., 2016; 2017), prediction of patch orderings (Doersch et al.
2015
Noroozi & Favaro 2016) and image transformations (Dosovitskiy et al.) [014] Gidaris et al.
2018
Caron et al. | 2019) et al. 2019). Compared heuristic pretext task designs, the proposed
PCL
is a more general learning framework with better theoretical justification.
3 PROTOTYPICAL CONTRASTIVE LEARNING
3.1 PRELIMINARIES
Given training set X= (2..,,.... of images, unsupervised visual representation learning aims to learn an embedding function So (realized via a DNN) that maps X to V = {u), U2, Un with 0. (zz), such that v; best describes r,. Instance-wise contrastive learning achieves this objective by optimizing a contrastive loss function, such as InfoNCE (Oord et al. 2018 He et al. 2020), defined as:
exp(1 v//T)
LInfNNC
log E exp(1; u///7)'
where ii and v/ are positive embeddings for instance i, and v/ includes one positive embedding and r negative embeddings for other instances, and r a temperature hyper-parameter. In MoCo (He et al. 2020), these embeddings are obtained by feeding to momentum encoder parametrized by for (zz), where 0' a moving average of
In prototypical contrastive learning, we prototypes instead of v', and replace the fixed tempera- ture 7 with a per-preoe per-poootyyp concentration estimation ø. An overview of our training framework is shown in Figure where clustering and representation learning performed iteratively at each epoch. Next, we will delincate the theoretical framework of PCL based on EM. A pseudo-code of our algorithm is given in appendix
3

---

Published as a conference paper at ICLR 2021
Input
Back Propagation
Encoder
ProtoNCE
M-step
Clustering
Momentum Encoder
E-step
Figure 2: Training framework of Prototypical Contrastive Learning.
3.2 PCL AS EXPECTATION
EXPEETATION-MAXXMPAATOO MAXIMIZATION
Our objective is to find the network parameters 0 that maximizes the log-likelihood function of the observed n samples:
arg max
log p(
(2)
We assume that the observed data {a; / are related to latent variable C' {e. which denotes the prototypes of the data. In this way, we can re-write the log- likelikelihood function as:
6* = max
log p( 0) = arg max
log Σ
p(z,,c,;
(3)
GEC
It is hard to optimize this function directly,
surrogate function lower-bound it:
n
p( c:0)
log pla cz;
log p(ss,,c;0) = Σ log ((c) CEEC
Σ Σ Q(cs)
(4)
((ci)
((ai
1=1
CLEC
1= CEE
where ((c.) denotes some distribution over c's Eg,ec Q(c:) 1), and the last step of derivation
uses Jensen's inequality. To make the inequality hold with equality, we require c( to be constant. Therefore, we have:
Q(e))
p(ai, q;
p(0i,,;;0 p(c;; p(r;;0)
((a)= = Ee
(5)
By ignoring the constant
Σ Egec Q(ci) log Q(c:) in eqn.44. should maximize:
Σ Σ ((c,) (og(l,,.0)
(6)
E1
E-step. In this step, we aim to estimate p(c; xy, 0). To this end, we perform k-means on the features
of for (z;) given by the momentum encoder obtain & clusters. We define prototype o as the
centroid for the i-th cluster. Then, we compute p(c.; 2,, 0) = 1(x; E c.), where ((x, E q.) = 1 if x; belongs the cluster represented by 6;; otherwise 1 (xx E cx) = 0. Similar to MoCo, we found features from the momentum encoder yield more consistent clusters.
M-step. Based on the E-step, we are ready maximize the lower-bound in eqn.
Σ Σ Q(c:)
logg(o;,
Σ
Σ p(c5,, log
i=1 C
1=1 cLEC
1(x E ci) log
&EC
Under the assumption of uniform prior over cluster centroids, we have: p(z,, c,; c):)) = p(z; c, 0)p(c;; k p(as c,. 8),
(8)
4

---

Published as a conference paper at ICLR 2021
where we set the prior probability p(c; 0) for each c₁ 1/k since we provided any samples. We assume that the distribution around each prototype is an isotropic Gaussian, which leads to:
(-(%) ce)2
k
(v cz)² 203 )
p(#; c.. 0) = exp
exp Σ
20²
where v; fz;) and x; E Cg. If we apply l n--oomaalization both and c, then (u-c)² 22-2v.c. we can write maximum log-likelihood estimation as
Combining this with eqn.
8
exp(u: ca/o.)
f" arg min
log
(10)
E exp(u; c//0,)'
where á x o² denotes the concentration level of the feature distribution around prototype and will be introduced later. Note that eqn.(TO) has similar form as the InfoNCE loss in eqn. (.. Therefore, InfoNCE can be interpreted prototype for feature U; is the augmented feature of from the same instance (i.e. e²), and the concentration of the feature distribution around each instance is fixed (i.e. Φ 7).
special case of the maximum log- sggliiklihood estimation, where the
In practice, we take the same approach as NCE and sample r negative prototypes to calculate the normalization term. We also cluster the samples M times with different number of clusters K (km)) m=11 which enjoys a more robust probability estimation of prototypes that encode the hierarchical structure. Furthermore, we add the InfoNCE loss to retain the property of local smoothness and help bootstrap clustering. Our overall objective, namely ProtoNCE, defined as
-1O0 exp(v: e//T) Ei= exp(1: v///T)
M
exp(u: c% /0/
LProtONCE
+
Σ 17
log
(11)
M
2j=0 exp(1; (47)
3.3 CONCENTRATION ESTIMATION
The distribution of embeddings around each prototype has different level of concentration. We use to denote the concentration estimation, where a smaller 6 indicates larger concentration. Here we calculate é using the momentum features [r'] that are within the same cluster as a prototype c. The desired & should be small (high concentration) if (1) the average distance between t' and is small, and (2) the cluster contains more feature points (i.e. large). Therefore, we define é as:
Σ 1/ cli2 Z log (Z + aa)
(12)
where is smooth parameter to ensure that small clusters do not have an overly-large ø. We normalize @ for each set of prototypes C such that they have mean of т.
In the ProtoNCE loss (eqn. ), 6th acts as scaling factor on the similarity between an embedding Us and its prototype eg With the proposed d. the similarity in loose cluster (larger () are down-scaled, pulling embeddings closer to the prototype. On the contrary, embeddings in tight cluster (smaller Φ) have an up-scaled similarity, thus less encouraged to approach the prototype. Therefore, learning with ProtoNCE yields more balanced clusters with similar concentration, as shown in Figure prevents a trivial solution where most embeddings collapse to a single cluster, problem that could only be heuristically addressed by data- resamplngg in DeepCluster (Caron et al. 2018).
3.4 MUTUAL INFORMATION ANALYSIS
has been shown that minimizing InfoNCE maximizing a lower bound on the mutual information (MI) between representations V and V (Oord et al.) 2018). Similarly, minimizing the proposed ProtoNCE can considered as simultaneously maximizing the mutual information between and all the prototypes (V', C² "..,,.,, M... This leads to better representation learning, for two reasons.
First, the encoder would learn the shared information among prototypes, and ignore the individual noise that exists in each prototype. The shared information is more likely to capture levee semantic knowledge. Second, we show that compared to instance features, prototypes have a larger mutual information with the class labels. We estimate the mutual information between the
5

---

Published as a conference paper at ICLR 2021
Histogram
20000
Estimated Concentration Fixed Concentration
MI Class Labels
15000
Prototypes (ProtoNCE) Protolypes (InIoNCE)
5000
4.5
Features (InfoNCE)
30-10 50-60 Cluster Size
70-150
Epoch
(b)
Figure 3: (a) Histogram of cluster size for PCL. (#elusters k=50000) with fixed or estimated concentration. Using a different 6 for each prototype yields more balanced clusters with similar sizes, which Icads to hetter representation learning. (b) Mutual info between instance features (or their assigned prototypes) and class labels of all images in ImageNet. Compared InfoNCE, our ProtoNCE learns better prototypes with semantics.
instance features (or their assigned prototypes) and the ground-truth class labels for all images in ImageNet (Deng et al. 2009) training set, following the method in (Ross 2014). We compare obtained MI of our method (ProtoNCE) and that of MoCo (InfoNCE). As shown in Figure b)) compared to instance features, the prototypes have a larger MI with the class labels due to the effect of clustering. Furthermore, compared to InfoNCE, training ProtoNCE can increase the MI of prototypes as training proceeds, indicating that better representations are learned to form more semantically-meanigguuu clusters.
3.5 PROTOTYPES AS LINEAR CLASSIFIE
Another interpretation of PCL can provide more insights into the nature of the learned prototypes. The optimization in eqn. 10 is similar to optimizing the muster-assignmen probability p(s: z4, 0) using the enosopy loss, where the prototypes e represent weights for a linear classifier. With k-means clustering, the linear classifier has fixed set of weights as the mean vectors for the representations in each cluster, 2/2 1=1 t A similar idea has been used for few-shot learning (Snell et al. 2017 where a non-maramttric prototypical classifier performs better than a parametric linear classifier.
4 EXPERIMENTS
We evaluate PCL on transfer learning tasks, based on the principle that good representations should transfer with limited supervision and fine-tuning. We follow the settings in MoCo, therefore direct comparisons with MoCo could demonstrate the improvement from the prototypical contrastive loss.
4.1 IMPLEMENTATION DETAILS
To enable a fair comparison, we follow the same setting as MoCo. We perform training on the ImageNet- 1M dataset. A ResNet-50 (He et al. 2016) is adopted the encoder, whose last fully- connected layer outputs a 128-D and L2-normalized feature. We follow previous works (He et al.) 2020 Wu al. 2018) and perform data augmentation with random crop, random color jittering, random horizontal flip, and random grayscale conversion. We use SGD as our optimizer, with a weight decay of 0.0001, momentum of 0.9, and a batch size of 256. We train for 200 epochs, where we warm-up the network in the first 20 epochs by only using the InfoNCE loss. The initial learning rate is 0.03, and is multiplied by 0.1 120 and 160 epochs. In terms of the hyper-parameters, we set r = 0.1, = 10, = 16000, and number of clusters K {25000, 50000, 100000). We also experiment with PCL using improvements introduced by Chen et al. (2020a (b). which includes MLP projection layer, stronger data augmentation with additional Gaussian blur, and temperature t 0.2. We adopt faiss (Johnson et al., 2017) for efficient k-means clustering. The clustering is performed per-epoch
center-cropped images. We find over-clustering to be beneficial. Recent advances in self-supervised learning have been propelled by huge compute which inaccessible to many researchers. We instead target more commonly accessible training resource for PCL with NVIDIA- V100 GPUs and approximately 5 days of training.

---

Published as a conference paper at ICLR 2021
VOCO7
Places205
Method
architecture
k=1
k=2
k=4
k=8
k=16
k=1
k=2
k=4
k=8
k=16
Random
8.0
8.2
8.2
8.2
8.5
0.7
0.7
0.7
0.7
0.7
ResNet-50
Supervised
54.3
67.8
73.9
79.6
82.3
14,9
21.0
26.9
32.1
36.0
Jigsaw
26.5
31.1
40.0
46.7
51.8
9.4
12.9
17.4
MoCo
ResNet-50
31.4
42.0
49.5
60.0
65.9
8.8
13.2
18.2
23.2
28.0
PCL (ours)
46.9
56.4
62.8
70.2
74.3
11.3
15.7
19.5
24.1
28.4
SimCLR
32.7
43.1
52.5
61.0
67.1
9.4
14.2
19.3
23.7
28.3
MoCo v2
ResNet-50-MLP
46.3
58.3
64.9
72.5
76.1
10.9
16.3
20.8
26.0
30.1
PCL v2 (ours)
47.9
59.6
66.2
74.5
78.3
12.5
17.5
23.2
28.1
32.3
Table Low-shot image classification on both VOCO7 and Places205 datasets using linear SVMs trained on fixed representations. All methods were pretrained on ImageNet- IM dataset for 200 epochs (except for Jigsaw trained on ImageNet-14N) 14M). We vary the number of labeled examples l and report the mAP (for VOC) and accuracy (for Places) across runs. We use the released pretrained model for MoCo, and re-implement SimCLR.
4.2
IMAGE CLASSIFICATION WITH LIMITED TRAINING DATA
Low-shot classification. We evaluate the learned representation on image classification tasks with few training samples per-category. We follow the setup in Goyal et al. (2019) and train linear SVMs using fixed representations on two datasets: Places205 (Zhou et al. 2014) for scene recognition and PASCAL VOC2007 (Everingham et 2010) for object classification. We vary the number of samples per-class and report the average result across independent runs (standard deviation is reported in appendix C). Table [ [sowws the results, which our method substantially outperforms both MoCo and SimCLR.
Semi-supervised image classification. We perform semi-supervised learning experiments to evalu- ate whether the learned representation can provide good basis for fine-tuning, Following the setup from Wu et al. (2018); Misra van der Maaten (2020), we randomly select subset (1% or 10%) of ImageNet training data (with labels), and fine-tune the self-supervised trained model these subsets. T reports the top-5 accuracy on ImageNet validation set. Our method sets a new state-of-the-art under 200 training epochs, outperforming both self supervised learning methods and supe-supe learning methods. The standard deviation across runs is low (< 0.6 for 1% labels).
#pretrain epochs
Top-5 Accuracy
Method
architecture
1%
10%
Random (Wu et al. 2018)
ResNet-50 ResNet-50
22.0
59.0
Supervised bascline Zhai et al 2019
48.4
80.4
Semi-supervised learning methods:
Pscudolabels (Zhai et al. 2019)
ResNet-50v2
51.6
82.4
VAT + Entropy Min. (Miyalo el al.
2019
ResNet-50v2
47.0
83.4
S Rotation (Zhai al. 2019)
ResNet-50v2
53.4
83.8
Self-supervisel learning methods:
Instance Discrimination (Wu et al.
2018
ResNet-50
200
39.2
77.4
Jigsaw (Noroozi & Favaro 2016
ResNet-50
90
45.3
79.3
AmnCLR(Chere et al. (2020a
ResNet-50-MLP
200
56.5
82.7
MoCo (He et al. 2020)
ResNet-50
200
56.9
83.0
MoCa v2 (Chen et al. 2020h
ResNet-50-MI.P
200
66.3
84.4
PCT. v2 (ours)
ResNet-50-MIL.P
200
73.9
85.0
PCI. (ours)
ResNet-50
200
75.3
85.6
PTRL. (Misra van der Maaten 2020
ResNet-50
800
57.2
83.8
SimCLR Chen et al. (2020a)
ResNet-50-MLP
1000
75.5
87.8
BYOL
Grill et al. 2020)
ResNet-50-MLL.
1000
78.4
89.01
SwAV Caron et
2020
ResNet-50-ML.P
800
78.5
89.9*
Table 2: Semi-supervised learning on ImageNet. We report top-5 accuracy the ImageNet validation set of self-supervised models that finetuned 1% 10% of labeled data. f: SimCLR, BYOL, and SWAV large batch size 4096. F; SwAV uses multi-crop augmentation.
7

---

Published as a conference paper at ICLR 2021
4.3 IMAGE CLASSIFICATION BENCHMARKS
Linear classifiers. Next, train linear classifiers fixed image representations using the entire labeled training data. We evaluate the performance of such linear classifiers on three datasets: ImageNet, VOC07, and Places205, Table 3 reports the results. PCL outperforms MoCo under direct comparison, which demonstrate the advantage of the proposed prototypical contrastive loss.
architecture (#params)
#pretrain epochs
Dalaset VOCO7
Method
ImageNet
Places205
Jigsaw (Norozii & Favaro.
2016
R:0 (24M)
90
45.7
64.5
41.2
Rotation Gidans el al. 2018
R50 (24M)
48.9
63.9
DeepCluster (Caron ct al. 2018
VGG(15M)
100
71.9
37.9
BigBiGAN (Donahue & Simonyan
2019
R50 (24M)
56.6
InstDise WWi et al. 2018
R50 (24M)
200
54.0
45.5
MoCo (He al. 2020
R50 (24M)
200
60.6
79.2"
48.9"
PCL (ours)
R50 (24M)
200
61.5
82.3
49.2
SimCLR (Chen et al.
2020a
R50-MLP (28M)
61.9
MoCo v2 (Chen et 2020b
R50-MLP (28M)
84.0"
PCI. v2 (ours)
R50-MLP (28M)
67.6
85.4
LocalAgg (Zhuang et al.
2019
R50 (24M)
200
60.2
50.1
Selfl.abel (Asano
2020)
R50 (24M)
400
61.5
CPC OOrd et al. 2018
R101 (28M)
CMC
al.. 2019
R50 (47M)
280
64.0
PIRL MMisra & van der Manten
2020
R50 (24M)
800
63.6
81.1
49.8
AMDIM (Bachman et al. 2019
Custom (626M)
150
68.11
55.0
SimCIR (Chen et al. 2020a
R50-MLP (28M)
1000
69.31
80.51
BYOL Grill et al. 2020
R50-MLP (355m (35M)
1000
74.3
SwAV
Caron et al. 2020
R50-MLP (28M)
75.3
88.9
56.71
Table 3:
Image classification with linear models. We report top-1 accuracy. Numbers with are from released pretrained model; all other numbers are adopted from corresponding papers...
: LocalAgg uses 10-cr 10-crop evaluation. ADMIM uses FastAutoAugment (Lim et al. 2019) that is supervised by ImageNet labels. SwAV uses multi-crop augmentation. SimCLR, BYO,, and SwAV use large batch size of 4096.
KNN classifiers. We perform k-nearest neighbor (kNN) classification ImageNet. For query image with feature v. we take its top & nearest neighbors from the momentum features, and perform weighted-cone comtbniatooiiiniionnnnn of their labels where the weights are calculated by exp(v v//T). Table reports the accuracy. Our method substantially outperforms previous methods.
Method
Inst. Disc, [W al. 2018)
MoCo (Hc al. 2020
LA (Zhuang al. 2019)
PCL (ours)
Accuracy 46.5
47.1
49.4
54.5
Table Image classification with kNN classifiers using ResNet-50 features on ImageNet.
4.4 CLUSTERING EVALUATION
evaluate the k-means clustering performance ImageNet using representations learned by different methods. PCL leads to substantially higher adjusted mutual information (AMI) score. Details are given in appendix
In Table 5
Method
DeepCluster (Caron et al. 2018
MoCo (He et al. 2020
PCT. (ours)
AMI
0.281
0.285
0.410
Table 5: AMI score for k-means clustering (k 25000) on ImageNet representation.
4.5 OBJECT DETECTION
We assess the representation object detection. Following Goyal et al. (2019), train Faster R-CNN (Ren et al. 2013) on VOCO7 or VOC07+12, and evaluate on the test set VOCO7. We keep the pretrained backbone frozen better evaluate the learned representation, and use the same
8

---

Published as a conference paper at ICLR 2021
schedule for all methods. Table reports the average mAP across three runs. Our method substantially closes the gap between self-supervised methods and supervised training. In appendix [D] we show the results for fine-tuning the pretrained model for object detection and instance segmentation COCO Lin et al. 2014), where PCL. outperforms both MoCo and supervised training.
Method
Pretrain Dataset
Architecture
Training data VOCO7
VOCU7+12
Supervised
ImageNet- IM ImageNet- 1M ImageNet-IM IM
Resnet-50-FPN
72.8
79.3
MoCo (He et al. 2020)
Resnet-50-FPN
66.4
73.5
PCT. (ours)
Resnet-50-FPN
71.7
78.5
Table 6: Object detection for frozen conv body VOC using Faster R-CNN.
VISUALIZATION OF LEARNED REPRESENTATION
In Figure 4 we visualize the unsupervised learned representation of ImageNet training images using t-SNE [Maaten Hinton 2008). Compared the representation learned by MoCo, the representation learned by the proposed PCI. forms more separated clusters, which also suggests representation of lower entropy.
MoCo Representation: Class 1.20
Representation: 1-20
MoCo Representation: 21-40
PCL Regresentation: Class 21-40
Figure 4: T-SNE visualization of the unsupervised learned representation for ImageNet training images from the first 40 classes. Left: MoCo; Right: PCL (ours). Colors represent classes.
CONCLUSION
This paper proposes Prototypical Contrastive Learning, a generic unsupervised representation learning framework that finds network parameters to maximize the log-likelihood of the observed data. We introduce prototypes as latent variables, and perform iterative clustering and representation learning in an EM-bdsed framework. PCL leams embedding space which encodes the semantic structure of data, by training on the proposed ProtoNCE loss. Our extensive experiments on multiple benchmarks demonstrate the advantage of PCL. for unsupervised representation learning.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. ICLR, 2020.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint orXiv:/906.0000 (ee.//906000900 00910, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, pp. 139-156, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In ICCV, pp. 2959-2968, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng. Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In ICCV, pp. 5880-5888, 2017.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003304299, 2020b.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248 244--55,2220 2009.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, pp. 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurlPS, pp. 10541-10551, 2019.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. In NIPS, pp. 766-774, 2014.
Mark Everingham, Lue Van Gool, Christopher Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88 (2):303-338. 2010.
Rong-En En Fan, Kai-Wei Chang, Cho-Jui Hsich, Xiang-Rou Rui Wang. and Chih-Jen Lin. LIBLINEAR: A library for large linear classification. JMLR, 9:1871-1874, 2008
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. SCAN: learning to classify images without labels. In ECCV, pp. 268-285, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron. 2018
https://githhl htt.://hw..c.caccccoooeeeeeeeeee
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. In ICCV, pp. 6391-6400, 2019.
10

---

Published as a conference paper at ICLR 2021
Jean-Bastir Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurlPS, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, pp. 297-304, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, pp. 2980-2988, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie -arcii--rrrhiloon. Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, pp. 9865-9874, 2019.
Jeff Johnson. Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint wwXiv:170.00772 2017.
Renjie Liao, Alexander Schwing, Richard Zemel, and Raquel Urtasun. Learning deep parsimo- nious representations. In NIPS, pp. 5076-00660000 20864, 2016.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NeurlPS, pp. 6662-6672, 2019.
Tsung- Yi Lin, Michael Maire. Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. ECCV, pp. 740-755, 2014.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9:2579-2605, 2008.
Ishan Misra and I Laurens van der Maaten. Self-supervised learning of prerexx-invariiatteeeeeseeeeet In CVPR, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 11(8):1979-1993, 2019.
Xuan Vinh Nguyen, Julien Epps. and James Bailey. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. J. Mach. Learn, Res., 11:2837-2854, 2010.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pp. 69-84, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.0374 2018
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In CVPR, pp. 2536-2544, 2016.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R- R-CNN: towards real-time object detection with region proposal networks. In NIPS, pp. 91-99, 2015.
Brian C Ross. Mutual information between discrete and continuous data sets. PloS one, 9(2). 2014.

---

Published as a conference paper at ICLR 2021
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. ICML, pp. 5628-5637, 2019.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NIPS, pp. 4077-4087, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 05849, 2019.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In ICLR, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pp. 3733-3742, 2018.
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. ICML, pp. 478 2016.
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards mmans-ffiendly spaces: Simultaneous deep learning and clustering. In ICML, pp. 3861-3870, 2017.
Jianwei Yang. Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and
image clusters. In CVPR, pp. 5147-5156, 2016.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, pp. 6210-6219, 2019.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In ICCV, pp. 1476-1485, 2019.
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In CVPR, pp. 6687-6696, 2020.
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. AET vs. AED: unsupervised representation learning by auto-encoding transformations rather than data. CVPR, 2019.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, pp- 649-666, 2016.
Richard Zhang, Phillip Isola, and Alexei Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. CVPR, pp. 1058-1067, 2017.
Bolei Zhou, Ágata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In NIPS, pp. 487-495, 2014.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In ICCV, pp. 6002-6012, 2019.
12

---

Published as a conference paper at ICLR 2021
APPENDIX A ABLATION ON PROTONCE
The proposed loss in eqn. contains terms: the instance-wise contrastive loss and the proposed prototypical contrastive loss. Here we study the effect each term on representation learning. Table 7 reports the results for low-resource fine-tuning and linear classification on ImageNet. The prototypical term plays an important role, especially in the fine-tuning experiment. The warm-up also improves the result by bootstrapping the clustering with better representations.
Method
I % fine-tuning (top-5 acc.)
linear classification (top-1 acc.)
instance only
56.9
60.6
proto only (w/o warm-up)
60.7
60.4
proto only (w/ warm-up)
72.3
60.9
instance proto (w/o warm-up)
74.6
61.3
instance + proto (w/ warm-up)
75.3
61.5
Table 7: Effect of instance-wise contrastive loss and prototypical contrastive loss.
APPENDIX B PSEUDO-CODE FOR PROTOTYPICAL CONTRASTIVE LEARNING
Algorithm 1: Prototypical Contrastive Learning.
Input: encoder fe. training dataset X, number of clusters K = (lim }
// initialize momentum encoder as the encoder
while not MaxEpoch do
/* E-step */
V' = for (X) for m = 1 to M do
// get momentum features for all training data // cluster V into kn clusters, return
C = k-means(V" km)
prototypes
6m Connentration V') // estimato the distribution concontration around each prototype with Equaticn 12 end /a M-step +/
for in Dataloader(X) do fo(x), v' (x) oncoder
// load minibatch 1/ forward pass through encoder and momentum
10
11
CrooeNCE {Cmm mm {(m) MM m
/1 calculate loss with Equation // update encoder paramoters // update momentum encoder
12
0 SGD(CPrstoNCE, 8)
0' 0.999 0' 0.001
14
end
end
APPENDIX C STANDARD DEVIATION FOR LOW-SHOT CLASSIFICATION
Table
we report the standard deviation for the low-shot classification experiments in Table
VOC07
Places205
Method
k=
k=2
k=4
k=8
k=16
k=
k=2
k=4
k=8
k=16
PCL
4.06
2.65
2.21
0.49
0.39
0.24
0.23
0.13
0.07
0.05
PCL v2
4.12
2.70
2.17
0.54
0.38
0.26
0.23
0.12
0.08
0.04
Table 8: Standard deviation across runs for low-shot image classification experiments.
13

---

Published as a conference paper at ICLR 2021
APPENDIX D COCO OBJECT DETECTION AND SEGMENTATION
Following the experiment setting in (He et al. backbone. We finetune all layers end-to-end on the COCO train2017 set and evaluate on va12017. The schedule is the default 2x in (Girshick et al. 2018). PCL outperforms both MoCo (He et al. 2020) and supervised pre-tree pre-training in all metrics.
2020). we use Mask R-CNN (He et al. 2017) with C4
Method
Aphb
AP 50
AP 75
Apmk
AP 50
AP
Supervised
40.0
59,9
43.1
34.7
56.5
36.9
MoCo (He al. 2020)
40.7
60.5
44.1
35.4
57.3
37.6
PCL (ours)
41.0
60.8
44.2
35.6
57.4
37.8
Table 9: Object detection and instance segmentation fine-tuned on COCO. We evaluate bounding-box AP (AP*)) and mask AP (APm)) on val2017.
APPENDIX E TRAINING DETAILS FOR TRANSFER LEARNING EXPERIMENTS
For training linear SVMs on Places and VOC, we follow the procedure in (Goyal et al., 2019) and use the LIBLINEAR (Fan et al. 2008) package. We preprocess all images by resizing to 256 pixels along the shorter side and taking 224 224 center crop. The linear SVMs are trained on the global average pooling features of ResNet-50.
For image classification with linear models, we use the pretrained representations from the global average pooling features (2048-D) for ImageNet and VOC, and the conv5 features (averaged pooled to ~99000-) for Places. We train a linear SVM for VOC, and a logistic regression classifier (a fully-comnected layer followed by softmax) for ImageNet and Places. The logistic regression classifier trained using SGD with a momentum of 0.9. For ImageNet, we train for 100 epochs with an initial learning rate of 10 and weight decay of O. Similar For Places, we train for 40 epochs with an initial learning rate of 0.3 and a weight decay of O.
used by (He et al. 2020).
For semi-supervised learning, we finetune ResNet-50 with pretrained weights on subset of ImageNet with labels. We optimize the model with SGD, using a batch size of 256, momentum of 0.9, and weight decay of 0.0005. We apply different learning rate to the ConvNet and the linear classifier. The learning rate for the ConvNet is 0.01. and the learning rate for the classifier is 0.1 (for 10% labels) or 1 (for 1% labels). We train for 20 epochs, and drop the learning rate by 0.2 12 and 16 epochs.
For object detection on VOC, We use the R50-FPN backbone for the Faster R-CNN detector available in the MMdetect on (Chen al., 2019) codebase. We freeze all the conv layers and also fix the BatchNorm parameters. The model is optimized with SGD, using batch size of 8, momentum of 0.9, and a weight decay 0.0001. The initial learning rate is set as 0.05, We finetune the models for 15 epochs, and drop the learning rate by 0.1 at 12 epochs.
APPENDIX F EVALUATION OF CLUSTERING
In order to evaluate the quality of the clusters produced by PCL., we compute the adjusted mutual information score (AMI) (Nguyen et al., (2000) between the clusterings and the ground-truth labels for ImageNet training data. AMI is adjusted for chance which accounts for the bias in MI to give high values to clusterings with a larger number of clusters. AMI has value of I when two partitions are identical, and an expected value of for random (independent) partitions. In Figure 5 we show the AMI scores for three clusterings obtained by PCL. with number of clusters K 25000, 50000, 100000}. In Table 5
we show that compared to DeepCluster (Caron et al. 2018 and MoCo (He et al. 2020), PCL produces clusters of substantially higher quality.
14

---

Published as a conference paper at ICLR 2021
0.5
0.4
0.3
AMI
K=25000
0.1
K=50000
K=100000
0.0
200
Epoch
Figure 5: Adjusted mutual information score between the clusterings generated by PCL the ground-truth labels for ImageNet training data.
APPENDIX G CONVERGENCE PROOF
Here we provide the proof that the proposed PCL would converge. Suppose let
pla;, G;
F(0) log p( 0)
log Σ p(a;,c;;))
log Q(es)
Q(c)
cEC
4EC
(13)
n
Σ ((c:) log p(1;;
Σ
Q(q)
i=1
CLEC
We have shown in Section 3.2 that the above inequality holds with equality when Q(c:) = p(c;;2; 0).
At the /-th E-step, we have estimated Q'(c)) = p(e; 0²). Therefore we have:
Σ (c:c) log Mar,e;;0% ('(c)
F(0")
Σ
(14)
i=1 4EC
At the 1-th M-step, we fix ('(c;) = p(cc( xz, 0²) and train parameter 0 to maximize Equation Therefore always have:
log p(o,,c00
Σ (cic log p(a,,c;0)) F(0"). (15)
F(0++1)
(1(c.)
Q*(c.)
i= CEC
1=1
cEC
The above result suggests that F(0*) monotonously increase along with more iterations. Hence the algorithm will converge.
APPENDIX H VISUALIZATION OF CLUSTERS
In Figure 6 we show ImageNet training images that are randomly chosen from clusters generated by the proposed PCL. PCL not only clusters images from the same class together, but also finds fine-grained patterns that distinguish sub-classes, demonstrating its capability to learn useful semantic representations.
15

---

Published as a conference paper at ICLR 2021
Figure 6: Visualization of randomly chosen clusters generated by PCL. Green boarder marks top-5 images that closest tineggrined prototypes (K 100k). Orange boarder marks images randomly chosen from coarse- coarse-grained clusters (K = 50k) that also cover the same green images. PCI. can discover hierarchical semantic structures within the data (e.g. images with horse and man form fine-grained cluster within the coarse-grained horse cluster.)
16


Reviews:
[{"review_id": "iE6NtOKSXc6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper mainly injects clustering into the instance discrimination work, MoCo, for unsupervised representation learning. The main idea is to maintain a list of cluster centers (i.e., prototypes), and use them to estimate the similarity between a key and query. Because the cluster centers also need to be learned, the proposed method is an iterative, EM-like algorithm where the clusters are produced every now and then.\n\nFrom the arXiv upload time, this is an improved version of the manuscript submitted for NeurIPS 2020. Back then I was pretty interested in this work and was able to reproduce some of the improvements (e.g., some minor improvements on ImageNet linear evaluation) observed in the paper using my own implementation. So this paper did a pretty good exploration and I would trust the results in the paper. However, the main concern is on significance now, as there are concurrent submissions like BYOL (which claims to implicitly perform bootstrapping representation learning), or more closely SwAV (an online-clustering algorithm that also aims at unsupervised learning). Both BYOL and SwAV are accepted at NeurIPS already with better performance on several evaluation benchmarks (e.g. semi-supervised setting, BYOL reports 78.4 top-5 accuracy with 1% labels), yet unfortunately this work does not. Nevertheless, I would still give an acceptance rating as:\n\n1) This is solid work, I have personally implemented this work and the observation is quite consistent with what's described in the paper, at least for some major experiments.\n2) Doing an EM-style clustering with momentum encoder/key is indeed something that has never been explored to my knowledge, and there is some novelty to it.\n3) It also proposes a prototype-specific T, which is quite new and well-motivated.\n\nFor writing and organization: I think the paper is written quite clearly (as I was able to write code based on this), and since this is already an improved version for the NeurIPS submission, I believe the clarity/organization has improved more.\n\nSuggestions/questions:\n1) Please include more recent numbers from other papers like BYOL or SwAV, for completeness of the assessment of the value of the work.\n2) Does more recent method (BYOL/SwAV) also have \"clustering\" effect? Like the mutual information with class labels on ImageNet?\n3) How  much does the \"cluster-varying T\" help performance? Can a single, global T used for protoNCE?  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "WOa60tysaGF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Overview]\nIn this paper, the authors tackle the problem of unsupervised visual representation learning by combining the deep clustering technique and momentum contrastive learning. Contrastive learning is widely used in instance-level SSL. However, considering it cannot model the inter-sample structures, the authors proposed a new method called prototypical contrastive learning (PCL), which boosts the original contrastive learning with clustering technique. It can be interpreted as an EM procedure. Though the high-level idea is similar to deep clustering and its variants, the authors argued that PCL can address the issue of reinitialization and potential cluster collapse, as demonstrated in the formulas. In the experiments, the authors performed extensive experiments to demonstrate the efficiency of PCL.\n\n[Strength]\n1. The authors presented a well-written paper with some theoretical explanations and extensive experiments. It is easy to follow with some good insights in the words.\n2. The proposed PCL is formulated as an EM procedure. Though see it as an EM algorithm at a high-level is not novel, this paper has a relatively thorough derivation and give a good intuition behind the proposed PCL method. \n3. The authors demonstrated the effectiveness of the proposed PCL on various datasets and settings. Compared with MoCo, its main counterpart, PCL achieve substantial improvement over it across the board. \n\n[Concerns]\n1. Introducing a EM-like procedure to mitigate the reinitialization issue has been proposed in previous works. Though not on large-scale dataset, JULE (Yang et al 2016) proposed to use a forward and backward pass for alternative learning, which corresponds to the E-step and M-step in this paper, respectively. To date, there are also some papers that proposed a similar strategy to address the issue of oscillation, such as \"Online Deep Clustering for Unsupervised Representation Learning. 2020\".  \n2. The proposed PCL combined momentum contrast and deep cluster technique, as formulated in Eq. (11). This formula prompts the interaction of intra-instance learning and inter-instance learning. From the experiments, it is not clear how they cooperate with each other during the training. Does the PCL (2nd term in Eq. (11)) solely perform poorly in representation learning?\n\n[Summary]\nOverall, I think this paper presents a clean formula to reconcile the intra-instance and inter-instance unsupervised learning methods, which seems to be a promising direction for further exploration. Based on my above comments, I think this is a good paper, and recommend acceptance. I would suggest the authors can address my above concerns and provide more insights about how these two losses interact with each other during the learning process in the rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PYhm7PJT1aF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nIn this paper, the authors propose a method for contrastive learning inspired on current instance discrimination approaches. The propose method, called Prototypical Contrastive Learning, can be seen as an augmentation of MoCo where, besides discriminating each instance, they also introduce prototypes as latent variables. A EM-like algorithm is introduce to train the model with latent variables.\nThe model is trained by considering both instance discrimination (as in MoCo) and prototypical discrimination. The authors provide results in different setting (low-shot image classification, semi-supervised learning and image classification with fixed features).\n\n### Strengths\n+ The paper is nicely written and provides a nice read.\n+ The paper is well motivated and points to valid issue with current issues with (unsupervised) contrastive learning approaches. The idea of providing latent 'prototype' variables to capture the semantic structure of data.\n+ The model is built on top of MoCo and the experiments show constant improvement over MoCo on different range of experiments.\n\n### Weakness\n- My main issue with this paper is the lack of comparison with current similar model, SwAV (Caron et al., NeurIPS20). Although quickly mentioned on the related works section, the authors do compare with them on experimental section. \n- I feel that some ablation studies to better understand the components of the method is missing (the number of clusters, the importance of the concentration estimation and the choices of its parameters, etc.). The qualitative visualization from the appendix are very nice and it is unfortunate that they cannot go on main manuscript.\n\n### Comments\nAlthough I generally like the paper, i dont understand why the authors did not include most recent methods in the experimental results (eg. BYOL, SwAV, etc). In particular, no mention or comparison with SwAV---a method that is similar to the proposed approach---appears in experimental section. \nI don't particularly mind the fact that SwAV outperform the proposed approach (the fact the method is built on the top of MoCo AND outperforms it is not bad). However, I dont think the method should be omitted from the experimental results. I would be much more happy to see more comparison (similarities/differences) between the two approaches.\n\nBecause of this, my current rate is \"Marginally below acceptance threshold\"\n\n### Post-rebuttal updates\nGiven the answer of the authors. I will raise my reviews to 7 assuming the authors will add more about comapring with current sota methods, as discussed on this review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7hl4PIIA8L", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposes a new self-supervised representation learning loss which is a combination of InfoNCE and EM-clustering. Rather than encouraging similarity to another augmentation of the same image, it clusters images and encourages similarity of images to their cluster centers. The evaluated approach ProtoNCE actually consists of a sum of the standard InfoNCE loss and the proposed novel loss. The approach is evaluated on the most common ImageNet linear classification benchmark and a few others: ImageNet semi-supervised, VOC07, Places 205, including some detection tasks. The paper compares to reasonable baselines like MoCo-v2 and SimCLR.\n\n\n##########################################################################\n\nReasons for score:\n\nThe proposed approach seems relatively complicated to implement compared to baseline MoCo-v2 while not being sufficiently better on the most established ImageNet linear eval benchmark. MoCo-v2 produces 67.5±0.1 (five runs from github repo: {67.7, 67.6, 67.4, 67.6, 67.3}, see https://github.com/facebookresearch/moco) while the proposed method gets 67.6 - not a very persuasive advantage. Moreover, the approach essentially compares with with 200-epoch ablation of MoCo-v2 without comparing to the main 800-epoch run. I couldn't find other numbers in MoCo-v2 paper https://arxiv.org/pdf/2003.04297.pdf surpassed by the numbers in this paper - although both papers evaluate on detection as well. \n\n\n##########################################################################\n\nPros:\n\n1. Interesting idea to compare images to EM cluster centers.\n2. Reasonable amount of experiments.\n\n##########################################################################\n\nCons:\n\n1. Results are within variance of MoCo-v2 number on ImageNet linear eval. Other numbers from MoCo-v2 paper are not directly compared against.\n2. It does not follow from the title/abstract/introduction that the evaluated approach is actually a sum of the standard InfoNCE and the novel loss. This is a clarity issue which needs to be addressed.\n3. The paper seems to make an implicit assumption about the computational budget under which it tries to obtain the best possible results. For example, it doesn't try to beat 1000-epoch-big-batch results of SimCLR. I don't have an issue with this assumption, but it needs to be directly articulated early on. Something like \"recent advances in self-supervised learning were propelled by huge compute, which is not available to the majority of labs; we instead target more commonly reproducible/achievable training regimes - X GPUs, Y days\". For example, MoCo-v2 paper makes these compute considerations clear upfront, as early as abstract - which is good for the reader.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above.\n\n#########################################################################\n\nMinor suggestions and typos: \n\n(1) c_s is not explained anywhere and it's a quite confusing notation for the center of the cluster which x_i belongs to\n\n(2) the fact that the paper uses data augmentations is not mentioned in the main text at all\n\n(3) \"where v'i is a positive embedding for instance i,\" - should be without '?\n\n(4) \"PCL not only learns low-level features for the task of instance discrimination, but\nmore importantly, it implicitly encodes semantic structures of the data into the\nlearned embedding space.\" - it is quite confusing to talk about semantic structures here, without clarifying what is meant by that. The clusters might have good mutual information with respect to unseen labels, but there is still no way to know that dog is a dog until an example of a dog is seen.\n\n(5) it might be nice (although not required by ICLR rules) to mention BYOL https://arxiv.org/abs/2006.07733 in something like Concurrent Work section", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension to previous unsupervised feature learning work, with an EM-style latent variable model with momentum encoders. The paper is well-written and provides a nice read. It has been noted that it is easy to follow and provides good insights. On the experimental side, compared with MoCo, the proposed approach achieve noticeable improvements. One of the reviewers noted the easy reproducibility of the proposed approach.

Some reviewers noted some comparisons were lacking from the original manuscript, but the authors have update the draft to include those. As noted in the reviews, the field of SSL in vision is moving at a very quick pace, making it hard to clearly state what is the SOTA at time t.

Overall, most questions raised by the reviewers were properly addressed during rebuttal - and given the ratings, I suggest acceptance.
Decision: Accept (Poster)