CO2: Consistent Contrast for Unsupervised Visual Representation Learning
Authors: Chen Wei
Huiyu Wang
Wei Shen
Alan Yuille
Venue: ICLR 2021 Poster (2021)
Keywords: unsupervised representation learning
contrastive learning
consistency regularization

Abstract: Contrastive learning has recently been a core for unsupervised visual representation learning. Without human annotation, the common practice is to perform an instance discrimination task: Given a query image crop, label crops from the same image as positives, and crops from other randomly sampled images as negatives. An important limitation of this label assignment is that it can not reflect the heterogeneous similarity of the query crop to crops from other images, but regarding them as equally negative. To address this issue, inspired by consistency regularization in semi-supervised learning, we propose Consistent Contrast (CO2), which introduces a consistency term into unsupervised contrastive learning framework. The consistency term takes the similarity of the query crop to crops from other images as unlabeled, and the corresponding similarity of a positive crop as a pseudo label. It then encourages consistency between these two similarities. Empirically, CO2 improves Momentum Contrast (MoCo) by 2.9% top-1 accuracy on ImageNet linear protocol, 3.8% and 1.1% top-5 accuracy on 1% and 10% labeled semi-supervised settings. It also transfers to image classification, object detection, and semantic segmentation on PASCAL VOC. This shows that CO2 learns better visual representations for downstream tasks.

Published as a conference paper at ICLR 2021
CO2: CONSISTENT CONTRAST FOR UNSUPERVISED VISUAL REPRESENTATION LEARNING
Chen Wei', Huiyu Wang', Wei Shen Alan Yuille'
Johns Hopkins University SSanghai Jiao Tong University
ABSTRACT
Contrastive learning has been adopted as a core method for unsupervised visual representation learning. Without human annotation, the common practice is perform an instance discrimination task: Given a query image crop, this task labels crops from the same image as positives, and crops from other randomly sampled images strategy is that it can not reflect the heterogeneous similarity between the query crop and each crop from other images, taking them equally negative, while some of them may even belong to the same semantic class as the query. To ad- dress this issue, inspired by consistency regularization in semi-supervised learn- ing on unlabeled data, we propose Consistent Contrast (CO2), which introduces consistency regularization term into the current contrastive learning framework. Regarding the similarity of the query crop to each crop from other images as "un- labeled", the consistency term takes the corresponding similarity of a positive crop as a pseudo label, and encourages consistency between these two similarities. Em- pirically, CO2 improves Momentum Contrast (MoCo) by 2.9% top-1 accuracy on ImageNet linear protocol, 3.8% and 1.1 % top-5 accuracy on 1 % and 10% labeled semi-supervised settings. It also transfers to image classification, object detection, and semantic segmentation on PASCAL VOC. This shows that CO2 learns better visual representations for these downstream tasks.
negatives. An important limitation of this label assignment
INTRODUCTION
Unsupervised visual representation learning has attracted increasing research interests for it unlocks the potential of large-scale pre-training for vision models without human annotation. Most of recent works learn representations through one or more pretext tasks, in which labels are automatically generated from image data itself. Several early methods propose pretext tasks that explore the in- herent structures within a single image. For example, by identifying spatial arrangement (Doersch) et al. 2015), orientation (Gidaris al. 2018). or chromatic channels (Zhang et al., 2016), models learn useful representations for downstream tasks. Recently, another line of works (Wu et al. 2018 Bachman | 2019 Hjelm et al., 2018: Tian al., 2019 He al. [2200] Misra & van der Maaten 2020] Chen et al. 2020a), (.)) e.g. Momentum Contrast (MoCo), falls within the framework of con- trastive learning (Hadsell et al., (006), which directly learns relations of images the pretext task. In practice, contrastive learning methods show better generalization in downstream tasks.
Although designed differently, most contrastive learning methods perform an instance discrimina- tion task, i.e., contrasting between image instances. Specifically, given query crop from one image, a positive sample an image crop from the same image; negative samples are crops randomly sam- pled from other images in the training set. Thus, the label for instance discrimination is a one-hot encoding over the positive and negative samples. This objective is to bring together crops from the same image and keep away crops from different images in the feature space, forming an instance discrimination task.
However, the one-hot label used by instance discrimination might be problematic, since it takes all the crops from other images as equally negative, which cannot reflect the heterogencous similarities between the query crop and each of them. For example, some "negative" samples semantically similar to the query, or even belong to the same semantic class as the query. This is referred to as
corresponndig author

---

Published as a conference paper at ICLR 2021
"class collision" in Saunshi al. (2019) and "sampling bias" in Chuang et al. (2020). The ignorance of the heterogeneous similarities between the query crop and the crops from other images can thus raise an obstacle for contrastive methods to learn good representation. A recent work, supervised contrastive learning (Khosla et al,, 2020). fixes this problem by using human annotated class labels and achieves strong classification performance. However, in unsupervised representation learning, the human annotated class labels are unavailable, and thus it is more challenging to capture the similarities between crops.
In this paper, we propose to view this instance discrimination task from the perspective of semi- supervised learning. The positive crop should be similar to the query for sure since they are from the same image, and thus can be viewed as labeled. On the contrary, the similarity between the query and each crop from other images is unknown, or unlabeled. With the viewpoint of semi-supervised learning, we introduce Consistent Contrast (CO2), consistency regularization method which fits into current contrastive learning framework. Consistency regularization (Sajjadi et al. (2166) is at the core of many oftte-of-thh-arr semi-supervised learning algorithms (Xie et al. 2019 Berthelot al.) 2019b Sohn et al. 2020). It generates pseudo labels for unlabeled data by relying on the assump- tion that good model should output similar predictions on perturbed versions of the same image. Similarly, in unsupervised contrastive learning, since the query crop and the positive crop naturally form two perturbed versions of the same image, we encourage them to have consistent similarities to each crop from other images. Specifically, the similarity of the positive sample predicted by the model is taken as a pseudo label for that of the query crop.
Our model is trained with both the original instance discrimination loss term and the introduced consistency regularization term. The instance discrimination label and the pseudo similarity label jointly construct a virtual soft label on-the-fly, and the soft label further guides the model itself in a bootstrap manner. In this way, CO2 exploits the consistency assumption on unlabeled data, mitigates the "class collision" effect introduced by the one-hot labels, and results in a better visual representation. More importantly, our work brings new perspective of unsupervised visual repre- sentation learning. It relaxes the stereotype that the pretext task can only be self-supervised which aims to construct artificial labels for every sample, c.g., a specific degree of rotation (Gidaris et al.) 2018), configuration of jigsaw puzzle (Noroozi & Favaro 2016), and one-hot label that indicates whether a crop comes from the same instance or not (Wu et al. (2018). In contrast, the pretext task also be seiieeeemspp allowing the task itself to be partially labeled. This relaxation is especially helpful when information for artificial label construction is not enough and imposing a label is harmful, such as the case of imposing the one-hot labels in instance discrimination.
This simple modification brings consistent gains on various evaluation protocols. We first bench- mark CO2 on ImageNet (Deng et al.) 2009) linear classification protocol. CO2 improves MoCo by 2.9% on top-1 accuracy. It also provides 3.8% and 1.1% top-5 accuracy gains under the semi- supervised setting on ImageNet with 1% and 10% labels respectively, showing the effectiveness of the introduced consistency regularization. We also evaluate the transfer ability of the learned repre- sentations on three different downstream tasks: image classification, object detection and semantic segmentation. CO2 models consistently surpass their MoCo counterparts, showing that CO2 can improve the generalization ability of learned representation. Besides, our experiments on ImageNet- 100 (Tian et al.) 2019) demonstrate the efficacy of CO2 on SimCLR (Chen et al., 2020a), showing the generality of our method on different contrastive learning frameworks.
2 METHOD
In this section, we begin by formulating current unsupervised contrastive learning instance discrimination task. Then, we propose our consistency regularization term which addresses the ignorance of the heterogeneous similarity between the query crop and each crop of other images in the instance discrimination task.
2.1 CONTRASTIVE LEARNING
Contrastive learning (Hadsell et al. 2006) is recently adopted as an objective for unsupervised learn- ing of visual representations. Its goal is to find a parametric function fo : RD + Rd that maps an input vector x feature vector fo (x) E Rd with D > d, such that simple distance measure (e.g., cosine distance) in the low-dimensional feature space can reflect complex similarities in the high-dimensional input space.
2

---

Published as a conference paper at ICLR 2021
One-Hot Label
Similarity
Softmax Froh.
Prob.
R-P q., 9
00 ... qiay
P-0:
Cross-Entropi Loss
Consistency Loss
Similarity
Similarity Q
Sofumax Prob
Prob.
11-00
(a)
(b)
Figure I: Illustration of instance discrimination and (b) our consistency regularization term. q is query and p is positive key, both are encoded from crops of the same image. {nx ) negative keys, encoded from random crops. In (a). the similarities are softmax cosine distances between q and each key. (p and (x)))) These similarities one-hot label which identifies p among all keys. However, some negatives can be semantically similar but not reflected by the one-hot label (e.g,, the one rounded by red box). In (b), our proposed consistency regularization encourages the agreement between P, the positive-negative similarities, and Q, the query-negatie similarities, reflecting the heterogeneous similarities between the query/positive and the negatives.
optimized towards artificial
For each input vector x; in the training set S, the similarity measure the input space is defined by subset of training vectors S; C S, called similarity set. The sample x; is deemed similar to samples in the similarity set S, but dissimilar to samples encourages fx(x;) to be close to fo fx()) in the feature space if xj S,, and otherwise to be distant.
\ S, Then, the contrastive objective
By training with contrastive loss, similarities defined by the similarity set determine character- istics of the learned representation and the mapping function fe. For example, if the similarity is defined as samples from the same semantic class, then fo will probably learn invariances to other factors, e.g., object deformation. In the supervised setting, this definition of similarity requires large amount of human labeling. On the contrary, unsupervised contrastive learning exploits simi- larities with need of human labels. One natural definition of unsupervised similarity is multiple views of an image, as explored by many recent methods. For example, random augmented crops (Wul al. 2018 Ye et al. 2019; He al. 2020, Chen al. 2020a b) of image could be defined similanty set. In this case, the contrastive objective is effectively solving an instance discrimination task (Wu et al., 2018) as illustrated in Figure Ia
The training of this instance discriminator involves randomly sampling query crop x S,, positive crop x" E S; from the same image, and K negative crops (x E S S from other images. These K + 2 crops (the query, the positive, and K negatives) are encoded with fo re- spectively, q = fa(x9)
fe(x²), ng = fo(x*). Then, an effective contrastive loss function, InfoNCE (Hjelm et al. 2018 is written
exp(q p/
Lins log
exp(q q
p/Tins) + Σk exp(q (g/TTin))
where Tins is a temperature hyper-parameter (Hinton et al.) 2015). This loss can be interpreted as cross entropy loss that trains the model to discriminate the positive crop (labeled as 1) from negative crops (labeled as 0) given the query crop. We denote this loss as Line as it performs an instance discrimination task. One direct instantiation of InfoNCE loss, represented by SimCLR (Chen al., 2020a)), formulates fa as an end-to-end encoder. this case, two crops of the same image are exchangeable or symmetric each other both are encoded by fo. The final loss is also symmetric
3

---

Published as a conference paper at ICLR 2021
with either one of the two crops as the query and the other crop as the positive. Another popular instantiation, represented by MoCo (He et al. 2020). encodes the query with fe and encodes the positive and the negatives with for which is the moving average of fo- In this case, only q can propagate gradients, which causes Lins to be asymmetric.
2.2 CONSISTENT CONTRAST
The one-hot labels used by InfoNCE loss is effective, showing good generalization ability across tasks and datasets (Chen et al., 2020b a). Nevertheless, we argue that the hard, zero-one labels is uninformative. Specifically, crops from other images are taken as equally negative as they are all labeled as O. This is contradictory the fact that some so-called "negative" crops can be similar or even in the same semantic class, especially when K is large. For example, SimCLR (Chen et al., 2020a) uses 16,382 negative samples batch, and MoCo (He et al. 2020 Chen al. 2020b) uses memory bank of 65,536 features as negative samples. Even worse, the current objective forces negatives to be as far from the query as possible, with larger weights for closer negatives since they are "hard negatives". However, these "hard negative" crops in fact tend to be semantically close. These issues impair good representation learning because the one-hot labels can not faithfully reflect the heterogeneous similarities between the query crop and the crops from other images.
Although generating labels based on instance discrimination trivial revealing the similarity be- tween two arbitrary crops is exactly what want to learn from unsupervised pre-training. There- fore, the label of the similarity between the query crop and each crop from other images is of little hope to get. This situation is similar to the usage of unlabeled data in semi- sumi-supervised learning set- ting, in which consistency regularization is widely used to propagate knowledge from labeled data to discover the structures in unlabeled data. Inspired by this, propose encourage the consistency between the similarities of crops from the same image, i.c., the query crop and the positive crop. We illustrate the consistency regularization in Figure lb
First, we denote the similarity between the query and the negatives ni(i E {l,..., K}) as:
.....,,K))
exp(q ni/TTon)
Q(1) = Ek exp(q ng/Teon))
where Teon is also a temperature hyper-parametee Q(i) the probability that the query q selects n, as its match from [n Similarly, the similarity between the positive p and the negatives is written as:
exp(p ni/Toom)
P(i)
Σ exp(p n& (xoT))))
We impose the consistency between the probability distributions P and Q by using symmetric Kullback-Leibler (KL) Divergence as the measure of disagreement:
Lron DKL. (PII)) + (KIP)
(4)
When and q are encoded by the same end-to-end encoder fo. it is natural to use symmetric KL as their disagreement measure, since p and q are exchangeable. Even when p and n, are encoded by the momentum encoder f/ symmetric KL empirically works well as forward KL, ie., D (Pl as shown in Section 3.5 Thus, we use symmetric KL as a unified objective for both cases.
The total loss a weighted average of the original instance discrimination loss term and the consis- tency regularization term:
(5) where a denotes the coefficient to balance the two terms. possible to merge the two terms by creating unique label containing information of both the one-hot label and the pseudo similarity label, but we find the weighted average can already get good performance and is easy to control.
L Lins + C-Ceon
The pseudo label is informative to reveal the similarity between the query q and each n,, while the one-hot label is unable to provide such information, since it only describe co-occurrence within one image. Note that, the pseudo label is also dynamic since the embedding function fa is updated in every training step, and thus generating better pscudo labels during training. It indicates that the unsupervised embedding function and the soft similarity labels give positive feedback each other.
4

---

Published as a conference paper at ICLR 2021
Table I: Linear classification protocol on ImageNet-IK
Pretext Task
Arch.
Head
#epochs
Top-1 Acc. (%)
ImageNet Classification
R50
90
76.5
Exemplar (Dosovitskiy et al. 2014)
R50w3x
35
46.0
Relative Position (Doersch al. 2015
R50w2x
35
51.4
Rotation Gidaris et al. 2018)
Rv50w4x
35
55.4
Jigsaw Noroozi Favaro 2016)
R50
90
45.7
Methods based on contrustive learning:
InsDisc Wu et al.
2018
R50
Linear
200
54.0
Local Agg. (Zhuang
2019)
R50
Linear
200
58.2
CPC v2 (Hénaff et al. 2019)
RI R170
200
65.9
CMC (Tian et al. (2019)
R50
Liner
240
60.0
AMDIM (Bachman et al. 2019
AMDIM
150
68.1
PIRL (Misra van der Maaten
2020
R50
Linear
800
63.6
SimCL. (Chen et al. 2020a
R50
MLP
1000
69.3
MoCo (He et al.
2020)
R50
Linear
200
60.6
MoCo (He et al.
2020)
+ CO2
R50
Linear
200
63.5
MoCo v2
Chen et al.
2020b
R50
MLP 200
67.5
MoCo v2
CCen et al.
2020b
CO2
R50
MLP 200
68.0
Table Top-5 accuracy for semi-supervised learning on ImageNet
Pretext Task
1% labels
10% labels
Supervised Baseline
48.4
80.4
InsDise (Wu et al.
2018)
39.2
77.4
PIRL (Misra van der Maaten
2020
57.2
83.8
MoCo (He et al.
2020)
62.4
84.1
MoCo (He et al.
2020)
CO2
66.2
85.2
MoCo v2
Chen ci al.
2020b
69.5
85.1
MoCo v2
Chen et al.
20206
CO2
70.6
85.4
Our method is simple and low-cost. It captures the similarity to each n, while introducing unno- ticeable computational overhead with only one extra loss term computed. This is unlike clustering based unsupervised learning methods, which are costly, since they explicitly compute the similarity sets in the training set after every training epoch (Caron et al. 2018 Zhuang et al. 2019: Li et al. 2020 Caron et al. 2020).
EXPERIMENTS
Herein, we first report our implementation details and benchmark the learned representations on ImageNet. Next, we examine how the unsupervised pre-trained models transfer to other datasets and tasks. We then analyze the characteristies our proposed method.
3.1 LINEAR CLASSIFICATION
Setup We mainly evaluate CO2 based on MoCo (He et al. 2020) and MoCo v2 (Chen et al. 2020b). Both of them use instance discrimination as pretext task, while MoCo v2 adopts more sophisticated design choices on projection head architecture, learning rate schedule and data aug- mentation strategy. We test CO2 MoCo for its representativeness and simplicity. On MoCo v2, we evaluate how CO2 compatible with advanced design choices. We also demonstrate the impact of CO2 the end-to-end contrastive framework in Section 3.5
The unsupervised training is performed on the train split of ImageNet- IK (Deng et al. 2009) without using label information. We keep aligned every detail with our baseline MoCo effectively pin- point the contribution of our approach, except the number of GPUs (MoCo uses GPUs while 4). A further search MoCo-related hyper-parameters might lead better results of
5

---

Published as conference paper at ICLR 2021
Table Transfer learning performance on PASCAL VOC datasets
Image Classification
Object Detection
Semantic Segmentation
Pretext Task
mAP
AP50
APall
AP15
mloU
ImageNet Classification
88.0
81.3
53.5
58.8
74.4
Rotation Gidaris et al. 2018
63.9
72.5
46.3
49.3
Jigsaw Noroozi & Favaro 2016
64.5
75.1
48.9
52.9
InsD Wu et al. 2018)
76.6
79.1
52.3
56.9
PIRL (Misra & van der Maaten
2020
81.1
80.7
54.0
59.7
SimCLR (Chen el al. 20200
81.8
55.5
61.4
BYOL Grill et al. 2020)*
81.4
55.3
61.1
SwAV Caron et al.
2020
81.5
55.4
61.4
SimSiam (Chen & He
z020
82.4
57.0
63.7
MoCo He
al.
2020
81.5
55.9
62.6
72.5
MoCo He et al.
2020
(our impl.)
79.7
81.6
56.2
62.4
72.6
MoCo (He
al.
2020
CO2
82.6
81.9
56.0
62.6
73.3
MoCo v2
(Chen et al.
2020b)
85.0
82.4
57.0
63.6
74.2
MoCo v2 (Chen al.
20206 + CO2
85.2
82.7
57.2
64.1
74.7
Results reported in Chen & He 2020).
method. For the hyper-parameters of CO2, we set Teon as 0.04, a as 10 for MoCo-based CO2,
and Tcon 0.05, 0.3 for MoCo v2-based CO2. Please refer the appendix for more detailed implementation description.
3.2 LINEAR CLASSIFICATION
We first benchmark the learned representations on the common linear classification protocol. After the unsupervised pre-training stage, we freeze the backhone network including the batch normaliza- tion parameters, and train linear classifier consisting of a fully-connected layer and softmax layer on the 2048-D features following the global average pooling layer. Table [ summaries the single- crop top-1 classification accuracy improves by 2.9% on MoCo and by 0.5% on MoCo v2. We also list several top-performing methods in the table for reference. These results indicate that the representation more linearly separable ImageNet with consistency regularization, since the consistency regularization mitigates the "class collision" effect caused by semantically similar negative samples.
the validation set of ImageNet-1K: Our method consistently
3.3
SEMI-SUPERVISED LEARNING
We next perform semi-supervised learning on ImageNet to evaluate the effectiveness of the pre- trained network in data-efficient settings. Following (Wu et al. 20188) Misra & van der Maaten 2020 Chen et al., 2020a), we finetune the whole pre-trained networks with only 1% and 10% labels which are sampled in class-balanced way. Tabmee]summaries the mean of the top-5 accuracy on the validation set of ImageNet-IK over three runs. The results for MoCo and MoCo v2 are produced by us using their officially released models. The proposed consistency regularization term provide 3.8% and 1.1% 1% top-5 accuracy gains for MoCo with 1 1% and 10% labels respectively. CO2 also improves from MoCo v2 by 1.1 % top-5 accuracy with % labels, and by 0.3% with 10% labels.
3.4 TRANSFER LEARNING
To further investigate the generalization ability of our models across different datasets and tasks, we evaluate the transfer learning performance on PASCAL VOC (Everingham et al.) 2015) with three typical visual recognition tasks, i.e., image classification, object detection and semantic seg- mentation. Table 3 reports the transfer learning performance comparing with other methods using ResNet-50. CO2 shows competitive or better performance comparing with the corresponding base- lines, In addition, it achieves better performance than oate-of-theeart unsupervised representation learning methods.
Image Classification Following the evaluation setup in Goyal et al. (201)) we train a linear SVM (Boser al. 1992) on the frozen 2048-D features extracted after the global average pool-

---

Published as a conference paper at ICLR 2021
Top-1 1o-1. Accuracy (%)
Top-1 Accuracy (%)
68
68
66
66
64
0
10
20
25
30
0.02
0.04
0.06
0.08
0.10
(a) Effect of varying the coefficient at
(b) Effect of varying the temperature Teon-
Figure 2: Ablation on the effect of hyper-parameters.
100
MoCo
MoCo
MoCo CO2
MoCo CO2
accuracy (%)
80
l055
lo55
40
20:
MoCo
MoCo + CO2
50
100
150
200
50
100
150
200
50
100
150
200
epochs
epachs
epochs
(a) Lins
(h) Leon
(c) Instance discrimination acc.
Figure 3: Training curves of ResNet-18 18 on ImageNet-100.
ing layer. The results of MoCo are produced by with their official models. In this case, CO2 is 2.9% better than MoCo, and 0.2% than MoCo v2.
Object Detection Following the detection benchmark set up in He al. (2020), use Faster R-CNN (Ren et al. 2015) object detector and ResNet-50 C4 (He et al. (017)) backbone, and all the layers are finetuned including the batch normalization parameters. The numbers of our method are averaged over three runs. Our reproduced results for MoCo are also listed in the table for reference. CO2 provides 0.3% AP gains on both MoCo and MoCo v2.
Semantic Segmentation We follow the settings in He et al. (2020) for semantic segmentation. Results are average over three runs. Similarly, reference. The result of MoCo v2 is produced by us using its officially released model. CO2 gives 0.9% mloU improvement upon MoCo, and 0.5% upon MoCo v2, which finally surpasses its supervised counterpart.
include
reproduced results of MoCo
The overall transfer learning improvements, though consistent, are smaller than linear classification and semi-supervised learning. Similar observations have also been made in Chen et al. (2020b). We hypothesize that the current unsupervised contrastive methods, which bring close different crops from the same image, reduce the representation's sensitivity to location which is useful for tasks like detection. It is still an open question which properties of unsupervised representation benefit the transfer ability to various downstream tasks.
3.5 ANALYSIS
In this section, we study the characteristics of the proposed method on a smaller backbone ResNet- 18 and smaller dataset ImageNet-1000 due to the consideration of the computational resource. ImageNet-100 100 firstly used Tian et al. (2019) and consists of 100 randomly selected classes from all 1, 000 classes of ImageNet.
Hyper-parameter Our method introduces two new hyper-parameters, the coefficient of consis- tency regularization term and its temperature Teon- In Figure 2 we show the top-1 accuracy of a linear classifier on models pre-trained by CO2 with different hyper- paper-parameters. In Figure 2a we fix the temperature Toon as 0.04 and vary the coefficient by using the consistency regularization term, the linear classification accuracy can be boosted from 63.6% to 69.2%. Increasing to 20 and beyond causes performance degeneration. We hypothesize that the model is over-regularizer by the consistency loss, and thus it loses some discrimination among different instances. In Figure 2b fix the coefficient to be 10 and varying the tempera- ture. As other consistency regularization methods (e.g.. Berthelot et al. (2019))) temperature Teon effectively influences the quality of the learned representation, and the best to use is 0.04.
The best coefficient is 10. We see that

---

Published as a conference paper at ICLR 2021
Training Curves In Figure 3 we show the training curves of the instance discrimination loss Linss the consistency loss Lcon and the instance discrimination accuracy. Instance discrimination accu- racy represents the percent of query crops which successfully select their corresponding positive crops, i.e., successfully identify their instances. MoCo is trained with Lins only and its Leon is just calculated out for comparison. We see that Lins of MoCo drops quickly from the beginning at the cost of jump of Lcon. As the training proceeds, Ceon of MoCo decreases spontancously, possibly because more semantic knowledge has been learned, but is still relatively high. Training with Leon and Line together, i.e., MoCo + CO2, Leon is kept very low from beginning, and Lcon increases gradually since the model trained discriminate between images at the same time. At the end of the training, Leon stays much lower than Loon of MoCo.
We also notice that with CO2, the instance discrimination accuracy drops from 97.57% to 95.26%. Although CO2 results in lower instance discrimination accuracy, it still does better in the down- stream classification task. The linear classification accuracy improves from 63.6% to 69.2%, as shown in Figure 2a It suggests again that there is gap between instance discrimination and the downstream tasks.
Comparison with Label Smoothing With the consistency regularization term, our approach as- signs soft pseudo labels to crops from other images. This looks similar to label smoothing regu- larization on supervised classification (Szegedy et al., 2016), a useful trick which assigns small constant value to the labels of all the negative classes to avoid overconfidence. We equip MoCo with label smoothing, i.c., assigning a small constant value to crops from other images (the "negatives"). Surprisingly, reports 61.2% linear classification accuracy, 2.4% lower than MoCo alone. This sug- gests that assigning a constant value as label smoothing can be harmful for unsupervised contrastive learning, since it ignores the heterogeneous similarity relationship. And is better to assign labels according to the similarities our consistency regularization.
End-to-Enu Encoder To further verify the effectiveness of the proposed consistency regularization term on differ- ent contrastive learning frameworks, we apply CO2 to Sim- CLR (Chen et al. 2020aa,, representative method with an end-to-end encoder (without momentum encoder). The results are presented in Table 4 On ImageNet- 100 (Tian
Table 4: Linear classification accuracy using end-to-end encoder and with different choices of Lcon- The results are summarized mean and standard deviation over three different runs.
(ct al.] 2019) with a ResNet- 18, SimCLR obtains 68.9% top-1 linear classification accuracy with batch size 128 and temperature Tins 0.1. Equipped with CO2 whose coeffi- cient a 0.07 and temperature Toon is 1.0, the linear clas- sification accuracy is boosted to 72.3%. The improvement demonstrates that CO2 can be applied to different unsu- pervised contrastive frameworks and improve the quality of the learned representation regardless of whether using a momentum encoder or not.
Method
Acc. (%)
SimCLR
68.920.06
SimCLR + CO2
72.3
MoCo
63. 20:99
MoCo + Forward KL
69.6
MoCo + Reverse KL
65. 1
MoCo + CO2
69.7
Varying the choices of Lcon We ablate on different variants of Leon (Eq. 4 MoCo including forward KL (Dk1. (PI(Q) reverse KL (DRr. (Q) and the objective of CO2, i.e., symmetric KL. Each of models uses coefficient a of 10 and temperature Teon of 0.04. We present the linear classification accuracy in Table 4 Our CO2 (symmetric KI.) improves over the baseline MoCo by a large margin, from 63.1% 1% to 69.7%. Forward KL alone improves similarly to 69.6%. And reserve KL alone can also provide a nontrivial 2.0% gain in accuracy.
RELATED WORK
Our method falls in the area of unsupervised visual representation learning, especially for image data. In this section, we first revisit various design strategies of pretext tasks for unsupervised learning. Then we elaborate on the pretext tasks based on contrastive learning, which is the focus of our work. Next, we review the methods using consistency regularization in semi-supervised learning, which inspire our work.
Unsupervised Learning and Pretext Tasks of pretext tasks have been established. The models can be taught to specify the relative position of patch (Doersch et al. 2015), solve spatial jigsaw puzzles (Noroozi & Favaro 2016 Wei et al.,
To learn from unlabeled image data, wide range
8

---

Published as a conference paper at ICLR 2021
2019 colorize gray scale images (Zhang et al. 2016 Larsson et al. 2017). inpaint images Pathak
et al. 2016), count objects (Noroozi et al. 2017). discriminate orientation
Gidaris et al. 2018
iteratively cluster (Caron et 2018| Zhuang et al. 2019; Asano al. 2019: Zhong et al.
2020
generate images (Donahue et al. 2016 Donahue & Simonyan 2019), etc.
Doersch & Zisserman
(2017) evaluates the combination of different pretext tasks. Kolesnikov et al.
2019) Gnyal al.
2019 revisit and benchmark different pretext tasks.
Contrastive Learning Contrastive learning (Hadsell et al. 2006) recently puts new perspective on the design of pretext task and holds the key to most state-of-the-a methods. Most of them per- form an instance discrimination task while differ in i) the strategies to synthesize positives and neg- atives, and ii) the mechanisms to manage a large amount of negatives. The synthesizing can base on context with patches (Hjelm et al. 2018, 2019), random resized crops with data augmentation (Wul et al.
He et al. 2020; Chen et al. 2020a), jigsaw puzzle luminance-chrominang decomposition (Tian) itt al. 2019). Regarding the mechanisms to maintain negative features, some methods (Wu et al., 2018 Misra & van der Maaten 2020) keep tracking the features of all images, some directly utilize the samples within the minibatch (Chen et al. 2020a; Tian et al. 2019 Ye et al.) 2019), and He al.) (2020) proposes to use a momentum encoder. Grill et al. (2020) recently proposes to only use positive examples without negatives. Recently, [Li et al. (2020) argues that the lack of semantic structure is one fundamental weakness of instance discrimination, and proposes to generate proto- types by k-means clustering. However, the computational overhead and the degeneration introduced by clustering are difficult to address. Chuang et al. (2020) also points out the possible sampling bias of instance discrimination, and proposes a debiased objective.
2018 Ye et al., 2019; Bachhmn et al. 2019
transformation (Misra & der Maaten 2020)
Consistency Regularization Consistency regularization an important component of many suc- cessful semi-supervised learning methods. It is firstly proposed in SSajadi et al. (2016), encouraging similar predictions on perturbed versions of the same image. Besides the consistency regulariza- tion on unlabeled data, the model is simultaneously trained with a supervised loss on a small set of labeled data. Several works made improvements on the way of perturbation, including using an ad- versarial transformation (Miyato al.) 2018). using the prediction of a moving average previous model (Tavainnn & Valpola| 2017: Laine& & Aila 2017). and using strong data augmentation (Xic eet al. 2019). Recently, several larger pipelines are proposed (Berthelot al. 2019h Sohn al., 2020 in which consistency regularization still serves as a core component.
The instance discrimination loss in unsupervised contrastive learning is analogous to the super- vised loss in semi-supervised learning, as both of them rely on some concrete information, i.e., co- occurrence in one image and human annotation, respectively. Meanwhile, CO2 on the similarities between crops is analogous to consistency regularization on unlabeled samples of semi-supervised methods as their labels are both unknown. The main difference. however, that semi-supervised methods crucially rely on the supervised loss to warm up the model, while there is no human annota- tion all in unsupervised contrastive learning. Our work presents an example that model learned completely without human annotations can also generate surprisingly effective pseudo labels for similarities between different crops and benefit from consistency regularization.
5 DISCUSSION
Unsupervised visual representation learning has shown encouraging progress recently, thanks to the introduction of instance discrimination and the contrastive learning framework. However, in this paper, we point out that instance discrimination is ignorant of the heterogeneous similarities between image crops. We address this issue with consistency regularization term on the similarities between crops, inspired by semi-supervised learning methods which impose consistency regularization on unlabeled data. In such a simple way, the proposed CO2 consistently improves on supervised and semi-supervised image classification. also transfers to other datasets and downstream tasks.
More broadly, we encourage researchers to rethink label correctness in existing pretext tasks. Taking instance discrimination as an example, we show that a pretext task itself could be, in fact, semi- supervised learning task. It might be harmful to think of the pretext task simple pure supervised task by assuming the unknown labels are negatives. In addition, our work relaxes the stereotype restriction that pretext task labels should always be known and clean. We hope this relaxation can give rise to novel pretext tasks which exploit noisy labels or partially-availabe labels, making a better usage of the data without human annotation.

---

Published as a conference paper at ICLR 2021
REFERENCES
YM Asano, C Rupprecht, and A Vedaldi. Self-labelling via simultaneous clustering and representa- tion learning. In ICLR, 2019.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatth: Supe-supe learning with distribution matching and augmenta- tion anchoring. In ICIRR, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. MixMatch: A holistic approach to semi- semi-supervised learning. In NeurlPS, 2019b.
Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on computational learning theory, 1992.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint 1.Xiv:2011.10566 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprins arXiv:2003.0422) 042997,, 2020b.
Ching-Yan Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. arXiv preprint arXiv:2007.00224 2020.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Carl Doersch and Andrew Zisserman. Multi-task self self-supervised visual learning. In ICCV, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ECCV, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurIPS, 2019.
Jeff Donahue, Philipp Krähenbühhl, and Trevor Darrell. Adversarial feature learning. ICLR, 2016.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NeurIPS, 2014.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. In ICCV, 2019.
10

---

Published as a conference paper at ICLR 2021
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Seman- tic contours from inverse detectors. In ICCV, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint 1905.09999 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurlPS Deep Learning and Representation Learning Workshop. 2015.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: /00::/888.000000 06670, 2018.
Karan Grewal, Phil Bachman, Adam
Devon Hjelm, Alefx Fedorov, Samuel Marchiln,,,,,,,on,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In /CLR, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.1 / 1322, 2020.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In CVPR, 2019.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
G. Larsson, M. Maire, and G. Shakhnarovich. Colorization as proxy task for visual understanding. In CVPR, 2017.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04900 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
llya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2016.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. TPAMI, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
11

---

Published as a conference paper at ICLR 2021
Mchdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning count. In ICCV,, 2017.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Mchdi Sajjadi, Mchran Javammardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep cemi-supervised learning. In NeurIPS, 2016.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICMI,, 2019.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. FixMatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, arXiv preprint arXiv:/906.000 05849, 2019.
Chen Wei, Lingxi Xie, Xutong Ren, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, and Alan L Yuille. Iterative reorganization with weak spatial constraints: Solving arbitrary jigsaw puzzles for unsu- pervised representation learning. In CVPR, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Qizhe Xie. Zihang Dai, Eduard Hovy, Minh-Tham Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848 2019.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. ECCV, 2016.
Huasong Zhong, Chong Chen, Zhongming Jin, and Sian-Sheng Hua. Deep robust clustering by contrastive learning. arXiv preprint 2rXi::2008.03000, 2020.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. CVPR, 2019.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A. IMPLEMENTATION DETAILS OF CONTRASTIVE PRE- TRAINING
We evaluate our approach based on MoCo (He et al. 2020). MoCo has two different encoders to encode queries and keys respectively. The query encoder is updated with respect to the loss function, while the key encoder is an exponential moving average of the query encoder. The keys are stored in dynamic memory bank, whose entries are updated at every training step with the current mini- batch enqueued and the oldest mini-batch dequeued. The backbone is a standard ResNet-50 (He et al. 2016), and features after the global average pooling layer are projected to 128-D vectors (Wu et al. 2018), normalized by &g norm. The size of the memory bank (i.c., the number of negative samples) is 65,536 and the momentum to update the key encoder is 0.999. Tine is 0.07 for MoCo variants and 0.2 for MoCo v2 variants, which are the default settings of these two methods.
We use momentum SGD with momentum 0.9 and weight decay le.4. The batch size is 256 on GPUs. To prevent potential information leak with Batch Normalization (BN) (loffe & Szegedy) 2015), shuffling BN (He al., 2020) is performed. The model is trained for 200 epochs with the initial learning rate of 0.03. The learning rate is multiplied by 0.1 after 120 and 160 epochs for MoCo vl, while cosine decayed (Loshchilov & Hutter 2016) for MoCo v2. We keep aligned all training details with MoCo except the number of GPUs. This could be problematic since it changes the per- worker minibatch size, which is related potential information leaks pointed by [H et al. (2020). However, we do not notice much difference when reproducing MoCo with 4 GPUs. Our reproduced MoCo v2 with GPUs reaches the accuracy of 67.6% on the linear classification protocol, 0. % higher than 67.5% reported in its paper. For the hyper-parameters of the proposed consistency term, we set Teon. as 0.04 and a as 10 for the MoCo v1 vb-aaded CO2, and Teon as 0.05, a as 0.3 for the MoCo v2-based variant.
A.2 IMPLEMETATION DETAILS OF DOWNSTREAM TASKS
Linear Classification We freeze the backbone network including the batch normalization param- eters, and train linear classifier consisting of fully-connected layer followed by softmax on the 2048-D features following the global average pooling layer. We train for 100 epochs. The learning rate is initialized as 15 and decayed by 0.1 every 20 epoch after the first 60 epochs. We set weight decay as 0 and momentum as 0.9. Only random cropping with random horizontal flipping is used as data augmentation.
Semi-Supprrvsed Learning We finetune the pre-trained model for 20 epochs with learning rate starting from 0.01 for the base model and 1.0 for the randomly initialized classification head, de- cayed by 0.2 after 12 and 16 epochs. Momentum is set to 0.9. Weight decay is 5e-4 for MoCo vI and le-4 for MoCo v2. Only random cropping with random horizontal flipping is used as data augmentation.
Classification on PASCAL VOC a linearSVM (Boser et al. 1992) on the frozen 2048-D features extracted after the global average pooling layer. The models are trained on trainval2007 split and tested on test The hyper-parameters are selected based on held-out subset of the training set.
Following the evaluation setup Goyal et al. (019)) we train
Detection on PASCAL VOC Following the detection benchmark set up in He et al. 2020), we use FasterR- RNNN (Ren et al., (2155) object detector and ResNet-50 C4 (He et al. 2017 backbone, implemented in Detectron2 (Wu et al., 2019). We finetune all the layers including the batch nor- malization parameters for 24k iterations on the trra wwn.nla++++. The typer-paramettess are the same as the counterpart with supervised ImageNet initialization and MoCo. To calibrate the small feature magnitude due to the output normalization in the unsuper- vised pre-training stage, two extra batch normalization layers are introduced, one is followed by the regional proposal head whose gradients are divided by 10 and the other is followed by the box prediction head.
split and test on test set.
Segmentation on PASCAL VOC Following the setup in He et al. (2020), an FCN-based (onng jet al. 2015) architecture with atrous convolutions (Chen et al.) (017) () is used and Re is the backbone. The training set is train.aug2012 (Hariharan et al. 2011) and the testing set is val va12012. Initialized with CO2 models, we finetune all layers for 50 epochs 33k iterations) with batch size 16, initial learning rate 0.003, weight decay le-4 and momentum 0.9.
13


Reviews:
[{"review_id": "n5uje1SEcg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nThis paper proposes an extension, coined as CO2, to InfoNCE contrastive loss used in semi/unsupervised methods. CO2 is based on the premise that the query-negative crop similarity distribution and positive-negative crop similarity distribution should be alike. The proposed method yields significant improvements for the linear classification protocol using ImageNet while the improvements for downstream transfer learning tasks such as object detection is marginal. Aside from the performance improvements, the paradigm where the researchers think of the pretext task as a downstream task and improve the pseudo labels by using pseudo-pseudo labels is very interesting. Proposed CO2 method provides a relatively simple way to achieve this. \n\n\nAuthors build upon the intuition that among the many crops the algorithm uses as negatives, it is highly likely that at least a few positive samples exist. These unknown “positive” crops should yield a high similarity to the query, but this is not possible to enforce as these crops’ labels are, by definition, unknown. Instead the authors suggest that positive-negative similarities and query-negative similarities should be alike. A KL divergence term (between positive-negative similarity distribution and query-negative distribution) is used to implement this constraint.\n\n*Novelty*\nTo the best of my knowledge this is a novel approach. It is also surprising that using pseudo-pseudo labels to correct the wrong assumptions of pseudo labels is working reasonably well.\n\n*Impact*\nI believe this paper will have a significant impact. Accuracy improvements on downstream tasks are diminished when the authors use MoCo-v2, suggesting that their method may not always yield significant benefits. However, aside from the numeric accuracy improvements on the downstream tasks, the proposed idea is very simple, seems easy to incorporate in other methods and likely opens new research directions. \n\n*Clarity*\nThe paper is well written and easy to follow. It is also generally clear but for some of my questions/comments please see below.\n\n*Evaluation*\nAuthors use their loss function with MoCo and MoCo-v2 and report relatively small improvements over MoCo-v2. It is an open question whether the proposed loss function would result in large performance gains for other methods. \nOn the transfer learning side, authors report marginal improvements for image classification, object detection and semantic segmentation tasks. It is not possible to form an opinion on how statistically significant these improvements are. Still, the authors provide a comparison with label smoothing which implies that CO2 is a beneficial addition.\n\n*Strengths (Reasons to accept)*\nThis is a relatively simple loss function extension, applicable to other methods.\nExperiments are implying that the method works well and improves upon the state of the art (under reasonable resource constraints).\nAs pointed out in the discussion, the paradigm of relaxing the pretext task’s label constraints (in a way, a learned label smoothing) is likely to open new research directions.\n\n*Weaknesses (Reasons to reject)*\nTransfer learning improvements are marginal (and in the object detection case CO2 results in an unexplained 0.2% drop in AP). I would expect to see a discussion about the reasons behind the discrepancy between the large improvements for semi-supervised learning (or linear classification) vs. the marginal improvements for transfer learning.\nOnly the object detection and semantic segmentation task experiments have been repeated (3 times). The rest of the experiments are, I believe, single runs. This makes the reader question the significance of the reported results.\nA more in depth explanation for the choice of the particular loss in Eq. 4  would benefit the reader. Why use symmetric loss? Please see my questions below.\n\n*Questions and other comments to the authors*\nIn Figure 1 I would like to see where the labels (both One-hot and Pseudo) come from, at least in the caption. The same is true for the similarity graphs, and the authors should consider adding axis-labels and named ticks. Finally, increasing the arrowhead sizes would make everything easier to follow. \nEven though we can not expect the authors to reproduce 1000 epoch and >4000 batch-size methods common to recent semi-supervised techniques, I would still like to see the impact of CO2 using a vanilla ResNet backbone (without the momentum encoder).\nI believe in Eq. 4, the first term alone should be enough to ensure that the “learned” query extractor distribution (Q) matches the “ground truth” distribution (P). In the next few paragraphs authors state P to be dynamic, but with just the first term P would be dynamic as both Q and P depend on the same network. As such, it is not clear to me why the authors chose a symmetric divergence.\nFinally, authors should refrain from using the letter P both for the distribution and for the pseudo label. This is an unnecessary overloading of notation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lLzFn8K8kMN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to add a new consistency loss term to the momentum contrast (MoCo) framework for self-supervised visual representation learning.  A common strategy for self-supervised learning, employed by MoCo as well as others, is to learn invariance to a class of transforms.  Here, a deep network is trained on an instance discrimination task: among distractor images and a transformed (e.g. via data augmentation) version of the input (query) image, correctly identify the transformed example (a classification problem).\n\nConsistency regularization formulates an alternate objective: learn a similarity function between images and, given a gallery of images, encourage the input query and positive example (transformed variant) to have a similar distribution of similarity over the gallery.  The similarity distribution can be treated as a pseudo-label.\n\nThe paper trains a variant of MoCo that combines the standard instance discrimination loss with this consistency regularization loss.  For optimal hyperparameters of this combination, experiments show small but consistent accuracy gains over the baseline MoCo in multiple scenarios: classification and semi-supervised learning on ImageNet, as well as classification, object detection, and semantic segmentation on PASCAL.\n\nIn terms of overall impact, the contribution of this paper appears to be an incremental improvement to the current self-supervised learning paradigm.  The consistency loss itself is not a new idea, as the paper cites Sajjadi et al. (2016).  In fact, Sajjadi et al. examine consistency loss in the context of self-supervised learning, which appears to further limit the novelty of the current paper to the specific contribution of doing so with MoCo.\n\nAs another limitation, the overall gains are small and it is not clear that they necessarily make MoCo+CO2 the top method.  For example, on ImageNet classification, SimCLR outperforms MoCov2+CO2 (69.3 to 68.0 in Table 1), though SimCLR is trained for more epochs.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dwb0yAPNN52", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n<Summary>\n\nThis paper addresses the problem of unsupervised contrastive learning for visual representation. Its key idea is to use a consistency regularization method to resolve the issue of one-hot labels for instance discrimination on which most previous work have relied. The proposed CO2 method is implemented on top of MoCo and MoCo v2 and improves their representation performance on multiple classification and detection tasks. \n\n<Strengths> \n\n1. It borrows  a simple consistency regularize method from semi-supervised learning literature to tackle one issue of recent unsupervised contrastive learning for visual representation – one-hot label that cannot discriminate the semantic closeness to a query between negative samples.\n\n2. The proposed approach is applied to recent SOTA MoCo methods and further improves their performance on image classification, detection and semantic segmentation tasks. \n\n<Weakness>\n\n1. Although this paper can clearly alleviate one issue of recent practice of contrastive learning, the technical novelty is limited. \n\n(1) This paper proposes a new use of an existing technique (consistence regularization) to a new problem (unsupervised contrastive learning). Given that the technique is basic and well-known in semi-supervised learning literature, the proposal of its simple use (with no extension) bears little technical novelty.\n\n(2) In my opinion, the proposal could be a good practice for unsupervised contrastive learning, but its contribution may not be sufficient enough to make this work as a legitimate full paper on ICLR as one of the top premier ML conferences. \n\n2. Experimental evaluation should be improved. \n\n(1) The proposed approach shows the SOTA performance on multiple computer vision tasks, but it largely attributes to the strong performance of MOCO variants. \n\n(2) The performance gain of the proposed method over MOCO v2 is rather marginal as shown in Table 2 and 3. \n\n(3) The proposed CO2 is only implemented on MOCO variants. In order to show the generality of the proposed method, it should be tested with multiple contrastive learning methods and show whether it can consistently improve the methods.\n\n<Conclusion>\n\nMy initial decision is ‘reject’ mainly because the contribution is somewhat limited and more empirical justification for the method is required. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UXUDaD_Cn7Q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposed a consistency regularization for unsupervised visual representation learning. This paper argues that the instance discrimination task performed by most contrastive learning methods merely uses one-hot labels, which cannot reflect the similarities between the query sample and negative samples. In order to tackle this problem, this paper proposed a consistency regularization method to generate pseudo labels and encourages the query sample and its positive sample to have consistent similarities to negative samples. The proposed consistency regularization method is simple and low-cost compared to clustering-based methods.\n\n\n##########################################################################\n\nReasons for score: \n\nOverall, I vote for acceptance, but some concerns on implementations lower my score. The idea of consistency regularization is simple and low-cost, but achieves significant improvement on the popular MoCo baseline. My main concerns are that (1) the improvement on MoCo v2 is much smaller than MoCo while the weight of the consistency regularization term is small (10 for MoCo, 0.3 for MoCo v2), (2) the usage of symmetric KD divergence, (3) the discussion about Mean Teacher [1]. I will change my rating depending on the feedback.\n\n \n##########################################################################\n\nPros: \n\nP1: The research problem, contrastive learning for unsupervised visual representation learning, is popular and important in the CV community.\n\nP2: The proposed consistent contrast (CO2) method is simple but effective. Compared to clustering-based methods, CO2 is less time-consuming. \n\nP3: The performance of linear classification, semi-supervised learning, transfer learning is reported.\n\nP4: The analysis with a smaller backbone and dataset is provided as an ablation study.\n \n##########################################################################\n\nCons: \n\nC1: Section 3.1 states that CO2 can be easily applied to other contrastive learning mechanisms. It is acceptable to me that CO2 is evaluated with MoCo. However, it is not convincing that the application of other methods is easy. The reason is that MoCo and SimCLR have different a design of key encoders. CO2 works well with the momentum updated encoder. However, there is no evidence that CO2 also works well with the end-to-end encoder, since the query and key encoders are updated simultaneously. I doubt that this conclusion is misleading and overstating.\n\nC2: In Section 3.1, $\\alpha$ is set as 10 for MoCo while 0.3 for MoCo v2. The improvement of CO2 over MoCo is 2.9% while only 0.5% for MoCo v2. It is not clear why $\\alpha$ is set at different levels and why the improvement over MoCo v2 is not significant.\n\nC3: This paper is closed to Mean Teacher [Tarvainen & Valpola, 2017]. The contrastive loss (instance discrimination) is like the classification cost in Mean Teacher, while the symmetric KL Diverfebce (consistency regularization) is like the consistency cost. The comparison, especially the difference, between CO2 and Mean Teacher should be discussed.\n\n##########################################################################\n\nQuestions during the rebuttal period: \n\nQ1: It is not clear to me why a symmetric KD divergence (i.e., D(P|Q)+D(Q|P)) is used in Eq. (4) rather than asymmetric KD divergence (i.e., D(Q|P)). In MoCo, the encoder for computing keys is momentum-updated rather than updated by back-propagation. Therefore, Eq.(4) becomes\n\n$\\mathcal{L}_{con}=\\frac{1}{2}\\sum^K_i-P(i)\\log Q(i)+Q(i)\\log Q(i)-Q(i)\\log P(i)$\n\nSince the gradient of $P(i)$ is stopped, $P(i)\\log P(i)$ is ignored. If the gradient of $Q$ in $D_{KL}(Q||P)$ is stopped, then $D_{KL}(Q||P)$ does not contribute to back-propagation. If not, $Q(i)\\log Q(i)$ works like a regularization term as minimizing $Q(i)\\log Q(i)$ may encourage a sharp $Q(i)$. I wonder whether the sharpness implicitly boosts the performance. I hope that more explanation about the symmetric KD divergence, especially (1) why \"symmetric\" rather than only $D_{KL}(Q||P)$. If \"symmetric\" is necessary, additional ablations and references/citations will help. (2) Whether $Q(i)\\log Q(i)$ is minimized. If so, more explanation is expected.\n\nQ2: In section 2.2, Eq. (2) and (3) only consider the similarity between the query and negatives. I wonder why the similarity between the query and positive is ignored. Although the latter is used in Eq.(1), I wonder whether there is any difference if the similarity between the query and positive is also considered in Eq. (2) and (3).\n \n#########################################################################\n\nSuggestions:\n\nS1: In Page 3, this paper states that $q$, $p$ and $n_k$ shares the same encoder $f_\\theta$. However, $q$, $p$ and $n_k$ are obtained with different encoders using MoCo. This part could be carefully revised to be more general.\n\n#########################################################################\n\nReferences:\n\n[1] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.\n\n\n\n\n\n\n------------------------------------------------------------------------------\n\nThank the authors for their good job during rebuttal.\n\nMost of my questions have been addressed. However, my main concern is still that the improvement of CO2 on MoCo v2 is incremental. During the rebuttal stage, the authors did not include more empirical evidence on whether CO2 can improve MoCo v2 well, or theoretical analysis on why the improvement is incremental, both of which are accepteable to me. This severely limits the contribution of this paper. In all, I would not change my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple yet effective approach to improve self-supervised contrastive approaches like MoCo. There are concerns with respect to novelty/simplicity and low improvements over MoCov2. AC believes that simplicity is good and while gains might not be as huge, they still show usefulness of new loss. It might also provide insights for future papers on self-supervised learning. Overall, the sentiment is that paper is above the bar.
Decision: Accept (Poster)