Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations
Authors: Wouter Van Gansbeke
Simon Vandenhende
Stamatios Georgoulis
Luc Van Gool
Venue: NeurIPS 2021 Poster (2021)
Keywords: representation learning
unsupervised learning
self-supervised learning
contrastive learning
computer vision

Abstract: Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection. However, current methods are still primarily applied to curated datasets like ImageNet. In this paper, we first study how biases in the dataset affect existing methods. Our results show that an approach like MoCo works surprisingly well across: (i) object- versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets. Second, given the generality of the approach, we try to realize further gains with minor modifications. We show that learning additional invariances - through the use of multi-scale cropping, stronger augmentations and nearest neighbors - improves the representations. Finally, we observe that MoCo learns spatially structured representations when trained with a multi-crop strategy. The representations can be used for semantic segment retrieval and video instance segmentation without finetuning. Moreover, the results are on par with specialized models. We hope this work will serve as a useful study for other researchers.

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations
Wouter Van Gansbeke Simon V
Stamatios Georgoulis'
Luc Van Gool 1.2
KU Leuven/ESAT-PSI
ETH Zurich/CVL
Abstract
Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection. However, current methods are still primarily applied to curated datasets like ImageNet. In this paper, we first study how biases in the dataset affect existing methods. Our results show that approach like MoCo [19] works surprisingly well across: (i) object- versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets. Second, given the generality of the approach, realize further gains with minor modifications. We show that learning additional invariances through the use of multi-scale cropping, stronger augmentations and nearest neighbors improves the representations. Finally, we observe that MoCo learns spatially structured representations when trained with multi-crop strategy. The representations can be used for semantic segment retrieval and video instance segmentation without finetuning. Moreover, the results are on par with specialized models. We hope this work will serve as useful study for other researchers. The code and models available
try
Introduction
Self-supervised learning (SSL) [25] aims to learn powerful representations without relying human annotations. The representations can used for various purposes, including transfer learning [19], clustering [1, 43, 44] semi-supervised learning [6]. Recent self-supervised methods [4, 5, 17, 19] learn visual representations by imposing invariances to various data transformations. popular way of formulating this idea is through the instance discrimination task [47] which treats each image as a separate class. Augmentations of the same image are considered as positive examples of the class, while other images serve negatives. To handle the large number of instance classes, the task is expressed as a non-parama non-parametric classification problem using the contrastive loss [18, 34].
Despite the recent progress, most methods still train on images from ImageNet [12]. This dataset has specific properties: (1) the images depict a single object in the center of the image, (2) the classes follow a uniform distribution and (3) the images have discriminative visual features. To deploy self-supervised learning into the wild, we need to quantify the dependence on these properties. Therefore, in this paper, we first study the influence of dataset biases on the representations. We take a utilitarian view and transfer different representations to a variety of downstream tasks.
Our results indicate that an approach like MoCo [19] works well for both object- and scene-centric datasets. We delve deeper understand these results. key component is the augmentation strategy which involves random cropping. For an object-centric dataset like ImageNet, two crops from the same image will show a portion the same object and no other objects. However, when multiple
"Equal contribution
"Code: attps://githur http:://gh...ccccoaaaaaaaaaaaaaa htwp:raaaattttteottttttttteeeeee
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

objects are present, a positive pair of -veerapalg crops could lead us to wrongfully match the feature representations of different objects. This line of thought led recent studies [38, 39] to believe that contrastive SSL. benefits from object-centric data.
So how can contrastive methods learn useful representations when applied more complex, scene- centric images? We propose a hypothesis that is two-fold. First, the default parameterization of the augmentation strategy avoids non-overaapping views. As a result, positive pairs will share information, which means that we can match their representations. Second, when applying more aggressive cropping, we only observe a small drop in the transfer learning performance. Since patches within the same image are strongly correlated, maximizing the agreement between non-overlapping views still provides a useful learning signal. We conclude that, in visual pretraining, combining the instance discrimination task with random cropping is universally applicable.
The common theme of recent advances is to learn representations that are invariant to different transformations. Starting from this principle, we try to improve the results obtained with an existing framework [19]. More specifically, we investigate three ways of generating more diverse set of positive pairs. First, the multi-crop transform from [4] is revisited. Second, we examine the use of a stronger augmentation policy. Third, leverage nearest neighbors mined online during training as positive views. The latter imposes invariances which are difficult to learn using handcrafted image transformations. The proposed implementation requires only few lines of code and provides a simple, yet effective alternative to clustering based methods [4, 30J. Each of the proposed additions found to boost the performance of the representations under the transfer learning setup.
The multi-crop transform realizes significant gains. We probe into what the network learns to explain the improvements. The multi-crop transform maximizes the agreement between smaller (local) crops and a larger (global) view of the image. This forces the model to learn a more spatially structured representation of the scene. As a result, the representations can be directly used solve several dense prediction tasks without any finetuning. In particular, we observe that the representations already model class semantics and dense correspondences. Furthermore, the representations are competitive with specialized methods [24, 53]. In conclusion, the multi-crop setup provides a viable alternative to learn dense representations without relying on video data [24, 29] or handcrafted priors [23, 44, 53].
In summary, the overall goal of this paper is to learn more effective representations through contrastive self-supervised learning without relying too much specific dataset biases. The remainder of this paper is structured as follows. Section 2 introduces the framework. Section 3 whether the standard SimCLR augmentations transfer across different datasets. This question is answered positively. Based upon this result, Section 4 then studies the use of additional invariances to further improve the learned representations. We hope this paper will provide useful insights other researchers...
2 Framework
We briefly introduce the contrastive learning framework. The idea is to generate feature representa- tions that maximize the agreement between similar (positive) images and minimize the agreement between dissimilar (negative) images. Let z be an image. Assume that set of positives for x can be acquired, denoted by X+. Similarly, a set of negatives X is defined. We learn an embedding function f that maps each sample on normalized hypersphere. The contrastive loss [18, 34] takes the following form
explf(x)" exp [f(z) f(x))//] exp f(x)T . f(x))/=]
Leontrastive
Σ log x
(1)
where 7 temperature hyperparameter. We will further refer to the image r as the anchor.
SSL methods obtain positives and negatives by treating each image as a separate class [47]. More specifically, augmented views of the same image are considered positives, while other images are used as negatives. The data augmentation strategy is an important design choice as it determines the invariances that will be learned. Today, most works rely on a similar set of augmentations that consists of (1) cropping, (2) color distortions, (3) horizontal flips and (4) Gaussian blur
In this paper, we build upon MoCo [19] widely known and competitive framework. However, our findings are expected to apply to other related methods (e.g. SimCLR [5]) as well. The embedding function of with parameters 0y consists of backbone g (e.g. ResNet [20]) and a projection MLP

---

Table 1: Overview of the training datasets. We sample uniform and long-tailed (LT) subset of 118K images from ImageNet. On OpenImages, we sample a random subset I18 images. The complete train splits are used for COCO and BDDI00K, The figure shows some examples.
MISCICC
Pretrain Data
#lmgs
#Ohj/Img Uniform Discriminative
ImageNet- 18K 18K [12]
118
1.7
ImageNet-118K-I1 18Ke-T [12]
118 K
1.7
COCO [31]
118 K
7.3
X
Opentizagte
OpenImages- 1 18K [28]
118 K
8.4
BDDI00K [52]
90 K
X
X
Table 2: Comparison of linear classification models trained on top of frozen features (400 epochs pretraining).
Pretrain Data
CIFAR10
Cars
Food
Pets
Places
SUN
VOC
IN-118K
83.1
35,9
62.2
68.9
45.0
50.0
75.8
COCO
77.4 (-5.7)
33.9 (-2.0)
62.0 (-0.2)
62.6 (-633)
47.3 (-2.3)
53.6 (43.6)
80.9 (4-5.))
O1-118K
74.0 (-9.))
32.2 (-3.7)
58.4 (-38)
59.3
46.6 (-1.6)
52.3 (+2.3)
75.9 (400.)
IN-118K-IT
83.2 (+001)
36.1 (40.2)
62.1 (-0.1)
69.1 (40.2)
45.3 (-0.3)
50.4 (+0.4)
76.1 (40.33
Table Comparison of different representations under transfer learning setup (400 epochs pretraining).
Semantic seg, (mloU)
Detection (AP)
Vid. seg. (J&FF)
Depth (rmse)
Pretrain Data
VOC
Cityscapes
NYUD
VOC
DAVIS
NYUD
IN-118K
68.9
70.1
37.7
53.0
63.5
0..25
COCO
69.1 (+0.2)
70.3 (40.22
39.3 (+1.6)
53.0 (+000)
65.1 (+116)
0.612 (- (-0113)
O1-118K
67.9 (-10)
70.9 (+0.8)
38.4 (+0.7)
53.1 (+003)
64.8 (+L.3)
0.609 (- 0.016
IN-118K-LT
69.0 (40.1)
70.1 (+0.0)
37.9 (40.2)
53.0 (+0.0)
63.7 (+0.2)
0.622 (-0.003)
BDD100K
70.1 (+0.0)
head h. The contrastive loss is applied after the projection head h. MoCo queue and moving-averaged encoder f' to keep a large and consistent set of negative samples. The parameters Or of f' are updated as: 0p m0y (1 m)05 with m momentum hyperparameter, The momentum-averrgee encoder f' takes as input the anchor æ, while the encoder f is responsible for the positives x+, The queue maintains the encoded anchors as negatives. We refer to [19] for more details.
3 Contrastive Learning in the Wild
Most contrastive self-supervised methods train on unlabeled images from ImageNet [12]. This is curated dataset with unique characteristics. First, the images are object-centric, i.e. they depict only a single object in the center of the image. This differs from other datasets [28, 31] which contain more complex scenes with several objects. Second, the underlying classes are uniformly distributed. Third, the images have discriminative visual features. For example, ImageNet covers various bird species which can be distinguished by a few key features. In contrast, domain domain-speciff specific datasets (e.g. BDD100K [52]) contain less discriminative scenery showing the same objects like cars, pedestrians, etc. In this section, we study the influence of dataset biases for contrastive self-supervised methods.
Setup. We train MoCo-v2 [7] on a variety of datasets. Table I shows an overview. The represen- tations are evaluated on six downstream tasks: linear classification, semantic segmentation, object detection, video instance segmentation and depth estimation. We adopt the following target datasets for linear classification: CIFAR10 10 [27], Food-101 [26]. Pets [35], Places365 [56], Stanford Cars [26]. SUN397 [48] and VOC 2007 [16]. The semantic segmentation task is evaluated on Cityscapes [10].
PASCAL VOC [16] and NYUD [40]. We use PASCAL VOC [16] for object detection. The DAVIS- 2017 benchmark [37] used for video instance segmentation. Finally, depth estimation performed on NYUD [40]. The model, i.e. ResNet-50 backbone, is pretrained for 400 epochs using batches
of size 256. The initial learning rate set to 0.3 and decayed using a cosine schedule. We use
default values for the temperature (r 0.2) and momentum (m 0.999) hyperparameters.

---

100
Ciasseaps (toli) VOc SVM (mAP)
MEax IoU
betweez vlews
0.6
Losr awniap
ligh arndap
Thonder peeforiaance (36)
DAVIS (J,,)
80
30
Triining
70
100 (Defoult)
0.44
20
20
30
50
100
betwors cropu
ToUl (pretracminn))
Training progress
Figure 1: IoU between random re- sized crops for existing frameworks.
Figure 2: Transfer results when Figure 3: Training curves when thresholding the IoU between crops.
thresholding the loU between crops.
3.1 Object-centric Versus Scene-centric
We compare the representations trained on an object-centric dataset i.e. ImageNet (IN-118K)- against the ones obtained from two scene-centric datasets - i.e. COCO and Openlmages (0I-118K). Tables 2-3 show the results under the linear classification and transfer learning setup.
Results. The linear classification model yields better results CIFARIO. 10, Cars, Food and Pets when pretraining the representations on ImageNet. Differently, when tackling the classification task on Places, SUN and VOC, the representations from COCO and OpenImages are better suited. The first group of target benchmarks contains images centered around a single object, while the second group contains cenneccnttic images with multiple objects. We conclude that, for linear classification, the pretraining dataset should match the target dataset in terms of being object- or scene-centric.
Next, we consider finetuning. Perhaps surprisingly, we do not observe any significant disadvantages when using more complex images from COCO or Openlmages for pretraining. In particular, for the reported tasks, the COCO pretrained model even outperforms its ImageNet counterpart. We made similar observation when increasing the size of the pretraining dataset (see suppl. materials).
Discussion. In contrast prior belief [38, 39], our results indicate that contrastive self-supervised methods do not suffer from pretraining on scene-centric datasets. How can we explain this inconsis- tency with earlier studies? First, the experimental setup in [38] only considered the linear evaluation protocol for an object-centric dataset (i.e. PASCAL cropped boxes). This analysis [38] does not show us the full picture. Second, the authors conclude that existing methods suffer from using scene-centric datasets due to the augmentation strategy, which involves random cropping. They argue that aggressive cropping could yield non-overlapping views which contain different objects. this case, maximizing the feature similarity would be detrimental for object recognition tasks. However, the default cropping strategy barely yields non-overlapping views". This is verified by plotting the intersection over union (IoU) between pairs of crops (see Figure 1). We conclude that the used example of non-overlapping crops [38, 39] seldom occurs.
The above observation motivates us to reconsider the importance of using overlapping views. We pretrain on COCO while forcing the IoU between views to be smaller than a predefined threshold. Figures 2-3 show the transfer performance and training curves for different values of the threshold. The optimization objective is harder satisfy when applying more aggressive cropping (i.e. the training loss increases when lowering the IoU). However, Figure 2 shows that the transfer performance remains stable. Patches within the same image were observed at the same point time and space, which means that they will share information like the camera viewpoint, color, shape, etc. As result, the learning signal is still meaningful, even when less overlapping crops are used as positives.
3.2 Uniform Versus Long-tailed
Next, we study whether MoCo benefits from using a uniform (IN-118K) versus long-tailed (IN-118K LT) dataset. We adopt the sampling strategy from [32] to construct long-tailed version of ImageNet. The classes follow the Pareto distribution with power value a 6. Tables 2-3 indicate that MoCo
We use the RandomBessizedCrop in PyTorch with scaling (0.2, 1.0) and output size 224 x 224.

---

Table 4: Ablation of different components. Models pretrained for 200 epochs COCO using the settings We indicate the differences with MoCo. The best model is trained with additional invariances.
from Section
Setup
Semantic seg. (mloU)
Classification (mAP/ Acc. / Acc.)
Method
MC
CC m.
A
f
NN
VOC
Cityscapes
NYUD
VOC
ImageNet
Places
Rand. init.
39.2
65.0
24.4
MoCo
X
X
66.2
70.3
38.2
76.1
49.3
45.1
Sec. 4.1
X
X
X
X
69.9
(+3.7)
70.9 (++06)
39.4 (+1.2)
81.3 (+5.2)
53.4 (+4.1))
47.7 (+2.6)
x
X
X
70.2
(+4.0)
70.9 (40.6)
39.5 (+1.3)
82.1 (4660)
54.0 (4-..7)
47.9 (+2.8)
<
v
v
X
X
70,9
(+4.7)
71.3 (++10)
39.9 (+1.7)
82.8 (4-6.)
54.8 (+5.55)
48.1 (+3.0)
Sec. 4.2
V
X
71.4
(+5.2)
72.0 (+1.7)
40.0 (+1.8)
83.7 (+7.6)
55.5 (+-622)
48.2 (+3.1)
Sec. 4.3
V
V
V
71.9
(+5.7)
72.2 (+1.)
40.9 (+2.7)
85.1 (49.0)
55,9 (+-6.6)
48.5 (+3.4)
MC: Multi-crop. CC: Constrained multi-crop. m4: Lower momentum,
Stronger augmentations. NN: Nearest neighbors
is robust to changes in the class distribution of the dataset. In particular, the IN-1 IN-118K-LT model performs on par or better compared to its IN-1 counterpart across all tasks. We conclude that is not essential use uniformly distributed dataset for pretraining.
3.3 Domain-Specific Datasets
Images from ImageNet have discriminative visual features, e.g. the included bird species can be recognized from their beak or plumage. In this case, the model could achieve high performance on the most discriminative feature in the image.
the instance discrimination task by solely focusing Differently, in urban scene understanding, we deal with more monotonous scenery - i.e. all images contain parked cars, lane markings, pedestrians, etc. We try measure the impact of using less discriminative images for representation learning.
We pretrain on BDD 100K dataset for urban scene understanding. Table 3 compares the repre- sentations for the semantic segmentation task on Cityscapes when pretraining on BDDI00K versus IN-118K. The BDD100K model performs on par with the IN-118K pretrained baseline (BDDI0OK, +0.00%). This shows that MoCo can be applied to domain-specific data as well. Two models trained on very different types of data perform equally well. A recent study [54] showed that it is mostly the low- and mid-level visual features that are retained under the transfer learning setup. This effect explains how two different representations produce similar results. Finally, we expect that further gains can be achieved by finetuning the augmentation strategy using domain knowledge.
4 Learning Invariances
In Section 3, showed that MoCo works well for a large variety of pretraining datasets: scene- centric, long-tailed and domain-specific images. This result shows that there is no obvious need to use more advanced pretext task [39] or use an alternative source of data like video [38] to improve the representations. Therefore, instead of developing a new framework, we try to realize further gains while sticking with the approach from MoCo. We realize this objective by learning additional invariances. We study three such mechanisms, i.e. multi-scale constrained cropping, stronger augmentations and the use of nearest neighbors. We concentrate on the implementation details and uncover several interesting qualities of the learned representations. Table 4 shows an overview of the results.
4.1 Multi-Scale Constrained Cropping
The employed cropping strategy proves crucial to obtain powerful visual representations. So far, we used the default two-crop transform, which samples two image patches to produce an anchor and a positive. The analysis Section 3 showed that this transformation yields highly-overlapping views. As a result, the model can match the anchor with its positive by attending only to the most discriminative image component, while ignoring other possibly relevant regions. This observation leads us to revisit the multi-cron crop augmentation strategy from [4].
5

---

84
VOC (mLAP)
343
80
200 cpordas
600 eppodos
73
800 upeodis
2: 224
MoCo
MoDo
0.58
3x
2%
4%
Xx.
Crops
timee (relative)
Figure The constrained multi-crop. Smaller crops forced to lic within the anchor image. Invalid/valid crops are colored red/green.
Figure Transfer performance versus time for the multi-crop (MC) model.
Setup. The multi circcroppp transform samples N additional positive views for each anchor. Remem- ber, the views are generated by randomly applying resized cropping, color distortions, horizontal flipping and Gaussian blur. We adjust the scaling parameters and resolution of the random crop transform to limit the computational overhead. The two default crops (i.e. the anchor and a positive) contain between 20% and 100% of the original image area, and are resized to 160 x 160 pixels. Differently, the N additional views contain between 5% and 14% of the original image area, and are resized to 96 x 96 pixels. We maximize the agreement between all N + positives and the anchor. Note that the N smaller, more aggressively scaled crops are not used as anchors, i.c. they are not shown to the momentum-updated encoder and are not used negatives in the memory bank.
Further, we consider two additional modifications. First, we enforce the smaller crops overlap with the anchor image. In this way, all positives can be matched with their anchor as there is shared information. The new transformation referred to as constrained multi-croppn cropppicoppin..... Figure illustrates the concept. Second, since increasing the number of views facilitates faster training, we reduce the momentum hyperparameter m from 0.999 0.995. We pretrain for 200 epochs on COCO.
Results. Table 4 benchmarks the modifications. Both the multi (MC) and constrained multi-crop (CC) significantly improve the transfer performance. Reducing the momentum (m.)) yields further gains. Figure plots the training time versus performance for the multi-crop model. The multi-crop improves the computational efficiency w.r.t. the two-crop transform.
Discussion. We investigate what renders the multi-cron and constrained multi crop effec- tive. The network matches the representations of the large anchor image with the small crops. Since the small crops show random subparts of the anchor image, the network is forced to encode as much information as possible, such that all image regions be retrieved from the representation of the scene. As a result, the representations will be informative of the spatial layout and different objects in the image rather than attend only to the most discriminative image component.
The latter verified by visualizing the class activation maps (CAMs) [55] of different representations (see Figure 6). The CAMs of the multi-croo model segment the complete object, while the swo-crop model only looks at few discriminative components. Additional examples can found in the suppl. materials.
Table 5: DAVIS 2017 video instance segmentation. We use the publicly available from [24] to evaluate our frozen representations. Qualitative results are shown for MoCo trained with the multi-crop (MC) transform.
Method
Data
Jm
Fm
DenseCL. [45]
COCO
60.6
63.9
MAST [29]
YT:VOS
63.3
67.6
STC [24]
Kinetics
64.8
70.2
MoCo
COCO
61.6
66.6
MoCo MC
COCO
64.3
69.4

---

Table 6: VOC semantic segment retricval. Each image partitioned into number of segments by running K-Means (K=15) on the spatial features. Then we adopt region descriptor computed by averaging embeddings of all pixels within a segment to obtain nearest neighbors of the validation regions from the train set. Qualitative results are shown for MoCo trained with the multi crop (MC) augmentation strategy (K=5).
Segm. Retr. VOC
Method
7 classes (loU)? 21 classes (loU)*
SegSort [53]
10.2
SSL. HG [53]
24.6
DenseCL. [45]
48.4
35.1
MoCo
41.8
28.1
McCo + MC
48.1
35.1
Further, the representations are evaluated on two dense prediction tasks without any finetuning. We consider the tasks of video in- stance segmentation on DAVIS 2017 [37] and semantic segment retrieval on VOC [16]. The multi coulti-cree crop model is found superior at modeling dense correspondences for the video instance seg- mentation task in Table 5 (MoCo vs. MoCo + MC). Similarly, Table shows that the multi-crop model achieves higher per- formance the semantic segment retrieval task compared to its
Malx Mahs-Dap
two-crop counterpart. This indicates that the pixel embeddings are better disentangled according to the semantic classes. Finally, the MoCo mul ti-crop model is competitive with other methods that were specifically designed for the tasks. We conclude that the multi- crop setup provides a viable alternative to learn dense representations without supervision. Moreover, this setup does not rely on video [24, 29] or handcrafted priors [23, 44, 53].
Figure 6: Class activation maps for the two-crop multi-crop model.
4.2 Stronger Augmentations
As mentioned before, the image crops are augmented by random color distortions, horizontal flips and Gaussian blur. Can do better by using stronger augmentations? We investigate the use of AutoAugment [11] an advanced augmentation policy obtained using supervised learning on ImageNet. We consider three possible strategies to augment the positives: (I) standard augmentations,
(2) AutoAugment and (3) randomly applying either (1) or (2).
Table 7 compares the representations under the linear evaluation pro- tocol on PASCAL VOC. Replacing the standard augmentations with AutoAugment degrades the performance (from 82.8% 79.9%). However, randomly applying either the standard augmentations AutoAugment does improve the result (from 82.8% 83. 7%). Chen el al. [5] showed that contrastive SSL benefits from using strong color augmentations as two random crops from the same image will share a similar color distribution. AutoAugment applies fewer color distortions, resulting in lower performance. This compensated when combining both augmentation strategies. Finally, Table 4 shows that combin- ing our custom augmentation policy (A+) with the model from Section 4.1 results in considerable improvements on all tasks.
Table 7: Ablation of augmentation policies applied random crops.
Augmentation policy
vOC SVM (mAP)
Standard
82,8
Augment
79.9
Standard or AutoAugment
83.7
4.3 Nearest Neighbors
Prior work [43] showed that the model learns to map visually similar images closer together than dissimilar ones when tackling the instance discrimination task. In this section, we build upon this insight by imposing additional invariances between neighboring samples. By leveraging other samples as positives, we can capture a rich set of deformations that are hard to model via handcrafted augmentations. However, well-known problem with methods that group different samples together
7

---

Anchor
Algorithm Pseudocode for KNN-MoCo
MLP
8. the mncnentun-updatee backbone
gradient
h, (nonentun-update)) proj.
X: batch of anchor images
#
# x batch of positives
BBL
queua 2g
C,
Qi queue xa
Pusilives
EMA
2y x - g' gg(t))
BxCy B-Lxxy
ANN
Indiions
Select
X
Kh KA - b (x,) (ax)) e BxCk B-LxC, " xy detachh Xj. da.dattchh()
4pu" hen(xy, vi,.(,, C,.) (ie lmn mn(x, (o)).vie((B,1..
gradient
logits catt( lare ,iim=)) # Bxla (X+1)
MLP
indices (xpp( .0,) Li, diiz-2)
Figure 7: KNN-MoCo setup. Q, and Qn maintain aligned queues of backbone and oulput features. As before, the en- coder f needs match positives with their anchor (green arrow). Additionally, we match the positives with their k nearest neighbors obtained from the queue Q2 (blue arrow).
loss: (1) sharpen with 5, (2) apply
lossise: = CE(logits/t, zeros((B,,))
loss.. multi label
Laall___aaee__dElaggr _00(__/t indicas)
1os8sst 10s8:ast laabdaa 10s8g.
bam: batch matrix mull.; mairix mult;; cat: cuncate-
nate; topk: indices of top kelements; cross-entropy
is preventing representation collapse. Therefore, we formulate our intuition as an auxiliary loss that regularizes the representations keeping the instance discrimination task as the main force.
Setup. Recall that the encoder f consists of backbone and projection head h. MoCo maintains a queue 2n, of encoded anchors {g6 - processed by the momentum encoder f". We now introduce second equally sized and aligned queue 2, which maintains the features from before the projection head {16- 4 ) The queue of backbone features 2g can be used to mine nearest neighbors on-the-fly during training. In particular, for a positive x processed by the encoder f the of its nearest neighbors N₁r {ae sim(g(x*). 4 top & E 2g) is computed w.r.t. the queue Qg. The cosine similarity measure is denoted by sim. Finally, we use the contrastive loss maximize the agreement between + and its nearest neighbors N₁+ after the projection head:
exp 1 f(x*)//]
L Σ
Σ 9EN
log explg" f/T -x-ex- Ex- exp la" f//=)//|
(2)
The total loss the sum of the instance discrimination loss Lins and the nearest neighbors loss Cun from Eq. 2: Linst + ALm- Figure shows schematic overview of our k-Nearest Neighbors based Momentum Contrast setup (kNN-MoCo). Algorithm contains the pseudocode (see also suppl.).
Table Ablation study number neighbors & and weight for linear classifier on VOC. represents the multi-crop model from Section 4.1. Models are trained for 200 epochs on COCO.
k
5
10
20
50
0.0
0.1
0.2
0.4
0.8
MAP (%)
83.6
84.0
84.1
84.2
84.3
MAP (%)
82.8
83.8
84.1
84.2
80.7
Table 9: State-of-the-art comparison. MoCo and DenseCL are trained for 800 epochs on COCO. VirTex is trained COCO captions [8]. MoCo is trained while imposing various additional invariances.
Semantic seg. (mloU)
Classification (mAP/ Acc. Asc.)
Vid. Seg. (J&FF)
Depth (rmse)
Method
vOC
Cityseapes
NYUD
vOC
ImageNet
Places
DAVIS
NYUD
Rand. init.
39.2
65.0
24.4
40.8
0.867
DenseCL. [45]
73.2
73.5
42.1
83.5
49.9
45.8
61.8
0.589
VirTex [13]
72.7
72.5
40.3
87.4
53.8
61.3
0.613
MoCo
71.1
71.3
40.0
81.0
49.8
44.7
63.3
0.606
+ CC
72.2
71.6
40.4
84.0
54.6
46.1
65.5
0.595
CC+ A+
72.7
71.8
40.7
85.0
56.0
47.0
65.7
0.590
+ + KNN
73.5
72.3
41.3
85.9
56.1
48.6
66.2
0,580
CC: Constrained multi-crop (Sec. 4.1), A+: Stronger augmentations (Sec. 4.2). KNN: nearest neighbors (Sec. 4.3).

---

Results & Discussion. Table 8 contains an ablation study of the number of neighbors i and the weight X. The performance remains stable for large range of neighbors i (A fixed 0.4). However, increasing the number of neighbors k positively impacts the accuracy. We use k = 20 for the remainder of our experiments. Further, the representation quality degrades when using a large weight (e.g. À = 0.8). This shows the importance of using the instance discrimination task as our main objective. Also, not shown in the table, we found that it important to mine the neighbors using the features before the projection head (84.2% vs 82.8%). Finally, Table 4 shows improved results on all tasks when combining the nearest neighbors loss with our other modifications. In conclusion, we have successfully explored the data manifold to learn additional invariances. The proposed implementation can be seen as a simple alternative to clustering baaeeddgg-aeeddddddee methods [2, 4, 30, 46].
4.4 Discussion
We retrain our final model for 800 epochs on COCO and compare with two other methods, i.e. DenseCL [45] and Virtex [13]. We draw the following conclusions from the results in Table 9. First, our model improves over the MoCo baseline. The proposed modifications force the model to learn useful features that can not be learned through standard data augmentations, even when increasing the training time. Second, our representations outperform other works on several downstream tasks. These frameworks used more advanced schemes [45] or caption annotations [13]. Interestingly, DenseCL reports better results for the segmentation tasks on Cityscapes and NYUD, but performs worse on other tasks. In contrast, the performance of our representations is better balanced across tasks. We conclude that generic pretraining is still an unsolved problem.
Related Work
Contrastive learning. The idea of contrastive learning [18, 34] is to attract positive sample pairs
and repel negative sample pairs. Self-supervised methods [5, 17, 19, 33, 41, 42, 47, 51] have used the contrastive loss to learn visual representations from unlabeled images. Augmentations of the same image are used as positives, while other images are considered as negatives.
A number of extensions were proposed to boost the performance. For example, group of works [21, 36, 44, 45] applied the contrastive loss the pixel-level learn dense representations. Others improved the representations for object recognition tasks by re-identifying patches [50] or by maximizing the similarity of corresponding image regions in the intermediate network layers [49]. Finally, Selvaraju et al. [39] employed an attention mechanism to improve the visual grounding abilities of the model. In contrast to these works, we do not employ more advanced pretext task to learn spatially structured representations. Instead, we adopt standard framework [19] and find that the learned representations exhibit similar properties when modifying the cropping strategy. Further, we expect that our findings can be relevant for other contrastive learning frameworks too.
Clustering. Several works combined clustering with self-labeling [2, 3] or contrastive learning [4, 30, 46] learn representations in self supervisdd way. Similar to the nearest neighbors loss (Eq. 2). these frameworks explore the data manifold to learn invariances. Differently, we avoid the use of clustering criterion like K-Means by computing nearest neighbors on-the-fly fly w.r.t. memory bank. few other works [22, 43] also used nearest neighbors as positive pairs in an auxiliary loss. However, the neighbors had to be computed lif-line at fixed intervals during training. Concurrent to our work, Dwibedi et al. [14] adopted nearest neighbors from a memory bank under the BYOL [17] framework. The authors focus on image classification datasets. conclusion, we propose a simple, yet effective alternative to existing clustering methods.
Other. Contrastive SSL has been the subject of several recent surveys [15, 38, 54]. We list the most relevant ones. Similar to our work, Zhao et al. [54] pretrain on multiple datasets. They investigate what information is retained under the transfer learning setup, which differs from the focus of this paper. Purushwalkam and Gupta [38] study the influence of the object-centric dataset bias, but their experimental scope is rather limited, and their conclusions diverge from the ones in this work. Eriesson et al. [15] compare several ImageNet pretrained models under the transfer learning setup. In conclusion, we believe our study can complement these works.

---

Conclusion
In this paper, we showed that we can find generic of sugmeentaiions/ivaaianees that allows us to learn effective representations across different types of datasets (i.e., scene-centric, non-uniform, -pmaainspeciffc). We provide empirical evidence to support this claim. First, Section 3, we show that the standard SimCLR augmentations can be applied across several datasets. Then, Section 4 studies the use of additional invariances to improve the results for a generic dataset (i.e., MS-COCO). In this way, reduce the need for dataset or domain-specific expertise to learn useful representations self-supervised way. Instead, the results show that simple contrastive frameworks apply to a wide range of datasets. We believe this is encouraging result. Finally, our overall conclusion differs from a few recent works [38, 39], which investigated the use of more advanced pretext task or video to learn visual representations.
Our paper also yields a few interesting follow-up questions. Modalities. Can we reach similar conclusions for other modalities like video, text, audio, etc.? Invariances. What other invariances can be applied? How do we bias the representations focus more on texture, shape or other specific properties? Compositionality. Can we combine different datasets to learn better representations?
Broader Impact
The goal of this work to study and improve contrastive self-supervised methods for learning visual representations. Supe-vupe learning aims to learn useful representations without relying human annotations. Our analysis indicates that existing methods can be applied to large variety of datasets. This observation could benefit applications where annotated data is scarce, like medical imaging, or where large amounts of unlabeled data are readily available, like autonomous driving. Our work also improves the learned representations, and thus benefits many downstream tasks like semantic segmentation, classification, etc. These tasks are relevance to many applications. At this point, hard to assess all possible societal implications of this work. After all, the advantages or disadvantages of new applications using the studied technology will depend on the intentions of the users or inventors.
The authors thankfully acknowledge support by Toyota via the TRACE project and MACCHINA (KU Leuven, C14/18/065). This work also sponsored by the Flemish Government under the Flemish AI programme.
Acknowledgment.
10

---

References
[1] Asano, Y.M.. Patrick. M., Rupprecht, C., Vedaldi, A.: Labelling unlabelled videos from scratch with multi-modal supe-super In: Advances in Neural Information Processing Systems (NcurIPS) (2020)
[2] Asano, Y.M., Rupprecht, C., Vedaldi, A.: Self-labelling via simultancous clustering and representation
learning. In: International Conference on Learning Representations (ICLR) (2020) 9
[3] Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: European Conference on Computer Vision (ECCV) (2018) 9
Caron, M., Misra, I,, Mairal, J.. Goyal. P., Bojanowski, P., Joulin, A:: Unsupervised learning of visual features by contrasting cluster assignments. In: Advances Neural Information Processing Systems (NeurIPS) (2020) 1, 2,9
[5] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. International Conference on Machine Learning (ICML) (2020) 1. 7.9
[6] Chen, T., Kornblith, S., Swersky, K., Norouzi, M., Hinton, G.: Big supf-suue models are strong sumi-supervised learners. In: Advances in Neural Information Processing Systems (NcurIPS) (2020) 1
[7] Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0429 202909973.49777 (2020) 3
Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.I.:: Microsoft coco captions:
Data collection and evaluation server. arXiv preprint 15X:v:1504.00525 (2015)
[9] Chen, X., He, K.: Exploring simple siamese representation learning, In: Conference Computer Vision
and Pattern Recognition (CVPR) (2021) 9
[10] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding, In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016) 3
[11] Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, V.: Autoaugment: Learning augmentation strategies from data. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)* 7
[12] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image
database. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009) 1, 3
[13] Desai, K., Johnsonn J.: Virtex: Learning visual representations from textual annotations. In: Conference on
Computer Vision and Pattern Recognition (CVPR) (2021) 8,
[14] Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: With a little help from my friends: Nearest--eiihhor contrastive learning of visual representations. arXiv preprint arXiv:2104. 14548 (2021) 9
[15] Ericsson, L., Gouk, H., Hospedales, T.M.: How well do superviid models transfer? Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[16] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International Journal of Computer Vision (ICV) (2010) 3,7
[17] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Docrsch, C., Pires, B.A., Guo, Z.D., Azar, M.G., et al.: Bootstrap your own latent: A new approach to self-supervised learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) 1,9
[18] Gutmann, M., Hyvärinen, A.: Noise-contrastive estimation: A new estimation principle for unnormalized
statistical models. In: AISTATS (2010) 1, 9
[19] He, K., Fan, H. Wu, Y. Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation
learning. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020) 1, 3,
[20] He. K. Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016) 2
[21] Hénaff, O.J., Koppula, S., Alayrac, J.B., Oord, A.v.d., Vinyals, O., Carreira, J.: Efficient visual pretraining with contrastive detection. arXiv preprint arXiv:2103. 10957 (2021)
[22] Huang, J., Dong, Q.. Gong, S., Zhu, X.: Unsupervised deep learning by neighbourhood discovery. In: International Conference on Machine Learning (ICML) (2019)
[23] Hwang, J.J., Yu, S.X., Shi, J., Collins, M.D., Yang, T.J., Zhang, X., Chen, L.C.: Segsort: Segmentation by discriminative sorting of segments. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 2.7
[24] Jabri, A., Owens, A., Efros, A.A.: Space-time correspondence contrastive random walk. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) 2, 6,
[25] Jing, L.. Tian, Y.: Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) (2020)
11

---

[26] Krause, J., Deng, J., Stark, M., Fei-Fei, L.: Collecting large-scale dataset of fine-grained cars (2013) 3
[27] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009) 3
[28]
Kuznelsova, A., Rom, H., Alldrin, N., Uijlings. J., Krasin, I,, Pont-Tuset, J., Kamali, S., Popov, s., Malloci, M., Kolesnikov, A., et al.: The open images dataset v4. International Journal of Computer Vision (ICV) (2020) 3
[29] Lai, Z., Lu, E., Xie, W.: Mast: A memory-augmmented self-supervised tracker. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020) 2, 6.
[30] Li, J.. Zhou, P.. Xiong, C., Socher, R., Hoi, S.C.: Prototypical contrastive learning of unsupervised
representations. In: International Conference on Learning Representations (ICLR) (2021) 2,9
[31] Lin, T.Y.,, Maire. M.. Belongie. S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference on Computer Vision (ECCV) (2014) 3
[32] Liu, Z., Miao, Z., Zhan, X., Wang. J.. Gong, B., Yu, S.X.: Large-scale long-tailed recognition in an open world. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 4
[33] Misra, L., van der Maaten, L.: Self-supervised learning of gretext-invarian representations. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2020) 9
[34] Oord, v.v.d. Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807 .07448 (2018) 2,
[35] Parkhi, O.M., Vedaldi, A., Zisserman, A.. Jawahar, C.: Cats and dogs. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012) 3
[36] Pinhciro, P.O., Almahairi, A., Benmaleck, R. Y., Golemo, F,, Courville, A.: Unsupervised lcarning of dense visual representations. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) 9
[37] Pont- Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Honee Hormung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv:1 704.00675 (2017) 3,
[38] Purushwalkam, S., Gupta, A.; Demystifying contrastive self- supervied learning: Invariances, augmenta- tions and dataset biases. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) 2, 4, 5,
9, 10
[39] Selvaraju, R.R., Desai, K., Johnson, J., Naik, N.: Casting your model: Learning to localize improves self-supervised representations. In: Conference Computer Vision and Pattern Recognition (CVPR) (2020) 2, 4, 5, 10
[40] Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: European Conference on Computer Vision (ECCV) (2012)
[41] Tian, Y., Krishnan, D., Isola, P:: Contrastive multiview coding. In: European Conference Computer Vision (ECCV) (2020) 9
[42] Tian, Y. Sun, C.,, Poole, B., Krishnan, D., Schmid, C., Isola, P.: What makes for good views for contrastive learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2020) 9
[43] Van Gansbeke, W.. Vandenhende, S., Georgoulis, S. Proesmans, M., Van Gool, L.: Scan: Learning to classify images without labels. In: European Conference Computer Vision (ECCV) (2020) 1,9,9
[44] Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Van Gool, L.: Unsupervised semantic segmentation by contrasting object mask proposals, arXiv preprint arXiv:2102.061199 (2021) 1, 2, 7,9
[45] Wang, X., Zhang, R., Shen, C., Kong, T., Li, L.: Dense contrastive learning for self-supervised visual pre- pre-training. In: Conference Computer Vision and Pattern Recognition (CVPR) (2020) 6, 8, 9
[46] Wang, X., Liu, Z. Yu, S.X.: Unsupervised feature learning by cross-level discrimination between instances and groups. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)* 9
[47] Wu, Z.. Xiong. Y., Yu. S.X., Lin, D.: Unsupervised feature learning via non- porpamametricc instance discrimi- nation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018) 2.
[48] Xiao, J., Hays, J., Ehinger, K.A., Oliva, A.. Torralba, A.: Sun database: Large-scale scene recognition from abbey zoo. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2010) 3
[49] Xiao, T., Reed, C.J., Wang, X., Keutzer, K., Darrell, T.: Region similarity representation learning. arXiv
preprint arXiv:2103.12900 (2021)* 9
[50] Yang. C., Wu, Z., Zhou, B., Lin, S.: Instance localization for self- superviee detection pretraining, In: Conference Computer Vision and Pattern Recognition (CVPR) (2021)
[51] Ye, M., Zhang, X., Yuen, P.C.. Chang, S.F.: Unsupervised embedding learning via invariant and spreading instance feature. In: Conference Computer Vision and Pattern Recognition (CVPR) (2019) 9
12

---

[52] Yu, F., Chen, H., Wang, X.. Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T.: Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In: Conference Computer Vision and Pattem Recognition (CVPR) (2020)
[53] Zhang, X., Maire, M.: Self-supervised visual representation learning from hierarchical grouping. In:
Advances in Neural Information Processing Systems (NeurIPS) (2020) 7.7
[54] Zhao, N., Wu, Z., Lau, R.W. Lin, S.: What makes instance discrimination good for transfer learning? In:
International Conference on Learning Representations (ICLR) (2020) 5, 9
[55] Zhou, Khosla, Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
[56] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Transactions Pattern Analysis and Machine Intelligence (T-PAMI) (2017) 3
13


Reviews:
[{"review_id": "t3rgDT3jKt2", "reviewer": "Reviewer_syro", "summary": "- Current contrastive methods mainly focus on curated iconic datasets like ImageNet.\n- The paper studies the effect of dataset biases (like centered object, uniform data distribution) on performance of existing self-supervised learning methods.\n- The paper shows that MoCo works well across a wide range of datasets. The paper analyses these datasets by separating them into different categories.\n- In addition to the analysis, the paper proposes improvement by modifications to multi-scale cropping, stronger augmentations and a NN based methods. In particular, the authors observe that random cropping for instance discrimination tasks works well across scenarios. \n- The paper shows that multi-crop transform leads to significant gains and gives rise to spatially structured representations which are useful for tasks like semantic segment retrieval and video instance segmentation.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper investigates the effect of dataset biases on self-supervised learning approaches. Further, the focus on uncurated datasets is an important, and use of these datasets for self-supervised learning has only started recently.\n- The authors quantitatively verified that small IoU in scene centric dataset rarely occurs. As pointed out by the authors, this would effectively ensure that for decently sized objects, the two crops would have a significant overlap between the positives.\n- The observation that transfer performance doesn't change much wrt changing Max IoU is also quite interesting.\n- MC and CC forces the network to focus less on the discriminative parts but rather the whole object. This is also qualitatively shown using gradcam masks. \n- The paper proposes a simple neat trick to mine for more examples by finding the nearest neighbors for anchors using features before the projection layer.\n- The paper seems good improvements using a combination of the three proposed modifications to MoCo. \n- The paper is an empirical one that shows some interesting analysis before delving into some easy fixes to improve performance.\n\nConcerns:\n- Would you have a significant memory penalty since you are storing pre projection features for the entire queue and these are typically of a larger dimension (2048D?) than the features after the final projection layer (typically 128D)\n- The novelty of the proposed approaches is a bit lacking since it builds on top of methods that have been proposed previously (Multi-crop, Autoaugment). But, the combination of these approaches and the simple fixes to improve performance will be beneficial to the community. Further, the analysis on dataset biases and self-supervised learning will be beneficial to the community. \n- The paper should mention the motivation behind choosing a 118K sized subset for ImageNet and OpenImages dataset. It looks like this is motivated from the size of the COCO dataset but it should be clarified.\n- The authors note that the improvements start to reduce when the size of the dataset increases. It will be beneficial to show a $\\Delta$ from a baseline trained with the same amount of data.\n- L136 : I am curious if you have some experiments to show that this is indeed what's happening. While I understand that camera parameters would remain the same, wouldn't that be an incorrect signal for the self-supervised learning task, ie a shortcut that we don't want the network to rely on ? Further, for color/shape, it makes sense if the two crops are a part of the same object but that cannot be ensured for datasets like OpenImages.\n- L145 : How do you explain the better performance of the long tailed model in Tables 2/3 ?\n- L246 : should be $g(x^+)$ ? Also, the equation is a bit confusing. It might help to have a call to the projection layer explicitly here.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors have adequately addressed the potential negative societal impact of their work.\n- The supplementary material includes a discussion on the drawbacks of the proposed approach.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qDwA5v7gh76", "reviewer": "Reviewer_1MyT", "summary": "The paper studies the current stream of instance discrimination-based self-supervised learning algorithm (MoCo) beyond the typical object-centric dataset, i.e., ImageNet. Experiments are conducted on scene-centric datasets, long-tail datasets, and general-domain datasets. Results show that MoCo works well on these datasets. Furthermore, the authors studied multi-crop and stronger data augmentation for MoCo and showed that these enhanced augmentation techniques improved performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\tThe paper is clearly rewritten and very easy to follow.\n-\tThe paper studies some of the essential problems in SSL. SSL models are trained mostly on ImageNet dataset, which by itself has inductive bias from data collecting procedure. Studying on other types of datasets, i.e., scene-centric and general-domain datasets, is an important steppingstone to better scale up the training of SSL models.\n-\tExperiments are thorough and convincing. Benchmarks are properly set up and follow the common practice.\n\nWeaknesses:\n-\tMissing supervised baselines. Since most experiments are done on datasets of scale ~100k images, it is reasonable to assume that full annotation is available for a dataset at this scale in practice. Even if it isn’t, it’s an informative baseline to show where these self-supervised methods are at comparing to a fully supervised pre-trained network.\n-\tThe discussion in section 3 is interesting and insightful. The authors compared training datasets such as object-centric versus scene-centric ones, and observed different properties that the model exhibited. One natural question is then what would happen if a model is trained on \\emph{combined} datasets. Can the SSL model make use of different kinds of data?\n-\tThe authors compared two-crop and multi-crop augmentation in section 4, and observed that multi-crop augmentation yielded better performance. One important missing factor is the (possible) computation overhead of multi-crop strategies. My estimation is that it would increase the computation complexity (i.e., slowing the speed) of training. Therefore, one could argue that if we could train the two-crop baseline for a longer period of time it would yield better performance as well. To make the comparison fair, the computation overhead must be discussed.\nIt can also be seen from Figure 7, for the KNN-MoCo, that the extra positive samples are fed into the network \\emph{that takes the back-propagated gradients}. It will drastically increase training complexity as the network not only performs forward passing, but also the backward passing as well.\n-\tSection 4.2 experiments with AutoAugment as a stronger augmentation strategy. One possible trap is that AutoAugment’s policy is obtained by supervise training on ImageNet. Information leaking is likely.\n\nQuestions\n-\tIn L114 the authors concluded that for linear classification the pretraining dataset should match the target dataset in terms of being object or-scene centric. If this is true, is it a setback for SSL algorithms that strive to learn more generic representations? Then it goes back again to whether by combining two datasets SSL model can learn better representations.\n-\tIn L157 the authors discussed that for transfer learning potentially only low- and mid-level visual features are useful. My intuition is that low- and mid-level features are rather easy to learn. Then how does it explain the model’s transferability increasing when we scale up pre-training datasets? Or the recent success of CLIPs? Is it possible that \\emph{only} MoCo learns low- and mid-level features?\n\nMinor things that don’t play any role in my ratings.\n-\t“i.e.” -> “i.e.,”, “e.g.” -> “e.g.,”\n-\tIn Eq.1, it’s better to write L_{contrastive}(x) = instead of L_{contrastive}. Also, should the equation be normalized by the number of positives? \n-\tL241 setup paragraph is overly complicated for an easy-to-explain procedure. L245/246, the use of x+ and x is very confusing.\n-\tIt’s better to explain that “nearest neighbor mining” in the intro is to mine nearest neighbor in a moving embedding space in the same dataset.\n\nOverall, I like the objective of the paper a lot and I think the paper is trying to answer some important questions in SSL. But I have some reservation to confidently recommend acceptance due to the concerns as written in the “weakness” section, because this is an analysis paper and analysis needs to be rigorous. I’ll be more than happy to increase the score if those concerns are properly addressed in the feedback.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors didn't discuss the limitations of the study. I find no potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "jpeedP2Ibst", "reviewer": "Reviewer_vQPG", "summary": "This paper presents rigorous analyses of self-supervised learning from different aspects. Here are their findings:\n\n1. The pretraining dataset should match the target dataset in terms of being object- or scene-centric.\n1. MoCo is robust to changes in the class distribution of the dataset (pre-trained on uniform ImageNet vs long-tailed ImageNet)\n1. The model trained on data with discriminative features (ImageNet) performs equally well with the model trained on data without (BDD100K) in the fine-tuning setting for semantic segmentation tasks.\n\nThe authors also propose three strategies to improve MoCo:\n1. Multi-scale constrained cropping data augmentation\n1. Mixture of standard augmentation and AutoAugment\n1. Including nearest neighbors as positive pairs", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n- This paper provides comprehensive and rigorous studies for different aspects of the pre-training data being used for SSL.\n- Results in Tab. 2 challenge the observation in [38, 39]. The authors identify that the non-overlapping between views is in fact not the major issue. What really matters is to match the pre-training and the target dataset in terms of being object- or scene-centric. Analyses in Fig. 1 and Fig. 2 further support their claims.\n- The proposed augmentation and training strategies in Sec. 4 are effective for improving MoCo's performance.\n\n### Weakness\n\n- The connection between the first part of the paper (Sec. 3) and the second part of the paper (Sec. 4) is weak. It looks like a concatenation of two papers. The reviewer expects to see how the analyses from Sec. 3 motivate the strategies proposed in Sec. 4.\n\n- Is the proposed strategies in Sec. 4 generalizable to other SSL methods? For example, a clustering-based method like SwAV.\n\n- For fine-tuning results, the reviewer suggests including baseline methods of random init for reference.\n\n- In Sec. 3.3, the experiment of BDD on linear evaluation is missing. Is it possible that training on BDD has a negative effect in the linear evaluation setting?\n\n- In Sec 4.2, the authors only explain why using AutoAugment leads to worse results. The explanations/analyses of why standard + auto-augment leads to better performance are not provided.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- (copy and paste from above) Is the proposed strategies in Sec. 4 generalizable to other SSL methods? For example, a clustering-based method like SwAV.\n\n- It is still unclear what kind of feature, bias, invariance, etc, the model captures when pre-trained on different datasets given only the performance in the linear evaluation setting and the fine-tuning setting presented in Sec. 3. For example, does the model learn a more robust representation? Is the model biased to shape or to texture?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "e79_tLvB0_", "reviewer": "Reviewer_KpV4", "summary": "The paper provides a comprehensive study and a set of improvements for the self-supervised method MoCo. First, the authors challenge the common assumptions that the training dataset has to be object-centric, not long-tailed and the crops overlapping, which provides very useful insights. Then, the authors propose a bag of tricks to improve the performance of MoCo in practice.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Good:\n\n- Very interesting and practically useful analysis of MoCo in Section 3. I would love to see a more extensive study on this matter in the paper.\n- Extremely useful for practitioners bag of tricks presented in Section 4. I particularly appreciate k-NN positives presented in Section 4.3.\n\n### Bad:\n\n- Submission to the wrong venue. While the paper contains very useful content, there is no technical novelty at all, rather a set of experiments applying one model to different datasets and providing a bag of tricks for better performance. This would be a perfect fit for representation learning workshop, or perhaps a purely vision conference.\n- The paper is lacking a holistic message. While Section 3 studies the inner working of MoCo and how it performs in different settings, Section 4 proposes technical improvements that have little to do with discoveries in Section 3. The paper has a feel of \"we've made these experiments and wrote about all of them in this paper\".\n- It is not clear whether the improvements will translate to other self-supervised models rather than MoCo and whether it will translate to longer training schedules on full-size training datasets, i.e. the mode where SoTA self-supervised results are reported.\n\n### Questions:\n\n- It is surprising for me that having littler overlap between crops in scene-centric training datasets does not affect the performance on the target dataset (see Figure 2-3). While I can understand this effect to some extent when testing on a scene-centric dataset, what would happen if the test dataset is object-centric?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper had thorough reviews, rebuttals, and discussion. The reviewers all agreed that the work presents a very interesting analysis and set of conclusions, along with some simple methods to improve results of MoCo-learned representations. While individual contributions are simple and not necessarily novel from a technical perspective, as agreed by the reviewers the set of narratives, experiments, interesting findings, and methods globally provide an interesting novel perspective to the community. Analysis of invariances and especially dataset types (e.g. object/scene-specific) is indeed interesting and one that may exist as common knowledge but has not been thoroughly and rigorously analyzed. In other words, while the paper is largely an empirical investigation it is well-executed in making claims/hypothesis and resulting analysis. 

One of the main concerns expressed by multiple reviewers is the lack of a strong connection and holistic perspective offered across the two sections of the paper. In the rebuttal, the authors provided several arguments and expanded this narrative and connections, which the reviewers were satisfied with. The authors should comprehensively incorporate this (and other great suggestions made by the reviewers) into the final version. While not mandatory, one of the remaining weaknesses that have not been addressed is the applicability of the findings to other self-supervised learning methods. We encourage the authors to add this if at all possible, as it would significantly increase the impact of the paper. 
Decision: Accept (Poster)