Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations

1 Introduction . Self-supervised learning ( SSL ) [ 25 ] aims to learn powerful representations without relying on human annotations . The representations can be used for various purposes , including transfer learning [ 19 ] , clustering [ 1 , 43 , 44 ] or semi-supervised learning [ 6 ] . Recent self-supervised methods [ 4 , 5 , 17 , 19 ] learn visual representations by imposing invariances to various data transformations . A popular way of formulating this idea is through the instance discrimination task [ 47 ] - which treats each image as a separate class . Augmentations of the same image are considered as positive examples of the class , while other images serve as negatives . To handle the large number of instance classes , the task is expressed as a non-parametric classification problem using the contrastive loss [ 18 , 34 ] . Despite the recent progress , most methods still train on images from ImageNet [ 12 ] . This dataset has specific properties : ( 1 ) the images depict a single object in the center of the image , ( 2 ) the classes follow a uniform distribution and ( 3 ) the images have discriminative visual features . To deploy self-supervised learning into the wild , we need to quantify the dependence on these properties . Therefore , in this paper , we first study the influence of dataset biases on the representations . We take a utilitarian view and transfer different representations to a variety of downstream tasks . Our results indicate that an approach like MoCo [ 19 ] works well for both object- and scene-centric datasets . We delve deeper to understand these results . A key component is the augmentation strategy which involves random cropping . For an object-centric dataset like ImageNet , two crops from the same image will show a portion of the same object and no other objects . However , when multiple ∗Equal contribution 2Code : https : //github.com/wvangansbeke/Revisiting-Contrastive-SSL 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . objects are present , a positive pair of non-overlapping crops could lead us to wrongfully match the feature representations of different objects . This line of thought led recent studies [ 38 , 39 ] to believe that contrastive SSL benefits from object-centric data . So how can contrastive methods learn useful representations when applied to more complex , scenecentric images ? We propose a hypothesis that is two-fold . First , the default parameterization of the augmentation strategy avoids non-overlapping views . As a result , positive pairs will share information , which means that we can match their representations . Second , when applying more aggressive cropping , we only observe a small drop in the transfer learning performance . Since patches within the same image are strongly correlated , maximizing the agreement between non-overlapping views still provides a useful learning signal . We conclude that , in visual pretraining , combining the instance discrimination task with random cropping is universally applicable . The common theme of recent advances is to learn representations that are invariant to different transformations . Starting from this principle , we try to improve the results obtained with an existing framework [ 19 ] . More specifically , we investigate three ways of generating a more diverse set of positive pairs . First , the multi-crop transform from [ 4 ] is revisited . Second , we examine the use of a stronger augmentation policy . Third , we leverage nearest neighbors mined online during training as positive views . The latter imposes invariances which are difficult to learn using handcrafted image transformations . The proposed implementation requires only a few lines of code and provides a simple , yet effective alternative to clustering based methods [ 4 , 30 ] . Each of the proposed additions is found to boost the performance of the representations under the transfer learning setup . The multi-crop transform realizes significant gains . We probe into what the network learns to explain the improvements . The multi-crop transform maximizes the agreement between smaller ( local ) crops and a larger ( global ) view of the image . This forces the model to learn a more spatially structured representation of the scene . As a result , the representations can be directly used to solve several dense prediction tasks without any finetuning . In particular , we observe that the representations already model class semantics and dense correspondences . Furthermore , the representations are competitive with specialized methods [ 24 , 53 ] . In conclusion , the multi-crop setup provides a viable alternative to learn dense representations without relying on video data [ 24 , 29 ] or handcrafted priors [ 23 , 44 , 53 ] . In summary , the overall goal of this paper is to learn more effective representations through contrastive self-supervised learning without relying too much on specific dataset biases . The remainder of this paper is structured as follows . Section 2 introduces the framework . Section 3 whether the standard SimCLR augmentations transfer across different datasets . This question is answered positively . Based upon this result , Section 4 then studies the use of additional invariances to further improve the learned representations . We hope this paper will provide useful insights to other researchers . . 2 Framework . We briefly introduce the contrastive learning framework . The idea is to generate feature representations that maximize the agreement between similar ( positive ) images and minimize the agreement between dissimilar ( negative ) images . Let x be an image . Assume that a set of positives for x can be acquired , denoted by X+ . Similarly , a set of negatives X− is defined . We learn an embedding function f that maps each sample on a normalized hypersphere . The contrastive loss [ 18 , 34 ] takes the following form Lcontrastive = − ∑ x+∈X+ log exp [ f ( x ) T · f ( x+ ) /τ ] exp [ f ( x ) T · f ( x+ ) /τ ] +∑x−∈X− exp [ f ( x ) T · f ( x− ) /τ ] ( 1 ) where τ is a temperature hyperparameter . We will further refer to the image x as the anchor . SSL methods obtain positives and negatives by treating each image as a separate class [ 47 ] . More specifically , augmented views of the same image are considered as positives , while other images are used as negatives . The data augmentation strategy is an important design choice as it determines the invariances that will be learned . Today , most works rely on a similar set of augmentations that consists of ( 1 ) cropping , ( 2 ) color distortions , ( 3 ) horizontal flips and ( 4 ) Gaussian blur . In this paper , we build upon MoCo [ 19 ] - a widely known and competitive framework . However , our findings are expected to apply to other related methods ( e.g . SimCLR [ 5 ] ) as well . The embedding function f with parameters θf consists of a backbone g ( e.g . ResNet [ 20 ] ) and a projection MLP ImageNet MS-COCO head h. The contrastive loss is applied after the projection head h. MoCo uses a queue and a moving-averaged encoder f ′ to keep a large and consistent set of negative samples . The parameters θf ′ of f ′ are updated as : θf ′ = mθf ′ + ( 1 − m ) θf with m a momentum hyperparameter . The momentum-averaged encoder f ′ takes as input the anchor x , while the encoder f is responsible for the positives X+ . The queue maintains the encoded anchors as negatives . We refer to [ 19 ] for more details . 3 Contrastive Learning in the Wild . Most contrastive self-supervised methods train on unlabeled images from ImageNet [ 12 ] . This is a curated dataset with unique characteristics . First , the images are object-centric , i.e . they depict only a single object in the center of the image . This differs from other datasets [ 28 , 31 ] which contain more complex scenes with several objects . Second , the underlying classes are uniformly distributed . Third , the images have discriminative visual features . For example , ImageNet covers various bird species which can be distinguished by a few key features . In contrast , domain-specific datasets ( e.g . BDD100K [ 52 ] ) contain less discriminative scenery showing the same objects like cars , pedestrians , etc . In this section , we study the influence of dataset biases for contrastive self-supervised methods . Setup . We train MoCo-v2 [ 7 ] on a variety of datasets . Table 1 shows an overview . The representations are evaluated on six downstream tasks : linear classification , semantic segmentation , object detection , video instance segmentation and depth estimation . We adopt the following target datasets for linear classification : CIFAR10 [ 27 ] , Food-101 [ 26 ] , Pets [ 35 ] , Places365 [ 56 ] , Stanford Cars [ 26 ] , SUN397 [ 48 ] and VOC 2007 [ 16 ] . The semantic segmentation task is evaluated on Cityscapes [ 10 ] , PASCAL VOC [ 16 ] and NYUD [ 40 ] . We use PASCAL VOC [ 16 ] for object detection . The DAVIS2017 benchmark [ 37 ] is used for video instance segmentation . Finally , depth estimation is performed on NYUD [ 40 ] . The model , i.e . a ResNet-50 backbone , is pretrained for 400 epochs using batches of size 256 . The initial learning rate is set to 0.3 and decayed using a cosine schedule . We use the default values for the temperature ( τ = 0.2 ) and momentum ( m = 0.999 ) hyperparameters . 0 20 40 60 80 100 0 0.2 0.4 0.6 0.8 1 Low overlap High overlap IoU between crops C D F Figure 1 : IoU between random resized crops for existing frameworks . 10 30 50 70 100 60 70 80 90 Max IoU between views ( pretraining ) T ra n sf er p er fo rm an ce [ % ] Cityscapes ( IoU ) VOC SVM ( mAP ) DAVIS ( Jm & Fm ) Figure 2 : Transfer results when thresholding the IoU between crops . 0 1 3.1 Object-centric Versus Scene-centric . We compare the representations trained on an object-centric dataset - i.e . ImageNet ( IN-118K ) - against the ones obtained from two scene-centric datasets - i.e . COCO and OpenImages ( OI-118K ) . Tables 2-3 show the results under the linear classification and transfer learning setup . Results . The linear classification model yields better results on CIFAR10 , Cars , Food and Pets when pretraining the representations on ImageNet . Differently , when tackling the classification task on Places , SUN and VOC , the representations from COCO and OpenImages are better suited . The first group of target benchmarks contains images centered around a single object , while the second group contains scene-centric images with multiple objects . We conclude that , for linear classification , the pretraining dataset should match the target dataset in terms of being object- or scene-centric . Next , we consider finetuning . Perhaps surprisingly , we do not observe any significant disadvantages when using more complex images from COCO or OpenImages for pretraining . In particular , for the reported tasks , the COCO pretrained model even outperforms its ImageNet counterpart . We made a similar observation when increasing the size of the pretraining dataset ( see suppl . materials ) . Discussion . In contrast to prior belief [ 38 , 39 ] , our results indicate that contrastive self-supervised methods do not suffer from pretraining on scene-centric datasets . How can we explain this inconsistency with earlier studies ? First , the experimental setup in [ 38 ] only considered the linear evaluation protocol for an object-centric dataset ( i.e . PASCAL cropped boxes ) . This analysis [ 38 ] does not show us the full picture . Second , the authors conclude that existing methods suffer from using scene-centric datasets due to the augmentation strategy , which involves random cropping . They argue that aggressive cropping could yield non-overlapping views which contain different objects . In this case , maximizing the feature similarity would be detrimental for object recognition tasks . However , the default cropping strategy barely yields non-overlapping views3 . This is verified by plotting the intersection over union ( IoU ) between pairs of crops ( see Figure 1 ) . We conclude that the used example of non-overlapping crops [ 38 , 39 ] seldom occurs . The above observation motivates us to reconsider the importance of using overlapping views . We pretrain on COCO while forcing the IoU between views to be smaller than a predefined threshold . Figures 2-3 show the transfer performance and training curves for different values of the threshold . The optimization objective is harder to satisfy when applying more aggressive cropping ( i.e . the training loss increases when lowering the IoU ) . However , Figure 2 shows that the transfer performance remains stable . Patches within the same image were observed at the same point in time and space , which means that they will share information like the camera viewpoint , color , shape , etc . As a result , the learning signal is still meaningful , even when less overlapping crops are used as positives .

Summary: This paper presents rigorous analyses of self-supervised learning from different aspects. Here are their findings:  1. The pretraining dataset should match the target dataset in terms of being object- or scene-centric. 1. MoCo is robust to changes in the class distribution of the dataset (pre-trained on uniform ImageNet vs long-tailed ImageNet) 1. The model trained on data with discriminative features (ImageNet) performs equally well with the model trained on data without (BDD100K) in the fine-tuning setting for semantic segmentation tasks.  The authors also propose three strategies to improve MoCo: 1. Multi-scale constrained cropping data augmentation 1. Mixture of standard augmentation and AutoAugment 1. Including nearest neighbors as positive pairs