Center-wise Local Image Mixture For Contrastive Representation Learning

1 INTRODUCTION . Learning general representations that can be transferable to different downstream tasks is a key challenge in computer vision . This is usually achieved by fully supervised learning paradigm , e.g. , making use of ImageNet labels for pretraining over the past several years . Recently , self-supervised learning has attracted more attention due to its free of human labels . In self-supervised learning , the network aims at exploring the intrinsic distributions of images via a series of predefined pretext tasks ( Doersch et al. , 2015 ; Gidaris et al. , 2018 ; Noroozi & Favaro , 2016 ; Pathak et al. , 2016 ) . Among them , instance discrimination ( Wu et al. , 2018 ) based methods have achieved remarkable progress ( Chen et al. , 2020a ; He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ) . The core idea of instance discrimination is to push away different images , and encourage the representation of different transformations ( augmentations ) of the same image to be similar . Following this paradigm , self-supervised models are able to generate features that are comparable or even better than those produced by supervised pretraining when evaluated on some downstream tasks , e.g. , COCO detection and segmentation ( Chen et al. , 2020c ; b ) . In contrastive learning , the positive pairs are simply constrained within different transformations of the same image , e.g. , cropping , color distortion , Gaussian blur , rotation , etc .. Recent advances have demonstrated that better data augmentations ( Chen et al. , 2020a ) really help to improve the representation robustness . However , contrasting two images that are de facto similar in semantic space is not applicable for general representations . It is intuitive to pull semantically similar images for better transferability . DeepCluster ( Caron et al. , 2018 ) and Local Aggregation ( Zhuang et al. , 2019 ) relax the extreme instance discrimination task via discriminating groups of images instead of an individual image . However , due to the lack of labels , it is inevitable that the positive pairs contain noisy samples , which limits the performance . In this paper , we target at expanding instance discrimination by exploring local similarities among images . Towards this goal , one need to solve two issues : i ) how to select similar images as positive pairs of an image , and ii ) how to incorporate these positive pairs , which inevitably contain noisy assignments , into contrastive learning . We propose a new kind of data augmentation , named Centerwise Local Image Mixture , to tackle the above two issues in a robust and efficient way . CLIM consists of two core elements , i.e. , a center-wise positive sample selection , as well as a data mixing operation . For positive sample selection , the motivation is that a good representation should be endowed with high intra-class similarity , and we find that although MoCo ( He et al. , 2020 ) does not explicitly model invariance to similar images , the intra-class similarity becomes higher as the training process goes . Based on this observation , we explicitly enforce semantically similar images towards the center of clusters , and generate representation with higher intra-class similarity , which we find is beneficial for few shot learning . This is achieved by searching nearest neighbors of an image , and only retaining similar samples that are closer to the corresponding cluster center , which we denote as center-wise local sample selection . As a result , an image is pulled towards the center while do not break the local similarity . Once similar samples are selected , a direct way is to treat these similar samples as multiple positives for contrastive learning . However , since feature representation in high dimensional space is complex , the returned positive samples inevitably contain noisy assignments , which should not be overconfident . Instead , we rely on data mixing as augmented samples , which can be treated as a smoothing regularization in unsupervised learning . In particular , we apply Cutmix ( Yun et al. , 2019 ) , a widely used data augmentation in supervised learning , where patches are cut and pasted among the positive pairs to generate new samples . Benefit from the center-wise sample selection , the Cutmix augmentation is only constrained within the local neighborhood of an image , and can be treated as an expansion of current neighborhood space . In this way , similar samples are pulled together in a smoother and robust way , which we find is beneficial for general representation . Furthermore , we propose multi-resolution augmentation , which aims at contrasting the same image ( patch ) at different resolutions explicitly , to enable the representation to be scale invariant . We argue that although previous operations such as crop and resize introduce multi-resolution implicitly , they do not compare the same patch at different resolutions directly . As comparisons , multi-resolution incorporates scale invariance into contrastive learning , and significantly boosts the performance even based on a strong baseline . The multi-resolution strategy is simple but effective , and can be combined with current data augmentations for further improving performance . We evaluate the feature representation on several self-supervised learning benchmarks . In particular , on ImageNet linear evaluation protocol , we achieve 75.5 % top-1 accuracy with a standard ResNet50 . In few shot setting , when finetuned with only 1 % labels , we achieve 59.3 % top-1 accuracy , surpassing previous works by a large margin . We also validate its transferring ability on several downstream tasks , and consistently outperform the fully supervised counterparts . 2 RELATED WORK . Unsupervised Representation Learning . Unsupervised learning aims at exploring the intrinsic distribution of data samples via constructing a series of pretext tasks without human labels . These pretext tasks take many forms and vary in utilizing different properties of images . Among them , one family of methods takes advantage of the spatial properties of images , typical pretext tasks include predicting the relative spatial positions of patches ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ) , or inferring the missing parts of images by inpainting ( Pathak et al. , 2016 ) , colorization ( Zhang et al. , 2016 ) , or rotation prediction ( Gidaris et al. , 2018 ) . Recent progress in self-supervised learning mainly benefits from instance discrimination , which regards each image ( and augmentations of itself ) as one class for contrastive learning . The motivation behind these works is the InfoMax principle , which aims at maximizing mutual information ( Tian et al. , 2019 ; Wu et al. , 2018 ) across different augmentations of the same image ( He et al. , 2020 ; Chen et al. , 2020a ) , ( Tian et al. , 2019 ) . Data Augmentation . Instance discrimination makes use of several data augmentations , e.g. , random cropping , color jittering , horizontal flipping , to define a large view set of vicinities for each image . As has been demonstrated ( Chen et al. , 2020a ; Tian et al. , 2020 ) , the effectiveness of instance discrimination methods strongly relies on the type of augmentations . Hoping that the network holds invariance in the local vicinities of each sample . However , current data augmentations are mostly constrained within a single image . An exception is ( Shen et al. , 2020 ) , where image mixture is used for flattened contrastive predictions . However , such mixture strategy is conducted among all images , which destroys the local similarity when contrasting mixed samples that are semantic dissimilar . Beyond self-supervised learning , mixing samples from different images is widely used to help alleviate overfitting in training deep networks . In particular , Mixup ( Zhang et al. , 2017 ) combines two samples linearly on pixel level , where the target of the synthetic image was a linear combination of one-hot labels . Following Mixup , there are a few variants ( Verma et al. , 2018 ) as well as a recent effort named Cutmix ( Yun et al. , 2019 ) , which combined Mixup and Cutout ( DeVries & Taylor , 2017 ) by cutting and pasting patches . 3 METHOD . In this section , we start by reviewing contrastive learning for unsupervised representation learning . Then we elaborate our proposed CLIM data augmentation , which targets at pulling similar samples via center-wise similar sample selection , followed by a cutmix data augmentation . We also present multi-resolution augmentation that we observe further improves the performance , as well as detailed analysis with recent methods that share similar targets with our method . 3.1 CONTRASTIVE LEARNING . Contrastive learning targets at training an encoder to map positive pairs to similar representations while pushing away the negative samples in the embedding space . Given unlabeled training set X = { x1 , x2 , ... , xn } . Instance-wise contrastive learning aims to learn an encoder fq that maps the samples X to embedding space V = { v1 , v2 , ... , vn } by optimizing a contrastive loss . Take the Noise Contrastive Estimator ( NCE ) ( Oord et al. , 2018 ) as an example , the contrastive loss is defined as : Lnce ( xi , x′i ) = −log exp ( fq ( xi ) · fk ( x′i ) /τ ) exp ( fq ( xi ) · fk ( x′i ) /τ ) + ∑K j=1 exp ( fq ( xi ) · fk ( x′j ) /τ ) ) , ( 1 ) where τ is the temperature parameter , and x′i and x ′ j denote the positive and negative samples of xi , respectively . The encoder fk can be shared ( Chen et al. , 2020a ; Caron et al. , 2020 ) or momentum update of the encoder fq ( He et al. , 2020 ) . 3.2 CLIM : CENTER-WISE LOCAL IMAGE MIXTURE . In contrastive learning , each sample as well as its augmentations is treated as a separate class , while all other samples are regarded as negative examples and pushed away . In principle , semantically similar samples should have similar feature representation in the embedding space , while current contrastive strategies do not consider the semantic similarities among different samples , and only choose different views of the same sample as positive pairs . To solve this issue , we propose a new kind of data augmentation , termed as Center-wise Local Image Mixture , which pulls samples that are semantically similar in an efficient and robust way . The proposed CLIM augmentation consists of two elements , i.e. , center-wise local similar sample selection , and a cutmix data augmentation , which would be described in details in the following . 3.2.1 CENTER-WISE LOCAL POSITIVE SAMPLE SELECTION . As noted by ( Wang & Isola , 2020 ) , a good representation should satisfy both alignment and uniformity , which encourages similar images to have similar representation in the embedding space , and meanwhile , semantically similar features are well-clustered . Towards this goal , we propose a positive sample selection strategy that considers both local similarity and global aggregation . This is achieved by searching similar samples within a cluster that the anchor sample belongs to , and only retaining samples that are closer to the corresponding cluster center . We denote it as center-wise local selection as these samples are picked out towards the cluster center among the local neighborhood of an image . In this way , similar samples are progressively pulled to the predefined cluster centers , while do not break the local similarity . Specifically , given a set of unlabeled images X= { x1 , x2 , ... , xn } and the corresponding embedding V = { v1 , v2 , ... , vn } with encoder fθ , where vi = fθ ( xi ) . We cluster the representations V using a standard k-means algorithm , and obtainm centers C = { c1 , c2 , ... , cm } . Given an anchor xi with its assigned cluster c ( xi ) ∈ C , denote the sample set that belongs to c ( xi ) as Ω1 = { x|c ( x ) = c ( xi ) } . We search the k nearest neighbors of xi over the entire space with L2 distance , obtaining sample set Ω2 = { xi1 , ... , xik } . The positive samples are selected based on the following rule : Ωp = { x|d ( fθ ( x ) , vc ( xi ) ) ≤ d ( fθ ( xi ) , vc ( xi ) ) , x ∈ Ω1 ∩Ω2 } , ( 2 ) where d ( · , · ) denotes the L2 distance of two samples , and vc ( xi ) denotes the feature representation of the corresponding cluster center , respectively . In this way , the samples are aggregated towards the predefined clusters , and meanwhile maintaining the local similarity . Our method combines the advantages of cluster and nearest neighbor methods . An illustration comparing the three methods is shown in Fig . 2 . Cluster-based method regards all samples that belong to the same center as positive pairs , which breaks the local similarity among samples especially when the anchor is around the boundary . While nearest neighbor-based method independently pulling samples of an anchor , and does not encourage the well-clustered goal . As a result , the embedding space is not highly concentrated among multiple similar anchors . As comparisons , by center-wise sample selection , similar samples are progressively pulled to the predefined center as well as considering the local similarity . In the experimental section , we would compare the performance of the three methods , and validate the superior performance of our proposed selection strategy .

Summary: The paper addresses the problem of contrastive representation learning, and proposes a new data augmentation, dubbed CLIM, that leverages similarity between images. Instead of generating positives pairs using different transformation of the same image -as it is standard in contrastive learning-, positive pairs are generated using those similar images to the anchor image: after clustering the representation space using k-means, the nearest neighbours that are closer to the corresponding center of the cluster where the anchor belongs to are selected. Then, positive pairs are constructed following Cutmix, which can be seen as a regulariser, and which consists in cutting and pasting patches among these pairs to generate new samples. Finally, the paper also proposes a multi-resolution augmentation, which consists in random zooms in (ie. random crop + resize) at different scales to enable scale invariance.