Center-wise Local Image Mixture For Contrastive Representation Learning
Authors: Hao Li
XIAOPENG ZHANG
Ruoyu Sun
Hongkai Xiong
Qi Tian
Venue:  (2021)
Keywords: Self-supervised Learning
Data Mixing
Contrastive Learning

Abstract: Recent advances in unsupervised representation learning have experienced remarkable progress, especially with the achievements of contrastive learning, which regards each image as well its augmentations as a separate class, while does not consider the semantic similarity among images. This paper proposes a new kind of data augmentation, named Center-wise Local Image Mixture, to expand the neighborhood space of an image. CLIM encourages both local similarity and global aggregation while pulling similar images. This is achieved by searching local similar samples of an image, and only selecting images that are closer to the corresponding cluster center, which we denote as center-wise local selection. As a result, similar representations are progressively approaching the clusters, while do not break the local similarity. Furthermore, image mixture is used as a smoothing regularization to avoid overconfident the selected samples. Besides, we introduce multi-resolution augmentation, which enables the representation to be scale invariant. Integrating the two augmentations produces better feature representation on several unsupervised benchmarks. Notably, we reach 75.5% top-1 accuracy with linear evaluation over ResNet-50, and 59.3% top-1 accuracy when fine-tuned with only 1% labels, as well as consistently outperforming supervised pretraining on several downstream transfer tasks.

Under review as a conference paper at ICLR 2021
CENTER-WISE LOCAL IMAGE MIXTURE FOR CON- TRASTIVE REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in unsupervised representation learning have experienced re- markable progress, especially with the achievements of contrastive learning. which regards each image as well its augmentations as separate class, while does not consider the semantic similarity among images. This paper proposes a new kind of data augmentation, named Center-wise Local Image Mixture, pand the neighborhood space of an image. CLIM encourages both local similarity and global aggregation while pulling similar images. This is achieved by searching local similar samples of an image, and only selecting images that are closer to the corresponding cluster center, which we denote center-wise local selection. As a result, similar representations are progressively approaching the clusters, while do not break the local similarity. Furthermore, image mixture is used as smooth- ing regularization to avoid overvonfidence on the selected samples. Besides, we introduce multi-resolution augmentation, which enables the representation be scale invariant. Integrating the two augmentations produces better feature repre- sentation on several unsupervised benchmarks. Notably, we reach 75.5% top-1 accuracy with linear evaluation over ResNet-50, and 59.3% top-1 accuracy when fine-tuned with only 1% labels, as well as consistently outperforming supervised pretraining several downstream transfer tasks.
INTRODUCTION
Learning general representations that can be transferable to different downstream tasks is a key challenge in computer vision. This usually achieved by fully supervised learning paradigm, e.g., making use of ImageNet labels for pretraining over the past several years. Recently, self-supervised learning has attracted more attention due to its free of human labels. In self-supervised learning, the network aims at exploring the intrinsic distributions of images via series of predefined pre- text tasks (Doersch et al. 2015) Gidaris et al., 2018 Noroozi & Favaro 2016 Pathak et al.) 2016). Among them, instance discrimination (Wu al. 2018) based methods have achieved remarkable
progress (Chen et al.) 2020a He et al. 2020: Grill et al. 2020 Caron et al.] (2020). The core idea of instance discrimination is push away different images, and encourage the representation of different transformations (augmentations) of the same image to be similar. Following this paradigm, self-supervised models are able to generate features that are comparable or even better than those
produced by supervised pretraining when evaluated on some downstream tasks, e.g, COCO detec- tion and segmentation (Chen al., 2020c b).
In contrastive learning, the positive pairs are simply constrained within different transformations of the same image, e.g., cropping, color distortion, Gaussian blur, rotation, etc.. Recent advances have demonstrated that better data augmentations (Chen et al., 2020a) really help to improve the representation robustness. However, contrasting two images that are de facto similar in semantic space is not applicable for general representations. It is intuitive to pull semantically similar images for better transfcrability. DeepCluster (Caron et al.] (2018) and Local Aggregation (Zhuang et al.) 2019) relax the extreme instance discrimination task via discriminating groups of images instead of an individual image. However, due to the lack of labels, it is inevitable that the positive pairs contain noisy samples, which limits the performance.
In this paper, we target at expanding instance discrimination by exploring local similarities among images. Towards this goal, one need solve two issues: how to select similar images as positive

---

Under review as a conference paper at ICLR 2021
pairs of an image, and ii) how to incorporate these positive pairs, which inevitably contain noisy assigmments, into contrastive learning. We propose a new kind of data augmentation, named Center- wise Local Image Mixture, to tackle the above two issues in robust and efficient way. CLIM consists of two core elements, L..., a center-wise positive sample selection, as well as a data mixing operation. For positive sample selection, the motivation is that good representation should be endowed with high intra-class similarity, and we find that although MoCo (He et al. (2200) does not explicitly model invariance to similar images, the intra-class similarity becomes higher as the training process goes. Based on this observation, we explicitly enforce semantically similar images towards the center of clusters, and generate representation with higher intra-class similarity, which we find is beneficial for few shot learning. This is achieved by searching nearest neighbors of an image, and only retaining similar samples that are closer to the corresponding cluster center, which we denote as center-wise local sample selection. As result, an image is pulled towards the center while do not break the local similarity.
Once similar samples are selected, direct way is to treat these similar samples as multiple posi- tives for contrastive learning. However, since feature representation in high dimensional space is complex, the returned positive samples inevitably contain noisy assignments, which should not be overconfident. Instead, we rely on data mixing as augmented samples, which can be treated as smoothing regularization in unsupervised learning. In particular, we apply Cutmix (Yun et al., 2019), a widely used data augmentation in supervised learning, where patches are cut and pasted among the positive pairs to generate new samples. Benefit from the center-wise sample selection, the Cutmix augmentation is only constrained within the local neighborhood of an image, and can be treated an expansion of current neighborhood space. In this way, similar samples are pulled together in a smoother and robust way, which we find is beneficial for general representation.
Furthermore, we propose mustiirrssottion augmentation, which aims at contrasting the same image (patch) at different resolutions explicitly, to enable the representation to be scale invariant. We argue that although previous operations such as crop and resize introduce multi-resolution implicitly, they do not compare the same patch at different resolutions directly. As comparisons, multi-resolution in- corporates scale invariance into contrastive learning, and significantly boosts the performance even based strong baseline. The multi-resolution strategy is simple but effective, and can be com- bined with current data augmentations for further improving performance.
We evaluate the feature representation several self supersupe learning benchmarks. In particular, on ImageNet linear evaluation protocol, we achieve 75.5% top-1 accuracy with a standard ResNet- 50. In few shot setting, when finetuned with only 1% labels, we achieve 59.3% top-1 accuracy, surpassing previous works by a large margin. We also validate its transferring ability on several downstream tasks, and consistently outperform the fully supervised counterparts.
2 RELATED WORK
Unsupervised Representation Learning. Unsupervised learning aims at exploring the intrinsic distribution of data samples via constructing a series of pretext tasks without human labels. These pretext tasks take many forms and vary in utilizing different properties of images. Among them, one family of methods takes advantage of the spatial properties of images, typical pretext tasks include predicting the relative spatial positions of patches (Doersch et al. 2015 Noroozi & Favaro 2016), or inferring the missing parts of images by inpainting (Pathak al. (016). colorization (Zhang et al. 2016). or rotation prediction (Gidaris et al., 2018). Recent progress in self-supervised learn- ing mainly benefits from instance discrimination, which regards each image (and augmentations of itself) as one class for contrastive learning. The motivation behind these works is the InfoMax principle, which aims at maximizing mutual information (Tian et al. 2019 Wu et al. 2018) across different augmentations of the same image (He et al., 2020; Chen al. 2020a). (Tian al. 2019).
Data Augmentation. Instance discrimination makes use of several data augmentations, e,g., random cropping, color jittering, horizontal flipping, to define large view set of vicinities for each image. As has been demonstrated (Chen et al., 2020a| Tian et al., 2020), the effectiveness of instance dis- crimination methods strongly relies the type of augmentations. Hoping that the network holds invariance in the local vicinities of each sample. However, current data augmentations are mostly constrained within single image. An exception is (Shen et al. 2020), where image mixture is used
2

---

Under review as a conference paper at ICLR 2021
Center-wise Local Image Mixture
Multi- Restii-Rsooltion
query 12tvd
Encoder
Positive
key.
Positive
Neighbar
Encoder
key_2 128d
Momentum Update
Figure An illustration of the proposed CLIM and multi-resolution data augmentations.
for flattened contrastive predictions. However, such mixture strategy is conducted among all images, which destroys the local similarity when contrasting mixed samples that are semantic dissimilar.
Beyond self- sulervupee learning, mixing samples from different images is widely used to help alle- viate overfitting in training deep networks. In particular, Mixup (Zhang et al. (017) combines two samples linearly on pixel level, where the target of the synthetic image was a linear combination of one-hot labels. Following Mixup, there are few variants (Verma et al.) 2018) as well as a recent effort named Cutmix (Yun et al. 2019), which combined Mixup and Cutout (De & Taylor 2017) by cutting and pasting patches.
METHOD
In this section, we start by reviewing contrastive learning for unsupervised representation learning. Then elaborate proposed CLIM data augmentation, which targets at pulling similar samples via center-wise similar sample selection, followed by a cutmix data augmentation. We also present multi-resolution augmentation that observe further improves the performance, well as detailed analysis with recent methods that share similar targets with our method.
3.1 CONTRASTIVE LEARNING
Contrastive learning targets training an encoder to map positive pairs similar representations while pushing away the negative samples in the embedding space. Given unlabeled training set X {I1, 32, K.... Instance-wise contrastive learning aims to learn an encoder fa that maps the samples X embedding space V {U1, U2 .... Un by optimizing contrastive loss. Take the Noise Contrastive Estimator (NCE) (Oord et al. 2018 as an example, the contrastive loss is defined
exp(ff(z)) fo(4)/77) (xp(f((()))f(()))))) ΣØ= (x)( fa(a()/r))
Ence (It; 24) -log
(I)
where T is the temperature parameter, and x/ and n denote the positive and negative samples æ;, respectively. The encoder fı be shared (Chen et al. 2020a Caron al.] 2020) momentum update of the encoder fa (He et al. 2020).
3.2 CLIM: CENTER- WENS LOCAL IMAGE MIXTURE
In contrastive learning, each sample as well as its augmentations is treated as separate class, while all other samples are regarded negative examples and pushed away. In principle, semantically similar samples should have similar feature representation in the embedding space, while current contrastive strategies do not consider the semantic similarities among different samples, and only choose different views of the same sample as positive pairs. To solve this issue, we propose new kind of data augmentation, termed Center-wise Local Image Mixture, which pulls samples that

---

Under review as a conference paper at ICLR 2021
Cluster center
Cluster neighhorhood
Nearest neighborhood
Center wise local neighborhood
Cluster
K means method
Knn method
CLIM method
Figure 2: Comparison of three positive sample selection strategies, i.e., k-means, knn, and the pro- posed center-wise local sample selection.
semantically similar in an efficient and robust way. The proposed CLIM augmentation consists of two elements, i.e., center-wise local similar sample selection, and a cutmix data augmentation, which would be described in details in the following.
3.2.1 CENTER- LOCAL POSITIVE SAMPLE SELECTION
As noted by (Wang& Isola 2020), a good representation should satisfy both alignment and uni- formity, which encourages similar images to have similar representation in the embedding space, and meanwhile, semantically similar features are well-clustered. Towards this goal, we propose a positive sample selection strategy that considers both local similarity and global aggregation. This is achieved by searching similar samples within a cluster that the anchor sample belongs to, and only retaining samples that closer to the corresponding cluster center. We denote it as center- wise local selection as these samples are picked out towards the cluster center among the local neighbor- hood of an image. In this way, similar samples are progressively pulled to the predefined cluster centers, while do not break the local similarity.
Specifically, given set of unlabeled images X X= T2,,.., ..., Fnn and the corresponding embedding V= ( u2,... J with encoder fa. where 0; = fo(x,). We cluster the representations V using standard k-means algorithm, and obtain m centers assigned cluster e(z;)) C, denote the sample set that belongs to c(2;) as Ω2 {x]c(<>) = c(x,) } We search the & nearest neighbors of 2; over the entire space with L2 distance, obtaining sample set 52 = {_iI .... J. J. The positive samples are selected based on the following rule:
{cI, C2 ..., Cm Given anchor x; with its
S2 = {x]d([f(())) S d(fo(())) Pe(m)), E 92 n 222)
(2)
where d(:,-) denotes the L2 distance of two samples, and Va(z,) denotes the feature representation of the corresponding cluster center, respectively. In this way, the samples are aggregated towards the predefined clusters, and meanwhile maintaining the local similarity.
Our method combines the advantages of cluster and nearest neighbor methods. An illustration com- paring the three methods is shown in Fig. 2 Cluster- based method regards all samples that belong to the same center as positive pairs, which breaks the local similarity among samples especially when the anchor is around the boundary. While nearest neighbor-based method independently pulling samples of an anchor, and does not encourage the well-clustered goal. As result, the embedding space not highly concentrated among multiple similar anchors. As comparisons, by center-wise sample selection, similar samples are progressively pulled to the predefined center as well as con- sidering the local similarity. In the experimental section, we would compare the performance of the three methods, and validate the superior performance of our proposed selection strategy.
3.2.2 CUTMIX DATA AUGMENTATION
Once we obtain the positive samples of an anchor, one direct way is to treat these samples similar as the augmented ones for contrastive learning. However, similarity computation in high dimensional space inevitably contains noisy samples, which should not be overconfident for contrasting, To solve this issue, we make use of data mixture strategy, which aims at mixing patches from two different images as augmented samples for contrasting. Data mixing is widely used in supervised learning as label smoothing regularization. The highlight is that without image level labels, we are not able to assign new labels to the augmented samples. Instead, only mixing samples that similar
4

---

Under review as a conference paper at ICLR 2021
in representation, and the mixed samples can be treated augmented version of the anchor. In this way, these mixed samples, as well as traditional data augmentations, can be pulled together in contrastive learning. Specifically, given positive pair (z,, we conduct data mixing follows:
Umic MO x, + (1- M) o Z
(3)
where M E [0, denotes binary mask indicating the mixed rectangle region of image, iee, where to cutout the region in and replaced with a randomly selected patch from T; and W, denotes the wide and height of an image, respectively. 1 is a binary mask filled with ones, and C is the element-wise multiplication operation. For mask M generation, we follow the setting in (unn et al., 2019). For the mixed sample Fmizz the positive sample can be either 2; or Tt and we reformulate the contrastive learning as combing two NCE loss:
Emiz Fi) A- Lncc (I 24) + +(1
-nnce (In Tz).
Where the combination ratio is sampled from beta distribution Beta(a, ai) with parameter a. The proposed data mixing augmentation can be seamlessly incorporated into current contrastive learning. The advantages are twofold: first, mixed samples help to expand the neighborhood space of current anchor sample for better representation; second, minimizing the two terms simultaneously can help to maximize the mutual information between 3; and ã. in a soft manner and perform as smoothing regularization the prediction for selected positive samples.
3.3 MULTI-RESOLUTION DATA AUGMENTATION
Data augmentation plays a key role in current contrastive learning, among them crop augmentation is one of the most effective way (Chen et al., 2020a). In typical crop augmentation, sample with size x W is randomly cropped with ratio a, and resized Krain x Ktrain as augmented samples, where Ktrain X Ktrain denotes the input resolution for model training. Hence the scaling factor w.r.t sample x can be described as:
1 Ktroin
(5)
O VH*W
For crop augmentation, the parameter Ktrain is fixed, and the crop ratio o is randomly selected among positive pairs. As a result, different crop augmentations usually contain different contents, which can be regarded as modeling occlusion invariance to some extent, where each crop sees one view of an image. In this section, we propose a simple but effective data augmentation strategy, named multi-resolution augmentation, which enables the representation to be scale invariant of an example. The highlight is that it is better for contrasting positive pairs with the same content but different resolutions. Specifically, for each positive we keep the crop ratio a fixed, and adjust K Ktrain to different resolutions for contrastive loss. An illustration is shown in Fig. Using multi-resolution, the objective function can be generalized as:
Linr
Σ EfT
Lmiz (2], 2
where (r1...,,r,) indicates the resolution set. In this way, the encoder would be encouraged to discriminate the positive samples with different resolutions from series of negative keys, which will maximize the mutual information between inputs with different resolutions and discard redundant information brought by resolutions.
Relation with Multi-crop Augmentation. There exist recent works that aim at improving crop augmentations, including multi-crop (Caron et al. 2020) and jigsaw-crop (Misra & Maaten 2020). However, both methods target at reducing crop ratio in Eq and resolution Keraān simultancously to bridge different parts of object, and do not explicitly model scale invariance. As comparisons, our proposed multi- resttiiresolttoon strategy fixes the crop ratio to explicitly model scale invariance. In the experimental section, would compare these augmentations validate the difference.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Top-1 accuracies under linear evaluation on ImageNet, using ResNet-50 encoder
Table 2:
Semi-supervised learning with few shot ImageNet labels, using ResNet- 50 encoder (averaged by 5 trials)
Method
Accuracy (%)
Top- TTpp-5
Supervised
76.5
Method
1% labels
10%
labels
Colorization (Zhagg et al.
2016
39.6
Supervised
25.4
48.4
56.4
56.4
Jigsaw
Noroozi & Favaro
2016
45.7
PIRL
30.7
57.2
60.4
83.8
NPID
Wu et al.
2018
54.0
SimCLR
48.3
75.5
65.6
87.8
LA Zhuang et al.
2019)
58.8
MoCo v2
52.4
78.4
65.3
86.6
MoCo
He et al.
2020
60.6
BYOL.
53.2
78.4
68.8
89.0
SeLa
YM. et al.
2020
61.5
SWAV
53.9
78.5
70.2
89.9
PIRL Misra Maaten
2020
63.6
SimCLRv2
57.9
82.5
68.4
89.2
CPCv2 (Hénaff et al.
2019
63.8
CLIM
59.3
81.6
70.0
89.3
PCL (Li et al.
2020
65.9
Table 3: Transfer learning on VOC object detection (averaged by 5 trials).
SimCLR Chen et al.
2020a)
70,0
MoCo v2 Chen et al.
2020c
71.1
Accuracy (%)
Method
SimCLRv2 (Chen al.
2020b
71.7
APso
AP7s
InfoMin Tian al.
2020
73.0
Supervised
81.4
58.8
BYOL Grill et al.
2020
74.3
MoCo v2
82.5
64.0
SwAV Caron et
2020
75.3
SWAV
82.6
CLIM
75.5
CLIM
82.8
64.5
Relation with Fix-Res. The proposed multi-resolution augmentation is reminiscent of recent work FixRes (Touvron et al. (019). which also explores resolution issue of better representation, but they are different in both motivation and goal. FixRes is based on the observation that data augmentations induce a significant discrepancy between the size of the objects seen by the classifier at train and test time, and employs different train and test resolutions to fix the train-test resolution discrepancy. The goal is to require less scale invariance for the neural net in FixRes, While our multi-resolution augmentation aims to model the scale invariance explicitly, which not carefully considered in previous supevvsd learning.
4 EXPERIMENTAL RESULTS
In this section, we assess our pretrained feature representation on several unsupervised benchmarks. We evaluate it on ImageNet under linear evaluation and semi-supervised settings. Then we transfer the learned features to different downstream tasks. We also analyze the performance of our rep- resentation with detailed ablation studies. For brief expression, except the ablation study, we denote our method CLIM, which includes two kinds of data augmentations.
4.1 LINEAR EVALUATION ON IMAGENET
The feature representation is trained based on ImageNet 2012 (Russakovsky al., 2015), using
standard ResNet-50 structure as backbone. We follow the setting in MoCo v2 (Chen et al. 2020c),
and the training details are listed in Appendix [ We first evaluate our features by training linear classifier on top of the frozen representation, following common protocol in (He et al. 2020 Tian jt al., 2019). For linear classifier, the learning rate is initialized as 30 and decayed by 0.1 after 60, 80 epochs, respectively. Table T shows the top-1 accuracies with center crop evaluation. Our method achieves an accuracy of 75.3%, surpassing MoCo v2 baseline (71.1%) by 4.4%, and nearly approaching the supervised learning baseline (76.5%).

---

Under review as a conference paper at ICLR 2021
Table 4: Transfer learning on COCO detection and instance segmentation (averaged by 5 trials)
Mask R-CNN R50-FPN,DC
Mask R-CNN.RSO-FPN.IISSe
Method
1x schedule
2x schedule
1x
schedule
2x schedule
Ap
AP
AP
Apd
AP
AP
AP
AP
AP75
Apm
AP
AP75
Supervised
38.9
59.6
42.0
40.6
61.3
44.4
35.4
56.5
38.1
36.8
58.1
39.5
MoCo v2
39.2
59.9
42.7
41.5
62.2
45.3
35.7
56.8
38.1
37.5
59.1
40.1
CLIM
39.5
60.0
43.3
41.8
62.3
45.7
35.8
57.0
38.6
37.7
59.4
40.5
4.2
SEMI-SUPERVISED TRAINING ON IMAGENET
We also evaluate method by fine-tuning the pretrained model with small subset of labels, following the semi- supervise settings in (Grill et al.] 2020; Korblith et al.) [019 CChn et al. 2020a Caron al. 2020). For fair comparisons, we the same fixed 1% and 10% splits of training data as in (Chen et al. 2020a), and fine-tune all layers using SGD optimizer with momentum of 0.9, and learning rate of 0.0001 for backbone, 10 for the newly initialized fe layer. The fine-tune epochs is set as 60, and the learning rate is decayed by 0.1 after every 20 epochs. During training, only random cropping and flipping data augmentations are used for fair comparison. The results are reported in Table CLIM achieves 59.3% top-1 accuracy with only 1% labels, and 70.0% with 10% labels. The performance gains are larger with 1% labels, e.g., 6.1% higher than BYOL, and 5.4% better than SwAV, which demonstrates that the proposed feature representation is mainly suitable for extremely few shot learning. Note that SimCLR v2 makes use of other tricks like more MLP layers for better performance, while our method simply adds one fc layer, and still achieves better performance under both settings.
4.3 DOWNSTREAM TASKS
We also evaluate our feature representation on several downstream tasks, including object detection and instance segmentation, evaluate the transferability of the learned features. For fair compari- son, all experiments follow MoCo settings.
PASCAL VOC Object Detection. Following the evaluation protocol in (He et al. 2020), we use Faster R-CNN (Ren et al.] (215)) with R50-C4 as backbone. We fine-tune all layers on the trainval set of VOC07+12 for 2x schedule and evaluate on the test set of VOC2007. We report the performances under the metric of AP50 and AP75. As shown in Table on PASCAL VOC, CLIM achieves 82.8% and 64.5% mAP under AP50 and AP75 metric, which is 1.4 points and 5.7 points higher than the fully supervised counterparts, and is slightly better than the results of MoCo v2.
COCO Object Detection and Instance Segmentation. We also evaluate the representation learned on large scale COCO dataset. Following (He et al.) 2020), we choose Mask R-CNN with FPN as backbone, and fine-tune all the layers on the train set and evaluate on the val set of COC02017. In Table 4 we report results under both 1x and 2x schedules. We show that CLIM consistently outperforms the supervised pretrained model and MoCo v2. Under 2X schedule, we achieve 41.8% and 37.7% detection and segmentation accuracies, respectively, which is 1.2 points and 1.1 points better than the supervised couterparts, and also slightly better than the highly optimized MoCo v2.
LVIS Long Tailed Instance Segmentation. Different from VOC and COCO where the number of training samples is comparable, LVIS is a long-tailed dataset, which contains more than 1200 categories, among them some categories only have less than ten instances. The main challenge is to learn accurate few shot models for classes among the tail of the class distribution, for which little data available. We evaluate our features on this long-tailed dataset to validate how the unsupervised representation boosts the performance. Similarly, we fine-tune the model (Mask R-CNN, R50-FPN) on the train set and evaluate on the val set of Lvis v0.5. Table 5 shows the result under 2 schedule. CLIM outperforms the supervised pretrained model by a large margin and is slightly better than MoCo v2. We claim that it is mainly to the proposed data mixing data augmentation, which is able to learn generalized representations even with extremely few labeled data.
7

---

Under review as a conference paper at ICLR 2021
Table 5: Transfer learning on LVIS long-tailed instance segmentation (averaged by 5 trials)
Object Det
Instance Seg
Method
AP
AP
AP
Ap
AP50
AP
Supervised
24.1
39,4
25.0
24.2
37.8
25.1
MoCo v2
25.1
40.4
26.1
25.3
38.4
27.0
CLIM
25.5
41.2
26.7
25.6
39.5
27.5
Table 6: Impact of different sample selection
Table 7: Impact of different multiple resolutions
Accuracy (%)
Strategy
Method
Resolution
Accuracy (%)
no mixing +cutmix
MoCo v2
67.5
Multi-Crop
2x224 2x96
69.7
Random
62.3
67.1
r, E (224, 96}
70.4
KNN
68.3
69.5
r, {224, 128)
71.7
K-means
68.0
69.2
Multi-Reso
r, e (224, 160)
72.3
KNN n K-means
68.5
69.6
Center-wise
69.3
70.1
r' E {224, 224)
71.4
4.4 ABLATION STUDY
In this section, we present ablation studies to better understand how each component affects the per- formance. Detailed comparisons include 1) positive sample selection, 2) cutmix data augmentation, and 3) multi-resolution augmentation. Unless specified, we train the model for 200 epochs over the ImageNet-1000 1000 and report the top-1 classification accuracy under linear evaluation protocol.
Positive Sample Selection. We first analyze the advantages of our proposed center-wise local sam- ple selection strategy. The compared sample selection alternatives include:
Random selection: Randomly select a sample from all unlabeled data.
KNN selection: Use k-nearest neighbors to build the correlation map among samples, and ran- domly select sample from the Top-k (k 10) nearest neighbors as positive samples.
K-means selection: Use k-means clustering algorithm to obtain i cluster centers, and randomly select sample from the corresponding cluster positive samples.
KNN n K-means selection: Use K-means clustering algorithm to obtain & cluster centers, and randomly select nearest neighbor within the cluster positive samples.
The results are shown in the second column of T In order the inspect the influence of sample selection, do not conduct cutmix augmentation, and these positive samples simply pulled via a standard contrastive loss. It can be shown that comparing with the MoCo baseline, both KNN and cluster-based sample selection boost the performance, and notably, simply selecting the union of knn and k-means achieves 68.5% accuracy, which is comparable with result that directly using knn. Since for samples lie around the boundary, it equals to knn, and does not encourage intra-class compactness, As comparison, our proposed center- wise selection strategy outperforms all the above selection methods.
Cutmix Data Augmentation. Data mixing helps to expand the neighborhood space of the target sample, and acts as smoothing regularization for the prediction. As shown in the third column of Table cutmix augmentation consistently improve the performance, comparing with directly pulling similar samples in contrastive loss, and achieve 70.1% accuracy with only 200 training epochs. Notably, with randomly selected positive samples, cutmix operation accuracy, slightly lower than the MoCo baseline, while significantly better than no mixing with only 62.3% accuracy. This can be attributed the smoothing regularization of cutmix, which is able to alleviate the effect of noisy samples and update model in a more robust way.
obtains 67.1 07.1%
Multiple Resolution. Based CLIM, further add multi resoluirsoo data augmentation to vali- date its effectiveness. The results of introducing different resolutions are shown Table 7 Using multiple resolutions setting with r, {224, 160), method achieves accuracy of 72.3% with
8

---

Under review as a conference paper at ICLR 2021
only 200 epochs, which surpasses the baseline of MoCo by 4.8%, and even much better than the results of MoCo with 800 epochs (71.1%).
multi-resolution augmentation with multi-crop augmentation proposed in (Caron et al. 2020). 2 x 224 + 2 x 96 denotes using two 224 x 224 crops with crop-scale (0.22 1.0) and two 96 96 crops with (0.05, 0.14), referring to (Caron et al. 2020 The main difference is that, the multi-crop strategy targets at capturing relationship between local and global information, while our proposed multiple resolution target enabling the encoder with scale invariance. We find that multi-crop slightly deteriorates the performance of CLIM (70.1% versus 69. 69),%) partially because data mixing behaves like image cropping augmentation, and shares similarity with multi-crop strategy.
We also compare
5 CONCLUSION
In this work, we proposed CLIM data augmentation, to efficiently pull semantically similar samples for better representation in contrastive learning. The main contributions of CLIM consist of two elements, center-wiss positive sample selection, which considers both local similarity and global aggregation property. In such way, similar samples are progressively aggregated to a series of prede- fined clusters, while not breaking the local similarity; and data mixing augmentation, which expands the neighborhood space of an example by mixing two images, and acts as a smoothing regularization for contrastive loss. Furthermore, which explicitly model scale invariance to further improve the representation. Experiments evalu- ated several unsupervised benchmarks demonstrate the effectiveness of our method.
present simple but effective multi-resolutie augmentation,
REFERENCES
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09880 09882, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 2002:22002.05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04299 2020c.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint ftti:::100.04444
2017.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference computer vision, pp- 1422-1430, 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803,0778 2017 2018.
Bran-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint Ntl:22006607333. 2020.
Kaiming He. Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729- 2020.

---

Under review as a conference paper at ICLR 2021
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09277 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2661-2671,
2019.
Junnan Li, Pan Zhou, Caiming Xiong. Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint ar arXiv:2005.0499 04966, 2020.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: / 807.00378, 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, and Trevor Darrell. Rethinking image mixture for unsupervised visual representation learning. arXiv preprint arXiv:2003.05433. 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:/906.000 05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isol.. What makes for good views for contrastive learning. arXiv preprint 2xXi:22005.0023 2020.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems, pp. 8252 8262, 2019.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. arXiv preprint arXiv: /06.00200 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprint arXiv:2005. 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin, Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7732-3742, 2018
Asano YM.. Rupprecht C., and Vedaldi A. Self-labelling via simultancous clustering and represen- tation learning. In International Conference on Learning Representations (ICLR), 2020.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE International Conference Computer Vision, pp. 6023-6032. 2019.
10

---

Under review as a conference paper at ICLR 2021
ZhanyiZhangg Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 2017.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2002-6012, 2019.

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
A.1 IMPLEMENTATION DETAILS FOR CONTRASTIVE PRETRAINING
Architecture and Optimization. We follow the setting in MoCo v2 (Chen al. 2020c), which relies on two encoders, one for training and the other one for momentum update (m = 0.999) to store negative keys. Following SimCLR (Chen et al. 2020a), we replace the head with 2-layer MLP to project the output of the final pooling layer to 128-d. We use SGD as optimizer, with weight decay setting as 0.0001 and the momentum as 0.9. We use a mini-batch size of 512 on 16 V100 GPUs with cosine learning rate schedule decayed from 0.06. We train the model for 1200 epochs, as we introducing data mixing augmentation, and usually requires more epochs for better performance as in supervised learning Yun et al. 2019).
Image Augmentations. We combine the proposed augmentations with previous widely used basic augmentation strategies, following the settings in (Chen et al.) 2020a He et al. 2020 The basic augmentations are listed below, as well as the corresponding parameters.
RandomResizedCrop: A crop of random size (from 0.2 to 1.0) of the original size and random aspect ratio (from 3/4 to 4/3) of the original aspect ratio is made.
RandomFlip: Randomly horizontally flip the image with probability of 0.5.
ColorJitter: Randomly change the brightness, contrast and saturation of an image.
RandomGrayssclle Randomly convert RGB image grayscale with probability of 0.2. RandomGausiianBuu:: Randomly blur the image with probability of 0.5. The radius is randomly sampled from 0.1 to 2.0.
A.2 DETAILS OF POSITIVE SAMPLE SELECTION
We implement k-means and knn by faiss (Johnson al. 2019). For efficiency, we perform clus- tering and knn computation every 5 epochs, since each iteration can be finished within minutes, the extra computation cost is marginal comparing with the budget for model training. The number of clusters is set as 10K, and we select the top 40 nearest neighbors in knn. In order to balance the contribution of each image, we randomly select 10 positive samples for the following cutmix augmentations. For situations where there remained no more than 10 examples (e.g., the anchor is already around the cluster center), we simply select the most nearest samples among the remained top-40 knn samples.
B MORE ABLATION STUDIES
This section gives more detailed analysis w.r.t. some hyperparameters. Unless specified, we train the model for 200 epochs over the ImageNet- 1000 and report the top-1 classification accuracy under linear evaluation protocol.
Table 8: Impact of the number of clusters mn and k of knn
Number of Clusters (m)
5000
10000
20000
knn (k)
20
40
60
20
40
60
20
40
60
Accuracy (%)
69.1
69.5
69.4
70.0
70.1
69.7
69.6
69.9
69.5
The number of Clusters m and the k in Knn. Here inspect the impact of the number of clusters m in k-means and the l in knn to analyze their effect on the performance. In order to ensure local similarity, we restrict the nearest neighbors within range from 20 to 60. The results for different clusters and top-k neighbors are shown in Table g
We observe that CLIM consistently improves
It is hard for fair comparison w.r.t training epochs, since different methods make use of different epochs and batchsize. egg., BYOL and SimCIR report results on 1000 epochs, while MoCo and SwaV are 800 epochs. We empirically find that for MoCo, more training epnchs do not improve the performance further.
12

---

Under review as a conference paper at ICLR 2021
the performance comparing the baseline Moco 67.5%, and is relatively robust to different and k. Notably, the best performance is achieved when m = 10000, 40.
Hyperparameters in Cutmix. The combination À in cutmix is sampled from the beta distribu- tion Beta(a, a),, where a plays an important role in data mixing augmentation, which controls the strength of interpolation between the anchor and its positive pair. Here inspect how different We find that the performance is relatively robust different a, and the best performance is achieved when a is 2.
E 1,,, 2, 2.5) affect the representation. As shown in Table 9
Table 9: Impact of in cutmix
1.0
1.5
2.0
2.5
Accuracy (%)
69.7
69.9
70.1
69.8
Ablation study on mixing strategies. Our method targets at generating new samples that expanding the neighborhood of an anchor. Here we compare performance of using mixup data augmentation, a widely used method in supervised settings. We try different choices of beta distribution for Mixup (Zhang al. 2017) and choose the best one (a 0.2) for comparison. Table shows that Cutmix performs better than Mixup, partially because mixup destroys the real pixel distribution (destroys the naturality of pixels).
Table 10: Ablation study on the mixing methods
Method
Accuracy (%)
Mixup
69.5
Cutmix
70.1
Extra ablation experiments for longer training schedule. We compare the improvements brought by different components of our proposed method for longer training schedule (800 epochs). Table shows the top-1 accuracies under linear evaluation protocol. Our method consistently outperforms the MoCo v2 baseline, which demonstrates the effectiveness of our proposed method.
Table 11: Ablation study on the longer training schedule
Method
Accuracy (%)
MoCo v2
71.1
Center-wise + cutmix
73.7
Center-wise cutmix Multi-reso
75.2
C MORE EXPERIMENTAL RESULTS
Visualization of Feature Representation. We visualize the feature space to better understand how CLIM augmentation pulls similar samples. Specifically, we randomly choose 10 classes from the validation set and provide the r-sne visualization of feature representation generated by CLIM, su- pervised training and MoCo v2. As shown Fig. 3 the same color denotes features with the same label. It can be shown that CLIM takes on higher aggregation property comparing with MoCo, and the fully supervised learned representation reveals the highest aggregation due to it makes use of im- age labels. Furthermore, we compute the intra-class similarity as the average cosine distance among all intra-class pairwise samples, and report the average similarity across 1000 classes, as shown in Table 12 CLIM achieves intra-class similarity of 0.65, which is much higher than that in MoCo v2 with similarity only 0.58, As comparison, we also list the result of supervised learning, with a similarity metric of 0.75..
13

---

Under review as a conference paper at ICLR 2021
Potole
MoCo V2
CLIM
Supervised
Figure 3: I-sne visualization of representation learned by MoCo, CLIM and supervised learning.
Table 12: Intra-class similarity for different models
Table 13: Results of different training epochs
Method
Intra-class Similarity
Epochs Accuracy (%)
Supervised 0.75
200
72.3
MoCo v2
0.58
800
75.2
CLIM
0.65
1200
75.5
compare CLIM trained with different epochs. Our method achieves an accuracy of 72.3% with only 200 epochs, 75.2% with 800 epochs, and can be further improved to 75.5% when training with 1200 epochs.
Results of Different Training Epochs. In Table 13
14


Reviews:
[{"review_id": "bbSNoZKpyRZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an improved positive sample selection and data augmentation method for unsupervised, contrastive representation learning. Authors propose two improvements to be used in contrastive representation learning: a positive sample selection scheme (called center-wise sample selection), that improves over previously proposed kNN or k-means methods; and multi-resolution data augmentation which is an extension of a known crop augmentation technique with multiple scales.\n\nStrengths:\nThe impact of the proposed improvements was thoroughly evaluated on several unsupervised feature learning benchmarks. It was shown that the proposed improvements advance state-of-the-art. Results on linear classification using features learned in unsupervised way; in few-shot learning task (using 1% of available labels) on  ImageNet and in transfer learning on VOC object challenge show small but consistent improvement over state-of-the-art.\nA large number of recently proposed unsupervised representation learning approaches was included in the comparison. Ablation study proves positive impact of each of the two proposed improvements on the final performance.\n\nWeaknesss:\nThe proposed improvements show rather limited improvement over state-of-the-art. E.g. from 75.3 (SwAV) to 75.5 (proposed method) on linear classification on ResNet using features learned in unsupervised way; 82.6 (SwAV) to 82.8 (proposed method) in transfer learning on VOC object chanllenge.\n\nWording and writing style needs to be revised as some sentences are difficult to understand.\n\n \nIn introduction authors write:\n\"....instance discrimination (Wu et al., 2018) based methods are rapidly closing the performance\ngap comparing with the supervised counterparts\" and in the following sentence:\n\"Following this paradigm, self-supervised models are able to generate features that are comparable or even better than those produced by supervised pretraining.\"\nFrom the first sentence it sounds like unsupervised features still fall behind the features learned in a supervised way. The second sentence claims, that self-supervised models can be even better than those produces by supervised methods. But there's no reference to prove this claim.\n\n\"semantic similar images\" sounds wrong. Rather it should be \"semantically similar images\".\n\n\"Contrastive learning targets at learning an encoder that is able to map positive pairs to similar representations\nwhile push away those negative samples in the embedding space.\" does not sound right, especially usage of word \"those\", rephrase.\n\n\".... while current contrastive strategy does not consider the semantic similarities among different samples,\nsamples, which makes the optimization contradictory and hard for convergence. To solve this issue, we propose a new \nkind of data augmentation, termed as Center-wise Local Image Mixture....\"\nThis is an overstatement. It sounds, like there's a serious problem with previous contrastive representation learning approaches and the proposed methods solves them. Whereas, experimental results prove, that the proposed method is only a small improvement over the state-of-the-art.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_XgKz5eL0Mz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary \n\nThe paper addresses the problem of contrastive representation learning, and proposes a new data augmentation, dubbed CLIM, that leverages similarity between images. Instead of generating positives pairs using different transformation of the same image -as it is standard in contrastive learning-, positive pairs are generated using those similar images to the anchor image: after clustering the representation space using k-means, the nearest neighbours that are closer to the corresponding center of the cluster where the anchor belongs to are selected. Then, positive pairs are constructed following Cutmix, which can be seen as a regulariser, and which consists in cutting and pasting patches among these pairs to generate new samples. Finally, the paper also proposes a multi-resolution augmentation, which consists in random zooms in (ie. random crop + resize) at different scales to enable scale invariance.\n\n\n## Pros\n\nThe paper proposes a simple, yet effective, data augmentation method that seems to help learning stronger representations for some tasks.\n\nThe experimental section is good, which includes evaluations for different tasks and an ablation study that analyses the contribution of the different components of the proposed approach.\n\n\n## Cons\n\nI didn’t find that the proposed approach was properly motivated in the paper. After reading the paper I didn’t get a good understanding of why ensuring that the images are pulled towards the center of these clusters is a good property for learning good representations. Why is it better to pull samples towards the center instead of simply pulling them towards the nearest neighbours? I missed a more theoretical and extensive explanation about this point, which is basically the core idea of the proposed approach. The authors motivate this by saying that “it is better to encourage global aggregation property while pulling local similar samples”, which I couldn’t understand. I think the paper would benefit from an extended motivation where the authors could elaborate more on this.\n\nThe presentation and the writing could be improved.\n\nWhile the improvements that the CLIM augmentation brings seem to be quite good in the linear evaluation on ImageNet by boosting the results of MoCo v2 (which is used as a baseline if I understood correctly) by 4 points, the improvement of this representation on other downstream tasks (ie. detection on PASCAL, COCO, LVIS) is minimal. Results are on par with MoCo v2 in all the scenarios (+0.4% in the best case), which means that the CLIM augmentation is not really bringing a significant improvement to the representation in these cases.\n\nRelated to the point above, the authors claim in the experiments on LVIS that “CLIM outperforms […] MoCo v2 by a large margin”. I have to disagree with this statement, since I personally consider these results to be on par: it only outperforms MoCo by 0.4/0.3 points in AP. I would suggest the authors to rephrase this claim.\n\nIs MoCo v2 used as a baseline setting where CLIM augmentations are applied to? I understood this from the explanation in sections 4.1 and 4.3, but this is not clear for the semi-supervised training experiments in Section 4.2. If so, then what is the performance of MoCo v2 in semi-supervised training on ImageNet (table 2)? It would have been interesting to see what’s the improvement that CLIM brings rather than a direct comparison with other methods.\n\n\n## Recommendation\n\nMy initial recommendation is leaning towards reject. Although the augmentation proposed is simple (in the good sense) and seems to work well (in some cases), I found that the paper somehow failed at motivating it, which I think it’s important for a paper like this. I also think results are a bit underwhelming for downstream tasks since they are on par with the baseline.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PmHb63JS32u", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Center-wise Local Image Mixture For Contrastive Representation Learning\n\nThe paper introduces a new contrastive learning method for unsupervised representation learning. The main idea is to consider the semantic similarity between different images and incorporate it in the learning procedure, in contrast to the many contrastive learning methods which only used augmentations of the query image as positives.\nThe main contribution is 2-fold: a) use nearest neighbors from the same cluster which are closer to the centroid than the anchor as positive samples; b) use more complex augmentations, i.e. [CutMix] and multi-resolution during training. The proposed method achieves state-of-the-art results for unsupervised learning on Imagenet and transfer learning tasks on Pascal VOC, COCO, and LVIS.\n\n### Pros\n\n+ The paper is written well and easy to understand.\n+ The method is simple and yet powerful, achieving state-of-the-art results on several standard benchmarks.\n+ The method is easy to implement and reproduce.\n+ The paper shows the importance of better modeling of intra-class variance by means of sampling positives among the nearest neighbors.\n+ I appreciate quite detailed ablation studies (but not all of them).\n\n### Cons\n- The main ideas presented in the paper are not entirely new. \n  - The idea of using highly related images in the learned so far representation space as positives (vs single exemplar + its augmentations or just naive nearest neighbors) for unsupervised representation learning was already explored in [CliqueCNN]. \n  Since the proposed positive sampling method is the cornerstone contribution of this paper, it would be nice to see how it compares with the sampling method proposed in [CliqueCNN], where only the nearest samples which form a clique (mutually very similar samples) are used as positives.\n  - [CutMix] augmentation is a previously published work. And the contribution of this paper is applying CutMix in the context of contrastive learning, which is yet another augmentation among a huge variety of possibilities. E.g., CutOut, MixUp, Attentive CutMix, etc. The paper gives no justification for why CutMix is especially better in this context.\n  - Similar Multi-resolution augmentation and the importance of using different image resolutions during raining and testing were explored in [FixRes].\n\n- p.4 *\"Cluster-based method regards all samples that belong to the same center as positive pairs, which breaks the local similarity among samples especially when the anchor is around the boundary.\"* It is not very clear how the proposed method differs from the clustering-based methods in this sense. It seems like the proposed sampling method can also break local similarity around the cluster boundary because the anchor will be attracted to the cluster center increasing the distance to the samples from other clusters, which will result in very pronounced hubs in the representation space (as in Fig. 3). \n- It is not clear from the experiments what is the main performance booster compared to the KNN baseline in Tab. 6. Is it (A) the sampling of positives from the top 40 nearest neighbors within the cluster (as explained in A.2) or (B) discarding the positive which are further from the cluster centroid than the anchor?  The experiment in Sec. 4.4 do not answer this question, because the K-means baseline in Tab.6 randomly selects positive samples from the entire cluster and not from the top 40 closest samples within the cluster. To prove that (B) is crucial one would need to make an extra ablation study (A), where the positives are sampled from the top 40 nearest neighbors within the cluster (w/o discarding those which are further from the centroid than the anchor).\n- *Minor*. Would be nice to see and extra ablation experiment on the full training schedule (1200 epochs) where all paper contributions are enabled one by one, e.g.: (i)  baseline, (ii) baseline + proposed positive sampling, (iii) baseline + proposed positive sampling + CutMix, (iv) baseline + proposed positive sampling + CutMix + multi-res.\nI understand that it is computationally demanding, however, it would provide a better picture of the performance contribution of the final components (after tuning them using a shorter training schedule with 200 epochs), since improvements brought by some components on the shorter training schedule can become less significant when the network is trained longer.\n\n\n## After rebuttal\nAfter reading the authors comments' and other reviews, I think that this is a **borderline** paper that could benefit from more rigorous experimental validation.\n\n[CutMix] CutMix: Regularization Strategy to Train Strong Classifierswith Localizable Features, ICCV 2019.  \n[CliqueCNN] CliqueCnn: Deep unsupervised exemplar learning, Bautista et al., NeurIPS 2016.  \n[CutOut] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout, 2017.  \n[MixUp] Mixup: Beyond empirical risk minimization, Zhang et al., 2017.  \n[Attentive CutMix]  Attentive CutMix: An Enhanced Data AugmentationApproach for Deep Learning Based ImageClassification, Walawalkar et al., 2020.  \n[FixRes] Fixing the train-test resolution discrepancy, Touvron et al., NeurIPS 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "B_JMJWpYaCW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on contrastive learning for performing self-supervised network pre-training.  Two components are proposed: First, to select semantically similar images that are pulled together in the contrastive learning, the paper proposes \"center-wise local image mixture\" (CLIM) - both k-means clustering and knn neighbors are computed, and then for a given anchor image x, and images x' that fall within the same cluster and are a knn neighbor (and additionally closer to the cluster center than x) are selected as a positive match to x.  This is motivated from the perspective of allowing for consideration of both local similarity and global aggregation.  This selection is further modified by the use of cutmix data augmentation, where (x, x') are combined via a binary mask.  This is motivated from the perspective of allowing for some smoothing regularization to handle potentially noisy matches from CLIM.\n\nThe second component is multi-resolution data augmentation, a variant of crop augmentation that focuses specifically on enabling scale invariance by maintaining the same aspect ratio and performing the cutmix augmentation at different image resolutions.\n\nPositives:\n+ interesting proposed method for expanding the neighborhood space of considered positive matches for contrastive learning\n+ ablation study provided to show the improvement from each proposed component (sample selection, cutmix, multi-resolution)\n+ generally good empirical performance on several tasks - linear evaluation on imagenet, semi-supervised learning with few labels, transfer learning\n\nNeutral:\n- overall novelty is moderate; I would consider the main novelty to be in the selection of positive matches, as the cutmix and multi-resolution augmentations are largely leveraging existing ideas.  \n\nNegatives:\n- ablation studies only use 200 training epochs.  From appendix C, is it clear there is a big difference in accuracy from 200 to 800 or more epochs.  I would like to know how the improvements from the proposed methods still hold up after longer training.  \n- hyper-parameter selection: there are several different parameters to be set in the proposed work: multi-resolution scales, number of clusters and neighbors for CLIM, alpha in cut-mix, with the differences in accuracy between settings approaching the difference between say knn+cutmix and center-wise+cutmix.  It appears that these hyper-parameters were directly set using the ImageNet linear evaluation, which seems like it has some potentially for overfitting then.  I would have liked to know how well the results generalize if these hyper-parameters are set on some separation validation data.\n- as a more minor point, I'm curious how much the restriction in equation (2) matters for CLIM - in other words, what if instead of equation (2), simply let $\\Omega_p = \\Omega_1 \\cap \\Omega_2$?\n\nOverall summary:\n\nGiven the overall improvements from the proposed method, I'd be inclined toward accept, if the concerns I raised regarding the empirical evaluation were addressed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There are two main contributions in this paper. First, the use of NN from the same cluster as “views” of the data as understood in classical contrastive learning. Second, the use of additional augmentation techniques, namely cutMix and multi-resolution. The reviewers noted that the paper is written well and easy to understand, that the ablation study is conducted well and that the model shows good empirical performance on several tasks. 

At the same time, the somewhat limited novelty of the paper was also discussed. As noted by R4, all aspects of the present paper have been discussed in previous work. The difference with previously published clustering-based SSL methods was also not very clear. This was discussed in the rebuttal but without strong evidence supporting the claims. Moreover, the ablation study is conducted on models that are trained for 200 epochs. While this is understandable from a pragmatic point of view, the conclusions may be completely different when the model is fully optimised. 

Because of all the points raised in the discussions, this paper is a too close to borderline to be accepted. We recommend the authors improve the manuscript given the feedback provided in the reviews and discussion and resubmit to another venue.
Decision: Reject