Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary 

The paper addresses the problem of contrastive representation learning, and proposes a new data augmentation, dubbed CLIM, that leverages similarity between images. Instead of generating positives pairs using different transformation of the same image -as it is standard in contrastive learning-, positive pairs are generated using those similar images to the anchor image: after clustering the representation space using k-means, the nearest neighbours that are closer to the corresponding center of the cluster where the anchor belongs to are selected. Then, positive pairs are constructed following Cutmix, which can be seen as a regulariser, and which consists in cutting and pasting patches among these pairs to generate new samples. Finally, the paper also proposes a multi-resolution augmentation, which consists in random zooms in (ie. random crop + resize) at different scales to enable scale invariance.


## Pros

The paper proposes a simple, yet effective, data augmentation method that seems to help learning stronger representations for some tasks.

The experimental section is good, which includes evaluations for different tasks and an ablation study that analyses the contribution of the different components of the proposed approach.


## Cons

I didn’t find that the proposed approach was properly motivated in the paper. After reading the paper I didn’t get a good understanding of why ensuring that the images are pulled towards the center of these clusters is a good property for learning good representations. Why is it better to pull samples towards the center instead of simply pulling them towards the nearest neighbours? I missed a more theoretical and extensive explanation about this point, which is basically the core idea of the proposed approach. The authors motivate this by saying that “it is better to encourage global aggregation property while pulling local similar samples”, which I couldn’t understand. I think the paper would benefit from an extended motivation where the authors could elaborate more on this.

The presentation and the writing could be improved.

While the improvements that the CLIM augmentation brings seem to be quite good in the linear evaluation on ImageNet by boosting the results of MoCo v2 (which is used as a baseline if I understood correctly) by 4 points, the improvement of this representation on other downstream tasks (ie. detection on PASCAL, COCO, LVIS) is minimal. Results are on par with MoCo v2 in all the scenarios (+0.4% in the best case), which means that the CLIM augmentation is not really bringing a significant improvement to the representation in these cases.

Related to the point above, the authors claim in the experiments on LVIS that “CLIM outperforms […] MoCo v2 by a large margin”. I have to disagree with this statement, since I personally consider these results to be on par: it only outperforms MoCo by 0.4/0.3 points in AP. I would suggest the authors to rephrase this claim.

Is MoCo v2 used as a baseline setting where CLIM augmentations are applied to? I understood this from the explanation in sections 4.1 and 4.3, but this is not clear for the semi-supervised training experiments in Section 4.2. If so, then what is the performance of MoCo v2 in semi-supervised training on ImageNet (table 2)? It would have been interesting to see what’s the improvement that CLIM brings rather than a direct comparison with other methods.


## Recommendation

My initial recommendation is leaning towards reject. Although the augmentation proposed is simple (in the good sense) and seems to work well (in some cases), I found that the paper somehow failed at motivating it, which I think it’s important for a paper like this. I also think results are a bit underwhelming for downstream tasks since they are on par with the baseline.
review_title: Would like to see a more extensive explanation of the motivation behind the method

final_score: rating: 6: Marginally above acceptance threshold
review_content: Center-wise Local Image Mixture For Contrastive Representation Learning

The paper introduces a new contrastive learning method for unsupervised representation learning. The main idea is to consider the semantic similarity between different images and incorporate it in the learning procedure, in contrast to the many contrastive learning methods which only used augmentations of the query image as positives.
The main contribution is 2-fold: a) use nearest neighbors from the same cluster which are closer to the centroid than the anchor as positive samples; b) use more complex augmentations, i.e. [CutMix] and multi-resolution during training. The proposed method achieves state-of-the-art results for unsupervised learning on Imagenet and transfer learning tasks on Pascal VOC, COCO, and LVIS.

### Pros

+ The paper is written well and easy to understand.
+ The method is simple and yet powerful, achieving state-of-the-art results on several standard benchmarks.
+ The method is easy to implement and reproduce.
+ The paper shows the importance of better modeling of intra-class variance by means of sampling positives among the nearest neighbors.
+ I appreciate quite detailed ablation studies (but not all of them).

### Cons
- The main ideas presented in the paper are not entirely new. 
  - The idea of using highly related images in the learned so far representation space as positives (vs single exemplar + its augmentations or just naive nearest neighbors) for unsupervised representation learning was already explored in [CliqueCNN]. 
  Since the proposed positive sampling method is the cornerstone contribution of this paper, it would be nice to see how it compares with the sampling method proposed in [CliqueCNN], where only the nearest samples which form a clique (mutually very similar samples) are used as positives.
  - [CutMix] augmentation is a previously published work. And the contribution of this paper is applying CutMix in the context of contrastive learning, which is yet another augmentation among a huge variety of possibilities. E.g., CutOut, MixUp, Attentive CutMix, etc. The paper gives no justification for why CutMix is especially better in this context.
  - Similar Multi-resolution augmentation and the importance of using different image resolutions during raining and testing were explored in [FixRes].

- p.4 *"Cluster-based method regards all samples that belong to the same center as positive pairs, which breaks the local similarity among samples especially when the anchor is around the boundary."* It is not very clear how the proposed method differs from the clustering-based methods in this sense. It seems like the proposed sampling method can also break local similarity around the cluster boundary because the anchor will be attracted to the cluster center increasing the distance to the samples from other clusters, which will result in very pronounced hubs in the representation space (as in Fig. 3). 
- It is not clear from the experiments what is the main performance booster compared to the KNN baseline in Tab. 6. Is it (A) the sampling of positives from the top 40 nearest neighbors within the cluster (as explained in A.2) or (B) discarding the positive which are further from the cluster centroid than the anchor?  The experiment in Sec. 4.4 do not answer this question, because the K-means baseline in Tab.6 randomly selects positive samples from the entire cluster and not from the top 40 closest samples within the cluster. To prove that (B) is crucial one would need to make an extra ablation study (A), where the positives are sampled from the top 40 nearest neighbors within the cluster (w/o discarding those which are further from the centroid than the anchor).
- *Minor*. Would be nice to see and extra ablation experiment on the full training schedule (1200 epochs) where all paper contributions are enabled one by one, e.g.: (i)  baseline, (ii) baseline + proposed positive sampling, (iii) baseline + proposed positive sampling + CutMix, (iv) baseline + proposed positive sampling + CutMix + multi-res.
I understand that it is computationally demanding, however, it would provide a better picture of the performance contribution of the final components (after tuning them using a shorter training schedule with 200 epochs), since improvements brought by some components on the shorter training schedule can become less significant when the network is trained longer.


## After rebuttal
After reading the authors comments' and other reviews, I think that this is a **borderline** paper that could benefit from more rigorous experimental validation.

[CutMix] CutMix: Regularization Strategy to Train Strong Classifierswith Localizable Features, ICCV 2019.  
[CliqueCNN] CliqueCnn: Deep unsupervised exemplar learning, Bautista et al., NeurIPS 2016.  
[CutOut] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout, 2017.  
[MixUp] Mixup: Beyond empirical risk minimization, Zhang et al., 2017.  
[Attentive CutMix]  Attentive CutMix: An Enhanced Data AugmentationApproach for Deep Learning Based ImageClassification, Walawalkar et al., 2020.  
[FixRes] Fixing the train-test resolution discrepancy, Touvron et al., NeurIPS 2019.

review_title: Simple method, very strong results, but not very novel and some justifications are missing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on contrastive learning for performing self-supervised network pre-training.  Two components are proposed: First, to select semantically similar images that are pulled together in the contrastive learning, the paper proposes "center-wise local image mixture" (CLIM) - both k-means clustering and knn neighbors are computed, and then for a given anchor image x, and images x' that fall within the same cluster and are a knn neighbor (and additionally closer to the cluster center than x) are selected as a positive match to x.  This is motivated from the perspective of allowing for consideration of both local similarity and global aggregation.  This selection is further modified by the use of cutmix data augmentation, where (x, x') are combined via a binary mask.  This is motivated from the perspective of allowing for some smoothing regularization to handle potentially noisy matches from CLIM.

The second component is multi-resolution data augmentation, a variant of crop augmentation that focuses specifically on enabling scale invariance by maintaining the same aspect ratio and performing the cutmix augmentation at different image resolutions.

Positives:
+ interesting proposed method for expanding the neighborhood space of considered positive matches for contrastive learning
+ ablation study provided to show the improvement from each proposed component (sample selection, cutmix, multi-resolution)
+ generally good empirical performance on several tasks - linear evaluation on imagenet, semi-supervised learning with few labels, transfer learning

Neutral:
- overall novelty is moderate; I would consider the main novelty to be in the selection of positive matches, as the cutmix and multi-resolution augmentations are largely leveraging existing ideas.  

Negatives:
- ablation studies only use 200 training epochs.  From appendix C, is it clear there is a big difference in accuracy from 200 to 800 or more epochs.  I would like to know how the improvements from the proposed methods still hold up after longer training.  
- hyper-parameter selection: there are several different parameters to be set in the proposed work: multi-resolution scales, number of clusters and neighbors for CLIM, alpha in cut-mix, with the differences in accuracy between settings approaching the difference between say knn+cutmix and center-wise+cutmix.  It appears that these hyper-parameters were directly set using the ImageNet linear evaluation, which seems like it has some potentially for overfitting then.  I would have liked to know how well the results generalize if these hyper-parameters are set on some separation validation data.
- as a more minor point, I'm curious how much the restriction in equation (2) matters for CLIM - in other words, what if instead of equation (2), simply let $\Omega_p = \Omega_1 \cap \Omega_2$?

Overall summary:

Given the overall improvements from the proposed method, I'd be inclined toward accept, if the concerns I raised regarding the empirical evaluation were addressed.
review_title: new data augmentations for improved contrastive learning; questions on empirical validation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an improved positive sample selection and data augmentation method for unsupervised, contrastive representation learning. Authors propose two improvements to be used in contrastive representation learning: a positive sample selection scheme (called center-wise sample selection), that improves over previously proposed kNN or k-means methods; and multi-resolution data augmentation which is an extension of a known crop augmentation technique with multiple scales.

Strengths:
The impact of the proposed improvements was thoroughly evaluated on several unsupervised feature learning benchmarks. It was shown that the proposed improvements advance state-of-the-art. Results on linear classification using features learned in unsupervised way; in few-shot learning task (using 1% of available labels) on  ImageNet and in transfer learning on VOC object challenge show small but consistent improvement over state-of-the-art.
A large number of recently proposed unsupervised representation learning approaches was included in the comparison. Ablation study proves positive impact of each of the two proposed improvements on the final performance.

Weaknesss:
The proposed improvements show rather limited improvement over state-of-the-art. E.g. from 75.3 (SwAV) to 75.5 (proposed method) on linear classification on ResNet using features learned in unsupervised way; 82.6 (SwAV) to 82.8 (proposed method) in transfer learning on VOC object chanllenge.

Wording and writing style needs to be revised as some sentences are difficult to understand.

 
In introduction authors write:
"....instance discrimination (Wu et al., 2018) based methods are rapidly closing the performance
gap comparing with the supervised counterparts" and in the following sentence:
"Following this paradigm, self-supervised models are able to generate features that are comparable or even better than those produced by supervised pretraining."
From the first sentence it sounds like unsupervised features still fall behind the features learned in a supervised way. The second sentence claims, that self-supervised models can be even better than those produces by supervised methods. But there's no reference to prove this claim.

"semantic similar images" sounds wrong. Rather it should be "semantically similar images".

"Contrastive learning targets at learning an encoder that is able to map positive pairs to similar representations
while push away those negative samples in the embedding space." does not sound right, especially usage of word "those", rephrase.

".... while current contrastive strategy does not consider the semantic similarities among different samples,
samples, which makes the optimization contradictory and hard for convergence. To solve this issue, we propose a new 
kind of data augmentation, termed as Center-wise Local Image Mixture...."
This is an overstatement. It sounds, like there's a serious problem with previous contrastive representation learning approaches and the proposed methods solves them. Whereas, experimental results prove, that the proposed method is only a small improvement over the state-of-the-art.
review_title: Interesting but small contribution over sota

Meta-review: There are two main contributions in this paper. First, the use of NN from the same cluster as “views” of the data as understood in classical contrastive learning. Second, the use of additional augmentation techniques, namely cutMix and multi-resolution. The reviewers noted that the paper is written well and easy to understand, that the ablation study is conducted well and that the model shows good empirical performance on several tasks. 

At the same time, the somewhat limited novelty of the paper was also discussed. As noted by R4, all aspects of the present paper have been discussed in previous work. The difference with previously published clustering-based SSL methods was also not very clear. This was discussed in the rebuttal but without strong evidence supporting the claims. Moreover, the ablation study is conducted on models that are trained for 200 epochs. While this is understandable from a pragmatic point of view, the conclusions may be completely different when the model is fully optimised. 

Because of all the points raised in the discussions, this paper is a too close to borderline to be accepted. We recommend the authors improve the manuscript given the feedback provided in the reviews and discussion and resubmit to another venue.
Decision: Reject