Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new contrastive attraction and contrastive repulsion (CACR) loss for self-supervised representation learning. By formulating two conditional distributions, CACR considers different importance of positive/negative samples according to their distances to the query sample. Experimental results show superior performance of CACR against traditional contrastive loss and state-of-the-art methods.
main_review: Strengths:
1. This paper is well organized and easy to understand.
2. The idea of CACR for self-supervised representation learning is reasonable and shows promising performance.
3. The experiments and analysis are sufficient.

Weaknesses:
1. It’s not clear why the conditional distributions should be like Eq. 3&4. Are there any theories? For example, why the probability of “moving it to a positive sample” is high when query is far from its positive pair? More explanations are needed.
2. It shows that increasing number of positive samples (number of augmentations, controlled by hyperparameter “K”) benefits the performance. It would be better to know the performance curve in a large range of K. Currently only K=1,4 are evaluated.

summary_of_the_review: This paper is generally well motivated and well written. The experiments are sufficient and the proposed CACR shows impressive performance. My minor concern is better explanations of the conditional distributions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to improve the conventional contrastive learning (CL) by introducing a CACR loss, which takes the distance-based weights into the consideration for CL loss computation. Theoretical analysis is provided on the robustness property of the representation learned by CACR. Experiments are also presented to evaluate the performance of CACR compared with MoCo V2, SimCLR and other recent CL methods. 
main_review: Strength: 

1. The proposed method is intuitively correct. 

2. The experiments demonstrate the proposed method works better than SimCLR and MoCo V2. 

Weakness:

1. The novelty is limited. 

2. The paper is not very easy to follow. 

3. From the experiments, the performance advantage of the proposed method over existing CL methods (like BYOL) is not very significant. The compared baseline is not always consistent. The proposed method is compared against SimCLR in some experiments and MoCo in other experiments. 
summary_of_the_review: Overall, I feel this paper makes the presentation of the proposed method over-complicated and the resulted approach has limited novelty. The experiment is not very solid and convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present a new contrastive representation learning method. Specifically, a doubly contrastive learning (CL) approach is proposed, in which it contrasts positive and negative samples within themselves separately. The proposed method is termed "contrastive attraction and contrastive repulsion (CACR)". Extensive experimental, as well as theoretical analyses, show the effectiveness and robustness of the proposed method. The main contributions of this work are the proposed CACR strategy and the corresponding experimental analysis.
main_review: ### Strengths

\+ The authors provide a theoretical analysis of the proposed CACR.

\+ Extensive experimental analysis validate the effectiveness of the proposed method.

\+ The proposed CACR is shown to be more robust compared to other existing methods, when evaluated on label-imbalanced data.

\+ The writing of the paper is generally good.

### Weaknesses

\- The main contribution of this paper is the "double-contrast" strategy. Compare to the traditional contrastive loss, the novelty is a bit limited. Besides, it is unclear why such a "double-contrast" is necessary, and why have to use the multiplication weight as a selection approach instead of others.

\- The proposed method is motivated by the conventional CL's robustness on uncurated datasets, as claimed by the authors. But the experimental analysis does not validate this well. The "wild" uncurated data is not only data-imbalance, it is also related to the quality and noise level of the data. As a result, suggest the authors either tone down the claim or provide more convincing justifications, either empirical or theoretical.

\- Although the authors demonstrate the effectiveness of the proposed method in a controlled experiment (Sec. 5.1), when compared to SOTAs (Sec. 5.2), the performance is not significant, just comparable/incremental to prior works.

\- It would be better if the authors could present an analysis of the add on computational complexity / number of parameters, by using the proposed CACR strategy.

\- The proposed method is also related to (though not exactly the same scenario) the hard-negative and hard-positive sampling used in video-audio representation learning (e.g. [*1, *2, *3]) and it would be better to discuss the relationships.

[*1] "Cooperative learning of audio and video models from self-supervised synchronization", NeurIPS 2018

[*2] "Self-supervised contrastive video-speech representation learning for ultrasound", MICCAI 2020

[*3] "Spoken moments: Learning joint audio-visual representations from video descriptions", CVPR 2021.
summary_of_the_review: Although the proposed method seems to be an incremental improvement to the conventional contrastive learning, the "doubly contrastive strategy" is shown to be effective by performing extensive experimental evaluations. This might be of interest to a group of audiences in ICLR 2022 and contribute to the community. On the other hand, there are some unclear statements I hope the authors could address in rebuttal. As a result, I recommend a marginal acceptance at the current stage and may change my rating after rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Contrastive Attraction and Contrastive Repulsion (CACR) loss for self-supervised learning with a doubly contrastive strategy. It constructs two conditional distributions to model the importance of positive samples and negative samples according to their distances to the query. The importance, which is reflected by weights, guides the query to not only more strongly pull more distant positive samples, but also more strongly push away closer negative samples. Theoretical analysis and empirical results show that CACR generalizes the conventional CL loss and is more robust in more general cases. Superior performances are achieved on both curated and uncurated datasets, and on various downstream tasks.
main_review: Strength:
1. CACR models the importance of positive/negative samples, making the optimization more effective.	
2. CACR is more robust than conventional contrastive loss, according to the performance on label-imbalanced datasets.
3. Pretraining representations of CARC consistently perform better than conventional CL methods when finetuned on downstream tasks.
4. Theoretical analysis, empirical results and ablation studies are sufficient to support its effectiveness.

Weakness:
There are some minor problems.
1.	I can’t find the definition of \delta in eq. (5).
2.	In the last sentence on page 1, why both easy and hard negative samples are desired for contrast? What’s the reference?

summary_of_the_review: Though the proposed framework is closely related to previous CL methods on the ideas of multi-crop and hard negatives, it still shows novelty on modeling the intra-relation among positives/negatives. The paper provides detailed theoretical analysis for understanding the method. Experiments and ablation studies are also sufficient to demonstrate its effectiveness.
review_title: null

Meta-review: The submission receives mixed ratings initially. Reviewer WK7k stays positive, edUG and qbmC are borderline, and MXU1 is negative. They raise several issues including limited technical contribution, insufficient sota experimental comparison, and detailed theoretical proof. The authors have responded to these issues in the rebuttal. The final ratings of these reviewers do not alter. 

After checking the revised manuscript, all the reviews and responses, the AC feels the review from MXU1 lacks sufficient details and the claim shall be better supported with evidence. On the other hand, the issue of the technical contribution raised by edUG still exists. The modification upon original CL is marginal. Also, the experimental validation, although tested in several configurations, only includes ResNet 50 as the encoder backbone. The sufficient validation upon different scales of CNNs are common strategies used in sota SSL methods (e.g., MoCo, BYOL, SimCLR). Without a thorough evaluation, the effectiveness of CACR on different scales of CNN backbones is in doubt. Meanwhile, the experimental comparison upon sota methods (e.g., BYOL, SwAV+multi crop) does not clearly show the performance advantages of CACR. The small-scale dataset utilized for ablation study (i.e., CIFAR) is not as convincing as ImageNet. 

Overall, the AC feels that the marginal technical contribution is acceptable, but it shall be equipped with thorough experimental validation and thus serves as a fundamental baseline (i.e., potentially benefits the SSL community). Based on the current form, the authors are suggested to further improve the current submission and are welcome to submit for the next venue.
Decision: Reject