Contrastive Attraction and Contrastive Repulsion for Representation Learning

Abstract: Contrastive learning (CL) methods effectively learn data representations without label supervision, where the encoder needs to contrast each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on ImageNet, a well-curated dataset with balanced image classes. However, they tend to yield worse performance when pretrained on images in the wild. In this paper, to further improve the performance of CL and enhance its robustness on uncurated datasets, we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples. Theoretical analysis reveals that CACR generalizes CL's behavior by positive attraction and negative repulsion. It further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. Extensive large-scale experiments on standard vision tasks show that CACR not only consistently outperforms existing CL methods on benchmark datasets in representation learning, but also shows better robustness when generalized to pretrain on wild large image datasets.

Reviews and author responses:

 Thanks to the authors for their response. After reading the response from the authors as well as comments from other reviewers, most of my concerns are addressed while the novelty is still a bit limited. As a result, I would keep my original rating.

 The authors' responses have addressed my concerns. 
It's impressive to see that the propsed method can achieve even better performance with larger positive sampling siz, as shown in Fig. 9.

I would like to keep my rating.

 After reading other reviews and authors’ feedback, I keep my original score.

 Dear Reviewer MXU1,

Thank you again for your comments. We were wondering whether you have had the chance to read our response, which we believe has addressed all your concerns. We appreciate it if you could let us know if you have any further comments or suggestions.

Best,
Authors of Paper2940

 Thank you for your comments and questions, in the following we provide our response to address your questions.


> **Definition of $\delta$ in Eqn (5).**

Thanks for your suggestion. $\delta$ in the paper refers to the Dirac delta function and we use it to represent the empirical distribution. Following your comments, we re-formalize it into $\hat p(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M \delta(\mathbf{x}-\mathbf{x}_i)$ to ensure the rigorous expression and add corresponding explanation in the text. Please check our revision on Page 5 (marked in blue).

------

> **Why both easy and hard negative samples are desired for contrast?**

We aim to point out that only using easy or hard negative samples could result in undesired results. As suggested in various papers such as [Ref1], using neither too easy nor too hard negative samples has benefits for the contrasts. We have added corresponding references and revised that phrase to make sure the expression would not lead to misunderstanding.

[Ref1] Mike Wu,  Milan Mosse,  Chengxu Zhuang,  Daniel Yamins,  and Noah Goodman.   Conditionalnegative sampling for contrastive learning of visual representations. In International Conference on Learning Representations, 2021.





 Thank you for your comments and questions, to which we provide our point-to-point response in what follows.

> **The novelty of  doubly-contrast**

Please refer to “Response to common questions.”

----------

> **Why do we need doubly-contrast and why in this form?**

Different positive and negative samples provide different information and thus they can be more or less informative to the model. This motivates us to first have an intra-contrast within positive and negative samples respectively to make sure we can leverage the information sufficiently for the inter-contrast between positive and negative samples. This mechanism boosts the efficiency of the training, especially for imbalanced data. Empirically, we showed in Appendix B.2 (in Table 9) that without the intra-contrast, the model is much less effective. 


Please also refer to “Response to common questions” and Page 17,19 of the revised paper for more detailed discussion about the definition of the conditional distribution in intra-contrast. 


-------

> **Definition of uncurated datasets.**

Thank you for pointing this out and we agree with you that it is better to make the definition of uncurated data clear. Thus we narrow our claim to that CACR is more robust on datasets with label imbalance. We leave the testing of CACR on wilder uncurated datasets that suffer from noise or other types of contaminations to future study.

---------

> **Comparison with SOTAs**

Please refer to “Response to common questions.”, where we re-clarify that CACR achieves significant performance in both linear classification and transfer learning on detection/segmentation.  

---------

> **Computational complexity**

We have provided the corresponding computation complexity of baselines and CACR(K=1,2,3,4) on ImageNet in Table 16 (Appendix B.4). Additionally, we add more results of computation complexity by varying K on CIFAR-10 Table 11. 

---------

> **Related references**

Thanks for pointing them out. We have added and discussed the missed references in the new latest revision.


 Thank you for your questions. Please find our response to your questions below.

> **The novelty is limited and CACR compare with CL methods (like BYOL) is not very significant.**

Please refer to “Response to common questions,” where we have made a clarification about the novelty and the significance of the experiment part.  

-------


> **The paper is not very easy to follow.**

We appreciate it if you could be more specific about the parts that confuse the readers so that we can make improvements accordingly. 

------

> **The comparison with baselines is not consistent: the proposed method is compared against SimCLR in some experiments and MoCo in other experiments.**

Regarding SimCLR and MoCo, they are both compared in the large-scaled datasets (Tables 3 & 5). In these small datasets, the convention is not to compare with MoCo since it is designed for large-scale datasets. Please note we mainly aim to compare with baselines from the perspective of the objective function. SimCLR, MoCov2 and CMC all use InfoNCE loss (Eqn 1 in the paper). Thus in our analysis part, we are comparing CACR loss with InfoNCE (SimCLR and/or MoCo), AU-CL and HN-CL. 

On the imbalanced datasets, Table 4 does not involve the results of SimCLR since there are no existing results on such a setting. To make our comparison fair in this case, we run the MoCov2 framework with both the InfoNCE and CACR losses.


 Thank you for your comments and questions. Please find our response below.

>  **The definition of conditional probability**

The conditional probability here leverages a Bayesian strategy and an energy form to describe the dependency between the query and positive/negative samples. The details are presented in “Response to common questions.” Please refer to that part.

> **Results of varying number of positive pairs (K=2, 3, etc.)**

We have added the corresponding plots (Fig. 9 in the revision) in Appendix B.3 to show CACR results with more values of K, and also provided the corresponding computation complexity in Table 11 and 16 (appendix B.4). Thanks for your suggestions!


 ## Response to common questions:
We thank all reviewers for their valuable feedback. We first address two common questions:


> **The novelty of our paper and the improvement compared with SOTAs (e.g. BYOL) is not significant** (From Reviewer MXU1 and Reviewer edUG).

We would like first to make the following connections and distinctions in order to clarify the novelty of this paper:

Our paper studies the principles in contrastive loss (InfoNCE), as in our Eqn 1. Compared with the contrast between positive and negative samples, CACR provides an intra-contrast within positive samples and negative samples, respectively. This intra-contrast provides better efficiency and robustness to improve the original contrastive loss, where we provide detailed theoretical analysis and support them with extensive experiment results.

Theoretically, we formalize the intra-contrast with conditional probabilities and in Section 4 we demonstrate the properties of these conditional probabilities. The positive conditional distribution plays a role of the importance sampling distribution to make the learning more efficient. The negative conditional distribution complementarily takes the mutual information between query and negative samples into account to make the learning more robust.
 
Empirically, we show that CACR improves contrastive loss to achieve better results in linear classification when pretrained on both small-scale and large-scale datasets, as well as on label-imbalanced datasets. When compared with SOTAs, although recent non-contrastive methods that without using negative samples achieve better results than conventional contrastive methods in linear classification, CACR performs similarly to (or is even better than) them. On transferring tasks like object detection/segmentation, CACR achieves the best results among these baseline methods. Following the suggestion of Reviewer WK7K, we include additional results on more values of K in Fig. 9 in Appendix B.4, which shows CACR can further boost its performance by increasing K.

------

> **Why the conditional distributions in CACR are defined in the current form?** (From Reviewer WK7K and Reviewer edUG)

The conditional probability here leverages a Bayesian strategy in [Ref1], which generalizes the energy-based conditional distribution $P(Y|X) = \frac{exp(-t E(X,Y))}{\int_y exp(-t E(X,y))}$ (Eqn in page 2 of [Ref2]) to $P(Y|X) = \frac{exp(-t E(X,y)) p(y)}{\int_y exp(-t E(X,y))p(y)}$ and makes the normalizing denominator Q(X) easy to be integrated. The advantage here is that the energy function $E(X,Y)$ is flexible to be designed and is able to capture dependencies between variables. In our case, for simplicity, we define it as distance $d_t(x,y)$ for positive conditional distribution (Eqn 3) and negative distance $-d_t(x,y)$ for negative conditional distribution (Eqn 4). 

This formulation can be interpreted as follows: the distant positive pairs and close negative pairs have higher probabilities to be attracted and repulsed, respectively. This well accords with the typical intuition in contrastive learning.

Here we provide an additional empirical study to demonstrate the correctness of our design. At our current CACR setting, we obtain 86.48% accuracy for Cifar10. By contrast, 

1) if we make the probability of moving a positive sample to the query be LOW when it is far from the query, by letting $\pi_+ = exp(-d_{t^+}(x,y)) / Q_+$, then the performance drops to 83.91%; 

2) if we make the probability of moving away a negative sample be HIGH when it is far from the query, by letting $\pi_- = exp(+d_{t^-}(x,y)) / Q_-$, then the performance drops to 79.46%;

3) if we combine 1) and 2), then the performance further drops to 74.91%. 

We have added this discussion into the Appendix (Table 8) to further validate the proposed CACR framework.  



[Ref1] Huangjie Zheng and Mingyuan Zhou. Exploiting Chain Rule and Bayes’ Theorem to Compare Probability Distributions. arXiv preprint arXiv:2012.14100, 2020.

[Ref2] LeCun, Y. and Huang, F. (2005). Loss functions for discriminative training of energy-based models. In Proc. of the 10-th International Workshop on Artificial Intelligence and Statistics (AIStats’05).

----------

## Summary of revision:

To address the concerns of the reviewers, we have carefully revised our paper, with the main changes marked in blue. A brief summary of the revision is as follows:

- Page1: 
Clarification in the abstract and introduction.

- Page 2:
Related work: Additional reference and discussion.

- Page 5:
More clearly defining the delta function and adding explanations.

- Page 17:
Additional results on varying conditional probability definition (Table 8).

- Page 21:
Results of different positive sampling sizes in page 21 (Fig. 9) and corresponding computation complexity (Table 11)

Below we provide our point-to-point response to each reviewer.


This paper proposes a new contrastive attraction and contrastive repulsion (CACR) loss for self-supervised representation learning. By formulating two conditional distributions, CACR considers different importance of positive/negative samples according to their distances to the query sample. Experimental results show superior performance of CACR against traditional contrastive loss and state-of-the-art methods. Strengths:
1. This paper is well organized and easy to understand.
2. The idea of CACR for self-supervised representation learning is reasonable and shows promising performance.
3. The experiments and analysis are sufficient.

Weaknesses:
1. It’s not clear why the conditional distributions should be like Eq. 3&4. Are there any theories? For example, why the probability of “moving it to a positive sample” is high when query is far from its positive pair? More explanations are needed.
2. It shows that increasing number of positive samples (number of augmentations, controlled by hyperparameter “K”) benefits the performance. It would be better to know the performance curve in a large range of K. Currently only K=1,4 are evaluated.
 This paper is generally well motivated and well written. The experiments are sufficient and the proposed CACR shows impressive performance. My minor concern is better explanations of the conditional distributions.

This paper aims to improve the conventional contrastive learning (CL) by introducing a CACR loss, which takes the distance-based weights into the consideration for CL loss computation. Theoretical analysis is provided on the robustness property of the representation learned by CACR. Experiments are also presented to evaluate the performance of CACR compared with MoCo V2, SimCLR and other recent CL methods.  Strength: 

1. The proposed method is intuitively correct. 

2. The experiments demonstrate the proposed method works better than SimCLR and MoCo V2. 

Weakness:

1. The novelty is limited. 

2. The paper is not very easy to follow. 

3. From the experiments, the performance advantage of the proposed method over existing CL methods (like BYOL) is not very significant. The compared baseline is not always consistent. The proposed method is compared against SimCLR in some experiments and MoCo in other experiments.  Overall, I feel this paper makes the presentation of the proposed method over-complicated and the resulted approach has limited novelty. The experiment is not very solid and convincing. 

In this paper, the authors present a new contrastive representation learning method. Specifically, a doubly contrastive learning (CL) approach is proposed, in which it contrasts positive and negative samples within themselves separately. The proposed method is termed "contrastive attraction and contrastive repulsion (CACR)". Extensive experimental, as well as theoretical analyses, show the effectiveness and robustness of the proposed method. The main contributions of this work are the proposed CACR strategy and the corresponding experimental analysis. ### Strengths

\+ The authors provide a theoretical analysis of the proposed CACR.

\+ Extensive experimental analysis validate the effectiveness of the proposed method.

\+ The proposed CACR is shown to be more robust compared to other existing methods, when evaluated on label-imbalanced data.

\+ The writing of the paper is generally good.

### Weaknesses

\- The main contribution of this paper is the "double-contrast" strategy. Compare to the traditional contrastive loss, the novelty is a bit limited. Besides, it is unclear why such a "double-contrast" is necessary, and why have to use the multiplication weight as a selection approach instead of others.

\- The proposed method is motivated by the conventional CL's robustness on uncurated datasets, as claimed by the authors. But the experimental analysis does not validate this well. The "wild" uncurated data is not only data-imbalance, it is also related to the quality and noise level of the data. As a result, suggest the authors either tone down the claim or provide more convincing justifications, either empirical or theoretical.

\- Although the authors demonstrate the effectiveness of the proposed method in a controlled experiment (Sec. 5.1), when compared to SOTAs (Sec. 5.2), the performance is not significant, just comparable/incremental to prior works.

\- It would be better if the authors could present an analysis of the add on computational complexity / number of parameters, by using the proposed CACR strategy.

\- The proposed method is also related to (though not exactly the same scenario) the hard-negative and hard-positive sampling used in video-audio representation learning (e.g. [*1, *2, *3]) and it would be better to discuss the relationships.

[*1] "Cooperative learning of audio and video models from self-supervised synchronization", NeurIPS 2018

[*2] "Self-supervised contrastive video-speech representation learning for ultrasound", MICCAI 2020

[*3] "Spoken moments: Learning joint audio-visual representations from video descriptions", CVPR 2021. Although the proposed method seems to be an incremental improvement to the conventional contrastive learning, the "doubly contrastive strategy" is shown to be effective by performing extensive experimental evaluations. This might be of interest to a group of audiences in ICLR 2022 and contribute to the community. On the other hand, there are some unclear statements I hope the authors could address in rebuttal. As a result, I recommend a marginal acceptance at the current stage and may change my rating after rebuttal.

This paper introduces Contrastive Attraction and Contrastive Repulsion (CACR) loss for self-supervised learning with a doubly contrastive strategy. It constructs two conditional distributions to model the importance of positive samples and negative samples according to their distances to the query. The importance, which is reflected by weights, guides the query to not only more strongly pull more distant positive samples, but also more strongly push away closer negative samples. Theoretical analysis and empirical results show that CACR generalizes the conventional CL loss and is more robust in more general cases. Superior performances are achieved on both curated and uncurated datasets, and on various downstream tasks. Strength:
1. CACR models the importance of positive/negative samples, making the optimization more effective.	
2. CACR is more robust than conventional contrastive loss, according to the performance on label-imbalanced datasets.
3. Pretraining representations of CARC consistently perform better than conventional CL methods when finetuned on downstream tasks.
4. Theoretical analysis, empirical results and ablation studies are sufficient to support its effectiveness.

Weakness:
There are some minor problems.
1.	I can’t find the definition of \delta in eq. (5).
2.	In the last sentence on page 1, why both easy and hard negative samples are desired for contrast? What’s the reference?
 Though the proposed framework is closely related to previous CL methods on the ideas of multi-crop and hard negatives, it still shows novelty on modeling the intra-relation among positives/negatives. The paper provides detailed theoretical analysis for understanding the method. Experiments and ablation studies are also sufficient to demonstrate its effectiveness. 

Meta-review: The submission receives mixed ratings initially. Reviewer WK7k stays positive, edUG and qbmC are borderline, and MXU1 is negative. They raise several issues including limited technical contribution, insufficient sota experimental comparison, and detailed theoretical proof. The authors have responded to these issues in the rebuttal. The final ratings of these reviewers do not alter. 

After checking the revised manuscript, all the reviews and responses, the AC feels the review from MXU1 lacks sufficient details and the claim shall be better supported with evidence. On the other hand, the issue of the technical contribution raised by edUG still exists. The modification upon original CL is marginal. Also, the experimental validation, although tested in several configurations, only includes ResNet 50 as the encoder backbone. The sufficient validation upon different scales of CNNs are common strategies used in sota SSL methods (e.g., MoCo, BYOL, SimCLR). Without a thorough evaluation, the effectiveness of CACR on different scales of CNN backbones is in doubt. Meanwhile, the experimental comparison upon sota methods (e.g., BYOL, SwAV+multi crop) does not clearly show the performance advantages of CACR. The small-scale dataset utilized for ablation study (i.e., CIFAR) is not as convincing as ImageNet. 

Overall, the AC feels that the marginal technical contribution is acceptable, but it shall be equipped with thorough experimental validation and thus serves as a fundamental baseline (i.e., potentially benefits the SSL community). Based on the current form, the authors are suggested to further improve the current submission and are welcome to submit for the next venue.

Decision: Reject