Intriguing Properties of Contrastive Losses

Abstract: We study three intriguing properties of contrastive learning. First, we generalize the standard contrastive loss to a broader family of losses, and we find that various instantiations of the generalized loss perform similarly under the presence of a multi-layer non-linear projection head. Second, we study if instance-based contrastive learning (with a global image representation) can learn well on images with multiple objects present. We find that meaningful hierarchical local features can be learned despite the fact that these objectives operate on global instance-level features. Finally, we study the phenomenon of feature suppression among competing features shared across augmented views, such as "color distribution" vs "object class". We construct datasets with explicit and controllable competing features, and show that, for contrastive learning, a few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. In scenarios where there are multiple objects in an image, the dominant object would suppress the learning of smaller objects. Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features over others, and could suffer from learning saturation for scenarios where existing augmentations cannot fully address the feature suppression. This poses open challenges to existing contrastive learning techniques.


Reviews and author responses:

This paper studies properties of the contrastive losses. The contributions are three-folds:
First, the authors find that when using a deep multi-layer nonlinear projection head, the choice of objectives does not matter within a family of generalized losses, the same goes for the choice of batch size. The authors then focus on instance-based objectives, and find that they help learning meaning hierarchical local features despite the objectives being global.

The most interesting contribution is that the authors identify a phenomenon called "feature suppression", where a few dominating features can hinder the learning of other features, posing an open challenge to existing contrastive learning techniques.  __Significance__: this paper provides interesting empirical observations on properties of the contrastive loss. I find "feature suppression" the most interesting. My concerns are 1) the robustness of this finding, e.g. whether feature suppression is a result of improper optimization; 2) it seems that similar observations have been made in previous work, but this paper does not provide deeper insights as in why this is happening; please see detailed comments below.
Despite these concerns, I think the phenomena provided in this work are interesting and hence recommend an accept.

__Questions__:
* Results about Table 3:
  * the numbers are reported without an error bar; are they from a single run or that the variance is small?
  * for supervised learning: 1) how is it trained, i.e. which digit is selected as the class label? 2) why is in-grid worse than random placement? Why is there a sudden drop of performance for in-grid when going from 12 to 16 digits?
  * why is learning multiple objects surprising, especially when the model has sufficient capability?

* (Main concern) Feature suppression:
  * Is this a result of the optimization algorithm? Can this be resolved by an initial large learning rate? e.g. Figure 3 in "Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks" by Li et. al.
  * Citation 12 (Tian et. al.) made a similar observation in their "Colorful-Moving-MNIST" (which is similar to DigitOnImageNet dataset used in this work) section with "Multiple factors shared" that sharing the digits dominates learning compared to the position of the digit, which is a form of feature suppression. I agree that this paper is the first to explicitly identify this phenomenon and emphasizes it as a problem, but would appreciate more comments from the authors about the difference, implications, and deeper explanations.

* Clarification on "Number of unique digits" (Fig 6): what does the x-axis mean? Is this the total number of distinct digits appearing in the dataset?

* The feature clustering results seem to suggest that supervised & contrastive learning learn similar features, however the feature suppression experiments (especially MultiDigit) seem to say that contrastive learning is more susceptible to being suppressed by "dominating features" than supervised learning. Is this it a result of optimization or the objective?

__Writing__: the paper is clearly written and easy to follow. I'd appreciate more details on the training procedure (e.g. the learning rates used, and the variances in different runs), but the message is clear. A minor typo: line 269 "but we leave that as...".

====== 
Post-rebuttal update: I thank the authors for their responses and keep my score of recommending an accept. The authors discuss two limitations: first, this paper focuses only on methods with explicit negatives. This is not a problem for me since it is okay for an analysis paper to focus on one type of methods. The second limitation is that the datasets used in the experiments are not fully realistic. This again is not an issue for me, since 1) the datasets used in this paper are variants of ImageNet and MNIST which are both realistic, and 2) fully realistic datasets will make it hard to control multiple aspects of variation with precision.

I agree with the authors' judgement that there is no immediate societal impact.

 Thanks for the explanations. I will keep my score and still recommend for recommendation.

 I appreciate the authors' responses. After reading the comments from all reviews and the corresponding responses, my concerns are addressed and I will remain my original score and recommend to accept.

 I thank the authors for the detailed responses.
This paper brings up several interesting empirical findings. Though it would be better to include more explanations on these findings, I agree that identifying these phenomena is itself a good first step.
I stand by my original score and recommend an accept.

 Thanks for the updates. I looked through the updates and appreciate the clarification. It would be great to add the results of deeper projection head to the paper. After reading these and my co-reviewers comments, I still advocate for acceptance.

 We’d like to thank the reviewer for their valuable time, positive comments and constructive reviews. Please find our responses below.

**Generalized contrastive loss and projection head** 
> The paper discovers these properties, but does not explain them. For instance, Why can deep projection heads lead to such improvement? What if we keep increasing the depth beyond three layers? Will the performance drop at the end?

We agree with the assessment as we try to avoid premature explanation / analysis (which can be wrong and misleading), and instead focus on presenting solid results (empirical discovery of properties). We believe that more theoretical analysis / explanation can come later with stronger scientific rigor.

For the deep projection head, we speculate the reason it helps is that it can reduce overfitting / over-specialization of backbone network representation to the contrastive prediction / instance discrimination task. But again, a formal explanation may require more rigorous theoretical analysis. As for the number of layers for the project head, 3-layer seems pretty optimal in our experiments. We empirically found that layers reach diminishing returns and performance will eventually drop; they also require more compute (as large fully connected layers are very expensive).

**Feature suppression** 
> The feature suppression phenomenon has also been explored in supervised learning setting [3, 4]. As info-NCE loss can be interpreted as instance discrimination, the same analysis from [3, 4] might also be applicable here.

We thank the reviewer for bringing up both work to our attention, and we agree they are relevant. We actually mentioned the potential feature suppression effect in supervised learning in the related work section (line 256) and have already cited [4]; we will also add discussion to [3]. While feature suppression can exist for both supervised learning and unsupervised contrastive learning, one key difference is that current contrastive learning methods (for images) are highly dependent on data augmentation (i.e. augmented views should be random and different), while supervised learning does not require data augmentation (though it often helps). Therefore we expect a good/realistic analysis for unsupervised contrastive learning should use data augmentation in their core assumptions (which is not necessary for supervised learning).

> An interesting experiment might be varying the weight for the distribution loss in equation (2) to see whether larger \lambda increase the robustness against feature suppression.

The suggested experiment and analysis can be found in Appendix E. Increasing the weight in distribution loss has similar effects as lowering the temperature in the standard NT-Xent loss, as both encourage higher entropy in the representations. However, as shown in Figure E1, adjusting the weight does not really solve the feature suppression problem (on RandBit dataset), as the best obtainable performance with this tuning is still lower than the baseline without extra bits added. We have tuned this hyper-parameter in other datasets and similar results are observed.


 We’d like to thank the review for their valuable time and detailed constructive review. Please find our responses below.

**Generalized contrastive loss**
> The generalized contrastive loss seems heuristic and could be enhanced if the authors could elaborate as done in [16].

We believe the generalized contrastive loss is more than a heuristic, as it is a proxy of mutual information between two augmented views (also a generalization of InfoNCE lower bound on the mutual information). More specifically, as discussed in supplementary material (line 420), the mutual information I(U; V ) = H(U) − H(U|V), so (1) the alignment term is directly related to H(U|V) which aims to reduce uncertainty of the other views given one view of the example; and (2) the distribution matching term can be considered as a proxy to H(u) for maximizing the entropy in the representation. We will further elaborate this in the main paper.

> The alignment and distribution regularization seems to be closely related to variational autoencoder, where the alignment can be achieved with reconstruction. Is there any connection?

While the VAE loss does share a similar prior loss (but typically much loosely enforced) on the distribution of latent representations, the reconstruction term in VAE is very different from the alignment term in the contrastive loss. In particular, the alignment term in contrastive learning relies entirely on the data augmentation (e.g. the alignment loss would be trivial if two augmented views are identical), while the reconstruction term does not require data augmentation.

> if we choose the normal distribution as prior, are the feature vectors normalized like on hypersphere?

We do not apply l2-normalization when using normal distribution as prior. However the *learned* feature vectors (of projection head’s output) are near a hypersphere. This is because the distribution of learned representations are close to prior normal distribution, and in high dimension space, the majority of mass of a normal distribution is on a hypersphere. We will clarify this in the revision.

> In the paper, the SWD is chosen for the distribution matching, how does it perform when using other metrics like JSD (discriminator-based method), MMD (IPM-based method)?

We chose SWD for its relative simplicity over the other methods. We have not tried the more complicated methods - e.g. JSD requires a discriminator which can cause other complications during training and optimization. We agree that this can be an interesting future investigation to compare the performance of various distribution matching mechanisms.

**Feature suppression**
> As the authors mention the suppression, if we construct the positive samples to resist the feature competing, e.g., we take channel-wise concatenation as a way to augment samples, will the model be enhanced to resist? 

If we understood ‘channel-wise concatenation’ correctly (please feel free to follow-up in discussion if we misunderstood), in the context of the RandBit experiments, simply adding more extra channels will not help, but randomly changing the random bits independently for both augmented views will help (we disallow such operations to simulate the situation where we don't know what the right augmentation is for those random features). In general if we have the right augmentation (e.g. color jittering to resist color histogram features), they can be effective in encouraging the model to learn more relevant features. The main difficulty is how to find the right augmentation for all features of interest, and in some limited scenarios (e.g. multiple objects of different sizes in an image), it is not even clear whether some simple augmentation exists that can address the feature suppression.


 We’d like to thank the review for their valuable time and detailed constructive review. Please find our responses below.

**Generalized contrastive loss**
> Although this result seems to indicate that the contrastive loss formalization does not matter with a deep projection head, the fact that only SimCLR setting is tested makes the analysis less useful. … These architecture and algorithmic differences make the analysis possibly limited only to SimCLR-like settings.

Our formulation of generalized contrastive loss generalizes the popular InfoNCE loss family with *explicit negatives* (which is used in SimCLR but also many others such as AMDIM, CPCv1/v2, CMC, PIRL, InstDist, MoCo..), and we agree it is not directly applicable to other implicit contrastive methods like BYOL. Although BYOL does not have a distribution loss, the learning dynamics (with stop gradient, EMA and predictor) may maintain an implicit entropy term. The extension of our formulation to implicit contrastive methods is useful but can come later as it is non-trivial, and we believe our current formulation provides a solid initial step towards that.
 
We would also like to point out that other findings of our work (e.g. feature suppression study) are (directly or indirectly) generalizable to a broader set of self-supervised learning methods including implicit contrative techniques. For example, we did preliminary experiments on BYOL and observed similar limitations from feature suppression (as mentioned on line 240).

**Multiple objects and localized representations**
> I feel that the digit recognition problem is really too easy and poses no challenge for the unsupervised learning algorithms. Specifically, recognizing the digits may just require the network to be able to extract line-features… Multi-digit analysis is limited by the easy-to-learn datasets the authors construct and may not have clear indications on real-world learning scenarios.

We agree that the standard single digit recognition problem is not differentiating enough for comparing modern advanced models (though it has been used as a benchmark for decades so it is not trivial). However, our proposed dataset is a modified version with multiple (overlapping) digits of different sizes. Purushwalkam et al [30] and a few others conjectured that contrastive learning “gains from access to a clean object-centric training dataset” as it is based on instance-level discrimination (matching two crops of the global image instead of two crops of each object). Therefore, it is non-trivial to know whether they would work well or at all on images of multiple objects/digits (some of which can be highly overlapping). Indeed, current contrastive learning algorithms do NOT work well when some digits in the image are smaller among others (Section 4.3), or when random bits are added (Section 4.4), which show this seemingly “easy” dataset is actually not solved.

That said, we also agree that it would be more desirable to use more natural images with multiple objects. However, existing datasets with multiple objects (e.g. COCO) are not directly  controllable, i.e. we cannot manipulate the number of objects, size of objects easily, which are critical for our analysis on how different numbers of objects (and their sizes) affect the representation learning. In the future, we can look into simulation environments with more diverse objects.

> Although I agree that the presented visualization indeed shows the cluster phenomenon in the intermediate layers, I feel this result is not surprising at all, as the SimCLR network (and other contrastive learning algorithms) have proven useful in COCO tasks, which would require this type of information. This sub-analysis can also benefit from more quantitative metrics instead of just visualizations.

Previous results in COCO tasks were achieved via fine-tuning, which allows the pretrained model to change the representations towards specific tasks (degree depends on fine-tuning hyper-parameters such as learning rate). So it is possible that the localization properties in representations are derived from fine-tuning instead of pretraining. Our visualization results are obtained directly from un-modified representations of SimCLR pre-training (i.e. not fine-tuned) via clustering, which provides a *direct* evidence that the internal representations in the pretrained network exhibit localization properties despite the loss function is instance-level discrimination. 

We agree quantitative metrics can be beneficial but it is also difficult since the approach does not use labels at all. As a remedy, we have created an interactive web-UI demo for visualizing thousands of randomly chosen images (from ImageNet and COCO validation set) to make the qualitative study more comprehensive (will be linked after the reviewing process).

**Feature suppression**
> This analysis indeed looks interesting and inspiring to me… I think explicitly forcing the algorithm to seek for additional information that can separate the images might be needed and even beneficial for general learning.

We share the same high level thoughts as the review. One way of encouraging the model to capture the features of interest is using specific data augmentation (such as color jittering in SimCLR). But this requires the hand design of such augmentation and may not cover all limited scenarios (e.g. different sizes of objects in the same image). It is still an open problem on how to encourage the model to learn more useful features without much human priors, and we believe addressing this problem of feature suppression can lead to significant performance improvements across a wide range of settings.


 We’d like to thank the review for their valuable time and detailed constructive review. Please find our responses below.

**Learning with multiple objects**
> Results about Table 3 - the numbers are reported without an error bar; are they from a single run or that the variance is small?

The numbers are from a single run for each experiment. However we did multiple runs for a few baseline experiments and discovered that the standard deviation is within 0.1. Therefore we reported numbers up to precision of 0.1 (keeping one decimal place). We will clarify it in the paper.

> Results about Table 3 - for supervised learning: 1) how is it trained, i.e. which digit is selected as the class label? 2) why is in-grid worse than random placement? Why is there a sudden drop of performance for in-grid when going from 12 to 16 digits?

For supervised experiments in Table 3, we used multi-label cross entropy loss for all digits that appear in an image. In-grid becomes worse than random placement only when there are more than 8 digits in an image. The drop of performance when there are too many digits is because digits from all 10 classes can often appear in the same image, these images provide no learning signal for multi-label classification.

> why is learning multiple objects surprising, especially when the model has sufficient capability?

Most contrastive learning methods are based on instance-level discrimination, i.e. matching representations of two crops of the global image instead of two crops of each object. Purushwalkam et al [30] and a few others conjectured that contrastive learning “gains from access to a clean object-centric training dataset”, and “the underlying biases of pre-training dataset - Imagenet is an object-centric dataset which ensures that different crops correspond to different parts of the same object”. Therefore, it is not straight-forward whether they would work well or at all on images of multiple objects. 

Our results provide a different/counter perspective from Purushwalkam et al [30] and others, as we explicitly study the behavior on images of controllable multiple objects, and show that they work well when objects/digits are of the same size. But when some objects/digits in the image are smaller among others, current contrastive learning algorithms do NOT work well (Section 4.3). This issue cannot be solved with bigger model capabilities or longer training (as we have varied model sizes and training epochs and the performance gap persists). 

**Feature suppression**
> Is this a result of the optimization algorithm? Can this be resolved by an initial large learning rate? e.g. Figure 3 in "Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks" by Li et. al.

Feature suppression is NOT a result of the optimization algorithm. Firstly, we have tuned hyperparameters extensively (including learning rates and training epochs) for all experiments in this paper. This phenomenon is consistent across all datasets that we designed. Secondly, adding extra features (random bits / MNIST digits) to ImageNet images actually reduces the loss, sometimes even to almost zero (with a few dozens of random bits), but the learned representations become even worse.

> Citation 12 (Tian et. al.) made a similar observation in their "Colorful-Moving-MNIST" (which is similar to DigitOnImageNet dataset used in this work) section with "Multiple factors shared" that sharing the digits dominates learning compared to the position of the digit, which is a form of feature suppression. I agree that this paper is the first to explicitly identify this phenomenon and emphasizes it as a problem, but would appreciate more comments from the authors about the difference, implications, and deeper explanations.

Our DigitOnImageNet is only similar to Colorful-Moving-MNIST” ([12]) in that we both overlay digits on some background, but we are still very different: the “Colorful-Moving-MNIST” is a synthetic video/sequence dataset, and they control the type of “views” used in the next frame. Our DigitOnImageNet dataset is an image dataset (which is the most common scenario where SSL algorithms are applied), and we only manipulate the number of unique digits used in the dataset. Therefore, our DigitOnImageNet dataset has the same data format as other natural image datasets like ImageNet, and can be directly used as a benchmark for comparing existing and future contrastive learning methods on the feature suppression issue.

For the feature suppression phenomenon, several previous works mentioned it, including SimCLR [13], which mentioned color histogram features and designed the color jittering augmentation. Different from previous work (including [12]), we study this phenomenon more comprehensively in controlled settings (e.g. comparing different numbers of objects, different sizes of objects, different types of features, and quantitatively on the number of extra random bits), and show multiple kinds of feature suppression (big vs small objects, easy-to-learn vs hard-to-learn features, suppression amount vs number of random bits). Furthermore, the resulting datasets (MultiDigits, DigitOnImageNet, RandBits) can serve as benchmarks for future research in this direction.

> Clarification on "Number of unique digits" (Fig 6): what does the x-axis mean? Is this the total number of distinct digits appearing in the dataset?

Yes, this is the total number of distinct digits appearing in the training dataset. For example, when only one unique digit is used, all imagenet images are overlaid with that same digit. During evaluation, the entire MNIST test set is used.
> The feature clustering results seem to suggest that supervised & contrastive learning learn similar features, however the feature suppression experiments (especially MultiDigit) seem to say that contrastive learning is more susceptible to being suppressed by "dominating features" than supervised learning. Is this a result of optimization or the objective?

We believe that is mainly a result of objective. In standard supervised learning, the objective is to pick the true label among all labels; in unsupervised contrastive learning, the objective is to pick the augmented view of the same example among candidate images. They can learn similar features on datasets like ImageNet with proper data augmentation (for contrastive learning), in which case we believe that contrastive learning learns class-relevant features (similar to that learned in supervised learning) for instance-level instrimination. However, current contrastive learning methods (for images) entirely rely on data augmentation (trivial/random features can achieve minimal contrastive loss without data augmentation), while supervised learning does not require data augmentation (though it often helps a little). This makes contrastive learning more susceptible to feature suppression.


The paper empirically studies three properties of contrastive loss: (1) deep projection head, (2) learning under multiple objects, and (3) feature suppression. In particular, the studies on feature suppression uncover open challenges for contrastive representation learning. 
  Overall, I think this paper address an important problem that will be of interest to a wide audience who works on contrastive learning. The paper is well written, I can follow all the details for the paper. The founded three properties are non-trivial and could be influential to algorithm design. For instance, the feature suppression observation has motivate several follow-up works to address this issue [1,2]. It is also interesting to see that making the projection head deeper consistently improves the performance across several formulation. Figure 3 and 4 also provide a possible explanation to why contrastive learning also lead to decent performance beyond classification. The good local features make contrastive learning a promising tool for detection or segmentation. I really like the studies in this paper.

Here are a few questions and suggestions:

1. The paper discover these properties, but does not explain them. For instance, why can deep projection heads lead to such improvement? What if we keep increasing the depth beyond three layers? Will the performance drops at the end? By answering these question could make the study more complete. 

2. The feature suppression phenomenon has also been explored in supervised learning setting [3, 4]. As info-NCE loss can be interpreted as instance discrimination, the same analysis from [3, 4] might also be applicable here.

3. It seems that VAE is robust to the features suppression problem as the reconstruction loss encourage all the information about images be preserved. As [5] shows, the uniformity (distribution) loss can be interpreted as an empirical approximation to the entropy of the feature variable Z. In particular, the entropy H(Z) can be interepreted how much input information is preserved in the features as I(X;Z) = H(Z) where X is the input variable. Therefore, an interesting experiment might be varying the weight for the distribution loss in equation (2) to see whether larger \lambda increase the robustness agains feature suppression.

In conclusion, the paper conduct interesting experiments that could impact algorithm design. I did not observe obvious flaw in the paper.

[1] Robinson et al., Can contrastive learning avoid shortcut solutions?, 2021

[2] Li et al., Information-Preserving Contrastive Learning for Self-Supervised Representations, 2021

[3] Shah et al., The Pitfalls of Simplicity Bias in Neural Networks, 2020

[4] Hermann et al., What shapes feature representations? Exploring datasets, architectures, and training, 2020

[5] Wang and Isola., Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, 2020.
 The authors adequately addressed the limitations and potential negative societal impact.

This work presents three empirical analyses on contrastive learning algorithms. The first analysis shows that when using a multi-layer non-linear project head, the different ways of instantiating the generalized contrastive learning loss lead to final representations with similar downstream visual task performance. The second analysis finds that the instance-based contrastive learning algorithms can still achieve good performance on clustered images, even though these algorithms are based on global image representations. The third analysis is about the preference of contrastive learning algorithms on competing features that are easy-to-learn. In this analysis, the authors show that after manually adding diverse competing information into the images and linking them with image identity, the contrastive learning algorithms fail to capture the general statistics but chooses to only encode the competing information. These three empirical analyses seem to be unrelated and all have certain limitations making the analyses not general to all algorithms. Although the final analysis illustrates that the contrastive learning algorithms indeed have certain limitations, the authors have not proposed a better or modified algorithm to address that issue. In summary, I feel the analyses presented in this paper are useful for the community to be aware of, so I would weakly recommend acceptance. But the current paper can be much stronger if the authors can address the limitations in the analyses (see detailed review below) and even further leverage the analyses to propose even better algorithms resolving the presented issues in the analyses.  Below, I will describe the limitations for all three analyses in my view, hoping the authors to address them in the rebuttal phase or to motivate the authors to develop stronger analyses. 

1. In the loss formalization analysis, the authors start from the generalized contrastive loss definition, which is the sum of the alignment loss and the uniformity loss. They then test several ways of defining the “alignment” and the “uniformity” including different prior distributions (hypercube or hypersphere) and different ways of encouraging uniformity (SWD or typical LogSum) to show whether the formalization of the generalized contrastive loss definition actually influences the results. After training a ResNet-50 on CIFAR-10 and ImageNet with two or three hidden layers in the projection head using a SimCLR setting, the authors find that when the projection is deeper (3 layers), there is no significant difference between these different formalizations. Although this result seems to indicate that the contrastive loss formalization does not matter with a deep projection head, the fact that only SimCLR setting is tested makes the analysis less useful. The innovation in most of the newer contrastive learning algorithms is not in how the prior distribution is defined nor in how the uniformity is encouraged, like BYOL has no uniformity loss but has a momentum-encoder and a prediction head. These architecture and algorithmic differences make the analysis possibly limited only to SimCLR-like settings. 
2. In the learning from “crowded” images analysis, the authors show that the even learning from a multi-digit dataset, SimCLR can still extract useful features that can be leveraged later to identify single digit. However, I feel that the digit recognition problem is really too easy and poses no challenge for the unsupervised learning algorithms. Specifically, recognizing the digits may just require the network to be able to extract line-features and it is not difficult to imagine that this type of features are indeed needed for doing the instance-level contrastive learning on multi-digit images as the models need to use the edge information to further separate the images. However, if the images contain multiple objects with overlapping, it is unclear whether the models can still learn meaning high-level information that is useful for single object identification, instead of just learning multi-object features like how they overlap with each other. Therefore, this analysis is limited by the easy-to-learn datasets the authors construct and may not have clear indications on real-world learning scenario. The later sub-analysis in the same section is about whether intermediate features of the SimCLR model trained on ImageNet cluster according to meaningful semantic similarity. Although I agree that the presented visualization indeed shows the cluster phenomenon in the intermediate layers, I feel this result is not surprising at all, as the SimCLR network (and other contrastive learning algorithms) have proven useful in COCO tasks, which would require this type of information. This sub-analysis can also benefit from more quantitative metrics instead of just visualizations.
3. In the final analysis, the authors show that contrastive learning algorithms can be “misled” by the competing information in the images that is easier to learn. This analysis indeed looks interesting and inspiring to me. The basic result of this analysis is intuitive, as the contrastive learning algorithms seek to find cues to separate different images and the added information to the images is like the cues that can be easy to use to do the separation, especially as it varies more with respect to the image identity. The learning algorithms therefore has no further motivation to find the other features to separate the images as the digits are already enough. This feature can be used as a way to “attack” the contrastive learning algorithms through adding possibly “invisible” features to the images to make them mislead the learning algorithms. Therefore, clearly illustrating this issue is indeed important for finding a way to address this issue later. I think explicitly forcing the algorithm to seek for additional information that can separate the images might be needed and even beneficial for general learning.
 Yes.

The paper introduces an empirical study on contrastive loss and its variants. In the paper, the contrastive loss is generalized as a linear combination of the alignment between positive representation pairs and the regularization between representation and prior distribution. Based on this formulation, then the authors present some interesting properties of contrastive loss: 1) meaningful local features can be learned when there exist multiple objects in the image; 2) competing features across augmentations could suppress other parts and limit the effectiveness of contrastive loss.
  Strength:
Introducing the intriguing properties of contrastive losses looks interesting and important to me. It also provides new insights to the community. Both the motivation and the technical details in the experiment designs seem sound. The evaluation shows the effectiveness of the proposed method on both synthetic datasets and standard datasets. 

Weakness:
The generalized contrastive loss seems heuristic and could be enhanced if the authors could elaborate the as done in [16]. The alignment and distribution regularization seems to be closely related to variational autoencoder, where the alignment can be achieved with reconstruction. Is there any connection?

Some details are not clarified. For example, if we choose the normal distribution as prior, are the feature vectors normalized like on hypersphere?  

Some open questions:
As the authors mention the suppression, if we construct the positive samples to resist the feature competing, e.g., we take channel-wise concatenation as a way to augment samples, will the model be enhanced to resist?  In the paper, the SWD is chosen for the distribution matching, how does it perform when using other metrics like JSD (discriminator-based method), MMD (IPM-based method)?  The authors did not discuss the limitations and potential negative societal impact.

Meta-review: The paper studies three properties of contrastive learning.  Reviewers found that although these properties were shown in experiments the paper did not explain why they happen, but thought they can guide better algorithm design in contrastive learning and of interest to a wide audience. Reviewers agreed that this is a good paper. Please include in the final manuscript the additional explanation provided during the rebuttal.  Accept 

Decision: accept