Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper empirically studies three properties of contrastive loss: (1) deep projection head, (2) learning under multiple objects, and (3) feature suppression. In particular, the studies on feature suppression uncover open challenges for contrastive representation learning. 

main_review: Overall, I think this paper address an important problem that will be of interest to a wide audience who works on contrastive learning. The paper is well written, I can follow all the details for the paper. The founded three properties are non-trivial and could be influential to algorithm design. For instance, the feature suppression observation has motivate several follow-up works to address this issue [1,2]. It is also interesting to see that making the projection head deeper consistently improves the performance across several formulation. Figure 3 and 4 also provide a possible explanation to why contrastive learning also lead to decent performance beyond classification. The good local features make contrastive learning a promising tool for detection or segmentation. I really like the studies in this paper.

Here are a few questions and suggestions:

1. The paper discover these properties, but does not explain them. For instance, why can deep projection heads lead to such improvement? What if we keep increasing the depth beyond three layers? Will the performance drops at the end? By answering these question could make the study more complete. 

2. The feature suppression phenomenon has also been explored in supervised learning setting [3, 4]. As info-NCE loss can be interpreted as instance discrimination, the same analysis from [3, 4] might also be applicable here.

3. It seems that VAE is robust to the features suppression problem as the reconstruction loss encourage all the information about images be preserved. As [5] shows, the uniformity (distribution) loss can be interpreted as an empirical approximation to the entropy of the feature variable Z. In particular, the entropy H(Z) can be interepreted how much input information is preserved in the features as I(X;Z) = H(Z) where X is the input variable. Therefore, an interesting experiment might be varying the weight for the distribution loss in equation (2) to see whether larger \lambda increase the robustness agains feature suppression.

In conclusion, the paper conduct interesting experiments that could impact algorithm design. I did not observe obvious flaw in the paper.

[1] Robinson et al., Can contrastive learning avoid shortcut solutions?, 2021

[2] Li et al., Information-Preserving Contrastive Learning for Self-Supervised Representations, 2021

[3] Shah et al., The Pitfalls of Simplicity Bias in Neural Networks, 2020

[4] Hermann et al., What shapes feature representations? Exploring datasets, architectures, and training, 2020

[5] Wang and Isola., Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, 2020.

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents three empirical analyses on contrastive learning algorithms. The first analysis shows that when using a multi-layer non-linear project head, the different ways of instantiating the generalized contrastive learning loss lead to final representations with similar downstream visual task performance. The second analysis finds that the instance-based contrastive learning algorithms can still achieve good performance on clustered images, even though these algorithms are based on global image representations. The third analysis is about the preference of contrastive learning algorithms on competing features that are easy-to-learn. In this analysis, the authors show that after manually adding diverse competing information into the images and linking them with image identity, the contrastive learning algorithms fail to capture the general statistics but chooses to only encode the competing information. These three empirical analyses seem to be unrelated and all have certain limitations making the analyses not general to all algorithms. Although the final analysis illustrates that the contrastive learning algorithms indeed have certain limitations, the authors have not proposed a better or modified algorithm to address that issue. In summary, I feel the analyses presented in this paper are useful for the community to be aware of, so I would weakly recommend acceptance. But the current paper can be much stronger if the authors can address the limitations in the analyses (see detailed review below) and even further leverage the analyses to propose even better algorithms resolving the presented issues in the analyses.
main_review: Below, I will describe the limitations for all three analyses in my view, hoping the authors to address them in the rebuttal phase or to motivate the authors to develop stronger analyses. 

1. In the loss formalization analysis, the authors start from the generalized contrastive loss definition, which is the sum of the alignment loss and the uniformity loss. They then test several ways of defining the “alignment” and the “uniformity” including different prior distributions (hypercube or hypersphere) and different ways of encouraging uniformity (SWD or typical LogSum) to show whether the formalization of the generalized contrastive loss definition actually influences the results. After training a ResNet-50 on CIFAR-10 and ImageNet with two or three hidden layers in the projection head using a SimCLR setting, the authors find that when the projection is deeper (3 layers), there is no significant difference between these different formalizations. Although this result seems to indicate that the contrastive loss formalization does not matter with a deep projection head, the fact that only SimCLR setting is tested makes the analysis less useful. The innovation in most of the newer contrastive learning algorithms is not in how the prior distribution is defined nor in how the uniformity is encouraged, like BYOL has no uniformity loss but has a momentum-encoder and a prediction head. These architecture and algorithmic differences make the analysis possibly limited only to SimCLR-like settings. 
2. In the learning from “crowded” images analysis, the authors show that the even learning from a multi-digit dataset, SimCLR can still extract useful features that can be leveraged later to identify single digit. However, I feel that the digit recognition problem is really too easy and poses no challenge for the unsupervised learning algorithms. Specifically, recognizing the digits may just require the network to be able to extract line-features and it is not difficult to imagine that this type of features are indeed needed for doing the instance-level contrastive learning on multi-digit images as the models need to use the edge information to further separate the images. However, if the images contain multiple objects with overlapping, it is unclear whether the models can still learn meaning high-level information that is useful for single object identification, instead of just learning multi-object features like how they overlap with each other. Therefore, this analysis is limited by the easy-to-learn datasets the authors construct and may not have clear indications on real-world learning scenario. The later sub-analysis in the same section is about whether intermediate features of the SimCLR model trained on ImageNet cluster according to meaningful semantic similarity. Although I agree that the presented visualization indeed shows the cluster phenomenon in the intermediate layers, I feel this result is not surprising at all, as the SimCLR network (and other contrastive learning algorithms) have proven useful in COCO tasks, which would require this type of information. This sub-analysis can also benefit from more quantitative metrics instead of just visualizations.
3. In the final analysis, the authors show that contrastive learning algorithms can be “misled” by the competing information in the images that is easier to learn. This analysis indeed looks interesting and inspiring to me. The basic result of this analysis is intuitive, as the contrastive learning algorithms seek to find cues to separate different images and the added information to the images is like the cues that can be easy to use to do the separation, especially as it varies more with respect to the image identity. The learning algorithms therefore has no further motivation to find the other features to separate the images as the digits are already enough. This feature can be used as a way to “attack” the contrastive learning algorithms through adding possibly “invisible” features to the images to make them mislead the learning algorithms. Therefore, clearly illustrating this issue is indeed important for finding a way to address this issue later. I think explicitly forcing the algorithm to seek for additional information that can separate the images might be needed and even beneficial for general learning.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces an empirical study on contrastive loss and its variants. In the paper, the contrastive loss is generalized as a linear combination of the alignment between positive representation pairs and the regularization between representation and prior distribution. Based on this formulation, then the authors present some interesting properties of contrastive loss: 1) meaningful local features can be learned when there exist multiple objects in the image; 2) competing features across augmentations could suppress other parts and limit the effectiveness of contrastive loss.

main_review: Strength:
Introducing the intriguing properties of contrastive losses looks interesting and important to me. It also provides new insights to the community. Both the motivation and the technical details in the experiment designs seem sound. The evaluation shows the effectiveness of the proposed method on both synthetic datasets and standard datasets. 

Weakness:
The generalized contrastive loss seems heuristic and could be enhanced if the authors could elaborate the as done in [16]. The alignment and distribution regularization seems to be closely related to variational autoencoder, where the alignment can be achieved with reconstruction. Is there any connection?

Some details are not clarified. For example, if we choose the normal distribution as prior, are the feature vectors normalized like on hypersphere?  

Some open questions:
As the authors mention the suppression, if we construct the positive samples to resist the feature competing, e.g., we take channel-wise concatenation as a way to augment samples, will the model be enhanced to resist?  In the paper, the SWD is chosen for the distribution matching, how does it perform when using other metrics like JSD (discriminator-based method), MMD (IPM-based method)? 
limitations_and_societal_impact: The authors did not discuss the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies properties of the contrastive losses. The contributions are three-folds:
First, the authors find that when using a deep multi-layer nonlinear projection head, the choice of objectives does not matter within a family of generalized losses, the same goes for the choice of batch size. The authors then focus on instance-based objectives, and find that they help learning meaning hierarchical local features despite the objectives being global.

The most interesting contribution is that the authors identify a phenomenon called "feature suppression", where a few dominating features can hinder the learning of other features, posing an open challenge to existing contrastive learning techniques.
main_review: __Significance__: this paper provides interesting empirical observations on properties of the contrastive loss. I find "feature suppression" the most interesting. My concerns are 1) the robustness of this finding, e.g. whether feature suppression is a result of improper optimization; 2) it seems that similar observations have been made in previous work, but this paper does not provide deeper insights as in why this is happening; please see detailed comments below.
Despite these concerns, I think the phenomena provided in this work are interesting and hence recommend an accept.

__Questions__:
* Results about Table 3:
  * the numbers are reported without an error bar; are they from a single run or that the variance is small?
  * for supervised learning: 1) how is it trained, i.e. which digit is selected as the class label? 2) why is in-grid worse than random placement? Why is there a sudden drop of performance for in-grid when going from 12 to 16 digits?
  * why is learning multiple objects surprising, especially when the model has sufficient capability?

* (Main concern) Feature suppression:
  * Is this a result of the optimization algorithm? Can this be resolved by an initial large learning rate? e.g. Figure 3 in "Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks" by Li et. al.
  * Citation 12 (Tian et. al.) made a similar observation in their "Colorful-Moving-MNIST" (which is similar to DigitOnImageNet dataset used in this work) section with "Multiple factors shared" that sharing the digits dominates learning compared to the position of the digit, which is a form of feature suppression. I agree that this paper is the first to explicitly identify this phenomenon and emphasizes it as a problem, but would appreciate more comments from the authors about the difference, implications, and deeper explanations.

* Clarification on "Number of unique digits" (Fig 6): what does the x-axis mean? Is this the total number of distinct digits appearing in the dataset?

* The feature clustering results seem to suggest that supervised & contrastive learning learn similar features, however the feature suppression experiments (especially MultiDigit) seem to say that contrastive learning is more susceptible to being suppressed by "dominating features" than supervised learning. Is this it a result of optimization or the objective?

__Writing__: the paper is clearly written and easy to follow. I'd appreciate more details on the training procedure (e.g. the learning rates used, and the variances in different runs), but the message is clear. A minor typo: line 269 "but we leave that as...".

====== 
Post-rebuttal update: I thank the authors for their responses and keep my score of recommending an accept.
limitations_and_societal_impact: The authors discuss two limitations: first, this paper focuses only on methods with explicit negatives. This is not a problem for me since it is okay for an analysis paper to focus on one type of methods. The second limitation is that the datasets used in the experiments are not fully realistic. This again is not an issue for me, since 1) the datasets used in this paper are variants of ImageNet and MNIST which are both realistic, and 2) fully realistic datasets will make it hard to control multiple aspects of variation with precision.

I agree with the authors' judgement that there is no immediate societal impact.
review_title: null

Meta-review: The paper studies three properties of contrastive learning.  Reviewers found that although these properties were shown in experiments the paper did not explain why they happen, but thought they can guide better algorithm design in contrastive learning and of interest to a wide audience. Reviewers agreed that this is a good paper. Please include in the final manuscript the additional explanation provided during the rebuttal.  Accept 
Decision: Accept (Poster)