Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates feature learning in contrastive learning. It argues that feature suppression occurs in contrastive learning, where the model ignores a subset of the features. It argues that by varying the difficulty of the contrastive discrimination task, it is possible to incentivize the learning of particular features, but again at the expense of other features. To deal with this feature suppression problem, the authors propose a new method called implicit feature modification, where the basic idea is to continuously remove features from the embedding representation that are being used to discriminate negatives from positives. This encourages the model to learn new, additional features to perform the discrimination task. The authors show that this scheme makes the model learn more robust features (in the sense of Ilyas et al. (2019) --ref. 22--).
main_review: I think this is a technically sound, novel contribution to the contrastive learning literature. I think the proposed method conceptually makes sense. Some of the experiments seem to be pretty well done. However, some experiments do not have any error bars (e.g. Figure 7-8, Table 1). Are those effects significant? Also, the reported improvements are rather small and the need to tune an additional hyperparameter (epsilon) makes the proposed scheme practically unappealing. I also want to point out that the authors never do a proper cross-validated experiment: they just show the results for different epsilon values (e.g. Figure 7, Table 2), which again makes me wonder if these effects would survive a properly cross-validated experiment. 

I also had a more high level objection that I wanted to mention. This is more of an objection to contrastive learning in general, rather than this particular paper, but the point is that what papers like this really show for me is that instance discrimination and similar objectives as self-supervised learning objectives are fundamentally flawed (although they happen to work quite well at the scales where we're currently testing them), because there’s no guarantee that they will learn the “intended” representation. These objectives already require quite a bit of manual data augmentation engineering to cajole the model into learning the “correct” representation and papers like this suggest that even then these methods are not entirely successful at preventing shortcut learning. So, instead of trying to patch these issues with ad-hoc fixes like this paper tries to do, I think it would be more productive over the long term to adopt more principled self-supervised learning objectives in the first place (for example, likelihood-based generative objectives), where at least in the limit of large data, we know that they will learn the right features.
limitations_and_societal_impact: I mentioned my views on the limitations of this paper above. It would be great if the authors could address my concerns about the significance of the effects and cross-validation. I also think that in their experiments, the authors could use stronger baselines to convince the readers that their sophisticated method is really needed for the reported improvements. For example, could simply adding a drop-out layer (or a similar regularizing layer) before the embedding also encourage the model to use more features and lead to similar improvements?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, the authors assess whether contrastive instance discrimination can be modified to avoid models that learn “shortcut” solutions and discard important features.  They use a theoretical approach to explain why optimizing the InfoNCE loss that is associated with instance discrimination does not necessarily avoid shortcut solutions, and demonstrate that it is possible to adjust the difficulty of instance discrimination in a relatively straightforward way to tune the tradeoff between which features are learned.  The authors propose a method called Implicit Feature Modification (IFM) which encourages encoders to use multiple different input features to discriminate between instances in a cost-efficient manner that improves generalization and reduces suppression of useful features.  They provide a variety of empirical results supporting both the reasoning behind and the practical significance of their approach.
main_review: Originality:

While I am not the most familiar with all of the related work in this area, the approach appears novel and the differences between the current work and cited work is clearly explained.  The narrative arc taking the reader from analysis of feature suppression through representation tradeoffs to motivating IFM is compelling and the arguments are well-made.  


Quality:

The submission appears technically sound.  The authors use appropriate methods to make their arguments, provide experimental justification for the claims the paper makes, and provide detailed descriptions of their experimental protocols.  The combination of theoretical analysis and empirical results that combine to tell a clear story is compelling.  The authors have an explicit limitations and broader impact section that adequately evaluates the strengths and weaknesses of their work.  I would, however, appreciate an additional discussion of the computational cost of this approach relative to the overall training process, as this seems a critical component of the work’s practical applicability that I do not remember seeing in the paper.

Clarity:

I found the paper to be extremely well-written.  However, some of the mathematical exposition is notationally dense, and it may be possible to make the paper more accessible by simplifying the presentation.

Significance:

As a practitioner in computer vision, I find these results important and would certainly expect others to use these ideas.  The fundamental problem of models that take shortcuts to achieve high levels of performance on a training set, but perform poorly in practice, is extremely common and often addressed only in ad hoc ways.  The authors present one of the first methods of which I am aware by which this issue can be systematically addressed, they motivate/analyze it theoretically, and provide compelling empirical results to support its practical significance.  The fact that the authors show improvement not only on benchmark datasets, but also on a medical imaging dataset that reflects a real application, is particularly compelling.

Detailed Comments/Questions:

L 71-75: I very much appreciate the authors’ presentation of this point.

L 92: What is the “space U” in “measure \nu on a space U”?

L 83 - 95: It may be worth describing why the authors set up the problem in this way.  It is not immediately obvious what the benefit of this formulation is at this juncture.

L 112: Nit: “number of negatives m goes to infinity” → as the reader, I did not know what m was without reading the following lines.

L112: It’s not immediately clear to me how the LFS → RHS.  Perhaps worth working out in an appendix or citing directly.
  
L 133 - 135: Another explanatory sentence here may be useful to provide intuition.

L 229: Do you evaluate sensitivity to alpha at any point, or is alpha=1 generally a good choice?
Fig. 4 caption: Isn’t this moving the negative sample towards the anchor (rather than away, as the caption says)?

L 294: It seems that the difference in linear readout accuracy b/t IFM and standard SIMCLR is very small (if not overlapping) in many cases.  I would soften the statement that it improves linear readout accuracy across all three features for all temperature settings.

Fig. 7: What does L_epsilon only mean?  Is this just not using the standard InfoNCE loss at all?  Why is this better sometimes for MoCo-v2?  Potentially worth discussing.

Table 1: Please show confidence intervals here or some measure of spread.  Point estimates are not sufficient.

L 328: Typo in “previouos”

L 337: This should be Appendix C.5 not C.6, I think.

Table 2: Some of the bolded numbers are likely not statistically significant (e.g. mMRC column).  Please indicate exactly what bold means, and ensure that it accurately reflects statistical significance in some sense.

Discussion:  Extremely short for a section entitled “discussion.”  I would suggest calling this “conclusions” and adding a bit more material here to summarize the results presented.

limitations_and_societal_impact: The authors discuss these issues in their Appendix.  One potential addition would be a discussion and/or analysis of if this approach yields different levels of performance gains (or potential losses) on some classes versus others, rather than an analysis based purely on aggregate performance metrics.  Though this addition would be valuable, the authors’ existing discussion is informative.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides a theoretical study on the cause of feature suppression in contrastive learning, and proposes "implicit feature modification" which is a theoretically grounded, yet simple and effective way of improving (avoiding) feature suppression in contrastive learning.
main_review: **Pros:**

1. Interesting theoretical analysis to explain the feature suppression phenomenon in contrastive learning.
2. Well-designed experiments to demonstrate claims made in the paper, e.g. more difficult instance discrimination task makes the model learn more difficult features but at the same time suppressing the original learned features.
3. Simplicity of IFM. No computational overheads, theoretically supported and shown to be effective on a decent variety of datasets. It is also nice to see IFM helps learn robust features.

**Cons:**

The approach is not tested on larger scale datasets such as ImageNet-1k, probably due to computational constraints. I'm curious about the impact of the approach on larger datasets, where diversity of features and amount of images may naturally help contrastive learning to learn more features. The impact of IFM would be even more if it is shown to be effective for even larger datasets.
limitations_and_societal_impact: NA.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies how to avoid shortcut solutions in contrastive learning.  To this end, this work explores feature suppression in contrastive learning and explains why shortcut solutions happen. Then, this work proposes a simple method to handle this issue based on the observation on the difficulty of instance discriminations. Experimental results show the effectiveness of the proposed method.
main_review: Positive points:
1. This paper studies an important and practical issue in contrastive learning: how to avoid shortcut solutions.
2. This paper theoretically analyzes feature suppression in contrastive learning and explains why it may happen in contrastive self-supervised learning.
3. This paper proposed a simple and efficient method based on adjusting the difficulty of instance discrimination.

Negative points:
1. The notations are two complex, which makes the paper hard to follow. Moreover, some notations seem to not be defined well. For example, what is $\bar{z}$ in Definition 1? What is $S^c$ in Line 194.

2. It is would be better to conduct experiments on ImageNet-1k. All advanced contrastive learning methods are evaluated on ImageNET-1k, which can better evaluate the effectiveness of the proposed method.

3. Whether does the proposed method contribute to siamese architectures, like BYOL and Simsiam?

Typos: "an" seems "and" in Line 172. 
limitations_and_societal_impact: Please refer to the above main review.
review_title: null

Meta-review: The reviewers agreed that this paper addresses and important practical question (feature supression in contrastive learning), and provides a theoretically-motivated and empirically-validated way to address it.
Decision: Accept (Poster)