Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper is generally concerned with self-supervised contrastive learning, such as SimCLR. The authors posit that existing ways to generate (contrastive) views introduce out-of-distribution examples that actually harm performance. To remedy this, the authors propose a latent variable model they term UOTA that, based on importance sampling, suggest good view candidates. They show UOTA's theoretical properties and conduct extensive experiments, showing gains using UOTA against a large number of recent baselines. 
main_review: The motivation for the paper is good -- contrastive learning is a hot topic recently, and as far as I'm aware there is little work describing the harm of using existing view generation / data augmentation strategies. The paper is a bit hard to follow -- the flow and general exposition can be improved, and presenting UOTA as an algorithm block could improve readability. I felt the experimental evaluation was solid -- testing using MoCo, BYOL, and SwAV, with Focal loss, Generalized Cross Entropy, MIL-NCE, Debiased, etc. I felt the ablations were complete as well. The strong and complete empirical results push me to lean in favor of acceptance, however I encourage the authors to improve the flow of the first half of the paper. I did not thoroughly check the proofs in the Appendix, and I'm not too confident about the related works on this particular topic.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents an algorithm named UOTA for self-supervised representation learning. The method is based on the observation that some low-quality negative samples are generated during the learning procedure and these bad samples can confuse the deep networks. So, a Gaussian distribution is built to estimate the weight of each sample and added to the loss term. Experiments show consistent accuracy gain.
main_review: This paper presents an algorithm named UOTA for self-supervised representation learning. The method is based on the observation that some low-quality positive samples are generated during the learning procedure and these bad samples can confuse the deep networks. So, a Gaussian distribution is built to estimate the weight of each sample and added to the loss term. Experiments show consistent accuracy gain.

The major concern of this paper lies in the technical contribution. Though some ablations are made to validate the effectiveness of introducing a Gaussian distribution to estimate the quality of positive samples, this is actually a small add-on to the existing self-supervised learning approaches. I buy the motivation, the method, and the mathematical analysis, but I just think that the contribution does not achieve the standard of NeurIPS.

To add more words, I do think that the current self-supervised learning methods rely too much on the negative samples, which is not good. However, I really expect some efforts of alleviating this issue by (1) finding key objects in the image so that the positive samples can largely preserve the quality of these key objects; or (2) finding a new proxy that bypasses the requirement of positive or negative pairs. I know that it is unfair to ask the authors to follow the suggested path, yet I just think that adding a weight to each positive sample forms a limited contribution, so I cannot suggested acceptance for now.

BTW, there are some efforts in adding stronger augmentations to achieve better performance of self-supervised learning. Do you have any comments of applying the proposed approach, UOTA, to these stronger augmented samples?
[A] Wang et al., Contrastive Learning with Stronger Augmentations, 2020.
[B] Xu et al., Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning, 2020.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This contribution is based on two main claims: i- using data augmentation to create different views from an image might result in out-of-distribution instances; and ii- employing such out-of-distribution views as positive instances in self-supervised approaches that rely on comparing representations from positive and negative views of an example, might be harmful for downstream performance. To address these issues, the authors proposed to apply an out-of-distribution detection approach to estimate how likely it is for a positive view to be out-of-distribution and use this information to weight the contribution of each positive pair when computing the learning criterion. The proposed strategy, named Unsupervised OuTlier Arbitration (UOTA), is then incorporated to three previously proposed self-supervised learning approaches and empirically validated on ImageNet and MS-COCO. Comparisons with other out-of-distribution detection techniques were also provided. Overall, UOTA showed to marginally improve the downstream performance of self-supervised pretraining both with respect to the compared detection approaches as well as self-supervised models without any mechanism to compensate for potential out-of-distribution positive views. 
main_review: The main positive aspect of this work is to raise a discussion on using out-of-distribution detection as a tool for improving self-supervised methods that rely on contrastive learning between multiple views. I also appreciated the large number of experiments and ablation studies presented in the empirical analysis. In spite of that, I think this contribution is not ready for publication, mostly due to a lack of clear evidence to support its main claims and conclusions. In addition to that, I found the manuscript quite hard to follow (it was necessary to read some sections a few times to grasp at least their main content). In the following, I further detail my concerns and provide suggestions to address them.  


- Unsupported claims: 
  
  - The authors claim in line 30 "We find these produced views even semantically deviate from the original instance and thus behave like out of distribution (OOD) samples" and in line 92  "we reveal that OOD noise is inherently existing in positive view sampling pipelines for a broad range of mainstream SSL approaches". However, it is not clear to me which part (if any) of the discussion presented in Sections 3.1, 3.2, and 3.3 supports this claim. As far as I understood, the example provided in Figure 1 is the only evidence showing that augmentation approaches for generating positive views are introducing out-of-distribution instances. To address this issue in the current version of the manuscript, experiments such as showing whether linear classifiers trained on top of features extracted by self-supervised pretrained models are able to distinguish augmented and original examples could be interesting (i.e. estimating the $\mathcal{H}$-divergence [1] between augmented and original data distributions).

  - Moreover, in line 32:  "The model therefore fail to generalize in some certain parameter region due to the interference of the OOD samples". I also couldn't find in the manuscript no piece of evidence to support such claim. How exactly out-of-distribution views are hurting generalization performance? Aren't there cases where enforcing the model to match positive views that might be considered as coming from different distributions could actually be helping the encoder to discard irrelevant information for the downstream the task?

  - Line 369: "Empirical study well corroborates our hypothesis and shows strong advantage of our UOTA approach over the state-of-the-art SSL approaches." The results are not presented along with error bars or any statistical significance analysis, therefore, I don't think it is possible to conclude from the empirical evaluation that UOTA shows a *strong* advantage with respect to previous approaches. In Table 4, for example, the improvement obtained by SwAV+UOTA was less than 1% in comparison to SwAV when training both models during 200 epochs, which might be due to other factors rather than due to the use of UOTA. To further illustrate my point, in Figure 3b of [3], it is shown that after training SwAV for 200 epochs it reaches 73.9% top-1 accuracy on ImageNet1k, surpassing SwAV+UOTA with the same budget of epochs. To address this issue, I suggest the authors to make it clear in the manuscript that the experiments do not indicate that UOTA provides a *strong* advantage with respect to the considered baselines. 


- Questions and other concerns:
  -  Surprisingly, the introduction contains no reference to support several statements related to previous work. For example, "So far, the most prevailing and effective assumption, is to force views from the same instance to be invariant in the feature space." (line 25) need to be followed by the corresponding references. 

  - It seems to me that different augmentations will shift the data distribution in different ways and this aspect should also be discussed in the manuscript. More specifically, I'm looking forward to understanding whether it would be helpful to only consider in the pipeline augmentations that generate small shifts. In case only these augmentations are used to generate positive views, how would this affect the downstream performance of the self-supervised model, *i.e.* what is the interplay between the distribution shift induced by an augmentation pipeline and the generalization capability of the self-supervised model on downstream tasks? 

  - Although I understand that contrastive learning approaches are very common for developing self-supervised methods, I believe the authors should also acknowledge in the manuscript the related literature which is based on different strategies, such as clustering. Please cite, discuss, and include for comparison in the experiments approaches such as DeepClusterv2 [2, 3] (it showed strong performance on several experimental settings, and pretrained models along with the original implementation are available on github).

  - Would the proposed importance sampling approach be also useful to weight the importance of augmentations on purely supervised tasks?  

  - Line 109: specify that $\theta$ corresponds to the model's parameters.

  - Line 126: what is an M-estimator?

   - Line 180 "A radical sampling distribution on n can simply contradict the desired prerequisite hypothesis (invariance) at hand and increases bias". It is not clear to me what radical sampling means. Please clarify. 

- Minor (not affected the final score):
  - Typo in line 277: "raining" -> training
  - Line 374: "appropriate" -> inappropriate?


[1] Ben-David, Shai, et al. "A theory of learning from different domains", 2010.

[2] Caron, Mathilde, et al. "Deep clustering for unsupervised learning of visual features", 2018.

[3] Caron, Mathilde, et al. "Unsupervised learning of visual features by contrasting cluster assignments", 2020.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper suggests that the different augmentations used as a positive pair during self-supervised training often lack the same semantic structure as the original image, and thus should be treated as an OOD sample. Using such OOD samples, they believe can hurt the downstream task performance. The aim should be to choose augmentations that offer the most data variance but are subject to the least semantic deviation. For this, they propose an importance sampling approach to weigh the loss for different augmentations differently. Empirically they obtain improvement for different downstream tasks across different datasets.
main_review: The idea of certain augmentations samples lacking semantic consistency and thus being weighed differently seems interesting. The authors do a good job of motivating the paper and showing results across different datasets and downstream tasks. The ablation studies are also nicely conducted.

My main doubts/concerns regarding the paper are the following - 

1. While it is true that strong augmentations (such as RandomResizedCrop) often create OOD samples, but even for supervised settings people have been using such strong augmentations[1]. I believe the proposed approach can be extended even for such supervised settings. Can the reviewers show results over the supervised setting too, or otherwise comment on why their approach is not suitable for this direction.

2. Another work suggests explicitly generating OOD samples by augmentations such as Jigsaw, and then treats them as negatives during SSL training [2]. Can the authors compare their results to this paper?

3. The weights computed in the initial stages might not be useful, as the features extracted from the network are not well trained. What happens if you instead use a pre-trained network to determine the weights of the augmentations. Say for instance, you take a pre-trained MoCo-v2 and use the features extracted from this network to computed the weights of samples. How does this perform?


I will be willing to increase my score if the above points are addressed.

References - 
[1] https://github.com/pytorch/examples/blob/master/imagenet/main.py 
[2] Negative Data Augmentation. Sinha et al.
limitations_and_societal_impact: Yes the authors have addressed their work's societal impact.
review_title: null

Meta-review: There was a robust discussion amongst the reviewers about the merits of this work. There was some disagreement as to whether this work's potential impact on the field of semi-supervised learning is big enough. I decided to side with the reviewers that argued that this contribution could in fact be important. The reviewers have unanimously suggested that the new experimental results be included in the final version, as well improved flow & readability.
Decision: Accept (Poster)