Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new approach to self-supervised learning of representations, wherein similar to previous works, the objective function relates to two different views–as in prior works, augmentations here–of the same data point.  However, rather than encouraging the respective two embeddings to be similar, the approach here encourages the distribution of similarities to other data points of the two embeddings to be similar, along with a momentum update approach.  On a thorough set of empirical evaluations, the proposed approach performs strongly.
main_review: Overall, the proposed approach seems like an intuitive modification to recent self-supervised learning approaches, which performs well on a thorough range of empirical evaluations relative to recently proposed approaches, and therefore should be interesting to attendees and readers of this conference.  Some specific points:
- The proposed approach has a number of "tricks" (like the momentum approach, distribution sharpening, particular type of weak augmentation), however these appear to be thoroughly ablated in the empirical results section
- The idea of applying some objective over the matrix of pairwise distances between a data point and others in the dataset is not itself new... ideally this paper would have compared to e.g. kernel-based approaches and mapped connections there
- The writing is somewhat unclear and has grammatical issues throughout, especially making some of the upfront exposition parts of the paper a little hard to parse.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a loss function modification on top of the MoCoV2 framework and seem to provide consistent gain over the original MoCoV2 framework. To make it work better, the author also had to change how images are augmented during training.
main_review: Existing SSL (self-supervised learning) works usually use contrastive / consistency loss (either way it’s instance as class). One long-standing problem is the so-called collision problem where the model, during SSL, learns different instances as different classes while they may come from the same semantic class.

Suppose there is an image x, and x1, x2 are two augmented version of it.
The authors proposed to not directly use contrastive loss on the features of two augmented versions of the same image. Instead, they propose to first draw a similarity distribution for x1/x2 over K number of recent samples. Then use KL-divergence to align these two distributions. In various experiments, this modification to the contrastive loss seems to provide benefits to the original MoCo V2 framework.

Question 1.
Will the proposed method work for fine grained dataset such as CelebA (faces) or Birdsnap (different birds)? I’m interested to know because the foundation of the proposed loss function is: 
Given three samples, x,y,z, if x1 is similar to y but different to z, x2 should maintain such relationships.

Suppose x, y, z are three samples from three distinct dog classes. if the x-species dog is a mix breed from y and z. Then say x1 and x2 are two different crop augmentation of the x-dog. Then depending on where the crop happens (e.g., ears, paws, tail), x1 could indeed be similar to y and x2 could be actually more similar to z.

Question 2.
The SWaV number (69.1) in Table 5 seems low and I understand that the authors copied it from citation_10. But when I look at SWaV itself (https://arxiv.org/pdf/2006.09882.pdf), they report in their in Table 3with ResNet50 as 72.7 with 200 epochs. May I ask why is there such a gap?
limitations_and_societal_impact: Adequately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work focuses on self-supervised learning for image classification. Previous work assumes that images and the corresponding augmentation should have similar features (or cluster into the same class) as the objective function to train the model. In contrast, this paper proposes to make the contrastive loss (positive and negative) of two different weak augmentations of a sample to be consistency. Moreover, knowledge distillation is also adopted to sharpen the target distribution for better capturing the most important relationship. Experimental results show that the proposed approach outperforms state-of-the-art approaches, e.g, SimCLR, BYOL, SimSiam, MoCoV2, with less computation requirement.
main_review: Generally speaking, this paper is well-motivated and easy to follow. The results show promising improvement on the state-of-the-art methods. Several minor issues are listed below.
1) Some abbreviations require the full name to make them friendly for new readers, e.g., EMA.
2) Some experimental results lack of detailed descriptions. For example, Figure 3 shows the t-sne of MoCo-v2 and the proposed ReSSL. However, it is difficult to tell which one is better. The brown ones still locate cross the green ones. Most importantly, the visualization does not show the reason why the proposed approach is better. It is encouraged to show the ablation visualization, i.e., Full model, Full model without sharpening, full model using strong augmentation.
3) It is desirable to release the codes for reproducing the results.
limitations_and_societal_impact: No. I cannot find the statements of limitations in the supplementary materials, even though this paper states "Please refers to our supplementary materials" in line 402.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a self-supervised learning technique, ReSSL, which makes (1) similar samples more similar and (2) dissimilar ones be more dissimilar. Since the proposed method does not assume positive and negative samples strictly, it does not suffer from such a class collision problem. Extensive experiments demonstrate the effectiveness of the proposed method from small-scale (e.g., CIFAR) to large-scale (i.e., ImageNet) datasets.

main_review: Strengths
- The proposed idea is simple and novel.
- The proposed method demonstrates its effectiveness in both small- and large-scale benchmarks.

Weaknesses & Concerns & Suggestions
1. Few transfer learning experiments.
   - This paper only considers the PASCAL benchmark with linear evaluation for transfer learning scenarios. However, self-supervised learning aims to transfer the learned representations or whole network parameters into various downstream tasks. This is why many SSL methods are often evaluated in various transfer learning scenarios. Could you provide more transfer learning experiments, for example, linear evaluation and fine-tuning in fine-grained classification tasks [2,3], semi-supervised learning [2,3,5], and object detection/segmentation [1,4]?
2. Small number of training epochs
   - The convergence rate and final accuracy highly depend on methods. For example, as reported in [4, Appendix D and Table 4], SimCLR's accuracy is more rapidly increasing than SimSiam in CIFAR-10, but SimCLR's final accuracy (91.1% at epoch 800) is lower than SimSimam (91.8%). Therefore, the reported results in this paper only show that ReSSL has a higher convergence rate (i.e., higher efficiency) than others. I think longer training experiments would further verify ReSSL's effectiveness.
4. Ablation study
   - Student's augmentations
     - I agree that the weak augmentation is important for the teacher network, as shown in Table 3 and Table 9.
     - I wonder why this paper does not conduct a similar empirical study for student's augmentations. Like CLSA, Can ReSSL use stronger augmentations?
   - Although ReSSL achieves higher accuracy than other methods in various settings, it is hard to fully understand why the method succeed. I think more qualitative and quantitative analyses would provide a better fundamental understanding.
     - More t-SNE comparisons with negative-free approaches (e.g., SimSiam) and clustering-based ones (e.g., SwAV) on other datasets (e.g., STL10)
     - Could you provide the changes of the k-NN classification accuracy during pretraining and compare it with other methods, especially clustering-based one? In my opinion, the proposed method is somewhat similar to clustering. Hence, the method might have an advantage when using the k-NN accuracy instead of linear evaluation.
5. Is this method applicable to other domains?
   - SimCLR and MoCo have shown their effectiveness in non-vision domains such as tabular data and natural language. If ReSSL is also widely applicable, it will strengthen this paper's contribution.
6. Some important results are provided in only the supplementary material, not the main manuscript.
   - Multi-crop experiments (Appendix E)
   - Ablation study on the choice of data augmentations (Appendix C)
   - I think this paper does not use its space compactly; for example, Table 5 and Table 6 can be combined. Hence, it would strengthen this paper if the main manuscript contains all the experimental results.
7. Some concurrent works
   - There are concurrent works [6,7,8] that share similar concepts of the proposed method. It might be better if comparisons between them and ReSSL are provided. (since they are concurrent works, I think the information can be placed in the supplementary material).

[1] He et al., Momentum Contrast for Unsupervised Visual Representation Learning, CVPR 2020 \
[2] Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020 \
[3] Grill et al., Bootstrap Your Own Latent: A New Approach to Self-supervised Learning, 2020 \
[4] Chen & He, Exploring Simple Siamese Representation Learning, 2020 \
[5] Caron et al., Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, NeurIPS 2020 \
[6] Dwibedi et al., With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021 \
[7] Wang et al., Solving Inefficiency of Self-supervised Representation Learning, 2021 \
[8] Caron et al., Emerging Properties in Self-Supervised Vision Transformers, 2021

limitations_and_societal_impact: This paper does not address limitations and potential negative societal impacts.

**A suggestion for negative societal impact.**
The line of self-supervised learning works, including this paper, requires a large neural network, a huge training dataset, and a long training time. Hence, it could consume a lot of energy for training, i.e., result in some environmental issues. To solve the problems, we may need an energy-efficient approach. For example, curriculum learning would be one option for improving the efficiency of self-supervised learning.
review_title: null

Meta-review: This paper proposes a more nuanced take on contrastive self-supervised learning (SSL). Rather than frame the objective of SSL as a binary similar/not similar target, the authors propose to use multiple sampled augmentations as a similarity distribution. The goal of SSL becomes to embed the training data in a way that agrees with the distribution. The result can be viewed either as a relaxed form of contrastive SSL, in which multiple views of the same instance only need to be similar, not the same, or---as one reviewer pointed out---a kernel-based approach that operates on the similarities among instances.

While the idea is intuitively appealing, the authors show that simply plugging it into existing SSL setups gives very poor performance, so the authors provide a recipe for this alternative approach. In particular, only weak augmentations (as opposed to changing the instances heavily) leads to better results, in contrast with existing contrastive learning approaches where the augmentation must be strong to make the pretext task challenging. Experimental results show that the learned representations lead to significant gains on object classification tasks.

The reviewers generally agree that this paper is well written and makes a significant contribution. During the discussion phase, the authors introduced several follow up experiments, which the reviewers encourage them to include in the final version.
Decision: Accept (Poster)