Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper proposes a  knowledge distillation (KD) approach for self-supervised learning (SSL) with small neural network models. The authors first observe that the state-of-the-art contrastive learning-based SSL does not obtain good performance on small models, due  to the larger  model capacity required for instance discrimination. To tackle this problem, they propose a SEED, a  KD method where the smaller student model learns to mimic its larger teacher model’s similarity distribution between an instance and its augmented views, using a cross-entropy based objective. The authors perform various experiments to show that -- 1)  SEED obtains substantial improvement in SSL-based imagenet classification performance for small models as compared to SSL training without SEED, 2) the performance gains are also substantial for transfer learning on other classification tasks, 3) the performance gains are smaller for downstream tasks of object detection and instance segmentation, with performance gains reducing for the larger COCO dataset, as compared to VOC, 4) SEED is robust to choice of SSL method, and performs better than other KD approaches.

Strengths: The paper is clearly written and well-organized. The SEED approach  is well-motivated and sensible. Experimental validation and the ablation studies are quite thorough. Performance gains on classification tasks are substantial. The method is simple to implement. 

Weaknesses: 1. Performance gains on downstream tasks of detection and instance segmentation are much lower -- how would the authors propose to improve these? 2. If the primary goal is to improve SSL performance on small models, I would have liked to see more analysis on how different design choices of setting up contrastive learning affect model performance and if these could aid performance improvement, in addition to knowledge distillation. 

Questions and suggestions: 1. Adding fully-supervised baselines for small models in table 1 will be useful in understanding the gap between full supervision and SSL for these models. 2. In figure 3, does 100% (green line) represent the student network trained with 100% of labeled imagenet supervised data? It is hard to interpret what these numbers represent. 3. Minor point: Some citations, which should not be in parentheses, are in parentheses (e.g., Romero et al. page 8). Please fix this in the revision. 

review_title: Simple approach for SSL knowledge distillation with good results

final_score: rating: 7: Good paper, accept
review_content: The paper proposes to distill knowledge from large teacher networks to small student networks in self-supervised learning. Experimental results show significant improvements on small networks.

Concerns:

- During the self-supervised distillation phase, it is not clear to me only distillation loss is applied or self-supervised learning loss is combined with distillation loss for learning student network. If only distillation loss is applied, does it make sense to train a student network using both self-supervised learning loss and distillation loss, such as MoCo-v2 used in most experiments? 
- The caption of Figure 3 is confusing, it would be good to explain it more clearly.
- It seems that improvements on object detection and instance segmentation (Table 2) are relatively small compared to other experiments, are there any explanations? Could it be possible to use smaller student networks in this experiment as well? 
- In the experiment of different sizes of sample queue, does it mean that the larger the better? What is the intuition behind it?
- Strong data augmentation is needed for most of self-supervised methods, but normally for distillation, it is not common to use very strong data augmentation, why does the paper decide to use the same data augmentation for both self-supervised learning and distillation learning?
- It is quite similar to a very recent paper as shown in the following, it would be good to discuss the differences in the paper.

@inproceedings{koohpayegani2020compress,
  title={CompRess: Self-Supervised Learning by Compressing Representations},
  author={Koohpayegani, Soroush Abbasi and Tejankar, Ajinkya and Pirsiavash, Hamed},
  booktitle={NeurIPS},
  year={2020}
}
review_title: It is interesting to see distillation applied to self-supervised learning. Experimental results are comprehensive. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
The paper address the problem of knowledge distillation in self-supervised learning, where the representational knowledge from the larger model (i.e., teacher) is used to guide the learning of a smaller model (i.e, student). To achieve this, an instance queue is used to compute the similarity score between teacher model features and the feature of a given image, and the learning objective is to minimise the cross-entropy loss of the similarity between teacher and student models. The paper provides comprehensive empirical results to justify the efficacy of the proposed approach.

Justification of rating:
The paper provides a comprehensive evaluation of the proposed approach on various network architecture and downstream tasks. Overall, I feel this work does not have sufficient theoretical or algorithmic contributions. The key contribution is the idea of apply knowledge distillation for self-supervised learning.

Strengths:
+ This is the first work that addresses self-supervised learning (SSL) with knowledge distillation. It empirically shows SSL with a small model is challenging (consistent with finding from Chen et al., 2020a;b), and proposed a technique (SEED) to transfer knowledge.
+ This paper provides comprehensive experiment on image classification task (self-supervised, semi-supervised and supervised), object detection/segmentation, domain transfer, as well as provide ablation studies on various model architecture and parameters. The results show knowledge distillation is effective for self-supervised learning. 

Weakness: 
- The core novelty of this work is the idea of conduct knowledge distillation in self-supervised learning. The key weakness is that the knowledge distillation approach and the instance queue approach are previously proposed and known to the research community. This work empirically shows how it can be combined for the task on hand.

Minor comments: 
- In section 3.2, before eqn (2), it is best to change “the similarity score between $x_i$ and $\vec{d}_j$’s” to "the similarity between the extracted (teacher/student) feature $z_i$ and $\vec{d}_j$’s”. This is to avoid confusion as one might wonder how can the similarity between the input image and the $\vec{d}_j$ be computed.
- Please move the table’s caption to the top of the table.
review_title: Interesting work and solid results with various network architecture

Meta-review: There is definite consensus on this paper, with all reviewers expressing very favorable opinions. The author responses are very well articulated and address the main concerns expressed by the reviewers. The paper is very well-written and the ablation study well-executed. Some recent related work was missed in the original submission, but this was adequately addressed in rebuttal. The proposed approach is novel technique for feature representation learning. The clarifications to the manuscript and the new analyses are especially appreciated. 
Decision: Accept (Poster)