Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper draws an interesting research question: why instance discrimination (ID) pretraining good for transfer learning? 
The authors dissect ID for transfer learning by extensively comparing with supervised pretraining in several task combinations, so that they attempted to empirically answer what knowledge is learned by ID and transferred, what is differences with supervised pretraining, and when it is effective according to task relationships and dataset sizes.
Based on their findings, they propose a new supervised pretraining method, which has a good trade-off for transfer learning applications, and validates with other contexts such as few-shot classification and landmark localization.

The message has been deemed by researchers, but this paper shows empirical evidence with extensive experiments. Thus, the message contained in this work is worthwhile to report to our community.


Reasons for score: 

Overall, I vote for accepting. I like the messages the authors want to convey through this work. If it presents and analyzes the effects according to task semantic relationship more specifically, it would have been a stronger paper. Also, it seems that there is room to improve the paper presentation further. 


Pros:
- The paper opens and specifies which direction the transfer learning research should go.
- Also, the findings could be extended to few-shot, semi-supervised, and fully-supervised learning regimes.
- Extensive analysis and clear summarization of their findings

Cons:
- It would have been good if the presentation of the results is better organized and summarized. The current presentation is busy.


Other comments:
- This reviewer thinks that VOC and COCO datasets could be considered as a subset of the ImageNet dataset at a semantic level. Do the authors have any insight or conclusion from their experiments that can be drawn w.r.t the semantic inclusion relationship of the datasets?
- For the proposed supervised pretraining method, the authors intentionally draw a link from exemplar SVM. However, it turns out the final proposed method is a simple modification from [Wu et al. 2018]. It seems unnecessary to make a link with exemplar SVM.
- In sec 2.2, it seems that Table 8 is a typo, which is in the supplementary material. 
review_title: An insightful paper

final_score: rating: 7: Good paper, accept
review_content: 
**Summary:**

This work aims to explore why unsupervised contrastive pretraining works as well (if not better) than the tried-and-true Supervised ImageNet classification pretraining.  They explore a number of different transfer tasks to give some intuition:
1.  Interesting findings:
 - Augmentation doesn’t make much difference for supervised transfer (from imagenet) and is essential for unsupervised transfer, with the effect monotonically increasing as more augmentations are added.
 - Dataset semantics are less important for unsupervised pretraining:  They transfer from a variety of tasks, faces, objects, etc using supervised and unsupervised pretraining.
 - Imagenet pretraining leads to greater localization errors (using analysis of Hoiem 2012), and  more generally a loss of spatial information (tested via image reconstruction)
2. Propose a new supervised pretraining method to combine the idea of unsupervised contrastive training with supervised exemplar training. 
 - Extend MoCo to use supervised labels so the loss doesn’t contrast examples from the same class.
 - This improves supervised transfer performance from ImageNet to the other tasks.  
3. Finally, they look at the impact of this new pretraining on two other tasks
 - Few shot learning: The Supervised Exemplar model outperforms the unsupervised methods, and the original cross-entropy Imagenet supervised model.
 - Facial landmark prediction: Both the unsupervised and exemplar-supervised pretraining perform similarly and outperform either training from scratch or imagenet-supervised pretraining.  This again supports the observation that imagenet classification pretraining dilutes the spatial acuity of the model.

**Positives:**
- They introduce a supervised pretraining method that can transfer better than the unsupervised method and the original supervised imagenet method.
- Overall, this work is clearly written.
- Ultimately, I do believe I have a better understanding of the differences between the supervised and unsupervised pretrained models.

**Negatives:**
- The major insight into the differences is limited:  mainly that we pay a price when the low-level information is lost by the supervised pretrained model. 

**Recommendation:**
These analysis papers are always tricky to rate -- often quite a bit of work goes into what seems like a small insight (maybe even obvious in retrospect).  However, I do think that this work is worthwhile for the community because 1) it shines light on a somewhat mysterious exciting new technique and 2) already shows how the findings are useful by using it to improve supervised pretraining, and a new vocabulary for evaluating pretraining techniques.

**Minor comments:**
Eq (2), should $v_i$ be $q_i$?

review_title: What Makes Instance Discrimination Good for Transfer Learning?

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a detailed analysis of the task transfer abilities of an self-supervised representation, instance discrimination. The paper studies, in extensive detail, how these induced representations work for different tasks. The paper also proposes a new representation learning framework.

Overall, there are a lot of details in this paper, and it must have been a tremendous of work to put together. However, the "take home" message of the paper is not very clear. There are a lot of empirical findings throughout the paper, but it is left to the reader to decide what to do with the empirical findings. Which ones are important? What lessons should the field take away from these findings? This paper would be much stronger if it was re-organized to focus on the key take-away in this paper.

A central discussion point in the paper is about low, mid, or high-level features. These features were never fully defined. What is the difference between a mid or high-level feature? 
review_title: Official Review #4

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: The paper addresses the important topic of understanding why self-supervised learning methods show very good performance when used as pretraining for fine-tuning tasks. Authors analyse in detail the difference in performance between self-supervised and supervised pretraining and propose a new method to train model which improves over standard supervised models when used as pretraining.

Strengths:

- I think the paper addresses a relevant problem. Understanding difference between self-supervised and supervised pretraining is relevant to advance in this field. I particularly like the amount of evidence the paper provides to back all the claims, and the originality of some of the experiments such as Figure 2.

- The paper is well written and motivates the issue very well. I think it's particularly interesting to question traditional training techniques such as cross-entropy training, when the models are planed to be used for a different goal.

- The insights of the transferability experiments are useful for the community as point the strengths and weaknesses of each methods. I think it's good that authors analyse many different tasks such as classification, detection and segmentation with variate datasets. 

- The proposed learning loss utilising the labels to produce the negatives is simple and seems to produce promising results according to Table 4 and Table 5. 

- The extensive supplementary materials are also useful for additional information.

Weaknesses:

- It would have been interesting to see whether this presented results comparing MoCo with supervised pretraining hold with other self-supervised methods such as SimCLR. Do authors have any intuition on that? Are some of those effects from the particularities of the MoCo training or can we generalise to all self-supervised methods?

- Some Table references might be wrong. Section 2 refers to Table 7 and 8 which are in the supplemental (probably referring to Table 1 and Table 2).

- The face landmark task is a bit outside the main story of the paper. It is introduced very late in the paper and it is not clear where the proposed loss helps. I believe authors should clarify this points.


Conclusion: I believe the paper is strong enough for publication. I think it would be good for the reader if authors clarify a bit more the face landmark section and discuss a bit how would this compare to other self-supervison methods, but overall the paper is very good. 
review_title: I think the paper addresses a relevant problem with strong experimental results. In my opinion, a very good paper. 

Meta-review: The paper aims at understanding why self-supervised/contrastive learning methods  transfer well when used as pretraining for fine-tuning downstream tasks  (compared to e.g., supervised pretraining based on the cross-entropy loss). Three reviewers recommend acceptance, whereas one reviewer recommends borderline rejection, arguing the take home message of the paper is not very clear. While this is a legitimate concern, the AC agrees with the majority that the paper does shed light on the differences between supervised and self-supervised pretraining (based on interesting empirical findings) and recommends acceptance.
Decision: Accept (Poster)