Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The post provides a very well written and concise summary of SEED, and all of the background necessary to understand it. Unfortunately there isn't much added value beyond this summary. Reviewer y1Bh points out a number of ways that this post could add new value which we agree with. Because of this lack of new content, we recommend rejection. 
review_title: Great pedagogical overview of material, but doesn't provide new insights 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The submission clearly introduces and motivates the problem of learning smaller representation models, and in the process it takes time to give a general (and largely self-contained) presentation about self supervised learning and knowledge distillation, which are prerequisite material for understanding the paper: “SEED: SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION”.
It proceeds with an accurate and well-written description of the method that the paper introduces, and it concludes by proposing a general pipeline which incorporates SEED for the training of a small student model to perform well on downstream tasks.

The blog post would improve if:
* The authors provided insight as to why the improvements are observed in the MOCO vs SEED case.
* It contained a presentation and criticism of the empirical results in the paper.
* The proposed future direction about adapting SimCLR to knowledge distillation is not convincing as the authors of SEED already draw links between their method and InfoNCE in the appendix.
* Provided arguments on why “training a student network to match these feature representations would not be effective” (in KD for SSL section).

Strengths and weaknesses    
————————————  

\+ Motivation  
\+ Method and task description   
\+ Literature review  
\+ Good reading flow  
\- Lack of empirical result presentation  

Typos  
———  

* ‘would not **be** effective’ (at the final paragraph of KD for SSL section)
* ‘with supervised knowledge distillation, **and (?)** with the downstream task’ (at the final paragraph of Self-supervised vs Supervised KD section)

Decision recommendation  
————————————  

I believe that this blog post motivates well-enough the problem, introduces relevant literature concisely but clearly, and depicts accurately the presented method. Thus, I recommend acceptance. 
review_title: Great writing and summary, lack of presentation and criticism on empirical results

Decision: Reject