Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an unsupervised representation learning method. It follows a teacher-student paradigm. The teacher (ResNet-50) is learned by fitting samples to clusters defined by randomly initialized prototypes and fitting different views of the same sample to the same cluster. The views are created with feature dropouts instead of random geometric transformations. The student model (ResNet-18) is distilled from the teacher online, i.e., trained together with the teacher.  Adversarial learning is also applied to align the features between the student and teachers.  
main_review: Pros: 

1.	Online distillation is relatively underexplored in the particular setting of unsupervised representation learning.
2.	The proposed method is a reasonable combination of various techniques. Though the separate components are not new, a working combination of them is always valuable. 
3.	Experimental results in distilling ResNet50 to ResNet18 demonstrate that the proposed method outperforms previous distillation-based methods significantly and outperforms other unsupervised learning methods trained directly on ResNet18.  

Cons:

1.	Online teacher-student update is not new in the teacher-student paradigm.  
2.	While the benefits of using a small network are generally justifiable, constraining a method strictly to small networks makes its potential application narrow. The absence of results with larger networks also raises the question of whether the method can work on larger networks at all. The paper could be stronger if it explores settings beyond ResNet50->ResNet18. 
3.	The online distillation and adversarial learning ideas are orthogonal to how the other aspect of unsupervised learning. That means such a distillation strategy can be combined with other unsupervised learning methods, such as SimCLR, MOCO, BYOL, SwAV. While outperforming existing offline distillation methods, the proposed method might be a better mate to a different unsupervised learning method. 

---

Thanks for the authors' response.

limitations_and_societal_impact: The paper is about general unsupervised learning. I did not see particular limitations to address.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors target an interesting and valuable problem: how to train an unsupervised/self-supervised small model by distillation large models. They proposed the OSS framework, which is equipped with a distillation strategy, a multi-view generation strategy, and an adversarial feature generation strategy. The proposed framework is shown to work better than current SOTA self-supervised frameworks and distillation methods. In some specific regimes, the proposed framework works even better than the supervised models. This work makes a valuable exploration for unsupervised/self-supervised learning with light-weight models.
main_review: This work tackles an interesting problem of unsupervised learning with light-weight models. The proposed OSS framework is effective and the experimental results could address all the claims of this paper. However, I still have some concerns regarding the model design and experimental settings.

Model Design:
> - For Eq. (5), since $\hat{z}$ are the dropout features and they are derived from $\tilde{z}$ through dropout, and during a long time of training, different versions of $\hat{z}$ could be sampled. And the goal of the multi-view generation is to make the model can equally make use of the multi-view versions of the features. In other words, we want $\hat{z}$ and $\tilde{z}$ can work equally for the model. So, why is it necessary to include a cross-entropy loss term for $\tilde{z}$, or I may want to ask why the sum of $L_{c}(\hat{z})$ and $L_{c}(\tilde{z})$ can work as such a role?  
> - For Eq. (7), the adversarial training. How did this loss backward to the main feature models? Shall we use $L_{F} = -\sum \sum (log d_{T}^{(1)} + log d_S^{(0)})$? 

Experiments:
> - In Fig. 1 and Table 1, how about the performance of the "vanilla SwAV Teacher" model and the "Supervised Teacher Model" (Baseline)? More results can help to better understanding the results and the improvements. 
> - In line 186, the authors mentioned "A plausible reason for this is that large models easily overfit to the training data, so the static teachers provide less extra knowledge beyond hard annotations", so why OSS model is not influenced by this issue?
> - In Table 6, how about the performance of the variant KD-Adv?

Typos:
> - line 2 "greatly have reduced" --> have greatly reduced
limitations_and_societal_impact: In the related work, the authors summarized the unsupervised methods into several categories: construction-based methods, prediction-based methods, cluster-based methods and contrastive methods. In this paper, the authors mainly explore the proposed OSS framework on one of the cluster-based methods. More valuable explorations may be applied to other cluster-based methods and also methods on the other categories, at least the contrastive methods may also work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a self-supervised distillation method that aims to learn better-performing representations for small models.  In particular, the teacher model is trained to produce consistent cluster assignments between different small views of the image while the student model is encouraged to mimic the prediction of the teacher. Comprehensive experiments on ImageNet and other downstream tasks validate the effectiveness of the self-supervised distillation method.
main_review: The paper presents a distillation-based self-supervised representation method, which is largely based on the previous efforts of SEED [1] and CompRess [2]. In essence, the author proposes to train both the Teacher and Student architecture, with the Teacher conducting the clustering and providing the soft labels for the Student. The author also proposes to leverage the multi-view strategies similar to [1] and [3]. The whole paper is well-organized, with comprehensive experiments and ablations, and clear paper writing. The exhibited performances are competitive and show great improvement over previous self-supervised distillation baselines and self-supervised methods. 

The concerns from the reviewer mainly come from the following aspects of incremental technical novelty comparing to [1]; and the doubts that whether the over-performing numbers come at the great training cost of additional optimizing the bulky Teacher architecture; In addition, it is much preferred to show comparisons with SEED and CompRess instead of SWAV/SimCLR. 

Specific comments are listed as follows:

1. About the novelty of contribution: It seems the proposed methods are mostly inspired and originated from [1] and [2]. In particular, SEED also adopted the `"instance dictionary", which I think has the same effect as the prototype clustering. Also, it seems [1]'s clustering is also based on the representations from the Teacher and also the multi-view strategy. Comparing to [1], what are the main differences that distinguish OSS? In other words, what are the reasons that further improve the performances? Is it the Teacher's continuing optimization jointly with the Student?

2. Concerns about Figure 1. The figure demonstrates the advantages of using the self-supervised distillation methods comparing with previous state-of-the-art SSL methods. However, as a consensus, distillation usually leads to efficient convergencies and better performances under the same training epochs. Is it a fair comparison with SSL methods instead of SEED and CompRess? 

I strongly suggest the author instead show the numbers w.r.t. to the number of epochs of SEED and CompRESS instead of SWAV/SimCLR.

3. Additional training cost. OSS shows great performance improvement over SSL and [1], [2], however the requirement to keep the Teacher model trainable during the distillation&training. I understand that the longer training epochs exist in most previous SSL baselines, while this also unavoidably narrows the OSS' scope of applications. I.e., given a generic SSL learned Teacher arch, isn't OSS sacrificing the cost of training a cumbersome Teacher for better performances? 

Suggestion: report also the offline distillation numbers when the Teacher arch is frozen and not updated. This quantifies the improvements gained from the online/offline setting.


4. It is strongly suggested to append specific discussion about the differences of OSS with CompRess instead of SimCLR when no distillation technique is leveraged. The current draft majorly compares with MoCo/v2 and other SSL methods and over-emphasizes its position in SSL, which I think is a bit one-sided. 

Suggestion: focus more on the other self-supervised distillation and reduce the statement of comparison over SSL methods.

Overall, I appreciate the authors with their efforts for comprehensive experiments and the showed performance-boosting is inspiring. I'm very prone to further update my ratings with further responses from the authors and I still hold quite positive opinions for this submission after fixing my concerns.


References

[1]. Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-supervised distillation for visual representation. In International Conference on Learning Representations (ICLR), 2021.

[2]. Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. CompRess: Self380 supervised learning by compressing representations. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.

[3]. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi  Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
limitations_and_societal_impact:  the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: All reviewers were supportive of the ideas in the paper and recommended acceptance - I agree with them. I also believe that studying small networks is an important area in itself due to the need for lightweight models on "edge devices". The reviewers have provided a number of valuable comments which the authors should integrate into their final version.
Decision: Accept (Poster)