Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The major contribution of this paper is:
1) This paper proposes a new Knowledge Distillation / Transfer scheme called **SHAKE** (**SH**adow **K**nowl**E**dge) to bridge offline and online knowledge distillation / transfer schemes. The proposed method achieves noticeable improvements in image classification (CIFAR-100, ImageNet) tasks.
strengths_and_weaknesses: **Strengths:**
1) This paper is written clearly. It is easy to follow.
2) The proposed SHAKE framework obtains good improvements on image classification (CIFAR-100, ImageNet) benchmarks.

**Weaknesses:**
1) My major concern is that this paper lacks analysis. This paper talks about student-aware logits, but no analysis / visualization / quantitative results are provided to add merit to “student-aware” logits claim except the final KD accuracy. Given that this is purely an empirical paper, I’m not convinced regarding the student-aware logits claim. I’m happy to see that the proposed method obtains improvement, but this paper can significantly benefit from good analysis (There are ablation experiments done for CIFAR-100 which I feel is not about student-aware logits). Some useful analysis for “student-aware” logits:
- Given the proposed method shares the backbone and uses different heads for teacher and student, penultimate layer visualization / analysis could be useful (See [1, 2, 3, 4]) to understand the logits learnt by the teacher/ student using SHAKE. 

2) Object detection procedure lacks technical details: 
- For Object detection, classification is treated as a multi-label classification problem rather than a multi-class classification problem. Is KL divergence used for multi-label distillation for Object detection?
- How did the authors combine ground truth labels for bounding box, objectness score with soft-label predictions from the teacher? I.e.: If the teacher predicts 10% probability for an incorrect class (so no bounding box information in ground-truth), how did the authors perform KD subsequently? Can the authors provide loss function details?


**Minor concerns:**
1) Figure 6 is not an attention map: Given a sample and target class (usually ground truth), it shows pixel-level visual explanations. Showing the probabilities for the correct class across all 5 methods for these 3 samples could be more informative to understand Figure 6 (you can still get pixel-level explanations for wrong predictions, See [5]).

Empirical results of this paper are interesting. But given the lack of analysis, I feel that the weaknesses of this paper outweigh the strengths. I’m happy to change my opinion based on the rebuttal. 


References

[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. "When does label smoothing help?." Advances in neural information processing systems 32 (2019).

[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K. T., & Savvides, M. (2021). Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR


[3] Lukasik, M., Bhojanapalli, S., Menon, A., & Kumar, S. (2020, November). Does label smoothing mitigate label noise?. In International Conference on Machine Learning (pp. 6448-6458). PMLR.

[4] Chandrasegaran, K., Tran, N. T., Zhao, Y., & Cheung, N. M. (2022). Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?. In International Conference on Machine Learning

[5] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618-626).

questions: Please see Weaknesses section above for a list of questions. Further please consider answering the following questions:

1) Can the authors include Top-5 accuracies for Tables 2, 3?
2) I believe this is not a theoretical paper, not sure why the answer is [Yes] for questions in Checklist (2). 

limitations: Briefly mentioned in Checklist 1(C).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Offline distillation can employ existing models yet demonstrates inferior performance than online ones. This paper first empirically shows that the essential factor for their performance gap lies in the reversed distillation from student to teacher, rather than the training fashion. Offline distillation can achieve competitive performance gain by fine-tuning pre-trained teacher to adapt student with such reversed distillation. However, this fine-tuning process still costs lots of training budgets. To alleviate this dilemma, this paper proposes SHAKE, a simple yet effective SHAdow KnowlEdge transfer framework to bridge offline and online distillation, which trades the accuracy with efficiency. 
strengths_and_weaknesses: The proposed method allows the proxy teacher to share the student’s backbone but use an individual shadow head to preserve the diversity of logits representations.

SHAKE (share) presents three benefits: (1) more than 3× training acceleration than DML and fine-tuning whole teacher model. (2) No need for architecture selection costs for proxy teacher. (3) Additional accuracy gains than individual proxy teacher because knowledge inherited by proxy teachers also directly improves the representation of students’ backbone in the weight-sharing process.


In principle, SHAKE alters the chain of knowledge transfer from pre-trained teacher→student in
KD to pre-trained teacher→proxy teacherstudent.

I think the idea is interesting though there are some related works.

The conclusion from Table 1 is only based on one dataset. But the performance margin is not large. Could this be generalized?

What is the exact meaning of shadow head? More explaination and why this name is used should be given.

Statistical tests to show the performance difference of different approaches are needed.
questions: The conclusion from Table 1 is only based on one dataset. But the performance margin is not large. Could this be generalized?

What is the exact meaning of shadow head? More explaination and why this name is used should be given.
limitations: Statistical tests to show the performance difference of different approaches are needed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper shows that the merits of online knowledge distillation (DML) over the offline knowledge distillation come from the fact that the teacher model are optimized for the student model. Then, to reduce the training overhead of fine-tuning the teacher in online knowledge distillation, they propose SHAKE which distills knowledge from the teacher to the proxy teacher offline, and then perform DML between the proxy teacher and the student. Experiments on both detection and image classification are provided to demonstrate their effectiveness. 
strengths_and_weaknesses: Strengths.
1. I like the study in Line 37-49, which gives a surprising explanation of the effectiveness of online knowledge distillation.
2. The authors have given very sufficient studies on both classification and object detection, with the combination of the other KD methods, and different data augmentation settings.
3. Good paper writing.

Weakness.
1. About the motivation. In Line44-49, the authors claim that the teacher model in offline knowledge distillation does not work well because it has not been optimized for the student model. I can understand that this conclusion is drawn from the observations in Line41-43. However, even in if DML and the other offline knowledge distillation methods, the "teacher model" is trained to minimize the cross-entropy loss between teacher prediction and the labels, and the KL (of feat loss) between teacher prediction and student predictions. In these situations, the teacher model is also not optimized for the student model. To make it easy to be understood, when the teacher in DML minimizes the knowledge distillation loss, it does not optimize the loss for the student, it just optimizes it for itself. Hence, the problem that why an offline teacher works worse is still not clear.

2. In the proposed SHAKE, the teacher knowledge is firstly transferred to the proxy teacher, and then transferred to the student. Thus, it is similar to the teacher-assistant knowledge distillation (TAKD, AAAI2020)[1].  It will be better to discuss the difference and connection between SHAKE and TAKD.

3. One of the motivations of SHAKE is to reduce the fine-tuning overhead of the pretrained teacher. Recently, adapter-based methods have been proposed to efficiently fine-tune a pretrained model. It will be better to compare SHAKE with using these adapter methods for teacher fine-tuning.

4. In the detection experiments in Table 4., SHAKE+Review leads to 0.31 AP improvements than Review, which is not significant.  Besides, both KD (logit) and Review are not knowledge distillation methods for object detection. It will be better if results with detection-oriented knowledge distillation methods, such as [2-3].


[1] Improved knowledge distillation via teacher assistant. AAAI2020

[2] Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors.ICLR2020.

[3] General instance distillation for object detection. CVPR2021.


I am happy to increase my rating if my questions are answered in the rebuttal.
questions: Please refer to the weakness.
limitations: n/a
review_title: null

Meta-review: metareview: This paper addresses an interesting point: the reversed distillation from student to teacher matters the KD performance. Based on this finding, the authors propose SHAKE to make the best of the worlds of offline and online KD. The technical contribution is significant to the KD community and the authors also provide comprehensive SOTA results on various datasets and model architectures. During rebuttal, the authors addressed most of the reviewers' concerns. AC appreciates the discussions and recommends accept.
confidence: Certain
Decision: Accept