Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new KD framework named SFTN that optimizes both the teacher and the student at the same time for better knowledge transfer. The experiments show that the proposed method can improve the performance of many KD methods.
main_review: ### Pros
- The paper is well-written.
- The performance is pretty good - for different teacher-student pairs and different compression methods.
- The authors have made improvements since their ICML rejection and the added results are convincing.

### Cons
- Designing student branches is non-trivial. I would like to see more discussion about the impact of different student branch designs. This is also a weakness of the proposed method since it requires human expertise for this job.
- I'm concerned about the computational overheads. The student branches should have a large impact on the training speed and memory etc., especially when there are multiple branches (when one tries to compress a huge model). If I'm not mistaken, the overhead should be $O(n \log n)$, where $n$ is the number of intermediate features/student branches. Meanwhile, a traditional knowledge distillation method can easily scale because its overhead is $O(n)$. This is clearly a limitation that should be addressed in the paper.

### Questions for the Authors
- How to distill when the teacher and the student have the same basic blocks? For example, compressing a 12-layer BERT model to a 6-layer one (which is a classic model compression setting in NLP).
- I'm not sure if the authors heard about module replacing for model compression [1] which employs a very similar idea of training the student with the teacher. While the difference is obvious, I wonder is there any shared mechanism between these two methods?

### References
[1] https://arxiv.org/abs/2002.02925
limitations_and_societal_impact: Yes, but I hope to see more discussion about the additional computational overheads of the proposed method.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose student friendly teacher networks (SFTN) that can facilitate better knowledge transfer. Specifically, during the training of the teacher network, the snapshot of the student model is also provided. The teacher network is trained along with the student branches to learn student-friendly representations. Thereafter this trained student-aware teacher network is used for knowledge distillation. They show the efficacy of their method on CIFAR-100 and ImageNet datasets over Resnet, VGG and ShuffleNet architectures. They also do sensitivity analysis of the hyperparameters like temperature of softmax and weight of KL divergence loss.
main_review: Strengths:

- As the student is involved during training of the teacher network, the representations from these two networks are similar as evidenced by low kl divergence and high CKA values. 

- After the teacher training, any distillation method can be used on top of it. The method obtains improvements over the several distillation methods.

- The writing is clear and easy to read.

Weakness/Limitations:

- One of the key drawbacks is that the teacher network is tied to a particular student architecture. The major advantage of the knowledge distillation over other compression techniques like pruning is its architectural independence i.e. one can transfer knowledge from any architecture to any other architecture, which is being compromised here. In section 4.5, line 245, they limit their method to the student with either the same architecture or same capacity. 

- Their method is more suitable when a-priori knowledge of the student network is there during the teacher training. However, this is a very limited use case.

- In Table 3, the improvements are minor, not significant.

Queries:

- There is no BS1 block in Figure 2(a). Also, how to determine how many layers to make a block ?

limitations_and_societal_impact: One of the key limitations which authors also point out is the increase in training cost due to the inclusion of the student branch of the teacher training stage. They fix it up through the use of pretrained teacher networks. But still, the teacher networks require additional finetuning.

Suggestions:

- Mention systematic ways to modularize teacher and student networks.  Checking the impact of different modularization on distillation performance.

- Perform experiments to check the performance of SFTN (trained for a specific student architecture) on other student networks that neither have the same architecture nor the same capacity.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 1. 
This submission proposes that large models intended for distillation be regularized with auxiliary loss terms that represent a compromise between full collaborative learning, where independent teacher and student networks are trained jointly, and completely disjointed conventional knowledge distillation. The submission is an empirical paper, and considers a variety of network architectures and distillation procedures on CIFAR-100 and ImageNet. 

2. I think the main idea has the potential to be very impactful. Since this paper is entirely empirical, everything hinges on flawless execution of the experiments in order for the results to have any credence. Although the authors have clearly gone to great lengths to compare a broad variety of architectures and methods, they seem to have neglected some of the most fundamental priorities when comparing methods empirically. I would rather see a comparison between a handful of carefully tuned baselines than a large table of untuned comparisons. If my concerns can be addressed, I think this could be a very good paper. However in its current state I have enough doubt about the results that I cannot recommend acceptance.

UPDATE 09/02/2021
Due to a very strong response from the authors, I am increasing my score to 7, good paper.
main_review: 3. (Relevance/Significance)
Given recent work on distillation that seems to identify optimization as a crucial bottleneck in distillation, this submission is very relevant to the NeurIPS community. In the context of self-distillation, we *know* that the teacher representations can be perfectly emulated by an identical network, yet in practice self-distillation produces students that are very different from their teachers. Furthermore there is little evidence to suggest changing the distillation dataset, the augmentation procedures, or the optimizer can really resolve the issue  [2, 3]. Therefore papers that investigate interventions that alter the distillation loss surface in some way, either by giving the student information about intermediate activations (like FitNet) or make the teacher labels easier to match (like this paper) are very interesting. 

4. (Motivation)
The motivation to use auxiliary heads to regularize the teacher rather than an independent student network is reasonable. It is not hard to imagine that if a teacher is regularized to be similar to a small student then a ceiling on the teacher's performance is imposed that potentially negates the effects of having a large teacher in the first place. Furthermore the ceiling cannot be raised except by making the student larger, which may not be feasible if the student size is dictated by hardware constraints. Treating the student branches as part of the teacher gives more flexibility, although obviously the correspondence of the capacity of the student branches and the student network is probably still a relevant factor.

5. (Correctness)
I am concerned that the logit temperature is not treated correctly. If equations (1) and (4) are taken at face value, then the main teacher head logits are tempered not just for the auxiliary distillation loss term, but also for the standard cross-entropy loss term. I believe this is either a mistake in the implementation or an oversight in the writing; unfortunately since the code was not included I cannot tell which. Obviously for \tau=1 the mistake is fairly benign, but I think it potentially completely compromises the temperature ablation experiment. The result of the temperature ablation (\tau^* = 1) is completely opposed to similar results in distillation literature, which heightens my concern. In nearly every other similar experiment in the literature the best temperature is not 1 (see, for example, the ablations in [2] or [4]). 
This also raises a concern about what temperature was used in the vanilla KD baseline. If the vanilla KD baseline used a temperature of 1 it was almost certainly undertuned, which would be a significant error. 

6. (Empirical Rigor)

6.a) The number of architectures considered is commendable, along with the experiment with fine-tuned pre-trained teacher networks. 

6.b) The loss term coefficient ablations may be compromised by the apparent decision not to constrain the coefficients to a simplex (i.e. take the total loss to be a convex combination of the individual losses). Unless the learning rate is being adjusted, changing the scale of the loss by making unnormalized changes to the coefficients is potentially a serious confounder. In general it is simplest to simply constrain the coefficients to a simplex instead. 

6.c) Error bars appear to be given for only one experiment in line 32 of the appendix, in contradiction to the author’s response to the relevant checklist question. I would like to emphasize that section 3 of the checklist is not a mere formality. It explicitly states common expectations of empirical rigor for NeurIPS submissions. Large tables of experiments are not helpful if the reader cannot be confident that they represent the whole picture.

7. (Completeness)

The greatest omission from this submission is any attempt to explain or understand why SFTN works. Table 9 is arguably the most important experiment in the paper if the authors wish to argue that SFTN is doing something more profound than simply regularizing the teacher in some clever way, which indeed is exactly what is claimed in the first sentence of the abstract. Figure 1 in the appendix (note that the figure numbering should not be reset to avoid confusion with the main text) is also helpful by providing evidence that SFTN is not improving student accuracy simply by regularizing the teacher to generalize better. Here the authors have a natural opportunity to provide an explanation for why SFTN works, an opportunity that is not taken up. I would encourage the authors to carefully design a self-distillation experiment that gives insight into why SFTN teachers are easier to distill than normal teachers. One possible explanation is that SFTN does something similar to tempering, in that it artificially raises or lowers the entropy of the teacher labels. I would also like to see top-1 agreement reported in Table 9, since it is a simple, interpretable measure of functional correspondence between teacher and student. 

8. (Understanding of Related Work)

The idea is very similar to the auxiliary classifier heads proposed in the original Inception paper [1] (also known as auxiliary towers), a connection which is worth noting in the discussion of related work. 

---------------
References

[1] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., ... & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).

[2] Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., & Wilson, A. G. (2021). Does Knowledge Distillation Really Work?. arXiv preprint arXiv:2106.05945.

[3] Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., & Kolesnikov, A. (2021). Knowledge distillation: A good teacher is patient and consistent. arXiv preprint arXiv:2106.05237.

[4] Cho, J. H., & Hariharan, B. (2019). On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4794-4802).

limitations_and_societal_impact: 9. 
The work only explores distillation in the context of image classification.

10. 
This work does not report the sensitivity of their method to dataset shift, which is very relevant in the context of distillation, because distillation is usually motivated as a means to deploy ML models to small edge devices where dataset shift is almost inevitable. Hence it is extremely important to at least quantify the sensitivity to shift, rather than simply identify it as an unknown risk in the broader impacts section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The author presents a novel knowledge distillation approach to assist the learning of better teacher models that are friendly to students and consequently a better-performing student network. For this purpose, the purposed method is a two-stage learning schema where the teacher architecture is firstly trained jointly with the Student, and then for distillation. The learning objective during the two phases is easy while looks surprisingly effective. The experiments presented in the paper are comprehensive and sufficient. 
main_review: The method presented in the paper is easy and not investigated (as far as claimed by the author). The proposed two-stage training schema is simple while yet surprisingly effective. 

- Despite so, my major concern lies in intuitive explanations about why student-specific distillation makes the distillation more effective. In essence, the training of the Teacher is not only optimized by the original CE losses, but also to the same `"hidden space" of the Student because of an additional regularization term. I am curious about what exactly leads the Teacher better representations suitable for Students: why this KL regularization leveraging the Student block can be helpful? 
The paper indeed gives an explanation intuitively that more consistent Teacher&Student features will be approached, but in such a case, shouldn't the online distillation achieve the same effect? (i.e., allowing Teacher network to be further optimized during distillation stage for longer epochs). 

- Following my previous comment, an experiment can be: consider attaching more visualizations, or ablations studying the consistency of features across the Teacher/Student before/after SFTN, e.g.: T-SNE visualizations or statistical distance of features between T-S, comparing with the T-S without the SFTN, also the baseline of online distillation.

- From another step, adopting the two-stage training can potentially lead to further computational burdens as well as a limited applicable scope. As for most of the real scenarios, it will remain unknown to what particular student architectures will be leveraged. And choosing and training a new Teacher per student case can be luxury and costly. This also raises my concern about the applicability of the proposed student-specific learning paradigm.

- Discuss more recent ongoings bring a broader scope of the works, and is strongly suggested: 1. the neural-architecture-searching (NAS). As now the Teacher needs to be student-specific and training, it would be interesting to see if the NAS can be applied to the Teacher to search for an optimal Teacher architecture for a specific Student. 2. In addition, the recently emerging self-supervised distillation looks more appropriate&challenging for the student-specific distillation schema as the self-supervised efforts somehow reveal that consistency in self-supervised learning is critical. Some relevant works include BYOL[1], SEED[2] and etc. 

As I'm less familiar with supervised distillation, I'd much like to hear more from other reviewers & responses from authors then adjust my ratings accordingly.

[1]. Grill, Jean-Bastien, et al. "Bootstrap your own latent: A new approach to self-supervised learning." arXiv preprint arXiv:2006.07733 (2020).

[2]. Fang, Zhiyuan, et al. "Seed: Self-supervised distillation for visual representation." arXiv preprint arXiv:2101.04731 (2021).
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

Meta-review: This paper proposes an auxiliary loss that balances independent teacher and student training and joint training. It is largely an empirical submission, but helps address an interesting open question --- since it has been shown recently that the agreement between student and teacher are often very poor. The authors also put a heroic effort in their responses to reviewers. Please carefully account for all reviewer comments in your revisions. 
Decision: Accept (Poster)