Single Teacher, Multiple Perspectives: Teacher Knowledge Augmentation for Enhanced Knowledge Distillation

Authors: Md Imtiaz Hossain, Sharmen Akhter, Choong Seon Hong, Eui-Nam Huh
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: TeKAP, Teacher Knowledge Augmentation, Teacher Knowledge Perturbation, Single Teacher Multiple Perspectives, Synthetic Teacher, Knowledge Distillation, Ensemble Learning, Knowledge Transfer

Abstract:
Do diverse perspectives help students learn better? Multi-teacher knowledge distillation, which is a more effective technique than traditional single-teacher methods, supervises the student from different perspectives (i.e., teacher). While effective, multi-teacher, teacher ensemble, or teaching assistant-based approaches are computationally expensive and resource-intensive, as they require training multiple teacher networks. These concerns raise a question: can we supervise the student with diverse perspectives using only a single teacher? We, as the pioneer, demonstrate TeKAP, a novel teacher knowledge augmentation technique that generates multiple synthetic teacher knowledge by perturbing the knowledge of a single pretrained teacher i.e., Teacher Knowledge Augmentation via Perturbation, at both the feature and logit levels. These multiple augmented teachers simulate an ensemble of models together. The student model is trained on both the actual and augmented teacher knowledge, benefiting from the diversity of an ensemble without the need to train multiple teachers. TeKAP significantly reduces training time and computational resources, making it feasible for large-scale applications and easily manageable. Experimental results demonstrate that our proposed method helps existing state-of-the-art knowledge distillation techniques achieve better performance, highlighting its potential as a cost-effective alternative. The source code can be found in the supplementary.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730476825339

thread_type:
official_review

<thread_object>
creator: reviewer_fgou

<content>
summary:
The authors propose TeKAP, a novel teacher knowledge augmentation technique that generates diverse synthetic teacher knowledge by perturbing a single pretrained teacher. This plug-and-play module leverages simple perturbations to capture ensemble benefits without training multiple teachers. Experimental results demonstrate TeKAP's effectiveness in enhancing both logit and feature-based knowledge distillation methods.

strengths:
- The proposed plug-and-play module integrates seamlessly with existing KD methods, adding minimal computational burden.
- By augmenting knowledge from a single pretrained teacher network, the authors significantly reduce training time and resource demands while achieving ensemble-like effects.
- The approach is simple yet highly effective.

weaknesses:
- The proposed plug-and-play module was not well validated. Specifically, it was only applied to vanilla KD and CRD, even though there have been many advanced KD methods that can serve as baselines.
- The experiments omit numerous state-of-the-art single-teacher and multi-teacher KD methods; additional benchmark comparisons would - strengthen the evaluation.
- Details on dynamic noise perturbation are insufficient, with critical implementation information missing for reference.

questions:
-How can randomly distorted teacher logits provide diverse inter-class relationships if the distortion is truly random?
-What does h represent in Eq. 9?
-What is the scale of the random noise, and how should it be set? Detailed guidelines for noise settings are needed.
-There appears to be no discernible difference between Fig. 3(b) and Fig. 3(c).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730545131710

thread_type:
official_review

<thread_object>
creator: reviewer_zdp5

<content>
summary:
The paper proposes a new augmentation method to replace the ensemble approach for KD by adding noise to the features or logits of the teacher model. This increases the variability of predictions and reduces the generalization error.

strengths:
The proposed method is more efficient compared to other ensemble methods, and increasing the variability of the teacher's predictions is meaningful for knowledge distillation.

The paper is well-written and easy to follow.

weaknesses:
1) The paper proposes an effective method to replace ensemble approaches; however, there is a lack of comparison to other ensemble methods (such as multi-augmentations) to demonstrate its effectiveness. Additionally, TAKD is not the SOTA method (for example, DGKD [1]) and there is a lack of experimental details for TAKD. It is not clear what teacher models are used for TAKD.

2) The experiments in this paper are not sufficient, and the baselines are outdated. The proposed method only compares with vanilla KD (2015), TAKD (2020), and CRD (2019), and lacks comparisons with other new methods like DKD [2] and MLKD [3].


[1] Son, W.; Na, J.; Choi, J.; and Hwang, W. 2021. Densely guided knowledge distillation using multiple teacher assistants. In Proc. Int. Conf. on Computer Vision (ICCV)

[2] Zhao, B.; Cui, Q.; Song, R.; Qiu, Y.; and Liang, J. 2022. Decoupled Knowledge Distillation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)

[3] Jin, Y.; Wang, J.; and Lin, D. 2023. Multi-Level Logit Distillation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)

questions:
1) What is \mathcal{L}_{cel}​ in Equation 5?

2) In Equations 2 and 4, calculate the summation of the perturbation loss. Does \lambda need to be adjusted according to the number of perturbations?

3) What is the difference between the CAMs of TeKAP and the teacher in Figure 5? They look the same.

4) There is a lack of experimental details; even the learning rate and the number of training epochs are not mentioned in the paper.

5) For feature-level perturbation, which features are selected to add noise?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730763236764

thread_type:
official_review

<thread_object>
creator: reviewer_puyy

<content>
summary:
This manuscript introduces TeKAP, a novel teacher knowledge augmentation technique. It generates multiple synthetic teacher perspectives from a single pretrained teacher model by perturbing its knowledge with random noise. TeKAP operates at both the feature and logit levels, enhancing the student's generalization ability. By reducing the need for multiple teacher models, TeKAP decreases both training time and memory usage. Evaluations on standard benchmarks demonstrate TeKAP's effectiveness in improving the performance of existing knowledge distillation approaches

strengths:
1. This work uses a single pretrained teacher to simulate multiple teacher perspectives through perturbation, effectively circumventing the high computational costs of traditional multi-teacher setups.
2. The proposed method is simple yet demonstrated encouraging results.
3. The work includes a comprehensive evaluation of various aspects such as model compression, adversarial robustness, and transferability, which strengthens the credibility of the proposed method.
4. The extensive experiments also demonstrate TeKAP’s effectiveness in few shot learning and noisy data settings, suggesting a promising direction for advancing knowledge distillation.

weaknesses:
1. Despite TeKAP's impressive results, the theoretical analysis of the perturbation methods lacks depth. While Gaussian noise is introduced, there is limited discussion on the choice of perturbation parameters, such as the standard deviation, and how these settings impact the model’s performance. This omission could hinder reproducibility and generalizability of the approach. 
2. Additionally, while the experiments cover a range of baseline comparisons, the paper lacks a comprehensive evaluation against existing multi-teacher distillation methods and other state-of-the-art single-teacher methods, which would better highlight TeKAP’s relative strengths. 
3. Moreover, there is little discussion on the computational efficiency and scalability of TeKAP in practical applications, potentially raising concerns among readers regarding its feasibility in real-world scenarios.
4. Some statements are overclaimed in this manuscript. The authors should comprehensively review related works and give proper discriptions.

questions:
On page 4, the paper mentions the use of Gaussian noise for teacher perturbation but does not detail the criteria for choosing the noise parameters. How are these parameters optimized, and what is their impact on the diversity and quality of the generated teacher perspectives?
2.On page 5, the term ​ is introduced in the formula without a complete explanation or definition.
3.Is there a risk of overfitting to the perturbed features, especially when the noise parameters are not dynamically adjusted? 
4.How does TeKAP handle scenarios where certain classes are imbalanced? Is there a mechanism within the framework that ensures the augmented teachers do not bias the student towards overrepresented classes?
5.Could the following discussion be added to page 8? For instance:
1)What do these differences in inter-class correlations imply for the student's learning process?
2)How does the performance improvement of TeKAP in terms of inter-class correlation contribute to the overall effectiveness of the model?
6.In Figure 6, it is noted that the performance is best when the number of augmented teachers is 3. Does this imply that three teachers will be used in future applications? Additionally, the performance with two teachers seems normal; is there an explanation for this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731065331496

thread_type:
official_review

<thread_object>
creator: reviewer_nkek

<content>
summary:
The paper proposes a novel knowledge distillation method called TeKAP (Teacher Knowledge Augmentation via Perturbation), which generates diverse perspectives from a single teacher model. Instead of relying on multiple teacher models for supervision, TeKAP introduces diversity by perturbing both feature maps and output logits of a pretrained teacher network. This approach aims to simulate the benefits of multi-teacher distillation without the associated computational cost.

strengths:
- The paper provides thorough theoretical proof and experimental validation.
- The paper is well-structured and clear in its approach, with intriguing perspectives.
- The method proposed in the paper has a wide range of application scenarios.

weaknesses:
- There is a lack of comparison with recent multi-teacher distillation work.
- The explanation of the difference in usage scenarios between feature-level and logit-level may be insufficient..

questions:
- If more distillation methods could be included, it would be more convincing.
- I think the idea that different teacher models provide different perspectives is interesting. Would increasing the number of teacher models further improve performance?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces TeKAP, a novel knowledge distillation method that generates diverse teacher perspectives by perturbing the feature maps and logits of a single pretrained teacher model. It simulates the benefits of multi-teacher distillation while reducing computational cost, improving student model generalization, and demonstrating effectiveness on standard benchmarks.

The paper has received mixed scores: three weak positives (6, 6, 6) and one negative (5). The reviewers highlight some its strengths:

(1). Efficiency and Simplicity: The method uses a single pretrained teacher to simulate multiple teacher perspectives through perturbation, effectively circumventing the high computational costs of traditional multi-teacher setups. 

(2). Seamless Integration with Existing KD Methods: The plug-and-play module integrates well with existing knowledge distillation methods, adding minimal computational overhead. 

(3). Wide Range of Applications: The proposed method demonstrates promising results in various aspects, such as model compression, adversarial robustness, transferability, and few-shot learning, indicating its broad applicability.

(4). Clear Structure and Expression: The paper is well-structured, clear, and easy to follow, presenting intriguing perspectives.

Meanwhile, the reviewers also pointed out some key weaknesses of the paper, such as the lack of comparisons with recent multi-teacher distillation approaches and other state-of-the-art single-teacher methods, which makes it difficult to highlight the relative strengths of the proposed method. The paper also has insufficient theoretical analysis of perturbation methods and lacks sufficient details on implementation. After the rebuttal phase, most of the weaknesses have been addressed.

The final decision is acceptance based on the following primary reasons: method's efficiency and simplicity, and seamless integration with existing KD methods to improve performance. Besides, most weaknesses have been addressed after the rebuttal phase. The authors are required to include some of the improvements mentioned in the rebuttal, such as necessary experimental results and image enhancements, in the final version. Meanwhile, the authors should consider the reviewers' suggestions to further improve the quality of the final version of the paper.