Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher

Authors: Yong Guo, Shulian Zhang, Haolin Pan, Jing Liu, Yulun Zhang, Jian Chen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Knowledge Distillation; Model Expansion; Reparameterization

Abstract:
Knowledge distillation aims to transfer knowledge from a large teacher model to a compact student counterpart, often coming with a significant performance gap between them. Interestingly, we find that a too-large performance gap can hamper the training process.
To alleviate this, we propose a **Gap Preserving Distillation (GPD)** method that trains an additional dynamic teacher model from scratch along with the student to maintain a reasonable performance gap. To further strengthen distillation, we develop a hard strategy by enforcing both models to share parameters. Besides, we also build the soft bidirectional mappings between them through ***Inverse Reparameterization (IR)*** and ***Channel-Branch Reparameterization (CBR)***.
IR initializes a larger dynamic teacher with approximately the same accuracy as the student to avoid a too large gap in early stage of training. CBR enables direct extraction of an effective student model from the dynamic teacher without post-training. 
In experiments, GPD significantly outperforms existing distillation methods on top of both CNNs and transformers, achieving up to 1.58\% accuracy improvement. 
Interestingly, GPD also generalizes well to the scenarios without a pre-trained teacher, including training from scratch and fine-tuning, yielding a large improvement of 1.80\% and 0.89\% on ResNet18, respectively.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729136123648

thread_type:
official_review

<thread_object>
creator: reviewer_fcyl

<content>
summary:
This paper aims to improve the effectiveness of knowledge distillation (KD) when using a teacher model with a large performance gap relative to the student model. The authors propose a method called Gap Preserving Distillation (GPD). By adopting a dynamic teacher model created through a re-parameterization operation on the student model as a proxy teacher, the performance gap is narrowed. Experimental results demonstrate the effectiveness of this approach.

strengths:
- The problem of effectively performing KD with an extremely large teacher model is important and well-motivated.
- The paper is well-organized and easy to follow.

weaknesses:
1. The abstract is too lengthy, as it seems to emphasize every aspect in detail. I suggest revising the abstract to improve clarity and conciseness.

2. The KD process involves training a dynamic teacher model alongside the student, which requires frequent re-parameterization. I am curious about the training cost compared to baseline methods.

3. In Table 6, it appears that the larger ViT-L teacher model leads to worse student performance than the smaller ResNet-101 teacher model (as with DKD). This suggests that the proposed method may still struggle with extremely large teacher models, despite it aims to overcome this issue.

4. The method's description primarily focuses on convolutional models. Although experimental results are provided for transformer models, it would be helpful to explain how re-parameterization is achieved for these models.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730433155123

thread_type:
official_review

<thread_object>
creator: reviewer_j6y5

<content>
summary:
The paper proposes Gap Preserving Distillation (GPD), a framework designed to address the capacity gap between teacher and student networks. GPD achieves this by training an additional dynamic teacher alongside the student. To further improve performance and efficiency, GPD enforces parameter sharing between the dynamic teacher and the student using Inverse Reparameterization and Channel-Branch Reparameterization techniques. Experiments on ImageNet-1K classification primarily compare GPD with baselines using static teachers, showing preliminary effectiveness.

strengths:
1. The problem is well-motivated, particularly in lines 54-61, highlighting the importance of adaptively managing the performance gap.
2. The proposed approach is novel. It co-trains a dynamic teacher to alleviate the teacher-student capacity gap and applies inverse reparameterization and channel-branch reparameterization for efficient parameter sharing.
3. The paper is well-organized and easy to follow, with clear descriptions, intuitive figures, and rigorously detailed algorithms.

weaknesses:
1. **Lack of comparisons with Knowledge Distillation (KD) methods using teacher assistants**. These methods are closely related to the proposed GPD. Without these comparisons, the effectiveness of GPD is unclear or less convincing. Suggested comparisons include, but are not limited to:
   - Mirzadeh, Seyed Iman, et al. "Improved knowledge distillation via teacher assistant." *AAAI 2020*
   - Son, Wonchul, et al. "Densely guided knowledge distillation using multiple teacher assistants." *CVPR 2021*
   - Ganta, Durga Prasad, Himel Das Gupta, and Victor S. Sheng. "Knowledge distillation via weighted ensemble of teaching assistants." *ICBK 2021*
   - Zhou, Yuhang, and Wei Ai. "Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios." *arXiv:2406.05322* (2024)

2. **Inappropriate and unfair baselines in Section 4.2**. The paper compares GPD without a static teacher against networks trained without any KD techniques. Since GPD involves an additional dynamic teacher, it would be more appropriate to compare it with self-distillation methods.

3. **Practicality and efficiency concerns**. The size of the additional dynamic teacher must still be tuned by adjusting the branch and channel expansion ratios, making GPD less practical and efficient.

questions:
1. The paper claims that the dynamic teacher shares exactly the same accuracy as the student with the reparameterization techniques. However, Figure 1 shows that the weights of the dynamic teacher are modified with added noise, which seems to contradict this claim.
2. The claim in lines 68-69, “Since DT is a larger model and often converges faster than the student,” appears counterintuitive. Is there any evidence to support this?
3. The network pairs in Table 6 do not convincingly demonstrate the effect of the teacher-student capacity gap. For instance, Table 6 shows that ViT-L performs worse than ResNet101 when distilling to a ResNet18. This may be due not to the larger capacity gap between ViT-L and ResNet18, but rather to differences in architecture. It would be more consistent to use CNN or ViT architectures for both teacher and student.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730936685846

thread_type:
official_review

<thread_object>
creator: reviewer_49em

<content>
summary:
This paper introduces Gap Preserving Distillation (GPD) to address the challenge where large performance gaps between teacher and student models hinder effective knowledge transfer. GPD introduces a dynamic teacher model that trains alongside the student, maintaining an optimal performance gap to improve transfer efficiency. The performance gains from GPD are primarily due to its use of Inverse Reparameterization (IR) and Channel-Branch Reparameterization (CBR), which create flexible connections that enable parameter sharing between the dynamic teacher and student. Experimental results show that models trained with GPD outperform those using traditional fixed teacher models and perform robustly even in scenarios without a pre-trained teacher, such as training from scratch and fine-tuning.

strengths:
1. This paper identifies the performance gap between teacher and student models as a key challenge in knowledge distillation. By introducing a dynamic adjustment mechanism, the method effectively narrows this gap, facilitating more efficient knowledge transfer.

2. The proposed GPD demonstrates adaptability across various training settings, with significant improvements across multiple models and datasets. Extensive ablation studies further validate the method’s effectiveness.

weaknesses:
1. Needs More Theoretical Explanation
This paper could benefit from a deeper explanation of why a large performance gap between the teacher and student models makes knowledge transfer less effective. Right now, the idea that a big gap harms learning is mentioned, but it would be stronger if some theoretical reasoning or tests showed how this happens. Adding experiments that test different gap sizes could also help show how different levels of teacher-student gaps impact the student model’s final performance.

2. Overengineering in the Dynamic Teacher Model 
The necessity of the complex structure designed in this paper, including Inverse Reparameterization (IR) and parameter sharing, appears somewhat disconnected from the core problem. If the primary goal is to reduce the performance gap, it remains unclear whether the proposed architecture (specifically the IR and shared parameters) is required to achieve this. Making the connection between these design choices and the main goal a bit clearer could help the paper feel more cohesive.

questions:
1. How can we better understand and measure the impact of the performance gap on knowledge transfer? The paper mentions that keeping a reasonable performance gap between teacher and student models helps with effective knowledge transfer. However, it doesn’t fully explain why a larger gap might hurt the process. How exactly does the size of this gap affect the student model’s final performance? As mentioned in the weaknesses section, adding some theory or controlled experiments here could help clarify this point and give a clearer idea of the best range for this gap.

2. As mentioned in the weaknesses, is the complex design of the dynamic teacher model really necessary? To tackle the performance gap, the paper introduces a dynamic teacher model with intricate structures like Inverse Parameterization (IR) and parameter sharing. But has the author considered more straightforward options to close the gap? If this complex setup is essential, could they explain more about how it directly addresses the problem?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732331801619

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
# Response to all reviewers and area chairs

comment:
Dear Reviewers and Area Chair,

We thank all reviewers and area chairs for their insightful comments and valuable time. We are grateful to all reviewers for reaching agreement on the following aspects:

- **The Novel Approach**:
  - "The proposed approach is **novel**. It co-trains a dynamic teacher to alleviate the teacher-student capacity gap and applies inverse reparameterization and channel-branch reparameterization for efficient parameter sharing." (Reviewer J6Y5)

- **The Sound Methodology**:
  - "The problem is **well-motivated**, particularly in lines 54-61, highlighting the importance of adaptively managing the performance gap." (Reviewer J6Y5)
  - "The problem of effectively performing KD with an extremely large teacher model is important and **well-motivated**." (Reviewer FcyL)

- **Well-Written and Organized**:
  - "The paper is **well-organized** and **easy to follow**, with **clear descriptions**, **intuitive figures**, and **rigorously detailed algorithms**." (Reviewer J6Y5)
  - "The paper is **well-organized** and **easy to follow**." (Reviewer FcyL)  

- **Comprehensive and Effective Experimental Evaluation**:
  - "...the method **effectively narrows this gap**, facilitating more efficient knowledge transfer." (Reviewer 49EM)
  - "The proposed GPD demonstrates **adaptability across various training setting**s, with **significant improvements** across multiple models and datasets. **Extensive ablation studies** further validate the method’s effectiveness." (Reviewer 49EM)
 
We have taken each comment very seriously and put tremendous effort into addressing them. Moreover, we also carefully revised the paper. Here, we offer a summary:

- We provide **theoretical analysis** and **empirical evidence** explaining why large performance gaps hinder knowledge distillation.
- We demonstrate that a performance gap of approximately 3% (achieved with expansion ratio r=3 and branch number M=6) is optimal for effective knowledge transfer.
- We provide extensive comparisons with **teacher-assistant methods** and **self-distillation approaches**, demonstrating that GPD **consistently outperforms these methods**.
- We clarify that re-parameterization incurs **negligible cost** (0.021 ms per operation). 
- We provide the training costs, showing that our GPD introduces only 33\% additional overhead, while achieving a significant 1.58% performance improvement. Our GPD offers a **favorable trade-off** between performance and training cost.
- We demonstrate **robust performance across different expansion ratios** without task-specific tuning.
- We highlight **the necessity of IR and parameter sharing** to reduce the performance gap, with IR enabling generalization and parameter sharing establishing explicit constraints.
- We analyze GPD’s **effectiveness** in mitigating extreme teacher-student gaps.
- We clarify that the added noise leads to a **negligible accuracy difference** of approximately 0.002%.
- We provide mathematical formulation and illustrations showing how our re-parameterization method **naturally extends to transformer**.

Thanks again to all reviewers and area chairs. We are eager to have a fruitful discussion with all the reviewers and we look forward to any comments, questions, and suggestions.

Best Regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732606731084

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks to all reviewers and area chairs

comment:
Dear Reviewers and Area Chairs,

We sincerely thank all reviewers for their constructive feedback and acknowledgment on our work. We are highly encouraged by the positive consensus reached by all the reviewers: Reviewer 49EM initially providing a positive assessment (score of 6), Reviewer FcyL raising the score to 6, and Reviewer J6Y5 promising to raise the score to 6 after reading our rebuttal. We have carefully revised the paper according to the concerns from all the reviewers.

We are convinced that the idea of preserving a suitable performance gap to enhance the transfer of knowledge is novel and particularly effective. We believe that our work will serve as a strong baseline for both knowledge distillation and model initialization. We will release our code to facilitate the reproducibility and future research.

We appreciate your time and consideration throughout this review process.

Best regards,

Authors of #869
</content>
</thread_object>

</thread>


Metareview:
metareview:
After discussion, the major concerns about novelty, contribution and experiments of the proposed approach were mostly solved. The main contribution lies in a dynamic teacher approach to alleviate the teacher-student capacity gap and applies inverse re-parameterization and channel-branch re-parameterization for efficient parameter sharing. The paper defines an improved teacher-assistant method for the important self-distillation problem. 

As pointed out by the reviewer, there remains a large performance gap between the teacher and student models after applying the proposed distillation approach, which is not fully explained and not well solved after rebuttal. As shown in Table 10 in the revised paper, the performance improvement upon the ImageNet benchmark over a 2022-year baseline is marginal (72.71 vs. 72.60).