Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher
Authors: Yong Guo
Shulian Zhang
Haolin Pan
Jing Liu
Yulun Zhang
Jian Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Distillation; Model Expansion; Reparameterization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Knowledge distillation aims to transfer knowledge from a large teacher model to a compact student counterpart, often coming with a significant performance gap between them. Interestingly, we find that a too-large performance gap can hamper the training process.
To alleviate this, we propose a **Gap Preserving Distillation (GPD)** method that trains an additional dynamic teacher model from scratch along with the student to maintain a reasonable performance gap. To further strengthen distillation, we develop a hard strategy by enforcing both models to share parameters. Besides, we also build the soft bidirectional mappings between them through ***Inverse Reparameterization (IR)*** and ***Channel-Branch Reparameterization (CBR)***.
IR initializes a larger dynamic teacher with approximately the same accuracy as the student to avoid a too large gap in early stage of training. CBR enables direct extraction of an effective student model from the dynamic teacher without post-training. 
In experiments, GPD significantly outperforms existing distillation methods on top of both CNNs and transformers, achieving up to 1.58\% accuracy improvement. 
Interestingly, GPD also generalizes well to the scenarios without a pre-trained teacher, including training from scratch and fine-tuning, yielding a large improvement of 1.80\% and 0.89\% on ResNet18, respectively.

Published as a conference paper at ICLR 2025
GAP PRESERVING DISTILLATION BY BUILDING BIDI- RECTIONAL MAPPINGS WITH A DYNAMIC TEACHER
Yong Shulian Shulian Zhang? Haolin Pan', Jing Liu?, Yulun Zhang*, Jian Chen Max Planck Institute for Informatics, "South China University of Technology, Monash University, "Shanghai Jiao Tong University
gucyengcs@gmail i com, {seshul ian, seph1 @@mail www.iilsccuu.... .ww.t.eduuuuuo.u.. en, j jing. www.leonass...ocmmmmm edu, Melun100@gmali com, ellachen@scut. odu.
ABSTRACT
Knowledge distillation aims to transfer knowledge from a large teacher model compact student counterpart, often coming with significant performance gap between them. Interestingly, we find that a too-large performance gap can hamper the training process. To alleviate this, we propose Gap Preserving Distillation (GPD) method that trains an additional dynamic teacher model from scratch along with the student to maintain reasonable performance gap. To further strengthen distillation, we develop a hard strategy by enforcing both models to share pa- rameters. Besides, we also build the soft bidirectional mappings between them through Inverse Reparameterization (IR) and Gitannel-Brance Reparameteriza- tion (CBR). IR initializes a larger dynamic teacher with approximately the same accuracy as the student to avoid a too large gap in early stage of training. CBR enables direct extraction of an effective student model from the dynamic teacher without post-training. In experiments, GPD significantly outperforms existing distillation methods on top of both CNNs and transformers, achieving up to 1.58% accuracy improvement. Interestingly. GPD also generalizes well to the scenar- ios without a pre-trained teacher, including training from scratch and fine-tuning, yielding a large improvement of 1.80% and 0.89% ResNet18, ResNeet18,rsspectveell The code is available at Iatpps: /tt::///gtttt com gumm//goyygggsG
INTRODUCTION
Deep neural networks have achieved remarkable success across various domains (Gu & Dao 2024 Liu et al. 2023b Touvron et al. [203]] Wang et al. 2023 Zhang et al. 2023a). However, the high accuracy of these models often comes at the cost large model sizes. Recent studies have begun to explore methods to reduce model complexity. In parallel to model compression techniques (Liu al. 2023a Wei al. [022] Hu et al. [Xiao et al. 2023), knowledge distillation (KD) (Hinton al. 2015) offers a solution by transferring knowledge from complex, high-capacity models to simpler, more lightweight models, thereby achieving effective model compression.
The standard practice of KD is to train smaller student model to mimic the behavior or predictions
of a teacher model (Sun et al. 2024 Jin et al. 2023 Zhao et al. 2022aa] Li et al. 2023 Zhao et al.
2022a Passalis et al.) 2021 Tian al. 2020 Zagoruyko & Komodakis 20 Heo et al. 2019a Chen et al. 2021b] Yang et al.) 2021 Peng et al. 2019). However, existing methods often exploit fixed pre- trained teacher model but it does not always guide the training of student in the most effective way. To specific, has been shown that is often non-trin non-trivial for the student to obtain promising knowledgeeimproveemnnt from the teacher when there is a very large gap between the teacher and student models, especially in the early training stage. In contrast, a weaker teacher, together with a smaller performance gap from the student. has been shown be better choice (Son) et al. 2021: Yang et al. 2019b; Mirzadeh et al. 2020 Wang et al., 2022: Gao et al., (02T). In fact, can be theoretically proved by (Wang et al. (002)) that weak models "have higher mutual infor- mation regarding the input" compared to stronger teacher models, which can enhance knowledge
"Equal contribution.
Corresponning author.

---

Published as a conference paper at ICLR 2025
distillation. Interestingly, this phenomenon can be intuitively understood, just like it is hard for child to learn advanced mathematics directly from a senior professor. Instead, it would be better to teach him/her to count or learn elementary mathematics first. In other words, for distillation, student model should learn from a teacher that has an appropriate knowledggepprfformacc gap over it. Nevertheless, even with a relatively weak teacher that has small performance gap, a fixed model that is commonly used would eventually become useless since the student will gradually improve and surpass the teacher at some time. Thus, how to adaptively control the performance gap within a reasonable range throughout the whole distillation process is a critical problem.
To address this, we seek to explicitly control the performance gap to boost knowledge transfer. However, it is non-trivial to determine the best range of the performance gap, since this gap may vary significantly across different datasets and tasks. For example, a performance gap of 3% or 5% could be good for training classification models on ImageNet. But this gap will definitely not be a suitable value for training on MNIST since almost all the models have an accuracy of over 98%. More critically, if we consider other tasks, e.g. image restoration, accuracy cannot be directly used to measure the gap. Instead of seeking a universal "best gap". we propose to cast the problem of determining the best gap into building suitable gap in terms of model size between the student and the teacher. We propose to introduce a learnable dynamic teacher (DT) model besides the pre- trained static teacher model, training it from scratch together with the student. Since DT is larger model and often has higher accuracy than the student, we are able to keep a promising performance gap between DT and student during training. In addition, we hope build a stronger connection between DT and student to transfer knowledge in a more explicit way, in order to further enhance the performance. To achieve this, we develop a hard strategy for distillation that forces DT and the student to share their parameters and encourage parameter inheritance. In addition to hard strategy, we also build a soft bidirectional mapping between them via a novel reparameterization method.
In this paper, we make the following key contributions: 1) We propose a Gap Preserving Distil- lation (GPD) method that enhances distillation performance by introducing an additional dynamic teacher (DT)) We simultaneously train DT and the student to maintain a reasonable accuracy gap between them during the whole training process. In this way, it becomes possible to build a dynamic performance gap at every iteration between the student and teacher, which is essentially different from existing work. 2) We develop a hard strategy for distillation where the student and the teacher share the same set of parameters. The key idea is that, since the teacher is often easier to obtain high accuracy, it may also be easier for the student to get promising improvement if it directly inher- its well-learned parameters from the teacher, empirically shown in Table 4 3) We explicitly enhance knowledge transfer by building bidirectional mappings between DT and the student via In- verse Reparameterization (IR) and Channel-Bran the dynamic teacher model by expanding the student model with an arbitrary expansion ratio along both the channel and branch dimensions, while preserving the same accuracy as the student model. This guarantees that both DT and the student can start from the same initial point and thus avoid a too-large performance gap in early the training stage. Interestingly, IR is not designed just for distillation, but a general initialization method for building any-size pre-trained models, yielding additional contributions to the community beyond distillation. On the other hand, our CBR seeks to extract an effective student model from the shared parameters with DT without any post- post-training. Unlike existing reparameterization approaches, CBR does not equivalently transform a model into more compact one, but extracts an effective student out of DDT. 4) In experiments, GPD consis- tently outperforms existing distillation methods on top of both CNN and transformer architectures. We emphasize that GPD is very flexible in that it also generalizes well other training settings, including both training from scratch and fine-tuning, which is rarely reported by other methods.
Reparameterization (CBR). IR constructs
2 RELATED WORK
Knowledge distillation. Knowledge distillation (KD) (Hinton et al.] 2015) transfers knowledge
from a teacher to a smaller student model. Methods improve this by focusing on logits or inter- mediate features (Sun et al.] 2024; Jin et al.] 2023 Zhao et al. 2022a; Li et al.] 2023 Passalis et al., 2021; Tian et al. [0200] Zagoruyko & Komodakis 2017a; Heo et al. 2019a Chen et al. 2021b, Heo et al. 2019b; Kim et al. 2018). Standard methods prioritize fully converged teach- ers with high performance, yet the performance gap can hinder knowledge transfer (Wang et al.) 2022 Cho & Hariharan 2019 Yuan et al. 2019 Gao et al. 2021). Strategies to address this in-
2

---

Published as conference paper at ICLR 2025
(a) Reparameterization (IR)
LCR
4
W
Student.S S
Sharing Parameters
Mappings
CBR
Scaling
RD
Scaling: W
W
CcE
Addiing nois:: W;
f.
L
Channel- Reparameterization (CBR)
Dynamic Teacher 72
Scal no Layer
Layer
Static Teacher T.
Data Flow
Stop Gradient
Multipic Branches
Transtormation
Feature Map
Weight
Scaled Weight
Noise-Added Weight
Noise Added Weight with R Random Intialization
Replicated Weight
Figure 1: Overview of the proposed Gap Preserving Distillation (GPD) method. Besides the static teacher, we introduce an additional dynamic teacher and train it from scratch along with the student. The student model shares parameters with the dynamic teacher via Inverse Reparameterization (IR) and Channel-Branch Reparameterization (CBR). (a) The dynamic teacher constructed through IR (top right) from the student model. For any layer, we replicate the weights along the channel dimension to build a wider layer while introducing additional branches to construct a multi-branch architecture. In order to maintain the same accuracy the student, we only activate the first branch that contains the original student weights and zero out all the other extra branches, i.e., one-scaling and zero-scaling. (b) We extract promising student from the dynamic teacher via CBR. The ex- panded multi-branch architecture can be merged into the student's single-brance architecture using similar way proposed by OREPA (Hu al.) 2022]. After that, given an expansion ratio r, we directly extract the first 1/r parameters multiplied by scaling factor (see details in Section 3.3.1)
clude using intermediate-stage teachers (Cho & Hariharan
2019 Zhao et al. 2022b), , pre-training
student- friendly teacher model (Yang al. 2019a] Park et 2222 Dong et al. [0224) introducing
intermediate-sized assistant teachers (Mirzadeh et al.) 2020) Son et al. 2021) ) or introducing auxil-
iary networks (Gao et al.) 2021). These methods often rely on specially designed and pre-trained
intermediate models. Feature-based methods like DTSKD (Li et al., (024b) and DifKK HHang
al. 2023) focus on bridging semantic gaps denoising features. SCKD Zhu & Wang 2021
optimizes transfer using gradient similarity. Recent works refine soft labels Yuan et al. 2024| Rao
al.] 2023) or student's 's output entropy (Zhu et al.) (224a) to enhance knowledge transfer. In con-
trast, our GPD constructs trainable dynamic teacher based on the student model, maintaining an appropriate accuracy gap throughout distillation for effective knowledge transfer.
Reparameterization Structural reparameterization (Ding et al.) 222 b) has gained attention in tasks such as compact model design (Dosovitskiy et al. (202)). architecture search (Chen et al. 2019 Zhang et al. 2023c), and pruning (Ding et al.) 2020). RepVGG (Ding et al., (021b) trans- forms training-time structures into equivalent, simpler inference structures. Other methods like DiracNet (Zagoruyko & Komodakis 20175). ACB et al. 2019), DO-Conv(Car et al.) 2022) and ExpandNet (Guo et al. (020)) also achieve structural reparameterization. OREPA (Hu et al., 2022 reduces training costs by online reparameterization. In contrast, our CBR enables the extrac-
tion of an effective student model from the dynamic teacher, enhancing knowledge transfer to the compact student model.
Model expansion. Net2Net (Chen et al.) 2016) pioneered prnetron model expansion. bert2BRTT (Chen et al. (021a) applied this Transformers, with other works focusing on depth growth (Dong et al. 2020 [ et al.) 2018)) Yang et al.) 2020) Gong et al.) 2019). Staged Train- ing (Shen et al.] 2022) and LEMON (Wang et al.) 2024) further expanded both width and depth. ControlNet (Zhang al. 2023b) duplicates the model and adds zero convolution layers main- tain equivalence, freezing the original parameters for fine-tuning ControlNet (Zhang et al.) 2023b) duplicates the model, adds zero convolution layers, and freezes original parameters for fine-tuning. In contrast, our GPD expands the model and enables adaptive switching between compact and ex- panded models during training, maintaining appropriate performance gap.
3

---

Published as a conference paper at ICLR 2025
3 GAP PRESERVING DISTILLATION
In this work, we develop Gap Preserving Distillation (GPD) method that enhances knowledge distillation by preserving an appropriate performance gap between the teacher and student through- out the whole distillation process. Instead of directly defining how large the gap should be, propose to learn an additional dynamic teacher (DT) model as a proxy to maintain this gap, which is more flexible and controllable. Unlike existing methods, we build bidirectional mappings between DT and student to strengthen their connections. These two mappings can be achieved by the In- verse Reparameterization (IR) method and the Channel-Branch Reparameterization (CRB) method, respectively, The overview of our method is shown in Figure T and Algorithm [
3.1 DISTILLATION WITH DYNAMIC TEACHER
Popular KD methods often exploit static teacher T. to guide the training of the student S (Sun
ct al. 2024| Jin et al. 2023: Zhao et al., 2022a Li et al., 2023 Zhao et al.] 2022a Furlanello et al.]
2018 Zhao
2022a Romero et
2015 Passalis et al. 2021 Tian et al. 2020 Zagoruyko &
Komodakis 201 7a Heo et al. 2019a Chen et al. 2021b Heo et al. 2019b Kim et al. 2018). Unlike
them, our Gap Preserving Distillation (GPD) introduces a learnable dynamic teacher (DT) model Ta, as shown in Figure ! Moreover, we seek to boost knowledge transfer by not only optimizing the distillation objectives, but also by inheriting parameters from a better teacher model. To achieve this, we exploit parameter sharing techniques (Zhao et al., 2023 Xie et al. [022 Zhang et al., 2022 Ma et al., |022] Zhou et al. (022) and enforce Ta and S to share the same set of parameters, which turns out to be particularly effective (see effectiveness in Section 5)
As for training. besides the standard objective of KD methods, we introduce an additional loss related to DT LOPD. Let LeE be the cross-entropy loss, and S(x) denotes the student model's prediction based on input æ. We use vi()) to represent a function that extracts desired knowledge from models, which could be either logits features. LKD measures the discrepancy between the knowledge of two models. Given an image-label pair (x,y), the objective function of GPD with a dynamic teacher is formulated follows:
Lootal LCB LCe(S(z).V) v) CxD(v(((=))
LGPD
standard objoctive of KD metheds
As for LGPD, in Figure we seek to use Ta to guide the training of S and thus introduce a KD loss LKD (1))), (Tx(z))) Regarding the training of Ta. minimize both the cross-entrope loss LCe and a KD loss between Ta and T, via LKD (V (T,, Thus, LOPD becomes
LOED (cn(TT(r).)) + ACkD()(S(#)), & (Ta(z))) + Cxo (TTTa(())
Where À controls the importance of the distillation loss between the student and DT. We set À = 3 in all experiments. To avoid learning knowledge from a weaker model, we consider single-way knowledge transfer and apply dop-gradient on all the KD losses.
3.2 BUILD DYNAMIC TEACHER VIA INVERSE REPARAMETERIZATTIO
In order to build a suitable dynamic teacher, we develop an Inverse Reparameterization (IR) method to build larger model from the student with any expansion ratio r, see Figure [ (top right). To achieve this, we expand the model along both the channel and branch dimensions. One key charac- teristic is that the expanded dynamic teacher shares approximately the same accuracy the student. Notably, IR can serve as a general initialization method for building any-size pre-trained models, yielding additional contributions to the community beyond distillation.
3.2.1 CHANNEL--EEVER INVERSE REPARAMETERIZATTION
In the channel level, we seek to expand the number of channels each layer by an expansion ratio without sacrificing accuracy. To this end, we propose channel- inverse reparameterization strategy, as shown in Figure [ The key idea is to replicate the weights of the student and introduce scale factor compensate for the increased number of channels.
4

---

Published as a conference paper at ICLR 2025
Algorithm Training process of Gap Preserving Distillation (GPD).
Input: Student S, static teacher T., epochs N, step size 1), model parameters W, weight of standard
knowledge distillation loss X, knowledge function (--), training data (I, y)
Obtain dynamic teacher Ta: Wd t IR IR(W) Inverse Reparameterization
2: for = 1 to N do
3: Forward propagation using the dynamic teacher via Da Ta(T);
4: Compute gradients for dynamic teacher:
5: Gd t Vw, (ūd. y) + (ww,CDD(b((0)) (x)())))
6:
Obtain student S from dynamic teacher Ta:
7:
W f CBR W Channel- BereBBhhchhhh Reparamcterization
8:
Forward propagation using the student via U. S(x):
9:
Compute gradients for student:
G. Vw, (CE -W,,CCE(fi v) A (KD( (T)()))
11:
+V +Vw, LKD (ū.),, ), ((Ta())));
Update parameters Wd sharing between student S and dynamic teacher Ta:
13:
Wa + Wa (( + G,);
end for
JR Sieees
X₁
X2
X2
x
2G
W
W
W
h
N
2
Fg
2
20
W
2
W
W
2 6
co
W+
f
f2
f2
s
f
W RGL
layet 1)
Intermediate layer
Lastlayer
y/
y/
y/1
y/
(a) Channel expansion for each layer weight
(b) S ttudent dynamic teacher
Figure 2: Illustration of channel-level inverse reparameterization with expansion ratio of 2. (a) For the first layer, weights are scaled by 2 and replicated along the output channel dimension, ex- panding from C1 x Co to 20, x Co. For intermediate layers, weights are scaled by then replicated along both input and output dimensions, expanding from C₁ x Ct-1 1 to 2C₁ x 2C For the last layer, weights are replicated along the imput dimension, expanding from C2 C-111 to Cz, x 2CL-11 (b) Inverse re- re-arrmmeteriiing the student model (left) to construct the dynamic teacher model (right) by expanding channels from 2 to 4 following the procedures exemplified in (a), while preserving the initial input-output mapping.
Considering a student model with L convolutional layers, let W E [RCXCLIIXX be the weight of the 1-th layer. Here, Ct and Ci-1 represent the number of output and input channels, and K K denotes the kernel size. Let W E RrCxxrG, 1 K be the expanded weight matrix for layer I of the dynamic teacher, given the expansion ratio r. To better illustrate our method, layers into three groups, including the first layer, intermediate layers, and the last layer. In Figure take the expansion ratio 2 for example illustrate our method. For the first layer (1 1). we replicate W by times and scale them by 1/r. In this way, given the same input æ. the output would be L/r of the original one. Nevertheless, since the number of output channels has been extended from C1 to rC1, summing up all the channels would obtain the same value as the original output. For convenience, 1/r. For the last layer (1 = L),, the original WL is replicated r times along the input dimension, giving WL E 1.t.xrrr.IXXX...
divide all
use W BrCxxCox XCo K K to denote the weight scaled by
As for the intermediate layers, we scale Wi by 1/r and replicate the scaled weights r times along both output and input channel dimensions. yielding rC₁ output channels and rCj-1 imput channels. To avoid the trivial solution caused by symmetrrica/iddeniiaall replications during training, we introduce a small noise into each replication. The theoretical proof of the equivalence of channel-level inverse reparameterization is provided in Appendix
5

---

Published as a conference paper at ICLR 2025
Channel-Level Reparameterization
Branch-Level
Reparameterization
Student Forward Process
Scaling Layer
Scaling Layer
Dynamic Teacher Forward Process
Dynamic teacher
Expanded
Channel-Level Reparameterieed Weight
Channel Branch Reparametertted Weight
Student
Dynamic Teacher
Weight
F Forward Process
Forward Process
Figure 3: Illustration of the forward process for the student and dynamic teacher models. The dy- namic teacher performs direct forward pass, utilizing its increased capacity. The student model shares all parameters from the dynamic teacher and undergoes a two-step reparameterization pro- cess. First, channel-level reparameterizzttion adjusts the expanded channels to match the original channel dimensions of the student model. Second, branch-level reparameterization merges the ex- panded multi-branch units into single branch structure, thereby restoring the original topology of the student model while inheriting knowledge from the dynamic teacher.
3.2.2 BRANCH-LEVEL INVERSE REPARAMETERRIZATIO
Besides channel- expansion, we also seek to expand the student model along the branch dimen- sion. The branch-level inverse reparameterization aims to expand a single convolutional layer into multi-branch structure with increased capacity, while preserving the identical input- mapping. The key idea is introduce additional branches with extra convolutions but zero-out them to keep the output of multi- brati- architecture the same the single-branch counterpart.
Specifically, considering convolutional layer represented by the kernel W Rexx 1xKx where E C and Y E C xW are the input and output tensors, respectively. Thus, the convolution operation be represented by Y WX. As shown in Figure | (top right), we expand this single convolution into a multi-branch topology with M branches. The first branch consists of single convolutional layer, whose weights are initialized with the original convolution weights W, followed by a learnable linear scaling layer S E RC initialized as a vector ones. For the remaining M - branches, each branch comprises stack of convolutional layers, with their weights randomly initialized. All these branches are also followed by a learnable linear scaling layer S.. E C,, initialized with vector of zeros. In this way, the output of the multi-branch block is equivalent to the original single-branch convolution. Let fm (X) denote the computation of the m-th branch. The expanded model along the branch dimension can be formulated by
M
Y
Σ (mfm(X) = SXff WX.
(3)
3.3 PARAMETER SHARING WITH CHANNEL-BRANCE REPARAMETERIZATION
To further enhance knowledge transfer between the student S and the dynamic teacher Ta- we pro- pose a hard strategy for distillation that forces them share parameters. In this way, the student able to directly inherit well-trained parameters from larger model, i.e., dynamic teacher Tq. Based on the shared parameters, S(x) and Ta(x:) have different predictions due different forward propa- gation methods, as shown in Figure 3 Specifically, Td directly performs the forward pass, while the student conducts online reparameterization at both channel-level and branch-level.
3.3. CHANNEL-LEVER REPARAMETERIIATTIO
As for channel-level reparameterization, given expansion ratio r, directly take the first 1/r kernels and scale them by r. Interestingly, this mapping is exactly the inverse transformation of how conduct channel-level inverse parameterization (previously discussed in Section 3.2.1). Formally, let W m E C xro!- x denote the weight of the 1-th convolutional layer the m-thh branch of expanded layer, and rCL. and rCl-1 are the numbers of output and input channels, respec- tively. During training, our channel- level reparameterization strategy explicitly extracts a subset of the dynamic teacher's expanded parameters construct the student model's 's weights, enabling

---

Published as a conference paper at ICLR 2025
obtain a promising student model after training. Specifically, extract channel- slice from W m and apply a scaling operation:
W = r I: O Chal (..i.
(4)
where W E RR xxx denotes the reparameterized weight for the corresponding layer in the student model. The slicing operation |: C..:: Ch ;. extracts the first C² output chan- nels and Ch= input channels from W m. The scaling ratio ensures the extracted parameters appropriately scaled, aligning with the IR process applied during the construction of the dynamic teacher. For the first and last layers, the channel extraction is performed only over the output or input channel dimensions, respectively. Specifically, for the first layer, W m = rw [: C i. :]. while for the last layer, W W. ;, - as the last layer's weights are not scaled during IR process. Moreover, we found that parameter sharing with Batch Normalization (Ioffe & Szegedy 2015) layers requires special treatment, as detailed in Appendix D
3.3.2 BRANCH-LEVEL REPARAMETERIZATION
After channel-level reparameterizationn, we merge the expanded multi-branch units of the dy- namic teacher model into the student's single-branch structure via branch-level reparameteriza- tion. Following Hu et al. (2022), for the m-th branch that contains Lm convolutional layers W.. zation. These are then merged into a single weight W via the standard reparameterization, which is mathematically equivalent, proven in (Ding et al. (02Ib), by conducting sequential convolution operations: Wm. Wh W m ... W. Performing this for all M branches yiclds
m: W m2 mm:--y W we first obtain WT m. W m..... Whn.. through channel-level reparameteri-
W,, W W2,... ,,... W M. After that, sum up all of them to get the final new weight W -- Em= W for this layer in the student model. Through the branch reparameterization strategy, the student model effectively inherits well- trained parameters from the dynamic teacher model.
3.4 EFFECT OF THE PROPOSED DESIGNS IN CONTROLLING PERFORMANCE GAP
We highlight that the proposed designs are necessary to control the performance gap between stu- dent and teacher. Indeed, there exists a straightforward approach that uses a randomly initialized dynamic teacher without either IR or parameter sharing. In fact, it works well for training models from scratch, but it fails in finettuning scenarios since the randomly initialized teacher can eas- ily destroy the distillation of pre-trained student model. This motivates initializes the dynamic teacher to start with the same accuracy as the student, making our method effective for both training from scratch and fine-tuning settings. On the other hand, the parameter sharing mechanism is also important since it acts as a hard constraint to control the gap between student and dynamic teacher within a suitable range. In this way, is hard for the dynamic teacher to become significantly better than the student since they share/optimize the same set of parame- ters. Interestingly. parameter sharing also benefits the training of the student since the gradients come from optimizing both the dynamic teacher and the student. In fact, this has been empirically verified. Our ablation study (Table demonstrates that parameter sharing contributes an additional 0.35% accuracy improvement. Together with IR, our GPD is able to effectively maintain appropriate performance gaps throughout the entire training process.
design of IR, which
EXPERIMENTS
4.1
DISTILLATION WITH A STATIC TEACHER
(2021b) and put details in Appendix. Table T shows the consistent superiority of our GPD across diverse architecture when training from scratch with static teacher. Notably, GPD not only yields substantial accuracy improvements when combined with existing KD methods like ReviewKD and DKD, but also outperforms the lat- est state-of-the-art KD approaches. Specifically, in the ResNet34 + ResNet18 setting, GPD boosts ReviewKD from 71.61% % to 72.50%, and improves DKD by 1.01 %, reaching 72.71%. These results outperform the most recent methods. In ResNet50 MobileNet, GPD enhances ReviewKD by 0.65% and DKD by 1.58%. Similar gains are also observed in the transformer based setting RVT- S - RVT-Ti. These significant accuracy improvements across diverse architectures highlight the
We closely follow the settings of Zhao et al. (022a), Chen et al.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of the performance of various distillation methods across different architec- tures. "_" denotes the result that is not reported. - indicates a teacher model distilling knowledge to student model B. GPD consistently enhances the performance of standard distilla- tion methods across diverse architectures.
Teacher - Student
Model
ResNet34 - ResNer18
ResNet50 - MobileNet
RVT-S - RVT-Ti
Teacher Student
73.31
76.16
81.69 68.87
69.75
78.45
KD (Hinton
2015
70.66
68.58
AT
Zagoruyko & Komodaais
2017a
70.69
69.56
OFD
Hoo al.
20198
70.81
71.25
CRD
Taan et al
2020
71.17
71.37
RKD
Park ct al. 2019
70.40
68.5
WSLD
Zhou a
2021
72.04
71.52
SRRI.
Yang ct al
202T
71.73
72.49
SimKD
Chen ct
2022
71.59
72.25
DIST Tuang al.
72.07
73.24
NKD Yang
71.96
72.58
CAT-KD Guo cl
2023
71.26
72.24
KD+CTKD
71.38
71.16
MLKD Jin of al. 2025)
71.90
73.01
KD+CTKD+LS Sun et al. 2024
71.81
72.92
DKD+LSKD Sun cl al 2024
71.88
72.85
MLKD Sun el al 2024
72.08
73.22
CKD Zhu of al 2024b
72.24
72.97
ReviewKD [hen et al. 2021b
71.61
72.56
78.92
ReviewKD + GPD
72.50 (+0.89)
73.21 (+0.65)
80.01 (+1.09)
DKD Zhao et al. DKT) GPD
2022a
71.70
72.05
79.12
72.71 (+1.01)
73.63 (+1.58)
80.14 (+1.02)
Comparison of training from scratch without a static teacher model. GPD* denotes our method using only the dynamic teacher for distillation. As standalone method, our GPD consistently improves student model per- formance across diverse architectures.
Table 2:
Table Performance comparison of fine- tuming. GPD* denotes distillation using only the dynamic teacher model without the static teacher. Our method consistently outperforms the fine-tuning baseline with longer training across various architectures.
Model ResNet18
MobileNet
RVT-TI
Model
ResNet18
MobileNet
RVT-Ti
Pre-trained Model
69.75
68.87
78.45
Bascline
70.07
71.68
78.45
71.87 (+1.80)
73.07 (+1.39)
79.85 (+1.40)
Longer Training
GPD
71.12 (+0.89)
69.47 (+0.46)
78.84 (+0.23)
effectiveness of our proposed method. By introducing a dynamic teacher model to mitigate the sub- stantial gap between the student and powerful static teacher, our GPD enables the student to more effectively absorb knowledge from the teacher. This leads to substantial performance improvements when combined with existing KD methods, surpassing the most recent distillation techniques.
4.2 TRAIN FROM SCRATCH
In this experiment, we train ResNet18 and MobileNet for 100 epochs, and RVT-Ti for 300 epochs. Table 2 illustrrase the effectiveness of our GPD* method, which utilizes only the dynamic teacher for distillation, without a static teacher. Across various backbone architectures, GPD* consistently improves upon baseline models trained without knowledge distillation. For example, the ResNet- 18 model achieves significant accuracy boost from 70.07% to 71.87% with GPD*, indicating a notable 1.80% improvement. Similarly, with GPD*, the MobileNet achieves noteworthy improvement from 71.68% to 73.07%. These results highlight the versatility and effectiveness of our proposed method, which not only enhances the performance of existing KD methods when strong static teacher is available but also serves as an effective stand-alone training strategy in scenarios where pre-trained teacher models are unavailable.
4.3 MODEL
In this experiment, we fine-tune the pre-trained models for 50 epochs and set the initial learning rate 0.1x w.r.t. its base/standard value. As for GPD*, the dynamic teacher is constructed via Inverse Reparameterizatioo based on the student model itself. This process ensures that the dynamic teacher initially exhibits the same accuracy the student. In Table compared the accuracy of
8

---

Published as a conference paper at ICLR 2025
pre-trained model, GPD* consistently achieves performance improvements across different archi- tectures. Moreover, GPD* outperforms the longer training approach by up to 0.89% for R and similar performance gains are observed for MobileNet and RVT-Ti models as well.
FURTHER DISCUSSIONS
Gap preserving and parameter sharing. We investigate the contribution of the proposed accuracy gap preserving mechanism and the parameter sharing strategy in this part. The ablation study in valiiaea the critical role of these components in boosting performance. Individually, the gap preservation mechanism yields a 0.66% accuracy gain, demonstrating its effectiveness in guiding the student model's learning trajectory. When combined with parameter sharing, the synergistic effect leads to a substantial 1.01 % improvement over the baseline, underscoring the significance of these key innovations in facilitating knowledge transfer within the GPD framework.
Channel-brance reparameterization. Table 5 provides insights into the impact of different in- verse reparameterization strategies on the performance of our GPD. Both channel-level and branch- level reparameterization techniques individually contribute to performance improvements over the baseline DKD method, achieving accuracy gains of 0.86% and 0.61%, %, respectively. However, their combined application yields the highest performance boost, with remarkable 1.01% % accuracy gain. Utilizing both techniques enables the dynamic teacher to effectively guide the student model, maxi- mizing knowledge transfer within the GPD.
Table 4: Ablation studies on preserving the ac- curacy gap and parameter sharing. We take DKD as the baseline method to distill knowl- edge from ResNet34 to ResNet18. Preserving the accuracy gap alone improves performance over the baseline, and combining with pa- rameter sharing yields further gains.
Table 5: Impact of inverse reparameteriza- tion level on performance. We take DKD the baseline method to distill knowledge from ResNet34 to ResNet18. The com- bined channel-level and branch-level strategy achieves the highest accuracy.
Channel-level Branch-level
Acc. (%)
Preserve Gap
Share Param
Acc. (%)
Baseline
71.70
Baseline
71.70
72.56 (+0.86)
72.36 (+0.66)
72.31 (+0.61)
72.71 (+1.01)
72.71 (+1.01)
Branch and channel expansion ratio. Figure (left) shows that increasing the number of branches from to 6 gradually improves accuracy, peaking at M = 6. However, further increasing to 12 branches leads to reduced performance gains, indicating that too many branches may be hard to train and do not necessarily improve performance. Figure 4 (right) illustrates the impact of the channel expansion ratio r. Both r and r = 3 significantly outperform the DKD baseline, but improvement dramatically drops at 4. The main reason is that the performance gap would be very large again when we consider a large dynamic teacher. We highlight that our GPD method is highly practical and efficient, since the default setting with M 6 branches and = channel expansion ratio generalizes well to all the considered scenarios. In practice, we recommend using M and 2 and this setting generalizes well to diverse scenarios. As demonstrated Sec- tion 4 these configurations consistently yield significant improvements across various architectures and distillation scenarios. Based the above, building the dynamic teacher with M = 6 and can be used as good gap. More critically, GPD consistently enhances baseline performance across different branch and channel expansion ratios, demonstrating its robustness to parameter choices. Thus, it is unnecessary to carefully tune these hyper-parametts due to the high robustness.
Effect of teacher-student gap size. As shown in Table 6 when using increasingly larger teacher models to distill ResNet18, performance initially improves and then deteriorates. Specifically, knowledge transfer is most effective when we choose ResNet101 as the teacher, with performance gains of up to 1.16%. However, the gap further increases with the models R5sNet152 and Con- vNeXt. Base, the distillation effectiveness significantly drops. Similarly, Table 9 shows a similar phenomenon in the transformer architecture family when using larger teachers (RVT-S to ViT-L) to distill RVT-Ti. These observations across both architecture families justify that overly large per- formance gaps indeed hinder the distillation performance. We highlight that this phenomenon

---

Published as conference paper at ICLR 2025
Baseline (DKD)
Performance Gain
73.2
72.9
72.56
72.61 (+0.91)
(+1.01)
72.71
72.63
72.71 (+1.01)
(+1.17)
72.87
Accuracy (96
72.6
(+0.86)
(+0.93)
72.8
Accuracy
72.3
72.4
71.90 (+0.20)
72.0
72.0
71.7
3
12
2
4
Number of Branches (M)
Expansion Ratio (r)
Figure 4: Impact of branch expansion number and channel expansion ratio on model accuracy. Performance gains are shown above the baseline (DKD). Left: Increasing M to 6 yields significant improvements, with diminishing returns beyond that. Right: Channel expansion ratios of and show substantial gains, while ratio of 4 leads to degradation.
Impact of different teacher-student gap sizes on Table 7: Accuracy drop compar- ResNet18 distillation performance. GPD consistently improves ison when distilling ResNet18 from different teacher models
Table 6:
performance across varying teacher model sizes.
(ResNet34 vs. ViT-L).
Method
ResNet34
ResNet101
ResNet152
ConvNeXi Base
VIFL
Method
ReNNe34
(Ax- Drup)
Teacher 73.31
77.37
78.31
84.06
85.14
Teacher
73.31
85.14
Student
69.75
69.75
71.74
-1.27
DKD GPD
72.71 (+1.01)
72.90 (+1.6)
72.81 (+1.20)
72.78 (+1)22)
72.71 (+1.28)
72.71
-0.00 (72.71)
aligns well with both our empirical results and has been also observed in lot works (Son al. 2021 Yang et al.] 2019b Mirzadeh et al.) 2020). For a more intuitive illustration of the perfor- mance impact of the teacher-student gap, please refer to the Appendix G On the other hand, we highlight that our GPD effectively mitigates the issue of large teacher-student performance gaps across various teacher model sizes. As shown in Table 7 when using ViT-L instead ResNet34 as the teacher, DKD suffers a notable 0.27% accuracy drop (71.70-71143)) while GPD maintains the same accuracy (72.71%). Similarly, as presented in Table 8 when switching from ResNet-101 to ViT-L, DKD incurs a larger 0.31% % accuracy decline (71.74-71 71.43,, whereas GPD shows only a minor 0.19% drop (72.90-72.711. These results demonstrate GPD's effectiveness in handling large teacher-student gaps, ensuring stable student performance even when the teacher model is significantly larger.
Table 9: Performance of distilling RVT-Ti using trans- former teachers of varying sizes. While overly large performance gaps hinder distillation performance, GPD consistently improves performance across differ- ent teacher model capacities.
Table 8: Accuracy drop comparison when distilling ResNet18 from different teacher
models (ResNet101 ViT-L).
Method
ResNet101
VIT-L (Ace. Drop)
Teacher
77.37
85.14
Model
RVT-S
RVT-Base
VIT-L
Student
69.75
69.75
Teacher
81.69
82.51
85.15
DKD
71.74
-0.31 (71.43)
Student
78.45
78.45
78.45
DKD GPD
72.90
-0.19 (72.71)
DKD
79.12
79.
78.97
DKD + GPD
80.14 (+1.02)
80.27 (+1.10)
80.16 (+1.19)
6 CONCLUSION
In this paper, we propose Gap Preserving Distillation (GPD), a novel approach to bridging the ac- curacy gap between teacher and student models for more effective knowledge transfer. Our key contribution is the introduction of a dynamic teacher model that preserves an appropriate accuracy lead over the student during training. We propose Inverse Reparameterization losslessly expand the student model along the channel and branch dimensions, constructing the dynamic teacher with increased capacity. Furthermore, we devise parameter sharing strategy based on Channel-Brand Reparameterization, enabling the student to inherit parameters from the expanded dynamic teacher. This reduces computational costs while allowing the student to benefit from the teacher's enriched knowledge representations. The improved efficiency and performance of compact models facilitated by our approach could enable the deployment of deep learning solutions in resource resource-constrained en- vironments, thereby promoting wider accessibility to Al technologies. Comprehensive experiments on the ImageNet dataset validate the effectiveness of GPD in boosting the performance of standard knowledge distillation methods across various backbone architectures.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China (Grant No. 62376099) and Natural Science Foundation of Guangdong Province, China (Grant No. 202491515019989.
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proc. of the Computer Vision and Pattern Recognition, 2019.
Jinming Cao, Yangyan Li, Mingchao Sun. Ying Chen, Dani Lischinski, Daniel Cohen-Or, Baoquan Chen, and Changhe Tu. Do-conv: Depthwise over- parameterized convolutional layer. 1EEE Transactions on Image Processing, 31:3726-3736, 2022.
Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level residual net- works from dynamical systems view. International Conference on Learning Representations, 2018.
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. arXiv preprint 1rX1v:2110.07143, 2021a.
Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proc. of the Computer Vision and Pattern Recog- nition, 2022.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proc. of the Computer Vision and Pattern Recognition, 2021b.
Shoufa Chen, Yunpeng Chen, Shuicheng Yan, and Jiashi Feng. Efficient differentiable neural archi- tecture search with meta kernels. CoRR, abs/1912.047499 2019.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. International Conference on Learning Representations, 2016.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proc. of the
IEEE International Conference on Computer Vision, 2019.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel
skeletons for powerful enn via asymmetric convolution blocks. In Proc. of the IEEE International Conference on Computer Vision, 2019.
Xiaohan Ding, Tianxiang Hao, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Loss- less CNN channel pruning via gradient resetting and convolutional re- e--arameterizzatonn CoRR, (bbs/200.03226) 2020.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building convolution as an inception-like unit. In Proc. of the Computer Vision and Puttern Recognition, 2021a.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proc. of the Computer Vision and Patlern Recognition, 2021b.
Chengyu Dong. Liyuan Liu, Zichao Li, and Jingbo Shang. Towards adaptive residual network training: A neural-ode perspective. In International conference on machine learning, pp. 2616 2626, 2020.
Chengyu Dong. Liyuan Liu, and Jingbo Shang. Toward student-oriented teacher network training for knowledge distillation. In International Conference Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbor,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pp. 1607-1616, 2018.
Mengya Gao, Yujun Wang, and Liang Wan. Residual error based knowledge distillation. Neuro- computing, 433:154-161, 2021.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International conference on machine learning, pp. 2337-2346. PMLR, 2019.
Albert Gu and Tri Dao. Mamba: Linear-ting sequence modeling with selective state spaces, 2024.
Shuxuan Guo, Jose M. Alvarez, and Mathieu Salzmann. Expandnets: Linear over-param parammttttteiiiioonnnn to train compact convolutional networks. In Advances in Neural Information Processing Systems, volume 33, pp. 1298-1310, 2020.
Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In Proc. of the Computer Vision and Pattern Recognition, 2023.
Byeongho Hco, Jecsoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proc. of the IEEE International Conference on Computer Vision, 2019a.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proc. of the AAAI conference artificial intelligence, volume 33, pp. 3779-3787, 2019b.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1 503.02503........ 2015.
Mu Hu, Junyi Feng, Jiashen Hua, Baisheng Lai, Jianqiang Huang, Xiaojin Gong, and Xian-Sheng Hua. Online convolutional ce-parameterization. CoRR, abs/2204.00822 2022.
Wenzheng Hu, Zhengping Che, Ning Liu, Mingyang Li, Jian Tang, Changshui Zhang, and Jianqiang Wang. Catro: Channel pruning via class-aware trace ratio optimization. IEEE Transactions Neural Networks and Learning Systems, 1555):11595-11600 2024.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. Advances in Neural Information Processing Systems, 15:33716-33722 2022.
Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowl- edge diffusion for distillation. In Advances in Neural Information Processing Systems, volume 36, pp. 65299-65316, 2023.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456, 2015.
Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and II-Chul Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In Proc. of the Computer Vision and Pattern Recognition, 2021.
Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proc. of the Computer Vision and Pattern Recognition, 2023.
Jangho Kim, ScongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres-
sion via factor transfer. Advances in neural information processing systems, 31, 2018.
12

---

Published as a conference paper at ICLR 2025
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly fty native ensemble. In Advances in Neural Information Processing Systems, pp. 7528-75255 2228-753,,20 2018.
Guopeng Li, Qiang Wang. Ke Yan, Shouhong Ding, Yuan Gao, and Gui-Song Xia. Tas: Distilling arbitrary teacher and student via hybrid assistant. arXiv preprint N.KKv:2410.123444 2024a.
Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proc. of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1504 1512, 2023.
Zheng Li, Xiang Li, Lingfeng Yang. Renjie Song, Jian Yang, and Zhigeng Pan. Dual teachers for self-knowledge distillation. Pattern Recognition, 151:1 10422, 2024b.
Shih- Shih-Yang Liu, Zechun Liu, and Kwang- Ting Cheng. Oscillation free quantization for low-bit vision transformers. In Proc. of the 40th International Conference on Machine Learning, volume 202,
pp. 21813-21824 23-29 Jul 2023a.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: data-free quantization aware training for large language models. CoRR, abs/2305.17888. 2023b.
Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. In Proc. of the Computer Vision and Patlern Recognition, 2022.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proc. the Computer Vision and Pattern Recognition, 2022.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proc. of the AAAI Con- ference on Artificial Intelligence, volume 34, pp. 5191-5198, 2020.
Dae Young Park, Moon-Hyun Cha, Changwook Jeong, Daesin Kim, and Bohyung Han. Learning student-friendly teacher networks for knowledge distillation. In Advances in Neural Information Processing Systems, volume 34, pp. 13292-13303, 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proc. of the Computer Vision and Pattern Recognition, June 2019.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Probabilistic knowledge transfer for lightweight deep representation learning. IEEE Transactions on Neural Networks and Learning Systems, 32(5):2030-2033 20399 2021.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proc. of the IEEE Inter- national Conference on Computer Vision, October 2019.
Jun Rao, Xv Meng, Liang Ding, Shuhan Qi, Xuebo Liu, Min Zhang, and Dacheng Tao. Parameter- efficient and student-friende knowledge distillation. IEEE Transactions on Multimedia, 2023.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. In International Conference on Machine Learning, pp. 19893- 19908, 2022.
Ravid Shwartz- and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint
2017.
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distilla- tion using multiple teacher assistants. In Proc. of the IEEE International Conference on Computer Vision, 2021.
13

---

Published as a conference paper at ICLR 2025
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proc. of the Computer Vision and Pattern Recognition, 2024.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song. and Gao Huang. Efficient knowledge distillation from model checkpoints. In Advances in Neural Information Processing Systems, volume 35, pp- 607-619, 2022.
Chien- Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the -he- for real-time object detectors. In Proc. of the Computer Vision and Puttern Recognition, 2023.
Yite Wang, Jiahao Su, Hanlin Lu, Cong Xie, Tianyi Liu, Jianbo Yuan, Haibin Lin, Ruoyu Sun, and Hongxia Yang. LEMON: lossless model expansion. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:1 7402-17414, 2022.
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, volume 202, pp. 38087-38099, 2023.
Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, and Qi Tian. Weight-sharing neural architecture search: A battle to shrink the optimization gap. ACM Comput. Surv., 54(9):183:1-183:33 2022.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European conference on computer vision, pp. 588-604. Springer, 2020.
Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progres- sively stacking 2.0: A multi-stage layerwise training method for BERT training speedup. CoRR, abs/2011.13635, 2020.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L. Yuille. Training deep neural networks in generations: more tolerant teacher educates better students. In Proc. of the AAAI conference on artificial intelligence, pp. 5628-5635, 2019a
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L Yuille. Training deep neural networks in generations: A more tolerant teacher educates better students. In Proc. of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5628-5635, 2019b.
Jing Yang, Brais Martínez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. International Conference on Learning Represen- lutions, 2021.
Zhendong Yang, Ailing Zeng, Zhe Li. Tianke Zhang, Chun Yuan, and Yu Li. From knowledge dis- tillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proc. of the IEEE International Conference Computer Vision, 2023.
Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: teacher-free framework. CoRR, abs/19099 11723, 2019.
Mengyang Yuan, Bo Lang, and Fengnan Quan. Student-frit sundent-friedly knowledge distillation. Knowledge- Based Systems, 296:1 1 11915, 2024.
14

---

Published as a conference paper at ICLR 2025
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017a.
Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. CoRR, (bbs/1706.00388 2017b.
Hao Zhang. Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M. Ni, and Lei Zhang. Mp- former: Mask-pilote transformer for image segmentation. In Proc. of the Computer Vision and Pattern Recognition, 2023a.
Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proc. of the Computer Vision and Paltern Recognition, 2022.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proc. of the IEEE International Conference on Computer Vision, 2019.
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4388- 4403, 2021.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proc. the IEEE International Conference on Computer Vision, 2023b.
Mingyang Zhang, Xinyi Yu, Jingtao Rong, and Linlin Ou. Repnas: Searching for efficient parameterizing blocks. In IEEE International Conference on Multimedia and Expo, pp. 270-275, 2023c.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proc. of the Computer Vision and Pattern Recognition, 2022a.
Haoran Zhao, Xin Sun, Junyu Dong, Changrui Chen, and Zihe Dong. Highlight every step: Knowl- edge distillation via collaborative teaching. IEEE Trans. Cybern., 52(4):2070-2081 2022b.
Qi Zhao, Shuchang Lyu, Lijiang Chen, Binghao Liu, Ting-Bing Xu, Guangliang Cheng, and Wen- quan Feng. Learn by oneself: Exploiting weight-sharing potential in knowledge distillation guided ensemble network. IEEE Transactions on Circuits and Systems for Video Technology, 33(11):6661-661) 2023.
Yujie Zheng, Chong Wang, Chenchen Tao, Sunqi Lin, Jiangbo Qian, and Jiafei Wu. Restructuring the teacher and student in self-distillation. IEEE Transactions on Image Processing, 2024.
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang- Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In Inter- national Conference on Learning Representations, 2021.
Yuwei Zhou, Xin Wang, Hong Chen, Xuguang Duan, Chaoyu Guan, and Wenwu Zhu. Curriculum- nas: Curriculum weight-sharing neural architecture search. In International Conference Mul- timedia, pp. 6792-6801, 2022.
Songling Zhu, Ronghua Shang, Bo Yuan, Weitong Zhang, Wenjie Li, Yangyang Li, and Licheng Jiao. Dynamickd: An effective knowledge distillation via dynamic entropy correction- based dis- tillation for gap optimizing. Pattern Recognition, 153:1, 2024a.
Wencheng Zhu, Xin Zhou, Pengfei Zhu, Yu Wang, and Qinghua Hu. CKD: contrastive knowledge distillation from sample-wise perspective. CORR, abs/2404. 141009 2024b.
Yichen Zhu and Yi Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In Proc. of the IEEE International Conference Computer Vision 2021.
Yichen Zhu, Ning Liu, Zhiyuan Xu, Xin Liu, Weibin Meng, Louis Wang, Zhicai Ou, and Jian Tang. Teach less, learn more: On the undistillable classes in knowledge distillation. Advances Neural Information Processing Systems, 35:3201 1-32024, 2022.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A OVERVIEW AND OUTLINE
In this paper, we propose Gap Preserving Distillation (GPD) bridge the performance gap between large teacher and compact student models. GPD trains a dynamic teacher alongside the student, maintaining a reasonable gap throughout. We utilize parameter sharing and establish mappings via Inverse Reparameterization (IR) and Channel- Branch Reparameterization (CBR). The supple- mentary material provides detailed theoretical analyses and additional experimental information to support the main paper, organized as follows:
In Section B
we present a theoretical analysis of the teacher-student performance gap based information bottleneck theory.
In Section C
we provide mathematical proofs for Channel-Level Inverse Reparameteriza- tion in both CNNs and Transformer architectures, demonstrating the equivalence between the expanded model and the original model through detailed derivations.
In Section D we discuss our strategy for parameter sharing in batch normalization, illustrat- ing the importance of maintaining independent running statisties for student and dynamic teacher models to ensure stable training.
In Section we detail our experimental settings, covering various training scenarios in-
cluding distillation with static teachers, training from scratch and fine-tuning.
In Section we provide additional comparisons with teacher assistant methods and self- distillation approaches, demonstrating GPIY's superior performance.
In Section G
analyze how the teacher-studes capacity gap affects model performance, showing GPD's effectiveness in mitigating performance degradation with large teachers.
analyze the evolution of performance gap during training, providing in- sights into GPD's effectiveness.
Section
In Section we evaluate the computational overhead introduced by our proposed methods, providing comparative analyses that highlight the efficiency of our approach.
In Section we discuss the key advantages of GPD over existing KD methods.
B THEORETICAL ANALYSIS OF THE TEACHER-STUDENT PERFORMANCE GAP IN KNOWLEDGE DISTILLATION
The theoretical explanation for the performance gap issue has been rigorously proved by Wang al. (002) (in Section 4) and we further derive the key theoretical analysis to justify our arguments. To be specific, based on the information bottleneck (IB) theory (Shwartz-Ziv mutual information between two variables is defined as I(.,-). In this sense, the training of deep neural networks aims to maximize the mutual information I(Y: between learned features F and ground truth Y while minimizing the mutual information T(X; F) with input data X. In the context of KD, the goal of effective knowledge transfer can be expressed as retaining high mutual information between the teacher and student networks (Ahn al. 2019). Thus, the optimization goal for the student model in KD can be described as follows:
Tishby 2017), the
min (T(X; F.) (T(Y; F.) ([T(x; F.) T(X; P.)1 -
where F, and F, are the features extracted by the student and teacher networks, respectively. The terms [[x; F.) T(X; Fx) and I(Y; F,) FF)-T((- F.)] measure the information divergence between teacher and student networks across input and output representations. B and y are positive hyper- parameters to control the importance of these terms.
We highlight that highly accurate model is often over-confident, to have very large mutual infor- mation with the target I(Y; F)) but small mutual information with the input I [(; Nevertheless, I(X;F) can be viewed as type of "ddrr knowledge" (Wang et al.) 2022) that is also very impor- tant for effective KD. Here, we take the example mentioned in Wang et al. (2022) to illustrate this: "considering image with man driving car, although it may be uniquely labeled into the "car"
16

---

Published as a conference paper at ICLR 2025
category, it still contains features of the "people" category". Such weak but non-negligible features extracted from the input (measured by I(X; F)) are the most valuable knowledge for distilling stu- dent models. Intuitively, low-capacity student model should avoid becoming over-confident, which justifies why a highly accurate teacher model may hamper the distillation performance. One possible solution is maintain suitable mutual information with the input T(X;)) F).
Theoretically, for highly accurate teacher model, the mutual information with the target I(Y; very large, while the mutual information with the input [(;; F+) is relatively small. In this case, [(X; - [(X; Fs) in the second term becomes negative and I ;(X;Ft) is a constant that does not affect computing gradients for the student. Thus, we omit [(X; and the second term becomes miny)/((X; Ft) - [[;; F,)l = miny( F.) I(X;FF)
min?I((;;F.) As for
the third term, I(Y; FF) I(Y; F.) is positive because I(Y; Fi) is very large. Similarly, the term Ff) is a constant that will not affect computing gradients, we also omit it and the third term becomes miny//(YY; - T(Y; F,)l = min (Y; F.). Based on the above, the objective can be
reformulated as:
min (1 + (X; (8 +1)
(6)
This formulation demonstrates that the teacher model tends to aggressively compress the input- related information (i.e. (1 + (X( F.)), potentially causing the student model to lose valuable "dark knowledge" that is desired for effective KD. In contrast, a weak teacher model tends to have large mutual information with the target I(Y; Ft) as well large mutual information with the input T(X; Ft). In this case, I(x; F.) - I(X;F, Fx) in the second term is often positive and this term becomes min?/[(X; (X) I(X; F&)) (Ix; I(); min >I(X; F.). The
overall objective becomes
min - (1-7) (8 (Y;
Compared with the previous objective, the difference is that a highly accurate teacher accelerates the compression of the mutual information with the input (X; while weak teacher alleviates this issue. Therefore, we can reasonably assume that: more mutual information with the input data can be the main reason why a weak teacher model achieves better distillation performance than highly accurate teacher model.
This theoretical perspective motivates our proposed dynamic teacher model approach, which main- tains a reasonable performance gap to optimize the knowledge transfer process.
C PROOF OF CHANNEL-LEVEL INVERSE REPARAMETERIZATIO
C.I CHANNEL-LEVEL INVERSE REPARAMETEERIZATION FOR CNNS
To mathematically prove the equivalence of channel-level inverse reparameterization for CNNs, consider a model with three convolutions (see Figure 2 (b)), where the parameters are denoted as W 2 and W Using @ to represent the convolutional operation, the computation of this model becomes
Y 3
W & X.
Following the expansion rules in Figure (a). when performing channel-level expansion with a ratio of 2, we expand the number of output channels the first two convolutions W and W scaling them by 1/2. For the Last convolution W3, expand the number of input channels without any scaling. For simplicity, we omit the introduced noise e introduced to the expanded weights. Thus, the outputs of these three convolutions can be derived follows:
W W1/2 W
Y1
@ x
(9)
17

---

Published as a conference paper at ICLR 2025
W2/2 W
Y2
W W 8
W2/2
W
[ /2
[W//2 @ X
(10)
W2/2 /
[W2 2
W @
@ X
Y/ = [wa W7 @ Y
W & W ex
[W3
/ 8
& & W
(11)
W @ 2 @ /2 X
W & WWW
W3s @ 22 W1 x
Clearly, given the same input X, the expanded model has the same output as that of the original model, Le., Y3 Y.
C.2 CHANNEL-LEVEL INVERSE REPARAMETERIZATION FOR TRANSFORMER ARCHITECTURES
Our inverse re-parameterization (IR) can naturally extend to transformer architectures, as their key components (Feed-Forward Networks and Multi-Head Attention) primarily consist of linear layers, which can be mathematically viewed as Ixl convolutions. One key difference lies in the Multi-Head Attention (MHA) module. Note that our re-parameterization would increase the number of channels in QKV linear projections. Nevertheless, the expanded channels of QKV features/outputs are not directly used to compute attention. Instead, dimension, i.e., increasing the number of attention heads while maintaining the same dimension per head the student model. In this way, it can be guaranteed that the computation in each head the dynamic teacher still remains as consistent as possible with the one in the student. As for the ce-parameterizatioon process, we follow the strategy depicted in Figure 3 and directly omit the extra/expanded attention heads to obtain the student model from the dynamic teacher.
the expanded channels to expand the head
We illustrate this extension by considering two-head attention module with expansion ratio of 2 (omitting the noise term for clarity). As shown in Figure 5)
1) Student model: For input with two attention heads, the computation is defined as:
Head, Attention (XW? XW:K XW.)
Attention Qi, K,, Vi)
(12)
softmax (K.))
V
ddg
The final output is then computed by concatenating the attention heads and applying projection:
= Concat(Head), Head2)
(13)
2) Expanded Model: Following our re-parameterization principle, we duplicate and scale the Query, Key, and Value projection matrices: [w W [w /2 w:K/2)
WV WY/2). These duplicated weights form new attention heads, doubling the number of heads while maintaining the same dimension per head. Consequently, the resulting Query. Key, and Value are halved to those of the student model. To preserve mapping equivalence, the Query and Key are scaled by 2 before the dot product ensure identical softmax output, while the Value
18

---

Published as a conference paper at ICLR 2025
Projection
Projection
MotMul
MatMul
MatMul
MatMul
MatMul
MatMul
Softmma
Saftmax
Softmax
Softmax
Softmax
Softmax
MatMul
MatMul
MatMul
01
K2
V/2
W
W1
10
W2
W:
72
W
W
Figure 5: Illustration of channel-level inverse reparamcterization in Multi-Head Attention (MHA). The student model (left) with 2 heads is expanded to 4 heads (right) by duplicating and scaling the Query, Key, and Value projection matrices, while preserving the original input-output mapping.
remains halved to maintain proper scaling of each head's output. This results in:
Head, Attention Attention(X(WW//222222 /2),
/2)2, X(W /2))
Attention Qi, Ki, V1/2)
(14)
softmax Q((Ki) ddk
V2/2
Head,/2
To demonstrate the equivalence between expanded model and student model, we consider the out- put projection as the last layer. Following the last-layer re-parameterization principle. the input dimension is expanded by duplicating Wo without scaling (as detailed in Section 3.3.1). resulting
W
This leads to the following equivalent final output:
wo
Wo wo
Y' = Concat(Head', Head',, Head', Head')
Woj
= Concat(Head, Headg/2 Head, / Headz/2)
Wo
(15)
Wo
Concat Concat(Head,,co Head, )/2] [Concat( Connct(Head,,, H2a
Wo
D PARAMETER SHARING FOR BATCH NORMALIZATION
Batch normalization plays crucial role in alleviating the vanishing gradient problem during the training of deep neural networks, ensuring stable convergence. However, since the student and dy- namic teacher models may exhibit different data distributions during the training process, directly sharing the running statisties (i.e., running mean and running variance) the batch normalization layers would lead to mutual interference between the two models, potentially causing training insta- bility. To avoid this issue, we propose maintaining independent running means and variances (ja., o?) and (jt, o for the student and dynamic teacher models, rather than sharing the statistics.
As illustrated in Figure 6 each model independently calculates its batch mean and batch variance from the input data, denoted as (H,,, o for the student model and (μ4 of) for the dynamic teacher model. Subsequently, normalization is performed using these statistics, and the respective running statistics (j.,, σ.) and (ft, o?) are updated accordingly. The normalized outputs are then scaled
19

---

Published as a conference paper at ICLR 2025
X
Mg 0
44
Y/2
7X
+
B
vo,"
M
Channel Leval Reperamelertzation
R. o
14
x=
Figure 6: Illustration of parameter sharing for batch normalization in the student and dynamic teacher models. We maintain a separate set of running statistics for each model due to the dis- tribution difference.
and shifted using the learnable parameters 7 and B, which shared across the models through the channel ensemble strategy. This design effectively eliminates potential instabilities arising from sharing statistics, thereby guaranteeing the robustness of the entire training process.
E EXPERIMENTAL SETTINGS
E. DISTILLATION WITH A STATIC TEACHER
In this experiment, adopt the standard data pre-processing pipeline, including random cropping, resizing to
random horizontal flipping, and normalization. By default, we employ the SGD optimizer with initial learning rate of 0.1 and momentum of 0.9. For convolutional neural networks, the batch size is set to 256 on 4 Nvidia Tesla V100 GPUs, while for vision transformers (ViTs), the batch size is set to 256 8 Nvidia Tesla V100 GPUs. The models are trained for 100 epochs with learning rate decay factor of 0.1 applied every 30 epochs. The weight decay is set to le-4, and the weight for the KD loss between the student and the dynamic teacher set 3.0, while the weights for the other KD losses and the cross-entrope loss are both set to 1.0. For convolutional neural networks, we strictly follow the settings from Zhao et al. (2022a); Chen et al. 2021b). For the same architecture family, the teacher model is ResNet-34, and the student model is ResNet- 18. For different architecture families, the teacher model ResNet-50, and the student model MobileNet- V1. Additionally, we explore the vision transformer architecture RVT Mao et al. (2022), employing RVT-S as the teacher model and RVT-Ti as the student model. Notably, in our RVT experiments, we did not apply branch reparameterization. We found that channel reparameterizzation alone performed better than when branch reparameterization was included.
E.2 TRAIN FROM SCRATCH
To further evaluate the efficacy of our approach, we conducted experiments without the reliance on a pre-trained static teacher model. Instead, we construct the dynamic teacher model via Inverse Reparameterization of the student model, described in Sec. 3.2. Both the dynamic teacher and the student model are trained simultaneously from random initialization. During training, we leverage the additional loss terms introduced by our GPD, given by Eq. 2 to facilitate knowledge transfer from the dynamic teacher to the student model. We adopt the same data preprocessing and opti- mization strategies described in Sec. 4.1
E.3 MODEL PINE-TUNIN
In this part, we begin with pre-trained student models and aim further improve their performance through our proposed approach. The pre-trained models are fine-tuned for 50 epochs, with the initial learning rate set 0. the initial learning rate used during the pretraining stage. For the standard fine-tuning baseline, fine-tune the pre-trained models with cross entropy loss. Regarding our proposed GPD* method, we leverage the dynamic teacher model for distillation based fine- tuning. Benefiting from Inverse Reparameterization, the dynamic teacher model initially exhibits the same accuracy as the student model. During the fine-tuning process, the dynamic teacher maintains slightly higher accuracy than the student due to its increased capacity. We used the same loss function as in the experiment of training from scratch.
20

---

Published as a conference paper at ICLR 2025
F ADDITIONAL COMPARISONS WITH RELATED WORK
Comparison with Teacher Assistant Methods. compare with the state-of-the-art teacher- assistant-based KD methods. As shown in Table 10 GPD consistently achieves superior perfor- mance compared to these methods on ResNet18, demonstrating its effectiveness.
Comparison with Self-Distillation Methods. To provide more comparisons when training without static teachers, we evaluate GPD* against state-of-the- self-distillation methods. As shown in Ta- ble TT, GPD* consistently achieves superior performance compared to these methods ResNet18, further demonstrating its effectiveness.
Table 10: Comparison with KD methods us- ing teacher assistants on ImageNet.
Table Comparison with Self-
Distillation Methods on ImageNet.
Method
ResNet18
Method
ResNet18
Student
69.75
SSKD
Xu et al. 2020
71.62
TAKI (Mizadee al.
2020
71.37
USKD
Yang ct a.. 2025
70.79
DGKD (Son al. 2021
71.73
FRSKD
(J) el a 2021
70.17
RKD Gao al. 2021
71.46
BYOT Zhang el al. 2019
68.93
TDS
Lt
2024a
72.29
ONE Lan al.
70.55
ESKD+AT (Cho & Hariharan
2019
71.39
SD
hang
70.63
TILLM
(7hu et al. 2022
72.6
RSD Zheng 2024
70.70
DKD
GPD
72.71
GPD
71.87
G EFFECT OF TEACHER-STUDENS CAPACITY GAP ON MODEL PERFORMANCE
As visualized in Figure increasing teacher capacity within appropriate range can bring per- formance gains. However, DKD shows a clear declining trend as the teacher-student gap grows larger, suffering significant accuracy degradation with extremely large teachers like ViT-L. In con- trast, GPD shows only minor performance decline and effectively alleviates the large-gap challenge. consistently outperforming DKD by more than 1 % across all teacher models, even with the largest teacher ViT-L.
72.90 (1.10)
72.81
1.20
72 78
72.7 (110
72.71 (.228)
fg 70
71.74
71.81
DKD+GPD
DKD
71.49
71.43
ResNer34
ResNec101
ReeNer152
CorvNeXt Base
VTT-L
Figure 7: Performance comparison with increasingly larger teacher models. DKD peaks with ResNet101 but suffers significant degradation with excessively large teachers, while GPD effectively mitigates performance degradation and maintains consistent improvements (+>1 1%).
H PERFORMANCE GAP EVOLUTION DURING DISTILLATION
The dynamic teacher often has higher accuracy during most of the training process except the very early stage. As shown in Figure g (distill ResNet18 from ResNet34), the dynamic teacher has lower/similar accuracy than the student in the first 3% epochs but outperforms in the following training stages. Since GPD maintains reasonable performance gap in most of the training process, we are able to enhance the distillation performance by preserving a reasonable performance gap. More critically, observe that this phenomenon exists in all the considered experiments.
21

---

Published as a conference paper at ICLR 2025
80
Accuracy on ImageNet (%)
60
40
20
Student (ResNe:18)
Dynamic Teacher [M
r=2]
Accuracy Superiority Dividing Line
20
60
80
100
epoch
Figure 8: Convergence of student and dynamic teacher during R distillation with GPD. The dynamic teacher shows lower/similar accuracy in the first 3% of epochs but outperforms the student in the following training stages.
I ANALYSIS OF COMPUTATIONAL OVERHEAD IN GPD
We highlight that the re- e--arameterizationn operation is very efficient and the time cost is negligible. To be specific, for a convolutional layer with 128 input channels and 256 output channels, it only takes 0.021 milliseconds, which is negligible compared to the overall computation cost. Our GPD slightly increases the training cost but comes with significant performance improvement. For exam- ple, from Table 12 GPD only introduces 33% extra overhead while yielding a large performance improvement of 1.58%. In fact, the additional training cost primarily arises from training the dy- namic teacher model. Interestingly, the extra training overhead is marginal due to two reasons: 1) Since KD often relies on a large static teacher, conducting forward propagation through this model at each iteration already greatly increases the training cost; 2) Training both the student and dynamic teacher can increase the utilization ratio of GPU and benefits from its high parallelism in computa- tion. Interestingly, we observe that merely training a small student model tends to come with a very low utilization ratio, c.g., often lower than 50%. Because our GPD consistently obtains significant performance improvement across diverse architectures and settings (see Table I we believe that it a good trade-off between performance and training cost. From this point of view, our GPD could be a very strong baseline for distillation.
Table 12: Comparison of computation cost on the experiment of ResNet50 + MibileNeedistiiltioonn. distillation. We measure the training time 4 A100 GPUs with batch size of 512 on ImageNet.
Method
Acc. (%)
Training Time per Epoch (min)
DKD
72.05
12
DKD + GPD
73.63 (+1.58)
16 (+33%)
J ADVANTAGES OVER EXISTING METHODS
Our GPD has specific advantages and essentially different from existing distillation meth- ods (Wang et al. 2022 Mirzadeh et al. 2020; Cho & Hariharan] 2019 Yang et al.] 2019a] Son cct al. 2021 Zhao et al. 2022b) in several aspects. First, they maintain different performance gaps. Existing methods (Dong et al. 2024 Mirzadch et al.) 2020: Son et al. 2021 utilize fixed-accuracyy pre-trained teachers assistant models, which may hinder knowledge transfer as the gap still be too large, particularly in early training stages when student accuracy is low. GPD initializes the dy- namic teacher with the same accuracy as the student and trains both simultaneously, maintaining a small, dynamic gap throughout the process. Second, they construct the teacher assistants in differ- ent ways. Unlike methods (Mirzadeh et al. 2020 [onn et al.) 202T) that construct and train multiple intermediate-sio teacher assistants separately, which can computationally expensive. GPD builds a single dynamic teacher. We propose a novel IR technique for model expansion that maintains the same initial accuracy as the student. This approach is both computationally efficient and ensures a controlled start point for the distillation process. Third, they transfer knowledge in different ways. Existing methods (Mizaddeh et al.) 2020 Son et al., 2021) follow the standard distillation paradigm by incorporating KD loss for transferring knowledge. Besides this, our GPD enforces parameter sharing between the student and dynamic teacher, allowing direct inheritance of parameters. This process is facilitated by CBR, enabling more direct and effective knowledge transfer.
22


Reviews:
[{"review_id": "DgL6xq4cmG", "reviewer": "Reviewer_49EM", "summary": "This paper introduces Gap Preserving Distillation (GPD) to address the challenge where large performance gaps between teacher and student models hinder effective knowledge transfer. GPD introduces a dynamic teacher model that trains alongside the student, maintaining an optimal performance gap to improve transfer efficiency. The performance gains from GPD are primarily due to its use of Inverse Reparameterization (IR) and Channel-Branch Reparameterization (CBR), which create flexible connections that enable parameter sharing between the dynamic teacher and student. Experimental results show that models trained with GPD outperform those using traditional fixed teacher models and perform robustly even in scenarios without a pre-trained teacher, such as training from scratch and fine-tuning.", "questions": "1. How can we better understand and measure the impact of the performance gap on knowledge transfer? The paper mentions that keeping a reasonable performance gap between teacher and student models helps with effective knowledge transfer. However, it doesn’t fully explain why a larger gap might hurt the process. How exactly does the size of this gap affect the student model’s final performance? As mentioned in the weaknesses section, adding some theory or controlled experiments here could help clarify this point and give a clearer idea of the best range for this gap.\n\n2. As mentioned in the weaknesses, is the complex design of the dynamic teacher model really necessary? To tackle the performance gap, the paper introduces a dynamic teacher model with intricate structures like Inverse Parameterization (IR) and parameter sharing. But has the author considered more straightforward options to close the gap? If this complex setup is essential, could they explain more about how it directly addresses the problem?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper identifies the performance gap between teacher and student models as a key challenge in knowledge distillation. By introducing a dynamic adjustment mechanism, the method effectively narrows this gap, facilitating more efficient knowledge transfer.\n\n2. The proposed GPD demonstrates adaptability across various training settings, with significant improvements across multiple models and datasets. Extensive ablation studies further validate the method’s effectiveness.", "weaknesses": "1. Needs More Theoretical Explanation\nThis paper could benefit from a deeper explanation of why a large performance gap between the teacher and student models makes knowledge transfer less effective. Right now, the idea that a big gap harms learning is mentioned, but it would be stronger if some theoretical reasoning or tests showed how this happens. Adding experiments that test different gap sizes could also help show how different levels of teacher-student gaps impact the student model’s final performance.\n\n2. Overengineering in the Dynamic Teacher Model \nThe necessity of the complex structure designed in this paper, including Inverse Reparameterization (IR) and parameter sharing, appears somewhat disconnected from the core problem. If the primary goal is to reduce the performance gap, it remains unclear whether the proposed architecture (specifically the IR and shared parameters) is required to achieve this. Making the connection between these design choices and the main goal a bit clearer could help the paper feel more cohesive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dREB7BMwir", "reviewer": "Reviewer_J6Y5", "summary": "The paper proposes Gap Preserving Distillation (GPD), a framework designed to address the capacity gap between teacher and student networks. GPD achieves this by training an additional dynamic teacher alongside the student. To further improve performance and efficiency, GPD enforces parameter sharing between the dynamic teacher and the student using Inverse Reparameterization and Channel-Branch Reparameterization techniques. Experiments on ImageNet-1K classification primarily compare GPD with baselines using static teachers, showing preliminary effectiveness.", "questions": "1. The paper claims that the dynamic teacher shares exactly the same accuracy as the student with the reparameterization techniques. However, Figure 1 shows that the weights of the dynamic teacher are modified with added noise, which seems to contradict this claim.\n2. The claim in lines 68-69, “Since DT is a larger model and often converges faster than the student,” appears counterintuitive. Is there any evidence to support this?\n3. The network pairs in Table 6 do not convincingly demonstrate the effect of the teacher-student capacity gap. For instance, Table 6 shows that ViT-L performs worse than ResNet101 when distilling to a ResNet18. This may be due not to the larger capacity gap between ViT-L and ResNet18, but rather to differences in architecture. It would be more consistent to use CNN or ViT architectures for both teacher and student.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem is well-motivated, particularly in lines 54-61, highlighting the importance of adaptively managing the performance gap.\n2. The proposed approach is novel. It co-trains a dynamic teacher to alleviate the teacher-student capacity gap and applies inverse reparameterization and channel-branch reparameterization for efficient parameter sharing.\n3. The paper is well-organized and easy to follow, with clear descriptions, intuitive figures, and rigorously detailed algorithms.", "weaknesses": "1. **Lack of comparisons with Knowledge Distillation (KD) methods using teacher assistants**. These methods are closely related to the proposed GPD. Without these comparisons, the effectiveness of GPD is unclear or less convincing. Suggested comparisons include, but are not limited to:\n   - Mirzadeh, Seyed Iman, et al. \"Improved knowledge distillation via teacher assistant.\" *AAAI 2020*\n   - Son, Wonchul, et al. \"Densely guided knowledge distillation using multiple teacher assistants.\" *CVPR 2021*\n   - Ganta, Durga Prasad, Himel Das Gupta, and Victor S. Sheng. \"Knowledge distillation via weighted ensemble of teaching assistants.\" *ICBK 2021*\n   - Zhou, Yuhang, and Wei Ai. \"Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios.\" *arXiv:2406.05322* (2024)\n\n2. **Inappropriate and unfair baselines in Section 4.2**. The paper compares GPD without a static teacher against networks trained without any KD techniques. Since GPD involves an additional dynamic teacher, it would be more appropriate to compare it with self-distillation methods.\n\n3. **Practicality and efficiency concerns**. The size of the additional dynamic teacher must still be tuned by adjusting the branch and channel expansion ratios, making GPD less practical and efficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dfxWUd96OU", "reviewer": "Reviewer_FcyL", "summary": "This paper aims to improve the effectiveness of knowledge distillation (KD) when using a teacher model with a large performance gap relative to the student model. The authors propose a method called Gap Preserving Distillation (GPD). By adopting a dynamic teacher model created through a re-parameterization operation on the student model as a proxy teacher, the performance gap is narrowed. Experimental results demonstrate the effectiveness of this approach.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of effectively performing KD with an extremely large teacher model is important and well-motivated.\n- The paper is well-organized and easy to follow.", "weaknesses": "1. The abstract is too lengthy, as it seems to emphasize every aspect in detail. I suggest revising the abstract to improve clarity and conciseness.\n\n2. The KD process involves training a dynamic teacher model alongside the student, which requires frequent re-parameterization. I am curious about the training cost compared to baseline methods.\n\n3. In Table 6, it appears that the larger ViT-L teacher model leads to worse student performance than the smaller ResNet-101 teacher model (as with DKD). This suggests that the proposed method may still struggle with extremely large teacher models, despite it aims to overcome this issue.\n\n4. The method's description primarily focuses on convolutional models. Although experimental results are provided for transformer models, it would be helpful to explain how re-parameterization is achieved for these models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)