Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation
Year: 2023

Abstract:
It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledge transfer. In order to enhance the guidance of the teacher training process, we introduce the concept of distillation influence to determine the impact of distillation from each training sample on the student’s generalization ability. In this paper, we propose Learning Good Teacher Matters (LGTM), an efficient training technique for incorporating distillation influence into the teacher’s learning process. By prioritizing samples that are likely to enhance the student’s generalization ability, our LGTM outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark. 1

heading: 1 Introduction
text: The recent success of natural language processing (NLP) is driven by the adoption of large-scale pretrained language models (Devlin et al., 2019; Liu et al., 2019; Dai et al., 2019; Yang et al., 2019). As these models are scaling up in depth and width, they become increasingly computational and storage intensive, making deployment difficult. To address this issue, different methods have been proposed for crafting efficient models with minimal loss in performance, such as weight pruning (Fan et al., 2019; Li et al., 2021a), network quantization (Kim et al., 2021; Zhang et al., 2020), and knowledge distillation (KD) (Sun et al., 2019; Tang et al., 2019; Sun et al., 2020). Among these methods, KD has proven to be effective in various NLP applications (Jiao et al., 2020) and is widely adopted. The idea of KD involves asking a lightweight stu-
∗Equal contribution. † Work done while being at Amazon. 1Our code is public available at https://github.com/
twinkle0331/LGTM
dent model to mimic the output of a large teacher model so as to transfer the knowledge.
Ideally, a teacher with better performance should be able to transfer more knowledge to the student. Therefore in most knowledge distillation algorithms, the teacher network is trained to maximize its own performance. However, multiple studies (Wang et al., 2022a; Cho and Hariharan, 2019) have observed that a teacher with higher performance does not necessarily lead to a betterperforming student, and may even cause a performance degradation. Stanton et al. (2021) has attributed this inefficiency in knowledge distillation to challenges during optimization. As the model capacity gap between the student and the teacher increases, the optimization process becomes more likely to be trapped in local optima (Cho and Hariharan, 2019; Mirzadeh et al., 2020).
One way to address the performance degradation in KD is to update the teacher via feedback from student’s performance, also known as learning to teach (L2T) (Fan et al., 2018; Zhou et al., 2022). L2T allows the teacher model to adjust its “teaching agenda” by interacting with the student. Among the L2T algorithms, online distillation (Zhang et al., 2018; Zhu et al., 2018; Shi et al., 2020) trains the student and teacher concurrently and enforces similarity between their outputs on the training set. However, online distillation focuses on transferring the knowledge of the teacher to the student on training set without explicitly considering how well the student will perform on validation set. On the other hand, meta distillation (Zhou et al., 2022; Pham et al., 2021) takes the generalization ability of student on the held-out validation set into account, and guides the teacher’s learning process to maximize the generalization ability. However, the optimization objective of meta distillation may result in a degraded teacher model, as it only receives supervision from the student model.
It is well-known that humans are more efficient
1990
learners when their teachers provide guidance on the level of attention they should devote to certain problems based on their current knowledge. Similarly, it is possible that a student model could be trained more effectively if it receives such guidance from a teacher. To accomplish this goal, the teacher should prioritize samples that are likely to enhance the student’s generalization ability during training, thus allowing the student to perform better on the held-out validation set.
In this work, inspired by the concept of influence function (Pruthi et al., 2020; Koh and Liang, 2017), we propose distillation influence to estimate how distilling on each training sample impacts the student’s performance on the validation set. In addition, we are able to interpret existing L2T methods from the perspective of influence function, so as to gain a deeper understanding of their limitations. The optimization process of existing L2T methods are often impacted by outliers, because they assign all training samples in the mini-batch the same weight. Hence, we propose our L2T framework, Learning Good Teacher Matters (LGTM), which assigns loss weights of the training samples based on their distillation influence.
Extensive experiments have shown that LGTM enables more effective knowledge transfer.
In summary, our contributions are as follows:
1. We propose distillation influence to quantify how distilling from each training sample impacts the student’s generalization ability.
2. We introduce finite difference approximation to efficiently incorporate distillation influence into the teacher’s learning process.
3. Comparing to 10 common KD baselines, our proposed LGTM demonstrates consistently better performance on 6 text classification tasks in GLUE benchmark.

heading: 2 Notations
text: Suppose we have a teacher model denoted as T (·; θt) and a student model denoted as S(·; θs). The corresponding model parameters are θt and θs. ηt and ηs are the learning rates adopted for model update. We use |t| and |s| to denote the dimensions of θt and θs, i.e., θt ∈ R|t|×1 and θs ∈ R|s|×1. The time step before and after model parameter updates are denoted as m and m + 1, respectively. It is used to track the evolution of the model parameters during the training process.
Given a labeled training dataset Dtrain, a batch of Br training samples and their corresponding labels are referred to as zr = (xr,yr), where r indicates training. We index each sample in the training batch zr as zri . Similarly for validation dataset Dval, we define the batch of samples as ze = (xe,ye), where e indicates validation.
In addition, we introduce the notation of the Jacobian matrix in the context of working with the chain rule and gradient. In particular, let f : Rk → Rn be a differentiable function, and let v ∈ Rk be a vector. We use the notation ∂f∂v ∈ Rk×n to represent the Jacobian matrix of f , which has dimensions k × n. For simplicity, we annotate ∂f∂v as ∇v. We use X⊺ to denote the transpose of the matrix X .

heading: 3 Revisiting Learning to Teach
text: In this paper, we focus on task-specific distillation given pre-trained language models. Under this setting, the teacher model is already pre-trained in an unsupervised manner and the student model is either derived from part of the teacher model or pre-trained in an unsupervised manner as well.
Vanilla distillation The typical approach to knowledge distillation is a two-stage process. It involves first fine-tuning a pre-trained teacher model
to maximize its performance on a specific task. Once the teacher model has converged, a student model is trained to closely imitate the output of the teacher model on the training data. The optimization objective for the student model at each mini-batch is:
Ls(θs, θt, zr) = αLce(yr, S(xr; θs)) +(1− α)Lce(T (xr; θt), S(xr; θs)).
(1)
The update of the student follows:
θm+1s = θ m s − ηs∇θsLs(θms , θmt , zr). (2)
The limitation of vanilla distillation is that it does not allow teacher to adjust its behavior according to student’s feedback, as the teacher’s parameters are fixed during the distillation process.
Online distillation To achieve student-aware distillation, online distillation (Zhang et al., 2018; Zhu et al., 2018; Shi et al., 2020) is proposed which involves the simultaneous fine-tuning of both the student and teacher models in one-stage.
In addition to minimizing the cross-entropy loss with respect to the ground truth labels, the target distribution of the teacher model is constrained to be close to that of the student model through the minimization of the cross-entropy loss between the outputs of the teacher and student models:
Lt(θt, θs, zr) = αLce(yr, T (xr; θt)) +(1− α)Lce(T (xr; θt), S(xr; θs)).
(3)
The training process involves iteratively updating the parameters of both models:
θm+1t = θ m t − ηt∇θtLt(θmt , θms , zr) θm+1s = θ m s − ηs∇θsLs(θms , θm+1t , zr).
(4)
Through iterative update, the student model is able to learn from the learning curve of the teacher model (Shi et al., 2020), which improves its performance on the given task.
However, online distillation focuses on transferring the knowledge of the teacher to the student on training set without explicitly considering how well the student model will perform on unseen test data. This might lead to the student model only memorizing the training examples without generalizing well to new ones (Zhou et al., 2022).
Meta distillation Meta distillation (Zhou et al., 2022; Pham et al., 2021) is a technique that takes into account the feedback from the student model and guides the optimization of the teacher model to maximize the generalization ability of the student. The generalization error of the student model is measured by the cross-entropy loss computed between the ground truth labels and the predictions of the student model on the validation set:
Lval(θs, ze) = Lce(ye, S(xe, θs)). (5)
Meta distillation decomposes models’ learning process into two stages. The first stage is to finetune a good teacher on task-specific data similar to vanilla distillation, while the second stage involves iterative update of the teacher and student models. Note that compared to online distillation, meta distillation obtains the student feedback from validation data, not training data.
During the second stage, the student model is first updated through the standard distillation process by minimizing the distillation loss in eq. (1). Then the teacher model is optimized to minimize the updated student’s loss on the held-out validation set, which ensures it is able to guide the student towards better generalization. During this process, the teacher is only trained for the purpose of knowledge transfer. Formally, the student model is updated as follows:
θm+1s = θ m s − ηs∇θsLs(θms , θmt , zr). (6)
The teacher model is then updated as follows:
θm+1t = θ m t − ηt∇θtLval(θm+1s , ze), (7)
However, the optimization objective of meta distillation can result in a degraded teacher model because it only receives supervision from the student. This will prevent the teacher model from continuing to learn and improve in the second stage, thus impeding its ability to adapt to new data.

heading: 4 Methods
text: To overcome the aforementioned limitations, we introduce our L2T framework, Learning Good Teacher Matters (LGTM) to enable more effective knowledge distillation. We first introduce distillation influence, which estimates how much will the student’s performance on validation data change
if we put one training sample in the knowledge distillation process.
Afterwards, we introduce an efficient training method based on finite difference approximation for incorporating distillation influence into the teacher’s update. Finally, we interpret current L2T methods from the perspective of influence function.
Distillation influence Influence function (Pruthi et al., 2020; Koh and Liang, 2017) is a way of measuring the influence of training samples on the model’s predictions. It can be utilized to identify instances that have a disproportionate effect on the model’s behavior, whether due to their status as outliers or due to incorrect labeling (Jia et al., 2019; Ghorbani and Zou, 2019; Hara et al., 2019). By calculating the influence function for a particular example, it is possible to estimate the extent to which the model’s prediction would be altered as a result of operations on that sample.
In vanilla distillation, for the student model, we derive the distillation influence of zri as the gradient similarity between the training sample zri and the validation batch ze:
Idistill(zri ,ze) =∇θsLce(T (xri ; θmt ), S(xri ; θms ))⊺
∇θsLce(ye, S(xe; θm+1s )) (8)
The detailed derivation can be found in appendix A. The influence reflects how well the knowledge gained from a particular sample generalizes. It follows that the teacher should focus on teaching the student to capture training samples that have the highest distillation influences.
In order to incorporate the per-sample influence into knowledge distillation, we adjust the loss weight of each sample based on its distillation influence. This allows us to determine the relative importance of each sample, and helps to control how much each sample contributes to the teacher’s learning process. Samples that are deemed to be more beneficial for the student’s generalization are assigned higher weights. Then we propose training the teacher using the following objective:
Linfluence = 1 Br
Br∑
i=1
wiLce((T (xri ; θmt ), S(xri ; θms )), (9)
where wi = Idistill(zri , ze). By including the influence in the knowledge distillation loss function, we can tailor the training process to better suit the characteristics of the target task.
Algorithm 1 LGTM Require: student θs, teacher θt, training set Dtrain, validation
set Dval Require: ηs, ηt: learning rate for the student and the teacher Require: ϵ: a small scalar Require: M : the maximum number of the training steps 1: while step < M do 2: Sample a batch of training set zr = (xr,yr) ∼ Dtrain 3: Copy student parameter θs to student θ′s 4: Update θ′s: θ′s ← θs − ηs∇θ′sLs(θ ′ s, θt,z
r) 5: Sample a batch of validation set ze = (xe,ye) ∼ Dval 6: Calculate θ±s : θ±s = θs ± ϵLce(ye, S(xe; θ′s)) 7: Calculate the Distillation Influence with zr, θt, θ±s and ϵ: Linfluence ▷ eq. (10) 8: Update θt: θt ← θt− ηt∇θtLt(θt, θs,zr) ▷ eq. (11) 9: Update original θs: θs ← θs − ηs∇θsLs(θs, θt,zr)
10: step← step+ 1 11: end while
Finite difference approximation For standard neural network training, we often compute a consolidated gradient for a mini-batch of Br training samples to enhance computational efficiency. However, in the context of determining the distillation influence for each sample, the computation of per-sample gradient Lce(T (xri ; θmt ), S(xri ; θms )) will slow down the training by a factor of Br. In addition, a naive implementation is memory intensive, because it requires to keep a copy of ∇θsLce(ye, S(xe; θm+1s )).
To address this, we propose an efficient method for updating the teacher with the distillation influence by utilizing finite difference (Gleich, 2005), a technique commonly used in numerical analysis for approximating the derivative of a function at a given point. Similar to (Pham et al., 2021; Liu et al., 2018), we approximate Linfluence by
Linfluence ≈ L̂influence = 1 Br
Br∑
i=1
[Lce(T (xi; θmt ), S(xi; θ+s )) 2ϵ
− Lce(T (xi; θ m t ), S(xi; θ − s ))
2ϵ
] ,
(10)
where θ±s = θs ± ϵLce(ye, S(xe; θm+1s )) and ϵ is a small scalar. Our proposed method for evaluating the finite difference is computationally efficient, as it only requires two forward passes for θs and one backward pass for θt for a single batch, as opposed to a naive implementation which requires Br forward and backward passes for θs and one backward pass for θt. We provide more details of the derivation in appendix B.
Teacher’s auxiliary loss Inspired by (Pham et al., 2021), in order to balance the trade-off between self-evolution and transferability of the teacher
model, we incorporate the loss with respect to the ground truth as Laux into the final objective:
Lt(θt | θs,zr) = L̂influence + Laux, Laux = αLce(yr, T (xr; θt))+
(1− α)Lce(T (xr; θt), S(xr; θs)) (11)
where α is the loss ratio.
Overall, our method allows the teacher to adapt to the student’s abilities and provide more personalized guidance while improving the student’s generalization capability. We present the algorithm of LGTM in algorithm 1.
Relationship with other L2T methods Here we interpret current learning to teach methods from the perspective of influence function.
In the case of online distillation, it is assumed that all training samples possess an equivalent distillation influence and that the teacher model is responsible for reducing the transfer difficulty of all training samples.
In contrast, the key differentiating factor between meta distillation and online distillation is the utilization of a dynamic loss weight. We interpret this weight as a measure of the distillation influence of the current training batch zr on the generalization ability of the student model. Specifically, it reflects the similarity between the gradients of the training and validation batches, indicating the effect of the current training batch zr on the validation batch ze (as detailed in appendix C). However, it should be noted that this weight functions primarily as an adaptive learning rate, adjusting the gradient step proportionally to the degree of similarity in gradients. We illustrate the general workflow of vanilla distillation, online distillation, meta distillation and LGTM in fig. 1.

heading: 5 Experiments
text: In this section, we first describe our experiment setup including datasets and baselines in Sec. 5.1. Then we compare our proposed LGTM to meta distillation to gain some basic understanding of how to incorporate the student’s feedback in Sec. 5.2. To further verify the effectiveness of our method, in Sec. 5.3 we compare to 10 widely adopted knowledge distillation baselines and show consistently better results. Then we demonstrate how distillation influence works in Sec. 5.4, followed by ablation studies of LGTM in Sec. 5.5.

heading: 5.1 Experimental Setup
text: Datasets We evaluate our proposed approach on text classification tasks in GLUE (Wang et al., 2018): MRPC (Dolan and Brockett, 2005), RTE (Wang et al., 2018), SST-2 (Socher et al., 2013), MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016) and QQP (Chen et al., 2018). For MRPC and QQP, we report both F1 and accuracy. And for other datasets, we report accuracy.
Baselines We compare our LGTM with 10 baselines: 1) KD (Hinton et al., 2015) 2) PKD (Sun et al., 2019) 3) SKD (Guo et al., 2022) 4) DIST (Huang et al., 2022) 5) TAKD (Mirzadeh et al., 2020) 6) RCO (Jin et al., 2019) 7) DML (Zhang et al., 2018) 8) ProKT (Shi et al., 2020) 9) PESF-KD (Rao et al., 2022) and 10) Meta Distill (Zhou et al., 2022).
Training setup Following previous works (Sun et al., 2019; Zhou et al., 2022), we distill BERTBase (Devlin et al., 2019) to a 6-layer BERT model. For all two-stage baselines, we fine-tune the models on each task. For fair comparison, both Meta Distill and LGTM utilize feedback from the validation set in the calculation of the distillation loss.
Detailed training hyperparameters can be found in appendix D.

heading: 5.2 Comparison with Meta Distillation
text: Given our proposed LGTM is closely related to the meta distillation line of work, here we first conduct a comparison between LGTM and a specific meta distillation method, Meta Distill (Zhou et al., 2022), to demonstrate the benefit of adopting distillation influence.
We observe that for Meta Distill (blue curve) in fig. 2 (a) and (b), the validation loss of the student model gradually increases in later iterations while the validation accuracy keeps improving until a stable plateau. This clearly indicates that the student model is experiencing overfitting. One possible explanation is that excessive emphasis is placed on certain training samples that generate high loss, e.g., hard samples or outliers. This negatively impacts the generalization ability of the student model, which leads to overfitting.
The key difference between Meta Distill and our LGTM (orange curve) is that LGTM accounts for the per-sample distillation influence while Meta Distill treats all training samples in a batch equally. This enables the filtering of samples that have a detrimental effect on generalization performance of the student model, leading to a steady decrease of validation loss (fig. 2 (a)) and an improved validation accuracy (fig. 2 (b)).
In terms of teacher model, it should not only impart their current knowledge to the student, but also actively seek out new information and perspectives to improve their own understanding. As can be
seen in fig. 2 (c), LGTM allows for the effective transfer of knowledge from the teacher model by incorporating the teacher auxiliary loss. The validation accuracy of the teacher model keeps improving for LGTM, but drops quickly for Meta Distill.

heading: 5.3 Main Results
text: Here we show the results of our proposed method on the test set of text classification tasks in GLUE benchmark. As can be seen in table 1, LGTM outperforms all 10 baselines including recent strong KD methods (Guo et al., 2022; Huang et al., 2022; Rao et al., 2022; Zhou et al., 2022), which highlights the effectiveness of our method.
To be more specific, our proposed method achieves state-of-the-art performance in comparison to those rely on carefully designed training pipelines or loss functions, e.g., PKD (Sun et al., 2019), SKD (Guo et al., 2022) and DIST (Huang et al., 2022). PKD proposes two distillation schemes, to enable the student to learn from multiple intermediate layers of the teacher model for incremental knowledge extraction. SKD and DIST both modify the form of KL-divergence loss to narrow the gap between the teacher and student models. LGTM also does not require a series of teacher assistant models as TAKD (Mirzadeh et al., 2020) and RCO (Jin et al., 2019).
Compared to online distillation methods, LGTM performs better than DML (Zhang et al., 2018), ProKT (Shi et al., 2020) and PESF-KD (Rao et al., 2022). This highlights the importance of incorporating student’s feedback during the training process. An overemphasis on knowledge transfer from
the training set may lead to the student overfitting the teacher’s outputs, resulting in a reduction in its generalization abilities.
Furthermore, unlike meta distillation methods, e.g., Meta Distill (Zhou et al., 2022), our method allows for computing distillation influence of individual training samples, which enables filtering out samples that may hurt student’s generalization. Therefore, LGTM is able to help the student to develop general understanding of the overall task while alleviate the overfitting issue.

heading: 5.4 Analysis of Distillation Influence
text: We further explore the trend of the distillation influence of samples during the real training process. Here, we conduct experiments on the MRPC dataset. The task is to predict whether the sentences in a sentence pair are semantically equivalent (Wang et al., 2018).
First, we select two representative samples presented in fig. 3 to visualize the trend of the distillation influence and its relationship between the teacher’s and the student’s prediction.
On the left-side of fig. 3, we can see that during the initial stages of training, both the teacher (green) and the student (orange) have made wrong predictions. It might suggest that this sample poses a significant challenge for both models to learn. In this case, we do not want student model to mimic the output from teacher models too much because
teacher model is also wrong about this sample. Our method is able to gradually adjust the loss weight to negative, indicating we will filter out this misleading training sample for now to make both models learn faster. As a result, the student model first escapes this predicament. Then through student feedback on the validation set, the teacher model also learns to make the correct prediction. Finally as training progresses, it is observed that both the student and the teacher are able to correctly classify this sample, resulting in the distillation influence stabilizing at a near-zero value.
We present another example in fig. 3 right, where both the student and the teacher are able to accurately predict a given sample. It might suggest this sample is too easy for the teacher and the student. In this case, we want to give this sample a high positive weight to form a student-friendly decision boundary. This is similar to design a curriculum to learn from easy samples to hard ones in curriculum learning (Soviany et al., 2022).
We also visualize an average trend of distillation influence in fig. 4, based on 64 samples that are randomly chosen from MRPC. We observe that the distillation influence is usually insignificant in the beginning and end of the training, but fluctuates in the middle. This is reasonable since our method is assigning varying weights to each sample during training, with the goal of filtering difficult samples and focusing on samples better for generalization.

heading: 5.5 Ablation Study
text: Given limited space, we present three studies in this section and show more ablation studies in appendix E.
Finite difference approximation Recall in section 4, we introduce finite difference approximation (FDA) for estimating the distillation influence of each sample. It is designed to address the slowness of computing per-sample gradients. As shown in table 3, here we conduct an ablation experiment on the MRPC dataset to evaluate its usefulness. We show that with FDA, our method only requires 11 minutes to complete the training, while the naive training without FDA requires 117 minutes. Such a significant reduction in training time (i.e., more than 10× speedup) highlights the computational efficiency of the proposed FDA technique. Furthermore, we assess the performance on the validation
set of the MRPC dataset and observe that training with FDA result in an F1 score of 90.4, while training without FDA resulted in a score of 90.7. There is only a slight drop in performance with the approximation.
Distillation loss There are other distillation losses in the context of knowledge distillation. Here we want to evaluate whether LGTM can adapt to those objectives. In particular, we consider the modified loss used in DIST (Huang et al., 2022) and the common mean squared error (MSE). As can be seen in table 2, our LGTM consistently beats the original methods that utilize these distillation objectives, which validates the compatibility of LGTM to different distillation objectives.
Student model size Here we conduct experiments to evaluate the performance of our proposed method in scenarios where there is a larger capacity difference between the teacher and student models. Specifically, we perform knowledge distillation from a BERT-Base model (Devlin et al., 2019) to a 4-layer BERT model. As can be seen from table 4, LGTM consistently outperforms other baselines in most of the tasks except competitive results on SST-2. This indicates the robustness of our method which suggests its wide usage in various knowledge distillation settings.

heading: 6 Related Work
text: The core of knowledge distillation (Hinton et al., 2015) relies on how to formulate and transfer the knowledge from the teacher to student. Three
key aspects are typically considered: the teacher model from which knowledge is transferred (learning target), the data on which the model is trained (learning material), and the objective function that defines the learning objective. Efforts have been made to make knowledge distillation more studentfriendly by reducing the difficulties in these aspects(Li et al., 2021b).
On learning target, Jin et al. (2019); Mirzadeh et al. (2020) introduce teacher assistant models of intermediate timestep or training time step respectively to narrow the gap between the teacher and student models. Park et al. (2021); Shi et al. (2020) propose updating the teacher and student jointly to make the teacher aware of the student’s state. Rao et al. (2022) trains for more timestep to smooth the distribution of the teacher for a easier transfer.
In terms of learning material, TinyBERT (Jiao et al., 2020) suggests augmenting the training data to make it more diverse. Kim et al. (2022) proposes training the student with samples that are easy for the teacher but difficult for the student. With respect to learning objective, the most common approach is to match the probabilistic prediction scores of the teacher and student models using KL-divergence. However, this can cause problems during training, leading to poor performance. Guo et al. (2022); Huang et al. (2022) propose to soft the constraint by a more tolerated loss. Pham et al. (2021); Zhou et al. (2022) propose using the student’s performance as the optimization objective for the teacher model, allowing the teacher to optimize its knowledge transfer based on feedback from the student. Wang et al. (2022b) proposes to select the appropriate knowledge to guide the optimization of the student.

heading: 7 Conclusion
text: In this paper, we first revisit several learning to teach paradigms in knowledge distillation. Then we propose distillation influence to determine how distilling from each training sample impacts the student’s generalization ability. By visualizing how the distillation influence of each sample changes during training, we can see that a simple re-weighting using distillation influence is able to help student training, e.g., reduce overfitting. Built on top of distillation influence, we propose our learning to teach framework, LGTM, that consistently outperforms existing knowledge distillation methods on text classification tasks in the GLUE benchmark.
Limitations
Although LGTM has demonstrated superior performance in task-specific knowledge distillation, it is worth investigating the potential benefits of combining LGTM with pre-training knowledge distillation (Jiao et al., 2020; Wang et al., 2020). Additionally, while our experiments have been limited to text classification tasks, which are relatively simple for current pre-trained language models, future work should explore the application of LGTM to more complex text generation tasks.
Ethics Statement
During the training process, the teacher and student models are initialized from pre-trained models. However, pre-trained language models are vulnerable to potential ethical and social risk as mentioned by Bommasani et al. (2021) and Weidinger et al. (2021). Therefore, the teacher and student models can be exposed to similar social risks of large language models.

heading: Acknowledgements
text: We thank Yongfei Liu and Zhengkun Zhang for their insightful discussion and the anonymous reviewers for their helpful comments. This work was supported by the National Key R&D Program of China (2022YFB4701400/4701402), SZSTC Grant (JCYJ20190809172201639, WDZC20200820200655001), Shenzhen Key Laboratory (ZDSYS20210623092001004), and Beijing Key Lab of Networked Multimedia.

heading: A The Derivation of Distillation Influence
text: As described by Pruthi et al. (2020), the influence of a training sample z = (x, y) on a test sample z′ = (x′, y′) can be traced by examining the change in loss of model w on the test sample. The influence function is defined as the total reduction in loss on the test sample z′ induced by the training process whenever the training sample z is utilized:
I(z, z′) = ∑
t:zt=z
L(wt, z′)− L(wt+1, z′). (12)
where wt+1 = wt−ηwL(wt, z) and ηw is the learning rate and the model are parameterized by wt and wt+1.
In this context, we will focus on the influence of the current training batch on the student model’s performance on the validation data. To improve computation efficiency, a batch of samples is drawn from the validation set to evaluate the model’s generalization performance. As a result, the influence on a single validation sample, as described in eq. (12), is extended to a batch of validation samples ze. The influence of the current training batch zr on the validation batch ze is defined as follows:
I(zr,ze) = Lval(θms ,ze)− Lval(θm+1s ,ze) = Lce(ye, S(xe; θms ))− Lce(ye, S(xe; θm+1s )), (13)
where θm+1s = θ m s − ηsLs(θms , θmt , zr).
By applying the Taylor expansion, we can approximate Lval(θms , ze) as follows:
Lval(θms ,ze) = Lval(θm+1s ,ze) + (θms − θm+1s )⊺ ∇θsLval(θm+1s ,ze) +O(||θms − θm+1s ||2) ≈ Lval(θm+1s ,ze) + (ηs∇θsLs(θms , θmt ,zr))⊺ ∇θsLval(θm+1s ,ze) (14)
As a result, we approximate the I(zr, ze) as follows:
Lval(θms ,ze)− Lval(θm+1s ,ze) ≈ (ηs∇θsLs(θms , θmt ,zr))⊺∇θsLval(θm+1s ,ze) (15)
The contribution of a single sample zri = (x r i , y r i ) in the training batch zr is defined as follows:
I(zri ,ze) ≈ (ηs∇θsLs(θms , θmt ,zri ))⊺∇θsLval(ze, θm+1s ) (16)
By excluding loss irrelevant to the teacher in eq. (16), we define the distillation influence of zri to be:
Idistill(zri ,ze) =∇θsLce(T (xri ; θmt ), S(xri ; θms ))⊺
∇θsLce(ye, S(xe; θm+1s )) (17)

heading: B Approximation Methods
text: Here, we efficiently approximate this gradient similarity using a Taylor expansion:
∇θt 1
Br
Br∑
i=1
wiLce(T (z r i , θt), S(z r i , θs))
= 1
Br
Br∑
i=1
∇θtLce(T (xri ; θmt ), S(xri ; θms ))
∇θsLce(ye, S(xe; θm+1s ))⊺ ∇θsLce(T (xri ; θmt ), S(xri ; θms ))
≈ 1 Br
Br∑
i=1
∇2θs,θtLce(T (xri ; θmt ), S(xri ; θms ))
∇θsLce(ye, S(xe; θm+1s ))
≈ ∇θt 1
Br
Br∑
i=1
[Lce(T (xri ; θmt ), S(xri ; θ+s )) 2ϵ −
Lce(T (xri ; θmt ), S(xri ; θ−s )) 2ϵ
]
(18)
where θ±s = θs ± ϵLce(ye, S(xe; θm+1s )) and ϵ is a small scalar.

heading: C A Closer Look at Meta Distillation
text: In meta distillation, the loss on the validation set with respect to the teacher can be derived as follows:
∇θtLce(ye, S(xe; θm+1s )) = ∇θtLce(ye, S(xe; θms − ηs∇θsLs(θms , θmt , zr))) = ∇θt(θms − ηs∇θsLs(θms , θmt , zr))∇θsLce(ye, S(xe; θm+1s )) = ∇θt(−ηs∇θsLs(θms , θmt , zr))∇θsLce(ye, S(xe; θm+1s )) = ∇θt(−ηs(1− α)∇θsLce(T (xr; θmt ), S(xr; θms )))
∇θsLce(ye, S(xe; θm+1s )) = −ηs(1− α)∇2θs,θtLce(T (xr; θmt ), S(xr; θms ))
∇θsLce(ye, S(xe; θm+1s )) ≈ −ηs(1− α)∇θtLce(T (xr; θmt ), S(xr; θms ))
∇θsLce(T (xr; θmt ), S(xr; θms ))⊺
∇θsLce((ye, S(xe; θm+1s ))) ≈ −ηs(1− α)h∇θtLce(T (xr; θmt ), S(xr; θms )),
(19) where
h =∇θsLce(T (xr; θmt ), S(xr; θms ))⊺
∇θsLce(ye, S(xe; θm+1s )).

heading: D Hyperparameters
text: Hyperparameter
For our method, online distillation and meta distillation baselines, we fix the teacher learning rate at 3e− 5.

heading: E More ablation study
text: E.1 Datasets for Student’s Feedback
In our method, we utilize the feedback from the student model on the provided validation set of GLUE datasets directly. In this section, we investigate the impact of utilizing feedback derived from a new validation set that has been separated from the original training set.
We random sample 5 % and 10 % samples of the training set to generate a new validation set respectively. Then we apply our method to the new training set.
The data used to measure the generalization of the student, whether it be from an existing validation set or a newly separated set, remains informative in both cases. As such, it is reasonable to expect that the feedback provided by the student to the teacher would not exhibit significant differences between the two sources.
Our experiments demonstrate that utilizing feedback from a validation set, whether pre-existing or newly separated from the training set, does not lead to significant variations in performance. However,
it should be noted that the number of training samples may play a role in the results. When a subset of the training set is selected to form a new validation set, the number of training samples is reduced. This reduction may lead to overfitting in datasets of small or medium size, as there is not enough data information provided to the model. Conversely, in large datasets, the number of samples is sufficient to encompass a substantial portion of the data information, thus having minimal impact on the results.
E.2 Ratio of Teacher’s Self-evolution
A student-friendly teacher should strike a balance between self-evolution and knowledge transfer. It is believed that an excessive focus on self-evolution may result in neglect of feedback provided by the student, leading to instruction that is not centered on the student’s needs. Conversely, inadequate focus on self-evolution may prevent the teacher from improving their own abilities, resulting in suboptimal instruction for the student. In either scenario, the outcome is not conducive to fostering a student-friendly environment.
Therefore, we ablate on the ratio of the teacher’s self-evolution to see how it contributes to the performance of the student. α is the ratio of the teacher’s loss with respect to ground truth in eq. (11). We set it from {1.0,0.8,0.6,0.4}.
In table 7, the performance of the student exhibits a unimodal distribution, which is in agreement with our proposed assumption. Specifically, the results indicate that when the ratio of the teacher’s self-evolution is set at 0.6, the performance of the student is optimal.

heading: F Analysis
text: We further discuss some design choices of current methods, including the initialization state of the teacher and the updating order of the teacher and student models. Following (Guo et al., 2022), we apply the entropy gap to evaluate these design choices.
F.1 Impact of the Teacher’s initial state
While vanilla distillation and meta distillation employ a two-stage training approach, online distillation and LGTM employ a one-stage joint training strategy for the teacher and student models. The key difference is whether to involve fine-tuning the teacher network on target task. In this study, we
investigate the impact of the teacher network’s state on the student network.
A teacher network initialized in the same state as the student network can maintain the student network’s progress at all times, but its capabilities may be relatively weak. In contrast, a converged teacher network has superior performance but also a larger gap, which can prevent the student network from gaining knowledge effectively.
As show in fig. 5, a lower initial confidence gap between the teacher model and the student model leads to more efficient knowledge transfer. When the initial ability gap is relatively high, it takes more iterations for the student model to catch up to the fine-tuned teacher model. In contrast, when the initial ability gap is lower, a teacher model initialized at the same state as the student model is able to transfer knowledge to the student more quickly. Specifically, in the early stages, the teacher model focuses more on self-evolution than knowledge transfer, causing the entropy gap to increase. Then, the teacher model shifts its focus towards knowledge transfer, resulting in an increasing and then decreasing trend in the entropy gap.
F.2 Prioritizing the Teacher or Student
Online distillation and meta distillation and LGTM all use bi-level optimization. However, online distillation and LGTM updates the teacher network followed by the student network, while meta distillation updates the student network followed by the teacher network. In this section, we study the optimal order for updating the teacher network and student network in knowledge distillation.
As shown in fig. 6, updating the teacher model first could lead to a lower entropy gap and faster convergence speed. We assume that the teacher could formulate an appropriate ‘teaching plan’ for the student in this updating order.
The teacher should strive to guide the student to identify the most important samples and information, to help the student develop a deep and general understanding of the task. Furthermore, the teacher should also take into consideration that some samples may be difficult for the teacher itself to classify or understand. And for those samples, a lower criterion should be set for the student, which may form a more student-friendly decision boundary.
Therefore, the teacher’s output serves as a dynamic learning target for each sample. By updating based on the student’s feedback in advance, the
teacher is able to reach a state that is optimal for the student’s learning. In this case, the teacher could provide an appropriate learning signal. Leveraging this updated supervision signal, the student could make up for the ability gap faster. For the other two updating orders, the teacher hasn’t updated yet, lacking of making trade-offs between the samples that are more beneficial for generalization and those that are more challenging to learn from. This may lead to a certain degree of lag in knowledge transfer, resulting in a larger entropy gap between the student and the teacher.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Yes. Section "Limitations"
3 A2. Did you discuss any potential risks of your work? Yes. Section "Ethics Statement"
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract + End of Section 1: Introduction
3 A4. Have you used AI writing assistants when working on this paper? Checking the presentation style of some sentences via ChatGPT. Use prompt like "help me rephrase XXX". However, sometimes ChatGPT will generate very wordy sentences and we haven’t used many recommendations.
B 3 Did you use or create scientific artifacts? Section 5.1. We use the GLUE benchmark.
3 B1. Did you cite the creators of artifacts you used? Section 5.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We haven’t discussed the term + license explicitly since they are in the GLUE paper and other papers we cited.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? GLUE has widely been used by the research community. We are writing a research paper so we haven’t used spaces to discuss the intended usage of GLUE.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We are using the datasets from GLUE.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. We are using the datasets from GLUE.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.1 and Appendix D
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No, we reported results on the test set of the GLUE benchmark. There is limitation to the total number of submissions each person can make for the GLUE benchmark.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5 and Appendix D
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Additionally, while our experiments have been limited to text classification tasks, which are relatively simple for current pre-trained language models, future work should explore the application of LGTM to more complex text generation tasks