Continual Knowledge Distillation for Neural Machine Translation
Year: 2023

Abstract:
While many parallel corpora are not publicly accessible for data copyright, data privacy and competitive differentiation reasons, trained translation models are increasingly available on open platforms. In this work, we propose a method called continual knowledge distillation to take advantage of existing translation models to improve one model of interest. The basic idea is to sequentially transfer knowledge from each trained model to the distilled model. Extensive experiments on Chinese-English and German-English datasets show that our method achieves significant and consistent improvements over strong baselines under both homogeneous and heterogeneous trained model settings and is robust to malicious models.1

heading: 1 Introduction
text: Current neural machine translation (NMT) systems often face such a situation: parallel corpora are not publicly accessible but trained models are more readily available. On the one hand, many data owners are usually unwilling to share their parallel corpora with the public for data copyright, data privacy and competitive differentiation reasons, leading to recent interests in federated learning for NMT (Wang et al., 2021b; Roosta et al., 2021). On the other hand, trained NMT models are increasingly available on platforms such as Hugginface (https://huggingface.co) and Opus-MT (https://opus.nlpl.eu/Opus-MT) since these models can be directly used without public access to the original training data.
As a result, a question naturally arises: can we take advantage of increasingly available trained NMT models to enhance one NMT model of interest? In this work, we propose a method called Continual Knowledge Distillation (CKD) to address
∗Corresponding authors. 1The source code is available at https://github.com/
THUNLP-MT/CKD.
this problem for NMT. As shown in Figure 1, we assume that multiple trained NMT models (i.e., teachers) are available to “educate” one NMT model of interest (i.e., student) in a sequential manner, which means that teacher models to arrive in the future are not accessible at the current time step. We also assume that the training set of the student model, a transfer set, and a test set are available, but the training set of the teachers are unavailable. CKD aims to continually improve the translation performance of the student model on the test set by sequentially distilling knowledge from each incoming teacher model to the student model.
As its name suggests, CKD is an intersection of knowledge distillation (Hinton et al., 2015) and continual learning (Kirkpatrick et al., 2017). On the one hand, CKD differs from standard knowledge distillation in that the knowledge is transferred from teacher models to the student model asynchronously instead of synchronously. As a result, the knowledge transferred to the student model from previous teacher models can be overridden by an incoming teacher model, which is often referred to as the catastrophic forgetting problem (Kirkpatrick et al., 2017). The situation aggravates when not all teacher models convey knowledge benefi-
7978
cial to the student model. On the other hand, CKD is different from conventional continual learning methods by focusing on learning one task (i.e., enhancing the student model) rather than learning many different tasks. The learning process is still very challenging as compared with standard continual learning because the original training data of teacher models is inaccessible to the student model. Consequently, we have to resort to knowledge distillation at each time step to make the most of teacher models.
To address these aforementioned challenges, we propose to fuse two knowledge sources for the student model at each time step: filtering the new knowledge from the current teacher model (i.e., knowledge filtration) and inheriting the old knowledge from the previous student model (i.e., knowledge inheritance) simultaneously. Experimental results show that our method significantly and consistently outperforms strong baselines under both homogeneous and heterogeneous teacher settings for Chinese-to-English and German-to-English translation. And it is also robust to malicious teachers.

heading: 2 Approach

heading: 2.1 Problem Statement
text: Let Θ = {θ∗1,θ∗2, . . . } be a sequence of frozen trained NMT models (i.e., teacher models), where θ∗t denotes the t-th teacher model. Let ϕ0 be an NMT model of interest (i.e., student model) and ϕt be the student model at time step t. We use x = x1, . . . , xI to denote a source-language sentence and y = y1, . . . , yJ to denote a target-language sentence. We use y<j = y1, . . . , yj−1 to denote a partial translation. Dtrain = {⟨x(m),y(m)⟩}Mm=1 represents the training set of the student model. Dtrans = {⟨x(n),y(n)⟩}Nn=1 represents the transfer set that a teacher model uses to “educate” the student model. Dtest is a test set used to evaluate the student model. We use BLEU(Dtest,ϕt) to denote the BLEU score the student model at time step t obtains on the test set.
Given an initial student model ϕ0, our goal is to maximize BLEU(Dtest,ϕt) by taking advantage of Θ, Dtrain, and Dtrans.

heading: 2.2 Training Objective
text: As shown in Figure 1, the student model ϕt at time step t is determined by the current teacher model θ∗t that encodes new knowledge and the previous learned student model ϕ̂t−1 that encodes
previously learned knowledge. Therefore, the overall training objective of CKD is composed of three loss functions:
ℓ(ϕt,θ ∗ t , ϕ̂t−1, Dtrain, Dtrans)
= ℓCE(ϕt, Dtrain) + λℓKF(ϕt,θ ∗ t , Dtrans)
+ (1− λ)ℓKI(ϕt, ϕ̂t−1, Dtrans), (1)
where ℓCE(ϕt, Dtrain) is the standard cross entropy loss defined as
ℓCE(ϕt, Dtrain)
= − M∑
m=1
J(m)∑
j=1
P (y (m) j |y (m) <j ,x (m);ϕt), (2)
Note that J (m) is the length of the m-th target sentence y(m). In Eq. 1, ℓKF(ϕt,θ∗t , Dtrans) is a knowledge filtration loss (see Sec. 2.3) that filters the knowledge transferred from θ∗t , ℓKI(ϕt, ϕ̂t−1, Dtrans) is a knowledge inheritance loss (see Sec. 2.4) that inherits the knowledge transferred from ϕ̂t−1, and λ is a hyper-parameter that balances the preference between receiving new and inheriting old knowledge.
Therefore, the learned student model at time step t can be obtained by
ϕ̂t = argmin ϕt
{ ℓ(ϕt,θ ∗ t , ϕ̂t−1, Dtrain, Dtrans) } .
(3)

heading: 2.3 Knowledge Filtration
text: In standard knowledge distillation (Hinton et al., 2015), an important assumption is that the teacher model is “stronger” than the student model, which means that the teacher model contains knowledge that can help improve the student model. Unfortunately, this assumption does not necessarily hold in our problem setting because it is uncertain what the next incoming teacher model will be. As a result, there are two interesting questions:
1. How do we know whether the teacher model contains knowledge useful to the student model?
2. How do we locate and transfer the useful knowledge from the teacher model to the student model?
Intuitively, the teacher and the student can do the same “test paper” in order to find where the

heading: Teacher > Student Teacher Student
text: teacher can help the student. Figure 2 shows an example. Given a (Romanized) Chinese sentence and its English translation, both the teacher and student models predict every target word yj given the source sentence x and the partial translation y<y. The quality of a prediction can be quantified as a real-valued number. If the teacher model performs better than the student model on a target word (e.g., “Prof.”), it is likely that the teacher model contains knowledge useful to the student model in this case. On the contrary, the teacher model is probably not more knowledgable than the student model regarding this case if its prediction is worse than that of the student (e.g., “will”).
More formally, we use Q(yj ,y<j ,x,ϕ) to quantify how well a student model predicts a target token. It can be defined in the following ways: 2
1. Token entropy: calculating the entropy of target tokens without using the ground truth token.
Q(yj ,y<j ,x,ϕ)
= − ∑
y∈Y P (y|y<j ,x;ϕ)
× logP (y|y<j ,x;ϕ) (4)
where Y is the vocabulary of the target language.
2Note that it is also possible to define sentence-level quantification functions Q(y,x,ϕ). If the teacher performs better than the student at sentence-level predictions, all target words with the sentence are considered positive instances for knowledge transfer. As our preliminary experiments show that the more fine-grained word-level quantification functions are much better than its sentence-level counterparts, we omit the discussion of sentence-level quantification functions due to the space limit.
2. Hard label matching: checking whether the predicted token is identical to the ground truth.
Q(yj ,y<j ,x,ϕ)
= δ ( yj , argmax
y P (y|y<j ,x;ϕ)
) (5)
where δ(y, y′) returns 1 if y is identical to y′ and 0 otherwise.
3. Token-level cross entropy: calculating tokenlevel cross entropy using the given model.
Q(yj ,y<j ,x,ϕ) = − logP (yj |y<j ,x;ϕ) (6)
The quantification function for a teacher model Q(yj ,y<j ,x,θ) can be defined likewise.
Since the transfer set Dtrans can be equivalently seen as a collection of tuples {〈
y (n) j ,y (n) <j ,x
(n) 〉∣∣∣j ∈ [1, J (n)], n ∈ [1, N ] } ,
(7) it can be divided into two parts depending on the comparison between the predictions of teacher and student models: a positive subset D+trans and a negative subset D−trans. A tuple ⟨yj ,y<j ,x⟩ belongs to D+trans if Q(yj ,y<j ,x,θ ∗ t ) is greater than Q(yj ,y<j ,x,ϕt). Otherwise, it is a negative instance that belongs to D−trans.
After splitting the transfer set into two parts, it is natural to apply standard knowledge distillation using the positive subset D+trans:
ℓKD(ϕt,θ ∗ t , D + trans)
= ∑
⟨yj ,y<j ,x⟩∈D+trans
KL ( P (yj |y<j ,x;θ∗t ) ∣∣∣∣
P (yj |y<j ,x;ϕt) ) (8)
However, one problem is that D+trans may be very small in most cases in practice, making training efficiency very low.
Therefore, instead of discarding the negative subset D−trans, we introduce a new loss function to make the most of negative instances. In analogy to humans, teachers can educate students by telling them what not to do. We expect that the student model can learn from D−trans in the same way. Our intuition is that erroneous tokens with a high probability in teacher model’s output distribution are critical because the student is prone to make the same mistakes. Pushing the output distribution of the student model away from the poor target distribution may enable the student model to avoid making the same mistakes. As a result, D−trans can be leveraged effectively and the overall learning efficiency will be improved significantly. Accordingly, the negative KD loss function on the negative subset is defined as
ℓNEG(ϕt,θ ∗ t , D − trans)
= min ( 0, α− ℓKD(ϕt,θ∗t , D−trans) ) (9)
where α is a hyper-parameter that controls the activation of the loss.
Finally, the knowledge filtration loss is the combination of the two functions:
ℓKF(ϕt,θ ∗ t , Dtrans)
= ℓKD(ϕt,θ ∗ t , D + trans)+ℓNEG(ϕt,θ ∗ t , D − trans).
(10)

heading: 2.4 Knowledge Inheritance
text: To circumvent the catastrophic forgetting problem, we introduce a loss function to inherit knowledge learned from previous time step for the current student model:
ℓKI(ϕt, ϕ̂t−1, Dtrans)
= ∑
⟨yj ,y<j ,x⟩∈Dtrans KL
( P (yj |y<j ,x; ϕ̂t−1 ∣∣∣∣
P (yj |y<j ,x;ϕt) ) . (11)

heading: 3 Experiments
text: To evaluate the effectiveness of our method, we conduct experiments on Chinese-to-English and German-to-English translation under three representative settings including homogeneous, heterogeneous and malicious teacher settings.

heading: Model Domain Training Dev. Test

heading: 3.1 Setup
text: Configurations. For the Chinese-to-English translation experiments under the homogeneous teacher setting, both the teachers and the student are Transformer-base models (Vaswani et al., 2017). Besides model architecture, there are a few other factors that may affect performance, e.g., teacher performance, student performance, model domain, and the order that the teachers arrive. To investigate the impact of model performance and model domain, we leverage five parallel corpora of representative domains as shown in Table 1, among which two are in million scale, one is in middle scale, and the other two are in small scale. Correspondingly, five Transformer-base models are trained on these corpora, denoted as A, B, C, D, and E, respectively. Intuitively, A and B are welltrained while D and E are under-trained due to the training data sizes. To investigate the impact of the order of teachers, we enumerate all the six permutations of A, B and C. In addition, we append D and E to the end of each permutation to simulate the “weak” teacher scenario. Therefore, we have six configurations in total.
Specially, we use a string like “ABDE → C” to denote a configuration, which means C is the student, A, B, D and E are the teachers and A arrives first, then B and so on. For simplicity, we use the training set of C as both the training set Dtrain and the transfer set Dtrans in CKD, and the test set of C is leveraged as Dtest. The goal in this configuration is to improve the performance of C on Dtest. In summary, the six configurations are “BCDE→A”, “CBDE→A”, “ACDE→B”, “CADE→B”, “ABDE→C”, and “BADE→C”.
For clarity, the differences of other aforementioned settings with this one will be given in the corresponding sections later.
Evaluation. We leverage the following two metrics to evaluate our method:
• BLEU (Papineni et al., 2002) 3: the most widely used evaluation metric for machine translation.
• Accumulative Degradation (AD): measuring the accumulative occasional quality degradation in all steps, which should be avoided as much as possible. AD from step 1 to t is defined as follows:
AD = t∑
k=1
max(0,B(ϕk−1)−B(ϕk)), (12)
where B(·) denotes BLEU(Dtest, ·). Baselines. Our method is compared with the following baseline methods:
• Knowledge Distillation (KD) (Khayrallah et al., 2018) for NMT which applies vanilla knowledge distillation on each token trivially.
• Elastic Weight Consolidation (EWC) (Saunders et al., 2019; Thompson et al., 2019) which is a representative continual learning method that adds an EWC term as a penalty to alleviate catastrophic forgetting.
• Continual Learning for NMT (CL-NMT) (Cao et al., 2021) which is a representative work on multi-step continual learning in NMT.
3BLEU score is computed using multi-bleu.perl on the corresponding test set for each student model.

heading: 3.2 Implementation Details
text: We use byte pair encoding (BPE) (Sennrich et al., 2016) with the vocabulary size of 32k. The hyperparameters of the Transformer-base models are set mostly following Vaswani et al. (2017). We use Adam (Kingma and Ba, 2014) optimizer, in which β1 = 0.9, β2 = 0.98. During training, learning rate is 7 × 10−4 and dropout rate is 0.1. Batch size is 6, 000. λ in Eq. 1 in step t is defined as λ = 0.9991−0.999
t−1 1−0.999t following (Cao et al., 2021).
More details of hyper-parameters are provided in Appendix B.

heading: 3.3 Quantification Function Selection
text: We first evaluate the three candidates for the quantification function Q defined in Sec. 2.3. A proper Q should correlate well with model performance and generalize well to a wide range of domains. To this end, we collect six widely used datasets of different domains and varying sizes and evaluate the correlations between the candidates and corpuslevel BLEU scores on them. The Pearson correlation coefficients between token entropy (Eq. 4), hard label matching (Eq. 5) and token-level cross entropy (Eq. 6) are −0.5622, 0.8091 and 0.7792, respectively. Both hard label matching and tokenlevel cross entropy are strongly correlated with
corpus-level BLEU. However, hard label matching can not break a tie when both the teacher and student models’ predictions are correct or incorrect. Therefore, we adopt token-level cross entropy as Q in the rest of this work. Examples and more discussions can be found in Appendix C.

heading: 3.4 Chinese-to-English Translation
text: Homogeneous Teacher Setting. In this setting, all the student and teacher models are of the same model architecture, which is Transformer-base. For space limitation, we only show results of three configurations in Table 2. The full results for all configurations can be found in Appendix D.1. From Table 2 we can observe that:
(1) Our method achieves improvements over the initial student model in all steps and configurations, and outperforms all baselines significantly. It indicates that our method is effective for leveraging diverse teacher models to continually improve the performance of the student model on its test dataset.
(2) Our method achieves zero or near-zero accumulative performance degradation (AD) scores in all configurationss, indicating our method is also effective to retain acquired knowledge. Especially, when encountering model D (step 3), nearly all baselines face severe quality degradation compared with step 2, while our method even achieves gain in ACDE → B, which further justifies the effectiveness of our method.
(3) All baselines perform poorly after four steps of distillation, indicating that the problem we aim to resolve is challenging. Specifically, KD, the worst one, suffers from severe performance degradation as averaged ∆BLEU and AD scores are −7.09 and 9.88, respectively. We argue this is due to KD implicitly assumes that the teacher models are helpful such that it is prone to less beneficial knowledge provided by them. EWC is designed to alleviate catastrophic forgetting and achieves better ∆BLEU and AD scores than KD. However, EWC still fails to achieve improvement over the initial student model, i.e., all ∆BLEU scores are negative. CL-NMT is specially designed for multi-step continual learning in NMT and achieves the best ∆BLEU and AD scores among baselines. Nevertheless, its average ∆BLEU score is significantly smaller than ours (0.40 v.s. 1.70) and its average AD score is significantly worse than ours (2.77 v.s. 0.05). Overall, the problem to be resolved is challenging and our method is remarkably effective
than baselines. (4) Despite the promising results, slight performance degradation can still be observed occasionally for our method. Therefore, there is still room for further improvement on retaining acquired knowledge.
Heterogeneous Teacher Setting. Using logits as the medium to transfer and retain knowledge, our approach is model-agnostic and scalable. To justify that, we replace the Transformer-base teacher models with RNN (Bahdanau et al., 2014) and Transformer-big (Vaswani et al., 2017) models, and repeat the experiments in Table 2 with other settings remaining identical. Table 3 shows similar results as Table 2 that our method outperforms all baselines significantly and also achieves zero or near-zero AD scores, indicating that our method is extensible to different model architectures. Interestingly, all the baselines encounter serious performance degradation while the ∆BLEU of our method is nearly zero, indicating that distilling knowledge from a teacher of a completely different architecture may be extremely difficult. It deserves more thoughtful investigation and we leave it as future work.
Malicious Teacher Setting. Robustness to malicious models is critical in our scenario as only the parameters rather than training data of teachers are available. We simulate malicious teacher models by shuffling the outputs of a well-trained model within a batch so that the model answers almost completely wrong with high confidence. We repeat the experiments in Table 2 with other settings remaining identical. As shown in Table 4, our approach is far less affected by the malicious model with three different teacher model architectures. Moreover, it could be further explored to detect and skip malicious models to save computational resources directly.

heading: 3.5 Larger Scale Chinese-to-English Translation
text: We scale up the dataset size of the Chinese-toEnglish translation experiment under the homogeneous teacher setting from one million to ten million. Other settings are similar to the original experiments and are detailed in Appendix D.2. As shown in Table 5, our method remains effective while all baseline methods fail to achieve positive quality gain (∆BLEU). This demonstrates that the performance of the baseline methods does not improve as the size of the data and performance of the models increase, while our method remains valid. Thus, it shows that our method is scalable for corpus of different sizes.

heading: 3.6 German-to-English Translation
text: We also conduct experiments on German-toEnglish datasets. Models are trained on four different datasets from different domains. Other settings are similar to the Chinese-to-English experiments and are detailed in Appendix D.3. The average values among each of the homogeneous, heterogeneous, and malicious teacher settings are reported in Table 6. Due to the large domain differences
of the datasets, only our method consistently obtains BLEU gains and zero or near zero AD scores, exceeding the baselines, demonstrating that our approach is effective for different language pairs.

heading: 3.7 Ablation Study
text: Table 7 shows the effect of the negative KD loss ℓNEG (Eq. 9) in knowledge filtration and the knowledge inheritance loss ℓKI. Results at the beginning (t = 1) and later step (t = 4) for Chinese-toEnglish translation under the homogeneous teacher setting are reported. We can observe that:
1. Removing either ℓNEG (row 2) or ℓKI (row 4) hurts the performance, indicating both of them are effective.
2. Comparing row 1 with row 2, we can conclude that the negative subset of the transfer set where the teacher performs worse than the student (D−trans) also contains valuable nontrivial knowledge. Furthermore, trivially applying vanilla KD loss ℓKD on D−trans (row 2 v.s. 3) brings no gain. Therefore, our proposed negative KD loss is effective for making less beneficial knowledge play a good role.
3. Without ℓKI, the performance drops severely, especially at a later step, verifying that knowledge inheritance is essential for retaining acquired knowledge.

heading: 3.8 Comparison with Multi-teacher Knowledge Distillation
text: Multi-teacher KD (Freitag et al., 2017), aka ensemble KD, generally requires all teachers available at the same time, which violates the definition of our problem and may result in enormous computational and memory cost as teacher number grows. Moreover, it is also non-trivial to adapt it to our scenarios due to potential unbeneficial knowledge provided by teachers. Therefore, we do not include it as a major baseline in the experiments above. Nevertheless, in this section, we still provide a comparison of our method with vanilla multi-teacher KD which averages the outputs of all teachers as the target distribution for analysis. The BLEU score of vanilla multi-teacher KD averaged over six configurations is 30.49, lower than our 31.18, indicating that our method is superior to vanilla multi-teacher KD although the comparison is more favorable to it. More details on comparison in terms of task definition, robustness and storage requirement are analyzed in Appendix D.4.

heading: 4 Related Work
text: Knowledge Distillation. Knowledge distillation (KD) is the most widely used technique for transferring knowledge between models (Hinton et al., 2015). Despite of their effectiveness, conventional KD methods usually implicitly assume that the teacher model is superior or complementary to the student model (Gou et al., 2021). Although recently Qin et al. (2022) allow a big model to learn from small models, they still require that the small models are better than the big model for the given tasks and datasets. However, the assumption does not necessarily hold in our scenario due to the diversity of teacher models. Multi-teacher KD (Freitag et al., 2017; You et al., 2017; Fukuda et al., 2017; Mirzadeh et al., 2020; Liu et al., 2020), which distills knowledge from multiple teachers simultaneously, is highly related to this work. Generally, multi-teacher KD requires all teachers to be available at the same time, which will result in enormous extra memory consummation as the number of teachers grows. More importantly, new teachers may be released constantly (Wolf et al., 2020), which can not be seen in advance. Therefore, multi-teacher KD methods are not feasible to our scenario. L2KD (Chuang et al., 2020) leverages sequential KD to continually learn new tasks, having different goal and challenges compared with
our scenario. Another line of related work is selective distillation (Gu et al., 2020; Wang et al., 2021a; Shi and Radu, 2022), which selects data and losses to accelerate KD or enhance model robustness. In contrast, we select data for conducting different ways of distillation in our proposed method.
Continual Learning. Continual learning (CL) for neural machine translation (NMT) aims at learning knowledge of new domains (Thompson et al., 2019; Liang et al., 2021; Cao et al., 2021) or languages (Neubig and Hu, 2018; Garcia et al., 2021; Huang et al., 2022) without forgetting old knowledge. Our scenario also requires learning new knowledge but focuses on improving performance of the student on its test set instead. Moreover, alleviating the negative impact of the less beneficial knowledge conveyed by “weak” teachers is essential in our scenario, which is hardly explored in CL for NMT. While our scenario is a multi-step process, multi-step CL is less explored in NMT (Cao et al., 2021; Liang et al., 2021). Zeng et al. (2019) address a similar task of adapting from multiple out-of-domain models to a single in-domain model. Nevertheless, they assume the training data for the out-of-domain models are available, which is inaccessible in our scenario. Besides, leveraging highresource language NMT models to improve lowresource language translation has also attracted intensive efforts (Neubig and Hu, 2018; Lakew et al., 2019; Liu et al., 2021; Huang et al., 2022), which can be a future extension of our method.

heading: 5 Conclusion and Future Work
text: To take advantage of increasingly available trained neural machine translation (NMT) models to improve one model of interest, we propose a novel method named continual knowledge distillation. Specially, knowledge from the trained models is transferred to the interested model via knowledge distillation in a sequential manner. Extensive experiments on two language pairs under homogeneous, heterogeneous, and malicious teacher settings show the effectiveness of our proposed method.
In the future, we will further explore the effect of the teacher model order. It is also worth involving more sophisticated methods in knowledge filtration, such as gradient-based and meta-learning-based methods. Moreover, it is also a promising research direction to exchange knowledge among all the models such that all of them achieve improvement.

heading: Limitations
text: There are some limitations that have yet to be addressed. Since we use the predicted probability distributions of the model output as a medium for continual KD for NMT, the vocabulary of multiple models needs to be consistent. Overcoming it allows continual KD for NMT to be extended to models with different language pairs and different modalities. Also, although our approach is robust to malicious models, there are more diverse and sophisticated attacks in real-world that require more research on defense. In addition, the teacher and student models must be trained on the same language pair. Further studies can consider more general scenarios without the above limitations. There are other approaches worth exploring in order to address the transfer of knowledge from models rather than their training data besides sequential manner. For example, it is also possible to explore various distillation methods like organizing teacher models into batches or pipelines.

heading: Ethics Statement
text: In practice, a provider may publicly release a model but may not wish its knowledge to be transferred into another one. Applying our method on such models will result in model stealing (He et al., 2022) related ethical concerns. How to detect this kind of misconduct still needs further exploration. Although sharing knowledge without exposing private data is one of the potential benefits of our method, models produced by our method are still vulnerable to attacks such as membership inference (Hisamoto et al., 2020), and the private training data could still be stolen from the model.

heading: Acknowledgments
text: This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Natural Science Foundation of China (No. 61925601, 62276152, 62236011). We thank all the reviewers for their valuable and insightful comments. We also thank Weihe Gao, Kaiyu Huang and Shuo Wang for their help.

heading: A Datasets for Chinese-to-English Translation
text: The statistics of the datasets have been shown in Table 1. The training data for the news, oral, internet, speech and subtitle domains are randomly sampled from LDC 4, AI Challenger 2018 (Wu et al., 2017), translation2019zh 5, TED transcripts (Tiedemann, 2012) and Subtitles (Lison and Tiedemann, 2016), respectively. Newstest 2018 and NIST 02-09 from LDC are used as the development and test set for the news domain. The AI Challenger 2017 dataset is used as the test set for the oral domain. For the other corresponding domains, the development and test sets provided along with the training sets are used as development and test sets accordingly.

heading: B Hyper-parameter Search
text: For hyper-parameter α in Eq. 9, we try multiple values in Table 8, and choose 0.1 as the default value.
Since α regulates how much the student output distribution is pushed away from the negative distribution, we also try to regulate the proportion of positive and negative KD losses that work in a similar way. We modify Eq. 10 to:
ℓKF(ϕt,θ ∗ t , Dtrans) =
kaℓKD(ϕt,θ ∗ t , D + trans)+kbℓNEG(ϕt,θ ∗ t , D − trans)
(13) 4LDC2002E18, LDC2003E07, LDC2003E14, part of
LDC2004T07, LDC2004T08 and LDC2005T06 5https://github.com/brightmart/nlp_chinese_ corpus, version 1.0.
By adjusting ka and kb, we can regulate the weights of positive and negative losses. As shown in Table 9, we still use the original settings since no significant performance improvement is found when adjusting ka : kb.

heading: C Exploring Knowledge Filtration Quantification Function

heading: C.1 Examples
text: In Table 10, we show three examples to demonstrate how the default quantification function (token-level cross entropy) works in knowledge filtration.
• In the first case, we apply standard knowledge distillation because the teacher model assigns a higher probability of the ground truth token “decorations” than student, indicating a better distribution from the former.
• In the second case, the output from the teacher model is discarded because the negative KD loss exceeds the threshold. It might be a reasonable choice since the output of the teacher is too far from the ground truth token.
• In the third case, the teacher model have slightly worse predictions than students, motivating the student model not to make similar error-prone mistakes.

heading: C.2 Alternatives to Quantification Function
text: The advantage of token-level cross entropy is that the predictions corresponding to the tokens in the transfer set Dtrans can be divided into two mutually disjoint parts depending on the comparison between the predictions of teacher and student models. In contrast, hard label matching divides Dtrans according to whether the teacher and student models predict the ground-truth token correctly, which will result in four parts due ties as shown in Table 11.
Are the advantages of these two metrics beneficial for our task? Is it possible to combine the beneficial properties? To answer the questions, we define several metrics in Table 11 to compare these two metrics at a fine-grained level. And the effects of these metrics are shown in Table 12. It could be found that token-level cross entropy always performs better because fewer samples are discarded such that more knowledge is transferred in knowledge distillation.
token yj and candidates matching with yj are highlighted in yellow . “Student” and “Teacher” show the top 5 predicted candidate tokens and their corresponding probabilities. α = 0.1 is the threshold here.

heading: D Detailed Results

heading: D.1 Chinese-to-English Translation
text: The full results for the Chinese-to-English homogeneous model setting for all configurations are shown in Table 13. Our experiments are conducted on NVIDIA A100 GPUs. Each distillation process requires 48 GPU hours and is run only once due to computational budgets.

heading: D.2 Larger Scale Chinese-to-English Translation
text: To illustrate the scalability of our method, we repeat Chinese-to-English translation experiments under homogeneous model setting (Sec. 3.4) using larger and complete corpora without sampling. As shown in Table 15, we choose five full datasets for news, oral, internet, speech and subtitle domains, respectively: WMT20, AI Challenger 2018, translation2019zh, TED transcripts and Subtitles. Compared with Table 1, only the training set of model F ,G and H are replaced with bigger datasets. And the validation sets, test sets and other settings are kept identical. F , G, H , I , J are all Transformerbase models. The three models F , G and H are combined in different orders to form six groups of experiments. To simulate the most challenging scenario, I and J with weaker performance are added at the end of these experiments to test the performance of our method on poor models. According

heading: Model Domain Training Dev. Test
text: to the task definition illustrated above, the transfer set of each set of experiments is the training set of the according student model.
The results are shown in Table 14. Similar to results in Sec. 3.4, our method outperforms baselines significantly and consistently, justifying that our method is scalable and generalizes to different corpus sizes. For example, without using any extra data, our method yield up to +1.49, +0.20, +0.35 BLEU scores on WMT20, AI Challenger 2018 and translation2019zh datasets, respectively.

heading: D.3 German-to-English Translation
text: As shown in Table 17, the training data for Germanto-English translation are from WMT16, TildeMODEL v2018 (Rozis and Skadin, š, 2017), Tanzil v1 and Europarl (Koehn, 2005), respectively. We sample 4, 000 sentences from the original corpus as the development set and 8, 000 sentences as the test set. Model N has weaker performance. Other settings are kept identical to the Chinese-toEnglish translation experiments. K, L, M , N are all Transformer-base models. The three models K, L and M are combined in different orders to form six groups of experiments. To simulate the most challenging scenario, N with weaker performance is added at the end of these experiments to test the performance of our method on poor models. According to the task definition illustrated above, the transfer set of each set of experiments is the training set of the according student model.
As shown in Table 16, our method performs similarly on the German-to-English language pair as it does on the Chinese-to-English language pair under the homogeneous model setting. We also repeat the experiments under the heterogeneous and ma-
licious model settings. As shown in Table 18 and Table 19, our method is also superior to the baselines under these settings for German-to-English translation.

heading: D.4 Comparison with Multi-teacher Knowledge Distillation
text: Multi-teacher distillation differs with our method in three aspects:
• Applicable scenario. Vanilla multi-teacher distillation averages the outputs of all teacher models as the target distribution, which requires all teacher models available at the same time, violating the task definition that a se-
quence of teacher models are distilled in many steps. It is impossible to get all teacher models at early steps. Thus, vanilla multi-teacher distillation cannot be used as a baseline method.
• Robustness. Vanilla multi-teacher distillation averages the output of all teacher models and is vulnerable to D−trans.
• Storage requirement. The memory footprint of vanilla multi-teacher distillation will exceed the available memories of GPUs as the number of teacher models increases. A straightforward way to alleviate the problem is storing the output of teachers in a similar way as the aforementioned knowledge inheritance. However, it is non-trivial to achieve a good balance between storage requirement and performance. Let |Dtrans| be token numbers of target sentences in Dtrans, Nstep be the number of steps, and NV be the output vocabulary size. The following three high-potential methods all face problems:
– Storing logits: It leaves room for integrating knowledge filtration. However, its storage requirement is|Dtrans| ·NV · Nstep, which is impractical since Nstep can be arbitrarily large.
– Storing top-1 tokens: It requires constant storage of |Dtrans|. However, knowledge filtration is hard if not impossible to be developed.
– Storing moving average of the logits: Its storage requirement is also constant, i.e., |Dtrans| · NV . However, it is prone to D−trans and knowledge filtration is also hard to be integrated.
In contrast, our knowledge inheritance has been shown to be effective and requires a constant size of storage (|Dtrans| ·NV ). Moreover, we still try our best to apply multi-teacher distillation on continual KD for NMT by storing the output logits of all teacher models. The
results in Table 20 show no performance advantage over our method.
Overall, our proposed method is superior to vanilla multi-teacher distillation for continual KD for NMT.

heading: E Exploring Loss Function of Knowledge Filtration
text: In this section, we will study the effects of inverse KL loss and modifying sources of knowledge filtration loss. All experiments are conducted under the homogeneous model setting on the Chinese-toEnglish language pair.
E.1 Inverse KL Loss
Trivial KL loss is zero-avoiding and concentrates on a single mode, while inverse KL loss covers the broad range and is zero-pursing.
KL(P1||P2) = − ∑
i
P1(i) ln P2(i)
P1(i) (14)
InvKL(P1||P2) = − ∑
i
P2(i) ln P1(i)
P2(i) (15)
The BLEU scores of the trivial KL loss and inverse KL loss are 31.07 and 30.29, respectively. Therefore, the trivial KL loss performs better in our task.

heading: E.2 Effect of Knowledge Filtration Loss
text: In the knowledge filtration loss ℓKF, ℓKD motivates the model to learn from student, while ℓNEG motivates the model to learn against student. Intuitively, ℓNEG is calculated from poor predictions from teacher models, and these error-prone distributions might provide empirical knowledge that motivates the student model not to make the same mistakes as the teachers. The output distributions from the student model could be improved by pushing them away from poor distributions of corresponding tokens output by teacher models. Conversely, random distributions are not beneficial to the student model. To verify this, we replace ℓNEG with the following noise:
Source of Noise Sample Sample Size BLEU
Uniform distribution 1 30.91 Normal distribution 1 30.86 Shuffled Batch (Attached) 1 29.98 Shuffled Batch (Attached) 5 29.99 Shuffled Batch (Detached) 1 30.52 Shuffled Batch (Detached) 5 30.62
Negative KD Loss 31.07
As shown in Table 21, performance is degraded when negative KD loss is derived from noise, thus illustrating that negative samples assist student models in avoiding errors and improving the efficiency of knowledge distillation.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
"Limitation" Section
3 A2. Did you discuss any potential risks of your work? "Ethics Statement" Section
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 3, Appendix B, Appendix C, Appendix D and Appendix E
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3.2, Appendix B and Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3.2, Appendix A, Appendix B and Appendix D
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3.2, Appendix A, Appendix B, caption of Table 2, caption of Table 3, caption of Table 4, caption of Table 5, caption of Table 6 and caption of Table 7
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.2, Appendix A and Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will further explore the effect of the teacher model order. It is also worth involving more sophisticated methods in knowledge filtration, such as gradient-based and meta-learning-based methods. Moreover, it is also a promising research direction to exchange knowledge among all the models such that all of them achieve improvement