Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation (2024)

Abstract: Incrementally expanding the capability of an existing translation model to solve new domain tasks over time is a fundamental and practical problem, which usually suffers from catastrophic forgetting. Generally, multi-domain learning can be seen as a good solution. However, there are two drawbacks: 1) it requires having the training data for all domains available at the same time, which may be unrealistic due to storage or privacy concerns; 2) it requires re-training the model on the data of all domains from scratch when adding a new domain and this is time-consuming and computationally expensive. To address these issues, we present a semi-supervised contrastive distillation framework for incremental neural machine translation. Specifically, to avoid catastrophic forgetting, we propose to exploit unlabeled data from the same distributions of the older domains through knowledge distillation. Further, to ensure the distinct domain characteristics in the model as the number of domains increases, we devise a cross-domain contrastive objective to enhance the distilled knowledge. Extensive experiments on domain translation benchmarks show that our approach, without accessing any previous training data or re-training on all domains from scratch, can significantly prevent the model from forgetting previously learned knowledge while obtaining good performance on the incrementally added domains.

heading: 1 Introduction
text: In the real scenario, translating an out-of-domain sentence is a common situation while it usually cannot work well due to domain discrepancy. An effective solution is to incrementally expand the capability of the existing translation model, i.e., continual learning (Silver et al., 2013). However, the biggest challenge is catastrophic forgetting when
∗ Work was done when Liang was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China.
† Jinan Xu is the corresponding author.
the model learns new knowledge and it would forget the previously acquired knowledge (Goodfellow et al., 2013; Gu and Feng, 2020). A theoretically good technique is multi-domain learning, which usually requires having all the training data available at the same time and re-training the model on all domains from scratch (Liang et al., 2023b). Nevertheless, in practice, it may be unfeasible because we sometimes cannot access the previous data due to storage or privacy concerns, and retraining would bring more training and resource consumption.
To overcome these drawbacks, many efforts have been devoted that fall into three categories, i.e., constructing pseudo data of previous domains/tasks, adding task-specific adapters, and regularizationbased learning. (i) The first category aims to create pseudo data of the previous task and mix them with the new task data for joint training (Kim and Rush, 2016; Liu et al., 2021; Ko et al., 2021). Although intuitive and effective, they generally require obtaining a large training data of previous tasks and are not flexible in practice. (ii) The second category is to add additional task-specific layers for new tasks and only optimizes these parameters with the new task data, having achieved impressive performance (Bapna and Firat, 2019a; Aharoni and Goldberg, 2020; Escolano et al., 2021; Liang et al., 2021a; Cao et al., 2021; Gu et al., 2019, 2021). However, the task-specific adapters may increase the difficulty of the model to be aware of which tasks the input belongs to and thus neglect the distinct task characteristics, which limits its application in practice. (iii) The third category essentially searches a trade-off between the new task and the previous ones through multi-objective training with an extra penalty item (e.g., L2 or EWC regularization) on the parameters (Khayrallah et al., 2018; Thompson et al., 2019). Therefore, previous methods usually lead to under- or over-constraint problems and achieve a suboptimal performance. Be-
10914
sides, they typically require the parallel data of the previous tasks/domains (Gu et al., 2022) and the time and space cost for computing the penalty item is expansive, especially with new tasks/domains appearing (Cao et al., 2021).
In this paper, to address the above issues, we present a Semi-supervised Contrastive Distillation (named SCD) framework for incremental neural machine translation. Specifically, to memorize the learned knowledge from previous domains, we propose to exploit unlabeled data from the same distributions of the older domains through knowledge distillation. To this end, we utilize the sourceside data related to the previous domains, e.g., the source-side data of validation set1, which is smallscale and easy to obtain compared to requiring parallel data. Furthermore, to guarantee distinct domain characteristics in the model as new domains appear, we devise a cross-domain contrastive objective to enhance the distilled knowledge, which encourages the model to learn to keep different domain characteristics and thus benefits translation for various domains.
We validate our proposed SCD framework on the commonly-used machine translation benchmark (Aharoni and Goldberg, 2020), which contains five domains. We incrementally add a single domain at each time to simulate the real-world situation. Extensive experiments show that our model effectively addresses the catastrophic forgetting issue and significantly outperforms related strong methods in terms of BLEU (Papineni et al., 2002) scores, demonstrating its effectiveness.
In summary, our main contributions are2:
• We propose a novel continual learning framework for incremental neural machine translation without accessing any previous training data or re-training on all domains from scratch. We also propose a cross-domain contrastive objective to enhance the distilled knowledge to guarantee distinct domain characteristics in the model.
• We conduct extensive experiments and systemic analysis on a more general scenario where m streams of data from different domains are fed to the model sequentially, and our approach can significantly prevent the model from forgetting previously acquired knowledge while obtaining good performance on the newly added domains.
• We show that our method can also achieve better 1Note that the target-side data is not used. 2https://github.com/XL2248/SCD
performance only with a handful of unlabeled data than that using a large of parallel data.

heading: 2 Methods
text: In this section, we first describe the problem definition § 2.1. Secondly, we introduce the proposed semi-supervised distillation method in § 2.2, which prevents the model from forgetting the previously learned knowledge. Then, to further ensure the domain characteristics, we present a cross-domain contrastive objective to enhance the distilled knowledge § 2.3. Finally, we elaborate on the training and inference in § 2.4.

heading: 2.1 Problem Statement
text: Domain-incremental training (Cao et al., 2021) aims to simulate training of the NMT model on real-world time streaming data, where the training domain data come from different times and is fed to the model in chronological order. And we indicate (Xa, Ya) and (Xb, Yb) as the training translation pairs for domain A and B, respectively. For example, as shown in Fig. 1, the model ΘA is firstly trained on a domain A. After a period of time, a new domain data B comes. Then, a model ΘAB , which needs to deal with both domains, is
trained incrementally based on Model ΘA without accessing the previous domain data A. The rest of the training process is similar to adding domain B.

heading: 2.2 Semi-supervised Distillation
text: Motivation. To continually learn new domains for translation, we exploit the knowledge distillation (KD) (Hinton et al., 2015) framework. Without loss of generality, we assume that we have already trained a model ΘA to solve domain A in stage 1 and we want to update it to learn how to also solve a new domain B. As illustrated in Fig. 1, we start by creating a copy of ΘA for domains A and B, i.e., ΘAB . The original ΘA and ΘAB models act as the teacher and the student in the KD framework, respectively. During training, we fix the model ΘA and only update ΘAB with the objective of (1) learning the new domain from the training data of domain B and (2) preserving the older domain’s knowledge by minimizing the loss function:
LKDAB = LB + αLKDA , LB = CE(Yb,ΘAB(Xb)),
LKDA = CE(ΘA(Xa),ΘAB(Xa)), (1)
where CE denotes the cross-entropy loss and LKDA denotes the CE loss between the token probability distribution of the student on domain A and the soft targets of the teacher ΘA, and α is the balancing coefficient. Here LB serves to let the student learn how to solve a new domain and LKDA helps it in preventing catastrophic forgetting of the old one. In the standard application of KD to continual learning, LKDA is computed on the new domain data (Shmelkov et al., 2017; Cao et al., 2021): this assumes that the old and new domains have the same data distribution (Dakwale and Monz, 2017).
However, the assumption does not satisfy the real-world machine translation where different domains are typically defined on extremely different data distributions. If we use the new domain data to compute the distillation loss, the model will bias the translation toward the new domain style when translating the sentence of the old domain. Therefore, preventing catastrophic forgetting when using only the new domain data can be challenging.
Dealing with Different Domain Distributions. To address this issue, we propose to augment the KD learning process with a data distribution resembling the one used to train the teacher model to solve domain A. Our assumption is that while the original training material for domain A may no
longer be available, we can still observe a stream of unlabeled data (Xa) from the same distribution, which is easy to obtain, e.g., the validation set of domain A.
By this way, the loss function LKDA represents the discrepancy between the teacher and student predictions for the old domains on a set of unlabeled data. In practice, the unlabeled data Xa are automatically labeled by the teacher model ΘA to produce the soft targets dataset of domain A. This data will be used to compute the loss LKDA . Meanwhile, a new labeled dataset for domain B is used to compute LB. By doing so, the student model should be able to minimize the discrepancy with the teacher on the old domains (i.e., minimizing the catastrophic forgetting) while learning the new domains.
This methodology can be trivially extended to the general case where the teacher is already trained on n domains and the student needs to solve a new domain. In this setting, we need to prevent the catastrophic forgetting of n different domains. We assume the availability of an unlabeled stream of data for each of the old domains to compute the individual distillation losses. For example, for three domains as the stage 3 shown in Fig. 1, the total loss is written as:
LKDABC = LC + α(LKDA + LKDB ), LC = CE(Yc,ΘABC(Xc)), LKDA = CE(ΘAB(Xa),ΘABC(Xa)), LKDB = CE(ΘAB(Xb),ΘABC(Xb)). (2)
In this way, the student model will maintain the relevant knowledge to solve the n domains by distilling it from the teacher on the unlabeled data stream, while also learning how to solve the new domain on the labeled data.

heading: 2.3 Cross-domain Contrastive Objective
text: In domain-incremental NMT, we require the model to simultaneously handle multiple domains and generate domain-aware translations. To guarantee the domain characteristics, we further propose a cross-domain contrastive objective to enhance the distilled knowledge. Particularly, as the stage 2 shown in Fig. 2, we use the output feature of the student model as an anchor feature HABa,i , and push it close to its original domain representation HAa,i provided by the teacher model. In contrast, we push apart the irrelevant pairs, e.g., the random one in the mini-batch HABa,j , j ̸= i. However, the simple negative sample cannot work well in distinguishing domain characteristics because they are different instances but come from the same domain. Therefore, we design a hard negative that is the same instance but encoded with another model for domain B. In this way, the only difference is that they are encoded by different domain models and thus we can distinguish the domain characteristics between domain A and B. That is, our negative samples include two parts: 1) Easy Negatives Xja (j ̸= i) randomly sampled and encoded by domain model ΘB; 2) Hard Negative Xia encoded with domain model ΘB . This forces the model ΘAB to capture and distinguish well domain A and domain B. Formally, the cross-domain contrastive training objective is defined by (N is the batch size):
LCCOAB = − log esim(H
AB a,i ,H A a,i)/τ
esim(H AB a,i ,H A a,i)/τ + N∑ j=1 esim(H AB a,i ,H B a,j)/τ
,
(3)
where sim(·, ·) is the cosine similarity and τ denotes a temperature hyperparameter.
Similarly, as the number of domains increases, we can easily extend Eq. 3 to a general setting. For example, for three domains as the stage 3 shown in Fig. 2, we require two cross-domain contrastive objectives LCCOAC and LCCOBC for domains A&C and B&C, respectively.

heading: 2.4 Training and Inference
text: At training, we train our model with the following objective at stage 2:
J = LKDAB + βLCCOAB , (4)
where β is the balancing hyper-parameter. Note that when training model ΘAB , the model ΘA and ΘB are frozen. During inference, only
the model ΘAB is used to generate translations for domains A and B. The rest of the training process is similar to the stage 2.

heading: 3 Experiments

heading: 3.1 Datasets
text: We use the domain translation dataset proposed by Koehn and Knowles (2017) to simulate the incremental multi-domain setting. The dataset mainly covers five diverse domains: IT, Koran, Law, Medical, and Subtitles, which are available in OPUS (Aulamo and Tiedemann, 2019). Following previous work (Gu and Feng, 2020; Gu et al., 2022), we use the new data splitting released by Aharoni and Goldberg (2020), and perform German to English translation (De→En). Please refer to Tab. 7 of Appendix C for detailed data statistics.

heading: 3.2 Metric
text: For a fair comparison, we follow previous work (Gu et al., 2022) and adopt the 4-gram casesensitive BLEU with the SacreBLEU tool3 (Post, 2018) and report the statistical significance test (Koehn, 2004).

heading: 3.3 Implementation Details
text: Following Gu et al. (2022), we use the mBART50nn (Tang et al., 2020) as our baseline model. Please refer to Appnedix A for detailed settings.

heading: 3.4 Comparison Models
text: Our comparison models consist of two parts: noncontinual learning methods and continual learning methods. Please refer to Appendix B for details.

heading: 3.5 Main Results

heading: 3.5.1 Adding a Second Domain
text: We investigate different methods for adding a new domain to a model already trained on one domain. In detail, we first fine-tune the mBART50-nn model on one domain. Then, we add another domain to the model through the proposed approach without accessing any training labels for the first domain. The results of all models are shown in Tab. 1.
As hypothesized, when adding the Koran domain to a model fine-tuned on the IT domain, in the regularization-based setting (mBART50-nn (L2Reg or EWC)) the models are not able to learn the IT domain by only adjusting the model weights
3BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+ version.1.4.13
with constraint (the BLEU of old domain is about 4 points below the single-domain fine-tuning). Alternatively, the mBART50-nn (TKD) method also cannot prevent the catastrophic forgetting of the previous domain, as demonstrated by the drop of about 2.5 points in terms of the BLEU score. This is happening to various degrees to all the old domains in all the pairs. We note that the same pattern can also be found for the other domain pairs (e.g., [IT→Law]). Compared with them, the mBART50nn (LFR-OM) method, to some extent, can keep the performance of the previous domain because they only update these parameters which does not harm the performance of the previous domain. However, this method first needs some parallel data to search such parameters. Given that the drop we ob-
serve for mBART50-nn (L2-Reg)&mBART50-nn (EWC)&mBART50-nn (TKD) is generally higher than mBART50-nn (LRF-OM), we will not report their results in the following sections.
In sum, computing the distillation loss with our proposed semi-supervised distillation and crossdomain contrastive objective largely mitigates the catastrophic forgetting issue and keeps the capability of the model to learn the new domain. When adding Koran to an IT-trained model, our model even surpasses the MDL or single-domain finetuning methods after the second stage when we use the unlabeled development set (we only use source-side data) of IT domain for distillation (the drop of mBART50-nn (LFR-OM) is about 1.3%). Additionally, the BLEU scores of all models on the Koran when added as a new domain are comparable with each other. 1) This means that the model is able to retain the general linguistic knowledge required to learn the new domain, while also preserving its knowledge of the older domain. Meanwhile, we observe a similar trend in the reverse setting, where we add IT to a model fine-tuned on the Koran. Finally, this pattern is consistent in other domain pairs as well (e.g., adding IT to Medical or Subtitles).

heading: 3.5.2 Adding Third and Fourth Domains
text: We further investigate the effectiveness of SCD by incrementally learning three and four domains, and we report the results with different domain orders in Tab. 2. Results show that our SCD is able to provide useful information to retain the knowledge in the model. For instance, when adding Law to IT and Koran (i.e., It→Koran→Law setting), the BLEU score of IT drops about 5.70% with the mBART50-nn (LFR-OM), while using SCD the drop is only about 1.60% compared to the singledomain fine-tuning model. Notice that this pattern is consistent in almost every domain combination we experimented with.
When adding the fourth domain, we also observe a similar trend to adding the third domain. Besides, we find that 2) the performance of the first domain gets lower with the domain increases, including all methods. This shows that there is much room for further improvement using other more advanced continuing learning methods.

heading: 3.5.3 Incremental Addition of Five Domains
text: In this section, we explore the effectiveness of SCD by incrementally adding five domains. We also list the results of adding the second, third, and fourth
domains for comparison in Tab. 3. The results show a similar pattern that we observed in Tab. 1 and Tab. 2. That is, our SCD still outperforms mBART50-nn (LFR-OM) in this setting. Incrementally adding a new domain gradually contributes to the forgetting of older domains for both mBART50-nn (LFR-OM) and SCD methods, especially for mBART50-nn (LFR-OM). For example, IT performance drops at each stage, resulting, at the last stage, in a total drop of about 9% drop in BLEU. The reason may be that 3) it is difficult for the mBART50-nn (LFR-OM) method to search such regions that can be freely updated for the previous four domains. That is the updatable parameters for several domains may be conflicting or none. Even for our proposed SCD, the drop still is 6% BLEU scores, showing that incrementally learning many domains still remains a challenge and is worth studying in the future.
Besides, we report the results of two additional task ordering in the last two blocks of Tab. 3, i.e., [Koran→Law→Medical→Subtitles→IT] and [Law→Medical→Subtitles→IT→Koran]. We observe that despite changing the order of the domain, the outcome is the same. We also find a similar pattern when we experimented with another domain order different from the mentioned ones. Our proposed model has the ability to limit catastrophic forgetting happening to some extent in the continual learning setting.

heading: 4 Analysis

heading: 4.1 Ablation Study
text: We conduct ablation studies to investigate how well semi-supervised distillation and cross-domain contrastive objective of SCD works. We conclude two findings from the results in Tab. 4.
(1) “w/o semi-supervised distillation”: i.e., without using the unlabeled data of the same distribution of the previous domain and using the data of the current domain, the model performance greatly degrades on the older domain and slightly improves the result of the current domain. It shows the necessity of using the data of the same distribution of the previous domain to prevent catastrophic forgetting. Besides, we also find that there is a performance trade-off between older domains and new domains, where the phenomenon is introduced by the hyperparameter α in Eq. 1. We investigated it in Tab. 9 of the Appendix and actually different hyperparameters have different impacts, which mainly affect the trade-off between older and new domains.
(2) “w/o LCCOAB ”: the model performance becomes worse on both domains. This shows that our cross-domain contrastive learning indeed can enhance the distilled knowledge and guarantee the distinct domain characteristics, which thus benefits the model performance on both domains.

heading: 4.2 Analysis of Adaptation to New Languages
text: To investigate whether our approach can apply to new language pairs, we follow Gu et al. (2022) and conducted such experiments on introducing new language pairs, i.e., Greek (El)↔English (En) and Slovak (sk)↔English (En). The results are shown in Tab. 5.
The results show that our approach can significantly surpass the continual learning method, i.e., mBART50-nn (LFR-OM), demonstrating the effectiveness and generality of our method.

heading: 4.3 Analysis of Model Variants
text: In our work, the additional domain model on Nk+1 is used to provide a hard sample representation for cross-domain contrastive learning. In this setting, we have tried additional three settings: 1) finetuning on the Nk+1 domain with the previously learned domain model (denoted as FT); 2) utilizing vanilla knowledge distillation (VKD), i.e., using the arbitrary unlabeled data; 3) using vanilla contrastive learning (VCL; i.e., only using the sample in the batch as the negative).
The results in 6 show that directly fine-tuning on the target domain without considering previous domains (FT), using vanilla knowledge distillation (VKD) or vanilla contrastive learning (VCL) cannot fully exert their advantage for domain translation. In comparison, cross-domain contrastive distillation has a positive impact on the model performance.

heading: 4.4 Effect of Using a Little Unlabeled Data
text: To further find out how much unlabeled data can achieve a good performance, we randomly sample
10, 50, 100, 500, 1000, and 2000 unlabeled examples from the validation set and use the remaining validation data to choose model checkpoints for evaluating on the test set. In Fig. 3, we observe that the model performance gradually improves and reaches stability as the used unlabeled data increases. Interestingly, we find that the model rapidly achieves a higher result only with a handful of unlabeled data, i.e., 50 and 100 instances for Koran and IT, respectively. It even surpasses the mBART50-nn (LFR-OM) which uses all labeled data in the Koran→IT setting. This shows the superiority of using unlabeled data of the same distribution of the older domain, which can largely help the model retain the learned knowledge of the older domain and prevent catastrophic forgetting. It again indicates the effectiveness of our approach. We also provide a case study to intuitively show how it works in Appendix F.

heading: 5 Related Work
text: Continual Learning of Translation. Recent studies on continual learning of machine translation mainly includes data memory-based method, taskspecific adapters, and regularization-based method. Specifically, (1) the data memory-based methods (Chu et al., 2017; Bapna and Firat, 2019a; Xu et al., 2020; Liu et al., 2021) usually require maintaining part or all of the training data of the previous domains/task, which is not flexible in practice and maybe not realistic in the real world due to data privacy. For example, Liu et al. (2021) produce many mixed-language sentences via a bilingual dictionary. (2) The task-specific adapter methods (Bapna and Firat, 2019a; Zeng et al., 2018, 2019; Gu et al., 2019; Cao et al., 2021; Wang et al., 2024; Gu et al., 2021; Liang et al., 2021a) typically require assigning additional model parameters to different domains/tasks, which requires the model to know which task the input comes from. (3) The regularization-based methods (Khayrallah et al., 2018; Thompson et al., 2019; Dakwale and Monz, 2017) reduce forgetting by introducing an additional penalty term in the learning objective, which may suffer from under- or over-constraint issues. For example, Gu et al. (2022) firstly utilize the previous parallel data to search the low forgetting risk regions and then only update these parameters within the region to largely maintain the performance of the previous domain. Unlike the above work, our method is flexible and free
to the requirement of parallel data of the previous domains compared with (1) and (3). Besides, our model does not explicitly lead to model separation against (2). Knowledge Distillation. KD (Hinton et al., 2015) is to transfer the knowledge (e.g., soft targets outputs) of the stronger model (aka. the teacher model) to the small model (aka. the student model), which has achieved impressive results in the literature (Kim and Rush, 2016; Wu et al., 2020; Wang et al., 2021; Lee et al., 2019). In neural machine translation, the KD-related work mainly focuses on how to effectively distill the knowledge of the teacher to the student. For example, Zhang et al. (2023) investigate where the knowledge comes from and then carefully design a method to contrapuntally distill the target knowledge. In this work, we aim to utilize the unlabeled development data of the previous domain to prevent catastrophic forgetting of the previous tasks via KD. Contrastive Learning. The idea of contrastive learning aims to learn effective representation by pulling semantically close neighbors together and pushing apart non-neighbors (Hadsell et al., 2006), which has verified its superiority in many fields, such as model compression (Sun et al., 2020), sentence embedding (Gao et al., 2021), summary (Liu and Liu, 2021; Wang et al., 2023; Liang et al., 2023a), pre-training (Zhou et al., 2023), and translation (Pan et al., 2021; Liang et al., 2021b; Lee et al.; Liang et al., 2022; Cheng et al., 2022). For example, in neural machine translation, Cheng et al. (2022) propose a contrastive translation memory to enhance the model performance and Pan et al. (2021) utilize the contrastive learning to improve the multilingual neural machine translation. Differently, we introduce a cross-domain contrastive objective to enhance the distilled knowledge, which further guarantees the distinct domain characteristics and thus improves the model performance for several domains. To our knowledge, we are the first that introduce it to prevent catastrophic forgetting.

heading: 6 Conclusion
text: In this paper, we propose a new continual learning framework for incremental neural machine translation without accessing any previous training data or re-training on all domains from scratch. To maintain the performance of the previous domain, we propose to utilize small-scale source-side development data of the previous domain via knowledge
distillation. To further ensure distinct domain characteristics in a model, we devise a cross-domain contrastive objective to enhance the distilled knowledge. Extensive experiments on a more general scenario show that our method can achieve significant improvements over several strong baselines. In the future, we would like to investigate our method for other tasks, e.g., cross-lingual summarization (Wang et al., 2022).
Limitations
While we show that the SCD achieves significant performance in continual learning of domain adaption translation, there are some limitations worth considering to study in future work: (1) In this study, we only conduct experiments on sequentially five domains, and future work could extend our method to more domains, languages and modalities; (2) This work does not conduct experiments on more real-world applications, e.g., sequentially adding different translation tasks (first sentencelevel machine translation and then document-level machine translation and more) or multilingual translation task.

heading: Acknowledgements
text: The research work described in this paper has been supported by the National Nature Science Foundation of China (No. 62376019, 61976015, 61976016, 61876198, and 61370130). The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve this paper.

heading: B Comparison Models
text: Our comparison models consist of two parts: noncontinual learning methods and continual learning methods.

heading: 1) non-continual learning methods :
text: • Scratch: We train a vanilla transformer (Vaswani et al., 2017) from scratch only with the training data from the new domain task. • mBART50-nn (Tang et al., 2020) is a large scale pre-trained NMT model. All the following systems are implemented based on this model. • mBART50-nn (FT) (Luong and Manning, 2015): We fine-tune the mBART50-nn model only on individual domain training data. • mBART50-nn (MDL) fine-tune the mBART50nn model with all domain training data, which is considered the upper bound in the field of continual learning. We use the temperature-based sampling function to oversample the validation datasets (Arivazhagan et al., 2019).
2) Continual learning methods , which aim to get a good balance between previous and new domains. • mBART50-nn (TKD) (Dakwale and Monz, 2017): Besides minimizing the training loss of the new domain, this method also minimizes the distillation loss for the previous domain, which is
computed on the new domain’s training data, i.e., without using any previous data. The training objective based on the mBART50-nn model is:
LTKDAB = LB + αCE(ΘA(Xb),ΘAB(Xb)). (5) • mBART50-nn (L2-Reg) (Barone et al., 2017) adds an L2-norm regularization on the mBART50nn model to alleviate the catastrophic forgetting when adding a new domain. • mBART50-nn (EWC) (Thompson et al., 2019) first models the importance of the parameters of the mBART50-nn model with Fisher information matrix (Ly et al., 2017) and then puts more constraints on the important parameters to let them stay close to the original values. • mBART50-nn (Adapter) (Bapna and Firat, 2019b) inject the domain-specific adapter layers into the mBART50-nn model and only update the adapters for different domains. • mBART50-nn (LFR-OM) (Gu et al., 2022) aims to update the parameters within the low forgetting risk regions with the output-based method, which requires the parallel data of the previous domain to search the low forgetting risk regions first.

heading: C Additional Results of Adding a Second Domain
text: In Tab. 8, we find the same trend as observed in Tab. 1. Besides, we also find that our model always achieves the best results on the older domains while sometimes performing slightly worse on the newly added domain compared with some baselines, e.g., mBART50-nn (TKD). The reason may be that our proposed method (knowledge distillation on the unlabeled data with the same distribution as previous domains and contrastive learning) aims to prevent catastrophic forgetting and does not obtain a better trade-off between previous and new tasks to some extent. Through tuning different hyper-parameters, α and β in the training loss, we observe a further improvement on previous domains without sacrificing the performance on new domains (see Tab. 9). Actually, with more domains added, the advantages of our approach are more evident (Tab. 2 and Tab. 3). Anyway, our method can always achieve the best average results, showing its effectiveness.

heading: D Effect of Hyperparameters α and β
text: We have investigated the impact of hyperparameters, i.e., α and β. Indeed, different hyperparam-
eters have different impacts, which mainly affect the trade-off between older and new domains. For example, in IT→Koran direction, the results are shown in Tab. 9. In our experiments, we choose α = 0.5 and β = 0.5 to achieve a better trade-off performance between older and new domains.

heading: E Training Efficiency
text: All our experiments are conducted on 8 V100 GPUs. The average running time is listed as follows (corresponding to different models in the Koran→IT setting of Table 1 with 10 epochs).
The results show that our method consumes slightly more time to train our model while achieving a significantly better performance. The inference time of all models costs the nearly same time due to the same model architecture.

heading: F Case Study
text: We listed an example here and will add more case studies in the new version. In the IT→Koran setting, we first trained a model on the IT domain denoted as model-1. Then, we fine-tune model-1 on the Koran domain denoted as Model-2. Model3 and Model-4 indicate mBART50-nn (LFR-OM) and our proposed method, respectively. The instance below is from the test set of the IT domain.
We can observe that model-1 can translate the domain word “Speicher” well after training on the IT domain. Unfortunately, after further finetuning on the Koran domain, the model forgets the previously learned domain knowledge and incorrectly translates “Speicher” to “storage”. Besides, Model-3, which aims to update the parameters within the low forgetting risk regions with
the output-based method to prevent forgetting, still cannot address this case. However, our model can accurately translate it, which demonstrates that our model indeed can prevent from forgetting of previously learned domain knowledge and alleviate the forgetting problem compared to other methods.

heading: Future Work
text: In the future, we would like to investigate our method for other tasks, e. g. , cross-lingual summarization (Wang et al. , 2022). Limitations
While we show that the SCD achieves significant performance in continual learning of domain adaption translation, there are some limitations worth considering to study in future work: (1) In this study, we only conduct experiments on sequentially five domains, and future work could extend our method to more domains, languages and modalities; (2) This work does not conduct experiments on more real-world applications, e. g. , sequentially adding different translation tasks (first sentencelevel machine translation and then document-level machine translation and more) or multilingual translation task