Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation
Year: 2023

Abstract:
In the real-world scenario, a longstanding goal of multilingual neural machine translation (MNMT) is that a single model can incrementally adapt to new language pairs without accessing previous training data. In this scenario, previous studies concentrate on overcoming catastrophic forgetting while lacking encouragement to learn new knowledge from incremental language pairs, especially when the incremental language is not related to the set of original languages. To better acquire new knowledge, we propose a knowledge transfer method that can efficiently adapt original MNMT models to diverse incremental language pairs. The method flexibly introduces the knowledge from an external model into original models, which encourages the models to learn new language pairs, completing the procedure of knowledge transfer. Moreover, all original parameters are frozen to ensure that translation qualities on original language pairs are not degraded. Experimental results show that our method can learn new knowledge from diverse language pairs incrementally meanwhile maintaining performance on original language pairs, outperforming various strong baselines in incremental learning for MNMT.1

heading: 1 Introduction
text: Multilingual neural machine translation (MNMT) aims at handling multiple translation directions in a single model and achieves great success in recent years (Wenzek et al., 2021; Goyal et al., 2022; Cheng et al., 2022). However, the powerful MNMT models need to be retrained from scratch
∗Corresponding authors: Peng Li (lipeng@air.tsinghua. edu.cn) and Yang Liu (liuyang2011@tsinghua.edu.cn)
1Our code will be released at https://github.com/ THUNLP-MT/ktnmt

heading: Method en→uk en→bn
text: using a mixture of original and incremental training data when new language pairs arrive (Tang et al., 2020). Considering that the original training data of MNMT models is often large-scale (Fan et al., 2021; Costa-jussà et al., 2022), and thus the method that utilizes original data to train incrementally is time-consuming and cumbersome (Ebrahimi and Kann, 2021). Therefore, a practical scenario is that these models can continually support new language pairs while preserving the previously learned knowledge without accessing previous training data, which belongs to incremental learning, and has drawn much attention recently (Dabre et al., 2020; Gu et al., 2022; Zhao et al., 2022).
In this scenario, existing studies attempt to overcome the issue of catastrophic forgetting (French, 1993) on original language pairs, such as replaybased methods (Garcia et al., 2021; Liu et al., 2021) and regularization-based methods (Huang et al., 2022; Zhao et al., 2022). However, the methods primarily focus on balancing performance between old and new translation directions and use only incremental data to acquire new knowledge, restricting the development of new language pairs (Escolano et al., 2021; Ke and Liu, 2022). As shown in Table 1, prior incremental learning methods cannot
15286
achieve comparable performance on new translation directions, compared with training a bilingual translation model from scratch. Therefore, is it possible to leverage external knowledge without increasing the amount of incremental training data to facilitate the learning procedure of new language adaptation?
Fortunately, the development of Transformerbased language models unveils that Feed-Forward Networks (FFN) might be a core component that stores the knowledge (Geva et al., 2021; Dai et al., 2022; Geva et al., 2022; Vázquez et al., 2022). In these efforts, external knowledge is injected into FFN layers to enhance the performance of pre-trained language models. More importantly, it opens the door to leveraging the knowledge from neural models in adapting MNMT models to incremental language pairs.
In this work, considering the knowledge in neural networks, we propose a knowledge transfer (KT) method that can efficiently adapt MNMT models to diverse incremental language pairs. First, we convert incremental training data into continuous representation by additional parameters, forming pluggable modules. Then the pluggable modules can be flexibly introduced in the embedding layer and FFN layers of original MNMT models, respectively. The two stages are regarded as a process of knowledge transfer and equip original models with knowledge of unseen languages before adaptation, alleviating the representation gap between original models and introduced parameters. And the knowledge transfer method can further provide better optimization than training from scratch for the introduced parameters. Moreover, except for the pluggable modules, all the parameters of the original model are frozen. Therefore, our architecture can also retain previously learned knowledge from the original translation model to completely maintain the translation qualities on original language pairs. To sum up, our contributions are as follows:
• We propose a knowledge transfer method with pluggable modules to acquire more knowledge of new languages, which achieves competitive translation qualities on incremental language pairs.
• Our architecture can efficiently adapt to diverse language pairs in incremental learning and naturally retain the performance on origi-
nal language pairs when the original training data is not available.
• Experiments show that our method can learn knowledge from the other large-scale translation models for adapting original models with different sizes to new language pairs.

heading: 2 Related Work
text: Replay-Based Methods. The first branch of works utilizes previous training data or create pseudo data that is essentially a replay on old tasks (de Masson D’Autume et al., 2019; Liu et al., 2021; Kanwatchara et al., 2021). Specifically, previous data sometimes cannot be accessed due to data protection and security (Feyisetan et al., 2020; Qu et al., 2021). In this scenario, Sun et al. (2019) replay pseudo samples of previous tasks, which can avoid forgetting previously learned knowledge. However, the pseudo data with noise significantly hurts the performance for both old and new tasks, and the data generation procedure requires additional time costs, restricting the efficiency of incremental learning (Peng et al., 2020; Garcia et al., 2021). In contrast to these methods, our approach does not require extra data and is more flexible in the real-world scenario.
Regularization-Based Methods. The second branch of works introduces additional penalty terms to the learning objective on the parameters, alleviating the issue of catastrophic forgetting (Kirkpatrick et al., 2017; Thompson et al., 2019; Castellucci et al., 2021; Gu et al., 2022). In particular, Shao and Feng (2022) propose a complementary online knowledge distillation method that utilizes previous models (teachers) to supplement current model (student) training. In contrast to these methods, our architecture can naturally avoid forgetting previously learned knowledge and retain the performance of old tasks. It allows our method to focus solely on learning new knowledge better instead of preserving old knowledge.
Parameter-Isolation Based Methods. The third branch of works introduces extra parameters to support new tasks and freeze all original parameters to completely retain the performance on previous tasks (Bapna and Firat, 2019; Madotto et al., 2021; Zhu et al., 2021). However, the methods only utilize incremental training data to optimize the additional parameters which are randomly initialized (Escolano et al., 2021; He et al., 2021),
hindering old models from learning knowledge of incremental languages (Dabre et al., 2020; Ke and Liu, 2022). Chalkidis et al. (2021) combine pseudo data with the prompt-tuning method to alleviate this issue for multilingual tasks. Our method attempts to exploit the potentiality of incremental training data to acquire new knowledge via knowledge transfer while not leveraging extra data, compared with existing methods.

heading: 3 Method
text: In this work, we aim to completely maintain the performance of previous translation tasks without original training data. As shown in Figure 1, we introduce additional components for new language pairs and adopt a strategy that does not disturb the parameters of the original model. We hope to minimize the impact on the original model during the incremental learning process. As a result, we exclusively concentrate on how to handle the situation of learning new language pairs. Furthermore, the additional components are transferred from parameters in another pre-trained translation model, in a similar way to a pluggable module, rather than randomly initialized, as shown in Figure 2. It can also reduce the cost of learning new language pairs during the training stage, enhancing the practicability and efficiency of incremental learning methods in the real-world scenario.

heading: 3.1 Task Definition
text: An ideal requirement is that original MNMT models can be continually updated to support new language pairs while retaining translation qualities on original language pairs without accessing previous training data, as shown in Figure 1.
Formally, an MNMT model is trained on initially selecting a set of available parallel data D = {D1, ...,Di, ...,DN} which covers N languages, and Di represents the original parallel training corpus on the i-th language pair. The training objective of the initial MNMT model is to maximize the log-likelihood L:
LD(θ) = ∑
Di∈D
∑
(x,y)∈Di log p(y|x; θ) (1)
where θ represents the trainable parameters of MNMT models. The source sentence is denoted as x, while the target sentence is denoted as y. In order to specify the source and target languages, two language tokens are added at the beginning of each source and target sentence, respectively.
Incremental learning is updating the original MNMT model on an updated set of parallel data D(U) = {D1, ...,DN , ...,DM} which covers M languages, and N > M . A dilemma, though, is that the original training data D is often unavailable due to data security. Thus, we can only utilize the new data D′ = {DN+1, ...,Dj , ...,DM} to incrementally train the original MNMT model, and Dj represents the incremental parallel training corpus on the j-th language pair. The optimization objective in incremental learning is given by:
LD′(θ) = ∑
Dj∈D′
∑
(x,y)∈Dj log p(y|x; θ) (2)
As a result, the number of language pairs that the MNMT model support increases from N to M .

heading: 3.2 Knowledge Transfer via Pluggable Modules
text: Based on the original MNMT model, we open up additional spaces for adapting to new language pairs, as shown in Figure 2. However, the additional spaces with initialized parameters are weak in their abilities to capture the shared linguistic features among different languages. Because the linguistic distribution of the original model is fitted previously, which leads to a representation gap between the original model and introduced spaces. Thus, we not only leverage new training data but also exploit the potentiality of these data by introducing two types of pluggable modules to bridge the representation gap.
Vocabulary Adaptation. On the one hand, if the new language has a distinct script with the
set of original languages, a certain proportion of the out-of-vocabulary (OOV) tokens with unclear semantics will occur due to different character sets between the original and incremental languages, which hinders performance on new language pairs (Zhang et al., 2022). However, the external model is not troubled by this situation and covers sufficient tokens for the incremental language pairs. Therefore, we expand an extra space in the embedding layer and concatenate the embeddings of non-overlap tokens between the original model and the external model, bridging the representation gap through vocabulary adaptation.
Feed-Forward Adaptation. On the other hand, FFN layers can be seen as key-value memories, which has previously been investigated by (Sukhbaatar et al., 2019). Each FFN layer consists of two linear networks with a non-linearity activation function and is given by:
FFN(H) = f(H ·K⊤) · V (3)
where K,V ∈ Rdffn×dmodel are parameter matrices, dffn is the dimension of the FFN, H is the input hidden state of FFN layers and the dimension is dmodel, f represents the non-linearity activation function, e.g., GELU and ReLU.
The first linear network is regarded as the keys
and activates a set of intermediate neurons. The second linear network integrates the corresponding value matrices by weighted sum, using the activated neurons as weights. The FFN layers store knowledge in the key-value memories manner. Thus, we leverage the continuous representation in the FFN layers of the external model that stores useful language knowledge and transfer the knowledge into the FFN layers of the original model, forming a type of pluggable module. We combine the output of the original FFN layers and the injected pluggable modules to share linguistic knowledge, alleviating the representation gap through FeedForward layers adaptation. The fusion FFN output H(f) is given by:
H(f) = FFNoriginal(H) + FFNexternal(H) (4)

heading: 3.3 Training and Inference
text: During the training stage, previously learned knowledge can be naturally preserved with a frozen training strategy, which can avoid the issue of catastrophic forgetting. The training procedure of our method is divided into two stages, as shown in Figure 2.
Stage 1: External Model Training. To convert incremental training data into continuous representation by additional parameters. We first leverage
the incremental training data to train an external Transformer-based neural network.
LD′(θ̂) = ∑
Dj∈D′
∑
(x,y)∈Dj log p(y|x; θ̂) (5)
where θ̂ represents the trainable parameters of the external neural models. We only retain the parameters in the embedding layer (θ̂e) and FFN layers (θ̂f ) of the external model as the pluggable modules for the next training stage.
Stage 2: Pluggable Module Tuning. Directly transferring the additional parameters limits the MNMT model capacity, especially for the language pairs with sufficient data. Therefore, we further train the pluggable modules in the second stage:
LD′(θ̂e, θ̂f ) = ∑
Dj∈D′
∑
(x,y)∈Dj log p(y|x; θ̂e, θ̂f )
(6) where θ̂e and θ̂f represent the trainable parameters of pluggable modules in the embedding layer and FFN layers, respectively.
Inference. For the inference stage, the original translation directions follow the original model without any pluggable modules while the incremental translation directions require the concatenation of the original model and pluggable modules.

heading: 4 Experiments

heading: 4.1 Datasets
text: To ensure the reliability of the experiments, the original MNMT model is implemented on a multilingual machine translation dataset2 (WMT-7) that covers seven languages (Farhad et al., 2021). And we provide four incremental languages considered for incremental adaptation3. An extensive description and comprehensive information regarding the datasets for all languages can be found in Appendix A. All training data are sourced from the WMT (Workshop on Machine Translation) and FLoRes datasets, ensuring reliable quality.
Language Choice. As contrasted to previous studies for incrementally adapting translation models to new languages, we further provide a comprehensive language setting. Previous works often investigate the situation of the related languages
2https://www.statmt.org/ 3https://data.statmt.org/cc-matrix/
which are similar language families and scripts to the original languages. In our setting, the incremental languages have distinct scripts and belong to several language families, which leads to a serious language representation gap. Please refer to Appendix A.2 for more details of language consideration in our setting.

heading: 4.2 Implementation Details
text: Baselines. We implement a vanilla Transformer for original languages as the initial model which is trained on multiple parallel data jointly (Johnson et al., 2017). And we compare the proposed method with different architectures for adapting the original model to new language pairs. All methods utilize the preprocessing script of a shared BPE model with 32k tokens based on the Sentencepiece library4. The baselines can be listed as follows:
From-scratch (Johnson et al., 2017): A vanilla Transformer is trained from scratch on the incremental languages with the multilingual training strategy. Note that the models do not support the original translation directions.
Adapter (Bapna and Firat, 2019): We follow previous adapter architectures and introduce extra parameters in each FFN layer of the original MNMT model. All original parameters are frozen and only the adapters are trainable.
Extension (Lakew et al., 2018): On the basis of the adapter architecture, we extend the original vocabulary (VP ) for new languages adaptation. Initially, a supplementary vocabulary (VQ) is created using the standard Byte-Pair Encoding (BPE) procedure from the incremental training data. Subsequently, VP and VQ are combined to form a unified vocabulary V , which is defined as V = VP ∪ VQ. The embeddings of the original models are expanded to match the size of the complete vocabulary (V), and the additional embeddings are initialized using a Gaussian distribution.
Serial/Parallel (Zhu et al., 2021): We follow Zhu et al. (2021) to introduce adapters in the serial or parallel connection manner. Our pluggable modules in the FFN layers can also be converted into a serial manner.
Training Setup. We implement all models based on the open-source toolkit fairseq5 (Ott et al., 2019). For a fair comparison, we employ the same configuration of Transformer-Big (Vaswani et al., 2017)
4https://github.com/google/sentencepiece 5https://github.com/pytorch/fairseq
in our experiments. All original parameters are frozen during the incremental learning procedure. More model training details are provided in Appendix B.1.
Evaluation. We report the detokenized casesensitive BLEU of models by the SacreBLEU evaluation script (Post, 2018)6. We show the training time of each method in terms of kiloseconds and use the beam search decoding algorithm with a beam size of 5 and a length penalty of 1.0.

heading: 4.3 Main Results

heading: Adding A Single Language
text: As shown in Table 2, we investigate the translation qualities when a new language pair arrive. The results demonstrate that our proposed method (KT) outperforms several baselines in terms of average BLEU scores for all incremental translation directions. Specifically, KT achieves an average BLEU score of 27.14 for xx→en and 21.32 for en→xx translations. In particular, based on KT, the pluggable modules that are injected in a parallel manner further improve the performance over the serial manner on all incremental language pairs.
6Signature: nrefs:1 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.2.0.
The Adapter methods are more vulnerable to adapting original models to some incremental language pairs, e.g., 0.12 BLEU scores on en→bn and 16.94 BLEU scores on bn→en. Because the methods with an unaltered vocabulary result in the sentence being broken up into semantically meaningless OOV tokens. Although the Extension methods can alleviate the issue of OOV tokens and fragmentary semantics by rebuilding embedding layers, the
extended parameters are still hard to optimize. The knowledge transfer method can further guide additional parameters to achieve greater improvements on these language pairs.
Considering the different introduced manners of pluggable modules, based on the baselines, parallel modules tend to be weaker than serials. It demonstrates that parallel architecture is more difficult to learn new knowledge from limited training data. And the results show that the knowledge transfer method mitigates this issue and explores the potential of parallel architectures, achieving obvious improvement on all eight translation directions, even outperforming the model training from scratch.

heading: Adding Multiple Languages Simultaneously
text: As shown in Table 3, we examine the translation qualities in incremental learning when eight new language pairs arrive simultaneously. The results show that our proposed method can also achieve better performance compared with the baselines. Notably, in the low resource scenario (ro and uk), our method of adding multiple languages obtains better performance compared with adding a single language.
Besides, adding multiple languages simultaneously in incremental training makes more training samples available and it facilitates the optimization of challenging pluggable modules in a parallel manner. In this setting, the parallel pluggable modules of all methods demonstrate better performance than the serial. Moreover, the situation of incremental language pairs that are difficult to learn is still alive with the Adapter. It even shows more severe degeneration on the other incremental languages (17.24 BLEU on en→de of adding a sin-
gle language while 6.56 BLEU of adding multiple languages simultaneously). However, our method does not significantly been disturbed by different conditions in incremental learning, which exhibits good stability, as shown in Table 2 and Table 3.

heading: Degeneration in Incremental Learning.
text: As shown in Table 4, to demonstrate the reliability and effectiveness, we investigate the degeneration on the original translation directions, compared with various outstanding continual learning methods. The results demonstrate that our method achieves competitive performance on the incremental translation directions and even outperforms the fine-tuning strategy (up to +0.57/+0.40 for ro→en and en→ro respectively). Please refer to Appendix 4.6 and B.2 for more details of the original models and all baselines.
Besides, the results also show that prior replaybased and regularization-based methods still suffer from pronounced degeneration on the original translation directions without the original data. Although no degradation has occurred using Prompt and Prefix, they are vulnerable to learning new knowledge from updated training samples incrementally. More importantly, considering the reliability of the comparison, we have only selected the translation directions between Romanian and English. Because previous methods cannot obtain comparable results when the incremental languages are not related to the set of original languages.

heading: 4.4 Results on Pre-trained Models
text: As shown in Table 5, we leverage pre-trained M2M100 models (Fan et al., 2021) as the external model and investigate the effectiveness of different knowl-

heading: Method ro→en en→ro bn→en en→bn
text: edge transfer methods. Knowledge distillation (KD) (Hinton et al., 2015) is a widely used technique to transfer knowledge between models. The results show that only utilizing KD cannot achieve comparable performance for incremental language adaptation. However, KD can be arbitrarily integrated into our method (KT) and further facilitate the procedure of knowledge transfer. The combination of KD and KT achieves better translation qualities than only using one alone based on all model settings.
Besides, both KD and KT are better at learning knowledge from the large pre-trained models. It proves that the large pre-trained model contains more useful knowledge. And we find that the size between models also determines the performance on incremental translation directions. The small M2M-100 model (0.4B) is beneficial for the same size original model (0.4B) but is insufficient to support the large original model (1.2B). In contrast, the large M2M-100 model (1.2B) plays a positive role in both small and large original models by knowledge transfer. However, the small original model (0.4B) limits learning sufficient knowledge from the large M2M-100 model according to the comparison between No.6 and No.12, as shown in Table 5.

heading: 4.5 Ablation Studies

heading: Effects on Transfer Areas
text: As shown in Table 6, we further investigate the effectiveness of our method in different transfer
areas. The results demonstrate that our method can help each pluggable module to be better optimized separately and achieves better performance when both two pluggable modules are injected through knowledge transfer for all incremental languages. Specifically, the method improves translation qualities related to Romanian and Ukrainian when it affects the pluggable module in the embedding layer. On the contrary, it is more effective to transfer the knowledge for the pluggable modules in the FFN layers on translation directions related to German and Bengali, according to the comparison between 2 and 3. A possible reason is that the resource of different language pairs influences the efficiency of knowledge transfer.

heading: Effects on Pluggable Modules
text: Previous parameter-isolation based methods propose various components to introduce additional parameters in the hidden layers (He et al., 2021). As shown in Table 7, inspired by them, we modify the usage of the pluggable modules in the hidden layers and our method is stable on the four translation directions. In particular, we also inject the pluggable modules in the Self-Attention layer. However, the special modification of pluggable modules does not demonstrate effective performance in incremental learning for MNMT.

heading: 4.6 Results on Original Language Pairs
text: To demonstrate the validity and reliability of our method, we build two powerful MNMT models as the original models. As shown in Table 8, the original models achieve state-of-the-art performance on all original translation directions, compared with the other powerful MNMT models.

heading: 4.7 More Comparisons
text: Due to space limitation, we provide a more detailed analysis of our method in Appendix C, including the training cost of the incremental learning, the
visualization of sentence representations on all language pairs, and the case study on new language pairs, demonstrating the effectiveness of the knowledge transfer method in incremental learning for new language adaptation.

heading: 5 Conclusion
text: In this work, we propose a knowledge transfer method in incremental learning for MNMT, which leverages the knowledge from neural models. It can encourage original models to learn new knowledge from updated training data while naturally mitigating the issue of degradation on previous translation directions. Moreover, it is more efficient to utilize the knowledge transfer scheme than introducing randomly initialized parameters in incremental learning. Experimental results demonstrate that the proposed method outperforms several strong baselines in the comprehensive language consideration.

heading: Limitations
text: In this work, we attempt to extend an existing MNMT model to support new language pairs with an acceptable expense. In addition to the advantages, our method has the following limitations:
(1) Additional introduced parameters. We utilize the parameter-isolation based method to support new language pairs. The total parameters of the MNMT model have been increased by pluggable modules to achieve better performance than prior studies. In the future, we will compress the number of parameters to the same size of original models meanwhile preserve the performance on all translation directions.
(2) The gap between our scenario and the realworld scenario. Our proposed method is a whitebox service in incremental learning. Thus, we train
a powerful MNMT model as the original model instead of directly utilizing existing models from the Internet. And we only consider eight incremental language pairs due to the limitation of computation resources. We try our best to simulate the realworld scenario and we will apply our proposed method for large-scale pre-trained MNMT models (e.g., NLLB 54.5B and M2M 12B) to validate the effectiveness in industrial scenarios.

heading: Acknowledgements
text: This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Natural Science Foundation of China (No. 61925601, 62276152, 62236011). We sincerely thank the reviewers for their insightful comments and suggestions to improve the quality of the paper.

heading: A Dataset Details
text: We utilize six language pairs to train the original MNMT model that covers 12 translation directions and 7 languages (WMT-7). All the original training data comes from the recent WMT general translation track. And we conduct eight incremental language pairs in incremental learning from the WMT news translation track and FLoRes. All data follow the license that can be freely used for research purposes (Farhad et al., 2021). The license of FLoRes dataset is CC-BY-SA 4.0. In addition, we follow Fan et al. (2021) to clean the training sample. We introduce the characteristics of different languages to analyze the linguistic diversity, as shown in Table 9. All language pairs are Englishcentric and the statistics of training data are shown in Table 10.

heading: A.1 Data Statistics
text: As the general setting, all language pairs are divided into three categories in terms of the amount of parallel data, including high resource (>10M), medium resource (1M~10M), and low resource (100k~1M). Specifically, the original language pairs are, High resource: Japanese and Polish; Medium resource: Icelandic and Pashto; Low resource: Hausa and Tamil. And the incremental language pairs are, Medium resource: German and Bengali; Low resource: Ukrainian and Romanian. Note that incremental training data is often a nonhigh resource in the real-world scenario.

heading: A.2 Language Consideration
text: In this work, we explore a more complex and comprehensive scenario for MNMT in incremental learning, taking into account the diversity of incremental languages. These incremental languages differ from the original languages in terms of their scripts and belong to different language families, which leads to a serious vocabulary and linguistic gap. Inspired by Zhang et al. (2022), if the incremental language has a distinct script with the set of original languages, a certain proportion of OOV tokens with unclear semantics will occur between the original and incremental languages and hinder the performance on new language pairs. Moreover, it is important to note that a language family refers to a group of languages that share a common ancestry, known as the proto-language7. This concept
7https://en.wikipedia.org/wiki/Languagefamily

heading: Code Language Genus Script Order
text: highlights the historical connections among languages and their evolution over time. Additionally, differences in grammar and word order can be observed across distinct language families8. These linguistic variations further contribute to the existing gap between incremental languages, making their translation more challenging.
In our setting, the 4 incremental languages include: Bengali, which is not related to any of the original 7 languages, and has a distinct script; Ukrainian, which is related to the original language Polish with the language family Slavic, but has a distinct script with Cyrillic; Romanian, is Romance language that is not related to all the original languages, but has a share script with Latin characters; German, which is similar to the original languages in the language families and scripts. The statistics and details of datasets for original and incremental languages are shown in Table 9.

heading: B Model Details

heading: B.1 Training Setup
text: We implement Transformer translation models in all our experiments. In particular, the small original model (0.4B) consists of 6 stacked encoder layers, 6 stacked decoder layers, and 16 multiattention heads, followed by the configuration of Transformer-Big (Vaswani et al., 2017). The dimensions of dmodel and dffn are 1024 and 4096 respectively. The large original model (1.2B) consists of 24 stacked encoder layers, 24 stacked decoder layers, and 16 multi-attention heads, followed by the configuration of M2M-100 (Fan et al., 2021). The
8https://wals.info
dimensions of dmodel and dffn are 1024 and 8192 respectively. We use Adam (Kingma and Ba, 2014) and a half-precision training scheme to optimize the parameters of all MNMT models. In addition, we reset the optimizer and learning scheduler in incremental learning and use the temperature-based sampling scheme (Arivazhagan et al., 2019) with a temperature of T = 5 to balance the training data between diverse language pairs. We adopt the early stop (patience is 10) strategy in incremental learning and the batch size is 4096×4 in all training procedures. To eliminate the randomness of the result, we report the mean BLEU scores of the models that are trained in five seeds. All incremental models are trained on 2 NVIDIA A100 GPUs.

heading: B.2 Continual Learning Baselines
text: We compare our method with various representative baselines in continual learning. The baselines are as follows:
• Replay (Sun et al., 2019): creating pseudo data for the original language pairs and training new language pairs jointly with the pseudo data and incremental training data.
• EWC (Kirkpatrick et al., 2017): computing the importance of the parameters with Fisher matrix and employing an additional penalty into the loss function to preserve original knowledge.
• Self-KD (Castellucci et al., 2021): utilizing the original models as the teacher model to distill old knowledge.
• LFR (Gu et al., 2022): constraining the parameters of original models with low forget-
ting risk regions. We choose the LRF-CM for adapting new language pairs.
• Prompt (Chalkidis et al., 2021): prepending prompts to the input embedding in the first layer.
• Prefix (Li and Liang, 2021): prepending prefixes to the keys and values of the attention at every layer.

heading: C More Comparisons

heading: C.1 Training Cost
text: To further illustrate the efficiency of our method, we investigate the training time compared with the stronger baselines, as shown in Figure 3. The results show that the knowledge transfer method can reduce the training time of incremental learning, which is more efficient and practical than the other methods.
C.2 Visualization of Multilingual Representations
As shown in Figure 4, we visualize the sentence representations on xx-to-English translation directions to investigate the representation gap between languages. Due to comparability in one representation space, we need multi-source sentences that represent the same meaning in different languages.
We use “FLoRes” and reduce the 1024-dim representations to 2-dim with t-SNE (Van der Maaten and Hinton, 2008) for visualization.
As Figure 4 shows, the sentence representations using our method are drawn closer than the standard Adapter method (one of the baselines). It demonstrates that our method can well adapt to the new language. Moreover, previous studies have shown that if sentences with similar semantics are closer together in the representation space, it can usually improve the translation performance of zero-shot translation. Experimental results in
two translation directions show that our method can achieve better performance for zero-shot translation, which is consistent with our visualization.

heading: C.3 Case Study
text: We present several translation examples to provide a comprehensive understanding of the knowledge transfer method, as shown in Table 12. The examples demonstrate that our method can effectively adapt original models to new languages especially when the incremental language is not related to the set of original languages. In particular, due to
the vocabulary gap, the Adapter method is vulnerable to learning incremental languages that have a distinct script with Latin. Although the Extension alleviates this issue by expanding the embedding layer, the additional parameters are not fully optimized to suffer from the off-target problem for MNMT.

heading: D Potential Risks of Our Method
text: Since our proposed method can increase the unlimited number of translation directions, it is possible for some malicious users to use the MNMT model to provide translation services for politically sensitive languages. For instance, a malicious user may utilize our model to generate hateful or offensive sentences in some politically sensitive languages.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We provide the limitations of our work in the section ’Limitation’.
3 A2. Did you discuss any potential risks of your work? We provide the potential risks of our work in Appendix D.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The paper’s main claims are summarized in the section ’Abstract’ and the section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We provide the dataset and open toolkit in the section 4.1, 4.2, and Appendix A.
3 B1. Did you cite the creators of artifacts you used? We cite the dataset and open toolkit in the section 4.1, 4.2, and Appendix A.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We discuss the license of dataset in Appendix A.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discuss the existing artifact was consistent with their intended use n the section 4.2 and Appendix A.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We check the details of dataset in Appendix A.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We provide the details of domains and languages in Appendix A.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We carefully provide the statistics of all data in Appendix A.
C 3 Did you run computational experiments? We provide the computational experiments in Appendix B and C.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We provide all implementation details and training setup in section 4, Appendix B.1 and Appendix B.2. The section 4.4 also contains the number of parameters in the models used. We report the training cost in Appendix C.1.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? We provide the experimental setup with hyper-parameters and configuration in Appendix B.1
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report the statistics about our results in Appendix B.1.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We used existing packages of scripts and toolkit and we report these details in section 4.2.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will compress the number of parameters to the same size of original models meanwhile preserve the performance on all translation directions. (2) The gap between our scenario and the realworld scenario. Our proposed method is a whitebox service in incremental learning. Thus, we train
a powerful MNMT model as the original model instead of directly utilizing existing models from the Internet. And we only consider eight incremental language pairs due to the limitation of computation resources. We try our best to simulate the realworld scenario and we will apply our proposed method for large-scale pre-trained MNMT models (e. g. , NLLB 54. 5B and M2M 12B) to validate the effectiveness in industrial scenarios