Continual Learning for Multilingual Neural Machine Translation via Dual Importance-based Model Division
Authors: Junpeng Liu
Kaiyu Huang
Hao Yu
Jiuyi Li
Jinsong Su
Degen Huang
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual Neural Machine Translation
Continual Learning
Model Pruning
TL;DR: We effectively adapt the existing multilingual neural machine translation to new language pairs and promote the translation performance on both supervised and zero-shot directions.

Abstract: A persistent goal of multilingual neural machine translation (MNMT) is to continually adapt the model to support new language pairs or improve some current language pairs without accessing the previous training data. To achieve this, the existing methods primarily focus on preventing catastrophic forgetting by making compromises between the original and new language pairs, leading to sub-optimal performance on both translation tasks. To mitigate this problem, we propose a dual importance-based model division method to divide the model parameters into two parts and separately model the translation of the original and new tasks. Specifically, we first remove the parameters that are negligible to the original tasks but essential to the new tasks to obtain a pruned model, which is responsible for the original translation tasks. Then we expand the pruned model with external parameters and fine-tune the newly added parameters with new training data. The whole fine-tuned model will be used for the new translation tasks. Experimental results show that our method can efficiently adapt the original model to various new translation tasks while retaining the performance of the original tasks. Further analyses demonstrate that our method consistently outperforms several strong baselines under different incremental translation scenarios.

Continual Learning for Multilingual Neural Machine Translation via
Dual Importance-based Model Division
Junpeng Liu'
Kaiyu Huang? Hao Yu!
Jiuyi Li"
Jinsong Su
Degen Huang
DDliann University of Technology
Institute for Al Industry Research, Tsinghua University *Xiamen University
(liujumpeng_nlp. yuhao_dlut, 91e.91]mmm
edu.cc
wttpyyuuiii..stiguuaaaaua jssu@xmu. edu. cn
huangkaiyu@air.. tsinghua. edu. cn
huangdg@dlut. edu. cn
Abstract
ABLBU vs. Modal
A persistent goal of multilingual neural ma- chine translation (MNMT) to continually adapt the model support new language pairs or improve some current language pairs with- out accessing the previous training data. To achieve this, the existing methods primarily fo- cus on preventing catastrophic forgetting by making compromises between original and new language pairs, leading sub-optimal per- formance on both translation tasks. To mitigate this problem, we propose a dual importance- based model division method to divide the model parameters into two parts and separately model the translation of the original and new tasks. Specifically, we first remove the parame- ters that are negligible to the original tasks but essential to the tasks obtain pruned model, which is responsible for the original translation tasks. Then we expand the pruned model with external parameters and fine- tune the newly added parameters with new training data, The whole fine-tuned model will used for the new translation tasks. Experimental re- sults show that our method can efficiently adapt the original model various new translation tasks while retaining the performance of the original tasks. Further analyses demonstrate that method consistently outperforms sev- eral strong baselines under different incremen- translation scenarios."
-1.2
WMT-9/242M
mBART550mmx611M
-L.8
13 20
25
Weight Pruming Ratio (9)
Figure I: The BLEU decline of different MNMT mod- els on the en++xx under different pruning ratios, We perform weight pruming with the general magnitude- based method (See et al., 2016).
practical problem is how efficiently extend those MNMT models support new languages im- prove some existing translation directions when the new training data is available. Considering that the current MNMT models are generally trained with a large amount of data (Liu et al., 2021; Ebrahimi and Kann, 2021), it will be extremely resource and time-consuming to retrain the model from scratch. Therefore, the method that continually adapt the previously learned model tasks while retaining the original knowledge with- out accessing the previous training data, i.e., con- tinual learning, has drawn increasing research at- tention over the past few years (Gu et al., 2022: Zhao et al., 2022; Sun et al., 2023; Huang et al., 2023).
translation
1 Introduction
Multilingual neural machine translation (MNMT) (Johnson et al., 2017; Zhang et al., 2020; Liu et al., 2022: Goyal al., 2022) handles multiple transla- tion directions scale MNMT models, such as mBART50-nn (Tang et al., 2020), M2M- 100 (Fan et al., 2021) and NLLB (Costa-jussà et al., 2022) have demonstrated promising translation performance. However,
The biggest challenge for continual learning is catastrophic forgetting (French, 1993), which re- sults in severe performance decline on the previous tasks when adapting to new tasks. To alleviate this problem, some works attempt to balance the perfor- mance between the original and incremental tasks, such the replay-based methods (Ko et al., 2021: Liu et al., 2021: Garcia al., 2021) and regulariza- tion methods (Khayrallah et al., 2018; Thompson et al., 2019; 1 Zhao et al., 2022). However, as the whole parameter spaces are shared across different
single model and recent large-
*Corresponding Author
!Our cnde is available at https://githur com/ raburabu91/BvPACC

---

translation tasks, those methods are vulnerable to inter-task interference, especially when the incre- mental tasks are largely different from the original ones. Other works (Bapna and Firat, 2019; Es- colano et al., 2021) employ additional task-specific parameters for new task adaptation while keeping the parameters in the original model fixed, which can naturally avoid the forgetting problem. How- ever, those methods increase the model size and the structure of the task-specific modules requires specialized manual design.
roles of the model parameters in incremental learning and preserve the performance of the previous translation tasks without accessing the original training data.
We conduct extensive experiments in different incremental translation scenarios. Experimen- tal results show that our method can be easily applied to the pre-trained MNMT models and improve the zero-shot translation.
Previous studies (Bau et al., 2019; Voita et al., 2019; Gu et al., 2021; Liang et al., 2021) show that some parameters in the network are not important to the original tasks so that they can be pruned with- out causing obvious performance degradation. As shown in Figure 1, the current MNMT models also have such unimportant parameters, which provides the possibility of using those negligible parameters to learn the knowledge of new tasks instead of intro- ducing additional parameters. However, this model pruning strategy focuses more on the importance of parameters to the original tasks while neglecting their roles in new task adaptation, which restricts the model's ability to learn the specific knowledge of the incremental languages.
2 Related Work
Replay-based Methods. The replay-based meth- ods utilize the training data or pseudo data of the previous tasks to replay the old knowledge while training on new tasks (Lakew et al., 2018; Sun al., 2019; de Masson D' Autume et al., 2019; Liu et al., 2021; Garcia al., 2021). However, those meth- ods will bring more training consumption, espe- cially for the large-scale pre-trained MNMT model. Moreover, the previous training data are sometimes unavailable due to data privacy or storage limita- tions (Feyisetan et al., 2020; Gu et al., 2022). The noise in pseudo data will also hurt the performance of the previous and new tasks. Compared to those methods, our method does not need to access the previous data, and thus it is more flexible and effi- cient for incremental learning.
In this work, considering the importance of pa- rameters on the original and new translation tasks, we propose dual importance-based model divi- sion method that can efficiently adapt the origi- nal MNMT model to various incremental language pairs. First, we fine-tune the original MNMT model with the new training data and propose two meth- ods to evaluate the importance of parameters in different tasks. Then find the parameters that are unimportant to the original lasks but important to the new tasks and remove them from the origi- nal model to obtain a pruned model. Finally, we expand the pruned model to its original size and fine-tune the newly added parameters with new training data. In this way, we can better capture the specific knowledge of incremental translation tasks and avoid catastrophic forgetting on the orig- inal translation tasks. Our main contributions are summarized as follows:
Regularization-bant Methods. In addition to the original training objective, the regularization- based methods usually introduce additional penalty terms to balance the performance the previous and new tasks. Khayrallah al. (2018) and Thomp- son et al. (2019) employ regularization terms to constrain the change of all the parameters from their original values. Castellucci et al. (2021) uti- lizes the original model as the teacher to prevent catastrophic forgetting via knowledge distillation. Gu et al. (2022) updates the parameters within the low forgetting risk regions with hard constraint. By contrast, our method only prunes the param- eters that are not essential the previous tasks so that it can prevent the forgetting problem. It also allows our method to focus on learning the new tasks without making compromises with the previous tasks.
We propose dual importance-basee model division method to improve the model's abil- ity to learn the knowledge of new language pairs, which achieves competitive translation qualities on incremental translation tasks.
Parameter Isolation- Methods. The pa- rameter isolation based methods generally allocate the model parameters to different tasks. Of this kind, some works (Bapna and Firat, 2019; Madotto
Our method can efficiently reorganize the

---

Uningurtant
Imporiant Paramation Non Tisk
Frizen Paramation
importance
Pruning
Model Expansion Fine-tuning
Task aat--ipeeiik Model
New Task
Exaluation
Porameter Intilalization
Tosk-specjic Fine tuning
Importance Weight Pruning
Model Exponsion
Iecomeets Lergage Pan
Primed Morkel
Modal
Original Model
Figure 2: The training process of the proposed method. Our method consists of three steps: parameter importance evaluation, model pruning and model expansion & fine-tur fine-tuning.
et al., ; Zhu et al., 2021) add extra task-specific parameters to the original model for new task learn- ing, Despite retaining the performance on previous tasks, those methods inevitably increase the num- ber of model parameters. Other works (Liang et al.. 2021; Gu et al., 2021) prune the model based on the parameter importance to the original tasks for do- main adaptation. Different from those methods, our method explores the potentiality of task-specific pa- rameters to improve the performance and efficiency of the new task adaptation without additional pa- rameters.
parameters have varying degrees importance for the previous and new translation tasks. Based on this, we utilize the parameters that are not impor- tant to the previous tasks but essential to the new tasks perform our method involves the following steps as shown in Figure First, we fine-tune the original MNMT model on the new training data to obtain a task- specific model, and then evaluate the importance of parameters in the original MNMT model and the task-specific model separately. Second, we remove the parameters that are unimportant to the original tasks but important to the new tasks from the origi- nal MNMT model obtain a pruned model, and we make responsible for the translation of the original tasks. The pruned model can retain the per- formance of the original tasks since the removed parameters are not necessary. Last, we expand the pruned model to its original size and fine-tune the newly added parameters to learn the new transla- tion tasks. As those new parameters essential to the tasks, they can learn the knowledge of the new tasks more efficiently.
task adaptation. Specifically,
Model Pruning. Model Pruning usually aims to reduce the model size and has been widely used in natural language processing tasks. Sun et al. (2020) highlights the importance of the sparse net- work structure. Lin et al. (2021) focuses on finding the important parameters for each language per- form language-specific modeling in MNMT. Gu al. (2021) utilizes the unimportant parameters to learn the in-domain knowledge. Compared those methods, our method considers the importance of parameters on the previous and incremental tasks simultaneously during the model pruning stage, which yields better performance on new tasks while retraining the knowledge of the previous tasks.
3.1 Importance Evaluation
The importance evaluation procedure is to search for the negligible parameters the original model that can be erased and the essential parameters that contribute more to the new task adaptation. To achieve this, we perform importance evaluation for the original and new translation tasks separately.
3 Method
In this work, we aim prevent the catastrophic per- formance degradation on the previous translation tasks without accessing their training data while adapting the model to new translation tasks effi- ciently. The main idea of our method is that the
Original Task Evaluation. To evaluate the pa- rameter importance in the original model, we adopt the general magnitude- based approach (See et al.,

---

2016) which regards the absolute value of each parameter its own importance:
Specifically, based on the importance evaluation re- sults on the original lasks, we first select a% param- eters with the smallest magnitude from each weight matrix d to form the pruning candidate set M. Since the parameters in M are relatively unim- portant to the original tasks, they can be pruned without causing catastrophic forgetting. Then we determine which parameters in M should be fi- nally pruned based on the importance evaluation results on the new tasks. According to different importance evaluation methods for the new tasks and pruning distributions each weight matrix, we propose three model pruning strategies. Uniform Magnitude-baseed Pruning (UMP) adopts the magnitude-based method and sorts the parameters in M by their magnitude in weight ma- trix W then p% parameters with largest magni- tude in each weight matrix will be pruned (p < a). Uniform Variation-based Pruning (UVP) adopts similar pruning method UMP except that the importance evaluation method is changed to the inttation-baseee onc.
In(wi] = lugil wij E Wo
(1)
where wi denotes the i-th row and j-th column parameter of a certain weight matrix W in the original model. In this way, the smaller the absolute value, the less important the parameter is.
New Task Evaluation. To obtain the important parameters for the new translation tasks, first fine-tune the original model with the new training data to obtain a task-specific model. Considering that the new languages may have different scripts from the original languages, we extend the embed- ding layer of the original model to avoid the out- of of-vocabulary problem. The vocabulary extension method is depicted in Appendix B. After that, we propose evaluation methods based on the mag- nitude and variation of the parameters, respectively. On the one hand, fine-tuning the original model on the new translation tasks will amplify the magni- tude of the important parameters and diminish that of the unimportant parameters. Therefore, we can simply utilize similar magnitude-basee method to Equation (I) for the new task evaluation. On the other hand, the variation of each parameter from the original MNMT model the task-specific model can also illustrate their importance. Intuitively, the value variations of the important parameters should be larger than those of the unimportant parame- ters since they capture the specific knowledge of the new tasks. Based on this, the variation buasee evaluation method is formulated as
Blind Variation-based Pruning (BVP) also adopts the variation based method, but different from UVP, it selects 5% parameters with the largest variation from each weight matrix W form the important parameter set N. Then the parameters in the intersection set of M and A will be pruned.
Note that the pruning ratios in each weight ma- trix the same for the first two methods while varying for the third method. The pruned model will be used to generate the translation of the origi- nal language pairs in the inference stage.
3.3 Transfer-based Model Expansion
Le(wit)= = I - - wty E W W",,ff w/
W (2)
In this step, we add new parameters to the pruned model and expand it to the original size. Moreover. the embedding layer of the pruned model is also tended with the embeddings of new languages sim- ilar to the task-specific fine-tuning in Section 3.1. Considering the potential representation gap be- tween the newly added parameters/embeddingg and the parameters in the pruned model, we propose a transfer- based parameter adaptation that initializes the new parameters/embedding. with their counter- parts in the fine-tuned task-specific model. On the one hand, those parameters have been fitted well with the original model during the task-specific fine-tuning stage so that they do not suffer from the representation gap. On the other hand, the task- specific knowledge contained in those parameters
where w/z and wif represent the i-th row and j-th column parameter of the weight matrices Wo and W*. The weight matrices w and W denote dif- ferent parts of the original and fine-tuned specific model, respectively.
3.2
Dual Importance-based Model Pruning
Based on the importance of parameters to the origi- nal and new translation tasks, we determine which parameters in the original MNMT model should be pruned. To make the most of the model's ability to learn the knowledge of new tasks while preserving the performance on the original tasks, we propose a dual importance-bnl based model pruning method.

---

will benefit the new task adaptation. We fine-tune the expanded model on the new training data (D) and the training objective
Replay (Sun et al., 2019) This method creates pseudo training data for the original language pairs and trains new models on the combina- tion the pseudo training data and incremen- tal training data.
C() - Σ (XY)ED
1og ON)
(3)
Fine-Tuning (Luong and Manning, 2015) This model is trained based on the original MNMT model only with the incremental train- ing data.
where ON denotes the trainable parameters includ- ing the newly added parameters (fp) and embed- dings (0g), and Oo denotes the pruned model which is kept frozen during this training stage. In the in- ference stage, all the parameters in the expanded model are employed to generate the translation of the new tasks.
L2-Reg (Miceli Barone et al., 2017) This method adds an additional 1.2-norm regular- ization term to the training objective.
EWC (Kirkpatrick al., 2017) This method evaluates the importance of parameters with the Fisher information matrix and adds a penalty term to the training objective to re- tain the original knowledge.
4 Experiments
4.1 Datasets
To ensure the reliability of the experiments, we first perform continual learning based on a self- trained MNMT model and then employ the pre- trained mBART50-nn model (Tang et al., 2020) the initial model to evaluate real-world scenario. For the former, the original MNMT model is trained a multilingual trans- lation dataset (WMT-9) covering language pairs. And then we select another four language pairs for incremental learning. For the latter, we adopt mBART50-nn as the initial MNMT model and per- form continual learning in two translation tasks: the language adaptation task and the language en- hancement task. The language adaptation task aims enable the model support the translation of new languages, while the language enhancement task aims to improve the translation performance of some existing language pairs that are already sup- ported by the model. The detailed descriptions of datasets for the original and incremental languages are in Appendix A.
LFR (Gu al., 2022) This method updates the parameters in the low forgetting risk re- gions with a hard constraint. We reproduce the LFR-OM model in the experiments.
method in the
Adapter (Bapna and Firat, 2019) This method freezes the whole network of the original model and employs additional networks to learn the knowledge of incremental language pairs. The dimension of the projection layer is set to 128 in the experiments.
PTE (Gu et al., 2021) This method com- presses the original model to learn new tasks and introduces a distillation loss to retain the old knowledge. We omit the knowledge distil- lation step as the original training data is not available in settings.
Other baselines in the experiments based on the mBART50-mn model are as follows:
4.2 Implementation Details
mBART50-nn (Tang et al., 2020) The large- scale pre-trained MNMT model, which is the baseline model for the language enhancement task. Other systems implemented based this model.
Baselines. The original MNMT model is trained based the vanilla Transformer (Vaswani al., 2017) architecture with multiple parallel data fol- lowing Johnson et al. (2017). We compare our method with various previous methods in continual learning. The baselines are as follows:
mBART550mn+LSST (Berard, 2021) This method inserts a new language-specific em- bedding layer (LSE) for the new languages. we adopt it as the baseline model for the lan- guage adaptation task and implement other methods based on it.
Scratch This method trains the initial MNMT model on the original 8 language pairs for continual learning and the bilingual model on each incremental language pair from scratch.

---

Original Language Puins
Incremental Language Pains
Model
en->xx
xx-ten
AVGI
en mm
m-
en-
de-ren
en->la
la la-hen
en en-
ga-een
AVG2
Scratch
18,90
23,21
21,06
23.42
31.29
24.78
30.13
9.95
14.41
12.43
12,61
19.88
Fine-Tuning
0.50
0.50
0.50
24.56
33.81
25.39
30.61
10.43
15.63
19.09
21.01
22.57
Replay
15.27
22.90
19.09
19.75
29.66
22.16
28.45
10.14
15.10
17.06
19.88
20.28
1.2-Reg
14.80
23.89
19.35
21.94
31.83
17.87
27.27
8.53
16.40
12.97
20.69
19.69
EWC
16.40
23.32
19.86
22.77
32.18
19.12
27.52
8.68
16.84
13:40
21.41
20.24
LFR
17.35
23.43
20.39
22.90
32.46
18.93
27.91
9.12
16.98
13.54
21.55
20.42
Adapter
18,90
23.21
21.06
23.89
33,36
21.89
29.14
9.49
16.31
16.01
21.81
21.49
PTE
18.87
23.04
21,01
24.03
33.42
22.09
28.65
9.69
16.46
17.55
21,46
21.67
UMP
18.69
22.63
20.66
24.70
33.78
22.64
29.31
9.54
16.19
18.53
22.89
22.19
UVP
18.85
23.09
20.97
24.63
33.82
22.76
29.19
9.68
16.73
19.58
22,37
22.35
BVP
18.86
23.10
20.98
24.60
33.98
22.94
29.13
9.92
16.69
19.36
22.42
22.38
Table I: The BLEU scores of adding a single language pair for MNMT incremental learning. "AVGI" and "AVG2" denote the average BLEU scores on the original language pairs and incremental language pairs, respectively. The best BLEU scores among all the incremental learning methods are marked in bold.
Training Setup. Our translation models are built on the Transformer using the open-source Fairseq implementation (Ott et al., 2019)². We employ the same configuration of Transformer-Bit (Vaswani et al., 2017) in our experiments for fair compar- isons. The details about model settings are in Ap- pendix B.
are weak in their ability learn the knowledge of new language pairs, leading to a performance gap compared to the Fine-Tuning model. In contrast to those prior continual learning methods, our method achieves the best overall translation performance.
Evaluation. We report the detokenized case- sensitive BLEU offered by SacreBLEU (Post, 2018)3. We employ the best checkpoint on the val- idation dataset for evaluation, and perform beam search decoding with beam size of 4 and length penalty of 1.0.
Adapting to multiple language pairs. We next investigate the translation qualities when adapting the original MNMT model to multiple language pairs simultaneously. The results are summarized in Table 2. In this scenario, the regularization- based methods retain their performance original language pairs at the cost of the insuf- ficient ability learn the incremental language pairs, which leads inferior performance the Scratch model. We also find that the performance degradation on English-to-Irie -rish (en- transla- tion is more severe than other language pairs. We ascribe this to the larger linguistic difference be- tween Irish (ga) and other languages, as Irish has distinct dominant word order (VSO) from other languages (SVO or SOV). This difference may lead to interference acToss languages, which increases the difficulty of the regularization-be based methods to balance the translation performance between the original and incremental language pairs. By con- trast, our method can better deal with the increased language divergence, and yield superior translation performance to the Scratch and PTE models. As we consider the incremental language pairs during the parameter importance evaluation stage, our method can capture the specifie knowledge of each incre- mental language pair more efficiently than other methods.
the
4.3 Main Results
Adapting to a single language pair. We first study the translation performance when adapting the original MNMT to single language pair. The results are summarized in Table 1. Compared with the Scratch model, our method achieves remark- able performance improvement the incremental language pairs (up to +2.50 BLEU average) and the translation qualities are also competitive with the Fine-Tuning model. Moreover, our method prevents catastrophic forgetting on the original lan- guage pairs. Despite prior attempts by the replay- based and regularization-batt based methods to make a balance between the original and incremental language pairs, they still suffer from obvious per- formance degradation on the original en-xxx trans- lation tasks. Although no pronounced degradation has occurred in the Adapter and PTE models, they
Ptt tps: //gi wwsithub.com... wwwmppytoor//fairsseeeoo 'Signature: BI. Aw..L//cacc.ixcccduumne. 4smno +smnoth.cxp= tok. 13a4versionn115.. 1.5...

---

Original Language Puins
Incremental Language Pains
Model
en->xx
xx-ten
AVGI
en -m
m-
en-
de-ten
en->la
la le-hen
en en-
ga-een
AVG2
Scratch
18,90
23,21
21,06
23.42
31.29
24.78
30.13
9.95
14.41
12.43
12.61
19.88
Fine- Fine-Tuning
0.40
1.74
1.07
24.68
34.85
26.08
31.34
10.11
16.81
14.03
22.81
22.59
L2-Reg
16.82
23.60
20.21
16.55
25.51
14.66
25.44
6.26
14.08
2.63
9.88
14.38
EWC
16.97
23.46
20.22
17.00
26.54
18.50
26.99
6.50
14.08
2.51
9.42
15.19
LFR
17.16
23.49
20.33
18.21
28.58
16.56
26.24
6,56
14.37
4.20
12.75
15.93
PTE
18.79
22.80
20.79
23.06
32.16
20.47
27.79
8.12
15.38
12.04
20.12
19.89
UMP
18.74
22.85
20.80
23.27
33.09
21.08
28.49
8.84
16.02
12.40
21.20
20.55
UVP
18.87
23.11
20.99
23.54
32.92
21.23
28.61
9.05
15.23
12.74
20.94
20.53
BVP
18.85
23.16
21.01
23.53
32,92
21.34
28.49
8.97
15.88
12.78
21.39
20.66
Table The BLEU scores adding all eight language pairs simultaneously MNMT incremental learning. The best results among all the continual learning methods are highlighted in bold.
5 Analysis
5.1 Ablation Studies
Effects of Different Pruning Ratios. To better balance the translation performance on the orig- inal and incremental translation tasks, we quan- tify the trade-off between the pruning ratio and the translation performance. The results are depicted in Figure 3. We also plot the PTE model in the figure for comprehensive comparisons. For each method, the performance on the original language pairs degrades with the increase of the pruning ra- tio, while the results on the incremental language pairs are quite reversed. Specifically, the proposed three pruning methods consistently outperform the PTE model on the incremental language pairs un- der the same pruning ratios. Moreover, the UVP and BVP models can achieve better results with smaller pruning ratio of 6% than the PTE model of 18%. These results demonstrate that training the in- cremental language pairs with their corresponding important parameters can improve the efficiency of continual learning, which makes it possible to adapt the original model to more new translation tasks. The UVP and BVP models can better re- tain the performance the original language pairs while the UMP model suffers from more apparent performance degradation. Considering the overall performance under different settings, we employ the BVP model for the following experiments.
Wagg Thaaing Radl-
(a) Original Languages
(b) Incremental Languages
Figure 3: Comparisons of PTE, UMP, UVP and BVP under different pruning ratios.
Method
XX-hen
entedde
entekl
Old
18.87
23.04
28.73
25.37
13.08
19.51
New
0.03
0.04
29.36
26.98
13.07
20.96
BVP
18.86
23.10
29.29
26.04
13.31
20.89
Ablation study on dual importance- based prun- ing. "Old" and "New" indicate the model pruning meth- ods only based the parameter importance on the old task and new task, respectively.
Table 3:
knowledge of new language pairs, leading to infe- rior performance to our BVP model. In contrast, when pruning only based on the new task impor- tance, although the model performs slightly better than BVP on some new language pairs, it fails on the original language pairs. We find that most (>98%) pruned parameters in this method are also important to the old language pairs. Therefore, the performance on original language pairs decreases apparently if those parameters are pruned. Our BVP model achieves the best overall performance across all language pairs, showing the necessity of the iual-importance.
Effects of Dual importance-based Pruning. We study the dual-importance pruning strategy with our BVP method under the pruning ratio of 6%. As shown in Table 3, when pruning only based on the old task importance, the model can maintain per- formance on the original language pairs. However, it is weak in the ability of capturing the specific
Effects Different Initialization methods. We implement three initialization methods to investi-

---

Model
en++xx
en-
Xx-
Model
ent
entOro
ent>la
Scratch
21.06
17.65
Scratch
21.06(-0.00)
27.36(-0.00)
12.18
Fine-Tuuing
2.25
19.51
5.18
Fine-Tuning
0.31 (-31(-0.19)
(234(-28.85)
12.86
L2-Reg
17.27
14.46
3.54
L2- Re-Reg
(-.17))
26.52
11.90
EWC
18.45
15.93
3.67
EWC
19.39
26.59
12.17
LFR
18.72
16.69
3.84
LFR
19.37
26.81 (-0.87)
12.27
PTE 20.98
18.12
6.16
PTE
20.89
28.42
12.59
Our)( 20.98
18.85
6.67
Ours(BVP)
(0.00))0.08)
29.18 -0.111
13.05
Table 4: The BLEU results on zero-shot translation. "xx" represent the 8 original languages and "yy" represent the 4 incremental languages in Table 9. The best results among all the continual learning methods are in bold.
Table 5: The BLEU results of the sequential language incremental learning. The value brackets represents the performance decline before and after training with the en-+*a data. The best results among all the continual learning methods are in bold.
gate the effects of different parameter initialization methods in the model expansion stage. The results indicate that initializing the newly added param- eters with their counterparts in the task-specific fine-tuned model enables the BVP model to at- tain optimal translation performance with minimal training time for incremental learning. The de- tailed comparisons and analyses provided in Appendix C.
parameters before training the new language pairs. For the L2-Reg and EWC methods, we em- ploy the model trained with the ent+ro data as the new initial model to compute the regularization loss when continually training the en<-xta task. Be- sides, we also recompute the Fisher information matrix and update the low forgetting risk regions for the EWC and LFR methods, respectively. For our methods, we evaluate the importance of pa- rameters on enttta and prune the parameters that are important en tt task but unimportant all the previous tasks including ent>ro translation. The results are summarized in Table 5. The Fine- Tuning method still encounters the catastrophic for- getting problem so that its performance on ent+ro degrades sharply after training with the en<+tla data. Although the regularization-basel methods achieve relatively better overall performance, they also suf- fer from performance degradation different de- grees. In contrast to those methods, our method only has slight decline in the previous tasks and outperforms other continual learning methods on all the translation tasks in this scenario.
5.2
Results on Zero-shot Translation
To investigate the transfer ability of our method, we further conduct zero-shot translation from the original languages to the incremental languages. To make the most of the ability to learn the knowl- edge of the incremental languages, we adapt the original MNMT model to only one incremental language each time. We adopt the Flores (Goyal et al., 2022) test sets for evaluation and the results are summarized in Table 4. For the regularization- based methods, the translation quality on the incre- mental language pairs and zero-shot language pairs lags far behind that of the Fine-tuning model. The results indicate that the constraint on the param- eters limits the transfer ability of those methods, leading to inferior performance on zero-shot trans- lation. Compared with the PTE model, our method achieves better zero-shot translation performance since capture the specific knowledge of the new languages more efficiently.
5.4 Results on Pre-trained MNMT models
We further employ the pre-trained mBART50-m model (Tang et al., 2020) as the original MNMT model and conduct two incremental learning tasks in the scenario that closer to the real world.
5.3 Sequential Language Adaptation
I Language adaptation task. In this scenario, we adapt the model support the Eneek++English (el+>en) and Slovak (sk+xen). Follow- ing Gu al. (2022), we conduct experiments based on the mBART50-nn+LSSE model (Berard, 2021) and report the tokenized BLEU in Table 6. Com- pared with all the previous continual learning meth-
In this scenario, enable the original MNMT model adapt the incremental language pairs in sequential manner. We first train the model with the en data and then with the en<*tt data. For each method, we extend the embedding layer of the previous model and update the specific hyper-

---

Original Language Pairs
Incremental
Language Pairs
Model
AVG
XX->en
en-
AVG1
el-ten
en-+el
sk->en
en-
AVG2
mBART50-mmHLSE
26.73
21.66
24.20
27.32
16.20
36.28
28.85
27.16
25.68
Fine- mmn-inging
20.40
1.88
11.14
30.88
27.80
35.32
33.69
31.92
21.53
Le-Reg
27.32
18.78
23.05
28.05
19.99
36.17
30.86
28.77
25.91
EWC
27.02
18.41
22.72
27.73
20.27
36.27
30.96
28.80
25.76
LFR
27.12
20.16
23.64
28.47
20.43
36.30
31.04
29.06
26.34
(urP(
26.58
21.58
24.08
29.62
25.64
34.80
32.29
30.59
27.34
"AVGI" and "AVG2" denote the average BLEU on the original and incremental language pairs, respectively. "AVG" is the overall performance computed by (AVG1+AVG2y2. We report the tokenized BLEU following Gu al. (2022). The best scores among all the continual learning methods are highlighted in bold.
Table 6: The BLEU scores of the language adaptation task.
Original Language Pairs
Incremental Language Pairs
Model
AVG
Xx xx->en
en-Xx
AVGI
de *fr
fr- zh-vvi
vi-+ah
AVG2
mBART50-nn
24.88
15.63
20.26
15.88
10.17
13.35
6.48
11,47
15.87
Fine-Tuning
0.96
2.79
1.88
29.88
28.57
30.02
30.37
29.71
15.80
L2-Reg
20.55
15.48
18.02
29.05
23.75
23.69
25.86
25.59
21.81
EWC
22.40
15.54
18.97
29.22
23.41
23.89
26.02
25.64
22.30
LFR
22.68
15.45
19.07
30.71
24.30
24.61
26.37
26.50
22.79
Ours(BVP)
24.78
15,56
20.17
32.59
26.55
28.89
29.86
29.47
24.82
Table 7: The BLEU scores of the language enhancement task. The best results among all the continual learning methods are highlighted in bold.
ods, method achieves better overall translation performance on the original and incremental tasks. Specifically, our method narrows the performance gap between the previous continual learning mod- els and the Fine-Tuning model on the incremental language pairs, especially for the en-tel direction. These results show that our method can also effi- ciently adapt to the new language even if it is quite different from the previous languages.
not available during the incremental learning stage, weakening the ability of the model trans- late into English. In contrast to the regularization- based methods, our method still retains the perfor- mance on the original language pairs and achieves better translation results on the incremental lan- guage pairs.
Conclusion
Language enhancement task. As mBART50-nn is trained the English-centric data, the trans- lation quality of non-English language pairs gen- erally lags far behind that of the English-centric language pairs. To this end, we aim to improve the translation performance on two non-English language pairs: Germank> Shineee++Veetman Fermank-French French (dex+fr) and (zh+vví). The results are summarized in Table 7. The regularization-bant methods suffer from performance degradation in the xx- directions, which is different from that in the language adaptation task. We ascribe this to the different training data distributions, where half the training set uses English as the target during the pre-training stage, while the English sentences
In this paper, propose dual importance-based model division method in continual learning for MNMT, which is based the importance of pa- rameters the original and incremental translation tasks. We search for the parameters that are unim- portant to the previous tasks but essential to the incremental tasks and utilize those parameters for new task adaptation. Experimental results show that our method can efficiently adapt the original MNMT model to various incremental translation tasks and consistently outperforms several previ- ous continual learning methods. Further analyses demonstrate that our method can also be applied to the pre-trained MNMT model and benefit the zero-shot translation.

---

Limitations
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 1538- 1548. Hong Kong, China. Association for Computa- tional Linguisties.
In this work, we attempt to extend an existing MNMT model to support new languages and im- prove the translation of some language pairs. In addition to the advantages, our method has the fol- lowing limitations.
Authony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Iden- tifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations.
(1) Insufficient model capacity. In order to sup- port new translation tasks while keeping the model size fixed, we only utilize small number of pa- rameters in the original MNMT model for new task adaptation. When adapting to multiple translation tasks simultaneously, the translation performance on the incremental translation tasks still lags behind that of the Fine- Tunin model due to the limited modeling capacity. In the future, we will improve the translation on the new tasks with better trans- fer method while not increasing the number of pa- rameters.
Alexandre Berard. 2021. Continual learning in multilin- gual NMT via language-specific embeddings. In Proceedings of the Sixth Conference Machine Translation, pages 542-565, Online. Association for Computational Linguistics.
Giuseppe Castellucci, Simone Filice, Danilo Croce, and Roberto Basili. 2021. Learning to solve NL.P tasks in incremental number of languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pages 837-847, Online. Association for Computational Linguisties.
(2) Single direction of knowledge transfer. Our method is essentially a parameter isolation- based model in which the parameters contribute differ- ently to different translation tasks. We utilize all the parameters for the translation of new tasks while for the original tasks, we only use the parameters in the pruned model. Therefore, the knowledge can be easily transferred from the original language pairs the new language pairs. However, the orig- inal language pairs can hardly acquire knowledge from the new language pairs. In the future, we will explore methods that can facilitate the knowledge transfer to the original languages to further improve the translation performance on the original tasks.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke 7ettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- loos-linguul representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbaayd, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint -XXix2207.04677
A Acknowledgements
We sincerely thank all the anonymous reviewers for their insightful comments and suggestions to improve the paper. This work was supported by the National Key Research and Development Pro- gram of China (Grant No. 20204400108004) and the Key Research and Development Program of Yunnan Province (Grant No. 202203AA080004).
Cyprien de Masson D' Autume. Sebastian Ruder, Ling- peng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32.
Abteen Ebrahimi and Katharina Kann. 2021. How to adapt your pretrained multilingual model to 1600 languages. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers). pages 4555-4567, Online, Association for Computa- tional Linguistics.
References
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. arXiv preprint 1907000099
Carlos Escolano. Marta Costa-Jussà, and José AR Fonollosa. 2021. From bilingual to multilingual neural-based machine translation by incremental training. Journal of the Association for Information Science and Technology, 72(2): 190-
Ankur Bapna and Orhan Firat. 2019. Simple, seal- able adaptation for neural machine translation. In

---

Angela Fan, Shruti Bhosale. Holger Schwenk, Zhiyi Ma, Ahmed El- Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centrie multi- lingual machine translation. The Journal of Machine Learning Research, 22(1 ):4839448866 4886.
Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018. Regularized training objec- tive for continued training for domain adaptation in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 36-44.
Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and tex- tual analysis via calibrated multivariate perturbations. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 178-186.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 1012:6442.69800.
Tom Diethe. 2020. Privacy-and
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska- Barwinska, et al. 2017. Over- coming catastrophic forgetting neural networks. Proceedings of the national academy of sciences. 1141133332133356
Robert French. 1993. Catastrophic interference in con- nectionist networks: Can it be predicted, can it be prevented? Advances in Neural Information Process- ing Systems, 6.
Xavier Garcia, Noah Constant, Ankur Parikh, and Orhan Firat. 2021. Towards continual learning for multilin- gual machine translation via vocabulary substitution. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1 1184-1192, Online. Association for Computa- tional Linguistics.
Wei-Jen Ko, Ahmed El-Kishky, Adithya Renduchin- tala, Vishrav Chaudhary, Naman Goyal, Francisco Guzmán, Pascale Fung, Philipp Koehn, and Mona Diab. 2021. Adapting high-resource NMT models to translate low-resource related languages without par- allel data. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing ( (Volume I: Long Papers), pages 802-812, Online. Association for Computa- tional Linguistics.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato. Francisco Guzmán, and Angela Fan. 2022. The Flores- 101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-533 538.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Shuhao Gu, Yang Feng, and Wanying Xie. 2021. Pruming-thee then-expanding model for domain adapta- tion of neural machine translation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages 3942-3952, Online, Association for Computational Linguistics.
Surafel M. Lakew, Aliia Erofeeva, Matteo Negri, Mar- cello Federico, and Marco Turchi. 2018. Transfer learning in multilingual neural machine translation with dynamic vocabulary. In Proceedings of the 15th International Conference on Spoken Language Trans- lation, pages 54-61, Brussels. International Confer- ence on Spoken Language Translation.
Shuhao Gu, Bojie Hu, and Yang Feng. 2022. Continual learning of neural machine translation within low for- getting risk regions. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1707-1718, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Jianze Liang, Chengqi Zhao, Mingxuan Wang, Xipeng Qiu, and Lei Li. 2021. Finding sparse structures for domain specific neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13333-13342.
Kaiyu Huang, Peng Li, Jin Ma, Ting Yao, and Yang Liu. 2023. Knowledge transfer in incremental learn- ing for multilingual neural machine translation. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 15286-1558 -13344, Toronto, Canada. Association for Computational I. Linguistics.
Zehui Lin, Liwei Wu, Mingxuan Wang. and Lei Li. 2021. Learning language specific sub-network for multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 293-305, Online. Association for Computational Linguisties.
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado. Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system; En- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351.
Junpeng Liu, Kaiyu Huang. Jiuyi Li, Huan Liu, Jinsong Su, and Degen Huang. 2022. Adaptive token-level cross-lingual feature mixing for multilingual neural

---

machine translation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 10097-101 13, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Fan-Keng Sun, Cheng-Hao Ho, and Hung- Yi Lee. 2019. Lamol: Language modeling for lifelong language learning. In International Conference on Learning Representations.
Simeng Sun, Maha Elbayad, Anna Sun, and James Cross. 2023, Efficiently upgrading multilingual ma- chine translation models support more languages. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages
Zihan Liu, Genta Indra Winata, and Pascale Fung, 2021, Continual mixed-language pre-training for extremely low-resource neural machine translation. In Find- ings of the Association for Computational Linguis- fics: ACL-IJCNLP 2021, pages 2706-2718, Online. Association for Computational Linguisties.
Dubrovnik, Croatia. Association for Computational Linguistics.
Migh-TTang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spo- ken language domains. In Proceedings of the 12th International Workshop on Spoken Language Trans- lation: Evaluation Campaign, pages 76-79.
Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. Learning sparse sharing architectures for multiple tasks. In Proceedings of the AAAT conference on artificial intelligence, volume 34, pages 8936 8943.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Se- ungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eu- njoon Cho. Pascale Fung, and Zhiguang Wang. 2021. Continual learning in task-oriented dialogue systems. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7452-7467, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Yuqing Tang, Chau Tran. Xian Li, Peng-Jen Chen, Na- man Goyal. Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual translation with exten- sible multilingual pretraining and finetuning. CoRR, Abs/2008.00401.
Brian Thompson, Jeremy Gwinnup. Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Volume 1 (Long and Short Papers), pages 2062-2068. Minneapolis, Min- nesota. Association for Computational Linguistics.
Antonio Valerio Miceli Barone, Barry Haddow, UIrich Germann, and Rico Sennrich. 2017. Regularization techniques for ine-tuning in neural machine trans- lation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1 44991144, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Proceedings of the 31st International Conference Neural Information Processing Sys- tems, pages 6000-6010.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion Computational Linguistics (Demonstrations) pages 48-53, Minneapolis, Minnesota. Association for Computational L Linguistics.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lift- ing, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics. pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Matt Post. 2018 A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1628- 1639, Online. Association for Computational Linguis- tics.
Abigail See, Minh-Thang Luong, and Christopher D. Manning. 2016. Compression of neural machine translation models via pruning. In Proceedings of the 20th SIGNLL Conference on Computational Natu- ral Language Learning, pages 291-301, Berlin, Ger- many. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Yang Zhao, Junnan Zhu, Lu Xiang. Jiajun Zhang Yu Zhou, Feifei Zhai, and Chengqing Zong. 2022. Life-long learning for multilingual neural machine translation with knowledge distillation.
Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingx- uan Wang, and Lei Li. 2021. Counter-intefference

---

adapter for multilingual machine translation. In Find- ings of the Association for Computational Linguis- rics: EMNLP 2021, pages 2812-2823, Punta Cana, Domimican Republic. Association for Computational Linguistics.

---

A Dataset Details
Code
Language
Genus
Script
Order
Spanish
Romance
Latin
SVO
A.1
WMT-9
nu
Russian
Slavie
Cyrillic
SVO
Original Language Pairs. select 8 English- centric language pairs of different data sizes from the WMT benchmark to train the original MNMT model that covers 16 translation directions and languages. Specifically, the original language pairs cover languages with different amounts of training data: high-resource (> (>10M): Russian, Spanish and Japanese; medium-ressouce (1M-10M): Finnish and Pashto; and low-resource (<IM);: Lithuanian, Hindi and Hausa. Moreover, those languages be- long to different language branches and vary in several linguistic characteristics such writing sys- tem and dominant word order, which increases the language diversity in the original MNMT model.
Japanese
Japanese
Kanji
SOV
Finnish
Finnic
Latin
SVO
Pashto
Iranian
Arahic
SOV
Lithuanian
Baltic
Latin
SVO
Hindi
Indic
Devanagari
SOV
ha
Hausa
West Chadic
Latin
SVO
ro
Romanian Romance
Latin
SVO
dc German
Germanic
Latin
SVO
Tamil
Dravidian
Tamil
SOV
Irish
Celtic
Latin
VSO
Table 8: The characteristics of languages in experi- ment. The top half part represents the original languages while the bottom half part represents the incremental languages.
Language Enhancement Task. In this transla- tion scenario, we aim to enhance the translation performance of mBART50-mn on German< Frrmanc French (det and Chinese Vietnamese (zh<>vi) direc- tions. The training data are from WMT22 and CCMatrix for de++fr and zht+vi, respectively, We also use the validation/test sets from Flores (Goyal et al., 2022) for checkpoint selection and evalua- tion.
Incremental Language Pairs. We select another four language pairs for incremental learning con- sidering that the incremental languages are usually non-high resource in the real-world translation sce- nario. The four languages include: Romanian (low- resource), which is highly related the original language Spanish with the same language branch Romance, and shares the same script and word order; German condiuum-esooucee which is sim- ilar to the original languages in script and word order, but comes from different language branch; Tamil (low-resource), which is not related to any original languages and has different scripts; Irish (low-resource), has completely different word or- der from all the original languages despite using the shared script of Latin characters.
B Implementation Details
Vocabulary Extension. For the experiments on WMT-9, apply byte pair encoding (BPE) al- gorithm (Sennrich al., 2016) using Sentence- Piece (Kudo and Richardson, 2018)6 to preprocess the sentences of original language pairs with a joint multilingual vocabulary of 64K. For each incre- mental language, we train distinct vocabulary of 32K and combine it with the original multilingual dictionary by removing the overlapped tokens. We utilize the combined vocabulary for incremental learning to avoid the out-of-vocabuee As all the methods in our experiments are built on the same extended vocabulary, we do not take these extra parameters into account when comparing the newly added parameters across different methods in this paper.
The characteristics of each language are summa- rized in Table 8 and the detailed statistics of the training dataset are shown in Table 9.
A.2 mBART50
We perform incremental learning based on the pre- trained mBART50-m model for two translation scenarios. The detailed statistics of the training dataset are shown in Table 10.
problem.
Language Adaptation Task. In this transla- tion scenario, we adapt the model to support the Englishe+Greek (ent>el) and Englishh+Slovak (ent>>sk) translation directions following Gu al. (2022). We use the data from OPUS-100 (Zhang al., 2020) for model training and the valida- tion/test sets from Flores (Goyal et al., 2022) for checkpoint selection and model evaluation.
For the experiments based on mBART50-nn. all the sentences are preprocessed using the Sentence- Piece model provided by XLM-R (Conneau et al., 2020). For the language adaptation task, insert
hhttps: htttp:://ww.ssatttee org/wmt22
cnttps://datas statmt. weattt/oo//cccrre/eee
htttps: thub. cctm//oogle/senteecppiee

---

Data Source
Samples
Language Pair
Train
Valid
Test
Train
Valid
Test
en-ru
WMT19
WMTL3
WMT19
38,492, 126
3000
2000
en-es
WMT13
WMT13
3MT13
15,182,374
3003
3000
en-ja
WMT21
WMT20
WMT21
18.012.559
993
1005
en-fi
WMT19
WMT18
WMT19
6,587,448
3000
1996
en-ps
WMT20
WMT20
WMT20
1,155,944
2698
2719
en-lt
WMT19
WMT19
WMT19
635,146
3000
1000
en-hi
WMT14
WMT14
WMT14
313,748
520
2507
en-ha
WMT21
WMT21
WMT21
752,357
2000
997
en-ro
WMT16
WMT16
WMT16
610,320
1999
1999
en-de
WMT14
WMT13
WMT14
4,508, 78.5
3000
3003
en-ta
WMT20
WMT20
WMT20
660,818
1989
997
en-ga
OPUS-100
Flores
Flores
289,524
997
1012
Table The statistics of train, valid and test data for the original language pairs and incremental language pairs in WMT-9.
Data Source
# Samples
Language Pair
Train
Valid
Test
Train
Valid
Test
en-el
OPUS-
Flores
Flores
IM
1012
en-sk
OPUS-100
Flores
Flores
1M
997
1012
de-fr WMT22
Flores
Flores
18.1
997
1012
zh-vi
CCMatrix
Flores
Flores
8.05M
997
1012
Table The statisties of train, valid and test data used continual learning based on mBART50-nn. The top half part represents the incremental language pairs in the language adaptation task while the bottom half part represents the incremental language pairs in the language enhancement task.
language-specific embedding layer (LSE) the original mBART50-nn model to implement the mBART50-nn-L. model (Berard, 2021) as the baseline model following Gu et al. (2022). For the language enhancement task, we do not need to extend the embedding layer since the languages involved are supported by mBART50-nn.
models on 8 Nvidia RTX A6000 GPUs with batch of 4096 and adopt the early stop (patience 10) strategy. For the task-specific fine-tuning in the importance evaluation stage, we fix all the layer normalization layers and the embedding layers the original MNMT model since we find that it brings better results. For model pruning, we also do not prune the layer normalization layers and embedding layers in the encoder and decoder.
Model settings for experiments on WMT-9 dataset. All the models follow the configuration of Transformer-Big (Vaswani et al., 2017). which consists of 6 stacked encoder/decoder layers and 16 attention heads. The model size dmodel and feed-forward dimension din are set 1024 and 4096, respectively. For model training, we use the temperature-based sampling strategy to bal- ance the training data distribution with a temper- ature of T - 5 (Arivazhagan et al., 2019), and use share-all coure-all-embeed -lbeeeddd-m in Fairseq to save pa- rameters. All the model parameters are optimized using Adam optimizer (Kingma and Ba, 2014) (81 0.9, 333 0.98) with label smoothing of 0.1. The learning rate is scheduled as Vaswani et (2017) with a warm-up step of 4000 and a peak learning rate of 0.0005. We train all the MNMT
Model settings for experiments based mBART50-nn. The mBART500nn model con- sists of 12 stacked encoder/decoder layers and 16 attention heads. The model size dmodel and feed- forward dimension dit are set to 1024 and 4096, respectively. Other model configurations are the same as those in the experiments on WMT-9.
Pruning ratio settings. Based on the results in Figure 1, in order to retain the performance on the original tasks, the hyper-paramette a% is set 25% in all the experiments. For the BVP method, tune the sarem--ar 6% in the range of B = (5%, 25%, 50%, 75%] on each incremental language pair. The final pruning ratios are summa- rized in Table 12. For the PTE, UMP and UVP

---

ID
Method
Embedding (0g)
Parameters (0p)
encoro
ent-dde
enc-tt
entoga
AVG
Handom-Randows
Random
Random
0.13
0.04
0,05
0.04
0.06
Transfer-Random
Transfer
Random
0.23
0.12
0.05
0.06
Random-Transee
Random
Transfer
29.16
26.27
13.16
18.48
21.76
(4)
Copy-Copy
Copy
Copy
29.25
25.97
13.27
18.13
21.68
Transfer
Transfer
29.29
26.04
13.31
18.36
21.75
Table 11: The performance on the incremental language pairs under different initialization methods.
Trining Time (x 10 S)
24
40
24
30
18
30
18
22.5
12
20
12
15
10
7.5
en-de
en-ta
en-ga
random-random transfer-random
random-transfer
copy-copy
transfer-transfer
Figure 4: The training time of the BVP model with different parameter initialization methods in incremental learning for MNMT.
Language Pair
5%
25%
50%
75%
ods:
en<>rro
1.2
6.1
12.3
18.6
the newly added parameters or em- bedding layer are randomly initialized with Gaussian distribution.
Random:
ent+de
1.2
6.0
12.2
18.5
ent->aa
1.2
6.1
12.3
18.6
entgga
1.3
6.4
12.6
18.8
Copy: the newly added parameters are ini- tialized with their counterparts in the original MNMT model and the extended embeddings are initialized with the embeddings of the orig- inal MNMT model by sampling.
Table 12: The pruning ratios p% each language pair along with different hyper-parameter 1%. The hyper- parameter a% is set to 25%.
methods, we keep the same as the pruning ratio of the BVP method for fair comparisons. In our experiments, the hyper- parameter 5% is set to 50% when adapting the orig- inal MNMT model to multiple language pairs si- multaneously (Table 2 in Section 4.3). Otherwise, 5% is set to 25% in other experiments.
5% the Transfer:
the newly added parameters and the extended embeddings are initialized with their counterparts in the fine-tuned task-specific model.
The results summarized in Table 11. We find that the translation results are more sensitive to the initialization method for the new parameters (0p). compared with the embedding layer (0g). Specifi- cally, when the new parameters (fp) are initialized randomly, it fails to adapt the model to the incre- mental language pairs. This result suggests that the representation gap between the original parameters and newly added parameters cannot be climinated during incremental training. By contrast, the copy initialization and transfer initialization methods do not suffer from this problem since the parameters have been well trained to fit the model advance. The initialization method for the embedding layer
More Results
Effects of different initialization methods
As illustrated in Section 3.3, when adapting to the incremental tasks, we add new parameters to the pruned model to expand the model to its original size and extend the embedding layer of the original MNMT model to avoid the out-ol-vocabulary prob- lem. In this experiment, we investigate the effects of different initialization methods on the transla- tion performance the incremental language pairs. Specifically, we study the three initialization meth-

---

does not apparently affect the translation perfor- mance on the incremental language pairs.
We also study the training time of the model with different initialization methods. As shown in Figure 4, the transfer initialization method can re- markably reduce the training time of incremental learning, which is more efficient than other meth- ods. Based on the above results, we adopt the transfer method to initialize the embedding layer (0p) and new parameters (fg) in the experiments.


Reviews:
[{"review_id": "3Bs2dBsBlI", "reviewer": "Reviewer_5LWe", "summary": "", "questions": "A. How do you balance the parameters between the old tasks and new tasks, if they share very similar features? Such as the transition tasks for close languages. In this situation, the weights of the fine-tuning model on the new tasks may share highly similar to the original tasks.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! We address your concerns below:\n\n**Regarding the degradation on original language pairs**\n> 1. The original tasks’ performance degenerates to some extent and underperforms the baseline of the Adapter, which indicates the negative influence of removing some parts of the original networks.\n\nOur method does encounter a slight performance decline compared with the Adapter method, because our method utilizes a small group of parameters in the original MNMT model to learn the knowledge of the new language pairs, while the Adapter method fixes the whole original MNMT model. However, unlike the Adapter mothed, our method does not require extra parameters for incremental learning. Moreover, the performance decline is much smaller than that of the previous replay-based and regularization-based methods, which is relatively acceptable. The performance decline can also be alleviated by reducing the pruning ratio or increasing the model capacity, as shown in Figure 1.\n\n**Regarding the limited model capacity**\n\n>2. The proposed method may encounter a limitation if the users continuously add new languages because of the limited model capacity.\n\nThe capacity bottleneck is a common problem in MNMT when many languages are accommodated in a single model. In our case, there are two possible solutions to alleviate that problem. One solution is to directly enlarge the model capacity. As shown in Figure 1, although mBART50-nn covers much more languages than our WMT-9 model (50 vs. 9), it suffers from smaller performance degradation since it has a larger model capacity. The results show that there are more unimportant parameters in the mBART50-nn model that can be used to adapt to more new language pairs. The other solution is to vary the pruning ratios for different incremental language pairs. As shown in Figure 3(b), the BLEU score on incremental language pairs does not increase when the pruning ratio is larger than 12%. Therefore, we can allocate different pruning ratios to different language pairs in order to accommodate more new language pairs.\n\n**Regarding the questions**\n\n> A. How do you balance the parameters between the old tasks and new tasks, if they share very similar features? Such as the transition tasks for close languages. In this situation, the weights of the fine-tuning model on the new tasks may share highly similar to the original tasks.\n\nFor the cases where the new languages are very similar to the original languages, we can utilize a smaller pruning ratio to retrain more common features between original and new languages. Although similar languages share more common features, Lin et al. [1] show that they still differ from each other on their language specific sub-networks. In our experiment on WMT-9 dataset, Spanish (es) is one of the original languages and we adapt the original MNMT model to Romanian (ro) and German (de), respectively. The three languages are similar as they are all from the Indo-European language family. The translation performance on en-es is not apparently affected when adapting to en-ro and en-de (The performance on en-es declines <0.1 BLEU). Moreover, we find that the overlap in the pruned parameters of en-ro and en-de only accounts for 26.8%, which means that 73.2% of the pruned parameters are non-overlapping, indicating there is a big difference between en-ro and en-de language pairs. Our fine-tuning model can capture those difference and yield different pruned parameters. Given the above observations, our method can deal with the case where the old tasks and new tasks share similar features.\n\n[1] Lin et al. Learning Language Specific Sub-network for Multilingual Machine Translation. In ACL2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to follow.\n\n2. The proposed method the catastrophic forgetting problem to some extent as indicated by the experiment results, through freezing the original parameters.", "reasons_to_reject": "1. The original tasks’ performance degenerates to some extent and underperforms the baseline of the Adapter, which indicates the negative influence of removing some parts of the original networks.\n\n2. The proposed method may encounter a limitation if the users continuously add new languages because of the limited model capacity.", "questions_to_authors": "A. How do you balance the parameters between the old tasks and new tasks, if they share very similar features? Such as the transition tasks for close languages. In this situation, the weights of the fine-tuning model on the new tasks may share highly similar to the original tasks.", "paper_topic_and_main_contributions": "The research problem in this paper is the alleviate the catastrophic forgetting problem during learning the new translation tasks for new languages for the MNMT model. To do that, the authors propose a method named dual importance-based model division to divide the model parameters for old tasks and new tasks respectively. Experiment results demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none", "Typos_Grammar_Style_and_Presentation_Improvements": "none"}, "extra_text": {}}, {"review_id": "pa2xfGyY0z", "reviewer": "Reviewer_Tywa", "summary": "", "questions": "- In some instances (Table 1) the performance on original LP's degrade with significant bleu scores (>1). Can the authors rephrase some statements or clearly state the cases where the proposed approach degrade for pre-existing LP's?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! We address your concerns below:\n\n**Regarding additional parameters**\n\n> 1. Model expansion step seems to defeat the original motivation of the work (i.e. not adding additional parameters when tuning a model to new translation direction)\n\nIn the model expansion step, we introduce two newly added parameters: new parameter $\\theta_P$ and new embedding $\\theta_E$. The new parameter $\\theta_P$ is used to expand the pruned model to its original size so that model parameters are not increased. The new embedding $\\theta_E$ is used only when the incremental languages are not supported by the original MNMT model. In this case, it increases the number of model parameters in the embedding layer. In our experiments, we add the new embedding $\\theta_E$ for all the baselines by default on WMT-9 dataset to avoid the out-of-vocabulary problem. Therefore, we do not take the embeddings into account when discussing the additional parameters. We will make it clear in the updated version to avoid misunderstanding.\n\n**Regarding the missing experiments for unseen language pairs**\n\n> 2. Missing experiments for LP's unseen in the parent model, this has been regarded as one of the mai challenge previous works have been trying to address. \n\nWe discuss three translation scenarios of unseen language pairs in our experiments. First, adapting the parent model to new languages. In this scenario, we adapt the WMT-9 model to four new languages (ro, de, ta, ga) and perform translation between those four languages and English (en) (See Table 1). Second, based on the first scenario, we further perform translation from non-English languages in the parent model to the new languages (See Table 3). In this scenario, the language pairs are also unseen in the parent model (e.g. es-ro). Third, adding language pairs that the parent model is not trained on their supervised data (See Table 6). For example, de and fr are already supported by mBART50-nn, but the mBART50-nn model has not been trained on de-fr data and it performs translation between de and fr in zero-shot fashion. Therefore, the de-fr language pair also belongs to unseen language pairs in the parent model. \n\n**Regarding the questions**\n\n> 1. In some instances (Table 1) the performance on original LP's degrade with significant bleu scores (>1). Can the authors rephrase some statements or clearly state the cases where the proposed approach degrade for pre-existing LP's?\n\nWe would like to explain why the performance on original language pairs degrades with a significant BLEU in Table 1. When adapting the original MNMT model to new language pairs, the L2-Reg, EWC and LFR methods fine-tune all the parameters in the model. Although they utilize regularizer or hard constraint to let the parameters stay close to their original values, the change of all parameters still results in the performance decline to some extent, especially when the new languages are similar to the original languages. For example, when adapting to en-ro language pair, the original language pair en$\\rightarrow$es declines over 10 BLEU which is remarkably larger than other language pairs, leading to over 1 average BLEU decline on the en-xx direction in the original translation task. For our method, we prune a small group of parameters in the original MNMT model based on the parameter importance. Similar to previous studies, the pruning step will naturally result in some performance decline. However, the performance loss for each original language pair is relatively smaller (at most 0.3 BLEU).\n\n**Missing references**\n\nThanks for pointing out these related studies. We will include a discussion of these studies in our updated version based on your feedback.\n\n**Typos**\n\nThanks for catching the typos! The word “utilize” in L103 should be removed. We have fixed it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Incremental model tuning strategy doesn't require original training data, which could have been a constraint for leveraging most pre-trained MNMT models. \n- Paper is well written, authors did a great job making the work  easily understandable, along with the illustration provided. \n- Extensive experiment and discussions.", "reasons_to_reject": "- Model expansion step seems to defeat the original motivation of the work (i.e. not adding additional parameters when tuning a model to new translation direction)\n- Missing experiments for LP's unseen in the parent model, this has been regarded as one of the mai challenge previous works have been trying to address. See comments for the rest. ", "questions_to_authors": "- In some instances (Table 1) the performance on original LP's degrade with significant bleu scores (>1). Can the authors rephrase some statements or clearly state the cases where the proposed approach degrade for pre-existing LP's?", "paper_topic_and_main_contributions": "This work propose an multilingual NMT model fine-tuning and parameter pruning approaches, with the goal to improve new language pair (LP) translation while maintaining quality for pre-existing LP's. Different from other works, authors propose model parameter weighting or importance evaluation strategies to prune parameters. Three different pruning approaches are discussed by evaluating parameters importance for pre-existing LP's and new LP. Experimental results show, the proposed approach to improve translation for new LP's.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L103: \"Finally, we utilize expand the pruned model ...\"? remove utilize ?"}, "extra_text": {"Missing_References": "- Missing related works utilizing dynamic adaptation of MNMT models:\n -Garcia et al. Towards continual learning for multilingual machine translation via vocabulary substitution\n -Lakew et al., Transfer learning in multilingual neural machine translation with dynamic vocabulary"}}, {"review_id": "DBkpCqtb4S", "reviewer": "Reviewer_5Yj3", "summary": "", "questions": "see above.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address each of your concerns below:\n\n**Regarding the results of Fine-Tuning method**\n\n> 1. The experimental results of fine-tuning presented in this paper are not convincing. For instance, in Table 1, the performance of the Original Language Pairs after fine-tuning decreased from 23.21 to 0.50. Similar phenomena can also be observed in Tables 2, 5, and 6. A BLEU score of 0.5 indicates that the model no longer has knowledge of this language. Such scores are difficult to believe. Furthermore, fine-tuning is an important step in identifying important parameters.\n\nWe would like to clarify the translation results of the Fine-Tuning method. As described in line 949 in Appendix B, **the Fine-Tuning method trains the incremental language pairs based on the pre-trained MNMT model only with the training data of new language pairs (the training data of original language pairs are not available)**. In this setting, the model will only focus on learning the knowledge of new language pairs while forgetting the knowledge of original language pairs, which is known as the catastrophic forgetting problem. Therefore, the model can hardly output correct target words or fluent sentences when translating across original language pairs, leading to the remarkable performance decline (in Tables 1, 2, 5, 6). Such serious performance decline is also reported in some recent studies (Gu et al. [1], Huang et al. [2] and Zhao et al. [3]). Take the results in Table 5 for example, our results on original language pairs are similar to those reported by Gu et al. [1] under the same experimental settings, and we put the comparisons blow. Moreover, Huang et al. [2] also reported that the performance on original language pairs drops to 1.14 BLEU from 15.14 BLEU for en-xx direction (-14.00 BLEU) and 1.68 BLEU from 20.19 BLEU for xx-en direction (-18.51 BLEU). Given the above comparisons, the results of our Fine-Tuning method are reasonable. The Fine-tuning method is actually an important step in identifying important parameters for new language pairs and it will not be affected by the poor performance on the original language pairs.\n\n|Method                   |    xx$\\rightarrow$en    |    en$\\rightarrow$xx    |\n|:-                             |      -:                            |                -:                   |\n|mBART50-nn+LSE (Gu et al. [1])  |   25.83   |  21.48 |\n|Fine-Tuning (Gu et al. [1])     |    18.37  |   1.15  |\n|mBART50-nn+LSE (Ours reimplemetation)      |   26.73      |   21.66   | \n|Fine-Tuning  (Our reimplemetation)       |     20.40    |     1.88   | \n\n\n**Further analysis on the importance of the parameters**\n\n> 2.1. The range of change for different parameters is related to their location. For example, the change in high-level parameters may be greater than that of embedding parameters.\n\nWe study the parameter change value in the en-ro adaptation scenario to show more analysis about our method. We fine-tune the original MNMT model on en-ro training data and calculate the average value change of each weight matrix. As you say, the range of change for different parameters is actually related to their location and the change in high-level parameters tends to be greater than that of the low-level parameters. However, the changes for all the weight matrices are mostly in the range of (0.015, 0.025). Similar results can also be found in language adaptation task (e.g. en-ta, en-de). These results show that all the weight matrices contribute to the new language pair, so we consider all the weight matrices for weight pruning. Despite that the changes in some weight matrices are relatively larger, we could not prune all the parameters in those weight matrices since those parameters may also be important to the original language pairs. Otherwise, the performance on original language pairs will decline remarkably. To this end, we only prune the parameters that are important to the new tasks while not important to the old tasks to make a compromise between the old and new tasks. Moreover, we can increase the hyper-parameter $b$ in our BVP method to add more parameters to the pruning candidate $\\mathcal{N}$ to cover more parameters with larger value changes. In fact, we find that the final pruning ratios of each weight matrix are different from each other in our BVP method, which also indicates that the effects of various parameters are related to their location.\n\n> 2.2. Selecting parameters based on their absolute value may lead to over-adjustment. In an extreme case where no adjustment is needed, selecting parameters based on their absolute value may still result in some being adjusted. How can we determine which parameters are truly important for the new task and have minimal impact on the original model?\n\nAs the analysis on en-ro shows, all the weight matrices contribute to the new language pairs. Therefore, the adjustment in each weight matrix is necessary. We agree that some parameters may not need to be adjusted since the performance on new language pairs will not be further improved when the pruning ratio is larger than 12% (See Figure 3(b)). We try to alleviate this problem by carefully tuning the pruning ratio. Previous studies (Gu et al. [4] and Xie et al. [5]) also try other methods such as Taylor Expansion for model pruning, however, the performance is not always better than the absolute value method.\n\n**Regarding the pre-trained model selection**\n\n> 3. This paper tries to extend an existing MNMT model to support new languages and improve the translation of some language pairs. There are many pre-trained MNMT models available, such as M2M-100, NLLB, and BLOOM. However, this paper trains a model from scratch or uses mBART50-nn (2020). The rationale for this selection is unclear.\n\nWe start our experiments from a self-trained MNMT model since we can select the languages during the pre-training and incremental learning stages, which makes it easier for us to study some characteristics that are possible to affect the incremental learning results in MNMT such as scripts, dominant order, data size and language relatedness. Our results show that the translation performance on incremental language pairs is actually related to those characteristics. We select mBART50-nn as the pre-trained model to perform incremental learning in the real-world scenario. Adapting the original model to new languages is regarded as one important task for the incremental learning in MNMT. Although the newly released pre-trained MNMT models such as M2M100 and NLLB are also available, it is difficult to further add new languages to them because they have covered over 100 languages. Moreover, our experiments on mBART50-nn can also be implemented in other pre-trained models such as M2M100. Given the above, we implement our method based on the mBART50-nn and perform two incremental tasks (language adaptation task and language enhancement task) that are the most common cases in the real world.\nDespite that, we perform incremental learning of de-fr and zh-vi language pairs based on the M2M100 (1.2B) model as you suggested. The results are as follows. We only report the performance on en-xx language pair for original tasks due to the time limitation. Our BVP model still consistently outperforms the vanilla M2M100 model on new language pairs without causing obvious performance decline on the original language pairs. The results show that our method generalizes to other pre-trained models.\n\n|Method    |    xx-en  |\ten-xx   |    fr-de    |   de-fr   |   zh-vi\t   |    vi-zh    |\n|  :-           |    -:         |      -:          |     -:         |    -:       |     -:        |      -:       |\n|M2M100(1.2B)\t|  23.54 |\t16.99    |   28.19 |    35.42   |   28.42 |\t27.66 |\n|Ours(BVP)           |   23.47  |\t16.96 |\t29.36  |   35.94   |   29.68 |\t30.40 |\n\n\n[1] Gu et al. Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions. In EMNLP2022.\n\n[2] Huang et al. Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation. In ACL2023.\n\n[3] Zhao et al. Life-long Learning for Multilingual Neural Machine Translation with Knowledge Distillation\n\n[4] Gu et al. Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation. In NAACL2021.\n\n[5] Xie et al. Importance-based Neuron Allocation for Multilingual Neural Machine Translation. In ACL2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces an algorithm designed to address the problem of catastrophic forgetting in language translation tasks. \n\nThe algorithm is straightforward and easy to implement, making it a practical solution for mitigating this issue. By following the steps outlined in the proposed approach, it is possible to improve the performance of new tasks while minimizing the impact of catastrophic forgetting.\n\n", "reasons_to_reject": "1. **The experimental results of fine-tuning presented in this paper are not convincing.** For instance, in Table 1, the performance of the Original Language Pairs after fine-tuning decreased from 23.21 to 0.50. Similar phenomena can also be observed in Tables 2, 5, and 6. A BLEU score of 0.5 indicates that the model no longer has knowledge of this language. Such scores are difficult to believe.  **Furthermore, fine-tuning is an important step in identifying important parameters.**\n\n\n2. **Further analysis is needed to determine the importance of various parameters.** The calculation of importance is based on the absolute change value, which raises several concerns:\n+ The range of change for different parameters is related to their location. For example, the change in high-level parameters may be greater than that of embedding parameters.\n+ Selecting parameters based on their absolute value may lead to over-adjustment. In an extreme case where no adjustment is needed, selecting parameters based on their absolute value may still result in some being adjusted. How can we determine which parameters are truly important for the new task and have minimal impact on the original model?\n\n3. This paper tries to extend an existing MNMT model to support new languages and improve the translation of some language pairs.  There are many pre-trained MNMT models available, such as M2M-100, NLLB, and BLOOM. However, this paper trains a model from scratch or uses mBART50-nn (2020). **The rationale for this selection is unclear.**\n", "questions_to_authors": "see above.", "paper_topic_and_main_contributions": "This paper presents an algorithm to address the issue of catastrophic forgetting in language translation tasks. The proposed approach involves a three-step process that balances the performance of the model on both original and new language pairs.\n\nThe first step involves fine-tuning a pre-trained model to identify parameters that are essential for new tasks but have minimal impact on the original tasks. This results in a pruned model. In the second step, the pruned model is expanded with additional parameters, which are then fine-tuned using new training data. Experimental results demonstrate that this approach is effective in retaining the performance of the model on original tasks while also improving its performance on new translation tasks.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ovmrc3EEIE", "reviewer": "Reviewer_eCJC", "summary": "", "questions": "- Is there any ablation study about dual-importance? (e.g., a comparison with single-step pruning?)\n- While the paper provided results about sequential training of two new languages, are there any results with sequential training involving more languages (e.g., 10)? Does BVP still outperform conventional continual learning methods such as EWC?\n- In addition to the adaptor, is there any comparison with recently proposed parameter-efficient fine-tuning techniques?\n- Is there any reason for freezing embedding and layernorm parameters? Adding detailed explanations would help for a deeper understanding.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback! We address your questions below:\n\n**Q1: Ablation study about dual-importance**\n\n> 1. Is there any ablation study about dual-importance? (e.g., a comparison with single-step pruning?)\n\nWe perform the ablation study on the dual-importance pruning strategy as you suggested. The pruning ratio for each method is 6% and the results are shown below. When pruning only based on the old task importance, the model can maintain its performance on the original language pairs. However, it is weak in the ability of capturing the specific knowledge of new language pairs, leading to inferior performance to our BVP model. In contrast, when pruning only based on the new task importance, although the model performs slightly better than BVP on the new language pairs, it fails on the original language pairs. We find that most (>98%) pruned parameters in this method are also important to the old language pairs. Therefore, the performance on original language pairs decreases apparently if those parameters are pruned. Our BVP model achieves the best overall performance across all language pairs, showing the necessity of the dual-importance.\n\n\n| Method                                |  en-xx  | xx-en   |  en-ro  |  en-de  |   en-ta    |   en-ga |\n| :-                                          |     -:      |  -:        |  -:        |  -:          |  -:          |  -:         |  \n| Only old task importance    |   18.87  |  23.04  | 28.73 |  25.37  |   13.08   |  19.51 |\n|Only new task importance   |    0.03  |   0.04  |   29.36 |  26.98  |   13.07  |   20.96 |\nOurs (BVP)                          |   18.86  | 23.10   | 29.29    | 26.04  |   13.31  |   20.89 |\n\n**Q2: Sequential training for more languages**\n\n> 2. While the paper provided results about sequential training of two new languages, are there any results with sequential training involving more languages (e.g., 10)? Does BVP still outperform conventional continual learning methods such as EWC?\n\nWe further sequentially adapt the model to en-de and en-ga based on the experiment in Table 4 (Section 5.3). Due to the time limitation, we cannot adapt the model to more language pairs. We compare our method with EWC and the results are as follows. As is shown, our BVP model can still balance the performance between original and new language pairs, and consistently outperforms the EWC model.\n\nResults after adapting to en-de:\n\n| Method       |     en-xx   |    en-ro    |    en-ta     |    en-de   |\n| :-                 |     -:          |  -:            |  -:             |  -:           |\n|Scratch         |     21.06   |    27.36   |   12.18     |    27.46  |\n|EWC            |     18.21   |     25.59  |      11.65   |     17.15  |\n|Ours(BVP)   |      20.77   |    28.87   |     12.72   |     25.90   |\n\nResults after adapting to en-ga:\n| Method       |     en-xx   |    en-ro    |    en-ta     |    en-de   |    en-ga  |\n| :-                 |     -:          |  -:            |  -:             |  -:           |  -:           |\n|Scratch       |     21.06    |    27.36   |     12.18   |     27.46  |     12.52  |\n|EWC           |     17.72    |    24.65   |    11.13    |     16.22  |     11.11   |\n|Ours(BVP)  |     20.67   |     28.83  |     12.69   |      25.27 |      11.87   |\n\n**Q3: More comparisons with other parameter-efficient fine-tuning method**\n\n> 3. In addition to the adaptor, is there any comparison with recently proposed parameter-efficient fine-tuning techniques?\n\nWe evaluate some recent parameter-efficient fine-tuning (PEFT) methods including prompt-tuning [1], prefix-tuning [2] and LoRA [3] on en-ro language pair. The results are as follows. Similar to the Adapter model, those PEFT methods do not degrade the performance on original language pairs since they employ additional parameters for en-ro adaptation. However, they yield inferior performance to the Adapter model on the new language pairs (en-ro) and also underperform our BVP model.\n\n|Method      |   en$\\rightarrow$xx    |     xx$\\rightarrow$en   |    en$\\rightarrow$ro   |    ro$\\rightarrow$en  | \n| :-                |     -:          |  -:            |  -:             |  -:           | \n|Prompt       |  18.90    |    23.21     |       19.85      |    30.20   |   \n|Prefix          |   18.90   |     23.21      |      20.24     |     30.48   |   \n|LoRA          |   18.90   |     23.21      |      22.95      |    31.86   |   \n|Adapter       |   18.90   |    23.21     |        23.89      |    33.36   |   \n|Ours(BVP)  |    18.86   |     23.10     |       24.60     |     33.98   |   \n\n**Q4: Explanations for freezing embedding and layernorm parameters**\n\n> 4. Is there any reason for freezing embedding and layernorm parameters? Adding detailed explanations would help for a deeper understanding.\n\nWe freeze the embedding and layernorm parameters during the training stage since we find that it benefits the task-specific finetuning in the parameter importance evaluation stage and yields better final translation performance on new language pairs. Moreover, Gu et al. [4] find that it helps to alleviate the forgetting problem on the original language pairs. We will make it clear in the updated version as you suggested.\n\n\n[1] Lester et al. The power of scale for parameter-efficient prompt tuning. In EMNLP2021.\n\n[2] Li and Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL2021.\n\n[3] Hu et al. LoRA: Low-rank adaptation of large language models.\n\n[4] Gu et al. Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions. In EMNLP2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is an interesting work on bridging pruning and incremental learning.\n- The effectiveness of the proposed method is supported with thorough experiments and analysis.\n- Their method provides promising translation performance compared to previous incremental learning techniques.", "reasons_to_reject": "- Their method requires complex training pipelines, necessitating an additional training phase for model pruning.", "questions_to_authors": "- Is there any ablation study about dual-importance? (e.g., a comparison with single-step pruning?)\n- While the paper provided results about sequential training of two new languages, are there any results with sequential training involving more languages (e.g., 10)? Does BVP still outperform conventional continual learning methods such as EWC?\n- In addition to the adaptor, is there any comparison with recently proposed parameter-efficient fine-tuning techniques?\n- Is there any reason for freezing embedding and layernorm parameters? Adding detailed explanations would help for a deeper understanding.", "paper_topic_and_main_contributions": "This paper studies the incremental learning of new languages in multilingual NMT (MNMT). Specifically, to effectively learn new languages while preserving translation performance on original languages, it proposes a dual importance-based parameter division method. The experimental results exhibit competetive BLEU in both new and original languages, across various translation directions and experimental settings (e.g., adding multiple languages or sequentially adding languages).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of adding language pairs to multilingual MT models. It proposes
a method that detectes parameters in the base model which are both unneeded for the original
languages, and suitable for fine-tuning in the incremental languages. Experiments show that the
method achieves bleu improvements on the incremental languages, whilst mostly preserving
performance on the original languages.

The reviewers are in broad agreement about the soundness of the paper, with two 3s and two 4s. I cannot
see any substantive issues raised by the reviewers which would justify the 3 score. 
he issue that I would raise is that
this paper's claims and conclusions depend heavily on bleu scores. I would expect to see additional
results on a trainable neural metric (eg comet) to validate the results.

There is some disagreement about the excitement, with reviewer 5Yj3 giving the paper a 2. However their
first 2 reasons to reject seem to be based on misunderstandings, and the third reason (the choice
of base model) is addressed in the rebuttal. Both eCJC and Tywa rate the excitement at 4 and liked
the interesting method, and the thorough experiments and analysis.

Overall, an interesting paper, with strong experiments, except for their over-reliance on bleu scores.
Decision: Accept-Main