Learn and Consolidate: Continual Adaptation for Zero-Shot and Multilingual Neural Machine Translation
Authors: Kaiyu Huang
Peng Li
Junpeng Liu
Maosong Sun
Yang Liu
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual Neural Machine Translation
Zero-shot Machine Translation
Continual Learning
TL;DR: We effectively promote both supervised and zero-shot performance for multilingual neural machine translation model using newly available data while maintaining original stronger performance on English-Centric directions for continual adaptation.

Abstract: Although existing multilingual neural machine translation (MNMT) models have demonstrated remarkable performance to handle multiple translation directions in a single model and achieved zero-shot translation between language pairs unseen in training, they still suffer from relatively poor translation qualities for some language pairs. A practical scenario is that how to continually update MNMT models for both supervised and zero-shot translations when limited new data arrives. To this end, we propose a two-stage approach that encourages original models to acquire language-agnostic multilingual representations from new data, and preserves the model architecture without introducing parameters. Experimental results and further analysis demonstrate that our method can efficiently improve performance of existing MNMT models in translation directions where they are initially weak, and mitigates the degeneration in the original well-performing translation directions, offering flexibility in the real-world scenario.

Learn and Consolidate: Continual Adaptation for Zero-Shot and Multilingual Neural Machine Translation
Kaiyu Huang', Peng Li Junpeng Liu', Maosong Sun', Yang Liu" 1,2,3 Instittte for AI Industry Research (AIR), Tsinghua University, Beijing, China Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Shanghai Artificial Intelligence Laboratory, Shanghai, China
DDalian University of Technology, Liaoning, China (huangkaiyu, liteyyu, weeaar..ssinnin tsinghua. wwiinnua.edu.coo cdu liuyang 1@ @issinhuaa edu. cn
Abstract
Numerous Parallel Data
Multilinual Machine Traensiation Model
Although existing multilingual neural ma- chine translation (MNMT) models have demon- strated remarkable performance handle mul- tiple translation directions in single model and achieved zero-shot translation between lan- guage pairs unseen in training, they still suffer from relatively poor translation qualities for some language pairs. A practical scenario is that how continually update MNMT models for both supervised and zero-shot translations when limited new data arrives. To this end, we propose two-stage approach that encourages original models to acquire language-agnoste multilinguall representations from new data, and preserves the model architecture without intro- ducing parameters. Experimental results and further analysis demonstrate that our method can efficiently improve performance of exist- ing MNMT models translation directions where they are initially weak, and mitigates the degeneration in the original well well-perrorming translation directions, offering flexibility real-world scenario.'
bable
Trani
passage Lime
Updated
Abeed?
Site BEINTTE
you
JiP
Figure 1: We aim at adapting existing MNMT models to both supervised and zero-shot translation directions with newly available parallel data while maintaining the stronger performance in Centric directions. where dashed lines represent no parallel corpus.
guages, e.g., German which can be used to facilitate translation directions with poor performance through supervised learning (Costa- jussà et al., 2022). In addition updated data, obtaining parallel data between some languages, e.g.. Chineee encounters difficulties the world (Siddhant et al., 2022). We aspire to achieve many-to-one translation with cross-lingual transferability of MNMT models in zero-shot manner (Chen et al., 2022), when the "one" tar- get language related data is updated. As shown in Figure 1, assume three translation directions for continual adaptation: new supervised transla- tions, new zero-shot translations, and original well- performing translations (typically -eglish-Centric). In this scenario, we aim to continually improve per- formance for both new supervised and zero-shot translations while retaining previously acquired knowledge in the other translation directions ing only new data.
1 Introduction
Existing multilingual neural machine translation (MNMT) models, such as mBART (Liu et al.. 2020) and M2M-100 (Fan et al., 2021), have show- cased significant advancements to handle multi- ple translation directions in single model, espe- cially enabling zero-shot translation directions be- tween languages not encountered during training with implicit lingull transfer (Johnson et al., 2017: Tan et al., 2019; Zhang et al., 2020; Cheng et al., 2022). However, they still perform poorly on language pairs without sufficient parallel cor- pus (Goyal et al., 2022).
In accordance with this scenario, intuitive lution is to introduce additional parameters for con- tinual adaptation, regarded as parameter-isolation based methods (He et al., 2021; Madotto et al., 2021). The drawback of these methods is that they increase the size of the base models, and as
Fortunately, new parallel sentence pairs will continually emerge between high-resource lan-
"Corresponding authors: Peng Li (lipeng @ air tsinghua. edu.cn) and Yang Liu (liuyang2011 (esinghua..du.cc) 'Our code is released at https::/// //gi thub. con/ koukaiu/ LCCA

---

new data continues to accumulate, the total number of parameters may expand indefinitely (Escolano al., 2021). To preserve the model architecture, an- other solution is to leverage the continual learning paradigm (Garcia et al., 2021; Huang et al., 2022b; Gu et al., 2022; Huang et al., 2022c). Although these methods propose multi-objeccive learning to alleviate the issue of catastrophic forgetting, they necessitate full parameter fine-tuning (Tang al., 2020), resulting in relatively low training ef- ficiency (Ke and Liu, 2022). Besides, a significant pitfall of the above-mentioned methods is that they lack the ability to promote zero-shot translation.
models of different sizes to new parallel data.
Related Work
Zero-Shot Translation with MNMT Models MNMT models have demonstrated their ability to facilitate knowledge transfer across languages and enable zero-shot translations between language pairs that are not covered in training data (Lakew et al., 2018; Tan et al., 2019; Zhang et al., 2020; Chen et al., 2022). To further enhance the per- formance of zero-shot translation, some studies investigate language-agnose representations (Ari- vazhagan et al., 2019a; Pham et al., 2019; Liu et al., 2021) and language-specifie features (Wang et al., 2019; Yang et al., 2021) for zero-shot translations. For instance, Pan et al. (2021) align cross-lingual representations using additional dictionaries and contrastive learning. This work continues to ex- plore the potential of multilingual representations, but focusing on addressing the continual adapta- tion for new zero-shot translations within an well- performed representation space while avoid catas- trophic forgetting.
In this work, we propose a two-stage method consisting of a learning stage and consolidation stage for continual adaptation (LCCA), which effi- ciently adapts MNMT models to diverse translation directions. We first introduce flexible pluggable module in the penultimate encoder and decoder layer, respectively, as the multilingual adaptation space. And the introduced multilingual space is optimized with contrastive learning to make rep- resentation language-agnosee for zero-shot trans- lations, realizing the learning stage. Then we at- tempt to compress the additional parameters to the same size of base MNMT models, regarded consolidation stage. The second stage adopts the information matrix and collaborative distillation to facilitates original components to learn introduced modules in a specific range. Furthermore, aside from the related components, all the parameters of the original model remain fixed, allowing for parameter-efficient manner to large-scale MNMT models. The two stages also isolated process designed adapt to diverse application require- ments in real-world scenarios.
Continual Learning for MNMT Some previous methods of continual learning attempt to address the issue of catastrophic forgetting when only the new data is accessible (De Lange et al., 2019; Wu et al., 2021), including replay-based methods (Sun et al., 2019; Tang et al., 2020; Garcia et al., 2021), regularization--ased methods (Kirkpatrick et al., 2017: Castellucci al., 2021; Huang al., 2022a; Zhao al., 2022), and parameter-efficien transfer (PET) methods (Bupna and Firat, 2019; Zhu et al., 2021; Huang et al., 2023). On the one hand, the first two methods adopt the multi-objective learn- ing (Thompson et al., 2019; Peng et al., 2020) to balance performance between old and new tasks in continual learning through full parameters tuning, which is inefficient and time-consuming.
To sum up. our contributions are as follows:
We propose LCCA, two-stage approach that encourages to learn new knowledge for contin- ual adaptation, while mitigating performance degeneration without introducing additional parameters.
On the other hand, PET methods introduce addi- tional task-specific parameters and freeze all origi- nal parameters to completely retain performance on previous tasks (Chalkidis et al., 2021: Li and Liang, 2021; Huang et al., 2023). However, they introduce extra parameters to adapt to new tasks and lack the ability to improve zero-shot performance, limiting their application in practice (Dabre et al., 2020). Differing from these approaches, our method miti- gates the issue of zero-shot translations only using new data, and achieves efficient continual adap- tation without introducing additional parameters.
The ability of the original MNMT model to translate into target language enhanced via acquiring language-agnostic representa- tion, which improves performance for zero- shot translations continual learning.
Experimental results demonstrate the effi- ciency and flcxibility of our approach in adapt- ing various powerful and soen-source MNMT

---

Learning Stage
FER
frozen
Penuttimate Encoder Layer
Negadive
Positive
trainable
cE
Penultimate Decoder
Feed-Forward Network
Copy
Learning Module
Feed-Forward Network
Copy
Learning Module
Cross-Attentin
Self Self-Attention
Sell-Attention
+
Souurce Nyx-11
Source A]
(Targget
Target YIN-11
Target Vis-a
[Source
du
zu
2
?
[Targat P]
[Source X,]
Die
Arbeit
angenommen
Y
[Target V,]
[Source Xx.
Argentinien
gewinnt
die
Welleessterschhf
Tarrget Vx-1
Consolidation Stage
Panultimate
Penultimate
Consolidation Module
Decoder Layer
Encoder
Consolidation Module
Feed-Forward
Learning Module
Inference
Feed-Forward Network
Learning Module
Cross Anttennntn
Inference
Self-Attention
Self-Atrention
[Source id Source Sentences]
Target id
Target Sentences]
Figure 2: Illustration of our approach LCCA. Source A, Target P denotes a positive sample that is the bilingual or code-switching sentence pair, while Target Y1--1 denotes N 1 negative samples selected from the same batch for contrastive learning. The blue modules are frozen and the red modules are trainable during their own stage.
This enhanced flexibility makes it well-suited for real-world scenarios.
3.1 Task Definition
Multilingual translation models can provide high- quality translation services many language pairs and are trained selecting available parallel data with multiple language pairs. Given the anticipated growth available data, it becomes possible continually update the multilingual models for both their original and newly emerging translation direc- tions, as illustrated in Figure 1.
3
Method
Our scenario is to efficiently improve performance for some particular translation directions without compromising previous well-performing transla- tions. To achieve this, we propose two-stage method to alleviate the issues of continual adapta- tion in particular translation directions at different stages, as shown in Figure 2. Specifically, we first introduce an additional space to capture language- agnostic features for new target languages, as the learning stage. Then compress the additional space back into the original models with collabo- ration, the consolidation stage. And we calculate an information matrix to measure the parameters optimization in a smooth region across multiple languages.
Formally, the training process of an MNMT model commences with an initial set of available parallel data denoted D D1, .... D,.... DM, encompassing total of M languages, Each D, represents the training corpus for the i-th language pair. this framework, the primary MNMT model, given an input sentence x, undergoes optimization by maximizing the log-likelihood L for the ground- truth sequence This is mathematically expressed as:
Cn(0)
Σ Σ D.ED (xx)eD,
logp(y)> 0)
(1)

---

Here, represents the parameters of the MNMT model. For language identification, a specific lan- guage token is prepended to the beginning of both source and target sentences, following the conven- tion introduced in prior work (Liu et al., 2020).
where J the length of the target sentence. Our scenario also focuses on facilitating the abil- ity of the original MNMT model to translate into a target language, achieving zero-shot translation. We argue that during the learning stage for contin- ual adaptation, the additional learning module in the encoder lacks to align parallel sentences of su- pervision that can bridge the representation gap across different languages. To this end, in- troduce contrastive learning loss as an auxiliary supervision to learn woaguugge-gnosstee representa- tions for the encoder. Given a sentence pair (x, y) from new training data D', we denote as a pos- itive example and select a set of target sentences [yr]er N from the same batch as negative exam- ples. The contrastive loss is given by:
We aim to achieve supervised translation direc- tions between high-resource languages (L, and L() and zero-shot translation from M languages I1, I2,,... L to the target language L, with the help of only the newly available data D' yx.y). Due to the unavailability of the original collection parallel data, the optimization objective in con- tinual learning is given by:
(D)
Σ (x,y)En'
logp(y|x;0)
(2)
As a result, we aspire to continually update MNMT models for both supervised and zero-shot transla- tions using limited new data.
(sim(H
CCTR Σ
log
Zy (xim(H
(Xy))D)
(5)
3.2 Learning Stage
where sim is the cosine similarity function and r is the temperature parameter which is set to 0.1.
One of the crucial steps in this task involves ac- quiring new knowledge from newly available data, Previous studies show that the FFN layers can be seen as key-value memories and store knowledge in this manner (Sukhbaatar et al., 2019). Thus, the FFN might be a core component which stores cross- lingual transfcrability for multilingual translations. As shown in Figure 2, to adapt original models to new data, open up two additional blocks in the hidden layers, the learning modules. The two blocks are replicated from FFN, thus are the same as corresponding FFN layers before tuning. Con- sidering that the original models normally have a large number of parameters, introducing additional parameters in every layer would increase training costs and reduce training efficiency. Therefore, we only introduce learning modules in the penultimate layer of the original model, as the knowledge stored in the FFN of this layer closer to the high-level se- mantic information. To further leverage knowledge from the original model, we combine the original FEN and the learning modules, which can share linguistic knowledge. The fusion output of hidden states Hy is given by:
3.3 Consolidation Stage
Due to introduced parameters in the learning stage, the architecture of original models is modified. Not only the model increases the total number of param- eters, but also it requires determine the specific task which the input sentence belongs. There- fore, we aim to compress the model from the pre- vious stage to the same size original models, preserving the model architecture without introduc- ing parameters. By employing a knowledge con- solidation approach, we integrated the knowledge from the learning module into the original mod- els. In the consolidation stage, assume the two separated modules (i.e., original FFN and previ- ous learning module) cooperative relationship. where each module beneficial for translating into different target languages. In the form of distilla- tion, knowledge can be transferred to the distilled model through the distribution (Hinton et al., 2015). And we propose collaborative distillation to facil- itate original components to learn from the lcarning modules. The collaborative knowledge distillation is given by:
H2 f FFNorigimal original H + FFN H
LCKD (f,, 0;)=
The learning module is optimized by minimizing cross-entropy loss of the parallel sequence pairs:
Σ
KI. (P
CCE(O) (:00=-1 - Σ
log 1ogp(u,lye,,x;10) (4)
[|P(s|yes,x000) (6)

---

where of and 0j represent the parameters of original FFN and learning modules, respectively. In addition to retrospecting the original knowl- edge, we also intervene in the stage of consolida- tion to make it relatively smooth. Given the model parameters and the model distribution p(z[0), it is natural to maximize the likelihood function to optimize 0. To evaluate the optimization for 0, we define score function s(0) = V log p()]()) and its measure of uncertainty that regarded as the covariance of the models, representing the degree correlation between two arbitrary variables that change together:
experiments, we provide the German-Chinese (de- zh) bilingual data considered for continual adapta- tion, as the newly available training corpus'. Con- sidering the issue of data quality, we performed data cleaning and sampling, resulting in a corpus of 10 million sentence pairs. As a result, we de- note many-to-English (xx->en) as original transla- tion directions, Hormann-to-hineee (de->zh) and Chinese-to-German (zh~>de) as supervised translation directions, many-to-German (xx-+de) and Kany-to-Chines (xx->zh) as new zero-shot translation directions. The validation data from FLoRes (Goyal et al., 2022) and we only choose the validation set in the new supervised translation direction for model selection.
F = Ep]= VV log p( V logp(z)0)1] (7)
In this task, we can approximate the expectation in F using empirical distribution ((x), which is given by the parallel training data:
Model Configuration In scenarios, we have chosen employ the mBART50-1n (Tang et al., 2020) the base MNMT model. The mBART50- nn is an English-centric multilingual model capa- ble of handling 50 different languages. To ensure consistency, we tokenize the parallel data using the same SentencePiece (Kudo and Richardson, 2018) model as mBART50-nn, which boasts a shared vocabulary consisting of 250,000 tokens. The mBART50-nn structured as Transformer- based model, featuring 12 encoder layers, 12 de- coder layers, and 16 multi-attention hoads. a more comprehensive understanding of model con- figuration, additional details are available in Ap- pendix A.
F- Eay. V log No
log V
(8)
The role of is measure of curvature of the optimization and has a connection to our LCKD- This gives rise to natural gradient loss LCKD with the information matrix F which can define the local curvature in distribution space:
LCKD r(0,,0:)
We compare our proposed LCCA with various representative methods in continual learn- ing and transfer learning for continual adaptation. The baselines can be listed as follows:
Baselines
N Σ (0) 07)
A
(CKD(O,,0)
(9)
N
where À is a hyper-parameter to balance the origi- nal parameters and learning modules. 84 represents the original parameters of FFN with computing the information matrix. Thus we can optimize the orig- inal FFN in a smooth region, preserving the previ- ously acquired knowledge. Note that we utilize small-scale set corresponding to the previous task to calculate the F matrix.
Scratch (Vaswani et al., 2017): training a bilin- gual Transformer-big model from scratch in the supervised translation directions.
mBART50-m (Tang et al., 2020): implement- ing all translation directions based on this orig- inal MNMT model directly.
Fine-Tuning (Luong and Manning, 2015): tun- ing based on the original models with new data. All original parameters are trainable.
4 Experiments
4.1 Experiment Settings
Mixed-FT (Sun et al., 2019): mixing the exter- nal training data related to original translation directions with the new training data to train the model jointly.
Parallel Data In this work, focus on con- tinual adaptation for MNMT models, aiming to enable the model improve performance of super- vised and zero-shot translation using only new data. To ensure the reliability and reproducibility of the

---

Old Directions
New Directions
Old Directions
New Directions
Method
AVG.
AVG.
xx-
de
xx
xx-en
zi-ade
xx-+de
Scratch
0.03
29.99
2.13
1.08
1.07
15.17
1.12
1.10
mBARTS0-an
24.72
3,40
5,23
14,98
24.72
6.92
8,46
16.59
Vine-Tuning
0.03
33,77
2.52
1.28
0.12
20.29
0.49
0.31
Mixed-FT
19.33
29.31
6,36
12.85
20.21
15.31
4.37
12.29
0.21
30.33
18.48
9.35
0.98
17.22
10.92
5.95
SixT
0.14
30.05
19.40
9.77
0.72
16.85
12.04
6,38
EWC
23.01
29.93
19,47
21.24
23.19
15,54
13.97
18.58
LFR FR-CM
23.11
29.91
21.13
22.12
23.13
16.74
14.97
19.05
LFR-OM
23.03
29.97
20.93
21.98
23.32
16.13
14.74
19.03
LCCA w/o LS
23.47
29.98
20.30
21.89
22.97
15.23
13.74
18.36
LCCA
24.03
31.17
22.09
23.06
24.81
18.99
18.13
21,47
Table 1: The overall BLEU scores of the particular adaptation directions. "w/o LS" represents that directly the consolidation stage without learning stage. "xx" denotes the languages supported by mBARTS0-mn. "Avg" denotes the average BLEU scores on all 100 translation directions. "f" indicates the stage one in the SixT. The highest score is highlighted in bold and the second highest score is underlined.
34
Pull Tre-iuning 33.77
25
29
Nictelarrg
de-roh
22:11
Hock furing
Adapse
29.27
BLEU
BLEU
LECA
19
PUB Suning
2h-ddo
13.0n
Firteiluingg
54.11
Prrtellute
1009
14:37
Fine-tuned Perameters
Fine ane-tuned Parameters (%)
(a) Supervised Translation
Zero-Shot Translation
Figure 3: Performance of aurameter-isolation methods for continual adaptation task in multiple translation directions. The x-axis represents the proportion of tuned parameters during tneettunig relative to the total parameters.
SixT (Chen et al., 2022): freezing model pa- rameters of encoder and decoder layers in dif- ferent stages, which beneficial for zero-shot translations.
Adapter (Bapna and Firat, 2019): introducing additional parameters in the FFN layer of orig- inal models and freezing all base parameters.
Training and Evaluation We evaluate the perfor- mance of translation directions using the FLo- Res testsets (Costa-jussà et al., 2022), covering 50 languages from 10 language groups. The per- formance of translation is measured by the detok- enized case-sensitive BLEU score, calculated us- ing the SacreBLEU evaluation script (Post, 2018)². The training time of each method is reported in kiloseconds. All models are implemented using the open-source toolkit fairseq* (Ott al., 2019). For more detailed information, please refer to Ap- pendix B.
EWC (Kirkpatrick et al., 2017): adding a penalty with Fisher matrix to preserve pre- vious knowledge.
LFR (Gu et al., 2022): constraining trainable parameters within low forgetting risk regions in continual learning, including LFR-CM and LFR-OM, respectively.
LoRA (Hu et al., 2022): injecting low-rank matrices into every attention layer, deploying independent tasks of fine-tuned models.
Prefix- Tuning (Li and Liang, 2021): prepend- ing prefixes to the keys and values at every self-attention layer.
Signature: nrefs:1 casermixed effino tok:13a smoothvexp version:2.2.0. Many-to-Chinese: nrefs:1 casermixed leffmo tok:zh smooth:exp version:2.2.0.. "https: //gi thub. com/pytorch/fairsee

---

Translation Directions
Translation Directions
Model
Method
#Size
AvG.
AVG.
xx-ten
de-
xx xx-+th
xx-+en
zh-
xx-
Base
0.6B
24.72
3.40
5.23
14.98
24.72
6.92
8.46
16.59
mBART
LCCA
0.6B
24.03
31.17
22.09
23.06
24,81
18.99
18.13
21.47
Base
1.2B
24.67
28.96
22.64
23.66
24.67
17.92
18.98
21.83
M2M
LCCA
1.2B
24.00
31.88
25.42
24.71
24.82
18.84
19.76
22.29
Base
1.3B
37.00
26.86
24.87
30.94
37.00
17.97
22.82
29.91
NLLB
LCCA
1.3B
36.13
30.99
26.37
31.30
35.98
18.97
24.27
30.13
Base
3.3B
38.43
26.80
26.13
32.28
38,43
19,42
24.57
31.50
NLLB
I.CAA
3.3B
38.17
31.35
28.93
33.55
38.02
20.03
25.43
31.73
Table The overall BLEU scores based on various MNMT models. "Base" indicates the original performance on MNMT models. The better score of each MNMT model is highlighted in bold.
4.2
Main Results
methods, they lack encouragement to learn new knowledge from updated data, while LCCA the consolidation stage outperforms the baselines parameter-isolation, comparing the overall results.
As presented in Table 1, we conducted an extensive assessment of translation quality across multiple directions as new data became available. The re- sults indicate that our proposed approach, referred to as LCCA, outperforms several baseline meth- ods, particularly in terms of average BLEU scores and new many-to-
4.3 Results on Various MNMT Models
As indicated in Table 2, we have employed range of pre-trained MNMT models as the initial models and examined the effectiveness of LCCA in the adaptation process. The outcomes underscore the versatility of our approach, showing that it can suc- cessfully adapt diverse, powerful models of vary- ing sizes to incorporate newly available data. This adaptability underscores the efficiency and flexibil- ity of our approach in real-world applications. Even if the original models already exhibit strong capa- bilities in certain translation directions, method can further enhance the performance through learn- ing knowledge from new data, achieving a com- prehensive upgrade in that particular translation direction. We also perform a new comparison be- tween the performance of NLLB 1.3B and 3.3B, and there is no indication that as the model be- comes larger, the effectiveness of LCCA reduces significantly.
for the original many-to-Fnng target translation directions. Specifically, LCCA attains average BLEU score of 22.09 for xx- and 24.03 for -en The trend persists when the target language is switched German, in which LCCA even demonstrates improvements in old language translation directions. Although the Fine- Tuning method achieves the best results in super- vised translation (de ++ zh) for continual adapta- tion, it suffers from catastrophic forgetting, which makes it unsuitable for my scenario. Furthermore, most of the baseline methods are more vulnerable adapting original models to the new zero- shot translation, while our approach achieves a notice- able improvement. And the results also show that LCCA exhibits no significant difference from the traditional regularization based approach without the learning phase. Therefore, it involves pro- cess of bringing the added parameters closer to the original parameter space with LCCA.
4.4 Ablation Studies
Effects of Different Learning Modules
As shown Figure we also investigate trans- lation qualities of parameter-isolation methods in multiple directions. The results show that LCCA in the learning stage achieves better performance with fewer additional parameters in both supervised and zero-shot translations for continual adaptation. Moreover, the improvement more noticeable in the zero-shot translation directions in terms of aver- age BLEU scores, e.g., 22.81 scores on xx -x- and 19.09 scores on xx-+de Although no performance degradation has occurred using parameter-isolation
As shown Table 3, we further investigate differ- ent strategies and modules in the learning stage. To learn new knowledge from new data, we introduce the learning modules in the serial or parallel con- nection manner, representing their relation to the layers of the original models. The results demon- strate that the parallel manner achieves slightly bet- ter performance, regardless of whether contrastive learning methods are employed, between and 3 (or and 4). Besides the comparison results be- tween and (or and 4) show that the contrastive

---

Learning Module
New Directions
No.
Manner
Leta
de-
xx-+zh
zh-+de
xx-
1
serial
x
31,08
22.63
18.70
18.81
2
serial
31.11
22.78
18.67
19.03
3
parallel
X
31.35
22.64
19.32
18.79
4
parallel
31.38
22.81
19.32
19.09
Table Results different strategies and modules in the learning stage. The highest score is highlighted in bold.
Method
xx-+en
de-+ch
XX-+th
XX-+en
sh
XX-+de
Adapter
24.01
29.64
20.09
23.99
16.08
13.78
LoRA
24.00
28.79
19.45
24.26
16.70
13.12
Prefix
23.32
26.89
19.39
23.46
15.96
12.85
Ours
24.03
31.17
22.09
24.81
18.99
18.13
Table Results on different PET methods learning stage. The highest score is highlighted bold.
Effects of Hyper-parameter
As shown in Figure 4, we compare the consolida- tion stage with different hyper-parameters À which control the trade-off between the performance of original and new translations. The larger the hyper- parameter X, the less the decrease for the previous tasks and the less À is to facilitate to learn new tasks. We compare our method with EWC (Kirk- patrick al., 2017) and LFR (Gu et al., 2022). For these methods, search the range from 0.001. Although these methods are sensitive hyper-parameters, which restricts the robustness. our method achieves better performance and rela- tively less affected by hyper-parameters, according to the downtrend of the curve in Figure 4.
Original Translations (=ee))
22
LCCA
EWC
LFR
New Shot Translations (-zhh
Figure 4: Results using different hyper-parameters. The x-axis and y-axis denote the BLEU of the zero-shot translations (xx->zh) and original translations (xx-ten), respectively. The point closer to the upper right corner indicates better result.
4.5 Visualization of Representations
To investigate the zero-shot translations for con- tinual adaptation, we present a visualization multilingual representations. It studies the shared representations for MNMT models, which can ob- serve their semantic equivalents among multiple languages (Cheng et al., 2022). The results of vi- sualization show that the learning stage of LCCA plays an important role in optimizing multilingual representation spaces and the consolidation stage does not undermine efforts of the previous stage. The details of the representation analysis are pro- vided in Appendix C.I.
learning is more important to learn the knowledge from new data for zero-shot translations, and the impact of supervised translation is minimal.
As shown in Table 4, we also supply an experi- ment to validate the effectiveness of LCCA at the learning phase, compared with different PET-based methods. The results show that the effectiveness of consolidation phase is related to the first learning phase. The higher performance during the learning stage, the better performance on the consolidation stage. Because the role of the consolidation stage is collaboratively integrate the newly added pa- rameters from the learning stage into the original model.
4.6 Training Cost
We further investigate the training cost compared with the stronger baselines. The results show that LCCA can improve the efficiency to continually adapt original models to new languages. LCCA can

---

the model compresses. Further analyses reveal that our method effectively captures linguistic features and bridges the gap shared representation space for comprehensive continual adaptation.
de
Limitations
In this work, aspire to continually improve per- formance of MNMT models in translation direc- tions where they are initially weak, and alleviate the issue of degeneration in the well-performing translation directions. In addition to the advantages mentioned, our method does have certain limita- tions as follows:
Fine Fine-Tuning
(1) Limited available data: We only utilize the parallel data
language pair for continual adaptation in this work. There many diverse datasets available, including monolingual data and parallel data on extremely low-resource target lan- guage.
LCCA
(2) Sequential adaptation: This work only con- siders adapting the original models to new data continually once. However, multiple parallel data are available in sequential manner. Due to the uncertainty about potential data exposure from the newly available data the sets, we plan carefully design and explore this scenario in future work.
Figure 5: T-SNE visualizations of encoder representa- tions on xx-> translations directions.
acquire and consolidate new knowledge efficiently for continual adaptation, which is more practical the real-world application. Since we only add additional parameters during the two stages in one layer, the training cost does not significantly in- crease the depth of the model increases. More details provided in Appendix C.2.
Ethies Statement
Regarding the datasets, we utilize CC-Matrix the newly available data and multilingual FLoRes datasets for evaluation (Costa-jussà al., 2022). All datasets which we use are available in public and widely used by researchers. Similarly, regard- ing the pre-trained MNMT models, we leverage mBART (Liu et al., 2020). M2M-100 (Fan et al., 2021) and NLLB (Costa-jusså et al., 2022). These models come from the open-source community. The generated process has the potential show inappropriate and misleading translation results, which can be mitigated by modifying the test sets before inference.
5 Conclusion
this work, we propose a two-stage method of learning and consolidation to improve performance of pre-trained MNMT models when new data ar- rives in both supervised and zero-shot translations. It extends multilingual language-agnone with the contrastive learning for new data adap- tation in the learning stage, and then adopts a col- laborative distillation with an information matrix to consolidate both previously and newly learned knowledge for module integration. As a result, our proposed method, LCCA, achieves continual adap- tation to multiple translation directions while keep- ing the model architecture intact. Experimental results demonstrate that the learning stage encour- ages original models learn new knowledge from updated parallel data and the consolidation stage mitigates the performance degradation on previ- ous well-performing translation directions when
space
A Acknowledgements
This work is supported by the National Key R&D Program of China (2022D00160502) and the Na- tional Natural Science Foundation of China (No. 61925601, 62276152). We sincerely thank GT- COM provide the computing resource and the reviewers for their insightful comments and sug- gestions to improve the quality of the paper.

---

References
and Tinne Tuytelaars. 2019, Continual learning: A comparative study on how to defy forgetting in clas- sification tasks. arXiv preprint urXiv:1999.080 2(6).
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a. The missing ingredient in zero-shot neural machine translation. CoRR, abs/1999..7091.
Carlos Escolano, Marta R Costa-Jusså, and José AR Fonollosa. 2021. From bilingual to multilingual neural-based machine translation by incremental training. Journal of the Association for Information Science and Technology. 72(2):190-22 203.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019h. Massively multitingual neu- ral machine translation in the wild: Findings and challenges. arXiv preprint arviv: 0000199000000 050199
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky. Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric mul- tilingual machine translation. J. Mach. Learn. Res.. 22(107):1-45
Ankur Bapna and Orhan Firat, 2019. Simple, scal- able adaptation for neural machine translation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538-1548.
Akhbardeh Farhad, Arkhangorodsky Arkady, Biesialska Magdalena, Bojar Ondiej, Chatterjee Rajen, Chaud- hary Vishrav, Marta R Costa-jussa, España-Bonet Cristina, Fan Angela, Federmann Christian, et al. 2021. Findings of the 2021 conference on machine translation (wmt21). In Proceedings of the Sixth Conference on Machine Translation, pages 1-88. As- sociation for Computational Linguistics.
Giuseppe Castellucci, Simone Filice, Danilo Croce, and Roberto Basili. 2021. Learning to solve nlp tasks in an incremental number of languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 837-847,
Xavier Garcia, Noah Constant, Ankur Parikh, and Orhan Firat. 2021. Towards continual learning for multilin- gual machine translation via vocabulary substitution. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 118-11992.
Thas Chalkidis, Manos Fergadiotis, and Ion Androut- sopoulos. 2021. Multieurlex-a multi- Mallti-inngul and multi-label legal document classification dataset for zero-shot gross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6974-6996.
Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, and Furu Wei. 2022. Towards making the most cross- longuli transfer for zero-shot neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 142-157.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek. Da Ju. Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2022. The flores- 101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
Yong Cheng, Ankur Bapna, Orhan Firat, Yuan Cao, Pidong Wang, and Wolfgang Macherey. 2022. Multi- lingual mix: Example interpolation improves multi- lingual neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4092-4102.
Shuhao Gu, Bojie Hu, and Yang Feng. 2022. Con- tinual learning of neural machine translation within low forgetting risk regions. urXiv:2211.01542
arXiv preprint
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficien transfer learning. In International Conference on Learning Representa- tions.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.0466
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503002330.,, 2(7).
Raj Dabre. Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine trans- lation. ACM Computing Surveys (CSUR), 53(5): 38.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learn- ing Representations.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh,

---

Kaiyu Huang, Jingxiang Cao, Zhuang Liu, and Degen Huang. 2022a. Word-based method for chinese part- of-speech via parallel and adversarial network. Chi- nese Journal of Electronics, 31333337-344.
Xiang Lisa Li and Percy Liang. 2021. Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597.
Kaiyu Huang, Peng Li, Jin Ma, and Yang Liu. 2022b. Entropy-based vocabulary substitution for incremen- tal learning in multilingual neural machine transla- tion. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 10537-10550, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Danni Liu, Jan Niehues, James Cross, Francisco Guzmán, and Xian Li. 2021. Improving zero-shot translation by disentangling positional information. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 1259-1273, Online. Association for Computational Linguistics.
Kaiyu Huang, Peng Li, Jin Ma, Ting Yao, and Yang Liu. 2023, Knowledge transfer in incremental learn- ing for multilingual neural machine translation. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 55286-15304 Toronto, Canada. Association for Computational Linguisties.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Yichong Huang, Xiaocheng Feng, Xinwei Geng, and Bing Qin. 2022c. Omniknight: Multilingual neu- ral machine translation with language-specifg self- distillation. arXiv preprint arxiv:2205.0666 22200.20..
Minh- Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spo- ken language domains. In Proceedings of the 12th International Workshop on Spoken Language Trans- lation: Evaluation Campaign, pages 76-79.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, -339-351..
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Se- ungwhan Moon, Paul A Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, and Zhiguang Wang. 2021. Continual learning in task-oriented dialogue systems. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing. pages 7452-7461
Zixuan Ke and Bing Liu. 2022. Continual learning of natural language processing tasks: A survey. arXiv preprint arXiv:2211.122 127001.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan. Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling, In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations), pages 48 18-53, Minneapolis, Minnesota. Association for Computational Linguistics.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Bariiikaa et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3521-3526.
Xiao Pan, Mingxuan Wang. Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume I: Long Papers), pages 244-258, Online. Asso- ciation for Computational Linguistics.
Taku Kudo and John Richardson. 2018 Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71.
Wei Peng, Chongxuan Huang, Tianhao Li, Yun Chen, and Qun Liu. 2020. Dictionary- -based data augmen- tation for cross-domain neural machine translation. arXiv preprint 2rXiv:2004.02577.
Surafel Melaku Lakew, Mauro Cettolo, and Marcello Federico. 2018. A comparison of transformer and recurrent neural networks on multilingual neural ma- chine translation. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, pages 641-652, Santa Fe, New Mexico. USA. Asso- ciation for Computational Linguistics.
Ngoc-Quan Pham, Jan Nichues, Thanh-Le Ha, and Alexander Waibel. 2019. Improving zero-shot trans- lation with inguagg--ndependde constraints. In Pro-

---

ceedings of the Fourth Conference on Machine Trans- lation (Volume 1: Research Papers), pages 13-23, Florence, Italy, Association for Computational Lin- guistics.
method. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 1213-1223, Florence, Italy. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. 2021. Pre- trained language model in continual learning: A com- parative study. International Conference on Learn- ing Representations.
Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Chen, Isaac Caswell, and Xavier Garcia. 2022. Towards the next 1000 languages in multilin- gual machine translation: Exploring the synergy be- tween supervised and self-supervised learning, arXiv preprint htth:2201.031111
Yilin Yang, Akiko Eriguchi, Alexandre Muzio, Prasad Tadepalli, Stefan Lee, and Hany Hassan. 2021. Im- proving multilingual translation by representation and gradient regularization. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 7266-7279, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lam- ple, Herve Jegou, and Armand Joulin. 2019. Aug- menting self-attention with persistent memory. arXiv preprint arXiv:1907.0m 01470.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2020. Share or not? learning schedule language-specific capacity for multilingual transla- tion. In International Conference Learning Rep- resentations.
Fan-Kem Ken-Keng Sun, Cheng- Ho, and Hung- Yi Lee. 2019. Lamol: Language modeling for lifelong language learning. In International Conference Learning Representations.
Yang Zhao, Junnan Zhu, Xiang, Jiajun Zhang. Yu Zhou, Feifei Zhai, and Chengqing Zong. 2022. Life-long learning for multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv:2212.02800
Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie- Yan Liu, 2019. Multilingual neural machine translation with language clustering. Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 963-973.
Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingx- Wang. and Lei Li. 2021. Counter-interference adapter for multilingual machine translation. In Find- ings of the Association for Computational Linguistics: EMNLP 2021, pages 2812 2823.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual translation with exten- sible multilingual pretraining and finetuning, arXiv preprint 200:02008.00401.. 00401.
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long and Short Papers), pages 2062-2066.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, pages 6000-6010.
Yining Wang, Long Zhou, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. 2019. com- pact and language-sensitive multilingual translation

---

A Model Details
Code Language
Family
Script
Order
Afrikaans
Germanic
Latin
SVO
We utilize Adam (Kingma and Ba, 2014) 31 - 0.9 and 82 = 0.98 to optimize the trainable parameters the MNMT models and adopts the temperature- based sampling (Arivazhagan et al., 2019b) with temperature of T 20 to balance the training data for the "Mixed-FT" methods. The dropout and attention-dropout
ar
Arabic
Semitic
Arabic
SOV
Azerhaijani
Turkic
Latin
SOV
bn
Bengali
Indo-Aryan
Bengali
SOV
Czech
Balto-So Palto-Slavic
Latin
SVO
German
Germanic
Latin
SVO
English
Germanic
Latin
SVO
Spanish
Italic
Latin
SVO
et 0.3 and 0.1 in all experiments, respectively. For regularization-baaee method (LFR), we follow the basic setting in Gu et al. (2022). The models are trained on NVIDIA A 100 GPUs and we report the mean BLEU scores of the our methods that are trained in three different seeds to ensure results reliable which have random- ness. The batch size is 4.096x 4 in the procedures of parameter-isolatioon based methods and 2,048 x in the procedures of fully tuning methods.
et
Estonian
Finnic
Latin
SVO
Persian
Irunian
Arabic
SOV
Finnish
Finnic
Latin
SVO
French
Italic
Latin
SVO
Galician
Italic
Latin
SVO
gu
Gujarati
Indo-Aryan
Gujarati
SOV
Hehrew
Semitic
Hebrew
SVO
Hindi
Indo-Aryan
Devanagari
SOV
Croatian
Balto-Slavic
Latin
SVO
Indonesian
Austronesian
Latin
SVO
Italian
Italic
Latin
SVO
Japanese
Japanesic
Japanese
SOV
Georgian
Kartvelian
Georgian
SOV
kk
Kazakh
Turkic
Cyrillic
SOV
In our experiments, we have implemented sev- eral pre-trained MNMT models with varying sizes and configurations. Specifically:
Khmer
Austronesian
Khmer
SVO
Korean
Korean
Hangul
SOV
Lithuanian
Balto-Slavic
Latin
SVO
Larvian
Balto-Slavic
Latin
SVO
The mBART50-nn (Liu et al., 2020) model (610 million parameters) comprises 12 stacked encoder layers, 12 stacked decoder layers, and 16 multi- attention heads. The di- mensions hidden states and FFN are 1,024 and 4,096, respectively.
mk
Macedonian
Balto-Slavic
Cyrillic
SVO
ml
Malayalam
Dravidian
Malayalam
SOV
Mongolian
Mongolic
Cyrillic
SOV
Marathi
Indo-Aryan
Devanagari
SOV
my
Burmese
Sino- Sino-Tibetan
Myanmar
SOV
Nepali
Indo-Aryan
Devanagari
SOV
Dutch
Germanic
Latin
SVO
pl
Polish
Balto-Slavic
Latin
SVO
Pushto
Irunian
Arabic
SOV
The M2M-100 (Fan et al., 2021) model (1.2 billion parameters) features 24 stacked en- coder layers, 24 stacked decoder layers, and 16 multi- attention heads. The dimensions of hidden states and FFN are 1,024 and 8,192. respectively.
Portuguese
Italic
Latin
SVO
Romanian
Italic
Latin
SVO
Russian
Balto-Slavic
Cyrillic
SVO
Sinbula
Indo-Aryan
Sinhula
SOV
Slovenian
Balto-Slavic
Latin
SVO
Swedish
Germanie
Latin
SVO
Tamil
Dravidian
Tamil
SOV
Thai
Tai- Kadai
Thai
SVO
Tagalog
Austronesian
Latin
VSO
The NLLB (Costa-jussà et al., 2022) model (3.3 billion parameters) consists 24 stacked encoder layers, 24 stacked decoder layers, and 16 hidden states and FFN set at 2,048 and 8.192, respectively.
Turkish
Turkic
Latin
SOV
uk
Ukrainian
Balto-Slavic
Cyrillie
SVO
Undu
Indo-Aryan
Arabic
SOV
Vietnamese
Austronesian
Latin
SVO
heads, with dimensions of
Xhosa
Atlantic-Cone Congo
SVO
Chinese
Sino-Tibea
Han
SVO
Table 5: The characteristics of languages for evaluation.
For optimizing the trainable parameters of the MNMT models, we employ the Adam opti- mizer (Kingma and Ba, 2014) with 31 = 0.9 and B2 - 0.98. We also utilize temperature-bait based sampling (Arivazhagan et al., 2019b) with a tem- perature of T 20 to balance the training data for the "Mixed-FT" methods. Dropout and atten- tion dropout rates both 0.3 and 0. re- spectively, in all experiments. During training, the batch size set to 4,096 4 for parameter-isolation based methods and 2,048 x for fully tuning meth-
"S, V, O" represent subject, verb and objective, respec- tively.
ods with 8 NVIDIA A100 GPUs. And LCCA are trained with three different seeds to ensure the reli- ability of results considering inherent randomness.

---

220
Training Time ( x 10P S)
165
110
Fine-Tuning
Mixed-FT
LFR
LoRA
Adapter
LCCA (Ours)
Figure 6: The training time of various methods for adapting mBART50-nm xx-+zh and xx xxe respectively.
B Dataset Details
necessitates the use multi-source sentences con- veying the same meaning in different languages with t-SNE (Van der Maaten and Hinton, 2008). As evident in Figure 5, the sentence representations produced by LCCA exhibit a closer proximity, il- lustrating the adeptness of our model at adapting data for zero-shot translations. Consistent with this visualization, Table further substanti- ates that our method excels achieving improved performance zero-shot translation scenarios.
In this work, we utilize parts of the FLoRes de- vtest as our test set, which covers 50 languages and follows the policy in a multi-source setting. The test set contains 1012 sentences in each language and is divided into different language groups with linguistic diversity for evaluation. It is worth not- ing that language family refers to collection of languages that share common ancestral language, the proto-language*. Varia-
often referred tions in grammar and word order can be observed across different language families And perform- ing cross- lonsu-l transfer and zero-shot translation between languages with significant differences is challenging. The FLoRes follows the C--BY-SA 4.0 license that can be freely used for research pur- poses (Farhad et al., 2021). The characteristics of different languages and language codes used in this paper shown in Table 5.
C.2 Training Cost
To further underscore the efficiency of LCCA, we examine the training time in comparison to the more robust baseline models, as depicted in Fig- ure 6. Although our method has two stages, the results show that training time of LCCA is close that of the parameter-efficient methods and shorter than the other methods with fully tuming, which is more efficient and practical for continual adapta- tion.
More Comparisons
C.1
Visualization of Representations
As depicted Figure 5, we visualize sentence rep- resentations in the context of xx xx-+zh to probe the representation gap between languages. Achieving comparability within single representation space
Phttps: //en. wikipedia. wtwpaapeiia.oog/iiiiiii..... chttps://walss info


Reviews:
[{"review_id": "DiZqfQWW7t", "reviewer": "Reviewer_HGki", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback.\n\n> Q1: There could be more analysis on how the model weight changed and benchmark the two-stage training efficiency (i.e. efficiently adapting model weights without significant retraining)\n\n(1) In this paper, to investigate the effectiveness for continual adaptation, we present a visualization of multilingual representations. And we will add more visualizations at different stages. The results of visualization show that the learning stage of LCCA plays an important role in optimizing multilingual representation spaces and the consolidation stage does not undermine the efforts of the previous stage. In principle, it involves a process of bringing the added parameters closer to the original parameter space.\n\n(2) As larger models are typically more substantial and have varying depths, to ensure method generality and efficiency, we make a sacrifice in the learning stage's performance, **where new parameters are only added in the penultimate layer**. Moreover, the effectiveness and trainable parameter amount of the consolidation phase are influenced by the learning phase. We will add more analysis of the two-stage training efficiency with different model sizes and we hope our study could shed light on developing novel continual learning for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work is one of the first to adapt existing MNMT model to new data without hurting the existing performance or increasing model size.\n* Their results look very promising. They conduct experiment on strong public available pretrained MNMT weights and eval set.\n", "reasons_to_reject": "* There could be more analysis on how the model weight changed and benchmark the two-stage training efficiency (i.e. efficiently adapting model weights without significant retraining)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies how to efficiently scale MNMT model to new data without increasing model size. It proposes a two-stage method with learning and then consolidation.\nResults show that their approach beats all earlier approach w/ strong pretrained model and on common baseline (FLoRes eval set).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DCCJgYPssm", "reviewer": "Reviewer_3Rbs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback and constructive comments.\n\n- For each of your concerns on over-claim:\n\n> Q1: In line 105, \"... without introducing additional parameters.\" However, at the learning stage, a new feed-forward module is required to capture the new knowledge.\n\nThanks for pointing this out. Our goal is to **preserve the original model's architecture** **and keep the number of parameters** after the entire two-stage training process rather than indicating that new parameters are not introduced during the intermediate process. Although our method increases the number of parameters at the first stage, LCCA specifically designs to reduce this added parameters at the second stage. The advantages are: (1) no increase overhead in inference; (2) parameters are non-isolated, eliminating the need to select specific adapters for specific languages. We will modify this misunderstanding in future versions to make it more rigorous.\n\n> Q2: In line 110, \"which improves performance for zero-shot translations in continual learning.\" This claim is too general: it implies that the proposed approach improves zero-shot translation. But the performance is mainly measured on the translation directions related to the language pairs of the new training data.\n\nIn this paper, we investigate a particular zero-shot translation proposed by [1], where not using direct parallel sentence pairs are considered as zero-shot translation. In this scenario, the model is trained with parallel dataset of only one language pair (such as de-zh). The trained model is tested on many-to-one language pairs (like cs/fi/fr-zh) in a zero-shot manner.\n\n> Q3: The proposed approach is not well ablated. Current results don't demonstrate the necessity of the two-stage strategy.\n\nThank you for your constructive suggestion. Just as the title suggests, the second stage serves as a consolidation phase. In principle, it involves a process of bringing the added parameters closer to the original parameter space. If we solely adopt the strategy of the second stage, the model will be optimized from a randomly initialized space. To validate this statement, we conduct the experiment using only the second-stage approach\n\n| Method                  | xx->en | de->zh | xx->zh | xx->en | zh->de | xx->de |\n| ----------------------- | ------ | ------ | ------ | ------ | ------ | ------ |\n| EWC                     | 23.01  | 29.93  | 19.47  | 23.19  | 15.54  | 13.97  |\n| LCCA w/o learning phase | 23.47  | 29.98  | 20.30   | 22.97  | 15.23  | 13.74  |\n| LCCA                    | 24.03  | 31.17  | 22.09  | 24.81  | 18.99  | 18.13  |\n\nThe \"LCCA w/o learning phase\" means that we only adopt the second stage process. The results show that the method exhibits no significant difference from the traditional regularization-based approach without the learning phase. Therefore, we believe that the two-stage approach is necessary.\n\n> Q4: In Section 3.2, the authors argue that L_CTR could induce language-agnostic representations. But, in Table 3, we find that L_CTR benefits translation very little. Do we need it?\n\n(1) **The effectiveness of \"L_CTR\" in the learning phase.** While the improvement might seem limited on average, in certain directions, there are notable differences in BLEU scores between the methods w/ and w/o \"L_CTR\". Besides, we counted the **number** of zero-shot translation directions that achieved performance improvement, as the Table shown:\n\n| Zero-Shot Directions | Increased by less than 1 BLEU | Increased by more than 1 BLEU | Passing significance testing (p<0.05) | No improvement | Zero-shot translation directions |\n| -------------------- | ----------------------------- | ----------------------------- | ------------------------------------- | -------------- | -------------------------------- |\n| xx->zh               | 13                            | 25                            | 23                                    | 11             | 49                               |\n| xx->de               | 13                            | 24                            | 19                                    | 12             | 49                               |\n\nThe light increase in the average BLEU is attributed to the remaining 18 directions and the average BLEU downplays the impact of \"L_CTR\". For example, on xx->zh directions, although we achieve performance improvements of more than 1 BLEU in 31 directions with \"L_CTR\", the directions with limited or no improvement diluted the overall performance.\n\nWe will provide these detailed results on all the translation directions in the revised version.\n\n(2) **The effectiveness of \"L_CTR\" in the consolidation phase.** We also add a further comparsion to validate the effectiveness of \"L_CTR\". If we use the learning module w/o \"L_CTR\" in the consolidation phase, the zero-shot performance decline is greater than the learning module w/ \"L_CTR\". For xx->zh directions, LCCA w/o \"L_CTR\" degrade to 21.01 average BLEU (reduced by 1.73 compared to the learning phase) in the consolidation phase, and LCCA w/ \"L_CTR\" degrade to 22.09 average BLEU (reduced by 0.72 compared to the learning phase) in the consolidation phase. For xx->de directions, we can draw similar comparsion (-2.65 vs -0.96). We will add this ablation study in the revised version.\n\n> Q5: The authors didn't explore the necessity of the separation of the two stages. What if removing the new learnable module and directly optimizing the FFN layer with L_CE and L_CKD-F?\n\nAs mentioned above (Q3), we have added these additional experiments to validate the significance of the learning phase. Please refer to that statement.\n\n- For each of your concerns on experimental results:\n\n> Q6: In Figure 3, LCCA outperforms almost all efficient tuning methods in all conditions at the learning stage. But the learning stage only copies the last-layer FFN module. Why would it surpass Adapter and LoRA, particularly considering the latter two have more parameters? Is it because the initialization?\n\nAlthough the adapter and LoRA have more parameters, the additional parameters in their method are evenly distributed across each layer. And some of these layers play a limited role. As you rightly pointed out, the advantage of copying is evident in initialization, which has been demonstrated by [2], compared with the other PET methods. Although it might be better to extend to all layers rather than just the penultimate layer, we propose LCCA to ensure method generality and efficiency for various pre-trained models with differnet sizes.\n\n> Q7: In Table 2, as model becomes larger, the effectiness of LCCA reduces. It seems to suggest that model scaling might have a big role here and even change the conclusion. \n\nThanks for your insights. The primary intention of this table is to demonstrate that our method can adapt various powerful models of different sizes to newly available data, offering efficiency and flexibility in the real-world. And we will answer your questions from the following three points: \n\n(1) These three models cannot be directly compared since the training data and details for the base models (mBART / M2M / NLLB) differ. To address this, we performed a new comparison between the performance of nllb 1.2B and 3.3B, and there is no indication that as the model becomes larger, the effectiveness of LCCA reduces significantly. For instance, when de->zh arrives, NLLB-1.2B improves 0.31 average BLEU and NLLB-3.3B improves 1.27 average BLEU with LCCA. When zh->de arrives, NLLB-1.2B improves 0.21 average BLEU and NLLB-3.3B improves 0.23 average BLEU with LCCA\n\n| Model | Method | Size | xx->en | de->zh | xx->zh | xx->en | zh->de | xx->de |\n| ----- | ------ | ---- | ------ | ------ | ------ | ------ | ------ | ------ |\n| M2M   | Base   | 1.2B | 24.67  | 28.96  | 22.64  | 24.67  | 17.92  | 18.98  |\n| M2M   | LCCA   | 1.2B | 24.00  | 31.88  | 25.42  | 24.82  | 18.84  | 19.76  |\n| NLLB  | Base   | 1.3B | 37.00  | 26.86  | 24.87  | 37.00  | 17.97  | 22.82  |\n| NLLB  | LCCA   | 1.3B | 36.13  | 30.99  | 26.37  | 35.98  | 18.97  | 24.27  |\n| NLLB  | Base   | 3.3B | 38.43  | 26.80  | 26.13  | 38.43  | 19.42  | 24.57  |\n| NLLB  | LCCA   | 3.3B | 38.17  | 31.35  | 28.93  | 38.02  | 20.03  | 25.43  |\n\n(2) When de->zh arrives, the reduction in the effects you mentioned may not exist between M2M (1.05 improvement) and NLLB (1.27 improvement). Although the improvement of LCCA based on NLLB-3.3B is limited when zh->de arrives. We still achieved improvements in 41 zero-shot directions. The calculation process is as follows: (33.77 [positive] + (-8.37) [negative] / 49) = 0.86). And the average BLEU downplays the comparison. \n\n(3) The reason of better performance on mBART is that the mBART is a English-centric model, the newly available data plays an important role for non-English-centric directions. But M2M and NLLB are non-English-centric and have already encountered a majority of language pairs, making the impact of the newly available data less significant.\n\nWe will carefully revise Table 2 to make the paper clearer.\n\n> Q8: Without distillation, the models from the learning stage already achieves super great performance as shown in Table 3.\n\nAlthough adding extra parameters can improve performance, as pointed out by [3] and the above mentioned, there are limitations to this approach with **introducing extra parameters**. Moreover, the parameters will continue to increase when new languages continue to arrive. Therefore, we believe that maintaining the architecture of the original model is necessary. This leads us to propose a consolidation phase to preserve architecture. And our approach offers greater flexibility – we can decide whether or not to use the second stage. Each option has its pros and cons, and the choice can be made based on the user's actual needs.\n\n> Q9: This paper would benefit from a proof-reading.\n\nThank you for your comments. We appreciate your suggestion to have the paper proof-read for improvements. We will certainly take your questions into consideration and work towards enhancing the paper's overall quality.\n\n[1] Chen et al. Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation. In ACL 2022\n\n[2] Huang et al. Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation. In ACL 2023\n\n[3] Gu et al. Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions. In EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors introduce a new approach that facilitates continual learning for MNMT. Improvements were reported in both target-related supervised directions and zero-shot directions.\n", "reasons_to_reject": "1) Some statements in this paper are over-claimed.\n  * In line 105, \"... without introducing additional parameters.\" However, at the learning stage, a new feed-forward module is required to capture the new knowledge. \n  * In line 110, \"which improves performance for zero-shot translations in continual learning.\" This claim is too general: it implies that the proposed approach improves zero-shot translation. But the performance is mainly measured on the translation directions related to the language pairs of the new training data.\n2）The proposed approach is not well ablated. Current results don't demonstrate the necessity of the two-stage strategy.\n  * In Section 3.2, the authors argue that L_CTR could induce language-agnostic representations. But, in Table 3, we find that L_CTR benefits translation very little. Do we need it? \n  * The authors didn't explore the necessity of the separation of the two stages. What if removing the new learnable module and directly optimizing the FFN layer with L_CE and L_CKD-F? \n3) Some experimental results are hard to understand.\n  * In Figure 3, LCCA outperforms almost all efficient tuning methods in all conditions at the learning stage. But the learning stage only copies the last-layer FFN module. Why would it surpass Adapter and LoRA, particularly considering the latter two have more parameters? Is it because the initialization?\n  * In Table 2, as model becomes larger, the effectiness of LCCA reduces. It seems to suggest that model scaling might have a big role here and even change the conclusion.\n  * Without distillation, the models from the learning stage already achieves super great performance as shown in Table 3.\n4) This paper would benefit from a proof-reading.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on continual learning for multilingual neural machine translation (MNMT). To leverage new knowledge in new available parallel data as well as avoid catastrophic forgetting, the authors propose a two-stage approach: 1) encoding knowledge into a new module and 2) compressing and shifting the learned knowledge into the original model. Results with several open-source MNMT models show improved target-related supervised and zero-shot performance over a couple of strong baselines.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LtpF5Nn7tr", "reviewer": "Reviewer_ZXCV", "summary": "", "questions": "1. It is not clear in Section 3.3. e.g., \n\na)\thow to get equation 9 from equation 8, \n\nb)\twhat does $\\theta_f^*$ mean in equation 9\n\nc)\tIn equation 9, what does ‘i’ mean \n\nd)\tIn equation 9, during inference, which one is used, $\\theta_f$ or $\\theta_l$\n\nMore clarifications and comparisons are needed to evaluate the effectiveness of the proposed method. \n \n2.\tThe selections of baselines are not reasonable. The key topic in this paper is continuous learning, however, some baselines are unrelated while some baselines are missing. For example, SixT is not designed for the continuous learning. The methods proposed in [1] and [2] should be also considered as baselines. \n\n3.\tMore experiments should be added in the ablation study. For example, why is FFN selected as the learned module instead of other modules? What are the comparisons results among directly fine-tuning FFN, adapter-based, LoRA-based, prefix tuning method?  \n\n4.\tOnly one language pair de-zh is examined in this paper. More language pairs should be examined. \n\n\n[1] Overcoming catastrophic forgetting during domain adaptation of neural machine translation\n[2] Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation\n\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and insights.\n\n> Q1: It is not clear in Equation 9.\n\nWe can compute the information matrix to approximate the Hessian matrix with Equation 8 and the information matrix is Integrated into $\\theta^{*}_{f}$ in Equation 9. \n\nThe $\\theta^{*}_{f}$ means the original parameters of FFN with computing the information matrix, and the $i$ means the index of weight matrix. During inference, the $\\theta_l$ in the learning phase is integrated into the $\\theta_f$, thus we utilize the updated $\\theta_f$. We will revise Equation 9 to alleviate any confusion regarding this formula.\n\n> Q2: The selections of baselines are not reasonable. The key topic in this paper is continuous learning, however, some baselines are unrelated while some baselines are missing. For example, SixT is not designed for the continuous learning. The methods proposed in [1] and [2] should be also considered as baselines.\n\nWe choose the baselines based on the following reasons.\n\n(1) The key topic in this paper not only the continual learning but also the zero-shot translation (the setting of SixT). Thus, our scenario is an extension of SixT in the context of continual learning. And SixT also mentioned that the approach has the ability to avoid catastrophic forgetting. \n\n(2) The approach of cross-attention adds extra parameters after the encoder, which is not suitable for our scenarios. Further, our consolidation stage can not be directly applied to their method. \n\n(3) The differences between [1] and EWC are not significant, and we believe it won't affect the conclusion of our results. Due to the absence of open-source code and time limitation, we reproduce the method on de-to-zh and will add this baseline in future versions.\n\n| Method | xx->en | de->zh | xx->zh |\n| ------ | ------ | ------ | ------ |\n| EWC    | 23.01  | 29.93  | 19.47  |\n| [1]    | 22.11  | 30.12  | 18.53  |\n| LCCA   | 24.03  | 31.17  | 22.09  |\n\n> Q3: More experiments should be added in the ablation study. For example, why is FFN selected as the learned module instead of other modules? What are the comparisons results among directly fine-tuning FFN, adapter-based, LoRA-based, prefix tuning method?\n\n(1) **The reason for choosing the FFN** is that the FFN might be a core component. The development of Transformer-based language models unveils that the FFN stores the knowledge [2]. External knowledge is injected into FFN layers to enhance the performance of pre-trained language models. Therefore, we consider this aspect as crucial. Thus, we only introduce new parameters in the FFN. Huang et al. [3] also investigate the effectiveness of different modules.\n\n(2) We add an experiment to validate the effectiveness of our method at the learning phase, **compared with different PET-based methods**.\n\n| Method          | xx->en | de->zh | xx->zh | xx->en | zh->de | xx->de |\n| --------------- | ------ | ------ | ------ | ------ | ------ | ------ |\n| Fine-tuning FFN | 12.14  | 31.66  | 3.29   | 0.13   | 19.01  | 0.43   |\n| Adapter-Based   | 24.01  | 29.64  | 20.09  | 23.99  | 16.08  | 13.78  |\n| LoRA-Based      | 24 .00    | 28.79  | 19.45  | 24.26  | 16.70   | 13.12  |\n| Prefix-Based    | 23.32  | 26.89  | 19.39  | 23.46  | 15.96  | 12.85  |\n| LCCA (Ours)     | 24.03  | 31.17  | 22.09  | 24.81  | 18.99  | 18.13  |\n\nThe results show that the effectiveness of consolidation phase is related to the first learning phase. The higher performance during the learning stage, the better performance on the consolidation stage. Because the role of the consolidation stage is to collaboratively integrate the newly added parameters from the learning stage into the original model. Moreover, directly fine-tuning FFN may forgets previous knowledge, which makes it unsuitable in our scenario.\n\n> Q4: Only one language pair de-zh is examined in this paper. More language pairs should be examined.\n\n(1) In the test process, we **examine 100 translation directions** for a target language instead of only one language pair. \n\n(2) In the training process, we aim to continually improve performance for both new supervised and zero-shot translation. Therefore, the core idea is how to leverage abundant parallel corpora (X->Y) to enhance the performance of many-to-Y translation. In this paper, **we prodived two target languages with a Latin script and a Chinese script which are totally different**. And in the Limitation section, we have explained the limitations of the current data. We only utilize parallel data on one language pair for continual adaptation in this work. In the future, we will attempt to investigate the effectiveness of our method on extremely low-resource target language. \n\n> Q5: The equation 9 is similar with the equation 5 in [1]. The comparison with this baseline should be added\n\nWe have already chosen the [4] as a baseline and make a comparison in Table 1.\n\n[1] Thompson et al. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In NAACL 2019\n\n[2] Geva et al. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. In EMNLP 2022\n\n[3] Huang et al. Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation. In ACL 2023\n\n[4] Gu et al. Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions. In EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The research problem and proposed idea are interesting. ", "reasons_to_reject": "The paper needs more clarifications and experiments to better illustrate the details and prove the effectiveness of the proposed method.", "questions_to_authors": "1. It is not clear in Section 3.3. e.g., \n\na)\thow to get equation 9 from equation 8, \n\nb)\twhat does $\\theta_f^*$ mean in equation 9\n\nc)\tIn equation 9, what does ‘i’ mean \n\nd)\tIn equation 9, during inference, which one is used, $\\theta_f$ or $\\theta_l$\n\nMore clarifications and comparisons are needed to evaluate the effectiveness of the proposed method. \n \n2.\tThe selections of baselines are not reasonable. The key topic in this paper is continuous learning, however, some baselines are unrelated while some baselines are missing. For example, SixT is not designed for the continuous learning. The methods proposed in [1] and [2] should be also considered as baselines. \n\n3.\tMore experiments should be added in the ablation study. For example, why is FFN selected as the learned module instead of other modules? What are the comparisons results among directly fine-tuning FFN, adapter-based, LoRA-based, prefix tuning method?  \n\n4.\tOnly one language pair de-zh is examined in this paper. More language pairs should be examined. \n\n\n[1] Overcoming catastrophic forgetting during domain adaptation of neural machine translation\n[2] Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation\n\n\n", "paper_topic_and_main_contributions": "The paper focuses on the problem of continually updating MNMT models with limited new data. They propose a two-stage method consisting of a learning stage and a consolidation stage for continual adaptation (LCCA). The experiments show the effectiveness of the proposed method. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The equation 9 is similar with the equation 5 in [1]. The comparison with this baseline should be added. \n\n[1] Gu et al. 2022. Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions\n"}}]

Author rebuttal:

Meta-review: This paper proposes a new approach of continual learning in the context of multilingual neural machine translation (MNMT). It utilizes newly available parallel data for zero-shot translation without harming the already well-performed directions by a two-stage strategy: 1) learning new knowledge from the new data by introducing a new module, and 2) compressing and transferring the acquired knowledge back into the original model. The outcomes, as demonstrated using various open-source MNMT models, improve supervised and zero-shot translation, surpassing the performance of several strong baselines.

Pros: 
- To my knowledge, this represents a novel approach within the research domain of continual learning for MNMT. It offers several advantages, such as enhancing both zero-shot and supervised translations without introducing additional parameters post-training. These improvements are substantiated by strong results. The concept is innovative and intriguing. The experimental design demonstrates thoughtfulness and is supported by reasonable justifications.
- Should the paper incorporate the rebuttals, it would serve as an effective mean to enhance MNMT in light of newly available parallel data.
- The paper conducts a concise yet comprehensive literature review.
- It provides a thorough reporting and comparison of various baselines, resulting in a robust conclusion regarding the proposed method.
- Well-written.

Cons:
- There were some unclear details, but these have been thoroughly addressed in the rebuttal phase.
Decision: Accept-Main