MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks
Authors: Shangjie Li
Xiangpeng Wei
shaolin Zhu
Jun Xie
Baosong Yang
Deyi Xiong
Venue: EMNLP 2023 Main (2023)
Keywords: MoE
Multilingual Machine Translation
Modularizing

Abstract: Mixture-of-Experts (MoE) based sparse architectures can significantly increase model capacity with sublinear computational overhead, which are hence widely used in massively multilingual neural machine translation (MNMT). However, they are prone to overfitting on low-resource language translation. 
In this paper, we propose a modularized MNMT framework that is able to flexibly assemble dense and MoE-based sparse modules to achieve the best of both worlds. The training strategy of the modularized MNMT framework consists of three stages: (1) Pre-training basic MNMT models with different training objectives or model structures, (2) Initializing modules of the framework with pre-trained couterparts (e.g., encoder, decoder and embedding layers) from the basic models and (3) Fine-tuning the modularized MNMT framework to fit modules from different models together. We pre-train three basic MNMT models from scratch: a dense model, an MoE-based sparse model and a new MoE model, termed as MoE-LGR that explores multiple Language-Group-specifc Routers to incorporate language group knowledge into MNMT. The strengths of these pre-trained models are either on low-resource language translation, high-resource language translation or zero-shot translation. Our modularized MNMT framework attempts to incorporate these advantages into a single model with reasonable initialization and fine-tuning. Experiments on widely-used benchmark datasets demonstrate that the proposed modularized MNMT framwork substantially outperforms both MoE and dense models on high- and low-resource language translation as well as zero-shot translation. Our framework facilitates the combination of different methods with their own strengths and recycling off-the-shelf models for multilingual neural machine translation. Codes are available at https://github.com/lishangjie1/MMNMT.

MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks
Shangjie Li Tianjin University sj_li@tju. edu. cn
Xiangpeng Wei
Shaolin Zhu
pemywei@gmail.com com
Tianjin University
zhushaolinen inttju. edu. cn
Jun Xie stiffxj@gmail.comm com
Baosong Yang' bllzct..boosong@@mailccco com
Deyi Xiong'
Tianjin University
dyxiongêtju. edu. cn
Abstract
Introduction
Multilingual neural machine translation translates multiple languages within a single model via multi- task learning, facilitating the deployment of ma- chine translation service in practice and improving low/zero-resource language translation (Johnson et al., 2017; Aharoni et al., 2019; Zhang et al., 2020; Fan al., 2021; Team et al., 2022). How- ever, as the number of languages translated by an MNMT model increase, the capacity of the model has to increased accordingly, otherwise trans- lation quality will degrade for all language pairs, especially for high-resource languages. This phe- nomenon is referred to as capacity bottleneck (Aha- roni et al., 2019; Zhang et al., 2020). MoE models provide an effective way to increase model capac- ity while the computation cost is sublinear the number of parameters. Due to this advantage of MoE models over dense models, sparse architec- tures built on MoE models are widely explored for massively multilingual neural machine translation that requires large model capacity. However, MoL models suffer from overfitting on low-resource lan- guages (Team et al., 2022), which is not observed in MNMT built dense models.
Mixture-of-Expeee Efpee-tttsstsssos (MoE) based sparse archi- tectures can significantly increase model ca- pacity with sublinear computational overhead, which are hence widely used in massively mul- tilingual neural machine translation (MNMT). However, they are prone to overfitting on low- resource language translation. In this paper, propose a modularized MNMT framework that able to flexibly assemble dense and MoE- based sparse modules to achieve the best of both worlds. The training strategy of the mod- ularized MNMT framework consists of three stages: (1) Pre-training basic MNMT mod- els with different training objectives model structures, (2) Initializing modules of the frame- work with pre-trained couterparts (e.g., en- coder, decoder and embedding layers) from the basic models and (3) Fine-tuning the modular- ized MNMT framework
modules from dif- ferent models together. We pre-train three basic MNMT models from scratch: a dense model, an MoE-based sparse model and new MoE model, termed MoE-LGR that explores mul- tiple Language- Language--rroup-speeeee Group-specife Routers to incor- porate language group knowledge into MNMT. The strengths of these pre- pre-trained models are ei- ther on low-resource language translation, high- resource language translation zero-shot trans- lation. Our modularized MNMT framework attempts to incorporate these advantages into a single model with reasonable initialization and fine-tuning. Experiments on widely-use benchmark datasets demonstrate that the pro- posed modularized MNMT framwork substan- tially outperforms both MoE and dense models on high- and low-resource language translation well zero-shot translation. Our frame- work facilitates the combination different methods with their own strengths and recy- cling off-the-shelf models for multilingual neu- ral machine translation. Codes are available at https://git thub. www./owo/aaaaeecccc http/lissanngieee 1////NT... /MMNMT.
This inspires us ask question: can we achieve the best of both worlds of dense and MoE- based sparse architectures for multilingual NMT? To answer this question, we propose MMNMT that Modularizes Multilingual NMT with flexibly assembled dense and MoE blocks. Specifically, the training strategy of MMNMT consists of three stages:
Pre-training basic multilingual NMT models with different architectures, attempting to ex- plore the strengths of different models on low-resource language translation (c.g., dense models), high-resource language translation (c.g.. MoE models).
"Corresponding authors.
Initializing the modules of the proposed mod-

---

ularized MNMT model with blocks from the pre-trained basic multilingual NMT models, e.g., using the pre-trained dense encoder to initialize the encoder of MMNMT. Such ini- tialization can be done in a flexible module assembling way.
Experiments the OPUS-100 and PC32 dataset demonstrate that the proposed MM- NMT achieves significant improvements over both MoE and dense models on all language directions, especially on low-resource and zero-shot translation.
Fine-tuning the MMNMT model make the assembled modules fit together.
2 Related Work
Using dense modules to initialize MoE models in our training strategy is in line with our prelimi- nary experiments and recent studies on MoE-based language models. These studies find that training MoE-based language models from an off off-the-shelf dense model (c.g.. g.. T5 (Raffel et al., 2020)) is more efficient than training from scratch (Nie et al., 2022; Komatsuzaki et al., 2023). Our preliminary experi- ments also demonstrate that dense MNMT models are superior to MoE-based sparse models on low- resource language translation.
Multilingual Neural Machine Translation Mul- tilingual neural machine translation has been gain- ing increasing interest in recent years (Johnson et al., 2017; Aharoni et al., 2019; Zhang et al.. 2020; Fan al., 2021; Team al., 2022). How- ever, MNMT models tend to be inferior bilingual NMT counterparts on translating high-resoures lan- guages due to the capacity bottleneck (Zhang et al., 2020). A wide variety of approaches have been pro- posed to alleviate this issue, e.g., deepening mod- els to increase model capacity (Zhang et al., 2020), exploring lightweight language-specific modules (Philip et al., 2020; Zhang et al., 2021: Zhu et al., 2021), word alignments (Lin al., 2020), con- trastive learning (Pan et al., 2021) and Mixture-of- Experts (Lepikhin et al., 2020; Fedus al., 2022; Nie et al., 2022). Mixture-of-Expert based MNMT models usually suffer from overfitting problem (Team et al., 2022). Alleviating the overfitting prob- lem is a key focus of our proposed modularized MNMT framework.
In order diversify the options of pre-trained basic models and improve zero-shot translation in MMNMT, we further propose MoE-LGR that in- corporates a Language Group Router into MoF models. Routing mechanism plays an important role in token dispatch and resource allocation for MoE models. Previous works in this line mainly focus load balancing across experts in MoE (Lewis et al., 2021: Roller et al., 2021; Fedus et al.. 2022; Zuo al., 2022; Zhou et al., 2022) to pre- vent experts from being not specialized or overly specialized. However, the language information of each token is not fully explored in routers, which could enhance the generation ability of the decoder and mitigate the tfr-target problem in zero-shot translation. To address this issue, introduce MoE-LGR, Particularly, categorize languages into multiple groups according linguistic typol- ogy and language embedding clustering, and learn a router per language group to enhance the differ- ence of routing for different language groups.
Mixture-of-Expert Sparsely-gated Mixture-of- Experts (MoE) (Lepikhin et al., 2020; Fedus et al., 2022) selects the top-K experts in MoE layers through routing mechanism. A strand of re- search focuses on load balancing, such as Base Layer (Lewis et al., 2021), Hash Layer (Roller et al.. 2021), THOR (Zuo al., 2022), expert choice rout- ing (Zhou et al., 2022). In this work, we propose language group routing mechanism to provide language information and enhance the difference of routing across language groups, which signifi- cantly improves the performance of zero-shot trans- lation. Recently, Komatsuzaki et al. (2023) use a pre-trained T5 dense model to initialize an MoE language model, achieving improvements in the computational cost but no obvious performance gains. Our work explores modularized initializa- tion of different parts of MoE models with various pre-trained basic models for multilingual NMT, which flexible and effective.
The main contributions of this work are summa- rized as follows:
We propose MMNMT to modularize multi- lingual NMT, which is capable of assembling modules from both dense and sparse models and fitting them together to achieve the best of both worlds.
We present MoE-LGR with language group routers, which is able to significantly improve zero-shot translation in multilingual NMT.

---

Any2En
En2Any
Model
High
Medium
Low
Avg
High
Medium
Low
Avg
Dense
29.6
40.4
33.5
32.3
22.5
33.2
29.7
26.3
MoE
32.6
41.9
32.3
33.7
25.5
37.9
33.0
29.6
Table 1: BLEU scores of the dense
MoE model on both Any-to-English and English-to-Any translation.
2.1
arabmurat aryzin lon #7287y hig
e1/0m Jeys
1.
19
srylle loa
#12ay Ni1
1.9
3.6
15-
14
26
14
20
Taaiing Stope
1shing Sroce
(a) The dense model
(b) The MoE model
Figure 1: The validation loss of the dense model and MoE model during training.
3 Preliminary Experiments and Findings
MoE and dense models. The training strategy of MMNMT consists of basic model pre-training, module initializing and fine-tuning.
To have a deep understanding on the strengths and weaknesses of dense and MoE models in multilin- gual NMT, conducted preliminary experiments compare them. Details of these experiments can be found in section 5.1. Results are shown in Ta- ble 1. We observe that the MoE model significantly outperforms the dense model on bigh-resource lan- guage translation, and achieves consistent improve- ments over the dense model on all En-to-Any di- rections. This demonstrates the advantage of MoE sparse models over dense models in terms of model capacity. However, also find that MoE models are prone overfitting on low-resource language translation, which resonates with the finding of (Team al., 2022). In Any-to-English translation, the dense model outperforms the MoE model by 1.3 BLEU (33.5 vs. 32.3) on low-resource language tanslation. In addition to this, we plot the valida- tion loss of the dense model and MoE model during training in Figure 1. It obvious that the validation losses of both mny-to-English and English-to-AAn low-resource language translation initially decrease and later increase, confirming the existence of the overfitting problem.
4.1 Basic Model Pre-training
We introduce three types of basic models for our general framework, namely dense model, MoE model and the proposed MoE-IGGR model. These basic models are trained from scratch with cross- entropy objectives on training data.
Dense Model The dense model is encoder- decoder backbone network contains 12 Trans- former encoder blocks and 12 Transformer decoder blocks. The encoder and decoder have shared embedding layer.
The MoE model substitutes the feed-forward network (FFN) sublayer in the Dense model with an MoE layer that consists of multiple FFN experts [FFN,) to expand model capacity. The MoE model uses a token router p (typically a Top-K gate) to perform token dispatch to experts.
MoE Model
To explore language information and improve zero-shot translation capability in the de- coder, we add multiple language-group-oee group-specific routers (one router per language group) into MoE layers of the decoder to enhance the difference of routing across language groups.
MoE-LGR
4 Methodology
To mitigate the overfitting issue and achieve the best of both worlds, we propose general frame- work MMNMT to modularize multilingual NMT so as to assemble desirable modules from both
The inputs to the language group router are a token representation x and a language ID lid. A language group will automatically identified

---

according the language ID via the language group identification operation LGI. The shared router and the identified language-grouppe -pocp-picicii router is aggregated to yield the output of the language group router, which is computed as follows:
Roxting Thevitddss
Shored
Language Escale Radars
Lergunge
Icenansstion
W sharee + W WLGI(id)
Takan Reprosstaaior
p(x) TopK
(1)
Figure 2: Diagram the proposed language group router.
where W share is the weight of share router across all languages, WLGI(Hd) the weight of the language-group group-sppcciiieeeeee router identified by LGI(lid).
Add-Nerraloe
For language groups, we use both external lin- guistic typology knowledge and internal language token embeddings to group languages. First, we categorize languages into multiple groups follow- ing external linguistic typology from Ethnologue (Lewis et al., 2009). which is one of the most authoritative language family taxonomy. Specifi- cally, as we use the OPUS-100 (Zhang et al., 2020) dataset in our experiments, which is English-centric and contains 99 language pairs (as the test sets in OPUS- 100 only cover 94 language pairs, we only use 94 language pairs in supervised training), we divide 95 languages (including English) into 22 groups according Ethnologue. The majority of languages are from the Indo audo-eur family. Some languages are assigned to a group with only one language of their own. For example, Thai is from the Kai-K group and no other languages in the OPUS-100 dataset are assigned to this group. In order to balance the corpus size across groups, need to further restructure language groups ac- cording language similarity. Language embed- ding based clustering (Tan al., 2019) solu- tion to automatically cluster languages into groups by using language token embeddings learned by MNMT. However, this method may generate low- quality language clusters due to low quality of low- resource language embeddings. Hence, we cluster languages on the basis of linguistic typology (i.e., based 22 groups defined in Ethnologue). We calculate group embeddings follows:
NAE
Layel
Light
Figure 3: Illustration of the initialization from the coder the pre-trained dense model to the encoder of MMNMT. The decoder initialized in the way, which is omitted here.
In this way, the impact of the low-quality lan- guage embeddings of low-resource languages is reduced. Then, we perform hicrarchical clustering with computed group embeddings and set a maxi- mum group corpus size Tmax to prevent excessive clustering. Specifically, let M the number of the ini- tial groups after the linguistic typology knowledge based grouping. In each iteration, we merge the two closest groups a and whose total corpus size is less than Tiaxx removing the two old groups and forming a new group. After this, the number of groups decreases from M to M-1. The group em- bedding of the new group is obtained by weighting the old group embedding based the corpus size:
Ta E. + Tb ED
EG
Σ IEG
T
E
(2)
Enew
T. + Tb
(3)
TC
where Eg is the group embedding, E, the lan- guage embedding of language group G, Ti
When grouping no longer changes, we obtain the final grouping result. In this way, we obtain 4
is the corpus size of the language i, TG is the total corpus size of all languages in group G.
groups for OPUS- 100 dataset at last. Please refer to Appendix A for more details.

---

Wall Time
Any2kn
Em2Any
Zero-Shof
ID
Model
(Hour)
High
Medium
Avg
High
Medium
Avg
BLEC
LangAcc(S:)
Basic Models
(1)
28
Dense
29.6
40.4
33.5
32.3
22.5
33.2
29.7
26.3
4.0
16.2
(2)
79
Dense
30.1
40.9
33,5
32.7
23.1
33,9
30.4
26.9
4.2
16.5
79
MoE
32.6
41.9
32.3
33.7
25.5
37.9
33.0
29.6
5,4
24.7
158
MoE
32.9
42.5
32.5
34.1
25.8
38.5
33,5
30.0
5.3
24.0
79
McE-LGR
32.5
41.7
31.9
33.5
25.5
38.0
32.8
29.6
7.1
32.9
158
MoE-LGR
33.0
42.3
32.3
34.0
25.9
38.5
33.4
30.0
7.2
34.2
MMNMT
Models
(7)
28+79
Ene(y DoGran
32.8
43.3
35.1
34,9
25.5
37,9
33.6
29.9
4.6
17.2
(8)
28+130
Ene()) Decrand
33.0
43.6
34.9
35.0
25.6
38.5
33,9
30.1
4.7
20.1
(
28+79
Ene()) Dec())
32.7
42.9
34.3
34.5
25.3
38.0
34.4
30.0
5.9
27.6
(10)
28+130
Enc(1) (ee(1)
32.9
43.1
34.2
34.6
25.6
38.1
34.6
30.2
6.2
29.8
(11)
79479
Ens() Dee(s)
33.3
42.1
32.2
34.0
26.2
33.2
30.2
7.0
32.6
(12)
79+79
()c()) Dec(s)
33.3
42.4
32.9
34.3
26.4
38.8
33.2
30.3
9.6
44.9
Table 2: Results of different models on the OPUS-100 test set. A subscript indicator denotes that the corresponding module of the MMNMT model initialized by the counterpart the basic model indexed by the indicator. LangACC: the proportion of translations in the correct target language among all translations.
4.2 Module Initialization
dense model and decoder of the MoE-LGR model to initialize the MMNMT model, which could improve the ability of low-resource language translation as well as zero-shot translation.
Once basic models are pre-trained, we can use pa- rameters of these pre-traine pre-trained models to initialize modules of the general MMNMT framework the entire framework in different ways.
4.3 Fine-tuning
Initialization From Dense Model When we use the pre-trained dense model to initialize MMNMT, we perform initialization in layer-wise way. As shown in Figure 3, we initialize the parameters of MMNMT layers with those of the dense model layers that have the same structure as the corre- sponding MMNMT layers, e.g., self-attention layer, cross-attention layer, embedding layer. For FFN ex- perts initialization, we randomly masks parameters of the corresponding FFN in the dense model and choose initialize FFN expert from MMNMT in the same layer with the remaining parameters, In doing so, we can enhance the diversity of experts (Nie et al., 2022).
After initializing the MMNMT model, need to further fine-tune the entire model on the same training data used in the basic model pre-training to ensure that the various modules in the model co- operate with each other, especially under the mixed initialization condition. We fine-tune the MMNMT model with cross-entropy and load-balance objec- tives.
Experiments
We used the MoE branch of fairseq' to implement our MoE models. We conducted extensive experi- ments to examine the effectiveness of the proposed MMNMT against MoE and dense model baselines in multilingual machine translation.
Initialization from MoE Model This be done in straightforward way as they have the same architecture. The initialization performed for the entire model.
5.1 Experiment Setting for Many-to-Many Translation
Dataset We used the publicly available OPUS- 100 dataset which contains approximately 55M sentence pairs and 99 language pairs in our ex- periments. Since the test sets in OPUS- 100 only covers 94 language pairs, we only used the data of these 94 language pairs for training. We employed the temperature-based sampling strategy (Aharoni et al., 2019) with T = to balance corpus size for various language pairs. We used SentencePiece
Initialization from MoF-LGR Model As the MoE-LGR has multiple routers in the decoder, we need add multiple Bunguagc-group-spcciff routers in the decoder of MMNMT accordingly to accommodate the initial- ization.
Mixed Initialization
To combine the strengths of these basic models, we can perform mixed initial- ization. For example, we can use the encoder the

---

(Kudo and Richardson, 2018) for tokenization and the vocabulary size was 64K.
Results are shown in Table 2. The Wall Time is the training cost of models. The dense model and MoE model require 28 and 79 hours to com- plete 100K training steps respectively. For a fair comparison, we also provide results for a dense model that continues to train in 79 hours with 260K steps. The wall time of MMNMT model is in x+y format, x denotes the pre-training time of the cor- responding basic model and y is the time of fine- tuning. We also extend the training time of some MMNMT models to conduct fair comparisons in terms of training hours. It can be seen that the MoE-LGR model is on a par with the MoE base- line on Any-to-Enn and En-to-Any translation, but achieves substantial improvements of 1.9 BLEU and 10.2% LangAce over the MoE model on zero- shot translation (see (4) and (5) in Table 2). This is in line with our motivation in developing the MoE-LGR model to improve zero-shot translation routers. All MMNMT
Our dense MNMT model contained 12 Transformer encoder blocks and 12 Transformer decoder blocks, where the model di- mension was 1024, the number of attention heads was 8. For the MoE model, MoE layer frequency of the MoE MNMT model was 2, the capacity factor was 1 1.25 in training and the eval-capacity- factor was set to 0.75 in inference. Every MoE layer had 32 experts and Top-2 routing strategy was used for expert routing. Other configurations were the same the dense model. For the MoE-LGR model, we set max 15M and used the language embeddings learned by the dense model. The 95 languages were finally clustered into language groups
Model Configurations
Training Configurations We conducted all ex- periments on 8 NVIDIA Tesla V100 32GB GPUs. For basic model pre-training and MMNMT fine- tuning, we used the Adam optimizer (Kingma and Ba, 2014) with learning rate = 2c 4 a = 0.9, B 0.98. The learning schedule was Polynomial de- cay with the number of warm-up steps being set to 4000, the end learning rate was 1e-5, and the total number of updates was 100K. We set the training max tokens 4096 per GPU and accumulated the gradients every 4 steps. The training objects were cross-entropy loss and foad-balance loss with a weight of 1. The best checkpoint was selected ac- cording to the perplexity (ppl) on the validation set, The BLEU score was computed via sacrebleu (Post, 2018), and we employed langdetect toolkit for lan- guage identification the OPUS- 100 dataset to calculate language accuracy of target translations.
via tanguage-group-ppee models with different assembled modules are supe- rior to both dense and MoE models. The MMNMT model Enc(1) Decraud- where the encoder is ini- tialized with the encoder of the dense model and the decoder is randomly initialized, achieves average improvement of 0.9 BLEU over the MoE model (see (4) and (8) in Table 2). The initializa- tion from the encoder of the dense model signifi- cantly improves low-resource language translation by 2.4 BLEU on Any-to-En translation over the MoE model and is on par with it on En-to-Any translation. Enc(()) Dec(1) which uses the pre- trained dense model to initialize both encoder and decoder, outperforms the MoE baseline by erage of 0.5 BLEU Any-to-English translation and 0.2 BLEU on English-to-Any translation. Es- pecially, E)c(1) Dec(1) achieves 34.6 BLEU on English-to Any Low-resource language translation.
5.2 Main Results
The Enc(1) - Dec(3) and Enc(1) Dec(s) model achieve significant improvements on high- language translation. Specifically, Enc(1) Dec(s) outperforms the MoE baseline by 0.4 and 0.6 BLEU on Any-to-English and English-to-Any high- resource language translation. These results sug- gest that the proposed MMNMT initialized with blocks from basic models is able to benefit from the advantages of these basic models.
In order to examine the effectiveness of different models, we evaluated the translation performance on the OPUS-100 test set and OPUS- 100 zero-shot test set. We grouped languages into three categories according to the size of training data available for them in the training dataset: High (500K size < IM, 100 directions), Medium (200K < size < 500K, 24 directions) and Low (< 200K, 64 direc- tions). The zero-shot test set covers six languages (Arabic, Chinese, Dutch, French, German, and Rus- sian) and 30 zero-shot directions.
5.3 Zero-Shot Results
Zero-shot translation enabled by multilingual NMT usually suffers from the off off-target translation issue

---

180
-1
-5
130
(a) The dense model
(b)
The MoE model
(c) The MMNMT model
Figure 4: Visualization of the encoder top layer representations of different models.
1.90
visualize the representations of the top layer of the encoder Figure 4. We used the Flores-200 (Team et al., 2022) dataset, which is a many-to- many multilingual benchmark including 204 lan- guages. We selected high-resource languages (German, French, Russian and Chinese) and 4 low- resource languages (Afrikaans, Amharic, Tamil and Telugu) in Flores-200, and encoded all sen- tences by the encoder of three different models: (a) the dense model (b) the MoE model trained from scratch and (c) the MMNMT (Dense) model (i.e., Eue(1) Decrand in Table 2). We averaged the sequential representations of the sentences from the above 8 languages over the sequence dimen- sion, and applied the t-SNE (Laurens and Hinton, 2008) dimensionality reduction algorithm to reduce the 1024 dimensions two dimensions. Then we ploted the bivariate kernel density estimation based on the reduced 2-dim representations. According to Figure 4, the encoder top layer representations of the dense model are more compact than those of the MoE model. Russian representations seem to over- lap with those of German, French, and Afrikaans. The MMNMT (Dense) model whose encoder is ini- tialized by the encoder of the dense model, shows obviously better unified representations than the other models. All 8 languages are close to each other in the shared space and gradually merged into a cluster.
1.5
28
Figure 5: The validation loss of the MMNMT model
during training.
(Zhang et al., 2020), where tokens in target translation are not in the right language. We evalu- ated baselines and our methods on OPUS-100 zero- shot test set (Zhang et al., 2020). Results are shown in Table 2. The MoE model achieves 5.3 BLEU and 24.0% language accuracy, slightly mitigating the off-target problem compared with the dense model. The Moli-LGR model improves BLEU and language accuracy from 5.3 7.2 and 24.0% to 34.2% respectively. The language group router en- courages similar token routing behavior within a language group. This makes the token-to-experts assignment in the decoder more language-specifis therefore improving the accuracy of language gen- eration. The Enc(1) Dec(s) model achieves the best result zero-shot translation, improving the BLEU score from 5.3 to 9.6 and language accuracy from 24.0% to 44.9%. This model initializes de- coder with the decoder of MoE-LGR, which itself achieves good performance on zero-shot transla- tion.
We also plot the validation loss curve of the MM- NMT (Dense) model during training in Figure 5. We observe an overall decrease in the validation loss of the MMNMT model compared with that of the MoE model ((b) in Figure 1), especially on low- resource language translation. This explains the significant improvements gained by the MMNMT model on low-resource language translation.
5.4 Analysis on the Encoder Representations and Validation Loss Curve
order to take a deep look into the improvements obtained by the initialization from the dense model,

---

0.00
0.06
0.04
0.04
20
0.02
(a)
(b)
0.08
0.08
0.00
0.05
10
15 20 25
30
10 15 20 30
(c)
(d)
Figure 6: The texpeesss assignments of the last MoE layer of the decoder for the MoE model and the MMNMT (MoE-LGR) model. (a): zero-shot translation by the Moï. model. (b): X-to-English translation by the MoB model. (c): zero-shot translation by the MMNMT model. (d): X-to-English translation by the MMNMT model. Brightness represents the probability of experts being routed.
5.5 Analysis on the Token-to-Experts Assignments the Decoder
5.6 Adapting Off-the-Shelff Dense Checkpoints to Sparse Architectures Via MMNMT
A wide variety of models have been proposed to im- prove dense multilingual machine translation in the past few years (Johnson et al., 2017: Aharoni et al., 2019; Zhang et al., 2020; Lin et al., 2020; Pan al., 2021). Our framework facilitates the adaptation of these off-the-shelf dense checkpoints to MoE- based sparse architectures, the current dominant formalism for massively multilingual NMT. We consider public MNMT model checkpoints with available multilingual parallel training data. For this, we choose mRASP2 (Pan et al., 2021) for our dense-to-sparse adaptation experiments with off the--sh--eff models.
To analyse the improvements of the MMNMT (MoE-LGR) model (i.e., Enc(1) Dec(G) in Ta- ble 2) gained on zero-shot translation, we report detailed results of zero-shot translation in Table Compared with MoE baselines, the MMNMT (MoF-LGR) model improves BLEU scores by an average of 4.2 and language accuracy by an aver- age of 20.3%. Arabic, Russian, and Chinese have the most noticeable improvements, which may be due to the fact that these languages not cate- gorized into the same group English, making them more differentiated in routing and effectively alleviating the off off-target issue. We visualize the token-tor dolen-to-cxye tx-expsrts assignments zcro-shot transla- tion in the last MoE-layer of the decoder for MoE and MMNMT (MoE-LGR) respectively in Figure 6, where the x-axis represents 32 experts, and the y- axis denotes the target language (e.g., the first row in (a) is the average loken-to-experts assignments of De-Ar, Fr-Ar, NI-Ar, Ru-Ar, Zh-Ar). As zero- shot translation usually suffers from the off-target issue (i.e., easily translated into English), we used the same source sentences to translate them into English to observe changes in token-to-experts as- signments. We observe that the MoE model has a zero-shot
Experiment Settings We used PC32 dataset (Lin et al., 2020) which is used by mRASP2. PC32 is a English-centric multilingual parallel corpus which includes 32 language pairs and 64 translation direc- tions in total. The details of the dataset, model and training configuration can be found in Appendix B.
Results Results are shown in Table 3. We com- pared with five baselines, and the numbers in bracket are the number model parameters and activated parameters at inference time. The dense and MoE models were trained on the PC32 dataset from scratch, and their model configuration were the same mRASP2. mRASP2 (Pan et al., 2021), M2M- 100 (Fan al., 2021) and NLLB (Team al., 2022) are all publicly released MNMT dense checkpoints, which were downloaded and evalu- ated without any changes. We applied our frame- work to both dense and mRASP2 models to obtain MMNMT (Dense) and MMNMT (mRASP2) re-
similar token-to-expers assignments and X-to-English translation (see (a) and (b) in Figure 6), while MMNMT learns different assign- ments (see (c) and (d) Figure 6). Our proposed routers route tokens from different language groups to different experts, miti- gating the off-target issue of zhro translation.

---

Any2En
Em2Any
Model (#params/ #activated params)
High
Medium
Low
Fx-L
Avg
High
Medium
Low
Fx-Low
Avg.
Baselines
Dense (0.4B / 0.4B)
29.9
28.1
24.2
15.0
25.5
37.7
22.3
16.6
8.7
20.1
MoE (3.5B/ 005B)
31.5
30.4
26.5
15.8
27.5
39.5
24.7
18.7
11.2
22.4
mRASP2 (0.4B/ 0.4B)
30.0
30.7
26.1
17.5
27.7
40.3
25.6
19.8
11.2
M2M-100 (1,2B/ 1.2B)
30.0
27.3
22.3
17.0
24.8
34.6
22.2
16.3
12.7
20.3
NLLB (1.3B/ 1.3B)
32.3
33.1
31.2
28.7
31.9
36.3
26.6
21.7
19.8
24.6
MMNMT
MMNMT (Dense) (3.58 /0.58)
32.1
31.0
27.4
17.0
28.3
39.8
24.8
19.3
10.7
22.6
MMNMT (mRASP2) (3.5B/ 0.5B)
32.9
32.1
28.8
18.8
29.5
41.0
26.1
20.4
12.3
23.9
Table 3: Results of the off-the-shell densc-to-spare adaptation experiments on the test sets of PC32.
Model
Ar
Zh
Avg
MoE
5.7(52.9%
5.6(25,49) 4%)
7.0(22.8%)
5.2(17.4%8
5.1(17.4%)
5.3(24.0% 9.1(42.4%)
3.2(8.0%)
MMNMT
12.0(80.9%)
7.8(35.0%)
10.2(36.9%)
6.8(25.1 1%)
11.7(49.4%)
9.6(44.9%)
Table Detailed results of zero-shot translation. The numbers outside the brackets are the BLEU scores, and the numbers in the brackets are the language accuracy.
spectively, where both the encoder and decoder of the adapted models were initialized by the counter- part modules from the corresponding models. As we can see in Table 3, mRASP2 is a very strong baseline and is consistently better than both dense and MoE baselines. MMNMT (Dense) achieves consistent improvements compared with MoE base- lines Any-to-English translation (28.3 27.5). MMNMT (mRASP2) can further significantly im- prove translation performance over mRASP2, es- pecially on Any-to-English translation (29.5 vs. 27.7). On English-to-Any translation, MMNMT (mRASP2) gains an average improvement of 0.7 BLEU and substantial improvement of 1.1 BLEU extremely low-resource language translation. These results suggest that our modularized MNMT framework can be successfully applied off-the- shelf MNMT models to recycle and upgrade them. Our MMNMT model achieves comparable perfor- mance with NLLB-1.3B on low-resource language translation. NLLB-1.31 uses far more training data than ours (only 108K sentence pairs in total) on extremely low-resource languages. It is hence rea- sonable that model cannot compete with it these languages. Note that framework can also use the modules of NLLB-1.3B achieve further performance improvements.
both worlds for high/low-resouree language trans- lation and zero-shot translation. Experiments and in-depth analyses demonstrate that our framework combined with diffcrent modules significantly out- performs the MoE and dense baselines on high- and low-resource language translation as well zero-shot translation.
Limitations
The MMNMT model is able to incorporate differ- ent modules from different basic models to com- bine their strengths. However, these modules might be not consistent with each other, making them not able to fit together in single model. We leave this to future work for exploring strategies.
Acknovledgments
The present research was partially supported by the Key Research and Development Program of Yunnan Province (Grant No. 202203A We would like to thank the anonymous reviewers for their insightful comments.
References
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
6 Conclusion
In this paper, we have presented MMNMT, mod- ularized multilingual neural machine translation framework that is capable of flexibly assembling both dense and sparse blocks to achieve the best of

---

Angela Fan, Shruti Bhosale. Holger Schwenk, Zhiyi Ma, Ahmed El- Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Ar- mand Joulin. 2021. Beyond english-centric multilin- gual machine translation. Journal of Machine Learn- ing Research, 22(107):1-48.
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu. Jiangtao Feng, Hao Zhou, and Lei Li. 2020. Pre- training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Langwage Processing (EMNLP), pages 2649- 2663, Online. Association for Computational Lin- guistics.
William Fedus, Barret Zoph. and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39.
Xiaonan Nie, Shijie Cao, Xupeng Miao, Lingxiao Ma. Jilong Xue, Youshan Miao, Zichao Yang, Zhi Yang. and Bin CUI, 2022. Dense-to-sparse gate for mixture- of-experts.
Xiao Pan, Mingxuan Wang. Liwei Wu, and Lei Li. 2021. Contrastive learning for many -o--y--toonnee to---tt---m many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- rational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 244-258, Online. Asso- ciation for Computational Linguistics.
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: En- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351. -311.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980.
Jerin Philip, Alexandre Berard, Mathhas Gallé, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4465 4470, Online. Association for Computational Linguistics.
Aran Komatsuzaki, Joan Puigcerver, James Leorph Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslic, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. Sparse upeycling: Training mixturs-of-eeperts from dense checkpoints. In The Eleventh International Conference on Learning Representations.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguisties.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text- to-ext transformer. Journal of Machine Learning Research, 21(140):1-67.
Van Der Maaten Laurens and Geoffrey Hinton. 2008 Visualizing data using t-sne. Journal of Machine Learning Research, (7665))2579-2665
Stephen Roller. Sainbayar Sukhbaatar, Arthur Szlam, and Jason E Weston. 2021. Hash layers for large sparse models. In Advances in Neural Information Processing Systems.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional com- putation and automatic sharding.
Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tic-Yan Liu. 2019. Multilingual neural machine translation with language clustering. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 963-973, Hong Kong, China. Association for Computational Lin- guistics.
M Paul Lewis, Gary F Simons, and Charles D Fennig. 2009. Ethnologue: languages of the world, dallas, texas: Sil international. Online version: http://www. ethnologue. com, 12112).0010.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. Base layers: Simplifying training of large, sparse models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265-6274 PMLR.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heañell, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti,

---

John Hoffman. Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Sehwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share not? learning schedule language-specific capacity for multilingual transla- tion.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 1628- 1639, Online. Association for Computational Linguis- tics.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y Zhao, Andrew Dai, Zhifeng Chen, Quoc V Le, and James I Laudon. 2022. Mixture- of-experts with expert choice routing. In Advances Neural Information Processing Systems.
Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingx- uan Wang, and Lei Li. 2021, Counter-interfernnce adapter for multilingual machine translation. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 2812-2823, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations.

---

A Languages Grouping Result
We built language tegguage-rouu--ye group-specific routers for MoE following section 4.1, the grouping results are shown in Table 5.
B
The Experiment Setting for mRASP2 Adapting Experiments
Dataset We downloaded the binary file of PC32 dataset without RAS and checkpoint from the project website". We grouped these languages into four categories (excluding English-Mongolian lan- guage pair) according their corpus size: High (IOM - size, 4 directions), Medium (1M < size 10M, directions), Low (100K Σ size IM, 16 directions), Extremely Low (size < 100K, 10 direc- tions). We omitted Maltese and Esperanto to make all multilingual translation models comparable in test. We report the validation and test set used in mRASP2 adapting experiments in Table 6.
Our dense MNMT model contained 12 Transformer encoder blocks and 12 Transformer decoder blocks, where the model di- mension was 1024, the number of attention heads was 16. The position embeddings of encoder and decoder were learned during training. For the MoE model, MoE layer frequency of the MoE MNMT model was 2, the capacity factor 1.25 in train- ing and the eval-capacity-factor inference. Every MoE layer had 32 experts and Top-2 routing strategy was used for expert routing. Other configurations were the same as the dense model.
Model Configurations
set to 0.75 in
Training Configurations We conducted all ex- periments 8 NVIDIA Tesla V100 32GB GPUs. For dense model pre-training and MMNMT fine- tuning, we used the Adam optimizer (Kingma and Ba, 2014) with learning rate 2c -4 = 0.9, 3 0.98. The learning schedule was inverse sqrt decay with the number of warm-up steps being set to 4000. The total number of updates was 100K. We set the training max tokens to 2000 per GPU and accu- mulated the gradients every 23 steps. The training objects were cross-entropy loss and load-balance loss with weight of 0.1. The best checkpoint was selected according to the perplexity (ppl) on the validation set. The BLEU score was computed vias sacrebleu (Post, 2018).

---

Groups
Languages
2
3
4
Table 5: Grouping results on the OPUS- 1000 dataset.
Lang Pair
Validation set n
Test set
en-fr
newstestl 14 newstest14
en-de
newstest13
en-zh
newsdev11 17
newstest17
en-ro
newsdev 16
newstest16
en-cs
newstest15
newstest16
en-tr
newsdev16 16
newstest16
en-ru
newstest18
newstest19
en-fi
newstest16
newstest17
en-es
newstest12
newstest13
en-it
newssyscomb2009
newstest2009
others
OPUS-100--vaid
OPUS-100-test
Table 6: The validation and test used in mRASP2 adapting experiments


Reviews:
[{"review_id": "tXKcNtQC3F", "reviewer": "Reviewer_yF46", "summary": "", "questions": "1.\tThe authors should provide more details on comparison of the size of the models. Do the performance gains result from the larger size of the model? \n2.\tCan the proposed modularized structure generalize to other multilingual tasks (e.g., XNLI and XQuAD)?\n3.\tOne of the main idea of this paper is to add language-specific modules to improve the multilingual performances, which is similar to the recent works (e.g. [1]). The authors should explain the differences to show the uniqueness of the proposed ideas. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and valuable suggestions. Our answers to your questions and comments are as follows.\n\nAnswer to RejectionReason1: \n\nWe have recently conducted experiments against 2 additional strong baseline models for a more in-depth comparison, namely NLLB-1.3B[1] and M2M-100[2]. The new results are as follows:\n|            Model (#params / #activated params)             |      |        | Any2En |        |           |      |        | En2Any |        |      |\n|:---------------------------:|:----:|:------:|:------:|:------:|:---------:|:----:|:------:|:------:|:------:|:----:|\n|                             | High | Medium |   Low  | Ex-Low |    Avg    | High | Medium |   Low  | Ex-Low |  Avg |\n|                             |      |        |        |        | Baselines |      |        |        |        |      |\n|     Dense (0.55B / 0.55B)     | 29.9 |  28.1  |  24.2  |  15.0  |    25.5   | 37.7 |  22.3  |  16.6  |   8.7  | 20.1 |\n|       MoE (3.7B / 0.65B)      | 31.5 |  30.4  |  26.5  |  15.8  |    27.5   | 39.5 |  24.7  |  18.7  |  11.2  | 22.4 |\n|     mRASP2 (0.55B / 0.55B)    | 30.0 |  30.7  |  26.1  |  17.5  |    27.7   | 40.3 |  25.6  |  19.8  |  11.2  | 23.2 |\n|     M2M-100 (1.2B / 1.2B)     | 30.0 |  27.3  |  22.3  |  17.0  |    24.8   | 34.6 |  22.2  |  16.3  |  12.7  | 20.3 |\n|       NLLB (1.3B / 1.3B)      | 32.3 |  33.1  |  31.2  |  28.7  |    31.9   | 36.3 |  26.6  |  21.7  |  19.8  | 24.6 |\n|                             |      |        |        |        |   MMNMT   |      |        |        |        |      |\n|  MMNMT (Dense) (3.7B / 0.65B) | 32.1 |  31.0  |  27.4  |  17.0  |    28.3   | 39.8 |  24.8  |  19.3  |  10.7  | 22.6 |\n| MMNMT (mRASP2) (3.7B / 0.65B) | 32.9 |  32.1  |  28.8  |  18.8  |    29.5   | 41.0 |  26.1  |  20.4  |  12.3  | 23.9 |\n\nThe numbers in bracket are the number of model parameters and activated parameters at inference time. It is worth noting that the newly compared models have larger model size, larger training data and longer training time than our models, while our method improves translation quality to a level comparable to them.\n\nOur MMNMT model achieves comparable performance with NLLB-1.3B on low-resource languages translation. NLLB has far more training data than ours (only 108K sentence pairs in total) on extremely low-resource languages. It is hence reasonable that our model cannot compete with it on these languages. Note that our framework can also use the modules of NLLB-1.3B to achieve further performance improvements. We’ll provide these results in the next version.\n\nAnswer to question 1: \nNo, the performance gains are NOT from the larger size of the model. We compared MMNMT with the MoE model of the same size, and achieved a significant improvement on the performance of low-resource and high-resource language translation. In addition, although the MoE model is larger, the number of model parameters activated at inference time is much smaller than those of baselines (e.g., 0.65B vs 1.3B). Hence, the performance gains do not result from the larger size of the model. We will conduct more experiments on model size and provide results in the next version.\n\nAnswer to question 2:\nYes. Although our work currently focuses on the multilingual translation task, we believe that as the size of the model becomes larger, the problem of overfitting on low-resource languages is widespread. For this, our method can be effectively used to improve other multilingual tasks including XNLI and XQuAD. We’d like to extend our approach to these tasks in our future work.\n\nAnswer to question 3:\nWe add language-group-specific routers to route tokens from different language groups to different experts. Different from other methods which add language-specific modules (Could you please provide the paper title of the reference [1] in your question 3 ?), our method only imposes a soft constraint on routing. Tokens can still be routed to any appropriate experts, which avoids the conflict between different languages in a gentler way and will not damage the capacity of MoE models. Compared with other methods, our method only needs to add a very small number of model parameters (~1M). We’ll explain the differences from recent works as you suggested in the next version.\n\n[1] Costa-jussà M R, Cross J, Çelebi O, et al. No language left behind: Scaling human-centered machine translation[J]. arXiv preprint arXiv:2207.04672, 2022.\n\n[2] Fan A, Bhosale S, Schwenk H, et al. Beyond english-centric multilingual machine translation[J]. The Journal of Machine Learning Research, 2021, 22(1): 4839-4886.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper is well-structured with pilot experiments to clear illustrate their motivations. And the main ideas of this paper are simple and easy to follow.\n2.\tExtensive experiments are conducted to prove the effectiveness of MMNMT and they well support the listed challenges. \n3.\tThe proposed framework is proved to be flexibly adapted to the off-the-shelf models to support the recycling.", "reasons_to_reject": "1.\tThe authors should include more sota baselines in MNMT to make the comparisons. ", "questions_to_authors": "1.\tThe authors should provide more details on comparison of the size of the models. Do the performance gains result from the larger size of the model? \n2.\tCan the proposed modularized structure generalize to other multilingual tasks (e.g., XNLI and XQuAD)?\n3.\tOne of the main idea of this paper is to add language-specific modules to improve the multilingual performances, which is similar to the recent works (e.g. [1]). The authors should explain the differences to show the uniqueness of the proposed ideas. ", "paper_topic_and_main_contributions": "This paper to modularize multilingual NMT with assembled modules from both dense and sparse models, named MMNMT. Experiments prove the effectiveness of the proposed modules on high-resource, low-resource, and zero-shot translations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zc6e3tBDgo", "reviewer": "Reviewer_bR2A", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and valuable suggestions. Our answers to your questions and comments are as follows.\n\nAnswer to RejectionReason1: \n\nWe have recently conducted experiments against 2 additional strong baseline models for a more in-depth comparison, namely NLLB-1.3B[1] and M2M-100[2]. We haven’t compared our model with Lego-MT[3] because we could not reproduce it as its public released model files are partially missing. The new results are as follows:\n\n|            Model (#params / #activated params)             |      |        | Any2En |        |           |      |        | En2Any |        |      |\n|:---------------------------:|:----:|:------:|:------:|:------:|:---------:|:----:|:------:|:------:|:------:|:----:|\n|                             | High | Medium |   Low  | Ex-Low |    Avg    | High | Medium |   Low  | Ex-Low |  Avg |\n|                             |      |        |        |        | Baselines |      |        |        |        |      |\n|     Dense (0.55B / 0.55B)     | 29.9 |  28.1  |  24.2  |  15.0  |    25.5   | 37.7 |  22.3  |  16.6  |   8.7  | 20.1 |\n|       MoE (3.7B / 0.65B)      | 31.5 |  30.4  |  26.5  |  15.8  |    27.5   | 39.5 |  24.7  |  18.7  |  11.2  | 22.4 |\n|     mRASP2 (0.55B / 0.55B)    | 30.0 |  30.7  |  26.1  |  17.5  |    27.7   | 40.3 |  25.6  |  19.8  |  11.2  | 23.2 |\n|     M2M-100 (1.2B / 1.2B)     | 30.0 |  27.3  |  22.3  |  17.0  |    24.8   | 34.6 |  22.2  |  16.3  |  12.7  | 20.3 |\n|       NLLB (1.3B / 1.3B)      | 32.3 |  33.1  |  31.2  |  28.7  |    31.9   | 36.3 |  26.6  |  21.7  |  19.8  | 24.6 |\n|                             |      |        |        |        |   MMNMT   |      |        |        |        |      |\n|  MMNMT (Dense) (3.7B / 0.65B) | 32.1 |  31.0  |  27.4  |  17.0  |    28.3   | 39.8 |  24.8  |  19.3  |  10.7  | 22.6 |\n| MMNMT (mRASP2) (3.7B / 0.65B) | 32.9 |  32.1  |  28.8  |  18.8  |    29.5   | 41.0 |  26.1  |  20.4  |  12.3  | 23.9 |\n\nThe numbers in bracket are the number of model parameters and activated parameters at inference time. It is worth noting that the newly compared models have larger model size, larger training data and longer training time than our models, while our method improves translation quality to a level comparable to them.\n\nOur MMNMT model achieves comparable performance with NLLB-1.3B on low-resource languages translation. NLLB has far more training data than ours (only 108K sentence pairs in total) on extremely low-resource languages. It is hence reasonable that our model cannot compete with it on these languages. Note that our framework can also use the modules of NLLB-1.3B to achieve further performance improvements. We’ll provide these results in the next version.\n\nA possible limitation of our MoE framework might be that modules from different basic models might be not consistent with each other, making them not able to fit together in a single model losslessly. We leave this to our future work for exploring new strategies.\n\n[1] Costa-jussà M R, Cross J, Çelebi O, et al. No language left behind: Scaling human-centered machine translation[J]. arXiv preprint arXiv:2207.04672, 2022.\n\n[2] Fan A, Bhosale S, Schwenk H, et al. Beyond english-centric multilingual machine translation[J]. The Journal of Machine Learning Research, 2021, 22(1): 4839-4886.\n\n[3] Yuan F, Lu Y, Zhu W H, et al. Lego-MT: Towards Detachable Models in Massively Multilingual Machine Translation[J]. arXiv preprint arXiv:2212.10551, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. interesting topic: mixture-of-experts (MoE) is currently a popular topic and the authors explore it in multilingual machine translation.\n2. comprehensive analysis: the authors conduct extensive experiment and analysis to show the effectiveness of the proposed method.", "reasons_to_reject": "1. missing comparison: this paper lacks comparison to strong multilingual machine translation baselines, e.g., NLLB[1], Lego-MT[2]. Can MMNMT outperforms these strong baselines? Is there a limitation on current MoE framework?\n\n[1] No language left behind: Scaling human-centered machine translation.NLLB team. arXiv:2207.04672.\n\n[2] Lego-MT: Towards detachable models in massively multilingual machine translation. Yuan et al. arXiv:2212.10551.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims at improving multilingual machine translation. The authors propose modularized MNMT framework which is able to flexibly assemble dense and MoE-based sparse modules. Experiment results on OPUS-100 dataset show that the proposed technique can bring improvement against the MoE baseline. Overall, the authors make a good contribution on improving MoE framework.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Vf5uzf1mD", "reviewer": "Reviewer_E1x3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and valuable suggestions. Our answers to your questions and comments are as follows.\n\nAnswer to RejectionReason1: \n\nWe have recently conducted experiments against 2 additional strong baseline models for a more in-depth comparison, namely NLLB-1.3B[1] and M2M-100[2]. The new results are as follows:\n|            Model (#params / #activated params)             |      |        | Any2En |        |           |      |        | En2Any |        |      |\n|:---------------------------:|:----:|:------:|:------:|:------:|:---------:|:----:|:------:|:------:|:------:|:----:|\n|                             | High | Medium |   Low  | Ex-Low |    Avg    | High | Medium |   Low  | Ex-Low |  Avg |\n|                             |      |        |        |        | Baselines |      |        |        |        |      |\n|     Dense (0.55B / 0.55B)     | 29.9 |  28.1  |  24.2  |  15.0  |    25.5   | 37.7 |  22.3  |  16.6  |   8.7  | 20.1 |\n|       MoE (3.7B / 0.65B)      | 31.5 |  30.4  |  26.5  |  15.8  |    27.5   | 39.5 |  24.7  |  18.7  |  11.2  | 22.4 |\n|     mRASP2 (0.55B / 0.55B)    | 30.0 |  30.7  |  26.1  |  17.5  |    27.7   | 40.3 |  25.6  |  19.8  |  11.2  | 23.2 |\n|     M2M-100 (1.2B / 1.2B)     | 30.0 |  27.3  |  22.3  |  17.0  |    24.8   | 34.6 |  22.2  |  16.3  |  12.7  | 20.3 |\n|       NLLB (1.3B / 1.3B)      | 32.3 |  33.1  |  31.2  |  28.7  |    31.9   | 36.3 |  26.6  |  21.7  |  19.8  | 24.6 |\n|                             |      |        |        |        |   MMNMT   |      |        |        |        |      |\n|  MMNMT (Dense) (3.7B / 0.65B) | 32.1 |  31.0  |  27.4  |  17.0  |    28.3   | 39.8 |  24.8  |  19.3  |  10.7  | 22.6 |\n| MMNMT (mRASP2) (3.7B / 0.65B) | 32.9 |  32.1  |  28.8  |  18.8  |    29.5   | 41.0 |  26.1  |  20.4  |  12.3  | 23.9 |\n\nThe numbers in bracket are the number of model parameters and activated parameters at inference time. It is worth noting that the newly compared models have larger model size, larger training data and longer training time than our models, while our method improves translation quality to a level comparable to them.\n\nOur MMNMT model achieves comparable performance with NLLB-1.3B on low-resource languages translation. NLLB has far more training data than ours (only 108K sentence pairs in total) on extremely low-resource languages. It is hence reasonable that our model cannot compete with it on these languages. Note that our framework can also use the modules of NLLB-1.3B to achieve further performance improvements. We’ll provide these results in the next version.\n\nWe have provided the number of model parameters of each baseline and MMNMT in Section 5 and the number of parameters actually activated during inference in the above table. We will provide all these details and adjust the size of Figure 1, 4 and 5 according to your suggestion in the next version.\n\n[1] Costa-jussà M R, Cross J, Çelebi O, et al. No language left behind: Scaling human-centered machine translation[J]. arXiv preprint arXiv:2207.04672, 2022.\n\n[2] Fan A, Bhosale S, Schwenk H, et al. Beyond english-centric multilingual machine translation[J]. The Journal of Machine Learning Research, 2021, 22(1): 4839-4886.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The model architecture is interesting especially the training method for the model. The description of the training method is also clear and simple to implement. The evaluations of different tasks and results cover multi parts of the model and show the effectiveness.", "reasons_to_reject": "However, there are some problems in this paper.\n\nAuthors have introduced some similar works in the related work while Section 5 only gives results of mRASP2. Authors should give more results of existing works for evaluation. The authors should also give the amount of parameters in Section 5.\n\nFigure 1, Figure 4 and 5 are too small.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an MNMT framework that combines the advantages of dense and MoE models to achieve the best of both worlds and also introduces a three-stage training method for the model.  Experimental results on two benchmarks show the effectiveness of the proposed model in which the model outperforms the dense and MoE.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposes a modular MMT framework that is able to use both dense and sparse MoE modules, which brings together the benefits of both sparse and dense MMT models. The authors do various experiments in different settings to compare to as baselines and compare across various language pairs and resource settings. The authors and reviewers had good discussion and reviewers are in consensus for accept, in particular the authors added additional critical baselines. The authors do note differences to existing work such as NLLB in terms of amount of data, which is a reasonable point in terms of performance.
Decision: Accept-Main