Sparse MoE with Language Guided Routing for Multilingual Machine Translation
Authors: Xinyu Zhao
Xuxi Chen
Yu Cheng
Tianlong Chen
Venue: ICLR 2024 poster (2024)
Keywords: Sparse Mixture-of-Experts
Multilingual Machine Translation
Language Guided Routing
Primary area: general machine learning (i.e., none of the above)
TL;DR: We power up SMoE with language-guided rounting based on language hierarchy and difficulty, leading to superior machine translation performance.

Abstract: Sparse Mixture-of-Experts (SMoE) has gained increasing popularity as a promising framework for scaling up multilingual machine translation (MMT) models with negligible extra computational overheads. However, current SMoE solutions neglect the intrinsic structures of the MMT problem: ($a$) $\textit{Linguistics Hierarchy.}$ Languages are naturally grouped according to their lingual properties like genetic families, phonological characteristics, etc; ($b$) $\textit{Language Complexity.}$ The learning difficulties are varied for diverse languages due to their grammar complexity, available resources, etc. Therefore, routing a fixed number of experts (e.g., $1$ or $2$ experts in usual) only at the word level leads to inferior performance. To fill in the missing puzzle, we propose $\textbf{\texttt{Lingual-SMoE}}$ by equipping the SMoE with adaptive and linguistic-guided routing policies. Specifically, it ($1$) extracts language representations to incorporate linguistic knowledge and uses them to allocate experts into different groups; ($2$) determines the number of activated experts for each target language in an adaptive and automatic manner, according to their translation difficulties, which aims to mitigate the potential over-/under-fitting issues of learning simple/challenges translations. Sufficient experimental studies on MMT benchmarks with {$16$, $50$, $100$} language pairs and various network architectures, consistently validate the superior performance of our proposals. For instance, $\texttt{Lingual-SMoE}$ outperforms its dense counterpart by over $5\%$ BLEU scores on $\texttt{OPUS-100}$ dataset.

Published as a conference paper at ICLR 2024
SPARSE MOE WITH LANGUAGE-GUIDEI ROUTING FOR MULTILINGUAL MACHINE TRANSLATION
Xinyu Zhao' Xuxi Chen? Yu Cheng'
Tianlong Chen
'The University of North Carolina at Chapel Hill
"The University of Texas at Austin
The Chinese University of Hong Kong
*IIT
"Harvard University
(xinyu, (nanoog)@ unc. edu
xxchen@utexas.. edu chengyulcse. www.://sssee.cc....oeooee
ABSTRACT
Sparse Mixture-of-Experts (SMoE) has gained increasing popularity as a promis- ing framework for scaling up multilingual machine translation (MMT) models with negligible extra computational overhead. However, current SMoE solutions neglect the intrinsic structures of the MMT problem: (a) Linguistics Hierarchy. Languages are naturally grouped according to their linguistic properties such as language families, phonological features, etc; (6) Language Complexity. Learning difficulties vary for different languages due to their available resources, grammar complexity etc. Therefore, routing fixed number of experts (e.g., or 2 experts in usual) only at the word level leads to inferior performance. To fill in the miss- ing puzzle, we propose Lingual-SMo SMoE by equipping the SMoE with adaptive and linguistics- guiieed routing policies. Specifically, it (1) extracts language rep- resentations to incorporate linguistic knowledge and uses them to allocate experts into different groups; (2) determines the number of activated experts for each get language in an adaptive and automatic manner, according to their difficulty level determined by data abundance, which aims to mitigate the potential over- /under-fitting problems of learning easy/difficult translations. Sufficient experi- mental studies on MMT benchmarks with {16, 50, 100} languages and various network architectures, consistently validate the superior performance of our pro- posals. For instance, Lingual SMoB outperforms its dense counterpart by over 5% BLEU scores the OPUS 100 dataset.
INTRODUCTION
Multilingual Machine Translation (MMT) aims to re- solve multiple translation directions simultaneously one unified model and has attracted considerable at- tention in both academia and industry. Aharoni et al. (2019); Johnson et al. (2016); Aharoni et al. (2019) veal that MMT models are capable of adapting to low- resource scenarios, benefiting from the joint optimiza- tion of multiple translations. Nevertheless, as the num- ber of languages involved in MMT increases (e.g., 50), the story starts to change. As illustrated by Conneau al. (2019); Sachan & Neubig (2018), the language in- terference emerges and poses an obstacle to achieving satisfactory performance, which a troublesome conse- quence of the competing gradients among different lan- guages. Unfortunately, such problem will be exacer- bated by severe data imbalance, leading to over-fitting on low-resource translation directions or catastrophic forgetting on previously trained samples (Lakew et al., 2018; Elbayad al., 2023).
Dense
32
Vanillaa SMoE
30
Lingual-SMoE
28
26
22
20
16
100
Number languages
Figure 1: The MMT performance of (dense model, vanilla SMoE, Lingual-SMoE (Ours)) on OPUS-1C0 with different amount of languages pairs E {16, 50, 100).
Our code is

---

Published as a conference paper at ICLR 2024
A natural remedy is to scale up the model capacity, which has been demonstrated an effective way to improve multilingual machine translation (Shazeer et al., 2018; Radford & Narasimhan, 2018; Devlin et al., 2019). However, these gigantic language models with massive parameter counts are extremely computationally intensive. For example, training the popular GPT-based model (Brown et al., 2020) with billions of parameters typically requires thousands of GPU days. To provide ef- ficient alternative, pioneering researchers introduce the Sparsely-gated framework (Lepikhin et al., 2021; Fedus al., 2021; Shazeer et al., 2017: Zoph et al., 2022), which is designed with an input-dependent conditional computing fashion. Specifically, SMoE only acti- vates the relevant model pieces given an input sample, and this dynamic sparse computing facilitates the training of huge language models with feasible resource costs.
Mixture-of-Experte (SMoE)
However, most existing practices adapt the SMoE algorithm in a straightforward way and overlook intrinsic structures in the MMT problem, i.e., Linguistic Hierarchy and Heterogeneous Language Complexity. @ Even incorporating linguistic knowledge in an exemplary way, helps. For example, Fan et al. (2021); ); Zhang et al. (2020) customize several language-specing components in the trans- lation model and enjoy an enhanced performance. Kudugunta et al. (2021) takes one step further by routing input samples to different experts in the SMoE based translation language representations. Although these initial efforts achieve great results, the exploitation of prior linguistic knowledge is highly insufficient. Languages have their hierarchy. Specifically, different languages can be orga- nized in a tree structure according to their language family, grammar, phonological features, etc. How to leverage such linguistic priors for SMoE design in MMT, question. @ Another limitation of current SMoE approaches is the neglect of the heterogeneous complexity of diverse language translations. Due the variance of factors such as grammatical complexity and available resources, the translation difficulties can vary significantly (Goyal et al., 2021: Team et al., 2022; Heffernan et al., 2022). However, existing SMoEs use fixed model size (e.g.. or 2 experts) to handle all translation directions. It potentially compromises certain transla- tions since excessive or insufficient model capacity could result in over-fitting or under-fitting issues, respectively. In addition, manual adjustments of the expert capacity for each language are labori- ous and suboptimal due to the interplay among multiple translation objectives. Then, how to take the heterogeneity of language difficulty into consideration adaptively determine the appropriate network capacity, is a necessary yet beneficial step towards superior SMoE models.
challenging yet rewarding
To answer the aforementioned questions, we propose novel Singuull-MMol for MMT, by design- ing language-guidee guided routing policies. It allocates experts in a hierarchical manner and enables the language-specifie model capacity, which brings significant performance improvements as demon- strated in Figure 1. The advantages of our effective strategy are multi-fold: (1) injecting rich linguis- tic knowledge the expert routing process via encouraging experts to specialize in certain language families; (2) mitigating potential over-/under-fiting on translation directions with different difficulty levels determined by data availability by dynamically adjusting the expert number based on training performance. Our contributions can be summarized as follows:
*
We propose an innovative SMoE framework for multilingual machine translation, i.e., Li ngual- by considering two unique properties of linguistic hierarchy and her- erogeneous language complexity in MMT problems.
We design a hierarchical routing policy in Lingual- SMOE that learns to route input sam- ples with multi-granularity information of (language family, language, and token).
We introduce a dynamic expert allocation mechanism to adaptively determine adequate expert capacity for each language translation with distinctive difficulty levels. Training dynamics are monitored to enable automatic adjustments.
Extensive experiments with different data resources and number of languages consistently evidence the effectiveness of Li nggual 1 F. For example, our language-guides routing proposals outperform its dense baseline by a clear performance margin 5% on OPUS-100
2
RELATED WORKS
Multilingual Machine Translation (MMT). Multilingual machine translation extends neural ma- chine translation to the scenario with multiple language pairs, which is popular paradigm in natural language processing (NLP). Bojar et al. (2018); Dabre et al. (2020) have demonstrated that training with multilingual data enhances the translation of low-resource languages. Related MMT research
2

---

Published as a conference paper at ICLR 2024
can be roughly divided into two categories: (1) multi-way translation, which supports many-to- many translation through parameter sharing, multilingual representation learning, and custom joint training techniques (Aharoni et al., 2019; Yang et al., 2021; Pan et al., 2021; Tan et al., 2019); (2) low-resource translation, enhancing MMT under limited parallel corpora, monolingual data, or even unseen languages (Ranathunga al., 2021; Neubig & Hu, 2018).
Spneuige Designs in MMT. Among the rich literature on MMT, language-specinn pa- rameter sharing for multi-way translation is the most relevant one to our work. An effective parameter-sharing algorithm needs to decide how many parameters to share and how to share (Dabre et al., 2020). The typical manner is to share a fixed part and learn extra language-specifie modules. For instance, Pires et al. (2023) assumes language-speciffs decoders and applies architecture search to determine the best composition of shared and sunguage-specifie encoder layers. Purason & Tättar (2022) advocates that combining shared, language-specifeg and layers benefits low-resource languages without harming high-resource languages. In contrast, an- other group of studies shares most of the translation model with lightweight language-specific mod- ules that adaptively inject linguistic knowledge. For example, Zhang et al. (2021) inserts conditional language-specinn specific (CLSR) layers in each encoder and decoder block, with a binary gate function that collects hidden representations from the shared and specialized part. Lin et al. (2021) learns a sub- network for each translation direction with shared parameters.
encoding
Sparse Mixture of Experts (SMoE). The concept of mixture-of-exppee (MoE) can be traced back several decades (Jacobs et al.. 1991; Jordan & Jacobs, 1994). It contains a series of network sub- modules that are utilized conditional on the input samples. The Sparsely-gate -aatedd Mixture-of ceatuureoof-Expeeee of-Exxerts (SMoE) is an efficient variant of MoE, which only activates a few expert networks for each input, allowing a significant amount of increase in model parameter counts yet with minimal extra comput- ing overheads (Shazeer et al., 2017). Numerous successes of plugging SMoE into transformer-base language models have been demonstrated in diverse NLP and computer vision applications (Fedus et al., 2022; Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2021; Zuo et al., 2022; Jiang et al., 2021; Zoph et al., 2022; Riquelme et al., 2021; Yang et al., 2019).
Routing Designs and Expert Capacity in SMoE. Routing policy is one of the major compo- nents of SMoE, which plays an essential role in its achievable performance. Various design options are introduced to pursue improved allocation of experts for each input sample. The classic one is a learnable router network that selects the top-k experts given an input token (Lepikhin et al., 2021; Fedus al., 2021). However, it suffers from the routing imbalance issue. Many techniques are designed to promote balanced expert assignments: injecting Gaussian noise into router net- works (Shazeer et al., 2017); adding an auxiliary balancing loss regularize routing (Lepikhin et al., 2021; Fedus et al., 2021); solving routing as a linear assignment problem (Lewis et al., 2021); using reinforcement learners (Clark et al., 2022): routing top-k input tokens to each expert instead of choosing top experts per token (Zhou et al., 2022); or directly replacing learnable gates with random routing (Zuo et al., 2022; Chen et al., 2023b; Roller et al., 2021). group of studies pioneer input- specific routing. Some studies language-specific routing (Kudugunta et al., 2021), others investigate routing with input domain information (Gururangan et al., 2022; Li et al., 2022). Linguistic charac- teristies like its hierarchy remain underexplored. While most studies train SMoE model with fixed top-k experts, some recent designs propose to change the expert capacity during training to adapt to multitask lifelong learning scenarios (Chen al., 2023c;a).
3 METHODOLOGY
3.1 PRELIMINARIES AND NOTATIONS
Multilingual Machine Translation. MMT is formulated as a sequence-to--sqqueec task, where a source language sequence is fed into encoder, and the target language sequence will be generated from a decoder conditioned on the encoder output (Sutskever et al., 2014). The translation objective CMT is adopted to maximize the probability of the generated sequence in the target language given the source sequence. In our case, Transformer-Bane baseline, and our approaches are established on top of it by inserting well-designed SMoE layers.
(Vaswani et al., 2017) is used as our dense
Sparse Mixture-of- Expertss (SMoE). In design, we replace every other transformer block with an SMoE block for both the encoder and decoder, following the default configuration in Lep- ikhin al. (2021). The SMoE block consists of n experts (E,, , . that are ferd-forward
3

---

Published as a conference paper at ICLR 2024
Token Embeddings
Expert
Target Language Embeddings
Lingual
Expert 2
Limburgish (Hard)
German (Easy)
Russian (Easy)
Bclarusian (Hard)
Router
Expert
Language Speci fie Esperts
Linguistie-guided
Token Routing in Vanilla SMoE
Language Router
Shared Expert
Routing
Token
Embeddings
Token Router
Group g
Group g(3)
Group
Group g
M Egi
Eg
E8
Expert Allocation Based on the Translation Difficulty Level
Figure 2: The overview of our proposed Lingual-SMoo
For the vanilla SMoE, routers take
each token as input and select top-k experts for the following execution. (m) A hierarchical routing design is adopted in Lingual SMoE. The linguistic-guided router first selects group of experts for each target language. Note that the group size is varied based on the difficulty level transla- tion. Then, another token router allocates experts from the language-specific expert group.
networks. Given an input embedding æ, is fed into a router network g(-) and assigned to the most relevant experts for further processing, as shown in Figure 2 (n). The dominant design of router networks in the literature is a fully connected layer, as described below:
where Wg are tunable parameters and top-k(-) is selection function that outputs the largest k values. The final output of an SMoE block will be a weight summarization of the features from activated experts, i.e. E!Sl g; (xx). S denotes the index set of experts selected by the routing policy. Usually, to encourage more uniform routing decision, an auxiliary load balancing loss Lg (Lepikhin et al., 2021; Shazeer et al., 2017) will be adopted for SMoE training.
3.2 LINGUAL-SMA SMcE EQUIPPING SMOE WITH LANGUAGE-GUIDEE ROUTING
In this section, we detail our proposal, i.e., Lingual- EMOE, As shown in Figure 2 (1) it consists of two main components (1) linguistic-guiin routing (LGR) and dynamic expert allocation (DEA).
Linguisties-Guided Routing (LGR). We start from a pilot investigation to whether vanilla SMoE model naturally captures similar routing patterns for closely related languages, e.g., languages from the same linguistic family. Specifically, we train top-2 routing SMoE with the language- routing policy, following the default configurations in Kudugunta et al. (2021). A subset of 16 lan- guages from OPUS-10 00 dataset is used for training and evaluation. From these translation pairs, we choose 8 languages from three different language families, i.e., {{bg, sk, sl, hr), {nb, de), (as, mr)) for visualizations. The first four languages (bq~hr) belong to the Slavic language group, while nb and de are of Germanic origin, and as and mr fall into the Iranian category. To measure the similarities between routing decisions, we compute the cosine distance among different routing outputs of the corresponding en-xx (xxE(bg><r)) translation directions, in the last de- coder SMoE layer. Results in Figure tell us that the routing choices show neither differentiability across language groups nor similarity between languages within the same group. It implies that the vanilla language- routing cannot learn the desired linguistic knowledge.
To fill in the research gap. design hierarchical routing policy with two-level router networks, guided by linguistic priors. In detail, for each input sequence: ① Extracting language embedding. We feed the target language ID into one embedding layer and two fully connected layers to produce a 512 112-dimensional language emhedding. 2 Language routing at the first level. In each SMoE layer, language router takes the language embedding as input and outputs language-dependen expert vector. Then, top-kı(-) function is applied on top of it to narrow down all experts to language- specific candidate experts as Sr. 3 Token routing at the second level. Lastly, we allocate experts from St at the token level, to generate the final activated expert set S. In summary, our routing policy executed as E!S ||S GI. 9j Ex)(2r), where is the input sample.
4

---

Published as a conference paper at ICLR 2024
1,00
0.09
0.77
0.76
0.54
0.60
0.77
0.71
14
0.69
1.00
0.50
0.09
0.17
0.63
0.89
0.94
17.5
0.77
0.58
1.00
0.73
06
0.87
0.09
0.54
10
de nb hr
0.76
0.89
0.73
1.00
0.38
0.27
0.90
0.84
pera loiiyty
15.0
Train
08
12.5
Valid (en-se)
0.54
0.17
6/6
0.38
1.00
0.63
0.32
0.20
Valid (en-2h)
10.0
0.66
0.63
0.87
0.77
0.63
1.00
0.75
0.55
-34
7.3
mr as
0.77
0.00
0.69
0.90
0.32
0.75
1.00
0.85
-22
5.0
0.71
0.94
0.54
0.04
0.20
0.55
0.85
1.00
bg
hr
de
60k
80k
100k
Training Steps
Figure 3: Routing similarities of the last de- coder layer through the language routing. Three groups of target languages (bg, sk, sl, hr). {nb, de), {as, mr} are presented. Darker blocks imply higher similarity.
Figure 4: Train and validation perplexity of different language pairs in vanilla SMoE models. The over- fitting issue pair (en-se) se) than in high-resource one (en-zh).
pronounced in the low-resource
To promote similar routing decisions between closer languages, 2-step language grouping is pro- posed in our Miiguull-SMon First, we train the embedding layer and two fully connected layers that convert target language ID) embedding with all target language samples. We use a trastive loss on the language families classes to maximize the margin of embeddings of different families, 2. Second, at the follow-up training phase, language grouping loss Li is added to the above translation, which encourages the expert allocation to be specialized in a particular language family or group. In each forward process, we compute the contrastive loss using the language routers' output. The distance measure is cosine similarity. More details about the language group- ing are included in Appendix Al and A9. The final objective function L is depicted below:
L = LMT + X Lg + C2 Li,
(2)
where ei and 02 the hyperparameters to control the regularization effects from the load balancing loss Lg and language grouping loss L,, respectively. In our experiments, C1 and 02 are set to 0.05, which is determined by a grid search.
Dynamic Expert Allocation (DEA). If SMoE translates multiple languages with diverse complexities and fixed model size, potential over- or under-fitting hap- due to excessive or insufficient model capacity in simple or complicated scenar- ios respectively. It will be amplified by data imbalance. For example, as shown in Elbayad al. (2023), SMoE models are prone to over-fitting on low-resource tasks, i.e. languages, or translation direc- tions with less training data in the case of machine translation. To provide clear
Algorithm 1 DEA in our proposed Lingual-SMa
Input: The language subset index j. validation set Di of the subset j. the number of experts per language ki, metric function P, and a expert number growing threshold X, number of updates N, maximum updates Nmax. the ratio of expert number exploring updates r. for each language subset do
2:
3: Initial indicator Improved True;
Initial the current best metric p while N S Nmax do val(best) P < for An iterations then ki, ki, Improved False
vallbest) f- 00;
5:
if pd
7:
else
picture of this severe problem, we vi- sualize the validation perplexity of high- resource (en-zh) and low-resource (en- se) language translation, as presented 12: end while
9:
p val((raa))
Plu Improved + Truc;
10:
end if
11:
Continue training until the next validation.
Figure 4. The results are collected from a vanilla SMoE model for MMT. It shows
13:
if not Improved then
14:
ki ki,j 1; rerun An iterations.
15:
end if
that the validation perplexity of the high- resource direction decreases continuously.
16: Fix ki,; train until Nmax iterations.
17:
end for
but the low-resource translation direction
18: Output: 1.1 Igual 1 SMOR with top-ki,5 routing.
strongly overfits after 20K steps.
Since aado-Furopean languages form majority of the OPUS- 1 00 (58 out of 100) dataset, treat their subfamilies the class label for the calculation of contrastive training loss.
5

---

Published as a conference paper at ICLR 2024
To address the issue and avoid laborious manual tuning, Lingual-SMor offers adaptive algo- rithm to decide the exact k for the top-ky expert assignment in the language router, as illustrated in Algorithm 1. We divides different languages into three difficulty level groups, according to the data richness. In addition, we explore grouping languages considering both grammar complexity and data availability, see Appendix A2 for more details, but decide not to include grammar complexity in the difficulty metire because of its subjectiveness. Specifically, we split the 94 validation language pairs in OPUS 100 into three groups based on their training data size: high-resource (> 0.9M, 45 lan- guages). low-resource (< 0.1M, 26 languages), and et al., 2020). At the validation step per iterations on each subset, we compute the perplexity P If If does not decrease for a certain threshold, the number of candidate experts for the associated language router will be increased by updating kt. = kt. + 1. After the exploration stage, if P still does not decrease, we reset this iteration with kt, = ki,j 1 and fixed it in the rest training.
resource (other, 28 languages) (Zhang
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Datasets. We evaluate the proposed Lingual-SMoE on the representative multilingual neural machine translation dataset, i..., OPUS 100 (Zhang et al., 2020) that contains 100 languages and 94 validation and test language pairs. To testify our methodology across varying language quan- tities, we extract datasets comprised of 16 and 50 languages from OPUS-100, respectively. This results in two smaller datasets, namely OPUS-16 6 and OPUS-50, both of which have equivalent ratios of high, medium, and low resource languages. The three datasets are further processed using SentencePiece (Kudo & Richardson, 2018), which sets the vocabulary size to 32, 000 for OPUS- 1oos-16 6 and OPUS-50 and 64, 000 for OPUS-100 We attach a target language ID with the source and target language sentences to identify its translation direction. See Appendix A1 for more details.
Models and Baselines. We compare our method with the Transformer- Base model (as Dense) and its SMoE variants that have encoder and decoder layers, 32 experts. The input and hidden dimensions of all feed-forward networks are 512 and 2048. Meanwhile, all considered SMoE mod- els fall into three types. @ For the vanilla SMoE model, Switch Transformer (Fedus et al., 2021) with top-1 token-based routing (as ST-SMoE) and GShard (Lepikhin et al., 2021) with top-2 token-based routing (as GS-SMoE) are adopted. @ We consider three improved SMoE models that incorporate language information parameter sharing: (1) Language-specinnn fixed routing (as LS-SMoE) inspired by Pires et al. (2023), assigning 2 non-overlapping experts for tokens according to their source language in the encoder and target language in the decoder; (2) Hybrid SMoE model (as Hybrid-SMooE from Kudugunta et al. (2021), with top-2 token routing in the encoder and top-2 target language routing in the decoder side; (3) Residual SMoE model (as Rasidual-SMobl from Elbayad et al. (2023); Rajbhandari et al. (2022); Zhang et al. (2021) that augments each SMoE layer with a shared ford-fowward network through a binary gate function. Note that the shared and SMoE branches are weighted accordingly for computing the final features. & The third group is our proposed Lin FM IGR-SMoE stands for an SMoE model with our linguistic-guide the second-level token router activates the top sequentially. In addition, to examine an interest- ing combination between the residual expert (Elbayad et al., 2023; Rajbhandari et al., 2022) and our linguistic-guiie -uidedd routing, we organically integrate them as LGR+tes reeetresSSMaa -SMoE. For consistency of computational cost, in -GR -SMEE, the second- level top-2 token routing is replaced by a top-1 routing. Li Lingual- SSMo further adopts the dynamic expert allocation on top of LGR-SMoE.
SMoE model with
routing, where the first-level language router selects the top 8 experts and
Training and Evaluation Details. The training processes have 35K, 100K, and 200K iterations for OPUS-1 6, OPUS-50, and OPUS-100, respectively. With a learning rate of x 10 4 we optimize models with Adam using c2 = (0.9, 0.98, 10 (Kingma & Ba, 2015). The learning rate schedule follows the Inverse Square Root with a specific number of warm-up steps set to 4. 000. A cemperature-based data sampling strategy is utilized to train our models (Aharoni et al., 2019). The temperature is set to 1.5 for OPUS - 16, and 5 for OPUS 50 and OPUS-100 The dynamic expert allocation uses value of An equal to 5, 000 iterations for experiments on OPUS-16, OPUS-50, 0, and 10, 000 iterations for OPUS-100. In addition, the ratio of expert number exploring updates is set to 0.8, and the threshold controlling expert capacity number X is 0.1 for OPUS-10 6, OPUS-50 and 0.01 for OPUS-100, For the memory efficiency purpose, we employ the fp1 in training all models (Ott al., 2018).

---

Published as a conference paper at ICLR 2024
Table 1: Multilingual machine translation performance on OPUS16 dataset. Average BLEU scores for each translation direction and win-ratio are reported. We classify languages according to data amount into three groups: high (> 0.9M), low (< 0.1M), and medium. We compare Lingual a B- and its variants [LGR- LLGR-SMoE,
with dense and SMoE baselines (Dense, GS S7-SMoE), and modified SMoE models (LS-SMoE, Hybrid-SMoE, Rexidual-SMoE.. The total number of experts for all SMoE mod- els 32. The number activated experts for each token level router is 2 except ST-SMoE. The number of candidate experts is 8. The best two performances are bold and underlined.
Methods
Avg.
en-xx
XX xx-en
win-rate
Avg.
high
medium
low
Avg.
high
medium
low
Dense
28.79
26.92
25.37
39.12
14.78
30.67
28.81
39.24
21.21
GS-SMoF
30.29
28.28
25.55
40.71
18.97
32.31
28.77
41.70
29.25
77%
ST.SMoE
31.79
29.68
26.55
43.04
20.23
33.89
29.94
44.00
30.94
100% %
LS-SMoE
26.75
22.99
20.06
37.31
11.72
30.50
23.69
42.99
32.01
33%
Hybrid-SMoE:
28.35
24.23
24.56
31.70
13.38
32.45
29.59
41.74
27.81
53%
Residual- SMoF
31.97
30.06
26.52
43.49
21.58
33.88
29.84
43.94
31.25
100% %
LGR-SMoE
32.32
31,20
26.46
46.68
23.23
33.44
29.76
43.15
30.28
97%
LGR -MMEE
32.61
31.42
27.06
46. 25
23.29
33.79
29.70
43.52
31.74
100 %
Lingual-SMa 32.71
31.46
27.11
46.24
23.34
33,96
29.87
43.49
32.02
100 %
To assess all algorithms, we calculate the BLEU scores test sets via "eETe (Post, 2018).
The scores encompass the average ratings across all language pairs, such as English-to-Any (en- xx), and Any-to-Englist (xx-en) the OPUS-100 000 dataset. Furthermore, we exhibit the win rate, indicating the fraction of language pairs where a method outperforms its dense counterpart (Zhang al., 2020). Experiments are conducted using Fairseq (Ott et al., 2019) with 8 RTX A6000 GPUs.
4.2 LINGUAL- SMcE IMPROVES MULTI-LINGUAL MACIINE TRANSLATION
Comparisons with Previous -tate-of-the-art (SoTA) Approaches. We compare our proposed -SME with baselines and existing SoTA SMoE algorithms for MMT. Specifically, all models are trained and assessed on the OPUS-16 6 dataset with Transformer- Base as the backbone architecture. The results are summarized in Table 1. Several observations can be drawn:
① Lingual-SMo outperforms the Dense and vanilla SMoE multilingual machine transla- tion baselines with distinct advantage. Specifically, Linguistis--Gudde Routing (LGR-SMots alone achieves f3.53%, 4. 4.28% 2.77%), (2.03%, 2.92%, 1.13%) improvements in BLEU scores for [All (Avg.), English to Any (en-xx), Any to English (xx-en)) translation directions for Dense and GS-SMoE, respectively. This clear advantage confirms the effectiveness of introducing linguistic knowledge into the routing of SMoEs. Furthermore, the English-to-Ai translation direction hibits a more notable improvement compared to the Any-to-English direction. Note that the former has diverse target language options and the latter has a single target language option, i.e., English. It implies that the language router performs a superior expert assignment when it receives diverse target language representations. Similar observations can be found in the comparison between LGR-SMoE and ST-SMoE. LGR-SMoE surpasses ST-SMoE by a clear performance margin in terms of { (Avg., e translation, while has comparable result in the case of (xx-en).
2 Lingual--MMe consistently surpasses other modified SMoE models (i.e., LS-SMoE, Hybrid- SMoE, and Residual-SMofEs demonstrating the advantage of Language-guided routing in balancing the competition between shared and language-specific parameters and improving previous substan- dard routing. More detailed analyses lie as follows. (1) The LS-SMoE model trains experts sepa- rately with different language samples, resulting in nearly 2% lower average BLEU score than the dense counterpart, especially when translating into languages besides English, which is about 4% lower. A possible reason is that sentence pairs where English is the target language, are the ma- jority the datasets, which leads to a highly imbalanced training of LS-SMoE. In other words, the English-specific parameters will receive much more attention compared to the rest languages. In contrast, Lingual-SMal approaches language-specific designs in a softer way by identifying and pre-routing set of experts for each language. The language-specing expert sets are allowed to be overlapped among different languages. (2) Similarly, Hybrid-SMoF also suffers from imbalanced
BB. Signature: nrefs: case:mixed eff:no| tok:133 smootheexp version:2.3.1

---

Published as a conference paper at ICLR 2024
Table 2: Multilingual machine translation performance OPUS50 and CPUS- dataset. Average BLEU scores for each translation direction and win-rate are reported. We compare 1, ninguall -SMOE and its variants {LGRRSMME, LGR+ -es
L.GEE with [Dnse,, GS-SMoE).. The hest two performances are bold and underlined.
Methods
Datasets
Avg.
XX
xx-en
wate
Avg.
high
medium
low
Avg.
high
medium
low
Dense
24.09
21.06
17.48
25.60
23.02
27.
22.88
30.81
31.26
GS-SMoE
26.45
23.15
19.23
29.17
24.14
29.76
26.16
33.61
32.49
83%
L.G-SMoE LGR -SMoE
OPUS50
27.75
26.19
22.05
32.92
26.84
26.27
33.76
30.34
92%
27.31
25.38
21.59
32.14
25.30
29.25
25.71
33.27
31,69
94%
LingLal-SMal
27.84
26.21
22.09
32.90
26.86
29.46
26.28
33.75
30.91
95 %
Dense
22.21
19.03
16.51
22.01
20.68
25.39
22, 83
27.82
27.76
GS-SMoE
24.82
20.85
16.88
24.66
23.99
28,78 78
26.73
32.11
28.81
79%
LGR-SMoE
0EUS100
27.50
25. 58
22.85 88
30.66
24.80
29, 42
27.99
32.45
28.59
94%
L.G -Mo
27.44
25. 49
22.68
30.35
25.18
29. 38
27.77
32.32
29.00
96%
Lingwal -ang SNGE
27.67
25.70
22.93
30.66
25. 10
29.65
28.23
32.47
29.05
97% %
training even though it employs a routing mechanism since routing only by language imposes a rigid constraint on expert selection, which is mitigated by our hierarchical routing from Lingual- -SMoE. (3) Residual-SMe -Moo outperforms Dense and vanilla SMoE baseline, but its fixed routing inferior compared to the flexibility of our hierarchical routing in L. Lingua SMMEE.
@ Inspired by Residual-SMMr SMoo,, we further combine a shared expert with top-1 linguistic-guide routing (.GR+tee-SMoE) On OPUS-16 6, it enhances the performance of LGR-SMoE by around 0.3% and increases the win-rate over the dense baseline to 100%. Extra validations about whether the fixed shared expert is necessary are presented in Table 2.
> With Dynamic Expert Allocation, Lingual -Magu SMoE automatically adjusts the appropriate network capacity to resolve language translations with varied complexity, by activating additional amounts of model parameters. It improves average performance on BLEU score by about 0.4% over fixed LGR-SMoE. We that DEA is particularly helpful for low-resource scenarios, providing an over 2% increase on low-resource xx-en directions, which is consistent with our intuition.
Evaluation across Different Number of Languages. To examine whether Lingual- Matiaal-Maa SMoEE re- tains its advantage on datasets with more language pairs, we choose Dense, SMoE baselines, and better-performing methods, and train them on QPUS-50 and OPUS-100. The results are recorded in Table 2. @ We found that Lingual EMoE continues to reach the best, outscoring the dense baseline by ~ 5% BLUE scores on OPUS-100. 100. It further verifies the effectiveness of our adaptive Language-guided routing. @ Meantime, while the LGR+res-SMo model outperforms linguistics- guided routing in small datasets, as the number of languages increases, LGR-SMok overtakes its residual counterpart at most cases. A possible explanation is that a single shared residual expert starts to be insufficient capture increased common language knowledge when the number of translation directions keeps boosting.
4.3 ABLATION STUDY AND EXTRA INVESTIGATION.
In this section, we further conduct an in-depth analysis of Lingual-SMof regarding: i) the con- tributions of its various components, ii) language router designs, tii) the number of experts and their specialization. All experiments are carried out on OPUS-16 with the same training configurations.
Contribution of Different Components in Linguistics- Guided Routing, Linguistics-guidee routing is divided into two steps: (1) training the language representation module and (2) training the translation model with lan- guage grouping loss and language representation initial- ization. Therefore, we train and evaluate models without both or without one of them, as shown in Table We see that language grouping loss (Loss) is more beneficial than language representation learning (Emb). An organic combination leads to the best performance. It again con-
Table 3: Ablation (1) language embed- dings initialization (Emb) and (2) language grouping loss (Loss) in 1 Lingual-SMae
Emb
Loss
Avg.
en-xx xx-en
X
X
29.18
27. .47
30,90
V
x
30.13
29.35
30.91
X
32.10
31.17
33.03
32.32
31.20
33.44
firms that linguistic guidance helps SMoE to reach better translation performance.
Comparison among Language Router Designs. In addition to the linguistic-uuider router (learned), testify two other routing designs of: (1) fixed, the first-level router always assigns
8

---

Published as a conference paper at ICLR 2024
8 fixed experts based on the target language for each input sample; (2) random, the first-level router randomly selects 8 experts for each input sample. The results are recorded in Table 4. Our routing policy achieves superior performance, compared to its linguistic-agnoste counterparts. This high- lights the need of appropriate sharing and specialization of model parameters for different languages in the multilingual machine translation.
Correlation between # Allocated Experts and Lin- guistic Difficulty. To understand the working mecha- nism of dynamic expert allocation, we track the num- ber of selected experts for each complexity level E as col-
Table 4: Ablation studies on language router designs of i,i ninaaggllsaa All fixed expert capacity S = =.
Method
Avg.
en-xx
xx-en
(high, med, low} in training Lingual-SMon lected in Figure 5. We observe that along with the train- ing, all levels gradually incorporate additional experts. Notably, the hightresource group witnesses the most sig- nificant surge in the number of experts, increasing from 8 to 11, while that of low-resource group allocated expert remains 8. It is within our expectations since the high-resouree case requires a larger network capacity to process more samples, whereas the low-resource tive mainly because it expands the exploration space of SMoE by enlarging # expert candidates | for each language I. Note that there is no additional computing cost since it still only activates two experts at the second-leve -evell token routing.
fixed random
16.38
20.94
11.82
28.15
26.49
29.81
learned
32.32
31.20
33 44
needs a smaller number of experts. Dynamic Expert Allocation can be effec-
Different Number of Sppeuige Experts. The quantity of expert candidates at the first-level language routing is a crucial hyper-paramenn impact on the exploration scope of the routers. The ablation results of Lí sinuaal 1 - with (4, 8, 16, 32) expert capaci- ties in the language router are presented in Table 5. If the value of S is set to 32 (i.e., using full experts), it degrades to a clas- sic t routing. Lingual-Ski
Low High
Red
numoer of expert
in our SMoE due to its significant
10
with S; seems to be a "sweet point" for superior results.
Traiing
Staps
Figure 5: The dynamics of if expert.
Routing Visualization. To investigate whether our routing decisions are grouped based on language similarity and vice versa, we visualize the expert assignments for different languages of the final encoder (Fig- ure A7) and decoder (Figure A8) layers. We vi- sualize routing decisions of three language fam- ilies: Slavic languages (bg, sk, s1, hr). Germanic languages {nb,, de), and Indo- Iranian languages (as, mr). As shown in the visualizations, the expert distributions within each group, i.e., the heatmap color patterns, are similar but not identical. For example, the Slavic language group ([bg ~ hr) from the first to the fourth row) prefers experts 9 and 10 in the encoder and experts 0, 10, and 11 in the decoder. It evidences that our proposals indeed capture the linguistic hierarchy in the routing, where both language and language family types affect each token's 's final expert selection.
Linguistics-Guides
Table 5: Ablation on # language-specific experts
(Si) in Lingual -SMoE The SMoE baseline Top-2 is equivalent to the one of Lingual-SSM with S; = 32,
Method
Avg.
en-xx
xx-en
Top-2 (w. S 32)
30.29
28.28
32.31
LGR w. S 16
30.21
28.87
31.55
LGR w. S =
32.32
31.20
33.44
LGR S: = 4
30.24
29.03
31.45
5 CONCLUSIONS
Sparse ofxture-of-Expertt (SMoE) is a practical approach for multilingual machine translation as it allows a significant model capacity scaling while minimizing the extra computational overhead. Nevertheless, current practices overlook the linguistic characteristics that languages are hierarchi- cally grouped and differ in complexity. In this work, we introduce a novel SMoE design for mul- tilingual machine translation, named winnnual-MMas Our approach incorporates linguistic infor- mation into the routing process using a hierarchical router at both language and token levels. Ad- ditionally, we propose a flexible expert allocation mechanism that adjusts the number of candidate experts based on training dynamics and conditional on the translation difficulty. Numerous stud- ies various dense and SMoF. architectures consistently showcase the performance improvements from our framework. Future plans include the extension to multiple modality scenarios.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The authors have devoted a considerable amount of effort to ensure the methods and results in this paper are reproductive. Section 4.1 and Appendix AI, A9 provide details about the datasets and preprocessing. Section 4.1 guides the readers through the experimental procedure and evaluation metries. The implementation of Lingual- EMc,, along with the Dense and SMoE baselines are presented in Section 4. I as well. In addition, the codes to train and evaluate our methods are included supplementary materials.
REFERENCES
Roce Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3874-3884, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. Findings of the 2018 conference on machine translation (wmt18) 8). In Confer- ence on Machine Translation, 2018
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829544
Tianlong Chen, Xuxi Chen, Xianzhi Du, Abdullah Rashwan, Fan Yang. Huizhhng Chen, Zhangyang Wang, and Yeqing Li. Adamv-moe: Adaptive multi-task vision mixture-of-experts. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17346-17357, October 2023a.
Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, and Zhangyang Wang. Sparse moe as the new dropout: Scaling dense and self-slimmable transformers. In The Eleventh International Conference on Learning Representations, 2023b.
Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with isstrbutionnspeciilizee experts. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 5383-5395. PMLR, 23-29 Jul 2023c.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Menseh, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George den Driessche, Eliza Rutherford, T. W. Hennigan, Matthew G. Johnson, Katie Millican, Albin Cas- sirer, Chris Jones, Elena Buchatskaya, David Budden, L.. Sifre, Simon Osindero, Oriol Vinyals, Jack W. Rae, Erich Elsen, Koray Kavukcuoglu. and Karen Simonyan. Unified scaling laws for routed language models. In International Conference on Machine Learning. 2022.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics, 2019.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. survey of multilingual neural machine trans- lation. ACM Comput. Surv., 53(5), sep 2020. ISSN 0360--330..
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics, 2019.
10

---

Published as a conference paper at ICLR 2024
Maha Elbayad, Anna Sun, and Shruti Bhosale. Fixing MoE over-fitting low-resource languages in multilingual machine translation. In Findings of the Association for Computational Linguis- fics: ACL 2023, pp. 14237-14253, Toronto, Canada, July 2023. Association for Computational Linguistics.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1), jan 2021. ISSN 1532-4435.
William Fedus, Barret Zoph, and Noam M. Shazeer. Switch transformers: Scaling to trillion param- eter models with simple and efficient sparsity. J. Mach. Learn. Res., 13:120:1-120000 2021.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. ArXiv, abs/2209.0166 2022.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjan Krishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Associa- lion for Compulational Linguistics, 10:522-538, 2021.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zeutlemoyer. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5557-5576, Seattle, United States, July 2022. Association for Com- putational Linguistics.
Kevin Heffernan, Onur Çelebi, and Holger Schwenk. Bitext mining using distilled sentence rep- resentations for low-resource languages. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pp. 2101-2112, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3:79-87, 1991.
Hao Jiang, Ke Zhan, Jianwei Qu, Yongkang Wu, Zhaoye Fei, Xinyu Zhang, Lei Chen, Zhicheng Dou, Xipeng Qiu, Zi-Han Guo, Ruofei Lai, Jiawen Wu, Enrui Hu, Yinxia Zhang, Yantao Jia, Fan Yu, and Zhao Cao. Towards more effective and economic sparsely-activated model. ArXiv, abs/21 10.07431, 2021.
Melvin Johnson, Mike Schuster, Quoc V. Le. Maxim Krikun, Yonghui Wu, Z. Chen, Nikhil Tho- rat, Fernanda B. Viégas, Martin Wattenherg, Gregory Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351, 2016.
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6:181-214, 1994.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Conference on Empirical Methods in Natural Language Processing, 2018
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3577-3599, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

---

Published as a conference paper at ICLR 2024
Surafel Melaku Lakew, Aliia Erofeeva, Matteo Negri, Marcello Federico, and Marco Turchi. Trans- fer learning in multilingual neural machine translation with dynamic vocabulary. In International Workshop on Spoken Language Translation, 2018.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. haar: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- tations, 2021.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, 2021.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zenteemyer. Branch- tranc--triin-merg:: Embarrassingly parallel training of expert language models. In First Workshop on Interpolation Regularizers and Beyond at NeurIPS 2022, 2022.
Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. Learning language specific sub-netwont for multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing Vollme 1: Long Papers), pp. 293 305, Online, August 2021. Association for Com- putational Linguistics.
Graham Neubig and Junjie Hu. Rapid adaptation of neural machine translation to new languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 875-880, Brussels, Belgium, October-Nove November 2018. Association for Computational Lin- guistics.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference Machine Translation: Research Pupers, pp. 1-9, Brus- sels, Belgium, October 2018. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In North American Chapter of the Association for Computational Linguistics, 2019.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. Contrastive learning for many-to-many multi- lingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 244-258. Online, August 2021. Association for Com- putational Linguistics.
Telmo Pires, Robin Schmidt, Yi-Hsiu Liao, and Stephan Peitz. Learning language- specific layers for multilingual machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pp. 14767-14783, Toronto, Canada, July 2023. Association for Computational Linguistics.
Matt Post. A call for clarity in reporting bleu scores. In Conference on Machine Translation, 2018.
Taido Purason and Andre Tättar. Multilingual neural machine translation with the right amount of sharing. In European Association for Machine Translation Conferences/Workshppp 2022.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed- moe: Advancing mixture-of-expert inference and training to power next-generation scale. In International Conference on Machine Learning. 2022.
Surangika Ranathunga, En-Shium Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. Neural machine translation for low-resource languages: A survey. ACM Computing Surveys, 55:1 37, 2021.
12

---

Published as a conference paper at ICLR 2024
Carlos Riquelme, Joan Puigcerver. Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Neural Information Processing Systems, 2021.
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In Neural Information Processing Systems, 2021.
Devendra Singh Sachan and Graham Neubig. Parameter sharing methods for multilingual self- attentional translation models. In Conference on Machine Translation, 2018.
Noam Shazeer, * Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hecht- man. Mesh-tensorflow: deep learning for supercomputers. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Systems, 18PS'18, pp. 10435-10444, Red Hook, NY, USA, 2018. Curran Associates Inc.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 14, pp. 3104-3112, Cambridge, MA, USA, 2014. MIT Press.
Xu Tan, Yi Ren, Di He, Tao Qin. Zhou Zhao, and Tie- Yan Liu. Multilingual neural machine trans- lation with knowledge distillation. ArXiv, abs//902.
2019.
NLLB Team, Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan. Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loie Barrault, Gabriel Mejia Gonza- lez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm' an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang- No language left behind: Scaling human-centered machine translation. ArXiv, abs/2207.04677 2022.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Brandon Yang, Gabriel Bender, Quoc V. Le, and Jiquan Ngiam. Condconv: Conditionally parame- terized convolutions for efficient inference. In Neural Information Processing Systems, 2019.
Yilin Yang, Akiko Eriguchi, Alexandre Muzio, Prasad Tadepalli, Stefan Lee, and Hany Hassan. Improving multilingual translation by representation and gradient regularization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7266-7279, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1628-1639, Online, July 2020. Association for Computational Linguistics.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations, 2021.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-expert with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
13

---

Published as a conference paper at ICLR 2024
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam M. Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. ArXiv, abs/2202.08906, 2022.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations, 2022.
14

---

Published as a conference paper at ICLR 2024
A1
MORE IMPLEMENTATION DETAILS
Table A6: The statistics of the OPUS- 100 datasets and its sub-datasets.
Languages
Datascts
Groups
Train
Validation
Test
All
high
med
low
OPUS-16
16
8
17, 559, 950
30 x 1000
30 1000
0P1S-50
17
50
24
13
13
54, 444, 772
96 x 1000
96 1000
0PUS-100
26
100
45
28
21
107, 826
188 x 1000
188 x 1000
Language Grouping Loss Details. Given the list of all language embeddings {æ;} as input, each has corresponding language family label y:- The objective of the language grouping loss Ci is to minimize the embedding distances for language embedding pairs of the same class, while maximiz- ing those for pairs of different classes. The computation of Lt is illustrated below:
2; 2j
8
(3)
a 2 |ayyle
Li( 2y. 1y: yol ||1 st + 1 [ + yo ||
For each pair of language embeddings æ; and Zjj, their cosine similarity St,j is computed as measure of the embedding distance. Then the language grouping loss Li of this embedding pair is 1 - Si,j if they belong to the same language family, and Si,j if they are from different language families.
Table A7: Comparison of routing mechanism of SMoE baseline models and variants of Lingual-Stie SMoE. Routing Granularity: router input in encoder and decoder SMoE layer. Top-k: number of router activated experts. Router:
whether the router is lcarnable or fixed. LGR: whether to enable Language Guided Routing. Shared: whether to enable shared expert.
Routing Granularity
Top-k
Model
Router
LGR
Shared
Encoder
Decoder
Language
Token
GS-SMoE
Token
Token
None
Learnable
X
X
ST-SMoE
Token
Token
None
Learaalee
X
x
15-SMoE
Source
Target
Nome
2
Fixed
X
X
Hybrid-SMoF
Token
Target
None
2
Learnable
X
X
Residual-SMal SMoE
Token
Token
None
2
Learnable
X
LGR-SMot
Target
Target
8
2
Learnable
X
LGR
Targot
Target
8
2
Learnable
Liaguel-8McE SMcE
Target
Target
Dynamic
2
Learnalle
Model Efficiency Details. The model size and the number of tera floating point operations (TFLOPs) are reported to measure the computational cost. The TFLOPs are evaluated of 128 identical samples in the OPUS dataset, with an input sequence length of 31 and a target sequence length of 25. For inference efficiency, report average tokens processed per second (token/s) on the same test set. For training efficiency, we report the average second cost per step (s/step). We report the model efficiency metries in Table2 of our Lingual wnngall-Maas top of of the current SOTA SMoE models GS-SMoE. As shown in Table A8, Our design improves translation performance with only marginal additional parameters. Table A8: Model efficiency of Lingual-SMor and GS-SMoE.
set
Model
Model Size
TELPPs
Inference token's
Training s/step
GS-
148.8M
1.05
469.84
1.208
Lingual -SKOE
149.48M
1.05
456.83
1.205
A2 MORE EXPERIMENT RESULTS
Extra Evaluations. We provide the ChrF, COMET, ROUGE-L, and METEOR scores of Dense, GS-SMoE, and Lingual-SMoE trained on OPUS-50 50 datasets in Table A10, which show that the advantages of our Lingual-SMoE across different metries. The ChrF score is computed using Sacre- BLEU. The COMET score is calculated using the COMET framework. The ROUGE-L, and ME- TEOR scores calculated with the HuggingFace evaluate library.
A15

---

dz nilo-saharan Published as a conference paper at ICLR 2024
Table A9: All languages in OPUS-1 and their corresponding abbreviations (abbr.) and language
groups.
Language
abbr.
Group
Language
abbr.
Group
Hebrew
afroasiatic
Portuguese
pt
indo-curopean romance
Arahic
ar
afroasiatic
Romanian ro
indo-curopean romance
Maltese
afroasiatic mt
Spanish
es
indo-curopean romance
Hausa
ufrousiatic ha
French
indo-curopean romance
Amharic
am afroasiatic
Italian
it
indo-curopean romance
Vietnamese
vi austroasiatic
Catalan
indo-european romance
Khmer
km austroasiatic
Galician
indo-european romance
Malay
ms
austroasiatic
Walloon
wa
indo-european romance
Indonesian
id
austroasiatic
Occitan
indo-curopean romance
Malagasy
mg
austroasiatic
Aragonese
indo-curopean romance
Mongolian
mn
mongolic
Bulgarian
bg
indo-curopean slavic
Sinhala
si
dravidian
Slovak
indo-curopean slavic
Malayalam
dravidian
Slovenian
indo-curopean slavic
Tamil
dravidian
Croatian
indo-curopean slavic
Telugu
te
dravidian
Polish
pl
indo-european slavic
Kannada
dravidian
Ukrainian
uk indo-european slavic
Lithuanian
indo-curopean baltic
Russian
indo-curopean slavic
Latvian
lv
indo indo-european baltic
Bosnian
indo-european slavic
Irish
ga
indo- auooeeurpeaa celtic
Serbian
sr
indo-european slavic
Welsh
cy
indo-european celtic
Czech
cs
indo-curopean slavic
Breton
br
indo-curopean celtic
Macedonian
mk
indo-curopean slavic
Scottish Gaelic
gd
indo-european celtic
Serbo-Croatian
sh
indo-european slavic
German
de
mdd-eeurpean germanic
Belarusian
be
indo-european slavic
Danish
da indo-european germanic
Basque
isolate
Dutch
nl indo-european germanic
Japanese
japonic
English
indo-curopean germanic
Georgian
ka kartvelian
Swedish
indo-european germanic
Korean
ko
koreanic
is Norwegian Nynorsk
Icelandic
indo-european germanie
Kinyarwanda Xhosa
niger-congo
Norwegian
indo-european germanic
xh
niger-congo
Norwegian Bokmal
indo-european germanic
Igbo
i
niger-congo
Afrikaans
indo-european germanic
Zulu
zu
niger-congo
nn
indo-european germanic
Yoruba
yo
niger-congo
Western Frisian
fy indo-european germanic
Chinese
sino-tibetan
Yiddish
yi
indo-european germanic
Burmese
my
sino-tibetan
Limburgish
li indo-european germanic
Thai
tai kai-kadai
Dzongkha
Turkish
tr
turkic
Persian
fa indo-curopean indo-iranian
Azerbaijani
az
turkic
Bangla
bn indo-curopean indo-iranian
Uzbek
uz
turkic
Assumese
indo-european indo aado-iranian
Uyghur
ug
turkic
Gujarati
indo-european indo-iranian
Kyrgyz
ky
turkic
Tajik
indo-european indo-iranian
Kazakh
kk
turkic
Nepali
indo indo-european ando-iranian
Tatar
turkic
Punjabi
pa hi indo-european indo-iranian
indo-european indo ando-iranian
Turkmen
turkic
Urdu
indo-european indo-iranias
Hungarian
hu
uralic
Hindi
Estonian
uralic
Marathi
indo-european indo-iranian
Finnish
uralic
Pashto
ps or indo-curopean indo-iranian
indo-european indo-iranian
Northern Sami
uralic
Kurdish
indo-curopean indo-iranian
Esperanto
co constructed
Odia
Greek
indo-curopean hellenic
Armenian
hy indo-curopean armenian
Albanian
indo-curopean albanian
Table A10: Multilingual machine translation performance on OPUS5C datasct.
Model
ChrF
COMET
METEOR ROUGE-L
Dense
43.14
72.82
40.99
41.47
GS-SMoE
45.36
74.46
42,38
43.15
Lingual-SMoß
46.56
75.52
43.06
43.96
Dynamic Expert Allocation with Grammar Complexity and Data Abundance. As outlined in the introduction, our study examines language difficulty from two perspectives: grammatical com- plexity and resource availability, which are combined by assigning corresponding scores. (1) In terms of available data, following (Zhang al., 2020), languages with sample sizes exceeding 0.9
A16

---

Published as a conference paper at ICLR 2024
Table A11: Performance comparisons of i, nggallll--- -MMoeetrrr with dynamic routing according language difficulty defined by grammar complexity and data abundance, compared Dense and vanilla SMoE GS-SMoF. Average BLEU scores for each direction are reported.
Methods
Avg.
xx-en
Avg.
high
medium
low
Avg.
high
medium
low
Dense
28.79
26.92
25.37
39.12
14.78
30.67
28.81
39.24
24.21
GS-SMoE
30.29
28.
25.55
40.71
18.97
32.31
28.77
41.70
29.25
Lingual MaopallSMMBBarrr
32.55
31.36
27.05
45.82
23.59
33.73
29.95
43.61
30.66
million samples are categorized as high-resource languages, while those with less than 0.1 million samples are considered low-resource languages. Languages falling between the two thresholds are classified as medium-resource languages. Scores of 0, 1, and 2 are assigned to low, medium, and high resource languages, respectively, with higher scores indicating greater data scareity. (2) Re- garding grammatical complexity, each language is rated on a scale of 1 to 5 using GPT-4, reflecting the level of difficulty from easy to hard. (3) Then the language difficulty metric is computed by summing the scores derived from resource availability and grammatical complexity. During train- ing, languages are categorized as easy, medium, and hard based on their language difficulty scores, maintaining the same distribution of high, medium, and low resource languages. Next, train Mingual-SMool SMoE with the language difficulty groups in the OPUS- 16 settings and compare to the dense and SMoE baselines. As shown in Table S1, Lingual-SMee Eeatrrr consistently out- performs the baseline models due to the incorporation of both grammar and data information. The reason why we exclude language difficulty when combining the grammar complexity score and the data availability score is twofold. First, the amount of data is an objective metric, but grammar com- plexity is relatively subjective for people with different first languages and education levels. Also, the definition of grammar complexity varies from GPT to human, and even from case to case within GPT. So we decide not to include grammar complexity in our Lingual -MMEE.
69
.0
0.96
0.96
91
0.94
0. 94
0.92
0.94
00
0.95
0.96
0.
0.97
0.94
0.94
0.90
0.92
Sk
0.96
1.00
0.96
0.95
0.91
0.82
0.88
0.91
1.05
0.35
1.00
0.95
0.95
0.93
0.93
0.90
0.92
1.05
0.96
1.00
0.98
0.93
0.93
0.92
0.94
96
0.95
1.00
0.97
0.92
0.94
0.90
0.92
1.00
1.00
0.95
0.98
0.95
0.92
0.95
0.97
0.95
0.00
0.94
0.95
0.91
0.92
0.93
0.91
nb
0.34
2.00
0.95
0.95
de
0.90
a5
0.90
0.95
mr
0.93
0..5
sk
s1
hr
nb
de
as
mr
bg
sk
hr
nb
de
mr
(a) Encoder Routing Decision Similarity.
(b) Decoder Routing Decision Similarity.
Figure A6: Routing decision similarities of the last encoder and decoder SMoE layer of Lingual-SMoB trained on OPUS-100 en-xx language pairs. Three groups target languages {bg, sk, sl, hr). (nb, de), {as, mr) are presented, Darker blocks imply higher similarity.
A17

---

Published as a conference paper at ICLR 2024
0.07
0.05
00004
0.03
0..02
Figure A7: Linguistics routing decisions of the encoder's 's last layer for en en-xx xX language pairs. Lingual -MoE is trained on OPUS- 1 100. We visualize 3 languages groups
0.040
0.033
0.030
0,034
0.020
0.048
30 31
{bg, sk, sl, hr), (nb, de), (as, mr) ). The darker cell is, the more likely the language router chooses the expert for translating that target language.
0.266
0.024
0.022
0.025
0.029
0:075
0.036
0.029
0:03
0.033
0.036
0.029
29
0.033
0.026
0033
8200
0.04
28
0.255
0.027
0.233
0:300
0.025
0.028
27
0.024
0.039
0.02
0.030
26
0.019
0332
0227
0338
0.050
25
0.032
0.034
0229
0.029
0.355
0.5 0.027
24
02020
023
0.028
010
0:021
23
0.041
0.042
0.037
0.037
0.029
22
0.022
0.025
0.299
0.030
0.036
0.038
21
049
032
024
0.024
20
D40 0.036
0.029
0.0266
0.037
0.00
0.045
19
081
0.044 0 032
0344
027
18
0.06
0.027
0.040
0.053
17
650 0.039
0,031
042 0,020
0.032
0.031
0.041 039 0.028
16
Expert
0.036
0 045
0.82
14 15
0.025
0.012
0.028 0.020
0.19
0.039 0.032
0.024
0.039
0.22
0.034
13
0.177
025
0.18
0038
0.0D
0.023
0.020
12
0.022
0.030
0.032
0.033
0.023
0.043 0.022
0.017
11
0.047
0.045
0.044
0.021
0.022 0.027 0.052 0.028
10
0.022
0442
0.004
048
0.028
9
-0 0.031 036 0.022 0:026 030 0.023 0:027
0.020
0 022 0.019 0.035 0. 022 0.034 0.028 D. 042 0.028 0.031
0.026 0,020
0,051 0.033
0.030
0:028
8
0.037
030
0.030 0,027 0. 031
0.027 0.014 D 051
-0 027 0.027 0.030 0 025 0.033 0.022
0.024
-0 017 0:035 0.00 0:021
5
0.047
033 0.033 0,029 0. 034
019 0.039 0.031 0 028
0.035 0.044
2
15 3% bg
qu
mt as
A18 A18

---

Published as a conference paper at ICLR 2024
0.07
0.06
0.03
-0.02
We visualize 3 languages groups
5,027
5.03
5.000
0.023
5.029
5.011
31
(bg, sk, sl, hr), [eb, de). {as, me). The darker a cell is, the more likely the language router chooses the expert for translating to that target language.
0.02
0.024
0.03
0.02
0.03
6.03
30
0,50
0,22
0.04
00023
6.233
0,033
29
0,27
02223
0.356
0.227
0.30
28
0032
0.27
0:34
6:33
6:23
6233
27
Figure A8: Routing decisions the decoder's last layer for en- XX language pairs. Lingual- SMoE is trained OFUS-2 00.
0025
0.22
0.33
024
020
0.26
26
0.228
0.231
0.331
0:224
0:336
0.024
25
3:222
5.255
5.002
0.228
5.255
5.007
24
0.032
5.00
0.03
23
0.01
0.030
0.32
0.035
0.03
0.00
22
5,36
5,006
5,03
2035
5032
21
6233
02337
0023
0:223
0.233
20
0015
0015
0012
0022
0.03
19
0.026
0:35
0.03
0024
0:33
18
0.220
0.22
0.20
0.225
17
2:081
0.003
5.000
5.224
2.03
16
Expert
0.035
0.000
0.044
0003
033
0.03
15
5025
0025
53338
5023
0,02
14
0.008
5.023
5.003
0.09
13
0.033
0.02
0.023
0.023
0:03
0.03
12
0123
0:95
0:032
0.23
0.20
0.02
11
0,225
0:285
0:255
0.34
0.23
0.333
10
3.033
0.008
3.000
2.025
5.000
5.003
9
0022
0135
0223
0.32
8
0.225
0.220
0:35
0232
0:20
5.031
5,005
5.007
5:03
0.022
0.027
0.022
5
0,52
0,02
0010
0022
6.25
2 4
0.225
0:337
0,335
0.225
0.225
0.355
0035
0:25
0:25
0222
0324
6323
0,20
0.023
0.022
01044
0.033
0:944
950
0:225
0.223
0.023
b9
S1
mi Se de nb
aanuggge
Language
A
A19


Reviews:
[{"review_id": "l9u4fN147G", "reviewer": "Reviewer_BJsk", "summary": "The authors propose Lingual-SMoE, an MoE model suited to multilingual MT that overcomes limitations of previous work. It features:\n1) Has hierarchical gating that leverages linguistic groupings of the languages on which the model is trained.\n2) Uses a Dynamic Expert Allocation during training to determine the correct number of experts to allocate to each target language.\n\nThey conduct experiments on the OPUS-100 dataset with 16-100 languages, and extensively compare Lingual-SMoE to the baselines.", "questions": "- wrt result with DEA added in Tables 1 and 2, do you have any analysis on why adding this is worse on LRLs for OPUS-16,100 compared to OPUS-50 (ie, the trend is reversed)? is, say, the number of experts per low resource language suddenly worse for 16,100? are there outliers for the 50 case? a trend reserval of +/-0.2 is not bad, but this is an order of magnitude reversal of the trend\n\n- wrt Figure 4, is this equally an issue for dense models? (it's known that this is an issue in general, but it is particularly different than dense models?)\n- What are the inference implications of your method? I think even theoretical trade-offs and discussion of this would be interesting, since the most related works discuss this extensively", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors did several experiments to justify their design choices and overcome the limitations of previous work.\n- I think the Dynamic Expert Allocation technique is intriguing and intuitive, and the results+analysis seem good. (See Questions section, though)\n- The authors did extensive ablations on their proposed techniques, and compared to a number of baselines.\n- The analysis based on empirical results is also good, and answer most of the questions a reader would have.", "weaknesses": "- Notation for the baselines is confusing - eg. from saying LS-SMoE or Hybrid-SMoE is not clear which baseline you are referring to - it made understanding the tables cumbersome. I suggest using slightly longer names like Switch MoE or GShard MoE or Hybrid TaskMoE etc (and preferrably organize as a list or table)\n- I think you should reframe the language complexity part as language resourcedness. It's not that any language is more or less complex, it's the amount of data available and handling under/overfitting. This will also match all the analysis and description of DEA in rest of the paper.\n\n\nSome papers to cite:\n- https://arxiv.org/abs/2108.05036, https://arxiv.org/abs/2208.03306 - while these papers are for multi-domain setting, I believe these are related in spirit given that they leverage differences in input to allocate experts.\n- https://proceedings.mlr.press/v202/chen23aq.html - related to DEA, but for a different purpose\n\n\nMinor: \n- In figure 5, are medium resource languages and low resource languages mixed up? If not this is counter intuitive and worth elaborating on/doing analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CGZ8SijtdH", "reviewer": "Reviewer_mayu", "summary": "This paper proposes a modification of sparse mixture-of-experts (SMoE) for multilingual MT that makes routing dependent on language-specific representations (and hierarchical), and automatically adjusts the number of activated experts for each language. These two modifications are introduced to more explicitly allow for grouping by languages based on their similarity, and for allocating experts based on complexity. The new SMoE variant is tested on a range of subsets of the OPUS-100 dataset and compared against various previous SMoE variants. A set of ablations further investigates the functionality of the expert allocation strategies.\n\nScore was raised after author response.", "questions": "- [answered] Are the baseline models reimplemented? Are they coded in the same codebase as the proposed model?\n- [answered] Can you quantify the similarity of routing patterns within language groups with cosine similarity in 4.3 as you did in Figure 3? Visual inspection is still hard.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The hypothesis is convincing and the design of the methods follows the hypothesis and is thoughtfully laid out and is original.\n- The set of empirical evaluations and ablations is rich (if one disregards the problems with the benchmark, see below) and sound, aligns well with the hypothesis, and presents the proposed SMoE solution as an empirically successful method and favourable in comparison to other SMoE variants.", "weaknesses": "- Empirical comparisons:\n   - [authors clarified: reimplementation] The baseline scores for the Dense model on OPUS100 are much lower than what was reported in previous papers, in particular Zhang et al. 2021 (https://openreview.net/pdf?id=Wj4ODo0uyCF), even though it’s supposed to be the same architecture and training data and evaluation metric. For example, Zhang et al.’s M2O model scores on average 29.27 BLEU, while this paper reports 25.39 in Table 2, analogously for O2M 20.93 vs 19.03. Perhaps I missed an obvious difference in modeling that could explain the difference in results, but it is questionable whether the dense baseline was tuned sufficiently, and the same for all sparse baselines.\n  - [authors promised more results on OPUS100] The prominently discussed results are reported on a subset of the languages in OPUS100 (Table 1), and the two most competitive baselines (ST-SMoE, Residual-SMoE - the two baselines with 100% win ratio) are left out for the full evaluation on all languages (Table 2). This leaves the competitiveness of the proposed solution in a realistically large multilingual setup to be questioned.\n- [authors provided WMT benchmark] Choice of benchmark: Unfortunately, OPUS100 is designed in such a way that different languages contain data for different domains, both in training and in test sets. This arises from the fact that many low-resource languages are only covered by religious datasets or tech localization datasets on OPUS. Therefore, there is a relatively strong domain interference, where higher resource languages also have more complex and diverse data, and lower resource languages have less complex and more repetitive data. In past work (Kreutzer et al. 2021 (https://arxiv.org/pdf/2110.06997.pdf), this was already suspected to interfere with language-as-a-task modeling), and it also explains why low-resource languages in the aggregated results often have higher avg BLEU than mid-resource languages (cf. Table 2), as they have simpler test sentences with less domain diversity. The effect seems less drastic with the selected subset of 16 languages (Table 1), but this is a less realistic setup in general. \nThe similarity between domains can interfere with language hierarchies that are supposed to be modeled here, i.e. data points sampled from distant languages but from the same domain can potentially be more similar to each other than data points from similar languages but different domains. This might blur the linguistics-guided routing visualization in 4.3 - it could be that routing was rather based on domains than languages, or a combination of both. With this dataset, it is unfortunately impossible to tell.\nFurthermore, the amount of data per language is therefore not a suitable (sufficient) metric for task complexity or linguistic difficulty as argued in 4.3.\nI would strongly recommend the authors to redo the experiments on a domain-controlled multilingual benchmark, such as a combination of WMT datasets across languages, as e.g. in Cheng et al. 2022 (https://arxiv.org/pdf/2203.07627.pdf). It might be a lot of effort, but eventually worth it to present the success of the proposed method with less interference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ksp6LgPCY8", "reviewer": "Reviewer_xU8U", "summary": "This paper introduces a novel approach known as Lingual-SMoE, aimed at enhancing multilingual machine translation. In contrast to conventional methods, Lingual-SMoE takes into consideration the distinct linguistic features and complexities of various languages. The experimental findings are promising in specific translation directions, as Lingual-SMoE consistently outperforms traditional techniques in multilingual translation tasks. For instance, it exhibits an increase of over 5% in BLEU scores on the OPUS-100 dataset. This underscores Lingual-SMoE's capability to effectively handle translation tasks for various languages, improving overall accuracy", "questions": "1.\tAs depicted in Figure 2, how is the difficulty of a language determined?\n\n2.\tBased on your findings in Table 1 and Table 2, it appears that your method has demonstrated improved performance in the en-xx language directions. However, the performance in the xx-en language directions appears to be less favorable than previous results, particularly in the context of medium to high-resource translation tasks. Have you conducted a detailed analysis of this phenomenon, or is it possible that your method still has limitations when it comes to xx-en tasks?\n\n3.\tRegarding Eq. (2) in Section 3, it is mentioned that \\alpha and \\beta are empirically set to 0.05. However, in the Implementation Details section, \\alpha and \\beta are set to 0.98. Could you please clarify whether these refer to the same parameters?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tConsideration of Linguistic Characteristics: This approach takes into account the linguistic hierarchy and the complexity of various languages, thus enhancing its performance in multilingual machine translation tasks. The initiative is highly motivated and intriguing in terms of how it organizes models within the MoE framework to match the specific characteristics of language pairs.\n\n2.\tHierarchical Routing Strategy: In order to implement this proposed concept, it introduces a hierarchical routing strategy that takes into consideration language families and token-level information. This results in a more efficient allocation of experts, optimizing the utilization of resources and routing decisions.\n\n3.\tAdaptive Expert Allocation: Another noteworthy proposal is the mechanism for adaptive expert allocation, which can automatically adjust the capacity of experts based on the translation difficulty of each language. This helps address issues related to overfitting and underfitting. The method is both intuitive and technically robust.\n\n4.\tEmpirical Validation: Extensive experiments, carried out on a variety of language pairs and scales, have consistently demonstrated the effectiveness of this approach across different data resources and language quantities. It consistently leads to a significant improvement in performance, particularly on the OPUS-100 dataset.", "weaknesses": "1.\tIn the paper, the authors claim to address the issues of Linguistic Hierarchy and Language Complexity. The former refers to the categorization of languages based on their language families, while the latter pertains to \"grammar complexity\" or \"language difficulty.\" However, I believe that the second claim may not be appropriate, as it appears to relate more to the availability of training data rather than addressing the inherent challenges of the languages themselves. Additionally, there is a lack of experiments or analyses to support the claim that their method genuinely considers the intrinsic difficulty of languages.\n\n2.\tIn the experiments, for the purpose of fair comparisons between the baselines and the proposed methods, it is recommended that the authors report the sizes of the models. It seems that the improvements may be attributed to the increased number of experts. Therefore, providing and discussing the size of model parameters, along with the computational complexity of the proposed hierarchical framework in terms of training and decoding time, would be beneficial.\n\n3.\tSome technical details require further clarification, and I would like to refer to the specific questions raised in the Questions.\n\n4.\tThere are several typos that need correction, such as:\n\n   - There are issues with citation formats and grammatical agreement in the last paragraph of the Related Work section.\n   - In Section 4.2, \"Table 1\" is incorrectly cited as \"Table 4.2.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9t6hP9xBHV", "reviewer": "Reviewer_BUYj", "summary": "The presented work proposes to add linguistic information into the mixture-of-expert routing process and shows the effectiveness on the OPUS-100 dataset compared to other gating choices.", "questions": "- p.6: What is $\\alpha$ for the Adam algorithm? Shouldn't this be either $\\beta_1$, $\\beta_2$ or $\\epsilon$? (see Table 4 in [Schmidt et al., 2021](https://proceedings.mlr.press/v139/schmidt21a.html)). Also, please cite the original Adam paper accordingly.\n- Is the code going to be open-sourced?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting work adding linguistic features and language complexity into the routing behaviour\n- Innovative design of an adaptive expert allocation mechanism\n- The paper is well written and easy to follow\n- Nice ablations for Linguistically Guided routing (Table 3) and the Language Router Designs (Table 4)", "weaknesses": "### Weaknesses\n\n- **[major]**: Please provide the `sacrebleu` hash that is used for evaluation such that scores can be reproducible\n- **[major]**: While BLEU is still widely used, there are now better metrics to use for Machine Translation that correlate better with human judgment as seen in [Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust](https://aclanthology.org/2022.wmt-1.2) (Freitag et al., WMT 2022), specifically I'd recommend including chrF and COMET scores.\n- **[major]**: One of the key takeaways of [Pires et al. 2023](https://aclanthology.org/2023.acl-long.825/) is that target language specific routing is helpful in the encoder and not only in the decoder and only source language routing the encoder hinders learning. This is true for both shared as well as language-specific decoders according to their experiments (see their Table 1 + Table 5). I think the baseline (1) in the presented work could be improved with two approaches to more closely match their setup by 1) converting the source language routing in the top 25% of encoder layers to target language routing and 2) deploying their proposed dense pre-training approach.\n- **[minor]**: Building on top of the previous point, I think even LGR-SMoE could benefit from the dense pre-training as this was also shown to be very beneficial in both [Pires et al. 2023](https://aclanthology.org/2023.acl-long.825/) and [Tricks for Training Sparse Translation Models](https://aclanthology.org/2022.naacl-main.244) (Dua et al., NAACL 2022).\n- **[minor]**: The justification on p.7 that (1) performs much more biased towards English doesn't make much sense as the results in Table 1 only include `X` $\\leftrightarrow$ `en` scores and OPUS-100 is an English-centric corpus? In any case, adding per language pair scores for all evaluation directions in the appendix would be beneficial for all methods. It might also be worth to have a LS baseline where only the decoder is language-specific for the target language and we do not have any language specific routing in the encoder since this granularity might be too much for the small english-centric OPUS-100 dataset.\n- **[minor]**: How big is the inference speed / parameter count / memory consumption overhead from the additional language router? I think some concrete benchmarking numbers would be helpful here.\n- **[minor]**: There are no details about the data epochs, are all of the presented models trained until convergence and is it single epoch? How was checkpoint selection done since we likely can see overfitting for some of the lower resource languages in e.g. LS-SMoE?\n- **[major]**: It is unclear how the proposed approach influences the zero-shot translation quality and/or code-switched payloads.\n\n---\n### Typos & Minor Comments\n\n- Change citation of \"Nllbteam\" to proper capitalization and spacing\n- Double check bibliography some works such as e.g. [Massively Multilingual Neural Machine Translation](https://aclanthology.org/N19-1388) (Aharoni et al., NAACL 2019) or [Learning Language-Specific Layers for Multilingual Machine Translation](https://aclanthology.org/2023.acl-long.825) (Pires et al., ACL 2023) have published version available that should be cited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)