Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation
Authors: Biao Zhang
Ankur Bapna
Rico Sennrich
Orhan Firat
Venue: ICLR 2021 Oral (2021)
Keywords: language-specific modeling
conditional computation
multilingual translation
multilingual transformer

Abstract: Using a mix of shared and language-specific (LS) parameters has shown promise in multilingual neural machine  translation (MNMT), but the question of when and where LS capacity matters most is still under-studied. We offer such a study by proposing conditional language-specific routing (CLSR). CLSR employs hard binary gates conditioned on token representations to dynamically select LS or shared paths. By manipulating these gates, it can schedule LS capacity across sub-layers in MNMT subject to the guidance of translation signals and budget constraints. Moreover, CLSR can easily scale up to massively multilingual settings. Experiments with Transformer on OPUS-100 and WMT datasets show that: 1) MNMT is sensitive to both the amount and the position of LS modeling: distributing 10%-30% LS computation to the top and/or bottom encoder/decoder layers delivers the best performance; and 2) one-to-many translation benefits more from CLSR compared to many-to-one translation, particularly with unbalanced training data. Our study further verifies the trade-off between the shared capacity and LS capacity for multilingual translation. We corroborate our analysis by confirming the soundness of our findings as foundation of our improved multilingual Transformers. Source code and models are available at https://github.com/bzhangGo/zero/tree/iclr2021_clsr.

Published as a conference paper at ICLR 2021
SHARE OR NOT? LEARNING TO SCHEDULE LANGUAGE-SPECIHIC CA- PACITY FOR MULTILINGUAL TRANSLATION
Biao Zhang" Ankur Bapna", Rico Sennrich* 911 Orhan Firat
1 School of Informatics, University of Edinburgh
2 Google Research
3 Department of Computational Linguistics, University of Zurich
b. zhang@ed. ac. uk, ankurbpn.google com, sennrichecl uzh, ch, orhanfégocgle.com
ABSTRACT
Using mix of shared and language-specific (LS) parameters has shown promise in multilingual neural machine translation (MNMT), but the question of when and where LS capacity matters most still under-studied. We offer such a study by proposing conditional language-specife routing (CLSR). CLSR em- ploys hard binary gates conditioned on token representations to dynamically se- lect LS shared paths. By manipulating these gates, it can schedule LS capacity across sub-layers in MNMT subject the guidance of translation signals and budget constraints. Moreover, CLSR can casily scale up massively multilin- gual settings. Experiments with Transformer on OPUS-100 and WMT datasets show that: 1) MNMT is sensitive both the amount and the position of LS modeling: distributing 10%-30% LS computation to the top and/or bottom en- coder/decoder layers delivers the best performance; and 2) one-to-many transla- tion benefits more from CLSR compared to many-to-one translation, particularly with unbalanced training data. Our study further verifies the trade-off between the shared capacity and LS capacity for multilingual translation. We corroborate our analysis by confirming the soundness of our findings as foundation of our improved multilingual Transformers. Source code and models are available at https:/ //g thub. wwwpbzaanggo// zero/ zero/tree/icteeeee r2021
INTRODUCTION
Model architecture design injects inductive biases neural network layouts, allowing a learning algorithm to favor certain representations over others, independent of the observed data (Mitchell, 1980). In multilingual neural machine translation (MNMT), where the learning objective com- monly cast as a multi-task learning problem (Firat et al., 2016a; Ha et al., 2016; Johnson et al., 2017), the inductive bias researchers usually study is deciding on which components of the neural network to share between tasks (languages), and which components to leave specific to the task or language. These components can be entire layer stacks, individual layers or even some sub-layers (Sachan & Neubig, 201 8; Blackwood et al., 2018; Wang et al., 2019; Zhu et al., 2020). Noticeably, the search space of which parameters to share and which granularity grows rapidly, as we make neural net- works large or increase the number of tasks (languages). This rapid expansion of the search space prevents us from exhaustively exploring the choice of sharing patterns in MNMT.
The incapability of full-space exploration motivates methods relying on heuristics (Sachan & Neu- big, 2018), that lack flexibility when more languages are covered, or meta-learning (Platanios et al., 2018), that often hard scale. These limitations hinder their generalization large-scale mul- tilingual models, which is the very focus of our study. In large scale multilingual models, also known as massively multilingual models (Aharoni al., 2019; Arivazhagan et al., 2019; Zhang et al., 2020b), hundreds of languages with varying amounts of training data, difficulty and linguistic properties are jointly trained together in a multi-task setup. While the joint training enables positive
Work done while Biao Zhang was interning at Google Research.

---

Published as a conference paper at ICLR 2021
Encoder
Decodar
CLSR Layor
Curput
CLBR
Food Forword Layor
Languaga Speciiic
Shared
Fowd-Forward Layer
CLSR
yh
Solt Attemtion
CLSR
SellArarttonn
x6
Emb
The model architecture used for our experiments. We introduce CLSR layer after every transformer sub-layer in the encoder and the decoder. The gating layer learns to route every input through either the LS projection layer, or a shared projection layer. We analyze the outputs of the gating layers to develop a MNMT architecture with LS projections.
Figure I:
transfer across languages, it also introduces inssffnteree between dissimilar languages (Ari- vazhagan al., 2019; Wang et al., 2020a;b) and a capacity bottleneck emerges due the increased number of languages and data (Huang et al., 2019; Zhang et al., 2020b).
In this paper we adopt an end-to-end data driven approach (conditional language-speciffs routing, or CLSR) which permits directly probing large section of the search space. We let the network learn the sharing structure from the data itself, by learning to route between language-specifie (LS) or shared pathways. These two routes determine the mode of operation for the network: when the LS branch is selected, the model is given access to a set of LS layers (implemented as simple projections per language) and when the shared branch is chosen, the computation is routed to layer that is used by all languages. By guiding the (gating) decision process with token level activation information, the network flexibly learns to alternate between the two modes and naturally lends itself to a conditional computation approach for multilingual processing (Bengio et al., 2013; Davis & Arel, 2013; Bapna al., 2020). The gate states optimized towards maximizing translation quality, but regularized with a budget constraint to control the amount of LS c Reducing the available budget results in fewer gates routing through the LS paths, enforcing CLSR identify the most crucial sub-layers which allows us to observe and study the importance of each sub-layer for multilingual processing. Our approach visually depicted in Figure 1.
We verify our proposal on WMT and the massively multilingual OPUS-100 dataset, with models building on the Transformer architecture (Vaswani et al., 2017). We explore target-specific and source-specific modeling for one-to one-to-many" and many-to-one translation, respectively. To measure the degree of each sub-layer's tendency to be language-specific, we propose LSScore metric. Our results show that CLSR successfully navigates the trade-offss in LS modeling, outperforming several strong baselines. Our main findings are summarized below:
Both the amount and the position of LS layers matter for MNMT. The best perfor- mance is achieved by distributing 10%-30% LS computation to the top and/or bottom en- coder/decoder layers.
Feed-forward sub-layers utilize more LS capacity compared other sub-layers on one-to- many translation.
One-to-many translation benefits more from CLSR (with target LS parameters) compared many-to-one translation (with source LS parameters), particularly when the training data is imbalanced.
The induced sharing pattern learned by CLSR is highly similar across languages.
We use the term "the amount LS capacity" to refer the proportion of open gates where CLSR selects route information through the 1.S path instead of its shared counterpart, which directly regularized and guided by the budget constraint p as in Fq.
In one-to-many machine translation setup, single source side language (commonly English) is tasked be translated into multiple target languages, one time.
2

---

Published as a conference paper at ICLR 2021
We can the learned patterns to hard-code better parameter sharing strategies for multi- lingual Transformers.
2 RELATED WORK
Our work closely relates to language-specifie (LS) modeling for multilingual NMT and conditional computation for sequential data which we will recap both here. Early research on MNMT focused on improving shared capacity for separate bilingual models to enhance cross- pooss-lingual transfer. These efforts included sharing encoders for one-to-many translation (Dong al., 2015), sharing decoders for many-to-one translation (Zoph& & Knight, 2016; Lee et al., 2017) and sharing sub-layers (atten- tion) for many-to-many translation (Firat et al., 201 These studies corroborated the feasibility of accommodating multiple languages with shared NMT sub-components, motivating researchers to explore universal MNMT. Ha et al. (2016) and Johnson et al. (2017) proposed such an implemen- tation that performs multilingual translation with single monolithic NMT model where the entire network is shared across languages, thanks to a target language token informing the model which language to translate into. Although this paradigm shows great scalability (Aharoni et al., 2019), the language token alone affords little flexibility in handling language diversity with a rigid share-all layout. Follow-up studies thus resort to LS modeling in an attempt to seek better trade-off between sharing and not sharing. Methods in this category involve specializing neural attentions (Blackwood et al., 2018; Sachan & Neubig, 2018; Wang al., 2019), broadening encoder outputs and normal- izations (Zhang et al., 2020b), decoupling multilingual encoders and/or decoders (Vázquez et al., 2019; Escolano al., 2020), using fixed mix of LS and shared parameters (Wang et al., 2018), in- serting lightweight adapters (Bapna & Firat, 2019) and separately modeling languages for different clusters (Tan et al., 2019), to name a few. Nevertheless, these methods heavily depend on heuristics, providing little evidence about how to optimally distribute LS capacity across the model.
By contrast, our proposed CLSR forces the model to learn LS behaviour. can be treated as a sim- plified differentiable neural architecture search (NAS) model (Liu et al., 2019) with search space defined by the presence/absence of LS projections after every transformer sub-layer. However, in contrast with NAS, we utilize conditional computation (Bengio et al., 2013) to make the choice of executing the shared or LS path conditional on the input representations. This allows us to compare and contrast the choice of LS vs shared paths on different inputs and languages. Conditional com- putation has previously been successfully applied to adapt the amount of computation to the input in recurrent models (Graves, 2016) and transformers (Dehghani et al., 2019; Elbayad et al., 2020), or to significantly scale up model capacity by utilizing sparsely-gated al., 2017; Lepikhin et al., 2020). Zhang et al. (2020a) applied conditional computation to sparsify encoder outputs in sequence- tonicc-to-seqeence models in order to reduce attention costs, while Bapna et al. (2020) introduced conditional execution of Transformer sub-layers control the amount of computation expended by the model at inference. Sukhbaatar et al. (2019) learn parameters that limit the attention spans, in order to make the attention operation more efficient, while Fan al. (2020) utilize structured dropout to prune transformer layers at inference. Ruder et al. (2019) propose the sluice network that learns the inter-task (shared) sub-spaces on top of task-specific models. By con- trast, CLSR starts with a totally shared model, and learns how to inject task-specific projections into it, which scales more easily to massively multilingual settings. Different from previous studies, explore conditional computation capacity for MNMT. Utilizing conditional computation to search for better LS sharing patterns in multilingual translation, to the best of our knowledge, has never been investigated before.
coatuure-of-Epperre layers (Shazeer
analysis tool to understand the ideal arrangement of LS
3 BACKGROUND: MNMT
Given a source sentence X' = {I1, 22, and its target translation Y {311 .2..... ..)). we follow Johnson et al. (2017) to reuse standard bilingual NMT models for multilingual trans- lation by altering the source input with language token lang, i.e. changing X' to X {lang, I1, .2,..., } Note that lang denotes the target language in one-to-many translation but source language in many- many-to-one translation.
We model translation from X to Y with Transformer (Vaswani et al., 2017). Transformer relies on structure (He al., 2015; Ba et al., 2016) smooth informa-
the following
3

---

Published as a conference paper at ICLR 2021
tion flow and avoid gradient vanishing and explosion:
z LN (z' + (2)),,
where I denotes layer depth and I.)()) is layer normalization (Ba et al., 2016). Function f(-) rep- resents the basic building block in Transformer, such as attention network or feed-forward network. The encoder in Transformer is stack of L identical layers, with each layer involving a self-attention sub-layer (SAN) and a feed-forwart sub-layer (FFN). The decoder uses a similar structure except for extra cross-attention sub-layer (CAN) inserted in-between the above two sub-layers.
4 CONDITIONAL LANGUAGE- ROUTING (CLSR)
The success of MNMT comes at the cost of expressivity and model's ability to capture language- specific characteristics. It has been empirically shown that the language signals from language indicator tokens alone are not sufficient (Arivazhagan et al., 2019), making architectures dedicated LS modeling a necessity (Blackwood et al.. 2018; Sachan & Neubig, 2018; Zhang et al., 2020b). Nevertheless, the question when and where LS modeling matters most in MNMT still remains to be answered. To this end, we propose conditional language-specifie routing (CLSR) which specializes f(-) and changes the formulation in Equation as follows:
++ LN (z' + CLSR (ʃ (2)))).
CLSR learns hard binary (scalar- valued)) gate g(-) for each input token based on its hidden repre- sentation z' E Rd. These gates endow each sub-layer in Transformer with the capability of routing information selectively through either LS path hlono or shared path h hahared,
CLSR (f (z')) = g(z') g(z')h 'aan + (1 g(2') ()zz)))" 1) shared with h lang = f (z') Wlang, h hshared = f (z') Washare
(4)
where Washared a weight matrix shared across languages, while parameter Wilng only used for modeling language lang which endows NMT with source or target LS modeling capacity. Intu- itively, a closed gate, corresponding to shared capacity, encourages maximal cross- lonss-inggul informa- tion transfer; open gate, corresponding to LS capacity instead, improves language awareness for translation albeit blocks knowledge transfer. CLSR balances between the two modes as controlled by the gates.
Following Bapna et al. (2020), we parameterize the gate g(-) with a two-layer feed-forward network G()), and inject zero-meat Gaussian noise during training to discretize it (Chiu & Raffel, 2018):
g(z')= = a (G(z') + at)WN (0, 1)), G(z') = Relu w/W1 + b) W2.
(5)
where (--) is the logistic-sigmoid function, dg is the gating feed-forward hidden dimension, and W, E Rdxd, W2 E Rd are trainable parameters. a is linearly increased along with training steps t. In this way, the gating parameters can be optimized with accurate gradients when a is small at the beginning so to measure the degree to which each position in each sub-layer benefits from LS modeling. As training progresses, a grows larger, forcing the gating network to emit hard outputs. At inference time, we discretize the gate based on simple decision rule: g( 5 (G(z') 2 0), where 8(-) is Dirac measure.
We train the gates based on the standard maximum likelihood objective, along with an additional budget regularization term that enables control over the amount of LS capacity used for translation. Let the set of all CLSR layers in the encoder be Mene and the decoder be M Then the amount of LSS computation utilized by sentence pair YX,Y) is given by G(x,y) = Σzex Eme..... Im (x+)+ + Eyer EmeMacc Im (v). Given budget constraint p [0, 1] and batch of sentence pairs, B, the training loss function of CLSR is formulated below:
Exxres G(XY) (Xx)) + + MallM))
(B)
Σ MLE(X,Y), (XY)EB
We maintain a set of language- specific weight matrices in order to compute h lang each language. To make the number parameters manageable, we share the set LS matrices Wieng across all encoder or decoder sub-layers, but distinguish Wilnng and Wrlnng I.S matrices for the encoder and decoder, respectively.
4

---

Published as a conference paper at ICLR 2021
Intuitively, the budget constraint tries to regulate the amount of LS computation available all to- kens in the batch as a fraction of the total LS computation in the model. Given that we make binary decision for every input for every CLSR layer, this corresponds to search space of This gating space not only grows with respect to the model depth and sub-layer types, but also is highly input dependent. This dependency makes difficult to search the entire space using heuristic methods (Blackwood et al., 2018).
The constraint in Equation 6 is imposed upon the aggregated gates. As a consequence, the model can learn to trade-off LS capacity for certain layers and inputs for others. Decreasing the budget encourages gate closure, such that the LS path is chosen only in the critical sub-layers. Thus, a properly learned gating function reveals the activations of LS paths and helps gain insights into the model behavior.
5 EXPERIMENTS
Data and Evaluation We report results on two benchmarks: OPUS-100 (Zhang et al., 2020b) and WMT-14 (Barrault et al., 2019). OPUS-100 is a massively multilingual dataset collected from OPUS (Tiedemann, 2012). including 100 languages in total with 99 languages to-and-from English." It consists of 55M training sentence pairs with up to 1M samples per language pair, and covers 94 dev/test language pairs, each with 2000 samples at most. WMTT14 another multilingual English- centric dataset composed of 13 widely-used WMT benchmarks following Siddhant et al. (2020) but excluding Kazakh and Gujarati due to their poor parallel resource. Compared to OPUS-100, WMT- 14 involves much fewer languages but its training data distribution is highly skewed across diverse language pairs, ranging from 0.2M (En-Tr)) to 60M (En-Cs) training examples, thus posing severe challenges. We show more details about the train, dev and test data for WMT-14 in Appendix A.
We apply byte pair encoding (BPE) algorithm (Sennrich et al., 2016) using SentencePiece (Kudo & Richardson, 2018) to preprocess multilingual sentences with vocabulary size of 64K. We use BLEU (Papineni et al., 2002), offered by SacreBLEU (Post, 2018),, for translation evaluation. Fol- lowing Zhang et al. (2020b), we split the 94 test language pairs in OPUS-100 into three groups based on their training data size to ease model evaluation: high resource (>0.9M, 45 languages), low resource (<..IM, 26 languages) and medium resource (others, 28 languages). Similarly, we split the 13 test language pairs in WMT-14 as follows: High (>10M, 5), Low (<IM, 5) and Med (others, 3).
We perform experiments for one-to tany translation (O2M) and many-to-onn translation (M20). In addition to using the original training data as is, we also report results with a temperature-basee strategy to balance the training data distribution by over-sampling low-resource languages with temperature of T (Arivazhagan al., 2019).
We report average BLEU, and also show win ratio (Zhang et al., 2020b, WR), informing the propor- tion of language pairs on which our method beats our baseline. To evaluate how often each sub-layer is LS, we introduce a new metric, LSScore, formulated as follows:
LSScore (I.p)
Lh P. Σ PEP
PP
where Hf, denotes the gating value averaged over all test tokens for the 1-th sub-layer f(-) trained under budget of p. Recall that pose the budget constraint to the summed gate value instead of each individual gate, as in Equation 6. This gives CLSR the freedom to close more gates in some segguaggeinsensstivee sub-layers (i.e. o. < p) while preserve more for the others (i.e. Jf, > p). A larger LSScore, 0 in particular, indicates that this sub-layer utilizes more LS modeling.
d 512, 8 attention heads with FFN middle size of 2048. Dropout of rate 0.1 is applied to residual connections and attention weights. We optimize parameters using Adam (Kingma & Ba, 2015) (31 0.9, 32 0.98) with label smoothing of 0.1. Learning rate is scheduled according to the inverse square root of running steps with a warmup step of 4K (Vaswani et al., 2017). We limit training sequence length
Model Settings We adapt Transformer base for our experiments: L
"htt::/opps.nlpl http://ppps.nppl..uoouuuus-ssss wwwpplssoppuuooocopppppp 100.php
SSinnauur:: BI.FUtcase..iiee mixee- wwssoth.eep-ttokk wtwp:ohh.eppttoo...aaeeeeeeeeeee 3avversion. 1.4.10.
5

---

Published as a conference paper at ICLR 2021
to 100, and train all models with batch size of 1920. We set the maximum running step to 500K and 600K for OPUS-100 and WMTT-14, respectively. We perform beam search decoding with beam size of 4 and length penalty of 0.6. We average the last 5 checkpoints for evaluation. For CLSR, we set dg to 128, and linearly increase a from 0 to along training steps (Bapna et al., 2020). 6 We vary the budget p in the range of p {0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0) to study its impact on model performance.
5.1 RESULTS ON OPUS-100
A vs Wi (Baeei
BLET Ny McMtliged Baeeli
1.0
1.-
133
01-
1.2
1.
07-
-15
+ComongfrCaaa
00
03
0.5
67
09
38 01
8-
0.7
00 10
43
6.
07
06
10
00
01
03
05
47
09
10
Sharad
-
L5
Starrd
budget P
LS
Stared
badgst
-S
Stared
badget
15
BLEU for O2M
(b) BLEU for M20
Win Ratio for O2M
(d) Win Ratio for M20
Figure 2: Average BLEU 2(b).2(b) and win ratio 2(c).2(d) over all test language pairs for O2M and M20 on OPUS-100 100 when varying the budget p. Baseline: multilingual haseline on the original training data; Oversam- ple: oversampling low-resource data with a temperature of 5.
On the trade-off between shared and language-specific capacity. Using more LS modeling fails to deliver increasingly better translation performance, as shown in Figure 2(a) and 2(b) when p approaches 1.0. At the point of full LS capacity for Transformer, i.e. p 1.0, CLSR even un- derperforms its corresponding multilingual baseline by a large margin of 1.0-2.0 BLEU on M20, Figure 2(b). Similarly, sharing all model parameters across language pairs, p + 0.0, also yields suboptimal performance with either original or oversampled training data. CLSR achieves its best translation quality at p = 0.3 (+2.0/3.0 O BLEU) and p = 0.1 (+0.5/1.5 BLEU) for O2M and M20, re- spectively. The win ratio curves in Figure 2(c) and 2(d) further confirm the robustmess of these qual- ity improvements, where properly scheduling LS capacity outperforms the baselines on >~ 80% language pairs. These results clearly show the trade-off between these two kinds of capacity.
When does language-specifie capacity matter for multilingual translation? Results in Figure show that O2M favors more LS modeling and benefits more from it compared to M20 (p 0.3 p = 0.1, and +3.0 vs. +1.5 BLEU on the original training data). We conjecture that translations on M20 share the same target language (English), so information can be easily transferred through shared modeling; by contrast, MNMT has to handle languages of diverse typological features for O2M, demanding stronger capacity delivered to each translation direction. Results in Figure also show that CLSR yields more aggressive improvements on the original training data, compared to the oversampled counterpart (+3.0 vs. +2.0 BLEU on O2M and +1.5 vs. +0.5 BLEU on M20). Fine- grained analysis on each resource group, as shown in Figure (Appendix B), reveals that CL.RR yields more improvements for lew-resuurce translation where the oversampling strategy partially offiset these improvements.
Where should we add language-specific capacity in multilingual Transformer? Figure 2 sug- gests that CLSR uses 10-30% LS capacity to reach its best performance. We next study how CLSR schedules this capacity across all Transformer sub-layers, in order to determine the ideal arrange- ment of LS layers for translation. Figure 3 shows the results. Regardless of layer types, CLSR schedules more LS capacity the top and/or bottom encoder/decoder layers rather than the middle ones. We find that more LS capacity is allocated to feed-forwart sub-layers for O2M (both encoder Figure 3(a) and decoder Figure 3(b), while the cross-attention sub-layers use more LS capacity for M20 (decoder, Figure 3()). Regarding the encoder for M20, we observe no significant LSScore
'Noe that, there are 99 x 2 and 13 x 2 independent lann weight matrices when CLSR is used for OPUS- 100 and WMT-14, respectively. This corresponds to each language having weight matrix in the encoder and the decoder.

---

Published as a conference paper at ICLR 2021
0.5
Day ail SrL pndie
Degid dammon
Oipinl Edl
Deged Jomnice
Oigiil Dhepllen
LSScom
0L1
L.S5ccr
0L1
Drsuejk Feet hoodd
LSScev
1.Seve
0.1
Cratiods Fuo Fuinel
-32-
Oione badd Farved
Layer Depth
Layer Deyts
Lager Depit
Laye: Depth
(a) Encoder, O2M
(b) Decoder, 02M
(c) Encoder, M20
(d) Decoder, M20
LSScore of encoder and decoder sub sub-layers for O2M 3(a), 3(b) and M20 3(c), 3(d) on OPUS- 100. The solid lines correspond to models trained the original data, while the dashed lines the oversampled data. We also include red, dotted line indicate the I.SScore of O.
Figure 3:
Data Setting
O2M
M20
Model
High
Med
Low
WR
High
Med
Low
All
WR
Buseline
21.39
22.36
18.02
20.93
28.55
30.10
29.71
29.27
LS
+0.75
+1.83
+3.96
+1.79
94.68
-0.54
-0.16
-0.46
-0.41
32.98
CLSR-S
+0.06
-0.09
-0.34
-0.08
41.49
-0.14
-0.06
+0.02
-0.08
37.23
Original
CLSR-L
+0.39
+1.54
+4.37
+1.62
84.04
-1.13
-0.47
-0.46
-0.78
21.28
CLSR
+1.45
+2.83
+6.40
+2.97
95.74
+0.65
+1.52
+3.79
+1.61
96,81
Top- Bottom
+1.27
+2.71
+6.60
+2.89
96.81
+0.38
+1.16
+3.06
+1,21
78.72
Dedicated
+1.35
+2.75
+6.46
+2.90
97.87
+0.61
+1.50
+2.85
+1.38
89.36
Baseline
19.95
24.22
25.27
22.41
26.98
30.69
34,26
29.71
LS
+0.99
+1.30
+0.94
+1.07
90.43
-0.55
-0.61
-3.96
-1.33
12.77
CLSR-S
+0.02
+0.13
+0.30
+0.11
44.68
-0.04
+0.06
-0.84
-0.19
35.11
Over Sample
CLSR-L
+0.62
+0.60
+0.34
+0.55
69.15
-1.02
-1.11
-4.56
-1.84
7.45
CLSR
+1.76
+2.04
+1.94
+1.89
93.62
+0.82
+0.84
-0.13
+0.62
76.60
Top-Bottom
+1.73
+1.91
+1.83
+1.81
96.81
+0.83
+1.05
-1.36
+0.41
73.40
Dedicated
+1.79
+2.03
+2.07
+1.92
94.68
+0.99
+0,92
-0.88
+0.55
77.66
Table 1:
Translation quality for O2M and M20 on OPUS-100 with the original and oversampled training data. We list average BLEU for High, Med, Low and All language groups, as well as WRT over all language pairs. Baseline: the vanilla multilingual baseline; L the LS model proposed by Zhang al. (2020b); CLSR-S: CLSR always using shared modeling, i.c. g(z')
for all imputs: CLSR-L: CLSR but always using LS modeling, i.e. s(x') 1 for all inputs; CLSR* the best CLSR model; Top-Bottom: applying LS modeling only the top and bottom Transformer layers: Dedicated: dedicated model that allocates LS modeling based on LSScore distribution. Best results are highlighted in bold.
difference among different sub-layers as in Figure 3(c). Overall, CLSR tends to make the "M" side more LS, i.c. the encoder side of M20 (Figure 3(c)) and the decoder side of O2M (Figure 3(b)).
Does CLSR schedule language-specific capacity based on linguistic similarity? It is intriguing to explore how the capacity scheduled by CLSR is actualized, especially whether CLSR learns to organize LS capacity according to linguistic characteristics or not. The heatmap in Figure 6, Appendix B shows that the LSScore distribution over different sub-layers (y-axis) has only subtle difference across different language pairs (x-axis). This suggests that the capacity schedule has little to do with linguistic characteristics. More results in other settings are given in Figure 6, Appendix B. which reflect similar observation. In short, CLSR allocates LS capacity to specific sub-layers rather than specific languages. This could be ascribed to the design of CLSR. CLSR shares the gating parameters and the budget, p. across all languages, which might impose some inductive bias discouraging its LS behavior. Besides, the structure of the gating in CLSR (Eq. 5) might fail to offer enough flexibility for controlling the gates in different ways across languages and layers. We leave further study of LS gating to future work.
On detailed results and comparison other baselines. Table summarizes our results.? 7 Al- though LS (Zhang et al., 2020b) improves O2M, it fails to surpass the vanilla Baseline on M20, indicating that the position of its LS layer, i.e. on top of the encoder outputs, is sub-optimal for
We list the number of trainable parameters for each model in Table 6, Appendix D.

---

Published as a conference paper at ICLR 2021
many-to-one translation. Compared to LS", CLSR-S uses no LS modeling while CLSR-L injects LS projection into each sub-layer, both of which delivers inferior performance on 02M (against LS and Bascline) and M20 (against Baseline), echoing findings from Figure 2. By contrast, CLSR*, which uses an optimized budget p. yields promising results, beating Baseline on O2M and M20, underlining the importance of seeking balances between sharing and not sharing.
Can we use these insights to improve multilingual Transformer? We answer this question by transferring findings from Figure 3 into the following two Transformer variants: enhances the top and bottom encoder/decoder layers with LS projections (Top-Bottom), while the other makes high-LSScore sub-layers LS (Dedicated). 8 Results of these experiments, in Table demonstrate that incorporating our findings into Transformer helps to improve translation performance. The dedicated model in particular recovers and even surpasses the performance of CLSR*.
5.2
RESULTS ON WMT-14
RLBB ss Milliigud Roatire
100
10
2--
0.5
Demomportt... Donard
3.0
022
1.0
WFR
WR
0.4
0.5
1.0
0-
0.
07-
60
0.8
Stead
18
Stand
Shand
badgo
-15
Stand
1.5
(a) BLEU for O2M
(b) BLEU for M20
(c) Win Ratio for O2M
(d) Win Ratio for M20
Figure 4: Average BLEU 4(b).((b) and win ratio 4(c),4(d) over test language pairs for O2M and M20 on WMT-14 when varying the budget p.
Data Setting
O2M
M20
Model
High
Med
Low
All
WR
High
Med
Low
All
WR
Baseline
27.24
16.27
8.72
17.58
30.64
24.37
18.66
24.58
LS
+0.06
+0.66
+1.68
+0.83
84.62
-0.06
-0.37
-0.92
-0.46
23.08
CLSR-S
+0.00
+0.16
+0.20
+0.12
53.85
+0.04
+0.16
+0.28
+0.17
69.23
CLSR-L
-0.52
+0.43
+2.06
+0.70
61.54
-0.82
-0.50
+0.00
-0.43
15.38
Original
CLSR
+0,46
+1.46
+3.10
+1.71
100.0
+0.12
+0.73
+1.52
+0.80
84.62
Top-Bottom
+0.10
+0.96
+2.78
+1.34
84.62
-0.08
+0.70
+1,26
+0.62
61.54
Dedicated
+0.26
+1.00
+2.98
+1.48
92.31
+0.04
+0.86
+1.90
+0.95
84.62
Baseline
26.10
19.00
15.52
20.39
29.96
26.33
23.56
26.66
LS
+0.30
+0.60
+0.26
+0.36
84.62
-0.44
-0.56
-0.80
-0.61
00.00
CLSR-S
-0.12
+0.03
+0.04
-0.02
46.15
+0.06
+0.04
-0.26
-0.07
46.15
Over Sample
CLSR-L
-0.56
-0.10
-0.32
-0.36
7.69
-0.90
-1.00
-1.24
-1.05
00.00
CLSRR*
+0.50
+0.67
+0.64
+0,59
100.0
+0.26
+0.30
+0.50
+0.36
100.0
Top-Bottom Dedicated
+0.12
+0.40
+0.56
+0.36
84.62
-0.08
+0.00
-0.50
-0.22
30.77
+0.46
+0.57
+0.66
+0.56
100.0
+0.16
+0.17
+0.22
+0.19
84.62
Table 2:
Translation quality for O2M and M20 on WMT-14 with the original and oversampled training data.
Figure 4 shows the capacity trade-off on WMT-14, reconfirming the ability of CLSR. One noticeable difference is that the relative improvements become smaller. We ascribe this to the smaller number of language pairs WMT-14, where the effect of introducing LS capacity into the model is smaller compared the massively multilingual setting (Arivazhagan al., 2019). Table shows similar results as Table 1, where CLSR* outperforms both fully-shared (CLSR-S) and fully-LS (CLSR- L) baselines, and both hop-Bottom and Dedicated improve multilingual translation compared to Baseline (except for M20 with oversampling). More results are given in Appendix C. Our results demonstrate that the CL SR approach generalizes to different datasets, with varying number of lan-
Detail about which sub-layers are specialized in Dedicated is given in Table 4, Appendix
8

---

Published as a conference paper at ICLR 2021
guages and resource sizes. We provide additional experiments ablating different elements on CLSR in Table 7, Appendix E.
6 CONCLUSION AND FUTURE WORK
Share or not? This an open question when developing MNMT models. In this paper, we attempt answer this question by proposing conditional language-specifie routing (CLSR). Our empirical results demonstrate that CLSR learns to balance between shared and LS paths across all NMT sub- layers, improving the quality of multilingual translation. Our analysis OPUS- 100 and WMT- 14 suggest that both the position and the amount of LS capacity greatly affects MNMT. Scheduling 10%-30% LS layers to the top and/or bottom encoder/decoder layers reaches CLSR's s peak perfor- mance. We also demonstrate how our findings can be leveraged to design multilingual Transformer with an optimal sharing pattern. We believe that our work improves our understanding on the trade- off between sharing and not sharing, paving the way for better multilingual models.
In the future, we plan extend our study to many-to-many translation as well as larger-capacity models. We also plan to adapt CLSR to other multilingual multi-task learning settings to better handle knowledge transfer among different tasks, especially for cross-lingual downstream transfer.
ACKNOWLEDGEMENTS
We would like to thank Yuan Cao for his valuable feedback. We would also like to thank the Google Translate team for their constructive discussions and comments. We thank the reviewers for their insightful comments. Rico Sennrich has received funding from the Swiss National Science Foundation (project 176727)
REFERENCES
Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine transla- tion. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 3874-3884, Minneapolis, Minnesota, June 2019, Association for Computational Lin- guisties, doi: 10.1863//1/N19-133888 URL https:/ http://wwwwwwww.../.p...oo org wwwpggggggnttottttooooooo N19-1388.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively multilingual neural machine translation the wild: Findings and chal- lenges. CoRR, abs/ 00009, 2019. URL http: arxiiiiiivv org/ abs/ / 1907. 05019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607::6007006450 2016.
Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In Pro- ceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
pp- 5538-1548. Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1/D19-11661
URL https:///// www. ac web. org anthologyy /D1 111-111111111 1165.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Controlling computation versus quality for neural sequence models. arXiv preprint arXi::2002.07106 2020.
Loic Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Kochn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 con- ference on machine translation (WMT19). In Proceedings of the Fourth Conference on Ma- chine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, Au- URL
gust 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-552 https: www. wwwl web. org antotoooolooooo ogyy ogy///////oooo / 9-5301

---

Published as a conference paper at ICLR 2021
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:130..3433 2013.
Graeme Blackwood, Miguel Ballesteros, and Todd Ward. Multilingual neural machine translation with task-specific attention. In Proceedings of the 27th International Conference Computa- tional Linguistics, pp. 3112-3122, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https: wwwwww.ww................mo..... aclweb. org/ httoghoooooooocccoo 00tp://111111.3.....
Chung-Chon Boing-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. In International Confer- ence on Learning Representations, 2018. URL Hko85p1CW.
net fwrmm?ormmmaa ?id=
Andrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks, 2013.
Mostafa Dehghami, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https: / //openreview.. net./ forum? d=HyzdRiRgyy
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for mul- tiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguisties and the 7th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pp. 1723 1732, Beijing, China, July 2015. Association for Computational Linguisties. doi: 10.3115/v1/P15 enthologgyPP15115
URL https: / /ww. acwweb. org/
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In In- ternational Conference on Learning Representations, 2020. URL http:::tt////tt.tt.tt /tttps://opereeview
forum? 7KKVKK...
Carlos Escolano, Marta R Costa-jussă, José AR Fonollosa, and Mikel Artetxe. Multilingual ma- chine translation: Closing the gap between shared and language-specific encoder-decoders. arXiv preprint arXiv:2004.06575 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations 2020. URL net httt://oommmddyy11yyeee
petts::/oppezeeee
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine trans- lation with shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 866-875, San Diego, California, June 2016a. Association for Computational Lin- guistics. doi: 10.18653/1/N11--11011 URL https:/ wwww acl web.. wrg.anthooogggy////ooo N16-1101.
Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. Zero-resource translation with multi-lingua neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 268-277, Austin, Texas, November 2016b. Association for Computational Linguistics. doi: 10.18653/ 11/D16-1026 URL. https::/t///t/...t. wtwp///wwww/../////////////tt ac web www.wwe..roo.................... htt..///anooooogg/iii......
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint
arXiv:1603.08988 2016.
Thanh-Le Ha, Jan Niehues, and Alexander Waibel. Toward multilingual neural machine translation with universal encoder and decoder. Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT), Seattle, USA, December 8-9 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, wttp/////..e.. 513.03385, 2015. URL http: / org altp//bbs/1111 5121. 0.3.85.
10

---

Published as a conference paper at ICLR 2021
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pp- 103-112, 2019.
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas. Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339-351, 2017. doi: 10.1162/tacl a 00060.0005.0 URL hitps: www. aclweb. org/ anthology/ (ttpy//117
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Con- ference on Empirical Methods Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18- 2012. URL hwtpt:ttp//.mmm www. acl web aewebborg. wwwg/aattglogggooooooooooooooo//
2012.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine trans- lation without explicit segmentation. Transactions of the Association for Computational Lin- guistics, 5:365-378, 2017, doi: 10.1162/tac_a_000 a 00066.. URL https: /tt:://wwwwwt www..aclweb..... org/
anthology/ 1026.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16662 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL htttps:://pppeeeeessss
net/ forum? id=$1
Tom M Mitchell. The need for biases in learning generalizations, Department of Computer Science, Laboratory for Computer Science Research 1980.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- fion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguisties. doi: 10.3 10.3115/1073083.0033333 URL https:
//www. ac wwebb org/ ant http//anhoooog//22 1040.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. Contex- tual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 425-435, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18- 1039. URL https::ttt......m...t. / www. ac web. org/ antho oog 2ttp://11111111 1039.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https://p/// /www acl www.ww..w....................... waaorg///aoolllooo wontholoogo ogy/ W1 631 9.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task architecture learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 4822-4829, Jul. 2019. doi: 10. http://aaaiv3311.33333131100 index. http::/p/////attt//eeeeeeeeee/ee
Devendra Sachan and Graham Neubig. Parameter sharing methods for multilingual self-attentional translation models. Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 261-271, Brussels, Belgium, October 2018. Association for Computational Lin- guistics. doi: 10.18653w1/W118-9977 URL https: //www. aclweb. org hwtpp/gtggg/aooooooooeeoooo honooog W1 W18-6327.
11

---

Published as a conference paper at ICLR 2021
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.11665/11//11-11666 URL ht.ts::// www. ac web. org/anthologyy P-
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of- layer.
arXiv preprint arXiv: 70653888 2017.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Ari- vazhagan, and Yonghui Wu. Leveraging monolingual data with self-supervision for multilin- gual neural machine translation. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 2827-2835, Online, July 2020. Association for Computa- tional Linguistics. doi: 10. 10.18633/v1/2000 acl-main.252. URL httts:/: /wwww ac web. org/ anthology/ 2020 acl maacmmmminn...... 252.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguisties, pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. (0.18663/v1//19 1032. URL https: /wwwwwww....../.. acl web. org/ pergganntoloog/// P19-1032.
Xu Tan, Jiale Chen, Di Hc, Yingce Xia, Tao QIN, and Tin-Yan Liu. Multilingual neural ma- chine translation with language clustering. In Proceedings of the 2019 Conference Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 963-973, Hong Kong, China, Novem- ber 2019, Association for Computational Linguistics. doi: 10.18653/1//119-008)) URL https:/ http:://wwww.ttt http:/wwwcllww.../.. org/ wwwpg//nthologyyy/oo lot.ol/gg/DD111..-
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Nicoletta Calzolari (Conference Chair), Khalid Choukri. Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mar- iani, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12) 12), Istanbul, Turkey, may 2012. European Lan- guage Resources Association (ELRA). ISBN 978-2-9517408-7-77
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL http:// /tap:r//ppprrs....
attettttnnniinti aawooonni-allllllll -ll--al----ueeddeee--e pdf.
Raúl Vázquez, Alessandro Raganato, Jörg Tiedemann, and Mathias Creutz. Multilingual NMT with enguaageindependen attention bridge. In Proceedings of the 4th Workshop on Representa- fion Learning for NLP (RepLANLP-2019) pp. 33-39, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/W1194443 URL https:
acl web.
org/anthelogy / /W19 Www-W19--33 4305.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural ma- chine translation. arXiv preprini arXiv:2004.067444 2020a.
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. Three strategies to improve one-to-many multilingual translation. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 1955-2960, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DI8. 1326. URL https: /wwp://wwwww///// acl web hwtgg//athllllo ogyoogg////oo - 1326.
Yining Wang, Long Zhou, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. A compact and Language-sensitiie multilingual translation method. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1213-1223, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/P199 11111119991111.11111111111111111 URL https:/ / www.wwww.mmmmm////oo acl web.
web..rgg/eee orgg wttgg/nntology///e /P 9117.
12

---

Published as a conference paper at ICLR 2021
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual mod- els: Findings and a meta-learnng treatment. Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 4438 -458, Online, Novem- ber 2020b. Association for Computational Linguistics. doi: 10065///122222.mmmmmmmmiiii URL https::////.......t.m /wwwwwwww... ac web www.g//nthooooogoooooooooooooooo ogy emn 1 p-ma in ..59.
Biao Zhang, Ivan Titov, and Rico Senmrich. On sparsifying encoder outputs in -ooeeneee---sueecee models. arXiv preprint arXiv:2004.11854 2020a.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 1628-1639, Online, July 2020b. Association for Computational Linguistics. doi: 1..18653/11/002000 act- wac.:/accllma.eae... 148. URL https: /www.
ac web. erg/ antho hotplllgyy//// acl mm n 148.
Changfeng Zhu, Heng Yu, Shanbo Cheng, and Weihua Luo. Language-aware interlingua for mul- tilingual neural machine translation. In Proceedings of the 58th Anmual Meeting of the Associa- tion for Computational Linguistics, pp. 1650-1655, Online, July 2020. Association for Computa- tional Linguistics. doi:
URL https: / /wwww ac web. org/
anthology gallooyyy220022 2020 acl-mainn 150.
Barret Zoph and Kevin Knight. Multi-source neural translation. In Proceedings of the 2016 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 30-34, San Diego, California, June 2016. Association for Com- putational Linguistics. doi: 10.18653/v1/N16 1004. URL https: /wwww//wwww//w////////////////// ac web org/
mothology/N16-100
13

---

Published as a conference paper at ICLR 2021
A DATASET DETAILS
Table 3: Statistics of train, dev and test data for WMT-14.
Language
Data Sources
Samples
Pair
Train
Dev
Test
Train
Dev
Test
En-Cs
WMT19
WMT17
WMT18
64336053
3005
2983
En-Fr
WMT15
WMT13
WMT14
40449146
3000
3003
En-Ru
WMT19
WMT18
WMT19
38492126
3000
2000
En-Zh
WMT19
WMT18
WMT19
25986436
3981
2000
En-Es
WMT13
WMT13
WMT13
15182374
3004
3000
En-Fi
WMT19
WMT18
WMT19
6587448
3000
1996
En-De
WMT14
WMT13
WMT14
4508785
3000
3003
En-Et
WMT18
WMT18
WMT18
2175873
2000
2000
En-Ly
WMT17
WMT17
WMT17
637599
2003
2001
En-Lt
WMT19
WMT19
WMT19
635146
2000
1000
En-Ro
WMT16
WMT16
WMT16
610320
1999
1999
En-Hi
WMT14
WMT14
WMT14
313748
520
2507
En-Tr
WMT18
WMT17
WMT18
205756
3007
3000
Figure 5: Training data distribution over language pairs for OPUS- 100 and WMT-14.
Data Distribution in OPUS-100
Data Distribution in WMT-14
0.8
# instances
# instunces
0.6
High
Low resourec
High
Low
Table 3 lists the statistics for WMT-14. We collect these datasets following Siddhant et al. (2020). Figure shows the training data distribution. Training data in WMT-14 is more imbalanced across different language pairs compared OPUS-100.
B
MORE RESULTS ON OPUS-100
Table Selected sub-layers by Dedicated on OPUS-100. SAN/CAN: seffeeosnniinnnosssttttttinoonno FFN: feed-forward.
Encoder
Decoder
Data Setting
Bottom
Top
Bottom
Top
SAN
FFN
SAN
FFN
SAN
CAN
FFN
SAN
CAN
FFN
Original, 02M Original, M20
Oversample, 02M Oversample, M20
/
/ /
/
Figure 7 shows the translation performance of CLSR on each resource group when varying the budget p. We find that LS modeling greatly improves translation low-resource settings, although
14

---

Published as a conference paper at ICLR 2021
Figure 6: Heatmap of LSScore distribution OPUS- 100 trained w/ and w/o oversampling. X-axis denotes language pairs ranked by training data size, and y-axis denotes encoder (enc) decoder (dec) sub-layers with a format of "enc/dec.laye types.layer index". Darker color indicates a larger LSScore.
Language Patr
Traning Chats
Oversample, M20
09 00000000
(b) Oversample, O2M
Patrs Bonked
Trainin Dart Soe
(c) Original, 02M
01 100 10
Language Palis Runked by Thining Data Size
(d) Original, M20
this improvement is partially offset by the oversampling strategy. We observe that the success of oversampling is built on top of sacrificing quality on high-resource languages, where CLSR largely narrows even closes the performance gap against the multilingual baseline trained the original
15

---

Published as a conference paper at ICLR 2021
Figure 7: Impact of the budget p average BLEU for Highh/Me///Low-esoorrs languages OPUS-100.
High Languages, 02M
Med Languages, 02M
Low Langunges, O2M
A Moliiinuul Daseline
BL EU Moltstrarual Bosline
3-
Muisilngnual
0.5
0.0
2-
+
CLSR
Dersceple
-0.5
DrexuppkoLLS
1-
-1.0
Shared
budget P
1S
Shared C-
badget P
-
LS
Shared
budget
(a) BLEU for O2M
High Languages, M20
Med Languages, M20
Low Languages, M20
A ABEEU vs Munitingull Hoeelie
BLEEU vs Multlingual Basellie
Mot.tlllgggge oguua Baselie
Bustie -C.3
-Oanannde
0.0
-0.5
0.0
0.1
0.5
0.7
1.0
0.0
0.1
0.3
0.5
0.7
0.9
0.1
03
0.5
0.7
0.9
1.0
Shared
budget
Shared
Shared
budger
(b) BLEU for M20
data. In addition, we find no significant difference in term of the trade-off curve across resource groups compared to Figure 2.
Figure 6 shows the heatmaps for LSScore distribution. Regardless of translation directions (02M
or M20) and data settings (Original or Oversample), there is no clear linguistic pattern behind these heatmaps. The learning of CLSR is not linguistic driven.
C MORE RESULTS ON WMT-14
Table 5: Selected sub-layers by Dedicated on WMT-14.
Encoder
Decoder
Data
Setting
Bottom
Top
Bottom
Top
SAN
FFN
SAN
FFN
SAN
CAN
FFN
SAN
CAN
FFN
Original, 02M
d
J
Original, M20
/
/
V
V
/
Oversample, 02M Oversample, M20
/
V
/
V
/
/
/
We show more results WMT-14 here. Both heatmaps in Figure 9 and fine- tine-grained trade-off curves in Figure show very similar story to the one on OPUS-10P In terms of LSScore distribution in Figure 10(d), also have similar observation except that the cross-attention sub-layer in the decoder on M20 shows no clear LS preference.
These results suggest the high generalization of CLSR different data settings.
16

---

Published as a conference paper at ICLR 2021
Figure 8: Impact of the budget p on average test BLEU for HighhMeed/Low-resouee languages WMT-14, 14.
High Langunges, 02M
Med Languages, 02M
Law Languages, O2M
A Molsiimuua Daseline
Molth Irigunal Hasetine
80 Iolgill Baxcline
0.0
CL.S
-1.0
1.5
Shared 6-
badget
Shared
badget P
-
LS
Shared
budger P L.
(a) BLEU for O2M
High Languages, M20
Med Languages. M20
Low Languages. M20
BBEEU Multitingaal Baselin
BLEUU vs Muthilingual Baseline
BI EU vs Multitingua Raseline
2.0
1.5
Secti:: CLSR
1.0
Drasungle
0.5
DecnomplocC..t
-1.0-
0.0
-1.5
-0.5%
0.0
0.1
0.5
6.7
0.9
0.1
0.3
0.5
0.7
0.9
1.0
0.0
0.1
0.3
0.5
0.7
0.
Shared
Shardd
badger
Shared
(b) BLEU for M20
Figure 9: Heatmap of LSScore distribution on WMT-14 14.
Laguge Pain Romkrad Yootiin Soo
Lagoge Pain Ramked Teatnin Soo
Oversample, 02M
(b) Oversample, M20
Laggug Toining
Languge Thiining
(c) Original, 02M
(d) Original, M2O
17

---

Published as a conference paper at ICLR 2021
Depid fcd dameo
Ordia 64
Ternupp:
Deged Joneic
LSScom
0L1
L.Scccr
0LE
Dauujk Ind heodd
LSScov
0.2
L.Seeve
0.1
-37
Layer Depth
Layer Deyts
Lager Deptt
Laye: Depth
(a) Encoder, O2M
(b) Decoder, 02M
(c) Encoder, M20
(d) Decoder, M20
Figure 10: LSScore of encoder and decoder sub-layers for 02M 10(a), 10(b) and M20 10(c). 10(d) on WMT- 14.
Table 6: Number of parameters for cach model on OPUS-10K 100 and WMT- 14.
OPUS-100
100
WMT-14
Model
02M
M20
O2M
M20
Baseline
99M
106M
LS
129M
111M
CLSR-S
102M
109M
CLSR-L
148M
111M
CLSR
157M
121M
Top- Hop-Bottom
153M
116M
Dedicated (Original)
153M
154M
117M
117M
Dedicated (Oversample)
154M
155M
117M
117M
D MODEL. PARAMETERS
We list model parameters in Table 6. Note that the source/target vocabulary for O2M is the same as the target/source vocabulary for M20, thus models for O2M and M20 have the same number of trainable parameters except for Dedicated where the network structure differs.
E ABLATION STUDY
We provide variants of CLSR for ablation. The first one examines whether CLSR could induce an optimal budget automatically by eschewing the budget constraint in Eq. (CLSR w/o p); the second one further replaces the hard gating network with a vanilla soft gating network CLtRR
Table 7 summarizes the translation results. Without the regularization term, CLSR w/o p yields an average gating value of 0.37/0.16 and 0.39/0.19 19 for O2M/M20 on OPUS-100 and WMT-14, re- spectively", resonating with our finding that O2M requires more LS capacity compared to M20. Compared to CLSR* CLSR w/o p produces more open gates, achieving comparable but overall worse performance in all data settings. This highlights the importance of the added regularization term, where without such regularization the model is likely spending extra learning cycles to auto- matically optimize for the budget as with CLSR w/o p. This in return hints at the importance of optimization routines (learning) which we held constant in this study.
By contrast, switching to soft gating network, CLSR-Gate gains translation quality in most settings, although the quality gains are mostly less than 0.1 compared to CLSR* Note that, CLSRR-ate relies on continuous gates, indicating much higher flexibility in balancing parameter sharing patterns. However, we also notice that CLSR-Gas performs worse on M20, even fails to beat the vanilla Baseline (All) the oversampled WMT--14. Unlike CLSR-Gate, CLSR uses hard binary gates which eases capacity analysis and achieves consistent improvements across data settings.
We also average the gating value over the original and oversampled data settings.
18

---

Published as a conference paper at ICLR 2021
Data Setting
O2M
M20
Model
#Param
High
Med
Low
All
WR
High
Med
Low
All
WR
OPUS-100
Baseline CLSR
99M
21.39
22.36
18.02
20.93
28.55
30.10
29.71
29.27
Original
157M
+1.45
+2.83
+6.40
+2.97
95.74
40.65
+1.52
+3.79
+1.61
96.81
CLSR wlo p
157M
+1.37
+2.77
+6.28
+2.88
96.81
+0.53
+1.27
+3.17
+1.34
90.43
CL.SR-Gate
157M
+1.55
+3.13
97.87
+2.87
+1.48
93.62
Bascline CLSR
99M
19.95
24.22
25.27
22.41
26.98
30.69
34.26
29.71
Over Sample
157M
+1.76
+2.04
+1.94
+1.89
93.62
+0.82
+0.84
-0.13
+0.62
76.60
CLSR w/o
157M
+1.72
+1.92
+2.06
+1.86
97.87
40.73
+0.56
-1.10
+0.27
71.28
CLSR-Gate
157M
+2.00
+2.24
+2.35
+2.15
97.87
+1.12
+1.28
-1.04
+0.68
79.79
WMT-14
Baseline
106M
27.24
16.27
8.72
17.58
64
24.37
18.66
24.58
Original
CLSR*
121M
+0.46
+1.46
+3.10
+1.71
100.0
+0.12
+0.73
+1.52
+0.80
84.62
CLSR w/o
121M
+0.32
+1.23
+3.00
+1.57
100.0
-0.04
+0.56
+1.72
+0.78
76.92
CL.RR-Gate
121M
+0.46
+1.43
+3.14
+1.72
100.0
+0.02
+0.73
+1.68
+0.83
84.62
Baseline
106M
26.10
19.00
15.52
20.39
29.96
26.33
23.56
26.66
Over Sample
CLSR*
121M
+0.50
+0.67
+0.64
+0.59
100.0
+0.26
+0.30)
+0.50
+0.36
100.0
CLSR w/o p
121M
+0.48
+0.63
+0.66
+0.59
100.0
+0.02
+0.24
-0.12
+0.02
38.46
CLSR-Gate
121M
+0.52
+0.77
+0.78
+0.68
100.0
+0.14
+0.30
-0.64
-0.12
61.54
Table 7: Translation quality for O2M and M20 on OPUS-100 and WMT-14 with the original and oversampled training data. CLSR--Gate CL.SR with g(z') normal soft gating network; CT.SR w/o p: CLSR trained without budget constraint.
19


Reviews:
[{"review_id": "zvfRbxLUcGq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors present a conditional language-specific routing (CLSR) scheme for transformer-based multilingual NMT systems. They introduce a CLSR layer after every transformer encoder and decoder layer; each such layer is made up of hard gating functions conditioned on token representations that will either select a language-specific projection layer or a shared projection layer. Further, a budget is imposed on the language-specific capacity measured by aggregating the number of gates that allow for language-specific computations; this budget constraint forces the network to identify the sub-layers that will benefit most from being language-specific.\n\nThis is nice work. The proposed technique has been described clearly, the idea is intuitive and the experiments are pretty compelling. I have a couple of minor comments/suggestions for the authors.\n\n* The authors show heat-maps of LSScore distribution in Figure 6 (Appendix B) which suggest that the LS capacity schedule might have little to do with linguistic characteristics. However, this might have to do with the multilingual model being trained on as many as 94 different languages. It seems plausible that linguistic similarities might govern LS capacity scheduling when there are fewer training languages to learn from. To check for this, it might be interesting to redo this experiment with the medium resource and low resource buckets containing 26-28 languages each.\n\n* There are two (among many other) interesting things that stand out from the results in Tables 1 and 2. (1) From Table 1, the only setting where CLSR* (as well as \"Top-Bottom\" and \"Dedicated\") underperforms compared to the baseline is M2O for low-resource languages. It seems like the use of language-specific layers here has a strong adverse effect on performance (-4.56 with CLSR-L) which is largely offset by CLSR*. Some more insights based on the individual BLEU scores for each test language in the \"Low\" bin and whether there were certain languages that were largely responsible for the drop in performance would be interesting to the reader. (2) From M2O in Table 2, the win ratios of Top-Bottom are much lower when compared with Dedicated and CLSR* (61.54 vs. 84.62 vs. 84.62; 30.77 vs. 84.62 vs. 100).  Could the authors share their thoughts on why this drop might be appearing?", "extra_scores": {}, "extra_text": {}}, {"review_id": "l_Gg-p4Mt2r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work proposes a hybrid architecture that has: (1) language-specific (LS) components; (2) as well as the components that are shared across all the languages -- a trade-off between specificity and generality.  A key conclusion of the work is that the best architectures typically are. the ones that have ~10-30% language-specific capacity. \n\nIn terms of experimental work, the work uses WMT-14 and OPUS-100 datasets to show the proposed trade-off. \n\nIn terms of exposition of the ideas, it's a well-written paper for the most part. \n\nOne issue that the authors could improve on is clarifying how \"the amount of LS computation\" is measured. You have mentioned it several times in the abstract/intro and it's neither clear nor referenced (it could be the number of parameters, it could be the number of basic computations, etc). For a new reader, it takes quite a while to find that $p$ is defined in eq. 6 and defined as a budget contains. \n\nOne other quibble is that all the trade-off figures are shown based BLEU/automatic metrics, which are known to be inaccurate. It would be nice to repeat one of the included evaluation with human judgments. \n\nOverall, I view this as a good contribution to pave the way towards stronger, but reasonably-sized multilingual models. This is partially assuming that the authors will stay true to their promise that \"Source code and models will be released.\"\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lYGH7w5rKp_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a study of different aspects of language-specific model capacity for massively multilingual machine translation. To this end, language-specific behaviour is achieved via a combination of conditional computation to decide whether to use language-specific parameters or not and statically assigning experts for each languages. The language specific sub-layers are incorporated throughout the network. The training objective allow budgetary constraints on the amount of language-specific parameters. The paper does a systematic analysis on the role of language specific parameters using the proposed architecture. Based on the analysis, recommendations on design of multilingual NMT architectures are proposed and their efficacy validated experimentally. The study sheds light on the amount of language specific parameter sharing, their distribution in the network, impact of language, etc. \n\nI find the analysis presented in the paper very interesting and insightful - and distinguishes it from previous work in this area. The hypothesis are clearly stated and the experiments are well designed. The findings from the analysis are an important addition to the understanding of the role of language specific parameters in multilingual NMT. \n\nIn terms of modelling, the work follows in the line of recent work on language-specific parameters for multilingual NMT. The deviation from existing work is mixing elements of conditional computation with language specific computation. I see this work more as an analysis on language-specific parameters for a particular LS-model rather than a novel architecture. It is not clear how this model would compare to other models using language specific parameters (sparsely gated mixture of experts (Lepikhin et al 2020), light-weight adapters (Bapna et al 2019)  ).\n\n\nQuestions: \n\n- Previous work has tried to combine both language-specific and shared parameters (Wang et al 2018), rather than making a binary choice between these. Did the authors compare with such an approach? \n- Since a major part of the model contains shared parameters, was there a need for new set of shared parameters along with the language-specific parameters. The gating decision could have been to bypass the language-specific sublayer or not.\n\nReferences\nYining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. Three strategies to improve one-to-many multilingual translation. EMNLP. 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UUMF0tx7frV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Manual parameter sharing schemes are generally costly to come up with and when they are obtained for certain language pairs they do not necessarily generalize well to arbitrary language pairs in multilingual NMT. The idea of learning which parameters to share across languages in multilingual transformer models is original and potentially useful for designing and analyzing multilingual models in the context of NMT.  \n\n**Strengths**:\n\nThe paper is well-written and easy to follow. The idea was (reasonably) well-positioned with respect to prior work and clearly presented. \n\nThe technical merit is essentially in coming up with the budget constraint term in the loss function that forces the multilingual encoder-decoder \"super-network\" to use the desired percentage of language-specific computation using gating. \n\nA significant part of the contribution was in the analysis of the results, obtained by this learning-based parameter sharing approach, which was quite informative and revealed some interesting insights about where and when a language-specific computation is required. The takeaways should be of interest to researchers and practitioners interested in designing and analyzing multilingual NMT systems. \n\n**Weaknesses**: \n\n(1) Even though it is the first time such a method is applied in the context of NMT, the idea is not as much novel in the broader context of deep learning. Prior work has explored \"learning-to-share\"  strategies for parameter sharing in multi-task learning (see Ruder et al., AAAI 2018), and using gating/masking to control computational paths in a differentiable way (see Fan et al., ICLR 2019, Sukhbaatar et al., ACL 2019); it is clear that the focus is NMT but it should be worth mentioning/discussing such studies to better situate the work and to help the reader assess the actual contributions. \n\n(2) Another weakness is that the comparison with the vanilla and LS baselines does not seem to be properly controlled in terms of parameters. I appreciate that the authors do not read too much into it and focus more on the analysis of the results, but one thing that remains unanswered in this paper is how the proposed method fairs against multilingual baselines that utilize (roughly) the same number of parameters; currently, the best models outperform the LS baseline by ~28M and ~10M parameters on OPUS-100 and WMT-14 respectively. How important is this difference?\n\n(3) In the experiment about linguistic similarity, it appears that the capacity schedule is the same across languages and the authors conclude from this that the schedule has little to do with linguistic characteristics. However, the main driving force in the choice of the language-specific computation is currently a single hyper-parameter p which is the same across languages; so, this will lead to choices that are good on average for all language pairs involved for a given *universal* budget. Do you think the conclusion would be still the same if a language-specific hyper-parameter p_l was used instead? \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a conditional language-specific routing (CLSR)  mechanism for multilingual NMT, which also considers the trade-off between language specificity and generality.

All of the reviewers think this paper is interesting for both idea and empirical findings. Therefore, it is a clear acceptance.
Decision: Accept (Oral)