mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations
Authors: Jonas Pfeiffer
Francesco Piccinno
Massimo Nicosia
Xinyi Wang
Machel Reid
Sebastian Ruder
Venue: EMNLP 2023 Findings (2023)
Keywords: Modularity
Multilinguality
Adapters
Parameter-efficiency

Abstract: Multilingual sequence-to-sequence models perform poorly with increased language coverage and fail to consistently generate text in the correct target language in few-shot settings. To address these challenges, we propose mmT5, a modular multilingual sequence-to-sequence model. mmT5 utilizes language-specific modules during pre-training, which disentangle language-specific information from language-agnostic information. We identify representation drift during fine-tuning as a key limitation of modular generative models and develop strategies that enable effective zero-shot transfer. Our model outperforms mT5 at the same parameter sizes by a large margin on representative natural language understanding and generation tasks in 40+ languages. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%, thereby greatly alleviating the source language hallucination problem.

mm:5:: Modular Multilingual Tre-nirini Solves Source Language Hallucinations
Jonas Pfeiffer
Francesco Piccinno
Massimo Nicosia
Xinyi Wang
Machel Reid
Sebastian Ruder
Google DeepMind
Abstract
Multilingual sequence-to-seqqeece models perform poorly with increased language coverage and fail consistently generate text in the correct target language in few-shot settings. address these challenges, we propose mmT5, modular multilingual sequence-to-seqqence model. mmT5 utilizes sngeuage-speciffee modules during pre- tre-training, which disentangle language-xpecific informa- tion from imguage-agnosttc information. We identify representation drift during fine-tuning key limitation of modular generative models and develop strategies that enable effective zero-shot transfer. Our model outperforms mT5 at the same parameter sizes by large margin on representative natural language understanding and generation tasks in 40+ languages. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%, thereby greatly alleviating the source language hallucination problem.
UP
FF Up
Norn
FFUB
Fiedd Forward
Mulli-tead Atlention
Food
Forward
Norm
Masked Muli-
Mull- Hell-Head
Figure 1: Architecture of mmT5. Language-speciingg bottleneck modules (dark blue and green components) are placed after the feed forward component within each layer of the Transformer encoder-decoder model.
Natural language generation (NLG) tasks present another challenge for current multilingual models, which may overfit to the training languages and partially forget their generation ability in the target language (Vu et al., 2022), generating text with the correct meaning in the wrong language. We refer this as the "source language halluci- nation problem".
1 Introduction
Multilingual pre-trained models (Conneau et al., 2020a; Xue et al., 2021) have demonstrated im- pressive performance on natural language under- standing (NLU) tasks across different languages (Hu et al., 2020; Ruder al., 2021). These mod- els are typically trained on large amounts of unla- beled data in hundreds of languages. Recent large language models (Brown et al., 2020; Chowdhery et al., 2023) display surprising multilingual capa- bilities despite being pre-trained predominantly on English data. However, all of these models share a key limitation: representations of all languages compete for the model's limited capacity. As result, models perform poorly with an increasing number of pre-training languages and on languages with less pre-training data. This is also known as the "curse of multilinguality" (Conneau et al., 2020a).
To address these two limitations, we propose the modular multilingual T5 (mmT5, Figure 1), the first modular multilingual generative model. Dur- ing pre-training, mmT5 allocates small amount of language-speeiff parameters to increase capacity for multilingual modeling. At fine-tuning time, we freeze the language-speciff modules while tuning the shared parameters, allowing direct adaptation to target language by swapping to the corresponding language-specitn module.
However, we observe an additional challenge for mmT5: the fine-tuned shared representations

---

may drift away from the frozen modular represen- tations in the decoder. The modular model is thus susceptible to generating text in the incorrect lan- guage, similar to its non-modular counterparts. To ameliorate this, we propose to freeze a subset of shared decoder parameters, which shows large im- provements in zero-shot cross-lingual generation for modular generative models.
et al., 2017), which have been used scale mod- els trillion parameters (Fedus et al., 2022) and for domain-specific pre-training of LMs (Gururan- gan et al., 2022). The most related work to ours is X-Mod (Pfeiffer et al., 2022), which pre-trains an encoder-only BERT-style model in a modular fashion. Their model, however, cannot be used for natural language generation and underperforms our model NLU tasks (see Section 4).
In general, we find that mmT5 is an effective model that overcomes the two limitations of multi- lingual sequence-to-sequee leviates the curse of multilinguality by adding addi- tional model capacity to different languages during pre-training. It outperforms both standard baselines well as m55 (Xue al., 2021) at the same pa- rameter sizes on representative set of multilingual NLU and NLG tasks; 2) mmT5 resolves the source language ballucination problem with impressive ability on zero-shot cross-lingual text generation. Our analysis ($6.4) shows that mT5 only generates text in the target language 7% of the time for a zero-shot multilinguul summarization task, while mmT5 generates text in the correct language for 99% of examples.
Limitations of multilingual language models State-of-the-arr multilingual LMs are pre-trained on large amounts of multilingual data in around 100 languages. Prior work has demonstrated, however, that models' performance deteriorates with increas- ing language coverage given the same fixed capac- ity, known the curse of mulrilinguality (Conneau et al., 2020b). Prior studies also found that mod- els perform poorly on languages that are under- represented in pre-training (Wu and Dredze, 2020; Hu et al., 2020: Lauscher et al., 2020b; Artetxe et al., 2020; Pfeiffer et al., 2020, 2021b; Chau et al., 2020; Ponti et al., 2020). For natural language gen- eration, multilingual models have been observed to overfit to the source language and fail to generate text consistently in the correct target language (Vu et al., 2022).
models: 1) mmT5 al-
2 Related work
Modular language models Much work has fo- cused on post-hoc modularity of pre-trained mul- tilingual models, i.e., modular representations are added to existing dense models. The most com- monly used modules are known as adapters (Re- buffi et al., 2017, 2018; Houlsby et al., 2019). They enable specialization to new data settings (Chen et al., 2019; Rücklé et al., 2020), combination of new and existing knowledge (Stickland and Mur- ray, 2019; Wang et al., 2021a; Pfeiffer et al., 2021a; Lauscher et al.. 2020a; Mahabadi et al., 2021b; Poth et al., 2021), and adaptation to new cross- lingual (Pfeiffer et al., 2020, 2021b; Üstün et al., 2020; Vidoni et al., 2020; Ansell al., 2021a, 2022; Wang al., 2021b) and NMT scenarios (Bapna and Firat, 2019; Philip al., 2020; Chronopoulou et al., 2020; Le et al., 2021; Üstün et al., 2021; Stickland et al., 2021; Garcia et al., 2021; Dua et al., 2022; Zhang et al., 2021; Pires et al., 2023).
mmT5
Standard multilinuual models update the same model parameters for hundreds of languages dur- ing pre-training, resulting in the curse of multi- linguality where different languages compete for the limited model capacity (Conneau et al., 2020a). We propose mm75, the first modular sequence- to-sequence multilingual model that allocates lan- guage specific modules during pre-training In this section, we discuss the architecture of mmT5, its training and fine-tuning methods, and our strate- gies to resolve the source language hallucination problem with mmT5.
3.1 Modeling
First, we describe the overall architecture of mmT5. We augment a standard Transformer encoder- decoder model with language-specific modules at every transformer layer (see Figure 1). The selec- tion of modules (i.e., fixed routing; Pfeiffer et al., 2023) is performed via the language ID provided with each example'; all tokens of an example are
Our approach, in contrast, uses modularity pri- ori, i.e., modularity integrated into the module architecture as an inductive bias. Such modularity is similar to parameter sharing strategies commonly defined in mulli-task learning (Ruder, 2017) as well as to mixture-of experts approaches (MoE; Shazeer
'Our pre-training data contains such metadata; alterna- tively, automatic language ID methods can be used (see $6.4).

---

passed through the same language-specific module. We use bottleneck adapters as the language- specific module because they perform better at smaller model sizes compared to other modular methods such as continuous prompts (Mahabadi al., 202 la; He et al., 2022). We place module after the feed-forward component in each layer. In contrast to Pfeiffer et al. (2022) that only experi- mented with encoder-only models, we focus on a more general sequuncce-to-sqquennc model follow- ing the T3 architecture (Raffel et al., 2020).
In order to ameliorate this drift, we propose to freeze parts of the model, with a focus on the decoder. We find that freezing the decoder feed- forward parameters provides the biggest benefit (see $6.1 for the detailed ablation) and almost com- pletely eliminates the source language hallucina- tion problem in modular models.
Experiments
Pre-training Details We pre-train mmT5 on data from 100 languages in mC4 (Xue et al., 2021)
We add N L modular components the T5 architecture where L is the number of layers of the model and N corresponds to the number of languages which the model pre-trained on. The transformer weights are shared across languages while the modular component provides the model with manguage-specffic capacity. During forward pass, each input is first passed through the shared transformer weights and then routed through the corresponding language-specife module based on the language of the input. We follow this procedure for transformer layers until the representations are passed to the shared prediction head.
following the general pre-training setup of mT5 (Xue et al., 2021), if not specified otherwise. We pre-train mm75 at two model sizes: small (300M parameters), and base (580M parameters). We train model variants with input sequence length of 1024 and target sequence length of 256 for IM up- date steps with a batch size of 1024. The bottleneck size of each module is half of the hidden dimen- sion of the transformer model. For instance, as the base variant has a hidden dimension of 768, we set the bottleneck size to 384, We additionally pre- train a non-modular variant of our modular model, mT55, where all parameters are shared across all languages. The mT5 variant uses exactly the same hyper-parameters and pre-training setup mmT5. To ensure that the models are directly comparable and have exactly the same number of parameters, we add shared bottleneck layers to mT55 in the same configuration as in mmT5.
3.2 Modular Pre-training, Fine-tuning, and Inference
We pre-train both language-specifie modules and shared parameters jointly. During fine-tuning, we freeze all language-specific modules and only up- date the shared parameters. This paradigm allows to more effectively adapt the fine-tuned model to any of the languages included in the pre-traine pre-training data by simply switching to the corresponding language-spest speciffc module. At inference, the mod- ule corresponding to the target language is used together with the fine-tuned shared parameters.
Experimental setting We conduct experiments across datasets in zero-shot cross-lingual transfer and multilingual training scenarios. For zero-shot cross-lingual transfer, we train the model on sub- set of languages (e.g. only English) and evaluate the model on held-out data of the same task in other languages. In multilingual training, fine- tune the model on multiple languages of the same task, and evaluate the model on the same set of languages. As the Linguage-speciffe modular com- ponents are replaced at inference time, we do not update the parameters of the modular components. We do the same for our shared model variants, in order for the number of trainable parameters to be equal for comparable scenarios.' For each dataset, select the best model checkpoint based on per- formance on the validation set.
3.3 Overcoming Modular Representation Drift
When fine-tuning the modular model for transfer settings in $5, we observe a scenario of modular representation drift: we find that the shared param- eters that are updated during task-specific training drift away from the modular parameters and be- come thus less compatible with modules that used for inference. In practice, this leads to loss of compositional generalization where the modu- lar model generates text in the incorrect language, similar to its non modular counterparts (Vu et al., 2022); see $6.4.
We observe little benefit to freezing decoder parameters in non-modular models, however. We analyze impact of bottleneck sizes in $6.2. *Here, we follow the procodure of Pfciffer et al. (2022).

---

Mod. Params.
160
Model
Variant
Shared Params.
per Lang.
modal
mmn5
140
mBERT
Base
178M
size
X-Mod
Base
270M
7M
perploxrty
120
base
Base
270M
100
XLM-R
Large
550M
Small
300M
mT5
Base
580M
60
Small
300M 4M
mT558
0.0
0.4
D.B
Base
580M
14M
stap
Small
300M
4M 14M
mmT5
Base
580M
Figure 2: Perplexity (lower is better) different model sizes during pre-training for mmT5 and mT55, aver- aged across languages.
Table I: Number of shared and modular parameters of baselines and our models.
Decoder Freezing Configurations To over-
Evaluation Tasks For zero-shot cross-lingual transfer, we evaluate the XQuAD (Artetxe et al., 2020) and TyDi QA GoldP (Clark al., 2020) ques- tion answering datasets; on the XNLI (Conneau al., 2018) natural language inference dataset; on XL-Sum (Hasan et al., 2021) for summarization;" and MASSIVE (FitzGerald et al., 2023) for seman- tic parsing. 6 We mainly fine-tune the model on English training data and evaluate on the target lan- guages (Hu et al., 2020). For XL-Sum, we addition- ally evaluate in multi-souree zero-shot transfer setting where we train jointly on data in Arabic, En- glish, Japanese and Chinese ar,,umm,,,nnjj....
come the modular representation drift described $3.3, we experiment with different configurations of freezing parts of the model when fine-tuning the model on a downstream task. We experiment with freezing the LayerNorm (LN), self-attention (Att), cross-attention (CrossAtt) and feed-forward component (FFN) in the encoder (Enc) and decoder (Dec) parts of the transformer model. We ablate freezing configurations in $6.1 and report test re- sults the freezing configuration that performs best on the dev set for each dataset for mmTS. For dense models, we observe no impact with freezing and report results using full fine-tuning.
For multilingual training, we evaluate on seman- tic parsing (MASSIVE) and summarization (XL- Sum) datasets. For each dataset, we fine-tune and evaluate the model all languages jointly.
Results
5.1 Pre-training
We first compare the language modeling per- plexities differrent model sizes for mmT5 and mT55 during pre-training in Figure 2. We find that mmT5 significantly outperforms its fully shared counterpart during the early stages of pre-training and maintains the gap throughout the pre-training process. From efficiency perspective, mmT5 only requires 282k and 220k update steps respec- tively for the small and base versions achieve the same final perplexity as the mT55 models at IM update steps. This corresponds to a = 4x effi- ciency boost when training modular multilingual model compared to fully dense one.
Baselines Our main comparison method is mT5S, a shared model that pre-trained with the same parameters as our modular model. We also com- pare to the published results of the mT5 encoder- decoder model (Xue al., 2021). In addition, compare several encoder-only models including mBERT (Devlin al., 2019), X-Mod (Pfeiffer et al., 2022), and XLM-R (Conneau et al., 2020b), Encoder-only models are generally smaller as they lack a decoder but cannot easily be used for gener- ation tasks. We provide an overview of the model sizes of the baselines and our method in Table 1.
setup, and number of
5.2 Fine-tuning
'We do not evaluate on Oromo. Kirundi, Pidgin, and Tigrinya as they were not seen during pre-trane pre-training.
We present our main results on the test sets for zero-shot cross-lingual transfer and multilingual training scenarios in Tables and 3, respectively.
"Wo do not evaluate on Hebrew and Tagalog as they were not seen during pre-training.

---

Zero-Shot
XQuAD FI/EM
TyDiQA(GoldP) FI/EM
XNLI
XL-Sum
tenSuut RG, RGG, /RG,
MASSIVE
Acc
RG, / RGG / /RG,
EM
Encoder
mBERT
64.3/49.44
59.7/43.9 -/-
65.4
basc
X-Mod
72.8"
73.5
XIM--R
70.6/55.5
-1-
76.2
large
XLM-R
7 60.8
65.1 / 445.0
79.2
Encoder-dsxoddr
mT5
/4.1
33.2/23.2
67.5
small
mT55
61.9/ $1.9/46.2
33.5/
63.2
15.5/
2.2/ 14.2
17.0/ 15.1
21.7
mm'l'
50.4
00.8/36.33
68.5
4.6/ 14.4 16.7/
29.4 12.6/ 23.3
27.7
mT5
49.0
49.1
75.4
34.7
base
mT55
68.7/51.5
64.0 47.8
75.1
16.2/
2.8 4.5
18.6/ 6.0 16.7
39.9
mm75
7
69.0 / 53.2
77.8
19.6/
6.1 16.4
34.5 16.1 26.8
46.0
Table Zero-shot cross- lingull transfer test results averaged over all languages. mBERT and XLM-R scores are from (Hu al., 2020); XLM-R Base XNLI results are from (Conneau et al., 2020b); mT5 results are from (Xue et al., 2021); X-Mod results are from (Pfeiffer et al., 2022) avvragee is only on a subset of languages).
Multilingual
XL-Sum
MASSIVE EM
which only requires predicting single token, the
RG: / RG2 RG,
full multilingual generative capabilities of mmT5 are under-utilized. Overall, we see clear trend that our modular models significantly outperform their respective dense counterparts especially for generation tasks.
mT58
36.4/ 17.9 28.5
60.7
dec
small
mmT5
36.7 18.1 28.7
65.6
mT5S
39.1 20.3 30.5
64.6
base
mmT5
41.6 22.8/ 33.0
66.7
Table 3: Multilingual training test results averaged over languages.
Multilingual training For multilingual training in Table 3, we also find that the modular models outperform their dense counterparts across all tasks experiment with. Here we find the largest gains for semantic parsing (MASSIVE). For summariza- tion (XI--SUM,, we see smaller, but still consistent gains. These results indicate that modular repre- sentations are not only useful in transfer settings but that mm75 can also leverage labeled data in the target language to deliver superior performance compared to the standard non-modular models.
mmT5 outperforms both the original mT5 well mT55 across all model sizes. It achieves per- formance similar XLM-R at the same parameter size- size-despite encoder-decoder configuration- and significantly outperforms X-Mod, the only other modular model.
Zero-shot For zero shot cross- -oossll transfer scenarios, we see large gains for generative tasks in particular. For question answering (XQuAD and TyDiQA), we observe average relative Fl im- provement of 5.5 and 6.3 for the small and base models respectively. For summarization, larger zero-shot gains when jointly training on more than one language. We suspect that this is due to the increase in training data and due to pos- itive transfer during multi-source training, which modular methods are better able to harness. This is in line with previous findings that multi-souree training improves cross- lingull transfer in adapter- based setups (Ansell et al., 2021b). We also see gain of 6.1 EM points on MASSIVE. The small- est gains are achieved for the classification task XNLI Here, mmT5 improves over the baselines only by 1-2.4 accuracy points. We hypothesize that due to the constrained formulation of the task,
6 Analysis and Ablations
6.1 Impact of Freezing Configuration
We investigate the impact of the freezing configu- ration on the performance of the model. In Table 5, we compare the best-performing freezing configu- rations with a non-frozen baseline for mmT5 base (we show the results of all freezing configurations in Appendix A. We observe significant improve- ments when freezing the feed-forward layer of the decoder during fine-tuning, particularly in zero- shot scenarios. For multilingual training, freezing of the decoder has less effect on the performance. We also find that freezing parts of the decoder has no effect on the dense m model across all tasks (see Appendix A.

---

Zero-Shot
Multilingul XL-Sum dev Rga / Rga Rgo
XQuAD
XNLI
MASSIVE dev
dev (en)
Test
dev
Emb
ENCLN
DEELN
DecAr
DeCcrossn
DeCHEN
/ em
em
ucc
90.7 83.6
66.9/49.3
75.5
32.1
41.2/22.4/3.4
X X
91.9 85.1
75.8
75.6
43.2
41.2 /
X
X
91.8/ 85.1
75.8/
77.3
41.0
41.9 23,1 33,2
X
X
X
X
92.1 85,5
76.3/
76.1
45.4
40.8 / 22. 32.3
x
X
X
91.8 85.1
55.0/
77.7
39.9
41.8/ 23.0/ 33.1
Table 4: Results of different freezing configurations for mmT5 base different tasks. Dev results for most. We Art, and EN and always freeze End and Dec X indicates that this component is
always fine-tune An frozen during task-level fine-tuning.
78
Typc
76
mmn55
70-
Acorsscy
XGuAD
XNU
68
Dattieneck Size
96
66
50
Tusk
Figure 3: Comparison of bottleneck sizes of base mmT5 for XQuAD (F1) and XNLI (Accuracy).
Figure 4: Comparison of model sizes for XQuAD (FI) and XNLI (Accuracy).
6.2 Impact of Bottleneck Size
6.4 Source Language Hallucination
We experiment with different bottleneck sizes of the modular components to understand the impact of providing each language with capacity. We report results for XQuAD, and XNLI in Fig- ure using mmT5 base and bottleneck sizes of 96. 192, 384, and 768. We find that for all three tasks the bottleneck size has little effect on the downstream task performance, achieving only 0.5- 2 absolute points difference between the larger and the smaller bottleneck sizes. This suggests that it is sufficient provide the model with only small amount of language-specific parameters in order learn idiosyneratic information and miti- gate catastrophic interference, and highlights the parameter-efficienge modular models.
We perform analysis of the generated text on the XL-Sum dev sets for mT5$ and mmT5 models trained in zero-shot setting (L.Suomreen.jo.sil using full fine-tuning and decoder freezing con- figuration. We automatically detect the language of the generated text using the Language Detection from the Google Cloud Translation API? (Caswell et al., 2020). We show the results in Figure 6. We find that most models tend to generate text in one of the source languages (in this setting: Arabic, En- glish, Japanese, and Chinese). This holds true also for mmT5 when fine-tune the decoder. How- ever, when freezing the decoder we observe dra- matic improvement in the target language genera- tion rate from 1% to 99% of examples for mmT5, essentially solving the issue of source language hal- lucination in cross-lingual transfer scenarios. This improvement in language consistency also helps explain the significant improvement of the mod- ular model over its dense counterparts natural language generation tasks.
6.3 Impact of Model Size
In Figure 4, plot the performance difference of mm75 and S for the small and base variants. We find that the modular model outperforms the dense variant across model sizes with a similar gap, indicating that the positive effect of modularity may not diminish at scale.
'tttps: //c oud. goog e. com cttcctotaaatteee rans late/ docs /baatbaasaccc c/tseec//eeeeeee ing- language

---

Südkalifornien besteht aus [...] einer interna- tionalen Metropolregion und GroBstadtgebi- eten. Die Region ist die Heimat von zwei er- weiterten Metropolregionen mit jeweils mehr als fünf Millionen Einwohnern. [...] Question: Wie viele erweiterte Metropoire- gionen gibt e
[...] Analysen [...] waren irreführend, da es mehrere Jahre dauert, bis die Auswirkungen zu Veränderungen des Wirtschaftswachstums führen. [...]
Question: Wie lange dauert es. bis sich die Auswirkungen als Veränderungen des wirtschaftlichen Wachstums manifestieren?
mm75: T5: zwei m
mm
mmT5: mehrere Jahre m
two
more ere Jahre
Figure 5: XQuAD examples where mT5 generates tokens with the correct meaning but in the wrong language. For the same examples, mm'5 is able to generate tokens in the correct language when freezing parts of the decoder.
seen during pre-training. However, with more than 7000 languages spoken in the world (Joshi et al.. 2020), mm15 covers less than 1% of them. While extending the model to unseen languages is out of scope for this work", evaluate the poten- tial reusability of existing language modules for truly unseen languages with case study on Taga- log. We utilize the base mmT5 model fine-tuned on the English MASSIVE training dataset (see Ta- ble 2). As a Tagalog language module does not exist within mmT5, we test all existing other lan- guage modules when evaluating on the Tagalog test set. In Figure 7, we report the Exact Match (EM) zero-shot accuracies for all languages. The module performing best corresponds Javanese, which the most closely related language Tagalog as both belong to the Malayo-Polynesian subgroup of the Austronesian language family. This finding demonstrates the effectiveness of modular models; modular components specifically incorporate inter- pretable concepts, which can be re-used for unseen scenarios. Additionally, they be further fine- tuned adapted to the target domain if training data is available.
1.0
model
mmn5 ft ments froz
Percetage
0.6
0.4-
target
Detected Language
Figure 6: Detected languages of generated text of the development set of "nvenjoo...j, All models have base size. *ft indicates that the decoder was fine- tuned, *froz indicates that the decoder was partially frozen. High numbers desireable for the first of plots ("target language"), low numbers are desireable for the remaining four sets of plots ("ar", "en", "ja", "zh"), We only include zero-shot cross- lingual results, therefore exclude the four source languages; all models achieve 100% accuracy for those. For more granular re- sults see Appendix Table 10.
In addition, we manually analyze outputs of mT55 and mmT5 on XQuAD and find similar is- sues of source language hallucinations. We show examples in Figure 5. Although the task is extrac- tive QA, i.e., the answer is substring of the input, m tends to translate subwords into English (the source language). This does not happen to mmT5 when freezing parts of the decoder, partially ex- plaining the large improvements of mmT5 over m on TyDi QA Table 2.
7 Conclusion
We have proposed mmT5, a modular multilingual encoder-decoder model. During multilingual pre- training the majority of parameters of mmT5 are shared between tasks, but each language is pro- vided with small amount of parameters only cessible to the respective language. We demon- strated that integrating modularity as an architec-
6.5 Module Re-Use for Unseen Languages
8 PPrevious work has demonstrated that it is possible to extend multilingual models to unseen languages (Preiffer al., 2020, 202 2022).
the previous sections we have evaluated the cross- cross-lingual performance of mmT5 on languages

---

22.5
22.0
21.5
21.0
20.5
20.0
19.5
19.0
en 05
be be
el GR
oy Ge
CN
R9LAL
ar
kmn RU
IN
fa RR my iin, RD
kn Au tn TH
KR
am EELETT C IN
hy
da BL
a2
no No
sv of AL My
hu AU WLNN
sw FE
PL
Ir C.
mn MN
pt
de re_0E
my
or
mi
Figure 7: Average of the top-5 zero-shot EM accuracies on the Tagalog MASSIVE development set by varying the
input language ID. Tagalog not during mmT5 pre-training.
tural inductive bias significantly improves train- ing efficiency, where the same perplexity as an equivalent fully dense model is achieved at quar- ter of the update steps. mm15 considerably out- large number
Third, we performed fixed routing under the as- sumption that the language ID is easy to obtain. We chose this path, as coaming-to-route has many difficulties such as training instabilities (Pfeiffer et al., 2023). However, this architecture design lim- its the sharing of information (e.g. domains) across languages. Consequently, a combination of fixed routing and learned routing would allow the model to learn how to share information across subsets languages.
performs comparable models of tasks including Question Answering, Seman- tie Parsing, Summarization and Classification in both zero-shot as well as multilingual scenarios. Finally, we show that by freezing parts of the de- coder when fine-tuning mm'15 on a target task in source language, the model consistently generates text in the target language. Consequently, modular- ity arguably solves source language hallucinations in cross-lingual transfer scenarios.
Fourth, we did not try using mm75 for machine translation. Using modular design for this type of task setup is quite natural, modules from the encoder and decoder can be casily replaced with the source and target language components, respec- tively. The effectiveness of modular sequence-to- sequence models for NMT has been investigated previously (Bapna and Firat, 2019; Philip et al., 2020; Chronopoulou et al., 2020; Le et al., 2021; Üstün et al., 2021; Stickland et al., 2021; Garcia et al., 2021; Dua et al., 2022).
8 Limitations and Future Work
In this paper, we explored the use of modularity for multilingual language models. We showed that modularity significantly improves lingull per- formance on a number of generative tasks by miti- gating hallucinations in the source language. How- ever, there are still many avenues for future work. First, did not consider placing the modules in different parts of the model. We only exper- imented with placing bottleneck layers after the ferd-forward component of cach transformer layer. Previous work has demonstrated that depending on the modality, different placements perform better (Pfeiffer et al., 202 20 Eichenberg et al., 2022).
Finally, did not consider extending the model to languages beyond those we pre-trained on. While our preliminary results (see $ 6.5) suggest that there are benefits of reusing related language modules to learn unseen languages, this requires further experimentation. However, previous works have demonstrated that modular (Pfeiffer et al., 2022) well dense models can be adapted to new languages and scripts (Pfeiffer et al., 2020, 2021b). Alternatively, future work might con- sider using post-hoc adaptation techniques, such LoRA (Hu et al., 2022), to adapt modules to new languages.
Sccond, we only experimented with extending the vanilla transformer architecture with modular components. Future work might consider modular- izing different parts of the transformer, such as the attention attention-componee components or entire feed-forware layers like in Kudugunta et al. (2021).

---

Acknowledgements
Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Pars- ing with multilingual bert, a small treebank, and small corpus. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 1324-1334.
We thank Andrea Gesmundo, Marc' Aurelio Ran- zato, Srini Narayanan, and Emanuele Bugliarello for helpful feedback on a draft of this paper.
Vincent S. Chen, Sen Wu, Alexander J. Ratner, Jen Weng, and Christopher Ré. 2019. Slice-bused learning: A pro- gramming model for residual learning in critical duta slices. In Advances in Neural Information Processing Systems 32: Annual Conference on Newral Information Process- ing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9392-9402.
References
Alan Ansell, Edoardo Maria Ponti, Anna Kothonen, and Ivan Vulic. 2022. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Popers), ACI. 2022, Dublin, Ireland, May 22-27. 2022, pages 1778-1796. Association for Computational Linguistics.
Ankanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma. Gaurav Mishra. Adam Roberts. Paul Barham, Hyung Won Chung. Charles Sutton, Schas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du. Ben Hutchinson, Reiner Pope. James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ai Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gar- cia, Vedant Misra, Kevin Robinson. Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Bar- ret Zoph, Alexander Spiridonov, Ryan Sepassi, David Do- han, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez. Erica Moreira. Rewon Child, Oleksandr Polo- zov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bren- nan Saeta, Mark Diaz. Orhan Firat, Michele Catasta. Jason Wei. Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24:240:1 141240::1240:11 240:113.
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Auna Korhonen. 2021a. MAD-G: Multilingual adapter generation for effi- cient cross-lingual transfer. In l'idding of the Association for Computational Linguistics: EMNLP 2021, pages 4762- 4781. Punta Cana, Dominican Republic. Association for Computational Linguisties.
Alan Ansell, Edoardo Maria Ponti, Jonas Piciffer, Schastian Ruder, Goran Glavaš, Ivan Vulić, and Auna Korhonen. 202 MAD-G: Multilingual adapter generation for effi- cient cross-lingual transfer. In Findings of the Association for Computational Linguistics: EMNLP 2021. pages 4762- 4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mikel Artetxe, Schastian Ruder, and Dani Yogatama. 2020. On the cross- lingual transferability of monolingual repre- sentations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623- 4637, Online, Association for Computational Linguistics.
Alexandra Chronopoulou. Dario Stojanovski, and Alexan- der Fraser. 2020. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pages 2703-2711, Online. Association for Computational Linguistics.
Ankur Bapna and Orhan Firat. 2019. Simple, scalable adap- tation for neural machine translation. In Proceedings of Empirical Methods in Natural Language Processing and the 9th International Joins Con- ference on Natural Language Processing, EMNLP-IJCNI 2019, Hong Kong, China, November page. 1538- 1548. Association for Computational Linguistics.
the 2019 Conference
Jonathan H. Clark, Eunsol Choi, Michael Collins. Dan Gar- rette, Tom Kwiatkowski, Vitaly Nikolacv, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information- secking question answering in typologically diverse lan- guages. Transactions of the Association for Computational Linguistics, 470.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncc- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffeey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess. Jack Clark, Christopher Berner. Sam Mc- Candlish, Alec Radford, Hya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020. NewrlPS 2020, December 6-12, 2020, virtual.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Oit, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised tion learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
lingual representa-
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave Myle Out, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Unsupervised cross-lingual representa- tion learning at scale. In Proceedings of the 58th Con- ference of the Association for Computational Linguistics, ACI. 2020, Virtual Conferencee, July 6-8, 2020, pages 8440- 8451.
Isaac Caswell Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unexpected chal- lenges on the path to a thousund-language web text corpus. In Pruceedings of the 28th International Conference on Computational Linguistics, pages 6588-6608, Barcelona, Spain (Online). International Commitice on Computational Linguistics.

---

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 XNLI: Evaluating cross- boss-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computa- tional Linguisties.
Findings of the Association for Computational Linguistics: ACL-IJCNLR 2021. pages 4693-4703, Online. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards uni- fied view of parameter-efficiett transfer learning. In The Tenth International Conference on Learning Representa- fions, ICLR 2022, Virtual Event, April 25.29, 2022. Open- Review.net.
Jacob Devlin, Ming- Wei Chang. Kenton Lec, and Kristina Toutanova. 2019. BERT: pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLL 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume I (Long and Short Pa- pers), pages 4181
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzkebski, Bruna Momonn. Quentin de Laroussilhe. Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter- efficient wansfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 2790-2799.
Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan. 2022. Tricks for training sparse translation models. In Proceedings of the 2022 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Hunan Language Technologies, pages 3340 334S, Seattle, United States. As- sociation for Computational Linguisties.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language mod- els. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Constantin Eichenberg. Sidney Black, Samuel Weinbach, Leti- tia Parcalabescu, and Anette Frank. 2022. MAGMA multimodal augmentation of generative models through adapter-baset based finetuning. In Findings the Association for Computational Linguistics: EMNLP 2022, pages 2416- 2428, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig. Orhan Firat, and Melvin Johnson. 2020. XTREME: A mas- sively multilingual multi-task benehmark for evaluating cross-lingual generalisation. In Proceedings of the 37th In- ternational Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4411 14411. PMLR.
William Fedus, Barret Zoph, and Noam Shazeer, 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23:120:1-120:39
Pratik Joshi, Sebastin Santy, Amar Budhiraja. Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Compu- rational I Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 6282 6282-6293. Association for Computational Linguis- tics.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach. Vishesh Kakarala, Richa Singh, Swetha Ran- ganath. Laurie Crist, Misha Britan, Wouter Leeuwis. Gökhan Tür, and Prem Natarajan. 2023, MASSIVE: A Im-example multilingual natural language understanding datuset with 51 ppologically-dierss languages. In Pro- ceedings of the s1 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 4277-4302. Association for Computational Linguistics.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dinitry Lepikhin, Minh-Thane Thang Luong, and Orhan Firat. 2021. Beyond distillation: Task-level mixture-of- experts for efficient inference. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 3577-3599. Association for Computational Linguistics.
Xavier Garcia, Noah Constant, Ankur Parikh, and Orhan Firat. 2021. Towards continual learning for multilingual machine translation via vocabulary substitution. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 1184-1192, Online. Associ- ation for Computational Linguisties.
Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, Nikolai Rozanov, and Goran Glavaš. 2020a. Common sense or world knowledge? investigating adapter- based knowledge injection into pretrained transformers. In Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop Knowledge Extraction and Integration for Deep Learning Architecture, pages 43-49, Online. Association for Computational Linguistics.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. Demix layers: Dis- entangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022. pages 5557-5576. Associ- ation for Computational Linguisties.
Anne I Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020b. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transform- ers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (LMNLP), pages 4483 4499, Online.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rah- man, and Rifat Shahriyar. 2021. XI- Large-scale mul- tilingual abstractive summarization for 44 languages. In
Hang Le, Juan Miguel Pino, Changhan Wang. Jiatao Gu. Di- dier Schwab, and Laurent Besacier, 2021. Lightweight adapter tuning for multilingual speech translation. In Pro- ceedings of the 59th Annual Meeting of the Association for

---

Computational Linguistics und the 11th International Joint Conference on Natural Language Processing, ACL/JC- NLP 2021. ( (Volume 2: Short Pupers), Virtual Event, August 1-6, 2021, pages 817-824. Association for Computational Linguistics.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska. Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonnenns reasoning. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (LMNLP). pages 2362- 2376, Online. Association for Computational Linguisties.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021a. Compacter: Efficient low-rank hypercom- plex adapter layers. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural In- formation Processing Systems 2021, NeurlPS 2021, De- cember 6-14, 2021, virtual, pages 1035.
Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. 2021. What to pre-train on? efficient intermedi- ate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 10585- As- sociation for Computational Linguisties.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa De- highnni, and James Henderson. 2021b. Parameter-efficien multi-task me--uning for transformers via shared hyper- networks. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing, ACL/JCNLP 2021. ( (Volune 1: Long Pupers), Virtual Event, August 1-6, 2021. pages 565-576. Associa- tion for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1 11:140:1-140:6.
Alvise Rebufñ, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA. USA, pages 506-516.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transform- ers. In Proceedings of the 2022 Conference of North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479- 3495, Seattle, United States. Association for Computa- tional Linguistics.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2018. Efficient parametrization of multi-domain deep neu- ral networks. In 2018 IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 22, 2018, pages 81 19-8.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. 202 AdapterFu- sion; Non-destructive task composition for transfer Iearn- ing. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguis- ties; Muin Volume, pages 487-503, Online, Association for Computational Linguistics.
Andreas Rücklé, Jonas Pfciffer, and Iryna Gurevych. 2020. Multicqa: Zero-shot transfer of self-supervised text match- ing models on massive seale. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2471 2486. Association for Computational Linguis- tics.
Jonas Pfeiffer. Sebastian Ruder. Ivan Vulié, and
Edoardo Maria Ponti. 2023. Modular deep learning, arXiv
preprins.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference Empirical Methods in Natural Longuage Processing (FMNI.P), pages 7654-7673, Online. Association for Computational Linguistics.
Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint.
Schastian Ruder, Noah Constant, Jan Botha, Aditya Sid- dhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced mul- tilingual evaluation. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Process- ing, pages 10215-10244 Online and Punta Cana, Domini- can Repuhlic. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. 2021b. UNKs Everywhere: Adapting Multilin- gual Language Models to New Scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Online, November. 2021.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz. Andy Davis, Quoe V. Le, Geotfrey F. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely- gated mixture-of oo-.eperteeeeeeeeeeooeeeeee layer. In 5th International Con- ference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview..nt.
Jerin Philip, Alexandre Berard, Matthias Gallé, and I Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In Proceedings of the 2020 Confer- ence Empirical Methods in Natural Language Process- Online. Association for
ing (EMNLP), pages Computational Linguistics.
Telmo Pires. Robin M. Schmidt, Hi-Hsiu Liao, and Stephan Peitz. 2023. Learning language-specific layers for multi- lingual machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14767-14783. Association for Computational Linguistics.
Asa Cooper Stickland, Alexandre Berard, and Vassilina Nikoulina. 2021. Multilingual domain adaptation for NMT: decoupling lunguage and domain information with adapters. In Proceedings of the Sixth Conference on Machine Trans- lation, WMT@ @ EMNLP 2021, Online Event, November 10- 11, 2021, pages 378-598. Association for Computational Linguistics.

---

Asa Cooper Stickland and lain Murray. 2019. BERT and pals: Projected attention layers efficient adaptation multi- task learning. Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA. volume 97 of Pro- ceedings of Machine Learning Research, pages 5986-5995. PMI.R.
A
Appendix
A.1
Freezing combinations
We show results with different freezing combinations Table find that freezing the FFN component the Decoder results the biggest performance gains.
A.2 Language-ID prediction on Cross-lingual Summarization
Ahmet Üstüi,, Alexandre Berard, Laurent Besacier, and Matthias Gallé. 2021. Multilingual unsupervised neural machine translation with denoising adapters. In Proceed- ings of the 2021 Conference on Empirical Methods in Nat- ural Lunguage Prucessing, EMNLP 2021. Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. pages 6650-6662. Association for Computational Linguis- tics.
We report the languages predicted by the Language Detection model from the Google Cloud Translation API' (Caswell et al., 2020) for the XL-Sum" "r.en.ja.sh task in Table 10. We find that mmT5 achieves near perfect performance for all target languages when freezing parts of the decoder (57)-99% of the text is generated in the correct target sigifcayttgt outperfoming all other model variants. Interestingly, mmI'5 hallucinates in the source language when the decoder is tine- tuned (sl), resulting in drop down to only 2% in the correct target language. mT5 also benefits slightly from freezing parts of the decoder, with an improvement from 7% to 18% target language generation, however, this is where close to the performance of mmTS,
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302-2315, Online. Associa- tion for Computational Linguistics.
Marko Vidoni, Ivan Vulić, and Goran Glavaš. 2020. Orthogo- nal language and task adupters in zero-shot cross-lingual transfer. arxiv preprint.
A.3 Language-level Results
XNLI We report XNLI validation results in Table 11 and test results in Table 6.
Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant, 2022. Overcoming catastrophic forget- ting in zero-shot pross-lingual generation, In Proceedings of the 2022 Conference Empirical Methods Natu- ral Language Processing, pages 9279 9300, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
XQuAD. report XQuAD validation results in Table and test results Table 7.
MASSIVE. We report MASSIVE validation results in Table 17 and test results Table 8.
TyDiQA. We report TyDiQA validation results in Ta- ble 23.
Ruize Wang. Duyu Tang. Nan Duan, Zhongyu Wei, Xuan- jing Huang. Jianshu Ji, Guihong Cao. Daxin Jiang, and Ming Zhou. 20 K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Asso- ciation for Computational Linguistics: ACI/IICNIP 2021, Online Event, August 1-6, 2021, volume ACL/UJCNPP 2021 of Findings of ACL, pages 1405- 1418. Association for Computational Linguistics.
Multilingual XL-Sum We report XL-Sum validation results in Tables 18, 19,20, 21, and 22 and test results in Table 15.
Zeroshot XL-Sum We report XL-Sum validation re- sults in Table 14 and test results in Table 13.
Zeroshot XL-Sumar.en.ja.u
We report XL-Sum val- idation results in Table 16 and test results in Table 12.
Xinyi Wang. Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig. 2021b. Efficient test time adapter ensembling for low-resource language varieties. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 730-737 Punta Cana, Dominican Republic, Association for Computational Linguisties.
A.4 Language-level Pre-training Perplexities We report the language-level perplexities of the different model variants and sizes in Figures 8, 9, 10, 11, 12, 13, 14.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120- 130, Ouline. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffol. 2021. m'S: A massively multilingual pre-trained text-to- text transformer. In Proceedings of the 2021 Conference of the North American Chapier of the Association for Com- putational Linguisties: Human I Language Technologies, pages 483 498, Online. Association for Computational Linguistics.
Biao Zhang, Ankur Bapna, Rico Semnrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual translation. In 9th International Conference on Learning Representations, ICLR 2021, Vir- rual Event, Austria, May 3-7, 2021. OpenReview.net.
"https: oud. goog e com ctoc rans late/ docs bocs/basic//etees ing- language

---

Zero Shot
Multi Source
XQUAD
XNLI
XI Sum
XI -uummaaaaa der
MASSIVE
X1-Sum
MASSIVE
des (mm)
des
der
des
dry
Emnh
Enshs
DoRiN
DeCas
Delson
[ cm
Rgy
Rg. /Rg./
EM
Rg. /Rgy
EM
11
/8.6
66.91-40 7-433
75.5
15.4/20/140
187/6.1/66
32.1
41.2 112/224/324
12
X
90.7 00.7/83.4 183.4
05.57-48.0
75.3
78.3
41.0143.4
70.9
68
64.574921
77.5
15
K
91 24.
74.3/57.5
73.8
16
91 9785.1
75.8/59.5
73.6
43.3
41/2/224/12A
17
91.5 85.1
75.8/59.8
77.3
19.7/62/ 1644
34.7/ 16.2 26.9
41.0
41.9/2311 23.1 /33.2
18
91.2 01.2/84.5 84.5
75.01 59.3
13.3
19
2/84.5
74.6 58.8
73.8
x10
x
02.1 /
76.3
66.3
76.1
45.4
40.8 22.1/323
66.78
<11
90.91 84.0
74.8/5K3 58.8
73.1
s12
91.21 84.5
75.0 59.3
73.6
113
91.31 84.5
74,91589
76.3
114
91.8 95.1
75.01 59.2
77.7
399
41.8 23.03333
Table 5: Results with different freezing combinations mmT5 on different tasks. X indicates that the component is frozen in the respective configuration Dev results for most. We always finetune the attention in the encoder (Enс the feed forward layer in the encoder (EncyFx), and always freeze the modules in the encoder (EnC)rd) and decoder MDeC find that configurations sl-s4 strongly underperform the respective other configurations (s5-s14), suggesting that freezing the feed forward layer of the decoder is essential for good cross- lingull transfer performance.
model
bg
de
e5
th
avg
small
mmTS
65.3
71.9
70.2
70.5
74.7
73.4
70.1
64.2
59.1
68.5
small
m
63.8
69.1
67.6
68.1
80.0
71.6
69.3
60.4
68.7
53.2
64.1
59.1
58.4
63.9
64.4
65.5
base
mmT5
75.0
81.2
80.3
79.5
86.9
82.6
80.9
73.4
78.3
74.0
74.9
76.3
69.9
76.7
77.2
77.8
buse
m
72.7
78.7
77.6
77.3
80.7
78.5
70.7
77.7
73.0
72.8
67.7
73.8
73.5
75.1
Table 6: XNLI test results for all language. We select the checkpoint performing best on the validation set.
model
d:
Fi /EM
F1 /BM
/EM
F1
FEM
FEM
F1
/CM
/EM
/EM
FL
EM
/EM
F1
/EM
/EM
bone small
mTS
53.5/37.1
67.2/
59.5/411
81.7
69.8/53.7
54.7 40.8
628/44.5
50.3/37.9
57.2/393
59.7
64.754.0
61.91162
74.2/57.6
79.5 63,0
77.6/599
86.7
79.3/61.
72.4
69.3/59.3 77.61587
53.9
743/542
74.4/
76.3/6013
63.3/43.
75.9
63.3/403
843/7119
76.1 /58.7
62.8/47.1
6407/2.6
70.1/117
70.4550.
66.0
68.7/51.5
Table 7: XQuAD test set results for languages. select checkpoint performing best on the English development set.

---

Language
Exact Match (EM)
af 2A
57.5
am ET
29.6
ar ar_SA
38.3
az_A2
41.3
BD
37.2
ey
35.5
dK_DK DK
60.5
de_DE
55.3
el_GR
49.6
US
72.7
es ES
53.8
dg
FI
fR
48.2
[i_
54.4
85.6 77.6
[r_FR FR
51.4
mInTS
87.2 79.4
87.4 79.4
hi hi_
44.1
87.3 79.6
47,4
Small
s14
87.4 79.2
by AM
38.6
1d_
57.1
84.9 76.5
85.8 77.6
1s_
42.8
mISS
it_
51.7
s10
86. 77.9
ja_JP
42.5
77.6
jv_
38.1
ka
38.9
90.7 83.6
km KH
40.7
90.7 83.4
kn_
34.4
90.7 83.5
90.9 83.6
ko_ KR
39.1
912//4.1
1v IV
50.3
91.9 85.1
m1
36.0
TInTS
S7
91.8 85.1
MN
34.2
ms MY
52.5
Basc
#9
91.2/ 84.5
my MM
33.8
92.1
58.2
si
90.9 84.0
n1 NL
57.5
91.2 84,5
pl PL
52.9
s13
91.3 84.5
gt
56.0
$14
91.8 85.1
RO_RO
55.4
s1
89.9 82.5
RU_RO
50.9
mIsS
86
90.2 83.0
81 ST_
50.3
9Q
48.3
s10
90.2 82.8
90.4
9V_SE
58.9
43.0
ta_
37.1
Table XQuAD validation results English across the different freezing configurations.
te_ NN
35.4
th
50.1
tr_ER
47.9
ur PK
39.6
44.9
30.0
sh_5w
28.2
Average
46.0
Table 8: MASSIVE Exact Match (EM) test accuracies the best model (s10 modular) for all languages

---

13 long
bn
pred ling cTp
-114
08
68
31
27
36
35
37
60
24
05
60
33
28
67
63
30
20
0.00
0/00
1.0
0.0
11.00
0.00
1.00
0.00
0.00
1.91
0.05
0.96
0100
0.00
0:01
1.01
1.0
0.0
0.08
0.31
0.099
0.02
0.5
0.02
0.65
0.19
0.13
0.76
3.95
0.35
0.76
cfc
201
2h
26
miits
1.8
1.0
11.
0.0
0.03
0..33
0.11
0.00
0.03
11.65
0.01
01
0.0
0.05
0.10
0174
0.
0.00
0:25
0.01
11.67
0.01
0:01
0.0
0.0
10.00
0.03
0.0
0.95
0.01
0.76
0.94
(el bing cfg
6M
14
89
0.00
0.34
0.18
0.10
0.43
0.03
0.17
0.62
0.01
0.01
0,00
0.33
0.36
0.04
0.30
0.81
0.13
0.62
0.03
0.33
0.10
0.25
0133
0,62
0.17
milly
67
1,00
0.00
11,00
0.00
8100
1,00
0.00
(0.00
0.00
(1000
1.30
11000
(1.0)
01001
0.00
0.98
0.00
0.00
0.00
0.00
1,00
0.00
(100
0.00
0:00
mist
0.13
0.16
[1.01
0.56
0.17
1111
0.83
0.83
0.00
8.00
01.33
11:33
0.02
CU64
0.065
0.07
0.14
0.24
0.01
0.82
8.61
0.01
11.54
0.00
0.00
0.23
0.06
0..2
0.49
0.06
0.01
0..76
0.00
0.000
0.12
0.08
0.23
0U48
0.01
0.25
0.08
0.57
0.00
0.00
0.66
0.001
0.31
0.00
0.00
ty linp
km
by
111
my
peol ling cfp
28
31
37
30
31
2h
-17
24
51
vth
mmlly
1.0
0.0
1.0
1.00
0.00
(1.0)
0.00
01000
0,99
0.03
0.00
0114)
0.00
0.96
0110)
0.00
(100
0.00
1,00
(1.00
0.00
1.4
0.00
0.00
0.02
1.04
0.34
0.05
0.08
0.05
0.00
0.00)
0.00
0.08
0.24
0.07
0.19
0.20
p6
ling
cf
2h
28
XB
0.10
0:27
0.33
0.22
0.83
0.00
0.0
0.14
0.10
0.24
0.03
0100
0.21
0.0
50
50
mmts
<1
0.00
0.30
0.23
0.11
0.25
0.01
0.23
0.45
0.03
0.04
0.00
0.110
0.5M0
0,035
0.185
0.01
0.18
0.51
0.02
0.05
1101
0.36
01.16
0.14
0.33
mints
<7
1.00
0.00
11.00
0.00
0.00
1.00
0.00
(..0)
0.00
0.00
0.95
0.000
0.010
0.000
0,000
1.00
0.00
0,00
0.00
0.00
1.00
0.00
11.00
0.00
0.00
a1
0.02
0.48
(1.01
0.40
0.02
0.04
0.65
0.25
0.00
0.00
0.03
0.375
0.525
0,005
0.005
0.01
0.54
0.13
0.00
0.00
0.07
0.09
0.02
0.58
0.07
37
0.07
0.63
0.10
0.11
0.00
0.06
0.71
0.15
0.00
0.00
0.02
0.200
0.545
0.000
0,000
0.02
0.57
0.09
0.00
0.00
0.32
0.05
0.06
0.46
0.03
lal haze
14
11
14.
poal hay cfc
03
34
ah
$4
0.35
0.13
0.70
0.03
0.18
0.32
0.32
0.94
muiss
x7
L00
0.00
0.00
0.50
0.00
1.00
0.00
0.00
0.00
0.00
1,00
[100
1.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
100
0.00
0.00
0.00
0.00
mis?
0.01
0.10
0.01
0.49
0:00
1108
0.16
0.01
0.00
0.88
0.08
0.01
[1.4
C111
0.0
0.02
0.12
(100
0.0
0/22
0.56
0.1
0.20
0.14
mis?
0.31
0.38
0.05
0,12
0:55
0.29
0.00
0.00
0.50
0.37
0.00
0.27
0.34
0.00
0.18
0.41
0.31
0.00
0.20
0.20
0.06
0.24
0.01
>h
(etclwding
peo king
cfp
28
51
37
*h
sh
28
76
m
0.0
0.02
mis
RTSF
Table 10: Language prediction results the amm.jaa task setup. The generated summarization text is passed into the language prediction model. We report the percentage of text which the model predicts to be the correct target language, as well as each of the 4 source languages. It is possible that another language was predicted, the numbers therefore do not need to sum up to 1.0.

---

cfg
ar
bg
de
cl
cn
fr
hi
SW
th
tr
ur
vi
zh
avg
s1
63.7
68.4
68.4
66.7
77.6
71.6
69.3
60.8
66.3
60.2
62.2
62.5
56.2
64.5
63.4
65.4
mm115
56
63.5
69.2
69.2
68.2
81.7
73.9
71.7
62.2
67.1
60.8
64.8
62.3
54.7
61.3
64.2
66.3
Small
87
65.4
70.8
70.5
70.1
81.5
74.5
73.0
63.1
69.1
62.7
66.1
63.1
58.4
65.7
65.7
68.0
s10
63.9
69.6
68.9
69.4
80.6
73.7
70.4
62.4
67.8
59.9
65.2
61.4
54.6
62.6
65.6
66.4
s14
64.7
70.7
70.4
69.5
81.6
74.5
72.6
63.2
68.8
60.9
65.3
62.4
58.0
65.1
65.9
67.6
63.2
68.6
67.8
67.3
80.2
70.8
70.1
59.8
67.6
52.9
62.5
58.2
56.9
63.5
63.2
64.8
m
86
58.8
64.5
64.2
62.4
77.2
67.3
65.9
55.0
64.0
48.2
53.1
50.9
58.6
61.2
60.8
47
62.2
66.6
77.9
70.8
69.3
59.4
67.1
53.2
57.6
56.1
61.7
57.5
66.8
77.6
56.4
64.4
51.0
54.8
59.0
61.3
61.5
s14
61.8
66.9
67.7
77.6
71.0
69.3
59.4
67.6
53.3
61.6
58.0
56.3
61.1
62.9
64.1
73.5
77.9
77.6
77.7
84.3
79.2
77.8
72.2
75.3
71.7
72.9
73.5
68.9
75.0
74.6
75.5
73.3
77.1
77.2
77.1
84.1
79.8
76.9
71.2
75.2
70.7
72.8
72.6
68.7
74.2
74.9
75.0
&
74.4
79.9
79.7
78.2
85.8
81.3
79.5
73.6
76.7
72.9
75.0
74.4
69.2
76.2
77.0
76.9
34
75.9
79.8
79.5
79.6
86.2
81.9
80.1
73.5
78.0
73.5
74.3
75.2
70.1
77.0
77.4
77.5
72.4
75.5
75.4
76.1
83.3
79.2
76.8
770.9
73.4
71.5
69.9
70.7
67.
71.8
772.8
73.8
73.5
77.6
78.2
77.2
84.6
81.0
79.0
72.3
76.2
71.8
72.4
72.2
68.7
73.8
74.7
75.6
mmi15
79.8
86.3
79.4
73.0
74.7
75.8
72.2
76.4
73.7
83.2
73.2
70.2
73.5
69.7
70.7
67.1
71.7
73.3
Base
74.1
77.8
76.9
84.7
80.2
78.3
71.6
75.8
71.4
73.2
73.7
74.0
75.3
75.6
S10
74.1
77.7
78.0
77.5
84.3
81.4
78.3
72.9
75.9
73.2
73.5
73.7
69.4
75.1
76.2
76.1
s11
71.3
75.5
73.9
74.7
82.2
77.8
75.2
70.0
73.3
69.6
71.1
70.0
66.9
70.8
73.4
73.1
s12
74.1
78.0
77.5
77.4
84.6
79.4
78.0
71.3
75.7
71.8
73.4
73.5
69.3
74.7
75.7
75.6
s13
75.2
78.1
78.3
78.2
84.4
80.1
78.4
73.1
75.9
73.0
73.9
73.8
69.7
75.8
76.5
76.3
s14
76.0
80.6
81.0
79.0
86.9
82.4
79.9
73.2
77.6
74.0
74.9
75.7
70.0
76.3
77.6
77.7
81
72.0
76.0
76.4
76.3
84.4
78.0
78.2
69.8
74.7
66.1
71.3
71.1
67.8
73.0
72.4
73.8
mISS
86
71.2
75.0
75.0
75.7
84.3
78.7
78.0
68.3
74.0
61.6
71.0
69.6
64.9
70.9
71.9
72.7
72.5
77.6
77.5
76.6
85.0
79.2
79.5
70.6
75.2
65.5
72.6
72.1
68.2
73.5
73.6
74.6
s10
66.7
70.8
71.3
78.9
71.8
64.5
69.2
60.8
64.9
66.1
68.7
68.4
73.7
81.9
75.3
63.2
68.2
71.3
Table 11: XNLI validation results for all languages. We report the results of different freezing configurations.

---

lang
amharic
arabic
azerbuijani
bengali
burnese
chinese, simplified Rgi /Rg/ Rgs
chinese, traditional
Rg. / Rgz Rgg.
Rg: / Rya Rgc.
Rg: / Rea Rkgu/Ras Rga.
Rg: Rgg Rgg.
Rg: /gg/ Rgs
Ry, / Rgg Rgs.
mmi75
29.7 23.7/134//2.7
17.3/123//94
18.5/68/155
27.6/13.1/255
38.7/1821 30.0
19.47 33.3
39.3 33.6 39.1 33.4
small
13.213/ 12.2
37.//124/294
13.125//1.1
7.6/11774
12.7/20/ 12.1
38.2/ 1.3.1
mm75 mTS"
34//77.//55
42.3/222/3244
261/111/213
388/170//6.1
30.7
29.1
49.07 39.0/30.8449
base
15.0/45/ 13.7
40.2 20.11/155
14.0/4//122
119/34/11.2
15.7/3.7/ 14.5
44.2/ 26.1 /
45.2 27.2//77.1 39.4
lang
english
Irench Rez
gujurati Rgi
hausa Kgg
hindi Rgu Raz
igbu Rgi Rgu RRg
indonesian Rgo Kag 31.4/119/52 17.1/35/158
Rgs/ Kag.
Rg:
Raz
Rgi Rgu
mmT5
38.9 14.4/ 30.4
13.7/12.7/6.77 22.1/44/19 19.4
26.//122/115 8.8 8.5
/2.7/ 16.4/50/ 149
30.6/13.8/2.55 128/32/12.
/7.1/ 22.0/6.1/ 19.1
small
mT5
39.0 14.7/ 30.4
mu:15
19.2/192/44,
37.0 15.2 23.4/5.3/0..
32,0/ 1588/46 24.6
36.7/16.5225
34.9/169/27. 15.0/49/ 13.8
40.0/ 19.1 29.6
36.1/1522/25 20.8/5.5/1.3
mITS
42.6 18.3/333333.5
11.7 1.7//3//08 10.8
20.6/7.1 18.2
30.3/ 10.5 /44
lang
japanese
korean
kyrgyz
marathi
nepalli Rey Rgz Rgg
pashto
persian
Rg / Rgs / Rgg
Rg. / Rez Rge
Rgi Rey / Rgz.
Rg: /eg Rg:
Rei / Rga Rg:
Ry / Rga Rg
mui15
44.9 20../25.66 6 37.0
31.//333/258
7..1001
19.4/82/1599
29.6/13.8/20
16./169//2
/ 29.0
small
mITS$
41.0 2..0/217//33 33.1
8.3/23/7.7
10.3 1./977
11.7/36/ 10.7
8.8/14/86
18.1/3.4 16.2
21.45555 18.7
mmT5
48.2/
39.5
34.4/16.122
26.01 10.72112
25.5/114/203
34.6/170/ 26.9
42.1/19.8/ 30.7 2L1/48/188
41.0/ 19.6 30.6 23.6/6./
base
mil's
44.3/248//
109/43/99
123/24/115
134/5.1/120
120/31/111
lang
portaguese
punjahi Rez
missian Rgg/ Raz
scotiihh gaelic
serbian eyrillic Rg. Rga Raz
serhian, latin Rgi Rga Rgg
sinhala
Rgg / Rga Rgg
Rg. / Rga
Rg, Rga Raz
Rg Rga Rgg
mmi15
33.4/1222/2.22 26.2
14.1/240/242
254/83/20.4
29.3/115/243 1944/9917.3
21.4/65/179 11.1/99 11.0
108/3.1/95
28.6/14.2721
small
m115
23.1 /3.1//22//0 20.5
108/ 10/
12///4/12.
11.5/237 10.4
2.1/1/999
mun15 mTS*
38.31555528.2
17.6/19.1/7.5
20.8/113/237
13.6/13.72233
29.2/ 102/100/20
19.0/54/66.1
18.0/18.8/
base
25.5/655/11
13.7 02//3/1122 12.8
16.2/3.0 14.9
23.2/72/20.1
138/19 13.3
18.8/4.2/ 16.6
10.0 1.0/2//94
lang
somali Rg: / Rg.
spunish
swahili Rgi Rgs/ Rgi
tamil
telugu
thai
turkish
Rg / Rgo Rg.
/Rga /Rg:
Rgy /gg/ Rgg.
Rgi Rgy Rgg.
Rgy / Rgg
mm15
28.8/11./22.5
28.5/83/22.3 18.7/36/16.
308/1155/334 14.5/27/1.4
25.0/118/204
23.2/10.6/ 19.0
26.6/ 12.9//2./2
24.9 10.0//0//00 20.6 16.7/4.1 / 14.8
small
mT5
16.7/26/ 15.1
7.//1/77.
8.4 1./822
7.1/2.1 /66
mmi15
33.6/144/ 25,1
32.7/ /24.
/6.1/15.3 /6.77
29.0/144/230
26.5/12.922 12.9 21.1
30.3/15.3 1..5/3.5
31.3/140025 25.3
mTSS
18.0/36/ 16.2
21.0/46/18. 18.2
16.1/361147
10.5/3.699
1088/9/ 10.0
18.755.7113 16.3
lang
ukrainian Rg: / Rgz Rgg.
undu
uzbok
vistnamese
welsh
yoruba Re Rgu Rg.
are
Rg: / Ryz Rga.
Rg. / Rgy / Rga.
Rg / Rgo Rg:
Rgy / Rgu Rgg
Ry: / Rgu Rgs.
mm71
22.1 18.1/881
34./1555//6.
4.4/99/44 /4.2.
36./169/258
27.6/ 21.6
29.9/12/ 22.9
29.4/126//9 23.3 17.0/4.7/151
small
mITS$
114/12/111
13.5/3.0/22 12.5
9.5/11
16.7/32/15 15.3
18.2/3. 16.7
18.7/3.9/ 16.8
mm75
28.67 22,5
38.1 2.1/184/200
14.7 13.7
42.2/20
26.9 11.37
37.7 16.41 28.0
34.5 16.1 26.8 18.6/6.0/66. 16.7
base
14.0/24/13.1
15.5/3.6/442
11.9/23/113
23.1/64/19.7
22.//48/198
24./63//212
Table XL- Su.em.n.... test set results for all languages. We evaluate using the best performing model on the four source languages on the validation set.

---

lang
azerbaijani
bengali
burmese
chinese simplified
chinese traditional Rgi / /gg Rgz
Rg: Rgg Rgs.
Rgi / Rga Rgc
Rgi Rga Rgz
Rg. / Rgo Rgg.
Rgi Rga / Rgc.
Rgi / Rga Rgz
mm
1.73/17/11.7
169/33/154
13.7/36/118
8../2./7.7 98/047996
17.6/377/55 18.3/0.6 18.0
4.2/1.2337
5.3/20749
mTs*
13.6/ 122
15.2/066 14.9
139/13/127
55/12//53
59/17/55
m5
14./5/2/44
193/45/171
17.5/52/146
15.1 12.7
4.0//9/199
92/22/883
8.1/26/7.5
base
1377/1//131
158/13/15.
153/25/135
10.5/08/ 100
18.1/16//122
67/18166
69/23/66
lang
english
french
gujarati
hindi Rgi.
igbo
indonesian
Rg: Rga
Rgs.
Rgi / Rga Rgc
Ryi / Rga Rgz
Rg, / Rga Rgr
Rgy Rga
Rg / Rga Rgt
Rgi /Rg Rez
man15
41.5/17.9/2.5
27.3/8.3/222
11.8/266 10.3/0.3/ 10.1
10,9
19.8/6./ 17.5
13.8/322/2.7
28.2/9.7 22,9
22./6.1/19.1 18.1/28/ 16.1
mTg*
42.0/177 33.1
21.5/3/118 18.4
18.8/45/ 16.5
13.1/09/ 128
25.6/5.11 /2.1
mm75
45.//117/ 36.5
/5.9/5.6/ 13.7 27.7/8.3/2..0
27.8/100/2.3
169/46/150
25.6/9.0/ 20.8
26.2/82/215 19/2335/188
base
m
44.6/20055 25.513333..
215/38/180
10.9/06/ 10.6
20.2/52/175
14.1/7//134
26.0/5.//199
lang
japunese Rg: Rega Rgs
kyrgyz
topali
pashto
persian
Rgi / Rga Rgg
Rgi / Rgo Rgz
Rg, / Rgg Rgr
Rgy Rga
Rgi
Rgi / Rga Rg:
Rgi / Rgo Rgz
man15
4.0% 1./38/
17.2/48/15.
9.9/17/92
9.1/14/8.5
12.7/36//1.
714/3.1/16.J
18.2/40/166 16.8
mTg
4./1.//4
1.0/
12.7/066 12.3
11.1 /09/ 10.7
10.8/04/ 10.6
145/0.4/14 14.3
16.3/0.7 16.0
mm75
6.1/2.1/56
194/58/16.
13./21//177
11.5/2/105
13./36/122
184/44/162
23.0/68/
buse
m
46/12/44
15.423/144
129/08//22
120/1A/113
118/11/111
160/144/1.2
173/18/163
lang
portuguese
punjabi Rgi / Rga/ Rg.
russion Rgi / Rgu Rgi.
scotiish gaolic Rg: / Rgu Rg.
serbian eyrillic
serbian Jatin
Rg:
Rgg Rg..
Rgi RRz Rgo
Rgi /Rgz/ Rg:
Rgi / Rgy Rg:
small
man15 m
30.4/9.92444
12.5/3.3/00 10.9
10.//30/ 14.3/25/130
8.3/15/777
1L.1/22/100
14.1/3.6/22 12.9
23.5/5.01 20.2
1211/.3/119 11.9
14.6/11/138
21.7/5/21 19.0
13.3/0.7 12.9
17.1/22/ 15.3
10.8/0.// 10.6
mm75 m
30.9/1022224
180/59/148
18.7/38/162
150//03//00
15.5/28/13.9
15.6/35/13.2
17.5/ 14.8
buse
23.6/5.51 19.8
127/08/12.2
14///12/137
/2.7/633 19.2
138/09/13 13.1
173/23/15.1
10.7/ 10.4
lang
somadi
spanish Rgi / Rga Rg.
swahili
telugu
turkish
Rg: Rgu Rg..
Rgi / Rgu Rgi.
Rg: / Rgu Rg.
Ry RRy / Rgo
Rg / Rez Rg:
Rg: / Rgy Rg:
manT5 mT5
2 17.0
21.2/5.6/77.
22.0/5.8/188
13:4/42/120
12.8/3.5/117
13.0/40/116
15.7/4.8 /./4//1/344
small
18.6/2./ 16.2
19.3/3.3/16. 16.9
19.1 16.6
9.5/0.7/92
10.0/0.799
9./1.1/88
16.3/99/14.
mmn15 mT55
22.0/6//17 17.3
25.5/699/200
25.6/8.0220
17.3/4//14 144
15.2/52/13.0
14.3/4.112.
216/711/175
buse
19.6/26 16.7
19./338/174
20.1 143
9./11
L./99
10.0/ 10.0/ 1379.2
17.6/37/15.2
lang
ukrainian Rg: Rgu Rg.
uzhok
vietnamese
welsh /Rg/ /Rg/ Rgo
yoriha
avg
Rg / Rga Rg.
Rgi / Rgu Rgi
Rg / Rgu /gg.
Ry
Rg / Rgz Rg:
Rgi / /Ry Rg:
mm
107/1
166/38/150
86/066/84
29.8/114/2.1 21.2/36/18.1
20.5754/115
28.8 10.0//3.
16.7/46/ 14.4
small
mT5*
13.5/09//29
13.1/0.3/2.9
12.5/06/121
22.2/3.5/199
23.7 20.8
15.5//2/142
mm155
16.9/36/149
176/4.7/153
8.5707/82
3297138/236
22.8/64/1899
24.87 19.8
19.6/611164
buse
mT55
1..1/12/1..1
140/13/13.4
13.//8//12
12.1/48//85
23./44/199
23.2/3.9/ 19.9
16.2/28/45
Table 13: SL test results for all languages, We evaluate using the best performing model on the English language of the validation set.

---

amharic
arabic
azerbaijani
bengali
burmese
chinese simplified
chinese traditionnll
Rgi /ga/
Rgi
(2y/Rgz /Rg Rga.
Rgi / Rgs Rga
Rgy
Rga
Rg. / Rga
Rgi /Rg
Small
man15
s7
12.5/18/119
17./3.3/15.7 15.7
14.0/40//21
8.7/22/7.9
17.8/366/62
3./12/33
54/2.0/49
mTgg
13.6/12/130
15.3/06/ 15.0
14//19/127
95/033/94
18.1705/778
5.6/ 1.2544
5.8/166/5.6
12.6/02/124
150/0.3/14.7
129/147125
9,5/02/9.3
17.8/02/176
5.3/05/5.1
5.0/05/499
Base
min15
67
16.3/5.0/44 14.2
1981477/17
17.7/54/147
15.0/5.0/ 12.5
220/48/195
9.1/2778.1
81/25/73
mITS
81
1587147733
153
1547297138
997067995
17871.7760
6.67 L3763
68722763
13.7/14/13.1
13/16.
16.73.6//4.7
9.9/05/9.6
182/14/174
6./16/662
6./16/55.
english
french
gujarati
hausa
hindi
igbo
indonesian
Rgi Rgg Rgz.
Rgi / Rg: Rgg
Rgi / Rga Rgr.
Rgi / Rga Rg:
Rgy Rea Rgg
Rgi Rgg Rg:
Rgi / Rga Rgs
Small
27.8/8.7/225 11.6/170/326
120/299/111
19.3/6.6 17,2
13.63300/
28.1/9.7/22.9
22.5/660/ 19.1 18.1/28/
17.3/37./33.2
12.1/3.6/889
106/04/10.4
18.3/43/16.3
13.1/0.9//28
25.4/5.022.8
61
45.0//20./350
21.1/366/778
10.6702/ 10.4
20.1/5.5/177
12470.2/12.1
24.5/40/214
18.4/29/1 116.2
Base
mmn15
46.1/218/366
27.9/84/22.2
159/556 13.6
27.5/11//222
166/44/
25.499.//00
26.1/80/ /2.4
mTS
20.7
2141397182
1117087
1937497 16.9
1407
2627547
19277377 16N
87
43.3/ 18.774344
22.7/4./199 19.3
11./07//100
21.71631177
14.27 12271//18 13.6
27.1/6.5/228
20.9 /.3//4/183
jepaness
korcon
kyrgyz Ry / Res.
marathi
nepali
poshto
persian
Rgi Rga/ Rga.
Rgi Rgs/
Rgi /
Ry
Rg.
Rgg Rga
Rg
Re.
Smoll
manTs
37708//3
162145/148
103/16//9
89/16/84
130/38/116
17.7/32/16.1
182/40/10
mTs
sl
4.07 10739
136708713.
125703712.1
1127107 10.8
10.770.57 10.3
14.370.3715
16470.77 I6.T
3.3/04/3.2
13/2/04/129
12.2/03/118
109/0.3/106
10.5 10.4
14.5/0.1/144
159/02/1157
Base
mm/15
67
2.3/
19.75577
13.3/22/120
115/22/ 10.6
14.1/38/12 124
18.0/4./ 15.8
23.0/6.8/ 19.8
mT5
81
4.57 1.5/1114 44,4
154/25/
14.4
128/08/123
12//13//11
115/11/110
15.911/111 15.1
17.4/ 188 164
42/09/4.1
15/2/18/142 14.2
13.5/09/128
119/10/114
119/11/11
16.0/11/153
1///16/
poruguese
punjahi
scottish, gaelic
serhian ayrillic
serbian Jarin
sinhala
Rg:
Rg. Rge.
Rgi / Rga Rg.
Ry / Rea Rgs
Rgi RRy Rg.
Ryj Rga /Rg.
Rg: /Rg Rgg
Ry: / Rgu Rga
Smalt
m5
30.77 12.1/1//10 72.1
10.6
143/28/130
27,6/00./233
83/14/77
109/22/98
139/33/128
m=T3"
12.17037
21AT3.17
13.570.67 13.0
17.2122715.3
s1
24.6/5.5/20.
12.2702/12. 12.0
139/06/13.1
21.6/5./ 18.5
13.00.4/22 12.7
16.2/16/ 14.5
10.370.1 10.2
Bose
mm/T5
31.3/ 24.5
078/58/144
189/39/163
24.8/10.5/20 20.2
15.6728
13.8
15.8/3.5/ 13.6
17.5 17.7/ 14.8
23.7/5.6/199
12.5/0.7/2.0
148/13/137
22.3/633188
13.9/09/
17.3/2.3
10.3
m
67
24.916.5/208
12.1/04/110
15.6/ 14.4
24.6/8.3/20.
142/009 13.5
18.7/28/ 16.4
11.0/0./ 10.8
somali RgL Rga/ Rez
spanish
swahili
tamil Rgc.
telugu Rgg
thai RgL /gg RRz
turkish
Rg. / Rga
Rec
Ry, /
Rgz
Rg. / Rga
Re /
Rg. / Rga / Raz.
Small
mm/3
20.1/49/163
220/59/182
128/36//11
152/48/
mi13"
81
1922757765 16.5
19A734717.
19377477667
957077992
10.7066798
9.27 L.76
163728/143
19.5/27716.
199/35/11.0
18.5/32/162
9.0/02/8.7
9.7/0.2/95
8.9/0478.3
166626/ 14S
Boso
mm15
12.0/65/17.3
256/70/202
25.6/33/202
176/66/14.5
189/55/136
21.6/7//15
s1
19.6786766 16.8
200738/171
203743/172
19710794
103709798
RE07 12792
1767337151
/1.1/3.//1
20./43/1177
218/49/185
10.3/
10.6/09/ 10.2
9.87 L./9000
19.676/116
ukrrannian
urdu
uzhak
vietnamese
welsh
yoriha Rgu Re// Raz
avz
ReL Rga/ Rac.
Rg Rsz
Rez
Re /
/ Rez
Ry, Rea Rgs
Re, / Rga REg
Small
m5nT5
<7
1.799
168/38//1.2 13.1703713.0
8.9 8./06/88.7
116/21
20.6/54/17.6
30.0/ 10.5/24.2 23.87407200
16.7/4.7 14.4
1347097128
12870.7722
21273.771822
21375.3719
14:572271422
s1
13.3706/ 12.6
13.1/0.2/3.0
127704/122
213/33./183 18.1/34/18.1
185
2.9/3.6/21.3
15.4720 14.
Base
mm15
67
17.1/37/11 18.0
176146/ 15.4
86 86/07//81
327/13.//28
22.8166.7 19.1
24.8.18 1888 /2.0
19.7/6.2/ 16.4
14077:27134
13.77087128
16.17287143
57
15.0/1./14 14.0
14.2/ LD/
140/
L.//111333 13.2
24.715.7 13.0/5.4/19.
20.8
25.0/5.//1..
168/330 15.0
Table 14: SL validation results for each of the languages. We report results of different combinations of freezing the model.

---

amharic Rgi / Rgg/g//K Rgg
arabic
azerbaiijani
bengali
burmese
chinese, simplified Rgi Rgy Rea
chinese, traditional
Rgi / Rga Rgg
Rgi / Rez Rgs.
Rg. Reg / Rgt
Rg. Reg Rgg.
Rg / Rga Res.
small
mm75
37.5/ 19.2
36.6/ 16.6/ 28.5
15.4/195/ 23.3/134//25
28.0
43.3/23.1/34.1
36.71 17.9/3177 37.1 18,1 /32.1
37.9/ 19.6 38.1 19.8 32.3
mTS5
37.2/
19.2/ 28.1
36.9/ 16.7 28.9
27.8 13.223..
34.51 188//28
134/234// 34.0
mm15
42.9 /5.1/ 20////5/330 33.0
41.9 21.5/ 3300
336/18../844
40.7 32.4
29.9/29./000 39.6
/ 22.3
43,0 24.0 36.7 40.8/2233/36
buse
mTS
40.1 22.0//2.0/300 30.0
39.7/ 19.4/ 30.9
10.3/153/22.2
/7.//217/2955
4 /25.3 /
39.4//004/34.3
english
french
gujarati Rgj Rgu Rg,
hausa
hindi
igbo
indonesian
Rgi Rga Rg.
Rgy Rgg Rg.
Rg Rg. Rg:
Rg /gg// Rg..
Rgi /gg RRg.
Rg Rg. RRg.
small
mmi1
34.4/1.00 / 30.0
17.31172129 29.5
33.5 18.2/ 26.8
21.0/2.22.
39.1/210//1.. /
41.9/21.2 /3//1 /309
37.7/174/ 29.8 37.7 17.3/ 30.0
mil58
13.6
38.2 29.6
20.//02/31.8
43.1/218/31
mm75
42.4/ 17.7
42.6/ 22,1 333
38.3/ 23.1/
36.4
44.1 26.4
47.2 25.8 34,4
42.5/219 /1.9//4/4
mTS8
40.9/ 16.3/318
40.2/ 194/ 31.1
36.01 20 28.7
226/ /3.8
42.07 24.1 /33.7
45.0 23.6/ 32.9
40.5/ 19.9 32.1
japanese
korean Rgi Rga Rgr.
lengyz
marathi
nepali Rg: /Rgs/ Rgg.
pushto
persian
Rgi Rga / Rgg.
Rgy / Rgy Rgg
Rgy Rgo / Rgr
Rgi / /Rga/ Rgr.
Rgi / Rgi Rgg.
mun15
41.7 22.5
19.1/1999/11
32.6/18.1/66 26.5/11.9/1.2
38.7/22.1/31 39.0/22.3 31.6
/3.3/33.7
small
mIT5
41.7
22.5 33.8
37.4 20.2 32.0
26.8 12.4 21.5
32.7 18.5 / 26.7
12.4 /3.4
42.7 22.9/ / 33.3
base
mmil3
48.5 28.5 40.0 45.4/25.6/36
44.2 / 26.9 38.0
31.3/ /6.5/ 25.2
/3.3/ 30.6
44.4/27.7 36.3
/ 38.4 47.0//2533 336.0
47.0/ 26.4 37.1 45.3/245/355
mil5
39.0/2188
12.//129/
partugnese Rga Rgz
punjahi Rgu / Rga Rgg
scattish, gaelic
serbian cyrillic Rg. Rga Rgg 30.8 / 10.8/
serbian, latin Rgu Rgg Rgg
sinhala
Rgi
Rej / Rga Raz
Rg. Rga Raz.
Ry, Rga Rgz
mmT5
40.5 18.2/ 30.2
/0.6/218/ 30.0
32.9/13.5/55.
36.9 /7.5/ 28.0
30.2/ 10.7 22.6 /5.6/ 28.1
/6.1/218/ 28,4 28.4 35.8/215/
small
40.3 / 30.1
40.1 00.1//1141 29.8
32.2 13.//.3.
36.51 169 27.4
30.6 10.5 23.2
10.5 222.4
mun15
44.2/22.1 /3.3
45.4/27.0/
36.8/17.1/8.6
/3..0
35.0/ 14.7
41.7/2744/355
base
m
42.5 20.3 /
42.7 42.7/24.2/31 31.7
35.0/ 15.6/27..
38.21 18.6 /8.44
33.77 13.0 25.2
32.8/ /2.4/24.1
38.0 /
somali
spanish / Rgg.
swahill /
tamil
tellgu
thai
turkish
Ryi / Rgr.
Rg:
/Rga/ Rgr.
Rgi /Rga// Rgr.
/ Rgg.
mm115
37.3 17.3/173//7.3
31.8/11.2/237
19.//19//30
30.5
31.7 18.0 26.6
28.3/152//3,
31.7/ 11.7/1722/44
33.3/1711 28.0 33.2/ /
small
mIT5
37.5/ 12.0
32.1/11.2/24 24.0
39.2/
18.9 30.3
31.6/ 18.0/66.5 26.5
27.6/14.7/26
31.5/ / 24.9
mmT5
/1.2/210/22 30.3
15.0/ 27.4
23.6/236/344 34,5
38.0 2366 32.2
33.4/198/28 29.4
36.8 2..6/21.//0 29.3
38.4/217 32.6
base
m
39.3 19.1
15.4/
20.5//..//9.0
17.2/247
33.2/ 18.4
/9.7 30.4
ukrainian Rgi / Rga/ Rgs
undu Rgi Rga Rgz
whek
vietnamese
yoruba Rgl /Rg Rgg.
avg
Rgi Rga/ Rgg
Rg. Rga Rgt
Rg. /Rga/ Rgg
Ryi / Rga Rgg.
sanlll
mm:55
31.8 13.4//3./5500
22.3/22./ 33.0
28.5 13.0/ 23,3
44.8 30.8
41.2/20./331 31,1
43.7/21.1 32.0
36.7 18.1 /
mTS*
31.4/ 31.//3.2/248
41.5/22.1/32
28.0/ 13.0 22.9
39.9 19.9/19.4 30.3
43.1 / /31.
36.4/ 17.9 17.9//77.//8..
mm75
36./177.3/2
46.7 46.7/27.8/31
37.3
33377
47.5 25.0 38.6 13.9/23.1/33.
41.6/
base
m
34.1/15.5/2.99
44.9/25.6/35.
10.7/15.0/55.
47.6/25.6/2.7
42.5/2200313.
39.1 20.3 30.5
Table 15: Test results for XL-Sum in the multisource setup. We evaluate using the model which performed best on all the languages in the validation set.

---

lang
ambaric
arabie
azerhaijanl jani
bengali
burnese
chinese simplifted
chinese traditional
Reu / Rez / Res
Rs. RRz Res
Reu Rea/ Rgg.
Rey /
Rez
Rg. Rez Reg
Rey /Rey/ Rez
Rgg Rg Ryg
Small
57
29.1 13//13//22.2
37.7/17.3/296 37.6/17.3/944
199/72/159 12.1
228/128//22
383/17.2/297
38.3/19.1/335 38.5/19.553344
398/214/348
mT5*
13003.1/1..0
74711772
119/197114
39.9/2L6/4.1
Base
T5
61
47
34.0 16.6
9/4/03/9.3
41.2/ 319399993199
424/223//88
263/110/211
12.//1//122
345/180/267
83/04/88
40.1/19.44 30,9
15.1
64.1/5112/00
48.4/996/422
629//98/588
394/813/438
14974.27156
20.720173155
1597447141
1147317106
16374.17
44,31 26.27
10.333/99
20.5/20.1/11.
12.3/42/ 10.9
63/20/59
22.3/ 1999
41.5 1.5//119360 36.0
42.7/42/330 36.8
lang
english
gujarati
huusa
hindi
igbo
indomesian
Rg / Rga/ Rgt.
Rg: /Rg Rg:
Rgi Rego
Rg..
Ry /Rg. Ryi
Rg: RRg
Rg.
Rg. RRga Rgg
Rgy Rea Rg.
Small
TS
67
34.0/1455/00 30.6
354/142//74
271//25/217
284/117/229 1647507149
30.1/134/241
37.7/174//90
314/1L8/25.0
191774.7700
2217747199
9.7716788
12.77327120
19.76.2799.1
17173471577
s1
244/202/35.1
23.2/53/195
86/00484
153/447/100
10./04/10.5
22.9/5.1/20.
179/38/161
Bose
mm15
47
194//94//448
37.3/152/278
12.1/1.1//500
164/276
3488/62212.1
40.3/19.31 30.3
25.8/150/27
42.7 18.7357
23.1734720.
12374,17114
2037687180
1437487138
31.771167254
2097557
419/172/32.
239/64//205
9./29/8.7
119/42/108
134/4.1/126
213/822/12
193/59/170
lang
japanese
korean
kyrgyz
marahii
nepali
pashto
Rgi Rga Rgs
Rg. /Rg. Rgz
Rgi Rgg
Rgg.
Rgi ////////RR
Rg Reg Rgg.
Rgy Rgy Rg:
Rgi Rgg Rgg
Small
m5
47
450//56/37/
311//133/255
193767/ 16.5
192/811/60
30.17443/2477
38.1 /1.0/ 28.7
37.7 /7.7/170/291
mT1""
313721573577
86724778
997T4794
1137347103
K2714779
18.7766776.1
2137566788 18.
46./2700/3...
103/LD/100
109/0.6
90/05/888
9.170.3/9.0
18.7/3.//66
18.87437
Bave
mmnTS
47
49.0/ /40.3
345/1577224
26.1
/22/
26.1/11//207
35.//8.11/22
/ 30.9
/1.2/19//00 3002
mT5
47
45.27 7577577750 90.6
48.1/ 28/6//99.
1107437100
168/59/15.1
9.07 19/85
1347497 120
ILL/37/10. 10.1
12.7757THH
9.8/2999.
19.1/58/168
2L07497
27.3/ 12.2/222.
2367637 193
lang
portugueee
punjabii RR
mission Rei Rez Rgg.
scottish gnelic Rez Rey/ Raz
serbian, eyrillic
serbian, latin
sinhala
Rei Rez/ Rez.
Raz
Ra. RRg Rez.
Re Rgy Raz
Reg ReL
Small
TS
<7
33./125//20 26.4 23375.37700 20.7
13.//33//355
254/8.3/20. 13.77577225
29.67 12.1 12.1/245 2445 18.675.6766
21.7163/ 18.0
1L0/32/95 1137233102
22.0/44//2.2
mT5
51
186/L1/103
113709711 112
8.7/11783
249/67/21.0
1044077/103
13.7/10/100
17.4/44/154
119/08/
149/23/134
8.1/033/8
Base
388/158//84
378/179/267
307/11.1/335
145//44
286/94/221
190/52/160
344/190/255
WTT
51
256768/218
262/77/22.3
1322727/24
85/19/80
129/28/11.7
16.17297448
2000 1./177/175
218/167198
1387177131
106/1A/108 10.2
1847387162
73/18/65
102223797
55/13/52
lang
somali
spanish
swahili
tamil
telugu
thai
turkish
Rgi
Rg: /Res/Re:
Rgi Rgg Rg.
Ry / Rgs/Rs/R
Rga
Re
Rga Rg.
Small
mun
67
293/116/2227
285/882/22
30.0 10.92229
150//200/20
10.6/ 19,2
26./122//20
248/99/
51
168727715.1
1687377168
1467297153
8.77 18778776
837137833
1.712766
1647387147 14.7
mmn5
s1
15.9330/149 14.0
19.7/43/1700
16.4335/14, 14,5
7.5/06/73
83/044/8.1
74/066/70
16.034/144 14,0
Baxe
15.0/55
32./104//40
15,0//5.... 26.4
29.6/
148/112
1688/188/111
30.7/15.7/235
312/1338/2.11
m
s1
18.6338/ 16.4
212/43/183
16.8/4.1/ 15.2
10.633//97
10.8/29 8.7/2.7/80
10.2
109/19/99
18.4544 16.0
47
126/25/115
198//49/7.3
13.8/3.57 12.7
100/39/9.1
1L1/47798
158/67/154
lang
ukrainiaa
unda
uzbek
viernamese
welth Raz.
yorsha
Rgi
Rez.
/Rg/////R Raz
Rgi
Rga
Rgg
Rei /Rge/////
Rgs Raz
Rgs Rgg
Bave Small
<7
224/74/18.3 34.5/15.6/60
4,2/1014.0
366/169/260
278/112//117
10.6/13.//277 13.6/417173
29.4/126/23.
mT55
51
114712/111
1267607126
107157 10.0
1697327152
11.2730716 16.2
1700 LS.
12./10/12.1
135/166/12
10.70057 104
180/33/158
20.1/3.6/ 17.8
20/2/32/183
20.47 68//788
38.5/186/266
152/3.1714.
17.3/
T
67
41
142725/152
120/28/11.0
132//4//199
15.67377143
12.1217113
88/22/82
2L1/73/116
28276331197
2281497199
12.//33/112
26176667223
18.0/5.//144
17.8 65/154
18.77 6.77G8 1G8
Table 16: dn. validation set results for all languages. We report results for the different freezing configurations.

---

alZZA
amLET FT
II.SA
IR. IZ.AZ
bn,BD
ey_CiB
de.
dLCR
27.4
30.5
19.2
11.4
7.8
45.4
28.0
72.8
39.5
5.5
20.9
18.3
11.9
10.0
25.6
42.6
31.4
75,0
<10
58.6
6.9
19.9
19.0
12.4
9.2
47.0
41.5
30.2
72.2
414
58.6
6.6
20.1
15.4
13.6
10.3
46.3
45.5
30111
75.2
25.6
5.5
16.6
12.8
9..8
5.6
55.1
31.0
22.8
71.0
mT5
26.1
23.3
0.5
6.4
16.8
16.9
13.0
10.0
8.6
0.1
3.9
34.6
36.0
32.
30.6
25.0
25.1
72.4
21.3
17.3
714
21.7
27.4
27.0
22.0
$4.7
31.5
20.5
22.2
2006
34.4
54.3
25.4
55.4
30.6
79.2
40.3
36.5
33.7
58.0
$5.0
16.4
73.4
474
$1.6
21.4
33.8
33.7
28.5
25.5
55.3
53.1
401.
72.9
43.5
25.1
12.3
48.1
72.8
mT5
4228
14.7
16.3
27.32
28.5
29.2
27.6
26.5
34.1
11.6
9.5
49.9
51.6
$1.1
$1.5
411.7
10.4
74.0
73.0
43.
19.1
20.8
30.6
28.8
11:7
51.2
30.9
40L5
73.3
43.6
16.4
28.0
28.3
26.9
10.3
51.3
51.1
40.5
74.1
E8
IR
FL
fr. FR
Tr. HU
id. MD,ID
is. 18
29.6
15.9
25.4
31.0
12:7
21.0
10.8
28.5
18.1
29.3
13.0
36
44.0
43.8
10.2
14.1
24.6
33.5
43.1
20.2
30.1
19.6
39.1
26.9
41.6
474
24.1
25.0
14.7
42.6
21.7
31.9
18.6
-01.5
25.8
41.2
30.5
mIS
30.4
57.5
19.4
19.3
32.0
36.5
36.5
15.2
21.9
32.3
13.8
12.3
15.7
16.6
32.1
33.9
15.3
19.4
23.0
36.6
14.2
22.2
14.9
29
16.6
31.7
20.3
22.2
38.2
15.6
22.5
16.0
33.1
16.5
33.0
35.9
32.4
31.3
35.5
28.5
30.5
25.0
42.9
29.3
36.9
$1.9
$1.2
42.6
41.1
37.6
$4.1
41.2
$1.6
43.1
19.7
52.4
39.6
41.2
35.0
51.8
38.2
50.5
50.5
33.1
37.7
50.3
16.9
33.2
27.5
mISS
40.9
40.1
13.0
36.7
49.6
79.9
19.1
50.1
35.7
14.3
35.4
47.6
78.7
48.5
51.0
36.8
40.0
50.7
34.9
33.3
29.8
44.7
29.1
47.9
Ja_JP
Jv_ID
kmLKH
Iv.LM
mIL_MN MMN
23.5
14.4
11.9
16.1
10.3
14.9
17.1
12.7
8.4
25,6
26.7
16.0
16.2
21.0
11.7
16.7
25.9
15.3
9.3
32.6
23.8
20.3
16.2
22.2
13.8
18.0
29.4
14.6
10.55
36.8
18.3
18:1
21.5
18.9
29.0
24.5
0.5
13.0
15.7
10.8
13.5
1542
1.2
25.8
mi
7.1
23.5
7.7
12.3
16.5
10.8
14.2
16.2
10.2
7.0
26.2
24.3
7.3
12.5
10.6
10.5
15.4
16.3
11.3
7.2
28.9
33.1
311.1
33.1
39.8
4..1
36.6
36.1
36.1
29.6
37.8
43.5
33.8
27.8
50.7
$7.0
31.8
30.
S4.1
26.8
36.8
45.3
31.1
25.8
48.1
40.0
39-1
37.01
39022
33.3
42.0
51.3
35.4
33.0
$300
33.8
29.6
33.5
25.8
33.9
44.9
30
26.5
47.4
32.
10.5
23.7
23.1
21.8
25.8
30.1
25.6
16.3
$0.3
mI3
$7.0
16.1
26-7
25.5
29.9
30.0
21.0
36.0
31.1
30.7
30.7
29.0
27.0
17.3
17.5
41.0
41.2
29.
35.8
31.9
18.6
41.1
my MM
ab. NO
NL.
pl. PL
PT
RO
RU
+1. SL.
AL.
SE
25.1
13.0
42.5
42.5
35.4
36.4
36.0
28.5
27.0
46,9
14.4
46.0
45.9
16.
44.3
19.9
36.9
30.6
30.8
46.5
+10
144
42.9
49.5
37.7
43.2
19.5
37.2
30.3
30.1
46.
16.0
45.3
46.3
37.5
44.0
40.8
36.4
30.0
30.1
47.2
149
27-3
38.3
25.3
34.00
35.5
20.1
16.5
18.7
34.2
mIs8
13.0
13.2
20.7
32.0
38.5
17.2
25.6
28.7
35.6
14.2
30.5
20.8
20
29.2
17.4
16.1
18.6
19.1
37.5
36.
31.8
37.1
29.5
40.1
43.9
24.9
48.3
46
453
33.66
56.
$6.1
50.6
$5.3
$6.0
48.4
51
48.5
$8.6
414
23.3
$3.1
$3.5
47.3
$1.2
$1.8
46.8
45.4
42.4
58.9
20.3
49.4
44.1
31.5
491.5
mI53
29.3
90.8
47.7
45.4
50.7
50.5
44.6
45.0
50.0
48.5
47.4
47.4
40.5
39.6
34.0
36
34.1
32.9
$2.6
51.9
230
48.5
30.8
44.1
49.8
48.5
50.6
36.6
25.1
53.2
28.1
30.00
36
IN
dh
Pr_
vi_VN
zh.
sh,TW
1.3.2
13.9
13.8
10.7
24.1
20.4
9.5
16.5
17.5
16.3
20.5
18.4
12.9
33.6
21.9
12.7
15.8
19.1
17.3
27.8
474
18.1
17.9
11.4
34.2
26.7
14.1
19.0
19.9
18.3
28.3
11.4
16.7
10.4
29.6
17.9
10.0
13.6
16.4
1...5
20.9
mT5
K.M
14.2
9.6
S2.1
18.8
SLS
12.4
17.8
15.1
21.6
15.2
10.8
S0U8
17.7
0.9
13.0
18.3
14.2
21.3
15.7
9.8
32.2
18.5
9.1
12.8
15.1
22.1
25.3
25.4
21.7
37.0
33.7
25.4
31.6
25.3
25,4
32.1
34.5
30.4
S0.4
42.6
35.9
38.5
31.6
11.2
45.2
35.0
32.7
28.0
47.9
19
36.3
26.0
41.0
35.2
27.0
39.9
22.1
mIS
17.2
27.0
14.9
26.1
34.9
19.4
79.6
79.8
45.8
36.5
26.2
34.1
20.5
27.9
37.0
<14
17.6
26.6
35.1
42.5
36.4
24.5
3099
25.
21.2
35.4
Table 17: MASSIVE Exact Match (EM) dev accuracies of all models and settings for all languages.

---

lang
english Rg / Rgt
french
gujaruri
hausa
hindi
igho
indonesian Rg: / Rgz
Rg
Rg: / Rga/ Rgg
Rgy / Rgu/ Rgg
Rg / Rg / Rgc
Rg: / Rga / Rgg
Rgy / Re
Rgg
Rg
38.5/ /
31,5
13.5/220/31.2
41.9/ 32.6/66.0
39.0 29.0/31.2/30 35.7
43.0/118/ 34.5
38.0/ 3 33.8 35.6/ 28.1 33.2
66
36. / 28.4/ 33.7
38.0 29 3224
32.0/260033 31.9
39.4/ 30.9 36.1
36.8 29.5 35.3
38.2/28.8 34.6
29.1 33.55
37.9 29.1//933
228/26.5/31.8
40.4/314/37.1
38.1 30.4/ 35.8
41,0/30.8
26.5/ 34,1
s10
28.5 /3244
/2.5/22.5/31.2
39.47 30.7 /33.6
36.91 29,6/
38.5/29.3//38
28.3
614
26.7/28.77 333.3
38.3 29.5//93.
32.8
31.5
40.4/314/
37.8 30.2/35.7
41.3/31.//34 34.9
36.3/ 34,1
38.3729.87312
35.07
26.5799
32.37320734.2
37.77
30.77221
mT5e
56
36.2/ 28. 1 30.0
37.21 28.8 /3.3
/1.9/ 29.2
38.0/29.6/
/ 32.3
36.9 29.4/ 32.5
37.4/28.5/317
27.6 /3.7
26.8/28.5 / 30.7
37.6
28.9 /3.7
28.8 32.3
38.3/
37.01 29.6/ 32.9
38.4/28.7/31
27.5/ 31.0
$10
35.4/27.7 /30.6
36.8 28.6// 30.5
31.8/25.9/29.
37.2/28.9/3..6
36.1 28.8 /
35.6/274432..
34.3/27.0/30 30.9
s14
28.5//8.5/3311
37.6 /8.9/30.
29.5
38.71 30.1 /33.6
37.41 29.8 33.6
38.4/28.91 32.5
35.61 28.1 /31.5
s1
42.6/
/33,5
41.2 /3.5
38.//332//3.2
45.6/ 33.00
44.0 26.2 35.7
46.3/25.0/ 34.5 46.7
42.3/21.5/335
41.9/ 17.0/ 32.7
41.6 20.9 32.4
29.1/239//313
26.1/ /
4 25.7 35.3
41.6/210// 33.2
munTS
67
42.7/
/ 17.9 33.5
10188/333 33.0 /2.6/188/55
39.2/24.031..
26.9/26.2/317
44.0 26.4 35.8
27.3/26.003 35.0
42.6/2200 34,1
$10
41.6/ 16.7 32.44
20.7
18.3/23.031.2
35.66 45.6/
42.9 25.3 34.9
45.6/249/338
41.2/ 20.6 32.9
Base
s14
42.5/ 33.6
/
26.9/ 38.9/23.4/1.5
/.366
44.01 35.7
47.2/25.8/349
42.6/218//44
41.0/ /6.8
40.0 19.1 / 30.9
29.1 36,6/
44.1/ 34.2
42.0 24.0/ 33.7
43,5 /2.//119
40.5
20.0 32.1
39.4 34.0 /
259//2.8//92
41.7/20.7/33
22.6//2235 32.5
22.8/224/3..7
38.8/ 18.1 30.7
67
39.7/
33.7
40.1 19.1 / 3
35.9/ 20.7 28.8
42.8/
41.0 /3.9
15.3/229/113
39.2/ 18.7 3100
s10
39.4/ 14.6 33.6
39.6
18.9 30.5
35.8 /5.8/2100/ 29.3
42.0/210/
40.9
23.0 32.9
43.7
23.7/230/20 32.4
39.0/
18 30.9
s14
40.0/
/3..
39.9 19.2/ 30.8
/ 29.5
43.2/ 22.2/33.66
41.7 23.9/ 33.6
44.3 4.3//334/325
39.5/ /
Table 18: XL-Sum results. Different configurations of freezing.
lang
japuneee Rig Rgz
konean Rg. / Rea / Rgz
kyngz
marathi
nepali
paslito
persian
Rgt/
Rg1 / Rga
Rgg
Ryt Rga / Rgz
Rg; / Rga / Raz
Ryy / Rgg
Rgg.
Ry / Rga /Rgz
42.27 34.3/43333
386/33.2/66
26.7/11.1/238
32.6/265/302
39.6/32.6/ 36.5
44.4/44.1388
48.1/338//3.3
mmn15
56
32.5 / 39.9
37.9/ 38.8
25.6/212//55.
20.6/25.4/9.8
36.8/3.11336 36,9
42.7/33.1
41.4/226/36 36.8
44.55/4.3//
/9.3/340/9922
264/216/254
31.9/263/306
38.3/31.3367
43.7/33.9/88
32.0/32.73311
s10
39.8 32.9 / 39.5
38.33229/
26.3/218/25.2
30.9/25./29.7
37.3/ 30./300.//33 36.3
42.5/33.1 38.0
41.6/
/
Small
s14
41.4/340//4
39.4/440//9.2
26.1/21.//2.4
32.4/2666/0.5
38.3/1.1/37.3
43.4/33.6/
38.5
41.8/32.7/77,
42.97
34.779
38.633.9735.1
264721.4122.1
32.9/2697287
30.0731..6345
44.2/3407322
42.8733573588
mTS3
66
39.0 /3.2
/8.3 / 34.7
24.6/
23.6
31.1/25.7/55,
/1.3/
35.5
/1.8/31.9/ 34.2
30.1/33.1/37,
38.2/33.0/61 36.1
254/2111/237
31.9/262/2.33
37.3/30.//44.
42.4/33.0/35
32.5/325/344
s10
33.2731.
36./3155/34 34.9
23.9/20.0/22 23.9
21.9
22.5/
36.01 29.4/
36.0 40.4/113/
40.4/314/
34.6
s14
40.4/ /38.6
38.3
25.22110/23.7
22.0/26.222 28.8
37.3 30.20///444
/3.5
36.3
41.5/324/3500
s1
47.9/
/3.5
43.0 25.6//6//66 36.7
/4,1/14,9/234
37.1/224/30. 30.2
44.8/28.1/56 36.5
22.557
38.6
47.3/
48.5/ 28.39/9///999
45.0/278//8.
36.2211.5/9. 31.3/163/25.3
44.7/28.4/88
49.0
38.4
46.5/
36.8
mEnTS
67
49.8 29.8/29.8/20 /41.
45.4 28.3 39.2
16,4///4/
37.0/22.5/00 30.6
44.9
28.5 36.7
49.7 28.7 38.9
26.8 37.6
s10
48.0/ /39.5
44.9 64.62776/38.
20.//6.1/25.2
360/216//2.7
44.1127.7/36
48.6/21.511
38.0
25 36.6 47.4/26.7/
Base
$14
49.7/29.8/41.
25.4/28.2/30
39.2
318//677/254
37,2/2255/3.5
25.4/291/37.3
49./284/ 38.5
sl
N6.37 40.37 364379
33.1
18.773.5722.11
28.7
43.776.3751
34.3
27.77337776..
43.712993358
45.0/ 337.2
/0.5/2333/347
28.8 142/142
12.0/173/25
4L1/248/3366
458/246/355
/4.0/ 34,2
mT58
46.0 26.0 37.5
42.0 24.7 36.1
29.4/
23.7
34.8/20.11
28.3
25.925.4// 34.1
/6.2/25.00
35.8
/4.4//34/ 34.6
s10
4 25.6 137.5
40.4
33..3/34.9
14.6114.//399
46.2 25.0//
24.4/23.//4
s14
46.9/
/ /
43.2
/3.1
29.3 14.5/23.
35.2/ 20.8 28.8
42.61
26.1 34.8
46.9 / 25.7 36.3
44.9) 23.9 /35.0
Table 19: Validation set results for XL-Sum in the multisource setup for languages English, French, Gujarati, Hausa, Hindi, Igbo, and Indonesian. We report results for the different freezing contigurations.
lang
portuguese Rga Rg:
punjabi Rg. / Rga / Rgg
ruasian
scottish gaelic Rgi Rgg / Rg:
sesbian eyrillic Rg, / Rga / Rgg
serbian Jatin Rgy / Rga / Rgg.
sinkala
Rgi
Rgi / Rga Rgg
Rgi / Rga / Rg:
40.6 30.4/33.5
/9.7/29.0/30 33.3
/3.1/25.5/88 31.0/23.9/279
37.1/28.1 30.9
30.5/3.11 25.6
30.423.0 26.1
36.4/99.//1.9
mmnT'S
67
56
39.5/ /
38.6/ 29.0
38.8 0888/888/38
37.7/28.2333 33.7
31.7/24.4/88
34.//74//2.2
36.3/28.2/33
29.0
29.4/22.7/233
23.6/19.1/66. 23.5
26.1 26.1/205//7.0
35.2/
38.5/
/3.7
/35.0
s10
38.8/29.0/322
/2.3/28.333.2
31.1/23.//7.0
35.3/27./32.4
29.2/2.5//677
23.5/18.//555
35.5/28.4/34
Smail
s14
/9.4 /3.55
38.3/28.333355
31.0/23.//8.7
35.3/27.6/32.1
292/22.6/279
24.4/19.3/ 26.9
25.0/28.1/34.3
40.4
39.7/29.13113
32.2/248/299
369//799/2.3
20.9122///44
30.2/3300 24.6
35.8/279/300
mT53
66
38.4 28.6 30.6
37.9 30.9
31.1/23.//5.2
35.0/27.6/292
28.7/22.3/22
24.3/19.3/23.1
27.7 /31.2
38.6/ 28.8//1..11
38.3/28.1/00
30.7/23.5/2.6
/4.7/27.1/2 / 29.5
28./2188/ 24.9
25.1
14.//1.6//33
35.6/
28.4 30.9
s10
37.3/28.0/30.
/69/274/31.2
30.//3.5/25.4
26.1//6
/8.0/1..//29
184//88//33
37.9/27.1/31 /31.9
s14
38.8 28.9//8.//31.5
28.3/384/31.5
31.2/23.//2.7
34.9/294/3..
29.2122.3/22 22.3
25.9
20.4/2.44222 24.6
2..//283/31..
44.5/22.4/3355
442//59//3.3
37.3/17.//2888
41.0/ / 30.9
/43/13.5/55
34.8 14./2.1...
40. /5.5
mITTS
66
43.9/21.//2.9
36.2/66./27.9
42.1/23.0/32.2 32,2
35.6
14.6/27.3
/4.5/ /6.4
42.4 28.1 34.7
44.8/22.5/3.77
44.//66//339
/7.1/17.//8.88
/2.1/23.//2.1
364/
15.5
35.9/1521272
42.5
28.5/282/330
s10
43.6/212/32.7
438//556/33.2
3522/15./27.0
41.//3.1/32.44
14./14.1125
336/1388/
336/
25.5
41.8/
2
Base
s14
44.7/66./33.
37.2/17.3/2.7
42.3/234/32.
36.2 15.3/79
/ / 26.9
/
/3,3
42.9/20.772.1
22./2332/333
356675.6126.9
38.21192/2.33
325/12.0/44
13.//228/2..6
34.6/2407
30.4
mT58
56 41.3/ 18.7/0666
4L1/22.5/309
32.9/338/25
37.5 155/188//93
/2.0/22
30.1/113/23.1
38.5/244/313
41.9/
19.5/31.2
41.7//2.6/399
30.9
/
38.9/
19.7 19.7/
12.//223/249
31.//121//2.
24.7/
30.9
$10
41.7/ /9.1
415//230/312
33.1/14.//5.4
38.6/
/9.5
32.7
12.2/249
31.6/117/235
29.3/25.1/113
s14
42.3/ 19.8/315
22.6/336/315
14.//244
39.5/ 20.5/30.. 30.1
33.2/ 12.5/522
32,7 / /
/9.4/25.//1.8
Table 20: Validation set results for XI.-Sum in the multisource setup for languages Portuguese, Punjabi, Russian, Scottish Gaelic, Serbian, and Sinhala. We report results for the different freezing configurations.

---

lang
somali
spanish
swahili
tamil
telugu
thai
turkish
Rg: Rga
Rga
Rg. Rga / Rgs
Rg Rgg/ Rga.
Rg / Rga Rga
Rg Rgg / Rgg
Rg / Rea Rea.
Rgu Rga Rga
s1
38.4/ 30.6
31.3/23.5/2733
38.9/3000/33
318/266//1.2
31.5/248/28 29.0/23.6/755
28.2
/3.2/278/320
$7.0/27.4/ 30.8
22.2/2..2/2.11
17.7/288/33
28.9/422/3.3
26.8/2.1122
31.0/248/20 29.3
30.3/25.3313.
mmn
67
37.5/ /31.2
30.8/23.1/2.6
38.1/29.5/34.7
30.8/26.0/3..2
27.6/22.7/26
31.8/25.3/29.
$1.5/264/32..
s10
36.8/ / 30.2
319//339/267
37.9/289/333
28.8/ 24.0/1..
22.2/22.2/27.3
31.5/25.0 29.3
30.5/2256/31.
Suall
<14
37.6/27.6/310
31.1/23.3/27.5
38.3/29.4/3.3
30.4/25.6/2.1
27.7/22.6/22
31.2/26.1 /32.1 31.8/25.1/296
s1
38.07 29.07290
317725997256
38.772977317
$19726.7728.7
25,72252725.3
21.7724476.22
329721373022
mTgs
66
25.4/26.31 18.4
52.1/25.9/2511
/6.1/27.//3.3
24.7/2550/26.2
26.9/22.3/255 25.0
30.6/24.71 26.7
25.8 22.3
57
25.8/264/ /28.7
30.9/23.1/25
36.6/2811311
29.9/25.1/22 28..5
26.9/219/251
30.9/
00.9/24.5220
20.6/25.8
29.2
s10
25.6/25.8// / 28.7
31.6/23.6/266
35.3/22.0/111 31.5
21.5/22.7/28, 28.3
26.01
2..0/21.//...
20.4/242//7.
29.2/24.51/ 28.88
s14
36.1/262/29..
32.2/23.//2.6
37.1/28.3/3.1
30.2/25.3/29)
/
20.5/24../2.77
30.7 25.8 29.7
41.7/214/30.6
362/1488/7.3
32.3/22.6/33.
33.8/20.0/22 37.0/224/31.2
35.5/20.3//8.2
38.0/2133/22
manTS
56
41.5/2155/06 /30.8
36.0/ 14,5/2../
43.1/23.0/33 33.7
57.0/224/31.
33.9 20.1
/6.4/214//99
37.3/ /3155
21.7/2177/112
16./15.11274
44.2/240/ 34.7
23.9/234/32.
34.41
/
37.4/22.0/ 29.8
21.6 32.4
$10
41.0/20.9// /30.2
35.6/ 14.1/2.1126.77
43/2/22.9/3.6
36.7/ /31,0
29.6 33.31
36.4/212//9.3
36.9/ 20.1 31.0
Boso
$14
41.7
24.//236/34.3
23.1/2.1132.
20.6//8.22 34.41
37.2/2177196
38.1 /
39.67 19:47 29.0
3137/333/256
41.7/211731.7
34.772017250 28.7
13.67 11.717../2531
36.7719577 18.7268
30.2
m755
56
38.5/ 28.2
/ 12.2/25.1
40.4/202/3.3
51.5/17.8/6.2
30.6
17.0/25.0
33.5 18.6/ 26.7
33.9/
17.3 28.3
39.1 19.0
53.7/12.6/25.
40.7
14.1/19.9225 /1.5/
15. 30.81
33.9/
19.1
34.9/
18.31 29.2
s10
38.8/
18.7 28.7
34.1/12.//2.6
40.8
33.5
33.77 1 28.3
30.8
15.4
33.8 18.9 27.0
34.5/ 18.0/28.5 28.8
s14
39.3/
19.1 / 29.0
34.21
41.2/ 128/2566
32.1
20.6//2.//22 29.1
31.51 17.8/
34.7 / 19.6 27.7
35.4/ 18.8 29.7
Table 21: Validation set results for XL-Sum in the multisource setup for languages Somali, Spanish, Swahili, Tamil, Telugu, Thai, and Turkish We report results for the different freezing con configurations.
lang
ukrainian Rgg / Rgs.
undu
wzhek
vietnamese Rgy Rg, / Rgi.
welsh
yoruha
avg
Rg:
Rg: / Rga
Rga
Rg: / Rga / Rgs
Rg / Rgu / Rgg.
Rg: / Rgg / Rgg.
Rg: Rga.
Rg:
32.2/25.3
28.//3.5/226 33.3/33.0/37.3
411/31.0/340 44.8/310/34.44
45.2/33.2/356
3688 18.3/
mim15
60
/0.1/23.6:1
28.6
39.5/30.6/38 36.8
26.8/2211/7.
42.9/299/34/
214//2.3/333
42.9/31.8355
34.8/ 16.5 2
67
20.5/ 29.2
12.00/2//37.4
07.3/224/275
(5.8/30.3/344 34,8
39,5/298/344 34.0
44,0 32.0/ 36.6
35.8/
17.4 28.2
<10
20.1/23.7288
39.8/3009/36.
269//2.2/269
48.2/30.1 34.1
/7.7/28.//85
42.8/31.7/8.5
35.0/ 16.7 22.66
Smail
s14
30.5/23.8/29
40.5//3.5/37.33
26.8/22.0/22
43.7/30.3
39.3/29.9/4.1
43.5/32.3/3622
35.6/
17.2/
31.7/25.0727.
41.63327755.3
28.2/33112444
44.8/308/350
40.47304731.8
44.332..7742
36.67
16.02868
29.7/23.2/21 /
39.4/30.4/30
26.9/22.3/25
42.8/29./313
37.7/28.4300 30.0
41.//31.0/3.5
34.5/
/6.2//22.2
mTSS
29.9/23.3 / 26.0
/9.9/
34.4
27.7/
23.00/24.
43.1/29.6/32,
37.8/28.4
30.5
43.2/31.6/333
34,9
16.5 27.4
s10
29.1/22.9220 16.1
/8.5/29.6/34
26.8 /2..
42.1/29.1/322
/3..2
16.7 /3.9
/3.11/335 40.8
3
15.5 26.4
s14
29.9/ / 26.5
/31.1/11.// / 34.9
27.4/22.4/55 25.2
43.2/29.9
31.7 /2.66 30.7
42.9
02.9/115/33
35.0 16.6/
s1
36.8/ 18.0 19.2
47.0 2
2 37.3
33.0/
16.5/ 26.6
49.1/27.6/ 34,4
44.9 24.8 34.0
48.0 25.4 35.6
41.2/ 32.4
1mm15
28.6
46.4 2
3
48.5/27.0/34.1
43.9/23.1
23.7723./33.
47.8
41.2/ 3.3.6
67
36.8/ 17.9 29.2
33.9/77//1 175
49.1/27.8/4..
/3.0
48.3 /8.3/26.2330
41.9 23.1 / /3.22
$10
35.6/ 16.7 28.1
26.0//268//
36.4
16.8/168/66
48.4/22.0/34.
/ /33,5
25./25.2/35 35.55
40.8
22.1 32.3
Base
<14
17.7
/6.//778/3..
14.0/77//274
49.0/276/34.4
44.0/246/3.1
/6.5/26.4/36 36.2
23.8/23.0/33 33.1
SI
34,77 16.1
45.1725.6735.5
30.77 14.37447744
4738/259735.0
42.422217118
46.77 342
39.2/20.47 20.4 30.6
mT58
32.9/ 14.3/25.8
23.1/23.7/3.7
24.91444/246
46.1/242/31.9
10.2/194/30
45.2/22.9/33
37.9/ /
67
33.3
14.5
43.8/2444/444
30.1
/ 14.5
46.6/247132.
40.5 19.9 33.5
45.1/22.8/33.
38.5 19.8 30.2
$10
33.2/
14.6
43.6 24.2 34.1
15.0/25./25.1
46.3/266/32.2
4 30,9
23.2/33.2/33.5
38.4 / / 30.2
s14
33.9/
15.1 26.5
44.//25.2/49
30.9 15.2/252
47.1/25.213.7
41.2/20.4300 30.7
236//3.0/30 33.5
39.1 20.4 30.7
Table 22: Validation set results for XL-Sum in the multisource setup for languages Ukrainian, Urdu, Uzbek, Viet- namese, Welsh, and Yoruba. We report results for the different freezing configurations.
hn
f
id
ko /EM
T
SW
avg
cfg
Fl /MM
F1
EM
Fl /EM
EM
/
Fl /EM
Fl FM/EM
51.6/ 34,9
30.4/ 16.8
62.7/51.4
46.9/32.1
49.9/33.6
26.6/19.6
46.3/28.6
34.4/
22.5
22.3/29.2
s6
39.4
39.5/248
68.3
51.7/35.2
54.3/40.
15,9/159
58.236..
44./325
43.5/29.6
49./
57
63.2/46.0
40.6/26.5
60.5/60.5
59.3/41.9
57.9/434
30.7
60.3/39.9
36.4
38.1/23.9
30.8/36.3
s10
39.7/56.8
25.7/40.6
69.7
34.7/49.1
40.2/53.6
17.0/269
34.7/57.8
31.5/45.9
28.7/39.5
34.448.77
WISS Small
s14
63.1/47.6
40.5/25.7
71.0/
58.1/40.3
56.1/41.
30.4/18.1
59.3/37.9
40.9/29.9
33.9/22.9
/0.4/36.0
xl
45.6/ 30.2
25.5/
61.7/500
45.1/29.9
/3.7
27.1/20.3
30.8/35.0
33.1/21.8
18.5/13.6
/
47.3/31.6
30.0/
18.6
65.9/54.
514/35.7
52.1/35.2
26.9/18.1
57.1
40.1/28.1
19.0/ 13.8
/3.6
s7
514/31.4
31.3/
18.6
68.4
52./33.5
54.4/36.1
19.2
34.7/35.5
33.4/23.6
20.1 14.6
43.8/30.0
S10
46.5 30.7
30.8/
19.5
65.7
31.2/35.5
50.8 /35.9
/
53.6/38.1
43.1/29.5
19.9
43.7/31.11
s14
50.4 33.7
26.6 15.9
65.6
56.6 /
28.3 18.8
/9.9/40.8
36,2/23.0
17.3/12.4
34.5/31.1
sl 65.8/45.5
$1.3/32.7
73.1/63.4
65.2/50.9
50.2
54.2/44.2
/
61.8/44.3
53.0/37.2
61.1/4.5
/9.2/
37.9/38.1
65.7
7
62.8/62.3
54.6/42.0
67.4/45.7
59.8/52.1
34.7
65.2/514
x7
75.2/
59.7
67.5
73.3/59.0
59.2 77.46616
65.5/
69.6
61.6/45.1
53.2
s10
74.3/57.0
56.9/37.2
/
72.1
/2.6/62.3
56.0/43.8
66.1 /43.7
51.9
/
65.7
Daae
s14
73.7/58.0 58.0
59.2/38.1
67.0
58.6
58.9 /6.5/61
68.1 /5..1
68.5 50.3
61
61.6/45.0
68.5
44.1
41.4/25.7
72.5/61.1
64.1/48.0
70.3
/0.1
58.4336.5
48.2/41.3
$7.8/42. 48.7/39.2
s6
66.5 44.8
42.8/274
74.8/62.7
67./50.6
$3.9/40.9 66.7
49.7 64.6
55.0/42.0
62.3 46.2
47.7
52.7/32.7
74.8/64.
65.5
74.3/589
49,8/39.9
63.6
48.4140.1
56,8/46.2
63.2 46.9
s10
67.5
46.9
4
74.2/63.9
67.0/51.0
73.5/57.7
35.7
64.7
57.9/42.3
53.3/42.6
61.6 45.8
x14
68.3
55.4//8.1 45.8
45.8
75.7/64.1
62.8
75.3/60.4
52.8/40.
63.7 40..6
61.4743.1
55.8/44.7
64.0/ 47.8
Table 23: Results for the validation set of TyDiQA We report results for the different configurations of freezing.

---

100
100
nD
111
lost
200
209
200
1.0
150
150
0.2
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.0
1.0
X082
100
300
19113
1915
191
mi5
$200
Size
2008
200
200
200
1.0
pert
0.2
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
PV115
1115
m15
3605
200
200
200
1.0
350
0.4
0.6
0.0
1.0
0.0
0.2
2.0
0.2
0.4
0.6
0.0
1.0
model
modil
mn:5
250
113
111
250
m1
200
200
200
130
220
50
59
0.0
1.0
0.0
0.2
2.0
0.
0.0
1.0
model
modi
medi
W715
mm:5
250
113
250
250
f13
828
200
200
200
100
100
50-
60-
0.0
0.2
100
Figure 8: Per language perplexity of different model sizes for languages af-de.

---

100
300
250
nD
115
250
F13
lost
boce
200
200
200
100
100
100
0.2
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.0
1.0
100
300
19113
250
1115
250
1915
250
mi5
$200
2888
200
150
139
pert
300
100
0.2
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
02
0.4
0.6
0.8
1.0
PVi15
250
115
250
250
m15
3605
partoott
parstectt
150
15g
150
0.3
0.4
0.0
1.0
0.0
0.2
2.0
0.2
0.4
0.6
0.0
1.0
model
modil
mns5
250
113
253
111
250
m1
150
150
150
50
53
0.4
0.6
0.0
1.0
0.0
0.2
2.0
02
1.0
model
modi
medsi
W115
mm:5
250
113
250
250
f/3
150
253
150
50-
1.0
0.0
1.0
Figure 9: Per language perplexity of different model sizes for languages el-gl.

---

100
100
nD
F15
lost
200
200
200
1.5
150
150
0.2
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.0
1.0
X283
100
300
1915
$520
Size
2008
base
200
200
200
1.0
pert
0.2
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
PVi15
115
m15
3695
200
200
200
pantocte
1.0
350
0.4
0.6
0.0
1.0
0.0
0.2
2.0
0.2
0.4
0.6
1.0
model
moddl
FL115
mns5
250
113
111
250
m1
200
200
200
130
220
50
59
0.0
1.0
0.0
0.2
2.0
0.2
0.6
0.0
10
model
modi
W715
250
113
250
250
200
200
200
mm05
100
50-
60-
1.0
0.0
0.2
1.0
Figure 10: Per language perplexity of different model sizes for languages gu-ja.

---

100
100
n1
F11
200
200
200
1.5
150
150
0.2
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.0
1.0
2283
100
300
19113
1915
195
mi5
$220
Size
2008
200
200
200
1.0
150
pert
0.2
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.8
1.0
PVi15
115
m15
3695
200
pantocte
260
200
1.0
250
0.4
0.0
1.0
0.0
0.2
2.0
0.2
0.4
0.6
0.0
1.0
model
modil
FL115
mn:5
250
1133
111
250
m1
200
200
200
150
220
50
59
0.4
0.6
0.0
1.0
0.0
0.2
2.0
02
0.0
1.0
model
modi
medsi
W715
FL115
mm:5
250
113
250
250
f/3
828
200
200
200
100
100
50-
60-
50
1.0
0.0
0.2
1.0
Figure 11: Per language perplexity of different model sizes for languages ja-mg.

---

100
100
nD
lost
200
209
200
1.5
150
150
0.2
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.0
1.0
100
300
19113
1915
mi5
$200
Size
2008
200
200
200
1.0
pert
0.2
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
PVi15
111
m15
3605
200
200
200
1.0
250
0.4
0.6
0.0
1.0
0.0
0,2
0.4
2.0
0.2
0.4
0.6
1.0
model
modil
250
113
111
250
200
200
200
130
220
mynt5
m15
50
59
0.0
1.0
0.0
0.2
2.0
0.2
0.6
1.0
model
modi
medsi
W715
mm:5
250
113
250
250
f13
200
200
200
100
100
50-
60-
50
1.0
0.0
1.0
Figure 12: Per language perplexity of different model sizes for languages mi-gs.

---

100
100
nD
lost
200
209
200
1.5
150
150
0.2
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.0
1.0
2283
100
300
19113
1915
1915
mi5
$200
Size
2000
200
200
200
1.0
pert
0.2
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
PVi15
111
m15
3695
200
260
200
pantocte
1.0
250
0.4
0.0
1.0
0.0
0.2
2.0
0.2
0.4
0.6
0.0
1.0
moddl
mns5
250
111
250
m1
200
200
200
150
220
1115
59
0.4
0.6
0.0
1.0
0.0
0.2
2.0
0.2
0.6
0.0
100
model
modi
W715
FL115
250
113
250
250
200
200
200
mm15
100
160
50-
60-
1.0
0.0
0.2
100
Figure 13: Per language perplexity of different model sizes for languages pt-sg.

---

100
100
nD
F13
boce
200
209
200
1.5
150
150
0.2
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.5
0.6
1.0
0.5
0.2
0.4
0.6
0.0
1.0
100
300
19113
1915
195
mi5
$200
Size
2008
200
209
200
1.0
pera
0.2
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.5
0.2
0.4
0.6
0.8
1.0
PV115
111
m15
3695
200
200
200
pantoxtv
1.0
250
0.4
0.0
1.0
0.0
0.2
2.0
0.2
0.4
0.6
0.0
1.0
model
modil
250
113
111
250
200
200
200
130
220
mynt5
m15
50
59
0.0
1.0
0.0
0.2
0.6
2.0
0.2
0.6
1.0
model
modi
meds
W715
mm:5
250
113
250
250
f13
E28
200
200
200
100
100
50-
60-
50
0.0
0.8
1.0
Figure 14: Per language perplexity of different model sizes for languages sv-zh.


Reviews:
[{"review_id": "UHkwErM82p", "reviewer": "Reviewer_QXjY", "summary": "", "questions": "1.\tThe number of language-specific modules is fixed to 100 in this paper. What if a new language comes? How does the model generalize to the novel language? \n2.\tIt seems that the total parameter of mmT5-base is 580M+14M*100=1980M, which is larger than other baselines (e.g., XLM-R and mT5-base). Do the improvements merely come from the expansion in size?\n3.\tCompared with encoder-only model XLM-R, mmT5-base is not competitive in both XQuAD and XNLI. Also XLM-R is smaller than mmT5 in size, the authors should explain it more detailedly.\n4.\tAlso, the authors should provide more results on other popular tasks like machine translation.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt gratitude to the reviewers for their thoughtful engagement with our work and their valuable feedback. We have carefully considered each comment and question.\n\nBefore addressing the individual questions, it is essential to emphasize that our primary contribution lies in the freezing strategy applied to modular networks to effectively combat the challenge of generating text in the incorrect language. The pronounced positive impact of this strategy on the modular model, while having a neutral effect on dense models, is a testament to the delicate interplay between these two factors: modularity and training regime of frozen components.\n\nTo your questions and concerns:\n\n> ”One of the main contributions of this paper is the extra language-specific modules. However, such method is too straight-forward, by simply expanding the model size”,\n\nThe concern regarding the simplicity of our approach and the expansion in model size is noted. While it's true that our method involves increasing the model size by incorporating language-specific modules, we argue that simplicity in design is not inherently a reason for rejection. However, our approach is not solely reliant on model size expansion, instead we allocate model parameters in a \"smarter\" way by disentangling language and language-agnostic (shared) information in the model.. The incorporation of language-specific modules enables us to tackle the \"curse of multilinguality,\" a challenge prevalent in existing models.\n\n\n> “It seems that the total parameter of mmT5-base is 580M+14M*100=1980M, which is larger than other baselines (e.g., XLM-R and mT5-base). Do the improvements merely come from the expansion in size?”\n\nWhile it might appear that our model has a larger total number of parameters due to the incorporation of language-specific modules, the crucial aspect lies in the number of per-language trainable parameters, which remains the same between the shared (mT5^s) and modular (mmT5) models. In fact, fine-tuning on a target task results in the same number of trainable parameters as in a dense model, as only the relevant language modules are selected. Moreover, during inference, the modular model maintains efficiency by activating only the necessary language modules, ensuring comparable computational efficiency with a dense model. This architecture choice allows us to strike a balance between language-specific capacity and efficient inference.\n\nFurther, our choice of a modular architecture over a simple extension of dense parameters is rooted in several advantages. Firstly, dense models face challenges when extending to new languages due to limitations in the transformer's capacity. In contrast, modular networks inherently support the incorporation of new languages without compromising performance (see Section 5.2 in [1]). Secondly, the absence of language-specific capacity in dense models can lead to an imbalance where high-resource languages dominate the parameter space, disadvantaging low-resource languages. Our modular approach effectively addresses this concern.\n\n\n> “there lack more encoder-decoder models or decoder-only models for comparison, since they are more popular in multilingual tasks. \n\nOur comparison focuses on the encoder-decoder architecture setting. While encoder-only and decoder-only models are indeed popular in multilingual tasks, it's important to note that these models require significant architectural and training adjustments compared to encoder-decoder models. For instance, when transitioning from our encoder-decoder freezing approach to a decoder-only model, it is nontrivial to define what part of the decoder-only model needs to be frozen given that the input and output for encoder-decoder tasks are passed through the same set of parameters, i.e. there is no distinction between the encoder weights and decoder weights.\n\nGiven the constraints of compute resources and time, we focused on conducting a comprehensive evaluation within the encoder-decoder model family. This allowed us to deeply explore the potential of our approach while providing meaningful insights into its effectiveness.\nFurthermore, the seq2seq task formulation has become a standard approach to address a wide range of natural language processing tasks, including summarization, and question-answering. The standardized nature of the seq2seq formulation makes it an apt choice for our evaluation, aligning with the established practices in the field.\n\n\n> “ the number of language-specific modules is fixed to 100, it seems that it is difficult to generalize to other novel languages.” “The number of language-specific modules is fixed to 100 in this paper. What if a new language comes? How does the model generalize to the novel language?”\n\nWhile it's true that the number of language-specific modules in our mmT5 model is fixed to 100, we would like to emphasize that previous state-of-the-art multilingual models are also trained on a limited set of languages, often around 100. This suggests that a fixed set of languages is a common practice in the field, and our choice aligns with established precedents.\n\nFurther, we would like to point out that our approach is not limited to the fixed set of 100 languages. Recent works such as [X-Mod; 1],[MAD-X; 3], and [4], propose methods to extend both dense and modular models to more languages. However, [X-Mod; 1] empirically demonstrates that fully dense multilingual models can face catastrophic interference between languages, which our modular approach effectively mitigates. This suggests that the modular architecture can be more robust when extending to more languages, without suffering from degradation in performance. The modularity of our approach enables straightforward integration of new languages and opens up avenues for novel research in continued pretraining for new languages. This is an exciting direction that can lead to advancements in multilingual modeling and further enhance the capabilities of our mmT5 architecture.\n\n\n> “Also, the model is not tested on other popular tasks like machine translation, which are also suitable for sequence-to-sequence models.”, “Also, the authors should provide more results on other popular tasks like machine translation.”\n\nThe primary objective of our paper is to address the source language hallucination problem, which is particularly prominent in tasks where the input and output languages are the same. While machine translation is indeed a crucial sequence-to-sequence task, it differs significantly from the tasks we tackle in this paper. Machine translation involves translating text from one language to another, necessitating the handling of different language pairs for input and output. In contrast, our focus is on tasks where the input and output languages are identical, making the hallucination problem more pronounced. While it's plausible that our methods could also be relevant for machine translation tasks, we recognize the need for careful exploration and adaptation of our approach to suit the unique characteristics of machine translation. We acknowledge that machine translation is a valuable research direction and intend to investigate it in future work.\n\n\n> “Compared with encoder-only model XLM-R, mmT5-base is not competitive in both XQuAD and XNLI. Also XLM-R is smaller than mmT5 in size, the authors should explain it more detailedly.”\n\nWe want to emphasize that XLM-R and mmT5-base are not directly comparable due to several significant differences in their architecture and training setup (see section 3.2.4 in [7]).\n- Architecture: XLM-R is an encoder-only model, while mmT5-base is a encoder-decoder model. This architectural distinction alone leads to differences in the models' capabilities and performance on different tasks. For instance, for encoder-decoder models classification (e.g. XNLI) and extractive QA (e.g. XQuAD) tasks are framed as generative seq2seq tasks. On the other hand, encoder-only models cannot perform generation at all. Overall, encoder-decoder models are a lot more versatile.\n- Training Regime: XLM-R is trained for a significantly larger number of update steps compared to mmT5-base. This extended training contributes to differences in the models' learned representations and their subsequent performance.\n- Task Setup: The evaluation tasks in the XQuAD and XNLI benchmarks have distinct characteristics. For instance for XQuAD, XLM-R's performance on extractive question-answering and mmT5's performance on generative question-answering tasks may lead to varying performance levels due to the nature of the tasks themselves.\n\nWe include the scores of XLM-R and other encoder-only models in our paper merely as a means of completeness.\nFurther we want to emphasize that the original (highly impactful) mT5 model does not in fact outperform XLM-R, reasons, among others being the aforementioned points. It's important to note that our proposed approach effectively narrows the performance gap. For instance:\n- In the XQuAD benchmark, XLM-R-large achieves 76.6, mT5-base achieves 67.0, and our mmT5-base achieves 76.3. This indicates that our approach successfully closes the gap by 9.3 points, with only a difference of 0.3 points from XLM-R-Large.\n- Similarly, in the XNLI benchmark, XLM-R-large achieves 79.2, mT5-base achieves 75.4, and our mmT5-base achieves 77.8. This demonstrates a notable narrowing of the performance gap by 2.4 points, with only a difference of 1.4 points from XLM-R.\n\n\n[1] Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022)\n\n[3] MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (Pfeiffer et al., EMNLP 2020)\n\n[4] Efficient Test Time Adapter Ensembling for Low-resource Language Varieties (Wang et al., EMNLP Findings 2021)\n\n[7] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al. Journal of Machine Learning Research 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is well written and the main contributions are clear.\n2.\tThis paper addresses two limitations of current seq-to-seq methods with simple structures.\n", "reasons_to_reject": "1.\tOne of the main contributions of this paper is the extra language-specific modules. However, such method is too straight-forward, by simply expanding the model size. Also, the number of language-specific modules is fixed to 100, it seems that it is difficult to generalize to other novel languages.\n2.\tIn the experiment section, the selected baselines and provided performances are weak: 1) there lack more encoder-decoder models or decoder-only models for comparison, since they are more popular in multilingual tasks. 2) Compared with XLM-R, mmT5 is not competitive (e.g., in XQUAD and XNLI).\n3.\tAlso, the model is not tested on other popular tasks like machine translation, which are also suitable for sequence-to-sequence models.\n", "questions_to_authors": "1.\tThe number of language-specific modules is fixed to 100 in this paper. What if a new language comes? How does the model generalize to the novel language? \n2.\tIt seems that the total parameter of mmT5-base is 580M+14M*100=1980M, which is larger than other baselines (e.g., XLM-R and mT5-base). Do the improvements merely come from the expansion in size?\n3.\tCompared with encoder-only model XLM-R, mmT5-base is not competitive in both XQuAD and XNLI. Also XLM-R is smaller than mmT5 in size, the authors should explain it more detailedly.\n4.\tAlso, the authors should provide more results on other popular tasks like machine translation.", "paper_topic_and_main_contributions": "This paper proposes a modular multilingual sequence-to-sequence model named mmT5, with extra language-specific modules for each language. It is expected to address two limitations of current seq-to-seq models: (1) curse of multilinguality; (2) source language hallucination problem. Experiments on both zero-shot cross-lingual transfer and multilingual training scenarios show the effectiveness of the model.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fipzlPzRJU", "reviewer": "Reviewer_14xE", "summary": "", "questions": "Line 251-253 \"For each dataset, we select the best model checkpoint based on performance on the validation set.\"  The validation set uses examples in the same language as the training dataset or test dataset under the zero-shot cross-lingual transfer setting?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt gratitude to the reviewers for their thoughtful engagement with our work and their valuable feedback. We have carefully considered each comment and question.\n\nBefore addressing the individual questions, it is essential to emphasize that our primary contribution lies in the freezing strategy applied to modular networks to effectively combat the challenge of generating text in the incorrect language. The pronounced positive impact of this strategy on the modular model, while having a neutral effect on dense models, is a testament to the delicate interplay between these two factors: modularity and training regime of frozen components.\n\nTo your questions and concerns:\n\n> “The technical contribution is incremental. The model architecture is very similar to that of \"Multilingual unsupervised neural machine translation with denoising adapters. \" [6]  However, the authors fail to talk about the relations and differences. “\n\nWhile it's true that there are architectural similarities between our work and the referenced study [6], our primary contribution is not centered around introducing a new architecture. Instead, our central focus is on addressing the source language hallucination problem within multilingual pre-trained models, particularly in the context of zero-shot cross-lingual sequence-to-sequence tasks. The core problem we address is fundamentally different from the work referenced. We introduce modular language-specific modules to mitigate the hallucination problem and propose novel strategies for freezing parts of the decoder to enhance zero-shot transfer, while the referenced work focuses on adapting dense multilingual models for zero-shot machine translation.The differences in problem formulation, methodology, and application contexts result in unique challenges and solutions in our work. This combination of modularity and freezing methods sets our contribution apart and provides substantial value to the field.\n\nHowever, we appreciate the reviewer's point regarding discussing the relationships and differences more explicitly in the related work section. We intend to revise and expand our discussion of related work to provide a more comprehensive comparison between our approach and the referenced work. This will enable readers to better understand the distinct contributions and contexts of the two studies.\n\n> “ It is also recommended to compare the performance with \"Multilingual unsupervised neural machine translation with denoising adapters. \"[6] and \"MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer\"[3], which trains adapters on top of a well trained multilingual pretrained model.”\n\nWe want to emphasize that our work is rooted in addressing the source language hallucination problem within multilingual pretrained models, particularly in the context of sequence-to-sequence tasks. [X-Mod; 1] empirically demonstrated the concept of models being \"cursed by multilinguality\" if not pre-trained modularly from the outset. Their experiments revealed that adding adapter modules post-hoc to a non-modular dense model (following the [MAD-X; 3] method) underperforms compared to training a multilingual model modularly from scratch.  Additionally, the findings in [X-Mod; 1] indicated that stacking adapters underperforms fine-tuning the shared parameters. We observed similar trends in our initial experiments with mmT5. This, combined with our compute budget, led us to discontinue pure adapter-based experiments. The decision was driven by a combination of factors, including the modular architecture's effectiveness in tackling hallucination issues and our focus on fine-tuning shared parameters, which we found to yield better results.\n\nFurthermore, we would like to underscore that the setup of machine translation (as in [6]) is significantly different from the context we address in our paper. Our main contribution centers around the source language hallucination problem in natural language generation tasks, where the input and output languages are the same. While the methods we propose may have applicability in machine translation tasks, the distinct task formulations warrant careful considerations and specific experimental setups, which we leave for future work. \n\n> “Line 251-253 \"For each dataset, we select the best model checkpoint based on performance on the validation set.\" The validation set uses examples in the same language as the training dataset or test dataset under the zero-shot cross-lingual transfer setting?”\n\nIn our experimental setup, we adopt a strategy similar to previous studies in the field. For example, for the XNLI dataset, we evaluate our models on the validation set of all target languages and select the best checkpoint based on their averaged performance. For the XQuAD dataset, we evaluate using only the available validation data in English, based on which the best checkpoint is selected. \nOur approach to model checkpoint selection is consistent with strategies used in prior multilingual studies, which helps ensure comparability and consistency with existing research. We believe that this approach reflects a reasonable and widely accepted methodology for evaluating model performance across diverse languages.\n\nHowever, we want to highlight that despite averaging over the validation sets of target languages for some tasks, the checkpoint that consistently yielded the best results across languages was the same as when the checkpoint selection is only conducted on validation sets of the English language. This suggests that, even under the zero-shot cross-lingual transfer setting, the best performance on the English validation set generally aligns with the best performance averaged over all languages.\n\n[1] Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022)\n\n[3] MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (Pfeiffer et al., EMNLP 2020)\n\n[6] Multilingual Unsupervised Neural Machine Translation with Denoising Adapters (Üstün et al., EMNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 The performance is good compared with mT5.\n\n2 The paper is well written.", "reasons_to_reject": "1 The technical contribution is incremental. The model architecture is very similar to that of \"Multilingual unsupervised neural machine translation with denoising adapters. \" However, the authors fail to talk about the relations and differences. However, it is interesting to see the performance of such model on NLU and other NLG tasks.\n\n2 It is also recommended to compare the performance with \"Multilingual unsupervised neural machine translation with denoising adapters. \" and \"MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer\", which trains adapters on top of a well trained multilingual pretrained model.\n\n", "questions_to_authors": "Line 251-253 \"For each dataset, we select the best model checkpoint based on performance on the validation set.\"  The validation set uses examples in the same language as the training dataset or test dataset under the zero-shot cross-lingual transfer setting?", "paper_topic_and_main_contributions": "This paper enhances mT5 with language adapters. It then conducts experiments on zero-shot cross-lingual transfer and multilingual tasks for both NLU and NLG tasks. Experiments show that the model outperforms mT5 and performs very good on resolving the problem of source language hallucination.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "Ey8q55mpzG", "reviewer": "Reviewer_Uj9i", "summary": "", "questions": "A. Why do you think that freezing part of the parameters in the decoder brings gains? If it is because the model does not forget what it has learned during the pre-train process, would methods that mitigate catastrophic forgetting also helps (e.g. EWC)?\n\nB. The added components sounds like a mixture-of-experts model to me, what is the difference between your method and a mixture-of-experts model where each language have its own expert.\n\nC. What are the hyper-parameters for your generation setting? I found that the decoding strategy can have significant influence on the generation quality. I cannot seem to find it in the paper thus I have assigned a lower score on the reproducibility part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt gratitude to the reviewers for their thoughtful engagement with our work and their valuable feedback. We have carefully considered each comment and question.\n\nBefore addressing the individual questions, it is essential to emphasize that our primary contribution lies in the freezing strategy applied to modular networks to effectively combat the challenge of generating text in the incorrect language. The pronounced positive impact of this strategy on the modular model, while having a neutral effect on dense models, is a testament to the delicate interplay between these two factors: modularity and training regime of frozen components.\n\nTo your questions and concerns:\n\n> [1], which is on an encoder-only model to an encoder-decoder model. The main benefits come from the previous method. Although the authors also conducted decoder freezing, the idea of freezing part of the parameter mitigates the incorrect language problem is also studied in [2]. Therefore the paper offers limited contribution comparing to adding the findings of these two paper together.\n\nWe respectfully disagree that our contribution is a mere combination of the findings from these two papers. Our contributions extend beyond the individual concepts discussed in those works and present a unique approach for mitigating the source language hallucination problem within modular multilingual pretrained encoder-decoder models.\n\nRegarding the concept of freezing parts of the model, [2] primarily focuses on prompt tuning and its parameter-efficient approach for freezing parts of the model. However, our work goes beyond by demonstrating a specific freezing strategy that positively affects modular models while having a neutral effect on dense models. Our comprehensive experiments emphasize the importance of freezing the right components of the decoder to achieve optimal zero-shot cross-lingual generation, which aligns with the distinct challenges of our proposed architecture.\nIn particular, our approach pays special attention to achieving near-perfect target language generation, which is a key focus in our XLSUM experiments. Our method substantially enhances the ability of our model to generate text in the correct language under zero-shot settings, as demonstrated in our results. This outcome stands in contrast to the issues highlighted in [2] where they mention in their paper that \"we discover that both MODELTUNING and PROMPTTUNING often partially summarize non English articles into English instead of the target language.\" [2]. \n\nFinally, in the study conducted by [2], it was observed that their parameter-efficient prompt-tuning approach led to a decline in performance in the source language (English) when compared to configurations where all parameters were fine-tuned. In contrast, our proposed freezing strategy yields performance enhancements across all languages, including the source language.\n\n>  I wish the author could expand a bit more on section 6.1, where they studied freezing different parts of the model and provide insights on why does freezing the FFN component of the decoder brings the largest gains\n\nThe choice to freeze the FFN component of the decoder stems from its functional significance within the model architecture. This component houses the majority of parameters and is responsible for transforming the intermediate attention outputs into the final model outputs, often associated with generating language. Given its substantial parameter count, it is plausible that the FFN component encapsulates the model's language generative capabilities.\n\nOur observation of freezing the FFN component bringing the largest gains aligns with our hypothesis that during fine-tuning, models tend to overfit to generating text in the source language they were fine-tuned on. By freezing the FFN component, we effectively prevent the model from over-fitting on generating in the fine-tuning source language, allowing it to retain its capacity for cross-lingual generation. This hypothesis is supported by the fact that only fine-tuning the attention component forces the model to focus more on the input text, aiding in mitigating source language hallucination.\n\nInterestingly, our experiments indicate that this freezing strategy positively affects the modular model while having a neutral effect on the dense model. One potential interpretation of this outcome is that, during fine-tuning, the modular model might learn to disregard the language-specific modules when the FFN component is fine-tuned. This dynamic suggests that the modularity of our approach plays a role in the observed behavior.\n\nThe differential impacts of freezing strategies on modular and dense models point to an intriguing area for future research. Investigating the interplay between the components, the modularity of the architecture, and the specific fine-tuning dynamics could offer deeper insights into the mechanisms that drive our observed improvements.\n\n> A. Why do you think that freezing part of the parameters in the decoder brings gains? If it is because the model does not forget what it has learned during the pre-train process, would methods that mitigate catastrophic forgetting also helps (e.g. EWC)?\n\nYour point is well-taken. The benefits we observe from freezing parts of the decoder could indeed relate to mitigating catastrophic forgetting, as you've rightly indicated. However, we would like to highlight that the intention behind our freezing strategy is more nuanced. Our approach aims to prevent the model from overfitting during zero-shot cross-lingual fine-tuning, particularly to generate text in the target language. While there might be some conceptual overlap between the idea of catastrophic forgetting and overfitting, our strategy focuses on addressing the specific challenges posed by the source language hallucination problem in natural language generation tasks.\nThe suggestion to explore methods like Elastic Weight Consolidation (EWC) is indeed intriguing. EWC, by design, aims to prevent the model from forgetting previously learned tasks. However, it's important to consider the complexity of our setup, where the model would need to manage a multitude of \"tasks\" (i.e., 100s of languages) learned during pre-training. Retaining all aspects of the pre-training task (such as language modeling) might be complicated for EWC and not actually necessary for effective zero-shot cross-lingual generation. The challenge lies in determining which aspects to retain and how to best adapt them to the zero-shot fine-tuning context.\nWe hypothesize that the positive effects of freezing the decoder stem from preventing the model from being biased toward generating in the fine-tuning language. By doing so, we ensure that the model's generative capacity remains versatile and adaptable to multiple languages, reducing the likelihood of hallucinations.\n\n> B. The added components sounds like a mixture-of-experts model to me, what is the difference between your method and a mixture-of-experts model where each language have its own expert.\n\nWe appreciate your insightful comparison between our method and a mixture-of-experts (MoE) model with language-specific experts. While both approaches incorporate the concept of modularization, it is essential to clarify the distinct characteristics that set our method apart:\nThe core distinction between our method and a MoE model lies in the routing strategy (for more details see [5; Section 4]). In our approach, we adopt a fixed routing regime, where each module is associated with a specific language and selected a priori during inference. This fixed selection prevents overfitting issues and provides clear interpretability in selecting the appropriate module for each language.\nIn contrast, MoE models typically involve a learned routing process, where the model dynamically selects modules based on input data. This learned routing approach, while conceptually appealing, presents several challenges highlighted in [5; Section 4.2.1]. These challenges include \"training instabilities,\" \"module collapse,\" and \"overfitting,\" which can lead to suboptimal performance and architectural inefficiencies.\n\nOur fixed routing strategy not only mitigates the challenges associated with learned routing but also offers a significant advantage in terms of interpretability and efficiency. During inference, we already know which modules to activate based on the language being processed. This allows for a streamlined and efficient inference process, as only the relevant modules are engaged.\nIn contrast, a learned routing MoE-style setup could potentially involve the activation of multiple modules for each input, leading to higher computational demands. Additionally, the interpretability of a MoE-style model could be diminished, as the selection of modules would depend on learned factors that might be less transparent than our fixed routing approach.\n\n> C. What are the hyper-parameters for your generation setting? I found that the decoding strategy can have significant influence on the generation quality. I cannot seem to find it in the paper thus I have assigned a lower score on the reproducibility part.\n\nWe perform greedy decoding.\n\n> You might want to cite the following work in your section on similar methods in machine translation:\nShare or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (Zhang et al., ICLR 2021)\nLearning Language-Specific Layers for Multilingual Machine Translation (Pires et al., ACL 2023)\n\nThanks for the pointers, we will make sure to include them in the updated version of our paper.\n\n> Figure 4 would be better if it is a table or a bar chart since there are only two sizes.\n\nThank you for the suggestion, we will change the plot to a bar chart.\n\n\n\n\n\n[1] Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022)\n\n[2] Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (Vu et al., EMNLP 2022)\n\n[5] Modular Deep Learning (Pfeiffer et al., arXiv 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is simple and elegant. The experiment results are much better than the baseline at the lost of a few million parameters which is negligible compare to a few hundred million parameters of the pre-trained model.\n\n2. The authors conducted experiments on a wide range of tasks spanning Question Answering and generation, and conducted detailed ablation studies on which component to freeze during fine-tuning. Their findings provide insight to further research and practical settings on adapting language models to languages with limited task-specific training data.\n\n3. The paper is well written with clear notations and self-contained figures and tables.", "reasons_to_reject": "Limited Novelty: the paper extends the method in [1], which is on an encoder-only model to an encoder-decoder model. The main benefits come from the previous method. Although the authors also conducted decoder freezing, the idea of freezing part of the parameter mitigates the incorrect language problem is also studied in [2]. Therefore the paper offers limited contribution comparing to adding the findings of these two paper together. I wish the author could expand a bit more on section 6.1, where they studied freezing different parts of the model and provide insights on **why** does freezing the FFN component of the decoder brings the largest gains. \n\n[1] Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022)\n\n[2] Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (Vu et al., EMNLP 2022)", "questions_to_authors": "A. Why do you think that freezing part of the parameters in the decoder brings gains? If it is because the model does not forget what it has learned during the pre-train process, would methods that mitigate catastrophic forgetting also helps (e.g. EWC)?\n\nB. The added components sounds like a mixture-of-experts model to me, what is the difference between your method and a mixture-of-experts model where each language have its own expert.\n\nC. What are the hyper-parameters for your generation setting? I found that the decoding strategy can have significant influence on the generation quality. I cannot seem to find it in the paper thus I have assigned a lower score on the reproducibility part.", "paper_topic_and_main_contributions": "This paper proposes mmT5: adding language specific components to multilingual language pre-training and freezing the decoder in cross-lingual transfer fine-tuning. Comprehensive experiments show that their method outperform standard training by a large margin in classification tasks. Moreover, their method solves the problem of the model generating incorrect language in generation tasks for small models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 4 would be better if it is a table or a bar chart since there are only two sizes."}, "extra_text": {"Missing_References": "You might want to cite the following work in your section on similar methods in machine translation: \n\nShare or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (Zhang et al., ICLR 2021)\n\nLearning Language-Specific Layers for Multilingual Machine Translation (Pires et al., ACL 2023)"}}]

Author rebuttal:

Meta-review: This paper explores increasing the language coverage in multi-lingual seq2seq models. The authors propose a modular approach to handle this issue. 

Overall, the paper is sound: “The performance of the proposed approaches outperform its most similar precursor mT5” , and “The paper is well written”.  However, the paper could not inspire excitement. All reviewers agree that the paper is not presenting a strong contribution to the research community and mostly incremental. That said, the presented research is worth to be known by the community, as it could be useful for similar experiments.
Decision: Accept-Findings