Venue: NAACL 2022

# Lifting the Curse of Multilinguality

by Pre-training Modular Transformers

Jonas Pfeiffer\({}^{*1,2,3}\), Naman Goyal\({}^{3}\), Xi Victoria Lin\({}^{3}\), Xian Li\({}^{3}\),

**James Cross\({}^{3}\), Sebastian Riedel\({}^{3}\), Mikel Artetxe\({}^{3}\) \({}^{1}\)New York University, \({}^{2}\)TU Darmstadt, \({}^{3}\)Meta AI**

Work done while interning at Meta AI._Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479 - 3495 July 10-15, 2022 ©2022 Association for Computational Linguistics

###### Abstract

Multilingual pre-trained models are known to suffer from _the curse of multilinguality_, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our **Cross-**lingual **Modular** (**X-****Mod**) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.

## 1 Introduction

Recent work on multilingual NLP has focused on pre-training transformer-based models Vaswani et al. (2017) on concatenated corpora of a large number of languages Devlin et al. (2019); Conneau et al. (2020). These multilingual models have been shown to work surprisingly well in cross-lingual settings, despite the fact that they do not rely on direct cross-lingual supervision (e.g., parallel data or translation dictionaries; Pires et al. (2019); Wu and Dredze (2019); Artetxe et al. (2020); Hu et al. (2020); K et al. (2020); Rust et al. (2021).

However, recent work has uncovered fundamental limitations of multilingual transformers. Conneau et al. (2020) observe that pre-training a model with a fixed capacity on an increasing amount of languages only improves its cross-lingual performance up to a certain point, after which performance drops can be measured--a phenomenon known as _the curse of multilingquality_ (Figure 2). As such, prior work had to find a trade-off between supporting more languages and obtaining better performance on a smaller set of languages.

In this work, we address this problem by introducing language-specific, modular components during pre-training (Figure 1). Our **Cross-**lingual, **Modular** (**X-****Mod**) language model shares the majority of the transformer parameters between all pre-training languages, while providing each language with individual capacity to learn idiosyncratic information without increasing the total number of trainable parameters per language. While previous adapter-based approaches (Figure 2(a)) extend pretrained multilingual language models (LMs) with modular components _after_ pre-training, we add modular components _during_ pre-training, thereby

Figure 1: A transformer layer of our proposed modular architecture. The dark blue and green components illustrate the modular layers, which are language specific. The Multi-Head Attention and Feed-Forward components are shared by all languages.



preparing the model to be extended to new languages post-hoc. Our experiments on natural language inference (NLI), named entity recognition (NER), and question answering (QA) demonstrate that our modular architecture not only is effective at mitigating interference between languages, but also achieves positive transfer, resulting in improved monolingual and cross-lingual performance. In addition, we show that X-Mod can be extended to unseen languages, with no measurable drop in performance, by learning its corresponding modules and leaving the shared parameters frozen. All in all, we propose a multilingual architecture that can scale to a large number of languages without any loss in performance, and can be further extended to new languages after pre-training.1

Footnote 1: Code and pre-trained models are available at: [https://github.com/pytorch/fairseq/tree/main/examples/xmod](https://github.com/pytorch/fairseq/tree/main/examples/xmod).

## 2 Background and related work

We provide a background on multilingual and modular language modelling, as well as approaches that extend LMs to new languages.

### Multilingual transformers

Recent LMs Devlin et al. (2019); Conneau et al. (2020), based on transformer architectures Vaswani et al. (2017) and pre-trained on massive amounts of multilingual data, have surpassed (static) cross-lingual word embedding spaces Ruder et al. (2019); Glavas et al. (2019) for cross-lingual transfer in NLP Pires et al. (2019); Wu and Dredze (2019); Wu et al. (2020); Hu et al. (2020); K et al. (2020). Transformer-based models are **1)** pre-trained on textual corpora using Masked Language Modelling (MLM). They are then **2)** fine-tuned on labelled data of a downstream task in a _source_ language and **3)** directly applied to perform inference in a _target_ language Hu et al. (2020).

### Modular language models

Modular approaches have a long standing history in NLP, preceding pre-trained models Andreas et al. (2016). They have recently re-gained interest for transformer-based models, where mix

Figure 3: Our proposed architecture in comparison to adapter-based approaches. (a) Previous approaches © utilize non-modular pre-trained transformer models and ��⃃



ture of experts (MoE; Shazeer et al., 2017) approaches have enabled training trillion parameters models in a distributed fashion Fedus et al. (2021). More recently modular MoE approaches have been shown to improve domain-specific pretraining of LMs Gururangan et al. (2021). In a similar trend, 'expert' modules have been added to (non-modular) pre-trained LMs post-hoc, predominantly referred to as adapters Rebuffi et al. (2017, 2018); Houlsby et al. (2019). Next to being extremely parameter Houlsby et al. (2019); Mahabadi et al. (2021); He et al. (2022) and training efficient Pfeiffer et al. (2020a); Ruckle et al. (2021), these modular approaches allow models to be extended to new data settings Chen et al. (2019); Ruckle et al. (2020), where newly learned knowledge can be combined Stickland and Murray (2019); Wang et al. (2021); Pfeiffer et al. (2021); Lauscher et al. (2020); Mahabadi et al. (2021); Poth et al. (2021), or stacked for combinatory cross-lingual Pfeiffer et al. (2020b, 2021); Ustun et al. (2020); Vidoni et al. (2020); Ansell et al. (2021); Wang et al. (2021) as well as NMT scenarios Bapna and Firat (2019); Philip et al. (2020); Chronopoulou et al. (2020); Le et al. (2021); Ustun et al. (2021); Stickland et al. (2021); Garcia et al. (2021).

### Weaknesses, improvements, and extensions of language models

Next to the _curse of multilinguality_, recent works have shown substantially reduced cross-lingual and monolingual abilities of models for low-resource languages with smaller pre-training data Wu and Dredze (2020); Hu et al. (2020); Lauscher et al. (2020); Artetxe et al. (2020); Pfeiffer et al. (2020b, 2021); Chau et al. (2020); Ponti et al. (2020).

K et al. (2020); Artetxe et al. (2020) show that a shared vocabulary is not necessary for cross-lingual transfer. Chung et al. (2021) demonstrate that decoupling the input embeddings from the prediction head improves the performance on a number of downstream tasks. Dufter and Schutze (2020) show that the number of parameters and training duration is interlinked with the model's multilingual capability. Chung et al. (2020); Rust et al. (2021) show that the tokenizer plays an important role in the per-language downstream task performance, which Clark et al. (2022); Xue et al. (2022); Tay et al. (2021) take to the extreme by proposing tokenizer-free approaches.

To extend a monolingual LM to other languages, Artetxe et al. (2020) train a new embedding layer with a corresponding target-language tokenizer, while freezing the pre-trained transformer weights. Tran (2020) extend a monolingual model to new languages using bilingual corpora. Wang et al. (2020); Chau et al. (2020) extend the vocabulary of multilingual models with a small number of target-language tokens, to improve the performance in the target language. Muller et al. (2021) propose a transliteration based approach, Vernikos and Popescu-Belis (2021) propose subword mappings, and Pfeiffer et al. (2020b, 2021); Vidoni et al. (2020); Ansell et al. (2021) propose adapter-based approaches to extend multilingual models to unseen languages.

While these approaches achieve considerable performance gains over unseen languages, they are outperformed by standard full fine-tuning methods for seen languages. One can further argue that, as the pre-trained models have already been cursed by multilinguality, the adapter-based approaches build upon sub-optimal parameter initializations.2 In our work, we consequently aim to **1)** modularize the model from the start to prepare the model to be **2)** extendable to new languages post-hoc.

Footnote 2: We investigate this claim further in §6.2.

## 3 Proposed approach

We propose X-Mod, a modular multilingual architecture that combines shared and language-specific parameters. In contrast to prior work, we pre-train modular models from the get-go. Our models can be extended to new languages after pre-training, and used for cross-lingual transfer learning in downstream tasks.

Architecture.As illustrated in Figure 1, we extend the transformer-based architecture from mBERT Devlin et al. (2019) and XLM-R Conneau et al. (2020) by incorporating language-specific modules--bottleneck feed-forward layers-- at every transformer layer. We learn a separate module for each language, whereas the attention and feed-forward components are shared. While the total number of parameters of the model grows linearly with the number of languages, the training and inference cost does not increase (as measured in FLOPs), as only the module in the relevant language is used for each input. Inspired by the adapter3 architecture of Pfeiffer et al. (2021) we 

place our 'modules' after the LayerNorm of the feed-forward transformer block, and the residual connection is placed after the LayerNorm;4 the LayerNorm before and after the modular component is shared.5

Footnote 3: We find that the residual connection proposed by Pfeiffer et al. (2021) results in training instabilities when trained together with the transformer weights.

**Pre-training procedure.** Similar to Conneau et al. (2020), we pre-train our model on MLM on combined monolingual corpora in multiple languages. Examples of each language are passed through the shared embedding matrix as well as the multihead attention and feed-forward components at each layer. As each layer contains a language-specific modular component, the examples are routed through the respective designated modular bottleneck layer. Given that each example only requires access to a single module, modules can be efficiently stored on only a subset of GPUs in distributed training.

**Extending to new languages.** The modular design of our model allows us to extend it to new languages after pre-training. To that end, we learn new embeddings and adapter modules for the target language through MLM, while the rest of the components are frozen.6 Consequently, we are able to extend the model to a new language by learning a small number of new parameters, without affecting performance in the set of pre-trained languages. Following Pfeiffer et al. (2021), we learn a new subword vocabulary for the added languages, and initialize the embeddings of lexically overlapping tokens from the original embedding matrix.

Footnote 6: Preliminary results showed that sharing the LayerNorm results in better cross-lingual transfer performance.

**Fine-tuning on downstream tasks.** To transfer the models to cross-lingual downstream tasks, we fine-tune the shared weights only on the source language data, while keeping the modular components and the embedding layer frozen. We follow the standard fine-tuning procedure of adding a prediction head on top of the CLS token. We then replace the source language modules (as well as embedding layer for _added_ languages) with the target language parameters, passing the text of the target language through the model.7

Footnote 6: Following Artetxe et al. (2020) we train positional embeddings.

## 4 Experimental design

We detail the baseline and models (SS4.1), and their training (SS4.2) and evaluation settings (SS4.3).

### Model variants

We pre-train separate models for all combinations along the following axes:

**X-Mod vs. shared.** To evaluate the effectiveness of our X-Mod model, we aim to compare ourselves to a conventional non-modular architecture. However, simply removing the modular component would be unfair, as the number of FLOPs and trainable parameters per language would not be the same--both in terms of pre-training, as well as fine-tuning. Consequently, for our base-line model--where all parameters should be _fully_ shared between all languages--we include a single bottleneck layer right after the Feed-Forward component. Effectively, this is the same architecture as our X-Mod model, just with a single module that is shared by all languages. We refer to this as the shared model throughout this paper.8 To extend the shared model to unseen languages, we follow Artetxe et al. (2020) and only learn a new embedding layer, freezing the transformer parameters. To fine-tune the shared model on a downstream task, we freeze the embedding layer, as well as the (single) module, thereby fine-tuning an equal amount of parameters on the downstream task as the X-Mod model.9

Footnote 8: Extending the total number of shared parameters would be unfair, as X-Mod and shared would not have the same FLOPs nor the same number of trainable parameters when fine-tuning.

Footnote 9: Adapter-based approach such as MAD-X Pfeiffer et al., 2020b) would be an alternative. However, this would require training on languages twice—once during pre-training, and once when adding adapters—which is not directly comparable to X-Mod. Nonetheless, we report results in §6.2.

**13 vs. 30 vs. 60 vs. 75 languages.** So as to understand how each approach is affected by the curse of multilinguality, we pre-train the X-Mod and shared models on 4 increasing sets of languages. We start with an _initial_ set of 13 typologically diverse languages that we evaluate on, and add additional languages for larger sets of 30, 60, and 75 languages. In addition, we keep a set of 7 held-out languages that we extend the pre-trained models to. Table 1 lists the specific languages in each 

group. The selection and split of _initial_ as well as _added_ languages is motivated by typological and geographical diversity, as well as the availability of downstream task evaluation data.

**Controlling for total vs. per-language updates.**Conneau et al. (2020) investigated the effect of adding more languages during pre-training, while training on an equal number of update steps. However, increasing the number of languages while keeping the number of updates constant results in the model seeing less data in each individual language. As such, it remains unclear if the curse of multilinguality happens because of negative interference, or simply because the number of updates for each specific language is smaller. So as to understand this, we compare (1) training on an equal number of _update steps_ and (2) training on an equal number of _seen examples_ per language. We start with the set of 13 languages (Table 1) and train the respective models for 125k update steps. When adding more languages, we compare (1) training models on each set of languages for 125k update steps, and (2) increasing the number of update steps such that the models are trained on the same number of examples in each of the initial 13 languages. For the latter, this amounts to training for 195k, 265k and 269k update steps, respectively.

### Training details

**Data and hyperparameters.**We sample languages with \(\alpha=0.7\) and train our models with a batch size of 2048 across 64 V100 GPUs on the CC100 dataset Conneau et al. (2020) using fairseq Ott et al. (2019). All our models extend the _base_ transformer architecture, with 12 layers and 768 dimensions. Modules are implemented with a bottleneck size of 384. The shared transformer weights account for 270M parameters, whereas each individual module accounts for 7M parameters. We train our models with a linear learning rate decay peaking at \(7{\rm e}{-}4\) during pre-training and \(1{\rm e}{-}4\) when adding languages.

**Vocabulary.**As we aim to identify the impact of _modularity_ on the curse of multilinguality, we control for consistent tokenization across the different axes. We therefore tokenize using the XLM-R vocabulary for all our pre-training experiments.10 However, for languages added post-hoc, we learn a _new_ SentencePiece tokenizer for each of the target language,11 as the languages potentially use scripts unseen by the original tokenizer.

Footnote 10: Rust et al. (2021) have previously demonstrated the impact of the multilingual tokenizer on the downstream task performance: languages underrepresented in the sub-word vocabulary exhibit considerable performance drops when compared to vocabularies dedicated to the respective language.

Footnote 11: We train the new tokenizers for a vocabulary size of 30k.

### Evaluation

We conduct experiments on NLI, NER, and QA. In all cases, we fine-tune the model on English and measure the zero-shot transfer performance in other languages. For NLI we train on MultiNLI Williams et al. (2018) and evaluate on XLNI Conneau et al. (2018). For QA, we train on SQuAD Rajpurkar et al. (2016) and evaluate on XQuAD Artetxe et al. (2020) and MLQA Lewis et al. (2020). For NER, we use WikiANN Pan et al. (2017); Rahimi et al. (2019). We experiment with learning rates \(1{\rm e}{-}4\), \(3{\rm e}{-}4\), and \(5{\rm e}{-}4\) and train for 3 or 5 epochs for QA and 5 or 10 epochs for NER and NLI. For NER and NLI we take the hyperparameter setting performing best on the development sets, averaged across the pre-trained languages (Table 1). For SQuAD we take the best performing checkpoint evaluated on the English development set, and report the cross-lingual test set results.12 All results are averaged across 5 random seed runs.

Footnote 12: In contrast to NER and NLI, the cross-lingual evaluation benchmarks of SQuAD do not provide a development set for each target language on the basis of which the best checkpoint can be selected. Consequently, we select the checkpoint based

\begin{table}
\begin{tabular}{c l l} \hline \hline Pre-trained & \multirow{2}{*}{
\begin{tabular}{l} 13-LANGS \\ languages \\ \end{tabular} } & _en_, **ar**, **fr**, **hi**, **ko**, **ru**, **th**, **y**, **ti**, **iq**, **skv**, ka \\ \cline{2-3}  & 30-LANGS & 13-LANGS + cs, eu, hr, hu, hy, it, tl, ml, mn, ms, pl, ro, si, sk, sq, sv, tI \\ \cline{2-3}  & 60-LANGS & 30-LANGS + af, am, be, bn, ca, cy, da, ee, ef, fa, ga, gl, gu, ha, is, ku, la, Iv, mk, ne, nl, no, ps, pt, sa, sd, sl, so, sr, te \\ \cline{2-3}  & 75-LANGS & 60-LANGS + as, br, bs, fy, gd, jv, kn, mg, mr, om, or, pa, su, sh, yi, \\ \hline \multicolumn{2}{c}{Added languages} & **bg**, **de**, **el**, **es**, **tr**, **z****h**, \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Selection of languages. We pre-train different models on 4 sets of languages, and further extend them to a set of held-out languages post-hoc. We evaluate on XXLI (languages in bold), NER (underlined languages) and XQuAD/MLQA (languages in italic). For more details about the language selection, see Appendix C.

 



## 5 Results and discussion

We present results for pre-trained languages in SS5.1 and added languages in SS5.2.

### Pre-trained languages

In Figure 4 we plot downstream task results of models pre-trained on different amounts of languages. Table 2 reports the individual language performance for the models trained on 60 languages.

**The Curse of Multilinguality.**Conneau et al. (2020) showed that multilingual LMs trained on _increasing_ amounts of languages, while _maintaining_ the number of update steps, exhibit drops in downstream task XNLI performance. We reproduce these results, both in terms of language modelling perplexity (Figure 1(a)),13 as well as downstream task performance on XNLI _and_ NER (Figure 3(a)). We further find that the curse of multilinguality does not _only_ happen _because_ the total number of update steps per language decreases, but _also_ when all shared models are trained on the _same_ number of examples per language (Figure 3(b)). This confirms that fully shared architectures suffer from negative interference.

Footnote 13: For per-language perplexity see Appendix A.

**Lifting the Curse.** While for the shared model we witness negative interference between languages in terms of perplexity, the X-Mod model is able to _maintain_ performance, and even improves for a subset of languages. We observe similar patterns in the downstream task performance: In both our experimental setups--(1) we control for the number of update steps (Figure 3(a)); (2) we control for the number of per-language seen examples (Figure 3(b))--our X-Mod model--in contrast to the shared model--is able to maintain, or

Figure 4: Test set results on XNLI (top) and NER (bottom) for models trained on different numbers of languages. _Source Language (English)_ only includes scores of the source language. _Average Pre-Trained Languages_ includes all evaluation languages that the model was pre-trained on. _Average Added Languages_ includes all languages that were added to the model after pre-training. Scores are averaged across all languages and random seeds.



even outperform model variants trained on less languages. These results demonstrate that the added per-language capacity is sufficient for the model to adequately represent all languages.

Surprisingly, X-Mod not only maintains performance, but actually slightly improves while we increase the number of languages we pre-train on. This is even the case for settings where the model sees _less_ examples in the target language. This suggests that increasing the language diversity can have a positive impact on the model's cross-lingual representation capability.

**X-Mod vs shared.** Overall, the X-Mod model pre-trained on 60 languages achieves the best cross-lingual performance.14 Our results on XNLI, NER, MLQA, and XQuAD in Table 2 demonstrate consistent performance gains over the shared model for every task and across (almost) all high- as well as low-resource languages.

Footnote 14: We find that the X-Mod model trained on 75 languages is less stable than the versions trained on less languages. We think that this can be attributed to the 15 added languages being extremely low resource—we only train for an additional 4k update steps—resulting in the respective randomly initialized modules being updated very infrequently. This variance could potentially be mitigated by training for longer.

### Extending to unseen languages

We further evaluate the cross-lingual performance of languages added in the second step; (1) on the architectural side--comparing the shared with the X-Mod modelling variant--and (2) by comparing the performance when _pre-training_ on the language, vs. when _adding_ the language post-hoc.

**Modular vs Shared.** We evaluate if the additional per-language capacity improves the extendability of the X-Mod model. On the right in Figure 3(a) we plot the results for added languages on XNLI (top) and NER (bottom). Similarly, we plot the results for the models where we control for the number of seen examples per target language in Figure 3(b). We find that the X-Mod model consistently outperforms the shared model, with a peak performance when pre-training on 60 languages, demonstrating that the language specific capacity is beneficial for adding new languages post-hoc. We report results for the 60 language versions in Table 3, demonstrating the consistent advantage of the X-Mod over the shared model.

**Pre-training vs Adding Languages.** To evaluate if there is a measurable difference on downstream performance for languages that we _pre-train_ on vs. those we _add post-hoc_, we train 2 models on _different_ initial sets of languages, adding the respectively missing ones in the second step. So as to understand if the typological similarity of languages has impact on the downstream task performance, we split the _initial_ and _added_ languages (Table 1) of our previous experiments into two parts. The _first_ split consists of languages where the model was pre-trained on at least one language of the same language family (e.g. English vs. German). The _second_ split consists of languages that are part of a **unique** language family, i.e. the model was **not**

\begin{table}
\begin{tabular}{c c|c c c c c c c c c c c|c} \hline \hline  & & en & ar & fr & hi & ko & ru & th & vi & ta & id & fi & sw & ka & avg \\ \hline \multirow{2}{*}{NER} & X-Mod & 81.4 & **78.9** & **77.2** & **70.1** & **53.0** & **59.1** & 2.8 & **66.2** & 51.1 & 50.5 & **78.6** & **73.4** & **67.3** & **62.8** \\  & shared & **81.5** & 74.1 & 74.7 & 64.4 & 46.0 & 58.3 & **4.0** & 63.7 & **52.5** & **51.5** & 74.4 & 57.2 & 61.5 & 58.8 \\ \hline \multirow{2}{*}{XNLI} & X-Mod & **84.4** & **71.2** & **77.6** & **68.3** & - & **74.1** & **71.7** & **73.4** & - & - & - & **66.9** & - & **73.5** \\  & shared & 82.8 & 69.2 & 75.6 & 66.6 & - & 73.2 & 68.5 & 72.5 & - & - & - & 62.1 & - & 72.5 \\ \hline \multirow{2}{*}{XQuAD} & X-Mod & **85.1** & **68.1** & - & **67.5** & - & **75.0** & **66.3** & **74.9** & - & - & - & - & - & **72.8** \\  & shared & 83.8 & 64.6 & - & 65.8 & - & 72.7 & 63.0 & 72.6 & - & - & - & - & - & 70.4 \\ \hline \multirow{2}{*}{MLQA} & X-Mod & **80.1** & **58.6** & - & **60.7** & - & - & - & **67.5** & - & - & - & - & - & **66.7** \\  & shared & 79.6 & 53.6 & - & 58.7 & - & - & - & 64.9 & - & - & - & - & - & 64.2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Pre-trained language results for the modular and shared model variants, pre-trained on the set of 60 languages for 265k update steps. For NER and MLQA we report \(F_{1}\), for XNLI _accuracy_ scores. Scores are averaged across all 5 random seeds of the best hyperparameter setting, evaluated on the development set.

\begin{table}
\begin{tabular}{c l r r r r r r r r r} \hline \hline  & & bg & de & el & es & tr & ur & zh & avg \\ \hline NER & X-Mod & **77.6** & **75.1** & **75.2** & **71.9** & **72.6** & **54.7** & **21.6** & **64.1** \\  & shared & 74.9 & 66.3 & 69.6 & 49.1 & 64.8 & 50.4 & 9.2 & 54.9 \\ \hline \multirow{2}{*}{XNLI} & X-Mod & **77.4** & **75.4** & **76.2** & **78.5** & **72.4** & **64.9** & **73.8** & **74.1** \\  & shared & 76.3 & 74.1 & 74.9 & 77.3 & 71.0 & 64.3 & 71.4 & 72.8 \\ \hline \multirow{2}{*}{MLQ} & XL-Mod & - & **63.8** & - & **68.6** & - & - & **61.7** & **64.8** \\  & shared & - & 58.9 & - & 66.7 & - & - & 56.5 & 60.7 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results for added languages, for models pretrained on the set of 60 languages for 265k update steps. We report \(F_{1}\) and _accuracy_ scores which are averaged across all 5 random seeds of the best hyperparameter setting on the development set.

 

pre-trained on a language of the same family (Table 4). Consequently, we pre-train two models on two sets of languages, adding the respective other set post-hoc.15

Footnote 15: In previous experiments, the modular model trained on 60 languages achieved the best performance. Therefore, the models in these experiments are also trained on 60 languages. Both models are trained on the same additional languages, i.e. the 60-LANGS of Table 1, where only the 13-LANGS differ.

Our XNLI results (Figure 5) demonstrate that the per-language performance is on par when pretraining vs. when adding the language post-hoc.16 We also find that the family does not have a measurable effect on the performance of the language. Our results therefore suggest that it is sufficient to train X-Mod on only a subset of languages for which sufficient pre-training data exists. Essentially, X-Mod has the potential to cover all languages of the world, as the model has the capability to be adapted to new languages post-hoc.

Footnote 16: The models have seen an equal amount of examples in the respective languages in each case.

## 6 Further analysis

We further analyze the impact of the number of update steps on X-Mod (SS6.1) and compare our method to adapter-based approaches (SS6.2).

### The importance of update steps

In Figure 4 we have witnessed a slight edge of the shared model over the X-Mod model, when training on only 13 languages and only training for 125k update steps. Dufter and Schutze (2020) found that it requires a large number of update steps for a model pre-trained on multiple languages to become multilingual; with the added per-language capacity we hypothesize that update steps also play an important role for modular models. We compare the downstream task performance of models pre-trained on 13 languages, when training for 125k with 250k update steps in Figure 6. When training for longer we find that the X-Mod model begins to outperforms the shared model in the source language, while almost closing the gap in the cross-lingual setting. This supports the hypothesis that the X-Mod model requires more update steps when training only on a small number of languages, in order for modularity to "kick-in".

### X-Mod vs. Adapters

As illustrated in Figure 3, from an architecture perspective X-Mod is similar to previously proposed multilingual Adapter-based methods (MAD-X; Pfeiffer et al., 2020). MAD-X utilizes a pretrained massively multilingual transformer-based model and fine-tunes newly introduced adapter weights on languages the model has seen during pre-training, and ones the model has not been

\begin{table}
\begin{tabular}{l l l l l l} \hline Language & iso & Family & Script & Model 1 & Model 2 \\ \hline \hline English & en & IEC & Germanic & Latta & pre-train & add \\ German & de & IE & Germanic & Latin & add & pre-train \\ French & fr & DE & Romanic & Latin & pre-train & add \\ Spanish & es & IE: Romance & Latin & ad & pre-train \\ Russian & ren & IE: Slave & Cyclic & pre-train & add \\ Ukrainian & tx & IE: Slavé & Cyrilice & add & pre-train \\ Hindi & hi & IE: Trainan & Dreamgan 

trained on. For a fair comparison in terms of _seen examples_ and _number of update steps_ we train a transformer model without module components (_shared_nn_) for 100k update steps on the respective languages (Table 1). We subsequently train adapters on each of the target languages for another 25k update steps.17 We report results in comparison to X-Mod in Figure 7, here results for _shared_nm_ are for a model that was trained for 125k update steps to instantiate a fair comparison.

Footnote 17: We follow Pfeiffer et al. (2020b) and train adapter weights with a learning rate of 0.0001. While they have found that cross-lingual transfer performance of adapters converges at \(\sim\)20k update-steps, we would like to stress that our experimental setup is only one of multiple different valid versions. A more thorough investigation to find the optimal number of update steps for pre-training and subsequent adapter training is necessary, which was out of scope for this work.

Our results demonstrate that the additional capacity of adapters added _after_ pre-training is not able to mitigate the curse of multilinguality which has already had a catastrophic impact on the shared transformer weights; the performance of the adapters strongly correlates with the performance of the corresponding fully shared model _shared_nm_. Consequently, adding language-specific capacity _during_ pre-training is important, as the curse of multilinguality cannot be lifted post-hoc.

## 7 Conclusions

In this paper, we have evaluated the effectiveness of modular multilingual language modelling across multiple axes. We have demonstrated that by providing additional per-language capacity, while maintaining the total number of trainable parameters per language, we are not only able to mitigate negative interference between languages, but additionally achieve positive transfer. Our results suggest that it is sufficient to train our proposed X-Mod model only on a subset of languages for which sufficient amounts of textual data is available. Unseen languages can be added post-hoc, with no measurable drop in performance on XNLI. By _pre-training_ the model in a modular fashion, we thus mitigate negative interference of idiosyncratic information, while simultaneously preparing the model to be extendable to unseen languages.

While in this work we have simulated language adding scenarios with a held out set of languages, in future work we aim to evaluate the performance on truly low-resource languages such as MasakhaNER Adelani et al. (2021) and AmericasNLI Ebrahimi et al. (2021). We further aim to evaluate the cross-lingual transfer performance from typologically more diverse source languages, besides English.

## Acknowledgments

We thank Samuel Broscheit for insightful feedback and suggestions on a draft of this paper, as well as the ARR reviewers and meta-reviewers for their valuable comments.

## References

* Afedani et al. (2021) David Ifeoluwa Adelani, Jade Z. Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsudden Hassan Muhammad, Chris Chienney Emzeue, Joyce Nakatunba-Nabende, Perez Ogayo, Aremu Anouluwapo, Catherine Gitau, Derguene Mbaye, Jesjoba O. Alabi, Seid Muhie Yimam, Tajuddeen Gwadabe, Ignatius Ezeani, Rubung Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin P. Adewumi, Paul Rayson, Mofetlouwa Adeymeni, Gerald Muriaki, Emmanuel Aneib, Chiamak Chukwuneke, Nkiriuka Obtu, Eric Peter Wariagala, Samuel Overinde, Clemencia Siro, Tobius Saul Batesan, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiline, Ayodele Awokoya, Mohammadane Mboup, Dibora Gebreyohannes, Henko Tilayce, Klecchi Nowike, Degaga Wolde, Abdoulay Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Doson, Kelechi Ogueji, Thierno Ibrahim Diop, Abdoulaye Diallo, Adewake Akinfaderin, Tendai Menengeree, and Salomey Osei. 2021. MasakhaNER: Named Entity Recognition for African Languages. In _Transactions of the Association for Computational Linguistics 2021_.
* Andreas et al. (2016) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics:

Figure 7: Comparison on XNLI of X–Mod and _shared_ models with an Adapter baseline, all models are pre-trained for 125k update steps.

 

Human Language Technologies, San Diego California, USA, June 12-17, 2016_, pages 1545-1554. The Association for Computational Linguistics.
* Ansell et al. (2021a) Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulic. 2021a. Composable sparse fine-tuning for cross-lingual transfer. _arXiv preprint_.
* Ansell et al. (2021b) Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavas, Ivan Vulic, and Anna Korhonen. 2021b. MAD-G: Multilingual adapter generation for efficient cross-lingual transfer. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 4762-4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4623-4637, Online. Association for Computational Linguistics.
* Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 1538-1548. Association for Computational Linguistics.
* Chau et al. (2020a) Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020a. Parsing with multilingual BERT, a small corpus, and a small treebank. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1324-1334, Online. Association for Computational Linguistics.
* Chau et al. (2020b) Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020b. Parsing with multilingual bert, a small treebank, and a small corpus. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_, pages 1324-1334.
* Chen et al. (2019) Vincent S. Chen, Sen Wu, Alexander J. Ratner, Jen Weng, and Christopher Re. 2019. Slice-based learning: A programming model for residual learning in critical data slices. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 9392-9402.
* Chronopoulou et al. (2020) Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2020. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2703-2711, Online. Association for Computational Linguistics.
* Chung et al. (2021) Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Chung et al. (2020) Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 4536-4546. Association for Computational Linguistics.
* Clark et al. (2022) Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. CANINE: pre-training an efficient tokenization-free encoder for language representation. _Transactions of the Association for Computational Linguistics_, 10.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Conference of the Association for Computational Linguistics, ACL 2020, Virtual Conference, July 6-8, 2020_, pages 8440-8451.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Dufter and Schutze (2020) Philipp Dufter and Hinrich Schutze. 2020. Identifying elements essential for BERT's multilinguality. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4423-4437, Online. Association for Computational Linguistics.
* Ebrahimi et al. (2021) Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir, Gustavo A. Gimenez-Lugo, Eliseabeth Mager, Graham Neubig, Alexiya Palmer, Rolando A. Coto Solano, Ngoc Thang Vu, and Katharina Kann. 2021. Americas--NLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages. _arXiv preprint_ 

William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. _arXiv preprint_.
* Garcia et al. (2021) Xavier Garcia, Noah Constant, Ankur Parikh, and Orhan Firat. 2021. Towards continual learning for multilingual machine translation via vocabulary substitution. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1184-1192, Online. Association for Computational Linguistics.
* Glavas et al. (2019) Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 710-721.
* Gurungan et al. (2021) Suchin Gurungan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021. Demix layers: Disentangling domains for modular language modeling. _arXiv preprint_.
* 29, 2022_.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzkebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, pages 2790-2799.
* Hu et al. (2020)Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalization. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 12-18 July 2020, Virtual Conference_.
* K. K. et al. (2020) Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: an empirical study. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
* Lauscher et al. (2020a) Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, Nikolai Rozanov, and Goran Glavas. 2020a. Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers. In _Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 43-49, Online. Association for Computational Linguistics.
* Lauscher et al. (2020b) Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavas. 2020b. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4483-4499, Online.
* Le et al. (2021) Hang Le, Juan Miguel Pino, Changshan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2021. Lightweight adapter tuning for multilingual speech translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021_, pages 817-824. Association for Computational Linguistics.
* Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7315-7330, Online. Association for Computational Linguistics.
* Mahabadi et al. (2021a) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021a. Compacter: Efficient low-rank hypercomplex adapter layers. _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021_.
* Mahabadi et al. (2021b) Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021b. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 565-576. Association for Computational Linguistics.
* Muller et al. (2021) Benjamin Muller, Antonios Anastasopoulos, Benoit Sagot, and Djame Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 448-462, Online. Association for Computational Linguistics.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of 

the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* August 4, Volume 1: Long Papers_, pages 1946-1958.
* Pfeiffer et al. [2021a] Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckle, Kyunghyun Cho, and Iryna Gurevych. 2021a. AdapterFusion: Non-destructive task composition for transfer learning. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 487-503, Online. Association for Computational Linguistics.
* Pfeiffer et al. [2020a] Jonas Pfeiffer, Andreas Ruckle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterHub: A Framework for Adapting Transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (System Demonstrations), EMNLP 2020, Virtual Conference, 2020_.
* Pfeiffer et al. [2020b] Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020b. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7654-7673, Online. Association for Computational Linguistics.
* Pfeiffer et al. [2021b] Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2021b. UNKs Everywhere: Adapting Multilingual Language Models to New Scripts. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Online, November 2, 2021_.
* Philip et al. [2020] Jerin Philip, Alexandre Berard, Matthias Galle, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4465-4470, Online. Association for Computational Linguistics.
* Pires et al. [2019] Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 4996-5001.
* Ponti et al. [2020] Edoardo Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2362-2376, Online. Association for Computational Linguistics.
* Poth et al. [2021] Clifton Poth, Jonas Pfeiffer, Andreas Ruckle, and Iryna Gurevych. 2021. What to pre-train on? efficient intermediate task selection. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 10585-10605. Association for Computational Linguistics.
* Rahimi et al. [2019] Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 151-164.
* Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 2383-2392.
* Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In _Advances in Neural Information Processing Systems '30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_, pages 506-516.
* Rebuffi et al. [2018] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2018. Efficient parametrization of multidomain deep neural networks. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 8119-8127.
* Ruckle et al. [2021] Andreas Ruckle, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. Adapterdrop: On the efficiency of adapters in transformers. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 7930-7946. Association for Computational Linguistics.
* Ruckle et al. [2020] Andreas Ruckle, Jonas Pfeiffer, and Iryna Gurevych. 2020. Multicqa: Zero-shot transfer of self-supervised text matching models on a massive scale. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 2471-2486. Association for Computational Linguistics.
* Ruder et al. [2019] Sebastian Ruder, Ivan Vulic, and Anders Sogaard. 2019. A survey of cross-lingual embedding models. _Journal of Artificial Intelligence Research_, 65:569-631 

Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics.
* Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReviewen.net.
* Stickland et al. (2021) Asa Cooper Stickland, Alexandre Berard, and Vassilina Nikoulina. 2021. Multilingual domain adaptation for NMT: decoupling language and domain information with adapters. In Proceedings of the Sixth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021, pp. 578-598. Association for Computational Linguistics.
* Stickland and Murray (2019) Asia Cooper Stickland and Iain Murray. 2019. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Volume 97 of _Proceedings of Machine Learning Research_, pages 5986-5995. PMLR.
* Tay et al. (2021) Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Prakash Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Chartformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint.
* Tran (2020) Ke M. Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint.
* Ustin et al. (2021) Ahmet Ustin, Alexandre Berard, Laurent Besacier, and Matthias Galle. 2021. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6650-6662. Association for Computational Linguistics.
* Ustin et al. (2020) Ahmet Ustin, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998-6008.
* Vernikos and Popescu-Belis (2021) Giorgros Vernikos and Andrei Popescu-Belis. 2021. Subword mapping and anchoring across languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Vidoni et al. (2020) Marko Vidoni, Ivan Vulic, and Goran Glavas. 2020. Orthogonal language and task adapters in zero-shot cross-lingual transfer. In arXiv preprint.
* Wang et al. (2021a) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Davin Jiang, and Ming Zhou. 2021a. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 1405-1418. Association for Computational Linguistics.
* Wang et al. (2021b) Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig. 2021b. Efficient test time adapter ensembling for low-resource language varieties. In Findings of the Association for Computational Linguistics: EMNLP 2021, Panta Cana, Dominican Republic. Association for Computational Linguistics.
* Wang et al. (2020) Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649-2656, Online. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wu et al. (2020) Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Conference of the Association for Computational Linguistics, ACL 2020, Virtual Conference, July 6-8, 2020, pp. 6022-6034.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, benttz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 

833-844, Hong Kong, China. Association for Computational Linguistics.
* Wu and Dredze (2020)Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_, pages 120-130, Online. Association for Computational Linguistics.
* Xue et al. (2022)Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. _Transactions of the Association for Computational Linguistics 2022_.

## Appendix A Additional results

We report MLQA and XQuAD results on pretrained languages in Tables 5 and 6, respectively, and MLQA results on added languages in Table 7. Table 8 report NER results on more languages.

Figures 9, 10 and 11 report per-language results as we increase the amount of languages on language modeling perplexity, XNLI and NER, respectively.

## Appendix B Intermediate checkpoints

Our results in SS6.1 suggest that, when the number of languages is small, X-Mod becomes more competitive with shared as the number of training steps increases. So as to understand if this behavior also holds for models covering more languages, we evaluate intermediate checkpoints for the 60-LANG model on XNLI. As shown in Figure 8, we find that the X-Mod model continuously outperforms the shared model. This suggests that the shared model immediately suffers from negative interference between languages, while the added, language-specific components of the X-Mod model are able to mitigate the curse of multilinguality, resulting in considerable performance gains at all evaluated checkpoints.

## Appendix C Language selection

We provide more details about our selection of languages in Table 9.

\begin{table}
\begin{tabular}{l c c c|c c c|c} \hline  & zn & zn & zn & zn & zn & zn \\  & F\({}_{1}\)/ EM & F\({}_{1}\)/EM & F\({}_{1}\)/EM & F\({}_{1}\)/EM & F\({}_{1}\)/EM & F\({}_{1}\)/EM \\ \hline X-Mod & **83.17+48.9** & **68.87+50.3** & **76.87+50.3** & **64.37+50.3** & **76.97+50.4** & **76.97+50.4** & **72.78+50.5** \\ standard & 83.73+21.1 & 46.87+50.5 & 68.55+43.3 & 72.75+53.0 & 60.40 & 72.63+21.1 & 70.45+53.9 \\ \hline \end{tabular}
\end{table}
Table 6: Average F\({}_{1}\) and Exact Match results for **pre-trained languages**, on the test set of **XQuAD** for the X-Mod and shared model variants, pre-trained on the set of 60 languages for 265k update steps. **Bold** numbers indicate better performance for the respective language.

Figure 8: Results on XNLI using intermediate checkpoints of the models trained on 60 languages.

 



\begin{table}
\begin{tabular}{l c|c c c c c c c c c c c c c c c c c c c c c c} \hline  & en & af & ar & bu & et & eu & fa & fi & fr & hi & hu & id & it & ix & ka & bo & ru & sw & ta & th & vi & avg \\ \hline X-Mod & 81.4 & **78.9** & 43.5 & **63.2** & **76.2** & **62.2** & **44.3** & **78.6** & **77.2** & **70.1** & **78.3** & 50.5 & **78.7** & **67.3** & **53.0** & **59.1** & **73.4** & 51.1 & 2.8 & **66.2** & **62.8** \\ shared & **81.5** & 74.1 & **44.2** & 62.4 & 70.7 & 58.1 & 40.3 & 74.4 & 74.7 & 64.4 & 74.2 & **51.5** & 75.5 & 61.5 & 46.0 & 58.3 & 57.2 & **52.5** & **4.0** & 63.7 & 59.5 \\ \end{tabular}
\end{table}
Table VIII: Average F\({}_{1}\) results for **pre-trained languages**, on the test set of **NER** for the X-Mod and shared model variants, pre-trained on the set of 60 languages. **Bold** numbers indicate better performance for the respective language.

Figure 10: Testset results on **XNLI** of pre-trained (top) and added (bottom) languages trained on different numbers of languages. Models trained on more languages are trained for longer \(\rightarrow\) all models have seen the **same amount of examples** in each individual language. Scores are averaged across all random seeds.

Figure 9: Perplexity when training on more languages. Each model has seen the **same amount of examples** in each language. Lower perplexity indicates better performance.



* [150] M. C. Bresson, A. Krizhevsky, and J. Schmidhuber. Learning representations from a large dataset. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 238-246, 2017.
* [151] D. L. Kingma and P. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1912.03012_, 2019.
* [152] R. G. LeCun, Y. Bengio, and S. Bengio. Generative adversarial nets. In _Advances in neural information processing systems_, volume 10, pages 1038-1049, 2019.
* [153] T. Lin, X. Liu, and Z. Chen. Deep residual learning with deep convolutional networks. In _NIPS conference on computer vision_, pages 1338-1346, 2019.
* [154] H. Li, W. Dong, and Q. Weinberger. Learning generative models via graph embedding. In _CVPR 2016_, pages 1338-1349, 2016.
* [155] I. Goodfellow, I. Guyon, and J. Shlens. Imagenet classification by jointly connected layers. In _Advances in neural information processing systems_, volume 10, pages 1038-1049, 2016.
* [156] V. Mnih, N. Bahdan, A. Graves, A. Rabinovicius, and A. Rabinovicius. Very accurate explanations for image captioning. _CoRR_, abs/1412.03082, 2014.
* [157] E. Silverstein, O. Talvain, and A. Ruskov. The mnist layer: A review. _Neural machine translation_, 2(3):125-139, 2017.
* [158] F. Chollet, M. Devin, A. Courville, and J. Dean. Dropout: a database for data augmentation. In _ICLR 2015_, pages 1093-1098, 2015.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l c r