SlowGuess/NAACL_2022_OCR huggingface dataset:
Jonas Pfeiffer\*1,2,3, Naman Goyal3, Xi Victoria Lin3, Xian Li3, James Cross3, Sebastian Riedel3, Mikel Artetxe3
$^{1}$ New York University,  $^{2}$ TU Darmstadt,  $^{3}$ Meta AI
Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-MoD) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.
Recent work on multilingual NLP has focused on pre-training transformer-based models (Vaswani et al., 2017) on concatenated corpora of a large number of languages (Devlin et al., 2019; Conneau et al., 2020). These multilingual models have been shown to work surprisingly well in cross-lingual settings, despite the fact that they do not rely on direct cross-lingual supervision (e.g., parallel data or translation dictionaries; Pires et al., 2019; Wu and Dredze, 2019; Artetxe et al., 2020; Hu et al., 2020; K et al., 2020; Rust et al., 2021).
However, recent work has uncovered fundamental limitations of multilingual transformers. Conneau et al. (2020) observe that pre-training a model with a fixed capacity on an increasing amount of languages only improves its cross-lingual performance up to a certain point, after which perfor
Figure 1: A transformer layer of our proposed modular architecture. The dark blue and green components illustrate the modular layers, which are language specific. The Multi-Head Attention and Feed-Forward components are shared by all languages.
mance drops can be measured—a phenomenon known as the curse of multilinguality (Figure 2). As such, prior work had to find a trade-off between supporting more languages and obtaining better performance on a smaller set of languages.
In this work, we address this problem by introducing language-specific, modular components during pre-training (Figure 1). Our Cross-lingual, Modular (X-MOD) language model shares the majority of the transformer parameters between all pretraining languages, while providing each language with individual capacity to learn idiosyncratic information without increasing the total number of trainable parameters per language. While previous adapter-based approaches (Figure 3a) extend pretrained multilingual language models (LMs) with modular components after pre-training, we add modular components during pre-training, thereby
* Work done while interning at Meta AI.
Human Language Technologies, pages 3479 - 3495
(a) Mean Perplexity.
(b) Mean Performance on XNLI and NER.
preparing the model to be extended to new languages post-hoc. Our experiments on natural language inference (NLI), named entity recognition (NER), and question answering (QA) demonstrate that our modular architecture not only is effective at mitigating interference between languages, but also achieves positive transfer, resulting in improved monolingual and cross-lingual performance. In addition, we show that X-MOD can be extended to unseen languages, with no measurable drop in performance, by learning its corresponding modules and leaving the shared parameters frozen. All in all, we propose a multilingual architecture that can scale to a large number of languages without any loss in performance, and can be further extended to new languages after pre-training. $^{1}$
We provide a background on multilingual and modular language modelling, as well as approaches that extend LMs to new languages.
2.1 Multilingual transformers
Recent LMs (Devlin et al., 2019; Conneau et al., 2020), based on transformer architectures (Vaswani et al., 2017) and pre-trained on massive amounts of multilingual data, have surpassed (static) cross-lingual word embedding spaces (Ruder et al., 2019; Glavas et al., 2019) for cross-lingual transfer in NLP (Pires et al., 2019; Wu and Dredze, 2019; Wu et al., 2020; Hu et al., 2020; K et al., 2020). Transformer-based models are 1) pre-trained on textual corpora using Masked Language Modelling
Figure 2: Average (a) perplexity and (b) transfer performance on XNLI and NER across pre-trained languages when training on an increasing number of languages. Each model has seen the same amount of examples in each language. Lower perplexity and higher downstream score indicate better performance. Refer to Figure 4 for per-task performance, and Appendix A for per-language performance.
(a) Adapter-based
Figure 3: Our proposed architecture in comparison to adapter-based approaches. (a) Previous approaches  $①$  utilize non-modular pre-trained transformer models and  $②$  extend them with modular adapter components. (b) We  $①$  pre-train the transformer with modular units from the get-go, preparing the model to be  $②$  extended with additional modular units later on. Yellow and light blue components indicate standard Multi-Head Attention and Feed-Forward layers. The remaining (non-gray) components are bottleneck (modular) units. Grayed-out components are frozen.
(b) X-MOD
(MLM). They are then 2) fine-tuned on labelled data of a downstream task in a source language and 3) directly applied to perform inference in a target language (Hu et al., 2020).
2.2 Modular language models
Modular approaches have a long standing history in NLP, preceding pre-trained models (Andreas et al., 2016). They have recently regained interest for transformer-based models, where mix
$^{1}$ Code and pre-trained models are available at: https://github.com/pytorch/fairseq/tree/main/examples/xmod.
ture of experts (MoE; Shazeer et al., 2017) approaches have enabled training trillion parameters models in a distributed fashion (Fedus et al., 2021). More recently modular MoE approaches have been shown to improve domain-specific pretraining of LMs (Gururangan et al., 2021). In a similar trend, 'expert' modules have been added to (non-modular) pre-trained LMs post-hoc, predominantly referred to as adapters (Rebuffi et al., 2017, 2018; Houlsby et al., 2019). Next to being extremely parameter (Houlsby et al., 2019; Mahabadi et al., 2021a; He et al., 2022) and training efficient (Pfeiffer et al., 2020a; Rückle et al., 2021), these modular approaches allow models to be extended to new data settings (Chen et al., 2019; Rückle et al., 2020), where newly learned knowledge can be combined (Stickland and Murray, 2019; Wang et al., 2021a; Pfeiffer et al., 2021a; Lauscher et al., 2020a; Mahabadi et al., 2021b; Poth et al., 2021), or stacked for combinatory cross-lingual (Pfeiffer et al., 2020b, 2021b; Üstün et al., 2020; Vidoni et al., 2020; Ansell et al., 2021b,a; Wang et al., 2021b) as well as NMT scenarios (Bapna and Firat, 2019; Philip et al., 2020; Chronopoulou et al., 2020; Le et al., 2021; Üstün et al., 2021; Stickland et al., 2021; Garcia et al., 2021).
2.3 Weaknesses, improvements, and extensions of language models
Next to the curse of multilinguality, recent works have shown substantially reduced cross-lingual and monolingual abilities of models for low-resource languages with smaller pre-training data (Wu and Dredze, 2020; Hu et al., 2020; Lauscher et al., 2020b; Artetxe et al., 2020; Pfeiffer et al., 2020b, 2021b; Chau et al., 2020b; Ponti et al., 2020).
K et al. (2020); Artetxe et al. (2020) show that a shared vocabulary is not necessary for cross-lingual transfer. Chung et al. (2021) demonstrate that decoupling the input embeddings from the prediction head improves the performance on a number of downstream tasks. Dufter and Schütze (2020) show that the number of parameters and training duration is interlinked with the model's multilingual capability. Chung et al. (2020); Rust et al. (2021) show that the tokenizer plays an important role in the per-language downstream task performance, which Clark et al. (2022); Xue et al. (2022); Tay et al. (2021) take to the extreme by proposing tokenizer-free approaches.
To extend a monolingual LM to other languages,
Artetxe et al. (2020) train a new embedding layer with a corresponding target-language tokenizer, while freezing the pre-trained transformer weights. Tran (2020) extend a monolingual model to new languages using bilingual corpora. Wang et al. (2020); Chau et al. (2020a) extend the vocabulary of multilingual models with a small number of target-language tokens, to improve the performance in the target language. Muller et al. (2021) propose a transliteration based approach, Vernikos and Popescu-Belis (2021) propose subword mappings, and Pfeiffer et al. (2020b, 2021b); Vidoni et al. (2020); Ansell et al. (2021b) propose adapter-based approaches to extend multilingual models to unseen languages.
While these approaches achieve considerable performance gains over unseen languages, they are outperformed by standard full fine-tuning methods for seen languages. One can further argue that, as the pre-trained models have already been cursed by multilinguality, the adapter-based approaches build upon sub-optimal parameter initializations. In our work, we consequently aim to 1) modularize the model from the start to prepare the model to be 2) extendable to new languages post-hoc.
We propose X-MOD, a modular multilingual architecture that combines shared and language-specific parameters. In contrast to prior work, we pretrain modular models from the get-go. Our models can be extended to new languages after pretraining, and used for cross-lingual transfer learning in downstream tasks.
Architecture. As illustrated in Figure 1, we extend the transformer-based architecture from mBERT (Devlin et al., 2019) and XLM-R (Connieu et al., 2020) by incorporating language-specific modules—bottleneck feed-forward layers—at every transformer layer. We learn a separate module for each language, whereas the attention and feed-forward components are shared. While the total number of parameters of the model grows linearly with the number of languages, the training and inference cost does not increase (as measured in FLOPs), as only the module in the relevant language is used for each input. Inspired by the adapter<sup>3</sup> architecture of Pfeiffer et al. (2021a) we
2We investigate this claim further in  $\S 6.2$
3The term 'adapter' refers to newly introduced layers within a pre-trained (frozen) model. These layers adapt the
place our 'modules' after the LayerNorm of the feed-forward transformer block, and the residual connection is placed after the LayerNorm; the LayerNorm before and after the modular component is shared.
Pre-training procedure. Similar to Conneau et al. (2020), we pre-train our model on MLM on combined monolingual corpora in multiple languages. Examples of each language are passed through the shared embedding matrix as well as the multi-head attention and feed-forward components at each layer. As each layer contains a language-specific modular component, the examples are routed through the respective designated modular bottleneck layer. Given that each example only requires access to a single module, modules can be efficiently stored on only a subset of GPUs in distributed training.
Extending to new languages. The modular design of our model allows us to extend it to new languages after pre-training. To that end, we learn new embeddings and adapter modules for the target language through MLM, while the rest of the components are frozen.<sup>6</sup> Consequently, we are able to extend the model to a new language by learning a small number of new parameters, without affecting performance in the set of pre-trained languages. Following Pfeiffer et al. (2021b), we learn a new subword vocabulary for the added languages, and initialize the embeddings of lexically overlapping tokens from the original embedding matrix.
Fine-tuning on downstream tasks. To transfer the models to cross-lingual downstream tasks, we fine-tune the shared weights only on the source language data, while keeping the modular components and the embedding layer frozen. We follow the standard fine-tuning procedure of adding a prediction head on top of the CLS token. We then replace the source language modules (as well as embedding layer for added languages) with the target language parameters, passing the text of the target language through the model.<sup>7</sup>
We detail the baseline and models (§4.1), and their training (§4.2) and evaluation settings (§4.3).
4.1 Model variants
We pre-train separate models for all combinations along the following axes:
X-MOD vs. SHARED. To evaluate the effectiveness of our X-MOD model, we aim to compare ourselves to a conventional non-modular architecture. However, simply removing the modular component would be unfair, as the number of FLOPs and trainable parameters per language would not be the same—both in terms of pre-training, as well as fine-tuning. Consequently, for our baseline model—where all parameters should be fully shared between all languages—we include a single bottleneck layer right after the Feed-Forward component. Effectively, this is the same architecture as our X-MOD model, just with a single module that is shared by all languages. We refer to this as the SHARED model throughout this paper. $^{8}$  To extend the SHARED model to unseen languages, we follow Artetxe et al. (2020) and only learn a new embedding layer, freezing the transformer parameters. To fine-tune the SHARED model on a downstream task, we freeze the embedding layer, as well as the (single) module, thereby fine-tuning an equal amount of parameters on the downstream task as the X-MOD model. $^{9}$
13 vs. 30 vs. 60 vs. 75 languages. So as to understand how each approach is affected by the curse of multilinguality, we pre-train the X-MOD and SHARED models on 4 increasing sets of languages. We start with an initial set of 13 typologically diverse languages that we evaluate on, and add additional languages for larger sets of 30, 60, and 75 languages. In addition, we keep a set of 7 held-out languages that we extend the pre-trained models to. Table 1 lists the specific languages in each
representations of the pre-trained mode; we train these modular components together with the transformer weights, and therefore refer to them as modules.
<sup>4</sup>We find that the residual connection proposed by Pfeiffer et al. (2021a) results in training instabilities when trained together with the transformer weights.
5 Preliminary results showed that sharing the LayerNorm results in better cross-lingual transfer performance.
$^{6}$ Following Artetxe et al. (2020) we train positional embeddings.
We initially also experimented with stacking adapters on
top of the language modules similar to Pfeiffer et al. (2020b, 2021b). While this approach is considerably more parameter efficient, we find that fine-tuning all shared weights slightly outperformed the adapter-based approach.
8Extending the total number of shared parameters would be unfair, as X-Mod and SHARED would not have the same FLOPs nor the same number of trainable parameters when fine-tuning.
9Adapter-based approach such as MAD-X (Pfeiffer et al., 2020b) would be an alternative. However, this would require training on languages twice—once during pre-training, and once when adding adapters—which is not directly comparable to X-MOD. Nonetheless, we report results in §6.2.
Pre-trained languages	13-LANGS	en, ar, fr, hi, ko, ru, th, vi, ta, id, fi, sw, ka
30-LANGS	13-LANGS + cs, eu, hr, hu, hy, it, lt, ml, mn, ms, pl, ro, si, sk, sq, sv, tl
60-LANGS	30-LANGS + af, am, be, bn, ca, cy, da, eo, et, fa, ga, gl, gu, ha, is, ku, la, lv, mk, ne, nl, no, ps, pt, sa, sd, sl, so, sr, te
75-LANGS	60-LANGS + as, br, bs, fy, gd, jv, kn, mg, mr, om, or, pa, su, xh, yi,
Added languages	bg, de, el, es, tr, ur, zh,
Table 1: Selection of languages. We pre-train different models on 4 sets of languages, and further extend them to a set of held-out languages post-hoc. We evaluate on XNLI (languages in bold), NER (underlined languages) and XQuAD/MLQA (languages in italic). For more details about the language selection, see Appendix C.
group. The selection and split of initial as well as added languages is motivated by typological and geographical diversity, as well as the availability of downstream task evaluation data.
Controlling for total vs. per-language updates. Conneau et al. (2020) investigated the effect of adding more languages during pre-training, while training on an equal number of update steps. However, increasing the number of languages while keeping the number of updates constant results in the model seeing less data in each individual language. As such, it remains unclear if the curse of multilinguality happens because of negative interference, or simply because the number of updates for each specific language is smaller. So as to understand this, we compare (1) training on an equal number of update steps and (2) training on an equal number of seen examples per language. We start with the set of 13 languages (Table 1) and train the respective models for  $125\mathrm{k}$  update steps. When adding more languages, we compare (1) training models on each set of languages for  $125\mathrm{k}$  update steps, and (2) increasing the number of update steps such that the models are trained on the same number of examples in each of the initial 13 languages. For the latter, this amounts to training for  $195\mathrm{k}$ ,  $265\mathrm{k}$  and  $269\mathrm{k}$  update steps, respectively.
Data and hyperparameters. We sample languages with  $\alpha = 0.7$  and train our models with a batch size of 2048 across 64 V100 GPUs on the CC100 dataset (Conneau et al., 2020) using fairseq (Ott et al., 2019). All our models extend the base transformer architecture, with 12 layers and 768 dimensions. Modules are implemented with a bottleneck size of 384. The shared transformer weights account for 270M parameters, whereas each individual module accounts for 7M parameters. We train our models with a linear learning
rate decay peaking at  $7\mathrm{e} - 4$  during pre-training and  $1\mathrm{e} - 4$  when adding languages.
Vocabulary. As we aim to identify the impact of modularity on the curse of multilinguality, we control for consistent tokenization across the different axes. We therefore tokenize using the XLM-R vocabulary for all our pre-training experiments. $^{10}$  However, for languages added post-hoc, we learn a new SentencePiece tokenizer for each of the target language, $^{11}$  as the languages potentially use scripts unseen by the original tokenizer.
We conduct experiments on NLI, NER, and QA. In all cases, we fine-tune the model on English and measure the zero-shot transfer performance in other languages. For NLI we train on MultiNLI (Williams et al., 2018) and evaluate on XNLI (Connieu et al., 2018). For QA, we train on SQuAD (Rajpurkar et al., 2016) and evaluate on XQuAD (Artetxe et al., 2020) and MLQA (Lewis et al., 2020). For NER, we use WikiANN (Pan et al., 2017; Rahimi et al., 2019). We experiment with learning rates  $1\mathrm{e} - 4$ ,  $3\mathrm{e} - 4$ , and  $5\mathrm{e} - 4$  and train for 3 or 5 epochs for QA and 5 or 10 epochs for NER and NLI. For NER and NLI we take the hyperparameter setting performing best on the development sets, averaged across the pre-trained languages (Table 1). For SQuAD we take the best performing checkpoint evaluated on the English development set, and report the cross-lingual test set results.[12] All results are averaged across 5 random seed runs.
$^{10}$ Rust et al. (2021) have previously demonstrated the impact of the multilingual tokenizer on the downstream task performance: languages underrepresented in the sub-word vocabulary exhibit considerable performance drops when compared to vocabularies dedicated to the respective language.
11We train the new tokenizers for a vocabulary size of  $30k$
In contrast to NER and NLI, the cross-lingual evaluation benchmarks of SQuAD do not provide a development set for each target language on the basis of which the best checkpoint can be selected. Consequently, we select the checkpoint based
(a) All models are trained for 125k update steps. Models trained on more languages have seen less examples in each language.
(b) Models trained on more languages are trained longer. All models have seen the same amount of examples in each language.
Figure 4: Test set results on XNLI (top) and NER (bottom) for models trained on different numbers of languages. Source Language (English) only includes scores of the source language. Average Pre-Trained Languages includes all evaluation languages that the model was pre-trained on. Average Added Languages includes all languages that were added to the model after pre-training. Scores are averaged across all languages and random seeds.
We present results for pre-trained languages in §5.1 and added languages in §5.2.
5.1 Pre-trained languages
In Figure 4 we plot downstream task results of models pre-trained on different amounts of languages. Table 2 reports the individual language performance for the models trained on 60 languages.
The Curse of Multilinguality. Conneau et al. (2020) showed that multilingual LMs trained on increasing amounts of languages, while maintaining the number of update steps, exhibit drops in downstream task XNLI performance. We reproduce these results, both in terms of language modelling perplexity (Figure 2a), $^{13}$  as well as downstream
task performance on XNLI and NER (Figure 4a). We further find that the curse of multilinguality does not only happen because the total number of update steps per language decreases, but also when all SHARED models are trained on the same number of examples per language (Figure 4b). This confirms that fully shared architectures suffer from negative interference.
Lifting the Curse. While for the SHARED model we witness negative interference between languages in terms of perplexity, the X-MOD model is able to maintain performance, and even improves for a subset of languages. We observe similar patterns in the downstream task performance: In both our experimental setups—(1) we control for the number of update steps (Figure 4a); (2) we control for the number of per-language seen examples (Figure 4b)—our X-MOD model—in contrast to the SHARED model—is able to maintain, or
on the best performance on the English development set.
13For per-language perplexity see Appendix A.
en	ar	fr	hi	ko	ru	th	vi	ta	id	fi	sw	ka	avg
NER	X-MOD	81.4	78.9	77.2	70.1	53.0	59.1	2.8	66.2	51.1	50.5	78.6	73.4	67.3	62.8
SHARED	81.5	74.1	74.7	64.4	46.0	58.3	4.0	63.7	52.5	51.5	74.4	57.2	61.5	58.8
XNLI	X-MOD	84.4	71.2	77.6	68.3	-	74.1	71.7	73.4	-	-	-	66.9	-	73.5
SHARED	82.8	69.2	75.6	66.6	-	73.2	68.5	72.5	-	-	-	62.1	-	72.5
XQuAD	X-MOD	85.1	68.1	-	67.5	-	75.0	66.3	74.9	-	-	-	-	-	72.8
SHARED	83.8	64.6	-	65.8	-	72.7	63.0	72.6	-	-	-	-	-	70.4
MLQA	X-MOD	80.1	58.6	-	60.7	-	-	-	67.5	-	-	-	-	-	66.7
SHARED	79.6	53.6	-	58.7	-	-	-	64.9	-	-	-	-	-	64.2
Table 2: Pre-trained language results for the modular and shared model variants, pre-trained on the set of 60 languages for  ${265}\mathrm{k}$  update steps. For NER and MLQA we report  ${F}_{1}$  ,for XNLI accuracy scores. Scores are averaged across all 5 random seeds of the best hyperparameter setting, evaluated on the development set.
bg	de	el	es	tr	ur	zh	avg
NER	X-MOD	77.6	75.1	75.2	71.9	72.6	54.7	21.6	64.1
SHARED	74.9	66.3	69.6	49.1	64.8	50.4	9.2	54.9
XNLI	X-MOD	77.4	75.4	76.2	78.5	72.4	64.9	73.8	74.1
SHARED	76.3	74.1	74.9	77.3	71.0	64.3	71.4	72.8
MLQA	X-MOD	-	63.8	-	68.6	-	-	61.7	64.8
SHARED	-	58.9	-	66.7	-	-	56.5	60.7
Table 3: Results for added languages, for models pretrained on the set of 60 languages for 265k update steps. We report  $F_{1}$  and accuracy scores which are averaged across all 5 random seeds of the best hyperparameter setting on the development set.