SlowGuess/NAACL_2022_OCR huggingface dataset:
Processing (EMNLP), pages 2362-2376, Online. Association for Computational Linguistics.
Clifton Poth, Jonas Pfeiffer, Andreas Rückle, and Iryna Gurevych. 2021. What to pre-train on? efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 10585-10605. Association for Computational Linguistics.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 151-164.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 506-516.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2018. Efficient parametrization of multidomain deep neural networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 8119-8127.
Andreas Rückle, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. Adapterdrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7930-7946. Association for Computational Linguistics.
Andreas Rückle, Jonas Pfeiffer, and Iryna Gurevych. 2020. Multicqa: Zero-shot transfer of self-supervised text matching models on a massive scale. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2471-2486. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual embedding models. Journal of Artificial Intelligence Research, 65:569-631.
Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135, Online. Association for Computational Linguistics.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Asa Cooper Stickland, Alexandre Berard, and Vassilina Nikoulina. 2021. Multilingual domain adaptation for NMT: decoupling language and domain information with adapters. In Proceedings of the Sixth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021, pages 578-598. Association for Computational Linguistics.
Asa Cooper Stickland and Iain Murray. 2019. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5986-5995. PMLR.
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Prakash Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint.
Ke M. Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint.
Ahmet Üstün, Alexandre Berard, Laurent Besacier, and Matthias Galle. 2021. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6650-6662. Association for Computational Linguistics.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302-2315, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Giorgos Vernikos and Andrei Popescu-Belis. 2021. Subword mapping and anchoring across languages. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2633-2647, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Marko Vidoni, Ivan Vulić, and Goran Glavaš. 2020. Orthogonal language and task adapters in zero-shot cross-lingual transfer. In arXiv preprint.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021a. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 1405-1418. Association for Computational Linguistics.
Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig. 2021b. Efficient test time adapter assembling for low-resource language varieties. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 730-737, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2649-2656, Online. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging crosslingual structure in pretrained language models. In Proceedings of the 58th Conference of the Association for Computational Linguistics, ACL 2020, Virtual Conference, July 6-8, 2020, pages 6022-6034.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages
833-844, Hong Kong, China. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Computational Linguistics.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics 2022.
We report MLQA and XQuAD results on pretrained languages in Tables 5 and 6, respectively, and MLQA results on added languages in Table 7. Table 8 report NER results on more languages.
Figures 9, 10 and 11 report per-language results as we increase the amount of languages on language modeling perplexity, XNLI and NER, respectively.
B Intermediate checkpoints
Our results in §6.1 suggest that, when the number of languages is small, X-MOD becomes more competitive with SHARED as the number of training steps increases. So as to understand if this behavior also holds for models covering more languages, we evaluate intermediate checkpoints for the 60-LANG model on XNLI. As shown in Figure 8, we find that the X-MOD model continuously outperforms the SHARED model. This suggests that the SHARED model immediately suffers from negative interference between languages, while the added, language-specific components of the X-MOD model are able to mitigate the curse of multilinguality, resulting in considerable performance gains at all evaluated checkpoints.
C Language selection
We provide more details about our selection of languages in Table 9.
Figure 8: Results on XNLI using intermediate checkpoints of the models trained on 60 languages.
en
F1 / EM	ar
F1 / EM	hi
F1 / EM	vi
F1 / EM	avg
F1 / EM
X-MOD	80.1 / 66.9	58.6 / 38.9	60.7 / 42.4	67.5 / 46.1	66.7 / 48.6
SHARED	79.6 / 66.5	53.6 / 33.9	58.7 / 40.4	64.9 / 43.8	64.2 / 46.2
Table 5: Average  $F_1$  and Exact Match results for pre-trained languages, on the test set of MLQA for the X-MOD and SHARED model variants, pre-trained on the set of 60 languages for 265k update steps. Bold numbers indicate better performance for the respective language.
en
F1 / EM	ar
F1 / EM	hi
F1 / EM	ru
F1 / EM	th
F1 / EM	vi
F1 / EM	avg
F1 / EM
X-MOD	85.1 / 73.4	68.1 / 52.4	67.5 / 50.3	75.0 / 57.8	66.3 / 52.6	74.9 / 54.6	72.8 / 56.9
SHARED	83.8 / 72.1	64.6 / 48.5	65.8 / 48.3	72.7 / 54.5	63.0 / 48.0	72.6 / 52.1	70.4 / 53.9
Table 6: Average  $\mathrm{F}_{1}$  and Exact Match results for pre-trained languages, on the test set of XQuAD for the X-MOD and SHARED model variants, pre-trained on the set of 60 languages for 265k update steps. Bold numbers indicate better performance for the respective language.
de F1 / EM	es F1 / EM	zh F1 / EM	avg F1 / EM
X-MOD	63.8 / 48.9	68.8 / 50.3	61.7 / 36.4	64.8 / 45.2
SHARED	58.9 / 44.1	66.7 / 48.3	56.5 / 32.2	60.7 / 41.5
Table 7: Average  $\mathrm{F}_{1}$  and Exact Match results for added languages, on the test set of MLQA for the X-MOD and SHARED model variants, pre-trained on the set of 60 languages for 265k update steps. Bold numbers indicate better performance for the respective language.
en	af	ar	bn	et	eu	fa	fi	fr	hi	hu	id	it	ka	ko	ru	sw	ta	th	vi	avg
X-MOD	81.4	78.9	43.5	63.2	76.2	62.2	44.3	78.6	77.2	70.1	78.3	50.5	78.7	67.3	53.0	59.1	73.4	51.1	2.8	66.2	62.8
SHARED	81.5	74.1	44.2	62.4	70.7	58.1	40.3	74.4	74.7	64.4	74.2	51.5	75.5	61.5	46.0	58.3	57.2	52.5	4.0	63.7	59.5
Table 8: Average  $\mathrm{F}_{1}$  results for pre-trained languages, on the test set of NER for the X-MOD and SHARED model variants, pre-trained on the set of 60 languages. Bold numbers indicate better performance for the respective language.
Figure 9: Perplexity when training on more languages. Each model has seen the same amount of examples in each language. Lower perplexity indicates better performance.
(a) Pre-Trained Languages
(b) Added Languages
Figure 10: Testset results on XNLI of pre-trained (top) and added (bottom) languages trained on different numbers of languages. Models trained on more languages are trained for longer  $\rightarrow$  all models have seen the same amount of examples in each individual language. Scores are averaged across all random seeds.
(a) Pre-Trained Languages
(b) Added Languages
Figure 11: Testset results on NER of pre-trained (top) and added (bottom) languages trained on different numbers of languages. Models trained on more languages are trained for longer  $\rightarrow$  all models have seen the same amount of examples in each individual language. Scores are averaged across all random seeds.
Language	iso	Family	Script	13	30	60	75
Afrikaans	af	IE:Germanic	Latin	✓	✓
Albanian	sq	IE:Albanian	Latin	✓	✓	✓
Amharic	am	Afro-Asiatic	Amharic	✓	✓
Arabic	ar	Afro-Asiatic	Arabic	✓, (+)	✓, (+)	✓, (+)	✓, (+)
Armenian	hy	IE:Armenian	Armenian	✓	✓	✓
Assamese	as	IE:Iranian	Assamese	✓
Basque	eu	Isolate	Latin	✓	✓	✓
Belarusian	be	IE:Slavic	Cyrillic	✓	✓
Bengali	bn	IE:Iranian	Bengali	✓	✓
Bosnian	bs	IE:Slavic	Latin	✓
Breton	br	IE:Celtic	Latin	✓
Bulgarian	bg	IE:Slavic	Cyrillic	+	+	+	+
Catalan	ca	IE:Romance	Latin	✓	✓
Chinese	zh	Sino-Tibetan	Chinese	+	+	+	+
Croatian	hr	IE:Slavic	Latin	✓	✓	✓
Czech	cs	IE:Slavic	Latin	✓	✓	✓
Danish	da	IE:Germanic	Latin	✓	✓
Dutch	nl	IE:Germanic	Latin	✓	✓
English	en	IE:Germanic	Latin	✓, (+)	✓, (+)	✓, (+)	✓, (+)
Estonian	et	Uralic	Latin	✓	✓
Esperanto	eo	Constructed	Latin	✓	✓
Finnish	fi	Uralic	Latin	✓	✓	✓	✓
French	fr	IE:Romance	Latin	✓, (+)	✓, (+)	✓, (+)	✓, (+)
Frisian	fy	IE:Germanic	Latin	✓
Galician	gl	IE:Romance	Latin	✓	✓
Georgian	ka	Kartvelian	Georgian	✓	✓	✓	✓
German	de	IE:Germanic	Latin	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Greek	el	IE:Hellenic	Greek	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Gujarati	gu	IE:Iranian	Gujarati	✓	✓
Hausa	ha	Afro-Asiatic	Latin	✓	✓
Hebrew	he	Afro-Asiatic	Hebrew	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Hindi	hi	IE:Iranian	Devanagari	✓, (+)	✓, (+)	✓, (+)	✓, (+)
Hungarian	hu	Uralic	Latin	✓	✓	✓
Icelandic	is	IE:Germanic	Latin	✓	✓
Indonesian	id	Austronesian	Latin	✓	✓	✓	✓
Irish	ga	IE:Celtic	Latin	✓	✓
Italian	it	IE:Romance	Latin	✓	✓	✓
Japanese	ja	Japonic	Japanese	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Javanese	jv	Austronesian	Latin	✓
Kannada	kn	Dravidian	Kannada	✓
Korean	ko	Koreanic	Korean	✓, (+)	✓, (+)	✓, (+)	✓, (+)
Kurdish	ku	IE:Iranian	Latin	✓	✓
Latin	la	IE:Romance	Latin	✓	✓
Language	iso	Family	Script	13	30	60	75
Latvian	lv	IE:Slavic	Latin	✓	✓
Lithuanian	lt	IE:Slavic	Latin	✓	✓	✓
Macedonian	mk	IE:Slavic	Cyrillic	✓	✓
Malagasy	mg	Austronesian	Latin	✓
Malay	ms	Austronesian	Latin	✓	✓	✓
Malayalam	ml	Dravidian	Malayalam	✓	✓	✓
Marathi	mr	IE:Iranian	Devanagari	✓
Mongolian	mn	Mongolian	Cyrillic	✓	✓	✓
Nepali	ne	IE:Iranian	Devanagari	✓	✓
Norwegian	no	IE:Germanic	Latin	✓	✓
Oriya	or	IE:Iranian	Odia	✓
Oromo	om	Afro-Asiatic	Ge'ez	✓
Pashto	ps	IE:Iranian	Arabic	✓	✓
Persian	fa	IE:Iranian	Arabic	✓	✓
Polish	pl	IE:Slavic	Latin	✓	✓	✓
Portuguese	pt	IE:Romance	Latin	✓	✓
Punjabi	pa	IE:Iranian	Gurmukhi	✓
Romanian	ro	IE:Romance	Latin	✓	✓	✓
Russian	ru	IE:Slavic	Cyrillic	✓,+	✓,+	✓,+	✓,+
Sanskrit	sa	IE:Iranian	Devanagari	✓	✓
Scottish Gaelic	gd	IE:Germanic	Latin	✓
Serbian	sr	IE:Slavic	Cyrillic	✓	✓
Sindhi	sd	IE:Iranian	Arabic	✓	✓
Sinhala	si	IE:Iranian	Sinhala	✓	✓	✓
Slovak	sk	IE:Slavic	Latin	✓	✓	✓
Slovenian	sl	IE:Slavic	Latin	✓	✓
Somali	so	Afro-Asiatic	Latin	✓	✓
Spanish	es	IE:Romance	Latin	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Sundanese	su	Austronesian	Latin	✓
Swahili	sw	Niger-Congo	Latin	✓	✓	✓	✓
Swedish	sv	IE:Germanic	Latin	✓	✓	✓
Tagalog	tl	Austronesian	Latin	✓	✓	✓
Tamil	ta	Dravidian	Tamil	✓	✓	✓	✓
Telugu	te	Dravidian	Telugu	✓	✓
Thai	th	Kra-Dai	Thai	✓,+	✓,+	✓,+	✓,+
Turkish	tr	Turkic	Latin	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Ukrainian	uk	IE:Slavic	Cyrillic	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Urdu	ur	IE:Iranian	Arabic	+, (✓)	+, (✓)	+, (✓)	+, (✓)
Vietnamese	vi	Austroasiatic	Latin	✓,+	✓,+	✓,+	✓,+
Welsh	cy	IE:Celtic	Latin	✓	✓
Xhosa	xh	Niger-Congo	Latin	✓
Yiddish	yi	IE:Germanic	Hebrew	✓
Table 9: List of languages we pre-train  $\checkmark$  on or add + in the different sets (13, 30, 60, 75).  $(\cdot)$  indicates the respectively different pre-training/added languages of models 1 and 2 as described in §5.2 and Table 4. IE stands for Indo-European.