SlowGuess/NAACL_2021_OCR huggingface dataset:
Looking at the per-language breakdowns in Appendix B, we find that mT5 performs well on both high- and low-resource languages. For example, in table 7, we see mT5-XXL outperforms XLM-R by between +3 (English) and +9 (Swahili) points on each individual language on XNLI zero-shot. In table 12 we see similarly strong performance across languages on TyDi QA GoldP (including lower-resource languages like Swahili and Telugu), with mT5-XXL surpassing human performance in four of nine languages on the "in-language" setting.
We run six ablations, modifying various settings, using our Large model as a baseline: (i) increase dropout to 0.1 in hopes of mitigating overfitting on low-resource languages, (ii) decrease sequence length to 512 (as was used in T5), (iii) increase the average noise span length in the pre-training objective to 10 since we observe fewer characters per token than T5, (iv) adjust the language sampling exponent  $\alpha$  to  $\{0.2, 0.7\}$  as used in MMNMT (Arivazhagan et al., 2019) and mBERT (Devlin, 2018), respectively, (v) turn off the "line length filter" in the mC4 data pipeline, and (vi) supplement mC4 with Wikipedia data from 103 languages.
The effect of these ablations on XNLI zero-shot accuracy is shown in table 4. In each case, the average XNLI score is lower than the mT5-Large baseline, justifying our chosen settings. The line
We use the translation data provided by Hu et al. (2020) throughout. On the PAWS-X task, FILTER used translation data from the original task instead. Switching to this data would improve our scores slightly (mT5-XXL  $91.5\rightarrow 92.0$ ).
<sup>9</sup>We use the 2020 Wikipedia data from TensorFlow Datasets, selecting the same languages as mBERT. https://www.tensorflow.org/datasets/catalog/wikipedia
Model	Accuracy
Baseline (mT5-Large)	81.1
Dropout 0.1	77.6
Sequence length 512	80.5
Span length 10	78.6
α = 0.7	80.7
α = 0.2	80.7
No line length filter	79.1
Add Wikipedia data	80.3
length filter provides a  $+2$  point boost, corroborating the findings of Conneau et al. (2020) and Raffel et al. (2020) that filtering low-quality pages from Common Crawl is valuable. Increasing the language sampling exponent  $\alpha$  to 0.7 has the expected effect of improving performance in high-resource languages (e.g. Russian  $81.5 \rightarrow 82.8$ ), while hurting low-resource languages (e.g. Swahili  $75.4 \rightarrow 70.6$ ), with the average effect being negative. Conversely, lowering  $\alpha$  to 0.2 boosts one tail language slightly (Urdu  $73.5 \rightarrow 73.9$ ) but is harmful elsewhere. Detailed per-language metrics on XNLI and the results of our ablations on zero-shot XQuAD are provided in Appendix C, showing similar trends.
5 Zero-Shot Generation
Since mT5 is a generative model, it can output arbitrary text predictions in a free form fashion. This is in contrast to "encoder-only" models like mBERT and XLM(-R) that make a prediction by either extracting it from the input or producing a class label. We found that the lack of constraints during prediction caused mT5 to sometimes have trouble generating a well-formed prediction in a language unseen during fine-tuning. Focusing on XQuAD zero-shot, we find that many of these errors are due to "accidental translation" into the fine-tuning language (English). In this section, we characterize this behavior and demonstrate that it can be counteracted by mixing a small amount of our multilingual pre-training task into the fine-tuning stage.
5.1 Illegal Predictions
In using a generative model for span selection (as in extractive QA tasks), we hope the model learns to generate "legal" spans that are substrings of the provided context. However, unlike encoder-based models like BERT, this is not a hard constraint of
Table 4: Average XNLI zero-shot accuracy of various ablations on our mT5-Large model. Per-language metrics are shown in Appendix C.
Target	Prediction	Explanation
विरूधान्य	विरूधान्य	Decomposed Thai ‘\` into ‘+ \}
लोयर शे माङिजियर	लोयर शे माङिजियर	Decomposed Hindi तिल into तिल + \}
27 - 30 %	27 - 30 %	Replaced full-width percent sign
12 .a	12 .a	Removed superscript
الترجمة الحرفية	الترجمة الحرفية	Arabic “for anaerobic bacteria” ⇒ “anaerobic bacteria”
сторамп绛дз	сторам绛дз	Russian “bit strings (instrumental)” ⇒ “bit strings (nominative)”
seis años	six years	Translated from Spanish
Zweiten Weltkrieg	the Second World War	Translated from German
新英格兰爱国者队	New英格兰爱国者队	Partially translated Chinese “New England Patriots”
хлоро��act	chlorо��act	Partially translated Russian “chloroplast”
Table 5: Illegal mT5-XXL predictions on XQuAD zero-shot, illustrating normalization (top), grammatical adjustment (middle) and translation (bottom).
the model. Notably, T5 learns to always output legal spans on SQuAD, suggesting this is not a major issue for generative models in simple cases.
A more challenging case for generative models is zero-shot cross-lingual span selection. Here, a pretrained multilingual model is fine-tuned on English but tested on other languages. We want the model to generate legal non-English predictions despite having only seen English targets in fine-tuning.
In practice, while mT5 achieves SOTA on the zero-shot variants of XQuAD, MLQA and TyDi QA, illegal predictions are still a problem. For example, on zero-shot XQuAD, a non-trivial portion of mT5 mistakes are in fact illegal spans, for all model sizes (cf. fig. 4 "Baseline"). Through inspection, we find these illegal predictions mainly fall into three categories: (i) normalization, (ii) grammatical adjustment, and (iii) accidental translation. Table 5 provides examples of each type.
Normalization indicates predictions that would be legal, except that "equivalent" Unicode characters have been substituted, so a legal span may be recovered through Unicode NFKC normalization. This is particularly common in Thai, Chinese and Hindi, where most mT5-XXL illegal predictions are resolved by normalization, as seen in fig. 3b.
Grammatical adjustment involves minor morphological changes to the original text. We frequently observe these adjustments when the target span cannot stand as a well-formed answer on its own. For example, mT5-XXL's Arabic and Russian predictions in the middle rows of table 5 are judged by native speakers as correct and grammatical answers to the posed XQuAD questions, while the gold targets are judged as ungrammatical answers. This type of illegal prediction is most common in
(a) mT5-Small
(b) mT5-XXL
Figure 3: Per-language error rates on XQuAD zero-shot, sorted by illegal rate. Incorrect: Not matching the target span. Illegal: Missing from the input context. Illegal after norm: Illegal even after Unicode NFKC normalization is applied to the prediction and context.
languages with extensive grammatical case marking, such as Russian, Turkish and German.
Accidental translation involves the model translating part or all of a contextual span into English (the language of all fine-tuning data). On the one hand, it is remarkable that mT5 performs "spontaneous" translation despite never seeing parallel training data. On the other, as practitioners we would ideally be able to control this behavior.
We observe accidental translation across all model sizes and all XQuAD languages. The problem is most prevalent in mT5-Small and mT5-Base, where from manual inspection, half or more of the illegal predictions within each language exhibit accidental translation, with many of the illegal predictions coming from Greek and Russian, as shown in fig. 3a. While we do observe full phrase translations, a more common occurrence is partial translation, where the model outputs a token or two of English before reverting to the correct target language. The transition may even occur mid-word, as in the prediction "chloroJIaCT", where the first half of the target "XJIoPoIIaCT" (Russian: chloroplast) has been translated to English.
5.2 Preventing Accidental Translation
The most direct solution to avoiding accidental translation on span selection tasks would be to modify our inference procedure. As is common practice
with encoder-based models, we could devise a task-specific fine-tuning mechanism that restricts the model to perform ranking over legal spans, removing the possibility of illegal predictions entirely. While this would likely improve our zero-shot metrics, it is unsatisfying for two reasons: First, it implies taking a step backward from the general text-to-text interface, as different tasks would demand different types of inference. Second, this solution won't extend to more "open-ended" zero-shot generative tasks like summarization, where the legal output space can't be easily delimited.
For these reasons, we consider a more general solution that remains within the text-to-text framework and can apply to all zero-shot generation tasks. Our motivating intuition is that the reason the model outputs English when given a non-English test input is that it has never observed a non-English target during fine-tuning. As English-only finetuning proceeds, the model's assigned likelihood of non-English tokens presumably decreases, eventually reaching the point where English becomes the most likely answer to any question.
To prevent the model from "forgetting" how to generate other languages, we use a strategy inspired by domain/task-adaptive pre-training (Howard and Ruder, 2018; Gururangan et al., 2020): We simply mix in our unsupervised multilingual pre-training task during fine-tuning. A similar approach was explored by Liu et al. (2020b). We use the same mC4 task definition as in pre-training, with two adjustments: First, we remove all "sentinel" tokens (corresponding to non-masked spans in the input text) from the target sequence, as otherwise we observe occasional sentinels in downstream predictions. Second, we reduce the language sampling parameter  $\alpha$  from 0.3 to 0.1. This produces a near-uniform distribution of languages, encouraging the model to treat all languages as equally likely.[10]
With these changes, we mix a small amount of our unsupervised task (covering 101 languages) into XQuAD fine-tuning, at a ratio of just 1:100. Figure 4 shows the results on XQuAD zero-shot error rates. The addition of even this small amount of multilingual data has a marked effect on the mT5-Small and mT5-Base models (where accidental translation was most rampant), reducing the illegal prediction rates by more than  $70\%$  (relative), and contributing to an overall reduction in errors.
10 Alternatively, one could mix in unlabeled data only for a single language at a time. However, we believe this is contrary
Figure 4: Error rates of mT5 on XQuAD zero-shot. Baseline: Fine-tuning on XQuAD alone. Domain Preserving Training (DPT): Mixing in the unsupervised mC4 task with fine-tuning.
In this paper, we introduced mT5 and mC4: massively multilingual variants of the T5 model and C4 dataset. We demonstrated that the T5 recipe is straightforwardly applicable to the multilingual setting, and achieved strong performance on a diverse set of benchmarks. We also characterized illegal predictions that can occur in zero-shot evaluation of multilingual pre-trained generative models, and described a simple technique to avoid this issue. We release all code and pre-trained datasets used in this paper to facilitate future work on multilingual language understanding. $^{11}$
We thank Melvin Johnson for tips on the translate-train procedure for XTREME and Itai Rolnick for help with infrastructure.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguistics.
Diedre Carmo, Marcos Piau, Israel Campiotti, Rodrigo Nogueira, and Roberto Lotufo. 2020. PTT5: Pretraining and validating the t5 model on brazilian portuguese data. arXiv preprint arXiv:2008.09144.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2020. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. arXiv preprint arXiv:2007.07834.
Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. Rethinking embedding coupling in pre-trained language models. arXiv preprint arXiv:2010.12821.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454-470.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32, pages 7059-7069.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
David Crystal. 2008. Two thousand million? English today, 24(1):3-6.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT model. arXiv preprint arXiv:1912.09582.
Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a dutch RoBERTa-based language model. arXiv preprint arXiv:2001.06286.
Jacob Devlin. 2018. Multilingual BERT.  
README. https://github.com/google-research/bert/blob/master/multilingual.md.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association
to the spirit of multilingual models and zero-shot evaluation.
11https://goo.gl/mt5-code
for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. FILTER: An enhanced fusion method for cross-lingual language understanding. arXiv preprint arXiv:2009.05166.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080.
Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
Mihir Kale. 2020. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005.10433.
Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09286.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single QA system. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System
Demonstrations, pages 66-71, Brussels, Belgium.  
Association for Computational Linguistics.
Hang Le, Loic Vial, Jibril Frej, Vincent Segonne, Maxim Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model pre-training for French. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2479-2490, Marseille, France. European Language Resources Association.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettle-moyer. 2020a. Pre-training via paraphrasing. arXiv preprint arXiv:2006.15020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Patrick Lewis, Barlas O'guz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2019. MLQA: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2020b. Exploring fine-tuning techniques for pre-trained cross-lingual models via continual learning. arXiv preprint arXiv:2004.14218.
Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. 2020. Veco: Variable encoder-decoder pre-training for cross-lingual understanding and generation. arXiv preprint arXiv:2010.16046.
Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. Playing with words at the national library of sweden-making a swedish BERT. arXiv preprint arXiv:2007.01658.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoit Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pages 7203-7219, Online. Association for Computational Linguistics.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. WT5?! Training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546.
Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037-1042, Online. Association for Computational Linguistics.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 708-718, Online. Association for Computational Linguistics.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Crosslingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958, Vancouver, Canada. Association for Computational Linguistics.
Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, and Samuel R Bowman. 2020. English intermediate-task training improves zero-shot cross-lingual transfer too. arXiv preprint arXiv:2005.13013.
Marco Polignano, Pierpaolo Basile, Marco de Gemmis, Giovanni Semeraro, and Valerio Basile. 2019. AlBERTo: Italian BERT language understanding model for NLP challenging tasks based on tweets. In CLiC-it.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15-18, Minneapolis, Minnesota. Association for Computational Linguistics.
Noam Shazeer. 2020. GLU variants improve transformer. arXiv preprint arXiv:2002.05202.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.