SlowGuess/NAACL_2021_OCR huggingface dataset:
Linting Xue* Noah Constant* Adam Roberts* Mihir Kale Rami Al-Rfou Aditya Siddhant Aditya Barua Colin Raffel Google Research
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.
Current natural language processing (NLP) pipelines often make use of transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on a downstream task of interest (Ruder et al., 2019). The success of this paradigm is partially thanks to the release of parameter checkpoints for pre-trained models. These checkpoints allow members of the NLP community to quickly attain strong performance on many tasks without needing to perform expensive pre-training themselves. As one example, the pre-trained checkpoints for the "Text-to-Text Transfer Transformer" (T5) model released by Raffel et al. (2020) have been used to achieve state-of-the-art results on many benchmarks (Khashabi et al., 2020; Roberts et al., 2020; Kale, 2020; Izacard and Grave, 2020; Nogueira et al., 2020; Narang et al., 2020, etc.).
Unfortunately, many of these language models were pre-trained solely on English-language text.
This significantly limits their use given that roughly  $80\%$  of the world population does not speak English (Crystal, 2008). One way the community has addressed this English-centricity has been to release dozens of models, each pre-trained on a single non-English language (Carmo et al., 2020; de Vries et al., 2019; Le et al., 2020; Martin et al., 2020; Delobelle et al., 2020; Malmsten et al., 2020; Nguyen and Tuan Nguyen, 2020; Polignano et al., 2019, etc.). A more general solution is to produce multilingual models that have been pre-trained on a mixture of many languages. Popular models of this type are mBERT (Devlin, 2018), mBART (Liu et al., 2020a), and XLM-R (Conneau et al., 2020), which are multilingual variants of BERT (Devlin et al., 2019), BART (Lewis et al., 2020b), and RoBERTa (Liu et al., 2019), respectively.
In this paper, we continue this tradition by releasing mT5, a multilingual variant of T5. Our goal with mT5 is to produce a massively multilingual model that deviates as little as possible from the recipe used to create T5. As such, mT5 inherits all of the benefits of T5 (described in section 2), such as its general-purpose text-to-text format, its design based on insights from a large-scale empirical study, and its scale. To train mT5, we introduce a multilingual variant of the C4 dataset called mC4. mC4 comprises natural text in 101 languages drawn from the public Common Craw web scrape. To validate the performance of mT5, we include results on several benchmark datasets, showing state-of-the-art results in many cases. Finally, we characterize a problematic behavior of pre-trained generative multilingual language models in the zero-shot setting, where they erroneously translate part of their prediction into the wrong language. To address this "accidental translation", we describe a simple procedure that involves mixing in unlabeled pre-training data during fine-tuning and demonstrate that it dramatically alleviates this issue. We release our pre-trained models and code
*Equal Contribution. Please direct correspondence to lintingx@google.com, nconstant@google.com, adarob@google.com, and craffel@google.com https://goo.gl/mt5-code
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498 June 6-11, 2021. ©2021 Association for Computational Linguistics
so that the community can leverage our work.
2 Background on T5 and C4
In this section, we provide a short overview of T5 and the C4 pre-training dataset. Further details are available in Raffel et al. (2020).
T5 is a pre-trained language model whose primary distinction is its use of a unified "text-to-text" format for all text-based NLP problems. This approach is natural for generative tasks (such as machine translation or abstractive summarization) where the task format requires the model to generate text conditioned on some input. It is more unusual for classification tasks, where T5 is trained to output the literal text of the label (e.g. "positive" or "negative" for sentiment analysis) instead of a class index. The primary advantage of this approach is that it allows the use of exactly the same training objective (teacher-forced maximum-likelihood) for every task, which in practice means that a single set of hyperparameters can be used for effective fine-tuning on any downstream task. Similar unifying frameworks were proposed by Keskar et al. (2019) and McCann et al. (2018). Given the sequence-to-sequence structure of this task format, T5 uses a basic encoder-decoder Transformer architecture as originally proposed by Vaswani et al. (2017). T5 is pre-trained on a masked language modeling "span-corruption" objective, where consecutive spans of input tokens are replaced with a mask token and the model is trained to reconstruct the masked-out tokens.
An additional distinguishing factor of T5 is its scale, with pre-trained model sizes available from 60 million to 11 billion parameters. These models were pre-trained on around 1 trillion tokens of data. Unlabeled data comes from the C4 dataset, which is a collection of about 750GB of English-language text sourced from the public Common crawl web scrape. C4 includes heuristics to extract only natural language (as opposed to boilerplate and other gibberish) in addition to extensive dedduplication. The pre-training objective, model architecture, scaling strategy, and many other design choices for T5 were chosen based on a large-scale empirical study described in detail in Raffel et al. (2020).
3 mC4 and mT5
Our goal in this paper is to create a massively multilingual model that follows T5's recipe as closely as possible. Towards this end, we develop an ex
tended version of the C4 pre-training dataset that covers 101 languages and introduce changes to T5 to better suit this multilinguality.
3.1 mC4
The C4 dataset was explicitly designed to be English only: any page that was not given a probability of at least  $99\%$  of being English by langdetect² was discarded. In contrast, for mC4 we use c1d3³ to identify over 100 languages. Since some of these languages are relatively scarce on the internet, we make use of all of the 71 monthly web scrapes released so far by Common Crawl. This is dramatically more source data than was used for C4, for which the April 2019 web scrape alone was enough to provide plenty of English-language data.
An important heuristic filtering step in C4 was the removal of lines that did not end in an English terminal punctuation mark. Since many languages do not use English terminal punctuation marks, we instead apply a "line length filter" that requires pages to contain at least three lines of text with 200 or more characters. Otherwise, we follow C4's filtering by deduplicating lines across documents and removing pages containing bad words. Finally, we detect each page's primary language using c1d3 and remove those with a confidence below  $70\%$ .
After these filters are applied, we group the remaining pages by language and include in the corpus all languages with 10,000 or more pages. This produces text in 107 "languages" as defined by c1d3. However, we note that six of these are just script variants of the same spoken language (e.g. ru is Russian in Cyrillic script and ru-Latin is Russian in Latin script). A histogram of the page counts for each language is shown in fig. 1. Detailed dataset statistics including per-language token counts are shown in Appendix A.
3.2 mT5
The model architecture and training procedure that we use for mT5 closely follows that of T5. Specifically, we base mT5 on the "T5.1.1" recipe, which improves upon T5 by using GeGLU nonlinearities (Shazeer, 2020), scaling both  $\mathrm{d}_{\mathrm{model}}$  and  $\mathrm{d}_{\mathrm{ff}}$  instead
$^{2}$ https://pypi.org/project/langdetect/  
 $^{3}$ https://github.com/google/cld3  
 $^{4}$ https://github.com/LDNOOBW/  
 $^{5}$ https://github.com/google-research/text-to-text-transfer-transformer/blob/master/releases_checkpoints.md#t511
Figure 1: Page counts per language in mC4 (left axis), and percentage of mT5 training examples coming from each language, for different language sampling exponents  $\alpha$  (right axis). Our final model uses  $\alpha = 0.3$ .
Model	Architecture	Parameters	# languages	Data source
mBERT (Devlin, 2018)	Encoder-only	180M	104	Wikipedia
XLM (Conneau and Lample, 2019)	Encoder-only	570M	100	Wikipedia
XLM-R (Conneau et al., 2020)	Encoder-only	270M – 550M	100	Common Crawl (CCNet)
mBART (Lewis et al., 2020b)	Encoder-decoder	680M	25	Common Crawl (CC25)
MARGE (Lewis et al., 2020a)	Encoder-decoder	960M	26	Wikipedia or CC-News
mT5 (ours)	Encoder-decoder	300M – 13B	101	Common Crawl (mC4)
Table 1: Comparison of mT5 to existing massively multilingual pre-trained language models. Multiple versions of XLM and mBERT exist; we refer here to the ones that cover the most languages. Note that XLM-R counts five Romanized variants as separate languages, while we ignore six Romanized variants in the mT5 language count.
of just  $\mathrm{d_{ff}}$  in the larger models, and pre-training on unlabeled data only with no dropout. We refer to Raffel et al. (2020) for further details on T5.
A major factor in pre-training multilingual models is how to sample data from each language. Ultimately, this choice is a zero-sum game: If low-resource languages are sampled too often, the model may overfit; if high-resource languages are not trained on enough, the model will underfit. We therefore take the approach used in (Devlin, 2018; Conneau et al., 2020; Arivazhagan et al., 2019) and boost lower-resource languages by sampling examples according to the probability  $p(L) \propto |L|^{\alpha}$ , where  $p(L)$  is the probability of sampling text from a given language during pre-training and  $|L|$  is the number of examples in the language. The hyperparameter  $\alpha$  (typically with  $\alpha < 1$ ) allows us to control how much to "boost" the probability of training on low-resource languages. Values used by prior work include  $\alpha = 0.7$  for mBERT (Devlin, 2018),  $\alpha = 0.3$  for XLM-R (Conneau et al., 2020), and  $\alpha = 0.2$  for MMNMT (Arivazhagan et al., 2019). We tried all three of these values (ablation results in section 4.2) and found  $\alpha = 0.3$  to give a reasonable compromise between performance on high- and low-resource languages.
The fact that our model covers over 100 languages necessitates a larger vocabulary. Following XLM-R (Conneau et al., 2018), we increase the vocabulary size to 250,000 wordpieces. As in T5, we
use SentencePiece (Kudo and Richardson, 2018; Kudo, 2018) models trained with the language sampling rates used during pre-training. To accommodate languages with large character sets like Chinese, we use a character coverage of 0.99999 and enable SentencePiece's "byte-fallback" feature to ensure that any string can be uniquely encoded.
3.3 Comparison to Related Models
To contextualize our new model, we provide a brief comparison with existing massively multilingual pre-trained language models. For brevity, we focus on models that support more than a few dozen languages. Table 1 gives a high-level comparison of mT5 to the most similar models.
mBERT (Devlin, 2018) is a multilingual version of BERT (Devlin et al., 2019). Similar to our approach with mT5, mBERT follows the BERT recipe as closely as possible (same architecture, objective, etc.). The primary difference is the training set: Instead of training on English Wikipedia and the Toronto Books Corpus, mBERT is trained on up to 104 languages from Wikipedia. XLM (Conneau and Lample, 2019) is also based on BERT but applies improved methods for pre-training multilingual language models including explicitly crosslingual pre-training objectives. Many pre-trained versions of XLM have been released; the most massively-multilingual variant was trained on 100 languages from Wikipedia. XLM-R (Conneau
Model	Sentence pair	Structured	Question answering
XNLI	PAWS-X	WikiAnn NER	XQuAD	MLQA	TyDi QA-GoldP
Metrics	Acc.	Acc.	F1	F1 / EM	F1 / EM	F1 / EM
Cross-lingual zero-shot transfer (models fine-tuned on English data only)
mBERT	65.4	81.9	62.2	64.5 / 49.4	61.4 / 44.2	59.7 / 43.9
XLM	69.1	80.9	61.2	59.8 / 44.3	48.5 / 32.6	43.6 / 29.1
InfoXLM	81.4	-	-	- / -	73.6 / 55.2	- / -
X-STILTs	80.4	87.7	64.7	77.2 / 61.3	72.3 / 53.5	76.0 / 59.5
XLM-R	79.2	86.4	65.4	76.6 / 60.8	71.6 / 53.2	65.1 / 45.0
VECO	79.9	88.7	65.7	77.3 / 61.8	71.7 / 53.2	67.6 / 49.1
RemBERT	80.8	87.5	70.1	79.6 / 64.0	73.1 / 55.0	77.0 / 63.0
mT5-Small	67.5	82.4	50.5	58.1 / 42.5	54.6 / 37.1	36.4 / 24.4
mT5-Base	75.4	86.4	55.7	67.0 / 49.0	64.6 / 45.0	59.1 / 42.4
mT5-Large	81.1	88.9	58.5	77.8 / 61.5	71.2 / 51.7	68.4 / 50.9
mT5-XL	82.9	89.6	65.5	79.5 / 63.6	73.5 / 54.5	77.8 / 61.8
mT5-XXL	85.0	90.0	69.2	82.5 / 66.8	76.0 / 57.4	82.0 / 67.3
Translate-train (models fine-tuned on English data plus translations in all target languages)
XLM-R	82.6	90.4	-	80.2 / 65.9	72.8 / 54.3	66.5 / 47.7
FILTER + Self-Teaching	83.9	91.4	-	82.4 / 68.0	76.2 / 57.7	68.3 / 50.9
VECO	83.0	91.1	-	79.9 / 66.3	73.1 / 54.9	75.0 / 58.9
mT5-Small	72.0	79.9	-	64.3 / 49.5	56.6 / 38.8	49.8 / 35.6
mT5-Base	79.8	89.3	-	75.3 / 59.7	67.6 / 48.5	66.4 / 51.0
mT5-Large	84.4	91.2	-	81.2 / 65.9	73.9 / 55.2	75.7 / 60.1
mT5-XL	85.3	91.0	-	82.7 / 68.1	75.1 / 56.6	80.1 / 65.0
mT5-XXL	87.1	91.5	-	85.2 / 71.3	76.9 / 58.3	83.3 / 69.4
In-language multitask (models fine-tuned on gold data in all target languages)
mBERT	-	-	89.2	-	-	77.6 / 68.0
mT5-Small	-	-	86.4	-	-	74.0 / 62.7
mT5-Base	-	-	88.2	-	-	79.7 / 68.4
mT5-Large	-	-	89.7	-	-	85.3 / 75.3
mT5-XL	-	-	91.3	-	-	87.6 / 78.4
mT5-XXL	-	-	92.2	-	-	88.7 / 79.5
Table 2: Results on XTREME sentence-pair classification, structured prediction and question answering tasks. mBERT metrics are from Hu et al. (2020). Metrics for XLM, InfoXLM, X-STILTs and XLM-R are from Fang et al. (2020), though Conneau et al. (2020) report better performance of XLM-R on XNLI (80.9). All other metrics are from the original sources: FILTER (Fang et al., 2020), VECO (Luo et al., 2020) and RemBERT (Chung et al., 2020). For the "translate-train" setting, we include English training data, so as to be comparable with Fang et al. (2020) and Luo et al. (2020). This differs from the XTREME "translate-train" setup of Hu et al. (2020). For mT5 results on TyDi QA zero-shot, we report the median across five fine-tuning runs, as we observed high variance across runs. Full results for all languages in all tasks are provided in the appendix.
et al., 2020) is an improved version of XLM based on the RoBERTa model (Liu et al., 2019). XLM-R is trained with a cross-lingual masked language modeling objective on data in 100 languages from Commoncrawl. To improve the pre-training data quality, pages from Commoncrawl were filtered by an n-gram language model trained on Wikipedia (Wenzek et al., 2020). mBART (Liu et al., 2020a) is a multilingual encoder-decoder model that is based on BART (Lewis et al., 2020b). mBART is trained with a combination of span masking and sentence shuffling objectives on a subset of 25 languages from the same data as XLM-R. MARGE (Lewis et al., 2020a) is a multilingual encoder-decoder model that is trained to reconstruct a docu
ment in one language by retrieving documents in other languages. It uses data in 26 languages from Wikipedia and CC-News (Liu et al., 2019).
To validate the performance of mT5, we evaluate our models on 6 tasks from the XTREME multilingual benchmark (Hu et al., 2020): the XNLI (Conneau et al., 2018) entailment task covering 14 languages; the XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2019), and TyDi QA (Clark et al., 2020) reading comprehension benchmarks with 10,
6Standard deviations of mT5 models on TyDi QA zero-shot across five runs are: Small: 0.44, Base: 1.38, Large: 3.66, XL: 1.29, XXL: 0.20.
7, and 11 languages respectively; the Named Entity Recognition (NER) dataset of WikiAnn (Pan et al., 2017) restricted to the 40 languages from XTREME (Hu et al., 2020), and the PAWS-X (Yang et al., 2019) paraphrase identification dataset with 7 languages. We cast all tasks into the text-to-text format, i.e. generating the label text (XNLI and PAWS-X), entity tags and labels (WikiAnn NER), or answer (XQuAD, MLQA, and TyDi QA) directly in a generative fashion. For NER, if there are multiple entities, they are concatenated in the order they appear, and if there are no entities then the target text is "None". We consider three variants of these tasks: (1) "zero-shot", where the model is fine-tuned only on English data, (2) "translate-train", adding machine translations from English into each target language, and (3) "in-language multitask", training on gold data in all target languages. For brevity, we refer to Hu et al. (2020) for further details on these benchmarks.
Following the original T5 recipe, we consider five model sizes: Small ( $\approx$  300M parameters), Base (580M), Large (1.2B), XL (3.7B), and XXL (13B). The increase in parameter counts compared to the corresponding T5 model variants comes from the larger vocabulary used in mT5. Note that, because mT5 is an encoder-decoder model, it has roughly twice as many parameters as correspondingly-sized encoder-only models such as XLM-R. For example, the "Large" variant of XLM-R has 550 million parameters whereas mT5-Large has around 1 billion. However, the computational cost for text classification is roughly the same: In both cases, the model processes a length- $T$  input sequence with an encoder of approximately equal size. In an encoder-only model like XLM-R, the encoder processes one additional "CLS" token, which is used to generate the representation for classification. In mT5, the decoder typically produces two additional tokens: the class label and an end-of-sequence token. Since the decoder has the same architecture (ignoring encoder-decoder attention) as the encoder, the computational cost of classification with mT5 typically amounts to the cost of processing  $T + 2$  tokens compared to  $T + 1$  for an encoder-only model. However, encoder-decoder architectures have the additional benefit of being applicable to generative tasks like abstractive summarization or dialog.
We pre-train our mT5 model variants for 1 million steps on batches of 1024 length-1024 input
sequences, corresponding to roughly 1 trillion input tokens total. This is the same amount of pretraining as T5 and about  $\frac{1}{6}$  as much as XLM-R. $^{7}$  Note that our pre-training dataset is large enough that we only complete a fraction of an epoch for high-resource languages (e.g. only covering  $2\%$  of the English data). While XLM-R's pre-training corpus CC-100 is 20 times smaller than mC4, XLM-R nevertheless pre-trains for more steps, and sees over 6 times more tokens in pre-training.
We use the same inverse square-root learning rate schedule used by T5 during pre-training, with the learning rate set to  $1 / \sqrt{\max(n,k)}$  where  $n$  is the current training iteration and  $k = 10^4$  is the number of warm-up steps. Following the T5.1.1 recipe, we do not apply dropout during pre-training. We use the same self-supervised objective as T5, with  $15\%$  of tokens masked and an average noise span length of 3. We ablate some of these experimental details in section 4.2.
For fine-tuning, we use a constant learning rate of 0.001 and dropout rate of 0.1 for all tasks. We use a batch size of  $2^{17}$  for most tasks, but decrease to  $2^{16}$  for WikiAnn NER zero-shot, due to the small size of the training, and increase to  $2^{20}$  tokens for XNLI, which we found gave better performance. For early stopping, we save checkpoints every 200 steps and choose the checkpoint with the highest performance on the standard validation sets specified by XTREME.
Table 2 presents our main results, with per-language breakdowns for each task given in Appendix B. Our largest model mT5-XXL exceeds state-of-the-art on all classification and QA tasks and is near SOTA on NER (69.2 vs. 70.1). Note that unlike our model, InfoXLM (Chi et al., 2020) and VECO (Luo et al., 2020) benefit from parallel training data, while X-STILTs (Phang et al., 2020) leverages labeled data from tasks similar to the target task. Overall, our results highlight the importance of model capacity in cross-lingual representation learning and suggest that scaling up a simple pre-training recipe can be a viable alternative to more complex techniques relying on LM filtering, parallel data, or intermediate tasks.
In the "translate-train" setting, we exceed state
$^{7}$ XLM-R Large sees 6.3 trillion tokens during pre-training (1.5 million batches of 8192 sequences of 512 tokens), and uses a packing mechanism similar to T5 to minimize the number of "wasted" padding tokens.
T5	mT5
Small	87.2 / 79.1	84.7 / 76.4
Base	92.1 / 85.4	89.6 / 83.8
Large	93.8 / 86.7	93.0 / 87.0
XL	95.0 / 88.5	94.5 / 88.9
XXL	96.2 / 91.3	95.6 / 90.4
Table 3: Comparison of T5 vs. mT5 on SQuAD question answering (F1/EM).
Figure 2: Average F1 on the TyDi QA GoldP task across languages. Performance improves with increasing model capacity. The importance of in-language training data (whether gold In-Language Multitask or synthetic Translate-Train) decreases with model scale, as seen by Zero-Shot closing the quality gap.
of-the-art on all XTREME classification and QA tasks. For these tasks, we fine-tune on the combination of the labeled English data and machine translations thereof. $^{8}$  This allows direct comparison with both FILTER (Fang et al., 2020) as well as the XLM-R baseline of Fang et al. (2020). Note that this setup differs from XTREME "translate-train" (Hu et al., 2020), which excludes English.
Figure 2 shows that model capacity is key to improving performance on variants of the TyDi QA GoldP task in the absence of "gold" multilingual data: For the smallest model, training on gold datasets (in-language multitask) achieves dramatically better performance than using weakly supervised data (translate-train) or English-only data (zero-shot), whereas the gap between these three settings is much smaller for the largest model. For our two largest models, zero-shot and translate-train performance is nearly the same, showing that machine translations of the monolingual dataset bring diminishing returns as model capacity in-
creases. Overall, these trends point to the possibility of avoiding the costly step of annotating data in more than one language when using large models.
Massively multilingual models have been observed to underperform on a given language when compared to a similarly-sized "dedicated" model trained specifically for that language (Arivazhagan et al., 2019). To quantify this effect, we compare the performance of mT5 and T5 when fine-tuned on the SQuAD reading comprehension benchmark (Rajpurkar et al., 2016). The results are shown in table 3, with results for T5 reproduced from Raffel et al. (2020). While the Small and Base mT5 models fall short of their English T5 counterparts, we find that the larger models close the gap. This suggests there may be a turning point past which the model has enough capacity to effectively learn 101 languages without significant interference effects.