SlowGuess/NAACL_2022_OCR huggingface dataset:
Rachel Reid
machelreid@weblab.t.u-tokyo.ac.jp
Mikel Artetxe
artetxe@fb.com
Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora, and do not make use of the strong cross-lingual signal contained in parallel data. In this paper, we present PARADISE (PARALLEL & Denoising Integration in SEquence-to-sequence models), which extends the conventional denoising objective used to train these models by (i) replacing words in the noised sequence according to a multilingual dictionary, and (ii) predicting the reference translation according to a parallel corpus instead of recovering the original sequence. Our experiments on machine translation and cross-lingual natural language inference show an average improvement of 2.0 BLEU points and 6.7 accuracy points from integrating parallel data into pretraining, respectively, obtaining results that are competitive with several popular models at a fraction of their computational cost.
Multilingual sequence-to-sequence pretraining has achieved strong results both in cross-lingual classification (Xue et al., 2021) and machine translation (Liu et al., 2020). These models are usually pretrained on combined monolingual corpora in multiple languages using some form of denoising objective. More concretely, they noise each sequence  $x$  with a noisig function  $g_{\phi}$ , and maximize the probability of recovering  $x$  given  $g_{\phi}(x)$ :
$$
\ell_ {\text {m o n o}} (x) = - \log P \left(x \mid g _ {\phi} (x)\right) \tag {1}
$$
Common noising functions include sentence-permutation and span masking (Lewis et al., 2020; Liu et al., 2020).
While these methods obtain strong cross-lingual performance without parallel data, they are usually
trained at a scale that is prohibitive for most NLP practitioners. At the same time, it has been argued that the strict unsupervised scenario is not realistic (Artetxe et al., 2020), and parallel data could provide a stronger signal and make training more efficient.
Motivated by this, we propose PARADISE, a pretraining method for sequence-to-sequence models that exploits both word-level and sentence-level parallel data. The core idea of our approach is to augment the conventional denoising objective introduced above by (i) replacing words in the noised sequence according to a bilingual dictionary, and (ii) predicting the reference translation rather than the input sequence. Despite their simplicity, we find that both techniques bring substantial gains over conventional pretraining on monolingual data, as evaluated both in machine translation and zero-shot cross-lingual transfer. Our results are competitive with several popular models despite using only a fraction of the compute, providing strong support for the importance of the inclusion of parallel information in smaller-scale multilingual pretraining methods.
As illustrated in Figure 1, we propose two methods for introducing parallel data into pretraining: dictionary denoising and bitext denoising.
Dictionary denoising. Our first method encourages learning similar representations at the word-level by introducing anchor words through multilingual dictionaries (Conneau et al., 2020b). Let  $D_{l}(w)$  denote the translation of word  $w$  into language  $l \in L$  according to the dictionary  $D$ . Given the source sentence  $x = (x_{1}, x_{2}, \ldots, x_{n})$ , we define its noised version  $g_{\psi}(x) = (\tilde{x}_{1}, \tilde{x}_{2}, \ldots, \tilde{x}_{n})$ , where  $\tilde{x}_{i} = D_{l}(x_{i})$  with probability  $\frac{p_r}{|L|}$  and  $\tilde{x}_{i} = x_{i}$  otherwise (i.e. we replace each word with its translation into a random language with probability
$^{1}$ Source code available at https://github.com/machelreid/paradise
Human Language Technologies, pages 800 - 810
(a) Dictionary Denoising
(b) Bitext Denoising
Figure 1: Our proposed techniques for integrating parallel data into sequence-to-sequence pretraining.
$p_r)$ . We set  $p_r = 0.4$ . Given the dictionary-noised sentence, we train our model using the denoising auto-encoding objective in Eq. 1:
$$
\ell_ {\mathrm {d i c t}} (x) = - \log P \left(x \mid g _ {\phi} \left(g _ {\psi} (x)\right)\right) \tag {2}
$$
Bitext denoising. Our second approach encourages learning from both monolingual and parallel data sources, by including translation data in the pretraining process. Given a source-target bitext pair  $(x,y)$  in the parallel corpus, assumed to be semantically equivalent, we model the following:
$$
\ell_ {\text {b i t e x t}} (x, y) = - \log P \left(y \mid g _ {\phi} (x)\right) \tag {3}
$$
in which we optimize the likelihood of generating the target sentence  $y$  conditioned on the noised version of the source sentence,  $g_{\phi}(x)$ .<sup>2</sup>
Combined objective. Our final objective combines  $\ell_{\mathrm{mono}}$ ,  $\ell_{\mathrm{dict}}$  and  $\ell_{\mathrm{bitext}}$ . Given that our corpus contains languages with varying data sizes, we sample sentences using the exponential sampling technique from Conneau and Lample (2019). We use  $\alpha_{\mathrm{mono}} = 0.5$  to sample from the monolingual corpus, and  $\alpha_{\mathrm{bitext}} = 0.3$  to sample from the parallel corpus. To prevent over-exposure to English on the decoder side when sampling from the parallel corpus, we halve the probability of to-English directions and renormalize the probabilities. In addition, given that we have fewer amounts of parallel data (used for  $\ell_{\mathrm{bitext}}$ ) than monolingual data (used for  $\ell_{\mathrm{mono}}$  and  $\ell_{\mathrm{dict}}$ ), we sample between each task using  $\alpha_{\mathrm{task}} = 0.3$ .
We pretrain our models on 20 languages (English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese,
Hindi, Swahili, Urdu, Japanese, Basque, Romanian, Sinhala and Nepalese), and evaluate them on machine translation and cross-lingual classification.
3.1 Pretraining
Data. We use Wikipedia as our monolingual corpus, and complement it with OSCAR (Ortiz Suárez et al., 2020), and CC100 (Conneau et al., 2020a) for low-resource languages. For a fair comparison with monolingually pretrained baselines, we use the same parallel data as in our downstream machine translation experiments (detailed in §3.2). In addition, we train a separate variant (detailed below) using additional parallel data from ParaCrawl (Esplà et al., 2019), UNPC (Ziemski et al., 2016), CCAigned (El-Kishky et al., 2020), and OpenSubtitles (Lison and Tiedemann, 2016).<sup>4</sup> We tokenize all data using SentencePiece (Kudo and Richardson, 2018) with a joint vocabulary of 125k subwords. We use bilingual dictionaries from FLoRes<sup>5</sup> (Guzmán et al., 2019) for Nepalese and Sinhala, and MUSE<sup>6</sup> (Lample et al., 2018) for the rest of languages. Refer to Appendix A for more details.
Models. We use the same architecture as BART-base (Lewis et al., 2020), totaling  $\sim 196\mathrm{M}$  parameters, and train for  $100\mathrm{k}$  steps with a batch size of  $\sim 520\mathrm{k}$  tokens. This takes around a day on 32 NVIDIA V100 16GB GPUs. As discussed before, we train two variants of our full model: PARADISE, which uses the same parallel data as the machine translation experiments, and PARADISE++, which uses additional parallel data. To better understand the contribution of each objective, we train two additional models without dictionary denoising, which we name PARADISE ( $\mathbf{w}/\mathbf{o}$  dict.) and PARADISE++ ( $\mathbf{w}/\mathbf{o}$  dict.), as well as a model without bitext denoising, which we name PARADISE++ (only dict.). Finally, we train a baseline system using the monolingual objective alone,
2To make our pretraining sequence length consistent with  $\ell_{\mathrm{mono}}$  and  $\ell_{\mathrm{dict}}$  , we concatenate randomly sampled sentence pairs from the same language pair to fit the maximum length. 3We use the same noising function  $g_{\phi}$  used by Lewis et al. (2020) and Liu et al. (2020).
4We cap the size of each language pair to 2GB. 5https://github.com/facebookresearch/fores 6https://github.com/facebookresearch/MUSE
Languages	En-Vi	En-Tr	En-Ja	En-Ar	En-Ne	En-Ro	En-Si	En-Hi	En-Es	En-Fr
Data Source	IWSLT15	WMT17	IWSLT17	IWSLT17	FLoRes	WMT16	FLoRes	IITB	WMT13	WMT14
Size	133K	207K	223K	250K	564K	608K	647K	1.56M	15M	41M
Direction	← →	← →	← →	← →	← →	← →	← →	← →	← →	← →
Random init.	23.6	24.8	12.2	9.5	10.4	12.3	27.5	16.9	7.6	4.3	34.0	34.3	7.2	1.2	10.9	14.2	32.1	31.4	37.0	38.9
mBART (ours)	29.1	31.5	21.3	15.8	15.7	17.3	32.1	19.2	10.3	6.1	34.3	34.9	11.0	2.7	20.2	19.0	29.8	30.4	36.0	38.2
PARADISE	30.0	32.6	23.5	17.2	17.2	19.2	35.3	21.1	13.7	7.9	35.9	36.5	14.0	3.7	23.6	20.7	32.6	32.7	37.8	39.8
which we refer to as mBART (ours). This follows the original mBART work (Liu et al., 2020), but is directly comparable to the rest of our models in terms of data and hyperparameters.
3.2 Downstream settings
Machine translation. Following Liu et al. (2020), we evaluate our models on sentence-level machine translation from and to English using the following datasets: IWSLT (Cettolo et al., 2015, 2017) for Vietnamese, Japanese and Arabic, WMT (Callison-Burch et al., 2009a,b; Bojar et al., 2016, 2017) for Spanish, French, Romanian and Turkish, FLoRes (Guzmán et al., 2019) for Sinhala and Nepalese, and IITB (Kunchukuttan et al., 2018) for Hindi. We report performance in BLEU as detailed in Appendix C.
Cross-lingual classification. We evaluate our models on zero-shot cross-lingual transfer on XNLI (Conneau et al., 2018) and PAWS- $X^7$  (Yang et al., 2019), where we finetune on English data and test performance on other languages. We develop a new approach for applying sequence-to-sequence models for classification: feeding the sequence into both the encoder and decoder, and taking the concatenation of the encoder's  $
$  representation and the decoder's  $
$  representation as the input of the classification head. We provide an empirical rationale for this in Appendix E. We finetune all models with a batch size of 64 and a learning rate of  $2 \times 10^{-5}$  for a maximum of 100k iterations, performing early stopping on the validation set.
4.1 Machine translation
As shown in Table 1, PARADISE consistently outperforms our mBART baseline across all language pairs. Note that these two models have seen the exact same corpora, but mBART uses the parallel
Table 1: Machine translation results. Random initialization numbers taken from Liu et al. (2020).
Lang. pair (En-XX)	Tr	Ro	Si	Hi	Es	AvgΔ
mBART (ours)	15.8	34.9	2.7	19.0	30.4	20.6±0.0
PARADISE (w/o dict.)	16.8	36.2	3.2	20.5	32.4	21.8+1.2
PARADISE	17.2	36.5	3.7	20.7	32.7	22.2+1.6
PARADISE++	19.0	37.3	4.2	20.7	33.0	22.8+2.2
Lang. pair (XX-En)	Tr	Ro	Si	Hi	Es	AvgΔ
mBART (ours)	21.3	34.3	11.0	20.2	29.8	23.3±0.0
PARADISE (w/o dict.)	23.2	35.6	13.2	22.3	31.6	25.2+1.9
PARADISE	23.5	35.9	14.0	23.6	32.6	25.9+2.6
PARADISE++	24.9	36.8	15.1	23.5	32.9	26.6+3.3
Table 2: Ablation results on machine translation.
data for finetuning only, whereas PARADISE also uses it at the pretraining stage. This suggests that incorporating parallel data into pretraining helps learn better representations, which results in better downstream performance.
Table 2 reports additional ablation results on a subset of languages. As can be seen, removing dictionary denoising hurts, but is still better than our mBART baseline. This shows that both of our proposed approaches—dictionary denoising and bitext denoising—are helpful and complementary. Finally, PARADISE++ improves over PARADISE, indicating that a more balanced corpus with more parallel data is helpful.
4.2 Cross-lingual classification
We report XNLI results in Table 3 and PAWS-X results in Appendix F. Our proposed approach outperforms mBART in all languages by a large margin. To our surprise, we also observe big gains in English. We conjecture that this could be explained by bitext denoising providing a stronger training signal from all tokens akin to ELECTRA (Clark et al., 2020), whereas monolingual denoising only gets effective signal from predicting the masked portion. In addition, given that we are using parallel data between English and other languages, PARADISE ends up seeing much more English text compared to mBART—yet a similar amount in the rest of languages—which could also contribute to
$^{7}$ Following Hu et al. (2021), we use English, German, Spanish, French and Chinese for PAWS-X.
Model	en	zh	es	de	ar	ur	ru	bg	el	fr	hi	sw	th	tr	vi	avg
mBART (ours)	77.5	68.0	70.7	68.8	66.7	62.2	68.6	72.1	69.6	70.1	63.4	62.6	66.6	65.0	69.7	68.1
PARADISE++ (only dict.)	79.6	70.9	74.9	75.4	64.4	66.0	69.0	72.2	75.4	73.4	63.9	65.1	70.9	69.0	72.1	70.8
PARADISE	83.4	73.8	77.6	76.0	72.4	65.1	74.0	74.4	73.2	77.7	70.6	66.2	70.4	72.1	75.3	73.5
PARADISE++ (w/o dict.)	83.3	72.9	77.2	75.7	64.4	66.9	73.4	74.8	75.7	77.7	68.5	67.4	71.0	73.3	75.0	73.1
PARADISE++	83.0	74.0	79.0	76.5	68.5	66.8	74.3	76.0	76.4	77.7	70.2	70.5	72.3	74.2	75.4	74.3
Table 3: Accuracy of zero-shot crosslingual classification on the XNLI dataset.
Model	#Langs	Task	Params.	Est. GPU Days	Data (GB)	XNLI	PAWS-X	MT
mBERT (Devlin et al., 2019)†	104	MLM	179M (0.9x)	—	60	65.4	86.2	—
MMTE (Siddhant et al., 2019)†	102	Translation	375M (1.9x)	—	5000	67.4	85.6	—
mT5-small (Xue et al., 2021)	101	Eq. 1	300M (1.5x)	—	27000	67.5	85.8	—
mT6 (Chi et al., 2021a)	94	SC+PNAT+TSC	300M (1.5x)	40 (1.3x)	2120	64.7	86.6	—
AMBER (Hu et al., 2021)	104	MLM+TLM	179M (0.9x)	1000 (31x)	100	71.6	89.2	—
XLM-15 (Conneau and Lample, 2019)‡	15	MLM+TLM	250M (1.3x)	450 (14x)	100	72.6	88.0	—
XLM-R-base (Conneau et al., 2020a)‡	100	MLM	270M (1.4x)	13K (406x)	2400	73.4	87.4	—
mBART (Liu et al., 2020)	25	Eq. 1	680M (3.5x)	4.5K (140x)	2400	—	—	23.5
mBART (ours)	20	Eq. 1	196M (1.0x)	32 (1.0x)	72	68.1	85.4	21.1
PARADISE	20	Eq. 1, 2, 3	196M (1.0x)	32 (1.0x)	81	73.5	89.0	23.1
PARADISE++	20	Eq. 1, 2, 3	196M (1.0x)	32 (1.0x)	95	74.3	89.2	23.8
Table 4: Comparison with prior work. † denotes results taken from Hu et al. (2020). ‡ denotes results taken from Hu et al. (2021). 1 GPU day = 1 day on an NVIDIA V100 GPU.
its better performance in this language. Finally, we observe that all of our different variants perform similarly in English, but incorporating dictionary denoising and using additional parallel data both reduce the cross-lingual transfer gap.
4.3 Comparison with prior work
So as to put our results into perspective, we compare our models with several popular systems from the literature. As shown in Table 4, our proposed approach obtains competitive results despite being trained at a much smaller scale. Just in line with our previous results, this suggests that incorporating parallel data makes pretraining more efficient given that we outperform XLM-R base, mT5, and mBART despite using less data/compute/model size. Interestingly, our method also outperforms XLM-15, MMTE, and mT6 which also use parallel data, as well as AMBER, showing evidence contrary to Hu et al. (2021)'s suggestion that using dictionaries may hurt performance. Detailed per-language results for each task can be found in Appendix F.
Most prior work on multilingual pretraining uses monolingual data only (Pires et al., 2019; Conneau et al., 2020a; Song et al., 2019; Liu et al., 2020; Xue et al., 2021). There have been several proposals to incorporate parallel data into encoder-only
models (Lample and Conneau, 2019; Huang et al., 2019; Hu et al., 2021; Chi et al., 2021b), with some approaches replacing words according to a bilingual dictionary, similar to our dictionary denoising objective (Conneau et al., 2020b; Chaudhary et al., 2020; Dufter and Schütze, 2020). In contrast, we focus on sequence-to-sequence models, which we believe are more flexible and provide a more natural way of integrating parallel data. In that spirit, Siddhant et al. (2019) showed that vanilla machine translation models are already competitive in cross-lingual classification. Closer to our work, Chi et al. (2021a) incorporated parallel corpora into sequence-to-sequence pretraining by feeding concatenated parallel sentences to the encoder and using different masking strategies. In contrast, our approach feeds a noised sentence into the encoder, and tries to recover its translation in the decoder side, obtaining better results with a similar computational budget. Concurrent to our work, Kale et al. (2021) extended T5 to incorporate parallel corpora using a similar approach to our bitext denoising.
In this work, we proposed PARADISE, which introduces two new objectives to integrate parallel data into sequence-to-sequence pretraining. Experimental results on machine translation and cross-lingual classification show that PARADISE provides significant improvements over mBART-style pretraining
on monolingual corpora, obtaining results that are competitive with several popular models at a much smaller scale. Given these findings, we encourage use of parallel data in smaller-scale multilingual pretraining work. In the future, we look to see if our improvements also hold at a larger scale.
We thank Junjie Hu, Jungo Kasai, and Victor Zhong for useful suggestions and comments. MR is grateful to the Masason Foundation for their support
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020. A Call for More Rigor in Unsupervised Cross-lingual Learning. ArXiv.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169-214, Copenhagen, Denmark. Association for Computational Linguistics.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Néveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131-198, Berlin, Germany. Association for Computational Linguistics.
Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009a. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 1-28, Athens, Greece. Association for Computational Linguistics.
Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009b. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 1-28, Athens, Greece. Association for Computational Linguistics.
M. Cettolo, Marcello Federico, L. Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian. 2017. Overview of the iwslt 2017 evaluation campaign.
M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, R. Cattoni, and Marcello Federico. 2015. The iwslt 2015 evaluation campaign.
Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. 2020. Dict-mlm: Improved multilingual pre-training using bilingual dictionaries.
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang Xian-Ling Mao, Heyan Huang, and Furu Wei. 2021a. mt6: Multilingual pretrained text-to-text transformer with translation pairs. arXiv preprint arXiv:2104.08692.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021b. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576-3588, Online. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining Text Encoders as Discriminators Rather Than Generators. arXiv:2003.10555 [cs].
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Philipp Dufter and Hinrich Schütze. 2020. Identifying elements essential for BERT's multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423-4437, Online. Association for Computational Linguistics.
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. CCAligned: A massive collection of cross-lingual web-document pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5960-5969, Online. Association for Computational Linguistics.
Miquel Esplà, Mikel Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019. ParaCrawl: Web-scale parallel corpora for the languages of the EU. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 118-119, Dublin, Ireland. European Association for Machine Translation.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English. arXiv:1902.01382 [cs].
Junjie Hu, Melvin Johnson, Orhan First, Aditya Siddhant, and Graham Neubig. 2021. Explicit Alignment Objectives for Multilingual Bidirectional Encoders. arXiv:2010.07972 [cs].
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. arXiv:2003.11080 [cs].
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2485-2494, Hong Kong, China. Association for Computational Linguistics.
Mihir Kale, Aditya Siddhant, Noah Constant, Melvin Johnson, Rami Al-Rfou, and Linting Xue. 2021. nmt5 - is parallel data still relevant for pre-training