SlowGuess/NAACL_2022_OCR huggingface dataset:
massively multilingual language models? arXiv preprint arXiv:2106.02171.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhat-tacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Guillaume Lample and Alexis Conneau. 2019. Cross-lingual Language Model Pretraining. arXiv:1901.07291 [cs].
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portož, Slovenia. European Language Resources Association (ELRA).
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Pedro Javier Ortiz Suárez, Laurent Romary, and Benoit Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 371-376, Berlin, Germany. Association for Computational Linguistics.
Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Arivazhagan, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. 2019. Evaluating the crosslingual effectiveness of massively multilingual neural machine translation.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked Sequence to Sequence Pre-training for Language Generation. arXiv:1905.02450 [cs].
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv:2010.11934 [cs].
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3530-3534, Portoorz, Slovenia. European Language Resources Association (ELRA).
We list data sources used for pretraining PAR-ADISE++ in Table 5 (monolingual data) and Table 6 (parallel data).
B Pretraining hyperparameters
We use the Adam optimizer  $(\epsilon = 10^{-6}, \beta = (0.9, 0.98))$ , and warm up the learning rate to a peak of  $7 \times 10^{-4}$  after 10K iterations and then proceed to decay the learning rate with the polynomial decay schedule up until 100K iterations. All code and experiments are performed with fairseq (Ott et al., 2019). Following Liu et al. (2020), we add an additional layer-normalization layer on top of both the encoder and decoder to stabilize training with FP16 precision (Micikevicius et al., 2018). All models are trained on 32 V100 16GB GPUs and take 24 hours to finish training.
C Machine translation evaluation
Following Liu et al. (2020), we use detokenized SacreBLEU (Post, 2018) for all languages unless specified otherwise next. For Japanese we use KyTea $^{8}$ , for Nepalese, Sinhala, and Hindi we use Indic-NLP $^{9}$ , for Arabic we use the QCRI Arabic Normalizer $^{10,11}$ , and for Romanian we use Moses tokenization and script normalization following Sennrich et al. (2016); Liu et al. (2020).
D Machine translation finetuning
We finetune our models using the same setup as mBART, warming up the learning rate to  $3 \times 10^{-5}$  over 2500 iterations and then decaying with a polynomial schedule. We use 0.3 dropout and label smoothing  $\epsilon = 0.2$ .
E Comparison of finetuning approaches
Table 7 compares our proposed finetuning approach, which combines the representations from both the encoder and the decoder (see §3), to using either of them alone.12 While prior work either minimally used the decoder if at all (Siddhant et al.,
Language	Data source	Data size (GB)
En	Wiki	14G
De	Wiki	5.9G
Fr	Wiki	4.5G
Es	Wiki	3.7G
Ja	Wiki	3.0G
Ru	Wiki	6.2G
Ar	Wiki	1.7G
Ne	CC100	3.8G
Si	CC100	3.7G
Ro	Wiki+WLM	2.5G
Zh	Wiki+WLM	4.4G
El	Wiki+WLM	2.9G
Eu	Wiki+OSCAR	0.6G
Bg	Wiki+OSCAR	2.5G
Hi	Wiki+OSCAR	2.3G
Sw	Wiki+CC100	1.1G
Th	Wiki+OSCAR	2.4G
Ur	Wiki+OSCAR	1.9G
Vi	Wiki+OSCAR	2.8G
Tr	Wiki+OSCAR	2.4G
Total	—	72G
Table 5: Monolingual Data Statistics. Wiki refers to Wikipedia, and WLM refers to the News Crawl data from CommonCrawl used in WMT.
Language	Data source	Data size (GB)	# Pairs
Ar	UNPC	2.0G	5554595
Bg	ParaCrawl	1.9G	6470710
De	ParaCrawl	2.0G	9685483
El	ParaCrawl	2.0G	6676200
Es	ParaCrawl	2.0G	9138031
Eu	OPUS	0.1G	585210
Fr	ParaCrawl	2.0G	8485669
Hi	IITB	0.4G	1609682
Ja	JParaCrawl	2.0G	6366802
Ne	CCAligned	0.2G	487157
Ro	ParaCrawl	1.3G	6160525
Ru	ParaCrawl	1.6G	5377911
Si	CCAligned	0.2G	619730
Sw	OPUS	0.2G	699719
Th	OpenSubtitles	0.4G	3281533
Tr	OpenSubtitles	2.0G	32077240
Ur	CCAligned	0.3G	1371930
Vi	OpenSubtitles	0.2G	3505276
Zh	UNPC	2.0G	7706183
Total	—	23G	126882448
Table 6: Parallel Data Statistics
Model	avg	Δ
PARADISE++ (encoder-decoder)	74.3	—
decoder-only	73.8	-0.5
encoder-only	72.0	-2.3
Table 7: Ablation of finetuning methods on XNLI.
2019; Xue et al., 2021), or only added a classification head on top of the decoder (Lewis et al., 2020), we find that combining them both works best.
$^{8}$ http://www.phontron.com/kytea/ $^{9}$ https://github.com/anoopkunchukuttan/ indic_nlp_library  $\mathbf{\Pi}_{10}^{\mathrm{th}}$  https://github.com/qntfy/gomosesgo  $\mathbf{\Pi}_{11}^{\mathrm{th}}$  https://alt.qcri.org/tools/arabic-normalizer/ $^{12}$ For decoder-only, we feed the input sequence to both the encoder and the decoder, but add a classification head on top of the decoder only, following Lewis et al. (2020).
We list detailed results by language in this section with results on XNLI in Table 8, PAWS-X in Table 9, and our machine translation ablation (with mBART (Liu et al., 2020) results included) in Table 10. We note that mBART underperforms XLM-R-large on XNLI, however that may be attributed to the fact that XLM-R was trained for much longer rather than the architectural design.
Models	en	zh	es	de	ar	ur	ru	bg	el	fr	hi	sw	th	tr	vi	avg
mBERT	80.8	67.8	73.5	70.0	64.3	57.2	67.8	68.0	65.3	73.4	58.9	49.7	54.1	60.9	69.3	65.4
MMTE	79.6	69.2	71.6	68.2	64.9	60.0	66.2	70.4	67.3	69.5	63.5	61.9	66.2	63.6	69.7	67.5
mT5-small	79.6	65.8	72.7	69.2	65.2	59.9	70.1	71.3	68.6	70.7	62.5	59.7	66.3	64.4	66.3	67.5
AMBER	84.7	71.6	76.9	74.2	70.2	61.0	73.3	74.3	72.5	76.6	66.2	59.9	65.7	73.2	73.4	71.6
XLM-15 (MLM+TLM)	84.1	68.8	77.8	75.7	70.4	62.2	75.0	75.7	73.3	78.0	67.3	67.5	70.5	70.0	73.0	72.6
XLM-100	82.8	70.2	75.5	72.7	66.0	59.8	69.9	71.9	70.4	74.3	62.5	58.1	65.5	66.4	70.7	69.1
XLM-R-base	83.9	73.6	78.3	75.2	71.9	65.4	75.1	76.7	75.4	77.4	69.1	62.2	72.0	70.9	74.0	73.4
mBART	87.7	76.4	81.5	79.8	75.5	—	78.9	—	—	80.6	73.0	—	—	76.1	77.4	—
XLM-R-large	88.7	78.2	83.7	82.5	77.2	71.7	79.1	83.0	80.8	82.2	75.6	71.2	77.4	78.0	79.3	79.2
mBART (ours)	77.5	68.0	70.7	68.8	66.7	62.2	68.6	72.1	69.6	70.1	63.4	62.6	66.6	65.0	69.7	68.1
PARADISE (w/o dict.)	83.3	72.9	77.2	75.7	64.4	66.9	73.4	74.8	75.7	77.7	68.5	67.4	71.0	73.3	75.0	73.1
PARADISE	83.0	74.0	79.0	76.5	68.5	66.8	74.3	76.0	76.4	77.7	70.2	70.5	72.3	74.2	75.4	74.3
Table 8: Accuracy of zero-shot crosslingual classification on the XNLI dataset. Bold numbers highlight the highest scores across languages on the existing models (upper part) and PARADISE variants (bottom part). Results for previous work are sourced from Hu et al. (2020, 2021); Xue et al. (2021).
Model	de	en	es	fr	zh	Avg
mBERT	85.7	94.0	87.4	87.0	77.0	86.2
MMTE	85.1	93.1	87.2	86.9	75.9	85.6
mT5-small	86.2	92.2	86.1	86.6	77.9	85.8
AMBER	89.4	95.6	89.2	90.7	80.9	89.2
XLM-15	88.5	94.7	89.3	89.6	78.1	88.0
XLM-100	85.9	94.0	88.3	87.4	76.5	86.4
XLM-R-base	87.0	94.2	88.6	88.7	78.5	87.4
XLM-R-large	89.7	94.7	90.1	90.4	82.3	89.4
PARADISE++	89.1	94.3	89.6	90.6	82.3	89.2
Table 9: Accuracy of zero-shot cross-lingual classification on PAWS-X. Bold numbers highlight the highest scores across languages on the existing models (upper part) and PARADISE variants (bottom part). We source baseline results from Hu et al. (2020, 2021); Xue et al. (2021).
Lang. Pair	En-Tr	En-Ro	En-Si	En-Hi	En-Es	Tr-En	Ro-En	Si-En	Hi-En
mBART (ours)	15.8	34.9	2.7	19.0	30.4	21.3	34.3	11.0	20.2
PARADISE (w/o dict.)	16.8	36.2	3.2	20.5	32.4	23.2	35.6	13.2	22.3
PARADISE	17.2	36.5	3.7	20.7	32.7	23.5	35.9	14.0	23.6
PARADISE++	19.0	37.3	4.2	20.7	33.0	24.9	36.8	15.1	23.5
mBART	17.8	37.7	3.3	20.8	34.0	22.5	37.8	13.7	23.5
Table 10: Ablation results on machine translation. Note that mBART is trained with 140x more compute and 3.5x more parameters.