PARADISE: Exploiting Parallel Data for
Multilingual Sequence-to-Sequence Pretraining
Machel Reid
The University of Tokyo

Mikel Artetxe
Facebook AI Researchartetxe@fb.com

machelreid@weblab.t.u-tokyo.ac.jp

Abstract
Despite the success of multilingual sequenceto-sequence pretraining, most existing approaches rely on monolingual corpora, anddo not make use of the strong cross-lingualsignal contained in parallel data. In thispaper, we present PARADISE (PARAllel& Denoising Integration in SEquence-tosequence models), which extends the conventional denoising objective used to train thesemodels by (i) replacing words in the noised
sequence according to a multilingual dictionary, and (ii) predicting the reference translation according to a parallel corpus insteadof recovering the original sequence. Our experiments on machine translation and crosslingual natural language inference show anaverage improvement of 2.0 BLEU points
and 6.7 accuracy points from integrating parallel data into pretraining, respectively, obtaining results that are competitive with several popular models at a fraction of their computational cost.1

1

Introduction

Multilingual pretraining (Pires et al., 2019; Mulcaire et al., 2019; Conneau and Lample, 2019)has shown impressive performance in cross-lingualtransfer scenarios. Recently, this paradigm has
been extended for sequence-to-sequence models,achieving strong results both in cross-lingual classification (Xue et al., 2021) and machine translation(Liu et al., 2020).
These models are usually pretrained on combined monolingual corpora in multiple languages
using some form of denoising objective. More concretely, given a sequence x, they proceed to noise
x with a noising function gφ , and maximize theprobability of recovering x given gφ (x):`mono (x) = − log P x|gφ (x)(1)
1

https://github.com/machelreid/paradise

Common noising functions include sentencepermutation and span masking (Lewis et al., 2020;
Liu et al., 2020).
While these methods obtain strong cross-lingual
performance without using any parallel data, they
are usually trained at a scale that is prohibitive formost NLP practitioners. At the same time, it has
been argued that the strict unsupervised scenario
is not realistic, as there is usually some amount ofparallel data available (Artetxe et al., 2020), which
could potentially provide a stronger training signal
and reduce the computational budget required topretrain these models.
Motivated by this, we propose PARADISE, a pretraining method for sequence-to-sequence models
that exploits both word-level and sentence-level
parallel data. The core idea of our approach is toaugment the conventional denoising objective introduced above by (i) replacing words in the noised sequence according to a bilingual dictionary, and (ii)
predicting the reference translation rather than the
input sequence. Despite their simplicity, we findthat both techniques bring substantial gains overconventional pretraining on monolingual data, as
evaluated both in machine translation and zero-shotcross-lingual transfer. Our results are competitive
with several popular models, despite using only afraction of the compute.

2

Proposed method

As illustrated in Figure 1, we propose two methods for introducing parallel information at both
the word-level and the sentence-level: dictionary
denoising and bitext denoising.
Dictionary denoising. Our first method encourages learning similar representations at the wordlevel by introducing anchor words through multilingual dictionaries (Conneau et al., 2020b). Let
Dl (w) denote the translation of word w into language l ∈ L according to the dictionary D. Given

Their work is absolutely amazing 


Encoder
Their 仕事

est <mask> ‫ﺣﯾرت اﻧﮕﯾز‬Their work is absolutely amazing 


Decoder
 Their work

is absolutely amazing

Decoder

Encoder<mask> 仕事 <mask> 素晴らしい(a) Dictionary Denoising
 Their work

is absolutely amazing(b) Bitext Denoising

Figure 1: Our proposed techniques for integrating parallel data into sequence-to-sequence pretraining.
the source sentence x = (x1 , x2 , . . . , xn ), we define its noised version gψ (x) = (x̃1 , x̃2 , . . . , x̃n ),prwhere x̃i = Dl (xi ) with probability |L|and x̃i =xi otherwise (i.e. we replace each word with its
translation into a random language with probabilitypr ). We set pr = 0.4. Given the dictionary-noised
sentence, we train our model using the denoisingauto-encoding objective in Eq. 1:
`dict (x) = − log P x|gφ (gψ (x))(2)
Bitext denoising. Our second approach encourages learning from both monolingual and parallel
data sources, by including translation data in the
pretraining process. Given a source-target bitext
pair (x, y) in the parallel corpus, assumed to besemantically equivalent, we model the following:`bitext (x, y) = − log P y|gφ (x)(3)
in which we optimize the likelihood of generating
the target sentence y conditioned on the noised
version of the source sentence, gφ (x).2
Combined objective. Our final objective combines `mono , `dict and `bitext .3 Given that our corpus contains languages with varying data sizes, we
sample sentences using the exponential samplingtechnique from Conneau and Lample (2019). We
use αmono = 0.5 to sample from the monolingual
corpus, and αbitext = 0.3 to sample from the parallel corpus. To prevent over-exposure to English
on the decoder side when sampling from the parallel corpus, we halve the probability of to-English
directions and renormalize the probabilities. In addition, given that we have fewer amounts of parallel
data (used for `bitext ) than monolingual data (usedfor `mono and `dict ), we sample between each taskusing αtask = 0.3.
2

To make our pretraining sequence length consistent with`mono and `dict , we concatentate randomly sampled sentencepairs from the same language pair to fit the maximum length.
3
We use the same noising function gφ used by Lewis et al.
(2020) and Liu et al. (2020).

3

Experimental Settings

We pretrain our models on 20 languages (English,
French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese,
Hindi, Swahili, Urdu, Japanese, Basque, Romanian,Sinhala and Nepalese), and evaluate them on machine translation and cross-lingual classification.
3.1

Pretraining

Data. We use Wikipedia as our monolingual corpus, and complement it with OSCAR (Ortiz Suárez
et al., 2020), and CC100 (Conneau et al., 2020a)
for low-resource languages. For a fair comparison
with monolingually pretrained baselines, we use
the same parallel data as in our downstream machine translation experiments (detailed in §3.2). In
addition, we train a separate variant (detailed below) using additional parallel data from ParaCrawl(Esplà et al., 2019), UNPC (Ziemski et al., 2016),
CCAligned (El-Kishky et al., 2020), and OpenSubtitles (Lison and Tiedemann, 2016).4 We tokenize
all data using SentencePiece (Kudo and Richardson, 2018) with a joint vocabulary of 125K subwords. We use bilingual dictionaries from FLoRes5(Guzmán et al., 2019) for Nepalese and Sinhala,
and MUSE6 (Lample et al., 2018) for the rest oflanguages. Refer to Appendix A for more details.
Models. We use the same architecture as BARTbase (Lewis et al., 2020), totaling ∼196M parameters, and train for 100k steps with a batch sizeof ∼520K tokens. This takes around a day on
32 NVIDIA V100 16GB GPUs. As discussed before, we train two variants of our full model: PAR ADISE , which uses the same parallel data as the machine translation experiments, and PARADISE++,
which uses additional parallel data. To better understand the contribution of each objective, we train
two additional models without dictionary denois4

We cap the size of each language pair to 2GB.
https://github.com/facebookresearch/flores
6
https://github.com/facebookresearch/MUSE
5

Languages
Data Source
Size
Direction

En-Vi
En-Tr
En-Ja
En-Ar
En-Ne
IWSLT15 WMT17 IWSLT17 IWSLT17 FLoRes
133K
207K
223K
250K
564K
← → ← → ← → ← → ← →

En-Ro
WMT16
608K
← →

En-Si
FLoRes
647K
← →

En-Hi
IITB
1.56M
← →

En-Es
WMT13
15M
← →

En-Fr
WMT14
41M
← →

Random init. 23.6 24.8 12.2 9.5 10.4 12.3 27.5 16.9 7.6 4.3 34.0 34.3 7.2 1.2 10.9 14.2 32.1 31.4 37.0 38.9mBART (ours) 29.1 31.5 21.3 15.8 15.7 17.3 32.1 19.2 10.3 6.1 34.3 34.9 11.0 2.7 20.2 19.0 29.8 30.4 36.0 38.2
PARADISE

30.0 32.6 23.5 17.2 17.2 19.2 35.3 21.1 13.7 7.9 35.9 36.5 14.0 3.7 23.6 20.7 32.6 32.7 37.8 39.8

Table 1: Machine translation results. Random initialization numbers taken from Liu et al. (2020).
ing, which we name PARADISE (w/o dict.) and
PARADISE ++ (w/o dict.). Finally, we train a baseline system using the monolingual objective alone,which we refer to as mBART (ours). This follows
the original mBART work (Liu et al., 2020), but
is directly comparable to the rest of our models interms of data and hyperparameters.
3.2

Downstream Settings

Machine translation. Following Liu et al.
(2020), we evaluate our models on sentence-levelmachine translation from and to English using thefollowing datasets: IWSLT (Cettolo et al., 2015,2017) for Vietnamese, Japanese and Arabic, WMT(Callison-Burch et al., 2009a,b; Bojar et al., 2016,2017) for Spanish, French, Romanian and Turkish, FLoRes (Guzmán et al., 2019) for Sinhala and
Nepalese, and IITB (Kunchukuttan et al., 2018) for
Hindi. We report performance in BLEU (Papineni
et al., 2002) as detailed in Appendix C. We finetune
our models using the same setup as mBART, warming up the learning rate to 3 × 10−5 over 2500
iterations and then decaying with a polynomial
schedule. We use 0.3 dropout and label smoothing = 0.2.
Cross-lingual classification. We evaluate ourmodels on zero-shot cross-lingual transfer, where
we finetune on English data and test performanceon other languages. To that end, we use the XNLI
natural language inference dataset (Conneau et al.,2018) and the PAWS-X adversarial paraphrase identification dataset (Yang et al., 2019). Following Hu
et al. (2021), we use all the 15 languages in XNLI,
and English, German, Spanish, French and Chinese for PAWS-X. We develop a new approach for
applying sequence-to-sequence models for classification: feeding the sequence into both the encoderand decoder, and taking the concatenation of theencoder’s 
 representation and the decoder’s
 representation as the input of the classification head. We provide an empirical rationale forLang. pair (En-XX)

Ro

Si

Hi

Es

Avg∆

mBART (ours)
15.8 34.9
PARADISE (w/o dict.) 16.8 36.2
PARADISE
17.2 36.5
PARADISE ++19.0 37.3

2.7
3.2
3.7
4.2

19.0
20.5
20.7
20.7

30.4
32.4
32.7
33.0

20.6±0.0
21.8+1.2
22.2+1.6
22.8+2.2

Lang. pair (XX-En)

Tr

Ro

Si

Hi

Es

Avg∆

mBART (ours)
21.3 34.3
PARADISE (w/o dict.) 23.2 35.6
PARADISE
23.5 35.9
PARADISE ++24.9 36.8

Tr

11.0
13.2
14.0
15.1

20.2
22.3
23.6
23.5

29.8
31.6
32.6
32.9

23.3±0.0
25.2+1.9
25.9+2.6
26.6+3.3

Table 2: Ablation results on machine translation.
this in Table 4. We finetune all models with a batch
size of 64 and a learning rate of 2 × 10−5 for a maximum of 100k iterations, performing early stoppingon the validation set.

4

Results

We next report our results on machine translation(§4.1) and cross-lingual classification (§4.2), andcompare them to prior work (§4.3).
4.1

Machine Translation

We report our main results in Table 1. We observe that PARADISE consistently outperforms ourmBART baseline across all language pairs. Notethat these two models have seen the exact same
corpora, but mBART uses the parallel data for finetuning only, whereas PARADISE also uses it at the
pretraining stage. This suggests that incorporating
parallel data into pretraining helps learn better representations, which results in better downstream
performance.
Table 2 reports additional ablation results on asubset of languages. As can be seen, removing
dictionary denoising hurts, but is still better thanour mBART baseline. This shows that both of our
proposed approaches—dictionary denoising andbitext denoising—are helpful and complementary.
Finally, PARADISE++ improves over PARADISE,
indicating that a more balanced corpus with moreparallel data is helpful.

Models

en

zh

es

de

ar

ur

ru

bg

el

fr

hi

sw

th

tr

vi

avg

69.6
73.2
75.7
76.4

70.1
77.7
77.7
77.7

63.4
70.6
68.5
70.2

62.6
66.2
67.4
70.5

66.6
70.4
71.0
72.3

65.0
72.1
73.3
74.2

69.7
75.3
75.0
75.4

68.1
73.5
73.1
74.3

70.5
75.7
75.3
77.3

70.5
76.2
77.5
78.4

73.9
78.6
77.3
79.2

71.7
77.2
76.6
78.5

Finetune a multilingual model on the English training set (ZERO-SHOT)mBART (ours)

77.5
83.4
PARADISE++ (w/o dict.) 83.3
PARADISE++
83.0
PARADISE

68.0
73.8
72.9
74.0

70.7
77.6
77.2
79.0

68.8
76.0
75.7
76.5

66.7
72.4
64.4
68.5

62.2
65.1
66.9
66.8

68.6
74.0
73.4
74.3

72.1
74.4
74.8
76.0

Finetune a multilingual model on all machine translated training sets (TRANSLATE-TRAIN-ALL)
mBART (ours)

77.8
84.0
PARADISE++ (w/o dict.) 83.2
PARADISE++
84.8
PARADISE

72.0
77.6
77.2
78.3

74.0
81.2
79.7
81.7

72.6
79.4
78.5
80.5

69.5
75.9
72.0
76.0

66.5
68.0
68.3
70.6

70.9
76.8
76.5
78.8

74.3
79.1
78.2
80.4

72.7
79.0
79.2
81.3

73.8
79.9
79.3
80.6

68.9
73.4
73.3
74.9

68.2
72.6
73.3
74.2

Table 3: Accuracy of zero-shot crosslingual classification on the XNLI dataset.
Model

avg

∆

PARADISE++ (encoder-decoder)

74.3
73.8
72.0

—
-0.5
-2.3

decoder-only
encoder-only

Table 4: Ablation of finetuning methods on XNLI.

imally used the decoder if at all (Siddhant et al.,2019; Xue et al., 2021), or only added a classification head on top of the decoder (Lewis et al., 2020),
we find that taking the best of both worlds performs
best.
4.3

4.2

Cross-lingual Classification

We report XNLI results in Table 3 and PAWS-X
results in Appendix D. Our proposed approach outperforms mBART in all languages by a large margin. To our surprise, we also observe big gains inEnglish. We conjecture that this could be explainedby bitext denoising providing a stronger training
signal from all tokens akin to ELECTRA (Clark
et al., 2020), whereas monolingual denoising only
gets effective signal from predicting the masked
portion. In addition, given that we are using parallel data between English and other languages,
PARADISE ends up seeing much more English text
compared to mBART—yet a similar amount in
the rest of languages—which could also contributeto its better performance in this language. Interestingly, our improvements also hold when using
the TRANSLATE - TRAIN - ALL approach, which indirectly uses parallel data to train the underlying
machine translation system. Finally, we observethat all of our different variants perform similarly
in English, but incorporating dictionary denoisingand using additional parallel data both reduce thecross-lingual transfer gap.
Table 4 compares our proposed finetuning approach, which combines the representations fromboth the encoder and the decoder (see §3), to using
either of them alone.7 While prior work either min7
For decoder-only, we feed the input sequence to both theencoder and the decoder, but add a classification head on top

Comparison with prior work

So as to put our results into perspective, we compare our models with several popular systems fromthe literature. As shown in Table 5, our proposed
approach obtains competitive results despite being
trained at a much smaller scale.8 Just in line with
our previous results, this suggests that incorporating parallel data makes pretraining more efficient.
Interestingly, our method also outperforms XLM,MMTE and mT6, which also use parallel data, as
well as AMBER, showing evidence contrary to Hu
et al. (2021)’s suggestion that using dictionaries
may hurt performance. Detailed per-language results for each task can be found in Appendix D.

5Related Work

Most prior work on large-scale multilingual pretraining uses monolingual data only (Pires et al.,2019; Conneau et al., 2020a; Song et al., 2019;
Liu et al., 2020; Xue et al., 2021). XLM (Lample
and Conneau, 2019) was first to incorporate parallel data through its translation language modeling(TLM) objective, which applies masked language
modeling to concatenated parallel sentences. Unicoder (Huang et al., 2019), AMBER (Hu et al.,2021) and InfoXLM (Chi et al., 2021b) introduced
additional objectives over parallel corpora. Similar to our dictionary denoising objective, someprevious work has also explored replacing wordsof the decoder only, following Lewis et al. (2020).
8
1 GPU day = 1 day on an NVIDIA V100 GPU

model#LangsTask

Params.

mBERT (Devlin et al., 2019)†
MMTE (Siddhant et al., 2019)†
mT5-small (Xue et al., 2021)
mT6 (Chi et al., 2021a)
AMBER (Hu et al., 2021)
XLM-15 (Conneau and Lample, 2019)‡
XLM-100 (Conneau and Lample, 2019)†
XLM-R-base (Conneau et al., 2020a)‡
XLM-R-large (Conneau et al., 2020a)†mBART (Liu et al., 2020)

104
102
101
94
104
15
100
100
100
25

MLM
Translation
Eq. 1
SC+PNAT+TSC
MLM+TLM
MLM+TLM
MLM
MLM
MLM
Eq. 1

179M (0.9x)
375M (1.9x)
300M (1.5x)
300M (1.5x)
179M (0.9x)
250M (1.3x)
570M (2.9x)
270M (1.4x)
550M (2.8x)
680M (3.5x)———
40 (1.3x)1000 (31x)450 (14x)640 (20x)
13K (406x)
27K (844x)
4.5K (140x)

60
5000
27000
2120
100
100
60
2400
2400
2400

65.4
67.4
67.5
64.7
71.6
72.6
69.1
73.4
79.2
—

86.2
85.6
85.8
86.6
89.2
88.0
86.4
87.4
89.4
—

—
—
—
—
—
—
—
—
—
23.5

mBART (ours)

20
20
20

Eq. 1
Eq. 1, 2, 3
Eq. 1, 2, 3

196M (1.0x)
196M (1.0x)
196M (1.0x)

32 (1.0x)
32 (1.0x)
32 (1.0x)

72
81
95

68.1
73.5
74.3

85.4
89.0
89.2

21.1
23.1
23.8

PARADISE
PARADISE++

Est. GPU Days Data (GB) XNLI PAWS-X MT

Table 5: Comparison with prior work. † denotes results taken from Hu et al. (2020), and ‡ denotes results
taken from Hu et al. (2021). The rest of the numbers are taken from the original papers.
according to a bilingual dictionary (Conneau et al.,2020b; Chaudhary et al., 2020; Dufter and Schütze,2020). However, all these approaches operate withencoder-only models, while we believe sequenceto-sequence models are more flexible and provide
a more natural way of integrating parallel data.
In that spirit, Siddhant et al. (2019) showed thatvanilla machine translation models are already competitive in cross-lingual classification. Our approach combines translation with denoising and further incorporates bilingual dictionaries and monolingual corpora, obtaining substantially better results. Closer to our work, Chi et al. (2021a) incorporated parallel corpora into sequence-to-sequence
pretraining by feeding concatenated parallel sentences to the encoder and using different masking strategies, similar to TLM. In contrast, our approach feeds a noised sentence into the encoder,and tries to recover its translation in the decoderside, obtaining substantially better results with asimilar computational budget. Concurrent to ourwork, Kale et al. (2021) extended T5 to incorporate
parallel corpora using a similar approach to our
bitext denoising.

6

Conclusions

In this work, we proposed PARADISE, which introduces two new denoising objectives to integrate bilingual dictionaries and parallel corpora
into sequence-to-sequence pretraining. Experimental results on machine translation and cross-lingual
sentence classification show that PARADISE provides significant improvements over mBART-style
pretraining on monolingual corpora only, obtaining
results that are competitive with several popularmodels at a much smaller scale. In future work, welook to see whether these techniques and findings
scale with model size.

Acknowledgements
We thank Junjie Hu, Jungo Kasai, and Victor Zhong
for useful suggestions and comments. MR is grateful to the Masason Foundation for their support.

References
Mikel Artetxe, Sebastian Ruder, Dani Yogatama,
Gorka Labaka, and Eneko Agirre. 2020. A Callfor More Rigor in Unsupervised Cross-lingual
Learning. ArXiv.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian
Huang, Matthias Huck, Philipp Koehn, Qun Liu,
Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia,
and Marco Turchi. 2017. Findings of the 2017conference on machine translation (WMT17). In
Proceedings of the Second Conference on Machine Translation, pages 169–214, Copenhagen,Denmark. Association for Computational Linguistics.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias
Huck, Antonio Jimeno Yepes, Philipp Koehn,
Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin
Popel, Matt Post, Raphael Rubino, Carolina
Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findingsof the 2016 conference on machine translation.

In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics.

Kevin Clark, Minh-Thang Luong, Quoc V. Le, andChristopher D. Manning. 2020. ELECTRA: Pretraining Text Encoders as Discriminators Rather
Than Generators. arXiv:2003.10555 [cs].

Chris Callison-Burch, Philipp Koehn, Christof
Monz, and Josh Schroeder. 2009a. Findings ofthe 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop
on Statistical Machine Translation, pages 1–28,
Athens, Greece. Association for Computational
Linguistics.

Alexis Conneau, Kartikay Khandelwal, Naman
Goyal, Vishrav Chaudhary, Guillaume Wenzek,
Francisco Guzmán, Edouard Grave, Myle Ott,
Luke Zettlemoyer, and Veselin Stoyanov. 2020a.
Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual
Meeting of the Association for Computational
Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Chris Callison-Burch, Philipp Koehn, Christof
Monz, and Josh Schroeder. 2009b. Findings ofthe 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop
on Statistical Machine Translation, pages 1–28,
Athens, Greece. Association for Computational
Linguistics.
M. Cettolo, Marcello Federico, L. Bentivogli,
Niehues Jan, Stüker Sebastian, Sudoh Katsuitho,
Yoshino Koichiro, and Federmann Christian.
2017. Overview of the iwslt 2017 evaluation
campaign.
M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli,
R. Cattoni, and Marcello Federico. 2015. The
iwslt 2015 evaluation campaign.
Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. 2020. Dict-mlm: Improved multilingual pre-training using bilingual
dictionaries.
Zewen Chi, Li Dong, Shuming Ma, Shaohan
Huang Xian-Ling Mao, Heyan Huang, and Furu
Wei. 2021a. mt6: Multilingual pretrained textto-text transformer with translation pairs. arXiv
preprint arXiv:2104.08692.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, XianLing Mao, Heyan Huang, and Ming Zhou.
2021b. InfoXLM: An information-theoretic
framework for cross-lingual language model pretraining. In Proceedings of the 2021 Conferenceof the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576–3588, Online.
Association for Computational Linguistics.

Alexis Conneau and Guillaume Lample. 2019.
Cross-lingual language model pretraining. In
Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019,
December 8-14, 2019, Vancouver, BC, Canada,
pages 7057–7067.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger
Schwenk, and Veselin Stoyanov. 2018. XNLI:
Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conferenceon Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium.
Association for Computational Linguistics.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging
cross-lingual structure in pretrained language
models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association
for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-trainingof deep bidirectional transformers for language
understanding. In Proceedings of the 2019 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, Volume 1 (Longand Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational
Linguistics.
Philipp Dufter and Hinrich Schütze. 2020. Identifying elements essential for BERT’s multilinguality. In Proceedings of the 2020 Conference

on Empirical Methods in Natural Language Processing (EMNLP), pages 4423–4437, Online. Association for Computational Linguistics.
Ahmed El-Kishky, Vishrav Chaudhary, Francisco
Guzmán, and Philipp Koehn. 2020. CCAligned:
A massive collection of cross-lingual webdocument pairs. In Proceedings of the 2020
Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 5960–
5969, Online. Association for Computational
Linguistics.
Miquel Esplà, Mikel Forcada, Gema RamírezSánchez, and Hieu Hoang. 2019. ParaCrawl:
Web-scale parallel corpora for the languages of
the EU. In Proceedings of Machine Translation
Summit XVII Volume 2: Translator, Project andUser Tracks, pages 118–119, Dublin, Ireland.
European Association for Machine Translation.
Francisco Guzmán, Peng-Jen Chen, Myle Ott,
Juan Pino, Guillaume Lample, Philipp Koehn,
Vishrav Chaudhary, and Marc’Aurelio Ranzato.
2019. The FLoRes Evaluation Datasets for LowResource Machine Translation: Nepali-English
and Sinhala-English. arXiv:1902.01382 [cs].
Junjie Hu, Melvin Johnson, Orhan Firat, Aditya
Siddhant, and Graham Neubig. 2021. Explicit
Alignment Objectives for Multilingual Bidirectional Encoders. arXiv:2010.07972 [cs].
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Crosslingual Generalization. arXiv:2003.11080 [cs].
Haoyang Huang, Yaobo Liang, Nan Duan, Ming
Gong, Linjun Shou, Daxin Jiang, and Ming
Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingualtasks. In Proceedings of the 2019 Conferenceon Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 2485–2494, Hong Kong, China.
Association for Computational Linguistics.
Mihir Kale, Aditya Siddhant, Noah Constant,
Melvin Johnson, Rami Al-Rfou, and Linting
Xue. 2021. nmt5 – is parallel data still relevant
for pre-training massively multilingual language
models? arXiv preprint arXiv:2106.02171.

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent
subword tokenizer and detokenizer for neuraltext processing. In Proceedings of the 2018
Conference on Empirical Methods in Natural
Language Processing: System Demonstrations,pages 66–71, Brussels, Belgium. Association for
Computational Linguistics.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay
English-Hindi parallel corpus. In Proceedingsof the Eleventh International Conference on Language Resources and Evaluation (LREC 2018),
Miyazaki, Japan. European Language Resources
Association (ELRA).
Guillaume Lample and Alexis Conneau. 2019.
Cross-lingual Language Model Pretraining.
arXiv:1901.07291 [cs].
Guillaume Lample, Alexis Conneau, Marc’Aurelio
Ranzato, Ludovic Denoyer, and Hervé Jégou.
2018. Word translation without parallel data. In
6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada,
April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan
Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer.
2020. BART: Denoising sequence-to-sequence
pre-training for natural language generation,translation, and comprehension. In Proceedingsof the 58th Annual Meeting of the Association for
Computational Linguistics, pages 7871–7880,
Online. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora
from movie and TV subtitles. In Proceedingsof the Tenth International Conference on Language Resources and Evaluation (LREC’16),
pages 923–929, Portorož, Slovenia. European
Language Resources Association (ELRA).
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li,
Sergey Edunov, Marjan Ghazvininejad, Mike
Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machinetranslation. Transactions of the Association for
Computational Linguistics, 8:726–742.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii
Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018.
Mixed precision training. In 6th International
Conference on Learning Representations, ICLR
2018, Vancouver, BC, Canada, April 30 - May 3,
2018, Conference Track Proceedings. OpenReview.net.
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith.
2019. Polyglot contextual representations improve crosslingual transfer. In Proceedingsof the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
3912–3918, Minneapolis, Minnesota. Association for Computational Linguistics.
Pedro Javier Ortiz Suárez, Laurent Romary, andBenoît Sagot. 2020. A monolingual approachto contextualized word embeddings for midresource languages. In Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online.
Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela
Fan, Sam Gross, Nathan Ng, David Grangier,and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North
American Chapter of the Association for Computational Linguistics (Demonstrations), pages
48–53, Minneapolis, Minnesota. Association for
Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, andWei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages
311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette.
2019. How multilingual is multilingual BERT?
In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics,
pages 4996–5001, Florence, Italy. Association
for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting
BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra
Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings ofthe First Conference on Machine Translation:
Volume 2, Shared Task Papers, pages 371–376,
Berlin, Germany. Association for Computational
Linguistics.
Aditya Siddhant, Melvin Johnson, Henry Tsai,
Naveen Arivazhagan, Jason Riesa, Ankur Bapna,
Orhan Firat, and Karthik Raman. 2019. Evaluating the cross-lingual effectiveness of massively
multilingual neural machine translation.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and
Tie-Yan Liu. 2019. MASS: Masked Sequence toSequence Pre-training for Language Generation.
arXiv:1905.02450 [cs].
Linting Xue, Noah Constant, Adam Roberts, Mihir
Kale, Rami Al-Rfou, Aditya Siddhant, Aditya
Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv:2010.11934 [cs].
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason
Baldridge. 2019. PAWS-X: A cross-lingual
adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conferenceon Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 3687–3692, Hong Kong, China.
Association for Computational Linguistics.
Michał Ziemski, Marcin Junczys-Dowmunt, andBruno Pouliquen. 2016. The United Nations
parallel corpus v1.0. In Proceedings of theTenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–
3534, Portorož, Slovenia. European Language
Resources Association (ELRA).

A

Data

We list data sources used for pretraining PAR ADISE ++ in Table 9 (monolingual data) and Table 10 (parallel data).

B

Pretraining hyperparameters

We use the Adam optimizer ( = 10−6 , β =(0.9, 0.98)), and warm up the learning rate to a peakof 7×10−4 after 10K iterations and then proceed todecay the learning rate with the polynomial decay
schedule up until 100K iterations. All code and experiments are performed with fairseq (Ott et al.,2019). Following Liu et al. (2020), we add an additional layer-normalization layer on top of both theencoder and decoder to stabilize training with FP16
precision (Micikevicius et al., 2018). All models
are trained on 32 V100 16GB GPUs and takes 24hours to finish training.

C

Machine translation evaluation

Following Liu et al. (2020), we use detokenized
SacreBLEU (Post, 2018) for all languages unlessspecified otherwise next. For Japanese we use
KyTea9 , for Nepalese, Sinhala, and Hindi we use
Indic-NLP10 , for Arabic we use the QCRI Arabic
Normalizer11,12 , and for Romanian we use Moses
tokenization and script normalization following
Sennrich et al. (2016); Liu et al. (2020).

D

Additional results

We list detailed results by language in this section with results on XNLI in Table 8, PAWS-X
in Table 6, and our machine translation ablation(with mBART (Liu et al., 2020) results included)in Table 7. We note that on XNLI that mBART
underperforms XLM-R-large, however that may be
attributed to the fact that XLM-R was trained formuch longer rather than the architectural design.

9

http://www.phontron.com/kytea/
https://github.com/anoopkunchukuttan/
indic_nlp_library11
https://github.com/qntfy/gomosesgo
12
https://alt.qcri.org/tools/
arabic-normalizer/
10

Model

de

en

es

fr

zh

Avg

mBERT
MMTE
mT5-small
AMBER
XLM-15
XLM-100
XLM-R-base
XLM-R-large

85.7
85.1
86.2
89.4
88.5
85.9
87.0
89.7

94.0
93.1
92.2
95.6
94.7
94.0
94.2
94.7

87.4
87.2
86.1
89.2
89.3
88.3
88.6
90.1

87.0
86.9
86.6
90.7
89.6
87.4
88.7
90.4

77.0
75.9
77.9
80.9
78.1
76.5
78.5
82.3

86.2
85.6
85.8
89.2
88.0
86.4
87.4
89.4

PARADISE++

89.1

94.3

89.6

90.6

82.3

89.2

Table 6: Accuracy of zero-shot cross-lingual classification on PAWS-X. Bold numbers highlight thehighest scores across languages on the existingmodels (upper part) and PARADISE variants (bottom part). We source baseline results from Hu et al.
(2020, 2021); Xue et al. (2021).

Lang. Pair

En-Tr

En-Ro

En-Si

En-Hi

En-Es

Tr-En

Ro-En

Si-En

Hi-En

mBART (ours)
PARADISE (w/o dict.)
PARADISE ++

15.8
16.8
17.2
19.0

34.9
36.2
36.5
37.3

2.7
3.2
3.7
4.2

19.0
20.5
20.7
20.7

30.4
32.4
32.7
33.0

21.3
23.2
23.5
24.9

34.3
35.6
35.9
36.8

11.0
13.2
14.0
15.1

20.2
22.3
23.6
23.5

mBART

17.8

37.7

3.3

20.8

34.0

22.5

37.8

13.7

23.5

PARADISE

Table 7: Ablation results on machine translation. Note that mBART is trained with 140x more computeand 3.5x more parameters.
Models

en

zh

es

de

ar

ur

ru

bg

el

fr

hi

sw

th

tr

vi

avg

mBERT
MMTE
mT5-small
AMBER
XLM-15 (MLM+TLM)
XLM-100
XLM-R-base
mBART
XLM-R-large

80.8
79.6
79.6
84.7
84.1
82.8
83.9
87.7
88.7

67.8
69.2
65.8
71.6
68.8
70.2
73.6
76.4
78.2

73.5
71.6
72.7
76.9
77.8
75.5
78.3
81.5
83.7

70.0
68.2
69.2
74.2
75.7
72.7
75.2
79.8
82.5

64.3
64.9
65.2
70.2
70.4
66.0
71.9
75.5
77.2

57.2
60.0
59.9
61.0
62.2
59.8
65.4
—
71.7

67.8
66.2
70.1
73.3
75.0
69.9
75.1
78.9
79.1

68.0
70.4
71.3
74.3
75.7
71.9
76.7
—
83.0

65.3
67.3
68.6
72.5
73.3
70.4
75.4
—
80.8

73.4
69.5
70.7
76.6
78.0
74.3
77.4
80.6
82.2

58.9
63.5
62.5
66.2
67.3
62.5
69.1
73.0
75.6

49.7
61.9
59.7
59.9
67.5
58.1
62.2
—
71.2

54.1
66.2
66.3
65.7
70.5
65.5
72.0
—
77.4

60.9
63.6
64.4
73.2
70.0
66.4
70.9
76.1
78.0

69.3
69.7
66.3
73.4
73.0
70.7
74.0
77.4
79.3

65.4
67.5
67.5
71.6
72.6
69.1
73.4
—
79.2

mBART (ours)
PARADISE (w/o dict.)

77.5
83.3
83.0

68.0
72.9
74.0

70.7
77.2
79.0

68.8
75.7
76.5

66.7
64.4
68.5

62.2
66.9
66.8

68.6
73.4
74.3

72.1
74.8
76.0

69.6
75.7
76.4

70.1
77.7
77.7

63.4
68.5
70.2

62.6
67.4
70.5

66.6
71.0
72.3

65.0
73.3
74.2

69.7
75.0
75.4

68.1
73.1
74.3

PARADISE

Table 8: Accuracy of zero-shot crosslingual classification on the XNLI dataset. Bold numbers highlightthe highest scores across languages on the existing models (upper part) and PARADISE variants (bottompart). Results for previous work are sourced from Hu et al. (2020, 2021); Xue et al. (2021).

Language

Data source

Data size (GB)

En
De
Fr
Es
Ja
Ru
Ar
Ne
Si
Ro
Zh
El
Eu
Bg
Hi
Sw
Th
Ur
Vi
Tr
Total

Wiki
Wiki
Wiki
Wiki
Wiki
Wiki
Wiki
CC100
CC100
Wiki+WLM
Wiki+WLM
Wiki+WLM
Wiki+OSCAR
Wiki+OSCAR
Wiki+OSCAR
Wiki+CC100
Wiki+OSCAR
Wiki+OSCAR
Wiki+OSCAR
Wiki+OSCAR
—

14G
5.9G
4.5G
3.7G
3.0G
6.2G
1.7G
3.8G
3.7G
2.5G
4.4G
2.9G
0.6G
2.5G
2.3G
1.1G
2.4G
1.9G
2.8G
2.4G
72G

Table 9: Monolingual Data Statistics. Wiki refers
to Wikipedia, and WLM refers to the News Crawl
data from CommonCrawl used in WMT.

Language

Data source

Data size (GB)# Pairs

Ar
Bg
De
El
Es
Eu
Fr
Hi
Ja
Ne
Ro
Ru
Si
Sw
Th
Tr
Ur
Vi
Zh
Total

UNPC
ParaCrawl
ParaCrawl
ParaCrawl
ParaCrawl
OPUS
ParaCrawl
IITB
JParaCrawl
CCAligned
ParaCrawl
ParaCrawl
CCAligned
OPUS
OpenSubtitles
OpenSubtitles
CCAligned
OpenSubtitles
UNPC
—

2.0G
1.9G
2.0G
2.0G
2.0G
0.1G
2.0G
0.4G
2.0G
0.2G
1.3G
1.6G
0.2G
0.2G
0.4G
2.0G
0.3G
0.2G
2.0G
23G

5554595
647071096854836676200
91380315852108485669
1609682
6366802487157616052553779116197306997193281533
32077240
1371930
3505276
7706183126882448

Table 10: Parallel Data Statistics