Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual
Pretraining on Cross-lingual Transferability
Yoshinari Fujinuma,*1 Jordan Boyd-Graber,2 Katharina Kann3AWS AI Labs
University of Maryland
University of Coloardo Boulder
fujinumay@gmail.com jbg@umiacs.umd.edu katharina.kann@colorado.edu∗Work done while at University of Colorado Boulder
12

At a Glance

3

Results: Diverse Languages• How does the number of pretraining languages influencezero-shot performance on unseen target languages?
• Does the answer to that question change with modeladaptation?
• Do the findings for our first question change if the languages
used for pretraining are all related?

• Average cross-lingual zero-shot accuracy increases up to some point
POS Tagging Accuracy on Unseen Languages

Pretrained multilingual models enable zero-shot learning even for
unseen languages, and that performance can be further improved
via adaptation prior to finetuning.
However, it is unclear how the number of pretraining languages
influences a model’s zero-shot learning for languages unseen during
pretraining. To fill this gap, we ask the following research questions:

Experimental Setup

0.3• Using XLM-R base vocabulary are not truely unseen for the targetlanguages

0.2

Experiment Setup with Model Adaptation on Truely Unseen Languages• Use the following pretrained models in addition to the pretrained
language models up to 10 languages

0.1• XLM-17 (17 languages, pretrained on full Wikipedia) [5]• XLM-100 (100 languages, pretrained on full Wikipedia) [5]• XLM-R base (100 languages, pretrained on Common Crawl) [6]0enDiv-2 Div-3 Div-4 Div-5 Div-6 Div-7 Div-8 Div-9 Div-10(+ru) (+zh) (+ar) (+hi) (+es) (+el) (+fi) (+id) (+tr)

Div-5Div-6Div-8Div-10

XLM-17

XLM-100

XLM-R

1POS Accuracy

0.75

0.2

0.50

0.25

0.1

• Pretraining Corpus: CoNLL 2017 Wikipedia dump [1] downsampledto ≈ 200MB

0Maltese

Wolof

YorubaErzya

Nothern Sami

Target Language

0en

Div-2 Div-3 Div-4 Div-5 Div-6 Div-7 Div-8 Div-9 Div-10(+ru) (+zh) (+ar) (+hi) (+es) (+el) (+fi) (+id) (+tr)

Before Adaptation

Pretraining Languages• Transformer with same hyperparameters and vocabulary as XLM-R
base

Div-5

Div-6

Div-8

Div-10

XLM-17

XLM-100

XLM-R1• Choice of pretraining languages

Results: Model Adaptation• Diverse set of languages (Div-X)• Related set of languages (Rel-X)• Trend 1: More languages are better (French and Farsi)• Task Dataset: XTREME [2]• Trend 2: More languages does not necessarily improve (Vietnameseand Tamil)bg• Continued pretraining with Masked Language Modeling [3]0.5Maltese

Wolof

YorubaErzya

Nothern Sami

Target Language

0.0vi

IE: Romance

After Adaptation

Austro−Asiatic

1.0Conclusion

0.50.0tafa1.0✓ Match the script between pretraining and target languages if notadapting multilingual models

0.5✓ The more languages the better if adapting multilingual models

Dravidian

IE: Iranian+tr+id+fi+el+es+hi+zh+ru

en+tr+id+fi+el+es+hi+ar+zh

0.0en

en, deen, de, sv
en, de, sv, nl
en, de, sv, nl, da0fr

AccuracyDiv-2 en, ru
Div-3 en, ru, zh
Div-4 en, ru, zh, ar
Div-5 en, ru, zh, ar, hi
Div-6 en, ru, zh, ar, hi, es
Div-7 en, ru, zh, ar, hi, es, el
Div-8 en, ru, zh, ar, hi, es, el, fi
Div-9 en, ru, zh, ar, hi, es, el, fi, id
Div-10 en, ru, zh, ar, hi, es, el, fi, id, trAfro−Asiatic

1.0+ruPretraining Languages use in the first set of experiments are:Accuracy• Adaptation Corpus: JHU Bible Corpus [4]Accuracy

IE: Slavic

0.500.25

he+ar

For model adaptation on each target language:

POS Accuracy

0.75• Downstream Tasks: POS, NER, NLI

Rel-2
Rel-3
Rel-4
Rel-5

• Computationally intensive

Pretraining Languages

NER F1 Score on Unseen Languages• With model adaptation via continued pretraining, pretraining
on a larger number of languages often gives further
improvement, suggesting that model adaptation is crucial toexploit additional pretraining languages.

Limitations in the First Set of Experiments• Downsampled corpus per language (∼200MB)• Up to 10 pretraining languages

Our findings are• Without model adaptation, surprisingly, increasing the numberof pretraining languages yields better results up to adding
related languages, after which performance plateaus.

Results up to 100 Languages

References
Regression Analysis on RQ1• Predict the POS tagging accuracy Y using X which are the featuresof pretraining and target languages. Typological features areconverted to binary (1 if same, 0 if different).
• Script type match between pretraining languages and the target
language is the most important one[1] F. Ginter, et al., “CoNLL 2017 shared task - automaticallyannotated raw texts and word embeddings.”

Results: Related Languages[2] J. Hu, et al., “XTREME: A massively multilingual multi-task
benchmark for evaluating cross-lingual generalisation,” inProceedings of the International Conference of Machine
Learning.

• Limited cross-lingual transfer across language families
afdeennlru

IE: Germanicbgzh

IE: SlavicarSino−Tibetan

heAfro−Asiatic[3] A. Ebrahimi et al., “How to adapt your pretrained multilingual
model to 1600 languages,” in Proceedings of the Annual Meetingof the Association for Computational Linguistics and theInternational Joint Conference on Natural Language
Processing, 2021.

1.00.5

FeaturesCoef. p-value

Script.061
Family.022
Syntax.001
Phonology.021# pretrain langs .011

CI

< .001 [.050, .073].004 [.007, .036].905 [-.016, .018]< .001 [.009, .033].044 [.000, .022]

0.0hi

mruresIE: Indo−Aryan

fr

it

ptel

IE: Romanceet

IE: GreekfihuUralic1.00.5[4] A. D. McCarthy, et al., “The Johns Hopkins University Bible
corpus: 1600+ tongues for typological exploration,” in Proceedingsof the Language Resources and Evaluation Conference, 2020.

0.0
id
AustronesiantreujaTurkic

BasqueJaponic

1.0[5] G. Lample et al., “Cross-lingual language model pretraining,” inProceedings of Advances in Neural Information Processing
Systems.

0.0vi

ko

Austro−Asiaticta

Koreanic

tefa

Dravidian

IE: Iranian

1.00.5+da+nl+sv+deen+da+nl+sv+deen+da+nl+sv+deen+da+nl+sv+de0.0
enRegression analysis on the POS tagging

0.5[6] A. Conneau, et al., “Unsupervised cross-lingual representation
learning at scale,” in Proceedings of the Association for
Computational Linguistics, 2020.