Match the Script, Adapt if Multilingual: Analyzing theEffect of Multilingual Pretraining on Cross-lingual
Transferability
Yoshinari Fujinuma∗1 Jordan Boyd-Graber2 Katharina Kann3
1AWS AI Labs 2 University of Maryland 3 University of Colorado Boulder∗ Done while at University of Colorado Boulder

1/19

Introduction▶ Situation: Traveling in a country where NLP resources are scarce.
▶ Pretrained multilingual language models are applicable to wide variety oflanguages under zero-shot transfer setting.

▶ What happens to languages unseen during pretraining?

2/19

Motivation▶ Issues with multilingual pretraining for language models▶ Curse of multilinguality (Conneau et al., 2020)▶ Negative interference (Wang et al., 2020)▶ Are the above phenomenon also an issue for languages unseen duringpretraining?
▶ What happens if we adapt multilingual models to target languages?

3/19

Research Questions▶ RQ1: How does the number of pretraining languages influence zero-shot
performance on unseen target languages?
▶ RQ2: Do the findings of RQ1 change with model adaptation?
▶ RQ3: Do the findings of RQ1 change if the languages used for pretraining
are all related?

4/19

Research Questions▶ RQ1: How does the number of pretraining languages influence zero-shot
performance on unseen target languages?
▶ RQ2: Do the findings of RQ1 change with model adaptation?
▶ RQ3: Do the findings of RQ1 change if the languages used for pretraining
are all related?

4/19

Experiment Setup: Pretraining Languages▶ Diverse set of languages (Div-X): up to ten languages from diverse language
families for RQ1 and RQ2▶ Related set of languages (Rel-X): up to five Germanic Languages for RQ3
Div-2
Div-3Div-4
Div-5
Div-6
Div-7
Div-8
Div-9
Div-10en, ru
en, ru, zh
en, ru, zh, ar
en, ru, zh, ar, hi
en, ru, zh, ar, hi, es
en, ru, zh, ar, hi, es, el
en, ru, zh, ar, hi, es, el, fi
en, ru, zh, ar, hi, es, el, fi, id
en, ru, zh, ar, hi, es, el, fi, id, tr

Rel-2
Rel-3
Rel-4
Rel-5

en, deen, de, sv
en, de, sv, nl
en, de, sv, nl, da

5/19

Experiment Setup: Model, Data and Tasks▶ Transformer with the same architecture and vocabulary as XLM-R base
▶ Pretraining Corpus: CoNLL 2017 Wikipedia dump (Ginter et al., 2017)▶ Downsampled to ≈ 200MB (to the smallest pretraining language)▶ Task Dataset: XTREME (Hu et al., 2020)
For RQ2 (with model adaptation):▶ Continued pretraining with Masked Language Modeling (Ebrahimi andKann, 2021).
▶ Adaptation Corpus: JHU Bible Corpus (McCarthy et al., 2020)

6/19

RQ1: Results Without Model Adaptation

0.20.1

0en

Div-2 Div-3 Div-4 Div-5 Div-6 Div-7 Div-8 Div-9 Div-10(+ru) (+zh) (+ar) (+hi) (+es) (+el) (+fi) (+id) (+tr)
Pretraining Languages

NER Results

POS Tagging Accuracy on Unseen Languages

NER F1 Score on Unseen Languages▶ Cross-lingual zero-shot accuracy on using diverse set of pretraining languages
▶
0.3

0.2

0.1

0en

Div-2 Div-3 Div-4 Div-5 Div-6 Div-7 Div-8 Div-9 Div-10(+ru) (+zh) (+ar) (+hi) (+es) (+el) (+fi) (+id) (+tr)
Pretraining Languages

POS Tagging Results
7/19

RQ1: Results Without Model Adaptationin Acc)rac+ on Unseen Lan )a es

0.2

0.1

0en

Di+-2 Di+-3 Di+-4 Di+-5 Di+-6 Di+-7 Di+-8 Di+-9 Di+-10(+ru) (+zh) (+ar) (+hi) (+e() (+el) (+ i) (+id) (+)r)Pre)raining Language(NER Results

POS TaNER F1 Score on Un(een Language(▶ Cross-lingual zero-shot accuracy on using diverse set of pretraining languages▶ Average cross-lingual zero-shot accuracy increases up to a certain point
0.3

0.2

0.1

0en

Div-2 Div-3 Div-4 Div-5 Div-6 Div-7 Div-8 Div-9 Div-10(+r)) (+zh) (+ar) (+hi) (+es) (+el) (+fi) (+id) (+(r)
Pre(rainin Lan )a es

POS Tagging Results
7/19

RQ1: Results Without Model Adaptation on Each Language▶
▶
himrurIE: Indo−Aryan

0.5+tr+id+fi+el+es+hi+ar+zh+ru0.0
en

POS Accuracy

1.0Pretraining Languages

8/19

RQ1: Results Without Model Adaptation on Each Language▶ Large increase if adding a pretraining language from the same language▶
himrurIE: Indo−Aryan

0.5+tr+id+fi+el+es+hi+ar+zh+ru0.0
en

POS Accuracy

1.0

Pretraining Languages

8/19

RQ1: Results Without Model Adaptation on Each Language▶ Large increase if adding a pretraining language from the same language▶ But no significant gain e.g., in Urdu after adding Hindi
himrurIE: Indo−Aryan

0.5+tr+id+fi+el+es+hi+ar+zh+ru

0.0en

POS Accuracy

1.0

Pretraining Languages

8/19

RQ1: Results Without Model Adaptation: Regression Analysis▶ Predict the POS tagging accuracy Y using X which are the features ofpretraining and target languages▶ Same or different script / family▶ Syntax and phonology features are from URIEL (Littell et al., 2017)▶ Script match between pretraining and target languages is the mostimportant one
FeaturesCoef.

p-value

CIScript
FamilySyntaxPhonology# pretrain langs

.061
.022
.001
.021
.011

< .001.004.905< .001.044[.050, .073][.007, .036][-.016, .018][.009, .033][.000, .022]
9/19

RQ2: Results With Model Adaptation▶ Trend 1: More languages are better (French and Farsi)
frfr
IE: RomancePretraining Languages

POS Tagging Results+fi+id+tr+tr0.5+el+es+hi+ar+zh0.0en+tr+id+fi+el+es+hi+ar+zh0.01.0+ru

NER F1 score

0.5en+id

IE: Iranian

1.0+ru+elfa

IE: Iranian

POS Accuracy+fi+hifa+esen+ar0.0+tr+fi+id+el+hi+es+ar+zhen

0.00.5+zh0.51.0+ru

NER F1 score

1.0+ruPOS Accuracy

IE: Romance

Pretraining Languages

NER Results
10/19RQ2: Results With Model Adaptation▶ Trend 2: More languages does not necessarily improve (Vietnamese andTamil)
vivi

POS Tagging Results+fi+id+tr+id+tr+el

1.00.5+es+hi+ar+zhen

0.0+ruNER F1 score
Pretraining Languages+tr+id+fi+el+es+hi0.0+fi

ta
Dravidian+el+hi+es+ar+zhen

ta

0.5+ar0.0

Dravidian
1.0+zh0.5+tr+fi+id+el+hi+es+ar+zhen+ru

0.01.0+ru

NER F1 score

0.5en

POS Accuracy

Austro−Asiatic1.0+ru

POS Accuracy

Austro−Asiatic

Pretraining Languages

NER Results

11/19

RQ3: Results w/o Adaptation, Pretrain on Related Languages▶ Within the same language family: Better cross-lingual transfer▶ Limited cross-lingual transfer across language families (please see the paperfor details)
afdeennl

IE: Germanic

0.5+da+nl+sv+de0.0
en

POS Accuracy

1.012/19

NLI Results▶ Trend is less clear on NLI results▶ Reason: NLI requires larger pretraining corpus (Lauscher et al., 2020)
NLI Accuracy on Unseen LanguagesNLI Accuracy on Unseen Languages

0.40.30.20.10en

Div-2(+ru)

Div-3(+zh)

Div-4(+ar)

Div-5(+hi)

Pretraining Languages

Pretrained on Diverse Languages

0.30.20.10en

Rel-2(+de)

Rel-3(+sv)

Rel-4(+nl)

Rel-5(+da)

Pretraining Languages

Pretrained on Germanic Languages
13/19

Limitations in the Previous Experiments▶ Small scale due to pretraining being computationally intensive▶ Downsampled corpus per pretraining language (∼200MB)▶ Up to 10 languages▶ Using XLM-R base vocabulary are not truely unseen for the target languages

14/19

Scaling up to 100+ Languages▶ Use more pretrained models in addition to the pretrained language models
up to 10 languages▶ XLM-17 (Lample and Conneau, 2019)▶ 17 languages, pretrained on full Wikipedia▶ XLM-100 (Lample and Conneau, 2019)▶ 100 languages, pretrained on full Wikipedia▶ XLM-R base (Conneau et al., 2020)▶ 100 languages, pretrained on Common Crawl▶ Target languages unseen when building vocabulary for XLM-17, XLM-100,or XLM-R▶ Maltese, Wolof, Yoruba, Erzya, and Northern Sami
15/19

Results up to 100 Languages▶ More languages does notnecessarily improve if not adapting
Div-5

Div-6

Div-8

Div-10

XLM-17

XLM-100

XLM-R

1

POS Accuracy

0.75

0.50

0.25

0Maltese

Wolof

YorubaErzya

Nothern Sami

Target Language

Before Adaptation
16/19

Results up to 100 Languages▶ More languages does notnecessarily improve if not adapting
Div-6

Div-8

Div-10

XLM-17

XLM-100

XLM-R

Div-5

110.750.75
POS Accuracy

POS Accuracy

Div-5▶ More languages the better whenadapted

0.50

0.25

Div-6

Div-8

Div-10

XLM-17

XLM-100

XLM-R

0.50

0.25

00Maltese

Wolof

YorubaErzya

Target Language

Before Adaptation

Nothern Sami

Maltese

Wolof

YorubaErzya

Nothern Sami

Target Language

After Adaptation
16/19

Conclusion and Future Work
If pretraining a new multilingual model and applying to unseen languages:▶ If not adapting, small set of diverse pretraining languages is likely sufficient▶ Match the script and family▶ If adapting, at least train on 100 languages or possibly more
Future Work:▶ Different vocabulary (Muller et al., 2021; Mielke et al., 2021)▶ Recent models e.g., mT5 (Xue et al., 2021)▶ Beyond simple NLP tasks (e.g., generation tasks)

17/19

References I
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M.,
Zettlemoyer, L., and Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. InACL.
Ebrahimi, A. and Kann, K. (2021). How to adapt your pretrained multilingual model to 1600 languages. InACL-IJCNLP.
Ginter, F., Hajič, J., Luotolahti, J., Straka, M., and Zeman, D. (2017). CoNLL 2017 shared task - automaticallyannotated raw texts and word embeddings. LINDAT/CLARIAH-CZ digital library at the Institute of Formaland Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
Hu, J., Ruder, S., Siddhant, A., Neubig, G., Firat, O., and Johnson, M. (2020). XTREME: A massively
multilingual multi-task benchmark for evaluating cross-lingual generalisation. In ICML.
Lample, G. and Conneau, A. (2019). Cross-lingual language model pretraining. In NeurIPS.
Lauscher, A., Ravishankar, V., Vulić, I., and Glavaš, G. (2020). From zero to hero: On the limitations ofzero-shot language transfer with multilingual Transformers. In EMNLP.
Littell, P., Mortensen, D. R., Lin, K., Kairis, K., Turner, C., and Levin, L. (2017). Uriel and lang2vec:
Representing languages as typological, geographical, and phylogenetic vectors. In EACL.
McCarthy, A. D., Wicks, R., Lewis, D., Mueller, A., Wu, W., Adams, O., Nicolai, G., Post, M., and Yarowsky,
D. (2020). The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration. In LREC.

18/19

References II

Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W. Y., Sagot, B.,and Tan, S. (2021). Between words and characters: A brief history of open-vocabulary modeling andtokenization in NLP. CoRR, abs/2112.10508.
Muller, B., Anastasopoulos, A., Sagot, B., and Seddah, D. (2021). When being unseen from mBERT is just thebeginning: Handling new languages with multilingual language models. In NAACL-HLT.
Wang, Z., Lipton, Z. C., and Tsvetkov, Y. (2020). On negative interference in multilingual models: Findings
and a meta-learning treatment. In EMNLP.
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2021). mT5:
A massively multilingual pre-trained text-to-text transformer.

19/19