SlowGuess/NAACL_2022_OCR huggingface dataset:
Hypotheses. We hypothesize that the more donating pretraining sets will improve cross-lingual transfer in downstream tasks, and that more recipient languages will have better cross-lingual performance compared to least recipient languages. These can be formally articulated using Equations 9 and 10:
$$
\forall P: \mathcal {Z} _ {P} (R _ {h}) > \mathcal {Z} _ {P} (R _ {l}) \tag {9}
$$
$$
\mathcal {Z} _ {\text {M o s t D o n .}} (C) > \mathcal {Z} _ {\text {R a n d o m}} (C) > \mathcal {Z} _ {\text {L e a s t D o n .}} (C) \tag {10}
$$
6.2 Tasks
We evaluated all pretraining configurations detailed in Table 2 on two of XTREME's tasks: part of speech tagging (POS) and named entity recognition (NER). Both of which commonly appear in NLP pipelines such as CoreNLP (Manning et al., 2014) and spaCy (Honnibal and Montani, 2017). We aim to balance the data in both tasks across different finetune languages, so as not to skew results towards higher-resource languages.
For part-of-speech tagging, XTREMES borrows from universal dependencies (Nivre et al., 2020). Since XTREMES is imbalanced across languages, we truncated the data to 1000 sentences to fit the lower-resource languages, e.g., XTREMES annotates POS in 909 sentences in Hungarian. For NER, we applied a similar procedure, where XTREMES's data was taken from the Wiskiann (panx) dataset (Rahimi et al., 2019) which we truncated to 5000 sentences (the data size available for Hindi NER in XTREMES).
Experimental setup. We use the code and default hyperparameter default values provided by XTREME to train the downstream tasks (Hu et al., 2020), adapted for multilingual training.
Several key observations can be made based on the results for both POS tagging and NER across all training configurations, which are presented in Tables 3 and 4. For each configuration  $P$  in Most Donating, Least Donating, Random, Control we
12 github.com/SLAB-NLP/ linguistic-blood-bank# interactive-exploration
Base Pretrain Set
(Donors)	Shared Pretrain Set	Total Data	Summary
Most Recipient (Rh)	Least Recipient (Rl)
Most Donating	{ja, te, fi, ru}	{hi, de, hu}	{ar, el, ta}	10^8 characters	Most donating pretraining set.
Least Donating	{ne, my, hy, en}	{hi, de, hu}	{ar, el, ta}	10^8 characters	Least donating pretrain set.
Random	{he, ga, fr, sv}	{hi, de, hu}	{ar, el, ta}	10^8 characters	Random donating pretrain set.
Control	{}	{hi, de, hu}	{ar, el, ta}	10^8 characters	No additional donating languages.
Table 2: Four pretraining language configurations. Each consists of donating languages (first column) and recipient languages (second column). The control group has the same amount of data, equally distributed among its languages.
NER [%F1]	POS [%F1]
Avg. Monolingual	Avg. Zeroshot	Avg. Monolingual	Avg. Zeroshot
Most Donating	49.3±.4	15.6±.4	61.4±.1	28.1±.3
Random	49.2±.3	15.6±.1	61.3±.1	26.9±.3
Least Donating	48.8±.2	14.8±.3	60.9±.2	26.9±.6
Control	49.0±.2	15.6±.2	61.9±.1	27.4±.3
Table 3: Donation results for named entity recognition (NER) and part of speech tagging (POS) as mean and standard deviation over five random seeds. For each pretraining language group (Most Donating, Random, Least Donating, and Control), we report corresponding average monolingual and zero shot performance. Most Donating consistently outperforms Least Donating in both tasks, and in both monolingual and zeroshot performance. Most Donating is on par with Control in monolingual performance in NER, despite having less in-domain data.
NER [%F1]	POS [%F1]
Avg. Monolingual	Avg. Zereshot	Avg. Monolingual	Avg. Zereshot
Most Recipient (Rh)	50.3±.6	18.4±.6	64.1±.3	28.7±.7
Least Recipient (Rl)	47.9±.4	12.4±.4	58.6±.4	26.0±.7
Table 4: Recipience results for named entity recognition (NER) and part of speech tagging (POS) as mean and standard deviation over five random seeds. We report results across different training configurations for two groups of downstream recipient languages. In accordance with our pretraining results, the Most Recipient set does better than the Least recipient set across both tasks in zero-shot and monolingual performance.
calculated zero-shot transfer scores on  $C$ , using  $\mathcal{Z}_P(C)$  defined by Equation 2. Monolingual results under each pretrain set  $P$  were calculated by the average  $F1$  performance of each language in  $C$ :
$$
\frac {1}{| C |} \cdot \sum_ {l \in C} \varepsilon \left(M ^ {P, l}, l\right) \tag {11}
$$
Where  $\varepsilon(M^{P,l}, l)$  denotes the  $F1$  score of a model pretrained on  $P$ , finetuned on  $l$  and evaluated on  $l$ .
Pretraining configuration affects downstream cross-lingual transfer. In both tasks, we observe a variance in results when changing the pretraining configuration, despite all of them having similar amounts of data. This may imply that previous work has omitted an important interfering factor.
Recipience score correlates with downstream cross-lingual performance. We evaluated zero-shot transfer for each language set  $R \in \{R_{l}, R_{h}\}$  as the average zero-shot transfer scores over all pretraining configurations. Table 4 reveals that the Most Recipient set outperforms the Least Recipient
set in both tasks  $(+5.5\%)$  in NER,  $+2.7\%$  in POS tagging).
Multilingual pretraining can improve monolingual performance. As seen in Table 3, the Most Donating pretraining configuration achieved a monolingual score which is slightly higher than the control group, while the Least Donating configuration underperforms all other sets. This suggests that multilingual pretraining datasets can benefit monolingual downstream results compared to more data in a single language.
English might not be an optimal pretraining language. Corresponding with our previous results, if donation score is indicative of a language's contribution in pretraining, English's relative low donation score might indicate that it is not the best language to pretrain upon. English was also part of the Least Donating pretraining configuration which scored lower than Most Donating as seen in Table 3. Further research can ascertain this finding.
As with other works on cross-lingual transfer, our results are influenced by many hyperparameters. Below we explicitly define our design choices and how they can be explored in future work.
First, data scarcity in low-resource languages restricted us to small data amounts. Although our experiments showed a non-trivial signal for pretraining and downstream tasks, future work may apply our framework to larger data sizes.
Second, for efficiency's sake, we trained relatively small models to enable us to train a large number of language configurations, while ensuring convergence in 6 languages. Furthermore, we did not do any hyper-parameter tuning and used only values reported in previous work, and use only the BERT architecture. Future work may revisit any of these design choices to shed more light on their effect.
Third, similarly to other works, our data was scraped from Wikipedia, and we did not account for language contamination across supposedly monolingual corpora (e.g., due to code switching). Such contamination may confound with cross-lingual transfer, as was recently shown by Blevins and Zettlemoyer (2022).
Finally, our downstream analysis focused on POS tagging and NER since they were available for many languages and are common in many NLP pipelines. Further experimentation can test if our results hold for more NLP tasks.
To the best of our knowledge, this is the first work to control for the amount of data allocated for each language during pretraining and finetuning while evaluating on many languages.
Perhaps most related to our work, Turc et al. (2021) challenge the primacy of English as a source language for cross-lingual transfer in various downstream tasks. Their work shows that German and Russian are often more effective sources. In all of their experiments, they use mBERT's imbalanced pretraining corpus. Blevins and Zettlemoyer (2022) complement this hypothesis by showing that English pretraining data actually contains a significant amount of non-English text, which correlates with the model's transfer capabilities.
Wu and Dredze (2020) evaluate how mBERT performs on a wide set of languages, focusing on
the quality of representation for low-resource languages in various downstream tasks by defining a scale from low to high resource. They show that mBERT underperforms non BERT monolingual baselines for low resource languages while performing well for high resource ones.
While Pires et al. (2019); Limisiewicz and Mareček (2021) show that typology plays a significant role for mBERT's multilingual performance, this is not replicated in our balanced evaluation, and has lesser impact in Wu et al. (2022) as well.
Finally, Conneau et al. (2020a) introduce the transfer-interference trade-off where low resource languages benefit from multilingual training, up to a point where the overall performance on monolingual and cross-lingual benchmarks degrades.
We explored the effect of pretraining language selection on downstream zero-shot transfer.
We first choose a diverse pretraining set of 22 languages, and curate a pretraining corpus which is balanced across these languages.
Second, we devise an estimation technique, quadratic in the number of languages, projecting which pretraining languages will serve better in cross-lingual transfer and which specific downstream languages will do best in that setting.
Finally, we test our hypothesis on two downstream multilingual tasks, and show that the choice of pretraining languages indeed leads to varying downstream cross-lingual results, and that our method is a good estimation for downstream performance. Taken together, our results suggest that pretraining language selection should be a factor in estimating cross-lingual transfer, and that current practices which focus on high-resource languages may be sub-optimal.
We would like to thank Roy Schwartz for his helpful comments and suggestions and the anonymous reviewers for their valuable feedback. This work was supported in part by a research gift from the Allen Institute for AI. Tomasz Limisiewicz's visit to the Hebrew University has been supported by grant 338521 of the Charles University Grant Agency and the Mobility Fund of Charles University.
Giuseppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor.
Terra Blevins and Luke Zettlemoyer. 2022. Language contamination explains the cross-lingual capabilities of english pretrained models. ArXiv preprint, abs/2204.08110.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguistics.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: an empirical study. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, and Gabriel Stanovsky. 2021. Filling the gaps in Ancient Akkadian texts: A masked language modelling approach. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4682-4691, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tomasz Limisiewicz and David Mareček. 2021. Examining cross-lingual contextual embeddings with orthogonal structural probes. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4589–4598, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resources Association.
Charles A Perfetti and Ying Liu. 2005. Orthography to phonology and meaning: Comparisons across and within writing systems. Reading and Writing, 18(3):193-210.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164, Florence, Italy. Association for Computational Linguistics.
Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. 2021. Revisiting the primacy of english in zero-shot cross-lingual transfer. ArXiv preprint, abs/2106.16171.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv preprint, abs/1609.08144.
Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin. 2022. Oolong: Investigating what makes crosslingual transfer hard with controlled studies. ArXiv preprint, abs/2202.12312.
Network Graph Visualization of Lang-Lang Interactions
Full list of tokenized languages The full list of Wikipedia language codes for languages used in our tokenizer training is:
- pms, ga, ne, cy, fi, hy, my, hi, te, ta, ko, el, hu, he, zh, ar, sv, ja, fr, de, ru, en - languages that are also evaluated and trained. Elaborated in Table 1.
- af, am, ca, cs, da, es, id, is, it, mg, nl, pl, sk, sw, th, tr, ur, vi, yi - Additional languages corresponding to Afrikaans, Amharic, Catalan, Czech, Danish, Spanish, Indonesian, Icelandic, Italian, Malagasy, Dutch, Polish, Slovak, Swahili, Thai, Turkish, Urdu, Vietnamese, and Yiddish.
Transfer Distribution In the histogram of cross-lingual transfers (Figure 7), we observe that the distribution has multiple local maximums (modes). We distinguish four main levels of cross-lingual transfer described in Section 4.2 ( $\mathcal{F}(l_i \to l_j)$ ):
- negative transfer  $\mathcal{F}(l_i\to l_j) < -10$
- neutral transfer  $-10 \leq \mathcal{F}(l_i \to l_j) < 10$
positive transfer  $10\leq \mathcal{F}(l_i\to l_j) <   55$
- very positive transfer  $55 \leq \mathcal{F}(l_i \to l_j)$
The choice of division borders was done in order to separate distinct modes of the distribution and to obtain interpretable bins (e.g. neutral transfer centered around zero).
Figure 6: Our visualization tool, based on Streamlit (https://streamlit.io)
Figure 7: Histogram of cross-lingual transfers  $\mathcal{F}(l_i\rightarrow l_j)$ . Horizontal lines (at  $-10, 10,$  and  $55$ ) are the borders between four transfer levels.
src/trgt	de	en	he	ne	hi	ja
de	0.2801	0.3177	0.2881	0.2231	0.2685	0.3954
en	0.3401	0.2508	0.2761	0.2238	0.2615	0.3927
he	0.3527	0.3295	0.2612	0.2536	0.2912	0.4041
ne	0.3255	0.2861	0.2716	0.1510	0.2531	0.3887
hi	0.3221	0.2981	0.2873	0.2415	0.2083	0.4045
ja	0.373	0.3536	0.3194	0.2825	0.3232	0.3869
Table 5: Averaged MRR scores for five seeds. Bilingual training was done with five seeds over a group of six diverse languages to verify the results are stable. The table shows mean results. The column indicates the source languages, the row indicates the target languages.
src/trgt	de	en	he	ne	hi	ja
de	0.028	0.0031	0.0118	0.0013	0.0103	0.0068
en	0.0062	0.0229	0.0061	0.0054	0.0071	0.0086
he	0.0037	0.0036	0.0113	0.0019	0.0051	0.0023
ne	0.0287	0.0049	0.0047	0.0041	0.0094	0.0078
hi	0.0033	0.0035	0.0113	0.0276	0.0531	0.0196
ja	0.0057	0.0059	0.0062	0.0066	0.0061	0.0029
Table 6: standard deviations for MRR scores over five seeds. Bilingual training was done with five seeds over a group of six diverse languages to verify the results are stable. The table shows the standard deviation of the results. The column indicates the source languages, the row indicates the target languages.
my	ne	de	hi	en	hu	hy	ar	he	ru	zh	ta	ko	ga	ja	fi	cy	te	el	fr	pms
23.3	24.2	24.7	25.0	25.9	27.2	28.3	32.1	33.4	34.3	36.3	36.4	36.5	36.7	37.6	38.0	39.0	40.0	41.0	41.4	58.9
Table 7: Monolingual results (MRR scores) for all 22 languages in our study, ordered from low to high. Colors coding follows Figure 3 where  $O$  type languages are marked in red and  $AB +$  type languages languages are marked in blue. Monolingual performance explains some of the pretraining contribution, namely recipient languages appear near the low end of the spectrum while donors appear towards the end.