Venue: NAACL 2022

# A Balanced Data Approach for Evaluating Cross-Lingual Transfer:

Mapping the Linguistic Blood Bank

Dan Malkin\({}^{\diamondsuit}\)   Tomasz Limisiewicz\({}^{\clubsuit}\)    Gabriel Stanovsky\({}^{\diamondsuit}\)

\({}^{\diamondsuit}\) School of Computer Science, The Hebrew University of Jerusalem

\({}^{\spadesuit}\) Faculty of Mathematics and Physics, Charles University in Prague

{dan.malkinhueb,gabriel.stanovskyy}@mail.huji.ac.il limisiewicz@ufal.mff.cuni.cz

Work done while visiting the Hebrew University.Code and models are publicly available at: github.com/SLAB-NLP/linguistic-blood-bank

###### Abstract

We show that the choice of pretraining languages affects downstream cross-lingual transfer for BERT-based models. We inspect zero-shot performance in balanced data conditions to mitigate data size confounds, classifying pretraining languages that improve downstream performance as _donors_, and languages that are improved in zero-shot performance as _recipients_. We develop a method of quadratic time complexity in the number of languages to estimate these relations, instead of an exponential exhaustive computation of all possible combinations. We find that our method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. Our findings can inform developers of large-scale multilingual language models in choosing better pretraining configurations.1

Footnote 1: Code and models are publicly available at: github.com/SLAB-NLP/linguistic-blood-bank

+
Footnote †: 2022: _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4903 - 4915

## 1 Introduction

Pretrained language models are setting state-of-the-art results by leveraging raw texts during pretraining (PLMs; Peters et al., 2018; Devlin et al., 2019, inter alia). Interestingly, when pretraining on multilingual corpora, PLMs seem to exhibit _zero-shot_ cross-lingual abilities, achieving non-trivial performance on downstream examples in languages seen only during pretraining. For example, in the bottom of Figure 1, a named entity recognition model finetuned on Russian is capable of predicting correctly name entity tags for texts in English, seen only during pretraining [6, 7, 11, 12].

Previous analyses examined how several factors contribute to this emerging behavior. For example, parameter sharing and model depth are important in certain configurations [10, 11], as well as typological similarities between languages [6], and the choice of specific finetune languages [13].

In this work, we focus on an important factor that we find missing in prior work, namely the effect that _pretraining_ languages have on downstream zero-shot performance. In particular, we ask three major research questions: (1) Does the choice

Figure 1: We build a complete, directed graph over a diverse set of 22 languages. Weighted edges show the improvement of bilingual LM over monolingual performance (bold edges represent larger weights). Languages which consistently improve performance are termed “donors” and marked in red, while languages which benefit most are termed “recipients” (marked in blue). We show that our observations hold in several configurations on two downstream tasks.



of pretraining languages affect downstream cross-lingual transfer, and if so, to what extent? (2) Is English the optimal pretraining language, when controlling for confounding factors such as data size and domain? And finally, (3) Can we choose pretraining languages to improve downstream zero-shot performance?

In addressing these research questions, we aim to decouple the _language_ from its corresponding _dataset_. To the best of our knowledge, prior work has conflated pretrain corpus size and its domain with other examined factors, thus skewing results towards over-represented languages, such as English or German Joshi et al. (2020).2 To achieve this, we first construct a _linguistically-balanced_ pretraining corpus based on Wikipedia, composed of a diverse set of 22 languages. We carefully control for the amount of data and domain distribution in each of the languages (Section 3).

Footnote 2: For example, English was X100 more likely to be sampled in mBERT’s pretraining data than Icelandic.

Next, since the number of pretraining configurations grows exponentially with the number of languages \(n\) represented in the dataset, it is infeasible to exhaustively test all possible configurations, much less extend it for more languages.3 In Section 4 we propose a novel pretraining-based approach that is quadratic in the number of languages. This is achieved by training all \({n\choose 2}\) combinations of bilingual masked language models over our corpus, thus yielding a complete directed graph (Figure 1), where an edge \(l_{1}\to l_{2}\) estimates how much a language \(l_{1}\) contributes to zero-shot performance in language \(l_{2}\), based only on language modeling performance.

Footnote 3: There are \(2^{n}\) possible pretraining configurations taking into account inclusion and omission of every language.

In Section 5, we use the graph to identify languages which generally contribute as pretraining languages (termed "donors"), and languages which often benefit from training with other languages (termed "recipients"). Further, we use the graph to make observations regarding the effect of typological features on bilingual language modeling, and make available an interactive graph explorer.

Finally, our evaluations on two multilingual downstream tasks (part of speech tagging and named entity recognition) lead to three main conclusions (Section 6): (1) the choice of pretraining languages indeed leads to differences in zero-shot performance; (2) controlling for the amount of data allotted for each language during pretraining questions the primacy of English as the main _pretraining_ language; and (3) our hypotheses regarding donors and recipient language hold in both downstream tasks, and against two additional control groups.

## 2 Metrics for Pretraining-Aware Cross-Lingual Transfer

In this section, we extend existing metrics for zero-shot cross-lingual transfer to account for _pretraining_ languages. Intuitively, our metrics for a model \(M\) and a given downstream task take into account three factors: (1) \(P\), the set of languages seen during pretraining, (2) \(s\in P\), the _source_ language used for finetuning, and (3) \(t\in P\), the _target_ language, seen during inference.

Formally, we adapt the formulation of Hu et al. (2020) to define a _pretraining-aware_ bilingual zero-shot transfer score \(\mathcal{Z}\) as:4

Footnote 4: We opt not to normalize the score by the monolingual performance as done in Turc et al. (2021), as we do not want it to affect the score.

\[\mathcal{Z}_{P}(s\to t):=\varepsilon(M^{P,s},t) \tag{1}\]

Where \(M^{P,l}\) is a model pretrained on the set of languages \(P\) and finetuned on downstream task instances in the language \(l\in P\), and \(\varepsilon(M,l)\) is an evaluation of model \(M\) on instances in language \(l\) in terms of the downstream metric, e.g., word-label accuracy for part of speech tagging.

Following, we extend the definition of zero-shot transfer score to a set of downstream test languages \(D\subseteq P\) to measure \(P\)'s aggregated effect on zero-shot performance, by averaging over all bilingual transfer combinations in \(D\):

\[\mathcal{Z}_{P}(D)=\frac{1}{|D|^{2}-|D|}\cdot\sum_{\begin{subarray}{c}l_{1},l _{2}\in D\\ l_{1}\neq l_{2}\end{subarray}}\mathcal{Z}_{P}(l_{1}\to l_{2}) \tag{2}\]

In the following sections, we will use these metrics to evaluate how different choices for pretraining languages influence downstream performance.

## 3 Data Selection

We collect a pretraining dataset to test the effect of pretraining languages on cross-lingual transfer.

First, we choose a set of 22 languages from 9 language families, as listed in Table 1. These represent a wide variety of scripts, as well as typological 

and morphological features. We note that our approach can be readily extended to other languages beyond those selected in this study.

Second, we aim to balance the amount of data and control for its domain across languages, to mitigate possible confounders in our evaluations. Below we outline design choices we make toward this goal.

### Data Balancing

To achieve a balanced dataset across our languages, we sample consecutive sentences from every language's Wikipedia dump from November 2021, such that each language is represented by 10 million characters.5 This amount was chosen to align all languages to the lower-resource ones (e.g., Piedmonese or Irish) which comprise approximately of 10mb. We choose to sample texts from Wikipedia as it consists of roughly similar encyclopedic domain across languages, and is widely used for training PLMs Devlin et al. (2019).

Footnote 5: Wikipedia dump was obtained and cleaned using wikiextractor Attardi (2015).

Can we balance the amount of information across languages?We note that a possible confound in our study is that languages may encode different amounts of _information_ in texts of similar character count. This may happen due to differences in the underlying texts or in inherent language properties.6 To estimate the amount of information in each of our \(10^{7}\) character partitions, we tokenize each language partition \(l\) with the same word-piece tokenizer, and look at the ratio between the total number of tokens in \(l\) and the number of unique tokens in \(l\), finding a good correlation across all our languages (\(r=0.73\)), which may indicate that our dataset is indeed balanced in terms of information. Our intuition is that an imbalanced amount of information would lead the tokenizer to "invest" more tokens in some of the languages while neglecting the less informative ones.

Footnote 6: For example logographic or abjad writing systems may be more condensed than other scripts Perfetti and Liu (2005).

Is our sample representative of the full Wikipedia corpus in each language?Another concern may be that our sampled corpus per language is not indicative of the full corpus for that language, which may be much larger (see Table 1). To test this, we create three discrete length distributions. Two length distributions for sentences (in terms of words and tokens), and word length distribution in terms of characters. We then compare those three distributions between our sample and the full data using Earth Movers Distance. All means and standard deviations score below 0.001, indicating that indeed all samples are similarly distributed to their respective full corpus in terms of these metrics.

## 4 Bilingual Pretraining Graph

In this section, we describe a method for estimating the effect that different pretrain language combinations would have on downstream zero-shot performance. This is achieved by evaluating bilingual performance on the pretraining masked language modeling (MLM) task.

We begin by describing our experimental setup, hyperparameters and hardware configuration (Section 4.1). In Section 4.2, we outline our estimation method, yielding a complete graph structure over our languages, amenable for future exploration and analyses (Figures 1, 2). In the following sections,

\begin{table}
\begin{tabular}{p{8.5pt} p{28.5pt} p{28.5pt} r c} \hline \hline
**Language** & **Code** & **Family** & \multicolumn{2}{c}{**Size [M chars]**} \\  & & & Wiki & Sample \\ \hline Piedmonese & pins & Indoeuropean & 14 & 10 \\ Irish & ga & Indoeuropean & 38 & 10 \\ Nepali & ne & Indoeuropean & 78 & 10 \\ Welsh & cy & Indooreuropean & 85 & 10 \\ Finnish & fi & Uralic & 131 & 10 \\ Armenian & hy & Indoeuropean & 174 & 10 \\ Burmese & my & Sino-Tibetain & 229 & 10 \\ Hindi & hi & Indoeuropean & 473 & 10 \\ Telugu & te & Dravidian & 533 & 10 \\ Tamil & ta & Dravidian & 573 & 10 \\ Korean & ko & Korean & 756 & 10 \\ Greek & el & Indoeuropean & 906 & 10 \\ Hungarian & hu & Urlaic & 962 & 10 \\ Hebrew & he & Afroasiatic & 1,261 & 10 \\ Chinese & zh & Sino-Tibetain & 1,546 & 10 \\ Arabic & ar & Afrosaiatic & 1,695 & 10 \\ Swedish & sv & Indoeuropean & 1,744 & 10 \\ Japanese & ja & Japonese & 3,288 & 10 \\ French & fr & Indoeuropean & 4,958 & 10 \\ German & de & Indoeuropean & 6,141 & 10 \\ Russian & ru & Indoeuropean & 6,467 & 10 \\ English & en & Indoeuropean & 14,433 & 10 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The size of the full Wikipedia dump for the languages in our study (in millions of characters) versus our fixed sized sampling of it. This exemplifies both the linguistic diversity as well as the variance in data sizes in the original Wikipedia corpus, often used for pretraining PLMs. In contrast, we create a balanced pretraining dataset by sampling 10M characters from all languages such that they conform to the smallest language portion in our set (Piedmonese).

 

we use the graph to formulate a set of downstream cross-lingual hypotheses regarding how different languages will affect zero-shot performance, and validate these hypotheses on two downstream tasks.

### Experimental Setup

For all evaluations discussed below, we train a BERT model Devlin et al. (2019) with 4 layers and 4 attention heads, an MLM task head, and an embedding dimension of 512.7 We train a single wordpiece tokenizer Wu et al. (2016) on our entire dataset.8 We train the models with a batch size of 8 samples, with sentences truncated to 128 tokens.

Footnote 7: We use the implementation provided by Hugging Face: [https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased).

Footnote 8: To allow future exploration, we also tokenize over 22 additional languages (listed in the Appendix) which are sampled in the same manner but are not included in this study.

Each language model was trained up to 4 epochs. This was determined by examining the training loss on 6 diverse languages in our set and observing that they converge around 4 epochs. A subset of 6 languages was trained on 4 additional seeds to verify the stability of the results, as seen in Table 5 and Table 6 in the Appendix. Masks were applied with default settings, generating 15% mask tokens and 10% random tokens for each input sequence Devlin et al. (2019). We used a single GPU core (nvidia tesla M60, gtx 980, and RTX 2080Ti). Training time varied between 80 - 120 minutes.

### Building a Pretraining Language Graph

Intuitively, we measure MLM performance when pretraining on a pair of languages \((l_{1},l_{2})\) as a proxy to the extent of how \(l_{1}\) and \(l_{2}\) contribute to one another in zero-shot cross-lingual transfer.

This methodology relies on two assumptions. First, we assume that the cross-lingual zero-shot performance as defined in Equation 2 is _monotonic_, i.e., that adding pretraining languages will improve the average downstream performance. This is defined formally as:

\[P^{\prime}\subseteq P~{}~{}\Rightarrow~{}~{}\mathcal{Z}_{P^{\prime}}(D) \leq\mathcal{Z}_{P}(D) \tag{3}\]

Figure 2: Bilingual finetune scores between language pairs in our _balanced_ corpus. Coordinate \((i,j)\) represents \(\mathcal{F}(l_{i}\to l_{j})\), i.e., the performance in MRR[%] (which correlates with perplexity) of an LM pretrained on a bilingual corpus over languages \((l_{i},l_{j})\) and tested intrinsically on \(l_{j}\). The last column (marked _Don._) sums over each line, i.e., index \(i\) in the column represents how much language \(i\)_donated_ to all other languages. Similarly, the \(j\)’th index in the last row (marked _Recp._) sums over column \(j\) and represents how much language \(l_{j}\) improved in all configurations.

 

Following this assumption will allow us to extend our bilingual observations to a pretraining language set \(P\) of arbitrary size.

Second, we assume that MLM performance correlates with downstream task performance, which is often the assumption made when training PLMs to minimize perplexity Peters et al. (2018); Devlin et al. (2019).

Bilingual MLM finetune scoreFormally, for every language pair \(s,t\in P\), we compute the following finetune score, \({\cal F}\):

\[{\cal F}(s\to t):=\frac{\varepsilon(M^{s,t},t)-\varepsilon(M^{t},t)}{ \varepsilon(M^{t},t)} \tag{4}\]

Where \(M^{s,t}\) is a model pretrained on \(s,t\), and \(\varepsilon\) is an intrinsic evaluation metric for MLM.9 I.e., \({\cal F}(s,t)\) estimates how much the target language \(t\) "gains" in the MLM task from additional pretraining on the source language \(s\) compared to monolingual pretraining on \(t\).

Footnote 9: We specifically use mean reciprocal rank (MRR), which correlates with perplexity.

Figure 2 depicts a weighted adjacency matrix where coordinate \((i,j)\) corresponds to \({\cal F}(l_{i}\to l_{j})\). As shown in Figure 1, the same information can be conveyed in a complete directed weighted graph, where each node represents a language, and edges \((l_{1},l_{2})\) are weighted by \({\cal F}(l_{1}\to l_{2})\).

Language-Level donation and recipienceNext, for each language \(l\in P\) we compute a _Donation_ score, \({\cal D}\), as an aggregate over all of its finetune scores as a source language (i.e., how much it contributed to other languages), and similarly a _recipience_ score, \({\cal R}\), by aggregating over all its finetune scores as a target language, to measure how much \(l\) is contributed to by other languages. Formally:

\[{\cal D}(l) :=\sum_{\begin{subarray}{c}t\in P\\ t\neq l\end{subarray}}{\cal F}(l\to t) \tag{5}\] \[{\cal R}(l) :=\sum_{\begin{subarray}{c}s\in P\\ s\neq l\end{subarray}}{\cal F}(s\to l) \tag{6}\]

We depict both donation and recipience scores as aggregate row and column vectors in Figure 2.

Thus, based on the two assumptions above, our hypothesis is that the downstream cross-lingual transfer will be proportional to the sum of recipientence scores for all pretraining languages. Formally:

\[{\cal Z}_{P}(D)\propto\sum_{l\in D}{\cal R}(l) \tag{7}\]

Moreover, higher donation scores for languages in the pretrain set will result in higher scores in the downstream task. Formally:

\[\sum_{l\in P}{\cal D}(l)\leq\sum_{l\in P^{\prime}}{\cal D}(l)\Rightarrow{ \cal Z}_{P}(D)\leq{\cal Z}_{P^{\prime}}(D) \tag{8}\]

## 5 Pretraining Graph Analysis

We present several key observations based on the bilingual pretraining graph described in the previous section and summarized by the adjacency matrix in Figure 2, as well as an interactive exploration interface. In the following sections, we use these observations in our downstream evaluations.

Some language combinations are detrimentalNegative finetune scores are present in some of the target languages, e.g., between Korean (ko) and Arabic (ar), which means that initializing a

Figure 3: Our languages on a “donor” versus “recipient” axes. A positive coordinate on the “donor” score (X axis) represents a language that on average improved other languages’ performance in bilingual pretraining, while a negative score indicates a language which hurts other languages on average. Inversely, a positive score on the Y axis represents languages whose performance was improved by bilingual pretraining, while negative scores represent languages whose performance was hurt by it. The \(II\) quadrant represents O type languages (donating but not receiving), languages on the \(IV\)’s quadrant are AB+ type languages (receiving but not donating)



language model for Arabic with weights learned for Korean hurts MLM performance on Arabic, compared to an Arabic monolingual baseline. I.e., in these language configurations, initializing the model with another language model's weights leads to worse performance than random initialization.

**Bilingual MLM relations are _not_ symmetric.** In fact, we observe a moderate _negative_ correlation between \({\cal F}(l_{1}\to l_{2})\) and \({\cal F}(l_{2}\to l_{1})\), as shown in Figure 3. For example, for German and Finnish we get \(0.51={\cal F}(fi\to de)>{\cal F}(de\to fi)=-0.24\). I.e., Finnish initialization improves German MLM, while the inverse is detrimental for Finnish.

**Monolingual performance correlates with donation score.** Perhaps expectedly, relatively worsening models benefit most from the bilingual transfer, while better-performing monolingual models tend to be better donors, although to a lesser extent (Figure 4).10

Footnote 10: Correlations are statistically significant (\(p<0.05\) based on Student’s T-test).

Different script leads to larger variance in bilingual finetuning. However, language family does not affect it.We find that fine-tuning between languages with different scripts is a high-risk, high-reward scenario. The highest transfer scores occur in this setting, but the proportion of negative scores is also higher. A shared script is a safe setting with a high proportion of neutral or positive donations (Figure 4(a)). In contrast with recent findings [6], we did not observe a statistically significant influence for the language family (Figure 4(b)).

Finetuning as transfusion: mapping the linguistic blood-bank.The non-symmetric nature of the scores gives rise to a coarse-grained ontology loosely reminiscent of human blood types, depicted in Figure 3. Languages which on average donate but do not receive (\({\cal D}(l)>0\) and \({\cal R}(l)<0\)) are

Figure 4: Scatter-plot. Y-axis represents cross-lingual transfer \({\cal F}(l_{1}\to l_{2})\) for a each possible pair of languages, while the x-axis represents the monolingual MRR score for a source language (left) and the target language (right).

 

denoted _O type languages_, while the inverse (receiving but not donating) are denoted as _AB+ type_.

### Interactive Exploration

To allow further exploration of our bilingual pretraining graph, we develop a publicly available web-based interactive exploration interface.12 We enable exploration of interactions between different linguistic features, based on _The World Atlas of Language Structures_ (WALS) Dryer and Haspelmath (2013), allowing users to filter and focus on specific traits and analyze how they affect bilingual pretraining.

Footnote 12: github.com/SLAB-NLP/linguistic-blood-bank#interactive-exploration

## 6 Downstream Zero-Shot Performance

In this section, we validate our method for estimating the effect of pretraining language combinations on downstream performance. Towards that end, in Section 6.1, we construct several pretraining configurations, based on pretraining observations. Then, in Section 6.2 we describe the multilingual datasets we use for two downstream tasks. Finally, our results are presented in Section 6.3, showing the influence of pretraining configuration on downstream performance.

### Choosing Pretraining Sets

We use the donation scores to identify pretraining languages projected to lead to better downstream zero-shot performance, and the recipience score to find downstream languages which will perform better languages as source (finetune) languages. Our setup is summarized in Table 2.

Donating languages.We define three sets of languages for pretraining, using the donation score while keeping the sets linguistically diverse: (1) _Most Donating:_ Japanese, Telugu, Finnish, and Russian; (2) _Least Donating:_ Nepali, Burmese, Armenian, and English. We also include English's as it is a popular source language; and (3) _Random:_A randomly selected set of 4 languages: Hebrew, Irish, French and Swedish.

Recipient languages.To validate that lower recipientence scores indeed indicate that languages are less likely to improve via cross lingual transfer, we added 6 languages to all configurations described above: 3 _Most Recipient_ languages (\(R_{h}\)): Hindi, German, and Hungarian, and 3 _Least Recipient_ languages (\(R_{l}\)): Arabic, Greek, and Tamil. Finally, we add a fourth control configuration which was pretrained only on \(C:=R_{h}\cup R_{l}\).

Hypotheses.We hypothesize that the more donating pretraining sets will improve cross-lingual transfer in downstream tasks, and that more recipient languages will have better cross-lingual performance compared to least recipient languages. These can be formally articulated using Equations 9 and 10:

\[\forall P:\mathcal{Z}_{P}(R_{h})>\mathcal{Z}_{P}(R_{l}) \tag{9}\]

\[\mathcal{Z}_{MostDon.}(C)>\mathcal{Z}_{Random}(C)>\mathcal{Z}_{ LeastDOn.}(C) \tag{10}\]

### Tasks

We evaluated all pretraining configurations detailed in Table 2 on two of XTREME's tasks: part of speech tagging (POS) and named entity recognition (NER). Both of which commonly appear in NLP pipelines such as CoreNLP Manning et al. (2014) and spaCy Honnibal and Montani (2017). We aim to balance the data in both tasks across different finetune languages, so as not to skew results towards higher-resource languages.

For part-of-speech tagging, XTREME borrows from universal dependencies Nivre et al. (2020). Since XTREME is imbalanced across languages, we truncated the data to 1000 sentences to fit the lower-resource languages, e.g., XTREME annotates POS in 909 sentences in Hungarian. For NER, we applied a similar procedure, where XTREME's data was taken from the Wikiann (panz) dataset Rahimi et al. (2019) which we truncated to 5000 sentences (the data size available for Hindi NER in XTREME).

Experimental setup.We use the code and default hyperparameter default values provided by XtREME to train the downstream tasks Hu et al. (2020), adapted for multilingual training.

### Results

Several key observations can be made based on the results for both POS tagging and NER across all training configurations, which are presented in Tables 3 and 4. For each configuration \(P\) in _Most Donating, Least Donating, Random, Control_ we 

calculated zero-shot transfer scores on \(C\), using \({\cal Z}_{P}(C)\) defined by Equation 2. Monolingual results under each pretrain set \(P\) were calculated by the average \(F1\) performance of each language in \(C\):

\[\frac{1}{|C|}\cdot\sum_{l\in C}\varepsilon(M^{P,l},l) \tag{11}\]

Where \(\varepsilon(M^{P,l},l)\) denotes the \(F1\) score of a model pretrained on \(P\), finetuned on \(l\) and evaluated on \(l\).

Pretraining configuration affects downstream cross-lingual transfer.In both tasks, we observe a variance in results when changing the pretraining configuration, despite all of them having similar amounts of data. This may imply that previous work has omitted an important interfering factor.

Recipience score correlates with downstream cross-lingual performance.We evaluated zero-shot transfer for each language set \(R\in\{R_{l},R_{h}\}\) as the average zero-shot transfer scores over all pretraining configurations. Table 4 reveals that the _Most Recipient_ set outperforms the _Least Recipient_ set in both tasks (\(+5.5\%\) in NER, \(+2.7\%\) in POS tagging).

Multilingual pretraining can improve _mono-lingual_ performance.As seen in Table 3, the _Most Donating_ pretraining configuration achieved a monolingual score which is slightly higher than the control group, while the _Least Donating_ configuration underperforms all other sets. This suggests that multilingual pretraining datasets can benefit monolingual downstream results compared to more data in a single language.

English might not be an optimal pretraining language.Corresponding with our previous results, if donation score is indicative of a language's contribution in pretraining, English's relative low donation score might indicate that it is not the best language to pretrain upon. English was also part of the _Least Donating_ pretraining configuration which scored lower than _Most Donating_ as seen in Table 3. Further research can ascertain this finding.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **NER** [\(\% F_{1}\)] & \multicolumn{2}{c}{**POS** [\(\% F_{1}\)]} \\  & Avg. Monolingual & Avg. Zeroshot & Avg. Monolingual & Avg. Zeroshot \\ \cline{2-6}
**Most Recipient (\(R_{h}\))** & **50.3\(\pm.6\)** & **18.4\(\pm.6\)** & **64.1\(\pm.3\)** & **28.7\(\pm.7\)** \\
**Least Recipient (\(R_{l}\))** & 47.9\(\pm.4\) & 12.4\(\pm.4\) & 58.6\(\pm.4\) & 26.0\(\pm.7\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Recipience results for named entity recognition (NER) and part of speech tagging (POS) as mean and standard deviation over five random seeds. We report results across different training configurations for two groups of downstream recipient languages. In accordance with our pretraining results, the _Most Recipient_ set does better than the _Least recipient_ set across both tasks in zero-shot and monolingual performance.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline  & \multicolumn{3}{c}{**NER** [\(\% F_{1}\)]} & \multicolumn{2}{c}{**POS** [\(\% F_{1}\)]} \\  & Avg. Monolingual & Avg. Zeroshot & Avg. Monolingual & Avg. Zeroshot \\ \cline{2-6}
**Most Donating** & **49.3\(\pm.4\)** & **15.6\(\pm.4\)** & **61.4\(\pm.1\)** & **28.1\(\pm.3\)** \\
**Random Control** & **49.2\(\pm.3\)** & **15.6\(\pm.1\)** & 61.3\(\pm.1\) & 26.9\(\pm.3\) \\
**Least Denoting** & 48.8\(\pm.2\) & 14.8\(\pm.3\) & 60.9\(\pm.2\) & 26.9\(\pm.6\) \\
**Control** & 49.0\(\pm.2\) & **15.6\(\pm.2\)** & **61.9\(\pm.1\)** & 27.4\(\pm.3\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Donation results for named entity recognition (NER) and part of speech tagging (POS) as mean and standard deviation over five random seeds. For each pretraining language group (_Most Donating_, _Random_, _Least Donating_, and _Control_), we report corresponding average monolingual and zero shot performance. _Most Donating_ consistently outperforms _Least Donating_ in both tasks, and in both monolingual and zeroshot performance. _Most Donating_ is on par with _Control_ in monolingual performance in NER, despite having less in-domain data.

 



## 7 Limitations and Future Work

As with other works on cross-lingual transfer, our results are influenced by many hyperparameters. Below we explicitly define our design choices and how they can be explored in future work.

First, data scarcity in low-resource languages restricted us to small data amounts. Although our experiments showed a non-trivial signal for pretraining and downstream tasks, future work may apply our framework to larger data sizes.

Second, for efficiency's sake, we trained relatively small models to enable us to train a large number of language configurations, while ensuring convergence in 6 languages. Furthermore, we did not do any hyper-parameter tuning and used only values reported in previous work, and use only the BERT architecture. Future work may revisit any of these design choices to shed more light on their effect.

Third, similarly to other works, our data was scraped from Wikipedia, and we did not account for language contamination across supposedly monolingual corpora (e.g., due to code switching). Such contamination may confound with cross-lingual transfer, as was recently shown by blevins2022multilingual.

Finally, our downstream analysis focused on POS tagging and NER since they were available for many languages and are common in many NLP pipelines. Further experimentation can test if our results hold for more NLP tasks.

## 8 Related Work

To the best of our knowledge, this is the first work to control for the amount of data allocated for each language during pretraining and finetuning while evaluating on many languages.

Perhaps most related to our work, turc2021learning challenge the primacy of English as a source language for cross-lingual transfer in various downstream tasks. Their work shows that German and Russian are often more effective sources. In all of their experiments, they use mBERT's imbalanced pretraining corpus. blevins2022multilingual complement this hypothesis by showing that English pretraining data actually contains a significant amount of non-English text, which correlates with the model's transfer capabilities.

Wu:2020 evaluate how mBERT performs on a wide set of languages, focusing on the quality of representation for low-resource languages in various downstream tasks by defining a scale from low to high resource. They show that mBERT underperforms non BERT monolingual baselines for low resource languages while performing well for high resource ones.

While pires2019language, limisiewicz2021learning show that typology plays a significant role for mBERT's multilingual performance, this is not replicated in our balanced evaluation, and has lesser impact in wu2022multilingual as well.

Finally, conneau2020multilingual introduce the transfer-interference trade-off where low resource languages benefit from multilingual training, up to a point where the overall performance on monolingual and cross-lingual benchmarks degrades.

## 9 Conclusions

We explored the effect of pretraining language selection on downstream zero-shot transfer.

We first choose a diverse pretraining set of 22 languages, and curate a pretraining corpus which is balanced across these languages.

Second, we devise an estimation technique, quadratic in the number of languages, projecting which pretraining languages will serve better in cross-lingual transfer and which specific downstream languages will do best in that setting.

Finally, we test our hypothesis on two downstream multispinual tasks, and show that the choice of pretraining languages indeed leads to varying downstream cross-lingual results, and that our method is a good estimation for downstream performance. Taken together, our results suggest that pretraining language selection should be a factor in estimating cross-lingual transfer, and that current practices which focus on high-resource languages may be sub-optimal.

## Acknowledgements

We would like to thank Roy Schwartz for his helpful comments and suggestions and the anonymous reviewers for their valuable feedback. This work was supported in part by a research gift from the Allen Institute for AI. Tomasz Limisiewicz's visit to the Hebrew University has been supported by grant 338521 of the Charles University Grant Agency and the Mobility Fund of Charles University



## References

* Giusepppe Attardi (2015) Giuseppe Attardi. 2015. Wikiextractor. [https://github.com/attardi/wikiextractor](https://github.com/attardi/wikiextractor).
* Blevins and Zettlemoyer (2022) Terra Blevins and Luke Zettlemoyer. 2022. Language contamination explains the cross-lingual capabilities of english pretrained models. _ArXiv preprint_, abs/2204.08110.
* Conneau et al. (2020a) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau et al. (2020b) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6022-6034, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dryer and Haspelmath (2013) Matthew S. Dryer and Martin Haspelmath, editors. 2013. _WALS Online_. Max Planck Institute for Evolutionary Anthropology, Leipzig.
* Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
* Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 4411-4421. PMLR.
* Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6282-6293, Online. Association for Computational Linguistics.
* Karthikeyan et al. (2020) Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: an empirical study. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Lazar et al. (2021) Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, and Gabriel Stanovsky. 2021. Filling the gaps in Ancient Akkadian texts: A masked language modelling approach. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4682-4691, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Limisiewicz and Marecek (2021) Tomasz Limisiewicz and David Marecek. 2021. Examining cross-lingual contextual embeddings with orthogonal structural probes. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4589-4598, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
* Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zennan. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 4034-4043, Marseille, France. European Language Resources Association.
* Perfetti and Liu (2005) Charles A Perfetti and Ying Liu. 2005. Orthography to phonology and meaning: Comparisons across and within writing systems. _Reading and Writing_, 18(3):193-210.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001, Florence, Italy. Association for Computational Linguistics

Afishin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 151-164, Florence, Italy. Association for Computational Linguistics.
* Turc et al. (2021) Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. 2021. Revisiting the primacy of english in zero-shot cross-lingual transfer. _ArXiv preprint_, abs/2106.16171.
* Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_, pages 120-130, Online. Association for Computational Linguistics.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _ArXiv preprint_, abs/1609.08144.
* Wu et al. (2022) Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin. 2022. Oolong: Investigating what makes crosslingual transfer hard with controlled studies. _ArXiv preprint_, abs/2202.12312.

 



## Appendix A Appendix

Full list of tokenized languagesThe full list of Wikipedia language codes for languages used in our tokenizer training is:

* languages that are also evaluated and trained. Elaborated in Table 1.
* Additional languages corresponding to Afrikaans, Amharic, Catalan, Czech, Danish, Spanish, Indonesian, Icelandic, Italian, Malagasy, Dutch, Polish, Slovak, Swahili, Thai, Turkish, Urdu, Vietnamese, and Yiddish.

Transfer DistributionIn the histogram of cross-lingual transfers (Figure 7), we observe that the distribution has multiple local maximums (modes). We distinguish four main level of cross-lingual transfer described in Section 4.2 (\({\cal F}(l_{i}\to l_{j})\)):

* negative transfer \({\cal F}(l_{i}\to l_{j})<-10\)
* neutral transfer \(-10\leq{\cal F}(l_{i}\to l_{j})<10\)
* positive transfer \(10\leq{\cal F}(l_{i}\to l_{j})<55\)
* very positive transfer \(55\leq{\cal F}(l_{i}\to l_{j})\)

The choice of division borders was done in order to separate distinct modes of the distribution and to obtain interpretable bins (e.g. neutral transfer centered around zero).

Figure 6: Our visualization tool, based on Streamlit ([https://streamlit.io](https://streamlit.io))

Figure 7: Histogram of cross-lingual transfers \({\cal F}(l_{i}\to l_{j})\). Horizontal lines (at \(-10\), \(10\), and \(55\)) are the borders between four transfer levels.





\begin{table}
\begin{tabular}{l c c c c c c c c} \hline src/trgt & **de** & **en** & **he** & **ne** & **hi** & **ja** \\ \hline
**de** & 0.028 & 0.0031 & 0.0118 & 0.0013 & 0.0103 & 0.0068 \\
**en** & 0.0062 & 0.0229 & 0.0061 & 0.0054 & 0.0071 & 0.0086 \\
**he** & 0.0037 & 0.0036 & 0.0113 & 0.0019 & 0.0051 & 0.0023 \\
**ne** & 0.0287 & 0.0049 & 0.0047 & 0.0041 & 0.0094 & 0.0078 \\
**hi** & 0.0033 & 0.0035 & 0.0113 & 0.0276 & 0.0531 & 0.0196 \\
**ja** & 0.0057 & 0.0059 & 0.0062 & 0.0066 & 0.0061 & 0.0029 \\ \hline \end{tabular}
\end{table}
Table 6: standard deviations for MRR scores over five seeds. Bilingual training was done with five seeds over a group of six diverse languages to verify the results are stable. The table shows the standard deviation of the results. The column indicates the source languages, the row indicates the target languages.

\begin{table}
\begin{tabular}{l l l l l l l l l l l l l l l l} \hline
**my** & **ne** & **de** & **bi** & **en** & **bn** & **by** & **ar** & **be** & **ru** & **db** & **ts** & **lo** & **pa** & **ju** & **g** & **t** & **ct** & **fr** & **pms** \\ \hline
23.3 & 24.2 & 24.7 & 25.0 & 25.9 & 27.2 & 28.3 & 32.1 & 33.4 & 34.3 & 36.3 & 36.4 & 36.5 & 36.7 & 37.6 & 38.0 & 39.0 & 420.0 & 41.0 & 41.4 & 58.9 \\ \hline \end{tabular}
\end{table}
Table 7: Monolingual results (MRR scores) for all 22 languages in our study, ordered from low to high. Colors coding follows Figure 3 where _O type languages_ are marked in red and _AB+ type languages_ languages are marked in blue. Monolingual performance explains some of the pretraining contribution, namely recipient languages appear near the low end of the spectrum while donors appear towards the end.

\begin{table}
\begin{tabular}{l l l l l l l l l} \hline src/trgt & **de** & **en** & **he** & **ne** & **hi** & **ja** \\ \hline
**de** & 0.028 & 0.0031 & 0.0118 & 0.0013 & 0.0103 & 0.0068 \\
**en** & 0.0062 & 0.0229 & 0.0061 & 0.0054 & 0.0071 & 0.0086 \\
**he** & 0.0037 & 0.0036 & 0.0113 & 0.0019 & 0.0051 & 0.0023 \\
**ne** & 0.0287 & 0.0049 & 0.0047 & 0.0041 & 0.0094 & 0.0078 \\
**hi** & 0.0033 & 0.0035 & 0.0113 & 0.0276 & 0.0531 & 0.0196 \\
**ja** & 0.0057 & 0.0059 & 0.0062 & 0.0066 & 0.0061 & 0.0029 \\ \hline \end{tabular}
\end{table}
Table 5: Averaged MRR scores for five seeds. Bilingual training was done with five seeds over a group of six diverse languages to verify the results are stable. The table shows mean results. The column indicates the source languages, the row indicates the target languages.