SlowGuess/NAACL_2022_OCR huggingface dataset:
A Balanced Data Approach for Evaluating Cross-Linguial Transfer: Mapping the Linguistic Blood Bank
Dan Malkin
Tomasz Limisiewicz\*
Gabriel Stanovsky
$\diamond$  School of Computer Science, The Hebrew University of Jerusalem
$\spadesuit$  Faculty of Mathematics and Physics, Charles University in Prague
{dan.malkinhueb, gabriel.stanovsky}@mail.huji.ac.il limisiewicz@ufal.mff.cuni.cz
We show that the choice of pretraining languages affects downstream cross-lingual transfer for BERT-based models. We inspect zero-shot performance in balanced data conditions to mitigate data size confounds, classifying pretraining languages that improve downstream performance as donors, and languages that are improved in zero-shot performance as recipients. We develop a method of quadratic time complexity in the number of languages to estimate these relations, instead of an exponential exhaustive computation of all possible combinations. We find that our method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. Our findings can inform developers of large-scale multilingual language models in choosing better pretraining configurations.
Pretrained language models are setting state-of-the-art results by leveraging raw texts during pretraining (PLMs; Peters et al., 2018; Devlin et al., 2019, inter alia). Interestingly, when pretraining on multilingual corpora, PLMs seem to exhibit zero-shot cross-lingual abilities, achieving non-trivial performance on downstream examples in languages seen only during pretraining. For example, in the bottom of Figure 1, a named entity recognition model finetuned on Russian is capable of predicting correctly name entity tags for texts in English, seen only during pretraining (Pires et al., 2019; Conneau et al., 2020b; K et al., 2020; Conneau et al., 2020a; Lazar et al., 2021; Turc et al., 2021).
Previous analyses examined how several factors contribute to this emerging behavior. For example, parameter sharing and model depth are important in certain configurations (K et al., 2020; Conneau et al., 2020b), as well as typological similarities
1) Bilingual Pretraining Graph
2) Balanced Pretraining on a Subset of Languages
Based on the bilingual graph
3) Downstream Cross-Linguual Transfer
Finetune on donating, zero-shot evaluation on recipient
PER
0
0
C
Jobs founded Apple  $\longrightarrow$  PER, O, ORG
Figure 1: We build a complete, directed graph over a diverse set of 22 languages. Weighted edges show the improvement of bilingual LM over monolingual performance (bold edges represent larger weights). Languages which consistently improve performance are termed "donors" and marked in red, while languages which benefit most are termed "recipients" (marked in blue). We show that our observations hold in several configurations on two downstream tasks.
between languages (Pires et al., 2019), and the choice of specific finetune languages (Turc et al., 2021).
In this work, we focus on an important factor that we find missing in prior work, namely the effect that pretraining languages have on downstream zero-shot performance. In particular, we ask three major research questions: (1) Does the choice
*Work done while visiting the Hebrew University.
$^{1}$ Code and models are publicly available at: github.com/SLAB-NLP/linguistic-blood-bank
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 4903 - 4915
July 10-15, 2022 ©2022 Association for Computational Linguistics
of pretraining languages affect downstream cross-lingual transfer, and if so, to what extent? (2) Is English the optimal pretraining language, when controlling for confounding factors such as data size and domain? And finally, (3) Can we choose pretraining languages to improve downstream zero-shot performance?
In addressing these research questions, we aim to decouple the language from its corresponding dataset. To the best of our knowledge, prior work has conflated pretrain corpus size and its domain with other examined factors, thus skewing results towards over-represented languages, such as English or German (Joshi et al., 2020).<sup>2</sup> To achieve this, we first construct a linguistically-balanced pretraining corpus based on Wikipedia, composed of a diverse set of 22 languages. We carefully control for the amount of data and domain distribution in each of the languages (Section 3).
Next, since the number of pretraining configurations grows exponentially with the number of languages  $n$  represented in the dataset, it is infeasible to exhaustively test all possible configurations, much less extend it for more languages. In Section 4 we propose a novel pretraining-based approach that is quadratic in the number of languages. This is achieved by training all  $\binom{n}{2}$  combinations of bilingual masked language models over our corpus, thus yielding a complete directed graph (Figure 1), where an edge  $l_1 \rightarrow l_2$  estimates how much a language  $l_1$  contributes to zero-shot performance in language  $l_2$ , based only on language modeling performance.
In Section 5, we use the graph to identify languages which generally contribute as pretraining languages (termed "donors"), and languages which often benefit from training with other languages (termed "recipients"). Further, we use the graph to make observations regarding the effect of typological features on bilingual language modeling, and make available an interactive graph explorer.
Finally, our evaluations on two multilingual downstream tasks (part of speech tagging and named entity recognition) lead to three main conclusions (Section 6): (1) the choice of pretraining languages indeed leads to differences in zero-shot performance; (2) controlling for the amount of data allotted for each language during pretraining ques
tions the primacy of English as the main pretraining language; and (3) our hypotheses regarding donors and recipient language hold in both downstream tasks, and against two additional control groups.
2 Metrics for Pretraining-Aware Cross-Lingual Transfer
In this section, we extend existing metrics for zero-shot cross-lingual transfer to account for pretraining languages. Intuitively, our metrics for a model  $M$  and a given downstream task take into account three factors: (1)  $P$ , the set of languages seen during pretraining, (2)  $s \in P$ , the source language used for finetuning, and (3)  $t \in P$ , the target language, seen during inference.
Formally, we adapt the formulation of Hu et al. (2020) to define a pretraining-aware bilingual zero-shot transfer score  $\mathcal{Z}$  as:
$$
\mathcal {Z} _ {P} (s \rightarrow t) := \varepsilon \left(M ^ {P, s}, t\right) \tag {1}
$$
Where  $M^{P,l}$  is a model pretrained on the set of languages  $P$  and finetuned on downstream task instances in the language  $l \in P$ , and  $\varepsilon(M,l)$  is an evaluation of model  $M$  on instances in language  $l$  in terms of the downstream metric, e.g., word-label accuracy for part of speech tagging.
Following, we extend the definition of zero-shot transfer score to a set of downstream test languages  $D \subseteq P$  to measure  $P$ 's aggregated effect on zero-shot performance, by averaging over all bilingual transfer combinations in  $D$ :
$$
\mathcal{Z}_{P}(D) = \frac{1}{|D|^{2} - |D|}\cdot \sum_{\substack{l_{1},l_{2}\in D\\ l_{1}\neq l_{2}}}\mathcal{Z}_{P}(l_{1}\to l_{2}) (2)
$$
In the following sections, we will use these metrics to evaluate how different choices for pretraining languages influence downstream performance.
3 Data Selection
We collect a pretraining dataset to test the effect of pretraining languages on cross-lingual transfer.
First, we choose a set of 22 languages from 9 language families, as listed in Table 1. These represent a wide variety of scripts, as well as typological
4We opt not to normalize the score by the monolingual performance as done in Turc et al. (2021), as we do not want it to affect the score.
2For example, English was X100 more likely to be sampled in mBERT's pretraining data than Icelandic. 3There are  $2^{n}$  possible pretraining configurations taking into account inclusion and omission of every language.
and morphological features. We note that our approach can be readily extended to other languages beyond those selected in this study.
Second, we aim to balance the amount of data and control for its domain across languages, to mitigate possible confounders in our evaluations. Below we outline design choices we make toward this goal.
3.1 Data Balancing
To achieve a balanced dataset across our languages, we sample consecutive sentences from every language's Wikipedia dump from November 2021, such that each language is represented by 10 million characters. This amount was chosen to align all languages to the lower-resource ones (e.g., Piedmontese or Irish) which comprise approximately of 10mb. We choose to sample texts from Wikipedia as it consists of roughly similar encyclopedic domain across languages, and is widely used for training PLMs (Devlin et al., 2019).
Can we balance the amount of information across languages? We note that a possible confound in our study is that languages may encode different amounts of information in texts of similar character count. This may happen due to differences in the underlying texts or in inherent language properties. To estimate the amount of information in each of our  $10^{7}$  character partitions, we tokenize each language partition  $l$  with the same word-piece tokenizer, and look at the ratio between the total number of tokens in  $l$  and the number of unique tokens in  $l$ , finding a good correlation across all our languages ( $r = 0.73$ ), which may indicate that our dataset is indeed balanced in terms of information. Our intuition is that an imbalanced amount of information would lead the tokenizer to "invest" more tokens in some of the languages while neglecting the less informative ones.
Is our sample representative of the full Wikipedia corpus in each language? Another concern may be that our sampled corpus per language is not indicative of the full corpus for that language, which may be much larger (see Table 1). To test this, we create three discrete length distributions. Two length distributions for sentences (in
Language	Code	Family	Size [M chars]
Wiki	Sample
Piedmontese	pms	Indoeuropean	14	10
Irish	ga	Indoeuropean	38	10
Nepali	ne	Indoeuropean	78	10
Welsh	cy	Indoeuropean	85	10
Finnish	fi	Uralic	131	10
Armenian	hy	Indoeuropean	174	10
Burmese	my	Sino-Tibetian	229	10
Hindi	hi	Indoeuropean	473	10
Telugu	te	Dravidian	533	10
Tamil	ta	Dravidian	573	10
Korean	ko	Korean	756	10
Greek	el	Indoeuropean	906	10
Hungarian	hu	Uralic	962	10
Hebrew	he	Afroasiatic	1,261	10
Chinese	zh	Sino-Tibetian	1,546	10
Arabic	ar	Afroasiatic	1,695	10
Swedish	sv	Indoeuropean	1,744	10
Japanese	ja	Japanese	3,288	10
French	fr	Indoeuropean	4,958	10
German	de	Indoeuropean	6,141	10
Russian	ru	Indoeuropean	6,467	10
English	en	Indoeuropean	14,433	10
Table 1: The size of the full Wikipedia dump for the languages in our study (in millions of characters) versus our fixed sized sampling of it. This exemplifies both the linguistic diversity as well as the variance in data sizes in the original Wikipedia corpus, often used for pretraining PLMs. In contrast, we create a balanced pretraining dataset by sampling 10M characters from all languages such that they conform to the smallest language portion in our set (Piedmontese).
terms of words and tokens), and word length distribution in terms of characters. We then compare those three distributions between our sample and the full data using Earth Movers Distance. All means and standard deviations score below 0.001, indicating that indeed all samples are similarly distributed to their respective full corpus in terms of these metrics.
4 Bilingual Pretraining Graph
In this section, we describe a method for estimating the effect that different pretrain language combinations would have on downstream zero-shot performance. This is achieved by evaluating bilingual performance on the pretraining masked language modeling (MLM) task.
We begin by describing our experimental setup, hyperparameters and hardware configuration (Section 4.1). In Section 4.2, we outline our estimation method, yielding a complete graph structure over our languages, amenable for future exploration and analyses (Figures 1, 2). In the following sections,
$^{5}$ Wikipedia dump was obtained and cleaned using wikiextractor (Attardi, 2015).  
 $^{6}$ For example logographic or abjad writing systems may be more condensed than other scripts (Perfetti and Liu, 2005).
Figure 2: Bilingual finetune scores between language pairs in our balanced corpus. Coordinate  $(i,j)$  represents  $\mathcal{F}(l_i\rightarrow l_j)$ , i.e., the performance in MRR[\%] (which correlates with perplexity) of an LM pretrained on a bilingual corpus over languages  $(l_i,l_j)$  and tested intrinsically on  $l_{j}$ . The last column (marked Don.) sums over each line, i.e., index  $i$  in the column represents how much language  $i$  donated to all other languages. Similarly, the  $j^{\prime}$ th index in the last row (marked Recp.) sums over column  $j$  and represents how much language  $l_{j}$  improved in all configurations.
we use the graph to formulate a set of downstream cross-lingual hypotheses regarding how different languages will affect zero-shot performance, and validate these hypotheses on two downstream tasks.
For all evaluations discussed below, we train a BERT model (Devlin et al., 2019) with 4 layers and 4 attention heads, an MLM task head, and an embedding dimension of 512. We train a single wordpiece tokenizer (Wu et al., 2016) on our entire dataset. We train the models with a batch size of 8 samples, with sentences truncated to 128 tokens.
Each language model was trained up to 4 epochs. This was determined by examining the training loss on 6 diverse languages in our set and observing that they converge around 4 epochs. A subset
of 6 languages was trained on 4 additional seeds to verify the stability of the results, as seen in Table 5 and Table 6 in the Appendix. Masks were applied with default settings, generating  $15\%$  mask tokens and  $10\%$  random tokens for each input sequence (Devlin et al., 2019). We used a single GPU core (nvidia Tesla M60, GTX 980, and RTX 2080Ti). Training time varied between 80 - 120 minutes.
4.2 Building a Pretraining Language Graph
Intuitively, we measure MLM performance when pretraining on a pair of languages  $(l_{1}, l_{2})$  as a proxy to the extent of how  $l_{1}$  and  $l_{2}$  contribute to one another in zero-shot cross-lingual transfer.
This methodology relies on two assumptions. First, we assume that the cross-lingual zero-shot performance as defined in Equation 2 is monotonic, i.e., that adding pretraining languages will improve the average downstream performance. This is defined formally as:
$$
P ^ {\prime} \subseteq P \Rightarrow \mathcal {Z} _ {P ^ {\prime}} (D) \leq \mathcal {Z} _ {P} (D) \quad (3)
$$
<sup>7</sup>We use the implementation provided by Hugging Face: https://huggingface.co/bert-base-uncased.
To allow future exploration, we also tokenize over 22 additional languages (listed in the Appendix) which are sampled in the same manner but are not included in this study.
Following this assumption will allow us to extend our bilingual observations to a pretraining language set  $P$  of arbitrary size.
Second, we assume that MLM performance correlates with downstream task performance, which is often the assumption made when training PLMs to minimize perplexity (Peters et al., 2018; Devlin et al., 2019).
Bilingual MLM finetune score. Formally, for every language pair  $s,t\in P$  , we compute the following finetune score,  $\mathcal{F}$  ..
$$
\mathcal {F} (s \rightarrow t) := \frac {\varepsilon \left(M ^ {s , t} , t\right) - \varepsilon \left(M ^ {t} , t\right)}{\varepsilon \left(M ^ {t} , t\right)} \tag {4}
$$
Where  $M^{s,t}$  is a model pretrained on  $s, t$ , and  $\varepsilon$  is an intrinsic evaluation metric for MLM. I.e.,  $\mathcal{F}(s, t)$  estimates how much the target language  $t$  "gains" in the MLM task from additional pretraining on the source language  $s$  compared to monolingual pretraining on  $t$ .
Figure 2 depicts a weighted adjacency matrix where coordinate  $(i,j)$  corresponds to  $\mathcal{F}(l_i\to l_j)$ . As shown in Figure 1, the same information can be conveyed in a complete directed weighted graph, where each node represents a language, and edges  $(l_{1},l_{2})$  are weighted by  $\mathcal{F}(l_1\rightarrow l_2)$ .
Language-Level donation and recipience. Next, for each language  $l \in P$  we compute a Donation score,  $\mathcal{D}$ , as an aggregate over all of its finetune scores as a source language (i.e., how much it contributed to other languages), and similarly a recipience score,  $\mathcal{R}$ , by aggregating over all its finetune scores as a target language, to measure how much  $l$  is contributed to by other languages. Formally:
$$
\mathcal{D}(l):= \sum_{\substack{t\in P\\ t\neq l}}\mathcal{F}(l\to t) \tag{5}
$$
$$
\mathcal{R}(l):= \sum_{\substack{s\in P\\ s\neq l}}\mathcal{F}(s\to l) \tag{6}
$$
We depict both donation and recipience scores as aggregate row and column vectors in Figure 2.
Thus, based on the two assumptions above, our hypothesis is that the downstream cross-lingual transfer will be proportional to the sum of recipi-ence scores for all pretraining languages. Formally:
Figure 3: Our languages on a "donor" versus "recipient" axes. A positive coordinate on the "donor" score (X axis) represents a language that on average improved other languages' performance in bilingual pretraining, while a negative score indicates a language which hurts other languages on average. Inversely, a positive score on the Y axis represents languages whose performance was improved by bilingual pretraining, while negative scores represent languages whose performance was hurt by it. The  $II$  quadrant represents O type languages (donating but not receiving), languages on the  $IV$ 's quadrant are  $\mathrm{AB+}$  type languages (receiving but not donating)
$$
\mathcal {Z} _ {P} (D) \propto \sum_ {l \in D} \mathcal {R} (l) \tag {7}
$$
Moreover, higher donation scores for languages in the pretrain set will result in higher scores in the downstream task. Formally:
$$
\sum_ {l \in P} \mathcal {D} (l) \leq \sum_ {l \in P ^ {\prime}} \mathcal {D} (l) \Rightarrow \mathcal {Z} _ {P} (D) \leq \mathcal {Z} _ {P ^ {\prime}} (D) \tag {8}
$$
5 Pretraining Graph Analysis
We present several key observations based on the bilingual pretraining graph described in the previous section and summarized by the adjacency matrix in Figure 2, as well as an interactive exploration interface. In the following sections, we use these observations in our downstream evaluations.
Some language combinations are detrimental. Negative finetune scores are present in some of the target languages, e.g., between Korean (ko) and Arabic (ar), which means that initializing a
<sup>9</sup>We specifically use mean reciprocal rank (MRR), which correlates with perplexity.
Figure 4: Scatter-plot. Y-axis represents cross-lingual transfer  $\mathcal{F}(l_1\rightarrow l_2)$  for a each possible pair of languages, while the x-axis represents the monolingual MRR score for a source language (left) and the target language (right).
language model for Arabic with weights learned for Korean hurts MLM performance on Arabic, compared to an Arabic monolingual baseline. I.e., in these language configurations, initializing the model with another language model's weights leads to worse performance than random initialization.
Bilingual MLM relations are not symmetric. In fact, we observe a moderate negative correlation between  $\mathcal{F}(l_1\to l_2)$  and  $\mathcal{F}(l_2\rightarrow l_1)$ , as shown in Figure 3. For example, for German and Finnish we get  $0.51 = \mathcal{F}(fi\to de) > \mathcal{F}(de\to fi) = -0.24$ . I.e., Finnish initialization improves German MLM, while the inverse is detrimental for Finnish.
Monolingual performance correlates with donation score. Perhaps expectedly, relatively worse-performing models benefit most from the bilingual transfer, while better-performing monolingual models tend to be better donors, although to a lesser extent (Figure 4).<sup>10</sup>
Different script leads to larger variance in bilingual finetuning. However, language family does not affect it. We find that fine-tuning between languages with different scripts is a high-risk, high-reward scenario. The highest transfer scores occur in this setting, but the proportion of negative scores is also higher. A shared script is a safe setting with a high proportion of neutral or positive donations (Figure 5a). In contrast with recent findings (Pires et al., 2019), we did not observe a
(a) Sharing Script
Figure 5: We divide language pairs into four bins by bilingual finetune score  $(\mathcal{F}(l_1 \to l_2))$ . The figures present the percentage of pairs assigned to each bin for samples of language pairs: (a) written in the same or different script; (b) belonging to the same or different language family. Sharing the language family has no significant effect on the transfer score  $(p > 0.05)$ , while the effect of sharing scripts is significant  $(p < 0.05)$  (p-values based on Pearson's  $\chi^2$  test).
(b) Sharing Family
statistically significant influence for the language family (Figure 5b).
Finetuning as transfusion: mapping the linguistic blood-bank. The non-symmetric nature of the scores gives rise to a coarse-grained ontology loosely reminiscent of human blood types, depicted in Figure 3. Languages which on average donate but do not receive  $(\mathcal{D}(l) > 0$  and  $\mathcal{R}(l) < 0)$  are
10Correlations are statistically significant  $(p < 0.05$  based on Student's T-test).
11We motivate our choice of bins in Appendix.
denoted  $O$  type languages, while the inverse (receiving but not donating) are denoted as  $AB+$  type.
5.1 Interactive Exploration
To allow further exploration of our bilingual pretraining graph, we develop a publicly available web-based interactive exploration interface. We enable exploration of interactions between different linguistic features, based on The World Atlas of Language Structures (WALS) (Dryer and Haspelmath, 2013), allowing users to filter and focus on specific traits and analyze how they affect bilingual pretraining.
6 Downstream Zero-Shot Performance
In this section, we validate our method for estimating the effect of pretraining language combinations on downstream performance. Towards that end, in Section 6.1, we construct several pretraining configurations, based on pretraining observations. Then, in Section 6.2 we describe the multilingual datasets we use for two downstream tasks. Finally, our results are presented in Section 6.3, showing the influence of pretraining configuration on downstream performance.
6.1 Choosing Pretraining Sets
We use the donation scores to identify pretraining languages projected to lead to better downstream zero-shot performance, and the recipience score to find downstream languages which will perform better languages as source (finetune) languages. Our setup is summarized in Table 2.
Donating languages. We define three sets of languages for pretraining, using the donation score while keeping the sets linguistically diverse: (1) Most Donating: Japanese, Telugu, Finnish, and Russian; (2) Least Donating: Nepali, Burmese, Armenian, and English. We also include Englishs as it is a popular source language; and (3) Random:A randomly selected set of 4 languages: Hebrew, Irish, French and Swedish.
Recipient languages. To validate that lower reci- cipience scores indeed indicate that languages are less likely to improve via cross lingual transfer, we added 6 languages to all configurations described above: 3 Most Recipient languages  $(R_{h})$  : Hindi,
German, and Hungarian, and 3 Least Recipient languages  $(R_{l})$ : Arabic, Greek, and Tamil. Finally, we add a fourth control configuration which was pretrained only on  $C\coloneqq R_h\cup R_l$