Venue: NAACL 2022

# When is BERT Multilingual? Isolating Crucial Ingredients for

Cross-lingual Transfer

Ameet Deshpande

Department of Computer Science

Princeton University, USA asd@cs.princeton.edu &Partha Talukdar

Google Research

India partha@google.com &Karthik Narasimhan

Department of Computer Science

Princeton University, USA karthikn@cs.princeton.edu

###### Abstract

While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer, there is a lack of consensus in the community as to what shared properties between languages enable transfer on downstream tasks. Analyses involving pairs of natural languages are often inconclusive and contradictory since languages simultaneously differ in many linguistic aspects. In this paper, we perform a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four diverse natural languages and their counterparts constructed by modifying aspects such as the script, word order, and syntax. Among other things, our experiments show that the absence of sub-word overlap significantly affects zero-shot transfer when languages differ in their word order, and there is a strong correlation between transfer performance and word embedding alignment between languages (e.g., \(\rho_{s}=0.94\) on the task of NLI). Our results call for focus in multilingual models on explicitly improving word embedding alignment between languages rather than relying on its implicit emergence.1

Footnote 1: Code is available at [https://github.com/princeton-nlp/MultilingualAnalysis](https://github.com/princeton-nlp/MultilingualAnalysis)

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3610 - 3623

## 1 Introduction

Multilingual language models like XLM Conneau et al. (2020) and Multilingual-BERT Devlin (2019) are trained with masked-language modeling (MLM) objective on a combination of raw text from multiple languages. Surprisingly, these models exhibit decent cross-lingual zero-shot transfer, where fine-tuning on a task in a source language translates to good performance for a different language (target).

Requirements for zero-shot transferRecent studies have provided inconsistent explanations for properties required for zero-shot transfer (hereon, transfer). For example, while Wu and Dredze (2019) conclude that sub-word overlap is vital for transfer, K et al. (2020) demonstrate that it is not crucial, although they consider only English as the source language. While Pires et al. (2019) suggest that typological similarity (e.g., similar SVO order) is essential for transfer, other works Kakwani et al. (2020); Conneau et al. (2020) successfully build multilingual models for dissimilar languages.

Need for systematic analysisA major cause of these discrepancies is a large number of varying properties (e.g., syntax, script, and vocabulary size) between languages, which make isolating crucial ingredients for transfer difficult. Some studies alleviate this issue by creating synthetic languages which differ from natural ones only in specific linguistic properties like script K et al. (2020); Dufter and Schutze (2020). However, their focus is only on English as a source language, and the scale of their experiments is small (in number of tasks or pre-training corpora size), thus limiting the scope of their findings to their settings alone.

Our approachWe perform a systematic study of cross-lingual transfer on bilingual language models trained on a natural language and a systematically _derived_ counterpart. We choose four diverse natural languages (English, French, Arabic, and Hindi) and create _derived_ variants using four different transformations on structural properties such as inverting or permuting word order, altering scripts, or varying syntax (Section 3.2). We train models on each of the resulting sixteen language pairs, and evaluate zero-shot transfer on four downstream tasks - natural language inference (nll), named-entity recognition (ner), part-of-speech tagging (pos), and question-answering (qa).

Our experiments reveal the following:

1. Contrary to previous belief, the absence of sub-word overlap degrades transfer when languages

differ in their word order (e.g., by more than \(40\) F1 points on POS tagging, (SS 4.1)).
2. There is a strong correlation between token embedding alignment and zero-shot transfer across different tasks (e.g., \(\rho_{s}=0.94,p<.005\) for XNLI, Fig 4).
3. Using pre-training corpora from similar sources for different languages (e.g., Wikipedia) boosts transfer when compared to corpora from different sources (e.g., \(17\) F1 points on NER, Fig 3).

To our knowledge, we are the first study to quantitatively show that zero-shot transfer between languages is strongly correlated with token embedding alignment (\(\rho_{s}=0.94\) for nli). We also show that the current multilingual pre-training methods Doddapaneni et al. (2021) fall short of aligning embeddings even between simple natural and derived language pairs, leading to failure in zero-shot transfer. Our results call for training objectives that explicitly improve alignment using either supervised (e.g., parallel corpora or bilingual dictionaries) or unsupervised data.

## 2 Related work

Multilingual pre-training for TransformersThe success of monolingual Transformer language models Devlin et al. (2019); Radford et al. (2018) has driven studies that learn a multilingual language-model (LM) on several languages. Multilingual-BERT (M-BERT) Devlin (2019) is a single neural network pre-trained using the masked language-modeling (MLM) objective on a corpus of text from \(104\) languages. XLM Conneau and Lample (2019) introduced translation language-modeling, which performs MLM on pairs of parallel sentences, thus encouraging alignment between their representations. These models exhibit surprising zero-shot cross-lingual transfer performance Conneau and Lample (2019); K et al. (2020), a setup where the model is fine-tuned on a source language and evaluated on a different target language.

Analysis of cross-lingual transferWhile Pires et al. (2019), Conneau et al. (2020b), and K et al. (2020) showed that transfer works even without a shared vocabulary between languages Wu and Dredze (2019) discovered a correlation between sub-word overlap and zero-shot performance. Conneau et al. (2020b) and Artetxe et al. (2020a) showed that shared parameters for languages with different scripts were crucial for transfer. Pires et al. (2019) and Wu and Dredze (2019) observed that transfer for NER and POS tagging works better between typologically similar languages. However, a study conducted by Lin et al. (2019) showed that there is no simple rule of thumb to gauge when transfer works between languages. Hsu et al. (2019) observed that changing the syntax (SOV) order of the source to match that of the target does not improve performance.

Transfer between real and synthetic LanguagesK et al. (2020) create a synthetic language by changing English's script and find that transfer between it and Spanish works even without common sub-words. However, they use only English as their source language, test only on two tasks, and use a single natural-synthetic language pair. Dufter and Schutze (2020) study transfer between English and _synthetic_ English obtained by changing the script, word order, or model delimiters. However, they use a small corpus (\(228\)K words) compared to current standards (we use \(3\) orders more) and measure only embedding similarity and not zero-shot transfer. A contemporary work Wu et al. (2022) uses synthetic transformations to modify the GLUE dataset Wang et al. (2018) and analyze properties required for good zero-shot transfer, but they perform their experiments only on English and do not perform token embedding alignment analysis. We show that the latter is crucial for good transfer.

## 3 Approach

We first provide some background on bilingual language models (Section 3.1), followed by descriptions of our transformations (Section 3.2), and our training and evaluation setup (Section 3.3).

### Background

Bilingual pre-trainingThe standard setup Conneau and Lample (2019) trains a bilingual language model (_Bi-LM_) on raw text corpora from two languages simultaneously. _Bi-LM_ uses the masked language-modeling loss (\(\mathcal{L}_{\text{MLM}}\)) on the corpora from the two languages (\(\mathcal{C}_{1},\mathcal{C}_{2}\)) separately with no explicit cross-lingual signal:

A shared byte pair encoding tokenizer Sennrich et al. (2015) is trained on \(\mathcal{C}_{1}+\mathcal{C}_{2}\). A single batch contains instances from both languages, but each instance belongs to a single language 

Zero-shot transfer evaluationConsider a bilingual model (_Bi-LM_) pre-trained on two languages, _source_ and _target_. Zero-shot transfer involves fine-tuning _Bi-LM_ on downstream task data from _source_ and evaluating on test data from _target_. This is considered zero-shot because _Bi-LM_ is not fine-tuned on any data belonging to _target_.

### Generating language variants with systematic transformations

Natural languages typically differ in several ways, like the script, word order, and syntax. To isolate the affect of these properties on zero-shot transfer, we obtain _derived_ language corpora (hereon, _de-rived_ corpora) from _original_ (natural) language corpora by performing sentence level transformations (\({\cal T}\)) which change particular properties. For example, an "_inversion_" transformation could be used to invert each sentence in the corpus (_Welcome\({}_{1}\) too \({}_{2}\)NAACL\({}_{3}\) only NAACL\({}_{3}\) to \({}_{2}\) Welcome\({}_{1}\)_). Since the transformation (\({\cal T}\)) is applied on each sentence of the _original_ corpus, the size of the _original_ and the _derived_ corpus is the same. In the following sections, we will use the following notation:

\[\begin{split}{\cal C}_{\mathbf{orig}}&\equiv \text{Original corpus}\\ &=\left\{s_{i}~{}|~{}i=1:N,s_{i}=\text{sentence}\right\}\\ {\cal T}&\equiv\text{Sentence-level transformation}\\ {\cal C}_{\mathbf{deriv}}&\equiv\text{Derived corpus}\\ &=\left\{{\cal T}(\text{sent})~{}|~{}\forall~{}\text{sent}\in{ \cal C}_{\mathbf{orig}}\right\}\end{split}\]

Types of transformationsWe consider four transformations which modify different aspects of sentences (examples in Table 1):

1. **Inversion** (\({\cal T}_{\text{inv}}\)): Invert the order of _tokens_ in the sentence, like in Dufter and Schutze (2020). The first token becomes the last, and vice versa.
2. **Permutation** (\({\cal T}_{\mathbf{perm}}\)): Permute the order of tokens in a sentence uniformly at random. For a sentence of \(n\) tokens, we sample a random ordering with probability \(\frac{1}{n!}\).
3. **Transliteration** (\({\cal T}_{\mathbf{trans}}\)): Change the script of all tokens other than the special tokens (like [CLS]). This creates a _derived_ vocabulary (\({\cal V}_{deriv}\)) with a one-to-one correspondence with the original vocabulary (\({\cal V}_{orig}\)).
4. **Syntax** (\({\cal T}_{\mathbf{syn}}\)): Modify a sentence to match the syntactic properties of a different natural language by re-ordering the dependents of nouns and verbs in the dependency parse. These transformations are stochastic because of the errors in parsing and sampling over possible re-orderings Wang and Eisner (2016).

These transformations allow us to systematically evaluate the effect of corresponding properties on zero-shot transfer. We also consider composed transformations (SS4.2) which consecutively apply two transformations. We note that while real languages typically differ in more than one or two properties considered in our transformations, our methodology remains useful in isolating crucial properties that enable good transfer and can be extended to more transformations.

Transformations for downstream tasksWe obtain the downstream corpus in the _derived_ language (\({\cal D}_{\mathbf{deriv}}\)) by applying the same transformation (\({\cal T}\)) used during pre-training on the _original_ downstream corpus (\({\cal D}_{\mathbf{orig}}\)). Unlike pre-training corpora which contain raw sentences, instances in downstream tasks contain one or more sentences with annotated labels. For text classification tasks like

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Transformation** & **Instance (\(s\))** & **Transformed instance (\({\cal T}(s)\))** \\ \hline _Inversion_ (\({\cal T}_{\text{inv}}\)) & Welcome to NAACL at Seattle & Seattle at NAACL to Welcome \\ _Permutation_ (\({\cal T}_{\mathbf{perm}}\)) & This is a conference & a This conference is \\ _Transliteration_ (\({\cal T}_{\mathbf{trans}}\)) & I am Sam . I am & \(\bigstar_{(\textsc{I})}\bigtriangledown_{(\textsc{am})}\bigtriangleup_{( \textsc{Sam})}\bigtriangleup_{(\textsc{J})}\bigtriangleup_{(\textsc{J})} \bigtriangledown_{(\textsc{am})}\) \\ _Syntax_ (\({\cal T}_{\mathbf{syn}}\)) & Sara (S) ate (V) apples (O) & Sara (S) apples (O) ate (V) \\  & Une table (N) & Une ronde (A) table (N) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of our transformations applied to different sentences (without sub-word tokenization). _Inversion_ inverts the tokens, _Permutation_ samples a random reordering, and _Transliteration_ changes the script. We use symbols (\(\blacktriangle\)) to denote words in the new script and mention the corresponding original word in brackets. _Syntax_ stochastically modifies the syntactic structure. In the first example for _Syntax_, the sentence in Subject-Verb-Object (SVO) order gets transformed to SOV order, and in the second, the sentence in Noun-Adjective (NA) order gets transformed to the AN order. The examples are high probability re-orderings and other ones might be sampled too.

 

NLI, we apply the transformation on each sentence in every dataset instance. For token classification tasks (e.g., NER, POS), any transformation which changes the order of the tokens also changes the order of the labels. We present the mathematical specification in Appendix A.

### Model Training and Evaluation

We now describe our pre-training and zero-shot transfer evaluation setup. Figure 1 provides an overview of pre-training and fine-tuning, and Table 2 summarizes the evaluation metrics we use.

Pre-trainingLet \({\mathcal{C}_{\mathbf{orig}}}\) and \({\mathcal{C}_{\mathbf{deriv}}}\) be the _original_ and _derived_ language pre-training corpora. We train two models for each _original-derived_ pair:

1. **Bilingual Model (_Bi-LM_)**: A bilingual model pre-trained on the combined corpus (\({\mathcal{C}_{\mathbf{orig}}}+{\mathcal{C}_{\mathbf{deriv}}}\)) (Figure 0(a)).
2. **Monolingual Model (_Mono-LM_)**: A monolingual model trained only on \({\mathcal{C}_{\mathbf{orig}}}\) for the same number of steps as _Bi-LM_'s. _MonoLM_ is used as a baseline to measure zero-shot transfer of a model not pre-trained on _derived_.

EvaluationLet \({\mathcal{D}^{\mathbf{train}}_{\mathsf{orig}}}\) and \({\mathcal{D}^{\mathbf{dev}}_{\mathsf{orig}}}\) be the _original_ language training and development sets for a downstream task, and \({\mathcal{D}^{\mathbf{train}}_{\mathsf{deriv}}}\) and \({\mathcal{D}^{\mathbf{dev}}_{\mathsf{deriv}}}\) be the corresponding _derived_ language datasets. For evaluation, we first fine-tune the pre-trained models on a downstream training set and evaluate the resulting model on a development set (Figure 0(b)). Since our goal is to investigate the extent of zero-shot transfer, we require appropriate lower and upper bounds to make informed conclusions. To this end, we compute three metrics, all on the same development set (summarized in Table 2):

* **Bilingual zero-shot transfer (_BZ_)**: This is the standard zero-shot transfer score Conneau and Lample (2019) which measures how well a bilingual model fine-tuned on \({\mathcal{D}^{\mathbf{train}}_{\mathsf{orig}}}\) zero-shot transfers to the other language (\({\mathcal{D}^{\mathbf{dev}}_{\mathsf{deriv}}}\)).
* **Bilingual supervised synthetic (_BS_)**: This is the supervised learning performance on the _derived_ language obtained by fine-tuning _Bi-LM_ on \({\mathcal{D}^{\mathbf{train}}_{\mathsf{deriv}}}\) and evaluating it on \({\mathcal{D}^{\mathbf{dev}}_{\mathsf{deriv}}}\).
* **Monolingual zero-shot transfer (_MZ_)**: This measures the zero-shot performance of the baseline _Mono-LM_, which is not pre-trained on the _derived_ language, by fine-tuning _MonoLM_ on \({\mathcal{D}^{\mathbf{train}}_{\mathsf{orig}}}\) and evaluating it on \({\mathcal{D}^{\mathbf{dev}}_{\mathsf{deriv}}}\).

_BS_ uses fine-tuning train data from the _derived_ language and serves as an upper-bound on _BZ_ and _MZ_ 

which don't use it. _MZ_ doesn't pre-train on the _derived_ language and serves as a lower-bound on _BZ_ which does pre-train on it. For easier comparison of _BZ_ and _MZ_ with _BS_ (upper-bound), we report the following score differences (Table 2), which are both negative in our experiments.

\[\Delta_{(\text{BZ}-\text{BS})}=(\text{BZ}-\text{BS}) \tag{1}\] \[\Delta_{(\text{MZ}-\text{BS})}=(\text{MZ}-\text{BS}) \tag{2}\]

_BZ_ alone cannot capture the quality of the zero-shot transfer. A large and negative \(\Delta_{(\text{BZ}-\text{BS})}\) implies that bilingual zero-shot transfer is much worse than supervised fine-tuning on _derived_. Concurrently, \(\Delta_{(\text{BZ}-\text{BS})}\approx\Delta_{(\text{MZ}-\text{BS})}\) implies that _Bi-LM_ transfers as poorly as _Mono-LM_. **Thus, good zero-shot transfer is characterized by \(\Delta_{(\text{BZ}-\text{BS})}\approx 0\) and \(\Delta_{(\text{BZ}-\text{BS})}\gg\Delta_{(\text{MZ}-\text{BS})}\).**

### Experimental Setup

LanguagesWe choose four diverse natural languages: English Indo-European, Germanic), French Indo-European, Romance), Hindi (Indo-European, Indo-Iranian), and Arabic Afro-Asiatic, Semitic), which are represented in the multilingual XTREME benchmark Hu et al. (2020). For each language, we consider four transformations Section 3.2) to create _derived_ counterparts, giving us \(16\) different original-derived pairs in total. For the _Syntax_ transformation, we use Qi et al. (2020) for parsing. We modify the syntax of fr, hi, and ar to that of en, and the syntax of en to that of fr.

DatasetsFor the pre-training corpus (\(\mathcal{C}_{\mathbf{orig}}\)), we use a \(500\)MB (uncompressed) subset of Wikipedia (\(\approx\) 100M tokens) for each language. This matches the size of WikiText-103 Merity et al. (2016), a standard language-modeling dataset. For downstream evaluation, we choose four tasks from the XTREME benchmark Hu et al. (2020). Table 4 lists all the datasets and their evaluation metrics.

Implementation DetailsWe use a variant of RoBERTa Liu et al. (2019) which has \(8\) layers, \(8\) heads, and a hidden dimensionality of \(512\). We train each model on \(500\)K steps, a batch size of \(128\), and a learning rate of \(1e\)-\(4\) with a linear warmup of \(10\)K steps. We use an _original_ language vocabulary size of \(40000\) for all the models and train on \(8\) Cloud TPU v3 cores for \(32\)-\(48\) hours. For fine-tuning, we use standard hyperparameters Appendix F) from the XTREME benchmark and report our scores on the development sets.

## 4 Results

Our experiments reveal several interesting findings for bilingual models including the situational importance of sub-word overlap for zero-shot transfer (SS 4.1, 4.2), the effect of domain mismatch between languages (SS 4.3), and correlation of zero-shot performance with embedding alignment (SS 4.4). We connect our findings to zero-shot transfer results between natural languages in Section 4.5.

### Sub-word overlap is not strictly necessary for strong zero-shot transfer

Sub-word overlap is the number of common tokens between two different language corpora. If \(\mathcal{E}_{1}\) and \(\mathcal{E}_{2}\) are sets of tokens which appear in the two corpora, then: Sub-word overlap \(=|\mathcal{E}_{1}\cap\mathcal{E}_{2}|/|\mathcal{E}_{1}\cup\mathcal{E}_{2}|\)Pires et al. (2019). The _Transliteration_ transformation (\(\mathcal{T}_{\mathbf{trans}}\)) creates _original-derived_ language pairs that have \(0\%\) sub-word overlap (equivalently, different scripts), but follow the same word order.

Table 3 displays \(\Delta_{(\text{BZ}-\text{BS})}\) scores for \(\mathcal{T}_{\mathbf{trans}}\), averaged over four languages (Appendix B contains a breakdown). We observe that \(\Delta_{(\text{BZ}-\text{BS})}\!\approx 0\) for all tasks while \(\Delta_{(\text{MZ}-\text{BS})}\) is highly negative, implying that zero-shot transfer is strong and on par with supervised learning. This result indicates that zero-shot transfer is possible even when languages with different scripts have similar word orders (in line with K et al. (2020)). However, it is unrealistic for natural languages to differ only in their script and not other properties (e.g., word order).

Absence of sub-word overlap significantly hurts zero-shot performance when languages differ in their word-orders

To simulate a more realistic scenario, we create _original_ and _derived_ language pairs which differ both in their scripts (\(0\%\) sub-word overlap) and in word order. We achieve this by composing two transformations on the _original_ language corpus, one of which is _Transliteration_ (\(\mathcal{T}_{\mathbf{trans}}\)). We experiment with three different compositions, (a) \(\mathcal{T}_{\mathbf{trans}}\)\(\circ\)\(\mathcal{T}_{\mathbf{inv}}\), (b) \(\mathcal{T}_{\mathbf{trans}}\)\(\circ\)\(\mathcal{T}_{\mathbf{perm}}\), and (c) \(\mathcal{T}_{\mathbf{trans}}\)\(\circ\)\(\mathcal{T}_{\mathbf{syn}}\). Here, \(\boldsymbol{\alpha}\)\(\circ\)\(\beta\) means that transformation \(\beta\) is applied before \(\boldsymbol{\alpha}\). A composed transformation (\(\mathcal{T}_{\mathbf{trans}}\) 

\(\mathbf{\beta}\)) differs from its second constituent (\(\mathbf{\beta}\)) in that the former produces a _derived_ language which has \(0\%\) sub-word overlap with the _original_ language whereas the latter has a \(100\%\) sub-word overlap.

ResultsOur results (Figure 2, breakdown in Appendix C) show that zero-shot performance is significantly hurt for composed transformations when compared to its constituents. \(|\Delta_{\text{(BZ-BS)}}|\) is much larger for \(\mathcal{T}_{\mathbf{trans}}\circ\mathcal{T}_{\mathbf{inv}}\) when compared to \(\mathcal{T}_{\mathbf{trans}}\) or \(\mathcal{T}_{\mathbf{inv}}\) individually. For example, for XNLI, \(|\Delta_{\text{(BZ-BS)}}|=19\) for the composed transformation and just \(2\) and \(3\) for \(\mathcal{T}_{\mathbf{trans}}\) and \(\mathcal{T}_{\mathbf{inv}}\) individually. \(\mathcal{T}_{\mathbf{trans}}\circ\mathcal{T}_{\mathbf{perm}}\) is worse by \(\approx 20\) points on XNLI and NER, and over \(40\) points on POS when compared to \(\mathcal{T}_{\mathbf{perm}}\). \(\mathcal{T}_{\mathbf{trans}}\circ\mathcal{T}_{\mathbf{syn}}\) suffers lesser than the other two composed transformations, but it is still worse than \(\mathcal{T}_{\mathbf{syn}}\) by \(3\), \(6\), and \(1\) point on XNLI, NER, and POS. In conclusion, the absence of sub-word overlap significantly degrades zero-shot performance in the realistic case of languages with different word orders.

### Data from the same domain boosts bilingual performance

Previously, we considered transformations (\(\mathcal{T}\)) that modified the _original_ pre-training corpus to get a parallel corpus, \(\mathcal{C}_{\mathbf{deriv}}=\mathcal{T}\left(\mathcal{C}_{\mathbf{orig}}\right)\), such that there is a one-to-one correspondence between sentences in \(\mathcal{C}_{\mathbf{orig}}\) and \(\mathcal{C}_{\mathbf{deriv}}\) (we call this setting _parallel_). Since procuring large parallel corpora is expensive in practice, we consider two other settings which use different corpora for _original_ and _derived_.

SetupConsider two text corpora of the same size, \(\mathcal{C}^{\mathbf{1}}_{\mathbf{orig}}\) and \(\mathcal{C}^{\mathbf{2}}_{\mathbf{orig}}\). We compare two settings: (1) The _parallel_ setting pre-trains a bilingual model on \(\mathcal{C}^{\mathbf{1}}_{\mathbf{orig}}+\mathcal{T}(\mathcal{C}^{\mathbf{1 }}_{\mathbf{orig}})\), whereas the (2) _non-parallel_ corpus setting uses \(\mathcal{C}^{\mathbf{1}}_{\mathbf{orig}}+\mathcal{T}(\mathcal{C}^{\mathbf{2 }}_{\mathbf{orig}})\). We consider two variants of _non-parallel_, (1) _non-parallel (same)_ which uses different splits of Wikipedia data (hence, _same_ domain), and (2) _non-parallel (diff)_ which uses Wikipedia data for the _original_ and common crawl data (web text) for the _derived_ language (hence, _different_ domain). We use the _Transliteration_ transformation (\(\mathcal{T}_{\mathbf{trans}}\)) to generate the _derived_ language corpus and report \(|\Delta_{\text{(BZ-BS)}}|\) averaged over all languages in Figure 3.

ResultsWe observe consistently on all tasks that the _parallel_ setting (blue bar) performs better than both the non-parallel settings. _Non-parallel (same)_

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Dataset** & **Task** & **Metric** \\ \hline XNLI Conneau et al. (2018) & NLI Accuracy Wikian Pan et al. (2017) & NER F1 \\ UD v2.5 Nivre et al. (2018) & POS & F1 \\ XQuAD Artetxe et al. (2020) & QA & F1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: XTREME benchmark datasets used for zero-shot transfer evaluation. NLI=Natural Language Inference, NER=Xamed-entity recognition, POS=Part-of-speech tagging, QA=Question-Answering.

 

performs better than _non-parallel (diff)_, with gains ranging between \(2\) points on XQuAD to \(17\) points on NER. This result shows that even for _original_ and _derived_ language pairs which differ only in their script, having parallel pre-training corpora leads to the best zero-shot transfer. Since large-scale parallel unsupervised data is hard to procure, the best alternative is to use corpora from similar domains (Wikipedia) rather than different ones (Wikipedia v.s. web text).

### Zero-shot performance is strongly correlated with embedding alignment

Our previous results (SS 4.2, 4.3) showed cases where zero-shot transfer between languages is poor when there is no sub-word overlap. To investigate this further, we analyze the static word embeddings learned by bilingual models and find that zero-shot transfer between languages is strongly correlated with the alignment between word embeddings for the _original_ and _derived_ languages.

SetupThe _original_ and the _derived_ languages have a one-to-one correspondence between their sub-word vocabularies when we use _transliteration_ (\({\cal T}_{\mathbf{trans}}\)). For a token embedding in the _original_-language embedding matrix, its alignment score is \(100\%\) if it retrieves the corresponding token embedding in the _derived_ language when a nearest-neighbor search is performed, and \(0\%\) otherwise. We average the alignment score over all the tokens and call it _alignment_.

ResultsWe measure the _alignment_ of bilingual models pre-trained on different _original-derived_ language pairs created using _translation_, namely the composed transformations (SS 4.2), _parallel_, and _non-parallel_ (SS 4.3). We plot the _alignment_ along with the corresponding scores for XNLI in Figure 4. Results for other tasks are in Appendix E.

We observe that higher _alignment_ is associated with lower \(\Delta_{\text{(BZ-BS)}}\), implying better zero-shot transfer. _Alignment_ is lower for composed transformations like \({\cal T}_{\mathbf{trans}}\circ{\cal T}_{\mathbf{inv}}\) and \({\cal T}_{\mathbf{trans}}\circ{\cal T}_{\mathbf{perm}}\) which have large and negative \(\Delta_{\text{(BZ-BS)}}\). _Alignment_ also explains the results in Section 4.3, with _non-parallel_ variants having lower alignment scores than _parallel_, which is in line with their lower \(\Delta_{\text{(BZ-BS)}}\). Overall, we find a strong and significant Spearman's rank correlation between _alignment_ and \(\Delta_{\text{(BZ-BS)}}\), with \(\rho=0.94,p<.005\) for XNLI, \(\rho=0.93,p<.005\) for NER, and \(\rho=0.89,p<.01\) for POS, indicating that increasing the embedding alignment between languages helps improve zero-shot transfer 



### Connections to results on natural language pairs

Effect of sub-word overlapIn SS 4.2, we showed that when languages have different scripts (\(0\%\) sub-word overlap), zero-shot transfer significantly degrades when they additionally have different word orders. However, the zero-shot transfer is good when languages differ only in the script and have similar or the same word order. This is in line with anecdotal evidence in Pires et al. (2019), where zero-shot transfer works well between _English_ and _Bulgarian_ (different script but same subject-verb-object order - SVO), but is poor between _English_ and _Japanese_ (different script _and_ word order - SVO v.s. SOV). Our result also corroborates findings in Conneau et al. (2020b) that artificially increasing sub-word overlap between natural languages (which have different word orders) improves performance (e.g., \(3\) points on XNLI).

Effect of token embedding alignmentIn SS 4.4, we showed that zero-shot transfer is strongly correlated with word embedding alignment between languages. This explains the usefulness of recent studies which try to improve multilingual pre-training with the help of auxiliary objectives, which improve word or sentence embedding alignment.

DICT-MLM (Chaudhary et al., 2020) and RelateLM (Khemchandani et al., 2021) require the model to predict cross-lingual synonyms as an auxiliary objective, thus indirectly improving word-embedding alignment and the zero-shot performance on multiple tasks. Hu et al. (2021) add an auxiliary objective that implicitly improves word embedding alignment and show that they can achieve performance similar to larger models. Cao et al. (2019) explicitly improve contextual word embedding alignment with the help of word-level alignment information in machine-translated cross-lingual sentence pairs. Since they apply this post hoc and not during pre-training, the improvement, albeit significant, is small (\(2\) points on XNLI). While these studies do not fully utilize word and sentence embedding alignment information, our results lead us to posit that they are a step in the right direction and that baking alignment information more explicitly into pre-training will be beneficial.



Conclusion

Through a systematic study of zero-shot transfer between four diverse natural languages and their counterparts created by modifying specific properties like the script, word order, and syntax, we showed that (1) absence of sub-word overlap hurts zero-shot performance when languages differ in their word order, and (2) zero-shot performance is strongly correlated with word embedding alignment between languages. Some recent studies have implicitly or unknowingly attempted to improve alignment and have shown slight improvements in zero-shot transfer performance. However, our results lead us to posit that explicitly improving word embedding alignment during pre-training by using either supervised (e.g., parallel sentences and translation dictionaries) or unsupervised data will significantly improve zero-shot transfer. Although real languages typically differ in more ways than the set of properties considered in our transformations, our methodology is still useful to help isolate crucial properties for transfer. Future work can experiment with more sophisticated transformations and investigate closer connections with human language pairs.

## Acknowledgments

This work was funded through a grant from the Chadha Center for Global India at Princeton University. We thank Shunyu Yao, Vishvak Murahari, Tianyu Gao, Sadhika Malladi, and Jens Tuyls for reading our draft and providing valuable comments. We also thank the Google Cloud Research program for computational support in running our experiments.

## References

* Arettex et al. (2020a) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a. On the cross-lingual transferability of monolingual representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 4623-4637. Association for Computational Linguistics.
* Artetxe et al. (2020b) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. On the cross-lingual transferability of monolingual representations. _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
* Bowman et al. (2015) Samuel Bowman, Christopher Potts, and Christopher D Manning. 2015. Recursive neural networks can learn logical semantics. In _Proceedings of the 3rd workshop on continuous vector space models and their compositionality_, pages 12-21.
* Cao et al. (2019) Steven Cao, Nikita Kitaev, and Dan Klein. 2019. Multilingual alignment of contextual word representations. In _International Conference on Learning Representations_.
* Chaudhary et al. (2020) Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. 2020. DICT-MLM: improved multilingual pre-training using bilingual dictionaries. _CoRR_, abs/2010.12566.
* Conneau et al. (2020a) Alexis Conneau, Kartiky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 8440-8451. Association for Computational Linguistics.
* Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 7057-7067.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnlli: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Conneau et al. (2020b) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6022-6034.
* Devlin (2019) Jacob Devlin. 2019. Multilingual-BERT. [https://github.com/google-research/bert/blob/master/multilingual.md](https://github.com/google-research/bert/blob/master/multilingual.md). [Online; accessed 20-December-2020].
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Doddapaneni et al. (2021) Sumanth Doddapaneni, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra. 2021. A primer on pretrained multilingual language models. _arXiv preprint arXiv:2107.00676_ 

Philipp Dufter and Hinrich Schutze. 2020. Identifying elements essential for bert's multilinguality. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4423-4437.
* Hsu et al. (2019) Tsung-Yuan Hsu, Chi-Liang Liu, and Hung-Yi Lee. 2019. Zero-shot reading comprehension by cross-lingual transfer learning with multi-lingual language representation model. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5933-5940.
* Hu et al. (2021) Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. 2021. Explicit alignment objectives for multilingual bidirectional encoders. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3633-3643.
* Hu et al. (2020)Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In _International Conference on Machine Learning_, pages 4411-4421. PMLR.
* K. K. Zihan Wang, Stephen Mayhew, and Dan Roth (2020)Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: an empirical study. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchuktutan, Satish Golla, NC Gokul, Avik Bhattacharyya, Mitesh M Khapra, and Pratyush Kumar. 2020. nlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_, pages 4948-4961.
* Khemchandani et al. (2021) Yash Khemchandani, Sarvesh Mehtunai, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021. Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1312-1323, Online. Association for Computational Linguistics.
* Lin et al. (2019) Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. 2019. Choosing transfer languages for cross-lingual learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3125-3135.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretrained approach. _arXiv preprint arXiv:1907.11692_.
* Merity et al. (2016)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. _arXiv preprint arXiv:1609.07843_.
* Nivre et al. (2018) Joakim Nivre, Mitchell Abrams, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, and Mohammed Attia et al. 2018. Universal dependencies 2.2. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
* Pan et al. (2017) Xiaoman Pan, Boliano Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1946-1958.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 101-108.
* Radford et al. (2018)Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Sennrich et al. (2015) Rico Sennrich, Barry Hadlow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. _arXiv preprint arXiv:1508.07909_.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In _International Conference on Learning Representations_.
* Wang and Eisner (2016) Dingquan Wang and Jason Eisner. 2016. The galactic dependencies treebanks: Getting more data by synthesizing new languages. _Transactions of the Association for Computational Linguistics_, 4:491-505.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, benttz, becas: The surprising cross-lingual effectiveness of BERT. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 833-844. Association for Computational Linguistics 

Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin. 2022. OOLOong: Investigating what makes crosslingual transfer hard with controlled studies. _arXiv preprint arXiv:2202.12312_.

 



## Appendices

### Mathematical Specification for Transformation of Downstream Datasets

Text classificationText classification tasks like news classification or sentiment analysis typically have instances which contain a single sentence and a label. Instances in other classification tasks like natural language inference (NLI) Bowman et al. (2015) contain two sentences and one label. For such tasks, we apply the transformation (\({\mathcal{T}}\)) on each sentence within every instance, and leave the annotated label as is. Therefore, for a dataset of size \(n\) which contains \(m\) sentences per instance, we have:

\[{\mathcal{D}_{\mathtt{orig}}}=\left\{(s_{i1},\ldots,s_{im},y_{i})~ {}|~{}i=1:N\right\}\\ {\mathcal{D}_{\mathtt{deriv}}}=\left\{({\mathcal{T}}(s_{i1}), \ldots,{\mathcal{T}}(s_{im}),y_{i})~{}|~{}i=1:N\right\}\]

Token-classification tasks Tasks like namedentity recognition (NER) and part-of-speech tagging (POS tagging) have labels associated with _each_ token in the sentence. For these datasets, we ensure that any transformation (\({\mathcal{T}}\)) that changes the order of the tokens also changes the order of the corresponding labels.

We define a few quantities to express the transformation mathematically. Let \(s_{i}=(w_{i1},\ldots,w_{ik})\) be a sentence comprised of \(k\) tokens and \(y_{i}=(y_{i1},\ldots,y_{ik})\) be labels corresponding to the tokens in the sentence. We define a new transformation (\({\mathcal{T}}_{aug}\)) which operates on the label augmented sentence, \(s^{aug}_{i}=\left((w_{i1},y_{i1})\,,\ldots,(w_{ik},y_{ik})\right)\). Let \(s^{aug}_{i}[j]\) correspond to the \(j^{th}\) element in the sequence, and \(s^{aug}_{i}[j][\texttt{word}]\) and \(s^{aug}_{i}[j][\texttt{label}]\) correspond to the word and label of the \(j^{th}\) element. Let \({\mathcal{T}}_{aug}(s^{aug}_{i})[j][\texttt{orig}]\) denote the index of the \(j^{th}\) element in the transformed sequence with respect to the original sequence \(s^{aug}_{i}\). Then, the new transformation \({\mathcal{T}}_{aug}\) is such that,

\[{\mathcal{T}}_{aug}(s^{aug}_{i})[j][\texttt{orig}]={\mathcal{ T}}(s_{i})[j][\texttt{orig}]\]

Let \(\texttt{orig\_j}={\mathcal{T}}_{aug}(s^{aug}_{i})[j][\texttt{orig}]\)

\({\mathcal{T}}_{aug}(s^{aug}_{i})[j][\texttt{label}]=s^{aug}_{i}[\texttt{ orig\_j}][\texttt{label}]\)

We transform the dataset using \({\mathcal{T}}_{aug}\):

\[{\mathcal{D}_{\mathtt{orig}}}=\left\{s^{aug}_{i}~{}|~{}i=1:N\right\}\\ {\mathcal{D}_{\mathtt{deriv}}}=\left\{{\mathcal{T}}_{aug}(s^{ aug}_{i})~{}|~{}i=1:N\right\}\]

## Appendix B Zero-shot transfer results for different transformations

Table 5 in the appendix is the extended version of Table 3 in the main paper with a breakdown for all languages. It reports \(\Delta_{\text{(BZ$-BS)}}\), \(\Delta_{\text{(MZ$-BS)}}\), and \(BZ\) for different languages and transformations considered.





\begin{table}
\begin{tabular}{c c r r r r r} \hline \hline \multirow{2}{*}{**Transliteration**} & \multicolumn{3}{c}{\(\Delta_{\text{(BZ-BS)}}~{}(\uparrow)\)} \\  & \multicolumn{1}{l}{**Variant**} & \multicolumn{1}{l}{**XNLI**} & \multicolumn{1}{l}{**NER**} & \multicolumn{1}{l}{**POS**} \\ \hline
**Parallel** & -2.1 & -2.3 & -0.5 & 90.0 \\
**Trans \(\circ\) Syntax** & -5.7 & -14.2 & -2 & 57.3 \\
**Non-parallel** & -3.8 & -4.1 & -0.7 & 43.0 \\
**Non-parallel** & -5.7 & -14.3 & -1.5 & 11.8 \\
**Non-parallel** & -19.2 & -51.5 & -42.7 & 0.16 \\
**Trans \(\circ\) Perm** & -27.7 & -46.3 & -59 & 0.01 \\ \hline \end{tabular}
\end{table}
Table 7: \(\Delta_{\text{(BZ-BS)}}\) and _alignment_ scores for different _Transliteration_ variants. The table contains raw scores for results in Section 4.4 of the main paper. Rows are sorted in descending order based on _alignment_. We observe strong correlations between alignment and zero-shot transfer, with \(\rho_{s}=0.94,p<..005\) on XNLI, \(\rho_{s}=0.93,p<.005\) on NER, and \(\rho_{s}=0.89,p<.01\) on POS.





\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Task**} & \multirow{2}{*}{**Language**} & \multicolumn{1}{l}{**XNLI**} & \multicolumn{1}{l}{**NER**} & \multicolumn{1}{l}{**POS**} & \multicolumn{1}{l}{**XQuAD**} \\ \cline{3-6}  & & \(\Delta_{\rm(BZ-BS)}\) & \(\Delta_{\rm(BZ-BS)}\) & \(\Delta_{\rm(BZ-BS)}\) & \(\Delta_{\rm(BZ-BS)}\) & \(\Delta_{\rm(BZ-BS)}\) \\ \hline \multirow{4}{*}{**Parallel**} & **English** & -1.7 & -2.1 & -0.5 & -4 \\  & **French** & -1.6 & -3.1 & -0.2 & -1 \\  & **Hindi** & -0.1 & -0.5 & -0.1 & 10.6 \\  & **Arabic** & -0.4 & -1.9 & -0.8 & -0.5 \\ \cline{2-6}  & **Avg.** & **-1.0** & **-1.9** & **-0.4** & **1.3** \\ \hline \multirow{4}{*}{**Non-parallel (Same)**} & **English** & -3.8 & -4.1 & -0.7 & -6.9 \\  & **French** & -1 & -6.3 & -0.5 & -0.9 \\  & **Hindi** & -0.4 & -3.1 & -0.2 & 4.5 \\  & **Arabic** & -2 & -6.1 & -1.5 & 0.7 \\ \cline{2-6}  & **Avg.** & **-1.8** & **-4.9** & **-0.7** & **-0.6** \\ \hline \multirow{4}{*}{**Non-parallel (Diff)**} & **English** & -5.7 & -14.3 & -1.5 & -9.3 \\  & **French** & -10.9 & -30.3 & -10.5 & -5.2 \\ \cline{1-1}  & **Hindi** & -0.5 & -8.6 & -1 & 5 \\ \cline{1-1}  & **Arabic** & -6.3 & -34.7 & -3.7 & -1.9 \\ \cline{1-1}  & **Avg.** & **-5.9** & **-22.0** & **-4.2** & **-2.9** \\ \hline \hline \end{tabular}
\end{table}
Table 8: \(|\Delta_{\rm(BZ-BS)}|\) for \(\mathcal{T}_{\text{trans}}\) under different conditions on the source of _original_ and _derived_ language pretraining corpora (§ 4.3). Larger values imply worse zero-shot transfer. For all languages: (1) _Non-parallel (diff)_, which uses corpora from different domains is worse than (2) _Non-parallel (same)_, which uses different sets of sentences sampled from the same domain, which is in turn worse than (3) _Parallel_, which uses the same sentences.

Figure 5: Alignment v.s. \(\Delta_{\rm(BZ-BS)}\) plots for XNLI, NER, and POS. We observe strong correlations between alignment and zero-shot transfer, with \(\rho_{s}=0.94,p<.005\) on XNLI, \(\rho_{s}=0.93,p<.005\) on NER, and \(\rho_{s}=0.89,p<.01\) on POS.