SlowGuess/NAACL_2022_OCR huggingface dataset:
Ameet Deshpande
Department of Computer Science Princeton University, USA
asd@cs.princeton.edu
Google Research India
partha@google.com
karthikn@cs.princeton.edu
While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer, there is a lack of consensus in the community as to what shared properties between languages enable transfer on downstream tasks. Analyses involving pairs of natural languages are often inconclusive and contradictory since languages simultaneously differ in many linguistic aspects. In this paper, we perform a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four diverse natural languages and their counterparts constructed by modifying aspects such as the script, word order, and syntax. Among other things, our experiments show that the absence of sub-word overlap significantly affects zero-shot transfer when languages differ in their word order, and there is a strong correlation between transfer performance and word embedding alignment between languages (e.g.,  $\rho_{s} = 0.94$  on the task of NLI). Our results call for focus in multilingual models on explicitly improving word embedding alignment between languages rather than relying on its implicit emergence.
Multilingual language models like XLM (Connieu et al., 2020a) and Multilingual-BERT (Devlin, 2019) are trained with masked-language modeling (MLM) objective on a combination of raw text from multiple languages. Surprisingly, these models exhibit decent cross-lingual zero-shot transfer, where fine-tuning on a task in a source language translates to good performance for a different language (target).
Requirements for zero-shot transfer Recent studies have provided inconsistent explanations for properties required for zero-shot transfer (hereon,
transfer). For example, while Wu and Dredze (2019) conclude that sub-word overlap is vital for transfer, K et al. (2020) demonstrate that it is not crucial, although they consider only English as the source language. While Pires et al. (2019) suggest that typological similarity (e.g., similar SVO order) is essential for transfer, other works (Kakwani et al., 2020; Conneau et al., 2020a) successfully build multilingual models for dissimilar languages.
Need for systematic analysis A major cause of these discrepancies is a large number of varying properties (e.g., syntax, script, and vocabulary size) between languages, which make isolating crucial ingredients for transfer difficult. Some studies alleviate this issue by creating synthetic languages which differ from natural ones only in specific linguistic properties like script (K et al., 2020; Dufter and Schütze, 2020). However, their focus is only on English as a source language, and the scale of their experiments is small (in number of tasks or pre-training corpora size), thus limiting the scope of their findings to their settings alone.
Our approach We perform a systematic study of cross-lingual transfer on bilingual language models trained on a natural language and a systematically derived counterpart. We choose four diverse natural languages (English, French, Arabic, and Hindi) and create derived variants using four different transformations on structural properties such as inverting or permuting word order, altering scripts, or varying syntax (Section 3.2). We train models on each of the resulting sixteen language pairs, and evaluate zero-shot transfer on four downstream tasks – natural language inference (NLI), namedentity recognition (NER), part-of-speech tagging (POS), and question-answering (QA).
Our experiments reveal the following:
1. Contrary to previous belief, the absence of subword overlap degrades transfer when languages
$^{1}$ Code is available at https://github.com/princeton-nlp/MultilingualAnalysis
Human Language Technologies, pages 3610 - 3623
differ in their word order (e.g., by more than 40 F1 points on POS tagging, (\$ 4.1)).
2. There is a strong correlation between token embedding alignment and zero-shot transfer across different tasks (e.g.,  $\rho_{s} = 0.94, p < .005$  for XNLI, Fig 4).
3. Using pre-training corpora from similar sources for different languages (e.g., Wikipedia) boosts transfer when compared to corpora from different sources (e.g., 17 F1 points on NER, Fig 3).
To our knowledge, we are the first study to quantitatively show that zero-shot transfer between languages is strongly correlated with token embedding alignment ( $\rho_{s} = 0.94$  for NLI). We also show that the current multilingual pre-training methods (Doddapaneni et al., 2021) fall short of aligning embeddings even between simple natural and derived language pairs, leading to failure in zero-shot transfer. Our results call for training objectives that explicitly improve alignment using either supervised (e.g., parallel corpora or bilingual dictionaries) or unsupervised data.
Multilingual pre-training for Transformers
The success of monolingual Transformer language models (Devlin et al., 2019; Radford et al., 2018) has driven studies that learn a multilingual language-model (LM) on several languages. Multilingual-BERT (M-BERT) (Devlin, 2019) is a single neural network pre-trained using the masked language-modeling (MLM) objective on a corpus of text from 104 languages. XLM (Conneau and Lample, 2019) introduced translation language-modeling, which performs MLM on pairs of parallel sentences, thus encouraging alignment between their representations. These models exhibit surprising zero-shot cross-lingual transfer performance (Conneau and Lample, 2019; K et al., 2020), a setup where the model is fine-tuned on a source language and evaluated on a different target language.
Analysis of cross-lingual transfer While Pires et al. (2019), Conneau et al. (2020b), and K et al. (2020) showed that transfer works even without a shared vocabulary between languages, Wu and Dredze (2019) discovered a correlation between sub-word overlap and zero-shot performance. Conneau et al. (2020b) and Artetxe et al. (2020a) showed that shared parameters for languages with
different scripts were crucial for transfer. Pires et al. (2019) and (Wu and Dredze, 2019) observed that transfer for NER and POS tagging works better between typologically similar languages. However, a study conducted by Lin et al. (2019) showed that there is no simple rule of thumb to gauge when transfer works between languages. Hsu et al. (2019) observed that changing the syntax (SOV) order of the source to match that of the target does not improve performance.
Transfer between real and synthetic Languages K et al. (2020) create a synthetic language by changing English's script and find that transfer between it and Spanish works even without common sub-words. However, they use only English as their source language, test only on two tasks, and use a single natural-synthetic language pair. Dufter and Schütze (2020) study transfer between English and synthetic English obtained by changing the script, word order, or model delimiters. However, they use a small corpus (228K words) compared to current standards (we use 3 orders more) and measure only embedding similarity and not zero-shot transfer. A contemporary work (Wu et al., 2022) uses synthetic transformations to modify the GLUE dataset (Wang et al., 2018) and analyze properties required for good zero-shot transfer, but they perform their experiments only on English and do not perform token embedding alignment analysis. We show that the latter is crucial for good transfer.
We first provide some background on bilingual language models (Section 3.1), followed by descriptions of our transformations (Section 3.2), and our training and evaluation setup (Section 3.3).
Bilingual pre-training The standard setup (Connéau and Lample, 2019) trains a bilingual language model  $(Bi-LM)$  on raw text corpora from two languages simultaneously.  $Bi-LM$  uses the masked language-modeling loss  $(\mathcal{L}_{\mathrm{MLM}})$  on the corpora from the two languages  $(\mathcal{C}_1,\mathcal{C}_2)$  separately with no explicit cross-lingual signal:
$$
\mathcal {L} _ {\mathrm {B i - L M}} ^ {\theta} (\mathcal {C} _ {1} + \mathcal {C} _ {2}) = \mathcal {L} _ {\mathrm {M L M}} ^ {\theta} (\mathcal {C} _ {1}) + \mathcal {L} _ {\mathrm {M L M}} ^ {\theta} (\mathcal {C} _ {2})
$$
A shared byte pair encoding tokenizer (Sennrich et al., 2015) is trained on  $\mathcal{C}_1 + \mathcal{C}_2$ . A single batch contains instances from both languages, but each instance belongs to a single language.
Transformation	Instance (s)	Transformed instance (T(s))
Inversion (Tinv)	Welcome to NAACL at Seattle	Seattle at NAACL to Welcome
Permutation (TPerm)	This is a conference	a This conference is
Transliteration (Ttrans)	I am Sam . I am	♣(I)♥(am)◇(Sam)♣(.)♣(I)♥(am)
Syntax (TSyn)	Sara (S) ate (V) apples (O)	Sara (S) apples (O) ate (V)
Une table (N) rode (A)	Une rode (A) table (N)
Table 1: Examples of our transformations applied to different sentences (without sub-word tokenization). Inversion inverts the tokens, Permutation samples a random reordering, and Transliteration changes the script. We use symbols  $(\clubsuit)$  to denote words in the new script and mention the corresponding original word in brackets. Syntax stochastically modifies the syntactic structure. In the first example for Syntax, the sentence in Subject-Verb-Object (SVO) order gets transformed to SOV order, and in the second, the sentence in Noun-Adjective (NA) order gets transformed to the AN order. The examples are high probability re-orderings and other ones might be sampled too.
Zero-shot transfer evaluation Consider a bilingual model  $(Bi - LM)$  pre-trained on two languages, source and target. Zero-shot transfer involves finetuning  $Bi - LM$  on downstream task data from source and evaluating on test data from target. This is considered zero-shot because  $Bi - LM$  is not fine-tuned on any data belonging to target.
3.2 Generating language variants with systematic transformations
Natural languages typically differ in several ways, like the script, word order, and syntax. To isolate the affect of these properties on zero-shot transfer, we obtain derived language corpora (hereon, derived corpora) from original (natural) language corpora by performing sentence level transformations  $(\mathcal{T})$  which change particular properties. For example, an "inversion" transformation could be used to invert each sentence in the corpus ( $Welcome_1$  to  $2$  NAACL3  $\Rightarrow$  NAACL3 to  $2$  Welcome1). Since the transformation  $(\mathcal{T})$  is applied on each sentence of the original corpus, the size of the original and the derived corpus is the same. In the following sections, we will use the following notation:
$$
\begin{array}{l} \mathcal {C} _ {\text {o r i g}} \equiv \text {O r i g i n a l c o r p u s} \\ = \left\{s _ {i} \mid i = 1: N, s _ {i} = \text {s e n t e n c e} \right\} \\ \end{array}
$$
$$
\mathcal {T} \equiv \text {S e n t e n c e - l e v e l t r a n s f o r m a t i o n}
$$
$$
\begin{array}{l} \mathcal {C} _ {\mathrm {d e r i v}} \equiv \text {D e r i v e d c o r p u s} \\ = \left\{\mathcal {T} (\text {s e n t}) \mid \forall \text {s e n t} \in \mathcal {C} _ {\text {o r i g}} \right\} \\ \end{array}
$$
Types of transformations We consider four transformations which modify different aspects of sentences (examples in Table 1):
1. Inversion  $(\mathcal{T}_{\mathrm{inv}})$ : Invert the order of tokens in the sentence, like in Dufter and Schütze
(2020). The first token becomes the last, and vice versa.
2. Permutation  $(\mathcal{T}_{\mathrm{perm}})$ : Permute the order of tokens in a sentence uniformly at random. For a sentence of  $n$  tokens, we sample a random ordering with probability  $\frac{1}{n!}$ .
3. Transliteration  $(\mathcal{T}_{\mathrm{trans}})$ : Change the script of all tokens other than the special tokens (like [CLS]). This creates a derived vocabulary  $(\mathcal{V}_{\text{deriv}})$  with a one-to-one correspondence with the original vocabulary  $(\mathcal{V}_{\text{orig}})$ .
4. Syntax  $(\mathcal{T}_{\mathrm{syn}})$ : Modify a sentence to match the syntactic properties of a different natural language by re-ordering the dependents of nouns and verbs in the dependency parse. These transformations are stochastic because of the errors in parsing and sampling over possible re-orderings (Wang and Eisner, 2016).
These transformations allow us to systematically evaluate the effect of corresponding properties on zero-shot transfer. We also consider composed transformations (§4.2) which consecutively apply two transformations. We note that while real languages typically differ in more than one or two properties considered in our transformations, our methodology remains useful in isolating crucial properties that enable good transfer and can be extended to more transformations.
Transformations for downstream tasks We obtain the downstream corpus in the derived language  $(\mathcal{D}_{\mathrm{deriv}})$  by applying the same transformation  $(\mathcal{T})$  used during pre-training on the original downstream corpus  $(\mathcal{D}_{\mathrm{orig}})$ . Unlike pre-training corpora which contain raw sentences, instances in downstream tasks contain one or more sentences with annotated labels. For text classification tasks like
(b) Fine-tuning
Figure 1: (a) During pre-training, we obtain the derived language corpus  $(\mathcal{C}_{\mathrm{deriv}})$  by transforming the original language corpus  $(\mathcal{C}_{\mathrm{orig}})$ . The two corpora are combined and a bilingual model  $(Bi-LM)$  is learned using the MLM objective. (b) During fine-tuning, we obtain the derived dev dataset  $(\mathcal{D}_{\mathrm{deriv}}^{\mathrm{dev}})$  by transforming the original dev dataset  $(\mathcal{D}_{\mathrm{orig}}^{\mathrm{dev}})$ .  $Bi-LM$  is fine-tuned on the original train dataset  $(\mathcal{D}_{\mathrm{orig}}^{\mathrm{train}})$ , and evaluated on  $\mathcal{D}_{\mathrm{deriv}}^{\mathrm{dev}}$ , which is the standard zero-shot cross lingual setup.
Evaluation	Corpus source
Pre-train	Fine-tune (train)	Fine-tune (dev)
BZ	Corig + Cderiv	Dorig	Dderiv
BS	Corig + Cderiv	Dderiv	Dderiv
MZ	Corig	Dorig	Dderiv
Δ(BZ-BS) = (BZ - BS)
Δ(MZ-BS) = (MZ - BS)
Table 2: Summary of evaluation metrics defined in § 3.3.  $\mathcal{C}$  and  $\mathcal{D}$  denote the pre-training and downstream corpus respectively, and their subscript indicates their source (original or derived).  $\mathbf{BZ}$  and  $\mathbf{MZ}$  represent bilingual and monolingual zero-shot transfer scores, and BS is the supervised learning baseline on derived. The differences in the setting of  $\mathbf{BZ}$  and other scores are typeset in blue. We use  $\Delta_{(\mathbf{BZ} - \mathbf{BS})}$  and  $\Delta_{(\mathbf{MZ} - \mathbf{BS})}$  (defined in the last two rows) throughout our paper.
NLI, we apply the transformation on each sentence in every dataset instance. For token classification tasks (e.g., NER, POS), any transformation which changes the order of the tokens also changes the order of the labels. We present the mathematical specification in Appendix A.
3.3 Model Training and Evaluation
We now describe our pre-training and zero-shot transfer evaluation setup. Figure 1 provides an overview of pre-training and fine-tuning, and Table 2 summarizes the evaluation metrics we use.
Pre-training Let  $\mathcal{C}_{\mathrm{orig}}$  and  $\mathcal{C}_{\mathrm{deriv}}$  be the original and derived language pre-training corpora. We train two models for each original-derived pair:
1. Bilingual Model  $(Bi - LM)$ : A bilingual model pre-trained on the combined corpus  $(\mathcal{C}_{\mathrm{orig}} +$
$\mathcal{C}_{\mathrm{deriv}}$  (Figure 1a).
2. Monolingual Model (Mono-LM): A monolingual model trained only on  $\mathcal{C}_{\mathrm{orig}}$  for the same number of steps as Bi-LM's. MonoLM is used as a baseline to measure zero-shot transfer of a model not pre-trained on derived.
Evaluation Let  $\mathcal{D}_{\mathrm{orig}}^{\mathrm{train}}$  and  $\mathcal{D}_{\mathrm{orig}}^{\mathrm{dev}}$  be the original language training and development sets for a downstream task, and  $\mathcal{D}_{\mathrm{deriv}}^{\mathrm{train}}$  and  $\mathcal{D}_{\mathrm{deriv}}^{\mathrm{dev}}$  be the corresponding derived language datasets. For evaluation, we first fine-tune the pre-trained models on a downstream training set and evaluate the resulting model on a development set (Figure 1b). Since our goal is to investigate the extent of zero-shot transfer, we require appropriate lower and upper bounds to make informed conclusions. To this end, we compute three metrics, all on the same development set (summarized in Table 2):
- Bilingual zero-shot transfer  $(BZ)$ : This is the standard zero-shot transfer score (Conneau and Lample, 2019) which measures how well a bilingual model fine-tuned on  $\mathcal{D}_{\mathrm{orig}}^{\mathrm{train}}$  zero-shot transfers to the other language ( $\mathcal{D}_{\mathrm{deriv}}^{\mathrm{dev}}$ ).
- Bilingual supervised synthetic (BS): This is the supervised learning performance on the derived language obtained by fine-tuning BiLM on  $\mathcal{D}_{\mathrm{deriv}}^{\mathrm{train}}$  and evaluating it on  $\mathcal{D}_{\mathrm{deriv}}^{\mathrm{dev}}$ .
- Monolingual zero-shot transfer (MZ): This measures the zero-shot performance of the baseline Mono-LM, which is not pre-trained on the derived language, by fine-tuning Mono-LM on  $\mathcal{D}_{\mathrm{orig}}^{\mathrm{train}}$  and evaluating it on  $\mathcal{D}_{\mathrm{deriv}}^{\mathrm{dev}}$ .
$BS$  uses fine-tuning train data from the derived language and serves as an upper-bound on  $BZ$  and  $MZ$
which don't use it.  $MZ$  doesn't pre-train on the derived language and serves as a lower-bound on  $BZ$  which does pre-train on it. For easier comparison of  $BZ$  and  $MZ$  with  $BS$  (upper-bound), we report the following score differences (Table 2), which are both negative in our experiments.
$$
\Delta_ {\left(\mathrm {B Z} - \mathrm {B S}\right)} = (\mathrm {B Z} - \mathrm {B S}) \tag {1}
$$
$$
\Delta_ {\left(\mathrm {M Z} - \mathrm {B S}\right)} = (\mathrm {M Z} - \mathrm {B S}) \tag {2}
$$
$BZ$  alone cannot capture the quality of the zero-shot transfer. A large and negative  $\Delta_{\mathrm{(BZ - BS)}}$  implies that bilingual zero-shot transfer is much worse than supervised fine-tuning on derived. Concurrently,  $\Delta_{\mathrm{(BZ - BS)}}\approx \Delta_{\mathrm{(MZ - BS)}}$  implies that  $Bi - LM$  transfers as poorly as Mono-LM. Thus, good zero-shot transfer is characterized by  $\Delta_{\mathrm{(BZ - BS)}}\approx 0$  and  $\Delta_{\mathrm{(BZ - BS)}}\gg \Delta_{\mathrm{(MZ - BS)}}$
3.4 Experimental Setup
Languages We choose four diverse natural languages: English (Indo-European, Germanic), French (Indo-European, Romance), Hindi (Indo-European, Indo-Iranian), and Arabic (Afro-Asiatic, Semitic), which are represented in the multilingual XTREME benchmark (Hu et al., 2020). For each language, we consider four transformations (Section 3.2) to create derived counterparts, giving us 16 different original-derived pairs in total. For the Syntax transformation, we use Qi et al. (2020) for parsing. We modify the syntax of FR, HI, and AR to that of EN, and the syntax of EN to that of FR.
Datasets For the pre-training corpus  $(\mathcal{C}_{\mathrm{orig}})$ , we use a 500MB (uncompressed) subset of Wikipedia  $(\approx 100\mathrm{M}$  tokens) for each language. This matches the size of WikiText-103 (Merit et al., 2016), a standard language-modeling dataset. For downstream evaluation, we choose four tasks from the XTREME benchmark (Hu et al., 2020). Table 4 lists all the datasets and their evaluation metrics.
Implementation Details We use a variant of RoBERTa (Liu et al., 2019) which has 8 layers, 8 heads, and a hidden dimensionality of 512. We train each model on 500K steps, a batch size of 128, and a learning rate of  $1e - 4$  with a linear warmup of 10K steps. We use an original language vocabulary size of 40000 for all the models and train on 8 Cloud TPU v3 cores for 32-48 hours. For fine-tuning, we use standard hyperparameters (Appendix F) from the XTREME benchmark and report our scores on the development sets.
Our experiments reveal several interesting findings for bilingual models including the situational importance of sub-word overlap for zero-shot transfer (§ 4.1, 4.2), the effect of domain mismatch between languages (§ 4.3), and correlation of zero-shot performance with embedding alignment (§ 4.4). We connect our findings to zero-shot transfer results between natural languages in Section 4.5.
4.1 Sub-word overlap is not strictly necessary for strong zero-shot transfer
Sub-word overlap is the number of common tokens between two different language corpora. If  $\mathcal{E}_1$  and  $\mathcal{E}_2$  are sets of tokens which appear in the two corpus, then: Sub-word overlap  $= |\mathcal{E}_1\cap \mathcal{E}_2| / |\mathcal{E}_1\cup \mathcal{E}_2|$  (Pires et al., 2019). The Transliteration transformation  $(\mathcal{T}_{\mathrm{trans}})$  creates original-derived language pairs that have  $0\%$  sub-word overlap (equivalently, different scripts), but follow the same word order.
Table 3 displays  $\Delta_{\mathrm{(BZ - BS)}}$  scores for  $\mathcal{T}_{\mathrm{trans}}$ , averaged over four languages (Appendix B contains a breakdown). We observe that  $\Delta_{\mathrm{(BZ - BS)}} \approx 0$  for all tasks while  $\Delta_{\mathrm{(MZ - BS)}}$  is highly negative, implying that zero-shot transfer is strong and on par with supervised learning. This result indicates that zero-shot transfer is possible even when languages with different scripts have similar word orders (in line with K et al. (2020)). However, it is unrealistic for natural languages to differ only in their script and not other properties (e.g., word order).
4.2 Absence of sub-word overlap significantly hurts zero-shot performance when languages differ in their word-orders
To simulate a more realistic scenario, we create original and derived language pairs which differ both in their scripts (0% sub-word overlap) and in word order. We achieve this by composing two transformations on the original language corpus, one of which is Transliteration ( $\mathcal{T}_{\mathrm{trans}}$ ). We experiment with three different compositions, (a)  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{inv}}$ , (b)  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{perm}}$ , and (c)  $\mathcal{T}_{\mathrm{trans}} \circ \mathcal{T}_{\mathrm{syn}}$ . Here,  $\alpha \circ \beta$  means that transformation  $\beta$  is applied before  $\alpha$ . A composed transformation ( $\mathcal{T}_{\mathrm{trans}} \circ$
$^4$ XQuAD is a question-answering task where the correct answer is a contiguous span. We do not report scores on XQuAD for  $\mathcal{T}_{\mathrm{perm}}$  and  $\mathcal{T}_{\mathrm{syn}}$  because they can potentially reorder individual words in the contiguous answer, thus distributing them throughout the transformed sentence and making the question unanswerable. On the other hand,  $\mathcal{T}_{\mathrm{inv}}$  and  $\mathcal{T}_{\mathrm{trans}}$  do not have this issue because they maintain the spans.
Task	Inversion (Tinv)	Permutation (Tperm)	Syntax (Tsyn)	Transliteration (Ttrans)
Δ(BZ-BS)	Δ(MZ-BS)	BZ	Δ(BZ-BS)	Δ(MZ-BS)	BZ	Δ(BZ-BS)	Δ(MZ-BS)	BZ	Δ(BZ-BS)	Δ(MZ-BS)	BZ
XNLI	-10.2	-13.0	58.4	-3.6	-8.6	62.6	-0.9*	-1.1	67.8	-1.0*	-36.7	69.3
NER	-49.1	-46.7	37.9	-26.3	-35.4	47.3	-14.6	-16.6	62.9	-1.9*	-82.6	83.7
POS	-30.2	-36.2	64.2	-11.2	-25.2	73.6	-4.4	-7.6	89.4	-0.4*	-95.0	95.4
XQuAD4	-32.8	-31.0	22.8	-4	-	-	-4	-	-	0.0*	-55.9	61.2