SlowGuess/ACL_2020_OCR huggingface dataset:
Mikel Artetxe\*, Sebastian Ruder‡, Dani Yogatama‡  
†HiTZ Center, University of the Basque Country (UPV/EHU)  
‡DeepMind  
mikel.artexe@ehu.eus  
{ruder, dyogatama}@google.com
State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective—freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.
Multilingual pre-training methods such as multilingual BERT (mBERT, Devlin et al., 2019) have been successfully used for zero-shot cross-lingual transfer (Pires et al., 2019; Conneau and Lample, 2019). These methods work by jointly training a
transformer model (Vaswani et al., 2017) to perform masked language modeling (MLM) in multiple languages, which is then fine-tuned on a downstream task using labeled data in a single language—typically English. As a result of the multilingual pre-training, the model is able to generalize to other languages, even if it has never seen labeled data in those languages. Such a cross-lingual generalization ability is surprising, as there is no explicit cross-lingual term in the underlying training objective. In relation to this, Pires et al. (2019) hypothesized that:
... having word pieces used in all languages (numbers, URLs, etc), which have to be mapped to a shared space forces the co-occurring pieces to also be mapped to a shared space, thus spreading the effect to other word pieces, until different languages are close to a shared space.
...mBERT's ability to generalize cannot be attributed solely to vocabulary memorization, and that it must be learning a deeper multilingual representation.
Cao et al. (2020) echoed this sentiment, and Wu and Dredze (2019) further observed that mBERT performs better in languages that share many subwords. As such, the current consensus of the cross-lingual generalization ability of mBERT is based on a combination of three factors: (i) shared vocabulary items that act as anchor points; (ii) joint training across multiple languages that spreads this effect; which ultimately yields (iii) deep cross-lingual representations that generalize across languages and tasks.
In this paper, we empirically test this hypothesis by designing an alternative approach that violates all of these assumptions. As illustrated in Figure 1, our method starts with a monolingual transformer trained with MLM, which we transfer to a new language by learning a new embedding matrix through MLM in the new language while freezing parameters of all other layers. This approach only learns new lexical parameters and does not rely on shared
*Work done as an intern at DeepMind.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(a) English pre-training
(b)  $L_{2}$  embedding learning
(c) English fine-tuning
Figure 1: Four steps for zero-shot cross-lingual transfer: (i) pre-train a monolingual transformer model in English akin to BERT; (ii) freeze the transformer body and learn new token embeddings from scratch for a second language using the same training objective over its monolingual corpus; (iii) fine-tune the model on English while keeping the embeddings frozen; and (iv) zero-shot transfer it to the new language by swapping the token embeddings.
(d) Zero-shot transfer to  $L_{2}$
vocabulary items nor joint learning. However, we show that it is competitive with joint multilingual pre-training across standard zero-shot cross-lingual transfer benchmarks (XNLI, MLDoc, and PAWS-X).
We also experiment with a new Cross-lingual Question Answering Dataset (XQuAD), which consists of 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 (Rajpurkar et al., 2016) translated into ten languages by professional translators. Question answering as a task is a classic probe for language understanding. It has also been found to be less susceptible to annotation artifacts commonly found in other benchmarks (Kaushik and Lipton, 2018; Gururangan et al., 2018). We believe that XQuAD can serve as a more comprehensive cross-lingual benchmark and make it publicly available at https://github.com/deepmind/xquad. Our results on XQuAD show that the monolingual transfer approach can be made competitive with mBERT by learning second language-specific transformations via adapter modules (Rebuffi et al., 2017).
Our contributions in this paper are as follows: (i) we propose a method to transfer monolingual representations to new languages in an unsupervised fashion  $(\S 2)^{1}$ ; (ii) we show that neither a shared subword vocabulary nor joint multilingual training is necessary for zero-shot transfer and find that the effective vocabulary size per language is an important factor for learning multilingual models ( $\S 3$  and  $\S 4$ ); (iii) we show that monolingual models learn abstractions that generalize across languages ( $\S 5$ ); and (iv) we present a new cross-lingual question answering dataset ( $\S 4$ ).
2 Cross-lingual Transfer of Monolingual Representations
In this section, we propose an approach to transfer a pre-trained monolingual model in one language  $L_{1}$  (for which both task supervision and a monolingual corpus are available) to a second language  $L_{2}$  (for which only a monolingual corpus is available). The method serves as a counterpoint to existing joint multilingual models, as it works by aligning new lexical parameters to a monolingually trained deep model.
As illustrated in Figure 1, our proposed method consists of four steps:
1. Pre-train a monolingual BERT (i.e. a transformer) in  $L_{1}$  with masked language modeling (MLM) and next sentence prediction (NSP) objectives on an unlabeled  $L_{1}$  corpus.
2. Transfer the model to a new language by learning new token embeddings while freezing the transformer body with the same training objectives (MLM and NSP) on an unlabeled  $L_{2}$  corpus.
3. Fine-tune the transformer for a downstream task using labeled data in  $L_{1}$ , while keeping the  $L_{1}$  token embeddings frozen.
4. Zero-shot transfer the resulting model to  $L_{2}$  by swapping the  $L_{1}$  token embeddings with the  $L_{2}$  embeddings learned in Step 2.
We note that, unlike mBERT, we use a separate subword vocabulary for each language, which is trained on its respective monolingual corpus, so the model has no notion of shared subwords. However, the special [CLS], [SEP], [MASK],
This is particularly useful for low-resource languages, since many pre-trained models are currently in English.
[PAD], and [UNK] symbols are shared across languages, and fine-tuned in Step 3. We observe further improvements on several downstream tasks using the following extensions to the above method.
Language-specific position embeddings. The basic approach does not take into account different word orders commonly found in different languages, as it reuses the position embeddings in  $L_{1}$  for  $L_{2}$ . We relax this restriction by learning a separate set of position embeddings for  $L_{2}$  in Step 2 (along with  $L_{2}$  token embeddings).<sup>3</sup> We treat the [CLS] symbol as a special case. In the original implementation, BERT treats [CLS] as a regular word with its own position and segment embeddings, even if it always appears in the first position. However, this does not provide any extra capacity to the model, as the same position and segment embeddings are always added up to the [CLS] embedding. Following this observation, we do not use any position and segment embeddings for the [CLS] symbol.
Noised fine-tuning. The transformer body in our proposed method is only trained with  $L_{1}$  embeddings as its input layer, but is used with  $L_{2}$  embeddings at test time. To make the model more robust to this mismatch, we add Gaussian noises sampled from the standard normal distribution to the word, position, and segment embeddings during the finetuning step (Step 3).
Adapters. We also investigate the possibility of allowing the model to learn better deep representations of  $L_{2}$ , while retaining the alignment with  $L_{1}$  using residual adapters (Rebuffi et al., 2017). Adapters are small task-specific bottleneck layers that are added between layers of a pre-trained model. During fine-tuning, the original model parameters are frozen, and only parameters of the adapter modules are learned. In Step 2, when we transfer the  $L_{1}$  transformer to  $L_{2}$ , we add a feedforward adapter module after the projection following multi-headed attention and after the two feedforward layers in each transformer layer, similar to Houlsby et al. (2019). Note that the original transformer body is still frozen, and only parameters of
the adapter modules are trainable (in addition to the embedding matrix in  $L_{2}$ ).
Our goal is to evaluate the performance of different multilingual models in the zero-shot cross-lingual setting to better understand the source of their generalization ability. We describe the models that we compare (§3.1), the experimental setting (§3.2), and the results on three classification datasets: XNLI (§3.3), MLDoc (§3.4) and PAWS-X (§3.5). We discuss experiments on our new XQuAD dataset in §4. In all experiments, we fine-tune a pre-trained model using labeled training examples in English, and evaluate on test examples in other languages via zero-shot transfer.
We compare four main models in our experiments:
Joint multilingual models (JOINTMULTI). A multilingual BERT model trained jointly on 15 languages<sup>4</sup>. This model is analogous to mBERT and closely related to other variants like XLM.
Joint pairwise bilingual models (JOINTPAIR). A multilingual BERT model trained jointly on two languages (English and another language). This serves to control the effect of having multiple languages in joint training. At the same time, it provides a joint system that is directly comparable to the monolingual transfer approach in §2, which also operates on two languages.
Cross-lingual word embedding mappings (CLWE). The method we described in §2 operates at the lexical level, and can be seen as a form of learning cross-lingual word embeddings that are aligned to a monolingual transformer body. In contrast to this approach, standard cross-lingual word embedding mappings first align monolingual lexical spaces and then learn a multilingual deep model on top of this space. We also include a method based on this alternative approach where we train skip-gram embeddings for each language, and map them to a shared space using VecMap (Artetxe et al., 2018). We then train an English BERT model using MLM and NSP on top of the frozen mapped embeddings. The model is
4We use all languages that are included in XNLI (Conneau et al., 2018b).
5We use the orthogonal mode in VecMap and map all languages into English.
2The rationale behind this is that special symbols are generally task dependent, and given that the fine-tuning in downstream tasks is done exclusively in English, we need to share these symbols to zero-shot transfer to other languages.
3We also freeze the  $L_{1}$  position embeddings in Step 3 accordingly, and the  $L_{2}$  position embeddings are plugged in together with the token embeddings in Step 4.
then fine-tuned using English labeled data while keeping the embeddings frozen. We zero-shot transfer to a new language by plugging in its respective mapped embeddings.
Cross-lingual transfer of monolingual models (MONOTRANS). Our method described in §2. We use English as  $L_{1}$  and try multiple variants with different extensions.
3.2 Setting
Vocabulary. We perform subword tokenization using the unigram model in SentencePiece (Kudo and Richardson, 2018). In order to understand the effect of sharing subwords across languages and the size of the vocabulary, we train each model with various settings. We train 4 different JOINTMULTI models with a vocabulary of  $32\mathrm{k}$ ,  $64\mathrm{k}$ ,  $100\mathrm{k}$ , and  $200\mathrm{k}$  subwords. For JOINTPAIR, we train one model with a joint vocabulary of  $32\mathrm{k}$  subwords, learned separately for each language pair, and another one with a disjoint vocabulary of  $32\mathrm{k}$  subwords per language, learned on its respective monolingual corpus. The latter is directly comparable to MONOTRANS in terms of vocabulary, in that it is restricted to two languages and uses the exact same disjoint vocabulary with  $32\mathrm{k}$  subwords per language. For CLWE, we use the same subword vocabulary and investigate two choices: (i) the number of embedding dimensions—300d (the standard in the crosslingual embedding literature) and 768d (equivalent to the rest of the models); and (ii) the self-learning initialization—weakly supervised (based on identically spelled words, Søgaard et al., 2018) and unsupervised (based on the intralingual similarity distribution, Artetxe et al., 2018).
Pre-training data. We use Wikipedia as our training corpus, similar to mBERT and XLM (Conneau and Lample, 2019), which we extract using the WikiExtractor tool. We do not perform any lowercasing or normalization. When working with languages of different corpus sizes, we use the same upsampling strategy as Conneau and Lample (2019) for both the subword vocabulary learning and the pre-training.
Training details. Our implementation is based on the BERT code from Devlin et al. (2019). For adapters, we build on the code by Houlsby et al. (2019). We use the model architecture of
BERT<sub>BASE</sub>, similar to mBERT. We use the LAMB optimizer (You et al., 2020) and train on 64 TPUv3 chips for 250,000 steps using the same hyperparameters as You et al. (2020). We describe other training details in Appendix A. Our hyperparameter configuration is based on preliminary experiments on the development set of the XNLI dataset. We do not perform any exhaustive hyperparameter search, and use the exact same settings for all model variants, languages, and tasks.
Evaluation setting. We perform a single training and evaluation run for each model, and report results in the corresponding test set for each downstream task. For MONOTRANS, we observe stability issues when learning language-specific position embeddings for Greek, Thai and Swahili. The second step would occasionally fail to converge to a good solution. For these three languages, we run Step 2 of our proposed method (§2) three times and pick the best model on the XNLI development set.
3.3 XNLI: Natural Language Inference
In natural language inference (NLI), given two sentences (a premise and a hypothesis), the goal is to decide whether there is an entailment, contradiction, or neutral relationship between them (Bowman et al., 2015). We train all models on the MultiNLI dataset (Williams et al., 2018) in English and evaluate on XNLI (Conneau et al., 2018b)—a cross-lingual NLI dataset consisting of 2,500 development and 5,000 test instances translated from English into 14 languages.
We report our results on XNLI in Table 1 together with the previous results from mBERT and XLM. We summarize our main findings below.
JOINTMULTI is comparable with the literature. Our best JOINTMULTI model is substantially better than mBERT, and only one point worse (on average) than the unsupervised XLM model, which is larger in size.
A larger vocabulary is beneficial. JOINTMULTI variants with a larger vocabulary perform better.
More languages do not improve performance. JOINTPAIR models with a joint vocabulary perform comparably with JOINTMULTI.
${}^{7}$  mBERT covers 102 languages and has a shared vocabulary of  ${110}\mathrm{k}$  subwords. XLM covers 15 languages and uses a larger model size with a shared vocabulary of  ${95}\mathrm{k}$  subwords, which contributes to its better performance.
$^{6}$ https://github.com/attardi/wikiextractor
en	fr	es	de	el	bg	ru	tr	ar	vi	th	zh	hi	sw	ur	avg
Prev work	mBERT	81.4	-	74.3	70.5	-	-	-	-	62.1	-	-	63.8	-	-	58.3	-
XLM (MLM)	83.2	76.5	76.3	74.2	73.1	74.0	73.1	67.8	68.5	71.2	69.2	71.9	65.7	64.6	63.4	71.5
CLWE	300d ident	82.1	67.6	69.0	65.0	60.9	59.1	59.5	51.2	55.3	46.6	54.0	58.5	48.4	35.3	43.0	57.0
300d unsup	82.1	67.4	69.3	64.5	60.2	58.4	59.2	51.5	56.2	36.4	54.7	57.7	48.2	36.2	33.8	55.7
768d ident	82.4	70.7	71.1	67.6	64.2	61.4	63.3	55.0	58.6	50.7	58.0	60.2	54.8	34.8	48.1	60.1
768d unsup	82.4	70.4	71.2	67.4	63.9	62.8	63.3	54.8	58.3	49.1	57.2	55.7	54.9	35.0	33.9	58.7
JOINT MULTI	32k voc	79.0	71.5	72.2	68.5	66.7	66.9	66.5	58.4	64.4	66.0	62.3	66.4	59.1	50.4	56.9	65.0
64k voc	80.7	72.8	73.0	69.8	69.6	69.5	68.8	63.6	66.1	67.2	64.7	66.7	63.2	52.0	59.0	67.1
100k voc	81.2	74.5	74.4	72.0	72.3	71.2	70.0	65.1	69.7	68.9	66.4	68.0	64.2	55.6	62.2	69.0
200k voc	82.2	75.8	75.7	73.4	74.0	73.1	71.8	67.3	69.8	69.8	67.7	67.8	65.8	60.9	62.3	70.5
JOINT PAIR	Joint voc	82.2	74.8	76.4	73.1	72.0	71.8	70.2	67.9	68.5	71.4	67.7	70.8	64.5	64.2	60.6	70.4
Disjoint voc	83.0	76.2	77.1	74.4	74.4	73.7	72.1	68.8	71.3	70.9	66.2	72.5	66.0	62.3	58.0	71.1
MONO TRANS	Token emb + pos emb + noising + adapters	83.1	73.3	73.9	71.0	70.3	71.5	66.7	64.5	66.6	68.2	63.9	66.9	61.3	58.1	57.3	67.8
83.8	74.3	75.1	71.7	72.6	72.8	68.8	66.0	68.6	69.8	65.7	69.7	61.1	58.8	58.3	69.1
81.7	74.1	75.2	72.6	72.9	73.1	70.2	68.1	70.2	69.1	67.7	70.6	62.5	62.5	60.2	70.0
81.7	74.7	75.4	73.0	72.0	73.7	70.4	69.9	70.6	69.5	65.1	70.3	65.2	59.6	51.7	69.5
Table 1: XNLI results (accuracy). mBERT results are taken from the official BERT repository, while XLM results are taken from Conneau and Lample (2019). We bold the best result in each section and underline the overall best.
A shared subword vocabulary is not necessary for joint multilingual pre-training. The equivalent JOINTPAIR models with a disjoint vocabulary for each language perform better.
CLWE performs poorly. Even if it is competitive in English, it does not transfer as well to other languages. Larger dimensionalities and weak supervision improve CLWE, but its performance is still below other models.
MONOTRANS is competitive with joint learning. The basic version of MONOTRANS is 3.3 points worse on average than its equivalent JOINTPAIR model. Language-specific position embeddings and noised fine-tuning reduce the gap to only 1.1 points. Adapters mostly improve performance, except for low-resource languages such as Urdu, Swahili, Thai, and Greek. In subsequent experiments, we include results for all variants of MONOTRANS and JOINTPAIR, the best CLWE variant (768d ident), and JOINTMULTI with  $32\mathrm{k}$  and  $200\mathrm{k}$  voc.
3.4 MLDoc: Document Classification
In MLDoc (Schwenk and Li, 2018), the task is to classify documents into one of four different genres: corporate/industrial, economics, government/social, and markets. The dataset is an improved version of the Reuters benchmark (Klementiev et al., 2012), and consists of 1,000 training and 4,000 test documents in 7 languages.
We show the results of our MLDoc experiments in Table 2. In this task, we observe that simpler
models tend to perform better, and the best overall results are from CLWE. We believe that this can be attributed to: (i) the superficial nature of the task itself, as a model can rely on a few keywords to identify the genre of an input document without requiring any high-level understanding and (ii) the small size of the training set. Nonetheless, all of the four model families obtain generally similar results, corroborating our previous findings that joint multilingual pre-training and a shared vocabulary are not needed to achieve good performance.
3.5 PAWS-X: Paraphrase Identification
PAWS is a dataset that contains pairs of sentences with a high lexical overlap (Zhang et al., 2019). The task is to predict whether each pair is a paraphrase or not. While the original dataset is only in English, PAWS-X (Yang et al., 2019) provides human translations into six languages.
We evaluate our models on this dataset and show our results in Table 2. Similar to experiments on other datasets, MONOTRANS is competitive with the best joint variant, with a difference of only 0.6 points when we learn language-specific position embeddings.
4 XQuAD: Cross-lingual Question Answering Dataset
Our classification experiments demonstrate that MONOTRANS is competitive with JOINTMULTI and JOINTPAIR, despite being multilingual at the embedding layer only (i.e. the transformer body is trained
MLDoc	PAWS-X
en	fr	es	de	ru	zh	avg	en	fr	es	de	zh	avg
Prev work	mBERT	-	83.0	75.0	82.4	71.6	66.2	-	93.5	85.2	86.0	82.2	75.8	84.5
CLWE	768d ident	94.7	87.3	77.0	88.7	67.6	78.3	82.3	92.8	85.2	85.5	81.6	72.5	83.5
JOINT	32k voc	92.6	81.7	75.8	85.4	71.5	66.6	78.9	91.9	83.8	83.3	82.6	75.8	83.5
MULTI	200k voc	91.9	82.1	80.9	89.3	71.8	66.2	80.4	93.8	87.7	87.5	87.3	78.8	87.0
JOINT	Joint voc	93.1	81.3	74.7	87.7	71.5	80.7	81.5	93.3	86.1	87.2	86.0	79.9	86.5
PAIR	Disjoint voc	93.5	83.1	78.0	86.6	65.5	78.1	80.8	94.0	88.4	88.6	87.5	79.3	87.5
MONO	Token emb	93.5	84.0	76.9	88.7	60.6	83.6	81.2	93.6	87.0	87.1	84.2	78.2	86.0
+ pos emb	93.6	79.7	75.7	86.6	61.6	83.0	80.0	94.3	87.3	87.6	86.3	79.0	86.9
TRANS	+ noisig	88.2	81.3	72.2	89.4	63.9	65.1	76.7	88.0	83.3	83.2	81.8	77.5	82.7
+ adapters	88.2	81.4	76.4	89.6	63.1	77.3	79.3	88.0	84.1	83.0	81.5	73.5	82.0
Table 2: MLDoc and PAWS-X results (accuracy). mBERT results are from Eisenschlos et al. (2019) for MLDoc and from Yang et al. (2019) for PAWS-X, respectively. We bold the best result in each section with more than two models and underline the overall best result.
exclusively on English). One possible explanation for this behaviour is that existing cross-lingual benchmarks are flawed and solvable at the lexical level. For example, previous work has shown that models trained on MultiNLI—from which XNLI was derived—learn to exploit superficial cues in the data (Gururangan et al., 2018).
To better understand the cross-lingual generalization ability of these models, we create a new Crosslingual Question Answering Dataset (XQuAD). Question answering is a classic probe for natural language understanding (Hermann et al., 2015) and has been shown to be less susceptible to annotation artifacts than other popular tasks (Kaushik and Lipton, 2018). In contrast to existing classification benchmarks, extractive question answering requires identifying relevant answer spans in longer context paragraphs, thus requiring some degree of structural transfer across languages.
XQuAD consists of a subset of 240 paragraphs and 1190 question-answer pairs from the development set of SQuAD v1.1 together with their translations into ten languages: Spanish, German, Greek, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, and Hindi. Both the context paragraphs and the questions are translated by professional human translators from Gengo. In order to facilitate easy annotations of answer spans, we choose the most frequent answer for each question and mark its beginning and end in the context paragraph using special symbols, instructing translators to keep these symbols in the relevant positions in
their translations. Appendix B discusses the dataset in more details.