SlowGuess/NAACL_2021_OCR huggingface dataset:
Junjie Hu<sup>1</sup>, Melvin Johnson<sup>2</sup>, Orhan First<sup>2</sup>, Aditya Siddhant<sup>2</sup>, Graham Neubig<sup>1</sup>  
<sup>1</sup>Carnegie Mellon University, <sup>2</sup>Google Research
{junjieh,gneubig}@cs.cmu.edu,{melvinp,orhanf,adisid}@google.com
Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020a) have proven impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages. This success comes despite the fact that there is no explicit objective to align the contextual embeddings of words/sentences with similar meanings across languages together in the same space. In this paper, we present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bidirectional EncodeR). AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. We conduct experiments on zero-shot cross-lingual transfer learning for different tasks including sequence tagging, sentence retrieval and sentence classification. Experimental results on the tasks in the XTREME benchmark (Hu et al., 2020) show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLM-R-large model which has  $3.2\mathrm{x}$  the parameters of AMBER. Our code and models are available at http://github.com/junjiehu/amber.
Cross-lingual embeddings, both traditional noncontextualized word embeddings (Faruqui and Dyer, 2014) and the more recent contextualized word embeddings (Devlin et al., 2019), are an essential tool for cross-lingual transfer in downstream applications. In particular, multilingual contextualized word representations have proven effective in reducing the amount of supervision needed in a variety of cross-lingual NLP tasks such as sequence labeling (Pires et al., 2019), question answering (Artetxe et al., 2020), parsing (Wang et al.,
2019), sentence classification (Wu and Dredze, 2019) and retrieval (Yang et al., 2019a).
Some attempts at training multilingual representations (Devlin et al., 2019; Conneau et al., 2020a) simply train a (masked) language model on monolingual data from many languages. These methods can only implicitly learn which words and structures correspond to each-other across languages in an entirely unsupervised fashion, but are nonetheless quite effective empirically (Conneau et al., 2020b; K et al., 2020). On the other hand, some methods directly leverage multilingual parallel corpora (McCann et al., 2017; Eriguchi et al., 2018; Conneau and Lample, 2019; Huang et al., 2019; Siddhant et al., 2020), which gives some degree of supervision implicitly aligning the words in the two languages. However, the pressure on the model to learn clear correspondences between the contextualized representations in the two languages is still implicit and somewhat weak. Because of this, several follow-up works (Schuster et al., 2019; Wang et al., 2020; Cao et al., 2020) have proposed methods that use word alignments from parallel corpora as the supervision signals to align multilingual contextualized representations, albeit in a post-hoc fashion.
In this work, we propose a training regimen for learning contextualized word representations that encourages symmetry at both the word and sentence levels at training time. Our word-level alignment objective is inspired by work in machine translation that defines objectives encouraging consistency between the source-to-target and target-to-source attention matrices (Cohn et al., 2016). Our sentence-level alignment objective encourages prediction of the correct translations within a minibatch for a given source sentence, which is inspired by work on learning multilingual sentence representations (Yang et al., 2019a; Wieting et al., 2019). In experiments, we evaluate the zero-shot crosslingual transfer performance of AMBER on four dif
*Work partially done at Google Research.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3633-3643 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: (a-b) show the computation of the target-to-source attention matrix used for the word alignment objective: (a) Masked attention for source/target (blue/green) sentences on the  $l$ -th layer; (b) Attention from  $y$  to  $x$  on the top layer. (c) shows the separate encoding of source/target sentences for the sentence alignment objective.
ferent NLP tasks in the XTREME benchmark (Hu et al., 2020) including part-of-speech (POS) tagging, paraphrase classification, and sentence retrieval. We show that AMBER obtains gains of up to 1.1 average F1 score on cross-lingual POS tagging, up to 27.3 average accuracy score on sentence retrieval, and achieves competitive accuracy in paraphrase classification when compared with the XLM-R-large model. This is despite the fact that XLM-R-large is trained on data  $23.8\mathrm{x}$  as large<sup>1</sup> and has  $3.2\mathrm{x}$  parameters of AMBER. This shows that compared to large amounts of monolingual data, even a small amount of parallel data leads to significantly better cross-lingual transfer learning.
2 Cross-lingual Alignment
This section describes three objectives for training contextualized embeddings. We denote the monolingual and parallel data as  $\mathcal{M}$  and  $\mathcal{P}$  respectively.
Masked Language Modeling (MLM) A masked language modeling objective takes a pair of sentences  $x, y$ , and optimizes the prediction of randomly masked tokens in the concatenation of the sentence pair as follows:
$$
\ell_ {\mathrm {M L M}} (x, y) = - \mathbb {E} _ {s \sim [ 1, | z | ]} \log P \left(z _ {s} \mid z _ {\backslash s}\right), \tag {1}
$$
where  $z$  is the concatenation of the sentence pair  $z = [x;y]$ ,  $z_{s}$  are the masked tokens randomly sampled from  $z$ , and  $z_{\backslash s}$  indicates all the other tokens except the masked ones.
In the standard monolingual setting,  $x, y$  are two contiguous sentences in a monolingual corpus. In Conneau and Lample (2019),  $x, y$  are two sentences in different languages from a parallel cor
pus, an objective we will refer to as Translation Language Modeling (TLM).
Sentence Alignment Our first proposed objective encourages cross-lingual alignment of sentence representations. For a source-target sentence pair  $(x,y)$  in the parallel corpus, we separately calculate sentence embeddings denoted as  $\mathbf{c}_x,\mathbf{c}_y$  by averaging the embeddings in the final layer as the sentence embeddings. We then encourage the model to predict the correct translation  $y$  given a source sentence  $x$ . To do so, we model the conditional probability of a candidate sentence  $y$  being the correct translation of a source sentence  $x$  as:
$$
P (y | x) = \frac {e ^ {\mathbf {c} _ {x} ^ {T} \mathbf {c} _ {y}}}{\sum_ {y ^ {\prime} \in \mathcal {M} \cup \mathcal {P}} e ^ {\mathbf {c} _ {x} ^ {T} \mathbf {c} _ {y ^ {\prime}}}}. \tag {2}
$$
where  $y^\prime$  can be any sentence in any language. Since the normalization term in Eq. (2) is intractable, we approximate  $P(y|x)$  by sampling  $y^\prime$  within a mini-batch  $\mathcal{B}$  rather than  $\mathcal{M} \cup \mathcal{P}$ . We then define the sentence alignment loss as the average negative log-likelihood of the above probability:
$$
\ell_ {\mathrm {S A}} (x, y) = - \log P (y | x). \tag {3}
$$
Bidirectional Word Alignment Our second proposed objective encourages alignment of word embeddings by leveraging the attention mechanism in the Transformer model. Motivated by the work on encouraging the consistency between the source-to-target and target-to-source translations (Cohn et al., 2016; He et al., 2016), we create two different attention masks as the inputs to the Transformer model, and obtain two attention matrices in the top layer of the Transformer model. We compute the target-to-source attention matrix  $\mathbf{A}_{y\rightarrow x}$  as follows:
${}^{2}$  In comparison, mBERT encodes a sentence pair jointly, then uses the CLS token embedding to perform its next sentence prediction task.
$^{1}$ AMBER is trained on 26GB parallel data and 80GB monolingual Wikipedia data, while XLM-R-large is trained on 2.5TB monolingual CommonCrawl data.
$$
\mathbf {g} _ {y _ {i}} ^ {l} = \operatorname {A t t n} (\mathbf {Q} = \mathbf {g} _ {y _ {i}} ^ {l - 1}, \mathbf {K V} = \mathbf {g} _ {[ y <   i: x ]} ^ {l - 1}; W ^ {l}), \tag {4}
$$
$$
\mathbf {g} _ {x _ {j}} ^ {l} = \operatorname {A t t n} \left(\mathbf {Q} = \mathbf {g} _ {x _ {j}} ^ {l - 1}, \mathbf {K V} = \mathbf {g} _ {x} ^ {l - 1}; W ^ {l}\right), \tag {5}
$$
$\operatorname{Attn}(\mathbf{QKV};W) = \operatorname{softmax}(\mathbf{QW}^q (\mathbf{KW}^k)^T)\mathbf{VW}^v$  (6)
$$
A _ {y \rightarrow x} [ i, j ] = \mathbf {g} _ {y i} ^ {L} \cdot \mathbf {g} _ {x j} ^ {L}. \tag {7}
$$
where  $\mathbf{g}_{y_t}^l$  is the embedding of the  $t$ -th word in  $y$  on the  $l$ -th layer,  $A_{y \to x}[i,j]$  is the  $(i,j)$ -th value in the attention matrix from  $y$  to  $x$ , and  $W = \{\mathbf{W}^q, \mathbf{W}^k, \mathbf{W}^v\}$  are the linear projection weights for  $Q, K, V$  respectively. We compute the source-to-target matrix  $\mathbf{A}_{x \to y}$  by switching  $x$  and  $y$ .
To encourage the model to align source and target words in both directions, we aim to minimize the distance between the forward and backward attention matrices. Similarly to Cohn et al. (2016), we aim to maximize the trace of two attention matrices, i.e.,  $\operatorname{tr}(\mathbf{A}_{y\rightarrow x}^T\mathbf{A}_{x\rightarrow y})$ . Since the attention scores are normalized in [0, 1], the trace of two attention matrices is upper bounded by  $\min(|x|, |y|)$  and the maximum value is obtained when the two matrices are identical. Since the Transformer generates multiple attention heads, we average the trace of the bidirectional attention matrices generated by all the heads denoted by the superscript  $h$
$$
\ell_ {\mathrm {W A}} (x, y) = 1 - \frac {1}{H} \sum_ {h = 1} ^ {H} \frac {\operatorname {t r} \left(\mathbf {A} _ {y \rightarrow x} ^ {h} {} ^ {T} \mathbf {A} _ {x \rightarrow y} ^ {h}\right)}{\min (| x | , | y |)}. \tag {8}
$$
Notably, in the target-to-source attention in Eq (4), with attention masking we enforce a constraint that the  $t$ -th token in  $y$  can only perform attention over its preceding tokens  $y_{< t}$  and the source tokens in  $x$ . This is particularly useful to control the information access of the query token  $y_{t}$ , in a manner similar to that of the decoding stage of NMT. Without attention masking, the standard Transformer performs self-attention over all tokens, i.e.,  $Q = K = \mathbf{g}_{z}^{h}$ , and minimizing the distance between the two attention matrices by Eq. (8) might lead to a trivial solution where  $\mathbf{W}^q \approx \mathbf{W}^k$ .
Combined Objective Finally we combine the masked language modeling objective with the alignment objectives and obtain the total loss in Eq. (9). Notice that in each iteration, we sample a minibatch of sentence pairs from  $\mathcal{M} \cup \mathcal{P}$ .
$$
\begin{array}{l} \mathcal {L} = \mathbb {E} _ {(x, y) \in \mathcal {M} \cup \mathcal {P}} \ell_ {\mathrm {M L M}} (x, y) \tag {9} \\ + \mathbb {E} _ {(x, y) \in \mathcal {P}} \left[ \ell_ {\mathrm {S A}} (x, y) + \ell_ {\mathrm {W A}} (x, y) \right] \\ \end{array}
$$
Model	Data	Langs	Vocab	Layers	Parameters	Ratio
AMBER	Wiki & MT	104	120K	12	172M	1.0
mBERT	Wiki	104	120K	12	172M	1.0
XLM-15	Wiki & MT	15	95K	12	250M	1.5x
XLM-100	Wiki	100	200K	12	570M	3.3x
XLM-R-base	CommonCrawl	100	250K	12	270M	1.6x
XLM-R-large	CommonCrawl	100	250K	24	550M	3.2x
Unicoder	CommonCrawl & MT	100	250K	12	270M	1.6x
Table 1: Details of baseline and state-of-the-art models.
3.1 Training setup
Following the setting of Hu et al. (2020), we focus on zero-shot cross-lingual transfer where we fine-tune models on English annotations and apply the models to predict on non-English data.
Models: Table 1 shows details of models in comparison. We adopt the same architecture as mBERT for AMBER. Notably, AMBER, XLM-15 and Unicoder are trained on the additional parallel data, while the others are trained only on monolingual data. Besides, XLM-R-base/large models have  $2.6 \times 4.8 \times$  the parameters of AMBER and are trained on the larger CommonCrawl corpus. We use a simple setting for our AMBER variants in the ablation study to show the effectiveness of our proposed alignment objectives without other confounding factors such as model sizes, hyper-parameters and tokenizations in different existing studies.
Pre-training: We train AMBER on the Wikipedia data for 1M steps first using the default hyperparameters as  $\mathrm{mBERT}^3$  except that we use a larger batch of 8,192 sentence pairs, as this has proven effective in Liu et al. (2019). We then continue training the model by our objectives for another 1M steps with a batch of 2,048 sentence pairs from Wikipedia corpus and parallel corpus which is used to train XLM-15 (Conneau and Lample, 2019). We use the same monolingual data as mBERT and follow Conneau and Lample (2019) to prepare the parallel data with one change to maintain truecasing. We set the maximum number of subwords in the concatenation of each sentence pair to 256 and use 10k warmup steps with the peak learning rate of 1e-4 and a linear decay of the learning rate. We train AMBER on TPU v3 for about 1 week.
Cross-lingual Part-Of-Speech (POS) contains data in 13 languages from the Universal Dependencies v2.3 (Nivre et al., 2018).
$\mathbf{PAWS - X}$  (Yang et al., 2019b) is a paraphrase detection dataset. We train on the English data (Zhang et al., 2019), and evaluate the prediction accuracy on the test set translated into 4 other languages.
XNLI (Conneau et al., 2018) is a natural language inference dataset in 15 languages. We train models on the English MultiNLI training data (Williams et al., 2018), and evaluate on the other 14.
Tatoeba (Artetxe and Schwenk, 2019) is a testbed for parallel sentence identification. We select the 14 non-English languages covered by our parallel data, and follow the setup in Hu et al. (2020) finding the English translation for a given a non-English sentence with maximum cosine similarity.
3.3 Result Analysis
In Table 2, we show the average results over all languages in all the tasks, and show detailed results for each language in Appendix A.3. First, we find that our re-trained mBERT (AMBER with MLM) performs better than the publicly available mBERT on all the tasks, confirming the utility of pre-training BERT models with larger batches for more steps (Liu et al., 2019). Second, AMBER trained by the word alignment objective obtains a comparable average F1 score with respect to the best performing model (Unicoder) in the POS tagging task, which shows the effectiveness of the word-level alignment in the syntactic structure prediction tasks at the token level. Besides, it is worth noting that Unicoder is initialized from the larger XLM-R-base model that is pre-trained on a larger corpus than AMBER, and Unicoder improves over XLM-R-base on all tasks. Third, for the sentence classification tasks, AMBER trained with our explicit alignment objectives obtain a larger gain (up to 2.1 average accuracy score in PAWS-X, and 3.9 average accuracy score in XNLI) than AMBER with only the MLM objective. Although we find that AMBER trained with only the MLM objective falls behind existing XLM/XLM-R/Unicoder models with many more parameters, AMBER trained with our alignment objectives significantly narrows the gap of classification accuracy with respect to XLM/XLM-R/Unicoder. Finally, for sentence retrieval tasks, we find that XLM-15 and Unicoder are both trained on additional parallel data, outperforming the other existing models trained only on monolingual data. Using additional parallel data, AMBER with MLM and TLM objectives also significantly improves over AMBER
Model	POS	PAWS-X	XNLI	Tatoeba
mBERT (public)	68.5	86.2	65.4	45.6
XLM-15	68.8	88.0	72.6	77.2
XLM-100	69.5	86.4	69.1	36.6
XLM-R-base	68.8	87.4	73.4	57.6
XLM-R-large	70.0	89.4	79.2	60.6
Unicoder	71.7	88.1	74.8	72.2
AMBER (MLM)	69.8	87.1	67.7	52.6
AMBER (MLM+TLM)	70.5	87.7	70.9	68.2
AMBER (MLM+TLM+WA)	71.1	89.0	71.3	68.8
AMBER (MLM+TLM+WA+SA)	70.5	89.2	71.6	87.9
Table 2: Overall results on POS, PAWS-X, XNLI, Tatoeba tasks. Bold numbers highlight the highest scores across languages on the existing models (upper part) and AMBER variants (bottom part).
with the MLM objective by 15.6 average accuracy score, while combining our word-level alignment objective yields a marginal improvement over AMBER with MLM and TLM objectives. However, adding the sentence-level alignment objective, AMBER trained by the combined objective can further improve AMBER with the MLM and word-level alignment objectives by 19.1 average accuracy score. This confirms our intuition that the explicit sentence-level objective can effectively leverage the alignment supervision in the parallel corpus, and encourage contextualized sentence representations of aligned pairs to be close according to the cosine similarity metric.
3.4 How does alignment help by language?
In Figure 2, we investigate the improvement of the alignment objectives over the MLM objective on low-resourced and high-resourced languages, by computing the performance difference between AMBER trained with alignment objectives and AMBER (MLM). First, we find that AMBER trained with alignment objectives significantly improves the performance on languages with relatively small amounts of parallel data, such as Turkish, Urdu, Swahili, while the improvement on high-resourced languages is marginal. Through a further analysis (Appendix A.3), we observe that AMBER (MLM) performs worse on these low-resourced and morphologically rich languages than on high-resourced Indo-European languages, while AMBER trained with alignment objectives can effectively bridge the gap. Moreover, AMBER trained with our word-level alignment objective yields the highest improvement on these low-resourced languages on the POS task, and AMBER trained with sentence-level alignment performs the best on XNLI.
Methods	en	bg	de	el	es	fr	Avg.
Cao et al. (2020)	80.1	73.4	73.1	71.4	75.5	74.5	74.7
AMBER (full)	84.7	74.3	74.2	72.5	76.9	76.6	76.5
Table 3: F1 scores of AMBER trained with all objectives and Cao et al. (2020) on 6 languages on XNLI.
AMBER (MLM+TLM) AMBER (MLM+TLM+WA) AMBER (MLM+TLM+WA+SA)
(a) Part-of-Speech
(b) XNLI
Figure 2: Performance difference between AMBER trained with alignments on parallel data and AMBER (MLM). Languages are sorted by no. of parallel data (Million) used for training AMBER with alignments.
3.5 Alignment with Attention vs Dictionary
Recent studies (Cao et al., 2020; Wang et al., 2020) have proposed to use a bilingual dictionary to align cross-lingual word representations. Compared with these methods, our word-level alignment objective encourages the model to automatically discover word alignment patterns from the parallel corpus in an end-to-end training process, which avoids potential errors accumulated in separate steps of the pipeline. Furthermore, an existing dictionary may not have all the translations for source words, especially for words with multiple senses. Even if the dictionary is relatively complete, it also requires a heuristic way to find the corresponding substrings in the parallel sentences for alignment. If we use a word alignment tool to extract a bilingual dictionary in a pipeline, errors may accumulate, hurting the accuracy of the model. Besides, Wang et al. (2020) is limited in aligning only fixed contextual embeddings from the model's top layer. Finally, we also compare AMBER trained with all the objectives and Cao et al. (2020) on a subset of languages on XNLI in Table 3. We find that our full model obtains a gain of 1.8 average F1 score.
While cross-lingual alignment is a long-standing challenge dating back to the early stage of research in word alignment (Brown et al., 1993), cross-lingual embeddings (Faruqui and Dyer, 2014;
Xing et al., 2015; Devlin et al., 2019; Conneau et al., 2020a) are highly promising in their easy integration into neural network models for a variety of cross-lingual applications. Analysis studies on recent cross-lingual contextualized representations (Pires et al., 2019; Wu and Dredze, 2019; Hu et al., 2020; Siddhant et al., 2020) further demonstrates this advantage for zero-shot cross-lingual transfer in a representative set of languages and tasks. In particular to improve cross-lingual transfer, some attempts directly leverage multilingual parallel corpus to train contextualized representations (McCann et al., 2017; Eriguchi et al., 2018; Conneau and Lample, 2019; Huang et al., 2019) with the hope of aligning words implicitly. The other line of work uses word alignments from parallel corpora as the alignment supervision in a posthoc fashion (Cao et al., 2020; Wang et al., 2020). Notably, AMBER does not rely on any word alignment tools, and explicitly encourage the correspondence both on the word and sentence level.
In this paper, we demonstrate the effectiveness of our proposed explicit alignment objectives in learning better cross-lingual representations for downstream tasks. Nonetheless, several challenging and promising directions can be considered in the future. First, most existing multilingual models tokenize words into subword units, which makes the alignment less interpretable. How to align a span of subword units with meaningful semantics at the phrase level deserves further investigation. Second, several studies (Ghader and Monz, 2017; Li et al., 2019) have shown that attention may fail to capture word alignment for some language pairs, and a few works (Legrand et al., 2016; Alkhouli et al., 2018) proposed neural word alignment to improve the word alignment quality. Incorporating such recent advances into the alignment objective is one future direction. Third, how to fine-tune a well-aligned multilingual model on English annotations without catastrophic forgetting of the alignment information is a potential way to improve cross-lingual generalization on the downstream applications.
We'd like to thank Yinfei Yang and Wei-Cheng Chang for answering our questions on the data and code. JH and GN were supported in part by a Google Faculty Award, and NSF Award #1761548.