Venue: NAACL 2021

# Explicit Alignment Objectives for Multilingual Bidirectional Encoders

 Junjie Hu1, Melvin Johnson2, Orhan Firat2, Aditya Siddhant2, Graham Neubig1

###### Abstract

Pre-trained cross-lingual encoders such as mBERT Devlin et al. (2019) and XLM-R Conneau et al. (2020) have proven impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages. This success comes despite the fact that there is no explicit objective to align the contextual embeddings of words/sentences with similar meanings across languages together in the same space. In this paper, we present a new method for learning multilingual encoders, amberA Multilingual Bidirectional EncoderR). amber is trained on additional parallel data using two _explicit_ alignment objectives that align the multilingual representations at different granularities. We conduct experiments on zero-shot cross-lingual transfer learning for different tasks including sequence tagging, sentence retrieval and sentence classification. Experimental results on the tasks in the XTREME benchmark Hu et al. (2020) show that amber obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLM-R-large model which has 3.2x the parameters of amber. Our code and models are available at [http://github.com/junjiehu/amber](http://github.com/junjiehu/amber).

+
Footnote †: Work partially done at Google Research.

## 1 Introduction

Cross-lingual embeddings, both traditional noncontextualized word embeddings Faruqui and Dyer (2014) and the more recent contextualized word embeddings Devlin et al. (2019), are an essential tool for cross-lingual transfer in downstream applications. In particular, multilingual contextualized word representations have proven effective in reducing the amount of supervision needed in a variety of cross-lingual NLP tasks such as sequence labeling Pires et al. (2019), question answering Artetxe et al. (2020), parsing Wang et al. (2019), sentence classification Wu and Dredze (2019) and retrieval Yang et al. (2019).

Some attempts at training multilingual representations Devlin et al. (2019); Conneau et al. (2020) simply train a (masked) language model on monolingual data from many languages. These methods can only _implicitly_ learn which words and structures correspond to each-other across languages in an entirely unsupervised fashion, but are nonetheless quite effective empirically Conneau et al. (2020); K et al. (2020). On the other hand, some methods directly leverage multilingual parallel corpora McCann et al. (2017); Eriguchi et al. (2018); Conneau and Lample (2019); Huang et al. (2019); Siddhant et al. (2020), which gives some degree of supervision implicitly aligning the words in the two languages. However, the pressure on the model to learn clear correspondences between the contextualized representations in the two languages is still implicit and somewhat weak. Because of this, several follow-up works Schuster et al. (2019); Wang et al. (2020); Cao et al. (2020) have proposed methods that use word alignments from parallel corpora as the supervision signals to align multilingual contextualized representations, albeit in a _post-hoc_ fashion.

In this work, we propose a training regimen for learning contextualized word representations that encourages symmetry at both the word and sentence levels _at training time_. Our word-level alignment objective is inspired by work in machine translation that defines objectives encouraging consistency between the source-to-target and target-to-source attention matrices Cohn et al. (2016). Our sentence-level alignment objective encourages prediction of the correct translations within a minibatch for a given source sentence, which is inspired by work on learning multilingual sentence representations Yang et al. (2019); Wieting et al. (2019). In experiments, we evaluate the zero-shot cross-lingual transfer performance of amber on four dif

ferent NLP tasks in the XTREME benchmark (Hu et al., 2020) including part-of-speech (POS) tagging, paraphrase classification, and sentence retrieval. We show that amber obtains gains of up to 1.1 average F1 score on cross-lingual POS tagging, up to 27.3 average accuracy score on sentence retrieval, and achieves competitive accuracy in paraphrase classification when compared with the XLM-R-large model. This is despite the fact that XLM-R-large is trained on data 23.8x as large1 and has 3.2x parameters of amber. This shows that compared to large amounts of monolingual data, even a small amount of parallel data leads to significantly better cross-lingual transfer learning.

Footnote 1: amber is trained on 26GB parallel data and 80GB monolingual Wikipedia data, while XLM-R-large is trained on 2.5TB monolingual CommonCrawl data.

## 2 Cross-lingual Alignment

This section describes three objectives for training contextualized embeddings. We denote the monolingual and parallel data as \({\mathcal{M}}\) and \({\mathcal{P}}\) respectively.

**Masked Language Modeling (MLM)** A masked language modeling objective takes a pair of sentences \(x,y\), and optimizes the prediction of randomly masked tokens in the concatenation of the sentence pair as follows:

\[\ell_{\text{MLM}}(x,y)=-\mathbb{E}_{s\sim[1,|z|]}\log P(z_{s}|z_{\setminus s}), \tag{1}\]

where \(z\) is the concatenation of the sentence pair \(z=[x;y]\), \(z_{s}\) are the masked tokens randomly sampled from \(z\), and \(z_{\setminus s}\) indicates all the other tokens except the masked ones.

In the standard monolingual setting, \(x,y\) are two contiguous sentences in a monolingual corpus. In Conneau and Lample (2019), \(x,y\) are two sentences in different languages from a parallel corpus, an objective we will refer to as Translation Language Modeling (TLM).

**Sentence Alignment** Our first proposed objective encourages cross-lingual alignment of sentence representations. For a source-target sentence pair \((x,y)\) in the parallel corpus, we separately calculate sentence embeddings denoted as \(\mathbf{c}_{x},\mathbf{c}_{y}\) by averaging the embeddings in the final layer as the sentence embeddings.2 We then encourage the model to predict the correct translation \(y\) given a source sentence \(x\). To do so, we model the conditional probability of a candidate sentence \(y\) being the correct translation of a source sentence \(x\) as:

Footnote 2: In comparison, mBERT encodes a sentence pair jointly, then uses the CLS token embedding to perform its next sentence prediction task.

\[P(y|x)=\frac{e^{\mathbf{c}_{x}^{T}\mathbf{c}_{y}}}{\sum_{y^{\prime}\in{ \mathcal{M}}\cup{\mathcal{P}}}e^{\mathbf{c}_{x}^{T}\mathbf{c}_{y^{\prime}}}}. \tag{2}\]

where \(y^{\prime}\) can be any sentence in any language. Since the normalization term in Eq. (2) is intractable, we approximate \(P(y|x)\) by sampling \(y^{\prime}\) within a mini-batch \({\mathcal{B}}\) rather than \({\mathcal{M}}\cup{\mathcal{P}}\). We then define the sentence alignment loss as the average negative log-likelihood of the above probability:

\[\ell_{\text{SA}}(x,y)=-\log P(y|x). \tag{3}\]

**Bidirectional Word Alignment** Our second proposed objective encourages alignment of word embeddings by leveraging the attention mechanism in the Transformer model. Motivated by the work on encouraging the consistency between the source-to-target and target-to-source translations (Chun et al., 2016; He et al., 2016), we create two different attention masks as the inputs to the Transformer model, and obtain two attention matrices in the top layer of the Transformer model. We compute the target-to-source attention matrix \(\mathbf{A}_{y\to x}\) as follows:

Figure 1: (a-b) show the computation of the target-to-Source attention matrix used for the **word alignment** objective: (a) Masked attention for source/target (blue/green) sentences on the \(l\)-th layer; (b) Attention from \(y\) to \(x\) on the top layer. (c) shows the separate encoding of source/target sentences for the **sentence alignment** objective.

 

\[{\bf g}^{l}_{y_{t}}={\rm Attn}({\bf Q}={\bf g}^{l-1}_{y_{u}},{\bf K }{\bf V}={\bf g}^{l-1}_{|y_{<t;t;t]}};W^{l}), \tag{4}\] \[{\bf g}^{l}_{x_{j}}={\rm Attn}({\bf Q}={\bf g}^{l-1}_{x_{j}},{\bf K }{\bf V}={\bf g}^{l-1}_{z};W^{l}),\] (5) \[{\rm Attn}({\bf QKV};W)={\rm softmax}({\bf QW}^{\#}({\bf KW}^{k})^{T }){\bf VV}^{v}\] (6) \[A_{y\to x}[i,j]={\bf g}^{L}_{y_{t}}.~{}{\bf g}^{L}_{x_{j}}. \tag{7}\]

where \({\bf g}^{l}_{y_{t}}\) is the embedding of the \(t\)-th word in \(y\) on the \(l\)-th layer, \(A_{y\to x}[i,j]\) is the \((i,j)\)-th value in the attention matrix from \(y\) to \(x\), and \(W=\{{\bf W}^{q},{\bf W}^{k},{\bf W}^{v}\}\) are the linear projection weights for \(Q,K,V\) respectively. We compute the source-to-target matrix \({\bf A}_{x\to y}\) by switching \(x\) and \(y\).

To encourage the model to align source and target words in both directions, we aim to minimize the distance between the forward and backward attention matrices. Similarly to Cohn et al. (2016), we aim to maximize the trace of two attention matrices, i.e., \({\rm tr}\left({{\bf A}_{y\to x}}^{T}{\bf A}_{x\to y}\right)\). Since the attention scores are normalized in \([0,1]\), the trace of two attention matrices is upper bounded by \(\min(|x|,|y|)\), and the maximum value is obtained when the two matrices are identical. Since the Transformer generates multiple attention heads, we average the trace of the bidirectional attention matrices generated by all the heads denoted by the superscript \(h\)

\[\ell_{\text{WA}}(x,y)=1-\frac{1}{H}\sum_{h=1}^{H}\frac{{\rm tr}\left({{\bf A}^ {h}_{y\to x}}^{T}{\bf A}^{h}_{x\to y}\right)}{\min(|x|,|y|)}. \tag{8}\]

Notably, in the target-to-source attention in Eq (4), with attention masking we enforce a constraint that the \(t\)-th token in \(y\) can only perform attention over its preceding tokens \(y_{<t}\) and the source tokens in \(x\). This is particularly useful to control the information access of the query token \(y_{t}\), in a manner similar to that of the decoding stage of NMT. Without attention masking, the standard Transformer performs self-attention over all tokens, i.e., \(Q=K={\bf g}^{h}_{z}\), and minimizing the distance between the two attention matrices by Eq. (8) might lead to a trivial solution where \({\bf W}^{q}\approx{\bf W}^{k}\).

**Combined Objective** Finally we combine the masked language modeling objective with the alignment objectives and obtain the total loss in Eq. (9). Notice that in each iteration, we sample a mini-batch of sentence pairs from \(\mathcal{M}\cup\mathcal{P}\).

\[\mathcal{L}= \mathbb{E}_{(x,y)\in\mathcal{M}\cup\mathcal{P}}~{}\ell_{\text{ MLM}}(x,y) \tag{9}\] \[+\mathbb{E}_{(x,y)\in\mathcal{P}}~{}[\ell_{\text{SA}}(x,y)+\ell_{ \text{WA}}(x,y)]\]

## 3 Experiments

### Training setup

Following the setting of Hu et al. (2020), we focus on _zero-shot cross-lingual transfer_ where we fine-tune models on English annotations and apply the models to predict on non-English data.

**Models**: Table 1 shows details of models in comparison. We adopt the same architecture as mBERT for amber. Notably, amber, XLM-15 and Unicoder are trained on the additional parallel data, while the others are trained only on monolingual data. Besides, XLM-R-base/large models have 2.6x/4.8x the parameters of amber and are trained on the larger CommonCrawl corpus. We use a simple setting for our amber variants in the ablation study to show the effectiveness of our proposed alignment objectives without other confounding factors such as model sizes, hyper-parameters and tokenizations in different existing studies.

**Pre-training**: We train amber on the Wikipedia data for 1M steps first using the default hyperparameters as mBERT3 except that we use a larger batch of 8,192 sentence pairs, as this has proven effective in Liu et al. (2019). We then continue training the model by our objectives for another 1M steps with a batch of 2,048 sentence pairs from Wikipedia corpus and parallel corpus which is used to train XLM-15 Conneau and Lample (2019). We use the same monolingual data as mBERT and follow Conneau and Lample (2019) to prepare the parallel data with one change to maintain rurecaling. We set the maximum number of subwords in the concatenation of each sentence pair to 256 and use 10k warmup steps with the peak learning rate of 1e-4 and a linear decay of the learning rate. We train amber on TPU v3 for about 1 week.

Footnote 3: [https://github.com/google-research/bert](https://github.com/google-research/bert)

### Datasets

**Cross-lingual Part-Of-Speech (POS)** contains data in 13 languages from the Universal Dependencies v2.3 Nivre et al. (2018).

\begin{table}
\begin{tabular}{l c r r l l l} Model & Data & Langs & Vocab & Layers & Parameters & Ratio \\ amber & Wiki \& MT & 104 & 120K & 12 & 172M & 1.0 \\ mBERT & Wiki & 104 & 120K & 12 & 172M & 1.0 \\ XLM-15 & Wiki \& MT & 15 & 95K & 12 & 250M & 1.5x \\ XLM-100 & Wiki & 100 & 200K & 12 & 570M & 3.3x \\ XLM-R-base & CommonCrawl & 100 & 250K & 12 & 270M & 1.6x \\ XLM-R-Large & CommonCrawl & 100 & 250K & 24 & 550M & 3.2x \\ Unicoder & CommonCrawl \& MT & 100 & 250K & 12 & 270M & 1.6x \\ \end{tabular}
\end{table}
Table 1: Details of baseline and state-of-the-art models.

 



**PAWS-X**Yang et al. (2019) is a paraphrase detection dataset. We train on the English data Zhang et al. (2019), and evaluate the prediction accuracy on the test set translated into 4 other languages.

**XNLI**Conneau et al. (2018) is a natural language inference dataset in 15 languages. We train models on the English MultiNLI training data Williams et al. (2018), and evaluate on the other 14.

**Tatoeba**Artetxe and Schwenk (2019) is a testbed for parallel sentence identification. We select the 14 non-English languages covered by our parallel data, and follow the setup in Hu et al. (2020) finding the English translation for a given a non-English sentence with maximum cosine similarity.

### Result Analysis

In Table 2, we show the average results over all languages in all the tasks, and show detailed results for each language in Appendix A.3. First, we find that our re-trained mBERT (amber with MLM) performs better than the publicly available mBERT on all the tasks, confirming the utility of pre-training BERT models with larger batches for more steps Liu et al. (2019). Second, amber trained by the word alignment objective obtains a comparable average F1 score with respect to the best performing model Unicoder) in the POS tagging task, which shows the effectiveness of the word-level alignment in the syntactic structure prediction tasks at the token level. Besides, it is worth noting that Unicoder is initialized from the larger XLM-R-base model that is pre-trained on a larger corpus than amber, and Unicoder improves over XLM-R-base on all tasks. Third, for the sentence classification tasks, amber trained with our explicit alignment objectives obtain a larger gain (up to 2.1 average accuracy score in PAWS-X, and 3.9 average accuracy score in XNLI) than amber with only the MLM objective. Although we find that amber trained with only the MLM objective falls behind existing XLM/XLM-R/Unicoder models with many more parameters, amber trained with our alignment objectives significantly narrows the gap of classification accuracy with respect to XLM/XLM-R/Unicoder. Finally, for sentence retrieval tasks, we find that XLM-15 and Unicoder are both trained on additional parallel data, outperforming the other existing models trained only on monolingual data. Using additional parallel data, amber with MLM and TLM objectives also significantly improves over amber with the MLM objective by 15.6 average accuracy score, while combining our word-level alignment objective yields a marginal improvement over amber with MLM and TLM objectives. However, adding the sentence-level alignment objective, amber trained by the combined objective can further improve amber with the MLM and word-level alignment objectives by 19.1 average accuracy score. This confirms our intuition that the explicit sentence-level objective can effectively leverage the alignment supervision in the parallel corpus, and encourage contextualized sentence representations of aligned pairs to be close according to the cosine similarity metric.

### How does alignment help by language?

In Figure 2, we investigate the improvement of the alignment objectives over the MLM objective on low-resourced and high-resourced languages, by computing the performance difference between amber trained with alignment objectives and amber (MLM). First, we find that amber trained with alignment objectives significantly improves the performance on languages with relatively small amounts of parallel data, such as Turkish, Urdu, Swahili, while the improvement on high-resourced languages is marginal. Through a further analysis (Appendix A.3), we observe that amber (MLM) performs worse on these low-resourced and morphologically rich languages than on high-resourced Indo-European languages, while amber trained with alignment objectives can effectively bridge the gap. Moreover, amber trained with our word-level alignment objective yields the highest improvement on these low-resourced languages on the POS task, and amber trained with sentence-level alignment performs the best on XNLI.

\begin{table}
\begin{tabular}{l|c c c c} \hline Model & POS & PAWS-X & XXNI & Tatoeba \\ \hline mBERT (public) & 68.5 & 86.2 & 65.4 & 45.6 \\ XLM-15 & 68.8 & 88.0 & 72.6 & **77.2** \\ XLM-100 & 69.5 & 86.4 & 69.1 & 36.6 \\ XLM-R-base & 68.8 & 87.4 & 73.4 & 57.6 \\ SLM-R-large & 70.0 & **89.4** & **79.2** & 60.6 \\ Unicoder & **71.7** & 88.1 & 74.8 & 72.2 \\ \hline amber (MLM) & 69.8 & 87.1 & 67.7 & 52.6 \\ amber (MLM+TLM) & 70.5 & 87.7 & 70.9 & 68.2 \\ amber (MLM+TLM+WA) & **71.1** & 89.0 & 71.3 & 68.8 \\ amber (MLM+TLM+WA+SA) & 70.5 & **89.2** & **71.6** & **87.9** \\ \hline \end{tabular}
\end{table}
Table 2: Overall results on POS, PAWS-X, XNLI, Tatoeba tasks. Bold numbers highlight the highest scores across languages on the existing models (upper part) and amber variants (bottom part).





### Alignment with Attention vs Dictionary

Recent studies Cao et al. (2020); Wang et al. (2020) have proposed to use a bilingual dictionary to align cross-lingual word representations. Compared with these methods, our word-level alignment objective encourages the model to automatically discover word alignment patterns from the parallel corpus in an end-to-end training process, which avoids potential errors accumulated in separate steps of the pipeline. Furthermore, an existing dictionary may not have all the translations for source words, especially for words with multiple senses. Even if the dictionary is relatively complete, it also requires a heuristic way to find the corresponding substrings in the parallel sentences for alignment. If we use a word alignment tool to extract a bilingual dictionary in a pipeline, errors may accumulate, hurting the accuracy of the model. Besides, Wang et al. (2020) is limited in aligning only fixed contextual embeddings from the model's top layer. Finally, we also compare amber trained with all the objectives and Cao et al. (2020) on a subset of languages on XNLI in Table 3. We find that our full model obtains a gain of 1.8 average F1 score.

## 4 Related Work

While cross-lingual alignment is a long-standing challenge dating back to the early stage of research in word alignment Brown et al. (1993), cross-lingual embeddings Faruqui and Dyer (2014); Xing et al. (2015); Devlin et al. (2019); Conneau et al. (2020a) are highly promising in their easy integration into neural network models for a variety of cross-lingual applications. Analysis studies on recent cross-lingual contextualized representations Pires et al. (2019); Wu and Dredze (2019); Hu et al. (2020); Siddhant et al. (2020) further demonstrates this advantage for zero-shot cross-lingual transfer in a representative set of languages and tasks. In particular to improve cross-lingual transfer, some attempts directly leverage multilingual parallel corpus to train contextualized representations McCann et al. (2017); Eriguchi et al. (2018); Conneau and Lample (2019); Huang et al. (2019) with the hope of aligning words implicitly. The other line of work uses word alignments from parallel corpora as the alignment supervision in a post-hoc fashion Cao et al. (2020); Wang et al. (2020). Notably, amber does not rely on any word alignment tools, and explicitly encourage the correspondence both on the word and sentence level.

## 5 Discussion and Future Work

In this paper, we demonstrate the effectiveness of our proposed explicit alignment objectives in learning better cross-lingual representations for downstream tasks. Nonetheless, several challenging and promising directions can be considered in the future. First, most existing multilingual models tokenize words into subword units, which makes the alignment less interpretable. How to align a span of subword units with meaningful semantics at the phrase level deserves further investigation. Second, several studies Ghader and Monz (2017); Li et al. (2019) have shown that attention may fail to capture word alignment for some language pairs, and a few works Legrand et al. (2016); Alkhouli et al. (2018) proposed neural word alignment to improve the word alignment quality. Incorporating such recent advances into the alignment objective is one future direction. Third, how to fine-tune a well-aligned multilingual model on English annotations without catastrophic forgetting of the alignment information is a potential way to improve cross-lingual generalization on the downstream applications.

## Acknowledgements

We'd like to thank Yinfei Yang and Wei-Cheng Chang for answering our questions on the data and code. JH and GN were supported in part by a Google Faculty Award, and NSF Award #1761548.

Figure 2: Performance difference between amber trained with alignments on parallel data and amber (MLM). Languages are sorted by no. of parallel data (Million) used for training amber with alignments.





## References

* T. Alkhouli, G. Bretschner, and H. Ney (2018)On the alignment problem in multi-head attention-based neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium, pp. 177-185. External Links: Link Cited by: SS3.4.
* M. Artetxe and H. Schwenk (2019)Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics7, pp. 597-610. External Links: Document, ISSN 0021-909X Cited by: SS2.
* P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer (1993)The mathematics of statistical machine translation: parameter estimation. Computational Linguistics19 (2), pp. 263-311. External Links: ISSN 0021-909X Cited by: SS2.
* S. Cao, N. Kitaev, and D. Klein (2020)Multilingual alignment of contextual word representations. In International Conference on Learning Representations, External Links: Link Cited by: SS2.
* T. Cohn, C. D. Vu Hoang, E. Vymolova, K. Yao, C. Dyer, and G. Haffari (2016)Incorporating structural alignment biases into an attentional neural translation model. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, pp. 876-885. External Links: ISBN 978-3-319-

Hong Kong, China. Association for Computational Linguistics.
* K. K. K. Wang, S. Mayhew, and D. Roth (2020)Cross-lingual ability of multilingual bert: an empirical study. In International Conference on Learning Representations, Cited by: SS1, SS3.4.
* J. Legrand, M. Auli, and R. Collobert (2016)Neural network-based word alignment through score aggregation. In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, Berlin, Germany, pp. 66-73. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* X. Li, G. Li, L. Liu, M. Meng, and Shuming Shi (2019)On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 1293-1303. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* Y. Lin, C. Chen, J. Lee, Z. Li, Y. Zhang, M. Xia, S. Rijhwani, J. He, Z. Zhang, X. Ma, A. Anastasopoulos, P. Littell, and Graham Neubig (2019)Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3125-3135. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1, SS2.
* B. McCann, J. Bradbury, C. Xiong, and R. Socher (2017)Learned in translation: contextualized word vectors. In Advances in Neural Information Processing Systems, Vol. 30, Curran Associates, Inc.. Cited by: SS1, SS2.
* J. Nivre, M. Abrams, Z. Agic, L. Ahrenberg, L. Antonsen, M. Jesus Aramzabe, G. Arutie, M. Asahara, U. Ateyah, M. Attia, et al. (2018)Universal dependencies 2.2. Cited by: SS1, SS2.
* T. Pires, E. Schlinger, and D. Garrette (2019)How multilingual is multilingual BERT?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 4996-5001. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* T. Schuster, O. Ram, R. Barzilay, and A. Globerson (2019)Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 1599-1613. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* A. Siddhaant, M. Johnson, H. Tsai, N. Ari, J. Riesa, A. Bapna, O. Firat, and K. Raman (2020)Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, pp. 8854-8861. Cited by: SS1, SS2.
* Y. Wang, W. Che, J. Guo, Y. Liu, and T. Liu (2019)Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 5721-5727. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* Z. Wang, J. Xie, R. Xu, Y. Yang, G. Neubig, and J. G. Carbonell (2020)Cross-lingual alignment vs joint training: a comparative study and a simple unified framework. In International Conference on Learning Representations, Cited by: SS1, SS2.
* J. Wieting, K. Gimpel, G. Neubig, and I. Berg-Kirkpatrick (2019)Simple and effective paraphrastic similarity from parallel translations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 4602-4608. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* A. Williams, N. Nangia, and S. Bowman (2018)A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 1112-1122. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* S. Wu and M. Dredze (2019)Beto, benttz, beccas: the surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 833-844. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* C. Xing, D. Wang, C. Liu, and Y. Lin (2015)Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, pp. 1006-1011. External Links: ISBN 978-3-540-3213-3, Link, Document Cited by: SS1, SS2.
* Y. Yang, G. A. Abrego, S. Yuan, M. Guo, Q. Shen, D. Cer, Y. Sung, B. Strope, and R. Kurzweil (2019)Improving multilingual sentence embedding using bi-directional dual encoder with additive margin soft 

max. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_, pages 5370-5378. International Joint Conferences on Artificial Intelligence Organization.
* Yang et al. (2019b) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019b. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
* Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.

 

Appendices

### Training Details for Reproducibility

Although English is not the best source language for some target languages Lin et al. (2019), this zero-shot cross-lingual transfer setting is still practical useful as many NLP tasks only have English annotations. In the following paragraphs, we show details for reproducing our results on zero-shot cross-lingual transfer setting.

Model:We use the same architecture as mBERT for amber, and we build our amber trained with the alignment objectives on top of the original mBERT implementation at [https://github.com/google-research/bert](https://github.com/google-research/bert), and are released at [http://github.com/junjiehu/amber](http://github.com/junjiehu/amber).

Pre-training:We first train the model on the Wikipedia data for 1M steps using the default hyper-parameters in the original repository except that we use a larger batch of 8,192 sentence pairs. The max number of subwords in the concatenation of each sentence pair is set to 256. To continue training amber with additional objectives on parallel data, we use 10K warmup steps with the peak learning rate of 1e-4, and use a linear decay of the learning rate. All models are pre-trained with our proposed objectives on TPU v3, and we use the same hyper-parameter setting for our amber variants in the experiments. We follow the practice of mBERT at [https://github.com/google-research/bert/blob/master/multilingual.md#data-source-and-sampling](https://github.com/google-research/bert/blob/master/multilingual.md#data-source-and-sampling) to sample from mutllingual data for training. We select the checkpoint of all models at the 1M step for a fair comparison. It takes about 1 week to finish the pre-training.

Fine-tuning:For fine-tuning the models on the downstream applications, we use the constant learning rate of 2e-5 as suggested in the original paper Devlin et al. (2019). We fine-tune all the models for 10 epochs on the cross-lingual POS tag prediction task, and 5 epochs on the sentence classification task. We use the batch size of 32 for all the models. All models are fine-tuned on 2080Ti GPUs, and the training can be finished within 1 day.

Datasets:We use the same parallel data that is used to train XLM-15. The parallel data can be processed by this script: [https://github.com/facebookresearch/XLM/blob/master/get-data-para.sh](https://github.com/facebookresearch/XLM/blob/master/get-data-para.sh). All the datasets in the downstream applications can be downloaded by the script at [https://github.com/google-research/xtreme/blob/master/scripts/download_data.sh](https://github.com/google-research/xtreme/blob/master/scripts/download_data.sh). Table 4 lists all the data statistic of parallel data by languages.

 



\begin{table}
\begin{tabular}{l l c c c c c c c c c} \hline \hline  & ISO & \# Parallel & \# Wikipedia & \multirow{2}{*}{Script} & Language & Diacritics/ & Extensive & Bound & \multirow{2}{*}{Inflec-} & Deviatron \\ Language & 639-1 & sentences (in) & articles (in millions) & & family & & & characters & ing & clities & tion \\ \hline Arabic & ar & 9.8 & 1.02 & Arabic & Afro-Asiatic & X & & X & \multicolumn{1}{c}{X} & \multicolumn{1}{c}{} \\ Bulgarian & bg & 0.6 & 0.26 & Cyrillic & IE: Slavic & X & & \multicolumn{1}{c}{} & X & \multicolumn{1}{c}{} \\ English & en & 40.2 & 5.98 & Latin & IE: Germanic & & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ French & fr & 13.2 & 2.16 & Latin & IE: Romance & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ German & de & 9.3 & 2.37 & Latin & IE: Germanic & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Greek & el & 4.0 & 0.17 & Greek & IE: Greek & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Hindi & hi & 1.6 & 0.13 & Devanagari & IE: Indo-Aryan & X & X & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Mandarin & zh & 9.6 & 1.09 & Chinese ideograms & Sino-Tibetn & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Russian & ru & 11.7 & 1.58 & Cyrillic & IE: Slavic & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Spanish & es & 11.4 & 1.56 & Latin & IE: Romance & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Swahili & sw & 0.2 & 0.05 & Latin & Niger-Congo & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Thai & th & 3.3 & 0.13 & Brahnic & Kra-Dai & X & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Turkish & tr & 0.5 & 0.34 & Latin & Turkic & X & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Urdu & ur & 0.7 & 0.15 & Perso-Arabic & IE: Indo-Aryan & X & X & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ Vietnamese & vi & 3.5 & 1.24 & Latin & Austro-Asiatic & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \hline \end{tabular}
\end{table}
Table 4: Statistics about languages used for pre-training with our alignment objectives. Languages belong to 7 language families, with Indo-European (IE) having the most members. Diacritics / special characters: Language adds diacrtics (additional symbols to letters). Compounding: Language makes extensive use of word compounds. Bound words / clitiics: Function words attach to other words. Inflection: Words are inflected to represent grammatical meaning (e.g. case marking). Derivation: A single token can represent entire phrases or sentences.

Figure 3: Performance difference between amber trained with alignments on parallel data and amber (MLM) on XNLI task. Languages are sorted by no. of parallel data used for training amber with alignments.





\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c|c} \hline models & ar & bg & de & el & en & es & fr & hi & ru & tr & ur & vi & zh & Avg \\ \hline mBERT (public) & 14.9 & 85.2 & 89.3 & 82.8 & 95.3 & 85.7 & 84.1 & 65.1 & 86.0 & 67.5 & 57.4 & 18.5 & 58.9 & 68.5 \\ XLM-15 & 17.5 & 86.1 & 89.3 & 85.4 & 95.7 & 85.9 & 84.9 & 63.9 & 86.8 & 69.3 & 55.1 & 18.0 & 57.2 & 68.8 \\ XLM-100 & 17.1 & 85.8 & 89.3 & 85.7 & 95.4 & 85.3 & 84.3 & 67.0 & 87.1 & 65.0 & 62.0 & 19.2 & **60.2** & 69.5 \\ XLM-R-base & 17.6 & **88.5** & 91.1 & **88.2** & 95.8 & 87.2 & 85.7 & 70.1 & 88.9 & 72.7 & 61.6 & 19.2 & 27.9 & 68.8 \\ XLM-R-large & **18.1** & 87.4 & **91.9** & 87.9 & **96.3** & **87.8** & **87.3** & **76.1** & **89.9** & **74.3** & **67.6** & **19.5** & 26.5 & **70.0** \\ \hline amber (MLM, our mBERT) & 15.4 & 86.6 & 90.1 & 84.3 & 95.5 & 86.5 & 84.6 & 68.2 & 86.8 & 69.0 & 59.2 & 18.7 & 62.1 & 69.8 \\ amber (MLM+TLM) & **16.0** & **87.2** & **91.5** & **86.4** & **95.7** & 86.9 & 85.2 & 67.7 & **87.9** & 72.9 & 57.9 & 19.1 & **62.7** & 70.5 \\ amber (MLM+TLM+WA) & 14.8 & 86.9 & 90.4 & 84.9 & 95.6 & 86.7 & 84.8 & **72.5** & 87.8 & **73.9** & **63.8** & **19.5** & 62.3 & **71.1** \\ amber (MLM+TLM+WA+SA) & 14.6 & 87.1 & 90.6 & 85.9 & 95.5 & **87.0** & **86.0** & 68.6 & 87.4 & 73.4 & 60.2 & 18.8 & 61.8 & 70.5 \\ \hline \end{tabular}
\end{table}
Table 6: F1 scores of part-of-speech tag predictions from the Universal Dependency v2.3. Bold numbers highlight the highest scores across languages on the existing models (upper part) and amber variants (bottom part).

Figure 4: Performance difference between amber trained with alignments on parallel data and amber (MLM) on POS task. Languages are sorted by no. of parallel data used for training amber with alignments.