Multilingual Alignment of Contextual Word Representations

1 INTRODUCTION . Embedding alignment was originally studied for word vectors with the goal of enabling cross-lingual transfer , where the embeddings for two languages are in alignment if word translations , e.g . cat and Katze , have similar representations ( Mikolov et al. , 2013a ; Smith et al. , 2017 ) . Recently , large pretrained models have largely subsumed word vectors based on their accuracy on downstream tasks , partly due to the fact that their word representations are context-dependent , allowing them to more richly capture the meaning of a word ( Peters et al. , 2018 ; Howard & Ruder , 2018 ; Radford et al. , 2018 ; Devlin et al. , 2018 ) . Therefore , with the same goal of cross-lingual transfer but for these more complex models , we might consider contextual embedding alignment , where we observe whether word pairs within parallel sentences , e.g . cat in “ The cat sits ” and Katze in “ Die Katze sitzt , ” have similar representations . One model relevant to these questions is multilingual BERT , a version of BERT pre-trained on 104 languages that achieves remarkable transfer on downstream tasks . For example , after the model is fine-tuned on the English MultiNLI training set , it achieves 74.3 % accuracy on the test set in Spanish , which is only 7.1 % lower than the English accuracy ( Devlin et al. , 2018 ; Conneau et al. , 2018b ) . Furthermore , while the model transfers better to languages similar to English , it still achieves reasonable accuracies even on languages with different scripts . However , given the way that multilingual BERT was pre-trained , it is unclear why we should expect such high zero-shot performance . Compared to monolingual BERT which exhibits no zero-shot transfer , multilingual BERT differs only in that ( 1 ) during pre-training ( i.e . masked word prediction ) , each batch contains sentences from all of the languages , and ( 2 ) it uses a single shared vocabulary , formed by WordPiece on the concatenated monolingual corpora ( Devlin et al. , 2019 ) . Therefore , we might wonder : ( 1 ) How can we better understand BERT ’ s multilingualism ? ( 2 ) Can we further improve BERT ’ s cross-lingual transfer ? In this paper , we show that contextual embedding alignment is a useful concept for addressing these questions . First , we propose a contextual version of word retrieval to evaluate the degree of alignment , where a model is presented with two parallel corpora , and given a word within a sentence in one corpus , it must find the correct word and sentence in the other . Using this metric of alignment , we show that multilingual BERT achieves zero-shot transfer because its embeddings are partially aligned , as depicted in Figure 1 , with the degree of alignment predicting the degree of downstream transfer . Next , using between 10K and 250K sentences per language from the Europarl corpus as parallel data ( Koehn , 2005 ) , we propose a fine-tuning-based alignment procedure and show that it significantly improves BERT as a multilingual model . Specifically , on zero-shot XNLI , where the model is trained on English MultiNLI and tested on other languages ( Conneau et al. , 2018b ) , the aligned model improves accuracies by 2.78 % on average over the base model , and it remarkably matches translate-train models for Bulgarian and Greek , which approximate the fully-supervised setting . To put our results in the context of past work , we also use word retrieval to compare our finetuning procedure to two alternatives : ( 1 ) fastText augmented with sentence and aligned using rotations ( Bojanowski et al. , 2017 ; Rücklé et al. , 2018 ; Artetxe et al. , 2018 ) , and ( 2 ) BERT aligned using rotations ( Aldarmaki & Diab , 2019 ; Schuster et al. , 2019 ; Wang et al. , 2019 ) . We find that when there are multiple occurences per word , fine-tuned BERT outperforms fastText , which outperforms rotation-aligned BERT . This result supports the intuition that contextual alignment is more difficult than its non-contextual counterpart , given that a rotation , at least when applied naively , is no longer sufficient to produce strong alignments . In addition , when there is only one occurrence per word , fine-tuned BERT matches the performance of fastText . Given that context disambiguation is no longer necessary , this result suggests that our fine-tuning procedure is able to align BERT at the type level to a degree that matches non-contextual approaches . Finally , we use the contextual word retrieval task to conduct finer-grained analysis of multilingual BERT , with the goal of better understanding its strengths and shortcomings . Specifically , we find that base BERT has trouble aligning open-class compared to closed-class parts-of-speech , as well as word pairs that have large differences in usage frequency , suggesting insight into the pre-training procedure that we explore in Section 5 . Together , these experiments support contextual alignment as an important task that provides useful insight into large multilingual pre-trained models . 2 RELATED WORK . Word vector alignment . There has been a long line of works that learn aligned word vectors from varying levels of supervision ( Ruder et al. , 2019 ) . One popular family of methods starts with word vectors learned independently for each language ( using a method like skip-gram with negative sampling ( Mikolov et al. , 2013b ) ) , and it learns a mapping from source language vectors to target language vectors with a bilingual dictionary as supervision ( Mikolov et al. , 2013a ; Smith et al. , 2017 ; Artetxe et al. , 2017 ) . When the mapping is constrained to be an orthogonal linear transformation , the optimal mapping that minimizes distances between word pairs can be solved in closed form ( Artetxe et al. , 2016 ; Schonemann , 1966 ) . Alignment is evaluated using bilingual lexicon induction , so these papers also propose ways to mitigate the hubness problem in nearest neighbors , e.g . by using alternate similarity functions like CSLS ( Conneau et al. , 2018a ) . A recent set of works has also shown that the mapping can be learned with minimal to no supervision by starting with some minimal seed dictionary and alternating between learning the linear map and inducing the dictionary ( Artetxe et al. , 2018 ; Conneau et al. , 2018a ; Hoshen & Wolf , 2018 ; Xu et al. , 2018 ; Chen & Cardie , 2018 ) . Incorporating context into alignment . One key challenge in making alignment context aware is that the embeddings are now different across multiple occurrences of the same word . Past papers have handled this issue by removing context and aligning the “ average sense ” of a word . In one such study , Schuster et al . ( 2019 ) learn a rotation to align contextual ELMo embeddings ( Peters et al. , 2018 ) with the goal of improving zero-shot multilingual dependency parsing , and they handle context by taking the average embedding for a word in all of its contexts . In another paper , Aldarmaki & Diab ( 2019 ) learn a rotation on sentence vectors , produced by taking the average word vector over the sentence , and they show that the resulting alignment also works well for word-level tasks . In a contemporaneous work , Wang et al . ( 2019 ) align not only the word but also the context by learning a linear transformation using word-aligned parallel data to align multilingual BERT , with the goal of improving zero-shot dependency parsing numbers . In this paper , we similarly align not only the word but also the context , and we also depart from these past works by using more expressive alignment methods than rotation . Incorporating parallel texts into pre-training . Instead of performing alignment post-hoc , another line of works proposes contextual pre-training procedures that are more cross-lingually-aware . Wieting et al . ( 2019 ) pre-train sentence embeddings using parallel texts by maximizing similarity between sentence pairs while minimizing similarity with negative examples . Lample & Conneau ( 2019 ) propose a cross-lingual pre-training objective that incorporates parallel data in addition to monolingual corpora , leading to improved downstream cross-lingual transfer . In contrast , our method uses less parallel data and aligns existing pre-trained models rather than requiring pretraining from scratch . Analyzing multilingual BERT . Pires et al . ( 2019 ) present a series of probing experiments to better understand multilingual BERT , and they find that transfer is possible even between dissimilar languages , but that it works better between languages that are typologically similar . They conclude that BERT is remarkably multilingual but falls short for certain language pairs . 3 METHODS . 3.1 MULTILINGUAL PRE-TRAINING . We first briefly describe multilingual BERT ( Devlin et al. , 2018 ) . Like monolingual BERT , multilingual BERT is pre-trained on sentences from Wikipedia to perform two tasks : masked word prediction , where it must predict words that are masked within a sentence , and next sentence prediction , where it must predict whether the second sentence follows the first one . The model is trained on 104 languages , with each batch containing training sentences from each language , and it uses a shared vocabulary formed by WordPiece on the 104 Wikipedias concatenated ( Wu et al. , 2016 ) . 3.2 DEFINING AND EVALUATING CONTEXTUAL ALIGNMENT . In the following sections , we describe how to define , evaluate , and improve contextual alignment . Given two languages , a model is in contextual alignment if it has similar representations for word pairs within parallel sentences . More precisely , suppose we have N parallel sentences C = { ( s1 , t1 ) , ... , ( sN , tN ) } , where ( s , t ) is a source-target sentence pair . Also , let each sentence pair ( s , t ) have word pairs , denoted a ( s , t ) = { ( i1 , j1 ) , ... , ( im , jm ) } , containing position tuples ( i , j ) such that the words si and tj are translations of each other.1 We will use f to represent a pre-trained model such that f ( i , s ) is the contextual embedding for the ith word in s. 1These pairs are called word alignments in the machine translation community , but we use the term “ word pairs ” to avoid confusion with embedding alignment . Also , because BERT operates on subwords while the corpus is aligned at the word level , we keep only the BERT vector for the last subword of each word . As an example , we might have the following sentence pair : s = { 0 I 1 ate 2 the 3 apple 4 . } t = { 0 Ich 1 habe 2 den 3 Apfel 4 gegessen 5 . } a ( s , t ) = { ( 0 , 0 ) , ( 1 , 4 ) , ( 2 , 2 ) , ( 3 , 3 ) , ( 4 , 5 ) } Then , using the parallel corpus C , we can measure the contextual alignment of the model f using its accuracy in contextual word retrieval . In this task , the model is presented with two parallel corpora , and given a word within a sentence in one corpus , it must find the correct word and sentence in the other . Specifically , we can define a nearest neighbor retrieval function neighbor ( i , s ; f , C ) = argmax t∈C , 0≤j≤len ( t ) sim ( f ( i , s ) , f ( j , t ) ) , where i and j denote the position within a sentence and sim is a similarity function . The accuracy is then given by the percentage of exact matches over the entire corpus , or A ( f ; C ) = 1 N ∑ ( s , t ) ∈C ∑ ( i , j ) ∈a ( s , t ) I ( neighbor ( i , s ; f , C ) = ( j , t ) ) , where I represents the indicator function . We can perform the same procedure in the other direction , where we pull target words given source words , so we report the average between the two directions . As our similarity function , we use CSLS , a modified version of cosine similarity that mitigates the hubness problem , with neighborhood size 10 ( Conneau et al. , 2018a ) . One additional point is that this procedure can be made more or less contextual based on the corpus : a corpus with more occurrences for each word type requires better representations of context . Therefore , we also test non-contextual word retrieval by removing all but the first occurrence of each word type . Given parallel data , these word pairs can be procured in an unsupervised fashion using standard techniques developed by the machine translation community ( Brown et al. , 1993 ) . While these methods can be noisy , by running the algorithm in both the source-target and target-source directions and only keeping word pairs in their intersection , we can trade-off coverage for accuracy , producing a reasonably high-precision dataset ( Och & Ney , 2003 ) .

Summary: The paper proposed a pre-training method for strengthening the contextual embeddings alignment. Given parallel sentences from a different language, the authors proposed to enforce corresponding words that have a similar representation by minimizing the squared error loss. The authors also proposed to use the an regulation that prevents the learned embedding from drift too far. The authors evaluated the proposed pre-training on the contextual alignment metric and show the BERT has variable accuracy depends on the language. The proposed method improved significantly on zero-shot XNLI compares to the base model.