Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework

1 INTRODUCTION . Continuous word representations ( Mikolov et al. , 2013a ; Pennington et al. , 2014 ; Bojanowski et al. , 2017 ) have become ubiquitous across a wide range of NLP tasks . In particular , methods for crosslingual word embeddings ( CLWE ) have proven a powerful tool for cross-lingual transfer for downstream tasks , such as text classification ( Klementiev et al. , 2012 ) , dependency parsing ( Ahmad et al. , 2019 ) , named entity recognition ( NER ) ( Xie et al. , 2018 ; Chen et al. , 2019 ) , natural language inference ( Conneau et al. , 2018b ) , language modeling ( Adams et al. , 2017 ) , and machine translation ( MT ) ( Zou et al. , 2013 ; Lample et al. , 2018a ; Artetxe et al. , 2018b ; Lample et al. , 2018b ) . The goal of these CLWE methods is to learn embeddings in a shared vector space for two or more languages . There are two main paradigms for learning CLWE : cross-lingual alignment and joint training . The most successful approach has been the cross-lingual embedding alignment method ( Mikolov et al. , 2013b ) , which relies on the assumption that monolingually-trained continuous word embedding spaces share similar structure across different languages . The underlying idea is to first independently train embeddings in different languages using monolingual corpora alone , and then learn a mapping to align them to a shared vector space . Such a mapping can be trained in a supervised fashion using parallel resources such as bilingual lexicons ( Xing et al. , 2015 ; Smith et al. , 2017 ; Joulin et al. , 2018b ; Jawanpuria et al. , 2019 ) , or even in an unsupervised2 manner based on distribution matching ( Zhang et al. , 2017a ; Conneau et al. , 2018a ; Artetxe et al. , 2018a ; Zhou et al. , 2019 ) . Recently , it has been shown that alignment methods can also be effectively applied to contextualized word representations ( Schuster et al. , 2019 ; Aldarmaki & Diab , 2019 ) . ∗Equal contribution . Ordering determined by dice rolling . 1As of the time of submission Sept 25 , 2019 . Source code is available at https : //github.com/ thespectrewithin/joint-align . 2In this paper , “ supervision ” refers to that provided by a parallel corpus or bilingual dictionaries . Another successful line of research for CLWE considers joint training methods , which optimize a monolingual objective predicting the context of a word in a monolingual corpus along with either a hard or soft cross-lingual constraint . Similar to alignment methods , some early works rely on bilingual dictionaries ( Ammar et al. , 2016 ; Duong et al. , 2016 ) or parallel corpora ( Luong et al. , 2015 ; Gouws et al. , 2015 ) for direct supervision . More recently , a seemingly naive unsupervised joint training approach has received growing attention due to its simplicity and effectiveness . In particular , Lample et al . ( 2018b ) reports that simply training embeddings on concatenated monolingual corpora of two related languages using a shared vocabulary without any cross-lingual resources is able to produce higher accuracy than the more sophisticated alignment methods on unsupervised MT tasks . Besides , for contextualized representations , unsupervised multilingual language model pretraining using a shared vocabulary has produced state-of-the-art results on multiple benchmarks3 ( Devlin et al. , 2019 ; Artetxe & Schwenk , 2019 ; Lample & Conneau , 2019 ) . Despite a large amount of research on both alignment and joint training , previous work has neither performed a systematic comparison between the two , analyzed their pros and cons , nor elucidated when we may prefer one method over the other . Particularly , it ’ s natural to ask : ( 1 ) Does the phenomenon reported in Lample et al . ( 2018b ) extend to other cross-lingual tasks ? ( 2 ) Can we employ alignment methods to further improve unsupervised joint training ? ( 3 ) If so , how would such a framework compare to supervised joint training methods that exploit equivalent resources , i.e. , bilingual dictionaries ? ( 4 ) And lastly , can this framework generalize to contextualized representations ? In this work , we attempt to address these questions . Specifically , we first evaluate and compare alignment versus joint training methods across three diverse tasks : BLI , cross-lingual NER , and unsupervised MT . We seek to characterize the conditions under which one approach outperforms the other , and glean insight on the reasons behind these differences . Based on our analysis , we further propose a simple , novel , and highly generic framework that uses unsupervised joint training as initialization and alignment as refinement to combine both paradigms . Our experiments demonstrate that our framework improves over both alignment and joint training baselines , and outperforms existing methods on the MUSE BLI benchmark . Moreover , we show that our framework can generalize to contextualized representations such as Multilingual BERT , producing state-of-the-art results on the CoNLL cross-lingual NER benchmark . To the best of our knowledge , this is the first framework that combines previously mutually-exclusive alignment and joint training methods . 2 BACKGROUND : CROSS-LINGUAL REPRESENTATIONS . Notation . We assume we have two different languages { L1 , L2 } and access to their corresponding training corpora . We use VLi = { w j Li } nLij=1 to denote the vocabulary set of the ith language where each wjLi represents a unique token , such as a word or subword . The goal is to learn a set of embeddings E = { xj } mj=1 , with xj ∈ Rd , in a shared vector space , where each token w j Li is mapped to a vector in E. Ideally , these vectorial representations should have similar values for tokens with similar meanings or syntactic properties , so they can better facilitate cross-lingual transfer . 2.1 ALIGNMENT METHODS . Given the notation , alignment methods consist of the following steps : Step 1 : Train an embedding set E0 = EL1 ∪ EL2 , where each subset ELi = { x j Li } nLij=1 is trained independently using the ith language corpus and contains an embedding xjLi for each token w j Li . Step 2 : Obtain a seed dictionary D = { ( wiL1 , w j L2 ) } Kk=1 , either provided or learnt unsupervised . Step 3 : Learn a projection matrix W ∈ Rd×d based on D , resulting in a final embedding set EA = ( W · EL1 ) ∪ EL2 in a shared vector space . To find the optimal projection matrix W , Mikolov et al . ( 2013b ) proposed to solve the following optimization problem : min W∈Rd×d ‖WXL1 −XL2‖F ( 1 ) 3https : //github.com/google-research/bert/blob/master/multilingual.md where XL1 and XL2 are matrices of size d × K containing embeddings of the words in D. Xing et al . ( 2015 ) later showed further improvement could be achieved by restricting W to an orthogonal matrix , which turns the Eq . ( 1 ) into the Procrustes problem with the following closed form solution : W ∗ = UV T , ( 2 ) with UΣV T = SVD ( XL2X T L1 ) ( 3 ) where W ∗ denotes the optimal solution and SVD ( · ) stands for the singular value decomposition . As surveyed in Section 5 , different methods ( Smith et al. , 2017 ; Conneau et al. , 2018a ; Joulin et al. , 2018b ; Artetxe et al. , 2018a ) differ in the way how they obtain the dictionary D and how they solve for W in step 3 . However , most of them still involve solving the Eq . ( 2 ) as a crucial step . 2.2 JOINT TRAINING METHODS . Joint training methods in general have the following objective : LJ = L1 + L2 +R ( L1 , L2 ) ( 4 ) where L1 and L2 are monolingual objectives and R ( L1 , L2 ) is a cross-lingual regularization term . For example , Klementiev et al . ( 2012 ) use language modeling objectives for L1 and L2 . The term R ( L1 , L2 ) encourages alignment of representations of words that are translations . Training an embedding set EJ = EL1 ∪ EL2 is usually done by directly optimizing LJ . While supervised joint training requires access to parallel resources , recent studies ( Lample et al. , 2018b ; Devlin et al. , 2019 ; Artetxe & Schwenk , 2019 ; Lample & Conneau , 2019 ) have suggested that unsupervised joint training without such resources is also effective . Specifically , they show that the cross-lingual regularization term R ( L1 , L2 ) does not require direct cross-lingual supervision to achieve highly competitive results . This is because the shared words between L1 and L2 can serve implicitly as anchors by sharing their embeddings to ensure that representations of different languages lie in a shared space . Using our notation , the unsupervised joint training approach takes the following steps : Step 1 : Construct a joint vocabulary VJ = VL1 ∪ VL2 that is shared across two languages . Step 2 : Concatenate the two training corpora and learn an embedding set EJ corresponding to VJ . The joint vocabulary is composed of three disjoint sets : V 1J , V 2 J , V s J , where V s J = VL1 ∩ VL2 is the shared vocabulary set and V iJ is the set of tokens that appear in the ith language only . Note that a key difference of existing supervised joint training methods is that embeddings corresponding to V sJ are not shared between EL1 and EL2 , meaning that they are disjoint , as in alignment methods . 2.3 DISCUSSION . While alignment methods have had great success , there are still some critical downsides , among which we stress the following points : 1 . While recent studies in unsupervised joint training have suggested the potential benefits of word sharing , alignment methods rely on two disjoint sets of embeddings . Along with some possible loss of information due to no sharing , one consequence is that finetuning the aligned embeddings on downstream tasks may be sub-optimal due to the lack of crosslingual constraints at the finetuning stage , whereas shared words can fulfill this role in jointly trained models . 2 . A key assumption of alignment methods is the isomorphism of monolingual embedding spaces . However , some recent papers have challenged this assumption , showing that it does not hold for many language pairs ( Søgaard et al. , 2018 ; Patra et al. , 2019 ) . Also notably , Ormazabal et al . ( 2019 ) suggests that this limitation results from the fact that the two sets of monolingual embeddings are independently trained . On the other hand , the unsupervised joint training method is much simpler and doesn ’ t share these disadvantages with the alignment methods , but there are also some key limitations : 1 . It assumes that all shared words across two languages serve implicitly as anchors and thus need not be aligned to other words . Nonetheless , this assumption is not always true , leading to misalignment . For example , the English word “ the ” will most likely also appear in the training corpus of Spanish , but preferably it should be paired with Spanish words such as “ el ” and “ la ” instead of itself . We refer to this problem as oversharing . 2 . It does not utilize any explicit form of seed dictionary as in alignment methods , resulting in potentially less accurate alignments , especially for words that are not shared . Lastly , while the supervised joint training approach does not have the same issues of unsupervised joint training , it shares limitation 1 of the alignment methods . We empirically compare both joint training and alignment approaches in Section 4 and shed light on some of these pros and cons for both paradigms ( See Section 4.3.1 ) .

Summary: This paper compares to approaches to bilingual lexicon induction, one which relies on joint training and the other which relies on projecting two languages' representations into a shared space. It also shows which method performs better on which of three tasks (lexicon induction, NER and MT). The paper includes an impressive number of comparisons and tasks which illuminate these two popular methods, making this a useful contribution. It also compares contextualized to non-contextualized embeddings.