Title: CROSS-LINGUAL ALIGNMENT VS JOINT TRAINING: A COMPARATIVE STUDY AND A SIMPLE UNIFIED FRAMEWORK

Abstract: Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly. In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse crosslingual tasks. Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is taskdependent. Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches. Extensive experiments demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We further show that this framework can generalize to contextualized representations such as Multilingual BERT, and produces state-of-the-art results on the CoNLL cross-lingual NER benchmark. 1 * Equal contribution. Ordering determined by dice rolling. 1 As of the time of submission Sept 25, 2019. Source

1. INTRODUCTION:

Continuous word representations (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) have become ubiquitous across a wide range of NLP tasks. In particular, methods for crosslingual word embeddings (CLWE) have proven a powerful tool for cross-lingual transfer for downstream tasks, such as text classification (Klementiev et al., 2012) , dependency parsing (Ahmad et al., 2019) , named entity recognition (NER) (Xie et al., 2018; Chen et al., 2019) , natural language inference (Conneau et al., 2018b) , language modeling (Adams et al., 2017) , and machine translation (MT) (Zou et al., 2013; Lample et al., 2018a; Artetxe et al., 2018b; Lample et al., 2018b) . The goal of these CLWE methods is to learn embeddings in a shared vector space for two or more languages. There are two main paradigms for learning CLWE: cross-lingual alignment and joint training.

The most successful approach has been the cross-lingual embedding alignment method (Mikolov et al., 2013b) , which relies on the assumption that monolingually-trained continuous word embedding spaces share similar structure across different languages. The underlying idea is to first independently train embeddings in different languages using monolingual corpora alone, and then learn a mapping to align them to a shared vector space. Such a mapping can be trained in a supervised fashion using parallel resources such as bilingual lexicons (Xing et al., 2015; Smith et al., 2017; Joulin et al., 2018b; Jawanpuria et al., 2019) , or even in an unsupervised 2 manner based on distribution matching (Zhang et al., 2017a; Conneau et al., 2018a; Artetxe et al., 2018a; Zhou et al., 2019) . Recently, it has been shown that alignment methods can also be effectively applied to contextualized word representations (Schuster et al., 2019; Aldarmaki & Diab, 2019) .

Another successful line of research for CLWE considers joint training methods, which optimize a monolingual objective predicting the context of a word in a monolingual corpus along with either a hard or soft cross-lingual constraint. Similar to alignment methods, some early works rely on bilingual dictionaries (Ammar et al., 2016; Duong et al., 2016) or parallel corpora (Luong et al., 2015; Gouws et al., 2015) for direct supervision. More recently, a seemingly naive unsupervised joint training approach has received growing attention due to its simplicity and effectiveness. In particular, Lample et al. (2018b) reports that simply training embeddings on concatenated monolingual corpora of two related languages using a shared vocabulary without any cross-lingual resources is able to produce higher accuracy than the more sophisticated alignment methods on unsupervised MT tasks. Besides, for contextualized representations, unsupervised multilingual language model pretraining using a shared vocabulary has produced state-of-the-art results on multiple benchmarks3 (Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019) .

Despite a large amount of research on both alignment and joint training, previous work has neither performed a systematic comparison between the two, analyzed their pros and cons, nor elucidated when we may prefer one method over the other. Particularly, it's natural to ask: (1) Does the phenomenon reported in Lample et al. (2018b) extend to other cross-lingual tasks? (2) Can we employ alignment methods to further improve unsupervised joint training? (3) If so, how would such a framework compare to supervised joint training methods that exploit equivalent resources, i.e., bilingual dictionaries? (4) And lastly, can this framework generalize to contextualized representations?

In this work, we attempt to address these questions. Specifically, we first evaluate and compare alignment versus joint training methods across three diverse tasks: BLI, cross-lingual NER, and unsupervised MT. We seek to characterize the conditions under which one approach outperforms the other, and glean insight on the reasons behind these differences. Based on our analysis, we further propose a simple, novel, and highly generic framework that uses unsupervised joint training as initialization and alignment as refinement to combine both paradigms. Our experiments demonstrate that our framework improves over both alignment and joint training baselines, and outperforms existing methods on the MUSE BLI benchmark. Moreover, we show that our framework can generalize to contextualized representations such as Multilingual BERT, producing state-of-the-art results on the CoNLL cross-lingual NER benchmark. To the best of our knowledge, this is the first framework that combines previously mutually-exclusive alignment and joint training methods.

2. BACKGROUND: CROSS-LINGUAL REPRESENTATIONS:

Notation. We assume we have two different languages {L 1 , L 2 } and access to their corresponding training corpora. We use

V Li = {w j Li } n L i

j=1 to denote the vocabulary set of the ith language where each w j Li represents a unique token, such as a word or subword. The goal is to learn a set of embeddings E = {x j } m j=1 , with x j ∈ R d , in a shared vector space, where each token w j Li is mapped to a vector in E. Ideally, these vectorial representations should have similar values for tokens with similar meanings or syntactic properties, so they can better facilitate cross-lingual transfer.

2.1. ALIGNMENT METHODS:

Given the notation, alignment methods consist of the following steps:

Step 1: Train an embedding set

E 0 = E L1 ∪ E L2 , where each subset E Li = {x j Li } n L i

j=1 is trained independently using the ith language corpus and contains an embedding x j Li for each token w j Li .

Step 2: Obtain a seed dictionary D = {(w i L1 , w j L2 )} K k=1 , either provided or learnt unsupervised.

Step 3: Learn a projection matrix W ∈ R d×d based on D, resulting in a final embedding set

E A = (W • E L1 ) ∪ E L2 in a shared vector space.

To find the optimal projection matrix W , Mikolov et al. (2013b) proposed to solve the following optimization problem: min

W ∈R d×d W X L1 -X L2 F (1)

where X L1 and X L2 are matrices of size d × K containing embeddings of the words in D. Xing et al. (2015) later showed further improvement could be achieved by restricting W to an orthogonal matrix, which turns the Eq.( 1) into the Procrustes problem with the following closed form solution:

EQUATION

with

EQUATION

where W * denotes the optimal solution and SVD(•) stands for the singular value decomposition.

As surveyed in Section 5, different methods (Smith et al., 2017; Conneau et al., 2018a; Joulin et al., 2018b; Artetxe et al., 2018a) differ in the way how they obtain the dictionary D and how they solve for W in step 3. However, most of them still involve solving the Eq.( 2) as a crucial step.

2.2. JOINT TRAINING METHODS:

Joint training methods in general have the following objective:

L J = L 1 + L 2 + R(L 1 , L 2 ) (4)

where L 1 and L 2 are monolingual objectives and R(L 1 , L 2 ) is a cross-lingual regularization term. For example, Klementiev et al. (2012) use language modeling objectives for L 1 and L 2 . The term R(L 1 , L 2 ) encourages alignment of representations of words that are translations. Training an embedding set E J = E L1 ∪ E L2 is usually done by directly optimizing L J .

While supervised joint training requires access to parallel resources, recent studies (Lample et al., 2018b; Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019) have suggested that unsupervised joint training without such resources is also effective. Specifically, they show that the cross-lingual regularization term R(L 1 , L 2 ) does not require direct cross-lingual supervision to achieve highly competitive results. This is because the shared words between L 1 and L 2 can serve implicitly as anchors by sharing their embeddings to ensure that representations of different languages lie in a shared space. Using our notation, the unsupervised joint training approach takes the following steps:

Step 1: Construct a joint vocabulary V J = V L1 ∪ V L2 that is shared across two languages.

Step 2: Concatenate the two training corpora and learn an embedding set E J corresponding to V J .

The joint vocabulary is composed of three disjoint sets: V 1 J , V 2 J , V s J , where V s J = V L1 ∩ V L2 is the shared vocabulary set and V i J is the set of tokens that appear in the ith language only. Note that a key difference of existing supervised joint training methods is that embeddings corresponding to V s J are not shared between E L1 and E L2 , meaning that they are disjoint, as in alignment methods.

2.3. DISCUSSION:

While alignment methods have had great success, there are still some critical downsides, among which we stress the following points:

1. While recent studies in unsupervised joint training have suggested the potential benefits of word sharing, alignment methods rely on two disjoint sets of embeddings. Along with some possible loss of information due to no sharing, one consequence is that finetuning the aligned embeddings on downstream tasks may be sub-optimal due to the lack of crosslingual constraints at the finetuning stage, whereas shared words can fulfill this role in jointly trained models.

2.:

A key assumption of alignment methods is the isomorphism of monolingual embedding spaces. However, some recent papers have challenged this assumption, showing that it does not hold for many language pairs (Søgaard et al., 2018; Patra et al., 2019) . Also notably, Ormazabal et al. (2019) suggests that this limitation results from the fact that the two sets of monolingual embeddings are independently trained.

On the other hand, the unsupervised joint training method is much simpler and doesn't share these disadvantages with the alignment methods, but there are also some key limitations: 1. It assumes that all shared words across two languages serve implicitly as anchors and thus need not be aligned to other words. Nonetheless, this assumption is not always true, leading to misalignment. For example, the English word "the" will most likely also appear in the training corpus of Spanish, but preferably it should be paired with Spanish words such as "el" and "la" instead of itself. We refer to this problem as oversharing.

2. It does not utilize any explicit form of seed dictionary as in alignment methods, resulting in potentially less accurate alignments, especially for words that are not shared.

Lastly, while the supervised joint training approach does not have the same issues of unsupervised joint training, it shares limitation 1 of the alignment methods.

We empirically compare both joint training and alignment approaches in Section 4 and shed light on some of these pros and cons for both paradigms (See Section 4.3.1).

3. PROPOSED FRAMEWORK:

Motivated by the pros and cons of both paradigms, we propose a unified framework that first uses unsupervised joint training as a coarse initialization and then applies alignment methods for refinement, as demonstrated in Figure 1 . Specifically, we first build a single set of embeddings with a shared vocabulary through unsupervised joint training, so as to alleviate the limitations of alignment methods. Next, we use a vocabulary reallocation technique to mitigate oversharing, before finally resorting back to alignment methods to further improve the embeddings' quality. Lastly, we show that this framework can generalize to contextualized representations.

3.1. UNIFYING ALIGNMENT WITH JOINT TRAINING:

Our proposed framework mainly involves three components and we discuss each of them as follows.

Joint Initialization. We use unsupervised joint training (Lample et al., 2018b) to train the initial CLWE. As described in Section 2.2, we first obtain a joint vocabulary V J and train its corresponding set of embeddings E J on the concatenated corpora of two languages. This allows us to obtain a single set of embeddings that maximizes sharing across two languages. To train embeddings, we used fastText4 (Bojanowski et al., 2017) in all our experiments for both word and subword tokens.

Vocabulary Reallocation. As discussed in Section 2.3, a key issue of unsupervised joint training is oversharing, which prohibits further refinement as shown in Figure 1 . To alleviate this drawback, we attempt to "unshare" some of the overshared words, so their embeddings can be better aligned in the next step. Particularly, we perform a vocabulary reallocation step such that words appearing mostly exclusively in the ith language are reallocated from the shared vocabulary V s J to V i J , whereas words that appear similarly frequent in both languages stay still in V s J . Formally, for each token w in the shared vocabulary V s J , we use the ratio of counts within each language to determine whether it belongs to the shared vocabulary:

EQUATION

where C Li (w) is the count of w in the training corpus of the ith language and

T Li = w C Li (w)

is the total number of tokens. The token w is allocated to the shared vocabulary if

EQUATION

)

where γ is a hyper-parameter. Otherwise, we put

w into either V 1 J or V 2 J

, where it appears mostly frequent. The above process generates three new disjoint vocabulary sets V 1 J , V 2 J , V s J and their corresponding embeddings E 1 J , E 2 J , E s J that are used thereafter. Note that, V J = V J and E J = E J .

Alignment Refinement. The unsupervised joint training method does not explicitly utilize any dictionary or form of alignment. Thus, the resulting embedding set is coarse and ill-aligned in the shared vector space, as demonstrated in Figure 1 . As a final refinement step, we utilize any off-the-shelf alignment method to refine alignments across the non-sharing embedding sets, i.e. mapping E 1 J to E 2 J and leaving E s J untouched. This step could be conducted by either supervised or unsupervised alignment method and we compare both in our experiments.

3.2. EXTENSION TO CONTEXTUALIZED REPRESENTATIONS:

As our framework is highly generic and applicable to any alignment and unsupervised joint training methods, it can naturally generalize to contextualized word representations by aligning the fixed outputs of a multilingual encoder such as multilingual BERT (M-BERT) (Devlin et al., 2019) . While our vocab reallocation technique is no longer necessary as contextualized representations are dependent on context and thus dynamic, we can still apply alignment refinement on extracted contextualized features for further improvement. For instance, as proposed by Aldarmaki & Diab (2019) , one method to perform alignment on contextualized representations is to first use word alignment pairs extracted from parallel corpora as a dictionary, learn an alignment matrix W based on it, and apply W back to the extracted representations. To obtain W , we can solve Eq.( 1) as described in Section 2.1, where the embedding matrices X L1 and X L2 now contain contextualized representations of aligned word pairs. Note that this method is applicable to fixed representations but not finetuning.

4. EXPERIMENTS:

We evaluate the proposed approach and compare with alignment and joint training methods on three NLP benchmarks. This evaluation aims to: (1) systematically compare alignment vs. joint training paradigms and reveal their pros and cons discussed in Section 2.3, (2) show that the proposed framework can effectively alleviate limitations of both alignment and joint training, and (3) demonstrate the effectiveness of the proposed framework in both non-contextualized and contextualized settings.

4.1. EVALUATION TASKS:

Bilingual Lexicon Induction (BLI) This task has been the de facto evaluation task for CLWE methods. It considers the problem of retrieving the target language translations of source langauge words. We use bilingual dictionaries complied by Conneau et al. (2018a) and test on six diverse language pairs, including Chinese and Russian, which use a different writing script than English. Each test set consists of 1500 queries and we report precision at 1 scores (P@1), following standard evaluation practices (Conneau et al., 2018a; Glavas et al., 2019) .

Name Entity Recognition (NER):

We also evaluate our proposed framework on cross-lingual NER, a sequence labeling task, where we assign a label to each token in a sequence. We evaluate both en-es es-en en-fr fr-en en-de de-en en-it it-en en-ru ru-en en-zh zh-en avg Alignment Methods (1) MUSE (Conneau et al., 2018a) 81 where we train a model on English and directly apply it to each of the other 3 languages.

Unsupervised Machine Translation (UMT) Lastly, we test our approach using the unsupervised MT task, on which the initialization of CLWE plays a crucial role (Lample et al., 2018b) . Note that our purpose here is to directly compare with similar studies in Lample et al. (2018b) , and thus we follow their settings and consider two language pairs, English-French and English-German, and evaluate on the widely used WMT'14 en-fr and WMT'16 en-de benchmarks.

4.2. EXPERIMENTAL SETUP:

For the BLI task, we compare our framework to recent state-of-the-art methods. We obtain numbers from the corresponding papers or Zhou et al. (2019) , and use the official tools for MUSE (Conneau et al., 2018a) , GeoMM (Jawanpuria et al., 2019) and RCSLS (Joulin et al., 2018b) to obtain missing results. We consider the method of Duong et al. (2016) for supervised joint training based on bilingual dictionaries, which is comparable to supervised alignment methods in terms of resources used. For unsupervised joint training, we train uncased joint fastText word vectors of dimension 300 on concatenated Wikipedia corpora of each language pair with default parameters. The hyperparameter γ is selected from {0.7, 0.8, 0.9, 0.95} on validation sets. For the alignment refinement step in our proposed framework, we use RCSLS and GeoMM to compare with supervised methods, and MUSE for unsupervised methods. In addition, we include an additional baseline of joint training, denoted as Joint -Replace, which is identical to unsupervised joint training except that it utilizes a seed dictionary to randomly replace words with their translations in the training corpus.

Following standard practices, we consider the top 200k most frequent words and use the crossdomain similarity local scaling (CSLS) (Conneau et al., 2018a) as the retrieval criteria. Note that a concurrent work (Artetxe et al., 2019) proposed a new retrieval method based on MT systems and produced state-of-the-art results. Although their method is applicable to our framework, it has high computational cost and is out of the scope of this work.

For the NER task: (1) For non-contextualized representations, we train embeddings the same way as in the BLI task and use a vanilla Bi-LSTM-CRF model (Lample et al., 2016) . For all alignment steps, we apply the supervised Procrustes method using dictionaries from the MUSE library for simplicity.

(2) For contextualized representations, we use M-BERT, an unsupervised joint training model, as en-es es-en en-fr fr-en en-de de-en en-it it-en en-ru ru-en en-zh zh-en avg Unsupervised (1) MUSE (Conneau et al., 2018a) 77 our base model and apply our proposed framework on it by first aligning its extracted features and then feeding them to a task-specific model (M-BERT Feature + Align). Specifically, we use the sum of the last 4 M-BERT layers' outputs as the extracted features. To obtain the alignment matrices, one for each layer, we use 30k parallel sentences from the Europarl corpus for each language pair and follow the procedure of Section 3.2. We feed the extracted features as inputs to a task-specific model with 2 Bi-LSTM layers and a CRF layer (see appendix A). We compare our framework to both finetuning (M-BERT Finetune), which has been studied by previous papers, and feature extraction (M-BERT Feature). Lastly, we also compare against XLM, a supervised joint training model.

For the UMT task, we use the exact same data, architecture and parameters released by Lample et al. (2018b) 7 . We simply use different embeddings trained with the same data as inputs to the model.

4.3. RESULTS AND ANALYSIS:



4.3.1. ALIGNMENT VS. JOINT TRAINING:

We compare alignment methods with joint training on all three downstream tasks. As shown in Table 1 and Table 3 , we find alignment methods significantly outperform the joint training approach by a large margin in all language pairs for both BLI and NER. However, the unsupervised joint training method is superior than its alignment counterpart on the unsupervised MT task as demonstrated in 2(c). While these results demonstrate that their relative performance is task-dependent, we conduct further analysis to reveal three limitations as discussed in Sec 2.3.

First, their poor performance on BLI and NER tasks shows that unsupervised joint training fails to generate high-quality alignments due to the lack of a fine-grained seed dictionary as discussed in its limitation 2. To evaluate accuracy on words that are not shared, we further remove test pairs of the same surface form (e.g. (hate, hate) as a test pair for en-de) of the BLI task and report their results in Table 2 . We find unsupervised joint training (row 2) to achieve extremely low scores which shows that emebddings of non-shared parts are poorly aligned, consistent with the PCA visualization shown in Figure 1 .

Moreover, we delve into the relative performance of the two paradigms on the MT task by plotting their test BLEU scores of the first 20 epochs in Figure 2 (a) and 2(b). We observe that the alignment method actually obtains higher BLEU scores in the first few epochs, but gets surpassed by joint training in later epochs. This shows the importance of parameter sharing as discussed in limitation 1 of alignment methods: shared words can be used as a cross-lingual constraint for unsupervised joint training during fine-tuning but this constraint cannot easily be used in alignment methods. The lack of sharing is also a limitation for the supervised joint training method, which performs poorly on the MT task even with supervision as shown in Figure 2(c) .

Lastly, we demonstrate that oversharing can be sub-optimal for unsupervised joint training as discussed in its limitation 2. Specifically, we conduct ablation studies for our framework in Table 1 .

Applying alignment refinement on unsupervised joint training without any vocabulary reallocation does not improve its performance (row 13). On the other hand, simple vocabulary reallocation alone boosts the performance by quite a margin (row 11). This shows some words are shared erroneously across languages in unsupervised joint training, thereby hindering its performance. In addition, while utilizing a seed dictionary (row 10) improves the performance of unsupervised joint train- ing, it still suffers from the oversharing problem and performs worse compared to supervised joint training (row 9).

4.3.2. EVALUATION OF PROPOSED FRAMEWORK:

As shown in Table 1 , Table 3 , and Figure 2 , our proposed framework substantially improves over the alignment and joint training baselines on all three tasks. In particular, it outperforms existing methods on all language pairs for the BLI task (using the CSLS as retrieval metric) and achieves state-of-the-art results on 2 out of 3 language pairs for the NER task. Besides, we show that it alleviates limitations of alignment and joint training methods shown in the previous section.

First, the proposed framework largely improves the poor alignment of unsupervised joint training, especially for non-sharing parts. As shown in Table 1 , the proposed Joint Align framework achieves comparable results to prior methods in the unsupervised case (row 12) and it outperforms previous state-of-the-art methods in the supervised setting (row 15). Specifically, our proposed framework can generate well-aligned embeddings after alignment refinement is applied to the initially ill-aligned embeddings, as demonstrated in Figure 1 . This is further verified by results in Table 2 , where our proposed framework largely improves accuracy on words not shared between two languages over the unsupervised joint training baseline (row 3 and 6 vs row 2).

Besides, our ablation study in Table 1 further shows the effectiveness of the proposed vocabulary reallocation technique, which alleviates the issue of oversharing. Particularly, we observe no improvement compared to unsupervised joint training baseline (row 8) when an alignment refinement step is used without vocabulary reallocation (row 13), while a vocabulary reallocation step alone significantly improves the performance (row 11). This is consistent with Figure 1 and shows that the oversharing is a bottleneck for applying alignment methods to joint training. It also suggests detecting what to share is crucial to achieve better cross-lingual transfer.

Lastly, while supervised joint training shares the limitation 1 of alignment methods and performs poorly when finetuned, our proposed framework exploits the same idea of vocabulary sharing used in unsupervised joint training. In the MT tasks, our framework obtains a maximum gain of 2.97 BLEU over baselines we ran and consistently performs better than results reported in Lample et al. (2018b) . In addition, Figure 2 shows that Joint Align not only converges faster in earlier training epochs but also consistently outperforms the two baselines thereafter. These empirical findings demonstrate the effectiveness of our proposed methods in the non-contextualized case.

4.3.3. CONTEXTUALIZED WORD REPRESENTATIONS:

As can be seen in Table 3 , when using our framework (M-BERT Feature + Align), we achieve state-of-the-art results on cross-lingual NER on 2 out of 3 languages and the overall average. This shows that our framework can effectively generalize to contextualized representations. Specifically, our framework improves over both the M-BERT feature extraction and finetuning baselines on all three language pairs. However, when compared to non-contextualized results, the gain of using alignment refinement on top of unsupervised joint training is much smaller. This suggests that, as the contextualized unsupervised joint training model performs very well already even without 2016) is a supervised method that we include for analysis purpose only and is not directly comparable to other results in this table.

any supervision, it is harder to achieve large improvements. While alignment refinement relies on word alignment, a process that is noisy itself, a better alignment approach may be warranted. Lastly, the reason why a supervised joint training model, XLM, performs worse than its unsupervised counterpart, M-BERT, is likely that XLM uses an uncased vocabulary, where casing information is important for NER tasks.

5. RELATED WORK:

Word embeddings (Mikolov et al., 2013a; Ruder et al., 2019) are a key ingredient to achieving success in monolingual NLP tasks. However, directly using word embeddings independently trained for each language may cause negative transfer (Wang et al., 2019) in cross-lingual transfer tasks. In order to capture the cross-lingual mapping, a rich body of existing work relying on cross-lingual supervision, including bilingual dictionaries (Mikolov et al., 2013a; Faruqui & Dyer, 2014; Artetxe et al., 2016; Xing et al., 2015; Duong et al., 2016; Gouws & Søgaard, 2015; Joulin et al., 2018a) , sentence-aligned corpora (Kočiskỳ et al., 2014; Hermann & Blunsom, 2014; Gouws et al., 2015) and document-aligned corpora (Vulić & Moens, 2016; Søgaard et al., 2015) .

Besides, unsupervised alignment methods aim to eliminate the requirement for cross-lingual supervision. Early work of Cao et al. (2016) matches the mean and the standard deviation of two embedding spaces after alignment. Barone (2016); Zhang et al. (2017a; b) ; Conneau et al. (2018a) adapted a generative adversarial network (GAN) (Goodfellow et al., 2014) to make the distributions of two word embedding spaces indistinguishable. Follow-up works improve upon GAN-based training for better stability and robustness by introducing Sinkhorn distance (Xu et al., 2018) , by stochastic self-training (Artetxe et al., 2018a) , or by introducing latent variables (Dou et al., 2018) .

While alignment methods utilize embeddings trained independently on different languages, joint training methods train word embeddings at the same time. Klementiev et al. (2012) train a bilingual dictionary-based regularization term jointly with monolingual language model objectives while Kočiskỳ et al. (2014) defines the cross-lingual regularization with the parallel corpus. Another branch of methods (Xiao & Guo, 2014; Gouws & Søgaard, 2015; Ammar et al., 2016; Duong et al., 2016) build a pseudo-bilingual corpus by randomly replacing words in monolingual corpus with their translations and use monolingual word embedding algorithms to induce bilingual representations. The unsupervised joint method by Lample & Conneau (2019) simply exploit words that share the same surface form as bilingual "supervision" and directly train a shared set of embedding with joint vocabulary. Recently, unsupervised joint training of contextualized word embeddings through the form of multilingual language model pretraining using shared subword vocabularies has produced state-of-the-art results on various benchmarks (Devlin et al., 2019; Artetxe & Schwenk, 2019; Lample & Conneau, 2019; Pires et al., 2019; Wu & Dredze, 2019) .

A concurrent work by Ormazabal et al. (2019) also compares alignment and joint method in the bilingual lexicon induction task. Different from their setup which only tests on supervised settings, we conduct analysis across various tasks and experiment with both supervised and unsupervised conditions. While Ormazabal et al. (2019) suggests that the combination of the alignment and joint model could potentially advance the state-of-art of both worlds, we propose such a framework and empirically verify its effectiveness on various tasks and settings.

6. CONCLUSION:

In this paper, we systematically compare the alignment and joint training methods for CLWE. We point out that the nature of each category of methods leads to certain strengths and limitations.

The empirical experiments on extensive benchmark datasets and various NLP tasks verified our analysis. To further improve the state-of-art of CLWE, we propose a simple hybrid framework which combines the strength from both worlds and achieves significantly better performance in the BLI, MT and NER tasks. Our work opens a promising new direction that combines two previously exclusive lines of research. For future work, an interesting direction is to find a more effective word sharing strategy.

en-es es-en en-fr fr-en en-de de-en en-it it-en en-ru ru-en en-zh zh-en avg Alignment Methods (1) MUSE (Conneau et al., 2018a) 82 and the output of the Bi-LSTM, and use Adam with default parameters and a learning rate of 0.0001 for optimization. We train the model for 40 epochs with a batch size of 10, and evaluate the model per 150 steps. For prediction, we feed the outputs of the Bi-LSTM that correspond to the first subword tokens of each word to the CRF model. This is identical to finetuning BERT on the NER task, except that in our case the outputs that correspond to the first subword token are fed into a CRF, rather than a linear layer as done in BERT.

B BLI TEST PAIRS:

The official evaluation script8 of MUSE only includes test pairs whose source words and target words both appear in their corresponding vocabularies, leaving out those that are OOV on either side. Since our proposed vocabulary reallocation step modifies both the source and target vocabularies, the script may exclude some test pairs when evaluating our model. For example, the word "age" from the test pair (age, age) for en-fr could be allocated as an en (not shared) word, so it is OOV on the fr side and the pair would thus be left out by the script. As a result, the total number of test pairs would be smaller, thereby changing the denominator when we calculate accuracy. To ensure fair comparison, we include these OOV pairs so the total number of test pairs stays the same. Specifically, if the source word of a test pair is OOV, we retrieve itself. Otherwise, we count it as incorrect.

In addition, we further investigate which pairs are left out and found that the MUSE benchmark contains some noisy test data. Specifically, we find that the majority of these pairs are in the same surface form, such as (sit, sit), but many target words are not actual translations of the source words.

For example, we found test pairs such as {(age, age), (century, century)} for en-fr and {(mickey, mickey), (uncredited, uncredited)} for en-zh. Clearly, these are English words and should not be considered as appropriate translations for French or Chinese. In Table 1 , we mark these pairs as incorrect for our framework to ensure fair comparison, while in fact our framework correctly allocates these words to English. To reveal the full picture, we also conduct BLI experiments without test pairs that got left out due to vocabulary reallocation. The results are shown in Table 4 . We observe that our proposed framework obtains a gain of 3.9 accuracy on average over the RCSLS baseline.

:

https://github.com/google-research/bert/blob/master/multilingual.md

https://github.com/facebookresearch/fastText

We found that the official evaluation script of MUSE ignores test pairs that are out-of-vocabulary (OOV) on the source or the target sides, which could occur occasionally when evaluating our model due to the proposed vocabulary reallocation step. To ensure fair comparison, we include these OOV pairs. Specifically, if the source word is OOV, we retrieve itself; otherwise, we count the pair as incorrect. Please see appendix B for details.

https://github.com/facebookresearch/UnsupervisedMT

https://github.com/facebookresearch/MUSE/blob/master/src/evaluation/ word_translation.py.

Figures and Tables:

Figure 1: PCA visualization of English and Spanish embeddings learnt by unsupervised joint training as in Lample et al. (2018b). As shown by plots (a) and (b), most words are shared in the initial embedding space but not well-aligned, hence the oversharing problem. Plots (b) and (c) shows that the vocabulary reallocation step effectively mitigates oversharing while the alignment refinement step further improves the poorly aligned embeddings by projecting them into a close neighborhood.

Figure 2: (a)(b): Results on MT of Align, Joint and our framework for the first 20 training epochs. Results after 20 epochs have similar patterns. (c): BLEU scores for the MT task. Results evaluated on the WMT'14 English-French and WMT'16 German-English. All training settings are the same for each language pair except the embedding initialization. Note that we are not trying to outperform state-of-the-art methods (Song et al., 2019) but rather to observe improvements afforded by embedding initialization. † Results reported by Lample et al. (2018b). Our results are obtained using the official code released by the author. ‡ Duong et al. (2016) is a supervised method that we include for analysis purpose only and is not directly comparable to other results in this table.

Precision@1 for the BLI task on the MUSE dataset 6 . Within each category, unsupervised methods are listed at the top while supervised methods are at the bottom. The best result for unsupervised methods is underlined while bold signifies the overall best. "IN" refers to iterative normalization proposed inZhang et al. (2019), "AR" refers to alignment refinement and "VR" refers to vocabulary reallocation. non-contextualized and contextualized word representations on the CoNLL 2002 and 2003 benchmarks(Tjong Kim Sang, 2002;Tjong Kim Sang & De Meulder, 2003), which contain 4 European languages. To measure the quality of CLWE, we perform zero-shot cross-lingual classification,

Table content: <table><tr><td/><td>.7</td><td>83.3</td><td>82.3</td><td>82.1</td><td>74.0</td><td>72.0</td><td>77.7</td><td>78.2</td><td>44.0</td><td>59.1</td><td>32.5</td><td>31.4</td><td>66.5</td></tr><tr><td>(2) VECMAP (Artetxe et al., 2018a)</td><td>82.3</td><td>84.7</td><td>82.3</td><td>83.6</td><td>75.1</td><td>74.3</td><td>-</td><td>-</td><td>49.2</td><td>65.6</td><td>0.0</td><td>0.0</td><td>-</td></tr><tr><td>(3) DeMa-BWE (Zhou et al., 2019)</td><td>82.8</td><td>84.9</td><td>83.1</td><td>83.5</td><td>77.2</td><td>74.4</td><td>-</td><td>-</td><td>49.2</td><td>65.7</td><td>42.5</td><td>37.9</td><td>-</td></tr><tr><td>(4) Procrustes (Smith et al., 2017)</td><td>81.4</td><td>82.9</td><td>81.1</td><td>82.4</td><td>73.5</td><td>72.4</td><td>77.5</td><td>77.9</td><td>51.7</td><td>63.7</td><td>42.7</td><td>36.7</td><td>68.7</td></tr><tr><td>(5) GeoMM (Jawanpuria et al., 2019)</td><td>81.4</td><td>85.5</td><td>82.1</td><td>84.1</td><td>74.7</td><td>76.7</td><td>77.9</td><td>80.9</td><td>51.3</td><td>67.6</td><td>49.1</td><td>45.3</td><td>71.4</td></tr><tr><td>(6) RCSLS (Joulin et al., 2018b)</td><td>84.1</td><td>86.3</td><td>83.3</td><td>84.1</td><td>79.1</td><td>76.3</td><td>78.5</td><td>79.8</td><td>57.9</td><td>67.2</td><td>45.9</td><td>46.4</td><td>72.4</td></tr><tr><td>(7) RCSLS + IN (Zhang et al., 2019)</td><td>83.9</td><td>-</td><td>83.9</td><td>-</td><td>78.1</td><td>-</td><td>79.1</td><td>-</td><td>57.9</td><td>-</td><td>48.6</td><td>-</td><td>-</td></tr><tr><td/><td/><td/><td colspan="3">Joint Traing Methods</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(8) Unsupervised Joint</td><td>33.4</td><td>36.6</td><td>42.2</td><td>47.4</td><td>39.5</td><td>41.4</td><td>36.8</td><td>38.8</td><td>4.0</td><td>3.5</td><td>17.9</td><td>10.2</td><td>29.3</td></tr><tr><td>(9) Supervised Joint (Duong et al., 2016)</td><td>79.7</td><td>79.8</td><td>78.1</td><td>76.7</td><td>67.5</td><td>68.9</td><td>74.4</td><td>74.1</td><td>41.8</td><td>51.8</td><td>46.7</td><td>43.3</td><td>65.2</td></tr><tr><td>(10) Joint -Replace</td><td>48.2</td><td>47.7</td><td>49.4</td><td>52.1</td><td>46.5</td><td>46.9</td><td>43.8</td><td>45.8</td><td>20.3</td><td>36.6</td><td>32.7</td><td>34.1</td><td>42.0</td></tr><tr><td/><td/><td/><td colspan="3">Joint Align Framework</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(11) Joint Align (w/o AR)</td><td>55.9</td><td>62.8</td><td>61.8</td><td>67.0</td><td>49.1</td><td>54.6</td><td>50.2</td><td>51.4</td><td>8.7</td><td>8.2</td><td>19.4</td><td>18.2</td><td>42.3</td></tr><tr><td>(12) Joint Align + MUSE</td><td>81.4</td><td>84.2</td><td>82.8</td><td>83.6</td><td>74.2</td><td>72.2</td><td>77.5</td><td>81.5</td><td>45.0</td><td>58.3</td><td>36.1</td><td>35.3</td><td>67.7</td></tr><tr><td>(13) Joint Align + RCSLS (w/o VR)</td><td>34.2</td><td>37.0</td><td>41.2</td><td>46.8</td><td>34.0</td><td>35.6</td><td>35.3</td><td>35.1</td><td>7.7</td><td>5.2</td><td>20.2</td><td>15.7</td><td>29.0</td></tr><tr><td>(14) Joint Align + GeoMM</td><td>82.6</td><td>85.7</td><td>82.5</td><td>84.2</td><td>75.5</td><td>77.2</td><td>78.2</td><td>81.4</td><td>52.4</td><td>67.7</td><td>50.4</td><td>46.5</td><td>72.0</td></tr><tr><td>(15) Joint Align + RCSLS</td><td>84.7</td><td>87.9</td><td>83.5</td><td>85.6</td><td>79.6</td><td>78.0</td><td>80.6</td><td>84.0</td><td>59.8</td><td>67.8</td><td>54.3</td><td>48.7</td><td>74.5</td></tr></table>

Precision@1 for the BLI task on the MUSE dataset with test pairs of same surface form removed. The best result for unsupervised methods is underlined while bold signifies the overall best.

Table content: <table><tr><td/><td>.1</td><td>82.5</td><td>76.4</td><td>78.0</td><td>67.4</td><td>67.8</td><td>72.5</td><td>77.5</td><td>42.7</td><td>50.2</td><td>28.7</td><td>29.1</td><td>62.5</td></tr><tr><td>(2) Unsupervised Joint</td><td>3.7</td><td>10.2</td><td>5.1</td><td>10.7</td><td>8.5</td><td>10.5</td><td>7.8</td><td>8.1</td><td>0.4</td><td>2.7</td><td>2.5</td><td>6.4</td><td>6.4</td></tr><tr><td>(3) Joint Align + MUSE</td><td>77.5</td><td>83.0</td><td>77.0</td><td>79.5</td><td>66.7</td><td>68.0</td><td>70.9</td><td>78.0</td><td>43.5</td><td>55.1</td><td>32.3</td><td>32.7</td><td>63.7</td></tr><tr><td/><td/><td/><td/><td colspan="2">Supervised</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(4) RCSLS (Joulin et al., 2018b)</td><td>78.0</td><td>83.9</td><td>76.0</td><td>78.6</td><td>68.2</td><td>68.4</td><td>71.8</td><td>78.2</td><td>50.7</td><td>56.9</td><td>51.0</td><td>41.7</td><td>67.0</td></tr><tr><td>(5) Supervised Joint (Duong et al., 2016)</td><td>76.8</td><td>80.8</td><td>73.4</td><td>76.1</td><td>60.1</td><td>61.7</td><td>69.7</td><td>76.2</td><td>41.0</td><td>51.8</td><td>52.3</td><td>43.3</td><td>63.6</td></tr><tr><td>(6) Joint Align + RCSLS</td><td>82.1</td><td>84.6</td><td>78.1</td><td>80.4</td><td>68.4</td><td>70.4</td><td>73.7</td><td>79.0</td><td>59.0</td><td>66.8</td><td>51.4</td><td>45.7</td><td>70.0</td></tr></table>

F1 score for the crosslingual NER task. "Adv" refers to adversarial training. ‡ denotes results that are not directly comparable due to different resources and architectures used.

Table content: <table><tr><td>denotes supervised XLM model</td></tr><tr><td>trained with MLM and TLM ob-</td></tr><tr><td>jectives. Its Dutch (nl) result is</td></tr><tr><td>blank because the model is not</td></tr><tr><td>pretrained on it. Bold signifies</td></tr><tr><td>state-of-the-art results. We re-</td></tr><tr><td>port the average of 5 runs.</td></tr></table>

Precision@1 for the BLI task on the MUSE dataset using test set produced by vocabulary reallocation. Within each category, unsupervised methods are listed at the top while supervised methods are at the bottom. Bold signifies the overall best results. "AR" refers to alignment refinement and "VR" refers to vocabulary reallocation.

Table content: <table><tr><td/><td>.1</td><td>83.5</td><td>82.6</td><td>83.1</td><td>74.1</td><td>71.8</td><td>77.4</td><td>79.8</td><td>44.1</td><td>59.1</td><td>34.1</td><td>31.6</td><td>66.9</td></tr><tr><td>(4) Procrustes (Smith et al., 2017)</td><td>81.6</td><td>82.0</td><td>80.5</td><td>81.5</td><td>74.2</td><td>73.3</td><td>77.0</td><td>77.8</td><td>50.8</td><td>63.5</td><td>44.2</td><td>37.0</td><td>68.6</td></tr><tr><td>(5) GeoMM (Jawanpuria et al., 2019)</td><td>82.1</td><td>86.9</td><td>82.3</td><td>85.1</td><td>75.3</td><td>77.2</td><td>78.6</td><td>82.2</td><td>51.7</td><td>67.8</td><td>50.5</td><td>45.6</td><td>72.1</td></tr><tr><td>(6) RCSLS (Joulin et al., 2018b)</td><td>82.8</td><td>84.3</td><td>82.4</td><td>83.3</td><td>78.6</td><td>75.6</td><td>78.3</td><td>81.0</td><td>57.7</td><td>66.8</td><td>48.3</td><td>45.6</td><td>72.1</td></tr><tr><td/><td/><td/><td colspan="3">Joint Traing Methods</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(7) Unsupervised Joint</td><td>33.2</td><td>36.3</td><td>41.5</td><td>46.8</td><td>39.1</td><td>40.7</td><td>35.8</td><td>38.1</td><td>4.1</td><td>3.7</td><td>8.2</td><td>5.7</td><td>27.8</td></tr><tr><td>(8) Supervised Joint (Duong et al., 2016)</td><td>80.1</td><td>80.5</td><td>78.6</td><td>77.1</td><td>67.2</td><td>68.3</td><td>74.6</td><td>74.5</td><td>41.7</td><td>51.8</td><td>47.2</td><td>44.0</td><td>65.5</td></tr><tr><td/><td/><td/><td colspan="3">Joint Align Framework</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(9) Joint Align (w/o AR)</td><td>56.8</td><td>63.2</td><td>62.2</td><td>67.2</td><td>49.2</td><td>55.1</td><td>50.6</td><td>51.9</td><td>8.7</td><td>8.2</td><td>19.5</td><td>18.4</td><td>42.6</td></tr><tr><td>(10) Joint Align + MUSE</td><td>82.4</td><td>85.0</td><td>83.5</td><td>84.7</td><td>74.6</td><td>72.9</td><td>78.2</td><td>82.6</td><td>46.1</td><td>58.7</td><td>39.9</td><td>36.2</td><td>68.7</td></tr><tr><td>(11) Joint Align + RCSLS (w/o VR)</td><td>34.3</td><td>36.8</td><td>41.0</td><td>47.0</td><td>34.3</td><td>35.9</td><td>35.5</td><td>35.2</td><td>7.6</td><td>5.2</td><td>21.3</td><td>16.1</td><td>29.2</td></tr><tr><td>(12) Joint Align + GeoMM</td><td>83.9</td><td>86.2</td><td>83.1</td><td>85.2</td><td>76.1</td><td>77.9</td><td>79.0</td><td>82.8</td><td>53.7</td><td>68.3</td><td>54.8</td><td>48.0</td><td>73.3</td></tr><tr><td>(13) Joint Align + RCSLS</td><td>87.1</td><td>88.5</td><td>84.2</td><td>86.6</td><td>80.1</td><td>78.7</td><td>81.3</td><td>85.2</td><td>61.3</td><td>68.3</td><td>59.6</td><td>50.7</td><td>76.0</td></tr></table>

Peer review (iclr2020):
This paper proposes a new and simple framework for learning cross-lingual embeddings that, based on well-supported insights, unifies two standard frameworks: joint learning and alignment-based approaches. While I like and acknowledge the fact that the paper examines these frameworks critically and has also some didactic value, I still have some concerns regarding the current experiments, and would like to see some additional analyses in the paper. Honestly, this type of work would work better as a short ACL/EMNLP paper, as the core methodological contribution is a very simple combination of the existing techniques. With joint methods, it is true that shared words can be used as implicit bilingual learning signal which gets propagated further in the model. Even in the case of alignment-based methods, it was shown that this signal can assist in learning shared cross-lingual spaces, but: 1) such spaces are of lower-quality than the spaces learned by relying on seed dictionaries (see e.g., the work of Vulic and Korhonen (ACL 2016)), 2) this is useful only for similar language pairs which use the same script. The paper fails to provide an insightful discussion on how the scores differ when we move towards more distant languages. For instance, English-Chinese is included as a more distant language pair, and the combined method seems to work fine, but the reader is left wondering why that happens. The same is true for English-Russian. The paper should definitely provide more information and insights here. One experiment that would be quite interesting imho is to take seed dictionaries to initialise the joint training method. It will not be unsupervised any more, but it would be very useful to report the differences in results between fully unsupervised joint initialisation (which, as mentioned should work only with more similar languages) and such weakly supervised init: e.g., the method could take all one-to-one translation pairs from the seed dictionary as *shared words*. It would also be interesting to combine such *shared words* and words that are really shared (identically spelled words) as initialisation and measure how it affects the results, also in light of differences in language distance. Adding one such experiment would make the paper more interesting. Some recent strong baselines are missing: e.g., I wonder how the model that does self-learning on top of seed dictionaries (similar to the work of Vulic et al., EMNLP 2019) compares to the proposed method. Also, can we use the same self-learning technique with the combined method here? Would that lead to improved results? Another work I would like to see comparisons to is the work of Zhang et al. (ACL 2019) and despite the fact that the authors explicitly stated that they decided not to compare to the work of Artetxe et al. (ACL 2019) as they see it as concurrent work, I would still like to see that comparison as the model of Artetxe et al. seems very relevant to the presented work. I do not see the extension to contextualized representations (described in Section 3.2) as a real contribution: this is a very straightforward method to apply an alignment-based method on multilingual BERT representations, and very similar techniques have been probed in previous work (e.g., by Aldarmaki & Diab), only the bilingual signal/dictionary came from parallel data instead. Finally, as mentioned before, one of the must-have experiments is including more (distant and diverse) language pairs and analysing the results across such language pairs, with an aim to further understand how the levels of sharing, over-sharing, and non-isomorphism affect performance. Also, while the authors state that reduced isomorphism is the key problem of alignment-based methods, I still fail to see how exactly the alignment refinement step does not suffer from that problem? More discussion is needed here. Other comments: - It would be useful to also point to the survey paper of Ruder et al. (JAIR 2019) where the difference between alignment-based and joint models is described in more detail and could inform an interested reader beyond the confines of the related work section in this paper. Similarly, differences between joint models and alignment-based models have also been analysed by Upadhyay et al. (ACL 2016); Vulic and Korhonen (ACL 2016). - I like how Figure 1 clearly visualizes the main intuitions behind the proposed framework, and how mitigating the oversharing problem leads to better alignments. This was very nice. - In light of the problems with silver standard MUSE datasets (see also the recent work of Kementchedjhieva, EMNLP 2019), I would suggest to run BLI experiments on more language pairs: e.g., there are BLI datasets of Glavas et al available for 28 language pairs.