Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
Authors: Zirui Wang*
Jiateng Xie*
Ruochen Xu
Yiming Yang
Graham Neubig
Jaime G. Carbonell
Venue:  (2020)
Keywords: Cross-lingual Representation

Abstract: Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly. In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse cross-lingual tasks. Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent. Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches. Extensive experiments demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We further show that this framework can generalize to contextualized representations such as Multilingual BERT, and produces state-of-the-art results on the CoNLL cross-lingual NER benchmark.

Published as a conference paper at ICLR 2020
CROSS-LINGUAL ALIGNMENT VS JOINT TRAINING: A COMPARATIVE STUDY AND A SIMPLE UNIFIED FRAMEWORK
Zirui Wang; Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, Jaime Carbonell Language Technologies Institute, Camegie Mellon University, Pittsburgh, PA 15213, USA fziruiw, jiatengx, zuochenx, yiming, gneubig, igc} cmu, edu
ABSTRACT
Learning multilingual representations of text has proven successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training. which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly. In this paper, we first conduct direct compar- isons of representations learned using both of these methods across diverse cross- lingual tasks. Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training task- dependent. Stemming from this analysis, we propose a simple and novel frame- approaches. Exten-
work that combines these two previously mutually-excluse sive experiments demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We further show that this framework can generalize to contextualized representations such as Multilingual BERT, and pro- duces sate-of-the-all results on the CoNLL cross-lingual NER benchmark
INTRODUCTION
Continuous word representations (Mikolov et al. [013a Pennington al. 2014 Bojanowski al. 2017) have become ubiquitous across a wide range of NLP tasks. In particular, methods for cross- lingual word embeddings (CLWE) have proven a powerful tool for cross-lingual transfer for down-
stream tasks, such as text classification (Klementiev et al. 2012). dependency parsing (Ahmad et al.)
2019) named entity recognition (NER) (Xie et al. 20188] Chen al. 2019), natural language
ference (Conneau et al. 2018b), language modeling (Adams et al. (017)) and machine translation
(MT) (ou al. 2013 Lample et al., 20188a] Artetxe et al.) 2018b Lample al. 2018b). The goal
of these CLWE methods is to learn embeddings in a shared vector space for two or more languages.
There are two main paradigms for learning CLWE: cross-lingual alignment and joint training.
The most successful approach has been the cross-lingual embedding alignment method (Mikolov) let al., 2013b), which relies on the assumption that monolingually-traing continuous word embed- ding spaces share similar structure across different languages. The underlying idea is to first inde- pendently train embeddings in different languages using monolingual corpora alone, and then learn mapping align them to shared vector space. Such mapping be trained in supervised fashion using parallel resources such as bilingual lexicons (Xing et al. 2013] Smith et al.) [017] distribu- tion matching (Zhang et al. 201 Conneau et al., 2018a| Artetxe et al. 2018a Zhou et al. 2019) Recently, it has been shown that alignment methods also be effectively applied to contextualized
Joulin et al.] 8018b| Jawanpuria et al. 2019), even in unsupervised manner based
word representations (Schuster et al. 2019 Aldarmaki & Diab 2019).
"quual contribution. Ordering determined by dice rolling.
of the time of submission Sept 25, 2019. Source code available at htt.pa://githubb.oo
t thespect rewi thin/joint inn
"In this paper, "supervision" refers that provided by parallel corpus bilingual dictionaries.

---

Published as a conference paper at ICLR 2020
Another successful line of research for CLWE considers joint training methods, which optimize monolingual objective predicting the context of a word in a monolingual corpus along with either hard soft cross-lingual constraint. Similar to alignment methods, some early works rely on bilingual dictionaries (Ammar et al. 2016; Duong et al. 2016) or parallel corpora [Luong et al.) 2013 Gouws et al.] 2013) for direct supervision. More recently, seemingly naive wnsupervised joint training approach has received growing attention due to its simplicity and effectiveness. In par- ticular, [ample et al. (0188) reports that simply training embeddings on concatenated monolingual corpora of two related languages using a shared vocabulary without any cross- tross-lingual resources is able to produce higher accuracy than the more sophisticated alignment methods on unsupervised MT tasks. Besides, for contextualized representations, unsupervised multilingual language model pretraining using a shared vocabulary has produced state-of-the-an results on multiple benchmark: (Devlin et al. 2019: Artetxe & Schwenk 2019; Lample & Conneau 2019).
Despite a large amount of research on both alignment and joint training. previous work has neither performed systematic comparison between the two, analyzed their pros and cons, nor elucidated when we may prefer one method over the other. Particularly, it's natural to ask: (1) Does the phenomenon reported in Lample et al. (2018b) extend to other cross- loossllnngual tasks? (2) Can we em- ploy alignment methods to further improve unsupervised joint training? (3) If so, how would such a framework compare to supervised joint training methods that exploit equívalent resources, i.e., bilin- gual dictionaries? (4) And lastly, can this framework generalize to contextualized representations?
In this work, we attempt to address these questions. Specifically, we first evaluate and compare alignment versus joint training methods across three diverse tasks: BLI, cross-lingual NER, and un- supervised MT. We seek characterize the conditions under which one approach outperforms the other, and glean insight on the reasons behind these differences. Based on our analysis, we further propose a simple, novel, and highly generic framework that uses unsupervised joint training as ini- tialization and aligmment as refinement to combine both paradigms. Our experiments demonstrate that our framework improves over both alignment and joint training baselines, and outperforms exist- ing methods on the MUSE BI.I benchmark. Moreover, we show that our framework can generalize to contextualized representations such as Multilingual BERT, producing state-of-the-art results on the CoNLL cross-lingual NER benchmark. To the best of our knowledge, this is the first framework that combines previously mutually- aligmment and joint training methods.
BACKGROUND: CROSS REPRESENTATIONS
Notation. We assume we have two different languages {L, L23 and access to their corresponding training corpora. We use Vz, [w[,))= to denote the vocabulary set of the ith language where each wL,, represents a unique token, such as a word or subword. The goal is to learn set of em- beddings E fa> }-= with x E Rdd in a shared vector space, where each token w is mapped to vector in E. Ideally, these vectorial representations should have similar values for tokens with similar meanings or syntactic properties, so they can better facilitate cross-lingual transfer.
2.1 ALIGNMENT METHODS
Given the notation, alignment methods consist of the following steps:
Step 1: Train an embedding set Eg = Er, U Er.g, where each subset Er. = 1 is trained independently using the ith language corpus and contains an embedding æ²., for each token wf.-- Step 2: Obtain a seed dictionary D = {(wi,, w/ ) either provided or learnt unsupervised. Step 3: Learn projection matrix W E Raxd based D, resulting in final embedding set
EA = (W. . FI,) U Ft., in a shared vector space.
To find the optimal projection matrix W. Mikolov et al. 2013b) proposed to solve the following optimization problem:
min W WeBaxd
[WX Xıallr
.dd
2

---

Published as a conference paper at ICLR 2020
where XL, and L2 matrices of size d x K containing embeddings of the words in D. Xing et al. (2015) later showed further improvement could be achieved by restricting W to an orthogonal matrix, which turns the Eq. [ into the Procrustes problem with the following closed form solution:
W = =VV²
with UEV* = =VV((X,,X))
(3)
where W* denotes the optimal solution and SVD(-) stands for the singular value decomposition.
As surveyed in Section 5 different methods (Smith et al. 2017 Conneau et al. 201 8a, Joulin et al., 20186 Artetxe al., 8a) differ in the way how they obtain the dictionary D and how they solve for W in step 3. However, most of them still involve solving the Eq. as a crucial step.
2.2 JOINT TRAINING METHOSS
Joint training methods in general have the following objective:
c-= L2 + R(L1, L2)
(4)
where L1 and L2 are monolingual objectives and R(L,1 I2) is cross-lingual regularization term. For example, Klementiev et al. (2012) use language modeling objectives for L1 and L2. The term R(L1, L2) encourages alignment of representations of words that are translations. Training an em- bedding set E, EL, U L, is usually done by directly optimizing L..
While supervised joint training requires access to parallel resources, recent studies (Lample al. 2018b Devlin et al. 22199; Artette Schwenk] 2019; Lample & Conncau 2019) have suggested that unsupervised joint training without such resources also effective. Specifically, they show that the cross-lingual regularization term R[,,, L2) does not require direct cross-lingual supervision achieve highly competitive results. This because the shared words between Lı and L2 can serve implicitly as anchors by sharing their embeddings to ensure that representations of different languages lie shared space. Using our notation, the unsupervised joint training approach takes the following steps:
Step 1: Construct a joint vocabulary V,, V2, VV, that shared across two languages.
Step 2: Concatenate the two training corpora and learn embedding set F, corresponding to V,.
The joint vocabulary is composed of three disjoint sets: VJ. V2, Vj. where Vj Vz, n V2, the shared vocabulary set and Vj is the of tokens that appear in the ith language only. Note that key difference of existing supervised joint training methods is that embeddings corresponding to Vi are not shared between EL, and EL,, meaning that they are disjoint, as in alignment methods.
2.3 DISCUSSTON
While alignment methods have had great success, there are still some critical downsides, among which we stress the following points:
1.
While recent studies in unsupervised joint training have suggested the potential benefits of word sharing, alignment methods rely two disjoint sets of embeddings. Along with some possible loss of information due to no sharing, one consequence is that finetuning the aligned embeddings on downstream tasks may be sub-optimal due to the lack of cross- lingual constraints at the finetuning stage, whereas shared words can fulfill this role in jointly trained models.
2. A key assumption of alignment methods is the isomorphism of monolingual embedding spaces. However, some recent papers have challenged this assumption, showing that it does not hold for many language pairs (Søgaard et al.) 2018 Patra et al., 2019). Also notably, Ormazabal al. (2019) suggests that this limitation results from the fact that the two sets of monolingual embeddings are independently trained.
On the other hand, the unsupervised joint training method is much simpler and doesn't share these disadvantages with the alignment methods, but there are also some key limitations:
3

---

Published as a conference paper at ICLR 2020
Une peraled p-4t
Waag vand jal
Tedneran:
Align
(8]
(6)
Figure 1: PCA visualization of English and Spanish embeddings learnt by unsupervised joint train- ing as in Lample et al. (2018b). As shown by plots (a) and (b), most words are shared in the initial embedding space but well-aligned, hence the oversharing problem. Plots (b) and (c) shows that the vocabulary reallocation step effectively mitigates oversharing while the alignment refinement step further improves the poorly aligned embeddings by projecting them into a close neighborhood.
1. It assumes that all shared words across two languages serve implicitly as anchors and thus need not be aligned other words. Nonetheless, this assumption is not always true, leading to misalignment. For example, the English word "the" will most likely also appear in the training corpus of Spanish, but preferably it should be paired with Spanish words such as in potentially less accurate alignments, especially for words that are not shared.
"el"" and "la" instead of itself. We refer to this problem oversharing.
2. It does not utilize any explicit form of seed dictionary as in alignment methods, resulting
Lastly, while the supervised joint training approach does not have the same issues of unsupervised joint training, it shares limitation I of the alignment methods.
We empirically compare both joint training and alignment upproaches in Section 4 and shed light on some of these pros and cons for both paradigms (See Section 4.3.1)
PROPOSED FRAMEWORK
Motivated by the pros and cons of both paradigms, we propose unified framework that first uses unsupervised joint training as a coarse initialization and then applies alignment methods for refine- ment, as demonstrated in Figure [ Specifically, shared vocabulary through unsupervised joint training, so as to alleviate the limitations of alignment methods. Next, use vocabulary reallocation technique mitigate oversharing, before finally resorting back to alignment methods to further improve the embeddings' quality. Lastly, we show that this framework can generalize to contextualized representations.
first build single set of embeddings with
3.1 UNIFYING ALIGNMENT WITH JOINT TRAINING
Our proposed framework mainly involves three components and we discuss each of them as follows.
Joint Initialization. We use unsupervised joint training (Lample et al. (0188) train the initial CLWE. As described Section we first obtain a joint vocabulary Vy and train its corresponding of embeddings F, the concatenated corpora of two languages. This allows single set of embeddings that maximizes sharing across two languages. To train embeddings, we used fastTex(" (Bojanowski al. 2017) in all our experiments for both word and subword tokens.
obtain
Vocabulary Reallocation. As discussed in Section 2.3
key issue of unsupervised joint training oversharing, which prohibits further refinement as shown in Figure [ To alleviate this drawback, attempt to "unshare" some of the overshared words, their embeddings be better aligned in the next step. Particularly, we perform a vocabulary reallocation step such that words appearing

---

Published as a conference paper at ICLR 2020
mostly exclusively in the ith language reallocated from the shared vocabulary Vi to Vj, whereas words that appear similarly frequent both languages stay still in Vj. Formally, for each token w in the shared vocabulary Vj, use the ratio of counts within each language to determine whether it belongs to the shared vocabulary:
Tıo CL (w) Tn CL2 (w))
(5)
where CL, (w) is the count of w in the training corpus of the ith language and T2, = E CL,(w) the total number of tokens. The token w is allocated the shared vocabulary if
1 <r 7 y
(6)
where y is hyper-parametts Otherwise, put into either VJ or V7, where it appears mostly frequent. The above process generates three new disjoint vocabulary sets V VÝ Vj and their corresponding embeddings EJ Ey Eÿ that are used thereafter. Note that, V = VJ and E'j = E,-
Alignment Refinement. The unsupervised joint training method does not explicitly utilize any dictionary or form of alignment. Thus, the resulting embedding set is coarse and ill-aligned in the shared vector space, as demonstrated in Figure [ As a final refinement step, we utilize any off-the-shelf' alignment method to refine alignments across the non-sharing embedding sets, i.e. mapping E to E and leaving Eý untouched. This step could be conducted by either supervised or unsupervised alignment method and we compare both in our experiments.
3.2 EXTENSION TO CONTEXTUALIEES REPRESENTATIONS
As our framework is highly generic and applicable to any alignment and unsupervised joint train- ing methods, it can naturally generalize to contextualized word representations by aligning the fixed outputs of a multilingual encoder such as multilingual BERT (M-BERT) (Devlin et al. (019). While our vocab reallocation technique is longer necessary as contextualized representations are depen- dent on context and thus dynamic, we can still apply alignment refinement on extracted contextual- ized features for further improvement. For instance, as proposed by Aldarmaki Diab (019), one method to perform alignment on contextualized representations is to first use word alignment pairs extracted from parallel corpora dictionary. learn an alignment matrix W based on it, and apply W back to the extracted representations. To obtain W, we can solve Eq. [1] as described in Sec- tion 2.T where the embedding matrices XL, and X now contain contextualized representations of aligned word pairs Note that this method is applicable to fixed representations but not finetuning.
EXPERIMENTS
We evaluate the proposed approach and compare with alignment and joint training methods on three NLP benchmarks. This evaluation aims to: (1) systematically compare alignment vs. joint training paradigms and reveal their pros and cons discussed in Section 2.33 (2) show that the proposed frame- work can effectively alleviate limitations of both alignment and joint training, and (3) demonstrate the effectiveness of the proposed framework in both non-contextualized and contextualized settings.
4.1 EVALUATION TASKS
Bilingual Lexicon Induction (BLI) This task has been the de facto evaluation task for CLWE methods. It considers the problem of retrieving the target language translations of source langauge words. We use bilingual dictionaries complied by [Cnneau et al. (2118) and test on six diverse language pairs, including Chinese and Russian, which use different writing script than English. Each test set consists of 1500 queries and we report precision at / scores (P@) 1), following standard evaluation practices (Conneau et al. 2018a Glavas et al.) 2019).
Name Entity Recognition (NER) We also evaluate our proposed framework on cross- cross-lingual NER, sequence labeling task, where we assign label to each token in sequence. We evaluate both
5

---

Published as a conference paper at ICLR 2020
codl
Iron
de-es
ca-ch
ebean
Aligemen Metheds
81.7
85.3
82.3
X2.1
74.0
72.0
2222
762
4400
35.1
12.5
31.4
66.5
(2) VECMAY
Aute X0
K2.5
86.7
92.3
85.6
75.1
74.5
49.2
65.6
0.0
0.0
(3) DcMM-BBB
2013
82.8
94.9
13.1
83.5
72.2
74.4
49.2
6.57
42.5
37.9
81.4
81.1
82.4
73.5
72.4
77.5
77.9
51.7
03.7
42.7
36.7
GaaMM
74.7
77.9
80.0
51.3
45.3
Ponp
78.5
57.9
(7) RCSLS
Zhzzg 21 20:9119 2019
83.9
78.1
79.3
57.9
48.6
Joent Trung
Unappersionl Irim
16.6
42.2
47.4
19.5
41.4
16.8
18.8
4.0
17.9
29.3
(9) Supervisci Iiitt Donng otall 2016
76.7
79.8
7%.1
76.7
67.5
66.9
74.4
74.1
41.8
51.8
46.7
45.3
65.2
(10) Joant Replace
48.2
47.7
49.4
52.1
46.5
46.9
43.8
45.8
20.3
36.6
32.7
34.1
42.0
Join Align Framewors
(11) Jlig (win AR)
55.9
62.8
61.8
67.0
45.1
54.6
50.2
51.4
8.7
8.2
19.4
18.2
42.3
(12) IsiatAlign MUSE
81.4
84.2
82.8
93.6
74,2
72.2
77.5
91.5
45.0
58.3
36.1
35.3
67.2
JouatAlign RCSLS V8)
34.2
34.0
35.6
35.3
7.7
5.2
2910
Aliga GaaMM
$4.2
78.2
81.4
52.4
50.4
(15) Jomt,Align RCSLS
79.6
80.6
59.8
Table 1: Precision@1 1 for the BLI task on the MUSE datase Within each category, unsuper- vised methods are listed at the top while supervised methods are at the bottom. The best result for unsupervised methods underlined while bold signifies the overall best. "IN" refers to iterative normalization proposed in Zhang et al. (2019), "AR" refers to alignment refinement and "VR" refers to vocabulary reallocation.
non-contextualized and contextualized word representations on the CoNLL 2002 and 2003 bench- marks (Tjong Kim Sang 2002; Tjong Kim Sang & De Meulder 2003), which contain 4 European languages. To measure the quality of CLWE, we perform zero-shot cross-lingual classification, where we train a model on English and directly apply it to each of the other 3 languages.
Unsupervised Machine Translation (UMT) Lastly, we test our approach using the unsupervised MT task, on which the initialization of CLWE plays a crucial role (Lample et al. 2018b). Note that our purpose here is to directly compare with similar studies in Lample et al. (b)88)) and thus we follow their settings and consider two language pairs, English- French and Germma,, and evaluate on the widely used WMT' 14 en-fr and WMT' 16 en-de benchmarks.
4.2 EXPERIMENTAL SETUP
For the BLI task, we compare framework recent state-of-ther ofr methods. We obtain numbers from the corresponding papers or Zhou et al. (2019), and use the official tools for MUSE (Conneau et al., 2018a), GeoMM (Jawanpuria et al. 2 and RCSLS (Joulin et al. 2018b) to obtain miss- ing results. We consider the method of Duong et al. (2016) for supervised joint training based on bilingual dictionaries, which comparable used. For unsupervised joint training, we train uncased joint fastText word vectors of dimension 300 on concatenated Wikipedia corpora of each language pair with default parameters. The hyper- parameter y is selected from {0.7, 0.8, 0.9, 0.95) on validation sets. For the alignment refinement step in our proposed framework, we use RCSLS and GeoMM to compare with supervised meth- ods, and MUSE for unsupervised methods. In addition, we include an additional baseline of joint training, denoted as Joint - Replace, which is identical to unsupervised joint training except that utilizes seed dictionary to randomly replace words with their translations in the training corpus. Following standard practices, we consider the top 200k most frequent words and use the cross- domain similarity local scaling (CSLS) (Conneau al. 2018a) the retrieval criteria. Note that concurrent work (Artetxe et al., 2019) proposed a new retrieval method based on MT systems and produced of--ee---th--rrr art results. Although their method is applicable to framework, it has high computational cost and is out of the scope of this work.
supervised alignment methods in terms of resources
For the NER task: (1) For non-contextualized representations, train embeddings the same way as in the BII task and use a vanilla Bi-LSTM-CRR model (Lample et al. 2016). For all alignment steps, we apply the supervised Procrustes method using dictionaries from the MUSE library for simplicity. (2) For contextualized representations, we use M-BERT, an unsupervised joint training model, as
We found that the official evaluation script of MUSE ignores test pairs that are of vocabulary (OOV) on the source or the target sides, which could occur occasionally when evaluating our model due to the proposed vocabulary reallocation step. To ensure fair comparison, include these O0V pairs. Specifically, if the source word is OOV, we retrieve itself; otherwise, we count the pair incorrect. Please appendix B for details.

---

Published as a conference paper at ICLR 2020
fr-en
zh:50
Unsupervised
MUSE Conneot
77.1
R2.5
76.4
78.0
67.4
67.8
72.5
77.5
42.7
5012
25.7
29.1
3,7
8.5
Align
Supervised
RCSLS spl POBM
68.2
(5) Superviest maa Daaing 2016
60.1
61.7
69.7
762
41.0
51.8
(6) JoiggAlign RUSES
$2.1
84.6
78.1
80.4
68.4
70.4
73.7
79.0
59.0
66.8
51.6
45.7
70.0
Table 2: Precision@) I for the BLI task on the MUSE dataset with test pairs of same surface form removed. The best result for unsupervised methods is underlined while bold signifies the overall best.
our base model and apply our proposed framework on it by first aligning its extracted features and then feeding them to a task-specific model (M-BERT Feature Align). Specifically, we use the sum of the last 4 M-BERT layers' outputs as the extracted features. To obtain the alignment matrices, one for each layer, we use 30k parallel sentences from the Europarl corpus for each language pair and follow the procedure of Section 3.2 We feed the extracted features inputs to task-specific model with 2 Bi-LSTM layers and a CRF layer (see appendix A We compare our framework to both finetuning (M-BERT Finetune), which has been studied by previous papers, and feature extraction (M-BERT Feature). Lastly, we also compare against XLM, a supervised joint training model.
For the UMT task,
the exact same data, architecture and parameters released by Lample al. We simply use different embeddings trained with the same data as inputs to the model.
2018b
4.3
RESULTS AND ANALYSIS
4.3.1
ALIGNMENT VS. JOINT TRAINING
We compare alignment methods with joint training on all three downstream tasks. As shown in Table and Table 3 find alignment methods significantly outperform the joint training approach by large margin in all language pairs for both BLI and NER. However, the unsupervised joint training method is superior than its alignment counterpart on the unsupervised MT task demonstrated in While these results demonstrate that their relative performance is task-dependent, we conduct further analysis reveal three limitations as discussed in See 2.3
1
2(c)
First, their poor performance BLI and NER tasks shows that unsupervised joint training fails to generate high-quality alignments due to the lack of fine-grained seed dictionary as discussed in its limitation 2 To evaluate accuracy on words that are not shared, we further remove test pairs of the same surface form (e.g. (hate, hate) as a test pair for en-de) of the BLI task and report their results in T We find unsupervised joint training (row 2) to achieve extremely low scores which shows that emebddings of non-shared parts are poorly aligned, consistent with the PCA visualization shown in Figure
Moreover, we delve into the relative performance of the two paradigms on the MT task by plotting their test BLEU scores of the first 20 epochs in Figure 2(a) and 2(b) We observe that the alignment method actually obtains higher BLEU scores the first few epochs, but gets surpassed by joint training in later epochs. This shows the importance of parameter sharing as discussed in limitation [ of alignment methods: shared words can be used as a cross- cross-lingual constraint for unsupervised joint training during fine-tuning but this constraint cannot easily be used in alignment methods. The lack of sharing is also limitation for the supervised joint training method, which performs poorly on the MT task even with supervision as shown in Figure 2(c)
Lastly, we demonstrate that oversharing can be sub-optimal for unsupervised joint training as dis- cussed in its limitation 2 Specifically, we conduct ablation studies for our framework in Table [ Applying alignment refinement on unsupervised joint training without any vocabulary reallocation does not improve its performance (row 13). On the other hand, simple vocabulary reallocation alone boosts the performance by quite margin (row 11). This shows some words are shared erroneously across languages in unsupervised joint training, thereby hindering its performance. addition, while utilizing seed dictionary (row 10) improves the performance of unsupervised joint train-
sedM!

---

Table F1 score for the cross- Published as a conference paper at ICLR 2020
05
n]
dc
lingual NER task. "Adv" refers adversarial training. i de- notes results that are not directly
contextualized
Unsupervised Joint
50.28
42.77
21.49
38.18
Supervised Joint (Duong et al.
2016
63.16
63.60
36.24
54.33
Joint Replace
65.28
68.44
51.59
61.77
Align
69.00
71.33
52.17
64.17
comparable due to different re-
Align al.
70.46
72.
56.47
66.34
sources and architectures used.
71.67
70.90
57.43
66.67
denotes supervised XLM model
et
2019
73.50
72:40
56.00
trained with MLM and TLM ob- jectives. Its Dutch (nl) result is blank because the model is not
Contextualized
(aampe Conneau
2019
63.18
M.BERT Finetine
Pires of al 2019
73.59
77.36
69.74
73.56
M-BERT Finetune
Wi Dredz 2019
74.96
77.57
69.56
74.03
pretrained on it. Bold signifies
M-BERT Finetune
Keung
2019
75.00
77.50
68.60
73.70
M-BERT Finetune + Adv Keung g cf
2019
74.30
77.60
71.90
74.60
state-of-the-ae therrr results. We re- port the average of 5 runs.
M-BERT Feature
74.23
78.65
67.63
73.50
M BERT Feature + Align
75.77
79.03
70.54
75.11
ing, it still suffers from the oversharing problem and performs worse compared to supervised joint training (row 9).
4.3.2 EVALUATION OF PROPOSED FRAMEWORK
As shown in Table 1 Table 3 and Figure 2 our proposed framework substantially improves over the alignment and joint training baselines on all three tasks. In particular, outperforms existing methods on all language pairs for the BLI task (using the CSLS as retrieval metric) and achieves state-of-the-art results on out of language pairs for the NER task. Besides, we show that alleviates limitations of alignment and joint training methods shown in the previous section.
First, the proposed framework largely improves the poor alignment of unsupervised joint training, especially for non-sharing parts. As shown in Table the proposed Joint_Align framework achieves comparable results to prior methods the unsupervised case (row 12) and it outperforms previ- state-of-the-are methods in the supervised setting (row 15). Specifically, our proposed frame- work can generate well-aligned embeddings after alignment refinement is applied to the initially ill-aligned embeddings, demonstrated in Figure [ This is further verified by results in Table 2 where our proposed framework largely improves accuracy on words not shared between two lan- guages over the unsupervised joint training baseline (row and row 2).
Besides, our ablation study in Table T further shows the effectiveness of the proposed vocabulary reallocation technique, which alleviates the issue of oversharing. Particularly, we observe no im- provement compared to unsupervised joint training baseline (row 8) when an alignment refinement step is used without vocabulary reallocation (row 13), while a vocabulary reallocation step alone significantly improves the performance (row 11). This is consistent with Figure [ and shows that the oversharing is bottleneck for applying alignment methods to joint training. also suggests detecting what to share is crucial to achieve better cross-lingual transfer.
Lastly, while supervised joint training shares the limitation | of alignment methods and performs poorly when finetuned, proposed framework exploits the same idea of vocabulary sharing used in unsupervised joint training. In the MT tasks, our framework obtains a maximum gain 2.97 BLEU over baselines we ran and consistently performs better than results reported in Lample et al. (20 Sb). In addition, Figure? 2 shows that Joint Align not only converges faster earher training epochs but also consistently outperforms the two baselines thereafter. These empirical findings demonstrate the effectiveness of our proposed methods in the non-contextualized case.
4.3.3 CONTEXTUALIZED WORD REPRESENTATIONS
As can be seen in Table 3 when using our framework (M-BERT Feature + Align), we achieve state-of-the-ar results on cross-lingual NER on 2 out of 3 languages and the overall average. This shows that our framework can effectively generalize to contextualized representations, Specifically, our framework improves over both the M-BERT feature extraction and finetuning baselines on all three language pairs. However, when compared to con-contextualized results, the gain of using alignment refinement on top of unsupervised joint training is much smaller. This suggests that, the contextualized unsupervised joint training model performs very well already even without
8

---

Published as a conference paper at ICLR 2020
en-fi
fr-ea
ea-de
de-en
Unsupervised Joint' Join'
25.14
24.18
17.16
21.00
Align'
22,00
21.30
Unsupervised Joint
23.68
23.00
15.04
19.21
Align
20.01
19.09
12.86
17.05
JointAligg
25.74
25.24
17.20
22.18
Supervised Joint
17.54
16.90
11.02
13.88
Epoch
(a) fi-en
(b) en-de
(c)
Figure 2: (a)(b): Results on MT of Align, Joint and our framework for the first 20 training epochs. Results after 20 epochs have similar patterns, (c): BLEU scores for the MT task. Results evaluated on the WMT 14 English-French and WMT 16 German-English. All training settings are the same for each language pair except the embedding initialization. Note that we are not trying to outperform state-of-ther aht methods (Song et al.) (2019) but rather to observe improvements afforded by embedding initialization. 'Results reported by Lample et al. 8b).. Our results are obtained using the official code released by the author. Duong al. (2016) is supervised method that
include for analysis purpose only and not directly comparable to other results in this table.
any supervision, it is harder achieve large improvements. While alignment refinement relies on word alignment, a process that is noisy itself, a better alignment approach may be warranted. Lastly, the reason why supervised joint training model, XLM, performs worse than its unsupervised counterpart, M-BERT, is likely that XLM uses an uncased vocabulary, where casing information is important for NER tasks.
RELATED WORK
Word embeddings (Mikolov et al., 2013a, Ruder et al. 2019) are a key ingredient to achieving success in monolingual NLP tasks. However, directly using word embeddings independently trained for each language may cause negative transfer (Wang et al. 2019) in cross pooss-lingual transfer tasks. In
order capture the cross-lingual mapping. a rich body of existing work relying on cross- looss-l
supervision, including bilingual dictionaries (Mikolov et al.) 2013a Faruqui & Dyer 2014 Artetxe al. 2016: Xing al. 2015 Duong et 2016 Gouws Søgaard 2015
al. 201 8a),
sentence-alignee corpora (Kočiský et al., 2014 Hermann & Blunsom 2014 Gouws et al. 2015)
and document-alii alligne corpora (Vulié Moens 2016 Søgaard et al. 2015).
Besides, unsupervised alignment methods aim to eliminate the requirement for cross-lingual su- pervision. Early work CCao et al. (2016) matches the mean and the standard deviation of two embedding spaces after alignment. [Barone (2016); Zhang et al. (201 b); Conneau et al. (2018)) Sa) adapted a generative adversarial network (GAN) (Goodfellow et al.) 22144] to make the distribu- tions of two word embedding spaces indistinguishable. Follow-up works improve upon GAN-based training for better stability and robustness by introducing Sinkhorn distance (Xu et al., 2018), 3 by stochastic self-training AAtetxe et al. (2188)) or by introducing latent variables (Dou al. 2018).
While alignment methods utilize embeddings trained independently on different languages, joint training methods train word embeddings at the same time. Klementiev et al. (2012) train bilin- gual dictionary-baas based regularization term jointly with monolingual language model objectives while Kočiský al. (2014) defines the cross- cross-lingual regularization with the parallel corpus. Another branch of methods (Xiao & Guo, 2014; Gouws & Søgaard) 2015): Ammar et al,, D et al.) 2016) build pseudo-bilingual corpus by randomly replacing words in monolingual corpus with their translations and use monolingual word embedding algorithms to induce bilingual represen- tations. The unsupervised joint method by Lample & Conneau (2019) simply exploit words that share the same surface form as bilingual "supervision" and directly train shared set of embedding with joint vocabulary. Recently, unsupervised joint training of contextualized word embeddings through the form of multilingual language model pretraining using shared subword vocabularies has produced state-of-ther -ftthe-r results on various benchmarks (Devlin et al. 2019 Artetxe Schwenk. 2019 Lample & Conneau 2019 Pires et al. 2019 Wu & Dredze 2019).

---

Published as a conference paper at ICLR 2020
A concurrent work by Ormazabal al. (2019) also compares alignment and joint method in the bilingual lexicon induction task. Different from their setup which only tests on supervised settings, conduct analysis across various tasks and experiment with both supervised and unsupervised conditions. While Ormazabal et al. (2019) suggests that the combination of the alignment and joint model could potentially advance the state-of-art of both worlds, we propose such framework and empirically verify its effectiveness on various tasks and settings.
6 CONCLUSION
In this paper, we systematically compare the alignment and joint training methods for CLWE. We point out that the nature of each category of methods leads certain strengths and limitations. The empirical experiments on extensive benchmark datasets and various NLP tasks verified our analysis. To further improve the state-of-art of CLWE, we propose a simple hybrid framework which combines the strength from both worlds and achieves significantly better performance in the BLI, MT and NER tasks. Our work opens promising new direction that combines two previously exclusive lines of research. For future work, an interesting direction is to find a more effective word sharing strategy.
This research was sponsored by Defense Advanced Research Projects Agency Information Innovation Office (12O) under the Low Resource Languages for Emergent Incidents (LORELEI) program, issued by DARPA/120 under Contract No. HR0011-15-C0114 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. gov- ermment is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation here on.
Acknowledgments:
REFERENCES
Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pa- pers, pp. 937-947, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https: www ac www.ww.wwbbb.... org/ antho www.thhooooyyyye.o ogt/e1////1//1
Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai- Wei Chang, and Nanyun Peng. On difficulties of cross- lingual transfer with order differences: A case study on dependency pars- ing. In Meeting of the North American Chapter of the Association for Computational Linguisties (NAACL), Minneapolis, USA, June 2019.
wwwpexiii.rrraaaoo /as//////////oooo/a///.///eaaeer 1811 00570
Hanan Aldarmaki and Mona Diab. Context-aware cross-lingual mapping. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACI), Minneapo- lis, USA, June 2019. URL https:/tt:://// /tatps://arxiv.. org/ 1903. 03243
Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. Massively multilingual word embeddings. arXiv preprint 16020:160..01925, 2016.
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019.
Mikel Artetxe, Gorka Labaka, and Encko Agirre. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.
2016.
Mikel Artetxe, Gorka Labaka, and Encko Agirre. A robust self- method for fully unsuper- vised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 789-798, 2018a.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In International Conference on Learning Representattions 2018b. URL https: //openrvvis ew. htt.:///oorrmmmmm/ysleeleeeeeeee
10

---

Published as a conference paper at ICLR 2020
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Bilingual lexicon induction through unsupervised machine translation. In Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, pp. 5002-5007, 2019.
Antonio Valerio Miceli Barone. Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders. In Proceedings of the 1st Workshop on Representation Learning for NLP. pp. 121-126, 2016.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Trunsactions of the Association for Computational Linguistics, 5:135-146,
2017.
Hailong Cao, Tiejun Zhao, Shu ZHANG, and Yao Meng. A distribution-based model learn bilingual word embeddings. In Proceedings of COLING 2016, the 26th International Confer- ence on Computational Linguisties: Technical Papers, pp. 1818-1827, Osaka, Japan, Decem- ber 2016. The COLING 2016 Organizing Committee. URL. https:/ //ww:///wwwww/// ac org/
anthoology/C16-1111
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. Multi- source cross-lingual model transfer: Learning what share. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pp. 12, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 00.18663/11/P11-1299 URL
https: www. ac www.wwe...oroo.. www../ee.brr//acooooggo///////// 1-911999
Alexis Conneau, Guillaume Lample, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learming Representations (ICLR), 2018a.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2018b.
Jacob Devlin, Ming- Bing-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Zi-Yi Dou, Zhi-Hao Zhou, and Shujian Huang. Unsupervised bilingual lexicon induction via la- tent variable models. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 621 621-626, 2018
Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. Learning crosslingual word embeddings without bilingual corpora. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1285-1295, 2016
Chris Dyer, Victor Chahuneau, and Noah Smith. A simple, fast, and effective reparameterizatiio of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 648, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 462-471. 2014.
Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. How to (properly) evaluate cross- lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. arXiv preprint arXiv:1902.00500 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 267222680,.20 2014.

---

Published as a conference paper at ICLR 2020
Stephan Gouws and Anders Søgaard. Simple task-specific bilingual word embeddings. In Proceed- ings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1386- 2015.
Stephan Gouws, Yoshua Bengio, and Greg Corrado. Bilbowa: Fast bilingual distributed representa- tions without word aligmments. In International Conference on Machine Learning, pp. 748-756, 2015.
Karl Moritz Hermann and Phil Blunsom. Multilingual models for compositional distributed seman- tics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 58-658886 28-68,222 2014.
Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukuttan, and Bamdev Mishra. Learning mul- tilingual word embeddings in latent metric space: a geometric approach. Transactions of the Association for Computational Linguistics, 7:107-120, 2019.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2979-2984, Brussels, Belgium, October-November 2018a. Association for Computational Linguistics. URL
https: wwwwwwa.. www.wblwwwbbb..... org/ hwt.a//www..../.atoooo/o////////
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in trans-
lation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2979-2984, 20.8b.
Phillip Keung, Yichao Lu, and Vikas Bhardwaj. Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and ner. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), November 2019. URL https:
7/arxiv. www.wvorr.g..oo.. abs/ / 909 00153
Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. Inducing crosslingual distributed repre- sentations of words. In Proceedings of COLING 2012, pp. 1459-1474 1474, Mumbai, India, Decem- ber 2012. The COLING 2012 Organizing Committee. URL https::///wett http://wwwwwb.bb..////.//// org/
anthology/CI2-I11
Tomáš Kočiský, Karl Moritz. Hermann, and Phil Blunsom. Learning bilingual word representations by marginalizing alignments. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 224-229, 2014.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. In Proceedings of NeurlPS, 2019.
2019.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Kevin Knight, Ani Nenkova, and Owen Rambow (cds.). NAACL, pp. 260-270. The Association for Computational Linguis- tics, 2016. ISBN 978-1-941643-911 URL http:/ pttt:://acl/ee /ww/web......./...//.. org/ http//ttoooo///////G/ N16-1030.1 pdf
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations, 201 8a. URL htt.:tts//ppeereiiieee htt.://wwwnneeoooooeoooott.t.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, al. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5039-5049, 2018b.
Thang Luong, Hieu Pham, and Christopher D Manning. Bilingual word representations with mono- lingual quality in mind. In Proceedings of the 1st Workshop Vector Space Modeling for Natural Language Processing, pp. 151-159, 2015.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. ICIR, 2013a.
12

---

Published as a conference paper at ICLR 2020
Tomas Mikolov, Quoe V Le, and llya Sutskever. Exploiting similarities among languages for ma- chine translation. arXiv preprint arXiv:1309.4168 2013b.
Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. Analyzing the limitations of cross-linguat word embedding mappings. arXiv preprint (eXXvv:906.050000 2019.
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In The 57th Annual Meeting of the Association for Computational Linguisties (ACL), Florence, Italy, July 2019. URLhttps:: / /www:/wwww.../..oo/ ac web. wwg. haarg//anholoooy/PPeeeo
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543, 2014.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? In The 57th Anmual Meeting of the Association for Computational Linguistics (ACL), July 2019. URL
https::/aaa http:///axxxvv...g////. attp/////////// 01502
Sebastian Ruder, Ivan Vulié, and Anders Søgaard. A survey of cross-lingual word embedding mod- els. Journal of Artificial Intelligence Research, 65:569-65 63199-63112000 2019.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of con- textual word embeddings, with applications zero-shot dependency parsing. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), Minneapo- lis, USA, June 2019. URL BtRLhttss:/// /tttpp:///axii....../// org/ /bsg//bs// 902900222 09492
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. hotthtts: http::///ppeeeeweeww net forum?ir
rlAab85gg
Anders Søgaard, Željko Agié. Héctor Martínez Alonso, Barbara Plank, Bernd Bohnet, and Anders Johannsen. Inverted indexing for cross- cooss-lingual nip. In The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference of the Asian Federation of Natural Language Processing (ACL-IJCNLP 2015), 2015.
Anders Søgaard, Sehastian Ruder, and Ivan Vulić. On the limitations of unsupervised bilin- gual dictionary induction. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 778-788, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/118-10777 1072. URL https:
7/ www. wwwwww.ww....... org/ anthology/P18 1072
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence sequence pre-training for language generation. In International Conference on Machine Learning, pp- 5926-5936. 2019.
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-independee named entity recognition. In CoNLL, pp. 1-4, 2002. doi: 10.31151111883311111
URL
https: //oii woog//...00000 301.....1.111 / 8853. 16653.111 1118877
Erik F Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: seaguugge-independee named entity recognition. In CoNLL, pp. 142-147, 2003.
Ivan Vulié and
Moens. Bilingual distributed word representations from document- aligned comparable data. Journal of Artificial Intelligence Research, 55:953-999 994, 2016.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding nega- tive transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11193-111331 1302, 2019.
Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising tross-lingull effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), November 2019. URL
09077
13

---

Published as a conference paper at ICLR 2020
Min Xiao and Yuhong Guo. Distributed word representation learning for cross-lingual depen- deney parsing. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning. pp. 119-129, 2014.
Jiateng Xie, Zhilin Yang, Graham Neubig. Noah A Smith, and Jaime Carbonell. Neural cross- lingual named entity recognition with minimal resources. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 369-379, 2018.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 1006-1011, 2015.
Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. Unsupervised cross- cross-lingual transfer of word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2465-2474, 2018.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume I, pp. 1959-1970, 2017a.
Meng Zhang. Yang Liu, Huanbo Luan, and Maosong Sun. Earth mover's 's distance minimization for unsupervised bilingual lexicon induction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1944-1945, 2017b.
Mozhi Zhang, Keyulu Xu, Ken-ichi Kawarabayashi, Stefanie Jegelka, and Jordan Boyd-Graber. Are girls neko sh';= ojo? cross-lingual alignment of non-isomorphic embeddings with iterative normalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 31803189, 2019.
Chunting Zhou, Xuezhe Ma. Di Wang, and Graham Neubig. Density matching for bilingual word embedding. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1588-1598, 2019.
Will Zou, Richard Socher, Daniel Cer, and Christopher Manning. Bilingual word embed- dings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empir- ical Methods in Natural Language Processing, pp. 1393-1398, Seattle, Washington, USA, Oc- tober 2013. Association for Computational Linguistics. URL https: / wwww www.ww.wwwbbo/g////o
anthology/D13- 1141
APPENDIX
A NER EXPERIMENT DETAILS
Here we include some additional details for the NER experiments with contextualized representa- tions:
we apply word alignment methods, such as fastal- ign (Dyer et al., 2013), on parallel data to extract word-aligned pairs for learning the align- ment matrix. As M-BERT is based on subword tokens, we use the average of the repre- sentations of all subword tokens that correspond to word as the representation for that word. For instance, assume that an English word "Resumption" is aligned to a German word "Wiederaufnahme" and they are tokenized by M-BERT as "Res". "##sumption" and "Wie", "##derr", " "##uf"" "##nahmm" respectively. Then the representation for "Resump- tion" is the average of the representations of subword tokens "Res" and "##sumption" and the same goes for "Wiederaufnahme".
Alignment As described in Section 3.2
Hyperparameters For the task- sask-specific NER model, we use a 2-layer Bi-LSTM with a hidden size of 768 followed by CRF layer. We apply dropout rate of 0.5 the input
14

---

Published as a conference paper at ICLR 2020
codi
Iren
de-ee
ebean
Aligemen Metheds
STSUUE
Jxmeses of al. K0
83.5
8266
K5.1
74.1
71.8
77.4
7038
44.1
3911
$4.1
31.6
66.3
(4) Pascuute
Samiin A017
KL.
8233
80.5
31.5
74.2
13.5
17.0
77.8
50.5
63.3
44.2
37.0
68.6
GeoMM
Diverpar 2019
82.1
86.9
82.3
851 1
75.3
77.2
78.6
82.2
51.7
67.8
90.5
45.6
72.1
Jonn
Methods
35%
(8) Supervisal Jnst Donng otal. 2016
861
786
67.3
74.6
191 (10) lont,Align + MUSE
672
49.3
5056
3.7
1935
62.4
85.0
85.5
86.7
74.6
72.9
78.2
82.6
46.1
58.
39.9
36.2
68.7
(11) Joni, Align + SC (oto
54.5
16.8
41.0
47.0
14.1
55.0
55.5
35.2
7.6
53
21.3
16.
292
(12) Align GeobIM
83.9
86.2
83.1
85.2
76.1
77.9
79.0
82.8
53.7
68.3
54.8
48.0
73.3
(13) Joia, RCSLS
87.1 88.5
84.2
86.6
$0.1
78.7
81.3
85.
61.3
08.3
99.6
50.7
76.8
Table Precision @1 1 for the BLI task on the MUSE dataset using test set produced by vocabu- lary reallocation. Within each category, unsupervised methods are listed at the top while supervised methods are at the bottom. Bold signifies the overall best results. "AR" refers to alignment refine- ment and "VR" refers to vocabulary reallocation.
and the output of the Bi-LSTM, and use Adam with default parameters and learning rate 0.0001 for optimization. We train the model for 40 epochs with a batch size of 10, and evaluate the model per 150 steps. For prediction, we feed the outputs of the Bi-LSTM that correspond to the first subword tokens each word to the CRF model. This is identical to finetuning BERT the NER task, except that in the first subword token are fed into CRF, rather than a linear layer as done in BERT.
the outputs that correspond to
B BLI TEST PAIRS
The official evaluation scrip MUSE only includes test pairs whose source words and target words both appear in their corresponding vocabularies, leaving out those that OOV on either side. Since our proposed vocabulary reallocation step modifies both the source and target vocabularies, the script may exclude some test pairs when evaluating our model. For example, the word "age" from the test pair (age, age) for en-fi could be allocated as an en (not shared) word, so it is OOV on the fr side and the pair would thus be left out by the script. As result, the total number of test pairs would be smaller, thereby changing the denominator when we calculate accuracy. To ensure fair comparison, we include these OOV pairs so the total number of test pairs stays the same. Specifically, the source word of test pair is OOV, we retrieve itself. Otherwise, we count it incorrect.
In addition, we further investigate which pairs are left out and found that the MUSE benchmark contains some noisy test data. Specifically, we find that the majority of these pairs are in the same surface form, such as (sit, sit), but many target words are not actual translations of the source words. For example, found test pairs such as {(age, age), (century, century)) for en-fr and {(mickey, mickey), (uncredited, uncredited)} for en-zh. Clearly, these are English words and should not be considered appropriate translations for French Chinese. In Table 1 mark these pairs in- correct for our framework to ensure fair comparison, while in fact our framework correctly allocates these words English. To reveal the full picture, we also conduct BLI experiments without test pairs that got left out due to vocabulary reallocation. The results are shown in Table We observe that our proposed framework obtains gain of 3.9 accuracy on average over the RCSLS bascline.
http:///tw..tttteeee/cc/e//ss//s cord_transil bord_ttraslattion.pp an
15


Reviews:
[{"review_id": "rkxAK85sKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper compares to approaches to bilingual lexicon induction, one which relies on joint training and the other which relies on projecting two languages' representations into a shared space. It also shows which method performs better on which of three tasks (lexicon induction, NER and MT). The paper includes an impressive number of comparisons and tasks which illuminate these two popular methods, making this a useful contribution. It also compares contextualized to non-contextualized embeddings.\n\nI have some questions:\n-can you goldilocks the amount of sharing in an alignment method? Put a different way, how much is performance affected if you perform alignment w/variously sized sub-partitions of the seed dictionary? Can you find a core lexicon (perhaps most common words shared cross-lingually) that will provide a good-enough alignment to joint-align with? If you were very ambitious, you could try artificially vary the amount of lexicon disjointness across languages (for camera-ready) and report how much your results are affected by incomplete overlap in translation variants.\n-for Table 1, do you have any ideas about why certain languages do better on eng--> them and others do better on them-->eng?\n-do you have any analysis exploring what is shared? wrt this sentence \"It also suggests detecting what to share is crucial to achieve better cross-lingual transfer.\"\n\nPlease address:\n- I would like more intuitive motivation for (6).\n\nSmall notes: \n-Fig. 1 font is too small, you could also remove excess axes. There's also overlap between the labeled arrow and the y-axis label btw (a) and (b).\n-\"oversharing\" should be typographically delimited as a definition in \"We refer to this problem as oversharing\"\n-typos: bottom of p4: \"Secition 2.3\"\n-\"e.g.\" or \"i.e.\" should be followed by a \",\"\n-list which languages you use on bottom of p5\n-Table 3 caption looks crazy, what happened to your spacing there?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HJeJ2SZ6FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper compares two approaches to learn cross-lingual word embeddings: cross-lingual alignment (where separately trained embeddings in different languages are mapped into a shared space) and joint training (which combines the training data in all languages and jointly learns a cross-lingual space). The authors argue that each approach has its own advantages, and propose a \"unified framework\" that essentially applies them sequentially (first train jointly, and then further align them after the necessary vocabulary reallocation).\n\nI have generally positive feelings about the paper. To be honest, I do not like the way the authors frame their work (e.g. the way the method is motivated in Section 2.3 or calling it a \"unified framework\"), but the actual method they propose does make sense, the experimental evaluation is solid, and the results are generally convincing.\n\n\nStrengths:\n\n- Good coverage of related work, including recent publications.\n\n- Thorough evaluation in 3 different tasks, much better than what is common in the area (usually limited to BLI).\n\n- The authors experiment with contextual embeddings in addition to conventional word embeddings.\n\n- Generally convincing results with relevant ablations and reasonable baselines.\n\n\nWeaknesses:\n\n- I feel that framing this as a \"unified framework\" for cross-lingual alignment and joint training is going a bit too far. The proposed method is as simple as first doing the joint training and then the cross-lingual alignment (with a special treatment for vocabulary reallocation). This is just a sequential application of two class of methods, and not what I would understand as a \"unified framework\" (which suggests some form of generalization or at least a closer interaction to me). At the same time, the only \"joint training\" that the authors explore is training regular embeddings over concatenated monolingual corpora, which as far as I know has only been explored by Lample et al. (2018), and I would not consider as a representative example of this family of methods.\n\n- It is not clear how the different vocabulary spaces are handled in BLI. My understanding is that if the query is in the source vocabulary subset the retrieval is done over the target subset, and if it is in the shared subset it is the same query word that is returned, but this is not clear at all.\n\n- I think that the issue of \"oversharing\" is magnified in the paper. I understand that this is directly connected to the reallocation step in the proposed method, and it of course makes sense to also map words that predominantly appear in a single language. However, in relation to the downstream tasks themselves, oversharing only seems relevant for BLI, where one needs to delimit the retrieval space and avoid returning the query word (which is of course its own nearest neighbor) unless it actually exists in the target language. This is connected to my previous point, and I think should be discussed in isolation.\n\n\nOther minor things to improve:\n\n- Please make Figure 2c an actual table instead of an image.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByezsH96YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new and simple framework for learning cross-lingual embeddings that, based on well-supported insights, unifies two standard frameworks: joint learning and alignment-based approaches. While I like and acknowledge the fact that the paper examines these frameworks critically and has also some didactic value, I still have some concerns regarding the current experiments, and would like to see some additional analyses in the paper. Honestly, this type of work would work better as a short ACL/EMNLP paper, as the core methodological contribution is a very simple combination of the existing techniques.\n\nWith joint methods, it is true that shared words can be used as implicit bilingual learning signal which gets propagated further in the model. Even in the case of alignment-based methods, it was shown that this signal can assist in learning shared cross-lingual spaces, but: 1) such spaces are of lower-quality than the spaces learned by relying on seed dictionaries (see e.g., the work of Vulic and Korhonen (ACL 2016)), 2) this is useful only for similar language pairs which use the same script. The paper fails to provide an insightful discussion on how the scores differ when we move towards more distant languages. For instance, English-Chinese is included as a more distant language pair, and the combined method seems to work fine, but the reader is left wondering why that happens. The same is true for English-Russian. The paper should definitely provide more information and insights here. \n\nOne experiment that would be quite interesting imho is to take seed dictionaries to initialise the joint training method. It will not be unsupervised any more, but it would be very useful to report the differences in results between fully unsupervised joint initialisation (which, as mentioned should work only with more similar languages) and such weakly supervised init: e.g., the method could take all one-to-one translation pairs from the seed dictionary as 'shared words'. It would also be interesting to combine such 'shared words' and words that are really shared (identically spelled words) as initialisation and measure how it affects the results, also in light of differences in language distance. Adding one such experiment would make the paper more interesting.\n\nSome recent strong baselines are missing: e.g., I wonder how the model that does self-learning on top of seed dictionaries (similar to the work of Vulic et al., EMNLP 2019) compares to the proposed method. Also, can we use the same self-learning technique with the combined method here? Would that lead to improved results? Another work I would like to see comparisons to is the work of Zhang et al. (ACL 2019) and despite the fact that the authors explicitly stated that they decided not to compare to the work of Artetxe et al. (ACL 2019) as they see it as concurrent work, I would still like to see that comparison as the model of Artetxe et al. seems very relevant to the presented work.\n\nI do not see the extension to contextualized representations (described in Section 3.2) as a real contribution: this is a very straightforward method to apply an alignment-based method on multilingual BERT representations, and very similar techniques have been probed in previous work (e.g., by Aldarmaki & Diab), only the bilingual signal/dictionary came from parallel data instead.\n\nFinally, as mentioned before, one of the must-have experiments is including more (distant and diverse) language pairs and analysing the results across such language pairs, with an aim to further understand how the levels of sharing, over-sharing, and non-isomorphism affect performance. Also, while the authors state that reduced isomorphism is the key problem of alignment-based methods, I still fail to see how exactly the alignment refinement step does not suffer from that problem? More discussion is needed here.\n\nOther comments:\n- It would be useful to also point to the survey paper of Ruder et al. (JAIR 2019) where the difference between alignment-based and joint models is described in more detail and could inform an interested reader beyond the confines of the related work section in this paper. Similarly, differences between joint models and alignment-based models have also been analysed by Upadhyay et al. (ACL 2016); Vulic and Korhonen (ACL 2016).\n\n- I like how Figure 1 clearly visualizes the main intuitions behind the proposed framework, and how mitigating the oversharing problem leads to better alignments. This was very nice.\n\n- In light of the problems with silver standard MUSE datasets (see also the recent work of Kementchedjhieva, EMNLP 2019), I would suggest to run BLI experiments on more language pairs: e.g., there are BLI datasets of Glavas et al available for 28 language pairs.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer worries include: whether the approach scales to distant language pairs, overselling of the paper as a "framework", a few citations and comparisons missing. I agree and encourage the authors not to use the word "framework" here. I would also encourage the authors to evaluate on more interesting language pairs, and analyze what vocabularies are relocated, as well as what their method is better at compared to previous work. 
Decision: Accept (Poster)