Multilingual Alignment of Contextual Word Representations
Authors: Steven Cao
Nikita Kitaev
Dan Klein
Venue:  (2020)
Keywords: multilingual
natural language processing
embedding alignment
BERT
word embeddings
transfer

Abstract: We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

Published as a conference paper at ICLR 2020
MULTILINGUAL ALIGNMENT OF CONTEXTUAL WORD REPRESENTATIONS
Steven Cao, Nikita Kitaev & Dan Klein Computer Science Division
University of California, Berkeley
{stevencao, kitaev, k wweenneeeeeeeeeeeeeeee..
ABSTRACT
We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits sig- nificantly improved zero-shot performance on XNLI compared to the base model, remarkably matching feeuud--uull-supervvii
translate-train models for Bulgar- ian and Greek. Further, to measure the degree of alignment, we introduce con- textual version of word retrieval and show that correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that corrected by the alignment procedure. These results support contextual alignment as useful concept for understanding large multilingual pre-trained models.
INTRODUCTION
Base BERT
Aligned BERT
2-
0-
year
woltee wented
Frage
kb
-0-
Gelegenhert
-8-
opportunity
12 10
Figure I-SNE (Maaten Hinton 2008
visualization the embedding space of multilingual German word pairs (left: pre-alignment, right: post-alignment). Each point is
BERT for English-German different instance of the word in the Europarl corpus. This figure suggests that BERT begins already somewhat aligned out-of out-of-the-box but becomes much more aligned after our proposed procedure.
Embedding alignment was originally studied for word vectors with the goal of enabling cross-lingual transfer, where the embeddings for two languages are in alignment if word translations, e.g. cat and Katze, have similar representations (Mikolov al. 2013a Smith et al., 2017). Recently, large pre- trained models have largely subsumed word vectors based on their accuracy on downstream tasks, partly due to the fact that their word representations are context-dependent, allowing them to more richly capture the meaning of word (Peters et al.) 2018 Howard & Ruder 2018 Radford et al.) 2018 Devlin et al., (018). Therefore, with the same goal of cross-lingual transfer but for these more complex models, word pairs within parallel sentences, e.g. cat in "The cat sits" and Katze "Die Katze sitzs," have similar representations.
might consider contextual embedding alignment, where observe whether

---

Published as a conference paper at ICLR 2020
One model relevant to these questions is multilingual BERT, version of BERT pre- mre-trained on 104 languages that achieves remarkable transfer on downstream tasks. For example, after the model is fine-tuned on the English MultiNLI training set, it achieves 74.3% accuracy the test set in Span- ish, which is only 7.1 % lower than the English accuracy (Devlin et al., 2018, Conneau et al. (0188). Furthermore, while the model transfers better to languages similar to English, it still achieves rea- sonable accuracies even on languages with different scripts.
However, given the way that multilingual BERT was pre-trained, it unclear why we should expect such high zero-shot performance. Compared to monolingual BERT which exhibits no zero-shot transfer, multilingual BERT differs only in that (1) during pre- pre-training (i.e. masked word prediction), each batch contains sentences from all of the languages, and (2) it uses a single shared vocabulary, formed by WordPiece on the concatenated monolingual corpora (Devlin et al.) 2019). Therefore,
we might wonder: (1) How can we hetter understand BERT's multilingualism? (2) Can we further improve BERT'S cross-lingual transfer?
In this paper, we show that contextual embedding alignment is useful concept for addressing these questions. First, we propose a contextual version of word retrieval to evaluate the degree of alignment, where a model is presented with two parallel corpora, and given a word within sentence in one corpus, it must find the correct word and sentence in the other. Using this metric of alignment, show that multilingual BERT achieves zero-shot transfer because its embeddings are partially aligned, as depicted in Figure I with the degree of alignment predicting the degree of downstream transfer.
Next, using between 10K and 250K sentences per language from the Europarl corpus as parallel data (Koehn) (005), we propose a fine-tuing-bbased alignment procedure and show that signifi- cantly improves BERT as a multilingual model. Specifically, on zero-shot XNLI, where the model trained on English MultiNLI and tested on other languages (Conneau et al. 2018b). the aligned model improves accuracies by 2.78% on average over the base model, and it remarkably matches translate- train models for Bulgarian and Greek, which approximate the fully-supervised setting.
To put our results in the context of past work, we also use word retrieval to compare our fine- tuning procedure to two alternatives: (1) fastText augmented with sentence and aligned using rota- tions (Bojanowski et al.] 2017: Rücklé al. 22018; AAttetx et al. (2018), and (2) BERT aligned using rotations (Aldarmaki & Diab 2 Schuster et al.) 2019; Wang et al. (019)). We find that when there are multiple occurences per word, fine-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT. This result supports the intuition that contextual aligmment is more difficult than its non-contextual counterpart, given that rotation, at least when applied naively, is no longer sufficient to produce strong alignments. In addition, when there is only one occurrence per word, fine-tuned BERT matches the performance of fastTeext Given that context disambiguation is no longer necessary, this result suggests that our fine-tuning procedure is able to align BERT at the type level degree that matches non-contextual approaches.
Finally, we use the contextual word retrieval task to conduct finer-grained analysis of multilingual BERT, with the goal of better understanding its strengths and shortcomings. Specifically, find that base BERT has trouble aligning open-class compared to closed-class parts-of-speech, as well as word pairs that have large differences in usage fireuuenc, suggesting insight into the pre-training procedure that we explore in Section 5 Together, these experiments support contextual alignment as an important task that provides useful insight into large multilingual pre- trained models.
2 RELATED WORK
Word vector alignment. There has been a long line of works that learn aligned word vectors from varying levels of supervision (Ruder et al., 2019]. One popular family of methods starts with word vectors learned independently for each language (using a method like skip-gram with negative sampling (Mikolov et al. 2013b)), and it learns a mapping from source language vectors to target language vectors with bilingual dictionary
supervision (Mikolov et al.) 2013a: Smith et al.) 2017 Artetxe et al. (017). When the mapping is constrained to be an orthogonal linear transfor- mation, the optimal mapping that minimizes distances between word pairs can be solved in closed form (Artetxe et al., 2016, Schonemann] 1966). Aligmment is evaluated using bilingual lexicon in- duction, these papers also propose ways to mitigate the hubness problem in nearest neighbors,
2

---

Published as a conference paper at ICLR 2020
e.g. by using alternate similarity functions like CSLS (Conneau et al. 2018a). A recent set of works has also shown that the mapping can be learned with minimal to no supervision by starting with
some minimal seed dictionary and alternating between learning the linear map and inducing the die-
tionary (Artetxe et al. 2018 Conneau et al. 2018a Hoshen & Wolf 2018: Xu et al. 2018 Chen & Cardie 2018).
Incorporating context into alignment. One key challenge in making alignment context aware is that the embeddings are now different across multiple occurrences of the same word, Past papers have handled this issue by removing context and aligning the "average sense" of a word. In one such study, SScuuster et al. (2019) learn rotation align contextual ELMo embeddings (Peters) iet al.] (2118) with the goal of improving zero-shot multilingual dependency parsing, and they handle context by taking the average embedding for word in all of its contexts. In another paper, AAl- darmaki & Diab (2019) learn rotation on sentence vectors, produced by taking the average word vector over the sentence, and they show that the resulting alignment also works well for word-level tasks. In contemporaneous work. Wang et al. (2019) align not only the word but also the context by learning a linear transformation using word-aligned parallel data to align multilingual BERT, with the goal of improving zero-shot dependency parsing numbers. In this paper, we similarly align not only the word but also the context, and we also depart from these past works by using more expressive alignment methods than rotation.
Incorporating parallel texts into pre-training. Instead of performing alignment post-hoe, an- other line of works proposes contextual pre-training procedures that are more tross-ingguall-awarre Wieting et al. (2019) pre-train sentence embeddings using parallel texts by maximizing similar- ity between sentence pairs while minimizing similarity with negative examples. [Lmmpee & Con- neau (219)) propose a cross-lingual pre-training objective that incorporates parallel data in addi- tion to monolingual corpora, leading to improved downstream gross-lingua transfer. In contrast, our method uses less parallel data and aligns existing pre-traine trained models rather than requiring pre- training from scratch.
Analyzing multilingual BERT. Pires et al. (2019) present series of probing experiments to better understand multilingual BERT, and they find that transfer possible even between dissimilar lan- guages, but that it works better between languages that are typologically similar. They conclude that BERT is remarkably multilingual but falls short for certain language pairs.
3 METHODS
3.1 MULTILINGUAL PRE- TRAINING
We first briefly describe multilingual BERT (Devlin et al. 2018). Like monolingual BERT, mul- tilingual BERT is pre-trained on sentences from Wikipedia to perform two tasks: masked word prediction, where must predict words that are masked within a sentence, and next sentence predic- tion, where it must predict whether the second sentence follows the first one. The model is trained on 104 languages, with each batch containing training sentences from each language. and it uses a shared vocabulary formed by WordPiece on the 104 Wikipedias concatenated (Wu et al. 2016).
3.2 DEFINING AND EVALUATING CONTEXTUAL ALIGNMENT
In the following sections, we describe how to define, evaluate, and improve contextual align- ment. Given two languages, model is in contextual alignment if it has similar representations for word pairs within parallel sentences. More precisely, suppose we have N parallel sentences C C= ((s.,t)).... (s*V {,)). where (s, t) is a source-target sentence pair. Also, let each sentence pair (s, t) have word pairs, denoted a(s,t) t) { J1), (im.jmu }m)) containing position tuples (i,j) such that the words s; and ty are translations of each other We will use f to represent a pre- pre-trained model such that fii,s) is the contextual embedding for the ith word in s.
'These pairs are called word alignments in the machine translation community, but we use the term "word pairs" to avoid confusion with embedding alignment. Also, hecause BERT operates on subwords while the corpus is aligned at the word level, we keep only the BERT vector for the last suhword of each word.
3

---

Published as a conference paper at ICLR 2020
As example, we might have the following sentence pair:
s = (7 ate the apple .) t= {Tch habe den Apfel gegessen ") -}
0 1 2 3
0
3
a(s,t) {(0, 0). (1, (2, 2), (3, (4,5))
Then, using the parallel corpus C, we can measure the contextual alignment the model f using its accuracy in contextual word retrieval. In this task, the model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must find the correct word and sentence in the other. Specifically, we can define nearest neighbor retrieval function
neighbor(i, s; f.C)= =
argmax tEC, 0S.iSen((t)
si,
where i and j denote the position within a sentence and sim is a similarity function. The accuracy then given by the percentage exact matches over the entire corpus, or
A(f:C)= -
Σ Σ
([neighborli.s: (. (i.t),
(S.t) ((J))a((.tt
where I represents the indicator function. We can perform the same procedure in the other direction, where we pull target words given source words, so we report the average between the two directions. As our similarity function, we use CSLS, modified version of cosine similarity that mitigates the hubness problem, with neighborhood size 10 (Conneau et al. 2018ap. One additional point that this procedure can be made more or less contextual based on the corpus: a corpus with more occurrences for each word type requires better representations of context. Therefore, we also test non-contextual word retrieval by removing all but the first occurrence of each word type.
Given parallel data, these word pairs can procured in an unsupervised fashion using standard techniques developed by the machine translation community (Brown al.) 1993). While these methods can be noisy, by running the algorithm both the source-target and target-source directions and only keeping word pairs in their intersection, we can trade-off coverage for accuracy, producing a reasonably high-precision dataset (Och & Ney 2003).
3.3 ALIGNING PRE- PRE-TRAINER TRAINED CONTEXTUAL EMBEDDINGS
To improve the aligmment of the model with respect the corpus C, we can encapsulate alignment in the loss function
L(fC)= -- E Σ
sim(f(i, f(.tt)
(s.t)EC (i.))Ea(s.e)
where we sum the similarities between word pairs. Because the CSLS metric is not easily optimized, instead use the squared error loss, sim(s(i, f(i, (i, fjj t)
However, note that this loss function does not account for the informativity of f: for example, is zero if f is constant. Therefore, at a high level, would like minimize L(f: while main- taining some aspect of f that makes it useful, e.g. its high accuracy when fine-tuned on downstream tasks. Letting Jo denote the initial pre-trained model before alignment, we achieve this goal by defining regularization term
len(t) Σ
-(: LEC E=1
which imposes a penalty if the target language embeddings stray from their initialization. Then, and take gradient steps of the function L(f: + XR(f: B) di-
we sample minibatches rectly on the weights of f. which moves the source embeddings toward the target embeddings while preventing the latter from drifting too far. In our experiments, we set 1.
In the multilingual case, suppose we have k parallel corpora C²l C'...., C* where each corpus has different source language with the target language as English. Then, we sample equal-sized batches B² Oi from each corpus and take gradient steps on Σk= L(f: ) + XR(f; B*), which moves all of the non-English embeddings toward English.
4

---

Published as a conference paper at ICLR 2020
Note that this alignment method departs from prior work, in which each oon-English language is rotated to match the English embedding space through individual learned matrices. Specifically, the most widely used post-hoc alignment method learns rotation W applied to the source vectors to minimize the distance between parallel word pairs, or
min Σ Σ
I f(i,t113 s.t. W W W
(s,t)CC (a.))
This problem is known as the Procrustes problem and can be solved in closed form (Schonemann) 1966). This approach has the nice property that the vectors only rotated, preserving distances and therefore the semantic information captured by the embeddings (Artetxe et al.] 2016). However, rotation requires the strong assumption that the embedding spaces are roughly isometric (Søgaard itt al. 2018), an assumption that may not hold for contextual pre-trained models because they rep- resent more aspects of a word than just its type, i.c. context and syntax. which are less likely to be isomorphic between languages. Given that past work has also found independent alignment per language pair to be inferior to joint training (Heyman et al. (019)) another advantage of our method is that the alignment for all languages is done simultaneously.
As our dataset, we use the Europarl corpora for English paired with Bulgarian, German, Greek, Spanish, and French, the languages represented in both Europarl and XNLI (Koehn) 2005). After tokenization (Koehn et al. 2007); we produce word pairs using fastAlign and keep the one-to-one pairs in the intersection (Dyer et al. 2013). We use the most recent 1024 sentences the test set, the previous 1024 sentences as the development set, and the following 250K sentences as the training set. Furthermore, the training set. We also remove any exact matches, e.g. punctuation and numbers, because BERT is already aligned for these pairs due to its shared vocabulary. Given that parallel data may be limited for low-resource language pairs, we also report numbers for 10K and 50K parallel sentences.
modify the test set accuracy calculation to only include word pairs not seen in
3.4 SENTENCE-AUGMENN
NON-CONTEXTUAL BASELINE
Given that there has been a long line of work on word vector alignment (Artetxe et al.) 2018 Con- neau et al. 201 Smith et al. 2017 inter alia), we also compare BERT to centence-augmenter Aarlowing/Aretxe TastText baseline (Bojanowski et al.) (017). Following Artetxe al. (018), we first normalize, then mean-center, then normalize the word vectors, and we then learn a rotation with the same parallel data as in the contextual case, as described in Equation [| We also strengthen this baseline by in- cluding sentence information: specifically, during word retrieval, we concatenate each word vector with a vector representing its sentence. Following RRücklé et al. (2018), we compute the sentence vector by concatenating the average. maximum, and minimum vector over all of the words in the sentence, a method that was shown to be state-of-the-arr for a suite of mooss-lnngual tasks. We also experimented with other methods, such first retrieving the sentence and then the word, but found this method resulted in the highest accuracy. As a result, the fastText vectors are 2200-dimensional, while the BERT vectors are vis-iiimesiooal.
3.5 TESTING ZERO- SHOT TRANSFER
The next step is to determine whether better alignment improves cross-lingual transfer. As our downstream task, we use the XNLI dataset, where the English MultiNL development and test sets are human-translated into multiple languages (Conneau et al. 2018b Williams et al. 2018). Given a pair of sentences, the task is predict whether the first sentence implies the second, where there are three labels: entailment, neutral, or contradiction. Starting from either the base or aligned multi- lingual BERT, we train on English and evaluate on Bulgarian, German, Greek, Spanish, and French, the XNLI languages represented in Europarl.
As our architecture, following Devlin et al.) (2018), we apply a linear layer followed by softmax on the [CLS] embedding of the sentence pair, producing scores for each of the three labels. The model trained using cross-entropy loss and selected based on its development set accuracy aver- aged across all of the languages. As a fully-supervised ceiling, we also compare to models trained and tested on the same language, where for the non- -on-English training data, we use the machine trans- lations of the English MultiNLI training data provided by Conneau et al. (2018b). While the quality of the training data is affected by the quality of the MT system, this comparison nevertheless serves as good approximate imption for the sully-super setting.
5

---

Published as a conference paper at ICLR 2020
English
Bulgarian
German
Greek
Spanish
French
Average
Translate-
Base BERT
81.9
73.6
75.9
71.6
77.8
76.8
76.3
Zero-Sho
Base BERT
80.4
68.7
67.0
74.5
73.4
72.4
Sentence-aligned BERT (rotation)
81.1
68.9
71.2
66.7
74.9
73.5
72.7
Word-aligned BERT (rotation)
78.8
69.0
71.3
67.1
74.3
73.0
72.2
Word-aligned BERT (fine-tuned)
80.1
73.4
73.1
71.4
75.5
74,5
74.7
XLM (MLM TLM)
77.4
77.8
76.6
78.9
78.7
79.1
Table I: Accuracy on the XNLI test set, where compare base BERT (Devlin al. 2018) and two rotation-base based methods, sentence alignment (Aldarmaki & Diab 2019) and word align- ment (Wang al.] (2019). We also include the current state-of-the art zero-shot achieved by XLM (Lample & Conneau 2019). Rotation-based methods provide small gains on some languages but not others. On the other hand, after fine-tuning- alignment, Bulgarian and Greek match the translate-tren train ceiling, while German, Spanish, and French close roughly one-third of the gap.
Note that the zero-shot Base BERT numbers are slightly different from those reported in Devlin et al. 2019) because we select single model using the average accuracy across the six languages. This selection method also accounts for the varying English accuracies across the zero-shot methods.
Sentences
English
Bulgarian
German
Greek
Spanish
French
Average
None
80.4
68.7
70.4
67.0
74.5
73.4
72.4
10K
79.2
71.0
71.8
67.5
75.3
74.1
73.2
50K
81.1
73
72.6
69.6
75
74.5
74.3
250K
80.1
73.4
73.1
71.4
75.5
74.5
74.7
Table 2: Zero-shot accuracy the XNLI test set, where we align BERT with varying amounts of parallel data. The method scales with the amount of data but achieves large fraction the gains with 50K sentences per language pair.
4 RESULTS
4.1 ZERO-SHOT XNLI TRANSFER
First, we test whether alignment improves multilingual BERT by applying the models to zero-shot XNLI, as displayed in Table [ We see that our alignment procedure greatly improves accuracies, with all languages seeing gain of at least 1%. In particular, the Bulgarian and Greek zero-shot numbers are boosted by almost 5% each and match the translate-train numbers, suggesting that the alignment procedure is especially effective for languages that initially difficult for BERT. We also run alignment for more distant language pairs (Chinese, Arabic, Urdu) and find similar results, which we report in the appendix.
Comparing to rotation-based methods (Aldarmaki & Diab 2019 Wang et al. 2019), we find that a rotation produces small gains for some languages, namely Bulgarian, German, and Spanish, but is sub-optimal overall, providing evidence that the increased expressivity of our proposed procedure is beneficial for contextual alignment. We explore this comparison more in Section 5.1
4.2 ALIGNMENT WITH LESS DATA
Given that our goal is zero-shot transfer, we cannot expect to always have large amounts of par- allel data. Therefore, also characterize the performance of our alignment method with varying amounts of data, as displayed in Table [] We find that improves transfer with as little as 10K sentences per language, making it promising approach for low-resource languages.

---

Published as a conference paper at ICLR 2020
bg-en
de-en
el-en
es-en
fr-en
Average
Contextual
Aligned fastText sentence
44.0
46.4
42.0
48.6
44.5
45.1
Base BERT
19.5
26.1
13.9
32.5
28.3
24.1
Word-aligned BERT (rotation)
29.8
31.6
20.8
36.8
31.0
30.0
Word-aligned BERT (fine-tuned)
50.7
51.3
49.8
51.0
48.6
50.3
Non-Contextur
Aligned fastText sentence
61.3
65.4
61.6
71.1
64.8
64.8
Base BERT
29.1
37.0
22.3
46.5
41.8
35.3
Word-aligned BERT (rotation)
39.6
43.6
32.4
51.4
46.1
42.6
Word-aligned BERT (fine-tuned)
62.8
64.3
67.5
68.4
66.3
65.9
Word retrieval accuracy for the aligned sentence-augmented fastText baseline and BERT pre- and post-alignment. Across languages, base BERT has variable accuracy while fine-tuning- aligned BERT is consistently effective. Fine-tuned BERT also matches fastText in a version of the task where context not necessary, suggesting that our method matches the type-level alignment of fastText while also aligning context.
Table 3:
ANALYSIS
5.1 WORD RETRIEVAL
In the following sections, we present word retrieval results to both compare our method to past work and better understand the strengths and weaknesses of multilingual BERT. Table 3 displays the word retrieval accuracies for the aligned sentence- augmented fastText baseline and BERT pre- and post- alignment. First, we find that contextual retrieval, fine-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT. This result supports the intuition that aligning large pre-trained models is more difficult than aligning word vectors, given that a rotation, least when applied naively, produces sub- par alignments. In addition, fine-tuned BERT matches the performance of fastText in non-contextual retrieval, suggesting that our alignment procedure overcomes these chal- lenges and achieves type-level alignment that matches non-contextual approaches. In the appendix, we also provide examples of aligned BERT disambiguating between different meanings word, giving qualitative evidence of the benefit of context alignment.
We also find that before alignment, BERT's performance varies greatly between languages, while after alignment it consistently effective. In particular, Bulgarian and Greek initially have very low accuracies. This phenomenon is also reflected in the XNLI numbers (Table where Bulgarian and Greek receive the largest boosts from alignment. Examining the connection between alignment and zero-shot more closely, we find that the word retrieval accuracies are highly correlated with downstream zero-shot performance (Figure 2 supporting our evaluation measure predictive of cross-lingual transfer.
The language discrepancies also consistent with a hypothesis by Pires al. (2019) explain BERT's multilingualism. They posit that due to the shared vocabulary, shared words between lan- guages, e.g. numbers and names, are forced to have the same representation. Then, due to the masked word prediction task, other words that co-occur with these shared words also receive similar representations. If this hypothesis is true, then languages with higher lexical overlap with English are likely to experience higher transfer. As extreme form of this phenomenon, Bulgarian and Greek have completely different scripts and should experience worse transfer than the common lan- guages, intuition that is confirmed by the word retrieval and XNLI accuracies. The fact that all languages are equally aligned with English post-alignment suggests that the pre-training procedure suboptimal for these languages.

---

Published as a conference paper at ICLR 2020
Lexical Overlap
Numeral
Punctuation
Proper Noun
Average
Base BERT
0.90
0.88
0.80
0.86
Aligned BERT
0.97
0.96
0.95
0.96
Closed-Class
Determiner
Preposition 0.72
Conjunction
Pronoun
Auxiliary 0.61
Average
Base BERT 0.76
0.71
0.70
0.70
Aligned BERT
0.91
0.86
0.89
0.89
0.84
0.88
Open- ClassClass
Noun
Adverb
Adjective
Verb
Average
Base BERT
0.61
0.57
0.50
0,49
0.54
Aligned BERT
0.90
0.88
0.90
0.89
0.89
Accuracy by part-of-speech tag for non-contextual word retrieval. To achieve better word type coverage, we do not remove word pairs seen in the training set. The tags are grouped into lexically overlapping, closed-class, and open-class groups. The "Particle," and "Other" tags are omitted.
Table 4:
"Particle""Syymbol" "Symbol," "Interjection,"
fr
142
V
74
XNLI 2010 shot accuracy
72
Freetiin corrwct
0.83
70
68
6.70
bg
Aligned BERT
Base BERT
66
Contextual word retricval accurncy
[Targel frequency rank Sourve frequency zank]
Figure 2: XNLI zero-shot versus word re- trieval accuracy for base BERT, where each point is language paired with English. This plot suggests that alignment correlates well with cross-linge linssll transfer.
Figure Contextual word retrieval accuracy plotted against difference in frequency rank between source and target. The accuracy of base BERT plummets for larger differences, suggesting that its alignment de- pends on word pairs having similar usage statistics.
5.2 WORD RETRIEVAL PART-OF-SPEECH ANALYSIS
Next, to gain insight into the multilingual pre-training procedure, we analyze the accuracy broken down by part-of-speech using the Universal Part-of-Speech Tagset (Petrov al. 2012). annotated using polyglot (Al-Rfou et al. 2013) for Bulgarian and spaCy (Honnibal & Montani 201 2017) for the other languages, as displayed Table Unsurprisingly, multilingual BERT has high alignment tut-of-the-bon for groups with high lexical overlap, e.g. numerals, punctuation, and proper nouns, due to its shared vocabulary. We further divide the remaining tags into closed-class and open-class, where closed-class parts-of-speech correspond to fixed sets of words serving grammatical functions (e.g. determiner, preposition, conjunction, pronoun, and auxiliary), while open-class parts-of-speed. correspond to lexical words (c.g. noun, adverb, adjective, verb). Interestingly, see that base BERT has consistently lower accuracy for closed-class versus open-clase categories (0.70 vs 0.54), but that this discrepancy disappears after alignment (0.88 vs 0.89).
5.3 USAGE HYPOTHESIS FOR ALIGNMENT
From this closed-class open-class difference, hypothesize that BERT's alignment of par- ticular word pair is influenced by the similarity of their usage statistics. Specifically, given that BERT is trained through masked word prediction, its embeddings in large part determined by
8

---

Published as a conference paper at ICLR 2020
the co-occurrences between words. Therefore, two words that used in similar contexts should be better aligned. This hypothesis provides an explanation of the closed-class vs open-class difference: closed-class words typically grammatical, similar languages. Furthermore, these words cannot be substituted for one another due to their grammatical function. Therefore, their usage statistics are strong signature that be used for alignment. On the other hand, open-class words can be substituted for one another: for example, in most sentences, the noun tokens could be replaced by wide range of semantically dissimilar nouns with the sentence remaining syntactically well-formed. By this effect, many nouns have similar co-occurrences, making them difficult align through masked word prediction alone.
they used in similar ways across typologically
To further test this hypothesis, we plot the word retrieval accuracy versus the difference between the frequency rank of the target and source word, where this difference measures discrepancies in usage, as depicted in Figure 3 We see that accuracy drops off significantly as the source-target difference increases, supporting our hypothesis. Furthermore, this shortcoming remedied by alignment, revealing another systematic deficiency of multilingual pre-training.
CONCLUSION
Given that the degree of alignment is causally predictive downstream cross-lingual transfer, con- textual alignment proves to be useful concept for understanding and improving multilingual pre- trained models. Given small amounts of parallel data, our alignment procedure improves multilin- gual BERT and corrects many of its systematic deficiencies. Contextual word retrieval also provides useful new insights into the pre-training procedure, opening up new avenues for analysis.
REFERENCES
Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. Polyglot: Distributed word representations for multilingual nlp. In Proceedings of the Seventeenth Conference on Computational Natural Lan- guage Learning, pp. 183-192, Sofia, Bulgaria, August 2013. Association for Computational Lin- guistics. URL waLhhtp:: www. ac web.
3520
Hanan Aldarmaki and Mona Diab. Context-aware cross- rrngual mapping. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 3906-3911, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1 1391. aww.allwelll
URL
1391
Mikel Artetxe, Gorka Labaka, and Eneko Agiree Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2289-2294, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/1//DI--1 1250. URL https: //www.co
1250
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 451-462, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v 1/P17-1042.
URL http::::// www. aclweb.
bre//antholooy/P17--11
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsu- pervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pp. Melbourne, Australia, July 2018. Association for Computational Linguistics. URL https:
// wwwww. wcwweb......... wr..orrgaae hergg/anhologgoo
1073
Piotr Bojanowski, Edouard Grave, Anmand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135- 146. 2017. doi: 10.1162/tacl a 00051. URL hhtpp:: //www. aclweb. ozq/antholoqy/
Q17-1010

---

Published as a conference paper at ICLR 2020
Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Comput. Linguist., 19(2): 263-311, June 1993. ISSN 0891-2017. BRLttt/://cl acm. org/ /wgg/iitaiion.. cfm?id= 972470 972474
Xilun Chen and Claire Cardie. Unsupervised multilingual word embeddings. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 261-270, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/
1024. URL htthttss::/ wwtp:/awwww..o...om wtt../ew...gg/aaoooggg////.///ee
ac
1024
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Herve J'egou. Word translation without parallel data. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), 2018a. URL
24087.pd7
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings the the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475 2485, Brussels, Belgium, Noobbbr-November 2018b. Association for Computational Lin- guistics. doi: 10.18653/v1/D18- 1269.
aclweb. org hagg/nthhooogy// URL https:: wtt.:/wwwww....b.b........o.....
DI8- 1269
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. //Xiv: [cs.CI.], October 2018.
URL htR.httpp//arxiv.orrs 008000000001
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training
of deep bidirectional transformers for language understanding. https: /tte:///gtthue com/ md. 2019.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameterization of IBM model 2. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644 648, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL httph::t////// //ww.
aclweb. www.:ow.....aa...........eee antho logy
Andreas Eisele and Chen. MultiUN: A multilingual corpus from united nation documents. In Proceedingy of the Seventh International Conference on Language Resources and Eval- uation (LREC' 10), Valletta, Malta, May 2010. European Language Resources Association
(ELRA). URL http: www. www.w.rrecc wre.- conf http://oo....rooeeeeeeeeee//e//a
Paper.pa .
Geert Heyman, Bregt Verreet, Ivan Vulié, and Marie-Francine Moens. Learning unsupervised mul- tilingual word embeddings with incremental multilingual hubs. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 1890-1902, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/119-111 URL https: /tw::///wwwwwww ac web. orq/ www.annoooooolla...ooo
Matthew Honnibal and Ines Montani. spaCy Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear, 2017.
Yedid Hoshen and Lior Wolf. aovessreralrial unsupervised word translation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 469-478, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/
v1/D18- 1043. URL httlh:ttp:/// www. ac webb org/
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 928-339, Association for Computational Linguistics, 2018. URL
httt:://ae web. org/ antto 1
10

---

Published as a conference paper at ICLR 2020
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Conference Pro- ceedings: The Tenth Machine Translation Summit, pp. 79-86, Phuket, Thailand, 2005. AAMT.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bo- jar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical ma- chine translation. In Proceedings of the 45th Annual Meeting of the Association for Com- putational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp. 177-180, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: http::/wwwwwccccccc www.walwwwbbogo... http://rrrraaooooo///////////
Guillame Lample and Alexis Conneau. Cross-lingua language model pretraining. 2019. URL
Laurens van der Maaten and Geoffrey Hinton.
Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9:2579 2605, 2008. URLhttp://wwet jmlr julr.orrg/pppers////
vande meteeeermatennBaaaoe html
Tomas Mikolov, Quoc Le, and llya Sutskever. Exploiting similarities among languages for ma- chine translation. 2013a. 3a. URL https: http:ss//aaxtt..or//p///////////
1168.6.0 pdf
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 2, NIPS' 13, pp. 3111-3119, USA, 3b. Curran Associates Inc. URI. http:/ / acm. 2999792.22 2999959
cfmiid
Franz. Josef Och and Hermann Ney. systematic comparison of various statistical alignment models. Comput. Linguist., 29(1):1-55,, March 2003. ISSN 0891-2017. doi: 10.1162/ 089120010321337721, URL http:/ /ttp://dxxx doi. 11..1........... 111.11///0910013233333311333
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 1N18-120. URL
https: www ac
Slav Petrov, Dipanjan Das, and Ryan McDonald. A universal part-of-speech tagset. In Proceed- ings of the Eighth International Conference on Language Resources and Evaluation (LREC- 2012), pp. 2089-2096, Istanbul, Turkey, May 2012. European Languages Resources Association (ELRA). URL pattp://l wwwww/wwww.../////////////w.... 1rec- conf http:///ooeeeeeeeeee//////////// Paper. pdf
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4996-50000 5001, Florence, Italy, July 2019. Association for Computational Linguisties. URL
Alec Radford, Karthik Narasimhan. Tim Salimans, and Ilya Sutskever. ing language understanding by generative pre-training. 2018. URL
Improv-
https:
//s3-us-west-2 awa.onaaaaaaaaaaaaa.ammma andp://aaaaauuddeettaninninnnnno
pdf
Andreas Rücklé Steffen Eger, Maxime Peyrard, and Iryna Gurevych. Concatenated p-mean word embeddings as universal cross-lingual sentence representations. arXiv:1803.01400 [cs. CLJ, 2018.
URL URLKLtpp//ariv..oorabs///ee 01400
Sebastian Ruder, Ivan Vulié, and Anders Søgaard. A survey of cross-lingual word embedding mod- cls. J. Artif. Int. Res., 65(1):5696630, May 2019. ISSN 1076-9757. doi: 10.1613/jair. 1640. U ptttps::/doiii |tt
11640
11

---

Published as a conference paper at ICLR 2020
Peter H. Schonemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1599-1613, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/11--
URL https:
/ /www. wawwwweeeee www.wwe..rrgg... www.:ew.......t..ooooooooooooo// /NI 1166
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), 2017. URL https://
openzeview.
Anders Søgaard, Sebastian Ruder, and Ivan Vulić. On the limitations of unsupervised bilin- gual dictionary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 778-788, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 100118633/1//18--1 1072. URL https: www. ac web.
Jörg Tiedemann. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth In- ternational Conference on Language Resources and Evaluation 12), pp. 2214-2218, Is- tanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL http: //www. www.:-rec-coo conf. org http://ceeennn/eeeee2////////eee
Paper. Pde
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. Cross-lingual BERT trans- formation for zero-shot dependency parsing. In Proceedings of the 2019 Conference Em- pirical Methods Natural Language Processing and the 9th International Joint Conference on Hong Kong, China, Novem-
Natural Language Processing (EMNLP-IJCNLP), pp. ber 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-15555 URL
https: www. weeeborrg/atton waaecchooogyy/ooo ogy 9-155
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Bing-Kirkpatrick. Simple and ef- fective paraphrastic similarity from parallel translations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4602-4608. Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/1.965//11119--4553. URL https:
7/ 7/www.co www.wb.wwbbb....... org/ hwt.aatooooooootto/o 1453
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1 1112-1122, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10. (0.11863/11/N18-11001 URL httpt::/p///. / www. acl web.
hogganthollen watthhoooy/K111111111
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv: 1609.06 01..08144 44 [cs. CL., 2016.
Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. Unsupervised cross-lingual transfer of word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2465-2474, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 18.186 I/D 1268. URL https:// www aclawws web.
org//anthellogyDDB8-11
12

---

Published as a conference paper at ICLR 2020
English
Bulgarian
German
Greek
Spanish
French
Arabic
Chinese
Undu
Average
Translate-Tran
Base BERT
81.9
73.6
75.9
71.6
77.8
76.8
70.7
76.6
61.6
74,1
7cm Shot
Base BERT
80.4
68.7
70.4
67.0
74.5
73.4
70.6
60.3
70.1
Aligned BERT (20K sent)
80.8
71.6
72.5
68.1
74.7
73.6
66.3
71.5
61.1
71.1
Table Zero-shot accuracy on the XNLI test set with more languages, where we use 20K parallel sentences for each language paired with English. This result confirms that the alignment method works for distant languages and a variety of parallel corpora, including Europarl, MultiUN, and Tanzil, which contains sentences from the Quran (Koehn 2005 Eisele & Chen 2010 Tiedemann 2012).
A
APPENDIX
A.1 OPTIMIZATION HYPERPARAMETERS
For both alignment and XNLI optimization, rameters 3 (0.9, 0.98), = 10-9 and linear learning rate warmup for the first 10% of the training data. For alignment, the model is trained for one epoch, with each batch containing sentence pairs per language. For XNLI, each model is trained for 3 epochs with 32 examples per batch, and 10% dropout is applied the BERT embeddings.
learning rate of 10-5 with Adam hyperpa-
A.2 ALIGNMENT OF CHINESE, ARABIC, AND URDU
In T report numbers for additional languages, where we align single BERT model for all eight languages and then fine-tune on XNLI. We use 20K sentences per language, where we use the MultiUN corpus for Arabic and Chinese (Eisele & Chen 2010), the Tanzil corpus for Urdu (Tiede- mann 2012), and the Europarl corpus for the other five languages (Koehn) (005). This result con- firms that the alignment method works for variety of languages and corpora. Furthermore, the Tanzil corpus consists of sentences from the Quran, suggesting that the method works even when the parallel corpus and downstream task contain sentences from entirely different domains.
A.3 EXAMPLES OF CONTEXT-AWARE RETRIEVAL
In this section, qualitatively show that aligned BERT is able to disambiguate between different occurences of a word.
First, we find two meanings of the word "like" occurring in the English- English-German Europarl test set. Note also that the second and third example, the two senses "like" occur in the same sentence.
This empire did look for colonies far from home or overseas, like most Western Euro- pean States, but close by.
Dieses Reich suchte seine Kolonien nicht weit von Hause und in bersee wie die meisten westcuropäischen Staaten, sondern in der unmittelbaren Umgebung.
Like other speakers, would like to support the call for the arms embargo to remain.
Wie andere Sprecher, so möchte auch ich den Aufruf zur Aufrechterhaltung des Waffen- embargos untersttzen.
Like other speakers, would like support the call for the arms embargo to remain. Wie andere Sprecher, möchte auch ich den Aufruf zur Aufrechterhaltung des Waffen- embargos unterstizen.
would also like, although they are absent, to mention the Commission and the Council. Ich möchte mir sogar erlauben, die Kommission und den Rat zu nennen, auch wenn sie nicht anwesend sind.
13

---

Published as a conference paper at ICLR 2020
Multiple meanings of "order":
Moreover, the national political elite had to make a detour in Ambon in order to reach the civil governor's residence by warship.
In Ambon mußte die politische Spitze des Landes auch noch einen Umweg machen, um mit einem Kriegsschiff die Residenz des Provinzgouverreurs zu erreichen.
Although the European Union has an interest in being surrounded by large, stable regions, the tools it has available in order to achieve this are still very limited.
Der Europäischen Union ist zwar an großen stabilen Regionen in ihrer Umgebung gelegen, aber sie verfgt nach wie vor nur ber recht begrenzte Instrumente, um das zu erreichen.
We could reasonably expect the Indonesian govermment to take action in three funda- mental areas: restoring public order, prosecuting and punishing those who have blood on their hands and entering into political dialogue with the opposition.
Von der neuen indonesischen Regierung darf man mit Fug und Recht drei elementare MaB- nahmen erwarten: die Wiederherstellung der öffentlichen Ordnung, die Verfolgung und Bestrafung derjenigen, an deren Händen Blut klebt, und die Aufnahme des politischen Di- alogs mit den Gegnern.
Firstly. might mention the fact that the army needs to be reformed, secondly that stable system of law and order needs to be introduced. Ich nenne hier an erster Stelle die notwendige Reform der Armee, ferner die Einfhrung eines stabilen Systems rechtsstaatlicher Ordnung.
Multiple meanings of "support":
Financial support needed to enable poor countries to take part in these court activities. Arme Länder müssen finanziell unterstützt werden, damit auch sie sich an der Arbeit des Gerichtshofs beteiligen können.
We must help them and ensure that a proper action plan is implemented to support their work.
Es gilt einen wirklichen Aktionsplan auf den Weg zu bringen, um die Arbeit dieser Organ- isationen zu unterstützen.
So I hope that you will all support this resolution condemning the abominable conditions of prisoners and civilians in Djibouti.
Ich hoffe daher, daB Sie alle diese Entschließung befürworten, die die entsetzlichen Be- dingungen von Inhaftierten und Zivilpersonen in Dschibuti verurteilt.
It would be difficult to support a subsidy scheme that channelled most of the aid to the large farms the best agricultural regions.
Es wäre auch problematisch, ein Beihilfesystem zu befürworten, das die meisten Beihilfen in die großen Betriebe in den besten landwirtschaftliche Gebieten lenkt.
Multiple meanings of "close":
This empire did not look for colonies far from home or overseas, like most Western Euro- pean States, but close by.
Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee wie die meisten westeuropäischen Staaten, sondern in der unmittelbaren Umgebung.
In addition, if we are to shut down or refuse investment from every company which may have association with the arms industry, then would have to close virtually every American and Japanese software company on the island of Ireland with catastrophic con- sequences.
Wenn wir zudem jedes Unternehmen, das auf irgendeine Weise mit der Rstungsindustrie verbunden ist, schließen oder Investitionen dieser Unternehmen unterbinden, dann mßten wir so ziemlich alle amerikanischen und japanischen Softwarreuntemeehnen auf der irischen Insel schließen, was katastrophale Auswirkungen hätte.
14

---

Published as a conference paper at ICLR 2020
On the other hand, the deployment of resources left over in the Structural Funds from the programme planning period 1994 to 1999 is hardly worth considering as the available funds have already been allocated to specific measures, in this case in close collaboration with the relevant French authorities.
Die Verwendung verbliebener Mittel der Strukturfonds aus dem Programmpla- nungszeitraum 1994 bis 1999 dagegen kaum Erwägung zu ziehen, da die verfgbaren Mittel bereits bestimmten Maßnahmen zugewiesen sind, und zwar im konkreten Fall im engen Zasammenwirken mit den zuständigen französischen Behörden.
This is particularly justified given that, as already stated, many Member States have very close relations with Djibouti.
Zumal, wie erwähnt, viele Mitgliedstaaten sehr enge Beziehungen zu Dschibuti unterhal- ten.
Mr President, it is regrettable that, at the close of the 20th century, a century symbolised so positively by the peaceful women's 's revolution, there are still countries, such Kuwait and Afghanistan, where half the population, women that is, still denied fundamental human rights.
Herr Präsident! Es ist wirklich bedauerlich, daß so positiv von der friedlichen Revolution der Frauen geprägten Jahrhunderts, noch immer Länder wie Kuwait und Afghanistan gibt, in denen der Hälfte der Bevölkerung, den Frauen, die elementaren Menschenrechte verweigert werden.
Ende des 20. Jahrhunderts, eines
15


Reviews:
[{"review_id": "Hkeb5VNZ5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a pre-training method for strengthening the contextual embeddings alignment. Given parallel sentences from a different language, the authors proposed to enforce corresponding words that have a similar representation by minimizing the squared error loss. The authors also proposed to use the an regulation that prevents the learned embedding from drift too far. The authors evaluated the proposed pre-training on the contextual alignment metric and show the BERT has variable accuracy depends on the language. The proposed method improved significantly on zero-shot XNLI compares to the base model. \n\nThe paper is well written, and the proposed aligned loss makes sense and should augment the multi-lingual pre-training from a high level. The authors did a good job of analyzing the bert for multi-lingual. There some details may help the reader understand the paper better\n\n1: Why use L2 distance as the metric function, what is the performance of using the inner product as a metric function? and what is the difference here? \n\n2: The authors mentioned the word pairs are extracted from the existing method which may be noisy. I wonder is there any ablations study with respect to how the word pairs affect the pretraining? \n\n3: When finetuning on zero-shot transfer, what is the finetune setting? Is there any strategy to avoid the lower layer embedding from drifting away? \n\n4: In table 3, the Fully supervised Base Bert on English is close to the zero-shot setting and the base BERT model is better than Alignment bert, I wonder can the authors explain more on this? \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlv3F9nYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper conducts a series of experiments on the multilingual BERT model of Devlin et al., aiming to inject stronger bilingual knowledge into the model for improved 'Aligned BERT'. The knowledge originating from parallel (Europarl) data improves the model significantly as shown on tasks such as contextual and non-contextual word retrieval as well as in zero-shot XNLI task. The paper continues the line of work on cross-lingual contextualised word embeddings, and it brings several minor contributions, but overall I do not see it as a very inspiring piece of work, and it leaves open several very important questions, in particular its relationship to prior work and some potentially stronger baselines than the ones reported in the paper, plus more experiments with more distant language pairs.\n\nI am not exactly sure that the comparison between 'Aligned BERT' and the main baseline 'Aligned fastText + sentence' is completely fair. 'Aligned BERT' uses more than 2M Europarl sentences to learn the alignment, while the standard alignment methods for learning cross-lingual word embeddings (see e.g. Ruder et al.'s survey) typically rely only on 5k translation pairs or even less pairs. There is a huge difference in the strength of the bilingual signal between 2M parallel sentences and, say, 2k, word translation pairs. \n\nThe main goal of the paper is to improve alignment of the starting multilingual BERT model, but I wonder why the authors have not compared to a more suitable XLM baseline of Lample and Conneau (NeurIPS 2019; the paper has been on arXiv since January 2019) - the XLM model uses exactly the same resources as 'Aligned BERT': parallel sentences from Europarl, while the main baseline here uses only seed dictionaries to learn the mapping. Regarding the baselines, it is also not clear to me why the authors have not compared to previous work of Schuster et al. (2019) and Aldarmaki and Diab (2019) at least in tasks where the models can be directly compared (XNLI or non-contextual word retrieval). Also, another non-contextual model which is worth trying is a joint model which relies on parallel sentences (similar to Ormazabal et al., ACL-19).\n\nFor the 'Aligned fastText + sentence' baseline, it would be interesting to report numbers with another (hybrid) baseline model that combines aligned fastText vectors with sentence encodings produced by multilingual BERT or some other multilingual sentence encoder (such as LASER, see Schwenk et al., 2019). Simply taking min, max, and avg vectors over all the sentence words might not be the best way to encode the sentence, and I would like to see more experiments here.\n\nThe paper makes some claims on novelty which 1) partially overlap with prior work, or 2) it does not cite related work while it leans on its findings. For instance on Page 4, the authors claim that their \"(...) alignment method departs from prior work, in which each non-English language is rotated to match the English embedding space through individual learned matrices.\" However, there is at least one previous paper (Heyman et al., NAACL 2019) which did the same thing as the authors and showed that departing from learning projections only to English leads to more robust multilingual embeddings. Further, also on Page 4, the authors discuss that the assumption on learning good rotation matrices relies on the assumption of rough/approximate isomorphism without citing a body of related work that actually investigated this assumption such as the work of Sogaard et al. (ACL 2018). Also, the paper should do a better job in Section 2 and cover \"word vector alignment\" in more detail (e.g., a good starting point might be Ruder et al.'s survey paper on cross-lingual word embeddings).\n\nThe assumption of rough/approximate isomorphism is problematic also for non-contextual cross-lingual embeddings in settings with more distant language pairs. The authors mention that it may not hold for 'contextual pre-trained models given their increased complexity'. This is very imprecise writing taking place imho: 1) it is not clear why it should not hold in the case of contextual pre-trained models (at least for similar languages). Are there any properties of the contextual models that invalidate that assumption? It is also not exactly shown why contextual pre-trained models have increased complexity compared to e.g. fastText. How does one measure that 'model complexity' in objective terms? In fact, the paper would contribute immensely from more precise writing: e.g., on Page 3 contextual alignment of the model f is defined as accuracy in contextual word retrieval. This reads as defining a critical concept or a task as an evaluation measure (that measures the success of that task). In Introduction, the paper aims to \"better understand BERT’s multilingualism\", but I do not see how it contributes to our better understanding of BERT's multilingualism besides a pretty straightforward claim that it shows less multilingual potential when doing experiments with Greek and Bulgarian that use different scripts. Figure 2 and Figure 3 also do not bring anything new - the paper seems to just state known facts without proposing new solutions on how to e.g. learn better alignments for Greek or Bulgarian.\n\nOne important analysis aspect is missing from the paper: there are no experiments with more distant language pairs (the most distant language pair is English-Greek). I would like to see more experiments in this space. Another experiment which would contribute to the paper is the analysis of the importance of parallel corpora size. How much does the model lose in its performance by shrinking the parallel corpus? We cannot expect having 2M sentences for so many language pairs, and, even if we do have the data, the paper does not convince me that I should use 'Aligned BERT' instead of e.g. the XLM model of Lample and Conneau.\n\nMinor remarks:\nAs a variant of the contextual word retrieval, have the authors tested if a correct target language sentence can be retrieved only looking at the context of the source language word? This would provide some insight on the importance of modeling context via BERT versus via simple context averaging.\n\nRegarding the analysis between closed-class and open-class words performance, the difference in performance can be due to mere frequency: closed-class word types are very scarce, but their corpus frequency is quite high which also leads to learning better representations in the first place, as well as better alignments later on.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgZ3a56KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method to further align multilingual BERT by learning a transformation to minimize distances in a parallel corpus.\n\nI think that this is overall a solid work. Although simple, the proposed method is well-motivated, and the reported results are generally convincing. However, I think that the paper lacks an appropriate comparison with similar methods in the literature, and the separation between the real evaluation in a downstream task (XNLI) and the analysis on a rather artificial contextual word retrieval task (which favors the proposed system) is not clear enough.\n\nMore concretely, these are the aspects that I think the paper could (and should) improve:\n\n- You are not comparing to any baseline using parallel data with contextual embeddings. You should at least compare your method to Schuster et al. (2019) and/or Aldarmaki & Diab (2019), who further align multilingual BERT in a supervised manner as you do, as well as Lample and Conneau (2019), who propose an alternative method to leverage parallel data during the training of multilingual BERT. In fact, while you do improve over multilingual BERT, your results in XNLI are far from the current state-of-the-art, and this is not even mentioned in the paper.\n\n- The \"contextual word retrieval\" task you propose is rather artificial and lacks any practical interest. It is not surprising that your proposed method is strong at it, as this is essentially how you train it (you are even using different subsets of the exact same corpus for train/test). The task is still interesting for analysis -which is in fact one of the main strengths of the paper- but it should be presented as such. Please consider restructuring your paper and moving all these results to the analysis section, where they really belong.\n\n- I do not see the point of the \"non-contextual word retrieval\" task, when you are in fact using the context (the fact that there is only one occurrence per word type doesn't change that). This task is even more artificial than the \"contextual word retrieval\" one. Again, it can have some interest as part of the analysis (showing that the gap between aligned fasttext and aligned BERT goes down from table 1 to table 2), but presenting it as a separate task as if it had some value on its own looks wrong. From my point of view, the real \"non-contextual word retrieval\" task would be bilingual lexicon induction (i.e. dictionary induction), which is more interesting as a task (as the induced dictionaries can have practical applications) and has been widely studied in the literature.\n\n- I really dislike the statement that contextual methods are \"unequivocally better than non-contextual methods for multilingual tasks\" on the basis of the non-contextual word retrieval results. If you want to make such a strong statement, you should at least show that your method is better than non-contextual ones in a task where the latter are known to be strong (i.e. bilingual lexicon induction, see above). However, your comparison is limited to a new task you introduce that clearly favors your own method, and in fact requires using the non-contextual methods in a non-standard way (concatenating the word embeddings with the avg/max/min sentence embeddings). Please either remove this statement or run a fair comparison in bilingual lexicon induction (and preferably do both).\n\n- BERT works at the subword level but, from what I understand, your parallel corpus (both for train/test) is aligned at the word level. It is not clear at all how this mismatch in the tokenization is handled.\n\n\nMinor details that did not influence my score:\n\n- Calling \"fully-supervised\" to the \"translate-train\" system is misleading. Please simply call it \"translate-train\".\n\n- I assume you want to refer to Figure 3 instead of Figure 2 in Section 5.2", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to improve alignments of a multilingual contextual embedding model (e.g., multilingual BERT) using parallel corpora as an anchor. The authors show the benefit of their approach in a zero-shot XNLI experiment and present a word retrieval analysis to better understand multilingual BERT.

All reviewers agree that this is an interesting paper with valuable contributions. The authors and reviewers have been engaged in a thorough discussion during the rebuttal period and the revised paper has addressed most of the reviewers concerns.

I think this paper would be a good addition to ICLR so I recommend accepting this paper.
Decision: Accept (Poster)