言語処理学会 第26回年次大会 発表論文集 (2020年3月)

Questioning the Use of Bilingual Lexicon Induction
as an Evaluation Task for Bilingual Word Embeddings
Benjamin Marie
Atsushi Fujita
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{bmarie, atsushi.fujita}@nict.go.jp
1Introduction

BLI tasks, we identify several incorrect practices, suchas the use of inconsistent evaluation settings. One ofWord embeddings have been shown useful in a wide our most surprising findings is that the seminal work
range of natural language processing tasks. Recently, by Mikolov et al. (2013) still remains a very competimany methods have been proposed to transform mono- tive method for BLI.
lingual word embeddings into bilingual word embeddings (BWE), relying on a small seed bilingual lexicon 2 Methods Evaluatedas a weak supervision (Mikolov et al., 2013). More
We sampled a small number of popular methods forrecently, unsupervised methods, i.e., that do not use
learning BWE to be re-evaluated on BLI tasks. Notebilingual resources for training, have also been prothat since we aim at analyzing common practices inposed (Artetxe et al., 2017; Lample et al., 2018).
evaluating BWE, our conclusions would hold for other
Most of the methods for learning BWE are evaluatedBWE methods.
through bilingual lexicon induction (BLI) tasks, wherea test set comprises 1k–3k source words each paired 2.1 Weakly-supervised BWEwith one or several correct word translations in the tarMiko (Mikolov et al., 2013): a projection matrix isget language. For each source word, a translation istrained from source word embeddings to the tarretrieved from the target language vocabulary relying
get embedding space. This method requires theonly on the BWE to be evaluated. Then, the accuracy
lowest computational cost for training among theof the retrieval, i.e., the percentage of source words formethods we re-evaluate. It is also often considwhich one of the correct word translations has been reered as a weak baseline method for BLI tasks,
trieved, is computed as a quality measure of the BWE.
since it has been shown to underperform more rePrevious work on BWE has claimed the superiority ofcent methods.
a method for learning BWE if its accuracy is higher
than other evaluated methods. BLI remains the most Miko-c (Artetxe et al., 2016): a modified version of
Miko that pre-processes the word embeddings
used evaluation task since the work by Mikolov et al.
with mean centering and length normalization be(2013), mainly because of its low computational costfore projection.
and the relatively low cost of creating many test sets
for several language pairs.
VM16 (Artetxe et al., 2016): similar to Miko-c but
In this paper, we focus on assessing how general are
the projection is performed through orthogonal
the conclusions drawn from BLI-based evaluations ofmapping as proposed by Xing et al. (2015). ThisBWE. The purpose of this paper is to shed light on
method has been shown to outperform Miko.
the limits of current practices rather than proposing alternative evaluation methods or finding which BWE VM17-S (Artetxe et al., 2017): significantly different from Miko, since it relies on a self-learning
performs the best. We re-evaluate nine methods foralgorithm that first induces a poorly accurate
learning BWE through BLI tasks with various experibilingual lexicon and refines it iteratively. For
mental settings varying in the test set, the monolingual
initialization of the training, we used the sameword embeddings to transform, and the method for retraining dictionaries used for training othertrieving word translations. These are independent ofweakly-supervised methods.
the methods used for training BWE. In other words,
the methods and their parameters remain unchanged VM18-S (Artetxe et al., 2018a): a generalization ofprevious work that uses a multi-step framework toacross our experiments. Through our experiments topre-process and post-process projected word emhighlight the significant limits of evaluating BWE on― 1225 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

beddings. This method has been shown to outper- any overlap between the test data and the training
form previous work in BLI tasks.
data of the weakly-supervised methods, we used MuseMuse-S (Lample et al., 2018): similar to VM17-S training data when performing an evaluation on Musebut uses a different algorithm for inducing the test sets and Vecmap training data for an evaluation on
Vecmap test sets. These training data, also provided
bilingual lexicon in the refinement steps.
with Muse and Vecmap test sets, contain 5,000 wordpairs for very frequent source words.
2.2 Unsupervised BWE
For contrastive experiments with a non-European
VM17-U (Artetxe et al., 2017): similar to VM17-S
language pair, we added English-to-Japanese (en-ja).
but uses a bilingual dictionary made of pairs of
Word embeddings were trained as for Vecmap-emb on
numbers for initialization.
the NTCIR monolingual data in patent domain (GotoMuse-U (Lample et al., 2018): performs first adver- et al., 2013). The training and testing bilingual lexsarial training to compute the mapping of BWE icons were created by ourselves following the sameand generate a bilingual lexicon that is then used procedure proposed by Dinu et al. (2014).6 Evento train Muse-S and refine the mapping. This though Japanese and English are distant languages, we
method outperforms previous work but has been can expect a reasonable accuracy of BWE for this BLI
shown to be highly unstable by subsequent work task since all the used data are from the same con(Artetxe et al., 2018b; Søgaard et al., 2018).
trolled domain.
To train BWE, we used the Vecmap toolkit for
VM18-U (Artetxe et al., 2018b): an extension ofMiko,
Miko-c, VM16, VM17-S, VM17-U, VM18-S,
VM18-S and VM17-S for unsupervised BWE.
andVM18-U,
and the Muse toolkit for Muse-S andIt has been shown to consistently outperform
Muse-U.
previous work, even weakly-supervised BWE.
To retrieve the best translation in the target vocabulary of 200k words, we used the Vecmap toolkit
3 Datasets and Tools
with nearest neighbors (NN) or cross-domain similarity scaling (CSLS) (Lample et al., 2018).
We re-evaluated all the methods with publicly available frameworks and datasets for English-to-German 4 Impact of Using Consistent Settings(en-de), English-to-Spanish (en-es), English-to-Italian(en-it), and English-to-Finnish (en-fi). For each lan- The left half of Table 1 shows the results on all the test
guage, we used two sets of word embeddings for 200k sets obtained with Vecmap-emb and CSLS. Since eachwords (300 dimensions): Wikipedia-emb, trained method had been proposed to outperform the other prewith fastText on the Wikipedia data of the corre- existing methods, we grouped the methods in the tablesponding language,1 and Vecmap-emb, trained us- according to their (pre-)publication dates. Methods ining word2vec on diverse corpora.2,3 We expect BWE a lower group in the table should thus outperform thosetrained using Wikipedia-emb to be much more ac- presented in the upper groups.
First of all, none of the evaluated methods consiscurate than with Vecmap-emb, since Wikipedia corpora are comparable to some degree across languages, tently worked best for all the test sets. More imporwhile Vecmap-emb has been trained on much more di- tantly, previously proposed methods, such as Mikoand Miko-c, outperformed more recent methods forverse sets of monolingual corpora.
As test bilingual lexicon for evaluation, we used many test sets. This happens because different methfour Muse test sets,4,5 containing only frequent source ods have so far been compared in combination withwords of Wikipedia, and four Vecmap test sets, con- different translation retrieval methods, such as NN, intaining source words of various frequency. To avoid verted softmax (Smith et al., 2017), or CSLS. For instance, Artetxe et al. (2018b) concluded that VM18-U1https://github.com/facebookresearch/surpasses VM18-S, on the basis of a comparison befastText/blob/master/pretrained-vectors.md
2
Wacky corpora (English, Italian, German), Common Crawl tween VM18-S’s accuracy with inverted softmax andcorpus (Finnish), and News Crawl corpus (Spanish).
VM18-U’s accuracy with CSLS. In contrast, through3
https://github.com/artetxem/vecmap
We use the official Muse test sets.
5
https://github.com/facebookresearch/MUSE
4

6This procedure was also used to create Vecmap bilingual lexicons.

― 1226 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Method

Muse test set
en-de en-es en-fi en-it

CSLS
Vecmap test set
en-de en-es en-fi en-it

NTCIRen-ja

Muse test set
en-de en-es en-fi en-it

NNVecmap test set
en-de en-es en-fi en-it

NTCIRen-ja

Miko66.35 54.23 54.86 56.97

44.00 35.80 32.87 45.60

57.00

59.31 46.04 48.77 39.75

35.00 27.73 25.91 34.93

52.60

Miko-cVM16

66.49 54.23 55.89 59.85
65.13 54.63 52.53 58.04

43.47 35.33 33.64 46.53
47.27 35.73 33.92 45.07

55.67
54.93

59.44 45.03 49.11 48.99
63.71 51.21 50.82 51.41

35.07 26.47 25.91 38.47
41.87 31.40 30.62 39.27

51.47
55.27

VM17-S
VM17-U

60.93 48.32 44.38 53.22
60.05 46.98 45.07 53.89

47.13 33.07 30.76 43.87
46.60 31.60 29.85 44.33

55.53
53.67

59.44 45.37 43.84 47.72
59.38 43.09 44.45 47.65

40.80 28.07 28.86 39.33
40.73 28.47 26.33 38.47

54.53
54.40

Muse-S
Muse-U
VM18-S62.69 54.23 48.63 56.90
0.00 0.00 0.00 57.04
63.98 55.77 50.48 63.34

46.80 36.47 31.88 45.33
0.00 0.00 0.00 44.93
47.20 38.20 34.97 47.33

54.40
54.27
51.27

62.15 50.60 48.97 51.47
0.00 0.00 0.00 51.88
64.45 56.17 50.62 58.85

40.93 31.00 28.37 39.60
0.00 0.00 0.00 39.27
44.27 36.47 32.79 43.80

55.20
55.43
54.07

VM18-U

64.45 56.17 47.19 59.25

48.47 37.40 32.94 48.27

52.87

63.44 52.75 45.62 55.43

43.87 34.00 29.92 43.53

54.67

Table 1: Accuracy using Vecmap-emb. Bold indicates the best score in each column.

a consistent use of CSLS for all the evaluated methods, we conclude that weakly-supervised BWE still
remains competitive to or better than unsupervised
BWE in BLI. For instance, we observed differencesof more than 8 and 4 points of accuracy for Muse enfi and Muse en-it test sets, respectively. These results
point out that, for a fair comparison of BWE through
BLI tasks, we must use the same translation retrieval
method for all the evaluated BWE to draw meaningfulconclusions.
Muse-U achieved an accuracy of 0.00 in manytasks, whereas Lample et al. (2018) did not observe
any 0.00 accuracy thanks to Wikipedia-emb. It is muchharder to learn BWE with Muse-U when using embeddings trained on monolingual corpora from diverse
domains such as Vecmap-emb (see Section 6).

5

The accuracy in BLI tasks can be dramatically improved depending on the method for retrieving theword translations, given some BWE. For instance, inverted softmax and CSLS have both been shown toprovide a better accuracy than NN. Given that the retrieval method has no impact on the embeddings quality itself, we should separately evaluate BWE and theretrieval method for BLI.
In Table 1, we compare the accuracy obtained with
CSLS (left) and NN (right). NN leads to lower accuracy than CSLS for most of the tasks and the BWE
methods. Especially, while Miko appeared to be acompetitive BWE with CSLS, its accuracy dropped bymore than 4 points for all tasks with NN. A more notable finding is that the best BWE (and the ranking ofBWE methods) for each task is not consistent between
CSLS and NN. With NN, VM18-S performed the bestfor 7 out of 9 tasks.
NN and CSLS rely on different aspects of learnedMuse test set
en-de en-es en-fi en-it

Miko

6.27

Miko-c
VM16

8.13 27.60 -2.73 17.94
10.60 26.87 -2.13 18.94

10.27 25.14 -2.67 12.20
6.00 24.34 -5.76 14.20

VM17-S12.87 33.33 -2.33 23.19

5.87

Muse-S
Muse-U
VM18-S12.99 27.66 -0.86 20.67
74.87 82.13 43.13 20.99
10.53 25.13 -0.20 14.13

7.14 23.93 -2.39 14.28
54.93 60.07 28.23 14.34
6.47 22.00 -3.58 10.47

VM18-U

11.67 26.48

6.53

26.53 -6.20 18.80

3.67

19.86

Vecmap test set
en-de en-es en-fi en-it
8.60

24.00 -5.14 12.00

27.53 -5.27 16.73

24.20 -2.61 12.46

Table 2: Difference of corrected accuracy obtained with
Wikipedia-emb and Vecmap-emb through CSLS.

BWE, and from the above observations, we conclude
that the performance of some BWE in BLI tasks can becompared only with the same word translation retrieval
method.

6

Impact of the Retrieval Methods

MethodImpact of the Monolingual WordEmbeddings

Lample et al. (2018) reported much higher accuracies
when using Wikipedia-emb, for Muse test sets, than
the accuracies observed in Sections 4 and 5. We assume that Wikipedia-emb makes the BLI tasks easierand report in Table 2 on the relative changes of the corrected accuracy7 when using Wikipedia-emb insteadof Vecmap-emb.
BWE trained on Wikipedia-emb largely outperformed BWE trained on Vecmap-emb. For instance,
we obtained more than 20 accuracy points of improvements for en-es test sets, and an absolute accuracy
above 60.00 for all test sets of en-de, en-es, and en-it.
7

The percentage of source words in a test set whose correcttranslation is found by a particular setting. Since the Wikipediaemb and Vecmap-emb vocabularies partially cover the word pairs
in the Vecmap and Muse test sets, respectively, we compute thecoverage c of the given BWE, as defined in the Vecmap toolkit,
and the corrected accuracy by multiplying the accuracy by c. We
do not provide the results for VM17-U, since Wikipedia-emb does
not provide embeddings for numerals.

― 1227 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

51
50
Accuracy

49of Glavaš et al. (2019) also shows that we need tore-assess existing baselines and design new evaluation
protocols for BWE.

VM18-S
Miko

Miko-c
VM16

VM18-UMuse-S

48
47

References46

VM17-S
VM17-U

45
44
2013

2014

2015
2016
2017
Pre-publication date

2018

2019

Figure 1: Average accuracy across eight public test setsachieved by each method with Vecmap-emb and CSLS.
Muse-U is located far below.

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016.
Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of EMNLP.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017.
Learning bilingual word embeddings with (almost) nobilingual data. In Proceedings of ACL.

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a.
Generalizing and improving bilingual word embedding
As reported in Lample et al. (2018), Muse-U becomes
mappings with a multi-step framework of linear transformations. In Proceedings of AAAI.
competitive when trained on Wikipedia-emb.

One exception appears for en-fi test sets: using
Wikipedia-emb led to a lower accuracy for most of ourevaluated BWE. We assume that the Finnish Wikipedia
corpus was not large enough to train accurate monolingual word embeddings. In fact, this corpus was significantly smaller than the Wikipedia corpora for all theother languages, and than the Finnish Common Crawl
corpus used to train Finnish Vecmap-emb.
Another finding is that Wikipedia-emb helps unsupervised methods, i.e., Muse-U and VM18-U, much
more than their weakly-supervised counterparts, i.e.,Muse-S and VM18-S. Furthermore, with Wikipediaemb, we have yet another conclusion, i.e., unsupervised BWE are better than weakly-supervised BWE
for most of the tasks, in contradiction to what we concluded from Vecmap-emb (Section 4). As shown by
Søgaard et al. (2018), a possible explanation is thatthe comparability of Wikipedia corpora facilitates unsupervised training.

7Summary

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. A
robust self-learning method for fully unsupervised crosslingual mappings of word embeddings. In Proceedingsof ACL.
Georgiana Dinu, Angeliki Lazaridou, and Marco G Baroni.
2014. Improving zero-shot learning by mitigating thehubness problem. CoRR, abs/1412.6568.
Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan
Vulić. 2019. How to (properly) evaluate cross-lingual
word embeddings: On strong baselines, comparative
analyses, and some misconceptions. pages 710–721.
Isao Goto, Ka Po Chow, Bin Lu, Eiichiro Sumita, and Benjamin K. Tsou. 2013. Overview of the patent machinetranslation task at the NTCIR-10 workshop. In Proceedings of NTCIR Conference.
Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Wordtranslation without parallel data. In Proceedings ofICLR.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168.

We demonstrated that conclusions drawn by previous
work are not valid in various settings according to BLI Samuel L. Smith, David H. P. Turban, Steven Hamblin, andNils Y. Hammerla. 2017. Offline bilingual word vectors,tasks. In other words, the superiority of some BWE
orthogonal transformations and the inverted softmax. In
can only be demonstrated for particular BLI experiProceedings of ICLR.
mental settings and not in general as in done by previous work. Moreover, even though recent methods Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. On
the limitations of unsupervised bilingual dictionary infor BWE are optimized for BLI, using CSLS during
duction. In Proceedings of ACL.
training (Artetxe et al., 2018b; Lample et al., 2018),potentially overfitting to the BLI task, it remains un- Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015.
Normalized word embedding and orthogonal transform
clear whether any significant progress has really been
for bilingual word translation.
In Proceedings ofachieved in BWE for BLI in the past 5 years, as ilNAACL-HLT.
lustrated by Figure 1. Moreover, a concurrent work

― 1228 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.