Title: Should All Cross-Lingual Embeddings Speak English?

Abstract: Most of recent work in cross-lingual word embeddings is severely Anglocentric. The vast majority of lexicon induction evaluation dictionaries are between English and another language, and the English embedding space is selected by default as the hub when learning in a multilingual setting. With this work, however, we challenge these practices. First, we show that the choice of hub language can significantly impact downstream lexicon induction performance. Second, we both expand the current evaluation dictionary collection to include all language pairs using triangulation, and also create new dictionaries for under-represented languages. Evaluating established methods over all these language pairs sheds light into their suitability and presents new challenges for the field. Finally, in our analysis we identify general guidelines for strong cross-lingual embeddings baselines, based on more than just Anglocentric experiments.

1. Introduction:

Continuous distributional vectors for representing words (embeddings) (Turian et al., 2010) have become ubiquitous in modern, neural NLP. Cross-lingual representations (Mikolov et al., 2013) additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging (Zhang et al., 2016) , parsing (Ammar et al., 2016) , document classification (Klementiev et al., 2012) , and machine translation (Irvine and Callison-Burch, 2013; Artetxe et al., 2018b; Lample et al., 2018) .

Often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter BWE and MWE, respectively) . First, monolingual word embeddings are learned over large swaths of text; such pre-trained word embeddings, in fact, are available for several languages and are widely used, like the fastText Wikipedia vectors (Grave et al., 2018) . Second, a mapping between the languages is learned, in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision (Zou et al., 2013) , under minimal supervision e.g. using only identical strings (Smith et al., 2017) , or even in a completely unsupervised fashion (Zhang et al., 2017; Conneau et al., 2018) . Both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned to (hereinafter "the hub"). We outline the details in Section 2. Despite all the recent progress in learning cross-lingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. Notably, most MWE approaches essentially select English as the hub during training by default, aligning all other language spaces to the English one. We argue and empirically show, however, that English is a poor hub language choice. In BWE settings, on the other hand, it is fairly uncommon to denote which of the two languages is the hub (often this is implied to be the target language). However, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages. This Anglocentricity is even more evident at the evaluation stage. The lexica most commonly used for evaluation are the MUSE lexica (Conneau et al., 2018) which cover 45 languages, but with translations only from and into English. Even still, alternative evaluation dictionaries are also very English-and European-centric: Dinu and Baroni (2014) report results on English-Italian, Artetxe et al. (2017) on English-German and English-Finnish, Zhang et al. (2017) on Spanish-English and Italian-English, and Artetxe et al. (2018a) between English and Italian, German, Finnish, Spanish, and Turkish. We argue that cross-lingual word embedding mapping methods should look beyond English for their evaluation benchmarks because, compared to all others, English is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems (Aronoff and Fudeman, 2011) . These two factors allow for an overly easy evaluation setting which does not necessarily generalize to other language pairs. In light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages.

With this work, we attempt to address these shortcomings, providing the following contributions:

• We show that the choice of the hub when evaluating on diverse language pairs can lead to significantly different performance (e.g., by more than 10 percentage points for BWE over distant languages). We also show that often English is a suboptimal hub for MWE. • We identify some general guidelines for choosing a hub language which could lead to stronger baselines; less isometry between the hub and source and target embedding spaces mildly correlates with performance, as does typological distance (a measure of language similarity based on language family membership trees). For distant languages, multilingual systems should in most cases be preferred over bilingual ones. • We provide resources for training and evaluation on non-Anglocentric language pairs. We outline a simple triangulation method with which we extend the MUSE dictionaries to an additional 2352 lexicons covering 49 languages, and we present results on a subset of them.

We also create new evaluation lexica for under-resourced languages using Azerbaijani, Belarusian, and Galician as our test cases. We additionally provide recipes for creating such dictionaries for any language pair with available parallel data.

2 Cross-Lingual Word Embeddings and Lexicon Induction

In the supervised bilingual setting, as formulated by Mikolov et al. (2013) , given two languages L = {l 1 , l 2 } and their pre-trained row-aligned embeddings X 1 , X 2 , respectively, a transformation matrix M is learned such that:

M = arg min M ∈Ω X 1 -M X 2 .

The set Ω can potentially impose a constraint over M , such as the very popular constraint of restricting it to be orthogonal (Xing et al., 2015) . Previous work has empirically found that this simple formulation is competitive with other more complicated alternatives (Xing et al., 2015; Conneau et al., 2018) . The orthogonality assumption ensures that there exists a closed-form solution in the form of the Singular Value Decomposition (SVD) of X 1 X T 2 .1 Note that in this case only a single matrix M needs to be learned, because X 1 -M X 2 = M -1 X 1 -X 2 , while at the same time a model that minimizes X 1 -M X 2 is as expressive as one minimizing M 1 X 1 -M 2 X 2 , and easier to learn.

In the minimally supervised or even the unsupervised setting (Zhang et al., 2017) the popular methods follow an iterative refinement approach (Artetxe et al., 2017) . Starting with a seed dictionary (e.g. from identical strings (Zhou et al., 2019) or numerals) an initial mapping is learned in the same manner as in the supervised setting. The initial mapping, in turn, is used to expand the seed dictionary with high confidence word translation pairs. The new dictionary is then used to learn a better mapping, and so forth the iterations continue until convergence. We will generally refer to such methods as MUSE-like.

Similarly, in a multilingual setting, one could start with N languages L = {l 1 , l 2 , . . . , l N } and their respective pre-trained embeddings X 1 , X 2 , . . . , X N , and then learn N -1 bilingual mappings between a pre-selected target language and all others. Hence, one of the language spaces is treated as a target (the hub) and remains invariant, while all others are mapped into the (now shared) hub language space. Alternatively, those mappings could be jointly learned using the MAT+MPSR methods of Chen and Cardie (2018) -also taking advantage of the inter-dependencies between any two language pairs. Importantly, though, there is no closed form solution for learning the joint mapping, hence a solution needs to be approximated with gradient-based methods. MAT+MPSR generalizes the adversarial approach of Zhang et al. (2017) to multiple languages, and also follows an iterative refinement ειρηνικά pacifici, pacifica, pacifico approach very similar to that of MUSE-like methods. 2 In either case, a language is chosen as the hub, and N -1 mappings for the other languages are learned.

Other than MAT+MPSR, the only other unsupervised multilingual approach is that of Heyman et al. (2019) , who propose to incrementally align multiple languages by adding each new language as a hub. We decided, though, against comparing to this method, because (a) their method requires learning O(N 2 ) mappings for relatively small improvements and (b) the order in which the languages are added is an additional hyperparameter that would explode the experimental space. 3Lexicon Induction One of the most common downstream evaluation tasks for the learned crosslingual word mappings is Lexicon Induction (LI), the task of retrieving the most appropriate wordlevel translation for a query word from the mapped embedding spaces. Specialized evaluation (and training) dictionaries have been created for multiple language pairs, with the MUSE dictionaries (Conneau et al., 2018) most often used, providing word translations between English (En) and 48 other high-to mid-resource languages, as well as on all 30 pairs among 6 very similar Romance and Germanic languages (English, French, German, Spanish, Italian, Portuguese).

Given the mapped embedding spaces, the translations are retrieved using a distance metric, with Cross-Lingual Similarity Scaling (Conneau et al., 2018, CSLS) as the most common and best performing in the literature. Intuitively, CSLS decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). The retrieved pairs are compared to the gold standard and evaluated using precision at k (P@k, evaluating how often the correct translation is within the k retrieved nearest neighbours of the query). Throughout this work we report P@1, which is equivalent to accuracy, but we also provide results with P@5 and P@10 in the Appendix.

3. New LI Evaluation Dictionaries:

As other works have recently noted (Czarnowska et al., 2019) the typically used evaluation dictionaries cover a narrow breadth of the possible language pairs, with the majority of them focusing in pairs with English (as with the MUSE dictionaries) or among high-resource European languages. In this section, we first outline our method for creating new dictionaries for low resource languages. Then, we describe the simple triangulation process that allows us to create dictionaries among all 49 MUSE languages.

3.1. Low-Resource Language Dictionaries:

Our approach for constructing dictionaries is fairly straightforward, inspired by phrase table extraction techniques from phrase-based MT (Koehn, 2009) . Rather than manual inspection, however, which would be impossible for all language pairs, we rely on fairly simple heuristics for controlling the quality of our dictionaries.

The first step is collecting publicly available parallel data between English and the low-resource language of interest. We use data from the TED (Qi et al., 2018) , OpenSubtitles (Lison and Tiedemann, 2016) , WikiMatrix (Schwenk et al., 2019) , bible (Malaviya et al., 2017) , and JW300 (Agić and Vulić, 2019) datasets. 4 This results in 354k, 53k, and 623k English-to-X parallel sentences for Azerbaijani (Az), Belarusian (Be), and Galician (Gl) respectively. 5 We align the parallel sentences using fast align (Dyer et al., 2013) , and extract symmetrized alignments using the gdfa heuristic (Koehn et al., 2005) . In order to ensure that we do not extract highly domain-specific word pairs, we only use the TED, OpenSubtitles, and WikiMatrix parts for word-pair extraction. Also, in order to control for quality, we only extract word pairs if they appear in the dataset more than 5 times, and if the alignment probability is higher than 30%.

With this process, we end up with about 6k, 7k, and 38k word pairs for Az-En, Be-En, and Gl-En respectively. Following standard conventions, we sort the word pairs according to source-side frequency, and use the intermediate-frequency ones for evaluation, typically using the 5000-6500 rank boundaries. The same process can be followed for any language pair with enough volume of parallel data (needed for training a decent word alignment model). In fact, we can produce similar dictionaries for a large number of languages, as the combination of the recently created JW300 and WikiMatrix datasets provide an average of more than 100k parallel sentences in 300 languages. Our second method for creating new dictionaries is inspired from phrase table triangulation ideas from the pre-neural MT community (Wang et al., 2006; Levinboim and Chiang, 2015) . The concept can be easily explained with an example, visualized in Figure 1 . Consider the Portuguese (Pt) word trabalho which, according to the MUSE Pt-En dictionary, has the words job and work as possible En translations. In turn, these two En words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the transitive property (which translation should exhibit) we can identify the set of 7 possible Cs translations for the Pt word trabalho. Following this simple triangulation approach, we create 2352 new dictionaries over language pairs among the 49 languages of the MUSE dictionaries. 7 For consistency, we keep the same train and test splits as with MUSE, so that the source-side types are equal across all dictionaries with the same source language.

Triangulating through English (which is unavoidable, due to the lack of non-English-centric dictionaries) is suboptimal -English is morphologically poor and lacks gender information. As a result, several inflected forms in morphologically-rich languages map to the same English form. Similarly, gendered nouns or adjectives in gendered languages map to English forms that lack gender information. For example, the MUSE Greek-English dictionary lists the word peaceful as the translation for all ειρηνικός, ειρηνική, ειρηνικό, ειρηνικά, which are the male, female, and neutral (singular and plural) inflections of the same adjective. Equivalently, the English-Italian dictionary translates peaceful into either pacifico, pacifici, or pacifica (male singular, male plural, and female singular, respectively; see Table 1 ). When translating from or into English lacking context, all of those are reasonable translations. When translating between Greek and Italian, though, one should take gender and number into account.

Hence, we devise a filtering method for removing blatant mistakes when triangulating morphologically rich languages. We rely on automatic morphological tagging which we can obtain for most of the MUSE languages, using the StanfordNLP toolkit (Manning et al., 2014) . The morphological tagging uses the Universal Dependencies feature set (Nivre et al., 2016) making the tagging comparable across almost all languages. Our filtering technique iterates through the bridged dictionaries: for a given source word, if we find a translation word with the exact same morphological analysis, we filter out all other translations with the same lemma but different tags. In the case of feature mismatch (for instance, Greek uses 4 cases and 3 genders while Italian has 2 genders and no cases) or if we only find a partial tag match over a feature subset, we filter out translations with disagreeing tags. Coming back to our Greek-Italian example, this means that for the form ειρηνικός we would only keep pacifico as a candidate translation (we show more examples in Table 1 ).

Our filtering technique removes about 17% of the entries in our bridged dictionaries. Naturally, this filtering approach is restricted to languages for which a morphological analyzer is available. Miti-Table 2 : Lexicon Induction performance (measured with P@1) over 10 European languages (90 pairs). In each cell, the superscript denotes the hub language that yields the best result for that language pair. µ best : average using the best hub language. µ En : average using the En as the hub. The shaded cells are the only language pairs where a bilingual MUSE system outperforms MAT+MSPR. 

4. Lexicon Induction Experiments:

For our main MWE experiments, we train MAT+MPSR systems to align several language subsets varying the hub language. For BWE experiments, we compare MUSE with MAT+MPSR. The differences in LI performance show the importance of the hub language choice with respect to each evaluation pair. As part of our call for moving beyond Anglo-centric evaluation, we also present LI results on several new language pairs using our triangulated dictionaries. It is worth noting that we are predominantly interested in comparing the quality of the multilingual alignment when different hub languages are used. Hence, even slightly noisy dictionaries (like our low-resource language ones) are still useful. Even if the skyline performance (from e.g. a perfect system) would not reach 100% accuracy due to noise, the differences between the systems' performance can be revealing.

We first focus on 10 European languages of varying morphological complexity and data availability (which affects the quality of the pre-trained word embeddings): Azerbaijani (Az), Belarusian (Be), Czech (Cs), English (En), Galician (Gl), Portuguese (Pt), Russian (Ru), Slovak (Sk), Spanish (Es), and Turkish (Tr). The choice of these languages additionally ensures that for our three low-resource languages (Az, Be, Gl) we include at least one related higher-resource language (Tr, Ru, Pt/Es respectively), allowing for comparative analysis. Table 2 summarizes the best post-hoc performing systems for this experiment.

In the second setting, we use a set of 7 more distant languages: English, French (Fr), Hindi (Hi), Korean (Ko), Russian, Swedish (Sv), and Ukrainian (Uk). This language subset has large variance in terms of typology and alphabet. The best performing systems are presented in Table 3 .

Experimental Setup We train and evaluate all models starting with the pre-trained Wikipedia FastText embeddings for all languages (Grave et al., 2018) . We focus on the minimally supervised scenario which only uses similar character strings between any languages for supervision in order to mirror the hard, realistic scenario of not having annotated training dictionaries between the languages. We learn MWE with the MAT+MPSR method (Chen and Cardie, 2018) using the publicly available code. 8 We also use MAT+MPSR for BWE experiments, but we additionally train and compare to MUSE systems9 (Conneau et al., 2018) . We compare the statistical significance of the difference in performance from two systems using paired bootstrap resampling (Koehn, 2004) . Generally, a difference of 0.4-0.5 percentage points evaluated over our lexica is significant with p < 0.05. 

BWE::

The hub matters for distant languages When using MUSE, the answer is simple: the closed form solution of the Procrustes problem is provably direction-independent, and we confirm this empirically (we provide complete results on MUSE in Table 15 in the Appendix). However, obtaining good performance with such methods requires the orthogonality assumption to hold, which for distant languages is rarely the case (Patra et al., 2019) . In fact, we find that the gradient-based MAT+MPSR method in a bilingual setting over distant languages exhibits better performance than MUSE. Across Tables 2 and 3 , in only a handful of examples (shaded cells) does MUSE outperform MAT+MPSR for BWE.

On the other hand, we find that when aligning distant languages with MAT+MPSR, the difference between hub choices can be significant -in Az-En, for instance, using En as the hub leads to more than 7 percentage points difference to using Az. We show some examples in Table 4 . On the other hand, when aligning typologically similar languages, the difference is less pronounced. For example, we obtain practically similar performance for Gl-Pt, Az-Tr, or Uk-Ru when using either the source or the target language as the hub. Note, though, that non-negligible differences could still occur, as in the case of Pt-Gl. In most cases, it is the case that the higher-resourced language is a better hub than the lower-resourced one, especially when the number of resources defer significantly (as in the case of Az and Be against any other language). Since BWE settings are not our main focus, we leave an extensive analysis of this observation for future work. MWE: English is rarely the best hub language In multilingual settings, we conclude that the standard practice of choosing English as the hub language is sub-optimal. Out of the 90 evaluation pairs from our European-languages experiment (Table 2 ) the best hub language is English in only 17 instances (less than 20% of the time). In fact, the average performance (over all evaluation pairs) when using En as the hub (denoted as µ En ) is 1.3 percentage points worse than the optimal (µ best ). In our distant-languages experiment (Table 3 ) English is the best choice only for 7 of the 42 evaluation pairs (again, less than 20% of the time). As before, using En as the hub leads to an average drop of one percentage point in performance aggregated over all pairs, compared to the averages of the optimal selection. The rest of the section attempts to provide an explanation for these differences.

Expected gain for a hub language choice As vividly outlined by the superscript annotations in Tables 2 and 3 , there is not a single hub language that stands out as the best one. Interestingly, all languages, across both experiments, are the best hub language for some evaluation language pair. For example, in our European-languages experiment, Es is the best choice for about 20% of the evaluation pairs, Tr and En are the best for about 17% each, while Gl and Be are the best for only 5 and 3 language pairs respectively.

Clearly, not all languages are equally suited to be the hub language for many language pairs. Hence, it would be interesting to quantify how much better one could do by selecting the best hub language compared to a random choice. In order to achieve this, we define the expected gain G l of using language l as follows. Assume that we are interested in mapping N languages into the shared space and p m l is the accuracy10 over a specified evaluation pair m when using language l as the hub. The random choice between N languages will have an expected accuracy equal to the average accuracy when using all languages as hub: The gain for that evaluation dataset m when using language l as hub, then, is

E[p m ] = l p m l N . EN FR HI KO RU SV UK 0 1 2 0.9 1.1 1.2 1.3 1.4 1.7 1.1 0.1 0 -0.3 0 0.4 -0.2 0.1 G l AZ BE CS EN ES GL PT RU SK TR 0 1 2 1.1 1 1.2 1.5 1.5 0.9 1.5 1.5 1 1.5 -0.4 0 0.2 0.2 0.4 0.4 0.2 0 -0.2 0 G l when best overall

g m l = p m l -E[p m ]

. Now, for a collection of M evaluation pairs we simply average their gains, in order to obtain the expected gain for using language l as the hub:

G l = E[g l ] = m g m l M .

The results of this computation for both sets of experiments are presented in Figure 2 . The bars marked 'overall' match our above definition, as they present the expected gain computed over all evaluation language pairs. For good measure, we also present the average gain per language aggregated over the evaluation pairs where that language was indeed the best hub language ('when best' bars). Perhaps unsurprisingly, Az seems to be the worst hub language choice among the 10 European languages of the first experiment, with an expected loss (negative gain) of -0.4. This can be attributed to how distant Az is from all other languages, as well as to the fact that the Az pre-trained embeddings are of lower quality compared to all other languages (as the Az Wikipedia dataset is significantly smaller than the others). Similarly, Hi and Sv show expected loss for our second experiment.

Note that English is not a bad hub choice per se -it exhibits a positive expected gain in both sets of experiments. However, there are languages with larger expected gains, like Es and Gl in the European-languages experiment that have a twice-as-large expected gain, while Ru has a 4 times larger expected gain in the distant-languages experiment. Of course, the language subset composition of these experiments could possibly impact those numbers. For example, there are three very related languages (Es, Gl, Pt) in the European languages set, which might boost the expected gain for that subset; however, the trends stand even if we compute the expected gain over a subset of the evaluation pairs, removing all pairs that include Gl or Pt. For example, after removing all Gl results, Es has a slightly lower expected gain of 0.32, but is still the language with the largest expected gain.

Identifying the best hub language for a given evaluation set The next step is attempting to identify potential characteristics that will allow us make educated decisions with regards to choosing the hub language, given a specific evaluation set. For example, should one choose a language typologically similar to the evaluation source, target, or both? Or should they use the source or the target of the desired evaluation set as the hub?

Our first finding is that the best performing hub language will very likely be neither the source nor the target of the evaluation set. In our European-languages experiments, a language different than the source and the target yields the best accuracy for over 93% of the evaluation sets. Similarly, in the distant-languages experiment, there is only a single instance where the best performing hub language is either the source or the target evaluation language (for the Fr-Ru dataset), and for the other 97% of the cases the best option is a third language. We hypothesize that learning mappings for both language spaces of interest (hence rotating both spaces) allows for a more flexible alignment which leads to better downstream performance, compared to when one of the two spaces is fixed. Note that this contradicts the mathematical intuition discussed in Section 2 according to which a model learning a single mapping (keeping another word embedding space fixed) is as expressive as a model that learns two mappings for each of the languages.

Our second finding is that the downstream performance correlates with measures of distance between languages and language spaces. The typological distance (d gen ) between two languages can be approximated through their genealogical distance over hypothesized language family trees, which we obtain from the URIEL typological database (Littell et al., 2017) . Also, Patra et al. (2019) recently motivated the use of Gromov-Hausdroff (GH) distance as an a priori estimation of how well two language embedding spaces can be aligned under an isometric transformation (which is an assumption most methods rely on). The authors also note that vector space GH distance correlates with typological language distance. We refer the reader to Patra et al. (2019) for more details. We find that there is a positive correlation between downstream LI performance and the genealogical distances between the source-hub and target-hub languages. The average (over all evaluation pairs) Pearson's correlation coefficient between P@1 and d gen is 0.49 for the distant languages experiment and 0.38 for the European languages one. A similar positive correlation of performance and the sum of the GH distances between the source-hub and target-hub spaces. On our distant languages experiment, the coefficient between P@1 and GH is equal to 0.45, while it is slightly lower (0.34) for our European languages experiment. High correlation examples from each experiment, namely Gl-En and En-Hi, are shown in Figure 3 .

Bi-, tri-, and multilingual systems The last part of our analysis compares bilingual, trilingual, and multilingual systems, with a focus on the under-represented languages. Through multiple experiments (complete evaluations are listed in the Appendix) we reach two main conclusions. On one hand, when evaluating on typologically distant languages, one should use as many languages as possible. In Table 5 we present one such example with results on Az-Cs under various settings. On the other hand, when multiple related languages are available, one can achieve higher performance with multilingual systems containing all related languages and one more hub language, rather than learning diverse multilingual mappings using more languages. We confirm the latter observation with experiments on the Slavic (Be, Ru, Uk) and Iberian (Es, Gl, Pt) clusters, and present an example (Ru-Uk) in Table 5 .

5. Conclusion:

With this work we challenge the standard practices in learning cross-lingual word embeddings. We empirically showed that the choice of the hub language is an important parameter that affects lexicon induction performance in both bilingual (between distant languages) and multilingual settings. More importantly, we hope that by providing new dictionaries and baseline results on several language pairs, we will stir the community towards evaluating all methods in challenging scenarios that include under-represented language pairs. Towards this end, our analysis provides insights and general directions for stronger baselines for non-Anglocentric cross-lingual word embeddings.

A Does evaluation directionality matter?

We also explored whether there are significant differences between the evaluated quality of aligned spaces, when computed on both directions (src-trg and trg-src). We find that the evaluation direction indeed matters a lot, when the languages of the evaluation pair are very distant, in terms of morphological complexity and data availability (which affects the quality of the original embeddings). A prominent example, from our European-languages experiment, are evaluation pairs involving Az or Be. When evaluating on the Az-XX and Be-XX dictionaries, the word translation P@1 is more than 20 percentage points higher than when evaluating on the opposite direction (XX-Az or XX-Be). For example, Es-Az has a mere P@1 of 9.9, while Az-Es achieves a P@1 of 44.9. This observation holds even between very related languages (cf. Ru-Be: 12.8, Be-Ru: 41.1 and Tr-Az: 8.4, Az-Tr: 32.0), which supports our hypothesis that this difference is also due to the quality of the pre-trained embeddings. It is important to note that such directionality differences are not observed when evaluating distant pairs with presumably high-quality pre-trained embeddings e.g. Tr-Sk or Tr-Es; the P@1 for both directions is very close.

B Complete results for all experiments:

Here we provide complete evaluation results for our multilingual experiments. Tables 6-11 present P@1, P@5, and P@10 respectively, for the experiment on the 10 European languages. Similarly, results on the distant languages experiment are shown in Tables 12, 13 , and 14. Table 15 presents the P@1 of the bilingual experiments using MUSE. 

:

We refer the reader to(Mikolov et al.,

2013) for more details.

Note that MAT+MPSR has the beneficial property of being as computationally efficient as learning O(N) mappings (instead of O(N 2 )).We refer the reader toChen and Cardie (2018) for exact details.

We refer the reader to Table2fromHeyman et al. (2019) which compares to MAT+MPSR, and to Table7of their appendix which shows the dramatic influence of language order.

Not all languages are available in all these datasets.

Note that the anglocentricity in this step is by necessity -it is hard to find a large volume of parallel data in a language pair excluding English.

We will create these dictionaries and make them publicly available, along with the corresponding code.

Available at AnonymizedURL.

https://github.com/ccsasuke/umwe

https://github.com/facebookresearch/MUSE

This could be substituted with any evaluation metric

Figures and Tables:

Figure 1: Transitivity example.

Figure 2: Expected gain G l for the MWE experiments.

Figure3: The downstream accuracy generally correlates positively with the GH distance of the source and target language vector spaces to the hub language.

Triangulation and filtering example on Greek-Italian. All words are valid translations of the English word 'peaceful'. We also show filtered-out translations.

Table content: <table><tr><td colspan="2">Greek</td><td>Italian</td><td/><td/><td colspan="2">Bridged Greek-Italian Lexicon</td></tr><tr><td>word</td><td>tag</td><td>word</td><td>tag</td><td>Match</td><td>Greek</td><td>Italian</td></tr><tr><td>ειρηνικός</td><td>M;NOM;SG</td><td colspan="2">pacifico M;SG</td><td>M;SG</td><td colspan="2">ειρηνικός pacifico, pacifici, pacifica</td></tr><tr><td>ειρηνική</td><td>F;NOM;SG</td><td colspan="2">pacifici M;PL</td><td>F;SG</td><td colspan="2">ειρηνική pacifica, pacifico, pacifici</td></tr><tr><td colspan="4">ειρηνικό Neut;NOM;SG pacifica F;SG</td><td>SG</td><td colspan="2">ειρηνικό pacifica, pacifico, pacifici</td></tr><tr><td colspan="2">ειρηνικά Neut;NOM;PL</td><td/><td/><td>PL</td><td/><td/></tr></table>

En 35.1 Es 35.7 Es 48.0 Tr 32.7 Ru 41.5 En 29.8 Pt 31.7 Cs 32.0 Pt 33.7 31.7 Be 14.1 Cs -35.9 Tr 29.9 Pt 39.5 En 25.8 Es 34.4 Es 41.1 Gl 30.7 Ru 20.4 Pt 30.2 28.8 Cs 6.9 Es 9.3 Ru -61.0 Es 60.5 En 27.9 Pt 57.8 En 45.9 Pt 71.2 En 35.8 Sk 41.8 41.2 En 17.9 Es 18.4 Es 50.2 Es -77.5 Ru 36.3 Es 72.3 Sk 43.3 Pt 40.4 Tr 41.9 Pt 44.2 42.7 Es 12.1 En 10.1 Ru 47.4 Pt 74.6 Sk -37.5 Es 83.1 Gl 41.9 Tr 40.0 Es 38.6 Sk 42.8 41.4 Gl 5.5 En 3.6 Az 26.5 Tr 43.2 Es 60.8 Tr -52.9 Cs 23.8 Tr 26.8 Cs 19.7 Cs 29.2 27.7 Pt 5.8 Pt 8.6 Sk 47.2 Gl 71.3 En 88.1 Pt 37.1 Es -38.0 Es 38.7 Es 38.1 En 41.4 40.4 Ru 8.7 Es 12.8 Az 50.3 Gl 55.5 Tr 54.8 Cs 23.0 Pt 52.4 En -45.5 Tr 27.0 Be 36.7 35.9 Sk 4.0 Be 10.9 Ru 72.5 Be 55.6 Tr 53.9 En 28.4 En 52.0 Es 44.0 Gl -28.5 En 38.9 37.9 Tr 12.1 Sk 9.0 Az 41.8 Ru 51.1 Cs 55.0 En 18.4 Tr 51.6 En 34.6 En 29.4 is beyond the scope of this work, although it is unfortunately a common issue. For example,Kementchedjhieva et al. (2019) were able to manually correct (filter) five dictionaries (between English and German, Danish, Bulgarian, Arabic, and Hindi) but one would have to rely on automated annotation in order to scale to all languages.

Table content: <table><tr><td>src</td><td>Az</td><td>Be</td><td>Cs</td><td>En</td><td>Es</td><td colspan="2">Target Gl</td><td>Pt</td><td>Ru</td><td>Sk</td><td>Tr</td><td>µ best µ En</td></tr><tr><td>Az</td><td>-</td><td colspan="9">17.2 Es</td><td>-</td><td>33.7 33.0</td></tr><tr><td colspan="2">µ best 9.7</td><td>11.1</td><td>45.2</td><td>53.1</td><td colspan="2">59.8</td><td>29.7</td><td>55.3</td><td>38.0</td><td>39.4</td><td>31.3</td><td>37.3</td></tr><tr><td>µ En</td><td>9.1</td><td>9.9</td><td>43.3</td><td>51.0</td><td colspan="2">59.3</td><td>28.2</td><td>54.9</td><td>36.5</td><td>37.7</td><td>30.8</td><td>36.0</td></tr><tr><td colspan="3">gating this limitation</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Lexicon Induction performance (P@1) over MWEs from 7 typologically distant languages (42 pairs). See Table2for notation.

Table content: <table><tr><td>Source</td><td>En</td><td>Fr</td><td>Hi</td><td>Target Ko</td><td>Ru</td><td>Sv</td><td>Uk</td><td colspan="2">µ best µ En</td></tr><tr><td>En</td><td>-</td><td colspan="8">76.3 Ru 23.9 Uk 10.4 Fr 42.0 Uk 59.0 Hi 28.3 Ru 40.0 38.5</td></tr><tr><td>Fr</td><td>74.0 Uk</td><td>-</td><td colspan="7">19.0 Ru 7.5 Sv 40.8 Ru 51.8 En 28.8 En 37.0 36.4</td></tr><tr><td>Hi</td><td colspan="2">31.4 Fr 26.9 Ru</td><td>-</td><td colspan="6">2.1 En 14.6 Uk 17.3 En 10.5 Fr 17.1 16.2</td></tr><tr><td>Ko</td><td colspan="3">17.7 Sv 13.6 Sv 2.4 Fr</td><td>-</td><td colspan="4">7.9 En 7.2 Ru 3.6 Fr 8.8</td><td>7.9</td></tr><tr><td>Ru</td><td colspan="4">53.4 Ko 51.7 Ko 15.3 Uk 5.2 En</td><td>-</td><td colspan="4">41.3 Uk 56.3 Ko 37.2 36.2</td></tr><tr><td>Sv</td><td colspan="5">52.7 Uk 48.2 Ko 17.7 Ru 5.1 Uk 33.2 Fr</td><td>-</td><td colspan="3">24.1 Ru 30.2 29.2</td></tr><tr><td>Uk</td><td colspan="6">41.4 Ru 44.0 Hi 14.4 Sv 2.6 En 59.7 Hi 36.8 Ko</td><td>-</td><td colspan="2">33.2 32.4</td></tr><tr><td>µ best</td><td>45.1</td><td>43.5</td><td>15.5</td><td>5.5</td><td>33.0</td><td>35.6</td><td>25.3</td><td>29.1</td></tr><tr><td>µ En</td><td>42.7</td><td>42.5</td><td>14.5</td><td>5.1</td><td>32.4</td><td>34.9</td><td>24.5</td><td/><td>28.1</td></tr><tr><td colspan="3">4.1 Analysis and Takeaways</td><td/><td/><td/><td/><td/><td colspan="2">Table 4: The hub is important for BWE be-</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">tween distant languages.</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Test</td><td>Hub src</td><td>trg</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Az-Cs 22.7 29.1</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Az-En 13.2 20.7</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Az-Tr 30.1 30.1</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Gl-Pt 53.5 53.6</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Pt-Gl 39.0 36.7</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Uk-Ru 61.6 61.8</td></tr></table>

Comparison of bilingual, trilingual, and multilingual systems for distant (left) and related (right) languages. Multilinguality boosts performance significantly on distant languages.

Table content: <table><tr><td colspan="2">Results on Az-Cs</td><td colspan="2">Average</td><td/><td>Results on Ru-Uk</td><td>Average</td></tr><tr><td>Bilingual with hub:</td><td>Az 22.7 29.1 Cs</td><td/><td>25.8</td><td>Bilingual with hub:</td><td>Ru 58.0 57.0 Uk</td><td>57.5</td></tr><tr><td colspan="2">Trilingual Az, Cs, +hub:</td><td/><td/><td colspan="2">Trilingual Be, Ru, Uk with hub:</td></tr><tr><td colspan="3">Be 21.6 28.5 31.8 23.0 En Es Gl Pt Ru Sk Tr 29.6 27.4 30.4 32.9 Trilingual Az, hub:Cs, +extra:</td><td>28.2</td><td colspan="3">Be 59.2 58.9 58.4 Ru Uk Trilingual Ru, Uk, +hub: Az Cs En Es Fr 57.4 58.5 58.4 58.3 58.0 57.0 57.2 Hi Tr</td><td>58.8 57.8</td></tr><tr><td colspan="3">En 30.1 30.1 33.2 27.1 33.7 Es Pt Ru Tr Multilingual (10 languages) Az Be Cs En Es</td><td>30.8</td><td colspan="3">Multilingual Be, Ru, Uk, +hub: Cs En Es Gl Ko 58.0 58.1 58.5 58.8 57.0 58.3 58.2 Pt Sv</td><td>58.1</td></tr><tr><td colspan="3">33.7 34.0 32.3 34.5 35.1 Gl Pt Ru Sk Tr 34.0 34.8 34.5 32.9 33.7</td><td>33.9</td><td colspan="3">Multilingual Ru, Uk, En, Fr, Hi, Ko, Sv, with hub: En Fr Hi Ko Ru Sv Uk 55.3 56.1 55.8 56.3 55.3 55.3 54.9</td><td>55.6</td></tr></table>

All results from the European-languages MWE experiment: P@1 (part 1).

Table content: <table><tr><td>Test</td><td>Az</td><td>Be</td><td>Cs</td><td>En</td><td>Hub language Es Gl</td><td>Pt</td><td>Ru</td><td>Sk</td><td>Tr</td><td>µ</td></tr></table>

All results from the European-languages MWE experiment: P@5 (part 1).

Table content: <table><tr><td>Test</td><td>Az</td><td>Be</td><td>Cs</td><td>En</td><td>Hub language Es Gl</td><td>Pt</td><td>Ru</td><td>Sk</td><td>Tr</td><td>µ</td></tr></table>

All results from the European-languages MWE experiment: P@5 (part 2).

Table content: <table><tr><td>Test</td><td>Az</td><td>Be</td><td>Cs</td><td>En</td><td>Hub language Es Gl</td><td>Pt</td><td>Ru</td><td>Sk</td><td>Tr</td><td>µ</td></tr></table>

All results from the European-languages MWE experiment: P@10 (part 1).

Table content: <table><tr><td>Test</td><td>Az</td><td>Be</td><td>Cs</td><td>En</td><td>Hub language Es Gl</td><td>Pt</td><td>Ru</td><td>Sk</td><td>Tr</td><td>µ</td></tr></table>

All results from the European-languages MWE experiment: P@10 (part 2).

Table content: <table><tr><td>Test</td><td>Az</td><td>Be</td><td>Cs</td><td>En</td><td>Hub language Es Gl</td><td>Pt</td><td>Ru</td><td>Sk</td><td>Tr</td><td>µ</td></tr></table>

All results from the distant languages MWE experiment (P@1).

Table content: <table><tr><td>Test</td><td>En</td><td>Fr</td><td>Hub language Hi Ko Ru</td><td>Sv</td><td>Uk</td><td>µ</td></tr></table>

All results from the distant languages MWE experiment (P@5).

Table content: <table><tr><td>Test</td><td>En</td><td>Fr</td><td>Hub language Hi Ko Ru</td><td>Sv</td><td>Uk</td><td>µ</td></tr></table>

All results from the distant languages MWE experiment (P@10).

Table content: <table><tr><td>Test</td><td>En</td><td>Fr</td><td>Hub language Hi Ko Ru</td><td>Sv</td><td>Uk</td><td>µ</td></tr></table>

Peer review (iclr2020):
This paper explores the importance of the pivot language when learning multilingual word embeddings. While previous work has been severely English-centric, the paper introduces a new dataset covering all combinations of 49 languages, derived automatically from the MUSE dataset through triangulation, and analyze the performance of two existing mapping methods in it for different choices of the pivot language. I think that the paper is well written and technically sound, and both the new dataset and the analysis done are valuable contributions. However, I am not convinced that it has enough novelty and substance for the venue: a similar point was already made by Alaux et al. (ICLR*18), although the authors do not seem to be aware of it, the dataset was derived automatically from an existing one using an error-prone (and rather trivial) procedure, the analysis is somewhat limited (e.g. covering only two mapping methods) and, while still valuable, the results do not seem very significant to me. More concretely: - The main point that the paper tries to sell is that the choice of the pivot language is important, and always picking English, as previous work did, is suboptimal. I have been actively working on this area, so when I read the title and the abstract I was excited to learn more about this finding. However, as I read through the paper, my excitement went down, and I was left with the feeling that the pivot language was not that important after all. In Table 2 and 3, the difference between English and the best configuration is around 1 point, which is rather small for this task. To make things worse, even that difference is questionable, as the best configuration is chosen in the test set (more on this below). - I think that picking the best configuration in the test set, and comparing that to one of the specific configurations, is methodologically problematic. To make an analogy, let*s say that I want to claim that the random seed is important when learning cross-lingual word embeddings with a given method, and picking 5468721, as some implementation does by default, may be suboptimal. So I make multiple runs with different random seeds and show that, indeed, 5468721 is rarely the optimal random seed. This would not surprise anybody, as the same thing applies to pretty much any work in ML. So, coming back to your work, is the choice of the pivot language anyhow different from a random seed? I do believe that there is something more than just randomness in your results, but unless I am missing something, you are not controlling this factor, and the differences you report look too small to ignore it. - The whole idea of the paper is actually not very novel. Alaux et al. (ICLR*2019, https://openreview.net/forum?id=HJe62s09tX) also showed that mapping all embeddings into English was suboptimal, and proposed an alternative approach. The authors do not seem to be aware of this work, as they do not even cite it. I am not saying that there is nothing new or interesting in the paper, but this is clearly a weakness, as the work done is not put in context appropriately. - The evaluation is limited to two methods, and some influential systems, like VecMap, are left out. I understand that asking to evaluate dozens of methods would be not be reasonable, but I think that the limited scope of the analysis is a weakness if you want to make general claims on the limitations of previous work. - Somewhat ironically, the proposed dataset is automatically derived from the MUSE one in an English-centric way. The authors use a simple triangulation approach for that. This is error-prone, as the authors acknowledge and try to mitigate with a post-processing that detects and removes some wrong entries. To make things worse, previous work (Kementchedjhieva et al., EMNLP*2019) has also found issues in the original MUSE dataset, which your derived one will unavoidably inherit. For that reason, I am not entirely convinced about the value of the resulting dataset. - The paper focuses exclusively in Bilingual Lexicon Induction (BLI), which is common practice in this research area but has been recently questionned. Glavas et al. (ACL*2019) showed that BLI-centric evaluation was problematic and, as mentioned before, Kementchedjhieva et al. (EMNLP*2019) found issues in the MUSE dataset, which you automatically extend. This does not invalidate all your work, but I do see it as a weakness. Other details that did not influence my score but I think the authors should address: - The term *pivot* seems more appropriate than *hub*. *Hub* is usually used with a different meaning in the context of cross-lingual word embeddings (in relation to the *hubness problem*), and *pivot* seems more standard for what you mean. - Iterative refinement methods are named *MUSE-like*, which seems unfair, as this technique was not introduced by MUSE. Why not just say *iterative refinement* (or *self-learning*)? Otherwise you should credit the original authors that proposed this idea, which I believe are Artetxe et al. (ACL*2018). - *We hypothesize that learning mappings for both language spaces of interest (hence rotating both spaces) allows for a more flexible alignment which leads to better downstream performance [...] Note that this contradicts the mathematical intuition discussed in Section 2 according to which a model learning a single mapping (keeping another word embedding space fixed) is as expressive as a model that learns two mappings for each of the languages.* Assuming that the mappings are orthogonal, this is not just a *mathematical intuition*, it is a provable fact, so it does not make sense to propose a hypothesis that contradicts it. If you find a surprising pattern that you cannot explain, you should acknowledge it, but proposing a hypothesis that is known to be false does not make any sense. - This is somewhat subjective, but I dislike the term *European language* as it is used in the paper. It does not make much sense from a linguistic point of view (Turkish is not an Indoeuropean language), and it is even questionable from a geographical point of view, which should not be very relevant, anyway (e.g. English, Spanish and Portuguese have more speakers outside of Europe than within Europe, and most of Turkey is actually in Asia).