Venue: ACL 2020

# Should All Cross-Lingual Embeddings Speak English?

Antonios Anastasopoulos and Graham Neubig

Language Technologies Institute, Carnegie Mellon University

{aanastas,gneubig}@cs.cmu.edu

###### Abstract

Most of recent work in cross-lingual word embeddings is severely Anglocentric. The vast majority of lexicon induction evaluation dictionaries are between English and another language, and the English embedding space is selected by default as the _hub_ when learning in a multilingual setting. With this work, however, we challenge these practices. First, we show that the choice of hub language can significantly impact downstream lexicon induction and zero-shot POS tagging performance. Second, we both expand a standard English-centered evaluation dictionary collection to include all language pairs using triangulation, and create new dictionaries for under-represented languages.1 Evaluating established methods over all these language pairs sheds light into their suitability for aligning embeddings from distant languages and presents new challenges for the field. Finally, in our analysis we identify general guidelines for strong cross-lingual embedding baselines, that extend to language pairs that do not include English.

Footnote 1: Available at [https://github.com/antonisa/embeddings](https://github.com/antonisa/embeddings).

+
Footnote †: _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8658–8679

## 1 Introduction

Continuous vectors for representing words (embeddings) Turian et al. (2010) have become ubiquitous in modern, neural NLP. Cross-lingual representations Mikolov et al. (2013) additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging Zhang et al. (2016), parsing Ammar et al. (2016), document classification Klementiev et al. (2012), and machine translation Irvine and Callison-Burch (2013); Artetxe et al. (2018); Lample et al. (2018).

Often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter BWE and MWE, respectively). First, monolingual word embeddings are learned over large swaths of text. Such pre-trained word embeddings, such as the fastText Wikipedia vectors Grave et al. (2018), are available for many languages and are widely used. Second, a mapping between the languages is learned in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision Zou et al. (2013), under minimal supervision e.g. using only identical strings Smith et al. (2017), or even in an unsupervised fashion Zhang et al. (2017); Conneau et al. (2018). Both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned (hereinafter "the _hub_"). We outline the details in Section 2.

Despite all the recent progress in learning cross-lingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. Notably, most MWE approaches essentially select English as the _hub_ during training by default, aligning all other language spaces to the English one. We argue and empirically show, however, that English is a poor hub language choice. In BWE settings, on the other hand, it is fairly uncommon to denote which of the two languages is the _hub_ (often this is implied to be the target language). However, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages.

This Anglocentricity is even more evident at the evaluation stage. The lexica most commonly used for evaluation are the MUSE lexicio Conneau et al. (2018) which cover 45 languages, but with translations only from and into English. Alternative evaluation dictionaries are also very English- and European-centric: Dinu and Baroni (2014) report results on English-ItalianArtexte et al. (2017) on English-German and English-FinnishZhang et al. (2017) on Spanish-English and Italian-EnglishArtetxe et al. (2018) between English and ItalianGerman et al. (2018). We argue that cross-lingual word embedding mapping methods should look beyond English for their evaluation benchmarks because, compared to all others, English is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems Aronoff and Fudeman (2011). These two factors allow for an

overly easy evaluation setting which does not necessarily generalize to other language pairs. In light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages.

With this work, we attempt to address these shortcomings, providing the following contributions:

* We show that the choice of the _hub_ when evaluating on diverse language pairs can lead to significantly different performance for iterative refinement methods that use a symbolic-based seed dictionary (e.g., by more than 10 percentage points for BWE over distant languages). We also show that often English is a suboptimal hub for MWE.
* We identify some general guidelines for choosing a hub language which could lead to stronger performance; _less_ isometry between the hub and source and target embedding spaces mildly correlates with performance, as does typological distance (a measure of language similarity based on language family membership trees). For distant languages, multilingual systems should be preferred over bilingual ones if the languages share alphabets, otherwise a bilingual system based on monolingual similarity dictionaries is preferable.
* We provide resources for training and evaluation on language pairs that do not include English. We outline a simple triangulation method with which we extend the MUSE dictionaries to an additional 4704 lexicons covering 50 languages (for a total of 4900 dictionaries, including the original English ones), and we present results on a subset of them. We also create new evaluation lexic for under-resourced, under-represented languages using Azerbaijan, Belarusian, and Galician as our test cases. Finally, we provide recipes for creating such dictionaries for any language pair with available parallel data.

## 2 Cross-Lingual Word Embeddings and Lexicon Induction

Bilingual Word EmbeddingsIn the supervised BWE setting of Mikolov et al. (2013), given two languages \({\cal L}=\{l_{1},l_{2}\}\) and their pre-trained _row-aligned_ embeddings \({\cal X}_{1},{\cal X}_{2}\), respectively, a transformation matrix \(M\) is learned such that:

The set \(\Omega\) can potentially impose a constraint over \(M\), such as the very popular constraint of restricting it to be orthogonal Xing et al. (2015). Previous work has empirically found that this simple formulation is competitive with other more complicated alternatives Xing et al. (2015). The orthogonality assumption ensures that there exists a closed-form solution through Singular Value Decomposition (SVD) of \(X_{1}{\cal X}^{T}_{2}\).2 Note that in this case only a single matrix \(M\) needs to be learned, because \(\|{\cal X}_{1}-{\mathbf{M}}{\cal X}_{2}\|=\left\lVert{\mathbf{M}}^{-1}{\cal X}_{1}-{ \cal X}_{2}\right\rVert\), while at the same time a model that minimizes \(\|{\cal X}_{1}-{\mathbf{M}}{\cal X}_{2}\|\) is as expressive as one minimizing \(\|M_{1}{\cal X}_{1}-M_{2}{\cal X}_{2}\|\), with half the parameters.

Footnote 2: We refer the reader to Mikolov et al. (2013) for details.

In the minimally supervised or even the unsupervised setting, Zhang et al. (2017) and Conneau et al. (2018) reframe the task as an adversarial game, with a generator aiming to produce a transformation that can fool a discriminator. However, the most popular methods follow an iterative refinement approach Artetxe et al. (2017). Starting with a seed dictionary (e.g. from identical strings Zhou et al. (2019) or numerals) an initial mapping is learned in the same manner as in the supervised setting. The initial mapping, in turn, is used to expand the seed dictionary with high confidence word translation pairs. The new dictionary is then used to learn a better mapping, and so forth the iterations continue until convergence. The same iterative approach is followed by Artetxe et al. (2018), with one important difference that allows their model (VecMap) to handle language pairs with different alphabets: instead of identical strings, the seed dictionary is constructed based on the similarity of the monolingual similarity distributions over all words in the vocabulary.3

Footnote 3: We refer the reader to Artetxe et al. (2018) for details.

Multilingual Word EmbeddingsIn a multilingual setting, the simplest approach is to use BWE and align all languages into a target language (the _hub_). In this case, for \(N\) languages \({\cal L}=\{l_{1},l_{2},\ldots,l_{N}\}\) on has to learn \(N-1\) bilingual mappings Ammar et al. (2016). Rather than using a single hub space, Heyman et al. (2019) propose an incremental procedure that uses an Incremental Hub Space (IHS): each new language is included to the multilingual space by mapping it to all languages that have already been aligned (e.g. language \(l_{3}\) would be mapped to the aligned space of \(\{l_{1},l_{2}\}\)).

Alternatively, all mappings could be learned jointly, taking advantage of the inter-dependencies between any two language pairs. Importantly, though, there is no closed form solution for learning the joint mapping, hence a solution needs to be approximated with gradient-based methods. The main approaches are:

* Multilingual adversarial training with pseudo-randomized refinement Chen and Cardiie (2018), MAT+MPSR): a generalization of the adversarial approach of Zhang et al. (2017); Conneau et al. (2018) to multiple languages, also combined with an iterative refinement procedure.4 Footnote 4: NAT+MPSR has the beneficial property of being as computationally efficient as learning \(\mathcal{O}(N)\) mappings (instead of \(\mathcal{O}(N^{j})\)). We refer the reader to Chen and Cardiie (2018) for exact details.
* Unsupervised Multilingual Hyperalignment Alaux et al. (2 

which maps all languages to a single _hub_ space,5 but also enforces good alignments between all language pairs within this space.

Footnote 4: Note that Alaux et al. (2019) use the term _pivot_ to refer to what we refer to as the _hub_ language.

Even though the architecture and modeling approach of all MWE methods are different, they share the same conceptual traits: one of the language spaces remains invariant and all other languages are effectively mapped to it. In all cases, English is by default selected to be the hub. The only exception is the study of triplets alignments in Alaux et al. (2019), where Spanish is used as the Spanish-French-Portuguese triplet hub.

Lexicon InductionOne of the most common downstream evaluation tasks for the learned cross-lingual word mappings is Lexicon Induction (LI), the task of retrieving the most appropriate word-level translation for a query word from the mapped embedding spaces. Specialized evaluation (and training) dictionaries have been created for multiple language pairs. Of these, the MUSE dictionaries Conneau et al. (2018) are most often used, providing word translations between English (En) and 48 other high- to mid-resource languages, as well as on all 30 pairs among 6 very similar Romance and Germanic languages English, French, German, Spanish, Italian, Portuguese).

Given the mapped embedding spaces, the translations are retrieved using a distance metric, with Cross-Lingual Similarity Scaling Conneau et al. (2018), CSLS) as the most commonly used in the literature. Intuitively, CSLS decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). The retrieved pairs are compared to the gold standard and evaluated using precision at \(k\) (P@\(k\), evaluating how often the correct translation is within the \(k\) retrieved nearest neighbors of the query). Throughout this work we report P@1, which is equivalent to accuracy; we provide P@5 and P@10 results in the Appendix.

## 3 New LI Evaluation Dictionaries

The typically used evaluation dictionaries cover a narrow breadth of the possible language pairs, with the majority of them focusing in pairs with English (as with the MUSE or Dinu et al. (2015) dictionaries) or among high-resource European languages. Glavas et al. (2019), for instance, highlighted Anglocentricly as an issue, creating and evaluating on 28 dictionaries between 8 languages Croatian, English, Finnish, French, German, Italian, Russian, Turkish) based on Google Translate. In addition, Czarnowska et al. (2019) focused on the morphology dimension, creating morphologically complete dictionaries for 2 sets of 5 _genetically related_ languages (Romance: French, Spanish, Italian, Portuguese, Catalan; and Slavic: Polish, Czech, Slovak, Russian, Ukrainian). In contrast to these two (very valuable!) works, our method for creating dictionaries for low-resource languages (SS3.1) leverages resources that are available for about 300 languages. In addition, we propose a simple triangulation process (SS3.2), that makes it possible to create dictionaries for arbitrary language pairs, given that dictionaries into a pivot language (usually English) are available for both languages.

### Low-Resource Language Dictionaries

Our approach for constructing dictionaries is straightforward, inspired by phrase table extraction techniques from phrase-based MT Koehn (2009). This is an automatic process, and introduces some degree of noise. Rather than controlling this through manual inspection, which would be impossible for all language pairs, we rely on fairly simple heuristics for controlling the dictionaries' quality.

The first step is collecting publicly available parallel data between English and the low-resource language of interest. We use data from the TED Qi et al. (2018), OpenSubtitles Lison and Tiedemann (2016), WikiMatrix Schwenk et al. (2019), bible Malaviya et al. (2017), and JW300 Agic and Vulic (2019) datasets.6 This results in 354k, 53k, and 623k English-to-X parallel sentences for Azerbaijan (Az), Belarusian (Be), and Galian (Gu) respectively.7 We align the parallel sentences using fast_alignDyer et al. (2013), and extract symmetrized alignments using the gdfa heuristic Koehn et al. (2005). In order to ensure that we do not extract highly domain-specific word pairs, we only use the TED, OpenSubtitles, and WikiMatrix parts for wordpair extraction. Also, in order to control for quality, we only extract word pairs if they appear in the dataset more than 5 times, _and_ if the symmetrized alignment probability is higher than 30% in both directions.

Footnote 7: Not all languages are available in all these datasets.

With this process, we end up with about 6k, 7k, and 38k word pairs for Az-En, Be-En, and Glc-En respectively. Following standard conventions, we sort the wordpairs according to source-side frequency, and use the intermediate-frequency ones for evaluation, typically unsiting the [5000-6500] rank boundaries. The same process can be followed for any language pair with a sufficient volume of parallel data (needed for training a reasonably accurate word alignment model).8 



### Dictionaries for all Language Pairs through Triangulation

Our second method for creating new dictionaries is inspired by phrase table triangulation ideas from the pre-neural MT community (Wang et al., 2006; Levinboim and Chiang, 2015). The concept can be easily explained with an example, visualized in Figure 1. Consider the Portuguese (Pr) word trabalho which, according to the MUSE Pr-EN dictionary, has the words job and work as possible Ex translations. In turn, these two En words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the _transitive_ property (which translation should exhibit) we can identify the set of 5 possible Cs translations for the Pr word trabalho. Following this simple triangulation approach, we create 4,704 new dictionaries over pairs between the 50 languages of the MUSE dictionaries.9 For consistency, we keep the same train and test splits as with MUSE, so that the source-side types are equal across all dictionaries with the same source language.

Footnote 9: Available at [https://github.com/antonisa/embeddings](https://github.com/antonisa/embeddings).

Triangulating through English (which is unavoidable, due to the relative paucity of non-English-centric dictionaries) is suboptimal - English is morphologically poor and lacks corresponding markings for gender, case, or other features that are explicitly marked in many languages. As a result, several inflected forms in morphologically-rich languages map to the same English form. Similarly, gendered nouns or adjectives in gendered languages map to English forms that lack gender information. For example, the MUSE Greek-English dictionary lists the word peaceful as the translation for all eupnyuko, eupnyuki, eupnyuko, eupnyuko, which are the male, female, and neutral (singular and plural) inflections of the same adjective. Equivalently, the English-Italian dictionary translates peaceful into either pacifico, pacifici, or pacifica (male singular, male plural, and female singular, respectively; see Table 1). When translating from or into English _lacking context_, all of those are reasonable translations. When translating between Greek and Italian, though, one should at least take number into account (grammatical gender is a more complicated matter: it is not uncommon for word translations to be of different grammatical gender across languages).

Hence, we devise a filtering method for removing blatant mistakes when triangulating morphologically rich languages. We rely on automatic morphological tagging which we can obtain for most of the MUSE languages, using the StanfordNLP toolkit (Qi et al., 2020).10 The morphological tagging uses the Universal Dependencies feature set (Nivre et al., 2016) making the tagging comparable across almost all languages. Our filtering technique iterates through the bridged dictionaries: for a given source word, if we find a target word with the exact same morphological analysis, we filter out all other translations with the same lemma but different tags. In the case of feature mismatch (for instance, Greek uses 2 numbers, 4 cases and 3 genders while Italian has 2 numbers, 2 genders, and no cases) or if we only find a partial tag match over a feature subset, we filter out translations with disagreeing tags. We ignore the grammatical gender and verb form features, as they are not directly comparable cross-lingually. Coming back to our Greek-Italian example, this means that for the form eupnyuko? we would only keep pacifico as a candidate translation (we show more examples in Table 1).

Footnote 10: The toolkit has since been renamed to Stanza. See [https://stanfordnlp.github.io/stanfordnlp/](https://stanfordnlp.github.io/stanfordnlp/).

Our filtering technique removes about 60.4% of the entries in 2964 of the 4900 dictionaries.11 Unsurprisingly, we find that bridged dictionaries between morphologically rich languages require a lot more filtering. For instance more than 80% of the entries of the Urdu-Greek dictionary get filtered out. On average, the languages with more filtered entries are Urdu (62.4%), Turkish (61.1%), and German (58.6%). On the other hand, much fewer entries are removed from dictionaries with languages like Dutch (36.2%) or English (38.1%). Naturally, this filtering approach is restricted to languages for which a morphological analyzer is available. Mitigating this limitation is beyond the scope of this work, although it is unfortunately a common issue. For example, Kementchedjhieva et al. (2019) manually corrected five dictionaries (between English and German, Danish, Bulgarian, Arabic, Hindi) but one needs to rely

\begin{table}
\begin{tabular}{c c|c c||c c} \hline \hline  & Greek & Italian & \multicolumn{3}{c}{Bridged Greek–Italian Lexicon} \\ word & tag & word & tag & Match & Greek & Italian \\ \hline eupnyuko & M;NOM;SG & pacifico & M;SG & Eupnyukö & pacifico, paetifei, pacifica \\ exptynki & F;NOM;SG & pacifici & M;PL & F;SG & Eupnyukü & pacifico, pacificoi \\ exptynkö & Neut;NOM;SG & pacifica & F;SG & Sg & Eupnyukö & pacifica, pacifico \\ exptynkö & Neut;NOM;PL & & PL & Eupnyukö & pacifici, pacificae \\ \hline \hline \end{tabular}
\end{table}
Table 1: Triangulation and filtering example on Greek–Italian. All words are valid translations of the English word ‘peaceful’. We also show filtered—out-transitions.



on automated annotations in order to scale to all languages. Our method that uses automatically obtained morphological information combined with the guidelines proposed by Kementchedjhieva et al. (2019) (e.g. removing proper nouns from the evaluation set) scales easily to multiple languages, allowing us to create more than 4 thousand dictionaries.

## 4 Lexicon Induction Experiments

The aim of our LI experiments is two-fold. First, the differences in LI performance show the importance of the hub language choice with respect to each evaluation pair. Second, as part of our call for moving beyond Anglo-centric evaluation, we also present LI results on several new language pairs using our triangulated dictionaries.

### Methods and Setup

We train and evaluate all models starting with pre-trained Wikipedia FastText embeddings for all languages Grave et al. (2018). We focus on the minimally supervised scenario which only uses similar character strings between any languages for supervision in order to mirror the hard, realistic scenario of not having annotated training dictionaries between the languages. We learn MWE with the MAT+MPSR method using the publicly available code,12 aligning several language subsets varying the hub language. We decided against comparing to the incremental hub (IHS) method of Heyman et al. (2019), because the order in which the languages are added is an additional hyperparameter that would explode the experimental space.13 We also do not compare to UMH, as we consider it conceptually similar to MAT+MPSR and no code is publicly available. For BWE experiments, we use MUSEs14 (MUSE, semisupervised) and VecMap15 systems, and we additionally compare them to MAT+MPSR for completeness.

Footnote 12: [https://github.com/cccsasuke/umwe](https://github.com/cccsasuke/umwe)

Footnote 13: We refer the reader to Table 2 from Heyman et al. (2019) which compares to MAT+MPSR, and to Table 7 of their appendix which shows the dramatic influence of language order.

We compare the statistical significance of the performance difference of two systems using paired bootstrap resampling Koehn (2004). Generally, a difference of \(0.4{-}0.5\) percentage points evaluated over our lexica is significant with \(p<0.05\).

**Experiment 1** We first focus on 10 languages of varying morphological complexity and data availability (which affects the quality of the pre-trained word embeddings): AzerbaijanAzzbarianAzBelarusianBeCzechCsEnglishEnGalicAlionPortuguesePrbRussianRucSlovakSkuSpanishEsandTurkishTraNThechoice of these languages additionally ensures that for our three low-resource languages (Az, Be, Glt) we include at least one related higher-resource language (Tr, Ru, Pr/Es respectively), allowing for comparative analysis. Table 2 summarizes the best post-hoc performing systems for this experiment.

**Experiment 2** In the second setting, we use a set of 7 more distant languages: EnglishFrenchFrHindiHKoreanKoRussian, RussianSwedishSvD, and UkrainianUkrainVkInThis language subset has large variance in terms of typology and alphabet. The best performing systems are presented in Table 3.

### Analysis and Takeaways

#### Mwe: English is rarely the best hub language

In multilingual settings, we conclude that the standard practice of choosing English as the hub language is sub-optimal. Out of the 90 evaluation pairs from our 10-language experiment (Table 2) the best hub language is English in only 17 instances (less than 20% of the

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c} \hline \hline src & \multicolumn{6}{c}{Target} \\  & Az & Be & Cs & En & Es & Glc & Pr & Rv & Sk & Tr & \(\mu^{\text{test}}\) & \(\mu^{\text{Ex}}\) \\ \hline Az & – & 17.2\({}^{\text{Ext}}\) & 35.1\({}^{\text{Est}}\) & 35.7\({}^{\text{Est}}\) & 48.0\({}^{\text{Th}}\) & 32.7\({}^{\text{Rev}}\) & 41.5\({}^{\text{Ex}}\) & 29.8\({}^{\text{Pr}}\) & 31.7\({}^{\text{Cr}}\) & 32.0\({}^{\text{Pr}}\) & 33.7 & 31.7 \\ Be & 14.1\({}^{\text{Cl}}\) & – & 35.9\({}^{\text{Er}}\) & 29.9\({}^{\text{Br}}\) & 39.5\({}^{\text{Esc}}\) & 25.8\({}^{\text{Ra}}\) & 34.4\({}^{\text{ES}}\) & 41.1\({}^{\text{Helo}}\) & 30.7\({}^{\text{Re}}\) & 20.4\({}^{\text{Pr}}\) & 30.2 & 28.8 \\ Cs & 6.9\({}^{\text{Es}}\) & 9.3 \({}^{\text{Rel}}\) & – & 61.0\({}^{\text{Et}}\) & 60.5\({}^{\text{Er}}\) & 27.9\({}^{\text{Pr}}\) & 57.8\({}^{\text{L}}\) & 45.9\({}^{\text{Ir}}\) & 11.2\({}^{\text{Int}}\) & 35.8\({}^{\text{As}}\) & 41.8 & 41.2 \\ Ex & 17.9\({}^{\text{Bis}}\) & 18.4\({}^{\text{St}}\) & 50.2\({}^{\text{Ep}}\) & – & 17.5\({}^{\text{St}}\) & 36.3\({}^{\text{Te}}\) & 22.8\({}^{\text{Abs}}\) & 43.3\({}^{\text{Hy}}\) & 40.4\({}^{\text{Ta}}\) & 41.9\({}^{\text{Ar}}\) & 42.4 & 24.7 \\ Es & 12.1\({}^{\text{Ev}}\) & 10.1\({}^{\text{Kr}}\) & 47.4\({}^{\text{Pr}}\) & 21.6\({}^{\text{Gr}}\) & – & 37.5\({}^{\text{St}}\) & 33.1\({}^{\text{Li}}\) & 41.9\({}^{\text{An}}\) & 40.0\({}^{\text{Es}}\) & 38.6\({}^{\text{Ks}}\) & 42.8 & 41.4 \\ GL & 5.5\({}^{\text{Em}}\) & 3.6 \({}^{\text{Ax}}\) & 26.5\({}^{\text{Da}}\) & 43.2\({}^{\text{Ec}}\) & 60.8\({}^{\text{Ne}}\) & — & 52.9\({}^{\text{Ge}}\) & 23.8\({}^{\text{La}}\) & 26.6\({}^{\text{Sc}}\) & 19.7\({}^{\text{Ce}}\) & 29.2 & 27.7 \\ Pr & 5.8\({}^{\text{Pr}}\) & 8.6 \({}^{\text{Sx}}\) & 47.2\({}^{\text{Ga}}\) & 71.3\({}^{\text{She}}\) & 88.1\({}^{\text{Sr}}\) & 37.1\({}^{\text{Es}}\) & – & 38.0\({}^{\text{Ok}}\) & 38.7\({}^{\text{El}}\) & 38.1\({}^{\text{Eu}}\) & 41.4 & 40.4 \\ RU & 8.7\({}^{\text{Es}}\) & 12.8\({}^{\text{Avd}}\) & 50.3\({}^{\text{Se}}\) & 55.5\({}^{\text{Sh}}\) & 54.5\({}^{\text{Sa}}\) & 23.0\({}^{\text{Er}}\) & 52.4\({}^{\text{Ex}}\) & – & 14.5\({}^{\text{Sn}}\) & 27.0\({}^{\text{Me}}\) & 36.7 & 35.9 \\ SK & 4.0\({}^{\text{Bde}}\) & 10.9\({}^{\text{Rh}}\) & 75.5\({}^{\text{Sm}}\) & 55.6\({}^{\text{Sb}}\) & 53.9\({}^{\text{Es}}\) & 28.4\({}^{\text{Sz}}\) & 52.0\({}^{\text{Es}}\) & 44.0\({}^{\text{Oh}}\) & – & 28.5\({}^{\text{Sb}}\) & 38.9 & 37.9 \\ Ta & 12.1\({}^{\text{Sk}}\) & 9.0\({}^{\text{Ac}}\) & 41.8\({}^{\text{St}}\) & 51.1\({}^{\text{Ca}}\) & 55.0\({}^{\text{Es}}\) & 18.4\({}^{\text{St}}\) & 51.6\({}^{\text{Sch}}\) & 34.6\({}^{\text{De}}\) & 29.4\({}^{\text{Es}}\) & – & 33.7 & 33.0 \\ \hline \hline \end{tabular} \(\mu^{\text{test}}\) & 9.7 & 11.1 & 45.2 & 53.1 & 59.8 & 29.7 & 55.3 & 38.0 & 39.4 & 31.3 & 37.3 \\ \(\mu^{\text{Ex}}\) & 9.1 & 9.9 & 43.3 & 51.0 & 59.3 & 28.2 & 54.9 & 36.5 & 37.7 & 30.8 & 36.0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Lexicon Induction performance (measured with P@1) over 10 languages (90 pairs). In each cell, the superscript denotes the hub language that yields the best result for that language pair. \(\mu^{\text{msr}}\): average using the best hub language. \(\mu^{\text{Env}}\): average using the Env as the hub. The lightly shaded cells are the language pairs where a bilingualVecMap system outperformsMAT+MSPR; in heavy shaded cells both MUSEs and VecMap outperformMAT+MSPR.

 

time). In fact, the average performance (over all evaluation pairs) when using En as the hub (denoted as \(\mu^{\text{Ex}}\)) is 1.3 percentage points worse than the optimal (\(\mu^{\text{next}}\)). In our distant-languages experiment (Table 3) English is the best choice only for 7 of the 42 evaluation pairs (again, less than 20% of the time). As before, using En as the hub leads to an average drop of one percentage point in performance aggregated over all pairs, compared to the averages of the optimal selection. The rest of this section attempts to provide an explanation for these differences.

Expected gain for a hub language choiceAs vividly outlined by the superscript annotations in Tables 2 and 3, there is not a single hub language that stands out as the best one. Interestingly, all languages, across both experiments, are the best hub language for some evaluation language pair. For example, in our 10-languages experiment, Es is the best choice for about 20% of the evaluation pairs, Tr and En are the best for about 17% each, while Gl and Be are the best for only 5 and 3 language pairs respectively.

Clearly, not all languages are equally suited to be the hub language for many language pairs. Hence, it would be interesting to quantify how much better one could do by selecting the best hub language compared to a random choice. In order to achieve this, we define the expected gain \(G_{I}\) of using language \(l\) as follows. Assume that we are interested in mapping \(N\) languages into the shared space and \(p_{l}^{m}\) is the accuracy16 over a specified evaluation pair \(m\) when using language \(l\) as the hub. The random choice between \(N\) languages will have an expected accuracy equal to the average accuracy when using all languages as hub:

Footnote 16: This could be substituted with any evaluation metric.

\[\mathbb{E}[p^{m}]=\frac{\sum_{l}p_{l}^{m}}{N}.\]

The gain for that evaluation dataset \(m\) when using language \(l\) as hub, then, is \(g_{l}^{m}=p_{l}^{m}-\mathbb{E}[p^{m}]\). Now, for a collection of \(M\) evaluation pairs we simply average their gains, in order to obtain the expected gain for using language \(l\) as the hub:

\[G_{l}=\mathbb{E}[g_{l}]=\frac{\sum_{m}g_{l}^{m}}{M}.\]

The results of this computation for both sets of experiments are presented in Figure 2. The bars marked 'overall' match our above definition, as they present the expected gain computed over all evaluation language pairs. For good measure, we also present the average gain per language aggregated over the evaluation pairs where that language was indeed the best hub language ('when best' bars). Perhaps unsurprisingly, Az seems to be the worst hub language choice among the 10 languages of the first experiment, with an expected loss (negative gain) of -0.4. This can be attributed to how distant Az is from all other languages, as well as to the fact that the Az pre-trained embeddings are of lower quality compared to all other languages (as the Az Wikipedia dataset is significantly smaller than the others). Similarly, Hi and Sv show expected loss for our second experiment.

Note that English is not a _bad_ hub choice _per se_ - it exhibits a positive expected gain in both sets of experiments. However, there are languages with larger expected gains, like Es and Gl in the 10-languages experiment that have a twice-as-large expected gain, while Ru has a 4 times larger expected gain in the distant

Figure 2: Expected gain \(G_{I}\) for the MWE experiments.

 

languages experiment. Of course, the language subset composition of these experiments could possibly impact those numbers. For example, there are three very related languages (Es, Gl, Pr) in the 10 languages set, which might boost the expected gain for that subset; however, the trends stand even if we compute the expected gain over a subset of the evaluation pairs, removing all pairs that include GL or Pr. For example, after removing all GL results, Es has a slightly lower expected gain of 0.32, but is still the language with the largest expected gain.

Identifying the best hub language for a given evaluation setThe next step is attempting to identify potential characteristics that will allow us make educated decisions with regards to choosing the hub language, given a specific evaluation set. For example, should one choose a language typologically similar to the evaluation source, target, or both? Or should they use the source or the target of the evaluation set as the hub?

Our first finding is that the best performing hub language will very likely be neither the source nor the target of the evaluation set. In our 10-languages experiments, a language different than the source and the target yields the best accuracy for over 93% of the evaluation sets, with the difference being statistically significant in more than half such cases. Similarly, in the distant-languages experiment, there is only a single instance where the best performing hub language is either the source or the target evaluation language (for Fr-Ru), and for the other 97% of cases the best option is a third language. This surprising pattern contradicts the mathematical intuition discussed in Section 2 according to which a model learning a single mapping (keeping another word embedding space fixed) is _as expressive_ as a model that learns two mappings for each of the languages. Instead, we find that in almost all cases, learning mappings for both language spaces of interest (hence rotating both spaces) leads to better BLI performance compared to when one of the spaces is fixed.

Our second finding is that the LI performance correlates with measures of distance between languages and language spaces. The topological distance (\(d^{gen}\)) between two languages can be approximated through their genealogical distance over hypothesized language family trees, which we obtain from the URIEL typological database Littell et al. (2017). Also, Patra et al. (2019) recently motivated the use of Gromov-Hausdorff (GH) distance as an _a priori_ estimation of how well two language embedding spaces can be aligned under an isometric transformation (an assumption most methods rely on). The authors also note that vector space GH distance correlates with typological language distance.

We find that there is a _positive_ correlation between LI performance and the genealogical distances between the source-hub and target-hub languages. The average (over all evaluation pairs) Pearson's correlation coefficient between P@1 and \(d^{gen}\) is 0.49 for the distant languages experiment and 0.38 for the 10-languages one. A similar positive correlation of performance and the sum of the GH distances between the source-hub and target-hub spaces. On our distant languages experiment, the correlation coefficient between P@1 and GH is 0.45, while it is slightly lower (0.34) for our 10-languages experiment. Figure 3 shows two high correlation examples, namely Gl-En and En-Hi.

BWE: The hub matters for distant languagesMUSES implements a provably direction-independent closed form solution of the Procrustes problem, and we confirm empirically that the hub choice does not affect the outcome (we provide complete results on MUSEs in Table 7 in the Appendix). Similarly, because VecMap uses symmetric re-weighting and produces bidirectional dictionaries at its final step, the results are not dependent on the training direction. However, obtaining good performance with such methods requires the orthogonality assumption to hold, which for distant languages is rarely the case Patra et al. (2019). In fact, we find that the gradient-based MAT+MPSR method in a bilingual setting over typologically distant languages exhibits better performance than MUSEs or VecMap. Across Table 2, in only a handful of examples (shaded cells) doVecMap or MUSSE systems outperform MAT+MPSR for BWE (with the majority being among En, Es, Glc, and Pr, all related high-resource languages).

In the 7 distant languages setting, however, the results are different: VecMap outperforms MUSEs and the multilingual MAT+MPSR in the vast majority of the language pairs. The difference is more stark when the languages of the pair use completely different alphabets, where the same-character strings heuristic for bootstrapping the initial dictionary mapping fails. Instead, the monolingual similarity approach employed by VecMap is definitely more appropriate for settings such as those posed by languages like Korean or Hindi. This highlights the importance of actually evaluating and reporting results on such language pairs.

On the one hand, we find that when aligning distant

Figure 3: The Lexicon Induction accuracy generally correlates positively with the GH distance of the source and target language vector spaces to the hub language.

 

languages with MAT+MPSR, the difference between hub choices can be significant - in Az-En, for instance, using En as the hub leads to more than 7 percentage points difference compared to using Az. We show some examples in Table 5. On the other hand, when aligning typologically similar languages, the difference is less pronounced. For example, we obtain practically similar performance for Gl-Pr, Az-Tr, or Uk-Ru when using either the source or the target language as the hub. Note, though, that non-negligible differences could still occur, as in the case of Pr-Gl. In most cases, it is the case that the higher-resourced language is a better hub than the lower-resourced one, especially when the number of resources differ significantly (as in the case of Az and Be against any other language). Since BWE settings are not our main focus, we leave an extensive analysis of this observation for future work.

Bi-, tri-, and multilingual systemsThis part of our analysis compares bilingual, trilingual, and multilingual systems, with a focus on the under-represented languages. Through multiple experiments (complete evaluations are listed in the Appendix) we reach two main conclusions. On one hand, when evaluating on topologically distant languages, one should use as many languages as possible. In Table 4 we present one such example with results on Az-Cs under various settings. On the other hand, when multiple related languages are available, one can achieve higher performance with multilingual systems containing all related languages and one more hub language, rather than learning diverse multilingual mappings using more languages. We confirm the latter observation with experiments on the Slavic (Be, Ru, Uk) and Iberian (Es, GL, PT) clusters, and present an example (Ru-Uk) in Table 4.

## 5 Downstream Task Experiments

Differences in BLI performance do not necessarily translate to differences in other downstream tasks that use the aligned embeddings, so Glavas et al. (2019) advocate for actual evaluation on such tasks. We extend our analysis to an example downstream task of zero-shot POS tagging using the aligned embeddings for select language pairs. We show that indeed the choice of the hub language can have dramatic impact. Using Universal Dependencies data Nivre et al. (2016) we train simple bi-LSTM POS taggers on En and Pr using the respective embeddings produced from each MAT+MPSR run, and evaluate the zero-shot performance on Gi and

\begin{table}
\begin{tabular}{c c c c|c c c} \hline \multicolumn{8}{c}{Transfer from Ex} & \multicolumn{3}{c}{Transfer from Pr} \\ Hub & Es & Pr & Gt. & Hub & Es & Glc \\ \hline En & **38.7** & 21.8 & 19.4 & En & **48.4** & 32.9 \\ Es & 26.5 & 16.1 & **28.5*** & Es & 41.4 & 25.5* \\ Pt & 28.1 & 25.7 & 15.6 & Pr & 44.3† & **36.5** \\ Gl & 35.4 & 22.8 & 23.1 & Gli & 48.1 & 23.8 \\ Be & 35.6 & 30.5 & 13.2 & & & \\ Ru & 28.6† & **30.6** & 18.2 & †: best train-test hub \\ Sk & 24.2 & 30.2�� & 14.6 & for LI. & \\ \hline \end{tabular}
\end{table}
Table 6: The choice of hub can significantly affect downstream zero-shot POS tagging accuracy.

\begin{table}
\begin{tabular}{c c c c c} \hline \multicolumn{4}{c}{Results on Az–Cs} & Average \\ \hline _Bilingual_ & Az & Cs & & 25.8 \\ with hub: & 22.7 & 29.1 & & 25.8 \\ \hline _Tilingal_ & Az, Cs, +hb: & & & \\  & Be & En & Es & Gl. \\  & 21.6 & 28.5 & 31.8 & 23.0 \\  & Pr & Rv & Sk & Tx \\  & 29.6 & 27.4 & 30.4 & 32.9 \\ \hline _Tilingal_ & Az, hub;Cs, +extra: & & \\  & En & Es & Pr & Ru & Tr \\   30.1 & 30.1 & 33.2 & 27.1 & 33.7 \\ \hline _Multilingual (10 languages)_ & & & \\  & Az & Be & Cs & En & Es \\   33.7 & 34.0 & 32.3 & 34.5 & **35.1** \\ Gla & Pr & Rv & Sk & Rx \\   34.0 & **34.8** & 34.5 & 32.9 & 33.7 \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of bilingual, trilingual, and multilingual systems for distant (left) and related (right) languages. Multilinguality boosts performance significantly on distant languages.

\begin{table}
\begin{tabular}{c|r r|r r|r r l|r l} \hline \multirow{2}{*}{Test} & \multicolumn{3}{c|}{Hub} & \multirow{2}{*}{Test} & \multicolumn{3}{c}{Hub} \\  & & & & & & & & & & \\  & src & TRg & & & & & & & \\ \hline Az–Cs & 22.7 & 29.1 & Gl–Pr & 53.5 & 53.6 \\ Az–En & 13.2 & 20.7 & Pr–G & 39.0 & 36.7 \\ Az–Tr & 30.1 & 30.1 & Uk–Ru & 61.6 & 61.8 \\ \hline \end{tabular}
\end{table}
Table 5: The hub is important for BWE between distant languages with MAT+MPSR.

 

Es.17 Although all taggers achieve consistent accuracies \(>95\%\) on English and Portuguese regardless of the original En or Pr embeddings, the zero-shot performance on the test languages, as shown in Table 6, varies widely. For instance, using the embeddings produced from using Pr as a hub, we obtain the highest zero-shot accuracy on Gr. (36.5%), while using the ones from the Gi. hub lead to significantly worse performance (23.8%). It should be noted that the best hub for POS-tagging does not always coincide with the best hub for LI, e.g. the best LI hub for Pr-Gl is Es, which leads to 11 percentage points worse Gl POS tagging performance than the best system. In fact, for the language pairs that we studied we observe no correlation between the two tasks performance as we vary the hub (with an average Spearman's rank correlation \(\rho=0.08\)).

Footnote 17: Note that our goal is not to achieve SOTA in zero-shot POS-tagging, but to show that embeddings resulting from different hub choices have different qualities.

## 6 Conclusion

With this work we challenge the standard practice in learning cross-lingual word embeddings. We empirically show that the choice of the hub language is an important parameter that affects lexicon induction performance in both bilingual (between distant languages) and multilingual settings. More importantly, we hope that by providing new dictionaries and baseline results on several language pairs, we will stir the community towards evaluating all methods in challenging scenarios that include under-represented language pairs. Towards this end, our analysis provides insights and general directions for stronger baselines for non-Anglocentric cross-lingual word embeddings. The problem of identifying the best _hub_ language, despite our analysis based on the use of typological distance, remains largely unsolved. In the future, we will investigate a hub language ranking/selection model a la Lin et al. (2019).

## Acknowledgements

The authors are grateful to the anonymous reviewers for their exceptionally constructive and insightful comments, and to Gabriela Weigel for her invaluable help with editing and proofreading the paper. This material is based upon work generously supported by the National Science Foundation under grant 1761548.

## References

* Agic and Vulic (2019) Zeljko Agic and Ivan Vulic. 2019. Jw300: A wide-coverage parallel corpus for low-resource languages. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3204-3210.
* Alaux et al. (2019) Jean Alaux, Edouard Grave, Marco Cuturi, and Armand Joulin. 2019. Unsupervised hyperalignment for multilingual word embeddings. In _Proceedings of the International Conference on Learning Representations_.
* Ammar et al. (2016a) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2016a. Many languages, one parser. _Transactions of the Association for Computational Linguistics_, 4:431-444.
* Ammar et al. (2016b) Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. 2016b. Massively multilingual word embeddings. _arXiv preprint arXiv:1602.01925_.
* Aronoff and Fudeman (2011) Mark Aronoff and Kirsten Fudeman. 2011. _What is morphology?_, volume 8. John Wiley & Sons.
* Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 451-462, Vancouver, Canada. Association for Computational Linguistics.
* Artetxe et al. (2018a) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 789-798.
* Artetxe et al. (2018b) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. Unsupervised statistical machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, Brussels, Belgium.
* Chen and Cardie (2018) Xilun Chen and Claire Cardie. 2018. Unsupervised multilingual word embeddings. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 261-270. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2018. Word translation without parallel data. In _Proceedings of the Sixth International Conference on Learning Representations_.
* Czarnowska et al. (2019) Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. 2019. Don't forget the long tail! a comprehensive analysis of morphological generalization in bilingual lexicon induction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 974-983, Hong Kong, China. Association for Computational Linguistics.
* Dinu and Baroni (2014) Georgiana Dinu and Marco Baroni. 2014. How to make words with vectors: Phrase generation in distributional semantics. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 624-633.

 

Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In _Proceedings of the International Conference on Learning Representations, workshop track_.
* Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 644-648.
* Glavas et al. (2019) Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 710-721, Florence, Italy. Association for Computational Linguistics.
* Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_.
* Heyman et al. (2019) Geert Heyman, Bregt Verreet, Ivan Vulic, and Marie-Francine Moens. 2019. Learning unsupervised multilingual word embeddings with incremental multilingual hubs. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1890-1902, Minneapolis, Minnesota. Association for Computational Linguistics.
* Irvine and Callison-Burch (2013) Ann Irvine and Chris Callison-Burch. 2013. Combining bilingual and comparable corpora for low resource machine translation. In _Proceedings of the eighth workshop on statistical machine translation_, pages 262-270.
* Kementchedjhieva et al. (2019) Yova Kementchedjhieva, Mareike Hartmann, and Anders Sogaard. 2019. Lost in evaluation: Misleading benchmarks for bilingual dictionary induction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_.
* Klementeiv et al. (2012) Alexandre Klementeiv, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In _Proceedings of COLING 2012_, pages 1459-1474.
* Koehn (2004) Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
* Koehn (2009) Philipp Koehn. 2009. _Statistical machine translation_. Cambridge University Press.
* Koehn et al. (2005) Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne, Chris Callison-Burch, Miles Osborne, and David Talbot. 2005. Edinburgh system description for the 2005 iswlst speech translation evaluation. In _International Workshop on Spoken Language Translation (IWSLT) 2005_.
* Lample et al. (2018) Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, Brussels, Belgium.
* Levinboim and Chiang (2015) Tomer Levinboim and David Chiang. 2015. Multi-task word alignment triangulation for low-resource languages. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1221-1226, Denver, Colorado. Association for Computational Linguistics.
* Lin et al. (2019) Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3125-3135, Florence, Italy. Association for Computational Linguistics.
* Lison and Tiedemann (2016) Pierre Lison and Jorg Tiedemann. 2016. Opensubtitles2015: Extracting large parallel corpora from movie and tv subtitles. In _International Conference on Language Resources and Evaluation_.
* Littell et al. (2017) Patrick Littell, David R Mortensen, Ke Lin, Katherine Kairis, Carlise Turner, and Lori Levin. 2017. Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 8-14.
* Malaviya et al. (2017) Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, Copenhagen, Denmark.
* Mikolov et al. (2013) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv:1309.4168.
* Nivre et al. (2016)Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silivera, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_, pages 1659-1666.
* Patra et al. (2019) Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In _The 57th Annual Meeting of the Association for Computational Linguistics (ACL)_, Florence, Italy 

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv:2003.07082.
* Qi et al. (2018) Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In _Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL)_, New Orleans, USA.
* Schwenk et al. (2019) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzman. 2019. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv:1907.05791.
* Smith et al. (2017) Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In _Proceedings of the Fifth International Conference on Learning Representations_.
* Turian et al. (2010) Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 384-394, Uppsala, Sweden. Association for Computational Linguistics.
* Wang et al. (2006) Haifeng Wang, Hua Wu, and Zhanyi Liu. 2006. Word alignment for languages with scarce resources using bilingual corpora of other language pairs. In _Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions_, pages 874-881, Sydney, Australia. Association for Computational Linguistics.
* Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics.
* Zhang et al. (2017) Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1959-1970.
* Zhang et al. (2016) Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. 2016. Ten pairs to tag-multilingual pos tagging via coarse mapping between embeddings. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1307-1317.
* Zhou et al. (2019) Chunting Zhou, Xuezhe Ma, Di Wang, and Graham Neubig. 2019. Density matching for bilingual word embedding. In _Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL)_, Minneapolis, USA.
* Zou et al. (2013) Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1393-1398.

 

Does evaluation directionality matter?

We also explored whether there are significant differences between the evaluated quality of aligned spaces, when computed on both directions (src-trg and trg-src). We find that the evaluation direction indeed matters a lot, when the languages of the evaluation pair are very distant, in terms of morphological complexity and data availability (which affects the quality of the original embeddings). A prominent example, from our European-languages experiment, are evaluation pairs involving Az or Be. When evaluating on the Az-XX and Be-XX dictionaries, the word translation P@1 is more than 20 percentage points higher than when evaluating on the opposite direction (XX-Az or XX-Be). For example, Es-Az has a mere P@1 of 9.9, while Az-Es achieves a P@1 of 44.9. This observation holds even between very related languages (cf. Ru-be: 12.8, Be-Ru: 41.1 and Tr-Az: 8.4, Az-Tr: 32.0), which supports our hypothesis that this difference is also due to the quality of the pre-trained embeddings. It is important to note that such directionality differences are not observed when evaluating distant pairs with presumably high-quality pre-trained embeddings e.g. Tr-Sk or Tr-Es; the P@1 for both directions is very close.

## Appendix B Complete results for all experiments

Here we provide complete evaluation results for our multilingual experiments. Table 7 presents the P@1 of the bilingual experiments using MUSE, and Table 8 presents accuracy using VecMap. Tables 9-14 present P@1, P@5, and P@10 respectively, for the experiment on the 10 European languages. Similarly, results on the distant languages experiment are shown in Tables 15, 16, and 17.

 



\begin{table}
\begin{tabular}{c|r r r r r r r r r r} \hline \hline Source & \multicolumn{10}{c}{Target} \\  & Az & Be & Cs & En & Es & Gl. & Pr & Ru & Sk & Tr \\ \hline Az & – & 4.8 & 21.4 & 23.6 & 32.6 & 13.6 & 26.7 & 10.4 & 15.0 & 31.8 \\ Bre & 4.0 & – & 26.1 & 3.8 & 12.3 & 9.3 & 11.3 & 42.0 & 23.1 & 2.9 \\ Cs & 2.6 & 5.4 & – & 57.1 & 55.5 & 11.9 & 52.3 & 44.7 & 71.2 & 31.6 \\ Ens & 12.2 & 2.5 & 47.3 & – & 79.3 & 32.0 & 72.9 & 39.7 & 34.3 & 40.6 \\ Es & 7.8 & 2.4 & 45.0 & 76.7 & – & 37.1 & 83.4 & 38.9 & 34.3 & 38.2 \\ Glc & 2.7 & 1.8 & 14.0 & 38.5 & 61.2 & – & 53.3 & 11.4 & 12.9 & 8.5 \\ Pt & 2.9 & 2.3 & 44.9 & 72.2 & 88.7 & 36.3 & – & 33.7 & 33.7 & 34.6 \\ Ru & 1.7 & 12.0 & 48.6 & 50.2 & 49.4 & 6.6 & 46.8 & – & 44.6 & 21.1 \\ Sk & 0.3 & 5.2 & 71.8 & 48.0 & 46.4 & 9.3 & 44.4 & 43.2 & – & 21.2 \\ TR & 10.8 & 0.3 & 35.8 & 48.0 & 50.9 & 3.5 & 45.9 & 26.9 & 20.3 & – \\ \hline \end{tabular}
\end{table}
Table 7: BWE results (P@1) with MUSE

\begin{table}
\begin{tabular}{c|r r r r r r r r r} \hline \hline Source & \multicolumn{10}{c}{Target} \\  & Ex & Fr & Hi & Ko & Rv & Sv & Uk \\ \hline En & – & 80.3 & 17.9 & 9.5 & 39.7 & 60.0 & 25.9 \\ Fr & 76.6 & – & 11.9 & 5.1 & 38.0 & 52.4 & 26.8 \\ Hi & 24.2 & 17.0 & – & 0.4 & 3.1 & 3.3 & 2.3 \\ Ko & 12.4 & 7.1 & 0.4 & – & 2.5 & 2.2 & 0.6 \\ Ru & 50.2 & 47.3 & 3.2 & 1.6 & – & 35.8 & 58.8 \\ Sv & 53.3 & 47.8 & 5.2 & 2.3 & 27.8 & – & 19.9 \\ Uk & 37.4 & 40.3 & 4.1 & 0.3 & 60.7 & 30.2 & – \\ \hline \end{tabular}
\end{table}
Table 8: BWE results (P@1) with VecMap



























\begin{table}
\begin{tabular}{c|r r r r r r r|l} \hline \hline \multirow{2}{*}{Test} & \multicolumn{8}{c|}{Hub language} & \multirow{2}{*}{} \\  & \multicolumn{1}{c}{En} & \multicolumn{1}{c}{Fr} & \multicolumn{1}{c}{Hi} & \multicolumn{1}{c}{Ko} & \multicolumn{1}{c}{Ru} & \multicolumn{1}{c}{Sv} & \multicolumn{1}{c|}{Uk} & \multicolumn{1}{c|}{\(\mu\)} \\ \hline En–Fr & 75.1 & 75.3 & 75.2 & 75.8 & 76.3 & 75.5 & 75.4 & 75.5 \\ En–Hi & 20.9 & 23.5 & 21.0 & 21.4 & 23.5 & 21.4 & 23.9 & 22.2 \\ En–Ko & 9.2 & 10.4 & 9.1 & 9.8 & 9.8 & 10.1 & 10.0 & 9.8 \\ En–Ru & 41.8 & 42.0 & 41.8 & 41.5 & 42.0 & 41.8 & 42.0 & 41.8 \\ En–Sv & 57.0 & 57.5 & 59.0 & 56.6 & 57.8 & 57.6 & 58.4 & 57.7 \\ En–Uk & 26.9 & 27.5 & 26.9 & 26.9 & 28.3 & 27.8 & 26.2 & 27.2 \\ Fr–En & 72.5 & 72.0 & 71.6 & 72.7 & 72.9 & 73.4 & 74.0 & 72.7 \\ Fr–Hi & 18.7 & 16.0 & 14.8 & 17.3 & 19.0 & 17.8 & 17.5 & 17.3 \\ Fr–Ko & 6.9 & 6.7 & 5.8 & 5.5 & 5.8 & 7.5 & 6.0 & 6.3 \\ Fr–Ru & 39.9 & 38.3 & 40.3 & 40.4 & 40.8 & 40.0 & 39.6 & 39.9 \\ Fr–Sv & 51.8 & 49.3 & 50.5 & 51.1 & 49.4 & 48.2 & 51.8 & 50.3 \\ Fr–Uk & 28.8 & 27.0 & 27.8 & 28.5 & 28.7 & 27.7 & 26.1 & 27.8 \\ Hi–En & 27.8 & 31.4 & 27.9 & 28.6 & 30.4 & 29.3 & 29.3 & 29.3 \\ Hi–Fr & 25.6 & 23.1 & 25.1 & 23.3 & 26.9 & 25.5 & 24.2 & 24.8 \\ Hi–Ko & 2.1 & 1.7 & 1.3 & 1.6 & 1.6 & 1.4 & 1.8 & 1.6 \\ Hi–Ru & 13.9 & 14.2 & 14.3 & 13.6 & 14.3 & 13.5 & 14.6 & 14.0 \\ Hi–Sv & 17.3 & 16.8 & 16.3 & 15.9 & 17.0 & 15.9 & 16.6 & 16.6 \\ Hi–Uk & 10.3 & 10.5 & 9.1 & 9.1 & 9.8 & 9.5 & 9.6 & 9.7 \\ Ko–En & 15.1 & 16.6 & 15.2 & 17.0 & 16.6 & 17.7 & 16.4 & 16.4 \\ Ko–Fn & 11.9 & 10.2 & 10.9 & 10.9 & 12.6 & 13.6 & 10.8 & 11.6 \\ Ko–Hi & 1.8 & 2.4 & 1.2 & 1.6 & 2.0 & 1.8 & 2.0 & 1.9 \\ Ko–Ru & 7.9 & 6.6 & 6.0 & 5.7 & 6.9 & 6.8 & 7.3 & 6.7 \\ Ko–Sv & 6.8 & 6.6 & 5.9 & 5.9 & 7.2 & 5.6 & 7.2 & 6.5 \\ Ko–Uk & 3.5 & 3.6 & 3.4 & 3.2 & 3.5 & 3.5 & 3.1 & 3.4 \\ Ru–En & 50.2 & 53.2 & 52.2 & 53.4 & 52.5 & 52.6 & 52.1 & 52.3 \\ Ru–Fr & 51.1 & 49.6 & 50.7 & 51.7 & 51.0 & 50.6 & 50.3 & 50.7 \\ Ru–Hi & 14.6 & 15.0 & 12.0 & 14.6 & 13.3 & 14.8 & 15.3 & 14.2 \\ Ru–Ko & 5.2 & 4.6 & 4.4 & 3.6 & 4.3 & 4.1 & 5.0 & 4.4 \\ Ru–Sv & 40.7 & 40.9 & 40.1 & 41.0 & 39.8 & 36.7 & 41.3 & 40.1 \\ Ru–Uk & 55.3 & 56.1 & 55.8 & 56.3 & 55.3 & 54.9 & 55.6 \\ Sv–En & 51.2 & 51.1 & 52.3 & 51.9 & 52.0 & 50.7 & 52.7 & 51.7 \\ Sv–Fe & 47.9 & 45.7 & 46.8 & 48.2 & 47.1 & 46.6 & 47.4 & 47.1 \\ Sv–Hi & 17.2 & 16.3 & 15.0 & 16.0 & 17.7 & 15.9 & 17.0 & 16.4 \\ Sv–Ko & 4.9 & 4.2 & 4.0 & 3.8 & 5.0 & 4.0 & 5.1 & 4.4 \\ Sv–Ru & 31.5 & 33.2 & 32.4 & 33.0 & 31.8 & 30.2 & 31.8 & 32.0 \\ Sv–Uk & 22.4 & 23.8 & 23.0 & 23.5 & 24.1 & 21.0 & 21.9 & 22.8 \\ Uk–Ex & 39.5 & 40.8 & 40.3 & 40.7 & 41.4 & 40.2 & 40.2 & 40.4 \\ Uk–Fr & 43.6 & 42.3 & 44.0 & 43.3 & 43.0 & 43.3 & 40.6 & 42.9 \\ Uk–Hi & 13.8 & 13.8 & 12.8 & 12.8 & 12.7 & 14.4 & 13.0 & 13.3 \\ Uk–Ko & 2.6 & 2.5 & 2.4 & 2.0 & 2.0 & 2.4 & 2.6 & 2.4 \\ Uk–Rv & 59.4 & 58.9 & 59.7 & 58.7 & 59.1 & 58.4 & 58.6 & 59.0 \\ Uk–Sv & 35.8 & 35.5 & 35.8 & 36.8 & 35.4 & 32.7 & 35.1 & 35.3 \\ \hline \hline \end{tabular}
\end{table}
Table 15: All results from the distant languages MWE experiment (P@1).





\begin{table}
\begin{tabular}{c|r r r r r r r|l} \hline \hline \multirow{2}{*}{Test} & \multicolumn{8}{c|}{Hub language} & \multirow{2}{*}{} \\  & \multicolumn{1}{c}{En} & \multicolumn{1}{c}{Fr} & \multicolumn{1}{c}{Hi} & \multicolumn{1}{c}{Ko} & \multicolumn{1}{c}{Ru} & \multicolumn{1}{c}{Sv} & \multicolumn{1}{c|}{Uk} & \multicolumn{1}{c}{\(\mu\)} \\ \hline En–Fr & 87.3 & 88.2 & 87.8 & 88.4 & 88.3 & 88.0 & 87.7 & 88.0 \\ En–Hi & 37.2 & 39.4 & 36.5 & 37.1 & 39.3 & 38.7 & 39.9 & 38.3 \\ En–Ko & 23.4 & 24.6 & 22.6 & 23.4 & 24.3 & 25.9 & 25.0 & 24.2 \\ En–Ru & 63.5 & 65.3 & 65.1 & 64.8 & 66.9 & 64.6 & 65.9 & 65.2 \\ En–Sv & 74.8 & 76.1 & 76.3 & 75.8 & 75.4 & 75.6 & 76.5 & 75.8 \\ En–Uk & 47.7 & 49.8 & 49.3 & 47.9 & 49.3 & 48.5 & 47.7 & 48.6 \\ Fr–En & 85.3 & 84.5 & 83.7 & 84.5 & 85.4 & 85.1 & 84.6 & 84.7 \\ Fr–Hi & 32.7 & 30.0 & 29.5 & 30.6 & 33.4 & 32.2 & 31.6 & 31.4 \\ Fr–Ko & 14.9 & 14.5 & 14.0 & 14.6 & 16.0 & 15.3 & 15.2 & 14.9 \\ Fr–Ru & 61.0 & 59.5 & 61.9 & 61.7 & 62.1 & 60.6 & 60.9 & 61.1 \\ Fr–Sv & 69.6 & 68.1 & 68.8 & 69.1 & 68.6 & 68.0 & 71.1 & 69.0 \\ Fr–Uk & 45.6 & 44.2 & 44.8 & 45.6 & 45.8 & 45.0 & 44.1 & 45.0 \\ Hi–En & 44.5 & 47.0 & 46.3 & 44.3 & 47.0 & 46.3 & 46.7 & 46.0 \\ Hi–Fr & 41.7 & 39.3 & 41.6 & 39.6 & 42.7 & 41.2 & 42.3 & 41.2 \\ Hi–Ko & 5.3 & 4.8 & 3.4 & 3.5 & 4.7 & 5.1 & 5.0 & 4.5 \\ Hi–Ru & 27.6 & 29.6 & 27.6 & 28.1 & 27.9 & 28.8 & 29.5 & 28.4 \\ Hi–Sv & 31.7 & 31.7 & 30.8 & 30.7 & 32.7 & 30.2 & 32.0 & 31.4 \\ Hi–Uk & 21.4 & 21.9 & 19.9 & 20.1 & 20.8 & 20.4 & 20.2 & 20.7 \\ Ko–En & 28.9 & 28.7 & 27.0 & 28.1 & 30.1 & 33.1 & 28.6 & 29.2 \\ Ko–Fn & 21.9 & 21.6 & 19.7 & 20.4 & 24.0 & 24.4 & 21.3 & 21.9 \\ Ko–Hi & 4.3 & 4.8 & 3.9 & 4.1 & 4.6 & 4.8 & 5.0 & 4.5 \\ Ko–Ru & 16.2 & 15.3 & 12.9 & 13.4 & 15.8 & 15.7 & 16.3 & 15.1 \\ Ko–Sv & 16.2 & 14.1 & 13.9 & 13.8 & 15.6 & 13.9 & 16.3 & 14.8 \\ Ko–Uk & 9.7 & 8.0 & 8.6 & 8.6 & 9.3 & 8.2 & 8.8 & 8.8 \\ Ru–En & 69.8 & 71.1 & 70.9 & 71.0 & 70.2 & 71.1 & 71.3 & 70.8 \\ Ru–Fr & 65.7 & 66.2 & 67.7 & 67.9 & 67.0 & 66.6 & 67.2 & 66.9 \\ Ru–Hi & 27.3 & 27.6 & 24.7 & 26.7 & 25.6 & 26.6 & 28.7 & 26.7 \\ Ru–Ko & 12.1 & 10.4 & 10.1 & 10.0 & 11.1 & 10.4 & 12.4 & 10.9 \\ Ru–Sv & 58.8 & 58.9 & 58.2 & 58.2 & 58.8 & 56.1 & 59.9 & 58.4 \\ Ru–Uk & 68.3 & 68.8 & 69.2 & 68.0 & 68.8 & 68.6 & 66.9 & 68.4 \\ Sv–En & 65.4 & 66.2 & 66.3 & 65.7 & 65.1 & 64.4 & 65.9 & 65.6 \\ Sv–Fe & 62.5 & 60.1 & 60.3 & 61.1 & 60.7 & 59.8 & 61.3 & 60.8 \\ Sv–Hi & 28.2 & 28.0 & 26.6 & 27.4 & 29.3 & 27.1 & 28.6 & 27.9 \\ Sv–Ko & 11.7 & 10.7 & 10.9 & 9.8 & 11.5 & 11.6 & 11.4 & 11.1 \\ Sv–Ru & 50.5 & 51.0 & 50.7 & 50.9 & 50.3 & 47.8 & 49.9 & 50.2 \\ Sv–Uk & 40.2 & 42.1 & 41.6 & 41.6 & 41.7 & 38.3 & 39.2 & 40.6 \\ Uk–Ex & 56.3 & 58.1 & 57.5 & 57.2 & 59.1 & 58.1 & 56.1 & 57.5 \\ Uk–Fr & 58.3 & 56.4 & 58.5 & 58.7 & 58.9 & 58.0 & 56.4 & 57.9 \\ Uk–Hi & 27.2 & 25.8 & 24.0 & 25.4 & 26.5 & 25.8 & 25.3 & 25.7 \\ Uk–Ko & 7.4 & 7.2 & 6.8 & 6.0 & 7.3 & 7.3 & 7.3 & 7.0 \\ Uk–Rv & 71.0 & 71.0 & 71.2 & 70.1 & 70.4 & 70.7 & 70.5 & 70.7 \\ Uk–Sv & 53.3 & 53.3 & 52.5 & 53.1 & 53.7 & 48.9 & 53.1 & 52.5 \\ \hline \hline \end{tabular}
\end{table}
Table 16: All results from the distant languages MWE experiment (P@5).





\begin{table}
\begin{tabular}{c|r r r r r r r|l|l} \hline \hline \multirow{2}{*}{Test} & \multicolumn{8}{c|}{Hub language} & \multirow{2}{*}{} \\  & \multicolumn{1}{c}{En} & \multicolumn{1}{c}{Fr} & \multicolumn{1}{c}{Hi} & \multicolumn{1}{c}{Ko} & \multicolumn{1}{c}{Ru} & \multicolumn{1}{c}{Sv} & \multicolumn{1}{c|}{Uk} & \multicolumn{1}{c}{\(\mu\)} \\ \hline En–Fr & 90.8 & 91.3 & 90.1 & 91.0 & 91.1 & 91.1 & 90.7 & 90.9 \\ En–Hi & 44.0 & 45.9 & 43.3 & 43.1 & 45.0 & 45.2 & 45.6 & 44.6 \\ En–Ko & 31.1 & 31.5 & 28.4 & 30.5 & 31.6 & 33.7 & 32.1 & 31.3 \\ En–Ru & 70.1 & 71.7 & 71.0 & 70.7 & 72.4 & 71.1 & 72.3 & 71.3 \\ En–Sv & 80.0 & 81.1 & 80.9 & 80.4 & 80.8 & 80.4 & 81.2 & 80.7 \\ En–Uk & 55.3 & 57.5 & 56.5 & 55.2 & 57.4 & 56.4 & 54.6 & 56.1 \\ Fr–En & 87.6 & 87.8 & 86.6 & 87.7 & 88.0 & 87.9 & 88.0 & 87.6 \\ Fr–Hi & 39.1 & 35.3 & 35.5 & 36.5 & 38.6 & 38.1 & 38.5 & 37.4 \\ Fr–Ko & 20.1 & 18.4 & 18.4 & 19.6 & 20.3 & 19.4 & 19.7 & 19.4 \\ Fr–Ru & 67.1 & 65.9 & 68.1 & 67.5 & 66.8 & 66.8 & 67.4 & 67.1 \\ Fr–Sv & 74.4 & 73.3 & 74.2 & 74.8 & 73.3 & 73.3 & 75.5 & 74.1 \\ Fr–Uk & 51.7 & 49.7 & 51.3 & 51.8 & 52.0 & 51.2 & 49.9 & 51.1 \\ Hi–En & 50.0 & 52.3 & 53.0 & 50.8 & 52.7 & 51.7 & 52.3 & 51.8 \\ Hi–Fr & 49.0 & 45.5 & 46.8 & 46.8 & 48.3 & 48.1 & 48.9 & 47.6 \\ Hi–Ko & 7.9 & 7.2 & 5.1 & 5.1 & 6.4 & 6.6 & 7.2 & 6.5 \\ Hi–Ru & 34.5 & 35.3 & 34.5 & 34.7 & 33.6 & 35.3 & 36.3 & 34.9 \\ Hi–Sv & 38.0 & 37.5 & 36.1 & 37.9 & 38.9 & 36.3 & 38.5 & 37.6 \\ Hi–Uk & 27.3 & 27.6 & 25.8 & 25.4 & 26.2 & 25.9 & 25.5 & 26.3 \\ Ko–En & 34.2 & 34.3 & 32.3 & 35.2 & 37.1 & 38.4 & 35.4 & 35.3 \\ Ko–Fe & 27.0 & 25.9 & 23.7 & 24.6 & 28.5 & 30.1 & 26.4 & 26.6 \\ Ko–Hi & 6.2 & 6.9 & 5.6 & 6.0 & 6.7 & 6.7 & 6.9 & 6.4 \\ Ko–Ru & 21.2 & 19.3 & 16.4 & 18.2 & 20.4 & 20.9 & 20.8 & 19.6 \\ Ko–Sv & 20.9 & 18.1 & 17.8 & 17.5 & 21.1 & 18.4 & 20.6 & 19.2 \\ Ko–Uk & 12.9 & 12.1 & 11.5 & 11.3 & 12.6 & 12.0 & 11.7 & 12.0 \\ Ru–En & 74.9 & 75.8 & 75.4 & 75.5 & 75.5 & 76.2 & 75.6 & 75.6 \\ Ru–Fu & 71.8 & 72.5 & 73.0 & 72.2 & 72.7 & 72.7 & 72.6 & 72.5 \\ Ru–Hi & 33.0 & 32.9 & 30.1 & 32.1 & 31.9 & 32.1 & 34.6 & 32.4 \\ Ru–Ko & 17.2 & 14.6 & 13.2 & 13.5 & 15.9 & 15.0 & 16.7 & 15.2 \\ Ru–Sv & 64.7 & 64.7 & 63.6 & 64.6 & 64.2 & 62.5 & 64.6 & 64.1 \\ Ru–Uk & 73.3 & 72.8 & 73.1 & 72.0 & 73.1 & 72.9 & 71.7 & 72.7 \\ Sv–En & 69.5 & 70.4 & 71.0 & 70.6 & 70.9 & 69.3 & 70.0 & 70.2 \\ Sv–Fø & 67.0 & 64.2 & 65.0 & 65.3 & 65.5 & 64.2 & 65.7 & 65.3 \\ Sv–Hi & 33.6 & 32.6 & 32.0 & 30.9 & 33.3 & 31.9 & 33.2 & 32.5 \\ Sv–Ko & 15.7 & 14.7 & 14.0 & 12.9 & 15.7 & 14.9 & 15.6 & 14.8 \\ Sv–Ru & 57.2 & 56.4 & 56.5 & 56.2 & 56.4 & 53.8 & 56.4 & 56.1 \\ Sv–Uk & 47.5 & 47.9 & 47.7 & 47.7 & 48.5 & 44.8 & 46.4 & 47.2 \\ Uk–Ex & 61.6 & 63.4 & 62.9 & 62.2 & 63.5 & 62.7 & 61.1 & 62.5 \\ Uk–Fr & 63.5 & 62.4 & 63.9 & 63.4 & 64.3 & 63.5 & 61.9 & 63.3 \\ Uk–Hi & 32.7 & 32.3 & 28.6 & 30.2 & 31.7 & 31.5 & 30.7 & 31.1 \\ Uk–Ko & 10.6 & 10.2 & 9.5 & 8.7 & 10.1 & 10.4 & 10.2 & 10.0 \\ Uk–Rv & 74.5 & 73.8 & 74.1 & 73.9 & 74.5 & 74.1 & 73.9 & 74.1 \\ Uk–Sv & 59.1 & 58.8 & 58.8 & 58.7 & 59.3 & 55.2 & 57.8 & 58.2 \\ \hline \hline \end{tabular}
\end{table}
Table 17: All results from the distant languages MWE experiment (P@10).