Continuous distributional vectors for representing words (embeddings) BIBREF0 have become ubiquitous in modern, neural NLP. Cross-lingual representations BIBREF1 additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging BIBREF2, parsing BIBREF3, document classification BIBREF4, and machine translation BIBREF5, BIBREF6, BIBREF7.Often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter BWE and MWE, respectively). First, monolingual word embeddings are learned over large swaths of text; such pre-trained word embeddings, in fact, are available for several languages and are widely used, like the fastText Wikipedia vectors BIBREF8. Second, a mapping between the languages is learned, in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision BIBREF9, under minimal supervision e.g. using only identical strings BIBREF10, or even in a completely unsupervised fashion BIBREF11, BIBREF12. Both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned to (hereinafter “the hub"). We outline the details in Section SECREF2.Despite all the recent progress in learning cross-lingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. Notably, most MWE approaches essentially select English as the hub during training by default, aligning all other language spaces to the English one. We argue and empirically show, however, that English is a poor hub language choice. In BWE settings, on the other hand, it is fairly uncommon to denote which of the two languages is the hub (often this is implied to be the target language). However, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages.This Anglocentricity is even more evident at the evaluation stage. The lexica most commonly used for evaluation are the MUSE lexica BIBREF12 which cover 45 languages, but with translations only from and into English. Even still, alternative evaluation dictionaries are also very English- and European-centric: BIBREF13 report results on English–Italian, BIBREF14 on English–German and English–Finnish, BIBREF11 on Spanish–English and Italian–English, and BIBREF15 between English and Italian, German, Finnish, Spanish, and Turkish. We argue that cross-lingual word embedding mapping methods should look beyond English for their evaluation benchmarks because, compared to all others, English is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems BIBREF16. These two factors allow for an overly easy evaluation setting which does not necessarily generalize to other language pairs. In light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages.With this work, we attempt to address these shortcomings, providing the following contributions:We show that the choice of the hub when evaluating on diverse language pairs can lead to significantly different performance (e.g., by more than 10 percentage points for BWE over distant languages). We also show that often English is a suboptimal hub for MWE.We identify some general guidelines for choosing a hub language which could lead to stronger baselines; less isometry between the hub and source and target embedding spaces mildly correlates with performance, as does typological distance (a measure of language similarity based on language family membership trees). For distant languages, multilingual systems should in most cases be preferred over bilingual ones.We provide resources for training and evaluation on non-Anglocentric language pairs. We outline a simple triangulation method with which we extend the MUSE dictionaries to an additional 2352 lexicons covering 49 languages, and we present results on a subset of them. We also create new evaluation lexica for under-resourced languages using Azerbaijani, Belarusian, and Galician as our test cases. We additionally provide recipes for creating such dictionaries for any language pair with available parallel data.In the supervised bilingual setting, as formulated by BIBREF1, given two languages $\mathcal {L} = \lbrace l_1,l_2\rbrace $ and their pre-trained row-aligned embeddings $\mathcal {X}_1, \mathcal {X}_2,$ respectively, a transformation matrix $$ is learned such that:The set $\Omega $ can potentially impose a constraint over $$, such as the very popular constraint of restricting it to be orthogonal BIBREF17. Previous work has empirically found that this simple formulation is competitive with other more complicated alternatives BIBREF17, BIBREF12. The orthogonality assumption ensures that there exists a closed-form solution in the form of the Singular Value Decomposition (SVD) of $_1_2^T$. Note that in this case only a single matrix $$ needs to be learned, because $\left\Vert _1 - ^{}_2 \right\Vert =\left\Vert ^{-1}_1 - _2 \right\Vert $, while at the same time a model that minimizes $\left\Vert _1 - _2 \right\Vert $ is as expressive as one minimizing $\left\Vert _1_1 - _2_2 \right\Vert $, and easier to learn.In the minimally supervised or even the unsupervised setting BIBREF11 the popular methods follow an iterative refinement approach BIBREF14. Starting with a seed dictionary (e.g. from identical strings BIBREF18 or numerals) an initial mapping is learned in the same manner as in the supervised setting. The initial mapping, in turn, is used to expand the seed dictionary with high confidence word translation pairs. The new dictionary is then used to learn a better mapping, and so forth the iterations continue until convergence.Similarly, in a multilingual setting, one could start with $N$ languages $\mathcal {L} = \lbrace l_1,l_2,\ldots ,l_N\rbrace $ and their respective pre-trained embeddings $\mathcal {X}_1, \mathcal {X}_2,\ldots ,_N$, and then learn $N-1$ bilingual mappings between a pre-selected target language and all others. Hence, one of the language spaces is treated as a target (the hub) and remains invariant, while all others are mapped into the (now shared) hub language space. Alternatively, those mappings could be jointly learned using the MAT+MPSR methods of BIBREF19 – also taking advantage of the inter-dependencies between any two language pairs. Importantly, though, there is no closed form solution for learning the joint mapping, hence a solution needs to be approximated with gradient-based methods. MAT+MPSR generalizes the adversarial approach of BIBREF11 to multiple languages, and also follows an iterative refinement approach.Similar to BIBREF19, BIBREF20 note that it is important to use all $N^2$ language pairs when optimizing multilingual alignments, rather than just the $N-1$ pairs between the hub and all other languages, and propose a model (UMH) that implements this intuition within a Wasserstein-Procrustes approach. Even though the architecture and modeling approach of MAT+MPSR and of UMH are not the same, the two methods are conceptually similar, as in both cases a language is chosen as the hub, and $N-1$ mappings for the other languages are learned. In either case, English is by default selected to be the hub. The only exception is the study of triplets alignments in BIBREF20, where Spanish is used as the hub for the Spanish–French–Portuguese triplet, although the authors mention that they found little differences due the choice of the hub, in contrast to our general findings.Other than MAT+MPSR and UMH, another unsupervised multilingual approach is that of BIBREF21, who propose to incrementally align multiple languages by adding each new language as a hub. We decided, though, against comparing to this method, because (a) their method requires learning $\mathcal {O}(N^2)$ mappings for relatively small improvements and (b) the order in which the languages are added is an additional hyperparameter that would explode the experimental space.One of the most common downstream evaluation tasks for the learned cross-lingual word mappings is Lexicon Induction (LI), the task of retrieving the most appropriate word-level translation for a query word from the mapped embedding spaces. Specialized evaluation (and training) dictionaries have been created for multiple language pairs, with the MUSE dictionaries BIBREF12 most often used, providing word translations between English (En) and 48 other high- to mid-resource languages, as well as on all 30 pairs among 6 very similar Romance and Germanic languages (English, French, German, Spanish, Italian, Portuguese).Given the mapped embedding spaces, the translations are retrieved using a distance metric, with Cross-Lingual Similarity Scaling BIBREF12 as the most common and best performing in the literature. Intuitively, CSLS decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). The retrieved pairs are compared to the gold standard and evaluated using precision at $k$ (P@$k$, evaluating how often the correct translation is within the $k$ retrieved nearest neighbours of the query). Throughout this work we report P@1, which is equivalent to accuracy, but we also provide results with P@5 and P@10 in the Appendix.With this work we challenge the standard practices in learning cross-lingual word embeddings. We empirically showed that the choice of the hub language is an important parameter that affects lexicon induction performance in both bilingual (between distant languages) and multilingual settings. More importantly, we hope that by providing new dictionaries and baseline results on several language pairs, we will stir the community towards evaluating all methods in challenging scenarios that include under-represented language pairs. Towards this end, our analysis provides insights and general directions for stronger baselines for non-Anglocentric cross-lingual word embeddings.

Q: What evaluation metrics did they use?

Q: What is triangulation?
A: Answer with content missing: (Chapter 3) The concept can be easily explained with an example, visualized in Figure 1. Consider the Portuguese (Pt) word trabalho which, according to the MUSE Pt–En dictionary, has the words job and work as possible En translations. In turn, these two En words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the transitive property (which translation should exhibit) we can identify the set of 7 possible Cs translations for the Pt word trabalho.

Q: What languages are explored in this paper?