Cross-Lingual Word Embeddings and Lexicon Induction ::: Lexicon Induction	One of the most common downstream evaluation tasks for the learned cross-lingual word mappings is Lexicon Induction (LI), the task of retrieving the most appropriate word-level translation for a query word from the mapped embedding spaces. Specialized evaluation (and training) dictionaries have been created for multiple language pairs, with the MUSE dictionaries  most often used, providing word translations between English (En) and 48 other high- to mid-resource languages, as well as on all 30 pairs among 6 very similar Romance and Germanic languages (English, French, German, Spanish, Italian, Portuguese). Given the mapped embedding spaces, the translations are retrieved using a distance metric, with Cross-Lingual Similarity Scaling  as the most common and best performing in the literature. Intuitively, CSLS decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). The retrieved pairs are compared to the gold standard and evaluated using precision at $k$ (P@$k$, evaluating how often the correct translation is within the $k$ retrieved nearest neighbours of the query).

Should All Cross-Lingual Embeddings Speak English?	Most of recent work in cross-lingual word embeddings is severely Anglocentric. The vast majority of lexicon induction evaluation dictionaries are between English and another language, and the English embedding space is selected by default as the hub when learning in a multilingual setting. With this work, however, we challenge these practices. First, we show that the choice of hub language can significantly impact downstream lexicon induction performance. Second, we both expand the current evaluation dictionary collection to include all language pairs using triangulation, and also create new dictionaries for under-represented languages. Evaluating established methods over all these language pairs sheds light into their suitability and presents new challenges for the field. Finally, in our analysis we identify general guidelines for strong cross-lingual embeddings baselines, based on more than just Anglocentric experiments.

We identify some general guidelines for choosing a hub language which could lead to stronger baselines; less isometry between the hub and source and target embedding spaces mildly correlates with performance, as does typological distance (a measure of language similarity based on language family membership trees). For distant languages, multilingual systems should in most cases be preferred over bilingual ones. We provide resources for training and evaluation on non-Anglocentric language pairs. We outline a simple triangulation method with which we extend the MUSE dictionaries to an additional 2352 lexicons covering 49 languages, and we present results on a subset of them. We also create new evaluation lexica for under-resourced languages using Azerbaijani, Belarusian, and Galician as our test cases. We additionally provide recipes for creating such dictionaries for any language pair with available parallel data.

Both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned to (hereinafter “the hub"). We outline the details in Section SECREF2. Despite all the recent progress in learning cross-lingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. Notably, most MWE approaches essentially select English as the hub during training by default, aligning all other language spaces to the English one. We argue and empirically show, however, that English is a poor hub language choice. In BWE settings, on the other hand, it is fairly uncommon to denote which of the two languages is the hub (often this is implied to be the target language). However, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages. This Anglocentricity is even more evident at the evaluation stage. The lexica most commonly used for evaluation are the MUSE lexica  which cover 45 languages, but with translations only from and into English.

Even still, alternative evaluation dictionaries are also very English- and European-centric:  report results on English–Italian,  on English–German and English–Finnish,  on Spanish–English and Italian–English, and  between English and Italian, German, Finnish, Spanish, and Turkish. We argue that cross-lingual word embedding mapping methods should look beyond English for their evaluation benchmarks because, compared to all others, English is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems . These two factors allow for an overly easy evaluation setting which does not necessarily generalize to other language pairs. In light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages. With this work, we attempt to address these shortcomings, providing the following contributions: We show that the choice of the hub when evaluating on diverse language pairs can lead to significantly different performance (e.g., by more than 10 percentage points for BWE over distant languages). We also show that often English is a suboptimal hub for MWE.

Introduction	Continuous distributional vectors for representing words (embeddings)  have become ubiquitous in modern, neural NLP. Cross-lingual representations  additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging , parsing , document classification , and machine translation , , . Often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter BWE and MWE, respectively). First, monolingual word embeddings are learned over large swaths of text; such pre-trained word embeddings, in fact, are available for several languages and are widely used, like the fastText Wikipedia vectors . Second, a mapping between the languages is learned, in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision , under minimal supervision e.g. using only identical strings , or even in a completely unsupervised fashion , .

Conclusion	With this work we challenge the standard practices in learning cross-lingual word embeddings. We empirically showed that the choice of the hub language is an important parameter that affects lexicon induction performance in both bilingual (between distant languages) and multilingual settings. More importantly, we hope that by providing new dictionaries and baseline results on several language pairs, we will stir the community towards evaluating all methods in challenging scenarios that include under-represented language pairs. Towards this end, our analysis provides insights and general directions for stronger baselines for non-Anglocentric cross-lingual word embeddings.

Note that in this case only a single matrix $$ needs to be learned, because $\left\Vert _1 - ^{}_2 \right\Vert =\left\Vert ^{-1}_1 - _2 \right\Vert $, while at the same time a model that minimizes $\left\Vert _1 - _2 \right\Vert $ is as expressive as one minimizing $\left\Vert _1_1 - _2_2 \right\Vert $, and easier to learn. In the minimally supervised or even the unsupervised setting  the popular methods follow an iterative refinement approach . Starting with a seed dictionary (e.g. from identical strings  or numerals) an initial mapping is learned in the same manner as in the supervised setting. The initial mapping, in turn, is used to expand the seed dictionary with high confidence word translation pairs. The new dictionary is then used to learn a better mapping, and so forth the iterations continue until convergence.

Similar to ,  note that it is important to use all $N^2$ language pairs when optimizing multilingual alignments, rather than just the $N-1$ pairs between the hub and all other languages, and propose a model (UMH) that implements this intuition within a Wasserstein-Procrustes approach. Even though the architecture and modeling approach of MAT+MPSR and of UMH are not the same, the two methods are conceptually similar, as in both cases a language is chosen as the hub, and $N-1$ mappings for the other languages are learned. In either case, English is by default selected to be the hub. The only exception is the study of triplets alignments in , where Spanish is used as the hub for the Spanish–French–Portuguese triplet, although the authors mention that they found little differences due the choice of the hub, in contrast to our general findings. Other than MAT+MPSR and UMH, another unsupervised multilingual approach is that of , who propose to incrementally align multiple languages by adding each new language as a hub.

Similarly, in a multilingual setting, one could start with $N$ languages $\mathcal {L} = \lbrace l_1,l_2,\ldots ,l_N\rbrace $ and their respective pre-trained embeddings $\mathcal {X}_1, \mathcal {X}_2,\ldots ,_N$, and then learn $N-1$ bilingual mappings between a pre-selected target language and all others. Hence, one of the language spaces is treated as a target (the hub) and remains invariant, while all others are mapped into the (now shared) hub language space. Alternatively, those mappings could be jointly learned using the MAT+MPSR methods of  – also taking advantage of the inter-dependencies between any two language pairs. Importantly, though, there is no closed form solution for learning the joint mapping, hence a solution needs to be approximated with gradient-based methods. MAT+MPSR generalizes the adversarial approach of  to multiple languages, and also follows an iterative refinement approach.

Throughout this work we report P@1, which is equivalent to accuracy, but we also provide results with P@5 and P@10 in the Appendix.

We decided, though, against comparing to this method, because (a) their method requires learning $\mathcal {O}(N^2)$ mappings for relatively small improvements and (b) the order in which the languages are added is an additional hyperparameter that would explode the experimental space.

Cross-Lingual Word Embeddings and Lexicon Induction	In the supervised bilingual setting, as formulated by , given two languages $\mathcal {L} = \lbrace l_1,l_2\rbrace $ and their pre-trained row-aligned embeddings $\mathcal {X}_1, \mathcal {X}_2,$ respectively, a transformation matrix $$ is learned such that: The set $\Omega $ can potentially impose a constraint over $$, such as the very popular constraint of restricting it to be orthogonal . Previous work has empirically found that this simple formulation is competitive with other more complicated alternatives , . The orthogonality assumption ensures that there exists a closed-form solution in the form of the Singular Value Decomposition (SVD) of $_1_2^T$.

Question: What evaluation metrics did they use?
Answer: we report P@1, which is equivalent to accuracy, we also provide results with P@5 and P@10 in the Appendix