Venue: ACL 2020

# Non-Linear Instance-Based Cross-Lingual Mapping for

Non-Isomorphic Embedding Spaces

Goran Glavas\({}^{1}\) and Ivan Vulic\({}^{2}\)

\({}^{1}\) Data and Web Science Group, University of Mannheim, Germany

\({}^{2}\)Language Technology Lab, TAL, University of Cambridge, UK

goran@informatik.uni-mannheim.de iv250@cam.ac.uk

###### Abstract

We present InstaMap, an instance-based method for learning projection-based cross-lingual word embeddings. Unlike prior work, it deviates from learning a single global linear projection. InstaMap is a non-parametric model that learns a non-linear projection by iteratively: (1) finding a globally optimal rotation of the source embedding space relying on the Kabsch algorithm, and then (2) moving each point along an instance-specific translation vector estimated from the translation vectors of the point's nearest neighbours in the training dictionary. We report performance gains with InstaMap over four representative state-of-the-art projection-based models on bilingual lexicon induction across a set of 28 diverse language pairs. We note prominent improvements, especially for more distant language pairs (i.e., languages with non-isomorphic monolingual spaces).

## 1 Introduction and Motivation

Induction of cross-lingual word embeddings (CLWEs) [36, 47, 9, 3] has been one of the key mechanisms for enabling multilingual modeling of meaning and facilitating cross-lingual transfer for downstream NLP tasks. Even though CLWEs are recently being contested in cross-lingual downstream transfer by pretrained multilingual language models [23, 34, 35, 36], they are still paramount in word-level translation, that is, bilingual lexicon induction (BLI).

While earlier work focused on joint induction of multilingual embeddings from multilingual corpora, relying on word- [1, 31, 32, 33], sentence- [36, 37, 38, 39, 40, 5], but also include the use of multiple-word embeddings to improve the performance of our approach. In this paper we focus on post-hoc alignment of independently trained monolingual embeddings (the so-called _projection_ or _mapping_ approaches) [36, 35, 37, 38, 41].

Despite some recent evidence that joint CLWE induction may lead to better bilingual spaces [36], projection-based methods still dominate the field [16, 35, 34, 37, 36] due to their conceptual attractiveness: they operate on top of vectors produced with any embedding model and need at most a few thousand word pairs of supervision [38, 39].

Most projection-based CLWE models induce bilingual spaces by orthogonally projecting one monolingual space to another. Since orthogonal projections do not affect the topology of the source space, the performance of these methods is bound by the degree of isomorphism of the two monolingual spaces. Yet, evidence suggests that monolingual spaces, especially those of etymologically and typologically distant languages, are far from isomorphic [22, 39, 40]. What is more, unsupervised CLWE models [36, 41, 42, 37, 38, 43], which additionally exploit the isomorphism assumption when inducing initial translation dictionaries, have been shown to yield near-zero BLI results for pairs of distant languages [22, 39]. Following these theoretical limitations of effectiveness of orthogonal mapping between non-isomorphic spaces, Joulin et al. (2018) and Patra

et al. (2019) relax the orthogonality constraint and report BLI improvements. These models, however, still learn only a linear transformation, i.e., an oblique projection matrix. While oblique projections may scale or skew the source space, there still exists a strong topological similarity between the original space and its oblique projection.

In this work, we deviate from learning a linear projection matrix (i.e., a parametric model) and propose a non-parametric model which translates vectors by estimating instance-specific geometric translations. Our method, InstaMap, iteratively (1) applies the Kabsch algorithm Horn (1987) on the full training dictionary to learn a globally optimal rotation of the source space w.r.t. the target space; and then (2) translates each point along the instance-specific translation vector, which we compute from the translation vectors of the point's nearest neighbours from the training dictionary.

We extensively evaluate InstaMap on the benchmark BLI dataset Glavas et al. (2019) encompassing 28 diverse language pairs. Our results show the non-linear mappings with InstaMap to be substantially more robust than linear projections, both orthogonal Smith et al. (2017); Artetxe et al. (2018) and oblique Joulin et al. (2018); Patra et al. (2019). We also show that, unlike InstaMap, oblique projection models - RCSLS Joulin et al. (2018) and BLISS Patra et al. (2019) - cannot surpass the performance of the best-performing orthogonal projection model VecMap Artetxe et al. (2018) for distant languages (i.e., for low isomorphicity). Finally, we report additional significant gains by applying InstaMap on top of VecMap.

## 2 Instance-Based Mapping

The core idea of InstaMap is illustrated in Figure 1. We iteratively: (1) use the entire training dictionary to learn a single global rotation matrix and then (2) perform an instance-based computation of translation vectors.

### Globally Optimal Rotation

Let \({\bf X}\) and \({\bf Y}\) be monolingual embedding spaces of the source and target language, respectively, and let \(D=\{(w_{L1}^{i},w_{L2}^{i})\},i=1\ldots N\), be the training dictionary. We first transform each of the two spaces by (independently) performing a full PCA transformation (i.e., no dimensionality reduction): this way we represent vectors in each of the spaces as combinations of linearly uncorrelated principal components of that space, which facilitates the learning of the optimal rotation between the spaces. Let \({\bf X}_{D}=\{{\bf x}_{L1}^{i}\}_{i=1}^{N}\subset{\bf X}\) and \({\bf Y}_{D}=\{{\bf y}_{L2}^{i}\}_{i=1}^{N}\subset{\bf Y}\) be the dictionary-aligned subsets of the two monolingual spaces. We aim to learn the optimal rotation matrix between \({\bf X}\) and \({\bf Y}\), i.e., the matrix \({\bf W}_{R}\) that minimizes the sum of square distances between the source vector projections and corresponding target vectors, \(W_{R}=\arg\min_{W}||{\bf X}_{D}{\bf W}-{\bf Y}_{D}||\). If we constrain \({\bf W}_{\bf R}\) to be orthogonal, the optimal solution is obtained by solving the Procrustes problem Schonemann (1966) - adopted by most projection-based CLWE models Smith et al. (2017); Conneau et al. (2018); Artetxe et al. (2018). However, our aim is to avoid introducing the orthogonal constraint and learn only the optimal rotation between the spaces. To this end, we use the Kabsch algorithm Horn (1987), which computes the optimal rotation matrix \({\bf W}_{R}\) as follows:

\[{\bf W}_{R} ={\bf V}\,{\bf I}_{R}\,{\bf U}^{T},\text{with} \tag{1}\] \[{\bf U}{\boldsymbol{\Sigma}}{\bf V}^{T} =\textit{SVD}({\bf X}_{D}^{T}{\bf Y}_{D}), \tag{2}\]

where \({\bf I}_{R}\) is a modification of the identity matrix, in which the last element (i.e., last row, last column) is not \(1\), but rather the determinant of \({\bf V}{\bf U}^{T}\). Upon obtaining \({\bf W}_{R}\), we rotate \({\bf X}\) w.r.t. \({\bf Y}\), \({\bf X}_{R}={\bf X}{\bf W}_{R}\).

### Instance-Specific Translations

We then perform localized, instance-specific translations in a rotationally-aligned bilingual space. For each point from both \({\bf X}_{R}\) and \({\bf Y}\), we compute a "personalized" translation vector, as the weighted average of the translation vectors of its closest dictionary entries. That is, for some vector \({\bf x}\in{\bf X}_{R}\) let \({\bf x}_{1},\dots,{\bf x}_{K}\) be the set of \(K\) vectors from \({\bf X}_{D}{\bf W}_{R}\) (corresponding to words \(w_{L1}^{1}\), \(w_{L1}^{2},\dots,w_{L1}^{K}\) in \(D\)) which are closest to \({\bf x}\) in terms of cosine similarity and let \({\bf y}_{1},{\bf y}_{2},\dots,{\bf y}_{K}\) be the vectors of the corresponding dictionary translations \(w_{L2}^{1}\), \(w_{L2}^{2},\dots,w_{L2}^{K}\) from \(D\) from the target language space. We then compute the instance-based translation of \({\bf x}\), \({\bf x}^{\prime}\), as follows:

\[{\bf x}^{\prime}={\bf x}+\frac{\sum_{k=1}^{K}\cos({\bf x},{\bf x}_{k})\cdot({ \bf y}_{k}-{\bf x}_{k})}{\sum_{k=1}^{K}\cos({\bf x},{\bf x}_{k})} \tag{3}\]

We perform an instance-specific translation of the vectors from \({\bf Y}\) analogously. Let \({\bf y}_{1},\dots,{\bf y}_{K}\) be the set of vectors from \({\bf Y}_{D}\) that are closest to some vector \({\bf y}\in{\bf Y}\). The translation \({\bf y}^{\prime}\) is then as follows:

\[{\bf y}^{\prime}={\bf y}-\frac{\sum_{k=1}^{K}\cos({\bf y},{\bf y}_{k})\cdot({ \bf y}_{k}-{\bf x}_{k})}{\sum_{k=1}^{K}\cos({\bf y},{\bf y}_{k})}\] (4) 

Because we compute a different translation vector for each point in both vector spaces, the final mapping function between the two spaces is globally non-linear. Also, being based on \(K\) nearest neighbours in the training dictionary \(D\), InstaMap is, unlike all other projection-based CLWE models, a non-parametric model (i.e., the number of model parameters is not fixed, it depends on the number of entries in the training dictionary \(D\)).

### Training Dictionary Expansion

We repeat the two steps - global rotation and instance-based translation - aiming to obtain an iterative refinement of the non-linear mapping between the two spaces. Following the established practice found in other iterative models Conneau et al. (2018); Artetxe et al. (2018), we augment the training dictionary for the next iteration with the mutual nearest neighbours in the bilingual space induced in the previous iteration. Intuitively, with InstaMap being a non-parametric model, we expect it to benefit more from the dictionary augmentation than the parametric projection models, which have been shown to saturate in performance when training dictionaries exceed 5K-10K translation pairs Vulic and Korhonen (2016); Glavas et al. (2019).

## 3 Evaluation

We evaluate InstaMap on bilingual lexicon induction, the standard task for evaluating CLWEs.

### Experimental Setup

**Data.** We evaluate on the BLI benchmark dataset introduced by Glavas et al. (2019), containing 28 pairs between eight diverse languages: English (EN), German (DE), Italian (IT), French (FR), Russian (RU), Croatian (HR), Turkish (TR), and Finnish (FI).1 Compprising both close and distant language pairs, this dataset allows us to compare model performance in settings with varying degree of isomorphism between monolingual spaces. We start from monolingual FastText vectors trained on Wikipedias of respective languages,2 with vocabularies trimmed to the 200K most frequent words.

Footnote 1: We use the training dictionaries with 5K instances.

Footnote 2: [https://fasttext.cc/](https://fasttext.cc/)

* solution to the Procrustes problem (Proc), and three state-of-the-art projection-based models: (1) VecMap Artetxe et al. (2018) emerged in recent comparative evaluations Glavas et al. (2019); Vulic et al. (2019) as the best-performing orthogonal-projection model; (2) RCSLS Joulin et al. (2018) learns an oblique (i.e., non-orthogonal) projection and yields best performance overall in a recent comparative evaluation Glavas et al. (2019); (3) BLISS Patra et al. (2019) combines an orthogonal projection objective with an objective based on adversarial learning, inducing a weakly-orthogonal projection matrix.

**Model Variants and Hyperparameter Tuning.** We evaluate two variants of InstaMap: (1) the base model is applied directly on unaligned monolingual vector spaces; (2) IM \(\circ\) VM is the variant in which we apply InstaMap on top of the bilingual space induced withVecMapArtetxe et al. (2018): because VecMap induces an orthogonal projection, the topologies of the monolingual subspaces of the VecMap bilingual space are preserved compared to

Figure 1: Illustration of InstaMap: **(a)** unaligned monolingual embedding spaces (EN – blue; DE – yellow) with dictionary alignments \(D\) (EN – green; DE – red); **(b)** rotation-aligned spaces: rotation matrix is learned on the whole dictionary \(D\); **(c)**InstaMap bilingual space: each point’s translation vector (depicted in (b)) is computed from translation vectors of nearest entries in \(D\); _Nota bene_: for simplicity, each point in illustration (figure (b)) inherits the translation vector of the nearest dictionary entry; in the actual algorithm, however, the translation vector is computed as weighted average of translation vectors of \(K\) nearest neighbours in \(D\) (see the description in §2.2).

 

respective original monolingual spaces - this holds promise of no undesirable side-effects originating from the composition. InstaMap has only two hyperparameters:3 the number of nearest neighbours \(K\) from \(D\), and the number of algorithm iterations \(T\). We identified, via fixed-split cross-validation on the training dictionaries, that configuration \(K=70\) and \(T=4\) works best for most language pairs.4

Footnote 3: Competing models – VecMap, RCSLS, and BLISS – all come with much larger sets of hyperparameters.

Footnote 4: For some pairs other configurations yield slightly better results: for simplicity, we report the results with base configuration \(K=70;T=4\) for all pairs.

### Results

We show BLI performance (\(P@1\)), aggregated over several different sets of language pairs, in Table 1.5 Overall, InstaMap significantly outperforms all competing models6 Somewhat surprisingly, VecMap, which induces an orthogonal projection (i.e., more strongly relies on the assumption of isomorphism), significantly outperforms RCSLS and BLISS, models that relax the orthogonality constraint and induce oblique linear projections. Only InstaMap, by removing the constraint of having a global linear projection altogether and by inducing a non-linear mapping, is able to consistently yield improvements over the orthogonal projection (VecMap). What is more, the IM \(\circ\) VM composition yields even larger performance gains.

Footnote 5: We provide detailed results for each of 28 language language pairs in the supplemental material.

Footnote 6: Non-parametric shuffling test Yeh [2000] with the Bonferroni correction: \(\alpha<0.05\) in comparison with VecMap and \(\alpha<0.01\) in comparison with other models.

Analysis of results across different groups of language pairs identifies InstaMap as particularly beneficial for pairs of distant languages (setups No-EN and HARD) and languages with least reliable monolingual vectors (TR, HR). For example, while InstaMap alone and IM \(\circ\) VM yield gains of 0.9 and 2.6 points, respectively, w.r.t. VecMap across ALL language pairs, these gaps widen to 1.5 and 3.5 points on most challenging language pairs (HARD). In contrast, BLISS, a model specifically tailored to improve the mappings between non-isomorphic spaces, appears to be robust only on pairs of close languages (e.g., HR-RU) and pairs involving EN (setup EN-*). It exhibits barely any improvement over the baseline orthogonal projection (Proc) on distant language pairs (HARD) and a significant degradation w.r.t. VecMap, a state-of-the-art model based on orthogonal projection. RCSLS is more robust than BLISS on difficult language pairs, but still performs worse than VecMap.

Further Analysis.We further analyze the performance of InstaMap (applied on top of VecMap) with respect to: (1) size of the training dictionary \(|D|\) and (2) number of nearest dictionary neighbours \(K\). We analyze the performance of IM\(\circ\)VM for three language pairs with lowest BLI scores: DE-TR, TR-FI, and TR-HR. We prepare dictionaries with 2.5K to 12.5K entries (with a 2.5K step), following steps described in Glavas et al. [2019].7 Figure 2 shows the performance for different training dictionary sizes. We can see that adding InstaMap on top of VecMap yields stable improvements for all dictionary sizes. On the one hand, this shows that InstaMap is equally helpful for any number of available word translations. On the other hand, since InstaMap is not constrained to learning a single global projection, we hoped to see bigger gains for larger dictionaries, but this is not the case. With larger dictionaries, we are

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline Model & Projection & ALL & EN-* & No-EN & EASY & HARD & Best LPs \\ \hline Proc & Orthogonal & 32.26 & 38.07 & 29.97 & 50.96 & 20.93 & DE-RU, DE-IT, DE-FR, IT-FR, EN-DE \\ VecMap & Orthogonal & 36.08 & 42.09 & 33.77 & 53.69 & 24.24 & HR-IT, DE-IT, FI-HR, FIR-FR \\ \hline RCSLS & Oblique & 35.31 & 41.94 & 32.75 & 53.31 & 23.78 & DE-RU, DE-FI, DE-HR, DE-TR, DE-FR \\ BLISS & Oblique & 33.78 & 44.62 & 30.04 & 49.92 & 21.07 & EN-RU, EN-TR, HR-RU, EN-FR \\ \hline InstaMap & Non-linear & 36.94 & 42.42 & 34.87 & 53.99 & 25.71 & DE-HR, DE-TR, DE-RU, FI-IT, DE-FR \\ IM \(\circ\) VM & Non-linear & **38.69** & **44.82** & **36.43** & **55.01** & **27.72** & DE-HR, TR-HR, TR-FI, DE-FL, DE-TR \\ \hline \hline \end{tabular}
\end{table}
Table 1: BLI results aggregated over diverse language pairs. Setups: (a) ALL – all 28 language pairs from Glavas et al. [2019] (b) EN-* – 7 language pairs with English as the source language; (c) EASY – 6 (20%) least difficult language pairs (EN-DE, EN-IT, EN-FR, IT-FR, DE-IT, DE-FR), according to average ranking of all models in evaluation; (d) HARD – 6 (20%) most difficult language pairs (TR-HR, DE-TR, TR-FI, TR-RU, FI-HR, DE-HR). (e) BEST LPs – 5 language pairs for which each model yields best relative performance compared to other models.

 

more likely to find more semantically similar dictionary neighbours for each word - and this should lead to better performance. We speculate, however, that larger dictionaries also increase the likelihood of selecting spurious neighbours due to hubness Dinu et al. (2015); Conneau et al. (2018) and that this cancels out the positive effect promised by having more candidates to choose the neighbours from. This could perhaps be remedied by using hubness-aware similarity scores like CSLS Conneau et al. (2018) instead of simple cosine similarity.

Figure 3 illustrates how InstaMap performance (on top of VecMap, i.e., IM \(\circ\) VM) varies with different values for the number of dictionary neighbours \(K\). The best performance is typically reached for values of \(K\) between \(50\) and \(90\) and there are no further improvements for larger values of \(K\) (TR-FI, where \(K=130\) gives the best score, is an exception). For very small \(K\) performance drops are substantial and here InstaMap even degrades the quality of the input space produced by VecMap. We believe this happens because InstaMap in this case has too few dictionary neighbours to accurately model the meaning of any given word and, in turn, compute a reliable mapping vector.

## 4 Conclusion

We have proposed InstaMap, a simple and effective approach for improving the post-hoc cross-lingual alignment between non-isomorphic monolingual embedding spaces. Unlike existing projection-based CLWE induction models, which learn a global linear projection matrix, InstaMap couples global rotation with instance-specific translations. This way, we learn a globally non-linear projection. Our experiments show that (1) InstaMap significantly outperforms four state-of-the-art projection-based CLWE models on a benchmark BLI dataset with 28 language pairs and (2) that it yields largest improvements for pairs of distant languages with a lower degree of isomorphism between their respective monolingual spaces. We plan to extend this work in two directions. First, we will explore mechanisms for instance-specific translation that are more sophisticated than the aggregation of translation vectors of nearest dictionary neighbours. Second, we plan to couple instance-based mapping with other informative features (e.g., character-level features) in classification-based BLI frameworks Heyman et al. (2017); Karan et al. (2020). The InstaMap code is available at: [https://github.com/codogogo/instamap](https://github.com/codogogo/instamap).

## Acknowledgments

We thank the anonymous reviewers for their insightful suggestions. GG is supported by the Eliteprogramm of the Baden-Wurttemberg Stiftung (AGREE grant). IV is supported by the ERC Consolidator Grant LEXICAL (no 648909).

Figure 3: _InstaMap_ (IM \(\circ\) VM) performance w.r.t the number of nearest dictionary neighbours \(K\). Results shown for three language pairs – DE-TR, TR-FI, and TR-HR – and \(T=3\) InstaMap iterations.

Figure 2: Comparison of performance betweenVecMap and InstaMap applied on top it (IM \(\circ\) VM) for different sizes of the training dictionary (from 2.5K word pairs to 12.5K word pairs in steps of 2.5K pairs).

 



## References

* Alvarez-Melis and Jaakkola (2018) David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-Wasserstein alignment of word embedding spaces. In _Proceedings of EMNLP_, pages 1881-1890.
* Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In _Proceedings of ACL_, pages 789-798.
* Artetxe et al. (2019) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. _arXiv preprint arXiv:1910.11856_.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of ACL_.
* Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2018. Word translation without parallel data. In _Proceedings of ICLR_.
* Coulmance et al. (2015) Jocelyn Coulmance, Jean-Marc Marty, Guillaume Wenzek, and Amine Benhalloum. 2015. Trans-gram, fast cross-lingual word-embeddings. In _Proceedings of EMNLP_, pages 1109-1113.
* Dinu et al. (2015) Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In _ICLR (Workshop Track)_.
* Glavas et al. (2019) Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In _Proceedings of ACL_, pages 710-721.
* Gouws and Sogaard (2015) Stephan Gouws and Anders Sogaard. 2015. Simple task-specific bilingual word embeddings. In _Proceedings of NAACL-HLT_, pages 1386-1390.
* Grave et al. (2019) Edouard Grave, Armand Joulin, and Quentin Berthet. 2019. Unsupervised alignment of embeddings with Wasserstein Procrustes. In _Proceedings of AISTATS_, pages 1880-1890.
* Hermann and Blunsom (2014) Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. In _Proceedings of ACL_, pages 58-68.
* Heyman et al. (2017) Geert Heyman, Ivan Vulic, and Marie Francine Moens. 2017. Bilingual lexicon induction by learning to combine word-level and character-level representations. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 1085-1095.
* Horn (1987) Berthold K.P. Horn. 1987. Closed-form solution of absolute orientation using unit quaternions. _Journal of the Optical Society of America A_, 4(4):629-642.
* Hoshen and Wolf (2018) Yedid Hoshen and Licor Wolf. 2018. Non-adversarial unsupervised word translation. In _Proceedings of EMNLP_, pages 469-478.
* Joulin et al. (2018) Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Herve Jegou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In _Proceedings of EMNLP_, pages 2979-2984.
* Karan et al. (2020) Mladen Karan, Ivan Vulic, Anna Korhonen, and Goran Glavas. 2020. Classification-based self-learning for weakly supervised bilingual lexicon induction. In _Proceedings of ACL_.
* Klementiev et al. (2012) Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In _Proceedings of COLING_, pages 1459-1474.
* Kocisky et al. (2014) Tomas Kocisky, Karl Moritz Hermann, and Phil Blunsom. 2014. Learning bilingual word representations by marginalizing alignments. In _Proceedings of ACL_, pages 224-229.
* Levy et al. (2017) Omer Levy, Anders Sogaard, and Yoav Goldberg. 2017. A strong baseline for learning cross-lingual word embeddings from sentence alignments. In _Proceedings of EACL_, pages 765-774.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Biltingual word representations with monolingual quality in mind. In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_, pages 151-159.
* Mikolov et al. (2013) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. _CoRR, abs/1309.4168_.
* Mosadala and Rettinger (2016) Aditya Mosadala and Achim Rettinger. 2016. Bilingual word embeddings from parallel and non-parallel corpora for cross-language text classification. In _Proceedings of NAACL-HLT_, pages 692-702.
* Nakashole (2018) Ndapa Nakashole. 2018. NORMA: Neighborhood sensitive maps for multilingual word embeddings. In _Proceedings of EMNLP_, pages 512-522.
* Ormazabal et al. (2019) Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. 2019. Analyzing the limitations of cross-lingual word embedding mappings. In _Proceedings of ACL_, pages 4990-4995.
* Patra et al. (2019) Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. In _Proceedings of ACL_, pages 184-193

* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In _Proceedings of ACL_, pages 4996-5001.
* Ruder et al. (2018) Sebastian Ruder, Ryan Cotterell, Yova Kementchedjhieva, and Anders Sogaard. 2018. A discriminative latent-variable model for bilingual lexicon induction. In _Proceedings of EMNLP_, pages 458-468.
* Schonemann (1966) Peter H Schonemann. 1966. A generalized solution of the orthogonal Procrustes problem. _Psychometrika_, 31(1):1-10.
* Smith et al. (2017) Samuel L. Smith, David H.P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In _Proceedings of ICLR_.
* Sogaard et al. (2015) Anders Sogaard, Zeljko Agic, Hector Martinez Alonso, Barbara Plank, Bernd Bohnet, and Anders Johannsen. 2015. Inverted indexing for cross-lingual NLP. In _Proceedings of ACL_, pages 1713-1722.
* Sogaard et al. (2018) Anders Sogaard, Sebastian Ruder, and Ivan Vulic. 2018. On the limitations of unsupervised bilingual dictionary induction. In _Proceedings of ACL_, pages 778-788.
* Vuilc et al. (2011) Ivan Vuilc, Wim De Smet, and Marie-Francine Moens. 2011. Identifying word translations from comparable corpora using latent topic models. In _Proceedings of ACL_, pages 479-484.
* Vulic et al. (2019) Ivan Vulic, Goran Glavas, Roi Reichart, and Anna Korhenen. 2019. Do we really need fully unsupervised cross-lingual embeddings? In _Proceedings of EMNLP_, pages 4398-4409.
* Vulic and Karhonen (2016) Ivan Vulic and Anna Karhonen. 2016. On the role of seed lexicons in learning bilingual word embeddings. In _Proceedings of ACL_, pages 247-257.
* Vulic and Moens (2016) Ivan Vulic and Marie-Francine Moens. 2016. Bilingual distributed word representations from documented comparable data. _Journal of Artificial Intelligence Research_, 55:953-994.
* Wu et al. (2020) Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In _Proceedings of ACL_.
* Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In _Proceedings of EMNLP_, pages 833-844.
* Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In _Proceedings of NAACL-HLT_, pages 1006-1011.
* Yeh (2000) Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In _Proceedings of COLING_, pages 947-953.
* Zhang et al. (2019) Mozhi Zhang, Keyulu Xu, Ken-ichi Kawarabayashi, Stefanie Jegelka, and Jordan Boyd-Graber. 2019. Are girls nek or shojo? cross-lingual alignment of non-isomorphic embeddings with iterative normalization. In _Proceedings of ACL_, pages 3180-3189.
* Zou et al. (2013) Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In _Proceedings of EMNLP_, pages 1393-1398.

 



\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c c c} Model & Proj. & EN-DE & EN-TR & EN-FI & EN-HR & EN-RU & EN-IT & EN-FR & DE-TR & DE-FI & DE-HR & DE-RI & DE-IR & DE-RT & DE-FR & TR-FI \\ Proc & Orth. & 45.6 & 24.1 & 30.8 & 24.3 & 36.2 & 54.8 & 58.2 & 20.8 & 27.1 & 24.2 & 32.9 & 43.2 & 42.5 & 19.7 \\ VecMap & Orth. & 48.5 & 30.4 & 35.4 & 29.8 & 38.5 & 57.3 & 61.2 & 23.2 & 28.8 & 27.1 & 31.6 & 46.3 & 45.3 & 23.1 \\ RCSLS & Obliq. & 49.0 & 28.8 & 34.8 & 28.1 & 41.9 & 57.6 & 60.6 & 24.1 & 30.2 & 27.3 & 36.8 & 45.0 & 45.2 & 22.8 \\ BLISS & Obliq. & 47.0 & 32.7 & 35.8 & 30.6 & **45.9** & 58.9 & **63.9** & 17.5 & 20.3 & 17.4 & 20.5 & 30.9 & 33.3 & 21.6 \\ InstaMap & Non-lin. & 47.6 & 30.6 & 33.6 & 29.6 & 40.6 & 59.1 & 61.2 & 25.7 & 31.0 & 29.1 & 36.4 & 46.8 & **47.4** & 24.9 \\ IM \(\circ\) VM & Non-lin. & **49.3** & **33.0** & **37.7** & **32.5** & 42.9 & **59.8** & 63.1 & **26.8** & **33.5** & **31.1** & **37.6** & **47.4** & 46.9 & **27.4** \\  & & & & & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 2: BLI results detailed over the first batch of 14 language pairs.

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c c} Model & Proj. & TR-HR & TR-RU & TR-IT & TR-FR & FI-HR & FI-RU & FI-IT & FI-FR & HR-RU & HR-TI & HR-FR & RU-IT & RU-FR & IT-FR \\ Proc & Orth. & 18.4 & 20.9 & 25.2 & 25.9 & 21.4 & 25.4 & 27.3 & 28.1 & 28.8 & 27.9 & 29.2 & 40.1 & 38.6 & 61.5 \\ VecMap & Orth. & 21.9 & 23.5 & 30.6 & 31.6 & 26.7 & 30.3 & 32.6 & 34.0 & 33.1 & 35.4 & 34.9 & 42.8 & 42.9 & 63.5 \\ RCSLS & Obliq. & 20.2 & 24.3 & 28.2 & 29.5 & 23.8 & 29.1 & 29.9 & 30.8 & 31.9 & 30.7 & 32.5 & 41.8 & 41.0 & 62.6 \\ BLISS & Obliq. & 21.6 & 23.1 & 28.2 & 29.0 & 25.3 & 29.5 & 30.4 & 31.1 & 34.8 & 32.3 & 32.5 & 42.9 & 43.3 & **65.6** \\ InstaMap & Non-lin. & 23.9 & 24.4 & 31.4 & 31.7 & 26.2 & 30.6 & 35.0 & 34.4 & 33.4 & 35.1 & 34.2 & 44.4 & 44.5 & 61.9 \\ IM \(\circ\) VM & Non-lin. & **26.1** & **26.6** & **32.0** & **34.5** & **28.4** & **31.8** & **35.4** & **35.8** & **36.3** & **36.4** & **36.6** & **44.9** & **46.0** & 63.5 \\  & & & & & & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 3: BLI results detailed over the second batch of 14 language pairs.