SlowGuess/ACL_2020_OCR huggingface dataset:
We train and evaluate all models starting with pretrained Wikipedia FastText embeddings for all languages (Grave et al., 2018). We focus on the minimally supervised scenario which only uses similar character strings between any languages for supervision in order to mirror the hard, realistic scenario of not having annotated training dictionaries between the languages. We learn MWE with the MAT+MPSR method using the publicly available code,[12] aligning several language subsets varying the hub language. We decided against comparing to the incremental hub (IHS) method of Heyman et al. (2019), because the order in which the languages are added is an additional hyperparameter that would explode the experimental space.[13] We also do not compare to UMH, as we consider it conceptually similar to MAT+MPSR and no code is publicly available. For BWE
experiments, we use MUSEs $^{14}$  (MUSE, semisupervised) and VecMap $^{15}$  systems, and we additionally compare them to MAT+MPSR for completeness.
We compare the statistical significance of the performance difference of two systems using paired bootstrap resampling (Koehn, 2004). Generally, a difference of 0.4-0.5 percentage points evaluated over our lexica is significant with  $p < 0.05$ .
Experiment 1 We first focus on 10 languages of varying morphological complexity and data availability (which affects the quality of the pre-trained word embeddings): Azerbaijani (Az), Belarusian (Be), Czech (Cs), English (EN), Galician (GL), Portuguese (Pt), Russian (Ru), Slovak (Sk), Spanish (Es), and Turkish (Tr). The choice of these languages additionally ensures that for our three low-resource languages (Az, Be, GL) we include at least one related higher-resource language (Tr, Ru, Pt/Es respectively), allowing for comparative analysis. Table 2 summarizes the best post-hoc performing systems for this experiment.
Experiment 2 In the second setting, we use a set of 7 more distant languages: English, French  $(\mathrm{FR})$ , Hindi  $(\mathrm{Ht})$ , Korean  $(\mathrm{Ko})$ , Russian, Swedish  $(\mathrm{Sv})$ , and Ukrainian  $(\mathrm{Uk})$ . This language subset has large variance in terms of typology and alphabet. The best performing systems are presented in Table 3.
4.2 Analysis and Takeaways
MWE: English is rarely the best hub language In multilingual settings, we conclude that the standard practice of choosing English as the hub language is sub-optimal. Out of the 90 evaluation pairs from our 10-language experiment (Table 2) the best hub language is English in only 17 instances (less than  $20\%$  of the
$^{12}$ https://github.com/ccsasuke/umwe
<sup>13</sup>We refer the reader to Table 2 from Heyman et al. (2019) which compares to MAT+MPSR, and to Table 7 of their appendix which shows the dramatic influence of language order.
$^{14}$ https://github.com/facebookresearch/MUSE
15https://github.com/artetxem/vecmap
8662
Source	Target
EN	FR	Hi	Ko	RU	Sv	UK	μBEST	μEN
EN	-	76.3RU	23.9UK	10.4FR	42.0UK	59.0HI	28.3RU	40.0	38.5
FR	74.0UK	-	19.0RU	7.5SV	40.8RU	51.8EN	28.8EN	37.0	36.4
Hi	31.4FR	26.9RU	-	2.1EN	14.6UK	17.3EN	10.5FR	17.1	16.2
Ko	17.7SV	13.6SV	2.4FR	-	7.9EN	7.2RU	3.6FR	8.8	7.9
RU	53.4KO	51.7KO	15.3UK	5.2EN	-	41.3UK	56.3KO	37.2	36.2
Sv	52.7UK	48.2KO	17.7RU	5.1UK	33.2FR	-	24.1RU	30.2	29.2
UK	41.4RU	44.0HI	14.4SV	2.6EN	59.7HI	36.8KO	-	33.2	32.4
μBEST	45.1	43.5	15.5	5.5	33.0	35.6	25.3	29.1
μEN	42.7	42.5	14.5	5.1	32.4	34.9	24.5	28.1
Table 3: Lexicon Induction performance (P@1) over MWEs from 7 typologically distant languages (42 pairs). The lightly shaded cells are the only language pairs where a bilingual MUSE system outperforms MAT+MSPR; in heavy shaded cells a bilingual VecMap (but not MUSES) system outperform MAT+MSPR.
time). In fact, the average performance (over all evaluation pairs) when using EN as the hub (denoted as  $\mu^{\mathrm{EN}}$ ) is 1.3 percentage points worse than the optimal ( $\mu^{\mathrm{BEST}}$ ). In our distant-languages experiment (Table 3) English is the best choice only for 7 of the 42 evaluation pairs (again, less than  $20\%$  of the time). As before, using EN as the hub leads to an average drop of one percentage point in performance aggregated over all pairs, compared to the averages of the optimal selection. The rest of this section attempts to provide an explanation for these differences.
Expected gain for a hub language choice As vividly outlined by the superscript annotations in Tables 2 and 3, there is not a single hub language that stands out as the best one. Interestingly, all languages, across both experiments, are the best hub language for some evaluation language pair. For example, in our 10-languages experiment, Es is the best choice for about  $20\%$  of the evaluation pairs,  $\mathrm{Tr}$  and EN are the best for about  $17\%$  each, while GL and BE are the best for only 5 and 3 language pairs respectively.
Clearly, not all languages are equally suited to be the hub language for many language pairs. Hence, it would be interesting to quantify how much better one could do by selecting the best hub language compared to a random choice. In order to achieve this, we define the expected gain  $G_{l}$  of using language  $l$  as follows. Assume that we are interested in mapping  $N$  languages into the shared space and  $p_l^m$  is the accuracy over a specified evaluation pair  $m$  when using language  $l$  as the hub. The random choice between  $N$  languages will have an expected accuracy equal to the average accuracy when using all languages as hub:
$$
\mathbb {E} [ p ^ {m} ] = \frac {\sum_ {l} p _ {l} ^ {m}}{N}.
$$
The gain for that evaluation dataset  $m$  when using language  $l$  as hub, then, is  $g_{l}^{m} = p_{l}^{m} - \mathbb{E}[p^{m}]$ . Now, for a collection of  $M$  evaluation pairs we simply average their gains, in order to obtain the expected gain for using
Figure 2: Expected gain  $G_{l}$  for the MWE experiments.
language  $l$  as the hub:
$$
G _ {l} = \mathbb {E} \left[ g _ {l} \right] = \frac {\sum_ {m} g _ {l} ^ {m}}{M}.
$$
The results of this computation for both sets of experiments are presented in Figure 2. The bars marked 'overall' match our above definition, as they present the expected gain computed over all evaluation language pairs. For good measure, we also present the average gain per language aggregated over the evaluation pairs where that language was indeed the best hub language ('when best' bars). Perhaps unsurprisingly, Az seems to be the worst hub language choice among the 10 languages of the first experiment, with an expected loss (negative gain) of -0.4. This can be attributed to how distant Az is from all other languages, as well as to the fact that the Az pre-trained embeddings are of lower quality compared to all other languages (as the Az Wikipedia dataset is significantly smaller than the others). Similarly, Hi and Sv show expected loss for our second experiment.
Note that English is not a bad hub choice per se – it exhibits a positive expected gain in both sets of experiments. However, there are languages with larger expected gains, like Es and Gl in the 10-languages experiment that have a twice-as-large expected gain, while Ru has a 4 times larger expected gain in the distant-
This could be substituted with any evaluation metric.
8663
languages experiment. Of course, the language subset composition of these experiments could possibly impact those numbers. For example, there are three very related languages (Es, GL, Pr) in the 10 languages set, which might boost the expected gain for that subset; however, the trends stand even if we compute the expected gain over a subset of the evaluation pairs, removing all pairs that include GL or Pr. For example, after removing all GL results, Es has a slightly lower expected gain of 0.32, but is still the language with the largest expected gain.
Identifying the best hub language for a given evaluation set The next step is attempting to identify potential characteristics that will allow us make educated decisions with regards to choosing the hub language, given a specific evaluation set. For example, should one choose a language typologically similar to the evaluation source, target, or both? Or should they use the source or the target of the evaluation set as the hub?
Our first finding is that the best performing hub language will very likely be neither the source nor the target of the evaluation set. In our 10-languages experiments, a language different than the source and the target yields the best accuracy for over  $93\%$  of the evaluation sets, with the difference being statistically significant in more than half such cases. Similarly, in the distant-languages experiment, there is only a single instance where the best performing hub language is either the source or the target evaluation language (for  $\mathrm{Fr - Ru}$ ), and for the other  $97\%$  of cases the best option is a third language. This surprising pattern contradicts the mathematical intuition discussed in Section 2 according to which a model learning a single mapping (keeping another word embedding space fixed) is as expressive as a model that learns two mappings for each of the languages. Instead, we find that in almost all cases, learning mappings for both language spaces of interest (hence rotating both spaces) leads to better BLI performance compared to when one of the spaces is fixed.
Our second finding is that the LI performance correlates with measures of distance between languages and language spaces. The typological distance  $(d^{gen})$  between two languages can be approximated through their genealogical distance over hypothesized language family trees, which we obtain from the URIEL typological database (Littell et al., 2017). Also, Patra et al. (2019) recently motivated the use of Gromov-Hausdorff (GH) distance as an a priori estimation of how well two language embedding spaces can be aligned under an isometric transformation (an assumption most methods rely on). The authors also note that vector space GH distance correlates with typological language distance.
We find that there is a positive correlation between LI performance and the genealogical distances between the source-hub and target-hub languages. The average (over all evaluation pairs) Pearson's correlation coefficient between  $\mathrm{P@1}$  and  $d^{gen}$  is 0.49 for the distant languages experiment and 0.38 for the 10-languages one. A similar positive correlation of performance and the
Figure 3: The Lexicon Induction accuracy generally correlates positively with the GH distance of the source and target language vector spaces to the hub language.
sum of the GH distances between the source–hub and target–hub spaces. On our distant languages experiment, the correlation coefficient between  $\mathrm{P@1}$  and GH is 0.45, while it is slightly lower (0.34) for our 10-languages experiment. Figure 3 shows two high correlation examples, namely GL-EN and EN-HI.
BWE: The hub matters for distant languages MUSES implements a provably direction-independent closed form solution of the Procrustes problem, and we confirm empirically that the hub choice does not affect the outcome (we provide complete results on MUSES in Table 7 in the Appendix). Similarly, because VecMap uses symmetric re-weighting and produces bidirectional dictionaries at its final step, the results are not dependent on the training direction. However, obtaining good performance with such methods requires the orthogonality assumption to hold, which for distant languages is rarely the case (Patra et al., 2019). In fact, we find that the gradient-based MAT+MPSR method in a bilingual setting over typologically distant languages exhibits better performance than MUSES or VecMap. Across Table 2, in only a handful of examples (shaded cells) do VecMap or MUSES systems outperform MAT+MPSR for BWE (with the majority being among EN, ES, GL, and PR, all related high-resource languages).
In the 7 distant languages setting, however, the results are different: VecMap outperforms MUSES and the multilingual MAT+MPSR in the vast majority of the language pairs. The difference is more stark when the languages of the pair use completely different alphabets, where the same-character strings heuristic for bootstrapping the initial dictionary mapping fails. Instead, the monolingual similarity approach employed by VecMap is definitely more appropriate for settings such as those posed by languages like Korean or Hindi. This highlights the importance of actually evaluating and reporting results on such language pairs.
On the one hand, we find that when aligning distant
8664
Results on Az-Cs	Average
Bilingual with hub:	Az 22.7	Cs 29.1	25.8
Trilingual Az, Cs, +hub:	28.2
BE	EN	Es	GL
21.6	28.5	31.8	23.0
Pt	RU	SK	TR
29.6	27.4	30.4	32.9
Trilingual Az, hub:Cs, +extra:	30.8
EN Es	PT	RU	TR
30.1	33.2	27.1	33.7
Multilingual (10 languages)	33.9
AZ BE	Cs	EN	Es
33.7	34.0	32.3	34.5
GL Pt	RU	SK	TR
34.0	34.8	34.5	32.9
Results on Ru-UK	Average
Bilingual with hub:	RU 58.0	UK 57.0	57.5
Trilingual BE, RU, UK with hub:	BE 59.2	RU 58.9	UK 58.4	58.8
Trilingual Ru, UK, +hub:	57.8
Az 57.4	Cs 58.5	EN 58.4	Es 58.3	FR 58.0	Hi 57.0	TR 57.2
Multilingual BE, RU, UK, +hub:	58.1
Cs 58.0	EN 58.1	Es 58.5	GL 58.8	Ko 57.0	Pt 58.3	Sv 58.2
Multilingual Ru, UK, EN, FR, Hi, Ko, Sv, with hub:
EN 55.3	FR 56.1	HI 55.8	Ko 56.3	RU 55.3	Sv 55.3	UK 54.9	55.6
Table 4: Comparison of bilingual, trilingual, and multilingual systems for distant (left) and related (right) languages. Multilinguality boosts performance significantly on distant languages.
Test	Hub	Test	Hub
SRC	TRG	SRC	TRG
Az-Cs	22.7	29.1	GL-PT	53.5	53.6
Az-EN	13.2	20.7	PT-GL	39.0	36.7
Az-Tr	30.1	30.1	UK-RU	61.6	61.8
languages with MAT+MPSR, the difference between hub choices can be significant – in Az-EN, for instance, using EN as the hub leads to more than 7 percentage points difference compared to using Az. We show some examples in Table 5. On the other hand, when aligning typologically similar languages, the difference is less pronounced. For example, we obtain practically similar performance for GL-Pr, Az-Tr, or UK-Ru when using either the source or the target language as the hub. Note, though, that non-negligible differences could still occur, as in the case of Pr-GL. In most cases, it is the case that the higher-resourced language is a better hub than the lower-resourced one, especially when the number of resources differ significantly (as in the case of Az and BE against any other language). Since BWE settings are not our main focus, we leave an extensive analysis of this observation for future work.
Bi-, tri-, and multilingual systems This part of our analysis compares bilingual, trilingual, and multilingual systems, with a focus on the under-represented languages. Through multiple experiments (complete evaluations are listed in the Appendix) we reach two main conclusions. On one hand, when evaluating on typologically distant languages, one should use as many languages as possible. In Table 4 we present one such example with results on Az-Cs under various settings. On the other hand, when multiple related languages
Table 5: The hub is important for BWE between distant languages with MAT+MPSR.
Hub	Transfer from EN	Transfer from Pt
Es	Pt	GL	Hub	Es	GL
EN	38.7	21.8	19.4	EN	48.4	32.9
Es	26.5	16.1	28.5†	Es	41.4	25.5†
Pt	28.1	25.7	15.6	Pt	44.3†	36.5
GL	35.4	22.8	23.1	GL	48.1	23.8
BE	35.6	30.5	13.2
RU	28.6†	30.6	18.2	†: best train-test hub
SK	24.2	30.2†	14.6	for LI.
Table 6: The choice of hub can significantly affect downstream zero-shot POS tagging accuracy.
are available, one can achieve higher performance with multilingual systems containing all related languages and one more hub language, rather than learning diverse multilingual mappings using more languages. We confirm the latter observation with experiments on the Slavic (BE, RU, UK) and Iberian (Es, GL, Pt) clusters, and present an example (RU-UK) in Table 4.
5 Downstream Task Experiments
Differences in BLI performance do not necessarily translate to differences in other downstream tasks that use the aligned embeddings, so Glavaš et al. (2019) advocate for actual evaluation on such tasks. We extend our analysis to an example downstream task of zero-shot POS tagging using the aligned embeddings for select language pairs. We show that indeed the choice of the hub language can have dramatic impact. Using Universal Dependencies data (Nivre et al., 2016) we train simple bi-LSTM POS taggers on EN and PT using the respective embeddings produced from each MAT+MPSR run, and evaluate the zero-shot performance on GL and
8665
Es. $^{17}$  Although all taggers achieve consistent accuracies  $>95\%$  on English and Portuguese regardless of the original EN or  $\mathrm{Pt}$  embeddings, the zero-shot performance on the test languages, as shown in Table 6, varies widely. For instance, using the embeddings produced from using  $\mathrm{Pt}$  as a hub, we obtain the highest zero-shot accuracy on GL (36.5%), while using the ones from the GL hub lead to significantly worse performance (23.8%). It should be noted that the best hub for POS-tagging does not always coincide with the best hub for LI, e.g. the best LI hub for  $\mathrm{Pt - GL}$  is Es, which leads to 11 percentage points worse GL POS tagging performance than the best system. In fact, for the language pairs that we studied we observe no correlation between the two tasks performance as we vary the hub (with an average Spearman's rank correlation  $\rho = 0.08$ ).
With this work we challenge the standard practice in learning cross-lingual word embeddings. We empirically show that the choice of the hub language is an important parameter that affects lexicon induction performance in both bilingual (between distant languages) and multilingual settings. More importantly, we hope that by providing new dictionaries and baseline results on several language pairs, we will stir the community towards evaluating all methods in challenging scenarios that include under-represented language pairs. Towards this end, our analysis provides insights and general directions for stronger baselines for non-Anglocentric cross-lingual word embeddings. The problem of identifying the best hub language, despite our analysis based on the use of typological distance, remains largely unsolved. In the future, we will investigate a hub language ranking/selection model a la Lin et al. (2019).
The authors are grateful to the anonymous reviewers for their exceptionally constructive and insightful comments, and to Gabriela Weigel for her invaluable help with editing and proofreading the paper. This material is based upon work generously supported by the National Science Foundation under grant 1761548.
Zeljko Agić and Ivan Vulić. 2019. Jw300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3204–3210.
Jean Alaux, Edouard Grave, Marco Cuturi, and Armand Joulin. 2019. Unsupervised hyperalignment for multilingual word embeddings. In Proceedings of the
International Conference on Learning Representations.
Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2016a. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431-444.
Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. 2016b. Massively multilingual word embeddings. arXiv preprint arXiv:1602.01925.
Mark Aronoff and Kirsten Fudeman. 2011. What is morphology?, volume 8. John Wiley & Sons.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462, Vancouver, Canada. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
Xilun Chen and Claire Cardie. 2018. Unsupervised multilingual word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 261-270. Association for Computational Linguistics.
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proceedings of the Sixth International Conference on Learning Representations.
Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. 2019. Don't forget the long tail! a comprehensive analysis of morphological generalization in bilingual lexicon induction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 974-983, Hong Kong, China. Association for Computational Linguistics.
Georgiana Dinu and Marco Baroni. 2014. How to make words with vectors: Phrase generation in distributional semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 624-633.
<sup>17</sup>Note that our goal is not to achieve SOTA in zero-shot POS-tagging, but to show that embeddings resulting from different hub choices have different qualities.
8666
Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In Proceedings of the International Conference on Learning Representations, workshop track.
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648.
Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710–721, Florence, Italy. Association for Computational Linguistics.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
Geert Heyman, Bregt Verreet, Ivan Vulic, and Marie-Francine Moens. 2019. Learning unsupervised multilingual word embeddings with incremental multilingual hubs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1890-1902, Minneapolis, Minnesota. Association for Computational Linguistics.
Ann Irvine and Chris Callison-Burch. 2013. Combining bilingual and comparable corpora for low resource machine translation. In Proceedings of the eighth workshop on statistical machine translation, pages 262-270.
Yova Kementchedjhieva, Mareike Hartmann, and Anders Søgaard. 2019. Lost in evaluation: Misleading benchmarks for bilingual dictionary induction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Alexandre Klementiev, Ivan Titov, and Binod Bhattacharai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459-1474.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne, Chris Callison-Burch, Miles Osborne, and David Talbot. 2005. Edinburgh system description