The wide use and success of monolingual word embeddings in NLP tasks BIBREF0 , BIBREF1 has inspired further research focus on the induction of cross-lingual word embeddings (CLWEs). CLWE methods learn a shared cross-lingual word vector space where words with similar meanings obtain similar vectors regardless of their actual language. CLWEs benefit cross-lingual NLP, enabling multilingual modeling of meaning and supporting cross-lingual transfer for downstream tasks and resource-lean languages. CLWEs provide invaluable cross-lingual knowledge for, inter alia, bilingual lexicon induction BIBREF2 , BIBREF3 , information retrieval BIBREF4 , BIBREF5 , machine translation BIBREF6 , BIBREF7 , document classification BIBREF8 , cross-lingual plagiarism detection BIBREF9 , domain adaptation BIBREF10 , cross-lingual POS tagging BIBREF11 , BIBREF12 , and cross-lingual dependency parsing BIBREF13 , BIBREF14 .The landscape of CLWE methods has recently been dominated by the so-called projection-based methods BIBREF15 , BIBREF16 , BIBREF17 . They align two monolingual embedding spaces by learning a projection/mapping based on a training dictionary of translation pairs. Besides their simple conceptual design and competitive performance, their popularity originates from the fact that they rely on rather weak cross-lingual supervision. Originally, the seed dictionaries typically spanned several thousand word pairs BIBREF15 , BIBREF18 , BIBREF19 , but more recent work has shown that CLWEs can be induced with even weaker supervision from small dictionaries spanning several hundred pairs BIBREF20 , identical strings BIBREF21 , or even only shared numerals BIBREF22 .Taking the idea of reducing cross-lingual supervision to the extreme, the latest CLWE developments almost exclusively focus on fully unsupervised approaches BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 , BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 : they fully abandon any source of (even weak) supervision and extract the initial seed dictionary by exploiting topological similarities between pre-trained monolingual embedding spaces. Their modus operandi can roughly be described by three main components: C1) unsupervised extraction of a seed dictionary; C2) a self-learning procedure that iteratively refines the dictionary to learn projections of increasingly higher quality; and C3) a set of preprocessing and postprocessing steps (e.g., unit length normalization, mean centering, (de)whitening) BIBREF31 that make the entire learning process more robust.The induction of fully unsupervised CLWEs is an inherently interesting research topic per se. Nonetheless, the main practical motivation for developing such approaches in the first place is to facilitate the construction of multilingual NLP tools and widen the access to language technology for resource-poor languages and language pairs. However, the first attempts at fully unsupervised CLWE induction failed exactly for these use cases, as shown by sogaard2018on. Therefore, the follow-up work aimed to improve the robustness of unsupervised CLWE induction by introducing more robust self-learning procedures BIBREF24 , BIBREF32 . Besides increased robustness, recent work claims that fully unsupervised projection-based CLWEs can even match or surpass their supervised counterparts BIBREF23 , BIBREF24 , BIBREF27 , BIBREF33 , BIBREF34 .In this paper, we critically examine these claims on robustness and improved performance of unsupervised CLWEs by running a large-scale evaluation in the bilingual lexicon induction (BLI) task on 15 languages (i.e., 210 languages pairs, see Table 2 in § "Experimental Setup" ). The languages were selected to represent different language families and morphological types, as we argue that fully unsupervised CLWEs have been designed to support exactly these setups. However, we show that even the most robust unsupervised CLWE method BIBREF24 still fails for a large number of language pairs: 87/210 BLI setups are unsuccessful, yielding (near-)zero BLI performance. Further, even when the unsupervised method succeeds, it is because the components C2 (self-learning) and C3 (pre-/post-processing) can mitigate the undesired effects of noisy seed lexicon extraction. We then demonstrate that the combination of C2 and C3 with a small provided seed dictionary (e.g., 500 or 1K pairs) outscores the unsupervised method in all cases, often with a huge margin, and does not fail for any language pair. Furthermore, we show that the most robust unsupervised CLWE approach still fails completely when it relies on monolingual word vectors trained on domain-dissimilar corpora. We also empirically verify that unsupervised approaches cannot outperform weakly supervised approaches also for closely related languages (e.g., Swedish–Danish, Spanish–Catalan).While the “no supervision at all” premise behind fully unsupervised CLWE methods is indeed seductive, our study strongly suggests that future research efforts should revisit the main motivation behind these methods and focus on designing even more robust solutions, given their current inability to support a wide spectrum of language pairs. In hope of boosting induction of CLWEs for more diverse and distant language pairs, we make all 210 training and test dictionaries used in this work publicly available at: https://github.com/ivulic/panlex-bli.We now dissect a general framework for unsupervised CLWE learning, and show that the “bag of tricks of the trade” used to increase their robustness (which often slips under the radar) can be equally applied to (weakly) supervised projection-based approaches, leading to their fair(er) comparison.In short, projection-based CLWE methods learn to (linearly) align independently trained monolingual spaces $\mathbf {X}$ and $\mathbf {Z}$ , using a word translation dictionary $D_0$ to guide the alignment process. Let $\mathbf {X}_D \subset \mathbf {X}$ and $\mathbf {Z}_D \subset \mathbf {Z}$ be the row-aligned subsets of monolingual spaces containing vectors of aligned words from $D_0$ . Alignment matrices $\mathbf {X}_D$ and $\mathbf {Z}_D$ are then used to learn orthogonal transformations $\mathbf {W}_x$ and $\mathbf {W}_z$ that define the joint bilingual space $\mathbf {Z}$0 . While supervised projection-based CLWE models learn the mapping using a provided external (clean) dictionary $\mathbf {Z}$1 , their unsupervised counterparts automatically induce the seed dictionary in an unsupervised way (C1) and then refine it in an iterative fashion (C2).Unsupervised CLWEs. These methods first induce a seed dictionary $D^{(1)}$ leveraging only two unaligned monolingual spaces (C1). While the algorithms for unsupervised seed dictionary induction differ, they all strongly rely on the assumption of similar topological structure between the two pretrained monolingual spaces. Once the seed dictionary is obtained, the two-step iterative self-learning procedure (C2) takes place: 1) a dictionary $D^{(k)}$ is first used to learn the joint space $\mathbf {Y}^{(k)} = \mathbf {X{W}}^{(k)}_x \cup \mathbf {Z{W}}^{(k)}_z$ ; 2) the nearest neighbours in $\mathbf {Y}^{(k)}$ then form the new dictionary $D^{(k+1)}$ . We illustrate the general structure in Figure 1 .A recent empirical survey paper BIBREF17 has compared a variety of latest unsupervised CLWE methods BIBREF23 , BIBREF27 , BIBREF33 , BIBREF24 in several downstream tasks (e.g., BLI, cross-lingual information retrieval, document classification). The results of their study indicate that the vecmap model of artetxe2018robust is by far the most robust and best performing unsupervised CLWE model. For the actual results and analyses, we refer the interested reader to the original paper of glavas2019howto. Another recent evaluation paper BIBREF35 as well as our own preliminary BLI tests (not shown for brevity) have further verified their findings. We thus focus on vecmap in our analyses, and base the following description of the components C1-C3 on that model.C1. Seed Lexicon Extraction. vecmap induces the initial seed dictionary using the following heuristic: monolingual similarity distributions for words with similar meaning will be similar across languages. The monolingual similarity distributions for the two languages are given as rows (or columns; the matrices are symmetric) of $\mathbf {M}_x = \mathbf {X}\mathbf {X}^T$ and $\mathbf {M}_z = \mathbf {Z}\mathbf {Z}^T$ . For the distributions of similarity scores to be comparable, the values in each row of $\mathbf {M}_x$ and $\mathbf {M}_z$ are first sorted. The initial dictionary $D^{(1)}$ is finally obtained by searching for mutual nearest neighbours between the rows of $\sqrt{\mathbf {M}_x}$ and of $\sqrt{\mathbf {M}_z}$ .C2. Self-Learning. Not counting the preprocessing and postprocessing steps (component C3), self-learning then iteratively repeats two steps:1) Let $\mathbf {D}^{(k)}$ be the binary matrix indicating the aligned words in the dictionary $D^{(k)}$ . The orthogonal transformation matrices are then obtained as $\mathbf {W}^{(k)}_x = \mathbf {U}$ and $\mathbf {W}^{(k)}_z = \mathbf {V}$ , where $\mathbf {U}\mathbf {\Sigma }\mathbf {V}^T$ is the singular value decomposition of the matrix $\mathbf {X}^T\mathbf {D}^{(k)}\mathbf {Z}$ . The cross-lingual space of the $D^{(k)}$0 -th iteration is then $D^{(k)}$1 .2) The new dictionary $D^{(k+1)}$ is then built by identifying nearest neighbours in $\mathbf {Y}^{(k)}$ . These can be easily extracted from the matrix $\mathbf {P} = \mathbf {X}\mathbf {W}^{(k)}_x( \mathbf {Z}\mathbf {W}^{(k)}_z)^T$ . All nearest neighbours can be used, or additional symmetry constraints can be imposed to extract only mutual nearest neighbours: all pairs of indices ( $i, j$ ) for which $\mathbf {P}_{ij}$ is the largest value both in row $i$ and column $j$ .The above procedure, however, often converges to poor local optima. To remedy for this, the second step (i.e., dictionary induction) is extended with techniques that make self-learning more robust. First, the vocabularies of $\mathbf {X}$ and $\mathbf {Z}$ are cut to the top $k$ most frequent words. Second, similarity scores in $\mathbf {P}$ are kept with probability $p$ , and set to zero otherwise. This dropout allows for a wider exploration of possible word pairs in the dictionary and contributes to escaping poor local optima given the noisy seed lexicon in the first iterations.C3. Preprocessing and Postprocessing Steps. While iteratively learning orthogonal transformations $\mathbf {W}_{x}$ and $\mathbf {W}_{z}$ for $\mathbf {X}$ and $\mathbf {Z}$ is the central step of unsupervised projection-based CLWE methods, preprocessing and postprocessing techniques are additionally applied before and after the transformation. While such techniques are often overlooked in model comparisons, they may have a great impact on the model's final performance, as we validate in § "Results and Discussion" . We briefly summarize two pre-processing (S1 and S2) and post-processing (S3 and S4) steps used in our evaluation, originating from the framework of artetxe2018generalizing.S1) Normalization and mean centering. We first apply unit length normalization: all vectors in $\mathbf {X}$ and $\mathbf {Z}$ are normalized to have a unit Euclidean norm. Following that, $\mathbf {X}$ and $\mathbf {Z}$ are mean centered dimension-wise and then again length-normalized.S2) Whitening. ZCA whitening BIBREF36 is applied on (S1-processed) $\mathbf {X}$ and $\mathbf {Z}$ : it transforms the matrices such that each dimension has unit variance and that the dimensions are uncorrelated. Intuitively, the vector spaces are easier to align along directions of high variance.S3) Dewhitening. A transformation inverse to S2: for improved performance it is important to restore the variance information after the projection, if whitening was applied in S2 BIBREF31 .S4) Symmetric re-weighting. This step attempts to further align the embeddings in the cross-lingual embedding space by measuring how well a dimension in the space correlates across languages for the current iteration dictionary $D^{(k)}$ . The best results are obtained when re-weighting is neutral to the projection direction, that is, when it is applied symmetrically in both languages. In the actual implementation S1 is applied only once, before self-learning. S2, S3 and S4 are applied in each self-learning iteration.Model Configurations. Note that C2 and C3 can be equally used on top of any (provided) seed lexicon (i.e., $D^{(1)}$ := $D_0$ ) to enable weakly supervised learning, as we propose here. In fact, the variations of the three key components, C1) seed lexicon, C2) self-learning, and C3) preprocessing and postprocessing, construct various model configurations which can be analyzed to probe the importance of each component in the CLWE induction process. A selection of representative configurations evaluated later in § "Results and Discussion" is summarized in Table 1 .Evaluation Task. Our task is bilingual lexicon induction (BLI). It has become the de facto standard evaluation for projection-based CLWEs BIBREF16 , BIBREF17 . In short, after a shared CLWE space has been induced, the task is to retrieve target language translations for a test set of source language words. Its lightweight nature allows us to conduct a comprehensive evaluation across a large number of language pairs. Since BLI is cast as a ranking task, following glavas2019howto we use mean average precision (MAP) as the main evaluation metric: in our BLI setup with only one correct translation for each “query” word, MAP is equal to mean reciprocal rank (MRR).(Selection of) Language Pairs. Our selection of test languages is guided by the following goals: a) following recent initiatives in other NLP research (e.g., for language modeling) BIBREF39 , BIBREF40 , we aim to ensure the coverage of different genealogical and typological language properties, and b) we aim to analyze a large set of language pairs and offer new evaluation data which extends and surpasses other work in the CLWE literature. These two properties will facilitate analyses between (dis)similar language pairs and offer a comprehensive set of evaluation setups that test the robustness and portability of fully unsupervised CLWEs. The final list of 15 diverse test languages is provided in Table 2 , and includes samples from different languages types and families. We run BLI evaluations for all language pairs in both directions, for a total of 15 $\times $ 14=210 BLI setups.Monolingual Embeddings. We use the 300-dim vectors of Grave:2018lrec for all 15 languages, pretrained on Common Crawl and Wikipedia with fastText BIBREF41 . We trim all vocabularies to the 200K most frequent words.Training and Test Dictionaries. They are derived from PanLex BIBREF43 , BIBREF44 , which was used in prior work on cross-lingual word embeddings BIBREF45 , BIBREF46 . PanLex currently spans around 1,300 language varieties with over 12M expressions: it offers some support and supervision also for low-resource language pairs BIBREF47 . For each source language ( $L_1$ ), we automatically translate their vocabulary words (if they are present in PanLex) to all 14 target ( $L_2$ ) languages. To ensure the reliability of the translation pairs, we retain only unigrams found in the vocabularies of the respective $L_2$ monolingual spaces which scored above a PanLex-predefined threshold.As in prior work BIBREF23 , BIBREF17 , we then reserve the 5K pairs created from the more frequent $L_1$ words for training, while the next 2K pairs are used for test. Smaller training dictionaries (1K and 500 pairs) are created by again selecting pairs comprising the most frequent $L_1$ words.Training Setup. In all experiments, we set the hyper-parameters to values that were tuned in prior research. When extracting the unsupervised seed lexicon, the 4K most frequent words of each language are used; self-learning operates on the 20K most frequent words of each language; with dropout the keep probability $p$ is 0.1; CSLS with $k=10$ nearest neighbors BIBREF24 .Again, Table 1 lists the main model configurations in our comparison. For the fully unsupervised model we always report the best performing configuration after probing different self-learning strategies (i.e., +sl, +sl+nod, and +sl+sym are tested). The results for unsupervised are always reported as averages over 5 restarts: this means that with unsupervised we count BLI setups as unsuccessful only if the results are close to zero in all 5/5 runs. orthg-super is the standard supervised model with orthogonal projections from prior work BIBREF21 , BIBREF17 .Main BLI results averaged over each source language ( $L_1$ ) are provided in Table 3 and Table 4 . We now summarize and discuss the main findings across several dimensions of comparison.Unsupervised vs. (Weakly) Supervised. First, when exactly the same components C2 and C3 are used, unsupervised is unable to outperform a (weakly) supervised full+sl+sym variant, and the gap in final performance is often substantial. In fact, full+sl+sym and full+sl+nod outperform the best unsupervised for all 210/210 BLI setups: we observe the same phenomenon with varying dictionary sizes, that is, it equally holds when we seed self-learning with 5K, 1K, and 500 translation pairs, see also Figure 2 . This also suggests that the main reason why unsupervised approaches were considered on-par with supervised approaches in prior work BIBREF23 , BIBREF24 is because they were not compared under fair circumstances: while unsupervised relied heavily on the components C2 and C3, these were omitted when running supervised baselines. Our unbiased comparison reveals that there is a huge gap even when supervised projection-based approaches consume only several hundred translation pairs to initiate self-learning.Are Unsupervised CLWEs Robust? The results also indicate that, contrary to the beliefs established by very recent work BIBREF24 , BIBREF30 , fully unsupervised approaches are still prone to getting stuck in local optima, and still suffer from robustness issues when dealing with distant language pairs: 87 out of 210 BLI setups ( $=41.4\%$ ) result in (near-)zero BLI performance, see also Table 4 . At the same time, weakly supervised methods with a seed lexicon of 1k or 500 pairs do not suffer from the robustness problem and always converge to a good solution, as also illustrated by the results reported in Table 5 .How Important are Preprocessing and Postprocessing? The comparisons between orthg-super (and orthg+sl+sym) on the one hand, and full-super (and full+sl+sym) on the other hand clearly indicate that the component C3 plays a substantial role in effective CLWE learning. full-super, which employs all steps S1-S4 (see § "Methodology" ), outperforms orthg-super in 208/210 setups with $|D_0|$ =5k and in 210/210 setups with $|D_0|$ =1k. Similarly, full+sl+sym is better than orthg+sl+sym in 210/210 setups (both for $|D_0|$ =1k,5k). The scores also indicate that dropout with self-learning is useful only when we work with noisy unsupervised seed lexicons: full+sl+nod and full+sl+sym without dropout consistently outperform full+sl across the board.How Important is (Robust) Self-Learning? We note that the best self-learning method is often useful even when $|D_0|=5k$ (i.e., full+sl+sym is better than full-super in 164/210 setups). However, the importance of robust self-learning gets more pronounced as we decrease the size of $D_0$ : full+sl+sym is better than full-super in 210/210 setups when $|D_0|=500$ or $|D_0|=1,000$ . The gap between the two models, as shown in Figure 2 , increases dramatically in favor of full+sl+sym as we decrease $|D_0|$ .Again, just comparing full-super and unsupervised in Figure 2 might give a false impression that fully unsupervised CLWE methods can match their supervised counterparts, but the comparison to full+sl+sym reveals the true extent of performance drop when we abandon even weak supervision. The scores also reveal that the choice of self-learning (C2) does matter: all best performing BLI runs with $|D_0|=1k$ are obtained by two configs with self-learning, and full+sl+sym is the best configuration for 177/210 setups (see Table 4 ).Language Pairs. As suggested before by sogaard2018on and further verified by glavas2019howto and doval2019onthe, the language pair at hand can have a huge impact on CLWE induction: the adversarial method of conneau2018word often gets stuck in poor local optima and yields degenerate solutions for distant language pairs such as English-Finnish. More recent CLWE methods BIBREF24 , BIBREF30 focus on mitigating this robustness issue. However, they still rely on one critical assumption which leads them to degraded performance for distant language pairs: they assume approximate isomorphism BIBREF49 , BIBREF48 between monolingual embedding spaces to learn the initial seed dictionary. In other words, they assume very similar geometric constellations between two monolingual spaces: due to the Zipfian phenomena in language BIBREF50 such near-isomorphism can be satisfied only for similar languages and for similar domains used for training monolingual vectors. This property is reflected in the results reported in Table 3 , the number of unsuccessful setups in Table 4 , as well as later in Figure 4 .For instance, the largest number of unsuccessful BLI setups with the unsupervised model is reported for Korean, Thai (a tonal language), and Basque (a language isolate): their morphological and genealogical properties are furthest away from other languages in our comparison. A substantial number of unsuccessful setups is also observed with other two language outliers from our set (see Table 2 again), Georgian and Indonesian, as well as with morphologically-rich languages such as Estonian or Turkish.One setting in which fully unsupervised methods did show impressive results in prior work are similar language pairs. However, even in these settings when the comparison to the weakly supervised full-super+sym is completely fair (i.e., the same components C2 and C3 are used for both), unsupervised still falls short of full-super+sym. These results for three source languages are summarized in Figure 3 . What is more, one could argue that we do not need unsupervised CLWEs for similar languages in the first place: we can harvest cheap supervision here, e.g., cognates. The main motivation behind unsupervised approaches is to support dissimilar and resource-poor language pairs for which supervision cannot be guaranteed.Domain Differences. Finally, we also verify that unsupervised CLWEs still cannot account for domain differences when training monolingual vectors. We rely on the probing test of sogaard2018on: 300-dim fastText vectors are trained on 1.1M sentences on three corpora: 1) EuroParl.v7 BIBREF51 (parliamentary proceedings); 2) Wikipedia BIBREF52 , and 3) EMEA BIBREF53 (medical), and BLI evaluation for three language pairs is conducted on standard MUSE BLI test sets BIBREF23 . The results, summarized in Figure 4 , reveal that unsupervised methods are able to yield a good solution only when there is no domain mismatch and for the pair with two most similar languages (English-Spanish), again questioning their robustness and portability to truly low-resource and more challenging setups. Weakly supervised methods ( $|D_0|=500$ or $D_0$ seeded with identical strings), in contrast, yield good solutions for all setups.The superiority of weakly supervised methods (e.g., full+sl+sym) over unsupervised methods is especially pronounced for distant and typologically heterogeneous language pairs. However, our study also indicates that even carefully engineered projection-based methods with some seed supervision yield lower absolute performance for such pairs. While we have witnessed the proliferation of fully unsupervised CLWE models recently, some fundamental questions still remain. For instance, the underlying assumption of all projection-based methods (both supervised and unsupervised) is the topological similarity between monolingual spaces, which is why standard simple linear projections result in lower absolute BLI scores for distant pairs (see Table 4 and results in the supplemental material). Unsupervised approaches even exploit the assumption twice as their seed extraction is fully based on the topological similarity.Future work should move beyond the restrictive assumption by exploring new methods that can, e.g., 1) increase the isomorphism between monolingual spaces BIBREF54 by distinguishing between language-specific and language-pair-invariant subspaces; 2) learn effective non-linear or multiple local projections between monolingual spaces similar to the preliminary work of nakashole2018norma; 3) similar to vulic2016on and Lubin:2019naacl “denoisify” seed lexicons during the self-learning procedure. For instance, keeping only mutual/symmetric nearest neighbour as in full+sl+sym can be seen as a form of rudimentary denoisifying: it is indicative to see that the best overall performance in this work is reported with that model configuration.Further, the most important contributions of unsupervised CLWE models are, in fact, the improved and more robust self-learning procedures (component C2) and technical enhancements (component C3). In this work we have demonstrated that these components can be equally applied to weakly supervised approaches: starting from a set of only several hundred pairs, they can guarantee consistently improved performance across the board. As there is still no clear-cut use case scenario for unsupervised CLWEs, instead of “going fully unsupervised”, one pragmatic approach to widen the scope of CLWE learning and its application might invest more effort into extracting at least some seed supervision for a variety of language pairs BIBREF22 . This finding aligns well with the ongoing initiatives of the PanLex project BIBREF44 and the ASJP database BIBREF56 , which aim to collate at least some translation pairs in most of the world’s languages.Finally, this paper demonstrates that, in order to enable fair comparisons, future work on CLWEs should focus on evaluating the CLWE methods' constituent components (e.g, components C1-C3 from this work) instead of full-blown composite systems directly. One goal of the paper is to acknowledge that the work on fully unsupervised CLWE methods has indeed advanced state-of-the-art in cross-lingual word representation learning by offering new solutions also to weakly supervised CLWE methods. However, the robustness problems are still prominent with fully unsupervised CLWEs, and future work should invest more time and effort into developing more robust and more effective methods, e.g., by reaching beyond projection-based methods towards joint approaches BIBREF16 , BIBREF57 .This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909). The work of Goran Glavaš is supported by the Baden-Württemberg Stiftung (AGREE grant of the Eliteprogramm). Roi Reichart is partially funded by ISF personal grants No. 1625/18. We thank the three anonymous reviewers for their encouraging comments and suggestions.

Q: How are seed dictionaries obtained by fully unsupervised methods?

Q: How does BLI measure alignment quality?

Q: What methods were used for unsupervised CLWE?