SlowGuess/ACL_2020_OCR huggingface dataset:
{aanastas,gneubig}@cs.cmu.edu
Most of recent work in cross-lingual word embeddings is severely Anglocentric. The vast majority of lexicon induction evaluation dictionaries are between English and another language, and the English embedding space is selected by default as the hub when learning in a multilingual setting. With this work, however, we challenge these practices. First, we show that the choice of hub language can significantly impact downstream lexicon induction and zero-shot POS tagging performance. Second, we both expand a standard English-centered evaluation dictionary collection to include all language pairs using triangulation, and create new dictionaries for under-represented languages.<sup>1</sup> Evaluating established methods over all these language pairs sheds light into their suitability for aligning embeddings from distant languages and presents new challenges for the field. Finally, in our analysis we identify general guidelines for strong cross-lingual embedding baselines, that extend to language pairs that do not include English.
Continuous vectors for representing words (embeddings) (Turian et al., 2010) have become ubiquitous in modern, neural NLP. Cross-lingual representations (Mikolov et al., 2013) additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several downstream tasks such as Part-Of-Speech (POS) tagging (Zhang et al., 2016), parsing (Ammar et al., 2016a), document classification (Klementiev et al., 2012), and machine translation (Irvine and Callison-Burch, 2013; Artetxe et al., 2018b; Lample et al., 2018).
Often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter BWE and MWE, respectively). First, monolingual word embeddings are learned over
large swaths of text. Such pre-trained word embeddings, such as the fastText Wikipedia vectors (Grave et al., 2018), are available for many languages and are widely used. Second, a mapping between the languages is learned in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision (Zou et al., 2013), under minimal supervision e.g. using only identical strings (Smith et al., 2017), or even in an unsupervised fashion (Zhang et al., 2017; Conneau et al., 2018). Both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned (hereinafter "the hub"). We outline the details in Section 2.
Despite all the recent progress in learning crosslingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. Notably, most MWE approaches essentially select English as the hub during training by default, aligning all other language spaces to the English one. We argue and empirically show, however, that English is a poor hub language choice. In BWE settings, on the other hand, it is fairly uncommon to denote which of the two languages is the hub (often this is implied to be the target language). However, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages.
This Anglocentricity is even more evident at the evaluation stage. The lexica most commonly used for evaluation are the MUSE lexica (Conneau et al., 2018) which cover 45 languages, but with translations only from and into English. Alternative evaluation dictionaries are also very English- and European-centric: (Dinu and Baroni, 2014) report results on English-Italian, (Artetxe et al., 2017) on English-German and English-Finnish, (Zhang et al., 2017) on Spanish-English and Italian-English, and (Artetxe et al., 2018a) between English and Italian, German, Finnish, Spanish, and Turkish. We argue that cross-lingual word embedding mapping methods should look beyond English for their evaluation benchmarks because, compared to all others, English is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems (Aronoff and Fudeman, 2011). These two factors allow for an
Available at https://github.com/antonisa/ embeddings.
8658
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8658-8679 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
overly easy evaluation setting which does not necessarily generalize to other language pairs. In light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages.
With this work, we attempt to address these shortcomings, providing the following contributions:
- We show that the choice of the hub when evaluating on diverse language pairs can lead to significantly different performance for iterative refinement methods that use a symbolic-based seed dictionary (e.g., by more than 10 percentage points for BWE over distant languages). We also show that often English is a suboptimal hub for MWE.
- We identify some general guidelines for choosing a hub language which could lead to stronger performance; less isometry between the hub and source and target embedding spaces mildly correlates with performance, as does typological distance (a measure of language similarity based on language family membership trees). For distant languages, multilingual systems should be preferred over bilingual ones if the languages share alphabets, otherwise a bilingual system based on monolingual similarity dictionaries is preferable.
- We provide resources for training and evaluation on language pairs that do not include English. We outline a simple triangulation method with which we extend the MUSE dictionaries to an additional 4704 lexicons covering 50 languages (for a total of 4900 dictionaries, including the original English ones), and we present results on a subset of them. We also create new evaluation lexica for under-resourced, under-represented languages using Azerbaijani, Belarusian, and Galician as our test cases. Finally, we provide recipes for creating such dictionaries for any language pair with available parallel data.
2 Cross-Lingual Word Embeddings and Lexicon Induction
Bilingual Word Embeddings In the supervised BWE setting of Mikolov et al. (2013), given two languages  $\mathcal{L} = \{l_1, l_2\}$  and their pre-trained row-aligned embeddings  $\chi_1, \chi_2$ , respectively, a transformation matrix  $M$  is learned such that:
$$
M = \operatorname * {a r g   m i n} _ {M \in \Omega} \| \mathcal {X} _ {1} - M \mathcal {X} _ {2} \|.
$$
The set  $\Omega$  can potentially impose a constraint over  $M$ , such as the very popular constraint of restricting it to be orthogonal (Xing et al., 2015). Previous work has empirically found that this simple formulation is competitive with other more complicated alternatives (Xing et al., 2015). The orthogonality assumption ensures that there exists a closed-form solution through Singular
Value Decomposition (SVD) of  $\mathcal{X}_1\mathcal{X}_2^T$ . Note that in this case only a single matrix  $M$  needs to be learned, because  $\| \mathcal{X}_1 - M\mathcal{X}_2\| = \left\| M^{-1}\mathcal{X}_1 - \mathcal{X}_2\right\|$ , while at the same time a model that minimizes  $\| \mathcal{X}_1 - M\mathcal{X}_2\|$  is as expressive as one minimizing  $\| M_1\mathcal{X}_1 - M_2\mathcal{X}_2\|$ , with half the parameters.
In the minimally supervised or even the unsupervised setting, Zhang et al. (2017) and Conneau et al. (2018) reframe the task as an adversarial game, with a generator aiming to produce a transformation that can fool a discriminator. However, the most popular methods follow an iterative refinement approach (Artetxe et al., 2017). Starting with a seed dictionary (e.g. from identical strings (Zhou et al., 2019) or numerals) an initial mapping is learned in the same manner as in the supervised setting. The initial mapping, in turn, is used to expand the seed dictionary with high confidence word translation pairs. The new dictionary is then used to learn a better mapping, and so forth the iterations continue until convergence. The same iterative approach is followed by Artetxe et al. (2018a), with one important difference that allows their model (VecMap) to handle language pairs with different alphabets: instead of identical strings, the seed dictionary is constructed based on the similarity of the monolingual similarity distributions over all words in the vocabulary. $^3$
Multilingual Word Embeddings In a multilingual setting, the simplest approach is to use BWE and align all languages into a target language (the hub). In this case, for  $N$  languages  $\mathcal{L} = \{l_1,l_2,\dots ,l_N\}$  on has to learn  $N - 1$  bilingual mappings (Ammar et al., 2016b). Rather than using a single hub space, Heyman et al. (2019) propose an incremental procedure that uses an Incremental Hub Space (IHS): each new language is included to the multilingual space by mapping it to all languages that have already been aligned (e.g. language  $l_{3}$  would be mapped to the aligned space of  $\{l_1,l_2\}$ ).
Alternatively, all mappings could be learned jointly, taking advantage of the inter-dependencies between any two language pairs. Importantly, though, there is no closed form solution for learning the joint mapping, hence a solution needs to be approximated with gradient-based methods. The main approaches are:
- Multilingual adversarial training with pseudorandomized refinement (Chen and Cardie, 2018, MAT+MPSR): a generalization of the adversarial approach of Zhang et al. (2017); Conneau et al. (2018) to multiple languages, also combined with an iterative refinement procedure. $^4$
- Unsupervised Multilingual Hyperalignment (Alaux et al., 2019, UMH): an approach
2We refer the reader to Mikolov et al. (2013) for details.
3We refer the reader to Artetxe et al. (2018a) for details.
$^4$ MAT+MPSR has the beneficial property of being as computationally efficient as learning  $O(N)$  mappings (instead of  $O(N^2)$ ). We refer the reader to Chen and Cardie (2018) for exact details.
8659
which maps all languages to a single hub space, but also enforces good alignments between all language pairs within this space.
Even though the architecture and modeling approach of all MWE methods are different, they share the same conceptual traits: one of the language spaces remains invariant and all other languages are effectively mapped to it. In all cases, English is by default selected to be the hub. The only exception is the study of triplets alignments in (Alaux et al., 2019), where Spanish is used as the Spanish-French-Portuguese triplet hub.
Lexicon Induction One of the most common downstream evaluation tasks for the learned cross-lingual word mappings is Lexicon Induction (LI), the task of retrieving the most appropriate word-level translation for a query word from the mapped embedding spaces. Specialized evaluation (and training) dictionaries have been created for multiple language pairs. Of these, the MUSE dictionaries (Conneau et al., 2018) are most often used, providing word translations between English (EN) and 48 other high- to mid-resource languages, as well as on all 30 pairs among 6 very similar Romance and Germanic languages (English, French, German, Spanish, Italian, Portuguese).
Given the mapped embedding spaces, the translations are retrieved using a distance metric, with Cross-Lingual Similarity Scaling (Conneau et al., 2018, CSLS) as the most commonly used in the literature. Intuitively, CSLS decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). The retrieved pairs are compared to the gold standard and evaluated using precision at  $k$  ( $\mathrm{P} @ k$ , evaluating how often the correct translation is within the  $k$  retrieved nearest neighbors of the query). Throughout this work we report  $\mathrm{P} @ 1$ , which is equivalent to accuracy; we provide  $\mathrm{P} @ 5$  and  $\mathrm{P} @ 10$  results in the Appendix.
3 New LI Evaluation Dictionaries
The typically used evaluation dictionaries cover a narrow breadth of the possible language pairs, with the majority of them focusing in pairs with English (as with the MUSE or Dinu et al. (2015) dictionaries) or among high-resource European languages. Glavaš et al. (2019), for instance, highlighted Anglocentricity as an issue, creating and evaluating on 28 dictionaries between 8 languages (Croatian, English, Finnish, French, German, Italian, Russian, Turkish) based on Google Translate. In addition, Czarnowska et al. (2019) focused on the morphology dimension, creating morphologically complete dictionaries for 2 sets of 5 genetically related languages (Romance: French, Spanish, Italian, Portuguese, Catalan; and Slavic: Polish, Czech, Slovak, Russian, Ukrainian). In contrast to these two (very valuable!) works, our method for creating dictionaries
Figure 1: Transitivity example (Portuguese  $\rightarrow$  English  $\rightarrow$  Czech).
for low-resource languages ( $\S 3.1$ ) leverages resources that are available for about 300 languages. In addition, we propose a simple triangulation process ( $\S 3.2$ ), that makes it possible to create dictionaries for arbitrary language pairs, given that dictionaries into a pivot language (usually English) are available for both languages.
3.1 Low-Resource Language Dictionaries
Our approach for constructing dictionaries is straightforward, inspired by phrase table extraction techniques from phrase-based MT (Koehn, 2009). This is an automatic process, and introduces some degree of noise. Rather than controlling this through manual inspection, which would be impossible for all language pairs, we rely on fairly simple heuristics for controlling the dictionaries' quality.
The first step is collecting publicly available parallel data between English and the low-resource language of interest. We use data from the TED (Qi et al., 2018), OpenSubtitles (Lison and Tiedemann, 2016), WikiMatrix (Schwenk et al., 2019), bible (Malaviya et al., 2017), and JW300 (Agić and Vulić, 2019) datasets. This results in  $354\mathrm{k}$ ,  $53\mathrm{k}$ , and  $623\mathrm{k}$  English-to-X parallel sentences for Azerbaijani (Az), Belarusian (Be), and Galician (GL) respectively. We align the parallel sentences using fast_align (Dyer et al., 2013), and extract symmetrized alignments using the gdfa heuristic (Koehn et al., 2005). In order to ensure that we do not extract highly domain-specific word pairs, we only use the TED, OpenSubtitles, and WikiMatrix parts for word-pair extraction. Also, in order to control for quality, we only extract word pairs if they appear in the dataset more than 5 times, and if the symmetrized alignment probability is higher than  $30\%$  in both directions.
With this process, we end up with about 6k, 7k, and 38k word pairs for Az-EN, BE-EN, and GL-EN respectively. Following standard conventions, we sort the word pairs according to source-side frequency, and use the intermediate-frequency ones for evaluation, typically using the [5000-6500) rank boundaries. The same process can be followed for any language pair with a sufficient volume of parallel data (needed for training a reasonably accurate word alignment model).
6Not all languages are available in all these datasets.
7The anglocentricity in this step is by necessity - it is hard to find a large volume of parallel data in a language pair excluding English.
In fact, we can produce similar dictionaries for a large number of languages, as the combination of the recently cre
<sup>5</sup>Note that Alaux et al. (2019) use the term pivot to refer to what we refer to as the hub language.
Greek	Italian	Match	Bridged Greek-Italian Lexicon
word	tag	word	tag	Greek	Italian
ειρηνικός	M;NOM;SG	pacifico	M;SG	M;SG	ειρηνικός	pacifico, paeificei, pacifica
ειρηνική	F;NOM;SG	pacifici	M;PL	F;SG	ειρηνική	pacifica, pacifico, paeificei
ειρηνικό	Neut;NOM;SG	pacifica	F;SG	SG	ειρηνικό	pacifica, pacifico, paeificei
ειρηνικά	Neut;NOM;PL	PL	ειρηνικά	pacifici, pacifica, paeifico
Table 1: Triangulation and filtering example on Greek-Italian. All words are valid translations of the English word 'peaceful'. We also show filtered-out translations.
3.2 Dictionaries for all Language Pairs through Triangulation
Our second method for creating new dictionaries is inspired by phrase table triangulation ideas from the pre-neural MT community (Wang et al., 2006; Levinboim and Chiang, 2015). The concept can be easily explained with an example, visualized in Figure 1. Consider the Portuguese  $(\mathrm{Pr})$  word travaího which, according to the MUSE  $\mathrm{Pr} - \mathrm{EN}$  dictionary, has the words job and work as possible EN translations. In turn, these two EN words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the transitive property (which translation should exhibit) we can identify the set of 5 possible Cs translations for the  $\mathrm{Pr}$  word travaího. Following this simple triangulation approach, we create 4,704 new dictionaries over pairs between the 50 languages of the MUSE dictionaries. For consistency, we keep the same train and test splits as with MUSE, so that the source-side types are equal across all dictionaries with the same source language.
Triangulating through English (which is unavoidable, due to the relative paucity of non-English-centric dictionaries) is suboptimal – English is morphologically poor and lacks corresponding markings for gender, case, or other features that are explicitly marked in many languages. As a result, several inflected forms in morphologically-rich languages map to the same English form. Similarly, gendered nouns or adjectives in gendered languages map to English forms that lack gender information. For example, the MUSE Greek-English dictionary lists the word peaceful as the translation for all ἐρηνικός, ἐρηνική, ἐρηνικό, ἐρηνικά, which are the male, female, and neutral (singular and plural) inflections of the same adjective. Equivalently, the English–Italian dictionary translates peaceful into either pacifico, pacifici, or pacifica (male singular, male plural, and female singular, respectively; see Table 1). When translating from or into English lacking context, all of those are reasonable translations. When translating between Greek and Italian, though, one should at least take number into account (gram
matical gender is a more complicated matter: it is not uncommon for word translations to be of different grammatical gender across languages).
Hence, we devise a filtering method for removing blatant mistakes when triangulating morphologically rich languages. We rely on automatic morphological tagging which we can obtain for most of the MUSE languages, using the StanfordNLP toolkit (Qi et al., 2020). The morphological tagging uses the Universal Dependencies feature set (Nivre et al., 2016) making the tagging comparable across almost all languages. Our filtering technique iterates through the bridged dictionaries: for a given source word, if we find a target word with the exact same morphological analysis, we filter out all other translations with the same lemma but different tags. In the case of feature mismatch (for instance, Greek uses 2 numbers, 4 cases and 3 genders while Italian has 2 numbers, 2 genders, and no cases) or if we only find a partial tag match over a feature subset, we filter out translations with disagreeing tags. We ignore the grammatical gender and verb form features, as they are not directly comparable cross-lingually. Coming back to our Greek-Italian example, this means that for the form ἐρηνικός we would only keep pacifico as a candidate translation (we show more examples in Table 1).
Our filtering technique removes about  $60.4\%$  of the entries in 2964 of the 4900 dictionaries.[11] Unsurprisingly, we find that bridged dictionaries between morphologically rich languages require a lot more filtering. For instance more than  $80\%$  of the entries of the Urdu-Greek dictionary get filtered out. On average, the languages with more filtered entries are Urdu  $(62.4\%)$ , Turkish  $(61.1\%)$ , and German  $(58.6\%)$ . On the other hand, much fewer entries are removed from dictionaries with languages like Dutch  $(36.2\%)$  or English  $(38.1\%)$ . Naturally, this filtering approach is restricted to languages for which a morphological analyzer is available. Mitigating this limitation is beyond the scope of this work, although it is unfortunately a common issue. For example, Kementchedjhieva et al. (2019) manually corrected five dictionaries (between English and German, Danish, Bulgarian, Arabic, Hindi) but one needs to rely
ated JW300 and WikiMatrix datasets provide an average of more than 100k parallel sentences in 300 languages. Before publication, we plan to create these dictionaries and make them publicly available, along with the corresponding code.
<sup>9</sup>Available at https://github.com/antonisa/embeddings.
10The toolkit has since been renamed to Stanza. See https://stanfordnlp.github.io/stanfordnlp/.
<sup>11</sup>Due to the lack of morphological analysis tools, we were unable to filter dictionaries in the following 11 languages: aze, bel, ben, bos, lit, mkd, msa, sqi, tam, tha, tel.
8661
src	Target
Az	BE	Cs	EN	Es	GL	Pt	RU	Sk	TR	μBEST
Az	-	17.2EN	35.1ES	35.7ES	48.0TR	32.7RU	41.5EN	29.8PT	31.7CS	32.0PT	33.7
BE	14.1CS	-	35.9TR	29.9PT	39.5EN	25.8ES	34.4ES	41.1GL	30.7RU	20.4PT	30.2
Cs	6.9ES	9.3RU	-	61.0ES	60.5EN	27.9PT	57.8EN	45.9PT	71.2EN	35.8SK	41.8
EN	17.9ES	18.4ES	50.2ES	-	77.5RU	36.3ES	72.3SK	43.3PT	40.4TR	41.9PT	44.2
Es	12.1EN	10.1RU	47.4PT	74.6SK	-	37.5ES	83.1GL	41.9TR	40.0ES	38.6SK	42.8
GL	5.5EN	3.6AZ	26.5TR	43.2ES	60.8TR	-	52.9CS	23.8TR	26.8CS	19.7CS	29.2
PT	5.8PT	8.6SK	47.2GL	71.3EN	88.1PT	37.1ES	-	38.0ES	38.7ES	38.1EN	41.4
RU	8.7ES	12.8AZ	50.3GL	55.5TR	54.8CS	23.0PT	52.4EN	-	45.5TR	27.0BE	36.7
SK	4.0BE	10.9RU	72.5BE	55.6TR	53.9EN	28.4EN	52.0ES	44.0GL	-	28.5EN	38.9
TR	12.1SK	9.0AZ	41.8RU	51.1CS	55.0EN	18.4TR	51.6EN	34.6EN	29.4ES	-	33.7
μBEST	9.7	11.1	45.2	53.1	59.8	29.7	55.3	38.0	39.4	31.3	37.3
μEN	9.1	9.9	43.3	51.0	59.3	28.2	54.9	36.5	37.7	30.8
Table 2: Lexicon Induction performance (measured with P@1) over 10 languages (90 pairs). In each cell, the superscript denotes the hub language that yields the best result for that language pair.  $\mu^{\mathrm{BEST}}$ : average using the best hub language.  $\mu^{\mathrm{EN}}$ : average using the EN as the hub. The lightly shaded cells are the language pairs where a bilingual VecMap system outperforms MAT+MSPR; in heavy shaded cells both MUSEs and VecMap outperform MAT+MSPR.
on automated annotations in order to scale to all languages. Our method that uses automatically obtained morphological information combined with the guidelines proposed by Kementchedjhieva et al. (2019) (e.g. removing proper nouns from the evaluation set) scales easily to multiple languages, allowing us to create more than 4 thousand dictionaries.
4 Lexicon Induction Experiments
The aim of our LI experiments is two-fold. First, the differences in LI performance show the importance of the hub language choice with respect to each evaluation pair. Second, as part of our call for moving beyond Anglo-centric evaluation, we also present LI results on several new language pairs using our triangulated dictionaries.
4.1 Methods and Setup