Venue: NAACL 2018

# Evaluating bilingual word embeddings on the long tail

Fabienne Braune\({}^{1,2}\), Viktor Hangya\({}^{1}\), Tobias Eder\({}^{1}\), Alexander Fraser\({}^{1}\)

\({}^{1}\)Center for Information and Language Processing

LMU Munich, Germany

\({}^{2}\)Volkswagen Data Lab Munich, Germany

fabienne.braune@volkswagen.de {hangyav, fraser}@cis.uni-muenchen.de

tobias.eder@germanistik.uni-muenchen.de

###### Abstract

Bilingual word embeddings are useful for bilingual lexicon induction, the task of mining translations of given words. Many studies have shown that bilingual word embeddings perform well for bilingual lexicon induction but they focused on frequent words in general domains. For many applications, bilingual lexicon induction of rare and domain-specific words is of critical importance. Therefore, we design a new task to evaluate bilingual word embeddings on rare words in different domains. We show that state-of-the-art approaches fail on this task and present simple new techniques to improve bilingual word embeddings for mining rare words. We release new gold standard datasets and code to stimulate research on this task.

## 1 Introduction

Bilingual lexicon induction (BLI) is the task of generating accurate translations for each word in a list of source language words. Being able to perform BLI without parallel data is critical in many low resource scenarios. Bilingual word embeddings (BWEs) represent words from two different languages in the same vector space. BWEs have been shown to be very effective for BLI given a _small_ seed lexicon (around 5000 wordpairs) as the only bilingual signal. Until now, BWEs have been evaluated on frequent words from parliament proceedings or Wikipedia articles and reached good accuracies on these datasets. However, evaluations on rare and domain-specific words have not yet been provided even though such evaluation scenarios are critical for applications like machine translation (e.g., mining of translations for OOV (out-of-vocabulary) items) or bilingual terminology mining. In this paper, we design a novel evaluation scenario for BWEs: given (i) large amounts of monolingual data and (ii) a seed lexicon of frequent word-pairs, the goal is to create BWEs that enable accurate mining of rare words. As gold standard data, we release manually annotated pairs of rare words and their translations from three domains: (i) web crawls (ii) news commentaries (iii) medical texts. We show that state-of-the-art BWEs perform poorly on these data sets. We present simple techniques to build and combine BWEs that yield strong performance improvements. We study using fast-text to build BWEs, using ensembles of BWEs, and dealing with orthographic distance in BWEs, all of which improve results for the new task of rare word translation mining. A secondary contribution is improvements over state-of-the-art approaches on frequent words (which have been already extensively studied in previous work). We make our datasets and code publicly available1.

Footnote 1: [https://github.com/braunefe/BWEeval](https://github.com/braunefe/BWEeval)

## 2 Bilingual Induction of Rare Words

We briefly present how BLI is performed using BWEs and then introduce our new datasets.

**Bilingual Lexicon Induction.** The goal is to generate translations \(t\) in target language \(V_{t}\) of provided words \(s\) from source language \(V_{s}\). Given a BWE representing \(V_{s}\) and \(V_{t}\), an \(n\)-best list of translations for each word \(s\in V_{s}\) can be induced by taking the top \(n\) words \(t_{i}\in V_{t}\) whose representation \(x^{*}_{t_{i}}\) in the BWE is closest to the representation \(\vec{x_{s}}\) according to cosine distance.

**Datasets.** To create BWEs we use _post-hoc mapping_ which requires only monolingual texts and a small seed lexicon (see SS3). Our training set consists of two large monolingual corpora:

* general: 4,400,309 English and German sentences from parliament proceedings, news

commentaries and web crawls taken from the WMT 2016 shared task Bojar et al. (2016).
* medical: 3,108,183 English and German sentences from titles of medical Wikipedia articles, medical term-pairs, patents, documents from the European Medicines Agency.2 Footnote 2: This is taken from the in-domain part of: [https://ufal.mff.cuni.cz/ufal_medical_corpus](https://ufal.mff.cuni.cz/ufal_medical_corpus).

Seed Lexicons.Throughout the paper, we work with two lexicons. For each lexicon, we take the most common words and translate these by taking the top-ranked translation from a probabilistic dictionary.3 BWEs trained using this data are evaluated on our gold standards containing pairs of rare words (we will also report results on frequent words, as in previous work, see below).

Footnote 3: This word-level dictionary is taken from a standard phrase-based SMT system trained on WIT 2017 data.

* genLex: 4955 most frequent words from general
* medLex: 6079 most frequent words from medical

Gold Standards for Rare Words.We created gold standard data for rare words by randomly sampling words occurring between 3 and 5 times4 in general and medical. For general we sample rare words from news commentaries and web crawls separately, so we have two rare word data sets here. For each (English) sampled word, a German native speaker generated a German translation. We indicate the division into validation and test sets:

Footnote 4: Words with frequencies 1 and 2 are very often tokenization errors or borrowings from other languages, therefore we start at frequency 3. We did not consider tokenization errors as rare words and removed those from our data.

* crawLRARE: 1000 rare words from web crawls of general (250 validation, 750 test)
* newsRare: 1144 rare words from news commentaries of general (369 validation, 775 test)
* medRate: 2109 rare words from medical (1000 validation, 1109 test)

As English-German BLI of frequent words has not been studied before, following previous work, we annotated 2000 frequent English words taken from each of the General and Medical corpora with their German translations using the same probabilistic dictionary as was used to generate the Lexicon sets. These two silver standard datasets will also be released with the paper:

* genFreq: 2000 frequent words from general (1000 validation, 1000 test)
* medFree: 2000 frequent words from medical (1000 validation, 1000 test)

## 3Bilingual Word Embedding Creation

To create bilingual word embeddings, we use _post-hoc mapping_ (PHM), a method that projects monolingual words embeddings (MWEs) into a shared space using a linear transformation trained with a small seed lexicon Mikolov et al. (2013b); Faruqui and Dyer (2014); Xing et al. (2015); Lazaridou et al. (2015); Vulic and Korhonen (2016). Among methods to generate BWEs, PHM uses a very cheap bilingual signal.5

Footnote 5: Gouws and Sogaard (2015) and Duong et al. (2016) also leverage seed lexicons. However, in order to generate high quality BWEs, these approaches leverage much larger bilingual dictionaries.

Given MWEs in two languages \(V_{s}\) and \(V_{t}\), the goal of post-hoc mapping is to find a matrix \(\mathbf{W}\in\mathbb{R}^{d_{1}\times d_{2}}\) that maps each representation \(\vec{x_{s}}\in\mathbb{R}^{d_{1}}\) of a source word \(s\in V_{s}\) to the representation \(\vec{y_{t}}\in\mathbb{R}^{d_{2}}\) of its translation \(t\in V_{t}\). Typically, \(\mathbf{W}\) is learned using a seed lexicon \(L=\{(\vec{x_{1}},\vec{y_{1}}),\ldots,(\vec{x_{n}},\vec{y_{n}})\}\), where each pair \((\vec{x_{i}},\vec{y_{i}})\in V_{s}\times V_{t}\) are mutual translations. A common objective for cross-lingual mapping is ridge regression Mikolov et al. (2013b) (ridge), where \(\mathbf{W}\) is estimated by:

\[\mathbf{W}^{*}=\operatornamewithlimits{arg\,min}_{\mathbf{W}\in\mathbb{R}^{d_{ 1}\times d_{2}}}||\;\mathbf{X}\mathbf{W}-\mathbf{Y}\;||+\lambda\;||\;\mathbf{W}\;|| \tag{1}\]

where \(\mathbf{X}\) and \(\mathbf{Y}\) are stacked vectors of \(\vec{x_{i}}\) and \(\vec{y_{i}}\) respectively. Lazaridou et al. (2015) use a max-margin ranking loss (max-marg) to estimate \(\mathbf{W}\). For each \((\vec{x_{i}},\vec{y_{i}})\in L\), a candidate \(\vec{y_{i}}=\mathbf{W}\cdot\vec{x_{i}}\) is computed. The ranking loss is:

\[\sum_{j\neq i}^{k}\max\{0,\gamma+sim(\vec{y_{i}^{*}},\vec{y_{i}})- sim(\vec{y_{i}^{*}},\vec{y_{j}})\} \tag{2}\]

where \(\vec{y_{j}}\) is a randomly selected negative example, i.e., it is not a translation of \(\vec{x_{i}}\), \(k\) is the number of negative examples and \(sim(\vec{x},\vec{y})\) computes cosine similarity between \(\vec{x}\) and \(\vec{y}\). Hyperparameters \(\gamma\) and \(k\) are tuned on held-out validation data.6 



### Testing Previous Work

We reimplement Mikolov et al. (2013) as well as Lazaridou et al. (2015). To replicate their setup on English-German texts we first evaluate these on two standard tasks, mining frequent words from general and medical. We follow the approach of Heyman et al. (2017) and use English as the source language. First, we train 300 dimensional MWEs on the monolingual data using w2v7 with default parameters except that we lowered the minimum word frequency threshold to 3 Mikolov et al. (2013). To generate BWEs, we use mediocal and medLex for medRare and medFreq, while we use general and genLex for the rest of the test sets. We report results with the combination of skip-gram (w2vskip) or cbow (w2vcbow) and ridge or max-marg. As in previous work, we use top-1 (translation is the closest neighbor) and top-5 (translation is one of 5 closest neighbors) accuracies. The results in Table 1 show that the best performing setups are w2vskip with max-marg for genFreq and w2vcbow with ridge for medFreq. Accuracies are comparable to previous work (which was on different language pairs). The poor performance on MedFreq is consistent with Heyman et al. (2017), who introduced the task of mining frequent medical terms.

Footnote 7: [https://github.com/dav/word2vec](https://github.com/dav/word2vec)

## 4 Applying BWEs for Mining Rare Word-Pairs

We use the exact same BWEs training setup as above (SS3) and perform BLI on our new test sets of rare words. The results in Table 2 show that on **low frequency** word-pairs BWEs perform very poorly. Compared to standard evaluation scenarios (see Table 1) a massive performance decrease is observed. Low accuracy is clearly caused by the inability of context-based models (w2v) to build accurate embedding vectors for words occurring in very few contexts only. Through post-hoc mapping, these (poor) embeddings get projected randomly into the bilingual space which results in very poor performance on BLI especially for the medical domain.

### Using Subword Models

A first way to create BWEs that are better adapted to rare words is to generate MWEs that provide better vector representations for the words. One simple idea is to try to add subword information. We show empirically this helps BLI of rare words, which has not been shown before, to our knowledge. FastTextBojanowski et al. (2017) extends w2v by adding subword information \(s(w,c)\) to the context-based objective as follows:

\[s(w,c)=\sum_{g\in G_{w}}z^{\top}_{g}v_{c} \tag{3}\]

where \(G_{w}\subset\{1,...,G\}\) is a set of character \(n\)-gram indices corresponding to the n-grams that appear in the word \(w\), \(z_{g}\) is the vector representation of the \(n\)-gram and \(v_{c}\) is the vector of the context words. Subword information helps for rare words (by using \(n\)-gram information shared between words) and generates more accurate MWEs especially for morphologically rich languages like German. We create 300 dimensional MWEs using Fasttext skip-gram and cbow models with default parameters and with the same exception as before, i.e., we lowered the minimum word

\begin{table}
\begin{tabular}{l|l|r|r} Domain & mapping & w2v skip & w2v cobw \\ \hline \hline genFreq & ridge & 27.1 (43.7) & 24.0 (41.4) \\  & max-marg & **32.1 (47.7)** & 22.8 (40.0) \\ \hline medFreq & ridge & 14.9 (24.0) & **18.1 (30.1)** \\  & max-marg & 16.0 (27.2) & 16.8 (27.2) \\ \end{tabular}
\end{table}
Table 1: Bilingual lexicon induction of **frequent** word-pairs on **general and medical domain** data. We report top-1 and (top-5 in brackets) percentage accuracy. In this paper, bolding indicates a best result so far for a particular dataset.

\begin{table}
\begin{tabular}{l|l|r|r} Domain & mapping & ftt Skip & ftt chpow \\ \hline \hline crawIRare & ridge & 10.1 (14.7) & 4.7 (6.7) \\  & max-marg & **11.5 (15.9)** & 7.3 (12.1) \\ \hline newsRare & ridge & 23.2 (37.7) & 6.8 (13.5) \\  & max-marg & **25.3 (39.5)** & 15.1 (24.1) \\ \hline medRare & ridge & 12.2 (19.0) & 8.2 (14.2) \\  & max-marg & **12.5 (20.0)** & 8.8 (15.3) \\ \hline genFreq & ridge & 33.8 (51.4) & 16.2 (32.1) \\  & max-marg & **38.7 (56.5)** & 28.3 (45.3) \\ \hline medFreq & ridge & 17.8 (33.6) & 14.9 (26.7) \\  & max-marg & **29.3 (42.7)** & 19.9 (33.2) \\ \end{tabular}
\end{table}
Table 3: Bilingual lexicon induction using MWEs trained with fasttext (fitt).



frequency value to 3. We perform PHM using ridge and max-margin. The results in Table 3 show that this procedure yields impressive performance improvements. After evaluation8 we manually looked at the prediction of our models. We present examples in table 5. Examples 1-3 shows that the model improves non-trivial cases as well where the meanings of the incorrect predictions induced by W2V are not close to that of the input. We also show counterexamples 4 and 5 where subword elements cause errors by inducing hyponymies of the correct words. Generating BWEs with max-margin on these improved MWEs is particularly effective. By analyzing word similarities we saw that in BWEs acquired with ridge rare English words are often mapped near to noise. Because max-margin uses negative noisy word pairs as training examples this phenomenon is not as strongly present there.

Footnote 8: We added these examples to the camera-ready paper after the results were finalized.

### Model Ensembling

Although BWEs obtained with fasttext and max-margin clearly outperform other methods on rare words, a combination of BWEs obtained with different models can further improve performance by integrating several sources of information. We ensemble BWEs obtained using different MWEs as follows: we generate \(n\)-best lists (\(n=100\)) of translation candidates using each model. For each pair \((s,t)\) of candidate translations, we compute an ensemble weight given by a weighted sum of similarity scores \(Sim_{i}(s,t)\) ob

\begin{table}
\begin{tabular}{l|l|c|c|c|c} \hline Domain & mapping & ensemble & ensemble + edit & edit only & \% orth. close \\ \hline \hline crawI Rare & ridge & 10.3 (14.6) & 19.5 (22.1) & 19.0 (21.7) & 60.3 \\  & max-marg & 13.2 (17.6) & **19.5 (22.3)** & & \\ \cline{2-6} newsRare & ridge & 24.3 (39.9) & 32.0 (42.8) & & \\  & max-marg & 27.2 (40.0) & **32.8 (43.5)** & & \\ \hline medRate & ridge & 15.1 (20.6) & 25.5 (26.8) & & \\  & max-marg & 12.5 (21.2) & **26.3 (28.2)** & & \\ \hline genFreq & ridge & 42.9 (60.7) & 44.8 (62.0) & 16.0 (27.1) & 26.5 \\  & max-marg & 45.4 (63.2) & **47.2 (63.6)** & & \\ \hline medFreq & ridge & 31.6 (37.0) & 35.5 (44.2) & & \\  & max-marg & 37.7 (46.7) & **38.6 (47.4)** & & \\ \end{tabular}
\end{table}
Table 4: Bilingual lexicon induction of **low-frequency** and **frequent** word-pairs in different domains. _Ensemble_ denotes the results for ensembling all BWE models, _ensemble + edit_ shows results by adding orthographic similarity, _edit only_ denotes the results obtained by using only orthographic distance (all other weights set to 0) and % _orth. close_ shows the percentage of orthographically similar gold standard word pairs (whose normalized Levenshtein distance is at most \(0.3\)).

\begin{table}
\begin{tabular}{l|l|l|l|l|l} No. & source & model1 & model2 & corpus & glossary \\ \hline \hline  & \multicolumn{4}{c}{w2v skip} & ftt skip & \\ \hline
1. & snowstorms & skinhead & **schmeestürme** & crawI Rare & skinhead \\
2. & fire-extinguisers & goldztertifikate & **feueröschern** & crawI Rare & gold certificates \\
3. & tissue-specificity & bassimileu & **gewebespezizität** & medRate & base environment \\
4. & university & **université** & harvard–universalité & crawI Rare & Harvard University \\
5. & cabin & **kabine** & flugzeugkbaine & newsRare & airplane cabin \\  & \multicolumn{4}{c}{ftt skip} & ensemble & \\ \hline
6. & rubbish & milltonen & **mill** & newsRare & trashcan \\
7. & bathtub & badezimmer & **badewanne** & newsRare & bathroom \\
8. & parenthood & vaterschaft & **etherschaft** & newsRare & fatherhood \\
9. & cognitively & neurokognitiven & **kognityi** & medRate & neuro cognitive \\
10. & nanojoules & **nanojoule** & mkrotopfelchen & medRxiv & microdroplet \\  & \multicolumn{4}{c}{ensemble} & ensemble + edit & \\ \hline
11. & sled dogs & pferdeschitten/ahrten & **schlitenthunde** & crawI Rare & sled rides \\
12. & gnome-applets & gtkhtml & **genome-anwendungen** & crawI Rare & layout engine used by Gnome \\
13. & glutenins & getreidoproteinen & **glutetine** & medRate & grain proteins \\
14. & esterify & **verestern** & esteroil & medRate & ester oil \\ \hline \end{tabular}
\end{table}
Table 5: Examples comparing the predictions of the indicated models using ridge for the mapping where _model1_ and _model2_ shows the induced words for the given source. Bolding indicates the correct prediction and we give glosses for the incorrect predictions.

 

tained on each BWE:

\[\sum_{i=1}^{M}\gamma_{i}Sim_{i}(s,t) \tag{4}\]

\(Sim_{i}(s,t)\) is computed using cosine similarity. When a candidate pair \((s,t)\) is not in the \(n\)-best list generated by a model \(i\) then \(Sim_{i}(s,t)\) is set to 0. The weights \(\gamma_{i}\) for each test set are tuned on validation sets separately (presented in SS2) using grid search. The results (Table 4) show that ensembling yields significant gains over subword models alone for all data sets. We again looked for examples after evaluation (Table 5) where ensembling helped compared with the previous best setup (examples 6-9) and saw that the method again improves upon hard cases where the incorrect predictions are very close, in terms of meaning, to the gold annotation. Row 10 shows a counter-example. We note that this idea could be used in a supervised neural network for BLI as well, where information from multiple models could be integrated by concatenating embeddings from them for a given word.

### Adding Orthographic Distance

While subword information captures orthographic properties of words to a certain extent, it cannot precisely represent the orthographic distance of each word pairs in a predefined number of dimensions, especially not that of source and target word pairs when performing post-hoc mapping (MWEs are trained separately thus there is no such cross-lingual information). Thus, it is beneficial to strengthen BWEs by integrating a similarity measure between word strings directly. The BWEs ensemble in Equation 4 can easily be augmented with a weighted term \(\gamma_{M+1}OSim(s,t)\) that measures the orthographic similarity (which we define as one minus the normalized Levenshtein distance) between the surface-forms of words \(s\) and \(t\). We generate \(n\)-best lists of candidate translations using different BWE models as in SS4.2. In addition, we generate a list containing the \(n\) closest target words according to \(OSim(s,t)\) and ensemble all lists together. Results are shown in Table 4. To measure the impact of orthographic information alone, we also report results obtained when using this information only (all other ensemble weights set to \(0\)). For **low frequency** wordpairs, orthographic information leads to massive performance gains. We analyzed the gold standard word pairs in our datasets from the perspective of orthographic similarity. For crawLare and medRare the ratios of similar words are high which explains the large improvements obtained by adding this measure. Even though the ratio is not high for newsRare and the two frequent datasets, orthographic information still improves performance which shows the advantage of using the technique in all cases. Table 5 shows non-trivial examples (11-13) where orthographic distance improves performance. Example 11 shows the advantage of combining the vector representation with orthographic distance, i.e., our model could find translations of _sided dogs_ that have similar meaning, while in examples 12 and 13 orthographic distance helped to pick the correct translation which is the closest in terms of edit distance. On the other hand, in example 14 orthographic distance caused an error because the incorrect prediction is too close to the source word in orthographic distance.

## 5 Conclusion

We evaluated BWEs on the novel task of rare term mining in different domains. Our experiments show that previous approaches to bilingual lexicon induction fail when mining rare words. We have studied techniques for decreasing the impact of these problems. By ensembling different BWEs and combining those with orthographic cues, we have reached state-of-the-art results. By making our code and datasets publicly available, we hope to encourage other researchers to further enhance BWEs to perform well on this important task. In the future, we would like to work on BLI of multi-word translations and compound words.

## Acknowledgments

We would like to thank Helmut Schmid and the anonymous reviewers for their valuable input. This project has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement No 644402 (HimL). This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 640550) 



## References

* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _TACL_ .
* Bojar et al. (2016) Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In _Proceedings of the First Conference on Machine Translation_. Association for Computational Linguistics, Berlin, Germany, pages 131-198.
* Duong et al. (2016) Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2016. Learning crosslingual word embeddings without bilingual corpora. In _Proc. EMNLP_.
* Faruqui and Dyer (2014) Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In _Proc. EACL_.
* Gouws and Sogaard (2015) Stephan Gouws and Anders Sogaard. 2015. Simple task-specific bilingual word embeddings. In _Proc. NAACL_.
* Heyman et al. (2017) Geert Heyman, Ivan Vulic, and Marie-Francine Moens. 2017. Bilingual lexicon induction by learning to combine word-level and character-level representations. In _Proc. EACL_.
* Lazaridou et al. (2015) Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In _Proc. ACL_.
* Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In _ICLR_.
* Mikolov et al. (2013b) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. _CoRR_ abs/1309.4168.
* Mikolov et al. (2013c) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013c. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26_. Curran Associates, Inc., pages 3111-3119.
* Vulic and Korhonen (2016) Ivan Vulic and Anna Korhonen. 2016. On the role of seed lexicons in learning bilingual word embeddings. In _Proc. ACL_.
* Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In _Proc. NAACL_.