Venue: NAACL 2019

# Context-Aware Cross-Lingual Mapping

Hanan Aldarmaki\({}^{1}\)  and   Mona Diab\({}^{1,2}\)

\({}^{1}\)The George Washington University

\({}^{2}\)AWS, Amazon AI

aldarmaki@gwu.edu,diabmona@amazon.com

###### Abstract

Cross-lingual word vectors are typically obtained by fitting an orthogonal matrix that maps the entries of a bilingual dictionary from a source to a target vector space. Word vectors, however, are most commonly used for sentence or document-level representations that are calculated as the weighted average of word embeddings. In this paper, we propose an alternative to word-level mapping that better reflects sentence-level cross-lingual similarity. We incorporate context in the transformation matrix by directly mapping the averaged embeddings of aligned sentences in a parallel corpus. We also implement cross-lingual mapping of deep contextualized word embeddings using parallel sentences with word alignments. In our experiments, both approaches resulted in cross-lingual sentence embeddings that outperformed context-independent word mapping in sentence translation retrieval. Furthermore, the sentence-level transformation could be used for word-level mapping without loss in word translation quality.

## 1 Introduction

Cross-lingual word vector models aim to embed words from multiple languages into a shared vector space to enable cross-lingual transfer and dictionary expansion Upadhyay et al. (2016). One of the most common and effective approaches for obtaining bilingual word embeddings is by fitting a linear transformation matrix on the entries of a bilingual seed dictionary Mikolov et al. (2013). This approach is versatile and scalable: multilingual embeddings can be obtained by mapping the vector spaces of multiple languages into a shared target language, typically English. In addition, imposing an orthogonality constraint on the mapping ensures that the original pair-wise distances are preserved after the transformation and results in better word translation retrieval Artetxe et al. (2016); Smith et al. (2017).

While word vector spaces tend to be globally consistent across language variations Aldarmaki et al. (2018), individual words like homographs with unrelated senses (e.g. 'bank', 'coast') and phrasal verbs ('stand up', 'stand out') are likely to behave less consistently in multilingual vector spaces due to their different usage distributions. Consequently, using such words in the alignment dictionary may result in suboptimal overall mapping. We propose two approaches to counteract this effect by incorporating sentential context in the mapping process without explicit word sense disambiguation or additional linguistic resources. The first approach is based on the recently proposed contextualized embeddings from language models, ELMo Peters et al. (2018). Using a parallel corpus with word-alignments, we extract contextualized embeddings to construct a context-aware dictionary for mapping. The second approach is to learn a transformation between sentence embeddings rather than individual word embeddings. Since these embeddings include context that spans full sentences, we surmisie that a mapping learned at this level would be more robust to individual word misalignments.

We used a constrained set of parallel sentences ranging from one hundred to a million sentences for alignment. We then evaluated the resultant mappings on sentence translation retrieval among English, Spanish, and German as test languages. Our results show that context-aware mappings significantly outperform context-independent cross-lingual word mappings using reasonably-sized parallel corpora, particularly when using contextualized word embeddings. In addition, when averaging static word embeddings, the sentence-level mapping can still be used for word-level mapping without loss in word translation quality.



Related Work

For cross-lingual alignment, we follow the popular approach of fitting a linear transformation matrix between word vector spaces that are independently trained for each language. Aligning monolingual word vector spaces using a seed dictionary was originally proposed in Mikolov et al. (2013). In Artetxe et al. (2016) and Smith et al. (2017), it was shown that imposing an orthogonality constraint on the transformation leads to better word translation quality. Recently, contextualized word embeddings were proposed, where a sequential neural network is trained as a language model and then used to extract context-sensitive word representations from the hidden states Peters et al. (2018). We use parallel text in order to align independently-trained contextualized embeddings across languages. Schuster et al. (2019) independently proposed a cross-lingual alignment approach for contextualized embeddings without the use of parallel text.

## 3 Approach

### Orthogonal Bilingual Mapping

Given a dictionary of source to target pairs \(\langle x,y\rangle\) and matrix representations \(X\) and \(Y\) whose columns are vector representations of the corresponding dictionary items, we seek to find an orthogonal transformation matrix \(R\) that minimizes the distances between the transformed vectors in \(RX\) and \(Y\). Formally,

\[R=\operatorname*{arg\,min}_{\hat{R}}\lVert\hat{R}X-Y\rVert~{}~{}\text{s. t. }~{}\hat{ R}^{T}\hat{R}=I \tag{1}\]

where \(\lVert.\rVert\) denotes the Frobenius norm. The orthogonality constraint ensures that pair-wise distances in the original source vector space are preserved after the transformation. As shown in Schonemann (1966), the solution can be found by singular value decomposition of \(YX^{T}\)

\[YX^{T}=U\Sigma V^{T}\]

Then,

\[R=UV^{T} \tag{2}\]

The resultant transformation, \(R\), can then be used to transform additional vectors in the source vector space. The quality of the transformation depends on the size and accuracy of the initial dictionary, and it is typically evaluated on word translation precision using nearest neighbor search Smith et al. (2017).

### Mapping of Contextualized Embeddings

Word embeddings in a given language tend to have similar structures as their translations in a target language Aldarmaki et al. (2018), which enables orthogonal mappings of word vector spaces to generalize well across various languages. However, items in bilingual dictionaries typically refer to specific word senses. In a given dictionary pair, the source word may have multiple senses that are not consistent with its aligned target translation (and vise versa), which could result in suboptimal global mappings. Intuitively, better mappings could be obtained using sense-disambiguated word embeddings, which could be approximated from context. ELMo (Embeddings from Language Models) is a recently-proposed deep model for obtaining contextualized word embeddings, which are calculated as the hidden states of a bi-LSTM network trained as a language model Peters et al. (2018). The network can be used in lieu of static word embeddings within other models, which yields better performance in a range of tasks, including word sense disambiguation. Sentence embeddings can be obtained from ELMo by averaging the contextualized word embeddings Perone et al. (2018).

Since ELMo generates dynamic, context-dependent vectors, we cannot use a simple word-level dictionary to map the model across languages. Instead, we use a parallel corpus with word alignments, i.e using an IBM Model Brown et al. (1993), to extract a dynamic dictionary of aligned contextualized word embeddings. Depending on the size of the parallel corpus, a large dictionary can be extracted to learn an orthogonal mapping as described in Section 3.1, which is then applied post-hoc on newly generated contextualized embeddings.

### Sentence-Level Mapping

An alternative general approach for obtaining a context-aware mapping is to learn sentence-level transformations. Intuitively, a sentence is less ambiguous than stand-alone words since the words are interpreted within a specific context, so a mapping learned at the sentence-level is likely to be less sensitive to individual word inconsistencies. Therefore, we learn the mapping as described in 

Section 3.1 using a dictionary of aligned sentence embeddings. Over a large parallel corpus, the aggregate mapping can yield a more optimal global solution compared to word-level mapping. This approach can be applied using any model capable of generating monolingual sentence embeddings. In this work, we use the average of word vectors in each sentence, where the word vectors are either static or contextualized. For inference, monolingual sentence embeddings are generated first, then mapped to the target space using the sentence-level transformation matrix.1

Footnote 1: Since we use vector averaging, it doesn’t matter whether we apply the learned transformation to the word embeddings before averaging, or to the sentence embeddings after averaging.

## 4 Experiments

We used skip-gram with subword information, i.e FastText Bojanowski et al. (2017), for the static word embeddings, and ELMo for contextualized word embeddings. Sentence embeddings were calculated from ELMo as the arithmetic average of the contextualized embeddings 2. For FastText, we applied weighted averaging using smooth inverse frequency Arora et al. (2017), which works better for sentence similarity compared to other averaging schemes Aldarmaki and Diab (2018).

Footnote 2: We found that using the arithmetic average for ELMo yields better results than weighted averaging.

### Data and Processing

We trained and aligned all models using the same monolingual and parallel datasets. For monolingual training, we used the 1 Billion Word benchmark Chelba et al. (2014) for English, and equivalent subsets of \(\sim\)400 million tokens from WMT'13 Bojar et al. (2013) news crawl data. We trained monolingual ELMo and FastText with default parameters. We used the WMT'13 commoncrawl data for cross-lingual mapping, and the WMT'13 test sets for evaluating sentence translation retrieval. For all datasets, the only preprocessing we performed was tokenization.

### Evaluation Framework

We evaluated the cross-lingual mapping approaches on sentence translation retrieval, where we calculate the accuracy of retrieving the correct translation from the target side of a test parallel corpus using nearest neighbor search with cosine similarity. To assess the minimum bilingual data requirements of each approach and measure how the various models respond to additional data, we split the training parallel corpus into smaller subsets of increasing sizes, starting from 100 to a million sentences (we double the size at each step). Data splits and evaluation scripts are available at [https://github.com/h-aldarmaki/sent_translation_retrieval](https://github.com/h-aldarmaki/sent_translation_retrieval).

### Alignment Schemes

For ELMo, word embeddings need to be calculated from context, so we extracted a dictionary of contextualized words from the parallel corpora by first applying word-level alignments using Fast_Align Dyer et al. (2013). We then calculated the contextualized embeddings for source and target sentences, and extracted a dictionary from the aligned words that have a one-to-one alignment (i.e. we excluded phrasal alignments). Since this can result in a very large dictionary, we capped the number of dictionary words at 1M for efficiency. For a fair comparison with FastText word-level mapping, we extracted a dictionary from word alignment probabilities using the same parallel sets. For each word in the source language, we extracted its translation as the word with the maximum alignment probability if the maximum was unique3. As a baseline, we used static dictionaries from Conneau et al. (2017) to obtain word-level mappings (dict). All alignments were performed from the source languages to English.

Footnote 3: Using other dictionary pairs generally resulted in lower performance.

#### 4.3.1 Results

Sentence translation retrieval results in all language directions are shown in Figure 1 (note the x-axis denotes the size of the alignment corpus in log scale). The arrows indicate the translation direction from source to target, with \(en\) for English, \(es\) for Spanish, and \(de\) for German. For clarity, the legend shows the average accuracies in the final step (1M).

Overall, ELMo word alignment resulted in the highest sentence translation retrieval accuracies, even with small amounts of training data; it exceeded the static dictionary baseline at around 2K parallel sentences. Sentence-level mapping outperformed word-level mapping only when additional parallel data were used (over 50K sentences).

 

With 1M sentences, sentence-level mapping of FastText yielded an increase of \(\sim\)3% in all directions. Sentence-level ELMo underperformed in the \(\rightarrow{}en\) directions until we used 100K sentences, where we observed a sharp increase in accuracy compared to the previous step of 50K sentences. For ELMo, we note particular improvements in zero-shot translation retrieval between the source languages: \(es\) and \(de\), where ELMo-based models performed much higher than FastText. The opposite is true for the \(\rightarrow{}en\) directions, although the difference is not as notable. This is an interesting observation and may indicate that contextualized dictionaries result in a more balanced mapping, while context-independent embeddings overfit the mapping to the specific direction used for alignment.

#### 4.3.2 Word-level Evaluation

Cross-lingual word embeddings are typically evaluated in word-translation retrieval: the precision of correctly retrieving a translation from the vocabulary of another language. Since this is a context-free task, we evaluated the performance of static word embeddings, FastText, using word vs. sentence mapping (with 1M parallel sentences). The transformation matrix learned at the sentence level is used to transform the word embeddings.

We used the dictionaries from Conneau et al. (2017). We also evaluated on the SemEval'17 cross-lingual word similarity task Camacho-Collados et al. (2017), which is measured using the average of Pearson and Spearman correlation coefficients against human judgements. As shown in Tables 1 and 2, the mapping learned at

Figure 1: Nearest neighbor sentence translation accuracy as a function of (log) parallel corpus size. (word) refers to word-level mapping, (sent) to sentence-level mapping, and (dict) refers to the baseline (using a static dictionary for mapping). The legend shows the average accuracies of each model using 1M parallel sentences.

 

the sentence-level yields equivalent performance to word-level mapping. While word-level mapping was slightly better in translating from source languages (German and Spanish) to English, the sentence-level mapping was better when translating between the source languages. In the word similarity task, sentence-level mappings performed slightly better in two out of the three cases. Overall, the performance of both models are comparable, which indicates that a single transformation matrix learned at the sentence-level can be used for both word and sentence-level tasks.

## 5 Discussion and Conclusions

We introduced alternatives to the popular word mapping approach that incorporate context in the mapping process. Given parallel corpora, context-aware mappings were learned by mapping aligned contextualized word embeddings or directly mapping the parallel sentence embeddings. Experimental results showed significant gains in sentence translation retrieval using contextualized mappings compared to context-independent word mapping. While word-level mappings worked better with smaller parallel corpora, the performance of sentence-level mapping continued to increase with additional data until it outperformed word-level mapping. In future work, we will explore the viability of the sentence mapping approach on other sentence embedding models.

## References

* Aladarmaki and Diab (2018) Hanan Aldarmaki and Mona Diab. 2018. Evaluation of unsupervised compositional representations. _Proceedings of the 27th International Conference on Computational Linguistics_.
* Aldarmaki et al. (2018) Hanan Aldarmaki, Mahesh Mohan, and Mona Diab. 2018. Unsupervised word mapping using structural similarities in monolingual embeddings. _Transactions of the Association of Computational Linguistics_, 6.
* Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings.
* Artetxe et al. (2016) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Bojar et al. (2013) Ondrej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 workshop on statistical machine translation. In _Proceedings of the eighth workshop on statistical machine translation_, pages 1-44.
* Brown et al. (1993) Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. _Computational linguistics_, 19(2):263-311.
* Camacho-Collados et al. (2017) Jose Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. Semeval-2017 task 2: Multilingual and cross-lingual semantic word similarity. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 15-26.
* Chelba et al. (2014) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In _Fifteenth Annual Conference of the International Speech Communication Association_.
* Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2017. Word translation without parallel data. _arXiv preprint arXiv:1710.04087_.
* Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 644-648.
* Mikolov et al. (2013) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. _arXiv preprint arXiv:1309.4168_.
* Perone et al. (2018) Christian S Perone, Roberto Silveira, and Thomas S Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. _arXiv preprint arXiv:1806.06259_.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_.
* Schonemann (1966) Peter H Schonemann. 1966. A generalized solution of the orthogonal procrustes problem. _Psychometrika_, 31(1):1-10.
* Schuster et al. (2019) Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to 

zero-shot dependency parsing. _arXiv preprint arXiv:1902.09492_.
* Smith et al. (2017) Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. _arXiv preprint arXiv:1702.03859_.
* Upadhyay et al. (2016) Shyam Upadhyay, Manaal Faruqui, Chris Dyer, and Dan Roth. 2016. Cross-lingual models of word embeddings: An empirical comparison. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1.