Venue: NAACL 2021

# Improving the Lexical Ability of Pretrained Language Models

for Unsupervised Neural Machine Translation

Alexandra Chronopoulou, Dario Stojanovski, Alexander Fraser

###### Abstract

Successful methods for unsupervised neural machine translation (unmt) employ cross-lingual pretraining via self-supervision, often in the form of a masked language modeling or a sequence generation task, which requires the model to align the lexical- and high-level representations of the two languages. While cross-lingual pretraining works for similar languages with abundant corpora, it performs poorly in low-resource and distant languages. Previous research has shown that this is because the representations are not sufficiently aligned. In this paper, we enhance the bilingual masked language model pretraining with lexical-level information by using type-level cross-lingual subword embeddings. Empirical results demonstrate improved performance both on unmt (up to \(4.5\) BLEU) and bilingual lexicon induction using our method compared to a unmt baseline.

Center for Information and Language Processing, LMU Munich, Germany

{achron, stojanovski, fraser}@cis.lmu.de

## 1 Introduction

unmt is an effective approach for translation without parallel data. Early approaches transfer information from static pretrained cross-lingual embeddings to the encoder-decoder model to provide an implicit bilingual signal [30, 29]. Lample:2018:acl

2016b). We build _subword_ monolingual embeddings with _fastText_(Bojanowski et al., 2017). Then, we map the monolingual embeddings of the two languages to a shared space, using _VecMap_(Artetxe et al., 2018a), with identical tokens occurring in both languages serving as the initial seed dictionary, as we do not have any bilingual signal. This is different from the original _VecMap_ approach, which operates at the _word level_. We use the mapped embeddings of the two languages to initialize the embedding layer of a Transformer-based encoder (Vaswani et al., 2017).

### Masked Language Model Training

We initialize the token embedding layer of the mlm Transformer encoder with pretrained _VecMap_embeddings, which provide an informative mapping, i.e., cross-lingual lexical representations. We train the model on data from both languages, using masked language modeling. Training a masked language model enhances the cross-lingual signal by encoding contextual representations. This step is illustrated in Figure 1.

### Unsupervised nmt

Finally, we transfer the mlm-trained encoder Transformer to an encoder-decoder translation model. We note that the encoder-decoder attention of the Transformer is randomly initialized. We then train the model for nmt in an unsupervised way, using denoising auto-encoding (Vincent et al., 2008) and back-translation (Sennrich et al., 2016a), which is performed in an online manner. This follows work by Artetxe et al. (2018b); Lample et al. (2018a, c).

## 3 Experiments

**Datasets.** We conduct experiments on English-Macedonian (En-Mk) and English-Albanian (En-sq), as Mk, Sq are low-resource languages, where lexical-level alignment can be most beneficial. We use 3K randomly sampled sentences of SETIMES (Tiedemann, 2012) as validation/test sets. We also use 6SM En sentences from NewsCrawl. For Sq and MK we use all the CommonCrawl corpora from Ortiz Suarez et al. (2019), which are 4M Sq and \(2.4\)M Mk sentences.

**Baseline.** We use a method that relies on cross-lingual language model pretraining, namely xlm(Lample and Conneau, 2019). This approach trains a bilingual mlm separately for En-Mk and En-sq, which is used to initialize the encoder-decoder of the corresponding nmt system. Each system is then trained in an unsupervised way.

**Comparison to state-of-the-art.** We apply our proposed approach to re-lm(Chronopoulou et al., 2020), a state-of-the-art approach for low-resource unmt. This method trains a monolingual En mlm model (_monolingual pretraining step_). Upon convergence, a vocabulary extension method is used, that randomly initializes the newly added vocabulary items. Then, the mlm is fine-tuned to the two languages (_MLM fine-tuning step_) and used to initialize an encoder-decoder model. This method outperforms xlm on low-resource scenarios.

**Lexically aligned language models.** When applied to the baseline, our method initializes the embedding layer of xlm with unsupervised cross-lingual embeddings. Then, we train xlm on the two languages of interest with a masked language modeling objective. Upon convergence, we transfer it to the encoder and decoder of an nmt model, which is trained in an unsupervised way.

In the case of re-lm, our method is applied to the _MLM fine-tuning step_. Instead of randomly initializing the new embedding vectors added in this step, we use pretrained unsupervised cross-lingual embeddings. We obtain them by applying _VecMap_ to _fastText_ pretrained Albanian/Macedonian embeddings and the English mlm token-level embeddings. Then, the mlm is fine-tuned on both languages. Finally, it is used to initialize an encoder-decodernmt model.

Figure 1: Lexically aligned cross-lingual masked language model.

 



**Unsupervised _VecMap_ bilingual embeddings.** We build monolingual embeddings with the _fastText_ skip-gram model with 1024 dimensions, using our BPE-split Sennrich et al. (2016) monolingual corpora. We map them to a shared space, using _VecMap_ with identical tokens. We concatenate the aligned embeddings of the two languages and use them to initialize the embedding layer of xlm, or the new vocabulary items of re-lm.

**Preprocessing.** We tokenize the monolingual data and validation/test sets using Moses Koehn et al. (2006). For xlm Lample and Conneau (2019), we use BPE splitting with 32K operations jointly learned on both languages. For re-lm Chronopoulou et al. (2020), we learn 32K BPEs on En for pretraining, and then 32K BPEs on both languages for the fine-tuning and unmt steps. The BPE merges are learned on a subset of the En corpus and the full Sq or Mk corpus.

**Model hyperparameters.** We use a Transformer architecture for both the baselines and unmt models, using the same hyperparameters as xlm. For the encoder Transformer used for masked language modeling, the embedding and model size is 1024 and the number of attention heads is 8. The encoder Transformer has 6 layers, while the nmt model is a 6-layer encoder/decoder Transformer. The learning rate is set to \(10^{-4}\) for xlm and unmt. We train the models on 8 NVIDIA GTX 11 GB GPUs. To be comparable with re-lm, we retrain it on 8 GPUs, as that work reports umtt results with only 1 GPU. The per-GPU batch size is 32 during xlm and 26 during unmt. Our models are built on the publicly available xlm and re-lm codebases. We generate final translations with beam search of size 5 and we evaluate with SacreBLEU2 Post (2018).

Footnote 2: Signature “BLEU+c.mixed+#.1+s.exp+tok.13a+v.1.4.9”

## 4 Results

Table 1 shows the results of our approach compared to two pretraining approaches that rely on mlm training, namely xlm and re-lm. The lexically aligned xlm improves translation results over the baseline xlm model. We obtain substantial improvements on En-Sq in both directions, of at most \(2.2\) BLEU and \(1.4\) chrF1, while on En-Mk, we get an even larger performance boost of up to \(4.5\) points in terms of BLEU and \(1.4\) in terms of chrF1. Our lexically aligned re-lm also consistently outperforms re-lm, most notably in the En\(\rightarrow\)Mk direction, by up to \(1.7\) BLEU. At the same time, chrF1 score improves by up to \(1.8\) points using the lexically aligned pretraining approach compared to re-lm.

In the case of xlm, the effect of cross-lingual lexical alignment is more evident for En-Mk, as Mk is less similar to En, compared to Sq. This is mainly the case because the two languages use a different alphabet Latin for En and Cyrillic for Mk). This is also true for re-lm when translating out of En, showing that enhancing the _fine-tuning step_ of mlm with pretrained embeddings is helpful and improves the final unmt performance.

In general, our method provides better alignment of the lexical-level representations of the mlm, thanks to the transferred _VecMap_ embeddings. We hypothesize that static cross-lingual embeddings enhance the knowledge that a cross-lingual masked language model obtains during training. As a result, using them to bootstrap the pretraining procedure improves the ability of the model to map the distributions of the two languages and yields higher translation scores. Overall, our approach consistently outperforms two pretraining models for unmt, providing for the highest BLEU and chrF1 scores on all translation directions.

## 5 Analysis

We conduct an analysis to assess the contribution of lexical-level alignment in the mlm training. We present Bilingual Lexicon Induction (BLI) and

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Mk\(\to\)En**} & \multicolumn{2}{c}{**En\(\rightarrow\)Mk**} & \multicolumn{2}{c}{**SQ\(\rightarrow\)En**} & \multicolumn{2}{c}{**En\(\rightarrow\)Sq**} \\  & BLEU \(\uparrow\) & chrF1 \(\uparrow\) & BLEU \(\downarrow\) & chrF1 \(\uparrow\) & BLEU \(\uparrow\) & chrF1 \(\uparrow\) & BLEU \(\uparrow\) & chrF1 \(\uparrow\) \\ \hline \hline xlm & 20.7 & 48.5 & 19.8 & 42.4 & 31.1 & 56.8 & 31.3 & 56.2 \\ _lexically aligned XLM_ & 25.2 & 49.9 & 22.9 & 43.1 & 32.8 & 58.2 & 33.5 & 56.8 \\ \hline re-lm & 25.0 & 51.1 & 23.9 & 45.8 & 30.1 & 55.8 & 32.2 & 56.4 \\ _lexically aligned RE-LM_ & 25.3 & 51.5 & 25.6 & 47.6 & 30.5 & 56.0 & 32.9 & 56.7 \\ \hline \hline \end{tabular}
\end{table}
Table 1: unmt results for translations to and from English. The first column indicates the pretraining method used. The scores presented are significantly different (p ¡ 0.05) from the respective baseline. chrF1 refers to character n-gram F1 score Popovic (2015). The models in italics are ours.



BLEU 1-gram precision scores. We also investigate the best method to leverage pretrained cross-lingual embeddings during mlm training, in terms of final unmt performance.

**Bilingual Lexicon Induction (BLI).** We use BLI, a standard way of evaluating lexical quality of embedding representations Gouws et al. (2015); Ruder et al. (2019), to explore the effect of the alignment of our method. We compare the BLI score of different cross-lingual pretrained language models. We report precision@5 (P@5) using nearest neighbors (NN) and cross-lingual semantic similarity (CSLS). The results are presented in Table 2. We use the embedding layer of each mlm for this task. We also experimented with averages over different layers, but noticed the same trend in terms of BLI scores. We obtain word-level representations by averaging over the corresponding subword embeddings. It is worth noting that we compute the type-level representation of each vocabulary word in isolation, similar to Vulic et al. (2020).

In Table 2, we observe that lexical alignment is more beneficial for En-Mk. This can be explained by the limited vocabulary overlap of the two languages, which does not provide sufficient cross-lingual signal for the training of mlm. By contrast, initializing an mlm with pretrained embeddings largely improves performance, even for a higher-performing model, such as re-lm. In En-Sq, the effect of our approach is smaller yet consistent. This can be attributed to the fact that the two languages use the same script.

Overall, our method enhances the lexical-level information captured by pretrained mlms, as shown empirically. This is consistent with our intuition that cross-lingual embeddings capture a bilingual signal that can benefit mlm representations.

**1-gram precision scores.** To examine whether the improved translation performance is a result of the lexical-level information provided by static embeddings, we present 1-gram precision scores in Table 3, as they can be directly attributed to lexical alignment. The biggest performance gains (up to +\(10.4\)) are obtained when the proposed approach is applied to xlm. This correlates with the BLEU scores of Table 1. Moreover, the En-Mk language pair benefits more than En-Sq from the lexical-level alignment both in terms of 1-gram precision and BLEU. These results show that the improved BLEU scores can be attributed to the enhanced lexical representations.

**How should static embeddings be integrated in the mlm training?** We explore different ways of incorporating the lexical knowledge of pretrained cross-lingual embeddings to the second, masked language modeling stage of our approach (SS2.2). Specifically, we keep the aligned embeddings fixed (_frozen_) during slm training and compare the performance of the final unmt model to the proposed (_fine-tuned_) method. We point out that, after we transfer the trained mlm to an encoder-decoder model, all layers are trained for unmt.

Table 4 summarizes our results. The fine-tuning approach, which is adopted in our proposed method, provides a higher performance both in En-Mk and En-Sq, with the improvement being more evident in En-Sq. Our findings generally show that it is preferable to train the bilingual embeddings together with the rest of the model in the mlm step.

## 6 Related Work

Artetxe et al. (2018c); Lample et al. (2018a) initialize unmt models with word-by-word transla

\begin{table}
\begin{tabular}{l c c c c} \hline Alignment Method & \multicolumn{2}{c}{**En-Mk**} & \multicolumn{2}{c}{**En-Sq**} \\  & \(\leftarrow\quad\rightarrow\quad\leftarrow\quad\rightarrow\) \\ \hline \hline lexically aligned mlm & & & & \\ frozen embeddings & 24.7 & 22.1 & 31.0 & 32.1 \\ fine-tuned embeddings (ours) & **25.2** & **22.9** & **32.8** & **33.5** \\ \hline \end{tabular}
\end{table}
Table 4: BLEU scores using different initializations of the xlm embedding layer. xlm is then trained on the respective language pair and used to initialize a unmt system. Both embeddings are aligned using _VecMap_.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**En-Mk**} & \multicolumn{2}{c}{**En-Sq**} \\  & NN & CSLS & NN & CSLS \\ \hline \hline xlm & 6.3 & 6.5 & 43.0 & 40.7 \\ lexically aligned xmlm & **15.5** & **16.5** & **51.6** & **50.6** \\ \hline re-lm & 29.8 & 16.1 & 52.0 & 35.9 \\ lexically aligned re-lm & **32.0** & **17.2** & **53.0** & **36.9** \\ \hline \hline \end{tabular}
\end{table}
Table 2: P@5 results for the BLI task on the MUSE Lample et al. (2018b) dictionaries. We evaluate the alignment of the embedding layer of each trained mlm.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**En-Mk**} & \multicolumn{2}{c}{**En-Sq**} \\  & \(\leftarrow\quad\rightarrow\quad\leftarrow\quad\rightarrow\) \\ \hline \hline xlm & 53.1 & 41.4 & 62.1 & 60.4 \\ lexically aligned xmlm & **56.0** & **51.8** & **63.6** & **61.5** \\ \hline re-lm & 56.0 & 52.8 & 61.6 & 61.2 \\ lexically aligned re-lm & **56.6** & **53.9** & **62.0** & **61.7** \\ \hline \hline \end{tabular}
\end{table}
Table 3: BLEU 1-gram precision scores.

 

tions, based on a bilingual lexicon inducted in an unsupervised way by the same monolingual data, or simply with cross-lingual embeddings. Lample et al. (2018) also use pretrained embeddings, learned on joint monolingual corpora of the two languages of interest, to initialize the embedding layer of the encoder-decoder. Lample and Conneau (2019) remove pretrained embeddings from the unmt pipeline and align language distributions by simply pretraining a mlm on both languages, in order to learn a cross-lingual mapping. However, it has been shown that this pretraining method provides a weak alignment of the language distributions Ren et al. (2019). While that work identified as a cause the lack of sharing n-gram level cross-lingual information, we address the lack of cross-lingual information at the lexical level.

Moreover, most prior work on umnt focuses on languages with abundant, high-quality monolingual corpora. In low-resource scenarios though, especially when the languages are not related, pretraining a cross-lingual mlm for unsupervised nmt does not yield good results Guzman et al. (2019); Chronopoulou et al. (2020). We propose a method that overcomes this issue by enhancing the mlm with cross-lingual lexical-level representations.

Another line of work tries to enrich the representations of multilingual mlms with additional knowledge Wang et al. (2020); Pfeiffer et al. (2020) without harming the already-learned representations. In our work, we identify lexical information as a source of knowledge that is missing from mlms, especially when it comes to low-resource languages. Surprisingly, static embeddings, such as _fastText_, largely outperform representations extracted by multilingual mlms in terms of cross-lingual lexical alignment Vulic et al. (2020). Motivated by this, we aim to narrow the gap between the lexical representations of bilingual mlms and static embeddings, in order to achieve a higher translation quality, when transferring the mlm to an encoder-decoder umtt model.

## 7 Conclusion

We propose a method to improve the lexical ability of a Transformer encoder by initializing its embedding layer with pretrained cross-lingual embeddings. The Transformer is trained for masked language modeling on the language pair of interest. After that, it is used to initialize an encoder/decoder model, which is trained for umtt and outperforms relevant baselines. Results confirm our intuition that masked language modeling, which provides contextual representations, benefits from cross-lingual embeddings, which capture lexical-level information. In the future, we would like to investigate whether lexical knowledge can be infused to multilingual mlms. We would also like to experiment with other schemes of training the mlm in terms of how the embedding layer is updated, such as regularizer annealing strategies, which would enable keeping the embeddings relatively fixed, but still allow for some limited training.

## 8 Ethical Considerations

In this work, we propose a novel unsupervised neural machine translation approach, which is tailored to low-resource languages in terms of monolingual data. We experiment with unsupervised translation between English, Albanian and Macedonian.

For English, we use high-quality data from news articles. The Albanian and Macedonian monolingual data originates from the OSCAR project Ortiz Suarez et al. (2019). The corpora are shuffled and stripped of all metadata. Therefore, the data should not be easily attributable to specific individuals. Nevertheless, the project offers easy ways to remove data upon request. The En-Sq and EnMk parallel development and test data are obtained from OPUS Tiedemann (2012) and consist of high-quality news articles.

Our work is partly based on training type-level embeddings which are not computationally expensive. However, training cross-lingual masked language models requires significant computational resources. To lower environmental impact, we do not conduct hyper-parameter search and use well-established values for all hyper-parameters.

This project has received funding from the European Research Council under the European Union's Horizon \(2020\) research and innovation program (grant agreement #640550). This work was also supported by DFG (grant FR \(2829\)/4-1). We thank Katerina Margatina, Giorgos Vernikos and Viktor Hangya for their thoughtful comments/suggestions and valuable feedback 



## References

* Artetxe et al. (2018a) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 789-798, Melbourne, Australia. Association for Computational Linguistics.
* Artetxe et al. (2018b) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. Unsupervised statistical machine translation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 3632-3642.
* Artetxe et al. (2019) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_, pages 194-203. Association for Computational Linguistics.
* Artetxe et al. (2018c) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018c. Unsupervised neural machine translation. In _International Conference on Learning Representations_.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Chronopoulou et al. (2020) Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2020. Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2703-2711, Online. Association for Computational Linguistics.
* Gouws et al. (2015) Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. Bilbowa: Fast bilingual distributed representations without word alignments. In _Proceedings of the 32nd International Conference on Machine Learning_, volume 37 of _Proceedings of Machine Learning Research_, pages 748-756, Lille, France. PMLR.
* Guzman et al. (2019) Francisco Guzman, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing_, pages 6100-6113.
* Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. _ArXiv_.
* Koehn et al. (2006) Philipp Koehn, Marcello Federico, Wade Shen, Nicola Bertoldi, Ondrej Bojar, Chris Callison-Burch, Brooke Cowan, Chris Dyer, Hieu Hoang, Richard Zens, et al. 2006. Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding. In _Final Report of the 2006 JHU Summer Workshop_.
* Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In _Advances in Neural Information Processing Systems_, page 7057-7067.
* Lample et al. (2018a) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In _International Conference on Learning Representations_.
* Lample et al. (2018b) Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2018b. Word translation without parallel data. In _International Conference on Learning Representations_.
* Lample et al. (2018c) Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018c. Phrase-based & neural unsupervised machine translation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 5039-5049.
* Marchisio et al. (2020) Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When does unsupervised machine translation work? In _Proceedings of the Fifth Conference on Machine Translation_, pages 571-583, Online. Association for Computational Linguistics.
* Quarez et al. (2019) Pedro Javier Ortiz Suarez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In _Workshop on the Challenges in the Management of Large Corpora_.
* Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7654-7673, Online. Association for Computational Linguistics.
* Popovic (2015) Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Conference on Machine Translation: Research Papers_, pages 186-191.
* Press and Wolf (2017) Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In _Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics_, pages 157-163

* Ren et al. (2019) Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, and Shuai Ma. 2019. Explicit cross-lingual pre-training for unsupervised machine translation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 770-779. Association for Computational Linguistics.
* Ruder et al. (2019) Sebastian Ruder, Ivan Vulic, and Anders Sogaard. 2019. A survey of cross-lingual word embedding models. _J. Artif. Int. Res._, 65(1):569-630.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_, pages 86-96.
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_, pages 1715-1725.
* Tiedemann (2012) Jorg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC)_, pages 2214-2218.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, page 5998-6008.
* Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In _Proceedings of the International Conference on Machine Learning_, pages 1096-1103.
* Vulic et al. (2020) Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavas, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7222-7240, Online. Association for Computational Linguistics.
* Wang et al. (2020) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xu-anjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters.

 

Appendix

### Datasets

We remove sentences longer than \(100\) words after BPE splitting. We split the data using the fastBPE codebase3.

Footnote 3: [https://github.com/glample/fastBPE](https://github.com/glample/fastBPE)

### Model Configuration

We tie the embedding and output (projection) layers of both lm and nmt models Press and Wolf (2017). We use a dropout rate of \(0.1\) and GELU activations Hendrycks and Gimpel (2017). We use the default parameters of Lample and Conneau (2019) in order to train our models.

Regarding the runtimes for the En-Sq experiments: the baseline xlm was trained for 3 days on 8 GPUs while our approach for 6 days and 14h. The experiment with freezing the embeddings provided for faster training, 2 days and 17h. The three methods needed 23h, 21h, and 1d and 8h for the unmt part, respectively. Fine-tuning with re-lm took 2 days and 14h on 1 GPU and with our approach it took 1 day and 5h. unmt for these models took 2 days and 11h, and 13h, respectively. We get a checkpoint every \(50\)K sentences processed by the model.

### Validation Scores of Results

In Table 5 we show the dev scores of the main results, in terms of BLEU scores. This table extends Table 1 of the main paper.

In Table 6, we show the dev scores of the extra fine-tuning experiments we did for the analysis. The table corresponds to Table 4 of the main paper.

We note that the dev scores are obtained using greedy decoding, while the test scores are obtained with beam search of size \(5\). We clarify that we train each nmt model using as training criterion the validation BLEU score of the Sq, Mk\(\rightarrow\)En direction, with a patience of \(10\). We specifically use the multi-bleu.perl script from Moses.

### Joint vs VecMap embeddings.

Using joint embeddings to initialize the mlm, before training it on data from the respective language is less effective for unmt. This is mostly the case for En-Mk, since the two languages use a different alphabet Latin and Cyrillic). In this case, simply learning _fastText_ embeddings on the concatenation of the two corpora is not useful, because the languages do not have a big lexical overlap.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline Alignment Method & \multicolumn{2}{c}{**En-Mk**} & \multicolumn{2}{c}{**En-Sq**} \\  & \(\leftarrow\quad\to\quad\) & \(\leftarrow\quad\to\quad\) \\ \hline \hline lextically aligned mlm frozen embeddings fine-tuned embeddings (ours) & **24.8** & 23.0 & 31.0 & 32.1 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Development BLEU scores using different initializations of the xlm embedding layer.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline Alignment Method & \multicolumn{2}{c}{**En-Mk**} & \multicolumn{2}{c}{**En-Sq**} \\  & \(\leftarrow\quad\to\quad\) & \(\leftarrow\quad\to\quad\) \\ \hline \hline joint _fastText_ & 21.5 & 19.8 & 32.3 & 33.1 \\ VecMap & **25.2** & **22.9** & **32.8** & **33.5** \\ \hline \hline \end{tabular}
\end{table}
Table 7: BLEU scores using different initializations of the xlm embedding layer. xlm is then trained on the respective language pair and used to initialize a unmt system. _JointfastText embs_ refers to jointly learned embeddings following Lample et al. (2018).

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline  & \multicolumn{2}{c}{**En-Mk**} & \multicolumn{2}{c}{**En-Sq**} \\  & \(\leftarrow\quad\to\quad\) & \(\leftarrow\quad\to\quad\) \\ \hline \hline xlm & - & - & 30.7 & 32.0 \\ lexically aligned xlm & **24.6** & **23.3** & **31.9** & **33.8** \\ \hline re-lm & 25.0 & 25.7 & 29.9 & 32.8 \\ lexically aligned re-lm & **25.3** & **26.6** & **29.5** & **30.3** \\ \hline \hline \end{tabular}
\end{table}
Table 5: unmt BLEU scores on the development set.