SlowGuess/NAACL_2021_OCR huggingface dataset:
Brooke Cowan, Chris Dyer, Hieu Hoang, Richard Zens, et al. 2006. Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding. In Final Report of the 2006 JHU Summer Workshop.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. In Advances in Neural Information Processing Systems, page 7057-7067.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations.
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In International Conference on Learning Representations.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018c. Phrase-based & neural unsupervised machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5039-5049.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When does unsupervised machine translation work? In Proceedings of the Fifth Conference on Machine Translation, pages 571-583, Online. Association for Computational Linguistics.
Pedro Javier Ortiz Suárez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In Workshop on the Challenges in the Management of Large Corpora.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Conference on Machine Translation: Research Papers, pages 186-191.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pages 157-163.
Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, and Shuai Ma. 2019. Explicit cross-lingual pre-training for unsupervised machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 770-779. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. J. Artif. Int. Res., 65(1):569-630.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 86-96.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1715-1725.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC), pages 2214-2218.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, page 5998-6008.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the International Conference on Machine Learning, pages 1096-1103.
Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222-7240, Online. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters.
We remove sentences longer than 100 words after BPE splitting. We split the data using the fastBPE codebase<sup>3</sup>.
A.2 Model Configuration
We tie the embedding and output (projection) layers of both LM and NMT models (Press and Wolf, 2017). We use a dropout rate of 0.1 and GELU activations (Hendrycks and Gimpel, 2017). We use the default parameters of Lample and Conneau (2019) in order to train our models.
Regarding the runtimes for the En-Sq experiments: the baseline XLM was trained for 3 days on 8 GPUs while our approach for 6 days and 14h. The experiment with freezing the embeddings provided for faster training, 2 days and 17h. The three methods needed 23h, 21h, and 1d and 8h for the UNMT part, respectively. Fine-tuning with RE-LM took 2 days and 14h on 1 GPU and with our approach it took 1 day and 5h. UNMT for these models took 2 days and 11h, and 13h, respectively. We get a checkpoint every 50K sentences processed by the model.
A.3 Validation Scores of Results
In Table 5 we show the dev scores of the main results, in terms of BLEU scores. This table extends Table 1 of the main paper.
In Table 6, we show the dev scores of the extra fine-tuning experiments we did for the analysis. The table corresponds to Table 4 of the main paper.
En-Mk	En-Sq
←	→	←	→
XLM	-	-	30.7	32.0
lexically aligned XLM	24.6	23.3	31.9	33.8
RE-LM	25.0	25.7	29.9	32.8
lexically aligned RE-LM	25.3	26.6	29.5	30.3
We note that the dev scores are obtained using greedy decoding, while the test scores are obtained with beam search of size 5. We clarify that we train each NMT model using as training criterion the validation BLEU score of the Sq,  $\mathrm{Mk}\rightarrow \mathrm{En}$  direction, with a patience of 10. We specifically use the multi-bleu.perl script from Moses.
Table 5: UNMT BLEU scores on the development set.
Alignment Method	En-Mk	En-Sq
←	→	←	→
lexically aligned MLM
frozen embeddings	24.8	23.0	31.0	32.1
fine-tuned embeddings (ours)	24.6	23.3	31.1	32.0
A.4 Joint vs VecMap embeddings.
Using joint embeddings to initialize the MLM, before training it on data from the respective language is less effective for UNMT. This is mostly the case for En-Mk, since the two languages use a different alphabet (Latin and Cyrillic). In this case, simply learning fastText embeddings on the concatenation of the two corpora is not useful, because the languages do not have a big lexical overlap.
Table 6: Development BLEU scores using different initializations of the XLM embedding layer.
Alignment Method	En-Mk	En-Sq
← →	← →
joint fastText	21.5	19.8	32.3	33.1
VECMAP	25.2	22.9	32.8	33.5
Table 7: BLEU scores using different initializations of the XLM embedding layer. XLM is then trained on the respective language pair and used to initialize a UNMT system. Joint fastText embs refers to jointly learned embeddings following Lample et al. (2018c).
<sup>3</sup>https://github.com/glample/fastBPE