SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 3: BLEU scores of various supervised and unsupervised models on the FLoRes devtest sets. The bolded numbers are the best unsupervised scores and the underlined numbers represent the best supervised scores. For any  $\mathrm{X}\leftrightarrow \mathrm{Y}$  language pair, the  $\mathrm{X}\rightarrow \mathrm{Y}$  translation results are listed under each Y column, and vice-versa.
The parallel data we use came from a variety of sources, all available through WMT. We drew our English-Hindi parallel data from IITB (Kunchukuttan et al., 2017); English-Russian, English-Arabic, and English-Chinese parallel data from the UN Corpus (Ziemski et al., 2016); English-Tamil and English-Telugu from Wikimatrix (Schwenk et al., 2019). We used the scripts from Moses (Koehn, 2009) to normalize punctuation, remove non-printing characters, and replace the unicode characters with their non-unicode equivalent. We additionally use the normalizing script from Indic NLP (Kunchukuttan, 2020) for Gujarati, Nepali, Telugu, and Sinhala.
We concatenate two million lines of monolingual data for each language and use it to build a vocabulary with SentencePiece $^{8}$  (Kudo and Richardson, 2018) of 64,000 pieces. We then separate our data into SentencePiece pieces and remove all training samples that are over 88 pieces long.
6.2 Model architecture
All of our models were coded and tested in Tensorflow (Abadi et al., 2016). We use the Trans
former architecture (Vaswani et al., 2017) as the basis of our translation models. We use 6-layer encoder and decoder architecture with a hidden size of 1024 and an 8192 feedforward filter size. We share the same encoder for all languages. To differentiate between the different possible output languages, we add (learned) language embeddings to each token's embedding before passing them to the decoder. We follow the same modification as done in Song et al. (2019) and modify the output transformation of each attention head in each transformer block in the decoder to be distinct for each language. Besides these modifications, we share decoder parameters for every language.
6.3 Training parameters
We use three different settings, corresponding to each stage of training. For the first stage, we use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.0002, weight decay of 0.2 and batch size of 2048 examples. We use a learning rate schedule consisting of a linear warmup of 4000 steps to a value 0.0002 followed by a linear decay for 1.2 million steps. At every step, we choose a single dataset from which to draw a whole batch using the following process: with equal probability, choose either monolingual or parallel. If the choice
8We build the SentencePiece model with the following settings: vocab_size=64000, model_type=bpe, user_definedsymbols=[MASK], character_coverage=1.0, split_by Whitespace=true.
is monolingual, then we select one of the monolingual datasets uniformly at random. If the choice is parallel, we use a temperature-based sampling scheme based on the numbers of samples with a temperature of 5 (Arivazhagan et al., 2019b). In the second stage, we retain the same settings for both rounds of leveraging synthetic data except for the learning rate and number of steps. In the first round, we use the same number of steps, while in the second round we only use 240 thousand steps, a 1/5th of the original.
For the final phase, we bucket sequences by their sequence length and group them up into batches of at most 2000 tokens. We train the model with 8 NVIDIA V100 GPUs, assigning a batch to each one of them and training synchronously. We also use the Adamax optimizer instead, and cut the learning rate by four once more.
6.4Baselines
We compare with the state-of-the-art unsupervised and supervised baselines from the literature. Note all the baselines build language-specific models, whereas we have a single model for all the target unsupervised directions.
Unsupervised baselines: For the bilingual unsupervised baselines, we include the results of Kim et al. (2020) for  $\mathrm{En} \leftrightarrow \mathrm{Gu}$  and  $\mathrm{En} \leftrightarrow \mathrm{Kk}$  and of Guzmán et al. (2019) for  $\mathrm{En} \leftrightarrow \mathrm{Si}$ . We also report other multilingual unsupervised baselines. mBART (Liu et al., 2020) leverages auxiliary parallel data (e.g.  $\mathrm{En} \leftrightarrow \mathrm{Hi}$  parallel data for  $\mathrm{Gu} \rightarrow \mathrm{En}$ ) after pre-training on a large dataset consisting of 25 languages and the FLoRes dataset benchmark (Guzmán et al., 2019) leverages  $\mathrm{Hi} \leftrightarrow \mathrm{En}$  data for the  $\mathrm{En} \leftrightarrow \mathrm{Ne}$  language pair. All the unsupervised baselines that use auxiliary parallel data perform considerably better than the ones that don't.
Supervised baselines: In addition to the unsupervised numbers above, mBART and the FLORes dataset benchmarks report supervised results that we compare with. We additionally include one more baseline where we followed the training scheme proposed in stage 1, but also included the missing parallel data. We labeled this model "Mult. MT Baseline", though we emphasize that we also leverage the monolingual data in this baseline, as in recent work (Siddhant et al., 2020a; Garcia et al., 2020).
6.5 Evaluation
We evaluate the performance of our models using BLEU scores (Papineni et al., 2002). BLEU scores are known to be dependent on the data preprocessing (Post, 2018) and thus proper care is required to ensure the scores between our models and the baselines are comparable. We thus only considered baselines which report detokenized BLEU scores with sacreBLEU (Post, 2018) or report explicit pre-processing steps. In the case of the Indic languages (Gujarati, Nepali, and Sinhala), both the baselines we consider (Guzmán et al., 2019; Liu et al., 2020) report tokenized BLEU using the tokenizer provided by the Indic-NLP library (Kunchukuttan, 2020). For these languages, we follow this convention as well so that the BLEU scores
<sup>9</sup>Due to the limited literature on unsupervised machine translation on low-resource languages, this was the best bilingual unsupervised system we could find.
Data configuration	newsdev2019
Monolingual	Parallel	Kk↔En
Ru	Ru	6.8	9.5
Ru, Ar, Zh	Ru	7.3	14.8
Ru	Ru, Ar, Zh	9.6	18.4
Ru, Ar, Zh	Ru, Ar, Zh	9.8	18.6
Table 4: BLEU scores for a model trained with various configurations for the auxiliary data.
remain comparable. Otherwise, we follow suit with the rest of the literature and report detokenized BLEU scores through sacreBLEU<sup>10</sup>.
6.6 Results & discussion
We list the results of our experiments for the WMT datasets in Table 2 and for the FLoRes datasets in Table 3. After the first stage of training, we obtain competitive BLEU scores for  $\mathrm{X} \rightarrow \mathrm{En}$  translation directions, outperforming all unsupervised models as well as mBART for the language pairs  $\mathrm{Kk} \rightarrow \mathrm{En}$  and  $\mathrm{Gu} \rightarrow \mathrm{En}$ . Upon completion of the second stage of training, we see that the  $\mathrm{En} \rightarrow \mathrm{X}$  language pairs observe large gains, while the  $\mathrm{X} \rightarrow \mathrm{En}$  directions also improve. The final round of training further improves results in some language pairs, yielding an increase of +0.44 BLEU on average.
Note that in addition to considerably outperforming all the unsupervised baselines, our approach outperforms the supervised baselines on many of the language pairs, even matching the state-of-the-art on  $\mathsf{Ne} \rightarrow \mathsf{En}$ . Specifically, it outperforms the supervised mBART on six out of ten translation directions despite being a smaller model and Guzmán et al. (2019) on all pairs. Critically, we outperform our own multilingual MT baseline, trained in the same fashion and data as Stage 1, which further reinforces our assertion that unsupervised MT can provide competitive results with supervised MT in low-resource settings.
7 Further analysis
Given the substantial quality gains delivered by our proposed method, we set out to investigate what design choices can improve the performance of unsupervised models. To ease the computational burden, we further filter the training data to remove any sample which are longer than 64 SentencePiece $^{11}$  pieces long and cut the batch size in
10 BLEU + case.mixed + numrefs.1 + smooth.exp + tok.13a + version.1.4.14
<sup>11</sup>For all the experiments in this section, we use the same SentencePiece vocabulary as our benchmark model.
half for the first two stages. Additionally, we only do one additional round of training with synthetic data as opposed to the two rounds performed for the benchmark models. While these choices negatively impact performance, the resulting models still provide competitive results with our baselines and hence are more than sufficient for the purposes of experimental studies.
7.1 Increasing multilinguality of the auxiliary parallel data improves performance
It was shown in Garcia et al. (2020); Bai et al. (2020) that adding more multilingual data improved performance, and that the inclusion of auxiliary parallel data further improved the BLEU scores (Siddhant et al., 2020b). In this experiment, we examine whether further increasing multilinguality under a fixed data budget improves performance. For all configurations in this subsection, we utilize all the available English and Kazakh monolingual data. We fix the amount of auxiliary monolingual data to 40 million, the auxiliary parallel data to 12 million, and vary the number of languages which manifest in this auxiliary data.
We report the results on Table 4. It is observed that increasing the multilinguality of the parallel data is crucial, but the matter is less clear for the monolingual data. Using more languages for the monolingual data can potentially harm performance, but in the presence of multiple auxiliary language pairs with supervised data this degradation vanishes.
7.2 Synthetic data is critical for both stage 2 and stage 3 of training
In the following experiments, we evaluate the role of synthetic parallel data in the improved performance found at the end of stage 2 and stage 3 of our training procedure. We first evaluate whether the improved performance at the end of stage 2 comes from the synthetic data or the continued training. We consider the alternative where we repeat the same training steps as in stage 2 but without the synthetic data. We then additionally fine-tune these models with the same procedure as stage 3, but without any of the terms involving synthetic data. We report the BLEU scores for all these configurations in Table 5. The results suggest: the baseline without synthetic parallel data shows inferior performance across all language pairs compared to our approach leveraging synthetic parallel data.
Stage	newsdev2019 Gu ↔ En	newsdev2019 Kk ↔ En	newsdev2016 Tr ↔ En	FLoRes dev Ne ↔ En	FLoRes dev Si ↔ En
Baseline	First	5.0	23.4	4.0	16.1	6.3	17.7	2.8	15.1	1.3	12.0
Without synthetic data	Second	6.2	24.8	4.24	17.0	6.3	18.5	3.6	16.0	1.6	12.7
Third	12.9	26.2	6.1	16.3	12.9	19.5	5.9	16.1	5.2	13.1
With synthetic data	Second	19.6	29.8	10.6	20.0	16.7	23.8	7.3	17.4	8.3	16.6
Third	18.6	30.3	11.6	21.5	17.9	24.7	8.2	17.6	7.7	17.4
Table 5: BLEU scores of model configurations with or without synthetic data. Otherwise, we report the numbers after stage 2 for both models and use the results after stage 1 as a baseline.
Objectives	ΔBLEU
With synthetic data	BT	0.6
+ CT with Hi	2.1
+ CT with Ru and Ar	2.6
Without synthetic data	BT	0.0
+ CT with Hi	1.3
+ CT with Ru and Ar	1.4
Finally, we inspect whether the synthetic parallel data is still necessary in stage 3 or if it suffices to only leverage it during the second stage. We consider three fine-tuning strategies, where we either (1) only utilize on-the-fly back-translation (2) additionally include cross-translation terms for Gujarati, Nepali, and Sinhala using Hindi (3) additionally include a cross-translation terms for Turkish and Kazakh involving Arabic and Russian respectively. We compare all of the approaches to the vanilla strategy that only leverages on-the-fly back-translation and report the aggregate improvements in BLEU on the  $\mathrm{X}\rightarrow \mathrm{En}$  directions over this baseline in Table 6. We see two trends: The configurations that do not leverage synthetic data perform worse than those that do, and increasing multilinguality through the inclusion of cross-translation further improves performance.
7.3 Our approach is robust under multiple domains
We investigate the impact of data quantity and quality on the performance of our models. In this experiment, we focus on  $\mathrm{En} \leftrightarrow \mathrm{Gu}$  and use all available monolingual and auxiliary parallel data for all languages except Gujarati. We consider three configurations: (1) 500,000 lines from News Crawl (in-domain high-quality data); (2) 500,000 lines from Common Crawl (multi-domain data); (3) 100,000 lines from News Crawl. We present the results on
Table 6: Total BLEU increase for  $\mathrm{X} \rightarrow  \mathrm{{En}}$  over baseline fine-tuning strategy consisting of on-the-fly back-translation (BT) and no synthetic data. We refer to cross-translation as "CT".
Data Configurations	newstest2019	newsdev2019
Gu← En	Gu← En
500k News crawl	6.8	15.7	9.7	21.7
500k Common crawl	9.2	16.7	9.4	22.5
100k News crawl	3.6	10.0	5.4	12.4
mBART	-	13.8	-	-
Kim et al. (2020)	0.6	0.6	-	-
Table 7: BLEU scores for various configurations of Gujarati monolingual data, where we vary amount of data and domain. We include the best results of mBART and (Kim et al., 2020) for comparison.
both newstest2019 and newsdev2019 for  $\mathrm{En} \leftrightarrow \mathrm{Gu}$  on Table 7. We see that both Common Crawl and News Crawl configurations produce similar results at this scale, with the Common Crawl configuration having a small edge on average. Notice that even in this data-starved setting, we still outperform the competing unsupervised models. Once we reach only 100,000 lines, performance degrades below mBART but still outperforms the bilingual UNMT approach of Kim et al. (2020), revealing the power of multilinguality in low-resource settings.
In this work, we studied how multilinguality can make unsupervised translation viable for low-resource languages in a realistic setting. Our results show that utilizing the auxiliary parallel data in combination with synthetic data through our three-stage training procedure not only yields large gains over unsupervised baselines but also outperforms several modern supervised approaches.
Roeaharoni, Melvin Johnson, and Orhan Firat. 2019.
Massively multilingual neural machine translation. In NAACL.
Maruan Al-Shedivat and Ankur P Parikh. 2019. Consistency by agreement in zero-shot neural machine translation. In NAACL.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a. The missing ingredient in zero-shot neural machine translation. arXiv preprint arXiv:1903.07091.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019b. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. In ACL.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In ICLR.
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020. A call for more rigor in unsupervised cross-lingual learning. arXiv preprint arXiv:2004.14958.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Hongxiao Bai, Mingxuan Wang, Hai Zhao, and Lei Li. 2020. Unsupervised neural machine translation with indirect supervision. arXiv preprint arXiv:2004.03137.
Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zeroresource neural machine translation. arXiv preprint arXiv:1705.00753.
Orhan First, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In *NAACL*.
Orhan First, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268-277, Austin, Texas. Association for Computational Linguistics.
Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur P Parikh. 2020. A multilingual view of unsupervised machine translation. arXiv preprint arXiv:2002.02955.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018. Universal neural machine translation for extremely low resource languages. arXiv preprint arXiv:1802.05368.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6100-6113.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation Transactions of the Association for Computational Linguistics, 5:339-351.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700-1709.
Yunsu Kim, Miguel Graça, and Hermann Ney. 2020. When and why is unsupervised neural machine translation useless? arXiv preprint arXiv:2004.10581.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71.
Anoop Kunchukuttan. 2020. The indicnlp library.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2017. The iit bombay english-hindi parallel corpus. arXiv preprint arXiv:1710.02855.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In ICLR.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020a. Data-dependent gaussian prior objective for language generation. In ICLR.
Zuchao Li, Hai Zhao, Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2020b. Reference language based unsupervised neural machine translation. arXiv preprint arXiv:2004.02127.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When does unsupervised machine translation work? arXiv preprint arXiv:2004.05516.
Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. arXiv preprint arXiv:1808.04189.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Sujith Ravi and Kevin Knight. 2011. Deciphering foreign language. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 12-21.
Shuo Ren, Wenhu Chen, Shujie Liu, Mu Li, Ming Zhou, and Shuai Ma. 2018. Triangular architecture for rare language translation. arXiv preprint arXiv:1805.04813.
Shuo Ren, Yu Wu, Shujie Liu, Ming Zhou, and Shuai Ma. 2019. Explicit cross-lingual pre-training for unsupervised machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 770-779.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wikimatrix: Mining 135m parallel sentences in 1620
language pairs from wikipedia. arXiv preprint arXiv:1907.05791.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan First, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020a. Leveraging monolingual data with self-supervision for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2827-2835, Online. Association for Computational Linguistics.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020b. Leveraging monolingual data with self-supervision for multilingual neural machine translation. arXiv preprint arXiv:2005.04816.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pages 5926-5936.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2020. Knowledge distillation for multilingual unsupervised neural machine translation. arXiv preprint arXiv:2004.10171.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Chang Xu, Tao Qin, Gang Wang, and Tie-Yan Liu. 2019. Polygon-net: a general framework for jointly boosting multiple unsupervised neural machine translation models. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5320-5326. AAAI Press.
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3530-3534.