SlowGuess/ACL_2020_OCR huggingface dataset:
Deepening the Transformer also improves the modeling capacity  $(+1.88\mathrm{BLEU}_{94}, 4 \rightarrow 8)$ . Although deep Transformer performs worse than  $\mathrm{LALN} + \mathrm{LALT}$  under a similar number of model parameters in terms of BLEU  $(-1.49\mathrm{BLEU}_{94}, 7 \rightarrow 8)$ , it shows more consistent improvements across different language pairs  $(+6.4\%$  WR). We obtain better performance when integrating all approaches (9). By increasing the model depth to
24 (①), Transformer with our approach yields a score of 29.60  $\mathrm{BLEU}_{94}$  and 21.23  $\mathrm{BLEU}_4$ , beating the baseline (③) on  $92.6\%$  tasks and outperforming the base bilingual model (①) by 0.33  $\mathrm{BLEU}_4$ . Our approach significantly narrows the performance gap between multilingual NMT and bilingual NMT (20.90  $\mathrm{BLEU}_4 \rightarrow 21.23 \mathrm{BLEU}_4$ , ①→①), although similarly deepening bilingual models surpasses our approach by 1.52  $\mathrm{BLEU}_4$  (①→②).
6.3 Results on Many-to-Many Translation
We train many-to-many NMT models on the concatenation of the one-to-many dataset (English  $\rightarrow$  X) and its reversed version (X  $\rightarrow$  English), and evaluate the zero-shot performance on X  $\rightarrow$  X language pairs. Table 3 and Table 4 show the translation results for English  $\rightarrow$  X and X  $\rightarrow$  English, respectively. $^{10}$  We focus on the translation performance w/o ROBT in this subsection.
Compared to the one-to-many translation, the many-to-many translation must accommodate twice as many translation directions. We observe that many-to-many NMT models suffer more se
<sup>10</sup>Note that the one-to-many training and test sets were not yet aggressively filtered for sentence overlap as described in Section 5, so results in Table 2 and Table 3 are not directly comparable.
ID	Model Architecture	L	#Param	English→X	X→English
High	Med	Low	High	Med	Low
1	Transformer	6	110M	20.69	20.82	15.18	26.99	28.60	27.49
2	1 + MATT	6	103M	19.70	19.77	14.17	26.32	27.81	26.84
3	2 + LALN + LALT	6	133M	21.07	22.88	19.99	27.03	28.60	26.97
4	2	12	141M	21.67	22.17	16.95	28.39	30.24	29.26
5	3	12	173M	22.48	24.38	21.58	28.66	30.73	29.50
6	3	24	254M	23.69	25.61	22.24	30.29	32.58	31.90
Table 5: Test BLEU for High/Medium/Low (High/Med/Low) resource language pairs in many-to-many setting on OPUS-100 (100 languages). We report average BLEU for each category.
ID	Model Architecture	L	#Param	w/o ROBT	w/ ROBT
BLEUzero	ACCzero	BLEUzero	ACCzero
1	Transformer, Pivot & Bilingual	6	110M	12.98	84.87	-	-
2	Transformer	6	110M	3.97	36.04	10.11	86.08
3	2 + MATT	6	103M	3.49	31.62	9.67	85.87
4	3 + LALN + LALT	6	133M	4.02	45.43	11.23	87.40
5	3	12	141M	4.71	39.40	11.87	87.44
6	4	12	173M	5.41	51.40	12.62	87.99
7	4	24	254M	5.24	47.91	14.08	87.68
8	7 + Pivot	24	254M	14.71	84.81	14.78	85.09
Table 6: Test BLEU and translation-language accuracy for zero-shot translation in many-to-many setting on OPUS-100 (100 languages). "BLEUzero/ACCzero": average BLEU/accuracy over all zero-shot translation directions in test set, "Pivot": the pivot-based translation that first translates one source sentence into English (X→English NMT), and then into the target language (English→X NMT). Lower accuracy indicates severe off-target translation. The average Pearson correlation coefficient between language accuracy and the corresponding BLEU is 0.93 (significant at  $p < 0.01$ ).
rious capacity issues on English  $\rightarrow$  X tasks (-4.93 BLEU $_4$ , ①  $\rightarrow$  ② in Table 3 versus -1.95 BLEU $_4$  in Table 2), where the deep Transformer with LALN + LALT effectively reduces this gap to -0.45 BLEU $_4$  (①  $\rightarrow$  ⑦, Table 3), resonating with our findings from Table 2. By contrast, multilingual NMT benefits X  $\rightarrow$  English tasks considerably from the multitask learning alone, outperforming bilingual NMT by 2.13 BLEU $_4$  (①  $\rightarrow$  ②, Table 4). Enhancing model capacity further enlarges this margin to +4.80 BLEU $_4$  (①  $\rightarrow$  ⑦, Table 4).
We find that the overall quality of English  $\rightarrow$  X translation (19.50/23.96 BLEU94, ②/⑦, Table 3) lags far behind that of its X  $\rightarrow$  English counterpart (27.60/31.36 BLEU94, ②/②, Table 4), regardless of the modeling capacity. We ascribe this to the highly skewed training data distribution, where half of the training set uses English as the target. This strengthens the ability of the decoder to translate into English, and also encourages knowledge transfer for X  $\rightarrow$  English language pairs. LALN and LALT show the largest benefit for English  $\rightarrow$  X (+2.9 BLEU94, ③  $\rightarrow$  ④, Table 3), and only a small benefit for X  $\rightarrow$  English (+0.6 BLEU94, ③  $\rightarrow$  ④, Table 4). This makes sense considering that LALN
and LALT are specific to the target language, so capacity is mainly increased for English  $\rightarrow$  X. Deepening the Transformer yields benefits in both directions (+2.57 BLEU94 for English  $\rightarrow$  X, +3.86 BLEU94 for X  $\rightarrow$  English; ④  $\rightarrow$  ⑦, Tables 3 and 4).
6.4 Effect of Training Corpus Size
Our multilingual training data is distributed unevenly across different language pairs, which could affect the knowledge transfer delivered by language-aware modeling and deep Transformer in multilingual translation. We investigate this effect by grouping different language pairs in OPUS-100 into three categories according to their training data size: High ( $\geq 0.9\mathrm{M}$ , 45), Low ( $< 0.1\mathrm{M}$ , 18) and Medium (others, 31). Table 5 shows the results.
Language-aware modeling benefits low-resource language pairs the most on English  $\rightarrow$  X translation (+5.82 BLEU, Low versus +1.37/+3.11 BLEU, High/Med, ②→③), but has marginal impact on X  $\rightarrow$  English translation as analyzed in Section 6.3. By contrast, deep Transformers yield similar benefits across different data scales (+2.38 average BLEU, English  $\rightarrow$  X and +2.31 average BLEU, X  $\rightarrow$  English, ②→④). We obtain the best perfor
mance by integrating both  $(1) \rightarrow (6)$  with a clear positive transfer to low-resource language pairs.
6.5 Results on Zero-Shot Translation
Previous work shows that a well-trained multilingual model can do zero-shot  $\mathrm{X} \rightarrow \mathrm{Y}$  translation directly (First et al., 2016b; Johnson et al., 2017). Our results in Table 6 reveal that the translation quality is rather poor (3.97  $\mathrm{BLEU}_{\text {zero }}$ ,  $②$  w/o ROBT) compared to the pivot-based bilingual baseline (12.98  $\mathrm{BLEU}_{\text {zero }}$ ,  $①$ ) under the massively multilingual setting (Aharoni et al., 2019), although translations into different target languages show varied performance. The marginal gain by the deep Transformer with  $\mathrm{LALN} + \mathrm{LALT}(+1.44 \mathrm{BLEU}_{\text {zero }}$ ,  $② \rightarrow ⑥$ , w/o ROBT) suggests that weak model capacity is not the major cause of this inferior performance.
In a manual analysis on the zero-shot NMT outputs, we found many instances of off-target translation (Table 1). We use translation-language accuracy to measure the proportion of translations that are in the correct target language. Results in Table 6 show that there is a huge accuracy gap between the multilingual and the pivot-based method  $(-48.83\%)$ $\mathrm{ACC}_{zero}$ ,  $① \rightarrow ②$ , w/o ROBT), from which we conclude that the off-target translation issue is one source of the poor zero-shot performance.
We apply ROBT to multilingual models by finetuning them for an extra 100k steps with the same batch size as for training. Table 6 shows that ROBT substantially improves  $\mathrm{ACC}_{zero}$  by  $35\% \sim 50\%$ , reaching  $85\% \sim 87\%$  under different model settings. The multilingual Transformer with ROBT achieves a translation improvement of up to 10.11  $\mathrm{BLEU}_{zero}$  ( $②$  w/o ROBT  $\rightarrow$ $⑦$  w/ ROBT), outperforming the bilingual baseline by 1.1  $\mathrm{BLEU}_{zero}$  ( $①$  w/o ROBT  $\rightarrow$ $⑦$  w/ ROBT) and approaching the pivot-based multilingual baseline (-0.63  $\mathrm{BLEU}_{zero}$ ,  $⑧$  w/o ROBT  $\rightarrow$ $⑦$  w/ ROBT).<sup>11</sup> The strong Pearson correlation between the accuracy and BLEU (0.92 on average, significant at  $p < 0.01$ ) suggests that the improvement on the off-target translation issue explains the increased translation performance to a large extent.
Results in Table 3 and 4 show that ROBT's success on zero-shot translation comes at the cost of sacrificing  $\sim 0.50$ $\mathrm{BLEU}_{94}$  and  $\sim 4\%$  WR on English  $\rightarrow \mathrm{X}$  and  $\mathrm{X} \rightarrow \mathrm{English}$  translation. We also note that models with more capacity yield higher
Figure 1: Zero-shot average test BLEU for multilingual NMT models finetuned by ROBT. ALL = MATT + LALN + LALT. Multilingual models with ROBT quickly converge on zero-shot directions.
Setting	BLEUzero
6-to-6	11.98
100-to-100	11.23
Table 7: Zero-short translation quality for ROBT under different settings. "100-to-100": the setting used in the above experiments; we set  $\mathcal{T}$  to all target languages. "6-to-6":  $\mathcal{T}$  only includes the zero-shot languages in the test set. We employ 6-layer Transformer with LALN and LALT for experiments.
language accuracy  $(+7.78\% / +13.81\%)$ $\mathrm{ACC}_{zero}$ ,  $③ \rightarrow ⑤ / ③ \rightarrow ④$ , w/o ROBT) and deliver better zero-shot performance before  $(+1.22 / +0.53$ $\mathrm{BLEU}_{zero}$ ,  $③ \rightarrow ⑤ / ③ \rightarrow ④$ , w/o ROBT) and after ROBT  $(+2.20 / +1.56$ $\mathrm{BLEU}_{zero}$ ,  $③ \rightarrow ⑤ / ③ \rightarrow ④$ , w/ ROBT). In other words, increasing the modeling capacity benefits zero-shot translation and improves robustness.
Convergence of ROBT. Unlike prior studies (Gu et al., 2019; Lakew et al., 2019), we resort to an online method for backtranslation. The curve in Figure 1 shows that ROBT is very effective, and takes only a few thousand steps to converge, suggesting that it is unnecessary to decode the whole training set for each zero-shot language pair. We leave it to future work to explore whether different back-translation strategies (other than greedy decoding) will deliver larger and continued benefits with ROBT.
Impact of  $\mathcal{T}$  on ROBT. ROBT heavily relies on  $\mathcal{T}$ , the set of target languages considered, to distribute the modeling capacity on zero-shot directions. To study its impact, we provide a comparison by constraining  $\mathcal{T}$  to 6 languages in the zero-shot test set. Results in Table 7 show that the biased ROBT outperforms the baseline by  $0.75\mathrm{BLEU}_{zero}$ . By narrowing  $\mathcal{T}$ , more capacity is scheduled to the focused languages, which results in performance improvements. But the small scale of this improve
<sup>11</sup>Note that ROBT improves all zero-shot directions due to its randomness in sampling the intermediate languages. We do not bias ROBT to the given zero-shot test set.
ment suggests that the number of zero-shot directions is not ROBT's biggest bottleneck.
This paper explores approaches to improve massively multilingual NMT, especially on zero-shot translation. We show that multilingual NMT suffers from weak capacity, and propose to enhance it by deepening the Transformer and devising language-aware neural models. We find that multilingual NMT often generates off-target translations on zero-shot directions, and propose to correct it with a random online backtranslation algorithm. We empirically demonstrate the feasibility of back-translation in massively multilingual settings to allow for massively zero-shot translation for the first time. We release OPUS-100, a multilingual dataset from OPUS including 100 languages with around 55M sentence pairs for future study. Our experiments on this dataset show that the proposed approaches substantially increase translation performance, narrowing the performance gap with bilingual NMT models and pivot-based methods.
In the future, we will develop lightweight alternatives to LALT to reduce the number of model parameters. We will also exploit novel strategies to break the upper bound of ROBT and obtain larger zero-shot improvements, such as generative modeling (Zhang et al., 2016; Su et al., 2018; García et al., 2020; Zheng et al., 2020).
This project has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreements 825460 (ELITR) and 825299 (GoURMET). This project has received support from Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland. Rico Sennrich acknowledges support of the Swiss National Science Foundation (MUTAMUR; no. 176727).
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.
Maruan Al-Shedivat and Ankur Parikh. 2019. Consistency by agreement in zero-shot neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1184-1197, Minneapolis, Minnesota. Association for Computational Linguistics.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a. The missing ingredient in zero-shot neural machine translation. CoRR, abs/1903.07091.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019b. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs/1907.05019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Ankur Bapna, Mia Chen, Orhan First, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3028-3033, Brussels, Belgium. Association for Computational Linguistics.
Ankur Bapna and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538-1548, Hong Kong, China. Association for Computational Linguistics.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Graeme Blackwood, Miguel Ballesteros, and Todd Ward. 2018. Multilingual neural machine translation with task-specific attention. In Proceedings of the 27th International Conference on Computational
Linguistics, pages 3112-3122, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Anna Currey and Kenneth Heafield. 2019. Zeroresource neural machine translation with monolingual pivot data. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 99-107, Hong Kong. Association for Computational Linguistics.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1723-1732, Beijing, China. Association for Computational Linguistics.
Orhan First, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866-875, San Diego, California. Association for Computational Linguistics.
Orhan First, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268-277, Austin, Texas. Association for Computational Linguistics.
Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur P. Parikh. 2020. A multilingual view of unsupervised machine translation. ArXiv, abs/2002.02955.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2019. Improved zero-shot neural machine translation via ignoring spurious correlations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1258-1268, Florence, Italy. Association for Computational Linguistics.
Thanh-Le Ha, Jan Niehues, and Alexander Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. In Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT), Seattle, USA.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. CoRR, abs/1512.03385.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's
multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
Philipp Koehn. 2010. Statistical Machine Translation, 1st edition. Cambridge University Press, New York, NY, USA.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Surafel M. Lakew, Marcello Federico, Matteo Negri, and Marco Turchi. 2019. Multilingual Neural Machine Translation for Zero-Resource Languages. arXiv e-prints, page arXiv:1909.07342.
Surafel Melaku Lakew, Mauro Cettolo, and Marcello Federico. 2018. A comparison of transformer and recurrent neural networks on multilingual neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 641-652, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365-378.
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 84-92, Brussels, Belgium. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 425-435, Brussels, Belgium. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on
Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Devendra Sachan and Graham Neubig. 2018. Parameter sharing methods for multilingual self-attentional translation models. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 261–271, Brussels, Belgium. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. 2018. Variational recurrent neural machine translation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao QIN, and Tie-Yan Liu. 2019. Multilingual neural machine translation with language clustering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 963-973, Hong Kong, China. Association for Computational Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), Istanbul, Turkey. European Language Resources Association (ELRA).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Raul Vázquez, Alessandro Raganato, Jörg Tiedemann, and Mathias Creutz. 2019. Multilingual NMT with a language-independent attention bridge. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 33-39, Florence, Italy. Association for Computational Linguistics.