SlowGuess/ACL_2020_OCR huggingface dataset:
Note: The first column shows best-performed (till 2019) BLEU scores of supervised NMT (SNMT) systems reported in the corresponding WMT news translation task (http://matrix.statmt.org). The second and third column show BLEU scores reported in the corresponding papers. SM shows the UNMT single model on these two languages (our baseline); LBUNMT shows the multilingual model across all languages in the same language branch; MUNMT shows the multilingual model across all languages; SKD shows the multilingual model with self-knowledge distillation across all languages; LBKD shows the multilingual model with language branch knowledge distillation across all languages. Note that the results for En-Ro are evaluated on the dataset with diacritics removed in the reference text for all our implemented systems.
Table 2: BLEU scores of all models on the English to non-English language pairs.
Corpus	SNMT	Sen et al. (2019)	Xu et al. (2019)	SM	LBUNMT	MUNMT	SKD	LBKD
Cs-En	27.10	-	11.56	20.62	20.62	20.09	21.05	21.25
De-En	28.40	11.94	16.46	21.31	21.31	21.95	22.54	22.81
Es-En	31.40	15.45	20.35	25.53	25.77	25.37	26.15	26.59
Et-En	30.90	-	-	19.48	20.30	19.60	20.95	21.31
Fi-En	33.00	-	-	7.62	7.68	7.19	7.92	7.80
Fr-En	32.20	14.47	19.87	25.86	26.02	25.41	26.07	26.48
Hu-En	-	-	-	14.48	14.86	14.54	15.16	15.34
It-En	-	-	-	24.33	24.87	24.77	25.30	25.35
Lt-En	36.30	-	-	1.72	11.00	14.04	15.31	15.84
Lv-En	21.90	-	-	0.95	12.75	14.90	15.49	15.33
Ro-En	35.20	-	-	28.52	29.57	28.38	29.58	30.18
Tr-En	28.00	-	-	12.99	12.99	15.65	16.85	17.35
Average	-	-	-	16.95	18.98	19.32	20.20	20.47
Table 3: BLEU scores of all models on the non-English to English language pairs.
are as follows:
1) Our proposed LBUNMT model trained in the same language branch performed better than the single model (SM) because similar languages have a positive interaction during the training process. Moreover, SM performed very poorly on low-resource language pairs such as En-Lt and En-Lv in the Baltic language branch.
2) Our proposed MUNMT model trained in all languages significantly outperformed the previous work (Sen et al., 2019; Xu et al., 2019) by  $4\sim 12$  BLEU scores. Moreover, the MUNMT model could alleviate the poor performance achieved with
initialized with the same parameters of pretrained language model (just an encoder).
low-resource language pairs, such as En-Lt and En-Lv. However, the performance of MUNMT is slightly worse than SM in some language pairs.
3) Our proposed knowledge distillation methods outperformed the original MUNMT model by approximately 1 BLEU score. Moreover, our proposed MUNMT with knowledge distillation performed better than SM in all language pairs with fewer training iterations. Regarding our two proposed methods, LBKD achieved better performance since it could obtain much more knowledge distilled from LBUNMT model.
4) There is a gap between the performance of our proposed MUNMT model and that of the su
pervised NMT systems. To bridge this gap, relying solely on monolingual training data, is worthy of being studied in the future.
6.1 Zero-shot Translation Analysis
We also studied the zero-shot translation accuracy of the MUNMT model. Although MUNMT could be trained on all translation directions (ordered language pairs), it would require an extremely long training time. Our proposed MUNMT model was trained in 24 translation directions (all English and non-English language pairs, in each direction), whereas 156 translation directions exist. As the number of languages increases, the number of translation directions increases quadratically. Therefore, zero-shot translation accuracy is important to the MUNMT model.
Methods	→	Cs	De	Es	Fr
Xu et al. (2019)	-	11.16	11.29	10.61
Sen et al. (2019)	-	-	-	-
MUNMT	Cs	-	11.91	15.22	14.66
LBKD	-	13.16	16.63	16.28
SKD	-	16.96	20.52	20.14
Xu et al. (2019)	10.52	-	13.68	9.45
Sen et al. (2019)	-	-	7.40	6.78
MUNMT	De	10.56	-	16.15	15.85
LBKD	11.53	-	17.27	16.96
SKD	14.58	-	20.20	20.61
Xu et al. (2019)	8.32	11.20	-	24.13
Sen et al. (2019)	-	4.78	-	13.92
MUNMT	Es	10.04	11.87	-	21.90
LBKD	10.86	12.98	-	23.05
SKD	13.63	16.62	-	27.04
Xu et al. (2019)	8.89	11.24	23.88	-
Sen et al. (2019)	-	4.59	13.87	-
MUNMT	Fr	9.77	11.70	22.30	-
LBKD	10.48	12.67	22.65	-
SKD	13.04	16.31	25.92	-
Table 4 shows the performance of translation between non-English language pairs in the zero-shot translation scenario. Note that Xu et al. (2019) (2019) shows the results of direct translation between the two languages, not the result of zero-shot translation. Compared with previous works, our MUNMT model outperformed the previous systems in almost all translation directions, particu
larly the direct translation results reported in Xu et al. (2019). Compared with the original MUNMT model, our proposed knowledge distillation methods further improved the performance of zero-shot translation. Regarding our two proposed methods, SKD significantly outperformed LBKD by approximately 3 BLEU scores since the third language was introduced during SKD translation training for two language pairs, achieving much more cross-lingual knowledge.
6.2 Further Training (Fine-tuning) Analysis
To better assess the effectiveness of our proposed MUNMT model, we further trained the MUNMT and LBKD model individually on each language pair for 15,000 iterations. As shown in Tables 5 and 6, after further training, the model outperformed the original single model on each language pair by approximately 4 BLEU scores. Actually, the number of iterations of the whole process (including training the MUNMT model) is half that of the original single model. This demonstrates that our proposed MUNMT model is a robust system and contains substantial cross-lingual information that could improve translation performance.
Table 4: BLEU scores of the MUNMT model between pairs of non-English languages. The first two rows of each block are the reported BLEU scores from the corresponding papers.
Corpus	SM	MUNMT	+FT	LBKD	+FT
En-Cs	14.54	14.40	15.79	15.47	15.93
En-De	18.26	17.58	19.57	19.28	20.00
En-Es	25.14	25.05	27.59	26.79	27.80
En-Et	14.86	14.09	16.62	15.62	17.21
En-Fi	9.87	9.75	11.05	10.57	11.58
En-Fr	26.02	25.84	28.56	27.78	28.62
En-Hu	11.32	10.90	12.77	12.03	13.12
En-It	24.19	23.80	25.25	25.52	25.98
En-Lt	0.79	10.07	10.92	11.11	11.22
En-Lv	1.02	13.09	14.33	14.33	15.17
En-Ro	29.44	28.82	32.38	31.28	32.43
En-Tr	11.87	12.41	14.78	13.83	15.30
Average	15.61	17.15	19.13	18.63	19.53
Table 5: The +FT column shows BLEU scores from further training of the MUNMT and LBKD model on the English to non-English language pairs. The other columns show results from Table 2.
Multilingual NMT has attracted much attention in the machine translation community. Dong et al. (2015) first extended NMT from the translation of a single language pair to multiple language pairs, using a shared encoder and multiple decoders and
Corpus	SM	MUNMT	+FT	LBKD	+FT
Cs-En	20.62	20.09	21.50	21.25	22.17
De-En	21.31	21.95	22.41	22.81	23.07
Es-En	25.53	25.37	26.24	26.59	26.78
Et-En	19.48	19.60	21.61	21.31	22.61
Fi-En	7.62	7.19	8.06	7.80	8.34
Fr-En	25.86	25.41	26.30	26.48	26.76
Hu-En	14.48	14.54	15.99	15.34	16.07
It-En	24.33	24.77	25.54	25.35	25.86
Lt-En	1.72	14.04	15.27	15.84	16.86
Lv-En	0.95	14.90	15.57	15.33	15.87
Ro-En	28.52	28.38	29.61	30.18	30.39
Tr-En	12.99	15.65	18.47	17.35	19.48
Average	16.95	19.32	20.55	20.47	21.19
Table 6: The +FT column shows BLEU scores from further training of the MUNMT and LBKD model on the non-English to English language pairs. The other columns show results from Table 3.
multiple attention mechanisms, for each language. Luong et al. (2016) translated multiple source languages to multiple target languages using a combination of multiple encoders and multiple decoders. First et al. (2016) used a shared attention mechanism but multiple encoders and decoders for each language. Ha et al. (2016) and Johnson et al. (2017) proposed a simpler method to use one encoder and one decoder to translate between multiple languages. Recently, many methods (Lakew et al., 2018; Platanios et al., 2018; Sachan and Neubig, 2018; Blackwood et al., 2018; Lu et al., 2018; Wang et al., 2019a; Aharoni et al., 2019; Wang et al., 2019b; Wang and Neubig, 2019) have been proposed to boost multilingual NMT performance. In particular, Tan et al. proposed a knowledge distillation method (Tan et al., 2019b) and a language clustering method (Tan et al., 2019a) to improve the performance of multilingual NMT. Ren et al. (2018) propose a triangular architecture to tackle the problem of low-resource pairs translation by introducing another rich language.
To further tackle the problem of low-resource pairs translation, UNMT (Artetxe et al., 2018; Lample et al., 2018a) has been proposed, using a combination of diverse mechanisms such as initialization with bilingual word embeddings, denoising autoencoder (Vincent et al., 2010), back-translation (Sennrich et al., 2016a), and shared latent representation. Lample et al. (2018b) concatenated two bilingual corpora as one monolingual corpus, and used monolingual embedding pretraining in the initialization step, to achieve remarkable results with
some similar language pairs. Lample and Conneau (2019) achieved better UNMT performance by introducing a pretrained language model. Sun et al. (2019, 2020) proposed to train UNMT with cross-lingual language representation agreement, to further improve UNMT performance. Moreover, an unsupervised translation task that evaluated in the WMT19 news translation task (Barrault et al., 2019) attracted many researchers to participate (Marie et al., 2019; Li et al., 2019).
For Multilingual UNMT, Xu et al. (2019) exploited multiple auxiliary languages for jointly boosting UNMT models via the Polygon-Net framework. Sen et al. (2019) proposed an MUNMT scheme that jointly trains multiple languages with a shared encoder and multiple decoders. In contrast with their use of multiple decoders, we have constructed a simpler MUNMT model with one encoder and one decoder. Further, we have extended the four or five languages used in their work to thirteen languages, for training our MUNMT model.
In this paper, we have introduced a unified framework, using a single encoder and decoder, for MUNMT training on a large scale of European languages. To further enhance MUNMT performance, we have proposed two knowledge distillation methods. Our extensive experiments and analysis demonstrate the effectiveness of our proposed methods. In the future, we intend to extend the work to include language types such as Asian languages. We will also introduce other effective methods to improve zero-shot translation quality.
We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. The corresponding authors are Rui Wang and Tiejun Zhao. Rui Wang was partially supported by JSPS grant-in-aid for early-career scientists (19K20354): "Unsupervised Neural Machine Translation in Universal Scenarios" and NICT tenure-track researcher startup fund "Toward Intelligent Machine Translation". Tiejun Zhao was partially supported by National Key Research and Development Program of China via grant 2017YFB1002102. Masao Utiyama was partially supported by JSPS KAKENHI Grant Number 19H05660.
Roee Aharoni, Melvin Johnson, and Orhan First. 2019. Massively multilingual neural machine translation. In NAACL, pages 3874-3884, Minneapolis, Minnesota.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In ICLR, Vancouver, Canada.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In WMT, pages 1-61, Florence, Italy.
Graeme Blackwood, Miguel Ballesteros, and Todd Ward. 2018. Multilingual neural machine translation with task-specific attention. In *COLING*, pages 3112-3122, Santa Fe, New Mexico, USA.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In ACL, pages 1723-1732, Beijing, China.
Orhan First, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In *NAACL*, pages 866–875, San Diego, California.
Thanh-Le Ha, Jan Niehues, and Alexander H. Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. CoRR, abs/1611.04798.
Sangchul Hahn and Heeyoul Choi. 2019. Self-knowledge distillation in natural language processing. CoRR, abs/1908.01851.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In NIPS, pages 820-828, Barcelona, Spain.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In *NAACL*, pages 1367-1377, San Diego California, USA.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. TACL, 5:339-351.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR, San Diego, California.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL, pages 177-180, Prague, Czech Republic.
Surafel Melaku Lakew, Mauro Cettolo, and Marcello Federico. 2018. A comparison of transformer and recurrent neural networks on multilingual neural machine translation. In COLING, pages 641-652, Santa Fe, New Mexico, USA.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. CoRR, abs/1901.07291.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In ICLR, Vancouver, Canada.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In EMNLP, pages 5039-5049, Brussels, Belgium.
M Paul Lewis. 2009. Ethnologue: Languages of the world. SIL international.
Bei Li, Yinqiao Li, Chen Xu, Ye Lin, Jiqiang Liu, Hui Liu, Ziyang Wang, Yuhao Zhang, Nuo Xu, Zeyang Wang, Kai Feng, Hexuan Chen, Tengbo Liu, Yanyang Li, Qiang Wang, Tong Xiao, and Jingbo Zhu. 2019. The NiuTrans machine translation systems for WMT19. In WMT, pages 257-266, Florence, Italy.
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation. In WMT, pages 84-92, Belgium, Brussels.
Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR, San Juan, Puerto Rico.
Benjamin Marie, Haipeng Sun, Rui Wang, Kehai Chen, Atsushi Fujita, Masao Utiyama, and Eiichiro Sumita. 2019. NICT's unsupervised neural and statistical machine translation systems for the WMT19 news translation task. In WMT, pages 294-301, Florence, Italy.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In EMNLP, pages 425-435, Brussels, Belgium.
Shuo Ren, Wenhu Chen, Shujie Liu, Mu Li, Ming Zhou, and Shuai Ma. 2018. Triangular architecture for rare language translation. In ACL, pages 56-65, Melbourne, Australia.
Devendra Sachan and Graham Neubig. 2018. Parameter sharing methods for multilingual self-attentional translation models. In WMT, pages 261-271, Belgium, Brussels.
Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multilingual unsupervised NMT using shared encoder and language-specific decoders. In ACL, pages 3083-3089, Florence, Italy.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In ACL, pages 86-96, Berlin, Germany.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In ACL, pages 1715-1725, Berlin, Germany.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019. Unsupervised bilingual word embedding agreement for unsupervised neural machine translation. In ACL, pages 1235-1245, Florence, Italy.
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2020. Unsupervised neural machine translation with cross-lingual language representation agreement. TASLP.
Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao QIN, and Tie-Yan Liu. 2019a. Multilingual neural machine translation with language clustering. In EMNLP, pages 962-972, Hong Kong, China.
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and TieYan Liu. 2019b. Multilingual neural machine translation with knowledge distillation. In ICLR, New Orleans, LA, USA.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. JMLR, 11:3371-3408.
Rui Wang, Haipeng Sun, and Sumita Eiichiro Utiyama, Masao. 2020. A survey of advances and challenges in unsupervised neural machine translation. In ANLP, Mito, Japan.
Xinyi Wang and Graham Neubig. 2019. Target conditioned sampling: Optimizing data selection for multilingual neural machine translation. In ACL, pages 5823-5828, Florence, Italy.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham Neubig. 2019a. Multilingual neural machine translation with soft decoupled encoding. In ICLR, New Orleans, LA, USA.
Yining Wang, Long Zhou, Jiajun Zhang, Feifei Zhai, Jingfang Xu, and Chengqing Zong. 2019b. A compact and language-sensitive multilingual translation method. In ACL, pages 1213-1223, Florence, Italy.
Chang Xu, Tao Qin, Gang Wang, and Tie-Yan Liu. 2019. Polygon-net: A general framework for jointly boosting multiple unsupervised neural machine translation models. In *IJCAI*, pages 5320-5326.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018. Unsupervised neural machine translation with weight sharing. In ACL, pages 46-55, Melbourne, Australia.