言語処理学会 第26回年次大会 発表論文集 (2020年3月)

A Survey of Advances and Challenges
in Unsupervised Neural Machine Translation
Rui Wang, Haipeng Sun, Masao Utiyama, and Eiichiro Sumita
National Institute of Information and Communications Technology{wangrui, sun.haipeng, mutiyama, eiichiro.sumita}@nict.go.jp

1

Introduction

Unsupervised
cross-lingual
language
representation initialization methods, together
with mechanisms such as denoising and backtranslation, have advanced unsupervised neuralmachine translation (UNMT), which has achieved
impressive results.
Meanwhile, there are stillchallenges for UNMT. This paper ﬁrst introduces
the background and the latest progress of UNMT.
We then examine a number of challenges to UNMT
and analyze the future trend of UNMT.

2Advances

Recently, neural machine translation (NMT)
has been adapted to the unsupervised scenario
where NMT is trained without any bilingual data.
Unsupervised NMT (UNMT) [1, 2] only requires
monolingual corpora, using a combination of diverse
mechanisms such as an initialization with bilingual
word embeddings, denoising auto-encoder [3], backtranslation [4], and shared latent representation.
A shared encoder was used to encode the source
sentences and decode them from a shared latent
space [1, 2]. The diﬀerence is that [2] used a single
shared decoder and [1] leveraged two independent
decoders for each language. [5] used two independent
encoders for each language with a weight-sharing
mechanism to overcome the weakness of retaining
the uniqueness and internal characteristics ofeach language.
[6] achieved remarkable results
in several similar languages such as EnglishFrench by concatenating two bilingual corpora as

one monolingual corpus and using monolingual
embedding pre-training in the initialization step.
Recent adavances:[7] utilized auxiliary
languages to boost UNMT model. [8] proposed
an extract-edit approach, to extract and then edit
real sentences from the target monolingual corpora
instead of back-translation. [9] proposed to train
UNMT with bilingual word embedding agreement.
[10] introduced unsupervised pivot translation for
distant language pairs. More recently, [11, 12]
introduced the pre-trained cross-lingual language
model to achieve better UNMT performance.
USMT and UNMT: [6, 13] proposed analternative method, that is, unsupervised statisticalmachine translation (USMT) method.
[14, 15,16] combined UNMT and USMT to improve
unsupervised machine translation performance. In
2019, the unsupervised MT task (German-Czech)
ﬁrst-time became the oﬃcially task of WMT-2019,
and the system from NICT [17] won the ﬁrst
place and achieved state-of-the-art performances bycombining the USMT and UNMT.
Supervised and unsupervised NMT: [17]also explored the relationship between supervisedand unsupervised NMT. First, they found thatthe pseudo-parallel data generated by unsupervised
MT can be directly used as training data to train
a pseudo-supervised NMT system with signiﬁcant
improvement. In addition, they found that NICT’ssystem which achieved comparable performance with
online (supervised) systems [18], if it was ﬁne-tuned
by several thousands of parallel sentences.
We summarize the performances of recent UNMT
works in Table 1.

― 1069 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

表 1: Performance (BLEU score) of recent UNMT works.

3MethodFr-En

En-Fr

De-En

En-De

Ja-En

En-Ja

Artetxe et al. [1]

15.56

15.13

n/an/an/an/a

Lample et al. [2]

14.31

15.05

13.33

9.64

n/an/aYang et al. [5]

15.58

16.97

14.62

10.86

n/an/aLample et al. [6]

24.20

25.10

21.00

17.20

n/an/aSun et al. [9]

25.87

28.38

22.67

18.29

17.22

23.64

Lample et al. [11]

33.4

33.3

26.4

34.3

n/an/aMethods

4

State-of-the-art UNMT [6] can be decomposed

Challenges
Eﬀect on speciﬁc language:

Most existinginto four components: bilingual word embedding

works focus on modeling UNMT systems and few

initialization,

works have investigated the eﬀect of UNMT indenoisingauto-encoder,back-translation, and shared latent representations.

speciﬁc languages. The performances of UNMT in

BilingualCompared

word embedding initialization:with supervised neural machine

similar language pairs (French/German-English) are
dramatically better than that in distant languagetranslation, there is no bilingual data for training.

pairs (Chinese/Japanese-English) translation tasks.

Instead, the pre-trained unsupervised BWE provides

The UNMT performance in the distant language

a naive translation knowledge to enable the back-pairs needs to be solved.

translation to generate pseudo-parallel corpora at
the beginning of the training.
Denoising auto-encoder:

One of the syntactic structures of languages
are diﬀerent.

Without parallel supervision, it is

Noise obtained byvery diﬃcult for UNMT to learn the syntactic

randomly performing local substitutions and wordcorrespondence. Syntax information has been shown

reorderings [19, 20], is added to the input sentencesthat it can improve the performances of NMT [21,to improve model learning ability and regularization.

22, 23, 24, 25, 26, 27, 28].

Consequently, the input data are continuously

Domain adaptation methods for UNMT havemodiﬁed and diﬀerent at each epoch. The denoising

not been well-studied although UNMT has recently

auto-encoder model objective function would beachieved remarkable results in some speciﬁc domains

optimized by maximizing the probability of encoding

for several language pairs.

a noisy sentence and reconstructing it.

to inconsistent domains between training data andBack-translation: The back-translation plays atest data for supervised NMT, there also exist otherkey role in achieving unsupervised translation relying

inconsistent domains between monolingual training

only on monolingual corpora in each language.

data in two languages. Actually, it is diﬃcult for

The pseudo-parallel sentence pairs produced by thesome language pairs to obtain enough source andmodel at the previous iteration have been used totarget monolingual corpora from the same domain

train the new translation model.

in the real-world scenario.

Sharing latent representations: Encoders anddecoders are (partially) shared for two languages.

[29] empirically show diﬀerent scenarios for
unsupervised domain-speciﬁc neural machine

Therefore, the two languages must use the sametranslation. Based on these scenarios, they show

vocabulary. The entire training of UNMT needs toand analyze several potential solutions including

consider back-translation between the two languages

batch weighting, data selection, and ﬁne tuning

and their respective denoising processing.

methods, to improve the performances of domain-For UNMT, addition

speciﬁc UNMT systems [29].

― 1070 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Eﬃciency: Compared with NMT, the training
time of UNMT increased rapidly.
learning

sharing

latent[7] C. Xu, T. Qin, G. Wang, and T. Liu,In addition,

representations“Polygon-net: A general framework for jointly

ties

boosting multiple unsupervised neural machine

the performance of both translation directions,especially for distant language pairs, while denoising
dramatically delays convergence by continuouslytranslation models,” in IJCAI, 2019.
[8] J. Wu, X. Wang, and W. Y. Wang, “Extractand edit: An alternative to back-translation

modifying the training data. Eﬃcient training offor unsupervised neural machine translation,”

UNMT is also an issue that needs to be solved.

in Proceedings of the 2019 Conference of theNorth American Chapter of the Association for

5

Computational Linguistics: Human Language

Conclusion

Technologies,

In this paper, we ﬁrst survey the the background

Volume 1 (Long and ShortPapers), Minneapolis, Minnesota, Jun. 2019,

and the latest progress of UNMT. We then examinepp. 1173–1183.

a number of challenges to UNMT and and analyze[9] H. Sun, R. Wang, K. Chen, M. Utiyama,

the future trend of UNMT.

E. Sumita,bilingual

参考文献

and T. Zhao,

wordembedding“Unsupervisedagreement

forunsupervised neural machine translation,” in
ACL, Florence, Italy, Jul. 2019, pp. 1235–1245.

[1] M. Artetxe, G. Labaka, E. Agirre, and K. Cho,
“Unsupervised neural machine translation,” in[10] Y. Leng, X. Tan, T. Qin, X.-Y. Li, and T.-Y.
Liu, “Unsupervised pivot translation for distantICLR, 2018.

languages,” in ACL, 2019.
[2] G. Lample, A. Conneau, L. Denoyer, andM. Ranzato, “Unsupervised machine translation[11] G. Lample and A. Conneau, “Cross-lingual
language model pretraining,”

using monolingual corpora only,” in ICLR, 2018.

CoRR, vol.

abs/1901.07291, 2019.
[3] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio,
denoising[12] K. Song, X. Tan, T. Qin, J. Lu, and T. Liu,

autoencoders: Learning useful representations“MASS: masked sequence to sequence pre-in a deep network with a local denoisingtraining for language generation,” in ICML,criterion,”

2019.

andP.

Manzagol,

Journal“StackedofMachine

Learning

Research, vol. 11, pp. 3371–3408, 2010.
[4] R. Sennrich,

B. Haddow,[13] M. Artetxe,and A. Birch,

and E. Agirre,

in EMNLP, 2018.

“Improving neural machine translation models
with monolingual data,” in ACL, 2016.

G. Labaka,“Unsupervised statistical machine translation,”[14] B. Marie and A. Fujita, “Unsupervised neural“Unsupervised neural machine translation withmachine translation initialized by unsupervised
statistical machine translation,” CoRR, vol.

weight sharing,” in ACL, 2018.

abs/1810.12703, 2018.

[5] Z. Yang, W. Chen, F. Wang, and B. Xu,[6] G. Lample, M. Ott, A. Conneau, L. Denoyer,[15] S. Ren, Z. Zhang, S. Liu, M. Zhou, and S. Ma,and M. Ranzato, “Phrase-based & neural
unsupervised machine translation,” in EMNLP,“Unsupervised neural machine translation with
SMT as posterior regularization,” in AAAI,

2018.

2019.

― 1071 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

[16] M. Artetxe, G. Labaka, and E. Agirre, “Anmachine translation,” in Proceedings of theeﬀective approach to unsupervised machine

55th Annual Meeting of the Association fortranslation,” in ACL, Jul. 2019.

Computational Linguistics (Volume 1:[17] B. Marie,

H. Sun,R. Wang,Papers), Vancouver, Canada, 2017, pp. 698–

K. Chen,707. [Online]. Available:

A. Fujita, M. Utiyama, and E. Sumita, “NICT’s[26] K. Chen, R. Wang, M. Utiyama, L. Liu,translation task,” in WMT, 2019.

A. Tamura, E. Sumita, and T. Zhao, “Neuralmachine translation with source dependency[18] L. Barrault, O. Bojar, M. R. Costa-jussÃ,C.

Federmann,M.

Fishel,http://aclweb.org/anthology/P17-1065

unsupervised neural and statistical machinetranslation systems for the WMT19 news

LongY.

representation,” in Proceedings of the 2017

Graham,Conference on Empirical Methods in NaturalB. Haddow, M. Huck, P. Koehn, S. Malmasi,

Language Processing, Copenhagen, Denmark,C. Monz, M. MÃller, S. Pal, M. Post, andSeptember 2017,

M. Zampieri, “Findings of the 2019 conferenceAvailable: https://www.aclweb.org/anthology/

on machine translation (wmt19),” in ACL, 2019.
[19] F. Hill, K. Cho, and A. Korhonen, “Learning
distributed representations of sentences frompp. 2846–2852. [Online].

D17-1304[27] K. Chen, R. Wang, M. Utiyama, E. Sumita, andT. Zhao, “Syntax-directed attention for neuralunlabelled data,” in NAACL, 2016.

machine translation,” in Proceedings of theT. Liu, and W. Ma, “Dual learning for machine

AAAI Conference on Artiﬁcial Intelligence, NewOlreans, LA, 2018, pp. 4792–4799. [Online].

translation,” in NIPS, 2016.

Available:[20] D. He, Y. Xia, T. Qin, L. Wang, N. Yu,

https://www.aaai.org/ocs/index.

php/AAAI/AAAI18/paper/view/16060/16008[21] A. Eriguchi, K. Hashimoto, and Y. Tsuruoka,“Tree-to-sequence attentional neural machinetranslation,” in ACL, 2016.

[28] C. Ma, A. Tamura, M. Utiyama, E. Sumita,
and T. Zhao, “Improving neural machinetranslation with neural syntactic distance,”[22] A. Eriguchi,

Y. Tsuruoka,

and K. Cho,in Proceedings of the 2019 Conference of the“Learning to parse and translate improves

North American Chapter of the Association forneural machine translation,” in ACL, 2017.

Computational Linguistics: Human Language
Technologies,[23] H. Chen, S. Huang, D. Chiang, and J. Chen,

Volume 1 (Long and Short“Improved neural machine translation with aPapers). Minneapolis, Minnesota: Association

syntax-aware encoder and decoder,” in ACL,

for Computational Linguistics,2017.

pp. 2032–2037. [Online]. Available:

Jun. 2019,https://www.aclweb.org/anthology/N19-1205[24] J. Li, D. Xiong, Z. Tu, M. Zhu, M. Zhang,and G. Zhou, “Modeling source syntax for[29] H. Sun, R. Wang, K. Chen, M. Utiyama,neural machine translation,” in Proceedings ofE. Sumita,

the 55th Annual Meeting of the Association forstudy of domain adaptation for unsupervised

Computational Linguistics (Volume 1:

neuralLong

Papers), Vancouver, Canada, 2017, pp. 688–697. [Online]. Available: http://aclweb.org/and T. Zhao,machinetranslation,”“An empiricalCoRR,vol.

abs/1908.09605, 2019.

anthology/P17-1064[25] S. Wu, D. Zhang, N. Yang, M. Li, andM. Zhou,“Sequence-to-dependency neural― 1072 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.