English-Myanmar Supervised and Unsupervised NMT:
NICT’s Machine Translation Systems at WAT-2019
Rui Wang1∗ , Haipeng Sun2,1∗, Kehai Chen1 ,Chenchen Ding1 , Masao Utiyama1 , and Eiichiro Sumita1
1National Institute of Information and Communications Technology (NICT)
3-5 Hikaridai, Seika-cho, Souraku-gun, Kyoto, 619-0289, Japan
2
Harbin Institute of Technology, Harbin, China{wangrui, sun.haipeng, khchen, chenchen.ding, mutiyama, eiichiro.sumita}@nict.go.jp

Abstracttranslation directions, including the training
corpus “ALT” and “UCSY”, and the “ALT”
dev/test data. The statistics of our preprocessedparallel data are illustrated in Table 1.

This paper presents the NICT’s participation(team ID: NICT) in the 6th Workshop on AsianTranslation (WAT-2019) shared translationtask, specifically Myanmar (Burmese) English task in both translation directions.
We built neural machine translation (NMT)
systems for these tasks. Our NMT systems
were trained with language model pretraining.
Back-translation technology is adopted toNMT. Our NMT systems rank the thirdin English-to-Myanmar and the second in
Myanmar-to-English according to BLEU
score.

1#lines#tokens (My/En)

train(ALT)train(UCSY)
dev(ALT)test(ALT)

17.9K
208.6K
0.9K
1.0K

1.0M / 410.2K
5.8M / 2.6M
57.4K / 22.1K
58.3K / 22.7K

Table 1: Statistics of our preprocessed parallel data.

In WAT2019, two Myanmar monolingual
corpora consist of Myanmar Wikipedia andMyanmar Common Crawl.
For English
monolingual corpus, we randomly extracted
10 million sentences from WMT monolingualNews Crawl datasets.2 The statistics of ourpreprocessed monolingual data are illustrated inTable 2.

Introduction

This paper describes the neural machinetranslation (NMT) systems1 built for National
Institute of Information and Communications
Technology (NICT)’s participation in the the
6th Workshop on Asian Translation (WAT2019) translation task(Nakazawa et al., 2019),
specifically Myanmar (My) - English (En) for
both translation directions.
The remainder of this paper is organized asfollows. In Section 2, we present the data
preprocessing. In Section 3, we introduce thedetails of our NMT systems. Empirical results
obtained with our systems are analyzed in Section
4 and we conclude this paper in Section 5.

2Corpus

Corpus#lines#tokens

MyEn

6.7M
10.0M

125.6M
229.8M

Table 2: Statistics of our preprocessed monolingualdata.

We used Moses tokenizer and truecaser forEnglish. The truecaser was trained on the English
data, after tokenization. For Myanmar, we used
the original tokens. For cleaning, we only applied
the Moses script clean-n-corpus.perl toremove lines in the parallel data containing morethan 80 tokens and replaced characters forbidden
by Moses.

Data Preprocessing

As parallel data to train our systems, we used
all the provided parallel data for all our targeted∗

Rui and Haipeng have equal contribution to this paper.
This work was conductd when Haipeng visited NICT as aninternship student.
1This system is based on our WMT-2019 system (Marieet al., 2019).

2

http://data.statmt.org/news-crawl/

90
Proceedings of the 6th Workshop on Asian Translation, pages 90–93
Hong Kong, China, November 4, 2019. c 2019 Association for Computational Linguistics

3

MT Systems--lgs ’en-my’ --encoder onlyfalse --emb dim 1024 --n layers
6 --n heads 8 --dropout
0.1 --attention dropout
0.1 --gelu activation
true --tokens per batch2000 --batch size 32--bptt 256 --optimizer
adam inverse sqrt,beta1=0.9,beta2=0.98,lr=0.0001--eval bleu true

To build competitive NMT systems, we choseto rely on the Transformer architecture (Vaswaniet al., 2017) since it has been shown to outperform,
in quality and efficiency, the two other mainstreamarchitectures for NMT known as deep recurrent
neural network (deep-RNN) and convolutional
neural network (CNN). We chose to rely on
the Transformer-based NMT initialized by apretrained cross-lingual language model (Lampleand Conneau, 2019) to train our NMT systems
since it had been shown to be efficient in thelow-resource language pairs. In order to limit
the size of the vocabulary of the NMT model,
we segmented tokens in the training data into
sub-word units via byte pair encoding (BPE)(Sennrich et al., 2016b). We determined 60k BPE
operations jointly on the training data for English
and Myanmar, and used a shared vocabulary forboth languages with 60k tokens based on BPE.
3.1

Table 4: Parameters for training NMT.

3.3

We also tried back-translation method (Sennrichet al., 2016a) to make use of monolingual corpora
for English-to-Myanmar translation task. Parallel
data for training NMT can be augmented withsynthetic parallel data, generated through backtranslation, to significantly improve translation
quality. For back-translation generation, we used
an NMT system, trained on the parallel data
provided by the organizers, to translate target
monolingual sentences into the source language
to generate pseudo parallel corpora. Then, thepseudo parallel corpora were simply mixed withthe original parallel data to train from scratch a
new source-to-target NMT system.

TLM

Before training NMT, we used all training corpora
including parallel data and monolingual data totrain a translation language model (TLM) using
XLM3 in order to pretrain the NMT model on 8GPUs4 . The parameters for training the language
model were set as listed in Table 3.
--lgs ’en-my’ --mlm steps’en,my,en-my,my-en’--emb dim 1024 --n layers
6 --n heads 8 --dropout
0.1 --attention dropout
0.1 --gelu activation true--batch size 32 --bptt 256--optimizer adam,lr=0.0001

3.4UNMT

To the best of our knowledge, unsupervised NMT(UNMT) (Artetxe et al., 2018; Lample et al.,2018a; Yang et al., 2018; Lample et al., 2018b;
Sun et al., 2019; Lample and Conneau, 2019)has achieved remarkable results on some similar
language pairs. To obtain a better picture of thefeasibility of UNMT, we also set up a UNMT
system for one truly low-resource and distantlanguage pair: En-My. We tried to train aTransformer-based UNMT model that relies solely
on monolingual corpora, with the pre-trained
cross-lingual language model using XLM toolkit.
Note that this cross-lingual language model wastrained solely on monolingual corpora shown inSection 2.
We used these monolingual corpora to train theUNMT model for 50000 iterations. The En-My
UNMT system was trained on 8 GPUs, with theparameters listed in Table 6.

Table 3: Parameters for training TLM.

3.2

Back-translation

NMT

We trained a Transformer-based NMT model withthe pre-trained TLM using XLM toolkit. Our NMTsystem was consistently trained on 8 GPUs, withthe following parameters listed in Table 4.
We performed NMT decoding with a single
model according to the best BLEU (Papineni et al.,2002) and the perplexity scores.
3

https://github.com/facebookresearch/
XLM
4
NVIDIA @ Tesla @ V100 32Gb.

91

Systems
UNMT
NMT
NMT
NMT+TLM
NMT+TLM
NMT+TLM+back-translation

ALT

UCSY

X
X
X
X
X

X
X
X
X

MONO

My-En

En-My

X

0.81
8.06
14.97
18.42
21.33
29.89

0.31
10.50
14.15
16.12
19.73
19.01

X
X

Table 5: Results (BLEU-cased) of our MT systems on the test set. ALT denotes that ALT training data was usedin this system; UCSY denotes that UCSY training data was used in this system; MONO denotes monolingual
training data was used in this system. +TLM denotes that language model pretraining was used in this system;+back-translation denotes that back-translation was used in this system.

low-resource machine translation.
3) For My-En translation direction, backtranslation could further improve translation
performance, achieving 8 BLEU scoresimprovement.
However, back-translation for
En-My translation direction was unable toimprove or even harm the NMT performancesince the My monolingual data was noisy.

--lgs ’en-my’ --ae steps’en,my’ --bt steps’en-my-en,my-en-my’--word shuffle 3--word dropout 0.1--word blank 0.1 --lambda ae’0:1,100000:0.1,300000:0’--encoder only false--emb dim 1024 --n layers
6 --n heads 8 --dropout
0.1 --attention dropout
0.1 --gelu activation
true --tokens per batch2000 --batch size 32--bptt 256 --optimizer
adam inverse sqrt,beta1=0.9,
beta2=0.98,lr=0.0001--eval bleu true5

We presented in this paper the NICT’s
participation in the WAT-2019 shared translation
task. Our primary NMT submissions to the task
performed the third in English-to-Myanmar andthe second in Myanmar-to-English according
to BLEU score. Our results also confirmed thepositive impact of language model pretraining
in NMT. Moreover, our results for UNMT
highlighted that unsupervised machine translationis still very far from exploitable for low-resource
distant language pairs.

Table 6: Parameters for training UNMT.

4Conclusion

Results
References

Our systems are evaluated on the ALT test set andthe results5 are shown in Table 5. Our observations
from are as follows:
1) The results of UNMT are very low,highlighting that UNMT is still very far from
exploitable for low-resource distant languagepairs.
2) Language model pretraining showedsignificant improvement in the NMT systems forboth translation directions. This demonstrates
that language model pretraining is effective forMikel Artetxe, Gorka Labaka, Eneko Agirre, andKyunghyun Cho. 2018.
Unsupervised neuralmachine translation. In ICLR, Vancouver, Canada.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining.
CoRR,abs/1901.07291.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer,
and Marc’Aurelio Ranzato. 2018a. Unsupervisedmachine translation using monolingual corpora
only. In ICLR, Vancouver, Canada.
Guillaume Lample, Myle Ott, Alexis Conneau,
Ludovic Denoyer, and Marc’Aurelio Ranzato.
2018b.
Phrase-based & neural unsupervisedmachine translation. In EMNLP, pages 5039–5049,
Brussels, Belgium.

5

The results of BLEU are based on our own
evaluation.
For the official results, please refer tohttp://lotus.kuee.kyoto-u.ac.jp/WAT/evaluation/index.html.

92

Benjamin Marie, Haipeng Sun, Rui Wang, Kehai
Chen, Atsushi Fujita, Masao Utiyama, and Eiichiro
Sumita. 2019. NICT’s unsupervised neural andstatistical machine translation systems for theWMT19 news translation task. In WMT, pages 294–301, Florence, Italy.
Toshiaki Nakazawa, Chenchen Ding, Raj Dabre,Hideya Mino, Isao Goto, Win Pa Pa, Nobushige Doi,
Yusuke Oda, Anoop Kunchukuttan, ShantipriyaParida, Ondej Bojar, and Sadao Kurohashi.
2019. Overview of the 6th workshop on Asiantranslation. In Proceedings of the 6th Workshop
on Asian Translation, Hong Kong. Association for
Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic
evaluation of machine translation. In ACL, pages
311–318, Philadelphia, Pennsylvania, USA.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016a.
Improving neural machine translation
models with monolingual data. In ACL, pages 86–96, Berlin, Germany.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016b. Neural machine translation of rare words
with subword units. In ACL, pages 1715–1725,Berlin, Germany.
Haipeng Sun, Rui Wang, Kehai Chen, Masao
Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2019.
Unsupervised bilingual word embedding agreement
for unsupervised neural machine translation. InACL, pages 1235–1245, Florence, Italy.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In NIPS, pages 5998–6008, Long Beach,CA, USA.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu.
2018. Unsupervised neural machine translation withweight sharing. In ACL, pages 46–55, Melbourne,
Australia.

93