PACLIC 32 - WAT 2018

English-Myanmar NMT and SMT with Pre-ordering:
NICT’s Machine Translation Systems at WAT-2018
Rui Wang
Chenchen Ding
Masao Utiyama
Eiichiro Sumita
National Institute of Information and Communications Technology (NICT)
3-5 Hikaridai, Seika-cho, Souraku-gun, Kyoto, 619-0289, Japan{wangrui, chenchen.ding, mutiyama, eiichiro.sumita}@nict.go.jp

Abstract
This paper presents the NICT’s participation(team ID: NICT) in the 5th Workshop
on Asian Translation (WAT-2018) sharedtranslation task, specifically Myanmar(Burmese) - English task in both translationdirections. We built state-of-the-art neuralmachine translation (NMT) as well asphrase-based statistical machine translation(PBSMT) systems for these tasks. Our NMTsystems were trained with the Transformer
architecture on the provided parallel data.
Pre-ordering technology is adopted to both
NMT and PBSMT. Our NMT systems rank
the first in English-to-Myanmar and thesecond in Myanmar-to-English according tothe official human evaluation.

1

2

Data Preprocessing

As parallel data to train our systems, we used all theprovided parallel data for all our targeted translation
directions, including the training corpus “ALT” and“UCSY” and the“ALT” dev/test data. The statistics
of our preprocessed parallel data are illustrated inTable 1.

Table 1: Statistics of our preprocessed parallel data.

Introduction

This paper describes the machine translation
systems1 built for National Institute of Information
and Communications Technology (NICT)’s
participation in the the 5th Workshop on AsianTranslation (WAT-2018) translation task (Nakazawaet al., 2018), specifically Myanmar (My) - English(En) for both translation directions. All of oursystems are constrained, i.e., we used only theparallel adata provided by the organizers to train
and tune our systems.
The remainder of this paper is organized asfollows.
In Section 2, we present the data
preprocessing. In Section 3, we introduce the details
1of our NMT and SMT systems with pre-ordering
technology. Empirical results obtained with oursystems are analyzed in Section 4 and we conclude
this paper in Section 5.

This system is based on our WMT-2018 system (Marie etal., 2018).

Corpus#lines#tokens (My/En)

train(ALT)train(UCSY)
dev(ALT)test(ALT)

17.9K
208.6K
0.9K
1.0K

1.0M / 410.2K
5.8M / 2.6M
57.4K / 22.1K
58.3K / 22.7K

We used Moses tokenizer and truecaser forEnglish. The truecaser was trained on the English
data, after tokenization. For Myanmar, we used
the original tokens. For cleaning, we only applied
the Moses script clean-n-corpus.perl toremove lines in the parallel data containing morethan 80 tokens and replaced characters forbidden byMoses.

972
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Table 2: Results (BLEU-cased and official human evaluation) of our MT systems on the test set. We only submitted
the top 2 systems on BLEU for human evaluation.

System

My→En (BLEU)

En→My (BLEU)

My→En (Human)

En→My (Human)

Moses
Moses Pre-order

9.44
N/A

25.55
25.75

N/A
N/A

N/A
N/A

MarianMarian Pre-order

16.32
N/A

26.02
23.79

7.250
N/A

42.50
N/A

Marian ensemble

20.79

29.89

20.50

61.00

3MT Systems

3.1

NMT

To build competitive NMT systems, we choseto rely on the Transformer architecture (Vaswaniet al., 2017) since it has been shown tooutperform, in quality and efficiency, the twoother mainstream architectures for NMT known
as deep recurrent neural network (deep-RNN) andconvolutional neural network (CNN). We chose
Marian2 (Junczys-Dowmunt et al., 2018) to train
our NMT systems since it supports many stateof-the-art features and is one of the fastest NMT
frameworks publicly available.3
All our NMT systems were consistently trainedon 4 GPUs,4 with the following parameters forMarian:--type transformer --max-length
80 --dec-depth 6 --normalize 1--save-freq 5000 --workspace
8000 --disp-freq 500--beam-size 12 --overwrite--cost-type ce-mean-words--keep-best --enc-depth 6--transformer-dropout 0.1--valid-mini-batch 16 --valid-freq
5000 --learn-rate 0.0003--lr-decay-inv-sqrt 16000--lr-warmup 16000 --lr-report--sync-sgd --devices 0 1 2
3 --dim-vocabs 50000 50000--exponential-smoothing2

https://marian-nmt.github.io/, version 1.4.0
It is fully implemented in pure C++ and supports multiGPU training.
4NVIDIA® Tesla® P100 16Gb.
3--optimizer-params 0.9 0.98 1e-09--clip-norm 5 --tied-embeddings--mini-batch-fit --early-stopping
5 --label-smoothing 0.1--valid-metrics ce-mean-words
perplexity translation
We performed NMT decoding with an ensembleof a total of 4 models according to the best BLEU
(Papineni et al., 2002) and the perplexity scores,
produced by 4 independent training runs.
3.2

SMT

We also trained phrase-based SMT systems usingMoses.
Word alignments and phrase tables
were trained on the tokenized parallel data usingmgiza.
Source-to-target and target-to-source
word alignments were symmetrized with thegrow-diag -final-and heuristic. We simply
trained regular phrase-based models and used thedefault distortion limit of 6. We trained two 5gram language models on the entire target sideof the parallel data, with SRILM (Stolcke, 2002).
To tune the SMT model weights, we used MERT
(Och, 2003) and selected the weights giving the best
BLEU score on the development data.
3.3

Pre-ordering

We also tried a classic pre-ordering method forEnglish-to-Myanmar translation task. Specifically,
the dependency-based head finalization in Ding etal. (2014) is exactly reproduced in our experiment.
The source English part is pre-ordered before being
input into NMT and SMT systems.

973
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

4Results

Our systems are evaluated on the ALT test set andthe results5 are shown in Table 2. Our observations
from are as follows:1) Our NMT (Marian) system performed muchbetter than SMT (Moses) system in My-to-En. Thatis, nearly 7 BLEU scores. However, there is no
significant difference in En-to-My.
2) Pre-ordering did not show significant
improvement in the NMT systems, while it
improved translation quality by 0.2 BLEU points inthe SMT system.
3) Ensemble decoding significantly outperformed
decoding with a single NMT model.

5Conclusion

We presented in this paper the NICT’s participationin the WAT-2018 shared translation task. Our
primary NMT submissions to the task performed
the best submitted system according to the officialhuman evaluation. Our results also confirmed
the slight positive impact of using pre-ordering in
PBSMT.

References

Toshiaki Nakazawa, Shohei Higashiyama, ChenchenDing, Raj Dabre, Anoop Kunchukuttan, Win Pa Pa,
Isao Goto, Hideya Mino, Katsuhito Sudoh, and Sadao
Kurohashi. 2018. Overview of the 5th workshop on
asian translation. In Proceedings of the 5th Workshop
on Asian Translation (WAT2018), Hong Kong, China,December.
Franz Josef Och. 2003. Minimum error rate training
in statistical machine translation. In Proceedingsof the 41st Annual Meeting of the Association for
Computational Linguistics, pages 160–167, Sapporo,Japan.
Kishore Papineni, Salim Roukos, Todd Ward, andWei-Jing Zhu.
2002.
Bleu: a method forautomatic evaluation of machine translation.
In Proceedings of 40th Annual Meeting of theAssociation for Computational Linguistics, pages
311–318, Philadelphia, Pennsylvania, USA, July.
Association for Computational Linguistics.
Andreas Stolcke. 2002. Srilm-an extensible language
modeling toolkit.
In Proceedings International
Conference on Spoken Language Processing, pages
257–286, Seattle.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In I. Guyon, U. V. Luxburg, S. Bengio,
H. Wallach, R. Fergus, S. Vishwanathan, andR. Garnett, editors, Advances in Neural Information
Processing Systems 30.

Chenchen Ding, Ye Kyaw Thu, Masao Utiyama, Andrew
Finch, and Eiichiro Sumita.
2014.
Empirical
dependency-based head finalization for statisticalchinese-, english-, and french-to-myanmar (burmese)machine translation. In Proceedings of IWSLT, pages
184–191.
Marcin Junczys-Dowmunt, Roman Grundkiewicz,Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, TomNeckermann, Frank Seide, Ulrich Germann, AlhamFikri Aji, Nikolay Bogoychev, André F. T. Martins,and Alexandra Birch. 2018. Marian: Fast neuralmachine translation in C++. In Proceedings of ACL
2018, System Demonstrations, Melbourne, Australia.
Benjamin Marie, Rui Wang, Atsushi Fujita, Masao
Utiyama, and Eiichiro Sumita. 2018. NICT’s neuraland statistical machine translation systems for theWMT18 news translation task. In Proceedings ofthe Third Conference on Machine Translation, Volume
2: Shared Task Papers, Brussels, Belgium, October.
Association for Computational Linguistics.
5The results of BLEU are based on our own evaluation. For
the official results, please refer to http://lotus.kuee.
kyoto-u.ac.jp/WAT/evaluation/index.html.

974
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors