NAVER Machine Translation System for WAT 2015
Hyoung-Gyu Lee1 , Jaesong Lee1 , Jun-Seok Kim1 and Chang-Ki Lee2
NAVER LABS, NAVER Corp., Seongnam-si, Gyeonggi-do, South Korea
2
Kangwon National University, Chuncheon-si, Gangwon-do, South Korea{hg.lee,jaesong.lee,jun.seok}@navercorp.com
leeck@kangwon.ac.kr

1

Abstract

2.2

We used Moses tokenizer and Berkeley constituency parser1 for tokenizing and parsing an
English sentence. We used our own Japanese tokenizer and part-of-speech tagger for tokenizingand tagging a Japanese sentence. After running thetokenizer and the tagger, we make a token fromconcatenation of a word and its part-of-speech.

In this paper, we describe NAVER machine translation system for English toJapanese and Korean to Japanese tasks at
WAT 2015. We combine the traditional
SMT and neural MT in both tasks.

1

Introduction

This paper explains the NAVER machine translation system for the 2nd Workshop on Asian Translation (WAT 2015) (Nakazawa et al., 2015). We
participate in two tasks; English to Japanese (EnJa) and Korean to Japanese (Ko-Ja).
Our system is a combined system of traditional
statistical machine translation (SMT) and neuralmachine translation (NMT). We adopt the tree-tostring syntax-based model as En-Ja SMT baseline,
while we adopt the phrase-based model as Ko-Ja.
We propose improved SMT systems for each task
and an NMT model based on the architecture using recurrent neural network (RNN) (Cho et al.,2014; Sutskever et al., 2014).
We give detailed explanations of each SMT system in section 2 and section 3. We describe ourNMT model in section 4.

2

English to Japanese

2.1

Training data

Language Analyzer

2.3

Tree-to-string Syntax-based SMT

To determining the baseline model, we first performed comparative experiments with the phrasebased, hierarchical phrase-based and syntax-basedmodels. As a result, we chose the tree-to-string
syntax-based model.
The SMT models that consider source syntax
such as tree-to-string and forest-to-string broughtout better performance than the phrase-based andhierarchical phrase-based models in the WAT
2014 En-Ja task.
The tree-to-string model was proposed byHuang (2006) and Liu (2006). It utilizes the constituency tree of source language to extract translation rules and decode a target sentence. The translation rules are extracted from a source-parsed andword-aligned corpus in the training step. We use
synchronous context free grammar (SCFG) rules.
In addition, we used a rule augmentation
method which is known as syntax-augmented
machine translation (Zollmann and Venugopal,2006). Because the tree-to-string SMT makessome constraints on extracting rules by considering syntactic tree structures, it usually extracts
fewer rules than hierarchical phrase-based SMT(HPBSMT) (Chiang, 2005). Thus it is required toaugment tree-to-string translation rules. The ruleaugmentation method allows the training system
to extract more rules by modifying parse trees.
Given a parse tree, we produce additional nodes

We used 1 million sentence pairs that are contained in train-1.txt of ASPEC-JE corpus
for training the translation rule tables and NMT
models. We also used 3 million Japanese sentences that are contained in train-1.txt,train-2.txt,train-3.txt of ASPEC-JE
corpus for training the 5-gram language model.
We also used 1,790 sentence pairs of dev.txt
for tuning the weights of each feature of SMT linear model and as validation data of neural network.
We filtered out the sentences that have 100 or more
tokens from training data.

1
https://github.com/slavpetrov/berkeleyparser

69
Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 69‒73,
Kyoto, Japan, 16th October 2015.
2015 Copyright is held by the author(s).

by combining any pairs of neighboring nodes, notonly children nodes, e.g. NP+VP. We limit themaximum span of each rule to 40 tokens in therule extraction process. The tree-to-string decoder
use a chart parsing algorithm with cube pruning
proposed by Chiang (2005).
Our En-Ja SMT system was developed by using
the open source SMT engines; Moses and Giza++.
Its other specifications are as follows:

the shortest edit distance between an original word
and its suggestion word.

• Good-Turing discounting for smoothing
probabilities• Minimum Error Rate Training (MERT) fortuning feature weights

3.1

Training data

3.2

Language Analyzer

We used MeCab-ko3 for tokenizing a Korean sentence. We used Juman4 for tokenizing a Japanesesentence. We did not perform part-of-speech tagging for both languages.

• Cube-pruning-pop-limit = 3000
Handling Out-of-Vocabulary

In order to handle out-of-vocabulary (OOV)
words, we use two techniques; hyphen word split
and spell error correction. The former is to split aword with hyphen (-) to two separate tokens before running the language analyzer. The latter is toautomatically detect and correct spell errors in an
input sentence. We give a detailed description ofspell error correction in section 2.4.1.

3.3

Phrase-based SMT

As in the En-Ja task, we first performed comparative experiments with the phrase-based and hierarchical phrase-based models, and then adopt thephrase-based model as our baseline model.
For the Ko-Ja task, we develop two phrasebased systems; word-level and character-level. We
use word-level tokenization for the word-basedsystem. We found that setting the distortion limit
to zero yields better translation in aspect of both
BLEU and human evaluation. We use the 5-gram
language model. We use character-level tokenization for character-based system. We use the 10gram language model and set the maximum phrase
length to 10 in the phrase pair extraction process.
We found that the character-level system does
not suffer from tokenization error and out-ofvocabulary issue. The JPO corpus contains manytechnical terms and loanwords like chemical compound names, which are more inaccurately tokenized and allow a lot of out-of-vocabulary tokens to be generated. Since Korean and Japaneseshare similar transliteration rules for loanwords,
the character-level system can learn translation ofunseen technical words. It generally produces better translations than a table-based transliteration.

2.4.1 English Spell Correction
It is not easy to translate a word including errata,
because the erroneous word has only a slim chanceof occurrence in the training data. We discovered
a lot of spell errors among OOV words that appear in English scientific text. We introduce English spell correction for reducing OOV words in
input sentences. We developed our spell corrector
by using Aspell2 .
For detecting a spell error, we skip words thathave only capitals, numbers or symbols, because
they are likely to be abbreviations or mathematic
expressions. Then we regard words detected byAspell as spell error words.
For correcting spell error, we use only top-3
suggestion words from Aspell. We find that a large
gap between an original word and its suggestion
word makes wrong correction. To avoid excessive
correction, we introduce a gap thresholding technique, that ignores the suggestion word that has
3 or longer edit distance and selects one that has
2

Korean to Japanese

We used 1 million sentence pairs that are contained in JPO corpus for training phrase tables andNMT models. We also used Japanese part of thecorpus for training the 5-gram language model.
We also used 2,000 sentence pairs of dev.txt
for tuning the weights of each feature of SMT linear model and as validation data of neural network.
We did not filter out any sentences.

• Grow-diag-final-and word alignment heuristic

2.4

3

3
https://bitbucket.org/eunjeon/
mecab-ko/4
http://nlp.ist.i.kyoto-u.ac.jp/EN/
index.php?JUMAN

http://aspell.net/
70Moreover, we tested jamo-level tokenization5
for Korean text, however, the preliminary test did
not produce effective results.
We also investigated a parentheses imbalanceproblem. We solved the problem by filtering out
parentheses-imbalanced translations from the nbest results. We found that the post-processing
step can improve the BLEU score with low orderlanguage models, but cannot do with high orderlanguage models. We do not use the step for finalsubmission.
To boosting the performance, we combine theword-level phrase-based model (Word PB) and thecharacter-level phrase-based model (Char PB). If
there are one or more OOV words in an input sentence, our translator choose the Char PB model,otherwise, the Word PB model.
Our Ko-Ja SMT system was developed by using
the open source SMT engines; Moses and Giza++.
Its other specifications are as follows:

system removes the need to store explicit phrase
tables and language models. Lastly, the decoderof an NMT system is easy to implement. Despite
these advantages and promising results, NMT has
a limitation in handling a larger target vocabulary, as the complexity of training and decoding
increases proportionally to the number of targetwords.
In this paper, we propose a new approach to
avoid the large target vocabulary problem by preprocessing the target word sequences, encoding
them as a longer character sequence drawn froma small character vocabulary. The proposed approach removes the need to replace rare words
with the unknown word symbol. Our approach issimpler than other methods recently proposed toaddress the same issue (Luong et al., 2015; Jean etal., 2015).
4.1

In this paper, we use our in-house software of
NMT that uses an attention mechanism, as recently proposed by Bahdanau et al. (2015). Theencoder of NMT is a bi-directional recurrent neural network such that

• Grow-diag-final-and word alignment heuristic• Good-Turing discounting for smoothing
probabilities• Minimum Error Rate Training (MERT) fortuning feature weights

4

Model

Neural Machine Translationht = [h~t ; ~ht ](1)

h~t = fGRU (Ws we xt , h~t+1 )(2)~ht = fGRU (Ws we xt , ~ht−1 )(3)where ht is a hidden state of the encoder, xt isa one-hot encoded vector indicating one of thewords in the source vocabulary, Ws we is a weightmatrix for the word embedding of the source language, and fGRU is a gated recurrent unit (GRU)(Cho et al., 2014).
At each time, the decoder of NMT computes
the context vector ct as a convex combination of
the hidden states (h1 ,. . . ,hT ) with the alignment
weights α1 ,. . . ,αT :Neural machine translation (NMT) is a new approach to machine translation that has shownpromising results compared to the existing approaches such as phrase-based statistical machinetranslation (Cho et al., 2014; Sutskever et al.,2014; Bahdanau et al., 2015). An NMT system is asingle neural network that reads a source sentence
and generates its translation. Using the bilingualcorpus, the whole neural network is jointly trained
to maximize the conditional probability of a correct translation given a source sentence. NMT has
several advantages over the existing statistical machine translation systems such as the phrase-basedsystem. First, NMT uses minimal domain knowledge. Second, the NMT system is jointly trained
to maximize the translation performance, unlike
the existing phrase-based system which consists ofmany separately trained features. Third, the NMT

ct =TX

αti hi(4)i=1exp(etj )αti = PT
j=1 exp(etj )(5)

eti = fF F N N (zt−1 , hi , yt−1 )(6)where fF F N N is a feedforward neural networkwith a single hidden layer, zt−1 is a previous hidden state of the decoder, and yt−1 is a previous
generated target word (one-hot encoded vector).

5

Jamo is Korean alphabet letters which represent consonants and vowels. A Korean character can be usually separated to three jamoes; choseong, jungseong and jongseong.
71SYS
Tree-to-string SB+ Rule augmentation+ Parameter modification+ OOV handling

A new hidden state zt of the decoder which uses
GRU is computed based on zt−1 , yt−1 , and ct :zt = fGRU (yt−1 , zt−1 , ct )(7)

The probability of the next target word yt is thencomputed byp(yt |y<t , x) = ytT fsof tmax {Wz 0 y zt0+ Wzy zt + Wcy ct+ Wyy (Wt we yt−1 ) + by }(8)

zt0 = fReLU (Wzz 0 zt )

(9)

SYS
Word PBChar PB
Word PB + Char PB#Rules
57M
55M
57M & 55M

section 2.4 also caused a positive effect as shownin the last row of the table.
The decoding time of the rule-augmented treeto-string SMT is about 1.3 seconds per a sentencein our 12-core machine. Even though it is not aterrible problem, we are required to improve thedecoding speed by pruning the rule table or using
the incremental decoding method (Huang and Mi,2010).

Settings

5.2

Ko-Ja SMT

Table 2 shows the evaluation results of our Ko-Ja
traditional SMT system. We obtained the best result in the combination of two phrase-based SMTsystems.
5.3

NMT

Table 3 shows effects of our NMT model. “Human” indicates the pairwise crowdsourcing evaluation scores provided by WAT 2015 organizers. In the table, “T2S/PBMT only” is the final T2S/PBMT systems shown in section 5.1
and section 5.2. “NMT only” is the system using only RNN encoder-decoder without any traditional SMT methods. The last row is the combined system that reranks T2S/PBMT n-best translations by NMT. Our T2S/PBMT system outputs
100,000-best translations in En-Ja and 10,000best translations in Ko-Ja. The final output is 1best translation selected by considering only NMTscore.
NMT outperforms the traditional SMT in EnJa, while it does not in Ko-Ja. This result meansthat NMT produces a strong effect in the language pair with long linguistic distance. Moreover, the reranking system achieved a great synergy of T2S/PBMT and NMT in both task, even

Experimental Results

All scores of this section are reported in experiments on the official test data; test.txt of theASPEC-JE corpus.
5.1

BLEU
70.36
70.31
70.91

Table 2: Ko-Ja SMT.

We constructed the source word vocabulary withthe most common words in the source language
corpora. For the target character vocabulary, weused a BI (begin/inside) representation (e.g., 結/B,果/I), because it gave better accuracy in preliminary experiment. The sizes of the source vocabularies for English and Korean were 245K and60K, respectively, for the En-Ja and Ko-Ja tasks.
The sizes of the target character vocabularies forJapanese were 6K and 5K, respectively, for theEn-Ja and Ko-Ja tasks. We chose the dimensionality of the source word embedding and the target character embedding to be 200, and chosethe size of the recurrent units to be 1,000. Eachmodel was optimized using stochastic gradient descent (SGD). We did not use dropout. Training wasearly-stopped to maximize the performance on thedevelopment set measured by BLEU.

5#Rules
250M
1950M
1950M
1950M

Table 1: En-Ja SMT.

where fsof tmax is a softmax function, fReLU is a
rectified linear unit (ReLU), Wt we is a weight matrix for the word embedding of the target language,
and by is a target word bias.
4.2

BLEU
31.34
32.48
32.63
32.76

En-Ja SMT

Table 1 shows the evaluation results of our En-Ja
traditional SMT system. The first row in the tableindicates the baseline of the tree-to-string systaxbased model. The second row shows the systemthat reflects the tree modification described in section 2.3. The augmentation method drastically increased both the number of rules and the BLEU
score. Our OOV handling methods described in72

SYS
T2S/PBMT only
NMT only
T2S/PBMT + NMT reranking

En-Ja
BLEU Human
32.76
N/A
33.14
48.50
34.60
53.25

Ko-Ja
BLEU Human
70.91
6.75
65.72
N/A
71.38
14.75

Table 3: Effect of NMT.
if “NMT only” is not effective in Ko-Ja. From thehuman evaluation, we can be clear that our NMT
model produces successful results.

6Liang Huang and Haitao Mi. 2010. Efficient incremental decoding for tree-to-string translation.
In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing(EMNLP).

Conclusion

Liang Huang, Kevin Knight, and Aravind Joshi. 2006.
Statistical syntax-directed translation with extended
domain of locality. In Proceedings of the 7th biennial conference of the Association for MachineTranslation in the Americas (AMTA).

This paper described NAVER machine translation
system for En-Ja and Ko-Ja tasks at WAT 2015.
We developed both the traditional SMT and NMT
systems and integrated NMT into the traditional
SMT in both tasks by reranking n-best translations of the traditional SMT. Our evaluation results
showed that a combination of the NMT and traditional SMT systems outperformed two independent systems.
For the future work, we try to improve the space
and time efficiency of both the tree-to-string SMT
and the NMT model. We also plan to develop andevaluate the NMT system in other language pairs.

Sébastien Jean, Kyunghyun Cho, Roland Memisevic,and Yoshua Bengio. 2015. On using very large
target vocabulary for neural machine translation.
In Proceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics and the7th International Joint Conference of the AsianFederation of Natural Language Processing (ACLIJCNLP).
Yang Liu, Qun Liu, and Shouxun Lin. 2006. Treeto-string alignment template for statistical machinetranslation. In Proceedings of the joint conference ofthe International Committee on Computational Linguistics and the Association for Computational Linguistics (Coling-ACL).

Acknowledgments
We would like to thank Kevin Cho, who has visited for the summer internship program in NAVER
LABS, for his contribution for development of theEnglish spell corrector.

Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol
Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting ofthe Association for Computational Linguistics andthe 7th International Joint Conference of the AsianFederation of Natural Language Processing (ACLIJCNLP).

References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In Proceedings ofthe 3rd International Conference on Learning Representations (ICLR).

Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro Sumita.
2015. Overview of the 2nd workshop on asian translation. In Proceedings of the 2nd Workshop on AsianTranslation (WAT2015), Kyoto, Japan, October.

David Chiang. 2005. A hierarchical phrase-basedmodel for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL).

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.
Sequence to sequence learning with neural networks. In Proceedings of the Twenty-eighth Annual
Conference on Neural Information Processing Systems (NIPS).

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder–decoderfor statistical machine translation. In Proceedings ofthe 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar, October.

Andreas Zollmann and Ashish Venugopal. 2006. Syntax augmented machine translation via chart parsing. In Proceedings of the Workshop on Statistical
Machine Translation (WMT).

73