NAVER Machine Translation System for WAT 2015
Hyoung-Gyu Lee1, Jaesong Lee1, Jun-Seok Kim1 and Chang-Ki Lee2
1NAVER LABS, NAVER Corp. and 2Kangwon National University

Neural Machine Translation

System Overview

SourceVocab.

T2S N-best NMT
Decoding
RerankingTranslation

Input
sentence

Char-level
PBMT
Decoding
Word-level
PBMT
Decoding

N-bestNMT
RerankingTranslation

N-best Language Analyzer

 Handling OOV English: Moses tokenizer & Berkeley parser
 Japanese: In-house tokenizer and POS tagger

 Tree-to-String Syntax-based SMT
 Use source parse trees Synchronous context free grammar (SCFG) rules
 Char parsing decoder with cube pruning Rule Augmentation
 Known as Syntax-Augmented Machine Translation(Zollmann and Venugopal, 2006)
 Extract more rules by modifying source parse tree
 Given a parse tree, produce additional nodes

VP
NPNP

PRP

VBP

PRP

I
(0)

love
(1)

you(2)

 #Target word vocab.: 173K(En-Ja), 128K(Ko-Ja)
 #Target char. Vocab.: 6K(En-Ja), 5K(Ko-Ja)
 Src/Tgt embedding dimension: 200
 #Recurrent units: 1,000 Optimization: Stochastic gradient descent (SGD) Not use dropout
 Early-stopped to maximize BLEU on dev. data

(2) B-I representation of target characters Better accuracy in preliminary experiment e.g. 結/B 果/I

English to JapaneseS Our settings(1) Use target character sequence rather than word
sequence Removes the need to replace rare words withthe unknown word symbol
 Simpler than other methods recently
proposed to address the same issue

Training
CorpusTraining
CorpusOOV
Handling

Input
sentence IdeasKorean to Japanese (Ko-Ja)

English to Japanese (En-Ja)

Baseline nodes

Additional nodes

0-0 PRP
0-0 NP
1-1 VBP
2-2 PRP
2-2 NP
1-2 VP
0-2 S

1-2 VBP+PRP
0-2 PRP+VP
0-1 PRP++VBP(1) Hyphen Word Split
 e.g. nano-laminate  nano laminate(2) Spell Error Correction
 Use open source spell checker, Aspell Detection based on skip rule Capitals, numbers, symbols
 Correction based on edit distance Large gap causes wrong correction
 Select one with the shortest distance among top-3 suggestion(3) Modified RNN encoder-decoder Bi-directional RNN with an attention
mechanism (Bahdanau et al., 2015)

yt-1

yt

zt-1’

zt ’zt-1zt

 New hidden state of the decoder
 Prob. of next target word Example

By Aspell
VLSI … H2 …
remrakable …

correction

detection

remrakable< Modified RNN Encoder-Decoder>remarkable

Aspell Suggestion
1. remarkable
2. remakable
3. reamarkable

Experimental Results
 En-Ja SMT

 Ko-Ja SMT

Korean-to-Japanese Language Analyzer
 Korean: Mecab-ko w/o POS tagger Japanese: Juman w/o POS tagger

 Combination of Two PBMT Word-level PBMT 5-gram LM, max-phrase-len=7
 Char-level PBMT Char-level tokenization
 10-gram LM, max-phrase-len=10
 Methods for selecting the best hypothesis(1) Rule-based:- Char-level PB, if an input has an OOV- Word-level PB, otherwise(2) NMT:- Rerank n-best lists of two PBMT Why Char-level PBMT Mis-tokenization problem in word-level PBMT cause many OOVs
Ko/Ja representation of “Diisononyl phthalate”:
 Ko: 디 / 이소노 / 닐 / 프탈 / 레이트 (5 tokens, 1 OOV) Ja: ジイソノニルフタレート (1 token)

 Both Ko/Ja technical terms are usually transliterated
from the same foreign word
 Char-level PBMT implicitly learns transliteration rules

 Rule augmentation increases both BLEU and #Rules
 Hyphen word split and spell correction improve theperformance Experiment with NMT variations
NMT Model
RNN E.-D. using target word seq.
RNN E.-D. using target char. seq.
RNN E.-D. using target char. seq.
with B-I
Modified RNN E.-D. using targetchar. seg. with B/I

En-JaBLEU
29.78
31.25
32.05
33.14

 Character-level PB is comparable to Word-level PB
 Combined system is the best Effect of combination between T2S/PBMT andNMT NMT outperform T2S in En-Ja, while it does notoutperform PBMT in Ko-Ja
 NMT reranking give a great synergy between T2S/PBand NMT