Evaluating Neural Machine Translation in English-Japanese Task
Zhongyuan Zhu

Overview (Abstract)

Evaluation results in English-Japanese task

We evaluated Neural Machine Translation (NMT) models inEnglish-Japanese translation task. Various networkarchitectures with different recurrent units are tested.
Additionally, we examine the effect of using pre-reordereddata for the training. Our experiments show that even simple
NMT models can produce better translations compared withall SMT baselines. For NMT models, recovering unknownwords is another key to obtaining good translations. We
describe a simple workaround to find missing translations
with a back-off system. Surprisingly, performing prereordering on the training data hurts the model performance.
We provide a qualitative analysis demonstrates a specificerror pattern in NMT translations which omits partial
information and thus fail to preserve the complete meaning.

BLEU

RIBES

HUMAN

BASELINE T2S SMT

33.44

0.758

30.00

Ensemble of 2 LSTM Search

33.38

0.800

-

+ UNK replacing(submitted system 1)

34.19

0.802

43.50

+ System combination

35.97

0.807

-+ 3 pre-reordered ensembles(submitted system 2)

36.210.80953.75(JPO adequacy evaluation result of system 2: 3.81, best competitor: 4.04)

Experimental details

Findings‣ A comparison of two network architectures‣ Soft-attention models outperforms multi-layer
encoder-decoder models
The evaluation of valid perplexity shows that soft-attention models
outperforms simple encoder-decoder models with a substantial margin.
This matches our expectation as the alignment between English andJapanese are far more complicated than English-French pair.

multi-layer encoder-decoder model‣ Training models on pre-reordered data hurts theperformance

soft-attention model‣ Visualization of the training process for different models
LSTM Search:soft-attention model with
LSTM units
Pre-reordered LSTM
Search:soft-attention model with
LSTM units trained on prereordered data
GRU Search:soft-attention model with
GRU units
LSTM encoder-decoder:
4-layer encoder-decoder
model with LSTM units
IRNN Search:soft-attention model withIRNN units

Both the perplexity on valid
data and automatic evaluation
scores show that training softattention LSTM models on prereordered data degrades theperformance.

BLEU

RIBES

Single LSTM Search

32.190.797

Pre-reordered LSTM
Search

30.97

0.779

‣ NMT models tend to make grammatically valid but
incomplete translations

Input

NMT result(for model comparison, we use SGD algorithm to optimize the network,
details are presented in the paper)

this paper discusses some systematic uncertainties including
casimir force , false force due to electric force , and various
factors for irregular uncertainties due to patch field and detector
noise .
ここ で は ， Ｃａｓｉｍｉｒ 力 を 考慮 し た いく つ か の 系
統 的 不 確実 性 に つ い て 論 じ た 。
Ｃａｓｉｍｉｒ 力 や 電気 力 に よ る 偽 の 力 ， パッチ 場 やReference‣ Problem of unknown words
Replacing unknown words in the target side with “” (Luong etal., 2015) works well with soft-attention models trained on prereordered data. However, for models trained on data of natural order,
other sophisticated solutions are required.
A simple workaround is to find the missing word in the translation
result of a baseline system. As for the same target word, they usuallyshare similar context even in different translations.

検出 器 雑音 に よ る 不 規則 な 不確か さ の 種々 の 要因 を 含め ， 幾 つ か の 系統 的 不確か さ を 論 じ た 。

Retrospection
We conducted a detailed qualitative analysis on a held-out development
dataset. The existence of unknown words are found to drasticallydegrade the quality of translations. Even the missing word can beposteriorly recovered, some of the translations are still unnatural. In ourexperiments, we set vocabulary size to 80k and 40k for the input andoutput layer respectively. Increasing these numbers will significantly
slow down the training. Overcoming this problem is expected to be thekey of obtaining qualitative translations for NMT models.

Weblio Inc.