Evaluating Neural Machine Translation in English-Japanese Task
Zhongyuan Zhu
Weblio Inc.
chugen.shu@weblio.co.jp

Abstract

guage models to rescore SMT outputs generally gives better translation results (Auli andGao, 2014). Other approaches rescore with
RNNs that predict the next word by taking theword in current step and S as inputs (Kalchbrenner and Blunsom, 2013; Cho, Merrienboer, et al., 2014). Here, S is a vector representation summarizes the whole input sentence.
Neural machine translation is a brand-new
approach that samples translation results directly from RNNs. Most published models involve an encoder and a decoder in the network architecture (Sutskever, Vinyals, and Le,2014), called Encoder-Decoder approach. Figure 1 gives a general overview of this approach.
In Figure 1, the vector output S of the encoder
RNN represents the whole input sentence.
Hence, S contains all information required to
produce the translation. In order to boost up
the performance, (Sutskever, Vinyals, and Le,2014) used stacked Long Short-Term Memory(LSTM) units for both encoder and decoder,their ensembled models outperformed phrasebased SMT baseline in English-French translation task.

In this paper, we evaluate Neural
Machine Translation (NMT) models
in English-Japanese translation task.
Various network architectures with different recurrent units are tested. Additionally, we examine the effect of using pre-reordered data for the training. Our experiments show that evensimple NMT models can produce better translations compared with all SMT
baselines. For NMT models, recovering
unknown words is another key to obtaining good translations. We describe
a simple workaround to find missing translations with a back-off system. To our surprise, performing prereordering on the training data hurts
the model performance. Finally, we
provide a qualitative analysis demonstrates a specific error pattern in NMTtranslations which omits some information and thus fail to preserve the complete meaning.

1

Introduction

In the last two decades, Statistical MachineTranslation (SMT) with log-linear models in
the core has shown promising results in thefield. However, as stated in (Duh and Kirchhoff, 2008), log-linear models may suffer fromthe underfitting problem and thus give poor
performance. While for recurrent neural networks (RNNs), as demonstrated in (Mikolov etal., 2010), they brought significant improvement in Natural Language Processing tasks.
In their research, RNNs are shown to be capable of giving more prediction power comparedwith conventional language models when large
training data is given. Using these neural lan-

Figure 1: Basic neural network architecture in
Encoder-Decoder approach
Recently, by scaling up neural network models and incorporating some techniques during
the training, the performance of NMT models
have already achieved the state-of-the-art inEnglish-French translation task (Luong et al.,61

Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 61‒68,
Kyoto, Japan, 16th October 2015.
2015 Copyright is held by the author(s).

2015) and English-German translation task
(Jean et al., 2015).
In this paper, we describe our works on applying NMT to English-Japanese translation
task. The main contributions of this work aredetailed as follows:

Figure 2: An illustration of the computationalgraph of a RNN• We examined the effect of using different
network architecture and recurrent units
for English-Japanese translation

2.1 Long short-term memory
For RNN, as the internal state ht is completelychanged in each time step, BPTT algorithm
dilutes error information after each step ofcomputation. Hence, RNN suffers from the
problem that it is difficult to capture longterm dependencies.

• We empirically evaluated NMT models
trained on pre-reordered data
• We demonstrate a simple solution to recover unknown words in the translation
results with a back-off system

Long short-term memory units (Hochreiter and Schmidhuber, 1997) incorporate somegates to control the information flow. In addition to the hidden units in RNN, memory
cells are used to store long-term information,
which is updated linearly. Empirically, LSTM
can preserve information for arbitrarily long
periods of time.

• We provide a qualitative analysis on thetranslation results of NMT models

2

Recurrent neural networks

Recurrent neural network is the solution formodeling temporal data with neural networks.
The framework of widely used modern RNN isintroduced by Elman (Elman, 1990), it is also
known as Elman Network or Simple Recurrent
Network. At each time step, RNN updates its
internal state ht based on a new input xt andthe previous state ht−1 , produces an output
yt . Generally, they are computed recursively
by applying following operations:ht = f (Wi xt + Wh ht−1 + bh )(1)

yt = f (Wo ht + bo )(2)

Figure 3 gives an illustration of the computational graph of a basic LSTM unit. In which,input gate it , forget gate ft and output gate
ot are marked with rhombuses. “×” and “+”
are element-wise multiplication and elementwise addition respectively. The computational
steps follows (Graves, 2013), 11 weight parameters are involved in this model, comparedwith only 2 weight parameters in RNN. We
can see from Figure 3 that the memory cells
ct can keep unchanged when ft outputs 1 andit outputs 0.

Where f is an element-wise non-linearity,
such as sigmoid or tanh. Figures 2 illustrates
the computational graph of a RNN. Solid lines
in the figure mark out the Affine transformations followed with a non-linear activation.
Dashed lines indicate that the result of previous computation is just a parameter of next
operation. The bias term bh is omitted in theillustration.
RNN can be trained with Backpropagation
Through Time (BPTT), which is a gradientbased technique that unfolds the network
through time so as to compute the actual gradients of parameters in each time step.

Figure 3: An illustration of a basic LSTM unit62

2.2 Gated recurrent unit
Gated recurrent unit (GRU) is originally proposed in (Cho, Merrienboer, et al., 2014).
Similarly to LSTM unit, GRU also has gating
units to control the information flow. While
LSTM unit has a separate memory cell, GRU
unit only maintains one kind of internal states,
thus reduces computational complexity. The
computational graph of a GRU unit is demonstrated in Figure 4. As shown in the figure, 6weight parameters are involved.

Figure 5: Illustration of a basic neural networkarchitecture for NMT with stacked multi-layer
recurrent units.
size is limited.
Attentional models are first proposed in thefield of computer vision, which allows the recurrent network to focus on a small portion in
the image at each step. The internal state is
updated only depends on this glimpse. Softattention first evaluates the weights for all possible positions to attend, then make a weighted
summarization of all hidden states in the encoder. The summarized vector is finally used
to update the internal state of the decoder.
Contrary to hard-attention mechanism which
selects only one location at each step and thus
has to be trained with reinforce learning techniques, soft-attention mechanism makes thecomputational graph differentiable and thus
able to be trained with standard backpropagation.

Figure 4: An illustration of a GRU unit

3

Network architectures of Neural
Machine Translation

A basic architecture of NMT is called EncoderDecoder approach (Sutskever, Vinyals, andLe, 2014), which encodes the input sequence
into a vector representation, then unrolls it togenerate the output sequence. Then softmax
function is applied to the output layer in order to compute cross-entropy. Instead of using
one-hot embeddings for the tokens in the vocabulary, trainable word embeddings are used.
As a pre-processing step, “<eos>” token isappended to the end of each sequence. Whentranslating, the token with the highest probability in the output layer is sampled and input
back to the neural network to get next output.
This is done recursively until “<eos>” is observed. Figure 5 gives a detailed illustrationof this architecture when using stacked multilayer recurrent units.
3.1

Soft-attention models in NMT

As stated in (Cho, Merriënboer, et al., 2014),
two critical drawbacks exist in the basic
Encoder-Decoder approach: (1) the performance degrades when the input sentence getslonger, (2) the vocabulary size in the target

Figure 6: The recurrent network using softattention mechanism to predict next output.
63

of the aligned source word for the unknown
target word. The alignments can be obtainedby conventional aligners. The purpose of thisprocessing step put some cues for recovering
missing words into the output. By applying
this approach, they were able to surpass thestate-of-the-art SMT system in English-Frenchtranslation task.

The application of soft-attention mechanism
in machine translation is firstly described in(Bahdanau, Cho, and Bengio, 2014), which isreferred as “RNNsearch” in this paper. The
computational graph of a soft-attention NMT
model is illustrated in Figure 6. In which, theencoder is replaced by a bi-directional RNN,
the hidden states of two RNNs is finally concatenated in each input position. At eachtime step of decoding, an alignment weightai is computed based on the previous stateof the decoder and the concatenated hidden
state of position i in the encoder. The alignment weights are finally normalized by softmax function. The weighted summarizationof the hidden states in the encoder is then fed
into the decoder. Hence, the internal state ofthe decoder is updated based on 3 inputs: theprevious state, weighted summarization of theencoder and the target-side input token.
The empirical results in (Bahdanau, Cho,and Bengio, 2014) show that the performanceof RNNsearch does not degrade severely likenormal Encoder-Decoder approach.

4

5 Experiments
5.1 Experiment setup
In our experiments, we are curious to see how
NMT models work in English-Japanese translation and how well the existing approaches
for unknown words fit into this setting. As
Japanese language drastically differs from English in terms of word order and grammar
structure. NMT models must capture the semantics of long-range dependencies in a sentence in order to translate it well.
We use Japanese-English Scientific Paper
Abstract Corpus (ASPEC-JE) as training
data and focus on evaluating the models forEnglish-Japanese translation task. In order tomake the training time-efficient, we pick 1.5M
sentences according to similarity score then filter out long sentences with more than 40 wordsin either English or Japanese side. This processing step gives 1.1M sentences for training.
We randomly separate out 1,280 sentences asvalid data.
As almost zero pre-knowledge of NMT experiments in English-Japanese translation canbe found in publications, our purpose is to conduct a thorough experiment so that we canevaluate and compare different model architectures and recurrent units. However, thelimitation of computational resource and time
disallows us to massively test various models,training schemes, and hyper-parameters.
In our experiments, we evaluated four kindsof models as follow:

Solutions of unknown words

A critical practical problem of NMT is the
fixed vocabulary size in the output layer. As
the output layer uses dense connections, enlarging it will significantly increase the computational complexity and thus slow down the
training.
According to existing publications, twokinds of approaches are used to tackle
this problem: model-specific and translationspecific approach.
Well known modelspecific approaches are noise-contrastive training (Mnih and Kavukcuoglu, 2013) and classbased models (Mikolov et al., 2010). In (Jeanet al., 2015), another model-specific solution
is proposed by using only a small set of target vocabulary at each update. By using avery large target vocabulary, they were able
to outperform the state-of-the-art system inEnglish-German translation task.
Solutions of Translation-specific approach
usually take advantage of the alignment of tokens in both sides. For examples, the proposed
method in (Luong et al., 2015) annotates theunknown target words with “unkposi ” insteadof “unk”. Where the subscript i is the position• LSTM Search: Soft-attention modelwith LSTM recurrent units• pre-reordered LSTM Search: Sameas LSTM Search, but the model is trainedon pre-reordered corpus• GRU Search: Soft-attention model with
GRU recurrent units
64models with LSTM unit constantly outperforms the muti-layer Encoder-Decoder model.
This matches our expectation as the alignment
between English and Japanese is too complicated thus it is difficult for simple EncoderDecoder models to capture it correctly. Another observation is that the performance ofthe soft-attention model with GRU unit is significantly lower than that with LSTM unit.
As this is conflict with the results reported inother publications (Jozefowicz, Zaremba, andSutskever, 2015), one possible explanation isthat some implementation issues exist and further investigation is required.
One surprising observation is that using prereordered data to train soft-attention models
does not benefit the perplexity, but degrades
the performance by a small margin. We showthat the same conclusion can be drawn bymeasuring translation performance directly inlatter sections.

• LSTM Encoder-Decoder:
Basic
Encoder-Decoder model with 4 stacked
LSTM layers
Most of the details of these models are common. The recurrent layers of all the models
contain 1024 neurons each. The size of wordembedding is 1000. We truncate the sourceside and target-side vocabulary sizes to 80k
and 40k respectively. For all models, we insert a dense layer contains 600 neurons immediately before the output layer. We basicallyuse SGD with learning rate decay as optimization method, the batch size is 60 and initial
learning rate is 1. The gradients are clipped
to ensure L2 norm lower than 3. Although wesort the training data according to the input
length, the order of batches is shuffled before
training. For LSTM units, we set the bias offorget gate to 1 before training (Jozefowicz,
Zaremba, and Sutskever, 2015). During thetranslation, we set beam size to 20, if no validtranslation is obtained, then another trail withbeam size of 1000 will be performed.
5.2

5.3 Replacing unknown words
Initially, we adapt the solution described in(Luong et al., 2015), which annotate the unknown words with “unkposi ”, where i is the
position of aligned source word. We find thisrequire source-side and target-side sentences
roughly aligned. When testing on the softattention model with pre-reordered training
data, we found this method can correctly pointout the rough aligned position of a missing
word. This allows us to recover the missing
output words with a dictionary or SMT systems.
However, for the training data in naturalorder, the position of aligned words in two
languages differs drastically. The solution described above can hardly be applied as it annotates the unknown words with relative positions.

Evaluating models by perplexity

For our in-house experiments, the evaluationof our models mainly relies on the perplexity
measured on valid data, as a strong correlation between perplexity and translation performance is observed in many existing publications (Luong et al., 2015). The changing
perplexities of the models described in Section
5.1 are visualized in Figure 7.
6

LSTM SearchPre-reordered LSTM Search
GRU Search
LSTM encoder-decoder

Valid perplexity

5
4
3
2
1
1

2

3

Epochs4

5

Figure 8: Illustration of replacing unknown
words with a back-off system.

Figure 7: Visualization of the training for different models.
In Figure 7, we can see that soft-attention

Here, we propose a simple workaround for65

recovering the unknown words with a back-off
system. We translate the input sentence using
both a NMT system and a baseline SMT system. Assume the translation results are similar, then if we observe an unknown word in the
result of the NMT system, then it is reasonable to infer that the rarest word in the baseline result which is missing in the NMT result
should be this unknown translation. This isdemonstrated in Figure 8, the rarest word in
the baseline result is picked out to replace theunknown word in the NMT result. Practically,
the assumption will not be true, the resultsof NMT systems and conventional SMT systems differ tremendously. Hence, some incorrect word replacements are introduced. Thismethod can be generalized to recover multiple
unknown words by selecting the rarest word ina near position.
5.4

Table 1: Automatic evaluation results in
WAT2015
BLEU

RIBES

PB basline

29.80
32.56
33.44
32.19
30.97
33.38
34.19
35.97
36.21

0.691
0.746
0.758
0.797
0.779
0.800
0.802
0.807
0.809

HPB baseline
T2S baseline
Single LSTM SearchPre-reordered LSTM Search
Ensemble of 2 LSTM Search+ UNK replacing (S1)
+ System combination+ 3 pre-reordered ensembles (S2)tem combination based on perplexity scores.
We evaluate the perplexity for all outputs produced by a baseline system and the NMT
model. These two sets of perplexity score
are normalized by mean and standard deviation respectively. Then for each NMT result,
we rescore it with the difference of perplexity
against the baseline system. Intuitively, if the
NMT result is better than the baseline result,the new score shall be a positive number. In
our experiment, we pick the system describedin (Zhu, 2014) as baseline system. We pick
top-1000 results from NMT and the rest fromthe baseline system, this gives us a gain of 1.8
in BLEU.

Evaluating translation
performance

In this section, we describe our submitted systems and report the evaluation results in the
English-Japanese translation task of The 2nd
Workshop on Asian Translation 1 (Nakazawa
et al., 2015). We train these models with
AdaDelta for 5 epochs. Then, we fine-tune themodel with AdaGrad using an enlarged training data, that each sentence contains no morethan 50 words. With this fine-tuning step, weare able to achieve perplexity of 1.76 in valid
data.
The automatic evaluation results are shownin Table 1. Three SMT baselines are pickedfor comparison. In the middle of the table,
we list two single soft-attention NMT models
with LSTM unit. The results show that training models on pre-reordered corpus leads todegrading of translation performance, where
the pre-reordering step is done using the model
described in (Zhu, 2014).
Our submitted systems are basically an ensemble of two LSTM Search models trainedon natural-order data, as shown in the bottom of Table 1. After we replaced unknown
words with the technique described in 5.3, we
gained 0.8 BLEU on test data. This is our firstsubmitted system, marked with “S1”.
We also found it is useful to perform a sys1

Model

Table 2: Human evaluation results for submitted system in WAT2015
Model

HUMAN

T2S baseline

30.00043.50053.750Submitted system 1 (S1)
Submitted system 2 (S2)

Finally, we added 3 pre-reordered LSTM
Search models to the ensemble, results in a 5model ensemble. During the translation, thesethree models receive pre-reordered input, another two LSTM Search models receive input
in natural order. We gain 0.24 BLEU withthis setting, and this is the second submitted system, marked with “S2”. Human evaluation results of our submitted systems areshown in Table 2. As we already know thatpre-reordering does not help improving translation performance, a natural choice is to train

Our team ID is “WEBLIO MT”
66

Referencesmore normal LSTM Search models and put
into the ensemble. We failed to do it because
of insufficient time.
5.5

Auli, Michael and Jianfeng Gao (2014). “Decoder integration and expected bleu training for recurrent neural network language
models”. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL’14), pp. 136–
142.
Bahdanau, Dzmitry, Kyunghyun Cho,and Yoshua Bengio (2014). “Neural machine translation by jointly learning toalign and translate”. In: arXiv preprint
arXiv:1409.0473.
Cho, Kyunghyun, Bart van Merrienboer, etal. (2014). “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation”. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for
Computational Linguistics, pp. 1724–1734.
Cho, Kyunghyun, Bart van Merriënboer, etal. (2014). “On the Properties of Neural
Machine Translation: Encoder–Decoder Approaches”. In: Syntax, Semantics and Structure in Statistical Translation, p. 103.
Duh, Kevin and Katrin Kirchhoff (2008). “Beyond log-linear models: boosted minimum
error rate training for N-best Re-ranking”.
In: Proceedings of the 46th Annual Meeting of the Association for Computational
Linguistics on Human Language Technologies: Short Papers. Association for Computational Linguistics, pp. 37–40.
Elman, Jeffrey L (1990). “Finding structure intime”. In: Cognitive science 14.2, pp. 179–
211.
Graves, Alex (2013). “Generating sequenceswith recurrent neural networks”. In: arXiv
preprint arXiv:1308.0850.
Hochreiter, Sepp and Jürgen Schmidhuber(1997). “Long short-term memory”. In:
Neural computation 9.8, pp. 1735–1780.
Jean, Sébastien et al. (2015). “On Using Very
Large Target Vocabulary for Neural Machine Translation”. In: Proceedings of the
53rd Annual Meeting of the Association
for Computational Linguistics and the 7thInternational Joint Conference on Natural
Language Processing (Volume 1: Long Pa-

Qualitative analysis

To find some insights in the translation results of NMT systems, we performed qualitative analysis on a proportion of held-out development data. During the inspection, we foundmany errors share the same pattern. It turns
out that NMT model tends to make a perfecttranslation by omitting some information during the translation. In this case, the output
tends to be a valid sentence, but the meaning is partially lost. One example of this phenomenon is shown in the following snippet:
Input: this paper discusses some systematic
uncertainties including casimir force ,false force due to electric force , andvarious factors for irregular
uncertainties due to patch field anddetector noise .
NMT result: ここ で は ， Ｃａｓｉｍｉｒ 力 を考慮 し た いく つ か の 系統 的 不 確実 性に つ い て 論 じ た 。
Reference: Ｃａｓｉｍｉｒ 力 や 電気 力 に よる 偽 の 力 ， パッチ 場 や 検出 器 雑音 に
よ る 不 規則 な 不確か さ の 種々 の 要因
を 含め ， 幾 つ か の 系統 的 不確か さ
を 論 じ た 。

6 Conclusion
In this paper, we performed a systematic evaluation of various kinds for NMT models in thesetting of English-Japanese translation. Basedon the empirical evaluation results, we foundsoft-attention NMT models can already makegood translation results in English-Japanesetranslation task. Their performance surpasses
all SMT baselines by a substantial margin according to RIBES scores. We also found that
NMT models can work well without extra dataprocessing steps such as pre-reordering. Finally, we described a simple workaround to
recover unknown words with a back-off system.
However, a sophisticated solution for dealing with unknown words is still an open question in the English-Japanese setting. As somepatterns of mistakes can be observed from thetranslation results, there exists some space forfurther improvements.

67

pers). Beijing, China: Association for Computational Linguistics, pp. 1–10.
Jozefowicz, Rafal, Wojciech Zaremba, and Ilya
Sutskever (2015). “An Empirical Exploration of Recurrent Network Architectures”.
In: Proceedings of the 32nd International
Conference on Machine Learning (ICML15), pp. 2342–2350.
Kalchbrenner, Nal and Phil Blunsom (2013).
“Recurrent Continuous Translation Models”. In: Proceedings of the 2013 Conference on Empirical Methods in Natural
Language Processing. Seattle, Washington,
USA: Association for Computational Linguistics, pp. 1700–1709.
Luong, Thang et al. (2015). “Addressing theRare Word Problem in Neural MachineTranslation”. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Beijing, China: Association for Computational Linguistics, pp. 11–19.
Mikolov, Tomas et al. (2010). “Recurrent neural network based language model.” In: INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan,
September 26-30, 2010, pp. 1045–1048.
Mnih, Andriy and Koray Kavukcuoglu (2013).
“Learning word embeddings efficiently withnoise-contrastive estimation”. In: Advancesin Neural Information Processing Systems
26. Ed. by C.J.C. Burges et al. Curran Associates, Inc., pp. 2265–2273.
Nakazawa, Toshiaki et al. (2015). “Overview
of the 2nd Workshop on Asian Translation”.
In: Proceedings of the 2nd Workshop on
Asian Translation (WAT2015).
Sutskever, Ilya, Oriol Vinyals, and Quoc VV
Le (2014). “Sequence to sequence learning with neural networks”. In: Advancesin neural information processing systems,
pp. 3104–3112.
Zhu, Zhongyuan (2014). “Weblio Prereordering Statistical Machine Translation
System”. In: Proceedings of the 1st Workshop on Asian Translation (WAT2014).

68