Neural Reranking Improves Subjective Quality of Machine Translation:
NAIST at WAT2015
Graham Neubig, Makoto Morishita, Satoshi Nakamura
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5 Takayama-cho, Ikoma-shi, Nara, Japan{neubig,morishita.makoto.mb1,s-nakamura}@is.naist.jp
some correlation with translation quality,1 it is fairto hypothesize that these gains will carry over togains in human evaluation, but empirical evidence
for this hypothesis is still scarce. In this paper, we
attempt to close this gap by examining the gains
provided by using neural MT models to rerank the
hypotheses a state-of-the-art non-neural MT system, both from the objective and subjective perspectives.
Specifically, as part of the Nara Institute of
Science and Technology (NAIST) submission to
the Workshop on Asian Translation (WAT) 2015
(Nakazawa et al., 2015), we generate reranked andnon-reranked translation results in four language
pairs (Section 2). Based on these translation results, we calculate scores according to automatic
evaluation measures BLEU and RIBES (Isozaki etal., 2010), and a manual evaluation that involves
comparing hypotheses to a baseline system (Section 3). Next, we perform a detailed analysis of thecases in which subjective impressions improved ordegraded due to neural MT reranking, and identify
major areas in which neural reranking improves
results, and areas in which reranking is less helpful (Section 4). Finally, as an auxiliary result, we
also examine the effect that the size of the n-best
list used in reranking has on the improvement oftranslation results (Section 5).

Abstract
This year, the Nara Institute of Scienceand Technology (NAIST)’s submission to
the 2015 Workshop on Asian Translation
was based on syntax-based statistical machine translation, with the addition of areranking component using neural attentional machine translation models. Experiments re-confirmed results from previous
work stating that neural MT reranking provides a large gain in objective evaluation
measures such as BLEU, and also confirmed for the first time that these results
also carry over to manual evaluation. We
further perform a detailed analysis of reasons for this increase, finding that the main
contributions of the neural models lie in
improvement of the grammatical correctness of the output, as opposed to improvements in lexical choice of content words.

1

Introduction

Neural network models for machine translation (MT) (Kalchbrenner and Blunsom, 2013;
Sutskever et al., 2014; Bahdanau et al., 2015),
while still in a nascent stage, have shown impressive results in a number of translation tasks.
Specifically, a number of works have demonstrated gains in BLEU score (Papineni et al.,2002) over state-of-the-art non-neural systems,
both when using the neural MT model standalone (Luong et al., 2015a; Jean et al., 2015;
Luong et al., 2015b), or to rerank the output of
more traditional systems phrase-based MT systems (Sutskever et al., 2014).
However, despite these impressive results withregards to automatic measures of translation quality, there has been little examination of the effectthat these gains have on the subjective impressionsof human users. Because BLEU generally has

2 Generation of Translation Results
2.1 Baseline System
All experiments are performed on WAT2015 translation task from Japanese (ja) to/from English (en)and Chinese (zh). As a baseline, we used the
NAIST system for WAT 2014 (Neubig, 2014), astate-of-the-art system that achieved the highest
accuracy on all four tracks in the last year’s eval1

Particularly when comparing similar systems, such as thecase of when neural MT is used for reranking existing system
results.
35

Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 35‒41,
Kyoto, Japan, 16th October 2015.
2015 Copyright is held by the author(s).

uation.2 The details of construction are describedin Neubig (2014), but we briefly outline it here forcompleteness.
The system is based on the Travatar toolkit(Neubig, 2013), using tree-to-string statistical MT(Graehl and Knight, 2004; Liu et al., 2006),
in which the source is first syntactically parsed,
then subtrees of the input parse are converted
into strings on the target side. This translation
paradigm has proven effective for translation between syntactically distant language pairs such as
those handled by the WAT tasks. In addition, following our findings in Neubig and Duh (2014), to
improve the accuracy of translation we use forestbased encoding of many parse candidates (Mi etal., 2008), and a supervised alignment techniquefor ja-en and en-ja (Riesa and Marcu, 2010).
To train the systems, we used the ASPEC corpus provided by WAT. For the zh-ja and ja-zh systems, we used all of the data, amounting to 672k
sentences. For the en-ja and ja-en systems, weused all 3M sentences for training the language
models, and the first 2M sentences of the training
data for training the translation models.
For English, Japanese, and Chinese, tokenization was performed using the Stanford Parser(Klein and Manning, 2003), the KyTea toolkit(Neubig et al., 2011), and the Stanford Segmenter(Tseng et al., 2005) respectively. For parsing, we
use the Egret parser,3 which implements the latentvariable parsing model of (Petrov et al., 2006).4
For all systems, we trained a 6-gram language model smoothed with modified KneserNey smoothing (Chen and Goodman, 1996) using KenLM (Heafield et al., 2013). To optimize
the parameters of the log-linear model, we use
standard minimum error rate training (MERT; Och(2003)) with BLEU as an objective.

long short-term memory (LSTM; Hochreiter andSchmidhuber (1997)) recurrent networks. This results in an encoding vector hj for each word fj in
f . The model then proceeds to generate the target translation ê one word at a time, at each time
step calculating soft alignments ai that are used togenerate a context vector gi , which is referenced
when generating the target word
gi =

|f |
∑

ai,j hj .

(1)

j=1

Attentional models have a number of appealing properties, such as being theoretically able
to encode variable length sequences without worrying about memory constraints imposed by the
fixed-size vectors used in encoder-decoder models. These advantages are confirmed in empirical results, with attentional models performing
markedly better on longer sequences (Bahdanau etal., 2015).
To train the neural MT models, we used theimplementation provided by the lamtram toolkit.5
The forward and reverse LSTM models each had
256 nodes, and word embeddings were also set tosize 256. For ja-en and en-ja models we chose thefirst 500k sentences in the training corpus, and for
ja-zh and zh-ja models we used all 672k sentences.
Training was performed using stochastic gradientdescent (SGD) with an initial learning rate of 0.1,
which was halved every epoch in which the development likelihood decreased.
For each language pair, we trained two models and ensembled the probabilities by linearly interpolating between the two probability distributions.6 These probabilities were used to rerankunique 1,000-best lists from the baseline model.
To perform reranking, the log likelihood of theneural MT model was added as an additional feature to the standard baseline model features, andthe weight of this feature was decided by running
MERT on the dev set.

2.2 Neural MT Models
As our neural MT model, we use the attentionalmodel of Bahdanau et al. (2015). The model firstencodes the source sentence f using bidirectional

3 Experimental Results

2First, we calculate overall numerical results for oursystems with and without the neural MT reranking
model. As automatic evaluation we use the standard BLEU (Papineni et al., 2002) and reorderingoriented RIBES (Isozaki et al., 2010) metrics. In

Scripts to reproduce the system are available at http://phontron.com/project/wat2014.
3https://github.com/neubig/egret
4
In addition, for ja-en translation, we make one modification to the parser used in the previous year’s submission,
performing parser self-training (McClosky et al., 2006) using sentences from the training data that had a BLEU score
greater than 0.8, and selecting the tree corresponding to the
500-best hypothesis that had the best score according to
BLEU+1 (Lin and Och, 2004).

5

http://github.com/neubig/lamtram
More standard log-linear interpolation resulted in similar, or slightly inferior results.
636

System
Base
Rerank

B
36.6
38.2

en-ja
R
H
79.6 49.8
81.4 62.3

B
22.6
25.4

ja-en
R
H
72.3 11.8
75.0 35.5

B
40.5
43.0

zh-ja
R
H
83.4 25.8
84.8 35.8

B
30.1
31.6

ja-zh
R
81.5
83.3

H
2.8
7.0

Table 1: Overall BLEU, RIBES, and HUMAN scores for our baseline system and system with neural
MT reranking. Bold indicates a significant improvement according to bootstrap resampling at p < 0.05
(Koehn, 2004).
Type
Reordering
Deletion
Insertion
Substitution
Conjugation
Total

manual evaluation, we use the WAT “HUMAN”
evaluation score (Nakazawa et al., 2015), whichis essentially related to the number of wins overa baseline phrase-based system. In the case thatthe system beats the baseline on all sentences, the
HUMAN score will be 100, and if it loses on all
sentences the score will be -100.
From the results in Table 1, we can first see that
adding the neural MT reranking resulted in a significant increase in the evaluation scores for all
language pairs under consideration, except for the
manual evaluation in ja-zh translation.7 It should
be noted that these gains are achieved even though
the original baseline was already quite strong (outperforming most other WAT2015 systems without
a neural component). While neural MT reranking has been noted to improve traditional systems with respect to BLEU score in previous work
(Sutskever et al., 2014), to our knowledge this is
the first work that notes that these gains also carryover convincingly to human evaluation scores. In
the following section, we will examine the resultsin more detail and attempt to explain exactly what
is causing this increase in translation quality.

Impr.
55
20
19
15
8
117

Degr.
9
10
2
11
1
33

% Impr.
86%
67%
90%
58%
89%
78%

Table 2: A summary of the improvements anddegradations caused by neural reranking.

the error typology of Vilar et al. (2006) consisting of insertion, deletion, word conjugation, word
substitution, and reordering, as well as subcategories of each of these categories (the number of
sub-categories totalled approximately 40). If there
was more than one change in the sentence, only
the change that we subjectively felt had the largest
effect on the translation quality was annotated.
The number of improvements and degradations
afforded by neural MT reranking is shown in Table 2. From this figure, we can see that overall,neural reranking caused an improvement in 117
sentences, and a degradation in 33 sentences, corroborating the fact that the reranking process isgiving consistent improvements in accuracy. Further breaking down the changes, we can see that
improvements in word reordering are by far themost prominent, slightly less than three times thenumber of improvements in the next most common category. This demonstrates that the neural
MT model is successfully capturing the overall
structure of the sentence, and effectively disambiguating reorderings that could not be appropriately scored in the baseline model.
Next in Table 3 we show examples of the fourmost common sub-categories of errors that were
fixed by the neural MT reranker, and note the total number of improvements and degradations ofeach. The first subcategory is related to the general reordering of phrases in the sentence. As there

4 Analysis
To perform a deeper analysis, we manually examined the first 200 sentences of the ja-en part of theofficial WAT2015 human evaluation set. Specifically, we (1) compared the baseline and reranked
outputs, and decided whether one was better or ifthey were of the same quality and (2) in the casethat one of the two was better, classified the example by the type of error that was fixed or causedby the reranking leading to this change in subjective impression. Specifically, when annotating
the type of error, we used a simplified version of
7
The overall scores for ja-zh are lower than others, perhaps a result of word-order between Japanese and Chinese
being more similar than Japanese and English, the parser
for Japanese being weaker than that of the other languages,
and less consistent evaluation scores for the Chinese output(Nakazawa et al., 2014).

37

1. Reordering of Phrases (+26, -4)
In.
症例２においては，直腸がんの肝転移に対する化学療法中に，発赤，硬結，皮膚潰ようを生じた。
Ref.
In case 2, reddening, induration, and skin ulcer appeared during chemical therapy for liver metastasis of rectal
cancer.
Base.
In case 2, occurred during chemotherapy for liver metastasis of rectal cancer, flare, induration, skin ulcer.
Rerank In case 2, the flare, induration, skin ulcer was produced during the chemotherapy for hepatic metastasis of rectal
cancer.
2. Insertion/Deletion of Auxiliary Verbs (+15, -0)In.
これにより得られる支配方程式は壁面乱流のようなせん断乱流にも有用である。
Ref.
Governing equation derived by this method is useful for turbulent shear flow like turbulent flow near wall.
Base.
The governing equation is obtained by this is also useful for such as wall turbulence shear flow.
Rerank The governing equation obtained by this is also useful for shear flow such as wall turbulence.
3. Reordering of Coordinate Structures (+13, -2)In.
レーザー加工は高密度光束による局所的な加熱とアブレーションにより行う。
Ref.
Laser work is done by local heating and ablation with high density light flux.
Base.
The laser processing is carried out by local heating by high-density luminous flux and ablation.
Rerank The laser processing is carried out by local heating and ablation by high-density flux.
4. Conjugation of Verb Agreement (+6, -0)In.
ラングミュア‐ブロジェット法や包接化にも触れた。
Ref.
Langmuir-Blodgett method and inclusion compounds are mentioned.
Base.
Langmuir-Blodgett method and inclusion is also discussed.
Rerank Langmuir-Blodgett method and inclusion are also mentioned.

Table 3: An example of more common varieties of improvements caused by the neural MT reranking.
is a large amount of reordering involved in translating from Japanese to English, mistaken longdistance reordering is one of the more common
causes for errors, and the neural MT model was
effective at fixing these problems, resulting in 26
improvements and only 4 degradations. In the sentence shown in the example, the baseline systemswaps the verb phrase and subject positions, making it difficult to tell that the list of conditions arewhat “occurred,” while the reranked system appropriately puts this list as the subject of “occurred.”
The second subcategory includes insertions ordeletions of auxiliary verbs, for which there were
15 improvements and not a single degradation.
The reason why these errors occurred in the firstplace is that when a transitive verb, for example“obtained,” occurs on its own, it is often translated
as “X was obtained by Y,”8 but when it occurs asa relative clause decorating the noun X it will betranslated as “X obtained by Y,” as shown in theexample. The baseline system does not include
any explicit features to make this distinction between whether a verb is part of a relative clause ornot, and thus made a number of mistakes of thisvariety. However, it is evident that the neural MT
model has learned to make this distinction, greatly
reducing the number of these errors.
The third subcategory is similar to the first, but
explicitly involves the correct interpretation of co-ordinate structures. It is well known that syntactic parsers often make mistakes in their interpretation of coordinate structures (Kummerfeld et al.,2012). Of course, the parser used in our syntaxbased MT system is no exception to this rule, andparse errors often cause coordinate phrases to bebroken apart on the target side, as is the case in theexample’s “local heating and ablation.” The factthat the neural MT models were able to correct alarge number of errors related to these structures
suggests that they are able to successfully determine whether two phrases are coordinated or not,and keep them together on the target side.
The final sub-category of the top four is related
to verb conjugation agreement. Many of the examples related to verb conjugation, including the one
shown in Table 3, were related to when two singular nouns were connected by a conjunction. In
this case, the local context provided by a standard
n-gram language model is not enough to resolve
the ambiguity, but the longer context handled by
the neural MT model is able to resolve this easily.
What is notable about these four categories isthat they all are related to improving the correctness of the output from a grammatical pointof view, as opposed to fixing mistakes in lexical
choice or terminology. In fact, neural MT reranking had an overall negative effect on choice of terminology with only 2 improvements at the costof 4 degradations. This was due to the fact thatthe neural MT model tended to prefer more com-

8
This passivization is somewhat of a trait of the scientificpaper material used as material for this analysis.

38

−9.0

en-ja

1e3

−1.95

−9.1

Model Score

−9.2
−9.3

−6.25

−2.05

−6.30

−2.10

−9.5

−2.15

−9.7
−9.8
−9.9 0
10

1

10

2

10

3

10

38.0

BLEU

37.5

37.0

36.5
0

10

1

10

2

10

3

10

−6.20

−2.00

−9.4
−9.6

ja-en

1e4

ja-zh

1e4

−1.22

−6.35

−1.24
−1.26

−6.45
−6.50

−2.25

−6.55
1

10

2

10

3

10

−6.60 0
10

27.0

42.5

26.5

42.0

26.0

41.5

25.5

41.0

25.0

40.5

24.5

40.0

24.0 0
10

−1.20

−6.40

−2.20

−2.30 0
10

zh-ja

1e3

1

10

2

10

3

10

39.5 0
10

−1.28
1

10

2

10

3

10

−1.30 0
10

10

1

10

2

10

1

10

2

10

3

31.0

30.5

30.0

29.5
1

10

2

10

3

10

0

10

10

3

Figure 1: Model and BLEU scores after neural MT reranking for each n-best list size (log scale).
mon words, mistaking “radiant heat” as “radiationheat” or “slipring” as “ring.” While these tendencies will be affected by many factors such as
the size of the vocabulary or the number and sizeof hidden layers of the net, we feel it is safe to
say that neural MT reranking can be expected tohave a large positive effect on syntactic correctness of output, while results for lexical choice areless conclusive.

ing points. First, we can see that the improvement in scores is very slightly sub-linear in the log
number of hypotheses in the n-best list. In otherwords, every time we double the n-best list size
we will see an improvement in accuracy that isslightly smaller than the last time we doubled thesize. Second, we can note that in most cases thistrend continues all the way up to our limit of 1000best lists, indicating that gains are not saturating,
and we can likely expect even more improvementsfrom using larger lists, or perhaps directly performing decoding using neural models (Alkhouli
et al., 2015). The en-ja results, however, are an
exception to this rule, with BLEU gains more orless saturating around the 50-best list point.

5 Effect of n-best Size on Reranking
In the previous sections, we confirmed the effectiveness of n-best list reranking using neural MT
models. However, reranking using n-best lists(like other search methods for MT) is an approximate search method, and its effectiveness is limited by the size of the n-best list used. In order toquantify the effect of this inexact search, we performed experiments to examine the post-reranking
automatic evaluation scores of the MT results for
all n-best list sizes from 1 to 1000. Figure 1 shows
the results of this examination, with the x-axis referring to the log-scaled number of hypotheses in
the n-best list, and the y-axis referring to the quality of the translation, either with regards to modelscore (for the model including the neural MT likelihood as a feature) or BLEU score.9
From these results we can note several interest-

6 Conclusion
In this paper we described results applying neural MT reranking to a baseline syntax-based machine translation system in 4 languages. In particular, we performed an in-depth analysis of whatkinds of translation errors were fixed by neural
MT reranking. Based on this analysis, we foundthat the majority of the gains were related to improvements in the accuracy of transfer of correct
grammatical structure to the target sentence, withthe most prominent gains being related to errors
regarding reordering of phrases, insertion/deletionof copulas, coordinate structures, and verb agreement. We also found that, within the neural MT
reranking framework, accuracy gains scaled ap-

9

The BLEU scores differ slightly from Table 1 due to differences in tokenization standards between these experiments
and the official evaluation server.
39

proximately log-linearly with the size of the n-best
list, and in most cases were not saturated even after examining 1000 unique hypotheses.

Chin-Yew Lin and Franz Josef Och. 2004. Orange: amethod for evaluating automatic evaluation metrics
for machine translation. In Proc. COLING, pages
501–507.

Acknowledgments:

Yang Liu, Qun Liu, and Shouxun Lin. 2006. Treeto-string alignment template for statistical machinetranslation. In Proc. ACL, pages 609–616.

This work was supported by JSPS KAKENHI
Grant Number 25730136.

Minh-Thang Luong, Ilya Sutskever, Quoc Le, Oriol
Vinyals, and Wojciech Zaremba. 2015a. Addressing the rare word problem in neural machine translation. In Proc. ACL, pages 11–19.

References
Tamer Alkhouli, Felix Rietig, and Hermann Ney. 2015.
Investigations on phrase-based decoding with recurrent neural network language and translation models. In Proc. WMT, pages 294–303.

Thang Luong, Hieu Pham, and Christopher D. Manning. 2015b. Effective approaches to attentionbased neural machine translation. In Proc. EMNLP,
pages 1412–1421.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In Proc. ICLR.

David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In
Proc. HLT, pages 152–159.

Stanley F. Chen and Joshua Goodman. 1996. An empirical study of smoothing techniques for language
modeling. In Proc. ACL, pages 310–318.

Haitao Mi, Liang Huang, and Qun Liu. 2008. Forestbased translation. In Proc. ACL, pages 192–199.

Jonathan Graehl and Kevin Knight. 2004. Training
tree transducers. In Proc. HLT, pages 105–112.

Toshiaki Nakazawa, Hideki Mino, Isao Goto, Sadao
Kurohashi, and Eiichiro Sumita. 2014. Overview
of the 1st Workshop on Asian Translation. In Proc.
WAT.

Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H.
Clark, and Philipp Koehn. 2013. Scalable modified
Kneser-Ney language model estimation. In Proc.
ACL, pages 690–696.

Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro Sumita.
2015. Overview of the 2nd Workshop on AsianTranslation. In Proc. WAT.

Sepp Hochreiter and Jürgen Schmidhuber. 1997.
Long short-term memory. Neural computation,
9(8):1735–1780.

Graham Neubig and Kevin Duh. 2014. On the elements of an accurate tree-to-string machine translation system. In Proc. ACL, pages 143–149.

Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito
Sudoh, and Hajime Tsukada. 2010. Automatic
evaluation of translation quality for distant language
pairs. In Proc. EMNLP, pages 944–952.

Graham Neubig, Yosuke Nakata, and Shinsuke Mori.
2011. Pointwise prediction for robust, adaptable
Japanese morphological analysis. In Proc. ACL,
pages 529–533.

Sébastien Jean, Kyunghyun Cho, Roland Memisevic,
and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In
Proc. ACL, pages 1–10.

Graham Neubig. 2013. Travatar: A forest-to-stringmachine translation engine based on tree transducers. In Proc. ACL Demo Track, pages 91–96.

Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent
continuous translation models. In Proc. EMNLP,
pages 1700–1709.

Graham Neubig. 2014. Forest-to-string SMT for
Asian language translation: NAIST at WAT2014. In
Proc. WAT.

Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proc. ACL, pages
423–430.

Franz Josef Och. 2003. Minimum error rate training instatistical machine translation. In Proc. ACL, pages
160–167.

Philipp Koehn. 2004. Statistical significance tests formachine translation evaluation. In Proc. EMNLP,
pages 388–395.

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. BLEU: a method for automatic
evaluation of machine translation. In Proc. ACL,
pages 311–318.

Jonathan K Kummerfeld, David Hall, James R Curran, and Dan Klein. 2012. Parser showdown at thewall street corral: an empirical investigation of error types in parser output. In Proc. EMNLP, pages
1048–1059.

Slav Petrov, Leon Barrett, Romain Thibaux, and Dan
Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proc. ACL, pages 433–
440.
40

Jason Riesa and Daniel Marcu. 2010. Hierarchicalsearch for word alignment. In Proc. ACL, pages
157–166.
Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014.
Sequence to sequence learning with neural networks. In Proc. NIPS, pages 3104–3112.
Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel
Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter for SIGHAN
bakeoff 2005. In Proc. SIGHAN.
David Vilar, Jia Xu, Luis Fernando d Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proc. LREC, pages 697–702.

41