Stronger Baselines for Trustable Results in Neural Machine Translation

Michael Denkowski
Amazon.com, Inc.
mdenkows@amazon.com

Abstract
Interest in neural machine translation has
grown rapidly as its effectiveness has
been demonstrated across language anddata scenarios.
New research regularly introduces architectural and algorithmic improvements that lead to significant
gains over “vanilla” NMT implementations. However, these new techniques
are rarely evaluated in the context of previously published techniques, specificallythose that are widely used in state-of-theart production and shared-task systems.
As a result, it is often difficult to determine whether improvements from research will carry over to systems deployed
for real-world use. In this work, we recommend three specific methods that are
relatively easy to implement and result inmuch stronger experimental systems. Beyond reporting significantly higher BLEU
scores, we conduct an in-depth analysis ofwhere improvements originate and whatinherent weaknesses of basic NMT models are being addressed. We then compare the relative gains afforded by several
other techniques proposed in the literaturewhen starting with vanilla systems versus
our stronger baselines, showing that experimental conclusions may change depending on the baseline chosen. This indicatesthat choosing a strong baseline is crucial
for reporting reliable experimental results.

1

Introduction

In the relatively short time since its introduction,neural machine translation has risen to prominence in both academia and industry. Neural models have consistently shown top performance inGraham Neubig
Carnegie Mellon University
gneubig@cs.cmu.edu

shared evaluation tasks (Bojar et al., 2016; Cettolo et al., 2016) and are becoming the technology
of choice for commercial MT service providers(Wu et al., 2016; Crego et al., 2016). New workfrom the research community regularly introduces
model extensions and algorithms that show significant gains over baseline NMT. However, thecontinuous improvement of real-world translation
systems has led to a substantial performance gap
between the first published neural translation models and the current state of the art. When promising new techniques are only evaluated on very basic NMT systems, it can be difficult to determine
how much (if any) improvement will carry overto stronger systems; is new work actually solving
new problems or simply re-solving problems thathave already been addressed elsewhere?
In this work, we recommend three specific techniques for strengthening NMT systems and empirically demonstrate how their use improves reliability of experimental results. We analyze in depth
how these techniques change the behavior of NMT
systems by addressing key weaknesses and discuss
how these findings can be used to understand theeffect of other types of system extensions. Our
recommended techniques include: (1) a training
approach using Adam with multiple restarts andlearning rate annealing, (2) sub-word translation
via byte pair encoding, and (3) decoding with ensembles of independently trained models.
We begin the paper content by introducing atypical NMT baseline system as our experimental starting point (§2.1). We then present and examine the effects of each recommended technique:
Adam with multiple restarts and step size annealing (§3), byte pair encoding (§4), and independent
model ensembling (§5). We show that combining
these techniques can lead to a substantial improvement of over 5 BLEU (§6) and that results for several previously published techniques can dramati-

cally differ (up to being reversed) when evaluatedon stronger systems (§6.2). We then conclude bysummarizing our findings (§7).

2

Experimental Setup

2.1

Translation System

Our starting point for experimentation is a standard baseline neural machine translation system
implemented using the Lamtram1 and DyNet2
toolkits (Neubig, 2015; Neubig et al., 2017). Thissystem uses the attentional encoder-decoder architecture described by Bahdanau et al. (2015), building on work by Sutskever et al. (2014). The translation model uses a bi-directional encoder with asingle LSTM layer of size 1024, multilayer perceptron attention with a layer size of 1024, andword representations of size 512. Translation
models are trained until perplexity convergenceon held-out data using the Adam algorithm witha maximum step size of 0.0002 (Kingma and Ba,
2015; Wu et al., 2016). Maximum training sentence length is set to 100 words. Model vocabulary
is limited to the top 50K source words and 50K target words by frequency, with all others mapped toan unk token. A post-processing step replaces any
unk tokens in system output by attempting a dictionary lookup3 of the corresponding source word(highest attention score) and backing off to copying the source word directly (Luong et al., 2015).
Experiments in each section evaluate this systemagainst incremental extensions such as improved
model vocabulary or training algorithm. Evaluation is conducted by average BLEU score overmultiple independent training runs (Papineni et al.,
2002; Clark et al., 2011).
2.2

Data Sets

We evaluate systems on a selection of public data
sets covering a range of data sizes, language directions, and morphological complexities. Thesesets, described in Table 1, are drawn from sharedtranslation tasks at the 2016 ACL Conference on
Machine Translation (WMT16)4 and the 2016 International Workshop on Spoken Language Translation (IWSLT16)5 .
1

https://github.com/neubig/lamtram
https://github.com/clab/dynet
3Translation dictionaries are learned from the system’s
training data using fast align (Dyer et al., 2013).
4
http://statmt.org/wmt16 (Bojar et al., 2016)5https://workshop2016.iwslt.org, https://wit3.fbk.eu (Cettolo et al., 2012)
2

Scenario
WMT German-English

Size (sent) Sources
4,562,102 Europarl,
Common Crawl,
news commentary
WMT English-Finnish
2,079,842 Europarl,
Wikipedia titles
WMT Romanian-English
612,422 Europarl, SETimesIWSLT English-French
220,400 TED talks
IWSLT Czech-English
114,390 TED talks
Scenario
DE-EN
EN-FI
RO-EN
EN-FR
CS-EN

Validation (Dev) Set
News test 2015
News test 2015
News dev 2016
TED test 2013+2014
TED test 2012+2013

Test Set
News test 2016
News test 2016
News test 2016
TED test 2015+2016
TED test 2015+2016

Table 1: Top: parallel training data available forall scenarios. Bottom: validation and test sets.

3

Training Algorithms

3.1

Background

The first neural translation models were optimized
with stochastic gradient descent (Sutskever et al.,2014). After training for several epochs witha fixed learning rate, the rate is halved at prespecified intervals. This widely used rate “annealing” technique takes large steps to move parameters from their initial point to a promising part ofthe search space followed by increasingly smaller
steps to explore that part of the space for a good
local optimum. While effective, this approach
can be time consuming and relies on hand-crafted
learning schedules that may not generalize to different models and data sets.
To eliminate the need for schedules, subsequent
NMT work trained models using the Adadelta algorithm, which automatically and continuously
adapts learning rates for individual parameters
during training (Zeiler, 2012). Model performance is reported to be equivalent to SGD withannealing, though training still takes a considerable amount of time (Bahdanau et al., 2015; Sennrich et al., 2016b). More recent work seeks
to accelerate training with the Adam algorithm,
which applies momentum on a per-parameter basis and automatically adapts step size subject to auser-specified maximum (Kingma and Ba, 2015).
While this can lead to much faster convergence,
the resulting models are shown to slightly underperform compared to annealing SGD (Wu et al.,2016). However, Adam’s speed and reputation

of generally being “good enough” have made it
a popular choice for researchers and NMT toolkit
authors6 (Arthur et al., 2016; Lee et al., 2016; Britzet al., 2017; Sennrich et al., 2017).
While differences in automatic metric scores
between SGD and Adam-trained systems may berelatively small, they raise the more general question of training effectiveness. In the following section, we explore the relative quality of the optima
found by these training algorithms.
3.2

Results and Analysis

To compare the behavior of SGD and Adam, we
conduct training experiments with all data setslisted in §2.2. For each set, we train instancesof the baseline model described in §2.1 with both
optimizers using empirically effective initial settings.7 In the only departure from the describedbaseline, we use a byte-pair encoded vocabulary
with 32K merge operations in place of a limited
full-word vocabulary, leading to faster training andhigher metric scores (see experiments in §4).
For SGD, we begin with a learning rate of 0.5
and train the model to convergence as measured
by dev set perplexity. We then halve the learning rate and restart training from the best previous
point. This continues until training has been run
a total of 5 times. The choice of training to convergence is made both to avoid the need for handcrafted learning schedules and to give the optimizers a better chance to find good neighborhoods
to explore. For Adam, we use a learning rate
(maximum step size) of 0.0002. While Adam’s
use of momentum can be considered a form of“self-annealing”, we also evaluate the novel extension of explicitly annealing the maximum step
size by applying the same halving and restarting
process used for SGD. It is important to note thatwhile restarting SGD has no effect beyond changing the learning rate, restarting Adam causes the
optimizer to “forget” the per-parameter learning
rates and start fresh.
For all training, we use a mini-batch size of
512 words.8 For WMT systems, we evaluate dev
6
Adam is the default optimizer for the Lamtram, Nematus (https://github.com/rsennrich/nematus),and Marian toolkits (https://github.com/amunmt/
marian).
7
Learning rates of 0.5 for SGD and 0.0002 for Adam orvery similar are shown to work well in NMT implementations
including GNMT (Wu et al., 2016), Nematus, Marian, andOpenNMT (http://opennmt.net).
8
For each mini-batch, sentences are added until the word

set perplexity every 50K training sentences for the
first training run and every 25K sentences for subsequent runs. For IWSLT systems, we evaluate every 25K sentences and then every 6,250 sentences.
Training stops when no improvement in perplexity
has been seen in 20 evaluations. For each experimental condition, we conduct 3 independent optimizer runs and report averaged metric scores. Alltraining results are visualized in Figure 1.
Our first observation is that these experiments
are largely in concert with prior work: Adam without annealing (first point) is significantly fasterthan SGD with annealing (last point) and often
comparable or slightly worse in accuracy, withthe exception of Czech-English where SGD underperforms. However, Adam with just 2 restarts and
SGD-style rate annealing is actually both fasterthan the fully annealed SGD and obtains significantly better results in both perplexity and BLEU.
We conjecture that the reason for this is twofold.
First, while Adam has the ability to automaticallyadjust its learning rate, like SGD it still benefits from an explicit adjustment when it has begun to overfit. Second, Adam’s adaptive learning
rates tend to reduce to sub-optimally low values
as training progresses, leading to getting stuck ina local optimum. Restarting training when reducing the learning rate helps jolt the optimizer out of
this local optimum and continue to find parametersthat are better globally.

4

Sub-Word Translation

4.1

Background

Unlike phrase-based approaches, neural translation models must limit source and target vocabulary size to keep computational complexity manageable. Basic models typically include the mostfrequent words (30K-50K) plus a single unk token to which all other words are mapped. As described in §2.1, unk words generated by the NMT
system are translated in post-processing by dictionary lookup or pass-through, often with significantly degraded quality (Luong et al., 2015). Realworld NMT systems frequently sidestep this problem with sub-word translation, where models operate on a fixed number of word pieces that canbe chained together to form words in an arbitrarcount is reached. Counting words versus sentences leads tomore uniformly-sized mini-batches. We choose the size of
512 based on contrastive experiments that found it to be thebest balance between speed and effectiveness of updates during training.

WMT German-English

26

12

15
20
Training Sentences (millions)

Adam
SGD
11
8
10
12
14
Training Sentences (millions)IWSLT English-French

IWSLT Czech-English

34
BLEU

33
32

BLEU

13

Adam
SGD

Adam
SGD
31
2.0
2.5
Training Sentences (millions)WMT Romanian-English
27

BLEU

BLEU

28

WMT English-Finnish

14

32
30

15

25

Adam
SGD
4
6
Training Sentences (millions)

22
BLEU

34

20

Adam
SGD
18
1.25 1.50 1.75
Training Sentences (millions)

DE-EN
EN-FI
RO-EN
EN-FR
CS-EN

SGD
(final)
6.05
17.97
11.20
6.09
15.16

Adam
(2r)
5.86
17.30
10.15
5.85
12.92

Dev Set Perplexities

Figure 1: Results of training the NMT models with Adam and SGD using rate annealing. Each point
represents training to convergence with a fixed learning rate and translating the test set. The learning rateis then halved and training resumed from the previous best point. Vertical dotted lines indicate 2 Adam
restarts. The table lists dev set perplexities for the final SGD model and the 2-restart Adam model. All
reported values are averaged over 3 independent training runs.
ily large vocabulary. In this section, we examine the impact of sub-words on NMT, specificallywhen using the technique of byte pair encoding(Sennrich et al., 2016b). Given the full parallel
corpus (concatenation of source and target sides),BPE first splits all words into individual charactersand then begins merging the most frequently adjacent pairs. Merged pairs become single units thatare candidates for further merging and the processcontinues to build larger word pieces for a fixed
number of operations. The final result is an encoded corpus where the most frequent words are
single pieces and less frequent words are split into
multiple, higher frequency pieces. At test time,words are split using the operations learned during training, allowing the model to translate witha nearly open vocabulary.9 The model vocabulary
size grows with and is limited by the number ofmerge operations. While prior work has focusedon using sub-words as a method for translating
9

It is possible that certain intermediate word pieces will
not appear in the encoded training data (and thus the model’s
vocabulary) if all occurrences are merged into larger units.
If these pieces appear in test data and are not merged, they
will be true OOVs for the model. For this reason, we map
singleton word pieces in the training data to unk so the model
has some ability to handle these cases (dictionary lookup or
pass-through).

WMT
IWSLT
DE-EN EN-FI RO-EN EN-FR CS-EN
Words 50K
31.6
12.6
27.1
33.6
21.0
BPE 32K
33.5
14.7
27.8
34.5
22.6
BPE 16K
33.1
14.7
27.8
34.8
23.0

Table 2: BLEU scores for training NMT models
with full word and byte pair encoded vocabularies.
Full word models limit vocabulary size to 50K.
All models are trained with annealing Adam andscores are averaged over 3 optimizer runs.
unseen words in morphologically rich languages(Sennrich et al., 2016b) or reducing model size
(Wu et al., 2016), we examine how using BPE actually leads to broad improvement by addressing
inherent weaknesses of word-level NMT.
4.2

Results and Analysis

We measure the effects of byte pair encoding bytraining full-word and BPE systems for all data
sets as described in §2.1 with the incremental
improvement of using Adam with rate annealing(§3). As Wu et al. (2016) show different levelsof effectiveness for different sub-word vocabulary
sizes, we evaluate running BPE with 16K and 32K

merge operations. As shown in Table 2, sub-wordsystems outperform full-word systems across theboard, despite having fewer total parameters. Systems built on larger data generally benefit fromlarger vocabularies while smaller systems performbetter with smaller vocabularies. Based on theseresults, we recommend 32K as a generally effective vocabulary size and 16K as a contrastive condition when building systems on less than 1 million parallel sentences.
To understand the origin of these improvements,
we divide the words in each test set into classes
based on how the full-word and BPE models handle them and report the unigram F-1 score foreach model on each class. We also plot the fullword and BPE vocabularies for context. As shownin Figure 2, performance is comparable for themost frequent words that both models represent
as single units. The identical shapes on the leftmost part of each vocabulary plot indicate thatthe two systems have the same number of training instances from which to learn translations. For
words that are split in the BPE model, performance is tied to data sparsity. With larger data,
performance is comparable as both models haveenough training instances to learn reliable statistics; with smaller data or morphologically rich languages such as Finnish, significant gains can berealized by modeling multiple higher-frequency
sub-words in place of a single lower-frequency
word. This can be seen as effectively moving to
the left in the vocabulary plot where translations
are more reliable. In the next category of words
beyond the 50K cutoff, the BPE system’s ability
to actually model rare words leads to consistentimprovement over the full-word system’s relianceon dictionary substitution.
The final two categories evaluate handling oftrue out-of-vocabulary items. For OOVs thatshould be translated, the full-word system will always score zero, lacking any mechanism for producing words not in its vocabulary or dictionary.
The more interesting result is in the relatively lowscores for OOVs that should simply be copiedfrom source to target. While phrase-based systems can reliably pass OOVs through 1:1, fullword neural systems must generate unk tokens
and correctly map them to source words using
attention scores. Differences in source and target true vocabulary sizes and frequency distributions often lead to different numbers of unk to-

kens in source and target sentences, resulting inmodels that are prone to over or under-generating
unks at test time. BPE systems address theseweaknesses, although their performance is not always intuitive. While some OOVs are successfully translated using word pieces, overall scores
are still quite low, indicating only limited successfor the notion of open vocabulary translation often associated with sub-word NMT. However, theability to learn when to self-translate sub-words10
leads to significant gains in pass-through accuracy.
In summary, our analysis indicates that while
BPE does lead to smaller, faster models, it alsosignificantly improves translation quality. Rather
than being limited to only rare and unseen words,
modeling higher-frequency sub-words in place oflower-frequency full words can lead to significant
improvement across the board. The specific improvement in pass-through OOV handling can beparticularly helpful for handling named entities
and open-class items such as numbers and URLs
without additional dedicated techniques.

5

Ensembles and Model Diversity

The final technique we explore is the combinationof multiple translation models into a single, more
powerful ensemble by averaging their predictions
at the word level. The idea of ensemble averaging is well understood and widely used acrossmachine learning fields and work from the earliest encoder-decoder papers to the most recent system descriptions reports dramatic improvements
in BLEU scores for model ensembles (Sutskever
et al., 2014; Sennrich et al., 2016a). While thistechnique is conceptually simple, it requires training and decoding with multiple translation models, often at significant resource costs. However,these costs are either mitigated or justified whenbuilding real-world systems or evaluating techniques that should be applicable to those systems.
Decoding costs can be reduced by using knowledge distillation techniques to train a single, compact model to replicate the output of an ensemble(Hinton et al., 2015; Kuncoro et al., 2016; Kim
and Rush, 2016). Researchers can skip this timeconsuming step, evaluating the ensemble directly,
while real-world system engineers can rely on it
to make deployment of ensembles practical. To re10
Learning a single set of BPE operations by concatenating
the source and target training data ensures that the same word
will always be segmented in the same way whether it appearson the source or target side.

WordsBPE 32K

103

50K

Occurrences

105

0.8

WMT German-English

0.6
0.4

F1

WMT DE-EN German

107

0.2

101

t

WMT English-Finnish
Words
BPE 32K

0.4

F1

Occurrences

WMT EN-FI English

-P

-T

OOV

50K

OOV

103

Dict

105

Words
BPE 32K

0.6

Spli

0.0

104
102
Word ID

Full

106

100

0.2

101

WMT RO-EN Romanian

WMT Romanian-English

103

50K

0.4

F1

0.6

Occurrences

105

Words
BPE 32K

0.2

101

-P
OOV
-T
OOV

t

Dict

0.0

Spli

104
102
Word ID

Full

100

-P

OOV

-T

OOV

t

Dict

0.0

Spli

104
102
Word ID

Full

100

IWSLT English-French
Words
BPE 16K

0.6
F1

0.4
0.2

103

IWSLT CS-EN Czech

Words
BPE 16K

50K

IWSLT Czech-English
Words
BPE 16K

0.4
0.2

101

-P

OOV

-T

OOV
Dict

0.0

t
Spli

104
102
Word ID

Full

100

0.6
F1

Occurrences

105

-P

OOV

-T

OOV
Dict

t
Spli

Full

0.0

Figure 2: Effects of using subword units on model vocabulary
and translation accuracy for specific types of words.
Left figures: Source vocabulary
visualizations for NMT training
data using full words and bytepair encoded tokens. The numberof merge operations is set to either 32K or 16K, chosen by best
BLEU score. BPE reduces vocabulary size by 1-2 orders ofmagnitude and allows models tocover the entire training corpus.
Full-word systems for all scenarios use a much larger vocabulary
size of 50K (labeled horizontalline) that leaves much of the total vocabulary uncovered.
Right figures: Class-wise test set
unigram F1 scores for NMT systems using full words and bytepair encoded tokens. Scores arereported separately for the following classes: words in the vocabulary of both the full-wordand BPE models (Full), words
in the vocabulary of the fullword model that are split in the
BPE model (Split), words outside
the vocabulary of the full-wordmodel but covered by its dictionary (Dict), words outside the vocabulary of the full-word modeland its dictionary that should betranslated (OOV-T), and words
outside the vocabulary of the fullword model and its dictionarythat should be passed through(OOV-P). All reported scores areaveraged over 3 independent optimizer runs.

WMT
IWSLT
DE-EN EN-FI RO-EN EN-FR CS-EN
Vanilla30.2
11.8 26.4
33.2
20.2
Recommended 33.5
14.7 27.8
34.5
22.6+Ensemble
35.8
17.3 30.3
37.3
25.5

Table 3: Test set BLEU scores for “vanilla” NMT(full words and standard Adam), and our recommended systems (byte pair encoding and annealing Adam, with and without ensembling). Scoresfor single models are averaged over 3 independent
optimizer runs while scores for ensembles are theresult of combining 3 runs.
duce training time, some work ensembles different
training checkpoints of the same model rather thanusing fully independent models (Jean et al., 2015;
Sennrich et al., 2016a). While checkpoint ensembling is shown to be effective for improving BLEU
scores under resource constraints, it does so withless diverse models. As discussed in recent workand demonstrated in our experiments in §6, model
diversity is a key component in building strong
NMT ensembles (Jean et al., 2015; Sennrich et al.,2016a; Farajian et al., 2016). For these reasons,
we recommend evaluating new techniques on systems that ensemble multiple independently trainedmodels for the most reliable results. Results showing both the effectiveness of ensembles and theimportance of model diversity are included in thelarger experiments conducted in the next section.

6

On Trustable Evaluation

6.1

Experimental Setup

In this section, we evaluate and discuss the effects that choice of baseline can have on experimental conclusions regarding neural MT systems.
First, we build systems that include Adam with
rate annealing, byte pair encoding, and independent model ensembling and compare them to thevanilla baselines described in §2.1. As shownin Table 3, combining these techniques leads toa consistent improvement of 4-5 BLEU points
across all scenarios. These improvements are
the result of addressing several underlying weaknesses of basic NMT models as described in previous sections, leading to systems that behave muchcloser to those deployed for real-world tasks.
Next, to empirically demonstrate the importance of evaluating new methods in the context
of these stronger systems, we select several tech-

EN-FR

Adam
+Annealing +Ensemble
Word BPE Word BPE
BPE
Baseline
33.2 33.7 33.6 34.8
37.3
Dropout
33.9 33.9 34.5 34.7
37.2
Lexicon Bias
33.8 34.0 33.9 34.8
37.1
Pre-Translation
–
34.0
–
34.9
36.6
Bootstrapping
33.7 34.1 34.4 35.2
37.4
CS-EN

Adam
+Annealing +Ensemble
Word BPE Word BPE
BPE
Baseline
20.2 22.1 21.0 23.0
25.5
Dropout
20.7 22.7 21.4 23.6
26.1
Lexicon Bias
20.7 22.5 20.6 22.7
25.2
Pre-Translation
–
23.1
–
23.8
25.8
Bootstrapping
20.7 23.2 21.6 23.6
26.2

Table 4: Test set BLEU scores for several published NMT extensions. Entries are evaluated withand without Adam annealing, byte pair encoding,and model ensembling. A bold score indicates improvement over the baseline while an italic scoreindicates no change or degradation. Scores for
non-ensembles are averaged over 3 independent
optimizer runs and ensembles are the result ofcombining 3 runs.
niques shown to improve NMT performance andcompare their effects as baseline systems are iteratively strengthened. Focusing on English-French
and Czech-English, we evaluate the following
techniques with and without the proposed improvements, reporting results in Table 4:
Dropout: Apply the improved dropout technique for sequence models described by Gal andGhahramani (2016) to LSTM layers with a rate of
0.2. We find this version to significantly outperform standard dropout.
Lexicon bias: Incorporate scores from a pretrained lexicon (fast align model learned on
the same data) directly as additional weights whenselecting output words (Arthur et al., 2016). Target word lexicon scores are computed as weighted
sums over source words based on attention scores.
Pre-translation: Translate source sentences witha traditional phrase-based system trained on thesame data. Input for the neural system is the original source sentence concatenated with the PBMT
output (Niehues et al., 2016). Input words are
prefixed with either s or t to denote source ortarget language. We improve performance with anovel extension where word alignments are used
to weave together source and PBMT output so thateach original word is immediately followed by its

suggested translation from the phrase-based system. As pre-translation doubles source vocabulary
size and input length, we only apply it to sub-wordsystems to keep complexity reasonable.
Data bootstrapping: Expand training data by extracting phrase pairs (sub-sentence translation examples) and including them as additional training
instances (Chen et al., 2016). We apply a novel extension where we train a phrase-based system anduse it to re-translate the training data, providing anear-optimal phrase segmentation as a byproduct.
We use these phrases in place of the heuristicallychosen phrases in the original work, improving
coverage and leading to more fine-grained translation examples.
6.2

Experimental Results

The immediately noticeable trend from Table 4is that while all techniques improve basic systems, only a single technique, data bootstrapping,
improves the fully strengthened system for bothdata sets (and barely so). This can be attributed
to a mix of redundancy and incompatibility between the improvements we’ve discussed in previous sections and the techniques evaluated here.
Lexicon bias and pre-translation both incorporate scores from pre-trained models that are shown
to improve handling of rare words. When NMT
models are sub-optimally trained, they can benefit from the suggestions of a better-trained model.
When full-word NMT models struggle to learntranslations for infrequent words, they can learn
to simply trust the lexical or phrase-based model.
However, when annealing Adam and BPE alleviate these underlying problems, the neural model’s
accuracy can match or exceed that of the pretrained model, making external scores either completely redundant or (in the worst case) harmful bias that must be overcome to produce correct translations. While pre-translation fares better than lexicon bias, it suffers a reversal in one
scenario and a significant degradation in the otherwhen moving from a single model to an ensemble. Even when bias from an external model improves translation, it does so at the cost of diversity
by pushing the neural model’s preferences toward
those of the pre-trained model. These results further validate claims of the importance of diversity
in model ensembles.
Applying dropout significantly improves all
configurations of the Czech-English system andsome configurations of the English-French system, leveling off with the strongest. This trend
follows previous work showing that dropout combats overfitting of small data, though the point of
inflection is worth noting (Sennrich et al., 2016a;
Wu et al., 2016). Even though the English-Frenchdata is still relatively small (220K sentences),
BPE leads to a smaller vocabulary of more general translation units, effectively reducing sparsity, while annealing Adam can avoid getting stuckin poor local optima. These techniques alreadylead to better generalization without the need for
dropout. Finally, we can observe a few key properties of data bootstrapping, the best performing
technique on fully strengthened systems. Unlike
lexicon bias and pre-translation, it modifies only
the training data, allowing “purely neural” models to be learned from random initialization points.
This preserves model diversity, allowing ensembles to benefit as well as single models. Further,data bootstrapping is complementary to annealing
Adam and BPE; better optimization and a moregeneral vocabulary can make better use of the new
training instances.
While evaluation on simple vanilla NMT systems would indicate that all of the techniques inthis section lead to significant improvement for
both data sets, only evaluation on systems using
annealing Adam, byte pair encoding, and independent model ensembling reveals both the reversals
of results on state-of-the-art systems and nuanced
interactions between techniques that we have reported. Based on these results, we highly recommend evaluating new techniques on systems thatare at least this strong and representative of those
deployed for real-world use.

7

Conclusion

In this work, we have empirically demonstrated
the effectiveness of Adam training with multiple
restarts and step size annealing, byte pair encoding, and independent model ensembling both for
improving BLEU scores and increasing the reliability of experimental results. Out of four previously published techniques for improving vanilla
NMT, only one, data bootstrapping via phrase extraction, also improves a fully strengthened modelacross all scenarios. For these reasons, we recommend evaluating new model extensions and algorithms on NMT systems at least as strong as those
we have described for maximally trustable results.

References
Philip Arthur, Graham Neubig, and Satoshi Nakamura.
2016. Incorporating discrete translation lexicons
into neural machine translation. In Proceedings ofthe 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1557–1567.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In International Conference on Learning Representations.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann,
Yvette Graham, Barry Haddow, Matthias Huck,
Antonio Jimeno Yepes, Philipp Koehn, Varvara
Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt
Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos
Zampieri. 2016. Findings of the 2016 conferenceon machine translation. In Proceedings of the First
Conference on Machine Translation. Association for
Computational Linguistics, Berlin, Germany, pages
131–198.
Denny Britz, Anna Goldie, Thang Luong, and Quoc
Le. 2017.
Massive exploration of neural machine translation architectures.
arXiv preprint
arXiv:1703.03906 .
M Cettolo, J Niehues, S Stüker, L Bentivogli, R Cattoni, and M Federico. 2016. The iwslt 2016 evaluation campaign .
Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit3 : Web inventory of transcribed andtranslated talks. In Proceedings of the 16th Conference of the European Association for MachineTranslation (EAMT). Trento, Italy, pages 261–268.
Wenhu Chen, Evgeny Matusov, Shahram Khadivi,
and Jan-Thorsten Peter. 2016. Guided alignment
training for topic-aware neural machine translation.
AMTA 2016, Vol. page 121.
Jonathan H. Clark, Chris Dyer, Alon Lavie, andNoah A. Smith. 2011. Better hypothesis testing forstatistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational
Linguistics: Human Language Technologies. pages
176–181.
Josep Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov,Patrice Brunelle, Aurelien Coquard, Yongchao
Deng, Satoshi Enoue, Chiyo Geiss, Joshua Johanson, Ardas Khalsa, Raoum Khiari, Byeongil Ko,
Catherine Kobus, Jean Lorieux, Leidiana Martins,
Dang-Chuan Nguyen, Alexandra Priori, Thomas
Riccardi, Natalia Segal, Christophe Servan, Cyril
Tiquet, Bo Wang, Jin Yang, Dakun Zhang, Jing
Zhou, and Peter Zoldan. 2016.
SYSTRAN’s

pure neural machine translation systems.
abs/1610.05540.

CoRR

Chris Dyer, Victor Chahuneau, and Noah A. Smith.
2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013
Conference of the North American Chapter of the
Association for Computational Linguistics: Human
Language Technologies. pages 644–648.
M Amin Farajian, Rajen Chatterjee, Costanza Conforti, Shahab Jalalvand, Vevake Balaraman, Mattia A Di Gangi, Duygu Ataman, Marco Turchi, Matteo Negri, and Marcello Federico. 2016. Fbks neural machine translation systems for iwslt 2016. In
Proceedings of the ninth International Workshop on
Spoken Language Translation (IWSLT), Seattle, WA.
Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In D. D. Lee, M. Sugiyama,
U. V. Luxburg, I. Guyon, and R. Garnett, editors,
Advances in Neural Information Processing Systems
29, Curran Associates, Inc., pages 1019–1027.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.
Distilling the knowledge in a neural network. arXiv
preprint arXiv:1503.02531 .
Sébastien Jean, Kyunghyun Cho, Roland Memisevic,
and Yoshua Bengio. 2015. On using very large
target vocabulary for neural machine translation.
In Proceedings of the 53rd Annual Meeting of the
Association for Computational Linguistics and the
7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China,
pages 1–10.
Yoon Kim and Alexander M. Rush. 2016. Sequencelevel knowledge distillation. In Proceedings of the2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1317–1327.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng
Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers
into one mst parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language
Processing. Association for Computational Linguistics, Austin, Texas, pages 1744–1753.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. Fully character-level neural machinetranslation without explicit segmentation. CoRR
abs/1610.03017.
Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals,
and Wojciech Zaremba. 2015. Addressing the rare

word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language
Processing (Volume 1: Long Papers). pages 11–19.
Graham Neubig. 2015. lamtram: A toolkit for language and translation modeling using neural networks. http://www.github.com/neubig/lamtram.
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin
Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel
Clothiaux, Trevor Cohn, Kevin Duh, Manaal
Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji,
Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke
Oda, Matthew Richardson, Naomi Saphra, Swabha
Swayamdipta, and Pengcheng Yin. 2017. Dynet:
The dynamic neural network toolkit. arXiv preprint
arXiv:1701.03980 .
Jan Niehues, Eunah Cho, Thanh-Le Ha, and Alex
Waibel. 2016. Pre-translation for neural machinetranslation. CoRR abs/1610.05243.
Kishore Papineni, Salim Roukos, Todd Ward, andWei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for
Computational Linguistics, Philadelphia, Pennsylvania, USA, pages 311–318.
Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin
Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a Toolkit for Neural MachineTranslation. In Proceedings of the Demonstrations
at the 15th Conference of the European Chapter ofthe Association for Computational Linguistics. Valencia, Spain.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016a. Edinburgh neural machine translation systems for wmt 16. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 371–
376.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016b. Neural machine translation of rare words
with subword units. In Proceedings of the 54th
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1715–1725.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V.
Le, Mohammad Norouzi, Wolfgang Macherey,
Maxim Krikun, Yuan Cao, Qin Gao, Klaus
Macherey, Jeff Klingner, Apurva Shah, MelvinJohnson, Xiaobing Liu, Lukasz Kaiser, Stephan
Gouws, Yoshikiyo Kato, Taku Kudo, Hideto
Kazawa, Keith Stevens, George Kurian, Nishant
Patil, Wei Wang, Cliff Young, Jason Smith, Jason
Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado,
Macduff Hughes, and Jeffrey Dean. 2016. Google’s
neural machine translation system: Bridging the gap
between human and machine translation. CoRR
abs/1609.08144.
Matthew D. Zeiler. 2012. ADADELTA: an adaptive
learning rate method. CoRR abs/1212.5701.