Guiding Neural Machine Translation with Retrieved Translation Pieces
Jingyi Zhang1,2 , Masao Utiyama1 , Eiichro Sumita1
Graham Neubig3,2 , Satoshi Nakamura2
1
National Institute of Information and Communications Technology, Japan
2
Graduate School of Information Science, Nara Institute of Science and Technology, Japan
3
Language Technologies Institute, Carnegie Mellon University, USA
jingyizhang/mutiyama/eiichiro.sumita@nict.go.jp
gneubig@cs.cmu.edu, s-nakamura@is.naist.jp
Abstract
One of the difficulties of neural machine translation (NMT) is the recall and appropriate
translation of low-frequency words or phrases.
In this paper, we propose a simple, fast, andeffective method for recalling previously seentranslation examples and incorporating theminto the NMT decoding process. Specifically,
for an input sentence, we use a search engine
to retrieve sentence pairs whose source sides
are similar with the input sentence, and thencollect n-grams that are both in the retrieved
target sentences and aligned with words thatmatch in the source sentences, which we call“translation pieces”. We compute pseudoprobabilities for each retrieved sentence basedon similarities between the input sentence andthe retrieved source sentences, and use theseto weight the retrieved translation pieces. Finally, an existing NMT model is used to translate the input sentence, with an additional
bonus given to outputs that contain the collected translation pieces. We show our method
improves NMT translation results up to 6
BLEU points on three narrow domain translation tasks where repetitiveness of the target
sentences is particularly salient. It also causes
little increase in the translation time, and compares favorably to another alternative retrievalbased method with respect to accuracy, speed,
and simplicity of implementation.

1

Introduction

Neural machine translation (NMT) (Bahdanau
et al., 2014; Sennrich et al., 2016a; Wang et al.,2017b) is now the state-of-the-art in machinetranslation, due to its ability to be trained end-toend on large parallel corpora and capture complex
parameterized functions that generalize across avariety of syntactic and semantic phenomena.
However, it has also been noted that comparedto alternatives such as phrase-based translation(Koehn et al., 2003), NMT has trouble with lowfrequency words or phrases (Arthur et al., 2016;
Kaiser et al., 2017), and also generalizing acrossdomains (Koehn and Knowles, 2017). A number of methods have been proposed to ameliorate
these problems, including methods that incorporate symbolic knowledge such as discrete translation lexicons (Arthur et al., 2016; He et al.,2016; Chatterjee et al., 2017) and phrase tables(Zhang et al., 2017; Tang et al., 2016; Dahlmann
et al., 2017), adjust model structures to be more
conducive to generalization (Nguyen and Chiang,
2017), or incorporate additional information about
domain (Wang et al., 2017a) or topic (Zhang et al.,2016) in translation models.
In particular, one paradigm of interest is recent
work that augments NMT using retrieval-basedmodels, retrieving sentence pairs from the training
corpus that are most similar to the sentence that
we want to translate, and then using these to bias
the NMT model.1 These methods – reminiscentof translation memory (Utiyama et al., 2011) or
example-based translation (Nagao, 1984; Grefenstette, 1999) – are effective because they augment
the parametric NMT model with a non-parametrictranslation memory that allows for increased capacity to measure features of the target technicalterms or domain-specific words. Currently there
are two main approaches to doing so. Li et al.
(2016) and Farajian et al. (2017) use the retrievedsentence pairs to fine tune the parameters of the
NMT model which is pre-trained on the wholetraining corpus. Gu et al. (2017) uses the retrievedsentence pairs as additional inputs to the NMT
model to help NMT in translating the input sen1

Note that there are existing retrieval-based methods
for phrase-based and hierarchical phrase-based translation(Lopez, 2007; Germann, 2015). However, these methods do
not improve translation quality but rather aim to improve theefficiency of the translation models.

Input:requirements

Retrieved:

requirements

inVorschriften

für

in

relationtotheoperational

relation

to

the

die

Eignungvonsuitability
suitability

Um@@

of

bulk

of

terminals

schlags@@

carriers

anlagenFigure 1: A word-aligned sentence pair retrieved for an input sentence. Red words are unedited words obtainedby computing the edit distance between the input sentence and the retrieved source sentence. The blue part of theretrieved target sentence is collected as translation pieces for the input sentence. The target word “Umschlagsanlagen” is split into “Um@@”, “schlags@@” and “anlagen” by byte pair encoding.

tence. While both of these paradigms have been
proven effective, they both add significant complexity and computational/memory cost to the decoding process, and also to the training procedure.
The first requires the running of several training
iterations and rolling back of the model, which iscostly at test time, and the second requires entirely
changing the model structure which requires training the model separately, and also increases testtime computational cost by adding additional encoders.
In this paper, we propose a simple and efficient model for using retrieved sentence pairs to
guide an existing NMT model at test time. Specifically, the model collects n-grams occurring in the
retrieved target sentences that also match words
that overlap between the input and retrieved source
sentences, which we will refer to as “translation
pieces” (e.g., in Figure 1, the blue part of the retrieved target sentence is collected as translation
pieces for the input sentence). The method thencalculates a pseudo-probability score for each of
the retrieved example sentence pairs and weights
the translation pieces according to this value. Finally, we up-weight NMT outputs that contain thecollected translation pieces. Unlike the previous
methods, this requires no change of the underlying
NMT model and no updating of the NMT parameters, making it both simple and efficient to apply
at test time.
We show our method improved NMT translation results up to 6 BLEU points on three translation tasks and caused little increase in the translation time. Further, we find that accuracies arecomparable with the model of Gu et al. (2017), despite being significantly simpler to implement andfaster at test time.

2

Attentional NMT

Our baseline NMT model is similar to the attentional model of Bahdanau et al. (2014), which
includes an encoder, a decoder and an attention (alignment) model. Given a source sentence
X = {x1 , ...,
h xL←},ithe encoder learns an annotation hi = ~hi ; h i for xi using a bi-directional
recurrent neural network.
The decoder generates the target translationfrom left to right. The probability of generatingnext word yt is,2

PN M T yt |y1t−1 , X = sof tmax (g (yt−1 , zt , ct ))(1)
where zt is a decoding state for time step t, computed by,
zt = f (zt−1 , yt−1 , ct )(2)

ct is a source representation for time t, calculated
as,
L
X
ct =
αt,i · hi(3)i=1

where αt,i scores how well the inputs around position i and the output at position t match, computedas,exp (a (zt−1 , hi ))αt,i = L(4)P
exp (a (zt−1 , hj ))j=1

The standard decoding algorithm for NMT isbeam search. That is, at each time step t, we keep
n-best hypotheses. The probability of a complete
2g, f and a in Equation 1, 2 and 4 are nonlinear, potentially multi-layered, functions.

log PN M T (Y |X) =|Y |X

log PN M T yt |y1t−1 , X



t=1

(5)
Finally, the translation score is normalized by sentence length to avoid too short outputs.
log SN M T (Y |X) =3

log PN M T (Y |X)
|Y |

(6)

Guiding NMT with Translation Pieces

This section describes our approach, which mainly
consists of two parts:
1. retrieving candidate translation pieces from aparallel corpus for the new source sentencethat we want to translate, and then
2. using the collected translation pieces to guidean existing NMT model while translating thisnew sentence.
At training time, we first prepare the parallel corpus that will form our database used in the retrievalof the translation pieces. Conceivably, it could
be possible to use a different corpus for translation piece retrieval and NMT training, for example when using a separate corpus for domain
adaptation, but for simplicity in this work we use
the same corpus that was used in NMT training.
As pre-processing, we use an off-the-shelf wordaligner to learn word alignments for the parallel
training corpus.
3.1

Retrieving Translation Pieces

At test time we are given an input sentence X.
For this X, we first use the off-the-shelf searchengine Lucene to search the word-aligned parallel
training corpus and retrieve M source sentences{X m : 1 ≤ m ≤ M } that are similar to X. Y m
indicates the target sentence that corresponds tosource sentence X m and Am is word alignments
between X m and Y m .
For each retrieved source sentence X m , wecompute its edit distance with X as d (X, X m )using dynamic programming. We record theunedited words in X m as W m , and also note thewords in the target sentence Y m that correspondto source words in W m , which we can presume
are words that will be more likely to appear in
the translated sentence for X. According to Algorithm 1, we collect n-grams (up to 4-grams) from

Gm
X
Yes
Yes
No
No

n-grams
Vorschriften für die Eignung
die Eignung von
von Um@@ schlags@@ anlagen
Um@@ schlags@@ anlagenhypothesis is computed as,

Table 1: Examples of the collected translation pieces.

the retrieved target sentence Y m as possible translation pieces Gm
X for X, using word-level alignments to select n-grams that are related to X anddiscard n-grams that are not related to X. The final translation pieces GX collected for X are computed as,3
M
[
GX =
Gm
(7)
X
m=1

Table 1 shows a few n-gram examples contained in the retrieved target sentence in Figure 1
and whether they are included in Gm
X or not. Because the retrieved source sentence in Figure 1 ishighly similar with the input sentence, the translation pieces collected from its target side are highly
likely to be correct translation pieces of the input
sentence. However, when a retrieved source sentence is not very similar with the input sentence(e.g. only one or two words match), the translation pieces collected from its target side will beless likely to be correct translation pieces for the
input sentence.
We compute a score for each u ∈ GX to measure how likely it is a correct translation piece for
X based on sentence similarity between the retrieved source sentences and the input sentence asfollowing,
S

u, X,

M
[

!
{(X

m

, Gm
X )}
(8)

m=1

=

max

1≤m≤M ∧u∈GmXmsimi (X, X )where simi (X, X m ) is the sentence similarity
computed as following (Gu et al., 2017),
simi (X, X m ) = 1 −3

d (X, X m )max (|X| , |X m |)(9)Note that the extracted translation pieces are target
phrases, but the target words contained in one extracted translation piece may be aligned to discontiguous source words,
which is different from how phrase-based translation extracts
phrase-based translation rules.

Algorithm 1 Collecting Translation Pieces
m = k L0 , Y m = v L00 , Am ,Require: X = xL
1, X
1
1
Wm

Ensure: Gm
X
Gm
X =∅
for i = 1 to L00 do
for j = i to L00 do
if j − i = 4 thenbreak
if ∃p : (p, j) ∈ Am ∧ p ∈/ W m thenbreak
add vij into Gm
X
Output layer of NMT

Matched translation pieces

𝑣1𝑣2
𝑦1 , . . . , 𝑦𝑡−1

𝑣3...
Predict the next word

𝑣𝑁 Reward𝑣2
𝑣3

𝑦𝑡−1 , 𝑣3
𝑦𝑡−2 , 𝑦𝑡−1 , 𝑣3

Target vocabulary

Figure 2: A simple demonstration of adding rewards
for matched translation pieces into the NMT output
layer.

3.2

Guiding NMT with Retrieved Translation
PiecesIn the next phase, we use our NMT system totranslate the input sentence. Inspired by Stahlberg
et al. (2017) which rewards n-grams from syntactic translation lattices during NMT decoding,we add an additional reward for n-grams that occur in the collected translation pieces. That is, as
shown in Figure 2, at each time step t, we update the probabilities over the output vocabulary
and increase the probabilities of those that resultin matched n-grams according tolog SN M T updated yt |y1t−1 , X= log PN M T yt |y1t−1 , X +! (10)
4MX[tλδ yt−n+1, X,{(X m , GmX )} ,
n=1

m=1

where λ can be tuned on the development set andtδ (·) is computed as Equation 8 if yt−n+1∈ GX ,
otherwise δ (·) = 0.
To implement our method, we use a dictionary

Algorithm 2 Guiding NMT by Translation Pieces

Require: Output layer log PN M T yt |y1t−1 , X ,LX , DX
Ensure: Updated output layer
for u in LX dolog PN M T u|y1t−1 , X + = λDX (u)for i = 1 to 3 do
if t − i < 1 then
break
t−1
if yt−i,u ∈/ DX then
break


t−1
log PN M T u|y1t−1 , X +=λDX yt−i,u
DX to store translation pieces GX and their scoresfor each input sentence X. At each time step t, weupdate the output layer probabilities by checking
DX . However, it is inefficient to traverse all targetwords in the vocabulary and check whether they
belong to GX or not, because the vocabulary sizeis large. Instead, we only traverse target words thatbelong to GX and update the corresponding output
probabilities as shown in Algorithm 2. Here, LX
is a list that stores 1-grams contained in GX .4
As we can see, our method only up-weights
NMT outputs that match the retrieved translation
pieces in the NMT output layer. In contrast, Li
et al. (2016) and Farajian et al. (2017) use theretrieved sentence pairs to run additional training iterations and fine tune the NMT parameters
for each input sentence; Gu et al. (2017) runs the
NMT model for each retrieved sentence pair to obtain the NMT encoding and decoding informationof the retrieved sentences as key-value memory to
guide NMT for translating the new input sentence.
Compared to their methods, our method adds little
computational/memory cost and is simple to implement.

4

Experiment

4.1

Settings

Following Gu et al. (2017), we use version 3.0 of
the JRC-Acquis corpus for our translation experiments. The JRC-Acquis corpus contains the total
body of European Union (EU) law applicable in
the EU Member States. It can be used as a narrowdomain to test the effectiveness of our proposed
method. We did translation experiments on three
4

Note that our method does not introduce new states during decoding, because the output layer probabilities are simply updated based on history words and the next word.

dev
test

NMT
Ours
NMT
Ours

en-de
BLEU
44.08
50.81
43.76
50.15

METEOR
36.69
39.50
36.57
39.18

en-fr
BLEU
57.26
62.60
57.67
63.27

METEOR
43.51
45.83
43.66
46.24

en-es
BLEU
55.76
60.51
55.78
60.54

METEOR
42.53
44.58
42.55
44.64

Table 2: Translation results.

TRAIN
DEV
TEST
Average Length

en-de
674K
1,636
1,689
31

en-fr
665K
1,733
1,710
29

en-es
663K
1,662
1,696
29

Table 3: Data sets. The last line is the average lengthof English sentences.

directions: English-to-German (en-de), Englishto-French (en-fr) and English-to-Spanish (en-es).
We cleaned the data by removing repeated sentences and used the train-truecaser.perl
script from Moses (Koehn et al., 2007) to truecase
the corpus. Then we selected 2000 sentence pairs
as development and test sets, respectively. The rest
was used as the training set. We removed sentences longer than 80 and 100 from the training
and development/test sets respectively. The final
numbers of sentence pairs contained in the training, development and test sets are shown in Table 3.5 We applied byte pair encoding (Sennrichet al., 2016b) and set the vocabulary size to be
20K.
For translation piece collection, we use
GIZA++ (Och and Ney, 2003) and thegrow-diag-final-and heuristic (Koehn
et al., 2003) to obtain symmetric word alignments
for the training set.
We trained an attentional NMT model as ourbaseline system. The settings for NMT are shownin Table 4. We also compared our method with thesearch engine guided NMT model (SGNMT, Gu
et al. (2017)) in Section 4.5.
Word embedding
GRU dimension
Optimizer
Initial learning rate
Beam size

512
1024
adam
0.0001
5

Table 4: NMT settings.
5
We put the datasets used in our experiments on Github
https://github.com/jingyiz/Data-sampled-preprocessed

dev
test

NMT
Ours
NMT
Ours

en-de
1.000
1.005
0.995
1.004

en-fr
0.990
0.991
0.990
0.989

en-es
0.997
1.001
0.990
0.993

Table 5: Ratio of translation length to reference length.

For each input sentence, we retrieved 100 sentence pairs from the training set using Lucene as
our preliminary setting. We analyze the influenceof the retrieval size in Section 4.4. The weights oftranslation pieces used in Equation 10 are tuned on
the development set for different language pairs,
resulting in weights of 1.5 for en-de and en-fr, anda weight of 1 for en-es.
4.2

Results

Table 2 shows the main experimental results. We
can see that our method outperformed the baseline NMT system up to 6 BLEU points. As large
BLEU gains in neural MT can also often be attributed to changes in output length, we examined
the length (Table 5) and found that it did not influence the translation length significantly.
In addition, it is of interest whether how well the
retrieved sentences match the input influences thesearch results. We measure the similarity between
a test sentence X and the training corpus Dtrain
by computing the sentence similarities between X
and the retrieved source sentences assimi (X, Dtrain ) = max simi (X, X m ) .
1≤m≤M

(11)
The similarity between the test set Dtest and thetraining corpus Dtrain is measured as,Psimi (Dtest , Dtrain ) =X∈Dtest simi (X, Dtrain )|Dtest |(12)

Our analysis demonstrated that, expectedly, theperformance of our method is highly influenced by
the similarity between the test set and the training
set. We divided sentences in the test set into twoen-de
en-fr
en-es

whole
0.56
0.57
0.57

half-H
0.80
0.81
0.81

half-L
0.32
0.33
0.32

Table 6: Similarities between the training set and thewhole/divided test sets.

en-de
en-fr
en-es

NMT
Ours
NMT
Ours
NMT
Ours

whole
43.76
50.15
57.67
63.27
55.78
60.54

half-H
60.93
73.26
72.64
82.76
69.32
78.37

half-L
32.25
34.28
47.38
49.81
46.26
47.93

Table 7:
Translation results (BLEU) for thewhole/divided test sets.

parts: half has higher similarities with the training corpus (half-H) and half has lower similarities
with the training corpus (half-L). Table 6 shows
the similarity between the training corpus and thewhole/divided test sets. Table 7 shows translation
results for the whole/divided test sets. As we cansee, NMT generally achieved better BLEU scoresfor half-H and our method improved BLEU scoresfor half-H much more significantly than for halfL, which shows our method can be quite usefulfor narrow domains where similar sentences canbe found.
We also tried our method on WMT 2017
English-to-German News translation task. However, we did not achieve significant improvements
over the baseline attentional NMT model, likely
because the test set and the training set for the
WMT task have a relatively low similarity as
shown in Table 8 and hence few useful translation pieces can be retrieved for our method. In
contrast, the JRC-Acquis corpus provides test sentences that have much higher similarities with thetraining set, i.e., much more and longer translation
pieces exist.
To demonstrate how the retrieved translation
pieces help NMT to generate appropriate outputs,
Figure 3 shows an input sentence with reference,
the retrieved sentence pair with the highest sentence similarity and outputs by different systems
for this input sentence with detailed scores: log
NMT probabilities for each target word in T1 andT2 ; scores for matched translation pieces contained in T1 and T2 . As we can see, NMT as-

Similarity[0, 0.1)[0.1, 0.2)
[0.2, 0.3)
[0.3, 0.4)
[0.4, 0.5)
[0.5, 0.6)
[0.6, 0.7)
[0.7, 0.8)
[0.8, 0.9)
[0.9, 1)
1

WMT
Sent Percent
0
0%
415
13.8%
1399
46.5%
740
24.6%
281
9.3%
113
3.7%
29
0.9%
10
0.3%
10
0.3%
0
0%
7
0.2%

JRC-Acquis
Sent Percent
4
0.2%
141
8.3%
238
14.0%
194
11.4%
154
9.1%
156
9.2%
157
9.2%
156
9.2%
252
14.9%
237
14.0%
0
0%

Table 8: Statistics for similarities between each test
sentence and the training set as computed by Equation 11 for the WMT 2017 en-de task (3004 sentences)and our JRC-Acquis en-de task (1689 sentences).

dev

test

NMT
Ours
1/0 reward
NMT
Ours
1/0 reward

en-de
44.08
50.81
47.70
43.76
50.15
47.13

en-fr
57.26
62.60
61.15
57.67
63.27
62.14

en-es
55.76
60.51
58.92
55.78
60.54
58.66

Table 9: Translation results (BLEU) of 1/0 reward.

signs higher probabilities to the incorrect translation T1 , even though the retrieved sentence pairwhose source side is very similar with the input
sentence was used for NMT training.
However, T2 contains more and longer translation pieces with higher scores. The five translation pieces contained only in T2 are collectedfrom the retrieved sentence pair shown in Figure 3,
which has high sentence similarity with the input
sentence. The three translation pieces contained
only in T1 are also translation pieces collected forthe input sentence, but have lower scores, because
they are collected from sentence pairs with lower
similarities with the input sentence. This showsthat computing scores for translation pieces basedon sentence similarities is important for the performance of our method. If we assign score 1 to alltranslation pieces contained in GX , i.e., use 1/0
reward for translation pieces and non-translation
pieces, then the performance of our method decreased significantly as shown in Table 9, but still
outperformed the NMT baseline significantly.

Source
Reference
Retrieved
SourceReference
Translation
T1 (NMT)
NMT scores
Our scores

T2 (Ours)
NMT scores
Our scores

requirements in relation to the operational suitability of bulk carriers
Vorschriften für die betriebliche Eignung von Massen@@ gut@@ schiffen
requirements in relation to the suitability of terminals
Vorschriften für die Eignung von Um@@ schlags@@ anlagen
Anforderungenan
die betriebliche Eignung von Massen@@ gut@@ schiffen 
-0.81 -0.34 -0.02-0.69-0.13 -0.12
-0.33
-0.13
-0.71 -0.01
0.16 0.21
0.7
0.17
0.7
0.7
0.35
0.35
0.35
0.21
0.7
0.35
0.35
0.35
0.35
Green
0.35
0.35
Vorschriftenfür
die betriebliche Eignung von Massen@@ gut@@ schiffen 
-2.71 -0.72 -0.10-0.91-0.13 -0.12
-0.33
-0.12
-0.69 -0.02
0.7
0.7
0.7
0.17
0.7
0.7
0.35
0.35
0.35
0.7
0.7
0.35
0.7
0.35
0.7
0.35
0.35
0.35
Yellow
0.35

Figure 3: Translation examples. Red scores are log NMT probabilities. Green, yellow and blue scores are scoresof matched translation pieces contained only in T1 , contained only in T2 , contained in both T1 and T2 , respectively.
γ
en-de
en-fr
en-es

NMT
Ours
Ratio (Ours/NMT)
NMT
Ours
Ratio (Ours/NMT)
NMT
Ours
Ratio (Ours/NMT)

0
5834
5843
1.00
6983
7058
1.01
6500
6516
1.00

1
3193
5433
1.70
3743
5443
1.45
3430
4589
1.33

2
1988
3153
1.58
2637
3584
1.35
2292
2970
1.29

5
1196
1690
1.41
1563
1919
1.22
1346
1652
1.22

10
717
933
1.30
812
968
1.19
772
895
1.15

20
370
458
1.23
493
581
1.17
437
500
1.14

50
157
193
1.22
210
214
1.01
182
196
1.07

100
75
86
1.14
118
134
1.13
95
97
1.02

Table 10: Countγ

4.3

Infrequent n-grams

The basic idea of our method is rewarding ngrams that occur in the training set during NMT
decoding. We found our method is especially useful to help the translation for infrequent n-grams.
First, we count how many times a target n-gram u
occurs in the training set Dtrain as,
Occur (u) = |{Y : hX, Y i ∈ Dtrain ∧ u ∈ uniq (Y )}|(13)

where uniq (Y ) is the set of uniq n-grams (up to
4-grams) contained in Y .
 kGiven system
outputs
Z : 1 ≤ k ≤ K for kthe test set X : 1 ≤ k ≤ K with referenceY k : 1 ≤ k ≤ K , we count the number of cor-

rectly translated n-grams that occur γ times in the
training set as,Countγ =KXψ γ, Z k , Y k

(14)

k=1

where


ψ γ, Z k , Y k =
n

 
 
o
u : u ∈ uniq Z k ∩ uniq Y k∧ Occur (u) = γ(15)Table 10 shows Countγ for different system
outputs. As we can see, our method helped little for the translation of n-grams that do not occur

Base NMT decoding
Search engine retrieval
TP collection
Our NMT decoding

en-de
0.215
0.016
0.521
0.306

en-fr
0.224
0.017
0.522
0.287

en-es
0.227
0.016
0.520
0.289

0.3
0.28
0.26
0.24
0.22

Table 11: Translation time (seconds).
0.6

0.2
0

1

2

5

10

20

50

100

Figure 5: NMT decoding time (seconds) with different
search engine retrieval sizes.

0.45
0.3

63

0.15

62.5

0
1

2

5

10

20

50

100

Figure 4: Translation piece collection time (seconds)
with different search engine retrieval sizes.

62
61.5
61
1

in the training set, which is reasonable because weonly reward n-grams that occur in the training set.
However, our method helped significantly for thetranslation of n-grams that do occur in the training set but are infrequent (occur less than 5 times).
As the frequency of n-grams increases, the improvement caused by our method decreased. We
analyze that the reason why our method is especially helpful for infrequent n-grams is that NMT
is trained on the whole training corpus for maximum likelihood and tends to generate more frequent n-grams while our method computes scoresfor the collected translation pieces based on sentence similarities and does not prefer more frequent n-grams.
4.4

Computational Considerations

Our method only collects translation pieces to help
NMT for translating a new sentence and does not
influence the training process of NMT. Therefore,
our method does not increase the NMT training
time. Table 11 shows the average time needed fortranslating one input sentence in the development
set in our experiments. The search engine retrieval
and translation piece (TP) collection time is computed on a 3.47GHz Intel Xeon X5690 machineusing one CPU. The NMT decoding time is computed using one GPU GeForce GTX 1080.
As we can see, the search engine retrieval timeis negligible and the increase of NMT decoding
time caused by our method is also small. However,

2

5

10

20

50

100

Figure 6: Translation results (BLEU) with different
search engine retrieval sizes.

collecting translation pieces needed considerable
time, although our implementation was in Python
and could potentially be significantly faster in amore efficient programming language. The translation piece collection step mainly consists of twoparts: computing the edit distances between the
input sentence and the retrieved source sentencesusing dynamic
programming with time complexity O n2 ; collecting translation pieces using Algorithm 1 with time complexity O (4n).
We changed the size of sentence pairs retrieved by the search engine and analyze its influence on translation performance and time. Figure 4, 5 and 6 show the translation piece collection time, the NMT decoding time and translation BLEU scores with different search engine retrieval sizes for the en-fr task. As we can see,as the number of retrieved sentences decreased,
the time needed by translation piece collection decreased significantly, the translation performancedecreased much less significantly and the NMT
decoding time is further reduced. In our experiments, 10 is a good setting for the retrieval size,
which gave significant BLEU score improvements
and caused little increase in the total translation
time compared to the NMT baseline.

dev

test

NMTreported
SGNMTreported
NMT
Ours
NMTreported
SGNMTreported
NMT
Ours

en-de
44.94
49.26
45.18
50.61
43.98
48.80
44.21
50.36

en-fr
58.95
64.16
59.08
65.03
59.42
64.60
59.43
65.69

en-es
50.54
57.62
50.71
57.49
50.48
57.27
50.61
57.11

Table 12: Comparison with SGNMT.

4.5

Comparison with SGNMT

We compared our method with the search engine guided NMT (SGNMT) model (Gu et al.,2017). We got their preprocessed datasets andtested our method on their datasets, in order tofairly compare our method with their reported
BLEU scores.6 Table 12 shows the results of their
method and our method with the same settings forthe baseline NMT system. As we can see, ourmethod generally outperformed their method on
the three translation tasks.
Considering the computational complexity,
their method also performs search engine retrieval
for each input sentence and computes the edit distance between the input sentence and the retrieved
source sentences as our method. In addition, their
method runs the NMT model for each retrievedsentence pair to obtain the NMT encoding and decoding information of the retrieved sentences as
key-value memory to guide the NMT model fortranslating the real input sentence, which changes
the NMT model structure and increases both thetraining-time and test-time computational cost.
Specifically, at test time, running the NMT modelfor one retrieved sentence pair costs the same timeas translating the retrieved source sentence withbeam size 1. Therefore, as the number of the retrieved sentence pairs increases to the beam sizeof the baseline NMT model, their method doubles
the translation time.

5

Conclusion

This paper presents a simple and effective methodthat retrieves translation pieces to guide NMT for
narrow domains. We first exploit a search engine
to retrieve sentence pairs whose source sides are
similar with the input sentence, from which we
6

Only BLEU scores are reported in their paper.

collect and weight translation pieces for the input sentence based on word-level alignments andsentence similarities. Then we use an existing
NMT model to translate this input sentence andgive an additional bonus to outputs that containthe collected translation pieces. We show ourmethod improved NMT translation results up to
6 BLEU points on three narrow domain translation tasks, caused little increase in the translation
time, and compared favorably to another alternative retrieval-based method with respect to accuracy, speed, and simplicity of implementation.

Acknowledgments
We thank Jiatao Gu for providing their preprocessed datasets in Section 4.5.

References
Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 1557–
1567. https://aclweb.org/anthology/
D16-1162.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint
arXiv:1409.0473 https://arxiv.org/pdf/
1409.0473.pdf.
Rajen Chatterjee, Matteo Negri, Marco Turchi, Marcello Federico, Lucia Specia, and Frédéric Blain.
2017. Guiding neural machine translation decoding with external knowledge. In Proceedings of
the Second Conference on Machine Translation.
pages 157–168. http://www.aclweb.org/
anthology/W17-4716.
Leonard Dahlmann, Evgeny Matusov, Pavel
Petrushkov, and Shahram Khadivi. 2017. Neural machine translation leveraging phrase-basedmodels in a hybrid search. In Proceedings ofthe 2017 Conference on Empirical Methods in
Natural Language Processing. pages 1411–1420.
https://www.aclweb.org/anthology/
D17-1148.
M. Amin Farajian, Marco Turchi, Matteo Negri,and Marcello Federico. 2017.
Multi-domain
neural machine translation through unsupervised
adaptation. In Proceedings of the Second Conference on Machine Translation. pages 127–137.
http://www.aclweb.org/anthology/W/
W17/W17-4713.pdf.

Ulrich Germann. 2015.
Sampling phrase tables
for the moses statistical machine translation system. The Prague Bulletin of Mathematical Linguistics 104(1):39–50. https://ufal.mff.
cuni.cz/pbml/104/art-germann.pdf.
Gregory Grefenstette. 1999. The world wide web
as a resource for example-based machine translation tasks. In Proceedings of the ASLIB Conference on Translating and the Computer. volume 21. http://www.mt-archive.info/
Aslib-1999-Grefenstette.pdf.
Jiatao Gu, Yong Wang, Kyunghyun Cho, andVictor OK Li. 2017.
Search engine guided
non-parametric neural machine translation.
arXiv preprint arXiv:1705.07267
https://arxiv.org/pdf/1705.07267.pdf.
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang.
2016. Improved neural machine translation withsmt features. In AAAI. pages 151–157. https://www.aaai.org/ocs/index.php/AAAI/
AAAI16/paper/view/12189/11577.
Łukasz Kaiser, Ofir Nachum, Aurko Roy, andSamy Bengio. 2017. Learning to remember rare
events. arXiv preprint arXiv:1703.03129 https://arxiv.org/pdf/1703.03129.pdf.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra
Constantin, and Evan Herbst. 2007. Moses: Open
source toolkit for statistical machine translation. In
Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion
Volume Proceedings of the Demo and Poster Sessions. pages 177–180. http://www.aclweb.
org/anthology/P07-2045.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. pages 28–39. http://www.
aclweb.org/anthology/W17-3204.
Philipp Koehn, Franz Josef Och, and Daniel Marcu.
2003. Statistical phrase-based translation. In
Proceedings of the 2003 Conference of the North
American Chapter of the Association for Computational Linguistics on Human Language TechnologyVolume 1. pages 48–54. http://www.aclweb.
org/anthology/N/N03/N03-1017.pdf.
Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2016.
One sentence one model for neural machine translation. arXiv preprint arXiv:1609.06490 https://arxiv.org/pdf/1609.06490.pdf.
Adam Lopez. 2007. Hierarchical phrase-based translation with suffix arrays. In Proceedings of the2007 Joint Conference on Empirical Methods in
Natural Language Processing and Computational

Natural Language Learning (EMNLP-CoNLL).
pages 976–985. http://www.aclweb.org/
anthology/D/D07/D07-1104.
Makoto Nagao. 1984. A framework of a mechanicaltranslation between japanese and english by analogy
principle. Artificial and human intelligence pages
351–354. http://www.mt-archive.info/
Nagao-1984.pdf.
Toan Q Nguyen and David Chiang. 2017.
Improving lexical choice in neural machine translation. arXiv preprint arXiv:1710.01329 https://arxiv.org/pdf/1710.01329.pdf.
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational linguistics 29(1):19–
51. http://www.aclweb.org/anthology/
J/J03/J03-1002.pdf.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016a.
Edinburgh neural machine translation
systems for WMT 16.
In Proceedings ofthe First Conference on Machine Translation.
pages 371–376. http://www.aclweb.org/
anthology/W/W16/W16-2323.pdf.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016b. Neural machine translation of rare words
with subword units. In Proceedings of the 54th
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
pages 1715–1725. http://www.aclweb.org/
anthology/P/P16/P16-1162.pdf.
Felix Stahlberg, Adrià de Gispert, Eva Hasler, andBill Byrne. 2017. Neural machine translation byminimising the Bayes-risk with respect to syntactic translation lattices. In Proceedings of the 15th
Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short
Papers. pages 362–368. http://www.aclweb.
org/anthology/E/E17/E17-2058.pdf.
Yaohua Tang, Fandong Meng, Zhengdong Lu, Hang Li,
and Philip LH Yu. 2016. Neural machine translation with external phrase memory. arXiv preprint
arXiv:1606.01792 https://arxiv.org/pdf/
1606.01792.pdf.
Masao Utiyama, Graham Neubig, Takashi Onishi, and Eiichiro Sumita. 2011.
Searchingtranslation memories for paraphrases.
In Machine Translation Summit. volume 13, pages
325–331.
http://mt-archive.info/
MTS-2011-Utiyama.pdf.
Rui Wang, Andrew Finch, Masao Utiyama, and Eiichiro Sumita. 2017a. Sentence embedding forneural machine translation domain adaptation. In
Proceedings of the 55th Annual Meeting of the
Association for Computational Linguistics (Volume
2: Short Papers). pages 560–566. http://
aclweb.org/anthology/P17-2089.

Yuguang Wang, Shanbo Cheng, Liyang Jiang, Jiajun Yang, Wei Chen, Muze Li, Lin Shi, Yanfeng
Wang, and Hongtao Yang. 2017b. Sogou neural machine translation systems for wmt17. In Proceedings of the Second Conference on Machine Translation. pages 410–415. http://www.aclweb.
org/anthology/W17-4742.
Jiacheng Zhang, Yang Liu, Huanbo Luan, Jingfang
Xu, and Maosong Sun. 2017. Prior knowledge
integration for neural machine translation using
posterior regularization. In Proceedings of the
55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
pages 1514–1523. http://www.aclweb.org/
anthology/P/P17/P17-1139.pdf.
Jian Zhang, Liangyou Li, Andy Way, and Qun Liu.
2016. Topic-informed neural machine translation.
In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics:
Technical Papers. pages 1807–1817. http://
aclweb.org/anthology/C16-1170.