Learning to Translate in Real-time with Neural Machine Translation
†

‡

♦

†

Jiatao Gu , Graham Neubig , Kyunghyun Cho and Victor O.K. Li†

Abstract
Translating in real-time, a.k.a. simultaneous translation, outputs translation words
before the input sentence ends, which is achallenging problem for conventional machine translation methods. We propose aneural machine translation (NMT) framework for simultaneous translation in whichan agent learns to make decisions on whento translate from the interaction with apre-trained NMT environment. To trade
off quality and delay, we extensively explore various targets for delay and design
a method for beam-search applicable inthe simultaneous MT setting. Experiments
against state-of-the-art baselines on two
language pairs demonstrate the efficacyof the proposed framework both quantitatively and qualitatively.1

1

‡

♦

The University of Hong Kong Carnegie Mellon University New York University†♦{jiataogu, vli}@eee.hku.hk gneubig@cs.cmu.edu‡
kyunghyun.cho@nyu.edu

Introduction

Simultaneous translation, the task of translating
content in real-time as it is produced, is an important tool for real-time understanding of spoken
lectures or conversations (Fügen et al., 2007; Bangalore et al., 2012). Different from the typicalmachine translation (MT) task, in which translation quality is paramount, simultaneous translation
requires balancing the trade-off between translation quality and time delay to ensure that users
receive translated content in an expeditious manner (Mieno et al., 2015). A number of methods
have been proposed to solve this problem, mostly
in the context of phrase-based machine translation.
These methods are based on a segmenter, which
receives the input one word at a time, then decides
when to send it to a MT system that translates each1
Code and data can be found at https://github.
com/nyu-dl/dl4mt-simul-trans.

st ighnLa

Gestern
Abend
haben
wir
Herrn
X
ein
Bier
serviert
,
der
im
Laufe
der
Nacht
ge-storben
ist.
< eos >

t

ed

rv

we se

Mr

X

a

er

be

,

>
ng
t
os
o
ed ri
e
gh
wh di du
th ni
. <e

READ
WRITE

Figure 1: Example output from the proposed
framework in DE → EN simultaneous translation. The heat-map represents the soft alignment
between the incoming source sentence (left, upto-down) and the emitted translation (top, leftto-right). The length of each column represents
the number of source words being waited for before emitting the translation. Best viewed whenzoomed digitally.
segment independently (Oda et al., 2014) or with aminimal amount of language model context (Bangalore et al., 2012).
Independently of simultaneous translation, accuracy of standard MT systems has greatly improved with the introduction of neural-networkbased MT systems (NMT) (Sutskever et al., 2014;
Bahdanau et al., 2014). Very recently, there havebeen a few efforts to apply NMT to simultaneous translation either through heuristic modifications to the decoding process (Cho and Esipova,
2016), or through the training of an independent
segmentation network that chooses when to perform output using a standard NMT model (Satija
and Pineau, 2016). However, the former modellacks a capability to learn the appropriate timingwith which to perform translation, and the lattermodel uses a standard NMT model as-is, lacking a holistic design of the modeling and learning
within the simultaneous MT context. In addition,neither model has demonstrated gains over previ-ous segmentation-based baselines, leaving questions of their relative merit unresolved.
In this paper, we propose a unified design forlearning to perform neural simultaneous machinetranslation. The proposed framework is based on
formulating translation as an interleaved sequenceof two actions: READ and WRITE. Based on this,we devise a model connecting the NMT systemand these READ/WRITE decisions. An exampleof how translation is performed in this frameworkis shown in Fig. 1, and detailed definitions of the
problem and proposed framework are described in§2 and §3. To learn which actions to take when, we
propose a reinforcement-learning-based strategy
with a reward function that considers both quality and delay (§4). We also develop a beam-searchmethod that performs search within the translation
segments (§5).
We evaluate the proposed method on EnglishRussian (EN-RU) and English-German (EN-DE)
translation in both directions (§6). The quantitative results show strong improvements compared
to both the NMT-based algorithm and a conventional segmentation methods. We also extensively
analyze the effectiveness of the learning algorithm
and the influence of the trade-off in the optimization criterion, by varying a target delay. Finally,
qualitative visualization is utilized to discuss thepotential and limitations of the framework.

2

Problem Definition

Suppose we have a buffer of input words X ={x1 , ..., xTs } to be translated in real-time. We define the simultaneous translation task as sequentially making two interleaved decisions: READ or
WRITE . More precisely, the translator READ s asource word xη from the input buffer in chronological order as translation context, or WRITEs atranslated word yτ onto the output buffer, resulting
in output sentence Y = {y1 , ..., yTt }, and action
sequence A = {a1 , ..., aT } consists of Ts READs
and Tt WRITEs, so T = Ts + Tt .
Similar to standard MT, we have a measure
Q(Y ) to evaluate the translation quality, such as
BLEU score (Papineni et al., 2002). For simultaneous translation we are also concerned with thefact that each action incurs a time delay D(A).
D(A) will mainly be influenced by delay caused
by READ, as this entails waiting for a humanspeaker to continue speaking (about 0.3s per word
for an average speaker), while WRITE consists ofgenerating a few words from a machine transla-

Figure 2: Illustration of the proposed framework:at each step, the NMT environment (left) computes a candidate translation. The recurrent agent(right) will the observation including the candidates and send back decisions–READ or WRITE.
tion system, which is possible on the order of milliseconds. Thus, our objective is finding an optimal policy that generates decision sequences witha good trade-off between higher quality Q(Y ) andlower delay D(A). We elaborate on exactly how
to define this trade-off in §4.2.
In the following sections, we first describe how
to connect the READ/WRITE actions with the NMT
system (§3), and how to optimize the system toimprove simultaneous MT results (§4).

3 Simultaneous Translation
with Neural Machine Translation
The proposed framework is shown in Fig. 2, andcan be naturally decomposed into two parts: environment (§3.1) and agent (§3.2).
3.1

Environment

Encoder: READ The first element of the NMT
system is the encoder, which converts input words
X = {x1 , ..., xTs } into context vectors H ={h1 , ..., hTs }. Standard NMT uses bi-directional
RNNs as encoders (Bahdanau et al., 2014), but thisis not suitable for simultaneous processing as using a reverse-order encoder requires knowing thefinal word of the sentence before beginning processing. Thus, we utilize a simple left-to-right unidirectional RNN as our encoder:hη = φUNI - ENC (hη−1 , xη )(1)

Decoder: WRITE Similar with standard MT, we
use an attention-based decoder. In contrast, weonly reference the words that have been read fromthe input when generating each target word:cητ = φATT (zτ −1 , yτ −1 , H η )zτη = φDEC (zτ −1 , yτ −1 , cητ )p (y|y<τ , H η ) ∝ exp [φOUT (zτη )] ,(2)

where for τ , zτ −1 and yτ −1 represent the previous
decoder state and output word, respectively. H η
is used to represent the incomplete input states,
where H η is a prefix of H. As the WRITE action
calculates the probability of the next word on thefly, we need greedy decoding for each step:yτη = arg maxy p (y|y<τ , H η )(3)

Note that yτη , zτη corresponds to H η and is the candidate for yτ , zτ . The agent described in the next
section decides whether to take this candidate or
wait for better predictions.
3.2

Agent

Observation As shown in Fig 2, we concatenate
the current context vector cητ , the current decoder
state zτη and the embedding vector of the candidate
word yτη as the continuous observation, oτ +η =[cητ ; zτη ; E(yτη )] to represent the current state.
Action Similarly to prior work (Grissom II et al.,2014), we define the following set of actions:• READ: the agent rejects the candidate and waits
to encode the next word from input buffer;• WRITE: the agent accepts the candidate andemits it as the prediction into output buffer;
Policy How the agent chooses the actions basedon the observation defines the policy. In our setting, we utilize a stochastic policy πθ parameterized by a recurrent neural network, that is:πθ (at |a<t , o≤t ) ∝ gθ (st ) ,Require: NMT system φ, policy πθ , τMAX , input
buffer X, output buffer Y , state buffer S.
1: Init x1 ⇐ X, h1 ← φENC (x1 ) , H 1 ← {h1 }2:z0 ← φINIT H 1 , y0 ← hsi3:
τ ← 0, η ← 1
4: while τ < τMAX do
5:t←τ +η6:yτη , zτη , ot ← φ (zτ −1 , yτ −1 , H η )7:at ∼ πθ (at ; a<t , o<t ) , S ⇐ (ot , at )8:if at = READ and xη 6= h/si then9:xη+1 ⇐ X, hη+1 ← φENC (hη , xη+1 )10:
H η+1 ← H η ∪ {hη+1 }, η ← η + 1
11:
if |Y | = 0 then z0 ← φINIT (H η )12:else if at = WRITE then13:zτ ← zτη , yτ ← yτη
14:
Y ⇐ yτ , τ ← τ + 1
15:
if yτ = h/si then break4

A trainable agent is designed to make decisions
A = {a1 , .., aT }, at ∈ A sequentially based on
observations O = {o1 , ..., oT }, ot ∈ O, and thencontrol the translation environment properly.

st = fθ (st−1 , ot )Algorithm 1 Simultaneous Greedy Decoding(4)

where st is the internal state of the agent, and is
updated recurrently yielding the distribution of theaction at . Based on the policy of our agent, theoverall algorithm of greedy decoding is shown in
Algorithm 1, The algorithm outputs the translation
result and a sequence of observation-action pairs.

Learning

The proposed framework can be trained using reinforcement learning. More precisely, we use policy gradient algorithm together with variance reduction and regularization techniques.
4.1

Pre-training

We need an NMT environment for the agent to explore and use to generate translations. Here, we
simply pre-train the NMT encoder-decoder on full
sentence pairs with maximum likelihood, and assume the pre-trained model is still able to generatereasonable translations even on incomplete source
sentences. Although this is likely sub-optimal, ourNMT environment based on uni-directional RNNs
can treat incomplete source sentences in a manner
similar to shorter source sentences and has the potential to translate them more-or-less correctly.
4.2

Reward Function

The policy is learned in order to increase a reward
for the translation. At each step the agent will receive a reward signal rt based on (ot , at ). To evaluate a good simultaneous machine translation, areward must consider both quality and delay.
Quality We evaluate the translation quality using metrics such as BLEU (Papineni et al., 2002).
The BLEU score is defined as the weighted geometric average of the modified n-gram precision
BLEU0 , multiplied by the brevity penalty BP topunish a short translation. In practice, the vanilla

BLEU score is not a good metric at sentence level
because being a geometric average, the score will
reduce to zero if one of the precisions is zero. To
avoid this, we used a smoothed version of BLEU
for our implementation (Lin and Och, 2004).
BLEU(Y, Y ∗ ) = BP · BLEU0 (Y, Y ∗ ),(5)

where Y ∗ is the reference and Y is the output. We
decompose BLEU and use the difference of partial BLEU scores as the reward, that is:∆BLEU0 (Y, Y ∗ , t)
t<T
rtQ =
(6)
BLEU(Y, Y ∗ )t=T
where Y t is the cumulative output at t (Y 0 = ∅),and ∆BLEU0 (Y, Y ∗ , t) = BLEU0 (Y t , Y ∗ ) −BLEU0 (Y t−1 , Y ∗ ). Obviously, if at = READ, no
new words are written into Y , yielding rtQ = 0.
Note that we do not multiply BP until the end of
the sentence, as it would heavily penalize partial
translation results.
Delay As another critical feature, delay judges
how much time is wasted waiting for the translation. Ideally we would directly measure the actual
time delay incurred by waiting for the next word.
For simplicity, however, we suppose it consumes
the same amount of time listening for one more
word. We define two measurements, global andlocal, respectively:• Average Proportion (AP): following the definition in (Cho and Esipova, 2016), X, Y are
the source and decoded sequences respectively,
and we use s(τ ) to denote the number of sourcewords been waited when decoding word yτ ,1 X
0 < d (X, Y ) =s(τ ) ≤ 1
|X||Y | τ
(7)

0
t<T
dt =
d(X, Y ) t = T
d is a global delay metric, which defines the average waiting proportion of the source sentence
when translating each word.
• Consecutive Wait length (CW): in speechtranslation, listeners are also concerned withlong silences during which no translation occurs. To capture this, we also consider on how
many words were waited for (READ) consecutively between translating two words. For eachaction, where we initially define c0 = 0,ct−1 + 1 at = READ
ct =
(8)
0
at = WRITE• Target Delay: We further define “target delay”
for both d and c as d∗ and c∗ , respectively, as
different simultaneous translation applications
may have different requirements on delay. In
our implementation, the reward function for delay is written as:
rtD = α·[sgn(ct − c∗ ) + 1]+β·bdt −d∗ c+ (9)where α ≤ 0, β ≤ 0.
Trade-off between quality and delay A good
simultaneous translation system requires balancing the trade-off of translation quality and timedelay. Obviously, achieving the best translation
quality and the shortest translation delays are ina sense contradictory. In this paper, the trade-off
is achieved by balancing the rewards rt = rtQ +rtD
provided to the system, that is, by adjusting the coefficients α, β and the target delay d∗ , c∗ in Eq. 9.
4.3

Reinforcement Learning

Policy Gradient We freeze the pre-trained parameters of an NMT model, and train the agent
using the policy gradient (Williams, 1992). The
policy gradient maximizes the following
hPexpectediT
cumulative future rewards, J = Eπθ
r
,
t
t=1
whose gradient is" T#X∇θ log πθ (at0 |·)Rt(10)∇θ J = Eπθ
t0 =1

h

i
PT
Q
D is the cumulative future
Rt =
r
+
r
k=t
k
k
rewards for current observation and action. In
practice, Eq. 10 is estimated by sampling multiple action trajectories from the current policy πθ ,collecting the corresponding rewards.
Variance Reduction
Directly using the policy
gradient suffers from high variance, which makes
learning unstable and inefficient. We thus employ the variance reduction techniques suggested
by Mnih and Gregor (2014). We subtract from
Rt the output of a baseline network bϕ to obtainR̂t = Rt − bϕ (ot ), and centered re-scale the rewith a running average bward as R̃t = √R̂σt2−b+and standard deviation σ. The baseline network is
trained to minimize the squared loss as follows:" T#
X
Lϕ = EπθkRt − bϕ (ot ) k2
(11)
t=1

We also regularize the negative entropy of thepolicy to facilitate exploration.

Algorithm 2 Learning with Policy Gradient
Require: NMT system φ, agent θ, baseline ϕ
1: Pretrain the NMT system φ using MLE;
2: Initialize the agent θ;
3: while stopping criterion fails do
4:
Obtain a translation pairs: {(X, Y ∗ )};5:
for (Y, S) ∼ Simultaneous Decoding do
6:for (ot , at ) in S do
7:
Compute the quality: rtQ ;8:Compute the delay: rtD ;9:
Compute the baseline: bϕ (ot );
10:
11:
12:
13:

Collect the future rewards: {Rt };Perform variance reduction: {R̃t };Update: θ ← θ + λ1 ∇θ [J − κH(πθ )]Update: ϕ ← ϕ − λ2 ∇ϕ L

The overall learning algorithm is summarizedin Algorithm 2. For efficiency, instead of updatingwith stochastic gradient descent (SGD) on a single
sentence, both the agent and the baseline are optimized using a minibatch of multiple sentences.

5

Simultaneous Beam Search

In previous sections we described a simultaneous
greedy decoding algorithm. In standard NMT it
has been shown that beam search, where the decoder keeps a beam of k translation trajectories,
greatly improves translation quality (Sutskever etal., 2014), as shown in Fig. 3 (A).
It is non-trivial to directly apply beam-search insimultaneous machine translation, as beam search
waits until the last word to write down translation.
Based on our assumption WRITE does not cost delay, we can perform a simultaneous beam-searchwhen the agent chooses to consecutively WRITE:
keep multiple beams of translation trajectories in
temporary buffer and output the best path when
the agent switches to READ. As shown in Fig. 3(B) & (C), it tries to search for a relatively better
path while keeping the delay unchanged.
Note that we do not re-train the agent for simultaneous beam-search. At each step we simply input the observation of the current best trajectory
into the agent for making next decision.

6

Experiments

6.1

Settings

DatasetTo extensively study the proposed simultaneous translation model, we train and evaluate it on two different language pairs: “English-

Figure 3: Illustrations of (A) beam-search, (B) simultaneous greedy decoding and (C) simultaneous
beam-search.
German (EN-DE)” and “English-Russian (ENRU)” in both directions per pair. We use the parallel corpora available from WMT’152 for bothpre-training the NMT environment and learning
the policy. We utilize newstest-2013 as the validation set to evaluate the proposed algorithm. Boththe training set and the validation set are tokenized
and segmented into sub-word units with byte-pair
encoding (BPE) (Sennrich et al., 2015). We only
use sentence pairs where both sides are less than
50 BPE subword symbols long for training.
Environment & Agent Settings We pre-trained
the NMT environments for both language pairs
and both directions following the same settingfrom (Cho and Esipova, 2016). We further built
our agents, using a recurrent policy with 512
GRUs and a softmax function to produce the action distribution. All our agents are trained using policy gradient using Adam (Kingma and Ba,2014) optimizer, with a mini-batch size of 10. For
each sentence pair in a batch, 5 trajectories are
sampled. For testing, instead of sampling we pick
the action with higher probability each step.
Baselines
We compare the proposed methods
against previously proposed baselines. For faircomparison, we use the same NMT environment:• Wait-Until-End (WUE): an agent that starts to
WRITE only when the last source word is seen.
In general, we expect this to achieve the best
quality of translation. We perform both greedy
decoding and beam-search with this method.
• Wait-One-Step (WOS): an agent that WRITEs
after each READs. Such a policy is problematicwhen the source and target language pairs havedifferent word orders or lengths (e.g. EN-DE).
2

http://www.statmt.org/wmt15/

16
14
12
10
8
6
4
2
00

AP=0.3AP=0.5AP=0.710

CW=2.0CW=5.0CW=8.0

Wait Until End.
Wait One Step

20
30
40
Mini-batches (x1000)

50

1.0
0.8
0.6
0.4
0.2
0.00

(a) BLEU (EN → RU)

AP=0.3AP=0.5AP=0.710

CW=2.0CW=5.0CW=8.0

Wait Until End
Wait One Step

20
30
40
Mini-batches (x1000)

50

25
20
15
10
5
00

(b) AP (EN → RU)

AP=0.3AP=0.5AP=0.710

CW=2.0CW=5.0CW=8.0

Wait Until End
Wait One Step

20
30
40
Mini-batches (x1000)50(c) CW (EN → RU)

12
0.5 0.6 0.7 0.8 0.9 1.0
Average Proportion

20
18
16
14
12
10
8
6
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Average Proportion(a) EN→RU(b) RU→EN

16

BLEU

BLEU

15
14
13

BLEU

Figure 4: Learning progress curves for variant delay targets on the validation dataset for EN → RU.
Every time we only keep one target for one delay measure. For instance when using target AP, thecoefficient of α in Eq. 9 will be set 0.

19
18
17
16
15
14
13
12 0.6

0.7 0.8 0.9
Average Proportion

1.0(c) EN→DE

24

BLEU

22
20
18
16
140.5 0.6 0.7 0.8 0.9 1.0
Average Proportion

Figure 5: Delay (AP) v.s. BLEU for both language pair–directions. The shown point-pairs are the results of simultaneous greedy decoding and beam-search (beam-size = 5)
respectively with models trained for various delay targets:(J /: CW=8, N4: CW=5, ♦: CW=2, I .: AP=0.3, HO:
AP=0.5, : AP=0.7). For each target, we select the modelthat maximizes the quality-to-delay ratio ( BLEU
AP ) on the validation set. The baselines are also plotted (F: WOS FI:
WUE, ×: WID, +: WIW).

(d) DE→EN• Wait-If-Worse/Wait-If-Diff (WIW/WID): as
proposed by Cho and Esipova (2016), the algorithm first pre-READs the next source word,
and accepts this READ when the probability ofthe most likely target word decreases (WIW), or
the most likely target word changes (WID).

ward functions, we vary the target delay d∗ ∈{0.3, 0.5, 0.7} and c∗ ∈ {2, 5, 8} for Eq. 9 separately, and trained agents with α and β adjusted tovalues that provided stable learning for each language pair according to the validation set.

• Segmentation-based (SEG) (Oda et al., 2014):
a state-of-the-art segmentation-based algorithm
based on optimizing segmentation to achieve
the highest quality score. In this paper, we
tried the simple greedy method (SEG1) and thegreedy method with POS Constraint (SEG2).

Learning Curves
As shown in Fig. 4, we plot
learning progress for EN-RU translation with different target settings. It clearly shows that ouralgorithm effectively increases translation quality
for all the models, while pushing the delay close,
if not all of the way, to the target value. It canalso be noted from Fig. 4 (a) and (b) that there exists strong correlation between the two delay measures, implying the agent can learn to decrease
both AP and CW simultaneously.

6.2

Quantitative Analysis

In order to evaluate the effectiveness of our reinforcement learning algorithms with different re-

BLEU

18
16
14
12
10
8
6
4
20

5

10

16.0
15.5
15.0
14.5
14.0
13.5
13.0
12.5
12.01.0 1.5 2.0 2.5 3.0 3.5

15

20

Consecutive Wait Length25

Figure 6: Delay (CW) v.s. BLEU score for EN→ RU, (J /: CW=8, N4: CW=5, ♦: CW=2, I.: AP=0.3, HO: AP=0.5, : AP=0.7), against
the baselines (F: WOS F: WUE, +: SEG1, ×:
SEG2).
Quality v.s. Delay
As shown in Fig. 5, it isclear that the trade-off between translation qualityand delay has very similar behaviors across bothlanguage pairs and directions. The smaller delay(AP or CW) the learning algorithm is targeting, thelower quality (BLEU score) the output translation.
It is also interesting to observe that, it is more difficult for “→EN” translation to achieve a lower APtarget while maintaining good quality, compared
to “EN→”. In addition, the models that are optimized on AP tend to perform better than those
optimized on CW, especially in “→EN” translation. German and Russian sentences tend to belonger than English, hence require more consecutive waits before being able to emit the next English symbol.
v.s. Baselines
In Fig. 5 and 6, the points closer
to the upper left corner achieve better trade-off
performance. Compared to WUE and WOS which
can ideally achieve the best quality (but the worst
delay) and the best delay (but poor quality) respectively, all of our proposed models find a goodbalance between quality and delay. Some of the
proposed models can achieve good BLEU scores
close to WUE, while have much smaller delay.
Compared to the method of Cho and Esipova(2016) based on two hand-crafted rules (WID,
WIW), in most cases our proposed models findbetter trade-off points, while there are a few exceptions. We also observe that the baseline models
have trouble controlling the delay in a reasonablearea. In contrast, by optimizing towards a given
target delay, our proposed model is stable while
maintaining good translation quality.
We also compared against Oda et al. (2014)’s

state-of-the-art segmentation algorithm (SEG). As
shown in Fig 6, it is clear that although SEG canwork with variant segmentation lengths (CW), theproposed model outputs high quality translations
at a much smaller CW. We conjecture that this isdue to the independence assumption in SEG, while
the RNNs and attention mechanism in our model
makes it possible to look at the whole history to
decide each translated word.
w/o Beam-Search We also plot the results of simultaneous beam-search instead of using greedy
decoding. It is clear from Fig. 5 and 6 that mostof the proposed models can achieve an visible increase in quality together with a slight increase indelay. This is because beam-search can help toavoid bad local minima. We also observe that thesimultaneous beam-search cannot bring as much
improvement as it did in the standard NMT setting. In most cases, the smaller delay the model
achieves, the less beam search can help as it requires longer consecutive WRITE segments for extensive search to be necessary. One possible solution is to consider the beam uncertainty in theagent’s READ / WRITE decisions. We leave this to
future work.
6.3

Qualitative Analysis

In this section, we perform a more in-depth analysis using examples from both EN-RU and EN-DE
pairs, in order to have a deeper understanding of
the proposed algorithm and its remaining limitations. We only perform greedy decoding to simplify visualization.
EN→RU
As shown in Fig 8, since both English and Russian are Subject-Verb-Object (SVO)
languages, the corresponding words may share thesame order in both languages, which makes simultaneous translation easier. It is clear that thelarger the target delay (AP or CW) is set, the more
words are read before translating the corresponding words, which in turn results in better translation quality. We also note that very early WRITE
commonly causes bad translation. For example,
for AP=0.3 & CW=2, both the models choose
to WRITE in the very beginning the word “The”,
which is unreasonable since Russian has no articles, and there is no word corresponding to it. One
good feature of using NMT is that the more words
the decoder READs, the longer history is saved,
rendering simultaneous translation easier.

gn

ai

e
st f
e mp
Th co
o
thca

ly
al
g
ic
in aid y
be
p
b

s
ibsa

l
my sa

y
ar

as

>
os
n- tor
a se
a
. <e

Die
Kostenfürdie
Kampagne
werdenim
Grundegenommendurch
mein
Geh-alt
als
Sen-atorge-deckt.
< eos >

READ
WRITE

e
st f
Th co
o

n

ig

pa

e m
th ca

y
ll
d
ca ere
v
co
by

si

is
ba

ry
la
my sa
as

n- tora seaDie
Kostenfürdie
Kampagne
werdenim
Grundegenommendurch
mein
Geh-alt
als
Sen-atorge-deckt.
< eos >

>
os

. <e

READ
WRITE

(a) Simultaneous Neural Machine Translation(b) Neural Machine Translation

Figure 7: Comparison of DE→EN examples using the proposed framework and usual NMT system
respectively. Both the heatmaps share the same setting with Fig. 1. The verb “gedeckt” is incorrectlytranslated in simultaneous translation.
SourceThepeople
,
as
I
heard
in
the
countryside,wantaGovernment
that
is
not
made
up
of
thi-eves.
<eos>

AP=0.3
The Thep-- i-- ent the p-- ol-- s,

Summaryas Iя слышал

всельской местности

AP=0.7

CW=2
The

CW=8

p-- riv-ersЛюди, как я слышал

,

Люди

как
я слышал

всельской
местности

в

,
как я
слышал

сельской
,местности
хочуправительство

всельской

,местности
хочу

,

,
хотят

правительство

которое не производится
во-- ров .

, чтобы правительство ,которое не в-- меши-вается в во-- ры .

, которое не является
состав-- ной частью во-ров .

BLEU=39/ AP=0.46

BLEU=64/AP=0.77

BLEU=54/CW=1.76,
хотят , чтобы
правительство , котороене в-- меши-- вается в во-ры .
BLEU=64/CW=2.55

Figure 8: Given the example input sentence (leftmost column), we show outputs by models trained forvarious delay targets. For these outputs, each row corresponds to one source word and represents theemitted words (maybe empty) after reading this word. The corresponding source and target words are in
the same color for all model outputs.
DE→EN As shown in Fig 1 and 7 (a), where
we visualize the attention weights as soft alignment between the progressive input and output
sentences, the highest weights are basically along
the diagonal line. This indicates that our simultaneous translator works by waiting for enoughsource words with high alignment weights andthen switching to write them.

DE-EN translation is likely more difficult as
German usually uses Subject-Object-Verb (SOV)
constructions a lot. As shown in Fig 1, when a sentence (or a clause) starts the agent has learned suchpolicy to READ multiple steps to approach the verb
(e.g. serviert and gestorben in Fig 1). Such a policy is still limited when the verb is very far fromthe subject. For instance in Fig. 7, the simultane-ous translator achieves almost the same translation
with standard NMT except for the verb “gedeckt”
which corresponds to “covered” in NMT output.
Since there are too many words between the verb
“gedeckt” and the subject “Kosten für die Kampagne werden”, the agent gives up reading (otherwise it will cause a large delay and a penalty)
and WRITEs “being paid” based on the decoder’shypothesis. This is one of the limitations of the
proposed framework, as the NMT environment istrained on complete source sentences and it may
be difficult to predict the verb that has not been
seen in the source sentence. One possible way is
to fine-tune the NMT model on incomplete sentences to boost its prediction ability. We will leavethis as future work.

7

Related Work

Researchers commonly consider the problem ofsimultaneous machine translation in the scenario
of real-time speech interpretation (Fügen et al.,2007; Bangalore et al., 2012; Fujita et al., 2013;
Rangarajan Sridhar et al., 2013; Yarmohammadi
et al., 2013). In this approach, the incoming
speech stream required to be translated are firstrecognized and segmented based on an automatic
speech recognition (ASR) system. The translation
model then works independently based on each ofthese segments, potentially limiting the quality oftranslation. To avoid using a fixed segmentation
algorithm, Oda et al. (2014) introduced a trainable
segmentation component into their system, so thatthe segmentation leads to better translation quality.
Grissom II et al. (2014) proposed a similar framework, however, based on reinforcement learning.
All these methods still rely on translating each segment independently without previous context.
Recently, two research groups have tried to apply the NMT framework to the simultaneous translation task. Cho and Esipova (2016) proposeda similar waiting process. However, their waiting criterion is manually defined without learning.
Satija and Pineau (2016) proposed a method similar to ours in overall concept, but it significantly
differs from our proposed method in many details.
The biggest difference is that they proposed to use
an agent that passively reads a new word at eachstep. Because of this, it cannot consecutively decode multiple steps, rendering beam search difficult. In addition, they lack the comparison to any
existing approaches. On the other hand, we per-form an extensive experimental evaluation against
state-of-the-art baselines, demonstrating the relative utility both quantitatively and qualitatively.
The proposed framework is also related to somerecent efforts about online sequence-to-sequence(SEQ 2 SEQ) learning. Jaitly et al. (2015) proposed a SEQ 2 SEQ ASR model that takes fixedsized segments of the input sequence and outputs
tokens based on each segment in real-time. It istrained with alignment information using supervised learning. A similar idea for online ASR is
proposed by Luo et al. (2016). Similar to Satija
and Pineau (2016), they also used reinforcement
learning to decide whether to emit a token while
reading a new input at each step. Although sharing some similarities, ASR is very different fromsimultaneous MT with a more intuitive definition
for segmentation. In addition, Yu et al. (2016)
recently proposed an online alignment model tohelp sentence compression and morphological inflection. They regarded the alignment between the
input and output sequences as a hidden variable,and performed transitions over the input and output sequence. By contrast, the proposed READ and
WRITE actions do not necessarily to be performedon aligned words (e.g. in Fig. 1), and are learned
to balance the trade-off of quality and delay.

8

Conclusion

We propose a unified framework to do neural simultaneous machine translation. To trade off quality and delay, we extensively explore various targets for delay and design a method for beamsearch applicable in the simultaneous MT setting.
Experiments against state-of-the-art baselines on
two language pairs demonstrate the efficacy bothquantitatively and qualitatively.

Acknowledgments
KC acknowledges the support by Facebook,
Google (Google Faculty Award 2016) and NVidia(GPU Center of Excellence 2015-2016). GN acknowledges the support of the Microsoft CORE
program. This work was also partly supported
by Samsung Electronics (Project: ”Development
and Application of Larger-Context Neural Machine Translation”).

References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint
arXiv:1409.0473.
Srinivas Bangalore, Vivek Kumar Rangarajan Sridhar, Prakash Kolan, Ladan Golipour, and Aura
Jimenez. 2012. Real-time incremental speech-tospeech translation of dialogs. In Proceedings of the2012 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technologies, pages 437–445. Association for Computational Linguistics.
Kyunghyun Cho and Masha Esipova. 2016. Can neural machine translation do simultaneous translation?
arXiv preprint arXiv:1606.02012.
Christian Fügen, Alex Waibel, and Muntsin Kolss.
2007. Simultaneous translation of lectures andspeeches. Machine Translation, 21(4):209–252.
Tomoki Fujita, Graham Neubig, Sakriani Sakti,
Tomoki Toda, and Satoshi Nakamura. 2013. Simple, lexicalized choice of translation timing for simultaneous speech translation. In INTERSPEECH.
Alvin Grissom II, He He, Jordan Boyd-Graber, John
Morgan, and Hal Daumé III. 2014. Dont until
the final verb wait: Reinforcement learning for simultaneous machine translation. In Proceedings ofthe 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1342–
1352, Doha, Qatar, October. Association for Computational Linguistics.

Andriy Mnih and Karol Gregor. 2014. Neural variational inference and learning in belief networks.
arXiv preprint arXiv:1402.0030.
Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki
Toda, and Satoshi Nakamura. 2014. Optimizing segmentation strategies for simultaneous speechtranslation. In Proceedings of the 52nd Annual
Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 551–556,
Baltimore, Maryland, June. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic
evaluation of machine translation. In Proceedings ofthe 40th annual meeting on association for computational linguistics, pages 311–318. Association for
Computational Linguistics.
Vivek Kumar Rangarajan Sridhar, John Chen, Srinivas
Bangalore, Andrej Ljolje, and Rathinavelu Chengalvarayan. 2013. Segmentation strategies for streaming speech translation. In Proceedings of the 2013
Conference of the North American Chapter of the
Association for Computational Linguistics: Human
Language Technologies, pages 230–238, Atlanta,
Georgia, June. Association for Computational Linguistics.
Harsh Satija and Joelle Pineau. 2016. Simultaneousmachine translation using deep reinforcement learning. Abstraction in Reinforcement Learning Workshop, ICML2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2015. Neural machine translation of rare words withsubword units. arXiv preprint arXiv:1508.07909.

Navdeep Jaitly, Quoc V Le, Oriol Vinyals, Ilya
Sutskeyver, and Samy Bengio. 2015. An online sequence-to-sequence model using partial conditioning. arXiv preprint arXiv:1511.04868.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112.

Diederik Kingma and Jimmy Ba. 2014. Adam: A
method for stochastic optimization. arXiv preprint
arXiv:1412.6980.

Ronald J Williams. 1992. Simple statistical gradientfollowing algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256.

Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram
statistics. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics,
page 605. Association for Computational Linguistics.

Mahsa Yarmohammadi, Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Baskaran
Sankaran. 2013. Incremental segmentation anddecoding strategies for simultaneous translation. In
IJCNLP, pages 1032–1036.

Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, andIlya Sutskever. 2016. Learning online alignmentswith continuous rewards policy gradient. arXiv
preprint arXiv:1608.01281.
Takashi Mieno, Graham Neubig, Sakriani Sakti,
Tomoki Toda, and Satoshi Nakamura. 2015. Speedor accuracy? a study in evaluation of simultaneous
speech translation. In INTERSPEECH.

Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online segment to segment neural transduction. arXiv
preprint arXiv:1609.08194.

A

Joint learning

Following the notation, let the input pipe as X ={x1 , x2 , ..., xTs }, and the output pipe as Y ={y1 , y2 , ..., yTt }. At every time step t = τ + η, we
first pre-compute the context vector cητ , the current hidden state zτη , and output the pre-decoded
word yτη as the input ot of the agent. In our paper,
we used a greedy decoder to get the pre-decodedword:yτη = arg max pφ (y|y<τ , x≤η )yHere, we can also use sampling asyτη ∼ pφ (y|y<τ , x≤η )The agent θ outputs READ/WRITE actions as A ={a1 , a2 , ..., aT }, where T = Ts + Tt . When theagent chooses to WRITE, the pre-decoded wordyτη is picked as yτ . The agent receives rewards
R = {r1 , r2 , ..., rT }. Thus, we can learn theagent’s policy πθ using REINFORCE, by assuming the NMT model is fixed:" T#
X∇θ J = Eπθ∇θ log πθ (at |ot ) · Rt
t=1

P
where Rt = Tk=t rk . The idea of joint learningis that we can also optimize the underlying NMT
model with REINFORCE, by assuming the policyof the agent is fixed:"T#tX∇φ log pφ (yτ |y<τ , x≤η ) · Rτ +η
∇φ J = Epφτ =1where we use η to show the number of the corresponded source words at each decoded word yτ .
The logic is a little similar with actor-critic. The
agent tries to find the best policy πθ given the current model, and the model φ tries to output the besttranslation under the current policy. One questionis that I am not sure if we need to compute the likelihood for the pre-decoded words that are dropped.