Improved Training Techniques for Online Neural Machine Translation
Authors: Maha Elbayad
Laurent Besacier
Jakob Verbeek
Venue:  (2020)
Keywords: Deep learning
natural language processing
Machine translation

Abstract: Neural sequence-to-sequence models are at the basis of state-of-the-art solutions for sequential prediction problems such as machine translation and speech recognition. The models typically assume that the entire input is available when starting target generation. In some applications, however, it is desirable to start the decoding process before the entire input is available, e.g. to reduce the latency in automatic speech recognition. We consider state-of-the-art wait-k decoders, that first read k tokens from the source and then alternate between reading tokens from the input and writing to the output. We investigate the sensitivity of such models to the value of k that is used during training and when deploying the model, and the effect of updating the hidden states in transformer models as new source tokens are read. We experiment with German-English translation on the IWSLT14 dataset and the larger WMT15 dataset. Our results significantly improve over earlier state-of-the-art results for  German-English translation on the WMT15 dataset across different latency levels.

Under review as a conference paper at ICLR 2020
IMPROVED TRAINING TECHNIQUES FOR ONLINE NEURAL MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural sequence-to-sequene models are at the basis of state-of-the-ard solutions for sequential prediction problems such as machine translation and speech recog- nition. The models typically assume that the entire input available when starting target generation. In some applications, however, it is desirable to start the decod- ing process before the entire input available, e.g. to reduce the latency in au- tomatic speech recognition. We consider state-of-thh-ar "wait-k" decoders, that first read & tokens from the source and then alternate between reading tokens from the input and writing to the output. We investigate the sensitivity of such models that is used during training and when deploying the model, and
to the value the effect of updating the hidden states in transformer models as new source to- kens are read. We experiment with German- EngllshEglhhhhh translation on the IWSLT14 dataset and the larger WMT15 dataset. Our results significantly improve ear- lier state-of-the-ar results for German-English translation on the WMT15 dataset across different latency levels.
INTRODUCTION
Seuueecc-to-Sqqueee (S2S) models are sfate-of-the-ann for tasks where source and target sequences have different lengths, including automatic speech recognition, machine translation, speech transla- tion, text-to-speech synthesis, etc. The most common models are composed of an encoder that reads the entire input sequence, while a decoder (often equipped with attention mechanism) iteratively produces the next output token given the input and the partial output decoded so far. While these models perform very well in the typical offline decoding models are affected by low-latency constraints, and which architectures and strategies are the most efficient. Low-latency decoding is desirable for applications such as online speech recognition, and you-oue machine translation. In such scenarios, the decoding process starts before the entire input sequence available, and the output sequence produced in an on-the-fly manner. However, if we consider for instance machine translation, online prediction generally comes at the cost of re- duced translation quality and more research is needed to reach the grail of natural and high-quality online specch-to-speede interpretation.
case, few studies consider how S2S
In this paper we consider deterministic "wait-A" decoders that are state of the art for low-latency decoding (Ma et al. 2019 Zheng et al., (0119)). These decoders first read A tokens from the source, after which they proceed to alternatingly produce a target symbol and read another source sym- bol. We compare two architectures to implement such models: one based on a 2D-convolutional sequence-to-sequere
model (Elbayad et al. 2018), and one based on the attention-based trans- former architecture (Vaswani et al. 2017). For these models, we investigate the impact of the choice of & when training the models, and when using them to generate translations. For the transformer model, also consider the effect of updating the hidden states of previous target symbols based on the full source context that is available at any moment. These updates are inspired from the 2D convolutional model, where such "updates" are inherent consequence of the architecture.
In summary, our contributions are the following:
1. We compare transformer and 2D convolutional architectures for online machine translation. 2. We propose improved training techniques for wait-k by first using uni-directional encoders and training across multiple values of l:

---

Under review as a conference paper at ICLR 2020
3. We boost the translation quality in test-time by updating the prefix hidden-states in trans- former decoders.
4. We set a new state of the art for online translation on WMT15 CCormnn-Engline improving over the recent state-of-the-art results by Ma et al. (0199) and /Zheng et al. (2019b) across full range of latency levels.
The rest of this paper is organized as follows: Section presents related work on low-latency ma- chine translation; Section 3 details our low-latency models, which we evaluate in Section for German-English machine translation on the IWSLT14 dataset as well as the larger 5MT15 dataset. Section 5 concludes the paper.
2 RELATED WORK
In order study differences between translated and simultaneously interpreted text, et al. (2016a) produced a parallel corpus between both and showed that human interpreters regularly apply several tacties to reduce translation latency, including sentence segmentation and passivization. In ma- chine translation, after pioneering work from Fügen et al. 20077) Yarmohammadi et al. (2013) and He al. (2015) proposed methods increase the alignment monotomicity for statistical machine translation. To allow for more flexible segmentation, Grissom et al. (2014) introduced trainable segmentation module that decides whether to write or read tokens based on the prediction of the final verb and the next token in the source sequence. Similarly, Oda et al. (2015) use output of syntax-based -bata statistical translation system to find the optimal segmentation strategy that maximizes the quality of the translations via greedy search and dynamic programming.
One of the first works on online translation to use attention-based sequence-to-sequence models is that of Cho & Esipoval (2016), which uses manually designed non-trainable waiting criteria that dictate whether the model should commit to a read/write operation. The neural transducer of Jaitly et al. (2016) reads equally-sized chunks of the source sequence and generates output sub-sequences of variable lengths, each ending with a special token marking the end of the writing. For training, single segmentation is chosen optimize the likelihood of the full output sequence. |Raffel et al. (2177) propose an alternative to attention using monotonic alignments. Whereas attention requires access the full source sequence to compute the weights, monotonic alignments enable linear time computation of the weights and online decoding.
sequence.
Another line of research treats general alignments source and target sequence as a latent variable. Inspired from the HMM word alignment model used in statistical machine translation (Vogel et al.) 1996), the segment-to- -segmmentt neural transducer model Yu et (2016) integrates latent align- ment variable into an --asedd seueenceto-sequenee model where the transition probabilities are conditioned on the encoder-decode hidden states. During training the alignment is marginal- ized out with forward-backwart algorithm (Rabiner 1989). However, their approach is used as an alternative to attention-based models, not for low-latency translation. [uo et al. (2017) introduce recurrent sequence-to-seeee to-suuuencee model with binary stochastic decision variables either emit the next output token or advance in encoding the source sequence with a unidirectional LSTM that jointly encodes the partially generated output sequence. The stochastic decisions are optimized ing a standard policy gradient reinforcement learning approach. Guu et al. (2017) propose a trainable agent emitting read/write decisions modeled recurrent neural network fed with the encoder and decoder current hidden states. In their framework, a left-to-right recurrent sequence-to-sequee model first pre-trained on the full bi-texts and then fine-tuned with policy gradient optimize reward balancing the quality and the latency of the translations.
Dalvi al. (2018) used a static decoding algorithm that starts with read operations then alternates between blocks of / write operations and read operations. Albeit simple. this approach outperforms the information based criteria of [ho & Esipova (2016) and allows for complete control of the translation delay. Their attempt integrate incrementality in the training by pre-aligning the source- target sequences and then attending over a constrained span source positions failed to improve the translation quality. The work of Press & Smith (2018), although not tackling the simultaneous translation task, allows for emitting target tokens before reading the full input sequence. Similar to the incremental training Dalvi et al. (2018), it requires pre-aligning the bitexts. More recently, Ma et al. (209) trained a eqquence-to-seqeeme model based on the transformer architecture (Vaswani al. 2017) with an integrated "wait-k"" agent that first reads h source tokens then alternate single
2

---

Under review as a conference paper at ICLR 2020
10
1=1
K
k=
Figure I: Low-latency decoding as a sequence of reads (green) and writes (red) over grid spanned by the source (horizontal) and target sequence (vertical). Different panels show decoding paths defined by the of initial reads k, of the wait-k: decoder. For training we optimize the emissions probabilities for writes on single or multiple paths, or for shaded positions above the path.
wradewrites, similar to Dalvi et al. (2018) but using - Wait-k: approaches, were found most effective by Zheng al. (201 9b) when trained for the specific & that is used to generate translations. This, however, requires training separate models for each potential value of k used for translation. As an alternative, [Zheng et al. (2019a b) both learn adaptive policy to produce read/write decisions in pre-trained offline translation model. Zheng et al. (2019a) train the model for decoding along two wait-k paths and Zheng et al. (20T9b) supervised training based sequence derived from the pre-trained offline translation model. Different latency levels are achieved by thresholding the policy's 's confidence, however, on most regimes their model performs worse than using transformer models directly trained for wait-k online translation. In our work, we also use wait-k decoders, but unlike Ma et al. (2019); Zheng al. (2019a[b) we opted for uni-directional encoders and show that they are efficient to train in an online setup and can achieve better results if we update the prefix decoder states during inference.
oracle read/write
3 LOW-LATENCY SOUUENCE-TO-SQUEEREE MODELING
Let (x,z) be pair of souree-target sequences of respective lengths |z] and ly/- Low-latency de- coding consists of executing a sequence of interleaved reading and writing operations, consuming tokens from the source æ and producing tokens of the target y. Low-latency decoding paths can be represented on a ly] |æ|| grid, where we advance from top left the bottom right in total of [x] read steps and ly) write steps. See Figure for an illustration.
We consider two recent sequence-to-sequuenc models for low-latency decoding: the transformer architecture of Vaswani et al. (2017) and the 2D convolutional architecture of Elbaaad et al. (2018). We describe Section 3.1 how each of these architectures is adapted for the task of simultancous translation. In Section 3.2 we describe how these architectures can be trained for online decoding, based on one or more decoding paths.
3.1 LOW-LATENCY DECODING ARCHITECTURES
To formalize the low-latency decoding process we model as sequence of lyy write steps. At step t (I,..., |y]] decode the t-th target token 1/+, conditioning on the target prefix y and the source sequence read up that point ø The value of is dictated by the decoding path.
The decoder computes the distribution over the target token given the source and target contexts,
Po yt y T Tg Zct),
where 0 denotes the parameters of the writer. It depends on the architecture of the decoder whether, given 2t, the prediction y/ further depends on the full decoding z path that led up this point.
3.1.1
CONVOLUTION-BASER LOW-LATENCY DECODER
In order for the decoder to be trained efficiently for different source/target context sizes, we build upon the "pervasive attention" architecture of Elbayad et al. (018)) Their machine translation model

---

Under review as a conference paper at ICLR 2020
uses masked 2D convolutions across source-targe target grid, as in Figure The masked convolutions ensure that only information from past target tokens is used to predict the next one. Here we adapt the masking pattern as to construct the field-of-view in way that it only extends over previous positions in both source and target dimensions. More concretely, we use a 11 x1x11 11 filter where only the top-left 6x 6 weights are non-zero. In this manner features computed by a sequence of convolutional layers at a position (t,j) in the grid can be used to define pe(Vilye Tzy).
As input, at each site (t,j),, the network takes the concatenation of the (sub-)word embedding of the (-th token in target and the j-th token in the source sequence. The input is then progressively transformed along a number of convolutional network layers. Let H = (hez) for < < ly and 1 $ & [x] be the final features of the masked convolutional neural network. Elbayad et al. (2018) apply max- max-pooling across the source dimension to map H to a fixed-sized vector for each target position. To define (u (y et: Rzj) in our low-latency model, we use
fis- = Pool(HaI,s.)
(2)
where Pool(-) pools Hst,sj in the source and/or target direction, neither in which case fej huj- Preliminary experiments with the pervasive attention architecture showed that max-pooling across the source dimension only is the best option. For the remainder of this paper we have:
Jej Max-pool(HH,sse (H),,)))
From there we generate the emission probabilities by linearly projecting the feature fúy to the di- mension of the output vocabulary with the matrix W, followed by a soft-max normalization:
po(uely (> < .t,)) = softmax W ftz).
(4)
Due to the 2D convolutional structure of the network, the path leading up to position (t.j) in the network does not impact the prediction of V/- Therefore, we drop the dependence on zu; in Eq. (4)
3.1.2 ATTENTION- BAASED LOW-LATENCY DECODER
The second online decoder we consider is based on the "transformer" model of Vaswani et al. (2017). Both the encoder and decoder consist of stack of blocks. Each block consists of multi-heat self- attention followed by a position-wise fully connected feed-forward network, or equivalently a 1 x convolution. The decoder uses additional multi-head attention block that ranges the source token encodings.
Given a query vector Qt, an attention component aggregates set of value vectors ii in a weighted sum, based on scoring a corresponding set of key vectors k; against the query. Using the dot-product as the score, we obtain the attention aggregate a, as
Gt "n
Orj
Σ, etj
expeyj
Caj
4 kg
(5)
The self-attention block is designed such that only previous output tokens can be attended over, since future tokens are not available when generating targets, ensuring the decoder is autoregressive (AR).
In our work, the transformer model for online translation has the following properties:
1. A uni-directional encoder: we make the self-attention in the encoder autoregressive, so that it can encode all source prefixes in parallel during training, and encode them progressively during generation. This is achieved by substituting the mask used in Ma et al. (2019)
(referred to as STACL in Eq. (6) with the AR mask of Eq. (7).
2. A constrained encoder-decoder interaction: that when producing the t-th output token attention limited to the 2, source tokens read far. This is guaranteed with the enc-dec mask of Eq.
mask the source attention in the decoder, so
q.. ky, if 21 -00, otherwise
1t, ejj
STACL
(6)
AR q. kj, lfj 00, otherwise
q. ky. If otherwise
ene des
(7)
tj
1o,
4

---

Under review as a conference paper at ICLR 2020
Cached prefix
-
Updated prefix
Vt-1
f--2
Self-attention
f
2
P--1
Self-attention
(2
21-13
(a)
(b)
Figure 2: Caching (a) and updating (b) hidden states of the decoder for self-attention.
In this transformer- based decoder, the prediction of y/s depends on the full decoding path Eet. This is because the decoder self-attention ranges over decoder hidden states of previous time steps t' < t, which themselves used žy to attend over the source encoding.
In addition to the cunsformer-decodee described above, consider second transformer- based variant which removes the dependency of Yt on &<t- To achieve this, each time a source token is read, update all hidden states in the decoder based the source context æ2. available that point. Essentially, we will re-run the decoder across the entire output sequence, using the new source context. This makes the decoder self-attentio insensitive to 2<t given 2,, increasing the
decoding cost from O x| x [y]) to O(
(y)). Note that the encoder still needs be run only once, since its self-attention has been constrained to be autoregressive. We refer to this decoder as "transformer-upe update", see Figure for an illustration.
3.2 TRAINING LOW-LATENCY DECODERS
Ma et al. 2019). First source tokens are read, before alternating reading and writing a single token at time, until either the full source has been read, or the target generation has been terminated.
For our decoders we use pre-defined "wait-k" schedules (Dalvi al. 2018
To train our models, we consider several strategies. The first, similar to Ma et al. (219)) is to train the model using the same wait-k decoding path that will be used for generation. That is, to train the model, we optimize a sum of loss terms, each which corresponds to the negative log- ligelikeddoodd of predicting target token:
v In po pol: W 0 Rc2)-
L(O, v. =
(8)
Training with k=00 corresponds to an offline (wait-until-the-end) decoder, with the exception that here we still use autoregressive dependencies in the encoder along the source dimension in the 2D convolutional network.
In addition to training according to single wait-k decoding path, with multiple paths. 'he additional loss terms may provide a richer training signal, and potentially yield models that could perform well across range of values for i during decoding. Due to the dependence of yr on the full decoding path zet in the transformer-based model, it is not possible to improve over simply training in parallel across the different values of k. When training for multiple values of k, we encode the source sentence once, and then forward it multiple times in the decoder, once for each value of k.
use losses associated
The 2D CNN-based architecture, however, does not include this full dependency, and allows par- allel computation of all po(y y ce Fzz) in single evaluation of the CNN. For this architecture therefore use loss of the form
vl
L 2, 2. = Σ
4t In pe (y Be &zy),
where 4t weight the terms corresponding different positions in the decoding grid of Figure The weights 4t can be set to zero/one to reflect all positions corresponding to wait-k decoding paths for also consider the option of activating all loss terms
more values of k. In our experiments above a certain wait-k decoding path. The latter corresponds to the sum of all loss terms on all wait-k' decoding paths with k & k' 2c
Summmnng the losses for all paths, though, would weight shared loss importantly.
5

---

Under review as a conference paper at ICLR 2020
The transformer-updett update architecture also avoids the full dependency <ct, but in order to train from all loss terms above a certain decoding path, it requires a separate decoder run for each source context size, i.e. for each column of the decoding grid in Figure
4
EXPERIMENTAL EVALUATION
In this section we present our experimental setup, followed by quantitative results.
4.1 DATASETS AND EXPERIMENTAL SETUP
Datasets. We evaluate our approach on IWSLTI4 En-De (Cettolo et al. 2014) and on the WMT15 De-En datasets Replicating the setup of Elbayad et al. (0188) for IWSLT14 De-En, we train on 160K sentence pairs, develop on 7K held-out pairs and test on 6.7K pairs. The vocabulary consists of 8.8K types the source side and 6.6K types on the target side that were obtained from joint source and target byte pair encoding (BPE; Sennrich et al. 2016). For WMT15 De-En we use a joint vocabulary of 32k BPE types. We train on 4.5M pairs, develop on newstest2013 (3K pairs) and test on 1 5 (2.2K pairs).
Evaluation metries. We use beam-search decoding, with beam of size 5, for offline models and only greedy decoding online. We evaluate the translation quality by measuring case-sensitive tok- enized BLEU (Papineni et al. 2002) with mu t -bllleeupp decoding latency using the proportion of the source sentence which has been read when producing target tokens, and average this proportion across the tokens in the generated sentence:
Cho & Esipova (2016) measure the
y
1
AP
2/ acl
(10)
y
More recently, average lagging (AL) and differentiable average lagging (DAL), were proposed to measure translation latency (Ma et al. 2019: Cherry & Foster 2019 Arivazhagan al. 2019):
AT 1
t-1
1 DAT.
t t-1 1 2/{
if max(z) 21, 2/-1 + 2
y
7
where 7 = /yl/|x and r = min g(() = [ These metrics handle differing source and target lengths more properly, and have intuitive interpretation as the average by which the system lags behind an ideal instantaneous translator. We refer the reader to (Cherry & Foster 2019) for details.
We report the mean of these metric across translations. In this section report the average lagging (AL) metric, while the other metrics (AP, DAL, leading to a similar trend) are given in Appendix [ AL is chosen make our results comparable with those of Ma at. (2019) and Zheng al. (209h)) for German-FEnllish WMT15 dataset.
Architectures. For the pervasive attention model, to reduce the memory footprint, skip connections (He et al. 2016b) rather than the dense connections (Huang et al.] 2017) used by |Elbayad et al. (2018). See Appendix A for more details about other minor changes made to the original architecture. We consider an offline pervasive attention baseline (PA) that reads the source sequence bi-directionally via asymmetric convolution filters that are only masking the future targets. Note that this baseline is not directly applicable for low-latency decoding, due to the bi- directional source encoding. We use second baseline where convolutions are masked along the source dimension, which we refer to as masked-PA PA baseline (MPA).
residual
For the Transformer model, we use small architecture on IWSLT with an embedding dimension denc 512 for the encoder, date 256 for the decoder, and N = 6, dæ 1024. h 4, Panop 0.3. On WMT, we use Transformer hase (Vaswani et al., (017)) with tied embeddings. Similar to the pervasive attention baseline, we consider the offline baseline (T) with bidirectional self-attention in the source side and a masked baseline (MT) with left-to-right self-attention in the encoder.
http: /www wtw.::ssttttttrrroooo 15/
https: https://github.. wwm. http://ooosssssttmmmeeeeee http:///wo.ooooooo/oo/oo///////a
multi-bles multi-bleu.per

---

Under review as a conference paper at ICLR 2020
PA*
PA
MPA
MT
Dutatset
task
BS
G
BS
G
BS
BS
G
BS
IWSLT14
En-De
27.21
27.23
27.96
26.66
27.52
27.46
28.27
26.58
27.70
IWSLT14
De-En
33.86
33.06
34.02
32.21
33.39
33.84
34.94
32.81
33.79
WMT15 De-En
28.08
29.28
31.96
33.00
31.14
32.27
Table 1: Offline evaluation of the Pervasive Attention (PA), Masked Pervasive Attention (MPA),
Transformer (T) and masked Transformer (MT) baselines. PA* corresponds to results reported by
Elbayad et al. (2018). We evaluate using greedy decoding (G) and beam-search (BS).
32
32
30
30
30
30
28
28
28
26
26
26
24
24
24
24
22
22
22
20
20
20
k
18
18
18
18
4
23
23
Average logging
Avarage lapping
MPA wait-k
(b) MT wait-k
Figure 3: IWSLT De-En: wait-k online decoding with MPA (a) and MT (b). Each curve obtained with model trained single decoding path. Each model is evaluated with Keval (1, 5, 7, 7.9).
4.2 OFFLINE DECODING BASELINE RESULTS
In Table we report the offline performance of the Pervasive Attention (PA), Masked Pervasive Attention (MPA), Transformer (T) and Masked Transformer (MT) baselines. Each with greedy decoding (G) and with beam search (BS).
Interestingly, the masked and non-masked versions perform quite similar, across both datasets and architectures. This validates the use of autoregressive source encoders in the low-latency models we consider below. The transformer and convolutional architectures perform similarly on IWSLT14, while the former performs significantly better on WMT15.
4.3 ONLINE DECODING RESULTT
In our first experiment we evaluate the wait-k online decoding for different MT and MPA models, trained for different wait-k decoding paths. We denote with == the wait-until-end training where the full source is read before decoding, In each figure the offline results are added for reference, the offline model has a latency of AL [x). Here we show results for IWSLT De-En, similar trends observed for IWSLT En-De (see Appendix B.3).
Impact the architecture. Figure 3 presents the performance models trained for wait-k decod- ing across range of latencies kieval E 11,3, 5, 7,9). Each trained model is represented by curve, by evaluating across different wait-k decoding paths.
For the convolutional architecture observe that single model trained for a relatively large value of k, e.g. 7 or provides good performance when using it to generate using different & values for generation. The performance of the transformer-basee model is for training. This difference in sensitivity could be due to the use of input-independent convolution filters self-attention which relies scoring context to weight its features.
sensitive the value of k used
Given an appropriate value for during training, the transformer based models yield better transla- tions for given latency level.

---

Under review as a conference paper at ICLR 2020
32
32
30
30
30
30
30
.
28
28
28
28
28
28
26
26
26
26
26
26
24
24
24
24
24
+
22
22
22
22
13
Rosi
20
13
KE [1.55
0 KE 1.99
18
18 0
18 0
18
0
6
25
2
6
23
2
4
8
25
Average lagging
Averago wazing
Average waging
(a) MPA above path
(b) MT &s [1..5]
(c) MT E [1..9]
Figure 4: IWSLT De-En: wait-k online decoding with MT trained on a single path or multiple paths. Each model evaluated with keval {1,3: 5, 7,9). The keval curve combines different models, evaluating for each value of & the model trained for that value.
Training
Update
Keval
Keval 3
Keval
5
keval 7
keval
k 00
w/o with
22.8
26.3
29.0
30.6
31.6
k 00
24.7
27.8
29.9
31.4
32.1
k 7
w/o
22.9
27.1
29.6
30.6
31.0
k = 7
with
24.7
27.9
30.0
31.1
31.0
E [1..5]
w/o
22.6
25.7
27.9
28.9
28.7
E [..5]
with
25.0
28.0
29.1
29.1
28.6
E [1.9]
w/o
23.1
26.8
29.6
31.3
31.7
A [1.9]
with 24.5
28.0
30.1
31.4
31.7
Table 2: IWSLT De-En: Impact of hidden state updates in transformer-be low-latency decoders.
Impact of multiple path training. For the convolutional model (MPA), consider activating loss terms the area above the wait-k path. Training on this area covers a combinatorially large number of paths with latency higher than k. The results in Figure 4a show that for the MPA architecture, multiple path training can be beneficial compared to single path (k= vs. k B keval). More importantly, the model trained with all loss terms above the diagonal of the decoding grid (k: > 1), performs equal or better than models trained with only subset of the terms, across all tested latency levels.
For transformer model (MT), training for all decoding paths above certain wait-k path is pro- hibitively costly (since the cost linear the number of training paths). Therefore, we opted instead for joint training for a selected set of wait-k paths. Results in Figure 4b show that jointly training on the w paths from k= to k=5 improves over the training on individual paths low- latency regimes (Keval & 3). For higher latency relatively small drop in performance is observed compared to training with 5.
Where the convolutional model could be trained near optimal by training across all values of k, in Figure 4c we see that training the paths from k=1 to k=9 performs similarly to just training with k=9. The latter model is (near) optimal for AL 3, but for very low-latency the model trained on ee [1.5]] ] is better.
Impact of hidden state updates. The standard decoding approach with Transformer is to use previously evaluated hidden states to encode the target prefix written so far. This not the case for MPA where source context is integrated in the convolution for all previous time-steps. To bridge the gap between the two decoding paradigms, we update previous hidden states to account for new source contexts.
In Table 2 we compare transformer models with and without hidden state updates, trained along one multiple decoding paths. Among the 20 comparisons (4 models, and 5 wait-k decoders)
8

---

Under review as a conference paper at ICLR 2020
Ofline
Oftine
30
30
30
30
28
28
28
28
26
26
BLEU
24
24
24
24
22
MT ke [1..9]
22
22
22
MT c [1.55]
20
Mt at 2019 k Koos
20
MT A--z)
Zheng al 2019b [SL]
18
MT k-aa Update
18
18
Ours MT k- update
18
0
2
6
8
10
27
0
2
8
10
27
Average lagging
Average lagging
(a) Effect hidden state updates
(b) Comparison to state the art
Figure 5: Experimental results on WMT15 De-En.
observe only single case where state updates drop the BLEU score by O.I. In all other cases, the state updates lead to improvements, by up to 2.4 BLEU points in very low latency settings
Experiments on the WMT15 De-En corpus. In Figure 5a we present our results on the WMT15 De-En task using transformer model, which gave the best results the IWSLT experiments. The results confirm observations made IWSLT: (i) Hidden state updates consistently improve the performance of the tested models. (ii) When using hidden state updates, single low-latency model (Is = 7) performs better comparable to using separate models trained specifically for the value of k used for decoding (keval).
The main differences w.r.t. the IWSLT experimental results observed for the model trained for offline decoding (k = (0). Whereas on IWSLT it was competitive with the best models trained for online decoding, for this larger corpus with longer sentences, this is no longer the case and the offline model performs significantly worse than the ones specifically trained for online decoding.
Comparison to state-of-the-art. Figure 5b compares our results with state-of-the-arr performance reported by Ma et al. (2019) and Zheng et al. 2019b) for German-Enge English translation on the WMT15 dataset. Our model
7, with update) establishes new state of the art for this task, significantly improving over previous work across the full range of latency levels.
is important note that to obtain our results we use single trained model regardless of the latency level considered at decoding time, like Zheng et al. (019b) but unlike Ma et al. (2019). Moreover, we use simple wait-k decoders attain different latency levels. Zheng et al. (2019b). on the other hand, trained a data adaptive controller to decide the read/write actions, but used an underlying decoder trained for offline translation. Given our results, we expect that further gains are possible by training controllers top of model, which is trained specifically for online decoding.
Training time. We compare in Table 3 the training time of our models to the training time of the baseline and to that of our implementation of the original wait-k paper (Ma al. 2019). If training on a wait-k path, we report the average training time of & (1,3, 3. 7,9). Updating either bi-directionally or the prefix decoder states increases the training time
the encoder states dramatically. In fact, when training along the wait-k path, the encoder update scales the encoder forward- mardd-psssig time by min( [z] (x/)) the decoder update increases the decoder forward-passing time by the same factor to re-compute the states with growing context.
produce states for each context size. Similarly,
Decoding speed. We compare in Table 4 the decoding speed of our models with and without up- dating the decoder states. We also include in the comparison the decoding speed if we update the encoder states as we advance in reading the source sequence, similar to Ma et al. (2019). To factor out the training approach, we only consider the models trained for offline decoding (MT and T of
'Table 2 does not report the AI. values, which comparable across different models given fixed wait-k decoder, Le. per column. See Appendix B for results with AL values.

---

Under review as a conference paper at ICLR 2020
Dataset
Model
Training time
Our Waitkk
2.9 hours
Our wait-k with decoder update in training
121 hours
IWSLT'14 14 De-En
Wait-k with encoder update in training (Ma et al. 2019
116 hours
Uni-directional baseline (MT)
3.5 hours
Bi-directional baseline (T)
3.4 hours
Our Wait-k
389 hours
Uni dinitiirctional baseline (MT)
559 hours
1M De-En
Bi-directional baseline (T)
613 hours
Table 3: Training time of our models compared the baseline: IWSLT' 14 De-En models were trained on a single GPU with a batch-size of 4k tokens (x2 gradient accumulation), and WMT' 15 De-En models were trained on 2 GPUs with batch-size of 3.5k tokens (x64 64 gradient accumula- tion).
Table 1 and evaluate them on the wait-k paths. The reported speeds are the average of evaluating with & (,99..
Dataset
Model
Update
GPU
CPU
21.7k tokens/s
130.3 tokens/s
MT
decoder
13.4k tokens/s
97.3 tokens/s
IWSLT' De-En
encoder
7.3k tokens/s
54.2 tokens/s
encoder+decoder
5.5k tokens/s
46.3 tokens/s
Ø decoder
6.3k tokens/s
77.1 tokens/s
MT
3.5k tokens/s
40.7 tokens/s
1M De-En
encoder
2.9k tokens/s
31.9 tokens/s
T
encoder+decoder
1.6k tokens/s
20.4 tokens/s
Table 4:
Decoding speed tokens/s on both GPU (RTX2080Ti) and CPU. For each model we measure the decoding speed with and without updating the decoder states.
The decoding speeds show that updating the prefix decoder states costs less than updating the coder states with a bi-directional encoder on both GPU (IWSLT: x1.8, WMT: x1.2 speed-up) and CPU (IWSLT: x 1.8, WMT: 1.3 speed-up).
CONCLUSION
We compared transformer and 2D convolutional architectures for online machine translation with "wait-k" decoders, and proposed improved training techniques for these models. We find the trans- former architecture to perform best, and improved its performance using hidden state updates in- spired from the 2D convolutional architecture. We find that training a single model for relatively high values of k; e.g. 7 or 9, yields performance that is comparable to using separate models trained for each specific value of k. Training single model for multiple values of k improves performance for very low-latency regimes.
Our results establish new state of the art for online machine translation WMT15 De-En dataset. We improve over recent results obtained with "wait-k" decoders (Ma et al. 2019) and trained data- adaptive controllers (Zheng et al. 2019b) to schedule read/write actions. Our results, together with those of Zheng et al. (2019b). suggest that further performance improvements are possible by inte- grating models trained for online decoding with data-adaptive controllers.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Naveen Arivazhagan, Colin Cherry. Wolfgang Macherey, Chung- Chiu, Semih Yavuz, Ruom- ing Pang, Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultancous machine translation. In Proc. of ACL, 2019.
J. Ba, J. Kiros, and G. Hinton. Layer normalization. In NeurlPS Deep Learning Symposium, 2016.
M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and M. Federico. Report on the 11th IWSLT evaluation campaign. /WSLT, 2014.
Colin Cherry and George Foster. Thinking slow about latency evaluation for simultaneous machine translation. In ArXiv preprint, 2019.
K. Cho and M. Esipova. Can neural machine translation do simultaneous translation? In ArXiv preprint, 2016.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. Incremental decoding and training methods for simultaneous translation in neural machine translation. Proc. of NAACL-HLT, 2018.
M. Elbayad, L. Besacier, and J. Verbeek. Pervasive attention: 2D convolutional neural networks for sequence-to-sequeen prediction. In Proc. of CoNLL, 2018.
Christian Fügen, Alex Waibel, and Muntsin Kolss.
Simultaneous translation of lectures and
speeches. In Machine translation, 2007.
Alvin Grissom, He He, Jordan L. Boyd-Grabbe John Morgan, and Hal Daumé. Don't until the final verb wait: Reinforcement learning for simultaneous machine translation. In Proc. of EMNLP, 2014.
Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O. K. Li. Learning to translate in real-time with neural machine translation. In Proc. of EACL, 2017.
He He, Alvin Grissom, John Morgan, Jordan L. Boyd-Graber, and Hal Daumé. Syntax-based rewrit- ing for simultancous machine translation. Proc. of EMNLP, 2015.
He He, Jordan Boyd-Graber, and Hal Daumé III. Interpretese vs. translationese: The uniqueness of human strategies in simultaneous interpretation. In Proc. of NAACL-HLT, 2016a.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Proc. of CVPR, 2016b.
G. Huang, Z. Liu, L. van der Maaten, and K. Weinberger. Densely connected convolutional net- works. In Proc. of CVPR, 2017.
Navdeep Jaitly, Quoe Le, Oriol Vinyals, Ilya Sutskever, David Sussillo, and Samy Bengio. online sequence-to-sequeee model using partial conditioning. In Proc. of NeurIPS, 2016.
Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, and Ilya Sutskever. Learning online alignments with continuous rewards policy gradient. In Proc. of ICASSP, 2017.
Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. Stacl: Simultancous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. In Proc. of ACL, 2019.
Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Syntax- simultaneous translation through prediction of unseen syntactic constituents. In Proc. of ACL, 2015.
K. Papineni, S. Roukos, T. Ward, and W.-J Zhu. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL, 2002.
Ofir Press and Noah A Smith. You may need attention. In ArXiv preprint, 2018.

---

Under review as a conference paper at ICLR 2020
L. Rabiner. A tutorial on hidden Markov models and selected applications in speech recognition. In Proceedings of the IEEE, 1989.
Colin Raffel, Minh-Thame Luong, Peter Liu, Ron J Weiss, and Douglas Eck. Online and linear-time attention by enforeing monotonic alignments. In Proc. of ICML, 2017.
R. Sennrich, Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Proc. of NeurIPS, 2017.
Stephan Vogel, Hermann Ney, and Christoph Tillmann. HMM-based word alignment in statistical translation. In Proc. of COLING, 1996.
Mahsa Yarmohammadi, Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Baskaran Sankaran. Incremental segmentation and decoding strategies for simultaneous translation. In Proc. of NAACL-HLT, 2013.
Lei Yu, Jan Buys, and Phil Blunsom. Online segment to segment neural transduction. Proc. of EMNLP, 2016.
Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. Simultaneous translation with flexi- ble policy via restricted imitation learning. Proc. of ACL, 2019a.
Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. Simpler and faster learning of adap- tive policies for simultaneous translation. In Proc. of EMNLP, 2019b.
12

---

Under review as a conference paper at ICLR 2020
A DETAILS ON THE CONVOLUTIONAL ARCHITECTURE
In this section we provide more details on the modifications we made to the convolutional "pervasive attention" architecture of Elbayad et al. (2018) 8018).1 In Figure 6 we provide schematic overview of the
original convolutional architecture of Elbayad al. 2018
Linear d
Conv2D(k)
BN1
BN2
d d
(a) Building block
Linear 2d
Pervasive block
Pervasive block
Pervasive block
concat
Fonsall
(b) Overall dense architecture
Figure 6: Original convolutional architecture of Elbayad et al. (2018)
In our work, we made the following modifications to reduce the memory footprint of the model and to improve its performance.
1. We added feed-forward 1 convolutional layers after the masked convolutions for their important role in existing encoder architectures as they boost the representational power of the model.
2. We substituted the dense layer connections with residual ones to reduce the size of the features that keep increasing and costing more memory.
3. We opted for layer-normalization (Ba et al. 2016) instead of batch-normalization as it is more stable and more appropriate for causal sequences.
4. We use depth-wise separable convolutions for Conv2D(k) instead of ordinary convolutions.
In Figure
7 provide schematic overview of the adapted convolutional architecture.
Linear
Conv2D(k)
Linear
Linear
LN1
LN2
d
d
Chin
Building block
Linear
Pervasive block
Pervasive block 2
Pervasive block
concat
2d
(b) Overall residual architecture
Figure 7: Overview of the adapted convolutional architecture.
Finally, found that removing the layer normalizations all- all-together and summing the block outputs before projecting on the target vocabulary to give best results. The final high-level architecture with block output addition is given in Figure
Linear 2d
Pervasive block
Pervasive block
Pervasive block N
concat
Figure 8: Overview of the architecture with block output addition used in this paper.
13

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL EVALUATION RESULTS
In this appendix we provide additional evaluation results.
1. In addition results in the main paper measuring latency with AL, in Appendix B.] we provide results measuring latency using the AP and DAL metrics.
2. In Appendix B.2 we provide the numerical underlying the plots for IWSLT De-En in the main paper and appendix.
3. Finally, provide results for the reverse En-De translation direction for both IWSLT in Appendix B.3 and WMT in Appendix B.4
14

---

Under review as a conference paper at ICLR 2020
B.1
IWSLT DE-EN: AP AND DAL LATENCY
32
30
30
30
30
28
28
28
28
26
26
26
26
24
8
22
22
22
22
20
20
20
20
Rood
16.5
0.7
0.8
0.9"
23
18
1.5
0.8
0.7
0.9
Average proportirs
Acerae proportonn
(a) MPA wait-k (AP)
(b) MPA above path (AP)
32
Orin
32
32
30
30
30
30
28
28
28
28
26
26
26
26
24
24
--X
22
22
22
22
20
20
20
20
18
18
18
18
10
23
10
23
average lapingg
Diferentbble average Raaing
(c) MPA wait-k (DAL)
(d) MPA above path (DAL)
cmii
ceiil
O
32
32
32
32
32
32
30
30
30
30
30
28
28
28
.
28
28
28
26
26
26
26
26
27
26
24
24
24
24
22
22
22
22
20
20
20
k
20
20
0 ke
[1.5
20
18
18
k
1..9
0.5
0.7
18.5
0.7
0.9
18
0.5
0.7
0.9
18
prozor
proportin
Averago proportion
(e) MT wait-k (AP)
(f) MT [1..5] (AP)
(g) MT A [1..9] (AP)
32
32
32
32
32
32
30
30
30
28
28
28
28
28
28
26
26
26
26
26
26
24
24
24
24
24
24
22
22
k
k
20
20
20
20
20
-
11.5
20
18
18
K
[1..5
1..9
10"
23
18
8
23
18
18
18
sorrge legging
Dilerentiade swrrage legging
Celenetittle maerage leging
(h) MT wait-k (DAL)
MT [1..5] (DAL)
(i) MT [1..9] (DAL)
Figure 9: IWSLT De-En: Online decoding with single and multiple paths training. Mcasuring
latency with AP and DAL, corresponding to figures and in the main paper where AL is used.
15

---

Under review as a conference paper at ICLR 2020
B.2
IWSLT DE-E: NUMERICAL RESULTS
BLEU
DAL
BLEU
DAL
BLEU
DAL
BLEU
DAL
18.9
0.53
0.1
1.8
22.8
0.60
1.8
2.1
22.9
0.60
1.8
2.5
18.9
0.53
0.1
1.8
20.8
0.70
3.6
5.1
5
29.0
0.71
1.8
5.4
5
29.6
0.73
1.6
5.4
5
29.1
0.73
4.5
5.3
19.0
0.75
5.1
6.9
30.6
0.79
7.2
0.79
7.1
30.6
0.79
7.1
17.2
0.80
8.7
9
3166
0.84
8.9
31.0
0.83
8.8
31.5
0.84
8.9
(a) MT, A
(b) MT,
A
OO
(c) MT, k
(d) MT.
&
Keval
BLEU
AP
AL
DAL
k
BLEU
AP
AL
DAL
k
BLEU
AP
AL
DAL
BLEU
AP
AL
DAL
1.7
2.1
2.2
1.7
21.5
0.62
3.3
27.8
0.67
3.6
27.9
0.66
3.3
3.6
27.1
0.65
3.5
20.0
0.70
3.7
5.1
29.9
0.71
5.0
5.1
30.0
0.73
1.9
5.4
28.8
0.73
5.0
5.5
7.2
7.1
9
16.3
0.80
6.6
8.7
9
32.1
0.81
8.1
8.9
9
31.0
0.81
8.2
8.8
9
31.7
0.81
8.3
8.9
(c) MT, A 1- Update
(f) MT, AS
Update
(g) MT, lo 7 Update
(h) MT, A:
Keval Update
k
BLEU
AP
AI.
DAL
k
BLEU
AP
AL.
DAI
k
BLEU
AP
A1.
DAI.
18.8
0.50
-0.9
1.7
1
17.5
0.49
-1.0
1.6
17.2
0.49
-1.1
1.5
25.8
0.63
1.8
3.4
3
25.2
0.62
1.7
3.4
24.5
0.62
1.4
3.3
28.9
0.72
4.2
5.3
5
28.6
0.72
4.1
5.3
28.3
0.71
3.9
5.2
30.6
0.79
6.3
7.1
7
30.5
0.79
6.2
7.1
7
30.3
0.78
6.0
7.1
31.2
0.84
8.1
8.9
9
31.3
0.84
8.1
8.8
31.3
0.83
8.0
8.8
(i) MPA, & 3
I MPA, A > 5
(k) MPA, & 3 7
BLEU
DAL
BLEU
AI
DAL
BLEU
AP
AL
DAL
1
22.6
0.55
0.3
2.0
1
23.1
0.60
1.8
2.5
23.1
0.61
2.0
2.6
3
25.7
0.63
1.9
3.4
3
27.2
0.66
3.0
3.7
26.8
0.67
3.2
3.8
0.71
3.9
5.2
29.7
0.73
5.4
29.6
0.73
4.7
5.4
7
5.9
30.7
0.79
7.1
31.3
0.79
6.5
7.1
9
28.7
0.82
7.5
8.7
9
31.1
0.83
8.1
8.8
9
31.7
0.84
8.1
8.8
()) MT, & [1..5]
(m)
MT, & [3..7]
(n) MT, [1..9]
Update
k
BLEU
AP
AL
DAL
k
BLEU
AP
AL
DAL
BLEU
AP
AL
DAL
1 25.0
0.56
1.3
1.9
1
23.1
0.60
1.8
2.5
24.5
0.59
2.1
2.3
3
28.0
0.65
2.8
3.5
3
27.2
0.66
3.0
3.7
3
28.0
0.67
3.4
3.7
5
29.1
0.72
4.6
5.3
5
29.7
0.73
4.5
5.4
5
30.1
0.74
5.0
5.4
7
29.1
0.78
6.2
7.0
7
30.7
0.79
6.3
7.1
31.4
0.79
6.7
7.2
28.6
0.83
7.8
8.8
31.1
0.83
8.1
8.8
31.7
0.84
8.3
8.9
(0) MT,
MT, A [1.55] Update
(p) MT, & [3..]] Update
(q) MT, & [1..9] Update
Table 5: Numerical results of IWSLT De-En
B.3
IWSLT EN-DE
BLEU
AP
DAL
BLEU
AL
DAL
BLEU
AL
DAL
18.5
0.61
2.5
2.8
18.7
0.62
2.6
3.0
16.4
0.59
1.9
2.5
3
22.8
0.69
3.8
4.1
23.2
0.69
3.9
4.2
22.1
0.68
3.7
4.0
25.0
25.4
5.4
25.4
5.3
5.7
25.9
0.81
7.2
26.4
0.81
7.1
7.4
26.4
0.81
7.1
7.4
9
26.2
0.86
8.9
9.2
26.4
0.85
8.7
9.1
26.7
0.85
8.8
9.1
MT, le
(b) MT,
(c) MT,
Reval
BLEU
DAL
BLEU
AL
DAL
BLEU
AP
AL
DAL
21.2
0.60
21.4
2.3
2.5
16.6
0.57
1.6
2.1
3
21.3
0.68
3.8
4.0
21.7
0.68
3.7
3.9
22.9
0.67
3.5
3.7
25.8
0.75
5.4
5.7
26.2
0.75
5.3
5.6
25.6
0.75
5.2
5.5
7.0
7.3
26.3
0.81
7.0
7.3
26.4
0.86
8.9
9.2
26.0
0.85
8.6
9.0
26.2
0.85
8.6
9.0
(d) MT.
00 Update
(e) MT, & 7 Update
(0) MT, &
Reval Update
Table 6: Numerical results of IWSLT En-De
16

---

Under review as a conference paper at ICLR 2020
Ors
26
24
24
24
24
24
24
22
22
22
22
22
22
20
20
20
20
20
20
18
18
18
18
18
16.5
16
16
16
0.6
0.7
0.8
0.9
22
0
2
8
22
8
10
22
Asanag proparicn
Assragu lagahru
Offerendaale Seergg lagging
(a) MPA wait-k (AP)
(b) MPA wait-k (AL)
(c) MPA wait-k (DAL)
26
26
26
24
24
24
24
24
24
22
22
22
22
22
22
20
20
20
20
20
20
18
18
18
Newi
16.5
16
0.6
0.7
0.8
0.9
2
4
8
10
22
6
8
10
22
Avcroge proco fon
Ascrogo logairg
Oiforommzali avorage lapaing
(d) MPA above path (AP)
(e) above path (AL)
(f) MPA above path (DAL.)
28
26
26
26
26
24
24
24
24
24
24
22
22
22
22
22
22
20
20
20
20
20
20
18
16.5
0.6
0.7
0.8
0.9
16
10
22
16
8
10
22
Avenge propor
Average lagging
Cdlerentaade mwraage legging
(g) MT wait-k (AP)
MT wait-k (AL.)
(i) MT wait-k (DAL)
Figure 10:
IWSLT En-De: Online decoding with single and multiple paths training. Measuring
latency with AP, AL. and DAI.
17

---

Under review as a conference paper at ICLR 2020
B.4
WMT EN-DE
k
BLEU
AP
AL
DAL
k
BLEU
AP
AL
DAL
k
BLEU
AP
AL
DAL
18.3
0.53
0.3
18.3
0.56
2.4
21.3
0.62
3.3
4.1
24.0
3
24.6
0.62
3.6
23.1
0.68
1.7
5
27.0
0.68
4.3
5.4
5
27.1
0.69
4.3
5.4
24.4
0.73
6.3
7.0
7
28.4
0.75
6.2
7.2
7
28.4
0.75
6.2
7.2
25.6
0.78
7.8
8.5
9
29.3
0.79
8.0
9.0
9
29.0
0.79
8.1
9.0
9
26.8
0.82
9.4
10.0
(a) MT, k Reval
(b) MT, k =
(c) MT,
50
BLEU
DAL
k
BLEU
AP
DAL
BLEU
AP
AL
DAL
1
18.9
0.53
0.4
2.0
1
22.8
0.55
1.0
1.9
21.3
0.62
3.3
4.1
3
24.8
0.62
2.6
3.6
3
26.3
0.62
2.7
3.5
3
23.1
0.68
4.7
5.4
5
27.8
0.69
4.4
5.4
5
28.1
0.69
4.5
5.4
24.4
0.73
6.3
7.0
28.8
0.75
6.3
28.8
0.75
7.2
25.6
0.78
7.8
8.5
9
29.4
0.79
8.0
9.0
9
29.3
0.79
8.1
9.0
9
26.8
0.82
9.4
10.0
(d) MT,
Reva Update
MT. 7 Update
(f) MT.
00 Update
Table 7: Numerical results of WMT De-En underlying figures
3 and in the main paper.
Citine
30
30
30
30
28
28
26
26
26
24
24
24
24
k-7
MT k-?
22
MT k- updiate
22
MT k- oposae
K-n
20
MT k- update
20
20
MT k- updato
20
MT
MT K-::
18
Uppinte
18
18
Update
18
0.6
0.7
0.8
0.9
6 Difforentiable average lagging
27
Average praportion
(a) AP
(b) DAL
Figure 11: WMT De-En results with latency measures AP and DAL.
18


Reviews:
[{"review_id": "SyxQenqMcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Sorry, this is a very quick review.\n\nThe paper is about an improved method of training latency-limited (wait-k) decoders for transformer-based machine translation, in which the right context is limited to various numbers.  So it's a kind of augmentation method that's well matched to the test scenario.  At least, that is my understanding.\n\nI am not really an MT expert so cannot comment with much authority.  On the plus side the paper says it sets a new state of the art for latency-limited decoding for a German-English MT task, and it involves transformers, which are quite hot right now so the  attendees might find it interesting because of that connection.\nOn the minus side, it is all really quite task-specific.\nI am putting weak accept.. regular-strength accept might be my other choice.\nIt's all with low confidence.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJgHncMCtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work apply the wait-k decoding policy on the 2D CNN-based architecture and transformer.  In the transformer-based model the author proposed to recalculate the decoder hidden states when a new source token arrives. The author also suggested to train with multiple k at the decoder level with shared encoder output. The experiments showed that the transformer model provide the best quality on IWSLT14 En-De, De-En, and WMT15 De-EN.\n\nThe masking and using causal attention for the transformer has been proposed in previous works. The hidden state updates provide some gains for the model but also makes the decoder more expensive. The training with multiple k provides similar gain as training with one k larger than the value used at the inference time. Overall the contributions are limited.\n\nThere is quite some room for this paper to improve its clarify, especially in terms of annotations and explaining the proposed ideas.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xFB-O0FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends the idea of prefix-to-prefix in STACL and proposes two different variations. The authors did some interesting experiments between caching and updating decoder.\n\nMy questions are as follows:\n\n1) in section 3.1.2, the authors mentioned two adaptations. Is the proposed AR encoder uni-directional? If the AR encoder is uni-directional, then I would be surprised that the uni-directional encoder outperforms the bi-directional encoder in the original wait-k model. For the second bullet, I think the original wait-k also did the same thing(they mentioned this in the paper clearly). So there is nothing new about bullet 2. \n\n2) the idea mentioned in fig 1 is very similar to [1]. I suggest the authors compare with the aforementioned methods. \n\n3) updating the hidden state of the decoder introduces more complexity during the inference time. I recommend the authors to perform some analysis about decoding time with CPU and GPU.\n\n4) it is also interesting to show more comparison between different models' training time with the original STACL. \n\n[1] Zheng et al. \"Simultaneous Translation with Flexible Policy via Restricted Imitation Learning\" ACL 2019", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method of training latency-limited (wait-k) decoders for online machine translation. The authors investigate the impact of the value of k, and of recalculating the transformer's decoder hidden states when a new source token arrives. They significantly improve over state-of-the-art results for German-English translation on the WMT15 dataset, however there is limited novelty wrt previous approaches. The authors responded in depth to reviews and updated the paper with improvements, for which there was no reviewer response. The paper presents interesting results but IMO the approach is not novel enough to justify acceptance at ICLR. 

Decision: Reject