Year: 2019

Abstract:
Neural sequence-to-sequence models are at the basis of state-of-the-art solutions for sequential prediction problems such as machine translation and speech recognition. The models typically assume that the entire input is available when starting target generation. In some applications, however, it is desirable to start the decoding process before the entire input is available, e.g. to reduce the latency in automatic speech recognition. We consider state-of-the-art “wait-k” decoders, that first read k tokens from the source and then alternate between reading tokens from the input and writing to the output. We investigate the sensitivity of such models to the value of k that is used during training and when deploying the model, and the effect of updating the hidden states in transformer models as new source tokens are read. We experiment with German-English translation on the IWSLT14 dataset and the larger WMT15 dataset. Our results significantly improve over earlier state-of-the-art results for German-English translation on the WMT15 dataset across different latency levels.

heading: 1 INTRODUCTION
text: Sequence-to-Sequence (S2S) models are state-of-the-art for tasks where source and target sequences have different lengths, including automatic speech recognition, machine translation, speech translation, text-to-speech synthesis, etc. The most common models are composed of an encoder that reads the entire input sequence, while a decoder (often equipped with an attention mechanism) iteratively produces the next output token given the input and the partial output decoded so far. While these models perform very well in the typical offline decoding use case, few studies consider how S2S models are affected by low-latency constraints, and which architectures and strategies are the most efficient. Low-latency decoding is desirable for applications such as online speech recognition, and as-you-type machine translation. In such scenarios, the decoding process starts before the entire input sequence is available, and the output sequence is produced in an on-the-fly manner. However, if we consider for instance machine translation, online prediction generally comes at the cost of reduced translation quality and more research is needed to reach the grail of natural and high-quality online speech-to-speech interpretation.
In this paper we consider deterministic “wait-k” decoders that are state of the art for low-latency decoding (Ma et al., 2019; Zheng et al., 2019b). These decoders first read k tokens from the source, after which they proceed to alternatingly produce a target symbol and read another source symbol. We compare two architectures to implement such models: one based on a 2D-convolutional sequence-to-sequence model (Elbayad et al., 2018), and one based on the attention-based transformer architecture (Vaswani et al., 2017). For these models, we investigate the impact of the choice of k when training the models, and when using them to generate translations. For the transformer model, we also consider the effect of updating the hidden states of previous target symbols based on the full source context that is available at any moment. These updates are inspired from the 2D convolutional model, where such “updates” are an inherent consequence of the architecture.
In summary, our contributions are the following:
1. We compare transformer and 2D convolutional architectures for online machine translation. 2. We propose improved training techniques for wait-k by first using uni-directional encoders
and training across multiple values of k
3. We boost the translation quality in test-time by updating the prefix hidden-states in transformer decoders. 4. We set a new state of the art for online translation on WMT15 German-English, improving over the recent state-of-the-art results by Ma et al. (2019) and Zheng et al. (2019b) across a full range of latency levels.
The rest of this paper is organized as follows: Section 2 presents related work on low-latency machine translation; Section 3 details our low-latency models, which we evaluate in Section 4 for German-English machine translation on the IWSLT14 dataset as well as the larger WMT15 dataset. Section 5 concludes the paper.

heading: 2 RELATED WORK
text: In order to study differences between translated and simultaneously interpreted text, He et al. (2016a) produced a parallel corpus between both and showed that human interpreters regularly apply several tactics to reduce translation latency, including sentence segmentation and passivization. In machine translation, after pioneering work from Fügen et al. (2007), Yarmohammadi et al. (2013) and He et al. (2015) proposed methods to increase the alignment monotonicity for statistical machine translation. To allow for a more flexible segmentation, Grissom et al. (2014) introduced a trainable segmentation module that decides whether to write or read tokens based on the prediction of the final verb and the next token in the source sequence. Similarly, Oda et al. (2015) use output of a syntax-based statistical translation system to find the optimal segmentation strategy that maximizes the quality of the translations via greedy search and dynamic programming.
One of the first works on online translation to use attention-based sequence-to-sequence models is that of Cho & Esipova (2016), which uses manually designed non-trainable waiting criteria that dictate whether the model should commit to a read/write operation. The neural transducer of Jaitly et al. (2016) reads equally-sized chunks of the source sequence and generates output sub-sequences of variable lengths, each ending with a special token marking the end of the writing. For training, a single segmentation is chosen to optimize the likelihood of the full output sequence. Raffel et al. (2017) propose an alternative to attention using monotonic alignments. Whereas attention requires access to the full source sequence to compute the weights, monotonic alignments enable linear time computation of the weights and online decoding.
Another line of research treats general alignments of source and target sequence as a latent variable. Inspired from the HMM word alignment model used in statistical machine translation (Vogel et al., 1996), the segment-to-segment neural transducer model of Yu et al. (2016) integrates a latent alignment variable into an LSTM-based sequence-to-sequence model where the transition probabilities are conditioned on the encoder-decoder hidden states. During training the alignment is marginalized out with a forward-backward algorithm (Rabiner, 1989). However, their approach is used as an alternative to attention-based models, not for low-latency translation. Luo et al. (2017) introduce a recurrent sequence-to-sequence model with binary stochastic decision variables to either emit the next output token or advance in encoding the source sequence with a unidirectional LSTM that jointly encodes the partially generated output sequence. The stochastic decisions are optimized using a standard policy gradient reinforcement learning approach. Gu et al. (2017) propose a trainable agent emitting read/write decisions modeled as a recurrent neural network fed with the encoder and decoder current hidden states. In their framework, a left-to-right recurrent sequence-to-sequence model is first pre-trained on the full bi-texts and then fine-tuned with policy gradient to optimize a reward balancing the quality and the latency of the translations.
Dalvi et al. (2018) used a static decoding algorithm that starts with k read operations then alternates between blocks of l write operations and l read operations. Albeit simple, this approach outperforms the information based criteria of Cho & Esipova (2016) and allows for complete control of the translation delay. Their attempt to integrate incrementality in the training by pre-aligning the sourcetarget sequences and then attending over a constrained span of source positions failed to improve the translation quality. The work of Press & Smith (2018), although not tackling the simultaneous translation task, allows for emitting target tokens before reading the full input sequence. Similar to the incremental training of Dalvi et al. (2018), it requires pre-aligning the bitexts. More recently, Ma et al. (2019) trained a sequence-to-sequence model based on the transformer architecture (Vaswani et al., 2017) with an integrated “wait-k” agent that first reads k source tokens then alternate single
read-writes, similar to Dalvi et al. (2018) but using l “ 1. Wait-k approaches, were found most effective by Zheng et al. (2019b) when trained for the specific k that is used to generate translations. This, however, requires training separate models for each potential value of k used for translation. As an alternative, Zheng et al. (2019a;b) both learn an adaptive policy to produce read/write decisions in a pre-trained offline translation model. Zheng et al. (2019a) train the model for decoding along two wait-k paths and Zheng et al. (2019b) use supervised training based on an oracle read/write sequence derived from the pre-trained offline translation model. Different latency levels are achieved by thresholding the policy’s confidence, however, on most regimes their model performs worse than using transformer models directly trained for wait-k online translation. In our work, we also use wait-k decoders, but unlike Ma et al. (2019); Zheng et al. (2019a;b) we opted for uni-directional encoders and show that they are efficient to train in an online setup and can achieve better results if we update the prefix decoder states during inference.

heading: 3 LOW-LATENCY SEQUENCE-TO-SEQUENCE MODELING
text: Let px,yq be a pair of source-target sequences of respective lengths |x| and |y|. Low-latency decoding consists of executing a sequence of interleaved reading and writing operations, consuming tokens from the source x and producing tokens of the target y. Low-latency decoding paths can be represented on a |y| ˆ |x| grid, where we advance from the top left to the bottom right in a total of |x| read steps and |y| write steps. See Figure 1 for an illustration. We consider two recent sequence-to-sequence models for low-latency decoding: the transformer architecture of Vaswani et al. (2017) and the 2D convolutional architecture of Elbayad et al. (2018). We describe in Section 3.1 how each of these architectures is adapted for the task of simultaneous translation. In Section 3.2 we describe how these architectures can be trained for online decoding, based on one or more decoding paths.

heading: 3.1 LOW-LATENCY DECODING ARCHITECTURES
text: To formalize the low-latency decoding process we model it as a sequence of |y| write steps. At step t P t1, . . . , |y|u we decode the t-th target token yt, conditioning on the target prefix yăt and the source sequence read up that point xďzt . The value of zt is dictated by the decoding path.
The decoder computes the distribution over the next target token given the source and target contexts,
pθpyt|yăt,xďzt , zătq, (1) where θ denotes the parameters of the writer. It depends on the architecture of the decoder whether, given zt, the prediction of yt further depends on the full decoding zăt path that led up to this point.

heading: 3.1.1 CONVOLUTION-BASED LOW-LATENCY DECODER
text: In order for the decoder to be trained efficiently for different source/target context sizes, we build upon the “pervasive attention” architecture of Elbayad et al. (2018). Their machine translation model
uses masked 2D convolutions across a source-target grid, as in Figure 1. The masked convolutions ensure that only information from past target tokens is used to predict the next one. Here we adapt the masking pattern so as to construct the field-of-view in a way that it only extends over previous positions in both source and target dimensions. More concretely, we use a 11ˆ11 filter where only the top-left 6ˆ6 weights are non-zero. In this manner features computed by a sequence of convolutional layers at a position pt, jq in the grid can be used to define pθpyt|yăt,xďjq. As input, at each site pt, jq, the network takes the concatenation of the (sub-)word embedding of the t-th token in target and the j-th token in the source sequence. The input is then progressively transformed along a number of convolutional network layers. Let H “ phtjq for 1 ď t ď |y| and 1 ď j ď |x| be the final features of the masked convolutional neural network. Elbayad et al. (2018) apply max-pooling across the source dimension to map H to a fixed-sized vector for each target position. To define pθpyt|yăt,xďjq in our low-latency model, we use ftj “ PoolpHďt,ďjq, (2) where Poolp¨q poolsHďt,ďj in the source and/or target direction, or neither in which case ftj “ htj . Preliminary experiments with the pervasive attention architecture showed that max-pooling across the source dimension only is the best option. For the remainder of this paper we have: ftj “ Max-poolpHt,ďjq. (3) From there we generate the emission probabilities by linearly projecting the feature ftj to the dimension of the output vocabulary with the matrix W , followed by a soft-max normalization: pθpyt|yăt,xďjq “ softmaxpWftjq. (4) Due to the 2D convolutional structure of the network, the path leading up to position pt, jq in the network does not impact the prediction of yt. Therefore, we drop the dependence on zăt in Eq. (4).

heading: 3.1.2 ATTENTION-BASED LOW-LATENCY DECODER
text: The second online decoder we consider is based on the “transformer” model of Vaswani et al. (2017). Both the encoder and decoder consist of a stack of blocks. Each block consists of a multi-head selfattention followed by a position-wise fully connected feed-forward network, or equivalently a 1ˆ1 convolution. The decoder uses an additional multi-head attention block that ranges over the source token encodings.
Given a query vector qt, an attention component aggregates a set of value vectors vj in a weighted sum, based on scoring a corresponding set of key vectors kj against the query. Using the dot-product as the score, we obtain the attention aggregate at as
at “ ÿ
j
αtjvj , αtj “ exp etjř j exp etj , etj “ qJt kj . (5)
The self-attention block is designed such that only previous output tokens can be attended over, since future tokens are not available when generating targets, ensuring the decoder is autoregressive (AR).
In our work, the transformer model for online translation has the following properties:
1. A uni-directional encoder: we make the self-attention in the encoder autoregressive, so that it can encode all source prefixes in parallel during training, and encode them progressively during generation. This is achieved by substituting the mask used in Ma et al. (2019) (referred to as STACL in Eq. (6)) with the AR mask of Eq. (7).
2. A constrained encoder-decoder interaction: we mask the source attention in the decoder, so that when producing the t-th output token attention is limited to the zt source tokens read so far. This is guaranteed with the enc-dec mask of Eq. (7).
@t, eSTACLij “ " qJi kj , If i, j ď zt ´8, otherwise , (6)
eARij “ " qJi kj , If j ď i ´8, otherwise , e enc-dec tj “ " qJt kj , If j ď zt ´8, otherwise . (7)
In this transformer-based decoder, the prediction of yt depends on the full decoding path zăt. This is because the decoder self-attention ranges over decoder hidden states of previous time steps t1 ă t, which themselves used zt1 to attend over the source encoding.
In addition to the transformer-decoder described above, we consider a second transformer-based variant which removes the dependency of yt on zăt. To achieve this, each time a source token is read, we update all hidden states in the decoder based on the source context xzt available at that point. Essentially, we will re-run the decoder across the entire output sequence, using the new source context. This makes the decoder self-attention insensitive to zăt given zt, increasing the decoding cost from Op|x| ˆ |y|q to Op|x|2 ˆ |y|q. Note that the encoder still needs to be run only once, since its self-attention has been constrained to be autoregressive. We refer to this decoder as “transformer-update”, see Figure 2 for an illustration.

heading: 3.2 TRAINING LOW-LATENCY DECODERS
text: For our decoders we use pre-defined “wait-k” schedules (Dalvi et al., 2018; Ma et al., 2019). First k source tokens are read, before alternating reading and writing a single token at a time, until either the full source has been read, or the target generation has been terminated.
To train our models, we consider several strategies. The first, similar to Ma et al. (2019) is to train the model using the same wait-k decoding path that will be used for generation. That is, to train the model, we optimize a sum of loss terms, each of which corresponds to the negative log-likelihood of predicting a target token:
Lpθ,x,y, zq “ ´ |y| ÿ
t“1 ln pθpyt|yăt,xďzt , zătq. (8)
Training with k“8 corresponds to an offline (wait-until-the-end) decoder, with the exception that here we still use autoregressive dependencies in the encoder or along the source dimension in the 2D convolutional network.
In addition to training according to a single wait-k decoding path, we can use losses associated with multiple paths. The additional loss terms may provide a richer training signal, and potentially yield models that could perform well across a range of values for k during decoding. Due to the dependence of yt on the full decoding path zăt in the transformer-based model, it is not possible to improve over simply training in parallel across the different values of k. When training for multiple values of k, we encode the source sentence once, and then forward it multiple times in the decoder, once for each value of k.
The 2D CNN-based architecture, however, does not include this full dependency, and allows parallel computation of all pθpyt|yăt,xďjq in single evaluation of the CNN. For this architecture we therefore use a loss of the form
Lpθ,x,y, qq “ ´ |y| ÿ
t“1
|x| ÿ j“1 qtj ln pθpyt|yăt,xďjq, (9)
where qtj weight the terms corresponding to different positions in the decoding grid of Figure 1. The weights qtj can be set to zero/one to reflect all positions corresponding to wait-k decoding paths for one or more values of k. In our experiments we also consider the option of activating all loss terms above a certain wait-k decoding path. The latter corresponds to the sum of all loss terms on all wait-k1 decoding paths with k ď k1 ď |x|.1
1Summing the losses for all paths, though, would weight shared loss terms more importantly.
The transformer-update architecture also avoids the full dependency on zăt, but in order to train from all loss terms above a certain decoding path, it requires a separate decoder run for each source context size, i.e. for each column of the decoding grid in Figure 1.

heading: 4 EXPERIMENTAL EVALUATION
text: In this section we present our experimental setup, followed by quantitative results.

heading: 4.1 DATASETS AND EXPERIMENTAL SETUP
text: Datasets. We evaluate our approach on IWSLT14 En-De (Cettolo et al., 2014) and on the WMT15 De-En datasets.2 Replicating the setup of Elbayad et al. (2018) for IWSLT14 De-En, we train on 160K sentence pairs, develop on 7K held-out pairs and test on 6.7K pairs. The vocabulary consists of 8.8K types on the source side and 6.6K types on the target side that were obtained from a joint source and target byte pair encoding (BPE; Sennrich et al. 2016). For WMT15 De-En we use a joint vocabulary of 32k BPE types. We train on 4.5M pairs, develop on newstest2013 (3K pairs) and test on newstest15 (2.2K pairs).
Evaluation metrics. We use beam-search decoding, with a beam of size 5, for offline models and only greedy decoding online. We evaluate the translation quality by measuring case-sensitive tokenized BLEU (Papineni et al., 2002) with multi-bleu.pl.3 Cho & Esipova (2016) measure the decoding latency using the proportion of the source sentence which has been read when producing target tokens, and average this proportion across the tokens in the generated sentence:
AP “ 1|y| |y| ÿ
t
zt{|x|. (10)
More recently, average lagging (AL) and differentiable average lagging (DAL), were proposed to measure translation latency (Ma et al., 2019; Cherry & Foster, 2019; Arivazhagan et al., 2019):
AL “ 1 τ
τ ÿ
t
zt ´ t´ 1 γ , DAL “ 1|y| |y| ÿ
t
z1t ´ t´ 1 γ , z1t “ # zt if t “ 1 maxpzt, z1t´1 ` 1γ q ,
where γ “ |y|{|x| and τ “ mintt | gptq “ |x|u. These metrics handle differing source and target lengths more properly, and have an intuitive interpretation as the average by which the system lags behind an ideal instantaneous translator. We refer the reader to (Cherry & Foster, 2019) for details.
We report the mean of these metric across translations. In this section we report the average lagging (AL) metric, while the other metrics (AP, DAL, leading to a similar trend) are given in Appendix B. AL is chosen to make our results comparable with those of Ma et al. (2019) and Zheng et al. (2019b) for German-English WMT15 dataset.
Architectures. For the pervasive attention model, to reduce the memory footprint, we use residual skip connections (He et al., 2016b) rather than the dense connections (Huang et al., 2017) used by Elbayad et al. (2018). See Appendix A for more details about other minor changes made to the original architecture. We consider an offline pervasive attention baseline (PA) that reads the source sequence bi-directionally via asymmetric convolution filters that are only masking the future targets. Note that this baseline is not directly applicable for low-latency decoding, due to the bidirectional source encoding. We use a second baseline where convolutions are masked along the source dimension, which we refer to as masked-PA baseline (MPA).
For the Transformer model, we use a small architecture on IWSLT with an embedding dimension denc “ 512 for the encoder, ddec “ 256 for the decoder, and N “ 6, dff “ 1024,h “ 4,Pdrop “ 0.3. On WMT, we use a Transformer base (Vaswani et al., 2017) with tied embeddings. Similar to the pervasive attention baseline, we consider the offline baseline (T) with bidirectional self-attention in the source side and a masked baseline (MT) with left-to-right self-attention in the encoder.
2http://www.statmt.org/wmt15/ 3https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/
multi-bleu.perl

heading: 4.2 OFFLINE DECODING BASELINE RESULTS
text: In Table 1 we report the offline performance of the Pervasive Attention (PA), Masked Pervasive Attention (MPA), Transformer (T) and Masked Transformer (MT) baselines. Each with greedy decoding (G) and with beam search (BS).
Interestingly, the masked and non-masked versions perform quite similar, across both datasets and architectures. This validates the use of autoregressive source encoders in the low-latency models we consider below. The transformer and convolutional architectures perform similarly on IWSLT14, while the former performs significantly better on WMT15.

heading: 4.3 ONLINE DECODING RESULTS
text: In our first experiment we evaluate the wait-k online decoding for different MT and MPA models, trained for different wait-k decoding paths. We denote with k“8 the wait-until-end training where the full source is read before decoding. In each figure the offline results are added for reference, the offline model has a latency of AL “ |x|. Here we show results for IWSLT De-En, similar trends are observed for IWSLT En-De (see Appendix B.3).
Impact of the architecture. Figure 3 presents the performance of models trained for wait-k decoding across a range of latencies keval P t1, 3, 5, 7, 9u. Each trained model is represented by a curve, by evaluating it across different wait-k decoding paths.
For the convolutional architecture we observe that a single model trained for a relatively large value of k, e.g. 7 or 9, provides good performance when using it to generate using different k values for generation. The performance of the transformer-based model is more sensitive to the value of k used for training. This difference in sensitivity could be due to the use of input-independent convolution filters vs. self-attention which relies on scoring context to weight its features.
Given an appropriate value for k during training, the transformer-based models yield better translations for a given latency level.
0 2 4 6 818
20
22
24
26
28
30
32
1
3
5 7 9
Average lagging
B LE
U
k “ 1 k ě 1 k “ keval k ě keval
23 18
20
22
24
26
28
30
32Offline
(a) MPA above path
0 2 4 6 818
20
22
24
26
28
30
32
Average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k P r1..5s
23 18
20
22
24
26
28
30
32 Offline
(b) MT k P r1..5s
0 2 4 6 818
20
22
24
26
28
30
32
Average lagging
B LE
U
k “ 9 k P r1..5s k P r1..9s
23 18
20
22
24
26
28
30
32 Offline
(c) MT k P r1..9s
Impact of multiple path training. For the convolutional model (MPA), we consider activating loss terms in the area above the wait-k path. Training on this area covers a combinatorially large number of paths with latency higher than k. The results in Figure 4a show that for the MPA architecture, multiple path training can be beneficial compared to single path (k“keval vs. k ě keval). More importantly, the model trained with all loss terms above the diagonal of the decoding grid (kě 1), performs equal or better than models trained with only a subset of the terms, across all tested latency levels.
For transformer model (MT), training for all decoding paths above a certain wait-k path is prohibitively costly (since the cost is linear in the number of training paths). Therefore, we opted instead for joint training for a selected set of wait-k paths. Results in Figure 4b show that jointly training on the wait-k paths from k“1 to k“5 improves over the training on individual paths in lowlatency regimes (keval ď 3). For higher latency a relatively small drop in performance is observed compared to training with k “ 5. Where the convolutional model could be trained near optimal by training across all values of k, in Figure 4c we see that training the paths from k“1 to k“9 performs similarly to just training with k“9. The latter model is (near) optimal for ALě 3, but for very low-latency the model trained on k P r1..5s is better. Impact of hidden state updates. The standard decoding approach with a Transformer is to use previously evaluated hidden states to encode the target prefix written so far. This is not the case for MPA where new source context is integrated in the convolution for all previous time-steps. To bridge the gap between the two decoding paradigms, we update previous hidden states to account for new source contexts.
In Table 2 we compare transformer models with and without hidden state updates, trained along one or multiple decoding paths. Among the 20 comparisons (4 models, and 5 wait-k decoders) we
observe only a single case where state updates drop the BLEU score by 0.1. In all other cases, the state updates lead to improvements, by up to 2.4 BLEU points in very low latency settings.4
Experiments on the WMT15 De-En corpus. In Figure 5a we present our results on the WMT15 De-En task using transformer model, which gave the best results in the IWSLT experiments. The results confirm observations made on IWSLT: (i) Hidden state updates consistently improve the performance of the tested models. (ii) When using hidden state updates, a single low-latency model (k“7) performs better or comparable to using separate models trained specifically for the value of k used for decoding (keval).
The main differences w.r.t. the IWSLT experimental results are observed for the model trained for offline decoding (k“8). Whereas on IWSLT it was competitive with the best models trained for online decoding, for this larger corpus with longer sentences, this is no longer the case and the offline model performs significantly worse than the ones specifically trained for online decoding.
Comparison to state-of-the-art. Figure 5b compares our results with state-of-the-art performance reported by Ma et al. (2019) and Zheng et al. (2019b) for German-English translation on the WMT15 dataset. Our model (k“7, with update) establishes a new state of the art for this task, significantly improving over previous work across the full range of latency levels.
It is important to note that to obtain our results we use a single trained model regardless of the latency level considered at decoding time, like Zheng et al. (2019b) but unlike Ma et al. (2019). Moreover, we use simple wait-k decoders to attain different latency levels. Zheng et al. (2019b), on the other hand, trained a data adaptive controller to decide on the read/write actions, but used an underlying decoder trained for offline translation. Given our results, we expect that further gains are possible by training controllers on top of our model, which is trained specifically for online decoding.
Training time. We compare in Table 3 the training time of our models to the training time of the baseline and to that of our implementation of the original wait-k paper (Ma et al., 2019). If training on a wait-k path, we report the average training time of k P t1, 3, 5, 7, 9u. Updating either the encoder states to use bi-directionally or the prefix decoder states increases the training time dramatically. In fact, when training along the wait-k path, the encoder update scales the encoder forward-passing time by minp|x|, |y|q ´ k ` 1 to produce states for each context size. Similarly, the decoder update increases the decoder forward-passing time by the same factor to re-compute the states with growing context.
Decoding speed. We compare in Table 4 the decoding speed of our models with and without updating the decoder states. We also include in the comparison the decoding speed if we update the encoder states as we advance in reading the source sequence, similar to Ma et al. (2019). To factor out the training approach, we only consider the models trained for offline decoding (MT and T of
4Table 2 does not report the AL values, which are comparable across different models given a fixed wait-k decoder, i.e. per column. See Appendix B for results with AL values.
Table 1) and evaluate them on the wait-k paths. The reported speeds are the average of evaluating with k P t1, 3, 5, 7, 9u.
The decoding speeds show that updating the prefix decoder states costs less than updating the encoder states with a bi-directional encoder on both GPU (IWSLT: ˆ1.8, WMT: ˆ1.2 speed-up) and CPU (IWSLT: ˆ1.8 , WMT: ˆ1.3 speed-up).

heading: 5 CONCLUSION
text: We compared transformer and 2D convolutional architectures for online machine translation with “wait-k” decoders, and proposed improved training techniques for these models. We find the transformer architecture to perform best, and improved its performance using hidden state updates inspired from the 2D convolutional architecture. We find that training a single model for relatively high values of k, e.g. 7 or 9, yields a performance that is comparable to using separate models trained for each specific value of k. Training a single model for multiple values of k improves performance for very low-latency regimes.
Our results establish a new state of the art for online machine translation on WMT15 De-En dataset. We improve over recent results obtained with “wait-k” decoders (Ma et al., 2019) and trained dataadaptive controllers (Zheng et al., 2019b) to schedule read/write actions. Our results, together with those of Zheng et al. (2019b), suggest that further performance improvements are possible by integrating our models trained for online decoding with data-adaptive controllers.

heading: A DETAILS ON THE CONVOLUTIONAL ARCHITECTURE
text: In this section we provide more details on the modifications we made to the convolutional “pervasive attention” architecture of Elbayad et al. (2018). In Figure 6 we provide a schematic overview of the original convolutional architecture of Elbayad et al. (2018).
In our work, we made the following modifications to reduce the memory footprint of the model and to improve its performance.
1. We added feed-forward 1ˆ1 convolutional layers after the masked convolutions for their important role in existing encoder architectures as they boost the representational power of the model.
2. We substituted the dense layer connections with residual ones to reduce the size of the features that keep increasing and costing more memory.
3. We opted for layer-normalization (Ba et al., 2016) instead of batch-normalization as it is more stable and more appropriate for causal sequences.
4. We use depth-wise separable convolutions for Conv2D(k) instead of ordinary convolutions.
In Figure 7 we provide a schematic overview of the adapted convolutional architecture.
Finally, we found that removing the layer normalizations all-together and summing the block outputs before projecting on the target vocabulary to give best results. The final high-level architecture with block output addition is given in Figure 8.

heading: B ADDITIONAL EVALUATION RESULTS
text: In this appendix we provide additional evaluation results.
1. In addition to results in the main paper measuring latency with AL, in Appendix B.1 we provide results measuring latency using the AP and DAL metrics.
2. In Appendix B.2 we provide the numerical underlying the plots for IWSLT De-En in the main paper and appendix.
3. Finally, we provide results for the reverse En-De translation direction for both IWSLT in Appendix B.3 and WMT in Appendix B.4.
B.1 IWSLT DE-EN: AP AND DAL LATENCY
0.5 0.6 0.7 0.8 0.918
20
22
24
26
28
30
32
1
3
5 7 9
Average proportion
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
2318
20
22
24
26
28
30
32Offline
(a) MPA wait-k (AP)
0.5 0.6 0.7 0.8 0.918
20
22
24
26
28
30
32
1
3
5 7 9
Average proportion
B LE
U
k “ 1 k ě 1 k “ keval k ě keval
1 18
20
22
24
26
28
30
32Offline
(b) MPA above path (AP)
0 2 4 6 8 1018
20
22
24
26
28
30
32
1
3
5 7 9
Differentiable average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
23 18
20
22
24
26
28
30
32Offline
(c) MPA wait-k (DAL)
0 2 4 6 8 1018
20
22
24
26
28
30
32
1
3
5 7 9
Differentiable average lagging
B LE
U
k “ 1 k ě 1 k “ keval k ě keval
23 18
20
22
24
26
28
30
32Offline
(d) MPA above path (DAL)
0.5 0.6 0.7 0.8 0.9 18
20
22
24
26
28
30
32
1
3 5
7
9
Average proportion
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
1 18
20
22
24
26
28
30
32 Offline
(e) MT wait-k (AP)
0.5 0.6 0.7 0.8 0.918
20
22
24
26
28
30
32
Average proportion
B LE
U
k “ 1 k “ 3 k “ 5 k P r1..5s
1 18
20
22
24
26
28
30
32 Offline
(f) MT k P r1..5s (AP)
0.5 0.6 0.7 0.8 0.918
20
22
24
26
28
30
32
Average proportion
B LE
U
k “ 9 k P r1..5s k P r1..9s
1 18
20
22
24
26
28
30
32 Offline
(g) MT k P r1..9s (AP)
0 2 4 6 8 10 18
20
22
24
26
28
30
32
1
3 5
7
9
Differentiable average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
23 18
20
22
24
26
28
30
32 Offline
(h) MT wait-k (DAL)
0 2 4 6 818
20
22
24
26
28
30
32
Differentiable average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k P r1..5s
23 18
20
22
24
26
28
30
32 Offline
(i) MT k P r1..5s (DAL)
0 2 4 6 818
20
22
24
26
28
30
32
Differentiable average lagging
B LE
U
k “ 9 k P r1..5s k P r1..9s
23 18
20
22
24
26
28
30
32 Offline
(j) MT k P r1..9s (DAL)
Figure 9: IWSLT De-En: Online decoding with single and multiple paths training. Measuring latency with AP and DAL, corresponding to figures 3 and 4 in the main paper where AL is used.
B.2 IWSLT DE-E: NUMERICAL RESULTS
B.3 IWSLT EN-DE
k BLEU AP AL DAL
1 18.5 0.61 2.5 2.8 3 22.8 0.69 3.8 4.1 5 25.0 0.75 5.5 5.7 7 25.9 0.81 7.2 7.5 9 26.2 0.86 8.9 9.2
(a) MT, k “ 8
k BLEU AP AL DAL
1 18.7 0.62 2.6 3.0 3 23.2 0.69 3.9 4.2 5 25.4 0.75 5.4 5.8 7 26.4 0.81 7.1 7.4 9 26.4 0.85 8.7 9.1
(b) MT, k “ 7
k BLEU AP AL DAL
1 16.4 0.59 1.9 2.5 3 22.1 0.68 3.7 4.0 5 25.4 0.75 5.3 5.7 7 26.4 0.81 7.1 7.4 9 26.7 0.85 8.8 9.1
(c) MT, k “ keval k BLEU AP AL DAL
1 21.2 0.60 2.3 2.6 3 24.3 0.68 3.8 4.0 5 25.8 0.75 5.4 5.7 7 26.2 0.81 7.2 7.5 9 26.4 0.86 8.9 9.2
(d) MT, k “ 8 - Update
k BLEU AP AL DAL
1 21.4 0.60 2.3 2.5 3 24.7 0.68 3.7 3.9 5 26.2 0.75 5.3 5.6 7 26.3 0.81 7.0 7.3 9 26.0 0.85 8.6 9.0
(e) MT, k “ 7 - Update
k BLEU AP AL DAL
1 16.6 0.57 1.6 2.1 3 22.9 0.67 3.5 3.7 5 25.6 0.75 5.2 5.5 7 26.3 0.81 7.0 7.3 9 26.2 0.85 8.6 9.0
(f) MT, k “ keval - Update
Table 6: Numerical results of IWSLT En-De
0.5 0.6 0.7 0.8 0.916
18
20
22
24
26
1
3
5
7 9
Average proportion
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
2216
18
20
22
24
26 Offline
(a) MPA wait-k (AP)
0 2 4 6 816
18
20
22
24
26
1
3
5
7 9
Average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
22 16
18
20
22
24
26 Offline
(b) MPA wait-k (AL)
0 2 4 6 8 1016
18
20
22
24
26
1
3
5
7 9
Differentiable average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
22 16
18
20
22
24
26 Offline
(c) MPA wait-k (DAL)
0.5 0.6 0.7 0.8 0.916
18
20
22
24
26
1
3
5
7 9
Average proportion
B LE
U
k “ 1 k ě 1 k “ keval k ě keval
1 16
18
20
22
24
26 Offline
(d) MPA above path (AP)
0 2 4 6 8 1016
18
20
22
24
26
1
3
5
7 9
Average lagging
B LE
U
k “ 1 k ě 1 k “ keval k ě keval
22 16
18
20
22
24
26 Offline
(e) MPA above path (AL)
0 2 4 6 8 1016
18
20
22
24
26
1
3
5
7 9
Differentiable average lagging
B LE
U
k “ 1 k ě 1 k “ keval k ě keval
22 16
18
20
22
24
26 Offline
(f) MPA above path (DAL)
0.5 0.6 0.7 0.8 0.916
18
20
22
24
26
1
3
5
Average proportion
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
1 16
18
20
22
24
26 Offline
(g) MT wait-k (AP)
0 2 4 6 8 1016
18
20
22
24
26
1
3
5
Average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
22 16
18
20
22
24
26 Offline
(h) MT wait-k (AL)
0 2 4 6 8 1016
18
20
22
24
26
1
3
5
Differentiable average lagging
B LE
U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “ 8
22 16
18
20
22
24
26 Offline
(i) MT wait-k (DAL)
Figure 10: IWSLT En-De: Online decoding with single and multiple paths training. Measuring latency with AP, AL and DAL
B.4 WMT EN-DE

Review summaries:
This work apply the wait-k decoding policy on the 2D CNN-based architecture and transformer.  In the transformer-based model the author proposed to recalculate the decoder hidden states when a new source token arrives. The author also suggested to train with multiple k at the decoder level with shared encoder output. The experiments showed that the transformer model provide the best quality on IWSLT14 En-De, De-En, and WMT15 De-EN.