Learning to Use Future Information in Simultaneous Translation
Authors: Xueqing Wu
Yingce Xia
Lijun Wu
Shufang Xie
Weiqing Liu
Tao Qin
Tie-Yan Liu
Venue:  (2021)
Keywords: sequence learning
simultaneous machine translation

Abstract: Simultaneous neural machine translation (briefly, NMT) has attracted much attention recently. In contrast to standard NMT, where the NMT system can access the full input sentence, simultaneous NMT is a prefix-to-prefix problem, where the system can only utilize the prefix of the input sentence and thus more uncertainty and difficulty are introduced to decoding. Wait-k inference is a simple yet effective strategy for simultaneous NMT, where the decoder generates the output sequence $k$ words behind the input words. For wait-k inference, we observe that wait-m training with $m>k$ in simultaneous NMT (i.e., using more future information for training than inference) generally outperforms wait-k training. Based on this observation, we propose a method that automatically learns how much future information to use in training for simultaneous NMT. Specifically, we introduce a controller to adaptively select wait-m training strategies according to the network status of the translation model and current training sentence pairs, and the controller is jointly trained with the translation model through bi-level optimization. Experiments on four datasets show that our method brings 1 to 3 BLEU point improvement over baselines under the same latency. Our code is available at https://github.com/P2F-research/simulNMT .

Under review as a conference paper at ICLR 2021
LEARNING TO USE FUTURE INFORMATION IN SIMULTANEOUS TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Simultaneous neural machine translation (briefly, NMT) has attracted much atten- tion recently. In contrast standard NMT, where the NMT system can access the full input sentence, simultaneous NMT is a prefix-to-prefix problem, where the system can only utilize the prefix of the input sentence and thus more uncertainty and difficulty are introduced to decoding. Wait-k (Ma et al., 2019) inference is a simple yet effective strategy for simultaneous NMT, where the decoder generates the output sequence k words behind the input words. For wait-k inference, we observe that wait-m training with m in simultaneous NMT (i.e., using more future information for training than inference) generally outperforms wait-k train- ing. Based on this observation, we propose a method that automatically learns how much future information to use in training for simultaneous NMT. Specifically, we introduce a controller to adaptively select wait-m training strategies according to the network status of the translation model and current training sentence pairs, and the controller is jointly trained with the translation model through bi-level optimization. Experiments on four datasets show that method brings 1 to 3 BLEU point improvement over baselines under the same latency. Our code is available httpt::///// /tttps://githul com/P2F research/ aassearchssimulMM simulNMT
INTRODUCTION
Neural machine translation (NMT) an important task for the machine learning community and many advanced models have been designed (Sutskever et al.] 2014: Bahdanau al. 2014 Vaswani ett al. 2017). In this work, we work on a more challenging task in NMT, simultaneous translation (also known simultaneous interpretation), which widely used in international conferences, summits and business. Different from standard NMT. simultancous NMT has a stricter requirement for latency. We cannot wait to the end of a source sentence but have to start the translation right after reading the first few words. That is, the translator is required to provide instant translation based on partial source sentence.
Simultaneous NMT is formulated as a prefix-to-prefix problem (Ma et al. 2019 2020 Xiong et al. 2019), where a prefix refers to a subsequence starting from the beginning of the sentence to be translated. In simultaneous NMT, we face more uncertainty than conventional NMT, since the translation starts with a partial source sentence rather than the complete one. Wair-k inference (Ma) cet al. 2019) is simple yet effective strategy in simultaneous NMT where the translation is k words behind the source input. Rather than instant translation of each word, wait-k inference actually leverages l more future words during inference phase. Obviously, larger i can bring more future information, and therefore results in better translation quality but at the cost of larger latency. Thus, when used in real-world applications, we should have a relatively small k for simultaneous NMT.
While only small l values are allowed in inference, we observe that wait-m training with m > will lead better accuracy for wait-k inference. F the results of training with wait-m but test with wait-3 on IWSLT' 14 Banglish-+Germme
translation dataset. If training with m = 3, we will obtain 22.79 BLEU score. If set m to larger values such as 7, 13 or 21 and test with wair-3, we can get better BLEU scores. That is, the model can benefit from the availability of more future information training. This is consistent with the observation in (Ma et al. 2019).
The challenge is how much future information we should use in training. As shown in Figure I using more future information does not monotonically improve the translation accuracy of wail-k inference,

---

Under review as a conference paper at ICLR 2021
mainly because that more future information results in larger mismatch between training and inference (i.... m k more words are used in training than inference). Besides, due to the diversity of the nat- ural language, intuitively, using different m's for different sentences will lead to better performance. Even for the same sentence pair, the optimal for training might vary in different training stages. In this work, we propose an algorithm that can auto- matically determine how much future information to use in training for simultaneous NMT. Given a pre-defined k; we want to maximize the performance of wait-k inference. We have a set of training strategies wait-m with different waiting thresholds m (m E (1.2 M)). We introduce a controller such that given training sample, the controller dy- namically selects one of these training strategies so maximize the validation performance on wail- k inference. Which wait-m training strategy to select based on the data itself and the network status of the current translation model. The controller and the translation model are jointly trained, and the learning process is formulated as bi-level optimization problem (Sinha et al., (018)) where one optimization problem is nested within another.
23.0
22.9
22.5
Figure Preliminary exploration of IWSLT English-to-Germanl simultaneous NMT. x- axis represents the waiting threshold m dur- ing training and y-axis represents the BLEU scores testing with wait-3.
Our contribution is summarized follows:
(1) We propose new method for simultancous NMT. where a controller is introduced to adaptively determine how much future information to use for training. The controller and the translation model are jointly learned through bi-level optimization.
(2) Experiments on four datasets show that our method improves the wait-k baseline by to BLEU scores, and also consistently outperforms several heuristic baselines leveraging future information.
RELATED WORK
Previous work on simultaneous translation can categorized by whether using a fixed decoding scheduler or adaptive one. Fixed policies usually use pre-defined rules to determine when to read or to write a new token (Dalvi et al. 2018 Ma et al. 2019). Wait-k is the representative method for fixed scheduler (Ma et al.] 2019), where the decoding always & words behind the source input. Wair-k k achieves good results in terms of translation quality and controllable latency, and has been used in speech-related simultaneous translation (Zhang al. 2019
Ren et al., 2020).
proposed wait-if-worse (WIW) and wait-if-diff (WID) methods which generate new target word if probability does not decrease (for WIW) or the generated word is unchanged (for WID) after reading new source token. Crissom II al. (2014) and Gu et al. (2017) used reinforcement learning train the read/write controller, while Zheng et al. (2019)) obtained it in a supervised way. Alinejad et al. (2018) added a "predict" operator to the controller that can anticipate future source inputs. Zheng et al. (2019b) introduced "delay" token into the target vocabulary indicating that the model should read new word instead of generating a new one. |Arivzhhagan al. (2019) proposed monotonic infinite lookback attention (MILk), which first used a hard attention model to determine when to read new source tokens, and then soft attention model perform translation. Ma et al. (2020) extended MILk into a multi-head version and proposed monotonic multihead attention (MMA) with two variants: MMA-IL (Infinite Lookback) which has higher translation quality by looking back at all available source tokens, and MMA-H(ard) which is more computational efficient by limiting the attention span.
For methods that use adaptive schedulers, Cho & Esipova 2016
Besides, Zheng et al. (020a) extended wait-k to an adaptive strategy by training multiple wait-m models with different m 's and adaptively selecting decoding strategy during inference. Zheng et al. (20205) explored a new setting, where at each timestep, the translation model over generates the target words and corrects them in timely fashion.
2

---

Under review as a conference paper at ICLR 2021
3 PROBLEM FORMULATION AND BACKGROUND
In this section, we first introduce the notations used in this work, followed by the formulation of wait-k strategy, and then we introduce network architecture adapted from (Ma et al. 2019).
3.1 NOTATIONS AND FORMULATION
Let X' and y denote the source language domain and target language domain. For any a E X and y E V. let 2; and Yi denote the i-th token x and y respectively. Ta and Ly are the numbers of tokens in and Zggt represents prefix of I, which is the subsequence æ1; "2,"". Tt, and similarly for Vge. Let Dr and va denote the training and validation sets, both of which are collections of bilingual sentence pairs.
The wail-k strategy (Ma al., 2019) defined as follows: given an input x E X, the generation of the translation y is always tokens behind reading æ. That is, at the I-th decoding step, we generate token Yt based on æ<< B=<(+k-1 (more strictly, 2 (t-nnttk--al }). Our goal is to obtain a model f: X + y with parameter 0 that achieve better results with wait-k inference.
3.2 MODEL ARCHITECTURE
Our model for simultaneous NMT is based on Transformer model (Vaswani et al. (017). The model includes an encoder and decoder, which are used for incrementally processing the source and target sentences respectively. Both the encoder and decoder are stacked of L blocks. We mainly introduce the differences compared with the standard Transformer.
(1) Incremental encoding: Let ly, denote the output of the 1-th position from block L. For ease of reference, let Hz denote (h4, hi.... h,], and let h? denote the embeddings of the t-th token, An attention model attnnqg K, V), takes a query E IRd, set of keys and values V as inputs. K and V of equal size, q E Id where 24 the dimension, ki Rd and oz IRd are the i-th key and value. attn is defined as follows:
K
[K|
a.Wo a, exp((W (WWk:)) Z Σ exp((W,g) (Wak))) (W.sk))
attn(q, Σ
Z
where W's the parameters to be optimized. In the encoder side, hț is obtained in unidirectional way: h/ attn 11' ". That is, the model can only attend the previously generated hidden representations, and the computation complexity ((L2). In comparison, (Ma et al. 2019) still leverages bidirectional attention, whose computation complexity is O(L2). We find that unidirectional attention is much more efficient than bidirectional attention without much accuracy drop (see Appendix [DI] for details).
(2) Incremental decoding: Since we use wait-k strategy, the decoding starts before reading all inputs. At the i-th decoding step, the decoder can only read &- When 1 T,p k: the decoder greedily generates token at each step, i.e., the token Yt arg max P(wlyse-- H H 1). where V is the vocabulary of the target language. When { > L, k. the model has read the full input sentence and can generate words using beam search (a et al. 2019)
OUR METHOD
We first introduce our algorithm in Section 4.1 and then we discuss its relationship with several other heuristic algorithms that leverage future information in Section 4.2
4.1 ALGORITHM
Let f(... :)) denote a translation model parameterized by 6, and let denote the controller parame- terized by to guide the training process of f- f(-.. 8* (w)) is the translation model obtained under the guidance the controller cp... ; w), where 8* (w) is the corresponding parameter. For each train- ing data (Ir, v). the controller yo adaptively assigns a training task wail-m, where E (1.2,... M).
3

---

Under review as a conference paper at ICLR 2021
and M E Z+, is pre-defined hyperparameter. The imput of consists of two parts: (i) the information of the training data (z, y); (ii) the network status of the translation model f. For case of reference, denote these imput features as 1. We will discuss how to design Iz,,,f in Section 5.1
Let MD(Dvai 0" (w)) denote the validation metric, which evaluated on the validation set Dva with model f(... 0 (w)) and wait-k inference. We formulate the training process of and yp bi-level optimization, where two optimization problems are nested together. In the inner-optimization, given the training
w. want to obtain the model f(.....'' (w)) that can minimize the loss function set Dur under the guidance of the controller (p-... ,w). In the outer-optimization, given a translation model #* (w), we optimize to maximize the validation performance Mg. The mathematical formulation is shown as follows:
max M& (Dva: 0" (w));;
s.t. (w) = arg min
Σ (2.g)~-D
Emwolteng (o ((x, y, n,m,;
Durl
(2)
where ((2, y, m;
Σ log P(ylz;;00mm (a.9)
Σ (x.y)
log P(o:lBst-1,
We optimize Equation 2 in an alternative way, where we first optimize 0 with given w, and then update w using the REINFORCE algorithm. We repeat the above process until convergence. Details are in Algorithm
Algorithm 1: The optimization algorithm.
Input: Training episode E; internal update iterations T; learning rate no of the translation model; learning rate ?hw of the controller, batch size B; initial parameters w, 8; do
for e 1 1
Init buffer to store states and actions: B = ():
for t <- 1 : T T do
Randomll sample
Mini-batch&e of data De, from Du with batch size B;
Assign a wait-m task to each data: ò = (xx, y, m2)(2, y) E De,t, m ~ ()) where the batch size is B, and m is sampled from to the output distribution of y (......mmlll m2) y, m) D):
Update the buffer: B B {(I.. f.
Update the translation model: 0 + 0 (no V (x.g.m)E (1x, y, y,m;
Calculate the validation performance as the reward: R. ~M (Da:: 0er)): Update the controller: w t w + n. E( V. log P(y(I:w) = m).
10
Return 0.
Algorithm D consists E episodes (i.e., the outer loop), and each episode consists of update iterations (I.e., the inner loop). The inner loop (from step 4 step 8) aims optimize the 9, where we can update the parameter with any gradient based optimizer like momentum SGD, Adam (Kingmal & Bal 2 etc. The outer loop (from step 2 to step 10) aims optimize w. (o can be regarded as a policy network, where the state is Iz.2.f. the action is the choice of the task wait-m, m {1, 2. .M), }. and the reward the validation performance R, (step 9). At the end of each episode, we update w using REINFORCE algorithm (step 10).
4.2 DISCUSSION
To use more information and to obtain better wait-k inference, there are several heuristic methods:
(1) Wait-k" We train M translation models using the wait-m strategy, 21,2, .M,, select the best waiting threshold k- according to the validation performance, and use the corresponding model for wait-k inference.
(2) Random sampling (briefly, Random): For each training data, randomly choose m from (1,2, M) with equal probability and using wait-m training.
4

---

Under review as a conference paper at ICLR 2021
(3) Currieulum learning (briefly, CL): We gradually decrease from M to the threshold &s we will use in the inference (see Appendix A.2 for mathematical definition).
However, common drawback of them is that they cannot dynamically adjust is according to the sentence representation and the model status. In wait-k", during training, each sentence pair is treated with the same k". For Random and CL., the selection of waiting threshold is predefined and not adjusted according to the training. We overcome this difficulty by introducing a controller, which is trained via reinforcement learning maximize the validation performance, and is able to adaptively determine how much exploration the model requires and how long we should use a specific wait-m strategy. Therefore, our method expected to outperform wair-k" Random and CL.
We also extend our method to an adaptive version by combining with Zheng et al. (2020a). With set of pre-trained wait-m models with different m values. Zheng et al. (2020a) adaptively selects the waiting threshold during inference. Therefore, we can combine ZZhnng et al. (2020a) with our method, where the wait-m models are obtained through our strategy.
5 EXPERIMENTS
We work on the text-to-text simultaneous NMT in this paper. Let us briefly denote English, German, Vietnamese and Chinese as En, De, Vi and Zh respectively. We conduct experiments on three small- scale datasets: IWSLT" 14 En->e,, I5 En and IWSLT' 17 En-*Zh, and large-scale dataset: 1M En translation.
5.1 SETTINGS
Datasets: For IWSLT' 14 En-*De, following (Edunov et al., 2018), we split 7k sentences from the training corpus for validation, and the test set is the concatenation of (s12010, 1s/2011, 1s/2012, dev20/0 and dev2012. For IWSLT' En-+Vi, Vi, following (Ma et al., 2020), we use 1s12012 as the validation set and tst2013 as the test set. For IWSLT' 17 En ZZh, we concatenate tsr2013, tst2014 and tst2015 as the validation set and use tst2017 as the test set. For WMT' 15 En->e, following (Ma) et al. 2019 Arivazhagan et al., 2019), we use newstest20/3 as the validation set and use newstesf2015 as the test set. More details about datasets can be found at Appendix [B.1]
Models: The translation model f is based on Transformer. For IWSLT En-ZZh and En-Vi,, we use the transformer small model, where the embedding dimension, feed-forward layer dimension, number of layers are 512, 1024 and 6 respectively. For IWSLT De, we use the same architecture but change the embedding dimension into 256. For WMT 15 En- we use the transformer big setting, where the above three numbers are 1024, 4096 and 6 respectively. The controller y for each task is a multilayer perceptron (MLP) with one hidden layer and the tanh activation function. The size of the hidden layer is 256.
Input features of o: The input Ig.y is 7-dimension vector with the following features: (1) the ratios between the lengths of the source/target sentences to the average source/target sentence lengths in all training data (2 dimensions), i.e., La/ (Exexx Lay (|)) and Ly, Lyy/(Eye Ly /D/)) (2) the training loss over data (I, 3) evaluated by wait-k; (3) the average of historical training losses; (4) the validation loss of the previous epoch; (5) the average of historical validation loss; (6) the ratio of current training step to the total training iteration.
Choice of M2: The validation performance M is the negative validation loss with wait-k strategy. To stabilize the training, we minus a baseline to the R. in step 9 of algorithm I which is the validation loss of the previous episode, i.e., R.-1- 1. That is, the reward signal at episode is R. Re-1- 1. Ro the negative validation loss of the randomly initialized model.
Baselines: We implement the wait-k", Random and CL baselines discussed in Section 4.2 We also compare our algorithm with several adaptive methods, including Wait-if-Worss (WIW). Wait-if-Diff (WID), MILK, MMA-IL and MMA-H (refer to Section 2 for a brief introduction). As discussed in Section 4.2] we also compare and combine our method with /Zheng et al. (2020a)) We leave the training details of all algorithms (optimizer, hyperparameter selection, etc) in Appendix B.2 and the implementation details of baseline algorithms in Appendix B.3
5

---

Under review as a conference paper at ICLR 2021
Evaluation: We use BLEU to measure the translation quality, and use Average Proportion (AP) and Average Lagging (AL) to evaluate translation delay. AP measures the average proportion of source symbols required for translation, and and AL measures the average number of delayed words, which is complementary to AP (see Appendix A. 1 for details). Following the common practice (Ma et al. 2019 2020), show the BLEEU-P and BLEU-AL. curves to demonstrate the tradeoff between quality and latency. For IWSLT" 14 En-De and IWSLT' 15 En-+Vi, we use mult iillluu perl evaluate the BLEU scores; for IWSLT" 17 En En-+Zh and WMT' 15 En->ee, we use sacreBLEU evaluate the detokenized BLEU scores.
5.2
RESULTS
We first compare our method with the baseline methods on IWSLT datasets. The BLEU-latency curves are shown in Figure and the BLEU scores of En->Vi under different wait-k inference are reported in T The BLEU scores of all language pairs are left Appendix
270
16.
205
200
275
22
Wait-k"
Wait-k
27
CL
365
Random
360
16
25.
25
(a) BLEU-AP, En De.
(b) BLEU-AP, En-+Vi.
(c) BLEU-AP, En En-+Zh.
76.0
26
36.
26.
72
27.
27.0
30.
260
16
25.
(d) BLEU-AL, En De.
(e) BLEU-AL, En-VV
(1) BLEU-AL, En-+Zh.
Figure 2: Translation quality against latency metrics (AP and AL. on IWSLT'14 14 En De, IWSLT'15 En and IWSLT' 17 En-+Zh tasks.
Test k
wait-k
wait-k'/ k.'
CL
Random
Ours
25.14
26.14/5
26.01
26.12
27.03
27.17
28.25 /5
26.37
28.54
29.01
5
28.29
28.44/9
27.97
28.61
28.91
7
28.31
28.38 / 13
28.31
28.77
29.17
28.39
28.39
28.31
28.70
29.06
Table 1: BLEU scores on IWSLT En-+Vi simultaneous NMT tasks.
We have the following observations:
(I) Generally, our method consistently performed the best across different translation tasks in terms of both translation quality and controllable latency. As shown in Table our method achieves the highest BLEU scores among all baselines. We perform significance test on En and find that our method significantly outperforms wait-k (p 0.01 for wait-1.3 and 9; p < 0.05 for wair-5 and 7). In Figure 2 the curve for our method (i.e., the red one) is on the top in most cases, which indicates that given specific latency (e.g., AP AL), achieve the best translation quality.

---

Under review as a conference paper at ICLR 2021
(2) Baselines like wait-k", Random and CL also outperform the vanilla wait-k, which demon- strates the effectiveness of leveraging future information. However, the improvements are not consistent, and it is hard to tell which baseline is better. On En En->De, the performance of the three baselines is similar and CL slightly outperforms the other two. On En En-+Zh, wait-k" performs the best followed by CL which performs well at higher latency. In comparison, the improvement brought by our method is much more consistent.
(3) The improvement brought by our method more significant with smaller k's than that with bigger k's. We observe that all baselines perform well with bigger k, where more information is available during inference. That is, the advantages of leveraging future information are less significant. We provide analysis on the training and validation loss in Appendix ??, which shows that leveraging future information can improve generalization ability.
The results of WMT" 15 En->De, whose training corpus is larger, are shown in Figure 3 The heuristic methods (wait-k*, Random, CL) do bring much improvement compared wair-k. Our method consistently outperforms all baselines, which demonstrates the effectiveness of our method on large datasets. We further evaluate them on WMT' 14 and WMT" 16 test sets and obtain similar conclusions (see Figure 7
in Appendix C for details).
20
20
Wait-k Wait-k
22
22
CL
Random
Ours
0.60
0.65
0.70
0.75
0.80
3
6
AP
AL
(a) BLEU-AP
(b) BLEU-AL
Figure 3: Translation quality against latency metries (AP and AL) WMT 15 En->De.
We further compare our method with WIW, WID, MILK, MMA-IL and MMA-H on IWSLT Vi- The BLEU-AL curves are shown in Figure 4(a) and the BLEU-AP curves are in Appendix C When AL 2 5.0, method outperforms all baseline models, and when AL < 5.0, method performs slightly worse than MMA-IL and MMA-H, since ours does not take reducing latency into consideration explicitly but focuses improving performance under given waiting thresholds.
To verify the effectiveness of the adaptive extension of our method, we compare and combine our method with Zheng al. (2020a). where the wait-m models obtained through the vanilla wait-k (denoted by "Zheng et al.") and our strategy respectively (denoted by "+Ours". The BLEU-AL curves are shown in Figure 4b) and the BLEU-AP curves in Appendix C We can see that: (1) our method catches up with (Zheng et al., 2020a), which is built upon 10 models in total (wait- to wait-10); (2) after combing our approach with Zheng et al. (2020a). the performance can be further improved, which shows that our method is complementary to adaptive inference strategies like Zheng al. (020a)..
5.3
COMPUTATIONAL OVERHEAD
To evaluate the additional computational overhead brought by our method, we compare the training speed of wair-3 (measured by batch second) of standard wait-k and our method. Results on IWSLT datasets are summarized in Table Our method requires 20% ~ 30% additional training time, which is acceptable considering the improvements of performance. The main overhead is from computing the training loss by wait-k. To verify that, we record the training speed of our method without the second and third input features of (o, which are the training loss over data (I, y) evaluated by wait-k; and the average of historical training losses. Without them, the training speed of our method is similar to wait-k (i.e., the "Ours w/o feature (2,3)").
7

---

Under review as a conference paper at ICLR 2021
29
29
27
26
25
Wait-k
Ours
Ours
WIW
Zheng et al.
22
MMA-H
Zheng et al. ours
21
25
2.5
5.0
7.5
10.0
125
15.0
17.5
20.0
10
12
14
16
18
AL
AL
(a) Comparison of our method, WIW. WID,
(b) Comparison combination of our method and ZZeng et al. (2020a).
MILk, MMA-IL and MMA-H.
Figure BLEU-AL comparison between our method and baselines on En-
Task
wair-k
Ours
Ours w/o feature (2,3)
En mnn->Da De
5.3
4.0 (-23%)
5.2 (-2%)
En Vi
1.5
1.1 (-27%)
1.4 (-7%)
En +Zh
2.5
1.8 (-28%)
2.4 (-4%)
Table Comparison of training speed (batch / sec) between wair-k and our methods.
5.4 ANALYSIS
(I) Strategy analysis: In Figure 5
we visualize the distribution of wait-m training strategies obtained by algorithm for wait-3 and wait-9 inference IWSLT En-+Zh task. We show the frequency of each wail-m strategy sampled by the controller yp at the Oth, 1st, 5th, 10th and 40th episode.
0.2
0.4
3.7
0.0
Walt
1=10
2
k-13
Figure 5: An illustration of the strategies for wait-3 and wait-9 9 En- dataset.
We observed that the controller uniformly samples different m's at first, and then the strategies converge within 10 episodes. After convergence, the controller mainly samples several specific actions, i.c., m 5, 9, 12 for wail-3, and m 5, 12 for wait-9. The action that both controllers prefer most is Generally, these two strategies assign most of the sampling frequency to large m, which again shows the importance of using future information. However, it is worth noting that the controller also samples smaller m, which means that the past information also utilized. For example, the controller for wait-9 still samples 5 with probability about 0.02. Our conjecture is that the of past.
12, which is close the wait-k* strategy (k* = 11) for both wait-3 and wair-9.
8

---

Under review as a conference paper at ICLR 2021
information helps mitigate the mismatch between training and inference. If the model is always trained with future information, this mismatch will be large.
(II) Action space selection: In previous experiments, both future information and past information are leveraged. We want to study the effect of using past information or future information only. For any wait-k, we build two other action spaces for yp: K {1, ... A); 13;: 11, K).
We evaluate wait-3, 5 on IWSLT" 14 En->De with the above two action spaces.
The results reported in Table 3 We observe that method with full action space significantly outperforms that with (k) and slightly outperforms that using (k). This shows that leveraging both kinds of information help improve the performance.
k
3 AP
BLEU
AL
BLEU
AP
AL
Full action space
23.91
0.650
3.252
26.27
0.723
4.887
Kyy(k)
23.70
0.655
3.386
26.04
0.730
5.134
(k))
22.80
0.645
3.078
25.58
0.726
4.979
Table 3: Ablation study for feature selection on 14 En->De dataset.
(III) Feature selection: emphasize the importance of the selected features Section we provide four groups of ablation study, where in each group some specific features are excluded: (i) source and target sentence lengths; (ii) current training loss and average historical training loss; (iii) current validation loss and average historical validation loss; (iv) training step. We work on IWSLT' 14 En task and study the effect to wait-3, 7,
The results shown in Table We report the BLEU scores only, since the latency metrics (AP and AL) are not significantly influenced. Removing any feature causes performance drop, indicating that they all contribute to the decision making. Specifically, network status information including validation performance (feature iii) and training stage (feature iv) more important than input data information including sequence length (feature i) and data difficulty (feature ii).
k
k 5
k 7
Ours
23.91
26.27
27.19
-i)
23.67 (-1.00%, rank 3)
26.03 (-0.91%, rank 3)
26.92 (-0.99%, rank 4)
- (ii) 23.70 (-0.88% rank 4)
26.04 (-0.88%, rank 4)
26.91 (-1.03%, rank 3)
-iii)
23.57 (-1.42%, rank 1)
25.92 (-1.33%, rank 2)
26.72 (-1.73%, rank 1)
-(iv)
23.65 (-1.09%, rank 2)
25.63 (-2.44%, rank 1)
26.86 (-1.21%, rank 2)
Table 4: Ablation study for feature selection on IWSLT' 14 En-*De dataset.
CONCLUSION AND FUTURE WORK
In this work, we proposed new approach for simultaneous NMT. Motivated by the fact that wait-k benefits from future information, we introduced controller, which adaptively assigns a training task wait-m to each input. bi-level optimization method leveraged to jointly obtain the translation model and the controller. Experiments on four translation tasks demonstrate the effectiveness of our approach. For future work, first, we will enhance the objective function Eqn. (2) beyond using translation quality only and explicitly introduce the latency constraint. Second, we will combine our method with the adaptive decoding methods (Arivazhagan et al., 2019 Ma et al. 2020). Third, we will apply the idea in this work to more applications like action prediction (Kong et al. 2020 Cai et al. 2019) weather forecasting, game Al (Li eiet al. 2020 Vinyals et al. 2019). etc.
REFERENCES
Ashkan Alinejad, Maryam Siahbani, and Anoop Sarkar. Prediction improves simultancous neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural

---

Under review as a conference paper at ICLR 2021
Language Processing, pp.
Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.186533v1/D11- 1337. URL sestts://wwee aclweb #rt/annthology/D1B-133
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung- Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultaneous machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1313-1323, Florence, Italy, 2019. Association for Computational Linguistics. URL
https: // www. ac web www.ooog///aaaoo ogy /1t//119--- 1126
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2014. URL http:/ss///ccci..rr///b/aa/
0473
Yijun Cai, Haoxin Li, Jian Jian-Fang Hu, and Wei-Shi Zheng. Action knowledge transfer for action prediction with partial videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 81 18-8125, 2019.
Kyunghyun Cho and Masha Esipova. Can neural machine translation do simultaneous translation? CoRR, fess/600.02012, 2016. holLtttpp///xxi.oree wttg/bbb//60s.e//// 02012
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. Incremental decoding and training
methods for simultaneous translation in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 2 (Short Papers), pp. 493-499, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.186 2079. URL https
7/www. ac web. org/ antho I ogy/ N18-201800
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Mare' Aurelio Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 355-3664 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N188 1033. URL https
7/ 7/www. www.:aawwwwboggg.ee.....
wrg. http://ww.eeeeaattooooo///------
Alvin Grissom 11, He He, Jordan
John Morgan, and Hal Daumé III. Don't until the final verb wait: Reinforcement learning for simultaneous machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1342-1352, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-11 140.
URL https:: / a wwwwww.wbb..m ong. www.ebbozr/aanhooogo//o..o..
1140
Jiatao Gu, Graham Neubig, Kyunghyun Cho. and Victor O.K. Li. Learning to translate in real-
time with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 1053- 1062, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https / www. www.wb.lwbbcco org/ wwtp://....gaatttooooooeooe /erlooogyy//1199999
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL httph::://///. http://axxxx.xg//dr/// 6980. pdf
Y. Kong, Z. Tao, and Y. Fu. Adversarial action prediction networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(3):539-55 2020.
Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie-Yan Liu, and Hsiao-Wuee Hon. Suphx: Mastering mahjong with deep reinforcement learning. 2020.
arXiv preprint arXiv:2003.
Mingbo Ma, Liang Huang. Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He. Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to to-fix-t framework. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/199 1289. URL.ttts.: www. ac http:///wwwbbbb.rr///ttooolooo/p 91
10

---

Under review as a conference paper at ICLR 2021
Xutai Ma, Juan Pino, James Cross, Liezl Puzon, and Jiatao Gu. Monotonic multihead attention. In 8th International Conference on Learning Representations, 2020.
Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tic- Yan Liu. Simulspeech: End-to-end simultaneous speech text translation. In ACL, 2020.
A. Sinha, P. Malo, and K. Deb. A review on bilevel optimization: From classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22(2):276-2 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27. Curran Associates, Inc., 2014. URL htttp://papers.nttt http:/ppprrs.ips....p.pppppppssp www.nipppp..pper..//..ppp Learning with neural networss...
5346 to---saeeene-e-t--eeeeeeee
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in stareraft ii using multi-agent reinforcement learning. Nature, 575(7782):3300 350-354, 2019.
Hao Xiong, Ruiging Zhang, Chuanqiang Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. Dutongchuan: Context-aware translation model for simultancous interpreting. arXiv preprint arXiv:/907.1298% 2019.
Chen Zhang, Xu Tan, Jinglin Liu, Yi Ren, Tao Qin, and Tie- Yan Liu. Simuls2s: End-to-end simultaneous speech speech translation. Openreview, 2019, URL htttpss:/opeeeeiiiatit 56EYvS
net met/ppffidd-3
Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. Simpler and faster learning of adaptive
policies for simultaneous translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language
Processing (EMNLP-IJCNLP), pp. 1349-1354, Hong Kong, China, November 2019a. Association for Computational Linguistics. doi: 10. 10.18653/11/D19-11 1137. URL https: www. aclweb
httg/anthology/D... D
Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. Simultaneous translation with flexible policy via restricted imitation learning. In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5816-5822, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10. N..18653/11/199111 1582. URL https: / www. ac leeb
arg/anthology/PPe PPpp:/11911111111
Baigong Zheng, Kaibo Liu, Renjie Zheng, Mingbo Ma, Hairong Liu, and Liang Huang. Simultaneous translation policies: From fixed to adaptive. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2847-2853, 2020a. doi: 10.18653/v1/2020.
acl-main.254 URL httpt:t/////.. h/wp::://wwwww www acl web. orbb httphhhlllyy/2222a.acoa
254
Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, and Liang Huang. Opportunistic decoding with timely correction for simultaneous translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 437-442, Online, July 2020b. Association for Computational Linguistics. doi: 10. 1e.1886//vll 1/2020/.//222acl///// 12a.03aaam..mm..mmmmmma.mm URL https: / /////w/w www. . ac web
org/antho I ogy//020 acc 1awaaacaaaa. -ma -mmmnnn.... 42
A MATHEMATICAL DEFINITIONS
A.1 LATENCY METRICS DEFINITIONS
Given the input sentence x and the output sentence y. let Ly and L, denote the length of x and y respectively. Define function g(l) of decoding step i, which denotes the number of source
11

---

Under review as a conference paper at ICLR 2021
tokens processed by the encoder when deciding the target token yt. For wait-k strategy, g(t) min k - ) The definition of Average Proportion (AP) and Average Lagging (AL) are listed
in Equation 3 and Equation
v
(2,( = |alla
g(t);
(3)
(ael)
1
AL 1)
g(t)
(4)
To (1]))
1
where Ty (|r) min (l)(())(() |x/]].
We use the scripts provided by Ma et al. (2019) calculate AP and AL scores.
A.2 MATIEMATICCAL FORMULATION CURRICULUM LEARNING
In the curriculum learning (briefly, CL) baseline, we gradually decrease m from K to the threshold k: which will used in the test setting. The mathematical formulations are shown as follows:
= M M- (M k + 1)]
(5)
where T denotes the total update number, denotes the current update number (t 2,.,.... .......)))
T).
B MORE DETAILED SETTINGS ABOUT EXPERIMENTS
B.1 DETAILED INTRODUCTION OF THE DATASETS
For IWSLT' 14 En->e,, following Edunov et al. (2018), we lowercase all words, tokenize them and apply BPE with 10k merge operations jointly to the source and target sequences. We split 7k sentences from the training corpus for validation and the remaining 160k sequences are left as the training set. The test set is the concatenation of 1s/2010, Ist2011, 1st2012, dev2010 and dev2012, which consists of 6750 sentences.
For IWSLT' 15 En- Vi, following Ma et al. (220). tokenize the data and replace words with frequency less than by <unk We use 1s/20/2 as the validation set and ts/2013 as the test set. The training, validation and test contains 1331, 1268 and 1553 sentences respectively.
For IWSLT' 17 En >Zh, we tokenize the data and apply BPE with 10k merge operations independently the source and target sequence We the concatenation of tst2013, tst2014 and 5x/2015 the validation set and use Is/2017 as the test set. The training, validation and test sets contains 235k, 3874 and 1459 sentences respectively. For WMT' 15 En we follow the setting Ma et al. (2019); Arivazhagan et al. (2019). We tokenize the data, apply BPE with 32k merge operations jointly to the source and target sentences, and get training corpus with 4.5M sentences. We use newstes1201. as the validation set and use newstesf2015 as the test set.
B.2 DETAILED TRAINING STRATEGY
For the translation model, we use Adam (Kingma Ba 2015) optimizer with initial learning rate 5 x 1 and inverse sgrt scheduler (see Section 5.3 of Vaswani et al. 2017) for details). The batch size and the number of GPUs of IWSLT En En+De, En En-VVi, En En+
+Zh and WMT' 15 En-+ee are 4096 1GPU, 16000 1GPU, 4000 x 1GPU and 3584 x 16GPU respectively. For IWSLT tasks, the learning rate is grid searched from {5 x 10-4 5 x 10-5 5 x 10-6,_ x 10-7) with vanilla SGD optimizer, and the internal update iteration T is grid searched from (, 2f), where t is the number of updates in an epoch of the translation model training. For WMT En-Dee, due to resource limitation, do not train the translation model from scratch. The translation model is
started from pretrained wait-k model, the learning rate is set as 5 x 10-5, and the internal update iteration T 16.
'The is downloaded from fttphttpps///t...satttrrrrdoooor tokenized already.
which has been
"The Chinese sentences tokenized using Jieha
httpp:://github.comm/
12

---

Under review as a conference paper at ICLR 2021
B.3 DETAILED BASELINE IMPLEMENTATION
In this section, we introduce how we reproduce bascline models and get the results on En-+Vi task.
For MILk (Arivahhagan et al. 2019 and MMA (Ma et al. 2020), we do not reproduce the results, but directly use the results reported Ma al. 2020). Note that the results for MILk is reproduced by Ma et al. (2020).
For WIW and WID (Cho & Esipova, 2016), we pre-train standard translation model, using the exact same architecture and training hyperparameters as our method and in (Ma et al. 2020). For fair comparison, we adopt bi-directional attention.
For Zheng et al. (2020a), we use the pre-trained wait-k model and the model obtained through our results (k: = 1, 2,3, .... 10). We use only single model rather than ensemble. As in Zheng et al. (2020a), the thresholds of different k values are obtained in this way: Pi P1 (i- p1o)/9, where we test with P1 E {0.2, 0.4, 0.6, 0.8), Pi0 O; and f1 = 1, Pio E (0.002.004.06600088)
C SUPPLEMENTAL RESULTS
In this section, report the specific BLEU scores and some additional results. The BLEU scores for IWSLT tasks and WMT' 15 En->De task are reported in Table 53 and Table respectively We further evaluate the baselines and our methods on WMT' 14 and WMT' 18 test sets, and report the
BLEU-latency curves in Figure 7 The BLEU-AL curves of our methods and baselines IWSLT' 15
En are reported in Figure ((a) the BLEU-AL. curves of our method and ZZeng et (020aa) are
in Figure 6(b)
Task
wail-k
wait-k*/ best h**
CL
Random
Ours
En fnn+De De (k = 1)
16.75
19.11/9 /
18.23
18.53
19.10
En-DDe
22.79
23.36
23.41
23.50
23.91
En-tDe (I: 5)
25.34
25.76/11
25.88
25.84
26.27
En (k = 7)
26.74
26.87/9
26.85
26.88
27.19
En (k: 9)
27.25
27.54/11
27.48
27.07
27.68
En Vi (f: 1)
25.14
26.14/5
26.01
26.12
27.03
En-Vi Vi (k 3)
27.17
28.25/5
26.37
28.54
29.01
En (k 5)
28.29
28.44/9
27.97
28.61
28.91
En (k 7)
28.31
28.38 / 13
28.31
28.77
29.17
En En (k 9)
28.39
2
28.31
28.70
29.06
En ZZ (k = I)
14.24
19.34/9
17.26
18.02
19.21
En Zh (k 3)
19.90
21.66 11
21.64
21.36
22.18
En Zn
5)
21.45
23.57
23.62
22.59
23.70
En >Zh (K-
23.23
24.95/11
24.32
23.15
24.35
En >Zh (k- = 9)
23.93
24.83
24.55
23.55
24.78
Table 5: BLEU scores IWSLT simultaneous NMT tasks.
k
wait-k
wait-k*/ best h*
CL
Random
Ours
1 17.07
19.83 /
19.41
17.59
18. 18.14
22.86
23.14/7
22.51
22.76
23.58
5
25.52
26.09 /7
25.51
25.66
26.18
27.32
27.50/9
26.80
26.91
27.89
9
28.05
/9
28.20
27.82
28.42
Table 6:
BLEU scores on 15 En DDe dataset.
13

---

Under review as a conference paper at ICLR 2021
27
28
Walt-k Ours
Wait-k
MILK
23
Ours
WIW
Zheng et al.
22
MMAH
Zheng et +
21
25
3.6
0.7
0.8
6.9
1.0
0.65
0.70
0.75
0.80
0.55
050
0.55
160
AP
AP
(a) Comparison of method, WTW, WID,
(b) Comparison and combination of method and Zheng et al. (2020a).
MILK, MMA-IL and MMA-H.
Figure BLEU-AP comparison between our method and baselines En-
22
20
Wait-k
26
Wait-k
18
Random
16
Ours
0.60
0.95
0.70
0.75
0.80
0.00
0.65
0.70
0.75
0.80
AP
(a) BLEU-AP, WMT 14 En
(b) BLEU-AP, WMT' 16 En De
22
28
BLEU
26
20
18
16
AL
(c) BLEU-AL, WMT' 14 En
(d) BLEU-AL, WMT 16 En-DDe
Figure 7: Translation quality against latency metrics (AP and AL) on WMT' and 16 English -Germann test sets.
14

---

Under review as a conference paper at ICLR 2021
D ADDITIONAL ABLATIONS AND ANALYSIS
D.1 MODEL ARCHITECTURE SELECTION
As mentioned in Section 3 of the main content, we adopt unidirectional attention instead of bidi- rectional attention in the encoder side. We compare the performance the wait-k model with two attention types IWSLT' 14 En->De dataset, and the results are in Figure [[a] and Figure 8(b) On IWSLT' 14, we observe that the performance of waitk with unidirectional attention slightly drops than that with bidirectional attention. On 1MT" En-+ dataset, our implementation of wair-k with unidirectional attention is slightly better than that of bidirectional attention reported Ma et al. (2019). However, the computational cost of bidirectional attention is much larger than unidirectional attention. For example, the inference speed of unidirectional wait-9 model is 57.39 sentences / second, while the inference speed of bidirectional attention is 6.48 sentences / second.
28
28
24
Unidirectional attention
18-
Bidirectional attention
0.60
0.65
0.70
0.75
0.50
0.85
(a) BLEU-AP, IWSLT 14 En De
(b) BLEU-AL, IWSLT" 14 En
Figure 8: Ablation study of different model architectures on IWSLT' 14 En-*De dataset and WMT' 15
En-+De dataset.
D.2 CASE STUDY
To analyze the effect of using future information, we present two translation examples for En Eh wair-3 translation Table Table We observe that all methods tend to anticipate when the future information is lacking (Table 7). Wait-3 makes more mistake (Table 7 and even makes wrong anticipation where there is no need to anticipate (Table 8), while wait-k" and Ours anticipate more appropriately (Table 77. However, in Table wait-k" sometimes generates repeated information, therefore increasing the overall latency. This might be resulted from the gap between training and testing, as wait-k* is trained to produce higher latency. Our method can leverage the advantages of both methods, and produces translations with the best quality.
D.3 MORE HEURISTIC BASELINES
In this section, we conduct ablation studies on two more heuristic baselines to demonstrate the effectiveness of our method.
(1) Randomly selecting in a window around k* We implement another baseline which is combination of wait-k" and Random. After obtaining k* instead of sampling wait-m on all possible 21, ,M,, we sample on a smaller region around k. We conduct experiments on IWSLT En and the results are in Figure 9 We can that this variant achieves similar results with wait-k", but is not as good as our method. This shows the importance of using an adaptive controller to guide the training.
(2) A variant CL: We implement self self-paced learning (SPL) for wait-k, a CL method based on loss function: within each minibatch, remove the 7% sentences with the largest loss, where is gradually decreased from 40 to On IWSLT En DDe,, when & = 3, the BLEU/AL/AP for SPL are 02..8700666... which are worse than conventional CL (23.41/0.66/3.48))
15

---

Under review as a conference paper at ICLR 2021
2
4
5
8
10
11
12
13
14
15
16
I
born
with
opi 1e
xy
and
intellectual disubility
wart-3
@
Xn e
1 @
was horn
in
Full of
epilepsy
Not word
country
born
country full opilopsy Not word.
Wiit-k
03
was boen
when
suffered from
epilepsy
and intellectual disability
When I was born, sutfered from spilepsy and intellectual disubility.
Ours
Hi
W9 intellectual
RQ
was born when
with
epilepsy
and
disability
When I was born, was accompanicd by epilopsy and intellectual disability.
Table Example 1 for En- wait-3 translation. this example and the next example, different colors represent different meanings. Specifically, green and red represents information that does not exist in the source sentence (i.e., anticipated by the model), where green represents information that is consistent with the input sentence (i.e. correctly anticipated), and red represents information that is inconsistent with the input sentence (i.e., wrongly anticipated).
At step 5, Wair3 anticipates "ØF "Æ\~~" (in a), while wait-k" and Ours anticipates "#1 [ (when) and "Iif" (when) respectively. The anticipation generated by wait-k" and Ours are more appropriate within the context, while wait-3 makes mistakes.
7
13 staring right baek
17
opened
up
website
and
there
was
my face
at
me
wair-3
12
1438 f found
St BL
A1
3R
opened
website
face
opened
my
Wait-k"
FTH T
1435
IT# T
R
W
32
opened the website
opened the website
POS
face
PROG PROG looking
opened website, opened
my was looking
Ours
77 T
1435
M:
#t
ff
R2
F1
# PROG
opened
the website
then
there
POS
T openeed the website, and then there my face
Table 8: Example 2 for En-ZZh wait-3 translation, where POS indicates possessive forms, and PROG indicates progressive tense. In this example, there need anticipate. However, wair-3 still anticipates " "Il"" (found) and makes a mistake. Wait-k" makes a mistake by repeating "JT#F *t" (opened the website). Ours generates the best translation.
26
26
22
22
Wait-k
Wait-k
Ours
18
Random in window
(a) BLEU-AP, IWSLT' 14 En
(b) BLEU-AL, IWSLT" 14 En-+De
Figure 9: Randomly selecting in a window around k.
16

---

Under review as a conference paper at ICLR 2021
(3) An annealing strategy: Inspired by Figure we design baseline where randomly sample the waiting threshold m from a distribution (m(m) at each training step t. The distribution pe (m) linearly anneals from uniform distribution to a distribution which prefers larger We expect single annealing strategy can train reasonably good models for different inference-time k values. Suppose the minimal m value is minn the maximal value is maxx- minn and Mmax and m E mmn Miin + 1, mmax The total training step is denote T' (m) is mathematically defined as follows:
integers
(mm) (1 +) Pini (m)) + T
Plinal (m,)
172
Pinit (m)
Prinal (m)
Mlmmx 770mn
The results are shown in Figure 10 which shows this baseline brings limited improvement compared to wair-k. A possible reason is that this baseline cannot guarantee the best "annealing" strategy for each separate k, while our method can adaptively find the optimal strategy. Besides, shown in Figure the learned strategies for different wait-k inference are pretty different.
28.5
25.5
20.0
28.0
27.5
27.0
27.0
Wait-k
26.9
26.0
Ours
25.5
Annealing
25.5
(a) BLEU-AP, IWSLT' 14 DD
(b) BLEU-AL, IWSLT" 14 En-De
Figure 10: Randomly selecting le in a window around k-
17


Reviews:
[{"review_id": "Xzgy6V12UU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new training method for wait-k simultaneous translation. Rather than training on prefix pairs where the target prefix lags the source by k tokens, it uses an RL controller to determine an optimal lag for each sentence pair. The controller uses a small set of features intended to capture training progress, and is trained with REINFORCE to minimize wait-k loss on a validation set, in alternation with main training steps. This method shows consistent gains over various wait-k training heuristics, and some gains over other approaches that adapt the lag at inference time.\n\nThe paper is very well written and organized. The method makes sense, and the experiments are quite thorough, comparing to a competitive set of heuristic baselines, and showing credible - though fairly modest - gains in this setting.\n\nThe comparisons to adaptive baselines are less convincing. They are shown only for one small-data language pair, no implementation details are given (for instance, architectures and model capacities), and the relative results are very different from those in the literature (MILK << wait-k, WIW, WID). I think the paper would be stronger if these were simply omitted.\n\nGiven its ease of implementation and efficiency, I think there is room for a paper focusing on improving wait-k training, even if wait-k isn’t quite state of the art. But having to set up an RL controller detracts from this picture, especially since the gains over various heuristics - in particular the random heuristic that works for any inference-time k - aren’t spectacular. My main reaction to this work is that it should be possible to get most of the gains using some predetermined curriculum inspired by figure 5 (random sampling at the beginning, annealing to some m > k) that is effective for a broad range of inference-time k’s. Note that this is quite different from the CL baseline included in the results. As it stands, I fear this paper risks being a dead end: too complex to be worth implementing, as the field moves on past wait-k.\n\nQuestions and suggestions:\n\n1. Controller feature (6): will this value ever repeat during an entire training run?\n2. For Mk, since this runs only over the validation set and you are using RL, why not use the actual wait-k BLEU score?\n3. You need to include all heuristic baselines in figure 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "E0QnlM00xUq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors observed that some lookahead information during training time is helpful to improve the translation accuracy for simultaneous translation. Base on this observation, this paper proposes to use RL-based methods to learn a certain number of lookahead words during the training of the wait-k-based simultaneous translation model.\n\nThis paper proposes a new approach for improving the translation quality and the results indeed show some improvements over the baseline methods. However, I still have the following concerns:\n1) I think the proposed RL-based methods are very completed (in terms of hyperparameter searching, extra training time compared with other baselines) but the improvements are quite marginal compared with wait-k* and random.\n\n2) For the experiments, the authors did not compare with other agent-based or adaptive methods. \n\n3) This paper only designs a controller for training. I think we could also have a controller for the inference. In this way, I believe we could use regular wait-k training, and use a controller to decide a smaller k during inference. Then it will be very similar to Gu. et al's RL-based methods. I suggest the authors include this method's experiments as well.\n\n4) If I did not misunderstand, the m of wait-m is defined on each training pair, but I think this will dramatically increase the training time and hard to do batch training. How slow is your training compared with baseline wait-k? I believe the baseline wait-k is already very slow.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BXmSWQDi7qu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper improves the wait-k based simultaneous NMT by training on an adaptive wait-m policy. The proposed method and experiments are clearly described. Experiments demonstrate that the proposed method is significant better than the wait-k baseline.\n\nHowever, compared to heuristic-based baselines, seemingly the proposed method is not significantly better in most cases, especially on the WMT En-De dataset. Given that the WMT dataset is much larger than IWSLT datasets, does this suggest that the proposed method may not work well on larger dataset?\n\nI’m also curious about the reason of using adaptive wait-m only during training since it is a more straightforward idea to apply adaptive policy during both training and inference. Would it be better if the adaptive policy (by re-design features) was used on inference as well? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3pHCV0Rw_Z2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a training strategy for simultaneous translation to choose appropriate amount of look-ahead information for each decoding. Based on the observation that the wait-k method can be improved by training with longer information, the method introduces a function to determine its length given the current example (source x, target y, and the translation model f). It is used as only a guidance during training, and it remains the same decoding criterion at inference.\n\nThe method is interesting and has promising improvements compared with bare wait-k methods according to the experiments, but the paper seems to have some major questions which should affect the conclusion. I recommend to revise the paper appropriately, especially to resolve following concerns:\n\n- The proposed method is intuitively strange because of mismatching between training/inference strategies. It is also unclear to choose this method rather than adaptive wait-k methods, e.g., one referred as Zheng et al. (2020a), which may solve a similar problem directly. The paper needs at least some comparison of these kind of methods to figure out the advantages of the proposed method.\n- Algorithm 1 involves a suspicious use of the development set: it is used directly to optimize a parameter. Specifically, since \\omega is optimized using the D_va, it brings information of the D_va into f, resulting that the training process does not include any strategy to avoid overfitting (in other words, your training set is actually D_tr + D_va, and there is no so-called development set).\n\nMinor comments:\n\n- The title sounds misleading: the proposed method still does not learn how to use future information because it uses only k look-ahead information at inference (same as usual wait-k methods), i.e., there is nothing special to represent \"future\".\n- Figure 1 involves some common mistakes of using bar charts: it must use 0 as the origin, and must not shorten the bar. If you want to focus on differences between each value, you should use other chart.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper improves the wait-k based simultaneous NMT by training on an adaptive wait-m policy with a controller determining the lag for sentence pair.  The controller is trained with RL to minimize the loss on a validation set. The overall model is reasonable, which is well presented. I however have the following two concerns
1. There is a clear mismatch between training/inference strategies, which raises two problems
    1. The motivation:  the authors tried to explain that in discussion,  but it is not convincing enough
    2. The title is misleading since there is no future information to use during inference 
2. The experiments is not convincing enough in that a) the improvement over baseline is modest, and b) comparison to adaptive wait-k and other strong baseline is insufficient 

In conclusion I would suggest to reject this paper.


Decision: Reject