Monotonic Multihead Attention
Authors: Xutai Ma
Juan Miguel Pino
James Cross
Liezl Puzon
Jiatao Gu
Venue:  (2020)
Keywords: Simultaneous Translation
Transformer
Monotonic Attention

Abstract: Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approach for this task either apply a fixed policy on transformer, or a learnable monotonic attention on a weaker recurrent neural network based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which introduced the monotonic attention mechanism to multihead attention. We also introduced two novel interpretable approaches for latency control that are specifically designed for multiple attentions. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach.


Published as a conference paper at ICLR 2020
MONOTONIC MULTIHEAD ATTENTION
Xutai Ma² Juan Pino', James James Cross', Liezl Puzon', Jiatao Gu
Facebook
Johns Hopkins University
xutai_ma jhu edu, puzonêcs stanford. (juancarabina, jeross, igu fb. com
ABSTRACT
Simultaneous machine translation models generating target sequence be- fore they have encoded the source sequence. Recent approaches for this task ei- ther apply a fixed policy on a state-of-the Transformer model, or learnable monotonic attention on a weaker recurrent neural network-based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead atten- tion. We also introduce two novel and interpretable approaches for latency con- trol that are specifically designed for multiple attention heads. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeofffs compared to MILk. the previous state-of-the-ar approach. We analyze how the latency controls affect the attention span and we study the relationship between the speed of a head and the layer it belongs to. Finally, motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency
INTRODUCTION
Simultancous machine translation adds the capability of a live interpreter to machine translation: simultaneous model starts generating translation before it has finished reading the entire source sentence. Such models are useful in any situation where translation needs to be done in real time. For example, simultaneous models translate live video captions facilitate conversations between people speaking different languages. In a usual translation model, the encoder first reads the entire sentence, then the decoder writes the target sentence. On the other hand, simultaneous neural machine translation model alternates between reading the input and writing the output using either a fixed or learned policy.
Monotonic attention mechanisms fall into the flexible policy category, in which the policies are au- tomatically learned from data. Recent work exploring monotonic attention variants for simultancous translation include: hard monotonic attention (Raffel et al., 2017), monotonic chunkwise attention (MoChA) (Chiu & Raffel, 2018) and monotonic infinite lookback attention (MILk) (Arivazhagan al. 2019). MILk in particular has shown better quality/latency trade-offs than fixed policy ap- proaches, such as wait-k (Ma et al., 2019) or wait-if-* (Cho & Esipova 2016) policies. MILK also outperforms hard monotonic attention and MoChA; while the other two monotonic attention mech- anisms only consider a fixed window, MILk computes a softmax attention over all previous encoder states, which may be the key to its improved latency-quality tradeoffs. These monotonic attention approaches also provide a closed-form expression for the expected alignment between source and target tokens.
However, monotonic attention-bart models, including the state-of-the-arr MILk, were built on top of RNN-based models. LNNN-bsee models have been outperformed by the recent state-of-the-art Transformer model (Vaswani et al. 2017), which features multiple encoder-decoder attention layers and multihead attention at each layer.
Work conducted during an internship at Facebook
'The code is available https: / thub. httpbbb.com/ttttcc//ftt.t.ce http://///.ccrraaaaaaaeeeeeeeeee exampl mu wtw.://mmmaaoouusutrrooo.... Laaeoos

---

Published as a conference paper at ICLR 2020
We thus propose monotonic multihead attention (MMA), which combines the high translation qual- ity from multilayer multihead attention and low latency from monotonic attention. We propose two variants, Hard MMA (MMA-H) and Infinite Lookback MMA (MMA-IL). MMA-H is designed with streaming systems in mind where the attention span must be limited. MMA-II. emphasizes the qual- ity of the translation system. We also propose two novel lateney regularization methods. The first encourages the model to be faster by directly minimizing the average latency. The second encour- ages the attention heads maintain similar positions, preventing the latency from being dominated by single few heads.
The main contributions of this paper are: (1) A novel monotonic attention mechanism, monotonic multihead attention, which enables the Transformer model to perform online decoding. This model leverages the power of the Transformer and the efficiency of monotonic attention.
(2) Better la- tency/quality tradeoffs compared to the MILk model, the previous oftte-of- on two standard translation benchmarks, IWSLT15 English-Viemaeess
Vietnamese (En-Vi) and WMT15 German- Englishh (De- En). (3) Analyses on how model is able to control the attention span and the relationship between the speed head and the layer belongs to. We motivate the design of our model with an ablation study the number of decoder layers and the number of decoder heads.
2 MONOTONIC MULTIHEAD ATTENTION MODEL
In this section, we review the monotonic attention-based approaches in RNN-based encoder-decoder models. We then introduce the two types of Monotonic Multihead Attention (MMA) for Transformer models: MMA-H and MMA-IL. Finally, we introduce strategies to control latency and coverage.
2.1 MONOTONIC ATTENTION
The hard monotonic attention mechanism (Raffel et al. 2017) was first introduced in order to achieve online linear time decoding for RNN-based encoder-decoder models. We denote the input sequence as x = {zi .T), and the corresponding encoder states as m = {m²,, mi)) with T being the length of the source sequence. The model generates a target sequence y , yo with being the length of the target sequence. At the i-th decoding step, the decoder only attends to encoder state my, with t; j. When generating new target token yr, the decoder chooses whether to move one step forward or to stay at the current position based on a Bernoulli selection probability Pi,j- that t; 2 44-1. Denoting the decoder state at the i-th position, starting from j f--1t;-11 1, + 2, ..., this process be calculated follows:
Ei MonotonicEnergyy:::
(1)
Più Sigmoid (ex,)
(2)
Zi.j Bernoulli(ps.))
When æ5,j 1, we set j and start generating target token yı; otherwise, we set t; = -j+ 1 and repeat the process. During training, an expected alignment introduced to replace the softmax attention. It can be calculated in recurrent manner, shown in Equation 4]
II 1=k
(1 pr.))
0i = Pi.
04-1k
(4)
Oiij-1 + 03-1,5
= Pi.j
(1-
Pi.j-
Pi.j-1
Raffel et al. (2017) also introduce closed-form parallel solution for the recurrence relation in Equation 5
0i-1: 04 = Ps. cumprocd1 - Pt.)) cumsum cumprod(1 - -P..)
(5)
where cumprod(x) [1, .1, 22 In practice, the denominator Equation 5 is clamped into a range of [c, 1] to avoid numerical in- stabilities introduced by cumprod. Although this monotonic attention mechanism achieves online
I [ æ.] and cumsum(z) [x1, 21 I22 Σ E 2.}. [=
Note that during training, to encourage discreteness, Raffel et al. 2017) added zero mean, unit variance pre-sigmoid noise Ej
2

---

Published as a conference paper at ICLR 2020
linear time decoding, the decoder only attend to encoder state. This limitation can diminish translation quality as there may be insufficient information for reordering.
Moreover, the model lacks mechanism to adjust latency based different requirements at decod- ing time. To address these issues, Chiu & Raffel (2018) introduce Monotonic Chunkwise Attention (MoChA), which allows the decoder to apply softmax attention to fixed-length subsequence of encoder states, Alternatively, AAriazhagan et al. (2019) introduce Monotonic Infinite Lookback At- tention (MILK) which allows the decoder access encoder states from the beginning of the source sequence. The expected attention for the MILk model is defined in Equation
Oii exp(ui.))
(6)
Ei= exp(ur.)
2.2 MONOTONIC MULTIHEAD ATTENTION
Previous monotonic attention approaches are based on RNN encoder-decoder models with a single attention and haven't explored the power of the Transformer model. The Transformer architec- ture Vaswani et al., 2017) has recently become the state-of-the-ar for machine translation (Barrault al. 2019). An important feature of the Transformer the of separate multihead attention module at each layer. Thus, we propose a new approach, Monotonic Multihead Attention (MMA), which combines the expressive power of multihead attention and the low latency of monotonic at- tention.
Multihead attention allows each decoder layer have multiple heads, where each head com- pute different attention distribution. Given queries Q. keys K and values V, multihead attention MultiHead K, V) defined in Equation 7
MultiHead Q, K, V) = Concat(headd.... head, W
where head,, = Attention (QW?. KW VWN.
The attention function is the scaled dot- not-product attention, defined in Equation 8
QKT
Attention(Q. V) = Softmax
(8)
vd
There are three applications multihead attention the Transformer model:
1. The Encoder contains self-attention layers where all of the queries, keys and values come from previous layers.
2. The Decoder contains self-attention layers that allow each position in the decoder attend to all positions in the decoder up to and including that position.
3. The Encoder-Decoder attention contains multihead attention layers where queries come from the previous decoder layer and the keys and values come from the output of the en- coder. Every decoder layer has separate encoder-decoder attention.
For MMA, we assign each head to operate as a separate monotonic attention in encoder-decoder attention.
For transformer with decoder layers and H attention heads per layer, we define the selection process of the h-th head eecoder-decoder attention in the 1-th decoder layer as
my W.K (* W.. )r
Vdu
1.
Sigmoid(ez.)
(10)
J.
Bernoulli(pa.)
(11)
*MILK was based on a strengthened RNN based model called RNMT+ The original RNMT+ model (Chen
al. 2018) uses multihead attention, computes attention only once, and then concatenates that single attention
layer to the output of each decoder layer block. However, the RNMT+ model used for MILk in Arivazhagan
2019) only single head.

---

Published as a conference paper at ICLR 2020
Figure 1: Monotonic Attention (Left) versus Monotonic Multihead Attention (Right).
where Wi, is the input projection matrix, dg the dimension of the attention head. We make the selection process independent for each head in each layer. We then investigate two types of MMA, M(M-H((ar)) and MMA-IL(infiiite lookback). For MMA-H, we use Equation 4 in order to
L.k. For MMA-II, we calculate calculate the expected alignment for each layer each head, given Pag- the softmax energy for each head follows:
(m= ( 1,
my W.K (8i
SoftEnergy
(12)
ddz
1
and then use Equation 6 to calculate the expected attention. Each attention head in MMA-H hard- attends to one encoder state. On the other hand, each attention head in MMA-IL can attend to all previous encoder states. Thus, MMA-IL allows the model to leverage more information for transla- tion, but MMA-H may be better suited for streaming systems with stricter efficiency requirements. Finally, our models unidirectional encoders: the encoder self-attention can only attend to previ- ous states, which is also required for simultaneous translation.
At inference time, our decoding strategy is shown in Algorithm 1 For each at decoding step we apply the sampling processes discussed in [subsection 2. indiviiuull and set the encoder step at 2 dh
Then hard alignment or partial softmax attention from encoder states, shown in Equation 13 will be retrieved feed into the decoder to generate the i-th token. The model will write new target token only after all the attentions have decided to write. In other words, the heads that have decided write must wait until the others have finished reading.
(oncat(c()) d.1 ,6, b.... Cj
marh dhh
MMA-H
where d.a Scontexa (h, (/
exp
1ike U
(13)
MMA-IL
==1 exp
Figure illustrates a comparison between our model and the monotonic model with one attention head. Compared with the monotonic model, the MMA model is able set attention to different positions so that it can still attend to previous states while reading each new token. Each head can adjust its speed on-the-fly. Some heads read new inputs, while the others can stay in the past to retain the source history information. Even with the hard aligmment variant (MMA-H), the model still able to preserve the history information by setting heads past states. In contrast, the hard monotonic model, which only has one head, loses the previous information at the attention layer.
2.3 LATENCY CONTROL
Effective simultaneous machine translation must balance quality and latency. At a high level, latency measures how many source tokens the model has read until a translation generated. The model we have introduced in subsectioo 2.2 is not able to control latency on its own. While MMA allows simultaneous translation by having a read or write schedule for each head, the overall latency is determined by the fastest head, i.e. the head that reads the most. It is possible that head always reads new input without producing output, which would result in the maximum possible latency. Note that the attention behaviors in MMA-H and MMA-IL can be different. In MMA-IL, head reaching the end the sentence will provide the model with maximum information about the source sentence. On the other hand, in the case of MMA-H, reaching the end of sentence for head only
4

---

Published as a conference paper at ICLR 2020
Algorithm MMA monotonic decoding. Because each head is independent, we compute line 3 to 16 parallel
Input: source tokens, h encoder states, i 1, 1, td 1, StartOfSequence
1: while 1/-1 o EndOrSequence do
Lmax
3: h = -empty sequence
4: for + 1 to L do
5:
for h to H do
for :--1 |a]
Pi. Sigmoid (MonooonicEneergy(ss-.....
P > 0.5 then
9:
h
10:
fountea (h, (.
11:
Break
12:
else
13:
if
> Leax then Read token 2;
15:
Calculate state by and append to h
16:
fan = j c = Concat(c, (1 c
17:
L.H. = DecoderLayer ( (s1 11)1
18:
19:
(st) i
20:
gives a hard alignment to the end-of-sentence token, which provides very little information to the decoder. Furthermore, it is possible that MMA-H attention head stays at the beginning of sentence without moving forward. Such a head would not cause latency issues but would degrade the model quality since the decoder would not have any information about the input. In addition, this behavior is not suited for streaming systems.
To address these issues, we introduce two latency control methods. The first one is weighted average latency, shown in Equation 14
exp(g 1= exp( 1,
9,
J
(14)
where
jasj. Jay Then we calculate the latency loss with a differentiable latency metric C.
Lavg Cgg
(15)
Like Arivazhagan al. 2019), we use the Differentiable Average Lagging. It is important note [Equai,nnuaioon
that, unlike the original latency augmented training in AAivazhagan et al. (2019), |Equatin 15 is not expected latency metric given but weighted average on all attentions. The real expected latency is max1,a (g instead of but using this directly would only affect the speed of the fastest head. |Equation 15 control every head way that the faster heads will automatically assigned to larger weights and slower heads will also be moderately regularized. For MMA-H models, we found that the latency of are mainly due to outliers that skip almost every token. The weighted average latency loss is not sufficient control the outliers. We therefore introduce the head divergence loss, the average variance of expected delays at each step, defined Equation 16)
H
Lvar
LH
g
91
(16)
I=1
h=1
where g. = LH Eg. go The final objective function is presented in Equation 17
L(#)= = log(3
x: 0) + Xavy Avar Lear
(17)
where Aavgs Avar are hyperparameters that control both losses. Intuitively, while Aavy controls the overall speed, Avar controls the divergence of the heads. Combining these two losses, are able to dynamically control the range attention heads so that we can control the latency and the reading buffer. For MMA-IL model, only use Teagi for MMA-H only use Tavar-
5

---

Published as a conference paper at ICLR 2020
3 EXPERIMENTAL SETUP
3.1 EVALUATION METRICS
We evaluate our model using quality and latency. For translation quality, tokenized BLEU for IWSLTI5 En-Vi and detokenized BLEU with SacreBLEU (Post) 2018) for WMTI5 De-En, For latency, we use three different recent metries, Average Proportion (AP) (Cho & Esipova 2016),
Average Lagging (AL) (Ma et al., 2019) and Differentiable Average Lagging (DAL)(())))) (Arivazhagan We remind the reader of the metric definitions in Appendix A.2
al. 2019
3.2 DATASETS
Dataset
Train
Validation
Test
IWSLT15 En-Vi
133k
1268
1553
WMT15 De-En
4.5M
3000
2169
Table 1: Number of sentences in each split.
Dataset
RNN
Transformer
IWSLT1S En-Vi
25.6
28.7
WMT15 De-En 28.4 (Arivazhagan al. 2019
32.3
Table 2: Offline model performance with unidirectional encoder and greedy decoding.
Dataset
Beam Search
Bidirectional Encoder
Unidirectional Encoder
32.6
32.3
WMT15 De-En
4
33.0
33.0
28.7
29.4
IWSLT15 En-Vi
28.8
29.5
Table 3: Effect of using unidirectional encoder and greedy decoding to BLEU score.
We evaluate our method on two standard machine translation datasets, IWSLT14 En- Vi and WMT15 De-En. Statistics of the datasets can be found Table I] For each dataset, we apply tokenization with the Moses (Koehn et al.) 2007) tokenizer and preserve casing.
IWSLT15 Diglish-Vietnamest TED talks from IWSLT 2015 Evaluation Campaign (Cettolo al., 2016). We follow the settings from Luong & Manning 2015) and Raffel et al. (2017). We replace words with frequency less than 5 by <unk>> We use 1st2012 validation set tst20T3 as a test set.
WMT15 German-English We follow the setting from Arivazhagan al. 2019). We apply byte pair encoding (BPE) (Sennrich et al.] 2016) jointly on the source and target to construct a shared vocabulary with 32K symbols. We use newstest2013 validation set and newstest2015 as test set.
3.3 MODELS
We evaluate MMA-H and MMA-IL models on both datasets. The MILK model we evaluate on IWSLTI5 En Vi based on Luong al. (2015) rather than RNMT+ (Chen et 2018). In general, our offline models use unidirectional encoders, i.e. the encoder self-attention can only attend to previous states, and greedy decoding. We report offline model performance in Table and the effect of using unidirectional encoders and greedy decoding in Table 3 For MMA models, the encoder-decoder layers with MMA and keep other hyperparameter settings the same as the offline model. Detailed hyperparameter settings can be found in subsection A.1 We use the Fairseq library (Ott et al. 2019) for our implementation.
replace
4 acquire data from htts::///p..a http/foorr..ttooord....o. www.fooo...u/pooocce..o..cocoooe edu/projeccs//mm// which is tokenized. not have tokenizer which processed this data, thus we report tokenize d BLEU for IWSLT15
stanford.
'Latency metrics computed on BPE tokens for WMT15 De-Fn consistent with Arivazhagan al.
2019) and on word tokens for TWSLT15 En-Vi.
Luong & Manning (2015)) report BI.UU score of 23.0 but they didn't 't mention what type of BLEUU score
they used. This score from our implementation the data aquired from

---

Published as a conference paper at ICLR 2020
4 RESULTS
In this section, we present the main results of our model in terms of latency-quality tradeoffs, abla- tion studies and analyses. In the first study, we analyze the effect of the variance loss on the attention span. Then, we study the effect of the number of decoder layers and decoder heads on quality and latency. We also provide case study for the behavior of attention heads in an example. Finally, we study the relationship between the rank of an attention head and the layer belongs to.
IWSLT15 English- Vietnamese
30
30
30
28
28
28
BLEU
26
26
26
24
24
24
0.6
0.7
0.8
2
8
10
2
4
6
10
WMT15 German-Englisn
32
32
32
30
30
30
28
28
28
MILL MMA-II
26
26
26
24
24
24
0.5
0.6
0.7
0.8
2
4
10
12
10
15
Average Proportion
Average Lagging
Differentiable Average Lagging
Figure 2: Latency-quality tradeoff's for MILK (Arivazhagan et al. 2019) and MMA on IWSLT15 En-V Vi and WMT15 De-En. Black dashed line indicates the unidirectional offline transformer model with greedy search.
4.1 LATENCY-QUALIE TRADEOFFS
We plot the quality-latency curves for MMA-H and MMA-IL in FFiuure The BLEU and latency scores on the test sets are generated by setting a latency range and selecting the checkpoint with best BLEU score the validation set. We use differentiable average lagging (Arivazhagan al., 2019) when setting the latency range. We find that for given latency. our models obtain better translation quality. While MMA-IL tends to have decrease quality as the latency decreases, MMA-H has small gain in quality as latency decreases: a larger latency does not necessarily mean an increase in source information available to the model. In fact, the large latency from the outlier attention heads, which skip the entire source sentence and point to the end of the sentence. The outliers not only increase the latency but they also do not provide useful information. We introduce the attention variance loss to eliminate the outliers, such loss makes the attention heads focus on the current context for translating the new target token.
It is interesting observe that MMA-H H has better latency-quality tradcoff than MILL even though each head only attends to only one state. Although MMA-H is not yet able to handle an arbitrarily long input (without resorting segmenting the input), since both encoder and decoder self-attention have an infinite lookback, that model represents good step in that direction.
4.2 ATTENTION SPAN
In subsection 2.3] we introduced the attention variance loss to MMA-H in order to prevent out- lier attention heads from increasing the latency increasing the attention span. We have already evaluated the effectiveness of this method on latency in (suseectoon 4.1 We also want to measure the difference between the fastest and slowest heads each decoding step. We define the average
Thee numbers of MILk WMT15 De-En from Arivazhagan al. 2019

---

Published as a conference paper at ICLR 2020
attention span in Equation 18
S y
Σ max 1.
min 2.b 1,b
(18)
estimates the reading buffer we need for streaming translation. We show the relation between the As expected, the average attention span is reduced
average attention span versus Avar in Figure 3 as we increase Avor-
IWSLT15 En-
WMT15 De-En
20
20
15
15
in
10
10
5
0.2
0.4
0.6
0.8
1
0
0.5
1
1.5
Aver
Avar
Figure Effect of Avar on the average attention span. The variance loss works as intended by reducing the span with higher weights.
4.3 EFFECT ON NUMBER OF LAYERS AND NUMBER OF HEADS
One motivation to introduce MMA to adapt the Transformer, which the current state-of-the-ar model for machine translation, to online decoding. Important features of the Transformer architec- ture include having a separate attention layer for each decoder layer block and multihead attention. In this section, test the effect of these two components the offline, MMA-H, and MMA-IL models from a quality and latency perspective. We report quality as measured by detokenized BLEU and latency measured by DAL on the WMT13 validation in Figure 4 We set Xang 0.2 for MMA-II. and Apar = 0.2 for MMA-H.
The offline model benefits from having more than decoder layer. In the case of decoder layer, increasing the number of attention heads is beneficial but in the case of and decoder layers, we do see much benefit from using more than 2 heads. The best performance obtained for 3 layers and 2 heads (6 effective heads). The MMA-IL model behaves similarly to the offline model, and the best performance observed with 6 layers and heads (24 effective heads). For MMA-H, with layer, performance improves with more heads. With 3 layers, the single-head setting is the most effective (3 effective heads). Finally, with 6 layers, the best performance reached with 16 heads (96 effective heads).
The general trend we observe is that performance improves as we increase the number of effective heads, either from multiple layers multihead attention, up to certain point, then either plateaus or degrades. This motivates the introduction the MMA model.
We also note that latency increases with the number of effective attention heads. This is due to having fixed loss weights: when more heads are involved, we should increase Avar or Aang to better control latency.
4.4 ATTENTION BEHAVIORS
We characterize attention behaviors by providing a running example of MMA-H and MMA-II, shown in Figure 5) Each curve represents the path that an attention head goes through at inference time. For MMA-H, shown in Figure 5a we found that when the source and target tokens have the same order, the attention heads behave linearly and the distance between fastest head and slowest head small. For example, this can be observed from partial sentence pair "I also didn't 't know that" and target tokens "Töi cüng không biét rång", which have the same order. However, when the source tokens and target tokens have different orders, such as "the second step" and "buôe (step) thú hai (second)", the model will generate "butec (step)" first and some heads will stay in the past to retain
8

---

Published as a conference paper at ICLR 2020
Offline Model
MMA-H
MMA-H (latency)
32
28
10
31
BLBU
26
DAL
6
30
decader
decoder
decader
29
dlcodder layers dacosder layor
24
decederr
2
decnnbrr toyms
decedkrr
daconder Layes
28
12
4
16
12
8
16
12
4
16
# heads
MMA-II
MMA-IL (latency)
30
BLEU
DAL
2 1
28
decode:
decodees
27
12
4
8
16
12
4
16
Figure Effect the number of decoder attention heads the number of decoder attention layers on quality and latency, reported on the WMT13 validation set.
the information for later reordered translation "thú hai (second)"- We also that the attention heads have a near-diagonal trajectory, which is appropriate for streaming inputs.
The behavior of the heads in MMA-IL models is shown in Figure 5b Notice that we remove the partial softmax alignment in this figure. We don't It expect streaming capability for MMA-IL: some heads stop at early position of the source sentence retain the history information. Moreover, because MMA-IL has more information when generating a new target token, it tends to produce translations with better quality. In this example, the MMA-II. model has a better translation on "isolate the victim" than MMA-H ("là cô låp nan nhân" vs "là tách biỘt nan nhån")
140
adag
<rc
41
(a) MMA-H, Trar 1.0
(b) MMA-II, Laeg 0.2
Figure S: Running examples on IWSLTIS English-Vietnanne Vietnamese datasel
4.5 RANK OF THE HEADS
calculate the average and standard deviation of rank of each head when generating every target token. For MMA-IL, we find that heads lower layers tend to have higher rank and thus slower. However, in MMA-H, the difference of the average rank smaller. Furthermore, the standard deviation is very large which means that the order of the heads in MMA-H changes frequently the inference process.
In Figure 6

---

Published as a conference paper at ICLR 2020
13H
190
131
185
1961
1991
134
1310
1311
1344
1212
11
25
21
36
31
Fonk
Kank
(a) MMA-H, Lvar 1.0
(b) MMA-IL, Long 0.2
Figure 6: The average rank of attention heads during inference on IWSLT15 En-Vi. Error bars indicate the standard deviation. L indicates the layer number and H indicates the head number.
RELATED WORK
Recent work on simultancous machine translation falls into three categories. In the first one, models use a rule-based policy for reading input and writing output. Cho & Esipova (2016) propose a Wait- If_* policy enable offline model to decode simultaneously. Ma et al. (2019) propose wait-k: policy where the model first reads A tokens, then alternates between read and write actions. Dalvi al. 2018) propose an incremental decoding method, also based on rule-based schedule. In the second category, a flexible policy is learnt from data. Grissom II et al. (2014) introduce a Markov chain to phrase- based machine translation models for simultaneous machine translation, in which they apply reinforcement learning to learn the read-write policy based on states. Gu et al. (2017) introduce an agent which learns make decisions on when translate from the interaction with pre-trained offline neural machine translation model. Luo et al. (2017) used continuous rewards policy gradient for online alignments for speech recognition. Lawson et al. (2018) proposed a hard alignment with variational inference for online decoding. Allinejad
al. (2018) propose new
operation "predict" which predicts future source tokens. Zheng et al. (2019) 9b)) introduce a restricted dynamic oracle and restricted imitation learning for simultaneous translation. Zheng al. (2019a) train the agent with an action sequence from labels that are generated based on the rank of the gold target word given partial input. Models from the last category leverage monotonic attention and replace the softmax attention with an expected attention calculated from a stepwise Bernoulli selection probability. Raffel et al. (017) first introduce the concept of monotonic attention for online linear time decoding, where the attention only attends to one encoder state at a time. CCiu & Raffel (0188 extended that work let the model attend to a chunk of encoder state, AArivzzhagan et al. 2019) also make use of the monotonic attention but introduce an infinite lookback to improve the translation quality.
CONCLUSION
In this paper, we propose two variants of the monotonic multihead attention model for simultaneous machine translation. By introducing two new targeted loss terms which allow us to control both latency and attention span, we are able to leverage the power of the Transformer architecture to achieve better quality-latency trade-offs than the previous state-of-the-art model. We also present detailed ablation studies demonstrating the efficacy and rationale of our approach. By introducing these stronger simultancous sequence-to-ssequeee models, hope to facilitate important applica- tions, such as high-quality real-time interpretation between human speakers.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Ashkan Alinejad, Maryam Siahbani, and Anoop Sarkar. Prediction improves simultaneous neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3022-3027, 2018.
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung- Chenng Chiu, Semih Yavuz, Ruom- ing Pang. Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultaneous machine translation. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguisties, pp. 1313-1323, Florence, Italy, July 2019. Association for Computational htwaanhhhlooyyyppp
Linguistics. URL https: www. aaw.lllwwee......
Loic Barrault, Ondfej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 con- ference on machine translation (WMT19). In Proceedings of the Fourth Conference on Ma- chine Translation (Volume 2: Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, Au- gust 2019. Association for Computational Linguistics. doi: 10. 10.18655/11/W11-53301. URL https:// http//www..rr acl web. org hat/tontoologs /1 0000///19-53371
Mauro Cettolo, Niehues Jan, Stüker Sebastian, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. The iwslt 2016 evaluation campaign. In International Workshop on Spoken Language Translation, 2016.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, et al. The best of both worlds: Combin- ing recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 76 86, 2018.
Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. 2018. URL https:/
openreview wwtp://wweee.tf.eeeeeeeeee Hkkodfdpp
Kyunghyun Cho and Masha Esipova. Can neural machine translation do simultaneous translation?
arXiv preprint arXiv: 020100 2016.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. Incremental decoding and train- ing methods for simultaneous translation in neural machine translation. In Proceedings of the 2018 Conference of the North American Chupter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 2 (Short Papers). pp. 493-499, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10.18653/w1/NIs/ 11..:///////111001 2079. 8 2079
URL https:/ / / wwwwwwwww.kkmm a acl web.
Alvin Grissom II, He He, Jordan Boyd-Graber, John Morgan, and Hal Daumé III. Don't until the final verb wait: Reinforcement learning for simultaneous machine translation. In Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP), pp. 1342- 1352, 2014.
Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor OK Li. Learning translate in real-time with neural machine translation. In 15th Conference of the European Chapter of the Associa- tion for Computational Linguisties, EACL 2017, pp. 1053-1062. Association for Computational Linguistics (ACL), 2017.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison- Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondfej Bo- jar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical ma- chine translation. In Proceedings of the 45th Annual Meeting of the Association for Com- putational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp. 177-180, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: http://wwww.oo aww.allleebl org/ wwtaaanooooooo.o. PO77 -045
Dieterich Lawson Chung-Cheng Chiu, George Tucker, Colin Raffel, Kevin Swersky, and Navdeep Jaitly. Learning hard alignments with variational inference. In 2018 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 5799-5803. IEFE, 2018.

---

Published as a conference paper at ICLR 2020
Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, and llya Sutskever. Learning online alignments with continuous rewards policy gradient. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 2801-2805. IEEE, 2017.
Minh-Thang Luong and Christopher D Manning. Stanford neural machine translation systems for spoken language domains. 2015.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. arXiv preprint arXiv: 2015.
Mingbo Ma. Liang Huang, Hao Xiong. Renjie Zheng. Kaibo Liu, Baigong Zheng. Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. STACL: Simultancous translation with implicit anticipation and controllable latency using prefi--t-preet framework. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1022-3033 Florence, Italy, July 2019. Association for Computational Linguistics. URL
https: www www.aalllao wwtp://wwww.oo./aooooooo.o.m....
1289
Myle Ott, Sergey Edunov, Alexci Bacvski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https: /www acl www. www. www.eb.ogg/antttooooooeo
W18 6319
Colin Raffel, Minh-Thang Thang Luong, Peter Liu, Ron J Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. In Proceedings of the 34th International Confer- ence on Machine Learning- Volume 70, pp. 2837-2846. JMLR. org, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1725-1725, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp.
2017.
Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. Simpler and faster learning of adaptive policies for simultaneous translation. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 1349-1354, Hong Kong, China, Novem- ber 2019a. Association for Computational Linguistics. doi: 10.18655/v1/D19--1133 URL.
www httpe://ww..c..eeb.orrgg hww.:acceeee.oog//attoooooggggg hattooootooooooooooooooooo DD1p://19911111
Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. Simultaneous translation with flexi-
ble policy via restricted imitation learning. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pp. for Computational Linguistics. doi: 10. 11511866111111111155 1582. URL. https:: / wwww.w acl web. arg/anthology. 1582
Florence, Italy, July 2019b. Association
A APPENDIX
A.1 HYPERPARAMETERS
The hyperparameters we used for offline and monotonic transformer models are defined in Table 4
A.2
LATENCY METRICS DEFINITIONS
Given the delays = {gi, 92, M of generating each target token, AP, AL and DAL are defined Table 5
12

---

Published as a conference paper at ICLR 2020
Hyperparameter
WMT15 German-Ennlish
IWSLT Engllinn-iiieeeneeess Vietnameee
encoder embed dim encoder ffn embed dim encoder attention heads encoder layers
1024
512
4096
1024
16
4
decoder embed dim decoder ffn embed dim decoder attention heads decoder layers
1024
512
4096
1024
16
4
dropout
0.3
optimizer
adam
adam-3
(0.9,0.98)
clip-norm Ir
0.0005
scheduler
inverse sqrt 4000
warmup-updates
warmup-init-Ir
le-07
sabel-smoothing
0.1
max tokens
3584 x 8 x 8 2
16000
Table 4: Offline and monotonic models hyperparameters.
Latency Metric
Calculation
y
1
Average Proportion
gi
y
i=1
1
7
Average Lagging
T
9:
y
[
where arg max; (gi =a=l))
i
gi
/
[c]
Differentiable Average Lagging
im
i
where of
|y/
max gi, 06-1
i
Table The calculation latency metrics, given source ær target and delays g
A.3 DETAILED RESULTS
We provide the detailed results in Figure 2 as Table 6 and Table 7
A.4 THRESHOLLD OF READING ACTION
We explore simple method that can adjust system's latency at inference time without training new models. Al/Algoithm I line 8, 0.5 was used as an threshold. One can set different threshold during the inference time to control the latency. We run the pilot experiments IWSLT15 En- Vi dataset and the results are shown as Table 8 Although this method doesn't 't require training new model, dramatically hurts the translation quality.
13

---

Published as a conference paper at ICLR 2020
BLEU
AP
AL.
DAL
Xavg
MMA-IL
0.05
30.7
0.78
10.91
12.64
0.1
30.5
0.70
7.42
8.82
0.2
30.1
0.63
5.17
6.41
0.3
30.3
0.60
4.18
5.35
0.4
29.2
0.59
3.75
4.90
0.5
26.7
0.59
3.69
4.83
0.75
25.5
0.58
3.40
4.46
1.0
25.1
0.56
3,00
4.03
Apar
MMA-H
0.1
28.5
0.74
8.94
10.83
0.2
28.9
0.69
6.82
8.622
0.3
29.2
0.64
5.45
7.03
0.4
28.5
0.59
3,90
5.21
0.5
28.5
0.59
3.88
5.19
0.6
29.6
0.56
3.13
4.32
0.7
29.1
0.56
2.93
4.10
Table 6: Detailed results
MMA-H and MMA-II. on WMT15 DeEn
BLEU
AP
AL
DAL
MILK
0.1
24.62
0.71
5.93
7.19
0.2
24.68
0.67
4.90
5.97
0.3
24.31
0.65
4.45
5.43
0.4
23.73
0.64
4.28
5.24
Aavg
MMA-IL IL
0.02
28.28
0.76
7.09
8.29
0.04
28.33
0.70
5.44
6.57
0.1
28.42
0.67
4.63
5.65
0.2
28.47
0.63
3.57
4.44
0.3
27.9
0.59
2.98
3.81
0.4
27.73
0.58
2.68
3.46
Avar
MMA-H
0.02
27.26
0.77
7.52
8.71
0.1
27.68
0.69
5.22
6.31
0.2
28.06
0.63
3.81
4.84
0.4
27.79
0.62
3.57
4.59
0.8
27.95
0.60
3.22
4.19
Table 7: Detailed results for MILk, MMA-H and MMA-IL on IWSLT15 En-Vi
A.S AVERAGE FOR MMA-H
We explore applying simple average instead of weighted average loss to MMA-H. The results are shown in Figure 7 and Table 9 We find that even with very large weights, we are unable to reduce the overall latency. In addition, we find that the weighted average loss severely affectt the translation quality negatively. On the other hand, the divergence loss we propose in Eqquaion 16 can efficiently reduce the latency while retaining relatively good translation quality for MMA-H models.
14

---

Published as a conference paper at ICLR 2020
Reading Threshold
Weighted Average Latency Loss
p
BLEU
AP
AL
DAL
Lavg
BLEU
AP
AL
DAL
0.5
25.5
0.792387
7.13673
8.27187
0.02
25.5
0.792387
7.13673
8,27187
0.4
25.25
0.73749
5.72003
6,85812
0.04
25.68
0.728107
5.52856
6.61744
0.3
23.06
0.697398
4.88087
6.03342
0.3
24.9
0.602703
2.90054
3.68039
0.2
18.37
0.678298
4.71099
5.94636
0.2
25.3
0.636914
3.54577
4.38623
0.1
8.73
0.696452
5.5225
7.20439
0.1
25.48
0.684424
4.57901
5,54102
Table 8: Comparison between setting threshold for reading action and weighted average latency loss.
WMT15 Germen-English
30
30
1000
deod
0P0
0000
BLEU
20
20
Average Weighted Average
Divergence
10
10
0.6
0.8
10
15
5
10
20
Figure 7: Effect average loss, weighted average loss and variance loss MMA-H WMTI5 DeEn develop-
set.
BLEU
AP
AL
DAL
Aava
Average
0.05
28.5
0.862581
14.3847
17.3702
0.1
27.8
0.855435
13.974
17.02
0.2
28
0.835324
12.8531
15.908
0.4
28.1
0.819408
11.9816
14,9763
1.0
28.2
0.810609
11.7528
14.6695
2.0
28.1
0.800258
11.1763
14.0761
8.0
28.4
0.806439
11.5289
14.6431
Aaeg
Weighted Average
0.02
28.24
0.773922
10.2109
12.2274
0.04
24.35
0.685834
7.06716
8.64069
0.06
7.80
0.875825
16.2046
19.0892
0.08
9.51
0.57372
3.92011
6.1421
0.1
9.78
0.556585
3.3007
5.46142
Avar
Divergence
0.1
27.35
0.736025
8.70968
10.5253
0.2
27.64
0.681491
6.63914
8,3856
0.3
27.37
0.6623
6.04902
7.71922
0.4
27.62
0.638188
5.31672
6.86834
0.5
27.50
0.625759
4.93044
6.38998
1.0
27.1
0.582194
3.64864
4.90997
Table 9; Detailed numbers on average loss, weighted average loss head divergence loss on WMT15 De-En development set
15


Reviews:
[{"review_id": "rkeK-Biu9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach for simultaneous neural machine translation. While prior works deal with recurrent models, the authors adopt previous approaches for Transformer. Specifically, for decoder-encoder attention they introduce Monotonic Multihead Attention (MMA) designed to deal with several attention heads. Each attention head operates as a separate monotonic attention head similar to the ones from previous works by Raffel et al. (2017) and Arivazhagan et al. (2019). MMA has two versions: MMA-H (hard), where each head attends only to one token, and MMA-IL (infinite lookback), where each head attends to all tokens up to a selected position. \n\nThe main novelty of the work in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions (L_var).\n\nExperimental results on two translation tasks (IWSLT En-Vi and WMT De-En) show better quality-latency trade-offs compared to the recurrent analogs. Experiments with different number of attention heads would be helpful, but in the current state are rather confusing (see the comments below).\n\n\nI can not recommend accepting this paper due to the two main reasons.\n\n1) The proposed solution lacks novelty. \nMMA-H and MMA-IL attentions, introduced in this work, are straightforward applications of previous works by Raffel et al. (2017) and Arivazhagan et al. (2019) respectively to each of the heads. Novelty is in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions. However, these alone are unlikely to make a large impact.\n\n2) The results are experimentally weak, evaluation is questionable. \nThe current baseline is the recurrent model, but since the main contribution of this work is in how to deal with several heads the proper baselines would be (i) the same, but with single-head attention, (ii) the same, but without L_var (summing latency penalty over heads is straightforward). However, these baselines are absent and the improvement is likely to be due to the replacement of RNN with the Transformer - this would be a limited contribution.\n\n\n\nOther comments on the experiments.\n\n1) MMA-IL was tested only on one small dataset (IWSLT En-Vi), MMA-H only on 2 datasets - more experiments would help.\n\n2) Figure 2 shows that when increasing lagging, performance first improves, then drops. While the former is expected, the latter is not: one would expect a model to reach the performance of its offline version when given maximum latency, but not to drop. The MILK model (Arivazhagan et al., 2019) behaves as expected, but the proposed MMA does not: does this mean that there is some problem with the model? An explanation of this behavior would be helpful.\n\n3) Number of heads, Figure 4: the results are supposed to show that performance improves when increasing the number of heads, but the figure shows the opposite. If the decoder has more than one layer (which Transformer usually has), for 4 heads perform worse than one, 8 heads (which the standard Transformer setting) - not better than one, and we get the improvement only when using 16 heads with 6 layers. Could you elaborate on this? The text says “quality generally tends to improve with more layers and more heads”.\n\n4) I can see the motivation for using the L_var loss term in this setting, but forcing heads in MMA-H attend to similar positions seems counterintuitive: there is evidence that making attention heads less correlated improves quality (see for example Li et al, EMNLP 2018 https://www.aclweb.org/anthology/D18-1317/), but L_var may end up doing the opposite. How different is the behavior of the learned heads? Figure 3 shows the average attention span, but this does not tell how many of the heads are doing different things. Some analysis would be nice.\n\n\nComments on the presentation.\n\n1) It would be better to mention that encoder self-attention is limited only to previous positions earlier in the text and more prominently (now it’s on the bottom of the sixth page). A good place would be in Section 2.2 when talking about different attention types in the Transformer and modifications required for simultaneous translation.\n\n2) Figure 2: it would be helpful to see offline model performance on these pictures (e.g., a horizontal line showing the BLEU scores). One of the main points of these experiments is to see how much each model drops compared to its offline version.\n\n3) Figure 4: it is not clear from the figure which models it corresponds (for example, what is the difference between the first and the second figures: language pairs? models?)\n\n4) Table 2 shows the BLEU scores for Transformer with unidirectional encoder and greedy decoding. It would be helpful to see also the BLEU scores of Transformer in the standard setting (full attention in the encoder, beam search inference). For now, it is not clear how much one loses by replacing the encoder to unidirectional: if much, then probably it makes sense to work also on the encoder (read and encode source sentence with a latency).\n\nLanguage and typos.\nThe text is quite raw and could use an extra work; a lot of typos starting from the abstract (e.g., “...for multiple attentions heads”).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJg74-IRYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a fully transformer-based monotonic attention framework that extends the idea of MILK. Though the idea of monotonic multi-head attention sounds interesting, I still have some questions below:\n\nAbout the method:\n   1. Is that possible that the MMA would have worse latency than MILK since all the attention heads need to agree to write while MILK only has one attention head?\n   2. Is there any attention order between different attention head?\n   3. I think the MMA only could control the latency during training time, which would produce different models with different latency. Is there any way that enables MMA to control the latency during inference time? Can we change the latency for on given model by tuning the requirements mentioned in (1)?\n\nAbout the experiments:\n    1. Do you have any explanation of why both MMA-H and MMA-IL have better BLEU when AL is small? The results in fig 2 seem counterintuitive. \n    2. I suggest the authors do more analysis of the difference between different attention heads to prove the effectiveness of MMA. \n    3. For the left two figures in fig 4, which one is the baseline, and which one is the proposed model?\n\nI also suggest the authors present more real sample analysis and discussions about the experiments.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxUOzOkqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper applies monotonic attention to the multiheaded (self-attention) mechanisms used in a Transformer. It also proposes a few new losses which encourage low-latency alignments. Experiments are carried out on WMT EnDe and IWSLT EnVi translation, with evaluation using BLEU and latency-related metrics. \n\nReview: Applying monotonic attention mechanisms to the Transformer architecture is an obvious and necessary idea, and as such this paper constitutes an important contribution. The application of monotonic attention to the Transformer is as I would have expected. The latency-reduction losses are novel, however. I think there are various things the authors could do to clarify their results as well as the presentation of their methods, which I discuss below. Overall, I think this is a solid accept, especially with some improvements to the presentation.\n\nSpecific comments & suggestions:\n- There is a typo in the abstract: \"multiple attentions heads\"\n- \"the denominator in Equation 5 is clamped into a range of (\\eps, 1]\" Technically this doesn't need to be an open range on the left.\n- \"which allows the decoder to apply softmax attention over a chunk (subsequence of encoder positions).\" It may be clearer if you just say \"which allows the decoder to apply to a fixed-length subsequence of encoder states preceding $t_i$\".\n- I think more could be done to distinguish the behavior of the encoder self-attention, the encoder-decoder attention, and the decoder self-attention. You write \"The model will write a new target token only after all the attentions have decided to write\" but also \"Some heads read new inputs, while the others can stay in the past to retain the source history information.\" If an attention head is \"staying in the past\", then the model will not be able to write a new target token. I think in one of these cases you are referring to (encoder) self-attention and in the other you are referring to decoder attention. Please clarify.\n- \"Although MMA-H is not quite yet streaming capable since both the encoder and decoder self-attention have an infinite lookback, that model represents a good step in that direction.\" I think you should distinguish between online and streaming translation. It sounds like when you say streaming you mean that the utterances may continue arbitrarily, so infinite lookback is impractical. However, one could truncate the source sequence whenever the decoder outputs an end-of-sentence token, or something. I'm not sure people usually make a strong distinction here.\n- For completeness it would be useful to include at least wait-k, and potentially a line corresponding to offline attention performance, in the plots in Figure 2.\n- Why don't you include MMA-IL in WMT'15 EnDe? (figure 2)\n- Are you copying the results from (Arivazhagan et al., 2019) or did you reimplement MILk in your codebase?\n- The results in Figure 2 could be made much more informative if there was some way of communicating the multiplier (the \"lambdas\") of the different latency losses for each of the different dots on the plot. This would make it much more obvious how important these losses are and how the effect the quality/latency trade-off.\n- There are some additional possible references for online seq2seq, like CTC, the Neural Transducer, \"Learning Hard Alignments with Variational Inference\", \"Learning online alignments with continuous rewards policy gradient”, etc.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends previous models for monotonic attention to the multi-head attention used in Transformers, yielding "Monotonic Multi-head Attention." The proposed method achieves better latency-quality tradeoffs in simultaneous MT tasks in two language pairs.

The proposed method is a relatively straightforward extension of the previous Hard and Infinite Lookback monotonic attention models. However, all reviewers seem to agree that this paper is a meaningful contribution to the task of simultaneously MT, and the revised version of the paper (along with the authors' comments) addressed most of the raised concerns.

Therefore, I propose acceptance of this paper.
Decision: Accept (Poster)