Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes an approach for simultaneous neural machine translation. While prior works deal with recurrent models, the authors adopt previous approaches for Transformer. Specifically, for decoder-encoder attention they introduce Monotonic Multihead Attention (MMA) designed to deal with several attention heads. Each attention head operates as a separate monotonic attention head similar to the ones from previous works by Raffel et al. (2017) and Arivazhagan et al. (2019). MMA has two versions: MMA-H (hard), where each head attends only to one token, and MMA-IL (infinite lookback), where each head attends to all tokens up to a selected position. 

The main novelty of the work in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions (L_var).

Experimental results on two translation tasks (IWSLT En-Vi and WMT De-En) show better quality-latency trade-offs compared to the recurrent analogs. Experiments with different number of attention heads would be helpful, but in the current state are rather confusing (see the comments below).


I can not recommend accepting this paper due to the two main reasons.

1) The proposed solution lacks novelty. 
MMA-H and MMA-IL attentions, introduced in this work, are straightforward applications of previous works by Raffel et al. (2017) and Arivazhagan et al. (2019) respectively to each of the heads. Novelty is in the way of dealing with multiple heads: (i) average over heads latency instead of single-head latency, (ii) penalty which encourages different heads point to similar positions. However, these alone are unlikely to make a large impact.

2) The results are experimentally weak, evaluation is questionable. 
The current baseline is the recurrent model, but since the main contribution of this work is in how to deal with several heads the proper baselines would be (i) the same, but with single-head attention, (ii) the same, but without L_var (summing latency penalty over heads is straightforward). However, these baselines are absent and the improvement is likely to be due to the replacement of RNN with the Transformer - this would be a limited contribution.



Other comments on the experiments.

1) MMA-IL was tested only on one small dataset (IWSLT En-Vi), MMA-H only on 2 datasets - more experiments would help.

2) Figure 2 shows that when increasing lagging, performance first improves, then drops. While the former is expected, the latter is not: one would expect a model to reach the performance of its offline version when given maximum latency, but not to drop. The MILK model (Arivazhagan et al., 2019) behaves as expected, but the proposed MMA does not: does this mean that there is some problem with the model? An explanation of this behavior would be helpful.

3) Number of heads, Figure 4: the results are supposed to show that performance improves when increasing the number of heads, but the figure shows the opposite. If the decoder has more than one layer (which Transformer usually has), for 4 heads perform worse than one, 8 heads (which the standard Transformer setting) - not better than one, and we get the improvement only when using 16 heads with 6 layers. Could you elaborate on this? The text says “quality generally tends to improve with more layers and more heads”.

4) I can see the motivation for using the L_var loss term in this setting, but forcing heads in MMA-H attend to similar positions seems counterintuitive: there is evidence that making attention heads less correlated improves quality (see for example Li et al, EMNLP 2018 https://www.aclweb.org/anthology/D18-1317/), but L_var may end up doing the opposite. How different is the behavior of the learned heads? Figure 3 shows the average attention span, but this does not tell how many of the heads are doing different things. Some analysis would be nice.


Comments on the presentation.

1) It would be better to mention that encoder self-attention is limited only to previous positions earlier in the text and more prominently (now it’s on the bottom of the sixth page). A good place would be in Section 2.2 when talking about different attention types in the Transformer and modifications required for simultaneous translation.

2) Figure 2: it would be helpful to see offline model performance on these pictures (e.g., a horizontal line showing the BLEU scores). One of the main points of these experiments is to see how much each model drops compared to its offline version.

3) Figure 4: it is not clear from the figure which models it corresponds (for example, what is the difference between the first and the second figures: language pairs? models?)

4) Table 2 shows the BLEU scores for Transformer with unidirectional encoder and greedy decoding. It would be helpful to see also the BLEU scores of Transformer in the standard setting (full attention in the encoder, beam search inference). For now, it is not clear how much one loses by replacing the encoder to unidirectional: if much, then probably it makes sense to work also on the encoder (read and encode source sentence with a latency).

Language and typos.
The text is quite raw and could use an extra work; a lot of typos starting from the abstract (e.g., “...for multiple attentions heads”).

review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: Summary: This paper applies monotonic attention to the multiheaded (self-attention) mechanisms used in a Transformer. It also proposes a few new losses which encourage low-latency alignments. Experiments are carried out on WMT EnDe and IWSLT EnVi translation, with evaluation using BLEU and latency-related metrics. 

Review: Applying monotonic attention mechanisms to the Transformer architecture is an obvious and necessary idea, and as such this paper constitutes an important contribution. The application of monotonic attention to the Transformer is as I would have expected. The latency-reduction losses are novel, however. I think there are various things the authors could do to clarify their results as well as the presentation of their methods, which I discuss below. Overall, I think this is a solid accept, especially with some improvements to the presentation.

Specific comments & suggestions:
- There is a typo in the abstract: "multiple attentions heads"
- "the denominator in Equation 5 is clamped into a range of (\eps, 1]" Technically this doesn't need to be an open range on the left.
- "which allows the decoder to apply softmax attention over a chunk (subsequence of encoder positions)." It may be clearer if you just say "which allows the decoder to apply to a fixed-length subsequence of encoder states preceding $t_i$".
- I think more could be done to distinguish the behavior of the encoder self-attention, the encoder-decoder attention, and the decoder self-attention. You write "The model will write a new target token only after all the attentions have decided to write" but also "Some heads read new inputs, while the others can stay in the past to retain the source history information." If an attention head is "staying in the past", then the model will not be able to write a new target token. I think in one of these cases you are referring to (encoder) self-attention and in the other you are referring to decoder attention. Please clarify.
- "Although MMA-H is not quite yet streaming capable since both the encoder and decoder self-attention have an infinite lookback, that model represents a good step in that direction." I think you should distinguish between online and streaming translation. It sounds like when you say streaming you mean that the utterances may continue arbitrarily, so infinite lookback is impractical. However, one could truncate the source sequence whenever the decoder outputs an end-of-sentence token, or something. I'm not sure people usually make a strong distinction here.
- For completeness it would be useful to include at least wait-k, and potentially a line corresponding to offline attention performance, in the plots in Figure 2.
- Why don't you include MMA-IL in WMT'15 EnDe? (figure 2)
- Are you copying the results from (Arivazhagan et al., 2019) or did you reimplement MILk in your codebase?
- The results in Figure 2 could be made much more informative if there was some way of communicating the multiplier (the "lambdas") of the different latency losses for each of the different dots on the plot. This would make it much more obvious how important these losses are and how the effect the quality/latency trade-off.
- There are some additional possible references for online seq2seq, like CTC, the Neural Transducer, "Learning Hard Alignments with Variational Inference", "Learning online alignments with continuous rewards policy gradient”, etc.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a fully transformer-based monotonic attention framework that extends the idea of MILK. Though the idea of monotonic multi-head attention sounds interesting, I still have some questions below:

About the method:
   1. Is that possible that the MMA would have worse latency than MILK since all the attention heads need to agree to write while MILK only has one attention head?
   2. Is there any attention order between different attention head?
   3. I think the MMA only could control the latency during training time, which would produce different models with different latency. Is there any way that enables MMA to control the latency during inference time? Can we change the latency for on given model by tuning the requirements mentioned in (1)?

About the experiments:
    1. Do you have any explanation of why both MMA-H and MMA-IL have better BLEU when AL is small? The results in fig 2 seem counterintuitive. 
    2. I suggest the authors do more analysis of the difference between different attention heads to prove the effectiveness of MMA. 
    3. For the left two figures in fig 4, which one is the baseline, and which one is the proposed model?

I also suggest the authors present more real sample analysis and discussions about the experiments.
review_title: Official Blind Review #2

Meta-review: This paper extends previous models for monotonic attention to the multi-head attention used in Transformers, yielding "Monotonic Multi-head Attention." The proposed method achieves better latency-quality tradeoffs in simultaneous MT tasks in two language pairs.

The proposed method is a relatively straightforward extension of the previous Hard and Infinite Lookback monotonic attention models. However, all reviewers seem to agree that this paper is a meaningful contribution to the task of simultaneously MT, and the revised version of the paper (along with the authors' comments) addressed most of the raised concerns.

Therefore, I propose acceptance of this paper.
Decision: Accept (Poster)