Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a monotonic attention to improve the latency of decoding. The attention is an improvement over the attention based on Gaussian mixture model, allowing the attention weights to depend on the encoder outputs. The experimental numbers are strong.

I am leaning towards accepting the paper, but I do have a few concerns that the paper fails to address. Some of the concerns can be fixed, and some will need additional experiments. I will revise the score based on the authors' response.

The first concern is simply that there are errors and ambiguous statements in the paper. For example, equation (14) is obviously wrong, and in section 2.4 it is unclear how many frames the encoder needs to see before the decoder can predict the next token. The latter is crucial because streaming is the major reason behind the use of monotonic attention. A detailed list is given at the end.

The second concern is that many confounding factors are introduced in the experiments, presenting us from concluding that the improvement is solely coming from the use of SAGMM. For example, in the experiments, CTC is used as an additional loss to guide the learning of monotonic attention. During training, the paper also stochastically introduces an all one vector at the end of the input. It is unclear whether these two have an impact on learning monotonic attentions. It would be more convincing to have results without these additions, and would make the comparison to others more meaningful (assuming that others did not use the same additions).

One minor concern is that the learned representation depends on the attention used. The paper did mention this in the translation experiments, but it is under developed and no further hypotheses or evidence are provided. This would require almost a separate paper, but I believe this is key to understanding why regular content-based attention fails and whether we actually solves the problem.

Here are the detailed comments.

> ... the proposed attention mechanism solves the online and long-form speech recognition problems ...

This is a strong claim. I don't think the paper presents enough evidence that the problem is solved.

> equation (1), (2), and (3)

This is minor, but the variables soft_\alpha, head^h, and Multihead can be better typeset.

> equation (4)

This is again minor. \Sigma_i is a scalar, and it might be better to use the lower case \sigma_i.

> equation (13)

What are I and J?

> \mu_i = \mu_{i-1} + relu3(\Delta_i)

What is relu3?

> equation (14)

This equation is wrong in many ways. I assume j \to \infty means having infinitely many time steps, and \delta_j \to 0 applies to \delta_j for all j.

> 2.4 SAGMM-tr for online inference

In streaming mode, while the frames are coming in, how does the decoder know when to produce the next token?

> We tested the window width c = 15 centered by \mu_i ...

What is c?

> ... we randomly concatenate the 1-vector after the end of source sentence with probability p_{eos} ...

This trick is not implemented in other approaches. Have the authors tried without using this trick? How do know if the improvement is purely due to this trick and not about the proposed SAGMM?

> We adopt the CTC loss on the encoder output ...

The CTC loss is meant to help learn monotonic attention. Why bother using the CTC loss if the proposed attention is already monotonic? Does this suggest that the CTC loss has other added benefits?

> Streamable model, bi-directional enc. in Table 2

How are bidirectional encoders in the streaming case? Using a bidirectional encoder by definition cannot be used for streaming unless I missed something.

> Transformer (121M) (Ours) in Table 2

I assume this uses a regular content-based attention. Please clarify. I would even group the Transformer, GMM, and SAGMM-tr rows together, since they provide the control experiments to support the usefulness of the proposed method. The other rows are less useful, since the architectures are different. The comparison of absolute numbers tell us little, except that the numbers provided for the proposed approach are in the right ballpark.

review_title: Good paper if errors and confounding factors are resolved

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper introduces “source-aware” GMM attention and applies it to offline, online, long-form ASR.  The value of source-aware GMM attention appears to be its ability to “ignore” long segments of silence in the input audio, which could potentially be more difficult to do using other attention mechanisms.  Fairly competitive results are presented for offline ASR.  For online ASR, the results are state-of-the-art amongst sequence-to-sequence-based models. 

Reasons for score: 
The main contribution of this paper seems to be the addition of predicted weights for each encoder step that allow the monotonic GMM attention mechanism to ignore certain input frames.  This seems to be directly useful in ASR; however, I’m not sure if it has any direct use outside of ASR.  The paper is also fairly hard to follow and not well motivated.  Because it is so ASR specific, I feel that it would be a better fit for a speech conference.  


High-level Comments:
* Despite being a relatively minor tweak, the use of source-aware weights for each encoder step is a cool idea that integrates nicely into existing GMM-based attention mechanisms. 
* As mentioned above, I felt that the paper was fairly hard to follow and not well motivated.  It took me until Figure 2 on page 6 to realize that the main benefit of using the source-aware mechanism was to ignore long segments of silence.  It would be good to provide this motivation earlier in the paper. 


Detailed Comments:
* I can’t figure out why there are 4 attention plots in Figure 2 (it says it’s single-head).  It might be a good idea to clarify in the caption. 
* I would recommend having your submission proof-read for English style and grammar issues.  The issues are subtle but addressing them would help to improve readability. 
* Eq. (14) seems unnecessary (it follows from basic calculus and doesn’t apply in actual usage).  Is there a reason it was included? 
* End of Section 2.2, “Moreover, the GMM attention score is uni-modal and cannot discard the non-informative tokens in attention window”: I'm not sure what is meant by "uni-modal" here.  A mixture of unimodal distributions (such as a GMM) is multi-modal. 
* The addition of section 4.3 felt a bit tacked-on.  To me, it doesn’t seem worthwhile to attempt to apply a monotonic attention mechanism specifically designed for ASR to a non-monotonic task like MT.  If indeed there is an interesting contribution to be made via positive constraint relaxation, I would save all of Section 4.3 for the followup paper rather than inserting it at the end of this paper. 

Update (2020-12-03):  Increasing score from 4 to 5. 
review_title: Cool tweak to GMM attention, but very specific to ASR. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel Gaussian mixture-based attention mechanism by incorporating the source (key) information, enabling a flexible representation of the Gaussian attention pattern with the well-described formulation. The effectiveness of the proposed method was validated by 1) artificially created long utterances, 2) Librispeech (segmented and concatenated utterances), and 3) machine translation. The paper also includes several attention visualizations to show the reasonable attention patterns obtained by the proposed method. The paper is well written. The precise monotonic attention gains a lot of attention for streaming ASR and simultaneous machine translation, and this paper would gein broad interests. My concerns about this paper are that 1) the novelty and the effectiveness are a little bit incremental, and 2) the paper requires more clarifications about the formulations and experimental descriptions (see my detailed comments).

Detailed comments:
- In the introduction, "The GMM attention is a pure location-aware algorithm in which the model selects attention windows cumulatively without considering source contents.": This is a little bit hard to understand. I understood it after I followed Section 2. It would be better to make this expression more understandable.
- I'm curious about the GMM's behavior (how the Gaussian parameters would be changed for each layer and each head). These are described in the appendix, but the authors may consider discussing it in the main paper. It is well known that the general soft attentions' behaviors are very different in the lower and higher layers. The lower-layer attentions are blur and less monotonic. I would like to ask the authors to discuss how the GMM parameters can adapt to capture such behaviors in the main paper.
- Equation (4): It's better to explain why the softplus function is used for \Delta and \Sigma. (monotonicity and positive constraint of the variance, right?)
- Equation (9): It's better to have some explanation about why $\delta$ is bounded by [0, 1].
- Equation (14): Is there proof?
- Section 2.4: Does the truncation make training faster?
- Section 4: The authors should clarify why they did not use a language model. Besides, it's better to clarify that the method does not use the language model in the table's caption (Is that correct? All the results listed in the table are without LMs, right?). The readers remember the Librispeech number with the LM, and they may confuse it.
- Why does the table 4 result not include the GMM attention result? It's better to compare the normal GMM and the proposed method in the online mode.
review_title: A new Gaussian-based attention mechanism by incorporating the source (key) information

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper describes a simple extension to the location-only monotonic GMM attention mechanism from Graves (2013), which takes the source/key context into account when computing attention weights.  The proposed method improves ASR performance over model using the baseline GMM attention which does not take source-content into account,  generalizes better to input sequences much longer than those seen during training, while also obtaining competitive performance to other streaming seq2seq ASR models on "matched" test sets.

## Pros:

1. Incorporating source content is an obvious and useful extension to monotonic GMM attention, combining the strengths of content-based approaches such as additive or dot-product attention.

2. It improves performance and generalization while being simpler than existing techniques in the literature (e.g. Mocha, CTC/transducer models which have more complex loss functions).

## Cons:

1. The description of the proposed mechanism is inconsistent with existing literature and is very unclear and confusing in parts.

2. Experiments are somewhat incomplete/missing important comparisons, e.g. comparing baseline GMM attention to the proposed "source-aware" variant in Tables 1,2,5, and comparing to other location-based attention mechanisms, even if non-monotinic, e.g. from http://papers.nips.cc/paper/5847-attention-based-models-for-speech-recognition.pdf
 
3. Overall the writing/language use could use improvement.

## Detailed comments

At the high level, the idea of incorporating the source keys K into GMM attention is a good one.  The proposed method seems to work, and be  simpler to implement than alternative monotonic alignment mechanisms used in seq2seq ASR models.  However, given the current state of the text, with many confusing details, I feel that the paper is not yet ready for publication without significant revisions. 

Many details in the paper, especially Section 2, are unclear:

- Sec 2.2. and throughout the paper:  The described "GMM" and "SA-GMM" attention always use a single component, so don't really count as a Gaussian *mixture*.  Using multiple components would explicitly allow for multimodal attention weights for each output step.  Moreover, since the mixing weights are generally computed independently at each step, using multiple components makes it possible for the base GMM attention to "discard the non-informative tokens".   This mechanism, which would be more precisely called "Gaussian attention", is strictly less flexible than the base GMM attention mechanism that was originally described in Graves, 2013.

- This claim is repeated in paragraph 2 of Sec 3: "uni-modal similar to conventional GMM attention".  When using multiple mixture components, GMM attention is not unimodal.

- Eq 8: The notation here is unclear.  Why is there a softmax over $\psi_i^h$ (a scalar AFAICT)?  Is the softmax computed over all attention heads?   Why is this necessary?  It seems to impair the training of some heads, at least for SAGMM-tr according to paragraph 4 of Sec. 4.2

- Figure 1 is difficult to interpret.  The two plots have difference horizontal axes and therefore don't seem to be directly comparable...  It's not clear what the "key width" in Figure 1b is trying to convey since there is always going to be a single weight per (discrete) encoder step j.

- Sec 2.3.  There is no particular motivation given for the proposed method for integrating source keys.  Why not include $K_j$ in the computation for the standard deviation $\Sigma_i$ as well?  And why is the same weight $\delta_j$ used as a scaling factor (eq (12)) and the mean offset in eq (10)?  These design choices deserve more explanation, and possibly empirical justification.

- Sec 2.4: Is it possible to train SAGMM-tr from scratch?  Or does it need to be first trained using SAGMM and then fine-tuning with truncation enabled?

Experiments:

- Are the different GMM attention variants used encoder self-attention layers as well?  Or does the encoder use conventional "soft attention"?

- As above, it seems unfair not to include any experiments using multiple components when comparing different variants of GMM attention.

- Sec 4.2, Table 1:  Please clarify the differences between the three models labeled (Ours).  Is the difference only in the encoder-decoder attention layer?

English usage.  Just a few examples of grammar errors and unclear text, as there are too many to list.

- page 1, "attend subset of long sequences" is missing a preposition, e.g., "attend to a subset".  It seems that "long sequences" is meant  to refer a single source sequence.

- page 1, "mismatch between attention parameters from decoders and information distribution in encoder outputs".  This sentence is difficult to parse.  What is the mismatch here?   Why would the source encoding and decoder query vector need to "match", especially in a purely location-based attention scheme?

- page 4: "fixed length with hyperparameters"  What are they hyperparameters being referred to here?
- page 5, Sec. 4: "enables early inference".  What does "early inference" mean?  
- page 5, Sec. 3: "for the inference" -> "for inference" 
- page 5, Sec 4.1:  "1 second speeches" -> "1 second long utterances"
- page 5, Sec 4.1: "from 30 vocabulary" -> "from a vocabulary of 30 words"


Other comments:

- Sec 2.2: Sutskever et al., 2014 did not use content-based attention.

review_title: Nice idea, but details unclear and presentation needs polish

Meta-review: The paper proposed a useful incremental extension to the monotonic GMM attention by incorporating source content.
It has shown comparable performance for online and long-form speech recognition, but falls behind on the machine translation task. For online ASR, it would be more convincing to include latency comparisons across different streaming models besides WERs. 
The presentation of the paper can be further improved although it already got better based on reviewers' comments.
As in the discussion, a more accurate description of the method would be "multi-head Gaussian attention" instead of GMM attention.

The main factor for the decision is limited novelty and clarity can be further improved.
Decision: Reject