Learning Monotonic Alignments with Source-Aware GMM Attention
Authors: Tae Gyoon Kang
Ho-Gyeong Kim
Min-Joong Lee
Jihyun Lee
Seongmin Ok
Hoshik Lee
Young Sang Choi
Venue:  (2021)
Keywords: Monotonic alignments
sequence-to-sequence model
aligned attention
streaming speech recognition
long-form speech recognition

Abstract: Transformers with soft attention have been widely adopted in various sequence-to-sequence (Seq2Seq) tasks. Whereas soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which is crucial for monotonic Seq2Seq tasks. Learning monotonic alignments between input and output sequences may be beneficial for long-form and online inference applications that are still challenging for the conventional soft attention algorithm. Herein, we focus on monotonic Seq2Seq tasks and propose a source-aware Gaussian mixture model attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence. We experimentally demonstrate that the proposed attention mechanism improved the performance on the online and long-form speech recognition problems without performance degradation in offline in-distribution speech recognition.

Under review conference paper at ICLR 2021
LEARNING MONOTONIC ALIGNMENTS WITH SOURCE-AWAR AWARE GMM ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers with soft attention have been widely adopted in various sequence-to sequence (Seq2Seq) tasks. Whereas soft attention is effective for learning semantic simi- larities between queries and keys based their contents, it does not explicitly model the order of elements in sequences which is crucial for monotonic Seq2Seq tasks. Learning monotonic alignments between input and output sequences may be beneficial for long- form and online inference applications that are still challenging for the conventional soft attention algorithm. Herein, we focus on monotonic Seq2Seq tasks and propose a source- aware Gaussian mixture model attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence. We experimen- tally demonstrate that the proposed attention mechanism improved the performance on the online and long-form speech recognition problems without performance degradation in offline in-distribution speech recognition.
INTRODUCTION
In recent years, transformer models with soft attention have been widely adopted in various sequence gener- ation tasks (Raffel et al. 2019; Vaswani et al., 2017 Parmarr et al., 2018 Karita et al.) 2019). Soft attention does not explicitly model the order of elements in sequence and attends all encoder outputs for each de- coder step. However, the order of elements crucial for understanding monotonic sequence-to-sequene (Seq2Seq) tasks, such automatic speech recognition (ASR), video analysis, and lip reading. Learning monotonic alignments enables the model to attend to a subset of the encoder output without performance degradation in these tasks. In comparison, soft attention not suitable for streaming inference applications because the softmax operation needs to wait until all encoder outputs are produced. Figure [ (b) shows the attention plot for soft-attention. Soft attention learns the aligmments between queries and keys based on their similarities; it requires all encoder tokens prior to the attention score calculation. Furthermore, soft attention cannot easily decode long-form sequences that are not considered in the training corpus.
The Gaussian Mixture Model (GMM) attention Graves 2013); Battenberg et al. (2020); Chiu et al. (2019) have been proposed for learning the monotonic mapping between encoder and decoder states for long-form sequence generation. The GMM attention is pure location-awan algorithm in which encoder contents are not considered during attention score calculation. However, each element in the encoder output sequence contains different amounts of information and should be attended considering their contents. In figure (c), the GMM attention fails to leam the detailed alignments and attends to many tokens simultaneously.
In this study, adopted the GMM attention mechanism to the modern transformer structure and proposed the Source-Aware Gaussian Mixture Model (SAGMM) attention which considers both contents and orders of source sequences. Each component in the SAGMM is multi-modal and discards non-informative to- kens in the attention window. For online inference, we propose a truncated SAGMM (SAGMM-tr) that discards the long-tail of the attention score in the SAGMM. To the best of our knowledge, this is the first

---

Under review conference paper at ICLR 2021
attempt to adopt GMM-based attention to online sequence generation tasks. Learning accurate monotonic alignments enables the SAGMM-tr to attend to relevant subset of sequences for each decoder step and improves the performance of the model in terms of streaming and long-form sequence generation tasks. Figure (d) shows the monotonic alignments learned by the SAGMM-tr, enabling online inference. Exper- iments involving streaming and long-form ASR showed substantial performance improvements compared with conventional algorithms without performance degradation in offline in-distribution ASR. Furthermore, we tested the SAGMM-tr a machine translation task and demonstrated the performance of the proposed algorithm in monoonnt tasks.
pime apiio Hoig
a
(b)
(d)
Figure 1: Input speech waveform (a) and attention plots between speech waveform and transcription learned by (b) soft attention, (c) GMM attention, and (d) SAGMM-tr attention.
SOURCE-AWARE GMM ATTENTION
2.1 SOFT ATTENTION
Herein, we abbreviate the head index h during attention score calculation for simplicity. In dot-product multi-head soft attention Vaswani et al.] (2017) without relative positional encoding. the attention score from soft attention OSoft derived from the query matrix E IR /xx and key matrix K RJdd xd as follows:
@Soft softmax (QKT/Va
(1)
where d, I, and J are the feature dimension, decoder and encoder sequence length, respectively. The atten- tion context matrix from the h-th head H and the multi-head output M are expressed as
Hh OSSoft V
(2)
M = concat [H'; (H';..;;""M Wo
(3)
where A SSoft denotes the OSooft for the h-th head, V E J the value matrix for the h-th head, and ny the number of heads. In this study, we adopted relative positional encoding Shaw et al.] (2018): [ et al. (2019)
which provides stronger baseline for long-form sequence generation for self-attention layers.
2.2 GMM ATTENTION
The previous studies regarding GMM attention Battenberg et al. (2020; Chiu et al. (2019) were based on early content-based attention (Cho et al.) 2014). Li et al.] (2020) adopted the GMM attention to the transformer framework, but did provide detailed descriptions. Herein, we adopt v2 model in Battenberg
et al. (2020) which improved the performance of the original GMM attention mechanism Graves (2013).
We define the GMM attention component as an attention score of single-head in a multi-head mechanism. In the study by [Battenberg et al. (2020), the value matrix was shared for all Gaussian components, whereas multi-head value matrices were multiplied with the probability from corresponding components in this study to attend to information from different representation subspaces (Vaswani al. 2017). Hence, the multi-head GMM attention introduced here is more generalized algorithm compared with the early GMM attention.
variant of multi-head attention by considering Gaussian distribution
Let us denote the i-th row Q as Qi E Rixdd The normal distribution parameters for the i-th step are expressed
Aj di de = (QQWW)) <(QW,), Q.
(4)
#4 A, + Pi-1
(5)
2

---

Under review conference paper at ICLR 2021
where ((x) is softplus function of æ; WA Rdx1 W Rdx1, and Wo. R The softplus function
was adopted, similar to the study of Battenberg et al. (2020) in which softlus activation demonstrated better
performances than the exponential operation. A mixture component of the GMM attention, from the i-th
decoder step to the j-th encoder token, is derived from the OGMM M Mls..
is defined as follows:
(j 14))
N 01.
V2TT, crp
(6)
20:
OGMMIliJ =
N (j. (j.Hi.0 0.)
(7)
H softmax,, (o Σ daMMI, V/.
where softmax, denotes the softmax function over heads.
The conventional GMM attention mechanism analogous to integral of source sequences with uniform axis spacing as shown in F (a). In this figure, each rectangle denotes the attention score with specified Gaussian component parameters. The GMM attention assumes that each encoder output equally impor- tant. However, this assumption is not satisfied for many input modalities, e.g. speech and videos from real environments. Moreover, the number of modes in the GMM attention is limited by the number mixture components. To learn robust alignments for monotonic Seq2Seq tasks, we propose the SAGMM which considers both contents and locations for the attention mechanism.
2.3 SOURCE-AWARD GMM ATTENTION
Figure 2 (b) shows the scheme of the proposed SAGMM attention. Compared with the GMM attention, the SAGMM is analogous to an integral of normal distribution with non-uniform spacing based on encoder contents. In the figure 2 (b), the width of rectangle 5, controls the model selectively attend the infor- mative tokens during the attention score calculation. This content-aware property of the SAGMM enables the model to learn stable monotonic alignments from the training corpus. Furthermore, SAGMM can easily discard non-informative tokens and aggregate information distributed over several remote tokens.
In the SAGMM, normal distribution parameter Ai, oi Oi, and μa derived from equation equation 5 To encode the contents of the source sequences, the weight for each encoder output ó, is provided from the j- row of the key matrix K follows:
8, sigmoid (K,Ws).
where Ws E Rdx1 and the sigmoid function are introduced to smoothly bound the maximum weight 85 similar to Dong Xu (2020).
N U.P.P E,)
N N E,)
a
8
(b)
V2
Figure 2: Schemes (a) conventional GMM and (b) SAGMM attention score calculation which are similar the integral of the normal distribution uniform and non-uniform axis spacing, respectively.
3

---

Under review conference paper at ICLR 2021
Subsequently, the probability of normal distribution (P3i p4s σ4) is calculated from its cumulative sum by expressed as
Vj = 6j + リー
(10)
(v/ 1k,)²
N (1))
exp
(11)
V2TG,
20,
Finally, the SAGMM attention score OSAGMM is defined as follows:
@SAGMMIl, 6j N (Vz; [4i ot)
(12)
where 8, where 8j is multiplied to describe uneven step sizes in summation.
the carly stage training, we introduced a length penalty loss to facilitate alignment learning between μ and v; it is expressed
Llength Alentth * (wu)- min (1, J)) + (My min (1,)))).
(13)
with Atength = 0.0005 where I and J denote the length of the decoder and encoder sequences, respectively. We turned off the Liength after 200K of training steps in the experiments. Finally, we modified [5 as 14;
fi-1 + min (man(A i 3) to scale (4; and Vj similar to the token indices. This modification facilitates the interpretability of μ and v.
The attention score for each encoder token in the SAGMM was determined independently because they do not rely on the softmax operation over the encoder outputs. is noteworthy that Σ, 6, N (Vy; (Us, (,, approx- imates the integral of the Gaussian distribution. Hence, the sum of the attention weights is approximately thereby facilitating numerical stability and learning without using softmax.
2.4 SAGMM-TR FOR ONLINE INFERENCE
Since the attention score in SAGMM is generated without softmax normalization over all encoder tokens, we can simply build the attention for streaming inference by cropping the long-tail of the Gaussian distribution. In SAGMM-tr, the normal distribution is truncated to limit the past and future contexts as follows:
1
cxp
(vg (us=),,") 20:
if p4 - 22-0 <V p4 2
Nor (153/1; oz)
V2na;
(14)
0.
else.
@SAGMM-rllks 8j Nor (Vzi oz)
(15)
H softmax, (ot)
Σ jEM 2 TT
ASMMMM-ulk, VAA
(16)
Discarding the tokens with threshold of /o; removes approximately 5% of the attention score. In online inference with the SAGMM-tr, H can be calculated after V, exceeds μ4, + 2 It noteworthy that when v 2 μ4 + 2 satisfies in for a current token j = 3,, then the equation satisfies for all j > 3r and H can be emitted without for waiting future contexts.
We started from the SAGMM model and fine-tuned SAGMM-tr until the performance converged. We dis- covered that the models with the SAGMM-tr demonstrated slightly better performances than the SAGMM; hence, we mainly report the results involving the SAGMM-tr model herein.
For online speech recognition using the SAGMM-tr, randomly concatenated the 1-vector after the end of the source sequence with a probability Peos in the training stage and performed training to emit the end- of of-sequence (EOS) token only with the utterances containing the 1-vector. The 1-vector concatenation
4

---

Under review conference paper at ICLR 2021
suppresses the EOS token in the long silence part. Finally, we built unidirectional encoder whose maxi- mum latency was similar to those of Zhang et al. (2020); Dong et al. (2019) by adopting block-wise mask on self attention layers. detailed explanation on the block-wiss wise masking provided in the Appendix.
Algorithm I in the Appendix shows a pseudo code for the SAGMM-tr -r attention in inference stage.
The number of tokens required to 16 was determined by model parameters. We trained and tested the SAGMM-tr with fixed attention window width c, to demonstrate the performance in environments with maximum latency constraint. In this version, equation 14 16 were modified as follows:
= arg max N (U5; Pe-SjjJ
(17)
(-
(-(ur-ni) (u--1;)²)
No (03:41000 oi)
V2xo: erp
20;
if r $ < < 7
(18)
0,
else.
@SAGMM+rlls = 63 Ner (Uzi 0.)
(19)
H =
aSAGMMirllig Vj P
Σ
(20)
In equation 20 wait 5 additional tokens before producing the output. We compared the performance of the SAGMM-U with adaptive and fixed window widths through experiments.
3 RELATED WORKS
Chiu & Raffel (2018) propose monotonic chunkwise attention (MoChA) based on hard monotonic sampling over attention window (Raffel et al.) 2017). The monotonic
attention Ma et al. (2019): [Inaguma] jet al. (2020) and monotonic infinite lookback attention Arivazhagan et al. (2019) are based on the concept similar to that reported by (Chiu & Raffel 2018). MoChA replaces hard sampling operation to probabil- ity distribution over memory in the training stage. By contrast, the SAGMM uses the same operations for training and inference. Furthermore, MoChA relies on a numerically unstable cumulative product that re- quires additional effort for training (Miao al. 2019 Raffel et al. 2017). By contrast, the SAGMM adopts a stable cumulative sum to describe monotonic alignments. Finally, MoCha not effective for long-form speech recognition compared with GMM attention Chiu al. (2019). Meanwhile, our model outperfomeed the GMM in the "test-long" set in experiments.
Dong Xu (2020) introduce the encoder output weight for ASR task. By contrast, our attention mechanism considers the decoder content during attention score calculation. Furthermore, the SAGMM allows the overlap between successive attention windows, facilitating the semantics that span several decoder tokens. Lee et al. (2020) proposed an online softmax-free attention mechanism by aggregating the information of the encoded sequence using an update gate. Similar to soft attention, their approach does not explicitly model monotonic alignments. Furthermore, the update gate for online inference in
Lee et al. (2020) is uni-modal.
The location-aware attention Chorowski et al. (0035); Watanabe et al. (201)): Moritz et al. (2019) which introduces previous attention score as additional argument to the current step was proposed to employ po- sitional information a content-based attention mechanism. The location-aware attention improved the performance of offline inference, whereas we introduced monotonicity in our study for online inference.
Non-attentive neural network-based approaches that do not rely on encoder-decoder attention have been investigated based on Connectionist Temporal Classification (CTC) Graves et al. (2006); Liptchinsky al. (2017): Li et al. (2019), IRNN-transduce (RNN-T)Gravess Graves (2012). Transformer-raansduce (Transformer- T) Zhang al. 2020); Yeh al. (2019), and Imputer Chan al. 2200)) Herein, focus the Seq2Seq approach which does not require an assumption regarding sequence lengths and uses the simple beam search algorithm for inference. Block inference has been widely investigated using manually defined decoding
5

---

Under review conference paper at ICLR 2021
blocks Jaitly al. (2015); Tsunoo et al. (2020) or joint decoding using CTC model Moritz et al. (2020).
We doid not use joint CTC decoding Kim et al. (2016) nor human supervision for the blocked inference.
4 EXPERIMENT
We conducted experiments in speech command Warden (2018) and LibriSpeech Panayotov al. (2015) datasets to compare the performances of our model in standard, online and long-form ASR tasks. We only compared the end-to-end speech recognition model without external language model (LM) rescoring. Furthermore, we performed preliminary experiments on the translation task to demonstrate the performance of the SAGMM-ur on monotonic tasks.
4.1 EXPERIMENTS ON CONCATENATED SPEECH COMMAND DATASET
First, we performed a speech recognition experiment with limited vocabulary demonstrate the difficulty of using the typical soft attention algorithm in decoding utterances with unseen sequence lengths. The speech command dataset consists of second long speeches from various speakers uttering single words from a vo- cabulary of 30 words. We built 100K utterances in the training corpus Warden (2018) by concatenating ran- domly selected 5-9 utterances in the training data and 500 test utterances by concatenating {3, 7, 10, 15, 20) randomly selected words in the test set. We cropped both the start and end of selected utterances randomly from 0.05s to 0. 15s before concatenation to prevent the SAGMM from yielding a trivial solution.
We compared the word error rate (WER) of transformers with soft and SAGMM attentions test sets. As shown in Table [ the soft attention mechanism tends to memorize the sequence length distribution from the training corpus and fails to decode correctly in the unseen sequence length. Mcanwhile, the proposed SAGMM algorithm was robust to sequence length mismatches. Detailed hyperparameters for the experi- ment, decoded examples for soft attention and the SAGMM are shown in the Appendix.
Table I: WERs (%) concatenated speech command dataset with various number of concatenated words.
7
10
15
20
soft attention
40,00
5.71
13.20
39.80
56.05
SAGMM
4.67
6.29
5.50
5.40
6.45
4.2 EXPERIMENTS ON LIBRISPEECH DATASET
We conducted experiments on LibriSpeech whose training dataset comprised 960h of read audio books Panaayotvv et al., 2015). We validated the performance of the models on development sets and reported the WERs in the tests sets with a fixed beam width of 4.
First, compared the performance of the SAGMM-tr with conventional algorithms in offline speech recog- nition. We trained the multi-heat transformers similarly to previous experiments. The encoder comprised 10 layers of self-attention blocks, with 768 hidden nodes and 3,072 filters. The decoder was constructed by stacking 4 self-attention blocks with the same node and filter sizes. The encoder and decoder self-attention employed soft-attention with relative positional encoding (Dai et al., 2019). We trained and tested models that employed soft attention, GMM attention, and SAGMM-trr attention mechanisms as the encoder-decode attention. We adopted the auxiliary CTC loss on the encoder output with Aete = 0.1 for a fair comparison with the previous attention-basse models Lee et al. 2020); Dong & Xu (2000) Kim et al. (2019). The WER of the SAGMM-tr model without CTC loss was 3.84% on test-clean dataset. Detailed hyperparameters used in the experiment and the performance of SAGMM-tr without CTC loss are shown in the Appendix.
Table 2 shows the performance of various non str-smarae models. In the non-streamable models, our base- line transformer demonstrated performances similar to those of previous studies considering the number of parameters. Furthermore, we compared the performances of GMM and SAGMM-tr models with a bi- directional encoder those of other streamable decoder algorithms. The performance of the SAGMM-tr
6

---

Under review conference paper at ICLR 2021
Table 2: WERs (%) of offline speech recognition models without LM on LibriSpeech (with number of parameters when available).
test-clean
test-other
Non-streamable
LAS Zeyer et al. 2018)
4.70
15.20
Location-aves aware attention
Moritz et al.
2019
6.1
20.0
LAS (180M) Park et al.
2019)
3.4
8.3
LAS (361M) (Park et al.
2019)
2.8
6.8
Transformer (270M) [Synnaeve al. 2020
2.89
6.98
Streamable decoders,
RNN-T (130M) (Zhang et al. 2020)
3.2
7.8
bi-directional encoders
Transformer-T (45.7M) (Yeh al. 2019
6.08
13.89
Transformer-T (139M) (Zhang et al. 2020)
2.4
5.6
DecGRC (Lee et al.
(2020)
4.83
14,90
Mocha (Lee et al.
2020)
4.95
15.32
CIF (Dong & Xu
20200)
3.41
12.62
Transformer((urs)
Soft attention (12TM)
3.20
7.96
GMM (121M) (Ours)
3,71
8.52
SAGMM-tr (121M) (Ours)
3.21
7.99
attention was consistent with that of our transformer model, whereas was better compared with those of other streamable Seq2Seq models. The results showed that the SAGMM-tr successfully described the monotonic alignments between speech and transcription without attending to encoder outputs.
Next, investigated the performance of the SAGMM-tr tr in online ASR task. We trained single-head and multi-head SAGMM-tr models using the block-wise masked encoder. Additionally, the truncated GMM model was tested, but discarded in experiments because the model attended to the future context as shown in [A.4] and failed to learn the correct aligmments. the multi-head SAGMM-tr, several heads in the first decoder layer were not trained and softmax(o!") approached zero. Therefore, we pruned these heads after the performance on the development set had converged and fine-tuned for 30K additional steps. We believe that training and head pruning with of can prevent over-fitting when the desired monotonic alignments do not require many heads, particularly in the first decoder layer.
Table 3 shows the performances of models based various online inference approaches. As shown, the single-head SAGMM-tr outperformed other CTC and attention- -baed algorithms. The multi-head mecha- nism further improved the performance SAGMM-tr. Adopting unidireciotmal encoder with block-wir mask increased the WER on the test-clean set by 0.31%. Our model demonstrated slightly worse perfor- mance than state-of-the-arr algorithms with transformer-T (Zhang et al., 2020). Subword regularization Kudo (2018): Hannun et al. (2019) and sequence-level loss [abour et al. (2019); Prabhavalkar et al.) (2018) may enable the SAGMM-tr to achieve performances similar to the best transducer models. Furthermore, the SAGMM-tr can be adopted in various monotonic Seq2Seq tasks easily; this is because it can be inferred via a simple beam search without probability marginalization transducers and does not require the assumption that the target should be equal or shorter than the source.
Figure shows the attention plots the single-head SAGMM-tr which successfully captures alignments between speech and transcription for utterance with long silence. Furthermore, Figure shows an
Figure 3:
Attention plots for single-head SAGMM-tr model with a speech utterance with a long silence. Each subfigure shows attention plot from each encoder-decoder attention layer in decoder with four blocks.
7

---

Under review conference paper at ICLR 2021
Table 3: WERs (%) of online speech recognition models without LM on LibriSpeech (with number of parameters when available).
test-clean
test-other
CTC variants
Gated ConvNet (Liptchinsky et al.
2017)
6.7
CTC (Chan al. 20200)
4.6
13.0
Imputer(DP)(Chr han et al.
2020)
4.0
11.1
Jasper (333M) (Li et al.
2019)
3.86
11.95
Transducers
Transformer-T (45. 4M) Yeh al.
2019
6.47
15.79
ConvT-T (67M) (Huang et al. (020)
3.5
8.3
Transformer-T (139M) (Zhang al.
2020
3.0
7.7
Streamable
Triggered Attention (Moritz et al. 2019
7.4
19.2
Seq2Seq
Mocha Kim et al.
2019)
6.30
18.41
Mocha (Lee et al. 2020)
5.15
16.45
DecGRC [Lee et al. 2020)
5.87
17.04
CIF (Dong & Xu 2020)
3,96
11.19
SAGMM-Ir (121M) (SI2IIM))snnneeeeeeee (sigle-head)
3.73
9.92
SAGMM-tr (121M) (multi-head)
3.52
9.29
Table 4: Experimental results of (a) SAGMM-tr with fixed window width and (b) long-form speech recog-
nition on "test-long" set. (sh) and (mh) denote single-head and multi-head attention, respectively.
(a)
(b)
test-clean
test-other
test-clean
test-long
SAGMM-tr (sh)
3.73
9.92
Soft attention
3.20
32.77
SAGMM-tr (mh)
3.52
9.29
GMM
3.71
24.71
SAGMM-tr fixed
3.67
9.76
SAGMM-tr (sh)
3.73
5.60
SAGMM-tr (mh,fixed)
4.45
10.34
SAGMM-tr(mh)
3.52
8.13
+fine-tuning
3.60
9.42
attention alignments from multi-head soft attention, truncated GMM attention and SAGMM-tr attentions(the full plots are shown in Appendix(A.4) In soft attention, several heads attend to the speech absence frames constantly for every decoder step. These heads might facilitate in the utterance-level normalization the attention context vectors. In future studied, the role of attenndgg heads soft attention should be investigated and introduced to the SAGMM-tr attention. The truncated GMM model failed learn the alignment and attended to a wide range of encoder tokens. Finally, the SAGMM-tr attention learned the monotonic alignments successfully. The SAGMM-tr attention plots the lower layers were more blurred compared with the higher layers. We suppose that the lower layers found the relevant subsets in the sequence and the higher layers utilized the context vectors from the lower encoder-decodee attentions and relatively more focused, partly resembling the behavior of soft attention.
The multi-head SAGMM-Ir model showed more blurred alignments compared with the single-head model. Our hypothesis for this phenomenon is that several heads attended larger windows to provide context in- formation. The fixed attention window width in equation 17 20 or regularization on a; can minimize the maximum latency of online inference. In this study, we tested the fixed attention window width. We began with the SAGMM-tr models trained for the online ASR experiment and measured the WERs before and after fine-tuning with constant window width (c 15 frames). From the results in Table 4 (a), the SAGMM-tr attention mechanism can be adopted to the models with a fixed attention window width.
Finally, concatenated utterances from the same speaker in the "test-clean" set to build "test-long" set for the long-form speech recognition. The average length of utterances in the test-long set was 54s. In the SAGMM-tr model, the beam search algorithm EOS token until the attention window encompassed Vr.J- The performance of the GMM attention did not improved by the same modification since GMM attention attends to V/.J prior to the end of sentence.
slightly modified from Algorithm to suppress the
8

---

Under review conference paper at ICLR 2021
As shown in Table (b), the SAGMM-tr outperformed the conventional soft and GMM attention models in long-form speech recognition. Comparing the SAGMM-tr models, the multi-heat SAGMM-tr showed worse performance than the single-head model. The performance degradation of the multi-head mechanism on the "test-long" set might arise from accumulated attention windows range mismatches, Attention window ranges multi-headed models should investigated in future studies.
4.3 EXPERIMENTS ON NON-MONOTONIC TASK
The proposed algorithm focuses on strict monotonic tasks that do not allow reordering. However, the self- attention layers the encoder can reorder the semantics imput sequences before monotonic alignments from the SAGMM-tr. We conducted machine translation experiments on the WMT EN-DE environment to demonstrate the performance of the SAGMM-tr attention in non-monotonic tasks. Whereas the machine translation is non-monotonic task, various approaches have been proposed improve the performance of online inference and simultaneous translation (Arivazhagan et al., 2019; Elbayad et al. 2020). this study, we conduct simple experiments to figure out if the GMM attention can manage non mon-monoto tasks.
We trained the encoder-decoder model similar to the transformer- base in Vaswani et al. (2017) except that the head size was in the encoder-decoder and decoder self-attention We trained model with 4.5M pairs of sentences and measured the BLEU score on the newstest2013 set after performing 100K of training steps. For the uni-directional encoder, block-wiss mask with M = 5 was applied to the SAGMM-tr model with a bi-directional encoder and then fine-tuned for 40K additional steps.
Table 5 shows the performances of soft and SAGMM-Ir attention models on the newstest2013 set. As expected, the performance of the SAGMM-tr was not consistent with that of conventional transformer models with soft attention. Furthermore, the uni-directional encoder de- teriorated the performance of the SAGMM-tr, similar the speech recognition experiments. Because the SAGMM-tr does not consider local reordering, it is dif- ficult for the proposed algorithm to learn locally non-monotonic functions. our future studies, we will improve the performance of SAGMM-tr for local reordering tasks by positive constraint relaxation A,
Table 5: BLEU scores on newstest2013 set for WMT EN-DE environment.
EN-DE
Transformer (Vaswani et al. 2017
25.8
Soft attention (Ours)
25.7
SAGMM-tr (bi-enc)
24.0
SAGMM-tr (uni-enc)
23.7
5 CONCLUSION
We proposed the SAGMM, which attends the subset of source sequences according to the normal distribution on a weighted encoder output axis, considering both contents and order of elements in sequences. Further- more, we proposed the SAGMM-tr for online/real-time inference applications. Based on results on various speech recognition tasks, it was discovered that the proposed attention mechanism can learn monotonic alignments between source and target sequences without human supervision. The performance of trans- former with the SAGMM-tr improved for online and long-form speech recognition without performance degradation in the standard offline ASR task. In future studies, we plan to adopt SAGMM-based attention mechanisms for natural language processing tasks that allow local reordering during sequence generation. We are also interested in latency minimization for online inference by controlling μ and in both the training and inference stages.
REFERENCES
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung- Cheng Chiu, Semih Yavuz, Ruoming Pang. Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultaneous machine translation.
ACL, 2019.
9

---

Under review conference paper at ICLR 2021
Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. ICASSP,
William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. Imputer: Se-
quence modelling via imputation and dynamic programming. ICML,, 2020.
Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. ICLR, 2018.
Chung-Cheng Chiu, Wei Han, Yu Zhang, Ruoming Pang, Sergey Kishchenko, Patrick Nguyen, Arun Narayanan, Hank Liao, Shuyuan Zhang, Anjuli Kannan, Rohit Prabhavalkar, Zhifeng Chen, Tara Sainath, and Yonghui Wu. A comparison of end-to-end models for long-form speech recognition. ASRU, 2019.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statisti- cal machine translation. EMNLP, 2014.
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. Neurips, 2015.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. ACL, 2019.
Linhao Dong and Bo Xu. Cif: Continuous integrate-and-fing for end-to-end speech recognition. ICASSP, 2020.
Linhao Dong. Shuang Xu, and Bo Xu. Speech-transformer:: A no-recurrence sequence-to-sequene model for speech recognition. ICASSP, 2018.
Linhao Dong Feng Wang, and Bo Xu. Self-attention aligner: A latency-control end-to-end model for asr using self-attention network and chunk-hopping, ICASSP, 2019.
Daniel Duckworth, Arvind Neelakantan, Ben Goodrich, Lukasz Kaiser, and Samy Bengio. Parallel sched- uled samplings. ArXiv, abs/1906.04333 2019.
Maha Elbayad, Laurent Besacier, and Jakob Verbeek. Efficient wait-k models for simultaneous machine translation. ArXiv, abs/2005.08595, 2020.
Alex Graves. Sequence transduction with recurrent neural networks. ArXiv, aat.a/////111111.11111111111111 2012.
Alex Graves. Generating sequences with recurrent neural networks. ArXiv, htts/1308.0850 2013.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. onnectionist tem- tem-poral clas- sification: labelling unsegmented sequence data with recurrent neural networks. ICML, 2006.
Awni Hannun, Ann Lee, Qiantong Xu, and Ronan Collobert. Sequence-tor seuuence-o--eqqeece speech recognition with time-depth separable convolutions. ArXiv, 0tts/1994.02619, 2019.
Wenyong Huang, Wenchao Hu, Yu Ting Yeung, and Xiao Chen. Conv- transforrfr transducer: Low latency, low frame rate, streamable end-to-end speech recognition. Interspeech, 2020.
Hirofumi Inaguma, Masato Mimura, and Tatsuya Kawahara. Enhancing monotonic multihead attention for streaming asr. ArXiv, hbs/2005.09394, 2020.
Navdeep Jaitly, David Sussillo, Quoe V. Le, Oriol Vinyals, Ilya Sutskever, and Samy Bengio. A neural transducer. ArXūv, abs/1511 1551111110486.3 2015.
10

---

Under review conference paper at ICLR 2021
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, Shinji Watanabe, Takenori Yoshimura, and Wangyou Zhang. A comparative study on transformer vs mn in speech applications.
ASRU, 2019.
Kwangyoun Kim, Kyungmin Lee, Dhananjaya Gowda, Junmo Park, Sungsoo Kim, Sichen Jin, Young- Yoon Lee, Jinsu Yeo, Daehyun Kim, Seokyeong Jung. Jungin Lee, Myoungji Han, and Chanwoo Kim. Attention based on-device streaming speech recognition with large speech corpus. ASRU, 2019.
Suyoun Kim. Takaaki Hori, and Shinji Watanabe. Joint etc-attention based end-to-end speech recognition using multi-task learning. ArXiv, (tss/1009.06777 2016.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. ACL, 2018.
Hyeonseung Lee, Woo Hyun Kang, Sung Jun Cheon, Hyeongju Kim, and Nam Soo Kim. Gated recurrent context: Softmax-free attention for online encoder-decoder speech recognition. ArXiv, abs/2007.05214, 2020.
Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. Jasper: An end-to-end convolutional neural acoustic model. Interspeech, 2019.
Naihan Li, Yanqing Liu, Yu Wu, Shujie Liu, Sheng Zhao, and Ming Liu. Robutrans: A robust transformer- based text-to-speech model. AAAI, 2020.
Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Letter-based speech recognition with gated convnets. ArXiv, abs/1771.09444, 2017.
Xutai Ma. Juan Pino, James Cross, Liezl Puzon, and Jiatao Gu. Monotonic multihead attention. ArXiv, abs/1909. 12406, 2019.
Haoran Miao, Gaofeng Cheng, Pengyuan Zhang, Ta Li, and Yonghong Yan. Online hybrid ctc/attention architecture for end-to-end speech recognition. Interspeech, 2019.
Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Triggered attention for end-to-end speech recognition. ICASSP, pp. 5666-5670, 2019.
Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Streaming automatic speech recognition with the trans- former model. ArXiv, abs/2001 .02674, 2020.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. ICASSP, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. ICML, 2018
Rohit Prabhavalkar, Tara N. Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung- Cheng Chiu, and Anjuli Kannan. Minimum word error rate training for attention-based sequence-to-sequence models. ICASSP. 2018
Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. ICLR, 2017.
11

---

Under review conference paper at ICLR 2021
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, (tbs/1910.1068.. 2019.
Sara Sabour, William Chan, and Mohammad Norouzi. Optimal completion distillation for sequence learning. ICLR, 2019.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. NAACL, 2018.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end ast. from supervised to semi- supervised learning with modern architectures. Workshop on Self-supervision in Audio and Speech,, 2020.
Emiru Tsunoo, Yosuke Kashiwagi. and Shinji Watanabe. Streaming transformer asr with blockwise syn- chronous inference. ArXiv, abs/2006. 149411, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neurips, 2017.
Pete Warden. Speech commands:
dataset for limited-vocabulary speech recognition. ArXiv,
hbss/184403309,, 2018.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11 (8):1240-1255 1253, 2017. doi: 10.100//STSP2017774455..
Ching-Feng Yeh, Jay Mahadeokar, Kaustubh Kalgaonkar, Yongqiang Wang, Duc Le, Mahaveer Jain, Kjell Schubert, Christian Fuegen, and Michael L.. Seltzer. Transformer--raanddueer End-to-end speech recogni- tion with self-attention. ArXiv, abs/1910.12977/ 2019.
Albert Zeyer, André Merboldt, Ralf Schlüter, and Hermann Ney. A comprehensive analysis on attention models. NIPS: Workshop IRASL, 2018
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and mn-t loss. ArXiv, abs/2002.02562. 2020.
12

---

Under review conference paper at ICLR 2021
A APPENDIX
A.1 BLOCK-WISE WISE MASK FOR ENCODER SELF-ATTENTION LAYER
Figure 4: Encoder self-attention mask between query (vertical axis) and key (horizontal axis) in M 3 case.
The block- wise uni-directional mask was adopted for encoder self-attention layers to block future context more than M tokens. 4 shows the block-wise masking with M = casc. In the experiment with SAGMM- tr, we chose M = 30 whose gmaximum latency is comparable to other studies with uni-directional encoders (Zhang al. 2020 Dong et al. 2019). The blcok-wise masking is different from lookahead operation since encoder outputs from the block-wiss wise masking are not guaranteed to look future M token depends on their position in block while the lookahead operation provides maximum future context for all encoder outputs.
A.2 PSEUDO CODE FOR ONLINE INFERENCE WITH SAGMM-TR ATTENTION
Algorithm 1: Decoding process of SAGMM-tr attention mechanism for a h-th head.
Data: encoder key and value vectors K/. V for j E {1, 2,. 1, n = v = 0, yo = SOS while yi-1 # EOS do Q feedforward (11-1)
Lower decoder layers +/ A o dt GMMparam (Q!), u 1 1 + min (max (),, / equation 4 *
pos μ 2/ pos ut 22 Attend 0
for j = do
8/ sigmoid (K)W)), 1n 2 + +
if > pos then
Break
else if 1 > " then
a d'AGCM--rlk,, = 55 Nor (Mfiptiot)
/* equation 14
15
*/
Attend = Attend" + +aSAGMMArll V/
y Output (Decoder (softmax, (d+t) Allend))). i=i+1
A.3 EXPERIMENT CONFIGURATIONS
For all speech recognition experiments, the 80-dimensional log-Mel filterbank features extracted with a 25ms window and shifted every 10ms. Each input vector stacked log-Mel feature vectors, downsampled 30ms frame rate.
the speech command dataset experiment, We trained the Dong al. (2018) with soft attention and SAGMM attention for encoder-decoder alignments. We adopt the
transformer Vaswani et al.) (2017):
13

---

Under review conference paper at ICLR 2021
Figure 5: Attention alignments plots from multi-head truncated GMM model.
relative positional encoding for both the encoder and decoder self-attention (Dai et al. 2019). The encoder consisted layers of self-attention block with 512 nodes and 2,048 filter size, respectively. The decoder was constructed by stacking 3 self-attention block with the same node and filter sizes. The number of heads for encoder self attention, encoder-decoder attention, and decoder self attention were {8, 4, 4}. respectively. We employed label smoothing of value 0.2 after 100k (Duckworth et al. 2019). We used the Adam optimizer with B1 0.9, B3 0.98 and = 1. The learning rate was linearly increased to 0.1 until 16k steps and set constant in following training steps.
0.15, and applied parallel scheduled sampling with probability
used the tensorflow framework and trained the model until 500K step with 4 m40 GPUs. We averaged 5 last checkpoints before the inference. The dropout ratio for attention weights, rectified linear units, output of sub-layers, and neural network input after positional encoding were 0.2, 0.1, 0.2, and 0.2, respectively. The Specaug algorithm Park et al. (2019) also applied during training. The width of beam was set to in all experiments. The output token segments are grepheme of the reference.
In the LibriSpeech experiment, The learning rate was linearly increased to 0.1 until 16k steps and constant until the model converges on dev-clean dataset. Then, we lowered learning rate to 0.02 and fine-tuned until the model converges. We used the tensorflow framework and the model trained from p40 GPUs. We averaged (5, 10, 15} last checkpoints before the performance evaluation. The 1K wordpiece vocabularies extracted from training dataset is used for output token segmentation. Other parameters are the same with those in the speech command dataset experiment.
10 days 8
A.4 ATTENTION PLOT FOR VARIOUS ENCODER-DECODEE MODELS
To show the SAGMM-tr able to learn alignments between encoder and decoders, we plot the attention score for single and multi-head models. We used the utterance "Chapter eleven ilong pausez the morrow brought a very sober looking morning the sun making only few efforts to appear and Catherine augured from it everything most favourable to her wishes" in LibriSpeech.
The truncated GMM model fails to learn the true alignment attended several distinct windows in Surpris- ingly, the performance of truncated GMM was 4.03% and 10.16% for test-clean and test-other, respectively. The decent performance of truncated GMM implies that the GMM learns to attend as many frames as pos- sible instead of learning alignments between sequences. Therefore, the truncated GMM attention not suitable for streaming inference.
Figure 6: Attention alignments plots from multi-head soft attention model.
14

---

Under review conference paper at ICLR 2021
Layer
Layer
Layer
Layer 3
to
Figure 7: Attention alignments plots for single-head SAGMM-tr for the utterance "Chapter eleven jlong pause; the morrow brought very sober looking morning the sun making only few efforts to appear and Catherine augured from it everything most favourable to her wishes" in LibriSpeech. The attention plot in time domain longer than the original wave due to 1-vector concatenation for E0$
Figure 8: Attention alignments plots from multi-head SAGMM-tr model with adaptive window width. The empty plots denote the pruned head in the first layer
Figure 9: Attention alignments plots from multi-head SAGMM-tr attention model with fixed window width. The empty plots denote the pruned head in the first layer
In the soft attention model, the several heads learns monotonic alignments between inputs and outputs as shown in Figure 6 However, other heads attend to the whole speech presence or absence intervals. We assume these heads help utterance-level normalization. While few heads show monotonicity, the soft attention is not suitable for online inference due to softmax operation over all frames.
The attention alignment figure for single and multi-headed SAGMM-tr model is shown in Figures 9 From the figures, both models learn the optimal alignments between input wave and transcription and distinguish the word boundaries without relying human knowledge. The width of attention windows in multi-head SAGMM-tr with adaptive window width are longer than those in the single-head model. The additional objective function for regularizing d; would be helpful to reduce the window width in figure g
g and
15

---

Under review conference paper at ICLR 2021
In the multi-head model with fixed window width, the model learns the multi-head monotonic alignment without increasing the window width. Considering the attention window width as hyperparameter from external knowledge, it shown that additional information could improve the model latency SAGMM-tr while the model can also be trained without human knowledge.
A.5 ABLATION STUDIES
We tested the three ablation experiments on SAGMM-tr attention Figure First, we tested the model without CTC loss encoder output. The CTC loss improves the word error rate performances of model sim- ilar to previous studies, but the model showed decent performance without the CTC loss. Second, we tested the SAGMM-tr model with uni-directional encoder without initialization from the bi-directional encoder. The result shows slightly worse performance compared to the original SAGMM-tr. Finally, we train the SAGMM model in the training and decoded with the SAGMM-tr decoding scheme without fine-tuning. The performance degradation in 6 shows that the mismatch between training and test stages should be removed by the fine-tuning with truncated normal distribution.
Table 6: The word error rate (%) of ablation experiments for SAGMM-tr attention mechanism in Lib- riSpeech.
test-clean
test-other
SAGMM-Ir
3.52
9.29
SAGMM-tr w/o CTC loss
3.84
9.86
SAGMM-Ir uni-encoder from scratch
3.69
9.79
SAGMM-tr w/o truncated attention fine-tuning
3.76
9.96
A.6 ATTENTION PLOTS FOR SAGMM-TR MACHINE TRANSLATION EXPERIMENTS
We plot the first three decoder layers of the SAGMM-tr model with bi-directional encoder for example utterance "The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that not long-term life threatening, and opt- ing instead, in such cases, for active monitoring of the disease." and model output "Der Patient muss wirklich verstanden werden, um den Grad des Risikos seines Krebses zu verstehen, indem er ihm die Möglichkeiten zur Verfügung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen und sich stattdessen solchen Fällen für eine aktive Überwachung der Krankheit entscheiden.".
We also plot the first three mecoder-decoder attention and encoder-self attention layers of the SAGMM-tr model with uni-directional encoder for an example utterance "The patient really needs be made to un- derstand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, such cases, for active monitor- ing of the disease." and model output "Der Patient muss wirklich verstanden werden, um das Risiko seines Krebses zu verstehen, indem er ihm die Möglichkeiten zur Verfügung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen, und stattdessen in solchen Fällen für eine aktive Überwachung der Krankheit zu entscheiden.".
From the figures IO and and 12 the transformers with SAGMM-tr approximates the machine translation task as strict monotonic task and finds optimal alignments under the assumption. Since the machine translation is not strictly monotonie task, the performance of the SAGMM-tr attention deteriorates compared to the soft attention. However, the transformer with SAGMM-tr attention and uni-directional encoder enables online inference for simultaneous translation.
16

---

Under review conference paper at ICLR 2021
Figure 10: Encoder-decoder attention alignments plots from multi-head SAGMM-tr model with bi- directional encoder in machine translation experiment.
Encoder self-attention alignments plots from multi-head SAGMM-tr model with uni-directional encoder in machine translation experiment.
Figure 11:
A.7 DECODED EXAMPLES OF SOFT ATTENTION AND SAGMM MODELS ON SPEECH COMMAND AND LIBRISPEECH DATASETS
Table 7 and 8 show the decoded examples from the models with unseen sequence lengths. From the tables, soft attention has difficulty generate the sequences that are significantly shorter longer than the training corpus distribution. In contrast, the SAGMM-tr is robust to the sequence length mismatch and generates the transcription arbitrary length.
17

---

Under review conference paper at ICLR 2021
Figure 12: Encoder-decoder attention alignments plots from multi-head SAGMM-tr model with uni- directional encoder in machine translation experiment.
Table 7: The example results for attention based models in concatenated speech command dataset.
Transcription
Reference
seven three stop seven three stop stop seven three stop two right two cat bird right left tree four one wow marvin left
Soft attention
SAGMM
Reference
Soft attention
marvin left two right two cat bird right left tree four tree one wow marvin left
two right two cat bird right left
SAGMM
18

---

Under review conference paper at ICLR 2021
Table 8: The example results for attention- models test-clean-long set from LibriSpeech dataset. (sh) and (mh) denote the single-head and multi-head attention, respectively.
Transcription
Reference
was idle for him to move himself to be generous towards them to tell himself that he ever came to their gates stripped of his pride beaten and in beggar's weeds that they would be generous towards him loving him themselves idle and embittering finally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in a chord words was it their colours they were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races
Soft attention
it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves to love him as ourselves with the same amount and intensity of love the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and martialed races
SAGMM-Ir(sh)
it was idle for him to move himself to be generous towards them to tell himself that he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him themselves idle and embittering finally to argue against his own dispassionate certitude that the commandment of love bade not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleys and wood be girt and set and of intrenched and marshalled races was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar's weeds that they would be generous towards him loving him themselves idle and embittering finally to argue against his own dispassionate certitude that the command of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay there beyond the irish sea europe of strange tongues and valleyed and wood girt and citadel and of intrenched and marched and races
SAGMM-tr(mh)
19


Reviews:
[{"review_id": "bu1M1cEdNr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel Gaussian mixture-based attention mechanism by incorporating the source (key) information, enabling a flexible representation of the Gaussian attention pattern with the well-described formulation. The effectiveness of the proposed method was validated by 1) artificially created long utterances, 2) Librispeech (segmented and concatenated utterances), and 3) machine translation. The paper also includes several attention visualizations to show the reasonable attention patterns obtained by the proposed method. The paper is well written. The precise monotonic attention gains a lot of attention for streaming ASR and simultaneous machine translation, and this paper would gein broad interests. My concerns about this paper are that 1) the novelty and the effectiveness are a little bit incremental, and 2) the paper requires more clarifications about the formulations and experimental descriptions (see my detailed comments).\n\nDetailed comments:\n- In the introduction, \"The GMM attention is a pure location-aware algorithm in which the model selects attention windows cumulatively without considering source contents.\": This is a little bit hard to understand. I understood it after I followed Section 2. It would be better to make this expression more understandable.\n- I'm curious about the GMM's behavior (how the Gaussian parameters would be changed for each layer and each head). These are described in the appendix, but the authors may consider discussing it in the main paper. It is well known that the general soft attentions' behaviors are very different in the lower and higher layers. The lower-layer attentions are blur and less monotonic. I would like to ask the authors to discuss how the GMM parameters can adapt to capture such behaviors in the main paper.\n- Equation (4): It's better to explain why the softplus function is used for \\Delta and \\Sigma. (monotonicity and positive constraint of the variance, right?)\n- Equation (9): It's better to have some explanation about why $\\delta$ is bounded by [0, 1].\n- Equation (14): Is there proof?\n- Section 2.4: Does the truncation make training faster?\n- Section 4: The authors should clarify why they did not use a language model. Besides, it's better to clarify that the method does not use the language model in the table's caption (Is that correct? All the results listed in the table are without LMs, right?). The readers remember the Librispeech number with the LM, and they may confuse it.\n- Why does the table 4 result not include the GMM attention result? It's better to compare the normal GMM and the proposed method in the online mode.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aVr1NevVjQ", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a monotonic attention to improve the latency of decoding. The attention is an improvement over the attention based on Gaussian mixture model, allowing the attention weights to depend on the encoder outputs. The experimental numbers are strong.\n\nI am leaning towards accepting the paper, but I do have a few concerns that the paper fails to address. Some of the concerns can be fixed, and some will need additional experiments. I will revise the score based on the authors' response.\n\nThe first concern is simply that there are errors and ambiguous statements in the paper. For example, equation (14) is obviously wrong, and in section 2.4 it is unclear how many frames the encoder needs to see before the decoder can predict the next token. The latter is crucial because streaming is the major reason behind the use of monotonic attention. A detailed list is given at the end.\n\nThe second concern is that many confounding factors are introduced in the experiments, presenting us from concluding that the improvement is solely coming from the use of SAGMM. For example, in the experiments, CTC is used as an additional loss to guide the learning of monotonic attention. During training, the paper also stochastically introduces an all one vector at the end of the input. It is unclear whether these two have an impact on learning monotonic attentions. It would be more convincing to have results without these additions, and would make the comparison to others more meaningful (assuming that others did not use the same additions).\n\nOne minor concern is that the learned representation depends on the attention used. The paper did mention this in the translation experiments, but it is under developed and no further hypotheses or evidence are provided. This would require almost a separate paper, but I believe this is key to understanding why regular content-based attention fails and whether we actually solves the problem.\n\nHere are the detailed comments.\n\n> ... the proposed attention mechanism solves the online and long-form speech recognition problems ...\n\nThis is a strong claim. I don't think the paper presents enough evidence that the problem is solved.\n\n> equation (1), (2), and (3)\n\nThis is minor, but the variables soft_\\alpha, head^h, and Multihead can be better typeset.\n\n> equation (4)\n\nThis is again minor. \\Sigma_i is a scalar, and it might be better to use the lower case \\sigma_i.\n\n> equation (13)\n\nWhat are I and J?\n\n> \\mu_i = \\mu_{i-1} + relu3(\\Delta_i)\n\nWhat is relu3?\n\n> equation (14)\n\nThis equation is wrong in many ways. I assume j \\to \\infty means having infinitely many time steps, and \\delta_j \\to 0 applies to \\delta_j for all j.\n\n> 2.4 SAGMM-tr for online inference\n\nIn streaming mode, while the frames are coming in, how does the decoder know when to produce the next token?\n\n> We tested the window width c = 15 centered by \\mu_i ...\n\nWhat is c?\n\n> ... we randomly concatenate the 1-vector after the end of source sentence with probability p_{eos} ...\n\nThis trick is not implemented in other approaches. Have the authors tried without using this trick? How do know if the improvement is purely due to this trick and not about the proposed SAGMM?\n\n> We adopt the CTC loss on the encoder output ...\n\nThe CTC loss is meant to help learn monotonic attention. Why bother using the CTC loss if the proposed attention is already monotonic? Does this suggest that the CTC loss has other added benefits?\n\n> Streamable model, bi-directional enc. in Table 2\n\nHow are bidirectional encoders in the streaming case? Using a bidirectional encoder by definition cannot be used for streaming unless I missed something.\n\n> Transformer (121M) (Ours) in Table 2\n\nI assume this uses a regular content-based attention. Please clarify. I would even group the Transformer, GMM, and SAGMM-tr rows together, since they provide the control experiments to support the usefulness of the proposed method. The other rows are less useful, since the architectures are different. The comparison of absolute numbers tell us little, except that the numbers provided for the proposed approach are in the right ballpark.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T_mB6hAshhy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces “source-aware” GMM attention and applies it to offline, online, long-form ASR.  The value of source-aware GMM attention appears to be its ability to “ignore” long segments of silence in the input audio, which could potentially be more difficult to do using other attention mechanisms.  Fairly competitive results are presented for offline ASR.  For online ASR, the results are state-of-the-art amongst sequence-to-sequence-based models. \n\nReasons for score: \nThe main contribution of this paper seems to be the addition of predicted weights for each encoder step that allow the monotonic GMM attention mechanism to ignore certain input frames.  This seems to be directly useful in ASR; however, I’m not sure if it has any direct use outside of ASR.  The paper is also fairly hard to follow and not well motivated.  Because it is so ASR specific, I feel that it would be a better fit for a speech conference.  \n\n\nHigh-level Comments:\n* Despite being a relatively minor tweak, the use of source-aware weights for each encoder step is a cool idea that integrates nicely into existing GMM-based attention mechanisms. \n* As mentioned above, I felt that the paper was fairly hard to follow and not well motivated.  It took me until Figure 2 on page 6 to realize that the main benefit of using the source-aware mechanism was to ignore long segments of silence.  It would be good to provide this motivation earlier in the paper. \n\n\nDetailed Comments:\n* I can’t figure out why there are 4 attention plots in Figure 2 (it says it’s single-head).  It might be a good idea to clarify in the caption. \n* I would recommend having your submission proof-read for English style and grammar issues.  The issues are subtle but addressing them would help to improve readability. \n* Eq. (14) seems unnecessary (it follows from basic calculus and doesn’t apply in actual usage).  Is there a reason it was included? \n* End of Section 2.2, “Moreover, the GMM attention score is uni-modal and cannot discard the non-informative tokens in attention window”: I'm not sure what is meant by \"uni-modal\" here.  A mixture of unimodal distributions (such as a GMM) is multi-modal. \n* The addition of section 4.3 felt a bit tacked-on.  To me, it doesn’t seem worthwhile to attempt to apply a monotonic attention mechanism specifically designed for ASR to a non-monotonic task like MT.  If indeed there is an interesting contribution to be made via positive constraint relaxation, I would save all of Section 4.3 for the followup paper rather than inserting it at the end of this paper. \n\nUpdate (2020-12-03):  Increasing score from 4 to 5. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "J6VnSfEbZ60", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a simple extension to the location-only monotonic GMM attention mechanism from Graves (2013), which takes the source/key context into account when computing attention weights.  The proposed method improves ASR performance over model using the baseline GMM attention which does not take source-content into account,  generalizes better to input sequences much longer than those seen during training, while also obtaining competitive performance to other streaming seq2seq ASR models on \"matched\" test sets.\n\n## Pros:\n\n1. Incorporating source content is an obvious and useful extension to monotonic GMM attention, combining the strengths of content-based approaches such as additive or dot-product attention.\n\n2. It improves performance and generalization while being simpler than existing techniques in the literature (e.g. Mocha, CTC/transducer models which have more complex loss functions).\n\n## Cons:\n\n1. The description of the proposed mechanism is inconsistent with existing literature and is very unclear and confusing in parts.\n\n2. Experiments are somewhat incomplete/missing important comparisons, e.g. comparing baseline GMM attention to the proposed \"source-aware\" variant in Tables 1,2,5, and comparing to other location-based attention mechanisms, even if non-monotinic, e.g. from http://papers.nips.cc/paper/5847-attention-based-models-for-speech-recognition.pdf\n \n3. Overall the writing/language use could use improvement.\n\n## Detailed comments\n\nAt the high level, the idea of incorporating the source keys K into GMM attention is a good one.  The proposed method seems to work, and be  simpler to implement than alternative monotonic alignment mechanisms used in seq2seq ASR models.  However, given the current state of the text, with many confusing details, I feel that the paper is not yet ready for publication without significant revisions. \n\nMany details in the paper, especially Section 2, are unclear:\n\n- Sec 2.2. and throughout the paper:  The described \"GMM\" and \"SA-GMM\" attention always use a single component, so don't really count as a Gaussian *mixture*.  Using multiple components would explicitly allow for multimodal attention weights for each output step.  Moreover, since the mixing weights are generally computed independently at each step, using multiple components makes it possible for the base GMM attention to \"discard the non-informative tokens\".   This mechanism, which would be more precisely called \"Gaussian attention\", is strictly less flexible than the base GMM attention mechanism that was originally described in Graves, 2013.\n\n- This claim is repeated in paragraph 2 of Sec 3: \"uni-modal similar to conventional GMM attention\".  When using multiple mixture components, GMM attention is not unimodal.\n\n- Eq 8: The notation here is unclear.  Why is there a softmax over $\\psi_i^h$ (a scalar AFAICT)?  Is the softmax computed over all attention heads?   Why is this necessary?  It seems to impair the training of some heads, at least for SAGMM-tr according to paragraph 4 of Sec. 4.2\n\n- Figure 1 is difficult to interpret.  The two plots have difference horizontal axes and therefore don't seem to be directly comparable...  It's not clear what the \"key width\" in Figure 1b is trying to convey since there is always going to be a single weight per (discrete) encoder step j.\n\n- Sec 2.3.  There is no particular motivation given for the proposed method for integrating source keys.  Why not include $K_j$ in the computation for the standard deviation $\\Sigma_i$ as well?  And why is the same weight $\\delta_j$ used as a scaling factor (eq (12)) and the mean offset in eq (10)?  These design choices deserve more explanation, and possibly empirical justification.\n\n- Sec 2.4: Is it possible to train SAGMM-tr from scratch?  Or does it need to be first trained using SAGMM and then fine-tuning with truncation enabled?\n\nExperiments:\n\n- Are the different GMM attention variants used encoder self-attention layers as well?  Or does the encoder use conventional \"soft attention\"?\n\n- As above, it seems unfair not to include any experiments using multiple components when comparing different variants of GMM attention.\n\n- Sec 4.2, Table 1:  Please clarify the differences between the three models labeled (Ours).  Is the difference only in the encoder-decoder attention layer?\n\nEnglish usage.  Just a few examples of grammar errors and unclear text, as there are too many to list.\n\n- page 1, \"attend subset of long sequences\" is missing a preposition, e.g., \"attend to a subset\".  It seems that \"long sequences\" is meant  to refer a single source sequence.\n\n- page 1, \"mismatch between attention parameters from decoders and information distribution in encoder outputs\".  This sentence is difficult to parse.  What is the mismatch here?   Why would the source encoding and decoder query vector need to \"match\", especially in a purely location-based attention scheme?\n\n- page 4: \"fixed length with hyperparameters\"  What are they hyperparameters being referred to here?\n- page 5, Sec. 4: \"enables early inference\".  What does \"early inference\" mean?  \n- page 5, Sec. 3: \"for the inference\" -> \"for inference\" \n- page 5, Sec 4.1:  \"1 second speeches\" -> \"1 second long utterances\"\n- page 5, Sec 4.1: \"from 30 vocabulary\" -> \"from a vocabulary of 30 words\"\n\n\nOther comments:\n\n- Sec 2.2: Sutskever et al., 2014 did not use content-based attention.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a useful incremental extension to the monotonic GMM attention by incorporating source content.
It has shown comparable performance for online and long-form speech recognition, but falls behind on the machine translation task. For online ASR, it would be more convincing to include latency comparisons across different streaming models besides WERs. 
The presentation of the paper can be further improved although it already got better based on reviewers' comments.
As in the discussion, a more accurate description of the method would be "multi-head Gaussian attention" instead of GMM attention.

The main factor for the decision is limited novelty and clarity can be further improved.
Decision: Reject