End-to-End Speech Recognition with Local
Monotonic Attention
Andros Tjandra, Sakriani Sakti, Satoshi Nakamura
Graduate School of Information ScienceNara Institute of Science and Technology{andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp

Abstract
Most attention mechanism in sequence-to-sequence model is based on a global
attention property which requires a computation of a weighted summarization of
the whole input sequence generated by encoder states. However, it is computationally expensive and often produces misalignment on the longer input sequence.
Furthermore, it does not fit with monotonous or left-to-right nature in speech
recognition task. In this paper, we propose a novel attention mechanism that has
local and monotonic properties. Various ways to control those properties are also
explored. Experimental results demonstrate that encoder-decoder based ASR withlocal monotonic attention could achieve significant performance improvements
and reduce the computational complexity in comparison with the one that used thestandard global attention architecture.

1

Introduction

Recently, end-to-end ASR based model approach allows the model to directly learn the mapping
between variable-length speech to text [Chorowski et al., 2014, Chan et al., 2016]. It allow us toreplaces the conventional ASR component such as a acoustic model, a pronunciation lexicon, and alanguage model, into a single integrated neural network. There are three main modules for end-to-end
training for sequence-to-sequence task: (1) Encoder module which represent source information,(2) Decoder module which produces output sequence, (3) Attention module which help decoder toextract related information from encoder representation.
However, mostly these attention module used today has a “global" property [Bahdanau et al., 2014,
Luong et al., 2015]. Every time the decoder needs to predict the output given the previous output, it
must compute a weighted summarization of the whole input sequence generated by the encoder states.
However, although the global attention mechanism has often improved performance in some tasks, it
is very computationally expensive. Furthermore, global attention does not fit with monotonous orleft-to-right natures in speech recognition tasks and focus on the audio’s specific timing. Therefore,
the attention needs two important characteristics to address these problems: local and monotonicity
properties. The local property helps our attention module focus on certain parts that the decoder
wants to transcribe, and the monotonicity property strictly generates alignment left-to-right frombeginning to the end of speech.
In this paper, we propose a novel attention module that has local and monotonicity properties1 .
We explore various ways to control these properties on the attention-based encoder-decoder model.
Experimental results demonstrate that an encoder-decoder based ASR with local monotonic attention
significantly improved performance and reduced the computational complexity more than one thatused the standard global attention architecture.
1
The long version of this paper with title "Local Monotonic Attention Mechanism for End-to-End Speech andLanguage Processing" has been accepted at 8th International Joint Conference on Natural Language Processing(IJCNLP) 2017

31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

Figure 1: Local monotonic attention.

2

Attention-based Encoder Decoder Neural Network

The encoder-decoder model is a neural network that directly models conditional probability p(y|x),where x = [x1 , ..., xS ] is the source sequence with length S and y = [y1 , ..., yT ] is the target sequencewith length T . The encoder task processes input sequence x and outputs representative information
he = [he1 , ..., heS ] for the decoder. The attention module produces context information ct at the time t
based on the encoder and decoder hidden states:
SX
exp(Score(hes , hdt ))(1)
ct =at (s) ∗ hes
whereat (s) = Align(hes , hdt ) = PSe ds=1 exp(Score(h s , ht ))s=1
There are several variations for score functions: e dhh , h i,
 e|s t de dScore(h s , ht ) = h s W s ht ,V s| tanh(W s [he , hd ]),s t

dot productbilinear
MLP(2)where Score : (R M × RN ) → R, M is the number of hidden units for encoder and N is the number ofhidden units for decoder. Finally, the decoder task, which predicts the target sequence probability
atPTtime t based on previous output and context information ct can be formulated log p(y|x) =t=1 log p(yt |y<t , ct ).

3

Locality and Monotonicity Properties

Figure 1 illustrates the overall mechanism of our proposed local monotonic attention, and details aredescribed blow.
1. Monotonicity-based Prediction of Central Position
First, we define how to predict the next central position of the alignment illustrated in Part
(1) of Figure 1. Assume we have source sequence with length S , represented by S encodedstates he = [he1 , ..., heS ]. At time t, we want to decode the t-th target output given the source
sequence, yt−1 , and current decoder states hdt . In standard approaches, we use hidden states
hdt to predict the position difference ∆pt with a multilayer perceptron (MLP). We use variable∆pt to determine how far we should move the center of the alignment compared to previous
center pt−1 .
In this paper, we propose two different formulations for estimating ∆pt :• Constrained position prediction:
We limit maximum range from ∆pt with hyperparameter Cmax with the following
equation:|∆pt = Cmax ∗ sigmoid(V p tanh(W p hdt ))(3)
However, it requires us to handle hyperparameter Cmax .
• Unconstrained position prediction:
Compared to a previous formulation, since we do not limit the maximum range of ∆pt ,here we can ignore hyperparameter Cmax and use exponential (exp) function instead ofsigmoid. We formulate unconstrained position prediction with following equation:|∆pt = exp(V p tanh(W p hdt ))(4)2

Both equations guarantee monotonicity properties since ∀t ∈ [1..T ], pt ≥ (pt−1 + ∆pt ).
Additionally, we also used scaling variable λt to scale the unnormalized Gaussian distributionthat depends on ht . The main objective of this step is to generate a scaled Gaussiandistribution aNt in Eq.5.
!
(s − pt )2
Nat (s) = λt ∗ exp −.
(5)2σ2
where pt is the mean and σ is the standard deviation. In this paper, we treat σ as a
hyperparameter.
2. Locality-based Alignment Generation
After calculating new position pt , we generate locality-based alignment, as shown in Part(2) of Figure 1.Here, we just need calculate the scores (Eq. 2) and generate alignment aSt
only within [pt − 2σ, pt + 2σ]:aSt (s) = Align(hes , hdt ),∀s ∈ [pt − 2σ, pt + 2σ].

(6)Compared to the standard global attention, we can reduce the decoding computationalcomplexity O(T ∗ S ) into O(T ∗ σ) where σ  S and σ is constant, T is total decoding step,
S is the length of the encoder states.
3. Context Calculation
SIn the last step, we calculate context ct with alignments aN
t and at , as shown in Part (3) ofFigure 1:
ct =(pXt +2σ)
SeaNt (s) ∗ at (s) ∗ h s(7)s=(pt −2σ)Context ct and current hidden state hdt will later be utilized
Overall, we can rephrase the first step as generating “prior" probabilities aNt based on the previous pt−1
position and the current decoder states. Then the second step task generates “likelihood" probabilities
aSt by measuring the relevance of our encoder states with the current decoder states. In the third step,
we combine our “prior" and “likelihood" probability into an unnormalized “posterior" probability atand calculate expected context ct . for calculating current output yt . For more detailed explanation,please refer to the long version of our paper Tjandra et al. [2017]2 .

4Experiment and Result Discussion

We conducted our experiments on the TIMIT [Garofolo et al., 1993] dataset with the same set-up
for training, development, and test sets as defined in the Kaldi s5 recipe [Povey et al., 2011]. We
used 40-dimensional fbank as the speech features and 39 phoneme class for our target. On theencoder sides, we used a linear layer followed by three Bi-LSTM with 512 hidden units. We reduced
the source sequence length by a factor of 4 with subsampling. On the decoder sides, we used twoLSTMs with 512 hidden units. Hyperparameter σ was set to 1.5, and Cmax for constrained position
prediction (see Eq. 3) was set to 5. Both hyperparameters were empirically selected and generally
gave consistent results across various settings in our proposed model. In the recognition phase, wegenerated transcriptions with best-1 (greedy) search from the decoder. We did not use any language
model in this work to rescore our transcription result.

2

https://arxiv.org/abs/1705.08091

3

Table 1: Result from baseline and proposed models on TIMIT.
Test
PER (%)

Model

Global Attention Model (Baseline)
Att Enc-Dec [Pereyra et al., 2017]
23.2
Att Enc-Dec [Luo et al., 2016]
24.5
Att Enc-Dec with MLP Scorer (Our baseline)
23.8
Local Attention Model (Proposed)
Monotonicity
Locality
Test
Alignment
Pos Prediction
FunctionPER (%)∆ptType
Score(hes , hdt )Const (sigmoid)No23.2
Const (sigmoid)
Yes
Bilinear
21.9
Const (sigmoid)
Yes
MLP
21.7
Unconst (exp)No
23.1
Unconst (exp)
Yes
Bilinear
20.9
Unconst (exp)
Yes
MLP
21.4

In Table 1, we summarizes the experiments on our proposed local attention models and compares
them to the baseline model using several possible scenarios. We found that it is more beneficial touse the unconstrained position prediction formulation since it gives better performance and we do
not need to handle the additional hyperparameter Cmax . We also found out that the scorer functionis essential for our proposed models. Overall, our proposed encoder-decoder model significantly
improved the performance and reduced the computational complexity in comparison with one thatused standard global attention mechanism. The best performance achieved by our proposed modelwith unconstrained position prediction and bilinear scorer, and provided 12.2% relative error rate
reduction to our baseline.

5Related Works

Aharoni and Goldberg [2016] proposed a monotonic attention by introducing an additional step
symbol to control the latest attention position. However, they need the complete target alignment totrain the model. Compared to our approach, we do not require any explicit alignment. Raffel et al.
[2017] also proposed a method for producing a monotonic alignment by using Bernoulli randomvariable to control when the attention mechanism should stop and generate output. Compare to ourproposed method, we did not restrict the area of the attention window. In the other hand, it is also
possible to combine their method with ours to predict the central position pt location.

6Conclusion

This paper demonstrated a novel attention mechanism for encoder decode model that ensures monotonicity and locality properties. We explored various ways to control these properties, including
monotonicity-based position prediction and locality-based alignment generation. We test our proposed model with speech recognition task. Our result revealed that we significantly improved theperformance and reduced the computational complexity more than one that used standard global
attention architecture.

Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP17K00237.

References
R. Aharoni and Y. Goldberg. Sequence to sequence transduction with hard monotonic attention.
arXiv preprint arXiv:1611.01487, 2016.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align andtranslate. arXiv preprint arXiv:1409.0473, 2014.
W. Chan, N. Jaitly, Q. Le, and O. Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Acoustics, Speech and Signal Processing (ICASSP),
2016 IEEE International Conference on, pages 4960–4964. IEEE, 2016.
4

J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio. End-to-end continuous speech recognition using
attention-based recurrent NN: First results. arXiv preprint arXiv:1412.1602, 2014.
J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett. Darpa TIMIT acousticphonetic continous speech corpus cd-rom. NIST speech disc 1-1.1. NASA STI/Recon technicalreport n, 93, 1993.
Y. Luo, C.-C. Chiu, N. Jaitly, and I. Sutskever. Learning online alignments with continuous rewards
policy gradient. arXiv preprint arXiv:1608.01281, 2016.
M.-T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machinetranslation. arXiv preprint arXiv:1508.04025, 2015.
G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton. Regularizing neural networks bypenalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek,
Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely. The Kaldi speech recognitiontoolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. IEEE
Signal Processing Society, Dec. 2011. IEEE Catalog No.: CFP11SRW-USB.
C. Raffel, T. Luong, P. J. Liu, R. J. Weiss, and D. Eck. Online and linear-time attention by enforcing
monotonic alignments. arXiv preprint arXiv:1704.00784, 2017.
A. Tjandra, S. Sakti, and S. Nakamura. Local monotonic attention mechanism for end-to-end speech
recognition. arXiv preprint arXiv:1705.08091, 2017.

5