Local Monotonic Attention Mechanism
for End-to-End Speech and Language Processing
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura
Graduate School of Information Science
Nara Institute of Science and Technology, Japan{andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp

Abstract

modalities (i.e., text-to-text sequence (Bahdanau
et al., 2014; Sutskever et al., 2014), speech-totext sequence (Chorowski et al., 2014; Chan et al.,2016), image-to-text sequence (Xu et al., 2015),
etc).
One popular approaches in the end-to-end mapping tasks of different modalities is based on
encoder-decoder architecture. The earlier versionof an encoder-decoder model is built with only two
different components (Sutskever et al., 2014; Choet al., 2014b): (1) an encoder that processes thesource sequence and encodes them into a fixedlength vector; and (2) a decoder that produces thetarget sequence based on information from fixedlength vector given by encoder. Both the encoderand decoder are jointly trained to maximize theprobability of a correct target sequence given asource sequence. This architecture has been applied in many applications such as machine translation (Sutskever et al., 2014; Cho et al., 2014b),
image captioning (Karpathy and Fei-Fei, 2015),
and so on.
However, such architecture encounters difficulties, especially for coping with long sequences.
Because in order to generate the correct target sequence, the decoder solely depends only on thelast hidden state of the encoder. In other words,
the network needs to compress all of the information contained in the source sequence into asingle fixed-length vector. (Cho et al., 2014a)
demonstrated a decrease in the performance ofthe encoder-decoder model associated with an increase in the length of the input sentence sequence.
Therefore, (Bahdanau et al., 2014) introduced attention mechanism to address these issues. Insteadof relying on a fixed-length vector, the decoder
is assisted by the attention module to get the related context from the encoder sides, depends onthe current decoder states.
Most attention-based encoder-decoder model

Recently, encoder-decoder neural networks have shown impressive performance on many sequence-related tasks.
The architecture commonly uses an attentional mechanism which allows the model
to learn alignments between the source
and the target sequence. Most attentional
mechanisms used today is based on a
global attention property which requires
a computation of a weighted summarization of the whole input sequence generated by encoder states. However, it iscomputationally expensive and often produces misalignment on the longer input
sequence. Furthermore, it does not fit
with monotonous or left-to-right nature inseveral tasks, such as automatic speech
recognition (ASR), grapheme-to-phoneme(G2P), etc. In this paper, we propose anovel attention mechanism that has local
and monotonic properties. Various ways
to control those properties are also explored. Experimental results on ASR, G2P
and machine translation between two languages with similar sentence structures,
demonstrate that the proposed encoderdecoder model with local monotonic attention could achieve significant performance improvements and reduce the computational complexity in comparison withthe one that used the standard global attention architecture.

1

Introduction

End-to-end training is a newly emerging approachto sequence-to-sequence mapping tasks, that allows the model to directly learn the mapping between variable-length representation of different
431

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 431–440,
Taipei, Taiwan, November 27 – December 1, 2017 c 2017 AFNLP

used today has a “global” property (Bahdanau
et al., 2014; Luong et al., 2015). Every time thedecoder needs to predict the output given the previous output, it must compute a weighted summarization of the whole input sequence generated by
the encoder states. This global property allows
the decoder to address any parts of the source sequence at each step of the output generation andprovides advantages in some cases like machinetranslation tasks. Specifically, when the source
and the target languages have different sentence
structures and the last part of the target sequence
may depend on the first part of the source sequence. However, although the global attention
mechanism has often improved performance insome tasks, it is very computationally expensive.
For a case that requires mapping between long sequences, misalignments might happen in standard
attention mechanism (Kim et al., 2017). Furthermore, it does not fit with monotonous or left-toright natures in several tasks, such as ASR, G2P,
etc.
In this paper, we propose a novel attention
module that has two important characteristics toaddress those problems: local and monotonicity
properties. The local property helps our attention
module focus on certain parts from the source sequence that the decoder wants to transcribe, andthe monotonicity property strictly generates alignment left-to-right from beginning to the end of
the source sequence. In case of speech recognition task that need to produces a transcription
given the speech signal, the attention module is
now able to focus on the audio’s specific timing and always move in one direction from thestart to the end of the audio. Similar way canbe applied also for G2P or machine translation(MT) between two languages with similar sentences structure, i.e., Subject-Verb-Object (SVO)
word order in English and French languages. Experimental results demonstrate that the proposed
encoder-decoder model with local monotonic attention could achieve significant performance improvements and reduce the computational complexity in comparison with the one that used thestandard global attention architecture.

2

Figure 1: Attention-based encoder-decoder architecture.

p(y|x), where x = [x1 , ..., xS ] is the source sequence with length S and y = [y1 , ..., yT ] is thetarget sequence with length T . Figure 1 shows theoverall structure of the attention-based encoderdecoder model that consists of encoder, decoder
and attention modules.
The encoder task processes input sequence x
and outputs representative information he =[he1 , ..., heS ] for the decoder. The attention module
is an extension scheme for assisting the decoder
to find relevant information on the encoder side
based on the current decoder hidden states (Bahdanau et al., 2014; Luong et al., 2015). Usually,
attention modules produces context information ct
at the time t based on the encoder and decoder hidden states:

ct =Attention-based Encoder Decoder
Neural Network

S
X
s=1

at (s) ∗ hes(1)

at (s) = Align(hes , hdt )exp(Score(hes , hdt ))= PSe ds=1 exp(Score(hs , ht ))The encoder-decoder model is a neural network that directly models conditional probability
432

(2)

There are several variations for score functions:
e d
dot producthhs , ht i,e|e ddScore(hs , ht ) = hs Ws ht ,bilinear |edVs tanh(Ws [hs , ht ]), MLP(3)where Score : (RM × RN ) → R, M is the numberof hidden units for encoder and N is the numberof hidden units for decoder. Finally, the decodertask, which predicts the target sequence probability at time t based on previous output and context
information ct can be formulated:
log p(y|x) =TXlog p(yt |y<t , ct )

(4)

t=1

For speech recognition task, most common input x
is a sequence of feature vectors like Mel-spectral
filterbank and/or MFCC. Therefore, x ∈ RS×D
where D is the number of features and S is the total frame length for an utterance. Output y, which
is a speech transcription sequence, can be either
phoneme or grapheme (character) sequence. In
text-related task such as machine translation, x andy are a sequence of word or character indexes.

3

Figure 2: Local monotonic attention.
Part (1) of Figure 2. Assume we have source
sequence with length S, which is encoded by
the stack of Bi-LSTM (see Figure 1) into S
encoded states he = [he1 , ..., heS ]. At time t,
we want to decode the t-th target output given
the source sequence, previous output yt−1 ,and current decoder hidden states hdt ∈ RN .
In standard approaches, we use hidden states
hdt to predict the position difference ∆pt witha multilayer perceptron (MLP). We use variable ∆pt to determine how far we should
move the center of the alignment compared
to previous center pt−1 .

Locality and Monotonicity Properties

In the previous section, we explained the standard
global attention-based encoder-decoder model.
However, in order to control the area and focus
attention given previous information, such mechanism requires to apply the scoring function into
all the encoder states and normalizes them with asoftmax function. Another problem is we cannot
explicitly enforce the probability mass generated
by the current attention modules that are always
moving incrementally to the end of the source sequence. In this section, we discuss and explain
how to model the locality and monotonicity properties on the attention module. This way, we could
improve the sensitivity of capturing regularities
and ensure to focus only an important subset instead of whole sequence.
Figure 2 illustrates the overall mechanism of
our proposed local monotonic attention, and details are described blow.

In this paper, we propose two different formulations for estimating ∆pt to ensure a forward or monotonicity movement:• Constrained position prediction:
We limit maximum range from ∆pt withhyperparameter Cmax with the following equation:∆pt = Cmax ∗ sigmoid(Vp| tanh(Wp hdt ))(5)

1. Monotonicity-based Prediction of Central
PositionFirst, we define how to predict the next central position of the alignment illustrated in

Here we can control how far our
next center of alignment position pt
433

relies on our datasets and guarantee
0 ≤ ∆pt ≤ Cmax . However, it requires
us to handle hyperparameter Cmax .

alignment aSt only within [pt − 2σ, pt + 2σ]:

• Unconstrained position prediction:
Compared to a previous formulation,
since we do not limit the maximum
range of ∆pt , here we can ignore hyperparameter Cmax and use exponential (exp) function instead of sigmoid.
We can also use another function (e.gsoftplus) as long as the function satisfy
f : R → R+
0 and the result of ∆pt ≥0. We formulate unconstrained position
prediction with following equation:

Since pt is a real number and the indexes forthe encoder states are integers, we convert pt
into an integer with floor operation. After weknow the center of the position pt , we only
need to calculate the scores (Eq. 3) for each
encoder states in [pt −2σ, .., pt +2σ] then calculate the context alignment scores (Eq. 2).

∆pt = exp(Vp| tanh(Wp hdt ))aSt (s) = Align(hes , hdt ),∀s ∈ [pt − 2σ, pt + 2σ].

Compared to the standard global attention,
we can reduce the decoding computational
complexity O(T ∗ S) into O(T ∗ σ) whereσ  S and σ is constant, T is total decoding
step, S is the length of the encoder states.

(6)

3. Context Calculation
In the last step, we calculate context ct with
Salignments aN
t and at , as shown in Part (3)of Figure 2:

Here Vp ∈ RK×1 , Wp ∈ RK×N , N is thenumber of decoder hidden units and K is thenumber of hidden projection layer units. We
omit the bias for simplicity. Both equations
guarantee monotonicity properties since ∀t ∈[1..T ], pt ≥ (pt−1 + ∆pt ).

(pt +2σ)ct =(s − pt )2−
2σ 2

Overall, we can rephrase the first step as generating “prior” probabilities aNt based on the previous pt−1 position and the current decoder states.
Then the second step task generates “likelihood”
probabilities aSt by measuring the relevance of our
encoder states with the current decoder states. In
the third step, we combine our “prior” and “likelihood” probability into an unnormalized “posterior” probability at and calculate expected context
ct .

(7)


.


S
e
aN
t (s) ∗ at (s) ∗ hs (10)Context ct and current hidden state hdt will
later be utilized for calculating current output
yt .

where Vλ ∈ RK×1 . In our initial experiments, we discovered that we improved ourmodel performance by scaling with λt foreach time-step. The main objective of thisstep is to generate a scaled Gaussian distribution aNt :aNt (s) = λt ∗ exp

X

s=(pt −2σ)

Additionally, we also used scaling variable λt
to scale the unnormalized Gaussian distribution that depends on ht . We calculated λtwith following equation:
λt = exp(Vλ| tanh(Wp hdt ))

(9)

(8)

4

Experiment on Speech Recognition

We applied our proposed architecture on ASR
task. The local property helps our attention module focus on certain parts from the speech that the
decoder wants to transcribe, and the monotonicity
property strictly generates alignment left-to-rightfrom beginning to the end of the speech.

where pt is the mean and σ is the standard
deviation, both of which are used to calculate
the weighted sum from the encoder states togenerate context vector ct later. In this paper,
we treat σ as a hyperparameter.
2. Locality-based Alignment Generation
After calculating new position pt , we generate locality-based alignment, as shown in Part(2) of Figure 2. Based on predicted position
pt , we follow (Luong et al., 2015) to generate

4.1

Speech Data

We conducted our experiments on the TIMIT 1(Garofolo et al., 1993) dataset with the same set1

434

https://catalog.ldc.upenn.edu/ldc93s1

figurations follow the above descriptions, except
the baseline model that does not have an MLP for
generating ∆pt and λt .

up for training, development, and test sets as defined in the Kaldi s5 recipe (Povey et al., 2011).
The training set contains 3696 sentences from 462
speakers. We also used another sets of 50 speakers for the development set and the test set contains 192 utterances, 8 each from 24 speakers. For
every experiment, we used 40-dimensional fbank
with delta and acceleration (total 120-dimension
feature vector) extracted from the Kaldi toolkit.
The input features were normalized by subtracting the mean and divided by the standard deviationfrom the training set. For our decoder target, we
re-mapped the original target phoneme set from 61
into 39 phoneme class plus the end of sequencemark (eos).
4.2

5Result and Discussion for Speech
Recognition

Table 1 summarizes our experiments on our proposed local attention models and compares themto the baseline model using several possible scenarios.
5.1

Constrained vs Unconstrained Position
Prediction

Considering the use of constrained and unconstrained position prediction ∆pt , our results showthat the model with the unconstrained position
prediction (exp) model gives better results than
one based on the constrained position prediction(sigmoid) model on both MLP and bilinear scorers. We conclude that it is more beneficial touse the unconstrained position prediction formulation since it gives better performance and we do
not need to handle the additional hyperparameter
Cmax .

Model Architectures

On the encoder sides, we projected our input features with a linear layer with 512 hidden units followed by tanh activation function. We used threebidirectional LSTMs (Bi-LSTM) for our encoderwith 256 hidden units for each LSTM (total 512
hidden units for Bi-LSTM). To reduce the computational time, we used hierarchical subsampling(Graves, 2012; Bahdanau et al., 2016), applied it
to the top two Bi-LSTM layers, and reduced theirlength by a factor of 4.
On the decoder sides, we used a 64dimensional embedding matrix to transform the
input phonemes into a continuous vector, followedby two unidirectional LSTMs with 512 hidden
units. For every local monotonic model, we used
an MLP with 256 hidden units to generate ∆pt andλt . Hyperparameter 2σ was set to 3, and Cmax forconstrained position prediction (see Eq. 5) was set
to 5. Both hyperparameters were empirically selected and generally gave consistent results acrossvarious settings in our proposed model. For ourscorer module, we used bilinear and MLP scorers (see Eq 3) with 256 hidden units. We used
an Adam (Kingma and Ba, 2014) optimizer with alearning rate of 5e − 4.
In the recognition phase, we generated transcriptions with best-1 (greedy) search from the decoder. We did not use any language model in thiswork. All of our models were implemented on the
Chainer framework (Tokui et al., 2015).
For comparison, we evaluated our proposedmodel with the standard global attention-based
encoder-decoder model and local-m attention (Luong et al., 2015) as the baseline. Most of the con-

5.2

Alignment Scorer vs Non-Scorer

Next we investigate the importance of the scorer
module by comparing the results between a modelwith and without it. Our results reveal that, by
only relying on Gaussian alignment aNt and setaSt = 1, our model performance’s was worse than
one that used both the scorer and Gaussian alignment. This might be because the scorer modules
are able to correct the details from the Gaussianalignment based on the relevance of the encoder
states in the current decoder states. Thus, we conclude that alignment with the scorer is essential forour proposed models.
5.3

Overall comparison to the baseline

Overall, our proposed encoder-decoder modelwith local monotonic attention significantly improved the performance and reduced the computational complexity in comparison with one thatused standard global attention mechanism (wecannot compare directly with (Chorowski et al.,2014) since its pretrained with HMM state alignment). We also tried local-m attention from (Luong et al., 2015), however our model cannot converge and we hypothesize the reason is because
ratio length between the speech and their corresponding text is larger than 1, therefore the435Table 1: Results from baseline and proposed models on ASR task with TIMIT test set.
Model

Test
PER (%)

Global Attention Model (Baseline)
Att Enc-Dec (pretrained with HMM align)
18.6(Chorowski et al., 2014)
Att Enc-Dec (Pereyra et al., 2017)
23.2
Att Enc-Dec (Luo et al., 2016)
24.5
Att Enc-Dec with MLP Scorer (ours)
23.8
Att Enc-Dec with local-m (ours)(Luong et al., 2015)
Local Attention Model (Proposed)
Monotonicity
Locality
Alignment
Pos Prediction
Func.
Test
∆pt
Type
PER (%)
Score(hes , hdt )Const (sigmoid)No
23.2
Const (sigmoid)
Yes
Bilinear
21.9
Const (sigmoid)
Yes
MLP
21.7
Unconst (exp)No
23.1
Unconst (exp)
Yes
Bilinear
20.9
Unconst (exp)
Yes
MLP
21.4

∆pt cannot be represented by fixed value. The
best performance achieved by our proposed modelwith unconstrained position prediction and bilinear scorer, and provided 12.2% relative error rate
reduction to our baseline.

6

Experiment on Grapheme-to-Phoneme

We also investigated our proposed architecture on
G2P conversion task. Here, the model need togenerate corresponding phoneme given small segment of characters and its always moving from left
to right. The local property helps our attention
module focus on certain parts from the graphemesource sequence that the decoder wants to convert into phoneme, and the monotonicity property strictly generates alignment left-to-right frombeginning to the end of the grapheme source sequence.
6.1

select one of the references that has lowest PER
between compared to our hypothesis, and if the
hypothesis completely match with one of thosereferences, then the WER is not increasing. For
our encoder input, we used 26 letter (A-Z) + single quotes (’). For our decoder target, we used 39
phonemes plus the end of sequence mark (eos).
6.2

Model Architectures

On the encoder sides, the characters input were
projected into 256 dims using embedding matrix.
We used two bidirectional LSTMs (Bi-LSTM) for
our encoder with 512 hidden units for each LSTM
(total 1024 hidden units for Bi-LSTM). On the decoder sides, the phonemes input were projected
into 256 dims using embedding matrix, followedby two unidirectional LSTMs with 512 hidden
units. For local monotonic model, we used an
MLP with 256 hidden units to generate ∆pt andλt . For this task, we only used the unconstrained
formulation because based on previous sections,
we able to achieved better performance and we
didn’t need to find optimal hyperparameter for
Cmax . For our scorer module, we used MLP
scorer with 256 hidden units.
In the decoding phase, we used beam searchstrategy with beam size 3 to generate thephonemes given the character sequences. For
comparison, we evaluated our model with standard global attention and local-m attention model(Luong et al., 2015) as the baseline.
6.3

Result Discussion

Table 2 summarizes our experiment on proposed
local attention models. We compared our proposed models with several baselines from otheralgorithm as well. Our model significantly improving the PER and WER compared to encoderdecoder, attention-based global softmax and localm attention (fixed-step size). Compared to BiLSTM model which was trained with explicit
alignment, we achieve slightly better PER and
WER with larger window size (2σ = 3).

DatasetHere, we used the CMUDict dataset2 . It contains
113438 words for training and 12753 for testing(12000 unique words). For validation, we randomly select 3000 sentences from the training set.
The evaluation metrics for this task are phoneme
error rate (PER) and word error rate (WER). In theevaluation process, there are some words has multiple references (pronunciations). Therefore, we

7Experiment on Machine Translation

We also conducted experiment on machine translation task, specifically between two languages
with similar sentences structure. By using our proposed method, we able to focus only to a small
related segment on the source side and the target

2

CMUdict:
https://sourceforge.net/
projects/cmusphinx/files/G2P%20Models/phonetisaurus-cmudict-split.tar.gz

436mulation for local monotonic experiment. For ourscorer module, we used MLP scorer with 256 hidden units. In the decoding phase, we used beam
search strategy with beam size 5 and normalized
length penalty with α = 1 (Wu et al., 2016). For
comparison, we evaluate our model with standard
global attention and local-m attention model (Luoet al., 2016) as the baseline.

Table 2: Results from baseline and proposed
method on G2P task with CMUDict test set
Model
Baseline
Enc-Dec LSTM (2 lyr)
(Yao and Zweig, 2015)
Bi-LSTM (3 lyr)
(Yao and Zweig, 2015)
Att Enc-Dec with
Global MLP Scorer (ours)
Att Enc-Dec with local-m(ours) (Luong et al., 2015)
ProposedAtt Enc-Dec + Unconst (exp)(2σ = 2)Att Enc-Dec + Unconst (exp)(2σ = 3)

PER
(%)

WER
(%)

7.63

28.61

5.45

23.55

5.96

25.55

5.64

24.32

5.45

23.15

5.43

23.19

Table 3: Results from baseline and proposed
method on English-to-France and Indonesian-toEnglish translation tasks.
Model
BLEU
BTEC English to France
Baseline
Att Enc-Dec with
49.0
Global MLP Scorer
Att Enc-Dec with local-m
50.4(ours) (Luong et al., 2015)
ProposedAtt Enc-Dec + Unconst (exp)51.2(2σ = 4)Att Enc-Dec + Unconst (exp)51.1(2σ = 6)BTEC Indonesian to English
Baseline
Att Enc-Dec
38.2
with Global MLP Scorer
Att Enc-Dec with local-m
39.8(ours) (Luong et al., 2015)
ProposedAtt Enc-Dec + Unconst (exp)40.9(2σ = 4)Att Enc-Dec + Unconst (exp)41.8(2σ = 6)

generation process usually follows the source sentence structure without many reordering process.
7.1

DatasetWe used BTEC dataset (Kikui et al., 2003)
and chose English-to-France and Indonesian-toEnglish parallel corpus. From BTEC dataset, we
extracted 162318 sentences for training and 510
sentences for test data. Because there are no default development set, we randomly sampled 1000
sentences from training data for validation set. For
all language pairs, we preprocessed our dataset
using Moses (Koehn et al., 2007) tokenizer. For
training, we replaced any word that appear lessthen twice with unknown (unk) symbol. In details, we keep 10105 words for French corpus,
8265 words for English corpus and 9577 words forIndonesian corpus. We only used sentence pairs
where the source is no longer than 60 words intraining phase.
7.2

7.3

Result Discussion

Table 3 summarizes our experiment on proposed
local attention models compared to baseline global
attention model and local-m attention model (Luong et al., 2015). Generally, local monotonic attention had better result compared to global attention on both English-to-France and Indonesianto-English translation task. Our proposed model
were able to improve the BLEU up to 2.2 points on
English-to-France and 3.6 points on Indonesianto-English translation task compared to standard
global attention. Compared to local-m attention
with fixed step size, our proposed model able
to improve the performance up to 0.8 BLEU on

Model Architecture

On both encoder and decoder sides, the input
words were projected into 256 dims using embedding matrix. We used three Bi-LSTM for our encoder with 512 hidden units for each LSTM (total
1024 hidden unit for Bi-LSTM). For our decoder,we used three LSTM with 512 hidden units. For
local monotonic model, we used an MLP with 256
hidden units to generate ∆pt and λt . Same as previous section, we only used the unconstrained for437

using Bernoulli random variable to control when
the alignment should stop and generate output.
However, it cannot attend the source sequence outside the range between previous and current position. In contrast with our approach, we are able tocontrol how large the area we want to attend basedon the window size.
(Chorowski et al., 2014) also proposed a soft
constraint to encourage monotonicity by invoking
a penalty based on the current alignment and previous alignments. However, the methods still did
not guarantee a monotonicity movement of the attention.
To the best of our knowledge, only few studies have explored about local and monotonicity
properties on an attention-based model. This work
presents a novel attention module with locality andmonotonicity properties. Our proposed mechanism strictly enforces monotonicity and locality
properties in their alignment by explicitly modeling them in mathematical equations. The observation on our proposed model can also possibly act as regularizer by only observed a subset of encoder states. Here, we also explore various ways to control both properties and evaluate the impact of each variations on our proposedmodel. Experimental results also demonstrate thatthe proposed encoder-decoder model with local
monotonic attention could provide a better performances in comparison with the standard global attention architecture and local-m attention model(Luong et al., 2015).

English-to-France and 2.0 BLEU on Indonesianto-English translation task.

8

Related Work

Humans do not generally process all of the information that they encounter at once. Selective attention, which is a critical property in human perception, allows attention to be focused on particular information while filtering out a range of otherinformation. The biological structure of the eye
and the eye movement mechanism is one part ofvisual selective attention that provides the ability
to focus attention selectively on parts of the visual
space to acquire information when and where it isneeded (Rensink, 2000). In the case of the cocktail party effect, humans can selectively focus theirattentive hearing on a single speaker among various conversation and background noise sources(Cherry, 1953).
The attention mechanism in deep learning has
been studied for many years. But, only recently
have attention mechanisms made their way into
the sequence-to-sequence deep learning architectures that were proposed to solve machine translation tasks. Such mechanisms provide a model withthe ability to jointly align and translate (Bahdanau
et al., 2014). With the attention-based model, theencoder-decoder model significantly improved theperformance on machine translation (Bahdanau
et al., 2014; Luong et al., 2015) and has successfully been applied to ASR tasks (Chorowski et al.,2014; Chan et al., 2016).
However, as we mentioned earlier, most of
those attention mechanism are based on “global”
property, where the attention module tries tomatch the current hidden states with all the statesfrom the encoder sides. This approach is inefficient and computationally expensive on longer
source sequences. A “local attention” was recently introduced by (Luong et al., 2015) which
provided the capability to only focus small subsetof the encoder sides. They also proposed monotonic attention but limited to fixed step-size andnot suitable for a task where the length ratio between source and target sequence is vastly different. Our proposed method are able to elevated thisproblem by predicting the step size dynamicallyinstead of using fixed step size. After we constructed our proposed framework, we found workby (Raffel et al., 2017) recently that also proposeda method for producing monotonic alignment by

9Conclusion

This paper demonstrated a novel attention mechanism for encoder decoder model that ensuresmonotonicity and locality properties. We exploredvarious ways to control these properties, including
dynamic monotonicity-based position prediction
and locality-based alignment generation. The results reveal our proposed encoder-decoder modelwith local monotonic attention significantly improved the performance on three different tasks
and able to reduced the computational complexity
more than one that used standard global attention
architecture.

10

Acknowledgement

Part of this work was supported by JSPS
KAKENHI Grant Numbers JP17H06101 and
JP17K00237.
438

References

Suyoun Kim, Takaaki Hori, and Shinji Watanabe.
2017. Joint ctc-attention based end-to-end speech
recognition using multi-task learning. In Acoustics, Speech and Signal Processing (ICASSP), 2017
IEEE International Conference on, pages 4835–
4839. IEEE.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint
arXiv:1409.0473.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk,
Philemon Brakel, and Yoshua Bengio. 2016. Endto-end attention-based large vocabulary speech
recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 4945–4949. IEEE.

Diederik Kingma and Jimmy Ba. 2014. Adam: A
method for stochastic optimization. arXiv preprint
arXiv:1412.6980.

William Chan, Navdeep Jaitly, Quoc Le, and Oriol
Vinyals. 2016. Listen, attend and spell: A neuralnetwork for large vocabulary conversational speech
recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 4960–4964. IEEE.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, et al. 2007. Moses: Open source
toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on
interactive poster and demonstration sessions, pages
177–180. Association for Computational Linguistics.

E Colin Cherry. 1953. Some experiments on therecognition of speech, with one and with two ears.
The Journal of the acoustical society of America,
25(5):975–979.

Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, andIlya Sutskever. 2016. Learning online alignmentswith continuous rewards policy gradient. arXiv
preprint arXiv:1608.01281.

Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder–decoder
approaches. Syntax, Semantics and Structure in Statistical Translation, page 103.

Minh-Thang Luong, Hieu Pham, and Christopher D
Manning. 2015. Effective approaches to attentionbased neural machine translation. arXiv preprint
arXiv:1508.04025.
Gabriel Pereyra, George Tucker, Jan Chorowski,Łukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output
distributions. arXiv preprint arXiv:1701.06548.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. 2014b. Learningphrase representations using RNN encoder-decoder
for statistical machine translation. arXiv preprint
arXiv:1406.1078.

Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas
Burget, Ondrej Glembek, Nagendra Goel, Mirko
Hannemann, Petr Motlicek, Yanmin Qian, Petr
Schwarz, Jan Silovsky, Georg Stemmer, and Karel
Vesely. 2011. The Kaldi speech recognition toolkit.
In IEEE 2011 Workshop on Automatic Speech
Recognition and Understanding. IEEE Signal Processing Society. IEEE Catalog No.: CFP11SRWUSB.

Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho,and Yoshua Bengio. 2014. End-to-end continuous
speech recognition using attention-based recurrent
NN: First results. arXiv preprint arXiv:1412.1602.
John S Garofolo, Lori F Lamel, William M Fisher,
Jonathon G Fiscus, and David S Pallett. 1993. Darpa
TIMIT acoustic-phonetic continous speech corpus
cd-rom. NIST speech disc 1-1.1. NASA STI/Recon
technical report n, 93.

Colin Raffel, Thang Luong, Peter J Liu, Ron J Weiss,and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. arXiv
preprint arXiv:1704.00784.

Alex Graves. 2012. Supervised sequence labelling. In
Supervised Sequence Labelling with Recurrent Neural Networks, pages 5–13. Springer.

Ronald A Rensink. 2000. The dynamic representationof scenes. Visual cognition, 7(1-3):17–42.

Andrej Karpathy and Li Fei-Fei. 2015. Deep visualsemantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages
3128–3137.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence-to-Sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112.
Seiya Tokui, Kenta Oono, Shohei Hido, and Justin
Clayton. 2015. Chainer: a next-generation open
source framework for deep learning. In Proceedingsof Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on
Neural Information Processing Systems (NIPS).

Genichiro Kikui, Eiichiro Sumita, Toshiyuki
Takezawa, and Seiichi Yamamoto. 2003. Creating
corpora for speech-to-speech translation. In Eighth
European Conference on Speech Communication
and Technology.

439

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V
Le, Mohammad Norouzi, Wolfgang Macherey,
Maxim Krikun, Yuan Cao, Qin Gao, Klaus
Macherey, et al. 2016.
Google’s neural machine translation system: Bridging the gap between
human and machine translation. arXiv preprint
arXiv:1609.08144.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun
Cho, Aaron C. Courville, Ruslan Salakhutdinov,
Richard S. Zemel, and Yoshua Bengio. 2015. Show,
attend and tell: Neural image caption generation
with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, ICML
2015, Lille, France, 6-11 July 2015, pages 2048–
2057.
Kaisheng Yao and Geoffrey Zweig. 2015. Sequenceto-sequence neural net models for grapheme-tophoneme conversion. In Sixteenth Annual Conference of the International Speech Communication Association.

440