Attention Forcing for Sequence-to-sequence Model Training
Authors: Qingyun Dou
Yiting Lu
Joshua Efiong
Mark J.F. Gales
Venue:  (2020)
Keywords: deep learning
sequence-to-sequence model
attention mechanism
speech synthesis
machine translation

Abstract: Auto-regressive sequence-to-sequence models with attention mechanism have achieved state-of-the-art performance in many tasks such as machine translation and speech synthesis. These models can be difficult to train. The standard approach, teacher forcing, guides a model with reference output history during training. The problem is that the model is unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. Several approaches deal with this problem, largely by guiding the model with generated output history. To make training stable, these approaches often require a heuristic schedule or an auxiliary classifier. This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in a stable fashion, without the need for a schedule or a classifier. In addition, it allows the model to generate output sequences aligned with the references, which can be important for cascaded systems like many speech synthesis systems. Experiments on speech synthesis show that attention forcing yields significant performance gain. Experiments on machine translation show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history is challenging, while guiding the model with reference attention is beneficial.

Under review as a conference paper at ICLR 2020
ATTENTION FORCING FOR SEQUENCE-TO-SEQUENGE MODEL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Auto-regressive sequence- -ooeence----euuenee models with attention mechanism have achieved state-of-the-ar performance in many tasks such as machine translation and speech synthesis. These models can be difficult to train. The standard ap- proach, teacher forcing, guides a model with reference output history during train- ing. The problem is that the model is unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. Several approaches deal with this problem, largely by guiding the model with generated output history. To make training stable, these approaches often require a heuris- tic schedule
auxiliary classifier. This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in stable fash- ion, without the need for a schedule or classifier. In addition. it allows the model to generate output sequences aligned with the references, which can be important for cascaded systems like many speech synthesis systems. Experiments on speech synthesis show that attention forcing yields significant performance gain. Experi- ments on machine translation show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history is challenging, while guiding the model with reference attention is beneficial.
INTRODUCTION
Auto-regressive sequence-to-sequence (seq2seq) models with attention mechanism are widely used in a variety of areas including Neural Machine Translation (NMT) (Neubig) 2017 Huang et al,, 2016) and speech synthesis (Shen et al. 2018 Wang et al.) 2018), also known as Text- To--eehSpeech (TTS). These models excel at connecting sequences different length, but can be difficult to train. A standard approach is teacher forcing, which guides model with reference output history during training. This makes the model unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. One alternative is to train the model in free running mode, where the model is guided by generated output history. This approach often struggles to converge, especially for attention-based models, which need to infer the correct output and align it with the input at the same time.
Several approaches are introduced tackle the above problem, namely scheduled sampling (Bengio et al. 2015) and professor forcing (Lamb et al.) 2016). Scheduled sampling randomly decides, for each time step, whether the reference or generated output token is added the output history. The probability of choosing the reference output token decays from 1 to 0 with heuristic schedule. natural extension is sequence-level scheduled sampling, where the decision is made for each se- quence instead of token. Professor forcing views the seq2seq model as generator. During training, the generator operates both teacher forcing mode and free running mode. teacher forcing mode, it tries to maximize the standard likelihood. In free rumning mode, it tries to fool a discriminator, which is trained to tell if the model running in teacher forcing mode. To make training stable, the above approaches require either well tuned schedule, or well trained discriminator.
This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach makes training stable by decoupling the learning of the output and that of the alignment. There no need for a schedule or a discriminator. Furthermore, for cascaded systems like many TTS systems, attention forcing can be particularly useful. A model

---

Under review as a conference paper at ICLR 2020
trained with attention forcing can generate (in attention forcing mode) output sequences aligned with the references. These output sequences can be used to train a downstream model, enabling it to fix
upstream errors. The TTS experiments show that attention forcing yields significant gain in speech quality. The NMT experiments show that for tasks where various re-orderings of the output valid, guiding the model with generated output history can be problematic, while guiding the model with reference attention yields slight but consistent gain in BLEU score (Papineni et al, 2002).
SEQUENCE-TO-SEQUENCC GENERATION
Sequence-to coaennce-to-sqqeeee sequence generation can be defined as the problem of mapping an input sequence #1:L to an output sequence yI.T. From probabilistic perspective, a model 0 estimates the distribution of y given æI.L, typically product of distributions conditioned output history:
p 21:2; 0) = I 11:::91:11110 #1:L; 0)
Ideally, the model is trained through minimizing the KL-divergence between the true distribution p(yr:r ) and the estimated distribution:
0 argmin Ex, - KL
0))
0
argmin Ba..))( Pyn (((.)))))))) log (p(y1: /tt))// [11:L; 0))
0
In practice, this is approximated by minimizing the Negative Log- Likelihoodd (NLL) of some training (n) (m) LN sampled from the true distribution:
data (y vi.
① argmin Ex=1 log p(y (m)
(n) 1.1. 0)
(3)
While L and T are functions of n, the subscripts are omitted to simplify notations, i.c. Ln and T, are written as and T. At inference stage, given an input æi.L. the output Ú1.7 can be obtained through searching for the most probable sequence from the estimated distribution:
y1.T argmax p(y1i//a (i.z.
v1.r The exact search is computationally expensive, and is often approximated by greedy search if the output space is continuous, or beam search if the output space is discrete (Bengio et al. 2015).
2.1 EYTENTION-BASER SEQ2SEQ MODEL.
Attention mechanisms (Bahdanau al. 2014 Chorowski al. 2015) are commonly used con- nect sequences of different length. This paper focuses on attention- based encoder-decoder models. For these models, the probability p(y, y1 æ1:1; 0) is estimated as:
p( 1ovilvaa-1, L1.L; 0) p 11.11; ar :I.L: p(yilst,
(5)
8, = f()).-1:0.)
(6)
C 21.:
(7)
0 = 0g. a is an alignment vector (a set of attention weights). 8t is a state vector representing the output history f1:t-1, and c, a context vector summarizing #1.L for the prediction of y:- The following equations, as well as figure [ give a more detailed illustration of how a, St and e can be computed:
h1.F f 0n)
8 = f(8 1:-1; 0.)
(9)
a₁ s(s hi:1; 0n)
(10)
C, = Σt. ar
(11)
ÿt ~ p(y 81, cz; 0y)
(12)
First the encoder maps æ1:L to an encoding sequence hILL- For each decoder time step, is updated with yt-1. Based on h and 8;, the attention mechanism computes at, and then e as the weighted sum of hT.L. Finally, the decoder estimates distribution based optionally generates an output token ÿt by either sampling or taking the most probable token. Note that the output history F1---1 plays important role, as it impacts p(y St, c;; fy) through both 8, and c, Also note that there are many forms of attention-based encoder-decoder models. While attention forcing is illustrated with this particular form, it is not limited to it.
and cr, and
2

---

Under review as a conference paper at ICLR 2020
Encoder
Attention
Decoder
Figure I: Illustration of an attention-bane -based encoder-decoder model
2.2 TRAINING APPROACHES
As shown in equations 2 and 3 minimizing the Kl.-divergence between the true distribution and the model distribution can be approximated by minimizing the NLL. This motivates the approach to train the model teacher forcing mode, where p (1t--1 T1:L; 0) is computed with the correct output history 91.t-1, shown in equations 5 6
In this case, the loss can be written as:
(0) = Σn log (og((1y |1.. (n)
T logp(y: (m) 1 0) (13)
0)
This approach yields the correct model (zero KL-divergence) if the following assumptions hold: 1) the model is powerful enough 2) the model is optimized correctly; 3) there is enough training data to approximate the expectation shown equation In practice, these assumptions are often not true, hence the model is prone to make mistakes. To illustrate the problem, suppose there is reference output yirr for the test input oi. Due to data sparsity in high-dimensional space, wiz, is likely to be unseen during training. If the probability p(y 1 zi.z: 0) is wrongly estimated to be small at time step t. the probability of the reference output sequence p(yi.r [ 0) will also |
be small, i.e. it will be unlikely for the model to generate vï.--
In practice, the model can be assessed by some loss D(vi.: )1r)) between the reference out- put Vir and the generated output Ú1.7- Taking the expected value yields the Bayes risk: E Pyrropleeirree o D(yi.r (1:T). This motivates training the model with the following loss:
( = E E D "erreeeenn
D(y (n)
111 D1:r)
(14)
MN MM p( (n.m)
x 1.t: O)D())))))))) (. (()"))
(n)
((nm) is sampled from the estimated distribution P(yrrl (n) 0). is minimal when the two sequences are equal. So the model is trained to not only assign high probability to the reference
sequences in the training data, but also assign low probability to other sequences. This makes
minimum Bayes risk training prone to overfitting.
Very often, is computed at sub-sequence level. Examples include BLEU score for NMT, word еrror rate for speech recognition and root mean square error for TTS. So if an approach trains the model to predict the reference output, based on erroneous output history, it will indirectly reduce the Bayes risk. One example is to train the model in free running mode, where p(y, 31.-11 is estimated with the generated output history:
p 11 0) p(y "
T1::: p(yl cr; 8: f
(15)
(16)
9. obtained from the estimated distribution p(ye 8;, c;: 0y), as shown in equation 12 (The ap- proaches discussed in this section are designed for all auto-regressive models, with without atten- tion mechanism. So the realization C₁ is not shown.) The corresponding loss function is:
(0) ()) - -. Et logp(v)") (n) 91.1-1 11.1; (1,;;
(17)
Note that if there is enough data and modeling power, and the model is optimized correctly, the distribution I p(u v. 11.--1 0) can be the same the true distribution P(MIrT/2122) The problem with this approach is that training often struggles to converge. One concern is that the model needs learn to infer the correct output and align that with the input the same time. Therefore, several approaches, namely scheduled sampling and professor forcing, are proposed to train the model in mode between teacher forcing and free running.
Scheduled sampling (Bengio et al, 2015) randomly decides, for each time step, whether the reference generated output token is added to the output history -1:--1- For this approach,
3

---

Under review as a conference paper at ICLR 2020
p(y 21:11 0) is estimated as:
F1.1 0) Re ply M1.-1, 0ILLi))
plyeloe, cr; 0g)
(18)
8: f
(19)
y y with probability
(20)
yr with probability
1-6
gradually decays from 1 to O with heuristic schedule. Considering that during training, V1:t-1 is mostly an inconsistent mixture of the reference output and the generated oulput, natural extension
is sequence-level scheduled sampling (Bengio et al. 2013), where the decision is made for each sequence instead of token:
V1:11
with probability
(21)
V1:-1
with probability
1-c
This type training improves the results of many experiments, but sometimes leads to worse results (Wang et al.) 2017: [Bengio et al. 2015). One concern is that the decay schedule does not fit the learning pace of the model.
Professor forcing (Lamb et al., 2016) is alternative trade-off. During training, the model 0 is viewed as a generator, which generates two output sequences for each input sequence, respectively in teacher forcing mode and free running mode] For the training example {V1.7
(n) af.2. ). let (x)
denote the output generated teacher forcing mode, and y the output generated free running forcing mode, this can be expressed as:
V, Ye ((n)
~ ( (n) (n)
(22)
(x) (1).
(23)
In addition to the final output, some intermediate output sequences are saved. Let 3 and a
denote the intermediate output sequences generated respectively in teacher forcing and free running mode. These generated sequences form dataset {y1.7
(n) ( 91:1: PIT11 that is used to train 11TT-
[n) ((x))
discriminator 4b. + trained to predict the probability that a group of sequences is generated in teacher forcing mode, and the loss function is:
( = - Ew (lgg (f(r ((n) (((m) :b) + log (1 - P +1)))))))))))
+)))
(24)
While this loss function is optimized w.r.t sb, it depends on 0, hence the notation 1 For the generator 0. there are three training objectives. The first one is the standard likelihood shown in equation 13 The second one to fool the discriminator free running mode:
C (0 (b) Em= log
((m): vb)
(25)
The third one, which is optional, is to fool the discriminator in teacher forcing mode:
c (0 = En log (1 f
(() , :b))
(26)
This approach makes the distribution p(y: W1.--1 1, L1:L: 0) estimated in free rumning mode similar to the corresponding distribution / : : 0) estimated teacher forcing mode. In addition,
it regularizes some hidden layers, encouraging them to behave if in teacher forcing mode. The disadvantage is that requires designing and training the discriminator.
3
ATTENTION FORCING
3.1 GUIDING THE MODEL WITH ATTENTION
For ceanttoo-bbsedd seq2seq generation, propose new algorithm: attention forcing. The basic idea is to use reference attention (i.e. reference alignment) and generated output to guide the model during training. In attention forcing mode, the model does need learn to simultaneously infer
'The term "teacher forcing". well as "attention forcing". can refer to either an operation mode, or the approach to train a model in that operation mode. An operation mode can be used not only to train a model, but also to generate from it. For example, in teacher forcing mode, given the reference output yır., model can generate guided output Ví.r. without evaluating the loss. ví.T is likely to be different but similar to yI:r, and useful for training the discriminator.
4

---

Under review as a conference paper at ICLR 2020
Figure 2: Illustration of attention forcing
the output and align it with the input. As the reference alignment is known, the decoder focus on inferring the output, and the attention mechanism can focus on generating the correct alignment.
Let i denote the model that trained in attention forcing mode, and later used for inference. In attention forcing mode, p(y W 1i;7; 0) is estimated with the generated output ÿ1æ-1 and the reference alignment at. and equation [ becomes:
p(y 11.--1, 01:L; Rd ply plyl-r ar, FILL: I
0) se p(y (âr, éq; 0,y
(27)
8; and én denote the state vector and context vector generated by Ó. Details of attention forcing can be illustrated by figure 2 as well as the following equations:
1 1 0p.
h = f(x 0%.
(28)
8: [( 1, 11-11 1;0,)
8, f(6,-1, -t- 0,)
(29)
a: = f(; h1:1: 0.)
a, = f(8,, h1:1: 0a)
(30)
é = dlEL= cr
(31)
Yt p(yel 82, cq: 0g)
(32)
The right side of the equations forcing model i operates. h; and o denote the encoding and alignment vectors generated by ô. & computed with -1 While an alignment à; is generated by Ô, is not used by the decoder, because é is computed with the reference alignment ax,, In most cases, ar is not available. One option of obtaining it shown by the left side of equations 28 to 30 which the same as equations The option is to generate a; from teacher forcing model 6. 0 is trained in teacher forcing mode, as described in section 2.2 Once trained, it can generate or, again in teacher forcing mode.
28
30 well as equations 31 and 32 show how the attention
8 to 10
During inference, the attention foreing model operates in free running mode. In this case, equation 31 becomes é Σl= du. The decoder guided by â;, instead of a.,
During training. there are two objectives: to infer the reference output and to imitate the reference alignment. For the first objective, the loss function is:
C(S (e, θ) = Ew Σ log p(y) (n) 10 (n) 1 0,0)
(33)
For the second objective, as an alignment corresponds to categorical distribution, the loss function is the average KL-divergence between the reference alignment and the generated alignment:
((A) (0, = -. II KI(a, (1) [|a (n) = I
EL EL Oi log
(n)
(n)
(34)
2())
The two losses can be jointly optimized (AA) Ly + is a scaling factor that should be set according to the dynamic range of the two losses, which roughly indicates the norm of the gradient. The alignment loss (!A)) can be interpreted as regularization term, which encourages the attention mechanism of Ô to behave like that of O. Our default optimization option is as follows. is trained teacher forcing mode, with the loss E shown in equation 13 and then fixed to generate the reference attention. θ is trained with the joint loss C(A) In our experiments, this option makes training more stable, most probably because the reference attention is the same from epoch to epoch. There are several alternative options. One example is to tie 0 and Ô, i.c. use only one set of model parameters, and train it with the joint loss ((4) This option is less stable, but more efficient.
5

---

Under review as a conference paper at ICLR 2020
3.2 COMPARISON WITH RELATED APPROACHES
Intuitively, attention forcing, as well as scheduled sampling and professor forcing, is in the middle of teacher forcing and free running. Unlike scheduled sampling, attention forcing does not require decay schedule, which can be difficult to tune. While the scaling factor ? is hyper parameter, it can be set according to the dynamic ranges of the two losses, as described in section 3.1 In addition, can be tuned according the alignment vector, which an interpretable indicator of how well the attention mechanism works.
Beam Search Optimization (BSO) (Wiseman & Rush 2016) is an alternative approach to dealing with the discrepancy between training and inference. The basic idea is to approximate beam search during training and penalize the reference output falling off the beam. A major difference between BSO and attention forcing is that BSO is designed for tasks where the output space is discrete, so that beam search can be used. In contrast, attention forcing is agnostic to whether the output space continuous discrete.
Ranzato et al. (2013) and Bahdanau et al. (2016) introduced approaches where Reinforcement Learn- ing (RL) is adopted to deal with the discrepancy. The basic idea is pretrain the model with teacher forcing, and the refine the model with RI. For these approaches, pretraining is essential because RL can be challenging in large action spaces, which is common for many seq2seq tasks includ- ing TTS, NMT and ASR. If the RL reward is defined as the evaluation metric at inference stage, these approaches can be considered type of minimum Bayes risk training. Compared with these approaches, attention forcing is expected to be more stable as the training reward is naturally less sparse.
In terms of regularization, attention forcing is similar to professor forcing. The output layer of the attention mechanism, which can be viewed as special hidden layer, is encouraged to behave as in teacher forcing mode. The difference is that attention forcing does not require discriminator to learn a loss function, as the KL-divergence The effect of regularization on the attention mechanism has been studied in previous work (Yu) ct al. 2017: Liu et al.] 2016; [ao et al., 2018), where alternative approaches of obtaining reference attention introduced. The approaches in |ao et al. (2018) and Yu al. (2017) require collecting extra data for reference attention, and that in Liu et al. (2016) uses a statistical machine translation model to estimate them. In contrast, we propose to generate reference attention with a teacher forcing model, which can be trained simultaneously with the attention forcing model.
natural loss function for the alignment vector.
A limitation of attention forcing is that it is less general than the approaches described in section 2.2 which are well defined for all auto-regressive models, with or without attention mechanism. To apply attention forcing to a model without attention mechanism, attention needs to be defined first. For convolutional neural networks, for example, attention maps can be defined based on activation or gradient (Zagoruyko Komodakis 2016).
4 APPLICATION TO SPEECH SYNTHESIS
Attention forcing has a feature that is essential for many cascaded systems: when the reference alignment available, the output can be generated in attention forcing mode, and will be aligned with the reference. TTS is a typical example. For TTS, the task is to map sequence of characters F1LL sequence of waveform samples Wi.J. Directly mapping T1.L W1.J is difficult because the two sequences are not aligned and are orders of magnitude different in length. (10 characters can correspond to more than 1000 waveform samples.) As shown in figure TTS is often realized by first mapping 11 to a vocoder feature sequence yı.r, and then mapping yi.r to WI.J. The vocoder feature sequence is a compact and interpretable representation of the waveform; vocoder can be used to map vocoder features to waveform reversely, with a series of signal processing techniques. Each feature frame corresponds to a window of waveform samples, i.c. each time step in the feature sequence corresponds to fixed number of time steps in the waveform sequence.
The model mapping V1.L to f1.1 can be referred to as the frame-level model 0, and the model mapping F1.r to W1.J can be referred to the waveform-level model &. Conventionally, is vocoder, and is not learnable. contains a text processing frontend, a duration model and a feature model (Li et al. 2018). The text processing frontend extracts linguistic features from æ1 the

---

Under review as a conference paper at ICLR 2020
Waveform sequetice
Mad
Waveform model
Feature sequa711
Frome saoe--level mode
Cransder
Figure 3: Illustration of a speech synthesis system
duration model predicts the duration of each linguistic feature; the feature model maps the linguistic features to y1.r. This paper focuses on the state-of-the-ar approach, where 0, as well as Φ. is neural network. & can be considered a neural vocoder, which is not limited by the assumptions made by the conventional vocoders (Lorenzo-Trueed an attention- bassod seq2seq model, described in section 2.1 Compared with the conventional approach. the attention- based model has several advantages, such as performance gain and less need for data labeling (Wang et al. 2017). Note that shown in figure O learns not only to map character sequence to a feature sequence, but also to align them. In contrast, Φ does not align its input and output (Shen et al.
et al. 2018 Kalchbrenner et al. 2018). is
2018 Oord et al., 2016).
(n)) L usually contains pairs waveform w(t) and text iit (To The training dataset {w] 1.E
(n)
simplify notations, the superscript (n) be omitted by default in the following discussion.) For each WI.J. vocoder feature sequence y1:7 can be extracted. The frame-level model 0 is trained with (31.7, 3...). The waveform-level model ø can be trained with {wi. V1.T ). or {Wi.. 9i.T ). where U1.T is generated by 0. Training with ý1.r allows Φ to fix some mistakes made by 0, but this is only possible when F1:T is aligned with WI.J. To ensure the alignment, the standard approach is train in teacher forcing mode, and then generate from in the same mode. This paper proposes an alternative approach: to use attention forcing instead of teacher forcing. As analyzed in section attention forcing
3.1 training with attention forcing improves its performance. Furthermore, mode, each output Yt is predicted based on -1:-11 (instead of y1a--1). 1), hence ÿI. is more likely (than in teacher forcing mode) to contain errors that makes at inference stage. Training with V1-T enable it correct the errors, improving the quality of the waveform. Note that if 0 is trained with scheduled sampling or professor forcing, it often not possible to predict, based only on generated output history, a vocoder feature sequence aligned with the reference waveform. Also note that & is trained in teacher forcing mode, as it does not have attention mechanism. Hence the rest of this section focuses discussing training stage and inference stage.
During training, is often assumed that the output tokens follow a certain type of distribution, so that minimizing the loss c(A shown in equation 33 be approximated by minimizing some distance metric between f1.r and &1.r. For example, assuming that the distribution shown in equation 27 is
a Laplace distribution, minimizing c(AA) is equivalent to minimizing the average é distance:
argmin ((A) (0, 0) argmin E 0.0
ΣT=
1y (n)
0.." 112
(35)
0.0
9: argmax ar, Z111:
(36)
y
The notation is the same in section 3.1 denotes the attention forcing model; denotes the teacher forcing model generating reference alignment. Equation replaces equation 32 In this case, ÿr not sampled, and always the mode of the predicted distribution. During inference, the exact search (equation 4) is approximated by greedy search: (Note that for TTS, the main difference between training and inference is the alignment, which influences duration more than quality.)
Ve 9t = argmax 0 64, ai.5:0
(37)
y:
7

---

Under review as a conference paper at ICLR 2020
70.9
TeacberForcing
AllentionForcing
popres
Figure 4: Result of the listening test comparing teacher forcing and attention forcing
5 EXPERIMENTS
5.1 SPEECH SYNTHESIS
The TTS experiments are conducted on IJ dataset (Ito) from single speaker. The utterances vary in length from I to 10 seconds, totaling approximately 24 hours. A transcription is provided for each waveform, and the corresponding vocoder features extracted with PML vocoder (Degottex et al. 2016)) The training-validationn validation-test split is 13000- 50-50. The waveform-lewe model is the Hierarchical Recurrent Neural Network (HRNN) neural vocoder (Mehri al. 2016: Dou et al. 2018). The model structure is exactly the same as described in Dou et al. (018). and the model configuration is adjusted for efficiency. The frame-level model is very similar to Tacotron (Wang al. 2017). The model structure and configuration are the described in Wang et al. (2017), except that: 1) the decoder target is vocoder features; 2) the attention location-based) attention (Chorowski al. 2013); 3) each decoding step predicts 5 vocoder feature frames. The neural vocoder is always trained with teacher foreing. The frame-level model is trained with either teacher forcing or attention forcing. Details of the setup (data, models and training) are presented in appendix A.2.1
2017), which contains 13,100 utterances
as
mechanism is the hybrid (content-basedd
Two TTS systems are built: a teacher forcing system and an attention forcing system. For the teacher forcing system, the frame-level model 0 is trained in teacher forcing mode. The neural vocoder & is trained with the vocoder features generated (in teacher forcing mode) by O. For the attention forcing system, the frame-level model is trained in attention forcing mode, with reference attention generated (in teacher forcing mode) by 0. At this stage, is updated, while is fixed. The neural vocoder & is trained with the vocoder features generated (in attention forcing mode) by Ó. At inference stage, all the models operate in free-running mode.
For TTS, human perception is the gold-standard. The two systems are compared in subjective listening test. Over 30 workers from Amazon Mechanical Turk are instructed listen to pairs of utterances, and indicate which one they prefer in terms of overall quality. Each comparison includes pairs of utterances randomly selected among all the test utterances. Figure 4 shows the result of the listening test. Each number indicates the percentage of certain preference. Most participants prefer attention forcing. We strongly encourage readers to listens to the generated utterances is obvious that attention forcing yields utterances that are significantly more natural and expressive.
5.2 MACHINE TRANSLATION
The NMT experiments conducted on the English-to- -iibt--ooooettesseeeee task in IWSLT 2015. It a low resource NMT task, where training set contains 133K sentence pairs. The Stanford pre-processed data is used. The TED tst2012 is used as a validation set, and BLEU scores on TED tst2013 are reported. The scores use a 4-gram corpus level BLEU with equal weights. Google's 's iitention-basee encoder-decoder LSTM model (Wu et al. 2016) is adopted. Details of the setup (data, model and training) are presented in appendix A.2.2
Our initial experiments show that directly applying attention forcing NMT can degrade the per- formance. One concern is that for translation, various re-orderings of the output sequence are valid. In this case, guiding the model with generated output can be problematic, as the reference output can take an ordering that is different from the generated output. To see if this is the reason, we tried a modified attention forcing mode, where the model guided with reference attention and refer- ence output. The right side of equation 29 becomes: å; = ((:-1, 0.). & is computed with
"Generated test utterances are randomly selected and made available at http: //m /mi .eng. cam. ac.uk/ =g2212/cc122000 2amppppppammm11 sampl html
8

---

Under review as a conference paper at ICLR 2020
the reference output 91.t-1, and matches the reference attention <x; Other parts of attention forcing (equations] 28 to B1 stay the same, hence Dt is predicted with 91-t-1 and ax.-
In the following experiments, two NMT models compared: one is trained in teacher forcing mode, with the NLL loss in equation T3| the other is trained in the modified attention forcing mode described above, with both the NLL loss and the attention loss in equation 34 An ensemble of 10 models are trained with teacher foreing. Then each model generates reference attention for a corresponding model trained with additional attention loss. The average performance of the teacher forcing models is 26.35 BLEU, and adding the attention loss yields an average +0.35 BLEU gain. 9 of out 10 times, the performance improves. The slight but consistent gain shows that for NMT, guiding the model with generated output is indeed the cause degrading the performance. It also shows that guiding the model with reference attention can be beneficial. One possible reason that the attention loss regularizes the attention mechanism. Another is that the model does not need to learn to simultaneously infer the output and align with the input.
6 CONCLUSION
This paper introduces attention forcing, which guides a seq2seq model with generated output history and reference attention. This approach can train the model recover from its mistakes, in stable fashion, without the need for schedule or classifier. In addition, it allows the model to generate output sequences aligned with the reference output sequences, which can be important for cascaded systems like many TTS systems. The TTS experiments show that attention forcing yields significant gain in speech quality. The NMT experiments show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history can be problematic, while guiding the model with reference attention yields slight but consistent gain in BLEU score.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.047. 2014.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086. 2016.
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. arXiv preprint arXiv: 09367, 2018.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Sys- tems, pp. 1171-1179, 2015.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. bbsed models for speech recognition. In Advances in Neural Information Processing Systems, pp. 577-585, 2015.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1rXXi::412.35555 2014.
Gilles Andre Degottex, Pierre Kim Lanchantin, and Mark John Gales. A pulse model in log-domain for a uniform synthesizer. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 230-236. IEEE, 2016.
Qingyun Dou, Moquan Wan, Gilles Degottex, Zhiyi Ma, and Mark JF Gales. Hierarchical rnns for waveform-level speech synthesis. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 618-625. IEFE, 2018.
Po- Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. Ateettio-bbsse multi- modal neural machine translation. In Proceedings of the First Conference on Machine Transla- fion: Volume 2, Shared Task Papers, pp. 645, 2016.

---

Under review as a conference paper at ICLR 2020
Keith Ito. The speech dataset. httppp////leaaa.ooooooo//aaeaaaa
2017.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. arXiv preprint arXiv:1802.08445, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pp. 4601-4603 2016.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, Ming Liu, and Ming Zhou. Close to human quality Its with transformer. arXiv preprint arXiv: 1809 889.08895, 2018.
Lemao Liu, Masao Utiyama, Andrew Finch, and Eicchiro Sumita. Neural machine translation with supervised attention. In Proceedings of COLING 2016, the 26th International Confer- ence on Computational Linguistics: Technical Papers, pp. 3093-3102, Osaka, Japan, Decem- ber 2016. The COLING 2016 Organizing Committee. URL https::/ aclweb. org
anthology/CC6- C16 1291
Jaime crrenzo-Trucba, Thomas Drugman, Javier Latorre, Thomas Merritt, Bartosz Putryez, and Roberto Barra-Chicote. Robust universal neural vocoding. arXiv preprint wrXiv:181.06699 2018.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. arXiv preprint arXiv: 150..00000.0 2015.
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.008877, 2016.
Graham Neubig. Neural machine translation and preprint arXiv:1703.0160 2017.
models: tutorial. arXiv
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.0334 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 31 311-318. Association for Computational Linguistics, 2002.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint 1rXiv:1511.0673: 2015.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4779-4782 IEEE, 2018.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 100.0:1500.00387. 2015.
Yuxuan Wang. RJ Skerry- Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio. et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint
2017.
Yuxuan Wang. Daisy Stanton, Yu Zhang, RJ Skemy- Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Fei Ren, Ye Jia, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. arXiv preprint arXiv: /803.0000 2018
10

---

Under review as a conference paper at ICLR 2020
Sam Wiseman and Alexander M Rush. Sequence-to-sequees learning beam-search optimization. arXiv preprint arXiv: wew.0096600 2016.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.081 2016.
Youngjae Yu, Jongwook Choi, Yeonhwa Kim, Kyung Yoo, Sang-Hun Lee, and Gunhee Kim. Su- pervising neural attention models for video captioning by human gaze data. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 490-498, 2017.
Sergey Zagoruyko and Nikos Komodakis, Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint vrXiv:1612.03926 2016.
A APPENDIX
A.I DETAILS OF SEQUENCE- GENERATION
The exact search shown in equation is computationally expensive, and is often approximated by greedy search if the output space continuous, or beam search if the output space is discrete (Bengio et al. 2015). For greedy search, the model generates the output sequence one token at a time based on previous output tokens, until special end-of-sequence token is generated. For beam search, a heap of b best candidate sequences is kept. At each time step, the candidates are updated by extending each candidate by one step, and pruning the heap to only keep b best candidates. The beam search stops when no new sequences are added.
A.2 DETAILS OF EXPERIMENTAL SETUP
A.2.1 SPEECH SYNTHESIS
The TTS experiments are conducted on LJ dataset (to 2017). This public domain dataset contains 13,100 utterances from a single speaker reading passages from 7 non-fiction books. The utterances vary in length from to 10 seconds and have a total length of approximately 24 hours. A tran- scription (character sequence) is provided for each utterance (waveform sequence). The waveforms are resampled to 66HZ to increase the efficiency of neural vocoders. Corresponding vocoder fea- tures are extracted at the frame rate of 0.2kHz, using a PML vocoder (Degottex et al., 2016). The vraining-vllddation-tee split is 13000-50-50.
The frame-level model is very similar to Tacotron (Wang et al. 2017). powerful attention-based decoder model. The differences are: 1) the decoder target vocoder features; 2) the at- tention mechanism is the hybrid (content-based + recation-based attention described in Chorowski et al. 2015); 3) the reduction factor is 5, i.e. each decoding step predicts 5 vocoder feature frames. Apart from these, the model structure is the same as described in Wang et (201)). The input char- acters are represented as one-hot vectors. The encoder has an embedding layer mapping the one-hot vectors to continuous vectors, bottleneck layer with dropout, and a CBHG module generating the final encoding sequence. The CBHG module consists of a bank of 1-D convolutional filters, fol- lowed by highway networks (Srivastava et al., 2015) and a bidirectional GRU. The decoder has stack of GRUs with vertical residual connections and generates the intermediate vocoder features. These features are post-processed by another CBHG module, yielding the final vocoder features. The model configuration is the same as described by Table in Wang et al. (2017).
The waveform-level model is the Hierarchical Recurrent Neural Network (HRNN) neural vocoder (Mehri et al., 2016 Dou et al., 2018). The HRNN structure is a hierarchy tiers; each tier includes several neural network layers and operates at different frequency. The lowest tier operates at waveform- frequency, and outputs distributions of waveform samples. Each higher tier operates at lower frequency, and supervises the tier below it. The model configuration is as follows. Tier 0 a 4-layer DNN, including three fully connected layers with ReLU activation and a softmax output layer; the dimension is 1024 for the first two fully connected layers, and is 256 for the other two

---

Under review as a conference paper at ICLR 2020
layers. The other tiers all 1-layer RNNs; Gated Recurrent Unit (GRU) (Chung al. 2014 is used and the dimension 512 for all layers. The frequencies for tiers 0 to are respectively 16kHz, 8kHz, 2kHz and 0.4kHz. This neural vocoder models each waveform sample with categorical distribution. Hence the waveform samples are quantized into 256 integer values.
The frame-level model is trained with either teacher foreing attention forcing. In both cases, the & loss shown in equation 35 is used for both the decoder and post-processing CBHG. The two losses have equal weights. For attention forcing, the additional alignment loss shown in equation (4 is used for the attention mechanism, and the scaling factor ? is 50. The neural vocoder is always trained in teacher forcing mode, and the loss function is shown equation [3] For all experiments, the optimizer is Adam (Kingma & Ba 2014), and the initial learning rate is 0.001.
A.2.2 MACHINE TRANSLATION
The NMT experiments are conducted on the English-to-Vietmammsse Vietnamese task in IWSLT 2015. It a low resource NMT task, with the parallel training set containing 133K sentence pairs. The Stanford pre- processed data (https: //nlp.//ppppp st soodd... edu
//mt// is used. The attention- based encoder-decoder LSTM model (Wu al.) 2016) is adopted. The model is simplified with smaller number of LSTM layers due to the small scale of data: the encoder has 2 layers of bi-LSTM and the decoder has layers of uni-LSTM: the general form of Luong attention [Lunng et al.) (2015) is used; both English and Vietnamese word embeddings have 200 dimensions and are randomly initialised. Adam optimiser used with learning rate of 0.002 and the maximum gradient norm
set to be 1. Dropout is used with probability of 0.2. During inference, predictions are made using beam search with width 10.
12


Reviews:
[{"review_id": "r1gXxa_TtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for fixing exposure bias (ie. training vs generated distribution mismatch) in seq2seq modeling with attention, particularly for the application of speech synthesis where reference alignments are available.\n\n\nRelated Work is missing:\n\n- Another paper that studies fixing exposure bias in seq2seq learning:\n\nWiseman & Rush. Sequence-to-Sequence Learning as Beam-Search Optimization\nhttps://arxiv.org/pdf/1606.02960.pdf\n\n- Other papers that try to enforce attention to attend to specific locations:\n\nBao et al. Deriving Machine Attention from Human Rationales. https://arxiv.org/abs/1808.09367\n\nLiu et al. Neural Machine Translation with Supervised Attention. https://www.aclweb.org/anthology/C16-1291/\n\nYu et al. Supervising Neural Attention Models for Video Captioning by Human Gaze Data. https://arxiv.org/abs/1707.06029\n\nWithout the comparison against other related papers that also aim to supervise attention mechanisms (there are other beyond the ones I cited above)s, it is unclear how much is novel about this paper.\n\n- Furthermore, it is conceptually clear to me that attention-forcing fully matches the training vs generated distributions. The authors should describe in greater detail why this happens this, or whether these distributions are not required to fully match in attention-forcing (and in this case, why this would be desirable).\n\n- The experiments are not very convincing (only 30 human evaluators for Speech synthesis with no other quantitative evaluation, NMT results that are not particularly promising).\n\n- Use of non-anonymous github link is questionable for blinded submissions.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgSxr_k5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel training scheme for seq2seq models where attention or reference alignment is used in combination with free-running mode for improving training.\n\nThe positives of this paper are that it is well written and very clear. It also is very relevant as seq2seq models can be hard to train and techniques like scheduled sampling and x-forcing algorithms are good heuristics but heuristics none-the-less.\n\nThe downside of this paper is in the experimental results and also complexity. It would’ve been good to see a broader set of experiments to really benchmark attention-forcing from other self-attention models.\n\nAttention forcing also requires a reference or ground-truth alignment, which is often not available. Hence the authors propose to simultaneously train another teacher-forcing model to estimate the reference alignment. However, this would incur twice the computation complexity. \n\nAttention forcing could also be used in conjunction with scheduled sampling. How does that compare with the reported results for attention forcing?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJejk40KFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an alternative mechanism of training the attention values of a sequence to sequence learning model as applied to tasks like speech synthesis and translation.  During training they compute two forms of attention: (1) the standard soft-attention from a decoder fed with teacher forced output, and (2) the inference-time attention from a decoder fed with predicted outputs.  Their training objective consists of two terms: The first is the token-wise cross entropy loss but by conditioning on the predicted output  but with teacher-forced attention.  The second is a KL distance between the above two types of attention distributions.   Experiments with mechanical  turks indicate that their attention forcing mechanism is strongly preferred over the existing teacher forced output and attention model.  On translation their method provides little or no improvement.\n\nI am inclined towards rejecting the paper because the experiment and related work section still requires a lot of work before 1. The claimed utility of the idea is established, and 2. The novelty over the many existing attention architectures is established.   I elaborate on each of these next.\n\nRelated work: Recently, many papers have directly or indirectly handled the problem of exposure bias that this paper attempts to address.  The paper does not discuss most of these.  Here are some that are missed from the paper:\n\n1.   Sequence level training with recurrent neural networks\nMA Ranzato, S Chopra, M Auli, W Zaremba, 2015.\nThis paper shows that the scheduled sampling method (discussed in the paper) is much worse than a reinforce-based training mechanism of handling exposure bias.  \n\n2. An actor-critic algorithm for sequence prediction\nD Bahdanau, P Brakel, K Xu, A Goyal, R Lowe\n\n3.  Posterior Attention Models for Sequence to Sequence Learning\nS Shankar, S Sarawagi - 2019\n\n4. Latent Alignment and Variational Attention\nYuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, Alexander M. Rush 2018\n\n5. Attention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings\nShaohui Kuang, Junhui Li, António Branco, Weihua Luo, Deyi Xiong\n\nExperiments:  Their experiments are rather sketchy and limited.\nThe TTS experiments are only on one dataset.  Their method is compared only with the standard seq2seq learning approach.  Even the scheduled sampling or professor forcing methods are not compared with.  In addition, state of the art TTS methods have gained significantly from hierarchical attention.  As such as far as the TTS task is concerned the significance of the improved quality over a baseline seq2seq method is limited.\n\nFor translation they consider only the English-Vietnamese task whereas there are tens of other translation tasks that are used in recent literature.\n\nOverall, the idea proposed seems quite incremental, experiments are limited, and related work discussion incomplete.\n\n*********\nI read the author response but I do not think the paper is ready for publication yet without the thorough comparison with related work.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed an attention-forcing algorithm that guides the sequence-to-sequence model training to make it more stable. But as pointed out by the reviewers, the proposed method requires alignment which is normally unavailable. The solution to address that is using another teacher-forcing model, which can be expensive. 

The major concern about this paper is the experimental justification is not sufficient:
* lack of evaluations of the proposed method on different tasks;
* lack of experiments on understanding how it interact with existing techniques such as scheduled sampling etc;
* lack of comparisons to related existing supervised attention mechanisms. 

Decision: Reject