Year: 2019

Abstract:
Auto-regressive sequence-to-sequence models with attention mechanism have achieved state-of-the-art performance in many tasks such as machine translation and speech synthesis. These models can be difficult to train. The standard approach, teacher forcing, guides a model with reference output history during training. The problem is that the model is unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. Several approaches deal with this problem, largely by guiding the model with generated output history. To make training stable, these approaches often require a heuristic schedule or an auxiliary classifier. This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in a stable fashion, without the need for a schedule or a classifier. In addition, it allows the model to generate output sequences aligned with the references, which can be important for cascaded systems like many speech synthesis systems. Experiments on speech synthesis show that attention forcing yields significant performance gain. Experiments on machine translation show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history is challenging, while guiding the model with reference attention is beneficial.

heading: 1 INTRODUCTION
text: Auto-regressive sequence-to-sequence (seq2seq) models with attention mechanism are widely used in a variety of areas including Neural Machine Translation (NMT) (Neubig, 2017; Huang et al., 2016) and speech synthesis (Shen et al., 2018; Wang et al., 2018), also known as Text-To-Speech (TTS). These models excel at connecting sequences of different length, but can be difficult to train. A standard approach is teacher forcing, which guides a model with reference output history during training. This makes the model unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. One alternative is to train the model in free running mode, where the model is guided by generated output history. This approach often struggles to converge, especially for attention-based models, which need to infer the correct output and align it with the input at the same time.
Several approaches are introduced to tackle the above problem, namely scheduled sampling (Bengio et al., 2015) and professor forcing (Lamb et al., 2016). Scheduled sampling randomly decides, for each time step, whether the reference or generated output token is added to the output history. The probability of choosing the reference output token decays from 1 to 0 with a heuristic schedule. A natural extension is sequence-level scheduled sampling, where the decision is made for each sequence instead of token. Professor forcing views the seq2seq model as a generator. During training, the generator operates in both teacher forcing mode and free running mode. In teacher forcing mode, it tries to maximize the standard likelihood. In free running mode, it tries to fool a discriminator, which is trained to tell if the model is running in teacher forcing mode. To make training stable, the above approaches require either a well tuned schedule, or a well trained discriminator.
This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach makes training stable by decoupling the learning of the output and that of the alignment. There is no need for a schedule or a discriminator. Furthermore, for cascaded systems like many TTS systems, attention forcing can be particularly useful. A model
trained with attention forcing can generate (in attention forcing mode) output sequences aligned with the references. These output sequences can be used to train a downstream model, enabling it to fix some upstream errors. The TTS experiments show that attention forcing yields significant gain in speech quality. The NMT experiments show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history can be problematic, while guiding the model with reference attention yields slight but consistent gain in BLEU score (Papineni et al., 2002).

heading: 2 SEQUENCE-TO-SEQUENCE GENERATION
text: Sequence-to-sequence generation can be defined as the problem of mapping an input sequence x1:L to an output sequence y1:T . From a probabilistic perspective, a model θ estimates the distribution of y1:T given x1:L, typically as a product of distributions conditioned on output history:
p(y1:T |x1:L;θ) = ∏T t=1 p(yt|y1:t−1,x1:L;θ) (1)
Ideally, the model is trained through minimizing the KL-divergence between the true distribution p(y1:T |x1:L) and the estimated distribution:
θ̂ = argmin θ
Ex1:L∼p(x1:L)KL ( p(y1:T |x1:L)||p(y1:T |x1:L;θ) ) = argmin
θ Ex1:L∼p(x1:L)Ey1:T∼p(y1:T |x1:L) log
( p(y1:T |x1:L)/p(y1:T |x1:L;θ) ) (2) In practice, this is approximated by minimizing the Negative Log-Likelihood (NLL) of some training data {y(n)1:T ,x (n) 1:L}N1 , sampled from the true distribution:
θ̂ = argmin θ − ∑N n=1 log p(y (n) 1:T |x (n) 1:L;θ) (3)
While L and T are functions of n, the subscripts are omitted to simplify notations, i.e. Ln and Tn are written as L and T . At inference stage, given an input x∗1:L, the output ŷ1:T can be obtained through searching for the most probable sequence from the estimated distribution:
ŷ1:T = argmax y1:T
p(y1:T |x∗1:L; θ̂) (4)
The exact search is computationally expensive, and is often approximated by greedy search if the output space is continuous, or beam search if the output space is discrete (Bengio et al., 2015).

heading: 2.1 ATTENTION-BASED SEQ2SEQ MODEL
text: Attention mechanisms (Bahdanau et al., 2014; Chorowski et al., 2015) are commonly used to connect sequences of different length. This paper focuses on attention-based encoder-decoder models. For these models, the probability p(yt|y1:t−1,x1:L;θ) is estimated as:
p(yt|y1:t−1,x1:L;θ) ≈ p(yt|y1:t−1,αt,x1:L;θ) ≈ p(yt|st, ct;θy) (5) st = f(y1:t−1;θs) (6) ct = f(αt,x1:L;θc) (7)
θ = {θy,θs,θc}. αt is an alignment vector (a set of attention weights). st is a state vector representing the output history y1:t−1, and ct is a context vector summarizing x1:L for the prediction of yt. The following equations, as well as figure 1, give a more detailed illustration of how αt, st and ct can be computed:
h1:L = f(x1:L;θh) (8) st = f(st−1,yt−1;θs) (9) αt = f(st,h1:L;θα) (10)
ct = ∑L l=1 αt,lhl (11)
ŷt ∼ p(yt|st, ct;θy) (12) First the encoder maps x1:L to an encoding sequence h1:L. For each decoder time step, st is updated with yt−1. Based on h1:L and st, the attention mechanism computes αt, and then ct as the weighted sum of h1:L. Finally, the decoder estimates a distribution based on st and ct, and optionally generates an output token ŷt by either sampling or taking the most probable token. Note that the output history y1:t−1 plays an important role, as it impacts p(yt|st, ct;θy) through both st and ct. Also note that there are many forms of attention-based encoder-decoder models. While attention forcing is illustrated with this particular form, it is not limited to it.

heading: 2.2 TRAINING APPROACHES
text: As shown in equations 2 and 3, minimizing the KL-divergence between the true distribution and the model distribution can be approximated by minimizing the NLL. This motivates the approach to train the model in teacher forcing mode, where p(yt|y1:t−1,x1:L;θ) is computed with the correct output history y1:t−1, as shown in equations 5 and 6. In this case, the loss can be written as:
L(T)y (θ) = − ∑N n=1 log p(y (n) 1:T |x (n) 1:L;θ) = − ∑N n=1 ∑T t=1 log p(y (n) t |y (n) 1:t−1,x (n) 1:L;θ) (13)
This approach yields the correct model (zero KL-divergence) if the following assumptions hold: 1) the model is powerful enough ; 2) the model is optimized correctly; 3) there is enough training data to approximate the expectation shown in equation 2. In practice, these assumptions are often not true, hence the model is prone to make mistakes. To illustrate the problem, suppose there is a reference output y∗1:T for the test input x ∗ 1:L. Due to data sparsity in high-dimensional space, x ∗ 1:L is likely to be unseen during training. If the probability p(y∗t |y∗1:t−1,x∗1:L;θ) is wrongly estimated to be small at time step t, the probability of the reference output sequence p(y∗1:T |x∗1:L;θ) will also be small, i.e. it will be unlikely for the model to generate y∗1:T .
In practice, the model can be assessed by some loss D(y∗1:T , ŷ1:T ) between the reference output y∗1:T and the generated output ŷ1:T . Taking the expected value yields the Bayes risk: Eŷ1:T∼p(y1:T |x∗1:L;θ)D(y ∗ 1:T , ŷ1:T ). This motivates training the model with the following loss:
L(B)y (θ) = ∑N n=1 Eŷ1:T∼p(y1:T |x(n)1:L;θ)D(y (n) 1:T , ŷ1:T )
≈ ∑N n=1 ∑M m=1 p(ŷ (n,m) 1:T |x (n) 1:L;θ)D(y (n) 1:T , ŷ (n,m) 1:T )
(14)
ŷ(n,m) is sampled from the estimated distribution p(y1:T |x(n)1:L;θ). D is minimal when the two sequences are equal. So the model is trained to not only assign high probability to the reference sequences in the training data, but also assign low probability to other sequences. This makes minimum Bayes risk training prone to overfitting.
Very often, D is computed at sub-sequence level. Examples include BLEU score for NMT, word error rate for speech recognition and root mean square error for TTS. So if an approach trains the model to predict the reference output, based on erroneous output history, it will indirectly reduce the Bayes risk. One example is to train the model in free running mode, where p(yt|y1:t−1,x1:L;θ) is estimated with the generated output history:
p(yt|y1:t−1,x1:L;θ) ≈ p(yt|ŷ1:t−1,x1:L;θ) ≈ p(yt|st, ct;θy) (15) st = f(ŷ1:t−1;θs) (16)
ŷt is obtained from the estimated distribution p(yt|st, ct;θy), as shown in equation 12. (The approaches discussed in this section are designed for all auto-regressive models, with or without attention mechanism. So the realization ct is not shown.) The corresponding loss function is:
L(F)y (θ) = − ∑N n=1 ∑T t=1 log p(y (n) t |ŷ (n) 1:t−1,x (n) 1:L;θ) (17)
Note that if there is enough data and modeling power, and the model is optimized correctly, the distribution ∏T t=1 p(yt|ŷ1:t−1,x1:L;θ) can be the same as the true distribution p(y1:T |x1:L). The problem with this approach is that training often struggles to converge. One concern is that the model needs to learn to infer the correct output and align that with the input at the same time. Therefore, several approaches, namely scheduled sampling and professor forcing, are proposed to train the model in a mode between teacher forcing and free running.
Scheduled sampling (Bengio et al., 2015) randomly decides, for each time step, whether the reference or generated output token is added to the output history ỹ1:t−1. For this approach,
p(yt|y1:t−1,x1:L;θ) is estimated as: p(yt|y1:t−1,x1:L;θ) ≈ p(yt|ỹ1:t−1,x1:L;θ) ≈ p(yt|st, ct;θy) (18)
st = f(ỹ1:t−1;θs) (19)
ỹt = { yt with probability ŷt with probability 1− (20)
gradually decays from 1 to 0 with a heuristic schedule. Considering that during training, ỹ1:t−1 is mostly an inconsistent mixture of the reference output and the generated output, a natural extension is sequence-level scheduled sampling (Bengio et al., 2015), where the decision is made for each sequence instead of token:
ỹ1:t−1 = { y1:t−1 with probability ŷ1:t−1 with probability 1− (21)
This type of training improves the results of many experiments, but sometimes leads to worse results (Wang et al., 2017; Bengio et al., 2015). One concern is that the decay schedule does not fit the learning pace of the model.
Professor forcing (Lamb et al., 2016) is an alternative trade-off. During training, the model θ is viewed as a generator, which generates two output sequences for each input sequence, respectively in teacher forcing mode and free running mode1. For the training example {y(n)1:T ,x (n) 1:L}, let y ′(n) 1:T denote the output generated in teacher forcing mode, and ŷ(n)1:T the output generated in free running forcing mode, this can be expressed as:
∀t y′(n)t ∼ p(yt|y (n) 1:t−1,x (n) 1:L;θ) (22)
∀t ŷ(n)t ∼ p(yt|ŷ (n) 1:t−1,x (n) 1:L;θ) (23)
In addition to the final output, some intermediate output sequences are saved. Let β′(n)1:T and β̂ (n) 1:T denote the intermediate output sequences generated respectively in teacher forcing and free running mode. These generated sequences form a dataset {y′(n)1:T ,β ′(n) 1:T , ŷ (n) 1:T , β̂ (n) 1:T }N1 that is used to train a discriminator ψ. ψ is trained to predict the probability that a group of sequences is generated in teacher forcing mode, and the loss function is:
Lψ(ψ|θ) = − ∑N n=1 ( log ( f(y ′(n) 1:T ,β ′(n) 1:T ;ψ) ) + log ( 1− f(ŷ(n)1:T , β̂ (n) 1:T ;ψ) )) (24)
While this loss function is optimized w.r.t. ψ, it depends on θ, hence the notation ψ|θ. For the generator θ, there are three training objectives. The first one is the standard likelihood shown in equation 13. The second one is to fool the discriminator in free running mode:
L(F)β (θ|ψ) = − ∑N n=1 log ( f(ŷ (n) 1:T , β̂ (n) 1:T ;ψ) ) (25)
The third one, which is optional, is to fool the discriminator in teacher forcing mode: L(T)β (θ|ψ) = − ∑N n=1 log ( 1− f(y′(n)1:T ,β ′(n) 1:T ;ψ) ) (26)
This approach makes the distribution p(yt|ŷ1:t−1,x1:L;θ) estimated in free running mode similar to the corresponding distribution p(yt|y1:t−1,x1:L;θ) estimated in teacher forcing mode. In addition, it regularizes some hidden layers, encouraging them to behave as if in teacher forcing mode. The disadvantage is that it requires designing and training the discriminator.

heading: 3 ATTENTION FORCING

heading: 3.1 GUIDING THE MODEL WITH ATTENTION
text: For attention-based seq2seq generation, we propose a new algorithm: attention forcing. The basic idea is to use reference attention (i.e. reference alignment) and generated output to guide the model during training. In attention forcing mode, the model does not need to learn to simultaneously infer
1The term ”teacher forcing”, as well as ”attention forcing”, can refer to either an operation mode, or the approach to train a model in that operation mode. An operation mode can be used not only to train a model, but also to generate from it. For example, in teacher forcing mode, given the reference output y1:T , a model can generate a guided output y′1:T , without evaluating the loss. y ′ 1:T is likely to be different but similar to y1:T , and can be useful for training the discriminator.
the output and align it with the input. As the reference alignment is known, the decoder can focus on inferring the output, and the attention mechanism can focus on generating the correct alignment.
Let θ̂ denote the model that is trained in attention forcing mode, and later used for inference. In attention forcing mode, p(yt|y1:t−1,x1:L; θ̂) is estimated with the generated output ŷ1:t−1 and the reference alignment αt, and equation 5 becomes: p(yt|y1:t−1,x1:L; θ̂) ≈ p(yt|ŷ1:t−1,αt,x1:L; θ̂) ≈ p(yt|ŝt, ĉt; θ̂y) (27) ŝt and ĉt denote the state vector and context vector generated by θ̂. Details of attention forcing can be illustrated by figure 2, as well as the following equations:
h1:L = f(x1:L;θh) ĥ1:L = f(x1:L; θ̂h) (28)
st = f(st−1,yt−1;θs) ŝt = f(ŝt−1, ŷt−1; θ̂s) (29)
αt = f(st,h1:L;θα) α̂t = f(ŝt, ĥ1:L; θ̂α) (30) ĉt = ∑L l=1 αt,lĥl (31)
ŷt ∼ p(yt|ŝt, ĉt; θ̂y) (32)
The right side of the equations 28 to 30, as well as equations 31 and 32, show how the attention forcing model θ̂ operates. ĥl and α̂t denote the encoding and alignment vectors generated by θ̂. ŝt is computed with ŷ1:t−1. While an alignment α̂t is generated by θ̂, it is not used by the decoder, because ĉt is computed with the reference alignment αt. In most cases, αt is not available. One option of obtaining it is shown by the left side of equations 28 to 30, which is the same as equations 8 to 10. The option is to generate αt from a teacher forcing model θ. θ is trained in teacher forcing mode, as described in section 2.2. Once trained, it can generate αt, again in teacher forcing mode.
During inference, the attention forcing model operates in free running mode. In this case, equation 31 becomes ĉt = ∑L l=1 α̂t,lĥl. The decoder is guided by α̂t, instead of αt.
During training, there are two objectives: to infer the reference output and to imitate the reference alignment. For the first objective, the loss function is:
L(A)y (θ, θ̂) = − ∑N n=1 ∑T t=1 log p(y (n) t |ŷ (n) 1:t−1,α (n) t ,x (n) 1:L;θ, θ̂) (33)
For the second objective, as an alignment corresponds to a categorical distribution, the loss function is the average KL-divergence between the reference alignment and the generated alignment:
L(A)α (θ, θ̂) = ∑N n=1 ∑T t=1 KL(α (n) t ||α̂ (n) t ) = ∑N n=1 ∑T t=1 ∑L l=1 α (n) t,l log α (n) t,l
α̂ (n) t,l
(34)
The two losses can be jointly optimized as L(A)y,α = L(A)y + γL(A)α . γ is a scaling factor that should be set according to the dynamic range of the two losses, which roughly indicates the norm of the gradient. The alignment loss L(A)α can be interpreted as a regularization term, which encourages the attention mechanism of θ̂ to behave like that of θ. Our default optimization option is as follows. θ is trained in teacher forcing mode, with the loss L(T)y shown in equation 13, and then fixed to generate the reference attention. θ̂ is trained with the joint loss L(A)y,α. In our experiments, this option makes training more stable, most probably because the reference attention is the same from epoch to epoch. There are several alternative options. One example is to tie θ and θ̂, i.e. use only one set of model parameters, and train it with the joint loss L(A)y,α. This option is less stable, but more efficient.

heading: 3.2 COMPARISON WITH RELATED APPROACHES
text: Intuitively, attention forcing, as well as scheduled sampling and professor forcing, is in the middle of teacher forcing and free running. Unlike scheduled sampling, attention forcing does not require a decay schedule, which can be difficult to tune. While the scaling factor γ is hyper parameter, it can be set according to the dynamic ranges of the two losses, as described in section 3.1. In addition, it can be tuned according the alignment vector, which is an interpretable indicator of how well the attention mechanism works.
Beam Search Optimization (BSO) (Wiseman & Rush, 2016) is an alternative approach to dealing with the discrepancy between training and inference. The basic idea is to approximate beam search during training and penalize the reference output falling off the beam. A major difference between BSO and attention forcing is that BSO is designed for tasks where the output space is discrete, so that beam search can be used. In contrast, attention forcing is agnostic to whether the output space is continuous or discrete.
Ranzato et al. (2015) and Bahdanau et al. (2016) introduced approaches where Reinforcement Learning (RL) is adopted to deal with the discrepancy. The basic idea is to pretrain the model with teacher forcing, and the refine the model with RL. For these approaches, pretraining is essential because RL can be challenging in large action spaces, which is common for many seq2seq tasks including TTS, NMT and ASR. If the RL reward is defined as the evaluation metric at inference stage, these approaches can be considered a type of minimum Bayes risk training. Compared with these approaches, attention forcing is expected to be more stable as the training reward is naturally less sparse.
In terms of regularization, attention forcing is similar to professor forcing. The output layer of the attention mechanism, which can be viewed as a special hidden layer, is encouraged to behave as if in teacher forcing mode. The difference is that attention forcing does not require a discriminator to learn a loss function, as the KL-divergence is natural loss function for the alignment vector. The effect of regularization on the attention mechanism has been studied in previous work (Yu et al., 2017; Liu et al., 2016; Bao et al., 2018), where alternative approaches of obtaining reference attention are introduced. The approaches in Bao et al. (2018) and Yu et al. (2017) require collecting extra data for reference attention, and that in Liu et al. (2016) uses a statistical machine translation model to estimate them. In contrast, we propose to generate reference attention with a teacher forcing model, which can be trained simultaneously with the attention forcing model.
A limitation of attention forcing is that it is less general than the approaches described in section 2.2, which are well defined for all auto-regressive models, with or without attention mechanism. To apply attention forcing to a model without attention mechanism, attention needs to be defined first. For convolutional neural networks, for example, attention maps can be defined based on activation or gradient (Zagoruyko & Komodakis, 2016).

heading: 4 APPLICATION TO SPEECH SYNTHESIS
text: Attention forcing has a feature that is essential for many cascaded systems: when the reference alignment is available, the output can be generated in attention forcing mode, and will be aligned with the reference. TTS is a typical example. For TTS, the task is to map a sequence of characters x1:L to a sequence of waveform samples w1:J . Directly mapping x1:L to w1:J is difficult because the two sequences are not aligned and are orders of magnitude different in length. (10 characters can correspond to more than 1000 waveform samples.) As shown in figure 3, TTS is often realized by first mapping x1:L to a vocoder feature sequence y1:T , and then mapping y1:T to w1:J . The vocoder feature sequence is a compact and interpretable representation of the waveform; a vocoder can be used to map vocoder features to waveform or reversely, with a series of signal processing techniques. Each feature frame corresponds to a window of waveform samples, i.e. each time step in the feature sequence corresponds to a fixed number of time steps in the waveform sequence.
The model mapping x1:L to y1:T can be referred to as the frame-level model θ, and the model mapping y1:T to w1:J can be referred to as the waveform-level model φ. Conventionally, φ is a vocoder, and is not learnable. θ contains a text processing frontend, a duration model and a feature model (Li et al., 2018). The text processing frontend extracts linguistic features from x1:L; the
duration model predicts the duration of each linguistic feature; the feature model maps the linguistic features to y1:T . This paper focuses on the state-of-the-art approach, where θ, as well as φ, is a neural network. φ can be considered a neural vocoder, which is not limited by the assumptions made by the conventional vocoders (Lorenzo-Trueba et al., 2018; Kalchbrenner et al., 2018). θ is an attention-based seq2seq model, as described in section 2.1. Compared with the conventional approach, the attention-based model has several advantages, such as performance gain and less need for data labeling (Wang et al., 2017). Note that as shown in figure 3, θ learns not only to map a character sequence to a feature sequence, but also to align them. In contrast, φ does not align its input and output (Shen et al., 2018; Oord et al., 2016).
The training dataset {w(n)1:J ,x (n) 1:L}N1 usually contains pairs of waveform w (n) 1:J and text x (n) 1:L. (To simplify notations, the superscript (n) is be omitted by default in the following discussion.) For each w1:J , a vocoder feature sequence y1:T can be extracted. The frame-level model θ is trained with {y1:T ,x1:L}. The waveform-level model φ can be trained with {w1:J ,y1:T }, or {w1:J , ŷ1:T }, where ŷ1:T is generated by θ. Training with ŷ1:T allows φ to fix some mistakes made by θ, but this is only possible when ŷ1:T is aligned with w1:J . To ensure the alignment, the standard approach is to train θ in teacher forcing mode, and then generate from it in the same mode. This paper proposes an alternative approach: to use attention forcing instead of teacher forcing. As analyzed in section 3.1, training θ with attention forcing improves its performance. Furthermore, in attention forcing mode, each output ŷt is predicted based on ŷ1:t−1 (instead of y1:t−1), hence ŷ1:T is more likely (than in teacher forcing mode) to contain errors that θ makes at inference stage. Training φ with ŷ1:T can enable it to correct the errors, improving the quality of the waveform. Note that if θ is trained with scheduled sampling or professor forcing, it is often not possible to predict, based only on generated output history, a vocoder feature sequence aligned with the reference waveform. Also note that φ is trained in teacher forcing mode, as it does not have attention mechanism. Hence the rest of this section focuses on discussing θ at training stage and inference stage.
During training, it is often assumed that the output tokens follow a certain type of distribution, so that minimizing the loss L(A)y shown in equation 33 can be approximated by minimizing some distance metric between y1:T and ŷ1:T . For example, assuming that the distribution shown in equation 27 is a Laplace distribution, minimizing L(A)y is equivalent to minimizing the average `1 distance:
argmin θ,θ̂ L(A)y (θ, θ̂) ≈ argmin θ,θ̂
∑N n=1 ∑T t=1 ||y (n) t − ŷ (n) t ||1 (35)
ŷt = argmax yt
p(yt|ŷ1:t−1,αt,x1:L; θ̂) (36)
The notation is the same as in section 3.1. θ̂ denotes the attention forcing model; θ denotes the teacher forcing model generating reference alignment. Equation 36 replaces equation 32. In this case, ŷt is not sampled, and is always the mode of the predicted distribution. During inference, the exact search (equation 4) is approximated by greedy search: (Note that for TTS, the main difference between training and inference is the alignment, which influences duration more than quality.)
∀t ŷt = argmax yt p(yt|ŷ1:t−1, α̂t,x∗1:L; θ̂) (37)

heading: 5 EXPERIMENTS

heading: 5.1 SPEECH SYNTHESIS
text: The TTS experiments are conducted on LJ dataset (Ito, 2017), which contains 13,100 utterances from a single speaker. The utterances vary in length from 1 to 10 seconds, totaling approximately 24 hours. A transcription is provided for each waveform, and the corresponding vocoder features are extracted with PML vocoder (Degottex et al., 2016). The training-validation-test split is 13000- 50-50. The waveform-level model is the Hierarchical Recurrent Neural Network (HRNN) neural vocoder (Mehri et al., 2016; Dou et al., 2018). The model structure is exactly the same as described in Dou et al. (2018), and the model configuration is adjusted for efficiency. The frame-level model is very similar to Tacotron (Wang et al., 2017). The model structure and configuration are the same as described in Wang et al. (2017), except that: 1) the decoder target is vocoder features; 2) the attention mechanism is the hybrid (content-based + location-based) attention (Chorowski et al., 2015); 3) each decoding step predicts 5 vocoder feature frames. The neural vocoder is always trained with teacher forcing. The frame-level model is trained with either teacher forcing or attention forcing. Details of the setup (data, models and training) are presented in appendix A.2.1.
Two TTS systems are built: a teacher forcing system and an attention forcing system. For the teacher forcing system, the frame-level model θ is trained in teacher forcing mode. The neural vocoder φ is trained with the vocoder features generated (in teacher forcing mode) by θ. For the attention forcing system, the frame-level model θ̂ is trained in attention forcing mode, with reference attention generated (in teacher forcing mode) by θ. At this stage, θ̂ is updated, while θ is fixed. The neural vocoder φ̂ is trained with the vocoder features generated (in attention forcing mode) by θ̂. At inference stage, all the models operate in free-running mode.
For TTS, human perception is the gold-standard. The two systems are compared in a subjective listening test. Over 30 workers from Amazon Mechanical Turk are instructed to listen to pairs of utterances, and indicate which one they prefer in terms of overall quality. Each comparison includes 5 pairs of utterances randomly selected among all the test utterances. Figure 4 shows the result of the listening test. Each number indicates the percentage of a certain preference. Most participants prefer attention forcing. We strongly encourage readers to listens to the generated utterances2. It is obvious that attention forcing yields utterances that are significantly more natural and expressive.

heading: 5.2 MACHINE TRANSLATION
text: The NMT experiments are conducted on the English-to-Vietnamese task in IWSLT 2015. It is a low resource NMT task, where training set contains 133K sentence pairs. The Stanford pre-processed data is used. The TED tst2012 is used as a validation set, and BLEU scores on TED tst2013 are reported. The scores use a 4-gram corpus level BLEU with equal weights. Google’s attention-based encoder-decoder LSTM model (Wu et al., 2016) is adopted. Details of the setup (data, model and training) are presented in appendix A.2.2.
Our initial experiments show that directly applying attention forcing to NMT can degrade the performance. One concern is that for translation, various re-orderings of the output sequence are valid. In this case, guiding the model with generated output can be problematic, as the reference output can take an ordering that is different from the generated output. To see if this is the reason, we tried a modified attention forcing mode, where the model is guided with reference attention and reference output. The right side of equation 29 becomes: ŝt = f(ŝt−1,yt−1; θ̂s). ŝt is computed with
2Generated test utterances are randomly selected and made available at http://mi.eng.cam.ac.uk/ ˜qd212/iclr2020/samples.html
the reference output y1:t−1, and matches the reference attention αt Other parts of attention forcing (equations 28 to 31) stay the same, hence ŷt is predicted with y1:t−1 and αt.
In the following experiments, two NMT models are compared: one is trained in teacher forcing mode, with the NLL loss in equation 13; the other is trained in the modified attention forcing mode described above, with both the NLL loss and the attention loss in equation 34. An ensemble of 10 models are trained with teacher forcing. Then each model generates reference attention for a corresponding model trained with additional attention loss. The average performance of the teacher forcing models is 26.35 BLEU, and adding the attention loss yields an average +0.35 BLEU gain. 9 of out 10 times, the performance improves. The slight but consistent gain shows that for NMT, guiding the model with generated output is indeed the cause degrading the performance. It also shows that guiding the model with reference attention can be beneficial. One possible reason is that the attention loss regularizes the attention mechanism. Another is that the model does not need to learn to simultaneously infer the output and align it with the input.

heading: 6 CONCLUSION
text: This paper introduces attention forcing, which guides a seq2seq model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in a stable fashion, without the need for a schedule or a classifier. In addition, it allows the model to generate output sequences aligned with the reference output sequences, which can be important for cascaded systems like many TTS systems. The TTS experiments show that attention forcing yields significant gain in speech quality. The NMT experiments show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history can be problematic, while guiding the model with reference attention yields slight but consistent gain in BLEU score.

heading: A APPENDIX
text: A.1 DETAILS OF SEQUENCE-TO-SEQUENCE GENERATION
The exact search shown in equation 4 is computationally expensive, and is often approximated by greedy search if the output space is continuous, or beam search if the output space is discrete (Bengio et al., 2015). For greedy search, the model generates the output sequence one token at a time based on previous output tokens, until a special end-of-sequence token is generated. For beam search, a heap of b best candidate sequences is kept. At each time step, the candidates are updated by extending each candidate by one step, and pruning the heap to only keep b best candidates. The beam search stops when no new sequences are added.
A.2 DETAILS OF EXPERIMENTAL SETUP
A.2.1 SPEECH SYNTHESIS
The TTS experiments are conducted on LJ dataset (Ito, 2017). This public domain dataset contains 13,100 utterances from a single speaker reading passages from 7 non-fiction books. The utterances vary in length from 1 to 10 seconds and have a total length of approximately 24 hours. A transcription (character sequence) is provided for each utterance (waveform sequence). The waveforms are resampled to 16kHz to increase the efficiency of neural vocoders. Corresponding vocoder features are extracted at the frame rate of 0.2kHz, using a PML vocoder (Degottex et al., 2016). The training-validation-test split is 13000-50-50.
The frame-level model is very similar to Tacotron (Wang et al., 2017), a powerful attention-based encoder-decoder model. The differences are: 1) the decoder target is vocoder features; 2) the attention mechanism is the hybrid (content-based + location-based) attention described in Chorowski et al. (2015); 3) the reduction factor is 5, i.e. each decoding step predicts 5 vocoder feature frames. Apart from these, the model structure is the same as described in Wang et al. (2017). The input characters are represented as one-hot vectors. The encoder has an embedding layer mapping the one-hot vectors to continuous vectors, a bottleneck layer with dropout, and a CBHG module generating the final encoding sequence. The CBHG module consists of a bank of 1-D convolutional filters, followed by highway networks (Srivastava et al., 2015) and a bidirectional GRU. The decoder has a stack of GRUs with vertical residual connections and generates the intermediate vocoder features. These features are post-processed by another CBHG module, yielding the final vocoder features. The model configuration is the same as described by Table 1 in Wang et al. (2017).
The waveform-level model is the Hierarchical Recurrent Neural Network (HRNN) neural vocoder (Mehri et al., 2016; Dou et al., 2018). The HRNN structure is a hierarchy of tiers; each tier includes several neural network layers and operates at a different frequency. The lowest tier operates at waveform-level frequency, and outputs distributions of waveform samples. Each higher tier operates at a lower frequency, and supervises the tier below it. The model configuration is as follows. Tier 0 is a 4-layer DNN, including three fully connected layers with ReLU activation and a softmax output layer; the dimension is 1024 for the first two fully connected layers, and is 256 for the other two
layers. The other tiers are all 1-layer RNNs; Gated Recurrent Unit (GRU) (Chung et al., 2014) is used and the dimension is 512 for all layers. The frequencies for tiers 0 to 3 are respectively 16kHz, 8kHz, 2kHz and 0.4kHz. This neural vocoder models each waveform sample with a categorical distribution. Hence the waveform samples are quantized into 256 integer values.
The frame-level model is trained with either teacher forcing or attention forcing. In both cases, the `1 loss shown in equation 35 is used for both the decoder and post-processing CBHG. The two losses have equal weights. For attention forcing, the additional alignment loss shown in equation 34 is used for the attention mechanism, and the scaling factor γ is 50. The neural vocoder is always trained in teacher forcing mode, and the loss function is shown in equation 13. For all experiments, the optimizer is Adam (Kingma & Ba, 2014), and the initial learning rate is 0.001.
A.2.2 MACHINE TRANSLATION
The NMT experiments are conducted on the English-to-Vietnamese task in IWSLT 2015. It is a low resource NMT task, with the parallel training set containing 133K sentence pairs. The Stanford preprocessed data (https://nlp.stanford.edu/projects/nmt/) is used. The attentionbased encoder-decoder LSTM model (Wu et al., 2016) is adopted. The model is simplified with a smaller number of LSTM layers due to the small scale of data: the encoder has 2 layers of bi-LSTM and the decoder has 4 layers of uni-LSTM; the general form of Luong attention Luong et al. (2015) is used; both English and Vietnamese word embeddings have 200 dimensions and are randomly initialised. Adam optimiser is used with a learning rate of 0.002 and the maximum gradient norm is set to be 1. Dropout is used with a probability of 0.2. During inference, predictions are made using beam search with a width of 10.

Review summaries:
This paper proposes an alternative mechanism of training the attention values of a sequence to sequence learning model as applied to tasks like speech synthesis and translation.  During training they compute two forms of attention: (1) the standard soft-attention from a decoder fed with teacher forced output, and (2) the inference-time attention from a decoder fed with predicted outputs.  Their training objective consists of two terms: The first is the token-wise cross entropy loss but by conditioning on the predicted output  but with teacher-forced attention.  The second is a KL distance between the above two types of attention distributions.   Experiments with mechanical  turks indicate that their attention forcing mechanism is strongly preferred over the existing teacher forced output and attention model.  On translation their method provides little or no improvement.