Title: ATTENTION FORCING FOR SEQUENCE-TO-SEQUENCE MODEL TRAINING Anonymous

Abstract: Auto-regressive sequence-to-sequence models with attention mechanism have achieved state-of-the-art performance in many tasks such as machine translation and speech synthesis. These models can be difficult to train. The standard approach, teacher forcing, guides a model with reference output history during training. The problem is that the model is unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. Several approaches deal with this problem, largely by guiding the model with generated output history. To make training stable, these approaches often require a heuristic schedule or an auxiliary classifier. This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in a stable fashion, without the need for a schedule or a classifier. In addition, it allows the model to generate output sequences aligned with the references, which can be important for cascaded systems like many speech synthesis systems. Experiments on speech synthesis show that attention forcing yields significant performance gain. Experiments on machine translation show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history is challenging, while guiding the model with reference attention is beneficial.

1. INTRODUCTION:

Auto-regressive sequence-to-sequence (seq2seq) models with attention mechanism are widely used in a variety of areas including Neural Machine Translation (NMT) (Neubig, 2017; Huang et al., 2016) and speech synthesis (Shen et al., 2018; Wang et al., 2018) , also known as Text-To-Speech (TTS). These models excel at connecting sequences of different length, but can be difficult to train. A standard approach is teacher forcing, which guides a model with reference output history during training. This makes the model unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. One alternative is to train the model in free running mode, where the model is guided by generated output history. This approach often struggles to converge, especially for attention-based models, which need to infer the correct output and align it with the input at the same time.

Several approaches are introduced to tackle the above problem, namely scheduled sampling (Bengio et al., 2015) and professor forcing (Lamb et al., 2016) . Scheduled sampling randomly decides, for each time step, whether the reference or generated output token is added to the output history. The probability of choosing the reference output token decays from 1 to 0 with a heuristic schedule. A natural extension is sequence-level scheduled sampling, where the decision is made for each sequence instead of token. Professor forcing views the seq2seq model as a generator. During training, the generator operates in both teacher forcing mode and free running mode. In teacher forcing mode, it tries to maximize the standard likelihood. In free running mode, it tries to fool a discriminator, which is trained to tell if the model is running in teacher forcing mode. To make training stable, the above approaches require either a well tuned schedule, or a well trained discriminator. This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach makes training stable by decoupling the learning of the output and that of the alignment. There is no need for a schedule or a discriminator. Furthermore, for cascaded systems like many TTS systems, attention forcing can be particularly useful. A model trained with attention forcing can generate (in attention forcing mode) output sequences aligned with the references. These output sequences can be used to train a downstream model, enabling it to fix some upstream errors. The TTS experiments show that attention forcing yields significant gain in speech quality. The NMT experiments show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history can be problematic, while guiding the model with reference attention yields slight but consistent gain in BLEU score (Papineni et al., 2002) .

2. SEQUENCE-TO-SEQUENCE GENERATION:

Sequence-to-sequence generation can be defined as the problem of mapping an input sequence x 1:L to an output sequence y 1:T . From a probabilistic perspective, a model θ estimates the distribution of y 1:T given x 1:L , typically as a product of distributions conditioned on output history:

p(y 1:T |x 1:L ; θ) = T t=1 p(y t |y 1:t-1 , x 1:L ; θ) (1)

Ideally, the model is trained through minimizing the KL-divergence between the true distribution p(y 1:T |x 1:L ) and the estimated distribution:

EQUATION

In practice, this is approximated by minimizing the Negative Log-Likelihood (NLL) of some training data {y

EQUATION

While L and T are functions of n, the subscripts are omitted to simplify notations, i.e. L n and T n are written as L and T . At inference stage, given an input x * 1:L , the output ŷ1:T can be obtained through searching for the most probable sequence from the estimated distribution:

ŷ1:T = argmax

EQUATION

The exact search is computationally expensive, and is often approximated by greedy search if the output space is continuous, or beam search if the output space is discrete (Bengio et al., 2015) .

2.1. ATTENTION-BASED SEQ2SEQ MODEL:

Attention mechanisms (Bahdanau et al., 2014; Chorowski et al., 2015) are commonly used to connect sequences of different length. This paper focuses on attention-based encoder-decoder models. For these models, the probability p(y t |y 1:t-1 , x 1:L ; θ) is estimated as:

p(y t |y 1:t-1 , x 1:L ; θ) ≈ p(y t |y 1:t-1 , α t , x 1:L ; θ) ≈ p(y t |s t , c t ; θ y ) (5) s t = f (y 1:t-1 ; θ s ) (6) c t = f (α t , x 1:L ; θ c ) (7) θ = {θ y , θ s , θ c }.

α t is an alignment vector (a set of attention weights). s t is a state vector representing the output history y 1:t-1 , and c t is a context vector summarizing x 1:L for the prediction of y t . The following equations, as well as figure 1 , give a more detailed illustration of how α t , s t and c t can be computed:

EQUATION

) First the encoder maps x 1:L to an encoding sequence h 1:L . For each decoder time step, s t is updated with y t-1 . Based on h 1:L and s t , the attention mechanism computes α t , and then c t as the weighted sum of h 1:L . Finally, the decoder estimates a distribution based on s t and c t , and optionally generates an output token ŷt by either sampling or taking the most probable token. Note that the output history y 1:t-1 plays an important role, as it impacts p(y t |s t , c t ; θ y ) through both s t and c t . Also note that there are many forms of attention-based encoder-decoder models. While attention forcing is illustrated with this particular form, it is not limited to it. As shown in equations 2 and 3, minimizing the KL-divergence between the true distribution and the model distribution can be approximated by minimizing the NLL. This motivates the approach to train the model in teacher forcing mode, where p(y t |y 1:t-1 , x 1:L ; θ) is computed with the correct output history y 1:t-1 , as shown in equations 5 and 6. In this case, the loss can be written as:

EQUATION

This approach yields the correct model (zero KL-divergence) if the following assumptions hold: 1) the model is powerful enough ; 2) the model is optimized correctly; 3) there is enough training data to approximate the expectation shown in equation 2. In practice, these assumptions are often not true, hence the model is prone to make mistakes. To illustrate the problem, suppose there is a reference output y * 1:T for the test input x * 1:L . Due to data sparsity in high-dimensional space, x * 1:L is likely to be unseen during training. If the probability p(y * t |y * 1:t-1 , x * 1:L ; θ) is wrongly estimated to be small at time step t, the probability of the reference output sequence p(y * 1:T |x * 1:L ; θ) will also be small, i.e. it will be unlikely for the model to generate y * 1:T . In practice, the model can be assessed by some loss D(y * 1:T , ŷ1:T ) between the reference output y * 1:T and the generated output ŷ1:T . Taking the expected value yields the Bayes risk: 1:L ; θ). D is minimal when the two sequences are equal. So the model is trained to not only assign high probability to the reference sequences in the training data, but also assign low probability to other sequences. This makes minimum Bayes risk training prone to overfitting.

Very often, D is computed at sub-sequence level. Examples include BLEU score for NMT, word error rate for speech recognition and root mean square error for TTS. So if an approach trains the model to predict the reference output, based on erroneous output history, it will indirectly reduce the Bayes risk. One example is to train the model in free running mode, where p(y t |y 1:t-1 , x 1:L ; θ) is estimated with the generated output history:

EQUATION

) ŷt is obtained from the estimated distribution p(y t |s t , c t ; θ y ), as shown in equation 12. (The approaches discussed in this section are designed for all auto-regressive models, with or without attention mechanism. So the realization c t is not shown.) The corresponding loss function is:

EQUATION

Note that if there is enough data and modeling power, and the model is optimized correctly, the distribution T t=1 p(y t | ŷ1:t-1 , x 1:L ; θ) can be the same as the true distribution p(y 1:T |x 1:L ). The problem with this approach is that training often struggles to converge. One concern is that the model needs to learn to infer the correct output and align that with the input at the same time. Therefore, several approaches, namely scheduled sampling and professor forcing, are proposed to train the model in a mode between teacher forcing and free running.

Scheduled sampling (Bengio et al., 2015) randomly decides, for each time step, whether the reference or generated output token is added to the output history y 1:t-1 . For this approach, p(y t |y 1:t-1 , x 1:L ; θ) is estimated as:

EQUATION

y t = y t with probability ŷt with probability 1 -

gradually decays from 1 to 0 with a heuristic schedule. Considering that during training, y 1:t-1 is mostly an inconsistent mixture of the reference output and the generated output, a natural extension is sequence-level scheduled sampling (Bengio et al., 2015) , where the decision is made for each sequence instead of token:

EQUATION

This type of training improves the results of many experiments, but sometimes leads to worse results (Wang et al., 2017; Bengio et al., 2015) . One concern is that the decay schedule does not fit the learning pace of the model.

Professor forcing (Lamb et al., 2016) is an alternative trade-off. During training, the model θ is viewed as a generator, which generates two output sequences for each input sequence, respectively in teacher forcing mode and free running mode1 . For the training example {y

(n) 1:T , x (n) 1:L }, let y (n) 1:T

denote the output generated in teacher forcing mode, and ŷ(n) 1:T the output generated in free running forcing mode, this can be expressed as:

EQUATION

In addition to the final output, some intermediate output sequences are saved. Let β

(n) 1:T and β(n) 1:T

denote the intermediate output sequences generated respectively in teacher forcing and free running mode. These generated sequences form a dataset {y

(n) 1:T , β (n) 1:T , ŷ(n) 1:T , β(n) 1:T } N

1 that is used to train a discriminator ψ. ψ is trained to predict the probability that a group of sequences is generated in teacher forcing mode, and the loss function is:

EQUATION

While this loss function is optimized w.r.t. ψ, it depends on θ, hence the notation ψ|θ. For the generator θ, there are three training objectives. The first one is the standard likelihood shown in equation 13. The second one is to fool the discriminator in free running mode:

EQUATION

The third one, which is optional, is to fool the discriminator in teacher forcing mode:

EQUATION

This approach makes the distribution p(y t | ŷ1:t-1 , x 1:L ; θ) estimated in free running mode similar to the corresponding distribution p(y t |y 1:t-1 , x 1:L ; θ) estimated in teacher forcing mode. In addition, it regularizes some hidden layers, encouraging them to behave as if in teacher forcing mode. The disadvantage is that it requires designing and training the discriminator.

3. ATTENTION FORCING:



3.1. GUIDING THE MODEL WITH ATTENTION:

For attention-based seq2seq generation, we propose a new algorithm: attention forcing. The basic idea is to use reference attention (i.e. reference alignment) and generated output to guide the model during training. In attention forcing mode, the model does not need to learn to simultaneously infer Let θ denote the model that is trained in attention forcing mode, and later used for inference. In attention forcing mode, p(y t |y 1:t-1 , x 1:L ; θ) is estimated with the generated output ŷ1:t-1 and the reference alignment α t , and equation 5 becomes: p(y t |y 1:t-1 , x 1:L ; θ) ≈ p(y t | ŷ1:t-1 , α t , x 1:L ; θ) ≈ p(y t |ŝ t , ĉt ; θy ) (27) ŝt and ĉt denote the state vector and context vector generated by θ. Details of attention forcing can be illustrated by figure 2 , as well as the following equations:

EQUATION

EQUATION

EQUATION

ŷt ∼ p(y t |ŝ t , ĉt ; θy ) (32)

The right side of the equations 28 to 30, as well as equations 31 and 32, show how the attention forcing model θ operates. ĥl and αt denote the encoding and alignment vectors generated by θ. ŝt is computed with ŷ1:t-1 . While an alignment αt is generated by θ, it is not used by the decoder, because ĉt is computed with the reference alignment α t . In most cases, α t is not available. One option of obtaining it is shown by the left side of equations 28 to 30, which is the same as equations 8 to 10. The option is to generate α t from a teacher forcing model θ. θ is trained in teacher forcing mode, as described in section 2.2. Once trained, it can generate α t , again in teacher forcing mode.

During inference, the attention forcing model operates in free running mode. In this case, equation 31 becomes ĉt = L l=1 αt,l ĥl . The decoder is guided by αt , instead of α t . During training, there are two objectives: to infer the reference output and to imitate the reference alignment. For the first objective, the loss function is:

EQUATION

1:L ; θ, θ) (33) For the second objective, as an alignment corresponds to a categorical distribution, the loss function is the average KL-divergence between the reference alignment and the generated alignment:

EQUATION

The two losses can be jointly optimized as

L (A) y,α = L (A) y + γL (A)

α . γ is a scaling factor that should be set according to the dynamic range of the two losses, which roughly indicates the norm of the gradient. The alignment loss L (A) α can be interpreted as a regularization term, which encourages the attention mechanism of θ to behave like that of θ. Our default optimization option is as follows. θ is trained in teacher forcing mode, with the loss L (T) y shown in equation 13, and then fixed to generate the reference attention. θ is trained with the joint loss L y,α . This option is less stable, but more efficient.

3.2. COMPARISON WITH RELATED APPROACHES:

Intuitively, attention forcing, as well as scheduled sampling and professor forcing, is in the middle of teacher forcing and free running. Unlike scheduled sampling, attention forcing does not require a decay schedule, which can be difficult to tune. While the scaling factor γ is hyper parameter, it can be set according to the dynamic ranges of the two losses, as described in section 3.1. In addition, it can be tuned according the alignment vector, which is an interpretable indicator of how well the attention mechanism works.

Beam Search Optimization (BSO) (Wiseman & Rush, 2016) is an alternative approach to dealing with the discrepancy between training and inference. The basic idea is to approximate beam search during training and penalize the reference output falling off the beam. A major difference between BSO and attention forcing is that BSO is designed for tasks where the output space is discrete, so that beam search can be used. In contrast, attention forcing is agnostic to whether the output space is continuous or discrete. Bahdanau et al. (2016) introduced approaches where Reinforcement Learning (RL) is adopted to deal with the discrepancy. The basic idea is to pretrain the model with teacher forcing, and the refine the model with RL. For these approaches, pretraining is essential because RL can be challenging in large action spaces, which is common for many seq2seq tasks including TTS, NMT and ASR. If the RL reward is defined as the evaluation metric at inference stage, these approaches can be considered a type of minimum Bayes risk training. Compared with these approaches, attention forcing is expected to be more stable as the training reward is naturally less sparse.

Ranzato et al. (2015) and:

In terms of regularization, attention forcing is similar to professor forcing. The output layer of the attention mechanism, which can be viewed as a special hidden layer, is encouraged to behave as if in teacher forcing mode. The difference is that attention forcing does not require a discriminator to learn a loss function, as the KL-divergence is natural loss function for the alignment vector. The effect of regularization on the attention mechanism has been studied in previous work (Yu et al., 2017; Liu et al., 2016; Bao et al., 2018) , where alternative approaches of obtaining reference attention are introduced. The approaches in Bao et al. (2018) and Yu et al. (2017) require collecting extra data for reference attention, and that in Liu et al. (2016) uses a statistical machine translation model to estimate them. In contrast, we propose to generate reference attention with a teacher forcing model, which can be trained simultaneously with the attention forcing model.

A limitation of attention forcing is that it is less general than the approaches described in section 2.2, which are well defined for all auto-regressive models, with or without attention mechanism. To apply attention forcing to a model without attention mechanism, attention needs to be defined first. For convolutional neural networks, for example, attention maps can be defined based on activation or gradient (Zagoruyko & Komodakis, 2016) .

4. APPLICATION TO SPEECH SYNTHESIS:

Attention forcing has a feature that is essential for many cascaded systems: when the reference alignment is available, the output can be generated in attention forcing mode, and will be aligned with the reference. TTS is a typical example. For TTS, the task is to map a sequence of characters x 1:L to a sequence of waveform samples w 1:J . Directly mapping x 1:L to w 1:J is difficult because the two sequences are not aligned and are orders of magnitude different in length. (10 characters can correspond to more than 1000 waveform samples.) As shown in figure 3 , TTS is often realized by first mapping x 1:L to a vocoder feature sequence y 1:T , and then mapping y 1:T to w 1:J . The vocoder feature sequence is a compact and interpretable representation of the waveform; a vocoder can be used to map vocoder features to waveform or reversely, with a series of signal processing techniques. Each feature frame corresponds to a window of waveform samples, i.e. each time step in the feature sequence corresponds to a fixed number of time steps in the waveform sequence.

The model mapping x 1:L to y 1:T can be referred to as the frame-level model θ, and the model mapping y 1:T to w 1:J can be referred to as the waveform-level model φ. Conventionally, φ is a vocoder, and is not learnable. θ contains a text processing frontend, a duration model and a feature model (Li et al., 2018) . The text processing frontend extracts linguistic features from x 1:L ; the Figure 3 : Illustration of a speech synthesis system duration model predicts the duration of each linguistic feature; the feature model maps the linguistic features to y 1:T . This paper focuses on the state-of-the-art approach, where θ, as well as φ, is a neural network. φ can be considered a neural vocoder, which is not limited by the assumptions made by the conventional vocoders (Lorenzo-Trueba et al., 2018; Kalchbrenner et al., 2018) . θ is an attention-based seq2seq model, as described in section 2.1. Compared with the conventional approach, the attention-based model has several advantages, such as performance gain and less need for data labeling (Wang et al., 2017) . Note that as shown in figure 3 , θ learns not only to map a character sequence to a feature sequence, but also to align them. In contrast, φ does not align its input and output (Shen et al., 2018; Oord et al., 2016) .

The training dataset {w

(n) 1:J , x (n) 1:L } N

1 usually contains pairs of waveform w

(n) 1:J and text x

(n) 1:L . (To simplify notations, the superscript (n) is be omitted by default in the following discussion.) For each w 1:J , a vocoder feature sequence y 1:T can be extracted. The frame-level model θ is trained with {y 1:T , x 1:L }. The waveform-level model φ can be trained with {w 1:J , y 1:T }, or {w 1:J , ŷ1:T }, where ŷ1:T is generated by θ. Training with ŷ1:T allows φ to fix some mistakes made by θ, but this is only possible when ŷ1:T is aligned with w 1:J . To ensure the alignment, the standard approach is to train θ in teacher forcing mode, and then generate from it in the same mode. This paper proposes an alternative approach: to use attention forcing instead of teacher forcing. As analyzed in section 3.1, training θ with attention forcing improves its performance. Furthermore, in attention forcing mode, each output ŷt is predicted based on ŷ1:t-1 (instead of y 1:t-1 ), hence ŷ1:T is more likely (than in teacher forcing mode) to contain errors that θ makes at inference stage. Training φ with ŷ1:T can enable it to correct the errors, improving the quality of the waveform. Note that if θ is trained with scheduled sampling or professor forcing, it is often not possible to predict, based only on generated output history, a vocoder feature sequence aligned with the reference waveform. Also note that φ is trained in teacher forcing mode, as it does not have attention mechanism. Hence the rest of this section focuses on discussing θ at training stage and inference stage.

During training, it is often assumed that the output tokens follow a certain type of distribution, so that minimizing the loss L (A) y shown in equation 33 can be approximated by minimizing some distance metric between y 1:T and ŷ1:T . For example, assuming that the distribution shown in equation 27 is a Laplace distribution, minimizing L (A) y is equivalent to minimizing the average 1 distance:

EQUATION

The notation is the same as in section 3.1. θ denotes the attention forcing model; θ denotes the teacher forcing model generating reference alignment. Equation 36 replaces equation 32. In this case, ŷt is not sampled, and is always the mode of the predicted distribution. During inference, the exact search (equation 4) is approximated by greedy search: (Note that for TTS, the main difference between training and inference is the alignment, which influences duration more than quality.) The TTS experiments are conducted on LJ dataset (Ito, 2017) , which contains 13,100 utterances from a single speaker. The utterances vary in length from 1 to 10 seconds, totaling approximately 24 hours. A transcription is provided for each waveform, and the corresponding vocoder features are extracted with PML vocoder (Degottex et al., 2016) . The training-validation-test split is 13000-50-50. The waveform-level model is the Hierarchical Recurrent Neural Network (HRNN) neural vocoder (Mehri et al., 2016; Dou et al., 2018) . The model structure is exactly the same as described in Dou et al. (2018) , and the model configuration is adjusted for efficiency. The frame-level model is very similar to Tacotron (Wang et al., 2017) . The model structure and configuration are the same as described in Wang et al. (2017) , except that: 1) the decoder target is vocoder features; 2) the attention mechanism is the hybrid (content-based + location-based) attention (Chorowski et al., 2015) ; 3) each decoding step predicts 5 vocoder feature frames. The neural vocoder is always trained with teacher forcing. The frame-level model is trained with either teacher forcing or attention forcing. Details of the setup (data, models and training) are presented in appendix A.2.1.

EQUATION

Two TTS systems are built: a teacher forcing system and an attention forcing system. For the teacher forcing system, the frame-level model θ is trained in teacher forcing mode. The neural vocoder φ is trained with the vocoder features generated (in teacher forcing mode) by θ. For the attention forcing system, the frame-level model θ is trained in attention forcing mode, with reference attention generated (in teacher forcing mode) by θ. At this stage, θ is updated, while θ is fixed. The neural vocoder φ is trained with the vocoder features generated (in attention forcing mode) by θ. At inference stage, all the models operate in free-running mode.

For TTS, human perception is the gold-standard. The two systems are compared in a subjective listening test. Over 30 workers from Amazon Mechanical Turk are instructed to listen to pairs of utterances, and indicate which one they prefer in terms of overall quality. Each comparison includes 5 pairs of utterances randomly selected among all the test utterances. Figure 4 shows the result of the listening test. Each number indicates the percentage of a certain preference. Most participants prefer attention forcing. We strongly encourage readers to listens to the generated utterances2 . It is obvious that attention forcing yields utterances that are significantly more natural and expressive.

5.2. MACHINE TRANSLATION:

The NMT experiments are conducted on the English-to- Our initial experiments show that directly applying attention forcing to NMT can degrade the performance. One concern is that for translation, various re-orderings of the output sequence are valid.

In this case, guiding the model with generated output can be problematic, as the reference output can take an ordering that is different from the generated output. To see if this is the reason, we tried a modified attention forcing mode, where the model is guided with reference attention and reference output. The right side of equation 29 becomes: ŝt = f (ŝ t-1 , y t-1 ; θs ). ŝt is computed with the reference output y 1:t-1 , and matches the reference attention α t Other parts of attention forcing (equations 28 to 31) stay the same, hence ŷt is predicted with y 1:t-1 and α t .

In the following experiments, two NMT models are compared: one is trained in teacher forcing mode, with the NLL loss in equation 13; the other is trained in the modified attention forcing mode described above, with both the NLL loss and the attention loss in equation 34. An ensemble of 10 models are trained with teacher forcing. Then each model generates reference attention for a corresponding model trained with additional attention loss. The average performance of the teacher forcing models is 26.35 BLEU, and adding the attention loss yields an average +0.35 BLEU gain. 9 of out 10 times, the performance improves. The slight but consistent gain shows that for NMT, guiding the model with generated output is indeed the cause degrading the performance. It also shows that guiding the model with reference attention can be beneficial. One possible reason is that the attention loss regularizes the attention mechanism. Another is that the model does not need to learn to simultaneously infer the output and align it with the input. The TTS experiments are conducted on LJ dataset (Ito, 2017) . This public domain dataset contains 13,100 utterances from a single speaker reading passages from 7 non-fiction books. The utterances vary in length from 1 to 10 seconds and have a total length of approximately 24 hours. A transcription (character sequence) is provided for each utterance (waveform sequence). The waveforms are resampled to 16kHz to increase the efficiency of neural vocoders. Corresponding vocoder features are extracted at the frame rate of 0.2kHz, using a PML vocoder (Degottex et al., 2016) . The training-validation-test split is 13000-50-50.

6. CONCLUSION:

The frame-level model is very similar to Tacotron (Wang et al., 2017) , a powerful attention-based encoder-decoder model. The differences are: 1) the decoder target is vocoder features; 2) the attention mechanism is the hybrid (content-based + location-based) attention described in Chorowski et al. (2015) ; 3) the reduction factor is 5, i.e. each decoding step predicts 5 vocoder feature frames.

Apart from these, the model structure is the same as described in Wang et al. (2017) . The input characters are represented as one-hot vectors. The encoder has an embedding layer mapping the one-hot vectors to continuous vectors, a bottleneck layer with dropout, and a CBHG module generating the final encoding sequence. The CBHG module consists of a bank of 1-D convolutional filters, followed by highway networks (Srivastava et al., 2015) and a bidirectional GRU. The decoder has a stack of GRUs with vertical residual connections and generates the intermediate vocoder features. These features are post-processed by another CBHG module, yielding the final vocoder features. The model configuration is the same as described by Table 1 in Wang et al. (2017) .

The waveform-level model is the Hierarchical Recurrent Neural Network (HRNN) neural vocoder (Mehri et al., 2016; Dou et al., 2018) . The HRNN structure is a hierarchy of tiers; each tier includes several neural network layers and operates at a different frequency. The lowest tier operates at waveform-level frequency, and outputs distributions of waveform samples. Each higher tier operates at a lower frequency, and supervises the tier below it. The model configuration is as follows. Tier 0 is a 4-layer DNN, including three fully connected layers with ReLU activation and a softmax output layer; the dimension is 1024 for the first two fully connected layers, and is 256 for the other two

:

The term "teacher forcing", as well as "attention forcing", can refer to either an operation mode, or the approach to train a model in that operation mode. An operation mode can be used not only to train a model, but also to generate from it. For example, in teacher forcing mode, given the reference output y1:T , a model can generate a guided output y 1:T , without evaluating the loss. y 1:T is likely to be different but similar to y1:T , and can be useful for training the discriminator.

Generated test utterances are randomly selected and made available at http://mi.eng.cam.ac.uk/ ˜qd212/iclr2020/samples.html

Figures and Tables:

Figure 1: Illustration of an attention-based encoder-decoder model

E ŷ1:T ∼p(y 1:T |x * 1:L ;θ) D(y * 1:T , ŷ1:T ). This motivates training the model with the following loss: m) is sampled from the estimated distribution p(y 1:T |x (n)

Figure 2: Illustration of attention forcing

. In our experiments, this option makes training more stable, most probably because the reference attention is the same from epoch to epoch. There are several alternative options. One example is to tie θ and θ, i.e. use only one set of model parameters, and train it with the joint loss L (A)

Figure 4: Result of the listening test comparing teacher forcing and attention forcing

Vietnamese task in IWSLT 2015. It is a low resource NMT task, where training set contains 133K sentence pairs. The Stanford pre-processed data is used. The TED tst2012 is used as a validation set, and BLEU scores on TED tst2013 are reported. The scores use a 4-gram corpus level BLEU with equal weights. Google's attention-based encoder-decoder LSTM model (Wu et al., 2016) is adopted. Details of the setup (data, model and training) are presented in appendix A.2.2.

Table content: <table/>

This paper introduces attention forcing, which guides a seq2seq model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in a stable fashion, without the need for a schedule or a classifier. In addition, it allows the model to generate output sequences aligned with the reference output sequences, which can be important for cascaded systems like many TTS systems. The TTS experiments show that attention forcing yields significant gain in speech quality. The NMT experiments show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history can be problematic, while guiding the model with reference attention yields slight but consistent gain in BLEU score.Sam Wiseman and Alexander M Rush. Sequence-to-sequence learning as beam-search optimization.arXiv preprint arXiv:1606.02960, 2016.Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.Youngjae Yu, Jongwook Choi, Yeonhwa Kim, Kyung Yoo, Sang-Hun Lee, and Gunhee Kim. Supervising neural attention models for video captioning by human gaze data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 490-498, 2017.The exact search shown in equation 4 is computationally expensive, and is often approximated by greedy search if the output space is continuous, or beam search if the output space is discrete(Bengio et al., 2015). For greedy search, the model generates the output sequence one token at a time based on previous output tokens, until a special end-of-sequence token is generated. For beam search, a heap of b best candidate sequences is kept. At each time step, the candidates are updated by extending each candidate by one step, and pruning the heap to only keep b best candidates. The beam search stops when no new sequences are added.

Table content: <table><tr><td>Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor-</td></tr><tr><td>mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928,</td></tr><tr><td>2016.</td></tr><tr><td>A APPENDIX</td></tr><tr><td>A.1 DETAILS OF SEQUENCE-TO-SEQUENCE GENERATION</td></tr><tr><td>A.2 DETAILS OF EXPERIMENTAL SETUP</td></tr><tr><td>A.2.1 SPEECH SYNTHESIS</td></tr></table>

Peer review (iclr2020):
This paper proposes an alternative mechanism of training the attention values of a sequence to sequence learning model as applied to tasks like speech synthesis and translation. During training they compute two forms of attention: (1) the standard soft-attention from a decoder fed with teacher forced output, and (2) the inference-time attention from a decoder fed with predicted outputs. Their training objective consists of two terms: The first is the token-wise cross entropy loss but by conditioning on the predicted output but with teacher-forced attention. The second is a KL distance between the above two types of attention distributions. Experiments with mechanical turks indicate that their attention forcing mechanism is strongly preferred over the existing teacher forced output and attention model. On translation their method provides little or no improvement. I am inclined towards rejecting the paper because the experiment and related work section still requires a lot of work before 1. The claimed utility of the idea is established, and 2. The novelty over the many existing attention architectures is established. I elaborate on each of these next. Related work: Recently, many papers have directly or indirectly handled the problem of exposure bias that this paper attempts to address. The paper does not discuss most of these. Here are some that are missed from the paper: 1. Sequence level training with recurrent neural networks MA Ranzato, S Chopra, M Auli, W Zaremba, 2015. This paper shows that the scheduled sampling method (discussed in the paper) is much worse than a reinforce-based training mechanism of handling exposure bias. 2. An actor-critic algorithm for sequence prediction D Bahdanau, P Brakel, K Xu, A Goyal, R Lowe 3. Posterior Attention Models for Sequence to Sequence Learning S Shankar, S Sarawagi - 2019 4. Latent Alignment and Variational Attention Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, Alexander M. Rush 2018 5. Attention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings Shaohui Kuang, Junhui Li, António Branco, Weihua Luo, Deyi Xiong Experiments: Their experiments are rather sketchy and limited. The TTS experiments are only on one dataset. Their method is compared only with the standard seq2seq learning approach. Even the scheduled sampling or professor forcing methods are not compared with. In addition, state of the art TTS methods have gained significantly from hierarchical attention. As such as far as the TTS task is concerned the significance of the improved quality over a baseline seq2seq method is limited. For translation they consider only the English-Vietnamese task whereas there are tens of other translation tasks that are used in recent literature. Overall, the idea proposed seems quite incremental, experiments are limited, and related work discussion incomplete. ********* I read the author response but I do not think the paper is ready for publication yet without the thorough comparison with related work.