More recently, a fully convolutional Seq2Seq architecture was investigated by Baidu Research   . The deepvoice architecture is composed of causal 1-D convolution layers for both encoder and decoder. They utilized query-key attention similar to that from the transformer architecure  . Another fully convolutional Seq2Seq architecture known as DCTTS was proposed  . In this architecture they employ modules composed of Causal 1-D convolution layers combined with Highway networks. In addition they introduced methods for help guide attention alignments early. As well as a forced incremental attention mechanism that ensures monotonic increasing of attention read as the model decodes during inference.

Recent work on neural TTS can be split into two camps, in one camp Seq2Seq models with recurrent architectures are used   . In the other camp, full convolutional Seq2Seq models are used  . Our model belongs in the first of these classes using recurrent architectures. Specifically we make the following contributions:

Related Work	Neural text-to-speech systems have garnered large research interest in the past 2 years. The first to fully explore this avenue of research was Google's tacotron  system. Their architecture based off the original Seq2Seq framework. In addition to encoder/decoder RNNs from the original Seq2Seq , they also included a bottleneck prenet module termed CBHG, which is composed of sets of 1-D convolution networks followed by highway residual layers. The attention mechanism follows the original Seq2Seq  mechanism (often termed Bahdanau attention). This is the first work to propose training a Seq2Seq model to convert text to mel spectrogram, which can then be converted to audio wav via iterative algorithms such as Griffin Lim  . A parrallel work exploring Seq2Seq RNN architecture for text-to-speech was called Char2Wav  . This work utilized a very similar RNN-based Seq2Seq architecture, albeit without any prenet modules. The attention mechanism is guassian mixture model (GMM) attention from Alex Grave's work. Their model mapped text sequence to 80 dimension vectors used for the WORLD Vocoder  , which invert these vectors into audio wave.

Proposed Improvements	Our proposed improvements come from the observation that employing generic Seq2seq models for TTS application misses out on further optimization that can be achieved when we consider the specific problem of TTS. Specifically, we notice that in TTS, unlike in applications like machine translation, the Seq2Seq attention mechanism should be mostly monotonic. In other words, when one reads a sequence of text, it is natural to assume that the text position progress nearly linearly in time with the sequence of output mel spectrogram. With this insight, we can make 3 modifications to the model that allows us to train faster while using a a smaller model.

Deep Text-to-Speech System with Seq2Seq Model	Recent trends in neural network based text-to-speech/speech synthesis pipelines have employed recurrent Seq2seq architectures that can synthesize realistic sounding speech directly from text characters. These systems however have complex architectures and takes a substantial amount of time to train. We introduce several modifications to these Seq2seq architectures that allow for faster training time, and also allows us to reduce the complexity of the model architecture at the same time. We show that our proposed model can achieve attention alignment much faster than previous architectures and that good audio quality can be achieved with a model that's much smaller in size. Sample audio available at https://soundcloud.com/gary-wang-23/sets/tts-samples-for-cmpt-419.

Model Overview	The architecture of our model utilizes RNN-based Seq2Seq model for generating mel spectrogram from text. The architecture is similar to that of Tacotron 2  . The generated mel spetrogram can either be inverted via iterative algorithms such as Griffin Lim, or through more complicated neural vocoder networks such as a mel spectrogram conditioned Wavenet  . Figure FIGREF3 below shows the overall architecture of our model.

Conclusion	We introduce a new architecture for end-to-end neural text-to-speech system. Our model relies on RNN-based Seq2seq architecture with a query-key attention. We introduce novel guided attention mask to improve model training speed, and at the same time is able to reduce model parameters. This allows our model to achieve attention alignment at least 3 times faster than previous RNN-based Seq2seq models such as Tacotron 2. We also introduce forced incremental attention during synthesis to prevent attention alignment mistakes and allow model to generate coherent speech for very long sentences.

Guided Attention Mask	Following the logic above, we utilize a similar method from  that adds an additional guided attention loss to the overall loss objective, which acts to help the attention mechanism become monotoic as early as possible. As seen from FIGREF24 , an attention loss mask, INLINEFORM0 , is created applies a loss to force the attention alignment, INLINEFORM1 , to be nearly diagonal. That is: DISPLAYFORM0  Where INLINEFORM0 , INLINEFORM1 is the INLINEFORM2 -th character, INLINEFORM3 is the max character length, INLINEFORM4 is the INLINEFORM5 -th mel frame, INLINEFORM6 is the max mel frame, and INLINEFORM7 is set at 0.2. This modification dramatically speed up the attention alignment and model convergence. Figure 3 below shows the results visually. The two images are side by side comparison of the model's attention after 10k training steps. The image on the left is trained with the atention mask, and the image on the right is not. We can see that with the attention mask, clear attention alignment is achieved much faster.

Forced Incremental Attention	During inference, the attention INLINEFORM0 occasionally skips multiple charaters or stall on the same character for multiple output frames. To make generation more robust, we modify INLINEFORM1 during inference to force it to be diagonal. The Forced incremental attention is implemented as follows: Given INLINEFORM0 , the position of character read at INLINEFORM1 -th time frame, where INLINEFORM2 , if INLINEFORM3 , the current attention is forcibly set to INLINEFORM4 , so that attention is incremental, i.e INLINEFORM5 .

Decoder	The decoder is an autoregressive recurrent neural network that predicts mel spectrogram from the encoded input sentence one frame at a time. The decoder decodes the hidden representation from the encoder, with the guidance of attention. The decoder is composed of two uni-directional LSTM/GRU with INLINEFORM0 hidden dimensions. The first LSTM/GRU, called the AttentionRNN, is for computing attention-mechanism related items such as the attention query INLINEFORM1 . DISPLAYFORM0  The second LSTM/GRU, DecoderRNN, is used to compute the decoder hidden output, INLINEFORM0 . DISPLAYFORM0  A 2-layer dense prenet of dimensions (256,256) projects the previous mel spectrogram output INLINEFORM0 into hidden dimension INLINEFORM1 . Similar to Tacotron 2, the prenet acts as an information bottleneck to help produce useful representation for the downstream attention mechanism. Our model differs from Tacotron 2 in that we jointly project 5 consequetive mel frames at once into our hidden representation, which is faster and unlike Tacotron 2 which project 1 mel frame at at time.

Text Encoder	The encoder acts to encoder the input text sequence into a compact hidden representation, which is consumed by the decoder at every decoding step. The encoder is composed of a INLINEFORM0 -dim embedding layer that maps the input sequence into a dense vector. This is followed by a 1-layer bidirectional LSTM/GRU with INLINEFORM1 hidden dim ( INLINEFORM2 hidden dim total for both directions). two linear projections layers project the LSTM/GRU hidden output into two vectors INLINEFORM3 and INLINEFORM4 of the same INLINEFORM5 -dimension, these are the key and value vectors. DISPLAYFORM0  where INLINEFORM0 .

The DecoderRNN's hidden state INLINEFORM0 is also projected to mel spectrogram INLINEFORM1 . A residual post-net composed of 2 dense layer followed by a tanh activation function also projects the same decoder hidden state INLINEFORM2 to mel spectrogram INLINEFORM3 , which is added to the linear projected mel INLINEFORM4 to produce the final mel spectrogram INLINEFORM5 . DISPLAYFORM0  A linear spectrogram INLINEFORM0 is also computed from a linear projection of the decoder hidden state INLINEFORM1 . This acts as an additional condition on the decoder hidden input. DISPLAYFORM0  A single scalar stop token is computed from a linear projection of the decoder hidden state INLINEFORM0 to a scalar, followed by INLINEFORM1 , or sigmoid function. This stop token allows the model to learn when to stop decoding during inference. During inference, if stop token output is INLINEFORM2 , we stop decoding. DISPLAYFORM0

Changes to Attention Mechanism	In the original Tacotron 2, the attention mechanism used was location sensitive attention  combined the original additive Seq2Seq  Bahdanau attention. We propose to replace this attention with the simpler query-key attention from transformer model. As mentioned earlier, since for TTS the attention mechanism is an easier problem than say machine translation, we employ query-key attention as it's simple to implement and requires less parameters than the original Bahdanau attention.

Introduction	Traditional text-to-speech (TTS) systems are composed of complex pipelines  , these often include accoustic frontends, duration model, acoustic prediction model and vocoder models. The complexity of the TTS problem coupled with the requirement for deep domain expertise means these systems are often brittle in design and results in un-natural synthesized speech. The recent push to utilize deep, end-to-end TTS architectures   that can be trained on <text,audio> pairs shows that deep neural networks can indeed be used to synthesize realistic sounding speech, while at the same time eliminating the need for complex sub-systems that neede to be developed and trained seperately. The problem of TTS can be summed up as a signal-inversion problem: given a highly compressed source signal (text), we need to invert or "decompress" it into audio. This is a difficult problem as there're multi ways for the same text to be spoken. In addtion, unlike end-to-end translation or speech recognition, TTS ouptuts are continuous, and output sequences are much longer than input squences.

We used their training steps to roughly estimate the training time of Tacotron 2 when attention roughly aligns. For all other models the training time is not comparable as they either don't apply (e.g parametric model) or are not reported (Tacotron griffin lim, Deepvoice 3). Direct comparison of model parameters between ours and the open-source tacotron 2, our model contains 4.5 million parameters, whereas the Tacotron 2 contains around 13 million parameters with default setting. By helping our model learn attention alignment faster, we can afford to use a smaller overall model to achieve similar quality speech quality.

Training and Loss	Total loss on the model is computed as the sum of 3 component losses: 1. Mean-Squared-Error(MSE) of predicted and ground-truth mel spectrogram 2. MSE of Linear Spectrogram 3. Binary Cross Entropy Loss of our stop token. Adam optimizer is used to optimize the model with learning rate of INLINEFORM0 . Model is trained via teacher forcing, where the ground-truth mel spectrogram is supplied at every decoding step instead of the model's own predicted mel spectrogram. To ensure the model can learn for long term sequences, teacher forcing ratio is annealed from 1.0 (full teacher forcing) to 0.2 (20 percent teacher forcing) over 300 epochs.

Experiment Dataset	The open source LJSpeech Dataset was used to train our TTS model. This dataset contains around 13k <text,audio> pairs of a single female english speaker collect from across 7 different non-fictional books. The total training data time is around 21 hours of audio. One thing to note that since this is open-source audio recorded in a semi-professional setting, the audio quality is not as good as that of proprietary internal data from Google or Baidu. As most things with deep learning, the better the data, the better the model and results.

Experiment Procedure	Our model was trained for 300 epochs, with batch size of 32. We used pre-trained opensource implementation of Tactron 2 (https://github.com/NVIDIA/tacotron2) as baseline comparison. Note this open-source version is trained for much longer (around 1000 epochs) however due to our limited compute we only trained our model up to 300 epochs

Evaluation Metrics	We decide to evaluate our model against previous baselines on two fronts, Mean Opnion Score (MOS) and training speed. Typical TTS system evaluation is done with mean opinion score (MOS). To compute this score, many samples of a TTS system is given to human evaluators and rated on a score from 1 (Bad) to 5 (Excellent). the MOS is then computed as the arithmetic mean of these score: DISPLAYFORM0  Where INLINEFORM0 are individual ratings for a given sample by N subjects. For TTS models from google and Baidu, they utilized Amazon mechanical Turk to collect and generate MOS score from larger number of workers. However due to our limited resources, we chose to collect MOS score from friends and families (total 6 people). For training time comparison, we choose the training time as when attention alignment start to become linear and clear. After digging through the git issues in the Tacotron 2 open-source implementation, we found a few posts where users posted their training curve and attention alignment during training (they also used the default batch size of 32).

Query-Key Attention	Query key attention is similar to that from transformers  . Given INLINEFORM0 and INLINEFORM1 from the encoder, the query, INLINEFORM2 , is computed from a linear transform of the concatenation of previous decoder-rnn hidden state, INLINEFORM3 , combined with attention-rnn hidden state, INLINEFORM4 ). DISPLAYFORM0  Given INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , the attention at each decoding step is computed by the scaled dot-product operation as: DISPLAYFORM0  Note that similar to transformers  , we apply a scale the dot-product by INLINEFORM0 to prevent softmax function into regions where it has extremely small gradients.

Question: Which modifications do they make to well-established Seq2seq architectures?
Answer: Replacing attention mechanism to query-key attention, and adding a loss to make the attention mask as diagonal as possible