Title: No title found

Abstract: In this work, we first propose ParaNet, a non-autoregressive seq2seq model that converts text to spectrogram. It is fully convolutional and obtains 46.7× speed-up over Deep Voice 3 (Ping et al., 2018b) at synthesis while maintaining comparable speech quality using a WaveNet vocoder. ParaNet also produces stable alignment between text and speech on the challenging test sentences by iteratively improving the attention in a layer-by-layer manner. Based on ParaNet, we build the first fully parallel neural text-to-speech system using parallel neural vocoders, which can synthesize speech from text through a single feed-forward pass. We investigate several parallel vocoders within the TTS system, including variants of IAF vocoders and bipartite flow vocoder. 1

1. INTRODUCTION:

Text-to-speech (TTS), also called speech synthesis, has long been a vital tool in a variety of applications, such as human-computer interactions, virtual assistant, and content creation. Traditional TTS systems are based on multi-stage hand-engineered pipelines (Taylor, 2009) . In recent years, deep neural networks based autoregressive models have attained state-of-the-art results, including highfidelity audio synthesis (van den Oord et al., 2016) , and much simpler sequence-to-sequence (seq2seq) pipelines (Sotelo et al., 2017; Wang et al., 2017; Ping et al., 2018b) . In particular, one of the most popular neural TTS pipelines consists of two components (Ping et al., 2018b; Shen et al., 2018) : (i) an autoregressive seq2seq model that generates mel spectrogram from text, and (ii) an autoregressive neural vocoder (e.g., WaveNet) that generates raw waveform from mel spectrogram. This pipeline requires much less expert knowledge and only needs pairs of audio and transcript as training data.

The autoregressive nature of these components leads to slow synthesis, because they operate sequentially at a high temporal resolution of waveform samples and spectrogram. Most recently, Parallel WaveNet (van den Oord et al., 2018) and ClariNet (Ping et al., 2018a) were proposed for parallel waveform synthesis by distilling an inverse autoregressive flow (IAF) (Kingma et al., 2016) from a pretrained WaveNet. However, they still rely on autoregressive or recurrent components to predict the frame-level acoustic features (e.g., 100 frames per second), which can be slow at synthesis on modern hardware optimized for parallel execution.

In this work, we introduce a fully parallel neural TTS system by proposing a non-autoregressive text-to-spectrogram model. Our major contributions are as follows:

1. We propose ParaNet, the first non-autoregressive attention-based architecture for TTS, which is fully convolutional and converts text to mel spectrogram. We compare the nonautoregressive ParaNet with its autoregressive counterpart (Ping et al., 2018b) in terms of speech quality, synthesis speed and attention stability. It achieves ∼ 46.7 times speed-up over Deep Voice 3 (DV3) at synthesis, while maintaining comparable speech quality using a WaveNet vocoder. ParaNet iteratively refines the attention alignment between text and spectrogram in a layer-by-layer manner, and it can produce stable attentions on the challenging test sentences as DV3 with attention masking.

2. Based on ParaNet, we build the first fully parallel neural TTS system by combining ParaNet with parallel neural vocoders. Our system can generate speech from text through a single feed-forward pass. We investigate several parallel vocoders within the parallel TTS system, including the distilled IAF vocoder (Ping et al., 2018a) and WaveGlow (Prenger et al., 2019) .

To explore the possibility of training IAF vocoder without distillation in this parallel system, we also propose an alternative approach, WaveVAE, which can be trained from scratch within the variational autoencoder (VAE) framework (Kingma and Welling, 2014) .

We organize the rest of this paper as follows. Section 2 discusses related work. We introduce the nonautoregressive text-to-spectrogram architecture in Section 3, and discuss the parallel neural vocoders in Section 4. We report experimental results in Section 5, and conclude the paper in Section 6.

2. RELATED WORK:

Neural speech synthesis has obtained the state-of-the-art results and gained a lot of attention. Several neural TTS systems were proposed, including WaveNet (van den Oord et al., 2016) , Deep Voice (Arık et al., 2017a) , Deep Voice 2 (Arık et al., 2017b) , Deep Voice 3 (Ping et al., 2018b) , Tacotron (Wang et al., 2017) , Tacotron 2 (Shen et al., 2018) , Char2Wav (Sotelo et al., 2017) , VoiceLoop (Taigman et al., 2018) , WaveRNN (Kalchbrenner et al., 2018) , Transformer TTS (Li et al., 2019) and ClariNet (Ping et al., 2018a) . In particular, Tacotron and Deep Voice 3 employ seq2seq framework with the attention mechanism (Bahdanau et al., 2015) , yielding much simpler pipeline compared to traditional multistage pipeline. Their excellent extensibility leads to promising results for several challenging tasks, such as voice cloning (Arik et al., 2018; Nachmani et al., 2018; Jia et al., 2018; Chen et al., 2019) .

All of these state-of-the-art TTS systems are based on autoregressive models.

RNN-based autoregressive models, such as Tacotron and WaveRNN (Kalchbrenner et al., 2018) , lack parallelism at both training and synthesis. CNN-based autoregressive models, such as Deep Voice 3 and WaveNet, enable parallel processing at training, but they still operate sequentially at synthesis since each output element must be generated before it can be passed in as input at the next time step. Recently, there are some non-autoregressive models proposed for neural machine translation. For example, Gu et al. (2018) trains a feed-forward neural network conditioned on fertility values, which is obtained from an external alignment system. Kaiser et al. (2018) proposes a latent variable model for fast decoding, while it remains autoregressive between latent variables. Lee et al. (2018) iteratively refines the output sequence through a denoising autoencoder framework. Arguably, non-autoregressive model plays a more important role in text-to-speech, where the output speech spectrogram consists of hundreds of time steps for a short text with a few words. To the best of our knowledge, our work is the first non-autoregressive seq2seq model for TTS and provides as much as 46.7 times speed-up at synthesis over its convolutional autoregressive counterpart.

Flow-based generative models (Rezende and Mohamed, 2015; Kingma et al., 2016; Dinh et al., 2017; Kingma and Dhariwal, 2018) transform a simple initial distribution into a more complex one by applying a series of invertible transformations. In previous work, flow-based models have obtained state-of-the-art results for parallel waveform synthesis (van den Oord et al., 2018; Ping et al., 2018a; Prenger et al., 2019; Kim et al., 2019; Yamamoto et al., 2019) .

Variational autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) has been applied for representation learning of natural speech for years. It models either the generative process of waveform samples (Chung et al., 2015; van den Oord et al., 2017) , or spectrograms (Hsu et al., 2019) . In previous work, autoregressive or recurrent neural networks are employed as the decoder of VAE (Chung et al., 2015; van den Oord et al., 2017) , but they can be quite slow at synthesis.

3 TEXT-TO-SPECTROGRAM MODEL Our parallel TTS system has two components: 1) a feed-forward text-to-spectrogram model, and 2) a parallel waveform synthesizer conditioned on spectrogram. In this section, we first present an autoregressive text-to-spectrogram model derived from Deep Voice 3 (DV3) (Ping et al., 2018b) . We then introduce ParaNet, a non-autoregressive text-to-spectrogram model (see Figure 1 ).

3.1. AUTOREGRESSIVE ARCHITECTURE:

Our autoregressive model is based on DV3, a convolutional text-to-spectrogram model, which consists of three components:

• Encoder: A convolutional encoder, which encodes text into internal hidden representation. The first attention block in decoder gets positional encoding as the query and is followed by non-causal convolution blocks and attention blocks.

• Decoder: A causal convolutional decoder, which decodes the encoder representation with an attention mechanism to log-mel spectragrams in an autoregressive manner with an 1 loss. It starts with 1 × 1 convolutions to preprocess the input log-mel spectrograms. • Converter: A non-causal convolutional post-processing network, which processes the hidden representation from the decoder using both past and future context information and predicts the log-linear spectrograms with an 1 loss. It enables bidirectional processing.

All these components use the same 1-D convolution with a gated linear unit as in DV3. The major difference between our model and DV3 is the decoder architecture. The decoder of DV3 has multiple attention-based layers, where each layer consists of a causal convolution block followed by an attention block. To simplify the attention distillation described in Section 3.3.1, our autoregressive decoder has only one attention block at its first layer. We find that reducing the number of attention blocks does not hurt the generated speech quality in general.

3.2. NON-AUTOREGRESSIVE ARCHITECTURE:

Our ParaNet (see Figure 2 ) uses the same encoder architecture as the autoregressive model. The decoder of ParaNet, conditioned solely on the hidden representation from the encoder, predicts the entire sequence of log-mel spectrograms in a feed-forward manner. As a result, both its training and synthesis can be performed in parallel. Specially, we make the following major architecture modifications from the autoregressive seq2seq model to the non-autoregressive model:

1. Non-autoregressive decoder: Without the autoregressive generative constraint, the decoder can use non-causal convolution blocks to take advantage of future context information and to improve model performance. In addition to log-mel spectrograms, it also predicts log-linear spectrograms with an 1 loss for slightly better performance. 2. No converter: Non-autoregressive model removes the non-causal converter since it already employs a non-causal decoder. Note that, the major motivation of introducing non-causal converter in DV3 is to refine the decoder predictions based on bidirectional context information provided by non-causal convolutions (Ping et al., 2018b) . 

3.3. PARALLEL ATTENTION MECHANISM:

It is challenging for the non-autoregressive model to learn the accurate alignment between the input text and output spectrogram. Note that, we need the full parallelism within the attention mechanism. For example, the location-sensitive attention (Chorowski et al., 2015; Shen et al., 2018) improves attention stability, but it performs sequentially at both training and synthesis, because it uses the cumulative attention weights from previous decoder time steps as an additional feature.

Previous non-autoregressive decoders rely on an external alignment system (Gu et al., 2018) , or an autoregressive latent variable model (Kaiser et al., 2018) . In this work, we present several simple & effective techniques, which could obtain accurate and stable alignment with the multi-step attention (Gehring et al., 2017) . In particular, our non-autoregressive decoder can iteratively refine the attention alignment between text and mel spectrogram in a layer-by-layer manner as illustrated in Figure 3 . Specially, the decoder adopts a dot-product attention mechanism and consists of K attention blocks (see Figure 2 ), where each attention block uses the per-time-step query vectors from convolution block and per-time-step key vectors from encoder to compute the attention weights (Ping et al., 2018b) . The attention block computes context vectors as the weighted average of the value vectors from the encoder. The non-autoregressive decoder starts with an attention block, in which the query vectors are solely positional encoding (see Section 3.3.2 for details). The first attention block then provides the input for the convolution block at the next attention-based layer.

3.3.1. ATTENTION DISTILLATION:

We use the attention alignments from the pretrained autoregressive model to guide the training of non-autoregressive ParaNet. Specially, we minimize the cross entropy between their attention distributions. We denote the attention weights from ParaNet as W (k) i,j , where i and j index the time-step of encoder and decoder respectively, and k refers to the k-th attention block within the decoder. Note that, the attention weights {W (k) i,j } M i=1 form a valid distribution. We compute the attention loss as the average cross entropy between the student and teacher's attention distributions:

EQUATION

where W T i,j are the attention weights from the autoregressive teacher, M and N are the lengths of encoder and decoder, respectively. Our final loss function is a linear combination of l atten and 1 losses from spectrogram predictions. We set the coefficient of l atten as 4, and other coefficients as 1.

3.3.2. POSITIONAL ENCODING:

We use a similar positional encoding as in DV3 at every attention block. The positional encoding is added to both key and query vectors in the attention block, which forms an inductive bias for monotonic attention. Note that, ParaNet solely relies on its attention mechanism to decode mel spectrograms from the encoded textual features, without any autoregressive input. This makes the positional encoding even more crucial in guiding the attention to follow a monotonic progression over time at training. The positional encodings are set as h p (i, k) = sin ( ωsi /10000 k /d ) (for even i), and cos ( ωsi /10000 k /d ) (for odd i), where i is the time-step index, k is the channel index, d is the total number of channels in the positional encoding, and ω s is the position rate which indicates the average slope of the line in the attention distribution and roughly corresponds to the speed of speech. We set ω s in the following ways:

• For the autoregressive teacher, ω s is set to one for the positional encoding of query. For the key, it is set to the averaged ratio of the time-steps of spectrograms to the time-steps of textual features, which is around 6.3 across our training dataset. Taking into account that a reduction factor of 4 is used to simplify the learning of attention mechanism (Wang et al., 2017; Ping et al., 2018b) , ω s is simply set as 6.3 /4 for the key at both training and synthesis. • For ParaNet, ω s is also set to one for the query, while ω s for the key is calculated differently.

At training, ω s is set to the ratio of the lengths of spectrograms and text for each individual training instance, which is also divided by a reduction factor of 4. At synthesis, we need to specify the length of output spectrogram and the corresponding ω s , which actually controls the speech rate of the generated audios (see Section III on demo website). F, we simply set ω s to be 6.3 /4 as in autoregressive model, and the length of output spectrogram as 6.3 /4 times the length of input text. Such a setup yields an initial attention in the form of a diagonal line and guides the non-autoregressive decoder to refine its attention layer by layer (see Figure 3 ).

3.3.3. ATTENTION MASKING:

Inspired by the attention masking in DV3 (Ping et al., 2018b) , we propose a different attention masking scheme for ParaNet at synthesis. For each query from decoder, instead of computing the softmax over the entire set of encoder key vectors, we compute the softmax only over a fixed window centered around the target position and going forward and backward several time-steps (e.g., 3). The target position is calculated as i query × 4/6.3 , where i query is the time-step index of the query vector, and is the rounding operator. We observe that this strategy reduces serious attention errors such as repeating or skipping words, and also yields clearer pronunciations, thanks to its more condensed attention distribution. Note that, this attention masking is shared across all attention blocks once it is generated, and does not prevent the parallel synthesis of the non-autoregressive model.

4. PARALLEL NEURAL VOCODER:

As an indispensable component in our parallel neural TTS system, the parallel neural vocoder converts the mel spectrogram returned from the non-autoregressive ParaNet into the raw waveform. In this section, we discuss several existing neural vocoders, and explore a new alternative in the system.

4.1. IAF VOCODER VS. BIPARTITE FLOW VOCODER:

Inverse autoregressive flow (IAF) (Kingma et al., 2016) is a special type of normalizing flow where each invertible transformation is based on an autoregressive neural network. IAF performs synthesis in parallel and can easily reuse the expressive autoregressive architecture, such as WaveNet (van den Oord et al., 2016) , which leads to the state-of-the-art results for speech synthesis (van den Oord et al., 2018; Ping et al., 2018a) . However, the likelihood evaluation in IAF is autoregressive and slow, thus previous training methods rely on probability density distillation from a pretrained autoregressive WaveNet (van den Oord et al., 2018; Ping et al., 2018a) . The two-stage distillation process complicates the training pipeline and may even introduce pathological optimization (Huang et al., 2019) . RealNVP (Dinh et al., 2017) and Glow (Kingma and Dhariwal, 2018) are different types of normalizing flows, where both synthesis and likelihood evaluation can be performed in parallel by enforcing bipartite architecture constraints. Most recently, both methods were applied as parallel neural vocoders and can be trained from scratch (Prenger et al., 2019; Kim et al., 2019) . However, these models are less expressive than their autoregressive and IAF counterparts, because half of the variables are unchanged after each transformation. In general, these bipartite flows require larger number of layers and hidden units, which lead to huge number of parameters. For example, a WaveGlow vocoder (Prenger et al., 2019) has 87.88M parameters, whereas IAF vocoder has much smaller footprint with only 2.15M parameters (Ping et al., 2018a) , making it more preferred in production deployment.

4.2. WAVEVAE:

Given a small-footprint IAF vocoder, it is interesting to investigate whether it can be trained without the density distillation (e.g., Huang et al., 2019) . Our method uses the VAE framework, thus it is termed as WaveVAE. In contrast to van den Oord et al. ( 2018); Ping et al. (2018a) , WaveVAE can be trained from scratch by jointly optimizing the encoder q φ (z|x, c) and decoder p θ (x|z, c), where z is latent variables and c is the mel spectrogram conditioner. We omit c for concise notation hereafter.

Encoder: The encoder of WaveVAE q φ (z|x) is parameterized by a Gaussian WaveNet (Ping et al., 2018a ) that maps the ground truth audio x into the latent representation z with the same length. Specifically, the Gaussian WaveNet models x t given the previous samples x <t as x t ∼ N µ(x <t ; φ), σ(x <t ; φ) , where the mean µ(x <t ; φ) and scale σ(x <t ; φ) are predicted by the WaveNet, respectively. The encoder posterior is constructed as,

EQUATION

Note that, the mean µ(x <t ; φ) and scale σ(x <t ; φ) are applied for "whitening" the posterior distribution. We introduce a trainable scalar ε > 0 to capture the global variation, which will ease the optimization process. Given the observed x, the q φ (z|x) admits parallel sampling of latents z. Note that, there is a connection between the encoder of WaveVAE and the teacher model of ClariNet, as both of them use a Gaussian WaveNet to guide the training of the inverse autoregressive flow (IAF) (Kingma et al., 2016) for parallel wave generation.

Decoder: Our decoder p θ (x|z) is is parameterized by the one-step-ahead predictions from an IAF (Ping et al., 2018a) . Specially, we let z (0) = z and apply a stack of IAF transformations from z (0) → . . . z (i) → . . . z (n) , and each transformation z (i) = f (z (i-1) ; θ) is defined as,

EQUATION

where µ

(i) t = µ(z (i-1) <t ; θ) and σ (i) t = σ(z (i-1) <t

; θ) are shifting and scaling variables modeled by a Gaussian WaveNet. As a result, given z (0) ∼ N (µ (0) , σ (0) ) from the Gaussian prior or encoder, the one-step-ahead prediction p(z

(n) t | z (0)

<t ) also follows Gaussian with scale and mean as,

EQUATION

Lastly, we set x = • σ tot + µ tot , where ∼ N (0, I). For the generative process, we use the standard Gaussian prior p(z) = N (0, I).

VAE objective::

We maximize the evidence lower bound (ELBO) for observed x in VAE, max

EQUATION

where the KL divergence can be calculated in closed-form as both q φ (z|x) and p(z) are Gaussians,

EQUATION

)

The reconstruction term in Eq. ( 5) is intractable to compute exactly. We approximate this term via stochastic optimization, by drawing a sample z from the encoder q φ (z|x) through the reparameterization trick, and evaluating the likelihood log p θ (x|z). Although z has the same size as x, it will be disentangled by minimizing this KL divergence. To avoid the "posterior collapse", in which the posterior distribution q φ (z|x) quickly collapses to the white noise prior p(z) at the early stage of training, we apply the annealing strategy for KL divergence, where its weight is gradually increased from 0 to 1, via a sigmoid function (Bowman et al., 2016) . Through it, the encoder can encode sufficient information into the latent representations at the early stage of training, and then gradually regularize the latent representation by increasing the weight of the KL divergence.

STFT loss: Similar to Ping et al. (2018a) , we also add a short-term Fourier transform (STFT) based loss to improve the quality of the synthesized speech. We define the STFT loss as the summation of an 2 loss on the magnitudes of STFT and an 1 loss on the log-magnitudes of STFT between the output audio and ground truth audio (Arık et al., 2019; Wang et al., 2019) . For STFT, we set the frame-shift to 12.5ms, Hanning window length to 50ms, and the FFT size to 2048. We consider two STFT losses in our objective: (i) the STFT loss between ground truth audio and reconstructed audio using encoder q φ (z|x); (ii) the STFT loss between ground truth audio and synthesized audio using the prior p(z), with the purpose of reducing the gap between reconstruction and synthesis. Our final loss is a linear combination of the terms in Eq. ( 5) and the STFT losses. The corresponding coefficients are simply set to be one in all of our experiments. 

5. EXPERIMENTS:

In this section, we first compare text-to-spectrogram ParaNet with its autoregressive counterpart DV3 in terms of speech quality, synthesis speed, and attention stability. We then present the results of parallel neural TTS system paired with various parallel neural vocoders. In our experiment, we use a proprietary English speech dataset containing about 20 hours of speech data from a female speaker with a sampling rate of 48 kHz. We downsample the audios to 24 kHz.

5.1. PARANET VS. DEEP VOICE 3:

For both ParaNet and DV3, we use the mixed representation of characters and phonemes (Ping et al., 2018b) . The default hyperparameters of ParaNet and DV3 are provided in Appendix A. Both ParaNet and DV3 are trained for 1M steps using Adam optimizer (Kingma and Ba, 2015) with batch size 16. We find that larger kernel width and deeper layers generally help improve the performance of ParaNet. Our best non-autoregressive ParaNet (17.61M) is 2.57 times larger than the autoregressive DV3 (6.85M) in terms of the number of parameters, but it obtains significant speedup at synthesis.

Speech quality::

We compare the speech quality of ParaNet and DV3 using WaveNet vocoder. We train 20-layer WaveNets with residual channel 256 conditioned on the predicted mel spectrogram from ParaNet and DV3, respectively. We apply two layers of convolution block to process the predicted mel spectrogram, and use two layers of transposed 2-D convolution (in time and frequency) interleaved with leaky ReLU (α = 0.4) to upsample the outputs from frame-level to sample-level. We use the crowdMOS toolkit (Ribeiro et al., 2011) for subjective Mean Opinion Score (MOS) evaluation. We report the MOS results in Table 1 . The non-autoregressive ParaNet can provide comparable quality of speech as the autoregressive DV3 using WaveNet vocoder.

Speedup at synthesis: We compare the largest ParaNet with the autoregressive DV3 in terms of inference latency. We construct a custom 15-sentence test set (see Appendix D) and run inference for 50 runs on each of the 15 sentences (batch size is set to 1). The average inference latencies over 50 runs and 15 sentences are 0.024 and 1.12 seconds on NVIDIA GeForce GTX 1080 Ti for the non-autoregressive and autoregressive models, respectively. Hence, our ParaNet brings about 46.7 times speed-up compared to its autoregressive counterpart at synthesis.

Attention error analysis::

In autoregressive models, there is a noticeable discrepancy between the teacher-forced training and autoregressive inference, which can yield accumulated errors along the generated sequence at synthesis (e.g., Bengio et al., 2015) . In neural TTS, this discrepancy leads to miserable attention errors at autoregressive inference, including (i) repeated words, (ii) mispronunciations, and (iii) skipped words (see Ping et al. (2018b) for detailed examples), which is a critical problem for online deployment of attention-based neural TTS systems. We perform an attention error analysis for our non-autoregressive ParaNet on a 100-sentence test set (see Appendix C), which includes particularly-challenging cases from deployed TTS systems (e.g. dates, acronyms, URLs, repeated words, proper nouns, foreign words etc.). In Table 2 , we find that the non-autoregressive ParaNet has much fewer attention errors than its autoregressive counterpart at synthesis (12 vs. 37) without attention mask. Although our ParaNet distills the (teacher-forced) attentions from an autoregressive model, it only takes textual inputs at both training and synthesis and does not have the similar discrepancy as in autoregressive model. In previous work, attention masking was applied to enforce the monotonic attentions and reduce attention errors, and was demonstrated to be effective in Deep Voice 3 (Ping et al., 2018b) . We find that our non-autoregressive ParaNet has slightly fewer attention errors than autoregressive DV3 (6 vs. 8), when both of them use the attention masking.

Ablation Study: We perform ablation studies to verify the effectiveness of several techniques used in ParaNet, including attention distillation, positional encoding, and stacking decoder layers to refine the attention alignment in a layer-by-layer manner. We evaluate the performance of a non-autoregressive ParaNet model trained without attention distillation and find that it fails to learn meaningful attention alignment. The synthesized audios are unintelligible and mostly pure noise. Similarly, we train 2 ). We conduct the same attention error analysis on the 100-sentence test set and the results are shown in Table 2 . We find that increasing the number of decoder layers for non-autoregressive ParaNet can reduce the total number of attention errors, in both cases with and without applying attention mask at synthesis.

5.2. TTS SYSTEM WITH PARALLEL VOCODERS:

We evaluate the neural TTS systems with parallel neural vocoders, including distilled IAF vocoder (Ping et al., 2018a) , the proposed WaveVAE, and WaveGlow (Prenger et al., 2019) . The experimental setup can be found in Appendix B. WaveGlow produces high-frequency artifacts for some utterances. To remedy this, we applied the denoising function with strength 0.1 in the repository. 2It is effective when the predicted mel spectrograms are from DV3, but not quite effective when the predicted mel spectrograms are from ParaNet. We report the MOS evaluation results of TTS systems in Table 3 . Note that, all of parallel neural vocoders are trained on ground-truth mel spectrograms. We find training them on predicted mel spectrograms does not stably improve the audio quality, which is different from using the WaveNet vocoder. The distilled IAF vocoder tends to be more robust than WaveVAE and WaveGlow in the parallel TTS system. Although both WaveVAE and WaveGlow can be trained from scratch, WaveVAE requires much fewer parameters than WaveGlow.

6. CONCLUSION:

In this work, we build a fully parallel neural TTS system by proposing a non-autoregressive textto-spectrogram model and combining it with several parallel vocoders. Our non-autoregressive text-to-spectrogram model obtains 46.7× speed-up over Deep Voice 3. In addition, we investigate various parallel vocoders in this TTS system. To explore the possibility of training IAF vocoder without distillation for parallel waveform synthesis, we propose an alternative approach, WaveVAE, which avoids the need for training a separate autoregressive WaveNet, and can be trained from scratch.

Appendices:

A HYPERPARAMETERS FOR TEXT-TO-SPECTROGRAM MODEL 

:

Synthesized speech samples are in: https://parallel-neural-tts-demo.github.io/.

https://github.com/NVIDIA/waveglow

https://github.com/NVIDIA/waveglow

Figures and Tables:

Figure 1: (a) Autoregressive text-to-spectrogram model. The dashed line depicts the autoregressive decoding of mel spectrogram at inference. (b) Non-autoregressive ParaNet model, which distills the attention from a pretrained autoregressive model.

Figure 2: Architecture of ParaNet. Its encoder provides key and value as the textual representation.The first attention block in decoder gets positional encoding as the query and is followed by non-causal convolution blocks and attention blocks.

Figure 3: Our ParaNet iteratively refines the attention alignment in a layer-by-layer way. One can see the 1st layer attention is mostly dominated by the positional encoding prior. It becomes more and more confident about the alignment in the subsequent layers.

Mean Opinion Score (MOS) ratings with 95% confidence intervals for comparison.

Table content: <table><tr><td>Model</td><td>MOS score</td></tr><tr><td>Deep Voice 3 + WaveNet</td><td>4.09 ± 0.26</td></tr><tr><td>ParaNet + WaveNet</td><td>4.01 ± 0.24</td></tr></table>

Attention error counts for text-to-spectrogram models on the 100-sentence test set. One or more mispronunciations, skips, and repeats count as a single mistake per utterance. The nonautoregressive ParaNet (17-layer decoder) with attention mask obtains the fewest attention errors in total. For ablation study, we include the results for two additional ParaNet models. They have 6 and 12 decoder layers and are denoted as respectively.

Table content: <table><tr><td>Model</td><td colspan="5">Attention mask Repeat Mispronounce Skip Total</td></tr><tr><td>Deep Voice 3</td><td/><td>12</td><td>10</td><td>15</td><td>37</td></tr><tr><td>ParaNet ParaNet-12</td><td>No</td><td>1 5</td><td>4 7</td><td>7 5</td><td>12 17</td></tr><tr><td>ParaNet-6</td><td/><td>4</td><td>11</td><td>11</td><td>26</td></tr><tr><td>Deep Voice 3</td><td/><td>1</td><td>4</td><td>3</td><td>8</td></tr><tr><td>ParaNet ParaNet-12</td><td>Yes</td><td>2 4</td><td>4 6</td><td>0 2</td><td>6 12</td></tr><tr><td>ParaNet-6</td><td/><td>3</td><td>10</td><td>3</td><td>16</td></tr></table>

Mean Opinion Score (MOS) ratings with 95% confidence intervals for comparison. ParaNet model without adding positional encoding in the attention block. The resulting model only learns very blurry attention alignment and cannot synthesize intelligible speech. Finally, we train two non-autoregressive ParaNet models with 6 and 12 decoder layers, respectively, and compare them with the default non-autoregressive ParaNet model which has 17 decoder layers (denoted as ParaNet in Table

Table content: <table><tr><td>Neural TTS system</td><td># parameters</td><td>MOS score</td></tr><tr><td>Deep Voice 3 + IAF (distilled)</td><td>8.99 M</td><td>3.93 ± 0.27</td></tr><tr><td>Deep Voice 3 + IAF (WaveVAE)</td><td>8.99 M</td><td>3.70 ± 0.29</td></tr><tr><td>Deep Voice 3 + WaveGlow</td><td>94.73 M</td><td>3.96 ± 0.25</td></tr><tr><td>ParaNet + IAF (distilled)</td><td>19.76 M</td><td>3.52 ± 0.28</td></tr><tr><td>ParaNet + IAF (WaveVAE)</td><td>19.76 M</td><td>3.25 ± 0.34</td></tr><tr><td>ParaNet + WaveGlow</td><td>105.49 M</td><td>3.21 ± 0.31</td></tr><tr><td>another non-autoregressive</td><td/><td/></tr></table>

Hyperparameters of autoregressive seq2seq model and non-autoregressive seq2seq model in the experiment. the distilled IAF vocoder, we use the same architecture as ClariNet(Ping et al., 2018a). It consists of four stacked Gaussian IAF blocks, which are parameterized by[10, 10, 10,30]-layer WaveNets respectively, with 64 residual & skip channels and filter size 3 in dilated convolutions. The IAF is conditioned on log-mel spectrograms with two layers of transposed 2-D convolution as in ClariNet. We use the same teacher-student setup for ClariNet as inPing et al. (2018a)  and we train a 20-layer Gaussian autoregressive WaveNet as the teacher model. For the encoder in WaveVAE, we also use a 20-layers Gaussian WaveNet conditioned on log-mel spectrograms. For the decoder, we use the same architecture as the distilled IAF. Both the encoder and decoder of WaveVAE share the same conditioner network. We use Adam optimizer with 1000K steps for distilled IAF. For WaveVAE, we train it for 400K because it converges much faster. The learning rate is set to 0.001 at the beginning and annealed by half for every 200K steps for both models. For WaveGlow, we use the open source SOMETIMES%ALL YOU NEED TO DO IS COMPLETELY MAKE AN ASS OF YOURSELF AND LAUGH IT OFF TO REALISE THAT LIFE ISN'T SO BAD AFTER ALL%. 93. SHE BORROWED THE BOOK FROM HIM MANY YEARS AGO AND HASN'T YET RE-TURNED IT%WHY WON'T THE DISTINGUISHING LOVE JUMP WITH THE JUVENILE%? 94. LAST FRIDAY IN THREE WEEK'S TIME I SAW A SPOTTED STRIPED BLUE WORM SHAKE HANDS WITH A LEGLESS LIZARD%THE LAKE IS A LONG WAY FROM HERE%. 95. I WAS VERY PROUD OF MY NICKNAME THROUGHOUT HIGH SCHOOL BUT TO-DAY%I COULDN'T BE ANY DIFFERENT TO WHAT MY NICKNAME WAS%THE METAL LUSTS%THE RANGING CAPTAIN CHARTERS THE LINK%. 96. I AM HAPPY TO TAKE YOUR DONATION%ANY AMOUNT WILL BE GREATLY APPRE-CIATED%THE WAVES WERE CRASHING ON THE SHORE%IT WAS A LOVELY SIGHT%THE PARADOX STICKS THIS BOWL ON TOP OF A SPONTANEOUS TEA%. 97. A PURPLE PIG AND A GREEN DONKEY FLEW A KITE IN THE MIDDLE OF THE NIGHT AND ENDED UP SUNBURNT%THE CONTAINED ERROR POSES AS A LOGICAL TARGET%THE DIVORCE ATTACKS NEAR A MISSING DOOM%THE OPERA FINES THE DAILY EXAMINER INTO A MURDERER%. 98. AS THE MOST FAMOUS SINGLER-SONGWRITER%JAY CHOU GAVE A PERFECT PER-FORMANCE IN BEIJING ON MAY TWENTY FOURTH%TWENTY FIFTH%AND TWENTY SIXTH TWENTY THREE ALL THE FANS THOUGHT HIGHLY OF HIM AND TOOK PRIDE IN HIM ALL THE TICKETS WERE SOLD OUT%. 99. IF YOU LIKE TUNA AND TOMATO SAUCE%TRY COMBINING THE TWO%IT'S REALLY NOT AS BAD AS IT SOUNDS%THE BODY MAY PERHAPS COMPENSATES FOR THE LOSS OF A TRUE METAPHYSICS%THE CLOCK WITHIN THIS BLOG AND THE CLOCK ON MY LAPTOP ARE ONE HOUR DIFFERENT FROM EACH OTHER%. 100. SOMEONE I KNOW RECENTLY COMBINED MAPLE SYRUP AND BUTTERED POP-CORN THINKING IT WOULD TASTE LIKE CARAMEL POPCORN%IT DIDN'T AND THEY DON'T RECOMMEND ANYONE ELSE DO IT EITHER%THE GENTLEMAN MARCHES AROUND THE PRINCIPAL%THE DIVORCE ATTACKS NEAR A MISSING DOOM%THE COLOR MISPRINTS A CIRCULAR WORRY ACROSS THE CONTROVERSY%. D 15-SENTENCE TEST SET The 15 sentences used to quantify the inference speed up in Section 5.2.1 are listed below (note that % corresponds to pause): 1. WHEN THE SUNLIGHT STRIKES RAINDROPS IN THE AIR%THEY ACT AS A PRISM AND FORM A RAINBOW%. 2. THESE TAKE THE SHAPE OF A LONG ROUND ARCH%WITH ITS PATH HIGH ABOVE%AND ITS TWO ENDS APPARENTLY BEYOND THE HORIZON%. 3. WHEN A MAN LOOKS FOR SOMETHING BEYOND HIS REACH%HIS FRIENDS SAY HE IS LOOKING FOR THE POT OF GOLD AT THE END OF THE RAINBOW%. 4. IF THE RED OF THE SECOND BOW FALLS UPON THE GREEN OF THE FIRST%THE RESULT IS TO GIVE A BOW WITH AN ABNORMALLY WIDE YELLOW BAND%. 5. THE ACTUAL PRIMARY RAINBOW OBSERVED IS SAID TO BE THE EFFECT OF SUPER IMPOSITION OF A NUMBER OF BOWS%. 6. THE DIFFERENCE IN THE RAINBOW DEPENDS CONSIDERABLY UPON THE SIZE OF THE DROPS%. 7. IN THIS PERSPECTIVE%WE HAVE REVIEWED SOME OF THE MANY WAYS IN WHICH NEUROSCIENCE HAS MADE FUNDAMENTAL CONTRIBUTIONS%. 8. IN ENHANCING AGENT CAPABILITIES%IT WILL BE IMPORTANT TO CONSIDER OTHER SALIENT PROPERTIES OF THIS PROCESS IN HUMANS%. 9. IN A WAY THAT COULD SUPPORT DISCOVERY OF SUBGOALS AND HIERARCHICAL PLANNING%. 10. DISTILLING INTELLIGENCE INTO AN ALGORITHMIC CONSTRUCT AND COMPARING IT TO THE HUMAN BRAIN MIGHT YIELD INSIGHTS%. 11. THE VAULT THAT WAS SEARCHED HAD IN FACT BEEN EMPTIED EARLIER THAT SAME DAY%. 12. ANT LIVES NEXT TO GRASSHOPPER%ANT SAYS%I LIKE TO WORK EVERY DAY%. 13. YOUR MEANS OF TRANSPORT FULFIL ECONOMIC REQUIREMENTS IN YOUR CHOSEN COUNTRY%. 14. SLEEP STILL FOGGED MY MIND AND ATTEMPTED TO FIGHT BACK THE PANIC%. 15. SUDDENLY%I SAW TWO FAST AND FURIOUS FEET DRIBBLING THE BALL TOWARDS MY GOAL%.

Table content: <table><tr><td>Hyperparameter</td><td colspan="2">Autoregressive Model Non-autoregressive Model</td></tr><tr><td>FFT Size</td><td>2048</td><td>2048</td></tr><tr><td>FFT Window Size / Shift</td><td>1200 / 300</td><td>1200 / 300</td></tr><tr><td>Audio Sample Rate</td><td>24000</td><td>24000</td></tr><tr><td>Reduction Factor r</td><td>4</td><td>4</td></tr><tr><td>Mel Bands</td><td>80</td><td>80</td></tr><tr><td>Character Embedding Dim.</td><td>256</td><td>256</td></tr><tr><td>Encoder Layers / Conv. Width / Channels</td><td>7 / 5 / 64</td><td>7 / 9 / 64</td></tr><tr><td>Decoder PreNet Affine Size</td><td>128, 256</td><td>N/A</td></tr><tr><td>Decoder Layers / Conv. Width</td><td>4 / 5</td><td>17 / 7</td></tr><tr><td>Attention Hidden Size</td><td>128</td><td>128</td></tr><tr><td>Position Weight / Initial Rate</td><td>1.0 / 6.3</td><td>1.0 / 6.3</td></tr><tr><td>PostNet Layers / Conv. Width / Channels</td><td>5 / 5 / 256</td><td>N/A</td></tr><tr><td>Dropout Keep Probability</td><td>0.95</td><td>1.0</td></tr><tr><td>ADAM Learning Rate</td><td>0.001</td><td>0.001</td></tr><tr><td>Batch Size</td><td>16</td><td>16</td></tr><tr><td>Max Gradient Norm</td><td>100</td><td>100</td></tr><tr><td>Gradient Clipping Max. Value</td><td>5.0</td><td>5.0</td></tr><tr><td>Total Number of Parameters</td><td>6.85M</td><td>17.61M</td></tr><tr><td colspan="2">B DETAILS OF PARALLEL NEURAL VOCODERS</td><td/></tr><tr><td colspan="3">implementation with default hyperparameters 3 , except sampling rate (22.05kHz→24kHz), FFT</td></tr><tr><td colspan="3">window length (1024→1200), and FFT window shift (256→300) for handling the 24kHz dataset. It</td></tr><tr><td>is trained for 1M steps.</td><td/><td/></tr><tr><td>C 100-SENTENCE TEST SET</td><td/><td/></tr><tr><td colspan="3">The 100 sentences used to quantify the results in Table 1 are listed below (note that % corresponds to</td></tr><tr><td>pause):</td><td/><td/></tr><tr><td>1. A B C%.</td><td/><td/></tr><tr><td>2. X Y Z%.</td><td/><td/></tr><tr><td>3. HURRY%.</td><td/><td/></tr><tr><td>4. WAREHOUSE%.</td><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposes improvements to the TTS architecture in DeepVoice3. For the most part (as far as I am aware), neural architectures for TTS are encoder-decoder based. The backbone can either be an RNN or a wavenet style model (autoregressive). Synthesizing audio from the feature representation (mel spectrogram) by a neural vocoder is usually an autoregressive model from the wavenet family. In the current work, the main novelties seem to be the following: 1) Replace the autoregressive component in the decoder (seemingly inspired by DeepVoice3) with a non-autoregressive model. This could be very advantageous because synthesis in autoregressive models can be quite slow owing to the sample level generations (and to overcome this defect, faster sampling with inverse autoregressive flows has been used in parallel wavenet, clarinet, etc. and probability density distillation). Here, if I can interpret the paper correctly, the authors use a trained attention model (autoregressive), to distill attention for the non-autoregressive setup used, which would otherwise have difficulties learning alignment. The paper claims speed up of ~50X over DeepVoice 3 which is very significant. 2) In the mel to audio converter (vocoder), the proposal is a VAE Wavenet, with appropriate modifications for the sequence modeling. The authors mention that there are similarities to the approach used in Clarinet (this has a closed form KLD between the distilled distributions, which makes things easy). Experiments: It is mentioned that poor attention alignments are the cause of many issues in these architectures (repeats, mispronunciations, skipped words, etc.), and they go on to show that their architecture fares well as regards these metrics, while maintaining a comparable MOS score with DeepVoice3+Wavenet. Evaluations: I am reasonably convinced by the audio demos provided. My thoughts: The paper is generally a good addition to the TTS literature. These are difficult to implement, brittle setups, and a practitioner could spend a lot of time debugging their broken attention curves. To that end, I feel that any technique that throws light into the modeling process would be useful for the practitioner. It is suggested that we can use a non-autoregressive model with speedups. Likewise, they also use a wavenet VAE, which they claim can be trained without distillation (could the authors please clarify this point?) I do, however, feel that the paper has a few drawbacks. 1) The presentation is not at all clear. This is a very subjective comment, but I feel that this work might be unreadable to someone who hasn*t studied the literature (starting from Tacotron, DeepVoice 1, 2, 3, wavenet, parallel wavenet, transformer, distillation, etc.). Furthermore, the paper does not seem to make things any clearer with succinct architecture diagrams. Just as a comparison, I would like to draw attention to Tacotron (1, 2) in which I think the details can actually be worked out with some effort. 2) Why are we using the WaveVAE - is this just to do away with the probability distillation in wavenet? Are we also using the IAF setup as described in Kingma*s work? 3) I really feel that we need more architecture diagrams for this work to be useful. That being said, the authors do provide a list of hyperparameters for potential use. 4) Distilling attention seems to require a previously trained autoregressive attention model. What if we don*t have one ready at hand? In summary, while I see that the work will definitely be useful to the Speech Synthesis practitioner, the clarity of the paper could be improved and we need a few more diagrams (maybe even code) to make it implementable.