Parallel Neural Text-to-Speech
Authors: Kainan Peng
Wei Ping
Zhao Song
Kexin Zhao
Venue:  (2020)
Keywords: text-to-speech
non-autoregressive model
parallel decoding

Abstract: In this work, we first propose ParaNet, a non-autoregressive seq2seq model that converts text to spectrogram. It is fully convolutional and obtains 46.7 times speed-up over Deep Voice 3 at synthesis while maintaining comparable speech quality using a WaveNet vocoder. ParaNet also produces stable alignment between text and speech on the challenging test sentences by iteratively improving the attention in a layer-by-layer manner. Based on ParaNet, we build the first fully parallel neural text-to-speech system using parallel neural vocoders, which can synthesize speech from text through a single feed-forward pass.  We investigate several parallel vocoders within the TTS system, including variants of IAF vocoders and bipartite flow vocoder.

Under review as a conference paper at ICLR 2020
PARALLEL NEURAL TEXT-TO-SPEECH
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we first propose ParaNet, con-autoreerrssivve seq2seq model that converts text to spectrogram. It is fully convolutional and obtains 46.7x x speed-up over Deep Voice 3 (Ping al., 201 at synthesis while maintaining comparable speech quality using a WaveNet vocoder. ParaNet also produces stable alignment between text and speech on the challenging test sentences by iteratively improving the attention in layer-by-layer manner. Based ParaNet, we build the first fully parallel neural text-to-spp soxeech system using parallel neural vocoders, which can synthesize speech from text through single feed-forward pass. We investigate several parallel vocoders within the TTS system, including variants of IAF vocoders and bipartite flow vocoder."
INTRODUCTION
Text-to-speech (TTS), also called speech synthesis, has long been vital tool in a variety of applica- tions, such as human-computer interactions, virtual assistant, and content creation. Traditional TTS systems are based on multi-stage hand-engineered pipelines (Taylor, 2009). In recent years, deep neural networks based autoregressive models have attained state-of-the-art results, including high- fidelity audio synthesis (van den Oord et al., 2016), and much simpler sgquence-o-sequemee pipelines (Sotelo et al., 2017; Wang et al., 2017; Ping et al., 201 8b). In particular, one of the most popular neural TTS pipelines consists two components (Ping al., 2018h; 8b; Shen al., 2018): (i) an autoregressive seq2seq model that generates mel spectrogram from text, and (ii) an autoregressive neural vocoder (e.g.. WaveNet) that generates raw waveform from mel spectrogram. This pipeline requires much less expert knowledge and only needs pairs of audio and transcript training data.
(seq2seq)
The autoregressive nature of these components leads to slow synthesis, because they operate sequen- tially at a high temporal resolution of waveform samples and spectrogram. Most recently, Parallel WaveNet (van den Oord et al., 2018) and ClariNet (Ping et al., 2018a) were proposed for parallel waveform synthesis by distilling inverse autoregressive flow (IAF) (Kingma et al., 2016) from pretrained WaveNet. However, they still rely on autoregressive or recurrent components to predict the frame-level acoustic features (e.g., 100 frames per second), which can be slow synthesis modern hardware optimized for parallel execution.
In this work, we introduce fully parallel neural TTS system by proposing a non-autoregresssvv text-to-spectrogram model. Our major contributions are as follows:
1. We propose ParaNet, the first non-autoregressive attention-based architecture for TTS, which is fully convolutional and converts text mel spectrogram. We compare the non- autoregressive ParaNet with its autoregressive counterpart (Ping et al., 2018b) in terms of speech quality, synthesis speed and attention stability. It achieves 46.7 times speed-up over Deep Voice 3 (DV3) at synthesis, while maintaining comparable speech quality using a WaveNet vocoder. ParaNet iteratively refines the attention alignment between and spec- trogram in soyeerby-layee manner, and it can produce stable attentions on the challenging test sentences as DV3 with attention masking.
2. Based ParaNet, we build the first fully parallel neural TTS system by combining ParaNet with parallel neural vocoders. Our system can generate speech from text through a single forward pass. We investigate several parallel vocoders within the parallel TTS system, including the distilled IAF vocoder (Ping et al., 2018a) and WaveGlow (Prenger et al., 2019).
Syythesizedd speech samples are in: httph:t/////... //tp///aaaaallaaa/ e neural -tt- tts--dmo..ee demo.gí thub. io/.

---

Under review as a conference paper at ICLR 2020
To explore the possibility of training IAF vocoder without distillation in this parallel system, we also propose an alternative approach, WaveVAE, which can be trained from scratch within the variational autoencoder (VAE) framework (Kingma and Welling, 2014).
We organize the rest of this paper as follows. Section 2 discusses related work. We introduce the non- autoregressive text-to-spectrogram architecture in Section 3, and discuss the parallel neural vocoders in Section 4. We report experimental results in Section 5, and conclude the paper in Section
RELATED WORK
Neural speech synthesis has obtained the state-of-the-art results and gained a lot of attention. Several neural TTS systems were proposed, including WaveNet (van den Oord et al., 2016). Deep Voice (Arik et al., 2017a), Deep Voice 2 (Arik et al., 2017b), Deep Voice 3 (Ping et al., 2018b), Tacotron (Wang et al., 2017), Tacotron 2 (Shen et al., 2018), Whar2Wav (Sotclo et al., 2017). VoiceLoop (Taigman et al., 2018), WaveRNN (Kalchbrenner al., 2018; 1). Transformer TTS (Li et al., 2019) and ClariNet (Ping et al., 201 8a). In particular, Tacotron and Deep Voice 3 employ seq2seq framework with the attention mechanism (Bahdanau et al., 2015), yielding much simpler pipeline compared traditional multi- stage pipeline. Their excellent extensibility leads to promising results for several challenging tasks, such voice cloning (Arik al., 2018; Nachmani et al., 2018; Jia al., 2018; Chen al., 2019). All of these state-of-the -hertre-rrrt TTS systems are based on autoregressive models.
RNN-based autoregressive models, such as Tacotron and WaveRNN (Kalchbrenner al., 2018), lack parallelism at both training and synthesis. CNN-based autoregressive models, such as Deep Voice 3 and WaveNet, enable parallel processing training, but they still operate sequentially synthesis since each output element must be generated before it can be passed in as input at the next time step. Recently, there are some non-autoregressive models proposed for neural machine translation. For example, Gu et al. (2018) trains ferd-forward neural network conditioned on fertility values, which is obtained from an external alignment system. Kaiser et al. (2018) proposes a latent variable model for fast decoding, while it remains autoregressive between latent variables. Lee et al. (2018) iteratively refines the output sequence through a denoising autoencoder framework. Arguably, non-autoregressive model plays more important role in text-to-speech, where the output speech spectrogram consists of hundreds of time steps for a short text with a few words. To the best of our knowledge, our work is the first non-autoregressive seq2seq model for TTS and provides as much as 46.7 times speed-up at synthesis over its convolutional autoregressive counterpart.
Flow-based generative models (Rezende and Mohamed, 2015; Kingma et al., 2016; Dinh et al., 2017; Kingma and Dhariwal, 2018) transform a simple initial distribution into a more complex one by applying series of invertible transformations. In previous work, flow-based models have obtained state-of-the-art results for parallel waveform synthesis (van den Oord et al., 2018; Ping et al., 201 8a; Prenger al., 2019; Kim et al., 2019; Yamamoto et al., 2019).
Variational autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) has been applied for representation learning of natural speech for years. It models either the generative process of waveform samples (Chung et al., 2015; van den Oord et al., 2017), or spectrograms (Hsu et al., 2019). In previous work, autoregressive or recurrent neural networks are employed as the decoder of VAE (Chung et al., 2015; van den Oord et al., 2017), but they can be quite slow synthesis.
3 TEXT-TO-SPECTROGRA. MODEL
Our parallel TTS system has two components: 1) a feed-forward speetr-spee model, and 2) a parallel waveform synthesizer conditioned on spectrogram. In this section, first present an autoregressive -xteto-speccrrogam model derived from Deep Voice (DV3) (Ping et al., 2018b). We then introduce ParaNet, non-autorcgressive text-to-spectrogram model (see Figure 1).
3.1 AUTOREGRESSIIV ARCHITECTURE
Our autoregressive model is based DV3, a convolutional text-to-spectrograa model, which consists of three components:
Encoder: A convolutional encoder, which encodes text into internal hidden representation.
2

---

Under review as a conference paper at ICLR 2020
Converter
inear output
Output
Decoder
Mel
Docoder
Stadent
Teacher
Alcention
Attention
Eucoddr
Encoder
Text
(a)
(b)
Figure I: (a) Autoregressive -ettto-spectrogram model. The dashed line depicts the autoregressive decoding of mel spectrogram at inference. (b) Non-autoregressive ParaNet model, which distills the attention from a pretrained autoregressive model.
(key, valuc)
Linear Output
Mei Output
Encnder
Deanier
Facader PostNot
X
Convolution Block
Attention Block
query
Convolution Block
PreNet
Positional
Emmhedding
query
Eacoding
Figure 2: Architecture of ParaNet. Its encoder provides key and value as the textual representation. The first attention block in decoder gets positional encoding as the query and is followed by non-can non-causal convolution blocks and attention blocks.
Decoder: A causal convolutional decoder, which decodes the encoder representation with an attention mechanism to log-mel spectragrams an auloregressive manner with an & loss. It starts with x convolutions to preprocess the input log-mel spectrograms.
Converter: A non-causal convolutional post-processing network, which processes the hidden representation from the decoder using both past and future context information and predicts the log-linear spectrograms with an & loss. It enables bidirectional processing.
All these components the same 1-D convolution with gated linear unit in DV3. The major difference between our model and DV3 is the decoder architecture. The decoder of DV3 has multiple attention-based layers, where each layer consists of causal convolution block followed by attention block. To simplify the attention distillation described in Section 3, our autoregressive decoder has only attention block at its first layer. We find that reducing the number of attention blocks does not hurt the generated speech quality in general.
3.2 NON-AUTOREGRRSSIVE ARCHITECTURE
Our ParaNet (see Figure 2) uses the same encoder architecture as the autoregressive model. The decoder of ParaNet, conditioned solely the hidden representation from the encoder, predicts the entire sequence of log-mel spectrograms and synthesis can be performed in parallel. Specially, we make the following major architecture modifications from the autoregressive seq2seq model to the non-autoregressive model:
feed-forward manner. As result, both its training
1. Non-autoregressive decoder: Without the autoregressive generative constraint, the decoder can use non-causal convolution blocks to take advantage of future context information and to improve model performance. In addition to log-mel spectrograms, it also predicts log-line log-linear spectrograms with an & loss for slightly better performance.
2. No converter: Non-autoregressive model removes the non-causal converter since it already employs non-causal decoder. Note that, the major motivation of introducing non-causal con- verter in DV3 is refine the decoder predictions based on bidirectional context information provided by non-causal convolutions (Ping et al., 2018b).
3

---

Under review as a conference paper at ICLR 2020
35
45
36
Figure 3: Our ParaNet iteratively refines the attention alignment in a layer- by-lerr way. One can see the 1st layer attention is mostly dominated by the positional encoding prior. It becomes more and more confident about the alignment in the subsequent layers.
3.3 PARALLEL ATTENTION MECHANISM
is challenging for the non-autoregressive model to learn the accurate alignment between the input text and output spectrogram. Note that, need the full parallelism within the attention mechanism. For example, the location-sensitio attention (Chorowski et al., 2015; Shen et al., 2018) improves attention stability, but it performs sequentially at both training and synthesis, because uses the cumulative attention weights from previous decoder time steps as an additional feature. Previous non-autoregressivv decoders rely on external alignment system (Gu et al., 2018), or an autoregressive latent variable model (Kaiser et al., 2018), In this work, we present several simple & effective techniques, which could obtain accurate and stable alignment with the multi-step attention (Gehring et al., 2017). In particular, our non- autoregressive decoder can iteratively refine the attention aligmment between text and mel spectrogram in layer-by-laye manner as illustrated in Figure 3. Specially, the decoder adopts a dot-product attention mechanism and consists of K attention blocks (see Figure 2), where each attention block uses the per-time-step query vectors from convolution block and per-time-step key vectors from encoder to compute the attention weights (Ping et al., 2018b). The attention block computes context vectors as the weighted average of the value vectors from the encoder. The non-autoregressive decoder starts with an attention block, in which the query vectors are solely positional encoding (see Section 3.3.2 for details). The first attention block then provides the input for the convolution block at the next attention-base layer.
3.3. ATTENTION DISTILLATION
We use the attention alignments from the pretrained autoregressive model to guide the training of non-autoregressive ParaNet. Specially, we minimize the cross entropy between their attention distributions. We denote the attention weights from ParaNet w:lk) where i and j index the time-step of encoder and decoder respectively, and is refers to the k-th attention block within the decoder. Note that, the attention weights (W!!) MM Ji=1 form a valid distribution. We compute the attention loss as the average cross entropy between the student and teacher's 's attention distributions:
K ΣΣΣ
1
W., log ((
Lattnn
KN
where i, are the attention weights from the autoregressive teacher, M and N are the lengths of encoder and decoder, respectively. Our final loss function linear combination of laten and & losses from spectrogram predictions. We set the coefficient of laten as 4, and other coefficients as 1.
3.3.2 POSITIONAL ENCODING
We use similar positional encoding as DV3 at every attention block. The positional encoding is added to both key and query vectors in the attention block, which forms an inductive bias for monotonic attention. Note that, ParaNet solely relies on its attention mechanism to decode mel spectrograms from the encoded textual features, without any autoregressive input. This makes the positional encoding even more crucial in guiding the attention to follow a monotonic progression over time at training. The positional encodings and cos ww.t/10000//4 "Ma) (for odd i). where is the time-step index, & is the channel index, is the total number of channels in the positional encoding, and w. the position rate which indicates the average
set as hyp (i, k) sin (w. (e03/10000*1e) (for even i),
4

---

Under review as a conference paper at ICLR 2020
slope of the line in the attention distribution and roughly corresponds to the speed of speech. We set we in the following ways:
For the autoregressive teacher, Wa is set to one for the positional encoding of query. For the key, it is set to the averaged ratio of the time-steps of spectrograms to the time-steps of textual features, which is around 6.3 across factor of 4 is used to simplify the learning of attention mechanism (Wang et al., 2017; Ping al., 2018b) W. is simply set as 6.3/2 for the key both training and synthesis. For ParaNet, wa is also set to one for the query, while wa for the key is calculated differently. At training, Wa is set the ratio of the lengths of spectrograms and text for each individual training instance, which is also divided by reduction factor of 4. At synthesis, we need to specify the length of output spectrogram and the corresponding wis which actually controls the speech rate of the generated audios (see Section III on demo website). F, we simply set to be 6.3/4 as in autoregressive model, and the length of output spectrogram as 6.3/4 times the length of input text. Such a setup yields an initial attention in the form of diagonal line and guides the non-autoregressive decoder to refine its attention layer by layer (see Figure 3).
training dataset. Taking into account that reduction
3.3.3 ATTENTION MASKING
Inspired by the attention masking in DV3 (Ping et al., 2018b), we propose a different attention masking scheme for ParaNet at synthesis. For each query from decoder, instead of computing the softmax over the entire set of encoder key vectors, we compute the softmax only over a fixed window centered around the target position and going forward and backward several time-steps (e.g.. 3). The target position is calculated as Liuuery x 4/6.3], where iquery is the time-step index of the query vector, and [] is the rounding operator. We observe that this strategy reduces serious attention errors such as repeating or skipping words, and also yields clearer pronunciations, thanks to its more condensed attention distribution. Note that, this attention masking is shared across all attention blocks once is generated, and does not prevent the parallel synthesis of the non-autoregressive model.
4 PARALLEI NEURAL VOCODER
As an indispensable component in our parallel neural TTS system, the parallel neural vocoder converts the mel spectrogram returned from the non-autoregressive ParaNet into the raw waveform. In this section, we discuss several existing neural vocoders, and explore new alternative in the system.
4.1 IAF VOCODER vs. BIPARTITE FLOW VOCODER
Inverse autoregressive flow (IAF) (Kingma et al., 2016) is special type of normalizing flow where each invertible transformation is based on an autoregressive neural network. IAF performs synthesis in parallel and can easily reuse the expressive autoregressive architecture, such as WaveNet (van den Oord et al., 2016), which leads to the state-of-the-art results for speech synthesis (van den Oord et al., 2018; Ping et al., 2018a). However, the likelihood evaluation in IAF is autoregressive and slow, thus previous training methods rely on probability density distillation from a pretrained au- toregressive WaveNet (van den Oord et al., 2018; Ping et al., 2018a). The two-stage distillation process complicates the training pipeline and may even introduce pathological optimization (Huang et al., 2019). RealNVP (Dinh et al., 2017) and Glow (Kingma and Dhariwal, 2018) are different types of normalizing flows, where both synthesis and likelihood evaluation can be performed in parallel by enforcing bipartite architecture constraints. Most recently, both methods were applied as parallel neural vocoders and can be trained from scratch (Prenger et al., 2019; Kim et al., 2019). However, these models are less expressive than their autoregressive and IAF counterparts, because half of the variables are unchanged after each transformation. In general, these bipartite flows require larger number of layers and hidden units, which lead to huge number of parameters. For example, a WaveGlow vocoder (Prenger et al., 2019) has 87.88M parameters, whereas IAF vocoder has much smaller footprint with only 2.M parameters (Ping et al., 2018a), making it more preferred in production deployment.
4.2 WAVEVAE
Given a amall-footpric IAF vocoder, it is interesting to investigate whether it can be trained without the density distillation (e.g., Huang et al., 2019). Our method uses the VAE framework, thus it is
5

---

Under review as a conference paper at ICLR 2020
termed VAve In contrast to van den Oord et al. (2018); Ping et al. (2018a), WaveVAE can be trained from scratch by jointly optimizing the encoder go (2 c) and decoder po(a) z c), where is latent variables and c is the mel spectrogram conditioner. We omit for concise notation hereafter.
Encoder: The encoder of Wav go (0(=(z)) is parameterized by a Gaussian WaveNet (Ping et al., 2018a) that maps the ground truth audio into the latent representation with the same length. Specifically, the Gaussian WaveNet models Da given the previous samples Zet as Fr N (N(( Φ), (xrec ))), where the mean u( Φ) and scale of 6) are predicted by the WaveNet, respectively. The encoder posterior is constructed as,
(( ) (o(2|x)) - II (z Fe agge); where go(z) xgg))= NV o
Note that, the mean uu( u(=< Φ) and scale ( Φ) are applied for "whitening" the posterior dis- tribution. We introduce trainable scalar E 0 to capture the global variation, which will ease the optimization process. Given the observed æ, the (o(2(r) admits parallel sampling of latents
2. Note that, there is a connection between the encoder of Wave and the teacher model of ClariNet, as both of them use Gaussian WaveNet to guide the training of the inverse autoregressive flow (IAF) (Kingma et al., 2016) for parallel wave generation.
Decoder: Our decoder (=]22) is is parameterized by the one-step-ahead predictions from an IAF (Ping al., 201 8a). Specially, we let ( and apply stack of IAF transformations from + (n), and each transformation 2+)) = f(z f(z(i-)) (): 1);)) is defined as,
(0)
(0) = 2 (i-1) (3) + tz
(3)
(=-1)=))) 0) are shifting and scaling variables modeled by a Gaussian WaveNet. As result, given (0) ~ N ( ((1)) from the Gaussian prior or encoder, the one-step-ahead prediction p(2 = also follows Gaussian with scale and mean as,
u(=21 (0:====1 0) and (0 = o 1),
where 14
I
1 for
Σ 1
II
(4)
i=0
i=0
j>i
Lastly, we set x = c. 10 + n0 where e ~ (0, I). For the generative process, we use the standard Gaussian prior p(z) NOO I).
VAE objective: We maximize the evidence lower bound (ELBO) for observed æ in VAE,
(x)(= log (a =)] (zs()))))
(5)
6.0
where the KL divergence can be calculated in closed-form as both 4o (z (z) and p(z) are Gaussians,
(L(((())=) Σ log ] ( - 1
-1(x)et))) ("= )2)
(6)
o(ret)
The reconstruction term in Eq. (5) is intractable to compute exactly. We approximate this term via stochastic optimization, by drawing a sample from the encoder qa (z|zz) x>) through the reparameteri- zation trick, and evaluating the likelihood log po z).. Although has the same size as æ, will be disentangled by minimizing this KI. divergence. To avoid the "posterior collapse", in which the posterior distribution gs (2(2);: quickly collapses to the white noise prior p(2) at the early stage of training, we apply the annealing strategy for KL divergence, where its weight gradually increased from O to 1, via a sigmoid function (Bowman et al., 2016). Through it, the encoder can encode sufficient information into the latent representations the early stage of training, and then gradually regularize the latent representation by increasing the weight of the KL divergence.
Similar to Ping et al. (2018a), we also add a short-term Fourier transform (STFT) based loss to improve the quality of the synthesized speech. We define the STFT loss as the summation of an & loss on the magnitudes of STFT and an & loss on the log-magnitudes of STFT between the output audio and ground truth audio (Arik et al., 2019; Wang et al., 2019). For STFT, we set the frame-shift to 12.5ms, Hanning window length to 50ms, and the FFT size to 2048 We consider two STFT losses in our objective: (i) the STFT loss between ground truth audio and reconstructed audio using encoder q0 ==)=;; (ti) the STFT loss between ground truth audio and synthesized audio using the prior p( (z)), with the purpose of reducing the gap between reconstruction and synthesis. Our final loss is a linear combination of the terms in Eq. (5) and the STET losses. The corresponding
STFT loss:
coefficients are simply set to be one in all of our experiments.

---

Under review as a conference paper at ICLR 2020
Table 1: Mean Opinion Score (MOS) ratings with 95% confidence intervals for comparison.
Model Deep Voice 3 WaveNet
MOS score
4.09 = 0.26
ParaNet + WaveNet
4.01 = 0.24
EXPERIMENTS
In this section, we first compare cox-tto-pecttrogram ParaNet with its autoregressive counterpart DV3 in terms of speech quality, synthesis speed, and attention stability. We then present the results of parallel neural TTS system paired with various parallel neural vocoders. In our experiment, we use a proprietary English speech dataset containing about 20 hours of speech data from female speaker with a sampling rate of 48 kHz. We downsample the audios to 24 kHz.
5.1 PARANET VS. DEEP VOICE 3
For both ParaNet and DV3, we use the mixed representation of characters and phonemes (Ping et al., 2018b). The default hyperparameters of ParaNet and DV3 are provided in Appendix A. Both ParaNet and DV3 are trained for 1M steps using Adam optimizer (Kingma and Ba, 2015) with batch size 16. We find that larger kernel width and deeper layers generally help improve the performance of ParaNet. Our best non-autoregressive ParaNet (17.6 (17.61M) is 2.57 times larger than the autoregressive DV3 (6.85M) in terms of the number of parameters, but it obtains significant speedup at synthesis.
Speech quality: We compare the speech quality of ParaNet and DV3 using WaveNet vocoder. We train 20-layer WaveNets with residual channel 256 conditioned on the predicted mel spectrogram from ParaNet and DV3, respectively. We apply two layers of convolution block to process the predicted mel spectrogram, and use two layers of transposed 2-D convolution (in time and frequency) interleaved with leaky ReLU (a 0.4) to upsample the outputs from frame-level sample-level. We use the crowdMOS toolkit (Ribeiro et al., 201 1) for subjective Mean Opinion Score (MOS) evaluation. We report the MOS results in Table The non-autoregressive ParaNet can provide comparable quality of speech as the autoregressive DV3 using WaveNet vocoder.
We compare the largest ParaNet with the autoregressive DV3 in terms of inference latency. We construct a custom 15-sentence test set (see Appendix D) and run inference for 50 runs on each of the 15 sentences (batch size is set to 1). The average inference latencies over 50 runs and 15 sentences are 0.024 and 1.12 seconds on NVIDIA GeForce GTX 1080 Ti for the non-autoregressive and autoregressive models, respectively. Hence, our ParaNet brings about 46.7 times speed-up compared to its autoregressive counterpart at synthesis.
Speedup synthesis:
Attention error analysis: autoregressive models, there a noticeable discrepancy between the teacher-forced training and autoregressive inference, which can yield accumulated errors along the generated sequence at synthesis (e.g., Bengio et al., 2015). In neural TTS, this discrepancy leads to miserable attention errors at autoregressive inference. including (i) repeated words, (ii) mispronun- ciations, and (iii) skipped words (see Ping et al. (2018b) for detailed examples), which is a critical problem for online deployment of attention-based neural TTS systems. We perform attention error analysis for our non-autoregressivve ParaNet on a 100-sentence test set (see Appendix C), which includes particulary-chaallenniii cases from deployed TTS systems (e.g. dates, acronyms, URLs, repeated words, proper nouns, foreign words etc.). In Table 2, we find that the non-autoregressive ParaNet has much fewer attention errors than its autoregressive counterpart at synthesis (12 vs. 37) without attention mask. Although our ParaNet distills the (teacher- forceed attentions from an autoregressive model, it only takes textual inputs at both training and synthesis and does not have the similar discrepancy as in autoregressive model. In previous work, attention masking was applied to enforce the monotonic attentions and reduce attention errors, and was demonstrated be effective in Deep Voice 3 (Ping et al., 2018b). We find that our con-autoreeressive ParaNet has slightly fewer attention errors than autoregressive DV3 (6 vs. 8), when both of them use the attention masking.
Ablation Study:
We perform ablation studies verify the effectiveness of several techniques used in ParaNet, including attention distillation, positional encoding, and stacking decoder layers to refine the attention alignment in a layer-by-layer manner. We evaluate the performance of a non-autoregressive ParaNet model trained without attention distillation and find that it fails to learn meaningful attention alignment. The synthesized audios are unintelligible and mostly pure noise. Similarly, train
7

---

Under review as a conference paper at ICLR 2020
Table 2: Attention error counts for text-to-spectrogram models on the 100-sentence test set. One or more mispronunciations, skips, and repeats count as single mistake per utterance. The non- autoregressive ParaNet (17-layer decoder) with attention mask obtains the fewest attention errors in total. For ablation study, we include the results for two additional ParaNet models. They have 6 and 12 decoder layers and are denoted as ParaNet-6 and ParaNet- 12, respectively.
Model
Attention mask
Repeat
Mispronounce
Skip
Total
Deep Voice 3
12
10
15
ParaNet
1
4
7
12
No
ParaNet-12
7
ParaNet-6
4
11
11
26
Deep Voice 3
4
ParaNet
2
4
Yes
ParaNet-12
4
6
12
ParaNet-6
3
10
16
Table 3: Mean Opinion Score (MOS) ratings with 95% confidence intervals for comparison.
Neural TTS system
parameters
MOS score
Deep Voice 3 + IAF (distilled)
8.99 M
3.93 = 0.27
Deep Voice 3 IAF (WaveVAE)
8.99 M
3.70 0.29
Deep Voice 3 + WaveGlow
94.73 M
3.96 + 0.25
ParaNet + IAF (distilled)
19.76
3.52 + 0.28
ParaNet + IAF (WaveVAR)
19.76 M
3.25 + 0.34
ParaNet + WaveGlow
105.49 M
3.21 + 0.31
another non-autoregressive ParaNet model without adding positional encoding in the attention block. The resulting model only learns very blurry attention alignment and cannot synthesize intelligible speech. Finally, we train two non- autorrrrerreeivve ParaNet models with 6 and 12 decoder layers, respectively, and compare them with the default non-autoregressive ParaNet model which has 17 decoder layers (denoted as ParaNet in Table 2). We conduct the same attention error analysis on the 100-sentence test set and the results are shown in Table We find that increasing the number of decoder layers for non-autoregressive ParaNet can reduce the total number of attention errors, in both cases with and without applying attention mask at synthesis.
5.2 TTS SYSTEM WITH PARALLEL VOCODERS
We evaluate the neural TTS systems with parallel neural vocoders, including distilled IAF vocoder (Ping et al., 2018a), the proposed WaveVAE, and WaveGlow (Prenger et al., 2019). The ex- perimental setup can be found in Appendix B. WaveGlow produces eigh-frequency artifacts for some utterances. To remedy this, we applied the denoising function with strength 0. in the repository. effective when the predicted mel spectrograms are from DV3, but not quite effective when the predicted mel spectrograms are from ParaNet. We report the MOS evaluation results of TTS systems Table 3. Note that, all of parallel neural vocoders are trained on ground-truth mel spectrograms. We find training them predicted mel spectrograms does not stably improve the audio quality, which different from using the WaveNet vocoder. The distilled IAF vocoder tends to be more robust than Wave VAE and WaveGlow in the parallel TTS system. Although both Wave VAE and WaveGlow can be trained from scratch, Wave VAE requires much fewer parameters than WaveGlow.
CONCLUSION
In this work, we build a fully parallel neural TTS system by proposing a non-autoregresssvve text- to-spectrogram model and combining it with several parallel vocoders. Our non-autoregressive text-to-spectroogram model obtains 46.7x speed-up over Deep Voice 3. In addition, we investigate various parallel vocoders in this TTS system. To explore the possibility of training IAF vocoder without distillation for parallel waveform synthesis, we propose an alternative approach, Wave VAE, which avoids the need for training separate autoregressive WaveNet, and can be trained from scratch.
Phtpps://git thub. www.mm/m/mmmmm /VI www.wwlwlll..oo Gw
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
S. O. Arık, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. J. Miller, J. Raiman,
S. Sengupta, and M. Shoeybi. Deep Voice: Real-time neural text-to-speech. In ICML, 2017a.
S. O. Arı, G. Diamos, A. Gibiansky, J. Miller, K. Peng, W. Ping, J. Raiman, and Y. Zhou. Deep Voice 2: Multi-speaker neural text-to-speech. In NIPS, 2017b.
S. O. Arik, J. Chen, K. Peng, W. Ping, and Y. Zhou. Neural voice cloning with few samples. In NIPS, 2018.
Ô. Ank,, Jun, and G. Diamos. Fast spectrogram inversion using multi-head convolutional neural networks. IEEE Signal Processing Letters, 2019.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICIRR, 2015.
Bengio, Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, 2015.
S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. 2016.
Y. Chen, Y. Assael, B. Shillingford, D. Budden. S. Reed, H. Zen, Q. Wang. L. C. Cobo, A. Trask, B. Laurie, et al. Sample efficient adaptive text-to-speech. In ICLR, 2019.
K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-bant models for speech recognition. In Advances in neural information processing systems, pages 577-585, 2015.
J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. In NIPS, 2015.
L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real NVP. In ICLR, 2017.
J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. Convolutional sequence to sequence learning. ICML, 2017.
Gu, Bradbury, C. Xiong, V. O. Li, and Socher. Non-aatooregressive neural machine translation. In ICLR, 2018.
W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia. Z. Chen, J. Shen, et al. Hierarchical generative modeling for controllable speech synthesis. In ICLR, 2019.
C.-W, Huang, F. Ahmed, K. Kumar, A. Lacoste, and A. Courville. Probability distillation: caveat and alternatives. In UAI, 2019.
Y. Jia, Y. Zhang, R. J. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen, P. Nguyen, R. Pang, and I. L. Moreno. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In NIPS, 2018
E. Kaiser, A. Roy, A. Vaswani, N. Pamar, S. Bengio, J. Uszkoreit, and N. Shazeer. Fast decoding in sequence models using discrete latent variables. JCML, 2018.
N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, Stimberg, A. v. Oord, S. Dieleman, and K. Kavukcuoglu. Efficient neural audio synthesis. In ICML, 2018.
S. Kim, S.-g. Lee, J. Song, and S. Yoon. FloWaveNet: A generative flow for raw audio. In ICML, 2019.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.
D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NIPS, 2018.
D. Kingma and M. Welling, Auto-encoding variational Bayes. In ICLR, 2014.

---

Under review as a conference paper at ICLR 2020
D. P. Kingma, Т. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Improving variational inference with inverse autoregressive flow. In NIPS, 2016.
J. Lee, E. Mansimov, and K. Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, 2018.
N. Li, S. Liu, Y. Liu, S. Zhao, M. Liu, and M. Zhou, Neural speech synthesis with transformer network. AAAI, 2019.
E. Nachmani, A. Polyak, Y. Taigman, and L. Wolf. Fitting new speakers based on a short untranscribed sample. In ICML, 2018.
W. Ping, K. Peng, and J. Chen. ClariNet: Parallel wave generation in end-to-end text-to-speech. 2018a.
arXiv preprini arXiv:
W. Ping, K. Peng, A. Gibiansky, O. Arik, A. Kannan, S. Narang, J. Raiman, and Miller. Deep Voice 3: Scaling text-to-speede with convolutional sequence learning, In ICLR, 2018b.
R. Prenger, R. Valle, and B. Catanzaro. WaveGlow: A flow-based generative network for speech synthesis. In ICASSP, 2019.
D. J. Rezende and S. Mohamed. Variational inference with normalizing flows. In JCMI., 2015.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. ICML, 2014.
Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. CrowdMOS: An approach for crowdsoureing mean opinion score studies. In ICASSP, 2011.
J. Shen, R. Pang. R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang. Z. Chen, Y. Zhang, Y. Wang, R. Skerry- Ryan, et al. Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In ICASSP, 2018.
Sotelo, S. Mchri, K. Kumar, J. F. Santos, Kastner, A. Courville, and Y. Bengio. Char2wav: End-to-end speech synthesis. ICLR workshop, 2017.
Y. Taigman, I.. Wolf, A. Polyak, and E. Nachmani. VoiceLoop: Voice fitting and synthesis via phonological loop. In ICLR, 2018.
Taylor. Speeech Synthesis. Cambridge University Press, 2009.
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016.
A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In NIPS, 2017.
A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. Driessche, E. Lockhart, L. C. Cobo, F. Stimberg, et al. Parallel WaveNet: Fast high-fidelity speech synthesis. In ICML, 2018
X. Wang, S. Takaki, and J. Yamagishi. Neural source-fitter-based waveform model for statistical parametric speech synthesis. In ICASSP, 2019.
Y. Wang, R. Skerry- D. Stanton, Y. Wu, R. Weiss, N. Jaitly, Z. Yang. Xiao, Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous. Tacotron: Towards end-to-end speech synthesis. Interspeech, 2017.
R. Yamamoto, E. Song, and J.-M. Kim. Probability density distillation with generative adversarial networks for high-quality parallel waveform generation. arXiv preprint arXiv:1904.0444 2019.
10

---

Under review as a conference paper at ICLR 2020
Appendices
A HYPERPARAMETERS FOR TEXI-TO-SPECTRRORAA MODEL.
Table 4: Hyperparameters of autoregressive seq2seq model and non-autoregressive seq2seq model in
the experiment.
Hyperparameter
Autoregressive Model
Non-autoregressive Model
FFT Size
2048
2048
FFT Window Size Shift
1200 300
1200 300
Audio Sample Rate
24000
24000
Reduction Factor
4
Mel Bands
80
80
Character Embedding Dim.
256
256
Encoder Layers Conv. Width / Channels
7/5/ 64
7/9/ 64
Decoder PreNet Affine Size
128, 256
N/A
Decoder Layers Conv. Width
4/5
17/7
Attention Hidden Size
128
128
Position Weight Initial Rate
1.0 6.3 6.3
1.066.3
PostNet Layers Conv. Width Channels
5 256
N/A
Dropout Keep Probability
0.95
1.0
ADAM Learning Rate
0.001
0.001
Batch Size
16
16
Max Gradient Norm
100
100
Gradient Clipping Max. Value
5.0
5.0
Total Number of Parameters
6.85M
17.61M
B DETAILS OF PARALLEL NEURAL VOCODERS
For the distilled IAF vocoder, we use the same architecture ClariNet (Ping et al., 2018a). It consists of four stacked Gaussian IAF blocks, which are parameterized by [10, 10, 10, 30]-layer WaveNets respectively, with 64 residual & skip channels and filter size 3 in dilated convolutions. The IAF is conditioned on log-mel spectrograms with two layers transposed 2-D convolution as in ClariNet. We use the same teacher-student setup for ClariNet in Ping et al. (201 8a) and we train 20-layer Gaussian autoregressive WaveNet as the teacher model. For the encoder in Vav, we also use 20-layers Gaussian WaveNet conditioned same architecture as the distilled IAF. Both the encoder and decoder of Wave VAE share the same conditioner network. We use Adam optimizer with 1000K steps for distilled IAF. For WaveVAE, train it for 400K because it converges much faster. The learning rate is set to 0,001 at the beginning and annealed by half for every 200K steps for both models. For WaveGlow, implementation with default hyperparameters except sampling rate (22.05kHz-24kHH))) window length (1024~>1200 and FFT window shift (256-3300) for handling the 24kHz dataset. is trained for 1M steps.
log-mel spectrograms. For the decoder,
the
use the open source FFT
C 100-SENTENCE TEST SET
The 100 sentences used to quantify the results in Table 1 are listed below (note that % corresponds to pause):
1. A B C%.
X Y Z%.
3. HURRY%.
WAREHOUSE%.
thub. com/Nv WWWIIIA//EER AOAA//avgllle
11

---

Under review as a conference paper at ICLR 2020
5. REFERENDUM%.
IS IT FREE%?
7. JUSTIFIABLE%.
8. ENVIRONMENT%.
A DEBT RUNS%.
10. GRAVITATIONAL
11. CARDBOARD FILM%. %..
12. PERSON THINKING%
13. PREPARED KILLER%
14. AIRCRAFT TORTURE%
15. ALLERGIC TROUSER%.
16. STRATEGIC CONDUCT%:
17. WORRYING LITERATURE%:
18. CHRISTMAS IS COMING%
19. A PET DILEMMA THINKS %.
20. HOW WAS THE MATH TEST%?
21. GOOD TO THE LAST DROP%
22, AN M B AGENT LISTENS%
23. A COMPROMISE DISAPPEARS%.
24. AN AXIS OF X OR FREEZES%.
25. SHE DID HER BEST TO HELP HIM%.
26. BACKBONE CONTESTS THE CHAOS%.
27. TWO A GREATER THAN TWO N NINE%.
28. DON'T STEP ON THE BROKEN GLASS%. %
29, A DAMNED FLIPS INTO THE PATIENTS %
30, TRADE PURGES WITHIN THE B B C%.
31. I'D RATHER BE A BIRD THAN FISH%.
32. HEAR THAT NANCY VERY PRETTY %.
33. I WANT MORE DETAILED INFORMATION
34. PLEASE WAIT OUTSIDE OF THE HOUSE&.
35. ASS EXPOSURE TUNES THE WAFFLE%.
36. A MIST DICTATES WITHIN THE MONSTER %
37. SKETCH ROPES THE MIDDLE CEREMONY %.
38. EVERY FAREWELL EXPLODES THE CAREER %.
39. SHE FOLDED HER HANDKERCHIEF NEATLY %.
40. AGAINST THE STEAM CHOOSES THE STUDIO%.
41. ROCK MUSIC APPROACHES AT HIGH VELOCITY %.
42. NINE ADAM BAYE STUDY ON THE TWO PIECES%
43. AN UNFRIENDLY DECAY CONVEYS THE OUTCOME%.
44. ABSTRACTION IS OFTEN ONE FLOOR ABOVE YOU%.
45. PLAYED LADY RANKS ANY PUBLICIZED PREVIEW %.
46. HE TOLD US A VERY EXCITING ADVENTURE STORY%. %.
47. ON AUGUST TWENTY EIGTH%MARY PLAYS THE PIANO%.
48. INTO A CONTROLLER BEAMS A CONCRETE TERRORIST%.
49. OFTEN SEE THE TIME ELEVEN ELEVEN ON CLOCKS%
50. IT WAS GETTING DARK%AND WE WEREN'T THERE YET%.
51. AGAINST EVERY RHYME STARVES A CHORAL APPARATUSE
52. EVERYONE WAS BUSY&SO WENT'TO THE MOVIE ALONE%.
53. I CHECKED TO MAKE SURE THAT HE WAS STILL ALIVE%
54. A DOMINANT VEGETARIAN SHIES AWAY FROM THE O P%,
55. JOE MADE THE SUGAR COOKIES&SUSAL DECORATED THEM%.
56. I WANT TO BUY A ONESIE KNOW IT WON'T SUIT ME%.
57. FORMER OVERRIDE OF Q W E T OUTSIDE THE POPE%.
58. B SAYS THAT CI SAYS%I'LL STAY AWAY FROM IT%.
59. ANY CLIMBING DISH LISTENS TO A CUMBERSOME FORMULASE
60. SHE WROTE HIM A LONG LETTER&BUL HE DIDN'T READ IT%.
61. DEAR&BEAUTY IS IN THE HEAT NOT PHYSICAL LOVE YOU%.
62. AN APPEAL ON JANUARY FIFTH DUPLICATES A SHARP QUEEN
63. FAREWELL SOLOS ON MARCH TWENTY THIRD SHAKES NORTH%.
12

---

Under review as a conference paper at ICLR 2020
64. HE RAN OUT OF MONEY%SO HE HAD TO STOP PLAYING POKER%
63. FOR EXAMPLE%A NEWSPAPER HAS ONLY REGIONAL DISTRIBUTION T%.
66. CURRENTLY HAVE FOUR WINDOWS OPEN UP%AND I DON'T KNOW WHY%
67. NEXT TO MY INDIRECT VOCAL DECLINES EVERY UNBEARABLE, ACADEMIC%
68. OPPOSITE HER SOUNDING BAG IS A M C'S CONFIGURED THOROUGHFARESA
69. FROM APRIL EIGHTH TO THE PRESENT% ONLY SMOKE FOUR CIGARETTES%
70. I WILL NEVER BE THIS YOUNG AGAIN%EVER%OL DAMN%I JUST GOT OLDER %.
71. A GENEROUS CONTINUUM OF AMAZON DOT COM IS THE CONFLICTING WORKER %
72. SHE ADVISED HIM TO COME BACK AT ONCE% WIFE LECTURES THE BLAST%.
73. A SONG CAN MAKE OR RUIN A PERSON'S DAY IF THEY LET IT GET TO THEM%.
74. SHE DID NOT CHEAT ON THE TEST&FOR WAS NOT THE RIGHT TO DO%. HOWERVEER MANY PEOPLE SAW HIM
75. HE SAID HE WAS NOT THERE
THERE%.
76. SHOULD WE START CLASS NOW&OR SHOULD WE WAIT FOR EVERYONE TO GET HERE%?
77. IF PURPLE PEOPLE EATERS ARE REAL& DO THEY FIND PURPLE PEOPLE TO EAT%?
78. ON NOVEMBER EIGHTEENTH EIGHTEEN TWENTY ONE%A GLITTERING GEM IS
NOT ENOUGH%.
79. ROCKET FROM SPACE X INTERACTS WITH THE INDIVIDUAL BENEATH THE SOFT FLAW%.
MALLS ARE GREAT PLACES TO SHOP%I CAN FIND EVERYTHING NEED UNDER ONE ROOF%.
81. THINK WILL BUY THE RED CAR%OR WILL LEASE THE BLUE TNE%THE FAITH NESTS%.
82. ITALY IS MY FAVORITE COUNTRY FACT%I PLAN TO SPEND TWO WEEKS THERE
NEXT YEAR%.
83. I WOULD HAVE GOTTEN W W DOT GOOGLE DOT COM&BUT MY ATTENDANCE
WASN'T GOOD ENOUGH%.
84. NINETEEN TWENTY IS WHEN WE ARE UNIQUE TOGETHER UNTIL WE REALISE&
ARE ALL THE SAME%.
85. MY MUM TRIES TO BE COOL BY SAYING H HTTP COLON SLASH SLASH W W W II EMAILED A S
U DOT COM%.
86. HE TURNED IN THE RESEARCH PAPER ON
D AT YAHOO DOT ORG%.
87. SHE WORKS TWO JOBS TO MAKE ENDS MEET%AT LEAST WAS HER REASON
FOR NOT HAVING TIME TO JOIN US%.
88. A REMARKABLE WELL PROMOTES THE ALPHABET INTO THE ADJUSTED
LUCK%THE DRESS DODGES ACROSS MY ASSAULT%
M OOOQRSTI U V X Y ONE TWO THREE FOUR FIVE SIX SEVEN EIGHT NINE TEN%.
89. A B CDEFGHIJ M.M.N
90. ACROSS THE WASTE PERSISTS THE WRONG PACIFIER% THE WASHED PASSENGER PARADES UNDER THE INCORRECT COMPUTER&.
91. IF THE EASTER BUNNY AND THE TOOTH FAIRY HAD BABIES WOULD THEY TAKE YOUR TEETH AND LEAVE CHOCOLATE FOR YOU%?
92. SOMETIMES YOU NEED TO DO IS COMPLETELY MAKE AN ASS OF YOURSELF
AND LAUGH IT OFF TO REALISE THAT LIFE ISN'T SO BAD AFTER ALL.%
93. SHE BORROWED THE BOOK FROM HIM MANY YEARS AGO AND HASN'T YET RE- TURNED IH% WON'T THE DISTINGUISHING LOVE JUMP WITH THE JUVENILE&??
94. LAST FRIDAY IN THREE WEEK'S TIME SAW A SPOTTED STRIPED BLUE WORM SHAKE HANDS WITH A LEGLESS LIZARD% LAKE IS LONG WAY FROM HERE%.
95. I WAS VERY PROUD OF MY NICKNAME THROUGHOUT HIGH SCHOOL BUT TO-
DAY%I COULDN'T BE ANY DIFFERENT TO WHAT MY NICKNAME WAS&THE METAL L THE RANGING CAPTAIN CHARTERS THE LINK%.
96. I AM HAPPY TO TAKE YOUR DONATIONS
AMOUNT WILL BE GREATLY APPRE-
CIATED& WAVES WERE CRASHING THE SHORE&IT WAS A LOVELY SIGHT&THE PARADOX STICKS THIS BOWL ON TOP OF SPONTANEOUS TEA%.
97. A PURPLE PIG AND A GREEN DONKEY FLEW A KITE IN THE MIDDLE OF THE NIGHT AND ENDED UP SUNBURNT% THE CONTAINED ERROR POSES AS A LOGICAL
13

---

Under review as a conference paper at ICLR 2020
THEGET& DIVORCE ATTACKS NEAR A MISSING DOOM% OPERA FINES THE
DAILY EXAMINER INTO A MURDERER%.
98. AS THE MOST FAMOUS
CHOU GAVE A PERFECT PER- FORMANCE IN BEIJING ON MAY TWENTY FOURTH&TWENTT FIFTH%AND TWENTY
SIXTH TWENTY THREE ALL THE FANS THOUGHT HIGHLY OF HIM AND TOOK PRIDE IN HIM AHE TICKETS WERE SOLD OUT%.
99. IF YOU LIKE TUNA AND TOMATO SAUCE%TRY COMBINING THE TWO%IT' REALLY
NOT AS BAD AS IT SOUNDS% BODY MAY PERHAPS COMPENSATES FOR THE LOSS OF TRUE METAPHYSICS%
CLOCK WITHIN THIS BLOG AND THE CLOCK ON MY LAPTOP ARE ONE HOUR DIFFERENT FROM EACH O.HER%. 100. SOMEONE/ KNOW RECENTLY COMBINED MAPLE SYRUP AND BUTTERED POP. CORN THINKING IT WOULD TASTE LIKE CARAMEL POPCORN DIDN'T AND THEY DON'T RECOMMEND ANYONE ELSE DO IT EITHER & GENTLEMAN MARCHES AROUND THE PRINCIPAL&THE DIVORCE ATTACKS NEAR A MISSING DOOM% COLOR MISPRINTS A CIRCULAR WORRY ACROSS THE CONTROVERSY%.
D 15-SENTENCE TEST SET
The 15 sentences used to quantify the inference speed up in Section 5.2.1 are listed below (note that % corresponds to pause):
WHEN THE SUNLIGHT STRIKES RAINDROPS IN THE AIR%THEY ACT AS PRISM
AND FORM A RAINBOW%.
THESE TAKE THE SHAPE OF A LONG ROUND ARCH&WITH ITS PATH HIGH ABOVE%AND ITS TWO ENDS APPARENTLY BEYOND THE HORIZON%
3. WHEN A MAN LOOKS FOR SOMETHING BEYOND HIS REACH%HIS FRIENDS SAY HE IS LOOKING FOR THE POT OF GOLD AT THE END OF THE RAINBOW%.
4. IF THE RED OF THE SECOND BOW FALLS UPON THE GREEN OF THE FIRST% THE RESULT TO GIVE BOW WITH AN ABNORMALLY WIDE YELLOW BAND%.
5. THE ACTUAL PRIMARY RAINBOW OBSERVED IS SAID TO BE THE EFFECT OF SUPER IMPOSITION OF A NUMBER OF BOWS%.
6. THE DIFFERENCE IN THE RAINBOW DEPENDS CONSIDERABLY UPON THE SIZE OF THE DROPS%.
IN THIS PERSPECTIVE&SE WE HAVE REVIEWED SOME OF THE MANY WAYS IN WHICH NEUROSCIENCE HAS MADE FUNDAMENTAL CONTRIBUTIONS%
8. IN ENHANCING AGENT CAPABILITIES@II WILL BE IMPORTANT TO CONSIDER OTHER SALIENT PROPERTIES OF THIS PROCESS IN HUMANS%.
IN A WAY THAT COULD SUPPORT DISCOVERY OF SUBGOALS AND HIERARCHICAL PLANNING%.
10. DISTILLING INTELLIGENCE INTO AN ALGORITHMIC CONSTRUCT AND COMPARING IT TO THE HUMAN BRAIN MIGHT YIELD INSIGHTS%.
11. THE VAULT THAT WAS SEARCHED HAD IN FACT BEEN EMPTIED EARLIER THAT
SAME DAY%.
12. LIVES NEXT TO GRASSHOPPERS: ANT SAYS% LIKE TO WORK EVERY DAY%.
13. YOUR MEANS OF TRANSPORT FULFIL ECONOMIC REQUIREMENTS IN YOUR
CHOSEN COUNTRY%
14. SLEEP STILL FOGGED MY MIND AND ATTEMPTED TO FIGHT BACK THE PANIC%.
15. SUDDENLYSA SAW TWO FAST AND FURIOUS FEET DRIBBLING THE BALL TOWARDS
MY GOAL%.
14


Reviews:
[{"review_id": "H1gad04AKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission belongs to the field of text-to-speech synthesis. In particular, it looks at the problem of efficient training and inference. Many current state-of-the-art approaches in the area employ an autoregressive component at one or another part of text-to-speech model making training and/or inference complicated/slow. The main idea behind this paper is to remove autoregressive components. \n\nI believe there is a great deal of interest for efficient and fast text-to-speech. I believe stirring more interest towards non-autoregressive approaches is the right direction. I believe that the approach proposed does indeed accomplish the task of removing autoregressive components from the text-to-speech model. I find however that the results with parallel neural vocoders for the proposed approach to lag behind (autoregressive) Deep Voice 3 with the same set of vocoders even though when an autoregressive vocoder is used the situation is different. \n\nI find that the presentation of this work to be lacking a balance. This work makes 2 contributions 1) non-autoregressive text-to-spectrogram and 2) non-autoregressive spectrogram-to-speech model. \n\n The first contribution is presented in a completely verbal manner to describe the complex process of text-to-speech mapping. For instance, I cannot find a proper technical description of 1) anywhere in the introduction, only statements that it is non-autoregressive and parallel. \n\nThe second contribution is squashed into a single page. Due to complexity involved I believe each of these contributions needs to be written (and assessed) separately. For instance, on page 6 you have a) encoder, b) decoder, c) VAE objective, d) STFT loss all discussed in very short details. \n\nOther comments:\n1) The Related work section would benefit from a bit more connected story than a list of RNN, flow and VAE based approaches. \n2) Block diagrams are helpful but not having a mathematical description makes them more ambiguous than they should be.\n3) \"residual channel 256 conditioned\", \"miserable attention errors\"\n4) What is significance between 6 and 8 attention errors made by ParaNet and DV3?\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgsIVtN9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes work on Parallel Neural Text-To-Speech.  \n\nThere are a number of efforts in this direction going on simultaneously.  FastSpeech is one that was noted in public comment.  I have seen other work under submission as well.  I don't believe this undermines the fundamentals of the work, but it will color claims of being \"first\" by each submission.\n\nWaveVAE is an interesting approach to a parallel neural vocoding.  However it performs works than the IAF distilled ClariNet parallel vocoder with the same amount of parameters.  It is not clear what advantages WaveVAE has over ClariNet.\n\nMore problematic, ParaNet is described as providing a speedup over Deep Voice while maintaining quality.  However, this claim is somewhat misleading.  The maintained quality is only achieved when using an autoregressive wavenet vocoder.  It appears as though the inference speed up is only measured on the feature to mel conversion, while the wall-clock inference time is (likely) dominated by wavenet inference (a notoriously computationally intensive process).  When evaluated with parallel vocoders (ClariNet, WaveVAE or WaveGlow) ParaNet performs quite poorly when compared to Deep Voice 3. (This is especially true when using WaveGlow -- the best performing vocoder for DV3 MOS=3.96, ParaNet MOS=3.21).  While there are fewer attention errors (cf. Table 2) this doesn't seem to impact MOS to a great degree.\n\nAttention Distillation is somewhat equivalent to using an external alignment or duration model as is done in classical TTS.  Requiring this external resource somewhat undermines the claim of being an \"end-to-end\" TTS system.  It is worth comparing attention distillation to, say, training a traditional alignment model based on forced alignment and feeding the target durations as a conditioning feature along with the text.  This is what ParallelWaveNet (https://arxiv.org/pdf/1711.10433.pdf) does.  From that perspective it is not clear why ParaNet+neural vocoder should be considered as \"more parallel\" than ParallelWaveNet or other IAF vocoding approaches paired with a traditional front end.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xmNSiAYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes improvements to the TTS architecture in DeepVoice3. \n\nFor the most part (as far as I am aware), neural architectures for TTS are encoder-decoder based. The backbone can either be an RNN or a wavenet style model (autoregressive). Synthesizing audio from the feature representation (mel spectrogram) by a neural vocoder is usually an autoregressive model from the wavenet family. \n\nIn the current work, the main novelties seem to be the following:\n\n1) Replace the autoregressive component in the decoder (seemingly inspired by DeepVoice3) with a non-autoregressive model. This could be very advantageous because synthesis in autoregressive models can be quite slow owing to the sample level generations (and to overcome this defect, faster sampling with inverse autoregressive flows has been used in parallel wavenet, clarinet, etc. and probability density distillation). Here, if I can interpret the paper correctly, the authors use a trained attention model (autoregressive), to distill attention for the non-autoregressive setup used, which would otherwise have difficulties learning alignment. The paper claims speed up of ~50X over DeepVoice 3 which is very significant.\n\n2) In the mel to audio converter (vocoder), the proposal is a VAE Wavenet, with appropriate modifications for the sequence modeling. The authors mention that there are similarities to the approach used in Clarinet (this has a closed form KLD between the distilled distributions, which makes things easy). \n\nExperiments: It is mentioned that poor attention alignments are the cause of many issues in these architectures (repeats, mispronunciations, skipped words, etc.), and they go on to show that their architecture fares well as regards these metrics, while maintaining a comparable MOS score with DeepVoice3+Wavenet. \n\nEvaluations: I am reasonably convinced by the audio demos provided. \n\nMy thoughts: \nThe paper is generally a good addition to the TTS literature. These are difficult to implement, brittle setups, and a practitioner could spend a lot of time debugging their broken attention curves. To that end, I feel that any technique that throws light into the modeling process would be useful for the practitioner. It is suggested that we can use a non-autoregressive model with speedups. Likewise, they also use a wavenet VAE, which they claim can be trained without distillation (could the authors please clarify this point?)\n\nI do, however, feel that the paper has a few drawbacks. \n\n1) The presentation is not at all clear. This is a very subjective comment, but I feel that this work might be unreadable to someone who hasn't studied the literature (starting from Tacotron, DeepVoice 1, 2, 3, wavenet, parallel wavenet, transformer, distillation, etc.). Furthermore, the paper does not seem to make things any clearer with succinct architecture diagrams. Just as a comparison, I would like to draw attention to Tacotron (1, 2) in which I think the details can actually be worked out with some effort. \n\n2) Why are we using the WaveVAE - is this just to do away with the probability distillation in wavenet? Are we also using the  IAF setup as described in Kingma's work?\n\n3) I really feel that we need more architecture diagrams for this work to be useful. That being said, the authors do provide a list of hyperparameters for potential use. \n\n4) Distilling attention seems to require a previously trained autoregressive attention model. What if we don't have one ready at hand?\n\nIn summary, while I see that the work will definitely be useful to the Speech Synthesis practitioner, the clarity of the paper could be improved and we need a few more diagrams (maybe even code) to make it implementable. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a non-autoregressive attention based encoder-decoder model for text-to-sepectrogram using attention distillation. It is shown to bring good speedup to conventional autoregressive ones. The paper further adopted VAE for the vocoder training which trains from scratch although performs worse than existing method (e.g. ClariNet). 

The main concerns for this paper come from the unclear presentation:
* As the reviewer pointed out, there're some misleading claims that the speedup gains was obtained without the consideration of the full context (i.e. not including the whole inference time).
* The paper failed to clear present the architectures developed/used in the paper and the differences from those used in the literature. The reviewers suggested the use of diagram to aid the presentation.
* The two contributions are unbalanced presented. Due to the complexities involved, it's better to explain things in more details. 
The authors acknowledged the reviewers comments during rebuttal, but did not make any changes to the paper.
Decision: Reject