ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech

1 INTRODUCTION . Speech synthesis , also called text-to-speech ( TTS ) , is traditionally done with complex multi-stage hand-engineered pipelines ( Taylor , 2009 ) . Recent successes of deep learning methods for TTS lead to high-fidelity speech synthesis ( van den Oord et al. , 2016a ) , much simpler “ end-to-end ” pipelines ( Sotelo et al. , 2017 ; Wang et al. , 2017 ; Ping et al. , 2018 ) , and a single TTS model that reproduces thousands of different voices ( Ping et al. , 2018 ) . WaveNet ( van den Oord et al. , 2016a ) is an autoregressive generative model for waveform synthesis . It operates at a very high temporal resolution of raw audios ( e.g. , 24,000 samples per second ) . Its convolutional structure enables parallel processing at training by teacher-forcing the complete sequence of audio samples . However , the autoregressive nature of WaveNet makes it prohibitively slow at inference , because each sample must be drawn from the output distribution before it can be passed in as input at the next time-step . In order to generate high-fidelity speech in real time , one has to develop highly engineered inference kernels ( e.g. , Arık et al. , 2017a ) . Most recently , van den Oord et al . ( 2018 ) proposed a teacher-student framework to distill a parallel feed-forward network from an autoregressive teacher WaveNet . The non-autoregressive student model can generate high-fidelity speech at 20 times faster than real-time . To backpropagate through random samples during distillation , parallel WaveNet employs the mixture of logistics ( MoL ) distribution ( Salimans et al. , 2017 ) as the output distribution for teacher WaveNet , and a logistic distribution based inverse autoregressive flow ( IAF ) ( Kingma et al. , 2016 ) as the student model . It minimizes a set of losses including the KL divergence between the output distributions of the student and teacher networks . However , one has to apply Monte Carlo method to approximate the intractable KL divergence between the logistic and MoL distributions , which may introduce large variances in gradients for highly peaked distributions , and lead to an unstable training in practice . In this work , we propose a novel parallel wave generation method based on the Gaussian IAF . Specifically , we make the following contributions : ∗These authors contributed equally to this work . Our method is named after the musical instrument clarinet , whose sound resembles human voice . 1Audio samples are in https : //clarinet-demo.github.io/ 1 . We demonstrate that a single variance-bounded Gaussian is sufficient for modeling the raw waveform in WaveNet without degradation of audio quality . In contrast to the quantized surrogate loss ( Salimans et al. , 2017 ) in parallel WaveNet , our Gaussian autoregressive WaveNet is simply trained with maximum likelihood estimation ( MLE ) . 2 . We distill a Gaussian IAF from the autoregressive WaveNet by minimizing a regularized KL divergence between their peaked output distributions . Our method provides closed-form estimation of KL divergence , which largely simplifies the distillation algorithm and stabilizes the training process . 3 . In previous studies , “ end-to-end '' speech synthesis actually refers to the text-to-spectrogram models with a separate waveform synthesizer ( i.e. , vocoder ) ( Sotelo et al. , 2017 ; Wang et al. , 2017 ) . We introduce the first text-to-wave neural architecture for TTS , which is fully convolutional and enables fast end-to-end training from scratch . In our architecture , the WaveNet module is conditioned on the hidden states instead of mel-spectrograms ( Ping et al. , 2018 ; Shen et al. , 2018 ) , which is crucial to the success of training from scratch . Our text-to-wave model significantly outperforms the separately trained pipeline ( Ping et al. , 2018 ) in naturalness . 4 . We also successfully distill a parallel neural vocoder conditioned on the learned hidden representation within the end-to-end architecture . The text-to-wave model with the parallel vocoder obtains competitive results as the model with an autoregressive vocoder . We organize the rest of this paper as follows . Section 2 discusses related work . We propose the parallel wave generation method in Section 3 , and present the text-to-wave architecture in Section 4 . We report experimental results in Section 5 and conclude the paper in Section 6 . 2 RELATED WORK . Neural speech synthesis has obtained the state-of-the-art results and gained a lot of attention recently . Several neural TTS systems were proposed , including Deep Voice 1 ( Arık et al. , 2017a ) , Deep Voice 2 ( Arık et al. , 2017b ) , Deep Voice 3 ( Ping et al. , 2018 ) , Tacotron ( Wang et al. , 2017 ) , Tacotron 2 ( Shen et al. , 2018 ) , Char2Wav ( Sotelo et al. , 2017 ) , and VoiceLoop ( Taigman et al. , 2018 ) . Deep Voice 1 & 2 retain the traditional TTS pipeline , which has separate grapheme-to-phoneme , phoneme duration , fundamental frequency , and waveform synthesis models . In contrast , Deep Voice 3 , Tacotron , and Char2Wav employ the attention based sequence-to-sequence models ( Bahdanau et al. , 2015 ) , yielding more compact architectures . In the literature , these models are usually referred to as “ end-to-end ” speech synthesis . However , they actually depend on a traditional vocoder ( Morise et al. , 2016 ) , the Griffin-Lim algorithm ( Griffin and Lim , 1984 ) , or a separately trained neural vocoder ( Ping et al. , 2018 ; Shen et al. , 2018 ) to convert the predicted spectrogram to raw audio . In this work , we propose the first text-to-wave neural architecture for TTS based on Deep Voice 3 ( Ping et al. , 2018 ) . The neural network based vocoders , such as WaveNet ( van den Oord et al. , 2016a ) and SampleRNN ( Mehri et al. , 2017 ) , play a very important role in recent advances of speech synthesis . In a TTS system , WaveNet can be conditioned on linguistic features , fundamental frequency ( F0 ) , phoneme durations ( van den Oord et al. , 2016a ; Arık et al. , 2017a ) , or the predicted mel-spectrograms from a text-to-spectrogram model ( Ping et al. , 2018 ) . We test our parallel waveform synthesis method by conditioning it on mel-spectrograms and hidden representation within the end-to-end model . Normalizing flows ( Rezende and Mohamed , 2015 ; Dinh et al. , 2014 ) are a family of stochastic generative models , in which a simple initial distribution is transformed into a more complex one by applying a series of invertible transformations . Normalizing flow provides arbitrarily complex posterior distribution , making it well suited for the inference network in variational autoencoder ( Kingma and Welling , 2014 ) . Inverse autoregressive flow ( IAF ) ( Kingma et al. , 2016 ) is a special type of normalizing flow where each invertible transformation is based on an autoregressive neural network . Thus , IAF can reuse the most successful autoregressive architecture , such as PixelCNN and WaveNet ( van den Oord et al. , 2016b ; a ) . Learning an IAF with maximum likelihood can be very slow . In this work , we distill a Gaussian IAF from a pretrained autoregressive generative model by minimizing a numerically stable variant of KL divergence . Knowledge distillation is originally proposed for compressing large models to smaller ones ( Bucilua et al. , 2006 ) . In deep learning ( Hinton et al. , 2015 ) , a smaller student network is distilled from the teacher network by minimizing the loss between their outputs ( e.g. , L2 or cross-entropy ) . In parallel WaveNet , a non-autoregressvie student-net is distilled from an autoregressive WaveNet by minimizing the reverse KL divergence ( Murphy , 2014 ) . Similar techniques are applied in non-autoregressive models for machine translation ( Gu et al. , 2018 ; Kaiser et al. , 2018 ; Lee et al. , 2018 ; Roy et al. , 2018 ) . 3 PARALLEL WAVE GENERATION . In this section , we present the Gaussian autoregressive WaveNet as the teacher-net and the Gaussian inverse autoregressive flow as the student-net . Then , we develop our knowledge distillation algorithm . 3.1 GAUSSIAN AUTOREGRESSIVE WAVENET . WaveNet models the joint distribution of high dimensional waveform x = { x1 , . . . , xT } as the product of conditional distributions using the chain rules of probability , p ( x | c ; θ ) = T∏ t=1 p ( xt | x < t , c ; θ ) , ( 1 ) where xt is the t-th variable of x , x < t represent all variables before t-step , c is the conditioner 2 ( e.g. , mel-spectrogram or hidden states in Section 4 ) , and θ are parameters of the model . The autoregressive WaveNet takes x < t as input , and outputs the probability distribution over xt . Parallel WaveNet ( van den Oord et al. , 2018 ) advocates mixture of logistics ( MoL ) distribution in PixelCNN++ ( Salimans et al. , 2017 ) for autoregressive teacher-net , as it requires much fewer output units compared to categorical distribution ( e.g. , 65,536 softmax units for 16-bit audios ) . Actually , the output distribution of student-net is required to be differentiable over samples x and allow backpropagation from teacher to student in distillation . As a result , one also needs to choose a continuous distribution for teacher WaveNet . Directly maximizing the log-likelihood of MoL is prone to numerical issues , and one has to employ the quantized surrogate loss introduced in PixelCNN++ . In this work , we demonstrate that a single Gaussian output distribution for WaveNet suffices to model the raw waveform . It might raise the modeling capacity concern because we use the single Gaussian instead of mixture of Gaussians ( Chung et al. , 2015 ) . We will demonstrate their comparable performance in experiment . Specifically , the conditional distribution of xt given previous samples is , p ( xt | x < t ; θ ) = N ( µ ( x < t ; θ ) , σ ( x < t ; θ ) ) , ( 2 ) where µ ( x < t ; θ ) and σ ( x < t ; θ ) are mean and standard deviation predicted by the autoregressive WaveNet , respectively . In practice , the network predicts log σ ( x < t ) and operates at log-scale for numerical stability . Given observed data , we do maximum likelihood estimation ( MLE ) for θ . Note that , the model may give very accurate prediction of 16-bit discrete xt without real-valued noise ( i.e. , µ ( x < t ) ≈ xt ) , then the log-likelihood calculation can become numerically unstable when it is free to minimize σ ( x < t ) . As a result , we clip the predicted log σ ( x < t ) at −9 ( natural logarithm ) before calculating the log-likelihood at training . 3 We discuss the importance of clipping constant for logscale in Appendix A . We also tried the dequantization trick by adding uniform noise u ∈ [ 0 , 265536 ] to the 16-bits samples similar as in image modeling ( e.g. , Uria et al. , 2013 ) . Indeed , these tricks are equivalent , in the sense that they both upper bound the continuous likelihood for modeling quantized data . We prefer the clipping trick , as it explicitly controls the model behavior and simplifies probability density distillation afterwards .

Summary: This paper proposes some modifications to established procedures for neural speech synthesis and investigates their effect experimentally. The proposed modifications are mostly fairly straightforward conceptually, but appear to work well, and this reviewer feels the paper has huge value in its experimental contributions extending and clarifying certain aspects of WaveNet training and distillation. The paper is well-written and fairly concise, with a short-and-sweet experimental results section.