DiffWave: A Versatile Diffusion Model for Audio Synthesis

1 INTRODUCTION . Deep generative models have produced high-fidelity raw audio in speech synthesis and music generation . In previous work , likelihood-based models , including autoregressive models ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ; Mehri et al. , 2017 ) and flow-based models ( Prenger et al. , 2019 ; Ping et al. , 2020 ; Kim et al. , 2019 ) , have predominated in audio synthesis because of the simple training objective and superior ability of modeling the fine details of waveform in real data . There are other waveform models , which often require auxiliary losses for training , such as flow-based models trained by distillation ( van den Oord et al. , 2018 ; Ping et al. , 2019 ) , variational auto-encoder ( VAE ) based model ( Peng et al. , 2020 ) , and generative adversarial network ( GAN ) based models ( Kumar et al. , 2019 ; Bińkowski et al. , 2020 ; Yamamoto et al. , 2020 ) . Most of previous waveform models focus on audio synthesis with informative local conditioner ( e.g. , mel spectrogram or aligned linguistic features ) , with only a few exceptions for unconditional generation ( Mehri et al. , 2017 ; Donahue et al. , 2019 ) . It has been noticed that autoregressive models ( e.g. , WaveNet ) tend to generate made-up word-like sounds ( van den Oord et al. , 2016 ) , or inferior samples ( Donahue et al. , 2019 ) under unconditional settings . This is because very long sequences need to be generated ( e.g. , 16,000 time-steps for one second speech ) without any conditional information . Diffusion probabilistic models ( diffusion models for brevity ) are a class of promising generative models , which use a Markov chain to gradually convert a simple distribution ( e.g. , isotropic Gaussian ) into complicated data distribution ( Sohl-Dickstein et al. , 2015 ; Goyal et al. , 2017 ; Ho et al. , 2020 ) . Although the data likelihood is intractable , diffusion models can be efficiently trained by optimizing the variational lower bound ( ELBO ) . Most recently , a certain parameterization has been shown successful in image synthesis ( Ho et al. , 2020 ) , which is connected with denoising score matching ( Song ∗Contributed to the work during an internship at Baidu Research , USA . 1Audio samples are in : https : //diffwave-demo.github.io/ & Ermon , 2019 ) . Diffusion models can use a diffusion ( noise-adding ) process without learnable parameters to obtain the “ whitened ” latents from training data . Therefore , no additional neural networks are required for training in contrast to other models ( e.g. , the encoder in VAE ( Kingma & Welling , 2014 ) or the discriminator in GAN ( Goodfellow et al. , 2014 ) ) . This avoids the challenging “ posterior collapse ” or “ mode collapse ” issues stemming from the joint training of two networks , and hence is valuable for high-fidelity audio synthesis . In this work , we propose DiffWave , a versatile diffusion probabilistic model for raw audio synthesis . DiffWave has several advantages over previous work : i ) It is non-autoregressive thus can synthesize high-dimensional waveform in parallel . ii ) It is flexible as it does not impose any architectural constraints in contrast to flow-based models , which need to keep the bijection between latents and data ( e.g. , see more analysis in Ping et al . ( 2020 ) ) . This leads to small-footprint neural vocoders that still generate high-fidelity speech . iii ) It uses a single ELBO-based training objective without any auxiliary losses ( e.g. , spectrogram-based losses ) for high-fidelity synthesis . iv ) It is a versatile model that produces high-quality audio signals for both conditional and unconditional waveform generation . Specifically , we make the following contributions : 1 . DiffWave uses a feed-forward and bidirectional dilated convolution architecture motivated by WaveNet ( van den Oord et al. , 2016 ) . It matches the strong WaveNet vocoder in terms of speech quality ( MOS : 4.44 vs. 4.43 ) , while synthesizing orders of magnitude faster as it only requires a few sequential steps ( e.g. , 6 ) for generating very long waveforms . 2 . Our small DiffWave has 2.64M parameters and synthesizes 22.05 kHz high-fidelity speech ( MOS : 4.37 ) more than 5× faster than real-time on a V100 GPU without engineered kernels . Although it is still slower than the state-of-the-art flow-based models ( Ping et al. , 2020 ; Prenger et al. , 2019 ) , it has much smaller footprint . We expect further speed-up by optimizing its inference mechanism in the future . 3 . DiffWave significantly outperforms WaveGAN ( Donahue et al. , 2019 ) and WaveNet in the challenging unconditional and class-conditional waveform generation tasks in terms of audio quality and sample diversity measured by several automatic and human evaluations . We organize the rest of the paper as follows . We present the diffusion models in Section 2 , and introduce DiffWave architecture in Section 3 . Section 4 discusses related work . We report experimental results in Section 5 and conclude the paper in Section 6 . 2 DIFFUSION PROBABILISTIC MODELS . We define qdata ( x0 ) as the data distribution on RL , where L is the data dimension . Let xt ∈ RL for t = 0 , 1 , · · · , T be a sequence of variables with the same dimension , where t is the index for diffusion steps . Then , a diffusion model of T steps is composed of two processes : the diffusion process , and the reverse process ( Sohl-Dickstein et al. , 2015 ) . Both of them are illustrated in Figure 1 . Algorithm 1 Training for i = 1 , 2 , · · · , Niter do Sample x0 ∼ qdata , ∼ N ( 0 , I ) , and t ∼ Uniform ( { 1 , · · · , T } ) Take gradient step on ∇θ‖ − θ ( √ ᾱtx0 + √ 1− ᾱt , t ) ‖22 according to Eq . ( 7 ) end for Algorithm 2 Sampling Sample xT ∼ platent = N ( 0 , I ) for t = T , T − 1 , · · · , 1 do Compute µθ ( xt , t ) and σθ ( xt , t ) using Eq . ( 5 ) Sample xt−1 ∼ pθ ( xt−1|xt ) = N ( xt−1 ; µθ ( xt , t ) , σθ ( xt , t ) 2I ) end for return x0 The diffusion process is defined by a fixed Markov chain from data x0 to the latent variable xT : q ( x1 , · · · , xT |x0 ) = T∏ t=1 q ( xt|xt−1 ) , ( 1 ) where each of q ( xt|xt−1 ) is fixed to N ( xt ; √ 1− βtxt−1 , βtI ) for a small positive constant βt . The function of q ( xt|xt−1 ) is to add small Gaussian noise to the distribution of xt−1 . The whole process gradually converts data x0 to whitened latents xT according to a variance schedule β1 , · · · , βT . 2 The reverse process is defined by a Markov chain from xT to x0 parameterized by θ : platent ( xT ) = N ( 0 , I ) , and pθ ( x0 , · · · , xT−1|xT ) = T∏ t=1 pθ ( xt−1|xt ) , ( 2 ) where platent ( xT ) is isotropic Gaussian , and the transition probability pθ ( xt−1|xt ) is parameterized asN ( xt−1 ; µθ ( xt , t ) , σθ ( xt , t ) 2I ) with shared parameter θ . Note that both µθ and σθ take two inputs : the diffusion-step t ∈ N , and variable xt ∈ RL . µθ outputs an L-dimensional vector as the mean , and σθ outputs a real number as the standard deviation . The goal of pθ ( xt−1|xt ) is to eliminate the Gaussian noise ( i.e . denoise ) added in the diffusion process . Sampling : Given the reverse process , the generative procedure is to first sample an xT ∼ N ( 0 , I ) , and then sample xt−1 ∼ pθ ( xt−1|xt ) for t = T , T − 1 , · · · , 1 . The output x0 is the sampled data . Training : The likelihood pθ ( x0 ) = ∫ pθ ( x0 , · · · , xT−1|xT ) · platent ( xT ) dx1 : T is intractable to calculate in general . The model is thus trained by maximizing its variational lower bound ( ELBO ) : Eqdata ( x0 ) log pθ ( x0 ) = Eqdata ( x0 ) logEq ( x1 , ··· , xT |x0 ) [ pθ ( x0 , · · · , xT−1|xT ) × platent ( xT ) q ( x1 , · · · , xT |x0 ) ] ≥ Eq ( x0 , ··· , xT ) log pθ ( x0 , · · · , xT−1|xT ) × platent ( xT ) q ( x1 , · · · , xT |x0 ) : = ELBO . ( 3 ) Most recently , Ho et al . ( 2020 ) showed that under a certain parameterization , the ELBO of the diffusion model can be calculated in closed-form . This accelerates the computation and avoids Monte Carlo estimates , which have high variance . This parameterization is motivated by its connection to denoising score matching with Langevin dynamics ( Song & Ermon , 2019 ; 2020 ) . To introduce this parameterization , we first define some constants based on the variance schedule { βt } Tt=1 in the diffusion process as in Ho et al . ( 2020 ) : αt = 1− βt , ᾱt = t∏ s=1 αs , β̃t = 1− ᾱt−1 1− ᾱt βt for t > 1 and β̃1 = β1 . ( 4 ) Then , the parameterizations of µθ and σθ are defined by µθ ( xt , t ) = 1 √ αt ( xt − βt√ 1− ᾱt θ ( xt , t ) ) , and σθ ( xt , t ) = β̃ 1 2 t , ( 5 ) where θ : RL × N→ RL is a neural network also taking xt and the diffusion-step t as inputs . Note that σθ ( xt , t ) is fixed to a constant β̃ 1 2 t for every step t under this parameterization . In the following proposition , we explicitly provide the closed-form expression of the ELBO . 2One can find that q ( xT |x0 ) approaches to isotropic Gaussian with large T in Eq . ( 11 ) in the Appendix A . Proposition 1 . ( Ho et al. , 2020 ) Suppose a series of fixed schedule { βt } Tt=1 are given . Let ∼ N ( 0 , I ) and x0 ∼ qdata . Then , under the parameterization in Eq . ( 5 ) , we have − ELBO = c+ T∑ t=1 κtEx0 , ‖ − θ ( √ ᾱtx0 + √ 1− ᾱt , t ) ‖22 ( 6 ) for some constants c and κt , where κt = βt2αt ( 1−ᾱt−1 ) for t > 1 , and κ1 = 1 2α1 . Note that c is irrelevant for optimization purpose . The key idea in the proof is to expand the ELBO into a sum of KL divergences between tractable Gaussian distributions , which have a closed-form expression . We refer the readers to look at Section A in the Appendix for the full proof . In addition , Ho et al . ( 2020 ) reported that minimizing the following unweighted variant of the ELBO leads to higher generation quality : min θ Lunweighted ( θ ) = Ex0 , , t ‖ − θ ( √ ᾱtx0 + √ 1− ᾱt , t ) ‖22 ( 7 ) where t is uniformly taken from 1 , · · · , T . Therefore , we also use this training objective in this paper . We summarize the training and sampling procedures in Algorithm 1 and 2 , respectively . Fast sampling : Given a trained model from Algorithm 1 , we noticed that the most effective denoising steps at sampling occur near t = 0 ( see Section IV on demo website ) . This encourages us to design a fast sampling algorithm with much fewer denoising steps Tinfer ( e.g. , 6 ) than T at training ( e.g. , 200 ) . The key idea is to “ collapse ” the T -step reverse process into a Tinfer-step process with carefully designed variance schedule . We provide the details in Appendix B .

Summary: This paper describes a neural vocoder based on a diffusion probabilistic model. The model utilizes a fixed-length markov chain to convert between a latent uncorrelated Gaussian vector and a full-length observation. The conversion from observation to latent is fixed and amounts to adding noise at each step. The conversion from latent to observation reveals slightly more of the observation from the latent at each step via a sort of cancellation. This process is derived theoretically based on maximizing the variational lower bound (ELBO) of the model and follows Ho et al. (2020) who derived it for image generation. Thorough experiments show that the model produces high quality speech syntheses on the LJ dataset (MOS comparable to WaveNet and real speech) when conditionally synthesizing from the true mel spectrogram, while generating much more quickly than WaveNet. Perhaps more interesting and surprising, however, is that it generates very high quality and intelligible short utterances with no conditioning, and also admits to global conditioning, e.g., with a digit label.