BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

1 INTRODUCTION . Deep generative models have shown a tremendous advancement in speech synthesis ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ; Prenger et al. , 2019 ; Kumar et al. , 2019a ; Kong et al. , 2020b ; Chen et al. , 2020 ; Kong et al. , 2021 ) . Successful generative models can mainly divide into the GAN ( Goodfellow et al. , 2014 ) based approach and the likelihood-based approach . The former uses an adversarial training procedure , but the training is sometimes unstable , and the relevant training objectives are not suitable to compare different GAN models ; the latter uses log-likelihood or surrogate objectives for training , but they also have intrinsic limitations . For example , the autoregressive models ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ) have exceedingly slow sampling speed and poor scaling properties on high-dimensional data , as its sampling is inherently a sequential process . Likewise , the flow-based models ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ; Chen et al. , 2018 ; Papamakarios et al. , 2021 ) rely on specialized architectures to build a normalized probability model . The model training is less parameter-efficient and typically more complex than other likelihoodbased generative models ( Bond-Taylor et al. , 2021 ) . Some other prior works use the surrogate objectives , such as the evidence lower bound in variational auto-encoders ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ; Maaløe et al. , 2019 ) and the contrastive divergence in energy-based models ( Hinton , 2002 ; Carreira-Perpinan & Hinton , 2005 ) . They improve inference speed but often yield lower-quality samples than autoregressive and GAN-based models and typically only work well for low-dimensional data ( Bond-Taylor et al. , 2021 ) . An up-and-coming class of likelihood-based models is the diffusion probabilistic models ( DPMs ) ( Sohl-Dickstein et al. , 2015 ) , which shed light on the generation of high-quality samples comparable or even superior ( Dhariwal & Nichol , 2021 ) to the state-of-the-art ( SOTA ) GAN-based models . Sohl-Dickstein et al . ( 2015 ) introduced the idea of using an iterative forward process for destroying the structure of a given distribution while learning the reverse process for restoring the structure in the data . With a different modeling objective , the score-based generative models ( Song & Ermon , 2019 ) used a neural network trained with the score matching ( Hyvarinen & Dayan , 2005 ) to produce samples via Langevin dynamics . As a seminal work of DPMs , Ho et al . ( 2020 ) proposed the denoising diffusion probabilistic models ( DDPMs ) , which exploited a connection ( Vincent , 2011 ) between the above two ideas that the diffusion framework can be viewed as a specific approach to provide support to score matching . For the first time , DDPMs present high-quality image synthesis results on par with GAN models , especially on several higher-resolution image benchmarks . Subsequently , score matching and denoising diffusion models were applied to speech syntheses , such as Wavegrad ( Chen et al. , 2020 ) and a parallel contribution Diffwave ( Kong et al. , 2021 ) , and demonstrated the capability of generating high fidelity audio samples outperforming non-autoregressive GAN-based models ( Yamamoto et al. , 2020 ; Kumar et al. , 2019b ; Yang et al. , 2020 ; Bińkowski et al. , 2020 ) meanwhile matching the quality of SOTA autoregressive methods ( Chen et al. , 2020 ) . Despite the compelling results , the diffusion generative models are two to three orders of magnitude slower than other generative models such as GANs and VAEs . Their primary limitation is that they require up to thousands of diffusion steps during training to learn the target distribution . Therefore a large number of reverse steps are often required at sampling time . Recently , extensive investigations have been conducted to reduce the sampling steps for efficiently generating high-quality samples , which we will discuss in the related work in Section 2 . Distinctively , we conceived that we might train a neural network to efficiently and adaptively estimate a much shorter noise schedule for sampling while achieving generation performances comparable or superior to the conventional DPMs . With such an incentive , after introducing the conventional DPMs as our background in Section 3 , we propose in Section 4 bilateral denoising diffusion models ( BDDMs ) , named after a bilateral modeling perspective – parameterizing the forward and reverse processes with a schedule network and a score network , respectively . We theoretically derive that the schedule network should be trained after the score network is optimized . For training the schedule network , we propose a novel objective to minimize the gap between a newly derived lower bound and the log marginal likelihood . We describe the training algorithm as well as the fast and high-quality sampling algorithm in Section 5 . The training of the schedule network converges very fast using our newly derived objective , and its training only adds negligible overhead to DDPM ’ s . In Section 6 , our neural vocoding experiments demonstrated that BDDMs could generate high-fidelity samples with as few as three sampling steps . Moreover , our method can produce speech samples indistinguishable from human speech with only seven sampling steps ( 143x faster than WaveGrad and 28.6x faster than DiffWave ) . 2 RELATED WORK . Prior works showed that noise scheduling is crucial for efficient and high-fidelity sample generation in diffusion probabilistic models . DDPMs ( Ho et al. , 2020 ) shared a linear noise schedule for both training and sampling , which , however , requires thousands of sampling iterations to obtain competitive results . To speed up the sampling process , one class of related work , including ( Chen et al. , 2020 ; Kong et al. , 2021 ; Nichol & Dhariwal , 2021 ) , attempts to use a different , shorter noise schedule for sampling . For clarity , we thereafter denote the training noise schedule as β ∈ RT and the sampling noise schedule as β̂ ∈ RN with N < T . In particular , Chen et al . ( 2020 ) applied a grid search ( GS ) algorithm to select β̂ . Unfortunately , GS becomes prohibitively slow when N grows large , e.g. , N = 6 took more than a day on a single NVIDIA Tesla P40 GPU . This is because the time costs of GS algorithm grow exponentially withN , i.e. , O ( 9N ) with 9 bins as the default setting in Chen et al . ( 2020 ) . Instead of searching , Kong et al . ( 2021 ) devised a fast sampling ( FS ) algorithm based on an expert-defined 6-step noise schedule for their score network . However , this specifically tuned noise schedule is hard to generalize to other score networks , tasks , or datasets . Another class of noise scheduling methods searches for a subsequence of time indices of the training noise schedule , which we call the time schedule . DDIMs ( Song et al. , 2021a ) introduced an accelerated reverse process that relies on a pre-specified time schedule . A linear and a quadratic time schedule were used in DDIMs and showed superior generation quality over DDPMs within 10 to 100 sampling steps . Nichol & Dhariwal ( 2021 ) proposed a re-scaled noise schedule for fast sampling , but this also requires pre-specifying the time schedule and the training noise schedule . Nichol & Dhariwal ( 2021 ) also proposed learning variances for the reverse processes , whereas the variances of the forward processes , i.e. , the noise schedule , which affected both the means and variances of the reverse processes , were not learnable . According to the results of ( Song et al. , 2021a ; Nichol & Dhariwal , 2021 ) , using a linear or quadratic time schedule resulted in quite different performances in different datasets , implying that the optimal choice of schedule varies with the datasets . So , there remains a challenge in finding a short and effective schedule for fast sampling on different datasets . Notably , Kong & Ping ( 2021 ) proposed a method to map a noise schedule to a time schedule for fast sampling . In this sense , searching for a time schedule becomes a sub-set of the noise scheduling problem , which resembles the above category of methods . Although DPMs ( Sohl-Dickstein et al. , 2015 ) and DDPMs ( Ho et al. , 2019 ) mentioned that the noise schedule could be learned by re-parameterization , the approach was not investigated in their works . Closely related works that learn a noise schedule emerged until very recently . San-Roman et al . ( 2021 ) proposed a noise estimation ( NE ) method , which trained a neural net with a regression loss to estimate the noise scale from the noisy sample at each time point , and then predicted the next noise scale . However , NE requires a prior assumption of the noise schedule following a linear or Fibonacci rule . Most recently , a concurrent work to ours by Kingma et al . ( 2021 ) jointly trained a neural net to predict the signal-to-noise ratio ( SNR ) by maximizing the variational lower bound . The SNR was then used for noise scheduling . Different from ours , this scheduling neural net only took t as input and is independent of the noisy sample generated during the loop of sampling process . Intrinsically , with limited information about the sampled data , the predicted SNR could deviate from the actual SNR of the noisy data during sampling . 3 BACKGROUND . 3.1 DIFFUSION PROBABILISTIC MODELS ( DPMS ) . Given i.i.d . samples { x0 ∈ RD } from an unknown data distribution pdata ( x0 ) , diffusion probabilistic models ( DPMs ) ( Sohl-Dickstein et al. , 2015 ) define a forward process q ( x1 : T |x0 ) =∏T t=1 q ( xt|xt−1 ) that converts any complex data distribution into a simple , tractable distribution after T steps of diffusion . A reverse process pθ ( xt−1|xt ) parameterized by θ is used to model the data distribution : pθ ( x0 ) = ∫ π ( xT ) ∏T t=1 pθ ( xt−1|xt ) dx1 : T , where π ( xT ) is the prior distribution for starting the reverse process . Then , the variational parameters θ can be learned by maximizing the standard log evidence lower bound ( ELBO ) : Felbo : = Eq [ log pθ ( x0|x1 ) − T∑ t=2 DKL ( q ( xt−1|xt , x0 ) ||pθ ( xt−1|xt ) ) −DKL ( q ( xT |x0 ) ||π ( xT ) ) ] . ( 1 ) 3.2 DENOISING DIFFUSION PROBABILISTIC MODELS ( DDPMS ) . As an extension to DPMs , denoising diffusion probabilistic models ( DDPMs ) ( Ho et al. , 2020 ) applied the score matching technique ( Hyvarinen & Dayan , 2005 ; Song & Ermon , 2019 ) to define the reverse process . In particular , DDPMs considered a Gaussian diffusion process parameterized by a noise schedule β ∈ RT with 0 < β1 , . . . , βT < 1 : qβ ( x1 : T |x0 ) : = T∏ t=1 qβt ( xt|xt−1 ) , where qβt ( xt|xt−1 ) : = N ( √ 1− βtxt−1 , βtI ) . ( 2 ) Based on the nice property of isotropic Gaussians , one can express xt directly conditioned on x0 : qβ ( xt|x0 ) = N ( αtx0 , ( 1− α2t ) I ) , where αt = t∏ i=1 √ 1− βi . ( 3 ) To revert this forward process , DDPMs employ a score network1 θ ( xt , αt ) to define pθ ( xt−1|xt ) : = N ( 1√ 1− βt ( xt − βt√ 1− α2t θ ( xt , αt ) ) , Σt ) , ( 4 ) 1Here , θ ( xt , αt ) is conditioned on the continuous noise scale αt , as in ( Song et al. , 2021b ; Chen et al. , 2020 ) . Alternatively , the score network can also be conditioned on a discrete time index θ ( xt , t ) , as in ( Song et al. , 2021a ; Ho et al. , 2020 ) . An approximate mapping of a noise schedule to a time schedule ( Kong & Ping , 2021 ) exists , therefore we consider conditioning on noise scales as the general case . where Σt is the co-variance matrix defined for the reverse process . Ho et al . ( 2020 ) showed that setting Σt = β̃tI = 1−α2t−1 1−α2t βtI is optimal for a deterministic x0 , while setting Σt = βtI is optimal for a white noise x0 ∼ N ( 0 , I ) . Alternatively , Nichol & Dhariwal ( 2021 ) proposed learnable variances by interpolating the two optimals with a jointly trained neural network , i.e. , Σt , θ ( x ) : = diag ( exp ( vθ ( x ) log βt + ( 1− vθ ( x ) ) log β̃t ) ) , where vθ ( x ) ∈ RD is a trainable network . Note that the calculation of the complete ELBO in Eq . ( 1 ) requires T forward passes of the score network , which would make the training computationally prohibitive for a large T . To feasibly train the score network , instead of computing the complete ELBO , Ho et al . ( 2020 ) proposed an efficient training mechanism by sampling from a discrete uniform distribution : t ∼ U { 1 , ... , T } , x0 ∼ pdata ( x0 ) , t ∼ N ( 0 , I ) at each training iteration to compute the training loss : L ( t ) ddpm ( θ ) : = ∥∥∥∥ t − θ ( αtx0 +√1− α2t t , αt ) ∥∥∥∥2 2 , ( 5 ) which is a re-weighted form ofDKL ( qβ ( xt−1|xt , x0 ) ||pθ ( xt−1|xt ) ) . Ho et al . ( 2020 ) reported that the re-weighting worked effectively for learning θ . Yet , we demonstrate it is deficient for learning the noise schedule β in our ablation experiment in Section 6.2 .

Summary: The paper aims at improving generative denoising diffusion models for audio synthesis. In particular, it proposes a method to learn a noise prediction or scheduling network that optimally chooses the noise scales (this is, the scales of the noise that is injected during the iterative synthesis process) at inference time. This allows for faster synthesis without much reduction in quality. The method, called Bilateral Denoising Diffusion Model (BDDM), relies on a slightly modified training objective compared to previous works, in which case learning the noise schedule corresponds to optimizing the objective towards a tighter bound on the data log-likelihood. The main score model is learned in a similar fashion like previous diffusion models, such that the objectives for the score model and the noise scheduling network decompose into two separate objectives that can be optimized essentially one after another. Experimentally, BDDM achieves audio generation with a quality almost as good as the current state-of-the-art, while requiring much less synthesis steps. It also outperforms other methods for accelerated sampling from diffusion models for audio synthesis.