BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

1 INTRODUCTION . Deep generative models have shown a tremendous advancement in speech synthesis ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ; Prenger et al. , 2019 ; Kumar et al. , 2019a ; Kong et al. , 2020b ; Chen et al. , 2020 ; Kong et al. , 2021 ) . Successful generative models can mainly divide into the GAN ( Goodfellow et al. , 2014 ) based approach and the likelihood-based approach . The former uses an adversarial training procedure , but the training is sometimes unstable , and the relevant training objectives are not suitable to compare different GAN models ; the latter uses log-likelihood or surrogate objectives for training , but they also have intrinsic limitations . For example , the autoregressive models ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ) have exceedingly slow sampling speed and poor scaling properties on high-dimensional data , as its sampling is inherently a sequential process . Likewise , the flow-based models ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ; Chen et al. , 2018 ; Papamakarios et al. , 2021 ) rely on specialized architectures to build a normalized probability model . The model training is less parameter-efficient and typically more complex than other likelihoodbased generative models ( Bond-Taylor et al. , 2021 ) . Some other prior works use the surrogate objectives , such as the evidence lower bound in variational auto-encoders ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ; Maaløe et al. , 2019 ) and the contrastive divergence in energy-based models ( Hinton , 2002 ; Carreira-Perpinan & Hinton , 2005 ) . They improve inference speed but often yield lower-quality samples than autoregressive and GAN-based models and typically only work well for low-dimensional data ( Bond-Taylor et al. , 2021 ) . An up-and-coming class of likelihood-based models is the diffusion probabilistic models ( DPMs ) ( Sohl-Dickstein et al. , 2015 ) , which shed light on the generation of high-quality samples comparable or even superior ( Dhariwal & Nichol , 2021 ) to the state-of-the-art ( SOTA ) GAN-based models . Sohl-Dickstein et al . ( 2015 ) introduced the idea of using an iterative forward process for destroying the structure of a given distribution while learning the reverse process for restoring the structure in the data . With a different modeling objective , the score-based generative models ( Song & Ermon , 2019 ) used a neural network trained with the score matching ( Hyvarinen & Dayan , 2005 ) to produce samples via Langevin dynamics . As a seminal work of DPMs , Ho et al . ( 2020 ) proposed the denoising diffusion probabilistic models ( DDPMs ) , which exploited a connection ( Vincent , 2011 ) between the above two ideas that the diffusion framework can be viewed as a specific approach to provide support to score matching . For the first time , DDPMs present high-quality image synthesis results on par with GAN models , especially on several higher-resolution image benchmarks . Subsequently , score matching and denoising diffusion models were applied to speech syntheses , such as Wavegrad ( Chen et al. , 2020 ) and a parallel contribution Diffwave ( Kong et al. , 2021 ) , and demonstrated the capability of generating high fidelity audio samples outperforming non-autoregressive GAN-based models ( Yamamoto et al. , 2020 ; Kumar et al. , 2019b ; Yang et al. , 2020 ; Bińkowski et al. , 2020 ) meanwhile matching the quality of SOTA autoregressive methods ( Chen et al. , 2020 ) . Despite the compelling results , the diffusion generative models are two to three orders of magnitude slower than other generative models such as GANs and VAEs . Their primary limitation is that they require up to thousands of diffusion steps during training to learn the target distribution . Therefore a large number of reverse steps are often required at sampling time . Recently , extensive investigations have been conducted to reduce the sampling steps for efficiently generating high-quality samples , which we will discuss in the related work in Section 2 . Distinctively , we conceived that we might train a neural network to efficiently and adaptively estimate a much shorter noise schedule for sampling while achieving generation performances comparable or superior to the conventional DPMs . With such an incentive , after introducing the conventional DPMs as our background in Section 3 , we propose in Section 4 bilateral denoising diffusion models ( BDDMs ) , named after a bilateral modeling perspective – parameterizing the forward and reverse processes with a schedule network and a score network , respectively . We theoretically derive that the schedule network should be trained after the score network is optimized . For training the schedule network , we propose a novel objective to minimize the gap between a newly derived lower bound and the log marginal likelihood . We describe the training algorithm as well as the fast and high-quality sampling algorithm in Section 5 . The training of the schedule network converges very fast using our newly derived objective , and its training only adds negligible overhead to DDPM ’ s . In Section 6 , our neural vocoding experiments demonstrated that BDDMs could generate high-fidelity samples with as few as three sampling steps . Moreover , our method can produce speech samples indistinguishable from human speech with only seven sampling steps ( 143x faster than WaveGrad and 28.6x faster than DiffWave ) . 2 RELATED WORK . Prior works showed that noise scheduling is crucial for efficient and high-fidelity sample generation in diffusion probabilistic models . DDPMs ( Ho et al. , 2020 ) shared a linear noise schedule for both training and sampling , which , however , requires thousands of sampling iterations to obtain competitive results . To speed up the sampling process , one class of related work , including ( Chen et al. , 2020 ; Kong et al. , 2021 ; Nichol & Dhariwal , 2021 ) , attempts to use a different , shorter noise schedule for sampling . For clarity , we thereafter denote the training noise schedule as β ∈ RT and the sampling noise schedule as β̂ ∈ RN with N < T . In particular , Chen et al . ( 2020 ) applied a grid search ( GS ) algorithm to select β̂ . Unfortunately , GS becomes prohibitively slow when N grows large , e.g. , N = 6 took more than a day on a single NVIDIA Tesla P40 GPU . This is because the time costs of GS algorithm grow exponentially withN , i.e. , O ( 9N ) with 9 bins as the default setting in Chen et al . ( 2020 ) . Instead of searching , Kong et al . ( 2021 ) devised a fast sampling ( FS ) algorithm based on an expert-defined 6-step noise schedule for their score network . However , this specifically tuned noise schedule is hard to generalize to other score networks , tasks , or datasets . Another class of noise scheduling methods searches for a subsequence of time indices of the training noise schedule , which we call the time schedule . DDIMs ( Song et al. , 2021a ) introduced an accelerated reverse process that relies on a pre-specified time schedule . A linear and a quadratic time schedule were used in DDIMs and showed superior generation quality over DDPMs within 10 to 100 sampling steps . Nichol & Dhariwal ( 2021 ) proposed a re-scaled noise schedule for fast sampling , but this also requires pre-specifying the time schedule and the training noise schedule . Nichol & Dhariwal ( 2021 ) also proposed learning variances for the reverse processes , whereas the variances of the forward processes , i.e. , the noise schedule , which affected both the means and variances of the reverse processes , were not learnable . According to the results of ( Song et al. , 2021a ; Nichol & Dhariwal , 2021 ) , using a linear or quadratic time schedule resulted in quite different performances in different datasets , implying that the optimal choice of schedule varies with the datasets . So , there remains a challenge in finding a short and effective schedule for fast sampling on different datasets . Notably , Kong & Ping ( 2021 ) proposed a method to map a noise schedule to a time schedule for fast sampling . In this sense , searching for a time schedule becomes a sub-set of the noise scheduling problem , which resembles the above category of methods . Although DPMs ( Sohl-Dickstein et al. , 2015 ) and DDPMs ( Ho et al. , 2019 ) mentioned that the noise schedule could be learned by re-parameterization , the approach was not investigated in their works . Closely related works that learn a noise schedule emerged until very recently . San-Roman et al . ( 2021 ) proposed a noise estimation ( NE ) method , which trained a neural net with a regression loss to estimate the noise scale from the noisy sample at each time point , and then predicted the next noise scale . However , NE requires a prior assumption of the noise schedule following a linear or Fibonacci rule . Most recently , a concurrent work to ours by Kingma et al . ( 2021 ) jointly trained a neural net to predict the signal-to-noise ratio ( SNR ) by maximizing the variational lower bound . The SNR was then used for noise scheduling . Different from ours , this scheduling neural net only took t as input and is independent of the noisy sample generated during the loop of sampling process . Intrinsically , with limited information about the sampled data , the predicted SNR could deviate from the actual SNR of the noisy data during sampling . 3 BACKGROUND . 3.1 DIFFUSION PROBABILISTIC MODELS ( DPMS ) . Given i.i.d . samples { x0 ∈ RD } from an unknown data distribution pdata ( x0 ) , diffusion probabilistic models ( DPMs ) ( Sohl-Dickstein et al. , 2015 ) define a forward process q ( x1 : T |x0 ) =∏T t=1 q ( xt|xt−1 ) that converts any complex data distribution into a simple , tractable distribution after T steps of diffusion . A reverse process pθ ( xt−1|xt ) parameterized by θ is used to model the data distribution : pθ ( x0 ) = ∫ π ( xT ) ∏T t=1 pθ ( xt−1|xt ) dx1 : T , where π ( xT ) is the prior distribution for starting the reverse process . Then , the variational parameters θ can be learned by maximizing the standard log evidence lower bound ( ELBO ) : Felbo : = Eq [ log pθ ( x0|x1 ) − T∑ t=2 DKL ( q ( xt−1|xt , x0 ) ||pθ ( xt−1|xt ) ) −DKL ( q ( xT |x0 ) ||π ( xT ) ) ] . ( 1 ) 3.2 DENOISING DIFFUSION PROBABILISTIC MODELS ( DDPMS ) . As an extension to DPMs , denoising diffusion probabilistic models ( DDPMs ) ( Ho et al. , 2020 ) applied the score matching technique ( Hyvarinen & Dayan , 2005 ; Song & Ermon , 2019 ) to define the reverse process . In particular , DDPMs considered a Gaussian diffusion process parameterized by a noise schedule β ∈ RT with 0 < β1 , . . . , βT < 1 : qβ ( x1 : T |x0 ) : = T∏ t=1 qβt ( xt|xt−1 ) , where qβt ( xt|xt−1 ) : = N ( √ 1− βtxt−1 , βtI ) . ( 2 ) Based on the nice property of isotropic Gaussians , one can express xt directly conditioned on x0 : qβ ( xt|x0 ) = N ( αtx0 , ( 1− α2t ) I ) , where αt = t∏ i=1 √ 1− βi . ( 3 ) To revert this forward process , DDPMs employ a score network1 θ ( xt , αt ) to define pθ ( xt−1|xt ) : = N ( 1√ 1− βt ( xt − βt√ 1− α2t θ ( xt , αt ) ) , Σt ) , ( 4 ) 1Here , θ ( xt , αt ) is conditioned on the continuous noise scale αt , as in ( Song et al. , 2021b ; Chen et al. , 2020 ) . Alternatively , the score network can also be conditioned on a discrete time index θ ( xt , t ) , as in ( Song et al. , 2021a ; Ho et al. , 2020 ) . An approximate mapping of a noise schedule to a time schedule ( Kong & Ping , 2021 ) exists , therefore we consider conditioning on noise scales as the general case . where Σt is the co-variance matrix defined for the reverse process . Ho et al . ( 2020 ) showed that setting Σt = β̃tI = 1−α2t−1 1−α2t βtI is optimal for a deterministic x0 , while setting Σt = βtI is optimal for a white noise x0 ∼ N ( 0 , I ) . Alternatively , Nichol & Dhariwal ( 2021 ) proposed learnable variances by interpolating the two optimals with a jointly trained neural network , i.e. , Σt , θ ( x ) : = diag ( exp ( vθ ( x ) log βt + ( 1− vθ ( x ) ) log β̃t ) ) , where vθ ( x ) ∈ RD is a trainable network . Note that the calculation of the complete ELBO in Eq . ( 1 ) requires T forward passes of the score network , which would make the training computationally prohibitive for a large T . To feasibly train the score network , instead of computing the complete ELBO , Ho et al . ( 2020 ) proposed an efficient training mechanism by sampling from a discrete uniform distribution : t ∼ U { 1 , ... , T } , x0 ∼ pdata ( x0 ) , t ∼ N ( 0 , I ) at each training iteration to compute the training loss : L ( t ) ddpm ( θ ) : = ∥∥∥∥ t − θ ( αtx0 +√1− α2t t , αt ) ∥∥∥∥2 2 , ( 5 ) which is a re-weighted form ofDKL ( qβ ( xt−1|xt , x0 ) ||pθ ( xt−1|xt ) ) . Ho et al . ( 2020 ) reported that the re-weighting worked effectively for learning θ . Yet , we demonstrate it is deficient for learning the noise schedule β in our ablation experiment in Section 6.2 .

Summary: The paper proposes to train a scheduling network in addition to the score network for fast sampling of diffusion models. To train such network a new loss is proposed. Results are presented for the task of neural vocoding (single-speaker and multi-speaker).