Denoising Diffusion Gamma Models

1 INTRODUCTION . Deep generative neural networks have shown significant progress over the last years . The main architectures for generation are : ( i ) VAE ( Kingma & Welling , 2013 ) based , for example , NVAE ( Vahdat & Kautz , 2020 ) and VQ-VAE ( Razavi et al. , 2019 ) , ( ii ) GAN ( Goodfellow et al. , 2014 ) based , for example , StyleGAN ( Karras et al. , 2020 ) for vision application and WaveGAN ( Donahue et al. , 2018 ) for speech ( iii ) Flow-based , for example Glow ( Kingma & Dhariwal , 2018 ) ( iv ) Autoregessive , for example , Wavenet for speech ( Oord et al. , 2016 ) and ( v ) Diffusion Probabilistic Models ( SohlDickstein et al. , 2015 ) , for example , Denoising Diffusion Probabilistic Models ( DDPM ) ( Ho et al. , 2020 ) and its implicit version DDIM ( Song et al. , 2020a ) . Models from this last family have shown significant progress in generation capabilities in the last years , e.g. , ( Chen et al. , 2020 ; Kong et al. , 2020b ) , and have achieved results comparable to state-of-the-art generation architecture for both images and speech . A DDPM is a Markov chain of latent variables . Two processes are modeled : ( i ) a diffusion process and ( ii ) a denoising process . During training , the diffusion process learns to transform data samples into Gaussian noise . Denoising is the reverse process and it is used during inference for generating data samples , starting from Gaussian noise . The second process can be conditioned on attributes to control the generation sample . To obtain high-quality synthesis , a large number of denoising steps is used ( i.e . 1000 steps ) . A notable property of the diffusion process is a closed-form formulation of the noise that arises from accumulating diffusion stems . This allows sampling arbitrary states in the Markov chain of the diffusion process without calculating the previous steps . In the Gaussian case , this property stems from the fact that adding Gaussian random variables leads to another Gaussian random variable . Other distributions have similar properties . For example , for the Gamma distribution , the sum of two random variables that share the scale parameter is a Gamma random variable of the same scale . The Poisson distribution has a similar property . However , its discrete nature makes it less suitable for DDPM . In DDPM , the mean of the Gaussian random variables is set at zero . The Gamma random variable , with its two parameters ( shape and scale ) , is better suited to fit the data than a Gaussian random variable with one degree of freedom ( scale ) . Furthermore , the Gamma random variable generalizes other distributions , and many other distributions can be derived from it ( Leemis & McQueston , 2008 ) . The added modeling capacity of the Gamma random variable can help speed up the convergence of the DDPM model . Consider , for example , a conventional DDPM model that was trained with Gaussian noise on the CelebA dataset ( Liu et al. , 2015 ) . The noise distribution throughout the diffusion process can be visualized by computing the histogram of the estimated residual noise in the generation process . The estimated residual noise ̂ is given by difference between x0 and the image xt after t DDPM steps ̂ = √ ᾱtx0−xt√ 1−|ᾱt| . The model is a pretrained DDPM ( Gaussian ) celebA ( 64x64 ) model . ( a ) The fitting of a Gaussian to the histogram of a typical image after t− 50 steps . ( b ) Fitting a Gamma distribution . ( c ) The fitting error to Gaussian and Gamma distribution , measured as the MSE between the histogram and the fitted probability distribution function . Each point is the average value for the generation of 100 images . The vertical error bars denote the standard deviation . ̂ = √ ᾱtx0−xt√ 1−|ᾱt| , where ᾱt is the noise schedule , x0 is the data point and xt is the estimate state at timestep t , as can be derived from Eq.4 from ( Song et al. , 2020a ) . Both a Gaussian distribution and Gamma distribution can then be fitted to this histogram , as shown in Fig . 1 ( a , b ) . As can be seen , the Gamma distribution provides a better fit to the estimated residual noise ̂ . Moreover , Fig . 1 ( c ) presents the mean fitting error between the histogram and the fitted probability distribution function . Evidently , the Gamma distribution is a better fit than the Gaussian distribution . While the model was trained to estimate Gaussian noise , at inference time it has to address a different distribution . In this paper , we investigate the non-Gaussian Gamma noise distribution . As noted , this distribution seems to fit the histogram of the generation error better than the Gaussian distribution , and it also has favorable properties such as its behavior under addition and scalar multiplication . The proposed models maintain the property of the diffusion process of sampling arbitrary states without calculating the previous steps . Our results are demonstrated in two major domains : vision and audio . In the first domain , the proposed method is shown to provide a better FID score for generated images . For speech data , we show that the proposed method improves various measures , such as Perceptual Evaluation of Speech Quality ( PESQ ) and short-time objective intelligibility ( STOI ) . 2 RELATED WORK . In their seminal work , Sohl-Dickstein et al . ( 2015 ) introduce the Diffusion Probabilistic Model . This model is applied to various domains , such as time series and images . The main drawback in the proposed model is that it needs up to thousands of iterative steps to generate a valid data sample . Song & Ermon ( 2019 ) proposed a diffusion generative model based on Langevin dynamics and the score matching method ( Hyvärinen & Dayan , 2005 ) . The model estimates the Stein score function ( Liu et al. , 2016 ) which is the gradient of the logarithm of data density . Given the Stein score function , the model can generate data points . Denoising Diffusion Probabilistic Models ( DDPM ) ( Ho et al. , 2020 ) combine generative models based on score matching and neural Diffusion Probabilistic Models into a single model . Similarly , in Chen et al . ( 2020 ) ; Kong et al . ( 2020a ) a generative neural diffusion process based on score matching was applied to speech generation . These models achieve state-of-the-art results for speech generation , and show superior results over well-established methods , such as Wavernn ( Kalchbrenner et al. , 2018 ) , Wavenet ( Oord et al. , 2016 ) , and GAN-TTS ( Bińkowski et al. , 2019 ) . Diffusion Implicit Models ( DDIM ) offer a way to accelerate the denoising process ( Song et al. , 2020a ) . The model employs a non-Markovian diffusion process to generate a higher quality sample . The model helps reduce the number of diffusion steps , e.g. , from a thousand steps to a few hundred . Algorithm 1 DDPM training procedure . 1 : Input : dataset d , diffusion process length T , noise schedule β1 , ... , βT 2 : repeat 3 : x0 ∼ d ( x0 ) 4 : t ∼ U ( { 1 , ... , T } ) 5 : ε ∼ N ( 0 , I ) 6 : xt = √ ᾱtx0 + √ 1− ᾱtε 7 : Take gradient descent step on : ‖ε− εθ ( xt , t ) ‖1 8 : until converged Algorithm 2 DDPM sampling algorithm 1 : xT ∼ N ( 0 , I ) 2 : for t= T , ... , 1 do 3 : z ∼ N ( 0 , I ) 4 : ε̂ = εθ ( xt , t ) 5 : xt−1 = xt− 1−αt√ 1−ᾱt ε̂ √ αt 6 : if t 6= 1 then 7 : xt−1 = xt−1 + σtz 8 : end if 9 : end for 10 : return x0 Dhariwal & Nichol ( 2021 ) find a better diffusion architecture through a series of exploratory experiments , leading to the Ablated Diffusion Model ( ADM ) . This model outperforms the state-of-the-art in image synthesis , which was previously provided by GAN based-models , such as BigGAN-deep ( Brock et al. , 2018 ) and StyleGAN2 ( Karras et al. , 2020 ) . ADM is further improved using a novel Cascaded Diffusion Model ( CDM ) . Our contribution is fundamental and can be incorporated into the proposed ADM and CDM architectures . Watson et al . ( 2021 ) proposed an efficient method for sampling from diffusion probabilistic models by a dynamic programming algorithm that finds the optimal discrete time schedules . Choi et al . ( 2021 ) introduces the Iterative Latent Variable Refinement ( ILVR ) method for guiding the generative process in DDPM . Moreover , Kong & Ping ( 2021 ) systematically investigates fast sampling methods for diffusion denoising models . Lam et al . ( 2021 ) propose bilateral denoising diffusion models ( BDDM ) , which take significantly fewer steps to generate high-quality samples . Huang et al . ( 2021 ) derive a variational framework for likelihood estimating of the marginal likelihood of continuous-time diffusion models . Moreover , Kingma et al . ( 2021 ) shows equivalence between various diffusion processes by using a simplification of the variational lower bound ( VLB ) . Song et al . ( 2020b ) show that score-based generative models can be considered a solution to a stochastic differential equation . Gao et al . ( 2020 ) provide an alternative approach for training an energy-based generative model using a diffusion process . Another line of work in audio is that of neural vocoders based on a denoising diffusion process . WaveGrad ( Chen et al. , 2020 ) and DiffWave ( Kong et al. , 2020a ) are conditioned on the melspectrogram and produce high-fidelity audio samples , using as few as six steps of the diffusion process . These models outperform adversarial non-autoregressive baselines . Popov et al . ( 2021 ) propose a text-to-speech diffusion base model , which allows generating speech with the flexibility of controlling the trade-off between sound quality and inference speed . Diffusion models were also applied to natural language processing tasks . Hoogeboom et al . ( 2021 ) proposed a multinomial diffusion process for categorical data and applied it to language modeling . Austin et al . ( 2021 ) generalize the multinomial diffusion process with Discrete Denoising Diffusion Probabilistic Models ( D3PMs ) and improve the generated results for the text8 and One Billion Word ( LM1B ) datasets . 3 DIFFUSION MODELS FOR GAMMA DISTRIBUTION . We start by recapitulating the Gaussian case , after which we derive diffusion models for the Gamma distribution . 3.1 BACKGROUND - GAUSSIAN DDPM . Diffusion networks learn the gradients of the data log density : s ( y ) = ∇y log p ( y ) ( 1 ) By using Langevin Dynamics and the gradients of the data log density ∇y log p ( y ) , a sample procedure from the probability can be done by : ỹi+1 = ỹi + η 2 s ( ỹi ) + √ ηzi ( 2 ) where zi ∼ N ( 0 , I ) and η > 0 is the step size . The diffusion process in DDPM ( Ho et al. , 2020 ) is defined by a Markov chain that gradually adds Gaussian noise to the data according to a noise schedule . The diffusion process is defined by : q ( x1 : T |x0 ) = T∏ t=1 q ( xt|xt−1 ) , ( 3 ) where T is the length of the diffusion process , and xT , ... , xt , xt−1 , ... , x0 is a sequence of latent variables with the same size as the clean sample x0 . The Diffusion process is parameterized with a set of parameters called noise schedule ( β1 , . . . βT ) , which defines the variance of the noise added at each step : q ( xt|xt−1 ) : = N ( xt ; √ 1− βtxt−1 , βtI ) , ( 4 ) Since we are using a Gaussian noise random variable at each step , the diffusion process can be simulated for any number of steps with the closed formula : xt = √ ᾱtx0 + √ 1− ᾱtε , ( 5 ) where αi = 1− βi , ᾱt = ∏t i=1 αi and ε = N ( 0 , I ) . Diffusion models are a class of generative neural network of the form pθ ( x0 ) = ∫ pθ ( x0 : T ) dx0 : T that learn to reverse the diffusion process . One can write that : pθ ( x0 : T ) = p ( xT ) T∏ t=1 pθ ( xt−1|xt ) ( 6 ) As described in ( Ho et al. , 2020 ) , one can learn to predict the noise present in the data with a network εθ and sample from pθ ( xt−1|xt ) using the following formula : xt−1 = xt − 1−αt√1−ᾱt εθ ( xt , t ) √ ᾱt + σtε , ( 7 ) where ε is white noise and σt is the standard deviation of added noise . ( Song et al. , 2020a ) use σt 2 = βt . The training procedure of εθ is defined in Alg.1 . Given the input dataset d , the algorithm samples , x0 and t. The noisy latent state xt is calculated and fed to the DDPM neural network εθ . A gradient descent step is taken in order to estimate the ε noise with the DDPM network εθ . The objective for the diffusion model is a variational bound on the model data log likelihood . The complete inference algorithm present at Alg . 2 . Starting from Gaussian noise and then reversing the diffusion process step-by-step , by iteratively employing the update rule of Eq . 7 . To perform generation with few denoising iterations one can use the update equation introduced in Song et al . ( 2020a ) . This work greatly improves the results of diffusion networks when performing sampling with few generative steps . xn−1 = √ ᾱn−1x̂0 , n + √ 1− ᾱn−1 − σ̃2nεθ ( xn , ᾱn ) + σ̃nε , ( 8 ) Intuitively this equation changes the added noise for the generative steps . It uses a blend of the noise from the previous state ( εθ ( xn , ᾱn ) and random noise ( ε ) . One can write σ̃ = η √ βn ( 1− ᾱn−1 ) ( 1− ᾱn ) this allows to have a simple parameter η to choose the ratio of the blend . On top of improving the results in short samplings regimes , this allows to generate in a deterministic way when using η = 0 .

Summary: ## Summary This paper explores the use of a non-Gaussian diffusion process for Diffusion Probabilistic Models.  Unlike the original work by Ho et al., the authors replace the diffusion process with a Markov chain with transition kernel defined by a Gamma distribution.  They show that the similar (and necessary) properties of Gaussian distribution that enable training DPM in practice also hold true for Gamma distribution.  The main motivation why Gamma distribution is used seems to be that Gamma distribution is more expressive than Gaussian due to having an extra parameter.  The authors experimentally verify the performance gains on a few datasets.