Denoising Diffusion Gamma Models
Authors: Eliya Nachmani
Robin San Roman
Lior Wolf
Venue: ICLR 2022 Submitted (2022)

Abstract: Generative diffusion processes are an emerging and effective tool for image and speech generation. In the existing methods, the underlying noise distribution of the diffusion process is Gaussian noise. However, fitting distributions with more degrees of freedom could improve the performance of such generative models. In this work, we investigate other types of noise distribution for the diffusion process. Specifically, we introduce the Denoising Diffusion Gamma Model (DDGM) and show that noise from Gamma distribution provides improved results for image and speech generation. Our approach preserves the ability to efficiently sample state in the training diffusion process while using Gamma noise. 

Under review as a conference paper at ICLR 2022
DENOISING DIFFUSION GAMMA MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative diffusion processes are emerging and effective tool for image and speech generation. In the existing methods, the underlying noise distribution of the diffusion process is Gaussian noise. However, fitting distributions with more degrees of freedom could improve the performance of such generative models. this work, investigate other types of noise distribution for the diffusion process. Specifically, we introduce the Denoising Diffusion Gamma Model (DDGM) and show that noise from Gamma distribution provides improved results for image and speech generation. Our approach preserves the ability to efficiently sample state in the training diffusion process while using Gamma noise.
INTRODUCTION
Deep generative neural networks have shown significant progress over the last years. The main architectures for generation are: (i) VAE (Kingma & Welling 2013) based, for example, NVAE (Vahdat & Kautz, 2020) and VQ- VQ- (Razavi al. 2019), (ii) GAN (Goodfellow et al. 2014) based, for example, StyleGAN (Karras et al. 2020) for vision application and WaveGAN (Donhhue et al. 2018) for speech (iii) Flow-based, for example Glow (Kingma Dhariwal, 2018) (iv) Autoregessive, for example, Wavenet for speech (Oord et al. 2016) and (v) Diffusion Probabilistic Models (Sohl- Dickstein et al. 2015), for example, Denoising Diffusion Probabilistic Models (DDPM) (Ho et al. 2020) and its implicit version DDIM (Song et al. 2020a).
Models from this last family have shown significant progress in generation capabilities in the last years, e.g., (Chen al. 2020 Kong et al., 2020b). and have achieved results comparable to state-of-the-an art generation architecture for both images and speech.
A DDPM is Markov chain of latent variables. Two processes modeled: (i) diffusion process and (ii) a denoising process. During training, the diffusion process learns to transform data samples into Gaussian noise. Denoising is the reverse process and it is used during inference for generating data samples, starting from Gaussian noise. The second process can be conditioned on attributes to control the generation sample. To obtain high-quality synthesis, large number of denoising steps is used (i.e. 1000 steps). A notable property of the diffusion process is a closed-form formulation of the noise that arises from accumulating diffusion stems. This allows sampling arbitrary states in the Markov chain of the diffusion process without calculating the previous steps.
In the Gaussian case, this property stems from the fact that adding Gaussian random variables leads to another Gaussian random variable. Other distributions have similar properties. For example, for the Gamma distribution, the sum of two random variables that share the scale parameter is a Gamma random variable of the same scale. The Poisson distribution has a similar property. However, its discrete nature makes it less suitable for DDPM.
In DDPM, the mean of the Gaussian random variables is set at zero. The Gamma random variable, with its two parameters (shape and scale), is better suited to fit the data than a Gaussian random variable with degree of freedom (scale). Furthermore, the Gamma random variable generalizes other distributions, and many other distributions can be derived from it (Leemis & McQueston 2008).
The added modeling capacity of the Gamma random variable can help speed up the convergence of the DDPM model. Consider, for example, a conventional DDPM model that was trained with Gaussian noise the CelebA dataset (Liu et al. 2015).
The noise distribution throughout the diffusion process can be visualized by computing the histogram of the estimated residual noise in the generation process. The estimated residual noise ? is given by

---

Under review as a conference paper at ICLR 2022
Mean Fitting Error (10
1.0
Gaussian Histogram
2.0
Gamma Histogram
Gausson Gamma
0.40
0.0
0.8
0.35
Dendly
Densiyy
0.30
0.5
0.4
0.4
0.0
0.2
0.2
0.15
0.0-2
0.0-2
Values of
Values of
steps
(a)
(b)
(c)
Figure 1: Fitting a distribution to the histogram of the generation error, which given by the scaled
difference between 2n and the image after t DDPM steps é = yazza =2t The model is a pretrained
VI- &eel
DDPM (Gaussian) celebA (64x64) model. (a) The fitting of Gaussian to the histogram of typical image after - 50 steps. (b) Fitting a Gamma distribution. (c) The fitting error to Gaussian
and Gamma distribution, measured the MSE between the histogram and the fitted probability distribution function. Each point is the average value for the generation of 100 images. The vertical error bars denote the standard deviation.
8 1/1aag-a where or is the noise schedule, æog is the data point and is the estimate state at V1-la,l
timestep I, be derived from Eq.4 from <Song al. 2020a). Both Gaussian distribution and Gamma distribution can then be fitted to this histogram, as shown in Fig- [a.b). As can be seen, the Gamma distribution provides better fit to the estimated residual noise Moreover, Fig. (c) presents the mean fitting error between the histogram and the fitted probability distribution function. Evidently, the Gamma distribution is better fit than the Gaussian distribution. While the model was trained to estimate Gaussian noise, at inference time it has to address different distribution.
In this paper, we investigate the non- Gaussian Gamma noise distribution. As noted, this distribution seems to fit the histogram of the generation error better than the Gaussian distribution, and it also has favorable properties such as behavior under addition and scalar multiplication. The proposed models maintain the property of the diffusion process of sampling arbitrary states without calculating the previous steps. Our results are demonstrated in two major domains: vision and audio. the first domain, the proposed method is shown provide better FID score for generated images. For speech data, we show that the proposed method improves various measures, such as Perceptual Evaluation of Speech Quality (PESQ) and short-time objective intelligibility (STOI).
RELATED WORK
In their seminal work, (Sohl-Dickstein al. (2015) introduce the Diffusion Probabilistic Model. This model is applied to various domains, such as time series and images. The main drawback in the proposed model is that needs up to thousands of iterative steps to generate a valid data sample. [Song & Ermon (2019) proposed a diffusion generative model based on Langevin dynamics and the score matching method (Hyvärinen & Dayan 2005). The model estimates the Stein score function (Liu et al. 2016) which is the gradient of the logarithm of data density. Given the Stein score function, the model can generate data points.
Denoising Diffusion Probabilistic Models (DDPM) (Ho et al. 2020) combine generative models based on score matching and neural Diffusion Probabilistic Models into a single model. Similarly, in Chen et al. (2020); Kong et al. (2020a) generative neural diffusion process based on score matching was applied to speech generation. These models achieve state-of-the-arr results for speech generation, and show superior results over well-established methods, such Wavernn (Kalchbrenner al. 2018), Wavenet (Oord et al. 2016), and GAN-TTS (Bnkkowski et al. 2019).
Diffusion Implicit Models (DDIM) offer way to accelerate the denoising process (Song al. 2020a). The model employs a non-Markovian diffusion process to generate a higher quality sample. The model helps reduce the number of diffusion steps, e.g., from thousand steps few hundred.
2

---

Under review as a conference paper at ICLR 2022
Algorithm 1 DDPM training procedure.
Algorithm DDPM sampling algorithm
1: Input: dataset d, diffusion process length T.
ør NNO,
noise schedule 31,..., BT
2: for t= T. do
2: repeat
z ~ 1O
3: 20 d(zo)
4:
é = (0(
4:
2t Vi-ai
5:
E e~ e~,(0,,)
21-1
Vai
6:
Ft VoyTO - d+E
6:
ift + then
7:
Take gradient descent step on:
7: 2t-1 2t-1 022
|| - (xx,,))) I)l
8: end if
8: until converged
9: end for
10: return 20
Dhariwal Nichol (202T) find a better diffusion architecture through a series of exploratory experi- ments, leading to the Ablated Diffusion Model (ADM). This model outperforms the state-of-the-an in image synthesis, which was previously provided by GAN based-models, such as BigGAN-deey (Brock al., 2018) and StyleGAN2 (Krrra al.) 2020). ADM is further improved using novel Cascaded Diffusion Model (CDM). Our contribution is fundamental and can be incorporated into the proposed ADM and CDM architectures.
Watson et al. 2 proposed an efficient method for sampling from diffusion probabilistic models by dynamic programming algorithm that finds the optimal discrete time schedules. Choi et al. (202T) introduces the Iterative Latent Variable Refinement (ILVR) method for guiding the generative process
in DDPM. Moreover, Kong & Ping (022)) systematically investigates fast sampling methods for
diffusion denoising models. Lam et al. (2021) propose bilateral denoising diffusion models (BDDM),
which take significantly fewer steps generate high-quality samples.
Huang et al. (202T) )) derive variational framework for likelihood estimating of the marginal likelihood of continuous time diffusion models. Moreover, Kingma et al. (2021) shows equivalence between various diffusion processes by using simplification of the variational lower bound (VLB).
Song al. (2020b) show that score-based generative models can be considered solution stochastic differential equation. [aao et al. (2020) provide alternative approach for training an energy-based generative model using a diffusion process.
Another line of work in audio is that of neural vocoders based a denoising diffusion process. WaveGrad (Chen et al.] 2020) and DiffWave (Kong et al. 2020a) are conditioned on the mel- spectrogram and produce high-fidelity audio samples, using as few six steps of the diffusion process. These models outperform adversarial non-autoregressive baselines. [Popov et al. (202T) propose a text-to-speech diffusion base model, which allows generating speech with the flexibility of controlling the trade-off between sound quality and inference speed.
Diffusion models were also applied natural language processing tasks. Hoogeboom et al. (2221)
proposed a multinomial diffusion process for categorical data and applied it to language modeling.
Austin et al. (202T) generalize the multinomial diffusion process with Discrete Denoising Diffusion Probabilistic Models (D3PMs) and improve the generated results for the text8 and One Billion Word (LMIB) datasets.
DIFFUSION MODELS FOR GAMMA DISTRIBUTION
We start by recapitulating the Gaussian case, after which we derive diffusion models for the Gamma distribution.
3.1 BACKGROUND GAUSSIAN DDPM
Diffusion networks learn the gradients of the data log density:
s(y) Vy log p(v)
3

---

Under review as a conference paper at ICLR 2022
By using Langevin Dynamics and the gradients of the data log density V log p(y), sample procedure from the probability can be done by:
91-1 = Do +
2 ((1)) + VT2
(2)
where M(0, 1) and > 0 is the step size.
The diffusion process DDPM (Ho et al., (020) is defined by a Markov chain that gradually adds Gaussian noise to the data according to noise schedule. The diffusion process is defined by:
(01r)(0)) II (()((=1))
(3)
where T is the length of the diffusion process, and IT ..... Lt, 2t-1,..., Zy is sequence of latent variables with the same size as the clean sample Ty. The Diffusion process is parameterized with a set of parameters called noise schedule (31:... Br), which defines the variance of the noise added at each step:
((< = Na,; v1 1,
Since we are using Gaussian noise random variable at each step, the diffusion process can be simulated for any number of steps with the closed formula:
Vogto vI ā
(5)
where 0; = 1 - 3; ā = -I Og and = (O,
Diffusion models are a class of generative neural network of the form (=())) pa (1ae ddror that learn reverse the diffusion process. One write that:
po( p(aT) II po 10(
(6)
As described in (Ho et al. 2020). one can learn to predict the noise present in the data with a network Eg and sample from py ( (2) using the following formula:
1-ar ca( t)
+ O,E,
at
where 6 is white noise and a, is the standard deviation of added noise. (Song et al. 2020a) use a,² = 3r.
The training procedure of éo is defined in Alg Given the input dataset d, the algorithm samples e, To and l. The noisy latent state v, is calculated and fed the DDPM neural network eg. A gradient descent step is taken in order to estimate the noise with the DDPM network Eg. The objective for the diffusion model a variational bound on the model data log likelihood.
The complete inference algorithm present at Alg. 2 Starting from Gaussian noise and then reversing the diffusion process step-by-step, by iteratively employing the update rule of Eq. 7 To perform generation with few denoising iterations one can use the update equation introduced in [Song et al. (2020a). This work greatly improves the results of diffusion networks when performing sampling with few generative steps.
Tn-1 Van-1aon + V1 - ān-1 (n an) + Tns,
Intuitively this equation changes the added noise for the generative steps. uses a blend of the noise from the previous state (Ea(En, ū,) and random noise (e). One can write ă = (1- mn ān) this allows to have simple parameter 17 to choose the ratio of the blend. On top of improving the results in short samplings regimes, this allows to generate deterministic way when using 7) = 0.
4

---

Under review as a conference paper at ICLR 2022
3.2 DENOISING DIFFUSION GAMMA MODELS (DDGM)
We expand the framework of diffusion generative processes by incorporating a new noise distribution, namely the Gamma Distribution. We call this new type of models Denoising Diffusion Gamma Models. First, we define the Gamma diffusion process, then we present a way to sample from this process, and finally we show how train those models by computing the variational lower bound and deriving a novel loss function from it.
3.2.
THE GAMMA MODEL
In the Gaussian case the diffusion equation (Eq. can be written as:
3, BBe
where & is the Gaussian noise of step t. One can denote [(k, 8) as the Gamma distribution, where k and are the shape and the scale respectively. We modify Eq, by adding, during the diffusion process, noise that follows a Gamma distribution:
V B (ge- B(gg))
(10)
Br a2002
where ~
vo,0o and k
Note that dg and Be are hyperparameters (and
the noise term has zero mean.)
Since the sum of Gamma random variables (with the same scale parameter) distributed as Gamma distribution, derive closed form for Tet i.e. an equation calculate Fe from zi:
Vārzo + (āe k2O,)
(11)
where de T(k2, 0c) and ki Ei= Kr.
Br
Lemma 1. Let Og R, Assuming VI (1....T). h4 Then Vt E {1...., T the following hold:
0: Vorto, and gr T(k,, 0t).
agt 2'
E(g. - E(gg)) = 0, V(: - E(gi)) = Be
(12)
2, = Vanuo + (g. E(g.))
(13)
where gr T(ke, 0c) and R; E h;
The complete proof for Lemmal is given in Appendix A.1
Similarly to Eq the inference is given by:
It 1-ar
(0( t)
9t E(g()
E1
+ 0. (V+
(14)
ar
In Algorithm 3 we describe the training procedure. As input we have the: (i) initial scale 6g. (ii) the dataset (iii) the maximum number of steps in the diffusion process and (iv) the noise schedule B1. 3r. The training algorithm sample: (i) an example To, (ii) number of step and (iii) noise Then it calculates It from Fo by using [T The neural network Eg has an input 24 and is conditional the time step f. Next, it takes a gradient descent step to approximate the normalized
-20 with the neural network Eg. The main changes between Algorithm and the single /1-la.1 Gaussian case (i.e. Alg. I are the following: (i) calculating the Gamma parameters, (ii) æ, update equation and (iii) the gradient update equation.
noise
The inference procedure is given in Algorithm
4 starts from a zero mean noise Zy sampled from
I Next, for T steps the algorithm estimates #1-1 from by using Eq 14 Note that as
in (Song et al. 2020a) a, = 3,. Algorithm 4 replaces the Gaussian version (i.e. Alg. () with the following: (i) the starting sampling point Ir, (ii) the sampling noise and (iii) the 2, update equation.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 3 Gamma Training Algorithm
Algorithm 4 Gamma Inference Algorithm
1: Input: initial scale 8g dataset d, diffusion process length T,, noise schedule 31...,, Sr
1: y ~ [(fr. kr)
2: 2T 7 0r lrT
2: repeat
3: for t= T, do
3:
d(zo)
vi c(a1.))
4:
-=., T})
4: 2t-1
Voa
5:
Sr ~ T(k,, Oz)
5:
if then
6:
Ft Voygo (ge R.
6:
~ 1(0,-11 k
7:
Take gradient descent step on:
2-01-1ke-
31 k, v 1-
Eo (0 t)
8:
F4 It-1 0x2
(1-(a))
9:
end if
8:
until converged
10: end for
3.2.2 THE REVERSE PROCESS FOR DDGM
The reverse process (( xx)) defines the underlying generation process. Therefore, in this section, we will obtain the reverse process for the Gamma denoising diffusion model. Furthermore, we will use the reverse process q 22) to obtain the variational lower bound and the appropriate loss function for the Gamma distribution denoising diffusion model.
Denote q(It-1/z0 z4) as the reverse process of the proposed Gamma diffusion model. Then, the reverse process is proportional to:
Lemma 2.
- c Xs/0r
kik 11 1-1 c- -x
1151-1/200 10000 ve. x
(15)
e
where
1. X1 = 2t -I- 3 kelt
2. X, = Ee - Van200 + Kr
3. X,-1 = xt-1 v0 + k1-10,-1
The complete proof for Lemma is given in Appendix A.I It states that the reverse process is proportional to three Gamma random variables X,, X,-1, and X, This observation allows us to develop the associated variational lower bound.
3.2.3 VARIATIONAL LOwER BOUND FOR DDGM
Denoising diffusion models (Ho al. 2020 trained by optimizing the usual variational bound negative log likelihood:
Po( = LVLB (
E ((ll < Eg
log logp(zr))
log
(16)
To get the variational lower bound for the proposed Gamma denoising diffusion model, one can use Eq.5 from Ho et al. (2020):
LVLB Ey L1 EL--1 Lu
(17)
where Lr. L.-1 and Lo define by:
LT DKI (x((q(rr((o)))((()))
2. T--1 DDKx(g(aee 1l sz))
3. T.g -log(po(z0(21))

---

Under review as a conference paper at ICLR 2022
It should be noted that in the Gaussian case the KL terms have closed form. Ly is constant and igmored during training since it doesn't have learnable parameters. Moreover, in (Ho et al. 2020) Lo modeled with discrete decoder, however, in our proposed model empirically found that the impact Lo is negligible and can be removed.
Therefore, to calculate the variatonal lower bound needs obtain:
L = DDL((((0-11al)) 1:) 2+))
(18)
where:
It v1 (2,0
80
(19)
vāt
Lemma 3. The for the proposed Gamma diffussin model upper bounded by the following L norm:
L--1 c1+C2+
C3
C4
1r0
-fol
(20)
Jt
Ic-1
where C1 C2 C3 and C₁ are constant terms.
The complete proof for Lemma is given in Appendix
As can be seen, the variational lower bound is bounded by some constant forms multiplied by the LI norm between the data point 2g and its estimation øo The constant terms C1, C2, C3s and C4 as well as Sr and 9t 1 are known values during the training.
3.2.4 LOss FUNCTION FOR DDGM
Denoising diffusion probabilistic models use the variational lower bound to minimize the negative log likelihood, As described in Sec 3.2.
minimize the variational lower bound by L, for t 2 1. To do so, one can minimize the LI norm from 34 Our model optimizes the LI norm between the sampled noise eg and the estimated noise Eq. This verified the following lemmas.
Lemma 4. Minimizing the variational lower bound for DDGM (i.e. foror t 2 1) is equivalent to minimizing the LI norm between the sampled noise and the estimated noise:
k,O,
C ft
Eo(I,, t)
(21)
a
The complete proof for Lemma
4 is given in See
A.4 at the appendix. Thus, the loss that is used in
0t
the Alg 3 is given by L
c0(0,,
v1 az.
4 EXPERIMENTS
4.1 SPEECH GENERATION
For our speech experiments we used version Wavegrad CChn et al.; 2020) based on this implementation Vovk (2020) (under BSD-3-Claune License). We evaluate our model with high-level perceptual quality of speech measurements, PESQ (Rix et al. (00I) and STOI (Taal al. 201T). We used the standard Wavegrad method with the Gaussian diffusion process baseline. We use two Nvidia Volta V100 GPUs to train our models.
For all the experiments, the inference noise schedules (Bo...,3)) were defined described in the Wavegrad paper (Chen et al. 2020). For 1000 and 100 iterations the noise schedule is linear, for 25 iterations comes from the Fibonacci and for 6 iterations search to find the best noise schedule parameters. For other hyper-paramen parameters (e.g. learning rate, batch size, etc)
performed model-dependent grid
the same as in Wavegrad (Chen et al. 2020h. Training performed using
Dr
the following form of Eq. 10 e.g. ft = vā,d and kn = o,,0²
Our best results were obtained using
6o 0.001.

---

Under review as a conference paper at ICLR 2022
Table 1: PESQ and STOI metrics for the IJ dataset for various Wavegrad-like models.
PESQ (t)
STOI (1)
Model Iteration
6
25
100
1000
6
25
100
1000
WaveGrad (Chen al. 2020
2.78
3.194
3.211 3.290 0.924 0.957 0.958 0.959
DDGM (ours)
3.07
3.208 3.214 3.308 0.948 0.972 0.969 0.969
Table 2: FID (4) score comparison for CelebA(6Ax64) dataset. Lower better.
Model
Iteration
10
20
50
100
1000
DDPM (Ho al.
2020
299.71
183.83
71.71
45.2
3.26
DDGM-
Gamma
Distribution DDPM (ours)
35.59
28.24
20.24
14.22
4.09
DDIM (Song al.
2020a)
17.33
13.73
9.17
6.53
3.51
DDGM Gamma Distrbbttin DDIM (ours)
11.64
6.83
4.28
3.17
2.92
Table FID (L) score comparison for LSUN Church (256x256) dataset. Lower better. The results of DDIM for T 1000 are not reported by [Song et al. (2020a).
Model Iteration
100
1000
DDPM (Ho et al.
2020
51.56
23.37
11.16
8.27
7.89
DDGM
Gamma Distribution DDPM (ours)
28.56
19.68
10.53
7.87
6.91
DDIM Song et al.
2020a)
19.45
12.47
10.84
10.58
NA 7.34
DDGM Gamma Distribution DDIM (ours)
18.11
11.32
10.31
8.75
Table 4: FID (L) score comparison for ImageNet (64x64) dataset. Lower is better.
Model Iteration
10
20
50
100
1000
DDIM (Song et al. 2020a)
42.88
35.40
31.98
30.74
28.81
DDGM Gamma Distribution DDIM (ours)
42.17
31.84
28.75
27.02
24.22
Table 5: FID (U) score comparison during the inference process for CelebA (64x64) dataset for the full inference procedure and for an earlier stage in the process.
Iteration Model
T=10
T=20
T=100
(first 5 steps/10 steps) (first 10 steps/20 steps) (first 50 steps/ 100 steps)
DDIM Song et al. 2020a)
54.32/17.33
43.35/13.73
36.68/6.53 35.20/3.17
DDGM (ours)
55.40/1 1.64
42.43/6.83
Results Tab. [0oTIpreee presents the PESQ and STOI measurement for the LJ dataset (Ito & Johnson 2017) As can be seen, for the proposed Gamma denoising diffusion model our results are better than the Wavegrad baseline for all number of iterations in both PESQ and STOI.
4.2 IMAGE GENERATION
Our model is based on the DDIM implementation available (Jiaming Song & Ermon 2020) (under
the MIT license). We trained our model on three image datasets (i) CclebA 64x64 (Liu et al. 2013).
(ii) L.SUN Church 256x256 (Yu et al., 2015) and (iii) ImageNet 64x64 (Deng et al.) 2009p. The Fréchet Inception Distance (FID) (Heusel et al., 2017) is used as the benchmark metric. For all experiments, similarly to previous work (Song et al. 2020a), we compute the FID score with 50, 000 generated images, using the torch-fidelity implementation (Obukhov et al. 2020). Similar to (Song et al. 2020a), the training noise schedule $1. 3r is linear with values raging from 0.0001 to 0.02. For other hyperparameters (c.g. learning rate, batch size etc) we the same parameters that appear in DDPM (Ho et al., 2020). We use eight Nvidia Volta V100 GPUs to train our models. The 8 parameter for Gamma distribution set to 0.001.
8

---

Under review as a conference paper at ICLR 2022
Figure 2: Typical examples of images generated with 100 iterations and v 0. For models trained with different noise distributions (i) First row Gaussian noise and (ii) Second row Gamma noise. All models start from the same noise instance.
Results We test our models with the inference procedure from DDPM (Ho et al. 2020) and DDIM (Song et al. (0200a). In Tab. we provide the FID score for CelebA (64x64) dataset (Liu et al. 2015 (under non-commereial research purposes license). As can be seen for DDPM inference procedure for 10, 20, 50, 100 steps, the best results were obtained from the Gamma model, which improves results by a gap of 264 FID scores for ten iterations. For 100 iterations, the Gamma model improves results by 31 FID scores. For 1000 iterations, the best results were obtained from the DDPM model. Nevertheless, our Gamma model obtains results that are closer to the DDPM by a gap of 0.83. For the DDIM procedure, the best results were obtained with the Gamma model for all number of iterations. Fig. 2
presents samples generated by the three models. Our models provide better quality images when compared to DDPM and DDIM methods.
In Tab 3 we provide the FID score for the LSUN church dataset Yu et al. 2015). As can be seen, the Gamma model improves results over the baseline for 10, 20, 50, 100, 1000 iterations.
Tab. . lists the FID score for the ImageNet 64x64 dataset. The Gamma model obtains better results than the DDIM baseline method for all iteration counts. Figure Baaureeooomme compares between our proposed
DDGM and the baseline DDIM method. The random samples are generated with 1000 iterations with the DDIM generation algorithm and rj = 0. each of the nine instances, the same random noise is used as 2 1000 for both models.
In Tab. we present results for training the network on a Markov chain of size T and perform all the processes until step 7/2, from which we infer zo in one inference step. As can be seen, after the T/2 first iterations, there no significant difference between Gaussian and Gamma networks. However, after the remaining T/2 iterations, the Gamma based model obtains performance GAP over the Gaussian model. This suggests that our method improving the results mostly during the final stage of the inference process.
CONCLUSIONS
We present a novel Gamma diffusion model. The model employs a Gamma noise distribution. A key enabler for using these distributions is closed-form formulation (Eq. (T) of the multi-step noising process, which allows for efficient training. We also present the reverse process and the variational lower bound for the Gamma diffusion model. The proposed model improves the quality of generated image and audio, as well as the speed of generation in comparison to conventional, Gaussian-basee diffusion processes. Our DDGM methods shows that diffusion model can benefit from non- Gau-ssssian noise distributions. This comes at a cost of adding one new hyperparameter to tune (0n). Working with other probability distributions, such as mixture models, may improve results even further.
REPRODUCIBILITY STATEMENT
We provide in the supplementary file the complete code that was used to perform all of our experi- ments. This archive includes audio samples and the code for both image and speech experiments. Hyperparameters choices are clearly stated in Sec. [ and the values obtained from publicly available implementation of previous work. The proof of all the theoretical results are available the appendix derived in the paper.

---

Under review as a conference paper at ICLR 2022
Figure 3: Comparison of generated samples with the (left) Gaussian noise (DDIM) and (right) Gamma noise (DDGM). In each of the nine cases, the two models start from the same noise instance, which leads to similar output images between the models.
REFERENCES
Jacob Austin, Daniel Johnson, Jonathan Ho, Danny Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. arXiv preprint urXiv:2107.03000 2021.
Mikolaj Binikowsi,, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. arXiv preprint arXiv: /646, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 18X.:1809..1096 2018.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint S1ti:220090007133 2020.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Condi- tioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.0298 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2/05.05233. 2021.
Chris Donahue, Julian McAuley, and Miller Puckette, Adversarial audio synthesis. arXiv preprint arXiv:1802.04208, 2018
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. arXiv preprint arXiv:2012.08125, 2020.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 1406v.1406.2660 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv: /506:/770.085555 08500, 2017.
10

---

Under review as a conference paper at ICLR 2022
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 11239, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non- au-oorreggresive language models. arXiv preprint arXiv:2102.053779 2021.
Chin-Wei Huang, Jae Hyun Lim, and Aaron Courville. A variational perspective on diffusion-based generative models and score matching. arXiv preprint arXiv:2106.02800 2021.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Keith Ito and Linda Johnson.
The Ij speech dataset.
http:://keithito.ccom
LJ St--Speech-Dataseet/ 2017.
Chenlin Meng Jiaming Song and Stefano Ermon. Denoising diffusion implicit models. https
/ddim. 2020.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp.
PMLR, 2018.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 11,9, 2020.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions.
arXiv preprint wrX.v:8807000333 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.0066: 2021.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132. 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0076/ 2020a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprini arXiv:2009.00 099701999999...... 2020b.
Max WY Lam, Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral denoising diffusion models. arXiv preprint arXiv:2108. /1514, 2021.
Lawrence M Leemis and Jacquelyn T McQueston. Univariate distribution relationships. The American Statistician, 62(1):45-55 53, 2008
Qiang Liu, Jason Lee, and Michael Jordan. kernelized stein discrepancy for goodness-of-fn tests. In International conference on machine learning, pp. 276 276-284. PMLR, 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Anton Obukhov, Maximilian Seitzer, Po-Wei Wu, Semen Zhydenko, Jonathan Kyl, and Elvis Yu-Jing
Lin. High-fidelity performance metries for generative models in pytorch, 2020. URL htthttss::/ Version: 0.2.0, DOI: 125388//2nooo..7766622
g wiithbb..o com wtt..////ttttttt..ooeooooeeeeeee
ity
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 160000609 2016.

---

Under review as a conference paper at ICLR 2022
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. arXiv preprint arXiv:210S.06337, 2021.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprini arXiv: (eee:990600444 2019.
A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 200! IEEE International Conference on Acousties, Speech, and Signal Processing. Proceedings (Car.
No.0/CH37221), volume 2, pp. 749-752 vol.2, 2001. doi: 10.1 N0.100///AASPP..000..00 9411033..
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv:1907.05600 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 0rXiv:2011.13455 2020b.
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen. An algorithm for intelligibility prediction of time-frequency weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7):2125-2136, 2011. doi: 10. 21119//ATL..211.211444
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898 2020.
Ivan Vovk. Wavegrad.
2020.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint (tt::2006.08002. 2021.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint wrXivv1506.03365 2015.
A PROOFS
A. PROOF OF LEMMA
Bt 2' 0: = Varlo, and gr Tike, 0c).
Lemma 1. Let 0g E R. Assuming Vt e {I. k2
a,d,
Then Vt {l,.., T3 the following hold:
Elge F(g:)) = 0, V(g - F(g.)) = Be
(12)
21 = Varuo + (g. E(g.))
(13)
where ge I'ke,, 0.) and k = Et
Proof. The first part of Eq. 12 is immediate. The variance part is also straightforward:
V(g. Flge)) k,O,² 3
12

---

Under review as a conference paper at ICLR 2022
Eq. 13 is proved by induction on t {1....T. For t = 1:
x = / 3100 + 91 -(9)))
since h k1 91 = 91- We also have that v 31 Vāy. Thus we have:
11 = Vāio + (01 -E()))
Assume Eq.
holds for some E (I....)) The next iteration obtained as
F++1 v1 Bet1Ze 9++1 E(g++1)
(22)
/ Be+1 (vrrg + (ge E(g))) + gt+1 - F((e+1)
(23)
V0 + v Bit10e + I++1 (V1 (ete)((r)) + E(ge+1))
(24)
remains to be proven that (i) / Be+1 St + 9t+1 = 9t+1 and (ii) v B E(ge) + E(g++1)
E(g++1). Since 3t ~ rik, 0,) hold, then:
V1 Bet10 [kk, V Bi+101) [kk,, 0ct1)
Therefore, we prove (i):
V 84419: : + 9++1 [hk + k 0(41) F(ke+1,0+1)
which implies that /1 Bi+19r ge+1 and 91+1 have the same probability distribution. Furthermore, by the linearity of the expectation, one can obtain (ii):
V Bet1 F(0)) + E(g+41) = E(V1 Bit15o 9c+1)
-E(g++))
Thus, have:
Tt+1 V04+100 + (gt+1 E(g++1))
which ends the proof by induction.
A.2 PROOF OF LEMMA 2
Denote q( 21) the reverse process of the proposed Gamma diffusion model. Then, the reverse process is proportional to:
Lemma 2.
-
204-2e-Xe/86 X./0. 8-h1-1 -
q(2--1 1 x2) x
(15)
X e 8,10,
where
1. X, Et v1 3,5,-1 kg
2. X, = Ft Var20 + Tr
3. X,-1 = xt-1 - 000-120 + 11-10--1
Proof. The reverse process is given by:
g(0:11/20, xx) = q(11/21-1,00)) ((
(25)
13

---

Under review as a conference paper at ICLR 2022
Next, one can calculate each of the three main components of the reverse process, ie. (i) (( zo), (ii) q(zx-1|20) and (iii) ( [m)). Since q is memoryless, ((2g|22-11,0)) =
((1)) Therefore, the first component (i) of Eq. 25 is the forward process. The forward process given by:
(((((111)== = p(gt = 2t v B + kigBe)
(26)
(xq v1 3,34-1 + k2O,) 1 (1hh----htt-ttheee
(27)
The second component of Eq is given by:
g(o)-1 (Iz-1 04 20 h 0e-1 )hn 1-le (+c von 1o0th, 1)/% 1
(28)
1(k
Similarly, the third component of Eq is given by:
((xx)2r0) p(g wz Varsn k.).) (et Varoo + k,O,) -o------ -a-
29
I(kx))²
Overall, the reverse process q 100 xx)) given by:
(xt - VT Fr + k, ha-
-l (ot / Mim 1 +44001/0
xz)
r(k:)0
(xt-1 Vaa- + 1--101100000 -
10
(30)
1)
(Ix Vanzo + k2O,) )a (osttohhtttt
One can denote:
1. X, 2e V1 315 h20
2. X, Zt Vā,200 + kr
3. X,-1 = It-1 /0t-1200 + 1
Thus, the reverse process gf (tx (r0, Ix) is proportional to:
e X./0, pkc -1
- 81-1/000
( 100-1 22) x
(31)
Xh: kk1-1 1
A.3 PROOF OF LEMMA
Lemma 3. The Lt-1 for the proposed Gamma diffusion model is upper bounded by the following L1
norm:
(c++Ca+
Ca
C4
L <
|:0 - -fol
(20)
gt
-r-1
where C1, C2, O3 and C4 are constant terms.
14

---

Under review as a conference paper at ICLR 2022
Proof. We can calculate the Le-1 with the exact form:
g( No. xz) p [0o xt))
zz)) L1 = (x- = E By 112 log
(32)
Using Eq 15 the RHS of Eq become:
q(() [o: 2tr)
X,-1 - Xx-1 (k. 1) log
X, X
log
= (ha-1 - 1) log
Po (1 (20, x21)
X
0--1
(33)
One can show that the four terms present in the previous equation can be upper bounded with the L1 distance between the predicted 4g and the ground truth zig
X, x,
= (200
20)
Vor 8c-1
S
C1
Eol
8c-1
X.-8 &
= =1
-00)
Vac
< C2/ ctol
De
2t v VāLzo + K2b
(ke
1) log
E1 X
(Ke - 1)log
=
Vārão + k:p8,
Varl - 10)
Var( -t))
log
1+
= Ca (:0 - -00
va,ê0 + k:0/
It v 2, + k,0=
(ki-1
1)l 811) = log
1+
v a, 1 200 80)
It-1
v C4 180 + k 18
/ - 200)
C4 100 Scl
It-1 vā--1 11-100-1
ge-1
The complete form of the L,-1 upper bound can be expressed as follows:
T S Pg(0r-1/00.a))
(00.0))
(a (1+
Cs +
C4
100
Sal =
(G+
Ci
+
C2
C3
+
Ca
rpl
-t-1
9t
9--1
(34)
A.4 PROOF OF LEMMA 4
Lemma 4. Minimizing the variational lower bound for DDGM (i.e. for t 2 1) is equivalent to minimizing the LI norm between the sampled noise and the estimated noise:
L
Gt
k,0= ae
Ea(
(21)
Proof. From Eq
34 the variational lower bound of DDGM is given by L
<
(C+CC= + Cs C.
I -Sol- Substitute Eq 19 and Eq 11 to the variational lower bound
15

---

Under review as a conference paper at ICLR 2022
have:
C3 +
C4
L
C1+ C2
20 Sol
(35)
9t -t-1
C3 + C4
It v1 (210
Ci C2
20
(36)
1t-1
Vāt
C3 + C4
C1+ C2 +
Vārzo - 21 + vI (r
(37)
9t-1
Vā
C3 + C4
1
Ci C1+C2+ +
|V0.0 - vārzo - (gt - -kelt) vI
area (2 t)
9t-1
Vān
(38)
C3 +
C4
C1+ C2
(g:
kglf)) v1 - (HEo(11;1) 1)
(39)
9t
-t-1
Vā
C1+ (1+C2+
C3 + C4
v1
9t k,O: v1 0z
co(22, t)
(40)
9t-1
Vā
Since we are minimizing the variational lower bound, one can drop the constant term I Therefore, minimizing the variational lower bound is equal to vāt
C C2 G 1t-1
minimizing the term
ūt k V1-
ea( 0)-
16


Reviews:
[{"review_id": "smiaSEYpdrZ", "reviewer": "Reviewer_y4Qp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The derivation of the proposed DDGM model is sound, but the experiments are not sufficient enough.\n\n1. Some obvious mistakes are needed to be corrected. For example, Eq. (6) and the denominator of Eq. (16).\n\n2. Eq. (13) represents the parameterization for the reverse process $p_{\\theta}(x_{t-1}|x_t)$. According to my understanding, all the terms of $\\bar{g}_t$ in Eq. (13) should be replaced with $g_t$, please have a check.\n\n3. The notations in Sec. 3.2.3 are confused. In Eq. (21), the reverse process is represented with $p_{\\theta}(\\cdot)$, but in the following part the reverse and the diffusion process are both denoted by $q(\\cdot)$, which are very hard to read and understand. What's more, some notations in this section are not explained, such as the $\\hat{x}_0$ in Eq.(24). In the related literatures of diffusion model, the diffusion and the reverse process are always denoted by $q(\\cdot)$ and $p(\\cdot)$ to distinguish. I suggest to reformulate this part following this common settings.\n\n4. The experiments is weaken. Even though the final FID scores justify the superiorities of the proposed DDGM, I can't intuitively understand the necessity of introducing the Gamma assumption in the diffusion model. If you can give more visual result to prove such necessity, it will be better.\n\n ", "summary_of_the_paper": "Tho goal of this work is to replace the original Gaussian noise distribution in DDPM with Gamma distribution, since the Gamma distribution is with more degrees of freedom. To achieve this goal, it reformulates the diffusion and the corresponding reverse process, and also deduces the variational lower bound.", "summary_of_the_review": "Even though the idea is interesting, this paper should be further improved from the mathematical formulations and the experiments. If so, I tend to increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qP1RlGyEmZ8", "reviewer": "Reviewer_nekw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths\n* Interesting extension to diffusion models, a new, relatively underexplored class of deep generative model.\n* Detailed derivations that are easy to follow.\n\n## Main concerns and questions\n* Lack of motivation:  I think it's great that the paper shows sample quality improvements on both image and audio dataset.  However, I have a few concerns.\n  * Towards higher iteration count, the gap between DDPM and DDGM seems to decrease (Tables 1, 2).  I'm curious if DDPM and DDGM eventually converge to a similar performance number, if we don't limit the iteration count.  If they do converge or the improvement becomes negligible, then DDGM feels more like a method to improve sampling efficiency (in the same vein as e.g. DDIM), rather than a way to actually improve the expressivity of the model.\n  * In Table 2, is there any explanation for why DDPM eventually outperforms DDGM at 1000 step count? \n  * I'm not quite sure if I understand Figure 1 and the explanation given in the Introduction.  In what way is the ability to fit the histogram of marginal pixel values better (presumably aggregated across all dimensions) related to the overall expressivity of the resulting DDGM?\n* Experimental rigor: As far as I understand, the original DDPM model was trained and evaluated on CIFAR10 and LSUN bedroom.  Presumably, the hyperparameters are tuned for those datasets.  So I'm a bit concerned that, while the model itself is based on the DDIM implementation, the hyperparameters are taken from the DDPM paper -- especially for LSUN Church, where there is very small gap between DDIM/DDPM and DDGM for several iteration counts.\n* (Minor comment) The current version of the manuscript I believe has an unnecessary amount of mathematical detail in the main text.  It'd improve the overall readability of the paper if some of the detailed derivations and algebraic manipulations are moved to the supplementary material, since they are not important part of the overall story.\n\n--------------------------------------------------------\nPOST REBUTTAL: Based on the authors' response, I raised the score to 5.", "summary_of_the_paper": "## Summary\nThis paper explores the use of a non-Gaussian diffusion process for Diffusion Probabilistic Models.  Unlike the original work by Ho et al., the authors replace the diffusion process with a Markov chain with transition kernel defined by a Gamma distribution.  They show that the similar (and necessary) properties of Gaussian distribution that enable training DPM in practice also hold true for Gamma distribution.  The main motivation why Gamma distribution is used seems to be that Gamma distribution is more expressive than Gaussian due to having an extra parameter.  The authors experimentally verify the performance gains on a few datasets.\n", "summary_of_the_review": "## Overall thoughts\nThe paper is well-written, and I had no difficulty understanding it for the most part.  I'm a bit concerned by the experimental methodology and (more importantly) the motivation for using Gamma distribution.  If the authors can better address these issues, I am willing to adjust my review accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jjjxgfkR_4_", "reviewer": "Reviewer_YK5T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n  - Overall a sound idea with some strong experimental results.\n\nWeakness:\n  - Experimental results are not well presented and discussed. More insights from the experimental results are expected.\n\nComments:\n  - Figure 2: Which model is used as the baseline, DDPM or DDIM? The samples from the baseline look significant worse than those from either Ho et al., 2020 or Song et al., 2020. Any idea?\n  - Image experiments: since experiments on higher resolution were conducted (LSUN Church, 256x256), can you provide such image samples besides the lower resolution experiments?\n  - Speech experiment: The audio samples with few iterations sounds significantly better than the baseline. However, all the audio samples from the proposed model with any number of iterations contain similar white-noise-like static noise, which does not present in the baseline. Any insights?\n  - It can be helpful to include discussion on both the pros and the cons of the proposed method.", "summary_of_the_paper": "This paper proposes to use Gamma distribution noise to replace Gaussian distribution noise in the denoising diffusion probabilistic models (DDPM). It includes experimental results conducted on speech generation and image generation.\n", "summary_of_the_review": "A sound idea with some strong experimental results. However, the presentation and the discussion can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I6pfi32x8OH", "reviewer": "Reviewer_C6QK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although Gaussian noise has many attractive properties, it may be natural to ask whether other noise processes might be as good or better than Gaussian noise.  From that point of view this is an interesting paper and novel. The paper is reasonably well written, but I believe it can be improved in several ways.  The formulation can be written more clearly.  it would also be useful to expand on the motivation for the use of Gamma noise in the first place.  There is some discussion of this issue but it is relatively superficial. The experiments are good, but not extensive relative to other recent papers on generative models, including diffusion and score-based models.\n\n1) Formulation:\n\nThe formulation of the variational lower bound in (22) involves KL terms like those in (26) for which there is no closed form expression.  The formulation in the paper instead provides an upper bound for the terms in (26), which is expressed in (27).  Once the terms in the lower bound are upper bounded, we no longer know whether the resulting objective remains a lower bound on the data log likelihood.  This appears problematic.\n\nIn the Gaussian case the KL terms of the objective, as shown by Ho et al, can be computed efficiently in closed form. In the Gamma case the computation of the objective is certainly more involved, which should be discussed explicitly in the paper.  It would be good to specifically address the computational cost of training DDGM vs DDPM and DDIM, for instance in Section 3.2.3.\n\nOne interesting property of the Gamma diffusion process is that repeated addition of independent Gamma distributed noise will become Gaussian over time due to the Central Limit Theorem. This implies that where the DDGM and DDPM deviate most is in the earliest steps of the forward diffusion process.  After 1000 steps, the noise distribution will have become a very close approximation to Gaussian noise.  As a consequence, it might seem somewhat surprising that DDGM and DDPM differ significantly in practice.  Is it possible to characterize over what range of time steps the Gaussian and Gamma cases are significantly different, and whether these differences are most significantly in the latest steps of the reverse process where fine-grained details are generated?\n\n\n2) Experiments:  I have several very minor comments on the experiments.\n\na) It would be very useful to see results from DDGM on ImageNet (even ImageNet 64x64) as it is much more challenging and diverse than CelebA.  Ideally it would be great to see results in ImageNet 256x256 for which FID numbers are available for a wide range of techniques.\n\nb)  In Table 3, how do FID numbers change as the number iterations increases to 500 or 1000?\n\nc) In Figure 2, the two rows of face from different models appear very similar.  The paper says they are generated from the same noise sample?  But there is not a priori reason to believe that two independently trained model should produce similar results given the same latent noise sample. It would be good to explain this more clearly.\n\n3) Minor issues of writing:\n\na) The paper states in multiple places that the sum of two Gaussian distributions is Gaussian and the sum of two Gamma distributions with the same scale parameter is also a Gamma distribution. This is not strictly true. The sum of two Gaussian random variables is Gaussian. But the sum of two Gaussian distributions is an unnormalized mixture model.  The same holds for Gamma random variables.  This is easy to fix of course.\n\nb) The first paragraph of Section 2 states that the score function is the logarithm of the data density.  It should be the gradient with respect to the data of the log data density.\n\nc) In Section 3.1 it might be useful state that the objective for the diffusion model is a variational bound on the model data log likelihood.  And in particular, it would be useful to remind readers that the KL terms in the objective have closed form expressions in the Gaussian case.\n\nd) It might help readers to explicitly state that the noise in (9) is mean-zero.\n\ne) Despite the result of Ho et al that DDPMs and denoising score matching are closely related, it does not appear that the most intuitive way to describe eqn (13) is in terms of Langevin dynamics.\n\nf) The reverse process is specified in (14) as q(x_t-1 | x_t, x_0). While a non-Markovian model is fine, it might be useful to mention this to readers and cite the DDIM work.  It might be good to explicitly state the differences between DDPM and DDIM inn Section 3.3 as both are used later in the experiments.\n\ng) People may find it more intuitive if the notation q(.) is reserved for the forward diffusion process, while the reverse diffusion model may be clearer if denoted by p_theta instead of q. For example, in the objective (Eqn 22) it is not clear at first glance where the parameterized model is. The definition of L0, for example, is given by q(x0 | x1), which is intractable, insted of the model p_theta(x0 | x1). The KL terms in (22), ie KL(q(x_{t-1} | x_t, x_0) || q(x_{t-1} | x_t, hat{x}_0)) hide the fact that hat{x}_0 is in fact an estimate obtained from x_t and the epsilon parameterization of the reverse process model. Improved notation in the paper would help the readability of the paper.\n\n------\n\nPost-Author Rebuttal:\nThank you for the clarification on the bound.  To avoid confusion it might be useful to bound the log likelihood rather than NLL and that way the bound is always a lower bound and my misunderstanding, mixing lower bounds in some places with upper bounds in others, can be avoided.  I also thank for the reviewers for the added quantitative results, etc in Table 3.\nWhile most of my concerns have been addressed by the authors, I remain somewhat concerned that the difference between Gamma and Gaussian noise processes ceases to be substantial after several steps of the additive diffusion process.  The authors have agreed that the most significant difference will be at the beginning of the forward process (ie the end of the reverse, generative process), which makes sense.  In the end, I am not sure how much impact this work will have in the longer term -- will people begin choosing to use Gamma noise instead of Gaussian noise in practice?\nWhile I can comfortably raise my score to 6 from 5, I still believe this is a borderline paper.", "summary_of_the_paper": "This paper formulates a denoising diffusion probabilistic model, but with Gamma distributed noise instead of Gaussian noise. The claim is that the Gamma noise model shares many of the same useful properties as the Gaussian model (eg a variational bound on data log likelihood, and repeated application of Gamma noise remains Gamma distributed, etc).  And they show, empirically, that the Gamma model produces superior results on image generation (eg on the CelebA and LSUN Church datasets) and on speech generation (eg on the LJ dataset).\n", "summary_of_the_review": "This paper describes a novel diffusion model based on Gamma noise instead of Gaussian noise.  The approach is shown to produce good empirical results compared to the Gaussian case.  But there are questions about the quality of the writing, the motivation for the use of Gamma diffusion, and the formulation, all of which the authors may want to address.  It would also be very useful to see more extensive experimental work on larger, more diverse data sets.\n\nPost-Author Rebuttal:  While most of my concerns have been addressed by the authors, I remain somewhat concerned that the difference between Gamma and Gaussian noise processes ceases to be substantial after several steps of the additive diffusion process.  The authors have agreed that the most significant difference will be at the beginning of the forward process (ie the end of the reverse, generative process), which makes sense.  In the end, I am not sure how much impact this work will have in the longer term -- will people begin choosing to use Gamma noise instead of Gaussian noise in practice?  While I can comfortably raise my score to 6 from 5, I still believe this is a borderline paper.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores replacing the Gaussian noise typically used in diffusion-based generative models with noise from other distributions, specifically the Gamma distribution. The effect of this change is studied empirically for both image and speech generation.

Reviewers welcomed the exploration of the design space of diffusion models, and several reviewers consider the study of alternative noise distributions in particular an important contribution. They also raised several issues with precision and clarity (several mistakes in the manuscript were pointed out), the quality of the experiments, and, especially, a lack of convincing motivation for this exploration / sufficient demonstration of its impact.

While the authors have made a significant effort to address the reviewers' comments and suggestions, which includes running additional experiments, all reviewers have nevertheless chosen borderline ratings, with half erring on the side of rejection, and the other half tentatively recommending acceptance.

I am inclined to agree that, as it stands, the benefit of the proposed change of noise distribution is not convincingly shown to outweigh the additional complexity this introduces, so I am also recommending rejection.
Decision: Reject