Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: ## Summary
This paper explores the use of a non-Gaussian diffusion process for Diffusion Probabilistic Models.  Unlike the original work by Ho et al., the authors replace the diffusion process with a Markov chain with transition kernel defined by a Gamma distribution.  They show that the similar (and necessary) properties of Gaussian distribution that enable training DPM in practice also hold true for Gamma distribution.  The main motivation why Gamma distribution is used seems to be that Gamma distribution is more expressive than Gaussian due to having an extra parameter.  The authors experimentally verify the performance gains on a few datasets.

main_review: 
## Strengths
* Interesting extension to diffusion models, a new, relatively underexplored class of deep generative model.
* Detailed derivations that are easy to follow.

## Main concerns and questions
* Lack of motivation:  I think it's great that the paper shows sample quality improvements on both image and audio dataset.  However, I have a few concerns.
  * Towards higher iteration count, the gap between DDPM and DDGM seems to decrease (Tables 1, 2).  I'm curious if DDPM and DDGM eventually converge to a similar performance number, if we don't limit the iteration count.  If they do converge or the improvement becomes negligible, then DDGM feels more like a method to improve sampling efficiency (in the same vein as e.g. DDIM), rather than a way to actually improve the expressivity of the model.
  * In Table 2, is there any explanation for why DDPM eventually outperforms DDGM at 1000 step count? 
  * I'm not quite sure if I understand Figure 1 and the explanation given in the Introduction.  In what way is the ability to fit the histogram of marginal pixel values better (presumably aggregated across all dimensions) related to the overall expressivity of the resulting DDGM?
* Experimental rigor: As far as I understand, the original DDPM model was trained and evaluated on CIFAR10 and LSUN bedroom.  Presumably, the hyperparameters are tuned for those datasets.  So I'm a bit concerned that, while the model itself is based on the DDIM implementation, the hyperparameters are taken from the DDPM paper -- especially for LSUN Church, where there is very small gap between DDIM/DDPM and DDGM for several iteration counts.
* (Minor comment) The current version of the manuscript I believe has an unnecessary amount of mathematical detail in the main text.  It'd improve the overall readability of the paper if some of the detailed derivations and algebraic manipulations are moved to the supplementary material, since they are not important part of the overall story.

--------------------------------------------------------
POST REBUTTAL: Based on the authors' response, I raised the score to 5.
summary_of_the_review: ## Overall thoughts
The paper is well-written, and I had no difficulty understanding it for the most part.  I'm a bit concerned by the experimental methodology and (more importantly) the motivation for using Gamma distribution.  If the authors can better address these issues, I am willing to adjust my review accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper formulates a denoising diffusion probabilistic model, but with Gamma distributed noise instead of Gaussian noise. The claim is that the Gamma noise model shares many of the same useful properties as the Gaussian model (eg a variational bound on data log likelihood, and repeated application of Gamma noise remains Gamma distributed, etc).  And they show, empirically, that the Gamma model produces superior results on image generation (eg on the CelebA and LSUN Church datasets) and on speech generation (eg on the LJ dataset).

main_review: Although Gaussian noise has many attractive properties, it may be natural to ask whether other noise processes might be as good or better than Gaussian noise.  From that point of view this is an interesting paper and novel. The paper is reasonably well written, but I believe it can be improved in several ways.  The formulation can be written more clearly.  it would also be useful to expand on the motivation for the use of Gamma noise in the first place.  There is some discussion of this issue but it is relatively superficial. The experiments are good, but not extensive relative to other recent papers on generative models, including diffusion and score-based models.

1) Formulation:

The formulation of the variational lower bound in (22) involves KL terms like those in (26) for which there is no closed form expression.  The formulation in the paper instead provides an upper bound for the terms in (26), which is expressed in (27).  Once the terms in the lower bound are upper bounded, we no longer know whether the resulting objective remains a lower bound on the data log likelihood.  This appears problematic.

In the Gaussian case the KL terms of the objective, as shown by Ho et al, can be computed efficiently in closed form. In the Gamma case the computation of the objective is certainly more involved, which should be discussed explicitly in the paper.  It would be good to specifically address the computational cost of training DDGM vs DDPM and DDIM, for instance in Section 3.2.3.

One interesting property of the Gamma diffusion process is that repeated addition of independent Gamma distributed noise will become Gaussian over time due to the Central Limit Theorem. This implies that where the DDGM and DDPM deviate most is in the earliest steps of the forward diffusion process.  After 1000 steps, the noise distribution will have become a very close approximation to Gaussian noise.  As a consequence, it might seem somewhat surprising that DDGM and DDPM differ significantly in practice.  Is it possible to characterize over what range of time steps the Gaussian and Gamma cases are significantly different, and whether these differences are most significantly in the latest steps of the reverse process where fine-grained details are generated?


2) Experiments:  I have several very minor comments on the experiments.

a) It would be very useful to see results from DDGM on ImageNet (even ImageNet 64x64) as it is much more challenging and diverse than CelebA.  Ideally it would be great to see results in ImageNet 256x256 for which FID numbers are available for a wide range of techniques.

b)  In Table 3, how do FID numbers change as the number iterations increases to 500 or 1000?

c) In Figure 2, the two rows of face from different models appear very similar.  The paper says they are generated from the same noise sample?  But there is not a priori reason to believe that two independently trained model should produce similar results given the same latent noise sample. It would be good to explain this more clearly.

3) Minor issues of writing:

a) The paper states in multiple places that the sum of two Gaussian distributions is Gaussian and the sum of two Gamma distributions with the same scale parameter is also a Gamma distribution. This is not strictly true. The sum of two Gaussian random variables is Gaussian. But the sum of two Gaussian distributions is an unnormalized mixture model.  The same holds for Gamma random variables.  This is easy to fix of course.

b) The first paragraph of Section 2 states that the score function is the logarithm of the data density.  It should be the gradient with respect to the data of the log data density.

c) In Section 3.1 it might be useful state that the objective for the diffusion model is a variational bound on the model data log likelihood.  And in particular, it would be useful to remind readers that the KL terms in the objective have closed form expressions in the Gaussian case.

d) It might help readers to explicitly state that the noise in (9) is mean-zero.

e) Despite the result of Ho et al that DDPMs and denoising score matching are closely related, it does not appear that the most intuitive way to describe eqn (13) is in terms of Langevin dynamics.

f) The reverse process is specified in (14) as q(x_t-1 | x_t, x_0). While a non-Markovian model is fine, it might be useful to mention this to readers and cite the DDIM work.  It might be good to explicitly state the differences between DDPM and DDIM inn Section 3.3 as both are used later in the experiments.

g) People may find it more intuitive if the notation q(.) is reserved for the forward diffusion process, while the reverse diffusion model may be clearer if denoted by p_theta instead of q. For example, in the objective (Eqn 22) it is not clear at first glance where the parameterized model is. The definition of L0, for example, is given by q(x0 | x1), which is intractable, insted of the model p_theta(x0 | x1). The KL terms in (22), ie KL(q(x_{t-1} | x_t, x_0) || q(x_{t-1} | x_t, hat{x}_0)) hide the fact that hat{x}_0 is in fact an estimate obtained from x_t and the epsilon parameterization of the reverse process model. Improved notation in the paper would help the readability of the paper.

------

Post-Author Rebuttal:
Thank you for the clarification on the bound.  To avoid confusion it might be useful to bound the log likelihood rather than NLL and that way the bound is always a lower bound and my misunderstanding, mixing lower bounds in some places with upper bounds in others, can be avoided.  I also thank for the reviewers for the added quantitative results, etc in Table 3.
While most of my concerns have been addressed by the authors, I remain somewhat concerned that the difference between Gamma and Gaussian noise processes ceases to be substantial after several steps of the additive diffusion process.  The authors have agreed that the most significant difference will be at the beginning of the forward process (ie the end of the reverse, generative process), which makes sense.  In the end, I am not sure how much impact this work will have in the longer term -- will people begin choosing to use Gamma noise instead of Gaussian noise in practice?
While I can comfortably raise my score to 6 from 5, I still believe this is a borderline paper.
summary_of_the_review: This paper describes a novel diffusion model based on Gamma noise instead of Gaussian noise.  The approach is shown to produce good empirical results compared to the Gaussian case.  But there are questions about the quality of the writing, the motivation for the use of Gamma diffusion, and the formulation, all of which the authors may want to address.  It would also be very useful to see more extensive experimental work on larger, more diverse data sets.

Post-Author Rebuttal:  While most of my concerns have been addressed by the authors, I remain somewhat concerned that the difference between Gamma and Gaussian noise processes ceases to be substantial after several steps of the additive diffusion process.  The authors have agreed that the most significant difference will be at the beginning of the forward process (ie the end of the reverse, generative process), which makes sense.  In the end, I am not sure how much impact this work will have in the longer term -- will people begin choosing to use Gamma noise instead of Gaussian noise in practice?  While I can comfortably raise my score to 6 from 5, I still believe this is a borderline paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use Gamma distribution noise to replace Gaussian distribution noise in the denoising diffusion probabilistic models (DDPM). It includes experimental results conducted on speech generation and image generation.

main_review: Strengths:
  - Overall a sound idea with some strong experimental results.

Weakness:
  - Experimental results are not well presented and discussed. More insights from the experimental results are expected.

Comments:
  - Figure 2: Which model is used as the baseline, DDPM or DDIM? The samples from the baseline look significant worse than those from either Ho et al., 2020 or Song et al., 2020. Any idea?
  - Image experiments: since experiments on higher resolution were conducted (LSUN Church, 256x256), can you provide such image samples besides the lower resolution experiments?
  - Speech experiment: The audio samples with few iterations sounds significantly better than the baseline. However, all the audio samples from the proposed model with any number of iterations contain similar white-noise-like static noise, which does not present in the baseline. Any insights?
  - It can be helpful to include discussion on both the pros and the cons of the proposed method.
summary_of_the_review: A sound idea with some strong experimental results. However, the presentation and the discussion can be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Tho goal of this work is to replace the original Gaussian noise distribution in DDPM with Gamma distribution, since the Gamma distribution is with more degrees of freedom. To achieve this goal, it reformulates the diffusion and the corresponding reverse process, and also deduces the variational lower bound.
main_review: The derivation of the proposed DDGM model is sound, but the experiments are not sufficient enough.

1. Some obvious mistakes are needed to be corrected. For example, Eq. (6) and the denominator of Eq. (16).

2. Eq. (13) represents the parameterization for the reverse process $p_{\theta}(x_{t-1}|x_t)$. According to my understanding, all the terms of $\bar{g}_t$ in Eq. (13) should be replaced with $g_t$, please have a check.

3. The notations in Sec. 3.2.3 are confused. In Eq. (21), the reverse process is represented with $p_{\theta}(\cdot)$, but in the following part the reverse and the diffusion process are both denoted by $q(\cdot)$, which are very hard to read and understand. What's more, some notations in this section are not explained, such as the $\hat{x}_0$ in Eq.(24). In the related literatures of diffusion model, the diffusion and the reverse process are always denoted by $q(\cdot)$ and $p(\cdot)$ to distinguish. I suggest to reformulate this part following this common settings.

4. The experiments is weaken. Even though the final FID scores justify the superiorities of the proposed DDGM, I can't intuitively understand the necessity of introducing the Gamma assumption in the diffusion model. If you can give more visual result to prove such necessity, it will be better.

 
summary_of_the_review: Even though the idea is interesting, this paper should be further improved from the mathematical formulations and the experiments. If so, I tend to increase my score.
review_title: null

Meta-review: This paper explores replacing the Gaussian noise typically used in diffusion-based generative models with noise from other distributions, specifically the Gamma distribution. The effect of this change is studied empirically for both image and speech generation.

Reviewers welcomed the exploration of the design space of diffusion models, and several reviewers consider the study of alternative noise distributions in particular an important contribution. They also raised several issues with precision and clarity (several mistakes in the manuscript were pointed out), the quality of the experiments, and, especially, a lack of convincing motivation for this exploration / sufficient demonstration of its impact.

While the authors have made a significant effort to address the reviewers' comments and suggestions, which includes running additional experiments, all reviewers have nevertheless chosen borderline ratings, with half erring on the side of rejection, and the other half tentatively recommending acceptance.

I am inclined to agree that, as it stands, the benefit of the proposed change of noise distribution is not convincingly shown to outweigh the additional complexity this introduces, so I am also recommending rejection.
Decision: Reject