Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at speeding up the reverse simulation process of score based denoising diffusion models through SDE. The main framework consists of two parts, where the first part is an MCMC based sampler that samples (t, x_t ~ p_t) jointly, where p_t is the marginal distribution of the forward process till time t. The second part is the standard reverse ODE simulation, where the simulation starts from this sampled time t and corresponding location x_t, and ends at t=0 to obtain the final sample. The main contribution comes from the first part where MCMC is implemented via Gibbs and stochastic gradient langevin with learned score function as the approximation. Experiments show that it can save the computation, while also improving several ODE-solver based approaches. 

strength_and_weaknesses: Strength:

- Improving the speed of the reverse process in denoising diffusion models is an important topic. 
- The proposed method is able to improve the efficiency, through the experimental justification.

Weakness:

- The novelty is somewhat limited, given the existing predictor-corrector algorithm. Using MCMC to obtain samples from p_t(x) is already done in Song et.al, 2021b. This paper directly uses MCMC to sample from p_t(x), which is a surrogate that has a trade-off between quality and the speed. But overall the technical contribution is limited as it is a relative straightforward extension of existing technique.

- There might be a potential issue for the first stage (i.e., using MCMC for p_t). The score function required for MCMC is approximated using the learned score function. It would be fine if the sample x is roughly around the high density region of p_t(x), however the initial samples fed to the score function might be quite far away. These would be the out-of-distribution samples for the neural parameterized score function, where the neural network may not be able to extrapolate that well. 

- Paper [1] is included but is not directly mentioned or compared. The authors are encouraged to provide more experimental comparison with [1]. 

Reference:

[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Lu et.al.


clarity,_quality,_novelty_and_reproducibility: Overall the paper is easy to follow and writing is Ok. The novelty is limited (see my comments above). The code is not provided, as the paper requires quite some tuning, the reproducibility might be an issue. 

summary_of_the_review: Overall this paper proposed a practical approach for speeding up the reverse process sampling of denoising diffusion models. The main concern comes from the novelty and technical correctness of the assumption/approximation. The experiment section can potentially be improved with more baseline comparison.

====
updated after the authors' response and more experimental details.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors propose to extend the state space of diffusion models to answer the following question: how much should we sample at a given noise level? The way they proceed is as follows. First, they put some prior on the space of noise level $p(\sigma)$ (in practice this distribution is chosen so that $p(\sigma) \propto 1/\sigma$ to put more emphasis on the distribution with small noise. This is because we ultimately want to sample from the original distribution  (without noise). Then we can easily write the joint distribution on the space of image and noise levels by using $p(x|\sigma)p(\sigma)$. Then we sample using a Gibbs procedure which can be described as follows:
* Given a noise level we can approximately sample from $p(x|\sigma)$ leveraging diffusion models. In particular, we use an Unadjusted Langevin Dynamics targetting $p(x|\sigma)$ where the gradient of the logarithm of this density is given by the learned score of the diffusion model. This part is really similar to a diffusion model
* Second we need to sample from $p(\sigma|x)$. To do so the authors train a neural network to predict the noise level. In fact even though we need to learn a probability distribution the authors found that just outputing one value of $\sigma$ corresponding to the mode of the distribution worked well.
The authors then alternate between the two steps. They illustrate their method on CelebA-HQ 256 and CIFAR 10.
strength_and_weaknesses: STRENGTHS:
* I think the ideas of the paper are clearly explained. I like the idea of extending the state space to also sample from the parameter s$\sigma$. I think this is an elegant idea that could lead to further developments.
* The experimental results seem to confirm that using the proposed method Denoising Langevin Gibbs (DLG) brings an advantage to existing sampling techniques.

WEAKNESSES:
* At the core level I think there is something weird with the strategy. Classical denoising diffusion models aim at denoising the image. They start with a Gaussian random variable and then go backward in time to reach the initial distribution. In that case the $\sigma$ decrease towards $0$ which makes sense as we are trying to go back to the data distribution. But here this is quite different as we are also sampling from $p(\sigma|x)$. At the end of the day (if things were perfect) we should sample from $p(x, \sigma)$. Hence, the noise should travel to the whole interval $[\sigma_{\mathrm{min}}, \sigma_{\mathrm{max}}]$ and not be stuck at $\sigma_{\mathrm{min}}$. Being stuck near $\sigma_{\mathrm{min}}$ has the advantage that then the samples are perceptually good but then we do not explore the whole distribution. It would be incredibly useful and I think key to the paper if the authors could provide the evolution of the sequence of $\sigma$. If this sequence is decreasing only then we are not exploring the distribution and the claim that we are doing a Markov chain in the augmented space is misleading. Maybe this is due to the fact that we use a classifier to learn the noise level $\sigma$. In any case, I think that the sampling on $\sigma$ which is the real methodological contribution of the paper is not investigated enough by the authors.
* Why do the authors focus on the Variance Exploding SDE? In the literature it has been shown that usually VP-SDE (which can be connected to DDPM) are more stable and yield better results than VE-SDE. This is because VE-SDE is just a Brownian motion whereas VP-SDE is a Ornstein-Uhlenbeck process (for which the variance does not explode and converge to the dimension). I am pretty sure that the analysis still holds in that case.
* I think that the authors missed a few references trying to predict the noise level in the sampling. For instance [1] also learned the covariance in DDPM models. In [2] the authors learn the noise level. The method seems very similar to what is introduced and is not discussed in the paper.
* I think that the claims on SOTA for FID for Celeba256x256 as some methods seem to achieve 3.25 see https://paperswithcode.com/sota/image-generation-on-celeba-256x256

OTHER QUESTIONS/COMMENTS:
* I don't really understand the discussion at the end of Section 3.2.
* In Equation (15) the authors seem to be using the Langevin dynamics targetting (the approximate) noised target distribution. This corresponds to the corrector step in [3]. Empirical analysis in [3] shows that using only the corrector step and not the predictor-corrector (or only predictor) step was very detrimental to the quality of the sampling. However it does not seem to affect the results of the present work. Could the authors give more details about this?

[1] Improved Denoising Diffusion Probabilistic Models - Nichol, Dhariwal
[2] Noise Estimation for Generative Diffusion Models - San Roman, Nachmani, Wolf
[3] Score-Based Generative Modeling through Stochastic Differential Equations - Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole
clarity,_quality,_novelty_and_reproducibility: Paper is clearly written. I think there is a bit of uncertainty regarding what really happens with the sequence $\sigma$ so the methodology is not perfectly clear.

The work is coherent and the methodology is a nice idea.

It seems to be novel although I really think that a similar idea was introduced in [1] and should be discussed.

The authors give experimental details.

[1] Noise Estimation for Generative Diffusion Models - San Roman, Nachmani, Wolf

summary_of_the_review: I think this work is an interesting contribution to the field.
However, as of now I think that there are a few elements missing:
* a discussion of the literature to assess the novelty of the idea
* a discussion of some methodological choices that seem to be quite non-classical (VE-SDE, corrector only step)
* verify claim on SOTA for Celeba256.
* most importantly, a study of the evolution of $\sigma$ to assess if we are really sampling from a Markov chain.
For these reasons, I think that as of now reasons to reject the paper outweight reasons to accept it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a faster algorithm, DMCMC, for integrating the reverse SDE/ODE associated with diffusion models. Unlike many existing works, DMCMC samples in the joint space of (x, sigma). Gibbs sampling is adopted for sampling in the joint space. Sampling x | sigma is just ordinary Langevin dynamics, and sampling sigma | x requires additional training of a time prediction network. After MCMC sampling, the algorithm will traverse to a time close to 0. Then, differential equation solvers can be adopted to generate the final sample. DMCMC can be combined with existing solvers, and it can generate high-quality samples in 10~20 NFE. 
strength_and_weaknesses: Strength:
- Sampling from diffusion models is an important problem;
- Sampling in the joint space is somewhat novel, and I am somewhat surprised that the authors are able to make that work.
- The proposed sampler is efficient.

Weaknesses:
- How does the proposed method work with more recent fast solvers, such as DPM-Solver and DEIS? I wonder whether the proposed approach can still help if the solver itself is fast enough. 
- The noise predictor network is a trained distribution rather than the true conditional p(sigma | x). With such an approximate conditional, MCMC can no longer guarantee to converge to the true stationary distribution (so rigorously it is not a Gibbs sampling algorithm). I wonder to what extent does this approximation affect the convergence / sample quality.
- It is not that intuitive to me how sampling in the joint space can be much faster than annealed MCMC. Doesn't the sampler still require many steps in the x space to mix? It would be better if the authors can better explain the mechanism behind the acceleration behavior. I understand that sampling from annealed distributions is better than directly sampling from p(x_0). But I don't quite understand why one should sample sigma rather than just using a deterministic schedule like other SDE solvers. Does it correspond to some adaptive solvers?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is mostly clear, though I think it would be better to include a pseudocode in the paper.
Quality: The technical quality is good and the results are decent.
Novelty: Sampling from the joint space differs from previous approaches of building faster solvers.
Originality: To the best of my knowledge, the proposed approach is novel. Though I could possibly miss some relevant works. 
summary_of_the_review: The paper proposes a novel way to accelerate the sampling of diffusion models. The proposed method can be combined with other approaches, which is an advantage. However, there are still some unclear aspects about the spectrum of the applicability / mechanism of the acceleration behavior.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new way of acceleration diffusion based generative models that combines MCMC on a augmented space with reverse-S/ODE integrators. More specifically, a Gibbs sampler procedure is introduced to traverse the augmented space based on a pre-trained classifier. Numerical experiments demonstate the effectiveness of the proposed method.
strength_and_weaknesses: Stength:

1. The paper is well written and organized.
2. The idea of using MCMC on the augment space for accelerating diffusion models is interesting and new.


Weaknesses:

1. The intialization of $x_0$ for DLG is not clearly stated. The author claimed that it needs to be set close to the image manifold. However, they also say $x_0$ is generated by sampling from a noise distribution, adding Gaussian noise of variance $0.25$, and runing Gibbs sampling a few iterations. I am confused with how $x_0$ is generated in practice. If it is generated by adding Gaussian noise to samples from real image data, the same trick can be applied to general diffusion models for acceleration which is not included in the ablation study.

2. I think the comparison to LD in section 5.1 is unfair. It would be better to compare with ALD with appropriate annealing schedules.

3. The biaseness towards high-density regions of $\mathcal{X}$ is controlled by the prior $\hat{p}(\sigma)$. Without appropriate choice of $\hat{p}(\sigma)$, the claim in section 3.2 needs not to hold.

4. Usually, MCMC takes some iterations for convergence. In practice, we want to use prior $\hat{p}(\sigma)$ the encourages small noise. There seems to be a trade-off between the mixing time of MCMC and the choice of $\hat{p}(\sigma)$ which is not discussed in this paper.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and well organized. Combining MCMC with reverse integrators is new for accelerating diffusion models.
summary_of_the_review: The idea of using MCMC to accelerate reverse integration of diffusion model is interesting and experiments show positive evidence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper develops a different way to sample from a diffusion-based generative model. Diffusion based generative modeling typically transforms noise to something that looks like a data by simulating an SDE. This simulation can be slow. To make generation faster the authors propose to augment the sampling space with variances or times and generate on the augmented space using Gibbs sampling. The idea is that favoring smaller times will keep samples closer to data rather than having to start from scratch. 

The positives are sampling from a diffusion faster is a very important problem and the idea of augmenting the space to sample the noise is fresh; the samples evaluate well with FID. 

The negatives are that the paper makes specific choices, such as using the variance exploding SDE, so it's unclear if the proposed method would generalize to other types of diffusions (reviewer uZQw) that often yield better results, such as the critically damped langevin difffusion. As noted in the author reply, not considering other diffusions seems to limit other techniques to speed up sampling. The other negatives are that the writing as evidenced by details queried by requested of the reviewers and that the prior on the variances ties the overall mixing of the gibbs sampler with the rate at which samples get accepted (when the noise is low) (reviewer DAST).

The author reply and revision cleans up some of these issues (one of the reviewers increased their score), but the majority of the changes are relegated to the appendix. The work would be stronger if some more of this material was weaved into the main text.
justification_for_why_not_higher_score: The negatives are that the paper makes specific choices, such as using the variance exploding SDE, so it's unclear if the proposed method would generalize to other types of diffusions that often yield better results, such as the critically damped langevin difffusion. As noted in the author reply, not considering other diffusions seems to limit other techniques to speed up sampling. The other negatives are that the writing as evidenced by details queried by requested of the reviewers and that the prior on the variances ties the overall mixing of the gibbs sampler with the rate at which samples get accepted (when the noise is low).
Decision: Reject