Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a framework to treat Denoising Diffusion Probabilistic Models (DDPMs) as solving differential equations on manifolds. This goal is faster sampling without significant loss of quality.
main_review: I thank the authors for this submission. I believe there is value in this work both from a theoretical and practical perspective. In general, I am willing to accept the paper. I have two main suggestions:

1. Please double-check the writing and notations. Some sentences are hard to understand and equations contain errors, e.g. Eq. 2, hat{alpha}_t, wrong index.

2. As DDPMs are relatively new, I was expecting a bit more elaborate introduction.

3. I was honestly quite disappointed with the presentation at times. Some equations are stated without any intuition. If you are short of space, I would rather move some of the experimental parts to a supplementary.

4. Sec. 4.3 and in general, I would also work with toy problems and easily visualizable data. This will provide additional insight for the reader.
summary_of_the_review: I am not very familiar with DDPMs but as far as I can tell, the numerical technique introduced makes sense and leads to improved results. I, therefore, recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This article studies pseudo numerical methods that improve and accelerate upon exiting numerical methods for Denoising Diffusion Probabilistic Models (DDPMs). The crux of this work is the observation that existing work on this topic does not take into consideration the structure of the high-density region of the data. 
main_review: The beginning of the article (Section 1 and Section 2) is well written and presents the motivation behind the introduction of these pseudo numerical methods in a very didactic way. However, Section 3 needs a lot of polishing. These are some comments and suggestions: 

1. The contribution of this work is based on the assumption that $\sigma_t = 0$. The conditions required to satisfy this assumption deserve to be commented.


2. The mention of the manifolds is very implicit and requires to be clarified: what is the definition of the manifold defined at the beginning of Section 3.2?

3. The paragraph at the beginning of Section 3.3, supposed to give the intuition behind the introduction of the transfer part defined in (11), is not clear and should be reformulated. For example, what do you mean by ‘We find that Equation (9) has the property that if ϵ is the precise noise in $x_{t}$, then the result of $x_{t−\delta}$ is also precise, no matter how big $\delta$ is’?

4. In Algorithm 2, PLMS and PRK were not defined


5. It would be preferable to put the theoretical results of Section 3.6 in a clearly stated theorem. This would highlight the theoretical contributions of this work. 







summary_of_the_review: The novelty of the methodology presented in this paper qualifies this work to be accepted to the conference conditionally to improve the clarity of Section 3.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new efficient method for denoising diffusion probabilistic models (DDPM) (generative models that optimize for the closest solution on a manifold) based on the observation that this can be seen a solving a set of differential equations on a manifold. This allows efficient pseudo numerical methods to be applied here which have many advantageous properties over classical optimization methods, including less optimization steps and guaranteed manifold solutions due to separating the gradient part from the transfer part in the optimization. Results are shown on four datasets comparing to two reasonable but not sota baselines. 
main_review: The paper introduces a class of pseudo numerical methods for DDPM, this is based on a previous work that already establishes the relationship between DDPMs and a certain class of differential equations on manifolds. The pseudo numerical methods separate the gradient and transfer step of classical numerical algorithms, and for each part choose the best of both worlds. With this they are able to provide faster converengence and efficient update steps because gradients do not need to be recomputed for every step. While I would not guarantee that methods do not exist already in classical optimization, I have not seen them in any related application. 

The paper is overall well written, and the contribution and main ideas are explained clearly. Starting at Section 3.3 (to 4) the derivations become slightly confusing, and it takes a lot of referencing back to find all the variable names again. A legend for the variables and a bit more redundant explanation would probably help here. The same goes for the derivations in the appendix which I could not follow completely. 

Experiments are done on four datasets with different resolutions using pretrained models for the manifold description. The results show that the introduce pseudo numerical methods converge much faster and provide better results in less iterations than the previous DDIM method and classical numerical methods. The qualitative examples in the appendix look good, but some of them are too smooth to compete with general sota generators (I am not sure if the smoothness here is related to the pretrained models that were used?). 

Minor comments:
- The authors claim that their implementation is in the supplementary. I was not able to find any supplementary (except the appendix) but I am honestly not sure if this is a failure of me using OpenReview... The implementation should definitely be published in the end though. 
- I think it is bad practice to move the related work section in the appendix. 
- The large figures in the appendix are very hard to understand because the subfigures are not separately titled. 
- There are some grammatical errors throughout the text which should be proofread again. 
summary_of_the_review: The idea of separating the gradient and transfer part is (to the best of my knowledge, I am not an expert though) novel, and I can see many applications besides the ones proposed in this paper. The shown results might not be exactly state-of-the-art in terms of generating images, but show clear advantages over traditional numerical methods. Therefore I recommend accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Highlighting the high computational complexity for sampling from Denoising Diffusion Probabilistic Models (DDPMs) (e.g. wrt GANs), authors build on the connection between diffusion processes and ODEs to propose efficient (pseudo-)numerical methods so as to sample data from the data manifold. The main idea is to combine the discrete update proposed in DDIMs, with a fourth-order gradient estimation given by the Runge-Kutta or linear multi-step methods. The motivation being that such gradient estimator should yield trajectories that stay closer to the data manifold. They empirically assess their method(s) on CIFAR10 and CelebA in terms of sample quality (measured by FID), and show that they get a ~x20 speed-up wrt DDIMs or a significant improvement in FID with the same number of steps.

main_review: ## Clarity.
Overall, I believe that the clarity of the submission should be enhanced.

First, the introduction can be improved to better stress the motivation of the submission. If I understand correctly, this work builds on Probability Flows (Song et al., 2020), which leverage the existence of a deterministic process whose trajectories have the same densities as the original diffusion process. This deterministic process satisfies an ODE that depends on the score original drift and diffusion terms but also on the score function. Consequently classical numerical ODE solvers (e.g. RK) can be leveraged to sample data from the probabilistic model. Authors then state that results obtained via this approach are subpar and suggest that this is due to the solvers tendency to sample data far from the data manifold.

1/ Why is this true? Would be necessary to give some intuition and to refer to a theoretical analysis

2/ What is the precise problem? Is the issue that the model oversample the data distribution's tail (i.e. fails to fit the distribution properly) or that the numerical methods fail (in what sense?) in these areas as the score is undefined (or hard to estimate)? SMLD (Song and Ermon, 2020)'s motivation to inject noise is built on the latter.

The authors then provide "pseudo numerical methods for diffusion models (PNDMs)" which produces trajectories that "iterate data on the high-density region of the data", hence tackling the aforementioned issue.

Also, perhaps this is question of taste, but I believe that the Background Section (and most of the paper)'s clarity could be greatly improved by taking the perspective of Song et al. (2020b), that is, a continuous diffusion process (forward / perturbating data) and the associated reverse diffusion process (generating data).

Additionally, Section 3.1 is challenging to follow. Would perhaps be better to put less equations but spend more time explaining why and how they matter.

## Strengths
First, the proposed method is conceptually simple and showcase previously proposed methods (DDIMs) as a special case where the update is a (first-order) Euler step.

Then, the submission shows strong empirical results on common datasets like CelebA, with faster convergences or significantly better FID (for the same number of steps) yielding SOTA. Authors report a ~x20 speed-up wrt DDIM, but this in number of steps and fourth order methods trade-off convergence speed for computational cost. Figure 3 suggests that the speed-up is around x15 for CIFAR which is still significant (although it would ideal to report the runtime directly in Table 2).

Finally, Figure 4 is quite nice as it empirically illustrate the proposed method ability to sample trajectories that like closer to the data manifold, which was the original motivation.

## Weaknesses
I think that the main weakness is the writing.
Although I believe that Section 4.3 would deserve a deeper empirical analysis as it is directly investigating the core motivation of this submission.

## Relation to prior work
- Perhaps worth citing Sohl-Dickstein et al., 2015 in Section 1?
- Citations for Runge-Kutta and the Linear Multi-Step methods appear to be missing.
- It is not entirely clear to me what method is meant by Probability Flows (Song et al., 2020b), is it with Variance Exploding SDE (SMLD) or Variance Exploding SDE (SMLD) (cf Table 1 from that paper).


## Additional feedback.
- "However, classical numerical methods (Sauer, 2017) have problems when they are applied to DDPMs." -> What class of numerical methods? The Euler and RK methods are ODE solvers, although extensions exist for SDEs. It is unclear how they can be applied to DDPMs. Or is it implicitly implied that they are used for the the corresponding deterministic process (probability flow)?
- "to iterate our data on the high-density region" -> would suggest to reformulate
- Eq 3: $\epsilon_\theta$ is not defined. Would advise to do so so for the paper to be self-contained, especially as $\epsilon_\theta$ is used through the entire paper.
- Table 2: Error bars / confidence intervals are missing. Bold is not defined. As methods have different computational cost per step would be very useful to additionally show this metric.
- Figure 3: Time has no unit.
- Figure 4: Axis names are missing.

summary_of_the_review: I personally find this submission interesting and significant, yet believe that clarity needs to be improved to enable readers take the most of the paper's insights.
review_title: null

Meta-review: This paper presents a new DDPM model based on solving differential equations on a manifold.  The resulting numerics appear to be favorable, with faster performance than past models.

Most of the reviews thought the main result was of interest and were impressed with the performance.  Reviewer c9bY points out some challenging issues and analytical questions that remain unanswered in the text; they also have some simpler textual revisions that seem less important.

In general, this paper has the misfortune of receiving reviews whose confidence appears to be low.  While partially this is a byproduct of the noisy machine learning review system, the difficulty of the text itself is substantial and made the paper less than approachable; the authors are encouraged to continue to revise their text based on feedback from as many readers as possible.  That said, the authors were quite responsive to reviewer comments during the rebuttal phase, which significantly improved the text.

Overall this is a borderline case, and the AC also had some difficulty following details of this technically dense paper.  Given the positive *technical* assessments of the work and at least one reviewer defending the paper's clarity, the AC is willing to give this paper the benefit of the doubt.
Decision: Accept (Poster)