Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: It has been showed recently that diffusion probabilistic models can be understood as ODE, which is useful to devise efficient prediction methods. This made it possible to strongly accelerate inference by several orders of magnitude compared to original SDE-based methods.

In this line of research, this paper shows that even more progress can be made by leveraging a decomposition of the solution as a sum of a linear and a non-linear part. The method can be used on off-the-shelf and already trained DPM, and doesn't require fine-tuning. 
strengths_and_weaknesses: The paper is both a challenging but an exciting read. It is challenging because it is pretty rigorous and the math often goes beyond my comfort zone. Still, it is exciting because it is self contained and really brings an interesting idea on the table, to devise a specific solver that allows effective inference with DPM.

Regarding the weaknesses, maybe I should say that the code provided in the supplementary is not really advertised in the paper and I think it could really benefit from some work. As such it is not documented at all so that it is really difficult to get into it. I think it would be worth it to spend some effort to make that better.
questions: * it would help me if you could provide some expression for \lambda(t). I guess this is really important in practice, and this is something that is missing. What does it mean for t when \lambda is uniformly picked ?
* I understand that your contribution is mostly relative to having faster inference, but do you think the insights you bring could actually be useful for training ?
limitations: nothing special
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a training-free few-step sampler of diffusion probabilistic models (DPMs) by leveraging semi-linear structure in the diffusion ODEs to approximate the solutions of the equation. The proposed method can accelerate the sampling of DPMs without sacrificing the sampling quality.

strengths_and_weaknesses: Strength

- (Quality) The authors effectively provide a theoretical derivation and error analysis of the proposed method and the experiments' details.
- (Clarity) The paper is well-written and clear to understand. The reviewer could understand the contents without difficulty.
- (Significance) Numerical experiments verify that DPM-solver significantly outperforms previous sampling methods in terms of quality with fewer function evaluations. The reviewer considers this paper deserves to be accepted by NeurIPS if the authors include the runtime comparison between the proposed and other methods.

Weakness

- (Originality) The proposed method shows excellent performance; however, its theoretical background is based on the variation of constants which is a classical idea in the literature of numerical differential equations, so one can think the originality is somewhat marginal.

***Updated Review***

The reviewer appreciates the authors' sincere answers. I have updated my score and hope the paper is accepted to the NeurIPS conference.
questions: Would you please include the comparison of runtime for sampling between the proposed method and other methods? The reviewer is concerned that the computational time would be a critical weak point in the proposed method.

limitations: The authors mentioned the limations of their work properly.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the slow-sampling problem in denoising diffusion generative models. Diffusion models have demonstrated outstanding synthesis quality, together with good sample diversity and mode coverage, but they are slow during sampling, since many neural networks calls are required to iteratively solve a generative differential equation and generate a single batch of data. To this end, the paper proposes a novel ODE solver, called *DPM-Solver*, that can very quickly solve the generative ODE of diffusion models. The generative ODE, also known as probability flow ODE, is of semi-linear form, consisting of both a linear and a non-linear (neural network) term. The paper proposes an ODE solver that solves the linear component analytically and exactly, while only requiring a numerical approximation for the non-linear term. The non-linear term, formally a time integral over the learnt score function with an exponential weighting, can be solved with different orders of accuracy (implying different accuracy-compute trade-offs). Overall, the framework can be interpreted as an exponential integrator approach tailored to diffusion models, with a Runge-Kutta-like technique applied to only the non-linear term. Experimentally, the paper demonstrates significantly accelerated diffusion model sampling for 3 low-resolution image generation tasks (CIFAR-10, CelebA 64x64, ImageNet 64x64), comparing to various well-established baselines.
strengths_and_weaknesses: **Stengths:**
- The paper proposes a rigorously derived and technically solid ODE integrator for fast sampling from diffusion models. The proposed method, *DPM-Solver*, is well-motivated based on techniques from the ODE literature (exponential integrators).
- The experimental results on low-resolution image synthesis successfully validate the method and show a clear advantage over the baselines.
- Overall, the paper is easy to follow and clearly explained (the English could be better, but that's okay; maybe some proofreading from a native speaker might help). The paper nicely discusses its differences compared to the most commonly used ODE solvers in diffusion models, this is, Runge-Kutta methods and DDIM [1].
- I think the contribution is sufficiently significant. The tackled problem, accelerating sampling from diffusion models, is very specific. However, this is a very relevant problem, considering that slow sampling currently is diffusion models' main weakness.

**Weaknesses:**

I do not think that the paper has major flaws. However, there are some points of criticism:
- The paper misses relevant baselines in its experimental evaluations: In particular, it would be very appropriate to quantitatively compare to "Pseudo Numerical Methods for Diffusion Models on Manifolds" [2], which is a published method that also accelerates diffusion model sampling with higher-order ODE methods. Furthermore, "FastDDIM" [3] would another appropriate baseline. Potentially, also Tachibana et al. [4] could be included. I actually think the proposed solver will outperform all these baselines. Nevertheless, I think a more rigorous evaluation with all possible baselines would strengthen the paper significantly, considering that accelerated sampling is the paper's main and only contribution.
- Beyond fast ODE samplers, the literature on diffusion models includes also many other ideas for accelerated sampling. For instance, sampling can be accelerated by working in a smooth latent space [5], by using GAN-based large-step denoising [6], of by devising tailored diffusion processes that can be denoised efficiently [7]. These works are not even cited. So, considering that this paper tackles accelerated diffusion model sampling, I think a more thorough discussion on the relevant literature is missing.
- Experimentally, the paper only uses low-resolution image datasets. The paper could be improved by also applying the method on other problems, such as high-resolution image synthesis or audio synthesis, where diffusion models have also been widely applied.

**Conclusion:**
Overall, I think this is a solid paper that deserves to be accepted. It successfully tackles a specific but relevant problem in the diffusion model literature in a convincing and technically solid way. It might be worth noting that there is highly related concurrent work that proposed a very similar method [8]. However, I don't think that this should have any negative impact here. I think exponential integration techniques are very natural to use in diffusion models, and having multiple works concurrently analyzing and demonstrating that independently in different ways is good. All that said, it also needs to be mentioned that the paper could still be significantly improved in various ways. In particular, more thorough experiments could be run and additional baselines could be considered (see weaknesses and questions below).

[1] Song et al., "Denoising Diffusion Implicit Models", ICLR 2021

[2] Liu et al., "Pseudo Numerical Methods for Diffusion Models on Manifolds", ICLR 2022

[3] Kong and Ping, "On Fast Sampling of Diffusion Probabilistic Models", arXiv 2021

[4] Tachibana et al., "Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives", arXiv 2021

[5] Vahdat et al., "Score-based Generative Modeling in Latent Space", NeurIPS 2021

[6] Xiao et al., "Tackling the Generative Learning Trilemma with Denoising Diffusion GANs", ICLR 2022

[7] Dockhorn et al., "Score-Based Generative Modeling with Critically-Damped Langevin Diffusion", ICLR 2022

[8] Zhang and Chen, "Fast Sampling of Diffusion Models with Exponential Integrator", arXiv 2022
questions: I also have some minor additional questions and suggestions:

- In the concurrent work [8], a very similar scheme, *DEIS*, is proposed. I think it would be interesting to compare and discuss the detailed differences between the method proposed here, DPM-sampler, and DEIS and also run detailed quantitative comparisons. My understanding is that DPM-Solver primarily leverages Runge-Kutta-like techniques to solve the non-linear term, while DEIS also explored muiltistepping methods that rely on past function evaluations. Furthermore, it seems the papers use different time-reparametrizations. I'd be curious how the $\rho$-parametrization in DEIS (Eq. (18) in their paper) and the $\lambda_t$-parametrization here are related. To this end, I am also wondering whether maybe the ideal future solver is based on a combination of ideas from both of these papers? As mentioned, since this is concurrent work, I don't consider this as a criterion for recommending acceptance here. But it would be nice and insightful for the community. I'd be curious about the authors' thoughts on this.

[8] Zhang and Chen, "Fast Sampling of Diffusion Models with Exponential Integrator", arXiv 2022
limitations: Limitations have been discussed only very briefly (the proposed ODE solver may not be suitable for also accelerating likelihood calcuation). It seems fine to me, although this could be extended a bit, ideally. In particular, the main limitation is that the sampler is still not fast enough to generate model samples so quickly that diffusion models can be used in real-time, interactive applications.

Potential negative societal impacts have been briefly mentioned, in a satisfactory manner.
review_title: null

Meta-review: metareview: The paper is very well-written, provides a very useful recipe for training diffusion models faster and combines that with extensive experiments. Good work!
confidence: Certain
Decision: Accept