DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
Authors: Cheng Lu
Yuhao Zhou
Fan Bao
Jianfei Chen
Chongxuan Li
Jun Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: diffusion probabilistic models
score-based generative models
fast sampling
ODE solver

Abstract: Diffusion probabilistic models (DPMs) are emerging powerful generative models. Despite their high-quality generation performance, DPMs still suffer from their slow sampling as they generally need hundreds or thousands of sequential function evaluations (steps) of large neural networks to draw a sample. Sampling from DPMs can be viewed alternatively as solving the corresponding diffusion ordinary differential equations (ODEs). In this work, we propose an exact formulation of the solution of diffusion ODEs. The formulation analytically computes the linear part of the solution, rather than leaving all terms to black-box ODE solvers as adopted in previous works. By applying change-of-variable, the solution can be equivalently simplified to an exponentially weighted integral of the neural network. Based on our formulation, we propose DPM-Solver, a fast dedicated high-order solver for diffusion ODEs with the convergence order guarantee. DPM-Solver is suitable for both discrete-time and continuous-time DPMs without any further training. Experimental results show that DPM-Solver can generate high-quality samples in only 10 to 20 function evaluations on various datasets. We achieve 4.70 FID in 10 function evaluations and 2.87 FID in 20 function evaluations on the CIFAR10 dataset, and a 4~16x speedup compared with previous state-of-the-art training-free samplers on various datasets.

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
Cheng Lu", Yuhao Zhou', Fan Bao", Jianfei Chen Chongxuan Li Jun Zhut* 'Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab "Tsinghua-Bosch Joint MI. Center, Tsinghua University, Beijing, 100084 China 'Gaoling School of Artificial Intelligence, Renmin University of China, Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China {lucheng 1c15, yuhaoz. cs Hagmail com; bf19@mails. wai.sguusngnaaa edu. chongxuanli iirucc edu. en; [jianfeic, dcsz))@tsinghut edu. on
Abstract
Diffusion probabilistic models (DPMs) are emerging powerful generative models. Despite their high-quality generation performance, DPMs still suffer from their slow sampling as they generally need hundreds or thousands of sequential function evaluations (steps) of large neural networks to draw a sample. Sampling from DPMs can be viewed alternatively as solving the corresponding diffusion ordinary differential equations (ODEs). In this work, we propose an exact formulation of the solution of diffusion ODEs. The formulation analytically computes the linear part of the solution, rather than leaving all terms black-box ODE solvers as adopted in previous works. By applying change-of-variable the solution be equivalently simplified to an exponentially weighted integral of the neural network. propose DPM-SP DPM-Solver, fast dedicated high-order solver for diffusion ODEs with the convergence order guarantee. DPM-Solver is suitable for both discrete-time and continuous-timg DPMs without any further training. Experimental results show that DPM-Solver can generate high-quality samples in only 10 20 function evaluations various datasets. We achieve 4.70 FID in 10 function evaluations and 2.87 FID 20 function evaluations on the CIFARIO dataset, and 4 ~ 16x speedup compared with previous state-of-the-art training-free samplers on various datasets
Based
formulation,
Introduction
Diffusion probabilistic models (DPMs) 11-3 are emerging powerful generative models with promis- ing performance on many tasks, such as image generation [4, video generation (6, text-to-image generation 77.. speech synthesis 8 9 and lossless compression [10] DPMs defined by discrete- time random processes 2 or continuous-time stochastic differential equations (SDEs) 3 which learn to gradually remove the noise added the data points. Compared with the widely-used genera- tive adversarial networks (GANs) [ and variational auto-encoders (VAEs) [2], DPMs can not only compute exact likelihood (.) but also achieve even better sample quality for image generation 4 However, to obtain high-quality samples, DPMs usually need hundreds or thousands of sequential steps of large neural network evaluations, thereby resulting in much slower sampling speed than the single-step GANs or VAFs. Such inefficiency is becoming critical bottleneck for the adoption of DPMs in downstream tasks, leading to an urgent request to design fast samplers for DPMs.
"Corresponding Author.
"Code available at
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

NFE NFE 15
NFE 20
NFE 100
NFE 10
DDIM 10
(b) DPM-Solver (ours)
Figure I: Samples by DDIM 19 with 10, 15, 20, 100 number of function evaluations (NFE), and DoM-Sr (ours) with only 10 NFE, using the pre-trained DPMs on ImageNet 256x256 with classifier guidance 4
Existing fast samplers for DPMs can be divided into two categories. The first category includes knowledge distillation [3 [4] and noise level or sample trajectory learning [15H18]. Such methods require a possibly expensive training stage before they can be used for efficient sampling. Furthermore, their applicability and flexibility might be limited. It might require nontrivial effort to adapt the method to different models, datasets, and number of sampling steps. The second category consists of training-free [1922]] samplers, which are suitable for all pre-trained DPMs in simple plug-and- play manner. Training-free samplers include adopting implicit [19] or analytical (21] generation
process, advanced differential equation (DE) solvers [3 20 24 and dynamic programming [18] However, these methods still require 50 function evaluations
21 generate high-quality samples
(comparable to those generated by plain samplers in about 1000 function evaluations), thereby are still consumnn.
In this work, we bring the efficiency of training-free samplers to a new level to produce high-quality samples in the "few-step sampling" regime, where the sampling can be done within around 10 steps of sequential function evaluations. We tackle the alternative problem of sampling from DPMs as solving the corresponding diffusion ordinary differential equations (ODEs) of DPMs, and carefully examine the structure of diffusion ODEs. Diffusion ODEs have semi-linear structure - they consist of linear function of the data variable and a nonlinear function parameterized by neural networks. Such structure is omitted in previous training- samplers B 20), which directly use black-box DE solvers. To utilize the semi-linear structure, we derive an exact formulation of the solutions of diffusion ODEs by analytically computing the linear part of the solutions, avoiding the corresponding discretization error. Furthermore, by applying change-of-variabe simplified to an exponentially weighted integral of the neural network. Such integral is very special and can be efficiently approximated by the numerical methods for exponential integrators (25].
the solutions can be equivalently
Based on our formulation of solutions, we propose DPM-Solver, a fast dedicated solver for diffusson ODEs by approximating the above integral. Specifically, we propose first-order, second-order and third-order versions of DPM-Solver with convergence order guarantees. We further propose an adaptive step size schedule for DPM-Solver. In general, DPM-Solver is applicable to both continuous- time and discrete-time DPMs, and also conditional sampling with classifier guidance (4. Fig. []] demonstrates the speedup performance of Denoising Diffusion Implicit Models (DDIM) [19 baseline and DPM-Solver, which shows that oPo-Solver can generate high-quality samples with as few as 10 function evaluations and is much faster than DDIM on the ImageNet 256x256 dataset (2
Our additional experimental results show that DPM-Solver can greatly improve the sampling speed of both discrete- time and continuous- DPMs, and it can achieve excellent sample quality around 10 function evaluations, which is much faster than all previous training-free samplers of DPMs.
2 Diffusion Probabilistic Models
We review diffusion probabilistic models and their associated differential equations in this section.

---

2.1 Forward Process and Diffusion SDEs
Assume that we have D-dimensional random variable 20 E RD with an unknown distribution 2o (2)).. Diffusion Probabilistic Models (DPMs) [0 define forward process (ax ( with
T > 0 starting with To, such that for any t E [0, the distribution of o, conditioned on æo satisfies
goe (ar (m0) N (V==)a(())) o² (I)T)
(2.1)
where (() R+ are differentiable functions of & with bounded derivatives, and we denote them as Okt, or for simplicity. The choice for aa and or is referred to as the noise schedule of a DPM. Let qe (2r,) denote the marginal distribution of æt, DPMs choose noise schedules ensure that qr (x(r)) = (x 00 I) for some 5 > 0, and the signal-to-me poaall-to-oiis-ratti (SNR) of /o is strictly decreasing w.r.t. [10]. Moreover, Kingma al. [TO] prove that the following stochastic differential equation (SDE) has the same transition distribution que (o, 2o) as in Eq. (2.1 for any E [0,
dæ. d(t) + g(t)dwr,
Do ~ qu(u)))
(2.2)
where W, E IR is the standard Wiener process, and
dl log Oz dt
do?
d log or oz.
f(t)
g²(1)
(2.3)
dt
dt
Under some regularity conditions, Song et al. B show that the forward process in Eq. (2.2) has an equivalent reverse process from time T to O, starting with the marginal distribution (zr):
dæ, [f(t)= /2 V log ge (x:))) + g(()dur ØT ~ qr())),
(2.4)
where W. is standard Wiener process in the reverse time. The only unknown term in Eq. (2.4) is the score function V log ge (Zz)) at each time t. In practice, DPMs use a neural network E (Zg t) parameterized by 9 to estimate the scaled score function: -o Va log (xx)). The parameter optimized by minimizing the following objective 22
33:
[(0;w(1)) = /- (()) (a))) [1e((1))))) 1) + o. V log
w(t) Bg(e) [10( t) - e dt dt +C L. 1
2
where wt) is weighting function, q(L) Ne(e I). x; a4a0 + ore, and C' is constant independent of 9. As Es(2,, t) can also be regarded as predicting the Gaussian noise added to a usually called the noise prediction model. Since the ground truth of ca(x,, t) is -o; V log qe (xx), DPMs replace the score function in Eq. (2.4) by - -Ep (x,, t)/a, and define a parameterized reverse process (diffusion SDE) from time T to 0, starting with ør N(0,02))
dx; = [10)a + g2(t) Ea ca( 1) dt + gt)ddw. ar ~ N(0,591).
(2.5)
Samples can be generated from DPMs by solving the diffusion SDE in Eq. 2.5 with numerical solvers, which discretize the SDE from T to O. Song al. (3] proved that the traditional ancestral sampling method for DPMs (2 can be viewed as first-order SDE solver for Eq. (.5).. However, these first-order methods usually need hundreds of or thousands of function evaluations converge B3) leading to extremely slow sampling speed.
2.2 Diffusion (Probability Flow) ODEs
When discretizing SDEs, the step size is limited by the randomness the Wiener process [27] Chap. 11]. A large step size (small number of steps) often causes non-convergence, especially in high dimensional spaces. For faster sampling, one can consider the associated probability flow ODE (3) which has the same marginal distribution at each time / as that of the SDE. Specifically, for DPMs, Song et
3 proved that the probability flow ODE of Eq. 2.4p is
dæ: dt
1 (tt log (x.) ET (r(
f(t)a,
(2.6)
3

---

where the marginal distribution of 2t is also qe (z:). By replacing the score function with the noise prediction model, Song al. 3 defined the following parameterized ODE (diffusion ODE):
dæ. dt ho f(()xz + g²(1)
(01,1))) (), Ør (10
(2.7)
20,
Samples can be drawn by solving the ODE from T to O. Comparing with SDEs, ODEs can be solved with larger step sizes as they have no randomness. Furthermore, we can take advantage of efficient numerical ODE solvers to accelerate the sampling. Song et al. K used the RK45 ODE solver (28
for the diffusion ODEs, which generates samples in ~ 60 function evaluations to reach comparable quality with 1000-step SDE solver for Eq. (.5) on the CIFAR-10 10 dataset 29) However, existing ceneral-purposs ODE solvers still cannot generate satisfactory samples in the few-step (~ 10 steps)
sampling regime. To the best of our knowledge, there is still a lack of training-free samplers for DPMs in the few-step sampling regime, and the sampling speed of DPMs is still critical issue.
3 Customized Fast Solvers for Diffusion ODEs
As highlighted in Sec. 2.2 discretizing SDEs is generally difficult in high dimensions [27] Chap. 11] and it is hard to converge within few steps. In contrast, ODEs are easier to solve, yielding a potential for fast samplers. However, mentioned in Sec. 2.2, the general black-box ODE solver used in previous work 3 empirically fails to converge in few steps. This motivates us to design dedicated solver for diffusion ODEs enable fast and nigh-quality few-step sampling. We start with a detailed investigation of the specific structure of diffusion ODEs.
3.1 Simplified Formulation of Exact Solutions of Diffusion ODEs
The key insight of this work is that given an initial value v, at time >> 0, the solution æ, at each time t < s of diffusion ODEs in Eq. (2.7) can be simplified into a very special exact formulation which can be efficiently approximated.
Our first key observation is that a part of the solution æ; can be exactly computed by considering the particular structure of diffusion ODEs. The r.h.s. of diffusion ODEs in Eq. (2.7) consists of two parts:
((())
the part (t)) xe is a linear function of æ,, and the other part 20. co (xt, t) is generally nonlinear function of xy because of the neural network cø ( t). This type of ODE is referred to as semi-lineat
ODE. The black-box ODE solvers adopted by previous work 33 are ignorant of this cemi-linear structure as they take the whole (a,,, I) in Eq. (2.7) as the input, which causes discretization errors of both the linear and nonlinear term. We note that for semi-linear ODEs, the solution time can be exactly formulated by the "variation of constants" " formula (30]:
Et = e.. f(F)dT x, + (f g2()
co(o..)) dr. 20,
(3.1)
This formulation decouples the linear part and the nonlinear part. In contrast to black-box ODE solvers, the linear part is now exactly computed, which eliminates the approximation error of the linear term. However, the integral of the nonlinear part is still complicated because it couples the coefficients about the noise schedule (i.e., ((),,(()). o.)) and the complex neural network eg. which still hard to approximate.
Our second key observation that the integral of the nonlinear part can be greatly simplified by introducing a special variable. Let X == log log(a, / (one half of the log-SNRA then A is a strictly decreasing function of t (due to the definition of DPMs as discussed in Sec. 2. We can rewrite g(t)
in Eq. (..)
dlog a d
dl o. =20² cllo oe
d di
dy,
(²(1) di
20
(3.2)
di
dl
Combining with f(t) = d log at dt in Eq. (2.3). 3, we can rewrite Eq. (3. as
04 F 04 /' (d)r Or co(x,,770 T)dr. dr ar
x
(3.3)
As A(t) = X, is a strictly decreasing function of t, it has an inverse function tx(+) satisfying t= (A(t).. We further change the subscripts of æ and Eg from to À and denote 2x = (x) E A) (2) Rewrite Eq. 333 by "change-of
for X, then we have:

---

Proposition 3.1 (Exact solution of diffusion ODEs), Given an initial value at time s >> 0, the solution 2, at time t [0, of diffusion ODEs in Eq. 277
0t U. at
Ft =
éa AJdA.
(3.4)
0s
We call the integral fe AJdA the exponentially weighted integral of ég. which very special and highly related to the exponential integrators in the literature of ODE solvers [25]. To the best of our knowledge, such formulation has not been revealed in prior work of diffusion models.
provides a new perspective for approximating the solutions diffusion ODEs. Specifically, given æ, time According to Eq. (3.4). approximating the solution at time is equivalent to directly approximating the exponentially weighted integral of ég from X to A, which avoids the
Eq. (3.4)
error of the linear terms and is well-studied in the literature of exponential integrators [25] 31. Based
on this insight, we propose fast solvers for diffusion ODEs, as detailed in the following sections.
3.2 High-Ordes Solvers for Diffusion ODEs
In this section, we propose high-order solvers for diffusion ODEs with convergence order guarantee by leveraging our proposed solution formulation Eq. (3.4). The proposed solvers and analysis are highly motivated by the methods of exponential integrators 25 31 in the ODE literature.
Specificallyy, given an initial value æт at time T and M + time steps MM decreasing from
to = T to tM 0. Let Ttg Zr be the initial value. The proposed solvers use M steps to iteratively compute a sequence ] Ji-o to approximate the true solutions at time steps { In particular,
the last iterate Ci, approximates the true solution at time 0.
In order to reduce the approximation error between Tts and the true solution time 0, need to reduce the approximation error for each Tt at every step [30]. Starting with the previous value ب time 44-1, according to Eq. (3.4), the exact solution Ft; 1-, at time 4 is given by
des
Cte Lt... Ctt
- Te(xx,A)da
(3.5)
Olt--1
Aes
Therefore, to compute the value Tt, for approximating Te-1- we need to approximate the exponentially weighted integral of éy from do__, to Ae,- Denote h4 == de, - and ef") (Øx. A) = d' (d)x(A))) the n-th order total derivative of é A) w.r.t. A. For 1, the (k 1) 1)-th order dAa
Taylor expansion of éo(ăs, A) w.r.t. at Ad, is
dr, ) ((m) (a (
( A)
a=0 n!
Substituting the above Taylor expansion into Eq. 3.5
yields
Ot:: -t.-n
(A dr, ,)" dd ax+ O(++))))) (3.6)
1 04;-1
(
n!.
where the integral f -- of integration-by-part (see Appendix B.2 Therefore, approximate It;-1-+,++ we only need to approximate the n-th order total derivatives e for S k - 1, which is well-studied problem in the ODE literature B1 32). By dropping the Ohh ++1+ error term and approximating the first (k- 1)-th total derivatives with the "stiff order conditions" [3 32), we can derive k-th-
yd can be analytically computed by repeatedly applying n times
ODE solvers for diffusion ODEs. We name such solvers DPM-Solver overall, and DPM-Solver-k
for specific order k. Here we take k- = for demonstration. In this case, Eq. (3.6) becomes
Ott Tt,_, a4, ( / e c
:(-1)
0t,11
Cit; -tj-1 - (r.(ch - 1)
01h3)
Ott-11
5

---

By dropping the high-order error term ( we can obtain approximation for -t,-1-4.- As k here, we call this solver DPM-Solver-,, and the detailed algorithm is following.
DPM-Solver-1. Given an initial value orr and M + time steps (4: =0 decreasing from 40 = T to tM = 0. Starting with Ltg = æT, the sequence {ā,, Ji=1 is computed iteratively as follows:
Ct -t1-1 - (c, (1 where h; = A A
3t,- =
(3.7)
01111
For 1: 2 2, approximating the first terms of the Taylor expansion needs additional intermediate points between t and [i]] The derivation is more technical defer it to Appendix [ Below propose algorithms for 2, and name them as DPM-Solver: and DPM-Son 3PM--olver-3, respectively.
Algorithm DPM-Solver-2.
Require: initial value æt, time steps (4:)-00 model eg
1: Rioo f- 2T M do
2: for
AGG-1 T:,-11
3: 5; f ta
2
0s; (e# -1) 6o (0(
4: ui s 0411
5: Tt. f cie--1
Ois 1t-11
St. (e - 1) Eo Si(
6: end for
7: return ZtM
Algorithm 2 DPM-SSlver-3.
Require: initial value æy, time steps [t model co
1: Tio +- TT, <- 3. 12 +-
2: for
M do
3: $2:-1 + la (Ae-, + trihi),
825 + ta (Jea-, + +r2hi)
4: u2;-1 & a-2-- T1: 0es (erth - 1) c c0( ((f4-119--11) 19/1-1)
an
5:
D2-1 t- 60 c0(U12111 82;-1) En (,t (0:1;-1:4411 (111
6:
U2 f a.2i 2221 - Jezi (E - 1) E0( t(-1) A2gi (2thi
11,11
1)
D
an
rghi
7:
D2i + 6g (U2, 82,) 60 [ 14-1)
on B1 Dta-1 Tt. (e 1) EB ( t4-1) 24 ( -1) D2s
8: 3t an
9: end for
10: return TtM
Here, tx(-) is the inverse function of A(t), which has an analytical formulation for the practical noise schedule used in [2] [6], as shown in Appendix D The chosen intermediate points are (s;, u,) for DPM-Solver-2 and (#2--1, u22-1) and (825, U2ss for DPM-Solver33, As shown in the algorithm, DPM-Solver-k requires & function evaluations
step for k = 1, Despite the more expensive 2, 3) are usually more efficient since they require much fewer steps
steps, higher-order solvers to converge, due their higher convergence order. We show that DPM-Solver- is k-th-order solver, as stated in the following theorem. The proof in Appendix B
Theorem 3.2 (DPM-Solver-k
k-th-order solver). Assume ca(a,, .) follows the regularity condi- tions detailed in Appendix B. then for k = 1, 3, DPM-Solver-k is a k-th order solver for diffusion ODEs, i.e,, for the sequence } computed by Solverr the approximation error time satisfies Dtst
To (hh where hima maX (>e, >1,.1).
Finally, solvers with & 2 4 need much more intermediate points as shown by previous work 33 32 for exponential integrators. Therefore, we only consider from in this work, while leaving solvers with higher & for future study.

---

3.3 Step Size Schedule
The proposed solvers in Sec. 3.2 need to specify the time steps [= in advance. We propose two choices of the time step schedule. One choice is handcrafted, which is to uniformly split the interval o,..., M. Note that this different from previous work 3 which chooses uniform steps for 4, Empirically, DPM-Solver with uniform time steps dr; can already generate quite good samples in few steps, where results are listed in Appendix E As the other choice, we propose an adaptive step size algorithm, which dynamically adjusts the step size by combining different orders of DPM-Solver. The adaptive algorithm is inspired by [20] and we defer its implementation details to Appendix
[Ar, ) i.e. X, ÀT + M (Ao Ar)
For few-step sampling, we need to use up all the number of function evaluations (NFE). When the NFE is not divisible by 3. we firstly apply DPM-Solver- as much as possible, and then add single step of DPM-Solved I or DPM-Solver-2 (dependent on the reminder of K divided by 3), as detailed in Appendix D In the subsequent experiments, we use such combination of solvers with the uniform step size schedule for NFE S 20, and otherwise the adaptive step size schedule.
3.4 Sampling from Discrete-Time DPMs
discrete-time DPMs 2 train the noise prediction model at N fixed time steps {t, JN and the noise prediction model is parameterized by (, n) for n = .....,N - 1, where each o, is corresponding the value time In+1- We can transform the discrete-time noise prediction model to
the continuous version by letting eg (x, t) = Eo Exr, - for all æ E Rd E [0, T]. Note that the input time of én may not be integers, but we find that the noise prediction model can still work well,
and we hypothesize that because the smooth time embeddings (e.g., position embeddings (2). By such reparameterization, the noise prediction model can adopt the continuous-tin steps input, and thus we can also use DPM-Solver for fast sampling.
4 Comparison with Existing Fast Sampling Methods
Here, we discuss the relationship and highlight the difference between DPM-Solver Solver and existing ODE-based fast sampling methods for DPMs. We further briefly discuss the advantage of ttraining-free samplers over those training- baased ones.
4.1 DDIM as DPM-Solver-1
Denoising Diffusion Implicit Models (DDIM) [19 design deterministic method for fast sampling from DPMs. For two adjacent time steps f--1 and f;, assume that we have a solution it,-; at time 4-11, then single step of DDIM from time 44-1 time t,
Ct.
Ott Tt.-1
Oit;
Tt Cit
co (4
(4.1)
Although motivated by entirely differrent perspectives, we show that the updates of DPM-Solver- and Denoising Diffusion Implicit Models (DDIM) [19] are identical. By the definition of X, we have
0t--1 A:z= and Dt, e ~,,, Plugging these and h; d Xo. Eq. (4. results in exactly or.-1
step of DPM-Solver-1 in Eq. (3.7). However, the allows for principled generalization to higher-order solvers and convergence order analysis.
ODE formulation of DPM-Solver
Recent work 133 also show that DDIM is a first-order discretization of diffusion ODEs by differenti- ating both sides of Eq. (4.1). However, they cannot explain the difference between DDIM and the first-order Euler discretization of diffusion ODEs. In contrast, by showing that DDIM is special case of DoM-Solver, we reveal that DDIM makes full use of the semi-linearity of diffusion ODEs. which explains its superiority over traditional Euler methods.
4.2 Comparison with Traditional Kunge-Kutta Methods
One can obtain high-order solver by directly applying traditional explicit Runge- Kutta (RK) methods to the diffusion ODE in Eq. (2. Specifically. RK methods write the solution of Eq. (2.7 in the
7

---

Table FID L on CIFAR-I0 for different orders of Runge-Kutte (RK) methods DPM-Solvers, varying number of function evaluations (NFE). For RK methods, we evaluate diffusion ODEs w.r.t both (Eq. 2.7 and
À (Eq. (E. p. We use uniform step size in for RK (t), and uniform step size in A for RK (A) and DPM-Solvers
Sampling method
NFE
12
18
24
30
36
42
48
RK2 (f)
16,40
7.25
3.90
3.63
3.58
3.59
3.54
RK2 (A)
107.81
42.04
17.71
7.65
4.62
3.58
3.17
DPM-Solver-2
5.28
3.43
3.02
2.85
2.78
2.72
2.69
RK3 (f)
48.75
21.86
10.90
6.96
5.22
4.56
4.12
RK3 (A)
34.29
4.90
3,50
3.03
2.85
2.74
2.69
DPM-Solver-3
6.03
2.90
2.75
2.70
2.67
2.65
2.65
following integral form:
Øt = xz + J." he hal = x, x,+ S: (f(T)a, +
g2(T) 20, Ea (c,, dr,
(4.2)
and use some intermediate time steps between [t. s] and combine the evaluations of hg these time steps to approximate the whole integral. The approximation error of explicit RK methods depends on ho, which consists of the error correpponding both the linear term f(r)æ, a and the nonlinear noise prediction model Cg. However, the error of the linear term may increase exponentially because the exact solution of the linear term has an exponential coefficient (as shown in Eq. (3.1). There are many empirical evidence [25] (31] showing that directly using explicit RK methods for semi-linear ODEs may suffer from unstable numerical issues for large step size. We also demonstrate the empirical difference of the proposed DPM-Solve and the traditional explicit RK methods in Sec. Sec. which shows that DPM-Solver have smaller discretization errors than the RK methods with the same order.
4.3 Training-basee based Fast Sampling Methods for DPMs
Samplers that need extra training or optimization include knowledge distillation 13 14 learning
the noise level or variance [15 16 33 and learning the noise schedule or sample trajectory
Although the progressive distillation method [3] can obtain fast sampler within steps, needs further training costs and loses part of the information in the original DPM (e.g., after distillation, the noise prediction model cannot predict the noise (score function) at every time step between [0, T)). In contrast, vaining-free samplers can keep all the information of the original model, and thereby can be directly extended to the conditional sampling by combining the original model and an external classifier [4] (e.g. see Appendix Dor the conditional sampling with classifier guidance).
Beyond directly designing fast samplers for DPMs, several works also propose novel types of DPMs which supports faster sampling. For instance, defining low-dimensional latent variable for DPMs 34; designing special diffusion processes with bounded score functions [35]; combining GANs with the reverse process of DPMs (36). The proposed DPM-Solver may also be suitable for accelerating the sampling these DPMs, and we leave them for future work.
5 Experiments
In this section, we show that as training-free sampler, DPM-Solver can greatly speedup the sampling of existing pre-trained DPMs, including both continuous-tim and discrete-time ones, with both linear noise schedule 12 19 and cosine noise schedule [16). We vary different number of function evaluations (NFE) which is the number calls the noise prediction model e (w,, t), and compare the sample quality between DPM-Solver and other methods. For each experiment, We draw 50K samples and use the widely adopted FID score 37 to evaluate the sample quality, where lower FID usually implies better sample quality.
Unless explicitly mentioned, we always use the solver combination with the uniform step size schedule in Sec. 3.3 if the NFE budget is less than 20, and otherwise the DPM-Solver-3 with the adaptive step size schedule in Sec. 3.3 We refer Appendix D for other implementation details of DPM-Solver and Appendix for detailed settings.

---

ODE (R845)
DOPM
DOPM
SDE (Edc)
Arokyi-D0PM
SDE (Allytte))
DOM
DOIM
Arahyic LIIM
100
20.
1522
16:8
16:2
1000
(a) CIFAR-10 (continuous)
(b) CIFAR-10 (discrete)
(c) CelebA 64x64 (discrete)
Sclver
D0PM
DOPM
DDPM
Acelyte-mmP
717M
DDIM
05
12
1000 15
200
15 15 29
(l) LSUN bedroom 256x256 (dis- crete)
(d) ImageNet 64x64 (discrete)
(e) ImageNet 128x128 (discrete)
Figure 2: Sample quality measured by FID of different sampling methods for DPMs on CIFAR-I0 with both continuous-time and discrete-time models, CelebA 64x64, ImageNet 64x64, ImageNet 128x and LSUN bedroom 256x256 with discrete-time modcls, varying the number of function evaluations (NFE). The method GGGDM 18] needs extra training to optimize the sample trajectory, while other methods are training-free To get the strongest baseline, we use the quadratic step size for DDIM on CelebA, which has a better FID than that of the uniform step size in the original paper [1]]
5.1 Comparison with Continuous- Time Sampling Methods
We firstly compare DPM-Solver with other continuous-ting sampling methods for DPMs. The compared methods include the Buler-Maruyann
discretization for diffusion SDEs [3), the adaptive
step size solver for diffusion SDEs 200 and the RK methods for diffusion ODEs B (88 in Eq. (2.7)
We compare these methods for sampling from a pre-trained continuous-timg "VP deep" model on
the CIFAR-10 10 dataset [29] with the linear noise schedule.
Fig. 2a shows the efficiency of compared solvers. We use uniform time steps with 50, 200, 1000 NFE for the diffusion SDE with Euler discretization, and vary the tolerance hyperparameter 3 20 for the adaptive step size SDE solver 200 and RK45 ODE solver (28] control the NFE. DPM-Solver can generate good sample quality within around 10 NFE, while other solvers have large discretization error even in 50 NFE, which shows that PPM-Solver can achieve ~5 speedup of the previous best solver. In particular, we achieve 4.70 FID with 10 NFE, 3.75 FID with 12 NFE, 3.24 FID with 15 NFE, and 2.87 FID with 20 NFE, which is the fastest sampler on CIFAR-10.
As an ablation study, we also compare the second-order and third-order DPM-Solver and RK methods, as shown in Ta We compare RK methods for diffusion ODEs w.r.t. both time t in Eq. (2.7 and half-log-SNR > by applying change-of- variiabl (see detailed formulations in Appendix E. The results show that given the same NFE, the sample quality of DPM-Solver is consistently better than RK methods with the same order. The superior efficiency of DPM-Solver is particularly evident in the few-step regime under 15 NFE, where RK methods have rather large discretization errors. This is mainly because DPM-Solver analytically computes the linear term, avoiding the corresponding discretization error. Besides. the higher order DPM-Solver-3 converges faster than 3PM-Solver-2. which matches the order analysis in Theorem 3,2
5.2 Comparison with Discrete-Tinee Sampling Methods
We use the method in Sec. 3.4.3.4 for using DPM-Solver in discrete-time DPMs, and then compare DPM-Solver with other discrete-time training- frree samplers, including DDPM 22, DDIM 19)
Analytic-DDPM (21), Analytic-DDIM [21] PNDM [22], FastDPM (38) and Itô-Taylo 24 We
also compare with GGDM [18] which uses the same pre-trained model but needs further training for
the sampling trajectory. We compare the sample quality by varying NFE from 10 to 1000.

---

Specifically, we use the discrete- model trained by simple in 2 on the CIFAR- 10 dataset with
linear noise schedule; the discrete-time model in [9] on CelebA 64x64
39 with linear noise
schedule; the discrete-time model trained by Lnybrid in [16] on ImageNet 64x64 26 with cosine
noise schedule; the cissreetetime model with classifier guidance in [4] on ImageNet 128x 128 [26]
with linear noise schedule; the discrete-time model in A on LSUN bedroom 256x256 [40] with
linear noise schedule. For the models trained on ImageNet, we only use their "mean" model and omit the "variance" model. As shown in Fig. 2 all datasets, DPM-Solver can obtain reasonable samples within 12 steps (FID 4.65 on CIFAR-IC 10, FID 3.71 on CelebA 64x64 and FID 19.97 on ImageNet 64x64, FID 4.08 on ImageNet 128x 128), which is ~ 16x faster than the previous fastest training- free sampler. DPM-Solver even outperforms GGDM, which requires additional training.
6 Conclusions
We tackle the problem of fast and training-free sampling from DPMs. We propose DPM-Solver, a fast dedicated training-free solver of diffusion ODEs for fast sampling of DPMs in around 10 steps of function evaluations. DPM-Solver leverages the semi-linearies of diffusion ODEs and directly approximates simplified formulation of exact solutions of diffusion ODEs, which consists of exponentially weighted integral of the noise prediction model. Inspired by numerical methods for exponential integrators, we propose first-order, second-order and third-order DPM- Solver to approximate the exponentially weighted integral of noise prediction models with theoretical convergence guarantee. We propose both handcrafted and adaptive step size schedule, and apply DPM-Solver for both continuous time and discrete- time DPMs. Our experimental results show that DPM-Solver can generate high-que high-quality samples in around 10 function evaluations on various datasets, and it can achieve 4 ~ 16x speedup compared with previous state-of-the of-thh-rrt training- free samplers.
Limitations and broader impact Despite the promising speedup performance, DPM-Solver is designed for fast sampling, which may be not suitable for accelerating the likelihood evaluations of DPMs. Besides, compared to the commonly-used GANs, diffusion models with DPM-Solver are still not fast enough for real-time applications. In addition, like other deep generative models, DPMs may be used to generate adverse fake contents, and the proposed solver may further amplify the potential undesirable influence of deep generative models for malicious applications.
Acknowledgements
This work was supported by National Key Research and Development Project of China (No. 2021ZD01 10502); NSF of China Projects (Nos. 62061 136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222, 62106120); Beijing NSF Project (No. JQ19016); Beijing Outstand- ing Young Scientist Program NO. BIJWZYJH01 201 9 9100020098; grant from Tsinghua Institute for Guo Qiang; the NVIDIA NVAIL Program with GPU/DGX Acceleration; the High Performance Computing Center, Tsinghua University; the Fundamental Research Funds for the Central Universi- ties, and the Research Funds of Renmin University of China (22XNKJ13). J.Z. is also supported by the XPlorer Prize.
References
[1] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics," in International Conference on Machine Learning. PMLR, 2015, pp. 2256-2265
[2] J. Ho, A. Jain, and Abbeel, "Denoising diffusion probabilistic models," in Advances in Neural Information Processing Systems, vol. 33, 2020. pp. 6840-6851
[3] Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," in International Conference Learning Representations, 2021.
[4] Dhariwal and A. Q. Nichol, "Diffusion models beat GANs on image synthesis," in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 8780-8794.
10

---

[5] C. Meng, Y. Song. J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, "SDEdit: Image synthesis and editing with stochastic differential equations," in International Conference on Learning Representations. 2022.
[6] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. Fleet, "Video diffusion models," arXiv preprint arXivv2200.03458, 2022.
[7] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with CLIP latents," arXiv preprint TrXiv:2204.06122 2022.
[8] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan, "Wavegrad: Estimating gradients for waveform generation," in International Conference on Learning Representations, 2021.
[9] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, N. Dehak, and W. Chan, "Wavegrad 2: Iterative refinement for text-to-speech synthesis," in International Speech Communication Association, 2021, pp. 3765-3769.
[10] D. P. Kingma, T. Salimans, B. Poole, and J. Ho, "Variational diffusion models," in Advances in Neural Information Processing Systems, 2021.
[11] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. "Generative adversarial nets," in Advances Neural Information Processing Systems, vol. 27, 2014, pp. 2672-2680.
[12] D. P. Kingma and M. Welling. "Auto-encoding variational bayes," in International Conference on Learning Representations, 2014.
[13] T. Salimans and J. Ho, "Progressive distillation for fast sampling of diffusion models," in International Conference on Learning Representations, 2022.
[14] E. Luhman and T. Luhman, "Knowledge distillation in iterative generative models for improved sampling speed," arXiv preprint arXiv:2101.02338 2021.
[15] R. San-Roman, E. Nachmani, and L. Wolf, "Noise estimation for generative diffusion models," arXiv preprint 104.020000000000, 2021.
[16] A. Q. Nichol and P. Dhariwal, "Improved denoising diffusion probabilistic models," in Interna- tional Conference on Machine Learning. PMLR, 2021, pp. 8162-8171.
[17] M. Lam, J. Wang, R. Huang. D. Su, and D. Yu, "Bilateral denoising diffusion models," arXiv preprint arXiv:2 11514, 2021.
[18) D. Watson, W. Chan, J. Ho, and M. Norouzi, "Learning fast samplers for diffusion models by dif- ferentiating through sample quality," in International Conference on Learning Representations, 2022.
[19] J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," in International Conference on Learning Representations, 2021.
[20] A. Selicoeur-Martinee K. Li, R. when generating data with score-based models." arXiv preprint arXiv:2105.14080 2021.
T. Kachman, and I. Mitliagkas, "Gotta go fast
[21] F. Bao, C. Li, J. Zhu, and B. Zhang, "Analytic-DPM: An analytic estimate of the optimal reverse variance in diffusion probabilistic models," in International Conference on Learning Representations, 2022.
[22] L. Liu, Y. Ren, Z. Lin, and Z. Zhao, "Pseudo numerical methods for diffusion models manifolds," in International Conference on Learning Representations, 2022.
[23] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, M. Kudinov, and J. Wei, "Diffusion-based voice conversion with fast maximum likelihood sampling scheme," in International Conference on Learning Representations, 2022.
11

---

[24] H. Tachibana, M. Go, M. Inahara, Y. Katayama, and Y. Watanabe, "Itó-Taylor sampling scheme for denoising diffusion probabilistic models using ideal derivatives, arXiv preprint arXiv:2112. 13333, 2021.
[25] M. Hochbruck and A. Ostermann, "Exponential integrators," Acta Numerica, vol. 19, pp. 209-286, 2010.
[26] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and I.. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009, pp. 248-255.
[27] E. Kloeden and E. Platen, Numerical Solution of Stochastic Differential Equations. Springer, 1992.
[28] J. R. Dormand and P. J. Prince, "A family of embedded Runge- Kutta formulae," " Journal of computational and applied mathematics, vol. 6, no. 1, pp. 19 26, 1980.
[29] A. Krizhevsky, "Learning multiple layers of features from tiny images," Tech. Rep., 2009.
[30] K. Atkinson, W. Han, and D. E. Stewart, Numerical solution of ordinary differential equations. John Wiley & Sons, 2011, vol. 108.
[31] M. Hochbruck and A. Ostermann, "Explicit exponential Runge-Kutta methods for semilinear parabolic problems," SIAM Journal Numerical Analysis, vol. 43, no. 3, pp. 1069-1090, 2005.
[32] V. T. Luan, "Efficient exponential Runge-Kutta methods of high order: Construction and implementation," " BIT Numerical Mathematics, vol. 61, no. 2, pp. 335-560, 2021.
[33] F. Bao, C. Li, J. Sun, J. Zhu, and B. Zhang, "Estimating the optimal covariance with imperfect mean diffusion probabilistic models," arXiv preprint urXiv:2206.07300 2022.
[34] A. Vahdat, K. Kreis, and J. Kautz, SScore-based generative modeling in latent space." in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 11 287-11 302.
[35] Dockhorn, A. Vahdat, and Kreis, "Score-based generative modeling with critically damped Langevin diffusion," in International Conference on Learning Representations, 2022.
[36] Z. Xiao, Kreis, and A. Vahdat, "Tackling the generative learning trilemma with denoising diffusion GANs," in International Conference on Learning Representations, 2022.
[37] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "GANs trained by a two time-scale update rule converge to local Nash equilibrium," in Advances in Neural Information Processing Systems, I. Guyon, U. von Luxburg. S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., vol. 30, 2017, pp. 6626-6637.
[38] Z. Kong and W. Ping, "On fast sampling of diffusion probabilistic models," arXiv preprint XrXiv:2106.00 00132, 2021.
[39] Z. Liu, P. Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3730-3738.
[40] Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and Xiao, "LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop," arXiv preprint arXiv:1506.03: 03365, 2015.
[41] Y. Song, C. Durkan, I. Murray, and S. Ermon, "Maximum likelihood training of score-based diffusion models," Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 1415-1428.
[42] K. Yang, J. Yau, L. Fei-Fei, J. Deng, and O. Russakovsky, "A study of face obfuscation in ImageNet," arXiv preprint arXiv:2 103.0/999 2021.
12

---

Checklist
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] We observe that the standard deviation of the FID evaluations of DPM-Solver are rather small (mainly less than 0.01) because the FID is The small 1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Ap- pendix B
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Code is attached in the supplemental materials, with the appendix.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Our method is training-free. But we also report the hyperparameters for evaluations used in our proposed solver.
already averaged over 50K samples, following existing work [8 20 21].
standard deviation does not change the conclusion.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] The GPU type and amount is detailed in Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/reeeusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We include code in the supplemental materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] All of the datasets used in the experiments are publicly available. (e) Did you discuss whether the data you using/curating contains personally identifiable
information or offensive content? [Yes] We mentioned the human privacy issues the
ImageNet dataset in Appendix H
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "s4QZTpc2meb", "reviewer": "Reviewer_Wm2X", "summary": "This paper proposes a training-free few-step sampler of diffusion probabilistic models (DPMs) by leveraging semi-linear structure in the diffusion ODEs to approximate the solutions of the equation. The proposed method can accelerate the sampling of DPMs without sacrificing the sampling quality.\n", "questions": "Would you please include the comparison of runtime for sampling between the proposed method and other methods? The reviewer is concerned that the computational time would be a critical weak point in the proposed method.\n", "limitations": "The authors mentioned the limations of their work properly.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- (Quality) The authors effectively provide a theoretical derivation and error analysis of the proposed method and the experiments' details.\n- (Clarity) The paper is well-written and clear to understand. The reviewer could understand the contents without difficulty.\n- (Significance) Numerical experiments verify that DPM-solver significantly outperforms previous sampling methods in terms of quality with fewer function evaluations. The reviewer considers this paper deserves to be accepted by NeurIPS if the authors include the runtime comparison between the proposed and other methods.\n\nWeakness\n\n- (Originality) The proposed method shows excellent performance; however, its theoretical background is based on the variation of constants which is a classical idea in the literature of numerical differential equations, so one can think the originality is somewhat marginal.\n\n***Updated Review***\n\nThe reviewer appreciates the authors' sincere answers. I have updated my score and hope the paper is accepted to the NeurIPS conference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6GFI5mf1VxC", "reviewer": "Reviewer_EaE9", "summary": "This paper tackles the slow-sampling problem in denoising diffusion generative models. Diffusion models have demonstrated outstanding synthesis quality, together with good sample diversity and mode coverage, but they are slow during sampling, since many neural networks calls are required to iteratively solve a generative differential equation and generate a single batch of data. To this end, the paper proposes a novel ODE solver, called *DPM-Solver*, that can very quickly solve the generative ODE of diffusion models. The generative ODE, also known as probability flow ODE, is of semi-linear form, consisting of both a linear and a non-linear (neural network) term. The paper proposes an ODE solver that solves the linear component analytically and exactly, while only requiring a numerical approximation for the non-linear term. The non-linear term, formally a time integral over the learnt score function with an exponential weighting, can be solved with different orders of accuracy (implying different accuracy-compute trade-offs). Overall, the framework can be interpreted as an exponential integrator approach tailored to diffusion models, with a Runge-Kutta-like technique applied to only the non-linear term. Experimentally, the paper demonstrates significantly accelerated diffusion model sampling for 3 low-resolution image generation tasks (CIFAR-10, CelebA 64x64, ImageNet 64x64), comparing to various well-established baselines.", "questions": "I also have some minor additional questions and suggestions:\n\n- In the concurrent work [8], a very similar scheme, *DEIS*, is proposed. I think it would be interesting to compare and discuss the detailed differences between the method proposed here, DPM-sampler, and DEIS and also run detailed quantitative comparisons. My understanding is that DPM-Solver primarily leverages Runge-Kutta-like techniques to solve the non-linear term, while DEIS also explored muiltistepping methods that rely on past function evaluations. Furthermore, it seems the papers use different time-reparametrizations. I'd be curious how the $\\rho$-parametrization in DEIS (Eq. (18) in their paper) and the $\\lambda_t$-parametrization here are related. To this end, I am also wondering whether maybe the ideal future solver is based on a combination of ideas from both of these papers? As mentioned, since this is concurrent work, I don't consider this as a criterion for recommending acceptance here. But it would be nice and insightful for the community. I'd be curious about the authors' thoughts on this.\n\n[8] Zhang and Chen, \"Fast Sampling of Diffusion Models with Exponential Integrator\", arXiv 2022", "limitations": "Limitations have been discussed only very briefly (the proposed ODE solver may not be suitable for also accelerating likelihood calcuation). It seems fine to me, although this could be extended a bit, ideally. In particular, the main limitation is that the sampler is still not fast enough to generate model samples so quickly that diffusion models can be used in real-time, interactive applications.\n\nPotential negative societal impacts have been briefly mentioned, in a satisfactory manner.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Stengths:**\n- The paper proposes a rigorously derived and technically solid ODE integrator for fast sampling from diffusion models. The proposed method, *DPM-Solver*, is well-motivated based on techniques from the ODE literature (exponential integrators).\n- The experimental results on low-resolution image synthesis successfully validate the method and show a clear advantage over the baselines.\n- Overall, the paper is easy to follow and clearly explained (the English could be better, but that's okay; maybe some proofreading from a native speaker might help). The paper nicely discusses its differences compared to the most commonly used ODE solvers in diffusion models, this is, Runge-Kutta methods and DDIM [1].\n- I think the contribution is sufficiently significant. The tackled problem, accelerating sampling from diffusion models, is very specific. However, this is a very relevant problem, considering that slow sampling currently is diffusion models' main weakness.\n\n**Weaknesses:**\n\nI do not think that the paper has major flaws. However, there are some points of criticism:\n- The paper misses relevant baselines in its experimental evaluations: In particular, it would be very appropriate to quantitatively compare to \"Pseudo Numerical Methods for Diffusion Models on Manifolds\" [2], which is a published method that also accelerates diffusion model sampling with higher-order ODE methods. Furthermore, \"FastDDIM\" [3] would another appropriate baseline. Potentially, also Tachibana et al. [4] could be included. I actually think the proposed solver will outperform all these baselines. Nevertheless, I think a more rigorous evaluation with all possible baselines would strengthen the paper significantly, considering that accelerated sampling is the paper's main and only contribution.\n- Beyond fast ODE samplers, the literature on diffusion models includes also many other ideas for accelerated sampling. For instance, sampling can be accelerated by working in a smooth latent space [5], by using GAN-based large-step denoising [6], of by devising tailored diffusion processes that can be denoised efficiently [7]. These works are not even cited. So, considering that this paper tackles accelerated diffusion model sampling, I think a more thorough discussion on the relevant literature is missing.\n- Experimentally, the paper only uses low-resolution image datasets. The paper could be improved by also applying the method on other problems, such as high-resolution image synthesis or audio synthesis, where diffusion models have also been widely applied.\n\n**Conclusion:**\nOverall, I think this is a solid paper that deserves to be accepted. It successfully tackles a specific but relevant problem in the diffusion model literature in a convincing and technically solid way. It might be worth noting that there is highly related concurrent work that proposed a very similar method [8]. However, I don't think that this should have any negative impact here. I think exponential integration techniques are very natural to use in diffusion models, and having multiple works concurrently analyzing and demonstrating that independently in different ways is good. All that said, it also needs to be mentioned that the paper could still be significantly improved in various ways. In particular, more thorough experiments could be run and additional baselines could be considered (see weaknesses and questions below).\n\n[1] Song et al., \"Denoising Diffusion Implicit Models\", ICLR 2021\n\n[2] Liu et al., \"Pseudo Numerical Methods for Diffusion Models on Manifolds\", ICLR 2022\n\n[3] Kong and Ping, \"On Fast Sampling of Diffusion Probabilistic Models\", arXiv 2021\n\n[4] Tachibana et al., \"Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives\", arXiv 2021\n\n[5] Vahdat et al., \"Score-based Generative Modeling in Latent Space\", NeurIPS 2021\n\n[6] Xiao et al., \"Tackling the Generative Learning Trilemma with Denoising Diffusion GANs\", ICLR 2022\n\n[7] Dockhorn et al., \"Score-Based Generative Modeling with Critically-Damped Langevin Diffusion\", ICLR 2022\n\n[8] Zhang and Chen, \"Fast Sampling of Diffusion Models with Exponential Integrator\", arXiv 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4k6Sc_zXRms", "reviewer": "Reviewer_DD9K", "summary": "It has been showed recently that diffusion probabilistic models can be understood as ODE, which is useful to devise efficient prediction methods. This made it possible to strongly accelerate inference by several orders of magnitude compared to original SDE-based methods.\n\nIn this line of research, this paper shows that even more progress can be made by leveraging a decomposition of the solution as a sum of a linear and a non-linear part. The method can be used on off-the-shelf and already trained DPM, and doesn't require fine-tuning. ", "questions": "* it would help me if you could provide some expression for \\lambda(t). I guess this is really important in practice, and this is something that is missing. What does it mean for t when \\lambda is uniformly picked ?\n* I understand that your contribution is mostly relative to having faster inference, but do you think the insights you bring could actually be useful for training ?", "limitations": "nothing special", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is both a challenging but an exciting read. It is challenging because it is pretty rigorous and the math often goes beyond my comfort zone. Still, it is exciting because it is self contained and really brings an interesting idea on the table, to devise a specific solver that allows effective inference with DPM.\n\nRegarding the weaknesses, maybe I should say that the code provided in the supplementary is not really advertised in the paper and I think it could really benefit from some work. As such it is not documented at all so that it is really difficult to get into it. I think it would be worth it to spend some effort to make that better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept