DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
Authors: Cheng Lu
Yuhao Zhou
Fan Bao
Jianfei Chen
Chongxuan Li
Jun Zhu
Venue: Submitted to ICLR 2023 (2023)
Keywords: diffusion probabilistic models
score-based generative models
fast sampling
guided sampling

Abstract: Diffusion probabilistic models (DPMs) have achieved impressive success in high-resolution image synthesis, especially in recent large-scale text-to-image generation applications. An essential technique for improving the sample quality of DPMs is guided sampling, which usually needs a large guidance scale to obtain the best sample quality. The commonly-used fast sampler for guided sampling is DDIM, a first-order diffusion ODE solver that generally needs 100 to 250 steps for high-quality samples. Although recent works propose dedicated high-order solvers and achieve a further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before. In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than  DDIM when the guidance scale grows large. To further speed up guided sampling, we propose DPM-Solver++, a high-order solver for the guided sampling of DPMs. DPM-Solver++ solves the diffusion ODE with the data prediction model and adopts thresholding methods to keep the solution matches training data distribution. We further propose a multistep variant of DPM-Solver++ to address the instability issue by reducing the effective step size. Experiments show that DPM-Solver++ can generate high-quality samples within only 15 to 20 steps for guided sampling by pixel-space and latent-space DPMs.


Under review as a conference paper at ICLR 2023
FAST SOLVER FOR GUIDED SAM- PLING OF DIFFUSION PROBABILISTIC MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Diffusion probabilistic models (DPMs) have achieved impressive success in high- resolution image synthesis, especially in recent large-scale text-to-image generation applications. An essential technique for improving the sample quality of DPMs is guided sampling, which usually needs a large guidance scale to obtain the best sample quality. The commonly-used fast sampler for guided sampling is DDIM, a first-order diffusion ODE solver that generally needs 100 to 250 steps for high- quality samples. Although recent works propose dedicated high-order solvers and achieve further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before. In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than DDIM when the guidance scale grows large. To further speed up guided sampling, we propose DPM-Solver+++ high-order solver for the guided sampling of DPMs. DPM--olver++ solves the diffusion ODE with the data prediction model and adopts thresholding methods to keep the solution matches training data distribution. We further propose a multistep variant of DPM-Solver++ address the instability issue by reducing the effective step size. Experiments show that DPM-Soller++ can generate high-quality samples within only 15 to 20 steps for guided sampling by pixel-space and latent-space DPMs.
INTRODUCTION
Diffusion probabilistic models (DPMs) (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al.
202Tb) have achieved impressive success on various tasks, such as high-resolution image synthe-
sis
(Dhariwal & Nichol, 2021
2021] Ho et al. 2022 Rombach et al. 2022). image editing (Meng et al.)
2022
Saharia et | 2022al] Zhao et al. 2022), text-to-image generation (Nichol et al. [022 Saharia
cet al.
|022b]|Rames Ramesh et al. 2022 Rombach et al. 2022; Gu et al.
(022), voice synthesis (Liu et al.
2022a
Chen et al. 2021a b), molecule generation (Xu et al, 2022 Hoogeboom et al., 2022 Wu et al.
2022)
and data compression (Theis et al., 2022 Kingma et al. (021). Compared with other deep
generative models such as GANs (Goodfellow et al. 2014) and VAES (Kingma & Welling 2014).
DPMs even achieve better sample quality by leveraging an essential technique called guided sam- pling (Dhariwal& & Nichol 2021] Ho & Salimans| 2021), which uses additional guidance models to improve the sample fidelity and the pandition-sampl alignment. Through it, DPMs in text-to-image and image-to-image tasks can generate high-resolution photorealistic and artistic images which are highly correlated the given condition, bringing a new trend artificial intelligence art painting.
The sampling procedure of DPMs gradually removes the noise from pure Gaussian random variables to obtain clear data, which can be viewed as discretizing either the diffusion SDEs (Ho al., 2020). Song et al.] 2021b) or the diffusion ODEs (Song et al. 2021b a) defined by parameterized noise prediction model or data prediction model (Ho et al. 2020 Kingma et al.) (021). )). Guided sampling unconditional model with a guidance model, where hyperparameter controls the scale of the guidance model (i.e. guidance scale). The commonly-use proven as a first-order diffusion ODE solver (Salimans & Ho 2022 Lu et al. 20022 and it generally needs 100 250 steps of large neural network evaluations to converge, which is time-comme timecconuumigg
of DPMs can also be formalized with such discretizations by combining
method for guided sampling is DDIM (Song al. 202 which is
Dedicated high-order diffusion ODE solvers (Lu et al. 2022 Zhang & Chen 2022) can generate high-quality samples in 10 to 20 steps for sampling without guidance. However, their effectiveness

---

Under review as a conference paper at ICLR 2023
DDIM (order= = 1) (ong et al. 2021a)
PNDM (order = 2) Liu al 2022b
DEIS-1 1 (order = 2) Zhang & Chen 2022
DEIS 2 (order 3) (hang & Chen 2022
DPM-Solver (order = 2)
DPM-Solver-3 (order = 3)
DDDMM (thresholding) Saharia al. (202hb)
DPM-Solvar++ (order = 2) (ours)
(Ln al. 2022)
[u et al. 2022)
Previous bigh-order solvers are unstable for guided sampling: Samples using the pre-trained DPMs (Dhariwal & Nichol 202T) on ImageNet 256 with a classifter guidance scale 8.0, varying different samplers (and different solver orders) with only 15 function evaluations. t: DDIM with the dynamic threshold- ing (Saharia al. 20226. Our proposed DPM-Solver++ (detailed in Algorithm 2 generate better samples than the first-order DDIM, while other high-order samplers are worse than DDIM.
Figure 1:
for guided sampling has not been carefully examined before. In this work, we demonstrate that previous high-order solvers for DPMs generate unsatisfactory samples for guided sampling, even worse than the simple first-order solver DDIM. We identify two challenges of applying high-order solvers to guided sampling: (I) the large guidance scale narrows the convergence radius of high-order solvers, making them unstable; and (2) the converged solution does not fall into the same range with the original data (a.k.a. "train-test mismatch" (Saharia et al., 2022bb)
Based on the observations, we propose DPM-Solver++, a training-free fast diffusion ODE solver for guided sampling. We find that the parameterization of the DPM critically impacts the solution quality. Subsequently, we solve the diffusion ODE defined by the data prediction model, which predicts the clean data given the noisy ones. We derive high-order solver for solving the ODE with the data prediction parameterization, and adopt dynamic thresholding methods (Saharia et al. (202bb) mitigate the train-test mismatch problem. Furthermore, we develop multistep solver which uses smaller step sizes to address instability.
As shown Fig. I DPM-Solver++ can generate high-quality samples in only 15 steps, which much faster than all the previous training- free samplers for guided sampling. Our additional experimental results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, for a wide variety of guided sampling applications, including both pixel-space DPMs and latent-space DPMs.
2 DIFFUSION PROBABILISTIC MODELS
In this section, we review diffusion probabilistic models (DPMs) and their sampling methods.
2.1 FAST SAMPLING FOR DPMS BY DIFFUSION ODES
Diffusion Probabilistic Models (DPMs) -ickssDicsttein et al. 2015 Ho et al., 2020 Song et al.
2021b) gradually add Gaussian noise D-dimensional random variable zu E R perturb the corresponding unknown data distribution go at time 0 to a simple normal distribution (#())) x N(rrr (0, 22 I) at time T > 0 for some ē 0. The transition distribution gıo (2%, (20) at
2

---

Under review as a conference paper at ICLR 2023
each time & [0 satisfies
(00(2;(20))
(2 o,
where Ct,, oi 0 and the signal-to-noise-ratio (SNR) of / /ot is strictly decreasing w.r.t. (Kingma et al. 2021 Eq. can be written as = a,20 + o,e, where e ~ I).
Parameterization: noise prediction and data prediction DPMs learn to recover the data Do based on the noisy input æт with a sequential denoising procedure. There define the model. The noise prediction model eg (zx, 1) attempts to predict the noise from the data Ty. which optimizes the parameter 0 by the following objective (Ho et al. 2020 Song et al. 2021b):
alternative ways to
min Emp. [w=()]len<xz: (x,, t)
<l
where æo ~ go(@ro), 6 ~ (O, I). ~ (O 1)). and ((t > 0 is a weighting function. Alternatvelyy, the data prediction model æa (2 t) predicts the original data 25 based on the noisy Zt. and its relationship with c0( t) is given by x t) = (x+ oz (Iz, f))/ar (Kingma et al. 2021
Diffusion ODEs
Sampling by DPMs can be implemented by solving the diffusion ODEs (Song al. 2021b a Liu al. 2022b Zhang & Chen 2022; Lu et al. (2022), which is generally faster than other sampling methods. Specifically, sampling by diffusion ODEs need to discretize the following
ODE (Song al. 20211) with changing from 'I' to 0:
dæ, dt
f(t)x + g2(1)
20, co (z),,,), ET ~ N2),
(3)
and the equivalent diffusion ODE w.r.t. the data prediction model ag is
dæ
(2( (1())
a1g²(t)
20T 2t.
wo (ate (), ØT *1~N(0,0"1),
(4)
dt
20
where the coefficients f(t)
log o
g2) =
do?
dog ao a de
of Kingma et al. 2021
de
dt
2.2 GUIDED SAMPLING FOR DPMS
2021 is a widely-used technique to apply DPMs for conditional sampling, which useful in ix-ate--mage, image-to-image. and class-to-image applications (Dhariwal & Nichol 2021 Saharia et al. 2022b Rombach et al. 2022 Nichol al. |021 Ramesh et al. 2022). Given condition variable c, guided sampling defines conditional noise prediction model 60(2,, 4, c). There are two types of guided sampling methods, depending whether they require classifier model. Classifier guidance (Dhariwal & Nichol 2021 leverages a pretrained classifier Po(c/z;; t) to define the conditional noise prediction model by:
Guided sampling DDhariwal & Nichol 2021 Ho & Salimans
éo(ar, 4, c) == ég (0 1)
V me log pa (c)
(5)
where & > 0 is the guidance scale. In practice, a large 8 is usually preferred for improving the condition-sample alignment (Rombach al. 2022 [Saharia et al., 2022b) for guided sampling. Classifier-fie frefferrr-reeeee guidance (Ho & Salimans 2021) shares the same parameterized model c ca f, c) for the unconditional and conditional noise prediction models, where the input for the unconditional model is a special placeholder Ø. The corresponding conditional model is defined by:
Eo Go c) == s (0 1, c) + +) 60 (D,, 1, 2).
Then, samples can be drawn by solving the ODE 3 with ca(2,, t, c) in place of c0( t). DDIM (Song al.) (2021a) is typical solver for guided sampling, which generates samples in a few hundreds of steps.
2.3 EXPONENTIAL INTEGRATORS AND HIGH-ORDER ODE SOLVERS
is shown in recent works (u et al. 2022 Zhang & Chen 2022) that ODE solvers based on exponential integrators (Hochbruck & Ostermann) 2010) converge much faster than the traditional
3

---

Under review as a conference paper at ICLR 2023
solvers for solving the unconditional diffusion ODE ct al. (2022) derive the solution 2y of the diffusion ODE ( at time as:
Given initial value œ, time 0.
Lu
Ea A)dA, Ckt xx ai /
3t
as
where the ODE is changed from the time (t) domain to the log- log-SNR (A) domain by the change-of- variables formula. Here, the log- d = log(ax/ (o)/))) is strictly decreasing function of t with
the inverse function (x),), and a = Fy (A)- Ed A) := eo E0 (2)= (A)) are the corresponding change-of-variabh forms for X. Lu et al. (2022) showed that DDIM is a first-order solver for Eq. 7 They further proposed high-order solver named "DPM-Solver", which can generate realistic samples for the unconditional model in only 10-20 steps.
Unfortunately, the outstanding efficiency of existing high-order solvers does transfer guided sampling, which we shall discuss soon.
3 CHALLENGES OF HIGH-ORDER SOLVERS FOR GUIDED SAMPLING
Before developing new fast solvers, we first examine the performance of existing high-order diffusion ODE solvers and highlight the challenges.
The first challenge is the large guidance scale causes high-order solvers to be instable. As shown in Fig. [ for large guidance scale s 8.0 and 15 function evaluations, previous high-order diffusion ODE solvers (Lu al.) 2022 Zhang& & Chen 2022 [iu et al., 2022b) produce low-quality images. Their sample quality is even worse than the first-order DDIM. Moreover, the sample quality becomes worse as the order of the solver gets higher.
Intuitively, large guidance scales may amplify both the output and the derivatives of the model Eg in Eq. (5) The derivatives of the model affect the convergence range of ODE solvers, and the amplification may cause high-order ODE solvers to need much smaller step sizes to converge, and thus the higher-order solvers may perform worse than the first-order solver. Moreover, high-order solvers require high-order derivatives, which are generally more sensitive to the amplifications. This further narrows the convergence radius.
The second challenge is the "train-tes! mismatch" problem Saharia et al. 2022b). The data lie in bounded interval (e-g- [-1, 1] for image data). However, the large guidance scale pushes the conditional noise prediction model ég ( f, c) away from the true noise, which in turns make the sample (i.e. the converged solution æo of diffusion ODEs) to fall out of the bound. In this case, the generated images are saturated and unnatural (Saharia et al. 2022b).
4 DESIGNING TRAINING-FRED FAST SAMPLERS FOR GUIDED SAMPLING
In this section, we design novel high-order diffusion ODE solvers for faster guided sampling. As discussed in Sec. 3 previous high-order solvers have instability and "train-test mismatch" issues for large guidance scales. The "train-ted mismatch" issue arises from the ODE itself, and find the parameterization of the ODE critical for the converged solution to be bounded. While previous high-order solvers designed for the noise prediction model Eg. solve the ODE 44) for the data prediction model æp. which itself has some advantages and thresholding methods are further available keep the samples bounded (Ho et al., 2020 Saharia et al. 2022b). We also propose multistep solver to address the instability issue.
4.1 DESIGNING SOLVERS BY DATA PREDICTION MODEL
We follow the notations in Lu et al. 2022 Given a sequence MM decreasing from to = T to tM 0 and initial value Tta [i I), the solver aims iteratively compute sequence ( 1M to approximate the exact solution at each time f,, and the final value Et,, is the approximated sample by the diffusion ODE. Denote h; == A At_. for 1..... M.
For solving the diffusion ODE w.r.t. Dg in Eq. (4) we firstly propose a simplified formulation of the exact solution of diffusion ODEs w.r.t. D6 below. Such formulation exactly computes
4

---

Under review as a conference paper at ICLR 2023
the linear term in Eq. (4 and only remains sroonentially-weighhhe integral of zœ- Denote
( : A) = 2z (x)) (x)) as the ohange-of-variabl form of zg for X, we have:
Proposition 4.1 (Exact solution diffusion ODEs 2g. proof in Appendix A)) Given an initial value Ig at time >> 0, the solution Et time E [0, of diffusion ODEs in Eq. is:
at Pt v 01 /
)
2 AjdA.
(8)
As the diffusion ODEs in Eq. 3 and Eq. 4 are equivalent, the exact solution formulations in Eq.
and Eq. (8) are also equivalent. However, from the prespective of designing ODE solvers, these two formulations are different. Firstly, Eq.
7 exactly computes the linear term o, E., while Eq. (8) exactly computes another the linear term Z. Moreover, design ODE solvers, Eq. 7 needs to d, approximate the integral f cgdA, while Eq. (8) needs to approximate f T'ddd,, and these two integrals different (recall that Zy (x, o,Ca)/a,). Therefore, the high-order solvers based on Eq. ( and Eq. (8) are essentially different. We further propose the general manner for design high-order ODE solvers based on Eq. (8) below.
Given the previous value -t;-1 at time 1,-1- the aim of our solver is to approximate the exact solution
time 44- Denote 2g (A) == (a).) the n-th order total derivatives of my w.r.t. logSNR
For & > 1, taking the (k: - 1)-th Taylor expansion at Xc,_ for Dy w.r.t À E [At,-1 Ar, and substitute it into Eq. (8) with s = 1 and = ti, we have
Ft. 04-1
Il.1 + ot,
Σ n=0
k-1
(m) 2g
(e dewn) S e (A dr. - 1)
dd OA+
(9)
n!!
estimated
analytically computed (Appendix
ommited
where the integral f e (A- " dA can be analytically computed by ontegral-by-penn (detailed in Appendix A.. Therefore, to design a k-th order ODE solver, we only need to estimate the n-th order
nt
derivatives zg (Ae, ) for n I k - after omitting the Ohh k+1 high-order error terms, which are
well-studied techniques and discussed in details in Sec. 4.2 A special case is k = 1, where the
solver is the same as DDIM (Song et al. 2021a), and we discuss in Sec. 5.1
For k == 2, we use similar technique DPM-Solver-2 (uu et al. 2022 to estimate the derivative a (#x, At;-1). Specifically, we introduce an additional intermediate time step 8; between 4-11 and 4; and combine the function values at 8; and 4-1 1 to approximate the derivative, which the standard manner for singlestep ODE solvers (Atkinson al. 2011). Overall, need 2M + time steps ((t,) M and }= which satisfies ty > $1 f1 > ... f---1 > $M > f.- The detailed algorithm is proposed in Algorithm T where combine the previous value Dt, at time 44-1 with the intermediate value u, at time 8; to compute the value 3, at time lç
Algorithm 2 SoveeSolver))(MM),
Require: initial value myy, time steps (4) data prediction model at..
Algorithm DPM--Sller+++25).
Require: initial value æ5,, time steps (t:)Mo (M M prediction model
Denote h;; Au, Ae... for = 1,..,, M. Initialize an empty buffer Q.
2: Rt,
1: Re, <- .ry.
3: Q buttes zo(ā... fo).
do 3: h Ad, Ari-1
Rú viu
"n.50 au (c-As to)
5: Q butter =o(ūe,,()
4:
+- In
2
5:
74-1 -i... an (e nihe xalds ki-
6:
D. (1 In (2 in 1--1 + (1).))
D. (1+ + soli i) to 2)
7: Jr, miat Heat an (e-
9:
ã., Bh... (0 D.
8: end for
9:
10: < M. then Q butter 20(....11
end for
12: return d...
We name the algorithm as (PM--olver+++2S)) which means that the proposed solver is a second-ordes singlestep method. We present the theoretical guarantee of the convergence order in Appendix A For
5

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison between high-order diffusion ODE solvers based on exponential integrators, including
DEIS (Zhang & Chen 2022), DPM-Solver (Lu et al 2022 and DPM- 3PM-Solver++ (ours).
DEIS
DPM-Solver
3PM-Solver++
Zhang & Chen 2022
(Lu et al. 2022]
(ours)
First-Order
DDIM
DDIM
DDIM
Model Type
6g
6/
Taylor Expansion
En for t
Eg for À
Tp for À
Solver Type (High-Order)
Multistep
Singlestep
Singlestep Multistep
2 as discussed in Sec. 3 high-order solvers may be unsuitable for large guidance scales, thus we mainly consider A: = 2 in this work, and leave the solvers for higher orders for future study.
Moreover, we provide theoretical comparison between DPM-Solver-2 (Lu et al. 2022 and DPM- (olver++(2S) in Appendix B We find that DPM-Soller+++<25 has a smaller constant before the high-order error terms, thus generally has smaller discretization error than 2.M--oller-2.
4.2 FROM SINGLESTEP TO MULTISTEP
At each step (from 4:-1 to 4;), the proposed singlestep solver needs two sequential function evaluations of the neural network xg. Moreover, the intermediate values u, are only used once and then discarded. Such method loses the previous information and may be inefficient. In this section, we propose another second-order diffusion ODE solver which uses the previous information at each step.
In general, to approximate the derivatives (n) in Eq. (9) for 2 1, there is another mainstream Zg approach (Atkinson al. 201Ty: multistep methods (such as Adams-Bashfern the previous values { 1-1 at time f;-1, multistep methods just reuse the previous values to approximate the high-order derivatives. Multistep methods are empirically more efficient than singlestep methods, especially for limited number of function evaluations. (Atkinson et al., 2011)
methods). Given
We combine the techniques for designing multistep solvers with the Taylor expansions in Eq. 9 and further propose multistep second-order solver for diffusion ODEs with my The detailed algorithm is proposed in Algorithm 2] where combine the previous values ă, and Tt,_; to compute the value Tt, without additional intermediate values. We name the algorithm as (2M)))) which means that the proposed solver is second-order multistep solver. We also present detailed theoretical guarantee the convergence order, which is stated in Appendix [A]
For fixed budget N of the total number of function evaluations, multistep methods use M = N steps, while k-th order singlestep methods can only use no more than M = N/kk steps. Therefore, each step size h; of multistep methods is around 1/k of that of singlestep methods, so the high-order error terms (1()) in Eq. (9) of multistep methods may also be smaller than those of singlestep methods. We show in Sec. 6.1 that the multistep methods are slightly better than singlestep methods.
4.3 COMBINING THRESHOLDING WITH +PM-SOLVER++
For distributions of bounded data (such as the image data), thresholding methods (Ho et al., 2020 Saharia al., 2022h) can push out-of-bound samples inwards and somehow reduce the adverse impact of the large guidance scale. Specifically, thresholding methods define clipped data prediction model d f, c) by elementwise clipping the original model Vy == (æ; - 0r:6)//r /a within the data bound, which results in better sample quality for large guidance scales (Saharia et al.) 2022b). As our proposed DPM--Sllve++ is designed for the Vg model, we can straightforwardly combine thresholding methods with DPM-Solver++.
combinn
RELATIONSHIP WITH OTHER FAST SAMPLING METHODS
In essence, all training free sampling methods for DPMs be understood either discretizing diffusion SDEs (Ho et al. 2020 Song et al. 20216 2021b Jolicoeur Marttnne et al.) [021] Tachibana et al.
2021 Kong& & Ping 2021 Bao al. 2022b Zhhag al. 2022) or discretizing diffusion ODEs (Song

---

Under review as a conference paper at ICLR 2023
2022 As DPM--Slver++ is designed and other diffusion ODE solvers. We further briefly discuss other fast sampling methods for DPMs.
al. 2021b Liu et al. 2022b Zhang & Chen 2022
Lu et al.
for solving diffusion ODEs, in this section, we discuss the relationship between
5.1 COMPARISON WITH SOLVERS BASED ON EXPONENTIAL INTEGRATORS
Previous state-of-thee art fast diffusion ODE solvers (Lu et al. 2022 Zhang & Chen 2022 leverages exponential integrators to solve diffusion ODEs with noise prediction models 6g. In short, these
solvers approximate the exact solution in Eq. 7 and include DDIM (Song et al. 202 as the first-order case. Below we show that the first-order case for DPM-Solverss also DDIM.
For l = 1, Eq. p becomes (after omitting the O(h+t)) terms)
+ 0z De 11411) frve eA
Tt. Tt.
0t, tt 04, (e 11--11amp 1)an (2t
04
Therefore, proposed DPM-Solver++ is the high-order generalization of DDIM w.r.t the data prediction model zg. To the best of our knowledge, such generalization has not been proposed before. We list the detailed difference between previous high-order solvers based exponential integrators and DPM-Solver++ in Table We emphasize that although the first-order version of these solvers equivalent, the high-order versions of these solvers rather different.
5.2
OTHER FAST SAMPLING METHODS
Samplers based on diffusion SDEs (Ho et al. 2020; Song et al. 202 Jolicoeur-Martineau et al.
2021 Tachibana et al. 2021 Kong Ping 2021] Bao al. 2022b; Zhang et al.) 2022) generally
needs more steps to converge than those based on diffusion ODEs (Lu et al. 2022), because SDEs
introduce more randomness and make denoising more difficult. Samplers based on extra training
include model distillation (Salimans & Ho 2022, Luhman & Luhman 2021). learning reverse process variances (San-Roman et al.
| [221] Nichol & Dhariwal [202]] [ao et al. [0.22] and learning
sampling steps (Lam et al. 2021; Watson et al. (002)). However, training- based samplers are hard to scale-up to pre-trained large DPMs (Saharia et al. [202b] Rombach et al. 2022: Ramesh et al. 2022). There other fast sampling methods by modifying the original DPMs to latent space (Vahdat et al. 2021 or with momentum (Dockhorn et al.] (202)). In addition. combining DPMs with GANs (Xiao
al.
2022 Wang al. 2022) improves the sample quality of GANs and sampling speed of DPMs.
6
EXPERIMENTS
In this section, we show that DPM-Solver++ can speed up both the pixel-space DPMs and the latent-space DPMs for guided sampling. We vary different number of function evaluations (NFE) which is the numebr of calls to the model eg (z f, e) or ze (Z,, c). and compare DPM-Solvers
with the previous state-of- the-art fast samplers for DPMs including DPM-Solver (Lu al. 2022). DEIS (Zhang & Chen 2022). PNDM (Liu et al.) 2022b) and DDIM (Song et al., (021a). We also convert the discrete-time DPMs to the continuous-timg and use these continuous time solvers. We
refer to Appendix[ the detailed implementations and experiment settings.
As previous solvers did not test the performance guided sampling, we also carefully tune the baseline samplers by ablating the step size schedule (i.e. the choice for the time steps (4) and the solver order. We find that
For the step size schedule, we search the time steps in the following choices: uniform t (a widely-used setting in high-resolution image synthesis), uniform (used in (Lu al.) 22022), uniform split of the power functions of (used in (Zhang & Chen 2022), detailed in Appendix C). and we find that the best choice is uniform l. Thus, we use uniform for the time steps in all of our experiments for all of the solvers.
We find that for a large guidance scale, the best choice for all the previous solvers is the second-order (i.e. DPM-Solver-2 and DEIS-1). However, for comprehensive comparison,
7

---

Under review as a conference paper at ICLR 2023
(DDMM) (DPMA Silve) Jadionder Na. (SisA-
(BDIM)
Lakondor (DBNM)
Anglstop (DPM odder ruttiop (1IPM (o
muttisee (DPM
The orter (DCM))
12s--adds rodiscp (DPM)))
15
(a) From 60 to 27o-
(b) From singlestep multistep.
(c) Thresholding
Figure 2: Ablation study for SoM--oover++.. Sample quality measured by FID I of different sampling methods
for DPMs on ImageNet 256x256 with guidance scale 8 8.0. varvine the number of function evaluations (NFE).
Vome
PRCU
PRCM
(a) Pixcl-space DPM
(h) Pixel-space DPM (s 4.0)
(c) Pixel-space DPM
(d) sppce DPM
(s 8.0)
(s 2.0)
7.5)
Figure 3: (a-c) Sample quality measured by FID + of different sampling methods for DPMs on ImageNet 256x256 with different guidance scale 8, varying the number of function evaluations (NFE). 1: results by combining the solver with dynamic thresholding method (Saharia et al.] (0226). (d) Convergence error measured by L2-norm L (divided by dimension) between different sampling methods and 1000-step DDIM, varying the number of function evaluations (NFE), for the latent-space DPM "stable-diffusic" MS-COC02014 validation set, with the default guidance scale = 7.5 in their official code.
(Rombach et al. 2022) on
we run all the orders of previous solvers, including DPM-Solver-2 and DPM-Solver-3; DEIS-1, DEIS-2 and DEIS-3 and choose their best result for each NFE in comparison.
We run both (2M) and (2M)))) and we find that for large guidance scales, the multistep (2M- performs better; and for a slightly small guidance scales, the singlestep DPM-Solver+++25 performs better. We report the best results of DPM-Solver++ and all of the previous samplers in the following sections, the detailed values are listed in Appendix [D]
6.1 PIXEL-SPACE DPMS WITH GUIDANCE
We firstly compare DPM- -oM-Solvr+++ with other samplers for the guided sampling with classifier- guidance on ImageNet 256x256 dataset by the pretrained DPMssDhariwal & Nichol 2021). We mea- sure the sample quality by drawing 10K samples and computing the widely-used FID score (Heusel) eet al. 2017). where lower FID usually implies better sample quality. We also adopt the dynamic thresholding method (Saharia
20226) for both DDIM and DPM-Solver+++. We vary the guidance scale in 8.0, 4.0 and 2.0, the results are shown in Fig. 3 Ba-cc.. We find that for large guidance scales, all the previous high-order samplers (DEIS, PNDM, DPM-Solver) converge slower than the first-order DDIM, which shows that previous high-order samplers are unstable. Instead, DPM-Solver++ achieve the best speedup performance for both large guidance scales and small guidance scales. Especially for large guidance scales, DPM-Solver+++ can almost converge within only 15 NEF.
As an ablation, we also compare the singlestep DPM-Solver-2, the singlestep DPM- (PM-Solver+++(S) and the multistep (PM- to demonstrate the effectiveness of our method. We use large guidance scale s = 8.0 and conduct the following ablations:
From eg De: As shown in Fig. 2a by simply changing the solver from eg to wy (i.e. from DPM-Solver-2 to OPM-Solver+++25)) the solver can achieve stable acceleration performance which faster than the first-order DDIM. Such result indicates that for guided sampling, high-order solvers w.r.t. Dg may be more preferred than those w.r.t. Eg.
From singlestep to multistep: As show in Fig. 26 the multistep DPM-Sollve+++ converges slightly faster than the singlestep DPM-Solver+++ which almost converges in
8

---

Under review as a conference paper at ICLR 2023
15 NFE. Such result indicates that for guided sampling with large guidance scale, multistep methods may be faster than singlestep methods.
With or without thresholding: We compare the performance of DDIM and DPM- + with / without thresholding methods in Fig. 2c Note that the thresholding method changes the model Bg and thus also changes the converged solutions of diffusion ODEs. Firstly, we find that after using the thresholding method, the diffusion ODE can generate higher quality samples, which is consistent with the conclusion in (Saharia et al. (002b)). Secondly, the sample quality of DPM-Solver++ with thresholding outperforms DPM-Solver++ without thresholding under the same NFE. Moreover, when combined with thresholding, DPM- Solver++ is faster than the first-order DDIM, which shows that DPM-Solver++ can also speed up guided sampling by DPMs with thresholding methods.
6.2 LATENT-SPACE DPMs WITH GUIDANCE
We also evaluate DPM-Solver++ the latent-space DPMs (Rombach al. 2022), which is recently popular among the community due to their official code "stable-diffusion" We use the default guidance scale latent code by training a pair of encoder and decoder, and then train a DPM for the latent code. As the latent code is unbounded, we do not apply the thresholding method.
7.5 in their official code. The latent-space DPMs map the image data with
Specifically, we randomly sample 10K caption-image pairs from the MS-COC02014 14 validation dataset and use the captions as conditions draw 10K images from the pretrained "stable-diffusion" " model, and we only draw a single image sample of each caption. following the standard evaluation procedures in (Nichol et al., 2021 Rombach et al., 2022). We find that all the solvers can achieve a FID around 15.00 to 16.0 even within only 10 steps, which is very close to the FID computed by the converged samples reported in the official page of "stable-diffusion." We believe it is due to the powerful pretrained decoder, which can map non-converged latent code to good image sample.
For latent-space DPMs, different diffusion ODE solvers directly affect the convergence speed on the latent space. To further compare different samplers for latent-space DPMs, we directly compare different solvers according to the convergence error on the latent space by the L2-norm between the sampled Øy and the true solution vū (and the error between them is || zúlla l/l VD. Specifically, firstly sample 10K noise variables from the standard normal distribution and fix them. Then sample 10K latent codes by different DPM samplers, starting from the 10K fixed noise variables. As all these solvers can be understood as discretizing diffusion ODEs, we compare the sampled latent codes by the true solution z from a 999-step DDIM with samples 2o by different samplers within different NFE, and the results are shown Fig. B(d). We find that the supported fast samplers (DDIM and PNDM) in "stable-diffusion" converge much slower than DPM-Solver++ and DEIS, and we find that the second-order multistep + and DEIS achieve quite close speedup on the latent space. Moreover, as -iabbe-diffusion" by default use PNDM with 50 steps, we find that DPM- Pole-Solvr+++ can achieve similar convergence error with only 15 to 20 steps. We also present an empirical comparison of the sampled images between different solvers in Appendix [ and we find that DPM-Solvert+ can indeed generate quite good image samples within only 15 20 steps.
7 CONCLUSIONS
We study the problem of accelerating guided sampling of DPMs. We demonstrate that previous high-order solvers based on the noise prediction models are abnormally unstable and generate worse- quality samples than the first-order solver DDIM for guided sampling with large guidance scales. To address this issue and speed up guided sampling, we propose DPM-Solver++, a training- ffree fast diffusion ODE solver for guided sampling. DPM-Solver++ is based on the diffusion ODE with the data prediction models, which can directly adopt the thresholding methods to stabilize the sampling procedure further. We propose both singlestep and multistep variants of OPM-Solver++ Experiment results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, applicable for pixel-space and latent-space DPMs.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Like other deep generative models such as GANs, DPMs may also be used to generate adverse fake contents (images). The proposed solver can accelerate the guided sampling by DPMs which can further be used for image editing and generate photorealistic fake images. Such influence may further amplify the potential undesirable affects of DPMs for malicious applications.
REPRODUCIBILITY STATEMENT
Our code is based on the official code of DPM-Solver (Lu al. 2022) and the pretrained checkpoints in Dhariwal & (ichol[(022] and Stable-Diffusion (Rombach et al.] 2022). We will release after the blind review. In addition, datasets used in experiments are publicly available. Our detailed experiment settings and implementations are listed in Appendix [ and the proof of the solver convergence guarantee are presented in Appendix A
REFERENCES
Kendall Atkinson, Weimin Han, and David E Stewart. Numerical solution of ordinary differential equations, volume 108. John Wiley Sons, 2011.
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint wxxiv:226.007309 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang, Analytic-DPM: An analytic estimate of the optimal reverse variance diffusion probabilistic models. International Conference on Learning Representations, 2022b.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021a
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. Wavegrad Iterative refinement for text-to-speech synthesis. International Speech Communication Association, pp. 3765-3769, 2021b.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Tim Dockhom, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped Langevin diffusion. In International Conference Learning Representations, 2022.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pp. 2672-2680, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 6626-6637, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
10

---

Under review as a conference paper at ICLR 2023
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209-288 286, 2010.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D. In International Conference on Machine Learning, pp. 8867-8887 PMLR, 2022.
Alexia Kolicoeur-Mattieea Ke Li, Rémi Piché- Fiche-Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta fast when generating data with score-based models. arXiv preprint arXiv:2105.1408) 2021.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, 2021.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint 1xXiv:2/06.00) 00132, 2021.
Max WY Lam, Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral denoising diffusion models. arXiv preprint arXiv:2108. 2021.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11020- 1028, 2022a.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint
2022b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0099 2022.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint MxXiv:2101.023800 2021.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Yun Zhu, and Stefano Ermon. SDEdit: Image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprini arXiv:21 /2.22/2/1111 12. 10741, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint arX1v:2244.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
11

---

Under review as a conference paper at ICLR 2023
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-ooiiaage diffusion models with deep language understanding. arXiv preprint arXiv:2205. I 1487, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models. arXiv preprint 1XXi::2044026000 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- lional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Hideyuki Tachibana, Mocho Go, Muneyoshi Inahara, Yotaro Katayama, and Yotaro Watanabe. Itô- Taylor sampling scheme for denoising diffusion probabilistic models using ideal derivatives. arXiv preprint arXiv:21 2021.
Lucas Theis, Tim Salimans, Matthew D Hoffman, and Fabian Mentzer. Lossy compression with gaussian diffusion. arXiv preprint wrxiv:2206.0888) 2022.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems, volume 34, pp. 11287-1132,, 2021.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan Training gans with diffusion. arXiv preprint arXiv:2206.02262, 2022.
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffuuson models by differentiating through sample quality. In International Conference on Learning Representations, 2022.
Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion-based molecule generation with informative prior bridges. arXiv preprint 0xX0:22009.00655, 2022.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations, 2022.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint (xXi:2203.002223 2022.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05366 2022.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via
mnergy-guided stochastic differential equations. arXiv preprint wwXiv:2207.06635, 2022.
12

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL PROOFS
A. PROOF OF PROPOSITION
Taking derivative w.r.t. in Eq. (8) yields
dæ: do as + dt
dhe do Lmm ( A)da+
dt co dc)
df dt O.
do at
dde
dl at + di
or
[(t) + g² 20 oz
(t) xe (e,,t)),
20
where the last inequality follows from the definitions f(t) dt dt
dllgga
gt)
dož
2 2dlog of
dt
A.2 CONVERGENCE OF ALGORITTMM
We make the following assumptions in Lu et al. 2002 for no, i.e.,
(0) (1) and T i Ii
(2) exist and are continuous (and hence are bounded).
The map x + xy x ax t) is L-Lipschitz
hmaz sax h1 O(1/M).
We also assume further
Ti > c <> for all i = 1, M.
Then, both algorithms are second-order:
Proposition A.1. Under the above assumptions, when hmaz is sufficiently small, have for both
Algorithms
land2 2 a ætn = O(h
A.2.1 CONVERGENCE OF ALGORITHM
The convergence proof this algorithm is similar to that DPM-Solver-2 (Lu al. 2022). We give it in this section for completeness.
First, Taylor's 's expansion gives
0's;
Et. Oss; (e-rc 1)00 (It, (--1111 11- 0(1)),
Fs:
at,-1
Ct Thi-1 as, hh 1)x0( t4-1) ar, (-e-hi- - h4 11+ x 200 (x t--1) + O(h)).
3t. a1,-1
Let A; = ||ãr, Lt, then l/ee - xx S + CCL, + Ch². Note that
1
xg (x),-1) rehs (Io(Ier, 81) - x 20( tz-1) S &Ch;-
Since is bounded away from zero, and -h: = 1- + h + Ohh?), we know
-h: -
(-e - h; + 1)x1 (1) (x (tt-iiti11)
(xo 5:) 2o
2r:
< CLh, Ai + ||u, x.,1l) 11) + Ch.
e -h 11 -1 hi +
+ Ti 2
|| 84) 100-ma 10
h;
< CLh,( + ||u. -x, 1)) + Ch + Ch²² (20 81) x 20(0)._114411)
< CL + C(L + +M:)h
13

---

Under review as a conference paper at ICLR 2023
where M, + supt; Sise, 1100 (xt, 1)ll- Then, A, could be estimated follows.
A S at - + Ch,(A,-1 h2)).
04111
Thus, A = =(h as long as hmaz is sufficiently small.
A.3 CONVERGENCE OF ALGORITHM
Following the same line of argument of the convergence proof of Algorithm convergence of Algorithm 2 Let A == || It, Taylor's expansion yields
|
we can prove the
It, ((o.. a
24,-1 - an, 1) ( 14-1) an, (-c te h4s + 1) 2 (0 schi
sCh),
where C/ a constant depends on x (2) Also note that
(1) (at: 1. 1(-1)
( 1,tt-1) - 20((11 2 2-= S CCh,
I
h4-1
Since r; is bounded away from zero, and e
= - hs + 2 + (hh,) ) we know
(-e-th h4 - -1 1)x0 (
(i-1)
e
hh
(
20 (2 41
2r:
(I Ch 1
e
hu -1
-e - 11s +
[100 1: wo 2 22))-2)
Pi
2
h4
C + Ch + Cha
No (101-21
CCh,(AI-1 A + CMA
where M, + sup, 8, Sist. || (1) (æt, t)ll. Then, A could be estimated follows.
A 5 0t. Ai-1 + ar, R. LA; + Oog (CMAQ CLh, (A-1 Ai-2)) Ch
04ty
Ot: A,-1 + Čh,(A +
Thus, A; = ( as long as hyar sufficiently small and Ap + Ai = =(hhaaz) which can be verified by the Taylor's expansion.
B COMPARISON BETWEEN DPM-SOLVER AND DPM-SOLVER++
In this section, we convert DPM-Solver++225 to the formulation w.r.t. the noise prediction model and compare with the second- order DPM-Solver (Lu et al. 2022).
At each step. the second-order DPM-Solver (DPM-Solver-2 (Lu et al. 2022)) has the following updating:
Oss Ct. Dz (e 1)E0 ( 11411)
U:
Ott:11
Oit: It.-1 (e 1))p(())..::tt_)))
Rt. 011-1
(10)
Tte (e 1) (Eo(us, 81) c is (--1))
2ri
while DPM-Solver++(2S) has the following updating:
Tsi Tii Oc.. -r-rss (e-uhs (T)
u;
0t:
(11)
Tt. -t:-1 an. -hi 1)
(1
1
oo (ăt, t--1) + co(
31; Ot:-1
2r,
2r,
14

---

Under review as a conference paper at ICLR 2023
Because
a - 0250
1 =x ds (2(;
Ot 04
we can rewrite DPM-Solver+++25) w.r.t. the noise prediction model (see Appendix B.1 for details):
as. It.-- Ts. (c
u;
at
at. :1,-1 - oz, (E - 1
St. On:
(12)
at: (E - 1) e e (co(u,,8s)
ca 11-
27
Comparing with Eq. 10 we can find that the only difference between DPM-Solver-2 and DPM- Solver++(2S) is that DPM-Solver+++25 has additional coefficient e e'r; at the second term (which is corresponding approximating the first-order total derivative 6 ()) Specifically, we have
co(u; 8;)
( (h+,)
As (2S multiplies a smaller coefficient into the O(h:) error term, the constant before the high-order error term of DPM-Solver++(SS) is smaller than that of DPM-Solver-2. As they both are equivalent to a second-orde, discretization the diffusion ODE, a smaller constant before the error term can result in smaller discretization eиror and reducing the numerical instabilities (especially for large guidance scales). Therefore, using the data prediction model is a key for stabilizing the sampling, and (25- is more stable than DPM-Solver-2.
B. DETAILED DERIVATION
We can rewrite (2M by:
Ust 1t.-
as re ( 1-1)
u,
01;
Os. T1-- atc-1
ass (e-tttth 1) + as as - 1) ( b-11)
Ot,-1
Oss Dt,-1
00 -1)e((()))
0t:: 1
and
0t: Dt.- - an -1) (1 (1-
1
Ct;
(:: ti-1) + xo xo(ui,
Tt.-1
2r:
2r;
04 01,- - (c,
at (e 11)(1x U1, 81) x 199(::_..14-11
4-11)
01,11
2r:
(it: Øtt-1 - 0e,
0t, (c - 2 (xo 20
4-1) +
at--1
2r;
and
A (Dg (e)( - xe x0l 20(
1
- - col - co(o), ,.11-))
ii:-1 - -or
81) te
xx
Arcoluu 8i) + to - eo
Ai-1
Ar Ep (--1) - -g
(Eo (60(0t,,, s0)u;;8:))
so we have
O4
1t, - 0 1)-
Tt. (e - 1)e -rchi (co(u,,))) - co
Tt; = au-1
2r;
15

---

Under review as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
C.I CONVERTING DISCRETE- TIME DPMS TO CONTINUOUS-
Discrete-time time DPMs (Ho et al. 2020) train the noise prediction model Eø at N fixed time steps [tn)n=) and the noise prediction model is parameterized by (x,, 1000m) for n = I,...,N, NN where each 2n corresponding the value at time t,- In practice, these discrete-time DPMs usually "f for n ,,...,N, N. The smallest time is N choose uniform time steps between [0, T), thus in N
Moreover, for the widely-used DDPM (Ho et al. 2020), we usually choose a sequence (B,, J which is defined by either linear schedule (Ho et (020) or cosine schedule (Nichol & Dhariwal 2021 After obtained the 3n sequence, the noise schedule an is defined by
On = II (n) 1=1
(13)
where each On is corresponding to the continuous-time l, = nT i.e. 04,, = On- To generalize N the discrete an to the continuous version, we use a linear interpolation for the function log ag. Specifically, for each & E [fn, 1n+1], we define
log 0z = log on + log an+1 - log an
fx))
(14)
hn+1 t,
Therefore, we can obtain a continuous-time noise schedule aa defined for all f E [ T]. and the std ot /1 of and the logSNR A log 04 - log ØN, Moreover, the logSNR A, is strictly decreasing for t, thus the change-of-variide variiable for À is still valid.
In practice, we usually have T 1 and N 1000, thus the smallest time is 10-3 Therefore, we solve the diffusion ODEs from time = 1 to time & = 10-3 to get our final sample. Such sampling can reduce the first-order discrete-time time DDIM solver when using uniform time step.
C.2 ABLATING TIME STEPS
Previous DEIS only tuned on low-resolutional data CIFAR-10, which may be not suitable for high- resolutional data such as ImageNet 256x256 and large guidance scales for guided sampling. For fair comparison with the baseline samplers, we firstly do ablation study for the time steps with the pretrained DPMs (Dhariwal & Nichol (20TT) on ImageNet 256x256 and vary the classifier guidance scale. In our experiments, we tune the time step schedule according to their power function choices. Specifically, let (M 10 and lo = the time steps (4:).M satisfies
M i M
M
where x is hyperparameter. Following Zhang & Chen (002), we search in 1, by DEIS, and the results are shown in Table We find that for all guidance scales, the best setting is к i.e. the uniform for time steps. We further compare uniform and uniform À and find that the uniform time step schedule is still the best choice. Therefore, in all of our experiments, we use the uniform / for evaluations.
C.3 EXPERIMENT SETTINGS
We use uniform time step schedule for all experiments. Particularly, as DPM-Solver (Lu et al.) 2022) is designed for uniform (the intermediate time steps are a half of the step size w.r.t. A), we also convert the intermediate time steps to ensure all the time steps are uniform (. We find that such conversion can improve the sample quality both the singlestep DPM-Solver the singlestep DPM-Solver++.
We run NFE 10, 15, 20, 25 for the high-order solvers and additional 50, 100, 250 for DDIM. For all experiments, solver diffusion ODEs from f to = 10-3 with the interpolation of noise schedule detailed in Appendix C.1 For DEIS, we use the "t-AB-k" methods for 1,2, 3, which is the fastest method in their original paper, and name them DEIS-K. respectively.
For the sampled image in Fig. 5 we use the prompt "A beautiful castle beside a waterfall in the woods, by Josef Thoma, matte painting, trending on artstation HQ".
16

---

Under review as a conference paper at ICLR 2023
Table Sample quality measured by on ImageNet 256x256 (discrete- time model DDhariwal Nichol
2021 ). varying the methods between DDIM (Song et al. 2021a> and different types DEIS (Zhang & Chen
2022
The number of function evaluations (NFE) is fixed by 10.
Method Guidance scale
8.0
7.0
6.0
5.0
4.0
2.0
0.0
DDIM
13.04
12.38
11.81
11.55
11.62
11.95
13.01
16.35
29.33
DEIS-2, -=
19.12
14.83
12.39
10.94
10.13
9.76
9.74
11.01
20.34
DEIS-2, k=
33.37
24.66
18.03
13.57
11.16
10.54
10.88
13.67
26.26
DEIS-2,
55.69
44.01
33.04
24.50
18.66
16.35
16.87
21.91
38.41
DBIS-3, k=
66.81
48.71
33.89
22.56
15.84
11.96
10.18
10.19
18.70
DEIS-3, == =
34.51
25,42
18.52
13.68
11.20
10.46
10.75
13.36
25.59
DEIS-3,
56.49
44.51
33.34
24.68
18.72
16.38
16.79
21.76
38.02
D EXPERIMENT DETAILS
We list all the detailed experimental results in this section.
Table
Sample quality measured by on ImageNet 256x256 (discrete-tin model Dhariwal Nichol varying the number function evaluations (NFF).
2021
Guidance Scale
Thresholding
Sampling Method ; NFE
10
15
20
25
50
100
250
DDIM (Snng et al
2021a
13.04
11.27
10.21
9.87
9.82
9.52
9.37
[LU al
20226
99.80
37.59
15.50
Dolveel1
Lu al.
2022
114.62
44.05
20.33
9.84
DPM-Solver-3
ml
21ZZ
164.74
91.59
No
DEIS-1 Zhang
15.20
10.86
10.26
10.01
8.0
DEIS-2 Zhang
19.12
11.37
10.08
9.75
DEIS-3 Zhang
66.86
24.48
12.98
10.87
DPM-Solver++(S) (ours)
12,20
9.85
9.19
9.32
DPM-Solver++MM) (ours)
14.44
9.46
9.10
9.11
DDIM (Song ct al.
2021a
10.58
9.53
9.12
8.94
8.58
8.49
8.48
Yes
DPM-Solver+++S)
(ours)
9.26
8.93
8.40
8.63
PPM-Solver-+<M) (ours)
9.56
8.39
DDIM (oog ct al
20
11.62
9.67
8.96
8.58
7.99
PNDM
22.71
10.03
8.69
8.47
DPM-Solver-2
2022
37.68
9.42
8.22
8.08
DPM-Solven-3
2122
74.97
1..665
9.99
8.15
No
DEIS-1 Zhang
10.55
9.47
8.88
8.65
4.0
DEIS-2 Zhang & hen
10.13
9.09
8.68
8.45
DEIS-3
Zhang & hien
2022
15.84
9.25
8.63
8.43
DPM-Solveri-+(S) (ours)
9.08
8.51
8.00
8.07
DPM-Solveri+<MM (ours)
8.98
8.26
8.06
8.06
DDIM (ong et al. 2021a
10.45
8.95
8.51
8.25
7.91
7.82
7.87
Yes
DPM-Solver++(S)
(ours)
8.94
8.26
7.95
7.87
DPM-Solver++MM) (ours)
8.91
8.21
7.99
7.96
DDIM Song
2021a
13.01
8.45
7.44
PNDM LLu
20226
11.58
8.48
8.17
7.97
DPM PPM--olver-Z
ef al.
2022
14.12
8.20
8.59
7.48
DPM-Solver-3 Lu al
2027
21.06
8.57
8.19
7.85
No
DEIS-1 (7hang & hem
2012
10.40
9.11
8.52
8.21
2.0
DEIS-2 Zhang & hen
2022
9.74
8.80
8.28
8.06
DEIS-3 Zhäng & Chen
2022
10.18
8.63
8.20
7.98
DPM-Solver-++(S) (ours)
9.18
8.17
7.77
7.56
DPM-Solver++<MM) (ours)
9.19
8.47
8.07
DDIM (ung 2021a
7.63
Yes
OPM--olver+++0 (ours)
9.23
8.18
7.81
7.60
DPM- (ours)
9.28
8.56
8.28
8.18
17

---

Under review as a conference paper at ICLR 2023
Table 4: Sample quality measured by MSE. on COCO2014 validation (discrete-time latent model (Rombach etal. 2022), varying the number function evaluations (NFE). Guidance is 7.5, which the recommended setling for stable-diffusion [Rombach et al. 2022).
Guidance Scale
Thresholding
Sampling Method NFE
10
15
20
25
50
100
250
DDIM (Song al.
2021a
0.59
0.42
0.48
0.45
0.34
0.23
0.12
PNDM Liu
20226
0.66
0.43
0.50
0.46
0.32
DPM-Sol 2
2022
0.66
0.47
0.40
0.34
0.20
DPM-Solver-3
0.59
0.48
0.43
0.37
0.23
7,5
No
DEIS-1 Zhang & Chen
2022
0.47
0.29
0.16
DEIS-2 2
Zhang & Chen
2022
0.48
0.40
0.34
0.29
0.15
DEIS-3 3
Zhang & Chen
2022
0.57
0.45
0.38
0.34
0.19
DPM- (PM-+Sllvr+++(S) (ours)
0.48
0.41
0.36
0.32
0.19
DPM-Solvar+++(M (ours)
0.49
0.40
0.34
0.29
0.16
DDIM (Song al.
2021a
0.83
0.78
0.71
0.67
PNDM \
Tau el 20020
0.99
0.87
0.79
0.75
DPM- JPM-Solver-2
2022
1.13
1.08
0.96
0.86
DETS-1 Zhang
0.84
0.58
15.0
No
DEIS-2 Zhang & Chen
0.63
DEIS-3
Zhang & Chin 2022
1.06
0.78
0.73
(PM--oller+++(S) (ours)
0.88
0.75
0.68
0.61
DPM-Solver (PMM-oller+++M) (ours)
0.84
0.72
0.64
0.58
DPM-Solver-2 (Eg, singelstep)
DPM-Solver++((2)) (æg, singlestep)
(OM--Slver++((MM (xr,, multistep, thresholdin)
Figure 4: Samples of different sampling methods for DPMs ImageNet 256x256 with guidance scale 8.0.
18

---

Under review as a conference paper at ICLR 2023
UU
DDIM (N 15)
DDIM (N 20) (Sogg al. 2021a
DDIM (N 50)
(ong al. 2021a
(Song al. 2021a
PNDM (N 15)
PNDM (LV 20) Liu al. 20226
PNDM (N 50) Liu 2022b
Liu 2022b
DDIM (N 999,
DPM-Solver-2 (N 15)
DPM-Solver-2 (N 20)
DPM-Solver-2 (N 50)
converged)
(Lu et al. 2022
(u et al. 2022)
Lu 2022)
Song al. 2021a
DEIS- 1 (N 15) Zhang & Chen 2022
DEIS-1 (N 20)
DEIS- (N 50)
ZZhag & Chen 2022
Zhang & Chen 2022
DPM-Solver++(2M) (N = 15)
(PM-Solver++(MM) (N = 20) (ours)
(2M) (N 50)
(ours)
(ours)
Figure 5: Samples using pre-trained space DPMs -ifffu--ii (Rombach 2022 with a classifier free guidance scale 7.5 (the default setting), varying different samplers and different number of function evaluations N.
19


Reviews:
[{"review_id": "xBWTAidbr", "reviewer": "Reviewer_8HFn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles fast sampling from diffusion models when using guidance. Guided sampling has become a crucial ingredient in conditional diffusion models to achieve high synthesis quality. However, most existing solvers for accelerated synthesis are not developed specifically with guidance in mind. In fact, the paper shows that some previous higher-order methods for fast sampling perform poorly in the guidance setting. The paper then proposes the DPM-Solver++, a higher-order solver that is based on an exponential integrator framework, similar to previous work. The approach is essentially a direct follow-up of DPM-Solver, tailored to the guidance setting. Compared to previous works, there are two crucial differences: (a) the new solver works with the x-prediction diffusion model parametrization, in contrast to previous works using epsilon-prediction. (b) This enables an easy application of thresholding algorithms to prevent samples from going out of bounds when performing the iterative synthesis process with large guidance weights. Moreover, the paper develops its novel solver both with a single-step framework and a multi-stepping framework.\n\nExperimentally, the novel DPM-Solver++ compares favourably to a variety of baselines on various fast synthesis tasks. The solver is applied on conditional pixel space diffusion models as well as on Stable Diffusion, a latent text-to-image diffusion model.", "summary_of_the_review": "In summary, the paper studies an important problem, fast sampling from conditional diffusion models using guidance with higher-order solvers. However, I think the claims made around other higher-order methods not being able to apply thresholding are either incorrect, or should be discussed in more detail to avoid confusion. Furthermore, the conceptual novelty of the paper is rather small. Experimentally, the results appear strong, but arguably guidance and thresholding should be applied for all baselines, if possible (see discussion on this above).\n\nIn conclusion, I believe the paper is not quite ready yet for publication in its current form. I am willing to increase my paper rating, if my concerns can be addressed and potential confusions resolved.", "strength_and_weaknesses": "**Strengths:**\n\n- Investigating fast higher-order solvers for diffusion model sampling specifically for the guidance scenario is a good idea and practically very important, due to the wide usage of guidance in the diffusion model literature.\n- The shown experimental results are strong and the new sampling approach works well in almost all settings. It usually outperforms all existing methods.\n\n**Weaknesses:**\n\n- The technical novelty of the solver is small. It is a direct follow-up of DPM-Solver, using the same framework. Changing from epsilon- to x-prediction is simple and applying thresholding is also trivial. The multistepping approach has also been used previously in the diffusion modeling literature.\n\n- Why exactly does x-prediction work better than epsilon-prediction (in isolation, without thresholding)? This should be properly discussed.\n\n- The paper points out that other higher-order baselines such as DEIS and DPM-Solver cannot use thresholding in their higher-order forms. I am not sure this is entirely correct. Unless I am misunderstanding something, I would think that also DEIS and DPM-Solver can always take their epsilon prediction, convert into x-prediction via $x_\\theta = (x_t - \\sigma_t \\epsilon_\\theta)/\\alpha_t$, apply the threshold, and convert back to epsilon prediction (and then just use the corresponding frameworks). I think the paper should analyze these details much more thoroughly and discuss in more detail. I would recommend the authors to lay out in detail how exactly thresholding is applied in DPM-Solver++ in the higher-order scheme and why this shouldn't be possible in the baselines --- because it looks to me like it was possible. Moreover, if the other methods can use thresholding, after all, the experimental baseline comparisons would have to be adapted accordingly.\n\n- The paper writes that it searches over step size schedules and then uses the same schedule in all experiments for all solvers. Maybe I am confusing something, but I am not sure this is an ideal approach. I think using the same step size schedule in all experiments could potentially be unfair to methods that may have been designed with other schedules in mind. I think for every baseline the best possible step size schedule should be chosen. It sounds as if only DEIS and DPM-Solver++ are considered when finding step size schedules. DPM-Solver and PNDM could potentially have other optimal choices.\n\nMinor question: In Figure 1, does DPM-Solver++ use thresholding or not? It would be interesting to add the synthesized image for DPM-Solver++ both with and without thresholding.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "s-Te5Gy2wGQ", "reviewer": "Reviewer_a1rL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on a practical aspect of diffusion-based generative models, i.e., training-free fast guided sampling of DDPM. It is a direct extension of the recently proposed fast unconditional sampler namely DPM-Solver.  The authors identify the unstable sampling trajectory of DPM-Solver for guided sampling, and present several tricks to resolve this issue. These include 1) reformulating the diffusion ODE with data prediction parameterization in lieu of noise prediction; 2) deriving the corresponding ODE higher-order solver leveraging the semi-linear structure of diffusion ODE and exponential integrators; 3) combining the solver with the thresholding trick. The numerical results show these techniques successfully resolve the convergence issue of the DPM-Solver for guided sampling, in both pixel space and latent space settings. ", "summary_of_the_review": "IMO, the ideas presented in this manuscript serve a nice contribution as a journal extension of its predecessor but are not appropriate to be published as another top-tier conference paper. ", "strength_and_weaknesses": "[Pros.]\n- Good practical extension of DPM-Solver for guided sampling\n- The presentation is clear and easy to follow\n\n[Cons.]\n- The technical novelty behind this work is rather incremental. The main techniques used in this paper are directly borrowed from its predecessor [A] (e.g, semi-linear diffusion ODE solver, exponential integrators) without significant deltas. The innovations introduced are the data prediction formulation of the diffusion ODE and the multistep solver, which is good but not surprising. It seems the key to stabilizing sampling is owing to the dynamic thresholding method, which also comes from another paper [B]. As a result, it's hard to evaluate the technical novelty of this paper, the proposed method looks sound but is more of an \"engineering\" trick\n- Could you please explain why the data prediction parameterization yields better results then noise prediction one even without using thresholding?\n\n[A] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022\n[B] Photorealistic text-to-image diffusion models with deep language understanding, arxiv 2022\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "au0d-Y8mmG", "reviewer": "Reviewer_s8Si", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this submission deals with accelerating guided sampling of DPMs. The challenge however is that high-order samplers based on noise prediction become unstable for large guidance steps, and they suffer from test-train mismatch. To address these challenges, this work proposes an ODE solver based on the data prediction model that approximates the integrator of the score with Taylor approximation and calculates the first-order and second-order derivatives. In addition, a clipping is used to keep the solution within the range. Experiments with both pixel and latent space DPMs indicate that it can generate high fidelity images in 15-20 steps. \n", "summary_of_the_review": "This paper addresses an interesting and timely topic. The contribution however seems limited. It simply switches to the model prediction method and uses the common clipping to avoid instability. None of this seems to be a surprise, and thus the significance and generality of the idea is not clear. ", "strength_and_weaknesses": "Strength\n- experimental results show significant improvement over DDIM for ImageNet in 15-20 steps\n\nWeakness\n- The idea is not principled and this reviewer is confused to find a principled way to think about the generality of the idea for handling instability for distributions beyond the reported experiments. It is not quite clear why switching to a data prediction model avoids the instability. Clipping is also a common method for such scenarios and not surprising.\n\n\nQuestions and comments\n- About latent diffusions and experiments with stable diffusions, is there instability observed as in the pixel space? The latent space seems to be smoother and as a result the higher order derivatives of the score could be more bounded. Can the instability still be a serious issue there?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q9U7g4dA5W", "reviewer": "Reviewer_zAj6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes a fast ODE solver for the guided sampling of diffusion probabilistic models with parameterization and thresholding. However, experiments and comparisons are not fair.", "summary_of_the_review": "The work proposes a fast ODE solver for the guided sampling of diffusion probabilistic models with parameterization and thresholding. However, experiments and comparisons are not fair. I would like to increase the score if authors can address the questions I raised.", "strength_and_weaknesses": "Strength:\n\n* Authors propose data prediction model from noise prediction model based on parameterization and its corresponding ODEs.\n\nWeakness:\n\n* The algorithm is quite similar to previous works, DEIS and DPM-Solver. The improvement is incremental.\n\n* It would be nice if the author could add a comparison between Algorithm1 and DPM-Solver-2 in the language of the same model and highlight the difference. Because the data prediction model is induced by $x_\\theta = (x_t - \\sigma_t \\epsilon) / \\alpha_t$, Algorithm1 should be able to be rewritten as noise prediction model too. \n\n* Table 1 is biased. First, both DEIS and DPM-Solver can be combined with thresholding easily. Notice that $x_\\theta = (x_t - \\sigma_t \\epsilon) / \\alpha_t$, we can infer corresponding \"thresholded\" $\\epsilon$ from \"thresholded\" $x_\\theta$ based on $\\epsilon = (x_t - \\alpha_t x_\\theta) / \\sigma_t $.\nSecond, DEIS can be shown as a high-order method and enjoy high-order convergence. If the reviewer understands it correctly, the analytical form can be obtained by exponential transformation introduced in DEIS~(see sec 4 in DEIS). \n\n* Therefore, most experiments are not comprehensive and need to show the performance of PNDM/DPM/DEIS with thresholding.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject