Fast Sampling of Diffusion Models with Exponential Integrator
Authors: Qinsheng Zhang
Yongxin Chen
Venue: ICLR 2023 poster (2023)
Keywords: Fast diffusion model
generative model

Abstract: The past few years have witnessed the great success of Diffusion models~(DMs) in generating high-fidelity samples in generative modeling tasks. A major limitation of the DM is its notoriously slow sampling procedure which normally requires hundreds to thousands of time discretization steps of the learned diffusion process to reach the desired accuracy. Our goal is to develop a fast sampling method for DMs with a much less number of steps while retaining high sample quality. To this end, we systematically analyze the sampling procedure in DMs and identify key factors that affect the sample quality, among which the method of discretization is most crucial. By carefully examining the learned diffusion process, we propose Diffusion Exponential Integrator Sampler~(DEIS). It is based on the Exponential Integrator designed for discretizing ordinary differential equations (ODEs) and leverages a semilinear structure of the learned diffusion process to reduce the discretization error. The proposed method can be applied to any DMs and can generate high-fidelity samples in as few as 10 steps. Moreover, by directly using pre-trained DMs, we achieve state-of-art sampling performance when the number of score function evaluation~(NFE) is limited,  e.g., 4.17 FID with 10 NFEs, 2.86 FID with only 20 NFEs on CIFAR10.

Published as a conference paper at ICLR 2023
FAST SAMPLING OF DIFFUSION MODELS WITH EXPO- NENTIAL INTEGRATOR
Qinsheng Zhang
Yongxin Chen
Georgia Institute of Technology qzhang4 1 98gatech. edu
Georgia Institute of Technology yongcheengattech. edu
ABSTRACT
The past few years have witnessed the great success of Diffusion models (DMs) in generating high-fidelity samples generative modeling tasks. A major limitation of the DM is its notoriously slow sampling procedure which normally requires hundreds to thousands of time discretization steps of the learned diffusion process to reach the desired accuracy. Our goal is to develop fast sampling method for DMs with fewer steps while retaining high sample quality. To this end, we sys- tematically analyze the sampling procedure in DMs and identify key factors that most cru-
affect the sample quality, among which the method of discretization cial. By carefully examining the learned diffusion process, we propose Diffusion Exponential Integrator Sampler (DEIS). It is based on the Exponential Integrator designed for discretizing ordinary differential equations (ODEs) and leverages a semilinear structure of the learned diffusion process to reduce the discretization eтror. The proposed method can be applied any DMs and can generate high- fidelity samples in as few as 10 steps. Moreover, by directly using pre-trained DMs, we achieve state-of-art sampling performance when the number of score function evaluation (NFE) is limited, e.g., 4.17 FID with 10 NFEs, 2.86 FID with only 20 NFEs on CIFAR1O. Project page and code: https://qsh-zh https:/ggshzh.giihub..iee github.io/dei..
INTRODUCTION
The Diffusion model (DM) (Ho al., 2020) is a generative modeling method developed recently that relies on the basic idea of reversing a given simple diffusion process. A time-dependent score function is learned for this purpose and DMs are thus also known as score-based models (Song et al., 2020b). Compared with other generative models such as generative adversarial networks (GANs), in addition great scalability, the DM has the advantage of stable training less hyperparameter sensitive (Creswell et al., 2018; Kingma & Welling, 2019). DMs have recently achieved impres- sive performances on a variety of tasks, including unconditional image generation (Ho et al., 2020; Song et al., 2020b; Rombach al., 2021; Dhariwal & Nichol, 2021), text conditioned image gen- cration (Nichol et al., 2021; Ramesh et al., 2022), text generation (Hoogeboom et al., 2021; Austin al., 2021), ), 3D point cloud generation (Lyu et al., 2021), inverse problem (Kawar et al., 2021; Song et al., 2021b), etc.
However, the remarkable performance of DMs comes at the cost of slow sampling; it takes much longer time to produce high-quality samples compared with GANs. For instance, the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) needs 1000 steps to generate one sam- ple and each step requires evaluating the learning neural network once; this is substantially slower than GANs (Goodfellow al., 2014; Karras et al., 2019). For this reason, there exist several stud- ies aiming at improve the sampling speed for DMs (More related works are discussed in App. A). One category of methods modify/optimize the forward noising process such that backward denois- ing process can be more efficient (Nichol & Dhariwal, 2021; Song et al., 2020b; Watson et al., 2021; Bao et al., 2022). An important and effective instance is the Denoising Diffusion Implicit Model (DDIM) (Song et al., 2020a) that uses a non-Markovian noising process. Another category of methods speed up the numerical solver for stochastic differential equations (SDEs) or ordinary differential equations (ODEs) associated with the DMs Dollicoeur-Martinee et al., 2021; Song al., 2020b; Tachibana et al., 2021). In (Song et al., 2020b), blackbox ODE solvers are used to solve a marginal equivalent ODE known the Probability Flow (PF), for fast sampling. In (Liu et al.,

---

Published as a conference paper at ICLR 2023
WORLD PEACE
WORLD PEACE
Figure 1:
Generated images with various DMs. Latent diffusion (Rombach et al., 2021) (Left), 256 x 256 image with text A shirt with inscriplion "World peace" (15 NFE). VE diffusion (Song et al., 2020b) (Mid), FFHQ 256 x 256 (12 NFE). VP diffusion (Ho et al., 2020) (Right), CIFARIO (7 NFE) and CELEBA (5 NFE).
2022), the authors combine DDIM with high order methods to solve this ODE and achieve further acceleration. Note that the deterministic DDIM can also be viewed as a time discretization of the PF as it matches the latter in the continuous limit (Song et al., 2020a; Liu et al., 2022). However, it unclear why DDIM works better than generic methods such Euler.
The objective of this work is to establish principled discretization scheme for the learned backward diffusion processes in DMs so as to achieve fast sampling. Since the most expensive part in sampling a DM is the evaluation of the neural network that parameterizes the backward diffusion, we seek discretization method that requires a small number of network function evaluation (NFE). We start with a family of marginal equivalent SDEs/ODEs associated with DMs and investigate numerical error sources, which include fitting error and discretization error. We observe that even with the same trained model, different discretization schemes have dramatically different performances in terms of discretization error. We then carry out a sequence of experiments to systematically inves- tigate the influences of different factors the discretization error. We find out that the Exponential Integrator (EI) (Hochbruck & Ostermann, 2010) that utilizes the semilinear structure of the back- ward diffusion has minimum error. To further reduce the discretization error, we propose either use high order polynomials to approximate the nonlinear term in the ODE or employ Runge Kutta methods on a transformed ODE. The resulting algorithms, termed Diffussio Exponential Integrator Sampler (DEIS), achieve the best sampling quality with limited NFEs.
Our contributions are summarized as follows: 1) We investigate family of marginal equivalent SDEs/ODEs for fast sampling and conduct systematic error analysis for their numerical solvers. 2) We propose DEIS, an efficient sampler that can be applied to any DMs to achieve superior sampling quality with a limited number of NFEs. DEIS can also accelerate data log-likelihood evaluation. 3) We prove that the deterministic DDIM a special case of DEIS, justifying the effectiveness of DDIM from discretization perspective. 4) We conduct comprehensive experiments to validate the efficacy of DEIS. For instance, with a pre-trained model (Song et al., 2020b), DEIS is able to reach 4.17 FID with 10 NFEs, and 2.86 FID with 20 NFEs CIFAR10.
2 BACKGROUND ON DIFFUSION MODELS
A DM consists of a fixed forward diffusion (noising) process that adds noise to the data, and learned backward diffusion (denoising) process that gradually removes the added noise. The backward diffusion is trained to match the forward in probability law, and when this happens, principle generate perfect samples from the data distribution by simulating the backward diffusion.
in
Forward noising diffusion: The forward diffusion of DM for D-dimensional data is linear diffusion described by the stochastic differential equation (SDE) (Särkkä & Solin, 2019)
da F,ædl + Gidw.
2

---

Published as a conference paper at ICLR 2023
Table 1: Two popular SDEs, variance preserving SDE (VPSDE) and variance exploding SDE
(VESDE). The parameter is decreasing with ao = 1, or 2 0, while or is increasing.
SDE
Ft
G:
VPSDE (Ho et al., 2020)
dlogg die
I
log
Vor
(1 ai)
VESDE (Song et al., 2020b)
O
d I
I
I?I
where F; E DxD denotes the linear drift coefficient, G. E DDD denotes the diffusion coef- ficient, and is a standard Wiener process. The diffusion Eq. (1) is initiated at the training data and simulated over a fixed time window [0, T. Denote by Pr (xt) the marginal distribution of and by Por (x (o)) the conditional distribution from æo to Zt. then po represents the underlying distribution of the training data. The simulated trajectories are represented by {æ,, JusieT. The pa- rameters F, and G2 are chosen such that the conditional marginal distribution Poe (2))//))) is simple Gaussian distribution, denoted (u, Σe), and the distribution (
Pr (r(r) ) is easy to sample from. Two popular SDFs in diffusion models (Song et al., 2020b) are summarized in Tab. 1. Here matrix notation for F, and G, to highlight the generality of our method. Our approach is applicable to any DMs, including the Blurring diffusion models (BDM) (Hoogeboom & Salimans, 2022: Rissanen al., 2022) and the critically- Langevin diffusion (CLD) (Dockhorn al., 2021) where these coefficients are indeed non-diagonal matrices.
Backward denoising diffusion: Under mild assumptions (Anderson, 1982; Song et al., 2020b), the forward diffusion Eq. (1) is associated with a reverse- time diffusion process
dx
FFad - G.G.T V log (::))d G,dw,
(2)
where w denotes a standard Wiener process in the reverse-time direction. The distribution of the trajectories of Eq. (2) with terminal distribution æт π coincides with that of Eq. (1) with ini- tial distribution 2o ~ that is, Eq. (2) matches Eq. (1) in probability law. Thus, in principle, we can generate new samples from the data distribution Po by simulating the backward diffusion Eq. (2). However, to solve Eq. (2), we need to evaluate the score function V Voo Pr (x)), which is not accessible.
The basic idea of DMs is to use time-dependent network so(æ,, f), known as score network, to approximate the score pl This achieved by score matching tech- niques (Hyvärinen, 2005; Vincent, 2011) where the score network 8q is trained by minimizing the denoising score matching loss
Training:
(()) Eni (ppp())pplleee (ar [||V log (xx so sa 1)
(3)
Here V log Por ( (g)) has a closed form expression as Por (a; (azg) is simple Gaussian distribution, and A, denotes time-dependent weight. This loss can be evaluated using empirical samples by Monte Carlo methods and thus standard stochastic optimization algorithms can be used for training. We refer the reader to (Ho et al., 2020; Song et al., 2020b) for more details choices of A, and training techniques.
3 FAST SAMPLING WITH LEARNED SCORE MODELS
Once the score network (x t) s V log (x) ( is trained, it can be used to generate new samples by solving the backward SDE Eq. (2) with V log p. (x) replaced by s0( t). It turns out there are infinitely many diffusion processes one can use. In this work, we consider a family of SDEs
dā [F. 1 X2
G.G. se (ă, t)]dt + XG,dw,
(4)
parameterized by À > O. Here we use d to distinguish the solution to the SDE associated with the learned score from the ground truth æ in Eqs. (1) and (2). When À 0. Eq. (4) reduces to ODE known as the probability flow ODE (Song et al., 2020b). The reverse-time diffusion Eq. (2) with an approximated score is special by Eq. (4) as (Øt JosesT and the marginal distributions as p,. The following Proposition (Zhang & Chen, 2021) (Proof in App. D) holds.
of Eq. (4) with À 1. Denote the trajectories generated
3

---

Published as a conference paper at ICLR 2023
Proposition 1. When sg(x,t)= = V log ( for all x, t, and p7. n, the marginal distribution it of Eq. (4) matches Pt of the forward diffusion Eq. for all 0 T.
The above result justifies the usage of Eq. (4) for generating samples. To generate a new sample, one can sample o from π and solve Eq. (4) to obtain a sample #t. However, in practice, exact solutions to Eq. (4) are not attainable and one needs to discretize Eq. (4) over time to get an approximated solution. Denote the approximated solution by æt, and its marginal distribution by Pt. then the error of the generative model, that is, the difference between po(x)) and Po(æ), is caused by two error sources, fitting error and discretization error. The fitting error is due to the mismatch between the learned score network so and the ground truth score V logg: (z:). The discretization error includes all extra errors introduced by the discretization in numerically solving Eq. (4). To reduce discretiza- tion error, one needs to use smaller stepsize and thus larger number of steps, making the sampling less efficient.
The objective of this work is to investigate these two error sources and develop a more efficient sampling scheme from Eq. (4) with less errors. In this section, we focus on the ODE approach with A = O. All experiments in this section are conducted based on VPSDE over the CIFARIO dataset unless stated otherwise. The discussions on SDE approach with À >> 0 are deferred App. C.
3.1
CAN WE LEARN GLOBALLY ACCURATE SCORE?
Since DMs demonstrate impressive empirical re- sults in generating high-fidelity samples, it is tempting to believe that the learned score network is able to fit the score of data distribution very well, that is. $ø(x,, t) V log (z()) for almost all E R²' and t E [0, Т]. This is, however, not true; the fitting error can be arbitrarily large on some ar f as illustrated in simple example below. In fact, the learned score models are not accurate for most t.
Figure 2: Fitting error on toy demo. Lighter areas represent higher probability region (left) and larger fitting error (right).
Consider a generative modeling task over I-dimensional space, i.e., D = 1. The data distribution is Gaussian concentrated with very small variance. We plot the fitting error' between a score model trained by minimizing Eq. (3) and the ground truth score in Fig. 2. As can be seen from the figure, the score model works well in the region where pe (ar) is large but suffers from large error in the region where (ar) is small. This observation can be explained by examining the training loss Eq. (3). In particular, the training data of Eq. (3) sampled from (ar). In regions with a low (z()) value, the learned score network is not expected to work well due to the lack of training data. This phenomenon becomes even clearer in realistic settings with high-dimensional data. The region with high Pe (=())) value is extremely small since realistic data is often sparsely distributed in R P; it is believed real data such as images concentrate on an intrinsic low dimensional manifold (Deng al., 2009; Pless & Souvenir, 2009; Liu et al., 2022).
As a consequence, to ensure do is close to F.. we need to make sure á; stays in the high p: region for all t. This makes fast sampling from Eq. (4) challenging task as it prevents us from taking an aggressive step size that is likely to take the solution to the region where the fitting error of the learned score network is large. A good discretization scheme for Eq. (4) should be able to help reduce the impact of the fitting error of the score network during sampling.
3.2 DISCRETIZATION ERROR
We next investigate the discretization error of solving the probability flow ODE (A 0)
dē
G.G.G tt
di F,ē- 2
(5)
The exact solution to this ODE is
1 G. Go (( i = s)z, + J. V(t,
T) ((((,T)]
(6)
Becauus the fitting error explodes when t + 0, we have scaled the fitting error for better visualization.
4

---

Published as a conference paper at ICLR 2023
Sx(X, 1) parameterization
5 parameterization
Eo|x, peremeterization
Eolx, paramettr zation
Euler
5=5
Euler EI
N=5
18
N=10
100
N-10
0.35
12
2=
13
N=20
<3 167
0.15
< 100
10
0.1
0.55
0.01
16-1
N
N
(a)
(b)
(c)
(d)
Figure 3: Fig. 3a shows average pixel difference Ap between ground truth 26 and numerical solution
Lo from Euler method and EI method. Fig. 3b depicts approximation error As along ground truth solutions. Fig. 3d shows A. can be dramatically reduced if the parameterization es () instead of s0( 1) is used. This parameterization helps the El method outperform the Euler method in Fig. 3c.
where (t, s) satisfying à (t, s) = (t,,s, s). (s, s) is known as the transition matrix from time s to associated with F,. Eq. (5) is semilinear stiff ODE (Hochbruck & Ostermann, 2010) that consists of linear term F and a nonlinear term 8a (1 t). There exist many different numerical solvers for Eq. (5) associated with different discretization schemes to approximate Eq. (6) (Griffiths & Higham, 2010). As the discretization step size goes to zero, the solutions obtained from all these methods converge to that of Eq. (5). However, the performances of these methods can be dramatically different when the step size is large. On the other hand, to achieve fast sampling with Eq. (5), need to approximately solve it with small number of discretization steps, and thus large step size. This motivates us to develop an efficient discretizaiton scheme that fits with Eq. (5) best. In the rest of this section, we systematically study the discretization error in solving Eq. (5), both theoretically and empirically with carefully designed experiments. Based on these results, we develop an efficient algorithm for Eq. (5) that requires small number of NFEs.
ät
Ingredient 1: Exponential Integrator over Fuler method. The Fuler method is the most elemen- tary explicit numerical method for ODEs and is widely used in numerical softwares (Virtanen al., 2020). When applied to Eq. (5), the Euler method reads
1 G.G. so (a(
ft- = i. - [F
This used in many existing works DMs (Song et al., 2020b; Dockhorn al., 2021). This approach however has low accuracy and is sometimes unstable when the stepsize is not sufficiently small. To improve the accuracy, we propose to use the Exponential Integrator (EI), a method that leverages the semilinear structure of Eq. (5), When applied to Eq. (5), the El reads
T; = (t - At, t) + 1f,
1 - - vt(( - At, GG, G.drr)sp 2
(8)
It is effective the nonlinear term Sy (#t, t) does not change much along the solution. In fact, for any given At, Eq. (8) solves Eq. (5) exactly if so 1) is constant over the time interval [t - At, (]-
To compare the EI Eq. (8) and the Euler method Eq. (7), we plot in Fig. 3a the average pixel difference Ap between the ground truth 2 and the numerical solution 2 obtained by these two methods for various number N of steps. Surprisingly, the El method performs worse than the Euler method.
This observation suggests that there are other major factors that contribute to the error Ap. In particular, the condition that the nonlinear term s @ t) does not change much along the solu- tion assumed for the El method does not hold. To see this, we plot the score approximation error (T( = [0 Fig. 3b2. can be seen that the approximation error grows rapidly as approaches 0. This is not strange; the score of realistic data distribution V log (xx should change rapidly as + (Dock-
(æ,, 7) so (æt, f)l1,, T
t), along the exact solution (Φ;) } to Eq. (5) in
horn et al., 2021).
'The } are approximated by solving ODE with high accuracy solvers and sufficiently small step size. For better visualization, we have removed the time discretization points in Fig. 3b and Fig. 3d, since A, 0 these points and becomes negative infinity in log scale.
5

---

Published as a conference paper at ICLR 2023
Ingredient 2: cø(2, t) over s9(2, t). The issues caused by rapidly changing score V Flogpo (z) do not only exist in sampling, but also appear in the training of DMs. To address these issues, a different parameterization of the score network is used. In particular, it is found that the parameterization (Ho et al., 2020) V log Pr (æ) & - '0 = t), where L, can be any matrix satisfying L 2x, leads to significant improvements of accuracy. The rationale of this parameterization is based on a reformulation of the training loss Eq. (3) as (Ho et al., 2020)
(()) = (oo [ E0(u4 + Lie, t) {),
with A, L, The network eq tries to follow which sampled from standard Gaussian and thus has a small magnitude. In comparison, the parameterization 89 -L, eg can take large value as L, + 0 as approaches O. It is thus better to approximate than 8o with a neural network.
A,
We adopt this parameterization and rewrite Eq. (5)
dæ
F₂ G. co,co(8,1))
(10)
dt
Applying the El to Eq. (10) yields
- A = 10(L - A1, 1) + 1// A-
V/L -
(11)
2
Compared with Eq. (8), Eq. (11) employs - - t) instead of sp x2,t) -L, co (x,, t) TT
ca(
to approximate the score sp(x,, over the time interval T E [t - At, This modification from L, to L.- turns out to be crucial; the coefficient L, changes rapidly over time. This is verified by Fig. 3d where we plot the score approximation error A, when the parameterization ea used, from which we see the error A, is greatly reduced compared with Fig. 3b. With this modification, the El method significantly outperforms the Euler method as shown in Fig. 3c. Next we develop several fast sampling algorithms, all coined as the Diffusion Exponential Integrator Sampler (DEIS), based on Eq. (11), for DMs.
Interestingly, the discretization Eq. (11) based on EI coincides with the popular deterministic DDIM when the forward diffusion Eq. (1) is VPSDE (Song et al., 2020a) as summarized below (Proof in App. E).
Proposition 2. When the forward diffusion Eq. (1) is set to be VPSDE (Fr, G. are specified in Tab. 1), the El discretization Eq. (11) becomes
#1-11
O dt 2.
1/1 111-10--- 04-11- 01-At
VI (2),)
(12)
at
at
which coincides with the deterministic DDIM sampling algorithm.
Our result provides an alternative justification for the efficacy of DDIM for VPSDE from a numerical discretization point of view. Unlike DDIM, our method Eq. (11) can be applied to any diffusion SDEs to improve the efficiency and accuracy of discretizations.
In the discretization Eq. (11), we use ea ( c t) to approximate ea ((, r) for all T [t- t). which a zero order approximation. Comparing Eq. (11) and Eq. (6) we see that this approximation error largely determines the accuracy of discretization. One natural question to ask is whether it is possible use better approximation of co (ă,, r) to further improve the accuracy? We answer this question affirmatively below with an improved algorithm.
Ingredient 3: Polynomial extrapolation of cg. Before presenting our algorithm, we investigate how en <0(x, () evolves along a ground truth solution { from T to / O. We plot the relative change 2-norm of 6o (ar,, I) in Fig. 4a. It reveals that for most time instances the relative change small. This motivates us to previous (backward) evaluations of Eq up t to extrapolate cø(x,, T) for r E [t --- At, t).
Inspired by the high-order polynomial extrapolation in linear multistep methods, we propose to use high-order polynomial extrapolation of 6g in our EI method. To this end, consider time discretization (ta 1. where to = 0, tN T. For each i, we fit polynomial (t(()) of degree with respect to the interpolation points (t++ co ( t++g)). 0 I j S r. This polynomial P. (t) has explicit expression

---

Published as a conference paper at ICLR 2023
relative change Eg
G:30
r-0
K2
16-
1=1
2
1.
10
1=3
1.00
x
(a)
(b)
(c)
Figure 4: Fig. 4a shows relative changes of co t) with respect to t are relatively small, especially
when & > 0.15. Fig. 4b depicts the extrapolation enror with N = 10. High order polynomial can
reduce approximation error effectively. Fig. 4c illustrates effects of extrapolation. When is small, higher order polynomial approximation leads to better samples.
Pt(t)
-,( EI1
Litk Lith
|co ( t(t5).
(13)
j=0 Keg 1ا-
We then use (t) to approximate eø cg(x,, over the interval [f,-1, ty]. For > N - r. we need to polynomials of lower order to approximate 6g. To the advantages of this approximation,
plot the approximate error (t) = [2; t) Pr (t)ll2 of co(æ,, t) by P, (t) along ground truth trajectories {æ:} in Fig. 4b. It can be seen that higher order polynomials can reduce approximation error compared with the case = O which uses zero order approximation as in Eq. (11).
zero order approximation of the score in Eq. (6), the update step of order is obtained by plugging the polynomial approximation Eq. (13) into Eq. (6). It can be written explicitly as
As in the El method Eq. (11) that
+
Σ Circc ts+i)l 3=0
(14)
Cij Cy- di- 1
()
T titk 1dr.
(15)
2
to-s t titk
kij
We remark that the update in Eq. (14) is linear combination of Si, and co (B li4g), where the weights (44-1,44) and Cij are calculated once for a given forward diffusion Eq. (1) and time discretization, and be reused across batches. For some diffusion Eq. (1), (144-1, 4). C.₃ have closed form expression. Even if analytic formulas are not available, one can use high accuracy solver obtain these coefficients. In DMs (e.g.. VPSDE and VESDE), Eq. (15) are normally 1- dimensional or 2-dimensional integrations and are thus easy to evaluate numerically. This approach resembles the classical Adams Bashforth (Hochbruck & Ostermann, 2010) method, thus we term tAB-DEIS. Here we use é to differentiate it from other DEIS algorithms we present later in Sec. based on a time-scaled ODE.
The (AB-DEIS algorithm is summarized in Algo 1. Note that the deterministic DDIM is special case of tAB-DER for VPSDE with r = 0. The polynomial approximation used in DEIS improves the sampling quality significantly when sampling steps N is small, shown in Fig. 4c.
4 EXPONENTIAL INTEGRATOR: SIMPLIFY PROBABILITY FLOW ODE
Next we present a different perspective to DEIS based on ODE transformations. The probability ODE Eq. (10) can be transformed into simple non-stiff ODE, and then off-the-shelf ODE solvers can he applied to solve the ODE effectively. To this end, we introduce variable ÿ, == (t,, 0) and rewrite Eq. (10) into
dÿ - VG,GG,GG en (VO,
(16)
di Note that, departing from Eq. (10), Eq. (16) does not possess semi-linear structure. Thus, we can apply off-the-shelf ODE solvers to Eq. (16) without accounting for the linear structure in al- gorithm design. This transformation Eq. (16) can be further improved by taking into account the
7

---

Published as a conference paper at ICLR 2023
Algorithm 1 tAB-DEIS
CIFAR1O FID
Input: i= r
Instantiate: 인tx Empty e-buffer
50
Calculate weights w, C' based on Eq. (15) for i in N, - 1...... do <-bufferappend(ce(@e,, t.)) end for
El
paramsterization
FID
+ Polynomial extropelation
10
IK45
Dt, < Eq. (14) with V, C, e-buffe
SDE Adaptiva
10 20:00 50 100
2000
NFE
Figure 5: Ablation study and comparison with other samplers. We notice switching from Eu- ler to Exponential Integrator worsens FID, which we explore and explain Ingredient 2 in Sec. 3. With EI, cg. polynomial extrapolation and optimizing timestamps can significantly improve the sam- pling quality. Compared with other samplers, ODE sampler based on RK45 (Song al., 2020b), SDE samplers based on Euler-Maruyy Murer-Maruyamaa (EM) (Song et al., 2020b) and SDE adaptive step size et al., 2021), DEIS can converge much faster.
solver Jolicoeur-Martingg
analytical form of L, G.,, L, Here present treatment for VPSDE; the results be extended to other (scalar) DMs such as VESDE.
Proposition For the VPSDE, with D: Vau /ant ir and the time-scaling 3(1) /ool V 1-0L
Eq. (10) can be transformed into
80
dý - 0.9 (p)), P (())).
col
(17)
dp
C0
After transformation, the ODE becomes black-box ODE that be solved by generic ODE solvers efficiently since the stiffness caused by the semi-linear structure is removed. This is the core idea of the variants of DEIS present next.
Based on the transformed ODE Eq. (17) and the above discussions, we propose two variants of the DEIS algorithm: şRK-DEIS when applying classical RK methods, and pAB-DEIS when applying Adams-Bashforth methods. We remark that the difference between (AB-DEIS and PAB-DEIS lies in the fact that tAB-DEIS fits polynomials in which may not be polynomials in p. Thanks to simplified ODEs, DEIS enjoys the convergence order guarantee as its underlying RK or AB solvers.
5 EXPERIMENTS
Abalation study: As shown in Fig. 5, ingredients introduced in Sec. 3.2 can significantly improve sampling efficiency on CIFAR1O. Besides, DEIS outperforms standard samplers by large margin.
DEIS variants: We include performance evaluations of various DEIS with VPSDE on CIFARIO in Tab. 2, including DDIM, pRK-DEIS, PAB-DEIS and tAB-DEIS. For pRK-DEIS, find Heun's method works best among second-order RK methods, denoted as p2Heun, Kutta method for third order, denoted as p3Kutta, and classic fourth-order RK denoted p4RK. For hdah-Brshforth methods, we consider fitting 1, 2,3 order polynomial in t, denoted as tAB and pAB respectively. We observe that almost all DEIS algorithms can generate high-fideltyy images with small NFE. Also, note that DEIS with high-order polynomial approximation can significantly outperform DDIM; the latter coincides with zero-order polynomial approximation. We also find the performance of high order dRK-DEIS is not satisfying when NFE is small but competitive as NFE increases. It is within ex expectation as high order methods enjoy smaller local truncation error and total accumulated when small step size is used and the advantage is vanishing as reduce the number of steps.
More comparisons: We conduct more comparisons with popular sampler for DMs, including DDPM, DDIM. PNDM (Liu et al., 2021), A-DDIM (Bao et al., 2022), FastDPM (Kong & Ping, 2021), and Ito Taylor (Tachibana et al., 2021). We further propose Improved PNDM (iPNDM) that avoids the expensive warming start, which leads to better empirical performance. We conduct
8

---

Published as a conference paper at ICLR 2023
SNPE
3.NN
6NFE
NFF
5NFE
SONFE
ZONFE
16NFE
10NFE
ONFE
100NFE
MON
3ONFE
ZUNFE
12NFE
DDIM
22Heun
AAB
1AB
Figure 6: Generated samples of DDIM and DEIS with unconditional 256 x 256 ImageNet pretrained model (Dhariwal Nichol, 2021)
FID for various DEIS
NFE
DDIM
p2Heunt
p3Kutta
p4RK
pAB1
pAB2
pAB3
(AB1
IAB2
1AB3
26.91
108+1
185+1
193+3
22.28
21.53
21.43
19.72
16.31
15.37
10
11.14
14.72
13.19
28.65 28.65+2
7.56
6.72
6.50
6.09
4.57
4.17
7.06
4.89*1
5.88
6.88+1
4.69
4.16
3.99
4.29
3.57
3.37
20
5.47
3.50
2.97+1
3.92
3.70
3.32
3.17
3.54
3.05
2.86
3.27
2.60
2,55
2.57+2
2.70
2.62
2.59
2.67
2.59
2.57
Table 2: More results of DEIS for VPSDE on CIFAR10 with limited NFE. For PRK-DEIS, the upper right number indicates extra NFEs used. Bold numbers denote the best performance achieved with similar NFE budgets. For fair comparison, discretization different algorithms with different NFE. We include a comparison given the same time discretization in App. H.3. t: The concurrent work (Karras et al., 2022) applies Heun method to rescaled DM. This is a special case of p2Heun (More discussion included in App. B).
report numbers based on their best time
comparison on image datasets, including 64 x 64 CelebA (Liu et al., 2015) with pre-trained model from Song et al. (2020a), class-conditioned 64 x 64 ImageNet (Deng et al., 2009) with pre-trained model (Dhariwal & Nichol, 2021), 256 x 256 LSUN Bedroom (Yu et al., 2015) with pre-trained Fig. 7 quanti- tatively, and show empirical samples in Fig. 6. More implementation details, the performance of various DMs, and many more qualitative experiments are included in App. H.
model (Dhariwal & Nichol, 2021). We compare DEIS with selected baselines
CelebA x
ImageNet x
Bedroom 256 x 256
p10mm Dore
BDM
ABOI
Ns1DPM
10.
In-3yter
20
20
1/1
300
100
13
20
NFE
NFE
NFE
Figure 7: Sample quality measured by FID of different sampling algorithms with pre-trained DMs.
6 CONCLUSION
In this work, we consider fast sampling problems for DMs. We present the diffusion exponential integrator sampler (DEIS), a fast sampling algorithm for DMs based on a novel discretization scheme of the backward diffusion process. In addition to its theoretical elegance, DEIS also works efficiently in practice; it is able to generate high-fidelity samples with less than 10 NFEs. Exploring better extrapolation may further improve sampling quality. More discussions included in App. B.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like thank the anonymous reviewers for their useful comments. This work is partially supported by NSF ECCS-1942523 and NSF CCF-2008513.
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Process. Appl., 12(3): 313-326, May 1982. ISSN 0304-4149, doi: 10. 121101603044-1499087700051333111 URL https: //doi. org/ 10. 10..1111000044111111111111
(82) 90051-5.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffussion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-1777 -17993, 2021.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytie-DPM: An Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models. 2022. URL http://arxiv: org/abs/ htt.aa//............... 06503.
Tianrong Chen, Guan Horng Liu, and Evangelos A Theodorou. Likelihood training of schrödinger bridge using forward-backward backward sdes theory. arXiv preprint urXiv:2110.1122 2021a.
Yongxin Chen, Tryphon T Georgiou, and Michele Pavon. Stochastic control liaisons: Richard sinkhor meets gaspard monge on a schrodinger bridge. SIAM Review, 63(2):249 2021b.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35 (1):53-65) 65, 2018.
Valentin De Bortoli, James Thornion, Jeremy Heng, and Arnad Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. 2021. URL 05233.
http:/ http://wwwwwv..g......///
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score- Generative Modeling with Critically- Damped Langevin Diffusson. pp. 1-13, 2021. URL http://arxiv.orr/aalt 07068.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
David Francis Griffiths and Desmond J Higham. Numerical methods for ordinary differential equations: initial value problems, volume 5. Springer, 2010.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 2020-Decem, 2020. ISBN 2006. I 1239v2. URL https:://// /ttp:/gitttt com/ ho jonathanho/ difffsion..
Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209-288 2010.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint arXiv:2209.05557, 2022.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34, 2021.
10

---

Published as a conference paper at ICLR 2023
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res., 6:695-708, 2005. ISSN 15337928.
Alexia JolicoeurMaartineau, Ke Li, Rémi Piché-Taillefer. Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.00364, 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips:
Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34, 2021.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527, 2021.
Diederik Kingma and Max Welling. An introduction to variational autoencoders. arXiv preprint 1rX.v:1906.02691, 2019.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132, 2021.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. 2021. URL com/ (lageassslc/Faaee
pytorchtt //axivv.///iiv/////a//aa/ http://bb/2110.00.0 04775.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo Numerical Methods for Diffiusion Models on Manifolds. (2021):1-22 2022. URL http::://axxvv...oope abs 2202.09778.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927) 2022.
Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan, and Dahua Lin. A conditional point diffusion-refinementt paradigm for 3d point cloud completion. arXiv preprint arXiv:2112.03530, 2021.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. ArXiv, abs/2102.09672, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE; Towards Photorcalistic Image Generation and Editing with Text-Guided Diffusion Models. 2021. URL http: waw.iv. htwpiiiorrr//b... 220.
Robert Pless and Richard Souvenir. survey of manifold learning for images. IPSJ Transactions Computer Vision and Applications, 1:83-94, 2009.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022.
Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissi- pation. arXiv preprint 2rXiv:2206.1339) 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and mm Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. 2021. URL http: / /rrxiv. org//bs 2212. 10752.
11

---

Published as a conference paper at ICLR 2023
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. University Press, 2019.
Cambridge
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. 2020a.
URL hww.tttpp//x.......t org/ abs/2010 02502.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score- -osre-Based Generative Modeling through Stochastic Differential Equations. ArXiv preprint, abs/201 1346666, 2020b. URL https: / /arxivivvv...////vv//av////iiivv org/ /asg//bb//// 2011 13456.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, 34, 2021a.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005, 2021b.
Hideyuki Tachibana, Mocho Go, Muneyoshi Inahara, Yotaro Katayama, and Yotaro Watanabe. Itô- taylor sampling scheme for denoising diffusion probabilistic models using ideal derivatives. arXiv preprint arXiv:2112.133: 2021.
Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving Schrödinger bridges via maximum likelihood. Entropy, 23(9): 134, 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput., 23(7):1661-167,, July 2011. ISSN 0899-7667, 1530-888X. doi: 20.1162/neco),aa
URL
httpss:/doii org/ 11 62 www../eecooooooee... __002______11....
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul Mul- bregt. and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-277 2020. doi: 10.1038/s41599 159-0019-191-19-11 019-0686-2.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via schrödinger bridge. In International Conference on Machine Learning, pp. 10794-10804, PMLR, 2021.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sam- ple from diffusion probabilistic models. arXiv preprint arXiv:2106. 03802, 2021.
Whalen, M. Brio. and J.V. Moloney, Exponential time-differencing with embedded Runge-Kutte adaptive step control. J. Comput. Phys., 280:579-601, January 2015. ISSN 0021 0021-9991. doi: j.10016/jjcp.0014.00 0.88. URL https: dei ott./...00/./////
038.
Fisher Yu, Ari Seff. Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint www.:iv15066030.0 2015.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published as a conference paper at ICLR 2023
A MORE RELATED WORKS
lot of research has been conducted to speed up the sampling of DMs. In (Kong & Ping, 2021; Watson et al., 2021) the authors optimize denosing process by modifying the underlying stochastic process. However, such acceleration can not generate high quality samples with small number of discretization steps. In (Song et al., 2020a) the authors use a non-Markovian forward noising. The resulted algorihtm, DDIM, achieves significant acceleration than DDPMs. More recently, the authors of (Bao et al., 2022) optimize the backward Markovian process to approximate the non- Markovian forward process and get an analytic expression of optimal variance in denoising process. Another strategy make the forward diffusion nonlinear and trainable (Zhang & Chen, 2021; Vargas et al., 2021; De Bortoli et al.,, 2021; Wang et al., 2021; ;; Chen et al., 2021a) a) in the spirit of Schrödinger bridge (Chen al., 2021b). This however comes with heavy training overhead.
More closely related to our method is (Liu et al., 2022), which interprets update step in deterministic DDIM combination of gradient estimation step and transfer step. modifies high order ODE methods to provide an estimation of the gradient and uses DDIM for transfer step. However, the decomposition of DDIM into two separate components is not theoretically justified. Based on our analysis on Exponential Integrator, Liu et al. (2022) uses Exponential Integral but with Euler discretization-baan -based approximation of the nonlinear term. This approximation is inaccurate and may suffer large discretization error if the step size is large as we show in Sec.
The semilinear structure presented in probability flow ODE has been widely investigated in physics and numerical simulation (Hochbruck & Ostermann, 2010: Whalen et al., 2015), from which we get inspirations. The stiff property of the ODEs requires more efficient ODE solvers instead of black- box solvers that are designed for general ODE problems. In this work, we investigate sovlers for differential equations in diffusion model and take advantage of the semilinear structure.
B DISCUSSIONS
1. Q Can DEIS help accelerate the likelihood evaluation of diffusiion models?
Theoretically, our methods can be used in likelihood evaluation as DEIS only changes nu- merical discretization. Practically, we can use pRK-DEL with Eq. (16) and Prop 3 to accelerate likelihood evaluation. We find NLL evaluation based on RK can converge with 36 NFE with order Kutta solver, which reaches 3.16 bits/dim compared with 3.15 bits/dim for RK45 (Song et al., 2020b) and achieves around 4 times acceleration.
Q Can the proposed method further be accelerated by designing an adaptive step size solver?
A The proposed pRK-DEIS can be combined with out-of-shelf adaptive step size solvers. However, we find that most ODE trajectories resulting from various starting points share similar patterns in curvature, and a tuned fixed step size works efficiently. Most existing adaptive step size strategies have some probability of getting rejected for the proposed step size, which will waste the NFE budget. Take the example of RK45, one rejection will waste 5 NFE. which is unacceptable when we try to generate samples in 10 NFE or even fewer steps.
Q - The proposed AB-DEIS and iPNDM use lower-order multistep solvers for computing the initial solution. Do they have convergence guarantee?
We use lower-order multistep for the first few steps to save computational costs. The strategy can help achieve similar sampling quality with less NFE we show in Tabs. and 5, which aligns with our goal of sampling with small NFE. Moreover, lower order Adams- Bashforth methods also enjoy a convergence guarantee, albeit with slower rate.
-
4. Q How is DEIS compared with the ODE sampling algorithm in Karras et al. (2022)?
13

---

Published as a conference paper at ICLR 2023
A - We note Karras et al. (2022) is concurrent work that introduces second-order Heun method in a rescaled ODE. The algorithm is a special case of pRK-DEIS with the second-ord -oderr Heun RK method. Below we show the equivalence. As the two works use different sets of notations, we use blue for notations from Karras et al. (2022) and orange for our notations.
Karras et al. (2022, Algorithm 1) investigates the diffusion model with forward process æ N (t) a(t)?), where (t)) is a scaling factor and att)² represents the variance. Karras et al. (2022, Sec 4) suggests the schedule s(t) (t) = I, which has the diffusion ODE
dæ
(18)
dt
where Dax, i) is trained to predict clean data given noise data æ at time t. They employ the second-order Huen method solve Eq. (18). Additionally, they show all isotropic diffusion models with arbitrary s(t) can be transformed into the suggested diffusion model with parameter schedule s(t) (t) t by proper rescaling. The rescaling in Karras et al. (2022) is equivalent to change-of ooange----variabless we introduce in Sec. 4, and Eq. (18) is the simplified ODE Eq. (17) we used that takes into account the analytical form of V Gt, Lt.
To further illustrate the point, consider the example with the popular VPSDE in Prop 3. In this case, the pRK-DEIS uses the time rescaling p(t) 1-as and the state rescaling ÿ
I de
(note 0g
in VPSDE). The forward process for úp becomes
a2 - ((ou. p²),
- = (e(p) N(Oo:
(19)
where t(p) is the inverse function of p(t) and the last equality holds due to c.g 9o- Com- paring Eq. (19) and the parameter schedule s(t) = (() = t in Karras et al. (2022), we conclude that d is equivalent to x, and is the same t. Morcover, =-D(w.)) is equivalent to
cal Ga- ÿ, B (p)) since both predict added white noise from noised data.
In summary, Karras et al. (2022, Algorithm 1) is special case of pRK-DEIS, which can be obtained by employing second-order Heun method Eq. (17). We include the empirical com- parison between other DEIS algorithms and Karras et al. (2022, Algorithm 1), which we denote as p2Heun. We find with relatively large NFE, third-order Kutta is better than second-order Heun. And tAB-DEIS outperforms pRK-DEIS when NFE is small.
5. Q How is DEIS compared with sampling algorithm in Lu et al. (2022)?
A - We note DPM-Solver (Lu et al., 2022) is concurrent work and it also uses the expo- nential integrator to reduce discretization error during sampling. Both start with the exact ODE solution but are different at discretization methods for nonlincar score parts. Below show the connections and differences. As the two works use different sets of notations, we use cyan for notations from Lu et al. (2022) and orange for our notations.
Exact ODE solution Lu et al. (2022) investigate diffusion model with forward noising ay <o M(araa, of). Lu et al. (2022, Proposition 3.1) propose the exact solution of ODE of æ; given initial value w. at time s 2 0
d e- (0 A)dd,
C.
(20)
where X log 0r is known as one half of log-SNR (a.ka. signal-to-noise-ratio and de (xxx,, A) = ca(,,, t) with corresponding given X. Similar to exponential Runge-Kutta
method (Hochbruck & Ostermann, 2010), Lu et al. (2022) approximate S Thee-n e->eo(ax, XjdA based on Taylor expansion and propose DPM-Solvers.
Eq. (20) shares lot of similarities with PKK-DEIS. Specifically, p(() = e X(t) since
for = det and VI 11-ar oe in VPSDE. Similar to Eq. (20), the exact solution
14

---

Published as a conference paper at ICLR 2023
in Eq. (17) follows
+ Vos I"
2.
é(E p)dp,
(21)
where p0(pp p) = eø (2t, b) with corresponding t given p. pRK-DEIS employs out-of-shelf'
Bunge-Kutta solvers for fp. p( p)dp.
An example of DPM-Solver2 To illustrate the connection and difference clearly, consider DPM-Solver-2 and pR pRK-DEIS with the standard middle point solver and compare their update schemes. To compare these two algorithms, we first introduce function FDDIM inspired by DDIM. In pRK-DEIS and DPM-Solver, FDDIM is defined as
Ot +
FDDMM 9, 8,1)
V
Us
[ 04
aslg
(22)
at Us + [ar(e" 1)]g
FDDM (.g,,,t)
where
de Ao
(23)
respectively.
With FDDIM. reformulate update schemes of DPMM-Sllve22 and pRK-DEIS with midpoint solver into Algo and 3. The two algorithms are only different the choice of midpoint 8; and 8t. In particular, 8; VPPPi4I.
Connection with Runge-Kutta Though both algorithms are inspired by El methods and Kutte, they are actually different even when there is no semi-linear structure diffusion flow ODE. Let us consider VESDE introduced in Karras et al. (2022) where = 1, = t. The VESDE has a simple ODE formulation,
dx ca (xx t)dt.
(24)
Eq. (24) does not have semi-linear structure. In this case, pRK-DEIS reduces to standard Runge-Kutt Kutta methods and has convergence order O for r-oodder RK methods. The DPM- solver uses the parametrization = log(!), and is different from standard Runge Kutta and reformulate Eq. (24) as
da = - ca (A))dd.
(25)
For K order DPMvr,lver, it has convergence order ((A))) under certain assumptions stated in Lu et al. (2022).
Empirical comparison We compare DoM-er,, DoM-e5, tAB-DEIS, and pRK- DEIS 64 64 class-conditioned ImageNet. We observe LAB-DEIS has the best sample quality most of time. We believe it is because multistep better than single-step methods when we have limited NFEs c.g.. 6. DPM-Solvers are better than pRK-DEIS in small NFE regions and the difference shrinks fastly as we increase sampling steps. We hypothesize that this is because DPM-Solvers are tailored for sampling with small NFEs. However, tRK-DEIS has slightly better FID when NFE is relatively large, although the difference in performance is small. The observation aligns with our experiments in CIFAR10, third order şRK-DEIS achieves 2.56 with 51 NFE while the third order DPM-Solver achieves 2.65 with 48 NFE (Lu et al., 2022). We include more visual comparison Figs. and
Algorithm pRK-DEIS with midpoint solver
Algorithm 2 2
1: Input: K;, 4G, li-1 and corresponding
1: Input: Bi, 11,4,44-1 and corresponding
Aey di-1
2: Output: I6-1
Pe-Pi-1
2: Output: Ø;-1
3: 8; tx( Ar
3: 8; ip (Pstpi-1
4: = eo 60(2;,t E
4: = co(æ;, ti)
5: u, FODIM Fzs g. 8;)
5: u; FDDIM (2i g t4, 8:)
6: = co (s)
6: g Eø(ui, si)
7: -;-1 1 FDDIM( g, 8i, 14-1)
7: H; 1 FDDIM(U, g. 8,, t.-1)
15

---

Published as a conference paper at ICLR 2023
6NFE
12NFE
18 N8NFE
24NFE
SONFE
pMid DPM-Solver2 pKutta
DPM-Solver3
tAB
pAB
Figure& 8: DPM-Solver DEIS with unconditional 256 x 256 ImageNet pretrained model (Dhari-
wal & Nichol, 2021). (Zoom in to see details)
10
12
14
16
18
20
30
50
tAB
6.65
3.99
3.67
3.49
3.21
3.10
2.81
2.69
pAB
9,28
6,46
5.02
4.34
3.74
3.39
2.87
2.66
DPM-Solver2
7.93
5.36
4.46
3.89
3.63
3.42
3.00
2.82
pMid
9.12
6.78
5.27
4.52
4.00
3.66
2.99
2.81
DPM-Solver3
5.02
3.62
3.18
3.06
2.84
2.72+
KKutta
13.12
5.18
3.63
3.16
2.82
2.71 +
Table 3: Comparison between DEIS and DPM-Solver Lu et al. (2022) ImageNet. The upper right + indicates one extra NFE used.
class-conditioned 64 x 64
16

---

Published as a conference paper at ICLR 2023
6NFE
NNF
8NFE
9NFE
10NFE
12NFE
12NFE
15NFE
14NFE
18 NFF
30NFE
30NFE
DPM-Solver2 tAB-DEIS
DPM-Solver3
Figure 9: DPM-Solver v.s. DEIS with unconditional 256 x 256 ImageNet pretrained model (Dhari-
wal & Nichol, 2021). (Zoom in to see details)
17

---

Published as a conference paper at ICLR 2023
The ODE solvers are sensitive to step size choice. Different works suggest different time discretization (Lu et al., 2022; Karras et al., 2022: Song et al., 2020a). How do compared algorithm and DEIS perform under different step size scheduling?
A The comparison given the same time discretization is included in App. H.3. We find differ- ent algorithms may prefer different time discretization. We provide a comparison for different sampling algorithms under their best time scheduling in Tab. 2. In most cases especially low NFE region, we find LAB-DEIS performs better than other approaches.
7. Q Can DEIS be generalized to accelerate SDE sampling for diffusion models?
A Some techniques developed in DEIS, such better parameterization and analytic treatment linear term, can be applied to SDE counterparts. However, SDE is more difficult to accelerate compared with ODE. We include more discussions in App. C.
C DISCRETIZATION ERROR OF SDE SAMPLING
In this section, consider the problem of solving the SDE Eq. (4) with À >> 0. As shown in Prop 1, this would also lead to a sampling scheme from DMs. The exact solution to Eq. (4) satisfies
Weight
1 x2 L. col + f A (L, G,dun / (t. T)
#; (1,, +
(26)
Linear term
Noise term
Nonlinear
where V is as before. The goal is to approximate Eq. (26) through discretization. Interestingly, the stochastic DDIM (Song et al., 2020a) turns out to be a numerical solver for Eq. (26) as follows (Proof in App. G).
Proposition 4. For the VPSDE, the stochastic DDIM is discretization scheme of Eq. (26).
How do we discretize Eq. (26) for general SDE Eq. (4)? One strategy is to follow what we did for the ODE (A = 0) in Sec. 3.2 and approximate eg (0, T) by a polynomial. However, we found this strategy does not work well in practice. We believe it is due to several possible reasons as follows. We do not pursue the discretization of the SDE Eq. (4) further in this paper and leave for future.
Nonlinear weight and discretization error. In Eq. (26), the linear and noise terms can be calculated exactly without discretizaiton error. Thus, only the nonlinear term E9 can induce eиrоr in the EI method. Compared with Eq. (11), Eq. (26) has larger weight for the nonlinearity term X > and is therefore more likely to cause larger errors. From this perspective, the ODE with À = 0 the best option since it minimizes the weight of nonlinear term. In Song al. (2020a), the authors also observed that the deterministic DDIM outperforms stochastic DDIM. Such observation is consistent with our analysis. Besides, we notice that the nonlinear weight VPSDE is significantly smaller than that in VESDE, which implies VPSDE has smaller discretization eтrоr. Indeed, empirically, VPSDE has much better sampling performance when N small. Additional noise. Compared with Eq. (11) for ODEs, Eq. (26) injects additional noise to the state when it is simulated backward. Thus, to generate new samples by denoising, the score model needs to not only remove noise in Øzy- but also remove this injected noise. For this reason, better approximation of ea may be needed.
D PROOF OF PROP
The proof is inspired by (Zhang & Chen, 2021). We show that the marginal distribution induced by Eq. (4) does not depend on the choice of and equals the marginal distribution induced by Eq. (2) when the score model is perfect.
Consider the distribution q induced by the SDE
1 X2
dx
F2x
G.G.T log (x))dt + GG.dw.
(27)
2
18

---

Published as a conference paper at ICLR 2023
Eq. (27) is simulated from t = T to = 0. According to the Lokker-Planck Kolmogorov (FPK)
Equation, solves the partial differential equation
(q)()) V [[Fxx
1+ X²
X²
8
G. v logg(x))a(2))) - G.
dr,du, (x)))
01
2
V. ([F.a X G. dr,dx, (())))
-
1
G. V log (log
{| (1~ G.G. V lo (
X²
2
where V. dedottss the divergence operator. Since
X² X2 G.G. V log (x)))(((())) v 1
X2 ( 2 G.G.
8²
dr,0r, (x)))) (28)
we obtain
(q. V. {1F G.G. log de (1))) 01
(29)
Eq. (29) shows that the above partial differential equation does not depend on A. Thus, the marginal distribution of Eq. (27) is independent of the value of X.
PROOF OF PROP 2
Thanks to, A straightforward calculation based on Eq. (6) gives that (t, for the VPSDE
at
(t s) = V a.
follows that
S. " G.
1
ai dlog ar
dr
dr
2
ax dr v1 - ax
1
da,
=Vai
2 a
a0, 00.5
=Var
VI
04
Olg
a22
Setting / 6- 1 A + we write Eq. (11)
01-At az
Et + I
04-A1
(4-At
V1
co. t).
a1
F PROOF OF PROP 3
We start our proof with Eq. (16), In VPSDE, Eq. (16) reduce to
dø
Oig log at
(O(V(O,))
(30)
dt 2 2\ dt V1-
Now we consider rescaled time p. which satisfies the following equation
dp 1
ao d log as.
(31)
dt
2V dt
vI Cz.
19

---

Published as a conference paper at ICLR 2023
Plugging Eq. (31) into Eq. (30), we reach
dÿ dp
en (V(00))..))
(32)
In VPSDE, we ar is monotonically decreasing function with respect to l. Therefore, there exists bijective mapping between P and based Eq. (31), which define B and p B(1).
Furthermore, we can solve Eq. (31) for |
1 at 0tt
((L) vaol
(33)
G
PROOF OF PROP 4
Our derivation uses the notations in (Song al., 2020a). The DDIM employs the update step
Tt-At Var-Sil v1 (x10((
+
V ai-Ai
172
- 04-
(1
at
)eo
Vaa
1 022
0r-Ar
1 04- (1
Ot
"V
)C2,
(34)
aq
at-
where 1) is a hyperparameter and 17 E [0, 1). When 0. Eq. (34) becomes determinstic and reduces to Eq. (12). We show that Eq. (34) is equivalent to Eq. (4) when 17 X and At -> O.
By Eq. (34), σi-Ai ~ (M,, I) where
2t v1 - ancola t) + V
1 - a1-A1 (1 Jea(x:: 1) 1 04 At 112
My = Von-Ai
Vax
az
04-At
o 1/²
-ac- (1
ai
1 1- az
ar-
It follows that
ai A1
My
lim
Ft.
At-0 At)
At
v Va1-AN
1-6x
a1
a1-Ai
172
1-a1
1 l-a:
21 (1
as
co c0(211)) 1)
At.
d dllog aq
1 122 doo an
1
r +
2
dt
dt
VI- at
1 1+
Fæ +
G.G. L, -T (x)
and
1-00-01 (1 2 1
dlogga at
lim dt
as 2l
G
di
Consequently, the continuous limit of Eq. (34) is
dæ = [Fxx + 1 n?
G.G. Li ce(xxt))dd - GGdw.
(35)
2
which is exactly Eq. (4) if X.
20

---

Published as a conference paper at ICLR 2023
H MORE EXPERIMENT DETAILS
H.1 IMPORTANT TECHNICAL DETAILS AND MODIFICATIONS
In Sec. 3, the ground-truth solutions { are approximated by solving ODE with high accuracy solvers and small step size. We empirically find solutions of RK4 converge when step size smaller than 10 VPSDE. We approximated ground-truth solutions by RK4 solutions with step size x 10-3
It is found that correcting steps and extra denoising step can improve image quality at additional NFE costs (Song et al., 2020b; comparison, disable the correcting steps, extra denoising step, or other heuristic clipping tricks for all methods and experiments this work unless stated otherwise.
et al., 2021). For a fair
Due to numerical issues, we set ending time to in DMs during sampling a non-zero number. Song al. (2020b) suggests to 10 for VPSDE and to 10 for VESDE. In practice, we find the value of lo and time scheduling have huge impacts on FIDs. This finding is not new and has been pointed out by existing works Kim et al.. 2021: Song et al., 2020a). Interestingly, we found different algorithms have different preferences
et al., 2021;
to and time scheduling. We report the best FIDs for each method among different choices of to and time scheduling in Tab. 2. We use to suggested by the original paper and codebase for different checkpoints and quadratic time scheduling suggested by Song et al. (2020a) unless stated otherwise. We include a comprehensive study about to and time scheduling in App. H.3
Because PNDM needs 12 NFE for the first 3 steps, we compare PNDM only when NFE is great than 12. However, our proposed iPNDM can work when NFE is less than 12.
We include the comparison against A-DDIM (Bao al., 2022) with its official checkpoints and implementation in App. H.5.
We only provide qualitative results for text-to-image experiment with pre-trained model (Ramesh al., 2022).
We include proposed r-th order iPNDM in App. H.2. We use r 3 by default unless stated otherwise.
H.2 IMPROVED PNDM
By Eq. (11), PNDM can be viewed as a combination of Exponential Integrator and linear multistep method based on the Euler method. More specifically, uses a linear combination of multiple score evaluations instead of using only the latest score evaluation. PNDM follows the steps
(3)
1
(55c, - 59E++ + 3777422 9c(+33)),
(36)
24
AF
01-A1
tt
11 14-11
a1- VI Vi-a orle)
(37)
Og
02
where e = co t). c+ = col + At). The coefficients Eq. (36) are derived based on black-box ODE Fuler discretization with fixed step size. Similarly, there exist lower order approxi- mations
(0)
= Et
(38)
(1)
3
1
Cu+Ai
(39)
2
2
(2) 1
(23G, 16E,- + 55(+2At).
(40)
12
Originally, PNDM uses Runge-Kutta for warming start and costs 4 score network evaluation for each the first 3 steps. To reduce the NEE in sampling, the improved PNDM (iPNDM) uses lower order multistep for warming start. We summarize iPNDM in Algo 4. We include comparison with tAB-DEIS Tabs. 4 and 5, we adapt uniform step size for tAB-DEIS when NFE=50 in CIFAR10 find its performance is slightly better than the quadratic one.
21

---

Published as a conference paper at ICLR 2023
Algorithm 4 Improved PNDM (iPNDM)
Input: {4:)==0- = i order Instantiate: y Empty e-buffer
for N, -
do
min(N -+
<-bufier.ppeed(ee@@... 1;)) Simulate e(,) based and e-buffer
it,- - Simulate Eq. (37) with ãt, and 2(5) t/2
end for
FID NFE
5
20
50
Method
PNDM
6.42
3.03
iPNDM
70.07
9.36
4.21
3.00
DDIM
30.64
11.71
6.12
4.25
IABI
20.01
6.09
3.81
3.32
IAB
16.53
4.57
3.41
3.09
tAB3
16.10
4.17
3.33
2.99
Table 4: PNDM and iPNDDM CIFARIO
H.3 IMPACT OF to AND TIME SCHEDULING FIDS
Ingredient 4: Optimizing time discretization. From Fig. 4 we observe that the approximation error is not uniformly distributed for all to t iy when uniform discretization over time is used; the error increases approaches O. This observation implies that, instead of a uniform step size (linear timesteps). a smaller step size should be used for t close to 0 to improve accuracy. One such option is the quadratic timestep suggested in (Song et al., 2020a) that follows linspace vtN, N + 1)?.
To better understand the effects of time discretization, we investigate the difference between the ground truth átū and the numerical solution a; with the same boundary value o
3 t. [ 1 (t, (t, G,G,G (e,(T)dT, ACo(T) co(#7, P.()).
(41)
Eq. (41) shows that the difference between the solutions क and &; is weighted sum of (c) We emphasize that Eq. (41) does not only contain the approximation error of P,(T) which discussed before, but also accumulation error. Indeed, since (T( is fitted on the solution {Øt} instead of ground truth trajectory (#:)) ). there exists accumulation error caused by past errors. A good choice of time discretization should balance the approximation error and the accumulation error.
We have options for time discretization, adaptive step size, and fixed timestamps. There exists one unique ODE for DMs and we find various ODE trajectories share a similar pattern of curva-
FID NFE
5
20
50
Method
PNDM
7.60
3.51
iPNDM
59.87
7.78
5.58
3.34
O-DEIS
30.42
13.53
6.89
4.17
1-DEIS
26.65
8.81
4.33
3.19
2-DEIS
25.13
7.20
3.61
3.04
3-DEIS 25.07
6.95
3.41
2.95
Table 5: PNDM and iPNDM CELEBA
22

---

Published as a conference paper at ICLR 2023
ture empirically. And the cost of rejected steps in adaptive step size solvers is not ignorable when our NFE is small, such as 10 or even 5. Thus, we prefer and explore fixed timestamps in DEIS. We experiment with several popular options for time discretization (Salimans & Ho, 2022; Song et al., 2020a) in H.3. Surprisingly, given the different budgets of NFF, we find various samplers have different preferences for timesteps. How to design time discretization in symmetrical ap- proach is an interesting problem: we leave it for future research. In Fig. 5, we show the effects of each ingredient we introduce. With Exponential Integrator, other ingredients can consistently im- prove sampling quality in terms of FID. Compared with other sampling algorithms, DEIS enjoys significant acceleration.
We present study about sampling with difference to and time scheduling based VPSDE. We con- sider two choices of lo (10-3, 10-1) and three choices for time scheduling. The first time scheduling follows the power function in
4
N '7 p
(42)
N
the second time scheduling follows power function in
N
P: N
Po
(43)
N
and the last time scheduling follows uniform step in log p space
N i
i
log P:
log Po + N
log PN
(44)
We include the comparison between different to and time scheduling in Tabs. 6 to 8. We notice to has a huge influence image FIDs, which is also noticed and investigated across different studies (Kim et al., 2021; Dockhorn et al., 2021). Among various scheduling, we observe LAB-DEIS has obvious advantages when NFE is small and ORK-DEIS is competitive when we NFE is relatively large.
H.4 MORE ABALATION STUDY
We include more quantitative comparisons of the introduced ingredients in Tab, 9 for Fig. 5. Since ingredients ca- ca-based parameterization and polynomial extrapolation only compatible with the exponential integrator, we cannot combine them with the Euler method. We also provide perfor- mance when applying quadratic timestamp scheduling Euler Tab. 10 directly. We find sampling with small NFE and large NFE have different preferences for time schedules.
We also report the performance of the RK45 ODE solver for VPSDE CIFARIO in Tab. 11 3 As a popular and well-developed ODE solver, RK45 has decent sampling performance when NFE 2 50. However, the sampling quality with limited NFE is not satisfying. Such results are within expectation as RK45 does not take advantage of the structure information of diffusion models. The overall performance of RK45 solver is worse than iPNDM and DEIS when NFE is small.
H.5 COMPARISON WITH ANALYTIC-DDIM, (A-DDIM) (BAO AL., 2022)
We also compare our algorithm with -aalytic-DDIM (A-DDIM) in terms of fast sampling perfor- mance. We failed to reproduce the significant improvements claimed in (Bao et al., 2022) in our default CIFAR10& checkpoint. There could be two factors that contribute to this. First, we use a score network trained with continuous time loss objective and different weights (Song et al., 2020b). However, Analytic-DDIM is proposed for DDPM with discrete times and finite timestamps. Sec- ond, some tricks have huge impacts on the sampling quality in A-DDIM. For instance, A-DDIM heavily depends clipping value in the last few steps (Bao et al., 2022). A-DDIM does not provide high-quality samples without proper clipping when NEE is low.
To compare with A-DDIM, conduct another experiment with checkpoints provided by (Bao et al., 2022) and integrate iPNDM and DEIS into the provided codebase; the results are shown in Tab. 12. We use piecewise linear function to fit discrete SDE coefficients in (Bao et al., 2022) for DEIS. Without any ad-hoc tricks, the plugin-and-play iPNDM is comparable or even slightly better than A-DDIM when the NFE budget small, and DEIS is better than both of them.
We use scipy. integrate. sol sol vejvpp and tune tolerance to get different performances on different NFE, We find different combinations of absolute tolerance and relative tolerance may result in the same NEE but different FID. We report the best FID in that case.
23

---

Published as a conference paper at ICLR 2023
FID for various DEIS with in Eq. (42)
NFE
DDIM
p2Heun
p3Kutta
p4RK
pAB1
pAB2
pAB3
fAB1
tAB2
tAB3
47.59
207*1
238+1
212+3
35.14
32.51
32.02
25.99
25.06
44.29
10
16.60
84.55
66.81
78.57
10.47
8.85
8.18
9.51
7.71
7.18
15
10.39
46.36
47.45
41.27*1
6.69
5.70
5.24
6.47
5.51
5.01
20
7.93
34.87
28.35+1
27.21
5.27
4.56
4.24
5.20
4.50
4.14
50
4.36
11.58
7.00
7.48
3.32
3.08
2.99
3.32
3.09
2.99
FID for various DEIS
with
2 in Eq.
(42)
NFF
DDIM
p2Heun
p3Kutta
p4RK
pAB1
pAB2
PAB3
IAB1
IAB2
1AB3
5
30.64
256*1
357+1
342+3
24.58
23.60
23.48
20.01
16.52
16.10
10
11.71
56.62
56.51+2
103#2
7.56
6.72
6.50
6.09
4.57
4.17
15
7.67
10.62
14.96
36.15
4.93
4.40
4.26
4.29
3.57
3.37
20
6.11
6.33
4.74+1
12.81
4.16
3.84
3.77
3.81
3.41
3.33
50
4.24
3.88
3.75+1
3.78+2
3.70
3.68
3.69
3.62
3.61
3.36
FID for various
DEIS
with in Eq.
(42)
NFE
DDIM
p2Heun
P3Kutta
p4RK
pAB1
pAB2
PAB3
1AB1
LAB2
1AB3
34.07
356+1
388+1
377+3
29.80
29.35
29.38
24.87
22.57
22.06
10
14.59
115
171+2
267+2
10.73
10.16
10.11
8.11
6.36
5.97
15
9.22
32.94+1
77.44
103#1
6.45
6.03
5.98
5.21
4.26
4.05
20
7.27
13.06
11.55+1
50.56
5.17
4.83
4.78
4.45
3.88
3.75
50
4.64
3.76
3.68+1
3.74+2
3.92
3.82
3.79
3.81
3.72
3.71
FID for various DEIS with Eq. (44)
NFE
DDIM
p2Heun
p3Kutta
p4RK
PAB1
pAB2
pAB3
fAB1
tAB2
tAB3
5
54.58
216*1
335+1
313+3
49.25
48.56
48.47
37.99
28.45
26.11
10
20.03
14.72
13.19
28.65
14.05
12.63
12.18
10.36
7.03
5.71
15
11.99
5.03+1
5.88
6.88+1
7.72
6.67
6.29
6.22
4.69
4.13
20
8.92
4.12
3.97+1
4.14
5.79
5.05
4.78
4.97
4.10
3.80
50
5.05
3.67
3.75+1
3.73+2
4.01
3.84
3.79
3.89
3.74
3.72
Table 6: DEIS for VPSDE on CIFAR10 with to = 10-3
H.6 SAMPLING QUALITY ON IMAGENET 32 32
We conduct experiments ImageNet 32 32 with pre-trained VPSDE model provided in (Song et al., 2021a). Again, we observe significant improvement over DDIM and iPNDM methods when the NFE budget is low. Even with 50 NFE, DEIS is able outperform blackbox ODE solver in terms of sampling quality.
H.7 DETAILS OF EXPERIMENTS ON IMAGENET 64 x 64 AND BEDROOM 256 256
We use popular checkpoints from guided-diffusion" for our class-conditioned ImageNet 64 x and 256 x 256 L.UN bedroom experiments. Though the models are trained with discrete time, we simply treat them as continuous diffusion models. Better performance is possible we have better time discretization scheme. We adopt time scheduling with = in Eq. (43) suggested by Karras et al. (2022) with P1 0.002, PN = 80.0, which gives a better empirical performance in class-conditioned ImageNet. We also use Eq. (44) time scheduling suggested by Lu et al. (2022) and = 0.002, PN = 80.0. Better sampling quality may be obtained with different time discretization.
H.8 MORE RESULTS ON VPSDE
We include mean and standard deviation for CELEBA in Tab. 14.
hhtpps: thub. wttpbb...o/oo....a..a// dedidd-ddii di ofusiion
did-di
24

---

Published as a conference paper at ICLR 2023
FID for various DEIS with in Eq. (42)
NFE
DDIM
p2Heun
p3Kutta
p4RK
pAB1
pAB2
pAB3
fAB1
tAB2
tAB3
42.38
239+1
232+1
199+3
33.09
31.06
30.67
26.01
20.57
42.35
10
17.23
143
95.13+2
130
12.44
11.04
10.41
12.01
10.57
8.04
15
12.06
99.76*
77.37
88.56*1
9.12
8.25
7.79
9.08
8.20
7.50
20
9.71
82.89
57.54+1
66.61
7.57
6.89
6,50
7.60
6.90
6.45
50
5.76
31.56
13.10
15.73
4.64
4.25
4.06
4.67
4.28
4.10
FID for various
DEIS
with
2 in Eq.
(42)
NFF
DDIM
p2Heun
p3Kutta
p4RK
pAB1
pAB2
PAB3
IAB1
ZAB2
1AB3
5
26.91
271+1
362+1
348+3
22.28
21.53
21.43
19.72
16.31
15.37
10
11.14
66.25
63.53+2
111#2
7.65
6.89
6.67
6.74
5.49
5.02
15
7.06
13.48
17.15
44.83*1
4.69
4.16
3.99
4.38
3.78
3.50
20
5.47
6.62
4.15+1
15.14
3.70
3.32
3.17
3.57
3.19
3.03
50
3.27
2.65
2.55+1
2.57+2
2.70
2.62
2.59
2.70
2.61
2.59
FID for various
DEIS with in Eq.
(42)
NFE
DDIM
p2Heun
p3Kutta
p4RK
PAB1
pAB2
pAB3
fAB1
tAB2
tAB3
32.11
364+1
393+1
383+3
28.87
28.58
28.62
25.78
23.66
23.38
10
13.18
135
199+2
298+2
9.89
9.38
9.33
7.74
6.20
5.77
15
7.92
42.04+1
99.64
122+1
5.41
4.99
4.91
4.48
3.65
3.37
20
5.92
17.05
16.66+1
64.40
4.04
3.69
3.60
3.54
3.05
2.86
50
3.36
2.77
2.57+1
2.71+2
2.73
2.63
2.60
2.67
2.59
2.57
FID for various DEIS with Eq. (44)
NFE
DDIM
p2Heun
p3Kutta
p4RK
PAB1
pAB2
pAB3
fAB1
tAB2
tAB3
5
54.85
230+1
382+1
370
51.94
51.62
51.58
43.84
39.91
38.76
10
19.80
23.35
25.08+2
82.17
14.63
13.43
13.07
11.14
7.78
6.02
15
11.29
5.63+1
7.46
8.90+1
7.31
6.28
5.90
5.89
4.35
3.71
20
7.91
3.84
3.05+1
4.14
4.91
4.19
3.91
4.23
3.35
3.00
50
3.82
2.60
2.56+1
2.59+2
2.86
2.70
2.64
2.79
2.63
2.58
Table 7: DEIS for VPSDE CIFARIO with to 10
in Eq. (42)
FID for various DEIS with K
7 in Eq. (43)
NFE
DDIM
p2Heun
P3Kutta
pARK
pAB1
pAB2
pAB3
IAB1
(AB2
LAB3
5
53.20
108+1
185+1
193-3
47.56
46.36
46.13
36.98
28.76
25.76
10
18.99
18.75
20.27+2
54.92+2
13.38
11.84
11.21
10.92
8.26
6.87
15
10.91
4.89+1
6.31
9.79+1
6.90
5.86
5.42
6.12
4.86
4.33
20
7.81
3.50
2.97 +1
3.92
4.84
4.10
3.80
4.48
3.69
3.38
50
3.84
2.60
2.58+1
2.59+2
2.86
2.69
2.64
2.82
2.66
2.61
Table 8: (2022)
DEIS for VPSDE on CIFAR10 with to and time scheduling suggested by Karras et al.
H.9 MORE REUSLTS ON VESDE
Though VESDE does not achieve the same accelerations as VPSDE, our method can significantly accelerate VESDE sampling compared with previous method for VESDE. We show the accelerated FID for VESDE on CIFAR10 Tab. 15 and sampled images in Fig. 10.
H.10
CHECKPOINT USED AND CODE LICENSES
Our code will be released in the future. We implemented our approach Jax and Py have also used code from number of sources in Tab. 16.
25

---

Published as a conference paper at ICLR 2023
FID with various NFE
Method
5
10
20
30
50
100
200
500
1000
Euler
246.16
90.52
27.38
14.99
8.46
4.96
3.54
2.81
2.62
+ EI
283.67
216.47
137.20
100.74
68.03
37.93
18.81
6.66
3.69
+EB
42.38
17.23
9.71
7.56
5.76
4.24
3.37
2.83
2.67
+ Poly
30.67
10.41
6.50
5.13
4.06
3.07
2.69
2.58
2.57
Opt (t.)
15.37
5.02
3.03
2.70
2.59
2.57
2.56
2.56
2.56
Table 9: Quantitative comparison in Fig. 5 for introduced ingredients, Exponential Integrator (EI), cø-based score parameterization, polynomial extrapolation, and optimizing time discretization (t,). where we change uniform stepsize to quadratic one to 10-4. We include Tabs. 6 to 8 for more ablation studies regarding time discretization.
FID with various NFE
Method
10
20
30
50
100
200
500
1000
Uniform
246.16
90.52
27.38
14.99
8.46
4.96
3.54
2.81
2.62
Quadratic
294.01
138.73
39.82
19.26
8.49
3.96
2.88
2.61
2.57
Table 10: Effects of different timesteps on the Euler method. We use to = 10 which has lower FID score compared with the default ty = 10 (Song et al., 2020b) in the experiments.
We list the used checkpoints and the corresponding experiments in Tab, 17.
MORE RESULTS FOR IMAGE GENERATION
26

---

Published as a conference paper at ICLR 2023
NFB
14
26
32
38
50
62
88.2
344
FID
61.11
36.64
15.18
9.88
6.32
2.63
2.56
2.55
Table 11: Quantitative performance of RK45 ODE solver with lg 10 Fig.
FID NFE
5
10
20
50
Method
A-DDIM
51,47
14.06
6.74
4.04
1-iPNDM
30.13
13.01
8.25
5.65
2-iPNDM
84.00
10.45
6.79
4.73
3-iPNDM
105.38
14.03
5.79
4.24
LAB1-DEIS
20.45
8.11
4.91
3.88
LAB2-DEIS
18.87
7.47
4.66
3.79
tAB3-DEIS
18.43
7.12
4.53
3.78
Table 12: Comparison with A-DDIM on the checkpoint and time scheduling provided by (Bao al., 2022) on CIFARIO
FID NFE
5
10
20
50
Method
iPNDM
54.62
15.32
9.26
8.26
DDIM
49.08
23.52
13.69
9.44
tAB1-DEIS
34.69
13.94
9.55
8.41
LAB2-DEIS
29.50
11.36
8.79
8.29
LAB3-DEIS
28.09
10.55
8.58
8.25
Table 13: Sampling quality on VPSDE ImageNet32 x 32 with the checkpoint provided by Song
al. (2021a). Blackbox ODE solver reports FID 8.34 with ODE tolerance 1 10-5 (NFE around 130).
FID NFE
Dataset
5
10
20
50
Method
PNDM
7.60
3.51
iPNDM
19.8
7.78-0.18
5.58
3.34-1.04
CELEBA
DDIM
30.42+0.87
13.53++0.48
6.8940.11
4.17:10.04
AABI-DEIS
26.65+0.63
8.81=0.23
4.33+0.07
3.19-10.03
tAB2-DEIS
25.13:0.56
7.20-0.21
3.61
3.04-10.02
LAB3-DEIS
25.07
6.95-0.09
3.41
2.95:10.03
Table 14: Mean and standard deviation of multiple rums with different random seeds on the check- point and time scheduling provided by Liu al. (2022) on CELEBA.
FID NFE
SDE
5
10
20
50
Method
IABÖ-DEIS
103.52.12.09
46.90-10.38
27.64
19.86
tABI-DEIS 56.3310.87
26.16+0.11
18.5240..33
16.6410.01
VESDE
TAB2-DEIS 58.6540.22
20.89-0.09
16.94+0.03
16.33#0.02
tAB3-DEIS
96.7010.90
25.01
16.5910.03
16.3110.02
Table 15: FID results of DEIS on VESDE CIFAR1O. We note the Predictor-Correed algorithm proposed in (Song et al., 2020b) have 2 100 FID if sampling with limited NFE budget (S 50).
27

---

Published as a conference paper at ICLR 2023
NFE-20
NFE-10
NFE-5
7=0
T=1
T == 2
r=3
Figure 10: Generated images with tABr-DEIS on VESDE CIFARIO.
URL http:///ggchbb..commmyyygoooooo http:://sttaaa..com///ppgg//////
License
Apache License 2.0
Apache License 2.0 MIT
Unknown
Table 16: Code Used and License
Experiment
Citation
License
CIFARIO Tabs. 2, 6 to 8, 11, 14 and 15, FFHQ Fig.
(Song al., 2020b)
Apache License 2.0
CIFAR10 Tab. 12
(Bao et al., 2022)
Unknown
CELEBA Tab. 14
(Liu al., 2022)
Apache License 2.0
ImageNet 32 x 32 Tab. 13
(Song et al., 202 la)
Unknown
Text-to-image
(Rombach al., 2021)
MIT
Table 17: Checkpoints for experiments
28

---

Published as a conference paper at ICLR 2023
Figure 11: Generated images with text "A artistic painting of snow trees by Pablo Picaso, oil on
canvas" (15 NFE)
Diffusion
Difussion
DIFFUSION
DIFFUSION
Figure 12: Generated images with text "A street sign that reads Diffusion" (15 NFE)
Figure 13: Generated images with text "The drawing of a funny husky" (15 NFE)
G
Figure 14: Generated images with text "Cyber punk oil painting" (15 NFE)
29

---

Published as a conference paper at ICLR 2023
NFE=10
NFE-5
7=0
T=1
T=2
r=3
Figure 15: Generated images with DEIS on VPSDE CIFARIO.
NFE-20
NFB-T
NFE-5
r=0 0
T=1
T= 2
r=3 3
Figure 16: Generated images with DEIS on VPSDE ImageNet 32 x 32.
30

---

Published as a conference paper at ICLR 2023
A
L
D
x
J
A
J
J
A
F
A
A
A
La
Kitk
A
T
A
A
Lan
Lane
A
Figure 17: Generated images with DEIS on VPSDE CelebA (NFE 5).
31

---

Published as a conference paper at ICLR 2023
Figure 18: Generated images with DEIS on VPSDE CelebA (NFE 10).
32

---

Published as a conference paper at ICLR 2023
Figure 19: Generated images with DEIS on VPSDE CelebA (NFE 20).
33


Reviews:
[{"review_id": "IsDXFEIBlH", "reviewer": "Reviewer_HoiQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper targets at improving the sampling efficiency of diffusion model via a semi-linear ODE methods, termed as Diffusion Exponential Integrator Sampler (DEIS). The paper is well motivated with experiment observations that the changes of \u000fnoise predicted by the neural network is small for most of the sampling steps as shown in Fig. 4a in the paper. Thus the values can be reused. The author also explores a variety of functions by measuring the  discretization error. The author finally proposes a principled guideline on achieving fast sampling of DM models with High order polynomial and . The experiments on CIFAR-10 and ImageNet with image size downsampled to 32x32 support the main claims of the paper.", "summary_of_the_review": "Overall, I like the paper's style and I think the authors have put in enough efforts on this work. However, the advantages of the proposed methods are not clean enough. The experiments are relatively weak in terms of both dataset and selected baselines. Please refer to the `strength & weakness' section for the details. Nevertheless, the paper have put in enough analysis that reveals some of the properties of DM. I would vote for board-line accept for this paper.", "strength_and_weaknesses": "Strengths:\n1. The paper is well written. The flow is easy to follow and the contributions of the paper is clearly presented.\n2. The background knowledge on diffusion model is well presented and the readers can easily get the main claims of the paper, which is the sampling efficiency.\n3. I like the analysis conducted by the authors. The claims are supported by the experiments results.\n\nWeakness:\n1. The paper only conducted experiment on easy & object centric datasets: CIFAR and ImageNet with 32x32 size. However, the main advantages of DM are on complex scenes where the generated images contain multiple objects. Besides, one of the well known disadvantages of DM is the high resolution images. Could the proposed algorithm achieve promising results on applied to the above mentioned cases?\n\n2. In Appendix B, authors claim that DPM-Solver is a concurrent work. The authors claim that DPM-solver is a special case of DEIS. So, how about the comparison with DPM-Solver? Does the more general exploration of DEIS outperform DPM-Solver?\n\n3. Recent popular LDM models use PLMS solver and achieves even faster sampling speed. How the proposed DEIS compared to PLMS sampler?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6cnKpHYm-S", "reviewer": "Reviewer_xTdk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a sampler for diffusion models based on the exponential integrator and the semi-linear structure of the probability flow ODE. The authors demonstrate that this sampler maintains strong performance for low numbers of function evaluations.   ", "summary_of_the_review": "Overall I think this is a well-motivated paper with convincing empirical results. A more thorough discussion and comparison of competing methods, as well as some restructuring to move results from the appendix would improve the paper further. \n\nEdit post-rebuttal: I'd like to thank the authors for their response. Some of the main criticism of the paper concerns concurrent work, which I feel the authors have attempted to address. Both the concurrent work and this paper constitute useful contributions, and I think this paper also deserves to appear in proceedings, even if the overall presentation is perhaps less clear. I've raised my score accordingly. ", "strength_and_weaknesses": "**Strengths** \nThe formulation of the solver is relatively well-structured and motivated. The three ingredients proposed in section 3.2 and the related ablation of each on the toy problem is neat. The explanation of DDIM in terms of a first order method for the semi-linear ODE is nice, although a similar argument was already made in DPM-Solver (Lu et al) (although I acknowledge it was concurrent work, more discussion below). The reparameterization in section 4 removing the linear term in section 4 is also neat, but again the discussion overlaps with Karras et al (also discussed more below). The experimental results, primarily Table 2, are convincing, especially showing the worth of the higher order methods. \n\n**Weaknesses** \nThe experiments section is quite concise. There is quite a lot of additional content in appendices G and H. It would be good to compare to more methods in the main paper (as well as the concurrent work discussed below) and not relegate this to the appendix. It would also be good to expand on the use of the proposed sampler for pretrained models like Stable Diffusion beyond qualitative samples -- showing improved performance here over competitive approaches would be a strong result. \n\nRelated to iPNDM: 'We further propose Improved PNDM (iPNDM) that uses lower order multistep methods for the first few steps in sampling to avoid expensive warming start.' Isn't it the case that for linear multistep methods the expensive warm start is necessary? i.e. if you don't warm start an order K methods with steps of at least order (K - 1), the resulting method is bottlenecked by the lowest order used for the warm start? \n\n**Nits**\n- Intro: 'DM is not hyperparameter sensitive' - maybe less sensitive than GANs, otherwise this is too strong a statement.\n- Intro: 'extremely slow sampling' - again, 'extreme' is probably too strong a statement, compared with e.g. AR sampling \n- Figure 2: the vertical-axis isn't labeled \n- 7: 'DEIS also works amazingly in practice' - again too strong ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4IW_DliYV-e", "reviewer": "Reviewer_5JHW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors present efficient integrator for diffusion models based on polynomial interpolation of the noise during trajectory estimation. ", "summary_of_the_review": "The paper should be simplified and made more clear, at this point it is too hard to read due to unnecessary mathematical overload and the reproducibility is under question.", "strength_and_weaknesses": "Strengths:\n1. Idea somewhat interesting from theoretical stand point\n2. Detailed motivation for ODE case why this should work\n\nWeakness:\n1. It's not clear why you mention application to SDE case as you mention that your method do not work for integrating SDE equation\n2. Notation is somewhat overcomplicated for no reason, e.g. \\Phi(s, t) -- why you need to mention hard matrix ODE as in your case it is always scalar multiplied by identity matrix? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0KATnHhI2L", "reviewer": "Reviewer_xzme", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a fast sampling method, termed as diffusion exponential integrator (DEIS), for diffusion models with a new discretization of the reverse process. In particular, this work first investigates the existing ODE solvers for diffusion models and found that reducing the discretization error is crucial for fast sampling. This work then proposes a new discretization that is mainly based on the exponential integrator (EI) and polynomial extrapolation of the noise prediction function. Besides, this work shows that DDIM is a special case of DEIS. Experiments show that the proposed method can achieve 4.17 FID with 10 function evaluations on CIFAR-10, outperforming other ODE discretization based methods. \n", "summary_of_the_review": "Overall, I think the idea is interesting and useful, but the idea and experimental results are poorly presented, which makes me not sure how it compares with previous relevant methods. My initial suggestion is “leaning to reject” but I’m open to adjusting my rating.\n", "strength_and_weaknesses": "Strengths:\n- The proposed idea from the exponential integrator perspective is interesting and novel to me. The motivation of reducing the discretization error using polynomial extrapolation is sound and intuitive. \n- This work provides a good insight into the understanding of DDIM from the numerical discretization perspective. That is, applying EI into the noise prediction reparameterization in VPSDE without the polynomial extrapolation becomes the DDIM sampling.\n- Experimental results on CIFAR-10 show the effectiveness of the proposed method: the best variant of DEIS largely outperforms previous ODE-based training-free methods (DDIM, A-DDIM, PNDM) when the NFE is less than 20.\n\nWeaknesses:\n- The presentation of the proposed method is not clear. I understand that this work tries to make the main idea well-motivated and general, but I think many design choices that do not work well can be moved to the appendix, such that we understand the core idea more easily. Below are some details: 1) if EI works better than Euler with the noise prediction reparameterization (while it works worse than Euler with score prediction reparameterization), we can present the idea with the noise prediction reparameterization, which has been commonly used in most diffusion models for images. The analysis for different network reparameterizations can be deferred to the appendix. 2) I don’t fully understand why we need Proposition 3. In other words, why do we need to transform Eq. (10) into “a simple non-stiff ODE”, if the resulting $\\rho$AB-DEIS is no better than $t$AB-DEIS (Table 2)? Also, I don’t fully understand why it is a good thing to apply generic ODE solvers without “worrying about the semi-linear structure”. 3) If DEIS does not work well for SDE, we can defer most details in Section 5 to the appendix. 4) I’m not exactly sure what “+optimizing timesteps” in Figure 5 means. It seems that the main text doesn’t directly explain its meaning.\n- The experiment settings and results are also confusing. Below are some details: 1) It mentions that the proposed method compares with other training-free baselines (DDIM, A-DDIM, PNDM) on several datasets. However, in the main text, we don’t see the results of A-DDIM and PNDM at all, and there is no quantitative result on other datasets except CIFAR-10. I suggest presenting these main experimental results in a more compact and coherent way instead of deferring most of them to the appendix. 2) In the experimental setting, I think several more recent training-free baselines (e.g. [1,2]) are missing. I noticed that the authors discussed their relationship with the proposed methods in the appendix. I would prefer to discuss these relevant baselines and show quantitative comparison results in the main text. 3) I appreciate that the authors considered ImageNet as one of the datasets to evaluate the proposed method. But I think ImageNet 32x32 is a bit low resolution and difficult to visually see the generation quality. I suggest similar to many previous works (e.g., [2,3]) that consider ImageNet with a resolution of at least 64x64 for obtaining both quantitative and qualitative results.\n- Minor things: In section 2, when describing the conditional marginal distribution $p_{0t}(x_t|x_0)$, what does $\\mu_t$ mean? I suggest replacing the term “Diffusion model” with “diffusion model”.\n\n\n[1] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022.\n[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022.\n[3] Progressive Distillation for Fast Sampling of Diffusion Models, ICLR 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster