gDDIM: Generalized denoising diffusion implicit models
Authors: Qinsheng Zhang
Molei Tao
Yongxin Chen
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Fast sampling
diffusion model

Abstract: Our goal is to extend the denoising diffusion implicit model (DDIM) to general diffusion models~(DMs) besides isotropic diffusions. Instead of constructing a non-Markov noising process as in the original DDIM, we examine the mechanism of DDIM from a numerical perspective.  We discover that the DDIM can be obtained by using some specific approximations of the score when solving the corresponding stochastic differential equation. We present an interpretation of the accelerating effects of DDIM that also explains the advantages of a deterministic sampling scheme over the stochastic one for fast sampling. Building on this insight, we extend DDIM to general DMs, coined generalized DDIM (gDDIM), with a small but delicate modification in parameterizing the score network. We validate gDDIM in two non-isotropic DMs: Blurring diffusion model (BDM) and Critically-damped Langevin diffusion model (CLD). We observe more than 20 times acceleration in BDM. In the CLD, a diffusion model by augmenting the diffusion process with velocity, our algorithm achieves an FID score of 2.26, on CIFAR10, with only 50 number of score function evaluations~(NFEs) and an FID score of 2.86 with only 27 NFEs.

Published as a conference paper at ICLR 2023
GDDIM: GENERALIZED DENOISING DIFFUSION IM- PLICIT MODELS
Qinsheng Zhang
Molei Tao
Georgia Institute of Technology qzhang4 1 9êgatech. edu
Georgia Institute of Technology mtacegatech. edu
Yongxin Chen
Georgia Institute of Technology yongchen@gatechi edu
ABSTRACT
Our goal is to extend the denoising diffusion implicit model (DDIM) to general diffusion models (DMs) besides isotropic diffusions. Instead of constructing a non-Markov noising process as in the original DDIM, we examine the mecha- nism of DDIM from numerical perspective. We discover that the DDIM be obtained by using some specific approximations of the score when solving the corresponding stochastic differential equation. We present an interpretation of the accelerating effects of DDIM that also explains the advantages of a determinis- tic sampling scheme over the stochastic one for fast sampling. Building on this insight, we extend DDIM to general DMs, coined generalized DDIM (gDDIM). with small but delicate modification in parameterizing the score network. We validate gDDIM in two non-isotropic DMs: Blurring diffusion model (BDM) and Critically-damper Langevin diffusion model (CLD). We observe more than 20 times acceleration in BDM. In the CLD. diffusion model by augmenting the diffusion process with velocity, our algorithm achieves an FID score of 2.26, on CIFAR10, with only 50 number of score function evaluations (NFEs) and FID score of 2.86 with only 27 NFEs. Project page and code: https:/github..comqqhh zh/gDDIM.
INTRODUCTION
Generative models based on diffusion models (DMs) have experienced rapid developments in the past few years and show competitive sample quality compared with generative adversarial net- works (GANs) (Dhariwal & Nichol, 2021; Ramesh al.; Rombach al., 2021). competitive nega- tive log likelihood compared with autoregressive models in various domains and tasks (Song et al., 2021; Kawar al., 2021). Besides, DMs enjoy other merits such stable and scalable training, and mode-collapsing resiliency (Song et al., 2021; Nichol & Dhariwal, 2021). However, slow and expensive sampling prevents DMs from further application in more complex and higher dimension tasks. Once trained, GANs only forward pass neural networks once to generate samples, but the vanilla sampling method of DMs needs 1000 or even 4000 steps (Nichol & Dhariwal, 2021; Ho et al., 2020; Song et al., 2020b) to pull noise back to the data distribution, which means thousands of neural networks forward evaluations. Therefore, the generation process of DMs several orders of magnitude slower than GANs.
How to speed up sampling of DMs has received significant attention. Building on the seminal work by Song et al. (2020b) on the connection between stochastic differential equations (SDEs) and diffusion models, a promising strategy based on probability flows (Song et al., 2020b) has been developed. The probability flows are ordinary differential equations (ODE) associated with DMs that share equivalent marginal with SDE. Simple plug-in of off fff-thh--heff ODE solvers can already achieve significant acceleration compared to SDEs-based methods (Song et al., 2020b). The guably most popular sampling method is denoising diffusion implicit model (DDIM) (Song et al., 2020a), which includes both deterministic and stochastic samplers, and both show tremendous im-

---

Published as a conference paper at ICLR 2023
provement in sampling quality compared with previous methods when only small number of steps is used for the generation.
Although significant improvements of the DDIM in sampling efficiency have been observed empiri- cally, the understanding of the mechanism of the DDIM is still lacking. First, why does solving prob- ability flow ODE provide much higher sample quality than solving SDEs, when the number of steps is small? Second, it is shown that stochastic DDIM reduces to marginal-equivalent SDE (Zhang & Chen, 2022), but its discretization scheme and mechanism of acceleration are still unclear. Finally, can we generalize DDIMs to other DMs and achieve similar or even better acceleration results?
In this work, we conduct a comprehensive study to answer the above questions, that we can generalize and improve DDIM. We start with an interesting observation that the DDIM can solve corresponding SDEs/ODE exactly without any discretization error in finite or even one step when the training dataset consists of only one data point. For deterministic DDIM, we find that the added noise perturbed data along the diffusion constant along an exact solution of probability flow ODE (see Prop 1). Besides, provided only one evaluation of log density gradient (a.k.a. score), we are already able to recover accurate score information for any datapoints, and this explains the acceleration of stochastic DDIM for SDEs (see Prop 3). Based on this observation, together with the manifold hypothesis, we present one possible interpretation to explain why the discretization scheme used in DDIMs is effective on realistic datasets (see Fig. 2). Equipped with this new interpretation, we extend DDIM to general DMs, which we coin generalized DDIM (gDDIM). With only a small but delicate change of the score model parameterization during sampling, gDDIM can accelerate DMs based on general diffusion processes. Specifically, we verify the sampling quality of gDDIM on Blurring diffusion models (BDM) (Hoogeboom & Salimans, 2022; Rissanen et al., 2022) and critically-damped Langevin diffusion (CLD) (Dockhorn et al., 2021) in terms of Fréchet inception distance (FID) (Heusel et al., 2017).
To summarize, we have made the following contributions: 1) We provide an interpretation for the DDIM and unravel its mechanism. 2) The interpretation not only justifies the numerical discretiza- tion of DDIMs but also provides insights into why ODE-based samplers are preferred over SDE- based samplers when NFE is low. 3) We propose gDDIM, a generalized DDIM that can accelerate large class of DMs deterministically and stochastically. 4) We show by extensive experiments that gDDIM can drastically improve sampling quality/efficiency almost for free. Specifically, when applied to CLD, gDDIM can achieve an FID score of 2.86 with only 27 steps and 2.26 with 50 steps. gDDIM has more than 20 times acceleration on BDM compared with the original samplers.
The rest of this paper is organized as follows. In Sec. 2 we provide brief inntroduction to diffusion models. In Sec. 3 we present an interpretation of the DDIM that explains its effectiveness in practice. Built on this interpretation, we generalize DDIM for general diffusion models in Sec. 4.
2 BACKGROUND
In this section, we provide a brief introduction to diffusion models (DMs). Most DMs are built on two diffusion processes continuous-ting one forward diffusion known as the noising process that drives any data distribution to a tractable distribution such as Gaussian by gradually adding noise to the data, and one backward diffusion known as the denoising process that sequentially removes noise from noised data to generate realistic samples. The continuous-time noising and denoising processes are modeled by stochastic differential equations (SDEs) (Särkkä & Solin, 2019).
In particular, the forward diffuusin is linear SDE with state (t) E IR
du = udt + Gidw,, / E [0,
(1)
where F,, G. E RDDD represent the linear drift coefficient and diffusion coefficient respectively, and w is a standard Wiener process. When the coefficients are piece-wise continuous, Eq. (1) admits a unique solution (Oksendal, 2013). Denote by Pa (1) the distribution of the solutions (u(t) ( JosesT (simulated trajectories) to Eq. (1) at time f, then Po is determined by the data distribution and pr is a (approximate) Gaussian distribution. That is, the forward diffusion Eq. (I) starts as data sample and ends as a Gaussian random variable. This can be achieved with properly chosen coefficients F,, G.. Thanks linearity of Eq. (1), the transition probability Pee (((t))((s) from u(s) to u(t) is a Gaussian distribution. For convenience, denote Po: (u()) (0) by (0), E,) where μr, Σ, e DDD.
2

---

Published as a conference paper at ICLR 2023
66 (u(t) 4: 1:30
(wir),
(x(t)) jOnE
T
Euler
RK45
L EI- Buler
RS El-Euler
R El-Multistep
NFK- NFE- NFE-
NPE-
NPR- 100
NFE-
NP
NFR-
NFE-
NPR-
Figure I: Importance of K, for score parameterization -) (w, 1) and acceleration of diffusion sampling with probability flow ODE. Trajectories of probability ODB for CLD (Dock- horn al., 2021) at random pixel locations (Left). Pixel value and output of cæ v channel with choice K, L, (Dockhorn et al., 2021) along the trajectory (Mid). Output of eg in æ, v channels with our choice RG (Right). The smooth network output along trajectories enables large stepsize and thus sampling acceleration. gDDIM based on the proper parameterization of K2 can accelerate more than 50 times compared with the naive Euler solver (Lower row).
The backward process from (T) to (0) of Eq. (I) the denoising process. can be characterized by the backward SDE simulated in eeverse-tiee direction (Song al., 2020b; Anderson, 1982)
du Fuudl G.G.T V log log)r(u))dl + G,dw
where w denotes a standard Wiener process running backward in time. Here V log Pt (w) is known as the score function. When Eq. (2) is initialized with u(T) pr, the distribution of the simulated trajectories coincides with that of the forward diffusion Eq. (1). Thus, u(0) of these trajectories are unbiased samples from po: the backward diffusion Eq. (2) is ideal generative model.
In general, the score function V log Pa (u) is not accessible. In diffusion based generative models, a dime-dependent network s sp t), known as the One effective approach to train 89 (o 1) is the denoising score matching (DSM) technique (Song al., 2020b; Ho et al., 2020; Vincent, 2011) that seeks minimize the DSM loss
network, is used to fit the score V log P. (u).
[0 (())) (O)) V log Poe (u(() - so (1)) 11
where 24(0, 2'] represents the uniform distribution over the interval [0, T'. The time-dependent weight A; is chosen to balance the trade-off between sample fidelity and data likelihood of learned generative model (Song al., 2021). It is discovered in Ho al. (2020) that reparameterizing the score network by
(4) with K.K.- E, leads better sampling quality. In this parameterization, the network tries to predict directly the noise added to perturb the original data. Invoking the expression A (0), Ex) of Poe Poe(u (u(t) u (0)). this parameterization results in the new DSM loss
sp(u, t) = -, (e t)
(())
Bul [l)le (0)) K,E, t)]| 1A, =].
(5)
Sampling: After the score network 8p is trained, one can generate samples via the backward SDE Eq. (2) with learned score, the marginal equivalent SDE/DDE (Song al., 2020b; Zhang &
3

---

Published as a conference paper at ICLR 2023
Chen, 202);
du = FFuu 1+ 12
X2
2 G,G. su, l)dt + XG,dw,
(6)
where À 2 0 is free parameter. Regardless of the value of X, the exact solutions to Eq. (6) produce
unbiased samples from Po if so(u, t) V log Pr for all t, When À = 1, Eq. (6) reduces to reverse-time diffusion in Eq. (2). When À Eq. (6) known as the probability flow flow ODE (Song et al., 2020b)
1 G. so(u, 2
du FFuu
Isotropic diffusion and DDIM: Most existing DMs are isotropic diffusions. popular DM is Denoising diffusion probabilistic modeling (DDPM) (Ho et al., 2020). For a given data distribution Pz)ta()), DDPM has u = a E Rd and sets po = (z). Though originally proposed in the discrete-time setting, it can be viewed as a discretization of continuous-time SDE with parameters
F, = 1 doggg Cr
dlog Ia di
Iai G=:
2 di
for a decreasing scalar function a satisfying ag = 1, = O. Here La represents the identity matrix of dimension d. For this SDE, K, is always chosen to be /1 a,,a
The sampling scheme proposed in DDPM inefficient; it requires hundreds or even thousands of steps, and thus number of score function evaluations (NFEs), to generate realistic samples. A more efficient alternative is the Denoising diffusion implicit modeling (DDIM) proposed in Song et al. (2020a). It proposes a different sampling scheme over grid {1:}
axti-1))= V ar art,) + (V a 02
an,, - -o,
V1 out 01
(co()(()),))) + C2
(9)
where } are hyperparameters and E NO, Ia). DDIM can generate reasonable samples within 50 NFEs. For the special case where ot. O, is recently discovered in Zhang & Chen (2022) that Eq. (9) coincides with the numerical solution to Eq. (7) using advanced discretization scheme known as the exponential integrator (EI) that utilizes the semi-linear structure of Eq. (7).
CLD and BDM: Dockhorn et al. (2021) propose critically- dampled Langevin diffusion (CLD), DM based on an augmented diffusion with an auxiliary velocity term. More specifically, the state v(()]
of the diffusion in CLD is of the form u(t) [()) E Rdad with velocity variable v(t) E Rd.
The CLD employs the forward diffusion Eq. (1) with coefficients
0
BM-1
F+
= B
-
"9 Ia G, = D - 2 Ia-
(10)
Here 1' > 0, > 0, M > O are hyperparameters, Compared with most other DMs such as DDPM that inject noise the data state æ directly, the CLD introduces noise the data state æ through the coupling between v and as the noise only affects the velocity component directly. Another interesting DM Blurring diffusion model (BDM) (Hoogeboom & Salimans, 2022). It can be shown the forward process in BDM can be formulated as SDE with (Detailed derivation in App. B)
F.
d VloggVa,VV V G.
V-
do
d dl
F.o - FiF.
(11)
dl
where VT' denotes a Discrete Cosine Transform (DCT) and V denotes the Inverse DCT. Diagonal matrices Ot of are determined by frequencies information and dissipation time. Though it is argued that inductive bias in CLD and BDM can benefit diffusion model (Dockhorn et al., 2021; Hooge- boom & Salimans, 2022), non-isotropic DMs are not easy to accelerate. Compared with DDPM, CLD introduces significant oscillation due to Z-- coupling while only inefficient ancestral sampling algorithm supports BDM (Hoogeboom & Salimans, 2022).
REVISIT DDIM: GAP BETWEEN THE EXACT SOLUTION AND NUMERICAL SOLUTION
The complexity of sampling from a DM is proportional to the NFEs used to numerically solve Eq. (6). To establish sampling algorithm with small NFEs, we ask the bold question:
4

---

Published as a conference paper at ICLR 2023
Can generate samples exactly from DM with finite steps if the score function is precise?
To gain some insights into this question, we start with the simplest scenario where the training data point æg. It turns out that accurate sampling from diffusion models on this toy example is not that easy, even if the exact score function is accessible. Most well-known numerical methods for Eq. (6), such as Runge Kutta (RK) for ODE, Buler-Maruyama (EM) for SDE, are accompanied by discretization error and cannot recover the single data point in the training set unless an infinite number of steps are used. Surprisingly, DDIMs can recover the single data point in this toy example in one step.
dataset consists of only
Built this example, we show how the DDIM can be obtained by solving the SDE/ODE Eq. (6) with proper approximations. The effectiveness of DDIM is then explained by justifying the usage of those approximations for general datasets the end of this section.
ODE sampling We consider the deterministic DDIM, that is, Eq. (9) with or, = 0. In view of Eq. (8), the score network Eq. (4) is (q, t) and the real score, denote the ground truth version of eg by EGT. In our toy example, the following property holds for EGT.
calu.t)
vi-aa" To differentiate between the learned score
Proposition 1. Assume po(u)) is a Dirac distribution. Lel u(l) be an arbitrary solution to the probability flow ODE Eq. (7) with coefficient Eq. (8) and the ground truth score, then (cT(u(()) -VI V log(:(u(!)) remains constant, which is V log (T)()), along u(t).
We remark that even though (GT(u(t), t) remains constant along an exact solution, the score v log Pappuuuu (u(t)) is time-varying. This underscores the advantage of the parameterization over sg. Inspired by Prop 1, we devise sampling algorithm as follows that can recover the exact data point in one step for our toy example. This algorithm turns out to coincide with the deterministic DDIM.
vi-ar and the approximation ea(u, r) se ()(((),,t) for T E [t, the solution to the probability flow ODE Eq. (7) with coefficient Eq. (8) is
Proposition 2. With the parameterization se(u, T) co(u.T)
ull- - Al) V V
att ull) + (V1 at v1 vl-
at-Ai (e)()()), 04
(12)
which coincides with deterministic DDIM.
When 6g = EGT as is the case in our toy example, there is no approximation error in Prop 2 and Eq. (12) is precise. This implies that deterministic DDIM recover the training data in one step in our example. The update Eq. (12) corresponds to a numerical method known as the exponential integrator to the probability flow ODE Eq. (7) with coefficient Eq. (8) and parameterization se su 7) = (v..7)
VI-ar" This strategy is used and developed recently in Zhang & Chen (2022). Prop and toy experiments in Fig. 2 provide sights on why such strategy should work.
SDE sampling The above discussions however do not hold for stochastic cases where 0 in Eq. (6) and o., > 0 in Eq. (9). Since the solutions to Eq. (6) from t = T to f = 0 are stochas- tic, neither V Vlog Plogpe(((() (u((t) nor (GT(u(t), t) remains constant along sampled trajectories; both are affected by the stochastic noise. The denoising SDE Eq. (6) is more challenging compared with the probability ODE since it injects additional noise to ut).. The score information needs to remove not only noise presented in uT)) but also injected noise along the diffusion. In general, one evaluation of c0(u, t) can only provide the information to remove noise in the current state u; it cannot predict the future injected noise. Can do better? The answer is affirmative on our toy dataset. Given only one score evaluation, it turns out that score at any point can be recovered.
Proposition 3. Assume SDE coefficients Eq. (8) and that po(u) a Dirac distribution. Given any evaluation of the score function V loo Ps (u(s)), one can recover V log Po(u) for any t, u as
1 0ss Vlog 1 aq
Of
Vlogpa(u(s))
1
uu
u u(s)).
(13)
as
1-au a
The major difference between Prop and Prop is that Eq. (13) retains the dependence of the score over the state u. This dependence is important in canceling the injected noise in the denoising SDE
5

---

Published as a conference paper at ICLR 2023
Trajectory (m())
t
(ecr(u(t), f)
Figure 2: Manifold hypothesis and Dirac distribution assumption. We model image dataset mixture well-separated Dirac distribution and visualize the diffusion process on the left. Curves in red indicate high density spanned by Poe Poe (I((t) (0)) by different mode and region surrounded by them indicates the phase when pt (u) is dominated by one mode while region surrounded by blue one is for the mixing phase, and green region indicates fully mixed phase. On the right, sampling trajectories depict smoothness of EGT along ODE solutions, which justifies approximations used in DDIM and partially explains its empirical acceleration.
Eq. (6). This approximation Eq. (13) turns out to lead to numerical scheme for Eq. (6) that coincide with the stochastic DDIM.
Theorem 1. Given the parameterization su(u,7) T) cu(u)) and the approximation sp sp(u, T) s VI-a,
1-0s. T-a, (= T-o, (u
Vo u()) for It At,t), t). the exact solution u(t At)
to Eq. (6) with coefficient Eq. (8) is
ult - At) ~ vV(
Dt u(t) + [1-v
ar-At
VI ai V at a of co(u(t),t), of
a4
(14)
with ox = =1--a-ss) 11-(1 (
a1- ) which is the same as the DDIM Eq.
(9).
Note that Thm with À = 0 agrees with Prop 2; both reproduce the deterministic DDIM but with different derivations. In summary, DDIMs can be derived by utilizing local approximations.
Justification of Dirac approximation While Prop and Prop require the strong assumption that the data distribution is Dirac, DDIMs in Prop 2 and Thm 1 work very effectively on realistic datasets, which may contain millions of datapoints (Nichol et al., 2021). Here we present one possible interpretation based on the manifold hypothesis (Roweis & Saul, 2000).
believed that real-world data lie on low-dimensional manifold (Tenenbaum et al., 2000) em- bedded in high-dimensional space and the data points are well separated in high-dimensional data space. For example, realistic images are scattered in pixel space and the distance between every two images can be very large if measured in pixel difference model this property, we consider a dataset consisting of M datapoints (m) ( The exact score
if they are similar perceptually. To
mm=1-
Poe (m))
V Vlg = Σ Wm log Por (m), Wm
(15)
Σm Pos Poe(ulut (u)
which can be interpreted as a weighted sum M score functions associated with Dirac distributions. This is illustrated in Fig. 2. In the red color region where the weights {w,} } are dominated by one
specific data (m)) and thus v p(()) V log Poe Poe (m")) Moreover, in the green region

---

Published as a conference paper at ICLR 2023
different modes have similar V log Por (u/u(m) as all of them close Gaussian and can be approximated by any condition score of any mode. The (ccT(((f), t)} trajectories in Fig. 2 validate our hypothesis as we have very smooth curves at the beginning and ending period. The phenomenon that score of realistic datasets can be locally approximated by the score of one datapoint partially justifies the Dirac distribution assumption in Prop 1 and 3 and the effectiveness of DDIMs.
4 GENERALIZE AND IMPROVE DDIM
The DDIM is specifically designed for DDPMs. Can we generalize it to other DMs? With the insights in Prop and 3, it turns out that with a carefully chosen K,, we can generalize DDIMs to any DMs with general drift and diffusion. We coin the resulted algorithm the Generalized DDIM (gDDIM).
4.1 DETERMINISTIC GDDIM WITH PROP
Toy dataset: Motivated by Prop I, we ask whether there exists an GGT that remains constant along a solution to the probability flow ODE Eq. (7). We start with a special case with initial distribution
po( (uu, Σo). It turns out that any solution to Eq. (7) is of the form
(tt) (t, 0)uu R4e.
(16)
with a constant and a time-varying parameterization coefficients R. RDDD that satisfies
Ro R = Eg and
dR: 1 G.G.T,) R dt (Fi (F.+ ;
(17)
Here st, is the transition matrix associated with F,; it is the solution to (V().)) F. F,V (8, In. Interestingly, R. satisfies R2 R. = E, like K, in Eq. (4). We remark K, v1 La a solution to Eq. (17) when the DM specialized to DDPM. Based on Eq. (16)
dt
and Eq. (17), we extend Prop I to more general DMs.
Proposition 4. Assume the data distribution po(u) is N(u,, Eo). Let u(t) be arbitrary lution to the probability flow ODE Eq. (7) with the ground truth score, then ECT(U(t),t) == - R. V log (u((t) remains constant along u(t).
Note that Prop 4 is slightly more general than Prop 1 in the sense that the initial distribution Pa is Gaussian instead Dirac. Diffusion models with augmented states such as CLD use a Gaussian distribution the velocity channel for each data point. Thus, when there is single data point, the initial distribution is Gaussian instead of a Dirac distribution. A direct consequence of Prop 4 is that we can conduct accurate sampling in one step in the toy example since we recover the score along any simulated trajectory given its value at = T, if K, in Eq. (4) is set to be R. This choice K; R₂ will make huge difference in sampling quality as we will show later. The fact provides guidance to design an efficient sampling scheme for realistic data.
Realistic dataset: As the accurate score not available for realistic datasets, we need to use learned score sa (u, t) for sampling. With our new parameterization eø co(u,t) f) -RR R. sy t) and the ap- proximation éo (u, 7) en es(u((), () for r E [t At, 4), we reach the update step for deterministic gDDIM by solving probability flow with approximator ca(u, T) exactly as
u(t At) (t - (t,t)((t) ((,))))) + +1f. --
2 T(t At, R2, dr)co(uu(t))), (18)
Multistep predictor-corrector for ODE: Inspired by Zhang & Chen (2022), we further boost the sampling efficiency of gDDIM by combining Eq. (18) with multistep methods (Hochbruck Os- termann, 2010; Zhang& & Chen, 2022; Liu et al., 2022). We derive multistep predictor-corrector methods to reduce the number of steps while retaining accuracy (Press et al., 2007; Sauer, 2005). Empirically. we found that using more NFEs in predictor leads to better performance when the total NFÈ is small. Thus, we only present multistep predictor for deterministic gDDIM. We include the proof and multistep corrector in App. B. For time discretization grid W where In ln T.
7

---

Published as a conference paper at ICLR 2023
the q-th step predictor from f; to f in term of cø parameterization reads
u(t;-1)= = (, + Σ (G)) t+t5)
(19a)
Cij - 11-1
bitk .dr.
(19b)
titk
We note that coefficients in Eqs. (18) and (19b) for general DMs can be calculated efficiently using standard numerical solvers if closed-form solutions are not available.
4.2 STOCHASTIC GDDIM WITH PROP 3
Following the same spirits, we generalize Prop
Proposition 5. Assume the data distribution po(a) is (uu. Eg). Given any evaluation of the score function V log Pa (2 (s)), one can recover log Pi u)) for any t, u as
v Pl = Ex (t, s) logp) E [u (t, s)u(s)]
(20)
Prop 5 is not surprising: in example, the has closed form. Eq. (20) not only provides an accurate score estimation for our toy dataset, but also serves as score approximator for realistic data.
Realistic dataset: Based on Eq. (20), with the parameterization sy (u, T) = =- -R, R.." co, T), we propose the following gDDIM approximator (0(u, r) for (u,
Co(u,T) T) V 8)
R,,,,),,,,(u( c, 8) R, lu Vo s){(s))
(21)
Proposition 6. With the parameterization eo (u, t) = - RT so t) and the approximator Eg(u, in Eq. (21), the solution to Eq. (6) satisfies
w(t) (t, s)u(s) + (t, (t, Re Pat),
(22)
where (t. 8) is the transition matrix associated with F, = F. 1+ G, G. and the covari- ance matrix Pst solves
dP., - P. + GG, P dr
(23)
Our stochastic gDDIM then uses Eq. (22) for update. Though the stochastic gDDIM and the deter- ministic gDDIM look quite different from each other, there exists a connection between them. Proposition Eq. (22) in stochastic gDDIM reduces to Eq. (18) in deterministic gDDIM when
== O.
5
EXPERIMENTS
As gDDIM reduces to DDIM for VPSDE and DDIM proves very successful, we validate the gener- ation and effectiveness of gDDIM on CL.D and BDM. We design experiments to answer the follow- ing questions. How to verify Prop 4 and 5 empirically? Can gDDIM improve sampling efficiency compared with existing works? What differences do the choice À and K, make? We conduct experiments with different DMs and sampling algorithms CIFAR10 for quantitative comparison. We include more illustrative experiments on toy datasets, high dimensional image datasets, and more baseline comparison in App. C.
Choice of K,: A key of gDDIM is the special choice K, = R, which is obtained via solv- ing Eq. (17). In CLD, Dockhorn et al. (2021) choose K, L, based Cholesky decomposition of Ee and does not obey Eq. (17). More details regarding L, are included in App. C. As it is shown in Fig. on real datasets with trained score model, randomly pick pixel locations and check the pixel value and ea output along the solutions to the probability flow ODE produced by the (L) (u, t; v) suffers from oscillation like Les co
high-resolution ODE solver. With the choice K,
8

---

Published as a conference paper at ICLR 2023
(R) value along time. However, Eo (u. 1) is much more flat. We further compare samples generated by Lg and R2 parameterizaiton in Tab. 1, where both use the multistep exponential solver in Eq. (19).
Table 1: L1 vs Rg (Our) on CLD
Table 2: X and integrators choice with NFE=50
FID at different NFE
FID at different À
K, 20
30
40
50
Method
0.0
0.1
0.3
0.5
0.7
1.0
L,
368
167
4.12
3.31
gDDIM
5.17
5,51
12.13
33
41
49
R2
3.90
2.64
2.37
2.26
EM 346
168
137
45
57
Choice A: We further conduct study with different À values. Note that polynomial extrapolation in Eq. (19) is not used here even when À = 0. As is shown in Tab. 2, increasing deteriorates the sample quality, demonstrating our claim that deterministic DDIM has better performance than its stochastic counterpart when a small NFE is used. We also find stochastic gDDIM significantly outperforms EM, which indicates the effectiveness of the approximation Eq. (21).
Accelerate various DMs: We present comparison among various DMs and various sampling algorithms. To make fair comparison, we compare three DMs with similar size networks while retaining other hyperparameters from their original works. We make two modifications to DDPM, including continuous-tine
training (Song al., 2020b) and smaller stop sampling time (Karras et al., 2022), which help improve sampling quality empirically. For BDM, we note Hoogeboom Salimans (2022) only supports the ancestral sampling algorithm, a variant With reformulated noising and denoising process as SDE Eq. (11), we can generate samples by solving corresponding SDE/ODEs. The sampling quality of gDDIM with 50 NFE can outperform the original ancestral sampler with 1000 NFE, more than 20 times acceleration.
EM algorithm.
Table 3: Acceleration various DMs with similar training pipelines and architecture. For RK45, we tune its tolerance hyperparameters so that the real NFE close but not equal to the given NFE. f: pre-trained model from Song et al. (2020b). ft: Karras et al. (2022) apply Heun method in rescaled DM, which is essentially a variant DEIS (Zhang & Chen, 2022)
FID (c) under different NFE
DM
Sampler
20
50
100
1000
EM
> >100
> >100
31.2
12.2
2,64
Prob.Flow, RK45
>10
52.5
6.62
2.63
2.56
DDPMT
2nd Heun i1
66.25
6.62
2.65
2.57
2.56
gDDIM
4.17
3.03
2.59
2.56
2.56
Ancestral sampling
>100
100
29.8
9.73
2.51
BDM
Prob.Flow, RK45
>100
68.2
7.12
2.58
2.46
gDDIM
4.52
2.97
2.49
2.47
2.46
EM
> 100
>> >00
57.72
13.21
2.39
CLD
Prob. Flow,Flow, RK45
>10
>
31.7
4.56
2.25
gDDMM
13.41
3.39
2.26
2.26
2.25
CONCLUSIONS AND LIMITATIONS
The more structural knowledge we leverage, the more efficient algorithms we obtain. In this work, we provide a clean interpretation of DDIMs based on the manifold hypothesis and the sparsity property on realistic datasets. This new perspective unboxes the numerical discretization used in DDIM and explains the advantage of ODE.-based sampler over SDE-based when NFE is small. Based on this interpretation, we extend DDIMs to general diffusion models. The new al- gorithm, gDDIM, only requires a tiny but elegant modification to the parameterization of the score model and improves sampling efficiency drastically. We conduct extensive experiments to validate the effectiveness of our new sampling algorithm. Limitation: There are several promising future directions. First, though gDDIM is designed for general DMs, we only verify it on three DMs. is beneficial to explore more efficient diffusion processes for different datasets, in which we believe gDDIM will play an important role in designing sampling algorithms. Second, more investigations needed to design efficient sampling algorithm by exploiting more structural knowledge in DMs. The structural knowledge can originate from different sources such as different modalities of datasets, and mathematical structures presented in specific diffusion processes.
Contribution:

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like thank the anonymous reviewers for their useful comments. This work is partially supported by NSF ECCS-1942523, NSF CCF-2008513, and NSF
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Process. Appl. 12(3):
313-326, May 1982. ISSN 0304-4149, doi: 10. 00.10166000-411900200095555555 URL https: //doi. oogii.rrg/11 10. 10311160000-4144 (82) 90051-5.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPMM DPM: An Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models. 2022. URL http://arxiv, org/abs/ 2201 06503.
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. 2021. URL 05233.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-Based Generative Modeling with Critically- Damped Langevin Diffusion. pp. 1-13, 2021. URL http://arxiv/ org/ hts//bbs/221 07068.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01360 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, volume 2020-Decem. 2020. ISBN 2006. I 1239v2. URL https:tp///// / //g. thu............ com / ho wwn.oaanno/tooooooooooo di ffusion.
Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209-286, 2010.
Emiel Hoogehoom and Tim Salimans. Blurring diffusion models. arXiv ar0p 2022.
Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021a.
Alexia Maficoeur-Martnnaal Ke Li, RR (e)) (e)(" Tal Kachman, and loannis Mitliagkas. Gotta Go Fast When Generating Data with Score-Based Models. May 2021b. doi: 10.4 10.485500/rxiv.21033 10080. URL http: /tt//vxivvv//////////////////avv org/ http//aab/21..111411 14080.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint X2i:22206000344, 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips:
Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34, 2021.
Zhifeng Kong and Wei Ping. On Fast Sampling of Diffusion Probabilistic Models. 2021a. URL http://arxiv orgoorg////.... absp//as.......////././///./..aa 00132.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv:2106.00132, 2021b.
arXiv preprint
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo Numerical Methods for Diffusson Models on Manifolds. (2021):1-23, 2022. URL httt::://axii.eee////e www.wiwwwwww..///////..b htt.b////222..99979.
10

---

Published as a conference paper at ICLR 2023
Cheng Lu. Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv:2206.00927, 2022.
Dpm-solver: A arXiv preprint
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint uIiv:2101.023388, 2021.
Siwei Lyu. Interpretation and generalization score matching. arXiv preprint 1tXiv:1205.261 2012.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. ArXiv, abs/2102.09672, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. 2021. URL http: http:://axxii/// http://awwxww.../aa/..aoa http://aaa////.11./. 11711.0111..11..
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Sci- ence & Business Media, 2013.
William H Press, Saul A Teukolsky, William T Vetterling, and Brian P Flannery. Numerical recipes 3rd edition: The art of scientific computing. Cambridge university press, 2007.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark (Chen OpenAI). Hierarchi- cal Text- Cext-Conditional Image Generation with CLIP Latents.
Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissi- pation. arXiv preprint arXiv:2206. 103.2797..1337,, 2022.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and B)()"(o)) Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. 2021. URL org/abs 2112.111211 10752.
Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290 5500:2323 6, 2000.
Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models. 2022.
org/ harg/bb///// 22022 C0512.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Timothy Sauer. Numerical analysis, 2005.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. 2020a. URL
org/ abs 2010.0000 02502.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-Based Generative Modeling through Stochastic Differential Equations. 2020b. URL http: //axiiv///xiv...///a org/ abs 2211 13456.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, 34, 2021.
Joshua B. Tenenbaum, Vin De Silva, and John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290 5500:2319-23, 2000.

---

Published as a conference paper at ICLR 2023
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-basets generative modeling in latent space. In Neural Information Processing Systems (NeurIPS). 2021.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput.
23(7):66111167, July 2011. ISSN 0899-7667, 1530-888X. doi: 10.1 10.1162/neco |ام -01142. URL https: /tt.///d..////////////////// .rrg www.:/1eeeeooaaaeaee
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sam- ple from diffusion probabilistic models. arXiv preprint arXiv:2106.03800 2021.
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality. 2022. URL http://arxiv, org/ hbs//b//// 2202 05830.
P. Whalen, M. Brio, and J.V. Moloney. Exponential lime-iifferencing with embedded Runge-Kutte Kutta adaptive step control. J. Comput. Phys., 280:579-601, January 2015. ISSN 0021-9991, doi: (..0066f..pp22014.0...3.
http:: /ttee:/ddooiiiee org /1..g/10.. 1ww./1016/..... jep. 2014. 09 038.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans, arXiv preprint 11200:2 2021.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Information Processing Systems, 34, 2021.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint 1rXiv:2204.39002, 2022.
12

---

Published as a conference paper at ICLR 2023
A MORE RELATED WORKS
Learning generative models with DMs via score matching has received tremendous attention re- cently
et al., 2015; Lyu, 2012; Song & Ermon, 2019; Song et al., 2020b; Ho et al., 2020; Nichol & Dhariwal, 2021). However, the sampling efficiency of DMs is still not satisfy- ing. Jalicoeur-Martineau et al. (2021a) introduced adaptive solvers for SDEs associated with DMs for the task of image generation. Song et al. (2020a) modified the forward noising process into a non-Markov process without changing the training objective function. The authors then proposed a family of samplers, including deterministic DDIM and stochastic DDIM, based on the modifica- tions. Both of the samplers demonstrate significant improvements over previous samplers. There are variants of the DDIM that aim to further improve the sampling quality and efficiency. Determin- istic DDIM in fact reduces to probability flow in the infinitesimal step size limit (Song et al., 2020a; Liu et al., 2022). Meanwhile, various approaches have been proposed to accelerate DDIM (Kong & Ping. 2021b; Watson et al., 2021; Liu et al., 2022). Bao et al. (2022) improved the DDIM by optimizing the reverse variance in DMs. Watson et al. (2022) generalized the DDIM in DDPM with learned update coefficients, which are trained by minimizing an external perceptual loss. Nichol & Dhariwal (2021) tuned the variance of the schedule of DDPM. Liu et al. (2022) found that the DDIM is a pseudo numerical method and proposed a pseudo linear multi-step method for it. Zhang & Chen (2022) discovered that DDIMs are numerical integrators for marginal-equivalent SDEs, and the deterministic DDIM is actually an exponential integrator for the probability flow ODE. They further utilized exponential multistep methods to boost sampling performance for VPSDE.
Another promising approach to accelerate the diffusion model is distillation for the probability flow ODE. Luhman & Luhman (2021) proposed learn the map from noise to data in teacher-student fashion, where supervised signals are provided by simulating the deterministic DDIM. The final student network distilled able to generate samples with reasonable quality within one step. Sali- mans & Ho (2022) proposed new progressive distillation approach to improve training efficiency and stability. This distillation approach relies on solving the probability flow ODE and needs extra training procedures. Since we generalize and improve the DDIM in this work, it will be beneficial to combine this distillation method with our algorithm for better performance in the future.
Recently, numerical structures of DMs have received more and more attention; they play important roles in efficient sampling methods. Dockhorn et al. (2021) designed Symmetric Splitting CLD Sampler (SSCS) that takes advantage of Hamiltonian structure of the CLD and demonstrated advan- tages over the naive Euler-Maruye Maruyaaa method. Zhang & Chen (2022) first utilized the semilinear structure presented in DMs and showed that the exponential integrator gave much better sampling quality than the Fuler method. The proposed Diffusion Exponential Integrator Sampler (DEIS) fur- ther accelerates sampling by utilizing Multistep and Runge Kutta ODE solvers. Similar DEIS, Lu et al. (2022): Karras et al. (2022) also proposed ODE solvers that utilize analytical forms of diffuson scheduling coefficients. Karras et al. (2022) further improved the stochastic sampling algorithm by augmenting ODE trajectories with small noise. Further tailoring these integrators to account for the stiff property of ODEs (Hochbruck & Ostermann, 2010; Whalen al., 2015) promising direction in fast sampling for DMs.
B PROOFS
Since the proposed gDDIM is a generalization of DDIM, the results regarding gDDIM in Sec. are generalizations of those in Sec. 3. In particular, Prop 4 generalizes Prop 1, Eq. (18) general- izes Prop 2, and Prop generalizes Prop 3. Thus, for the sake of simplicity, we mainly present proofs for gDDIM in Sec. 4.
B.1 BLURRING DIFFUSION MODELS
re regarding BDM proposed in Hoogeboom & Salimans (2022) and
We first review the formulations show it can be reformulated SDE in continuous time Eq. (11).
Hoogeboom & Salimans (2022) introduce an forwarding noising scheme where noise corrupts data in frequency space with different schedules for the dimensions. Different from existing DMs, the
13

---

Published as a conference paper at ICLR 2023
diffusion process is defined in frequency space:
( = (y σ.
yt V'
(24)
where a, o are Rdxd diagonal matries and control the different diffuse rate for data along different dimension. And is the mapping of æ; in frequency domain obtained through Discrete Cosine Transform (DCT) V We note V is the inverse DCT mapping and V
Based on Eq. (24), we are able to derive its corresponding noising scheme in original data space
p = (xx Va, V Zo. o.I).
(25)
Eq. (25) indicates BDM a non-isotropic diffusion process. Therefore, we are able to derive its forward process as a linear SDE. For general linear SDE Eq. (1), its mean and covariance follow
dm: Fim.
(26)
dt EE
dΣ,
(27)
Plugging Eq. (25) into Eq. (26), able to derive the drift and diffusion F,, G. for BDM as Eq. (11). As BDM admit a SDE formulation, we can use Eq. (3) to train BDM. For choice of hyperparameters az o, and pratical implementation, we include more details in App. C.
B.2 DETERMINISTIC GDDIM
B.2.1 PROOF OF EQ. (17)
Since we assume data distribution (u)) = N(un, Eo), the score has closed form
v log (u() = - -, -,-
(t,, 0)uo).
(28)
To make sure our construction Eq. (16) is solution to the probability flow ODE, we examine the condition for R. The LHS of the probability flow ODE is
du
(t, 0) + R (t, 00 utd dt + Rjedt
(t, O) + R,c]dt.
(29)
The RHS of the probability flow ODE
Fuu 1 V....'looppiuu)) F. 0)uo + F.R.c + G.G.T R.
2
2
F, (t, 0)uo + F&R,E + G.G R T R Redltt,
(30)
where the first equality is due v log pu() = - R. Since Eqs. (29) and (30) holds for each e, we establish
R = (F, G.G.T R RT R
(31)
2
(F. >
(32)
B.2.2 PROOF PROP 4
Similar to the proof of Eq. (17), over solution (u(t), t} the probability flow ODE, R, (u(()
(t, 0)wn) is constant. Furthermore, by Eq. (28),
v logt)(u(t))
- (u(t) - (t, 0)uo)
=R T ( (t, 0)ur)
(33)
Eq. (33) implies that R log (u(())) is constant and invariant with respect to f.
14

---

Published as a conference paper at ICLR 2023
B.2.3 PROOF OF EQs. (12) AND (18)
We derive Eq. (18) first.
The update step is based on the approximation (o(u, = co(u((),() for T E [t - -t, The resultant ODE with Eg reads
? F } co
(34)
which is a linear ODE. The closed-form solution reads
u(t;-1) = V(tz-1, + 1/, 2
(35)
where (t, s) is the transition matrix associated F,, that is, satisfies d (t, s) = (t, (ssss) ID.
(36)
dt
When the DM is specified he DDPM, derive Eq. (12) based on Eq. (18) by expanding the coefficients in Eq. (18) explicitly as
s1,
V as
vtt (t,t)
0t-At
at
1 V/(L - A1, (,,)),, R
Ol At d 0r
dr
2
2
V
Ot
dr
ar
ar
Br-ae
1 1
V0t-A11
at
aa
-V 11-04-40
04-A1
Oily
0t-At
B.2.4 PROOF OF MULTISTEP PREDICTOR- PREDICTOR-CORRECTTR
Our Multistep Predictor- Correctr method slightly extends the traditional linear multistep Predictor- Corrector method to incorporate the semilinear structure in the probability flow ODE with an expo- nential integrator (Press et al., 2007; Hochbruck & Ostermann, 2010).
Predictor:
For Eq. (7), the key insight of the multistep predictor is existing function eval- eg eu(u(ti+g-1), fi+q-1) and their timestamps 4i 1........ li+q-1 1 to fit q - order polynomial (t) approximate e0(u(7), With this approximator T) = <p(T) for 5 E [t), the multistep predictor step is obtained by solving
uations co(u((z)), ti), eatuut
Fuu + G. (o((,)) du
dt
2
Fuu+ G, R, (T))
(
(37)
which is a linear ODE. The solution to Eq. (37) satisfies
z L G. ey(ddd..
u(t;11) = ((t(-n,)))((t)) +
(38)
Based Lagrange formula, we write ep (T)
lith Eoo titk
(x))=
(39)
t+4j
= Kgj
15

---

Published as a conference paper at ICLR 2023
Plugging Eq. (39) into Eq. (38), we obtain
u(t;-1)= 1 ((4-1,)))
Σ r (((((t+)))tts)))))
(40)
PC = -/ 1111 1 ((th-,, R
litk
Idt,
(41)
tith
which are Eqs. (19a) and (19b). Here we use "C to emphasize these are constants used in the q-step predictor. The 1-step predictor reduces to Eq. (18).
Corrector:
Compared with the explicit scheme for the multistep predictor, the multistep corrector behaves like an implicit method (Press et al., 2007). Instead of constructing Ep(T) to extrapolate model out- put for T [f,-1.t]] as in the predictor, the q step conrector aims find <e(T) to interpolate (u(t++1), .... , (u(f:ta-2) (2tt2))tt)))) (itg-2) and their timestamps co co(u(t;-1); co(u(t;), ti), eo (,tt
1--1, b, 4441," " litg-2. Thus, u(tz-1) is obtained by solving
u(t = (tt, (t) + f.
G, G. R,",
(42)
2
Since <c(r) is defined implicitly it is Instead, practitioners eo (ū((tt1)) 1), 14-1). eo (u((t), eo (a(f++1), t++1). ... eg ( 22) where ū(t;-1) is obtained by the predictor in Eq. (38) and a(t+) u(t;), a(t++1) = u(t+t1), ... ((t+tg22)
easy
find <a(T), u(li-i).
bypass
the
difficulties by
interpolating
(1++g-2). Hence, we derive the update step for corrector based on
u u(t4_1) V0 (ttti1,)))((t))
G, R, R, R
(43)
2
11
where ET(T) is defined
9-2
ce(T) = Σ
T litk
(co(uenng t4ts).
(44)
tits - titk
1=11
kgj
Plugging Eq. (44) into Eq. (43), we reach the update step for the corrector
u(t+-1) ((tr-1,t))(tt))
rc ca (o(t)))))t44))
(45)
cc(a) = pi 11 ",, R.
Litk ldr.
(46)
tit t
k=
We use ccl) to emphasis constants used in the g-step corrector.
Exponential multistep Predictor-Correctoor
Here present the Exponential multistep one step corrector update step after an update step of the q-step predictor. The interested reader can easily extend the idea employ multiple update steps of corrector or different number of steps for the predictor and the corrector. We note coefficients "C, "C can be calculated using high resolution ODE solver and used everywhere.
algorithm. Specifically, we employ
16

---

Published as a conference paper at ICLR 2023
Algorithm Exponential multistep Predictor-Correcte
Input: Timestamps {4, {4:)/=0> step order y coefficients for predictor update PC, coefficients for corrector update OC
Instantiate: u (tx) (
for i in N - 1,... do
predictor update step
Ouur min(g, N + 1) handle warming start, use lower order multistep method
- f Simulate Eq. (40) with Qeu-Step predictor
# corrector update step
Yeur min(g, N - + 2) handle warming start, use lower order multistep method
u(t u(t,).-.. u uttt+ae--1) u(t--1), u(tt).... u(tr+gen-1)
-t + Simulate Eq. (45) with Stup corrector
end for
B.3 STOCHASTIC GDDIM
B.3.1 PROOF PROP 5
Assuming that the data distribution (o()) is Nuo, Eo) with a given Eo, we can derive the mean and covariance of (r(u)
1t = (1(,))
(47)
dE, F,Ex + F. + G.G.P
(48)
dt
Therefore, the ground truth score reads
v - (t, (t,, 0)uo).
(49)
We assume Eg is given but ug unknown. Fortunately, 24g can be inferred via one score evaluation as follows. Given evaluation V log p. (u(s)), we can recover uo as
Uo (0, s) E V log p. (u(s)) + us)].
(50)
Plugging Eq. (50) and (1(,,)) = (1, V (0, 8) into Eq. (49), recover Eq. (20).
B.3.2 PROOF OF PROP 6
With the approximator EØ (u( t) defined in Eq. (21) for E [s, f). Eq. (6) can be reformulated as
1 X2
du = F, udr +
G, G. R. ( + XG
2
=FF, 1
X2
G, G R R. uutt
2
1 X
G. R. R ((T 8) (R,cou((s)))) u()))dt + XG,du.
(51)
2
Define F, :- L+X G, G. R F, + G. and denote by (t, the
transition matrix associated with it. Clearly, Eq. (51) is a linear differential equation on u, and the conditional probability Par (s) associated with is Gaussian distribution.
Applying Särkkä & Solin (2019, Eq (6.6,6.7)). we obtain the exact expressions
Mean (tt,s)((s)
- 1f( (t, r)
x
(.G.
2
+Iʃ (t, 1 + X²
G.G.T.' V(T, dr c,co(u()),
(52)
for the mean of (u)(((t)
(s))). Its covariance Psr satisfies
dP F, F. + N.G.G P..
(53)
dr
Eq. (52) has closed form expression with the help of the following lemma.
17

---

Published as a conference paper at ICLR 2023
Lemma 1.
| ((,)))
1+12
2 ".G,E,"((T,s)) (t,,s) (tt,s)
Proof. For a fixed s, we define N(t) S- (t, 1+ G. G. and M() (t, 8)
(tt, s). It follows that
(N(() FTN(T) + 1 1
dr
= FF,(() + 1+ G, G. (N()) 8(T,8)
(54)
(T
(, 8) - (T,)(
dr
= =F,((T) 1+ X2
G, (T,
(55)
2
Define E(I) ((1) ((),, then
dE(t) dt
++ = (Fi
G.G. E(t). E(t)
(56)
2
On the other hand, N(s) M (s) = 0 which implies E(s) = 0. We thus conclude E(t) = 0 and (() = (().
Using lemma 1, we simplify Eq. (52) to
V((,))u(s) ()( VV(1, (coss))))
(57)
which is the mean in Eq. (22).
B.3.3 PROOF OF THM
We restate the conclusion presented Thm 1. The exact solution ult - At) to Eq. (6) with coeffi- cient Eq. (8) is
ull - Al) NV
ai-si u(l) + [-V
ar-
a1-At
VI
0r
V 04-A: - of ca co(u((),
at
at
(58)
x²
12
with o7 = ((1 ar- at)
1- 11-(-aao0
L-ac-ar I-ore
ar
which is the same as the DDIM Eq. (9).
Ore
Thm 1 is concrete application of Eq. (22) when the DM is DDPM and F,, G, are set Eq. (8).
Thanks to the special form of F,, V has the expression
0x1 Id. V a²
(t, =
(59)
and V satisfies
log (t, s) [ [ dr
d 0.
1 22 dlog a
1 dr
(60)
2
a,
1112
1/4
(t,
1 -
a2
(61)
1 as
04
18

---

Published as a conference paper at ICLR 2023
Mean:
Based on Eq. (22), we obtain the mean of Pat
1412
04
u(s)
+
04 Og
vI
a,
+
1 Oft
Ots
2
v1
as
E 8)
(62)
1 Ct
22
x²
04
((s)
+
04t
v1
03 + $1
1/1 (1
aa)
1 -----s)
at
(0(2()),
co
axe
ac-si
(63)
a2
uss) +
a1
02 + v - ar - o2
cs(u(s),
(64)
a2
where
12
0 = (1 - ar) 1
at Cx
axs
(65)
Setting (s, + (t,t At), we arrive at the mean update in Eq. (14).
Covariance: It follows from
dP.t. 2 dlog ar
log at 1 a,, 1 X
dlog
P.r
Xa
Idi Poo
dr
2dt
2 dr
dr
that
x2
X27
Pa = (1 at)
1
ax
axa
Setting (s, < (t, - At), we recover the covariance in Eq. (14).
B.3.4 PROOF OF PROP 7
When À = 0, the update step in Eq. (22) from s to reads
u(t = (t, ss)(s) + (t, s) ( s) ())(((())))
(66)
Meanwhile, the update step in Eq. (18) from & to is
utt = (t, ()((s) 1f" _ d,
(67)
Eqs. (66) and (67) equivalent
have the following lemma.
Lemma 2. When X = 0,
2 T (I, R d - - VT R.
Proof. We introduce two new functions
N / 1 [ (t, R,
(68)
M(t) = (1, - (T, x)] R...
(69)
First, (s) ((s) 0. Second, they satisfy
dN(() F.N(1)) 1 G.G. R.
(70)
dt
dM -F,0(,,)) F.V((I,)) R)
(71)
dt
-F.(() 1 G. )1,(
(72)
2
19

---

Published as a conference paper at ICLR 2023
Note L and R satisfy the same linear differential equation
1 G.G 2 (t, [. 2
dR,
-Fit G.G.
(73)
It is a standard result in linear system theory (see Särkkä & Solin (2019, Eq(2.34))) that (t, 8)
R, Plugging it and R, R = Li into Eq. (72) yields
dM(() F.M(( G.G.G. R
(74)
dt
Define E()) (1) ((), then it satisfies
E(s) = dE(() F,E()),
(75)
dt
which clearly implies that E() = 0. Thus, N(I) = M().
C MORE EXPERIMENT DETAILS
We present the practical implementation of gDDIM and its application to BMD and CLD. We in- clude training details and discuss the necessary calculation overhead for executing gDDIM. More experiments are conducted to verify the effectiveness of gDDIM compared with other sampling al- gorithms. We report image sampling performance over an average of 3 runs with different random seeds.
C.I BDM: TRAINING AND SAMPLING
Unfortunately, the pre-trained models for BDM are not available. We reproduce the training pipeline in BDM (Hoogeboom & Salimans, 2022) to validate the acceleration of gDDIM. The official pipeline is quite similar to the popular DDPM (Ho et al., 2020). We highlight the main differ- ence and changes in our implementation. Compared DDPM, BDM scheme Eqs. (11) and (25). The two key hyperparameterss {a,), for follow the exact same setup in Hoogeboom & Salimans (2022). whose details and python implementation can be found in Ap- pendix A (Hoogeboom & Salimans, 2022). In our implementation, we use Unet network architec- tures (Song et al., 2020b). We find our larger Unet improves samples quality. As a comparison, our SDE sampler can achieve FID as low as 2.51 while Hoogeboom & Salimans (2022) only has 3.17 on CIFARIO.
different forward noising
C.2 CLD: TRAINING AND SAMPLING
For CLD, Our training pipeline, model architectures and hyperparameters similar to those in Dockhorn et al. (2021). The main differences are in the choice of K, and loss weights K, A.K,T.
Denote by tu( [eo(u, a), co v)] for corresponding model parameterization. The thors of Dockhorn et al. (2021) originally propose the parameterization so(u, L. co(uu, t) where y; = L, is the Cholesky decomposition of the covariance matrix of por (()) Built DSM Eq. (3), they propose hybrid score matching (HSM) that is claimed to be advanta- geous (Dockhorn et al., 2021). It uses the loss
E E (0)) (o)) [|lc (x))) + Lec, 1,,,
(76)
With similar derivation (Dockhorn al., 2021), obtain the HSM loss with score parameterization so t) L, 60 (u, t) as
E E Ee ((0)) |lle (o( + Rie, (IRR)
(77)
Though Eqs. (76) and (77) look similar, cannot directly pretrained model provided in Dock- horn et al. (2021) for gDDIM. Due to the lower triangular structure of L, and the special G,, the solution to Eq. (6) only relies (u, v) and thus only eo fa(u, l; v) is learned in Dockhorn
20

---

Published as a conference paper at ICLR 2023
et al. (2021) via special choice of A4. In contrast, in our parametrization, both co co(u, t; x) and c0(u, t; v) are needed to solve Eq. (6). To train the score model for gDDIM, we R, R, I for simplicity, similar to the choice made in Ho et al. (2020). Our weight choice has reasonable performance and we leave improvement possibilities, such mixed score (Dockhorn et al., 2021), better A, weights (Song et al., 2021), for future work. Though we require a different training scheme of score model compared with Dockhorn et al. (2021), the modifications to the training pipeline and extra costs are almost ignorable.
We change from K L, K, = R.. Unlike L, which has a triangular structure and closed form expression as (Dockhorn et al., 2021)
E
Eee
L,
with Σ,
(78)
we rely on high accurate numerical solver to solve Rs. The triangular structure of Le and sparse pattern of Gt for CLD in Eq. (10) also have an impact on the training loss function of the score model. Due to the special structure of G., we only need to learn so u, t; v) signals present in the with L,"" -T being upper triangular. Thus we only need to train <g (Le) (u,, (; to recover so(u, 4; In contrast, R2 does not share the triangular structure L₁ both Eo Therefore, Dockhorn et al. (2021) sets loss weights
velocity channel. When K; Le, se(u, t) -L, L
(Rt) tu, 2x) cg
(R.)
(u, t; are needed to recover sø(u, t; e)
0
0
D, A,,,
=
0 1
@
la.
(79)
while we choose
[ 0
R,
2 Ia-
(80)
As a result, we need to double channels in the output layer in our new parameterization associated with R. though the increased number of parameters in last layer is negligible compared with other parts of diffusion models.
We include the model architectures and hyperparameters in Tab. 4. In additional to the standard size model on I0, we also train a smaller model for CELEBA to show the efficacy and advantages of gDDIM.
Table 4: Model architectures and hyperparameters
Hyperparameter Model
CIFARIO
CELEBA
EMA rate
0.9999
0.999
# of ResBlock per resolution
8
2
Normalization
Group Normalization
Group Normalization
Progressive input
Residual
None
Progressive combine
Sum
N/A
Finite Impluse Response
Enabled
Disabled
Embedding type # of parameters Training # of iterations
Fourier
Positional
~ 108M
R 62M
1m
150k
Optimizer
Adam
Adam
Learning rate
2x10-4
2x 10
Gradient norm clipping
1.0
1.0
Dropout
0.1
0.1
Batch size per GPU
32
32
GPUs
4 A6000
A6000
Training time
79h
& 16h
21

---

Published as a conference paper at ICLR 2023
C.3 CALCULATION CONSTANT COEFFICIENTS
In gDDIM, many coefficients cannot be obtained in closed-form. Here we present our approach to obtain them numerically. Those constant coefficients can be divided into two categories, solutions ODEs and definite integrals. We remark that these coefficients only need to be calculated once and then can be used everywhere. For CLD, each of these coefficient corresponds to a 2 x matrix. The calculation of all these coefficients can be done within min.
Type I: Solving ODEs The problem appears when we need to evaluate R, in Eq. (17) and (t, s)
in
(1, = (t,,, (s,,s)= =1 dt
(81)
Across our experiments, we use RK4 with a step size 10-6 to calculate the ODE solutions. For (t, s), we only need to calculate (t, 0) because (t, s) = (t, (), 0)] In CLD, F,, G2, R., Σg can be simplified to a tremely fast. We note (t s) and Σ, admit close-form formula (Dockhorn et al., 2021). Since the output of numerical solvers are discrete time, we employ linear interpolation to handle query in continuous time. Since R. V/ are determined by the forward SDE in DMs, the numerical results can be shared. In stochastic gDDIM Eq. (22), we apply the same techniques to solve P.c. In BDM, F; can be simplified into matrix whose shape align with spatial shape given images. We note that the drift coefficient of BDM is diagonal matrix, dimensional ODE. Thanks to parallel computation GPU, solving multiple one dimensional ODE efficient.
matrix; solving the ODE with small step size is ex-
decompose matrix ODE into multiple one
Type II: Definite integrals
The problem appears in the derivation of update step in Eqs. (18), (19b) and (46), which require coefficients such as PC(9) =c!") We use step size 10 for the integration from t to At. The integrand can be efficiently evaluated in parallel using GPUs. Again, the coefficients, such as PC!() <c(") are calculated once and used afterwards we need sample another batch with the same time discretization.
C.4 GDDIM WITH OTHER DIFFUSION MODELS
Though we only test gDDIM in several existing diffusion models, including DDPM, BDM, and CLD, gDDIM can be applied to any other pre-trained diffusion models long the full score function available. In the following, we list key procedures to integrate gDDIM sampler into general diffusion models. The integration consists of two stages, offline preparation of gDDIM (Stage I), and online execution of gDDIM (Stage IT).
Stage I: Offline preparation of gDDIM Preparation of gDDIM includes scheduling timestamps and the calculation of coefficients for the execution of gDDIM.
Step 1: Determine an increasing time sequence T {t,}
Step 2: Obtain (1, s) by solving ODE Eq. (81).
Step 3: Calculate R, by solving ODE Eq. (17)
Step 4: Obtain p((g) (() via applying definite integrator solvers on Eqs. (18), (19b) and (46)
How to solve ODEs Eqs. (17) and (81) and definite integrals Eqs. (18), (19b) and (46) has been discussed in App. C.3.
Stage II: Online execution of gDDIM Stage II employs high order El-based ODE solvers for Eq. (82) with K, = R, We include pseudo-code for simulating El-multistep solvers in Algo 1. It mainly uses updates Eq. (40) and Eq. (45).
22

---

Published as a conference paper at ICLR 2023
C.5 MORE EXPERIMENTS ON THE CHOICE OF SCORE PARAMETERIZATION
Here we present more experiments details and more experiments regarding Prop 1 and differences between the two parameterizations involving R, and Lt.
Toy experiments:
Trojetorry [u(ti]
(-T
T-T
t-T
Figure 3: Trajectory and of Probability Flow solution in CLD
R
H1 K.
Lr
11
K.=
Einles
NFE-13 15
NFE-22 25
NFE-35 35
NFE-60 60
NFC-20 200
Figure 4: Sampling challenging 2D example with the exact score V log (t)).. where data distribution is a mixture of Gaussian with small variance. Compared with Euler, algorithms based on Exponential Integrator (El) Eq. (82) have much better sampling quality. Among El-based sam- plers, different K, for score parameterization e(u,() - K log pr (uu) have different sampling performances when NFE small. Clearly, R. proposed by gDDIM enjoys better sampling quality given the same NFE budget.
Here present empirical results to demonstrate the advantage of proper K,,
In VPSDE, the advantage DDIM has been verified various models and datasets (Song et al., 2020a). To empirically verify Prop a mixture two one dimension Gaussian distributions. The ground truth V log (u()) known this toy example. As shown Fig. along the solution probability flow ODE, the score parameterization e(u, 1) = - V log Pe (w) /1 as enjoys smoothing property. We remark that R, = L, VI, VI- I, in VPSDE, and thus gDDIM is the same as DDIM; the differ- ences among R.. Lt, v E, only appear when Σ, is non-diagonal.
present toy example where the data distribution is
23

---

Published as a conference paper at ICLR 2023
FID / IS different NFE
K;
20
30
40
50
L
461.72 1.18
441.05 1.21
244.26 2.39
120.07 5.25
R.
16.74/ 8.67
9.73 8.98
6.32/9.24
5.17
L
368.38
1.32
232.39/2.44
99.45/4.92
57.06 77.06/6.70 6.70
R.
8.03
4.26/9.49
3.27
2.67 /9.65
L
463.33/ 1.17
166.90 3.556
4.12/9.25
3.31/9.38
R.
3.90/9.5
2.66/9.64
2.39 9.74
2.32/9.75
L
464.36 1.17
463.45/1.J1
1.17 463.32 1.17
240.45 2.29
R2
332.70 1.47
292.31/ 1.70
13.27 / 10.15
2.26 / 9.77
Table 5: More experiments on CIFAR10
FID at different NFE
q
K2
20
30
40
50
L. 446.56
430.59
434.79
379.73
0
R2
37.72
13.65
12.51
8.96
L
261.90
123.49
105.75
88.31
R:
12.68
7.78
5.93
5.11
L 446.74
277.28
6.18
5,48
2
R.
6.86
5.67
4.62
4.19
L 449.21
440.84
443.91
286.22
3
R
386.14
349.48
20.14
3.85
Table 6: More experiments on CELEBA
In CLD, we present a similar study, As the covariance E, in CLD is no longer diagonal, we find the difference of the L, parameterization suggested by (Dockhorn et al., 2021) and the R, parame- terization is large in Fig. 3. The oscillation of (c(L.)) prevents numerical solvers from taking large step sizes and slows down sampling. We also include a more challenging 2D example to illustrate the difference further in Fig. We compare sampling algorithms based on Exponential Integrator without multistep methods for a fair comparison, which reads
At
ult - At) = vt(l ()(() + +1f.
(82)
where (K.) (u, (2,
le Though have the exact score, sampling with Eq. (82)
will not give us satisfying results we use K, L, other than R2 when NFE is small.
Image experiments:
We present more empirical results regarding the comparison between L, and RŞ. Note that we use exponential integrator for the parametrization L, well, similar to DEIS (Zhang & Chen, 2022). We vary the polynomial order in multistep methods (Zhang & Chen, 2022) and test sampling performance CIFARIO and CELEBA. In both datasets, we generate 50k images and calcualte their FID. As shown in Tabs. 5 and 6, R, has significant advantages, especially when NFE is small.
We also find that multistep method with large q can harm sampling performance when NFE is small. This is reasonable; the method with larger assumes the nonlinear function is smooth in large domain and may rely on outdated information for approximation, which may worsen the accuracy.
C.6 MORE EXPERIMENTS THE CHOICE OF
To study the effects of X, visualize the trajectories generated with various À but the same ran- dom seeds in Fig. 5 on our toy example. Clearly, trajectories with smaller À have better smoothing property while trajectories with large À contain much more randomness. From the fast sampling per- spective, trajectories with more stochasticity are much harder to predict with small NFE compared with smooth trajectories.
24

---

Published as a conference paper at ICLR 2023
DDPM
CLD x)
= 0.0
0
A 0.3
> 0.5
= 0.7
= 1.0
Figure 5: Sampling with various and accurate score on a Toy example. Trajectories are shown from T to O.
We include more qualitative results on the choice of À and comparison between the Euler-Maruyama
(EM) method and the gDDIM in Figs. 8 and 9. Clearly, when NFE is small, increasing À has a negative effect on the sampling quality of gDDIM. We hypothesize that À = 0 already generates high-fidelity samples and additional noise may harm the sampling performance. With fixed number of function evaluations, information derived from score network fails to remove the injected noise as we increase A. On the other hand, we find that the EM method shows slightly better quality as we increase X. We hypothesize that the ODE or SDEs with small À has more oscillations than SDEs with large X. known that the EM method has a very bad performance for oscillations systems and suffers from large discretization error (Press et al., 2007). From previous experiments, we find that ODE in CLD is highly oscillated.
We also find both methods perform worse than Symmetric Splitting CLD Sampler (SSCS) (Dock- horn et al., 2021) when À 1. The improvement by utilizing Hamiltonian structure and SDEs structure significant. This encourages further exploration that incorporates Hamiltonian structure into gDDIM in the future. Nevertheless, we also remark that SSCS with = 1.0 performs much worse than gDDIM with À 0.
C.7 MORE COMPARISONS
We also compare the performance of the CLD model we trained with that claimed in Dockhorn al. (2021) Tab. 7. We find that our trained model performs worse than Dockhorn al. (2021) when a blackbox ODE solver or EM sampling scheme with large NFE are used. There may be two reasons. First, with similar size model, our training scheme not only needs to fit V. log Pe (u), but also Vx log (u)), while Dockhorn al. (2021) can allocate all representation resources of neural network to V, Po().. Another factor the mixed score trick on parameterization, which shown empirically have boost in model performance (Dockhorn et al., 2021) but we do not include it in our training.
We also compare algorithm with more accelerating sampling methods in Tab. 7. gDDIM has achieved the best sampling acceleration results among training-free methods, but still cannot compete with some distillation-basee acceleration methods. In Tab. 8, we compare Predictor-only method with Predictor-Correcte (PC) method. With the same number of steps N, PC can improve the quality of Predictor-only at the of additional N - score evaluations, which is almost two times slower compared with the Predictor-only method. We also find large g may harm the sam- pling performance in the exponential multistep method when NFE is small. We note high order polynomial requires more datapoints to fit polynomial. The used datapoints may be out-of-date and harmful sampling quality when we have large stepsizes.
C.8 NEGATIVE LOG LIKELIHOOD EVALUATION
Because method only modifies the score parameterization compared with the original CLD (Dockhom et al., 2021), we follow a similar procedure to evaluate the bound of negative log-likelihood (NI..). Specifically, can simulate probability ODE Eq. (7) to estimate the log-
25

---

Published as a conference paper at ICLR 2023
Table More comparison on CIFAR10. FIDs may be reported based on different training tech- niques and data augmentation. It should not be regarded as the only evidence to compare different algorithms.
Class
Model
NFE (
HID (L)
CLD-SGM (gDDIM)
50
2.26
our CLD-SGM (SDE, EM)
2000
2.39
CLD
CLD-SGM (Prob.Flow RK45)
155
2.86
our CLD-SGM (Poo,,Flow, RK45)
312
2.26
CLD-SGM (Prob.Flow. RK45) by Dockhorn al. (2021)
147
2.71
CLD-SGM (SDE, EM) by Dockhorn et al. (2021)
2000
2.23
DDIM by Song et al. (2020a)
100
4.16
Training-free Accelerated Score
Gotta Go Fast by Jolicoeu-Martineau et al. (2021b)
151
2.73
Analytic-DPa DPM by Bao et al. (2022)
100
3.55
FastDPM by Kong & Ping (2021a)
100
2.86
PNDM by Liu al. (2022)
100
3.53
DEIS Zhang & Chen (2022)
50
2.56
DPM-Solver by Lu et al. (2022)
50
2.65
Rescaled 2nd Order Heun by Karras et al. (2022)
35
1.97
DDSS Watson al. (2022)
25
4.25
training-needer
Progressive Distillation by Salimans & Ho (2022)
3.0
Accelerated Score
Knowledge distillation by Luhman & Luhman (2021)
9.36
LSGM by Vahdat et al. (2021)
138
2.10
Score+Others
LSGM-110M by Vahdat al. (2021)
131
4.60
Diffusion GAN by Xiao et al. (2021)
3.75
FID at different steps N
Method
20
30
40
50
Predictor
16.74
9.73
6.32
5.17
Predictor
8.03
4.26
3.27
2.67
PC
6.24
2.36
2.26
2.25
Predictor
3.90
2.66
2.39
2.32
2
PC
3.01
2.29
2.26
2.26
Predictor
332,70
292.31
13.27
2.26
PC
337.20
313.21
2.67
2.25
Table 8: Predictor-only vs Cooreetor (PC). Compared with Predictor-only, PC adds one more correcting step after each predicting step except the last step. When sampling with step, the Predictor-only approach requires score evaluation, while PC consumes 2N NFE.
26

---

Published as a conference paper at ICLR 2023
likelihood of given data (Grathwohl et al., 2018). However, diffusion model models the joint distribution pl uo) = p(z0, vo) on test data zo and augmented velocity data Vo- Getting marginal need integrate vo for each zo. To circumvent this issue, Dockhorn et al. (2021) derives lower bound on the log-likelihood,
distribution p( (2) from p(un) is challenging,
log p(x)))))))))))) - log( / p(
p(Zu, vo) ddv)
p
2 E ppp ((oo vo)l + H(p(vo)):
where (p(vo)) denotes the entropy of p(vo). We can then estimate the lower bound with the Monte Carlo approach.
Empirically, our trained model achieves a NLL upper bound 3.33 bits/dim, which is comparable with 3.31 bits/dim reported in the original CLD (Dockhorn et al., 2021). Possible approaches to further reduce the NLL bound include maximal likelihood weights (Song et al., 2021), and improved training techniques such as mixed score. For more discussions of log- likeelllioood and how to tighten the bound, we refer the reader to Dockhorn et al. (2021).
C.9 CODE LICENSES
We implemented gDDIM and related algorithms in Jax. We have used code from number of sources in Tab. 9.
CRL
Citation
License
Song et al. (2020b)
Apache License 2.0 NVIDIA License
https://github.com cwm/ htwp:/hww..coom/mmla/a/////////a
Dockhorn et al. (2021)
httpa://github. com/ wwt.:shommaccccaaee
Zhang & Chen (2022)
Unknown
Table 9: Code License
27

---

Published as a conference paper at ICLR 2023
dp
83
7 -=
==b b
NFE=10
NFE-20
NFE-30
NFE=50
-
NFF=10
NFE-20
NFE-30
NFE=50
Figure 6: Comparison between L (Upper) and R (Lower) with exponential integrator on CIFAR
28

---

Published as a conference paper at ICLR 2023
2 CD
1= b
NFE=10
NFE-20
NFE-30
NFE=50
== =b
2
NFF=10
NFE-20
NFE-30
NFE=50
Figure 7: Comparison between L (Upper) and R (Lower) with exponential integrator on CELEBA.
29

---

Published as a conference paper at ICLR 2023
NFE=50
NFE=30
NFE-20
A= 0.0
A=
A 0.3
== 0.5
NFE=50
NFE=30
NFE-20
0.0
0.1
0.3
À 0.5
Figure 8: Comparison between EM (Upper) and gDDIM (Lower) on CIFAR1O.
30

---

Published as a conference paper at ICLR 2023
NFE=50
NFE=30
NFE-20
A= 0.0
A=
==
A= 0.5
NFE=50
NFE-30
NFE-20
A 0.0
A 0.1
0.3
À = 0.5
Figure 9: Comparison between EM (Upper) and gDDIM (Lower) on CELEBA.
31


Reviews:
[{"review_id": "qlznWfZA-X", "reviewer": "Reviewer_PjHX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission deals with DDIMs. It proposes a generalization of DDIMs to general diffusion models beyond the isotropic denoising by modifying the parameterization of score. It links DDIMs to SDEs by a score approximation, and provides an interpretation for the success of DDIMs. It validates the results for BDM and CLD settings, where it achieves strong FID scores for CIFAR-10 sample generation under a small number of function evaluations compared with the original samplers. ", "summary_of_the_review": "This paper addresses an important and timely problem. It proposes a theoretically grounded gDDIM framework that generalized DDIM beyond isotropic denoising, and provides insightful interpretations for the good performance of DDIMs. The experiments also show significant improvement over original samplers. ", "strength_and_weaknesses": "Strength\n- This is the first work to generalize DDIMs (gDDIM) using a theoretically grounded framework\n- It provides insightful interpretations for the DDIMs that explains why ODE samplers perform well\n- The reported experiments show strong improvements over the original samplers\n- The paper is well written and contains sufficient details \n\nWeakness\n- The results could be more solid if tested with more realistic datasets beyond the toy CIFAR-10 such as for LSUN and ImageNet\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fCR8xst3om", "reviewer": "Reviewer_vCLj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explains that the exact solution can be numerically found in DDIM with fast sampling.\nExtending this explanation, this paper proposes \"generalized DDIM\" that modifies parameterization of the score networks of existing diffusion models to achieve 20x acceleration and comparable or much better FID even on non-isotropic diffusion models.", "summary_of_the_review": "This paper shows non-trivial advance of diffusion models in identifying principles of DDIM and enabling fast sampling. Though its content is valuable, the presentation can be nicer to be easier to follow.", "strength_and_weaknesses": "\nStrengths\n\n(+) The proposed generalized DDIM is versatile to different diffusion models.\n\n(+) Step-by-step logical flow: Dirac distribution to Gaussian distribution to real distribution, deterministic case to stochastic case, etc.\n\n(+) Straightforward illustration for understanding ideas.\n\nWeaknesses\n\n(-) Connection from Dirac distribution to the real distribution appears at the bottom of page 6. Bringing it to the top of Section 3 and describing \"empirical distribution with weights vs Dirac = real distribution vs one sample\" would provide the whole picture to the readers.\n(-) Missing link: how do we move from Gaussian and to real distributions? I do not find assumptions or proofs.\n\nMisc.\n(-) Please leave vspace for readability, e.g., Figure 1.\n(-) Please place figures close to its connected paragraphs, e.g., Figure 1.\n(-) In Table 4, please fix the typo \"# of arameters\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZGGOcljFReG", "reviewer": "Reviewer_85mY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a theoretical framework that extends DDIMs, which were derived only for DDPMs, to a more general form of diffusion SDEs.", "summary_of_the_review": "Overall, the contribution of this work is simple yet significant. While the proposed framework for building generalized DDIMs relies on several somewhat unrealistic assumptions (see first point in Weaknesses), these choices are at least partially justified by the promising empirical performance of the resulting samplers.", "strength_and_weaknesses": "Strengths:\n- The work proposes a generalized framework for deriving accelerated samplers from standard diffusion models.\n- The proposed framework performs well empirically, and allows one to make informed choices on how the learned score function should be reparameterized (e.g. CLD appears to be much improved in the low NFE scheme by choosing $K_t = L_t$).\n\nWeaknesses:\n- While the derivation produces the original DDIM, when applied to DDPM models, it involves several steps that make questionable assumptions (e.g., Section 3 assumes unperturbed data is a Dirac distribution on a single point, Section 4 assumes data is normally distributed.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O3fTl094c6", "reviewer": "Reviewer_4f5G", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a generalized framework based on DDIM that explains the acceleration led by DDIM and can be extended to non-isotropic diffusion models like Blurring diffusion model (BDM) and Critically-damped Langevin diffusion model (CLD). The authors build up their insights using a toy dataset consisting of only one data point, in which the parameterization of DDIM can produce smooth network output along trajectories and thus enable large step size to achieve sampling acceleration. Then this insight is generalized to real dataset by considering the manifold hypothesis and a mixture of Dirac distributions. As an application of this framework, authors first rewrite non-isotropic BDM and CLD into gDDIMs and show the acceleration achieved by the smoothed estimation along the trajectory.", "summary_of_the_review": "This paper is overall interesting but not appealing enough. Its merits and weaknesses are both clear. Authors make great efforts to provide a comprehensive analysis. However, there are numerous weaknesses that limit the scope of this paper. Based on the authors' response, I am happy to update my score accordingly.", "strength_and_weaknesses": "Strengths:\n\n1. The analysis toward understanding why DDIM accelerates the sampling process is well organized and inspiring. Following that, because complexity can be decomposed into simple cases, extending and interpreting the results in real datasets is natural. (even though in a less rigorous way) The research methodology is of merit, starting from a clear analysis of the simple and synthetic case and finally carefully verifying this on realistic datasets.\n2. Given the fact that authors train an adjusted CLD, (which is not training free to achieve the best results), the final results are pretty good.\n\nWeaknesses/Concerns:\n\n1. To apply the acceleration techniques, one first needs to formulate irregular diffusion models into a regular form provided by this paper and make a correspondence between less structured generalized diffusion model designs with parameters and formulations in this framework, which incurs an invisible bar for many practitioners. This is somehow not an off-the-shelf acceleration method.\n2. Authors spend a notable portion explaining the hyperparameters in CLD and the correspondence between the original CLD and newly designed in this paper, like the $L_t$ and $R_t$. In this case, the gDDIM is not a training free method. To achieve the best acceleration, training from scratch is necessary, which actually changes the model design for CLD.\n3. The writing of the introduction can be polished much further. Without good logical guidance, it is a bit confusing for readers to see many less structured references to the method section. The authors should compress the related works in the introduction, move this part to the related works, and use the additional space here to organize and present the *logic* of this paper.\n\nMinor comments:\n\n1. Related works should be presented in the main paper.\n2. The definition of $L_t$ should be introduced earlier in the main paper, as this is discussed and compared in the experiments but not clearly stated.\n3. It should be $N$ score evaluation in Table 8, not $n$ score evaluation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%