SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models
Authors: Martin Gonzalez
Nelson Fernandez
Thuy Vinh Dinh Tran
Elies Gherbi
Hatem Hajri
Nader Masmoudi
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Probabilistic Models
Exponential SDE methods
Image Generation
Generative Models
TL;DR: We propose a collection of versatile sampling SDE methods for Diffusion Models, with proven convergence guarantees, achieving optimal quality sampling faster than previous SDE methods.

Abstract: A potent class of generative models known as Diffusion Probabilistic Models
(DPMs) has become prominent. A forward diffusion process adds gradually noise
to data, while a model learns to gradually denoise. Sampling from pre-trained
DPMs is obtained by solving differential equations (DE) defined by the learnt
model, a process which has shown to be prohibitively slow. Numerous efforts on
speeding-up this process have consisted on crafting powerful ODE solvers.
Despite being quick, such solvers do not usually reach the optimal quality
achieved by available slow SDE solvers. Our goal is to propose SDE solvers that
reach optimal quality without requiring several hundreds or thousands of NFEs
to achieve that goal. We propose Stochastic Explicit Exponential
Derivative-free Solvers (SEEDS), improving and generalizing Exponential
Integrator approaches to the stochastic case on several frameworks. 
After carefully analyzing the formulation of exact
solutions of diffusion SDEs, we craft SEEDS to analytically compute the linear
part of such solutions. Inspired by the Exponential Time-Differencing method,
SEEDS use a novel treatment of the stochastic components of solutions,
enabling the analytical computation of their variance, and contains high-order
terms allowing to reach optimal quality sampling $\sim3$-$5\times$ faster than previous
SDE methods. We validate our approach on several image generation benchmarks,
showing that SEEDS outperform or are competitive with previous SDE solvers.
Contrary to the latter, SEEDS are derivative and training free, and we fully
prove strong convergence guarantees for them.

SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models
Martin Gonzalez" IRT SystemX
Nelson Fernandez Air Liquide
Thuy Tran IRT SystemX
Elies Gherbi IRT SystemX
Hatem Hajri IRT SystemX & Safran
Nader Masmoudi New York University
Abstract
A potent class of generative models known as Diffusion Probabilistic Models (DPMs) has become prominent. A forward diffusion process adds gradually noise data, while model learns gradually denoise. Sampling from pre-trained DPMs is obtained by solving differential equations (DE) defined by the learnt model, process which has shown to be prohibitively slow. Numerous efforts on speeding-up this process have consisted on crafting powerful ODE solvers. Despite being quick, such solvers do not usually reach the optimal quality achieved by available slow SDE solvers. Our goal is to propose SDE sol solvers that reach optimal quality without requiring several hundreds or thousands of NFEs to achieve that goal. We propose Stochastic Explicit Exponential Derivative-free Solvers (SEEDS), improving and generalizing Exponential Integrator approaches to the stochastic case on several frameworks. After carefully analyzing the formulation of exact solutions of diffusion SDEs, we craft SEEDS to analytically compute the linear part of such solutions. Inspired by the Exponential Time- time--ifffrrneingggg method, SEEDS use a novel treatment of the stochastic components of solutions, enabling the analytical computation of their variance, and contains high-order terms allowing to reach optimal quality sampling ~ 3-5x faster than previous SDE methods. We validate our approach several image generation benchmarks, showing that SEEDS outperform or are competitive with previous SDE solvers. Contrary to the latter, SEEDS derivative and training free, and fully prove strong convergence guarantees for them. Our code is publicly available in this link
Introduction
Diffusion Probabilistic Models (DPMs) 32 12 have emerged as a powerful category of generative models and have proven to quickly become SOTA for generative tasks such image, video, audio and more 229 36. These models employ a forward diffusion process where noise is gradually added to the data, and the model learns remove the noise progressively. However, sampling from most pre-trained DPMs is done by simulating the trajectories of associated differential equations (DE) and has been found be prohibitively slow [34]. Previous attempts to accelerate this process have mainly focused on developing efficient ODE solvers. On one hand, training-baat based methods speed-up sampling by using auxiliary training such as Progressive Distillation 31 and Fourier Neural Operators [42, learning the noise schedule, scaling, variance, or trajectories. are slower but are more versatile for
generation 8 13 3
the other hand, training-fres methods [6
Corresponding author: wtw.hih.gonzalleiit--s1-e-e fr
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

being employed on different models and achieve higher quality results than current training- based methods. Although these solvers are fast, they often fall short of achieving the optimal quality attained by slower SDE solvers [16]. The latter usually do not present theoretical convergence guarantees and, while being training-free, they often still require costly parameter optimization to achieve optimal results which might be difficult to estimate for large datasets.
Our objective is to introduce SDE solvers that can achieve optimal quality without requiring an excessively large number of function evaluations (NFEs). To accomplish this, we propose Stochastic Explicit Exponential Derivative-fre Solvers (SEEDS). These are off-the- off-the-shelf SDE samplers: they offer promising high-quality sampling without further training or parameter optimization. SEEDS enhance and generalize existing Exponential Integrator [22) [3] [1) approaches to the stochastic case on several practical frameworks and is based on the following 4 building blocks: (1) the exponential representation of semi-linear SDE exact solutions which isolate linear terms to be computed analytically; (2) a general change-of-variabbe in the solutions in order to better approximate the deterministic one: (3) method to analytically compute the variance of the stochastic one; (4) method to decompose the obtained stochastic components such a way that the resulting sequences of higher-stage numerical approximations yield Markov chains.
SDE
recipe to simplify the integrals involved
Overall, we make the following contributions: (a) our change-of-varre varviibel method allow us to re- frame the gDDIM solver [41 as special case of SEEDS and to craft bespoke solvers for the EDM-precondttioned DPMs in [16] which attain equivalent sampling quality twice faster than the previous SOTA sampling method [16]; (b) based on the Stochastic Exponential Time-Diffeen Lifeeeiffrreniin method, we analytically compute of our solver's stochastic components terms of the so-called o-ffncctons, allowing for efficient implementation; (c) our noise decomposition method (4), which is the key of success of SEEDS, is both theoretically grounded, experimentally shown to be optimal, and has no deterministic equivalent; (d) we provide full proofs of strong/weak convergence guarantees for our SDE solvers which, to our knowledge, has no precedent in the DPM literature. although the formula used for the truncated Ttylor expansion might seem similar to that of (2]. our convergence theorems leverage the full Itô- TayTaylor expansion of solutions, making our proofs not incremental and different from [22]: (c) we conduct extensive experiments demonstrating that SEEDS establishes SOTA results among available solvers on several image generation benchmarks, o is competitive with existing SDE solvers while being 2-5 times faster than the latter.
particular,
Although our solvers theoretically apply to certain non-isotropic DPMs such as Critically-dampe Langevin Dynamics (CLD) [0] (see Prop. 4.5 and Rem. 4.6), we will restrict our presentation to the isotropic case for which many notations become simpler.
2 Background on Diffusion Probabilistic Models
General Isotropic DE Formulation. The evolution of a data sample Xg E Rdd taken from an unknown data distribution Pasta into standard Gaussian noise can be defined forward diffusion process {xtheeo with T > 0, which is a solution to linear SDE:
f(t) = log ar
dlo g(t) ar dt
dx,= f(t)x,dt + g(t)dwr,
(I)
dt
where g() E Raxdd called the drift and diffusion coefficients respectively and is d- dimensional standard Wiener process, and a4 de E RD00 are differentiable functions with bounded derivatives. In practice, when specifying the SDE a acts as schedule controlling the noise levels of an input at time t, and O₂ as time-dependent signal scaling controlling its dynamic range.
By denoting P. (x.) the marginal distribution of x, at time 6, functions ar and a; are designed that the end-time distribution of the process process is pr (xT) R NN(x](0, d²La) for some ā > O. As (I) is linear, the transition probability Por (x)) from Xo to is Gaussian whose mean and variance can be expressed in terms of a and ot. For simplicity, we will denote it
Por (xx) xo) = - (xx;; 1480, Σx), Mt, Ex E Rddd
The evolution of the reverse time process of Jx50,11 (which we will still denote [xe]eejo,]] for simplicity) is then driven by a backward differential equation
1+ 6
dx, [f()),
g (t) log (x;))t lg(t)d
(2)

---

where df are negative infinitesimal time-steps and W. is now a Wiener process with variance ddt. In this article, we will concentrate the cases & 0, 1, known the literature as the Probability Flow ODE (PFO) and diffusion reverse SDE (RSDE), respectively.
Training. Denoising score-matching a technique to train a time-dependent model Dx(x,, t) to approach the score function Vx, (x.)) at each time t. Intuitively, as Dg approaches the score, it produces a sample which maximizes the log-likelihood. As such, this model is coined as a data prediction model. However, in practice DPMs can be more efficiently trained by reparameteriz- ing Dg into a different model Fe (x,, t) whose objective is to predict the noise to be removed from a sample time t. This noise prediction model is trained by means of the loss
1o.-2100.T/xxa-PPaaa - | Fo(1xx0 + K,c,t)lls Ky k, k,,,,,
where i/ time dependent weighting parameter and K,K,T Ex.
3 Accelerating Optimal Quality Solvers for Diffusion SDEs
Once Fg or Dg have been trained, its corresponding expression involving either one of these models. For instance, taking the noise prediction model and = sampling is conducted by simulating trajectories of SDE of the form
effectively solve 2 after replacing the score function by
dx: [A()), + (()) () + +())dw,,
for some functions A(t). b(t) which are usually not equal to g(t). In what follows, we consider a time discretization M going backwards in time starting from to T to tM 0 and ease the notation we will always denote < s for two consecutive time-steps f++1 < t4.
The usual representation of the analytic solution x, time of B w.r.t initial condition x, is:
X = x, + J. [A(T)x b(T)) ] + J. g(T)dw,
The numerical schemes we propose for approaching the trajectories of based representation are grounded in the 4 following principles:
1. The variation-of-pan parameetrss formula: representing analytic solutions with linear term ex- tracted from the integrand;
2. Exponentially weighted integrals: extracting the time-varying linear coefficient attached to the network from the integrand by means of a specific choice of change of variables which allows analytic computation of the leading coefficients in the truncated Itô-Taylor expansion associated (x,, T) up to any arbitrary order;
3. Modified Gaussian increments: after replicating such change of variables onto the stochastic integral, analytically computing its variance.
4. Markov-preservee noise decomposition: stochastic integrals need to be dependent on overlapping time intervals and independent on non-overlapping ones.
The first key insight this work is that, using the variation-of paraametesss formula, we can represent the analytic solution x; time of 3) with respect to an initial condition x, as follows:
Exponential representation of exact solutions of diffusion SDEs.
= (tt, + J: Qa (a((,)))(77 + S. (a(t,))g(T)de)
(5)
where st) exp A(7)dr) is called the transition matrix associated with A(t). The separation of the linear and nonlinear components is achieved by this formulation and also appears in [22 23 41. differs from black-box SDE solvers as enables the exact calculation of the linear portion, thereby removing any approximation errors associated with it. However, the integration of the nonlinear portion remains complex due to the interaction of the new coefficient a 7)b(T) and the intricate neural network, making it challenging to approximate.
3

---

Exponentially weighted integrals. Due to the regularity conditions usually imposed on the drift and diffusion coefficients of ( one can make several choices of Change-of-variable on the integral components in (5) in order to simplify it. Our second key insight is that there is specific choice of change variables allowing the analytic computation of the Itó- Taylor coefficients of (x 7) with respect to 7, and based at that will be used for crafting SEEDS. More specifically, this expansion reads
s) (x,, 8) Rn.
Fx
where the residual R., consists of deterministic iterated integrals of length greater than 1 and all iterated integrals with at least one stochastic component. As such, we obtain
[ u ((lt)))) Tdd
S:d (A(G,))((T)
(T (T-s) dr + Rn,
Σ k=U
F (x..
(6)
kd
where R₂ is easily obtained from R, and S (4, T)((() The third key contribution of our work is to rewrite, for any h > O, the integral J 1 A +)((T) (-= (r dr as an integral of the form Pir e (1- dλ since the latter is recursively analytically computed in terms of the ye-functions
(1) = E,; 9P++ (1) = f (1-T)e dr (Px(t) (xx(())
A 420.
b!
t
Modified Gaussian increments. In order for making such change of variables to be consistent on the overall system, one needs to replicate it accordingly in the stochastic integral f A (t, ())(r) As such, our last key contribution is to transform it into an expomentially weighted stochastic integral with integration endpoints A. ) and apply the Stochastic Exponential Time Differeneing (SETD) method 1 to compute its variance analytically, as illustrated in (14) below.
Let us test our methodology in two key examples. As we explained in Section sampling from pre-trained DPMs amounts on choosing schedule at, scaling at, and parameterized learnt approximation of the score function Vx log (xx). In what follows, we denote by 1x the inverse of chosen change of variables d and denote &2 ((), (o A) == (A)x (A)).
The VPSDE case.
Let Og Bat + Bmt, where Sa, 3m > 0 and E [0, 1). Then, by denoting
04 :=:=0 1:
Vx. log,,(x;) = o. Fo(x,,
we obtain the VP SDE framework from 34 and the following result.
Proposition 3.1. Let t < s. The analytic solution at time t of the RSDE with coefficients and
initial value x2 is
at X. 200e Lmm - A)dA V2au
Xx
dww
A - log(o)).
(8)
Oss
The change of variables of 8 is interesting as it allows to compute analytically the Itô- ié-Taylor coefficients in (6) by using, for d -'s9 the following key result which will be used in Prop. 4.2
de ee (2 -x, dA x 1
For instance, in the case when & 0, is easy to see that / -> or(e" - and Sote -d obeys normal distribution with zero mean, and one can analytically compute its variance:
of 2h 1).
2d
(10)
2

---

The EDM case.
Denote og the variance of the considered initial dataset and set
1
x
toa
X
log(t)
oo == L, == 1, Vx. lo (x:) ~
Fe + to
(11)
12
12+0
4
These parameters correspond to the preconditioned EDM framework introduced in [6 Sec. 5, App- B.6]. The following result is the basis for constructing our customized SEEDS in this case, and for which we report experimental results Table For simplicity, we will write Fo(X,, t) for the preconditioned model in (1I) and we refer to Appendix B for details.
Proposition 3.2. Let < s. The analytic solution at time t of 2 with coefficients and initial value Xa is, for
1 o
X 2(1 + + //d FoGR,,, (2(())-2((22__) oz) / dw>
X, = 82 + oo
(12)
where d == : llo
In the case when = 0, it is given by
OA 12-
x, + v oo //m Ae 3² + toa
12 + o
X1
c 1)d1, d := log arctan
(13)
0d
Remark 3.3. One can wonder about the generality of such change of variables. Our method is very general in that
always make such change of variables with very mild regularity conditions:
for [0, 7] + -200 integrable, with primitive C(t) > O, we have e(t) clog(e(t)) This means write c(t) C(t) ed d with d llog(C(t)). In other words, for such c, we have
[ c(F)dr = [ e² cA- = S/a ed
4 Higher Stage SEEDS for DPMs
In this section we present our SEEDS algorithms by putting together all the ingredients presented in the previous section. Let < s. all what follows, we consider the analytic solution time of the RSDE (2) with coefficients (7), h = X X and initial value x,. Plugging (9) with k = 0 and (IO) into the exact solution allow to infer the first SEEDS scheme, given by iterations the form
X
0s X,
Ao) -1)
20, 1)P
o. 2 le,
L()
(14)
The following Theorem gives strong order convergence guarantees for this method, which we call SEEDS-1, under mild conditions which apply to all our experiments. We stress out that its proof (App. C is a non-trivial result, is fundamentally different in nature from [22] and involves mathematical tools which have deterministic counterparts.
Theorem 4.1. Under Assumption C.I the numerical solution x, produced by the SEEDS-1 method converges to the exact solution X, of
dx, [f(t)x; + g² (1)o, Fø(xe, t)]dt + g(t)dwr, (a, 1/ō,)
(15)
with coefficients Mw..-MeannSaaee -quare sense with strong order 1.0:
there is constant C' > such that
sup [R xel? Log1g1
& Ch,,
as h 0.
Higher stage SEEDS. As announced, by fully exploiting the analytic computations enabled by the expansion 9 we now turn into crafting our multi-step SEEDS. Usually, SDE solvers are constructed by using the full Itô Hé-Taylor expansion of the SDE solutions and usually need a big number of evaluations of the network F to achieve higher order of convergence. As our main concern is to present stochastic solvers with minimal amount of NFE, we choose truncate such Itô-Taylor expansion that the neural networks only appear in the deterministic contributions.
5

---

Algorithm 1 Iterative procedure
Algorithm 2 SEEDS-I(Fg, s,,
Input: initial value xI, steps [4 model Fo Initialize Xtp f XI for i- = to M- 1 do
(0,1
X 0 23, (c - Fo( -. 1 12
h to X,- SEEDS- 8.
(t) + ti-
end for
Return X + last-step( 11,
Algorithm 3 SEEDS-2(Fo, R,, s,t)
f- (2 + 2) (- c- ((0,dd) @ (0.
oi 20,, c 1 Fo( 8) A,
A Foz Ven Iz
n,
R: "i x, 20. (u, 81) B. : = ōr(Ve2h ehaz² + Veh 12²)
Algorithm 4 SEEDS- X, 8, 8,t) with 0 < r1 < 12 <1
81 1a (A+++1rhh
+ +rh), 82 + (A, r2h), (2) 22, 2² N(0, Id)es
11 f "1x8, 2, (erth 1) F Fo R) Jss v /2rrch 72
A 2r 2r + Ve 1
u2 < " 25. (e 1) (o( 8) - 2 00g
c'2h -1) (Fo(un, - -Fo( s))) - A
rzh
B o.(v th Zrztiz! 1 + v @2rgh e22rtig2 + Vearih - T20)
X1 R. 2 (c - 1) Fa(ãe.;) 5) 2 ch - 11 (2)) - Fol B
Proposition 4.2. Assume that Fg is a C2n+1 C -function with respect to A. Then the truncated Itô- Itō-Taylor expansion of reads, for ~ NN
a
X; ag
x, 25
h + Pktt (h) (z) 8.v -ee Rn+1,
(16)
K=
with ((())
A) (x(x,,))) A), with LA is an infinitesimal operator defined in Appendix E.2.2
and Ru+1 consists the usual deterministic residue and all iterated integrals of length at greater or equal to in which there is at least one stochastic component among them.
Our approach for constructing derivative- free 2-stage and 3-st SEEDS schemes consists on exploiting the analytic computation of the Itô- Itô-Taylor coefficients in Proposition 4.2 and replace the (>(k) (xx, A) terms by well-adapted correction terms which do not need any derivative evaluation and dropping the R,+1 contribution as in the Runge-Kutta approach.
Markov-preserving noise decomposition. We use collocation methods for constructing higher- stage derivative-fiee ffree solvers. Although the chosen truncated Itô-Taylo expansion produces approxi- mations for the deterministic integral similar to (22), adding the corresponding noise contribution found by the SETD method at each step does not yield Markov chains in general. The reason is that stochastic integrals on overlapping time intervals need to be dependent, a phenomenon that has no deterministic counterpart. As such, our last and key element to construct SEEDS consists on a novel decomposition of stochastic integrals which enforces the Markov property for multi-stage SEEDS.
Algorithms T to 4 prescribe all SEEDS schemes obtained by this procedure in the VP case. We now show (see App. for the proofs) that all methods yield Markov chains and are weakly convergent. Proposition 4.3. The sequences }e induced by the choice of stochastic noise contributions presented in Algorithms3 I satisfy the Markov property.
Corollary 4.4. Under Assumption C.2 the numerical solutions 5n produced by the SEEDS methods (3) and 44) converge to the exact solution X, of (15) with coefficients (7) in weak sense with global order 1 in both cases: there is constant C' > 0 such that, for any continuous bounded function G:
E[ M1 & Ch.

---

Comparison with existing sampling methods. Let us now examine the connection between SEEDS and existing sampling techniques used for DPMs, emphasizing the contrasts between them.
The main distinctive feature of SEEDS is that they are off-the-shelf solvers. This means that, not only they are training-free, contrary to (9, but they do not require any kind of optimization procedure to achieve their optimal results. This is in contrast to methods such as: gDDIM, which is training- free but not off- thf-the-shelf as one needs to make preliminary optimization procedures such as simulating the transition matrix of their method in the CLD case; Heun-Like method from EDM (for all baseline models and the EDM -pttiiiid models for ImageNet) since they need preliminary optimization procedures on 4 parameters which actually break the convergence criteria. Moreover, neither gDDIM, EDM nor the SSCS method in [T0] present full proofs of convergence for their solvers. Also, both DEIS and gDDIM identify their methods with stochastic DDIM theoretically, but the poor results obtained by their stochastic solvers do not yield to further experimentation in their works. a way, SEEDS can be thought as improved and generalized DPM-Solver to SDEs. Nevertheless, such generalization is not incremental as the tools for proving convergence in our methods involve concepts which are exclusive to SDEs. We now make rigorous statements of the above discussion.
Proposition 4.5. Consider the SEEDS approximation of
(15) with coefficients 7 Then
1. If we set g 0 in 15 the resulting SEEDS do not yield DPM-Solver.
2. If we parameterize (5) in terms of the data prediction model Dg. the resulting SEEDS are not equivalent their noise prediction counterparts defined in Alg. 1o
3. The gDDIM solver [4 Th. 1] equals to SEEDS-1 in the data prediction mode, for l 1.
The first point makes it explicit that SEEDS are not incremental based on DPM-Solver. The second point in Prop. 4.3 is analog to the result in Appendix B of [23), where the authors compare DPM- Solver2 and (DM--Solvr+++(2S) that is the noise and data prediction approaches, and find that they do not equate. The last point exhibits gDDIM as a special case of SEEDS-1 for isotropic DPMs.
Remark 4.6. Building solvers from the representation of the exact solution in ( requires computing the transition matrix (1, 8), which cannot be analytically computed for non-isotropic DPMs such as CLD 10. Nevertheless, the SEEDS approach can be applied in this scenario in at least two different ways. On the one hand, the SSCS method from [TO] resides splitting ( s) into two separate terms. The first can be analytically computed. The second describes the evolution of comi-linear differential equation [10] Eq. 92]. While [10] approximates the latter by the Euler method, crafting exponential integrators for approximating such DE may yield an acceleration of the SSCS method. On the other hand, gDDIM [41 proposes an extension of DEIS sampling [40] to CLD by setting pre-sampling phase 41 App. C.4) in which they compute an approximation of 4 at,, s) in order to apply their method, and the latter was shown in Prop. [4.5 to be a special case of our method.
Unfortunately, the authors did not release pre-trained models in [41], and the latter are not the same as those in 10 Sampling in this scenario may also benefit from our approach.
5 Experiments
We compare SEEDS with several previous methods on discretely and continuously pre-trained DPMs. We report results of many available sources, such as DDPM [1], Analytic DDPM 22. PNDM 20). GGF [1). DDIM 33 gDDIM [1). DEIS 40 and DPM-Solved
22 Although we do not include training-based schemes here, we still included GENIE 9 which trains a small additional network but still solves the correct generative ODE at higher-order. For each experiment, we compute the FID score for 50K sampled images on multiple runs and report the minimum along different solvers. Details on model specifications and experiment illustrations are shown in Appendix F
Practical considerations. For continuously trained models, SEEDS use the EDM discretization [66] Eq. 5] with default parameters and does not use the last-step iteration trick, meaning that the last iteration of SEEDS is trivial. For discretely trained models, SEEDS use the linear step schedule in the interval [Aeo- )ty] interval following [22] Sec. 3.3, 3.4]. All the reported SEEDS results were obtained using the noise prediction mode. We conducted comparative experiments on SEEDS for both the data and noise prediction modes and found better results with the latter (see Tab. 3 for details). EDM solvers [16] Alg. 2] depend on four parameters controlling the amount of noise to
7

---

Table I: Sample quality measured by FID4 on pre- pre-trained DPMs. We report the minimum FID obtained by each model and the NFE at which it was obtained. For CIFAR, CelebA and FFHQ, we baseline pre-trained models 34 16]. For ImageNet, we the optimized pre-trained model from 16 *discrete-time model, "cntinuous-time" model, "recomputed FID for the non-deep model.
SAMPLING METHOD
FID
NFE
CIFAR-I0# VP-LNCOND.
SAMPLING METHOD
FID4
NFE
DDIM 33
3.95
1000
INALYTI--DDPM 2
3.84
1000
CELEBA-64 VP-UNCOND.
GENIE [
3.64
SNALYTIC-DDPM 2
5.21
1000
ANALYTIC-DDIE DDIM
2
3.60
200
DDIM 33
4.78
200
F-PNDM (LINEAR)
20
3.60
250
DDPM 12
3.50
1000
DPM-SOLVER'
3.48
GDDIM 4
3.85
50
F-PNDM (COSINE)
20
3.26
1000
ANALYTIC- DDIM
3.13
1000
DDPM [12]
3.16
1000
3-DEIS 40
2.95
SEEDS-3 (OURS)
3.08
201
F-PNDM (LINEAR)
20
2.71
250
CIFAR-IO VP-COND.
DPM-SOLVER 22
2.71
36
DPM-SOLVER 22
3.57
195
SEEDS-3 (OURS)
1.88
EDM (Sehum
16
2.48
35
FFHQ-64" VP-UNCOND.
SEEDS-3 (OuRS)
2.08
DPM-SOLVER' [22
3.52
CIFAR-IO" VP-UNCOND.
SEEDS-3 (OURS)
3.40
150
DPM-SOLVER'
2.59
EDM (S 0) 16
3.39
79
GGF [S]
2.59
180
IMAGENET-66 64 EDM-COND
GDDIM 41
2.56
100
DPM-SOLVER 22
3.01
270
DEIS ę3KUTTA 40
2.55
50
EDM (S
0) 16
2.22
511
EULER-MAL MARUYAMA
33
2.54
1024
SEEDS-3 (OURS)
1.38
270
STOCHASTIC EDM
[6
2.54
1534
EDM (OPTIMIZED)
1.36
511
SEEDS-3 (OuRS)
2.39
165
EDM (OPTIMIZED)
16
2.27
511
FID
FID
FID
1PM Sacvon
SEEDS.1 (Dus)
Sacvend
5.0
DE DEEOE
2.6
2.4
EDM 511 NFE
SPEDSS NFR
RUM: STLNEE
SE3015-2 (Dkas)
4.5
2.2
4.0
SEEDS-3 (Duns)
3.95
2.0
1.8
3.0
3.58
3.5
3.5g
2-46
3.4
3.0
3,03
1.4
1.36
1.2
NHH=50
100 150 200
250 300 350 400
NHk=16
322
128
256
512
1024
Saam m0
10
20
30
40
(a) Uncond. CIFAR, VP
(b) Uncond. CIFAR, VP
(c) Stochasticity on ImageNet-64
Figure 1: (a-b) Comparison sample quality measured by FID SEEDS, DPM-Solver and other methods for discretely trained DPMs on CIFAR 10 with varying number function evaluations. (c) Effect Seau on SEEDS-3 (at NFE = 270) and EDM method (at NFE 511) on conditioni ImageNet-64, bascline ADM model. EDM preconditioned model.
be injected in specific sub-interval of the iterative procedure. We consider three scenarios: when stochasticity is injected along all the iterative procedure, we denote it stochastic EDM, when no stochasticity injected we denote EDM (Seburn = and we denote EDM (Optimized) the case where such parameters were subject to an optimization procedure. To better evaluate sampling quality along the sample pre-trained DPM, we recalculate DPM-Solver for sampling from the non-deep VP continuous model on the CIFAR-10 dataset. All implementation details can be found in Appendix D
Comparison with previous works. In Table we compare SEEDS with other sampling methods
for pre-trained DPMs, and report the minimum FID obtained and their respective NPE. For each of reported pre-trained models in CIFAR-10, CelebA-64 and ImageNet-64, SEEDS outperform off-the-shelf methods in terms of quality with relatively low NFEs. For the discrete pre-trained DPM on CIFAR-10 (VP Uncond.) it is ~ 5x faster than the second best performant solver. Additionally, SEEDS remain competitive with the optimized EDM sampler. For ImageNet-64, it is nearly as good

---

discoyp, schodulewyp. scaling-vp
discwedm, schedulewyp. scalingeap
discmedi,, schedule=linear, scaling-none
35
SEEDS-3
Pixel state
STEP-D
Stochastic EDM
Pixel stile
STEP-0
10
20
30 50
30
40
50 10
20
30
Trajectories of 10 pixels (R channel) sampled from SEEDS (Ist line) and Stochastic EDM (2nd line) on the optimized pre-trained model [16] on ImageNet64. Schedule=scaling=vvp corresponds the VP coefficients in 7 and schedule=linear, scaling=none the EDM coefficients (11). We use the time discretizations disc=vp (linear) and disc=edm given in [6 Tab.
Figure 2:
the optimized EDM sampler while being almost twice faster than the latter. Figure (a) compares the FID score of SEEDS and DPM-Solver with varying NFEs. While DPM-Solver methods stabilize faster very low NFE regime, our methods eventually surpass them. Interestingly, after reaching their minimum, SEEDS methods tend to become worse at higher NFEs, a fact that is also visible Figure (b), where we notice that such phenomenon also present on other SDE methods. We report in Appendix F the results of our SEEDS methods in the low NFE regime and connect their behavior with their proven convergence rate.
Combining SEEDS with other methods. While being off-the-shelf method, SEEDS be combined with the Churn-like method used in EDM incurring into SDE solvers with an additional source of stochasticity. As done in [16]. evaluate the effect of this second kind of stochasticity, measured by a parameter denoted Sehurn Figure (c) shows that SEEDS and EDM show similar behavior, although SEEDS-3 is twice faster, more sensitive Schurn and quickly achieves compara- ble performance to EDM. This indicates that SEEDS could possibly outperform EDM after proper parameter optimization that will be left for future works. Nevertheless, highlight the fact that obtaining such optimal parameters is costly and might scale poorly.
Stiffiness reduction with SEEDS. In Fig. 2 we illustrate the impact of different choices of discretization steps, noise schedule and dynamic scaling SEEDS and stochastic EDM. We see that choosing the EDM discretization over the linear one has the effect of flattening the pixel trajectories latest stages of the simulation procedure. Also, choosing the parameters [II over those in (7) has the effect greatly changing the distribution variances as the trajectories evolve. Notice that all the perceptually more stable than those from EDM. It would be interesting relate this to the stiffness of the semi-linear DE describing these trajectories, and to the magnitude of the parameters involved in the noise injection for EDM solver amplifying this phenomenon.
SEEDS trajectories
Ablation studies. As said earlier, our principled use of the Chasles rule to enforce independence only on non-overlapping paths for SHEDS-2/3 ensures that the set of resulting iterations of our solvers satisfy the Markov property, is new and is the central key of success of SEEDS. To highlight this,

---

F19
RDD
STEIDS T-Eane
REEDS Naaee
360
307.00
270.85
261.20
180
90
65.06
46.1
22 80
2.63
2.33
2.30
NFE-S
120
150
NFE-36
90
120
150
(a) 2-stage SDE methods
(b) 3-stage SDE methods
Figure 3: Quality (measured by FID increasing NFEs) comparison of SEEDS-2/3 with the enumerated ablation versions of on CIFAR-10 the (baseline) VP conditional framework.
conduct ablation study on how different combinations of the noise components and B in Algorithms and have an impact on the sampling quality of SEEDS.
For simplicity, explain this for SEEDS-2 (Alg. 3). Set (2',22, 20) three independent standard Gaussian random variables. Denote A Js Vehh 1 for the noise contribution in the term. We have the following choices for the noise contribution in &₁:
noise combination B (V2hh chiz + Veh 1s?
SEEDS-2-Naive-I one noise per stage 5, (e eh Veh veh-1 x²
SEEDS-2-Naive-2: one noise per step B = ō (Ve2h eh Ve - z2
SEEDS-2-Naive-3; one noise per integral evaluation B = 5. (Veth - zh +
Veh - Is?
noises in inverse position B = 5 Veth-
1x
Figure 3 experimentally shows that all naive combinations of noises for SEEDS-2 and SEEDS-3 lead to FID/NFE curves with different behavior and a sharp drop of performance both in quality and speed in all of them, a phenomenon having no deterministic parallel.
6 Conclusions
Our focus is on addressing the challenge of training- sampling from DPMs without compromising sampling quality. To achieve this, introduce SEEDS, an off-the-shelf solution for solving diffusion SDEs. SEEDS capitalize on the semi-linearity of diffusion SDEs by approximating simplified formulation of their exact solutions. Inspired by numerical methods for stochastic exponential integrators, we propose three SEEDS schemes with proven convergence order. They transform the integrals involved in the exact solution into exponentially weighted integrals, and estimate the deterministic one while analytically computing the variance of the stochastic integral. We extend our approach handle other isotropic DPMs, and evaluate its performance on various benchmark tests. Our experiments demonstrate that SEEDS can generate images of optimal quality, outperforming existing SDE solvers while being 5x faster. Limitations and broader impact. While SEEDS prioritize optimal quality sampling, they may require substantial computational resources and energy consumption, making them less suitable for scenarios where speed is the primary concern. such cases, alternative ODE methods may be more appropriate. Additionally, as with other generative models, DPMs can employed to create misleading or harmful content, and our proposed solver could inad inadvertently amplify the negative impact of generative AI for malicious purposes.
10

---

Acknowledgments. This work has been supported by the French govermment under the "France 2030" program, as part of the SystemX Technological Research Institute.
References
[1] ADAMU, I. Numerical Approximation of SDEs and Stochastic Swift- Wwift-Hohenberg Equation (PhD Thesis,
Heriot Watt University), 2011.
[2] BAO, F., LI, C.. ZHU, J., AND ZHANG, B. Analytie-DPM: An Analytic Estimate Of The Optimal Reverse Variance In Diffusion Probabilistic Models. ICLR (2022).
[3] CHEN, N.. ZHANG, Y. ZEN, H.. WEISS, R. J., NOROUZI, M. AND CHAN, W. WaveGrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.0071: (2020).
[4] CHEN, N., ZHANG, Y., ZEN, H., WEISS, R. J., NOROUZI, M., DEHAK, N., AND CHAN, W. WaveGrad 2: Iterative refinement for text-to-specch synthesis. arXiv preprint (rXiv:2106,09660 (2021).
[5] CHEN, S., CHEWI, S., LI, J., LI, Y., SALIM, A., AND ZHANG, A. R. Sampling is as Easy as Learning the Score: Theory for Diffusion Models with Minimal Data Assumptions. arXiv:2209. 1225 (2022).
[6] CLARK, J. C.,, AND CAMERON, R. J. The Maximum Rate Convergence of Discrete Approximations for Stochastic Differential Equations. In Stochastic Differential Systems Filtering and Control (Berlin, Heidelberg, 1980), Springer Berlin Hcidelherg, pp. 162-171.
DEBRABANT, K., KVVRRNO, A., AND MATTSSON, Differential Equations. BIT Numerical Mathematics (2021), 381-409.
Runge- Kutta Lawson schemes for Stochastic
DHARIWAL, AND NICHOL, A. Diffusion Models Beat GANs On Image Synthesis. Advances in Neural
Information Processing Systems (2021), 8780-8794.
[9] DOCKHORN, T., VAHDAT, A., AND KREIS, K. GENIE: Higher-order Denoising Diffusion Solvers, arXiv preprint arXiv:2210.05477 (2022).
[10] DOCKHORN, T., VAHDAT, A., AND KREIS, K. Score-Based Generative Modeling with Critically-Damped Langevin Diffuusion In International Conference on Learning Representutions (ICLR) (2022).
[11] GONZALEZ, M., HAJRI, H., CANTAT, L., AND PETRECZKY, M. Noisy Learning of Neural ODEs Acts as a Robustness Locus Widening. In International Conference on Machine Learning (PODS Workshop) (2022).
[12] Ho. J., JAIN, A., AND ABBEEL, P. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems 33 (2020), 6840-6851.
[13] Ho, J., SALIMANS, T., GRITSENKO, A., CHAN, W., NOROUZI, M., AND FLEET, D. J. Video Diffussion Models. arXiv:2204.03458 (2022).
[14] HOCHBRUCK, M., OSTERMANN, A. Exponential Integrators. Acta Numerica 19 (2010), 209-286.
[15] JoLLCCOFUR-MRRTIEENNN A., Lt, K., PiCHÉ-TAILLLEFE TAIRLEFER, R., KACHMAN, T., AND MITLIAGKAS, I. Gotta Go Fast When Generating Data With Score- Based Models. arXiv preprint arXiv:2 14080 (2021).
[16] KARRAS, T., AITTALA, M., AILA, T., AND LAINE, S. Elucidating the Design Space of Diffusion- Beauusion-Based Generative Models. In Proc. NeurTPS (2022).
[17] KLOEDEN, P. E., AND PLATEN, E. Numerical Solution Of Stochastic Differential Equations. Stochastic Modelling And Applied Probability. Applications of Mathematics. Springer, 1992.
[18] KOMORI, Y. Exponential Runge- Kutta Methods for Stiff Stochastic Differential Equations. RIMS (2016). 128-140.
[19] KOMORI, Y.. COHEN, D., AND BURRAGE, K. Weak Second Order Explicit Exponential Runge-Kutta Methods For Stochastic Differential Equations. SIAM Journal Scientific Computing 39, (2017). A2857-A2878.
[20] LIU, L., REN, Y., LIN, 7., AND ZHAO, Z. Pseudo Numerical Methods for Diffusion Models on Manifolds. arXiv preprint arXiv:2202.00 2207.0978 (2022).
[21] LU, C., ZHENG, K., BAO, F., CHEN, J., LI, C., AND ZHU, J. Maximum Likelihood Training for Score-Based Diffusion ODEs by High Order Denoising Score Matching, In International Conference on Machine Learning (2022), PMLR, pp. 14429-14460.
[22] LU, C., ZHOU, Y., BAO, F., CHEN, J., LI, C., AND ZHU, J. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. Advances in Neural Information Processing Systems (2022).
[23] LU, C., ZHOU Y., BAO, F., CHEN, J., LI, C., AND ZHU, J. Solvertr:::+: Fast Solver For Guided Sampling Of Diffusion Probabilistic Models. arXiv preprint arXiv:22 (2022).
11

---

[24] MENG, C.. SONG, Y., SONG, J., WO, J.. ZHU, J.-Y., AND ERMON, S. SDEdit: Image Synthesis And Editing With Stochastic Differential Equations. arXiv preprini arXiv:2108.0107. (2021).
[25] MILSTEIN, G., AND TRETYAKOV, M. Stochastic Numerics for Mathematical Physics. Scientific Computation series, Springer, Berlin, 2021.
[26] NICHOL, A. Q.. AND DHARIWAL. Improved Denoising Diffusion Probabilistic Models. International Conference on Machine Learning (2021), PMLR, pp. $162-8177.
[27] NIE, W., GUO, B., HUANG, Y., XIAO, C., VAHDAT, A., AND ANANDKUMAR, A. Diftusion Models for Adversarial Purification. International Conference on Machine Learning (ICML) (2022).
[28] PENG, S., XU, W. CORNELIUS, C., HULL, M., LI, K., DUGGAL, R., PHUTE, M.. MARTIN, J., AND
CHAU, D. H. Robust Principles: Architectural Design Principles for Adversarially Robust CNNs. arXiv
preprint 2r3i 16258 (2023).
[29] POPOv, V., Vovk, I., GOGORYAN, V., SADEKOVA, T., AND KUDINOV, M. Grad-TTS: A Diffusion Probabilistic Model for -exteto-Speeeh. In Proceedings of the 38th International Conference Machine Learning (2021), M. Meila and T. Zhang, Eds., vol. 139, PMLR.
[30] RÖSSLER, A. Runge-Kutta Methods for the Strong Approximation of Solutions of Stochastic Differential Equations. SIAM Journal on Numerical Analysis 48, 3 (2010), 922-952.
[31] SALIMANS. T., AND HO, J. Progressive Distillation For Fast Sampling Of Diffusion Models. arXiv preprint arXiv:2202.00512 (2022).
[32] SOHL-DICKSTEIN, J., WEISS, E., MAHESWARANATHAN, N., AND GANGULI, S. Deep Unsupervised Learning Using Nonequilibrium Thermodynamics. In International Conference on Machine Learning (2015), PMLR, pp. 2256-2265.
[33] SoNG, J., MENG. C., AND ERMON, S. Denoising Diffusion Implicit Models. arXiv:2010.02502 (2020).
[34] SONG, Y., SOHL-DICKSTEIN, J.. KINGMA, D. P.. KUMAR, A., ERMON, S., AND POOLE, B. Score- Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations (2021).
[35] SZEGEDY, C., VANHOUCKE, V., IOFFE, S., SHLENS, J., AND WOINA, Z. Rethinking the Inception Architecture for Computer Vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 2818 188--82..
[36] VAHDAT, A., KREIS, K., AND KAUTZ, J. Score-based Generative Modeling Latent Space. Neural Information Processing Systems (NewrIPS) (2021).
[37] WANG, Z., PANG, T., DU. C., LIN, M., LIU, W., AND YAN, S. Better Diffusion Models Further Improve
Adversarial Training. In International Conference on Machine Learning (ICML) (2023).
[38] XU, Y., DENG, M., CHENG, X., TIAN, Y., LIU, 7., AND JAAKKOLA, T. Restart Sampling for Improving Generative Processes. arXiv:2306. 14878 (2023).
[39] YAN, H., DU, J., TAN, V. Y. F., AND FENG, J. On robustness of neural ordinary differential equations.
In International Conference on Learning Representations (2020).
[40] ZHANG, Q., AND CHEN, Y. Fast Sampling Diffusion Models with Exponential Integrator. arXiv preprint ArX:v:2204. 13902 (2022).
[41] ZHANG, Q.. Tao, M.. AND CHEN, Y. gDDIM: Generalized Denoising Diffusion Implicit Models. arXiv preprint arXiv:2206.00 055444 (2022).
[42] ZHENG, H., NIE, W., VAHDAT, A., AZIZZADENESHELI, K., AND ANANDKUMAR, A. Fast Sampling of Diffusion Models via Operator Learning, arXiv preprint arXiv:2211.13440 (2022).
12

---

Contents
Introduction
Background on Diffusion Probabilistic Models
Accelerating Optimal Quality Solvers for Diffusion SDEs
Higher Stage SEEDS for DPMs
5 Experiments
6
Conclusions
10
Discussion
14
Detailed Derivation of the SEEDS Design Space
16
The Isotropic General SDE framework
16
B.2
Re-franing and Generalizing Previous Exponential Solvers
18
B.2.1
The VP case
18
B.2.2
Proof of Proposition 3.1
18
B.2.3
Proof of Proposition 4.2
19
B.2.4
Generalization to the remaining data prediction and deterministic modes
20
B.2.5
Proof of Proposition 4.5
21
B.2.6
The VE, DDIM and iDDPM cases
23
B.2.7
The EDM case
23
B.2.8
Proof of Proposition 3.2
24
B.2.9
The SEEDS algorithms in the EDM Noise Prediction case
25
Convergence Proofs
26
Preliminaries
27
Convergence of SEEDS-1
28
C.2.1
Strong Itô Taylor approximation
................................
28
C.2.2
Continuous approximation SEEDS
29
C2.3
Proof of Theorem
31
C22.4
Discrete-time approximation
33
255
Useful Lemmas
34
C
Proof of Proposition 4.3
35
Proof of Corollary|4.4
36
1.411
Convergence of SEEDS-2.
36
C.4.2
Convergence of SEEDS-3
37
D
Implementation Details
38
13

---

D.I
Stabilization of the exponential terms
38
D.2 Noise schedules parameterizations
D.3 EDM discretization
D.4
Final sampling step
40
Reminders on Stochastic Exponential Integrators
40
Dervative-Iree exponential ODE schemes
40
E.1.1
Runge-Kutee Kutta approach
40
1.2
Exponential Runge-Kutte Kutta approach
41
Derivative-free exponential SDE schemes
E.2.1
Strong and Weak Stochastic Runge- Kutta approach
45
E.2.2
Truncated Itô Taylor expansions
45
E.2.
Integral approximations
47
Experiment Details
49
F.1
Pre-trained model specifications
49
F2
Noise vs. Data Prediction approaches
50
Low vs. High stage Solvers
50
Deterministic vs. Stochastic Solvers
51
155
Run-time Comparison
51
F6
Hardware configuration
52
F7
Licences
52
F8
Supporting samples
52
A
Discussion
Why do SEEDS exhibit high FID scores in the low NFE regime? In Appendix F.4 we provide all tables with FID values at increasing NFEs corresponding to Fig. Notice that SEEDS exhibit high FID scores in the low NFE regime. Let us discuss some theoretical facts that might explain this phenomenon. In (38 Theorem 1.], there is a formal explanation on why ODE samplers outperform SDE samplers the small NFE regime and fall short in the large NFE regime. In particular, theoretically shown that, at large step sizes, it is the discretization error that dominates sampling errors while, small step sizes, it the approximation error that dominates it. One can infer that in the large NFE regime, SDE methods (with proven convergence orders) will outperform ODE methods terms of sampling quality. would interesting to see combining SEEDS with the new ideas in (38 might improve SEEDS in the low NFE regime.
Additionally, SDE solvers and ODE solvers may have different sample quality, even the score models are optimal. Indeed, as proven in [21] Appendix B.J. even if the score model has been trained to the optimal score function, the distributions between SDEs and ODEs are still different. This is because the distribution at time T in the forward process is always not exactly standard normal distribution as in the reverse SDE/ODE process, and thus the distributions time 0 different.
Finally, 38 5) show that the overall generalization error divides into discretisation and approximation errors, indicating the theoretical role of the sampling error contribution to DPM's performance. This is also consistent with the suggestion in [16] Section 5], stating that more diverse datasets continue benefit from stochastic sampling rather than deterministic sampling.
Do SEEDS maintain good performances for higher resolution image generation? training- free and optimization-free SDE solvers, SEEDS naturally maintain good performances in higher
14

---

resolution images in the realm of unguided image generation (unconditional and conditional). For unconditional generation using Latent Diffusion Model, SEEDS are able to generate good quality images already at 100 NFEs. Now, for guided image generation, higher-stage SEEDS will, as expected, see their performance sharply drop as the guidance scale grows for the same reasons DPM-solver do (see [23] for details). Yet, SEEDS-1 still maintains high quality sampling in this scenario. In Appendix F.8 Fig. |5 exhibits a 512" image generated with SEEDS- at 90 NFEs on Stable Diffusion with default guidance scale.
Do SEEDS exhibit other distinguishing features beyond generation optimal quality? As SDE solvers, SEEDS have distinctive (although indirect) capability in the realm of adversarial robustness compared to DPM-Solver:
1. For more than 2 years, the leader-board of RobustBench has been dominated by Diffusion- based data augmentation techniques on top of Adversarial Training. The current SOTA [28 37 uses EDM- preconditioned DPMs and need to generate many as 50M images to achieve SOTA robustness results. For ImageNet-64, [37] use the EDM pretrained model and ppiimization-based stochastic EDM sampler for data augmentation, leading to a 5% robust accuracy improvement compared to doing so for the baseline ADM pretrained model. Since SEEDS reach same FID quality as EDM, but twice faster, we believe it will have a positive impact in this domain, making diffusion- based data augmentation schemes more affordable with limited computational capacity.
2. The work (27 uses DPM-based adversarial purification test-time adversarial defense. The idea is to use off-the-shelf DPMs to annihilate the adversarial content in inputs in test-time before feeding it to pretrained classifier. In 27] Table 6], one see that SDE solvers show substantial robustness capabilities compared to ODE solvers.
We'd like stress out that, although indirect, such capabilities are on the side of the sampling methods rather than on the side of intrinsic robustness properties of score-based learning method. More broadly, robustness properties of ML. models determined as neural differential equations (DPMs being in this scope) has been studied in [39
How do SEEDS compare to Stochastic Runge-Kufta methods? Contrary to the ODE case, there are many stochastic Runge- Kutta approaches, usually tailored for SDEs of a specific form. Nonetheless, a common way of distinguishing solvers with same strong order to assign them couples Ps), where p. is the (stochastic) strong convergence order and pa is the resulting (deterministic) order determined if setting g = 0 in the considered SDE i.e. when they are deterministic. For instance, [30 Tab. 6.2] determines solvers with orders (1,1.0) and (2,1.0) respectively and [30 Tab. 6.3] determines solvers SRAI I and SRA3 with orders (2,1.5) and (3,1.5) respectively. Many of these solvers' speed was already tested in [5 Table 3] CIFAR-10 10 (VP). Experimentally, SEEDS-1 shows to be 6.83x faster than the baseline Mullrr-aruyamma scheme.
To our knowledge. the only available strong order SERKs method for SDEs with in-homogeneous diffusion coefficients are the exponential Fuler Buler-Maruyamo (EEM) method [88 and the stochastic RK Lawson (SRKL) schemes 77] In short, the SRKL schemes only compute anal analytically the linear coefficient and use the Integrating Factor (IF) method to approximate the integrals in the representation of the exact solution given by the of variables (see 77 Alg. IJ), one can create exponential integrator versions of many SDE methods. We implemented our own version of the SRKL schemes, which we denote SRKL-1/211 1/2/3, to take into account that a,, a are not constant and used the Integrating Factor method to approximate the integrals in the representation of the exact solution given by the variation-off parameters prriaaitonoo-paammttterss formula and under the VP oechang--of-varibllle variabees. Interestingly, the SRKL schemes stabilize at increasing NFEs but at much higher FID values than their SETD counterparts.
formula. This way, by special change
In Table 2 below, we draw a comparison of SEEDS with current Stochastic RKL, methods (on the À temporal parameter space) on CIFAR-10 10 for the discretely trained DPM in the VP unconditional regime.
Is the proven convergence order of each of the SEEDS methods optimal? The proposed convergence orders for each SEEDS-1/2/3 optimal: this consequence of the general result from [6] about maximum convergence rates for SDE schemes with uncorrelated Gaussian increments.
15

---

Table 2: Comparison of SEEDS with adapted Stochastic RKL methods on CIFAR-IO in the VP unconditional discrete framework.
METHOD \ NFE
10
20
50
100
BEST KNOWN
SRKL-1(A)
332.52
282.96
33.42
8.62
SEEDS-1
303.48
153.21
22.70
7.97
(500 NFE) 3.13
SRKL-2(AA)
475.20
469.64
134.82
7.74
SEEDS-2
476.90
226.70
7.17
3.23
(90 NFE) 3.21
SRKL-3(A)
462.24
376.15
8.36
7.46
SEEDS-3
483.00
428.60
43.30
3.41
(201 NFE) 3.08
The underlying idea, fully detailed in Appendix E is that any solver with strong order 2 1.5 has to account for double stochastic integrals the non -ruturuceeeed Itó- Ito-Taylor expansion. ultimately forcing any SRK-like solver to use correlated random variables (see [30 Tab. 6.3] and [77] more generally). SEEDS avoid this additional complexity but an interesting future avenue would be extend SEEDS to the higher strong order case (and not in the IF approach but the SETD approach) as long as it doesn't 't incur into an explosion of needed NFEs per step to craft such solvers. Another interesting path would be to craft weak second order SERK methods for DPMs (the work [19] addressed this only for homogeneous semi-linear SDES with constant linear coefficients).
B Detailed Derivation of the SEEDS Design Space
In Sections 2 and we proposed simplified presentation the design space of diffusion models and of the ingredients that constitute our proposed SEEDS methodology. In this section, we further develop our presentation a technical manner, making explicit the formalization of our design choices.
B.1 The Isotropic General SDE framework
In Section 2 we presented a parametric family of differential equations (2) driving the generative process for DPMs, based on time-reversing the forward noising diffusion process ( While doing so, we presented two parameters the noise schedule ar and the scaling a, for which the effects on DPMs have been widely studied in [16].
As the shape of the trajectories of ① and (for f 0, 1) are defined by a and ot, start by writing down, for the scaling x, = 0, the scaled generalization of the proposed SDEs [16 Eq. 103] which unifies in single framework the forward and reverse trajectories:
to ddw dy ajoror V 1og ; + afor V x log log (o)) dt + aror
dx
(17)
Following [16], the previous VP, VE, iDDPM, DDIM and EDM frameworks all are unified as different choices of a;, dt, among other choices presented [T6] Tab. 1] and we will use this a basis for all the proofs contained in this Appendix. In particular, forward time means taking x for which the score vanishes in this context. Now set x,
The formulation in involves family of backwards differential equations controlled by parameter E [o. 1] which all yield reverse-timg processes for ( a fact that can be obtained by studying the Pokker-Planck equation for marginals ; of (17).
When = 1, 2 the obtained SDE is known as the reverse SDE (RSDE) and, when l = 0, we obtain an ODE that known as the Probability Flow ODE (PFO):
à x, a.
dx,
of Vx, log logp(X;; dt.
(18)
Now, finding minimum for the loss function in [6 Eq. 51] is formulated as a convex optimization problem. As such, for the ideal model D(x; σi) arg minp X(D; o1), the score function with
16

---

scaled input is expressed as
D(21500) 81
Xx, log p(x
This ideal model usually subtracted by a raw network F in the form of time-dependent precondi- tioning:
((x:)))) == (t)x, + (2())
c(t E R. i= 1...,44
As such, we can express the score function as two parameterizations involving D or F as follows:
0() X (c1(t) 1)8, + ((1(()()))))))) (19)
Vx, 1ogp({((:0))
a,o?
amo?
Let us now denote D/ == (x( for a pre-trained network approximating the ideal denoiser and let D2_- (o(ca(t)&x; ca(t)) be the corresponding score function in the RSDE and the PFO with each of these models yields four disferent differential equations with a neural network as one of their components. These are given, for i 1.2, by
pre-trained network. Substituting the
dx, = [A'(t)x+ + B (1)D,, + g(t)dor.
(20)
dx. = (1)x + (t)+
(21)
When D2, (resp. D2.)) is employed to replace the score function in 17 using 19). the resulting SDE (20) will be called data (resp. noise) prediction neural SDE. Proceeding analogously for the PFO (88 yield two ODEs (2I) which will be called data (resp. noise) prediction neural PFO. The general form of the A' and 3 coefficients determining each of these DEs is as follows:
de +
0t dt
òe
(t))
(+( -20g
(DP NRSDE)
at
Te Ôe B²(1) -20x Tt
å
2 ōt
(1
(1
(1))
(2
(NP NRSDE)
at 0t
A²(I) de
0z
(1) -0t ōz
(DP NPFO)
at.
(1 = du 8t (1 ci)
åz (*(1) - -at c2())
(NP NPFO)
Remark B.I. At first glance, it would seem misleading to differentiate four DEs as these essentially correspond different choices of at, ot, Ci (t)...., Ct)) But the reason why we do so is that, after applying the variation of constants formula, each of these DEs will yield different representation of their exact solutions (see (2) and (23) below). As we will see below, constructing exponential
integrators heavily depends such representation and will show to lead to four different modes of SEEDS solvers, each one showing different behavior and performance for sampling from pre-trained DPMs. As such, we will articulate this difference already at the DE formulation.
For < the variation of constants formulae for NSDEs allows to represent the exact solutions of 20 as
x, = (',,sx, + [ (1)))) + S' ('((,
i=
1,2
(22)
and those for NPFOs
X (1(11 + I ('(()))(())D
i 3,4
(23)
where
+ (t, = exp (L. <'((arr)
(24)
is called the transition matrix associated with (t) and is defined the solution to
i Φ A. (t. A (t,
$ (s. 8) Ia-
Øt
17

---

When (t) is constant and (t) 1, there is well-established literature on exponential ODE and SDE solvers with explicit stiff order conditions and prescribed by different forms of Butcher tableaux. When (1) is not constant, for the expression in (24) to make sense in the usual sense (in terms of exponential series expansion) instead of having to make use of time-ordered exponentials/ Magnus expansions. the f(t) (t) coefficients must satisfy [f(k) (t), (s)] 0. This condition is trivially satisfied here as the (t) considered here are d-dimensional diagonal matrices.
Notice that, if A' * A for some i + J. their associated transition matrices will not be equal. In particular, if A o A2, then the variances of the stochastic integrals in (22) are different for i = and 2. This the first step in explaining the statement in Rem. B.1 and refer the reader to
the proof of Proposition 4.5 where we put into evidence its validity.
B.2 Re-framing and Generalizing Previous Exponential Solvers
B.2.1 The VP case
Let ā₂ (3aT Bm)dT 1att²² + Bmt, where 3a > Bm 0. Set
((t) dlog oz dt
s(() an
/dlo
v =
1āx
1
dt
Vo
+1
Recall that in the VP case, and the noise prediction mode, base of the following ODE
22
construct exponential solvers on the
[(t)x + 5²(1)
dx.
ca (x):t) dt, t [T,0],
(25)
20,
where o, == 0g The ODE 25 identifies with that 16 for the VP case for which the authors identify the preconditioning
(t) c2(() - ca(()
+ ca(t) (M - (o,) (M 1)t.
As such, obtain the following coefficients for the NP NPFO:
A²(t)=
de B²(t)= - agdr.
(t) at
as
and
Do (x(x000)) -xa
(ci(t) 1) + (cs())((cs())
Vx logp(X2;;,))
ao!o
Fo(x, (M - 1)t)
a10z
B.2.2 Proof of Proposition 3.1
First of all, denote F& (xe, (M 1)t) Ea (xe, t).
We have
d log Oi f(t) = dt
22(1= = 20? (dlog or
dlogaa ar
dhe dt
20
e d
dt dt
This way, one can directly relate d with the signal-to-noise ratio SNR(!) = oz// also being used in [2]. As such, SNR(t) is strictly monotonically decreasing time. Thus, the analytic solution to
18

---

yields
S el (tr)an g(T))
l'" (())(r))))((T)) dw(()
= eLi 1(7)dx x +
o, cx(xx dr +
X
(²)
co(x+; + 04 l g(T) dō((r)
at
x, + a1
02
0,07
ar
0i x, a1
20 ddr
Eg (x,, (X,,,) r) +
{ g(T) dw(()
02g
0, r dr
ar
01 x,
T+ dd 201
g(T) dw)() d
Eg (X,, T) +
0+ dr
Ott
0t X,
dhr
V2nr L' ce-r V dr
ddr dōo(T).
201
ca (x, r)dr
0g
dr
By using the change of variables A(t), our equation now reads
d e (5,,, v2ar /mm d))
Oft
X
X; 204
(26)
as
2
Finally, notice that at
V Itc 2x
V 1462x, so that 26 is equivalent to and 0
âs xx. - -àar, /
de
e E012022,, v2ax L.m e dx)
X
This finishes the proof.
B.2.3 Proof Proposition 4.2
Recall that the functions fok are the integrals
*P++ 1 (1) = f//
( 8)1 O
dó.
k!
which satisfy (px(0) .. The truncated Itô- Itô-Taylor expansion of Eo with respect to À reads
Aa)
(x,,,, 2)
(xx, Ax) + +R,+1,
k!
(k) where here Eg
denotes the L operators defined in (58) applied ēg. On the one hand, since σt (e 1), we obtain by iteratively integrating by parts
Ait e Ad
a
(Xxx, Aa) //
)e - Aaa*
e
- col,, = Σ
(k) 6
dd + Ru+2
k!
k=0
σ, n
(1) (Xx,; da)h k+1 pk++ (h) + Ru+2-
0t
k=
On the other hand, we have > 1, d A 0. Note that since the stochastic integrals xS Ar-x,), they are independent
Aot d=a(( are measurable with respect to (A))) on disjoint time intervals by the independence of increments property of Brownian motion. Thus the random variable == Ea,t in our algorithms are independent on disjoint time intervals. We then write
d 0, e --2dA
AWWA))) = N
A
1
V
2h. - 22re, (~(,1)
22
1
0. Øt T + Ti
e, c~..L)
V2V
as a:
02
at
Øt
E2h
1c,
c~ (a),a.
2
19

---

In conclusion, the truncated Itó Itó-Taylor expansion of the analytic expression
X
0t X,
200t
E E8,2, 2jd1 V20,, e d=
(27)
ats
simplifies to
n
X₁
01 X,
20t
Σ
h Pk+1
(n) (Xx,, 2o) - os ee 1e + Rxt2, 6 (O,
0,
This finishes the proof.
B.2.4 Generalization to the remaining data prediction and deterministic modes
Propositions 3.1 and 4.2 consist the first steps for crafting SEEDS solvers in the VP case associated to the noise prediction neural RSDE (NP NRSDE), Generalizing the above procedure for crafting
SEEDS for the modes associated to (22 and yields the following sets of coefficients
de
Ōt B²(() -2axt
(1)
(DP NRSDE)
04
at
at
(1) de
(2 = 20,
(NP NRSDE)
de +
δe
(1) = - Okg 8t
(1)
(DP NPFO)
0
A 04
B αg²
(NPNPFO)
We readily obtain
ōi (1,
(1, o.a0
x -1, (1(
a2
0,
02002
Then, by setting the simpler change of variables d, == - log(a;)) obtain
J 55 (t, (()))) az [," a- a4 1' o,dr -0t //
)r
c -A = - -
a,
By recursion obtain
[ s' ((t)))) - Σ
n-1
pk42 o(h*tl).
In the same way, we obtain
J ('(( = 0:00 S'
-1 ôr dr
0:04 L' - dr
a,0+ o,
of
oia0 (hv
de
Add - -1).
Next,
J 0² (t = a, / a, a, -25,(E" - -1),
ar
and finally, as already shown Propositions 3.1 and 4.2
= J. (r(rt)))))(rrrr= ''
-20, dr oa
-2 ddr 2hr
oz js
dr
ofa; ('v 2Ad -a 22 1).
pit
20

---

Now, since (t) = 20,0,0 the stochastic integrals mean and variances:
for i = 1, 2, have zero
[ (''( = 0. {f oto? ( = =0,(1 - 22
[ (((())))))) oi 20, -0i 1).
We deduce from this the SEEDS-1 schemes in all four modes given by iterates:
da -a(e (x.. 8) + v1 2 e~ NO(,Ia))
x,
(28)
0.a.
x,
X, (ct -1)00(2;;)) σov 22h lc c~ I(,,aa)
(29)
02
x arle - -1)Do (Xs,
(30)
0s
at x, ōz(c" - 11)00(8,, 8).
X
(31)
02
Notice that the iterates
30 and 31 are exactly the iterates of the first stage solvers in 22 and 23
with Fe (xy: (M
1)t) Eo Xx, t). The iterates (29) coincide with the SEEDS-1 method presented
in (14) and 28) consist on SEEDS-1 method in the data prediction mode, which we will use in the following section.
B.2.5 Proof of Proposition 4.5
We will write in bold the statement to be proven.
If set g in (15), the resulting SEEDS solvers do not yield DPM-Solver. Indeed, if we set g O in 15 then the method (29) does not contain a noise contribution and we readily see that it cannot be equal
(31). As the latter has been shown to 3PM-Solver-1, the conclusion follows.
If we parameterize 155 in terms of the data prediction model Da, the resulting SEEDS solvers are not equivalent to their noise prediction counterparts defined in Alg. 1o
One can check that the SEEDS solver are not the same between noise and data prediction modes by simply noticing that the noise contributions in 228 and (29) do not equate.
The gDDIM solver [1 Theorem 1, for
equal to SEEDS-1 in the data prediction mode.
As shown in 29 our proposed method SEEDS- in the data prediction mode for the VP case has iterates of the form
ofa x. oza.
N
ar(e 1)Do e-
(32)
where Ia), o. as9 and h log J. As our notation and that of [4] Theorem 1] σ; overlap, we will use blue color when referring to their notation.
On the one hand, gDDIM constructs iterates over a representation of the exact solution of the following family of neural differential equations:
du, = f(t)u: + 1+ X2
g2(1)
co(u dt + Ag(t)d
(33)
2
-I- at
where dg decreases from ag 1 ar = 0, and with coefficients
f(t) = d at
g(t) = / dlog a
de
dt
In particular, they choose an approximation, for E [t - At, t) given by
su 7) = c (u,
1- at
ar (u((), 1 ar
(u
n(t))
v1 ar
ax
a2
21

---

The gDDIM iterates, for À = L, are then written as follows:
u(t -1) Ar-Ar 0t
u(l) +
Ol- At
+ V or ar of
ca(u(();1) + oze
ar
Okt
(34)
where c ~ (O) and
0 (1-a--r) ar)
1 - ac- 1-
)
at
(35)
Ctt
04-
Now set (s,t) - (t, t At). Then
1 O2
(1 - at)
1
as
u(t)
V
a2 u(s)
v 04
o
VI
ae
cø(u(s), 8) + oge.
Cta
Next, we identify at = Utn and oN
v1
deg
Then the variance of the noise
35 is
2
2
2
27
o?
1
0t
0x
06
0
0i
0g5
27
o/
1
σt
0s
07
Hence, by denoting X, = u(), the gDDIM iterate
34
reads, for ~ ~N(O
0+ x,
+
03 - - - e 2h)
-
0. c0( 8) + 1oo e 22
a3
at
Jo,ee-21. - aros] - ca (x, 8) + V1 c 2-
X, +
ag
01 X,
[o 0o co + 0 -2hc. 0. ot
oz
(36)
ag
On the other hand, the data and noise prediction models in our case are related by the following equation:
X
5)
+ (s)
0,60 (X., 8).
02s
0, one can rewrite 32 in terms eg as follows:
As such, and as h d, ~
log at
of x,
N
az(e 2h
1)Da (x,
8) +0.1 2h
0.as
oia
x
0.05
X
0tt
02
0xE0(8.18)
+ ee-hee
-2h
ofon
of
N, + a4
1
(2E0 8) J. e 2h
ohas
02
03
X, + āt
o
0.
60 (8,88) 8) + 0 2
(As
07
0ig
+ 0:
0:
0
Eo (x, 3) 8, / 2- 2he
02
which coincides with the gDDIM iterate in Equation
36
22

---

B.2.6 The VE, DDIM and iDDPM cases
Following [1]] Eq. 217], here aę = and ea (t) = 1 and so the only possibilities incurring into
semi-linear differential equations are
01 A (L) = 2 1x
82 (1 = -2 0t (²
(DP SDE)
(t) öt
(DP PFO)
dt
di
We obtain, again with the choice A = - log(or) and setting = x - Xę the following:
T (1,88) 0º
of
T (1, 8)
01
0,
J x BB = -o²
de 2d di =
1 (e-th - 1)
de
2
[ B = -01 //m edd = c hh 11
pr
Next,
[ (
of 1 f
-
22 -
o2
dr dr=
| 20 dr 0e o,, 1). o²
of 0+
We readily see that these cases are identical to the VP case with at 1. In particular, the obtained SEEDS-1 iterates are
o x (e-2h 1)Do((xio.) + -o..I-e-
03 x,
at (c- 1)Do Os
x,
Now denote tA + rh). for 0 < & 1, where ta = e There are two families of single-step one-parameter two-stage exponential ODE schemes:
N x, - (c -c -0(-1(((1 2r 1) 0x( + Do Do(X))00)))
0g
2r
1 1 + +1) (0)( (e(x
at X, (e ()5o(((; +
x,
with same supporting value
_ - (e- - (x))o))...)
X
0s
In the same vein we define a single-step two stage exponential SDE scheme:
X
o X,
-2hh
1)Do + t. Ve Ve-2rh 2rh lei
X, = X, (E-22h -(6-2h-1) -10(11221
Do (o(Xx:os)=
+ _ -0(311000.)
o²
2r
2r
to, Ve Ve-2h 2h - -rbhe + Ve- 2rh 162
B.2.7 The EDM case
In the EDM-preconditionee case 16. App. B.6), set o, = and ar We denote d == Odata the variance of the considered initial dataset and we set
o
loa
1
c1())
c2(1)
ca(1)
ca(1) log((),
12 o3
vp + o3'
23

---

so we obtain the following coefficients:
(t) 2
(1)
(DP NRSDE)
(t) 2
o2
2
B1(1) V12 + oz
(NP NRSDE)
1
12 + o
(1 1
(1)
(DP NPFO)
1
A i 12 o²
1 1
(1())
od
(NP NPFO)
12 + 0
In particular, the data prediction neural SDE/PFO are identical to those in the VE with or t.
So let us concentrate on the noise prediction regime, leading us to prove Proposition 3.2
B.2.8 Proof of Proposition 3.2
In the Noise Prediction case, we have
t + 0z
12 +
o
(1,,)
(1,
8² o
8² + 0
so we readily compute:
L @2 (T)) = (12 00)) J.
-2ad dr,
72 + oo +
J. (tttt)) = V + od S Od
dr.
V 7
Let us consider two different changes of variables:
A == log aretan ()) and
;= llg
(37)
day 12 +
that will be used for the (NP NPFO) and (NP NRSDE), respectively. For the former case, we have
64 dt
oadt
~ =
0 +12
1
Therefore, deduce that
S s (T()))
'/ V 72 + o
A+o -0d dr
VT2 + o²
d
+
-0d
dr
72 + oo
v +
ed
( (
v + of aarctann
(c 1).
For the latter case, we have
t
JJ 12 + 0 to
12 + oo
é dddde =
o2 + +0
03
o1(12 + 02)
(12 + 0
12 +
24

---

We then obtain
J. (2(1;T)B
a dr
12 77+02
VT² +0
L ed
2(1 + +og)
21 21 1
(e 1).
The stochastic integral J" (², (r)(d, in noise prediction case is a Gaussian random variable with zero mean and whose variance can be computed by Itô isometry
J. (1 = (o)))))
2rdr
(t2 + +0)
dr
[ + d
(12 + 0z²²
52 + +
t2 + 03
(t + 0 (s - -8)
(s² + +
Putting everything together, we obtain the analytic solution at time of 2 with coefficients 11 and initial value x2 for the (NP NRSDE):
12 12+ 03
X, 2(1 + o 1" (= (2piia--((((11)) + oz) + / e =>>ww
x
(38)
5² 7
where A = -log
For the (NP NPFO). is given by
t2 + 0
1 + Sh Eo A= log
arctan t
+
(39)
52 + o
0
This finishes the proof.
Remark B.2. From the above proof, we immediately deduce the SEEDS-1 iterates in the EDM- preconditioned noise prediction case. These are given, for the (NP NPFO) and (NP NRSDE) respectively, by
X, + V + o arctan (1) (L) (6" - (Xp:
R, =
(40)
3 202
f P+o2
x, + 2t + o² (c 1)) +1
(+2 0z) -t3)
R, =
(41)
x2 to
(s2 + +02)
0d
where Lo)
B.2.9
The SEEDS algorithms in the EDM Noise Prediction case
In the same way we presented the SEEDS algorithms 3 and 4 the VP case, the change-of-vvrialle optimized for the EDM framework in Proposition 3.2 induces [ggorithms33ooo SEEDS-2 and for SEEDS-3 respectively, under the Noise Prediction regime. This is the version SEEDS that we use achieve same performance as EDM solver but with twice less NFEs than the latter for ImageNet-64 (see Table and Fig. (c) in the main part of this article).
25

---

Algorithm 5 SEEDS-2 (EDM NP)
Input: initial value x7, steps [4)]= data prediction model Do. r 1/2 Initialize Xtp f- xI for i = to M 1 do
(t, + (t,, 1)
(x),
100
X(t) - log aaviRto
t(()+
0d f- Odtta
h + de- Ass Asy Ao rh,
81 t (x(,
6o (Rs; 8) < Do 82:07
43 Ez
2240
Xs, + 74od Xs + 2 11
Alyattn2 (e 1)eo (X,, +
(+ x²
(o (+
B Nx 2 X. 80002
20 (c" 1)en (Kin, 81) +
(12 (e -12) 2/ +
oa
(0
end for Return Xt last-step(xear 15 ta8-1
Algorithm 6 3EEDS-3 (EDM NP)
Input: initial value xy, steps (t.) data prediction model Do. r1 = 1/3, r2 = 2/3 Initialize Xip f- xT for i = 1 to M - 1 do (1, + (1,,4-1), (2t,22,²))
N((0,d) Id) 83
X(1) - log "av/Rto?
(x(2)
On f- Odata
h h de As toy As rih,
deg
As r2h,
ta(o))); 82 t (2(22))
eo (R.; - Do 82:055
o7 Sz
60
Xs. + +t Xs + 2
(0
(e
1)ee (Xs, s) +
xe
A
(13+ x 23))
(crihz) + + Pt en (1) Eu (xc,8) 8)
(1f+
2to & + 2
o0 (c 1)co (Rs, 8) +
2r2 22 o2to2
e'2h r2f
- 1) Pi Pi+ A
3u
B
110+0)103-
(x - 2 + Pa + 6o (Razs 82) c6 (x,,3)
(0g (2)
R+og X, 2405
X + B (c - 1)ee (Xe, 8) + 2 Wiito' ( - 1) P2 + B
end for
Return X ( 1
C Convergence Proofs
In this Section, we give detailed proofs of Theorem 4.1 Proposition in the main part of this paper. Let us start recalling its framework. We start by considering the NP NRSDE (15) with VP coefficients:
1x and Corollary 4.4 stated
f()xx+ + 20 ot
dx,
c)(x),)) b) dt g(t)das
(42)
[())x + 2afd1o, co(xxit) dt + g(t)d
ō.
dlloga dl
dloi] dt
x, + 20,ōreo(X1,1) 1) di + as
dw
Denote tA the inverse of X, = - log(o;) (which is a strictly decreasing function of t) and denote λソ == (()) (xx)), X) == (A)) We consider a time discretization (4))24 going backwards in time starting from to = T to ta 0 and to case the notation we will always
M+1
26

---

denote < 8 for two consecutive time- &, < 44-1- The analytic solution at time of the RSDE with coefficients and initial value X, reads
Ae /,m 2,
(ellx,,)dh V2ar L/m e d=)d((A).
tre
att X, 20z a2
X₁
=
(43)
Given initial condition Xea Xx, the SEEDS-1 iterates read, for h4 do, de- de Aet
0t x
20,(e ()e s) o, 2.V le e e~N(O,
x
ags
Assumption C.I.
1. The function 69 t) is continuous (and hence bounded) on [0, T). Lipschitz with respect to x and there a constant C such that, for 6, [0, 7] with < 8, we have
(x,(x, t) - cØ(y, t)
< L y|²
(44)
(x,(x, t) v (9))
< L2 xx(2)
(45)
(x( () Eo Ex s)12
Ta(1 xx|3]e s| s|?
(46)
2. h maxisisM h4l ~ O(1/M), where h4 = Aaa- A
Let CR , R) denote the family of / times continuously differentiable real-valued functions on Rd whose partial derivatives of order have polynomial growth and let C². (I x R, RR) be the space of functions g( such that, for all (r, E x R.4, x), E (T, and g(t, E C/ (IRd IR).
Assumption C.2. addition to Assumption C.1 assume that all the components of Eg belong to c/2 (RE x [0, R).
Before going into the proofs, we give some context that lead us to necessitate such assumptions.
C.1 Preliminaries
For an interval I= [fo, TJ. letx (x(t)), y the solution of the following SDE
dx(l) f(x((, (((()))))))=((dd(())))) (())
(47)
where s(t) (t) Idaa is considered here a diagonal matrix with identical diagonal entries ğ(t). Suppose that f. are continuous, and satisfy a linear growth and Lipsehitz condition so that the conditions of the Existence and Uniqueness Theorem are fulfilled for the SDE (47).
Let In = .lo..... "oo...,,,,,, ( be a time discretization of with step sizes hn, = ln+1 - la for n O,..., M - 1 and let h maxogneM hn A time discrete approximation scheme & = (Rn),, will be defined as a sequence X, x(lo) and
k
4 {Sx, In: In);
0. M 1,
where To is independent of &o. with I, w(m-+1 (n) Wiener increments drawn from the nor- mal distributions with zero mean and variance h1 and which are independent of Xg...., X, and fo....,,--..
scheme & converges in the strong (resp. weak) sense, with global order >> O, to the solution x of the SDE (47) if there is a constant C 0, independent of h and 8 0, such that, for each h Ej0, 6), we have
(B][x(7) RM A 12) < ChP, (resp.. (resp./E((e
Che, VG E C (2(++1) 1) (R. R)).
Notice that if condition, we have [[((((())) - -[]( (hP). Nevertheless, this not enough to infer the optimal weak order of convergence of such method.
8 O(P)) then for every function f satisfying Lipschitz
27

---

Strong convergence is concerned with the precision of the path, while the weak convergence is with the precision of the moments. As, for DPMs, the center of attention the evolution of the probability densities rather than that of the noising process of single data samples, weak convergence is enough to guarantee the wellicondii of our numerical schemes. Moreover, when the diffusion coefficient vanishes, then both strong and weak convergence (with the choice G id) reduce to the usual deterministic convergence criterion for ODEs.
Let us now state some useful results that will be used later on.
Assumption C.3. All moments of the initial value No exist, f is continuous, satisfy linear growth and globally Lipschitz condition.
In particular, since I is closed finite interval in R. f(., and are bounded by some constant.
Theorem C.4 25
addition
suppose that
E ((x(t1) - &1]l
& Ch
IE (ta) & 11/21 & Ch
with pa > 1/2 and pi p2 1/2. Then & of strong global order p p2 1/2.
Assumption C.5. All moments of the initial value &o exist, f is continuous, satisfy linear growth and Lipschitz condition with all their components belonging to Cpt c0++12(p+) (I x R R) and e (I.
Theorem C.6 ((25). In addition to
CI5 suppose that
1. for large enough N, the moments ([1 exist and are uniformingly bounded with respect to M and n O,...,M
2. for all G CRiP+ 1) (R4, R), 8, ((x),, for some K E CR IR), we have E[G(x(tn+1))] -BGGG ((x
Then & is weak global order p.
Lemma C.7. Suppose that Ro has moments of all orders and that, for 1,
E EE((En hns In) &n]] [ hn, In) Nn < (
< K(1 + 88, 1
where X,, has moments of all orders. Then Condition 1 in Theorem C.6 fulfilled.
C.2 Convergence of SEEDS-1
In this section we will prove that that SEEDS-1 described above is of global strong order 1.0.
C.2.1 Strong Itô- Taylor approximation
Then the truncated Itô- ttyTaylor expansion of the analytic solution x, of the VP NP NRSDE starting from x, is given, for NO, 1), by
X
x,
20r
V2at
dwax
a,
)
4
at
X,
(e" 1)60(x,.
-5,Ve22 le R
a.
01
25,(e" - -1),0(8,,5) o, ō.V622 1c O(h),
X,
0s
28

---

where the symbol (hh)) represent terms such that ||||| < [K(x)) for K E CP(RR, R) and small h > The SEEDS-1 scheme corresponds such truncated Itô-Taylor expansion containing only the time and Wiener integrals of multiplicity one. As such, since Gtg(t) 0 and assuming Lipschitz and linear growth conditions on Eg as in Assumption C.T X can be interpreted as an order 1.0 strong Itó- T6-Toyyorr approximation 17 Theorem 10.6.3) of the solution to (42
C.2.2 Continuous approximation of SEEDS-1
Let ã₂ := Bat + fmtt where 3a = .max
Amin 19.9, 3m Bmin 0.1. We have
o = V 11 ar = -
Vož +1
so, in particular, as T = we have Ütg = σ1 = (3max Bmin) + Smin = (Bmax + Smin) ~ 10.05 and 0it āg 0. We deduce 0t, a1 10-000 > and cxr+ do = 0. As such, Xto - -log(o0)) == Lo 0 and d - to0.
2 < 1, 0tsst ao 1. Next, O1 01 &
As such, we will set tm +M+1 + E the end time so that Aess = Lo is finite. This implies that, for AE Lo t00%, 0 < Lo < e < e and, for T Aeas dea et ede-d, & ed Now set, for X E Lo +o0l.
â 1+e-2x
V + T+t22
Then, as À increases, 4 increases starting from 0 < âxi and âd d
while at the same
time à decreases starting from &xx2 and ax A+0000
0. As such, we can rewrite the exact
solution
43 as
200x e Ex A)dA V20x A/m
de Ado
(48)
Notice that dx bounded for 1, by ax
âd < âx. 6x
and 0 < âx < 1 for all À E [oLo; +ool.
Recall that the SEEDS-1 is defined recursively as
at
Yt.-1 (e - 1)ca (y)o 44-1) - oz; 2 le;
Yto XT, Vt,
and, for simplicity, we will denote yx., for the iterates
(8
Define continuous approximation of SEEDS-1 follows. For h t - 8, write [s/h]h where [z]] denotes the largest integer lesser or equal to x and [|A] is the indicator function associated set A. We define the step function:
{(A)
Dig 1441 (Ydey
R30
and the continuous approximation
d 20
(2), v20r fr
y(1)= = Ct y(o))
S
--
a'oo
Proposition C.8. There are two constants C1, C2 independent of h such that, for all E [0, we have
E
sup
lx(t))²
Ci
E (((1) - 8(()
C
29

---

Proof. Recall the standard inequality (a + c)2 & 3(aa² - +b2 for R. Then
2
de S.me e E A)dx + +20
2
< 3
atto
ytu
+
4a
250
e
dww
Using the fact that or < have, by writing T Neas No and taking the expectation:
E
sup
[x(1)
Lo S
2
I.me e
27
Ar
Ox
E
11o [2]
+ +4E
+ +2a2B
e dws
Ot
A1e
2
Ot
(2)
+ 4F
1e 112
60 (2))) 211
+ 202
|e l
Olto
As0
3
Ctt
2
yto
2
+ 47
le 112
(00(s1a). 1017 aa + 20
(nmm Am
c c-21d
3K
E
yto
2
44
du (oR(A12Ai)) - 1)
200
3K
E
Vta
2
44 L2
(1+ [1((2))]]
dd+ (6 1)
2100
3K
E
yyto
2
+4TT22 + 2 -1 1812
/m E
[(((A)]]]
da
3K
E
ytu
2
+ 4TT.2 + 2t +c22-11) + -3KTTI2
aa
3K E
[2]
+ 4 L + 2 -- 12K
E
(r(r)) AAETEA
sup
dx,
where used the linearity of expectation, Hölder's inequality, Doob's martingale inequality, Itô isometry, the linear growth condition of ég. and we set K = max
2
o2. 17.1) As we
know that E [xx(t)]]] < xc, we apply Grönwall's inequality in the last line to obtain
(x) & C1, G = 3K E [x( + + 25 -1] 12ktls
20
E sup
LOS
Second, we have, for s = = u 1,41 and e [tx(4),
y() ((t) = at a₂
1
1) yz 20, / e o-
so that, using Hölder's inequality, we get
2 |yal
+ 4h (av
Ai le (xPP1e0(y),,,))))
(t(t) 2(0)2 < 3
+4h
axs
Now using Itô isometry we obtain:
2
+ 4hE f/ A 1 1eooxr 2.)
[y(t) - {(t))]
04
<
3
y
30

---

Now, using the bound E above, we get
max lyel" [
<
C1, the fact that Eo is bounded and the same arguments as
E
(v(1)-5(11P)
<
3
04 1
C1 + 4KhB
Ieoly,., :o) as
0g
( (1+CI)aN)
0 1
3
C1 + +4KhL2
ag
C1 + (1+4k22a(++ti))
<
3
0g
Finally, as we have
e
0s
0t V a
=
02 v ai
and 1
2 he -112 ~ ((2)). Now, by denoting |e 113
1, we obtain
a
Vi-a? M-20
Kgh², conclude that
[y(t) - 8(1)) & C2h?,
with C2 == 3K,03KKC1- + 4K C1). This finishes the proof.
C.2.3 Proof of Theorem 4.1
Let's now take a look at the approximation given yt, = -Xo- We have
A
yt X2
20
(8x,,)) --(o)))))))
(2)( dx.
Using the inequality S 1, the Lipschitz property of Eo, Assumption C.I and Hölder's inequality we deduce the bound:
ly
x
<
e - (xx, - - - (o 2) da
Amo
1 Ie
|e (0-xE0(3s,)) ))))(xxd)) 2 dd
84
A1o
Taking the expectation yields
-xxxx xsl]
E
sup ly. -
-c dd
SÂB
Now, for the first integral, by writing:
~'Ea( 2) - (5 2)
co(ox)) - - A) te - to - - (x),))))
te t0 A) - c - + - 1) -e -e - 2)
(0 c + (2() 20(x).))
+e (Eo(RA, 1) Eo(ys, 2) Eoly,,
\) - (o(Px:2))
31

---

we can state the following inequalities:
220 2) 2 'dx
E
(X2,
Aro
le x
-xpE [B0 2)1² dd 3E
de
1e-ip (0(2)))) E0(22. 20)² ai
c
Ado
Aea
A
le $12 (x 2) Eo(rx. A) +2o(yx,)) - 2)p }
+3E
Ada
<
le
e- 11p (os A)1² dA + +3KLB
(1+ [1+181131(1- 2))PdN
Ato
A11
de
le 11
(o 2) Co(yA A) 2)), - Eo(Fx, 2 da
+3E
Aag
+ E||sA 3 AP + 3KLSh" /// 12x1))d
c 1/12
< 3L2 A00
Ae
L1
E 81882- y2
+ y2 - - -3,1 d1.
A100
Thus, we obtain
1 (x,, A) - -6 2)p= as
E
A1o
3L2KT
- 1P((1 + KL C1)T
3Lth
+3KL
| - + +y -311] yp E
3L2 3InKT(e"
1 (1 C1) + +31gh2((1 GifT
| - y2l + + -8ai]ar da E
+6KL,
< - - +
+6KL,TChh2+ + +KL
E E| -y11] dp
3L2KTJe - 1p(1+ + ((1 + CIYi
t6KLTCCh + +KLL
E
sup
{x(r) - y(r)) da
Avo
- 1 Lah?l + 6KI,TC,12
+6KL, /
d, E
(x(r) x(r)) 2 dd
sup
LAnErEA
32

---

Putting everything together yields
E sup
ly. x,1²
LOG
TKT + Ci)(T - 1|2 + Lzh) + 6KI,fC2h7
+8T6KL
E
sup
[x(r) - -((r) aa
LAOEEEA
24K7
(1+ 112 + L3/h²] 48KL,
+48TKL
E
sup DOGTEA
[(() y(r)) dd.
Now, by denoting let 112 & Kzh?, we apply the continuous version of the Grönwall Lemma to obtain:
xal? < Coh²
E
sup Lo
ly:
where
Co [2 Ci) L3] + 48KT² L1 L 48t
Finally, using Lyapunov's inequality we obtain, for C' Vo
Ellyr xT|] [E] YT xT x 1/2 & Ch.
In other words, for C as stated above, have the following inequality
sup LISGEEM
xiP
Ch,
as -
Remark C.9. From the above, it is easy to induce that the following order for the one step error
(lx(t1) y, 12] O(h").
Now, since Gtg(t) = 0 in this case (additive noise), it is easy to see from the truncated Itô- to-Taylor expansion of x that [E[x((z)= yıll O(h3). As such, apply Theorem C.4 to conclude that SEEDS-1 has strong convergence of global order 1.0.
C.2.4 Discrete-time approximation
know that SEEDS-1, being of strong order 1.0, is immediately also of weak order 1. Nonetheless, let's give a discrete approach of this statement that we will use for the proofs of convergence for the remaining solvers stated in Corollary 4.4 Define the discrete time process:
By Theorem 4.1
Yto -- XT.Nt.
at
(e - 1)e(y 4-41) 220,,
(s)
Ott-11
We will prove that E xt is of order h as h - O. Note that (yt.)s has the same distribution as (x described in Algorithm since the stochastic integrals (Jyys e e=)((s)) are independent
1 du ve Ie with h; do. - e~ 10) and each Pi e (d) is distributed as /2 az,
We have:
Nhi frr e - c(yt,,,, t
Ot: (yt- X(,.)) ) 201 a:.-1
Yt. Xt.
For simplicity, what follows will denote a constant not dependent on the subdivision of [0. T] that may change from one line to the next by systematically denoting the maximum value of the
33

---

different constants appearing in the line before. Using the inequality as S 1, the Lipschitz property of e, we deduce the bound:
Are Lm e~>(((2)) - 44-
X1,
Okt:
Yt, 1 Xz, .100
->118xx
-r, 1
Acc1
Acc-
Notice that
Ar
E
1
ex - xx_i/dd +
E
e - - -o- 'dl
A44-1
De-1
cx1
Xx, -x,_ Chi
Vta-ib
and recall that Xu = (x) Using the fact that ε is increasing and Lemma C.11
we have:
Aes E 14-1
lf e" t-1/dd s Ch
<33 x1__, sc
Introduce U, = E(lyt. xt,ll. Since Ars (1)(ta(A) - - ch
an + Ch U U
U <
a11-1
Let a; on + Ch and b Ch By applying Lemma C.12 we have: UM S Ast
1
Ex= bb, with AM = Ilk 1 ax and A = AM/As 11 ag Note that Up = 0 since Yta XT and so:
M-1 UM Ch?
Σ
sup at)
1515M
Using our hypothesis, we can bound:
M-1
M-1
(exp(Ch) + Ch)* (exp(Ch) + Ch) (exp(Ch) Ch) -
Σ
sup ai)*
I5
k=
Am=
The quantity on the right of order C/h. Indeed, as goes to 0, exp(Ch) - Ch - is equivalent to 2Chh and (cxp(Ch) + Ch) 1/h converges to constant. This gives the bound UM v Ch, when h + 0 and by using Proposition C.8 and Theorem C.6 we conclude that SEEDS-1 is convergent of weak order 1.
C.2.5 Useful Lemmas
Lemma C.10. (Continuous Grönwall Lemma) Let [a, b] denote a compact interval of the real line with a b. Let 3, u be continuous real-valued functions defined 1. Assume B is non-negative, a is non-decreasing and u satisfies the integral inequality
u(t) s a(l) + S ((s) ds,
I
Then
ul() S all) exp (())))))) V.
Lemma C.11. Assume the following forward SDE is satisfied d.X, F(t, Xi)dt G(f)dW,, E
[0, TT. where T > O, F(t, x) is Lipschitz with respect to (t,x), G continuous and X, is an integrable random variable. Then, there exists C 0 such that for all < t 10, T with t
Bl|X, XaIl S CVt-
34

---

Proof. We take and apply the triangular inequality:
[X, -Xol - L (f,
+1 S. +
Setting u(l) = Bl|x, Xol] and taking the expectation, we deduce:
(t) & K 1a + F() + E G(u)dWa) pp + K S'' u(s)ds, 2
where K is positive constant. Note that BI[ G(u)dW.,)P) [So G2 by the Itô isometry property which is less than Cvt since is continuous. Thus, we have proved that:
u(t) S a(t) + K S. u(s)ds,
where a(t) = K a(t) (Kt) S a(t) exp(KT). Since a(t) s Cor E [0, 1], the lemma holds.
2 + t0 x,)l + CVE is non-decreasing. By Lemma C.10
u(t)
S
Lemma C.12. (Discrete Grönwall Lemma) Consider a real number sequence (un)n such that Un+1 & On+11lp + b 2 0
where (an) and (ba)a are two given sequences such that bn is positive. Then
Un S A, uo +
Σ Ak
k=1
where A, IT Og and Ag, An/Ax = 1Tp=k41 Oz-
C.3 Proof of Proposition 4.3
Let us prove this statement for SEEDS-2. On the one hand, have
u1 O'11 X, 25., -1= (x() - B!, B¹ = /20r prvn e -wwx
Now B¹ depends on the Brownian movement (wx., wx.) do, Sd. By the Markov property, this independent of (wx,) )..s,, Ex,- Since 8, a function of (wx, (u JA, we deduce that B² has to be independent of all (Ku)uzs- By Itō Isometry, we obtain
B¹ = o. V ch 12), x1 N.).L.)
On the other hand, the update x, is
Ba = 22ar fr e- dww
X
a2 x, 20 - F) B",
Hence, the Wiener process W fwix [A,, is predetermined on the interval [Ag do, Then, by using independent increments property of Wiener process W, we can deduce, for 0 < Xg < Xan < dç that
Wh, w>, and w>, wi.,, are independent.
Then, by the above Brownian independence property, the random variables 8 and B1 are
Indepeedeent non-overlapping time intervals
35

---

Dependent on overlapping intervals.
By the Chasles rule, we then decompose
Ba = V20n / e
ddux + 220 L e -->ww
the
and, as
[am dww
Aos
B'
V2as
we obtain
Ba
/ e
Ar
B = -200.
+
V20ss
Jan
Then have
An
(
de
B
v 22n
-200 ddw
c dux
1
22
Jsi V 2 2,) 1x= V2ar
c2((e-Ao)) 12
V2
V2
V ch 12 + 0 V 62h - 122
Jsi Veh 11 + ee -122
o; ( Ac Vel 1x + v 22h 1x?
(e Ve ch -2 + ee - 122
o, (Veth. zh + 2e - 1=?
which completes the proof.
The proof for SEEDS-3 is straightforward from the proof for SEEDS-2.
C.4 Proof of Corollary 4.4
C.4.1 Convergence of SEEDS-2:
Let 3s be the discrete stochastic process defined as follows:
du
du v2ar, ( "do(ss)
Yto XT,Nt, ae;
ar Vt: 200, د
" sijdu
Juu-
with 8; +- tx (Dem + hc) and
Olei yt.. 20., (e 1) c c(yt,,))t-x) V20s,
u; +
dd)(s)
at
Aep-1
Then {yt;); has the same distribution as f%t, in Algorithm 3 We can compute the difference:
Ct (yt: X1, .) +Tr r T2
Yt: Xt:
0t.-1
and:
phis - [E(u; s:) E(x,,, st)] du
I1 -20t J
phos
T2 -20t. xey-
c
[e([e 8i) - E(E,, u)] du
36

---

Similarly to the case k 1, 2|[ & Ch Note that Ei[T & ChA, with A, Ellu; - xs,
Introduce: U, Elyy,. x,, We will bound A, by a function of UZ-1. I. For this, recall:
Ass
M d)x(())
Aes
Yt,
20rs;
e 11 4-1-1)dd - V20s.
-
and write the difference:
pri E " u) EB (y (n-1))dd
Cto
U; Xs.
(yt-)) Xta-i) 1 + 200,
-1ts-1
By the triangular inequality:
(((2,) u) - c(Yu, du 1) C((xes(u) - X1, .1+ [x: Yo, .1 + (4x()- - -4-11)
and Elle(Rin u) E(Yr, 1s A m)I S C(V++ U2-1 + +1). Finally:
as Up-1 + Ch (vh+ U2- U--+++ + +h) SUL1+ Ch (v++Ui1)
A S 011-1
and B[[TT|]] < O + C(h + 1²) This gives the bound:
U. S Ch2 +
( + Ch2)U,-1-
0t
0t;-1
Now using Theorem C.6 the proof now be finished following in proof in Section C.2.4
C.4.2 Convergence of SEEDS-3:
Continuing with the same notations as before, we will prove by analogy that:
U, S Ch² + Ctty
+ Ch + Ch? + Uh
(49)
a6
so that, by Theorem C.6 we obtain the desired result.
Let yt. = X,,- We have:
Ct: vb.-1 2as, prv e (4
Aee
yt. at
201
dee e e->(1- de (c S21) (( + Noise,
h,T2 -11-1
201,; Pais - AjdA + Noise, where Noise is the same in both
and Ft; occ-1
Wt;
equations. From the proof for h it suffices to bound:
-- \e 52;)
A4-1
in the T,1 norm. By the Lipschitz property of E:
[[T]] s Ch(Blluu - T + El| - Fzzi 1 Elyy... xt,,,))) + Ch2
Ch(Ellu2i Tszill vh U--1) Chh.
Now, let us bound El I |l and for this, write
p e
Ao2i - (
Ctsas Vo
U2i a0,-1
20sx
Auzi é A-1(1 - )ewn))
E(U2i- 1, 82i-1) elyyt-11 t(-)) dd + Noise,
-20s 1e-11
r1b4
37

---

and
Aozi
Clssi
-1,-1
22
e ==(((X):))) + Noise,
0t.11
dec11
where Noise is the same both equations. So
Aozi
U2: D02c
(yt1-1 - Iti-.) 20sn
e -C(B,,,
0t
Aozi
-20 Jec-l
e c-1(1 )
where c(w2i-1.82i ( (1-1)
From the convergence proof of SEEDS-1, 1, and the Lipschitz
r1h;
property of 6, we obtain:
Ellu21 x |l SUS-11 ChU,-1 Ch² ChE[lugi-1 3/.-11).
Again, by the triangular inequality:
El st,-1ll S El[ Tsz- CVh+ U:-1.
and since
Augi-1 e c->(E(2,, 4(
0ss:: (3/t.1
U2i-1 Ten-
It(-1) 20
Ott;
A4-11
we have, as before,
E
U2:11
Nzn U--1 ChU,-1+ Ch
Combining the previous inequalities leads
(49
This finishes the proof of Corollary 4.4
D Implementation Details
The SEEDS solvers used in our experiments are exactly the variants we described in Algorithms 2 in the main part of the paper. In particular, SEEDS-2 and SEEDS-3 solvers are completely determined by one-parameter families of deterministic exponential integrators from 14 of order and respectively and prescribed by the following Butcher tableaux:
0
0
0
02
02.11.2 (1
C2
3
2
C2P112
2
4
42.3
0
9c,
22.3
0
0
202
P1
20g
P1
P1
52
0
7 P2
In experiments we fix the parameter co = 0.5 for SEEDS-2 and C2 for SEEDS-3. We point out that the Butcher tableau here associated to SEEDS-3 is the result of a weakening of the stiff order conditions so might suffer from order reduction in the deterministic case. We also point out that full theory of stiff order conditions for stochastic exponential Kunge--Kuta methods for semi-linear DEs with a Lime-varyine linear coefficient have not been yet developed to the authors knowledge. As such, this is only an analogy whose purpose is to clarify how our solvers relate to well-known solvers from the literature, such Butcher tableaux do not rigorously reflect their convergence order.
D.I Stabilization of the exponential terms
In the proposed algorithms, one subtle detail re-arrange all equations in order for them make only of the expm1(h) function which computes e with great numerical stability, specially
38

---

for small values of h. We have for values = 1/3, = 2/3, the following identity
Ve chi 23 + Vehi lt; = Vehi -1
(eelizz vi)
Veh -1
e + Us
v expmi(h:)
@xpm1
+ 1
28 + 2s
2
h
expmi(h:)
expm1
-
+
1
24 + expm1(h4)r)
2
Now by using
2(22-11)4s = (33-)) = ci = c²2hi
we now compute
2e erahi + Verah. - 12,
Verch: -
( Verahiz 2i
Vershe
2
= V (x2p)(r2 (erpppmmmrr11/ (/44))) + 1)21 + 2,))
V 2e 2b, - 22
erth, - c2r2h: 2
e V erahi 1z
v
+ 1)2
Finally, we obtain
ee 22rshi Vearhh. exahi 27 + Verzh, - -123 + 1)2! + (expm1 hi) + +))2 2))
= V
D.2 Noise schedules parameterizations
The inverses of A(t) are given in the (VP) linear and cosine schedules respectively by
2 llog(e + 1)
(x(A)
V min + 2(3max Pmin log(e
2A + 1) + 8min
2(1 +) arecos
(exp( log(e -2 + 1) + log cos
1
7S
(A)
2(1
8)
In EDM Noise Prediction case, the inverses of A(t), as given in 37 with respect to the NPFO and NRSDE are respectively given by
(A) ad tan(e->) and tx(A)
Od
oz--2x
D.3 EDM discretization
We follow Karras al. [16] to implement the EDM discretization timesteps (M -o;)) such that, for p > 0,
Fmax )
Oi<M
Tmax +
N -
min
and ON = 0.
39

---

From the definition, we note that do = mmx and 6M-1 miin, where min and max denote the minimum and maximum noise magnitude, respectively. We also keep default value p as in [16]. However, we figured out that when using EDM discretization with linear schedule, the noise schedule improvement in iDDPM pre-trained models (26) would result in two consecutive time steps of the 1 and large steps (M 2 61). Thus, for
same value, i.e. E 1++1 for some index = 1....., M
SEEDS-3 and DPM-Solver-3 [2 the usage of the function (22(h) e¹ h (see Appendix
h2
for more details) will cause zero division error where h Atz= X1, 0. Therefore, in our
implementaiion, we ignore the noise schedule improvement of iDDPM (26 models when using
solvers of order three.
D.4 Final sampling step
The sampling phase in DPMs using SEEDS follows the RSDE, which requires gradual computing through discretization time steps M and the latter goes from to T far O. our implementation, to avoid the logarithm of zero error at the last step, i.c. log tы. = log(0), we stop the sampling phase at step (M 1). Hence, the NFE used in run will be given as
NFE k (M 1),
where A represents the order of the solver. We also do not the "denoising" trick, i.e., ignoring the random noise at the last step and leave it to further research.
E Reminders on Stochastic Exponential Integrators
Let us consider a SDE the form
dx (L)
+(())(((()))))) g()dd)(
(50)
where : [0, + R and g |0, T) dxd In other words we concentrate to high-dimensional semi-linear autonomous SDEs with additive noise. The objective of this section is construct explicit stochastic exponential derivative-free methods for the above equation following the Runge- Kutta (RK) approach. These methods ideally should fulfill the following properties:
1. If f = 0, then 50 can be solved exactly;
2. If g 0, then SEEDS method for (50) identifies with an exponential RK (ERK) method;
3. If a = 0, then a SEEDS method for 50) identifies with a stochastic RK (SRK) method and if moreover g then identifies with a classical RK ODE method.
Before tackling the aimed SEFDS SEEDS problem let us rapidly recall elementary constructions of RK, ERK, weak and strong SRK methods. We will not deal with adme-ade methods here.
E.1 Derivative-free exponential ODE schemes
E.1.1 Runge-Kutta approach
Derivative-free schemes are obtained by comparing the Itô-Taylor expansion the above paragraph with expressions of x(t) in terms of its intermediate evaluations between and = + h and the Taylor expansions of such evaluations. As a simple example, in the ODE regime
dx(tt = f (x((t, t)dt.
(51)
As such, analytic solutions to the above equations are of the form
x(() = x(s) + S ((x(T),t)dr
40

---

Now, Taylor expansion gives, up to order 2:
h2 x(t) x(s) h x" (s) +s)
+hf 8) ((s)))f(((ss),) 8)) 0(h))
h²2
= x(s) + hf(((s)) 8) + 2
+
A straightforward recursion yields a Taylor expansion
x(t) = x(s)
s(((),))
Li (x(()
where we denote the generalized infinitesimal operator of the solution x of (51 by
Le(+) = ()) + f(x(t), t) (x)
Derivative-free methods seek to get rid of the derivatives L, by computing Taylor expansions at well-chosen points. In the explicit one-step case, this amounts of defining
x(1) x(5) + h 8,)))
where the & function does not contain derivatives of f. The general high order case is given by well-tuned coefficients in the following scheme
2(1)
2(5) + ail(xi, cali)
= 8(5) + h Σ
0i a [ cyh).
==1
By denoting [on (ol,.,.,],cc == (a... c and A = (di.j), these can be represented by a Butcher tableau of the form
A
In the following sections will present extended version of this tableau for representing more involved numerical schemes. As such, the Euler, midpoint, Heun and general second order explicit methods are respectively:
0
0
0
1/2
1/2
0
1
0
C2
C2
0 1
1/2
1/2
2ng
2ng
E.1.2 Exponential Runge-Kutta approach
We now concentrate non-autonomous semi-linear ODE of the form
dx(t) [f(t)x(t) ((x(t)))))cc
(52)
where f(t)= f(t)- Id, and g(((())) identical diagonal coefficients. In particular, have (t), ( 0. property will need to facilitate exponential matrix multiplications. As f(s) is a dddimensional diagonal matrix, the fundamental matrix for 152) usually given by the Seuno-Baker series, simplifies in this case the form
t) Idl are diagonal time-dependent matrices with
( rovarr"
@(t; = el' f(r)dr
Σ n! n=D
41

---

which satisfies (t; = f (t) s) and ((s;s) = 1. Then the exact solution for (22) is given, via the variation of constants formula, by
x(s) + / Φ (T)s)))(()
D(t:s)x(s)+ /' ((t:)))(((T))))))
x(t) = ((t: s)
In light of this integral form, one can formalize general class of exponential n-stage RK methods
x; (h + at fa)g(xy, cyh)
x(1) = St)(( + Σ f cch),
where "oo Ti, Oij. by are functions of the step-size h, fand fz(r) == Su f(s + r)dr. There are two possible approaches create exponential integrators, namely the exponential time-differencing (ETD) approach that uses the variation of constants formula and makes use of the y functions, and the Lawson approach, also know as integrating factor (IF), which makes change of variables on the above SDE thus avoiding the use of the &p functions but computing exponential factors step-wise.
The Lawson and the ETD approaches for Exponential Euler schemes
There are two choices one can make when computing first order approximations of
x(t) = (tt; s)x(s) + J. o ((((())(
First, by interpolating g(x(T),T) as (x()), 8) we obtain
x(t) $(t: s)x(s) + g(x((), S ((::)))s
Now two choices remain. Either J (1; 7 dd is computed exactly or we again interpolate Ø(4; r) as (t; s). Taking for simplicity f = to be constant and by denoting h = 1 8, the first case yields the ETD Fuler method
x(t)
+ (on
= ( + (pp
((s) - h (4b))(((()) {())
and the second yields the IF Euler (also called Lawson-Euler) method
x(t) = (()) +
=
Now, consider solution
x(t) (t: ()((s) + | (b: T)g(()s + T), r)dr.
Exponential methods then aim to approximate the term g(x(s + T), + r) by its interpolation polynomial in certain non-confluent quadrature nodes cI...., Cn.
The ETD approach
In this case, the variation of constants formula yields
stch e retch a f(x(T),T)dl
e a(r)dr +
x(s + ch)
sch
elch a(str)dt + eft alstr)dr f(x(= + r), + r)d7.
42

---

Now, as before, the Taylor expansion of yields
[(x(8 +), = Σ j=1
(x((),,
(j --1
%
(r 11)1))-11 (x TI). n)dd.
(g 10-11 1)!
Recall that the yp functions are given in an integral form as follows
S'' (1--))+ 8*
Ph+1 (t)
k! dó,
which satisfy (0) Now denote
at(t,a) = ti L of a(r)dr 75-1
1
(j- 1)! dr,
1>1.
a)) /// eff a(r)dr //² ch a())ard
The exact solution ch reads
x(s + ch) = eloxh aletr)drx(() +
Σ (x)))))
a)
j=1
/ ch cf alstr)dr
(%" (T -
(x(s + s n)ds) dn
+
(q- 1)!
Using the left endpoint rule yields
x(t) e5t ((xt))dxx(()) f(x(s), 1-' ef a elt (letr)dxx(s) + (h, ((x(s) O(h3).
Second order examples:
The condition b2(2)c2 p2(() implies by (22())/22 and we obtain
x = chA x(s) (ch))()))(((((s)
x(1) (^*x(s) + -
" ch). (53)
second method is obtained by weakening the above condition by(0)c2 = (2 1/2 giving
& = chA x(s) + chip (ch))))))
x(t) eAx((s) + hpp (hA) (1- 2t2 f(x(s), +
2c P: (hA))((( s +c).
(54)
In some cases this method can suffer from order reduction and not reach order convergence. Moreover, setting 1/2 gives the exponential midpoint method:
x(s) chyp (xh)))((((ss,)) cohA((s)
x(1)
<^^x(s) hep (IA)) s + ch).
(55)
The above order 1 and 2 exponential methods are represented by the following exponential Butcher tableaux
0
0
0
0
0
02
0241,
C2
C241
C24P1,2
f1
P1
P2
82
20g
P1
209
P1
43

---

We check that when formally setting A = 0 then the first two methods are identical and give the generic 2nd order RK method, the choice 1 gives the Heun method and the choice 1/2 gives the midpoint method.
Fourth order methods
can be shown that ERK methods need at least 5 stages to achieve order 4. By setting formally A = 0 these methods do not have non-exponential counterpart to our knowledge.
5-stage sequential We have a fourth-order ERK scheme given by
2 2 - 422,3
4223
P9,1 2422,1 1 2 24,, 205.2 05,4
P22,1
(P2,1
05,2
05,2
1
442,5
05,2
PP1 3:22 + 44ps
0
-Pp + 443
4:22 8ps
with
05,2 = C3422.5
(3,,4 + - C543,5-
This can also be represented by Rosenbrock-lik, Butcher tableau
2 11,
2P,
22,
P1
R22
P2
21,5
05,2
05.2
4
92.5
05,2
P1
0
0
4
P22
4,p2
84p3
with
05,2 - + c3ip 4p3.
This traduces into (recall that D; = (xx,,
f(x(s), 8))
x(1) = x()) + (()r(()) + h(4pp((hA) 422 D2))
-(44
= x(s) + (2/x)((c)).)))((((()..)))
X3 x(s) + hespp(()),,))((((()))))))))) + (cgghAA))
= x(s) + h4p (1A)))))((((() hp2((A)) + Dz)
X5 = x(s) + hes (x(s) 8) has.2 (h2) + D3) + ((G)p)(cshA) -a,,(hh)))
Inspired by this, we deduce the following fourth order Algorithm 7 specialized for DPMs the VP noise prediction regime.
E.2 Derivative-free exponential SDE schemes
Let x, be the path at the continuous limit - 0, and (x:) be the discretized numerical path. computed by numerical scheme with M 1/h steps of length h > O. Then, S has
1. strong order of convergence y there K such that
E| R., I1 < Kh
(56)
2. weak order of convergence 3 if there is K > 0 and a function class I. such that [E(((xxx)] ((
)] Kh.
Vo(-) E.
(57)
Strong convergence is concerned with the precision of the path, while the weak convergence is with the precision of the moments. As, for diffusion models, the center of attention the evolution of the probability densities rather than that of the noising process of single data samples, weak convergence is enough to guarantee the well-conditioning of our numerical schemes. Moreover, when the diffusion coefficient vanishes, then both strong and weak convergence (with the choice e id) reduce to the usual deterministic convergence criterion for ODEs.
44

---

Algorithm 7 DPM-Solver-
I: def DPM- 11 0.5):
2: hi f )r, 1
3:
la (dt;-1 + +1i)),
+ la (de=1 + rhi)
4:
$4 tx (1i-1 + +hs)
85 t ta (Dem-s + rhi)
5:
k f En Ext 1-1i-1)
6:
k2 as
aez Xt,-1
02 (c 1)k1
7:
k3 F
deg X1.-1
0sg (c - -1)k1 - Osa (4ehh -2) [co (k2, 82) k1]
h
8:
k4 t
ang X1,-1
0ss (e 1)k1
Uss cn -1 (oo 83) + co c0(k2, 82) 2
9:
A = Jo (e
1)k1 00 h - [k + eo (k2, 82) + c/ (k 8s)]
10:
B = Jss
c 1
[k] + 4co (22 82) + 4co kz 83) co s4)l
h
11:
C das
11-1+1(e" -1)-3h -1) 1) l c 22) c0 ckk3 $3) + (Ks s4)]
h
kg
aug [::-1
A B-C
an
D 01 (eh
1)kg - 0:
( -1 (AEo 85) - co 84) 3k:] [k1 + co s1) - 2eo (ks, ss)]
E = di - -2)
14:
hi
15:
X ai.-1
St1-1 D E
16:
Return X1;
E.2.1 Strong and Weak Stochastic Runge-Kutta approach
In all what follows is considered d-dimensional Wiener process (with identity diffusion matrix).
Consider the following SDE
(x(() f(x(t), + g(t)dw(t),
where g(t) g(t) Ida is considered here as a diagonal matrix with identical diagonal entries ğ()).
Given an initial value independent of w, the integral form of x() is given by
x(1) =x(s) S. [(x(T), J. g(())d(())
The idea underlying stochastic numerical schemes is very similar to the one used in the deterministic approach, that is to take expansions of the terms inside the integrals based at the integral's 's initial value and replace the obtained derivatives that appear by interpolated approximations. here is that as we have to consider Itô-Taylon expansions, the infinitesimal operators are different but most importantly most of the stochastic iterated integrals will need be approximated in an appropriate sense, whenever that is possible. We will develop this expansion up to triple integrals.
key difference
E.2.2 Truncated Itô- To-Taylor expansions
Applying Itō formula to h = f or 9 yields
h(x(t),t) n(x(s),s)+ '" g(T) ()((())
+ +f. (B.M(x(T)) + f(x(T). (xk(((T), (2(T)
(2.h(x(7).)) dr.
This allows us to define two differential operators T., G as
L 0 + f()(t), 0x (2()).. 02
(58)
G. g(t) 0x.
(59)
45

---

In particular Lig(t) = Oig(t) and Gig(t) = O. With this notation, we have
h(x(t),t) 1(x((s),)) +) + [ L)h(x(t))))dr+ + L: G()(x(t))
so our solution reads
x(t) x(5)+ L: ((((T)) + I. s(m))w(())
- x(6) S. (11x1000)) 1." [Ts((((2), 12)d + ["" GxI(((12),, ra)du((z)) dr
+ S. (00)) S." Lig(radrr2+ + J Gig(Ta)d((1a) du((T)
- x()) - [(((s),, 8)h + ((1)((()) - w(s)) + Ri.
Now G(g(T2) = 0 and
g(s) + [: Lig(Ta)drz g(s) + L." Org(1z)dmz = ((s) + - g(F))
So have
J (ol6) J. Lig(ra)dra+ J Geg(m))dw(ra) dw(())
and solution reads
x(t) x(s)
w(s) Rı.
where
Rí S. ( + 1(" [-" G.I(x(T2); (2)(())))))dduu) (T)
Now have
Le((x(t).t)
[)) ()x(()))))) r)d((T)
+ +f. (OLLI(((()))))) f(x(T), ( gx(((T),
02 (2LL)((((T),)) dr
2
L.((((s).s)+ + I 12((x(r),))dr+ I' GcIaf(((T)))))(()))) and G.I(x(1),1)
Gif(x(s),s) + I. ((x(1)))))) (acc)))))))aaGa((((()))))))))
[ (D.G.((((T))))) ((x))) -.G.)))))))GGG.(())))) ( (2 dr = Grf(x(s), + S L.G(((((T)))) S G?
Now, if we denote (T) dr, dw (T) dw(()) and
(() = S dw)(m)
Tass = 1/' S. dw (w)(d))
i e (0.1),
46

---

applying the same procedure to R1 leads to
R1
[
(x)(( 12)drzda + [ [." m))dx)))
[ [e5(x((),)++ + [." 1 { G.Laf(x(Ts)) rs)ss()s)) dr
+ [ (c,((((),))+ 1 (GG)(((s)))))) (a)d f." GF((((ss) rs) d
L)(((()))) [.=] S. dradni + +G,((x(())) S dw(r2)d7
+ [ (x(((()))) (a) f. (-) [-" GeInf(x(rs), ra)dw(r3) (drddr
[ [f ( midddddr)dr
+ J. J. ())( (f)ds)
(x(x()), C, L2 (G) + (0.10 + (3 + R2,
and with R2 consisting quadruple integrals. As such, our solution now reads
x()
x(5) ((x(s), + w(s))
(x)x + (z)
+ (x) + G.TeJ(x(s)) [
+ (xG),(((ss),)) 8) (o.1.0)) + G (x) )) R2.
Now, in the SDE regime, one cannot get rid of the iterated Itô integral and so stochastic RK methods cannot derived as simple extensions of their deterministic counterparts. In order to continue we now take into account the fact that the diffusion SDE has additive and diagonal noise. In this case, both the and the Stratonovich SDE coincide.
Iterated integrals
Now, for simplicity, set t = 0. We then have (o) = h I0.0) = So" Jo drædr = Now notice that
A2 3. (0.000))
ha 6
I(D) = in ~ (O),h)
[o.1) En (-¹ 1-" dw(r2)dr
lim h
ef g~~ (~ h
20000 72
İ=U j=l
Additionally, wn and én satisfy B[iig] h.
El(wan 2] E (%%0 Tdw((T)
1
h
E wn 1'' h
Ahi
E(vin2n]
rdw(())
= RR
7dT
1
E[ER]
B((wah 2n)²
112 2hrur 2n]
= 10
E.2.3 Integral approximations
Weak Approximations
47

---

When crafting weak stochastic approximations to SDEs one may replace multiple Itō integrals by other random variables satisfying the corresponding moment conditions. We will denote ia the approximation of To for a multi-index following [17 Corollary 5.12.1J. Of course, the deterministic integrals (qo....u) need not to be approximated.
First order approximations
The random variable (a) must satisfy for some constant K:
[B(al + (B(())))))+++ hi] & Kh²
Two possible choices for (1)) are either (a) h)) or (()) is two-pointed distributed discrete random variable with
P (()) = =Vb] 2
1
Second order approximations
The random variable ((1) must satisfy for some constant K:
[1) [B(((()) + [B(()))) [B(((())2 a] + (f)) -3h Kh
Two possible choices for (1) are either {(1) ~ NN h) or i()> is a three-pointed distributed random variable with
(a) a = +V33 = 6'
pl = =0]
The rest follows from the above calculations:
((0.1) 2 hla)
i= 0,
Third order approximations
One can choose (()) NN(,, h), (O.1) ~ N (0, )) satisfying Bl = 212.
Then, one can deduce the following:
f(1.0.0) = [ = (0.0.1) = 12
6
To.1.1 = [(1,0.1) = (1.100)) = @ h (PP) - -h)-
Thus, we can write the solution weak approximation as
122 )) R2
x(1) x(s)
+ =x(s)
+(x))[(())),)))))))))))))))))<<) 5)h + L(((((s),
13 + +GLLf((((s) i
+G.f(x(s), 3) (0.1) + (x((((s),
6
+L,,,f(x 3) h2 i + ()) 3) h).
(60)
An example of such pair (o.)) = (wn, 2n) can be easily obtained as follows
Wh
Vh
0
u1. U2 iid
NO.0 1).
hvT
hvh
U2
2
2/3
Indeed, for such pair we have
Wh
h A
2
2/
3
In light of the above expression, the truncated Taylor expansions we refer in the main part of the paper consists on the consideration only the coefficients L,C The only noise noise contribution we will consider corresponds to (
48

---

F Experiment Details
We evaluate the Fréchet inception distance (FID) after generating 50K samples with each solver, and compare with the statisties of real-data. In our experiments we make use of the code from [[6] for continuously trained models as well as their reference FID stats and that of [22] for discretely trained models.
All the experiments of SEEDS for continuous-time models are parameterized within the EDM framework with the discretization of type EDM, linear schedule, and scaling none described on [16] in noise prediction mode unless explicitly stated. We use the SEEDS-3 method that has 3 NFEs per step and fixed step-size and report FID scores at NFEs divisible by 3.
We leverage the explicit Langevin-like "churn" trick using in [16] to add or remove noise in the sampling phase. Specifically, [16] uses hyper-parameterd Schurn Smin: Smax and Suise in which Schum controls the overall amount of stochasticity added before giving the input to the SEEDS-3 method when the noise level (or time step in EDM configuration) is contained in the noise interval [Smin: Smax]- It means that the EDM proposed sampler is stochastic under some conditions of those nyper-parameters and deterministic otherwise, while our method is completely stochastic. In experiments, we set Sehum 0 except for ImageNet-64 EDM optimized model. We noticed that using the additional stochasticity indeed helps improve the image quality in Fig. (c). Moreover, setting Spoise slightly above I might correct the errors in earlier steps more effectively as indicated in [6]
F1 Pre-trained model specifications
For producing the CIFAR-10 time-continuous results in Table we use the VP DDPM++ continuous architecture. These models are publicly available in conditiona and unconditiona versions and were directly derived from [34] under the Apache 2.0 license. On the unconditional mode (Figure (a-b)), we first generate the FID curves of 3 types of DPM-Solver (with orders 1, 2 and 3) using the updates from their official implementatior in noise prediction mode. Taking profit of the tuning advancements proposed by [16), we used a linear noise schedule with 3a = 19.1 and Bini 0.1 that slightly differs from the original parameters from 34 but were proven beneficial. We set the end time of sampling to le-4 as recommended by (22 Appendix D.2]. The values of all benchmark
models for Figure (b) were taken directly from tables provided by (22].
In our FFHQ-64 experiments, we employ the unconditional VP pretrained model provided by 16
For the CelebA-64 experiments, we use the pre-trained VP unconditional model whose checkpoint is provided by (33. We use the Type-1 discretization proposed in [22] to ensure compatibility of our method with the prescribed trained steps of such model.
For ImageNet-64, we both use the baseline and the optimized pre- pre-trained models given in [6]. We note that the baseline is trained on the iDDPM class of model [26), which actually uses different preconditioning and thus the change of variables compared the optimized model. The Figure I (c) was obtained using the EDM precondiiiiii checkpoin [ The added noise settings of SEEDS-3 solver were not subject to grid-search optimization procedure. The chosen hyper-parameters were Sehurn = 11, Snoise 1.003, S.in 0.05, and Smax 15 but we are confident that this configuration can be optimized further improve our results.
http:://gtaabbboo//////gg//////p
sharing
pkl
49

---

F.2 Noise vs. Data Prediction approaches
In Appendix of [23]. the authors compare DPM-Solver2 and (2M- which amounts on comparing in framework the difference between the obtained exponential integrators for the PFO the noise and data prediction regimes to detect exactly a coefficient on the non-linear term that is absent in the noise prediction regime. The term they find corresponds exactly to the difference between applying the variation of constants formula before (instead of after) replacing the score function with the desired neural network. In Tab. we report both data and noise prediction SEEDS-3. At low NFEs the DP approach gives better results but stabilizes high NFEs at a FID score that is worse than the one the NP approach reaches.
Table 3: Comparison between noise prediction Fo,, and data prediction Dø. modes of SEEDS-3 on CIFAR-10 (VP uncond. cont.).
METHOD NFE
30
60
150
165
180
SEEDS-3 PREDICTION
60.75
22.42
12.47
2.95
2.51
2.54
2.55
SEEDS-3 NOISE PREDICTION
471.29
288.20
33.92
3.76
2.40
2.39
2.47
E3 Low High stage Solvers
Similar to DPM-Solver [22]. the FII) scores in Tab. 4 and show that at low NFEs, higher stage methods performs poorly while at higher NFEs, DPM-Solver-3 and SEEDS-3 are better than their counterparts with and 2 stages.
Table 4: FID comparison between SEEDS (Ours) and DPM-Solver for low NFEs CIFAR-I0 VP uncond. discrete. We recomputed the DPM-Solver score using the "non-deep" model while [22 reports results for the "deep" architecture. The symbol used when using NFE more and when using NFE less because the given NFE cannot be divided by 2 or 3. This corresponds to Figure (a).
METHOD \ NFE
10
12
15
20
30
40
50
100
DPM-SOLVER-1
22.90
17.73
13.36
9.78
6.87
5.77
5.17
4.22
DPM-SOLVER
12.22
6.52
*4.55
3.75
3.68
3.64
3.60
DPM-SOLVER-3
166.92
9.72
5.32
*3.83
3.66
*3.61
13.58
SEEDS-1
303.48
239.79
279.84
192,68
84.78
45.26
28.18
8.24
SEEDS-2
481.09
473.48
*430.98
305.88
223.01
51.41
11.10
3.19
SEEDS-3
1483.04
482.19
479.63
*462.61
280.48
1247.44
*62.62
13.53
Table 5: FID comparison between SEEDS (Ours) and DPM-Solver for high NFEs on CIFAR-10 VP uncond. discrete. We recomputed the DPM-Solver score using the "non-deep" model while 22 reports results for the "deep" architecture. The symbol is used when using 1 NFE more because the given NFE cannot be divided by 3. This corresponds to Figure (a).
METHOD \ NFE
150
300
510
DPM- SOLVER-3
3.59
*3.58
3,58
SEEDS-1
4.07
3.40
SEEDS-3
3.12
*3.08
3.14
3.24
50

---

Table 6: FID comparison between SEEDS-3 (Ours) and other solvers CIFAR-10 VP unconditional discrete. The symbol is used when using I NEE more and when using NFE less because the given NFE cannot be divided by 3. This corresponds to Figure [ (b).
METHOD NFE
10
12
15
20
50
200
1000
EULER- MARUYAMA
278.67
246.29
197.63
137.34
32.63
4.03
3.16
ANALYTIC DDPM
35.03
27.69
20.82
15.35
7.34
4.11
3.84
ANALYTIC DDIM
14.74
11.68
9.16
7.20
4.28
3.60
3.86
DDIM
13.58
11.02
8.92
6.94
4.73
4.07
3.95
DPM-SOLVER-3
16.92
9.72
5.32
*3.83
"3.61
*3.58
3EEDS-3
1483.04
482.19
479.63
*462.61
*62.62
*3.08
E,4 Deterministic vs. Stochastic Solvers
Deterministic solvers as DPM-Solver [22 are fast and well-adapted to applications in which speed is the most concern. As shown in [22] Table 4 and 5 DPM-Solver converges to local minimum at early steps and cannot be improved in large NFEs. Moreover, preconditioned deterministic solver in EDM gives optimal quality on unconditional CIFAR-10 [6, Fig. 5 (b)]. However, for more complicated datasets as ImageNet64, the stochasticity indeed helps improve the samples quality [16] Fig. [c)]. We can consider the random noise corrector that approaches better local or global minimum. At Sehum O, our SEEDS-3 3 with stochasticity gives lower FID score than EDM deterministic Heun (see Fig- 1 (c)). SEEDS-3 also reaches the best quality prior the number steps needed in Fuler Maruyama and other solvers as in Table Table 7able completes the analysis in Table forr CIFAR-10 showing FID scores for varying NFEs and multiple versions of the EDM solver, depending on the chosen optimization
Tables to I0 complete the findings in Table I for the remaining used pretrained models. In particular, report the exact FID values of SEEDS in the low NFE regime.
Table FID comparison of different solvers on CIFAR- 10--00oodd-woe (o--ucood-vpp-ootnnuee vp-continuous. The symbol is used when using NFE more and when using NFE less because the given NFE cannot be divided by 2 or This corresponds to Figure (b). All values for EDM are retrieved from the Latex source file SdePlotNfe.tex in the arXiv version of [16].
METHOD NPE
30
48
63
126
165
180
511
GOTTAGOFAST
82.42
2.75
2.44
EDM (Semaax 0)
*3.10
12.99
2.94
$2.98
2.93
EDM (STMIN,MAX + Sunnn 1)
*3.44
12.89
2.77
2.72
2.69
EDM (SHOISE
*3.99
13.13
2.90
2.60
2.55
EDM (STMIN.TMAX = [0, x0)
+3.43
12.87
2.71
2.52
2.54
EDM (OPTIMIZED)
*3.77
13.08
2.84
2.47
2.27
SOM-SOLVER (NON-DEEP)
2.95
2.90
2,89
SEEDS (Senoux 0)
*288.20
190.25
33.91
2.45
2.39
2.47
F.5 Run-time Comparison
As sanity check, provide run-time comparison table for experiments on different datasets (see Tab. [T]. One can see that the run-time is linear with respect to the NEE also for SEEDS, as the main advantage of the SETD method is to analytically compute the stochastic components in solver, making their computation cost negligible. Some mild improvements about repetitive terms allowed our implementation of SEEDS be slightly more effective than EDM and even DPM-Solver at same NFE.
51

---

Table 8: FID comparison of different solvers on CelebA 64x64 discrete. The symbol is used when using 1 NFE more and when using NFE less because the given NFE cannot be divided by
METHOD \ NFF
9
12
15
21
51
60
90
102
200
E-M
331022
227.16
207.97
1220.44
129.25
3.90
AN.-DDPM
*28.99
25.27
21.80
18.14 14
f11.23
6.51
AN.-DDIM
*15.62
13.90
12.29
10.45
16.13
3.46
DDIM
'10.85
9.99
7.78
16.64
15.23
4.78
OPM-SOLVER
6.92
4.20
3.05
2.82
2.72
SEEDS
460.87
374.48
301.66
261.87
3.84
6.58
1.88
1.97
Table 9: SEEDS-3 on CIFAA--10-com<xy--conttuuuu (using the baseline model in [66
METHOD NFE
30
60
120
129
1500
165
SEEDS
239.2
167.5
131.1
25.06
3.19
2.17
2.08
2.15
2.16
2.19
F.6 Hardware configuration
During the experiments, we used three Linux-based servers with 60GB memory each and 4 GPUs NVIDIA V100 32GB, 4 GPUs NVIDIA V100 16GB, and GPUs NVIDIA V100 32GB, respectively, Table 12 shows the detail of the configuration utilized for each experiment. We noted that when using the 4 GPUs configuration, the FID results were slightly lower (around 2%). even after using stacked fixed random seed. We run the experiments multiple times and reported the minimum FID value each time.
F.7 Licences
Pre-trained models:
CIFAR-10 models by 34 Apache V2.0 license
FFHQ-64 model by [16]: Creative Commons Atrribtio--onnCommeeraaahheeeee 4.0 International License.
CelebA 64 model by [33]: Apache V2.0 license
ImageNet-66 model by [6] Apache V2.0 license
Inception-ve model by 35 Apache V2.0 license
F.8 Supporting samples
In this subsection, we report the image grids supporting our claims in section 5
52

---

Table 10: Tailored SEEDS-3 for the EDM- EDM-preconditioned pretrained model [16 on ImageNet 64x64.
METHOD NFE
15
102
270
SEEDS
209.12
197.79
153.72
63.75
16.35
1.56
1.38
Table 11:
Run-time comparison (second/batch std) single NVIDIA V100 of EDM, DPM- Solver and SEEDS. Discretely-trained models are run with the implementation based on the code
from 22 Continuously-trainee models using the code from [16].
METHOD \ NFE
21
51
90
CIFAR-10 32 x32 CONTINUOUS (BATCH SIZE = 128)
EDM
2.096(10.003 4.891
11.592(=0.019)
21.213(110000)
DPM-SOLVER
2.099(10.002)
4.871(10.004)
11.841(20.014)
20.888(10.020)
SEEDS
2.086(#0.002)
4.867(=000003)
11.817(=0.006)
20.896(#0.028)
FFHQ 64x64 64 CONTINUOUS (BATCH SIZE 128)
EDM
4.361(10.005) 4.344(10.0055 4.303(10.003)
10.179(10.005)
24.738(10.018)
44.145(10.016)
DPM-SOLVER
10.148(10.009)
24.637(=0.007)
43.526(10.029)
SEEDS
10.157(110.004)
24.661()0.011)
43.537(100.03)
IMAGENET 64x64 CONTINUOUS (BATCH SIZE 128)
EDM
7.525(+:0.007)
17.579(#0.004)
42.696(=0.010)
76.175(+0.004)
DPM-SOLVER
7.535(++0.04) 17.556(+0.009) 7.429(+:0.06))
42.629(=0.018)
75.222(+00026)
SEEDS
17.572(+0.013)
42.654(=0.014)
75.245(+00.034)
CIFAR-10 32x32 DISCRETE (BATCH SIZE 128)
SOLVEOLVER
0.272(1:.005) 0,529(10.00)
1.32((=0.007)
2.632(110.04))
SEEDS
0.261(+0.002)
0.523(+0.002)
1.299(=0.003)
2.598(+0.003)
CELEBA 64 DISCRETE (BATCH SIZE = 128)
DPM-SOLVER
0.936(10.005) 0004) 1.812(110000)
4.558(-0.008)
9.108(10.008)
SEEDS
0.912(10.002)
1.800(1).004)
4.526(10.002)
9.033(10.004)
LSUN-BEDROOM
256x256 DISCRETE (BATCH SIZE = 32)
DPM-SOLVER
5.566(10.019) 11.124(10.018) 11.001(11.022) 27.503(=0.029)
27.815(=0.031)
55.648(10.021)
SEEDS
5.498(110008)
54.842(100.20)
LDM-CELEBAHQ
256x256 (BATCH SIZE 64)
8.648(#0.013) 17.492(20.013) 8.652(#00005)
39.569(=0.015)
68.205(+0.017)
SEEDS
17.469(+0.010)
39.524(=0.010)
68.240(110026)
STABLE DIFFUSION 512x512 (BATCH SIZE 16)
SOL-SOLVER
19.598(10.070) 40.679(10.107) 19.656(10.106)
92.914(20.079)
163.902(10.116)
SEEDS
40.781(20.136)
93.409(=0.123)
(11.45)(10.518)
Table 12: Details of GPUs utilized during the experiments.
EXPERIMENT
MODEL
NUMBER
GPU SIZF
CIFAR-10 CONTINUOUS
NVIDIA V100
16 GB
CIFAR-10 DISCRETE
NVIDIA V100
2
32 GB
FFHQ64
NVIDIA V100
4
16 GB
CELEBA64
NVIDIA A100
2
16 GB
IMAGENET64
NVIDIA V100
4
32 GB
53

---

NFE-9
NFE=21
NFE=30
NFE=90
NFE=129
NFE=165
NP
DP
Figure 4: Samples CIFAR-I4 10 from low to high NFEs by SEEDS-3 in Noise Prediction (NP)
and Data Prediction (DP) modes, using conditional VP continuous baseline model [6]]
NFE-9
NFE-21
NFE=30
NFE-90
NFE=129
NFE=165
NP
DP
Figure 5: Samples CIFAR- 10 from low to high NFEs by SEEDS-3
3 in Noise Prediction (NP)
and Data Prediction (DP) modes, using unconditional VP continuous baseline model [16].
54

---

NFE=9
NFE=21
NFE=30
NFE=60
NFE=90
NFE=150
SEEDO
DPM DPM-Solver Solver-3
Figure 6: Samples on CelebA-64 from low to high NFEs by SEEDS-3 and DPM- DPM-Solver-3, using pre-trained model from 33
NFE-9
NFE=21
NFE-30
NFE=60
NFE=90
NFE=150
NP
DP
Figure 7: Samples on FFHQ-64 from low to high NFEs by SEEDS-3
3 in Noise Prediction (NP)
and Data Prediction (DP) modes, using unconditional VP continuous baseline model [16].
55

---

NFE=30
NFE=60
NFE=90
NFE=150
NFE=210
NFE=270
MEAS
NP
DP
Figure 8: Samples ImageNet-644 from low to high NFEs in Noise Prediction (NP) and Data
Prediction (DP) modes, using conditional EDM optimized model [16].
NFE-30
NFE=60
NFE=90
NFE=150
NFE-210
NFE=270
NP
DP
Figure 9: Samples ImageNet-64 from low to high NFEs in Noise Prediction (NP) and Data
Prediction (DP) modes, using conditional EDM bascline model [[6].
56

---

Figure 10: Example of samples on CelebA-64 generated by SEEDS-3 in 90 NFEs, using pre-trained model from [33]
DPM-Solver-3 at 44 NFE.
(b) SEEDS-3 at 201 NFE.
Figure 11: Visual sample quality comparison between DPM-Solver-3 and SEEDS-3 using their optimal settings and unconditional CIFAR-10 discrete model 34
57

---

(a) DPM-Solver-3 at 80 NFF.
(b) EDM 80 NFF.
(c) SEEDS-3 at 150 NFE.
Figure 12: Visual sample quality comparison between DPM- DPM-Solver-3, EDM and SEEDS-3 using their optimal settings and unconditional VP FFHQ-64 continuous model 16
(a) DPM-Solver-3 at 30 NFE.
(b) EDM at 90 NFE.
(c) SEEDS-3 at 150 NFE.
Figure 13: Visual sample quality comparison between DPM-Solver-3, EDM and SEEDS-3 using their optimal settings and conditional VP CIFAR- 10 baseline continuous model [16].
58

---

Figure 14: Example of samples on by SEEDS-3 in 201 NFEs, using pre-trained
model from [8)
59

---

Figure 15: StableDiffusion5512 x 512 by SEEDS-1 with prompt "High quality photo of an astronaut riding a horse in space" and NFE- = 90.
60


Reviews:
[{"review_id": "BfIrjqxVKz", "reviewer": "Reviewer_nfDR", "summary": "This paper aims to accelerate the SDE solvers through exponential integrators. The authors compute the linear part of the SDE analytically, employ a more accurate interaction of the stochastic part. Experimentally, the proposed method improves over the previous ODE solvers.", "questions": "- Is this work a straightforward application of prior works ([1], [2]) to SDEs?\n\n- In Figure 1, why does the SEEDS solver have a such high FID score (worse sample quality) in the low NFE regime? Is it due to the error when simulating the noise term?", "limitations": "Discussed in Sec 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes to utilize the exponential integrator to solve the diffusion SDEs (Eq. (5)). In addition, it approximates the integral components through Taylor expansion.\n\n- The paper utilizes the stochastic exponential time differencing method to compute the variance of the stochastic term analytically.\n\n- Experimentally, the third-order solver yields better FID than previous ODE solvers while requiring significantly smaller NFEs than SDE solvers.", "weaknesses": "- It seems that the same formula of the exponential integrated version of SDE (Eq.5) has been proposed in prior works [1] (see their eq.17, arXiv version 1). Since the DPM-solver [2] is essentially the same as DEIS, and they also use a Talyer expansion of the score function, I wonder how this work differs from these prior works. \n\n- The authors show in Prop 4.4 that setting $g=0$ in Eq.15, the resulting SEEDs solvers do not equal to DPM-solver. I wonder why this is the case based on the point above. In addition, setting $g=0$ in Eq.15 would not lead to meaningful backward ODE/SDEs.\n\n- The SEEDS-3 solver indeed showcases great improvement over ODE solvers. However, when using an increased NFE, the EDM solvers perform better than SEEDS-3 (SEEDS-3 even gets worse when increasing NFE as shown in Fig.1). In addition, I notice the SEEDS solver does not even apply to a low NFE regime (NFE<100) on CIFAR-10. It might be helpful to utilize the techniques in a concurrent work [3] to further improve the sample quality when increasing the NFE, or to achieve better results in the low NFE regime. It also is helpful to discuss [3] as they also focus on improving the existing SDEs.\n\n### Minors\n\n- Line 74 : $\\mu_t \\to \\alpha_t$\n\n\n\n[1] Zhang, Qinsheng, and Yongxin Chen. \"Fast sampling of diffusion models with exponential integrator.\" arXiv preprint arXiv:2204.13902 (2022).\n\n[2] Lu, Cheng, et al. \"Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.\" Advances in Neural Information Processing Systems 35 (2022): 5775-5787\n\n[3] Xu, Yilun, et al. \"Restart Sampling for Improving Generative Processes.\" arXiv preprint arXiv:2306.14878 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer nfDR for their effort reading our work. \n\n> Is this work a straightforward application of prior works ([1], [2]) to SDEs?\n\nWe have clarified in a general rebuttal response what building principles in SEEDS are our own contributions. In particular:\n- Building principle of eq.5 will not be presented as novel in the revised version\n- The change-of-variables used in the EDM case (Prop. 3.2) is new and leads to the version of SEEDS achieving same performance as optimized EDM solver but twice faster than the latter.\n    - We point out that Fig.1(c) is not a FID/NFE curve.\n    - We refer to Fig.1 of the attached PDF file to this rebuttal to illustrate that SEEDS stabilizes at increasing NFEs.\n- The truncated Itô-Taylor expansion, the SETD method and specially the specific combination of stochastic terms that we used are novel and not incremental over [DPM-Solver,DEIS].\n- Although for small datasets, ODE methods have demonstrated to achieve optimal performance, already for ImageNet64 it has been shown in [Karras, Fig. 5(c)] the necessity of stochasticity to obtain optimal performance. It is precisely in that scenario that SEEDS show competitive results with twice less NFEs that EDM solver.\n- Setting $g=0$ in Eq. 15 yields a both a valid ODE and valid ODE solvers. But such ODE is the PFO of a forward SDE process which is not equivalent to the one we used (the induced marginal probability trajectories via Fokker-Planck do not follow the same PDE).\n\nWe hope this will convince you that our work is not a straightforward application of the mentioned prior works.\n\n> In Figure 1, why does the SEEDS solver have a such high FID score (worse sample quality) in the low NFE regime? Is it due to the error when simulating the noise term?\n\nWe thank the reviewer for pointing out the work [XY] that appeared after our submission (so cannot be considered as a baseline for this submission). Interestingly, in [XY, Th.1] there is a formal explanation on why ODE samplers outperform SDE samplers in the small NFE regime and fall short in the large NFE regime. In particular, it is theoretically shown that, at large step sizes, it is the discretization error that dominates sampling errors while, at small step sizes, it is the approximation error that dominates it. We will include this discussion in our manuscript, hoping that this will encourage the reviewer into promoting the acceptance of our work, and we definitely look forward to combine SEEDS with the new ideas in [XY].\n\nReferences:\n\n[DPM-Solver] Lu et al. DPM-solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 steps\n\n[DEIS] Zhang & Chen. Fast Sampling of Diffusion Models with Exponential Integrator\n\n[EDM] Karras et al. Elucidating the Design Space of Diffusion-based Generative Models\n\n[XY] Xu et al. Restart Sampling for Improving Generative Processes"}, {"review_id": "bBbB2kM9Gi", "reviewer": "Reviewer_CfgK", "summary": "This paper proposes a sampling method that achieves 3 to 5 times faster computation by exploiting the semi-linearity in the time-reversal stochastic differential equation (SDE). It analytically computes the linear part and reaches optimal quality sampling. This sampling method demonstrates comparable results to existing SDE solvers on datasets like CIFAR10 or Celeba without requiring any additional training methods.", "questions": "- Is it possible to maintain fast sampling speeds for datasets with resolutions higher than 256, such as CelebA-HQ or ImageNet?\n- Apart from the ability to reach optimal performance, are there any other distinguishing features compared to the DPM-solver? For instance, does it generate a more diverse set of samples or exhibit stronger capabilities in tasks like imputation?", "limitations": "The paper has effectively summarized its limitations.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is well-written in a clear and readable manner. It is easy to understand the paper without much difficulty, and the derivation of formulas is well-guided, focusing on the essential parts. One of the strengths of the paper is that it proposes a method that achieves approximately 3 to 5 times faster computation while still reaching optimal performance, which sets it apart from other ODE-based methods. Additionally, the paper provides a thorough and clear analysis of convergence, which is commendable.", "weaknesses": "In (5), the claim of a \"Novel representation of exact solutions of diffusion SDEs\" seems to be overclaimed as the equation is well-known in the form of the weak solution of the SDE. Therefore, it is difficult to consider this as a newly discovered novel representation. Moreover, while this integral representation leads to a practical sampling method, much of it is based on formulas used in the DPM-solver [1]. This suggests that the contribution of this work may be relatively weak. The main difference from the DPM-solver appears to be the inclusion of a stochastic term, but beyond that, significant distinctions are not apparent.\n\nThe claimed advantage of reducing the number of function evaluations (NFE) by 3 to 5 times seems less compelling when compared to ODE-based methods that can achieve sampling speeds more than 50 times faster and still attain optimal performance. For more persuasive results, it would be better if this method demonstrated similar NFE to ODE-based methods while reaching optimal performance.\n\nAdditionally, it is unclear whether this method addresses any meaningful issues beyond the problem of ODE-based methods not achieving optimal performance.\n\n[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, *NeurIPS*, (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer CfgK for their effort reading our work. We have clarified in a general rebuttal response what building principles in SEEDS are our own contributions. In particular,\n- eq.5 will not be presented as novel in the revised version\n- The change-of-variables used in the EDM case (Prop. 3.2) is new and leads to the version of SEEDS achieving same performance as the optimized EDM solver but twice faster than the latter.\n- The truncated Itô-Taylor expansion, the SETD method and specially the specific combination of stochastic terms that we used are novel and not incremental over prior works.\n- Although for small datasets, ODE methods achieve optimal performance, already for ImageNet64 it has been shown in [EDM (arXiv version v2), Fig. 5(c)] the necessity of stochasticity to obtain optimal performance. It is precisely in that scenario that SEEDS show competitive results with twice less NFEs than EDM solver.\n\n> Is it possible to maintain fast sampling speeds for datasets with resolutions higher than 256, such as CelebA-HQ or ImageNet?\n\nIndeed, SEEDS maintains fast sampling speeds on higher-resolution images. For unconditional generation using Latent Diffusion Model, SEEDS is able to generate good quality images already at 100 NFEs. In the attached PDF file to this rebuttal, we added a 512x512 image generated with SEEDS-1 at 90 NFEs on Stable Diffusion. This is consistent with the suggestion in [EDM, end of Section 5], stating that more diverse datasets continue to benefit from stochastic sampling rather than deterministic sampling. We hope this convinces the reviewer on the scalability of SEEDS.\n\n> Apart from the ability to reach optimal performance, are there any other distinguishing features compared to the DPM-solver? For instance, does it generate a more diverse set of samples or exhibit stronger capabilities in tasks like imputation?\n\nWe computed the Inception Score for SEEDS and saw only marginal improvements compared to DPM-Solver so both exhibit comparable sample diversity. Nevertheless, SEEDS have at least two distinctive capabilities in the realm of adversarial robustness showing substantial capabilities compared to DPM-Solver and EDM, which we hope will be relevant enough to the reviewer for promoting the acceptance of our work.\n\n- For more than 2 years, the leaderboard of RobustBench has been dominated by Diffusion-based data augmentation techniques + Adversarial Training. The current SOTA is [DMAT, Table 4], which uses EDM-preconditioned DPMs and need to generate as many as 50M images to achieve SOTA robustness results. For ImageNet-64, [DMAT] use the EDM pretrained model+stochastic sampler for data augmentation, leading to a 5% robust accuracy improvement compared to doing so for the baseline ADM pretrained model. Since SEEDS reach same FID quality as EDM, but twice faster, we believe it will have a positive impact in this domain, making diffusion-based data augmentation schemes more affordable with limited computational capacity.\n- The work [DiffPure] uses DPM-based adversarial purification as a test-time adversarial defense. The idea is to use off-the-shelf DPMs to annihilate the adversarial content in inputs in test-time before feeding them to a pretrained classifier. Below we reproduce [DiffPure, Table 6], showing standard and robust accuracy of a WideResNet-28-10 pretrained classifier with AutoAttack $l_\\infty$ ($\\epsilon$ = 8/255) for CIFAR-10 whose inputs are purified by a pretrained CIFAR-10-VP-unconditional DPM using ODE versus SDE sampling schemes. One can see that using SDE solvers show substantial robustness capabilities compared to ODE solvers.\n\n| Sampling Method      | Standard Acc | Robust Acc     |\n| :---        |    :----:   |        :----: |\n| VP-ODE      | 90.79       | 39.86   |\n| VP-SDE   | 89.02        | 70.64     |\n\nReferences:\n\n[EDM] Karras et al. Elucidating the Design Space of Diffusion-based Generative Models\n\n[DM-AT] Wang et al. Better Diffusion Models further Improve Adversarial Training\n\n[DiffPure] Nie et al. Diffusion Models for Adversarial Purification"}, {"review_id": "ohwu7kzZXh", "reviewer": "Reviewer_MHs8", "summary": "The paper proposed an efficient method for solving diffusion SDEs with some convergence guarantee. Experiments are done to verify their claims. ", "questions": "See weaknesses.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The experiments show SEEDS-3 achieves optimal results with minimal NFE on many data sets.", "weaknesses": "1. Authors claim they have found a novel representation of the exact solution of the SDE, page 3, line 106. However, it is just a simple technique that's commonly used when solving SDEs. For example, when solving OU process, you multiply some exponential multiplier like this. I don't find anything novel about this.\n2. Essentially, in the solution, the first and last terms are treated using a multiplier and change of variables. The second is rewritten using Taylor expansion. The convergence theorem is sort of too straightforward, as the Taylor expansion of course approximates the original function.\n3. The experiments only report the NFE taken when achieving optimal performance, which I am not sure is a commonly accepted metric of evaluation in this community. Please comment on this. I am especially open to modifying the score if this question is addressed sufficiently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer MHs8 for their comments. We have clarified in a general rebuttal response what building principles in SEEDS are our own contributions. In particular:\n- Building principle of eq.5 will not be presented as novel in the revised version\n- The truncated Itô-Taylor expansion, the SETD method and specially the specific combination of stochastic terms that we used are novel and not incremental over [DPM-Solver,DEIS].\n    - We point out that this already suggests that our convergence theorem is not straightforward. We refer the reviewer to Appedix B.2 in our manuscript for its proof. In particular, the use of Itô's Lemma, Itô isometry, Doob's martingale inequality, Hölder's, Lyapounov's and Grönwall's inequality are necessary to prove our theorem.\n\n> The experiments only report the NFE taken when achieving optimal performance, which I am not sure is a commonly accepted metric of evaluation in this community. \n\nActually, besides optimal FID, we also report FID/NFE curves (in Fig. 1) as evaluation metrics (with supporting FID\\NFE Table 3--5 in App. E.4). For sake of completeness, and in hope to fully address the reviewers question, we report below FID\\NFE tables for experiments that lead to the results in Table 1 and we will add them in our manuscript.\n\nIn what follows: †=discrete models. ‡=continuous models. (+) resp. (-) means value at n+1 resp. n-1 NFE.\n\n### CIFAR-10-uncond-vp‡\n\n(*value obtained with the non-deep vp pretrained model. The reported 2.59 FID at 51 NFE in Table 1 of our manuscript is obtained with the \"VP deep\" architecture)\n| Method\\NFE|11|15|24|30|48|63|126|165|180|511\n|:---|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|Euler-Maruyama|304.73(-)|248.13|\\ |\\ |66.32(NFE=50)|\\ |\\ |\\ |12.27(NFE=200)|2.44(NFE=1000)\n|GottaGoFast|\\ |\\ |\\ |\\ |82.42|\\ |\\ |2.73(NFE=151)|2.44|\\\n|EDM $(\\text{S}_{\\text{churn}}=0)$|18.41|6.52|3.52(-)|3.10(+)|2.99(-)|2.94|2.98(+)|\\ |\\ |2.93\n|EDM $(\\text{S}\\_{\\text{tmin,tmax}}+\\text{S}\\_{\\text{noise}}=1)$|34.31|11.26|4.59(-)|3.44(+)|2.89(-)|2.77|2.72|\\ |\\ |2.69\n|EDM $(\\text{S}_{\\text{noise}}=1)$|35.21|12.44|5.38(-)|3.99(+)|3.13(-)|2.90|2.60|\\ |\\ |2.55\n|EDM $(\\text{S}_{\\text{tmin,tmax}}=[0,\\infty])$|29.63|10.01|4.51(-)|3.43(+)|2.87(-)|2.71|2.52|\\ |\\ |2.54\n|EDM $(\\text{optimal})$|29.89|10.21|4.85(-)|3.77(+)|3.08(-)|2.84|2.47|\\ |\\ |**2.27**\n|DPM-Solver-3|9.04|3.76|3.00|2.95|2.90|2.89*|\\ |\\ |\\ |\\\n|SEEDS-3 $(\\text{S}_{\\text{churn}}=0)$|\\ |\\ |\\ |288.22(+)|90.25(-)|33.91|2.45|**2.39**|2.45|\\\n\n### CelebA 64x64†\n\n|Method\\NFE|9|12|15|21|51|60|90|102|201\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:| :---:|:---:\n|Euler-Maruyama|310.22(+)|227.16|207.97|120.44(-)|29.25(-)|\\ |\\ |\\ |3.90(-)\n|Analytic-DDPM|28.99(+)|25.27|21.80|18.14(-)|11.23(-)|\\ | \\ | \\ |6.51(-)\n|Analytic-DDIM|15.62(+)|13.90|12.29|10.45(-)|6.13(-)|\\ |\\  |\\ |3.46(-)\n|DDIM|10.85(+)|9.99|7.78|6.64(-)|5.23(-)|\\ |\\ |\\ |4.78(-)\n|DPM-Solver-3|6.92|4.20|3.05|2.82|**2.71**(NFE=36)\n|SEEDS-3|460.87|374.48|301.66|261.87|3.84|6.58|**1.88**|1.97|\\\n\nIn particular, the reported values for DPM-Solver-3 on CIFAR-10-vp-uncond (on the non-deep model) were computed by us, with sole purpose to propose a FID\\NFE baseline. For baseline CIFAR-10-vp-conditional and optimized ImageNet64-EDM-preconditioned pretrained DPM, we didn't find FID/NFE curves that we could use as a baseline but we still report various FID values at increasing NFEs so that future works can use them as baselines.\n\n### SEEDS-3 on other datasets\n\n|Dataset\\NFE|15|21|30|60|90|120|129|150|165|180\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:\n|CIFAR-10-cond-vp‡|239.23|167.58|131.09| 25.06|3.19|2.17|2.08|2.15|2.16|2.19\n\n|Dataset\\NFE|12|15|21|51|102|201|270\n|:---|:---:|:---:|:---:|:---:| :---:|:---:|:---:\n| ImageNet 64x64‡|209.12|197.79|153.72|63.75|16.35|1.56|1.38\n\n### Discussion on evaluation metrics for DPM sampling methods\n\nAll benchmarks on image generation tasks in paperswithcode.com classify papers by FID score and, when applicable, the corresponding NFE.\nWhile comparing solver's performance should ideally be done by comparing their FID/NFE curves at fixed datasets & pretrained DPMs, computing such curve is computationally expensive. As such, we have many FID/NFE curves on CIFAR-10, which become strong comparison baselines (one baseline per pretrained DPM on CIFAR). Usually, authors establish FID/NFE curves on small datasets to illustrate the solver's convergence behaviour and then proceed to report FID scores at specific NFE values. Some limitations:\n- [EDM] does not report FID/NFE curves for FFHQ, AFHQ and EDM-preconditioned ImageNet64, only showing the best obtained FID score for those. As such, using FID curves to compare SEEDS to EDM on those datasets is not possible unless we produce ourselves EDM curves. Additionally, [EDM] computes FID curves for CIFAR-10-VP uncond. cont. but not in the \"deep\" version (which has 8 instead of 4 residual blocks) and the FID points are given on different NFEs than [DPM-Solver] (which uses the \"deep\" architecture). Evaluating solvers is something that needs to be done with fixed pretrained DPMs and making FID/NFE tables needs to have FID values at same NFEs.\n\nWe hope to have convinced the reviewer that we did our best to use the ideal FID/NFE curve+table evaluation metric whenever it was feasible, and reported the best found FID+corresponding NFE for all experiments. We also hope we addressed a discussion on evaluation metrics that will satisfy the reviewer and help promoting the acceptance of our work.\n\nReferences:\n\n[DEIS] Zhang & Chen. Fast Sampling of Diffusion Models with Exponential Integrator\n\n[EDM] Karras et al. Elucidating the Design Space of Diffusion-based Generative Models\n\n[DPM-Solver] Lu et al. DPM-solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 steps"}, {"review_id": "JiT6rIQuE3", "reviewer": "Reviewer_daVP", "summary": "This paper proposes an off-the-shelf (i.e., no further training) few-step sampler for diffusion probabilistic models (DPMs). By isolating linear terms in the exact solution of diffusion SDEs and using a change-of-variable method, the proposed sampler, SEEDS, simplifies the integrals and can approximate the solution of stochastic equations. SEEDS can accelerate the sampling of DPMs without compromising the quality of the samples.", "questions": "To discuss the generation speed of the SEEDS-3 algorithm in Table 1, it is necessary to compare it quantitatively with the actual runtime, as well as with NFE. For example, in the case of the DPM-Solver, a comparison of DDIM with respect to runtime was made in Table 7 of Appendix E.7.", "limitations": "The authors mentioned the limations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- (Quality & Clarity) The authors have successfully provided a theoretical derivation and error analysis of their proposed method, as well as details of the experiments. The paper is well-written and the reviewer had no difficulty comprehending its contents.\n\n- (Significance) Numerical experiments show that SEEDS produces competitive results compared to previous sampling methods in terms of quality, with fewer function evaluations, in several benchmarks, namely CIFAR-10, CelebA-64, and ImageNet-64.", "weaknesses": "- (Originality) The proposed method has shown promising performance. However, its theoretical background is based on a classical idea of the *variation-of-parameters* formula in the literature of differential equations. Although the authors claim that equation (5) is a novel representation of diffusion SDEs, this kind of solution form is very well-known in the SDE community even for general Hilbert spaces (e.g. see Curtain & Falb and Da Prato & Zabczyk). Therefore, it seems that justification is needed to claim that the contribution of this paper regarding the solution representation suggested by the author is novel. Furthermore, the derivation of exponentially weighted integral is quite similar to the DPM-solver (Lu et al.), except for utilizing Itō-Taylor expansion due to the diffusion part in the solution representation. The convergence result of the proposed algorithm may not be an incremental result. However, this paper's argument on the contribution has an excessive aspect, and it is quite difficult for reviewers to agree with the author's claim.\n\n[Curtain & Falb] Stochastic Differential Equations in Hilbert Space, *Journal of Differential Equations*, (1971).\n\n[Da Prato & Zabczyk] Stochastic Equations in Infinite Dimensions, (2014).\n\n[Lu et al.] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, *NeurIPS*, (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank and agree with reviewer daVP on the poor choice of words for presenting the well-known the variation-of-parameters formula and we remove all mentions of novelty in this point giving the impression of overselling our work. Instead, we highlighted in the general rebuttal response what we find to be novel in our work.\n\nAs requested, we report below the average runtime of a batch on a single NVIDIA V100 of EDM, DPM-Solver and SEEDS for sampling (at increasing NFEs) by all diffusion models mentioned in Table 1, and additional higher dimension models: Latent Diffusion Model (LDM) and Stable Diffusion (SD). For reference, we added in Fig.2 of the appended PDF file a 512x512 image generated by Stable Diffusion using SEEDS-1 at only 90 NFEs to ensure that the runtime in this case follows an improvement in generation quality. We set batch sizes of 16 for SD, 32 for LSUN-Bedroom, 64 for CelebaHQ256 and 128 for all other datasets. One can see that the runtime is linear with respect to the NFE also for SEEDS, as the main advantage of the SETD method is to analytically compute the stochastic components in our solver, making their computation cost negligible. Some mild improvements about repetitive terms allowed our implementation of SEEDS to be slightly more effective than EDM and even DPM-Solver at same NFE.\n\n**Runtime comparison (second /batch + std) on a single NVIDIA V100 of EDM, DPM-Solver and SEEDS**\n\n(*Discretely-trained models, implementation based on the [DDIM] code.†computed using the [EDM] unmodified code. ‡our implementation in the [EDM] code.)\n| Sampling method\\NFE           | 9 | 21 | 51 | 90 | 99 |\n| :---              |    :---:   |    :---: |  :---: |  :---: |  :---: |  \n| **CIFAR-10 32x32** |\n| EDM† | 2.096(±0.003)|4.891(±0.004)|11.952(±0.019)|21.213(±0.009)|23.086(±0.010)|\n| DPM-Solver‡ | 2.099(±0.002) |\t4.871(±0.004) |\t11.841(±0.014) |\t20.888(±0.020) |\t23.006(±0.025)|\n| SEEDS‡ | 2.086(±0.002) |\t4.867(±0.003) |\t11.817(±0.006) |\t20.896(±0.028) |\t22.957(±0.009)|\n| **FFHQ 64x64** |\n| EDM† | 4.361(±0.005)|\t10.179(±0.005)|\t24.738(±0.018)|\t44.145(±0.016)|\t48.007(±0.020)|\n| DPM-Solver‡ | 4.344(±0.005)|\t10.148(±0.009)|\t24.637(±0.007)|\t43.526(±0.029)|\t47.857(±0.025)|\n| SEEDS‡ | 4.353(±0.003)|\t10.157(±0.004)|\t24.661(±0.011)|\t43.537(±0.013)|\t47.885(±0.018)|\n| **ImageNet 64x64** |\n| EDM† | 7.525(±0.007)|\t17.579(±0.004)|\t42.696(±0.010)|\t76.175(±0.024)|\t82.870(±0.032)|\n| DPM-Solver‡ | 7.535(±0.048)|\t17.556(±0.009)|\t42.629(±0.018)|\t75.222(±0.026)|\t82.725(±0.016)|\n| SEEDS‡ | 7.429(±0.006)|\t17.572(±0.013)|\t42.654(±0.014)|\t75.245(±0.034)|\t82.793(±0.036)|\n| **CIFAR-10 32x32*** |\n| DPM-Solver | 0.272(±0.004)|\t0.529(±0.007)|\t1.324(±0.007)|\t2.632(±0.004)|\t2.895(±0.004)|\n| SEEDS | 0.261(±0.002)|\t0.523(±0.002)|\t1.299(±0.003)|\t2.598(±0.003)|\t2.863(±0.008)|\n| **Celeba 64x64***|\n| DPM-Solver | 0.936(±0.004)|\t1.821(±0.003)|\t4.558(±0.008)|\t9.108(±0.008)|\t10.030(±0.008)|\n| SEEDS | 0.912(±0.009)|\t1.808(±0.004)|\t4.526(±0.002)|\t9.033(±0.004)|\t9.922(±0.012)|\n| **LSUN-Bedroom 256x256*** |\n| DPM-Solver | 5.566(±0.019)|\t11.124(±0.018)|\t27.815(±0.031)|\t55.648(±0.021)|\t61.168(±0.026)|\n| SEEDS | 5.498(±0.008)|\t11.001(±0.022)|\t27.503(±0.029)|\t54.842(±0.020)|\t60.262(±0.027)|\n| **LDM-CelebAHQ 256x256**|\n| DPM-Solver | 8.648(±0.013)|\t17.492(±0.013)|\t39.569(±0.015)|\t68.205(±0.017)| - |\n| SEEDS | 8.652(±0.005)|\t17.469(±0.010)|\t39.524(±0.010)|\t68.240(±0.026)| - |\n| **Stable Diffusion 512x512** |\n| DPM-Solver | 19.598(±0.070)|40.679(±0.107)|92.914(±0.079)|163.902(±0.116)|-|\n| SEEDS | 19.656(±0.106)|\t40.781(±0.136)|\t93.409(±0.123)|\t161.451(±0.518)| - |\n\nWe hope to have addressed all concerns from the reviewer necessary to promote the acceptance of our work.\n\nReferences:\n\n[DDIM] Song et al. Denoising Diffusion Implicit Models\n\n[EDM] Karras et al. Elucidating the Design Space of Diffusion-based Generative Models\n\n[DPM-Solver] Lu et al. DPM-solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 steps\n"}, {"review_id": "S9KjeSsGO9", "reviewer": "Reviewer_YvZd", "summary": "There are powerful ODE solvers to speed up the sampling process of diffusion models. Despite being quick, ODE solvers do not usually reach the optimal quality achieved by SDE solvers which are however slow. To tackle this problem, the paper proposes SEEDS, which is based on Exponential Integrator in the stochastic case. SEEDS are derivative and training-free solvers with proven convergence order.", "questions": "Would you please include the connection between the proposed method SEEDS-1/2/3 and the stochastic Runge-Kutta method properly combined with Exponential Integrator? \n\nAlso, the convergence order included is order 1 for SEEDS-1 in the mean-square sense and order 1 for SEEDS-2/3 in the weak sense, since the stochastic Runge-Kutta method can achieve higher convergence order even in the strong sense, the reviewer is concerned the convergence order provided is not optimal.", "limitations": "The author addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide exact solutions for diffusion SDEs and analytical computation of their variance.\n\n2. The paper contains rigorous theoretical derivation and convergence order guarantee for the proposed methods. It also points out its connection with gDDIM.\n\n3. The paper is overall well-written and self-contained.\n\n4. The method achieves outstanding sampling performance.", "weaknesses": "The proposed SEEDS method still needs around a hundred NFEs to achieve good sampling qualities, the quality will degrade fast when NFEs decrease, which limits its applications.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful lecture of our work. Below a discussion to the raised questions, which will be included in the paper.\n\n> Would you please include the connection between the proposed method SEEDS-1/2/3 and the stochastic Runge-Kutta method properly combined with Exponential Integrator?\n\nContrary to the ODE case, there are many stochastic Runge-Kutta approaches, usually tailored for SDEs of a specific form. Nonetheless, a common way of distinguishing solvers with same strong order is to assign them couples $(p_d,p_s)$, where $p_s$ is the stochastic order and $p_d$ is the resulting order determined if setting $g=0$ in the considered SDE i.e. when they are deterministic. Example: [R, Tab. 6.2] determines solvers with orders (1,1.0) and (2,1.0) respectively and [R, Tab. 6.3] determines solvers SRA1 and SRA3 with orders (2,1.5) and (3,1.5) respectively. Many of these solvers' speed was already tested in [GGF, Table 3] on CIFAR-10 (VP) which we reproduce below with our SEEDS method added to it.\n\n| Method      | Strong-Order | Speed     | \n| :---        |    :----:   |          :---: |\n| Euler-Maruyama (EM)      | 0.5       | Baseline speed  |\n|SEEDS | 1 | 6.83x faster |\n|Lamba EM (atol=1e-3)  |0.5 | 2x faster |\n|Lamba Euler-Heun  |0.5 | 1.75x faster |\n|Lamba EM (atol=1e-3, rtol=1e-3)  |0.5 | 1.27x faster |\n|Euler-Heun |0.5  | 1.86x slower |\n|SOSRA |1.5 | 5.92x slower |\n|SRA3  |1.5 | 6.93x slower |\n|SOSRI  |1.5 | 8.57x slower |\n|Lamba EM (default)  |0.5 | Diverged |\n|RKMil |1.0 | Diverged |\n|ImplicitRKMil  |1.0 | Diverged |\n|ISSEM | 0.5 | Diverged |\n\nTo our knowledge, the only available strong order SERKs method for SDEs with inhomogeneous diffusion coefficients are the exponential Euler-Maruyama (EEM) method [K] and the stochastic RK Lawson (SRKL) schemes [D]. In short, the SRKL schemes only compute analytically the linear coefficient and use the Integrating Factor (IF) method to approximate the integrals in the representation of the exact solution given by the variation-of-parameters formula. This way, by a special change of variables (see [D, Alg. 1]), one can create exponential integrator versions of many SDE methods. We implemented our own version of the SRKL schemes to take into account that $\\sigma,\\alpha$ are not constant and used the Integrating Factor method to approximate the integrals in the representation of the exact solution given by the variation-of-parameters formula (after properly changing variables). Interestingly, the SRKL schemes seem to stabilize at increasing NFEs but at much higher FID values than their SETD counterparts.\n\n**Comparison of SEEDS with current Stochastic RKL methods on CIFAR-10-vp-uncond (discrete)**\n\n| Method\\NFEs      | 10 | 20 | 50 | 100 | Best known \n| :---        |    :----:   |          :---: |   :---: |   :---: |   :---: |\n| SRKL1($\\lambda$)      |    332.52   |   282.96 |  33.42   |   8.62 | \\ |\n| SEEDS-1   |     303.48    |   153.21    |  22,70   |   7.97 | 3.13(500NFE) |\n| SRKL2($\\lambda$)      |    475.20   |   469.64 |  134.82   |   7.74 | \\ |\n| SEEDS-2   |     476.9    |   226.7    |  7.17   |   3.23  | 3.21(90NFE) |\n| SRKL3($\\lambda$)       |    462.24   |   376.15 |  8.36   |   7.46 | \\ |\n| SEEDS-3   |     483.0    |   428.6    |  43.3   |   3.41 | 3.08(201NFE) |\n\n> Also, the convergence order included is order 1 for SEEDS-1 in the mean-square sense and order 1 for SEEDS-2/3 in the weak sense, since the stochastic Runge-Kutta method can achieve higher convergence order even in the strong sense, the reviewer is concerned the convergence order provided is not optimal.\n\nThe proposed convergence order for each SEEDS-1/2/3 is optimal: this is a consequence of the general result from [CC] about maximum convergence rates for SDE schemes with uncorrelated Gaussian increments. The underlying idea is that any solver with strong order $\\geq 1.5$ has to account for double stochastic integrals in the non-truncated Itô-Taylor expansion, ultimately forcing any SRK-like solver to use correlated random variables (see [R, Tab. 6.3] and [KP] more generally). SEEDS avoids this additional complexity but an interesting future avenue would be to extend SEEDS to the higher strong order case (and not in the IF approach but the SETD approach). Another interesting path would be to craft weak second order SERK methods for DPMs (the work [KCB] addressed this only for homogeneous semi-linear SDEs).\n\nWe hope to have addressed all necessary concerns from the reviewer to promote the acceptance of our paper.\n\nReferences:\n\n[R] Rossler. Runge-Kutta Methods for the Strong Approximation of Solutions of Stochastic Differential Equations\n\n[K] Komori. Exponential Runge-Kutta Methods for Stiff Stochastic Differential Equations\n\n[D] Debrabant et al. Runge-Kutta Lawson Schemes for Stochastic Differential Equations\n\n[GGF] Jolicoeur-Martineau et al. Gotta Go Fast when Generating Data with Score-based Models\n\n[CC] Clark & Cameron. The Maximum Rate of Convergence of Discrete Approximations for Stochastic Differential Equations\n\n[KP] Kloeden & Platen. Numerical Solution of Stochastic Differential Equations\n\n[KCB] Komori et al. Weak Second Order Explicit Exponential Runge-Kutta methods for Stochastic Differential Equations"}]

Author rebuttal:
## General Rebuttal Response
As reviewers 1-4 pointed out, and with whom we fully agree, the widely known variation-of-parameters formula is not a contribution of ours and we will modify this in our manuscript. In our effort to clarify the building principles of SEEDS, we unintentionally stressed incorrectly which of those principles are novelties in our work. 

Let's clarify the 4 contributions :
- Using the stochastic exponential time differencing (SETD) method (l.130) to express the noise variance in terms of the $\varphi$ functions for the exponentially weighted stochastic integrals is new and not incremental over [DPM-solver]
- The truncated Itô-Taylor expansion is a new tool. It is different from - and not incremental over - Taylor expansions (as highlighted by reviewer daVP and fully detailed in App. D.2.2)
- Our principled use of the Chasles rule (l.182) to enforce dependence on overlapping paths for SEEDS-2/3 is new and is the central key of success of SEEDS. It also ensures that the set of resulting iterations of our solvers satisfy the Markov property. Furthermore, we claim that the specially chosen combination of stochastic terms in SEEDS is not incremental over any prior work. To highlight this, we added an ablation study on how choosing different noise decompositions (e.g. the values of A and B in Algorithm 4) has an impact on the performance of SEEDS. We consider 4 different combinations of noise components in SEEDS-2/3. For simplicity let's explain it at the SEEDS-2 (Alg. 3) level.
    - Set $(z^1,z^2)$ two independent standard Gaussian random variables. Denote $A=\sigma_{s_1} \sqrt{e^h - 1} z^1$ for the noise contribution in the $\mathbf{u}$ term. We have the following choices for the noise contribution for $\tilde{\mathbf{x}}_t$:
        - SEEDS-2: our noise combination $B=\sigma_t \left( \sqrt{e^{2 h} - e^h} z^1 + \sqrt{e^h - 1} z^2 \right)$
        - Naive v1: one noise per stage $B=\sigma_t \left( \sqrt{e^{2 h} - e^h} + \sqrt{e^h - 1} \right) z^2$
        - Naive v2: one noise per step $B=\sigma_t \left( \sqrt{e^{2 h} - e^h} + \sqrt{e^h - 1} \right) z^1$
        - Naive v3: one noise per integral evaluation $B=\sigma_t \left( \sqrt{e^{2 h} - e^h} z^3 + \sqrt{e^h - 1} z^2 \right)$ (where $z^3$ is also standard Gaussian)
        - Naive v4: noises in inverse position $B=\sigma_t \left( \sqrt{e^{2 h} - e^h} z^2 + \sqrt{e^h - 1} z^1 \right)$

     For each of these combinations, we generated FID/NFE curves (see Fig. 1 in the attached PDF file), showing that using all naive combinations of noises in SEEDS-2/3 lead to a sharp drop of performance both in quality and speed. Finally, our choice is enforced with a theoretical justification of it (using the independence property of stochastic integrals on disjoint intervals) that has no counterpart in the deterministic setting. All convergence proofs in Appendix B make intense use of formulae (like Itô Isometry or Doob's martingale inequality) that also have no deterministic equivalent.
- The change-of-variables optimized for the EDM framework is new (our Prop. 3.2) and is the one we use to achieve same performance as EDM solver but with twice less NFEs than the latter for ImageNet-64 (Fig. 1(c) in our manuscript).
To give an example of the difference in the obtained formulas for SEEDS in this case, we write below the SEEDS-2 iteration rule, and we can add it to the manuscript at the request of the reviewers:
    - $\lambda (t) := - \log \left( \frac{t}{\sigma_d \sqrt{t^2 + \sigma^2_d}}
\right) \qquad t_{\lambda} (\lambda) :=
\frac{\sigma_d}{\sqrt{\frac{1}{\sigma^2_d e^{- 2 \lambda}} - 1}}, \quad
\sigma_d = \sigma_{data}$
    - $h = \lambda_t - \lambda_s, \quad \lambda_{s_1} = \lambda_s +
rh , \quad s_1 = t_{\lambda} (\lambda_{s_1})$
    - $\epsilon_{\theta} \left( \widetilde{\mathbf{x}}_s, s \right)$ = $\left[ D\_{\theta}
\left( \frac{\widetilde{\mathbf{x}}_s}{\alpha_s} ; \sigma_s \right) -
\frac{\sigma_d^2}{s^2 + \sigma_d^2} \frac{\widetilde{\mathbf{x}}_s}{\alpha_s} \right]
\cdot \frac{\sqrt{s^2 + \sigma_d^2}}{s \sigma_d},$
    - $\widetilde{\mathbf{x}}_{s_1} = \frac{s_1^2 + \sigma_d^2}{s^2 + \sigma_d^2}
\widetilde{\mathbf{x}}_s + 2 \frac{s_1  \sqrt{s_1^2 + \sigma_d^2}}{\sigma_d} (e^{r h} - 1){\epsilon\_{\theta}}\left( \widetilde{\mathbf{x}}_s, s \right) + \sqrt{\frac{(s_1^2 + \sigma_d^2) (s^2 - s_1^2)}{(s^2 + \sigma_d^2)}} z^1$
    - $\widetilde{\mathbf{x}}_t = \frac{t^2 + \sigma_d^2}{s^2 + \sigma_d^2} \widetilde{\mathbf{x}}_s + 2 \frac{t  \sqrt{t^2 + \sigma_d^2}}{\sigma_d} (e^{ h} - 1) \epsilon\_{\theta}\left({\widetilde{\mathbf{x}}\_{s_1}}, s_1 \right) + \sqrt{\frac{(t^2 + \sigma_d^2) (s_1^2 - t^2)}{(s_1^2 + \sigma_d^2)}} (e^{r h} z^1 + z^2) $

We hope that this clarification both highlights: on the one hand, the building principles of SEEDS, and on the other hand, the original contributions of our work. Finally, we would like to emphasise that many available optimisation tools for sampling (for instance multi-stepping, dynamic thresholding,...) can be readily built on top of SEEDS to better address specific tasks like guided sampling.

References:

[DPM-Solver] Lu et al. DPM-solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 steps

Meta-review: This paper proposes a fast SDE solver while maintaining the optimal performance of the SDE in sampling from pre-trained diffusion probabilistic models. Especially, the proposed three SDE solvers (SEEDS) are derivative and training free with proven convergence guarantees and empirically demonstrates the optimal performances in terms of quality with small NFEs compared to the previous SDE solvers on image generation benchmarks.
Overall, the main contribution of this paper seems to the proposed analytic computation of the “stochastic” component and the corresponding “convergence guarantees”, which is different from the existing fast ODE solvers such as DPM-solver. I think this contribution is not marginal and the feasibility and the practicality have been sufficiently validated from experimental results. Therefore, I recommend this paper to be accepted.
Having said that, some reviewers claim that the derivation of the proposed solution for the stochastic term in the SDE seems to be based on common and general approaches and the resulting guaranteed convergence would be somewhat straightforward. Hence, the authors need to thoroughly and clearly address these points (more than the responses in the rebuttal) in the revision.
Decision: Accept (poster)