Title: SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion Probabilistic Models

Exponential SDE methods

Image Generation

Generative Models
Abstract: A potent class of generative models known as Diffusion Probabilistic Models
(DPMs) has become prominent. A forward diffusion process adds gradually noise
to data, while a model learns to gradually denoise. Sampling from pre-trained
DPMs is obtained by solving differential equations (DE) defined by the learnt
model, a process which has shown to be prohibitively slow. Numerous efforts on
speeding-up this process have consisted on crafting powerful ODE solvers.
Despite being quick, such solvers do not usually reach the optimal quality
achieved by available slow SDE solvers. Our goal is to propose SDE solvers that
reach optimal quality without requiring several hundreds or thousands of NFEs
to achieve that goal. We propose Stochastic Explicit Exponential
Derivative-free Solvers (SEEDS), improving and generalizing Exponential
Integrator approaches to the stochastic case on several frameworks. 
After carefully analyzing the formulation of exact
solutions of diffusion SDEs, we craft SEEDS to analytically compute the linear
part of such solutions. Inspired by the Exponential Time-Differencing method,
SEEDS use a novel treatment of the stochastic components of solutions,
enabling the analytical computation of their variance, and contains high-order
terms allowing to reach optimal quality sampling $\sim3$-$5\times$ faster than previous
SDE methods. We validate our approach on several image generation benchmarks,
showing that SEEDS outperform or are competitive with previous SDE solvers.
Contrary to the latter, SEEDS are derivative and training free, and we fully
prove strong convergence guarantees for them.

SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models

Abstract

A potent class of generative models known as Diffusion Probabilistic Models (DPMs) has become prominent. A forward diffusion process adds gradually noise to data, while a model learns to gradually denoise. Sampling from pre-trained DPMs is obtained by solving differential equations (DE) defined by the learnt model, a process which has shown to be prohibitively slow. Numerous efforts on speeding-up this process have consisted on crafting powerful ODE solvers. Despite being quick, such solvers do not usually reach the optimal quality achieved by available slow SDE solvers. Our goal is to propose SDE solvers that reach optimal quality without requiring several hundreds or thousands of NFEs to achieve that goal. We propose Stochastic Explicit Exponential Derivative-free Solvers (SEEDS), improving and generalizing Exponential Integrator approaches to the stochastic case on several frameworks. After carefully analyzing the formulation of exact solutions of diffusion SDEs, we craft SEEDS to analytically compute the linear part of such solutions. Inspired by the Exponential Time-Differencing method, SEEDS use a novel treatment of the stochastic components of solutions, enabling the analytical computation of their variance, and contains high-order terms allowing to reach optimal quality sampling ∼ 3 -5 × faster than previous SDE methods. We validate our approach on several image generation benchmarks, showing that SEEDS outperform or are competitive with previous SDE solvers. Contrary to the latter, SEEDS are derivative and training free, and we fully prove strong convergence guarantees for them. Our code is publicly available in this link.

1 Introduction

Diffusion Probabilistic Models (DPMs) [32, 12] have emerged as a powerful category of generative models and have proven to quickly become SOTA for generative tasks such as image, video, audio generation [8, 24, 13, 3, 4], and more [29, 36]. These models employ a forward diffusion process where noise is gradually added to the data, and the model learns to remove the noise progressively. However, sampling from most pre-trained DPMs is done by simulating the trajectories of associated differential equations (DE) and has been found to be prohibitively slow [34]. Previous attempts to accelerate this process have mainly focused on developing efficient ODE solvers. On one hand, training-based methods speed-up sampling by using auxiliary training such as Progressive Distillation [31] and Fourier Neural Operators [42], learning the noise schedule, scaling, variance, or trajectories. On the other hand, training-free methods [16, 22, 15, 40, 23] are slower but are more versatile for

Elies Gherbi IRT SystemX

Hatem Hajri IRT SystemX &Safran

being employed on different models and achieve higher quality results than current training-based methods. Although these solvers are fast, they often fall short of achieving the optimal quality attained by slower SDE solvers [16]. The latter usually do not present theoretical convergence guarantees and, while being training-free, they often still require costly parameter optimization to achieve optimal results which might be difficult to estimate for large datasets.

Our objective is to introduce SDE solvers that can achieve optimal quality without requiring an excessively large number of function evaluations (NFEs). To accomplish this, we propose Stochastic Explicit Exponential Derivative-free Solvers (SEEDS). These are off-the-shelf SDE samplers: they offer promising high-quality sampling without further training or parameter optimization. SEEDS enhance and generalize existing Exponential Integrator [22, 23, 41] approaches to the stochastic case on several practical frameworks and is based on the following 4 building blocks: (1) the exponential representation of semi-linear SDE exact solutions which isolate linear terms to be computed analytically; (2) a general change-of-variables recipe to simplify the integrals involved in the solutions in order to better approximate the deterministic one; (3) a method to analytically compute the variance of the stochastic one; (4) a method to decompose the obtained stochastic components in such a way that the resulting sequences of higher-stage numerical approximations yield Markov chains.

Overall, we make the following contributions: (a) our change-of-variables method allow us to reframe the gDDIM solver [41] as a special case of SEEDS and to craft bespoke solvers for the EDM-preconditioned DPMs in [16] which attain equivalent sampling quality twice faster than the previous SOTA sampling method [16]; (b) based on the Stochastic Exponential Time-Differencing method, we analytically compute of our solver's stochastic components in terms of the so-called φ -functions, allowing for efficient implementation; (c) our noise decomposition method (4), which is the key of success of SEEDS, is both theoretically grounded, experimentally shown to be optimal, and has no deterministic equivalent; (d) we provide full proofs of strong/weak convergence guarantees for our SDE solvers which, to our knowledge, has no precedent in the DPM literature. In particular, although the formula used for the truncated Itô-Taylor expansion might seem similar to that of [22], our convergence theorems leverage the full Itô-Taylor expansion of solutions, making our proofs not incremental and different from [22]; (e) we conduct extensive experiments demonstrating that SEEDS establishes SOTA results among available solvers on several image generation benchmarks, or is competitive with existing SDE solvers while being 2-5 times faster than the latter.

Although our solvers theoretically apply to certain non-isotropic DPMs such as Critically-damped Langevin Dynamics (CLD) [10] (see Prop. 4.5 and Rem. 4.6), we will restrict our presentation to the isotropic case for which many notations become simpler.

2 Background on Diffusion Probabilistic Models

General Isotropic DE Formulation. The evolution of a data sample x 0 ∈ R d taken from an unknown data distribution p data into standard Gaussian noise can be defined as a forward diffusion process { x t } t ∈ [0 ,T ] , with T > 0 , which is a solution to a linear SDE:

where f ( t ) , g ( t ) ∈ R d × d are called the drift and diffusion coefficients respectively and ω is a d -dimensional standard Wiener process, and α t , σ t ∈ R > 0 are differentiable functions with bounded derivatives. In practice, when specifying the SDE (1), σ t acts as a schedule controlling the noise levels of an input at time t , and α t as a time-dependent signal scaling controlling its dynamic range.

By denoting p t ( x t ) the marginal distribution of x t at time t , functions α t and σ t are designed so that the end-time distribution of the process process is p T ( x T ) ≈ N ( x T | 0 , ˜ σ 2 I d ) for some ˜ σ > 0 . As (1) is linear, the transition probability p 0 t ( x t | x 0 ) from x 0 to x t is Gaussian whose mean and variance can be expressed in terms of α t and σ t . For simplicity, we will denote it

The evolution of the reverse time process of { x t } t ∈ [0 ,T ] (which we will still denote { x t } t ∈ [0 ,T ] for simplicity) is then driven by a backward differential equation

where d t are negative infinitesimal time-steps and ¯ ω t is now a Wiener process with variance -d t . In this article, we will concentrate in the cases ℓ = 0 , 1 , known in the literature as the Probability Flow ODE (PFO) and diffusion reverse SDE (RSDE), respectively.

Training. Denoising score-matching is a technique to train a time-dependent model D θ ( x t , t ) to approach the score function ∇ x t log p t ( x t ) at each time t . Intuitively, as D θ approaches the score, it produces a sample which maximizes the log-likelihood. As such, this model is coined as a data prediction model. However, in practice DPMs can be more efficiently trained by reparameterizing D θ into a different model F θ ( x t , t ) whose objective is to predict the noise to be removed from a sample at time t . This noise prediction model is trained by means of the loss

where γ t is a time dependent weighting parameter and K t K ⊤ t = Σ t .

3 Accelerating Optimal Quality Solvers for Diffusion SDEs

Once F θ or D θ have been trained, one can effectively solve (2) after replacing the score function by its corresponding expression involving either one of these models. For instance, taking the noise prediction model and ℓ = 1 , sampling is conducted by simulating trajectories of a SDE of the form

for some functions A ( t ) , b ( t ) which are usually not equal to f ( t ) , g 2 ( t ) . In what follows, we consider a time discretization { t i } i M =0 going backwards in time starting from t 0 = T to t M = 0 and to ease the notation we will always denote t < s for two consecutive time-steps t i +1 < t i .

The usual representation of the analytic solution x t at time t of (3) w.r.t. an initial condition x s is:

The numerical schemes we propose for approaching the trajectories of (3) based on representation (4) are grounded in the 4 following principles:

The variation-of-parameters formula: representing analytic solutions with linear term extracted from the integrand;

Exponentially weighted integrals: extracting the time-varying linear coefficient attached to the network from the integrand by means of a specific choice of change of variables which allows analytic computation of the leading coefficients in the truncated Itô-Taylor expansion associated to F θ ( x τ , τ ) up to any arbitrary order;

Modified Gaussian increments: after replicating such change of variables onto the stochastic integral, analytically computing its variance.

Markov-preserving noise decomposition: stochastic integrals need to be dependent on overlapping time intervals and independent on non-overlapping ones.

Exponential representation of exact solutions of diffusion SDEs. The first key insight of this work is that, using the variation-of-parameters formula, we can represent the analytic solution x t at time t of (3) with respect to an initial condition x s as follows:

where Φ A ( t, s ) = exp ( ∫ t s A ( τ )d τ ) is called the transition matrix associated with A ( t ) . The separation of the linear and nonlinear components is achieved by this formulation and also appears in [22, 23, 41]. It differs from black-box SDE solvers as it enables the exact calculation of the linear portion, thereby removing any approximation errors associated with it. However, the integration of the nonlinear portion remains complex due to the interaction of the new coefficient Φ A ( t, τ ) b ( τ ) and the intricate neural network, making it challenging to approximate.

Exponentially weighted integrals. Due to the regularity conditions usually imposed on the drift and diffusion coefficients of (1), one can make several choices of change-of-variables on the integral components in (5) in order to simplify it. Our second key insight is that there is a specific choice of change of variables allowing the analytic computation of the Itô-Taylor coefficients of F θ ( x τ , τ ) with respect to τ , and based at s that will be used for crafting SEEDS. More specifically, this expansion reads

where the residual R n consists of deterministic iterated integrals of length greater than n +1 and all iterated integrals with at least one stochastic component. As such, we obtain

where ˜ R n is easily obtained from R n and ∫ t s Φ A ( t, τ ) b ( τ )d τ . The third key contribution of our work is to rewrite, for any k ⩾ 0 , the integral ∫ t s Φ A ( t, τ ) b ( τ ) ( τ -s ) k k ! d τ as an integral of the form ∫ λ t λ s e λ ( λ -λ s ) k k ! d λ since the latter is recursively analytically computed in terms of the φ -functions

Modified Gaussian increments. In order for making such change of variables to be consistent on the overall system, one needs to replicate it accordingly in the stochastic integral ∫ t s Φ A ( t, τ ) g ( τ )d ¯ ω τ . As such, our last key contribution is to transform it into an exponentially weighted stochastic integral with integration endpoints λ s , λ t and apply the Stochastic Exponential Time Differencing (SETD) method [1] to compute its variance analytically, as illustrated in (14) below.

Let us test our methodology in two key examples. As we explained in Section 2, sampling from pre-trained DPMs amounts on choosing a schedule σ t , a scaling α t , and a parameterized learnt approximation of the score function ∇ x t log p t ( x t ) . In what follows, we denote by t λ the inverse of a chosen change of variables λ t and we denote ̂ x λ := x ( t λ ( λ )) , ˆ F θ ( ̂ x λ , λ ) := F θ ( x ( t λ ( λ )) , t λ ( λ )) .

The VPSDE case. Let ˜ α t := 1 2 β d t 2 + β m t , where β d , β m > 0 and t ∈ [0 , 1] . Then, by denoting

we obtain the VP SDE framework from [34] and the following result.

Proposition 3.1. Let t < s . The analytic solution at time t of the RSDE (2) with coefficients (7) and initial value x s is

The change of variables of (8) is interesting as it allows to compute analytically the Itô-Taylor coefficients in (6) by using, for h = λ t -λ s , the following key result which will be used in Prop. 4.2:

For instance, in the case when k = 0 , it is easy to see that ∫ λ t λ s e -λ d λ = σ t ( e h -1) and ∫ λ t λ s e -λ d ¯ ω λ obeys a normal distribution with zero mean, and one can analytically compute its variance:

The EDM case. Denote σ 2 d the variance of the considered initial dataset and set

These parameters correspond to the preconditioned EDM framework introduced in [16, Sec. 5, App. B.6]. The following result is the basis for constructing our customized SEEDS in this case, and for which we report experimental results in Table 1. For simplicity, we will write F θ ( x t , t ) for the preconditioned model in (11) and we refer to Appendix B for details.

Proposition 3.2. Let t < s . The analytic solution at time t of (2) with coefficients (11) and initial value x s is, for ℓ = 1 ,

where λ t := -log [ t σ d √ t 2 + σ 2 d ] . In the case when ℓ = 0 , it is given by

Remark 3.3 . One can wonder about the generality of such change of variables. Our method is very general in that one can always make such change of variables with very mild regularity conditions: for c : [0 , T ] -→ R > 0 integrable, with primitive C ( t ) > 0 , we have c ( t ) = e log( c ( t )) . This means we can write c ( t ) = ˙ C ( t ) = e λ t ˙ λ t with λ t = log( C ( t )) . In other words, for such c , we have

4 Higher Stage SEEDS for DPMs

In this section we present our SEEDS algorithms by putting together all the ingredients presented in the previous section. Let t < s . In all what follows, we consider the analytic solution at time t of the RSDE (2) with coefficients (7), h = λ t -λ s and initial value x s . Plugging (9) with k = 0 and (10) into the exact solution (8) allow us to infer the first SEEDS scheme, given by iterations of the form

The following Theorem gives strong order convergence guarantees for this method, which we call SEEDS-1, under mild conditions which apply to all our experiments. We stress out that its proof (App. C) is a non-trivial result, it is fundamentally different in nature from [22] and involves mathematical tools which have no deterministic counterparts.

Theorem 4.1. Under Assumption C.1, the numerical solution ˜ x t produced by the SEEDS-1 method (14) converges to the exact solution x t of

with coefficients (7) in Mean-Square sense with strong order 1.0: there is a constant C > 0 such that

Higher stage SEEDS. As announced, by fully exploiting the analytic computations enabled by the expansion (9) we now turn into crafting our multi-step SEEDS. Usually, SDE solvers are constructed by using the full Itô-Taylor expansion of the SDE solutions and usually need a big number of evaluations of the network ˆ F θ to achieve higher order of convergence. As our main concern is to present stochastic solvers with a minimal amount of NFE, we choose to truncate such Itô-Taylor expansion so that the neural networks only appear in the deterministic contributions.

Algorithm 1 Iterative procedure

Input: initial value x T , steps { t i } i M =0 , model F θ Initialize ˜ x t 0 ← x T

(

t, s

)

(

t

i

, t

i

-

1

)

,

˜

x

t

=

end for

Return ˜ x t M ← last-step (˜ x t M -1 , t M -1 , t M )

Algorithm 3 SEEDS-2 ( F θ , ˜ x s , s, t )

Algorithm 2 SEEDS-1 ( F θ , ˜ x s , s, t )

Algorithm 4 SEEDS-3 ( F θ , ˜ x s , s, t ) with 0 < r 1 < r 2 < 1

Proposition 4.2. Assume that ˆ F θ is a C 2 n +1 -function with respect to λ . Then the truncated Itô-Taylor expansion of (8) reads, for ϵ ∼ N ( 0 , I d ) ,

with ˆ F ( k ) θ ( ̂ x λ , λ ) = L k λ ˆ F θ ( ̂ x λ , λ ) , with L λ is an infinitesimal operator defined in Appendix E.2.2 and R n +1 consists on the usual deterministic residue and all iterated integrals of length at greater or equal to 2 in which there is at least one stochastic component among them.

Our approach for constructing derivative-free 2-stage and 3-stage SEEDS schemes consists on exploiting the analytic computation of the Itô-Taylor coefficients in Proposition 4.2 and replace the ˆ F ( k ) θ ( ̂ x λ , λ ) terms by well-adapted correction terms which do not need any derivative evaluation and dropping the R n +1 contribution as in the Runge-Kutta approach.

Markov-preserving noise decomposition. We use collocation methods for constructing higherstage derivative-free solvers. Although the chosen truncated Itô-Taylor expansion produces approximations for the deterministic integral similar to [22], adding the corresponding noise contribution found by the SETD method at each step does not yield Markov chains in general. The reason is that stochastic integrals on overlapping time intervals need to be dependent, a phenomenon that has no deterministic counterpart. As such, our last and key element to construct SEEDS consists on a novel decomposition of stochastic integrals which enforces the Markov property for multi-stage SEEDS.

Algorithms 1 to 4 prescribe all SEEDS schemes obtained by this procedure in the VP case. We now show (see App. C for the proofs) that all methods yield Markov chains and are weakly convergent.

Proposition 4.3. The sequences { ˜ x t } t induced by the choice of stochastic noise contributions presented in Algorithms.3 and 4 satisfy the Markov property.

Corollary 4.4. Under Assumption C.2, the numerical solutions ˜ x t produced by the SEEDS methods (3) and (4) converge to the exact solution x t of (15) with coefficients (7) in weak sense with global order 1 in both cases: there is a constant C > 0 such that, for any continuous bounded function G :

SEEDS-k

(

F

θ

h

s

λ

, s, t

)

t

-

λ

s

,

˜

x

Comparison with existing sampling methods. Let us now examine the connection between SEEDS and existing sampling techniques used for DPMs, emphasizing the contrasts between them.

The main distinctive feature of SEEDS is that they are off-the-shelf solvers. This means that, not only they are training-free , contrary to [9], but they do not require any kind of optimization procedure to achieve their optimal results. This is in contrast to methods such as: gDDIM, which is training-free but not off-the-shelf as one needs to make preliminary optimization procedures such as simulating the transition matrix of their method in the CLD case; Heun-Like method from EDM (for all baseline models and the EDM-optimized models for ImageNet) since they need preliminary optimization procedures on 4 parameters which actually break the convergence criteria. Moreover, neither gDDIM, EDM nor the SSCS method in [10] present full proofs of convergence for their solvers. Also, both DEIS and gDDIM identify their methods with stochastic DDIM theoretically, but the poor results obtained by their stochastic solvers do not yield to further experimentation in their works. In a way, SEEDS can be thought as improved and generalized DPM-Solver to SDEs. Nevertheless, such generalization is not incremental as the tools for proving convergence in our methods involve concepts which are exclusive to SDEs. We now make rigorous statements of the above discussion.

Proposition 4.5. Consider the SEEDS approximation of (15) with coefficients (7) . Then

If we set g = 0 in (15) , the resulting SEEDS do not yield DPM-Solver.

If we parameterize (15) in terms of the data prediction model D θ , the resulting SEEDS are not equivalent to their noise prediction counterparts defined in Alg. 1 to 4.

The gDDIM solver [41, Th. 1] equals to SEEDS-1 in the data prediction mode, for ℓ = 1 .

The first point makes it explicit that SEEDS are not incremental based on DPM-Solver. The second point in Prop. 4.5 is analog to the result in Appendix B of [23], where the authors compare DPMSolver2 and DPM-Solver++(2S), that is the noise and data prediction approaches, and find that they do not equate. The last point exhibits gDDIM as a special case of SEEDS-1 for isotropic DPMs.

Remark 4.6 . Building solvers from the representation of the exact solution in (5) requires computing the transition matrix Φ A ( t, s ) , which cannot be analytically computed for non-isotropic DPMs such as CLD [10]. Nevertheless, the SEEDS approach can be applied in this scenario in at least two different ways. On the one hand, the SSCS method from [10] resides splitting Φ A ( t, s ) into two separate terms. The first can be analytically computed. The second describes the evolution of a semi-linear differential equation [10, Eq. 92]. While [10] approximates the latter by the Euler method, crafting exponential integrators for approximating such DE may yield an acceleration of the SSCS method. On the other hand, gDDIM [41] proposes an extension of DEIS sampling [40] to CLD by setting a pre-sampling phase [41, App. C.4] in which they compute an approximation of Φ A ( t, s ) in order to apply their method, and the latter was shown in Prop. 4.5. to be a special case of our method. Unfortunately, the authors did not release pre-trained models in [41], and the latter are not the same as those in [10]. Sampling in this scenario may also benefit from our approach.

5 Experiments

We compare SEEDS with several previous methods on discretely and continuously pre-trained DPMs. We report results of many available sources, such as DDPM [12], Analytic DDPM [2], PNDM [20], GGF [15], DDIM [33], gDDIM [41], DEIS [40] and DPM-Solver [22]. Although we do not include training-based schemes here, we still included GENIE [9], which trains a small additional network but still solves the correct generative ODE at higher-order. For each experiment, we compute the FID score for 50K sampled images on multiple runs and report the minimum along different solvers. Details on model specifications and experiment illustrations are shown in Appendix F.

Practical considerations. For continuously trained models, SEEDS use the EDM discretization [16, Eq. 5] with default parameters and does not use the last-step iteration trick , meaning that the last iteration of SEEDS is trivial. For discretely trained models, SEEDS use the linear step schedule in the interval [ λ t 0 , λ t N ] interval following [22, Sec. 3.3, 3.4]. All the reported SEEDS results were obtained using the noise prediction mode. We conducted comparative experiments on SEEDS for both the data and noise prediction modes and found better results with the latter (see Tab. 3 for details). EDM solvers [16, Alg. 2] depend on four parameters controlling the amount of noise to

NFE

=50

100

150

200

250

300

350

400

3

.

2

3

.

4

3

.

6

3

.

8

4

.

0

FID

3

.

87

3

.

60

3

.

58

3

.

40

3

.

19

3

.

08

DPM-Solver-1

DPM-Solver-2

DPM-Solver-3

SEEDS-1 (Ours)

SEEDS-2 (Ours)

SEEDS-3 (Ours)

NFE

=16

32

64

128

256

512

1024

3

.

0

3

.

5

4

4

.

5

5

.

0

FID

3

.

16

3

.

84

3

.

6

3

.

95

3

.

58

3

.

08

Euler-Maruyama

Analytic DDPM

Analytic DDIM

DDIM

DPM-Solver-3

SEEDS-3

S

churn

=0

10

20

30

40

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

2

.

4

2

.

6

FID

1

.

36

1

.

38

2

.

66

2

.

22

1

.

66

EDM

†

, 511 NFE

EDM

⋆

, 511 NFE

SEEDS

⋆

,

270 NFE

(a) Uncond. CIFAR, VP

(b) Uncond. CIFAR, VP

(c) Stochasticity on ImageNet-64

be injected in a specific sub-interval of the iterative procedure. We consider three scenarios: when stochasticity is injected along all the iterative procedure, we denote it stochastic EDM, when no stochasticity is injected we denote it EDM ( S churn = 0 ) and we denote EDM (Optimized) the case where such parameters were subject to an optimization procedure. To better evaluate sampling quality along the sample pre-trained DPM, we recalculate DPM-Solver for sampling from the non-deep VP continuous model on the CIFAR-10 dataset. All implementation details can be found in Appendix D.

Comparison with previous works. In Table 1 we compare SEEDS with other sampling methods for pre-trained DPMs, and report the minimum FID obtained and their respective NFE. For each of the reported pre-trained models in CIFAR-10, CelebA-64 and ImageNet-64, SEEDS outperform all off-the-shelf methods in terms of quality with relatively low NFEs. For the discrete pre-trained DPM on CIFAR-10 (VP Uncond.) it is ∼ 5 × faster than the second best performant solver. Additionally, SEEDS remain competitive with the optimized EDM sampler. For ImageNet-64, it is nearly as good

disc=vp, schedule=vp, scaling=vp

disc=edm, schedule=vp, scaling=vp

disc=edm, schedule=linear, scaling=none

SEEDS-3

STEP

=0

10

20

30

40

50

Pixel state

0

10

20

30

40

50

0

10

20

30

40

50

25

35

Stochastic EDM

STEP

=0

10

20

30

40

50

Pixel state

0

10

20

30

40

50

0

10

20

30

40

50

25

35

as the optimized EDM sampler while being almost twice faster than the latter. Figure 1 (a) compares the FID score of SEEDS and DPM-Solver with varying NFEs. While DPM-Solver methods stabilize faster in a very low NFE regime, our methods eventually surpass them. Interestingly, after reaching their minimum, SEEDS methods tend to become worse at higher NFEs, a fact that is also visible in Figure 1 (b), where we notice that such phenomenon is also present on other SDE methods. We report in Appendix F, the results of our SEEDS methods in the low NFE regime and connect their behavior with their proven convergence rate.

Combining SEEDS with other methods. While being an off-the-shelf method, SEEDS can be combined with the Churn-like method used in EDM incurring into SDE solvers with an additional source of stochasticity. As done in [16], we evaluate the effect of this second kind of stochasticity, measured by a parameter denoted S churn . Figure 1 (c) shows that SEEDS and EDM show similar behavior, although SEEDS-3 is twice faster, more sensitive to S churn , and quickly achieves comparable performance to EDM. This indicates that SEEDS could possibly outperform EDM after a proper parameter optimization that will be left for future works. Nevertheless, we highlight the fact that obtaining such optimal parameters is costly and might scale poorly.

Stiffness reduction with SEEDS. In Fig. 2, we illustrate the impact of different choices of discretization steps, noise schedule and dynamic scaling on SEEDS and stochastic EDM. We see that choosing the EDM discretization over the linear one has the effect of flattening the pixel trajectories at latest stages of the simulation procedure. Also, choosing the parameters (11) over those in (7) has the effect of greatly changing the distribution variances as the trajectories evolve. Notice that all the SEEDS trajectories seem perceptually more stable than those from EDM. It would be interesting to relate this to the stiffness of the semi-linear DE describing these trajectories, and to the magnitude of the parameters involved in the noise injection for EDM solver amplifying this phenomenon.

Ablation studies. As said earlier, our principled use of the Chasles rule to enforce independence only on non-overlapping paths for SEEDS-2/3 ensures that the set of resulting iterations of our solvers satisfy the Markov property, is new and is the central key of success of SEEDS. To highlight this,

NFE

=8

30

60

90

120

150

2

90

180

270

360

FID

2

.

33

2

.

36

261

.

24

45

.

1

65

.

06

106

.

73

18

.

53

23

.

89

SEEDS-2-Correct

SEEDS-2-Naive-1

SEEDS-2-Naive-2

SEEDS-2-Naive-3

SEEDS-2-Naive-4

NFE

=36

60

90

120

150

2

80

160

240

320

400

FID

2

.

08

2

.

19

307

.

62

275

.

88

90

.

80

6

.

68

2

.

43

SEEDS-3-Correct

SEEDS-3-Naive-1

SEEDS-3-Naive-2

SEEDS-3-Naive-3

SEEDS-3-Naive-4

(a) 2-stage SDE methods

(b) 3-stage SDE methods

we conduct an ablation study on how 4 different combinations of the noise components A and B in Algorithms 3 and 4 have an impact on the sampling quality of SEEDS.

For simplicity, we explain this for SEEDS-2 (Alg. 3). Set ( z 1 , z 2 , z 3 ) three independent standard Gaussian random variables. Denote A = ¯ σ s 1 √ e h -1 z 1 for the noise contribution in the u term. We have the following choices for the noise contribution B in ˜ x t :

SEEDS-2-Correct: our noise combination B = ¯ σ t ( √ e 2 h -e h z 1 + √ e h -1 z 2 )

SEEDS-2-Naive-1: one noise per stage B = ¯ σ t ( √ e 2 h -e h + √ e h -1 ) z 2

SEEDS-2-Naive-2: one noise per step B = ¯ σ t ( √ e 2 h -e h + √ e h -1 ) z 1

SEEDS-2-Naive-3: one noise per integral evaluation B = ¯ σ t ( √ e 2 h -e h z 3 + √ e h -1 z 2 )

SEEDS-2-Naive-4: noises in inverse position B = ¯ σ t ( √ e 2 h -e h z 2 + √ e h -1 z 1 ) .

Figure 3 experimentally shows that all naive combinations of noises for SEEDS-2 and SEEDS-3 lead to FID/NFE curves with different behavior and a sharp drop of performance both in quality and speed in all of them, a phenomenon having no deterministic parallel.

6 Conclusions

Our focus is on addressing the challenge of training-free sampling from DPMs without compromising sampling quality. To achieve this, we introduce SEEDS, an off-the-shelf solution for solving diffusion SDEs. SEEDS capitalize on the semi-linearity of diffusion SDEs by approximating a simplified formulation of their exact solutions. Inspired by numerical methods for stochastic exponential integrators, we propose three SEEDS schemes with proven convergence order. They transform the integrals involved in the exact solution into exponentially weighted integrals, and estimate the deterministic one while analytically computing the variance of the stochastic integral. We extend our approach to handle other isotropic DPMs, and evaluate its performance on various benchmark tests. Our experiments demonstrate that SEEDS can generate images of optimal quality, outperforming existing SDE solvers while being 3 ∼ 5 × faster. Limitations and broader impact. While SEEDS prioritize optimal quality sampling, they may require substantial computational resources and energy consumption, making them less suitable for scenarios where speed is the primary concern. In such cases, alternative ODE methods may be more appropriate. Additionally, as with other generative models, DPMs can be employed to create misleading or harmful content, and our proposed solver could inadvertently amplify the negative impact of generative AI for malicious purposes.

Acknowledgments. This work has been supported by the French government under the 'France 2030' program, as part of the SystemX Technological Research Institute.

References

ADAMU, I. Numerical Approximation of SDEs and Stochastic Swift-Hohenberg Equation (PhD Thesis, Heriot-Watt University), 2011.

BAO, F., LI, C., ZHU, J., AND ZHANG, B. Analytic-DPM: An Analytic Estimate Of The Optimal Reverse Variance In Diffusion Probabilistic Models. ICLR (2022).

CHEN, N., ZHANG, Y., ZEN, H., WEISS, R. J., NOROUZI, M., AND CHAN, W. WaveGrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713 (2020).

CHEN, N., ZHANG, Y., ZEN, H., WEISS, R. J., NOROUZI, M., DEHAK, N., AND CHAN, W. WaveGrad 2: Iterative refinement for text-to-speech synthesis. arXiv preprint arXiv:2106.09660 (2021).

CHEN, S., CHEWI, S., LI, J., LI, Y., SALIM, A., AND ZHANG, A. R. Sampling is as Easy as Learning the Score: Theory for Diffusion Models with Minimal Data Assumptions. arXiv:2209.11215 (2022).

CLARK, J. M. C., AND CAMERON, R. J. The Maximum Rate of Convergence of Discrete Approximations for Stochastic Differential Equations. In Stochastic Differential Systems Filtering and Control (Berlin, Heidelberg, 1980), Springer Berlin Heidelberg, pp. 162-171.

DEBRABANT, K., KVÆRNØ, A., AND MATTSSON, N. C. Runge-Kutta Lawson schemes for Stochastic Differential Equations. BIT Numerical Mathematics 61 (2021), 381-409.

DHARIWAL, P., AND NICHOL, A. Diffusion Models Beat GANs On Image Synthesis. Advances in Neural Information Processing Systems 34 (2021), 8780-8794.

DOCKHORN, T., VAHDAT, A., AND KREIS, K. GENIE: Higher-order Denoising Diffusion Solvers. arXiv preprint arXiv:2210.05475 (2022).

DOCKHORN, T., VAHDAT, A., AND KREIS, K. Score-Based Generative Modeling with Critically-Damped Langevin Diffusion. In International Conference on Learning Representations (ICLR) (2022).

GONZALEZ, M., HAJRI, H., CANTAT, L., AND PETRECZKY, M. Noisy Learning of Neural ODEs Acts as a Robustness Locus Widening. In International Conference on Machine Learning (PODS Workshop) (2022).

HO, J., JAIN, A., AND ABBEEL, P. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems 33 (2020), 6840-6851.

HO, J., SALIMANS, T., GRITSENKO, A., CHAN, W., NOROUZI, M., AND FLEET, D. J. Video Diffusion Models. arXiv:2204.03458 (2022).

HOCHBRUCK, M., AND OSTERMANN, A. Exponential Integrators. Acta Numerica 19 (2010), 209-286.

JOLICOEUR-MARTINEAU, A., LI, K., PICHÉ-TAILLEFER, R., KACHMAN, T., AND MITLIAGKAS, I. Gotta Go Fast When Generating Data With Score-Based Models. arXiv preprint arXiv:2105.14080 (2021).

KARRAS, T., AITTALA, M., AILA, T., AND LAINE, S. Elucidating the Design Space of Diffusion-Based Generative Models. In Proc. NeurIPS (2022).

KLOEDEN, P. E., AND PLATEN, E. Numerical Solution Of Stochastic Differential Equations. Stochastic Modelling And Applied Probability . Applications of Mathematics. Springer, 1992.

KOMORI, Y. Exponential Runge-Kutta Methods for Stiff Stochastic Differential Equations. RIMS (2016), 128-140.

KOMORI, Y., COHEN, D., AND BURRAGE, K. Weak Second Order Explicit Exponential Runge-Kutta Methods For Stochastic Differential Equations. SIAM Journal on Scientific Computing 39 , 6 (2017), A2857-A2878.

LIU, L., REN, Y., LIN, Z., AND ZHAO, Z. Pseudo Numerical Methods for Diffusion Models on Manifolds. arXiv preprint arXiv:2202.09778 (2022).

LU, C., ZHENG, K., BAO, F., CHEN, J., LI, C., AND ZHU, J. Maximum Likelihood Training for Score-Based Diffusion ODEs by High Order Denoising Score Matching. In International Conference on Machine Learning (2022), PMLR, pp. 14429-14460.

LU, C., ZHOU, Y., BAO, F., CHEN, J., LI, C., AND ZHU, J. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. Advances in Neural Information Processing Systems (2022).

LU, C., ZHOU, Y., BAO, F., CHEN, J., LI, C., AND ZHU, J. DPM-Solver++: Fast Solver For Guided Sampling Of Diffusion Probabilistic Models. arXiv preprint arXiv:2211.01095 (2022).

MENG, C., SONG, Y., SONG, J., WU, J., ZHU, J.-Y., AND ERMON, S. SDEdit: Image Synthesis And Editing With Stochastic Differential Equations. arXiv preprint arXiv:2108.01073 (2021).

MILSTEIN, G., AND TRETYAKOV, M. Stochastic Numerics for Mathematical Physics . Scientific Computation series, Springer, Berlin, 2021.

NICHOL, A. Q., AND DHARIWAL, P. Improved Denoising Diffusion Probabilistic Models. In International Conference on Machine Learning (2021), PMLR, pp. 8162-8171.

NIE, W., GUO, B., HUANG, Y., XIAO, C., VAHDAT, A., AND ANANDKUMAR, A. Diffusion Models for Adversarial Purification. In International Conference on Machine Learning (ICML) (2022).

PENG, S., XU, W., CORNELIUS, C., HULL, M., LI, K., DUGGAL, R., PHUTE, M., MARTIN, J., AND CHAU, D. H. Robust Principles: Architectural Design Principles for Adversarially Robust CNNs. arXiv preprint arXiv:2308.16258 (2023).

POPOV, V., VOVK, I., GOGORYAN, V., SADEKOVA, T., AND KUDINOV, M. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International Conference on Machine Learning (2021), M. Meila and T. Zhang, Eds., vol. 139, PMLR.

RÖSSLER, A. Runge-Kutta Methods for the Strong Approximation of Solutions of Stochastic Differential Equations. SIAM Journal on Numerical Analysis 48 , 3 (2010), 922-952.

SALIMANS, T., AND HO, J. Progressive Distillation For Fast Sampling Of Diffusion Models. arXiv preprint arXiv:2202.00512 (2022).

SOHL-DICKSTEIN, J., WEISS, E., MAHESWARANATHAN, N., AND GANGULI, S. Deep Unsupervised Learning Using Nonequilibrium Thermodynamics. In International Conference on Machine Learning (2015), PMLR, pp. 2256-2265.

SONG, J., MENG, C., AND ERMON, S. Denoising Diffusion Implicit Models. arXiv:2010.02502 (2020).

SONG, Y., SOHL-DICKSTEIN, J., KINGMA, D. P., KUMAR, A., ERMON, S., AND POOLE, B. ScoreBased Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations (2021).

SZEGEDY, C., VANHOUCKE, V., IOFFE, S., SHLENS, J., AND WOJNA, Z. Rethinking the Inception Architecture for Computer Vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 2818-2826.

VAHDAT, A., KREIS, K., AND KAUTZ, J. Score-based Generative Modeling in Latent Space. In Neural Information Processing Systems (NeurIPS) (2021).

WANG, Z., PANG, T., DU, C., LIN, M., LIU, W., AND YAN, S. Better Diffusion Models Further Improve Adversarial Training. In International Conference on Machine Learning (ICML) (2023).

XU, Y., DENG, M., CHENG, X., TIAN, Y., LIU, Z., AND JAAKKOLA, T. Restart Sampling for Improving Generative Processes. arXiv:2306.14878 (2023).

YAN, H., DU, J., TAN, V. Y. F., AND FENG, J. On robustness of neural ordinary differential equations. In International Conference on Learning Representations (2020).

ZHANG, Q., AND CHEN, Y. Fast Sampling of Diffusion Models with Exponential Integrator. arXiv preprint arXiv:2204.13902 (2022).

ZHANG, Q., TAO, M., AND CHEN, Y. gDDIM: Generalized Denoising Diffusion Implicit Models. arXiv preprint arXiv:2206.05564 (2022).

ZHENG, H., NIE, W., VAHDAT, A., AZIZZADENESHELI, K., AND ANANDKUMAR, A. Fast Sampling of Diffusion Models via Operator Learning. arXiv preprint arXiv:2211.13449 (2022).

Contents

A Discussion

Why do SEEDS exhibit high FID scores in the low NFE regime? In Appendix F.4, we provide all tables with FID values at increasing NFEs corresponding to Fig. 1. Notice that SEEDS exhibit high FID scores in the low NFE regime. Let us discuss some theoretical facts that might explain this phenomenon. In [38, Theorem 1.], there is a formal explanation on why ODE samplers outperform SDE samplers in the small NFE regime and fall short in the large NFE regime. In particular, it is theoretically shown that, at large step sizes, it is the discretization error that dominates sampling errors while, at small step sizes, it is the approximation error that dominates it. One can infer that in the large NFE regime, SDE methods (with proven convergence orders) will outperform ODE methods in terms of sampling quality. It would be interesting to see if combining SEEDS with the new ideas in [38] might improve SEEDS in the low NFE regime.

Additionally, SDE solvers and ODE solvers may have different sample quality, even if the score models are optimal. Indeed, as proven in [21, Appendix B.], even if the score model has been trained to the optimal score function, the distributions between SDEs and ODEs are still different. This is because the distribution at time T in the forward process is always not exactly a standard normal distribution as in the reverse SDE/ODE process, and thus the distributions at time 0 are different.

Finally, [38, 5] show that the overall generalization error divides into discretisation and approximation errors, indicating the theoretical role of the sampling error as a contribution to a DPM's performance. This is also consistent with the suggestion in [16, Section 5], stating that more diverse datasets continue to benefit from stochastic sampling rather than deterministic sampling.

Do SEEDS maintain good performances for higher resolution image generation? As trainingfree and optimization-free SDE solvers, SEEDS naturally maintain good performances in higher

resolution images in the realm of unguided image generation (unconditional and conditional). For unconditional generation using Latent Diffusion Model, SEEDS are able to generate good quality images already at 100 NFEs. Now, for guided image generation, higher-stage SEEDS will, as expected, see their performance sharply drop as the guidance scale grows for the same reasons DPM-solver do (see [23] for details). Yet, SEEDS-1 still maintains high quality sampling in this scenario. In Appendix F.8, Fig. 15 exhibits a 512 2 image generated with SEEDS-1 at 90 NFEs on Stable Diffusion with default guidance scale.

Do SEEDS exhibit other distinguishing features beyond generation optimal quality? As SDE solvers, SEEDS have a distinctive (although indirect) capability in the realm of adversarial robustness compared to DPM-Solver:

For more than 2 years, the leader-board of RobustBench has been dominated by Diffusionbased data augmentation techniques on top of Adversarial Training. The current SOTA [28, 37] uses EDM-preconditioned DPMs and need to generate as many as 50M images to achieve SOTA robustness results. For ImageNet-64, [37] use the EDM pretrained model and optimization-based stochastic EDM sampler for data augmentation, leading to a 5% robust accuracy improvement compared to doing so for the baseline ADM pretrained model. Since SEEDS reach same FID quality as EDM, but twice faster, we believe it will have a positive impact in this domain, making diffusion- based data augmentation schemes more affordable with limited computational capacity.

The work [27] uses DPM-based adversarial purification as a test-time adversarial defense. The idea is to use off-the-shelf DPMs to annihilate the adversarial content in inputs in test-time before feeding it to a pretrained classifier. In [27, Table 6], one can see that SDE solvers show substantial robustness capabilities compared to ODE solvers.

We'd like to stress out that, although indirect, such capabilities are on the side of the sampling methods rather than on the side of intrinsic robustness properties of score-based learning method. More broadly, robustness properties of ML models determined as neural differential equations (DPMs being in this scope) has been studied in [39, 11].

How do SEEDS compare to Stochastic Runge-Kutta methods? Contrary to the ODE case, there are many stochastic Runge-Kutta approaches, usually tailored for SDEs of a specific form. Nonetheless, a common way of distinguishing solvers with same strong order is to assign them couples ( p d , p s ) , where p s is the (stochastic) strong convergence order and p d is the resulting (deterministic) order determined if setting g = 0 in the considered SDE i.e. when they are deterministic. For instance, [30, Tab. 6.2] determines solvers with orders (1,1.0) and (2,1.0) respectively and [30, Tab. 6.3] determines solvers SRA1 and SRA3 with orders (2,1.5) and (3,1.5) respectively. Many of these solvers' speed was already tested in [15, Table 3] on CIFAR-10 (VP). Experimentally, SEEDS-1 shows to be 6.83x faster than the baseline Euler-Maruyama scheme.

To our knowledge, the only available strong order SERKs method for SDEs with in-homogeneous diffusion coefficients are the exponential Euler-Maruyama (EEM) method [18] and the stochastic RK Lawson (SRKL) schemes [7]. In short, the SRKL schemes only compute analytically the linear coefficient and use the Integrating Factor (IF) method to approximate the integrals in the representation of the exact solution given by the variation-of-parameters formula. This way, by a special change of variables (see [7, Alg. 1]), one can create exponential integrator versions of many SDE methods. We implemented our own version of the SRKL schemes, which we denote SRKL-1/2/3, to take into account that σ, α are not constant and used the Integrating Factor method to approximate the integrals in the representation of the exact solution given by the variation-of-parameters formula and under the VP λ -change-of-variables. Interestingly, the SRKL schemes stabilize at increasing NFEs but at much higher FID values than their SETD counterparts.

In Table 2 below, we draw a comparison of SEEDS with current Stochastic RKL methods (on the λ temporal parameter space) on CIFAR-10 for the discretely trained DPM in the VP unconditional regime.

Is the proven convergence order of each of the SEEDS methods optimal? The proposed convergence orders for each SEEDS-1/2/3 is optimal: this is a consequence of the general result from [6] about maximum convergence rates for SDE schemes with uncorrelated Gaussian increments.

The underlying idea, fully detailed in Appendix E, is that any solver with strong order ≥ 1 . 5 has to account for double stochastic integrals in the non-truncated Itô-Taylor expansion, ultimately forcing any SRK-like solver to use correlated random variables (see [30, Tab. 6.3] and [17] more generally). SEEDS avoid this additional complexity but an interesting future avenue would be to extend SEEDS to the higher strong order case (and not in the IF approach but the SETD approach) as long as it doesn't incur into an explosion of needed NFEs per step to craft such solvers. Another interesting path would be to craft weak second order SERK methods for DPMs (the work [19] addressed this only for homogeneous semi-linear SDEs with constant linear coefficients).

B Detailed Derivation of the SEEDS Design Space

In Sections 2 and 3 we proposed a simplified presentation of the design space of diffusion models and of the ingredients that constitute our proposed SEEDS methodology. In this section, we further develop our presentation in a technical manner, making explicit the formalization of our design choices.

B.1 The Isotropic General SDE framework

In Section 2, we presented a parametric family of differential equations (2) driving the generative process for DPMs, based on time-reversing the forward noising diffusion process (1). While doing so, we presented two parameters - the noise schedule σ t and the scaling α t - for which the effects on DPMs have been widely studied in [16].

As the shape of the trajectories of (1) and (2) (for ℓ = 0 , 1 ) are defined by α t and σ t , we start by writing down, for the scaling x t = α t ̂ x t , the scaled generalization of the proposed SDEs in [16, Eq. 103] which unifies in a single framework the forward and reverse trajectories:

Following [16], the previous VP, VE, iDDPM, DDIM and EDM frameworks all are unified as different choices of α t , σ t , among other choices presented in [16, Tab. 1] and we will use this as a basis for all the proofs contained in this Appendix. In particular, forward time means taking x + t for which the score vanishes in this context. Now set x t = x -t .

The formulation in (2) involves a family of backwards differential equations controlled by a parameter ℓ ∈ [0 , 1] which all yield reverse-time processes for (1), a fact that can be obtained by studying the Fokker-Planck equation for marginals p ( ̂ x + t ; σ t ) of (17).

When ℓ = 1 , (2) the obtained SDE is known as the reverse SDE (RSDE) and, when ℓ = 0 , we obtain an ODE that is known as the Probability Flow ODE (PFO):

Now, finding a minimum for the loss function in [16, Eq. 51] is formulated as a convex optimization problem. As such, for the ideal model D ( x t ; σ t ) = arg min D L ( D ; x t , σ t ) , the score function with

scaled input is expressed as

This ideal model is usually subtracted by a raw network F in the form of a time-dependent preconditioning:

As such, we can express the score function as two parameterizations involving D or F as follows:

Let us now denote D 1 θ,t := D θ ( ̂ x t ; σ t ) for a pre-trained network approximating the ideal denoiser and let D 2 θ,t := F θ ( c 3 ( t ) ̂ x t ; c 4 ( t )) be the corresponding raw pre-trained network. Substituting the score function in the RSDE and the PFO with each of these models yields four different differential equations with a neural network as one of their components. These are given, for i = 1 , 2 , by

When D 1 θ,t (resp. D 2 θ,t ) is employed to replace the score function in (17) using (19), the resulting SDE (20) will be called data (resp. noise) prediction neural SDE . Proceeding analogously for the PFO (18) yield two ODEs (21) which will be called data (resp. noise) prediction neural PFO . The general form of the A i and B i coefficients determining each of these DEs is as follows:

Remark B.1 . At first glance, it would seem misleading to differentiate four DEs as these essentially correspond to different choices of α t , σ t , c 1 ( t ) , . . . , c 4 ( t ) . But the reason why we do so is that, after applying the variation of constants formula, each of these DEs will yield a different representation of their exact solutions (see (22) and (23) below). As we will see below, constructing exponential integrators heavily depends on such representation and will show to lead to four different modes of SEEDS solvers, each one showing different behavior and performance for sampling from pre-trained DPMs. As such, we will articulate this difference already at the DE formulation.

For t < s , the variation of constants formulae for NSDEs allows to represent the exact solutions of (20) as

and those for NPFOs (21) as

where

is called the transition matrix associated with A i ( t ) and is defined as the solution to

When A i ( t ) is constant and B i ( t ) = 1 , there is a well-established literature on exponential ODE and SDE solvers with explicit stiff order conditions and prescribed by different forms of Butcher tableaux. When A i ( t ) is not constant, for the expression in (24) to make sense in the usual sense (in terms of exponential series expansion) instead of having to make use of time-ordered exponentials/ Magnus expansions, the f ( t ) := A i ( t ) coefficients must satisfy [ f ( k ) ( t ) , f ( l ) ( s )] = 0 . This condition is trivially satisfied here as the A i ( t ) considered here are d -dimensional diagonal matrices.

̸

̸

Notice that, if A i = A j for some i = j , their associated transition matrices will not be equal. In particular, if A 1 = A 2 , then the variances of the stochastic integrals in (22) are different for i = 1 and i = 2 . This is the first step in explaining the statement in Rem. B.1, and we refer the reader to the proof of Proposition 4.5 where we put into evidence its validity.

̸

B.2 Re-framing and Generalizing Previous Exponential Solvers

B.2.1 The VP case

Recall that in the VP case, and the noise prediction mode , [22] construct exponential solvers on the base of the following ODE

where ¯ σ t := α t σ t . The ODE (25) identifies with that in [16] for the VP case for which the authors identify the preconditioning

As such, we obtain the following coefficients for the NP NPFO:

and

B.2.2 Proof of Proposition 3.1

First of all, denote F θ ( x t , ( M -1) t ) = ϵ θ ( x t , t ) . We have

This way, one can directly relate λ t with the signal-to-noise ratio SNR( t ) = α 2 t / ¯ σ 2 t , also being used in [22]. As such, SNR( t ) is strictly monotonically decreasing in time. Thus, the analytic solution to

(2) yields

By using the change of variables to λ ( t ) , our equation now reads

Finally, notice that α t = √ 1 1+ e -2 λt and ¯ σ t = √ 1 1+ e 2 λt so that (26) is equivalent to

This finishes the proof.

B.2.3 Proof of Proposition 4.2

Recall that the functions φ k are the integrals

which satisfy φ k (0) = 1 k ! . The truncated Itô-Taylor expansion of ̂ ϵ θ with respect to λ reads

where here ̂ ϵ ( k ) θ denotes the L k t operators defined in (58) applied to ̂ ϵ θ . On the one hand, since ∫ λ t λ s e -λ dλ = ¯ σ t α t ( e h -1) , we obtain by iteratively integrating by parts

On the other hand, we have s > t, h = λ t -λ s > 0 . Note that since the stochastic integrals ∫ λ t λ s e -λ d ¯ ω ( λ ) are measurable with respect to ( ¯ ω ( λ ) -¯ ω ( λ s ) , 0 ≤ λ ≤ λ t -λ s ) , they are independent on disjoint time intervals by the independence of increments property of Brownian motion. Thus the random variable ϵ := ϵ s,t in our algorithms are independent on disjoint time intervals. We then write

In conclusion, the truncated Itô-Taylor expansion of the analytic expression

simplifies to

This finishes the proof.

B.2.4 Generalization to the remaining data prediction and deterministic modes

Propositions 3.1 and 4.2 consist on the first steps for crafting SEEDS solvers in the VP case associated to the noise prediction neural RSDE (NP NRSDE). Generalizing the above procedure for crafting SEEDS for the 4 modes associated to (22) and (23) yields the following sets of coefficients

We readily obtain

Then, by setting the simpler change of variables λ t := -log( σ t ) , we obtain

By recursion we obtain

In the same way, we obtain

Next,

and finally, as already shown in Propositions 3.1 and 4.2:

Now, since g 2 ( t ) = 2 α 2 t ˙ σ t σ t , the stochastic integrals ∫ t s Φ i ( t, τ ) g ( τ )d ¯ ω τ , for i = 1 , 2 , have zero mean and variances:

We deduce from this the SEEDS-1 schemes in all four modes as given by iterates:

Notice that the iterates (30) and (31) are exactly the iterates of the first stage solvers in [22] and [23] with F θ ( x t ; ( M -1) t ) = ϵ θ ( x t , t ) . The iterates (29) coincide with the SEEDS-1 method presented in (14) and (28) consist on our SEEDS-1 method in the data prediction mode , which we will use in the following section.

B.2.5 Proof of Proposition 4.5

We will write in bold the statement to be proven.

If we set g = 0 in (15) , the resulting SEEDS solvers do not yield DPM-Solver. Indeed, if we set g = 0 in (15), then the method (29) does not contain a noise contribution and we readily see that it cannot be equal to (31). As the latter has been shown to be DPM-Solver-1, the conclusion follows.

If we parameterize (15) in terms of the data prediction model D θ , the resulting SEEDS solvers are not equivalent to their noise prediction counterparts defined in Alg. 1 to 4.

One can check that the SEEDS solver are not the same between the noise and data prediction modes by simply noticing that the noise contributions in (28) and (29) do not equate.

The gDDIM solver [41] Theorem 1, for ℓ = 1 , is equal to SEEDS-1 in the data prediction mode.

As shown in (29), our proposed method SEEDS-1 in the data prediction mode for the VP case has iterates of the form

where ϵ ∼ N ( 0 , I d ) , ¯ σ t = α t σ t and h = log σ s σ t . As our notation and that of [41, Theorem 1] overlap, we will use blue color when referring to their notation.

On the one hand, gDDIM constructs iterates over a representation of the exact solution of the following family of neural differential equations:

where α t decreases from α 0 = 1 to α T = 0 , and with coefficients

In particular, they choose an approximation, for τ ∈ [ t -∆ t, t ] , given by

The gDDIM iterates, for λ = 1 = ℓ , are then written as follows:

where ϵ ∼ N ( 0 , I d ) and

Now set ( s, t ) ← ( t, t -∆ t ) . Then

Next, we identify α t = √ α t , and ¯ σ t = √ 1 -α t . Then the variance of the noise in (35) is

Hence, by denoting ˜ x t = u ( t ) , the gDDIM iterate (34) reads, for ϵ ∼ N ( 0 , I d ) :

On the other hand, the data and noise prediction models in our case are related by the following equation:

As such, and as h = λ t -λ s = log σ s σ t , one can rewrite (32) in terms of ϵ θ as follows:

which coincides with the gDDIM iterate in Equation (36).

B.2.6 The VE, DDIM and iDDPM cases

Following [16, Eq. 217], here α t = 1 and c 1 ( t ) = 1 and so the only possibilities incurring into semi-linear differential equations are

We obtain, again with the choice λ t = -log( σ t ) and setting h = λ t -λ s , the following:

Next,

We readily see that these cases are identical to the VP case with α t = 1 . In particular, the obtained SEEDS-1 iterates are

Now denote s 1 = t λ ( λ s + rh ) , for 0 < r ⩽ 1 , where t λ = e -λ . There are two families of single-step one-parameter two-stage exponential ODE schemes:

with same supporting value

In the same vein we define a single-step two stage exponential SDE scheme:

B.2.7 The EDM case

In the EDM-preconditioned case [16, App. B.6], we set σ t := t and α t := 1 . We denote σ d := σ data the variance of the considered initial dataset and we set

so we obtain the following coefficients:

In particular, the data prediction neural SDE/PFO are identical to those in the VE case with σ t = t . So let us concentrate on the noise prediction regime, leading us to prove Proposition 3.2.

B.2.8 Proof of Proposition 3.2

In the Noise Prediction case, we have

so we readily compute:

Let us consider two different changes of variables:

that will be used for the (NP NPFO) and (NP NRSDE), respectively. For the former case, we have

Therefore, we can deduce that

For the latter case, we have

We then obtain

The stochastic integral ∫ t s Φ 2 ( t, τ ) g ( τ )d ¯ ω τ in noise prediction case is a Gaussian random variable with zero mean and whose variance can be computed by Itô isometry as

Putting everything together, we obtain the analytic solution at time t of (2) with coefficients (11) and initial value x s for the (NP NRSDE):

where λ t := -log [ t σ d √ t 2 + σ 2 d ] . For the (NP NPFO), it is given by

This finishes the proof.

Remark B.2 . From the above proof, we immediately deduce the SEEDS-1 iterates in the EDMpreconditioned noise prediction case. These are given, for the (NP NPFO) and (NP NRSDE) respectively, by

where ϵ ∼ N ( 0 , I d ) .

B.2.9 The SEEDS algorithms in the EDM Noise Prediction case

In the same way we presented the SEEDS algorithms 3 and 4 for the VP case, the change-of-variables optimized for the EDM framework in Proposition 3.2 induces algorithms 5 for SEEDS-2 and 6 for SEEDS-3 respectively, under the Noise Prediction regime. This is the version of SEEDS that we use to achieve same performance as EDM solver but with twice less NFEs than the latter for ImageNet-64 (see Table 1 and Fig. 1 (c) in the main part of this article).

Algorithm 5 SEEDS-2 (EDM NP)

end for

Return ˜ x t M ← last-step (˜ x t M -1 , t M -1 , t M )

Algorithm 6 SEEDS-3 (EDM NP)

M M -1

C Convergence Proofs

In this Section, we give detailed proofs of Theorem 4.1, Proposition 4.3 and Corollary 4.4 stated in the main part of this paper. Let us start recalling its framework. We start by considering the NP NRSDE (15) with VP coefficients:

Denote t λ the inverse of λ t := -log( σ t ) (which is a strictly decreasing function of t ) and denote ̂ x λ := x ( t λ ( λ )) , ˆ ϵ θ ( ̂ x λ , λ ) := ϵ θ ( x ( t λ ( λ )) , t λ ( λ )) . We consider a time discretization { t i } M +1 i =0 going backwards in time starting from t 0 = T to t M +1 = 0 and to ease the notation we will always

denote t < s for two consecutive time-steps t i < t i -1 . The analytic solution at time t of the RSDE (2) with coefficients (7) and initial value x s reads

Given an initial condition ˜ x t 0 = x T , the SEEDS-1 iterates read, for h i = λ t i -λ t i -1 = λ t -λ s ,

Assumption C.1.

The function ϵ θ ( x , t ) is continuous (and hence bounded) on [0 , T ] , Lipschitz with respect to x and there is a constant C such that, for t, s ∈ [0 , T ] with t < s , we have

Let C l P ( R d , R ) denote the family of l times continuously differentiable real-valued functions on R d whose partial derivatives of order ⩽ l have polynomial growth and let C k,l P ( I × R d , R ) be the space of functions g ( · , · ) such that, for all ( t, x ) ∈ I × R d , g ( · , x ) ∈ C k ( I, R ) and g ( t, · ) ∈ C l P ( R d , R ) .

Assumption C.2. In addition to Assumption C.1, assume that all the components of ϵ θ belong to C 4 , 2 P ( R d × [0 , 1] , R ) .

Before going into the proofs, we give some context that lead us to necessitate such assumptions.

C.1 Preliminaries

For an interval I = [ t 0 , T ] , let x = ( x ( t )) I the solution of the following SDE

where g ( t ) = ˆ g ( t ) · Id d is considered here as a diagonal matrix with identical diagonal entries ˆ g ( t ) . Suppose that f, g are continuous, and satisfy a linear growth and Lipschitz condition so that the conditions of the Existence and Uniqueness Theorem are fulfilled for the SDE (47).

Let I h = { t 0 , . . . , t M } be a time discretization of I with step sizes h n = t n +1 -t n for n = 0 , . . . , M -1 and let h = max 0 ⩽ n<M h n . A time discrete approximation scheme ̂ x = ( ̂ x n ) I h will be defined as a sequence ̂ x 0 = x ( t 0 ) and

where I 0 is independent of ̂ x 0 , with I n = ω ( t n +1 -t n ) Wiener increments drawn from the normal distributions with zero mean and variance h n and which are independent of ̂ x 0 , . . . , ̂ x n and I 0 , . . . , I n -1 .

A scheme ̂ x converges in the strong (resp. weak) sense, with global order p > 0 , to the solution x of the SDE (47) if there is a constant C > 0 , independent of h and δ > 0 , such that, for each h ∈ ]0 , δ ] , we have

Notice that if ( E [ | x ( T ) -̂ x M | 2 ]) 1 / 2 = O ( h p ) then for every function f satisfying a Lipschitz condition, we have | E [ f ( x ( T ))] -E [ f ( ̂ x M )] | = O ( h p ) . Nevertheless, this is not enough to infer the optimal weak order of convergence of such method.

Strong convergence is concerned with the precision of the path, while the weak convergence is with the precision of the moments. As, for DPMs, the center of attention is the evolution of the probability densities rather than that of the noising process of single data samples, weak convergence is enough to guarantee the well-conditioning of our numerical schemes. Moreover, when the diffusion coefficient vanishes, then both strong and weak convergence (with the choice G = id ) reduce to the usual deterministic convergence criterion for ODEs.

Let us now state some useful results that will be used later on.

Assumption C.3. All moments of the initial value ̂ x 0 exist, f is continuous, satisfy a linear growth and globally Lipschitz condition.

In particular, since I is a closed finite interval in R , f ( · , x ) and g are bounded by some constant.

Theorem C.4 ([25]) . In addition to C.3, suppose that

with p 2 ⩾ 1 / 2 and p 1 ⩾ p 2 +1 / 2 . Then ̂ x is of strong global order p = p 2 -1 / 2 .

Assumption C.5. All moments of the initial value ̂ x 0 exist, f is continuous, satisfy a linear growth and Lipschitz condition with all their components belonging to C p +1 , 2( p +1) P ( I × R d , R ) and g ∈ C p +1 ( I, R ) .

Theorem C.6 ([25]) . In addition to C.5, suppose that

for large enough r ∈ N , the moments E [ | ̂ x n | 2 r ] exist and are uniformingly bounded with respect to M and n = 0 , . . . , M

for all G ∈ C 2( p +1) P ( R d , R ) , if ̂ x n = x ( t n ) , then for some K ∈ C 0 P ( R d , R ) , we have | E [ G ( x ( t n +1 ))] -E [ G ( ̂ x n +1 )] | ≤ K ( ̂ x n ) h p +1 .

Then ̂ x is of weak global order p .

Lemma C.7. Suppose that ̂ x 0 has moments of all orders and that, for h < 1 ,

where X n has moments of all orders. Then Condition 1 in Theorem C.6 is fulfilled.

C.2 Convergence of SEEDS-1

In this section we will prove that that SEEDS-1 as described above is of global strong order 1.0.

C.2.1 Strong Itô-Taylor approximation

Then the truncated Itô-Taylor expansion of the analytic solution x t of the VP NP NRSDE starting from x s is given, for ϵ ∼ N (0 , 1) , by

where the symbol O ( h p ) represent terms u such that ∥ u ∥ ⩽ | K ( x s ) | h p , for K ∈ C 0 P ( R d , R ) and small h > 0 . The SEEDS-1 scheme corresponds to such truncated Itô-Taylor expansion containing only the time and Wiener integrals of multiplicity one. As such, since G t g ( t ) = 0 and assuming Lipschitz and linear growth conditions on ϵ θ as in Assumption C.1, x t can be interpreted as an order 1.0 strong Itô-Taylor approximation [17, Theorem 10.6.3] of the solution to (42).

C.2.2 Continuous approximation of SEEDS-1

Let ˜ α t := 1 2 β d t 2 + β m t , where β d = β max -β min = 19 . 9 , β m = β min = 0 . 1 . We have

so, in particular, as T = 1 , we have ˜ α t 0 = ˜ α 1 = 1 2 ( β max -β min )+ β min = 1 2 ( β max + β min ) ≈ 10 . 05 and ˜ α t M +1 = ˜ α 0 = 0 . We deduce α t 0 = α 1 ≈ e -10 . 05 2 < 1 , α t M +1 = α 0 = 1 . Next, σ t 0 = σ 1 ≈ √ e 10 . 05 -1 > 1 and σ t M +1 = σ 0 = 0 . As such, λ t 0 = -log( σ 1 ) := -L 0 < 0 and λ t -→ t → t M +1 + ∞ .

As such, we will set t M = t M +1 + ε the end time so that λ t M = L 0 is finite. This implies that, for λ ∈ [ -L 0 , + ∞ [ , 0 < e -L 0 ⩽ e λ < 1 < e λ t M and, for ˆ T = λ t M -λ t 0 , e h = e λ t -λ s ⩽ e ˆ T . Now set, for λ ∈ [ -L 0 , + ∞ [ ,

Then, as λ increases, ˆ α λ increases starting from 0 < ˆ α λ t 0 < 1 and ˆ α λ -→ λ → + ∞ 1 while at the same time ˆ σ λ decreases starting from 0 < ˆ σ λ t 0 < 1 and ˆ σ λ -→ λ → + ∞ 0 . As such, we can rewrite the exact solution (43) as

Notice that ˆ α λt ˆ α λs is bounded for all t, s by and the continuous approximation

Proposition C.8. There are two constants C 1 , C 2 independent of h such that, for all t ∈ [0 , T ] , we have

and 0 < ˆ α λ < 1 for all λ ∈ [ -L 0 , + ∞ [ .

Recall that the SEEDS-1 is defined recursively as

and, for simplicity, we will denote y λ t i for the iterates (48).

Define a continuous approximation of SEEDS-1 as follows. For ˆ h = t -s , we write ˆ s = [ s/ ˆ h ] ˆ h where [ x ] denotes the largest integer lesser or equal to x and I [ A ] is the indicator function associated to a set A . We define the step function:

Proof. Recall the standard inequality ( a + b + c ) 2 ⩽ 3( a 2 + b 2 + c 2 ) for a, b, c ∈ R . Then

Using the fact that α t ⩽ 1 , we have, by writing ˆ T = λ t M -λ t 0 and taking the expectation:

where we used the linearity of expectation, Hölder's inequality, Doob's martingale inequality, Itô isometry, the linear growth condition of ˆ ϵ θ , and we set K = max ( ∣ ∣ ∣ α t α t 0 ∣ ∣ ∣ 2 , | e L 0 | 2 , ¯ σ 2 t , 1 ) . As we know that E [ | x ( t 0 ) | 2 ] < ∞ , we apply Grönwall's inequality in the last line to obtain

Second, we have, for s = t i , u = t i +1 and t ∈ [ t λ ( u ) , t λ ( s )[ ,

so that, using Hölder's inequality, we get

Now using Itô isometry we obtain:

Now, using the bound E [ max | y t | 2 ] ⩽ C 1 , the fact that ϵ θ is bounded and the same arguments as above, we get

Finally, as we have and 1 < √ 1 -α 2 s √ 1 -α 2 t -→ M →∞ 1 , we obtain ∣ ∣ ∣ α t α s -1 ∣ ∣ ∣ 2 ∼ | e h -1 | 2 ∼ O ( h 2 ) . Now, by denoting | e h -1 | 2 ⩽ 2

K 2 h , we conclude that with C 2 := 3 K 2 C 1 +4 KL 2 (1 + C 1 ) . This finishes the proof.

C.2.3 Proof of Theorem 4.1

Let's now take a look at the approximation given y t 0 = x t 0 . We have

Using the inequality α t ≤ 1 , the Lipschitz property of ̂ ϵ θ , Assumption (C.1), and Hölder's inequality we deduce the bound:

Taking the expectation yields

Now, for the first integral, by writing:

we can state the following inequalities:

Thus, we obtain

Putting everything together yields

Now, by denoting | e h -1 | 2 ⩽ K 2 h 2 , we apply the continuous version of the Grönwall Lemma to obtain:

where

Finally, using Lyapunov's inequality we obtain, for C = √ C 0

In other words, for C as stated above, we have the following inequality

Remark C.9 . From the above, it is easy to induce that the following order for the one step error

Now, since G t g ( t ) = 0 in this case (additive noise), it is easy to see from the truncated Itô-Taylor expansion of x that | E [ x ( t 1 ) -y 1 ] | = O ( h 2 ) . As such, we apply Theorem C.4 to conclude that SEEDS-1 has strong convergence of global order 1.0.

C.2.4 Discrete-time approximation

By Theorem 4.1, we know that SEEDS-1, being of strong order 1.0, it is immediately also of weak order 1. Nonetheless, let's give a discrete approach of this statement that we will use for the proofs of convergence for the remaining solvers as stated in Corollary 4.4. Define the discrete time process:

We will prove that E [ | y t M -x t M | ] is of order h as h -→ 0 . Note that ( y t i ) i has the same distribution as ( ˜ x t i ) i described in Algorithm 2 since the stochastic integrals ( ∫ λ t i λ t i -1 e -s d ¯ ω ( s ) ) i are independent and each ∫ λ t i λ t i -1 e -s d ¯ ω ( s ) is distributed as 1 √ 2 σ t i α t i √ e 2 h i -1 ϵ with h i = λ t i -λ t i -1 , ϵ ∼ N ( 0 , I d ) . We have:

For simplicity, in what follows C will denote a constant not dependent on the subdivision of [0 , T ] that may change from one line to the next by systematically denoting the maximum value of the

different constants appearing in the line before. Using the inequality α t ≤ 1 , the Lipschitz property of ϵ , we deduce the bound:

Notice that

and recall that ̂ x u = x t λ ( u ) . Using the fact that t λ is increasing and Lemma C.11, we have:

Introduce U i = E [ | y t i -x t i | ] . Since ∫ λ t i λ t i -1 e -λ | t λ ( λ ) -t i -1 | d λ ≤ Ch 2 :

Let a i = α t i α t i -1 + Ch and b i = Ch 2 . By applying Lemma C.12, we have: U M ≤ A M U 0 + ∑ M k =1 A k,n b k with A M = ∏ M k =1 a k and A k,M = A M /A k = ∏ M i = k +1 a i . Note that U 0 = 0 since y t 0 = x T and so:

Using our hypothesis, we can bound:

The quantity on the right is of order C/h . Indeed, as h goes to 0 , exp( Ch ) + Ch -1 is equivalent to 2 Ch and (exp( Ch ) + Ch ) 1 /h converges to a constant. This gives the bound U M ≤ Ch , when h → 0 and by using Proposition C.8 and Theorem C.6 we conclude that SEEDS-1 is convergent of weak order 1.

C.2.5 Useful Lemmas

Lemma C.10. (Continuous Grönwall Lemma) Let I = [ a, b ] denote a compact interval of the real line with a < b . Let α, β, u be continuous real-valued functions defined on I . Assume β is non-negative, α is non-decreasing and u satisfies the integral inequality

Then

Lemma C.11. Assume the following forward SDE is satisfied dX t = F ( t, X t ) dt + G ( t ) dW t , t ∈ [0 , T ] , where T > 0 , F ( t, x ) is Lipschitz with respect to ( t, x ) , G is continuous and X 0 is an integrable random variable. Then, there exists C > 0 such that for all s < t ∈ [0 , T ] with t -s ≤ 1 ,

Proof. We take s = 0 and apply the triangular inequality:

Setting u ( t ) = E [ | X t -X 0 | ] and taking the expectation, we deduce:

where K is a positive constant. Note that E [( ∫ t 0 G ( u ) dW u ) 2 ] 1 2 = [ ∫ t 0 G 2 ( s ) ds ] 1 2 by the Itô isometry property which is less than C √ t since G is continuous. Thus, we have proved that:

where α ( t ) = K t 2 2 + t E [ | F (0 , X 0 ) | ] + C √ t is non-decreasing. By Lemma C.10: u ( t ) ≤ α ( t ) exp( Kt ) ≤ α ( t ) exp( KT ) . Since α ( t ) ≤ C √ t for t ∈ [0 , 1] , the lemma holds.

Lemma C.12. (Discrete Grönwall Lemma) Consider a real number sequence ( u n ) n such that

where ( a n ) and ( b n ) n are two given sequences such that b n is positive. Then

where A n = ∏ n k =1 a k and A k,n = A n /A k = ∏ n i = k +1 a i .

C.3 Proof of Proposition 4.3

Let us prove this statement for SEEDS-2. On the one hand, we have

Now B 1 depends on the Brownian movement ( ω λ s 1 -ω λ s ) λ s 1 ⩾ λ s . By the Markov property, this is independent of ( ω λ u ) λ u ⩽ λ s . Since ˜ x s is a function of ( ω λ u ) λ u ⩽ λ s , we deduce that B 1 has to be independent of all (˜ x u ) u ⩾ s . By Itô Isometry, we obtain

On the other hand, the update ˜ x t is

Hence, the Wiener process W = { ω λ : λ ∈ [ λ s , λ t ] } is predetermined on the interval [ λ s , λ s 1 ] . Then, by using independent increments property of Wiener process W , we can deduce, for 0 ≤ λ s < λ s 1 < λ t , that

Then, by the above Brownian independence property, the random variables B 0 and B 1 are

Independent on non-overlapping time intervals

Dependent on overlapping intervals.

By the Chasles rule, we then decompose and, as

we obtain

Then we have

which completes the proof.

The proof for SEEDS-3 is straightforward from the proof for SEEDS-2.

C.4 Proof of Corollary 4.4

C.4.1 Convergence of SEEDS-2:

Let { y t i } i be the discrete stochastic process defined as follows:

Then { y t i } i has the same distribution as { ˜ x t i } i in Algorithm 3. We can compute the difference:

and:

Similarly to the case k = 1 , E [ | Γ 2 | ] ≤ Ch 2 . Note that E [ | Γ 1 | ] ≤ Ch ∆ i with ∆ i = E [ | u i -x s i | ] . Introduce: U i = E [ | y t i -x t i | ] . We will bound ∆ i by a function of U i -1 . For this, recall:

and write the difference:

By the triangular inequality:

and so: E [ | ̂ ϵ ( ̂ x u , u ) -ϵ ( y t i -1 , λ t i -1 ) | ] ≤ C ( √ h + U i -1 + h ) . Finally:

and E [ | Γ 1 | ] ≤ Ch 5 / 2 + C ( h + h 2 ) U i -1 . This gives the bound:

Now using Theorem C.6, the proof can now be finished following as in proof in Section C.2.4.

C.4.2 Convergence of SEEDS-3:

Continuing with the same notations as before, we will prove by analogy that:

so that, by Theorem C.6, we obtain the desired result.

Let y t i = ˜ x t i . We have:

and x t i = α t i α t i -1 x t i -1 -2 α t i ∫ λ t i λ t i -1 e -λ ̂ ϵ ( ̂ x λ , λ )d λ + Noise, where Noise is the same in both equations. From the proof for k = 1 , it suffices to bound:

in the L 1 norm. By the Lipschitz property of ϵ :

Now, let us bound E [ | u 2 i -x s 2 i | ] and for this, write

and

where Noise is the same in both equations. So

where D = ϵ ( u 2 i -1 ,s 2 i -1 ) -ϵ ( y t i -1 ,t i -1 ) r 1 h i . From the convergence proof of SEEDS-1, and the Lipschitz property of ϵ , we obtain:

Again, by the triangular inequality:

and since

we have, as before,

Combining the previous inequalities leads to (49). This finishes the proof of Corollary 4.4.

D Implementation Details

The SEEDS solvers used in our experiments are exactly the variants we described in Algorithms 2, 3, 4 in the main part of the paper. In particular, SEEDS-2 and SEEDS-3 solvers are completely determined by one-parameter families of deterministic exponential integrators from [14] of order 2 and 3 respectively and prescribed by the following Butcher tableaux:

In all experiments we fix the parameter c 2 = 0 . 5 for SEEDS-2 and c 2 = 1 3 for SEEDS-3. We point out that the Butcher tableau here associated to SEEDS-3 is the result of a weakening of the stiff order conditions so it might suffer from order reduction in the deterministic case. We also point out that a full theory of stiff order conditions for stochastic exponential Runge-Kutta methods for semi-linear DEs with a time-varying linear coefficient have not been yet developed to the authors knowledge. As such, this is only an analogy whose purpose is to clarify how our solvers relate to well-known solvers from the literature, but such Butcher tableaux do not rigorously reflect their convergence order.

D.1 Stabilization of the exponential terms

In the proposed algorithms, one subtle detail is to re-arrange all equations in order for them to make use only of the expm1 ( h ) function which computes e h -1 with great numerical stability, specially

for small values of h . We have for values r 1 = 1 / 3 , r 2 = 2 / 3 , the following identity

Now by using we now compute

Finally, we obtain

D.2 Noise schedules parameterizations

The inverses of λ ( t ) are given in the (VP) linear and cosine schedules respectively by

In EDM Noise Prediction case, the inverses of λ ( t ) , as given in (37), with respect to the NPFO and NRSDE are respectively given by

D.3 EDMdiscretization

We follow Karras et al. [16] to implement the EDM discretization timesteps { t i } i M =0 as t i = σ -1 ( σ i ) such that, for ρ > 0 ,

From the definition, we note that σ 0 = σ max and σ M -1 = σ min, where σ min and σ max denote the minimum and maximum noise magnitude, respectively. We also keep default value ρ = 7 as in [16]. However, we figured out that when using EDM discretization with linear schedule, the noise schedule improvement in iDDPM pre-trained models [26] would result in two consecutive time steps of the same value, i.e. t j = t j +1 for some index j = 1 , . . . , M -1 and large steps ( M ≥ 61 ). Thus, for

SEEDS-3 and DPM-Solver-3 [22], the usage of the function φ 2 ( h ) = e h -h -1 h 2 (see Appendix E for more details) will cause zero division error where h = λ t j +1 -λ t j = 0 . Therefore, in our implementation, we ignore the noise schedule improvement of iDDPM [26] models when using solvers of order three.

D.4 Final sampling step

The sampling phase in DPMs using SEEDS follows the RSDE, which requires gradual computing through discretization time steps { t i } i M =0 and the latter goes from t 0 = T to t M = 0 . In our implementation, to avoid the logarithm of zero error at the last step, i.e. log σ t M = log(0) , we stop the sampling phase at step ( M -1) . Hence, the NFE used in a run will be given as

where k represents the order of the solver. We also do not use the 'denoising' trick, i.e., ignoring the random noise at the last step and leave it to further research.

E Reminders on Stochastic Exponential Integrators

Let us consider a SDE of the form

where a, c : [0 , T ] → R and g : [0 , T ] → R d × d . In other words we concentrate to high-dimensional semi-linear non autonomous SDEs with additive noise. The objective of this section is to construct explicit stochastic exponential derivative-free methods for the above equation following the RungeKutta (RK) approach. These methods ideally should fulfill the following properties:

If f ≡ 0 , then (50) can be solved exactly ;

If g ≡ 0 , then a SEEDS method for (50) identifies with an exponential RK (ERK) method;

If a ≡ 0 , then a SEEDS method for (50) identifies with a stochastic RK (SRK) method and if moreover g ≡ 0 then it identifies with a classical RK ODE method.

Before tackling the aimed SEEDS problem let us rapidly recall elementary constructions of RK, ERK, weak and strong SRK methods. We will not deal with time-adaptive methods here.

E.1 Derivative-free exponential ODE schemes

E.1.1 Runge-Kutta approach

Derivative-free schemes are obtained by comparing the Itô-Taylor expansion of the above paragraph with expressions of x ( t ) in terms of its intermediate evaluations between s and t = s + h and the Taylor expansions of such evaluations. As a simple example, in the ODE regime

As such, analytic solutions to the above equations are of the form

Now, Taylor expansion gives, up to order 2:

A straightforward recursion yields a Taylor expansion

where we denote the generalized infinitesimal operator of the solution x of (51) by

Derivative-free methods seek to get rid of the derivatives in L t by computing Taylor expansions of f at well-chosen points. In the explicit one-step case, this amounts of defining

where the Φ function does not contain derivatives of f . The general high order case is given by well-tuned coefficients in the following scheme

By denoting α = [ α 1 . . . α n ] , c = [ c 1 . . . c n ] T and A = ( a i,j ) , these can be represented by a Butcher tableau of the form

In the following sections we will present extended version of this tableau for representing more involved numerical schemes. As such, the Euler, midpoint, Heun and general second order explicit methods are respectively:

E.1.2 Exponential Runge-Kutta approach

We now concentrate on a non-autonomous semi-linear ODE of the form

where f ( t ) = ¯ f ( t ) · Id d and g ( x ( t ) , t ) = ¯ g ( x ( t ) , t ) · Id d are diagonal time-dependent matrices with identical diagonal coefficients. In particular, we have [ f ( k ) ( t ) , f ( l ) ( s )] = 0 , a property we will need to facilitate exponential matrix multiplications. As f ( s ) is a d -dimensional diagonal matrix, the fundamental matrix for (52), usually given by the Peano-Baker series, simplifies in this case to the form

which satisfies Φ ′ ( t ; s ) = f ( t )Φ( t ; s ) and Φ( s ; s ) = 1 . Then the exact solution for (52) is given, via the variation of constants formula, by

In light of this integral form, one can formalize a general class of exponential n -stage RK methods

where γ 0 , γ i , a ij , b j are functions of the step-size h , f and f s ( r ) := ∫ r 0 f ( s + τ )d τ . There are two possible approaches to create exponential integrators, namely the exponential time-differencing (ETD) approach that uses the variation of constants formula and makes use of the φ functions, and the Lawson approach, also know as integrating factor (IF), which makes a change of variables on the above SDE thus avoiding the use of the φ functions but computing exponential factors step-wise.

The Lawson and the ETD approaches for Exponential Euler schemes

There are two choices one can make when computing first order approximations of

First, by interpolating g ( x ( τ ) , τ ) as g ( x ( s ) , s ) we obtain

Now two choices remain. Either ∫ t s Φ( t ; τ )d τ is computed exactly or we again interpolate Φ( t ; τ ) as Φ( t ; s ) . Taking for simplicity f ≡ A to be constant and by denoting h = t -s , the first case yields the ETD Euler method

and the second yields the IF Euler (also called Lawson-Euler) method

Now, consider a solution

Exponential methods then aim to approximate the term g ( x ( s + τ ) , s + τ ) by its interpolation polynomial in certain non-confluent quadrature nodes c 1 , . . . , c n .

The ETD approach

In this case, the variation of constants formula yields

Now, as before, the Taylor expansion of f yields

Recall that the φ functions are given in an integral form as follows

which satisfy φ k (0) = 1 k ! . Now denote

The exact solution at s + ch now reads

Using the left endpoint rule yields

Second order examples:

The condition b 2 ( z ) c 2 = φ 2 ( z ) implies b 2 = φ 2 ( z ) /c 2 and we obtain

A second method is obtained by weakening the above condition to b 2 (0) c 2 = φ 2 (0) = 1 / 2 giving

In some cases this method can suffer from order reduction and not reach order 2 of convergence. Moreover, setting c = 1 / 2 gives the exponential midpoint method:

The above order 1 and 2 exponential methods are represented by the following exponential Butcher tableaux

We check that when formally setting A = 0 then the first two methods are identical and give the generic 2nd order RK method, the choice c = 1 gives the Heun method and the choice c = 1 / 2 gives the midpoint method.

Fourth order methods

It can be shown that ERK methods need at least 5 stages to achieve order 4. By setting formally A = 0 these methods do not have a non-exponential counterpart to our knowledge.

5-stage sequential We have a fourth-order ERK scheme given by

with

This can also be represented by Rosenbrock-like Butcher tableau

with

This traduces into (recall that D i = f ( x i , s + c i h ) -f ( x ( s ) , s ) )

Inspired by this, we deduce the following fourth order Algorithm 7 specialized for DPMs in the VP noise prediction regime.

E.2 Derivative-free exponential SDE schemes

Let x t be the path at the continuous limit h → 0 , and { ̂ x t } t M t 0 be the discretized numerical path, computed by a numerical scheme S with M = 1 /h steps of length h > 0 . Then, S has

1.

weak order of convergence β if there is K > 0 and a function class K such that

Strong convergence is concerned with the precision of the path, while the weak convergence is with the precision of the moments. As, for diffusion models, the center of attention is the evolution of the probability densities rather than that of the noising process of single data samples, weak convergence is enough to guarantee the well-conditioning of our numerical schemes. Moreover, when the diffusion coefficient vanishes, then both strong and weak convergence (with the choice ϕ = id ) reduce to the usual deterministic convergence criterion for ODEs.

Algorithm 7 DPM-Solver-4

1: def DPM-Solver-4 ( ϵ θ , ˜ x t i -1 , t i -1 , t i , r = 0 . 5) :

2: h i ← λ t i -λ t i -1

5: k 1 ← ϵ θ (˜ x t i -1 , t i -1 )

15: ˜ x t i ← α t α t i -1 ˜ x t i -1 -D -E

16: Return x t i

E.2.1 Strong and Weak Stochastic Runge-Kutta approach

In all what follows ω is considered a d -dimensional Wiener process (with identity diffusion matrix). Consider the following SDE

where g ( t ) = ˆ g ( t ) · Id d is considered here as a diagonal matrix with identical diagonal entries ˆ g ( t ) . Given an initial value independent of ω , the integral form of x ( t ) is given by

The idea underlying stochastic numerical schemes is very similar to the one used in the deterministic approach, that is to take expansions of the terms inside the integrals based at the integral's initial value and replace the obtained derivatives that appear by interpolated approximations. A key difference here is that as we have to consider Itô-Taylor expansions, the infinitesimal operators are different but most importantly most of the stochastic iterated integrals will need to be approximated in an appropriate sense, whenever that is possible. We will develop this expansion up to triple integrals.

E.2.2 Truncated Itô-Taylor expansions

Applying Itô formula to h = f or g yields

This allows us to define two differential operators L, G as

In particular L t g ( t ) = ∂ t g ( t ) and G t g ( t ) = 0 . With this notation, we have

so our solution reads

Now G t g ( τ 2 ) = 0 and

So we have

and now our solution reads

where

Now we have

and

Now, if we denote d ω 0 ( τ ) = d τ , d ω 1 ( τ ) = d ω ( τ ) and

applying the same procedure to R 1 leads to

and with R 2 consisting on quadruple integrals. As such, our solution now reads

Now, in the SDE regime, one cannot get rid of the iterated Itô integral and so stochastic RK methods cannot be derived as simple extensions of their deterministic counterparts. In order to continue we now take into account the fact that the diffusion SDE has additive and diagonal noise. In this case, both the Itô and the Stratonovich SDE coincide.

Iterated integrals

Now, for simplicity, set t = 0 . We then have I (0) = h , I (0 , 0) = ∫ h 0 ∫ τ 1 0 d τ 2 d τ 1 = h 2 2 , I (0 , 0 , 0) = h 3 6 . Now notice that

Additionally, ˆ w h and ˆ z h satisfy E [ ˆ w 2 h ] = h ,

E.2.3 Integral approximations

Weak Approximations

When crafting weak stochastic approximations to SDEs one may replace multiple Itô integrals by other random variables satisfying the corresponding moment conditions. We will denote ˆ I α the approximation of I α for α a multi-index following [17, Corollary 5.12.1]. Of course, the deterministic integrals I (0 ,..., 0) need not to be approximated.

First order approximations

The random variable ˆ I (1) must satisfy for some constant K :

Two possible choices for ˆ I (1) are either ˆ I (1) ∼ N (0 , h ) or ˆ I (1) is a two-pointed distributed discrete random variable with

Second order approximations

The random variable ˆ I (1) must satisfy for some constant K :

Two possible choices for ˆ I (1) are either ˆ I (1) ∼ N (0 , h ) or ˆ I (1) is a three-pointed distributed random variable with

The rest follows from the above calculations:

Third order approximations

One can choose ˆ I (1) ∼ N (0 , h ) , ˆ I (0 , 1) ∼ N ( 0 , 1 3 h 3 ) satisfying E [ ˆ I (1) ˆ I (0 , 1) ] = 1 2 h 2 .

Then, one can deduce the following:

Thus, we can write the solution weak approximation as

An example of such a pair ( ˆ I (1) , ˆ I (0 , 1) ) = ( ˆ w h , ˆ z h ) can be easily obtained as follows

Indeed, for such a pair we have

In light of the above expression, the truncated Taylor expansions we refer in the main part of the paper consists on the consideration of only the coefficients in L k t . The only noise noise contribution we will consider corresponds to g ( s ) ˆ I (1) .

F Experiment Details

We evaluate the Fréchet inception distance (FID) after generating 50K samples with each solver, and compare with the statistics of real-data. In our experiments we make use of the code from [16] for continuously trained models as well as their reference FID stats 2 and that of [22] for discretely trained models.

All the experiments of SEEDS for continuous-time models are parameterized within the EDM framework with the discretization of type EDM, linear schedule, and scaling none as described on [16] in noise prediction mode unless explicitly stated. We use the SEEDS-3 method that has 3 NFEs per step and fixed step-size and report FID scores at NFEs divisible by 3.

We leverage the explicit Langevin-like 'churn' trick using in [16] to add or remove noise in the sampling phase. Specifically, [16] uses 4 hyper-parameters S churn , S min , S max and S noise in which S churn controls the overall amount of stochasticity added before giving the input to the SEEDS-3 method when the noise level (or time step in EDM configuration) t i is contained in the noise interval [ S min , S max ] . It means that the EDM proposed sampler is stochastic under some conditions of those hyper-parameters and deterministic otherwise, while our method is completely stochastic. In our experiments, we set S churn = 0 except for ImageNet-64 EDM optimized model. We noticed that using the additional stochasticity indeed helps to improve the image quality as in Fig. 1 (c). Moreover, setting S noise slightly above 1 might correct the errors in earlier steps more effectively as indicated in [16].

F.1 Pre-trained model specifications

For producing the CIFAR-10 time-continuous results in Table 1, we use the VP DDPM++ continuous architecture. These models are publicly available in conditional 3 and unconditional 4 versions and were directly derived from [34] under the Apache 2.0 license. On the unconditional mode (Figure 1 (a-b)), we first generate the FID curves of 3 types of DPM-Solver (with orders 1, 2 and 3) using the updates from their official implementation 5 in noise prediction mode. Taking profit of the tuning advancements proposed by [16], we used a linear noise schedule with β d = 19 . 1 and β min = 0 . 1 that slightly differs from the original parameters from [34] but were proven beneficial. We set the end time of sampling ε to 1 e4 as recommended by [22, Appendix D.2]. The values of all benchmark models for Figure 1 (b) were taken directly from tables provided by [22].

In our FFHQ-64 experiments, we employ the unconditional VP pretrained 6 model provided by [16].

For the CelebA-64 experiments, we use the pre-trained VP unconditional model whose checkpoint 7 is provided by [33]. We use the Type-1 discretization proposed in [22] to ensure compatibility of our method with the prescribed trained steps of such model.

For ImageNet-64, we both use the baseline and the optimized pre-trained models given in [16]. We note that the baseline is trained on the iDDPM class of model [26], which actually uses different preconditioning and thus the change of variables compared to the optimized model. The Figure 1 (c) was obtained using the EDM-preconditioned checkpoint 8 . The added noise settings of SEEDS-3 solver were not subject to a grid-search optimization procedure. The chosen hyper-parameters were S churn = 11 , S noise = 1 . 003 , S min = 0 . 05 , and S max = 15 but we are confident that this configuration can be optimized to further improve our results.

F.2 Noise vs. Data Prediction approaches

In Appendix B of [23], the authors compare DPM-Solver2 and DPM-Solver++(2S), which amounts on comparing in our framework the difference between the obtained exponential integrators for the PFO in the noise and data prediction regimes to detect exactly a coefficient on the non-linear term that is absent in the noise prediction regime. The term they find corresponds exactly to the difference between applying the variation of constants formula before (instead of after) replacing the score function with the desired neural network. In Tab. 3 we report both data and noise prediction SEEDS-3. At low NFEs the DP approach gives better results but stabilizes in high NFEs at a FID score that is worse than the one the NP approach reaches.

F.3 Low vs. High stage Solvers

Similar to DPM-Solver [22], the FID scores in Tab. 4 and 5 show that at low NFEs, higher stage methods performs more poorly while at higher NFEs, DPM-Solver-3 and SEEDS-3 are better than their counterparts with 1 and 2 stages.

F.4 Deterministic vs. Stochastic Solvers

Deterministic solvers as DPM-Solver [22] are fast and well-adapted to applications in which speed is the most concern. As shown in [22], Table 4 and 5, DPM-Solver converges to a local minimum at early steps and cannot be improved in large NFEs. Moreover, preconditioned deterministic solver in EDM gives optimal quality on unconditional CIFAR-10 [16, Fig. 5 (b)]. However, for more complicated datasets as ImageNet64, the stochasticity indeed helps improve the samples quality [16, Fig. 5 (c)]. We can consider the random noise as a corrector that approaches a better local or even global minimum. At S churn = 0 , our SEEDS-3 with stochasticity gives lower FID score than EDM deterministic Heun (see Fig. 1 (c)). SEEDS-3 also reaches the best quality prior to the number steps needed in Euler Maruyama and other solvers as in Table 6. Table 7 completes the analysis in Table 1 for CIFAR-10 showing FID scores for varying NFEs and multiple versions of the EDM solver, depending on the chosen optimization hyper-parameters. Tables 8 to 10 complete the findings in Table 1 for the remaining used pretrained models. In particular, we report the exact FID values of SEEDS in the low NFE regime.

F.5 Run-time Comparison

As a sanity check, we provide a run-time comparison table for our experiments on different datasets (see Tab. 11). One can see that the run-time is linear with respect to the NFE also for SEEDS, as the main advantage of the SETD method is to analytically compute the stochastic components in our solver, making their computation cost negligible. Some mild improvements about repetitive terms allowed our implementation of SEEDS to be slightly more effective than EDM and even DPM-Solver at same NFE.

F.6 Hardware configuration

During the experiments, we used three Linux-based servers with 60GB memory each and 4 GPUs NVIDIA V100 32GB, 4 GPUs NVIDIA V100 16GB, and 2 GPUs NVIDIA V100 32GB, respectively. Table 12 shows the detail of the configuration utilized for each experiment. We noted that when using the 4 GPUs configuration, the FID results were slightly lower (around 2%), even after using a stacked fixed random seed. We run the experiments multiple times and reported the minimum FID value each time.

F.7 Licences

Pre-trained models:

CIFAR-10 models by [34]: Apache V2.0 license

FFHQ-64 model by [16]: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

CelebA-64 model by [33]: Apache V2.0 license

ImageNet-64 model by [16]: Apache V2.0 license

Inception-v3 model by [35]: Apache V2.0 license

F.8 Supporting samples

In this subsection, we report the image grids supporting our claims in section 5.

NP

DP

NFE=9

NFE=21

NFE=30

NFE=90

NFE=129

NP

DP

NFE=9

NFE=21

NFE=30

NFE=90

NFE=129

SEEDS-3

DPM-Solver-3

NFE=9

NFE=21

NFE=30

NFE=60

NFE=90

NFE=150

NP

DP

NFE=9

NFE=21

NFE=30

NFE=60

NFE=90

NFE=150

NP

DP

NFE=30

NFE=60

NFE=90

NFE=150

NFE=210

NFE=270

NP

DP

NFE=30

NFE=60

NFE=90

NFE=150

NFE=210

NFE=270

57

(b) SEEDS-3 at 201 NFE.

(b) EDM at 80 NFE.

(c) SEEDS-3 at 150 NFE.

(b) EDM at 90 NFE.

(c) SEEDS-3 at 150 NFE.

私

Peer reviews:
comments: SUMMARY:
There are powerful ODE solvers to speed up the sampling process of diffusion models. Despite being quick, ODE solvers do not usually reach the optimal quality achieved by SDE solvers which are however slow. To tackle this problem, the paper proposes SEEDS, which is based on Exponential Integrator in the stochastic case. SEEDS are derivative and training-free solvers with proven convergence order.

STRENGTHS:
1. The authors provide exact solutions for diffusion SDEs and analytical computation of their variance.

2. The paper contains rigorous theoretical derivation and convergence order guarantee for the proposed methods. It also points out its connection with gDDIM.

3. The paper is overall well-written and self-contained.

4. The method achieves outstanding sampling performance.

WEAKNESSES:
The proposed SEEDS method still needs around a hundred NFEs to achieve good sampling qualities, the quality will degrade fast when NFEs decrease, which limits its applications.

QUESTIONS:
Would you please include the connection between the proposed method SEEDS-1/2/3 and the stochastic Runge-Kutta method properly combined with Exponential Integrator? 

Also, the convergence order included is order 1 for SEEDS-1 in the mean-square sense and order 1 for SEEDS-2/3 in the weak sense, since the stochastic Runge-Kutta method can achieve higher convergence order even in the strong sense, the reviewer is concerned the convergence order provided is not optimal.

LIMITATIONS:
The author addressed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an off-the-shelf (i.e., no further training) few-step sampler for diffusion probabilistic models (DPMs). By isolating linear terms in the exact solution of diffusion SDEs and using a change-of-variable method, the proposed sampler, SEEDS, simplifies the integrals and can approximate the solution of stochastic equations. SEEDS can accelerate the sampling of DPMs without compromising the quality of the samples.\n\nSTRENGTHS:\n- (Quality & Clarity) The authors have successfully provided a theoretical derivation and error analysis of their proposed method, as well as details of the experiments. The paper is well-written and the reviewer had no difficulty comprehending its contents.\n\n- (Significance) Numerical experiments show that SEEDS produces competitive results compared to previous sampling methods in terms of quality, with fewer function evaluations, in several benchmarks, namely CIFAR-10, CelebA-64, and ImageNet-64.\n\nWEAKNESSES:\n- (Originality) The proposed method has shown promising performance. However, its theoretical background is based on a classical idea of the *variation-of-parameters* formula in the literature of differential equations. Although the authors claim that equation (5) is a novel representation of diffusion SDEs, this kind of solution form is very well-known in the SDE community even for general Hilbert spaces (e.g. see Curtain & Falb and Da Prato & Zabczyk). Therefore, it seems that justification is needed to claim that the contribution of this paper regarding the solution representation suggested by the author is novel. Furthermore, the derivation of exponentially weighted integral is quite similar to the DPM-solver (Lu et al.), except for utilizing Itō-Taylor expansion due to the diffusion part in the solution representation. The convergence result of the proposed algorithm may not be an incremental result. However, this paper's argument on the contribution has an excessive aspect, and it is quite difficult for reviewers to agree with the author's claim.\n\n[Curtain & Falb] Stochastic Differential Equations in Hilbert Space, *Journal of Differential Equations*, (1971).\n\n[Da Prato & Zabczyk] Stochastic Equations in Infinite Dimensions, (2014).\n\n[Lu et al.] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, *NeurIPS*, (2022).\n\nQUESTIONS:\nTo discuss the generation speed of the SEEDS-3 algorithm in Table 1, it is necessary to compare it quantitatively with the actual runtime, as well as with NFE. For example, in the case of the DPM-Solver, a comparison of DDIM with respect to runtime was made in Table 7 of Appendix E.7.\n\nLIMITATIONS:\nThe authors mentioned the limations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
The paper proposed an efficient method for solving diffusion SDEs with some convergence guarantee. Experiments are done to verify their claims.

STRENGTHS:
The experiments show SEEDS-3 achieves optimal results with minimal NFE on many data sets.

WEAKNESSES:
1. Authors claim they have found a novel representation of the exact solution of the SDE, page 3, line 106. However, it is just a simple technique that's commonly used when solving SDEs. For example, when solving OU process, you multiply some exponential multiplier like this. I don't find anything novel about this.
2. Essentially, in the solution, the first and last terms are treated using a multiplier and change of variables. The second is rewritten using Taylor expansion. The convergence theorem is sort of too straightforward, as the Taylor expansion of course approximates the original function.
3. The experiments only report the NFE taken when achieving optimal performance, which I am not sure is a commonly accepted metric of evaluation in this community. Please comment on this. I am especially open to modifying the score if this question is addressed sufficiently.

QUESTIONS:
See weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a sampling method that achieves 3 to 5 times faster computation by exploiting the semi-linearity in the time-reversal stochastic differential equation (SDE). It analytically computes the linear part and reaches optimal quality sampling. This sampling method demonstrates comparable results to existing SDE solvers on datasets like CIFAR10 or Celeba without requiring any additional training methods.\n\nSTRENGTHS:\nThe paper is well-written in a clear and readable manner. It is easy to understand the paper without much difficulty, and the derivation of formulas is well-guided, focusing on the essential parts. One of the strengths of the paper is that it proposes a method that achieves approximately 3 to 5 times faster computation while still reaching optimal performance, which sets it apart from other ODE-based methods. Additionally, the paper provides a thorough and clear analysis of convergence, which is commendable.\n\nWEAKNESSES:\nIn (5), the claim of a \"Novel representation of exact solutions of diffusion SDEs\" seems to be overclaimed as the equation is well-known in the form of the weak solution of the SDE. Therefore, it is difficult to consider this as a newly discovered novel representation. Moreover, while this integral representation leads to a practical sampling method, much of it is based on formulas used in the DPM-solver [1]. This suggests that the contribution of this work may be relatively weak. The main difference from the DPM-solver appears to be the inclusion of a stochastic term, but beyond that, significant distinctions are not apparent.\n\nThe claimed advantage of reducing the number of function evaluations (NFE) by 3 to 5 times seems less compelling when compared to ODE-based methods that can achieve sampling speeds more than 50 times faster and still attain optimal performance. For more persuasive results, it would be better if this method demonstrated similar NFE to ODE-based methods while reaching optimal performance.\n\nAdditionally, it is unclear whether this method addresses any meaningful issues beyond the problem of ODE-based methods not achieving optimal performance.\n\n[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, *NeurIPS*, (2022).\n\nQUESTIONS:\n- Is it possible to maintain fast sampling speeds for datasets with resolutions higher than 256, such as CelebA-HQ or ImageNet?\n- Apart from the ability to reach optimal performance, are there any other distinguishing features compared to the DPM-solver? For instance, does it generate a more diverse set of samples or exhibit stronger capabilities in tasks like imputation?\n\nLIMITATIONS:\nThe paper has effectively summarized its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-31"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to accelerate the SDE solvers through exponential integrators. The authors compute the linear part of the SDE analytically, employ a more accurate interaction of the stochastic part. Experimentally, the proposed method improves over the previous ODE solvers.\n\nSTRENGTHS:\n- The paper proposes to utilize the exponential integrator to solve the diffusion SDEs (Eq. (5)). In addition, it approximates the integral components through Taylor expansion.\n\n- The paper utilizes the stochastic exponential time differencing method to compute the variance of the stochastic term analytically.\n\n- Experimentally, the third-order solver yields better FID than previous ODE solvers while requiring significantly smaller NFEs than SDE solvers.\n\nWEAKNESSES:\n- It seems that the same formula of the exponential integrated version of SDE (Eq.5) has been proposed in prior works [1] (see their eq.17, arXiv version 1). Since the DPM-solver [2] is essentially the same as DEIS, and they also use a Talyer expansion of the score function, I wonder how this work differs from these prior works. \n\n- The authors show in Prop 4.4 that setting $g=0$ in Eq.15, the resulting SEEDs solvers do not equal to DPM-solver. I wonder why this is the case based on the point above. In addition, setting $g=0$ in Eq.15 would not lead to meaningful backward ODE/SDEs.\n\n- The SEEDS-3 solver indeed showcases great improvement over ODE solvers. However, when using an increased NFE, the EDM solvers perform better than SEEDS-3 (SEEDS-3 even gets worse when increasing NFE as shown in Fig.1). In addition, I notice the SEEDS solver does not even apply to a low NFE regime (NFE<100) on CIFAR-10. It might be helpful to utilize the techniques in a concurrent work [3] to further improve the sample quality when increasing the NFE, or to achieve better results in the low NFE regime. It also is helpful to discuss [3] as they also focus on improving the existing SDEs.\n\n### Minors\n\n- Line 74 : $\\mu_t \\to \\alpha_t$\n\n\n\n[1] Zhang, Qinsheng, and Yongxin Chen. \"Fast sampling of diffusion models with exponential integrator.\" arXiv preprint arXiv:2204.13902 (2022).\n\n[2] Lu, Cheng, et al. \"Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.\" Advances in Neural Information Processing Systems 35 (2022): 5775-5787\n\n[3] Xu, Yilun, et al. \"Restart Sampling for Improving Generative Processes.\" arXiv preprint arXiv:2306.14878 (2023).\n\nQUESTIONS:\n- Is this work a straightforward application of prior works ([1], [2]) to SDEs?\n\n- In Figure 1, why does the SEEDS solver have a such high FID score (worse sample quality) in the low NFE regime? Is it due to the error when simulating the noise term?\n\nLIMITATIONS:\nDiscussed in Sec 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-01"}