Title: DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

fast sampling

ODE solver
Abstract: Diffusion probabilistic models (DPMs) have exhibited excellent performance for high-fidelity image generation while suffering from inefficient sampling. Recent works accelerate the sampling procedure by proposing fast ODE solvers that leverage the specific ODE form of DPMs. However, they highly rely on specific parameterization during inference (such as noise/data prediction), which might not be the optimal choice. In this work, we propose a novel formulation towards the optimal parameterization during sampling that minimizes the first-order discretization error of the ODE solution. Based on such formulation, we propose \textit{DPM-Solver-v3}, a new fast ODE solver for DPMs by introducing several coefficients efficiently computed on the pretrained model, which we call \textit{empirical model statistics}. We further incorporate multistep methods and a predictor-corrector framework, and propose some techniques for improving sample quality at small numbers of function evaluations (NFE) or large guidance scales. Experiments show that DPM-Solver-v3 achieves consistently better or comparable performance in both unconditional and conditional sampling with both pixel-space and latent-space DPMs, especially in 5$\sim$10 NFEs. We achieve FIDs of 12.21 (5 NFE), 2.51 (10 NFE) on unconditional CIFAR10, and MSE of 0.55 (5 NFE, 7.5 guidance scale) on Stable Diffusion, bringing a speed-up of 15\%$\sim$30\% compared to previous state-of-the-art training-free methods. Code is available at \url{https://github.com/thu-ml/DPM-Solver-v3}.

Abstract

Diffusion probabilistic models (DPMs) have exhibited excellent performance for high-fidelity image generation while suffering from inefficient sampling. Recent works accelerate the sampling procedure by proposing fast ODE solvers that leverage the specific ODE form of DPMs. However, they highly rely on specific parameterization during inference (such as noise/data prediction), which might not be the optimal choice. In this work, we propose a novel formulation towards the optimal parameterization during sampling that minimizes the firstorder discretization error of the ODE solution. Based on such formulation, we propose DPM-Solver-v3 , a new fast ODE solver for DPMs by introducing several coefficients efficiently computed on the pretrained model, which we call empirical model statistics . We further incorporate multistep methods and a predictorcorrector framework, and propose some techniques for improving sample quality at small numbers of function evaluations (NFE) or large guidance scales. Experiments show that DPM-Solver-v3 achieves consistently better or comparable performance in both unconditional and conditional sampling with both pixelspace and latent-space DPMs, especially in 5 ∼ 10 NFEs. We achieve FIDs of 12.21 (5 NFE), 2.51 (10 NFE) on unconditional CIFAR10, and MSE of 0.55 (5 NFE, 7.5 guidance scale) on Stable Diffusion, bringing a speed-up of 15% ∼ 30% compared to previous state-of-the-art training-free methods. Code is available at https://github.com/thu-ml/DPM-Solver-v3 .

1 Introduction

Diffusion probabilistic models (DPMs) [47, 15, 51] are a class of state-of-the-art image generators. By training with a strong encoder, a large model, and massive data as well as leveraging techniques such as guided sampling, DPMs are capable of generating high-resolution photorealistic and artistic images on text-to-image tasks. However, to generate high-quality visual content, DPMs usually require hundreds of model evaluations to gradually remove noise using a pretrained model [15], which is much more time-consuming compared to other deep generative models such as generative adversarial networks (GANs) [13]. The sampling overhead of DPMs emerges as a crucial obstacle hindering their integration into downstream tasks.

To accelerate the sampling process of DPMs, one can employ training-based methods [37, 53, 45] or training-free samplers [48, 51, 28, 3, 52, 56]. We focus on the latter approach since it requires no

(a) DPM-Solver++ [32] (MSE 0.60)

(b) UniPC [58] (MSE 0.65)

(c) DPM-Solver-v3 ( Ours , MSE 0.55)

extra training and is more flexible. Recent advanced training-free samplers [56, 31, 32, 58] mainly rely on the ODE form of DPMs, since its absence of stochasticity is essential for high-quality samples in around 20 steps. Besides, ODE solvers built on exponential integrators [18] converge faster. To change the original diffusion ODE into the form of exponential integrators, they need to cancel its linear term and obtain an ODE solution, where only the noise predictor needs to be approximated, and other terms can be exactly computed. Besides, Lu et al. [32] find that it is better to use another ODE solution where instead the data predictor needs to be approximated. How to choose such model parameterization (e.g., noise/data prediction) in the sampling of DPMs remains unrevealed.

In this work, we systematically study the problem of model parameterization and ODE formulation for fast sampling of DPMs. We first show that by introducing three types of coefficients, the original ODE solution can be reformulated to an equivalent one that contains a new model parameterization. Besides, inspired by exponential Rosenbrock-type methods [19] and first-order discretization error analysis, we also show that there exists an optimal set of coefficients efficiently computed on the pretrained model, which we call empirical model statistics (EMS). Building upon our novel ODE formulation, we further propose a new high-order solver for diffusion ODEs named DPM-Solverv3 , which includes a multistep predictor-corrector framework of any order, as well as some novel techniques such as pseudo high-order method to boost the performance at extremely few steps or large guidance scale.

We conduct extensive experiments with both pixel-space and latent-space DPMs to verify the effectiveness of DPM-Solver-v3. Compared to previous fast samplers, DPM-Solver-v3 can consistently improve the sample quality in 5 ∼ 20 steps, and make a significant advancement within 10 steps.

2 Background

2.1 Diffusion Probabilistic Models

Suppose we have a D -dimensional data distribution q 0 ( x 0 ) . Diffusion probabilistic models (DPMs) [47, 15, 51] define a forward diffusion process { q t } T t =0 to gradually degenerate the data x 0 ∼ q 0 ( x 0 ) with Gaussian noise, which satisfies the transition kernel q 0 t ( x t | x 0 ) = N ( α t x 0 , σ 2 t I ) , such that q T ( x T ) is approximately pure Gaussian. α t , σ t are smooth scalar functions of t , which are called noise schedule . The transition can be easily applied by x t = α t x 0 + σ t ϵ , ϵ ∼ N ( 0 , I ) . To train DPMs, a neural network ϵ θ ( x t , t ) is usually parameterized to predict the noise ϵ by minimizing E x 0 ∼ q 0 ( x 0 ) , ϵ ∼N ( 0 , I ) ,t ∼U (0 ,T ) [ w ( t ) ∥ ϵ θ ( x t , t ) -ϵ ∥ 2 2 ] , where w ( t ) is a weighting function. Sampling of DPMs can be performed by solving diffusion ODE [51] from time T to time 0 :

where f ( t ) = dlog α t d t , g 2 ( t ) = d σ 2 t d t -2 dlog α t d t σ 2 t [23]. In addition, the conditional sampling by DPMs can be conducted by guided sampling [10, 16] with a conditional noise predictor ϵ θ ( x t , t, c ) , where c is the condition. Specifically, classifier-free guidance [16] combines the unconditional/conditional model and obtains a new noise predictor ϵ ′ θ ( x t , t, c ) := s ϵ θ ( x t , t, c ) + (1 -s ) ϵ θ ( x t , t, ∅ ) ,

where ∅ is a special condition standing for the unconditional case, s > 0 is the guidance scale that controls the trade-off between image-condition alignment and diversity; while classifier guidance [10] uses an extra classifier p ϕ ( c | x t , t ) to obtain a new noise predictor ϵ ′ θ ( x t , t, c ) := ϵ θ ( x t , t ) -sσ t ∇ x log p ϕ ( c | x t , t ) .

In addition, except for the noise prediction, DPMs can be parameterized as score predictor s θ ( x t , t ) to predict ∇ x log q t ( x t , t ) , or data predictor x θ ( x t , t ) to predict x 0 . Under variance-preserving (VP) noise schedule which satisfies α 2 t + σ 2 t = 1 [51], 'v' predictor v θ ( x t , t ) is also proposed to predict α t ϵ -σ t x 0 [45]. These different parameterizations are theoretically equivalent, but have an impact on the empirical performance when used in training [23, 59].

2.2 Fast Sampling of DPMs with Exponential Integrators

Among the methods for solving the diffusion ODE (1), recent works [56, 31, 32, 58] find that ODE solvers based on exponential integrators [18] are more efficient and robust at a small number of function evaluations (<50). Specifically, an insightful observation by Lu et al. [31] is that, by changeof-variable from t to λ t := log( α t /σ t ) (half of the log-SNR), the diffusion ODE is transformed to

where ˙ α λ := d α λ d λ . By utilizing the semi-linear structure of the diffusion ODE and exactly computing the linear term [56, 31], we can obtain the ODE solution as Eq. (3) (left). Such exact computation of the linear part reduces the discretization errors [31]. Moreover, by leveraging the equivalence of different parameterizations, DPM-Solver++ [32] rewrites Eq. (2) by the data predictor x θ ( x λ , λ ) := ( x λ -σ λ ϵ θ ( x λ , λ )) /α λ and obtains another ODE solution as Eq. (3) (right). Such solution does not need to change the pretrained noise prediction model ϵ θ during the sampling process, and empirically outperforms previous samplers based on ϵ θ [31].

However, to the best of our knowledge, the parameterizations for sampling are still manually selected and limited to noise/data prediction, which are not well-studied.

3 Method

We now present our method. We start with a new formulation of the ODE solution with extra coefficients, followed by our high-order solver and some practical considerations. In the following discussions, we assume all the products between vectors are element-wise, and f ( k ) ( x λ , λ ) = d k f ( x λ ,λ ) d λ k is the k -th order total derivative of any function f w.r.t. λ .

3.1 Improved Formulation of Exact Solutions of Diffusion ODEs

As mentioned in Sec. 2.2, it is promising to explore the semi-linear structure of diffusion ODEs for fast sampling [56, 31, 32]. Firstly , we reveal one key insight that we can choose the linear part according to Rosenbrock-type exponential integrators [19, 18]. To this end, we consider a general form of diffusion ODEs by rewriting Eq. (2) as

where l λ is a D -dimensional undetermined coefficient depending on λ . We choose l λ to restrict the Frobenius norm of the gradient of the non-linear part w.r.t. x :

where p θ λ is the distribution of samples on the ground-truth ODE trajectories at λ (i.e., model distribution). Intuitively, it makes N θ insensitive to the errors of x and cancels all the 'linearty' of

N θ . With l λ = l ∗ λ , by the ' variation-of-constants ' formula [1], starting from x λ s at time s , the exact solution of Eq. (4) at time t is

where f θ ( x λ , λ ) := N θ ( x λ ,λ ) α λ . To calculate the solution in Eq. (6), we need to approximate f θ for each λ ∈ [ λ s , λ t ] by certain polynomials [31, 32].

Secondly , we reveal another key insight that we can choose different functions to be approximated instead of f θ and further reduce the discretization error, which is related to the total derivatives of the approximated function. To this end, we consider a scaled version of f θ i.e., h θ ( x λ , λ ) := e -∫ λ λs s τ d τ f θ ( x λ , λ ) where s λ is a D -dimensional coefficient dependent on λ , and then Eq. (6) becomes

Comparing with Eq. (6), we change the approximated function from f θ to h θ by using an additional scaling term related to s λ . As we shall see, the first-order discretization error is positively related to the norm of the first-order derivative h (1) θ = e -∫ λ λs s τ d τ ( f (1) θ -s λ f θ ) . Thus, we aim to minimize ∥ f (1) θ -s λ f θ ∥ 2 , in order to reduce ∥ h (1) θ ∥ 2 and the discretization error. As f θ is a fixed function depending on the pretrained model, this minimization problem essentially finds a linear function of f θ to approximate f (1) θ . To achieve better linear approximation, we further introduce a bias term b λ ∈ R D and construct a function g θ satisfying g (1) θ = e -∫ λ λs s τ d τ ( f (1) θ -s λ f θ -b λ ) , which gives

With g θ , Eq. (7) becomes

Such formulation is equivalent to Eq. (3) but introduces three types of coefficients and a new parameterization g θ . We show in Appendix I.1 that the generalized parameterization g θ in Eq. (8) can cover a wide range of parameterization families in the form of ψ θ ( x λ , λ ) = α ( λ ) ϵ θ ( x λ , λ ) + β ( λ ) x λ + γ ( λ ) . We aim to reduce the discretization error by finding better coefficients than previous works [31, 32].

Now we derive the concrete formula for analyzing the first-order discretization error. By replacing g θ ( x λ , λ ) with g θ ( x λ s , λ s ) in Eq. (9), we obtain the first-order approximation ˆ x λ t = α λ t e -∫ λt λs l λ d λ ( x λs α λs -∫ λ t λ s e ∫ λ λs ( l τ + s τ )d τ ( g θ ( x λ s , λ s ) + ∫ λ λ s e -∫ r λs s τ d τ b r d r ) d λ ) . As g θ ( x λ s , λ s ) = g θ ( x λ , λ ) + ( λ s -λ ) g (1) θ ( x λ , λ ) + O (( λ -λ s ) 2 ) by Taylor expansion, it follows that the first-order discretization error can be expressed as

where h = λ t -λ s . Thus, given the optimal l λ = l ∗ λ in Eq. (5), the discretization error ˆ x λ t -x λ t mainly depends on f (1) θ -s λ f θ -b λ . Based on this insight, we choose the coefficients s λ , b λ by solving

For any λ , l ∗ λ , s ∗ λ , b ∗ λ all have analytic solutions involving the Jacobian-vector-product of the pretrained model ϵ θ , and they can be unbiasedly evaluated on a few datapoints { x ( n ) λ } K ∼ p θ λ ( x λ ) via MonteCarlo estimation (detailed in Section 3.4 and Appendix C.1.1). Therefore, we call l λ , s λ , b λ empirical model statistics (EMS). In the following sections, we'll show that by approximating g θ with Taylor expansion, we can develop our high-order solver for diffusion ODEs.

ˆ

x

t

x

s

(

t

s

,

g

s

)

(

t

i

n

,

g

i

n

)

estimation

g

(0)

s

g

(1)

s

g

(

n

)

s

(

t

i

1

,

g

i

1

)

(a) Local Approximation

x

0

ˆ

x

i

-

2

ˆ

x

i

-

1

ˆ

x

i

ˆ

ϵ

i

-

2

ˆ

g

i

-

2

ˆ

ϵ

i

-

1

ˆ

g

i

-

1

ˆ

ϵ

i

ˆ

g

i

ˆ

x

c

i

Predictor

Corrector

cache

ˆ

g

·

=

a

·

,i

-

1

ˆ

x

·

+

b

·

,i

-

1

ˆ

ϵ

·

+

c

·

,i

-

1

t

0

= 1

t

i

-

2

t

i

-

1

t

i

ˆ

ϵ

·

:=

ϵ

θ

(ˆ

x

·

, t

·

)

(b) Multistep Predictor-Corrector

3.2 Developing High-Order Solver

In this section, we propose our high-order solver for diffusion ODEs with local accuracy and global convergence order guarantee by leveraging our proposed solution formulation in Eq. (9). The proposed solver and analyses are highly motivated by the methods of exponential integrators [17, 18] in the ODE literature and their previous applications in the field of diffusion models [56, 31, 32, 58]. Though the EMS are designed to minimize the first-order error, they can also help our high-order solver (see Appendix I.2).

For simplicity, denote A ( λ s , λ t ) := e -∫ λt λs l τ d τ , E λ s ( λ ) := e ∫ λ λs ( l τ + s τ )d τ , B λ s ( λ ) := ∫ λ λ s e -∫ r λs s τ d τ b r d r . Though high-order ODE solvers essentially share the same mathematical principles, since we utilize a more complicated parameterization g θ and ODE formulation in Eq. (9) than previous works [56, 31, 32, 58], we divide the development of high-order solver into simplified local and global parts, which are not only easier to understand, but also neat and general for any order.

3.2.1 Local Approximation

Firstly, we derive formulas and properties of the local approximation, which describes how we transit locally from time s to time t . It can be divided into two parts: discretization of the integral in Eq. (9) and estimating the high-order derivatives in the Taylor expansion.

Discretization. Denote g ( k ) s := g ( k ) θ ( x λ s , λ s ) . For n ≥ 0 , to obtain the ( n +1) -th order discretization of Eq. (9), we take the n -th order Taylor expansion of g θ ( x λ , λ ) w.r.t. λ at λ s : g θ ( x λ , λ ) = ∑ n k =0 ( λ -λ s ) k k ! g ( k ) s + O (( λ -λ s ) n +1 ) . Substituting it into Eq. (9) yields

Here we only need to estimate the k -th order total derivatives g ( k ) θ ( x λ s , λ s ) for 0 ≤ k ≤ n , since the other terms are determined once given λ s , λ t and l λ , s λ , b λ , which we'll discuss next.

High-order derivative estimation. For ( n +1) -th order approximation, we use the finite difference of g θ ( x λ , λ ) at previous n +1 steps λ i n , . . . , λ i 1 , λ s to estimate each g ( k ) θ ( x λ s , λ s ) . Such derivation is to match the coefficients of Taylor expansions. Concretely, denote δ k := λ i k -λ s , g i k := g θ ( x λ i k , λ i k ) , and the estimated high-order derivatives ˆ g ( k ) s can be solved by the following linear system:

Then by substituting ˆ g ( k ) s into Eq. (12) and dropping the O ( h n +2 ) error terms, we obtain the ( n +1) -th order local approximation:

where ˆ g (0) θ ( x λ s , λ s ) = g s . Eq. (13) and Eq. (14) provide an update rule to transit from time s to time t and get an approximated solution ˆ x t , when we already have the solution x s . For ( n +1) -th order approximation, we need n extra solutions x λ i k and their corresponding function values g i k . We illustrate the procedure in Fig. 2(a) and summarize it in Appendix C.2. In the following theorem, we show that under some assumptions, such local approximation has a guarantee of order of accuracy.

Theorem 3.1 (Local order of accuracy, proof in Appendix B.2.1) . Suppose x λ i k are exact (i.e., on the ground-truth ODE trajectory passing x s ) for k = 1 , . . . , n , then under some regularity conditions detailed in Appendix B.1, the local truncation error ˆ x t -x t = O ( h n +2 ) , which means the local approximation has ( n +1) -th order of accuracy.

Besides, we have the following theorem showing that, whatever the order is, the local approximation is unbiased given our choice of s λ , b λ in Eq. (11). In practice, the phenomenon of reduced bias can be empirically observed (Section 4.3).

Theorem 3.2 (Local unbiasedness, proof in Appendix B.4) . Given the optimal s ∗ λ , b ∗ λ in Eq. (11) , For the ( n +1) -th order approximation, suppose x λ i 1 , . . . , x λ i n are on the ground-truth ODE trajectory passing x λ s , then E p θ λs ( x s ) [ ˆ x t -x t ] = 0 .

3.2.2 Global Solver

Given M +1 time steps { t i } i M =0 , starting from some initial value, we can repeat the local approximation M times to make consecutive transitions from each t i -1 to t i until we reach an acceptable solution. At each step, we apply multistep methods [1] by caching and reusing the previous n values at timesteps t i -1 -n , . . . , t i -2 , which is proven to be more stable when NFE is small [32, 56]. Moreover, we also apply the predictor-corrector method [58] to refine the approximation at each step without introducing extra NFE. Specifically, the ( n +1) -th order predictor is the case of the local approximation when we choose ( t i n , . . . , t i 1 , s, t ) = ( t i -1 -n , . . . , t i -2 , t i -1 , t i ) , and the ( n +1) -th order corrector is the case when we choose ( t i n , . . . , t i 1 , s, t ) = ( t i -n , . . . , t i -2 , t i , t i -1 , t i ) . We present our ( n + 1) -th order multistep predictor-corrector procedure in Appendix C.2. We also illustrate a second-order case in Fig. 2(b). Note that different from previous works, in the local transition from t i -1 to t i , the previous function values ˆ g i k ( 1 ≤ k ≤ n ) used for derivative estimation are dependent on i and are different during the sampling process because g θ is dependent on the current t i -1 (see Eq. (8)). Thus, we directly cache ˆ x i , ˆ ϵ i and reuse them to compute ˆ g i in the subsequent steps. Notably, our proposed solver also has a global convergence guarantee, as shown in the following theorem. For simplicity, we only consider the predictor case and the case with corrector can also be proved by similar derivations in [58].

Theorem 3.3 (Global order of convergence, proof in Appendix B.2.2) . For ( n +1) -th order predictor, if we iteratively compute a sequence { ˆ x i } i M =0 to approximate the true solutions { x i } i M =0 at { t i } i M =0 , then under both local and global assumptions detailed in Appendix B.1, the final error | ˆ x M -x M | = O ( h n +1 ) , where | · | denotes the element-wise absolute value, and h = max 1 ≤ i ≤ M ( λ i -λ i -1 ) .

3.3 Practical Techniques

In this section, we introduce some practical techniques that further improve the sample quality in the case of small NFE or large guidance scales.

Pseudo-order solver for small NFE. When NFE is extremely small (e.g., 5 ∼ 10), the error at each timestep becomes rather large, and incorporating too many previous values by high-order solver at each step will cause instabilities. To alleviate this problem, we propose a technique called pseudoorder solver: when estimating the k -th order derivative, we only utilize the previous k +1 function values of g θ , instead of all the n previous values as in Eq. (13). For each k , we can obtain ˆ g ( k ) s by

solving a part of Eq. (13) and taking the last element:

In practice, we do not need to solve n linear systems. Instead, the solutions for ˆ g ( k ) s , k = 1 , . . . , n have a simpler recurrence relation similar to Neville's method [36] in Lagrange polynomial interpolation. Denote i 0 := s so that δ 0 = λ i 0 -λ s = 0 , we have

Theorem 3.4 (Pseudo-order solver) . For each k , the solution in Eq. (15) is ˆ g ( k ) s = k ! D ( k ) 0 , where

Proof in Appendix B.3. Note that the pseudo-order solver of order n > 2 no longer has the guarantee of n -th order of accuracy, which is not so important when NFE is small. In our experiments, we mainly rely on two combinations: when we use n -th order predictor, we then combine it with n -th order corrector or ( n +1) -th pseudo-order corrector.

Half-corrector for large guidance scales. When the guidance scale is large in guided sampling, we find that corrector may have negative effects on the sample quality. We propose a useful technique called half-corrector by using the corrector only in the time region t ≤ 0 . 5 . Correspondingly, the strategy that we use corrector at each step is called full-corrector .

3.4 Implementation Details

In this section, we give some implementation details about how to compute and integrate the EMS in our solver and how to adapt them to guided sampling.

Estimating EMS. For a specific λ , the EMS l ∗ λ , s ∗ λ , b ∗ λ can be estimated by firstly drawing K (1024 ∼ 4096) datapoints x λ ∼ p θ λ ( x λ ) with 200-step DPM-Solver++ [32] and then analytically computing some terms related to ϵ θ (detailed in Appendix C.1.1). In practice, we find it both convenient and effective to choose the distribution of the dataset q 0 to approximate p θ 0 . Thus, without further specifications, we directly use samples from q 0 .

Estimating integrals of EMS. We estimate EMS on N ( 120 ∼ 1200 ) timesteps λ j 0 , λ j 1 , . . . , λ j N and use trapezoidal rule to estimate the integrals in Eq. (12) (see Appendix I.3 for the estimation error analysis). We also apply some pre-computation for the integrals to avoid extra computation costs during sampling, detailed in Appendix C.1.2.

Adaptation to guided sampling. Empirically, we find that within a common range of guidance scales, we can simply compute the EMS on the model without guidance, and it can work for both unconditional sampling and guided sampling cases. See Appendix J for more discussions.

3.5 Comparison with Existing Methods

By comparing with existing diffusion ODE solvers that are based on exponential integrators [56, 31, 32, 58], we can conclude that (1) Previous ODE formulations with noise/data prediction are special cases of ours by setting l λ , s λ , b λ to specific values. (2) Our first-order discretization can be seen as improved DDIM. See more details in Appendix A.

4 Experiments

In this section, we show that DPM-Solver-v3 can achieve consistent and notable speed-up for both unconditional and conditional sampling with both pixel-space and latent-space DPMs. We conduct extensive experiments on diverse image datasets, where the resolution ranges from 32 to 256. First, we present the main results of sample quality comparison with previous state-of-the-art training-free

5

6

8

10

12

15

20

NFE

5

10

15

20

25

FID

†

DEIS

DPM-Solver++

UniPC

DPM-Solver-v3

(a) CIFAR10

5

6

8

10

12

15

20

NFE

0

5

10

15

20

25

30

FID

Heun's (EDM)

DPM-Solver++

UniPC

DPM-Solver-v3

(b) CIFAR10

5

6

8

10

12

15

20

NFE

4

6

8

10

12

14

16

18

FID

DPM-Solver++

UniPC

DPM-Solver-v3

(c) LSUN-Bedroom

(Latent-Diffusion, Latent DPM)

5

6

8

10

12

15

20

NFE

8

10

12

14

16

FID

DPM-Solver++

UniPC

DPM-Solver-v3

(a) ImageNet-256

(Classifier Guidance,

s

= 2

.

0

)

5

6

8

10

12

15

20

NFE

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

RMSE

DPM-Solver++

UniPC

DPM-Solver-v3

(b) MS-COCO2014

(Stable-Diffusion, Latent DPM)

(Classifier-Free Guidance,

s

= 1

.

5

)

5

6

8

10

15

20

NFE

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

MSE

DPM-Solver++

UniPC

DPM-Solver-v3

(c) MS-COCO2014

(Stable-Diffusion, Latent DPM)

(Classifier-Free Guidance,

s

= 7

.

5

)

methods. Then we illustrate the effectiveness of our method by visualizing the EMS and samples. Additional ablation studies are provided in Appendix G. On each dataset, we choose a sufficient number of timesteps N and datapoints K for computing the EMS to reduce the estimation error, while the EMS can still be computed within hours. After we obtain the EMS and precompute the integrals involving them, there is negligible extra overhead in the sampling process. We provide the runtime comparison in Appendix E. We refer to Appendix D for more detailed experiment settings.

4.1 Main Results

We present the results in 5 ∼ 20 number of function evaluations (NFE), covering both few-step cases and the almost converged cases, as shown in Fig. 3 and Fig. 4. For the sake of clarity, we mainly compare DPM-Solver-v3 to DPM-Solver++ [32] and UniPC [58], which are the most state-of-the-art diffusion ODE solvers. We also include the results for DEIS [56] and Heun's 2nd order method in EDM [21], but only for the datasets on which they originally reported. We don't show the results for other methods such as DDIM [48], PNDM [28], since they have already been compared in previous works and have inferior performance. The quantitative results on CIFAR10 [24] are listed in Table 1, and more detailed quantitative results are presented in Appendix F.

Unconditional sampling We first evaluate the unconditional sampling performance of different methods on CIFAR10 [24] and LSUN-Bedroom [55]. For CIFAR10 we use two pixel-space DPMs,

-

4

-

2

0

2

4

6

λ

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

l

λ

s

λ

b

λ

(a) CIFAR10 (ScoreSDE)

-

4

-

2

0

2

4

6

λ

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

l

λ

s

λ

b

λ

(b) CIFAR10 (EDM)

-

2

-

1

0

1

2

3

λ

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

l

λ

s

λ

b

λ

(c) MS-COCO2014

(Stable-Diffusion,

s

= 7

.

5

)

(a) DPM-Solver++ [32] (FID 18.59)

(b) UniPC [58] (FID 12.24)

(c) DPM-Solver-v3 (

Ours

) (FID 7.54)

one is based on ScoreSDE [51] which is a widely adopted model by previous samplers, and another is based on EDM [21] which achieves the best sample quality. For LSUN-Bedroom, we use the latentspace Latent-Diffusion model [43]. We apply the multistep 3rd-order version for DPM-Solver++, UniPC and DPM-Solver-v3 by default, which performs best in the unconditional setting. For UniPC, we report the better result of their two choices B 1 ( h ) = h and B 2 ( h ) = e h -1 at each NFE. For our DPM-Solver-v3, we tune the strategies of whether to use the pseudo-order predictor/corrector at each NFE on CIFAR10, and use the pseudo-order corrector on LSUN-Bedroom. As shown in Fig. 3, we find that DPM-Solver-v3 can achieve consistently better FID, which is especially notable when NFE is 5 ∼ 10. For example, we improve the FID on CIFAR10 with 5 NFE from 23 to 12 with ScoreSDE, and achieve an FID of 2 . 51 with only 10 NFE with the advanced DPM provided by EDM. On LSUN-Bedroom, with around 12 minutes computing of the EMS, DPM-Solver-v3 converges to the FID of 3.06 with 12 NFE, which is approximately 60% sampling cost of the previous best training-free method (20 NFE by UniPC).

Conditional sampling. We then evaluate the conditional sampling performance, which is more widely used since it allows for controllable generation with user-customized conditions. We choose two conditional settings, one is classifier guidance on pixel-space Guided-Diffusion [10] model trained on ImageNet-256 dataset [9] with 1000 class labels as conditions; the other is classifierfree guidance on latent-space Stable-Diffusion model [43] trained on LAION-5B dataset [46] with CLIP [41] embedded text prompts as conditions. We evaluate the former at the guidance scale of 2 . 0 , following the best choice in [10]; and the latter at the guidance scale of 1 . 5 (following the original paper) or 7 . 5 (following the official code) with prompts random selected from MS-COCO2014 validation set [26]. Note that the FID of Stable-Diffusion samples saturates to 15.0 ∼ 16.0 even within 10 steps when the latent codes are far from convergence, possibly due to the powerful image decoder (see Appendix H). Thus, following [32], we measure the mean square error (MSE) between the generated latent code ˆ x and the ground-truth solution x ∗ (i.e., ∥ ˆ x -x ∗ ∥ 2 2 /D ) to evaluate convergence, starting from the same Gaussian noise. We obtain x ∗ by 200-step DPM-Solver++, which is enough to ensure the convergence.

We apply the multistep 2nd-order version for DPM-Solver++, UniPC and DPM-Solver-v3, which performs best in conditional setting. For UniPC, we only apply the choice B 2 ( h ) = e h -1 , which performs better than B 1 ( h ) . For our DPM-Solver-v3, we use the pseudo-order corrector by default, and report the best results between using half-corrector/full-corrector on Stable-Diffusion ( s = 7 . 5 ). As shown in Fig. 4, DPM-Solver-v3 can achieve better sample quality or convergence at most NFEs, which indicates the effectiveness of our method and techniques under the conditional setting. It's worth noting that UniPC, which adopts an extra corrector, performs even worse than DPM-Solver++ when NFE<10 on Stable-Diffusion ( s = 7 . 5 ). With the combined effect of the EMS and the halfcorrector technique, we successfully outperform DPM-Solver++ in such a case. Detailed comparisons can be found in the ablations in Appendix G.

4.2 Visualizations of Estimated EMS

We further visualize the estimated EMS in Fig. 5. Since l λ , s λ , b λ are D -dimensional vectors, we average them over all dimensions to obtain a scalar. From Fig. 5, we find that l λ gradually changes from 1 to near 0 as the sampling proceeds, which suggests we should gradually slide from data prediction to noise prediction. As for s λ , b λ , they are more model-specific and display many fluctuations, especially for ScoreSDE model [51] on CIFAR10. Apart from the estimation error of the EMS, we suspect that it comes from the fluctuations of ϵ (1) θ , which is caused by the periodicity of trigonometric functions in the positional embeddings of the network input. It's worth noting that the fluctuation of s λ , b λ will not cause instability in our sampler (see Appendix J).

4.3 Visual Quality

We present some qualitative comparisons in Fig. 6 and Fig. 1. We can find that previous methods tend to have a small degree of color contrast at small NFE, while our method is less biased and produces more visual details. In Fig. 1(b), we can observe that previous methods with corrector may cause distortion at large guidance scales (in the left-top image, a part of the castle becomes a hill; in the left-bottom image, the hill is translucent and the castle is hanging in the air), while ours won't. Additional samples are provided in Appendix K.

5 Conclusion

We study the ODE parameterization problem for fast sampling of DPMs. Through theoretical analysis, we find a novel ODE formulation with empirical model statistics, which is towards the optimal one to minimize the first-order discretization error. Based on such improved ODE formulation, we propose a new fast solver named DPM-Solver-v3, which involves a multistep predictor-corrector framework of any order and several techniques for improved sampling with few steps or large guidance scale. Experiments demonstrate the effectiveness of DPM-Solver-v3 in both unconditional conditional sampling with both pixel-space latent-space pre-trained DPMs, and the significant advancement of sample quality in 5 ∼ 10 steps.

Limitations and broader impact Despite the great speedup in small numbers of steps, DPM-Solverv3 still lags behind training-based methods and is not fast enough for real-time applications. Besides, we conducted theoretical analyses of the local error, but didn't explore the global design spaces, such as the design of timestep schedules during sampling. And commonly, there are potential undesirable effects that DPM-Solver-v3 may be abused to accelerate the generation of fake and malicious content.

References

Kendall Atkinson, Weimin Han, and David E Stewart. Numerical solution of ordinary differential equations , volume 108. John Wiley & Sons, 2011.

Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In International Conference on Machine Learning , pages 1555-1584. PMLR, 2022.

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: An analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations , 2022.

Costas Bekas, Effrosyni Kokiopoulou, and Yousef Saad. An estimator for the diagonal of a matrix. Applied numerical mathematics , 57(11-12):1214-1229, 2007.

Mari Paz Calvo and César Palencia. A class of explicit multistep exponential integrators for semilinear problems. Numerische Mathematik , 102:367-381, 2006.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations , 2021.

Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations , 2022.

Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusionbased semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations , 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255. IEEE, 2009.

Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems , volume 34, pages 87808794, 2021.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. In Advances in Neural Information Processing Systems , 2022.

Alfredo Eisinberg and Giuseppe Fedele. On the inversion of the vandermonde matrix. Applied mathematics and computation , 174(2):1384-1397, 2006.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems , volume 27, pages 2672-2680, 2014.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , volume 33, pages 6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021.

Marlis Hochbruck and Alexander Ostermann. Explicit exponential Runge-Kutta methods for semilinear parabolic problems. SIAM Journal on Numerical Analysis , 43(3):1069-1090, 2005.

Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica , 19:209286, 2010.

Marlis Hochbruck, Alexander Ostermann, and Julia Schweitzer. Exponential rosenbrock-type methods. SIAM Journal on Numerical Analysis , 47(1):786-803, 2009.

Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation , 18(3):10591076, 1989.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems , 2022.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems , 2022.

Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems , 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Shengmeng Li, Luping Liu, Zenghao Chai, Runnan Li, and Xu Tan. Era-solver: Errorrobust adams solver for fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2301.12935 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 11020-11028, 2022.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations , 2021.

Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations , 2022.

Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning , pages 14429-14460. PMLR, 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems , 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpmsolver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 , 2021.

Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In NeurIPS 2022 Workshop on Score-Based Methods , 2022.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2022.

Eric Harold Neville. Iterative interpolation . St. Joseph's IS Press, 1934.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning , pages 16784-16804. PMLR, 2022.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei. Diffusion-based voice conversion with fast maximum likelihood sampling scheme. In International Conference on Learning Representations , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems , 2022.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 , 2023.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Advances in Neural Information Processing Systems , volume 34, pages 1415-1428, 2021.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Hideyuki Tachibana, Mocho Go, Muneyoshi Inahara, Yotaro Katayama, and Yotaro Watanabe. Itô-taylor sampling scheme for denoising diffusion probabilistic models using ideal derivatives. arXiv e-prints , pages arXiv-2112, 2021.

Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations , 2022.

Suttisak Wizadwongsa and Supasorn Suwajanakorn. Accelerating guided diffusion sampling with splitting numerical methods. In The Eleventh International Conference on Learning Representations , 2022.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations , 2022.

Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. In Advances in Neural Information Processing Systems , 2022.

Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A unified predictorcorrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867 , 2023.

Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. arXiv preprint arXiv:2305.03935 , 2023.

A Related Work

Diffusion probabilistic models (DPMs) [47, 15, 51], also known as score-based generative models (SGMs), have achieved remarkable generation ability on image domain [10, 21], yielding extensive applications such as speech, singing and video synthesis [6, 27, 14], controllable image generation, translation and editing [38, 42, 43, 35, 57, 8], likelihood estimation [50, 23, 30, 59], data compression [23] and inverse problem solving [7, 22].

A.1 Fast Sampling Methods for DPMs

Fast sampling methods based on extra training or optimization include learning variances in the reverse process [37, 2], learning sampling schedule [53], learning high-order model derivatives [11], model refinement [29] and model distillation [45, 33, 49]. Though distillation-based methods can generate high-quality samples in less than 5 steps, they additionally bring onerous training costs. Moreover, the distillation process will inevitably lose part of the information of the original model, and is hard to be adapted to pre-trained large DPMs [43, 44, 42] and conditional sampling [34]. Some of distillation-based methods also lack the ability to make flexible trade-offs between sample speed and sample quality.

In contrast, training-free samplers are more lightweight and flexible. Among them, samplers based on diffusion ODEs generally require fewer steps than those based on diffusion SDEs [51, 40, 3, 48, 32], since SDEs introduce more randomness and make the denoising harder in the sampling process. Previous samplers handle the diffusion ODEs with different methods, such as Heun's methods [21], splitting numerical methods [54], pseudo numerical methods [28], Adams methods [25] or exponential integrators [56, 31, 32, 58].

A.2 Comparison with Existing Solvers Based on Exponential Integrators

In this section, we make some theoretical comparisons between DPM-Solver-v3 and existing diffusion ODE solvers that are based on exponential integrators [56, 31, 32, 58]. We summarize the results in Table 2, and provide some analysis below.

Previous ODE formulation as special cases of ours. First, we compare the formulation of the ODE solution. Our reformulated ODE solution in Eq. (9) involves extra coefficients l λ , s λ , b λ , and it corresponds to a new predictor g θ to be approximated by Taylor expansion. By comparing ours with previous ODE formulations in Eq. (3) and their corresponding noise/data prediction, we can easily figure out that they are special cases of ours by setting l λ , s λ , b λ to specific values:

Noise prediction: l λ = 0 , s λ = -1 , b λ = 0

Data prediction: l λ = 1 , s λ = 0 , b λ = 0

It's worth noting that, though our ODE formulation can degenerate to previous ones, it may still result in different solvers, since previous works conduct some equivalent substitution of the same order in the local approximation (for example, the choice of B 1 ( h ) = h or B 2 ( h ) = e h -1 in UniPC [58]). We never conduct such substitution, thus saving the efforts to tune it.

Moreover, under our framework, we find that DPM-Solver++ is a model-agnostic approximation of DPM-Solver-v3, under the Gaussian assumption . Specifically, according to Eq. (5), we have

If we assume q λ ( x λ ) ≈ N ( x λ | α λ x 0 , σ 2 λ I ) for some fixed x 0 , then the optimal noise predictor is

It follows that σ λ ∇ x ϵ θ ( x λ , λ ) ≈ I , thus l ∗ λ ≈ 1 by Eq. (5), which corresponds the data prediction model used in DPM-Solver++. Moreover, for small enough λ (i.e., t near to T ), the Gaussian assumption is almost true (see Section 4.2), thus the data-prediction DPM-Solver++ approximately computes all the linear terms at the initial stage. To the best of our knowledge, this is the first explanation for the reason why the data-prediction DPM-Solver++ outperforms the noise-prediction DPM-Solver.

First-order discretization as improved DDIM Previous methods merely use noise/data parameterization, whether or not they change the time domain from t to λ . While they differ in high-order cases, they are proven to coincide in the first-order case, which is DDIM [48] (deterministic case, η = 0 ):

However, the first-order case of our method is

which is not DDIM since we choose a better parameterization by the estimated EMS. Empirically, our first-order solver performs better than DDIM, as detailed in Appendix G.

B Proofs

B.1 Assumptions

In this section, we will give some mild conditions under which the local order of accuracy of Algorithm 1 and the global order of convergence of Algorithm 2 (predictor) are guaranteed.

B.1.1 Local

First, we will give the assumptions to bound the local truncation error.

Assumption B.1. The total derivatives of the noise prediction model d k ϵ θ ( x λ ,λ ) d λ k , k = 1 , . . . , n exist and are continuous.

Assumption B.2. The coefficients l λ , s λ , b λ are continuous and bounded. d k l λ d λ k , d k s λ d λ k , d k b λ d λ k , k = 1 , . . . , n exist and are continuous.

Assumption B.1 is required for the Taylor expansion which is regular in high-order numerical methods. Assumption B.2 requires the boundness of the coefficients as well as regularizes the coefficients' smoothness to enable the Taylor expansion for g θ ( x λ , λ ) , which holds in practice given the smoothness of ϵ θ ( x λ , λ ) and p θ λ ( x λ ) . Assumption B.3 makes sure δ k and λ t -λ s is of the same order, i.e., there exists some constant r k = O (1) so that δ k = r k ( λ t -λ s ) , which is satisfied in regular multistep methods.

B.1.2 Global

Then we will give the assumptions to bound the global error.

Assumption B.4. The noise prediction model ϵ θ ( x , t ) is Lipschitz w.r.t. to x .

Assumption B.5. h = max 1 ≤ i ≤ M ( λ i -λ i -1 ) = O (1 /M ) .

Assumption B.6. The starting values ˆ x i , 1 ≤ i ≤ n satisfies ˆ x i -x i = O ( h n +1 ) .

Assumption B.4 is common in the analysis of ODEs, which assures ϵ θ (ˆ x t , t ) -ϵ θ ( x t , t ) = O (ˆ x t -x t ) . Assumption B.5 implies that the step sizes are rather uniform. Assumption B.6 is common in the convergence analysis of multistep methods [5].

B.2 Order of Accuracy and Convergence

In this section, we prove the local and global order guarantees detailed in Theorem 3.1 and Theorem 3.3.

B.2.1 Local

Proof. (Proof of Theorem 3.1) Denote h := λ t -λ s . Subtracting the Taylor-expanded exact solution in Eq. (12) from the local approximation in Eq. (14), we have

First we examine the order of A ( λ s , λ t ) and ∫ λ t λ s E λ s ( λ )( λ -λ s ) k d λ . Under Assumption B.2, there exists some constant C 1 , C 2 such that -l λ < C 1 , l λ + s λ < C 2 . So

Next we examine the order of ˆ g ( k ) θ ( x λs ,λ s ) -g ( k ) θ ( x λs ,λ s ) k ! . Under Assumption B.1 and Assumption B.2, since g θ is elementary function of ϵ θ and l λ , s λ , b λ , we know g ( k ) θ ( x λ s , λ s ) , k = 1 , . . . , n exist and are continuous. Adopting the notations in Eq. (13), by Taylor expansion, we have

Comparing it with Eq. (13), we have

From Assumption B.3, we know there exists some constants r k so that δ k = r k h, k = 1 , . . . , n . Thus

And finally, we have

Substitute Eq. (22), Eq. (23) and Eq. (27) into Eq. (21), we can conclude that ˆ x t -x t = O ( h n +2 ) .

B.2.2 Global

First, we provide a lemma that gives the local truncation error given inexact previous values when estimating the high-order derivatives.

Lemma B.7. (Local truncation error with inexact previous values) Suppose inexact values ˆ x λ i k , k = 1 , . . . , n and ˆ x s are used in Eq. (13) to estimate the high-order derivatives, then the local truncation error of the local approximation Eq. (14) satisfies

where ∆ · x · x · , h := λ t λ s

Proof. By replacing x · with ˆ x · in Eq. (13) and subtracting Eq. (12) from Eq. (14), the expression for the local truncation error becomes

And the linear system for solving g ( k ) θ ( x λ s , λ s ) , k = 1 , . . . , n becomes

where ˆ g · = g θ (ˆ x λ · , λ · ) . Under Assumption B.4, we know ˆ g · -g · = O (∆ λ · ) . Thus, under Assumption B.1, Assumption B.2 and Assumption B.3, similar to the deduction in the last section, we have

Besides, under Assumption B.2, the orders of the other coefficients are the same as we obtain in the last section:

Thus

Combining Eq. (29), Eq. (32) and Eq. (33), we can obtain the conclusion in Eq. (28).

Then we prove Theorem 3.3 below.

Proof. (Proof of Theorem 3.3)

As we have discussed, the predictor step from t m -1 to t m is a special case of the local approximation Eq. (14) with ( t i n , . . . , t i 1 , s, t ) = ( t m -n -1 , . . . , t m -2 , t m -1 , t m ) . By Lemma B.7 we have

It follows that there exists constants C, C 0 irrelevant to h , so that

Denote f m := max 0 ≤ i ≤ m | ∆ i | , we then have

Since α t m A ( λ t m -1 ,λ t m ) α t m -1 → 1 when h → 0 and it has bounded first-order derivative due to Assumption B.2, there exists a constant C 2 , so that for any C ≥ C 2 , α t m A ( λ t m -1 ,λ t m ) α t m -1 + Ch > 1 for sufficiently small h . Thus, by taking C 3 = max { C 1 , C 2 } , we have

Denote A m -1 := α t m A ( λ t m -1 ,λ t m ) α t m -1 + C 3 h , by repeating Eq. (37), we have

By Assumption B.5, h = O (1 /M ) , so we have

where σ = max n ≤ i ≤ M -1 α t i -1 C 4 α t i A ( λ t i -1 ,λ t i ) . Then denote β := max n +1 ≤ i ≤ M α t M A ( λ t i ,λ t M ) α t i , we have

Then we substitute Eq. (39) and Eq. (40) into Eq. (38). Note that M = O (1 /h ) by Assumption B.5, and f n = O ( h n +1 ) by Assumption B.6, finally we conclude that | ∆ M | ≤ f M = O ( h n +1 ) .

B.3 Pseudo-Order Solver

First, we provide a lemma that gives the explicit solution to Eq. (15).

The solution to Eq. (15) is

Proof. Denote

̸

Then the solution to Eq. (15) can be expressed as

where ( R -1 k ) kp is the element of R -1 k at the k -th row and the p -th column. From previous studies of the inversion of the Vandermonde matrix [12], we know

̸

Substituting Eq. (44) into Eq. (43), we finish the proof.

̸

Then we prove Theorem 3.4 below:

Proof. (Proof of Theorem 3.4) First, we use mathematical induction to prove that

̸

For k = 1 , Eq. (45) holds by the definition of D ( k ) l . Suppose the equation holds for k , we then prove it holds for k +1 .

Define the Lagrange polynomial which passes ( δ l + p , g i l + p -g i l ) for 0 ≤ p ≤ k :

̸

Then ˜ D ( k ) l = P ( k ) l ( x )[ x k ] is the coefficients before the highest-order term x k in P ( k ) l ( x ) . We then prove that P ( k ) l ( x ) satisfies the following recurrence relation:

By definition, P ( k -1) l +1 ( x ) is the ( k -1) -th order polynomial which passes ( δ l + p , g i l + p -g i l ) for 1 ≤ p ≤ k , and P ( k -1) l ( x ) is the ( k -1) -th order polynomial which passes ( δ l + p , g i l + p -g i l ) for 0 ≤ p ≤ k -1 .

Thus, for 1 ≤ p ≤ k -1 , we have

For p = 0 , we have

for p = k , we have

Therefore, ˜ P ( k ) l ( x ) is the k -th order polynomial which passes k +1 distince points ( δ l + p , g i l + p -g i l ) for 0 ≤ p ≤ k . Due to the uniqueness of the Lagrange polynomial, we can conclude that P ( k ) l ( x ) = ˜ P ( k ) l ( x ) . By taking the coefficients of the highest-order term, we obtain

where by the induction hypothesis we have D ( k -1) l +1 = ˜ D ( k -1) l +1 , D ( k -1) l = ˜ D ( k -1) l . Comparing Eq. (51) with the recurrence relation of D ( k ) l in Eq. (16), it follows that D ( k ) l = ˜ D ( k ) l , which completes the mathematical induction.

Finally, by comparing the expression for ˜ D ( k ) l in Eq. (45) and the expression for ˆ g ( k ) s in Lemma B.8, we can conclude that ˆ g ( k ) s = k ! D ( k ) 0 .

B.4 Local Unbiasedness

Proof. (Proof of Theorem 3.2) Subtracting the local exact solution in Eq. (9) from the ( n +1) -th order local approximation in Eq. (14), we have the local truncation error

where x λ is on the ground-truth ODE trajectory passing x λ s , and ( R -1 n ) kl is the element of the inverse matrix R -1 n at the k -th row and the l -th column, as discussed in the proof of Lemma B.8. By Newton-Leibniz theorem, we have

Also, since x λ i l , l = 1 , . . . , n are on the ground-truth ODE trajectory passing x λ s , we have

where

Note that s λ , l λ are the solution to the least square problem in Eq. (11), which makes sure E p θ τ ( x τ ) [ f (1) θ ( x τ , τ ) -s τ f θ ( x τ , τ ) -b τ ] = 0 . It follows that E p θ λs ( x λs ) [ f (1) θ ( x τ , τ ) -s τ f θ ( x τ , τ ) -b τ ] = 0 , since x τ is on the ground-truth ODE trajectory passing x λ s . Therefore, we have E p θ λs ( x λs ) [ g θ ( x λ , λ ) -g θ ( x λ s , λ s )] = 0 and E p θ λs ( x λs ) [ g θ ( x λ i l , λ i l ) -g θ ( x λ s , λ s ) ] = 0 . Substitute them into Eq. (52), we conclude that E p θ λs ( x λs ) [ ˆ x t -x t ] = 0 .

C Implementation Details

C.1 Computing the EMS and Related Integrals in the ODE Formulation

The ODE formulation and local approximation require computing some complex integrals involving l λ , s λ , b λ . In this section, we'll give details about how to estimate l ∗ λ , s ∗ λ , b ∗ λ on a few datapoints, and how to use them to compute the integrals efficiently.

C.1.1 Computing the EMS

First for the computing of l ∗ λ in Eq. (5), note that

Since diag ( l λ ) is a diagonal matrix, minimizing E p θ λ ( x λ ) [ ∥∇ x N θ ( x λ , λ ) ∥ 2 F ] is equivalent to minimizing E p θ λ ( x λ ) [ ∥ diag -1 ( ∇ x N θ ( x λ , λ )) ∥ 2 2 ] = E p θ λ ( x λ ) [ ∥ diag -1 ( σ λ ∇ x ϵ ( x λ , λ )) -l λ ∥ 2 2 ] , where

diag -1 denotes the operator that takes the diagonal of a matrix as a vector. Thus we have l ∗ λ = E p θ λ ( x λ ) [ diag -1 ( σ λ ∇ x ϵ ( x λ , λ )) ] .

However, this formula for l ∗ λ requires computing the diagonal of the full Jacobian of the noise prediction model, which typically has O ( d 2 ) time complexity for d -dimensional data and is unacceptable when d is large. Fortunately, the cost can be reduced to O ( d ) by utilizing stochastic diagonal estimators and employing the efficient Jacobian-vector-product operator provided by forward-mode automatic differentiation in deep learning frameworks.

For a d -byd matrix D , its diagonal can be unbiasedly estimated by [4]

where v k ∼ p ( v ) are d -dimensional i.i.d. samples with zero mean, ⊙ is the element-wise multiplication i.e., Hadamard product, and ⊘ is the element-wise division. The stochastic diagonal estimator is analogous to the famous Hutchinson's trace estimator [20]. By taking p ( v ) as the Rademacher distribution, we have v k ⊙ v k = 1 , and the denominator can be omitted. For simplicity, we use regular multiplication and division symbols, assuming they are element-wise between vectors. Then l ∗ λ can be expressed as:

which is an unbiased estimation when we replace the expectation with mean on finite samples x λ ∼ p θ λ ( x λ ) , v ∼ p ( v ) . The process for estimating l ∗ λ can easily be paralleled on multiple devices by computing ∑ ( σ λ ∇ x ϵ θ ( x λ , λ ) v ) v on separate datapoints and gather them in the end.

Next, for the computing of s ∗ λ , b ∗ λ in Eq. (11), note that it's a simple least square problem. By taking partial derivatives w.r.t. s λ , b λ and set them to 0, we have

And we obtain the explicit formula for s ∗ λ , b ∗ λ

which are unbiased least square estimators when we replace the expectation with mean on finite samples x λ ∼ p θ λ ( x λ ) . Also, the process for estimating s ∗ λ , b ∗ λ can be paralleled on multiple devices by computing ∑ f θ , ∑ f (1) θ , ∑ f θ f θ , ∑ f θ f (1) θ on separate datapoints and gather them in the end. Thus, the estimation of s ∗ λ , b ∗ λ involving evaluating f θ and f (1) θ on x λ . f θ is a direct transformation of ϵ θ and requires a single forward pass. For f (1) θ , we have

After we obtain l λ , ˙ l λ can be estimated by finite difference. To compute ϵ (1) θ ( x λ , λ ) , we have

which can also be computed with the Jacobian-vector-product operator.

In conclusion, for any λ , l ∗ λ , s ∗ λ , b ∗ λ can be efficiently and unbiasedly estimated by sampling a few datapoints x λ ∼ p θ λ ( x λ ) and using the Jacobian-vector-product.

C.1.2 Integral Precomputing

In the local approximation in Eq. (14), there are three integrals involving the EMS, which are A ( λ s , λ t ) , ∫ λ t λ s E λ s ( λ ) B λ s ( λ )d λ, ∫ λ t λ s E λ s ( λ ) ( λ -λ s ) k k ! d λ . Define the following terms, which are also evaluated at λ j 0 , λ j 1 , . . . , λ j N and can be precomputed in O ( N ) time:

Then for any λ s , λ t , we can verify that the first two integrals can be expressed as

which can be computed in O (1) time. For the third and last integral, denote it as E ( k ) λ s ,λ t , i.e.,

We need to compute it for 0 ≤ k ≤ n and for every local transition time pair ( λ s , λ t ) in the sampling process. For k = 0 , we have

which can also be computed in O (1) time. But for k > 0 , we no longer have such a simplification technique. Still, for any fixed timestep schedule { λ i } i M =0 during the sampling process, we can use a lazy precomputing strategy: compute E ( k ) λ i -1 ,λ i , 1 ≤ i ≤ M when generating the first sample, store it with a unique key ( k, i ) and retrieve it in O (1) in the following sampling process.

C.2 Algorithm

We provide the pseudocode of the local approximation and global solver in Algorithm 1 and Algorithm 2, which concisely describes how we implement DPM-Solver-v3.

Algorithm 1 ( n +1) -th order local approximation: LUpdate n +1

Require: noise schedule α , σ , coefficients l , s , b

Input: transition time pair ( s, t ) , x s , n extra timesteps { t i k } k =1 , g θ values ( g i n , . . . , g i 1 , g s ) at { ( x , t ) } n and ( x , s )

t t λ λ λ n λ i k i k k =1 s

{

t

i

n

,

g

i

n

}

, . . . ,

{

t

i

1

,

g

i

1

}

,

{

s,

x

s

,

g

s

}

, t

Output: ˆ x t

Algorithm 2 ( n +1) -th order multistep predictor-corrector algorithm

̸

Require: noise prediction model ϵ θ , noise schedule α t , σ t , coefficients l λ , s λ , b λ , cache Q 1 , Q 2 Input: timesteps { t i } i M =0 , initial value x 0 1: Q 1 cache ← x 0 2: Q 2 cache ← ϵ θ ( x 0 , t 0 ) 3: for m = 1 to M do 4: n m ← min { n +1 , m } 5: ˆ x m -n m , . . . , ˆ x m -1 fetch ← Q 1 6: ˆ ϵ m -n m , . . . , ˆ ϵ m -1 fetch ← Q 2 7: ˆ g l ← e -∫ λ l λ m -1 s τ d τ σ λ l ˆ ϵ l -l λ l ˆ x l α λ l -∫ λ l λ m -1 e -∫ r λ m -1 s τ d τ b r d r, l = m -n m , . . . , m -1 (Eq. (8)) 8: ˆ x m ← LUpdate n m ( { t m -n m , ˆ g m -n m } , . . . , { t m -2 , ˆ g m -2 } , { t m -1 , ˆ x m -1 , ˆ g m -1 } , t m ) 9: if m = M then 10: ˆ ϵ m ← ϵ θ (ˆ x m , t m ) 11: ˆ g m ← e -∫ λm λ m -1 s τ d τ σ λ m ˆ ϵ m -l λ m ˆ x m α λ m -∫ λ m λ m -1 e -∫ r λ m -1 s τ d τ b r d r (Eq. (8)) 12: ˆ x c m ← LUpdate n m ( { t m -n m +1 , ˆ g m -n m +1 } , . . . , { t m -2 , ˆ g m -2 } , { t m , ˆ g m } , { t m -1 , ˆ x m -1 , ˆ g m -1 } , t m ) 13: ˆ ϵ c m ← ˆ ϵ m + l λ m (ˆ x c m -ˆ x m ) /σ λ m (to ensure ˆ g c m = ˆ g m ) 14: Q 1 cache ← ˆ x c m 15: Q 2 cache ← ˆ ϵ c m 16: end if 17: end for Output: ˆ x M

D Experiment Details

In this section, we provide more experiment details for each setting, including the codebases and the configurations for evaluation, EMS computing and sampling. Unless otherwise stated, we utilize the forward-mode automatic differentiation ( torch.autograd.forward_ad ) provided by PyTorch [39] to compute the Jacobian-vector-products (JVPs). Also, as stated in Section 3.4, we draw datapoints

x λ from the marginal distribution q λ defined by the forward diffusion process starting from some data distribution q 0 , instead of the model distribution p θ λ .

D.1 ScoreSDE on CIFAR10

Codebase and evaluation For unconditional sampling on CIFAR10 [24], one experiment setting is based on the pretrained pixel-space diffusion model provided by ScoreSDE [51]. We use their official codebase of PyTorch implementation, and their checkpoint checkpoint_8.pth under vp/cifar10_ddpmpp_deep_continuous config. We adopt their own statistic file and code for computing FID.

EMS computing We estimate the EMS at N = 1200 uniform timesteps λ j 0 , λ j 1 , . . . , λ j N by drawing K = 4096 datapoints x λ 0 ∼ q 0 , where q 0 is the distribution of the training set. We compute two sets of EMS, corresponding to start time ϵ = 10 -3 (NFE ≤ 10) and ϵ = 10 -4 (NFE>10) in the sampling process respectively. The total time for EMS computing is ∼ 7h on 8 GPU cards of NVIDIA A40.

Sampling Following previous works [31, 32, 58], we use start time ϵ = 10 -3 (NFE ≤ 10) and ϵ = 10 -4 (NFE>10), end time T = 1 and adopt the uniform logSNR timestep schedule. For DPM-Solver-v3, we use the 3rd-order predictor with the 3rd-order corrector by default. In particular, we change to the pseudo 3rd-order predictor at 5 NFE to further boost the performance.

D.2 EDMon CIFAR10

Codebase and evaluation For unconditional sampling on CIFAR10 [24], another experiment setting is based on the pretrained pixel-space diffusion model provided by EDM [21]. We use their official codebase of PyTorch implementation, and their checkpoint edm-cifar10-32x32-uncond-vp.pkl . For consistency, we borrow the statistic file and code from ScoreSDE [51] for computing FID.

EMS computing Since the pretrained models of EDM are stored within the pickles, we fail to use torch.autograd.forward_ad for computing JVPs. Instead, we use torch.autograd.functional.jvp , which is much slower since it employs the double backward trick. We estimate two sets of EMS. One corresponds to N = 1200 uniform timesteps λ j 0 , λ j 1 , . . . , λ j N and K = 1024 datapoints x λ 0 ∼ q 0 , where q 0 is the distribution of the training set. The other corresponds to N = 120 , K = 4096 . They are used when NFE<10 and NFE ≥ 10 respectively. The total time for EMS computing is ∼ 3.5h on 8 GPU cards of NVIDIA A40.

Sampling Following EDM, we use start time t min = 0 . 002 and end time t max = 80 . 0 , but adopt the uniform logSNR timestep schedule which performs better in practice. For DPM-Solver-v3, we use the 3rd-order predictor and additionally employ the 3rd-order corrector when NFE ≤ 6. In particular, we change to the pseudo 3rd-order predictor at 5 NFE to further boost the performance.

D.3 Latent-Diffusion on LSUN-Bedroom

Codebase and evaluation The unconditional sampling on LSUN-Bedroom [55] is based on the pretrained latent-space diffusion model provided by Latent-Diffusion [43]. We use their official codebase of PyTorch implementation and their default checkpoint. We borrow the statistic file and code from Guided-Diffusion [10] for computing FID.

EMScomputing We estimate the EMS at N = 120 uniform timesteps λ j 0 , λ j 1 , . . . , λ j N by drawing K = 1024 datapoints x λ 0 ∼ q 0 , where q 0 is the distribution of the latents of the training set. The total time for EMS computing is ∼ 12min on 8 GPU cards of NVIDIA A40.

Sampling Following previous works [58], we use start time ϵ = 10 -3 , end time T = 1 and adopt the uniform t timestep schedule. For DPM-Solver-v3, we use the 3rd-order predictor with the pseudo 4th-order corrector.

D.4 Guided-Diffusion on ImageNet-256

Codebase and evaluation The conditional sampling on ImageNet-256 [9] is based on the pretrained pixel-space diffusion model provided by Guided-Diffusion [10]. We use their official codebase of PyTorch implementation and their two checkpoints: the conditional diffusion model

256x256_diffusion.pt and the classifier 256x256_classifier.pt . Weadopt their own statistic file and code for computing FID.

EMScomputing We estimate the EMS at N = 500 uniform timesteps λ j 0 , λ j 1 , . . . , λ j N by drawing K = 1024 datapoints x λ 0 ∼ q 0 , where q 0 is the distribution of the training set. Also, we find that the FID metric on the ImageNet-256 dataset behaves specially, and degenerated l λ ( l λ = 1 ) performs better. The total time for EMS computing is ∼ 9.5h on 8 GPU cards of NVIDIA A40.

Sampling Following previous works [31, 32, 58], we use start time ϵ = 10 -3 , end time T = 1 and adopt the uniform t timestep schedule. For DPM-Solver-v3, we use the 2nd-order predictor with the pseudo 3rd-order corrector.

D.5 Stable-Diffusion on MS-COCO2014 prompts

Codebase and evaluation The text-to-image sampling on MS-COCO2014 [26] prompts is based on the pretrained latent-space diffusion model provided by Stable-Diffusion [43]. We use their official codebase of PyTorch implementation and their checkpoint sd-v1-4.ckpt . We compute MSE on randomly selected captions from the MS-COCO2014 validation dataset, as detailed in Section 4.1.

EMScomputing We estimate the EMS at N = 250 uniform timesteps λ j 0 , λ j 1 , . . . , λ j N by drawing K = 1024 datapoints x λ 0 ∼ q 0 . Since Stable-Diffusion is trained on the LAION-5B dataset [46], there is a gap between the images in the MS-COCO2014 validation dataset and the images generated by Stable-Diffusion with certain guidance scale. Thus, we choose q 0 to be the distribution of the latents generated by Stable-Diffusion with corresponding guidance scale, using 200-step DPMSolver++ [32]. We generate these latents with random captions and Gaussian noise different from those we use to compute MSE. The total time for EMS computing is ∼ 11h on 8 GPU cards of NVIDIA A40 for each guidance scale.

Sampling Following previous works [32, 58], we use start time ϵ = 10 -3 , end time T = 1 and adopt the uniform t timestep schedule. For DPM-Solver-v3, we use the 2nd-order predictor with the pseudo 3rd-order corrector.

D.6 License

We list the used datasets, codes and their licenses in Table 3.

E Runtime Comparison

As we have mentioned in Section 4, the runtime of DPM-Solver-v3 is almost the same as other solvers (DDIM [48], DPM-Solver [31], DPM-Solver++ [32], UniPC [58], etc.) as long as they use the same NFE. This is because the main computation costs are the serial evaluations of the large neural network ϵ θ , and the other coefficients are either analytically computed [48, 31, 32, 58], or precomputed (DPM-Solver-v3), thus having neglectable costs.

Table 4 shows the runtime of DPM-Solver-v3 and some other solvers on a single NVIDIA A40 under different settings. We use torch.cuda.Event and torch.cuda.synchronize to accurately compute the runtime. We evaluate the runtime on 8 batches (dropping the first batch since it contains

extra initializations) and report the mean and std. We can see that the runtime is proportional to NFE and has a difference of about ± 1 %for different solvers, which confirms our statement. Therefore, the speedup for the NFE is almost the actual speedup of the runtime.

F Quantitative Results

We present the detailed quantitative results of Section 4.1 for different datasets in Table 1, Table 5, Table 6 and Table 7 respectively. They clearly verify that DPM-Solver-v3 achieves consistently better or comparable performance under various settings, especially in 5 ∼ 10 NFEs.

G Ablations

In this section, we conduct some ablations to further evaluate and analyze the effectiveness of DPM-Solver-v3.

G.1 Varying the Number of Timesteps and Datapoints for the EMS

Table 8: Ablation of the number of timesteps N and datapoints K for the EMS, experimented with ScoreSDE [51] on CIFAR10 [24]. We report the FID ↓ with different numbers of function evaluations (NFE), evaluated on 50k samples.

First, we'd like to investigate how the number of timesteps N and the number of datapoints K for computing the EMS affects the performance. We conduct experiments with the DPM ScoreSDE [51] on CIFAR10 [24], by decreasing N and K from our default choice N = 1200 , K = 4096 .

We list the FID results using the EMS of different N and K in Table 8. We can observe that the number of datapoints K is crucial to the performance, while the number of timesteps N is less significant and affects mainly the performance in 5 ∼ 10 NFEs. When NFE>10, we can decrease N to as little as 50, which gives even better FIDs. Note that the time cost for computing the EMS is proportional to NK , so how to choose appropriate N and K for both efficiency and accuracy is worth studying.

G.2 First-Order Comparison

As stated in Appendix A, the first-order case of DPM-Solver-v3 ( DPM-Solver-v3-1 ) is different from DDIM [48], which is the previous best first-order solver for DPMs. Note that DPM-Solver-v3-1 applies no corrector, since any corrector has an order of at least 2.

In Table 9 and Figure 7, we compare DPM-Solver-v3-1 with DDIM both quantitatively and qualitatively, using the DPM ScoreSDE [51] on CIFAR10 [24]. The results verify our statement that DPM-Solver-v3-1 performs better than DDIM.

Method

NFE

5

6

8

10

12

15

20

DDIM [48]

54.56

41.92

27.51

20.11

15.64

12.05

9.00

DPM-Solver-v3-1

39.18

29.82

20.03

14.98

11.86

9.34

7.19

NFE = 5

NFE = 10

NFE = 20

DDIM

[48]

FID 54.56

FID 20.11

FID 9.00

DPM-Solver-v3-1

(

Ours

)

FID 39.18

FID 14.98

FID 7.19

G.3 Effects of Pseudo-Order Solver

Wenowdemonstrate the effectiveness of the pseudo-order solver, including the pseudo-order predictor and the pseudo-order corrector.

Pseudo-order predictor The pseudo-order predictor is only applied in one case (at 5 NFE on CIFAR10 [24]) to achieve maximum performance improvement. In such cases, without the pseudoorder predictor, the FID results will degenerate from 12.76 to 15.91 for ScoreSDE [51], and from 12.21 to 12.72 for EDM [21]. While they are still better than previous methods, the pseudo-order predictor is proven to further boost the performance at NFEs as small as 5.

Pseudo-order corrector We show the comparison between true and pseudo-order corrector in Table 10. We can observe a consistent improvement when switching to the pseudo-order corrector. Thus, it suggests that if we use n -th order predictor, we'd better combine it with pseudo ( n +1) -th order corrector rather than ( n +1) -th order corrector.

G.4 Effects of Half-Corrector

We demonstrate the effects of half-corrector in Table 11, using the popular Stable-Diffusion model [43]. We can observe that under the relatively large guidance scale of 7.5 which is necessary for producing samples of high quality, the corrector adopted by UniPC [58] has a negative effect on the convergence to the ground-truth samples, making UniPC even worse than DPM-Solver++ [32]. When we employ the half-corrector technique, the problem is partially alleviated. Still, it lags behind our DPM-Solver-v3, since we further incorporate the EMS.

G.5 Singlestep vs. Multistep

As we stated in Section 3.2.2, multistep methods perform better than singlestep methods. To study the relationship between parameterization and these solver types, we develop a singlestep version of DPM-Solver-v3 in Algorithm 3. Note that since ( n +1) -th order singlestep solver divides each step into n +1 substeps, the total number of timesteps is a multiple of n +1 . For flexibility, we follow the adaptive third-order strategy of DPM-Solver [31], which first takes third-order steps and then takes first-order or second-order steps in the end.

Algorithm 3 ( n +1) -th order singlestep solver

̸

Since UniPC [58] is based on a multistep predictor-corrector framework and has no singlestep version, we only compare with DPM-Solver [31] and DPM-Solver++ [32], which uses noise prediction and data prediction respectively. The results of ScoreSDE model [51] on CIFAR10 [24] are shown in Table 12, which demonstrate that different parameterizations have different relative performance under singlestep and multistep methods.

Specifically, for singlestep methods, DPM-Solver-v3 (S) outperforms DPM-Solver++ (S) across NFEs, but DPM-Solver (S) is even better than DPM-Solver-v3 (S) when NFE ≥ 10 (though when NFE<10, DPM-Solver (S) has worst performance), which suggests that noise prediction is best strategy in such scenarios; for multistep methods, we have DPM-Solver-v3 (M) > DPM-Solver++ (M) > DPM-Solver (M) across NFEs. Moreover, DPM-Solver (S) even outperforms DPM-Solver++ (M) when NFE ≥ 20, and the properties of different parameterizations in singlestep methods are left for future study. Overall, DPM-Solver-v3 (M) achieves the best results among all these methods, and performs the most stably under different NFEs.

H FID/CLIP Score on Stable-Diffusion

In text-to-image generation, since the sample quality is affected not only by discretization error of the sampling process, but also by estimation error of neural networks during training, low MSE (faster convergence) does not necessarily imply better sample quality. Therefore, we choose the MSCOCO2014 [26] validation set as the reference, and additionally evaluate DPM-Solver-v3 on Stable-Diffusion model [43] by the standard metrics FID and CLIP score [41] which measure the sample quality and text-image alignment respectively. For DPM-Solver-v3, we use the full-corrector strategy when NFE<10, and no corrector when NFE ≥ 10.

The results in Table 13 show that DPM-Solver-v3 achieves consistently better FID and similar CLIP scores. Notably, we achieve an FID of 15.4 in 8 NFE, close to the reported FID of Stable-Diffusion v1.4.

Still, we claim that FID is not a proper metric for evaluating the convergence of latent-space diffusion models. As stated in DPM-Solver++ and Section 4.1, we can see that the FIDs quickly achieve 15.0 ∼ 16.0 within 10 steps, even if the latent code does not converge, because of the strong image decoder. Instead, MSE in the latent space is a direct way to measure the convergence. By comparing the MSE, our sampler does converge faster to the ground-truth samples of Stable Diffusion itself.

I More Theoretical Analyses

I.1 Expressive Power of Our Generalized Parameterization

Though the introduced coefficients l λ , s λ , b λ seem limited to guarantee the optimality of the parameterization formulation itself, we claim that the generalized parameterization g θ in Eq. (8) can actually cover a wide range of parameterization families in the form of ψ θ ( x λ , λ ) = α ( λ ) ϵ θ ( x λ , λ )+ β ( λ ) x λ + γ ( λ ) . Considering the paramerization on [ λ s , λ t ] , by rearranging the terms, Eq. (8) can be written as

We can compare the coefficients before ϵ θ and x λ in g θ and ψ θ to figure out how l λ , s λ , b λ corresponds to α ( λ ) , β ( λ ) , γ ( λ ) . In fact, we can not directly let g θ equal ψ θ , since when λ = λ s , we have ∫ λ λ s ( · ) = 0 , and the coefficient before ϵ θ in g θ is fixed. Still, ψ θ can be equalized to g θ by a linear transformation, which only depends on λ s and does not affect our analyses of the discretization error and solver.

Specifically, assuming ψ θ = ω λ s g θ + ξ λ s , by corresponding the coefficients we have

̸

Therefore, as long as α ( λ ) = 0 and α ( λ ) , γ ( λ ) have first-order derivatives, our proposed parameterization g θ holds the same expressive power as ψ θ , while at the same time enabling the neat optimality criteria of l λ , s λ , b λ in Eq. (5) and Eq. (11).

I.2 Justification of Why Minimizing First-order Discretization Error Can Help Higher-order Solver

The EMS s ∗ λ , b ∗ λ in Eq. (11) are designed to minimize the first-order discretization error in Eq. (10). However, the high-order solver is actually more frequently adopted in practice (specifically, thirdorder in unconditional sampling, second-order in conditional sampling), since it incurs lower sampling errors by taking higher-order Taylor expansions to approximate the predictor g θ .

In the following, we show that the EMS can also help high-order solver. By Eq. (52) in Appendix B.4, the ( n +1) -th order local error can be expressed as

By Newton-Leibniz theorem, it is equivalent to

We assume that the estimated EMS are bounded (in the order of O (1) , Assumption B.2 in Appendix B.1), which is empirically confirmed as in Section 4.2. By the definition of g θ in Eq. (8), we have g (1) θ ( x τ , τ ) = e -∫ τ λs s r d r ( f (1) θ ( x τ , τ ) -s τ f θ ( x τ , τ ) -b τ ) . Therefore, Eq. (11) controls ∥ g (1) θ ∥ 2 and further controls ∥ ˆ x t -x t ∥ 2 , since other terms are only dependent on the EMS and are bounded.

I.3 The Extra Error of EMS Estimation and Integral Estimation

Analysis of EMS estimation error In practice, the EMS in Eq. (5) and Eq. (11) are estimated on finite datapoints by the explicit expressions in Eq. (58) and Eq. (63), which may differ from the true l ∗ λ , s ∗ λ , b ∗ λ . Theoretically, on one hand, the order and convergence theorems in Section 3.2 are irrelevant to the EMS estimation error: The ODE solution in Eq. (9) is correct whatever l λ , s λ , b λ are, and we only need the assumption that these coefficients are bounded (Assumption B.2 in Appendix B.1) to prove the local and global order; on the other hand, the first-order discretization error in Eq. (10) is vulnerable to the EMS estimation error, which relates to the performance at few steps. Empirically, to enable fast sampling, we need to ensure the number of datapoints for estimating EMS (see ablations in Table 8), and we find that our method is robust to the EMS estimation error given only 1024 datapoints in most cases (see EMS computing configs in Appendix D).

Analysis of integral estimation error Another source of error is the process of integral estimation ( ∫ λ t λ s E λ s ( λ ) B λ s ( λ )d λ and ∫ λ t λ s E λ s ( λ ) ( λ -λ s ) k k ! d λ in Eq. (14)) by trapezoidal rule. We can analyze the estimation error by the error bound formula of trapezoidal rule: suppose we use uniform discretization on [ a, b ] with interval h to estimate ∫ b a f ( x )d x , then the error E satisfies

Under Assumption B.2, the EMS and their first-order derivative are bounded. Denote f 1 ( λ ) = E λ s ( λ ) B λ s ( λ ) , f 2 ( λ ) = E λ s ( λ ) ( λ -λ s ) k k ! , then

Since l λ , s λ , b λ , l ′ λ , s ′ λ , b ′ λ are all O (1) , we can conclude that f ′′ 1 ( λ ) = O ( h ) , f ′′ 2 ( λ ) = O ( h k -2 ) , and the errors of ∫ λ t λ s E λ s ( λ ) B λ s ( λ )d λ, ∫ λ t λ s E λ s ( λ ) ( λ -λ s ) k k ! d λ are O ( h 2 0 h 2 ) , O ( h 2 0 h k -1 ) respectively, where h 0 is the stepsize of EMS discretization ( h 0 = λ t -λ s n , n corresponds to 120 ∼ 1200 timesteps for our EMS computing), and h = λ t -λ s . Therefore, the extra error of integral estimation is under high order and ignorable.

J More Discussions

J.1 Extra Computational and Memory Costs

The extra memory cost of DPM-Solver-v3 is rather small. The extra coefficients l λ , s λ , b λ are discretized and computed at N timesteps, each with a dimension D same as the diffused data. The extra memory cost is O ( ND ) , including the precomputed terms in Appendix C.1.2, and is rather small compared to the pretrained model (e.g. only ∼ 125M in total on Stable-Diffusion, compared to ∼ 4G of the model itself).

The pre-computation time for estimating EMS is rather short. The EMS introduced by our method can be effectively estimated on around 1k datapoints within hours (Appendix D), which is rather short compared to the long training/distillation time of other methods. Moreover, the integrals of these extra coefficients are just some vector constants that can be pre-computed within seconds, as shown in Appendix C.1.2. The precomputing is done only once before sampling.

The extra computational overhead of DPM-Solver-v3 during sampling is negligible. Once we obtain the estimated EMS and their integrals at discrete timesteps, they can be regarded as constants. Thus, during the subsequent sampling process, the computational overhead is the same as previous training-free methods (such as DPM-Solver++) with negligible differences (Appendix E).

-

4

-

2

0

2

4

6

λ

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

s

λ

b

λ

∫

λ

λ

T

s

τ

d

τ

∫

λ

λ

T

b

τ

d

τ

J.2 Flexibility

The pre-computed EMS can be applied for any time schedule during sampling without re-computing EMS. Besides, we compute EMS on unconditional models and it can be used for a wide range of guidance scales (such as cfg=7.5 in Stable Diffusion). In short, EMS is flexible and easy to adopt in downstream applications.

Time schedule: The choice of N,K in EMS is disentangled with the timestep scheduler in sampling. Once we have estimated the EMS at N (e.g., 1200) timesteps, they can be flexibly adapted to any schedule (uniform λ /uniform t ...) during sampling, by corresponding the actual timesteps during sampling to the N bins. For different time schedule, we only need to re-precompute E ( k ) λ s ,λ t in Appendix C.1.2, and the time cost is within seconds.

Guided sampling: We compute the EMS on the unconditional model for all guided cases. Empirically, the EMS computed on the model without guidance (unconditional part) performs more stably than those computed on the model with guidance, and can accelerate the sampling procedure in a wide range of guidance scales (including the common guidance scales used in pretrained models). We think that the unconditional model contains some common information (image priors) for all the conditions, such as color, sketch, and other image patterns. Extracting them helps correct some common biases such as shallow color, low saturation level and lack of details. In contrast, the conditional model is dependent on the condition and has a large variance.

Remark J.1 . EMS computed on models without guidance cannot work for extremely large guidance scales (e.g., cfg scale 15 for Stable Diffusion), since in this case, the condition has a large impact on the denoising process. Note that at these extremely large scales, the sampling quality is very low (compared to cfg scale 7.5) and they are rarely used in practice. Therefore, our proposed EMS without guidance is suitable enough for the common applications with the common guidance.

J.3 Stability

As shown in Section 4.2, the estimated EMS s λ , b λ appear much fluctuating, especially for ScoreSDE on CIFAR10. We would like to clarify that the unstable s λ , b λ is not an issue, and our sampler is stable:

The fluctuation of s λ , b λ on ScoreSDE is intrinsic and not due to the estimation error. As we increase the number of samples to decrease the estimation error, the fluctuation is not reduced. We attribute it to the periodicity of trigonometric functions in the positional timestep embedding as stated in Section 4.2.

Moreover, we only need to consider the integrals of s λ , b λ in the ODE solution Eq. (9). As shown in Figure 8, the integrals of s λ , b λ are rather smooth, which ensures the stability of our method. Therefore, there is no need for extra smoothing.

J.4 Practical Value

When NFE is around 20, our improvement of sample quality is small because all different fast samplers based on diffusion ODEs almost converge. Therefore, what matters is that our method has a faster convergence speed to good sample quality. As we observed, the less diverse the domain is, the more evidently the speed-up takes effect. For example, on LSUN-Bedroom, our method can achieve up to 40% faster convergence.

To sum up, the practical value of DPM-Solver-v3 embodies the following aspects:

15 ∼ 30% speed-up can save lots of costs for online text-to-image applications. Our speedups are applicable to large text-to-image diffusion models, which are an important part of today's AIGC community. As the recent models become larger, a single NFE requires more computational resources, and 15 ∼ 30% speed-up can save a lot of the companies' expenses for commercial usage.

Improvement in 5 ∼ 10 NFEs benefits image previewing. Since the samples are controlled by the random seed, coarse samples with 5 ∼ 10 NFEs can be used to preview thousands of samples with low costs and give guidance on choosing the random seed, which can be then used to generate fine samples with best-quality sampling strategies. This is especially useful for text-to-image generation. Since our method achieves better quality and converges faster to the ground-truth sample, it can provide better guidance when used for preview.

K Additional Samples

We provide more visual samples in Figure 9, Figure 10, Figure 11 and Table 14 to demonstrate the qualitative effectiveness of DPM-Solver-v3. It can be seen that the visual quality of DPM-Solver-v3 outperforms previous state-of-the-art solvers. Our method can generate images that have reduced bias (less 'shallow'), higher saturation level and more visual details, as mentioned in Section 4.3.

NFE = 5

FID 12.76

DPM-Solver++

[32]

FID 28.53

FID 4.01

UniPC

[58]

FID 23.71

FID 3.93

DPM-Solver-v3

(

Ours

)

NFE = 10

FID 3.40

NFE = 5

NFE = 10

Heun's 2nd

[21]

FID 320.80

FID 16.57

DPM-Solver++

[32]

FID 24.54

FID 2.91

UniPC

[58]

FID 23.52

FID 2.85

DPM-Solver-v3

(

Ours

)

FID 12.21

FID 2.51

DPM-Solver++

[32]

(FID 11.02)

UniPC

[58]

(FID 10.19)

DPM-Solver-v3

(

Ours

)

(FID 9.70)

IGFV

Text Prompts

DPM-Solver++

[32]

(MSE 0.60)

UniPC

[58]

(MSE 0.65)

DPM-Solver-v3

(

Ours

)

(MSE 0.55)

'pixar movie still portrait photo of madison beer, jessica alba, woman, as hero catgirl cyborg woman by pixar, by greg rutkowski, wlop, rossdraws, artgerm, weta, marvel, rave girl, leeloo, unreal engine, glossy skin, pearlescent, wet, bright morning, anime, sci-fi, maxim magazine cover'

'oil painting with heavy impasto of a pirate ship and its captain, cosmic

horror painting, elegant intricate artstation concept art by craig mullins

detailed'

'environment living room interior, mid century modern, indoor garden with

fountain, retro, m vintage, designer furniture made of wood and plastic,

concrete table, wood walls, indoor potted tree, large window, outdoor

forest landscape, beautiful sunset, cinematic, concept art, sunstainable

architecture, octane render, utopia, ethereal, cinematic light'

'the living room of a cozy wooden house with a fireplace, at night, interior

design, concept art, wallpaper, warm, digital art. art by james gurney

and larry elmore. '

'Full page concept design how to craft life Poison, intricate details, in- fographic of alchemical, diagram of how to make potions, captions, directions, ingredients, drawing, magic, wuxia'

'Fantasy art, octane render, 16k, 8k, cinema 4d, back-lit, caustics, clean environment, Wood pavilion architecture, warm led lighting, dusk, Land- scape, snow, arctic, with aqua water, silver Guggenheim museum spire, with rays of sunshine, white fabric landscape, tall building, zaha hadid and Santiago calatrava, smooth landscape, cracked ice, igloo, warm lighting, aurora borialis, 3d cgi, high definition, natural lighting, realis- tic, hyper realism'

'tree house in the forest, atmospheric, hyper realistic, epic composition, cinematic, landscape vista photography by Carr Clifton & Galen Rowell, 16K resolution, Landscape veduta photo by Dustin Lefevre & tdraw, de- tailed landscape painting by Ivan Shishkin, DeviantArt, Flickr, rendered in Enscape, Miyazaki, Nausicaa Ghibli, Breath of The Wild, 4k detailed post processing, artstation, unreal engine'

'A trail through the unknown, atmospheric, hyper realistic, 8k, epic com- position, cinematic, octane render, artstation landscape vista photogra- phy by Carr Clifton & Galen Rowell, 16K resolution, Landscape veduta photo by Dustin Lefevre & tdraw, 8k resolution, detailed landscape paint- ing by Ivan Shishkin, DeviantArt, Flickr, rendered in Enscape, Miyazaki, Nausicaa Ghibli, Breath of The Wild, 4k detailed post processing, artsta- tion, rendering by octane, unreal engine'

'postapocalyptic city turned to fractal glass, ctane render, 8 k, exploration, cinematic, trending on artstation, by beeple, realistic, 3 5 mm camera, un- real engine, hyper detailed, photo-realistic maximum detai, volumetric light, moody cinematic epic concept art, realistic matte painting, hyper photorealistic, concept art, cinematic epic, octane render, 8k, corona render, movie concept art, octane render, 8 k, corona render, trending on artstation, cinematic composition, ultra-detailed, hyper-realistic, volumetric lighting'

''WORLDS': zoological fantasy ecosystem infographics, magazine lay- out with typography, annotations, in the style of Elena Masci, Studio Ghibli, Caspar David Friedrich, Daniel Merriam, Doug Chiang, Ivan Aivazovsky, Herbert Bauer, Edward Tufte, David McCandless'

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new sampling algorithm, DPM-Solver-v3, to solve the diffusion ODE. Applying the Rosenbrock-type exponential integrators, DPM-Solver-v3 pre-computes several coefficients (empirical model statistics) to minimize the norm of the gradient of non-linear term, which leads to reduced discretization error. Empirical results of DPM-Solver-v3 demonstrate its effectiveness in unconditional/conditional sampling and classifier-free guidance on Stable Diffusion.\n\nSTRENGTHS:\n- By introducing empirical model statistics arising from Rosenbrock-type exponential integrators, DPM-Solver-v3 has better model parametrization than data/noise prediction ones, which leads to lower discretization error.\n\n- Experiments show that DPM-Solver-v3 outperforms DPM-Solver++ and UniPC for both conditional and unconditional sampling. DPM-Solver-v3 also incurs a smaller MSE on Stable Diffusion, across NFEs.\n\nWEAKNESSES:\n- The sampling algorithm is based on the pre-computed empirical model statistics $l_{\\lambda}, s_{\\lambda}, b_{\\lambda}$ at each time step $\\lambda$, and their corresponding estimated integrals. This seems to lead to extra memory cost and makes the model not able to be flexibly adapted to new use cases (e.g., a different time schedule, guided sampling, etc.)\n\n- When developing the error order and convergence of the algorithm in section 3.2, the estimation error of EMS is not taken into account. However, the unstable $s_{\\lambda}$ w.r.t $\\lambda$ in Figure 5 (especially for the one estimated on ScoreSDE) indicates that the EMS estimation error is not ignorable. Is it possible to provide such analysis or guarantees given the error of EMS?\n\n- In the low FID region (which is closer to practical usage), the benefit of DPM-Solver-v3 seems not very significant compared with UniPC in Figure 3 and Figure 4\n\n- Applying Rosenbrock-type exponential integrators to diffusion ODE is a valid contribution, but other components (e.g. higher-order derivative estimation) of the method seem to be quite relevant to those proposed by existing works like the RK45 Butcher tableau or [1,2]. The real improvement of the Rosenbrock-type technique is unclear when compounded with many other components. Could the authors maybe ablate on its individual effect?\n\n- I think it would be very helpful to compare the FID score / CLIP score on Stable-Diffusion since low MSE does not necessarily imply better sample quality (the quality is affected not only by discretization error but also by estimation error of neural networks).\n\n- Could the authors also compares DPM-Solver-V3 with some typical stochastic samplers, like Alg 2 in EDM, Gonna Go Fast [3], or more recent works on stochastic samplers? Is it possible to integrate the solver in these stochastic samplers, as these samplers seem to provide significantly better sample quality compared to their deterministic counterparts on complex datasets?\n\n\n[1] Fast Sampling of Diffusion Models with Exponential Integrator, Zhang et al., ICML 2023\n\n[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Lu et al., NeurIPS 2022\n\n[3] Gotta go fast when generating data with score-based models.  A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas. CoRR, abs/2105.14080, 2021.\n\nQUESTIONS:\n- The author mentioned that EMS computed on the model without guidance can work for guided sampling cases within a common range of guidance scales. I'm wondering how large is the performance gap between algorithms using EMS computed on the model with guidance and the model without guidance. How is this affected by different magnitudes of guidance scales?\n\n- Theoretically and empirically, is the model robust to estimation error of EMS?\n\n- How is DPM-Solver-v3's performance affected by single-step versus multistep sampling, in comparison with the baselines?\n\n- The estimated $s_{\\lambda}$ seems to be unstable w.r.t $\\lambda$ in some cases. Would using moving average or other smoothing methods help with the sampling algorithm?\n\n- I think it would be very helpful to compare the FID score / CLIP score on Stable-Diffusion since low MSE does not necessarily imply better sample quality (the quality is affected not only by discretization error but also by estimation error of neural networks).\n\n- How well is DPM-Solver-v3 compatible with current SDE samplers?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This work proposes a method to speed up the DPM solver by using empirical model statistics (pre-trained model) as well as employing multistep methods with a predictor-corrector framework for improving sample quality. Promising results were reported.

STRENGTHS:
- This work addresses an important problem of fast sampling in diffusion models without retraining.
- The proposed parametric model is simple and seems neat with optimality criteria.

WEAKNESSES:
- The improvement of the proposed method over prior works such as DPM-solver++ seems incremental. It is unclear if the proposed empirical model statistics is good enough. While this work discusses an optimality under this parametric model, it is unclear if it is truly optimal in terms of modeling itself. This work may also discuss the prior work such as [H Zheng et al., Fast Sampling of Diffusion Models via Operator Learning, ICML 2023] and its arXiv version, which was able to achieve incredible results with only a single model evaluation.
- The experiment results seem to need more improvements and should ensure fairness. For example, Figures 3, 4 indicate that the proposed method is faster than other methods in early iterations, but in order to ensure reasonable FID, one may need a certain number of NFE and in those cases, the differences of FIDs look negligible. Moreover, the comparisons in Fig 6 seems unfair - will other methods achieve similar image quality and FID scores with 15-30% more evaluations? Incorporating the factor of computation should be incorporated for fair comparisons.

QUESTIONS:
Please address the comments in the Weaknesses section.

- Will the speed-up of 15-30% have a practical value? Please discuss.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes DPM-Solver-v3, an ODE solver for diffusion model inference. The method builds upon existing formulations involving exponential integrators, where the linear term of the ODE is canceled and only the noise predictor needs to be approximated. A new model parameterization is derived which involves an optimal set of coefficients, empirical model statistics (EMS). The authors describe methods for computing the EMS in practice, as well as a pseudo high-order method in the few NFE regime and a half-corrector technique when the guidance scale in guided sampling is large.\n\nSTRENGTHS:\n- Although the method builds upon existing methods involving exponential integrators in the diffusion literature, the model parameterization involving EMS is novel.\n- Experiments and ablations are thorough and clearly done. Results are compelling and the proposed method consistently outperforms existing ODE solvers across datasets/models, especially in few NFE regime.\n- Theoretical results for local/global error are clearly presented and assumptions seem reasonable.\n\nWEAKNESSES:\n- In the experiments, the authors mention that \"we tune the strategies of whether to use pseudo-order predictor/corrector at each NFE on CIFAR10\" (lines 254-255). This seems to be an important caveat and should be reiterated in the appendix where the detailed empirical results are presented (Table 4).\n- Similarly, for Stable Diffusion, s=7.5, the authors mention that the best results among no/half/full corrector are reported. This should be reiterated in the appendix in Table 7.\n- In general, I wonder if the presentation of the paper would be more compelling if some of the quantitative results could be moved to the main body from the appendix.\n\nQUESTIONS:\n- FID degenerating from 12.76 to 15.91 for ScoreSDE on CIFAR10 (line 822, G.3) in whether or not to use pseudo-order predictor/corrector seems quite substantial. In which cases is the pseudo-order predictor necessary? It might be compelling to produce a table similar to Table 10 (pseudo-order corrector experiments) examining the influence of the pseudo-order predictor.\n- How does choice of N, K for the EMS change as the timestep scheduler changes?\n\nLIMITATIONS:\nThe authors adequately address the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an improved/generalized version of DPM-solver. It include three more coefficients based on the semi-linear ODE solution proposed by DPM-solver, accounting for minimizing the 'linearity' for the nonlinear component of the ODE and minimizing the first-order discretization error respectively. Such optimal coefficients can be approximately calculated by Monte Carlo samples from the learned model. Together with multistep formulation, predictor-corrector framework by UniPC, and some practical techniques, the paper empirically demonstrated the proposed sample can outperform existing ones especially when NFE is very small.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow. \n- The paper built on DPM-solve formulation and generalized it with additional coefficients that are optimized based on trained model. The added coefficients are intuitive. Theoretical analysis has been carried out to ensure local/global convergence. \n- Two practical techniques has been proposed and seems to be useful in general. \n- Extensive experiments have been performed to justify the effectiveness.\n\nWEAKNESSES:\n- I'm not convinced by optimizing $s_\\tau$ with Eq. (11):\n  - In equation 10, the first coefficient inside the big integration should be $e^{\\int_{\\lambda_s}^\\lambda (l_\\tau + s_\\tau) d\\tau}$ instead of  $e^{\\int_{\\lambda_s}^\\lambda l_\\tau d\\tau}$. Therefore, the first-order discretization error not only depends on $f_\\theta^{(1)} - s_\\lambda f_\\theta - b_\\lambda$. There could be a chance that Eq. (11) is minimized but this term $e^{\\int_{\\lambda_s}^\\lambda (l_\\tau + s_\\tau) d\\tau}$ is amplified. \n  - Main results shown in this paper are not really by first-order solver but by higher-order solvers. A justification of why minimizing first-order discretization error can help higher-order solvers is needed. \n\n- A downside of the introduced coefficients is that the integrals in Eq. (14) become intractable, which has to be approximated by trapezoidal rule. It could introduce extra approximation error and add computational cost. I'd like to see a detailed analysis on the extra error and computational cost. \n\n-  It is understandable and acceptable that this work is heavily built on DPM-solver, but certain sentences in this paper is duplicated from DPM solver. E.g., line 94 our first key insight is..., line 105 our second key insight is ..., line 135-136: the proposed solvers and analysis ....The paper should at least rephrase these sentences. \n\n- The paper can do a better job on clarifying which is the unique contribution of this paper while which part has been proposed by previous works. For example, most techniques and theoretical conclusions in section 3.2 have already been developed in DPM-solver / DPM-solver++ / UniPC, which should be further admitted and clarified.\n\nQUESTIONS:\n- Why EMS without guidance can be directly applied to model with guidance? Does it still hold if the guidance weight is really high?\n- Why for conditional setting, the best setting is 2nd-solver instead of higher-order counterparts?\n\nLIMITATIONS:\nI'd expect to see more discussion on limitations due to the intractable integrals.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}