SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models
Authors: Shuchen Xue
Mingyang Yi
Weijian Luo
Shifeng Zhang
Jiacheng Sun
Zhenguo Li
Zhi-Ming Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model Sampler
Multi-step SDE Solver
TL;DR: We propose a multi-step SDE solver for fast sampling of diffusion models.

Abstract: Diffusion Probabilistic Models (DPMs) have achieved considerable success in generation tasks. As sampling from DPMs is equivalent to solving diffusion SDE or ODE which is time-consuming, numerous fast sampling methods built upon improved differential equation solvers are proposed. The majority of such techniques consider solving the diffusion ODE due to its superior efficiency. However, stochastic sampling could offer additional advantages in generating diverse and high-quality data. In this work, we engage in a comprehensive analysis of stochastic sampling from two aspects: variance-controlled diffusion SDE and linear multi-step SDE solver. Based on our analysis, we propose SA-Solver, which is an improved efficient stochastic Adams method for solving diffusion SDE to generate data with high quality. Our experiments show that SA-Solver achieves: 1) improved or comparable performance compared with the existing state-of-the-art (SOTA) sampling methods for few-step sampling; 2) SOTA FID on substantial benchmark datasets under a suitable number of function evaluations (NFEs).

SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models
Shuchen Xue!: Mingyang Yi Weijian L Zhang? ,Jiacheng Sun?,
Zhenguo Li Zhi-Ming Ma 1.4
University of Chinese Academy of Sciences Huawei Noah's Ark Lab Peking University AAcadmmy of Mathematics and Systems Science, Chinese Academy of Sciences
Abstract
Diffusion Probabilistic Models (DPMs) have achieved considerable success in generation tasks. As sampling from DPMs is equivalent solving diffusion SDE or ODE which is time-consuming, numerous fast sampling methods built upon improved differential equation solvers are proposed. The majority of such techniques consider solving the diffusion ODE due to its superior efficiency. How- ever, stochastic sampling could offer additional advantages in generating diverse and high-quality data. In this work, we engage in a comprehensive analysis of stochastic sampling from two aspects: variance-controlled diffusion SDE and linear multi-step SDE solver. Based improved efficient stochastic Adams method for solving diffusion SDE to generate data with high quality. Our experiments show that SA-Solver achieves: I) improved or comparable performance compared with the existing sate-of-the-ann (SOTA) sampling methods for few-step sampling; 2) SOTA FID substantial benchmark datasets under a suitable number of function evaluations (NFEs).
analysis. we propose SA-Solver, which is an
Introduction
Diffusion Probabilistic Models (DPMs) [1-3] have demonstrated substantial success across broad spectrum of generative tasks such as image synthesis [4-6), video generation [7, 8). text-to-image generation [9-11], speech synthesis [12, 13), etc. The primary mechanism of DPMs involves forward diffusion process that incrementally introduces noise into data. Simultaneously, reverse diffusion process is learned to generate data from this noise. Despite DPMs demonstrating enhanced generation performance in comparison to alternative methods such as Generative Adversarial Net- works (GAN) [14] or Variational Autoencoders (VAE) [15], the sampling process of DPMs demand hundreds of evaluations of network function evaluations (NFE) [2]. The substantial computation requirement poses a significant limitation to their wider application in practice.
The existing literature on improving the sampling efficacy of DPMs can be categorized into two ways, depending on whether conducting extra training on the DPMs. The first category necessitates supplementary training [16-21], which often emerges bottleneck, thereby limiting their practical application. Due to this, we focus on exploring the second category, which consists training-free free methods to improve the sampling efficiency of DPMs in this paper. Current training-free samplers employ efficient numerical schemes to solve the diffusion SDE/ODE[[2-26]] Compared with solving diffusion SDE (stochastic sampler) [25-27], solving diffusion ODE (deterministic sampler) [22-24]
"Work done during internship at Huawei Noah's Ark Lab. Email: xueshuchen1/@mai1. ucas. suoweijian@ste pku. edu. en "Coresponning authors: Mingyang Yi (yiminggangg@@hawei.cco
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

empirically exhibits better sampling efficiency. Existing stochastic samplers typically exhibit slower convergence speed. However, empirical observations [3, 27] indicate that the stochastic sampler has the potential to generate higher-quality data when increasing the sampling steps. This empirical observation motivates us to further explore the efficient stochastic sampler.
Owing to the observed superior performance of stochastic sampler [3, 27], we speculate that adding properly scaled noise in the diffusion SDE may facilitate the quality of generated data. Thus, instead of solving the vanilla diffusion SDE in [22], we propose to consider a family of diffusion SDEs which shares the same marginal distribution [28, 27] with different noise scales. Meanwhile, efficient stochastic solvers are not carefully studied, which could be the reason that diffusion ODE exhibits better sampling efficiency. To overeome this problem, we study the linear multi-step SDE solvers [29] and incorporate them in the sampling.
Based these studies, we propose SA-Solver with theoretical convergence order to solve the proposed diffusion SDEs. Our SA-Solver is based on the stochastic Adams method in [29]. by adapting it to the exponentially weighted integral and analytical variance. With the proposed diffusion SDEs and SA-Solver, we can efficiently generate data with controllable noise scales. We empirically evaluate our SA-Solver on plenty of benchmark datasets of image generation. The evaluation criterion is the Fréchet Inception Distance (FID) score [30] under different number of function evaluations (NFEs). The experimental results can be summarized as three folds: 1) Under small NFEs, our SA-Solver has improved or comparable FID scores, compared with baseline methods; 2) Under suitable NFEs our SA-Solver achieves the State-of-the-Ar FID scores over all benchmark datasets; 3) SA-Solver achieves superior performance over deterministic samplers when the model is not fully trained.
2 Related Works
The DPMs originate from the milestone work [1], and are further developed by [2] and [3] to successfully generate high-quality data, under the framework of discrete and continuous diffusion SDEs respectively. In this paper, we mainly focus on the latter framework. As mentioned in Section 1, plenty of papers are working on accelerating the sampling of DPMs due to their low efficiency, distinguished by whether conducting a supplementary training stage. Training- -based methods, e.g.. knowledge distillation [16-18], learning-to-sample [19], and integration with GANs [20, 21], have the potential to sampling for one or very few steps to enhance the efficiency, but their applicability is limited by the lack of a plug-and-play nature, thereby constraining their broad applicability across diverse tasks. Thus we mainly focus on the training-free methods in this paper.
Solving Diffusion ODE. Since the sampling process equivalent solving diffusion SDE (ODE), the training-free methods are mainly built on solving the differential equations via high-efficiency numerical methods. As ODEs are easier to solve compared with SDEs, the ODE sampler has attracted great attention. For example, Song et al. [22] provides an empirically efficient solver DDIM. Zhang and Chen [28] and Lu et al. [23] point out the remi-linear structure of diffusion ODEs, and develop higher-order ODE samplers based on it. Zhao et al. [24] further improve these samplers in terms of NFEs by integrating the mechanism of prdictor-cooreett method.
Solving Diffusion SDE. Though less explored than the ODE sampler, the SDE sampler exhibits the potential of generating higher-quality data [27]. Thus developing an efficient SDE sampler as we did
in this paper is meaningful topic. In the existing literature, researchers [2, 26. 3] solve the diffusion SDE by first-order discretization numerical method. The higher-order stochastic sampler of diffusion SDE has also been discussed in [25]. Karras et al. [27] proposes another stochastic sampler (which is not a general SDE numerical solver) tailored for diffusion problems. However, in contrast to our proposed SA-Solver, the existing SDE samplers are limited duc to their low efficiency [2, 26, 3] or sensitivity to hyperparameters [27].
We found a concurrent paper proposing an SDE sampler HHE-DPM-Solver+e [31] which is similar to our SA-Solv Though both methods develop multi-step diffusion SDE samplers, our Solver is different from SDE-DPM- DPM-Solver++ as follows: 1) SA-Solver incorporates the predictor-correctt method, which helps improve the quality of generated data [3, 32, 24]; 2) In contrast to SDE- DPM-Solver++, SA-Solver has theoretical guarantees with proved convergence order; 3) SDE-DPM- Solver++ special case of SA-Solver when the predictor step equals 2 with no corrector in our predictor-corrector method, while our solver supports arbitrary orders with analytical forms.

---

3 Preliminary
In the regime of the continuous stochastic differential equation (SDE), Diffusion Probabilistic Models (DPMs) [1-3, 33] construct noisy data through the following linear SDE:
dæ; = f(t) + g(t)dur
(1)
where w, E Rd represents the standard Wiener process, f(t) and g(t) respectively denote the drift and diffusion coefficients. For each time / E [0, (0, aan ~ (a r)
Let (xr)) denotes the marginal distribution of at,, the coefficients f(1) and g(1) meticulously selected to guarantee that the marginal distribution pr (ærr) closely approximates Gaussian distribu- (SNR) aj / strictly decreasing w.r.t. In the sequel, we follow the established notations in [33]:
tion, i.e., (0, I), and the iignal-tonnois--ree
d log at; g2( do dlog at oz.
((1)
dt
Anderson [34] demonstrates a pivotal theorem that the forward process (1) has an equivalent reverse- time diffusion process (from T 0) as the following equation, so that generating process can be equivalent to numerically solve the diffusion SDE [2. 3].
dæ, [f())=; - o² Ve log (xx)) de + g()ddo;
ØT pr(
where W. represents the Wiener process in reverse time, and Ve log Pc(x) the score function.
Moreover, Song et al. [3] also prove that there exists a corresponding deterministic process whose trajectories share the same marginal probability densities pa (xx) as (3), so that the ODE solver can be adopted for efficient sampling [23, 24]:
[())=
1 2 V lagm(a))) dl,
da
oT pT(TT)
(4)
To get the score function s by i to approximate it by optimizing the denoising score matching loss [3]:
Va (e in (3), we usually take neural network se sa(x, t) parameterized
0 arg min E. F2 Fx, lee [|| (x, V log Poi ( (1). am) 1
In practice, two methods are used to reparameterize the score based model [35]. The first approach utilizes noise prediction model such that cα t) - S0 [z, t), while the second employs data prediction model, represented by æg (Zt, I) = (ax; - orE0 (2,, /))/ar. The reparameterized models are plugged into the sampling process (3) or (4) according to their relationship with so (z:, ().
4 Variance Controlled Diffusion SDEs
As mentioned in Section 1, most of the existing training- frren efficient samplers are based on solving diffusion ODE (4), e.g.. [23, 22, 24], because of their improved efficiency compared with the solvers of diffusion SDE (3). However, the empirical observations in [27, 22] exhibit that the quality of data generated by solving diffusion SDE outperforms diffusion ODE given sufficient computational budgets. For example, in [22], the diffusion ODE sampler DDIM [22) significantly improve the FID) score of diffusion SDE sampler DDPM [2] (from 133.37 to 6,84) on CIFAR10 dataset [36] under 20 NFEs. However, under I 1000 NFEs, the DDPM beats the DDIM in terms of FID score (3.17 v.s. 4.04). There may be a trade-off between stochasticity and efficiency. Thus, we conjecture that adding proper scale noise during the generating process may improve the quality of generated data with few NFEs.
In this section, we explore family of variance-cone controlle diffusion SDEs, so that we can use proper noise scales during the sampling stage. Inspired by Proposition 1 in [28] and Eq. (6) in [27], we propose the following proposition to construct the aforementioned diffusion SDEs.
Proposition 4.1. For any bounded measurable function T()) : [0, T] + R, the following Reverse SDEs
dæ; = [[())= 10--1111 (1+r20) ((()) logp((z)) dt + (t)) or pp(TT) (6)
2
share the same marginal probability distributions with (4) and (3).
3

---

The proof can be found in Appendix 1. The proposition indicates that by solving any of the diffusion SDEs (6), we can sample from the target distribution. It is worth noting that the magnitude of noise respectively correspond to the diffusion ODE and SDE in [3]. Thus we can control the magnitdee of added noise during the sampling process by varying
varies with r =t), and (() = 0 or r (())
In practice, we numerically solve the diffusion SDEs (6) by substituting score function V log Pe (xx) in it with the "data prediction reparameterization model" xy (2,,, t) according V log pr (3r,) Rs - (xt a, (2t, t))/o} o as pointed out in Section 3. Then diffusion SDEs to be solved become
[ (1+ (t)
) dt + (t))g())dw
dx: = -[(t) +
20:
Remark 1. We reparameterize the score function in diffusion SDEs (6) with data prediction model (o( 1) to get Eq. (9). The equation can be also reparameterized by the "noise prediction model" co(2), 1) as discussed in Section 3. Though the obtained diffusion ODEs e.g., Eq. (9) are equivalent. the numerical solver applied to them will result in different solutions. For our proposed SA-Solver, we find the diffussoon SDEs reparameterized data prediction model significantly improves the quality of generated data. More details and theoretical explanations are in Sec. 6 and Appendix A.2.4. For the remaining part of the paper, we focus on data reparameterization.
We then solve the difffusion SDEs (9) with change-of-variable applying to it, i.e., changing time variable & to log-SNR A = log (0)/ Noting the following relationship in Eq. (2)
d log a dt
d dlog at 0, = -
ddp
f(t)
g²(t)= = dt dt
(8)
dl
and plugging them into (7), becomes
dæ; = [d at x, - (1 (1+ (())) a, (zr, t))
dhe
dl
dt + o())rr/ -2 -2
dw.
(9)
dt
di
di
The above equation has an explicit solution owing to its semi-lineat structure [37]. Proposition 4.2. Given a, for any time O, the solution æ, at time t [0, s] of (9)
xt - 1: (()dA v. + Fo.Fo oo, G(s,t),
J
Fo(s,t) Same e- S2' (R)dd (1 (1+ xo xo (IXx, d)
(10)
Sx.
G(s,t) [ cc - (())
-2 ddu dw
du
where t) is an Itô integral [38] with the special property
G,, ~ -1 ( ((jaa)).
(11)
The proof can be seen in Appendix A.2.2. With this proposition, we can sample from the diffusion model via numerically solving Eq. (10) starting from 2y approximated by a Gaussian distribution.
5 SA-Solver: Stochastic Adams Method to Solve Diffusion SDEs
Stochastic training-frree samplers for solving diffusion SDEs have not been studied as systematically as their deterministic ODE counterparts. This stems from the inherent challenges associated with designing numerical schemes for SDEs compared to ODEs [39]. Existing stochastic sampling
methods either use only variant of one-step discretization of diffusion SDEs [2, 26, 3]. or are specifically designed sampling procedures for diffusion processes [27] which are not general purpose SDE solvers. Pnlicoeur-Martinee
et al. [25] uses stochastic Improved Euler's 's method [40] with adaptive step sizes. However, it still necessitates hundreds of steps to yield a high-quality sample. As observed by [25], off-tth-shelf SDE solvers are generally ill-suited for diffusion models, often exhibiting inferior qualities or even failing to converge. We postulate that the current dearth of fast stochastic samplers is principally due to factor that existing methodologics predominantly tend to rely

---

Algorithm SA-Solver
==0- Require: data prediction model zo. timesteps (t,) (t=)==o initial value Eio- predictor step Sp, corrector
step Se: buffer B to store former evaluation of z0, rt) control variance.
1: B buffer
De Dg to)
2: for = 1 mar(sp- Sc) do
Warm-up
3: sample ~ (0,
4: calculate steps for warm-up " = min(i, (i, 8p), 8" min(i, 8o)
5: xt; -5p -step SA-Predictor(E:.. B. (Eq. (14))
b Prediction Step
6: B butter
2e 20(2 (x),, t4)
Evaluation Step
7:
Lt. + -seep SA-Corrector(r), tt. B, (Eq. (17))
b Correction Step
8:
for maax(pp $c) to M do
9:
sample I0,
10:
si; f Sy-Step SA-Predictor(o::. B,E) (Eq. (14))
P Prediction Step
buffer 11: B zo xx)
Evaluation Step
12: It, f st-tte SA Aa-Correctoo(l 01. B) (Eq. (17))
Correction Step
return Zty
on one-step discretization its variants, alternatively, on heuristic designs of stochastic samplers. To address this factor, we leverage advanced contemporary tools in numerical solutions for SDEs, specifically, stochastic Adams methods [29]. necessitates fewer evaluations compared Stochastic Runge- Kutta schemes, making it more suitable choice for problems which are computationally expensive- characteristic that diffusion sampling certainly exemplifies.
Next, we formally present our Stochastic Adams Solver (SA-Solver). To solve Eq. (9), we first take M + 1 time steps [t which is strictly decreased from tu T to tM 0.3 Then we can iteratively obtain the It, (so that zy approximates the required data) by the following relationship.
Met1 PQuudd) St + 11,11 Fa(li, 1) + 0 4(1,,4+1) Sx..
Ft;
(12)
As pointed out in Proposition 4.2, the Itô integral term Git,, f++1) in above equation follows Gaussian that can be directly sampled so we need to solve the deterministic integral term Fa(t,, 1++1).
We further combine Eq. (12) with the predictor-corretto method, which is a widely used numerical method. It works in two main steps. First, predictor step is taken make an initial approximation of
the solution. Second, corrector step will refine the predictor's approximation by taking the predicted value into account. It has been proven successful the wide application of numerical analysis [37]. Especially, there are some attempts to use the predictor-corrector method to help sample diffusion models [3, 32, 24). the subsequent Section 5.1 and Section 5.2, we will separately derive our SA-Predicton and SA-Corrector using Eq. (12). Our algorithm is outlined in Algorithm 1.
5.1 SA-Predictor
The fundamental idea behind stochastic Adams methods is to leverage preceding model evaluations
like æ0(#;, 4,), wa (xt, (144-11,.-- L (1) (4-(=1))) These evaluations can be retained with negligible cost implications. Given these preceding model evaluations, natural strategy for
estimating Fe (t,, t441) involves the application of Lagrange interpolations [37] of these evaluations. Lagrange interpolation of s points 2e (x xe ( (4-), xe (2 polynomial L(t) with degrees
1:
L(t) Σ (t) s,ts-)),
(13)
where (t) IR + R is the Lagrange basis. Lagrange interpolation is excellent approximation of Zo (x,, t) with the special property: I(tz-;) = 2g (Zt. (4-5), V S S $ 1. Thus a natural
Thee diffusion SDEs (7) are reverse-time SDEs, so that the here is increased.
5

---

way to estimate Fo f4-1) is to replace (x Xu) with L()), which is just a change-of- of (t). The formula for s-step SA-Predictor is then derived.
s-step SA-Predictor Given the initial value 2t, at time t, total of s former model evaluations 2o 2o (3
1,), 2 (2 44-1). (2 (o-(1- our s-step SA-Predictor is defined as:
Mell P(jja
outi e
Jnn
1t++1
Bt,
+
bi- a ( + o.g. & E~N(00,1; (14)
0t.
5=0
-2 Mett (A)dd where 8; = Ottt1 1-
according to Proposition 4.2 and b,_j is given by:
putti -- - P(jjk (1+ (A)) vo
bi-j = 04441
(1))
(15)
Jas
We show the convergence result in the following theorem. The proof can be found in Appendix B. Theorem 5.1 (Strong Convergence of s-step SA-Predictor). Under mild regularity conditions, our s-step SA- SA-Predictor (Eq. (14)) has a global error in strong convergence sense of O sup +t)) USIST
where h max (ta - t4-1).
5.2 SA-Corrector
Eq. (14) offers "prediction" = that relies information preceding coinciding with the time step since we only use x4 (x f4) along with other model evaluations antecedent to it, while the integration is over time [t,, t.41]- Then predictor-correctorr method can be incorporated to better estimate (o (++1 in Eq. (12). We perform model evaluation æg (a? (441) and construct the Lagrange interpolations of x (=f t++1). xe ( t(), ... xe (t, -w (a-
(x L(t) (t) 4+11 Σ (00 (in.
(16)
where (t=)(t) : + R is the Lagrange basis and can be different with in Eq. (13). The 8-step SA-Co La-Corrector derived by replacing ze ( Au) with ((A) which change-of-variabe of L()..
s-step SA-Corrector Given the initial value oa at time t,, a total of & former model evalu- ations wo (a( (4), (x (i.1)..... x -a 4-(--1)) model evaluation of "prediction"
To 20(2 Exp 14+1). our s-step SA- EA-Corrector is defined as:
Mu+t (A)dā
Ottt e O1,
Dt. + b (xf t+11) + > b [ (zu_) ti-s) 3.6. (17)
jm0
V Ve-e" -2
-25 2 72(j)
where AV(O,)), oz outs
according to Proposition 4.2 and the
coefficients bit1, bi- is given by:
b 0411
e 1
nest (A)dā (1+ (A)) e (A)AA vO
(18)
bi41 =041
J Auts (Ajak (1++2 (2)) 2 (A)i
Jxe,
We show the convergence result the following theorem. The proof can found in Appendix B. Theorem 5.2 (Strong Convergence of s-step SA-C AA-Correctorr. Under mild regularity conditions, $- step SA-Core Corrector (Eq. (17)) has a global error in strong convergence sense of sup -(t)
O5tST
where h = max (t: 1(-1). IS

---

5.3 Connection with other samplers
We briefly discuss the relationship between our SA-Solver and other existing solvers for sampling diffusion ODEs or diffusion SDEs.
Relationship with DDIM [22] DDIM generate samples through the following process:
Et ot. c
14++1 at++1
1V11aaf ai o ED (2 (.) + 5r,5,
(19)
Cig
where & ~ (0, I). ot, is a variable parameter. In practice, DDIM introduces a parameter such that when 0, the sampling process becomes deterministic and when 1, the sampling process
1- 1-02,
(1 (1- (1-aftt) oi
coincides with original DDPM [2]. Specifically, o,, =V
Corollary 5.3 (Relationship with DDIM). For any n in DDIM, there exists TT,(t) : a piecewise constant function such that DDIM-11 coincides with our 1-step SA-Predictor when (tt) = Tr (t) with data parameterization of our variance controll-co diffusion SDE.
- R which
The proof can be found in Appendix B.5.1.
Relationship with (2M))))ver++(2M) [31] DPM-Solver++ a high-order solver which solves diffusion ODEs for guided sampling. DPM- (BMM-Sovvr++((MM is special case of our 2-step SA-P -A-Predictor when r(t) O.
Relationship with UniPC [24] UniPC is a unified predictor-correcto framework for solving diffusion ODEs. UniPC-p is special case of our SA-Solver when rl) = 0 with predictor step p. corrector step in Algorithm 1.
6 Experiments
In this section, we demonstrate the effectiveness of SA-Solver over the existing sampling methods on both a small number of function evaluations (NFEs) settings and considerable number of NFEs settings, with extensive experiments. We use Fenchel Inception Distance (FID) [30] as the evaluation metric show the effectiveness of SA-Solver. Unless otherwise specified, 50K images are sampled for evaluation. The experiments are conducted on various datasets, with image sizes ranging from 32x32 256x256. We also evaluate the performance of various models, including ADM [4], EDM [27], Latent Diffusion [5], and DiT [41].
For of computation, we take 7(1) = 7 as a constant function piecewise constant function. We leave the detailed settings for (1). predictor step, and corrector step in Appendix E. For the following experiments, we first discuss the effectiveness of the data-prediction model. Then evaluate the performance of SA-Solver under different random noise scales T to demonstrate the principles for selecting 7 under few-steps and a considerable number of steps. Finally, we compare SA-Solver with the existing solver to demonstrate its effectiveness.
6.1 Comparison between Data- Inta-Prediction Model and Noise-Prediction Model
We first discuss the necessity of using a data- data-prediction model for SA-Solver. We test on ImageNet 256x256 (latent diffusion model) with 7(t) = 1. Results of the data-predie tata--pediction and noise- prediicrei model are shown in Table It can be seen that the data-prediction model can achieve better sampling quality values under different NFFss, thus we use the data- preitccrreiiiionnn model in the rest of the experiments. More detailed discussions and theoretical analysis can be seen in Appendix A.2.4.
Ablation Study on Predictor/Cooreccor Steps and Predictor-Correctoe Method
To verify the effectiveness of our proposed Stochastic Linear Multi-step Methods and Predictor- Corrector Method, we conduct an ablation study on the CIFARIO dataset as follows. We use EDM [27] baseline- VE pretrained checkpoint. Concretely, vary the number of predictor steps
7

---

Table Compared results by FID + under cntt-prrdicttion and noise prediictnni models, measured by
different NFEs. The latent diffusion model in ImageNet 256x256 is used for evaluation.
NFEs
Noise doise-prediction
Pata-prediction
20
310.5
3.88
40
5.85
3.47
60
3.54
3.41
80
3.41
3.38
Table 2: Compared results by FID under different predictor steps and corrector steps, measured by different NFEs. The VE-baseline model [27] in CIFARIO I0 32x32 is used for evaluation.
method setting (NFE, 7)
15,0.4
23,0.8
31,1.0
47,1 1.
Predictor 1-steps only
13.76
12.44
11.72
14.67
Predictor 1-steps, Corrector 1-step
8.49
6.87
6.13
6.75
Predictor 3-steps only
5.30
3.93
3.52
2.98
Predictor 3-steps, Corrector 3-steps
4.91
3.77
3.40
2.92
and meanwhile conduct them with/without corrector to separately explore the effect of the two components. As can be seen in Table 2, both Stochastic Linear Multi-step Methods (Predictor 1-steps only v.s. Predictor 3-steps only) and Predictor-oorrede Method (Predictor 1-steps only v.s. Predictor 1-steps, Corrector 1-step, and Predictor 3-steps only Predictor 3-steps, Corrector 3-steps) improve the performance of our sampler.
6.3 Effect on Magnitude of Stuchasticity
The proposed SA-Solver is evaluated on various types of datasets and models, including ImageNet 256x256 [43] (latent diffusion model [5]), LSUN Bedroom 256x256 [44] (pixel diffusion model [4]) ImageNet 64x64 (pixel diffusion model [4]), and CIFARIO 32x32 (pixel diffusion model [27]). The models corresponding to these datasets cover pixel-space and latent-space diffusion models, with unconditional, conditional, and classifier-free guidance settings (s 1.5 in ImageNet 256x256).
We used different constant values for SA-Solver, namely {0.0, 0.2, 0.4, ..., 1.6), where larger value of T correspond to larger magnitude of stochasticity. The FID results under different NFE and values are shown in Fig. 1. Note that for LSUN Bedroom, 10K images are sampled for evaluation. The experiments indicate that (I) under relatively small NFEs, smaller nonzero values can achieve better FID results; (2) under considerable number of steps (20-100), large T can achieve better FID. This phenomenon is consistent with the theoretical analysis we conducted in Appendix B and Appendix C, in which the sampling error with stuchasticity is dominated under small NFE, while larger significantly improve the quality of generated samples as the number of steps increases. In subsequent experiments, unless otherwise specified, we will report the results of a proper r()) value. Details can be found in Appendix E.
6.4 Comparison with State-of-the-Art
We compare SA-Solver with existing state-of-the-art sampling methods, including DDIM [22], DPM- Solver [23], UniPC [24], Heun sampler and stochastic sampler in EDM [27]. Unless otherwise specified, the methods are tested using the default pprem-paraeeees in the original papers or code.
Results on CIFAR10 32x32 and ImageNet 64x64. We use the EDM [27] baseline-Vi Ve model for the CIFARIO 10 32x32 experiments and the ADM [4] model for the ImageNet 64x64 experiments. We use EDM's 's timesteps selection for all samplers for fair comparisons. EDM introduces a certain type of SDE and corresponding stochastic sampler, which used for comparison. The experimental results are shown in Fig. 2(a-b). It can be seen that the proposed SA-Solver consistently outperforms other samplers and achieves state-of-the-art FID results. It should be noticed for EDM samplers, we report its optimal result which is searched over four hyper-parameters. In fact, at 95 NFEs, SA-Solver can achieve the best FID value of 2.63 CIFAR1O and 1.81 in ImageNet 64x64 which outperforms all other samplers.

---

Table 3: Sampling quality measured by FID of different sampling methods on DIT, Min-SNR ImageNet [41, 42] models. DiT-XL/2-G and ViT-XL--atch2-32 with = 1.5 are used.
Model
FID (L)
DDPM (NFE=250)
SA-Solver (Ours) (NFE=60)
DIT ImageNet 256x256
2.27
2.02
Heun (NFE=50)
SA-Solver (Ours) (NFE=20)
Min-SNR ImageNet 256x256
2.06
1.93
DDPM (NFE-250) SA-Solver (Ours) (NFE=60)
DiT ImageNet 512x512
3.04
2.80
CIAADD STASA thecl
Imagefilt beab6 thoe p19))
LSUN Bistrom 22txu2e D1901
traggsficl.ccccmmmoocoooooototce 23ttud2 (Laten) DPM)
1.0
6.2
12.
Figure 1: Sampling quality measured by FID + of SA-Solver under different number of function evaluations (NFE), varying the stochastic noise scale T. For LSUN Bedroom, 10K samples are used to evaluate FID.
Results on ImageNet 256x256 and 512x512. We evaluate with two classifier-free guidance models, one is the UNet-based latent diffusion model [5] in which the VQ-4 encoder-decoder model is adopted, and the other is the DiT [41] model using Vision Transformer based model with KL-8 encoder-decoder. The corresponding classifier-free guidance scale, namely s 1.5, is adopted for evaluation. For ImageNet 256x256 dataset with UNet based latent diffusion model, the results of different samplers are shown in Fig. 2(c). Under a considerable number of steps, SA-Solver achieves the best sample quality, in which the FID value is 3.87 with only 20 NFEs and below 3.5 with 40 NFEs or more. While for ODE solvers, the FID values cannot reach below 4, which shows the superiority of the proposed SDE solver.
Table 3 consists results of current SOTA models in ImageNet 256x256 and 512x512. Note that the (Min-SNR) DiT-XL/2-G models are adopted [41, 42]. It can be seen clearly that better FID results are achieved compared with baseline solvers used by corresponding methods. We achieve 1.93 FID value in Min-SNR DiT model at ImageNet 256x256, and 2.80 in DiT model at ImageNet 512x512, both of which are state-of-the-art results under existing DPMs.
Results of text-to-image image generation Fig. 3 shows the qualitative results on text-to-image generation. It can be seen that both UniPC and SA-Solver can generate images with more details. Our SA-Solver is able to generate more reasonable images with better details.
6.5 Effect of Stochasticity for Inaccurate Score Estimation
When the training data is not enough or the computational budget is limited, the estimated score is inaccurate. We empirically observed that the stochasticity significantly improve the sample quality under the circumstance. To further investigate this effect, we reproduce the early training stage of EDM [27] baseline-VE VE model for the CIFARIO 32x32 dataset and DIT-XL/2 [41] model for the ImageNet 256x256 dataset. We compare SA-Solver with different stochastic level r and existing state- of-the-art deterministic sampling methods. We use the same parpmeeprraeters as the corresponding experiment in section 6.4.
Figure 4 shows that SA-Solver outperforms deterministic sampling methods, especially in the early stage of the training process. Moreover, larger 7 value results in better performance. We also conduct thcoretical analysis that stochasticity can mitigate the error of estimation (see Appendix C).

---

CIFARIO (Pixel
ImaageNet 64x64
ImageNet (Latent
15.0
20.0
(ous)
EOM (SDS)
EDM (OOE)
UndPC
DPM Soreer
EDM (5DE) EDN (OB1)
5.0-
D0 Mig E)
12
FID
F10
6.2-
4.0
3--
255
3.5-
NFE
NFE
NFE
Figure 2: Sampling quality measured by FID 1 of different sampling methods of DPMs under different NFEs.
DDIM
UniPC
SA-Solver(Ours)
NFE=20
NFE=50
NFE=20
NPE=50
NFE=20
NPE=50
Figure 3: Qualitative comparisons between our SA- SA-Solver and previous state-of-the methods. All images are generated by Stable Diffusion v1.5 with the same random seed. The main part of the prompt is "portrait of curly orange haired mad scientist man". We set the guidance scale as 7.5. The proposed SA-Solver is able to generate images with more details.
(a) CIFAR (Pixel (NM)
20 ImageNet 256x256 (Latent DPM)(NFE
SA. Salver(t - 0.6) Solverit 1.0
SA-Saver'' 0.4)
SA 50lvertt 0.))
DDM
DRM-Selivr
FOMCDEE
10.
6-
8-
5-
1250
1300
14000
100
150
200
Epoch
Epoch
Figure 4: Sampling quality measured by FID 4 of different sampling methods of DPMs under different training epochs.
7 Conclusions
In this paper, we propose an efficient solver named SA-Solver for solving Diffusion SDEs, achieving high sampling performance in both minimal steps and a suitable number of steps. To better control the scale of injected noise, we propose Variance Controlled Diffusion SDEs based on noise scale function T(t) and propose the analytic form of the SDEs. Based on Variance Controlled Diffusion SDE, we propose SA-Solver, which is derived from the stochastic Adams method and uses exponentially weighted integral and analytical variance to achieve effficient SDE sampling. Meanwhile, SA-Solver has the optimal theoretical convergence bound. Experiments show that SA-Solver achieves state-of- the-art sampling performance in various pre-trained DPMs models. Moreover, SA-Solver achieves superior performance when the score estimation is inaccurate.
Although SA-Solver achieves optimal sampling performance, the noise scale r(l) selection under different NFEs needs further research. The paper proposes empirical criteria for selecting T()), more in-depth theoretical analysis is still needed.
10

---

References
[1] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamies," in International Conference on Machine Learning. PMLR, 2015, pp. 2256-2265.
[2] J. Ho, A. Jain, and P. Abbeel, "Denoising diffiusion probabilistic models," in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 6840-6851.
[3] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations." in International Conference on Learning Representations, 2021.
[4] Dhariwal and A. Q. Nichol, "Diffusion models beat GANs on image synthesis," in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 8780-8794.
[5] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthe- sis with latent diffusion models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 684
[6] J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and Salimans, "Cascaded diffusion models for high fidelity image generation." Journal of Machine Learning Research, vol. 23, no. 47, pp. 1-33, 2022.
[7] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, "Video diffusion models," in Advances in Neural Information Processing Systems, 2022.
[8] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, "Align your latents: High-resolution video synthesis with latent diffusion models," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[9] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, Sutskever, and M. Chen, "GLIDE: towards photorealistic image generation and editing with text- text-guided diffusion models," in International Conference on Machine Learning (ICML), 2022.
[10] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with clip latents," 2022.
[11] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, "Photorealistic text-to-image diffusion models with deep language understanding." " in Advances in Neural Information Processing Systems, 2022.
[12] M. W. Lam, J. Wang, D. Su, and D. Yu, "Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis," in International Conference on Learning Representations, 2022.
[13] K. Song, Y. Leng, X. Tan, Y. Zou, T. Qin, and D. Li, "Transcormer: Transformer for sentence scoring with sliding language modeling," in Advances Neural Information Processing Systems, 2022.
[14] J. J. Goodfellow, J. Pbuuet-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. "Generative adversarial nets," in Advances in Neural Information Processing Systems, vol. 27, 2014, pp. 2672-2680.
[15] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," in International Conference on Learning Representations, 2014.
[16] E. Luhman and T. Luhman, "Knowledge distillation in iterative generative models for improved sampling speed," arXiv preprint arXiv:2/01.02388, 2021.
[17] T. Salimans and J. Ho, "Progressive distillation for fast sampling of diffusion models," in International Conference Learning Representations, 2022.
11

---

[18] C. Meng, R. Gao, D. P. Kingma. S. Ermon, J. Ho, and T. Salimans. "On distillation of guided diffusion models," in NewrIPS 2022 Workshop on soore-Based Methods, 2022.
[19] D. Watson, W. Chan, J. Ho, and M. Norouzi, "Learning fast samplers for diffusion models by dif- ferentiating through sample quality," in International Conference Learning Representations, 2022.
[20] Z. Xiao, K. Kreis, and A. Vahdat, "Tackling the generative learning trilemma with denoising diffusion GANs," in International Conference on Learning Representations, 2022.
[21] Z. Wang, H. Zheng, P. He, W. Chen, and M. Zhou, "Diffusion-GAN: Training GANs with diffusion," in The Eleventh International Conference Learning Representations, 2023.
[22] Song. C. Meng, and S. Ermon, "Denoising diffusion implicit models," in International Conference on Learning Representations, 2021.
[23] C. Lu. Y. Zhou, F. Bao, J. Chen, C. LI. and J. Zhu, 'Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps," in Advances Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35, 2022, pp. 5775-5787.
[24] W. Zhao, L. Bai, Y. Rao. J. Zhou, and J. Lu, "Unipe: A unified predictor-corrector framework for fast sampling of diffusion models," arXiv preprint arXiv:2302.04867. 2023.
[25] A. Marttnna,,,,,,,,u,,, K. Li, R. Piché-Taillefer. T. Kachman, and I. Mitliagkas, "Gotta go fast when generating data with score-based models." arXiv preprint arXiv:2105.14080) 2021.
[26] Bao, C. Li, J. Zhu, and B. Zhang, "Analytic-DPM, An analytic estimate of the optimal reverse variance in diffusion probabilistic models," in International Conference on Learning Representations, 2022.
[27] T. Karras, M. Aittala, T. Aila, and S. Laine, "Elucidating the design space of diffusion based generative models," in Proc. NeurlPS, 2022.
[28] Q. Zhang and Y. Chen, "Fast sampling of diffusion models with exponential integrator," in The Eleventh International Conference on Learning Representations, 2023.
[29] E. Buckwar and R. Winkler, "Multistep methods for sdes and their application to problems with small noise," SIAM Journal on Numerical Analysis, 2006.
[30] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "GANs trained by two time-scale update rule converge to a local Nash equilibrium." in Advances in Neural Information Processing Systems, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., vol. 30, 2017, pp. 6626-6637.
[31] C. Lu, Y. Zhou, Bao, J. Chen, C. Li, and J. Zhu, "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models," 2023.
[32] S. Li, Liu, Z. Chai, R. Li, and Tan, "Era-solver: Error-robust adams solver for fast sampling of diffusion probabilistic models," 2023.
[33] D. P. Kingma, T. Salimans, B. Poole, and J. Ho, "Variational diffusion models," in Advances in Neural Information Processing Systems, 2021.
[34] B. D. Anderson, "Reverse-time diffussion equation models," Stochastic Processes and their Applications, vol. 12. no. 3, pp. 313-326, 1982.
[35] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with CLIP latents," arXiv preprint arXiv:2204.06125 2022.
[36] A. Krizhevsky, "Learning multiple layers of features from tiny images," " Tech. Rep., 2009.
[37] K. Atkinson, W. Han, and D. E. Stewart, Numerical solution of ordinary disferreniaal equations. John Wiley & Sons, 2011, vol. 108.
12

---

[38] B. Oksendal, Stochastic differential equations: an introduction with applications. Science & Business Media, 2013.
Springer
[39] E. Kloeden and E. Platen, Numerical Solution of Stochastic Differential Equations. 1992.
Springer,
[40] A. Roberts, "Modify the improved euler scheme to integrate stochastic differential equations." 2012.
[41] W. Peebles and S. Xie, "Scalable diffusion models with transformers." arXiv preprint arXiv:2212.09748, 2022.
[42] Hang, S. Gu, C. Li, Bao, Chen, H. Hu, X. Geng, and B. Guo, "Efficient diffusion training via min-snr weighting strategy," arXiv preprint arXiv:2303.09556 2023.
[43] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009, pp. 248-255.
[44] F. Yu, Y. Zhang, S. Song, A. Seff, and J. Xiao, "Lsun: Construction of large-scale image dataset using deep learning with humans in the loop." arXiv preprint
2015.
[45] E. Buckwar and R. Winkler, "Improved linear multi-step methods for stochastic ordinary differential equations," " Journal of Computational and Applied Mathematics, 2007.
13

---

A Derivations of Variance Controlled Diffusion SDEs
A.1 Proof of Proposition 4.1
Proposition 4.1 For any bounded measurable function r(t) : :0, [0, + R, the following Reverse
SDEs
dz; = [(t)z; ((1))=,(1117 -1+r2( g log dt + (t))((t)) ær ~ ppp(())) (20)
2
has the same marginal probability distributions with (4) and (3).
Proof. Denote p(æ, 1) R x [0. T] R the probability density function of 2; time 4, thus p(a, T) = pr (2). Fokker-Planck equation [38] determines a Partial Differential Equation (PDE)
that pt (x) satisfies:
(p((a)
0
[
[(0p(to)): - 1 11775 7 -11-2 2
(1)p((a))
1 log Pr (a:
01
Oxg
Ox:
d
d
g2
(()())(()))s)p()))))
2 1=1
dr,ūz,
(21)
f()))1()) - ( (1+r2() g (t)pp(()) Plog
Σ
dz,
2
Ox,
d
2 1=1
Oz;
Eq. (20) is a reverse-time SDE running from T to thus there two additional minus signs in
Eq. (21) before term (p)( and term f(t)p;(z)=;; (1+="(0) g² (t)pe(x)) p((w) compared
with vanilla Fokker-Planck equation in general cases. Here 6ig is the Dirac symbol satisfies 6iz = when i otherwise, dip O. Notice that
a
=- 21))) 2 10700mmmlo p())) + (1))
2 Pl (x
(22)
0z.
Ox,
Substituting Eq. (22) into Eq. (21), we obtain that
(x)
d
(1+r((t) g (t))p((*)
log Pe (x() Ox
f(t)p((z)2;
Ōt
2
1=1
1
θ algg (t)))))
(x:)
Σ
(23)
2
dx,
Σ
d
0
5(()p(()) } (1)p((())))
(logPe(2)
Oz;
which is independent of (l). With the same initial condition p(ar, 7) = (rr),, ) the family of Reverse SDEs in Ey. (20) have exactly the same evolutions of probability density function because they share the same Fokker- Planckk equation. Especially, when r(t) = 0, Eq. (20) degenerates to diffusion ODEs and when r(t) 1, Eq. (20) degenerates to diffusion SDEs.
A.2 Two Reparameterizations and Exact Solution under Exponential Integrator
In this subsection, we will show the exact solution of SDE in both data prediction reparameterization and noise prediction reparameterization. The noise term in data prediction has smaller variance than noise prediction ones, implying the necessity adopting data prediction reparameterization for the SDE sampler.
14

---

A.2.1 Data Prediction Reparameterization
After approximating V. (x with so(x,,t) and reparameterizing so(x,, t) with -(x; 0, o)) Eq. (20) becomes
dæ;- [1()az+ +
21 P ( (1 (t)
) dt dt+ ((t)g(t)d
(24)
20,
0t
Applying change-of- variable with log-SNR X log(as/o1) and substituting the following relation- ship
f(t) d log Oz
do d log ar
=207 dde dt
(²) =
(25)
dt
dt dt
Eq. (24) becomes
[dlogar at
di dt
-2 ddd
dx
=
It - (1+ + (at ax00(211,))
dt + r(t)ou
dw,
dt
dt
[ dlogay an
dde
ddp
(1 (1+ (t))
dt xt + +1)+ (x,, t)
dt
(26)
dt
dt
-2 ddi dw.
+ [t)oo
A.2.2 Proof of Proposition 4.2 Proposition 4.2
Given D, for any time 0, the solution æ, at time t [0, s] of Eq. (9)
Le Shi (A)dR n, + a. Fo + G(,,t).
J2
Fo(s,t)= = fhe c- Sh' ((jjd 72 (2)) e² xo (I A)
(27)
G(s,t) [" Pajja ((u)
dhu du
-2
dw..
where G( is an ltō integral [38] with the special property
o,G(s,t) ((0.0)) -25
P(jaa)).
(28)
Proof. Define y, = fig. ("
(dloga (1++"(0)) d2 )dv
xr, where to E [0, is constant. Differentiate
y, with respect to 1, we get
dy.
[olo aq
(1 (1+
dle dt
1: (dlex oe (w))
dAi )ae de a,
dt
+ S/g( dv
(1)+
dAu de
)dv dar
/dlogar aq
(1++2(1)
dAe
11 (Jlegne (1++>(u))
JAn de
Ide dt
dt
dt
+e- Sig (dlagaa (1))
ddv )av de
dl dlloga (1+)2(1)
dd
dt
dl
dl
+ [/
du
-(1+> 2ou )dv
(1+ arar (zz,, 1)
dd
dl.
(29)
dt
+ [/ [d (1++²(+)) Lay )dv T(()0z
-2 dli
dww; dt
fig (diogas ((tto=((4+2(1))) (1++ س2ي )de )de (1+ a,z0 (I,,
ddp
dt
dt
+ 1// (dlggg (1+7 سhا )du r(t)a
dli
-2 dw..
dt
15

---

Integrating both sides from s to
y: y, / ("
بu )e (1+ (w)) an (E
ddu
(2 u)
du
du
(30)
+ Fig dlggun (1+ حعر da T(u)on, V -2
ddu dww. du
Substituting the definition of ye into Eq. (30), we obtain Eq. (27)
C.ll (dlog
(1+r ( yeu xc
[i [1(dloroe (1++ 2: )d (1+ (ui) lavel(as u) ddu
du
e- [i(( (1 (1++ dAy )d (u))ou -2
dduu
dww
do
du
el'( dloxon (w) ddu )du x.
Si (aldaa - -1) (w) dr (1 + 7² r2()) auxol u)
diu
du
du
1/ [i(dllag 200 )o T())ou -2 du
dlu dw
+
c
0t
5/ (a) ddn du du x2
(31)
e
08
0: e f (w) de (1 + (u)) ( 4)
diu du
Tu
du
a: 7 d2 T(u)ou
ddu
-2 du
dw
Ou
(A)dd
0x
di
E e L (()ji (1 (1)) > no (2 A)dA
+ 0t
)
+ 0 I' c - 7(0) da de (())
din diw.. du
-2
The last term o, S e J (2) de do (w) -2 du dw of Eq. (31) is the Itō integral term. It follows a Gaussian distribution, which can be directly derived from two basic facts [38]: first, the definition of Itō integral is the limitation in L space; second, the limit of Gaussian Process in L2 space is still Gaussian. Then we can compute the mean as:
E σt [ e 18 (v) (v) dex (u))
ddu
2
dw
(32)
du
16

---

and the variance is
ot (" e (1)
th ((u)
ddu d
Var
du
27
1)
=B
01 Ju (1) dr ((1)
2 ddu d du
E
Øt
- (v)) de du =(()) -2 du
ddu dweu
2
-2 adu dwa du
fu (v) dv -(u)/
E
10
=0
1." 6 -2()
a2x du
du [(u)
2 ddu
du
du
=0 2
uhe e (0)
dv
do (1))
(-2 cddu
du
du
=o /
2e JN 27 (A)a) (
(33)
The expectation equals zero because the Itō integral is martingale [38]. The computation of variance uses the Itô Isometry, which crucial fact of Itô integral. We can further simplify the result by using the change of variable P(A) e 2A)A
Var of I' e- (v)
dhu du
20 du (())
-2
dww
2e -0- 10 (A)
oi
(34)
0 .(())) P(X) aP(2)
P(Ar)
P(A)
=o (P(M)) - P(A))
=0 -e- -2 (())a)))
A.2.3 Noise Prediction Reparameterization
After approximating V. (og with s0(2,, and reparameterizing so(x,,t) with -co (x)xt)))/oo Eq. (20) becomes
dæ; = [(0)==++ + (1+7(()) 8² (t) dt + (())
(35)
20:
Applying change-of-variable with log-SNR X = log (0/0///0) and substituting the following relation- ship
S(1) d log ag dt
doł d doogay a 02 -20? dd
(2(1)= dt dt
(36)
dt
Eq. (35) becomes
log ore xt - 1a.-(1+:) + 21)) 0,00 ar, t)
dhe
dd
dx;
dt + +1))a)
-2
dw..
(37)
dt
dt
dt
Eq (35) is the formulation noist prediction model. Similar with Proposition 4.2, Eq. (37) can be solved analytically, which is shown in the following propositon:
17

---

Proposition A.1. Given for any time > 0, the solution 2, at time t E [0, s) of (37) is
3t as Is aeFssss G(,tt,,,
Fo(s.t) (rv e- (1+1 co( ApdY
(38)
G(s,t)= = I @chur(u)y)
-2 ddu dw
du
where G(s,t) is an Itó integral [38] with the special property
a G(s,t) (Gss,t)~NNo,at N( /' 2e 2x, 2
(39)
Proof. Define Y, - -1o d or to t, we get
du
It. where tu [], is constant. Differentiate y, with respect
llo at dt e dig al du
a,dt + -ig dlog dæ:
dy,
d log 0t e -S alag du xidl + e Sig Llgg du d log us
x.dl dt
dt
Sig dlorae do (1+ +1++2 (t)) occo 1) dt
dhi
dt + Sig dlagay de (()))1))
dh,
e
dw,
dt
= =-e -hi dlegev du (1++ 0.
dli
fi aleg dv [(1)oe) dl te
-2 dhi
dww,
di
di
(40)
Integrating both sides from to
y: U f J d alg
dh du
(1+ r1+)((u) on u)
du
(41)
+
pt. Jig du
du T(u)o V
-2 ddu
dwo..
du
Substituting the definition of yr into Eq. (41), we obtain
2t = el algg des
x + e ft dlagau
(1+ ouco(In. 2)
dd du
du
dloxas
do
(u))
-2 ddu du
dwo
(42)
04
ar (1+ (1++2(u)) O.EEl2u u) dda du + I' at d' du
ddu
- x, +
-2
dw
0x au
au
du
at v. + at
(hr c- (1((1+2(())) c0 A)dd + as f' e e-var(ul)
ddu
diiu
0y
du
The Itō integral term e (uu -2 22d dw, follows Gaussian distribution. Following the derivation in Proposition 4.2, the mean of the Itô integral term is:
2 dhu
E 04
e c-dur(u)y
dw
=
(43)
du
18

---

and the expectation is
duu
Var
Ot
e
-2
d
du
21
deou )
dda
ddu
E
Au Tu))
d
E
at
e
e
-2
du
du
27
ddu
E
6
Nu Tu)
dwru
du
(44)
dda
0 EE
e (u)
du
du
-2 dhu
=0
22a (u)du
du
du
2e 22,22 (Ajdd
A.2.4 Comparison between Data and Noise Reparameterizations
In Table perform ablation study on data and noise reparameterizations, the experiment results show that under the same magnitude of stochasticity, the proposed SA-Solver in data reparameter- ization has better convergence which leads to better FID results under the same NFEs. In this subsection, we provide theoretical view of this phenomenon.
Corollary A.2. For any bounded measurable function r(t), the following inequality holds
de
of (1 e1-c-210 -2 +((jaa)
at
2e 2
(45)
Proof. It's equivalent show that
1 - - 222 +())d <
<e2M, 220 (/r, 2E 2e-227((A)dA
(46)
From the basic inequality 1 - - S v, have
> 2 1 +2(d)d
1-e 1- 212 (()dd
(47)
Thus it's sufficient to show that
2e -2 (A)dd 22 //a A
7())d
(48)
2
which is true because
2 (62()1-2)-11 -11 +2 2 0,
(49)
This corollary indicates that the same SDE under two different reparameterizations has different properties under the effect of the exponential integrator. Specifically, in the numerical scheme, the data reparameterization will inject smaller noise each step's updation. We speculate that this the reason that the data reparameterization has a better convergence, shown in Table 1.
19

---

B Derivations and Proofs for SA-Solver
B.1 Preliminary
We will first review some basic concepts and formulas in the numerical solutions of SDEs [39].
Suppose we have an It SDE dæ, f( 1)dt + g ()d,r, and time steps {4,}=1 M 4 E 00. 7 to
numerically solve the SDE. For a random variable Z, we define the In norm |2 L2 = E [Z]] the
Lg norm 1 [
where | the Euclidean norm. Denote h max (t, - 4-11). 15
Definition B.1. We shall say that time-discrete approximation .o. W. where æ, numerical approximation of Lt. converges strongly with order > 0, if there exists positive constant C, which does depend on and ho Ouch that
max | - -iill, S SC 4 S ho- US M
(50)
Definition B.2. say mean-square convergent with order < > if there exists a positive constant C, which does not depend on h and ho 0 such that
max - S SCO, 4h hn-
(51)
USSS M
Remark 2. To prove the strong convergence order 7 of numerical scheme, it's sufficient show the mean-square convergence order 7,. This is from Hölder Inequality E [| - E [1]]]]
| AE |1111
| Thus max OSSSM
|x - -xell ll, S max
Lt. Fiil L2-
0S:SM
We also need the following definition and assumptions, which usually holds in practical diffusion models.
Definition B.3. function Rdd x [0, T] Rdd satisfies linear growth condition if there exists constant K such that
[h(x,t) S K (2
(52)
Assumption B.4. The data prediction model Dg and derivatives such as 0,z0. Vn and Aag satisfy the linear growth condition.
Assumption B.5. The data prediction model ze satisfies uniform Lipschitz condition with respect
to x
[x - xg (I2, t) S L| x22, Yx, E Rd. [0,
(53)
B.2 Outline of the Proof
In the remaining part of this section, we will focus on our variance controlled SDE
dæ. = [( [dlooge a
dhe
dAi
(1)+²(()
(1 (1++2 +2))) au (x,, t)
dt
dl
di
dl
(54)
dl
+ +()oo
-2 -2 dw
dt
Consider the general case of the numerical scheme as follows:
A4t1 (Du)ddd
s-1
x:+1
Ohtt 6
0; +
Σ
b xo (c; (ei-jttt-))
Tt.
1=11
(55)
phil
e l 10:+,?(00) (a)d)
ddu dw...
+ Jttt
-2
du
in which Eq. (17) and Eq (14) are the special case of this scheme. We will provide proof of the mean-square convergence order of the numerical scheme max USISM
|| - x aill We define the local
20

---

enror of the numerical scheme Eq. (55) for the approximation of the SDE Eq. (54) as
M4+1
Ixe,
Lit1 Dt... 31+1
1tas1
It.
b ve (2t (Ii.
Oti
(56)
f Milll
ddu dw..
Sa
-2 (())
((u)
0t+t
du
Li+1 can be decomposed into R,+1 and Si+1- Then the mean-square convergence can be derived, which is summarized in the following theorem proved by [29]:
Theorem B.6 ([29), Theorem 1). The mean-square convergent of I; is bounded by
||Brllz + ||
}
max |__, 02ilL2 S OS M
11 + 0Siis-1
max
max SSISM
h2
(57)
In Eq. (57), D,, i O,... , are the initial error which we do consider. Given Theorem B.6, to show the convergence order O max (t)h + h*) of our s-step SA-Predictor and the convergence
US1ST'
order O max (t)h+h**)) of our s-step SA- SA-Corrector, we just need to prove the following lemmas. 0S1ST
Lemma B.7 (Convergence rate of s-step SA-Predictor). For
A4t1 (Au)dd
Di4t1 = 0u++1 e
Inns
Ct.
b ao f (oc-- ( +āg &~,(0,I),
Tt.
1=0
σ: J6i1 V Vi-e" -2544
Autt ,2(2)d
priss
-
Matt (ao)dd (1++² (Aa)) (ulj-j(udddd vo
b;-j = Jtts Jan
e
(58)
There exists an decomposition of local error L2 such that L, = R4 + S and
|Rll S h ,11 < max T (t)h
(59)
OSTET
Lemma B.8 (Convergence rate of s-step SA SA-Corrector). For
8-1
O1e-1 e
(dd,ddd, Snee
st; + (0 (2. (+11) + Σ ( + Ō
Pt+t1
O2.
1=
0; 01i11 V 1-E -28mg
Mi-1 =2()d
Nosts
Mit (do) (1++²(AN)) exv (w
b,-
ot.t1
e
SSn
vo s-
Nests
dut
bit1
e -San
((c)ddo
(1+ (Xu)) (
Thet1
Axs
(60)
There exists an decomposition of local error L, such that L, = R₁ S, and |alll,, I h ,1S < max
(tt)
(61)
0STET
Lemma B.7 and B.8 will be proved in Sec. B.4.
B.3 Lemmas for the Proof
To better analyze the local error here, we state the following definitions and results from [45]. For continuous function y Rdd x [0, [0, + Rd a general multiple Wiener integral over the subinterval
[t. + h] C [0, T is given by
1---1 y( 83 )dw,, (8z) ... dwr, (81),
Jd.++ Pir2nrr (y) =
(62)
51
21

---

where ri {0, 1.. 1.... d and (s) = ds. Then we have the following lemma.
Lemma B.9 (Bound of Wiener Integral). For any function y TRd x [0, T] Rd that satisfies growth condition in the form y(x t) S ((1 11 + 22) for any a E RRA,, and any E [0. T), such that + h E ; we have that
I
(w))F = is # ffr some E (0...,h).
(63)
1ethh frira-r, (w) = ( -
(64)
where 1 is the number of zero indices and la is the number of non-zero indices TV. Lemma B.10 (Property of Lagrange interpolation polynomial), For s + (l.41, 1141) (li,))), ... (4- (
points
the Lagrange interpolation polynomial
#11
L(t)
Σ
1xt)gkk
(65)
k-i-(s-1)
Then the following 8+ equalities hold
H+1
Σ
lu(u) = 1,
k=i-(s-1)
i11
xx
Σ
(u(u)
duz
duz.
(s--(s-1)
(4-(0-1)
(4-(0-1)
(u(u) f
Σ
du duzduz
ksi-(a-1)
2g
du ... dugduz
(1-te-1) (
(66)
Proof. For the first equality, consider ya = 1 i- is S si+1. The Lagrange interpolation
polynomial for these yks is constant function I(t) 1. We have L(u)
Eit Ek= (e(u) 1.
For the second equality, consider yk mial for these S&S is a polynomial of degree L(t) = t 4-(0-11)- We have L(u) =
_ to 1)
duz-
The Lagrange interpolation polyno-
Ei (u()) Sett dug f4-1(-(--)) Se--a- dug.
For equalities from the third to the last, without loss of generality, we prove the th equality, where < p I Sae Mi rua S (ur du.... ..uuduu.. The Lagrange interpolation polynomial for these
+ 1. Consider yk
yns is a polynomial of degree 1 L(() = - L(u)
fuz (e
... -- " dup ... dugdu2.
We have
-e- ( ... plp-1
It-te- dup ... duyduz
Si-a-n
f S " 11-1 ("p-- - du, ... -ugduz.
B.4 Proof of Lemma B.7 (for Theorem. 5.1) and Lemma B.8 (for Theorem. 5.2)
To simplify the notation, we will introduce two operators which will appear in the Itô formula.
Suppose we have It SDE dæ; = f(x,, t) + g( t)dw, and h(2, t) a twice continuously differentiable function. Let lo(-) 0.(-) + va(.ff. A-) and I2(+) Va(-)9 in which
22

---

Ve is the Jacobian matrix, and A is the Laplacian operator. With the notation here, we can express the Itō formula for h(x, .) as
h( 1) = h) + /. (To(()) dt /- ('2(k))
(67)
Given the above lemmas, we will analyze the local error Li+1 step by step. Inspired by Theorem B.6, for data data-prediction reparameterization model, Li+i can be estimated by decomposing the terms step by step. The first step of decomposition is summarized as the following lemma:
Lemma B.11 (First step of estimating local enror Li+1 in data- data-prediction reparameterization model).
Given the exact solution of data prediction model
0t SN1 x O2
(K)dd Bx + to Do,Fo(s, + (o.Foo(s.))oo
St
Fo(s,t) = pmve e e- 1-120 7(jja (1+ (2)) eº xo (axx, A)
phr
(68)
G(s,t) f.- e c-121 -(jas ((u)
ddu
ddw
-2
du
With proper ba, [i - (s - 1), i + 11. The local error Lit1 in Eq. (56) is
L R8 + +(())
(69)
where
111 =0
max (t)h1
OSIST
Ohn (frttt
e- -MM P(ajaa (1+ cdc
ddu du
Σ
x
du
(s--()-1)
Vrun To(zo)dt
(70)
plan
1eta Pajaa (1 (1+ 6t (f," [["ooao))tt)
ddu du
e- -aaa"
+ Oeet1
du
tx+1
Io(
A Ami-
Proof. The difference between Eq. (55) and Eq. (56) is that æ, is our numerical approximation. while Dty is the exact solution of SDE Eq. (54) at time t = tj. Substitute the exact solution Eq. (31) of Ftar have
Outt
Attt ("(jd Jaes
Et. + Thts (rr e- --1-44 P(a)as 7(a))
2 ddu dw
Li+1
Tt
du
Mutt
Aett aajaa (1 + 72 (Xu)) ed ( du du
Otit1
dit1
(((t) (()))
ddu dw
-tit1
Ju:
e
-2
du
Nottl (22)ddd It. JAn
(71)
01it
Σ bi- ti-;)
Tbi
1it1
6 -man
dett (2)dd (1 + +(()) e² m 1)
dhu du
=0++1
du
bi
23

---

Let f(x, t) = log dz. - +11+ dde
x (1 + +(()) a, 1) -
()())(())))))))))))))))(()) g(t) = -()))//-2200 -24h
dt
dr
By Ito's formula [38], have
12+1 (x-1)
(( (To(20)) (x(o)) dt
2o (D u) = (: (1: +
(72)
+ { (To(x0))+1)) (T(o))
T2(xo)du1+ + / Tz(
(8-1) 1/
i-j-1
Akti ((ll) (To(xe) (Tllo))+Ti(a (xx()))) dt
00( t/-s) 20 t (x-()-))
E
k= (s--(s-1)u1)
(73)
i-j-1
x11
Σ
k=i-(s-1)"
Substituting Eq. (72) and Eq. (73) into Eq. (71), we have
11-1
Otetl
dit1
2()dd (1 + (u]) E San -Sm
ex, ddu du
Σ bi-j
ve
(4-)
14
du
1=-1
i-1 Σ Oeen (frten
e -x A612 =aaad (1 (1+ (1)r cx
ddu
du
du
k=f-(--1)
(1ame) (To(x)) + ((())))
Σ Juet (frtee e- -h hue 72(jax (1 (1+ (1))))))++
cx ddu du du) du
(frma
glett
T2(x0)ddu)
1=--(--1)
1i1l
Neti 7(D)da (1+ e.. (f (To(xo)) + (xxo)) dt
dd du du
-
+ 01,41
E
s
titl
Nit 7(A))d (1+ ex (f. ("
diu du
+ 01.+1
c
du
binj
i-j-1
Σ
J.
1++1
(To(#o)
+I, dt +
8-1-1
Σ
I2(
1x-
(8-1) 1)
(74)
We will divide the local error L,41 into distinct terms. The first term has coefficient
Man 2())x (1 (1+ +())) cv ddu du
Ttal1
bi-g.
(75)
By Lemma 1.10, by constructed by the integral of Lagrange polynomial in Eq. (58) and Eq. (60)
satisfies by = (h) and the coefficient (75) is zero. Furthermore, we have g(l) = =())
-2 at

---

max r(t)
By Lemma B.9, we have the following estimations
0S1ST
-
Aasa A(A)d (1 (1+
et= ddu du x (fr (xx
0412
du
kmi (s-1)
=0
max (())12
OS1ST
i-1
Auti (A)dd (1+ (()) c² dduu du x (Fra (2((z))dū
Σ
e Ixm
O4tt
du
(=--(8-1)
-0
max T(t) OSSSS
Nit1
hest (Ajdd (1 (1))) ~²v (S." Ti (T(o)
ddu du 0
0tit1
JAu
max
?(t)12)
14
du
pltt
Sxu
nuti aajaa (1 (1+)(()) ca (1/ (2(20))duos
ddu du O
max ((1)h3 USEET
0tst1
e
H4
du
i-j-1
tEl1 (r(g)) 0
max ²t()h²
0S1ST
km--((- (s-1)
th
imj=1
le+1 (2(20)
0
max
(t)hh)
OSSSS
k=i-(s-1)
(76)
and the summation of the above terms is (!1) =
max 7t)b6
0S1ST
The remaining terms of local error are
(1)
Tem
e- --mn Aett (aaaa (1 + (w)) cto dia dudd)
(1.me Mx12
To(
141
du
(s--(--1)
plint
e --x
P(j)dA (1 (1+ et (L. To(
ddu du
0b+t1
du
1-1
i-j-1
1x+1
bi-j
To(
1=-1
ksi-(s-1)
1) lx
(77)
which completes the proof.
The remaining problem is to estimate the (11)) in Eq. (70). We can further expand the term (o)o) as following
[o( t)
f" b
(TT(()))))))) di + S" [2Fo(ro)dw
=To(2 (o +
(78)
Substituting the expansion of (x(o)), we perform the approximation of L441, which is summarized with the following lemma:
Lemma 1.12 (Second step of estimating L4t1 data-prediction reparameterization model). R(2
in Eq. (70) can be decomposed
(1)
R
1+1
(79)
25

---

where
1141
s(2) O
max (t)hh3 USSSS
R22)
=04++
Ni1l Autt (aaaA (1++ ex
E Ixm
( du2
ddu
du ti-
du
ditl
e-mam
San P(JJaA (1++((u)) eao (Jrcee f ToFo( (ous: Uz)
uu
ddu du
+0t:1
duyduu
.44
(1-(e-1)
du
s-1
p/a-
Di-j
duz x (2) ((---1))
(0-1
u2
bi-j
(ole (Zug) uz) duzduz
S--(--1)
(80)
Proof. We start with decomposing the term R241
(1)
dill
(1 (1)) c² (1 (2( u2 u2)duz
dd du du
=0t++1
e-
Jt:
Σ
bi-j
(F 12
5=-1
Nitt
Sxm
P(JdA (1 + 22)) ca (Sm duz
ddu du [o( to t)-t)-(--))
=04+1 Hi
du
pltt
aaaad (1 + (2( dto
+ 0 f
e
u2
dhu du du
(ToL'o(x0) + +rTo'(x0)) (æus, us) dugduz
dets (ajaa (1 + (2)) dtv
Sau
+ Oest Ju
(
12
dh du du
P2l'((z)) (æug uz) dwe us duz
4-1
bi-j
duz x To(#)) 6- (opt-----11----0 (0--1))
11 (1---1 be-
(ToLo(oo) + [)Too(z)) (Zags Us) duzduz
u2 de-s
bijj
P2To(#0) (æuy: ua) dii dug-
13 4---1
(81)
26

---

We further estimate the terms with 11 and T2.
lit1
e Sxu
Paydd (u)) e^
16
ddu du du
(iTo( (aug Uz) duzduz/
=0
max
- (())22
USIST
dill
A444 Pa)dA (1+ (u)) ex
0tie1
dAu du
(82)
U (1-(1-1)
(2T'o(00))
(Zusi Uz) duu
du
=0
max T(t)h 0stST
?(())))
bi-j
(oTo(H0) (Iugy us) duyduu =
max
11-(0-1) 41-1-11
(SSST
bi-j (-(--1)
(2T (Zuugy u3) dweng due
O
max T(t)
OSIST
The summation of the above terms is 5(2) =0 be omitted.
max (())h) USIST
Compared with 5(2), this term can
The remaining local error is
e -hha -utt 2())d (1 +1+ +(()) cx (1 duz
dda du
R 0t
du
I (2 ti--)))))))
h1t1 e Sxm
M (())d (1+ coo
+ +t+t
ddu du du
(o)o( (mug: ua) dugduu
(83)
hi-
Σ
b:-j
dug (o(2o)(<1; a i ((---i)))))
4-
22
Σ
bi-j
FoTo((0) (Zag, uz) dugdu2
which completes the proof.
Remark 3. With Lemma B.11 and B.12, the local error Li+1 can be decomposed to the term
S = =1 + 2it and the term R It is clear that S:+1 = O show that given be-j constructed by integral of Lagrange polynomial Eq. (58) and Eq. (60). R O (1²).
max (t)h and we will
OS1ST
27

---

By Lemma B.10, by constructed by the integral of Lagrange polynomial in Eq. (58) and Eq. (60)
satisfies that the coefficient for (z to 15-1 ((--1))
Otet J:
1ill A441 +2Ajd (1+ (1++(u)) e (fr duz
e- Jam
dit
du
Σ
8-1
fi duz.
1 (84)
equals zero. And the remaining term in R is O
Remark 4. We will show that the local ervor can be further decomposed such that L,+1 = R
Σj-1 s Sir In this case Si+1 -i-1 sl3) i+1 is the term such that S.+1
O
USEST (())h)
max
and
we will show that by our constructed bi R O (1++2).
Lemma B.13 (j - th step of estimating Li+1 in data ceata-preiitioo reparameterization model), For
jss+l, R 1-) in Eq. (70) can be decomposed as
R(i-)) RR0) + +S(
(85)
where
500) =
max (t)h 21+1 (ISIST
R())
1it1 Au24 (AJaA (1 (1+ cau ( "
Sx e
122
uuj
ddu du
=0h+11 Ju.
duy ... dugduz
du
To
(o( 4 (4-(111)
fill
Sa
A(A)d) (1 (1+ (u)) ex
+0t++1
Jr:
(man (4 11
ddu du du
1o... (o(20) (Iugei Uj+1) duyti ... duzduz
22
duy dugduz To Tol (x) (x) 1y- ti- (a-(s-1))
(--1-1))
di-j
a2
To...Fo(zg) (Fagsi U541) dus+1 ... duyduz
(86)
Furthermore, given that by is constructed by the integral of Lagrange polynomial in Eq. (58) and Eq. (60), R(d) O11+1)
Sketch of the proof (1) Use the Itô formula Eq. (67) to expand To... To(x)). (2) Use Lemma B.9 to estimate the stochastic term {() For the remaining term RG), by Lemma B. 1010 by constructed by the integral of Lagrange polynomial in Eq. (58) and Eq. (60) satisfies that the coefficient before
28

---

j-1
To... To(aro)( (1(--1))
plitt e- --Saa
M (a)da (1 (2 edu o
u " S duj duyduz
duu du
0t+t1
du
fr " duy duzduz.
bi-s
(87)
equals zero. And the remaining term in R O(h+t)).
The process can be repeated until the coefficient before To ... ('o()z) (o)- (- is
ditl
Jmm
(A)aA
(1 ++)) e I
22
dust1 ..r...-dduuuee duzduz
ddu du du
8-1
S f (1-(--1)
U2
bi-j
dust1 ... duzduz.
(1-(e-1) (
(88)
which equals zero. And the remaining term #+1 is O(h*+2)
We conclude with the proof of Lemma B.7 and B.8.
Proof for Lemma B.8 (Convergence for s-step SA-Corrector) Lemma B.10 prove that with b defined in Theorem 5.2, the coefficients of Eq. (75), Eq. (84), Eq. (87) and Eq. (88) equal zero. Thus the deterministic term R;+1 can be estimated as 22). The proof completed.
The stochastic term S141 can be
estimated as O max
(t)h OSEST
The stochastic term S,+1 can be estimated as O max (t)h from Eq. (76) and Eq. (82). Lemma B.10 prove that with b,- 0SEET
Proof for Lemma B.7 (Convergence for s-step SA-Predictor)
defined in Theorem 5.1, the coefficients of Eq. (75), Eq. (84), Eq. (87) and Eq. (88) equal zero except for the last term. This is because in s-step SA SA-Precictor we only have points in contrast to + points in s-step BA-Corrector, for which we can only obtain the first equalities in Lemma B. 10. Thus the deterministic term R can be estimated as O(1++1). The proof is completed.
B.5
Relationship with Existing Samplers
B.5.1 Relationship with DDIM
DDIM [22] generates samples through the following process:
Cti Ft. c t2)
Pt++1 Oct:+1
+ 11-ai oz - o., ( (.) E.
(89)
Ott
where & ~ (00 I), âr, is variable parameter. In practice, DDIM introduces parameter 7/ such that when = O, the sampling process becomes deterministic and when = 1, the sampling process
1-
of
coincides with original DDPM [2]. Specifically, o,, TV
1-
Corollary 5.3
For any y in DDIM, there exists Tr (1) :R R which is a piecewise constant function such that DDIM-1) coincides with our s-step SA-Predictor when r(1) = TT)( with data parameterization of our variance-controlledd diffusion SDE.
29

---

Proof. Our 1-step SA-Predictor when r(t) = T,t E [t,, with data parameterization of our curiance-controlled diffusion SDE
Ou+t
(A1441 Ac, Bt, + a#t+11 (1 (1- 10 (1+1)) -d1)) xo po((re..1)
Pt;41
Ot.
(90)
+ + - 1--e-22000 -di)g.
DDIM-1 generates samples through the following process
Rt.41 0t. 2o (Tt, t4)+ /1 az
fi1l
"N
2
a
(91)
If substitute or, with oapa V e- 27 -M1411 can verify that v
0p
02
Jun ( -Aa), The DDIM-1) then becomes
F4111 =0t: vo (0t, (2) 02sl - (Airee -Ai) [t, O42 (FL : 14)
Ot
+ V (Nrc -dri)g
Jtit1 e- 72 (Nrrs1 -he; xt. + Oee
Otti Ouin (Neess -dss)) 20 (z),
(92)
E 27 (2eis1 -xoi)
+ +
Ot+t1
e - (Mects -de,) De: asii (1 e (2+17) (2tr) xo( t.)
0ti
0tst1 1-- -27 -..))
which is exactly the same with our 1-step SA-Predictor. To find the Typ we solve the relationship
at ot
ai,
Outs V1 11-c
2rg(A -r.).
(93)
ai.m
The relationship between and is
2r (Yesss -2,))
log
1
-
of
1/2
1
at
of,
1) Ct.
Tn
(94)
-2
(deuts
dc.)
In concurrent paper [31], Lu el al. prove the result that their SDE--DPM-Solver++1 coincides with DDIM with special Their result special case of Corollary 5.3 when = and take special value, while our result holds for arbitrary n.
B.5.2 Relationship with DPM-Solver+++(M)
DPM-Solver++ [31] is a high-order solver which solves diffusion ODEs for guided sampling. DPM- Solver++(2M) is equivalent to 2-step Adams-Bashfone scheme combined with the exponential integrator. While our 2-step SA-Predictor is also equivalent to the 2-step combined with the exponential integrator when (()) 0. Thus (2M) special case of our 2-step SA-Predictor when 7(t) = 0.
scheme
30

---

B.5.3 Relationship with UniPC
UniPC [24] is unified predictor-corrector framework for solving diffusion ODEs. Specifically, scheme combined with the exponential integrator predictor and p-step Adams-Moulton scheme combined with the exponential integrator as corrector. While our p-step SA-Predictor is also equivalent to p-step glams-Bashforth scheme combined with the exponential integrator when r()) = 0 and our p-step SA-Corrector is also equivalent to p-step Adams-Moulton Moulton scheme combined with the exponential integrator when r(1) = Thus UniPC-p is special case of our SA-Solver when 7(t) = 0 with predictor step p, corrector step p in Algorithm 1.
UniPC-p uses a p-step
C Selection on the Magnitude of Stochasticity
In this section, will show that choose rtt) in number of NFEs. We will show that under certain conditions, the upper bound of KI. divergence between the marginal distribution and the true distribution can be minimized when 7t)) = 1.
Let (a)) denotes the marginal distribution of æs, by Proposition 4.1, we know that for any bounded measurable function r(t) []] + I. the following Reverse SDEs
dx. [(()a -
1 + (+t) -0tttttt0000 (² Ve log (1)) dt + (t)g(())d 2r ~ pr(x)), (95)
2
have the same marginal probability distributions. In practice, we substitute Va ((g) with se (2,, t) and substitute pr(TT) wiht ı to sample the reverse SDE.
dæ. [[0s (1 (1+z20 (t) 9² g²(1) (()))s((ef))))
1) at +())g()) s² n;
(96)
2
where π is a known distribution, specifically here a Gaussian. We have the following theorem under the Assumption in Appendix A in [3].
Theorem C.1. Let Po be the data distribution, which is the distribution if we sample from the ground truth reverse SDE (54) time O. Let P' =(1) be the distribution if we sample from the practical reverse SDE (96) at time O. Under the assumptions above, we have
DEL (plle(
<Del (PTl(T)+ + 8 1 (p)(a))
[ (TO)+ = (101 log s0(x, 1)|1 dt.
(97)
This evidence lower bound (ELBO) is minimized when 7 r(1)= =
Proof. Denote the path measure of Eq. (95) and Eq. (96) as and respectively. Both and uniquely determined by the corresponding SDEs due assumptions. Consider a Markov kernel
K ((=))eto.T) v)) 8(20 = y). Thus we have the following result
f K (fæileejaz) .²) djp ( (athheto.)) = po(x))
(98)
f K (10 (=.))e(0 2z) du (m)))))).m)= 7(1) (z))
(99)
By data processing inequality for KL divergence
DEL (pllpo")) = DEL (poliz51)
-DKL (ʃ K ((atheeoor) : 2) du ((xoheejo.1) 1lf K (f=T)eeto.) 2) du (127 relar)))
<DxL (plli))
(100)
31

---

By the chain rule of KL divergence, we have
DEL (plliv) DKL (prlI=) Exapr [DE (u(-12T 2 2)]|u(-|107 x)]]
(101)
By Girsanov Thoerem, DEL (ua(-kET 2)] x-100 =]) can be computed as
DEL
(1( 2) 2 =))
1
B
7(1) +
((() (Va log s0( ()) dw
2
=(1)
T 1 ((()) + r(t) )
(102)
+ E p
(t) logpt(x:) s0(x d)
2 4
1
(T(E)+
Ep (p)(())
T(t)+ + r(t)
( so(2, so(2,,))1 dt
8
D Implementation Details
For our 2-step SA-Predictor and 1-step SA-Corrector, we find that the coefficient will degenerate to simple case.
For 2-step A--Predictor, assume on (+41, r(t) is constant,
/ patte c((+12)A A dd,
bi = - - T Ouip (1 (1+
(103)
de; Ac
b =6 deat2 own (1 (1+72)
deitss
cll++9)a de, dd, Acc- - des
(104)
Aey
we have
Ma41 (A)dā
Obs+1 e
Jay
Pt.
Dt: b, 14) + bi wo i 1:) + ō.g
oti
Utitl e
-
Jan:
M1ll ²((jj
ot; (b: + bi- Jo ( 14)
(105)
Tte
(aoll 14) xo ( G.E.
Let h Aeai
dec we have
{ Aeuss e(2+7))ad
b. + bi = A4p1 0ta (1 (1++2)
= =0 - c1-c -h
(1(++") (1 + h2
b,-1 = O1t111
(1++?) - >x= 1)
(106)
a1+t1
1 1 (1+
(1
No. At
1+
0tt+1
- (1 +1++790 7 22) 0(h3)
1
No. >o;
0144 {1 + + and as at. (1 hh(+++))) - Note Thus we implement b;-1 as Ao, -
that substituting b;, 44-1 as b; bi-1 will maintain the convergence order result of 2-step SA Preddicco since the modified term is (13)). The implementation detail for 1-step SA-Corrector technically the same.
32

---

E Experiment Details
E.1 Details on r(), Predictor Steps and Corrector Steps
CIFARIO 32x32 For the CIFARIO experiment in Section 6.4, we use the pretrained baseline- unconditional-VE from [27]. It's an unconditional model with VE noise schedule. To fairly compare with results in [27], we use a piecewise constant function r(t) inspired by [27]. Concretely, denoting o.DM ax our r(t) is set to be constant T in the interval ((FDM))-((1) and be zero outside the interval. We find empirically that this piecewise constant function setting makes our SA-Solver converge better, especially in large noise scale cases. We use a 3-step SA-Predictor and 3-step SA-Corrector. For the CIFARIO experiment in Section 6.2 and 6.5, we also use the piecewise constant function (t) as above. The predictor steps and corrector steps vary to verify the effectiveness of our proposed method in Section 6.2, while they are both set to be 3-steps in Section 6.5.
ImageNet 64x64 For the ImageNet 64x64 experiment in Section 6.4, we use the pretrained model from [4]. It's conditional model with VP cosine noise schedule. To fairly compare with results in [27], we use piecewise constant function r(t) inspired by [27]. Concretely, denoting of EDDM
our 7()) is set to be constant the interval ((DEDM))
(DED)))((50)) and to be zero outside the interval. We find empirically that this piecewise constant function setting makes our SA-Solver converge better, especially in large noise scale cases. We use a 3-step SA-Predictor and a 3-step SA-Corrector
Other experiments For other experiments, we use constant function =(t) = not the optimal choice for each individual task, thus further fine- finegrrined tuning has the potential to improve the results. We aim to report the result of our SA-Solver without extra hyperparameter tuming. We use a 3-step SA-Predictor and 3-step SA- A--Corecctor under 20 NFEs and 2-step SA- Predictor and 1-step SA-Corrector beyond 20 NFEs.
It's generally
E.2 Details on Pretrained Models and Settings
VE froe [27]. It's an unconditional model with VE noise schedule. To fairly compare with results in [27), we follow the time step schedule it. Specifically, we set Omin = 0.02 and dmaz = 80 and select the step by o, = (o (o min mmz))) for SA-Solver and UniPC. We directly report the results of the deterministic sampler and stochastic sampler of EDM. To make it strong baseline, we report the results of the optimal setting for 4 super-parameters {Sehurns Stmin Stmary Snoise] and report its lowest observed FID. While for SA-Solver and UniPC, we report the averaged observed FID.
CIFARIO 32x32 For the CIFARIO experiment, we use the pretrained
ImageNet 64x64 For the ImageNet 64x64 experiment, we use the pretrained model? from [4]. It's a conditional model with VP cosine noise schedule. To fairly compare with results [27], we follow the time step schedule in it and use conditional sampling. Specifically, we set mmin 0.0064
and max = 80 and select the step by = (omaz + (0 mén -max)) )) for SA-Solver. UniPC, DPM-Solver and DDIM. We directly report the results of the deterministic sampler and stochastic sampler of EDM. To make it strong baseline, we report the results of the optimal setting for hyper-parameters (Schurn, Spmin, Stmaz, Spoise } and report its lowest observed FID. While for SA-Solver, UniPC, DPM-Solver and DDIM, we report the averaged observed FID.
"accpe:/tvvlaa-----cdd.w../ httpa//awwaa.////etttttt//aaaaaa baeeline-ccfarr--2223---moo--- pk1
'bttps:/opeemiipubbi..b.... core. windows. wwwdows.ne//ttiffuueee httplo/ss.t///ffffsssssaa//-///- sisf
core
htttp://avvaab--ff-cccc...d/
baselineecctaarr00-2222---oooo
https::/oppenaipubllcc....L core. windows. wtt.d//www.e////ffffaa/aa//u///a http:////....///.6......fsssao.. diffusion.pt
33

---

ImageNet 256x256 For the ImageNet 256x256 experiment, we use three different pretrained models:LDM handcrafted noise schedule) from [5], DVT-XL/2"(VP, linear noise schedule) from [41], Min-SNR (VP, cosine noise schedule) from [42]. We use classifier-free guidance of scale s 1.5 and uniform time step schedule because it's the most common setting for guided sampling for ImageNet 256x256.
ImageNet 512x512 For the ImageNet 256x256 experiment, we use the pre-trained model: DiT- XI/2 from [41]. We classifier-free guidance of scale 1.5 and uniform time step schedule following the settings of DiT [41].
LSUN Bedroom 256x256 For the LSUN Bedroom 256x256 experiment, we use the pretrained model from [4]. We use unconditional sampling and uniform lambda step schedule from [23].
F
Additional Results
We include the detailed FID results in Figure 1, Figure 2 and Figure 4 in the tables to 14. The ablation study shows that stochasticity indeed helps improve sample quality. We find that for small NFEs, the magnitude of stochasticity should be small while for large NFEs, large magnitude of stochasticity helps improve sample quality. can also be observed that in latent space, SDE converges faster in Table 13. With only 10 NFEs, 0.6 is better than 0. With 20 NFEs, our SA-Solver can achieve 3.87 FID, which outperforms all ODE samplers even with far more steps.
Table 4: Sample quality measured by FID J. on CIFAR10 32x32 dataset (VE-baseline model from [27]) varying the number of function evaluations (NFE). For the results from EDM', we reported its lowest observed FID.
Method NFF
11
15
23
47
63
95
DDIM(1 0)
18.28
12.23
7.93
6.45
5.27
4.83
4.42
DPM-Solver
9.26
5.13
4.52
4.30
4.02
3.97
3.94
UniPC
6.42
5.02
4.19
4.00
3.91
3.90
3.89
EDM(ODE)
13.46
5.62
4.04
3.82
3.79
3.80
3.79
EDM(SDE)
23.94
8.94
4.73
3.95
3.59
3.36
3.06
SA-Solver
6.46
4.91
3.77
3.40
2.92
2.74
2.63
Table 5: Sample quality measured by FID 1. on CIFAR1O 32x32 dataset (VE- baseline model from [27])
varying the number of function evaluations (NFE) and the magnitude of stochasticity (T).
SA-Solver
NFE
11
15
23
31
47
63
95
0.0
6.46
5,06
4.22
4.02
3.93
3.92
3,91
T = 0.2
6.54
5.01
4.14
3.95
3.89
3.84
3.83
0.4
6.79
4.91
4.03
3.81
3.76
3.74
3.67
T 0.6
7.34
4.91
3.85
3.65
3.60
3.56
3.57
0.8
8.61
5.28
3.77
3.48
3.45
3.43
3.50
T = 1.0
10.89
6.52
3.98
3.40
3.21
3.25
3.29
1.2
14.49
9.33
5.19
3.69
3.00
3.03
3.07
1.4
20.19
13.76
7.60
4.91
2.92
2.86
2.93
1.6
27.90
20.51
11.89
8.07
3.25
2.74
2.80
1.8
36.26
29.43
18.13
14.00
4.60
2.83
2.63
https//ss..aaapppc..c..cco/o/aes 0/ema_0. w99.9xxx9pxpp.. /ema___99999_x1.ptt
Phttps::/oppnaappbbbl blob. core. vindows wtt.::/wwww..tt///ddaauo/oooooos -0221/1/smmmmmmmm bedroom.
34

---

Table Sample quality measured by FID &. on ImageNet 64x64 dataset (model from [4]) varying the number of function evaluations (NFE). For the results from EDMT, we reported its lowest observed FID.
Method \ NFE
15
23
31
47
63
95
(D = 0)
8.48
5.39
4.27
3.46
3.17
2.95
DPM-Solver
3.49
3.04
2.88
2.80
2.76
2.74
UniPC
3.51
2.84
2.75
2.72
2.71
2.72
EDM(ODE)
4.78
3.12
2.84
2.73
2.73
2.67
EDM(SDE)
8.94
4.30
3.40
2.72
2.44
2.22
SA-Solver
3.41
2.61
2.23
1.95
1.88
1.81
Table 7: Sample quality measured by FII J. on ImageNet 64x64 dataset (model from [4]) varying the
number of function evaluations (NFE) and the magnitude of stochasticity (r).
SA-Solver NFE
15
23
31
47
63
95
0.0
3.48
2.72
2.72
2.66
2.64
2.71
T = 0.2
3.41
2.80
2.63
2.63
2.64
2.60
0.4
3.52
2.70
2.51
2.51
2.49
2.49
T = 0.6
3.98
2.61
2.44
2.39
2.34
2.35
0.8
5.80
2.68
2.32
2.24
2.19
2.21
= 1.0
10.06
3.38
2.23
2.09
2.08
2.08
1.2
18.39
5.52
2.52
1.95
1.97
2.00
= 1.4
32.42
10.37
3.83
2.05
1.89
1.89
1.6
52.31
19.64
7.10
2.60
1.88
1.81
G
Additional Samples
We include additional samples in this section. In Figure 5 and Figure we compare samples of our proposed SA-Solver with other diffusion samplers. In Figure 7 and Figure 8, we compare samples of our proposed SA-Solver under different NFEs and т. In Figure and Figure 10, we compare samples of our proposed SA-Solver with other diffusion samplers on text-to- to-- tasks. Our SA-Solver can generate more diverse samples with more details.
35

---

Table 8: Sample quality measured by FID & on CIFARIO I0 32x32 dataset (model trained by ourselves; see Section 6.5) varying the sampling method and the training epoch.
method (NFE = 31) \ epoch
1250
1300
1350
1400
1450
1500
DDIM
39.32
29.79
19.59
12.98
8.63
6.64
DPM-Solver
30.57
22.11
13.85
8.85
5.68
4.55
EDM(ODE)
27.51
19.82
12.37
8.03
5.33
4.32
SA-Solver(T = =0.6)
20.55
14.89
9.71
6.55
4.61
4.08
SA-Solver( =1.0)
13.62
10.01
6.79
4.81
3.70
3.47
Table 9: Sample quality measured by FID 1 on ImageNet 256x256 dataset (model trained by ourselves; Section 6.5) varying the sampling method and the training epoch.
method (NFE = 40) \ epoch
50
100
150
200
250
DDIM
19.40
9.61
6.75
5.86
5.12
DPM-Solver
18.75
8.96
6.15
5.28
4.62
SA-Solver(T 0.4)
17.93
8.39
5.69
4.84
4.24
SA-Solver(T = 0.8)
16.57
7.54
5.15
4.48
3.99
Table 10:
Sample quality measured by FID 4 on ImageNet 256x256 dataset(model from [5]) varying
the number of function evaluations (NFE).
Method \ NFE
5
10
20
40
60
80
100
DDIM(1 = 0)
58.68
16.32
6.82
4.71
4.45
4.28
4.23
DPM-Solver
166.88
6.19
5.51
4.17
4.18
4.21
4.15
UniPC
12.79
4.96
4.21
4.14
4.12
4.09
4.10
DDIM(1 1)
138.91
50.05
14.60
6.09
4.56
4.12
3.87
SA-Solver
11.46
4.82
3.88
3.47
3.37
3.37
3.33
Table 11: Ablation study on the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID on CIFARIO 32x32 dataset(model from (27]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(r).
NFE
23
31
47
63
127
0
4.84
4.11
3.94
3.86
3.88
3.87
3,87
0.2
4.96
4.04
3.84
3.75
3.74
3.79
3.75
0.4
5.27
4.00
3.87
3.64
3.71
3.70
3.62
0.6
6.05
3.95
3.61
3.49
3.46
3.53
3.43
0.8
7.40
4.12
3.53
3.28
3.37
3.32
3.30
1.0
10.00
4.49
3.41
3.18
3.24
3.17
3.15
1.2
13.58
5.14
3.59
3.10
3.02
2.97
3.05
1.4
17.88
6.55
3.94
3.04
3.01
2.89
2.95
1.6
22.42
8.44
4.69
3.20
3.02
2.94
2.89
36

---

Table 12: Ablation study on the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID & ImageNet 64x64 dataset(model from [4]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(T).
T NFE
20
40
60
80
100
0
3.30
2.83
2.78
2.79
2.82
0.2
3.32
2.77
2.72
2.74
2.79
0.4
3.37
2.68
2.63
2.62
2.59
0.6
3.61
2.57
2.49
2.49
2.47
0.8
4.19
2.51
2.40
2.34
2.30
1.0
5.55
2.54
2.32
2.21
2.20
1.2
7.93
2.77
2.29
2.14
2.14
1.4
11.55
3.20
2.40
2.14
2.08
1.6
16.15
3.97
2.60
2.20
2.09
Table 13: Ablation study the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID L on ImageNet 256x 256 doteset(model from [5]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(r).
NFE
10
20
40
60
80
100
0
11.46
5.04
4.30
4.16
4.12
4.10
4.16
0.2
11.88
4.89
4.29
4.05
4.02
4.01
4.03
0.4
12.69
4.84
4.14
3.86
3.84
3.83
3.84
0.6
14.84
4.82
3.99
3.63
3.62
3.63
3.61
0.8
18.82
5.09
3.87
3.55
3.50
3.47
3.47
1.0
25.96
6.06
3.88
3.47
3.41
3.39
3.38
1.2
37.20
8.23
3.92
3.47
3.37
3.37
3.33
1.4
53.03
12.93
4.08
3.53
3.40
3.38
3.36
1.6
71.30
24.08
4.43
3.56
3.44
3.45
3.33
Table 14: Ablation study on the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID on LSUN Bedroom 256x256 dataset(model from [4]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(t).
T NFE
20
40
60
80
100
0
3.60
3.14
3.06
3,09
3.07
0.2
3.51
3.12
3.00
2.99
2.99
0.4
3.70
3.09
2.97
3.03
3.16
0.6
4.10
3.08
2.95
2.99
3.03
0.8
4.75
3.11
2.97
2.89
2.99
1.0
6.18
3.28
2.98
2.90
2.91
1.2
8.54
3.53
3.12
2.86
3.00
12.14
4.25
3.24
2.98
2.93
1.6
16.63
5.50
3.75
3.18
3.10
37

---

NFE 15
NFE 23
NFE 47
NFE
DDIM(1 0)
DPM-Solver
UniPC
EDM(ODE)
EDM(SDE)
SA-Solver(ours)
Figure 5: Samples by DDIM, DPM-Solver, UniPC, EDM(ODE), EDM(SDE) and SA-Solver
with 15, 23, 47, 95 NFEs with the same random seed from CIFAR1O 32x32 VE baseline model [27]
38

---

NFE- 15
NFE 23
NFE = 47
NFE = 95
DDIM(1 0)
DPM-Solver
UniPC
SA-Solver(ours)
Figure Samples by DDIM, DPM-Solver, UniPC, and our SA-Solver with 15, 23, 47, NFEs with the same random seed from ImageNet 64x64 model [condcintill sampling)
39

---

NFE 20
NFE 40
NFE 60
NFE 100
SA-Solver(n 0)
SA-Solver(y = 0.2)
SA-Solver(n 0.4)
SA-Solver(1 = 0.6)
SA-Solver(y 0.8)
SA-Solver(y 1.0)
SA-Solver(n = 1.2)
Figure 7: Samples by SA-Solver with 20, 40, 60, 100 NFEs varying stochasticity(r) with the same random seed from L.SUN-Bedrom 256x256 model [4][unconditional sampling).
40

---

NFE = 60
SA-Solver(T = 0)
SA-Solver(T = 0.4)
SA-Solver(T = 0.8)
SA-Solver(T = 1.0)
Figure 8: Samples by SA-Solver with 60 NFEs varying stochasticity(r) with the same random seed
from ImageNet 512x512 DiT model [41] with classifer-free guidance scale s = 4.0(default setting to show image).
41

---

NFE 20
NFE = 50
NFE = 100
DDIM(1 = 0)
UniPC
SA-Solver(Ours)
Figure 9: Samples using Stable-Diffusion v1.5 [5] with classifier-free guidance scale 7.5 with different solvers and NFEs. Prompt:The Legend of Zelda landscape atmospheric, hyper realistic, 8k, epic composition, cinematic, octane render, artstation landscape vista photography by Carr Clifton Galen Rowell, 16K resolution, Landscape veduta photo by Dustin Lefevre tdraw, 8k resolution, detailed landscape painting by Ivan Shishkin, DeviantArt, Flickr, rendered in Enscape, Miyazaki, Nausicaa Ghibli, Breath The Wild, 4k detailed post processing, artstation, rendering by octane, unreal engine.
42

---

NFE 20
NFE =
NFE 100
DDIM(1) 0)
UniPC
SA-Solver(Ours)
Figure 10: Samples using Stable-Diffusiie v1.5 [5] with a classifier-fres guidance scale 7.5 with different solvers and NFEs. Prompt:glowway portrait of curly orange haired mad scientist man from borderlands 3, au naturel, hyper detailed, digital art, trending in artstation, cinematic lighting, studio quality, smooth render, unreal engine 5 rendered, octane rendered, art style by pixar dreamworks warner bros disney riot games and overwatch.
43


Reviews:
[{"review_id": "xoNO7P5tTP", "reviewer": "Reviewer_tWFq", "summary": "The paper presents a multistep SDE solver for diffusion models instead of ODE solvers. The main goal is to have diverse and high-quality samples while reducing the number of solver step required. To do this, the paper proposes a new SDE that includes an additional term, ensuring the marginal distribution unchanged. Based on this new SDE, the paper adopts stochastic Adam method, introducing two key component: SA-predictor, and SA-corrector. The convergences of these two are provided. Experiments are conducted in different scenario including varying stochastic noise scale, different models. The results support the claims on the reduction in NFEs.", "questions": "\nHow many of predictor step $s_p$ and corrector step $s_c$ is used?\n\nMinor points,\n- Line 199, please add the notation stepsize $h$\n- Eq (13), please add the description for the Lagrange basis $l(t)$.\n- Caption of Fig. 1, \"scholastic\" -> \"stochastic\"\n- There are some typos in Appendix", "limitations": "The paper mentioned a current limitation is that the optimal $\\tau(t)$ is unknown, which I agree. There many factors affecting this parameters including NFE, dataset and approximated score function.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has the following strong points:\n\n- The experiment shows state-of-the-art result in term of FID given a limited NFEs. This highlights the effectiveness of the proposed method in achieving impressive results with efficient resource utilization.\n- It provides theoretical studies on the convergence of proposed method, offering valuable insights ", "weaknesses": "The main weaknesses I find in this paper are:\n\n- In term of the contribution to difussion model research, the proposed method may be considered incremental. This is because the key factor leading to good result, such as predictor-corrector, multi-step have been well-established in the existing literature.\n- The paper presentation can be improved, in particular the exposure of stochastic Adam methods. I would be good to dedicate a small paragraph or section explicitly discussing ths method to provide readers with clear understand of how it is employed in the proposed approach. \n- The paper lacks clarity in establishing the connection between the variance-control SDE and the use of stochastic Adam methods. The discussion should be more thorough and elaborate, especially in addressing why setting $\\tau=1$ (falling back to DDPM) is the best choice in many cases. Then, stochastic Adam significantly contributes to the model's overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments and carefully reviews. Below are our responses to the raised questions:\n\nQ1: The proposed method may be considered incremental. This is because the key factor leading to good results, such as predictor-corrector, and multi-step have been well-established in the existing literature.\n\nA1: The multi-step solver and predictor-corrector method have been well-established and studied in ODE solvers of diffusion models, e.g. [1][2][3]. However, as we have claimed in lines 38-39, \"adding properly scaled noise in the diffusion SDE may facilitate the quality of generated data\". Thus we focus on developing a solver for diffusion SDEs in this paper, which has not been well developed in the existing literature. \n\nBack to your concern, although the mentioned techniques (e.g., multiple-step method) have been used in diffusion ODEs, generalizing it into diffusion SDEs is non-trivial. Concretely, the core technique of the multi-step method of diffusion SDEs is from numerical SDE (e.g., Ito-Taylor expansion), which is quite different from the one used in diffusion ODEs. \n\nQ2: The paper presentation can be improved, in particular the exposure of stochastic Adam methods. I would be good to dedicate a small paragraph or section explicitly discussing ths method to provide readers with a clear understanding of how it is employed in the proposed approach.\n\nA2: Thank you for your sincere advice. We will carefully revise the order of our content and make it more readable and make a discussion about the stochastic Adam methods right after line 168 as you suggested. And we consider adding a section to explicitly introduce the stochastic Adams method in the appendix to help readers better understand the method.\n\nQ3: The paper lacks clarity in establishing the connection between the variance-control SDE and the use of stochastic Adam methods.\n\nA3: Stochastic Adams method is a numerical method to solve SDEs and variance-controlled SDE is a specialized SDE. The connection has been clarified at the beginning of Section 5. \n\nQ4:  The discussion should be more thorough and elaborate, especially in addressing why setting $\\tau = 1$\n (falling back to DDPM) is the best choice in many cases.\n\nA4: Thank you for pointing out it. In fact, we want to clarify that $\\tau = 1$ for SA-Solver is generally not the optimal setting,  as in line 259. We uniformly use it over different datasets because we aim to make a fair comparison with other methods (no extra-tuned hyperparameters). As can be seen in the ablation study to $\\tau$ in Tables 6, 7, 8, and 9 in our supplementary material, SA-Solver with larger $\\tau$ (even larger than 1) exhibits improved results, especially under larger NFEs. \n\nTo further address your concern, we evaluate our SA-Solver with improved (compared with results in this paper) non-constant $\\tau(t)$ inspired by EDM [4]. Concretely, a $\\tau(t)=\\tau$ in the interval $[t_{min}, t_{max}]$, and $\\tau(t)=0$ otherwise, where $t_{min}$ and $t_{max}$ are selected as in [4]. The results are summarized below, and we will add them to the revised version. \n\n| method\\NFE on CIFAR10(EDM baseline-VE)    | 11       | 15       | 23       | 31       | 47       | 63       | 95       |\n|---------------------------|----------|----------|----------|----------|----------|----------|----------|\n| SA-Solver(vanilla)        | 7.49     | **4.84** | 4.04     | 3.41     | 3.18     | 3.24     | 3.17     |\n| SA-Solver(improved)       |**6.46**     | 4.91     | **3.77** | **3.40** | **2.92** | **2.74** | **2.63** |\n\n| method\\NFE on ImageNet64(ADM)  | 15       | 23       | 31       | 47       | 63       | 95       |\n|---------------------------|----------|----------|----------|----------|----------|----------|\n| SA-Solver(vanilla)        | 3.65     | 3.08     | 2.77     | 2.40     | 2.30     | 2.22     |\n| SA-Solver(improved)       | **3.41** | **2.61** | **2.23** | **1.95** | **1.88** | **1.81** |\n\nThe first row uses the original setting in the paper, which contains constant $\\tau(t)$, not tuned $\\tau$.\n\nThe second row uses the improved setting, which contains the piecewise constant $\\tau(t)$, tuned $\\tau$.\n\nQ5: How many of predictor step $s_p$ and corrector step $s_c$ is used?\n\n A5: As clarified in line 239, we use $s_p = 2$ and $s_c = 1$ in the experiment part as they are observed stable in practice. \n\n\n[1] Liu, L., Ren, Y., Lin, Z. and Zhao, Z., 2022. Pseudo numerical methods for diffusion models on manifolds.\n\n[2] Zhao, W., Bai, L., Rao, Y., Zhou, J. and Lu, J., 2023. UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models.\n\n[3] Li, S., Liu, L., Chai, Z., Li, R. and Tan, X., 2023. ERA-Solver: Error-Robust Adams Solver for Fast Sampling of Diffusion Probabilistic Models.\n\n[4] Karras, T., Aittala, M., Aila, T. and Laine, S., 2022. Elucidating the design space of diffusion-based generative models.\n \n\n\n "}, {"review_id": "hrovm1Mdl6", "reviewer": "Reviewer_UcRr", "summary": "This paper studies the problem of fast sampling of diffusion models. Since standard DDPM sampling is slow, this is a very important topic of late, with many competing methods. Many methods reformulate as a ODE solving problem, which makes it easier to do few-step sampling. However, it has been noted that sampling from the original SDE formulation (as opposed to ODE) can lead to better samples if there is budget for taking many diffusion steps. Therefore, this paper studies the problem of accelerating sampling of the SDE. \n\nThe method, named SA-Solver is a predictor corrector method. It incorporates some recent findings, such as semi-linearity, but mainly uses the method of Lagrange interpolations. More specifically, the predictor step involves (uniquely) fitting a polynomial of degree s-1 using s predictions and reading off the polynomial value at desired points. Finally, a correction step is incorporated that plug in the predicted value at the new timestep (t+1) back into the formula to choose a new predicted value (similar in spirit to heun's method). The paper draws connections to DDIM (deterministic version of 1-step SA-Predictor), DPM-Solver (deterministic version of 2-step SA-Predictor), and UniPC (deterministic version of SA_Solver)\n\nThe paper shows sampling results on cifar10, imagenet64, and imagenet256 (latent), demonstrating best FID scores (conditioned on the same number of NFEs) when using more than 30 NFEs.", "questions": "How much of the performance is due to the predictor/corrector approach, versus the lagrange interpolation? An ablation study in this regard would improve the paper and our understanding of the method.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The problem being studied is an important one, as sampling speed is arguably the biggest weakness of diffusion models (and diffusion models are being used in a huge number of applications right now).\n\nThe experimental results seem quite promising, showing better FID scores than DDIM/DPMSolver, which are very popular baselines.", "weaknesses": "Seems to be missing quantitative evaluation (e.g. CLIP score on stable diffusion) for text-to-image tasks. Also, evaluating on different domains besides image would improve the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank you for taking the time to carefully review our paper, acknowledgment of our novel contributions, and the insightful questions. Below we respond to the questions:\n\nQ1: Seems to be missing quantitative evaluation (e.g. CLIP score on stable diffusion) for text-to-image tasks.\n\nA1: In fact, for text-to-image tasks, papers on fast samplers rarely report the FID and CLIP scores, as all solvers achieve similar FID results [1][2]. These works attribute this to the powerful pretrained decoder, which can map a non-converged latent code to a good image sample [1][2]. We test 4 diffusion samplers: DDIM, DPM-Solver, UniPC, and our SA-Solver on Stable Diffusion v1.5. Following the standard evaluation procedure, we randomly draw 30k prompts from the MS-COCO validation set and report the FID results on the generated images. The results are provided in the table below, which shows similar conclusions as the previous works.\n\n| NFE\\method | DDIM  | DPM-Solver | UniPC | SA-Solver |\n|------------|-------|------------|-------|-----------|\n| 20         | 10.48 | 10.40      | 10.46 | 10.22     |\n| 60         | 10.30 | 10.47      | 10.40 | 10.33     |\n\nQ2: Also, evaluating on different domains besides image would improve the empirical results.\n\nA2: Thanks for your valuable advice! But we are not familiar with generation tasks in other domains. We will consider adding some experiments in other domains in the future.\n\nQ3: How much of the performance is due to the predictor/corrector approach, versus the Lagrange interpolation? An ablation study in this regard would improve the paper and our understanding of the method.\n\nA3: This is an interesting and valuable question. To explore this, we conduct an ablation study on the effect of Lagrange interpolation and predictor-corrector on the CIFAR10 dataset as follows. We use EDM baseline-VE pretrained checkpoint. Concretely, we vary the number of predictor steps and meanwhile conduct them with/without corrector to separately explore the effect of the two components. As can be seen, both Lagrange interpolation (Predictor 1-steps only v.s. Predictor 3-steps only) and predictor-corrector (Predictor 1-steps only v.s. Predictor 1-steps, Corrector 1-step, and Predictor 3-steps only v.s. Predictor 3-steps, Corrector 3-steps) improve the performance of our sampler.  \n\n| method\\NFE and $\\tau$                    | 15 0.4   | 23 0.8   | 31 1.0   | 47 1.4   |\n|------------------------------------------|----------|----------|----------|----------|\n| Predictor 1-step only                    | 13.76    | 12.44    | 11.72    | 14.67    |\n| Predictor 1-step with Corrector 1-step   | 8.49     | 6.87     | 6.13     | 6.75     |\n| Predictor 3-steps only                   | 5.30     | 3.93     | 3.52     | 2.98     |\n| Predictor 3-steps with Corrector 3-steps | **4.91** | **3.77** | **3.40** | **2.92** |\n\n[1] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C. and Zhu, J., 2022. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.\n\n[2] Zhao, W., Bai, L., Rao, Y., Zhou, J. and Lu, J., 2023. UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models.\n"}, {"review_id": "dFjWs4An6d", "reviewer": "Reviewer_txgZ", "summary": "The paper proposes a new solver for diffusion SDEs, termed SA-Solver, combining the ideas of a predictor-corrector scheme and the stochastic Adam solvers. The predictor and corrector utilize the Lagrange polynomials for extrapolation to lower the approximation error at future time stamps. Experimentally, SA -Solver improves over previous ODE and SDE across a wide range of image generation benchmarks.", "questions": "I copy-and-paste some questions from above:\n\n- Since the authors observe the benefit of larger $\\tau$ on larger NFE, why not use a varying $\\tau$ in Fig.2 based on NFE? \n\n- I wonder if this is an issue for using Lagrange polynomials in SDE, since the preceding predictions could be misleading. If not, could the author provide some intuitions? In addition, the authors mention that \"predictor step 2 and corrector step 1 is the most stable setup\". such small steps could be caused by the issue I raise.\n\n- Is it possible to give the theoretical order after combing the predictor and corrector?\n\n- Could the author compare different methods using the VP model (config A), as in [1] and [2], because the VP model is more commonly used in practice and offers better results?\n\n- Could the author also provide quantitative results on Stable Diffusion experiments, to better showcase the advantage of SA-Solver?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The considered problem of accelerating SDE solvers is of great practical value in the field, as SDE solver often delivers better sample quality than ODE solvers, but are hindered by their slow sampling speed.\n\n- Adapting the idea in Stochastic Adams is interesting, offering higher-order convergence with Lagrange polynomial extrapolation.\n\n- The fusion of the PC sampler and Lagrange polynomial yields superior empirical results on diffusion SDE, tested on a range of dataset resolutions from 32x32 to 256x256. It's nice to see the methods scale to different resolutions and architectures.", "weaknesses": "- Throughout the paper, the authors mention several times that empirically the quality of data generated by SDE has a better upper limit. I think some theoretical analysis in the main text to make the paper more self-contained. A concurrent [1] provides some theoretical arguments and comparisons between SDE and ODE, showing that the stochasticity in SDE can reduce overall sample errors. I think it would be beneficial to discuss them to better motivate the idea of the paper.\n\n- The authors didn't provide detailed discussions of the benefit of introducing $\\tau(t)$ (at the end the author set $\\tau(t)=1$ in experiments). Since the authors observe the benefit of larger $\\tau$ on larger NFE, why not use a varying $\\tau$ in Fig.2 based on NFE? In addition, the benefit of a larger $\\tau$ on larger NFE seems predictable based on the theory in [1].\n\n - The authors did some comparisons of epsilon-prediction and data prediction models. It seems that EDM [2] uses interpolation between these two by pre-conditioning. Does it offer better results?\n\n- Unlike the deterministic path in ODE, the randomness in SDE can change the sample trajectories quite a bit. For example, the trajectories of generating a dog can stray into generating a cat. I wonder if this is an issue for using Lagrange polynomials in SDE, since the preceding predictions could be misleading. If not, could the author provide some intuitions? In addition, the authors mention that \"predictor step 2 and corrector step 1 is the most stable setup\". such small steps could be caused by the issue I raise.\n\n- The paper separately gives the convergence order of predictor and corrector. Is it possible to give the theoretical order after combing the predictor and corrector?\n\n- I think the author didn't use the \"EDM VE\" model for CIFAR-10. In Table 2 of EDM [2], the unconditional VE (in config F, EDM VE) can achieve an FID of 1.98 using only 35 NFE, as opposed to the >3 FID in Fig 2 in the current paper. So I guess the author use their VE model in config A (baseline VE). Could the author compare different methods using the VP model (config A), as in [1] and [2], because the VP model is more commonly used in practice and offers better results?\n\n- The stochastic sampler in the concurrent work [1] obtains an FID ~ 1.8 using less than 100 NFE on ImageNet-64 with Pixel DPM, as opposed to the > 2.1 FID in Fig 2 of the current paper. I wonder if one could combine the multi-step idea in this paper with [1] to obtain further improvements.\n\n- Could the author also provide quantitative results on Stable Diffusion experiments, to better showcase the advantage of SA-Solver? Current visualized images do not seem compelling.\n\n\n### Minors\n\n- line 225, DPM-solver++ and UniPC are *ODE* solvers which would not naturally be special cases of the proposed *SDE* solver.\n\n\n\n[1] Restart Sampling for Improving Generative Processes, Xu et al, https://arxiv.org/abs/2306.14878\n\n[2] Elucidating the Design Space of Diffusion-Based Generative Models, Karras et al, https://arxiv.org/abs/2206.00364\n\n\n# Post rebuttal\n\nThanks the authors for the rebuttal. After another pass of the paper, it occurred to me that that paper is a direct application of the ideas (variation-of-parameter, or exponential integrator, and Langraian polynomial for approximation) in [1] to SDE samplers. The same formula of the exponential integrated version of SDE (Eq.5) has been proposed in prior works [1] (see their eq.17, arXiv version 1). In addition, [1] also uses a Lagrange polynomial to extrapolate. Due to it's straightforward extension of prior works, I will keep my score unchanged.\n\n[1] Zhang, Qinsheng, and Yongxin Chen. \"Fast sampling of diffusion models with exponential integrator.\" arXiv preprint arXiv:2204.13902 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank you for taking the time to carefully review our paper, acknowledgment of our novel contributions, and the insightful questions. Below we respond to the questions:\n\nQ1: I think some theoretical analysis in the main text to make the paper more self-contained. A concurrent [1] provides some theoretical arguments and comparisons between SDE and ODE.\n\nA1: Thank you for pointing out the interesting concurrent work [1], we will add a discussion to it in our revised version. In this work, the authors clarify the advantage of SDE over ODE by proving it has a lower upper bound on sampling error in Wasserstein-1 distance. An upper bound of sampling error in terms of KL divergence for variance-controlled SDE is also provided in our supplementary material, which indicates the advantage of SDE over ODE in KL divergence.\n\nQ2: The authors didn't provide detailed discussions of the benefit of introducing $\\tau(t)$. Why not use a varying $\\tau$ in Fig.2 based on NFE?\n\nA2: Thank you for pointing out it. In fact, $\\tau = 1$ for SA-Solver is not the optimal setting, we use it because we aim to make a fair comparison with other methods (no extra-tuned hyperparameters).\n\nWe evaluate our SA-Solver with non-constant $\\tau(t)$ inspired by EDM [2] and tuned $\\tau$ in $0, 0.2\\cdots,1.6, 1.8$, and we observe an improved performance. Concretely, a $\\tau(t)=\\tau$ in the interval $[t_{min}, t_{max}]$, and $\\tau(t)=0$ otherwise, where $t_{min}$ and $t_{max}$ are selected as in [2].\n\n|method\\NFE on CIFAR10(EDM baseline-VE)|11|15|23|31|47|63|95|\n|-|-|-|-|-|-|-|-|\n|SA-Solver(vanilla)|7.49|**4.84**|4.04|3.41|3.18|3.24|3.17|\n|SA-Solver(improved)|**6.46**|4.91|**3.77**|**3.40**|**2.92**|**2.74**|**2.63**|\n\n|method\\NFE on ImageNet64(ADM)|15|23|31|47|63|95|\n|-|-|-|-|-|-|-|\n|SA-Solver(vanilla)|3.65|3.08|2.77|2.40|2.30|2.22|\n|SA-Solver(improved)|**3.41**|**2.61**|**2.23**|**1.95**|**1.88**|**1.81**|\n\nThe first row uses the original setting in the paper, which contains constant $\\tau(t)$, not tuned $\\tau$.\n \nThe second row uses the improved setting, which contains the piecewise constant $\\tau(t)$, tuned $\\tau$.\n\nQ3: The authors did comparisons of epsilon and data prediction models. Does EDM's interpolation offer better results?\n\nA3: It is an interesting point! In Section A.2.4, we show that the data model injects a smaller variance compared with the noise model. Similarly, the injected noise of the interpolation model is larger than the one of the data model. Thus we speculate interpolation may not bring extra benefits.\n\nQ4: I wonder if this is an issue for using Lagrange polynomials in SDE since the sample trajectories change and the preceding predictions could be misleading.\n\nA4: The theoretical convergence result indicates at least the stochastic multistep method will converge in the distribution sense.\n\nQ5: Is it possible to give the theoretical order after combing the predictor and corrector?\n\nA5: The convergence result of the ODE predictor-corrector method is well established, e.g. see [5]. To the best of our knowledge, the theoretical convergence result of the stochastic multi-step method with p-c is not known yet.\n\nQ6: So I guess the author use their VE model in config A (baseline VE).\n\nA6: Yes. We will revise the expression to 'EDM baseline-VE' to avoid misunderstanding.\n\nQ7: Could the author compare different methods using the VP model (config A)?\n\nA7: We provide the results of the baseline-VP CIFAR10 model as below.\n\n|method\\NFE |11|15|23|31|47|63|\n|-|-|-|-|-|-|-|\n|DDIM|17.07|11.57|7.33|5.68|4.41|3.89|\n|DPM-Solver|**6.31**|**4.72**|3.46|3.28|3.07|2.99|\n|UniPC|7.05|5.59|3.08|2.88|2.88|2.88|\n|EDM-ODE|18.41|6.52|3.52|3.10|2.99|2.95|\n|EDM-SDE|29.90|10.21|4.85|3.77|3.08|2.84|\n|SA-Solver|7.05|5.59|**3.03**|**2.70**|**2.50**|**2.39**|\n\nQ8: The stochastic sampler in the concurrent work [1] obtains an FID ~ 1.8 using less than 100 NFE on ImageNet-64, as opposed to the $>$ 2.1 FID of the current paper. I wonder if one could combine the two ideas to obtain further improvements.\n\nA8: Thank you for your advice! We think the idea in [1] is interesting and the result is promising. We first want to point out that we have a typo in line 271, and it causes a misleading. We actually use EDM baseline-VE for CIFAR10 while ADM[4] for ImageNet64. We will correct this typo in the revised version. \n\nOur results for ImageNet64 are based on ADM which is slightly weaker than EDM, i.e., 1.55 v.s. 1.36 for SOTA FID. To fairly compare, we conduct our method with the non-constant $\\tau(t)$ in A2 on EDM ImageNet64 model and provide the results below. The idea that combines the two papers can be explored in the future. \n\n|method\\NFE|39|67|99|165|\n|-|-|-|-|-|\n|Restart|2.38|1.95|1.71|1.51|\n|SA-Solver|**1.80**($\\tau=1.0$)|**1.58**($\\tau=1.4$)|**1.49**($\\tau=1.8$)|**1.44**($\\tau=2.2$)|\n\nQ9: Could the author also provide quantitative results on Stable Diffusion experiments, to better showcase the advantage of SA-Solver? Current visualized images do not seem compelling.\n\nA9: We test on Stable Diffusion v1.5. Following the standard evaluation procedure, we randomly draw 30k prompts from the MS-COCO validation set. The results show that all solvers achieve similar FID results. We attribute this to the powerful pretrained decoder, which can map a non-converged latent code to a good image sample. This phenomenon has also been observed in section 7.2 in [3].\n\n|NFE\\method|DDIM|DPM-Solver|UniPC|SA-Solver|\n|-|-|-|--|-|\n|20|10.48|10.40|10.46|10.22|\n|60|10.30|10.47|10.40|10.33|\n\n[1]Xu, Yilun, et al. \"Restart Sampling for Improving Generative Processes.\"\n\n[2]Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\"\n\n[3]Lu, Cheng, et al. \"Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.\"\n\n[4]Dhariwal, Prafulla, and Alexander Nichol. \"Diffusion models beat gans on image synthesis.\"\n\n[5]Gragg, William B., and Hans J. Stetter. \"Generalized multistep predictor-corrector methods.\""}, {"review_id": "sNkri60rIT", "reviewer": "Reviewer_VoWY", "summary": "The paper proposed a stochastic Adam solver for solving diffusion SDEs in an efficient way with a convergence guarantee. Authors adapt the stochastic Adam from numerical literature and use Lagrange interpolation to predict unknown terms. They show strong convergence for both predictor and corrector. Numerical experiments are done to verify their claims. ", "questions": "See weaknesses.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. By deriving an explicit form of the solution of the SDE, authors are able to utilize stochastic Adam with Lagrange interpolation to find an efficient diffusion SDE solver, \n2. Theoretical analysis is well-written.\n3. Numerical experiments show improvements compared to other solvers, especially the diffusion ODE solvers.\n4. It sort of unifies previous methods in some sense. ", "weaknesses": "1. My main concern is, from Figure 2, it seems SA-solver only outperforms other methods within a certain range of NFE. This \"optimal\" range looks very different for different data sets. In reality, if I use SA-solver for the sake of doing fewer function evals, how do I know when it outperforms other methods? \n\n2. I don't think I fully understand how to choose the parameter tau(t) in the experiments. Please comment on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank you for taking the time to carefully review our paper, acknowledgment of our novel contributions, and the insightful questions. Below we respond to the questions:\n\nQ1: I don't think I fully understand how to choose the parameter tau(t) in the experiments. Please comment on this.\n\nA1: In our experiment, we use a constant function $\\tau(t) = \\tau, \\forall t \\in [0, T]$. In section 6.1, we use $\\tau = 1$ for the comparison of the data-prediction model and noise-prediction model. In section 6.2, we vary $\\tau$ from $\\{0.0, 0.2, \\cdots, 1.6\\}$ to explore the effect of different NFE and $\\tau$ to the result. In section 6.3, in relatively small NFEs, we set a proper small $\\tau$ value and use $\\tau = 1$ over 20 NFEs.\n\nIn fact, we want to clarify that $\\tau = 1$ for SA-Solver is generally not the optimal setting. We uniformly use it for our method in Tables 1 and 2 over different datasets because we aim to make a fair comparison with other methods (no extra-tuned hyperparameters). \n\nTo further address the concern about choosing the proper $\\tau(t)$, we conduct an ablation study in section 6.1 (see Figure 1) by varying the value of $\\tau$. Our empirical results suggest that increasing $\\tau$ with NFEs improves the FID. \n\nMoreover, we empirically observe that using a non-constant $\\tau(t)$ yields better results than the one presented in this paper. Thus we suggest using it in practice. The strategy of such improved $\\tau(t)$ is taking $\\tau(t)=\\tau$ in the interval $[t_{min}, t_{max}]$, and $\\tau(t)=0$ otherwise, where $t_{min}$ and $t_{max}$ are selected as in [1] Below are the results under improved $\\tau(t)$.\n\n| method\\NFE on CIFAR 10    | 11       | 15       | 23       | 31       | 47       | 63       | 95       |\n|---------------------------|----------|----------|----------|----------|----------|----------|----------|\n| SA-Solver(vanilla)        | 7.49     | **4.84** | 4.04     | 3.41     | 3.18     | 3.24     | 3.17     |\n| SA-Solver(improved)       | 6.46     | 4.91     | **3.77** | **3.40** | **2.92** | **2.74** | **2.63** |\n| Best over baseline method | **6.41** | 5.01     | 4.04     | 3.82     | 3.59     | 3.36     | 3.06     |\n\n| method\\NFE on ImageNet64  | 15       | 23       | 31       | 47       | 63       | 95       |\n|---------------------------|----------|----------|----------|----------|----------|----------|\n| SA-Solver(vanilla)        | 3.65     | 3.08     | 2.77     | 2.40     | 2.30     | 2.22     |\n| SA-Solver(improved)       | **3.41** | **2.61** | **2.23** | **1.95** | **1.88** | **1.81** |\n| Best over baseline method | 3.49     | 2.83     | 2.75     | 2.72     | 2.44     | 2.22     |\n\nQ2: My main concern is, from Figure 2, it seems SA-solver only outperforms other methods within a certain range of NFE. This \"optimal\" range looks very different for different data sets. In reality, if I use SA-solver for the sake of doing fewer function evals, how do I know when it outperforms other methods? \n\nA2: Thank you for pointing out this. Under the improved $\\tau(t)$ as in the A1, our method almost consistently beats the other baseline methods as presented in the above Tables. \n\n[1] Karras, T., Aittala, M., Aila, T. and Laine, S., 2022. Elucidating the design space of diffusion-based generative models.\n\n"}, {"review_id": "K9zRGHrfqf", "reviewer": "Reviewer_zg3j", "summary": "This paper extends UniPC to a stochastic manner to obtain a faster sampler called SA-Solver for diffusion models. Specifically, the authors start from the formulation of diffusion SDE and derive the SA-Predictor and SA-Corrector. Extensive experiments on CIFAR10, ImageNet, LSUN, etc demonstrate the effectiveness of the proposed SA-Solver.", "questions": "See the weaknesses.", "limitations": "The limitations have been fully discussed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is interesting to consider sampling from the SDE instead of ODE. This opens up a new direction for the fast sampling of DPMs. I am surprised that this simple practice can improve the sampling quality so much.\n2. The writing is clear and the derivation of SA-Solver is easy to understand. I especially appreciate Section 5.3, where the relationship with UniPC helps me to understand the proposed method at a glance.\n3. The experiments are thorough and convincing. Most required experiments are included, such as unconditional/conditional sampling with data/noise prediction with different image resolutions. ", "weaknesses": "1. It is questionable whether the novelty of SA-Solver is enough. As said in L228, SA-Solver can be viewed as an extension of UniPC with a non-zero $\\tau(t)$. However, I think the idea to introduce stochasticity back to the sampling of the diffusion models also contributes to the novelty. I just feel worried that this issue might be raised by other reviewers.\n2. Some experiments are lacking, such as comparing the convergence of different methods on COCO using a pre-trained text-to-image diffusion model. For now, only some qualitative results are provided. It would be better to also demonstrate some quantitative comparisons.\n3. Some minor format issues: too much space below Table 1. Maybe the authors can try to make it looks better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank you for taking the time to carefully review our paper, acknowledgment of our novel contributions, and the insightful questions. Below we respond to the questions:\n\nQ1: It is questionable whether the novelty of SA-Solver is enough.\n\nA1: As we have claimed in lines 38-39, \"adding properly scaled noise in the diffusion SDE may facilitate the quality of generated data\". Thus we focus on developing a solver for diffusion SDEs in this paper, which has not been well developed in the existing literature. \n\nThough our method originates from similar techniques as in existing methods of solving diffusion ODEs, i.e., multiple-steps and predictor-corrector. Generalizing the multiple-step method into diffusion SDEs is non-trivial. Concretely, the multi-step and predictor-corrector of diffusion SDEs are from numerical SDE, and the core idea behind them is quite different from their versions under diffusion ODEs (e.g., Ito-Taylor expansion v.s. Taylor expansion).    \n\nWe also clarify the difference between SA-Solver (ours) with the existing sampling methods in Sec. 5.3, while most of them aim to accelerate the sampling of diffusion ODEs. We show that these methods are the certain case of the proposed SA-Solver.\n\nQ2: Some experiments are lacking, such as comparing the convergence of different methods on COCO using a pre-trained text-to-image diffusion model.\n\nA2: We test 4 diffusion samplers: DDIM, DPM-Solver, UniPC, and our SA-Solver on Stable Diffusion v1.5. Following the standard evaluation procedure, we randomly draw 30k prompts from the MS-COCO validation set and report the FID results on the generated images. The results are provided in the table below. The results show that all solvers achieve similar FID results. We attribute this to the powerful pretrained decoder, which can map a non-converged latent code to a good image sample. This phenomenon has also been observed in section 7.2 in [1].\n\n| NFE\\method | DDIM  | DPM-Solver | UniPC | SA-Solver |\n|-|-------|------------|-------|-----------|\n| 20| 10.48 | 10.40      | 10.46 | 10.22     |\n| 60 | 10.30 | 10.47      | 10.40 | 10.33     |\n\nIn [1][2], the authors compare the convergence speed of different diffusion ODE samplers by reporting the l2-distance between the sampled latent and the ground truth latent (approximated by 999 steps DDIM) under the same random seed and initial noise. However, this examination method does not fit stochastic samplers since even given the same initial noise, the intrinsic stochastic property will guide the stochastic samplers to different sampled latent.\n\n\n\nQ3: Some minor format issues: too much space below Table 1. Maybe the authors can try to make it looks better.\n\nA3: Thank you for your sincere advice. We will revise it in the final version.\n\n[1] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C. and Zhu, J., 2022. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.\n[2] Zhao, W., Bai, L., Rao, Y., Zhou, J. and Lu, J., 2023. UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models."}, {"review_id": "DZKR6wT1NY", "reviewer": "Reviewer_vQfu", "summary": "The authors propose a sampling method for diffusion probabilistic models by solving an alternative SDE with the same marginal distribution. Approximation techniques are applied in constructing the computationally efficient solver and corrector . The results show a superior FID with less function evaluations.", "questions": "1. Could you elaborate on why the images are not consistent for different NFEs in your method?", "limitations": "The authors did not address the limitations.\n\nThere is no potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The construction of the alternative SDE and the approximation techniques, including score function approximation, change of variable and Lagrange interpolation, provide a complete framework for efficient DFM generation.\n2. The proposed method can achieve lower FID scores with less function evaluations which is promising.", "weaknesses": "1. The paper content seems not well organized yet.\n2. In the appendix, for other methods, with different NFEs, the image content is consistent. But for the proposed method, the image content varies a lot. It might be caused by the inaccurate approximations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank you for taking the time to carefully review our paper, acknowledgment of our novel contributions, and the insightful questions. Below we respond to the questions:\n\nQ1: The paper content seems not well organized yet.\n\nA1: Thank you for your sincere advice. The paper is organized in the following order. The introduction part is contained in Section 1. In Sections 2 and 3, we introduce the background and related work on the fast sampling of diffusion models. In Section 4, we propose a family of diffusion SDEs called variance controlled diffusion SDEs which shares the same marginal probability distribution. In Section 5, we introduce our SA-Solver, which utilizes the stochastic Adams method to solve the SDEs we proposed in Section 4. In Section 6, we conduct several ablation studies and experiments to demonstrate the effectiveness of our method. We will carefully finalize our draft to make it more readable.\n\nQ2: In the appendix, for other methods, with different NFEs, the image content is consistent. But for the proposed method, the image content varies a lot. It might be caused by inaccurate approximations.\n\nA2: Thank you for noting such an interesting phenomenon. In contrast to inaccurate approximations, we attribute the inconsistent content under different NFEs to the stochasticity of diffusion SDEs.\n\nConcretely, for diffusion ODEs, owing to its deterministic formulation, there is a one-to-one correspondence between starting noise and generated image, so that the same generated contents within the same starting noises under different NFEs (as shown in Figure 4). However, the one-to-one correspondence property does not hold for diffusion SDEs as it injected extra randomness during the sampling procedure. Under the same random seed, different NFEs in the setting of the stochastic sampler will sample different numbers of Gaussian noises, which is the reason the image content varies. This explains the observed inconsistent contents under different NFEs for stochastic sampling methods i.e., EDM (SDE) and our SA-Solver in Figure 4 in Appendix. "}]

Author rebuttal:

Meta-review: The reviews on this paper were overall positive. While some reviewers questioned the possibly incremental improvement upon existing SDE solvers (e.g., the predictor-corrector method for SDE has been known to be important for sample quality for a while now), I think experimental results presented in the paper make up for this. Also, this work contains some nice theoretical analysis. I encourage the authors to incorporate the feedback from the reviewers for the camera ready, in particular addressing the questions around the setting of $\tau$, and furthermore including the many ablation studies run in the rebuttal.
Decision: Accept (poster)