On Accelerating Diffusion-Based Sampling Processes via Improved Integration Approximation
Authors: Guoqiang Zhang
Kenta Niwa
W. Bastiaan Kleijn
Venue: ICLR 2024 poster (2024)
Keywords: diffusion
EDM
DDIM
DPM-Solver
optimal stepsize
Primary area: generative models
TL;DR: A new technique for accelerating diffusion-based sampling processes such as EDM, DDIM, and DPM-Solver by optimizing the stepsizes in front of the gradients

Abstract: A popular approach to sample a diffusion-based generative model is to solve an ordinary differential equation (ODE). In existing samplers, the coefficients of the ODE solvers are pre-determined by the ODE formulation, the reverse discrete timesteps, and the employed ODE methods. In this paper, we consider accelerating several popular ODE-based sampling processes (including EDM, DDIM, and DPM-Solver) by optimizing certain coefficients via improved integration approximation (IIA). We propose to minimize, for each time step, a mean squared error (MSE) function with respect to the selected coefficients.  The MSE is constructed by applying the original ODE solver for a set of fine-grained timesteps, which in principle provides a more accurate integration approximation in predicting the next diffusion state. The proposed IIA technique does not require any change of a pre-trained model, and only introduces a very small computational overhead for solving a number of quadratic optimization problems. Extensive experiments show that considerably better FID scores can be achieved by using IIA-EDM, IIA-DDIM, and IIA-DPM-Solver than the original counterparts when the neural function evaluation (NFE) is small (i.e., less than 25).

Published as a conference paper at ICLR 2024
ON ACCELERATING DIFFUSION-BASED SAMPLING PROCESSES VIA IMPROVED INTEGRATION APPROXI- MATION
Guoqiang Zhang
Kenta Niwa
W. Bastiaan Kleijn
Dept. of Computer Science
Communication Science Labs, NTT Japan
School of ECS
University of Exeter
Victoria Univ. of Wellington New Zealand
United Kingdom
R. zhanggexeterr uc.uk
kenta. Conta.niwaanre niwagntt..com
bast lann kles jnévux.ao.nz
ABSTRACT
A popular approach sample diffusion- based generative model is to solve ordinary differential equation (ODE). In existing samplers, the coefficients of the ODE solvers are pre-determined by the ODE formulation, the reverse discrete timesteps, and the employed ODE methods. In this paper, consider accelerating several popular ODE-based sampling processes (including DDIM, + and EDM) by optimizing certain coefficients via improved integration approxi- mation (IIA). We propose to minimize, for each time step, a mean squared error (MSE) function with respect to the selected coefficients. The MSE is constructed by applying the original ODE solver for set of fine-grained timesteps, which in principle provides more accurate integration approximation in predicting the next diffusion state. The proposed IIA technique does not require any change pre-trained model, and only introduces very small computational overhead for solving a number of quadratic optimization problems. Extensive experiments show that considerably better FID scores be achieved by using IIA-DDIM, HIA-DMM-Solver++ and IIA-EDM than the original counterparts when the neural function evaluation (NFE) small (i.e., less than 25).
(b)
(c)
DDIM
IIA-DDIM
Figure 1: Comparison of DDIM and proposed ITA-DDIM with timesteps for text-to-image generation over StableDiffusion V2. See Table for input texts, for FID evaluation, and Figs. R for more images.
INTRODUCTION
As one type of generative models Goodfellow et al. 2014): Arjovsky et al. (2017); Guirajani
al. (2017): [Suuer et al. (2022); Bishop (2066), diffusion probabilistic models (DPMs) have made significant progress in recent years. Following the pioneering work Sohl----Diksttinn et al. (2015). various learning and/or sampling strategies have been proposed to improve the performance of
DPMs, which include, for example, denoising diffusion probabilistic models (DDPMs) Ho al.
2020). denoising diffusion implicit models (DDIMs) Song et al. (2021a). improved DDIMs Nichol & Dhariwal (2021): Dhariwal & Nichol (2021). generalized DDIM Zhang et al. (2023b), latent diffusion
models (LDMs) Rombach et al. (2022), score matching with Langevin dynamics (SMLD) Song
& Ermon 2019 Song et al. 2202 analytic-DPMs. Bao et al. (2022baa) optimized denoising

---

Published as a conference paper at ICLR 2024
schedules Kingma et al. (2021);
): Chen et al. 2020): Lam et al. (2022), and guided diffusion strategies
Nichol et al. (2022) ; Kim et al. (2022). It is worth noting that DDIM can be interpreted as a first-order ODE solver, where its coefficients are pre-determined by the ODE formulation and the discrete reverse timesteps. See also Yang et al. (2021) for a detailed literature overview.
To further improve the sampling qualities in DPMs, one recent research trend is exploit high-order methods for solving the ordinary differential equations (ODEs) in the sampling processes. The authors Li et al. (2022) proposed pseudo linear multi-step (PLMS) sampling method, of which high-order polynomials of the estimated Gaussian noises from a score network are introduced per timestep to improve the sampling quality. The work Zhang & Chen (2022) further extends Liu et al. (2022) by refining the coefficients of the high-order polynomials of the estimated Gaussian noises, and proposes the diffusion exponential integrator sampler (DEIS). Recently, the authors of Lu et al. (2022a b) considered solving the ODEs of diffusion model differently from /Zhang & Chen (2022). In particular, a high-order Taylor expansion of the estimated Gaussian noises was employed to approximate the continuous solutions of the ODEs more accurately. The resulting sampling methods are referred to as DPM-Solver and DPM-Solver++. The work [Zhan et al. (202a)) improves the sampling performance of DDPM, DDIM, second order PLMS (S-PNDM), DEIS, and DPM-Solver by performing additional extrapolation on the estimated clean data at each reverse timestep. The recent work Karras al. (2022) achieves state-of-the-art (SOTA) sampling performance on CIFARIO and ImageNet64 by utilizing only the improved Fuler method Asscher & Petzold (1998) to solve an ODE of a refined diffusion model, referred as the EDM sampling procedure. Similarly to DDIM, the coefficients of EDM are pre-determined by the ODE formulation and the reverse timesteps.
In this paper, we make two main contributions. Firstly, we propose to optimize the stepsizes (or coefficients) in front of the selected gradient vectors in a number of promising ODE-based sampling processes (including DDIM, DPM-Solver++,, and EDM). Our basic idea to improve the accuracy of the integration approximation per timeslot when predicting the next diffusion state by minimising a mean squared error (MSH) function, referred to as the improved integration approximation (IIA) technique. The MSE per reverse timeslot is constructed by measuring the difference between the coarse- and fine- mee-grrined approximations of an ODE integration, where the fine- ane--grined approximation is obtained by applying the original ODE solver over set of fine-graines timesteps. The MSE is then minimized with respect to the considered stepsizes embedded in the coarse integration approximation. Our IIA technique renders more flexibility than the aforementioned existing ODE solvers, of which the update formats are always fixed for different pre- are-trained diffusion models. See Bar-Sinai et al. (2019) and Li et al. (2023) for related works on optimizing numerical integration parameters in the context of solving differential equations describing physical systems.
Secondly, verify the effectiveness of IIA-DDIM, HIA-DPM-Solver+++ and IIA-EDM via extensive experiments. For each method being applied to a pre-trained model with pre-defined timesteps, the optimal stepsizes are computed only once by minimising the constructed MSEs (MMSEs), and then stored for extensive sampling in the FID evaluation. To reduce computational overhead, the MMSEs are performed by solving set of quadratic functions based on a finite number of initial noise samples. In all our experiments, introducing the IIA technique into DDIM, and DPM-Solver++ and IIA-EDM significantly improves the sampling quality for small NFEs (see Figs.
3 9 10 and Table T). Computational overhead and sampling time (see Tables 10 and 9 were measured for HI-EDMM, showing that the overhead is negligible and the sampling time is roughly the same as that of EDM.
2 PRELIMINARY
Forward and reverse diffusion processes: distribution Patata (x) with a bounded variance. A forward diffusion process progressively adds Gaussian noises to the data samples æ to obtain 2; as t increases from 0 until T. The conditional distribution of z, given a can be represented as
Suppose the data sample x E Rd follows a data
Qnjoz (z) = N(z of1),
where o and a₂ are assumed to be differentiable functions of with bounded derivatives. We use q(z;; Ote, σt) denote the marginal distribution of 2, The samples of the distribution g(zT; oT) would be practically indistinguishable from pure Gaussian noises if oT >> QT.
The reverse process of diffusion model firstly draws sample 27 from (00 of I). and then progressively denoises it obtain sequence of diffusion states (zz, ~ p(2;00, d; ) } Jezo where
2

---

Published as a conference paper at ICLR 2024
the notation p(-) indicate that reverse sample distribution might not be identical to the forward distribution q(-) because of practical approximations. is expected that the final sample v is roughly distributed according Pdata (z), i.c., Pdata (x) =z p(zixi Oxx T(x) where tN = 0.
ODE formulation: In [ong et al. (2021c), Song et al. present a so-called probability flow ODE which shares the same marginal distributions 2, in (T). Specifically, with the formulation (T) for forward diffusion process, its reverse ODE form can be represented as
dlog at 2t 2 di
1 doo dl logg at oi
< log ((z) ar a.) dt,
dz=
2
(2)
di
d(=,0)
where V log ((2; aq, oz) oz) in is the score function Hyvarinen (2005) pointing towards higher density of data samples at the given noise level (av, and the gradient ds is represented by d(z, t).
As increases, the probability flow ODE (2) continuously reduces noise level of the data samples in the reverse process. In the ideal scenario where no approximations are introduced in (2). the sample distribution p(z; Oxe (x) approaches Pdata (x)) as goes from to O. As a result, the sampling process of diffusion model boils down to solving the ODE form ( where randomness is only introduced in the initial samples. This has opened up the research opportunity of exploiting different ODE solvers in diffusion-based sampling processes.
To be able to utilize 2 for sampling, one needs to specify a particular form of the score function V log q(z; aq, oz). One common approach is to train a noise estimator éo by minimizing the expected L2 error for samples drawn from Udota (see Ho et al. (020)); Song et al. 2021c ab):
Denoising score matching:
Egepaar Be-N(0,0f
o.c. t) cll
where (a,, o,) are from the forward process I With 3., the score function can then be represented in terms of éø Ezz; () as
V log ((: at oz) = - (o(zz;t)/o)
(4)
IMPROVED INTEGRATION APPROXIMATION (IIA) FOR DDIM AND DPM-SOLVER++
In this section, first briefly present the basic principle for incorporating IIA into existing ODE solvers. After that, we consider applying the IIA technique for both the conventional DDIM sampling and the classifier-free guided DDIM sampling developed for text-to-image generation. In the end, we explain how to design ML-DDDM---oveeee for text-to-imas generation.
Design principles for incorporating ITA into an ODE solver: To our best knowledge, the formats of existing diffusion samplers (c.g., DDIM, EDM and Solve++v) are fixed irrespective of the number of time steps and particular pre-trained models. It is likely that the coefficients of those ODE solvers are not always optimal. To improve the accuracy of the integration approximation per timestep in existing ODE solver, we propose to optimize the stepsizes (or coefficients) in front of certain selected quantities via MMSE. The quantities can be functions of, for example, estimated Gaussian noises and/or estimated clean data. The MSE per time-interval can be constructed by measuring the difference between the coarse- and fine-grained integration approximations, where the stepsizes principle, the
to be optimized are embedded in the coarse- coarse-grained integration approximation. selected quantities are appropriate as long as the residual error of the MSE after stepsize optimization reduced notably compared to that of the original ODE solver. As will be presented in the following, when we incorporate IIA into DDIM, DPM- we focus on a particular proper selection of the quantities for each sampling method instead of finding the optimal configuration of these quantities that produces the smallest residual error.
and EDM by following the above guidance,
ITA for conventional DDIM sampling: The conventional DDIM sampling procedure is in fact first-order solver for the ODE formulation 2 update expression is given by
(see) Lu et al. 2002a)) Zhang & Chen (022)). Its
(xx.)t))
Zi Te. Eo (Z1,41) 4i)
((4) + +o (0( ~0 ((le d(z,r)ds
2;41 = 00441
(5)
Ctty
3

---

Published as a conference paper at ICLR 2024
DECA
d(a. d[].
fine- fine-grained appro
3
1++1
I." dis, ridt oo (ma (u(a(R) af
+dulBala,, ()) fot-tt 4. 4.-)))
Figure 2: Coarse and fine-grained approximations of the integration f. d(z, t)dt. {dio, on are the introduced stepsizes in IIA-DDIM for conventional sampling. which are determined by (7).
where ((2,, t.) denotes an estimator for the clean image æ, and d(z, 7) is from (2)
We now introduce two additional quantities in computing #;+1, which are given by P (z..4)
1st quannitty
2i-1 Oeti (zi,, (.) + tu Eo Zi, (;) +0 (0(z), - (((-n))-1))
+ +h (Eo(Zi,t)) EØ (z 1,--1))))1 12)
(6)
2nd quantity
The first quantity in (6) can interpreted as a gradient vector pointing towards the data sample æ (see /Zhang et al. (2023a)). The second quantity in (6) is vector measuring the difference of the noise estimators at timesteps & and 14-1, which is inspired by high-order ODE solvers /Zhang & Chen (2022); Liu et al. 2022). The two stepsizes (o oa) in front of the two quantities in (6) are computed by the MMSE as follows:
(0%0 0/1)) = arg min Eeg ~(0.030))) 010.0111
L Pt. 44) + 20 - @ (22--1,
M
+ ten (,) 4,) 4-11))
tet + me (zi
mt lit x -iz 37
m==
where the expectation is over the probability distribution the initial state 2tp- The summation in the RHS of (7) from 0 until m M corresponds to applying DDIM over a fine-grained set of timesteps {1++ x2 mm=o within the time-interval of [4, 44+1] (see Fig. 2p. principle, when M goes to infinity, the summation would provide very accurate approximation of the integration in (7). The solution (0%, o1 makes the update 2 in (6) optimal with respect to the MMSE criterion of 7 over the probability distribution of the initial state Zto- We note that the FID score for measuring image quality is in fact also performed over the probability distribution of the initial state. In principle, the MSE on the RHS of (7) is indeed reduced due { Φū } the resulting FID would be improved. In practice, the expectation in (7) can be realized by utilizing set B of initial samples at fo- (o o11) can then be computed by solving a quadratic optimization problem. Once the optimal stepsizes are obtained, they can be stored and re-used later on for extensive sampling.
IIA for classifier-free guided DDIM sampling for text-to-image generation: The classifier-free guided DDIM method has been widely used in diffusion based text-to-image generation. The basic idea to evaluate the noise prediction model two times at each timestep 4, the first time with a text prompt: éØ 2,, Ø Pitz) (where P denotes the text prompt) and the second time with the null text prompt: éØ Ri, 6 = null;t,) The two predicted noises are then combined to obtain a refined noise (0 Ø P; f:). which is plugged into the DDIM update in computing the next diffusion state.
To apply IIA to the above text-to-image generation scenario, we optimize the stepsize (or coefficient) in front of Čo(2, o P,t,), which is found be preferable over the two terms in (6) for the conventional DDIM method:
tt (Zi, (2) 1, 1 PP;4),
where B, is the introduced stepsize and ěø o P: utilized in the DDIM update expression.
To optimize the stepsize 3; in (8), we construct an MSE estimate that averages over the probability distributions of both the initial noise vector 2t, and the text prompts. We assume that the text prompts
4

---

Published as a conference paper at ICLR 2024
follow non- parameetiiiic distribution that be approximated by sampling. The MSE can then be approximated as a quadratic function 3; by using finite samples of text prompts and 2
IIA for classifier-free guided DPM-Solver++ sampling for text-to-image generation: IIA-DPM- Solver++ can be designed in a similar way as IIA-DDIM for text-to-image generation presented above. Again the MSE for computing the optimal stepsizes in HA--DPM-Solver' should take into account the probability distributions of 2t, and the text prompts. See Appendix for details.
C
Remark 1. We have also considered the application of IIA to the high-order methods SPNDM and IPNDN iPNDN & Chen (202)). See the performance results in Appendix G
In summary, the IIA technique improves the sampling performance of SPNDM and IPNDM for certain pre-trained models.
IMPROVED INTEGRATION APPROXIMATION (IIA) FOR EDM
In this section, we first briefly review the EDM sampling procedure for solving the ODE 2 in Karras et al. (2022), which produces SOTA performance CIFARIO and ImageNet64. We then present our new IIA technique for solving the ODE more accurately, thus accelerating the sampling process.
4.1 REVIEW OF EDM SAMPLING PROCEDURE
The recent work Karras et al. (2022) reparameterizes the forward diffusion process to be
[r|0(=,|a=) = ((ziaaa, I)
(9)
where or of represented as d; = a, Let Do ž1, t) denote an estimator for the data sample at timestep t. It can be computed in terms of the noise estimator éo Dø t) zt/at (z, t). The resulting probability flow ODE takes the form
0z (z dt,
dz
(10)
at
Ōt
d(z,()
where the dot operation denotes a time derivative.
The work Karras al. (2022) proposed deterministic sampling procedure for solving (10) for arbitrary on and agg, Basically, the improved Euler method Ascher & Petzold (1998) was utilized for solving the ODE form. The resulting update expressions from time t; to 4 are given by
ă+11 = 2, + (t++1 - tx)
(11)
Z++1 2, (li+1 1.)( d, + 2222211
d' dl(taR)
(12)
= $ d(2,t)(
where (1+41 -t,) 1,) is the stepsize, d; = d(2,, 2,, (z), and d' = d( 11211,441 441). êeς1 is the intermediate estimate of the hidden state z at time f,+1- The final estimate 2;41 is computed by utilizing the average of the gradients d; and d' 1 We will explain in the next subsection how compute the optimal stepsize 112 instead of using the fixed one (f++1 42).
4.2 BASIC IIA FOR EDM (BIIA-EDM) VIA MMSE
In this subsection, we consider improving the accuracy of the integral approximation in (12 at timestep t. To do so, we propose to approximate the integration f d(2z, t)dt by utilizing the most recent set of gradients d = given by
J/ d(z,t)dt Σ Tin (d
di_ ++ d' d-1++1)-a)
(13)
A R)
where the set of coefficients {TH* o be interpreted as the stepsizes being multiplied by those gradients. We attempt to find a proper choice for {7ax 1 so that the integral approximation (13)
will become more accurate than the one in (12).
Our motivation for utilizing the most recent r+11 gradients in (33) is inspired by SGD with momentum
Sutskever al. (2013); Polyak (1964) and its variants Kingma Ba (2017); Zhang (2033 which
5

---

Published as a conference paper at ICLR 2024
computes and makes use of the exponential moving average of historical gradients in updating machine learning models. In general, the recent gradients provide additional directions pointing towards higher functional values. Proper exploration of those gradients can help to accelerate the diffusion sampling process.
We now in position to compute (Nit) in 113 timestep 4,- Our basic idea is first obtain a highly accurate approximation of ff d(z, t)dt, and then compute proper values of > so that the coarse approximation (13) is optimally close to the accurate approximation. Similarly to the design of IIA-DDIM, we approximate the integration f+++) d(z, l)dl by applying the improved Euler method over set of fine-graines timesteps {4i+ - J where 0 and .M correspond to the starting time and ending time fi+1, respectively, Mathematically, the integration f(++1 d(z, t) can be estimated more accurately over (1 x Jm=o as
M-1
M-1
d(2,1)dl=
Σ
m+11 d(z, l)dl
Σ
(4++ - 1 ( + _d
(5
37
m=U
m=0
fg (2i).
(14)
where we use A (z:) to denote the summation of the fine- fine-grained integration approximations.
We compute the optimal solution of {Ta in 13
via MMSE with regard to the difference of the
two approximations Di=o Vek (Z)-k) and 5g (z,):
t = arg min Es,, ((0.007
Σ k=0
Yikk (z - 550(2)))
(15)
where {zi-*)== are determined by the initial state 2to in the deterministic sampling procedure.
Similarly to the stepsize optimization in IIA-DDIM, one can solve the optimization problem (15) by utilizing a set B of initial samples at timestep ty to approximate the expectation operation. The
solution {nk 1r k= can then be easily computed by minimizing a quadratic function. Consider the
simple case of r = 0 as an example, where only the quantity A (z:) is employed in (15) The optimal be
solution %% easily
Ez, EB B (ii(z)) A (z))))
700
(16)
Ex,,EB A (2;))1
where (.) denotes inner product. For the general case > O, one can also easily derive the closed-form solution for {7 1 k= see Alg. in appendix for the sampling procedure of BIIA-EDM
after stepsize optimization.
4.3 ADVANCED IIA FOR EDM VIA MMSE
In this subsection, we present an advanced IIA technique for EDM. To do so, we reformulate the update expression for 2,+1 in The resulting update expression is summarized in lemma below: Lemma 1. The update expression for Zi+1 at timestep t; in EDM under the configuration of az = can be reformulated be
14+1
Da(2,, 4:]] + (l++1 - 4i)
Zi+1
2;
Do(2ū()) - (o(2-11;44))
(17)
26 2nd stepsize
1st quantity
2nd quantity
1st stepsise
where the detailed derivation is provided in Appendix B
Lemma I Inddicates that the integration approximation over the time interval [t,, fy+ for computing weighted summation of two quantities (see Zhang et al. (2023a)) )) [2;
Z:41 is realized
Do(2, t,]] and [De (2,+1; f241) Dæ(2;; t.).. The two stepsizes in front of the two quantities in
(17) functions of and 4,41, which are predetermined by the improved Euler method.
We propose to introduce new stepsizes front of the two quantities in [77. As will be explained be- low, the new stepsizes will be determined by the improved integration approximation (IIA) technique.

---

Published as a conference paper at ICLR 2024
Algorithm 1 IIA-EDM as extension of EDM in Karras et al. (2022)
1: Input:
2: number of time steps N, = 1, (1% ( } = O, h=l
pre-computed values via MMSE]
3: Sample 2o ~ (0, o2, 02. ()
4: for E [0.1..... N 11 do
5: d: < (z 44)
6: E:41 2, + (fe41 f.) )
7:
if 0t.+ + then
8:
d++11: + d( d( t++1)
9:
2:41 + + ~ DD
(a 2ith 642 ((:1)))
10: end if
11: end for
12: Output: NN
Specifically, we approximate the integration - d(z, t)dt at timestep t,
f211
t(
1=
= Si (1 P Do-o
(18)
is noted that a number of the most recent quantities are also included in providing additional gradient directions in the MSE minimisation.
18 for the purpose of
Next we compute the optimal stepsizes in the above function Si(-) by the following MMSE:
(0%0",
100:PPt PP 1 arg min E,,, ~V(0,.0 ||S; ((3 (R) A (2))))
(19)
where A (2,) is from (14. Since S:(-) is a linear function of its variables, the optimal solution in (99 can be easily computed by minimizing a quadratic function. Similarly the earlier subsection, the expectation operation in (19) can be approximated by utilizing a set B of initial samples at timestep fo. Again the optimisation 19) only needs to be performed once, and then can be used for extensive sampling.
By inspection of (15). (18) and (19). we can conclude that the optimisation (19) exploits the internal structure of the update for 2,+1 in EDM. For the special case of r = 0, (19) invol two variables (3h, 8 while (15) only consists of one variable "Ka- Informally, the residual error of (19 after
minimisation should be smaller than that of 15 which would lead to improved sampling quality.
EXPERIMENTS
We investigated the performance gain of the IIA technique when being incorporated into DDIM, and DPM-Solver++, and EDM sampling procedures. For the EDM sampling procedure, two IIA techniques are proposed. The associated sampling procedures are referred to as BIIA-EDM (see Alg. (2) and IIA-EMM. As we mentioned earlier, the optimal stepsizes for each pre-trained model over a particular set reverse timesteps were only computed once and were then stored and used for generating default of 50K images (unless specified otherwise) in the computation of the FID score. It is found that the IIA technique significantly improves the sampling qualities for low NFEs (e.g., less than 25). For text-to-image generation, ITA-DDIM and ITA-DPM-Solver significantly outperforms their counterparts.
5.1 PERFORMANCE OF BITA-EDM AND ITA-EDM
In this experiment, we tested four pre- pre-trained models for four datasets: CIFARIO, FFHQ, AFHQV2. and ImageNet64 (see Table in Appendix D.I) D. The set-size [B] for computing the optimal stepsizes when employing the IIA techniques was [3 200, which is also the default minibatch size for sampling in the EDM official open-source repository! [1] See Table 2 for the setup of other hyper- parameters. We note that r in BIIA-EDM and IIA-EDM was set to = I to save memory space.
https://github..e
abs/edm

---

Published as a conference paper at ICLR 2024
CIFAR10
FFHQ
AFHQV2
ImageNet64
EDM
12
BIAAEDM BA-EDM
25
BIIA.EMM IA-EDM
BIAJEDM WAVEDM
12
BIA-EOM
IIAEEM
10
20
10
10
15
B
10
15
20
25
30
36
20
50
40
20
30
40
20
30
40
NFES
NFEs
NFEs
NFE8
Figure 3: Sampling performance of EDM, BIIA-EDM, and IIA-EDM over four datasets.
CIFAR10
FFHQ
AFHQV2
ImageNe164
101
BII-EEDM
BIA-EDM I1A-FDDM
BIlA I1--FDM
EDM
1U-FFOMM
8V8 reodbual error
10
8VD. resiual enror
10
101
IIA-FTTM
100
re8/dua
10
SUS. resdual
10P
70
1011
10"
10
10
10
30
NFEs
NFES
NFEs
NFES
Figure 4: Comparison of average residual errors EDM, BIIA-EDM. IIA-EDM in Fig.
Table Comparison five methods for text-to-image generation over StableDiffusion V2 in terms of FID (the lower the better) and CLIP (the high the better) scores.
DDIM
IIADDIM
-olleere
DPM
Selvece+
DDIM
IIA-DDIM
DFM -Saleere
DPM Solverre
PLMS
10
FID
14.78
13.21
15.82
12.97
24.42
30 NFEs
FID
15.08
14.03
14.23
13.26
15.31
NFEs
CUP
24.86
25.32
23.85
CLIP
25.05
25.05
25.16
24.92
HD
14.65
13.14
13.85
12.77
14.30
FID
14.69
13.85
14.29
13.68
15.05
NFEs
CUP
25.00
25.08
25.02
25.21
24.80
NFEs
CLIP
25.01
25.05
25.05
25.12
24.95
Fig. 3 visualizes the FID scores for the four pre-trained models. is clear that EDM consistently outperforms the EDM sampling procedure when the NFE smaller than 25. This can explained by the fact that for small NFE, the integration approximation in EDM is not accurate. IIA-EDM improves the accuracy of the integration approximation by introducing optimal stepsizes. The performance of IIA-EDM is also superior to that of BIIA-EDM because IIA-EDM exploits the internal structure of the EDM update expressions (see Lemma L and (8)). making it more flexible.
Fig. 4 displays the average residual errors between the coarse- and fine- fine-grained integration approxi- mations for EDM, BIIA-EDM, and IIA-EDM. It is clear that for each NFE, IIA-EDM provides the smallest error while EDM yields the largest error. This indicates that the original stepsizes of EDM are not optimal for at least small NFEs. Our work provides one approach to compute better stepsizes via IIA for small NFEs.
is seen from the FID curves over ImageNet64 in Fig. that BIIA-EDM performs slightly worse than EDM when NFE is greater than 25, This may be because, for large NFES, an accurate integration approximation does not necessarily lead to a better sampling quality (e.g., see the sampling perfor- mance of Bao et al. (2022a). To the best of our knowledge, it is not clear from the literature why for large NFEs, there exists a discrepancy between FID scores and accurate integration approximation.
The sampling time of IIA-EDM and EDM can be found in Table Appendix [E] It can be concluded from the table that the two methods consume almost the same amount of time per mini-batch, demonstrating the efficiency of IIA-EMM. The computational overhead of II-EEDM is summarized in Table TO It is seen the time overhead is very small in comparison to the training or fine-tuning of typical DNN model.
Sampling time and computational overhead of IIA-EDM:
5.2 EVALUATION OF IIA-DDIM AND IIA D---DDM---LEEEEE
Text-to-image generation: In this experiment, performed FID and CLIP evaluation for IIA- DDIM, DPM-Soover++ DDIM, SPM-Solver++ and PLMS by using the validation set of COC02014 4 over StableDiffusion V2. For each sampling method, 20K images of size 512 x 512 were
8

---

Published as a conference paper at ICLR 2024
CIFAR10
LSUN bedroom
LSUN church
14
20
20
DDIM II-DDIM
DDIM
DDIM
12
IIA II-DDIM
IIA-DDIM
10
15
15
8
10
10
20
30
40
10
20
30
40
10
20
30
40
NFEs
NFEs
NFEs
Figure 5: Sampling performance of DDIM and IIA-DDIM for conventional pre-trained models.
generated in FID and CLIP evaluation by using 20K different text prompts. All five methods share the same set of text prompts and the same seed of the random noise generator. The obtained images were resized to a size of 256 x 256 before computing the FID and CLIP scores. The tested NFEs were (10, 20, 30, 40). The parameter M in IIA-DDIM was set to M 10, and the fine-grained timesteps were uniformly distributed within each time-interval. The set of quadratic functions for approximating the MSEs in IIA-DDIM and WAA-DPM-Solverse
were constructed by utilizing 20 different text-prompts from the validation set to compute the optimal stepsizes (see Fig. for the optimal stepsizes { } in IIA-DDIM). Once the optimal stepsizes for each sampling method with certain set of timesteps were obtained, 20K images were then generated accordingly.
Table D summarizes the FID and CLIP scores of the five methods. two new methods IA-DDIM and IIA-DPM-Solver+++ perform significantly better than the original counterparts in terms of FID performance. The facts that the FIDs of DPM-Solver++ and PLMS are larger than that of DDIM at 10 NFEs might be because the gradient statistics of the classifier-free guided diffusion sampling are different from those of the conventional diffusion sampling. As a result, the stepsizes in front of the gradients in DPM-Solver++ and PLMS might not be optimal for the scenario of the classifier-free guided diffusion sampling when NFE is small. On the other hand, different stepsizes in ILA-DDIM and IIA-DPM-Solvert+ are learned via MMSE for different pre-trained models no matter it is classificr- free guided diffusion sampling or conventional sampling.
clear from the table that our
Conventional DDIM sampling: In the second experiment, we studied the performance gain of IIA-DDIM in comparison to DDIM. We tested three pre-trained models (see IT in the appendix), one for a particular dataset: CIFAR 10, LSUN bedroom, and LSUN church. The set-size [B] for approximating the expectation operation in sampling in the computation of the FID scores. The hyper-parameter M in 7 was set to M = 3.
was set to 16, which is also the mini-batch size for
The performance results of DDIM and IIA-DDIM are shown in Fig. outperforms DDIM consistently for different NFEs and across different pre- pre-trained models. The performance of IIA-SPNDM and IIA-IPNDM is shown in the Appendix G
is seen that IIA-DDIM
To summarize, in all our experiments, the IIA based ODE solvers improve the image quality remark- ably when the NFE small (i.e., less than 25). The performance gain of the IIA technique is obtained by improving the accuracy of the integration approximation per timestep via stepsize optimization. When the NFE is large, the integration approximation of existing ODE solvers is accurate general, leaving little room to improve the accuracy to a large extent. Consequently, there is no significant improvement for large NEE when applying our IIA technique.
6 CONCLUSION
In this paper, we have proposed a new technique of improved integration approximation (IIA) to accelerate the ceffuusion-aased sampling processes. In particular, we have proposed to introduce new stepsizes (coefficients) in front of certain quantities in existing popular ODE solvers in order to improve the accuracy of integration approximation. The stepsizes at timestep i, are determined by encouraging a coarse integration approximation over [f., f441] to get closer to a highly accurate integration approximation over the same time slot. The optimal stepsizes only need to be computed once and can then be stored and reused later on for extensive sampling. Extensive experiments confirm that the IIA technique is able to significantly improve the sampling quality of EDM, DDIM, and DPM-Solver++ when the NFE is small (e.g., less than 25). This can be explained by the fact that the integration approximation in the original method for small NFE is a rough estimate. The employment of IIA has significantly improved the accuracy of the integration approximation.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein GAN. arXiv:100107877 [stat.ML], 2017.
U. M. Ascher and L. R. Petzold. Computer Methods for Ordinary Differential Equations and Differential-Aggebrue Equations. Soceity for Industrial and Applied Mathematics, 1998.
F. Bao, C. Li, Sun, J. Zhu, and B. Zhang. Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models. In ICML, 2022a.
F. Bao, Chongxuan Li, J. Zhu, and B. Zhang. Analytic- DPM: Reverse Variance in Diffusion Probabilistic Models. In ICLR, 2022b.
DDMM: an Analytic Estimate of the Optimal
Y. Bar-Sinai, S. Hoyer, J. Hickey, and M. P. Brenneru. Learning data driven discretizations for partial differential equations. arXiv: (at.i1008..90000 2019.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan. WaveGrad: Estimating Gradients for Waveform Generation. arXiv:2009.00711. September 2020.
Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. arXiv:2105.05233
[cs.LG], 2021.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. In Proceedings of the International Conference Neural Information Processing Systems, pp. 2672-2680, 2014.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin. and A. C. Courville. Improved training of wasserstein gans. Advances in neural information processing systems, pp. 5767-5777, 2017.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
A. Hyvarinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 24:695-709, 2005.
T. Karras, M. Aittala, T. Alia, and S. Laine. Elucidating the Design Space of Diffusion-Based Generative Models. In 36th Conference on Nueral Information Processing Systems (NeurIPS), 2022.
D. Kim, Y. Kim, S. Kwon, W. Kang, and I.-C. Moon. Refining Generative Process with Dis- criminator Guidance in Score-based Diffusion Models. arXiv preprint arXiv:2211.1700 [cs.CV], 2022.
D. P. Kingma and J. L. Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980v9, 2017.
arXiv preprint
D. P. Kingma, T. Salimans, Poole, and Ho. Variational diffusion models. arXiv:2107.00630. 2021.
arXiv: preprint
M. W. Y. Lam, Wang, D. Su, and D. Yu. BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis. In ICLR, 2022.
X. Li, F. Lu, M. Tao, and X.-F. Ye. NySALT: Nyström-Type Inference-Batte Schemes Adaptive to Large Time-Stepping. Journal of Computational Physics, 2023.
I. Liu, Yi Ren, 7. Lin, and 7. Zhao. Pseudo Numerical Methods for Diffusion Models on Manifolds. In ICLR, 2022.
C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Sampling in Around 10 Steps. In NewrIPS, 2022a.
C. Lu, Zhou, F. Bao, Chen. C. Li, and J. Zhu. DPM-Solver+4: Fast Solver for Guided Sampling of Diffusion Probabilistic Models. arXiv preprint arXiv:2211.01095 [es.LG], November 2022h.
10

---

Published as a conference paper at ICLR 2024
A. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
A. Nichol, P. Dharwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. GLIDE: Towards Photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4:1-17, 1964.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
A. Sauer, K. Schwarz, and A. Geiger. StyleGAN-XI: Scaling StyleGAN to large diverse datasets. In SIGGRAPH, 2022.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. ICML, 2015.
Song, C. Meng, and S. Ermon. Denoising Diffusion Implicit Models. In ICLR, 2021a
Y. Song and S. Ermon. Generative modeling by estimating gradients the data distribution. In Advances in neural information processing systems (NeurIPS), pp- 11895-11901, 2019.
Y. Song, C. Durkan, Murray, and S. Ermon. Maximum likelihood training of score-based diffusion models. In Advances in neural information processing systems (NeurIPS), 202
Y. Song. J. S.-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Based Generative Modeling Through Stochastic Differential Equations. ICLR, 2021c
H. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on Machine Learning (ICML), 2013.
L. Yang, Z. Zhang, S. Hong, R. Xu, Y., Shao, W. Zhang, M.-H. Yang, and B. Cui. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2102.09672 2021.
G. Zhang. On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performance. arXiv:2302.01029 [cs.LG], 2023.
G. Zhang. K. Niwa, and W. B. Kleijn. Lookahead Diffusion Probabilistic Models for Refining Mean Estimation. In Computer Vision and Pattern Recognition (CVPR), 2023a.
Q. Zhang and Y. Chen. Fast Sampling of Diffusion Models with Exponential Integrator. arXiv:2204.1323 13902 [cs.LG], 2022.
Q. Zhang, M. Tao, and Y. Chen. gDDIM: Generalized Denoising Diffusion Implicit Models. In ICLR, 2023b.

---

Published as a conference paper at ICLR 2024
A UPDATE PROCEDURE OF BIIA-EDM
Algorithm 2 BIIA-EDM as extension of EDM in Karras et al. (2022)
1: Input:
2: number of time steps N, P =
1, 1.17% kk = 0, 1
pre-computed values obtained via MMSE]
3: Sample 2g N(, a2, 000
4: for [0, 10......,N. N 1] do
mi Do (z1itz) 84
5: di 4 d(z
6: E:41 t 2: + (l++1 - 4.
7:
if 0411 # then
8:
d/+1 d(ž le41)
84:-1
+ de++1 Eesl be a4 De (Z:4154411
Jest1
9:
2:41 + 2, + (t+41 - (i) Eiso Vik
Vik ( di-x + 2dh [historical gradients are used]
10: end if
11: end for
Output: NN NN
Remark: Sampling Karras et al. 2022 recovered when (r6o 13 and (ri = 0/A * N)
B PROOF FOR LEMMA 1
Firstly, we rewrite the update expression for 2,41 in terms of z,, and the estimators Dz t.) and ( 1:+1):
*+1 2; + (te+1 t0.5d, + 0.5d.)
2i DD(x
2i+1 - Do(2i41,,
= + (te+1 - -t)
24g
21
2, Do(z,,tt) ti)
= + (fi+1 14)
21:
2i + (li+1 - (.)(2z (o(2,,4))/ 10(2+11,444) 4441
+ (f++1 - 12)
21,+1
2; + (ti+1 - -t,) 2i Do 2.+11)
(20)
t;
Next, we derive the expression for Do(zi, 15-1) in 20
To do so, we let
2: Do(z,, 2: -Doo(z),t))
2: 2t+(1
t.)(z.
DDo
4
21;
24
22 Do 2141) 0.5(2, 155(z,-Do(2;; Do +))) 1,2, (1:41-4;)22, Do -1) (o(
2t++1
-(li41 -;: 1, DD(0
< Do(2,, 2141) -0.5D((z,).) 1.)
21
Do(z,, -Do fi) t,(Do(zz,tt) ti) - t0(
21++1
1. (: (441) - Do(2i,tz))
+ DD (z,, ă(41) Do(2,,t))++ +
(21)
2t
Plugging
21 into 20 and rearranging the terms in the expression yields 17)
The proof is complete.
12

---

Published as a conference paper at ICLR 2024
C DESIGN OF IIA-DM-SOLVVER++ FOR CLASSIFIER-FREE GUIDED TEXT-TO-IMAGE GENERATION
In general, DPM-Solver has different implementations in the platform of StableDiffusion V2. The results in Table were obtained by using the multi-step 2nd-order DPM-Solver++e which is the default setup in StablcDiifusion. At cach timestep Êz. the pre-trained DNN model produces an estimator tØ(2).
P. (2)) of the clean-image, where P denotes the text prompt. In general, when i > O, the diffusion state 2 is computed by making estimators (2, @ = P, (t)) j = i -
of the two most recent clean-image as well as the current state 2;. For simplicity, let us
i) 1, denote the update expression of DPM--olver++ for computing 1 at timestep & as
Zi+1 11 (z,, (To (zj, Φ P.t,) = 1,4))
(22)
Next, we consider refining the estimation for 2i+1 in (2) by using the IIA technique. Similarly to the design of IIA-DDIM, we propose to compute Zi+1 at timestep t; by introducing two additional quantities into (22). which takes the form of
21+1 = - (ze, (#o(zz ø P.tz)
1, + P pn Φ P,4.). (23)
To optimize the two stepsizes (PPt, (p1) in (33g. we construct the following MSE function
(vrls ph))
= arg min E Pee (2), (00(2,, - - - 1,1))
+ PP002 + &i (Zz, o P.tz)
M-1
( ===
T 1h -4 m+ (zi+ (20 o= P,t,)) = mm m 1, - 2; 13 )
- -2,
(24)
where the expectation is taken over the distribution of initial Gaussian noise 21, ~ ~N o) and the distribution of the text-prompt P Ptest- The summation from 0 to m M - in the RHS of (24) corresponds to applying the original within [t., 1.. We use C' to denote set of finite pairs of (zt, P). The MSE in (24) can then be approximated by using the finite set C. In our experiment for both IIA-DDIM and IIA-DPM-Solver in the task of text-to-image generation, the set-size of C was set to 20. The optimal solution (4%% fi)) can be computed by solving quadratic optimization problem based on C. Once the optimal stepsizes (p ihi i=
over set of fine-grained timeslots
- computed for the first time, they then utilized for generating 20K images in FID and CLIP evaluation by feeding 20K different text-prompts from COC02014 validation set.
D ABLATION STUDY OF IIA-BASED ODE SOLVERS
In this section, we first present the setup of the hyperparameters in IIA when performing MMSE. We then study the performance of BIIA-EDM and IIA-EDM for different values. After that, we examine how the parameter M affects the performance of IIA-EDM with 11 and NFEs. Finally, we consider the performance of IIA-DDIM by optimizing one coefficient per timestep instead of two coefficients per step as being studied in
D.1 HYPER-PARAMETER PARAMETERS OF IIA WHEN PERFORMING MMSE
The reason we select study in Subsection D.2 over CIFAR 10 shows that the setup = 2 does not generally improve the FID performance. It is also noted in Table 2 that only for BIIA-EDM over FFHQ64, the setup r is being employed instead of r = 1. The ablation study in Table 3 shows that the performance of r 0 is slightly better than that of
0 in Table 2 for BIIA-EDM and IIA-EDM is because the ablation
in BIIА-EDM over FFHQ64.
For the experiment of IIADDDIM, IIA SPNDM, and IIA-IPDNM, the hyper-parameter M was set to M 3. Similarly, the fine-grained timestep 44 was computed f m = t; (441-1,)m
D.2 ABLATION STUDY FOR BIIA-EDM AND IIA-EDM WITH DIFFERENT VALUES
In this section, we perform an ablation study for BIIA-DDIM and IIA-EDM with r = and r =
The parameter M was set M which is the same the setup for obtaining Figure 3
13

---

Published as a conference paper at ICLR 2024
Table 2: Parameter-sctups when performing MMSE in BIIA-EDM and IIA-EDM. The four pre-trained models below were downloaded from official open source repository work Karras et al. (2022). The fine- grained timesteps {f.
JM were uniformly distributed within each time slot [fa, t In particular, f24
XF
(641-4)m
was computed as 44 57
M
pre-trained models
BIIA-EDM
IIA EDM
edm-cifar10-333 101
(M,r) (3,1)
(M,r) = (3, 1)
(M,r) (3.0)
(M,r) = (3,
(M,r) (3,1)
(M,r) (3.
(M,r) (3.1)
(M,r) (3,
Table FID comparison of the setups and r = BIIA-EDM FFHQ644 The parameter M was set
to .M
NFEs
11
13
15
19
BILA- EDM (r 1)
16.01
9.24
6.25
4.02
BIIA-EDM (r = 0)
12.78
7.33
5.09
EDM
29.34
15.87
9.85
5.26
Table and 5 summarize the averaged residual errors and the FID scores of the two methods, respectively. By inspection of the FID scores, it is seen that increases from to r = 2, the performance for each sampling method is slightly degraded in most cases. On the other hand, as increases, the averaged residual errors for each method in Table slightly decrease. The above analysis is bit counter-intuitive. It implies that when we introduce more gradient vectors in IIA- EDM or BIIA-EDM. it reduces the averaged residual error but does not always improve the FID score. This suggests that in addition to the residual errors of the MSEs, there are some unknown factors that affect the image sampling quality.
It is noted from Table 5 that for each sampling method, the differences of the FID scores for r = 1 and are very small. practice, it does not matter much regarding the FID scores one chooses 7= or = 2. However, the setup 2 would incur additional memory consumption. It is thus recommended to use 7 = as least CFIAR10 in Table
Table 4: Averaged residual errors of HII--DDM and IIA EDM with and 2.
BIA-EDM
BILA-EDM (r=1) 1)
IA-EDM
IA-EDMM
NFEs
EDM
(r
2)
(r 2)
(r 1)
CIFARIO
11
4.31
4.93
2,87
2.98
17.33
2.08
1.07
1.12
15
0.84
1.00
0.52
0.54
2.58
19
0.24
0.28
0.14
0.15
0.61
Table 5: FID evaluation of BIIA-EDM and IIA-EDM with and r
BIIA
BDM
HD
ED
NFEs
EDM
(r
CIFARIO
7.81
7.97
5.01
4.90
14.42
4.01
3.91
2.96
2,89
6.79
2.87
2.76
2.37
2.33
4.31
2.20
2.06
2.04
2.54
D.3 ABLATION STUDY OF IIA-EDM FOR DIFFERENT M VALUES
In this section, we study how the parameter M affects the performance of IIA-EDM over CIFAR10. The tested M values were M e {3, 10, 20, 50). Two NFEs were considered the ablation study, which were 11 and 19.
14

---

Published as a conference paper at ICLR 2024
The results are summarized in Table 6 It is clear from the table that the one-time computational overhead increases linearly as M increases from to 50. When the M value is larger than 10, the obtained FID scores and the averaged residual errors are roughly the same. This suggests that in practice, a reasonably large M value is sufficient when applying the IIA technique. For the results in Table M 10 is sufficiently large.
Table 6: Comparison of FID, computational overhead (in seconds), and averaged residual emror for different M values IIA-EDM over CIFARIO. The parameter to 1.
M
10
20
50
IA-EEM
4.90
3.76
3.71
3.70
(11 NFEs)
IA--EMM
computational
17.6
46.8
88.0
210.5
(11 NFEs)
overhead (s)
IIA-EDM
averaged residual errors
2.98
3.32
3.35
3.36
(11 NFEs)
IIA-EDM (19 NFEs)
FID
2.04
2.10
2.11
2.11
IIA
computational overhead (s) averaged residual errors
91.9
171.8
418.9
(19 NFEs)
IDA-EDM
0.146
0.179
0.181
0.182
(19 NFEs)
D.4 ABLATION STUDY OF IIA-DDIM BY OPTIMIZING ONE COEFFICIENT PER- TIMESTEP
In Section we have presented IIA-DDIM for both conventional diffusion sampling and text-to- image generation. For the case of conventional diffusion sampling, we optimize two stepsizes in front of two quantities for IIA-DDIM shown in (6). In this section, consider the sampling performance IIA-DDIM that involves only one quantity terms of the difference the estimated clean data. In particular, the update expression for 2 is computed
D 1 (R),t))
additionnl quantity
2;-1 C4 (2,, (2) + Oeep čo(z; t2) +o, (#(2,, t,) d ((2--1,4-1),,
(25)
where the optimal coefficient of can be obtained by MMSE. We omit the details for computing oj to avoid redundant derivation.
Experimental comparison: Table 7 summarizes the averaged residual errors of the associated MSES for DDIM, and IIA-DDIM with one quantity and two quantities. As we explained earlier, IIA-DDIM with one quantity has the update expression (25) while IIA-DDIM with two quantities has the update expression (6) clear from the table that IIA-DDIM with two quantities produces the smallest residual eтror for each number of timesteps. The residual errors of IIA-DDIM with one quantity are also consistently smaller than those of DDIM. By following the design principle of IIA in Section
3 both versions of IIA-DDIM are good sampling candidates.
Table 8 below summarizes the obtained FID scores the three sampling methods in Table 7 is clear from the table that both versions of IIA-DDIM perform much better than DDIM. Furthermore, IIA-DDIM with two quantities outperforms IIA-DDIM with one quantity. The above results are consistent with the behaviors of averaged residual errors of the three methods in Table
Table 7: Averaged residual errors of IIADDDIM with one quantity (see (6) and two quantities (see 25 ) for conventional diffusion sampling (sce also Fig. 5) The residual error of DDIM is also included a reference.
IIA IDAM
HA DDIM
timesteps
(ome quantity)
(two guantities)
DDIM
CIFARTO
10
8.20c-2
6.601e-2
1.36-1
8.00-5
4.50c-3
2.26622
40
8.50c
3,76e-4
3.10e
Study of the two quantities : We now show that the difference 2(2,, tx) - (2, t;-)) of the estimated clean data cannot be represented by using the difference of the estimated Gaussian noises
15

---

Published as a conference paper at ICLR 2024
Table 8: FID evaluation of IIA-DDIM with one quantity (see 66 and quantities (see
25
for conventional
diffusion sampling (see also Fig. 5
The performance of DDIM is also included as a reference.
IIA-DDIM (one quantity)
IAA DDIM
timesteps
DDIM
(two quantities)
CIFARIO
10
9.50
14.38
20
5.43
4.85
7,51
40
3.97
3.68
4,95
éz 1.) é0( 1111)
a(z,, t3) - =b(zi=111 14-1)
2i - Eo (zi, li)
Xi-1 - T4 60
at
Ohi-1
Zi
7
Tz. (1,
ou co (.((z-1..4-1)
(26)
0tty
01t;11
Otty
0tt-1
where in general additional quantities in 6 cannot be reduced one quantity together with linear combination of 2; and 21-1-
di + 0i-1
As a result, one can conclude from the above derivation that the
E SAMPLING TIME AND COMPUTATIONAL OVERHEAD OF IIA-EDM
Table 9: Comparison of sampling time between EDM and IIA-EDM seconds) over a GPU (NVIDIA RTX
2080Ti). The batchsize was set to 200, See Table 10
for the computational overhead of IIM-DDM.
NFEs
11
13
15
17
19
21
23
EDM
5.1
6.1
7.1
8.2
9.2
10.3
11.4
CIFARIO
1IA-EDM
5.2
6.3
7.3
8.4
9.3
10.4
11.4
EDM
12.4
15.0
17.4
20.0
22.4
24.9
27.3
FFHQ
IIA-EDM
12.6
15.2
17.5
20.0
22.5
25.0
27.4
EDM
12.6
15.0
17.5
19.9
22.3
24.9
27.4
AFHQV2
IIA-EDM
12.7
15.1
17.6
20.0
22.4
24.9
27.5
Table 10: One- time computational overhead (in seconds) of IIA-DDM for computing the optimal coefficients via MMSE.
NFEs
11
15
17
19
21
23
CIFAR10
17.6
21.9
25.9
30.0
34.1
37.3
41.8
FFHQ
42.8
52.6
62.0
71.9
80.3
91.6
102.1
AFHQV2
42.7
52.3
62.2
72.4
82.0
92.0
101.8
The GPU (NVIDIA RTX 2080Ti) utilized for measuring the processing time (in seconds). The hyper-parameter parameter (IB), M, r) was set to (M,r) = (200, 3, 1) as in the paper, where [B] denotes the number of samples in set B of initial noise vector 2 r 1 refers to the case that only the gradient of the most recent time step is being utilized in HI-EEMM, which should not take much memory space.
F TESTED PRE-TRAINED MODELS FOR IIA-DDIM, -IA-SPNDM AND IT-IPNDM
16

---

Published as a conference paper at ICLR 2024
Table 11: Tested pre-trained models in Fig. and Fig.
1.ddim cifar10.ckpt
2.ddim Isun bedroom.ckpt
3.ddim 3.ddim_lsun_chee hww___uuccccccckptt
G PERFORMANCE OF II-SPPNM AND IIA-IPNDM
G.1 DESIGN OF -IA-SPDDM
IIA-SPNDMM is designed to solve variance-preserving (VP) ODE (i.e., 02 a? in by following a similar procedure for IIA-DDIM presented in Section 3 We summarize the sampling procedure of IIA-SPNDM in Alg. 3 The only difference between IIA-SPNDM and SPNDM is the computation of 2,+1 for i = .. .N- 1, where two additional terms are introduced for better integration approximation. The two coefficients 2%aand 9oo: in Alg. can, in principle, be computed
by performing the following MMSE
(p%o: ph1) = arg min E, ~(0003311 2 Pue (Zi, (4) + 10:00 (2 2 f-2-10 1: 1sf-2) 2))
M-
2
+ ( 6(i-1:5-2)
-t m (zi+ 37 li+ 3
(27)
m=0
where L 4444 (zi, represents the update expression of SPNDM over the time interval [t,, f411)
given by
Pe (2), = a1+1 (2:
of é fis 11)/aa + t// Qp
(28)
and the summation DM-1 - 44 "5
(zi+ y le 3 in 27 provides highly accurate integra- tion approximation by applying SPNDM over fine- grained set of timesteps within the time interval [t,, t.1.. When the two coefficients are manually set to (yp% pfo) = (0, 0) for all i, IIA-SPNDM reduces SPNDM.
From Alg.B) we observe that the method SPNDM or IA-SPNDM exploits 2nd order polynomial of the estimated Gaussian noises (Eg(21-5.k i - f)}/ in estimation of 2;+1 at timestep i > 0. The coefficients (3/2,- 11/2) of the polynomial fixed across different timesteps.
17

---

Published as a conference paper at ICLR 2024
Algorithm 3 Sampling of IIA-SPNDM
Input: 2o ~N, I), (pio- ph P
for i do
2i41 di+1
2, 2 af + v c
a
E[++::]] (E0(2i, E0(2e11,++1)
(a)
t; (z, VI (:((+1)0)/
2i41 0141 / 1 [[t1a]
end for
Denote a u- = t0
for i
..., N do
[ (3c0 k,, 1) é0 12,-1,1-- 1))
a ia (2) v1 0 (1:1-1))/a
(b)
Zi+1 1 11 + - af Elus-1] + (0o( 20(-11-1)
+ t (citi-1) (((-1:1-2)
end for
output: zN
The update for 21 in (a) is referred to pscudo improved Euler step in Liu al. (2022)
The update for Z:+1 in (b) is referred to as pscudo linear multi step Liu 2022
IIA-SPNDM reduces to SPNDM when {veto 0, pi
0
G.2 SAMPLING PROCEDURE OF IIA-IPNDM
In brief, IPNDM is 4th-order ODE solver Zhang & Chen (2022) as an extension of the PNDM method Liu et al. 2022). At timestep i, the four most recent estimated Gaussian noises (co(zi-j,> j)} are linearly combined to produce more reliable estimated Gaussian noise εg,i. IPNDM then utilizes co,; and z, to compute the next diffusion state 2i+1-
We extend IPNDM to obtain II-IIPDNM, aiming to find out if the IIA technique can assist the sampling performance of IPNDM. The sampling procedure of IIA-PNDM is summarized in Alg. 1 The two coefficients (P% (i)) at iteration are pre-determined by the IIA technique via solving We omit the details here.
quadratic optimisation, which is constructed a similar way as 27
Algorithm 4 Sampling of IIA-IPNDM
Input: zo ~VO I), (plo: !
for i = do
à; (zi VI E0 (zi, i))/a;
Zi+1 04+ th v 1 60
end for
for i 1..., N do
if 1 then
Eg, = (3ea 2;, i) Eo co (2 1))/2
else if small i 2 then
εo. = (23eo(2;, - 16co (2) 5co(2. 2)
else
čo. (55co i) 59co (2-1,, 37 2) 9eo 8 3))
end if
à. = (z, v -fff0..)/a
Z:+1 = 4441 Ee; + v af + p/o - i-11) (Eo. E0,-1)
end for output:
IIAIIPDMM reduces to IPNDM when (5p% = 0, 0, = N01
18

---

Published as a conference paper at ICLR 2024
Algorithm 5 IPNDM
Input: EN (O( I) for i = .N do
Φ; (zi VI afee i))/a;
21-1 1;; + / a²- co(2;
end for
for i N ...., do
if N then
E = (3eo i) c0 co (z++1, 1))/2
else if small i = 2 then
ca. (23eo (): 16.a(z..1, 1) + (zo + 2))/12
else
E0., = (55e0(z1,)) i) - (Z 1) + 37 i + 2) - 9e0(zi4a, 3)//24
end if
d; (z) v1 - aaffta.)/aa
2;-1= 1 + V 0._ E0,3
end for output: 20
G.3 PERFORMANCE COMPARISON
CIFAR10
LSUN bedroom
LSUN church
12
SPNDM IIA-SPNDM
SPNDM
SPNDM
10
IIA-SPNDM
IIA-SPNDM
11
9
10
7
6
10
20
30
40
20
30
40
10
20
30
40
NFEs
NFEs
NFEs
CIFAR10
LSUN bedroom
LSUN church
IPNDM
IPNDM
13
IPNDM
IIAIINDDM
HA-IPNDM
IIA-PNDMM
9
12
8
11
7
10
9-
6
10
20
30
40
10
20
30
40
10
20
30
40
NFEs
NFEs
NFEs
Figure 6: Performance comparison of four sampling methods.
In this experiment, we investigate the sampling performance of four methods: SPNDM, IA--PPNDM. IPDNM, and IIA-IPNDM. The experimental setup follows that of IIA-DDIM in Subection 5.2 and
Section F The tested pre-trained models are listed in Table F
Fig. 6 summarizes the performance of the four sampling methods for small NFEs. It is clear that
IA-SPNDM outperforms SPNDM for CIFAR1O. For L.SUN-berrood and LSUNN-hurch, the performance of IIA-SPNDM and SPNDM is almost identical.
lI
Next, we consider the performance of IIAIPPDMM and IPNDM. It is seen from the figure that for CIFARIO, IIA-IPNDM produces slightly better performance. However, for LSUN-bedroom and LSUN-church, the IIA technique does not help the sampling procedure of IPNDM. This can be explained by the fact that for decreases and then quickly increases in the NFE range of [15 40), which is undesirable. This implies that as the NFE increases from 15 to 40, the accuracy of the integration approximation of
and LSUN-church, the FID score of IPNDM first
19

---

Published as a conference paper at ICLR 2024
IPNDM may not be monotonically increasing. We note that the IIA technique implicitly assumes that a highly accurate integration upproximation for each timeslot [ty, 1.+1] can obtained by performing IPNDM over set of fine- sreeggained timesteps within [t,, f Our above analysis suggests that the assumption of the IIA technique might be violated in the NFE range of [15, 40] for LSUN-bedroom and LSUN-church.
To summarize, the IIA technique improves the sampling performance of SPNDM and IPNDM for certain pre-trained models when the FID score decreases as the NFE increases. On the other hand, the IIA technique does not help with the sampling performance of SPNDM and IPNDM for those pre-trained models where the FID score first decreases and then quickly increases as the NFE increases.
H EXPERIMENTS ON TEXT-TO-IMAGE GENERATION
In our experiment, the pre-trained model used for text-to-image generation over StableDiffusion The three reference methods DDIM, PLMS and OPM-Solver+++ are implemented by StableDiffusion V2 itself.
is
Fig. below summarizes the obtained optimal 3 values (see in IIA-DDIM for the text-to- image generation task. As can be seen, for each NFE scenario, the optimal 3 values are different across different timestep indices. As 4 approaches to ε O, the optimal 3 parameter increases. Furthermore, as NFE increases from 10 to 40, the average of the beta values decreases. From the above analysis, conclude that it is time-consuming to manually tune the parameter B.
10 NFEs
NFEs
0.05
0.03
optimal & values
optimal values
0.04
0.02
0.03
0.02
0.01
0.01
*
0
0
5
10
5
10
15
20
timestep index
timestep index
30 NFEs
NFEs
0.02
0.02
optimal 3 values
optimal values
0.015
0.015
0.01
0.01
0.005
0.005
0
10
20
30
10
30
timestep index
timestep index
Figure 7: Optimal values in IIA-DDIM for classifier-free guided text-to-image sampling.
20

---

Published as a conference paper at ICLR 2024
I
ADDITIONAL IMAGE COMPARISONS
Table 12: text- pro-ppospss in Fig. and
(a)
A bench sitting along side of river next tree
(b)
A blonde boy stands looking happily at the camera
(C)
A double decker bus is moving along stretch of road
(d)
A large black bear standing in a forest
(c)
A blue and light green bus parked at a terminal
f
Two cats sleep together in open case
(g) a black bench and a green and blue bottle
(h)
The sheep graze and eat in a city field
A sheep with horns in grassy green field
Flowers in vase top of a wooden table
(k)
A large white bear standing near a rock
())
man in glasses wearing suit and tie
(m)
A cat that is looking a dog
(n)
A man dressed for the snowy mountain looks at the camera bird standing alone the water looking
(0)
21

---

Published as a conference paper at ICLR 2024
(d)
(e)
DDIM
FESENAS
IIA-DDIM
QUE
(g)
hh)
ii)
DDIM
IIA-DDIM
Figure 8: Comparison of images generated by DDIM and IIA-DDIM 10 timesteps StableDiffusion V2. See Table 12 for input texts.
22

---

Published as a conference paper at ICLR 2024
(i)
(k)
(1)
DDIM
IIA-DDIM
(m)
(n)
(0)
DDIM
IIA-DDIM
Figure 9: Comparison of images generated by DDIM and IIA-DDIM 10 timesteps StableDiffusion V2. See Table 12 for input texts.
23

---

Published as a conference paper at ICLR 2024
EDM
IIA-EDM
Figure 10: Comparison of images generated by EDM and IIA-EDM at 11 NFEs (or equivalently timesteps).
24


Reviews:
[{"review_id": "p7HqsDuMNu", "reviewer": "Reviewer_an6f", "summary": "This work proposes IIA solvers by estimating the coefficients of diffusion ODE solvers. Instead of using the previous analytical coefficients, this work use the ground truth solution (by solving with many steps) at each interval to estimate the coefficients with a MSE loss, and then further accelerate the sampling procedure of diffusion ODEs, which can be understood as \"distill the coefficients\".", "questions": "Please clarify the common design principle for IIA for different solvers. Is there any common principle such that we do not need to one-by-one design them?\n\n====================\n\nThanks for the authors' revisions. I think the revised version addressed my concerns to some extent, so I raise my score to 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is easy to understand and the writing is easy to follow.\n- The proposed method can be used for any previous ODE solvers and further improve them.", "weaknesses": "- Major:\n  - The design for IIA solvers seem to be lack of a principle. For example:\n    - In IIA-EDM, why $z_i-D_\\theta$ is the \"first gradient\"? It is the gradient of what? Because the \"2nd gradient\" is the difference of two $D_\\theta$, it is natural to understand the \"first gradient\" is $D_\\theta$ itself, but not $z_i-D_\\theta$. So what is the basic principle for designing it?\n    - Why DDIM use data-pred model and noise-pred model for IIA? As data-pred model can be equivalently rewritten to noise-pred model, it seems to be equivelent to a linear combination of $z_i$ and $\\epsilon_\\theta(z_i, t_i)$.\n- Minor:\n  - Table 1, Column 6, should be \"IIA-DPM-Solver\" instead of \"IIA-PDM-Solver\".\n  - As far as I known, the dpm variant for SD v2 is 2nd-order multi-step DPM-Solver++, not DPM-Solver. Please clarify the detailed setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ImadxCGdIn", "reviewer": "Reviewer_SzoU", "summary": "This paper proposed to use the improved integration approximation(IIA) technique to numerically solve the reverse ODEs that appears in the ODE-based sampling processes, including EDM, DDIM and DPM-Solver. The authors introduced numerical algorithms based on IIA and explain the algorithms from both the theoretical and experimental perspective. \n\n1. Theoretically, algorithms based on IIA are designed for EDM, DDIM and DPM-Solver. In these algorithms, gradient at each step is estimated by a linear combination of several most-recent gradients, which is expected to be more accurate than to only use one or two most recent gradients. The coefficients in the linear combination are obtained by solving an optimization problem, minimizing the MSE to some highly accurate integral approximation.\n\n2. Experimentally, they verify the effectiveness of these algorithms on EDM, DDIM and DPM-Solver. It is observed that the IIA-based algorithms improves the sampling qualities for low NFEs($\\le 25$).", "questions": "Questions:\n\n1. Comparing the integration approximation in $(9)$ and $(14)$, why do we preserve the factor $\\frac{1}{2}$ in $(9)$ but ignore the two step-size in $(13)$ when we derive $(14)$? If we include the two $\\frac{1}{2}$ factors into the coefficients in $(9)$, how would it affect the numerical results?\n\n2. The MSE optimization is based on a high-accuracy integration approximation. How to choose the $M$ and fine-grained time-step in the high accuracy integration? Would the pre-trained time-step affect the parameters in the high-accuracy integration approximation?\n\nComment:\n\n1. Typo in the second integral in $(10)$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper introduces new numerical algorithms for ODE-based sampling processes based on IIA. Although this paper focuses on EDM, DDIM and DPM-Solver, same idea applied to other ODE-based sampling processes as well. \n\n2. The theoretical formulations of the algorithms are clear.\n\n3. Experimental results are provided and they show the improvements of such algorithms in some cases(small NFEs).", "weaknesses": "1. The paper doesn't provide any theoretical result showing the effectiveness of the IIA-based algorithms.\n\n\n2. According to the experiments, the improvement of the algorithms in FID only happen when NFE is small. It is also not clear why there is no significant improvement on FID with big NFE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5XJEY9R5fv", "reviewer": "Reviewer_8gXX", "summary": "The paper proposes an improvement to the diffusion model integration procedure in order to make the sampling procedure faster. In particular, in order to accelerate the numerical time integration over an interval t_i to t_{i+1}, an optimization problem is solved that optimizes the weights over gradients evaluated at a coarse grid, in order to minimize the error with  respect to the integration using a fine grid. Using the optimized weights, the inference procedure becomes faster by decreasing the number of steps, while the accuracy is kept consistent. The paper illustrates the performance of the algorithm, highlighting its computational gain, over several experiments.", "questions": "Please see the comments above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper discusses an important and timely topic.\n- The contributions of the paper are clearly described \n- Extensive numerical experiments are provided to support the contributions.", "weaknesses": "- The choice of several hyper-parameters is not clear. How does one choose the number and location of grid points for the coarse and find grids? What are the limitations? What will be the computational overhead if one has to search for suitable parameters? How does the computational gain vary as one changes these parameters? \n\n- There is no discussion or comparison with other numerical integration methods. Numerical integration is a classical topic. It will be helpful to include discussion about why the existing methods are not appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "isEKgoUyUx", "reviewer": "Reviewer_t2so", "summary": "This paper proposes improved integration approximation (IIA) for diffusion model sampling. The core idea is to use past gradients in approximating the update term in diffusion ODEs. The parameters before past gradients are settled by minimizing MMSE to the fine-grained Euler approximation. The proposed IIA technique is only a sampling technique and does not require any modification of the pretrained model. Extensive experiments are conducted to illustrate the performance of this technique.", "questions": "Discussed above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The authors studies a sampling technique in diffusion models. The key idea is to approximate ODE more precise to generate better images. This work focuses on a small task but the originality is good.\n\nQuality: The presentation of this work is quite clear.\n\nSignificance: The technique is useful for generating high-quality images, and has potential to be integrated into SOTA diffusion realizations. However, as I'm not an expert in empirical diffusion models literature, the performance should be more carefully evaluated by other reviewers.", "weaknesses": "Some parts of the algorithm can be more clearly discussed:\n - In experiments (Table 2) the authors mostly use $r=1$, that is, only use past two gradients to estimate the integration term. It is better to discuss why to choose such a parameter, probably with some numerical illustrations. In addition, only for BIIA-EDM $r=0$ is chosen, I wonder if it is because of bad performance. Choice of fine-grained timesteps $M$ is also not discussed.\n - Personally I think the IIA idea resembles that of \"Anderson Acceleration\" in optimization literature, but that's not referred in the paper. The authors may gain some insights from it.\n - The authors could illustrate more about the difference between BIIA and IIA. It seems IIA only decomposes the gradient form $0.5d_i+0.5d'_{i+1|i}$ into two terms, and treat the coefficients of such two terms as optimization parameters so as to add flexibility. I wonder how the performance of IIA is compared to BIIA with larger $r$, i.e., with more previous approximations included in MMSE optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)