Enhanced Diffusion Sampling via Extrapolation with Multiple ODE Solutions
Authors: Jinyoung Choi
Junoh Kang
Bohyung Han
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion models
Primary area: generative models

Abstract: Diffusion probabilistic models (DPMs), while effective in generating high-quality samples, often suffer from high computational costs due to their iterative sampling process. To address this, we propose an enhanced ODE-based sampling method for DPMs inspired by Richardson extrapolation, which reduces numerical error and improves convergence rates. Our method, RX-DPM, leverages multiple ODE solutions at intermediate time steps to extrapolate the denoised prediction in DPMs. This significantly enhances the accuracy of estimations for the final sample while maintaining the number of function evaluations (NFEs). Unlike standard Richardson extrapolation, which assumes uniform discretization of the time grid, we develop a more general formulation tailored to arbitrary time step scheduling, guided by local truncation error derived from a baseline sampling method. The simplicity of our approach facilitates accurate estimation of numerical solutions without significant computational overhead, and allows for seamless and convenient integration into various DPMs and solvers. 
Additionally, RX-DPM provides explicit error estimates, effectively demonstrating the faster convergence as the leading error term's order increases. Through a series of experiments, we show that the proposed method improves the quality of generated samples without requiring additional sampling iterations.

Published as a conference paper at ICLR 2025
ENHANCED DIFFUSION SAMPLING VIA EXTRAPOLATION WITH MULTIPLE ODE SOLUTIONS
Jinyoung Choi', Junoh Kang' & Bohyung Han Computer Vision Lab., 'ECE & 'IPAI, Scoul National University, Korea junoh. kang, bhhan snu. snu.ac.k..
ABSTRACT
Diffusion probabilistic models (DPMs), while effective in generating high-quality samples, often suffer from high computational costs due to their iterative sam- pling process. To address this, we propose an enhanced ODE-based sampling method for DPMs inspired by Richardson extrapolation, which reduces numerical error and improves convergence rates. Our method, RX-DPM, leverages multiple ODE solutions at intermediate time steps to extrapolate the denoised prediction in DPMs. This significantly enhances the accuracy of estimations for the final sample while maintaining the number of function evaluations (NFEs). Unlike standard Richardson extrapolation, which assumes uniform discretization of the time grid, we develop more general formulation tailored arbitrary time step scheduling, guided by local truncation error derived from a baseline sampling method. The simplicity of our approach facilitates accurate estimation of numerical solutions without significant computational overhead, and allows for seamless and convenient integration into various DPMs and solvers. Additionally, RX-DPM provides explicit error estimates, effectively demonstrating the faster convergence as the leading error term's order increases. Through a series of experiments, we show that the proposed method improves the quality of generated samples without requiring additional sampling iterations.
INTRODUCTION
Diffusion probabilistic models (DPMs) have emerged as powerful framework for generating high-
quality samples in wide range of applications and domains for images (Ho al. 2020 Song et al.)
20216: Dhariwal & Nichol, 202T Rombach et al. (022). videos (Ho et al. 2022 Singer et al. 2022 Zhou et al. [022]] Wang al. (023). 3D shapes ZZegg et al. 2022), etc. While DPMs demonstrate impressive performance in data fidelity and diversity, they also have limitations, particularly their computational inefficiency due to the sequential nature of sampling. Addressing this issue is crucial
for enhancing the usability of DPMs in real-world scenarios, where time constraints are critical for practical deployment.
The generation process of DPMs can be formulated as a problem of finding solutions to SDEs or ODEs (Song et al. 20211), where the truncation errors of the numerical solutions are highly correlated to the quality of the generated samples. To enhance the quality of these samples, it is essential to reduce truncation errors, which can be achieved by adopting advanced solvers or numerical techniques that improve the accuracy of numerical estimations. In this context, aim lower truncation emror by applying numerical extrapolation to existing sampling methods for DPMs. The key ingredient of the proposed method is Richardson extrapolation, proven and widely used technique in the mathematical modeling of physical problems such as fluid dynamics and heat transfer, which demand high computational resources. While numerous variants and strategies have been studied (Richards) 1997 Botchev & Verwer 2009 Zlatev et al. 2010), its application to DPMs remains unexplored. The method uses a simple linear combination of multiple numerical estimates from different resolutions of a grid to approximate the ideal solution, where the estimates are expected to converge as the resolution becomes finer, ultimately reaching the target value in the limit.

---

Published as a conference paper at ICLR 2025
We propose extrapolation algorithm that is applied repeatedly every & denoising steps of an ODE- ba-ddeed sampling method to improve the accuracy of interme- diate denoising steps. This is achieved by utilizing an additional ODE solution, which is estimated by
4.22
(2)
a single step over an interval of à time steps. Fig- ure [ illustrates this concept with
[(2)
on time steps [t,, f;-1, fz-2], which forms unit block of our extrapolation-buet based sampling. Two ODE solutions single-step and two-step estimations at 4-2 from 4- can be leveraged achieve an approximation closer to the ideal solution _ which is unknown.
26
solution
n-step estimation
extrapolated cstimation
Figure Application of the proposed extrap- olation on two denoising steps (k: = 2) with
time steps of [t,, 14-2, a!"h) denotes
The standard Richardson extrapolation assumes uni- form discretization of the time grid. However, the uniform grid of denoising time steps might be sub- optimal for DPMs; a smaller time interval near the clean sample is often more beneficial (Karras et al.] 2022, Song et al., (021a) given the same number steps. Considering such characteristics of DPMs and the advantages offered by specific time step discretizations, we propose an algorithm applicable to arbitrary schedules, with coefficients determined by the chosen configuration. We observe that grid-aware approach yields better performance than conventional methods.
that steps used by the baseline sampler within the same interval. _|2_2 represents the extrapolated estimation using two ODE solutions at f--2, (1) and 21;
(2)
Although there exist other methods applying extrapolation techniques diffusion models, their usages of extrapolation are somewhat different from ours. For example, /Zhang & Chen (2023); Zhang al. (2023) utilize estimations from earlier steps improve the estimation at the time step, 4,, whereas our approach adopts two denoised estimations at the same time step, t,, to enhance accuracy 4, In addition, the main building block of our approach, Richardson extrapolation, is proven to enhance numerical accuracy and provides an explicit estimate of the error, which allows for clear understanding of the convergence behavior. Furthermore, the implementation of our algorithm is simple and cost-effective because it requires no additional network evaluations and insignificant computational overhead to perform the extrapolation. We refer to the proposed sampling algorithm RX-DPM.
Our main contributions are summarized below:
We introduce an improved diffusion sampler, RX-DPM, inspired by Richardson extrapo- lation, which effectively increases the order of accuracy of existing ODE- samplers without increasing NFEs.
We systematically develop an algorithm for general DPM solvers with arbitrary time step scheduling starting from the derivation of truncation error of the Euler method on non-uniform grid.
Our experiments across various well-known basclines demonstrate that RX-DPM exhibits strong generalization performance and high practicality, regardless of ODE designs, model architectures, and base samplers.
RELATED WORK
There exists substantial body of research that seeks to reduce the computational burden of DPMs while maintaining their performance. One approach in this direction involves exploring alternative modeling strategies for the reverse process. For example, the networks in (alimans & Ho (2022): Song et al. (2023): Kim et al. (2024) learn alternative objectives through knowledge distillation, using the outputs obtained by iterative inferences of the pretrained (teacher) networks. On the other hand, Bao al. (2022) model a more accurate reverse distribution by incorporating the optimal covariance, while Xiao et al. (2022); Kang et al. (204)) implicitly estimate the precise reverse distribution by utilizing GAN components.
While the aforementioned methods require model training, there is also a training free approach that interprets the generation process of diffusion model as solving ODE or SDE (Song al.) 2021b
2

---

Published as a conference paper at ICLR 2025
Karras al. 2022). For instance, DDIM (Song et al. 2021a) proposes to skip intermediate time steps, which is equivalent to solving an ODE using the Euler method with a large step size. To further
improve sampling quality, a large volume of research (Karras et al. 2022, Dockhorn et al. 2022 Liu
et al. 2022 Zhang & Chen 2023 Lu et al. 2022 2023) simply applies classical higher-order solvers customizes them for diffusion models. Specifically, Karras et al. (2022) adopt the second-order
Heun's is method (Süli & Mayers (203) and Dockhorn et al. (2022) apply the second-order Taylor
expansion. In addtion, [Luu et al. (0222) propose pseudo-numerical solver, which combines classical
high-order numerical methods and DDIM.
Building on ODE-based sampling methods, various numerical techniques have been employed to further improve the sample quality. For example, IIA (Zhang et al. (024) optimizes the coefficients of certain quantities to approximate the fine-grained integration. LA-DPM (Zhang al., (023) and DEIS (Zhang & Chen 2023) adopt extrapolation techniques as our method, but with notable differences. Specifically, LA-DPM linearly extrapolates the previous and current predictions of the solution at t = O-the original data manifold-while DEIS uses high-order polynomial extrapolation on the noise prediction function. The key distinction of our approach is that, while they adopt score (noise) predictions obtained at different time steps for each extrapolation, utilize multiple denoised outputs obtained at the same time step.
3 PRELIMINARIES
3.1 DIFFUSION PROBABILISTIC MODELS AS SOLVING AN ODE
For po Pdata and Rd Karras al. (2022) defines marginal distribution at as
(x) = s(t) "p(=/s(t);<(t))
(I)
where p(#; o) = Piata * *N (1) and s()) and (l) are non-negative functions satisfying The probability flow ODE,
s0) = 1, (0) 0, and lim;-+00
e(t) x(3)
dx = (8(1)/((t) (t) (ogpa///oo(///s(()((()))))
x(T) ~ pr)(x),
(2)
matches the marginal distribution. By adopting the specific choices, s(t) 1 and al) = t as
in Karras et al. 2022). Equation 2 is reduced as follows:
dæ
V p(æ; t)dl, ar(T) pr()).
Diffusion models now learn the score function V log p(:; t), which is the only unknown component
in the equation. For sufficiently large T', the marginal distribution (r can be approximated by
N(:: 0, I²)) and the generation process is equivalent to solving for (0) using Equation (3) with
the boundary condition, xTT) ~ (0, ( Since the analytic solution of Equation (3) cannot
be expressed a closed form, numerical methods are used to solve the ODE, Given the time step
scheduling, 0 = to < l1 <... < IN T, the solution is given by
x0)) = xT) + [
- tv log (og)((t
(4)
= xT)) +
- -v log p()
(5)
1= N
where each integration from f; to t can be approximated by ODE solvers such as the Euler method or Heun's method.
3.2 RICHARDSON EXTRAPOLATION
Let the exact and numerical solutions at f = be V* and Vhh), respectively, where h (0 < < 1) denotes the step size. If V lim, V(h) and the order of truncation error is known, Richardson extrapolation (Richardson 1911) identifies faster converging sequence, (h)) For instance, V(h) with truncation error in the order of O(hP) is expressed by
4 V(h) ch. + 0(19)
(6)
3

---

Published as a conference paper at ICLR 2025
for 0 x and * 0. Then, for fixed constant l >> 1,
V = (hhkk)+
kp h" + O(h).
From Equations 6 and (7). eliminating the h" terms, we obtain the solution
V(h, k) k (h/(k//) - (h)
(8)
kP
which has a truncation error of Ohh)), asymptotically smaller than (hh)
RX-DPM
Before discussing the proposed method, we first outline the algorithmic development process for the most simplified problem and then explore an extension to a general DPM solver. The application of our method, RX-DPM, to a specific solver will be referred as RX-[SolverName].
4.1 TRUNCATION ERROR OF EULER METHOD ON NON-UNIFORM GRID
We now derive the truncation error formula for the Euler method non-uniform grid, based on the local truncation error, which results from a single iteration. For intuitive clarity, we consider a mne-dimensional ODE of the form
dx = (ar, t)dl,
where / a smooth function. Suppose that the numerical solution obtained using the Euler method with the discretization points [t,, f......... tj-k] in reverse time order, given the initial condition at,t)) = Zt,, From now on, we denote s17) as the numerical solution at tú obtained by n iterations and zł, as the exact solution at ty. Given h t; ti-x and Xy 1 (t,-/+1 44-) for j 1,..., k. the local truncation error formula of the one-step Euler method, derived from the Taylor expansion, is expressed as
f Iz; (ehf((r, = 2 X,,, O
(9)
Then, the truncation етrоr of the two-step numerical solution is derived as
(2)
= ( - >22hf(f12,)
(10)
1/-
1:-1
1
= - 2",, 2712 + +0hh) - x2hf(2
(11)
1 Xphis(r
2 5/ >0, +0(1)) X2h((a(1))) + Agh (xg
(12)
1
1"
1
2 " Aghh2
2 3"th A212 0(h3) Equuatooo 9 and f smooth)
(13)
= x
2 34h (Ai + 23 +0hh) (:: is smooth).
"
(14)
Inductively, we can obtain the truncation error for the kestep solution as
(k)
2 It.
2 + 0(h3),
(15)
j=1
which approximates of_,, with a truncation error of (
4.2 RX-EULER
We now describe RX-Euler performing extrapolation every & steps on the Euler method. Extrapolation is executed linear combination of two different numerical solutions a(1)) and a(44)_ obtained by the Fuler solver over single step the grid [4i, 4- and i steps the grid (4,,4111.... (.k]))
4

---

Published as a conference paper at ICLR 2025
respectively. To calculate coefficients for extrapolation,
the truncation error derived in
Section 4.1, which can be also applied to Equation (3) in Section [.1 as the ideal score function is
considered smooth; its derivative is Lipschitz continuous, referring to the equation in Appendix B.3
of Karras al. (2022). From Equations
( and (15p, we derive the expressions for Li and (*))
for a constant e as tollows:
Et. li Ti + ch² + +hh²)
(1)
(16)
- Fi Σ x3 +011)
(k)
(17)
Then, by solving the linear system of Equations following extrapolation:
16 and 17
we approximate Ti through the
f=1 13 4-6
(18)
1 ード= A3
which involves a truncation error of O(h²), asymptotically smaller than O(h²).
In the sampling process, we set the initial condition at the next denoising step, ti-k. (k) and repeatedly perform the proposed extrapolation technique every k steps. Because this approach provides provably more accurate solutions at every steps, we can reduce error propagation and expect better quality of generated examples.
The proposed method is applicable to first-order methods in general, including DDIM (Song et al. 202 which is arguably the most widely used DPM sampler. In this context, we bring the interpretation of DDIM as the Euler method applied to the following ODE:
dy = ea (a()
(19)
where y(t xx)) / 1+ 71)² and 7() / - in the variance-preserving diffusion pro- af cess (Song et al. 2021b), i.e., (x] (a (1- (1-oa)) r)) Thus, instead of using the time grid, we compute X in Equation (18) in terms of (()'ss replacing & with the corresponding xt) in the computation, while the other procedures remain unchanged.
RX-Fuler (RX-DDIM) does not require additional NFEs beyond the number of time steps, as the first prediction every k-step-interval can be stored during the computation of (*() and reused to obtain (1)), The only extra computation involves a linear combination of two estimates, which is negligible compared to the forward evaluations of DPMs
4.3 RX-DPM WITH HIGHER-ORDER SOLVERS
We now present the algorithm for general ODE samplers of DPMs including high-order solvers. When the extrapolation occurs every k steps, as in Section 4.2 the error of - resulting from single iteration of an arbitrary ODE method, is given by
Eii = o + ch + 0(1))
(20)
for 0 < q and + O. For L. Equation (17) to obtain the following equation:
(k)
we extend the form of the linear error accumulation observed in
a c xT + e
(k)
Σ XpP NpP++0(m +
(21)
Note that Equation (1 does not hold in general; however, we consider this simplified assumption reasonable, as is consistent with the standard assumption of Richardson extrapolation under uniform
discretization (see Appendix A). Finally, by solving the linear system of Equations (00) and (1). the extrapolated solution is given by
(K) Thit
Σx
k j=1 N3a
(22)
X3
E==1
5

---

Published as a conference paper at ICLR 2025
which approximates = with truncation enror of Ohh)), asymptotically smaller than O(h").
A limitation of this approach is that estimating a T and a (k) t through naive applications of higher- order solvers requires additional network evaluations compared to baseline sampling. However, since Di.-N accessible from the network predictions made for the computation of a!_"), applying our method to higher-order solvers does not increase the NFE provided that intermediate predictions properly stored. We next discuss how this is achieved using specific examples of high-order ODE solvers; the generalization other solvers mostly straightforward.
Before moving forward, we note that high-order solvers typically rely on interpolation-basee -based tech- niques, such as the Runge-Kure Kuttaa method (Sül & Mayers/ (003) and linear multistep method (Timothy 2017), where the former employs evaluations at multiple intermediate points, while the latter leverages evaluations from previous steps.
We consider applying our method with k = 2 to the second-order Runge- Kutta method. A sequence of one-step estimates are given by
RX-Runge-Kutta
_ 2t, - (t: ta + 027,-6) and
(23)
$1
..11) (1:-1 11-2)( + 022;-1-6).
(24)
where 2j (x())(t)), for 1- <tjo6 S tj. Then, we can express the single combined-step estimate at f4-2 as
(1) Dt. Ut, (4 - 1 (a)2; + a22;-8))
(25)
Since is reusable after the calculation of a T only need to compute z,-5', which is approxi- mated as 2;-1 or 2:-1-6 depending the proximity of its time step. This approach allows to
(1)
efficiently extrapolate the solutions without compromising the quality of the generated samples.
TB-AAams-Bashhfrtd Suppose that, by the s-step Adams- method, extrapolation is performed on grid with an interval of h every ls steps. For predefined bj 's, we are given
( = L D4.241 2t1 +
bon
(26)
j=0
Then, we compute T with a step size of kh for extrapolation as
(1) G Ez, + kh Σ
(27)
which requires additional NFEs by storing the previous network evaluations.
Algorithm| I summarizess the procedure of the proposed method with a generic ODE solver under the assumption that N is multitude of is for simplicity; it is simple handle the last few steps by either adjusting & for the remaining steps or skipping the extrapolation.
4.4 ANALYSIS ON GLOBAL TRUNCATION ERRORS
We perform analysis global truncation errors of the Euler method and RX-Euler under the same NFEs. Assume that we solve an ODE satisfying Lipschitz condition from t = to = 0 with NFES N.
Euler Since the Euler method requires a single network evaluation for each time step, the number of time steps allowed is N. Since the local truncation error of the Euler method step size of = 1/N expressed as ch? + Ohh3), the global truncation error is given by
(ch2 +0(3)) = N +O(N-2) =(N-)).
(28)

---

Published as a conference paper at ICLR 2025
Algorithm 1 Sampling of RX-DPM
Require: co(:), NT, ... to = 1: Input: k; (-) (ODE solver),
2: 2T (r
3: for 1 do
4: Imitialization
5: if mod l = then
6:
+- NN-i+1 tN-i-k+1
7:
Pin f- Dty
8:
end
9:
X (tN-i+1 IN -
10:
Din-. + D(0iN-44 IN io
Store Eo(t)'s if neccessary.
Extrapolation (Equation if i mod k = 0 then
22
12:
a (1)
Iw- <- (aIN
N ths -N + h -2:6)
No NFE required.
(k) # ayy-1 2jz Nf
ik)
14:
a IN-=
1-2 -+1--1 2/
Biy (k)
15:
16: end if
17:
end for
return Tto
RX-Euler RX-Euur performs extrapolation every i steps, the extrapolation happens N/k times, where N is equal to the NFEs for the Euler method. The local truncation error for RX-Euler over each & steps, which has the interval of h k/N, expressed ch O and therefore the global truncation error is given by
N R2d K N -
(c'h3 + 0(hh))
(29)
RX-Euler exhibits a higher convergence rate of the global truncation error compared to the Euler method by one order of magnitude. Using the same approach, we can also demonstrate that the proposed method achieves faster convergence of the global truncation error for higher-order solvers.
EXPERIMENT
5.1
IMPLEMENTATION DETAILS
We conduct the experiment with EDM (Karras et al. 2022). Stable Diffusion v2 (Rombach et al. 2022), DPM-Solver (Lu et 2022), and PNDM (Liu al. (022) using their official implementations and provided pretrained models. Throughout all experiments, we retain the default settings from the official codebases, except for additional hyperparameters related the proposed method. For experiments with EDM, DPM-Solver, and PNDM backbones, we generate 50K images and compute FID (Heusel al. (017) using the evaluation code provided their implementations. To evaluate Stable Diffusion V2 results, use the Py PoTch implementation for the computation of FIL and CLIP score with the patch size of 32 x 32.
5.2 VALIDITY TEST
We first evaluate the effectiveness of RX-Euler under the EDM backbone for & E {2, 32, where smaller & values correspond to more frequent extrapolation over the same number of time steps. As
v2-1 -2-11512-emma-pruuedddd.
patch32

---

Published as a conference paper at ICLR 2025
shown in Figure 2 RX- consistently achieves significantly better FID scores than the Euler method across all values of k. In particular, when extrapolation is applied every two steps, i.e., k = 2. approach achieves the best performance across wide range of NFEs. This indicates that more frequent extrapolation leads to more accurate intermediate predictions, effectively mitigating error accumulation in the final samples. Meanwhile, the curves for k 2 3 remain closer that of k than to the Euler method, empirically validating the reduced truncation error derived in Equation (18) for general k. In other words, even sparse extrapolation still has significant impact on the output
quality. For the rest of our results, we set k = 2.
We also compare the proposed method with con- ventional Richardson extrapolation, as formu- lated in Equation (8) with = 2, which em- plays fixed coefficients. This baseline is labeled as Naive (k= 2) in 2 When comparing
CIFAR-10 (32x32)
FIHQ (64x54)
Fuler
Fuler
25
Eo-EEler (K=2)
20
Ex-Euler (f=2)
RX--Euler (le:3)
Ex- (j=3)
20
RX- Edler (led4
15
RX-Euler (fed)
RX-Euler (K = 2) with Naive (k = 2), we find that our method produces superior results. This implies that extrapolation coefficients adapted to arbitrary time step scheduling are more effective than fixed coefficients derived from uniformly discretized time steps. In other words, the pro- posed grid-aware coefficients enable more effec- tive extrapolation than deterministic ones, better capturing the varying importance of precision in DPMs over time (Karras et al. 2022)).
155
(k=22)
Narve (x=22
10
10
10
10
30
Figure 2: Effect of extrapolation on the Euler method with different k's.
5.3 QUANTITATIVE COMPARISONS ON EDM BACKBONE
We compare RX-Euler with other methods on four different datasets-CIIARR (Krizhevsky &
Hinton 2009, FFHQ (Karras et al. 2019). AFHQv2 (Choi et al. (020), and ImageNet (Deng et al. 2009) using the EDM (Karras et 2022) backbone. Following standard practice, we evaluate the performance of dlass-conditional generation on CIFAR-10 and ImageNet while testing unconditional generation on the other two datasets. this experiment, we include Heun's 's method, LA-DPM (Zhang et al. 2023), and IIA (Zhang et al. (024) for comparisons with our approach, RX-Euler. Note that both Heun's ' method and RX-Euler are second-order numerical solvers, offering higher accuracy than the Euler method, while LA-DPM and IIA are techniques refining baseline sampling. To reproduce the results of LA-DPM, we use the Euler method, as it achieves better performance than Heun's s method. For IIA (Zhang al. 2024). we present results from the better-performing variant, selected between IIA and BIIA, as indicated in the original paper.
Figure 3 presents comparison of FIID scores across the evaluated methods over broad range of NFEs. RX-Fuler consistently outperforms the other approaches, particularly at lower NFEs, demonstrating its effectiveness in fast sampling scenarios. While it occasionally falls behind LA- DPM or IIA on CIFAR-I0 within specific intervals, its overall performance remains more stable and superior across the other three datasets, which pose greater challenges due to higher resolutions and increased data diversity.
In the comparison between RX-Euler and Heun's method, while RX-Euler excels at lower NFEs. Heun's 's method performs slightly better at larger NFEs. This implies that selecting more appropriate solver for each interval- between RX-Fuler (extrapolation) and Heun's method (interpolation)- could yield better results. In this regard, one might expect that in the early where predictions are closer to noise and thus less accurate-interpolaatee Based on this reasoning, we experiment with hybrid approach of RX-Euler and Heun's method, labeled as RX+EDM in Figure We find strong performance of this approach when RX-Euler is selectively applied to the middle or last (low-noise few steps, outperforming both Heun's 's method and -ullrr This indicates that there is still room for improvement in our algorithm and provides another direction for future work.
tends to be more stable than extrapolation.
8

---

Published as a conference paper at ICLR 2025
CIFAR-10 cond. (32x32)
FFHQ (64x64)
ARHQQ2 (64x64)
ImageNet cond. (64x64)
18
18
EDM
EDM
EDM
EDM
26
LA-DPM
14
LA-DPM
LA-OPM
14
LA-DPM
IIAEDDM
RX-Euler
RX-Euler
RX-Fuler
RX--uler
RX+EDM
RX+EDM
18
RX+EDM
10
RX-EDM
10
10
100
15
20
10
20
30
40
1.0
20
30
40
1.0
20
30
40
Figure 3: FIDs of RX-Euler, Heun's method (labeled as EDM), LA-DPM and IIA (or BIIA) by varying NFEs on the CIFAR-10, FFHQ. AFHQv2, and ImageNet datasets using the EDM backbone.
NFES 15
NFES 50
NFES 15
NFEs 50
DDIM
RX-DDIM
"A red fire truck driving down road."
"A pair of giraffe standing in big open area."
Figure 4: Qualitative results of DDIM and RX-DDIM based on Stable Diffusion V2. RX-DDIM produces sharper and more detailed backgrounds, especially evident in the left example. On the right, RX-DDIM generates more realistic giraffes, whereas DDIM struggles at NFEs = 50, failing to properly render the giraffe.
Table 1: FID and CLIP scores of DDIM and RX-DDIM using Stable Diffusion V2.
NFEs
15
20
30
50
Method
Metric
FID ())
CLIP (t)
FID (J)
CLIP (t)
FID (L)
CLIP (1)
FID (1)
CLIP (t)
DDIM
19.15
31.727
18.43
31.716
19.00
31.750
18.65
31.711
RX-DDIM
17.24
31.629
17.12
31.721
17.62
31.781
17.83
31.727
5.4 COMPARISONS ON STABLE DIFFUSION
We apply RX-DDIM to Stable Diffusion V2, which provides various conditional generations. For evaluation, we generate 10K 512 512 images from unique text prompts the COC02014 (Lin et al.) 2014) validation set and compute FID and CLIP scores on resized 256 x 256 images. As shown in Table I. our method also performs well on large models. However, we observe that RX-DDIM yields lower CLIP scores at NFEs 15, which we attribute to classifier classffier-guidnce scales. According to Rombach et al. (2022), optimal classifier-fres guidance scales differ across models. Since the default setting tuned for DDIM, RX-DDIM may benefit from further optimization. Figure 4 presents qualitative comparisons between DDIM and RX-DDIM, highlighting the superior image quality of RX-DDIM. Notably, RX-DDIM generates images with more vivid colors, sharper textures, and more realistic object depictions, leading to an overall more natural appearance. We provide more examples for qualitative comparisons between DDIM and RX-DDIM in Figures 13 and 14 of Appendix

---

Published as a conference paper at ICLR 2025
Table 2: FID scores of DPM-Solvers (Lu al. 2022) and RX-DPMs applied to DPM-Solvers on CIFAR-10 and LSUN Bedroom datasets. All baseline results are reproduced under the same setting as RX-DPMs.
CIFAR- 10 (32x32)
LSUN Bedroom (256x256)
Method \ NFES
9
10
12
15
9
10
12
15
DPM-Solver-2
15.06
11.33
7.36
14.67
11,38
6.44
EX-DPM-Solverrs
12.94
9.80
6,53
12.66
10.13
5.72
DPM-Solver-3 12.39
6.76
5.00
8.79
5.37
4.04
RX-DPM-Solver3 11.50
6.62
4,85
8.12
5.18
4.04
Table 3: FID scores of two types of PNDM (Liu et al. 2022) and RX-DPMs applied to each solver on CIFAR-10, CelebA and LSUN Church datasets. Note that S-PNDM and F-PNDM require and 9 additional NFEs to the number of time steps, respectively. The baseline results are copied from PNDM (Liu et al. 2022).
CIFAR-10 (32x32)
CelebA (64x64)
LSUN Church (256x256)
Method of steps
10
20
5
10
20
5
10
20
S-PNDM
18.3
8.64
5.77
15.2
12.2
9.45
20.5
11.8
9,20
RX-S-PNDM
19.69
7.64
4.72
11.56
9.22
6.89
21.15
10.96
8.96
F.PNDM
18.2
7.05
4.61
11.3
7.71
5.51
14.8
8.69
9.13
RX-F-PNDM
6.60
3,99
7.10
4.99
8.85
9.41
5.5 COMPARISONS ON HIGHER-ORDER SOLVERS
We further apply the proposed method to advanced ODE samplers with higher-order accuracy. Table presents the effectiveness of RX-DPM when applied to DPM-Solvers (Lu et al. 2022) CIFAR- 10 and LSUN Bedroom Yu et al. 2015). Among the variations of DPM solvers, we utilize the single- step versions of DPM-Solver-2 and DPM-Solver-3. Note that, since single-step DPM-solver-n can be considered as an n.th -order Runge-Kutta-like Kutta-like solver, we apply RX-DPM with n + 1 in Equation 22 for the DPM-solver-n. Additionally, we compare our method with another accelerated diffusion sampler, DEIS (Zhang & Chen 2023). for class-conditioned image generation on ImageNett (64 64) in Table 4 of Appendix [..] As shown in the results, RX-DPM consistently achieves the best performance across all NFEs.
As another type of advanced sampler, we consider PNDM (Liu et al. 2022p. S-PNDM and F-PNDM employ linear multistep methods, specifically the 2-step and 4-step Claam-Baashforth methods, respectively, except for the initial few time steps. Accordingly, we apply RX-DPM with p = 3 and (22) for S-PNDM and F-PNDM, respectively. The results the CIFAR- CelebA (Liu et al., 2015), and LSUN Church (Yu et al.) 2015) datasets are presented in Table 3
p 5 in Equation
While RX-DPM improves performance in most cases, exception arises with F-PNDM the LSUN Church dataset, where RX-DPM does not provide a clear advantage. Upon analysis, we observe that the baseline performance of F-PNDM is highest when using 10 time steps and degrades as the number of steps increases (Liu et al. 2022). Since RX-DPM enhances accuracy by leveraging improvements in the baseline solver with finer time steps, its effectiveness is limited when the baseline itself deteriorates under finer discretization. A similar phenomenon with F-PNDM on LSUN datasets has also been reported in IIA (Zhang et al. 2024).
6 CONCLUSION
We introduced RX-DPM, an advanced ODE sampling method for DPMs that leverages extrapolation based on two ODE solutions derived from different discretizations of the same time interval. Our algorithm computes the optimal coefficients for arbitrary time step scheduling without additional training and incurs no extra NFEs by utilizing past predictions. This approach effectively reduces truncation errors, resulting in improved sample quality. Extensive experiments well-established baseline models and datasets confirm that RX-DPM surpasses existing sampling methods, offering a more efficient and accurate solution for DPMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was partly supported by Samsung Research, Samsung Electronics Co., Ltd., and the Institute of Information & communications Technology Planning & Evaluation (IIP)) grants [RS- 2022-11220959 NN..2022-0-00959) (Part Few-Shot Learning of Causal Inference in Vision and Language for Decision Making; No.RS-2021 RAS-2021-112122 12261008,, Al Innovation Hub (AI Institute. Seoul National University);
1343, Artificial Intelligence Graduate School Program (Seoul National University) funded by the Korea government (MSIT).
ETIICS STATEMENT
This paper utilizes pretrained diffusion probabilistic models to generate high-quality images while prioritizing efficiency. As result, our approach does inherently introduce hazardous elements. Nonetheless, we acknowledge the potential for unintended misuse, including the synthesis of inap- propriate or sensitive content.
REPRODUCIBILLIT STATEMENT
We provide detailed implementation instructions and reproducibility guidelines in Section 5.1 along with the pseudo-code of the core algorithm in Algorithm T The full implementation is available at
https:// wtw.ttthhcooommcaaemeeoeeeeeeoet com. wttp://o1n////////o/mmm dpm.
REFERENCES
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In ICML, 2022.
Mike A Botchev and Jan G Verwer. Numerical integration of damped maxwell equations. SIAM Journal on Scientific Computing, 31(2):1322-133 2009.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wer Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusson models beat GANs on image synthesis. In NeurIPS, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. GENIE: Higher-order denoising diffusion solvers. In NeurlPS, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. NeurlPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. NeurIPS, 2022.
Junoh Kang, Jinyoung Choi, Sungik Choi, and Bohyung Han. cettvvattionuuuidee diffusion proba- bilistic models. In CVPR, 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurIPS, 2022.
Dongjun Kim, Chich-Hsin Lai, WeiiHsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Ue- saka, Yutong He. Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. In ICLR, 2024.
11

---

Published as a conference paper at ICLR 2025
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll' a r, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. CoRR, abs/ 405.0312, 2014. URL
1405. 0312
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In ICCV, 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. In ICLR, 2023.
Shane Richards. Completed richardson extrapolation space and time. Communications in numerical methods in engineering, 13(7):573-582, 1997.
Lewis Fry Richardson. Ix. the approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application to the stresses in masonry dam.
Philosophical Transactions of the Royal Society of London, 4700459-400)3557--05 https://doi.org//II 1098/ssta.9911.0009
1911. doi:
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Videe Text-to-video generation without tixt-video data. In ICLR, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
Endre Süii and David Mayers. An Introduction to Numerical Analysis. Cambridge University Press, 1 edition, 2003. ISBN 0-521-00794-1.
Sauer Timothy. Numerical Analysis. Pearson, 3 edition, 2017. ISBN 2017028491, 9780134696454, 013469645X.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elScope text-to-video technical report. arXiv preprint 1.0kv:::308.6557. 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with
denoising diffusion GANs. In ICLR, 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint /506.
2015.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: Latent point diffusion models for 3DD shape generation. In NeurIPS, 2022.
12

---

Published as a conference paper at ICLR 2025
Guoqiang Zhang, Niwa Kenta, and W. Bastiaan Kleijn. Lookahead diffusion probabilistic models for refining mean estimation. In CVPR, 2023.
Guoqiang Zhang, Niwa Kenta, and W. Bastiaan Kleijn, On accelerating diffusion-based sampling process via improved integration approximation. In ICLR, 2024.
Qinsheng Zhang and Yongxin Chen, Fast sampling of diffusion models with exponential integrator. In ICLR, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv. Yizhe Zhu, and Jiashi Feng. MagicVides Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
Zahari Zlatev, István Faragó, and Ágnes Havasi. Stability of the richardson extrapolation applied together with the 6-method. Journal of Computational and Applied Mathematics, 235(2):507-517, 2010.
13

---

Published as a conference paper at ICLR 2025
A JUSTIFICATION ON EQUATION 21
Assuming a uniform grid as in the context of conventional Richardson extrapolation in Section 3.2 and the ODE solver with O of local truncation error formula, have
V = ((h) + chP + 0(1+t)) and
(30)
V = V +
c x +
0(h*t)
(31)
since we expect the ( of the global truncation error. Then, we have the following extrapolated solution with truncation error of O(hp-1) by Richardson extrapolation (Equation 8
(hh, k) khV(h/k) - (h)
(32)
kP
Now, considering the case of Equations
20 and 21
with uniform discretization, we have
V (hh + chht1
O(hh+2) and
(33)
V v( h ( ke j jptt O(h²²2)
(
(34)
each correspondingly. Then, the extrapolated solution Vours obtained from solving linear system of Equations (33) and (34) becomes
K. (h//k) -
ours
(35)
which turns to be exactly the same as Equation 32
Thus, we believe our approach can be considered to employ assumptions shared by those used in common practices of Richardson extrapolation and also can reduce global errors which is backed by experimental results as well.
B
MORE QUANTITATIVE RESULTS
B. COMPARISON WITH DEIS
We compare DEIS variants (Zhang & Chen 2023). DPM-Solvers, and RX-DPMs applied to DPM- Solvers on coadiicondi ImageNet (64x 64) Table g The results clearly demonstrate that our method outperforms all other approaches across all NFEs.
B.2 DPMS WITH OPTIMAL COVARIANCES
Although our method is designed for ODE solvers, we also conduct experiments with SN-DPM and NPR-DPM (Bao et al.) (202) CIFAR- 10 (Krizhevsky & Hinton 2009) and CelebA datasets (Liu et al. 2015) to evaluate its performance when applied to SDE solvers. We use the official codes and provided pretrained models as described in Section 5.1 SN-DPM and NPR-DPM are two different models that correct the imperfect mean prediction in the reverse process of existing models through optimal covariance learning.
To incorporate our method into stochastic sampling, we decompose it into deterministic sampling component and stochastic component, and apply our method to the deterministic sampling part. Specifically, within each s-step interval, we execute RX-DPM algorithm using the deterministic sampling component and then add the stochasticity term afterward. this way, our method uses the stochasticity of the baseline sampling method in limited manner compared to the baseline sampling with the same NFEs. For a better understanding of our implementation, we provide the diagrams of the proposed method in Appendix C
In T we show the results on NPR-DPM and SN-DPM along with the vanilla DDIM and LA- DPM (Zhang al.] 2023). which is another extrapolation-basee sampling method. We observe that our method outperforms the compared methods in most cases, although performance degradation is noted with RX SN-DDM the ODE might offset the benefits of the model's optimization. Despite this, we observe significant performance improvements in the most extreme case, NFEs 10.
the CIFAR-10 dataset. This implies that approach of solving
14

---

Published as a conference paper at ICLR 2025
Table 4: Comparisons of DEIS variants, DPM-Solvers and RX-DPMs applied to DPM-Solvers on class-conditioned ImageNet (64x64). All results of DEIS and DPM-Solvers are copied from DEIS (Zhang & Chen 2023) except for the result of DPM-Solver-3 with NFEs
NFEs
Method
10
12
18
30
IAB-DEIS
6.65
3.99
3.21
2.81
PAB-DEIS
9.28
6.46
3.74
2.87
DPM-Solver-2
7.93
5.36
3.63
3.00
pMid-DFIS
9.12
6.78
4.00
2.99
RX-DPM-Solver->
6.11
5.61
3.64
2.93
3
7.45
5.02
3.18
2.84
/Kutta-DEIS
13.12
3.63
2.82
3X-Der--olver-3
3.90
Table 5: FII scores for CIFAR-10 and CelebA on DDIM, NPR-DDIM and SN-DDIM models. The
values for each baseline and LA-DDIM results are copied from Zhang et al. (003).
Dataset
CIFAR- 10 (32x32)
CelehA (64 64)
Method
NFES
10
25
50
10
25
50
DDIM RX-DDIM
21.31
10.70
7.74
20.54
13.45
9.33
14.78
8.42
6.30
18.31
10.54
6.88
NPR-DDIM
13.40
5.43
3.99
14.94
9.18
6.17
LA-NPR-DDIN
10.74
4.71
3.64
14.25
8.83
5.67
RX EX-NPR-DDD
6.35
3,92
3,34
11.58
6.61
SN-DDIM
12.19
4,28
3,39
10.17
5.62
A-SNNDDIM
8.48
3.15
2.93
8.05
4.56
2.93
RX-SN-DDIM
7.50
5.12
4.40
5.20
2.72
2.25
C DIAGRAMS
Figure 5 compares the diagrams of an ODE solver, the proposed method with ODE solver, and the proposed method with an SDE solver.
D COMPUTATIONAL COST
We compare the computational time and GPU memory usage of the Fuler method and RX-Buler using the EDM backbone in Tables 6 and respectively. For measurements, set the batch size to 128 and use 10-step sampling on an A6000 GPU. The average runtime per batch is measured for computational time. The additional operations introduced by our method, which consist of linear combinations of precomputed values, result in negligible computational overhead compared to the time required for the network forward pass. Furthermore, as the model size increases, the relative overhead diminishes (e.g., only 0. % increase for ImageNet consiconniiional sampling). Similarly, GPU memory usage increases slightly with RX-Euler, primarily due to the storage of previous predictions. However, this increase is minimal and decreases as model or data size increases, showing a similar trend to that observed with computational time.
E LIMITATIONS
As our method is primarily designed for an ODE solver, integrate it with an SDE solver (or stochastic sampling method). we partially apply the stochasticity component of the SDE solver as demonstrated Appendix [2.2 Consequently,
some cases, the effectiveness offset because the full effects of stochasticity are not captured. However, in scenarios where NFE is very limited, which are of greater interest to us, the combined effect of RX-DPM and stochastic sampling has been empirically shown to be highly beneficial. We leave the development of methods that can perform better in more general cases for future work. Additionally, in the extension the RX-Eule algorithm to higher-order solver in Section Section 4.3 there remains room for improvement since impose assumptions about linear error propagation. We believe that relaxing these assumptions or deriving more accurate equations could further enhance the performance.
15

---

Published as a conference paper at ICLR 2025
(1)
Lt;
ODE solver
One step of an ODE solver
(1) Pti
ODE salver
ODE solver
(k)
Lt
Eq
(24)
E li
(1)
ODE salver
L t/
(b) l steps of RX-DPM applied to ODE solver
(k)
ODE solver
ODE solver
St+
Add stochasticity
Eti
Eq (24)
(1) E
ODE solver
t/-
(c) ls steps RX-DPM applied an SDE solver
Figure 5: Digarams of the baseline and the proposed sampling methods. The blue-bordered boxes in
(b) and (c) indicate that the corresponding operation does not require network evaluation. The ODE solver in (c) refers to the deterministic sampling component of the SDE solver.
Table Comparison of per-batch computation times between the Euler method and RX-Euler with the EDM backbone. The reported values represent the average runtime across 100 measurements (in seconds).
CIFAR- 10 cond. (32x32)
FFHQ (64x64)
ImageNet cond. (64x64)
Buler
1.737 +: 0.028
3.895 0.023
6.436 0.033
RX-Eule
1.743 0.031
3.903 0.025
6.443 0.039
Table 7: Comparison of GPU memory usage (MiB) during inference between the Buler method and RX-Euler with the EDM backbone.
CIFAR- cond. (32x32)
FFHQ (64x64)
ImageNet cond. (64x64)
Euler RX-Fuler
2929
9433
12661 12705
3033
9477
QUALITATIVE RESULTS
We provide qualitative results using the EDM backbone in Figures 6
12 and Stable Diffusion V2 in
Figures 13 and 14
16

---

Published as a conference paper at ICLR 2025
Euler (NFES= 10, FID: 15.88)
EDM (NFEs 11, FID: 14.46)
RX-Euler (NFEs 10, FID 4.35)
RX+EDM (NFES 10. FID 4.26)
Figure 6: Qualitative results of CIFAR-10 of different sampling methods with EDM backbone.
17

---

Published as a conference paper at ICLR 2025
Euler (NFEs 9, FID:: 24.27)
EDM (NFES 9, FID: 57.12)
RX-Eulee (NFEs FID: 12.94)
RX+EDM (NFEs FID: 6.83)
Figure 7: Qualitative results of FFHQ of different sampling methods with EDM backbone.
18

---

Published as a conference paper at ICLR 2025
Euler (NFES 11, FID: 18.30)
EDM (NFES 11, FID: 29.34)
RX-Euler (NFEs 10. FID: 7.20)
RX+EDM (NFEs 11. FID: 5.50)
Figure 8: Qualitative results of FFHQ of different sampling methods with EDM backbone.
19

---

Published as a conference paper at ICLR 2025
Huler (NFFs- 9, FID: 11.81)
EDM (NFEs 9, FID: 27.92)
RX-Euler (NFEs 8, FID: 6.49)
RX+EDM (NFEs FID: 3,88)
Figure 9: Qualitative results of AFHQv2 of different sampling methods with EDM backbone.
20

---

Published as a conference paper at ICLR 2025
Huler (NFEs 11, HID: 8.94)
EDM (NFEs = 11, FID: 13.61)
RX-Euler (NFEs 10. FID: 4.02)
RX+EDM (NFEs 10. FID: 3.72)
Figure 10: Qualitative results of AFHQv2 of different sampling methods with EDM backbone.
21

---

Published as a conference paper at ICLR 2025
Huler (NFEs 9, FID:: 20.49)
EDM (NFEs 9, FID: 35.34)
RX-Euler (NFEs FID: 12.38)
RX+EDM (NFEs FID: 7.94)
Figure 11: Qualitative results of ImageNet of different sampling methods with EDM backbone.
22

---

Published as a conference paper at ICLR 2025
Euler (NFEs 11, FID: 14.84)
EDM (NFEs 11, FID: 15.35)
RX-Euler (NFEs 10. FID: 6.95)
RX+EDM (NFEs 11. FID: 5.83)
Figure 12: Qualitative results of ImageNet of different sampling methods with EDM backbone.
23

---

Published as a conference paper at ICLR 2025
NFCs 15
NFEs 20
NFES 30
NFBs 50
DDIM
RX- -D-D
"A red fire truck driving down road." "
DDIM
RX RX-DDIM
Men and women are posing for photo a conference room."
DDIM
RX-DDIM
pair of giraffe standing a big open area."
Figure 13: Qualitative results Stable Diffusion V2 of DDIM and RX-DDIM.
24

---

Published as a conference paper at ICLR 2025
NFCs 15
NFEs 20
NFEs 30
NFEs 50
DOIN
RX
"Small wooden bench sitting the middle of a forrest."
DDIM
RX DDIM
"A plate with a crepe covered with fresh cut fruit."
DDIM
RX-DDIN
"Very many woolly sheep a field which dry."
Figure 14: Qualitative results on Stable Diffusion V2 of DDIM and RX-DDIM.
25


Reviews:
[{"review_id": "VQIgkEMUFs", "reviewer": "Reviewer_Fth1", "summary": "The work considers the problem of accelerating the inference of diffusion probabilistic models (DPM). Different from the distillation methods, the work adopts the ODE perspective of DPM and aims to propose an accelerated solver for solving the ODE associated with DPMs. The particular technique the work leverages is the Richardson extrapolation (RX), which is proven to achieve lower numerical errors given the same amount of iterations. Analogous to the RX in Euler, the authors derive the ODE solver for DPMs (eq. 22-23). Note that a rigorous mathematical justification of eq 22-23 is not presented, but a simple derivation of the truncation error in terms of total iteration number N is given (assuming eq. 22-23 to be true).\n\nTwo different experiments are conducted to demonstrate the performance of RX-DPMs. The first experiment focus on the RX-Euler case, which can be derived rigorously. The experiment clearly shows that RX-Euler leads to faster convergence. The second experiments focuses on RX-DPMs. The results demonstrate faster convergence of RX-DPMs than other DPMs, which, however, is not as significant as RX-Euler over Euler.", "questions": "---- Methodology ----\n1. The authors are advised to define what truncation error means in the first place, as readers may have different interpretations of the error.\n2. As pointed out in the weakness, the authors are advised to provide a more clear and rigorous justification of eq. 22-23.\n3. Some symbols are used before proper definition, such as $\\lambda$.\n\n---- Experiment -----\n1. [Fig.2] Please show the dimensionality of the images from CIFAR-10 and FFHQ.\n2. [Fig.3] The authors are advised to explain why not all algorithms start at NFE=0.\n3. A paragraph properly discussing the limitation of the proposed method is recommended. Besides RX may not be extended to SDE, another clear limitation of the method is the lack of rigorous mathematical justification.\n4. The current experiment does not include any other fast DPM methods. To help properly evaluate the performance, the authors are advised to compare RX-DPMs with either a distillation-model-type or consistency-model-type method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally easy to follow.\n2. The introduction of Richardson extrapolation is interesting.", "weaknesses": "1. The limitations of the proposed method is not well discussed.\n2. The justification of equation 22 needs improvement on its clarity and rigor.\n3. The second weakness leads to a disconnection in the logic flow from RX-Euler to RX-DPMs, which reduces the significance of the second contribution considered by the authors, that is, the proposal of an RX method on a non-uniform grid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7IlsS4xIBY", "reviewer": "Reviewer_6Qbz", "summary": "Inspired by the Richardson extrapolation, the author propose an enhanced ODE-based sampling method called RX-DPM for DPMs to reduce numerical errors and improve convergence rates. The author outline the algorithmic development process for the most simpliﬁed problem and explore an extension to a general DPM solver. The experiments show that RX-DPM can reduce the truncation error and achieve better sample qualities.", "questions": "1. How to apply propose method into the SDE sampler?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. propose an diffusion sampling method called RX-DPM, inspiring by richardson extrapolation. The method can achieve accurate estimation of numerical solutions without additional computational overhead.Besides, the method is applicable to arbitrary discretizations of time steps.\n\n2. The experiments show that RX-DPM can reduce the truncation error and achieve better sample qualities. RX-DPM shows strong generalization and practical effectiveness, performing well across various ODE designs, architectures, and base samplers.", "weaknesses": "1. The value of $p$ is not provided, and the experiment does not show how the sample quality changes as order $p$ increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XMZU3i8dfF", "reviewer": "Reviewer_UczB", "summary": "The authors present an enhanced diffusion sampler called RX-DPM, inspired by Richardson extrapolation, which significantly improves the accuracy of existing ODE-based samplers. They create an algorithm for arbitrary discretization specifically designed for DPMs.\n• In the paper they outline utilization of the algorithm utilizing Richardson extrapolation for general DPM solvers, accommodating arbitrary time step scheduling, beginning with the derivation of the truncation error formula for the Euler method on a non-uniform grid. They offer implementation details applicable to various diffusion samplers without adding extra function evaluations.\n•The experiments with several established baselines reveal that RX-DPM demonstrates robust generalization capabilities and high practicality, irrespective of the ODE structures, architectures, or foundational samplers used.", "questions": "What is the difference of the proposed method with alternative methods in the literature in terms of sampling time?\nWhy extrapolation does not require additional NFE? In which scenario us the method is applicable for SDE?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors introduce an innovative sampling technique for diffusion models. This method, RX-DPM, appears to surpass previous sampling methods due to its incorporation of Richardson extrapolation, specifically adapted for DPMs. The algorithm effectively reduces local truncation error, resulting in improved sample quality.\nThe authors demonstrated the effectiveness of RX-DPM through experiments with established baseline models and datasets, as well as by comparing RX-DPM to other sampling methods.", "weaknesses": "There is a notable absence of performance comparisons regarding computation time and FLOPS, as the study only evaluates the number of function evaluations (NFE) of the algorithm without contrasting the computational costs of each step of RX-Euler with the standard Euler method for DPMs.\nFurthermore, most of the experiments involving NFE variation focus on low to moderate-resolution images, not exceeding 256 by 256 pixels, while leading diffusion models are capable of generating high-resolution images, such as 512 by 512 pixels or more. Consequently, the study lacks a thorough quality comparison of the methods across high-resolution datasets like AFHQv2 and FFHQ, particularly for varying NFEs beyond 10 or 11", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d99ClI5lI4", "reviewer": "Reviewer_xBMM", "summary": "The authors proposed an enhanced ODE-based sampling method for DPMs based on Richardson extrapolation. They claimed that they used non-uniform discretization approach with arbitrary time step scheduling and utilized an additional ODE solution over an interval of k time steps.", "questions": "1. In page 4, line 127 p(x, \\sigma) = p_data \\dot N(0, sigma^2 I), the right hand side of the equation does not make sense to me.\n2. I believe the local truncation error at line 196/197 in the right of eqn. (10) should not have the first term. Please check. As a result, if this is a mistake, it propagates into the following equations, and might cause major technical issue for this paper. \n3. From eqn. (13) to eqn. (14) to eqn. (15), in page 4, the function f disappeared, the only reason provided was because function f is smooth. I wonder if there is an error here. They could have used the Appendix to explain the soundness of this algorithm.\n4. The authors mentioned that eqn. (17) in page 5 is obtained from eqn. (16) in page 4; however, eqn. (16) has k in both subscript and superscript, one can not randomly set k =1 for superscript, and keep k in subscript; which resulting in two different values.\n5. It is unclear how the authors interpret DDIM as Eqn. (20). Somehow they jumped from equation (3) to eqn. (20) without explanation.\n6. In section 4.3, the authors first proposed an estimate of x_{t_{i-k}}, then jumped to RX-Runge-Kutta in page 4, when they mentioned they \"consider the second order Runge-Kutta ..\", but it did not mention for what reason to consider it. From the notation, I guess is they use to to estimate the truncated error. Then they jumped again into algorithm 1, in which they brought up ode solver \\Phi that was not explained before. The whole section should be reorganized and be rewritten so that the algorithm can be clearly explained. \n7. The paper should explicitly state that their proposed numerical solution is backward although it was time reverse in diffusion model.\n8. The authors only mentioned that \"We acknowledge that there are some\nlimitations to our method’s application in SDE ...\". First of all, they proposed their algorithm for ODE, but they did not really extend it to SDE. Second, they did not state what is their limitation. Third, I would appreciate some discussion on the sensitivity of this algorithm as well as complexity cause by using the Richardson extrapolation because it is actually a more complicated way to calculate the derivative approximation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of using Richardson extrapolation for ODE-based sampling method is not done in diffusion model although the idea has been explored in solving odes.", "weaknesses": "1. The technical part (section 4) of the paper is poorly written. I am concerned that there might be some technical errors in the formulas that lead to the main algorithm. The authors need to check carefully and they also need to refer to some existing mathematical results on using Richardson extrapolation in numerically solving odes. Also the mentioned that one of the contribution is arbitrary discretization of time steps; however it is not clear if they have actually implemented it.   \n\n2. The experiment part is not well written either. It is ambiguous on the relationship between RX-DPM  and RX-EDM and RX-DDIM, \n \nI believe this paper needs a major revision before it is published. Please also refer to the following section - Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)