UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models
Authors: Wenliang Zhao
Lujia Bai
Yongming Rao
Jie Zhou
Jiwen Lu
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion models
fast sampling
predictor-corrector
training-free
TL;DR: We propose a new framework for the fast sampling of diffusion models called UniPC, consisting of a corrector (UniC) and a predictor (UniP) that share a unified analytical form and support arbitrary orders.

Abstract: Diffusion probabilistic models (DPMs) have demonstrated a very promising ability in high-resolution image synthesis. However, sampling from a pre-trained DPM is time-consuming due to the multiple evaluations of the denoising network, making it more and more important to accelerate the sampling of DPMs. Despite recent progress in designing fast samplers, existing methods still cannot generate satisfying images in many applications where fewer steps (e.g., $<$10) are favored. In this paper, we develop a unified corrector (UniC) that can be applied after any existing DPM sampler to increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC, we propose a unified predictor-corrector framework called UniPC for the fast sampling of DPMs, which has a unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps. We evaluate our methods through extensive experiments including both unconditional and conditional sampling using pixel-space and latent-space DPMs. Our UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet 256$\times$256 (conditional) with only 10 function evaluations. Code is available at https://github.com/wl-zhao/UniPC.

UniPC: A Unified Predictor-Correctnn Framework for Fast Sampling of Diffusion Models
Wenliang Zhao"
Lujia Bai"
Yongming Rao
Jie Zhou Jiwen Lu
Tsinghua University
Abstract
Diffusion probabilistic models (DPMs) have demonstrated very promising ability in high-resolution image synthesis. However, sampling from pre-trained DPM is consuminn due to the multiple evaluations of the denoising network, making it more and more important accelerate the sampling of DPMs. Despite recent progress in designing fast samplers, existing methods still cannot generate satisfy- ing images in many applications where fewer steps (e.g., <10) favored. In this paper, we develop a unified corrector (UniC) that can be applied after any existing DPM sampler increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC, we propose a unified predictor-corrector framework called UniPC for the fast sampling of DPMs, which has unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps. We evaluate our methods through extensive ex- periments including both unconditional and conditional sampling using pixel-space and autent-space DPMs. Our UniPC can achieve 3.87 FID on CIFAR 10 (uncondi- tional) and 7.51 FID on ImageNet 256 (conditional) with only function evaluations. Code is available at https: // github. com/ Eaazzzha/Unnn
1 Introduction
Diffusion probabilistic models (DPMs) 13 35 have become the new prevailing generative models and have achieved competitive performance on many tasks including image synthesis [8] 29 B video synthesis
(15) text-to-image generation 27 29 [2], voice synthesis (5), etc. Different from GANs [10 and VAES (20), DPMs are trained to explicitly match the gradient the data density (i.e., score), which is more stable and less sensitive to hyper-parameters. However, sampling from a pre-trained DPM usually requires multiple model evaluations to gradually perform denoising from Gaussian noise [33 consuming more inference time and computational costs compared with single-step generative models like GANs.
Recently, there have been increasing efforts to accelerate the sampling of DPMs 31 28
25
40
Among those, training-free methods 34 25 40 can be directly applied to off-the-shelf trained DPMs. Although these methods have significantly reduced the sampling steps from 1000 to less than 20 steps, the sampling quality with extremely few steps (e.g., <10) has been rarely investigated. Few-step sampling can be used in many scenarios where we need to efficiently obtain plausible samples, such as designing a proper prompt for text- to-image diffusion model (29 30 and computing losses the generated samples during the training of some diffusion-basee based visual systems [1] 6). However, current fast samplers usually struggle to generate high-quality samples within 10 steps (see Figure
enjoy wider usage in applications because they
'Equal contribution.
'Correspondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

DDIM (order = 1)
DEIS (order 2)
DPM-Solver++ (order 2)
UniPC (order (Ours)
34
40
Figure 1: Qualitative comparisons between our UniPC and previous methods. All images are generated by sampling from DPM trained on ImageNet 256x 256 with only 7 number of function evaluations (NFE) and a classifier scale of 8.0. We show that our proposed UniPC can generate more plausible samples with more visual details compared with previous first-order sampler [34] and high-order samplers [40 26]. Best viewed in color.
In this paper, we propose training-free framework for the fast sampling of DPMs called UniPC. We find that UniPC significantly outperforms existing methods within 5~ 10 NFE (number of function evaluations), and can also achieve better sampling quality with more sampling steps. Specifically, we first develop a unified corrector (UniC) which works by using the the model output Eø ( t.) at the current timestep t; to obtain a refined oi Different from the predictor-correctte paradigm in numerical ODE solver that requires a doubled NFE, our UniC reuses the model output eø (#t, 4.) the next sampling step, thus introducing no extra function evaluation. UniC can be applied after any existing DPM sampler to increase the order of accuracy, while the inference speed is almost unaffected. Interestingly, also find that by simply changing hyper-parameter in UniC, new family of predictors (UniP) can be further obtained.
Since UniC is method-agnostic samplers called UniPC. Different from previous fast solvers 26 25 40) that either have no higher- order (e.g., > 3) variants have explicit forms, UniPC supports arbitrary orders with unified analytical expression and are easy to implement. Benefiting from the universal design, variants of be easily derived. We theoretically
combine UniP and UniC to obtain
family of fast
UniPC (e.g., singlestep/multistep.. noise/data prediction) prove that UniPC enjoys higher convergence order and empirically demonstrate that UniPC has better sampling quality in variety of scenarios. We also show that the inference speed and memory usage of UniPC is similar to DPM-Solver++ [26). indicating that UniPC can achieve superior performance under the same computational budgets.
We conduct extensive experiments with both pixel-space and latent-space DPMs to verify the effectiveness of the proposed UniPC. Our results show that UniPC performs consistently better than previous ofte--of-theart methods on both unconditional and conditional sampling tasks. Notably, UniPC achieve 3.87 FID CIFAR10 (unconditional) and 7.51 FID ImageNet 256 256 (conditional) with only 10 function evaluations. We also demonstrate that UniC can improve the sampling quality of several existing fast samplers significantly with very few NFE (number of function evaluations). Some qualitative comparisons are shown in Figure I where we observe that our UniPC can generate images with more visual details than other methods.
Background and Related Work
2.1 Diffusion Probabilistic Models
For a random variable To with an unknown distribution go(x)), Diffusion Probabilistic Models (DPMs) [33 13 19] transit go (2x) at time normal distribution qr (r(x)) (N 52,))) at time T for some ā > 0 by gradually adding Gaussian noise to the observation æo. For each time t [0, T), and given dt, 0t > the Gaussian transition is
q: (xt/2x))) ) = =- ( of I.
where of (the signal-to-noise- (SNR)) is strictly decreasing w.r.t. t 19

---

Let (x t) denote the noise prediction model using data to predict the noise e, and the parameter 0 is obtained by minimizing
(12]
where 2i ~ 4o (o()))) E (/ 7J, and the weight function w(1) > Sampling from DPMs can be achieved by solving the following diffusion ODEs [35]:
da:
(2) 20.
= =(t))=
(0,,,,), [0. T), æT No0, 8
(I)
dt
where f(t)
log oi
(²)
do dt
gdloog log o7.
dt
dt
2.2 Fast Sampling of DPMs
Fast samplers of DPMs can be either training-based 31 37 or training-free
25
26
40
24
41
training-based samplers require further training costs while training-free methods directly use the original information without re-training and are easy to implement in conditional sampling. The essence of training-free samplers solving stochastic differential equations (SDEs)
or ODEs 26 40 24
34
25
Other fast sampling methods include modifying DPMs 9 and the
combination with GANs
38
36.-
Among others, samplers solving diffusion ODEs are found to converge faster for the purpose of sampling DPMs [34] 35). Recent works 40 25 26 show that ODE solvers built exponential
integrators 17 appear to have faster convergence than directly solving the diffusion ODE I The
solution 2, of the diffusion ODE given the initial value Ig can be analytically computed as 25
02 x.
///a Es A)dA, 2
at
a55
where we use the notation éω and Φ to denote changing from the domain of time(t) to the domain of half log-SNR(A), Le., A = log(ot//0)) at (A) and (o(:; == co(- (o(:xx(A)) (X)).
Based the exponential integrator, (5 proposes approximate ég via taylor expansion and views DDIM as DPM-Solver-1, i.e.,
Zt.-
at.
:t.. (0,(e" - (Q):)
(3)
at
26 considers rewriting 2 using te instead of és: [40 derives the taylor expansion formulae
with respect to t instead of the half log-SNR(A). (24) employs pseudo numerical methods such as
Runge-Kutta method directly for the updating of εœ of (3. Although many aforementioned high-order solvers are proposed, existing solvers of diffusion ODEs can be explicitly computed for orders not greater than 3, due to the lack of analytical forms.
3 A Unified Predictor-Corrector Solver
In this section, we propose a unified predictor-corrector solver of DPMs called UniPC, consisting of UniP and UniC. Our UniPC is unified in mainly two aspects: 1) the predictor (UniP) and the corrector (UniC) share the same analytical form; 2) UniP supports arbitrary order and UniC can be applied after off- shel fast samplers of DPMs increase the order of accuracy.
3.1 The Unified Corrector UniC-p
Modern fast samplers based on discretizing diffusion ODEs 25 34 40 aim leverage the previous points { 12 samplers, the quality of few-step sampling still has room for improvement. In this paper, we propose corrector called UniC-p to improve the initial estimation using not only the previous p points but also the current point. Formally, after obtaining the initial estimation Zt,, we perform the correction step through the following formula:
=k=1 to estimate Tt, with order of accuracy. Despite the rapid development of fast
De
or,( (t-))) o₁, B(h.)
Σ
am D
m.
0:.-1
m
3

---

Algorithm UniC-p
Algorithm UniP-p
Require: (re fr network, any p-order solver Solver-p. p. buffer Q {ea (1
Require: {ri]-= {Eo ( (i-b)
network, a buffer Q
his - Xt, At;-1. Lt; + Solver-por - for m = to do
h: s de, Xe,_1
for m = 110 do
tx(mnh homb )
t- (rmll Xr; 1)
Dm 60( : 8m) (01 t2-11
D.. Ep(@E : 5m) ea co 1:4i11
end for
end for
Compute ap < (hi)s, (lu) B((u), where Rp, Up are as defined in Theorem 3.1
Compute ap-1 + R 1p (h,,// (hh)). where Rg-1, pp-1 are as defined in Thcorem 3.1
Fi + 04
Fi on (e, (
it ac R1; oc, (c 1)Ed 1 44-1)
-0, B(h,) Em=1 Dm/mm
o. B(b)) Em=l am am/mm/m
Q buffer return: R;
F eo t.)
Q
buffer (0(
return: 3t;
where äs, denotes the corrected result, B(h) Oh) is non-zero function of h, h4 X X is the step size in the half-log-SNR(A)) domain, 71 < < ... < Tp are a non-zero increasing sequence, determining which previous points used. Specifically, between A to Xr to obtain the auxiliary timesteps 8m = (x h + ) 1,2..... The model outputs at these timesteps are used to compute Dm by
use (r:)==) - interpolate
Dm =o( 8m) co 4411.
(5)
We now describe how choose {am P pa in UniC-p to effectively increase the order of accuracy. The main idea is to cancel out low-order terms between the numerical estimation 4 and the theoretical solution (2). practice, we expand the exponential integrator in (2) as follows:
34,
Cit,
The-1 o0, 1)Ea (2
ar,-1
F
Te. hith (x+z((()))))) (a : + O(HPt2)
(6)
where E denotes the k-th derivative of ég., and (h) can be analytically computed [6]. The
{am m= be then determined by matching the coefficients between 44) and (6. In the following theorem, we show that UniC-p has an order of accuracy p + 1 (see Appendix E.3 for detailed proof).
Theorem 3.1 (The Order of Accuracy of UniC-p). For any non-zero sequence {r:)e=n and define
1
rih
rzh
rph (r (1-1)
Rn(h))
(rih)
(rgh)e-1
Let (h) = (0r (h), ... (h)) with (n) = 1:" (h), where sp is defined by the
recursive relation TTo::
in) (p)(h) - 1/n!
pa(h)) = c
h
For an increasing sequence T1 < r2 < < <rp = 1, suppose ap := (aI,..., ap) satisfies,
Ra(ha)a, B(ha) -
(b.h))) ((.))))) Ok)+1)
where . denotes the 1 norm for matrix. Then, under regularity conditions in Appendix E.2 of will have (p + 1) order of accuracy.
UniC-p
The monotonicity of {r.} ensures the invertibility of the Vandermonde matrix Rp. Therefore, can take ap- RR, (ha)) as the coefficient vector for (4) for simplicity, where Bhh)

---

can be any function of h such that B(h) (h), for example (h) = h, Bz(h) = e² - 1. The detailed implementation of UniC is shown in Algorithm [ Importantly, we cireumvent the extra evaluation of 6n (#. (4) by pushing 60 (0 44) into the buffer Q instead of =o(2;, 44). Taking full advantage of E 44) of previous results enables us to increase the order of accuracy without incurring significant increment of computation cost. This makes our method inherently different from the corrector methods in ODE literature [22], where the computational costs are doubled because an extra function evaluation on the corrected of, is required for each step.
3.2 The Unified Predictor UniP-p
We find that the order of accuracy of UniC does not depend on the specific choice of the sequence {riHez, which motivates us design p-order unified predictor (UniP-p) which only leverages the previous data points by excluding D, in 4 since Dp involves Tt. The order of accuracy is guaranteed by the following corollary.
Corollary 3.2 (The Order of Accuracy of UniP-p). For increasing sequence rı r2 <...<
Tp-1 < 1, the solver given in 4 dropping the term D, and using coefficients that satisfies
(h:) - dp- (b.))l = O12))
has p-th order of accuracy.
Due to the unified form of UniP and UniC, we can use UniP-p as the implementation of the Solver-p in UniC-p obtain a new family of solvers called UniPC-p. Theorem [.1] and Corollary 3.2 ensure that UniPC-p can achieve (p + 1 order of accuracy. Moreover, under additional regularity conditions in Appendix D] based on Theorem 3.1 and Corollary 3.2
we show that the order of
convergence of UniPC-p reaches + 1 (see Appendix D
3.3 Comparison with Existing Methods
Here we discuss the connection and the difference between UniPC and previous methods. When 1. UniPC will reduce to DDIM 34. Motivated by linear multistep approaches, PNDM [24]
proposes to use pseudo numerical methods for DDIM, while our UniPC makes use of information in
the ODE solution (2) and is specially designed for diffusion ODEs. DEIS 40) is built on exponential
integrators in the time domain, where the integral cannot be analytically computed and explicit
formulae for high-order solvers cannot be derived. By using the half log- -ig-SRR// X 25 26). it is shown that the application of pnegrationn-y-pare
can simplify the integration of ( and leads explicit expansion of a:, DPM-Solver-2 [25] lies in our UniPC framework as UniP-1, where they assume We find through our numerical analysis that B(h) can be any non-degenerate
B(h) = e function such that Bhh) = Oh). Furthermore, DPM-Solvers do not admit unified forms even for orders smaller than 3, which adds to the challenge of obtaining algorithms for higher orders. In contrast, our UniPC exploits the structure of exponential integrators w.r.t. half log-SNR and admits not only simple and analytical solutions for efficient computation but also unified formulations for easy implementation of any order.
3.4 Implementation
By setting m' = (At.--- X,)/ m -.....p- 1, the UniPC-p updates in a multistep manner, which reuses the previous evaluation results and proves to be empirically more efficient, especially for limited steps of model evaluation [6). while singlestep methods might incur higher computation cost per step. Therefore, we use multistep UniPC in our experiments by default. The detailed algorithms for multistep UniPC and the proof of convergence can be found in Appendix B For UniPC, the choices of [r.) determine different updating methods. all the values are in (0, 1], the UniPC will switch to singlestep. Notably, we find in experiments that our UniC consistently improves different updating methods. Besides, we find UniP-2 8 and UniC-1 7 degenerate to simple equation where only single a is unknown, where we find a₁ 0.5 can be a solution for both B (h) and B2(h) (see Appendix(F] [F independent of h. In Appendix C we provide another variant of UniPC called UniPC, where the coefficients do not depend on h for arbitrary order p.
In the conditional inference, guided sampling [14 is often employed. Recent works 300 26 find
that thresholding data prediction models can boost the sampling quality and mitigate the problem of
5

---

20
30
30
UniPC
UniPC
UniPC
5olvarr
Solverr
DPM Solverr
23-
16
25-
20-
12
15
10
10
10
NFE
NFE
NFE
(a) CIFARIO (Pixel DPM)
(b) LSUN Bedroom (Latent DPM)
(c) FFHQ (Latent DPM)
Figure 2: Unconditional sampling results. We compare our UniPC with DPM-Solver++ 25 on 10, LSUN Bedroom, and FFHQ We report the FIDI of the methods with different numbers of function evaluations (NFE). Experimental results demonstrate that our method is consistently better than previous ones both pixel-space DPMs and latent-space DPMs, especially with extremely few steps. For more results, we recommend refering to Table 10 in Appendix G
train-test mismatch. Our framework of UniPC can be easily adapted to the data prediction model, see Appendix for algorithms and theoretical analysis. The detailed algorithms for multistep UniPC for data prediction are in Appendix B Hence, UniPC with data prediction can achieve fast conditional sampling in extremely few steps through dynamic thresholding.
4 Experiments
In this section, we show that our UniPC can significantly improve the sampling quality through extensive experiments. Our experiments cover wide range of datasets, where the image resolution ranges from 32x32 to 256 256. Apart from the standard image-space diffusion models [35] 8). we also conduct experiments on the recent prevailing stable-diffusion [29] trained on latent space. We will first present our main results in Section 4.1 and then provide detailed analysis in Section 4.2
4.1
Main Results
We start by demonstrating the effectiveness of our UniPC on both unconditional sampling and conditional sampling tasks, with extremely few model evaluations (< (<10 NFE). For the sake of clarity, we compare UniPC with the previous Mate--of-the-ar method DPM-Solver++ [26]. We have also
conducted experiments with other methods including DDIM 34. DPMeSolver 225 DEIS 40 and
PNDM (24. However, since some of these methods perform very unstable in few-step sampling, we
leave their results in Section 4.2 and Appendix G
Unconditional sampling. We first compare the unconditional sampling quality of different methods on CIFARIO 10 (21) FFHQ [18], and LSUN Bedroom [39]. The pre-trained diffusion models are from B55 and 29 including both pixel-space and latent-space diffusion models. The results For DPM--olver++, we use the multistep 3-order version due to its better combination of UniP-3 3 and UniC-3, thus the order of accuracy
shown in Figure performance. For UniPC, is 4. As shown in Figure 3 we find that our UniPC consistently achieves better sampling quality than DPM-Solver++ DPM-Solver++, our UniPC improves the FID by 6.0, 5.9, and 8.5 on CIFAR 10, LSUN Bedroom, and FFHQ, respectively. These results clearly demonstrate that our UniPC can effectively improve the unconditional sampling quality with few function evaluations.
different datasets, especially with fewer NFE. Notably, compared with
Conditional sampling. Conditional sampling is more useful since it allows user-defined input to control the synthesized image. To evaluate the conditional sampling performance of our UniPC, we conduct experiments on two widely used guided sampling settings, including classifier guidance and classifier free guidance. For classifier guidance, we use the pixel-space diffusion model trained on ImageNet 256 256 7 provided by [8) Following DPM-Solver+++, we use dynamic thresholding (30) to mitigate the gap between training and testing. The results are shown in Figure 3a and 3b where we compare our UniPC with DPM-Solver++ (66 under different guidance scale (s 8.0/4.0). For DPM-Solver++,
the multistep 2-order version (2M), which achieves the best results

---

8.6.0
5
5=1.5
UniPC
UniPC
9.26
UNIPC
50-
S
Solver++
Solverolve+++
25-
9.24
kyDist
0.22
0.20
9:38
10
NFE
(a) ImageNet (Pixel DPM)
(b) ImageNet (Pixel DPM)
(c) MS-COC02014 (Latent DPM)
Figure 3: Conditional sampling results. (a)(b) We compare the sample quality measured by FID on ImageNet 256x256 with guidance scale 8 = 8.0/4.0; (c) We adopt the text-to-image model provided by stable-diffusion [29] compare the convergence error, which is measured by the l2 distance between the results of different methods and 1000-step DDIM. We show that our method outperforms previous ones with various guidance scales and NFE,
Table I: Ablation on the choice of B(h). We consider two implementations of B(h) and also provide the performance of +PM-Solver++ [26] for comparison. The results are measured by the FID(4) on CIFARIO (1I and FFHQ [1]. We show that while the UniPC with both the two forms of B(h) can outperform UPM--olver++, Bg (h) performs better at fewer sampling steps.
(a) CIFARIO (Pixel-space DPM)
(b) FFHQ (Latent-space DPM)
Sampling Method
NFE
Sampling Method
NFE
6
10
5
DPM-Solver++ UniPC (B) b)
29.22
13.28
5:21
4.03
DPM-Solvere+ + (2)
27.94
15.99
9.20
7.36
23.22
10.33
5.10
3.97
UniPC (8,
18.66
11.89
8.21
6.99
UniPC (h) - 1)
26.20
11.48
5.11
3.87
UniPC (Ba(h) - D)
21.66
13.21
8.63
7.20
according to the original paper. For our UniPC, we use UniP-2 and UniC-2. It can be seen that our UniPC generates samples with better quality and converges rather faster than other methods. For
classifier-free guidance, we adopt the latent-space diffusion model provided by stable-diffusion [29]
and set the guidance scale as 1.5 following their original paper. To obtain the input texts, we randomly sample 10K captions from MS- IS-CCC02201 validation dataset [23]. As discussed in (26), the FID of
the text-to-image saturates in <10 steps, possibly because the powerful decoder can generate good image samples from non-converged latent codes. Therefore, to examine how fast a method converges, we follow 26] to compute the ly-distance between the generated latent code æy and the true solution z (obtained by running a 999-step DDIM), i.e., | - 2/ll2/V VD, where D the dimension of the latent code. For each text input, we use the same initial value zi sampled from Gaussian distribution for all the compared methods. It can be seen in Figure 3e that our UniPC consistently has lower li-distance than DPM-Solver++, which indicates that UniPC converges faster in guided sampling.
4.2 Analysis
In this section, we will provide more detailed analyses to further evaluate the effectiveness of UniPC.
Ablation on the choice of B(h). In Section 3 we mentioned that B(h) is set to be any non-zero
function of h that satisfies (hh) = ((h). We now investigate how the choice of B(h) would affect the performance of our UniPC Specifically, we test two simple forms: B1 (h) = and
Bg (h) = " 1 and the results are summarized in Table [] where we also provide the performance of DoM- (26] for reference. We show that UnIPC with either implementation of B(h) can outperform DPM Solvert.t When the NFE extremely small (5~6), we observe that B, (h) consistently outperforms (h) by 1~33 in FID. On the other hand, as the NFE increases, the performance of B2 (h) catches up and even surpasses B, (h) in some experiments (e.g., on CIFARIO and LSUN Bedroom). As for the guided sampling, we find B (h) is worse than B2 (h) consistently (see Appendix for detailed results and discussions). These results also inspire us that our UniPC can be further improved by designing better B(h), which we leave to future work.
7

---

Table 2: Applying UniC to any solvers. We Table 3: Exploring the upper bound of UniC.
show that UniC can be a plug-and-play com- We compare the performance of UniC and UniC- ponent to boost the performance of both sin- oracle by applying them to the DPM Solver++. glestep/multistep solvers with different orders. Note that the NFE of UniC-oracle is twice the The sampling quality is measured by FID on
number of sampling steps. Our results show that
the CIFARIO dataset.
UniC still has room for improvement.
Sampling Method
Order
NHB
Sampling Method
Sampling Steps
8
10
DDIM (14)
55.04
41.81
27.54
20.02
LSUN Bedroom, Latent-suuce DPM
(Ours)
47.22
33.70
19:20
12.77
DPM Solver++ (26)
17.79
8.03
4.04
3.63
DPM--ol (2M) (2))
2
33.86
21.12
10:24
6,83
UniC
13.79
6.53
3.98
3.52
UniC (Ours)
3
31:23
17.96
8.09
5.51
UniCe
6.05
4.39
3.46
3.22
DPM-Solver++ (3S) (26)
3
$1.49
38.83
11.98
6.46
FFNQ. Latent-space DPM
4. UUnC (Qurs)
4
50.62
24.59
10:32
5.50
DPM Sulver++ (26)
27.15
15.60
8.98
7.39
29.22
4.03
UniC
21.73
13.38
1 UniC (Ours)
25:50
11.72
5.04
3.90
UniC-oracle
15.29
11.25
8.33
7.03
UniC for any order solver. As shown in Algorithm 1 our UniC-p can be applied after any p-order solver to increase the order of accuracy. To verify this, we perform experiments on a wide range of solvers. The existing solvers for DPM can be roughly categorized by the orders or the updating method (i.e, singlestep or multistep). Since DPM-Solver++ [26] by design has both singlestep and multistep variants of I~3 orders, we apply our UniC to different versions of DPM-Solver++ to
whether UniC can bring improvements. The results are reported in Table 2 where the sampling
quality is measured by FID on CIFARIO by sampling from a pixel-space DPM 35 We also provide the order of accuracy of each baseline method without/with our UniC. Apart from the DDIM (34), which can be also viewed as 1 1-order singlestep DPM-Solver++ we consider another 3 variants of + including 2-order multistep (2M), 3-order singlestep (3S) and 3-order multistep (3M). It can be found that our UniC can increase the order of accuracy of the baseline methods by I and consistently improve the sampling quality for the solvers with different updating methods and orders.
Exploring the upper bound of UniC. According to Algorithm [ our UniC works by leveraging the rough prediction Et, as another data point to perform correction and increase the order of accuracy. Note that to make sure there is no extra NFE, we directly feed co(#t,, (;) to the next updating step instead of re-computing a 6g (#t, 4,) for the corrected éf.- Although the error caused by the misalignment between ca (it,, f4) and Eø( E .t) has no influence on the order of accuracy (as proved in Appendix E.7), we are still interested in how this error will affect the performance. Therefore. we conduct experiments where we re-compute the [0 44) as the input for the next sampling step, which we name as "UniC-oracle". Due to the multiple function evaluations on each f, for both कt, and T; the real NFE for UniC-oracle is twice as the standard UniC for the same sampling steps. However, UniC-oracle very helpful to explore the upper bound of UniC, and thus can be used in pre-experiments to examine whether the corrector is potentially effective. We compare the performance of UniC and UniC-oracle in T where we apply them to the DPM Solver++ (26) on LSUN Bedroom (39] and FFH [18) datasets. We observe that the UniC-oracle can significantly improve the sampling quality over the bascline methods. Although the approximation error caused by the misalignment makes UniC worse than UniCloracle, we find that UniC can still remarkably increases the sampling quality over the baselines, especially with few sampling steps.
Customizing order schedule via UniPC. Thanks to the unified analytical form of UniPC, we able to investigate the performance of arbitrary-order solvers and customize the order schedule freely. As first attempt, we conduct experiments on CIFAR10 with our UniPC, varying the order schedule (the order at each sampling step). Some results are listed in Table where we test different order schedules with NFE=6/7 because the search space is not too big. Note that the order schedule in Table 4 represents the order of accuracy of the UniP, while the actual order is increased by I because of UniC. Our default order schedule follows the implementation of OPM-Solver++ [26). where lower-order solvers are used in the final few steps. Interestingly, we find some customized order schedules can yield better results, such as 123432 for NFE-6 and 1223334 for NFE=7. We also show that simply increasing the order as much as possible is harmful to the sampling quality.
Sampling diversity. Apart from the sampling quality, we are also interested in the diversity of the images generated by UniPC In Table 6 we compare the sampling diversity of UniPC and
8

---

Table 4:
Customizing order schedule via
Table 5: Comparisons with more NFE. We com-
UniPC. We investigate different order schedules
pare the sampling quality between UniPC and previous methods with 10-25 NFE on ImageNet 256 and show our UniPC still outperforms
with UniPC and find that some customized order
schedules behave better than the default settings,
while simply increasing the order as much as pos-
previous methods by a large margin.
sible is harmful to the sampling quality.
Sampling Method NFE
10
15
20
25
CIFRROO, NFE= 6
DDIM (34]
13.04
11.27
10.21
9.87
Order Schedule 123321 10.33
123432
123443
123456
DPM- Solver 25
114.62
44.05
20.33
9.84
FIDJ
9.03
11,23
22.98
PNDM (4
99.80
37.59
15.50
11.54
CIFARIO, 7
DEIS 40)
19.12
11.37
10.08
9.75
DPM-Solver++
26
9.56
8.64
8.50
8.39
Order Schedule
1233321
1223334
1234321
1234667
FIDJ
6.41
6.29
7.24
60.99
UniPC (Ours)
7,51
6.76
6.65
6.58
Table 6: Comparisons of sampling diversity. We
Table 7: Comparisons of inference time. We compare the inference time of sampling from a stable-diffusion (29] and find UniPC achieves a
compute the Inception Score (IS) on CIFARIO [2]]
and find UniPC can generate more diverse samples than DPM-SolverH. 126].
similar speed to DPM-Solver++ [26].
NFE
NFE
Inception Score (IS?)
Inference Time (s)
5
8
10
5
10
15
DPM-Solver++
26
7.27
8.62
9.52
9.69
DoM-Stl
0.48
0.77
1.07
UniPC
7.55
8.81
9.59
9.83
UniPC
0.49
0.78
1.07
measured by the inception score (IS) on CIFARIO dataset. We find that UniPC DPM-Solver++ [26]
consistently generates more diverse samples in a variant number of function evaluations.
Comparisons with more NFE. To further evaluate the effectiveness of our UniPC, we also perform experiments with 10~25 NFE. Specifically, we perform guided sampling on ImageNet 256x 7 with guidance scale 8.0 and compare our UniPC with more existing methods including DDIM, DPM- Solver, PNDM, DEIS, and DPM-Solver++. We summarize the results in Table[5] where some results of the previous methods are from (26). The results clearly demonstrate that our UniPC surpasses previous methods by large margin.
Inference speed and memory. We test the wall-clock time of UniPC by sampling from a stable- diffusion model [29] using single NVIDIA RTX 3090 GPU and the results are shown in Table 7 We find the actual inference time of UniPC is very similar to DPM-Solver++ [26]. As for memory usage, it is only related to how many previous model outputs stored. Therefore, our UniPC also costs similar memory to DPM-Solver++. For example, both UniPC-2 and DPM-Solver+++(MM) cost about 6.3GB of memory when sampling from stable-diffusion model.
Visualizations. We provide a qualitative comparison between our UniPC and previous methods with only 7 NFE, as shown in Figure I We use different methods to perform guided sampling with guidance scale of 8.0 from a DPM trained on ImageNet 256 x 256. We find DEIS 40 tends to crash with extremely few steps, while the sample generated by DDIM 34 is relatively blurry. Compared with DPM-Solver++, the state-of state-of-the-and art method in guided sampling, UniPC can generate more plausible samples with better visual details. We further compare the sampling quality of our method UniPC and DPM-Solver++ using 1024 x 1024 images. The results in Figure show that our method consistently generates more realistic images with fewer visual flaws.
a newly released model that can generate
Limitations and broader impact. Despite the effectiveness of UniPC, it still lags behind training- based methods such as [31] How to further close the gap between training-frr muining-free methods and training-bast based methods requires future efforts.
Conclusions
In this paper, we have proposed a new unified predictor-corrector framework named UniPC for the fast sampling of DPMs. Unlike previous methods, UniPC has unified formulation for its two components (UniP and UniC) for any order. The universality of UniPC makes it possible to customize arbitrary order schedules and to improve the order of accuracy of off-the-shelf fast sampler via UniC.

---

DPM-Solver++ 266]
UniPC (Ours)
DPM-Solver++ [66
UniPC (Ours)
"black and white cat is sitting on top of a wooden bench" T
"vase filled with flowers sitting on table" "
"litlee girl sitting on top of teddy bear"
"cat sitting on top of a bed in front of living room"
"baseball player swinging bat at a game"
"bathroom with a sink and a mirror"
"bowl of fruits and vegetables on it"
"man standing next to a dog"
Figure 4: Comparisons of text-to-image results between UniPC and SPM-Solvert+ Images sampled from the newly released Stable-Diflusion-L (1024 1024) using 15 NFE. We show that the images generated by DPM-Solverts contain visible artifacts while UniPC consistently produces images with better quality. Please view the images in color and zoom in for easier comparison.
Extensive experiments have demonstrated the effectiveness of UniPC on unconditional/oondiioona sampling tasks with pixel-space/llttnt-sppaa space pre-trained DPMs. We have also discovered several directions where UniPC can be further improved, such as choosing better (hh), estimating more accurate eg (ăt, 44), and designing better order schedule. We hope our attempt can inspire future work to further explore the fast sampling of DPMs very few steps.
10

---

Acknowledgments
This work was supported in part by the National Key Research and Development Program of China under Grant 20222D0100102. and in part by the National Natural Science Foundation of China under Grant 62321005, Grant 62336004, Grant 12271287 and Grant 62125603.
References
[1] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.0083 2021.
[2] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. ICML, 2022.
[3] Fan Bao, Chongxuan Li. Jun Zhu, and Bo Zhang. Analytic-dpm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICLR, 2022.
[4] Mari Paz Calvo and César Palencia. A class of explicit multistep exponential integrators for semilinear problems. Numerische Mathematik, 102(3):367-381, 2006.
[5] Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. ICLR, 2021.
[6] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.09788 2022.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. CVPR, pages 248-255, IEEE, 2009.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. NeurlPS, 34:8780-8794, 2021.
[9] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. ICLR, 2022.
[10] Ian Goodfellow, Jean Pouget- Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, pages 2672-2680, 2014.
[11] David Francis Griffiths and Desmond Higham. Numerical methods for ordinary differential equations: initial value problems, volume 5. Springer, 2010.
[12) Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pages 10696-10706, 2022.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
[14] Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. NeurIPS, 2021.
[15] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204. 034458, 2022.
[16] Marlis Hochbruck and Alexander Ostermann. Explicit exponential runge-kurte kutta methods for semilinear parabolic problems. SIAM Journal Numerical Analysis, 1369):1069-18 2005.
[17] Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209-288 2010.
[18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4411 2019.
[19] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. NeurIPS, 34:21696-21707, 2021.
[20] Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
[22] John Denholm Lambert al. Numerical methods for ordinary differential systems, volume 146. Wiley New York, 1991.
11

---

[23] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, pages 740-755. Springer, 2014.
[24]
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. ICLR, 2022.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022.
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:221 2022.
[27] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML, 2022.
[28] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML,, pages 8162-8171. PMIR, 2021.
[29] Robin Rombach, Andreas Blaumann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[30] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic texl-to-image diffusion models with deep language understanding. NeurIPS, 2022.
[31] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR, 2022.
[32] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
[33] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In ICML, pages 2256-2265. PMLR, 2015.
[34] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
[35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[36] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.0222 2022.
[37] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In ICLR, 2021.
[38] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. ICLR, 2022.
[39] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.00063333 03365, 2015.
[40] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.1390 2022.
[41] Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.08 055664 2022.
12

---

A UniPC for Data Prediction Model
A.1
Comparison of data prediction and noise prediction
The data prediction model is a simple linear transformation of the noise prediction model, namely 2g (æz 0,60)/ox [9]]. However, high-order solvers based on æy and eg essentially different 30 26] As we shall see, the formulae of UniPC for the data prediction model differ from those for the noise prediction model. On the other hand, for image data, ay is bounded [-1, 1J, while eæ is generally unbounded and thus can push the sample out of the bound. Therefore, solvers for data prediction models are preferred, since thresholding method 30 can be directly applied and alleviate the "train-test mismatch" problem.
A.2 Adapting UniPC to Data Prediction Model
As shown in the Proposition 4.1 of [26], for initial value w, at time 0, the solution time t e [0, s] of diffusion ODEs is
T+ Vs 0t S/me A [x.
( AjdA,
2:
Os
where we use the notation to and 인 to denote changing from the domain of time (f) to the domain of half log- SNR(A), i.e., Φ == lx)(A) and c A) == =x)= We are also able to adapt our UniPC to the data prediction model and utilize more information from previous data points. Recall that 14 dd. - d1 1. For any nonzero increasing sequence ri < r2 ... < Tp
Asm rmhh + Xr; 1- 8m i.((As.... m .P....P. The UniPC-p is given by
Ct,-
Th _t.n ats e )ao (it 11-1) + on; B(h+)
Σ
Om D
(Predictor) (10)
I'
m=1
1i = Th Dt: + Ot. - -e--6- t h1: )az 2o0l2t (īt, 1 (1111) + or. B(h))
Gm D
(Corrector) (11)
0t:-1
Tm
m=1
where D @. 8m) we (it, is 114) Importantly, the corrector (UniC) can be also applied to any solver for the data prediction model that outputs ã.,- Denote a, (a1, ap-1) Cp (c :G) Let
(h) = (h)) gr h" (1)))
(12)
where (h)( is defined by the recursive relation vha (h) 1/n (h) (ho e-h see Ap- pendix [..4 for details. The order of accuracy of UniPC-p for the data prediction model is given by the following proposition. The proof in Appendix E.4
Proposition A.1 (Order of Accuracy of UniPC-p for Data Prediction Model). For an increasing
sequence T < T2 < Tp 1, under regularity assumption E.3 assuming 0 + B(h) O(h).
R& (h (h+) (hgt)), and (Rp then the order of accuracy of UniPC-p is p pp+1.
9p-1 (1+)) O(+)) (13)
We list the algorithms for UniC-p UniP-p for the data prediction model separately, see Algorithm and Algorithm 4
B Detailed Algorithms of Multistep UniPC
This section offers detailed algorithms with warming-up for multistep UniPC for noise prediction model (Algorithm 5 Algorithm 6 and for data prediction model (Algorithm 7 Algorith 8).
C UniPC with varying coefficients (UniPC,,)
UniPC-p 4 uses vector ap to match simultaneously all the coefficients of the derivatives. Alter- 1p natively, we can use Ke to match each order of derivatives separately and obtain matrix of
13

---

Algorithm 3 UniC-p for data prediction model
Require: {ri)i=h 1.-1 data prediction model œø, any p-order solver Solver-p -p, a buffer Q (xa ( tikk) I==1'
h; + )1, - Tp f 1, Tt. t Solver-por Saaver-p(ot,,,. Q))
for m to do
8m la + Xo= 1)
D - xe colasmm Sm) - Iz (2 t/-1)
end for
Compute Cp + R2 (h,)g where Rg and 9p are as defined Theorem 3.1 and 12 o6 on 0455 + an, (1 - R is 11(1-1) 1) or, B(h,) E Cm D rm 0e-1
Q byffer (0(
return: Zi.
Algorithm 4 UniP-p for the data prediction model
Require: {r: 1P-1 data prediction model ææ, buffer Q {ao (T), +=
h4 + Ay. A
for m 1 to do
Sm tx + >)--1)
D < Fo ( 8m) Fol (it;- (1-1)
end for Compute ap-1 R 1 (--(()))pp- 1 B(h;), where Rp-1 and 9p-1 are as defined in Theo-
rem 3.1 and
Ltt. Oec-1 Tt, + or, (1 e -hi )20 (ir: 1s 11-11) 1) + an, B(h,) (P Om D T'
Q butter return: Lt,
+ Cal (i)
coefficients, Ap = (a1.p. ... ap.)) Consider
Ctt Wr. of, (e 1)Ea 1s 11111 or, Σ hepp+ 1hPp+1(h))), (hi)a
0t; Oit;11
Dp:
(14)
where D, = (D1//i, Dp/rp)) T and Dm is defined in (5). The following theorem guarantees
the order of accuracy of UniPC... The proof deferred to Appendix The convergence order is investigated in Appendix D Define
1
r2/2!
Tp 2
C
:
/pl
T2
/pl
1p-1pp
Let Ip be the d-imennioonal identity matrix. Theorem C.1. Under the conditions of Theorem 3.1 if
IC Ipl = Ohh")
max his 1515P
(15)
UniPC, is (p + 1)-th order accurate.
In fact, Cp is invertible. Note that Cp is the product diagonal matrix and Vandermonde matrix. namely
1
T1
...
Tp <Tp
Gp
1 1/p!
14

---

Algorithm 5 Detailed implementation of multistep UniC-p
Require: initial value ær, time steps {4: J noise prediction model 60, any p-order solver Solver-p
Denote h; := -., for i.... M. Tto t IT, Ti t- ZT. Initialize an empty buffer Q.
Q byffer co( to) for i to do
P: f ip
Lt
(1) - ari
Ti - oi, or -))ea(
dt. - Solver-p, (2; Q)
Tp +-1
DR, f Eg(Tr, 4i) EB E
for m to pi do
mm f- (A1 m
do 1)/li;
D Dm-1 t co( to-m) c
end for
Compute ap, Rp, (h:) Bh,)) where (ha) and op. defined in Theo-
rem
3.1 (1)
#f. Tie.
ot, B(hs) E Om D Dm
Q byffer ,
end for return: 3tM
Thus, we can simply take A, = Cp The advantage of UniPC,, is that Ap solely depends on (re]i The algorithm of UniPC, to replace the updating formulae of UniPC (Algorithm) and Algorithm 2 by (14) with A, Op See Appendix for its performance.
D Order of Convergence
In this section, we shall show that under mild conditions the convergence order of UniP-p is p and the convergence order UniPC-p is + for either the noise prediction model or data prediction
model. The proof is deferred to Appendix E
Definition D.1. For time steps {4: i= we say the order of convergence of sampler for DPMs is
M
Et
zol O(P)).
We start by introducing some additional regularity assumptions. Assumption D.2. The noise prediction model cx(x, s) Lipschitz continuous w.r.t. with Lipschitz constant
Assumption D.3. (1) = maxisisM hy maxigigM To4-1/au < b, oz < for all 1 S M.
O(IM)) (2) For constant
Assumption D.4. The starting values क.,, < i < k satisfies for some positive constant co.
F1 -,,, S h k
<
(16)
Assumption D.2 is common in the analysis of ODEs similar to 20 By Assumption D.2 we have
co( s) = c co( [.., s) + O(z, - x.). Assumption D.3 assures that there no significantly large step size and the signals
neither exploding degenerating. Assumption D.4 is common in the convergence analysis of multistep approaches 4
The following Propositions/ D.5 D.6 ensure the convergence order of UniP-p and UniPC-p. For general Solver-p such as DDIM (p 1), SPPM-SolverDDMM-SSllerrrt S 3), UniC-p can also increase the convergence order.
15

---

Algorithm 6 Detailed implementation of multistep UniP-p
Require: initial value ær, time steps {44 ) noise prediction model Denote h; := X1, A for = -,. M.
Ita f- IT. Initialize an empty buffer Q.
Q byfler E to)
for i = to M do
P: <- (p,
Ft. - on, -t,-1 - oe, - 1)
if pi then
Lt t Li.
Q 5 co(
continue
end if
for m = pj do
mm-1 f- (de; dt; 1)/h;
Dm-1 t
mi (1-m) -
end for
Compute ag
Ry. (h.))pp-11 (h,)) Blii), where 1 and Φp are as defined
in Theorem
31. + t
Ti. B(hs) Emsl am Dm/
buffer Eo(
Q end for return: RiM
Proposition D.5. Under the conditions of Theorem 3.1 Assumptions convergence of UniP-p p is p.
D.3 and D.4 the order of Assumptions
D.2
Proposition D.6. Under the conditions of Theorem the order of convergence of UniPC-p is
D.
D.3 and D.4 with
Remark D.7. After a careful investigation of the proof of Proposition D.6 and Proposition D.5 we
point out that for the singlestep updating when using UniPC-1 for the estimation of a ..... li
81, .... 8m < 4, the order of convergence for UniPC-p is + and the order of convergence for UniP-p p for p
find that UniC-p also increase the con- vergence order for general Solver-p for diffusion ODEs. The following is a direct corollary of Proposition D.6 which gives the order of convergence of UniPC.-p.
Following similar arguments of Proposition D.6
Corollary D.8. Under the conditions of Theorem C.1 Assumptions D.2 D.3 and D.4 with k the order of convergence of UniPC,-p p
The order of convergence for the data prediction model follows analogous to Proposition D.5 and Proposition under slightly different assumptions:
Assumption D.9. The noise prediction model no( s) is Lipschitz continuous w.r.t a with Lipschitz constant
Assumption D.10. (1) h maxigigM (0t,,//rt < b, b-1 < <a, < b, for all Sis
max h; O(LMM) (2) For constant b 0.
We list the results of the order of convergence of UniP and UniPC for the data prediction model as corollaries to Propositions [.5] and D.6 and omit their proofs for simplicity. Interested readers can refer to the arguments in Appendix E for the noise prediction model and derive the detailed proofs
for the data prediction model.
Corollary D.11. Under the conditions of Proposition A.1 Assumptions D.9
D.10 and D.4 with
k = the order of convergence of UniP-p.for data prediction model is Corollary D.12. Under the conditions of Proposition A.J Assumptions
D.9
10 and D. with
k the order of convergence of UniPC-p for data prediction model is
16

---

Algorithm 7 Detailed implementation of multistep UniC-p for data prediction model
Require: initial value ær, time steps {4e ( data prediction model ae any p-order solver Solver-p
Denote h; := -., for i.... M. Dto t- IT, Ti t- 2T. Initialize an empty buffer Q.
Q byffer xz( (2 to)
for i = to do
P: f mp
(1) - 0i-1 Tha + +a, - e1 (
dt. - Solver-p, (2; QQ)
Tp +-1
DR. f oo (i, , 1i) - xe 1 (i-1)
for m to pi do
mm-1 f- (A:;
\ 1)/lu;
DT t I (t ti-m) xz (20(Tie.1)) 11
end for
Compute Cp, (h:)ggp (hh)) where Rp, (h;) and gp. defined in Theorem 3.1 and (12).
(1) + ai, B(hs) E Cm mm/m
Q byffor xe co( ti) end for return: 3tM
E Proofs
In this section, we provide preliminaries of numerical analysis, regularity assumption, and the proof of
Theorem in the paper, the regularity assumption and the proof of Proposition A. Appendix A as well as detailed proofs for Theorem C.areemmC.iirr in Appendix C, for Proposition D.5 and Proposition D.6 in Appendix D Through this section, we use and C2 denote sufficiently large positive constants independent of h.
E.1 Preliminaries
We begin with introducing preliminary results and concepts necessary for the proof of Theorem 3.1
Expansion of exponential integrator. First, obtain the Taylor expansion of namely the exponentially weighted integral. Define the k-th order derivative of čo ( A) t (#x, A) = d ( x)/dx*. For S S T, r E [0, 1], let == d As À = A + rh. Assuming the existence of total derivatives of (k) (#x, A), 0 < k S n, we have the n-th order Taylor expansion of ( A) w.r.t. the half log-SNR
n hh" 20" ( 2a) 11
A
(17)
Using the result of Taylor expansion of [17). the exponential integrator of can be reduced
d
Eolox,
A)d1
02 01
Σ h+1
k!! (#x, Aa) + 0(h+t2)
dre
(k)
h k+1 //' c(l-r)h
dre
(k)
(Ø), Xo) + (1
a1 k=0
k!
Øt
h
x+ patt (h)) (
\ A.) + 01
(18)
km0
17

---

Algorithm 8 Detailed implementation of multistep UniP-p for data prediction model
Require: initial value ær, time steps {44 J data prediction model mg Denote h; := Xt, A for = -,.... M.
Ita f- 37. Initialize an empty buffer Q.
Q byfler (i.,,,,.0 to) for i = to M do
P: <- min
st - o4
Ft.-1 + on, - - (001))0 p 1i
dec
if pi then
Lt t Li;
Q 5 zo (te,, 1i)
continue
end if
for m pj do
mm-1 f- (Ae; )o; 1)/h;
D D=-1 t Fol (T
mi ((-m) ne (0 4-14
end for
Compute ag
(hr) and gp.-1 are as defined in R (1.))pp.-- (h.)/ (his), where R
Theorem and
Lt. + tar B(h,) Σ Pi am D I'
butfer end for return: Tts
Q
an (m( 14)
where Pk+1 (h) = So c(l-r)hr* dr can be computed via the recurrence relation (x++(22) = (px(22) bPx (0))/z,, (0) 1/k!, and up(())
[6] For example, the closed-forms of yox (h) for
== 1,2,3 are
hh -1
eh h
eh - 12 h -
s (h)
p2(h)
(aa(
22
h
Order of accuracy. In the following, we use the linear multistep method illustrate the order of accuracy. Consider the ODE
= f(x y), x E [To, b). y(zo) = yoo We say that f satisfies Lipschitz condition, there exists such that
(19)
(f v1) f(x, v2) S Lly1
V2 Vyn 1/2 E IR.
(20)
Suppose yxx) is the solution of Equation
19
The k-oodder linear multistep method is given by
k
Dno+k
Callnii + h
Σ 1=0
3i (Inti Un+s),
(21)
1=0
where buti is the approximation of y(xntt), Entt = En + ih, ais Br are constants, |aol + |30| >
0.
Definition E. The local truncation error of (1h on Intk is
Totk = f p(In+k)= Σ acu(In+s) - h Σ Bi ((0(4t)))((E44)))
(22)
i=0
T 1(#+1),, we say the order of accuracy of 21 p.
The local truncation error describes the error caused by iteration in the numerical analysis [2]
18

---

E.2
Regularity Assumption
Assumption E.2. The total derivatives d'
1, .... p exist and are continuous.
dAu
Assumption E.2 is required for the Taylor expansion which is also regular in high-order numerical methods.
E.3 Proof of Theorem 3.1 for arbitrary 2
We first give the local truncation error of UniC. Given Br S 4--1, are correct, with slight abuse of notation, define
Tt.
Ot::
Dt--1 - oc, (1)) 04, Bhh,)
Σ
Om
(Co(Exem Asm) - co de, Acm
01,11
I'
m=
" ( 1i) - Eo Eo( )
-Ox, B(h;)
Tp
Since Solver-p has order of accuracy p and B(h) -Oh), under Assumption D.2 have
Lt, 0tt-11
0t. It.-1 Fi, (e (2)) ti-1 oi, B(h,)
am
(ég (Co(it dam)
Tm
E Ao. .) + ((1+t2).
(23)
Suppose v, is the solution of the diffusion ODE Then, the local truncation error t, is given by 0t, lt, Further, similar to Definition E. 1 the order of accuracy of UniC is 1, if there exists sufficiently large positive constant such that
max IS M
Tt.
Tr, S Ch++1
max h4 ISiSM
In the following, we shall show that the order accuracy UniC-p p + 1. Combining 2 and we have
de.
It.-
Oits
01.1 - ai,
e E A)dA
01.1
0tt:
Bt.-1
Te. Σ
h xk+1 (k++1 (1)
(Dxu dmmn) 0(htt2)
a1,11
k=0
Cit
T1, (e 1) or, Σ h (b.))) (0 (Exe,,, Am_1)) + (hp+2).
a1,11
(24)
Let RTT, denote the k-th row of R,p. By 77. we have |a, (h.)) (h. s Co (h,))(t,) Under Assumption
E.2 by Taylor expansion, we obtain
am
(èg (
dom))
I'm
p
(1))
( de. 1) O(PPt))
Om
1t (n) Eg (8
(ixe_,, )e_,,) ((P+l)
am
n2!
hi
n!
a
R
p.
(2x,, ) 0(h+t1).
19

---

Thus, we have
am (ēo hemm Aom)) EB (BAg ,,,r-1)) -1()))-B00) B(h,)
1
(hsz)))))e) (Ex,, ( d
T'
k=1
< (B(b)) 11222 + h(t1).
(25)
Combining
23
and 25
given Bhh;) = O(hs). we have
D18X IS1SM
[t; Dt.
ot
h
k+1 (h:))) (dxxe 1 A.,
k=1
+ 01; B(ha)
Σ
Om (ég(2xxn )om) - co : + 0(het2
Fm
m==
(26)
Therefore, UniC-p is of (p 1)-th order of accuracy.
E.4 Proof of Proposition A.1
Regularity Assumption
Assumption E.3. The total derivatives d=xo(( k = 1, exist and are continuous.
Expansion of the Exponentially Weighted Integral. First, we obtain the Taylor expansion of (9) Define the k-th order derivative of @ A) (k) (æx,,) A) = ( A)/d>* For 0 < < $ S T. r E [0, 1]. let == d As À = As rh. Assuming the existence of total derivatives of (k)
Tg ( A). 0 < k < the n-th order Taylor expansion of ( A) w.r.t. the half log-SNR À is:
a
h* a
(a( A)
(k) (Ex, Ao) 0(h+t)).
(27)
9 Then, the exponential integrator of can be reduced to
//m >
eA-2, rb
c 2220(22,111)
A)dd
0t
h
h++
k! dr (ix,, Ao) 2()
(k)
σ1
k=0
0x
h h+1 ('' c(r-1)h
dr
(1))
(0: ( 2s) 2
Rd!
k=0
Q
h k+ Wh+
(# A.) (xx)
(28)
σ k=
where (h) So elr--
x
dr can be computed via the recurrence relation by integration-by E7
parts formula:
1 f² p.b (1++((2)
1/1
k (k 1)! dr
delr (r-1s) 1)s
u/x (2)
2 Ju hd!
kd!
and c/o(2) - [16]
For example, the closed-forms of thx (h) for 2, 3
(h) = 1-e-h h
(h) - 1
Va(h) 1 h +
42
h
Proof. In the subsequence analysis, we prove the order of accuracy of UniC for the data prediction model and the result of UniP for the data prediction model follows similarly. Suppose is the
20

---

solution of the diffusion ODE (T. For the data prediction model, given ær, S f--1. are correct, with slight abuse notation, define
0:: F4-1 + ar, (00 + 04, B(h,) Ft;
Σ
Cm (iol@rr, Aam) - 3o ( ( )t--1))
06-1
Im
+o B(ss)
Cp (zo( 1.) Bo @xxer de 1)).
Tp
Similar to 23 since Solver-p has order of accuracy p and B(h) = (h), under Assumption D.9 cm
we have Tt
0::-1 -t,-1 + (1 e - (20,
dec
(h+)) -m=1
(to A Amm
Pm
do (
A (h)+2). Then, the local truncation eπror on t, is given by (0t, Fc. Further, similar Definition we shall show that
max [tt, Tt. 0(17t2), h h= max h.
IS
Combining and 28 we have
= 0t. Dh.,. + ar, (1 - N4 (0 + ar,
k+1 h 1/++1 (h.) (h:)aeg (āx. )e,_,) ) + O(hot2).
(1())
k=1
Recall that R is the k-th row of R.p. By 13 have c, R (hs) B- kl))))))))
CB (h,)het1 for some constant C Under Assumption E.3 by Taylor expansion, we obtain
(0o (is(@nn (7xmm Aam dem) 0 (.te1)))
I'm
m=1
p
(1))
(i),, ) ;
O(h+tl)
2g
n
R=1 1 m=1
h4
Cp Rp (ho)at (ix)) ) + 0(h+t1).
n!
Thus, we have En 2m=1
(2 Aem) - 2e (n-1)) BBha)) BB(h)) Ek / Ui+1 (ha)aa (æ).. >e...) S (LB(h:)-1121 + l.tt). Given # B(h,) = O(hi), we have
max ISiSM
Ft, -4.1= |
h | wk4z (h,) (1,)2 ( ).ai)))
k=1
a,, B(h))
Cm ( Asm)
@ dmm 0(h++2)
Tin
3
Therefore, UniC-p for the data prediction model is of (p 1)-th order of accuracy.
E.5 Proof of Theorem
We first give the local truncation error UniPC,, Let Am. denote the element of Ap on row and column n. Given Er,, r < 1;-1, are correct, with slight abuse of notation, let
0t Th,,, or, (c -1)0((01,1114-1)
Lt.
a1.1
Ot
happ+1 (h:)
Σ
Am. (Eo (Eo( Nem) Eo Eo( d).-)))/Tm
m=1
0t.
hxpp+
ti) co( dewr))/
21

---

Similar to (2).. we have
at
oz, (e"1 1)Ea (0t, (it
Dt:
04
P
P Σ m=1
o2, Σ
Chspn4i(h))
Am, (6) ( Eo Eo( A + O(h+t)) (29)
n=1
Suppose We the solution of the diffusion ODE ! and the local truncation error on t, is given by [
L. Further, we shall show that the order of accuracy of UniPC,-P is p + 1, i.e.
max SSSM
I1. Lt,
O(htt), h
max h4 ISSM
By 24 we have
Ott
D1, on, 1)Ea(20, 1,t111 - oi, Σ
Gk+1 (h,))z ( (txe du 1) + O(h+t)).
3t:
h
0t:11
(30)
Under Assumption E.2
by Taylor expansion, we obtain
Am.k (Eo (E
hick+1(hh)) (hx)
Asm.) co c - det-1))
I'
m=1
hapk+1(h4))
Am,k
Eg ( ( +0
((())
n!
Aml
m=1
n=1
pn n-
(h+1(hi)
Am,
Eg
(1))
de. 0
n!!
n=1 1=1
m=1
4 (h.) (1 + ((x AG, ,) + 0(PPt)
n=l k=1
1 Pnt (hx) (()) (ax )t, ) + 0(h+t2).
n=1
Thus, we have
Dt, 1 0(Ppt))
15
Therefore, UniPC of (p+ 1) order of accuracy.
E.6 Proof of Proposition D.5
For UniP-p, p. we define
0t: Ft-r
oe, 1)co( 4-11) 41)
oz, (hh
(co( )om)) co( Ag-)))
at11
I'm
(31)
By Assumption D.2 we have (eo(#,, 8) - co(2c 8)s S L|e. azl. Therefore, for sufficiently large constants C, Ci > depending on {am) and frm we have
[ at + Lo - -1) CIph) p Fu 1 + CLh 0
T m--
D0;
m=1
(32)
For simplicity, define e; = |/,; - fn maxpsign lesl. Using Theorem 3.landd and
32
we obtain
< Ot, + (eh - 11) + CLph) e.1 + +Lh Σ 0:11
Gi-1-m I + Coret1
(33)
m=
22

---

Let 5 == aq + Lo (c 1) + CLph. Then, it follows that -
e, ei (CLph + 8e)f + Coh Since & CLph > 1 for sufficiently small h, the right hand side is also a trivial bound for fi-1, because 0ç > 0 is monotone decreasing thus 0t, < 0t-11 We then have
Ji S (CI.ph + Be) + Col
(34)
Let = MaxigisM ot, +2Cp. By elementary calculation, under Assumption D.3 we have
M
M
Look
II + CLph) s Ci ==p
II 1=p
ao, /o, + La/M) Ci att Os2 i=p
On, M II
S Oze
(35)
where C₁. C2 sufficiently large constants. Under Assumption
D.4 with k
p.
maxogisp-1 L Lzi & cghh. Repeat the argument of (34). under Assumption D.3 by (35)
there exists constant C3, C1 such that
fM S C2e f2-1- CyMhet1 & C.hh.
(36)
Therefore, the convergence order of UniP-p is
E.7 Proof' of Proposition D.6
For the sake of clarity, we use Ti St. Bh,)) E am (Ex dom))
ay Ihe- (e 1)
c ) for UniC-p and use Tye of (31) for UniP-p.
m==1 Fm
Similar to (32. we have for 2 P.
Zi,1
S (
0t;
+ Lo (c - CLph) " Itaill
p-1
+ CLh
D Ftammnl LhC1 Tt, zul.
(37)
where for the oracle UniPC (see Section
4.2 for definition), the UniP-p admits
Dr. Lt,l S
at
Lor -1) CLph) [ Immill
0t111
p-1
+ CLh
100 [1----1
Dtt 1l-
(38)
Define ef = 17,- 11, 1, ff maxoçisn lef). Write Bi = on + Lor, (e - 1) + C'phh. Let
as,..
= max oe, + 2Cp. As shown in the proof of Theorem Combining the local truncation error, it follows that
3.1
Ft. -xf. O(het2).
fo < f +
CLhpfi C2he+1) + Coh
< (1 (AC1) + + Col
Repeating this argument, similar 36), have
xol S f&r II [1 LACI)
+ CLhp))ff MC2h+42
==p
< Cge f--1 + MC2hP42
(39)
For 0 < p, we have
at
1i - Lt,1 5
+ Lor, - 1) + C'Lph) (if_. æt,-1l
Ct:
+ Chh
Dremmn
Ite-m-1 + LhC L., Ital-
m
23

---

Therefore, under Assumption D.4 with & = we have for 0 <isp-1
S S 3 C.C.pph + -hhet
Repeat this argument, we find for a constant C4 > 0 Combining
1 S Cahp+1
(40)
339 and 40). we have
F - Zol = 1(h
For the non-oracle solver, for the step of UniP-p we have
ir.. -2t, 5
ar
E Dt, 11+1++ (CLph + Lor, (e - 1)) [r., 02,
01
CLh
3t B4 al + C2hrt1
(41)
Let en L It,1 f: mix C₁. Under Assumption D.4 with k p. fp-1 cahP. For
0 S i < p. we have
1 Ft. -
0tt
|
0t.,1l+ (CIph Lo,, (c - 1)) ( Dta-1l
044-1
i-1
+ CLhh
Σ
0h-m-1 - Dti-m-il LhC1 [t. - It, + Cyhpt2,
m=1
Similar to 40). we obtain
fp O(hrtl). Iterating 37 and by Theorem 3.1 we have
[ val (hh+t)..
F Implementation Details
We now provide more details about our UniPC and the experiments.
F.1
Implementation Details about UniPC
Our UniPC implemented in multistep manner by default, as is illustrated in Algorithm 5| In this case, the extra timesteps that are used to obtain the estimation of higher-order derivatives {8m pp-1 are set to be larger than t;-1. In other words, frm | are all negative. We have also found that the conditions of UniP-2 and UniC-1 degenerate to simple equation where only a single a₁ unknown. Specifically, considering (8) and ( we find that
a1 Bh) 4/1 (h) O(h3),
(42)
where
(h) (1pa(h) ch - h -1
1
/ 11+0(11).
(43)
For B1 = is easy to show that when d1 0.5,
B(4) - 15n (h) = h - 0
(44)
For ch h (h+)). the derivation is similar and 1/2 also satisfies the condition. Therefore, we can directly set a1 = 1/2 for UniP-2 and UniC-1 without solving the equation. For higher orders, the vector ap is computed normally through the inverse of the R, matrix.
To provide enough data points for high-order UniPC, we need a warming up procedure in the first few steps, as is also used in previous multistep approaches (26) and is shown in Algorithm [16]. Since our UniC needs to compute 6g ( 44), i.e., the model output at the current timestep I to obtain the corrected result 2; performing our UniC at the last sampling step will introduce an extra function evaluation. Therefore, we do not use the corrector after the execution of the predictor for fair comparisons.
24

---

Table 8: More unconditional sampling results on CIFARIO 21
NFE
Sampling Method
5
7
8
9
10
DDIM
3
55.04
41.81
33.10
27.54
22.92
20.02
DDIM UniC-1
47.22
33.70
24.60
19.20
15.33
12.77
DPM-Solver-3
25
290.65
23.91
15.06
23.56
5.65
4.64
(2M-
26
33.86
21.12
13.93
10.24
7.97
6.83
DPM-Solvar++2MM
UniC-2
31.23
17.96
11.23
8.09
6.29
5.51
DPM-Solver++<3M))
26
29.22
13.28
7.18
5.21
4.40
4.03
DPM-Solver++(3M) UniC-3
25.50
11.72
6.79
5.04
4.22
3.90
(n)
23.22
10.33
6.41
5,10
4.29
3.97
UniPC-3-B2(h)
26.20
11.48
6.73
5.11
4.30
3.87
-
25.60
11.10
6.18
4.80
4.19
4.18
Table 9: More unconditional sampling results on FFHQ (18].
NFE
Sampling Method
5
8
9
10
DDIM
34
58.23
44.40
34.52
28.06
23.52
19.72
DDIM UniC
39.41
26.48
18.58
14.56
11.97
10.33
DPM-Solver-3
25
54.17
25.24
12.37
8.06
10.22
7.74
DPM-Solvar++(M)
26
32.50
20.32
14.25
11.30
9.45
8.28
DPM-Solver++(2M)
UniC
24.20
14.92
10.82
9.11
8.01
7.39
DPM-Solver+++3M)
(26]
27.15
15.60
10.81
8.98
7.89
7.39
DPM-Solver++(3M) UniC
21.73
13.38
10.06
8.67
7.89
7.22
(niPC-3-Bi,h) UniPC-3-B,
18.66
11.89
9.51
8.21
7.62
6.99
21.66
13.21
9.93
8.63
7.69
7.20
F22 Details about the experiments.
We now provide more details of our experiments. For unconditional sampling on CIFARIO 21 we the ScoreSDE 35 codebase and their pre-trained model, which is continuous-time +DPM+ wwwpddppppeppppeeee
config file, the same as the example provided by the official code DPM-Solver (25] To compute FID, adopt the statistic file provided by ScoreSDE (35] codebase. For unconditional sampling on LSUN (39 and FFHQ 18], we adopt the latent-space DPM provided by the stable-diffusion codebase (29 Since there is no statistic file for these two datasets in the codebase, we compute the dataset statistic of FFHQ using the script in the library pytorch-fid, and borrow the statistic file of LSUN Bedroom from the siided-diffusion codebase [8] For conditional sampling on pixel space, we implement our method in guided-diffusiin codebase [ and use pre-trained checkpoint for ImageNet 256x256 For conditional sampling on latent space, adopt the stable-diffusion codebase and use their sd-v1-3. ckpt checkpoint, which pre-trained on LAION [2] To obtain the text prompts, randomly sample 10K captions from the IS-COC02014 validation dataset (23). We sample 10K random latent code x for each caption and fix them when using different methods.
model [35]. More concretely, we use the
Bedroom
G More Results
In this section, we will provide more detailed results, including both quantitative and qualitative results.
G.1 More Quantitative Results
Unconditional Sampling. We start by demonstrating detailed results on CIFAR10 21 which are shown in Table The results our proposed method are highlighted in gray. Apart from the results already illustrated in Figure 2 we also include the performance of the DPM-Solver (25)
25

---

Table 10: More unconditional results on LSUN 39
NFE
Sampling Method
7
8
9
10
DDIM
3
40.40
25.56
17.93
13.47
10.77
8.95
DPM-Solver+++3MM
26
17.79
8.03
4.97
4.04
3.79
3.63
DPM-Solver++ + UniC
13.79
6.53
4.58
3.98
3.69
3.52
Uni
11.88
5.99
4.40
3.97
3.74
3.62
UniPC-3-Bg(1))
13.60
6.44
4.47
3.91
3.76
3.54
Table 11: More conditional sampling results on ImageNet 256x256.
NFE
Sampling Method
5
7
8
10
guidance scale
8.0
DDIM 34
38.11
25.31
17.71
14.50
11.51
10.34
DEIS 40
83.80
67.73
54.91
44.91
37.84
31.84
DPM-Solver++ 26
55.64
24.07
14.25
12.22
9.53
8.49
UniPC- B1 (h)
68.69
34.76
22.79
20.45
17.44
16.02
UniPC-Bale (2())
25.87
12.30
8.72
8.68
7.72
7.51
guidance scale 4.0
DDIM 34
27.41
18.47
13.52
11.19
9.66
8.76
DEIS 40
37.86
24.00
16.00
11.69
9,49
8.05
DPM-Solver++ 26
31.57
14.58
9.92
9.43
7.98
7.51
UniPC- MniPC-Bi(H) ( (h)
22.83
12.59
9.77
9.67
8.90
8.52
(2(PC-B2(h)
19.48
10.81
8.18
8.09
7.49
7.31
guidance scale 1.0
DDIM (34
36.08
27.02
21.38
17.59
15.20
13.40
DPM-Solver++ (26
26.10
17.75
13.95
13.10
11.88
11.11
UniPC-
UniPC-B2(h) B2(h)
22.22
15.79
12.72
12.28
11.30
10.84
and our UniPC,, which has varying coefficients (see Appendix C for detailed description). Firstly, we show that DPM-Solver performs very unstable with extremely few steps: the FID of NFE comes to 290.65, which means the solver crashes in this case. Besides, we also show that the DPM- Solver++(M)) [26] performs consistently better than DPM-Solver-3 [25], indicating the multistep method is more effective in few-step sampling setting. That is also why tend to DPM-Solver++ as our baseline method all of our experiments. Table g also shows the comparisons between variants of our UniPC, such UniPC with different instantiations of ((h) and UniPC..- Since the comparisons between B, (h) and (h have already been discussed in Section 4.2 we now focus the analysis of the performance of UniPC.. We find UniPC, achieves the best sampling quality with 7 9 NFE, while cannot beat other variants of UniPC with 5,6, 10 NFE. These results show that different variants of our UniPC may have different applications and should select the most suitable one according to the budget of NFE. We also include more experimental results of the unconditional sampling results FFHQ 18 in LSUN Bedroom 39. summarized in Table 9 and Table TO] Note that there are fewer results for LSUN because we performed most of our experiments on FFHQ and CIFAR 10 in the early stage. Nevertheless, the overall conclusions of the results on different datasets are aligned. To sum up, our results show (1) multistep methods behave better than singlestep ones when the NFE budget is extremely small. (2) our UniC can consistently improve the sampling quality of a wide range of off-the-shelf solvers. (3) selecting a proper variant of UniPC can yield better results in most cases.
Conditional Sampling. We also provide more results on guided sampling ImageNet 256x256, as shown in Table TT. We evaluate the performance DEIS 40 and find performs worse than both the DPM-Solver++ and our UniPC. Besides, we have compared the choice of B(h) on guided
26

---

DDIM 34
DPM-Solver++ 26
UniPC (Ours)
Figure 5: Comparisons between the images sampled from DPM pre-trained on ImageNet256 256
using DDIM 34], DPM-Solver++ (26) and our UniPC with only NFE.
27

---

sampling, where we find (h) significantly outperforms (h),), perhaps because B1 h is too simple and not suitable for the guided sampling. We have also added the results when the guidance scale = 1.0. The results show that our method can achieve better sampling quality with both large and small guidance scales with few sampling steps.
G.2 More Qualitative Results
We provide more visualizations to demonstrate the qualitative performance. First, we consider the conditoondd guided sampling, i.e., the conditional sampling on ImageNet 7 in Figure S Specifically, we compare the sampling quality of each method with only 7 NFE. Note that we randomly sample the class from the total 1000 classes in ImageNet for each image sample, but fix the initial noise for different methods.
28


Reviews:
[{"review_id": "l7XvGlKrdv", "reviewer": "Reviewer_uABf", "summary": "This paper proposes an ODE sampler for diffusion probabilistic models\n(DPM), exploiting the structure of exponential integrators.  The paper\nclaims the proposed sampler can use any existing DPM and achieve high\nsampling quality with very few (<10) number of function evaluations\n(NFE), and also improve upon related methods when using arbitrary\nNFEs.\n", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is well justified theoretically and the\n  experiments validate the proposed approach.\n- The paper clearly states the connections between the proposed method\n  and related existing solvers.\n- The experimental results include  both pixel-space and\n  latent-space diffusion and show consistent improvements when applying\n  the method on top of existing solvers.\n- The method performance is superior to the baselines in both low and\n  middle NFE regimes.\n", "weaknesses": "1) It would be nice if the paper included the statistical significance\n   of the FID scores reported in Tables 1-6. In the ~10 NFE regime\n   sometimes these are very close to the baseline, while for lower\n   quality low NFE one could expect the FID to be not very\n   informative. I plan to revisit my decision after considering the\n   statitstical significance of these results.\n2) The paper repeatedly claims being able to achive higher order p\n   than existing solvers. However I wonder how important it is in\n   practice to achieve order p>3. Table 4 shows when including higher\n   orders the results become poorer.  And also I'd be curious to see\n   this study for the higher quality NFE regimes (eg NFE=10).\n3) Minor:\n   - L. 39 \"output of the model output\"\n   - L. 68-69 extra subscript 0 on lhs?\n   - L. 71 \"obtained\"\n   - L. 97 (after period) Can this be rephrased in a less subjective manner?\n   - (5) I think $\\hat{\\epsilon}_\\theta^{(k)}$ is not defined. Shouldn't\n     the argument in this tem vary with $k$?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive comments on our work! We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About the statistical significance**\n\n**[Reply]** Thanks for the advice. We consider the following hypothesis testing problem:\n$$\nH_0: M_{\\rm DPM-Solver++}\\le M_{\\rm UniPC} \\quad \\text{versus} \\quad H_1: M_{\\rm DPM-Solver++}> M_{\\rm UniPC}\n$$\nwhere $M$ is the metric of interest, and a lower value of $M$ implies better performance.  We derive the statistical significance by running our method and the baseline method for independent trials on different datasets and conducting two-sample t-tests for the aforementioned hypothesis testing problem. For CIFAR10, FFHQ, and LSUN, the metric $M$ is FID50K,  and we compute the results of 10 independent runs. For MS-COCO2014, the metric $M$ is the  $\\ell_2$ distance between the generated latent codes and the ground truth (obtained by a 999-step DDIM), which is calculated by randomly selecting 200 captions as conditions. Note that for both the two metrics (FID50K and $\\ell_2$-Dist), the lower is the better. \n\nThe p-values of the two-sample t-tests on different datasets are presented in the following table (where we use NFE=8,10 as examples):\n\n|Dataset \\ $p$-value|NFE=8|NFE=10|\n|---------|---------|---------|\n|CIFAR10|$1.09\\times 10^{-3}$|$1.46\\times 10^{-3}$|\n|FFHQ|$3.01\\times 10^{-15}$|$3.66\\times 10^{-12}$|\n|LSUN Bedroom|$4.72\\times 10^{-4}$| $4.53\\times 10^{-5}$|\n|MS-COCO2014|$7.04\\times 10^{-5}$|$1.25\\times 10^{-5}$|\n\nWe can find that all the $p$-values are far smaller than $0.01$, which means that we can reject the null hypothesis of no improvement at the significance of $0.01$. In other words, UniPC performs significantly better than DPM-Solver++.\n\n **Q2: About the higher order**\n\n**[Reply]** Thanks for pointing out this. We prove theoretically that higher-order solvers enjoy better accuracy in solving a diffusion ODE. In addition, our method offers a unified framework to empirically investigate the performance of higher-order solvers. From our empirical study, the schemes '123432' for NFE 6 and '1223334' for NFE 7 outperform the low-order schemes with $p \\leq 3$ and all the existing solvers on CIFAR10. Our results also show that simply increasing the order by introducing more _**previous points**_ might not be beneficial to performance. We speculate that this is because the model output of previous points might be more inaccurate and thus would affect the subsequent sampling steps. On the other hand, our UniC increases the convergence order by re-using the _**current point**_, which can consistently improve the sampling quality over the baseline methods (see Table 2). Here we provide some results of different order schedules when NFE=10 on CIFAR10:\n\n|schedule|1223433321|1233343321|1234544321|\n|------------|-----------------|----------|------------|\n|FID$\\downarrow$|4.07 |4.14|4.76|\n\n|schedule|1234554322|1234565432|1234444443|\n|------------|-----------------|----------|------------|\n|FID$\\downarrow$|5.41|18.23|6.84|\n\n **Q3: About the minor issues**\n\n**[Reply]** Thanks for your careful reading. We will modify these as follows:\n- L. 39: \"output of the model output\" $\\rightarrow$ \"model output\"\n- L. 68-69: We use $q_{t0}$ to represent the transition probability from $x_0$ to $x_t$. We will change it into $q_{t|0}$ for better readability.\n- L. 71: \"obtain\"  $\\rightarrow$ \"obtained\"\n- L. 97: We will change this sentence into: \"Despite the rapid development of fast samplers, the quality in few-step sampling still has room for improvement. \"\n- (5): The superscript $k$ denotes the $k$-th derivative of $\\hat{\\epsilon}_\\theta$. We will add the notation in the revised paper.\n\n\n"}, {"review_id": "f7gFB1hFxj", "reviewer": "Reviewer_vcSP", "summary": "In this paper, the authors present a novel sampling solver called UniPC for diffusion models. UniPC consists of two parts: UniC and UniP, where UniC corrects the estimation with prediction of current timestep and it can be applied other sampling solvers and UniP is a special case of UniC and share the similar form for sampling. Compared with previous methods(e.g. DPMSolver), empirical evaluation have verified the effectiveness of UniPC in generating better results with less sampling steps.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposed method can significantly reduce the number of sampling steps to less than 10, without losing the quality of the results.\n-\tThe proposed method has some generality and can further improve the approximate accuracy of other existing DPM samplers.\n", "weaknesses": "- Experiments. Please conduct experiments on datasets with larger resolution images to measure inference time, quality, and diversity.\n- Recent work: Please add more details on the related wok about DPM solver and some related ODE-based sampling solver.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive comments on our work! We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About experiments on larger resolution images**\n\n**[Reply]** Thanks for your advice. In our original paper, we have already conducted experiments on $512\\times 512$ images (sampling from stable-diffusion). Here we compare our method and the baseline DPM-Solver++ using a larger model _**Stable-Diffusion-XL**_, which can produce $1024\\times 1024$ images. We randomly select 200 captions and generate the samples using UniPC and DPM-Solver++ implemented in the diffusers library. Due to the larger resolution, we evaluate both methods with NFE=15, and the results are listed in the following table (the evaluation protocol is the same as the original paper):\n\n|Method|Quality: $\\ell_2$-Dist ($\\downarrow$)|Diversity: IS($\\uparrow$)|Inference Time (s)|\n|------|----------|----------|---------|\n|DPM-Solver++|0.741|8.494|4.33$\\pm$0.02|\n|UniPC|0.669|8.909|4.26$\\pm$0.01|\n\nOur results show that UniPC achieves better performance in all three metrics, indicating that UniPC can also be a good choice when sampling from a large diffusion model like Stable-Diffusion-XL. We also provide some qualitative results in the attached one-page PDF, where we show that the images generated by UniPC are more realistic and have much fewer artifacts compared with DPM-Solver++.\n\n**Q2: About the recent work**\n\n**[Reply]** Thanks for the suggestion. We will elaborate on the details of the related work and below is another paragraph we plan to add to Section 2.2 (due to a display issue of OpenReview, we cannot type too many formulae here, but we will include more details in the revised paper): \n\nBased on the exponential integrator, [25] proposes to approximate $\\hat{\\epsilon}_{\\theta}$  via Taylor expansion and views DDIM as DPM-Solver-1. [26] considers the data prediction scheme by rewriting (2) and demonstrates its effectiveness in conditional sampling. [40] derives the Taylor expansion formulae with respect to t instead of the half log-SNR. [24] employs pseudo-numerical methods such as the Runge-Kutta method. Although many aforementioned high-order solvers are proposed, existing solvers of diffusion ODEs can be explicitly computed for orders not greater than 3, due to the lack of analytical forms. \n\n\n"}, {"review_id": "BzJzT1kSfK", "reviewer": "Reviewer_TtDL", "summary": "This paper presents a universal predictor-corrector method for faster sampling of diffusion models, with model retraining. The key idea is to further include the current point along with previous $p$ points while estimating the data point by adding a correction step. It shows that this model can achieve an order of accuracy $p+1$. Experiments on several image datasets show that this proposed method achieves strong performance in comparison to prior SOTA methods.\n\n", "questions": "Please see the weaknesses above", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is generally written clearly \n\nThe problem of accelerating diffusion model is critical to solve\n\nReasonably good performance achieved", "weaknesses": "**Limited novelty and design**:\n- The idea proposed here is marginal in the sense that at each step, the current estimate is additionally used along with some previous points  (the latte is the same as [25,34,40]); Given the sequential nature, the current point is actually already used by previous methods [25,34,40] in the next steps (with a step delay), whilst it is just that this point is used for more times. No extra information is used whilst also no clear understand on why this extra use of the current point can lead to increase in the order of accuracy. \n\n- Besides, in general, there are two families of diffusion models: DDPM (e.g., [A, 34]) and SGM (e.g., [35]). And the DDPM series are often easier to speed up due to adopting a variance preserving (VP) process, in contrast to SGM’s variance exploding (VE) process. It is not clear which family this proposed method is focused on, or both?\n\n- It looks the step of 10 is a milestone. Is there any relationship or theoretical implication of this proposed method on tackling this issue?\n\n- Also, the addition operations introduce some complex parameter, which adds further burden to the model tuning process. It is unclear if these parameters are general for different datasets. \n\n**Limited results gain**\n- Whilst the numeral results such as FID looks good in comparison, when it comes to the visual examples in supplementary, the generated images are of similar quality as previous methods such as DPM-Solver++. It is known that numerical metrics are some limited in the visual perception evaluation. Overall, the generation performance gain is some limited, not as convincing as what claimed.\n\n- It is some inconsistent when evaluating UniC/UniP/UniPC. How is this selection done?\n\n- If I am not missing, there is no exact evaluation on the benefit of using the current point in estimation, which is the key design of the whole  model. \n\n**References**:\n- [A] Jonathan et al. Denoising Diffusion Probabilistic Models. NeurIPS 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About the novelty and design**\n\n**[Reply]** \n- As discussed in Section 3.1 and proved in Appendix E.3, our UniC can indeed increase the order of accuracy of the sampling procedure. We would like to clarify that our usage of the current point is better than [25, 34, 40] because we reduce the error of the current point via a corrector step, while in previous methods there are more accumulative errors. We will add more discussion about how our method works in the revised paper.\n- Thanks for your advice. Our method is designed for the variance preserving (VP) diffusion models, similar to DPM-Solver. VP diffusion models are also more useful in practice, such as the Stable-Diffusion, DeepFloyd-IF, etc. We will clarify this in our paper.\n- We think that the proposed corrector (UniC) is the main reason why our method works better than previous methods when NFE<10. Existing methods would suffer from large accumulative errors with large step sizes. However, our method can mitigate this issue because we can obtain a better estimation for the current point due to the extra corrector step. Please also refer to Table.2, where we show that UniC can boost the performance of a variety of existing solvers.\n- Sorry for the confusion. Our method is totally training-free, containing no extra learnable parameters. Our method can serve as a drop-in replacement of existing samplers of diffusion models and accelerate the sampling process. \n\n**Q2: About the results gain**\n\n**[Reply]** \n- We would need to focus more on the detailed structure of the images when comparing the qualitative results. For example in Figure 4, it can be easily found that our UniPC generates more realistic images than DPM-Solver++ (some blurry or broken regions can be easily observed in the images generated by DPM-Solver++). We also encourage the reviewer to have a look at the results of sampling from a larger model _**Stable-Diffusion-XL**_ in the attached one-page PDF, where we show that UniPC can generate more realistic images than the baseline method.\n- For most of the experiments, we use the UniPC (which is the combination of UniP and UniC) to compare with other methods. In the ablation study of UniC (see Table 2), we directly add the UniC after a variety of existing solvers demonstrate the UniC can be a plug-and-play module to improve sampling quality. \n- Please refer to Table 2, where we clearly show that using the current point in the corrector step can consistently boost performance.\n"}, {"review_id": "EFanAAezI6", "reviewer": "Reviewer_5Qcj", "summary": "This paper develops a unified corrector (UniC) that can be applied after any existing DPM sampler to increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC,  a unified predictor-corrector framework called UniPC for the fast sampling of DPMs  has a unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps. ", "questions": "They claim many times on unified or model-agnostic but no experiments support their claim, the method only tested on DPM. Also the writing make me lost  many times when reading the paper, I suggest you rewrite your motivation part by which technical issue in DPM you try to solve, why your method works.  The theory on faster convergence property is also confused on accuracy, is it really useful?  how to  theoretically guarantee on the accuracy. ", "limitations": "See Weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "On both unconditional and conditional sampling using pixel-space and latent-space DPMs. UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet 256 256 (conditional) with only 10 function evaluations.", "weaknesses": "They claim many times on unified or model-agnostic but no experiments support their claim, the method only tested on DPM. Also the writing make me lost  many times when reading the paper, I suggest you rewrite your motivation part by which technical issue in DPM you try to solve, why your method works.  The theory on faster convergence property is also confused on accuracy, is it really useful?  how to  theoretically guarantee on the accuracy. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive comments on our work! We address the questions and clarify the issues accordingly as described below.\n\n**Q1: About model-agnostic**\n\n**[Reply]** Sorry for the confusion. By “model-agnostic” we mean that (1): our UniC can be applied after any existing solver, which is proved in Table 2; (2): like other training-free samplers, our method can be used to sample from any off-the-shelf DPMs, (pixel/latent space DPMs of different resolutions, see Figure 2,3). We will clarify this in the revised paper.\n\n**Q2: About the motivation**\n\n**[Reply]** Thanks for your suggestion. Our main motivation is that existing solvers for DPM often suffer from large accumulative error when sampling with <10 NFEs, due to the large step size. To overcome this issue, we propose to use a corrector than re-use the current point to reduce the error and increase the order of accuracy (which are proved theoretically and empirically). We will add the above discussion to make our motivation clearer.\n\n**Q3: About the accuracy** \n\n**[Reply]** The theoretical accuracy (i.e., order of accuracy/convergence order) is guaranteed by the Theorem 3.1 (which is proved in Appendix E.3). We have also conducted extensive experiments to further prove that our method can indeed improve the sampling quality of the DPMs. We will also improve our writing to highlight the novelty and usefulness of our newly proposed UniPC.\n"}, {"review_id": "3gYr9nbUWv", "reviewer": "Reviewer_xuvS", "summary": "This paper propose a predictor-corrector method to accelerate the diffusion sampling process, where a corrector is proposed to correct the initial estimation of x_t using previous and current points. The experiments are conducted on imagenet and cifar10 both of which outperform existing efficient sampler at very few sampling steps. ", "questions": "N/A", "limitations": "The authors adequately addressed the limitations ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Improving existing sampling method by a unified predictor-corrector solver is reasonable. \n2. Adequate theoretical and empirical analysis of the proposed methods. \n3. The paper is well organized and easy to understand. ", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive comments on our work, especially the appreciation of our newly proposed unified framework UniPC, our adequate theoretical and empirical analysis, and our superior performance. We hope our work can open a new avenue for improving the sampling quality in the few-step sampling scenario via a predictor-corrector paradigm and thus promote the application of AIGC."}]

Author rebuttal:
We sincerely thank the reviewers for the positive feedback and valuable comments on our work. As suggested by Reviewer vcSP, we further compare the sampling quality of our method UniPC and the baseline DPM-Solver++ using _**Stable-Diffusion-XL**_, a newly released model which can generate $1024\times 1024$ images. We highly encourage the reviewers to have a look at the attached one-page PDF for the qualitative results, where it can be found that our method consistently generates more realistic images with fewer visual flaws compared with DPM-Solver++.

Meta-review: This paper aims to improve the sampling speed of the diffusion model with a predictor-corrector framework. It still receives mixed scores (7, 6, 5, 5, 3) after the discussion period. The remaining concerns are about the novelty and experimental performance of this paper. However, this work receives multiple positive comments like " well justified theoretically", " the experiments validate the proposed approach", " reasonably good performance achieved", and " generally written clearly".

Overall, the AC thinks this work can be of large interest to the community working on diffusion model but still hopes the authors emphasize the novelty of this work, as suggested by the reviewers TtDL.
Decision: Accept (poster)