Title: UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

fast sampling

predictor-corrector

training-free
Abstract: Diffusion probabilistic models (DPMs) have demonstrated a very promising ability in high-resolution image synthesis. However, sampling from a pre-trained DPM is time-consuming due to the multiple evaluations of the denoising network, making it more and more important to accelerate the sampling of DPMs. Despite recent progress in designing fast samplers, existing methods still cannot generate satisfying images in many applications where fewer steps (e.g., $<$10) are favored. In this paper, we develop a unified corrector (UniC) that can be applied after any existing DPM sampler to increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC, we propose a unified predictor-corrector framework called UniPC for the fast sampling of DPMs, which has a unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps. We evaluate our methods through extensive experiments including both unconditional and conditional sampling using pixel-space and latent-space DPMs. Our UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet 256$\times$256 (conditional) with only 10 function evaluations. Code is available at https://github.com/wl-zhao/UniPC.

UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models

Abstract

Diffusion probabilistic models (DPMs) have demonstrated a very promising ability in high-resolution image synthesis. However, sampling from a pre-trained DPM is time-consuming due to the multiple evaluations of the denoising network, making it more and more important to accelerate the sampling of DPMs. Despite recent progress in designing fast samplers, existing methods still cannot generate satisfying images in many applications where fewer steps ( e.g ., < 10) are favored. In this paper, we develop a unified corrector (UniC) that can be applied after any existing DPM sampler to increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC, we propose a unified predictor-corrector framework called UniPC for the fast sampling of DPMs, which has a unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps. We evaluate our methods through extensive experiments including both unconditional and conditional sampling using pixel-space and latent-space DPMs. Our UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet 256 × 256 (conditional) with only 10 function evaluations. Code is available at https://github.com/wl-zhao/UniPC .

1 Introduction

Diffusion probabilistic models (DPMs) [33, 13, 35] have become the new prevailing generative models and have achieved competitive performance on many tasks including image synthesis [8, 29, 13], video synthesis [15], text-to-image generation [27, 29, 12], voice synthesis [5], etc . Different from GANs [10] and VAEs [20], DPMs are trained to explicitly match the gradient of the data density ( i.e ., score), which is more stable and less sensitive to hyper-parameters. However, sampling from a pre-trained DPM usually requires multiple model evaluations to gradually perform denoising from Gaussian noise [13], consuming more inference time and computational costs compared with single-step generative models like GANs.

Recently, there have been increasing efforts to accelerate the sampling of DPMs [31, 28, 34, 25, 40]. Among those, training-free methods [34, 25, 40] enjoy a wider usage in applications because they can be directly applied to off-the-shelf pre-trained DPMs. Although these methods have significantly reduced the sampling steps from 1000 to less than 20 steps, the sampling quality with extremely few steps ( e.g ., < 10) has been rarely investigated. Few-step sampling can be used in many scenarios where we need to efficiently obtain plausible samples, such as designing a proper prompt for a textto-image diffusion model [29, 30] and computing losses on the generated samples during the training of some diffusion-based visual systems [1, 6]. However, current fast samplers usually struggle to generate high-quality samples within 10 steps (see Figure 1).

In this paper, we propose a training-free framework for the fast sampling of DPMs called UniPC . We find that UniPC significantly outperforms existing methods within 5 ∼ 10 NFE (number of function evaluations), and can also achieve better sampling quality with more sampling steps. Specifically, we first develop a unified corrector (UniC) which works by using the the model output ϵ θ ( x t i , t i ) at the current timestep t i to obtain a refined x c t i . Different from the predictor-corrector paradigm in numerical ODE solver that requires a doubled NFE, our UniC reuses the model output ϵ θ ( x t i , t i ) to the next sampling step, thus introducing no extra function evaluation. UniC can be applied after any existing DPM sampler to increase the order of accuracy, while the inference speed is almost unaffected. Interestingly, we also find that by simply changing a hyper-parameter in UniC, a new family of predictors (UniP) can be further obtained.

Since our UniC is method-agnostic, we combine UniP and UniC to obtain a new family of fast samplers called UniPC. Different from previous fast solvers [26, 25, 40] that either have no higherorder ( e.g ., > 3 ) variants or have no explicit forms, our UniPC supports arbitrary orders with a unified analytical expression and are easy to implement. Benefiting from the universal design, variants of UniPC ( e.g ., singlestep/multistep, noise/data prediction) can be easily derived. We theoretically prove that UniPC enjoys higher convergence order and empirically demonstrate that UniPC has better sampling quality in a variety of scenarios. We also show that the inference speed and memory usage of UniPC is similar to DPM-Solver++ [26], indicating that UniPC can achieve superior performance under the same computational budgets.

We conduct extensive experiments with both pixel-space and latent-space DPMs to verify the effectiveness of the proposed UniPC. Our results show that UniPC performs consistently better than previous state-of-the-art methods on both unconditional and conditional sampling tasks. Notably, UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet 256 × 256 (conditional) with only 10 function evaluations. We also demonstrate that UniC can improve the sampling quality of several existing fast samplers significantly with very few NFE (number of function evaluations). Some qualitative comparisons are shown in Figure 1, where we observe that our UniPC can generate images with more visual details than other methods.

2 Background and Related Work

2.1 Diffusion Probabilistic Models

For a random variable x 0 with an unknown distribution q 0 ( x 0 ) , Diffusion Probabilistic Models (DPMs) [33, 13, 19] transit q 0 ( x 0 ) at time 0 to a normal distribution q T ( x T ) ≈ N ( x T | 0 , ˜ σ 2 I ) at time T for some ˜ σ > 0 by gradually adding Gaussian noise to the observation x 0 . For each time t ∈ [0 , T ] , and given σ t , α t > 0 , the Gaussian transition is

where α 2 t /σ 2 t (the signal-to-noise-ratio (SNR)) is strictly decreasing w.r.t. t [19].

Let ϵ θ ( x t , t ) denote the noise prediction model using data x t to predict the noise ϵ , and the parameter θ is obtained by minimizing

where x 0 ∼ q 0 ( x 0 ) , t ∈ U [0 , T ] , and the weight function ω ( t ) > 0 . Sampling from DPMs can be achieved by solving the following diffusion ODEs [35]:

where f ( t ) = dlog α t d t , g 2 ( t ) = d σ 2 t d t -2 dlog α t d t σ 2 t .

2.2 Fast Sampling of DPMs

Fast samplers of DPMs can be either training-based [31, 2, 37] or training-free [25, 26, 40, 24, 41]. Training-based samplers require further training costs while training-free methods directly use the original information without re-training and are easy to implement in conditional sampling. The essence of training-free samplers is solving stochastic differential equations (SDEs)[13, 35, 3, 41] or ODEs[26, 40, 24, 34, 25]. Other fast sampling methods include modifying DPMs [9] and the combination with GANs [38, 36].

Among others, samplers solving diffusion ODEs are found to converge faster for the purpose of sampling DPMs [34, 35]. Recent works [40, 25, 26] show that ODE solvers built on exponential integrators [17] appear to have faster convergence than directly solving the diffusion ODE (1). The solution x t of the diffusion ODE given the initial value x s can be analytically computed as [25]:

where we use the notation ˆ ϵ θ and ˆ x λ to denote changing from the domain of time( t ) to the domain of half log-SNR( λ ), i.e ., λ t = log( α t /σ t ) , ˆ x λ := x t λ ( λ ) and ˆ ϵ θ ( · , λ ) := ϵ θ ( · , t λ ( λ )) .

Based on the exponential integrator, [25] proposes to approximate ˆ ϵ θ via taylor expansion and views DDIM as DPM-Solver-1, i.e.,

[26] considers rewriting (2) using ˆ x θ instead of ˆ ϵ θ ; [40] derives the taylor expansion formulae with respect to t instead of the half log-SNR( λ ). [24] employs pseudo numerical methods such as Runge-Kutta method directly for the updating of ϵ θ of (3). Although many aforementioned high-order solvers are proposed, existing solvers of diffusion ODEs can be explicitly computed for orders not greater than 3, due to the lack of analytical forms.

3 A Unified Predictor-Corrector Solver

In this section, we propose a unified predictor-corrector solver of DPMs called UniPC, consisting of UniP and UniC. Our UniPC is unified in mainly two aspects: 1) the predictor (UniP) and the corrector (UniC) share the same analytical form; 2) UniP supports arbitrary order and UniC can be applied after off-the-shelf fast samplers of DPMs to increase the order of accuracy.

3.1 The Unified Corrector UniCp

Modern fast samplers based on discretizing diffusion ODEs [25, 34, 40] aim to leverage the previous p points { ˜ x t i -k } p k =1 to estimate ˜ x t i with p order of accuracy. Despite the rapid development of fast samplers, the quality of few-step sampling still has room for improvement. In this paper, we propose a corrector called UniCp to improve the initial estimation using not only the previous p points but also the current point. Formally, after obtaining the initial estimation ˜ x t i , we perform the correction step through the following formula:

Algorithm 1 UniCp

Algorithm 2 UniPp

where ˜ x c t i denotes the corrected result, B ( h ) = O ( h ) is a non-zero function of h , h i = λ t i -λ t i -1 is the step size in the half-log-SNR( λ ) domain, r 1 < r 2 < · · · < r p = 1 are a non-zero increasing sequence, determining which previous points are used. Specifically, we use { r i } p m =1 to interpolate between λ t i -1 to λ t i to obtain the auxiliary timesteps s m = t λ ( r m h + λ t i -1 ) , m = 1 , 2 , . . . , p . The model outputs at these timesteps are used to compute D m by

Wenowdescribe how to choose { a m } p m =1 in UniCp to effectively increase the order of accuracy. The main idea is to cancel out low-order terms between the numerical estimation (4) and the theoretical solution (2). In practice, we expand the exponential integrator in (2) as follows:

where ˆ ϵ ( k ) θ denotes the k -th derivative of ˆ ϵ θ , and φ k ( h ) can be analytically computed [16]. The { a m } p m =1 can be then determined by matching the coefficients between (4) and (6). In the following theorem, we show that UniCp has an order of accuracy p +1 (see Appendix E.3 for detailed proof).

Theorem 3.1 (The Order of Accuracy of UniCp ) . For any non-zero sequence { r i } p i =1 and h > 0 , define

Let ϕ p ( h ) = ( ϕ 1 ( h ) , · · · , ϕ p ( h )) ⊤ with ϕ n ( h ) = h n n ! φ n +1 ( h ) , where φ n ( h ) is defined by the recursive relation [16]:

For an increasing sequence r 1 < r 2 < · · · < r p = 1 , suppose a p := ( a 1 , . . . , a p ) ⊤ satisfies,

where | · | denotes the l 1 norm for matrix. Then, under regularity conditions in Appendix E.2, UniCp of (4) will have ( p +1) -th order of accuracy.

The monotonicity of { r i } p i =1 ensures the invertibility of the Vandermonde matrix R p . Therefore, we can take a p = R -1 p ( h i ) ϕ p ( h i ) /B ( h i ) as the coefficient vector for (4) for simplicity, where B ( h )

can be any function of h such that B ( h ) = O ( h ) , for example B 1 ( h ) = h , B 2 ( h ) = e h -1 . The detailed implementation of UniC is shown in Algorithm 1. Importantly, we circumvent the extra evaluation of ϵ θ (˜ x c t i , t i ) by pushing ϵ θ (˜ x t i , t i ) into the buffer Q instead of ϵ θ (˜ x c t i , t i ) . Taking full advantage of ϵ θ (˜ x t i , t i ) of previous results enables us to increase the order of accuracy without incurring significant increment of computation cost. This makes our method inherently different from the predictor-corrector methods in ODE literature [22], where the computational costs are doubled because an extra function evaluation on the corrected ˜ x c t i is required for each step.

3.2 The Unified Predictor UniPp

We find that the order of accuracy of UniC does not depend on the specific choice of the sequence { r i } p i =1 , which motivates us to design p -order unified predictor (UniPp ) which only leverages the previous p data points by excluding D p in (4) since D p involves ˜ x t i . The order of accuracy is guaranteed by the following corollary.

Corollary 3.2 (The Order of Accuracy of UniPp ) . For an increasing sequence r 1 < r 2 < · · · < r p -1 < 1 , the solver given in (4) dropping the term D p and using coefficients that satisfies

has p -th order of accuracy.

Due to the unified form of UniP and UniC, we can use UniPp as the implementation of the Solver-p in UniCp to obtain a new family of solvers called UniPCp . Theorem 3.1 and Corollary 3.2 ensure that UniPCp can achieve ( p + 1) -th order of accuracy. Moreover, under additional regularity conditions in Appendix D, based on Theorem 3.1 and Corollary 3.2, we show that the order of convergence of UniPCp reaches p +1 (see Appendix D).

3.3 Comparison with Existing Methods

Here we discuss the connection and the difference between UniPC and previous methods. When p = 1 , UniPC will reduce to DDIM [34]. Motivated by linear multistep approaches, PNDM [24] proposes to use pseudo numerical methods for DDIM, while our UniPC makes use of information in the ODE solution (2) and is specially designed for diffusion ODEs. DEIS [40] is built on exponential integrators in the time domain, where the integral cannot be analytically computed and explicit formulae for high-order solvers cannot be derived. By using the half log -SNR λ [25, 26], it is shown that the application of integration-by-parts can simplify the integration of (2) and leads to explicit expansion of x t . DPM-Solver-2 [25] lies in our UniPC framework as UniP1 , where they assume B ( h ) = e h -1 . We find through our numerical analysis that B ( h ) can be any non-degenerate function such that B ( h ) = O ( h ) . Furthermore, DPM-Solvers do not admit unified forms even for orders smaller than 3 , which adds to the challenge of obtaining algorithms for higher orders. In contrast, our UniPC exploits the structure of exponential integrators w.r.t. half log -SNR and admits not only simple and analytical solutions for efficient computation but also unified formulations for easy implementation of any order.

3.4 Implementation

By setting r m = ( λ t i -m -1 -λ t i ) /h i , m = 1 , . . . , p -1 , the UniPCp updates in a multistep manner, which reuses the previous evaluation results and proves to be empirically more efficient, especially for limited steps of model evaluation [11, 26], while singlestep methods might incur higher computation cost per step. Therefore, we use multistep UniPC in our experiments by default. The detailed algorithms for multistep UniPC and the proof of convergence can be found in Appendix B. For UniPC, the choices of { r i } p -1 i =1 determine different updating methods. If all the values are in (0 , 1] , the UniPC will switch to singlestep. Notably, we find in experiments that our UniC consistently improves different updating methods. Besides, we find UniP-2 (8) and UniC-1 (7) degenerate to a simple equation where only a single a 1 is unknown, where we find a 1 = 0 . 5 can be a solution for both B 1 ( h ) and B 2 ( h ) (see Appendix F) independent of h . In Appendix C, we provide another variant of UniPC called UniPC v where the coefficients do not depend on h for arbitrary order p .

In the conditional inference, guided sampling [14, 8] is often employed. Recent works [30, 26] find that thresholding data prediction models can boost the sampling quality and mitigate the problem of

5

6

7

8

NFE

5

10

15

20

25

30

FID

UniPC

DPM-Solver++

5

6

7

8

NFE

4

6

8

10

12

14

16

18

20

FID

UniPC

DPM-Solver++

(b) LSUN Bedroom (Latent DPM)

5

6

7

8

NFE

10

15

20

25

30

FID

UniPC

DPM-Solver++

train-test mismatch. Our framework of UniPC can be easily adapted to the data prediction model, see Appendix A for algorithms and theoretical analysis. The detailed algorithms for multistep UniPC for data prediction are in Appendix B. Hence, UniPC with data prediction can achieve fast conditional sampling in extremely few steps through dynamic thresholding.

4 Experiments

In this section, we show that our UniPC can significantly improve the sampling quality through extensive experiments. Our experiments cover a wide range of datasets, where the image resolution ranges from 32 × 32 to 256 × 256. Apart from the standard image-space diffusion models [35, 8], we also conduct experiments on the recent prevailing stable-diffusion [29] trained on latent space. We will first present our main results in Section 4.1 and then provide a detailed analysis in Section 4.2.

4.1 Main Results

We start by demonstrating the effectiveness of our UniPC on both unconditional sampling and conditional sampling tasks, with extremely few model evaluations ( < 10 NFE). For the sake of clarity, we compare UniPC with the previous state-of-the-art method DPM-Solver++ [26]. We have also conducted experiments with other methods including DDIM [34], DPM-Solver [25], DEIS [40], and PNDM [24]. However, since some of these methods perform very unstable in few-step sampling, we leave their results in Section 4.2 and Appendix G.

Unconditional sampling. We first compare the unconditional sampling quality of different methods on CIFAR10 [21], FFHQ [18], and LSUN Bedroom [39]. The pre-trained diffusion models are from [35] and [29], including both pixel-space and latent-space diffusion models. The results are shown in Figure 2. For DPM-Solver++, we use the multistep 3-order version due to its better performance. For UniPC, we use a combination of UniP-3 and UniC-3, thus the order of accuracy is 4. As shown in Figure 3, we find that our UniPC consistently achieves better sampling quality than DPM-Solver++ on different datasets, especially with fewer NFE. Notably, compared with DPM-Solver++, our UniPC improves the FID by 6.0, 5.9, and 8.5 on CIFAR10, LSUN Bedroom, and FFHQ, respectively. These results clearly demonstrate that our UniPC can effectively improve the unconditional sampling quality with few function evaluations.

Conditional sampling. Conditional sampling is more useful since it allows user-defined input to control the synthesized image. To evaluate the conditional sampling performance of our UniPC, we conduct experiments on two widely used guided sampling settings, including classifier guidance and classifier-free guidance. For classifier guidance, we use the pixel-space diffusion model trained on ImageNet 256 × 256 [7] provided by [8]. Following DPM-Solver++, we use dynamic thresholding [30] to mitigate the gap between training and testing. The results are shown in Figure 3a and 3b, where we compare our UniPC with DPM-Solver++ [26] under different guidance scale ( s = 8 . 0 / 4 . 0 ). For DPM-Solver++, we use the multistep 2-order version (2M), which achieves the best results

5

6

7

8

NFE

10

20

30

40

50

FID

s

=8.0

UniPC

DPM-Solver++

5

6

7

8

NFE

10

15

20

25

30

FID

s

=4.0

UniPC

DPM-Solver++

5

6

7

8

NFE

0.18

0.20

0.22

0.24

0.26

2-Dist

s

=1.5

UniPC

DPM-Solver++

(c) MS-COCO2014 (Latent DPM)

(a) CIFAR10 (Pixel-space DPM)

(b) FFHQ (Latent-space DPM)

according to the original paper. For our UniPC, we use UniP-2 and UniC-2. It can be seen that our UniPC generates samples with better quality and converges rather faster than other methods. For classifier-free guidance, we adopt the latent-space diffusion model provided by stable-diffusion [29] and set the guidance scale as 1.5 following their original paper. To obtain the input texts, we randomly sample 10K captions from MS-COCO2014 validation dataset [23]. As discussed in [26], the FID of the text-to-image saturates in < 10 steps, possibly because the powerful decoder can generate good image samples from non-converged latent codes. Therefore, to examine how fast a method converges, we follow [26] to compute the l 2 -distance between the generated latent code x 0 and the true solution x ∗ 0 (obtained by running a 999-step DDIM), i.e ., ∥ x 0 -x ∗ 0 ∥ 2 / √ D , where D is the dimension of the latent code. For each text input, we use the same initial value x ∗ T sampled from Gaussian distribution for all the compared methods. It can be seen in Figure 3c that our UniPC consistently has a lower l 2 -distance than DPM-Solver++, which indicates that UniPC converges faster in guided sampling.

4.2 Analysis

In this section, we will provide more detailed analyses to further evaluate the effectiveness of UniPC.

Ablation on the choice of B ( h ) . In Section 3, we mentioned that B ( h ) is set to be any non-zero function of h that satisfies B ( h ) = O ( h ) . We now investigate how the choice of B ( h ) would affect the performance of our UniPC. Specifically, we test two simple forms: B 1 ( h ) = h and B 2 ( h ) = e h -1 and the results are summarized in Table 1, where we also provide the performance of DPM-Solver++ [26] for reference. We show that UniPC with either implementation of B ( h ) can outperform DPM-Solver++. When the NFE is extremely small (5 ∼ 6), we observe that B 1 ( h ) consistently outperforms B 2 ( h ) by 1 ∼ 3 in FID. On the other hand, as the NFE increases, the performance of B 2 ( h ) catches up and even surpasses B 1 ( h ) in some experiments ( e.g ., on CIFAR10 and LSUN Bedroom). As for the guided sampling, we find B 1 ( h ) is worse than B 2 ( h ) consistently (see Appendix G for detailed results and discussions). These results also inspire us that our UniPC can be further improved by designing better B ( h ) , which we leave to future work.

UniC for any order solver. As shown in Algorithm 1, our UniCp can be applied after any p -order solver to increase the order of accuracy. To verify this, we perform experiments on a wide range of solvers. The existing solvers for DPM can be roughly categorized by the orders or the updating method ( i.e ., singlestep or multistep). Since DPM-Solver++ [26] by design has both singlestep and multistep variants of 1 ∼ 3 orders, we apply our UniC to different versions of DPM-Solver++ to see whether UniC can bring improvements. The results are reported in Table 2, where the sampling quality is measured by FID ↓ on CIFAR10 by sampling from a pixel-space DPM [35]. We also provide the order of accuracy of each baseline method without/with our UniC. Apart from the DDIM [34], which can be also viewed as 1-order singlestep DPM-Solver++, we consider another 3 variants of DPM-Solver++ including 2-order multistep (2M), 3-order singlestep (3S) and 3-order multistep (3M). It can be found that our UniC can increase the order of accuracy of the baseline methods by 1 and consistently improve the sampling quality for the solvers with different updating methods and orders.

Exploring the upper bound of UniC. According to Algorithm 1, our UniC works by leveraging the rough prediction ˜ x t i as another data point to perform correction and increase the order of accuracy. Note that to make sure there is no extra NFE, we directly feed ϵ θ (˜ x t i , t i ) to the next updating step instead of re-computing a ϵ θ (˜ x c t i , t i ) for the corrected ˜ x c t i . Although the error caused by the misalignment between ϵ θ (˜ x t i , t i ) and ϵ θ (˜ x c t i , t i ) has no influence on the order of accuracy (as proved in Appendix E.7), we are still interested in how this error will affect the performance. Therefore, we conduct experiments where we re-compute the ϵ θ (˜ x c t i , t i ) as the input for the next sampling step, which we name as 'UniC-oracle'. Due to the multiple function evaluations on each t i for both ˜ x t i and ˜ x c t i , the real NFE for UniC-oracle is twice as the standard UniC for the same sampling steps. However, UniC-oracle is very helpful to explore the upper bound of UniC, and thus can be used in pre-experiments to examine whether the corrector is potentially effective. We compare the performance of UniC and UniC-oracle in Table 3, where we apply them to the DPM Solver++ [26] on LSUN Bedroom [39] and FFHQ [18] datasets. We observe that the UniC-oracle can significantly improve the sampling quality over the baseline methods. Although the approximation error caused by the misalignment makes UniC worse than UniC-oracle, we find that UniC can still remarkably increases the sampling quality over the baselines, especially with few sampling steps.

Customizing order schedule via UniPC. Thanks to the unified analytical form of UniPC, we are able to investigate the performance of arbitrary-order solvers and customize the order schedule freely. As a first attempt, we conduct experiments on CIFAR10 with our UniPC, varying the order schedule (the order at each sampling step). Some results are listed in Table 4, where we test different order schedules with NFE=6/7 because the search space is not too big. Note that the order schedule in Table 4 represents the order of accuracy of the UniP, while the actual order is increased by 1 because of UniC. Our default order schedule follows the implementation of DPM-Solver++ [26], where lower-order solvers are used in the final few steps. Interestingly, we find some customized order schedules can yield better results, such as 123432 for NFE=6 and 1223334 for NFE=7. We also show that simply increasing the order as much as possible is harmful to the sampling quality.

Sampling diversity. Apart from the sampling quality, we are also interested in the diversity of the images generated by UniPC. In Table 6, we compare the sampling diversity of UniPC and

DPM-Solver++ [26], measured by the inception score (IS) on CIFAR10 dataset. We find that UniPC consistently generates more diverse samples in a variant number of function evaluations.

Comparisons with more NFE. To further evaluate the effectiveness of our UniPC, we also perform experiments with 10 ∼ 25 NFE. Specifically, we perform guided sampling on ImageNet 256 × 256 [7] with guidance scale 8 . 0 and compare our UniPC with more existing methods including DDIM, DPMSolver, PNDM, DEIS, and DPM-Solver++. We summarize the results in Table 5, where some results of the previous methods are from [26]. The results clearly demonstrate that our UniPC surpasses previous methods by a large margin.

Inference speed and memory. We test the wall-clock time of UniPC by sampling from a stablediffusion model [29] using a single NVIDIA RTX 3090 GPU and the results are shown in Table 7. We find the actual inference time of UniPC is very similar to DPM-Solver++ [26]. As for memory usage, it is only related to how many previous model outputs are stored. Therefore, our UniPC also costs similar memory to DPM-Solver++. For example, both UniPC-2 and DPM-Solver++(2M) cost about 6.3GB of memory when sampling from a stable-diffusion model.

Visualizations. We provide a qualitative comparison between our UniPC and previous methods with only 7 NFE, as shown in Figure 1. We use different methods to perform guided sampling with a guidance scale of 8.0 from a DPM trained on ImageNet 256 × 256. We find DEIS [40] tends to crash with extremely few steps, while the sample generated by DDIM [34] is relatively blurry. Compared with DPM-Solver++, the state-of-the-art method in guided sampling, UniPC can generate more plausible samples with better visual details. We further compare the sampling quality of our method UniPC and DPM-Solver++ using Stable-Diffusion-XL, a newly released model that can generate 1024 × 1024 images. The results in Figure 4 show that our method consistently generates more realistic images with fewer visual flaws.

Limitations and broader impact. Despite the effectiveness of UniPC, it still lags behind trainingbased methods such as [31]. How to further close the gap between training-free methods and training-based methods requires future efforts.

5 Conclusions

In this paper, we have proposed a new unified predictor-corrector framework named UniPC for the fast sampling of DPMs. Unlike previous methods, UniPC has a unified formulation for its two components (UniP and UniC) for any order. The universality of UniPC makes it possible to customize arbitrary order schedules and to improve the order of accuracy of off-the-shelf fast sampler via UniC.

Extensive experiments have demonstrated the effectiveness of UniPC on unconditional/conditional sampling tasks with pixel-space/latent-space pre-trained DPMs. We have also discovered several directions where UniPC can be further improved, such as choosing a better B ( h ) , estimating a more accurate ϵ θ (˜ x c t i , t i ) , and designing a better order schedule. We hope our attempt can inspire future work to further explore the fast sampling of DPMs in very few steps.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2022ZD0160102, and in part by the National Natural Science Foundation of China under Grant 62321005, Grant 62336004, Grant 12271287 and Grant 62125603.

References

Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.00390 , 2021.

Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. ICML , 2022.

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICLR , 2022.

Mari Paz Calvo and César Palencia. A class of explicit multistep exponential integrators for semilinear problems. Numerische Mathematik , 102(3):367-381, 2006.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. ICLR , 2021.

Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.09788 , 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255. IEEE, 2009.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS , 34:8780-8794, 2021.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. ICLR , 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , pages 2672-2680, 2014.

David Francis Griffiths and Desmond J Higham. Numerical methods for ordinary differential equations: initial value problems , volume 5. Springer, 2010.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR , pages 10696-10706, 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurIPS , 2021.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Marlis Hochbruck and Alexander Ostermann. Explicit exponential runge-kutta methods for semilinear parabolic problems. SIAM Journal on Numerical Analysis , 43(3):1069-1090, 2005.

Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica , 19:209-286, 2010.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR , pages 4401-4410, 2019.

Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. NeurIPS , 34:21696-21707, 2021.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR , 2014.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

John Denholm Lambert et al. Numerical methods for ordinary differential systems , volume 146. Wiley New York, 1991.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755. Springer, 2014.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. ICLR , 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS , 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpmsolver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML , 2022.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , pages 8162-8171. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS , 2022.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR , 2022.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR , 2021.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR , 2021.

Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.02262 , 2022.

Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In ICLR , 2021.

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. ICLR , 2022.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902 , 2022.

Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05564 , 2022.

A UniPC for Data Prediction Model

A.1 Comparison of data prediction and noise prediction

The data prediction model is a simple linear transformation of the noise prediction model, namely x θ = ( x t -σ t ϵ θ ) /α t [19]. However, high-order solvers based on x θ and ϵ θ are essentially different [30, 26]. As we shall see, the formulae of UniPC for the data prediction model differ from those for the noise prediction model. On the other hand, for image data, x θ is bounded in [ -1 , 1] , while ϵ θ is generally unbounded and thus can push the sample out of the bound. Therefore, solvers for data prediction models are preferred, since thresholding method [30] can be directly applied and alleviate the 'train-test mismatch' problem.

A.2 Adapting UniPC to Data Prediction Model

As shown in the Proposition 4.1 of [26], for an initial value x s at time s > 0 , the solution at time t ∈ [0 , s ] of diffusion ODEs is

where we use the notation ˆ x θ and ˆ x λ to denote changing from the domain of time ( t ) to the domain of half log -SNR( λ ), i.e ., ˆ x λ := x t λ ( λ ) and ˆ x θ ( · , λ ) := x θ ( · , t λ ( λ )) . We are also able to adapt our UniPC to the data prediction model and utilize more information from previous data points. Recall that h i = λ t i -λ t i -1 . For any nonzero increasing sequence r 1 < r 2 < · · · < r p = 1 , λ s m = r m h i + λ t i -1 , s m = t λ ( λ s m ) , m = 1 , . . . , p . The UniPCp is given by

where D x m = x θ (˜ x s m , s m ) -x θ (˜ x t i -1 , t i -1 ) . Importantly, the corrector (UniC) can be also applied to any solver for the data prediction model that outputs ˜ x t i . Denote a p = ( a 1 , · · · , a p -1 ) ⊤ , c p = ( c 1 , · · · , c p ) ⊤ . Let

where ψ n ( h ) is defined by the recursive relation ψ n +1 ( h ) = 1 /n ! -ψ n ( h ) h , ψ 0 ( h ) = e -h , see Appendix E.4 for details. The order of accuracy of UniPCp for the data prediction model is given by the following proposition. The proof is in Appendix E.4.

̸

Proposition A.1 (Order of Accuracy of UniPCp for Data Prediction Model) . For an increasing sequence r 1 < r 2 < · · · < r p = 1 , under regularity assumption E.3, assuming 0 = B ( h ) = O ( h ) ,

Welist the algorithms for UniCp and UniPp for the data prediction model separately, see Algorithm 3 and Algorithm 4.

B Detailed Algorithms of Multistep UniPC

This section offers detailed algorithms with warming-up for multistep UniPC for noise prediction model (Algorithm 5,Algorithm 6) and for data prediction model (Algorithm 7,Algorithm 8).

C UniPC with varying coefficients (UniPC v )

UniPCp (4) uses a vector a p to match simultaneously all the coefficients of the derivatives. Alternatively, we can use { a i,p } p i =1 to match each order of derivatives separately and obtain a matrix of

Algorithm 3 UniCp for data prediction model

Require: { r i } p -1 i =1 , data prediction model x θ , any p -order solver Solver-p , a buffer Q = { x θ (˜ x t i -k , t i -k ) } p k =1 .

for m = 1 to p do

end for

Compute c p ← R -1 p ( h i ) g p ( h i ) /B ( h i ) , where R p and g p are as defined in Theorem 3.1 and (12)

return: ˜ x c t i

Algorithm 4 UniPp for the data prediction model

Require: { r i } p -1 i =1 , data prediction model x θ , a buffer Q = { x θ (˜ x t i -k , t i -k ) } p k =1 .

h i ← λ t i -λ t i -1

end for

Compute a p -1 ← R -1 p -1 ( h i ) g p -1 ( h i ) /B ( h i ) , where R p -1 and g p -1 are as defined in Theorem 3.1 and (12)

return: ˜ x t i

coefficients, i.e ., A p = ( a 1 ,p , · · · , a p,p ) ⊤ . Consider

where D p = ( D 1 /r 1 , · · · , D p /r p ) ⊤ and D m is defined in (5). The following theorem guarantees the order of accuracy of UniPC v . The proof is deferred to Appendix E. The convergence order is investigated in Appendix D. Define

Let I p be the p -dimensional identity matrix.

Theorem C.1. Under the conditions of Theorem 3.1, if

UniPC v is ( p +1) -th order accurate.

In fact, C p is invertible. Note that C p is the product of a diagonal matrix and a Vandermonde matrix, namely

Algorithm 5 Detailed implementation of multistep UniCp

Require: initial value x T , time steps { t i } =0 , noise prediction model ϵ θ , any p -order solver Solver-p are as defined in Theo-

Thus, we can simply take A p = C -1 p . The advantage of UniPC v is that A p solely depends on { r i } p i =1 . The algorithm of UniPC v is to replace the updating formulae of UniPC (Algorithm 1 and Algorithm 2) by (14) with A p = C -1 p . See Appendix G for its performance.

D Order of Convergence

In this section, we shall show that under mild conditions the convergence order of UniPp is p and the convergence order of UniPCp is p +1 for either the noise prediction model or data prediction model. The proof is deferred to Appendix E.

Definition D.1. For time steps { t i } i M =0 , we say the order of convergence of a sampler for DPMs is p if

We start by introducing some additional regularity assumptions.

Assumption D.2. The noise prediction model ϵ θ ( x , s ) is Lipschitz continuous w.r.t. x with Lipschitz constant L .

Assumption D.3. (1) h = max 1 ≤ i ≤ M h i = O (1 /M ) (2) For a constant b > 0 , max 1 ≤ i ≤ M Lα t i -1 /α t i < b , b -1 < α t i < b , for all 1 ≤ i ≤ M .

Assumption D.4. The starting values ˜ x t i , 1 ≤ i ≤ k -1 satisfies for some positive constant c 0 ,

Assumption D.2 is common in the analysis of ODEs similar to (20). By Assumption D.2, we have ϵ θ (˜ x s , s ) = ϵ θ ( x s , s ) + O (˜ x s -x s ) . Assumption D.3 assures that there is no significantly large step size and the signals are neither exploding nor degenerating. Assumption D.4 is common in the convergence analysis of multistep approaches [4].

The following Propositions D.5 and D.6 ensure the convergence order of UniPp and UniPCp . For general Solver-p such as DDIM ( p = 1 ), DPM-Solver/DPM-Solver++( p ≤ 3 ), UniCp can also increase the convergence order.

Algorithm 6 Detailed implementation of multistep UniPp

Proposition D.5. Under the conditions of Theorem 3.1, Assumptions D.2, D.3 , and D.4, the order of convergence of UniPp is p .

Proposition D.6. Under the conditions of Theorem 3.1, Assumptions D.2, D.3, and D.4 with k = p , the order of convergence of UniPCp is p +1 .

Remark D.7 . After a careful investigation of the proof of Proposition D.6 and Proposition D.5, we point out that for the singlestep updating when using UniPC1 for the estimation of ˜ x s m , t i -1 < s 1 , · · · , s m < t i , the order of convergence for UniPCp is p +1 and the order of convergence for UniPp is p for p ≤ 3 .

Following similar arguments of Proposition D.6, we find that UniCp can also increase the convergence order for general Solver-p for diffusion ODEs. The following is a direct corollary of Proposition D.6 which gives the order of convergence of UniPC v -p .

Corollary D.8. Under the conditions of Theorem C.1, Assumptions D.2, D.3, and D.4 with k = p , the order of convergence of UniPC v -p is p +1 .

The order of convergence for the data prediction model follows analogous to Proposition D.5 and Proposition D.6 under slightly different assumptions:

Assumption D.9. The noise prediction model x θ ( x , s ) is Lipschitz continuous w.r.t. x with Lipschitz constant L .

Assumption D.10. (1) h = max 1 ≤ i ≤ M h i = O (1 /M ) (2) For a constant b > 0 , max 1 ≤ i ≤ M Lσ t i -1 /σ t i < b , b -1 < σ t i < b , for all 1 ≤ i ≤ M .

We list the results of the order of convergence of UniP and UniPC for the data prediction model as corollaries to Propositions D.5 and D.6 and omit their proofs for simplicity. Interested readers can refer to the arguments in Appendix E for the noise prediction model and derive the detailed proofs for the data prediction model.

Corollary D.11. Under the conditions of Proposition A.1, Assumptions D.9, D.10, and D.4 with k = p , the order of convergence of UniPp for data prediction model is p .

Corollary D.12. Under the conditions of Proposition A.1, Assumptions D.9, D.10, and D.4 with k = p , the order of convergence of UniPCp for data prediction model is p +1 .

Algorithm 7 Detailed implementation of multistep UniCp for data prediction model

Require: initial value x T , time steps { t i } i M =0 , data prediction model x θ , any p-order solver Solver-p

end for

Compute c p i ← R -1 p i ( h i ) g p i ( h i ) /B ( h i ) , where R -1 p i ( h i ) and g p i are as defined in Theorem 3.1 and (12).

end for

return:

˜ x t M

E Proofs

In this section, we provide preliminaries of numerical analysis, regularity assumption, and the proof of Theorem 3.1 in the paper, the regularity assumption and the proof of Proposition A.1 in Appendix A as well as detailed proofs for Theorem C.1 in Appendix C, for Proposition D.5 and Proposition D.6 in Appendix D. Through this section, we use C and C i to denote sufficiently large positive constants independent of h .

E.1 Preliminaries

We begin with introducing preliminary results and concepts necessary for the proof of Theorem 3.1.

Expansion of exponential integrator. First, we obtain the Taylor expansion of (2), namely the exponentially weighted integral. Define the k -th order derivative of ˆ ϵ θ (ˆ x λ , λ ) as ˆ ϵ ( k ) θ (ˆ x λ , λ ) := d k ˆ ϵ θ (ˆ x λ , λ ) / d λ k . For 0 ≤ t < s ≤ T , r ∈ [0 , 1] , let h := λ t -λ s , λ := λ s + rh . Assuming the existence of total derivatives of ˆ ϵ ( k ) θ (ˆ x λ , λ ) , 0 ≤ k ≤ n , we have the n -th order Taylor expansion of ˆ ϵ θ (ˆ x λ , λ ) w.r.t. the half log -SNR λ :

Using the result of Taylor expansion of (17), the exponential integrator of (2) can be reduced to

Algorithm 8 Detailed implementation of multistep UniPp for data prediction model

Require: initial value x T , time steps { t i } i M =0 , data prediction model x θ Denote h i := λ t i -λ t i -1 , for i = 1 , · · · , M .

˜ x t 0 ← x T . Initialize an empty buffer Q .

continue

end if

end for

Compute a p i -1 ← R -1 p i -1 ( h i ) g p i -1 ( h i ) /B ( h i ) , where R -1 p i -1 ( h i ) and g p i -1 are as defined in Theorem 3.1 and (12).

end for

return: ˜ x

t M

where φ k +1 ( h ) = ∫ 1 0 e (1 -r ) h r k k ! d r can be computed via the recurrence relation φ k +1 ( z ) = ( φ k ( z ) -φ k (0)) /z , φ k (0) = 1 /k ! , and φ 0 ( z ) = e z [16]. For example, the closed-forms of φ k ( h ) for k = 1 , 2 , 3 are

Order of accuracy. In the following, we use the linear multistep method to illustrate the order of accuracy. Consider the ODE

We say that f satisfies Lipschitz condition, if there exists L > 0 such that

Suppose y ( x ) is the solution of Equation (19).

The k -order linear multistep method is given by

where y n + i is the approximation of y ( x n + i ) , x n + i = x n + ih , α i , β i are constants, | α 0 | + | β 0 | > 0 . Definition E.1. The local truncation error of (21) on x n + k is

If T n + k = O ( h p +1 ) , we say the order of accuracy of (21) is p .

The local truncation error describes the error caused by one iteration in the numerical analysis [22].

E.2 Regularity Assumption

Assumption E.2. The total derivatives d k ˆ ϵ θ (ˆ x λ ,λ ) d λ k , k = 1 , · · · , p exist and are continuous.

Assumption E.2 is required for the Taylor expansion which is also regular in high-order numerical methods.

E.3 Proof of Theorem 3.1 for arbitrary p ≥ 1

We first give the local truncation error of UniC. Given x r , r ≤ t i -1 , are correct, with a slight abuse of notation, define

Since Solver-p has order of accuracy p and B ( h ) = O ( h ) , under Assumption D.2, we have

Suppose x t is the solution of the diffusion ODE (1). Then, the local truncation error on t i is given by | x t i -¯ x t i | . Further, similar to Definition E.1 the order of accuracy of UniC is l , if there exists a sufficiently large positive constant C such that

In the following, we shall show that the order of accuracy of UniCp is p +1 . Combining (2) and (17), we have

Let R ⊤ p,k denote the k -th row of R p . By (7), we have | a ⊤ p R p,k ( h i ) -B -1 ( h i ) h k i k ! φ k +1 ( h i ) | ≤ C 0 B -1 ( h i ) h p +1 . Under Assumption E.2, by Taylor expansion, we obtain

Thus, we have

Combining (23), (24) and (25) given B ( h i ) = O ( h i ) , we have

Therefore, UniCp is of ( p +1) -th order of accuracy.

E.4 Proof of Proposition A.1

Regularity Assumption

Assumption E.3. The total derivatives d k ˆ x θ (ˆ x λ ,λ ) d λ k , k = 1 , · · · , p exist and are continuous.

Expansion of the Exponentially Weighted Integral. First, we obtain the Taylor expansion of (9). Define the k -th order derivative of ˆ x θ (ˆ x λ , λ ) as ˆ x ( k ) θ (ˆ x λ , λ ) := d k ˆ x θ (ˆ x λ , λ ) / d λ k . For 0 ≤ t < s ≤ T , r ∈ [0 , 1] , let h := λ t -λ s , λ := λ s + rh . Assuming the existence of total derivatives of ˆ x ( k ) θ (ˆ x λ , λ ) , 0 ≤ k ≤ n , the n -th order Taylor expansion of ˆ x θ (ˆ x λ , λ ) w.r.t. the half log -SNR λ is:

Then, the exponential integrator of (9) can be reduced to

where ψ k +1 ( h ) = ∫ 1 0 e ( r -1) h r k k ! d r can be computed via the recurrence relation by integration-byparts formula:

and ψ 0 ( z ) = e -z [16]. For example, the closed-forms of ψ k ( h ) for k = 1 , 2 , 3 are

Proof. In the subsequence analysis, we prove the order of accuracy of UniC for the data prediction model and the result of UniP for the data prediction model follows similarly. Suppose x t is the

□

solution of the diffusion ODE (1). For the data prediction model, given x r , r ≤ t i -1 , are correct, with a slight abuse of notation, define

Similar to (23), since Solver-p has order of accuracy p and B ( h ) = O ( h ) , under Assumption D.9, we have ¯ x t i = σ t i σ t i -1 x t i -1 + α t i (1 -e -h i ) x θ ( x t i -1 , t i -1 )+ α t i B ( h i ) ∑ p m =1 c m r m (ˆ x θ (ˆ x λ sm , λ s m ) -ˆ x θ (ˆ x λ t i -1 , λ t i -1 )) + O ( h p +2 ) . Then, the local truncation error on t i is given by | x t i -¯ x t i | . Further, similar to Definition E.1, we shall show that

Combining (9) and (28), we have

Recall that R ⊤ p,k is the k -th row of R p . By (13), we have | c ⊤ p R p,k ( h i ) -B -1 ( h i ) h k i k ! ψ k +1 ( h i ) | ≤ CB -1 ( h i ) h p +1 for some constant C > 0 . Under Assumption E.3, by Taylor expansion, we obtain

̸

Thus, we have ∣ ∣ ∣ ∑ p m =1 c m r m (ˆ x θ (ˆ x λ sm , λ s m ) -ˆ x θ (ˆ x λ t i -1 , λ t i -1 )) -B ( h i ) -1 ∑ p k =1 h k +1 i ψ k +1 ( h i )ˆ x ( k ) θ (ˆ x λ t i -1 , λ t i -1 ) ∣ ∣ ∣ ≤ C 1 ( B ( h i ) -1 h p +2 + h p +1 ) . Given 0 = B ( h i ) = O ( h i ) , we have

Therefore, UniCp for the data prediction model is of ( p +1) -th order of accuracy. □

E.5 Proof of Theorem C.1

We first give the local truncation error of UniPC v . Let A m,n denote the element of A p on row m and column n . Given x r , r ≤ t i -1 , are correct, with a slight abuse of notation, let

Similar to (23), we have

Suppose x t is the solution of the diffusion ODE (1) and the local truncation error on t i is given by | x t i -ˇ x t i | . Further, we shall show that the order of accuracy of UniPC v -p is p +1 , i.e .,

By (24), we have

Under Assumption E.2, by Taylor expansion, we obtain

Thus, we have

Therefore, UniPC v -p is of ( p +1) -th order of accuracy.

E.6 Proof of Proposition D.5

For UniPp , we define

By Assumption D.2, we have | ϵ θ (˜ x s , s ) -ϵ θ ( x s , s ) | ≤ L | ˜ x s -x s | . Therefore, for sufficiently large constants C, C 1 > 0 depending on { a m } and { r m } , we have

For simplicity, define e i = | ˜ x t i -x t i | , f n = max 0 ≤ i ≤ n | e i | . Using Theorem 3.1 and (32), we obtain

□

Let β i := α t i α t i -1 + Lσ t i ( e h i -1) + CLph . Then, it follows that

Since β i + CLph > 1 for sufficiently small h , the right hand side is also a trivial bound for f i -1 , because α t > 0 is monotone decreasing thus α t i < α t i -1 . We then have

Let σ := max 1 ≤ i ≤ M σ t i +2 Cp . By elementary calculation, under Assumption D.3 we have

where C 1 , C 2 are sufficiently large constants. Under Assumption D.4 with k = p , f p -1 = max 0 ≤ i ≤ p -1 | ˜ x t i -x t i | ≤ c 0 h p . Repeat the argument of (34), under Assumption D.3 by (35), there exists constant C 3 , C 4 > 0 such that

Therefore, the convergence order of UniPp is p .

E.7 Proof of Proposition D.6

For the sake of clarity, we use ¯ x c t i = α t i α t i -1 x t i -1 -σ t i ( e h i -1) ϵ θ ( x t i -1 , t i -1 ) -σ t i B ( h i ) ∑ p m =1 a m r m (ˆ ϵ θ (ˆ x λ sm , λ s m ) -ˆ ϵ θ (ˆ x λ t i -1 , λ t i -1 )) for UniCp and use ¯ x t i of (31) for UniPp . Similar to (32), we have for i ≥ p ,

where for the oracle UniPC (see Section 4.2 for definition), the UniPp admits

Define e c i = | ˜ x c t i -x t i | , f c n = max 0 ≤ i ≤ n | e c i | . Write β i := α t i α t i -1 + Lσ t i ( e h i -1) + CLph . Let σ := max 1 ≤ i ≤ M σ t i + 2 Cp . As shown in the proof of Theorem 3.1, | x t i -¯ x c t i | = O ( h p +2 ) . Combining the local truncation error, it follows that

Repeating this argument, similar to (36), we have

For 0 ≤ i < p , we have

□

Therefore, under Assumption D.4 with k = p , we have for 0 ≤ i ≤ p -1

Repeat this argument, we find for a constant C 4 > 0

For the non-oracle solver, for the step of UniPp we have :

Let e n = | ˜ x t n -x t n | , f i = max 1 ≤ i ≤ n e i . Under Assumption D.4 with k = p , f p -1 ≤ c 0 h p . For 0 ≤ i < p , we have

Similar to (40), we obtain f c p -1 = O ( h p +1 ) . Iterating (37) and (41), by Theorem 3.1 we have | ˜ x c t M -x 0 | = O ( h p +1 ) . □

F Implementation Details

We now provide more details about our UniPC and the experiments.

F.1 Implementation Details about UniPC

Our UniPC is implemented in a multistep manner by default, as is illustrated in Algorithm 5,6. In this case, the extra timesteps that are used to obtain the estimation of higher-order derivatives { s m } p -1 m =1 are set to be larger than t i -1 . In other words, { r m } p -1 m =1 are all negative. We have also found that the conditions of UniP-2 and UniC-1 degenerate to a simple equation where only a single a 1 is unknown. Specifically, considering (8) and (7), we find that

where

For B 1 ( h ) = h , it is easy to show that when a 1 = 0 . 5 ,

For B 2 ( h ) = e h -1 = h + O ( h 2 ) , the derivation is similar and a 1 = 1 / 2 also satisfies the condition. Therefore, we can directly set a 1 = 1 / 2 for UniP-2 and UniC-1 without solving the equation. For higher orders, the vector a p is computed normally through the inverse of the R p matrix.

To provide enough data points for high-order UniPC, we need a warming-up procedure in the first few steps, as is also used in previous multistep approaches [26] and is shown in Algorithm 5,6,. Since our UniC needs to compute ϵ θ (˜ x t i , t i ) , i.e ., the model output at the current timestep t i to obtain the corrected result x c t i , performing our UniC at the last sampling step will introduce an extra function evaluation. Therefore, we do not use the corrector after the last execution of the predictor for fair comparisons.

Combining (39) and (40), we have

F.2 Details about the experiments.

We now provide more details of our experiments. For unconditional sampling on CIFAR10 [21], we use the ScoreSDE [35] codebase and their pre-trained model, which is a continuous-time DDPM++ model [35]. More concretely, we use the cifar10_ddpmpp_deep_continuous config file, the same as the example provided by the official code of DPM-Solver [25]. To compute FID, we adopt the statistic file provided by ScoreSDE [35] codebase. For unconditional sampling on LSUN Bedroom [39] and FFHQ [18], we adopt the latent-space DPM provided by the stable-diffusion codebase [29]. Since there is no statistic file for these two datasets in the codebase, we compute the dataset statistic of FFHQ using the script in the library pytorch-fid, and borrow the statistic file of LSUN Bedroom from the guided-diffusion codebase [8]. For conditional sampling on pixel space, we implement our method in guided-diffusion codebase [8] and use the pre-trained checkpoint for ImageNet 256 × 256. For conditional sampling on latent space, we adopt the stable-diffusion codebase and use their sd-v1-3.ckpt checkpoint, which is pre-trained on LAION [32]. To obtain the text prompts, we randomly sample 10K captions from the MS-COCO2014 validation dataset [23]. We sample 10K random latent code x ∗ T for each caption and fix them when using different methods.

G More Results

In this section, we will provide more detailed results, including both quantitative and qualitative results.

G.1 More Quantitative Results

Unconditional Sampling. We start by demonstrating detailed results on CIFAR10 [21], which are shown in Table 8. The results of our proposed method are highlighted in gray. Apart from the results already illustrated in Figure 2, we also include the performance of the DPM-Solver [25]

and our UniPC v which has varying coefficients (see Appendix C for detailed description). Firstly, we show that DPM-Solver performs very unstable with extremely few steps: the FID of 5 NFE comes to 290.65, which means the solver crashes in this case. Besides, we also show that the DPMSolver++(3M) [26] performs consistently better than DPM-Solver-3 [25], indicating the multistep method is more effective in few-step sampling setting. That is also why we tend to use DPM-Solver++ as our baseline method in all of our experiments. Table 8 also shows the comparisons between variants of our UniPC, such as UniPC with different instantiations of B ( h ) and UniPC v . Since the comparisons between B 1 ( h ) and B 2 ( h ) have already been discussed in Section 4.2, we now focus on the analysis of the performance of UniPC v . We find UniPC v achieves the best sampling quality with 7 9 NFE, while cannot beat other variants of UniPC with 5,6,10 NFE. These results show that different variants of our UniPC may have different applications and we should select the most suitable one according to the budget of NFE. We also include more experimental results of the unconditional sampling results on FFHQ [18] in LSUN Bedroom [39], as summarized in Table 9 and Table 10. Note that there are fewer results for LSUN because we performed most of our experiments on FFHQ and CIFAR10 in the early stage. Nevertheless, the overall conclusions of the results on different datasets are aligned. To sum up, our results show (1) multistep methods behave better than singlestep ones when the NFE budget is extremely small. (2) our UniC can consistently improve the sampling quality of a wide range of off-the-shelf solvers. (3) selecting a proper variant of UniPC can yield better results in most cases.

Conditional Sampling. We also provide more results on guided sampling on ImageNet 256 × 256, as is shown in Table 11. We evaluate the performance of DEIS [40] and find it performs worse than both the DPM-Solver++ and our UniPC. Besides, we have compared the choice of B ( h ) on guided

DDIM [34]

DPM-Solver++ [26]

S/AbC

mname

sampling, where we find B 2 ( h ) significantly outperforms B 1 ( h ) , perhaps because B 1 ( h ) = h is too simple and not suitable for the guided sampling. We have also added the results when the guidance scale s = 1 . 0 . The results show that our method can achieve better sampling quality with both large and small guidance scales with few sampling steps.

G.2 More Qualitative Results

We provide more visualizations to demonstrate the qualitative performance. First, we consider the class-conditioned guided sampling, i.e ., the conditional sampling on ImageNet [7] in Figure 5. Specifically, we compare the sampling quality of each method with only 7 NFE. Note that we randomly sample the class from the total 1000 classes in ImageNet for each image sample, but fix the initial noise for different methods.

Peer reviews:
comments: SUMMARY:
This paper propose a predictor-corrector method to accelerate the diffusion sampling process, where a corrector is proposed to correct the initial estimation of x_t using previous and current points. The experiments are conducted on imagenet and cifar10 both of which outperform existing efficient sampler at very few sampling steps.

STRENGTHS:
1. Improving existing sampling method by a unified predictor-corrector solver is reasonable. 
2. Adequate theoretical and empirical analysis of the proposed methods. 
3. The paper is well organized and easy to understand.

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
The authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper develops a unified corrector (UniC) that can be applied after any existing DPM sampler to increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC,  a unified predictor-corrector framework called UniPC for the fast sampling of DPMs  has a unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps.

STRENGTHS:
On both unconditional and conditional sampling using pixel-space and latent-space DPMs. UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet 256 256 (conditional) with only 10 function evaluations.

WEAKNESSES:
They claim many times on unified or model-agnostic but no experiments support their claim, the method only tested on DPM. Also the writing make me lost  many times when reading the paper, I suggest you rewrite your motivation part by which technical issue in DPM you try to solve, why your method works.  The theory on faster convergence property is also confused on accuracy, is it really useful?  how to  theoretically guarantee on the accuracy.

QUESTIONS:
They claim many times on unified or model-agnostic but no experiments support their claim, the method only tested on DPM. Also the writing make me lost  many times when reading the paper, I suggest you rewrite your motivation part by which technical issue in DPM you try to solve, why your method works.  The theory on faster convergence property is also confused on accuracy, is it really useful?  how to  theoretically guarantee on the accuracy.

LIMITATIONS:
See Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a universal predictor-corrector method for faster sampling of diffusion models, with model retraining. The key idea is to further include the current point along with previous $p$ points while estimating the data point by adding a correction step. It shows that this model can achieve an order of accuracy $p+1$. Experiments on several image datasets show that this proposed method achieves strong performance in comparison to prior SOTA methods.\n\nSTRENGTHS:\nThe paper is generally written clearly \n\nThe problem of accelerating diffusion model is critical to solve\n\nReasonably good performance achieved\n\nWEAKNESSES:\n**Limited novelty and design**:\n- The idea proposed here is marginal in the sense that at each step, the current estimate is additionally used along with some previous points  (the latte is the same as [25,34,40]); Given the sequential nature, the current point is actually already used by previous methods [25,34,40] in the next steps (with a step delay), whilst it is just that this point is used for more times. No extra information is used whilst also no clear understand on why this extra use of the current point can lead to increase in the order of accuracy. \n\n- Besides, in general, there are two families of diffusion models: DDPM (e.g., [A, 34]) and SGM (e.g., [35]). And the DDPM series are often easier to speed up due to adopting a variance preserving (VP) process, in contrast to SGM’s variance exploding (VE) process. It is not clear which family this proposed method is focused on, or both?\n\n- It looks the step of 10 is a milestone. Is there any relationship or theoretical implication of this proposed method on tackling this issue?\n\n- Also, the addition operations introduce some complex parameter, which adds further burden to the model tuning process. It is unclear if these parameters are general for different datasets. \n\n**Limited results gain**\n- Whilst the numeral results such as FID looks good in comparison, when it comes to the visual examples in supplementary, the generated images are of similar quality as previous methods such as DPM-Solver++. It is known that numerical metrics are some limited in the visual perception evaluation. Overall, the generation performance gain is some limited, not as convincing as what claimed.\n\n- It is some inconsistent when evaluating UniC/UniP/UniPC. How is this selection done?\n\n- If I am not missing, there is no exact evaluation on the benefit of using the current point in estimation, which is the key design of the whole  model. \n\n**References**:\n- [A] Jonathan et al. Denoising Diffusion Probabilistic Models. NeurIPS 2020\n\nQUESTIONS:\nPlease see the weaknesses above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
In this paper, the authors present a novel sampling solver called UniPC for diffusion models. UniPC consists of two parts: UniC and UniP, where UniC corrects the estimation with prediction of current timestep and it can be applied other sampling solvers and UniP is a special case of UniC and share the similar form for sampling. Compared with previous methods(e.g. DPMSolver), empirical evaluation have verified the effectiveness of UniPC in generating better results with less sampling steps.

STRENGTHS:
-	The proposed method can significantly reduce the number of sampling steps to less than 10, without losing the quality of the results.
-	The proposed method has some generality and can further improve the approximate accuracy of other existing DPM samplers.

WEAKNESSES:
- Experiments. Please conduct experiments on datasets with larger resolution images to measure inference time, quality, and diversity.
- Recent work: Please add more details on the related wok about DPM solver and some related ODE-based sampling solver.

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an ODE sampler for diffusion probabilistic models\n(DPM), exploiting the structure of exponential integrators.  The paper\nclaims the proposed sampler can use any existing DPM and achieve high\nsampling quality with very few (<10) number of function evaluations\n(NFE), and also improve upon related methods when using arbitrary\nNFEs.\n\nSTRENGTHS:\n- The proposed method is well justified theoretically and the\n  experiments validate the proposed approach.\n- The paper clearly states the connections between the proposed method\n  and related existing solvers.\n- The experimental results include  both pixel-space and\n  latent-space diffusion and show consistent improvements when applying\n  the method on top of existing solvers.\n- The method performance is superior to the baselines in both low and\n  middle NFE regimes.\n\nWEAKNESSES:\n1) It would be nice if the paper included the statistical significance\n   of the FID scores reported in Tables 1-6. In the ~10 NFE regime\n   sometimes these are very close to the baseline, while for lower\n   quality low NFE one could expect the FID to be not very\n   informative. I plan to revisit my decision after considering the\n   statitstical significance of these results.\n2) The paper repeatedly claims being able to achive higher order p\n   than existing solvers. However I wonder how important it is in\n   practice to achieve order p>3. Table 4 shows when including higher\n   orders the results become poorer.  And also I'd be curious to see\n   this study for the higher quality NFE regimes (eg NFE=10).\n3) Minor:\n   - L. 39 \"output of the model output\"\n   - L. 68-69 extra subscript 0 on lhs?\n   - L. 71 \"obtained\"\n   - L. 97 (after period) Can this be rephrased in a less subjective manner?\n   - (5) I think $\\hat{\\epsilon}_\\theta^{(k)}$ is not defined. Shouldn't\n     the argument in this tem vary with $k$?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}