SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenliang Zhao* Lujia Bai* Yongming Rao Jie Zhou Jiwen Lu†
Diffusion probabilistic models (DPMs) have demonstrated a very promising ability in high-resolution image synthesis. However, sampling from a pre-trained DPM is time-consuming due to the multiple evaluations of the denoising network, making it more and more important to accelerate the sampling of DPMs. Despite recent progress in designing fast samplers, existing methods still cannot generate satisfying images in many applications where fewer steps  $(e.g., < 10)$  are favored. In this paper, we develop a unified corrector (UniC) that can be applied after any existing DPM sampler to increase the order of accuracy without extra model evaluations, and derive a unified predictor (UniP) that supports arbitrary order as a byproduct. Combining UniP and UniC, we propose a unified predictor-corrector framework called UniPC for the fast sampling of DPMs, which has a unified analytical form for any order and can significantly improve the sampling quality over previous methods, especially in extremely few steps. We evaluate our methods through extensive experiments including both unconditional and conditional sampling using pixel-space and latent-space DPMs. Our UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet  $256 \times 256$  (conditional) with only 10 function evaluations. Code is available at https://github.com/wl-zhao/UniPC.
Diffusion probabilistic models (DPMs) [33, 13, 35] have become the new prevailing generative models and have achieved competitive performance on many tasks including image synthesis [8, 29, 13], video synthesis [15], text-to-image generation [27, 29, 12], voice synthesis [5], etc. Different from GANs [10] and VAEs [20], DPMs are trained to explicitly match the gradient of the data density (i.e., score), which is more stable and less sensitive to hyper-parameters. However, sampling from a pre-trained DPM usually requires multiple model evaluations to gradually perform denoising from Gaussian noise [13], consuming more inference time and computational costs compared with single-step generative models like GANs.
Recently, there have been increasing efforts to accelerate the sampling of DPMs [31, 28, 34, 25, 40]. Among those, training-free methods [34, 25, 40] enjoy a wider usage in applications because they can be directly applied to off-the-shelf pre-trained DPMs. Although these methods have significantly reduced the sampling steps from 1000 to less than 20 steps, the sampling quality with extremely few steps  $(e.g., < 10)$  has been rarely investigated. Few-step sampling can be used in many scenarios where we need to efficiently obtain plausible samples, such as designing a proper prompt for a text-to-image diffusion model [29, 30] and computing losses on the generated samples during the training of some diffusion-based visual systems [1, 6]. However, current fast samplers usually struggle to generate high-quality samples within 10 steps (see Figure 1).
DDIM (order  $= 1$  [34]
Figure 1: Qualitative comparisons between our UniPC and previous methods. All images are generated by sampling from a DPM trained on ImageNet  $256 \times 256$  with only 7 number of function evaluations (NFE) and a classifier scale of 8.0. We show that our proposed UniPC can generate more plausible samples with more visual details compared with previous first-order sampler [34] and high-order samplers [40, 26]. Best viewed in color.
DEIS (order  $= 2$  [40]
DPM-Solver++ (order  $= 2$  [26]
UniPC (order  $= 3$  ) (Ours)
In this paper, we propose a training-free framework for the fast sampling of DPMs called UniPC. We find that UniPC significantly outperforms existing methods within  $5\sim 10$  NFE (number of function evaluations), and can also achieve better sampling quality with more sampling steps. Specifically, we first develop a unified corrector (UniC) which works by using the model output  $\epsilon_{\theta}(\pmb{x}_{t_i}, t_i)$  at the current timestep  $t_i$  to obtain a refined  $\pmb{x}_{t_i}^c$ . Different from the predictor-corrector paradigm in numerical ODE solver that requires a doubled NFE, our UniC reuses the model output  $\epsilon_{\theta}(\pmb{x}_{t_i}, t_i)$  to the next sampling step, thus introducing no extra function evaluation. UniC can be applied after any existing DPM sampler to increase the order of accuracy, while the inference speed is almost unaffected. Interestingly, we also find that by simply changing a hyper-parameter in UniC, a new family of predictors (UniP) can be further obtained.
Since our UniC is method-agnostic, we combine UniP and UniC to obtain a new family of fast samplers called UniPC. Different from previous fast solvers [26, 25, 40] that either have no higher-order  $(e.g., > 3)$  variants or have no explicit forms, our UniPC supports arbitrary orders with a unified analytical expression and are easy to implement. Benefiting from the universal design, variants of UniPC (e.g., singlestep/multistep, noise/data prediction) can be easily derived. We theoretically prove that UniPC enjoys higher convergence order and empirically demonstrate that UniPC has better sampling quality in a variety of scenarios. We also show that the inference speed and memory usage of UniPC is similar to DPM-Solver++ [26], indicating that UniPC can achieve superior performance under the same computational budgets.
We conduct extensive experiments with both pixel-space and latent-space DPMs to verify the effectiveness of the proposed UniPC. Our results show that UniPC performs consistently better than previous state-of-the-art methods on both unconditional and conditional sampling tasks. Notably, UniPC can achieve 3.87 FID on CIFAR10 (unconditional) and 7.51 FID on ImageNet  $256 \times 256$  (conditional) with only 10 function evaluations. We also demonstrate that UniC can improve the sampling quality of several existing fast samplers significantly with very few NFE (number of function evaluations). Some qualitative comparisons are shown in Figure 1, where we observe that our UniPC can generate images with more visual details than other methods.
For a random variable  $\pmb{x}_0$  with an unknown distribution  $q_0(\pmb{x}_0)$ , Diffusion Probabilistic Models (DPMs) [33, 13, 19] transit  $q_0(\pmb{x}_0)$  at time 0 to a normal distribution  $q_{T}(\pmb{x}_{T})\approx \mathcal{N}(\pmb{x}_{T}|\pmb {0},\tilde{\sigma}^{2}\pmb {I})$  at time  $T$  for some  $\tilde{\sigma} >0$  by gradually adding Gaussian noise to the observation  $\pmb{x}_0$ . For each time  $t\in [0,T]$ , and given  $\sigma_t,\alpha_t > 0$ , the Gaussian transition is
$$
q _ {t | 0} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {0}) = \mathcal {N} (\boldsymbol {x} _ {t} | \alpha_ {t} \boldsymbol {x} _ {0}, \sigma_ {t} ^ {2} \boldsymbol {I}),
$$
where  $\alpha_{t}^{2} / \sigma_{t}^{2}$  (the signal-to-noise-ratio (SNR)) is strictly decreasing w.r.t.  $t$  [19].
Let  $\epsilon_{\theta}(\boldsymbol{x}_t,t)$  denote the noise prediction model using data  $\boldsymbol{x}_t$  to predict the noise  $\epsilon$ , and the parameter  $\theta$  is obtained by minimizing
$$
\mathbb {E} _ {\boldsymbol {x} _ {0}, \epsilon , t} [ \omega (t) \| \epsilon_ {\theta} (\boldsymbol {x} _ {t}, t) - \epsilon \| _ {2} ^ {2} ],
$$
where  $\pmb{x}_0 \sim q_0(\pmb{x}_0)$ ,  $t \in \mathcal{U}[0, T]$ , and the weight function  $\omega(t) > 0$ . Sampling from DPMs can be achieved by solving the following diffusion ODEs [35]:
$$
\frac {\mathrm {d} \boldsymbol {x} _ {t}}{\mathrm {d} t} = f (t) \boldsymbol {x} _ {t} + \frac {g ^ {2} (t)}{2 \sigma_ {t}} \boldsymbol {\epsilon} _ {\theta} (\boldsymbol {x} _ {t}, t), t \in [ 0, T ], \quad \boldsymbol {x} _ {T} \sim \mathcal {N} (\boldsymbol {0}, \tilde {\sigma} ^ {2} \boldsymbol {I}) \tag {1}
$$
where  $f(t) = \frac{\mathrm{d}\log\alpha_t}{\mathrm{d}t}, g^2(t) = \frac{\mathrm{d}\sigma_t^2}{\mathrm{d}t} - 2\frac{\mathrm{d}\log\alpha_t}{\mathrm{d}t}\sigma_t^2.$
2.2 Fast Sampling of DPMs
Fast samplers of DPMs can be either training-based [31, 2, 37] or training-free [25, 26, 40, 24, 41]. Training-based samplers require further training costs while training-free methods directly use the original information without re-training and are easy to implement in conditional sampling. The essence of training-free samplers is solving stochastic differential equations (SDEs)[13, 35, 3, 41] or ODEs[26, 40, 24, 34, 25]. Other fast sampling methods include modifying DPMs [9] and the combination with GANs [38, 36].
Among others, samplers solving diffusion ODEs are found to converge faster for the purpose of sampling DPMs [34, 35]. Recent works [40, 25, 26] show that ODE solvers built on exponential integrators [17] appear to have faster convergence than directly solving the diffusion ODE (1). The solution  $\boldsymbol{x}_t$  of the diffusion ODE given the initial value  $\boldsymbol{x}_s$  can be analytically computed as [25]:
$$
\boldsymbol {x} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \boldsymbol {x} _ {s} - \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \hat {\boldsymbol {\epsilon}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda}, \lambda) d \lambda , \tag {2}
$$
where we use the notation  $\hat{\epsilon}_{\theta}$  and  $\hat{x}_{\lambda}$  to denote changing from the domain of time(t) to the domain of half log-SNR(λ), i.e.,  $\lambda_{t} = \log (\alpha_{t} / \sigma_{t})$ $\hat{x}_{\lambda}\coloneqq x_{t_{\lambda}(\lambda)}$  and  $\hat{\epsilon}_{\theta}(\cdot ,\lambda)\coloneqq \epsilon_{\theta}(\cdot ,t_{\lambda}(\lambda))$
Based on the exponential integrator, [25] proposes to approximate  $\hat{\epsilon}_{\theta}$  via taylor expansion and views DDIM as DPM-Solver-1, i.e.,
$$
\tilde {\boldsymbol {x}} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \tilde {\boldsymbol {x}} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \left(e ^ {\lambda_ {t _ {i}} - \lambda_ {t _ {i - 1}}} - 1\right) \boldsymbol {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}\right). \tag {3}
$$
[26] considers rewriting (2) using  $\hat{x}_{\theta}$  instead of  $\hat{\epsilon}_{\theta}$ ; [40] derives the taylor expansion formulae with respect to  $t$  instead of the half log-SNR( $\lambda$ ). [24] employs pseudo numerical methods such as Runge-Kutta method directly for the updating of  $\epsilon_{\theta}$  of (3). Although many aforementioned high-order solvers are proposed, existing solvers of diffusion ODEs can be explicitly computed for orders not greater than 3, due to the lack of analytical forms.
3 A Unified Predictor-Corrector Solver
In this section, we propose a unified predictor-corrector solver of DPMs called UniPC, consisting of UniP and UniC. Our UniPC is unified in mainly two aspects: 1) the predictor (UniP) and the corrector (UniC) share the same analytical form; 2) UniP supports arbitrary order and UniC can be applied after off-the-shelf fast samplers of DPMs to increase the order of accuracy.
3.1 The Unified Corrector UniC- $p$
Modern fast samplers based on discretizing diffusion ODEs [25, 34, 40] aim to leverage the previous  $p$  points  $\{\tilde{\boldsymbol{x}}_{t_{i - k}}\}_{k = 1}^{p}$  to estimate  $\tilde{\boldsymbol{x}}_{t_i}$  with  $p$  order of accuracy. Despite the rapid development of fast samplers, the quality of few-step sampling still has room for improvement. In this paper, we propose a corrector called UniC-  $p$  to improve the initial estimation using not only the previous  $p$  points but also the current point. Formally, after obtaining the initial estimation  $\tilde{\boldsymbol{x}}_{t_i}$ , we perform the correction step through the following formula:
$$
\tilde {\boldsymbol {x}} _ {t _ {i}} ^ {c} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \tilde {\boldsymbol {x}} _ {t _ {i - 1}} ^ {c} - \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) \boldsymbol {\epsilon} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}\right) - \sigma_ {t _ {i}} B \left(h _ {i}\right) \sum_ {m = 1} ^ {p} \frac {a _ {m}}{r _ {m}} D _ {m}, \tag {4}
$$
Algorithm 1 UniC-p
Require:  $\{r_i\}_{i = 1}^{p - 1}$ $\epsilon_{\theta}$  network, any  $p$  -order solver Solver-p, a buffer  $Q = \{\epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i - k},t_{i - k})\}_{k = 1}^p$ $h_i\gets \lambda_{t_i} - \lambda_{t_{i - 1}},\tilde{\boldsymbol{x}}_{t_i}\gets \mathsf{S o l v e r - p}(\tilde{\boldsymbol{x}}_{t_{i - 1}}^c,Q)$
for  $m = 1$  to  $p$  do
$$
s _ {m} \leftarrow t _ {\lambda} \left(r _ {m} h + \lambda_ {t _ {i - 1}}\right)
$$
$$
D _ {m} \leftarrow \boldsymbol {\epsilon} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {s _ {m}}, s _ {m}\right) - \boldsymbol {\epsilon} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}\right)
$$
Compute  $\pmb{a}_p \gets \pmb{R}_p^{-1}(h_i) \phi_p(h_i) / B(h_i)$ , where  $\pmb{R}_p, \phi_p$  are as defined in Theorem 3.1
$$
\begin{array}{l} \tilde {\boldsymbol {x}} _ {t _ {i}} ^ {c} \gets \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \tilde {\boldsymbol {x}} _ {t _ {i - 1}} ^ {c} - \sigma_ {t _ {i}} (e ^ {h _ {i}} - 1) \epsilon_ {\theta} (\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}) \\ - \sigma_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p} a _ {m} D _ {m} / r _ {m} \\ \end{array}
$$
$$
Q \stackrel {\text {b u f f e r}} {\longleftarrow} \epsilon_ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i}}, t _ {i}\right)
$$
return:  $\tilde{x}_{t_i}^c$
Algorithm 2 UniP-  $p$
Require:  $\{r_i\}_{i = 1}^{p - 1}$ $\epsilon_{\theta}$  network, a buffer  $Q =$ $\{\epsilon_{\theta}(\tilde{x}_{t_{i - k}},t_{i - k})\}_{k = 1}^{p}$
$$
h _ {i} \leftarrow \lambda_ {t _ {i}} - \lambda_ {t _ {i - 1}}
$$
for  $m = 1$  to  $p - 1$  do
$$
\begin{array}{l} s _ {m} \leftarrow t _ {\lambda} \left(r _ {m} h + \lambda_ {t _ {i - 1}}\right) \\ D _ {m} \gets \epsilon_ {\theta} (\tilde {\boldsymbol {x}} _ {s _ {m}}, s _ {m}) - \epsilon_ {\theta} (\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}) \\ \end{array}
$$
Compute  $\pmb{a}_{p - 1} \gets \pmb{R}_{p - 1}^{-1}(h_i) \phi_{p - 1}(h_i) / B(h_i)$ , where  $\pmb{R}_{p - 1}, \phi_{p - 1}$  are as defined in Theorem 3.1
$$
\begin{array}{l} \tilde {\boldsymbol {x}} _ {t _ {i}} \gets \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \tilde {\boldsymbol {x}} _ {t _ {i - 1}} - \sigma_ {t _ {i}} (e ^ {h _ {i}} - 1) \boldsymbol {\epsilon} _ {\theta} (\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}) \\ - \sigma_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p - 1} a _ {m} D _ {m} / r _ {m} \\ \end{array}
$$
$$
Q \stackrel {\text {b u f f e r}} {\longleftarrow} \boldsymbol {\epsilon} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i}}, t _ {i}\right)
$$
return:  $\tilde{x}_{t_i}$
where  $\tilde{\pmb{x}}_{t_i}^c$  denotes the corrected result,  $B(h) = \mathcal{O}(h)$  is a non-zero function of  $h$ ,  $h_i = \lambda_{t_i} - \lambda_{t_{i-1}}$  is the step size in the half-log-SNR( $\lambda$ ) domain,  $r_1 < r_2 < \dots < r_p = 1$  are a non-zero increasing sequence, determining which previous points are used. Specifically, we use  $\{r_i\}_{m=1}^p$  to interpolate between  $\lambda_{t_{i-1}}$  to  $\lambda_{t_i}$  to obtain the auxiliary timesteps  $s_m = t_\lambda(r_m h + \lambda_{t_{i-1}})$ ,  $m = 1, 2, \dots, p$ . The model outputs at these timesteps are used to compute  $D_m$  by
$$
D _ {m} = \epsilon_ {\theta} \left(\tilde {\boldsymbol {x}} _ {s _ {m}}, s _ {m}\right) - \epsilon_ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}\right). \tag {5}
$$
We now describe how to choose  $\{a_m\}_{m=1}^p$  in UniC- $p$  to effectively increase the order of accuracy. The main idea is to cancel out low-order terms between the numerical estimation (4) and the theoretical solution (2). In practice, we expand the exponential integrator in (2) as follows:
$$
\begin{array}{l} \boldsymbol {x} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) \epsilon_ {\theta} \left(\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}\right) \\ - \sigma_ {t _ {i}} \sum_ {k = 1} ^ {p} h _ {i} ^ {k + 1} \varphi_ {k + 1} \left(h _ {i}\right) \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) + \mathcal {O} \left(h ^ {p + 2}\right). \tag {6} \\ \end{array}
$$
where  $\hat{\epsilon}_{\theta}^{(k)}$  denotes the  $k$ -th derivative of  $\hat{\epsilon}_{\theta}$ , and  $\varphi_{k}(h)$  can be analytically computed [16]. The  $\{a_m\}_{m=1}^p$  can be then determined by matching the coefficients between (4) and (6). In the following theorem, we show that  $\mathrm{UniC}-p$  has an order of accuracy  $p+1$  (see Appendix E.3 for detailed proof).
Theorem 3.1 (The Order of Accuracy of UniC-p). For any non-zero sequence  $\{r_i\}_{i=1}^p$  and  $h > 0$ , define
$$
\boldsymbol {R} _ {p} (h) = \left( \begin{array}{c c c c} 1 & 1 & \dots & 1 \\ r _ {1} h & r _ {2} h & \dots & r _ {p} h \\ \dots & \dots & \dots & \dots \\ (r _ {1} h) ^ {p - 1} & (r _ {2} h) ^ {p - 1} & \dots & (r _ {p} h) ^ {p - 1} \end{array} \right).
$$
Let  $\phi_p(h) = (\phi_1(h),\dots ,\phi_p(h))^{\top}$  with  $\phi_n(h) = h^n n!\varphi_{n + 1}(h)$ , where  $\varphi_{n}(h)$  is defined by the recursive relation [16]:
$$
\varphi_ {n + 1} (h) = \frac {\varphi_ {n} (h) - 1 / n !}{h}, \quad \varphi_ {0} (h) = e ^ {h}.
$$
For an increasing sequence  $r_1 < r_2 < \dots < r_p = 1$ , suppose  $\mathbf{a}_p \coloneqq (a_1, \ldots, a_p)^\top$  satisfies,
$$
\left| \boldsymbol {R} _ {p} \left(h _ {i}\right) \boldsymbol {a} _ {p} B \left(h _ {i}\right) - \phi_ {p} \left(h _ {i}\right) \right| = \mathcal {O} \left(h _ {i} ^ {p + 1}\right), \tag {7}
$$
where  $|\cdot|$  denotes the  $l_{1}$  norm for matrix. Then, under regularity conditions in Appendix E.2, UniC- $p$  of (4) will have  $(p + 1)$ -th order of accuracy.
The monotonicity of  $\{r_i\}_{i = 1}^p$  ensures the invertibility of the Vandermonde matrix  $R_{p}$ . Therefore, we can take  $\pmb{a}_p = R_p^{-1}(h_i)\phi_p(h_i) / B(h_i)$  as the coefficient vector for (4) for simplicity, where  $B(h)$
can be any function of  $h$  such that  $B(h) = \mathcal{O}(h)$ , for example  $B_{1}(h) = h$ ,  $B_{2}(h) = e^{h} - 1$ . The detailed implementation of UniC is shown in Algorithm 1. Importantly, we circumvent the extra evaluation of  $\epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i}^c, t_i)$  by pushing  $\epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i}, t_i)$  into the buffer  $Q$  instead of  $\epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i}^c, t_i)$ . Taking full advantage of  $\epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i}, t_i)$  of previous results enables us to increase the order of accuracy without incurring significant increment of computation cost. This makes our method inherently different from the predictor-corrector methods in ODE literature [22], where the computational costs are doubled because an extra function evaluation on the corrected  $\tilde{\boldsymbol{x}}_{t_i}^c$  is required for each step.
3.2 The Unified Predictor UniP-  $p$
We find that the order of accuracy of UniC does not depend on the specific choice of the sequence  $\{r_i\}_{i = 1}^p$ , which motivates us to design  $p$ -order unified predictor (UniP- $p$ ) which only leverages the previous  $p$  data points by excluding  $D_{p}$  in (4) since  $D_{p}$  involves  $\tilde{\boldsymbol{x}}_{t_i}$ . The order of accuracy is guaranteed by the following corollary.
Corollary 3.2 (The Order of Accuracy of UniP-p). For an increasing sequence  $r_1 < r_2 < \dots < r_{p-1} < 1$ , the solver given in (4) dropping the term  $D_p$  and using coefficients that satisfies
$$
\left| \boldsymbol {R} _ {p - 1} \left(h _ {i}\right) \boldsymbol {a} _ {p - 1} B \left(h _ {i}\right) - \phi_ {p - 1} \left(h _ {i}\right) \right| = \mathcal {O} \left(h _ {i} ^ {p}\right) \tag {8}
$$
has  $p$ -th order of accuracy.
Due to the unified form of UniP and UniC, we can use UniP-  $p$  as the implementation of the Solver-p in UniC-  $p$  to obtain a new family of solvers called UniPC-  $p$ . Theorem 3.1 and Corollary 3.2 ensure that UniPC-  $p$  can achieve  $(p + 1)$ -th order of accuracy. Moreover, under additional regularity conditions in Appendix D, based on Theorem 3.1 and Corollary 3.2, we show that the order of convergence of UniPC-  $p$  reaches  $p + 1$  (see Appendix D).
3.3 Comparison with Existing Methods
Here we discuss the connection and the difference between UniPC and previous methods. When  $p = 1$ , UniPC will reduce to DDIM [34]. Motivated by linear multistep approaches, PNDM [24] proposes to use pseudo numerical methods for DDIM, while our UniPC makes use of information in the ODE solution (2) and is specially designed for diffusion ODEs. DEIS [40] is built on exponential integrators in the time domain, where the integral cannot be analytically computed and explicit formulae for high-order solvers cannot be derived. By using the half log-SNR  $\lambda$  [25, 26], it is shown that the application of integration-by-parts can simplify the integration of (2) and leads to explicit expansion of  $x_{t}$ . DPM-Solver-2 [25] lies in our UniPC framework as UniP-1, where they assume  $B(h) = e^{h} - 1$ . We find through our numerical analysis that  $B(h)$  can be any non-degenerate function such that  $B(h) = \mathcal{O}(h)$ . Furthermore, DPM-Solvers do not admit unified forms even for orders smaller than 3, which adds to the challenge of obtaining algorithms for higher orders. In contrast, our UniPC exploits the structure of exponential integrators w.r.t. half log-SNR and admits not only simple and analytical solutions for efficient computation but also unified formulations for easy implementation of any order.
3.4 Implementation
By setting  $r_m = (\lambda_{t_{i - m - 1}} - \lambda_{t_i}) / h_i$ ,  $m = 1, \dots, p - 1$ , the UniPC- $p$  updates in a multistep manner, which reuses the previous evaluation results and proves to be empirically more efficient, especially for limited steps of model evaluation [11, 26], while singlestep methods might incur higher computation cost per step. Therefore, we use multistep UniPC in our experiments by default. The detailed algorithms for multistep UniPC and the proof of convergence can be found in Appendix B. For UniPC, the choices of  $\{r_i\}_{i = 1}^{p - 1}$  determine different updating methods. If all the values are in (0, 1], the UniPC will switch to singlestep. Notably, we find in experiments that our UniC consistently improves different updating methods. Besides, we find UniP-2 (8) and UniC-1 (7) degenerate to a simple equation where only a single  $a_1$  is unknown, where we find  $a_1 = 0.5$  can be a solution for both  $B_1(h)$  and  $B_2(h)$  (see Appendix F) independent of  $h$ . In Appendix C, we provide another variant of UniPC called UniPC $_v$  where the coefficients do not depend on  $h$  for arbitrary order  $p$ .
In the conditional inference, guided sampling [14, 8] is often employed. Recent works [30, 26] find that thresholding data prediction models can boost the sampling quality and mitigate the problem of
(a) CIFAR10 (Pixel DPM)
Figure 2: Unconditional sampling results. We compare our UniPC with DPM-Solver++ [25] on CIFAR10, LSUN Bedroom, and FFHQ. We report the FID↓ of the methods with different numbers of function evaluations (NFE). Experimental results demonstrate that our method is consistently better than previous ones on both pixel-space DPMs and latent-space DPMs, especially with extremely few steps. For more results, we recommend referring to Table 8-10 in Appendix G.
(b) LSUN Bedroom (Latent DPM)
(c) FFHQ (Latent DPM)
train-test mismatch. Our framework of UniPC can be easily adapted to the data prediction model, see Appendix A for algorithms and theoretical analysis. The detailed algorithms for multistep UniPC for data prediction are in Appendix B. Hence, UniPC with data prediction can achieve fast conditional sampling in extremely few steps through dynamic thresholding.