SlowGuess/NeurIPS_2023_OCR huggingface dataset:
High-order derivative estimation. For  $(n + 1)$ -th order approximation, we use the finite difference of  $\pmb{g}_{\theta}(\pmb{x}_{\lambda},\lambda)$  at previous  $n + 1$  steps  $\lambda_{i_n},\ldots ,\lambda_{i_1},\lambda_s$  to estimate each  $\pmb{g}_{\theta}^{(k)}(\pmb{x}_{\lambda_s},\lambda_s)$ . Such derivation is to match the coefficients of Taylor expansions. Concretely, denote  $\delta_k\coloneqq \lambda_{i_k} - \lambda_s$ ,  $\pmb{g}_{i_k}\coloneqq \pmb{g}_{\theta}(\pmb{x}_{\lambda_{i_k}},\lambda_{i_k})$  and the estimated high-order derivatives  $\hat{\pmb{g}}_s^{(k)}$  can be solved by the following linear system:
$$
\left( \begin{array}{c c c c} \delta_ {1} & \delta_ {1} ^ {2} & \dots & \delta_ {1} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ \delta_ {n} & \delta_ {n} ^ {2} & \dots & \delta_ {n} ^ {n} \end{array} \right) \left( \begin{array}{c} \hat {\boldsymbol {g}} _ {s} ^ {(1)} \\ \vdots \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(n)}}{n !} \end{array} \right) = \left( \begin{array}{c} \boldsymbol {g} _ {i _ {1}} - \boldsymbol {g} _ {s} \\ \vdots \\ \boldsymbol {g} _ {i _ {n}} - \boldsymbol {g} _ {s} \end{array} \right) \tag {13}
$$
Then by substituting  $\hat{g}_s^{(k)}$  into Eq. (12) and dropping the  $\mathcal{O}(h^{n + 2})$  error terms, we obtain the  $(n + 1)$ -th order local approximation:
$$
\frac {\hat {\boldsymbol {x}} _ {t}}{\alpha_ {t}} = \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \left(\frac {\boldsymbol {x} _ {s}}{\alpha_ {s}} - \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \boldsymbol {B} _ {\lambda_ {s}} (\lambda) \mathrm {d} \lambda - \sum_ {k = 0} ^ {n} \hat {\boldsymbol {g}} _ {s} ^ {(k)} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} \mathrm {d} \lambda\right) \tag {14}
$$
where  $\hat{\pmb{g}}_{\theta}^{(0)}(\pmb{x}_{\lambda_s},\lambda_s) = \pmb{g}_s$ . Eq. (13) and Eq. (14) provide an update rule to transit from time  $s$  to time  $t$  and get an approximated solution  $\hat{\pmb{x}}_t$ , when we already have the solution  $\pmb{x}_s$ . For  $(n + 1)$ -th order approximation, we need  $n$  extra solutions  $\pmb{x}_{\lambda_{i_k}}$  and their corresponding function values  $\pmb{g}_{i_k}$ . We illustrate the procedure in Fig. 2(a) and summarize it in Appendix C.2. In the following theorem, we show that under some assumptions, such local approximation has a guarantee of order of accuracy.
Theorem 3.1 (Local order of accuracy, proof in Appendix B.2.1). Suppose  $\boldsymbol{x}_{\lambda_{i_k}}$  are exact (i.e., on the ground-truth ODE trajectory passing  $\boldsymbol{x}_s$ ) for  $k = 1, \dots, n$ , then under some regularity conditions detailed in Appendix B.1, the local truncation error  $\hat{\boldsymbol{x}}_t - \boldsymbol{x}_t = \mathcal{O}(h^{n + 2})$ , which means the local approximation has  $(n + 1)$ -th order of accuracy.
Besides, we have the following theorem showing that, whatever the order is, the local approximation is unbiased given our choice of  $s_{\lambda}, b_{\lambda}$  in Eq. (11). In practice, the phenomenon of reduced bias can be empirically observed (Section 4.3).
Theorem 3.2 (Local unbiasedness, proof in Appendix B.4). Given the optimal  $s_{\lambda}^{*}, b_{\lambda}^{*}$  in Eq. (11), for the  $(n + 1)$ -th order approximation, suppose  $\pmb{x}_{\lambda_{i_1}}, \dots, \pmb{x}_{\lambda_{i_n}}$  are on the ground-truth ODE trajectory passing  $\pmb{x}_{\lambda_s}$ , then  $\mathbb{E}_{p_{\lambda_s}^{\theta}(\pmb{x}_s)}[\hat{\pmb{x}}_t - \pmb{x}_t] = 0$ .
3.2.2 Global Solver
Given  $M + 1$  time steps  $\{t_i\}_{i = 0}^M$ , starting from some initial value, we can repeat the local approximation  $M$  times to make consecutive transitions from each  $t_{i - 1}$  to  $t_i$  until we reach an acceptable solution. At each step, we apply multistep methods [1] by caching and reusing the previous  $n$  values at timesteps  $t_{i - 1 - n},\ldots ,t_{i - 2}$ , which is proven to be more stable when NFE is small [32, 56]. Moreover, we also apply the predictor-corrector method [58] to refine the approximation at each step without introducing extra NFE. Specifically, the  $(n + 1)$ -th order predictor is the case of the local approximation when we choose  $(t_{i_n},\dots ,t_{i_1},s,t) = (t_{i - 1 - n},\dots ,t_{i - 2},t_{i - 1},t_i)$ , and the  $(n + 1)$ -th order corrector is the case when we choose  $(t_{i_n},\dots ,t_{i_1},s,t) = (t_{i - n},\dots ,t_{i - 2},t_i,t_{i - 1},t_i)$ . We present our  $(n + 1)$ -th order multistep predictor-corrector procedure in Appendix C.2. We also illustrate a second-order case in Fig. 2(b). Note that different from previous works, in the local transition from  $t_{i - 1}$  to  $t_i$ , the previous function values  $\hat{\pmb{g}}_{i_k}$ $(1\leq k\leq n)$  used for derivative estimation are dependent on  $i$  and are different during the sampling process because  $g_{\theta}$  is dependent on the current  $t_{i - 1}$  (see Eq. (8)). Thus, we directly cache  $\hat{\pmb{x}}_i,\hat{\pmb{\epsilon}}_i$  and reuse them to compute  $\hat{\pmb{g}}_i$  in the subsequent steps. Notably, our proposed solver also has a global convergence guarantee, as shown in the following theorem. For simplicity, we only consider the predictor case and the case with corrector can also be proved by similar derivations in [58].
Theorem 3.3 (Global order of convergence, proof in Appendix B.2.2). For  $(n + 1)$ -th order predictor, if we iteratively compute a sequence  $\{\hat{\pmb{x}}_i\}_{i = 0}^M$  to approximate the true solutions  $\{\pmb {x}_i\}_{i = 0}^M$  at  $\{t_i\}_{i = 0}^M$  then under both local and global assumptions detailed in Appendix B.1, the final error  $|\hat{\pmb{x}}_M - \pmb {x}_M| = \mathcal{O}(h^{n + 1})$ , where  $|\cdot |$  denotes the element-wise absolute value, and  $h = \max_{1\leq i\leq M}(\lambda_i - \lambda_{i - 1})$ .
3.3 Practical Techniques
In this section, we introduce some practical techniques that further improve the sample quality in the case of small NFE or large guidance scales.
Pseudo-order solver for small NFE. When NFE is extremely small (e.g.,  $5\sim 10$ ), the error at each timestep becomes rather large, and incorporating too many previous values by high-order solver at each step will cause instabilities. To alleviate this problem, we propose a technique called pseudo-order solver: when estimating the  $k$ -th order derivative, we only utilize the previous  $k + 1$  function values of  $\pmb{g}_{\theta}$ , instead of all the  $n$  previous values as in Eq. (13). For each  $k$ , we can obtain  $\hat{\pmb{g}}_s^{(k)}$  by
solving a part of Eq. (13) and taking the last element:
$$
\left( \begin{array}{c c c c} \delta_ {1} & \delta_ {1} ^ {2} & \dots & \delta_ {1} ^ {k} \\ \vdots & \vdots & \ddots & \vdots \\ \delta_ {k} & \delta_ {k} ^ {2} & \dots & \delta_ {k} ^ {k} \end{array} \right) \left( \begin{array}{c} \cdot \\ \vdots \\ \frac {\dot {\mathbf {g}} _ {s} ^ {(k)}}{k !} \end{array} \right) = \left( \begin{array}{c} \mathbf {g} _ {i _ {1}} - \mathbf {g} _ {s} \\ \vdots \\ \mathbf {g} _ {i _ {k}} - \mathbf {g} _ {s} \end{array} \right), \quad k = 1, 2, \dots , n \tag {15}
$$
In practice, we do not need to solve  $n$  linear systems. Instead, the solutions for  $\hat{\pmb{g}}_s^{(k)}, k = 1, \dots, n$  have a simpler recurrence relation similar to Neville's method [36] in Lagrange polynomial interpolation. Denote  $i_0 \coloneqq s$  so that  $\delta_0 = \lambda_{i_0} - \lambda_s = 0$ , we have
Theorem 3.4 (Pseudo-order solver). For each  $k$ , the solution in Eq. (15) is  $\hat{\pmb{g}}_s^{(k)} = k!D_0^{(k)}$ , where
$$
D _ {l} ^ {(0)} := \boldsymbol {g} _ {i _ {l}}, \quad l = 0, 1, \dots , n
$$
$$
D _ {l} ^ {(k)} := \frac {D _ {l + 1} ^ {(k - 1)} - D _ {l} ^ {(k - 1)}}{\delta_ {l + k} - \delta_ {l}}, \quad l = 0, 1, \dots , n - k \tag {16}
$$
Proof in Appendix B.3. Note that the pseudo-order solver of order  $n > 2$  no longer has the guarantee of  $n$ -th order of accuracy, which is not so important when NFE is small. In our experiments, we mainly rely on two combinations: when we use  $n$ -th order predictor, we then combine it with  $n$ -th order corrector or  $(n + 1)$ -th pseudo-order corrector.
Half-corrector for large guidance scales. When the guidance scale is large in guided sampling, we find that corrector may have negative effects on the sample quality. We propose a useful technique called half-corrector by using the corrector only in the time region  $t \leq 0.5$ . Correspondingly, the strategy that we use corrector at each step is called full-corrector.
In this section, we give some implementation details about how to compute and integrate the EMS in our solver and how to adapt them to guided sampling.
Estimating EMS. For a specific  $\lambda$ , the EMS  $l_{\lambda}^{*}, s_{\lambda}^{*}, b_{\lambda}^{*}$  can be estimated by firstly drawing  $K$  (1024~4096) datapoints  $\boldsymbol{x}_{\lambda} \sim p_{\lambda}^{\theta}(\boldsymbol{x}_{\lambda})$  with 200-step DPM-Solver++ [32] and then analytically computing some terms related to  $\epsilon_{\theta}$  (detailed in Appendix C.1.1). In practice, we find it both convenient and effective to choose the distribution of the dataset  $q_{0}$  to approximate  $p_{0}^{\theta}$ . Thus, without further specifications, we directly use samples from  $q_{0}$ .
Estimating integrals of EMS. We estimate EMS on  $N$  ( $120 \sim 1200$ ) timesteps  $\lambda_{j_0}, \lambda_{j_1}, \dots, \lambda_{j_N}$  and use trapezoidal rule to estimate the integrals in Eq. (12) (see Appendix I.3 for the estimation error analysis). We also apply some pre-computation for the integrals to avoid extra computation costs during sampling, detailed in Appendix C.1.2.
Adaptation to guided sampling. Empirically, we find that within a common range of guidance scales, we can simply compute the EMS on the model without guidance, and it can work for both unconditional sampling and guided sampling cases. See Appendix J for more discussions.
3.5 Comparison with Existing Methods
By comparing with existing diffusion ODE solvers that are based on exponential integrators [56, 31, 32, 58], we can conclude that (1) Previous ODE formulations with noise/data prediction are special cases of ours by setting  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  to specific values. (2) Our first-order discretization can be seen as improved DDIM. See more details in Appendix A.
In this section, we show that DPM-Solver-v3 can achieve consistent and notable speed-up for both unconditional and conditional sampling with both pixel-space and latent-space DPMs. We conduct extensive experiments on diverse image datasets, where the resolution ranges from 32 to 256. First, we present the main results of sample quality comparison with previous state-of-the-art training-free
(a) CIFAR10
(ScoreSDE, Pixel DPM)
(b) CIFAR10
(EDM, Pixel DPM)
(c) LSUN-Bedroom
(Latent-Diffusion, Latent DPM)
(a) ImageNet-256
(Guided-Diffusion, Pixel DPM)
(Classifier Guidance,  $s = 2.0$ )
Figure 4: Conditional sampling results. We report the  $\mathrm{FID}\downarrow$  or  $\mathrm{MSE}\downarrow$  of the methods with different numbers of function evaluations (NFE), evaluated on 10k samples.
(b) MS-COCO2014
(Stable-Diffusion, Latent DPM)
(Classifier-Free Guidance,  $s = 1.5$ )
Figure 3: Unconditional sampling results. We report the FID $\downarrow$  of the methods with different numbers of function evaluations (NFE), evaluated on 50k samples.  $\dagger$  We borrow the results reported in their original paper directly.
(c) MS-COCO2014
(Stable-Diffusion, Latent DPM)
Classifier-Free Guidance,  $s = 7.5$
methods. Then we illustrate the effectiveness of our method by visualizing the EMS and samples. Additional ablation studies are provided in Appendix G. On each dataset, we choose a sufficient number of timesteps  $N$  and datapoints  $K$  for computing the EMS to reduce the estimation error, while the EMS can still be computed within hours. After we obtain the EMS and precompute the integrals involving them, there is negligible extra overhead in the sampling process. We provide the runtime comparison in Appendix E. We refer to Appendix D for more detailed experiment settings.
We present the results in  $5\sim 20$  number of function evaluations (NFE), covering both few-step cases and the almost converged cases, as shown in Fig. 3 and Fig. 4. For the sake of clarity, we mainly compare DPM-Solver-v3 to DPM-Solver++ [32] and UniPC [58], which are the most state-of-the-art diffusion ODE solvers. We also include the results for DEIS [56] and Heun's 2nd order method in EDM [21], but only for the datasets on which they originally reported. We don't show the results for other methods such as DDIM [48], PNDM [28], since they have already been compared in previous works and have inferior performance. The quantitative results on CIFAR10 [24] are listed in Table 1, and more detailed quantitative results are presented in Appendix F.
Unconditional sampling We first evaluate the unconditional sampling performance of different methods on CIFAR10 [24] and LSUN-Bedroom [55]. For CIFAR10 we use two pixel-space DPMs,
(a) CIFAR10 (ScoreSDE)
Figure 5: Visualization of the EMS  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  w.r.t.  $\lambda$  estimated on different models.
(b) CIFAR10 (EDM)
(c) MS-COCO2014
(Stable-Diffusion,  $s = 7.5$
(a) DPM-Solver++ [32] (FID 18.59)
(b) UniPC [58] (FID 12.24)
(c) DPM-Solver-v3 (Ours) (FID 7.54)
Figure 6: Random samples of Latent-Diffusion [43] on LSUN-Bedroom [55] with only NFE = 5.
Table 1: Quantitative results on CIFAR10 [24]. We report the FID $\downarrow$  of the methods with different numbers of function evaluations (NFE), evaluated on 50k samples.  $\dagger$ We borrow the results reported in their original paper directly.
Method	Model	NFE
5	6	8	10	12	15	20	25
†DEIS [56]	15.37	\	\	4.17	\	3.37	2.86	\
DPM-Solver++ [32]	ScoreSDE [51]	28.53	13.48	5.34	4.01	4.04	3.32	2.90	2.76
UniPC [58]	23.71	10.41	5.16	3.93	3.88	3.05	2.73	2.65
DPM-Solver-v3	12.76	7.40	3.94	3.40	3.24	2.91	2.71	2.64
Heun's 2nd [21]	320.80	103.86	39.66	16.57	7.59	4.76	2.51	2.12
DPM-Solver++ [32]	EDM [21]	24.54	11.85	4.36	2.91	2.45	2.17	2.05	2.02
UniPC [58]	23.52	11.10	3.86	2.85	2.38	2.08	2.01	2.00
DPM-Solver-v3	12.21	8.56	3.50	2.51	2.24	2.10	2.02	2.00
one is based on ScoreSDE [51] which is a widely adopted model by previous samplers, and another is based on EDM [21] which achieves the best sample quality. For LSUN-Bedroom, we use the latent-space Latent-Diffusion model [43]. We apply the multistep 3rd-order version for DPM-Solver++, UniPC and DPM-Solver-v3 by default, which performs best in the unconditional setting. For UniPC, we report the better result of their two choices  $B_{1}(h) = h$  and  $B_{2}(h) = e^{h} - 1$  at each NFE. For our DPM-Solver-v3, we tune the strategies of whether to use the pseudo-order predictor/corrector at each NFE on CIFAR10, and use the pseudo-order corrector on LSUN-Bedroom. As shown in Fig. 3, we find that DPM-Solver-v3 can achieve consistently better FID, which is especially notable when NFE is  $5\sim 10$ . For example, we improve the FID on CIFAR10 with 5 NFE from 23 to 12 with ScoreSDE, and achieve an FID of 2.51 with only 10 NFE with the advanced DPM provided by EDM. On LSUN-Bedroom, with around 12 minutes computing of the EMS, DPM-Solver-v3 converges to the FID of 3.06 with 12 NFE, which is approximately  $60\%$  sampling cost of the previous best training-free method (20 NFE by UniPC).
Conditional sampling. We then evaluate the conditional sampling performance, which is more widely used since it allows for controllable generation with user-customized conditions. We choose two conditional settings, one is classifier guidance on pixel-space Guided-Diffusion [10] model trained on ImageNet-256 dataset [9] with 1000 class labels as conditions; the other is classifier-free guidance on latent-space Stable-Diffusion model [43] trained on LAION-5B dataset [46] with CLIP [41] embedded text prompts as conditions. We evaluate the former at the guidance scale of 2.0, following the best choice in [10]; and the latter at the guidance scale of 1.5 (following the original paper) or 7.5 (following the official code) with prompts random selected from MS-COCO2014 validation set [26]. Note that the FID of Stable-Diffusion samples saturates to  $15.0\sim 16.0$  even within 10 steps when the latent codes are far from convergence, possibly due to the powerful image decoder (see Appendix H). Thus, following [32], we measure the mean square error (MSE) between the generated latent code  $\hat{x}$  and the ground-truth solution  $\boldsymbol{x}^*$  (i.e.,  $\| \hat{\boldsymbol{x}} -\boldsymbol{x}^{*}\|_{2}^{2} / D$ ) to evaluate convergence, starting from the same Gaussian noise. We obtain  $\boldsymbol{x}^*$  by 200-step DPM-Solver++, which is enough to ensure the convergence.
We apply the multistep 2nd-order version for DPM-Solver++, UniPC and DPM-Solver-v3, which performs best in conditional setting. For UniPC, we only apply the choice  $B_{2}(h) = e^{h} - 1$ , which performs better than  $B_{1}(h)$ . For our DPM-Solver-v3, we use the pseudo-order corrector by default, and report the best results between using half-corrector/full-corrector on Stable-Diffusion ( $s = 7.5$ ). As shown in Fig. 4, DPM-Solver-v3 can achieve better sample quality or convergence at most NFEs, which indicates the effectiveness of our method and techniques under the conditional setting. It's worth noting that UniPC, which adopts an extra corrector, performs even worse than DPM-Solver++ when NFE<10 on Stable-Diffusion ( $s = 7.5$ ). With the combined effect of the EMS and the half-corrector technique, we successfully outperform DPM-Solver++ in such a case. Detailed comparisons can be found in the ablations in Appendix G.
4.2 Visualizations of Estimated EMS
We further visualize the estimated EMS in Fig. 5. Since  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  are  $D$ -dimensional vectors, we average them over all dimensions to obtain a scalar. From Fig. 5, we find that  $l_{\lambda}$  gradually changes from 1 to near 0 as the sampling proceeds, which suggests we should gradually slide from data prediction to noise prediction. As for  $s_{\lambda}, b_{\lambda}$ , they are more model-specific and display many fluctuations, especially for ScoreSDE model [51] on CIFAR10. Apart from the estimation error of the EMS, we suspect that it comes from the fluctuations of  $\epsilon_{\theta}^{(1)}$ , which is caused by the periodicity of trigonometric functions in the positional embeddings of the network input. It's worth noting that the fluctuation of  $s_{\lambda}, b_{\lambda}$  will not cause instability in our sampler (see Appendix J).
4.3 Visual Quality
We present some qualitative comparisons in Fig. 6 and Fig. 1. We can find that previous methods tend to have a small degree of color contrast at small NFE, while our method is less biased and produces more visual details. In Fig. 1(b), we can observe that previous methods with corrector may cause distortion at large guidance scales (in the left-top image, a part of the castle becomes a hill; in the left-bottom image, the hill is translucent and the castle is hanging in the air), while ours won't. Additional samples are provided in Appendix K.
We study the ODE parameterization problem for fast sampling of DPMs. Through theoretical analysis, we find a novel ODE formulation with empirical model statistics, which is towards the optimal one to minimize the first-order discretization error. Based on such improved ODE formulation, we propose a new fast solver named DPM-Solver-v3, which involves a multistep predictor-corrector framework of any order and several techniques for improved sampling with few steps or large guidance scale. Experiments demonstrate the effectiveness of DPM-Solver-v3 in both unconditional conditional sampling with both pixel-space latent-space pre-trained DPMs, and the significant advancement of sample quality in  $5\sim 10$  steps.
Limitations and broader impact Despite the great speedup in small numbers of steps, DPM-Solver-v3 still lags behind training-based methods and is not fast enough for real-time applications. Besides, we conducted theoretical analyses of the local error, but didn't explore the global design spaces, such as the design of timestep schedules during sampling. And commonly, there are potential undesirable effects that DPM-Solver-v3 may be abused to accelerate the generation of fake and malicious content.
This work was supported by the National Key Research and Development Program of China (No. 2021ZD0110502), NSFC Projects (Nos. 62061136001, 62106123, 62076147, U19A2081, 61972224, 62106120), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J.Z is also supported by the XPlorer Prize.
[52] Hideyuki Tachibana, Mocho Go, Muneyoshi Inahara, Yotaro Katayama, and Yotaro Watanabe. Itô-taylor sampling scheme for denoising diffusion probabilistic models using ideal derivatives. arXiv e-prints, pages arXiv-2112, 2021.
[53] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2022.
[54] Suttisak Wizadwongsa and Supasorn Suwajanakorn. Accelerating guided diffusion sampling with splitting numerical methods. In The Eleventh International Conference on Learning Representations, 2022.
[55] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[56] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2022.
[57] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. In Advances in Neural Information Processing Systems, 2022.
[58] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A unified predictor-corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.
[59] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. arXiv preprint arXiv:2305.03935, 2023.
Diffusion probabilistic models (DPMs) [47, 15, 51], also known as score-based generative models (SGMs), have achieved remarkable generation ability on image domain [10, 21], yielding extensive applications such as speech, singing and video synthesis [6, 27, 14], controllable image generation, translation and editing [38, 42, 43, 35, 57, 8], likelihood estimation [50, 23, 30, 59], data compression [23] and inverse problem solving [7, 22].
A.1 Fast Sampling Methods for DPMs
Fast sampling methods based on extra training or optimization include learning variances in the reverse process [37, 2], learning sampling schedule [53], learning high-order model derivatives [11], model refinement [29] and model distillation [45, 33, 49]. Though distillation-based methods can generate high-quality samples in less than 5 steps, they additionally bring onerous training costs. Moreover, the distillation process will inevitably lose part of the information of the original model, and is hard to be adapted to pre-trained large DPMs [43, 44, 42] and conditional sampling [34]. Some of distillation-based methods also lack the ability to make flexible trade-offs between sample speed and sample quality.
In contrast, training-free samplers are more lightweight and flexible. Among them, samplers based on diffusion ODEs generally require fewer steps than those based on diffusion SDEs [51, 40, 3, 48, 32], since SDEs introduce more randomness and make the denoising harder in the sampling process. Previous samplers handle the diffusion ODEs with different methods, such as Heun's methods [21], splitting numerical methods [54], pseudo numerical methods [28], Adams methods [25] or exponential integrators [56, 31, 32, 58].
A.2 Comparison with Existing Solvers Based on Exponential Integrators