SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kaiwen Zheng\*†1, Cheng Lu\*1, Jianfei Chen1, Jun Zhu†123
$^{1}$ Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab
$^{1}$ Tsinghua-Bosch Joint ML Center, Tsinghua University, Beijing, China
$^{2}$ Shengshu Technology, Beijing  $^{3}$ Pazhou Lab (Huangpu), Guangzhou, China
{zkwthu,lucheng.lc15}@gmail.com; {jianfeic, dcszj}@tsinghua.edu.cn
Diffusion probabilistic models (DPMs) have exhibited excellent performance for high-fidelity image generation while suffering from inefficient sampling. Recent works accelerate the sampling procedure by proposing fast ODE solvers that leverage the specific ODE form of DPMs. However, they highly rely on specific parameterization during inference (such as noise/data prediction), which might not be the optimal choice. In this work, we propose a novel formulation towards the optimal parameterization during sampling that minimizes the first-order discretization error of the ODE solution. Based on such formulation, we propose DPM-Solver-v3, a new fast ODE solver for DPMs by introducing several coefficients efficiently computed on the pretrained model, which we call empirical model statistics. We further incorporate multistep methods and a predictor-corrector framework, and propose some techniques for improving sample quality at small numbers of function evaluations (NFE) or large guidance scales. Experiments show that DPM-Solver-v3 achieves consistently better or comparable performance in both unconditional and conditional sampling with both pixel-space and latent-space DPMs, especially in  $5\sim 10$  NFEs. We achieve FIDs of 12.21 (5 NFE), 2.51 (10 NFE) on unconditional CIFAR10, and MSE of 0.55 (5 NFE, 7.5 guidance scale) on Stable Diffusion, bringing a speed-up of  $15\% \sim 30\%$  compared to previous state-of-the-art training-free methods. Code is available at https://github.com/thu-ml/DPM-Solver-v3.
Diffusion probabilistic models (DPMs) [47, 15, 51] are a class of state-of-the-art image generators. By training with a strong encoder, a large model, and massive data as well as leveraging techniques such as guided sampling, DPMs are capable of generating high-resolution photorealistic and artistic images on text-to-image tasks. However, to generate high-quality visual content, DPMs usually require hundreds of model evaluations to gradually remove noise using a pretrained model [15], which is much more time-consuming compared to other deep generative models such as generative adversarial networks (GANs) [13]. The sampling overhead of DPMs emerges as a crucial obstacle hindering their integration into downstream tasks.
To accelerate the sampling process of DPMs, one can employ training-based methods [37, 53, 45] or training-free samplers [48, 51, 28, 3, 52, 56]. We focus on the latter approach since it requires no
arXiv:submit/5200479 [cs.CV] 27 Oct 2023
†Work done during an internship at Shengshu Technology
$\ddagger$  Corresponding author
(a) DPM-Solver++ [32] (MSE 0.60)
(b) UniPC [58] (MSE 0.65)
(c) DPM-Solver-v3 (Ours, MSE 0.55)
Figure 1: Random samples of Stable-Diffusion [43] with a classifier-free guidance scale 7.5, using only 5 number of function evaluations (NFE) and text prompt "A beautiful castle beside a waterfall in the woods, by Josef Thoma, matte painting, trending on artstation HQ".
extra training and is more flexible. Recent advanced training-free samplers [56, 31, 32, 58] mainly rely on the ODE form of DPMs, since its absence of stochasticity is essential for high-quality samples in around 20 steps. Besides, ODE solvers built on exponential integrators [18] converge faster. To change the original diffusion ODE into the form of exponential integrators, they need to cancel its linear term and obtain an ODE solution, where only the noise predictor needs to be approximated, and other terms can be exactly computed. Besides, Lu et al. [32] find that it is better to use another ODE solution where instead the data predictor needs to be approximated. How to choose such model parameterization (e.g., noise/data prediction) in the sampling of DPMs remains unrevealed.
In this work, we systematically study the problem of model parameterization and ODE formulation for fast sampling of DPMs. We first show that by introducing three types of coefficients, the original ODE solution can be reformulated to an equivalent one that contains a new model parameterization. Besides, inspired by exponential Rosenbrock-type methods [19] and first-order discretization error analysis, we also show that there exists an optimal set of coefficients efficiently computed on the pretrained model, which we call empirical model statistics (EMS). Building upon our novel ODE formulation, we further propose a new high-order solver for diffusion ODEs named  $DPM-Solver\upsilon3$ , which includes a multistep predictor-corrector framework of any order, as well as some novel techniques such as pseudo high-order method to boost the performance at extremely few steps or large guidance scale.
We conduct extensive experiments with both pixel-space and latent-space DPMs to verify the effectiveness of DPM-Solver-v3. Compared to previous fast samplers, DPM-Solver-v3 can consistently improve the sample quality in  $5\sim 20$  steps, and make a significant advancement within 10 steps.
2.1 Diffusion Probabilistic Models
Suppose we have a  $D$ -dimensional data distribution  $q_{0}(\pmb{x}_{0})$ . Diffusion probabilistic models (DPMs) [47, 15, 51] define a forward diffusion process  $\{q_t\}_{t=0}^T$  to gradually degenerate the data  $\pmb{x}_0 \sim q_0(\pmb{x}_0)$  with Gaussian noise, which satisfies the transition kernel  $q_{0t}(\pmb{x}_t|\pmb{x}_0) = \mathcal{N}(\alpha_t\pmb{x}_0,\sigma_t^2\pmb{I})$ , such that  $q_{T}(\pmb{x}_{T})$  is approximately pure Gaussian.  $\alpha_{t}, \sigma_{t}$  are smooth scalar functions of  $t$ , which are called noise schedule. The transition can be easily applied by  $\pmb{x}_t = \alpha_t\pmb{x}_0 + \sigma_t\pmb{\epsilon}, \pmb{\epsilon} \sim \mathcal{N}(\pmb{0},\pmb{I})$ . To train DPMs, a neural network  $\epsilon_{\theta}(\pmb{x}_t,t)$  is usually parameterized to predict the noise  $\epsilon$  by minimizing  $\mathbb{E}_{\pmb{x}_0 \sim q_0(\pmb{x}_0), \epsilon \sim \mathcal{N}(\pmb{0},\pmb{I}), t \sim \mathcal{U}(0,T)}[w(t)\| \epsilon_{\theta}(\pmb{x}_t,t) - \epsilon \|_2^2]$ , where  $w(t)$  is a weighting function. Sampling of DPMs can be performed by solving diffusion ODE [51] from time  $T$  to time 0:
$$
\frac {\mathrm {d} \boldsymbol {x} _ {t}}{\mathrm {d} t} = f (t) \boldsymbol {x} _ {t} + \frac {g ^ {2} (t)}{2 \sigma_ {t}} \boldsymbol {\epsilon} _ {\theta} (\boldsymbol {x} _ {t}, t), \tag {1}
$$
where  $f(t) = \frac{\mathrm{d}\log\alpha_t}{\mathrm{d}t}$ ,  $g^2 (t) = \frac{\mathrm{d}\sigma_t^2}{\mathrm{d}t} -2\frac{\mathrm{d}\log\alpha_t}{\mathrm{d}t}\sigma_t^2$  [23]. In addition, the conditional sampling by DPMs can be conducted by guided sampling [10, 16] with a conditional noise predictor  $\epsilon_{\theta}(\pmb {x}_t,t,c)$  where  $c$  is the condition. Specifically, classifier-free guidance [16] combines the unconditional/conditional model and obtains a new noise predictor  $\epsilon_{\theta}^{\prime}(\pmb {x}_t,t,c)\coloneqq s\epsilon_{\theta}(\pmb {x}_t,t,c) + (1 - s)\epsilon_{\theta}(\pmb {x}_t,t,\emptyset)$
where  $\emptyset$  is a special condition standing for the unconditional case,  $s > 0$  is the guidance scale that controls the trade-off between image-condition alignment and diversity; while classifier guidance [10] uses an extra classifier  $p_{\phi}(c|\boldsymbol{x}_t,t)$  to obtain a new noise predictor  $\epsilon_{\theta}^{\prime}(\boldsymbol{x}_t,t,c) \coloneqq \epsilon_{\theta}(\boldsymbol{x}_t,t) - s\sigma_t\nabla_{\boldsymbol{x}}\log p_{\phi}(c|\boldsymbol{x}_t,t)$ .
In addition, except for the noise prediction, DPMs can be parameterized as score predictor  $s_{\theta}(\pmb{x}_t,t)$  to predict  $\nabla_{\pmb{x}}\log q_t(\pmb{x}_t,t)$ , or data predictor  $\pmb{x}_{\theta}(\pmb{x}_t,t)$  to predict  $\pmb{x}_0$ . Under variance-preserving (VP) noise schedule which satisfies  $\alpha_{t}^{2} + \sigma_{t}^{2} = 1$  [51], "v" predictor  $\pmb{v}_{\theta}(\pmb{x}_t,t)$  is also proposed to predict  $\alpha_{t}\epsilon -\sigma_{t}\pmb{x}_{0}$  [45]. These different parameterizations are theoretically equivalent, but have an impact on the empirical performance when used in training [23, 59].
2.2 Fast Sampling of DPMs with Exponential Integrators
Among the methods for solving the diffusion ODE (1), recent works [56, 31, 32, 58] find that ODE solvers based on exponential integrators [18] are more efficient and robust at a small number of function evaluations ( $< 50$ ). Specifically, an insightful observation by Lu et al. [31] is that, by change-of-variable from  $t$  to  $\lambda_t \coloneqq \log(\alpha_t / \sigma_t)$  (half of the log-SNR), the diffusion ODE is transformed to
$$
\frac {\mathrm {d} \boldsymbol {x} _ {\lambda}}{\mathrm {d} \lambda} = \frac {\dot {\alpha} _ {\lambda}}{\alpha_ {\lambda}} \boldsymbol {x} _ {\lambda} - \sigma_ {\lambda} \epsilon_ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda), \tag {2}
$$
where  $\dot{\alpha}_{\lambda} \coloneqq \frac{\mathrm{d}\alpha_{\lambda}}{\mathrm{d}\lambda}$ . By utilizing the semi-linear structure of the diffusion ODE and exactly computing the linear term [56, 31], we can obtain the ODE solution as Eq. (3) (left). Such exact computation of the linear part reduces the discretization errors [31]. Moreover, by leveraging the equivalence of different parameterizations, DPM-Solver++ [32] rewrites Eq. (2) by the data predictor  $x_{\theta}(x_{\lambda}, \lambda) \coloneqq (x_{\lambda} - \sigma_{\lambda}\epsilon_{\theta}(x_{\lambda}, \lambda)) / \alpha_{\lambda}$  and obtains another ODE solution as Eq. (3) (right). Such solution does not need to change the pretrained noise prediction model  $\epsilon_{\theta}$  during the sampling process, and empirically outperforms previous samplers based on  $\epsilon_{\theta}$  [31].
$$
\frac {\boldsymbol {x} _ {t}}{\alpha_ {t}} = \frac {\boldsymbol {x} _ {s}}{\alpha_ {s}} - \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \boldsymbol {\epsilon} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) d \lambda , \quad \frac {\boldsymbol {x} _ {t}}{\sigma_ {t}} = \frac {\boldsymbol {x} _ {s}}{\sigma_ {s}} + \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {\lambda} \boldsymbol {x} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) d \lambda \tag {3}
$$
However, to the best of our knowledge, the parameterizations for sampling are still manually selected and limited to noise/data prediction, which are not well-studied.
We now present our method. We start with a new formulation of the ODE solution with extra coefficients, followed by our high-order solver and some practical considerations. In the following discussions, we assume all the products between vectors are element-wise, and  $\pmb{f}^{(k)}(\pmb{x}_{\lambda},\lambda) = \frac{\mathrm{d}^k\pmb{f}(\pmb{x}_{\lambda},\lambda)}{\mathrm{d}\lambda^k}$  is the  $k$ -th order total derivative of any function  $\pmb{f}$  w.r.t.  $\lambda$ .
3.1 Improved Formulation of Exact Solutions of Diffusion ODEs
As mentioned in Sec. 2.2, it is promising to explore the semi-linear structure of diffusion ODEs for fast sampling [56, 31, 32]. Firstly, we reveal one key insight that we can choose the linear part according to Rosenbrock-type exponential integrators [19, 18]. To this end, we consider a general form of diffusion ODEs by rewriting Eq. (2) as
$$
\frac {\mathrm {d} \boldsymbol {x} _ {\lambda}}{\mathrm {d} \lambda} = \underbrace {\left(\frac {\dot {\alpha} _ {\lambda}}{\alpha_ {\lambda}} - \boldsymbol {l} _ {\lambda}\right) \boldsymbol {x} _ {\lambda}} _ {\text {l i n e a r p a r t}} - \underbrace {\left(\sigma_ {\lambda} \epsilon_ {\theta} \left(\boldsymbol {x} _ {\lambda} , \lambda\right) - \boldsymbol {l} _ {\lambda} \boldsymbol {x} _ {\lambda}\right)} _ {\text {n o n - l i n e a r p a r t , : = N} _ {\theta} \left(\boldsymbol {x} _ {\lambda}, \lambda\right)}, \tag {4}
$$
where  $l_{\lambda}$  is a  $D$ -dimensional undetermined coefficient depending on  $\lambda$ . We choose  $l_{\lambda}$  to restrict the Frobenius norm of the gradient of the non-linear part w.r.t.  $x$ :
$$
\boldsymbol {l} _ {\lambda} ^ {*} = \underset {\boldsymbol {l} _ {\lambda}} {\operatorname {a r g m i n}} \mathbb {E} _ {p _ {\lambda} ^ {\theta} (\boldsymbol {x} _ {\lambda})} \| \nabla_ {\boldsymbol {x}} \boldsymbol {N} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) \| _ {F} ^ {2}, \tag {5}
$$
where  $p_{\lambda}^{\theta}$  is the distribution of samples on the ground-truth ODE trajectories at  $\lambda$  (i.e., model distribution). Intuitively, it makes  $N_{\theta}$  insensitive to the errors of  $\mathbf{x}$  and cancels all the "linearity" of
$N_{\theta}$ . With  $l_{\lambda} = l_{\lambda}^{*}$ , by the "variation-of-constants" formula [1], starting from  $\pmb{x}_{\lambda_s}$  at time  $s$ , the exact solution of Eq. (4) at time  $t$  is
$$
\boldsymbol {x} _ {\lambda_ {t}} = \alpha_ {\lambda_ {t}} e ^ {- \int_ {\lambda_ {s}} ^ {\lambda_ {t}} l _ {\lambda} \mathrm {d} \lambda} \left(\frac {\boldsymbol {x} _ {\lambda_ {s}}}{\alpha_ {\lambda_ {s}}} - \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} l _ {\tau} \mathrm {d} \tau} \underbrace {\boldsymbol {f} _ {\theta} (\boldsymbol {x} _ {\lambda} , \lambda)} _ {\text {a p p r o x i m a t e d}} \mathrm {d} \lambda\right), \tag {6}
$$
where  $\pmb{f}_{\theta}(\pmb{x}_{\lambda}, \lambda) \coloneqq \frac{N_{\theta}(\pmb{x}_{\lambda}, \lambda)}{\alpha_{\lambda}}$ . To calculate the solution in Eq. (6), we need to approximate  $\pmb{f}_{\theta}$  for each  $\lambda \in [\lambda_s, \lambda_t]$  by certain polynomials [31, 32].
Secondly, we reveal another key insight that we can choose different functions to be approximated instead of  $\pmb{f}_{\theta}$  and further reduce the discretization error, which is related to the total derivatives of the approximated function. To this end, we consider a scaled version of  $\pmb{f}_{\theta}$  i.e.,  $h_{\theta}(x_{\lambda},\lambda)\coloneqq e^{-\int_{\lambda_s}^{\lambda}\pmb{s}_{\tau}\mathrm{d}\tau}\pmb{f}_{\theta}(\pmb{x}_{\lambda},\lambda)$  where  $s_\lambda$  is a  $D$ -dimensional coefficient dependent on  $\lambda$ , and then Eq. (6) becomes
$$
\boldsymbol {x} _ {\lambda_ {t}} = \alpha_ {\lambda_ {t}} e ^ {- \int_ {\lambda_ {s}} ^ {\lambda_ {t}} l _ {\lambda} \mathrm {d} \lambda} \left(\frac {\boldsymbol {x} _ {\lambda_ {s}}}{\alpha_ {\lambda_ {s}}} - \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(l _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \underbrace {\boldsymbol {h} _ {\theta} (\boldsymbol {x} _ {\lambda} , \lambda)} _ {\text {a p p r o x i m a t e d}} \mathrm {d} \lambda\right). \tag {7}
$$
Comparing with Eq. (6), we change the approximated function from  $\pmb{f}_{\theta}$  to  $h_{\theta}$  by using an additional scaling term related to  $s_{\lambda}$ . As we shall see, the first-order discretization error is positively related to the norm of the first-order derivative  $\pmb{h}_{\theta}^{(1)} = e^{-\int_{\lambda_s}^{\lambda}\pmb{s}_{\tau}\mathrm{d}\tau}(\pmb{f}_{\theta}^{(1)} - \pmb{s}_{\lambda}\pmb{f}_{\theta})$ . Thus, we aim to minimize  $\| \pmb{f}_{\theta}^{(1)} - \pmb{s}_{\lambda}\pmb{f}_{\theta}\|_2$ , in order to reduce  $\| \pmb{h}_{\theta}^{(1)}\|_2$  and the discretization error. As  $\pmb{f}_{\theta}$  is a fixed function depending on the pretrained model, this minimization problem essentially finds a linear function of  $\pmb{f}_{\theta}$  to approximate  $\pmb{f}_{\theta}^{(1)}$ . To achieve better linear approximation, we further introduce a bias term  $\pmb{b}_{\lambda} \in \mathbb{R}^{D}$  and construct a function  $\pmb{g}_{\theta}$  satisfying  $\pmb{g}_{\theta}^{(1)} = e^{-\int_{\lambda_s}^{\lambda}\pmb{s}_{\tau}\mathrm{d}\tau}(\pmb{f}_{\theta}^{(1)} - \pmb{s}_{\lambda}\pmb{f}_{\theta} - \pmb{b}_{\lambda})$ , which gives
$$
\boldsymbol {g} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) := e ^ {- \int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {f} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) - \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r. \tag {8}
$$
With  $g_{\theta}$  , Eq. (7) becomes
$$
\boldsymbol {x} _ {\lambda_ {t}} = \alpha_ {\lambda_ {t}} \underbrace {e ^ {- \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {l} _ {\lambda} \mathrm {d} \lambda}} _ {\text {l i n e a r c o e f f i c i e n t}} \left(\frac {\boldsymbol {x} _ {\lambda_ {s}}}{\alpha_ {\lambda_ {s}}} - \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \underbrace {e ^ {\int_ {\lambda_ {s}} ^ {\lambda} (\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}) \mathrm {d} \tau}} _ {\text {s c a l i n g c o e f f i c i e n t}} \left(\underbrace {\boldsymbol {g} _ {\theta} (\boldsymbol {x} _ {\lambda} , \lambda)} _ {\text {a p p r o x i m a t e d}} + \underbrace {\int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r}\right) \mathrm {d} \lambda\right). \tag {9}
$$
Such formulation is equivalent to Eq. (3) but introduces three types of coefficients and a new parameterization  $g_{\theta}$ . We show in Appendix I.1 that the generalized parameterization  $g_{\theta}$  in Eq. (8) can cover a wide range of parameterization families in the form of  $\psi_{\theta}(x_{\lambda},\lambda) = \alpha (\lambda)\epsilon_{\theta}(x_{\lambda},\lambda) + \beta (\lambda)x_{\lambda} + \gamma (\lambda)$ . We aim to reduce the discretization error by finding better coefficients than previous works [31, 32].
Now we derive the concrete formula for analyzing the first-order discretization error. By replacing  $\pmb{g}_{\theta}(\pmb{x}_{\lambda},\lambda)$  with  $\pmb{g}_{\theta}(\pmb{x}_{\lambda_s},\lambda_s)$  in Eq. (9), we obtain the first-order approximation  $\hat{\pmb{x}}_{\lambda_t} = \alpha_{\lambda_t}e^{-\int_{\lambda_s}^{\lambda_t}l_\lambda\mathrm{d}\lambda}\left(\frac{\pmb{x}_{\lambda_s}}{\alpha_{\lambda_s}} -\int_{\lambda_s}^{\lambda_t}e^{\int_{\lambda_s}^{\lambda}(l_\tau +s_\tau)\mathrm{d}\tau}\left(\pmb{g}_{\theta}(\pmb{x}_{\lambda_s},\lambda_s) + \int_{\lambda_s}^{\lambda}e^{-\int_{\lambda_s}^r s_\tau \mathrm{d}\tau}\pmb{b}_r\mathrm{d}r\right)\mathrm{d}\lambda\right)$ . As  $\pmb{g}_{\theta}(\pmb{x}_{\lambda_s},\lambda_s) = \pmb{g}_{\theta}(\pmb{x}_{\lambda},\lambda) + (\lambda_s - \lambda)\pmb{g}_{\theta}^{(1)}(\pmb{x}_{\lambda},\lambda) + \mathcal{O}((\lambda - \lambda_s)^2)$  by Taylor expansion, it follows that the first-order discretization error can be expressed as
$$
\hat {\boldsymbol {x}} _ {\lambda_ {t}} - \boldsymbol {x} _ {\lambda_ {t}} = \alpha_ {\lambda_ {t}} e ^ {- \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {l} _ {\lambda} \mathrm {d} \lambda} \left(\int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {l} _ {\tau} \mathrm {d} \tau} (\lambda - \lambda_ {s}) \left(\boldsymbol {f} _ {\theta} ^ {(1)} (\boldsymbol {x} _ {\lambda}, \lambda) - \boldsymbol {s} _ {\lambda} \boldsymbol {f} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) - \boldsymbol {b} _ {\lambda}\right) \mathrm {d} \lambda\right) + \mathcal {O} (h ^ {3}), \tag {10}
$$
where  $h = \lambda_t - \lambda_s$ . Thus, given the optimal  $l_{\lambda} = l_{\lambda}^{*}$  in Eq. (5), the discretization error  $\hat{x}_{\lambda_t} - x_{\lambda_t}$  mainly depends on  $f_{\theta}^{(1)} - s_{\lambda}f_{\theta} - b_{\lambda}$ . Based on this insight, we choose the coefficients  $s_{\lambda}, b_{\lambda}$  by solving
$$
\boldsymbol {s} _ {\lambda} ^ {*}, \boldsymbol {b} _ {\lambda} ^ {*} = \underset {\boldsymbol {s} _ {\lambda}, \boldsymbol {b} _ {\lambda}} {\operatorname {a r g m i n}} \mathbb {E} _ {p _ {\lambda} ^ {\theta} (\boldsymbol {x} _ {\lambda})} \left[ \left\| \boldsymbol {f} _ {\theta} ^ {(1)} (\boldsymbol {x} _ {\lambda}, \lambda) - \boldsymbol {s} _ {\lambda} \boldsymbol {f} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) - \boldsymbol {b} _ {\lambda} \right\| _ {2} ^ {2} \right]. \tag {11}
$$
For any  $\lambda, l_{\lambda}^{*}, s_{\lambda}^{*}, b_{\lambda}^{*}$  all have analytic solutions involving the Jacobian-vector-product of the pretrained model  $\epsilon_{\theta}$ , and they can be unbiasedly evaluated on a few datapoints  $\{x_{\lambda}^{(n)}\}_{K} \sim p_{\lambda}^{\theta}(x_{\lambda})$  via Monte-Carlo estimation (detailed in Section 3.4 and Appendix C.1.1). Therefore, we call  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  empirical model statistics (EMS). In the following sections, we'll show that by approximating  $g_{\theta}$  with Taylor expansion, we can develop our high-order solver for diffusion ODEs.
(a) Local Approximation
(b) Multistep Predictor-Corrector
Figure 2: Illustration of our high-order solver. (a)  $(n + 1)$ -th order local approximation from time  $s$  to time  $t$ , provided  $n$  extra function values of  $\pmb{g}_{\theta}$ . (b) Multistep predictor-corrector procedure as our global solver. A combination of second-order predictor and second-order corrector is shown.  $a_{.,i - 1}, b_{.,i - 1}, c_{.,i - 1}$  are abbreviations of coefficients in Eq. (8).
3.2 Developing High-Order Solver
In this section, we propose our high-order solver for diffusion ODEs with local accuracy and global convergence order guarantee by leveraging our proposed solution formulation in Eq. (9). The proposed solver and analyses are highly motivated by the methods of exponential integrators [17, 18] in the ODE literature and their previous applications in the field of diffusion models [56, 31, 32, 58]. Though the EMS are designed to minimize the first-order error, they can also help our high-order solver (see Appendix I.2).
For simplicity, denote  $A(\lambda_s, \lambda_t) \coloneqq e^{-\int_{\lambda_s}^{\lambda_t} l_\tau \mathrm{d}\tau}$ ,  $E_{\lambda_s}(\lambda) \coloneqq e^{\int_{\lambda_s}^{\lambda} (l_\tau + s_\tau) \mathrm{d}\tau}$ ,  $B_{\lambda_s}(\lambda) \coloneqq \int_{\lambda_s}^{\lambda} e^{-\int_{\lambda_s}^{\tau} s_\tau \mathrm{d}\tau} b_r \mathrm{d}r$ . Though high-order ODE solvers essentially share the same mathematical principles, since we utilize a more complicated parameterization  $g_\theta$  and ODE formulation in Eq. (9) than previous works [56, 31, 32, 58], we divide the development of high-order solver into simplified local and global parts, which are not only easier to understand, but also neat and general for any order.
3.2.1 Local Approximation
Firstly, we derive formulas and properties of the local approximation, which describes how we transit locally from time  $s$  to time  $t$ . It can be divided into two parts: discretization of the integral in Eq. (9) and estimating the high-order derivatives in the Taylor expansion.
Discretization. Denote  $\pmb{g}_s^{(k)} \coloneqq \pmb{g}_\theta^{(k)}(\pmb{x}_{\lambda_s}, \lambda_s)$ . For  $n \geq 0$ , to obtain the  $(n+1)$ -th order discretization of Eq. (9), we take the  $n$ -th order Taylor expansion of  $\pmb{g}_\theta(\pmb{x}_\lambda, \lambda)$  w.r.t.  $\lambda$  at  $\lambda_s$ :  $\pmb{g}_\theta(\pmb{x}_\lambda, \lambda) = \sum_{k=0}^n \frac{(\lambda - \lambda_s)^k}{k!} \pmb{g}_s^{(k)} + O((\lambda - \lambda_s)^{n+1})$ . Substituting it into Eq. (9) yields
$$
\frac {\boldsymbol {x} _ {t}}{\alpha_ {t}} = \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \left(\frac {\boldsymbol {x} _ {s}}{\alpha_ {s}} - \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \boldsymbol {B} _ {\lambda_ {s}} (\lambda) \mathrm {d} \lambda - \sum_ {k = 0} ^ {n} \boldsymbol {g} _ {s} ^ {(k)} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} \mathrm {d} \lambda\right) + \mathcal {O} \left(h ^ {n + 2}\right) \tag {12}
$$
Here we only need to estimate the  $k$ -th order total derivatives  $\pmb{g}_{\theta}^{(k)}(\pmb{x}_{\lambda_s},\lambda_s)$  for  $0\leq k\leq n$ , since the other terms are determined once given  $\lambda_{s},\lambda_{t}$  and  $\pmb{l}_{\lambda},\pmb{s}_{\lambda},\pmb{b}_{\lambda}$ , which we'll discuss next.