SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Require: noise prediction model  $\epsilon_{\theta}$ , noise schedule  $\alpha_{t}, \sigma_{t}$ , coefficients  $l_{\lambda}, s_{\lambda}, b_{\lambda}$ , cache  $Q_{1}, Q_{2}$
2:  $Q_{2} \stackrel{c a c h e}{\leftarrow} \epsilon_{\theta}(\boldsymbol{x}_{0}, t_{0})$
3: for  $m = 1$  to  $M$  do
4:  $i_0 \gets (n + 1)(m - 1)$
5: for  $i = 0$  to  $n$  do
6:  $\hat{\pmb{x}}_{i_0},\dots ,\hat{\pmb{x}}_{i_0 + i}\stackrel {fetch}{\leftarrow}Q_1$
7:  $\hat{\epsilon}_{i_0},\dots ,\hat{\epsilon}_{i_0 + i}\stackrel {fetch}{\leftarrow}Q_2$
8:  $\hat{\pmb{g}}_l\gets e^{-\int_{\lambda_{i_0}}^{\lambda_l}\pmb{s}_{\tau}\mathrm{d}\tau}\frac{\sigma_{\lambda_l}\hat{\pmb{e}}_l - \pmb{l}_{\lambda_l}\hat{\pmb{x}}_l}{\alpha_{\lambda_l}} -\int_{\lambda_{i_0}}^{\lambda_l}e^{-\int_{\lambda_{i_0}}^r\pmb{s}_{\tau}\mathrm{d}\tau}\pmb {b}_r\mathrm{d}r,\quad l = i_0,\dots ,i_0 + i$  (Eq. (8))
9:  $\hat{\pmb{x}}_{i_0 + i + 1}\gets \mathrm{LUpdate}_{i + 1}(\{t_{i_0 + 1},\hat{\pmb{g}}_{i_0 + 1}\} ,\dots ,\{t_{i_0 + i},\hat{\pmb{g}}_{i_0 + i}\} ,\{t_{i_0},\hat{\pmb{x}}_{i_0},\hat{\pmb{g}}_{i_0}\} ,t_{i_0 + i + 1})$
10: if  $(i + 1)m\neq (n + 1)M$  then
11:  $\hat{\pmb{\epsilon}}_{i_0 + i + 1}\gets \pmb {\epsilon}_\theta (\hat{\pmb{x}}_{i_0 + i + 1},t_{i_0 + i + 1})$
12:  $Q_{1}\stackrel {cache}{\leftarrow}\hat{\pmb{x}}_{i_{0} + i + 1}$
13:  $Q_{2}\stackrel {cache}{\leftarrow}\hat{\epsilon}_{i_{0} + i + 1}$
14: end if
15: end for
16: end for
Output:  $\hat{\pmb{x}}_{(n + 1)M}$
Since UniPC [58] is based on a multistep predictor-corrector framework and has no singlestep version, we only compare with DPM-Solver [31] and DPM-Solver++ [32], which uses noise prediction and data prediction respectively. The results of ScoreSDE model [51] on CIFAR10 [24] are shown in Table 12, which demonstrate that different parameterizations have different relative performance under singlestep and multistep methods.
Specifically, for singlestep methods, DPM-Solver-v3 (S) outperforms DPM-Solver++ (S) across NFEs, but DPM-Solver (S) is even better than DPM-Solver-v3 (S) when  $\mathrm{NFE} \geq 10$  (though when  $\mathrm{NFE} < 10$ , DPM-Solver (S) has worst performance), which suggests that noise prediction is best strategy in such scenarios; for multistep methods, we have DPM-Solver-v3 (M) > DPM-Solver++ (M) > DPM-Solver (M) across NFEs. Moreover, DPM-Solver (S) even outperforms DPM-Solver++ (M) when  $\mathrm{NFE} \geq 20$ , and the properties of different parameterizations in singlestep methods are left for future study. Overall, DPM-Solver-v3 (M) achieves the best results among all these methods, and performs the most stably under different NFEs.
H FID/CLIP Score on Stable-Diffusion
Table 13: Sample quality and text-image alignment performance on MS-COCO2014 [26] prompts with Stable-Diffusion model [43] and guidance scale 7.5. We report the FID↓ and CLIP score↑ of the methods with different numbers of function evaluations (NFE), evaluated on 10k samples.
Method	Metric	NFE
5	6	8	10	12	15	20
DPM-Solver++ [32]	FID	18.87	17.44	16.40	15.93	15.78	15.84	15.72
CLIP score	0.263	0.265	0.265	0.265	0.266	0.265	0.265
UniPC [58]	FID	18.77	17.32	16.20	16.15	16.09	16.06	15.94
CLIP score	0.262	0.263	0.265	0.265	0.265	0.265	0.265
DPM-Solver-v3	FID	18.83	16.41	15.41	15.32	15.13	15.30	15.23
CLIP score	0.260	0.262	0.264	0.265	0.265	0.265	0.265
In text-to-image generation, since the sample quality is affected not only by discretization error of the sampling process, but also by estimation error of neural networks during training, low MSE (faster convergence) does not necessarily imply better sample quality. Therefore, we choose the MSCOCO2014 [26] validation set as the reference, and additionally evaluate DPM-Solver-v3 on Stable-Diffusion model [43] by the standard metrics FID and CLIP score [41] which measure the sample quality and text-image alignment respectively. For DPM-Solver-v3, we use the full-corrector strategy when NFE<10, and no corrector when NFE≥10.
The results in Table 13 show that DPM-Solver-v3 achieves consistently better FID and similar CLIP scores. Notably, we achieve an FID of 15.4 in 8 NFE, close to the reported FID of Stable-Diffusion v1.4.
Still, we claim that FID is not a proper metric for evaluating the convergence of latent-space diffusion models. As stated in DPM-Solver++ and Section 4.1, we can see that the FIDs quickly achieve  $15.0 \sim 16.0$  within 10 steps, even if the latent code does not converge, because of the strong image decoder. Instead, MSE in the latent space is a direct way to measure the convergence. By comparing the MSE, our sampler does converge faster to the ground-truth samples of Stable Diffusion itself.
I More Theoretical Analyses
I.1 Expressive Power of Our Generalized Parameterization
Though the introduced coefficients  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  seem limited to guarantee the optimality of the parameterization formulation itself, we claim that the generalized parameterization  $g_{\theta}$  in Eq. (8) can actually cover a wide range of parameterization families in the form of  $\psi_{\theta}(x_{\lambda}, \lambda) = \alpha(\lambda)\epsilon_{\theta}(x_{\lambda}, \lambda) + \beta(\lambda)x_{\lambda} + \gamma(\lambda)$ . Considering the parameterization on  $[\lambda_s, \lambda_t]$ , by rearranging the terms, Eq. (8) can be written as
$$
\boldsymbol {g} _ {\theta} \left(\boldsymbol {x} _ {\lambda}, \lambda\right) = e ^ {- \int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \frac {\sigma_ {\lambda}}{\alpha_ {\lambda}} \boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {x} _ {\lambda}, \lambda\right) - e ^ {- \int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \frac {\boldsymbol {l} _ {\lambda}}{\alpha_ {\lambda}} \boldsymbol {x} _ {\lambda} - \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r \tag {68}
$$
We can compare the coefficients before  $\epsilon_{\theta}$  and  $x_{\lambda}$  in  $\pmb{g}_{\theta}$  and  $\psi_{\theta}$  to figure out how  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  corresponds to  $\alpha(\lambda), \beta(\lambda), \gamma(\lambda)$ . In fact, we can not directly let  $\pmb{g}_{\theta}$  equal  $\psi_{\theta}$ , since when  $\lambda = \lambda_s$ , we have  $\int_{\lambda_s}^{\lambda} (\cdot) = 0$ , and the coefficient before  $\epsilon_{\theta}$  in  $\pmb{g}_{\theta}$  is fixed. Still,  $\psi_{\theta}$  can be equalized to  $\pmb{g}_{\theta}$  by a linear transformation, which only depends on  $\lambda_s$  and does not affect our analyses of the discretization error and solver.
Specifically, assuming  $\psi_{\theta} = \omega_{\lambda_s}\pmb{g}_{\theta} + \pmb{\xi}_{\lambda_s}$ , by corresponding the coefficients we have
$$
\left\{ \begin{array}{l} \boldsymbol {\alpha} (\lambda) = \boldsymbol {\omega} _ {\lambda_ {s}} e ^ {- \int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \frac {\sigma_ {\lambda}}{\alpha_ {\lambda}} \\ \boldsymbol {\beta} (\lambda) = - \boldsymbol {\omega} _ {\lambda_ {s}} e ^ {- \int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \frac {\boldsymbol {l} _ {\lambda}}{\alpha_ {\lambda}} \\ \boldsymbol {\gamma} (\lambda) = - \boldsymbol {\omega} _ {\lambda_ {s}} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r + \boldsymbol {\xi} _ {\lambda_ {s}} \end{array} \right. \Rightarrow \left\{ \begin{array}{l} \boldsymbol {\omega} _ {\lambda_ {s}} = e ^ {\lambda_ {s}} \boldsymbol {\alpha} (\lambda_ {s}) \\ \boldsymbol {\xi} _ {\lambda_ {s}} = \boldsymbol {\gamma} (\lambda_ {s}) \\ \boldsymbol {l} _ {\lambda} = - \sigma_ {\lambda} \frac {\beta (\lambda)}{\boldsymbol {\alpha} (\lambda)} \\ \boldsymbol {s} _ {\lambda} = - 1 - \frac {\boldsymbol {\alpha} ^ {\prime} (\lambda)}{\boldsymbol {\alpha} (\lambda)} \\ \boldsymbol {b} _ {\lambda} = - e ^ {- \lambda} \frac {\gamma^ {\prime} (\lambda)}{\boldsymbol {\alpha} (\lambda)} \end{array} \right. \tag {69}
$$
Therefore, as long as  $\alpha (\lambda)\neq 0$  and  $\pmb {\alpha}(\lambda),\pmb {\gamma}(\lambda)$  have first-order derivatives, our proposed parameterization  $g_{\theta}$  holds the same expressive power as  $\psi_{\theta}$ , while at the same time enabling the neat optimality criteria of  $l_{\lambda},s_{\lambda},b_{\lambda}$  in Eq. (5) and Eq. (11).
I.2 Justification of Why Minimizing First-order Discretization Error Can Help Higher-order Solver
The EMS  $s_{\lambda}^{*}, b_{\lambda}^{*}$  in Eq. (11) are designed to minimize the first-order discretization error in Eq. (10). However, the high-order solver is actually more frequently adopted in practice (specifically, third-order in unconditional sampling, second-order in conditional sampling), since it incurs lower sampling errors by taking higher-order Taylor expansions to approximate the predictor  $g_{\theta}$ .
In the following, we show that the EMS can also help high-order solver. By Eq. (52) in Appendix B.4, the  $(n + 1)$ -th order local error can be expressed as
$$
\begin{array}{l} \hat {\pmb {x}} _ {t} - \pmb {x} _ {t} = \alpha_ {t} \pmb {A} (\lambda_ {s}, \lambda_ {t}) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \pmb {E} _ {\lambda_ {s}} (\lambda) (\pmb {g} _ {\theta} (\pmb {x} _ {\lambda}, \lambda) - \pmb {g} _ {\theta} (\pmb {x} _ {\lambda_ {s}}, \lambda_ {s})) \mathrm {d} \lambda \\ \left. - \alpha_ {t} \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \sum_ {k = 1} ^ {n} \left(\sum_ {l = 1} ^ {n} \left(\boldsymbol {R} _ {n} ^ {- 1}\right) _ {k l} \left(\boldsymbol {g} _ {\theta} \left(\boldsymbol {x} _ {\lambda_ {i _ {l}}}, \lambda_ {i _ {l}}\right) - \boldsymbol {g} _ {\theta} \left(\boldsymbol {x} _ {\lambda_ {s}}, \lambda_ {s}\right)\right)\right) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} \mathrm {d} \lambda \right. \tag {70} \\ \end{array}
$$
By Newton-Leibniz theorem, it is equivalent to
$$
\begin{array}{l} \hat {\boldsymbol {x}} _ {t} - \boldsymbol {x} _ {t} = \alpha_ {t} \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \left(\int_ {\lambda_ {s}} ^ {\lambda} g _ {\theta} ^ {(1)} \left(x _ {\tau}, \tau\right) \mathrm {d} \tau\right) \mathrm {d} \lambda \tag {71} \\ - \alpha_ {t} \boldsymbol {A} (\lambda_ {s}, \lambda_ {t}) \sum_ {k = 1} ^ {n} \left(\sum_ {l = 1} ^ {n} \left(\boldsymbol {R} _ {n} ^ {- 1}\right) _ {k l} \int_ {\lambda_ {s}} ^ {\lambda_ {i _ {l}}} g _ {\theta} ^ {(1)} (x _ {\lambda}, \lambda) d \lambda\right) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \frac {(\lambda - \lambda_ {s}) ^ {k}}{k !} d \lambda \\ \end{array}
$$
We assume that the estimated EMS are bounded (in the order of  $\mathcal{O}(1)$ , Assumption B.2 in Appendix B.1), which is empirically confirmed as in Section 4.2. By the definition of  $\pmb{g}_{\theta}$  in Eq. (8), we have  $\pmb{g}_{\theta}^{(1)}(\pmb{x}_{\tau},\tau) = e^{-\int_{\lambda_s}^{\tau}\pmb{s}_r\mathrm{d}r}\left(\pmb{f}_{\theta}^{(1)}(\pmb{x}_{\tau},\tau) - \pmb{s}_{\tau}\pmb{f}_{\theta}(\pmb{x}_{\tau},\tau) - \pmb{b}_{\tau}\right)$ . Therefore, Eq. (11) controls  $\| \pmb{g}_{\theta}^{(1)}\|_2$  and further controls  $\| \hat{\pmb{x}}_t - \pmb{x}_t\|_2$ , since other terms are only dependent on the EMS and are bounded.
I.3 The Extra Error of EMS Estimation and Integral Estimation
Analysis of EMS estimation error In practice, the EMS in Eq. (5) and Eq. (11) are estimated on finite datapoints by the explicit expressions in Eq. (58) and Eq. (63), which may differ from the true  $l_{\lambda}^{*}, s_{\lambda}^{*}, b_{\lambda}^{*}$ . Theoretically, on one hand, the order and convergence theorems in Section 3.2 are irrelevant to the EMS estimation error: The ODE solution in Eq. (9) is correct whatever  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  are, and we only need the assumption that these coefficients are bounded (Assumption B.2 in Appendix B.1) to prove the local and global order; on the other hand, the first-order discretization error in Eq. (10) is vulnerable to the EMS estimation error, which relates to the performance at few steps. Empirically, to enable fast sampling, we need to ensure the number of datapoints for estimating EMS (see ablations in Table 8), and we find that our method is robust to the EMS estimation error given only 1024 datapoints in most cases (see EMS computing configis in Appendix D).
Analysis of integral estimation error Another source of error is the process of integral estimation  $(\int_{\lambda_s}^{\lambda_t}\pmb {E}_{\lambda_s}(\lambda)\pmb {B}_{\lambda_s}(\lambda)\mathrm{d}\lambda$  and  $\int_{\lambda_s}^{\lambda_t}\pmb {E}_{\lambda_s}(\lambda)\frac{(\lambda - \lambda_s)^k}{k!}\mathrm{d}\lambda$  in Eq. (14)) by trapezoidal rule. We can analyze the estimation error by the error bound formula of trapezoidal rule: suppose we use uniform discretization on  $[a,b]$  with interval  $h$  to estimate  $\int_{a}^{b}f(x)\mathrm{d}x$ , then the error  $E$  satisfies
$$
| E | \leq \frac {(b - a) h ^ {2}}{1 2} \max  \left| f ^ {\prime \prime} (x) \right| \tag {72}
$$
Under Assumption B.2, the EMS and their first-order derivative are bounded. Denote  $f_{1}(\lambda) = E_{\lambda_{s}}(\lambda)B_{\lambda_{s}}(\lambda), f_{2}(\lambda) = E_{\lambda_{s}}(\lambda)\frac{(\lambda - \lambda_{s})^{k}}{k!}$ , then
$$
\boldsymbol {f} _ {1} (\lambda) = e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r
$$
$$
\boldsymbol {f} _ {1} ^ {\prime} (\lambda) = \left(\boldsymbol {l} _ {\lambda} + \boldsymbol {s} _ {\lambda}\right) e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r + \boldsymbol {b} _ {\lambda} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {l} _ {\tau} \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r
$$
$$
\begin{array}{l} \boldsymbol {f} _ {1} ^ {\prime \prime} (\lambda) = \left(\boldsymbol {l} _ {\lambda} ^ {\prime} + \boldsymbol {s} _ {\lambda} ^ {\prime}\right) e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r + \left(\boldsymbol {l} _ {\lambda} + \boldsymbol {s} _ {\lambda}\right) ^ {2} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {B} _ {r} \mathrm {d} r \\ + \left(\boldsymbol {l} _ {\lambda} + \boldsymbol {s} _ {\lambda}\right) \boldsymbol {b} _ {\lambda} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {l} _ {\tau} \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r + \boldsymbol {b} _ {\lambda} ^ {\prime} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {l} _ {\tau} \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r \\ + \boldsymbol {b} _ {\lambda} \boldsymbol {l} _ {\lambda} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \boldsymbol {l} _ {\tau} \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r + \boldsymbol {b} _ {\lambda} ^ {2} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} (\boldsymbol {l} _ {\tau} - \boldsymbol {s} _ {\tau}) \mathrm {d} \tau} \int_ {\lambda_ {s}} ^ {\lambda} e ^ {- \int_ {\lambda_ {s}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r \tag {73} \\ \end{array}
$$
$$
\boldsymbol {f} _ {2} (\lambda) = e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !}
$$
$$
\boldsymbol {f} _ {2} ^ {\prime} (\lambda) = \left(\boldsymbol {l} _ {\lambda} + \boldsymbol {s} _ {\lambda}\right) e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} + e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k - 1}}{(k - 1) !}
$$
$$
\begin{array}{l} \boldsymbol {f} _ {2} ^ {\prime \prime} (\lambda) = \left(\boldsymbol {l} _ {\lambda} ^ {\prime} + \boldsymbol {s} _ {\lambda} ^ {\prime}\right) e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} + \left(\boldsymbol {l} _ {\lambda} + \boldsymbol {s} _ {\lambda}\right) ^ {2} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} \tag {74} \\ + \left(\boldsymbol {l} _ {\lambda} + \boldsymbol {s} _ {\lambda}\right) e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k - 1}}{k - 1!} + \left(\boldsymbol {l} _ {\lambda} + \boldsymbol {s} _ {\lambda}\right) e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k - 1}}{(k - 1) !} \\ + e ^ {\int_ {\lambda_ {s}} ^ {\lambda} \left(\boldsymbol {l} _ {\tau} + \boldsymbol {s} _ {\tau}\right) \mathrm {d} \tau} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k - 2}}{(k - 2) !} \\ \end{array}
$$
Since  $l_{\lambda}, s_{\lambda}, b_{\lambda}, l_{\lambda}^{\prime}, s_{\lambda}^{\prime}, b_{\lambda}^{\prime}$  are all  $\mathcal{O}(1)$ , we can conclude that  $f_1''(\lambda) = \mathcal{O}(h), f_2''(\lambda) = \mathcal{O}(h^{k-2})$ , and the errors of  $\int_{\lambda_s}^{\lambda_t} E_{\lambda_s}(\lambda) B_{\lambda_s}(\lambda) \, \mathrm{d}\lambda, \int_{\lambda_s}^{\lambda_t} E_{\lambda_s}(\lambda) \frac{(\lambda - \lambda_s)^k}{k!} \, \mathrm{d}\lambda$  are  $O(h_0^2 h^2), O(h_0^2 h^{k-1})$  respectively, where  $h_0$  is the stepsize of EMS discretization ( $h_0 = \frac{\lambda_t - \lambda_s}{n}$ ,  $n$  corresponds to 120~1200 timesteps for our EMS computing), and  $h = \lambda_t - \lambda_s$ . Therefore, the extra error of integral estimation is under high order and ignorable.
J More Discussions
J.1 Extra Computational and Memory Costs
The extra memory cost of DPM-Solver-v3 is rather small. The extra coefficients  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  are discretized and computed at  $N$  timesteps, each with a dimension  $D$  same as the diffused data. The extra memory cost is  $\mathcal{O}(ND)$ , including the precomputed terms in Appendix C.1.2, and is rather small compared to the pretrained model (e.g. only  $\sim 125\mathrm{M}$  in total on Stable-Diffusion, compared to  $\sim 4\mathrm{G}$  of the model itself).
The pre-computation time for estimating EMS is rather short. The EMS introduced by our method can be effectively estimated on around 1k datapoints within hours (Appendix D), which is rather short compared to the long training/distillation time of other methods. Moreover, the integrals of these extra coefficients are just some vector constants that can be pre-computed within seconds, as shown in Appendix C.1.2. The precomputing is done only once before sampling.
The extra computational overhead of DPM-Solver-v3 during sampling is negligible. Once we obtain the estimated EMS and their integrals at discrete timesteps, they can be regarded as constants. Thus, during the subsequent sampling process, the computational overhead is the same as previous training-free methods (such as DPM-Solver++) with negligible differences (Appendix E).
Figure 8: Visualization of the EMS  $s_{\lambda}$ ,  $b_{\lambda}$  and their integrals w.r.t.  $\lambda$ , estimated on ScoreSDE [51] on CIFAR10 [24].  $s_{\lambda}$ ,  $b_{\lambda}$  are rather fluctuating, but their integrals are smooth enough to ensure the stability of DPM-Solver-v3.
J.2 Flexibility
The pre-computed EMS can be applied for any time schedule during sampling without re-computing EMS. Besides, we compute EMS on unconditional models and it can be used for a wide range of guidance scales (such as  $\mathrm{cfg} = 7.5$  in Stable Diffusion). In short, EMS is flexible and easy to adopt in downstream applications.
- Time schedule: The choice of  $N$ ,  $K$  in EMS is disentangled with the timestep scheduler in sampling. Once we have estimated the EMS at  $N$  (e.g., 1200) timesteps, they can be flexibly adapted to any schedule (uniform  $\lambda$ /uniform  $t \dots$ ) during sampling, by corresponding the actual timesteps during sampling to the  $N$  bins. For different time schedule, we only need to re-precompute  $E_{\lambda_s,\lambda_t}^{(k)}$  in Appendix C.1.2, and the time cost is within seconds.
- Guided sampling: We compute the EMS on the unconditional model for all guided cases. Empirically, the EMS computed on the model without guidance (unconditional part) performs more stably than those computed on the model with guidance, and can accelerate the sampling procedure in a wide range of guidance scales (including the common guidance scales used in pretrained models). We think that the unconditional model contains some common information (image priors) for all the conditions, such as color, sketch, and other image patterns. Extracting them helps correct some common biases such as shallow color, low saturation level and lack of details. In contrast, the conditional model is dependent on the condition and has a large variance.
Remark J.1. EMS computed on models without guidance cannot work for extremely large guidance scales (e.g., CFG scale 15 for Stable Diffusion), since in this case, the condition has a large impact on the denoising process. Note that at these extremely large scales, the sampling quality is very low (compared to CFG scale 7.5) and they are rarely used in practice. Therefore, our proposed EMS without guidance is suitable enough for the common applications with the common guidance.
J.3 Stability
As shown in Section 4.2, the estimated EMS  $s_{\lambda}, b_{\lambda}$  appear much fluctuating, especially for ScoreSDE on CIFAR10. We would like to clarify that the unstable  $s_{\lambda}, b_{\lambda}$  is not an issue, and our sampler is stable:
- The fluctuation of  $s_{\lambda}, b_{\lambda}$  on ScoreSDE is intrinsic and not due to the estimation error. As we increase the number of samples to decrease the estimation error, the fluctuation is not reduced. We attribute it to the periodicity of trigonometric functions in the positional timestep embedding as stated in Section 4.2.
- Moreover, we only need to consider the integrals of  $s_{\lambda}, b_{\lambda}$  in the ODE solution Eq. (9). As shown in Figure 8, the integrals of  $s_{\lambda}, b_{\lambda}$  are rather smooth, which ensures the stability of our method. Therefore, there is no need for extra smoothing.
J.4 Practical Value
When NFE is around 20, our improvement of sample quality is small because all different fast samplers based on diffusion ODEs almost converge. Therefore, what matters is that our method has a faster convergence speed to good sample quality. As we observed, the less diverse the domain is, the more evidently the speed-up takes effect. For example, on LSUN-Bedroom, our method can achieve up to  $40\%$  faster convergence.