SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which can also be computed in  $\mathcal{O}(1)$  time. But for  $k > 0$ , we no longer have such a simplification technique. Still, for any fixed timestep schedule  $\{\lambda_i\}_{i=0}^M$  during the sampling process, we can use a lazy precomputing strategy: compute  $E_{\lambda_{i-1}, \lambda_i}^{(k)}$ ,  $1 \leq i \leq M$  when generating the first sample, store it with a unique key  $(k, i)$  and retrieve it in  $\mathcal{O}(1)$  in the following sampling process.
C.2 Algorithm
We provide the pseudocode of the local approximation and global solver in Algorithm 1 and Algorithm 2, which concisely describes how we implement DPM-Solver-v3.
Algorithm 1  $(n + 1)$ -th order local approximation: LUpdate $_{n + 1}$
Require: noise schedule  $\alpha_{t},\sigma_{t}$  , coefficients  $l_{\lambda},s_{\lambda},b_{\lambda}$
Input: transition time pair  $(s,t)$ ,  $x_{s}$ ,  $n$  extra timesteps  $\{t_{i_k}\}_{k=1}^n$ ,  $\pmb{g}_{\theta}$  values  $(g_{i_n},\dots,g_{i_1},g_s)$  at  $\{(x_{\lambda_{i_k}},t_{i_k})\}_{k=1}^n$  and  $(x_{s},s)$
Input Format:  $\{t_{i_n},\pmb {g}_{i_n}\} ,\dots ,\{t_{i_1},\pmb {g}_{i_1}\} ,\{s,\pmb {x}_s,\pmb {g}_s\} ,t$
1: Compute  $A(\lambda_s,\lambda_t),\int_{\lambda_s}^{\lambda_t}E_{\lambda_s}(\lambda)B_{\lambda_s}(\lambda)\mathrm{d}\lambda ,\int_{\lambda_s}^{\lambda_t}E_{\lambda_s}(\lambda)\frac{(\lambda - \lambda_s)^k}{k!}\mathrm{d}\lambda$  (Appendix C.1.2)
2:  $\delta_{k} = \lambda_{i_{k}} - \lambda_{s},\quad k = 1,\ldots ,n$
$$
\begin{array}{l} 3: \left( \begin{array}{c} \hat {\boldsymbol {g}} _ {s} ^ {(1)} \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(2)}}{2 !} \\ \vdots \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(n)}}{n !} \end{array} \right) \leftarrow \left( \begin{array}{c c c c} \delta_ {1} & \delta_ {1} ^ {2} & \dots & \delta_ {1} ^ {n} \\ \delta_ {2} & \delta_ {2} ^ {2} & \dots & \delta_ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ \delta_ {n} & \delta_ {n} ^ {2} & \dots & \delta_ {n} ^ {n} \end{array} \right) ^ {- 1} \left( \begin{array}{c} \boldsymbol {g} _ {i _ {1}} - \boldsymbol {g} _ {s} \\ \boldsymbol {g} _ {i _ {2}} - \boldsymbol {g} _ {s} \\ \vdots \\ \boldsymbol {g} _ {i _ {n}} - \boldsymbol {g} _ {s} \end{array} \right) (\text {E q . (1 3)}) \\ \begin{array}{l} 4: \hat {\boldsymbol {x}} _ {t} \leftarrow \alpha_ {t} \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \left(\frac {\boldsymbol {x} _ {s}}{\alpha_ {s}} - \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \boldsymbol {B} _ {\lambda_ {s}} (\lambda) \mathrm {d} \lambda - \sum_ {k = 0} ^ {n} \hat {\boldsymbol {g}} _ {s} ^ {(k)} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} \mathrm {d} \lambda\right) \\ (\text {E q . (1 4)}) \end{array} \\ \end{array}
$$
Output:  $\hat{x}_t$
Algorithm 2  $(n + 1)$ -th order multistep predictor-corrector algorithm
Require: noise prediction model  $\epsilon_{\theta}$ , noise schedule  $\alpha_{t}, \sigma_{t}$ , coefficients  $l_{\lambda}, s_{\lambda}, b_{\lambda}$ , cache  $Q_{1}, Q_{2}$
Input: timesteps  $\{t_i\}_{i = 0}^M$  , initial value  $x_0$
1:  $Q_{1} \xleftarrow{cache} x_{0}$
2:  $Q_{2}\stackrel {cache}{\leftarrow}\pmb{\epsilon}_{\theta}(\pmb{x}_{0},t_{0})$
3: for  $m = 1$  to  $M$  do
4:  $n_m \gets \min \{n + 1, m\}$
5:  $\hat{\pmb{x}}_{m - n_m},\dots ,\hat{\pmb{x}}_{m - 1}\stackrel {f e t c h}{\leftarrow}Q_1$
6:  $\hat{\epsilon}_{m - n_m},\dots ,\hat{\epsilon}_{m - 1}\stackrel {fetch}{\leftarrow}Q_2$
7:  $\hat{\pmb{g}}_l\gets e^{-\int_{\lambda_{m - 1}}^{\lambda_l}\pmb{s}_{\tau}\mathrm{d}\tau}\frac{\sigma_{\lambda_l}\hat{\pmb{\epsilon}}_l - \pmb{l}_{\lambda_l}\hat{\pmb{x}}_l}{\alpha_{\lambda_l}} -\int_{\lambda_{m - 1}}^{\lambda_l}e^{-\int_{\lambda_{m - 1}}^r\pmb{s}_{\tau}\mathrm{d}\tau}\pmb {b}_r\mathrm{d}r,\quad l = m - n_m,\dots ,m - 1$
(Eq. (8))
8:  $\hat{\pmb{x}}_m\gets \mathrm{LUpdate}_{n_m}(\{t_{m - n_m},\hat{\pmb{g}}_{m - n_m}\} ,\dots ,\{t_{m - 2},\hat{\pmb{g}}_{m - 2}\} ,\{t_{m - 1},\hat{\pmb{x}}_{m - 1},\hat{\pmb{g}}_{m - 1}\} ,t_m)$
9: if  $m \neq M$  then
$$
1 1: \quad \hat {\boldsymbol {g}} _ {m} \leftarrow e ^ {- \int_ {\lambda_ {m - 1}} ^ {\lambda_ {m}} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \frac {\sigma_ {\lambda_ {m}} \hat {\boldsymbol {\epsilon}} _ {m} - \boldsymbol {l} _ {\lambda_ {m}} \hat {\boldsymbol {x}} _ {m}}{\alpha_ {\lambda_ {m}}} - \int_ {\lambda_ {m - 1}} ^ {\lambda_ {m}} e ^ {- \int_ {\lambda_ {m - 1}} ^ {r} \boldsymbol {s} _ {\tau} \mathrm {d} \tau} \boldsymbol {b} _ {r} \mathrm {d} r \tag {Eq. (8)}
$$
10:  $\hat{\pmb{\epsilon}}_m\gets \pmb {\epsilon}_\theta (\hat{\pmb{x}}_m,t_m)$
12:  $\hat{\pmb{x}}_m^c\gets \mathrm{LUpdata}_{n_m}(\{t_{m - n_m + 1},\hat{\pmb{g}}_{m - n_m + 1}\} ,\dots ,\{t_{m - 2},\hat{\pmb{g}}_{m - 2}\} ,\{t_m,\hat{\pmb{g}}_m\} ,$
$\{\pmb{t}_{m - 1},\hat{\pmb{x}}_{m - 1},\hat{\pmb{g}}_{m - 1}\} ,t_m)$  13:  $\hat{\pmb{\epsilon}}_m^c\gets \hat{\pmb{\epsilon}}_m + \pmb {l}_{\lambda_m}(\hat{\pmb{x}}_m^c -\hat{\pmb{x}}_m) / \sigma_{\lambda_m}$  (to ensure  $\hat{\pmb{g}}_m^c = \hat{\pmb{g}}_m)$
14:  $Q_{1}\stackrel {cache}{\leftarrow}\hat{\pmb{x}}_{m}^{c}$
15:  $Q_{2}\stackrel {cache}{\leftarrow}\hat{\epsilon}_{m}^{c}$
16: end if
17: end for
Output:  $\hat{\pmb{x}}_M$
In this section, we provide more experiment details for each setting, including the codebases and the configurations for evaluation, EMS computing and sampling. Unless otherwise stated, we utilize the forward-mode automatic differentiation (torch.autograd.forward_ad) provided by PyTorch [39] to compute the Jacobian-vector-products (JVPs). Also, as stated in Section 3.4, we draw datapoints
$x_{\lambda}$  from the marginal distribution  $q_{\lambda}$  defined by the forward diffusion process starting from some data distribution  $q_{0}$ , instead of the model distribution  $p_{\lambda}^{\theta}$ .
D.1 ScoreSDE on CIFAR10
Codebase and evaluation For unconditional sampling on CIFAR10 [24], one experiment setting is based on the pretrained pixel-space diffusion model provided by ScoreSDE [51]. We use their official codebase of PyTorch implementation, and their checkpoint checkpoint_8.pth under vp/cifar10_ddpmmpp_deep連續 config. We adopt their own statistic file and code for computing FID.
EMS computing We estimate the EMS at  $N = 1200$  uniform timesteps  $\lambda_{j_0},\lambda_{j_1},\ldots ,\lambda_{j_N}$  by drawing  $K = 4096$  datapoints  $\pmb{x}_{\lambda_0}\sim q_0$  , where  $q_{0}$  is the distribution of the training set. We compute two sets of EMS, corresponding to start time  $\epsilon = 10^{-3}$  (NFE  $\leq 10$  ) and  $\epsilon = 10^{-4}$  (NFE>10) in the sampling process respectively. The total time for EMS computing is  $\sim 7h$  on 8 GPU cards of NVIDIA A40.
Sampling Following previous works [31, 32, 58], we use start time  $\epsilon = 10^{-3}$  (NFE  $\leq 10$ ) and  $\epsilon = 10^{-4}$  (NFE>10), end time  $T = 1$  and adopt the uniform logSNR timestep schedule. For DPM-Solver-v3, we use the 3rd-order predictor with the 3rd-order corrector by default. In particular, we change to the pseudo 3rd-order predictor at 5 NFE to further boost the performance.
D.2 EDM on CIFAR10
Codebase and evaluation For unconditional sampling on CIFAR10 [24], another experiment setting is based on the pretrained pixel-space diffusion model provided by EDM [21]. We use their official codebase of PyTorch implementation, and their checkpoint `edm-cifar10-32x32-uncond-vp.pykl. For consistency, we borrow the statistic file and code from ScoreSDE [51] for computing FID.
EMS computing Since the pretrained models of EDM are stored within the pickles, we fail to use torch.autograd.forward_ad for computing JVPs. Instead, we use torch.autogradfunctional.jvp, which is much slower since it employs the double backward trick. We estimate two sets of EMS. One corresponds to  $N = 1200$  uniform timesteps  $\lambda_{j_0}, \lambda_{j_1}, \dots, \lambda_{j_N}$  and  $K = 1024$  datapoints  $x_{\lambda_0} \sim q_0$ , where  $q_0$  is the distribution of the training set. The other corresponds to  $N = 120, K = 4096$ . They are used when NFE<10 and NFE≥10 respectively. The total time for EMS computing is  $\sim 3.5h$  on 8 GPU cards of NVIDIA A40.
Sampling Following EDM, we use start time  $t_{\mathrm{min}} = 0.002$  and end time  $t_{\mathrm{max}} = 80.0$ , but adopt the uniform logSNR timestep schedule which performs better in practice. For DPM-Solver-v3, we use the 3rd-order predictor and additionally employ the 3rd-order corrector when NFE  $\leq 6$ . In particular, we change to the pseudo 3rd-order predictor at 5 NFE to further boost the performance.
D.3 Latent-Diffusion on LSUN-Bedroom
Codebase and evaluation The unconditional sampling on LSUN-Bedroom [55] is based on the pretrained latent-space diffusion model provided by Latent-Diffusion [43]. We use their official codebase of PyTorch implementation and their default checkpoint. We borrow the statistic file and code from Guided-Diffusion [10] for computing FID.
EMS computing We estimate the EMS at  $N = 120$  uniform timesteps  $\lambda_{j_0},\lambda_{j_1},\dots ,\lambda_{j_N}$  by drawing  $K = 1024$  datapoints  $\pmb{x}_{\lambda_0}\sim q_0$  , where  $q_{0}$  is the distribution of the latents of the training set. The total time for EMS computing is  $\sim 12\mathrm{min}$  on 8 GPU cards of NVIDIA A40.
Sampling Following previous works [58], we use start time  $\epsilon = 10^{-3}$ , end time  $T = 1$  and adopt the uniform  $t$  timestep schedule. For DPM-Solver-v3, we use the 3rd-order predictor with the pseudo 4th-order corrector.
D.4 Guided-Diffusion on ImageNet-256
Codebase and evaluation The conditional sampling on ImageNet-256 [9] is based on the pretrained pixel-space diffusion model provided by Guided-Diffusion [10]. We use their official codebase of PyTorch implementation and their two checkpoints: the conditional diffusion model
256x256_diffusion.pt and the classifier 256x256_classifier.pt. We adopt their own statistic file and code for computing FID.
EMS computing We estimate the EMS at  $N = 500$  uniform timesteps  $\lambda_{j_0},\lambda_{j_1},\ldots ,\lambda_{j_N}$  by drawing  $K = 1024$  datapoints  $\pmb{x}_{\lambda_0}\sim q_0$  , where  $q_{0}$  is the distribution of the training set. Also, we find that the FID metric on the ImageNet-256 dataset behaves specially, and degenerated  $l_{\lambda}$ $(l_{\lambda} = 1)$  performs better. The total time for EMS computing is  $\sim 9.5\mathrm{h}$  on 8 GPU cards of NVIDIA A40.
Sampling Following previous works [31, 32, 58], we use start time  $\epsilon = 10^{-3}$ , end time  $T = 1$  and adopt the uniform  $t$  timestep schedule. For DPM-Solver-v3, we use the 2nd-order predictor with the pseudo 3rd-order corrector.
D.5 Stable-Diffusion on MS-COCO2014 prompts
Codebase and evaluation The text-to-image sampling on MS-COCO2014 [26] prompts is based on the pretrained latent-space diffusion model provided by Stable-Diffusion [43]. We use their official codebase of PyTorch implementation and their checkpoint sd-v1-4. ckpt. We compute MSE on randomly selected captions from the MS-COCO2014 validation dataset, as detailed in Section 4.1.
EMS computing We estimate the EMS at  $N = 250$  uniform timesteps  $\lambda_{j_0},\lambda_{j_1},\ldots ,\lambda_{j_N}$  by drawing  $K = 1024$  datapoints  $\pmb{x}_{\lambda_0}\sim q_0$ . Since Stable-Diffusion is trained on the LAION-5B dataset [46], there is a gap between the images in the MS-COCO2014 validation dataset and the images generated by Stable-Diffusion with certain guidance scale. Thus, we choose  $q_{0}$  to be the distribution of the latents generated by Stable-Diffusion with corresponding guidance scale, using 200-step DPM-Solver++ [32]. We generate these latents with random captions and Gaussian noise different from those we use to compute MSE. The total time for EMS computing is  $\sim 11\mathrm{h}$  on 8 GPU cards of NVIDIA A40 for each guidance scale.
Sampling Following previous works [32, 58], we use start time  $\epsilon = 10^{-3}$ , end time  $T = 1$  and adopt the uniform  $t$  timestep schedule. For DPM-Solver-v3, we use the 2nd-order predictor with the pseudo 3rd-order corrector.
D.6 License
Table 3: The used datasets, codes and their licenses.
Name	URL	Citation	License
CIFAR10	https://www.cs.toronto.edu/~kriz/cifar.html	[24]	\
LSUN-Bedroom	https://www.yf.io/p/lsun	[55]	\
ImageNet-256	https://www(image-net.org	[9]	\
MS-COCO2014	https://cocodataset.org	[26]	CC BY 4.0
ScoreSDE	https://github.com/yang-song/score_sde/pytorch	[51]	Apache-2.0
EDM	https://github.com/NVlabs/edm	[21]	CC BY-NC-SA 4.0
Guided-Diffusion	https://github.com/openai/guided-diffusion	[10]	MIT
Latent-Diffusion	https://github.com/CompVis/latent-diffusion	[43]	MIT
Stable-Diffusion	https://github.com/CompVis/stable-diffusion	[43]	CreativeML Open RAIL-M
DPM-Solver++	https://github.com/LuChengTHU/dpm-solver	[32]	MIT
UniPC	https://github.com/wl-zhao/UniPC	[58]	\
We list the used datasets, codes and their licenses in Table 3.
E Runtime Comparison
As we have mentioned in Section 4, the runtime of DPM-Solver-v3 is almost the same as other solvers (DDIM [48], DPM-Solver [31], DPM-Solver++ [32], UniPC [58], etc.) as long as they use the same NFE. This is because the main computation costs are the serial evaluations of the large neural network  $\epsilon_{\theta}$ , and the other coefficients are either analytically computed [48, 31, 32, 58], or precomputed (DPM-Solver-v3), thus having negligible costs.
Table 4 shows the runtime of DPM-Solver-v3 and some other solvers on a single NVIDIA A40 under different settings. We use torch.cuda.Event and torch.cuda}synchronize to accurately compute the runtime. We evaluate the runtime on 8 batches (dropping the first batch since it contains
Table 4: Runtime of different methods to generate a single batch (second / batch, ±std) on a single NVIDIA A40, varying the number of function evaluations (NFE). We don't include the runtime of the decoding stage for latent-space DPMs.
Method	NFE
5	10	15	20
CIFAR10 [24], ScoreSDE [51] (batch size = 128)
DPM-Solver++ [32]	1.253(±0.0014)	2.503(±0.0017)	3.754(±0.0042)	5.010(±0.0048)
UniPC [58]	1.268(±0.0012)	2.532(±0.0018)	3.803(±0.0037)	5.080(±0.0049)
DPM-Solver-v3	1.273(±0.0005)	2.540(±0.0023)	3.826(±0.0039)	5.108(±0.0055)
CIFAR10 [24], EDM [21] (batch size = 128)
DPM-Solver++ [32]	1.137(±0.0011)	2.278(±0.0015)	3.426(±0.0024)	4.569(±0.0031)
UniPC [58]	1.142(±0.0016)	2.289(±0.0019)	3.441(±0.0035)	4.590(±0.0021)
DPM-Solver-v3	1.146(±0.0010)	2.293(±0.0015)	3.448(±0.0018)	4.600(±0.0027)
LSUN-Bedroom [55], Latent-Diffusion [43] (batch size = 32)
DPM-Solver++ [32]	1.302(±0.0009)	2.608(±0.0010)	3.921(±0.0023)	5.236(±0.0045)
UniPC [58]	1.305(±0.0005)	2.616(±0.0019)	3.934(±0.0033)	5.244(±0.0043)
DPM-Solver-v3	1.302(±0.0010)	2.620(±0.0027)	3.932(±0.0028)	5.290(±0.0030)
ImageNet256 [9], Guided-Diffusion [10] (batch size = 4)
DPM-Solver++ [32]	1.594(±0.0011)	3.194(±0.0018)	4.792(±0.0031)	6.391(±0.0045)
UniPC [58]	1.606(±0.0026)	3.205(±0.0025)	4.814(±0.0049)	6.427(±0.0060)
DPM-Solver-v3	1.601(±0.0059)	3.229(±0.0031)	4.807(±0.0068)	6.458(±0.0257)
MS-COCO2014 [26], Stable-Diffusion [43] (batch size = 4)
DPM-Solver++ [32]	1.732(±0.0012)	3.464(±0.0020)	5.229(±0.0027)	6.974(±0.0013)
UniPC [58]	1.735(±0.0012)	3.484(±0.0364)	5.212(±0.0015)	6.988(±0.0035)
DPM-Solver-v3	1.731(±0.0008)	3.471(±0.0011)	5.211(±0.0030)	6.945(±0.0022)
extra initializations) and report the mean and std. We can see that the runtime is proportional to NFE and has a difference of about  $\pm 1\%$  for different solvers, which confirms our statement. Therefore, the speedup for the NFE is almost the actual speedup of the runtime.
F Quantitative Results
Table 5: Quantitative results on LSUN-Bedroom [55]. We report the FID  $\downarrow$  of the methods with different numbers of function evaluations (NFE), evaluated on 50k samples.
Method	Model	NFE
5	6	8	10	12	15	20
DPM-Solver++ [32]	18.59	8.50	4.19	3.63	3.43	3.29	3.16
UniPC [58]	Latent-Diffusion [43]	12.24	6.19	4.00	3.56	3.34	3.18	3.07
DPM-Solver-v3	7.54	4.79	3.53	3.16	3.06	3.05	3.05
We present the detailed quantitative results of Section 4.1 for different datasets in Table 1, Table 5, Table 6 and Table 7 respectively. They clearly verify that DPM-Solver-v3 achieves consistently better or comparable performance under various settings, especially in  $5\sim 10$  NFEs.
G Ablations
In this section, we conduct some ablations to further evaluate and analyze the effectiveness of DPM-Solver-v3.
Table 6: Quantitative results on ImageNet-256 [9]. We report the FID↓ of the methods with different numbers of function evaluations (NFE), evaluated on 10k samples.
Method	Model	NFE
5	6	8	10	12	15	20
DPM-Solver++ [32]	Guided-Diffusion [10]	16.87	13.09	9.95	8.72	8.13	7.73	7.48
UniPC [58]	(s = 2.0)	15.62	11.91	9.29	8.35	7.95	7.64	7.44
DPM-Solver-v3	15.10	11.39	8.96	8.27	7.94	7.62	7.39
Table 7: Quantitative results on MS-COCO2014 [26] prompts. We report the MSE $\downarrow$  of the methods with different numbers of function evaluations (NFE), evaluated on 10k samples.
Method	Model	NFE
5	6	8	10	12	15	20
DPM-Solver++ [32]	Stable-Diffusion [43]	0.076	0.056	0.028	0.016	0.012	0.009	0.006
UniPC [58]	0.055	0.039	0.024	0.012	0.007	0.005	0.002
DPM-Solver-v3	0.037	0.027	0.024	0.007	0.005	0.001	0.002
DPM-Solver++ [32]	Stable-Diffusion [43]	0.60	0.65	0.50	0.46	0.42	0.38	0.30
UniPC [58]	0.65	0.71	0.56	0.46	0.43	0.35	0.31
DPM-Solver-v3	0.55	0.64	0.49	0.40	0.45	0.34	0.29
G.1 Varying the Number of Timesteps and Datapoints for the EMS
Table 8: Ablation of the number of timesteps  $N$  and datapoints  $K$  for the EMS, experimented with ScoreSDE [51] on CIFAR10 [24]. We report the FID $\downarrow$  with different numbers of function evaluations (NFE), evaluated on 50k samples.
N	K	NFE
5	6	8	10	12	15	20
1200	512	18.84	7.90	4.49	3.74	3.88	3.52	3.12
1200	1024	15.52	7.55	4.17	3.56	3.37	3.03	2.78
120	4096	13.67	7.60	4.09	3.49	3.24	2.90	2.70
250	4096	13.28	7.56	4.00	3.45	3.22	2.92	2.70
1200	4096	12.76	7.40	3.94	3.40	3.24	2.91	2.71
First, we'd like to investigate how the number of timesteps  $N$  and the number of datapoints  $K$  for computing the EMS affects the performance. We conduct experiments with the DPM ScoreSDE [51] on CIFAR10 [24], by decreasing  $N$  and  $K$  from our default choice  $N = 1200$ ,  $K = 4096$ .
We list the FID results using the EMS of different  $N$  and  $K$  in Table 8. We can observe that the number of datapoints  $K$  is crucial to the performance, while the number of timesteps  $N$  is less significant and affects mainly the performance in  $5\sim 10$  NFEs. When NFE>10, we can decrease  $N$  to as little as 50, which gives even better FIDs. Note that the time cost for computing the EMS is proportional to  $NK$ , so how to choose appropriate  $N$  and  $K$  for both efficiency and accuracy is worth studying.
G.2 First-Order Comparison
As stated in Appendix A, the first-order case of DPM-Solver-v3 (DPM-Solver-v3-1) is different from DDIM [48], which is the previous best first-order solver for DPMs. Note that DPM-Solver-v3-1 applies no corrector, since any corrector has an order of at least 2.
In Table 9 and Figure 7, we compare DPM-Solver-v3-1 with DDIM both quantitatively and qualitatively, using the DPM ScoreSDE [51] on CIFAR10 [24]. The results verify our statement that DPM-Solver-v3-1 performs better than DDIM.
Table 9: Quantitative comparison of first-order solvers (DPM-Solver-v3-1 and DDIM [48]), experimented with ScoreSDE [51] on CIFAR10 [24]. We report the FID↓ with different numbers of function evaluations (NFE), evaluated on 50k samples.
Method	NFE
5	6	8	10	12	15	20
DDIM [48]	54.56	41.92	27.51	20.11	15.64	12.05	9.00
DPM-Solver-v3-1	39.18	29.82	20.03	14.98	11.86	9.34	7.19
DDIM [48]
FID 54.56
FID 20.11
FID 9.00
DPM-Solver-v3-1 (Ours)
FID 39.18
FID 14.98
FID 7.19
Figure 7: Random samples by first-order solvers (DPM-Solver-v3-1 and DDIM [48]) of ScoreSDE [51] on CIFAR10 dataset [24], using 5, 10 and 20 NFE.
G.3 Effects of Pseudo-Order Solver
We now demonstrate the effectiveness of the pseudo-order solver, including the pseudo-order predictor and the pseudo-order corrector.
Pseudo-order predictor The pseudo-order predictor is only applied in one case (at 5 NFE on CIFAR10 [24]) to achieve maximum performance improvement. In such cases, without the pseudo-order predictor, the FID results will degenerate from 12.76 to 15.91 for ScoreSDE [51], and from 12.21 to 12.72 for EDM [21]. While they are still better than previous methods, the pseudo-order predictor is proven to further boost the performance at NFEs as small as 5.
Pseudo-order corrector We show the comparison between true and pseudo-order corrector in Table 10. We can observe a consistent improvement when switching to the pseudo-order corrector. Thus, it suggests that if we use  $n$ -th order predictor, we'd better combine it with pseudo  $(n + 1)$ -th order corrector rather than  $(n + 1)$ -th order corrector.
G.4 Effects of Half-Corrector
We demonstrate the effects of half-corrector in Table 11, using the popular Stable-Diffusion model [43]. We can observe that under the relatively large guidance scale of 7.5 which is necessary for producing samples of high quality, the corrector adopted by UniPC [58] has a negative effect on the convergence to the ground-truth samples, making UniPC even worse than DPM-Solver++ [32]. When we employ the half-corrector technique, the problem is partially alleviated. Still, it lags behind our DPM-Solver-v3, since we further incorporate the EMS.
Table 10: Effects of pseudo-order corrector under different settings. We report the FID $\downarrow$  with different numbers of function evaluations (NFE).
Method	NFE
5	6	8	10	12	15	20
LSUN-Bedroom [55], Latent-Diffusion [43]
4th-order corrector	8.83	5.28	3.65	3.27	3.17	3.14	3.13
→pseudo (default)	7.54	4.79	3.53	3.16	3.06	3.05	3.05
ImageNet-256 [9], Guided-Diffusion [10] (s = 2.0)
3rd-order corrector	15.87	11.91	9.27	8.37	7.97	7.62	7.47
→pseudo (default)	15.10	11.39	8.96	8.27	7.94	7.62	7.39
MS-COCO2014 [26], Stable-Diffusion [43] (s = 1.5)
3rd-order corrector	0.037	0.028	0.028	0.014	0.0078	0.0024	0.0011
→pseudo (default)	0.037	0.027	0.024	0.0065	0.0048	0.0014	0.0022
Table 11: Ablation of half-corrector/full-corrector on MS-COCO2014 [26] prompts with Stable-Diffusion model [43] and guidance scale 7.5. We report the MSE↓ of the methods with different numbers of function evaluations (NFE), evaluated on 10k samples.
Method	Corrector Usage	NFE
5	6	8	10	12	15	20
DPM-Solver++ [32]	no corrector	0.60	0.65	0.50	0.46	0.42	0.38	0.30
UniPC [58]	full-corrector	0.65	0.71	0.56	0.46	0.43	0.35	0.31
→half-corrector	0.59	0.66	0.50	0.46	0.41	0.38	0.30
DPM-Solver-v3	full-corrector	0.65	0.67	0.49	0.40	0.47	0.34	0.30
→half-corrector	0.55	0.64	0.51	0.44	0.45	0.36	0.29
G.5 Singlestep vs. Multistep
Table 12: Quantitative comparison of single-step methods (S) vs. multi-step methods (M), experimented with ScoreSDE [51] on CIFAR10 [24]. We report the FID $\downarrow$  with different numbers of function evaluations (NFE), evaluated on 50k samples.
Method	NFE
5	6	8	10	12	15	20	25
DPM-Solver (S) [31]	290.51	23.78	23.51	4.67	4.97	3.34	2.85	2.70
DPM-Solver (M) [32]	27.40	17.85	9.04	6.41	5.31	4.10	3.30	2.98
DPM-Solver++ (S) [32]	51.80	38.54	12.13	6.52	6.36	4.56	3.52	3.09
DPM-Solver++ (M) [32]	28.53	13.48	5.34	4.01	4.04	3.32	2.90	2.76
DPM-Solver-v3 (S)	21.83	16.81	7.93	5.76	5.17	3.99	3.22	2.96
DPM-Solver-v3 (M)	12.76	7.40	3.94	3.40	3.24	2.91	2.71	2.64
As we stated in Section 3.2.2, multistep methods perform better than singlestep methods. To study the relationship between parameterization and these solver types, we develop a singlestep version of DPM-Solver-v3 in Algorithm 3. Note that since  $(n + 1)$ -th order singlestep solver divides each step into  $n + 1$  substeps, the total number of timesteps is a multiple of  $n + 1$ . For flexibility, we follow the adaptive third-order strategy of DPM-Solver [31], which first takes third-order steps and then takes first-order or second-order steps in the end.
Algorithm 3  $(n + 1)$ -th order singlestep solver
Input: timesteps  $\{t_i\}_{i = 0}^{(n + 1)M}$ , initial value  $\pmb{x}_0$
1:  $Q_{1}\stackrel {cache}{\leftarrow}x_{0}$