SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} | \tilde {\boldsymbol {x}} _ {t _ {i}} - \boldsymbol {x} _ {t _ {i}} | \leq \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} | \tilde {\boldsymbol {x}} _ {t _ {i - 1}} ^ {c} - \boldsymbol {x} _ {t _ {i - 1}} | + \left(C L p h + L \sigma_ {t _ {i}} (e ^ {h _ {i}} - 1)\right) | \tilde {\boldsymbol {x}} _ {t _ {i - 1}} - \boldsymbol {x} _ {t _ {i - 1}} | \\ + C L h \sum_ {m = 1} ^ {p - 1} \left| \tilde {\boldsymbol {x}} _ {t _ {i - m - 1}} - \boldsymbol {x} _ {t _ {i - m - 1}} \right| + C _ {2} h ^ {p + 1}. \tag {41} \\ \end{array}
$$
Let  $e_n = |\tilde{\boldsymbol{x}}_{t_n} - \boldsymbol{x}_{t_n}|$ ,  $f_i = \max_{1\leq i\leq n}e_i$ . Under Assumption D.4 with  $k = p$ ,  $f_{p - 1}\leq c_0h^p$ . For  $0\leq i < p$ , we have
$$
\begin{array}{l} \left| \tilde {\boldsymbol {x}} _ {t _ {i}} ^ {c} - \boldsymbol {x} _ {t _ {i}} \right| \leq \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \left| \tilde {\boldsymbol {x}} _ {t _ {i - 1}} ^ {c} - \boldsymbol {x} _ {t _ {i - 1}} \right| + \left(C L p h + L \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right)\right) \left| \tilde {\boldsymbol {x}} _ {t _ {i - 1}} - \boldsymbol {x} _ {t _ {i - 1}} \right| \\ + C L h \sum_ {m = 1} ^ {i - 1} \left| \tilde {\boldsymbol {x}} _ {t _ {i - m - 1}} - \boldsymbol {x} _ {t _ {i - m - 1}} \right| + L h C _ {1} \left| \tilde {\boldsymbol {x}} _ {t _ {i}} - \boldsymbol {x} _ {t _ {i}} \right| + C _ {0} h ^ {p + 2}. \\ \end{array}
$$
Similar to (40), we obtain  $f_{p - 1}^{c} = \mathcal{O}(h^{p + 1})$ . Iterating (37) and (41), by Theorem 3.1 we have  $|\tilde{\boldsymbol{x}}_{t_M}^c -\boldsymbol {x}_0| = \mathcal{O}(h^{p + 1})$ .
We now provide more details about our UniPC and the experiments.
F.1 Implementation Details about UniPC
Our UniPC is implemented in a multistep manner by default, as is illustrated in Algorithm 5.6. In this case, the extra timesteps that are used to obtain the estimation of higher-order derivatives  $\{s_m\}_{m=1}^{p-1}$  are set to be larger than  $t_{i-1}$ . In other words,  $\{r_m\}_{m=1}^{p-1}$  are all negative. We have also found that the conditions of UniP-2 and UniC-1 degenerate to a simple equation where only a single  $a_1$  is unknown. Specifically, considering (8) and (7), we find that
$$
a _ {1} B (h) - \psi_ {1} (h) = \mathcal {O} \left(h ^ {2}\right), \tag {42}
$$
$$
\psi_ {1} (h) = h \varphi_ {2} (h) = \frac {e ^ {h} - h - 1}{h} = \frac {1}{2} h + \mathcal {O} \left(h ^ {2}\right). \tag {43}
$$
For  $B_{1}(h) = h$ , it is easy to show that when  $a_1 = 0.5$
$$
a _ {1} B (h) - \psi_ {1} (h) = \frac {1}{2} h - \frac {1}{2} h + \mathcal {O} \left(h ^ {2}\right) = \mathcal {O} \left(h ^ {2}\right). \tag {44}
$$
For  $B_{2}(h) = e^{h} - 1 = h + \mathcal{O}(h^{2})$ , the derivation is similar and  $a_1 = 1/2$  also satisfies the condition. Therefore, we can directly set  $a_1 = 1/2$  for UniP-2 and UniC-1 without solving the equation. For higher orders, the vector  $\pmb{a}_{p}$  is computed normally through the inverse of the  $R_{p}$  matrix.
To provide enough data points for high-order UniPC, we need a warming-up procedure in the first few steps, as is also used in previous multistep approaches [26] and is shown in Algorithm 5.6.. Since our UniC needs to compute  $\epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i},t_i)$ , i.e., the model output at the current timestep  $t_i$  to obtain the corrected result  $\boldsymbol{x}_{t_i}^c$ , performing our UniC at the last sampling step will introduce an extra function evaluation. Therefore, we do not use the corrector after the last execution of the predictor for fair comparisons.
Table 8: More unconditional sampling results on CIFAR10 [21].
Sampling Method	NFE
5	6	7	8	9	10
DDIM [34]	55.04	41.81	33.10	27.54	22.92	20.02
DDIM + UniC-1	47.22	33.70	24.60	19.20	15.33	12.77
DPM-Solver-3 [25]	290.65	23.91	15.06	23.56	5.65	4.64
DPM-Solver++(2M) [26]	33.86	21.12	13.93	10.24	7.97	6.83
DPM-Solver++(2M) + UniC-2	31.23	17.96	11.23	8.09	6.29	5.51
DPM-Solver++(3M) [26]	29.22	13.28	7.18	5.21	4.40	4.03
DPM-Solver++(3M) + UniC-3	25.50	11.72	6.79	5.04	4.22	3.90
UniPC-3-B1(h)	23.22	10.33	6.41	5.10	4.29	3.97
UniPC-3-B2(h)	26.20	11.48	6.73	5.11	4.30	3.87
UniPCv-3	25.60	11.10	6.18	4.80	4.19	4.18
Table 9: More unconditional sampling results on FFHQ [18].
Sampling Method	NFE
5	6	7	8	9	10
DDIM [34]	58.23	44.40	34.52	28.06	23.52	19.72
DDIM + UniC	39.41	26.48	18.58	14.56	11.97	10.33
DPM-Solver-3 [25]	54.17	25.24	12.37	8.06	10.22	7.74
DPM-Solver++(2M) [26]	32.50	20.32	14.25	11.30	9.45	8.28
DPM-Solver++(2M) + UniC	24.20	14.92	10.82	9.11	8.01	7.39
DPM-Solver++(3M) [26]	27.15	15.60	10.81	8.98	7.89	7.39
DPM-Solver++(3M) + UniC	21.73	13.38	10.06	8.67	7.89	7.22
UniPC-3-B1(h)	18.66	11.89	9.51	8.21	7.62	6.99
UniPC-3-B2(h)	21.66	13.21	9.93	8.63	7.69	7.20
F.2 Details about the experiments.
We now provide more details of our experiments. For unconditional sampling on CIFAR10 [21], we use the ScoreSDE [35] codebase and their pre-trained model, which is a continuous-time DDPM++ model [35]. More concretely, we use the cifar10_ddpmmpp_deep連續 config file, the same as the example provided by the official code of DPM-Solver [25]. To compute FID, we adopt the statistic file provided by ScoreSDE [35] codebase. For unconditional sampling on LSUN Bedroom [39] and FFHQ [18], we adopt the latent-space DPM provided by the stable-diffusion codebase [29]. Since there is no statistic file for these two datasets in the codebase, we compute the dataset statistic of FFHQ using the script in the library pytorch-fid, and borrow the statistic file of LSUN Bedroom from the guided-diffusion codebase [8]. For conditional sampling on pixel space, we implement our method in guided-diffusion codebase [8] and use the pre-trained checkpoint for ImageNet  $256 \times 256$ . For conditional sampling on latent space, we adopt the stable-diffusion codebase and use their sd-v1-3. ckpt checkpoint, which is pre-trained on LAION [32]. To obtain the text prompts, we randomly sample 10K captions from the MS-COCO2014 validation dataset [23]. We sample 10K random latent code  $x_{T}^{*}$  for each caption and fix them when using different methods.
In this section, we will provide more detailed results, including both quantitative and qualitative results.
G.1 More Quantitative Results
Unconditional Sampling. We start by demonstrating detailed results on CIFAR10 [21], which are shown in Table 8. The results of our proposed method are highlighted in gray. Apart from the results already illustrated in Figure 2, we also include the performance of the DPM-Solver [25]
Table 10: More unconditional results on LSUN [39].
Sampling Method	NFE
5	6	7	8	9	10
DDIM [34]	40.40	25.56	17.93	13.47	10.77	8.95
DPM-Solver++(3M) [26]	17.79	8.03	4.97	4.04	3.79	3.63
DPM-Solver++(3M) + UniC	13.79	6.53	4.58	3.98	3.69	3.52
UniPC-3-B1(h)	11.88	5.99	4.40	3.97	3.74	3.62
UniPC-3-B2(h)	13.60	6.44	4.47	3.91	3.76	3.54
Table 11: More conditional sampling results on ImageNet  ${256} \times  {256}$  .
Sampling Method	NFE
5	6	7	8	9	10
guidance scale s = 8.0
DDIM [34]	38.11	25.31	17.71	14.50	11.51	10.34
DEIS [40]	83.80	67.73	54.91	44.91	37.84	31.84
DPM-Solver++ [26]	55.64	24.07	14.25	12.22	9.53	8.49
UniPC-B1(h)	68.69	34.76	22.79	20.45	17.44	16.02
UniPC-B2(h)	25.87	12.30	8.72	8.68	7.72	7.51
guidance scale s = 4.0
DDIM [34]	27.41	18.47	13.52	11.19	9.66	8.76
DEIS [40]	37.86	24.00	16.00	11.69	9.49	8.05
DPM-Solver++ [26]	31.57	14.58	9.92	9.43	7.98	7.51
UniPC-B1(h)	22.83	12.59	9.77	9.67	8.90	8.52
UniPC-B2(h)	19.48	10.81	8.18	8.09	7.49	7.31
guidance scale s = 1.0
DDIM [34]	36.08	27.02	21.38	17.59	15.20	13.40
DPM-Solver++ [26]	26.10	17.75	13.95	13.10	11.88	11.11
UniPC-B2(h)	22.22	15.79	12.72	12.28	11.30	10.84
and our  $\mathrm{UniPC}_v$  which has varying coefficients (see Appendix C for detailed description). Firstly, we show that DPM-Solver performs very unstable with extremely few steps: the FID of 5 NFE comes to 290.65, which means the solver crashes in this case. Besides, we also show that the DPM-Solver++(3M) [26] performs consistently better than DPM-Solver-3 [25], indicating the multistep method is more effective in few-step sampling setting. That is also why we tend to use DPM-Solver++ as our baseline method in all of our experiments. Table 8 also shows the comparisons between variants of our UniPC, such as UniPC with different instantiations of  $B(h)$  and  $\mathrm{UniPC}_v$ . Since the comparisons between  $B_1(h)$  and  $B_2(h)$  have already been discussed in Section 4.2, we now focus on the analysis of the performance of  $\mathrm{UniPC}_v$ . We find  $\mathrm{UniPC}_v$  achieves the best sampling quality with 7.9 NFE, while cannot beat other variants of UniPC with 5,6,10 NFE. These results show that different variants of our UniPC may have different applications and we should select the most suitable one according to the budget of NFE. We also include more experimental results of the unconditional sampling results on FFHQ [18] in LSUN Bedroom [39], as summarized in Table 9 and Table 10. Note that there are fewer results for LSUN because we performed most of our experiments on FFHQ and CIFAR10 in the early stage. Nevertheless, the overall conclusions of the results on different datasets are aligned. To sum up, our results show (1) multistep methods behave better than singlestep ones when the NFE budget is extremely small. (2) our UniC can consistently improve the sampling quality of a wide range of off-the-shelf solvers. (3) selecting a proper variant of UniPC can yield better results in most cases.
Conditional Sampling. We also provide more results on guided sampling on ImageNet  $256 \times 256$ , as is shown in Table 11. We evaluate the performance of DEIS [40] and find it performs worse than both the DPM-Solver++ and our UniPC. Besides, we have compared the choice of  $B(h)$  on guided
DDIM [34]
DPM-Solver++ [26]
UniPC (Ours)
Figure 5: Comparisons between the images sampled from a DPM pre-trained on ImageNet256 × 256 using DDIM [34], DPM-Solver++ [26] and our UniPC with only 7 NFE.
sampling, where we find  $B_{2}(h)$  significantly outperforms  $B_{1}(h)$ , perhaps because  $B_{1}(h) = h$  is too simple and not suitable for the guided sampling. We have also added the results when the guidance scale  $s = 1.0$ . The results show that our method can achieve better sampling quality with both large and small guidance scales with few sampling steps.
G.2 More Qualitative Results
We provide more visualizations to demonstrate the qualitative performance. First, we consider the class-conditioned guided sampling, i.e., the conditional sampling on ImageNet [7] in Figure 5. Specifically, we compare the sampling quality of each method with only 7 NFE. Note that we randomly sample the class from the total 1000 classes in ImageNet for each image sample, but fix the initial noise for different methods.