SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\psi_{n}(h)$  is defined by the recursive relation  $\psi_{n + 1}(h) = \frac{1 / n! - \psi_n(h)}{h}$ ,  $\psi_0(h) = e^{-h}$ , see Appendix E.4 for details. The order of accuracy of UniPC- $p$  for the data prediction model is given by the following proposition. The proof is in Appendix E.4.
Proposition A.1 (Order of Accuracy of UniPC- $p$  for Data Prediction Model). For an increasing sequence  $r_1 < r_2 < \dots < r_p = 1$ , under regularity assumption E.3, assuming  $0 \neq B(h) = \mathcal{O}(h)$ ,
$$
\left| \boldsymbol {R} _ {p} \left(h _ {i}\right) \boldsymbol {c} _ {p} B \left(h _ {i}\right) - \boldsymbol {g} _ {p} \left(h _ {i}\right) \right| = \mathcal {O} \left(h _ {i} ^ {p + 1}\right), a n d \left| \boldsymbol {R} _ {p - 1} \left(h _ {i}\right) \boldsymbol {a} _ {p - 1} B \left(h _ {i}\right) - \boldsymbol {g} _ {p - 1} \left(h _ {i}\right) \right| = \mathcal {O} \left(h _ {i} ^ {p}\right), \tag {13}
$$
then the order of accuracy of UniPC-  $p$  is  $p + 1$
We list the algorithms for UniC-  $p$  and UniP-  $p$  for the data prediction model separately, see Algorithm 3 and Algorithm 4.
B Detailed Algorithms of Multistep UniPC
This section offers detailed algorithms with warming-up for multistep UniPC for noise prediction model (Algorithm 5, Algorithm 6) and for data prediction model (Algorithm 7, Algorithm 8).
C UniPC with varying coefficients (UniPCv)
UniPC- $p$  (4) uses a vector  $\mathbf{a}_p$  to match simultaneously all the coefficients of the derivatives. Alternatively, we can use  $\{\mathbf{a}_{i,p}\}_{i = 1}^{p}$  to match each order of derivatives separately and obtain a matrix of
Algorithm 3 UniC-  $p$  for data prediction model
Require:  $\{r_i\}_{i = 1}^{p - 1}$ , data prediction model  $x_{\theta}$ , any  $p$ -order solver Solver-p, a buffer  $Q = \{\pmb{x}_{\theta}(\tilde{\pmb{x}}_{t_{i - k}},t_{i - k})\}_{k = 1}^{p}$ .
$$
\dot {h} _ {i} \leftarrow \lambda_ {t _ {i}} - \lambda_ {t _ {i - 1}}, r _ {p} \leftarrow 1, \tilde {\boldsymbol {x}} _ {t _ {i}} \leftarrow \operatorname {S o l v e r} - \mathrm {p} \left(\tilde {\boldsymbol {x}} _ {t _ {i - 1}} ^ {c}, Q\right)
$$
$$
\begin{array}{l} s _ {m} \leftarrow t _ {\lambda} \left(r _ {m} h _ {i} + \lambda_ {t _ {i - 1}}\right) \\ D _ {m} ^ {x} \leftarrow \boldsymbol {x} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {s _ {m}}, s _ {m}\right) - \boldsymbol {x} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}\right) \\ \end{array}
$$
Compute  $\pmb{c}_p \gets \pmb{R}_p^{-1}(h_i) \pmb{g}_p(h_i) / B(h_i)$ , where  $\pmb{R}_p$  and  $\pmb{g}_p$  are as defined in Theorem 3.1 and (12)
$$
\tilde {\pmb {x}} _ {t _ {i}} ^ {c} \gets \frac {\sigma_ {t _ {i}}}{\sigma_ {t _ {i - 1}}} \tilde {\pmb {x}} _ {t _ {i - 1}} ^ {c} + \alpha_ {t _ {i}} (1 - e ^ {- h _ {i}}) \pmb {x} _ {\theta} (\tilde {\pmb {x}} _ {t _ {i - 1}}, t _ {i - 1}) + \alpha_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p} c _ {m} D _ {m} ^ {x} / r _ {m}
$$
$$
Q \stackrel {\text {b u f f e r}} {\leftarrow} \boldsymbol {x} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i}}, t _ {i}\right)
$$
$$
\text {r e t u r n :} \tilde {\boldsymbol {x}} _ {i _ {c}} ^ {i}
$$
Algorithm 4 UniP-  $p$  for the data prediction model
Require:  $\{r_i\}_{i = 1}^{p - 1}$ , data prediction model  $x_{\theta}$ , a buffer  $Q = \{\pmb {x}_{\theta}(\tilde{\pmb{x}}_{t_{i - k}},t_{i - k})\}_{k = 1}^{p}$ .
Compute  $\pmb{a}_{p-1} \gets \pmb{R}_{p-1}^{-1}(h_i) \pmb{g}_{p-1}(h_i) / B(h_i)$ , where  $\pmb{R}_{p-1}$  and  $\pmb{g}_{p-1}$  are as defined in Theorem 3.1 and (12)
$$
\tilde {\boldsymbol {x}} _ {t _ {i}} \gets \frac {\sigma_ {t _ {i}}}{\sigma_ {t _ {i - 1}}} \tilde {\boldsymbol {x}} _ {t _ {i - 1}} + \alpha_ {t _ {i}} (1 - e ^ {- h _ {i}}) \boldsymbol {x} _ {\theta} (\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}) + \alpha_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p - 1} a _ {m} D _ {m} ^ {x} / r _ {m}
$$
$$
\text {r e t u r n}: \tilde {x} _ {t _ {i}}
$$
coefficients, i.e.,  $\pmb{A}_{p} = (\pmb{a}_{1,p},\dots ,\pmb{a}_{p,p})^{\top}$ . Consider
$$
\tilde {\boldsymbol {x}} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \tilde {\boldsymbol {x}} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) \boldsymbol {\epsilon} _ {\theta} \left(\tilde {\boldsymbol {x}} _ {t _ {i - 1}}, t _ {i - 1}\right) - \sigma_ {t _ {i}} \sum_ {n = 1} ^ {p} h _ {i} \varphi_ {n + 1} \left(h _ {i}\right) \boldsymbol {a} _ {n, p} ^ {\top} \boldsymbol {D} _ {p}, \tag {14}
$$
where  $\pmb{D}_{p} = (D_{1} / r_{1},\dots ,D_{p} / r_{p})^{\top}$  and  $D_{m}$  is defined in (5). The following theorem guarantees the order of accuracy of  $\mathrm{UniPC}_v$ . The proof is deferred to Appendix E. The convergence order is investigated in Appendix D. Define
$$
\boldsymbol {C} _ {p} = \left( \begin{array}{c c c c} 1 & 1 & \dots & 1 \\ r _ {1} / 2! & r _ {2} / 2! & \dots & r _ {p} / 2! \\ \vdots & \vdots & \ddots & \vdots \\ r _ {1} ^ {p - 1} / p! & r _ {2} ^ {p - 1} / p! & \dots & r _ {p} ^ {p - 1} / p! \end{array} \right).
$$
Let  $I_{p}$  be the  $p$ -dimensional identity matrix.
Theorem C.1. Under the conditions of Theorem 3.1, if
$$
\left| \boldsymbol {C} _ {p} \boldsymbol {A} _ {p} - \boldsymbol {I} _ {p} \right| = \mathcal {O} \left(h ^ {p}\right), \quad h = \max  _ {1 \leq i \leq p} h _ {i}, \tag {15}
$$
$UniPC_{v}$  is  $(p + 1)$ -th order accurate.
In fact,  $C_p$  is invertible. Note that  $C_p$  is the product of a diagonal matrix and a Vandermonde matrix, namely
$$
\boldsymbol {C} _ {p} = \left( \begin{array}{c c c} 1 & & \\ & \ddots & \\ & & 1 / p! \end{array} \right) \left( \begin{array}{c c c c} 1 & 1 & \dots & 1 \\ r _ {1} & r _ {2} & \dots & r _ {p} \\ \vdots & \vdots & \ddots & \vdots \\ r _ {1} ^ {p - 1} & r _ {2} ^ {p - 1} & \dots & r _ {p} ^ {p - 1} \end{array} \right), \quad r _ {1} <   \dots <   r _ {p}.
$$
Algorithm 5 Detailed implementation of multistep UniC-  $p$
Require: initial value  $\boldsymbol{x}_T$ , time steps  $\{t_i\}_{i=0}^M$ , noise prediction model  $\epsilon_{\theta}$ , any  $p$ -order solver  
Solver-p  
Denote  $h_i := \lambda_{t_i} - \lambda_{t_{i-1}}$ , for  $i = 1, \dots, M$ .  
 $\tilde{\boldsymbol{x}}_{t_0} \gets \boldsymbol{x}_T, \tilde{\boldsymbol{x}}_{t_0}^c \gets \boldsymbol{x}_T$ . Initialize an empty buffer  $Q$ .  
 $Q^{buffer} \in \epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_0}, t_0)$   
for  $i = 1$  to  $M$  do  
 $p_i \gets \min\{p, i\}$ $\tilde{\boldsymbol{x}}_{t_i}^{(1)} \gets \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}} \tilde{\boldsymbol{x}}_{t_{i-1}}^c - \sigma_{t_i}(e^{h_i} - 1)\epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$ $\tilde{\boldsymbol{x}}_{t_i} \gets \text{Solver-p}_i(\tilde{\boldsymbol{x}}_{t_{i-1}}^c, Q)$ $r_{p_i} \gets 1$ $D_{p_i} \gets \epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i}, t_i) - \epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$   
for  $m = 2$  to  $p_i$  do  
 $r_{m-1} \gets (\lambda_{t_{i-m}} - \lambda_{t_{i-1}})/h_i$ $D_{m-1} \gets \epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_{i-m}}, t_{i-m}) - \epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$   
end for  
Compute  $a_{p_i} \gets R_{p_i}^{-1}(h_i)\phi_{p_i}(h_i)/B(h_i)$ , where  $R_{p_i}^{-1}(h_i)$  and  $\phi_{p_i}$  are as defined in Theorem 3.1.  
 $\tilde{\boldsymbol{x}}_{t_i}^c \gets \tilde{\boldsymbol{x}}_{t_i}^{(1)} - \sigma_{t_i}B(h_i)\sum_{m=1}^{p_i}a_mD_m/r_m$ $Q^{buffer} \in \epsilon_{\theta}(\tilde{\boldsymbol{x}}_{t_i}, t_i)$   
end for  
return:  $\tilde{\boldsymbol{x}}_{t_M}$
Thus, we can simply take  $\mathbf{A}_p = \mathbf{C}_p^{-1}$ . The advantage of  $\mathrm{UniPC}_v$  is that  $\mathbf{A}_p$  solely depends on  $\{r_i\}_{i=1}^p$ . The algorithm of  $\mathrm{UniPC}_v$  is to replace the updating formulae of UniPC (Algorithm 1 and Algorithm 2) by (14) with  $\mathbf{A}_p = \mathbf{C}_p^{-1}$ . See Appendix G for its performance.
D Order of Convergence
In this section, we shall show that under mild conditions the convergence order of UniP-  $p$  is  $p$  and the convergence order of UniPC-  $p$  is  $p + 1$  for either the noise prediction model or data prediction model. The proof is deferred to Appendix E.
Definition D.1. For time steps  $\{t_i\}_{i = 0}^M$ , we say the order of convergence of a sampler for DPMs is  $p$  if
$$
\left| \tilde {\boldsymbol {x}} _ {t _ {M}} - \boldsymbol {x} _ {0} \right| = \mathcal {O} \left(h ^ {p}\right).
$$
We start by introducing some additional regularity assumptions.
Assumption D.2. The noise prediction model  $\epsilon_{\theta}(\pmb{x}, s)$  is Lipschitz continuous w.r.t.  $\pmb{x}$  with Lipschitz constant  $L$ .
Assumption D.3. (1)  $h = \max_{1 \leq i \leq M} h_i = \mathcal{O}(1 / M)$  (2) For a constant  $b > 0$ ,  $\max_{1 \leq i \leq M} L\alpha_{t_{i-1}} / \alpha_{t_i} < b, b^{-1} < \alpha_{t_i} < b$ , for all  $1 \leq i \leq M$ .
Assumption D.4. The starting values  $\tilde{\pmb{x}}_{t_i}$ ,  $1\leq i\leq k - 1$  satisfies for some positive constant  $c_{0}$
$$
\left| \boldsymbol {x} _ {t _ {i}} - \tilde {\boldsymbol {x}} _ {t _ {i}} \right| \leq c _ {0} h ^ {k}, 1 \leq i \leq k - 1. \tag {16}
$$
Assumption D.2 is common in the analysis of ODEs similar to (20). By Assumption D.2, we have  $\epsilon_{\theta}(\tilde{\boldsymbol{x}}_s,s) = \epsilon_{\theta}(\boldsymbol{x}_s,s) + \mathcal{O}(\tilde{\boldsymbol{x}}_s - \boldsymbol{x}_s)$ . Assumption D.3 assures that there is no significantly large step size and the signals are neither exploding nor degenerating. Assumption D.4 is common in the convergence analysis of multistep approaches [4].
The following Propositions D.5 and D.6 ensure the convergence order of UniP-  $p$  and UniPC-  $p$ . For general Solver-  $p$  such as DDIM ( $p = 1$ ), DPM-Solver/DPM-Solver++( $p \leq 3$ ), UniC-  $p$  can also increase the convergence order.
Algorithm 6 Detailed implementation of multistep UniP-  $p$
Require: initial value  $\boldsymbol{x}_T$ , time steps  $\{t_i\}_{i=0}^M$ , noise prediction model  $\epsilon_\theta$   
Denote  $h_i := \lambda_{t_i} - \lambda_{t_{i-1}}$ , for  $i = 1, \dots, M$ .  
 $\tilde{\boldsymbol{x}}_{t_0} \gets \boldsymbol{x}_T$ . Initialize an empty buffer  $Q$ .  
 $Q^{buffer} \epsilon_\theta(\tilde{\boldsymbol{x}}_{t_0}, t_0)$   
for  $i = 1$  to  $M$  do  
 $p_i \gets \min\{p, i\}$ $\tilde{\boldsymbol{x}}_{t_i}^{(1)} \gets \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}} \tilde{\boldsymbol{x}}_{t_{i-1}} - \sigma_{t_i}(e^{h_i} - 1)\epsilon_\theta(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$   
if  $p_i = 1$  then  
 $\tilde{\boldsymbol{x}}_{t_i} \gets \tilde{\boldsymbol{x}}_{t_i}^{(1)}$ $Q^{buffer} \epsilon_\theta(\tilde{\boldsymbol{x}}_{t_i}, t_i)$   
continue  
end if  
for  $m = 2$  to  $p_i$  do  
 $r_{m-1} \gets (\lambda_{t_{i-m}} - \lambda_{t_{i-1}})/h_i$ $D_{m-1} \gets \epsilon_\theta(\tilde{\boldsymbol{x}}_{t_{i-m}}, t_{i-m}) - \epsilon_\theta(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$   
end for  
Compute  $a_{p_i-1} = R_{p_i-1}^{-1}(h_i)\phi_{p_i-1}(h_i)/B(h_i)$ , where  $R_{p_i-1}^{-1}(h_i)$  and  $\phi_{p_i-1}$  are as defined in Theorem 3.1.  
 $\tilde{\boldsymbol{x}}_{t_i} \gets \tilde{\boldsymbol{x}}_{t_i}^{(1)} - \sigma_{t_i}B(h_i)\sum_{m=1}^{p_i-1}a_mD_m/r_m$ $Q^{buffer} \epsilon_\theta(\tilde{\boldsymbol{x}}_{t_i}, t_i)$   
end for  
return:  $\tilde{\boldsymbol{x}}_{t_M}$
Proposition D.5. Under the conditions of Theorem 3.1, Assumptions D.2, D.3, and D.4, the order of convergence of UniP- $p$  is  $p$ .
Proposition D.6. Under the conditions of Theorem 3.1, Assumptions D.2, D.3, and D.4 with  $k = p$ , the order of convergence of UniPC- $p$  is  $p + 1$ .
Remark D.7. After a careful investigation of the proof of Proposition D.6 and Proposition D.5, we point out that for the singlestep updating when using UniPC-1 for the estimation of  $\tilde{\boldsymbol{x}}_{s_m}$ ,  $t_{i - 1} < s_1,\dots ,s_m < t_i$ , the order of convergence for UniPC- $p$  is  $p + 1$  and the order of convergence for UniP- $p$  is  $p$  for  $p\leq 3$ .
Following similar arguments of Proposition D.6, we find that  $\mathrm{UniC - }p$  can also increase the convergence order for general Solver-p for diffusion ODEs. The following is a direct corollary of Proposition D.6 which gives the order of convergence of  $\mathrm{UniPC}_v - p$ .
Corollary D.8. Under the conditions of Theorem C.1, Assumptions D.2, D.3, and D.4 with  $k = p$ , the order of convergence of  $\mathrm{UniPC}_v - p$  is  $p + 1$ .
The order of convergence for the data prediction model follows analogous to Proposition D.5 and Proposition D.6 under slightly different assumptions:
Assumption D.9. The noise prediction model  $\pmb{x}_{\theta}(\pmb{x}, s)$  is Lipschitz continuous w.r.t.  $\pmb{x}$  with Lipschitz constant  $L$ .
Assumption D.10. (1)  $h = \max_{1 \leq i \leq M} h_i = \mathcal{O}(1 / M)$  (2) For a constant  $b > 0$ ,  $\max_{1 \leq i \leq M} L\sigma_{t_{i-1}} / \sigma_{t_i} < b, b^{-1} < \sigma_{t_i} < b$ , for all  $1 \leq i \leq M$ .
We list the results of the order of convergence of UniP and UniPC for the data prediction model as corollaries to Propositions D.5 and D.6 and omit their proofs for simplicity. Interested readers can refer to the arguments in Appendix E for the noise prediction model and derive the detailed proofs for the data prediction model.
Corollary D.11. Under the conditions of Proposition A.1, Assumptions D.9, D.10, and D.4 with  $k = p$ , the order of convergence of UniP- $p$  for data prediction model is  $p$ .
Corollary D.12. Under the conditions of Proposition A.1, Assumptions D.9, D.10, and D.4 with  $k = p$ , the order of convergence of UniPC- $p$  for data prediction model is  $p + 1$ .
Algorithm 7 Detailed implementation of multistep UniC-  $p$  for data prediction model
Require: initial value  $\boldsymbol{x}_T$ , time steps  $\{t_i\}_{i=0}^M$ , data prediction model  $\boldsymbol{x}_\theta$ , any p-order solver  
Solver-p  
Denote  $h_i := \lambda_{t_i} - \lambda_{t_{i-1}}$ , for  $i = 1, \dots, M$ .  
 $\tilde{\boldsymbol{x}}_{t_0} \gets \boldsymbol{x}_T$ ,  $\tilde{\boldsymbol{x}}_{t_0}^c \gets \boldsymbol{x}_T$ . Initialize an empty buffer  $Q$ .  
 $Q^{buffer}$ $\boldsymbol{x}_\theta(\tilde{\boldsymbol{x}}_{t_0}, t_0)$   
for  $i = 1$  to  $M$  do  
 $p_i \gets \min\{p, i\}$ $\tilde{\boldsymbol{x}}_{t_i}^{(1)} \gets \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}} \tilde{\boldsymbol{x}}_{t_{i-1}}^c + \alpha_{t_i}(1 - e^{-h_i}) \boldsymbol{x}_\theta(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$ $\tilde{\boldsymbol{x}}_{t_i} \gets \text{Solver-p}_i(\tilde{\boldsymbol{x}}_{t_{i-1}}^c, Q)$ $r_{p_i} \gets 1$ $D_{p_i} \gets \boldsymbol{x}_\theta(\tilde{\boldsymbol{x}}_{t_i}, t_i) - \boldsymbol{x}_\theta(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$   
for  $m = 2$  to  $p_i$  do  
 $r_{m-1} \gets (\lambda_{t_{i-m}} - \lambda_{t_{i-1}})/h_i$ $D_{m-1}^x \gets \boldsymbol{x}_\theta(\tilde{\boldsymbol{x}}_{t_{i-m}}, t_{i-m}) - \boldsymbol{x}_\theta(\tilde{\boldsymbol{x}}_{t_{i-1}}, t_{i-1})$   
end for  
Compute  $c_{p_i} \gets R_{p_i}^{-1}(h_i) g_{p_i}(h_i)/B(h_i)$ , where  $R_{p_i}^{-1}(h_i)$  and  $g_{p_i}$  are as defined in Theorem 3.1 and (12).  
 $\tilde{\boldsymbol{x}}_{t_i}^{c} \gets \tilde{\boldsymbol{x}}_{t_i}^{(1)} + \alpha_{t_i} B(h_i) \sum_{m=1}^{p_i} c_m D_m^x/r_m$ $Q^{buffer}$ $\boldsymbol{x}_\theta(\tilde{\boldsymbol{x}}_{t_i}, t_i)$   
end for  
return:  $\tilde{\boldsymbol{x}}_{t_M}$
In this section, we provide preliminaries of numerical analysis, regularity assumption, and the proof of Theorem 3.1 in the paper, the regularity assumption and the proof of Proposition A.1 in Appendix A as well as detailed proofs for Theorem C.1 in Appendix C, for Proposition D.5 and Proposition D.6 in Appendix D. Through this section, we use  $C$  and  $C_i$  to denote sufficiently large positive constants independent of  $h$ .
E.1 Preliminaries
We begin with introducing preliminary results and concepts necessary for the proof of Theorem 3.1.
Expansion of exponential integrator. First, we obtain the Taylor expansion of (2), namely the exponentially weighted integral. Define the  $k$ -th order derivative of  $\hat{\epsilon}_{\theta}(\hat{x}_{\lambda},\lambda)$  as  $\hat{\epsilon}_{\theta}^{(k)}(\hat{x}_{\lambda},\lambda)\coloneqq$ $\mathrm{d}^k\hat{\epsilon}_\theta (\hat{x}_\lambda ,\lambda) / \mathrm{d}\lambda^k$  . For  $0\leq t < s\leq T$ $r\in [0,1]$  , let  $h\coloneqq \lambda_t - \lambda_s$ $\lambda \coloneqq \lambda_{s} + rh$  Assuming the existence of total derivatives of  $\hat{\epsilon}_{\theta}^{(k)}(\hat{x}_{\lambda},\lambda)$ $0\le k\le n$  , we have the  $n$  -th order Taylor expansion of  $\hat{\epsilon}_{\theta}(\hat{x}_{\lambda},\lambda)$  w.r.t. the half log-SNR  $\lambda$  ..
$$
\hat {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda}, \lambda\right) = \sum_ {k = 0} ^ {n} \frac {r ^ {k} h ^ {k}}{k !} \hat {\epsilon} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s}}, \lambda_ {s}\right) + \mathcal {O} \left(h ^ {n + 1}\right). \tag {17}
$$
Using the result of Taylor expansion of (17), the exponential integrator of (2) can be reduced to
$$
\begin{array}{l} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \hat {\epsilon} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda}, \lambda) d \lambda = \frac {\sigma_ {t}}{\alpha_ {t}} \sum_ {k = 0} ^ {n} h ^ {k + 1} \int_ {0} ^ {1} e ^ {\lambda_ {t} - \lambda} \frac {r ^ {k}}{k !} d r \hat {\epsilon} _ {\theta} ^ {(k)} (\hat {\boldsymbol {x}} _ {\lambda_ {s}}, \lambda_ {s}) + \mathcal {O} (h ^ {n + 2}) \\ = \frac {\sigma_ {t}}{\alpha_ {t}} \sum_ {k = 0} ^ {n} h ^ {k + 1} \int_ {0} ^ {1} e ^ {(1 - r) h} \frac {r ^ {k}}{k !} \mathrm {d} r \hat {\pmb {\epsilon}} _ {\theta} ^ {(k)} (\hat {\pmb {x}} _ {\lambda_ {s}}, \lambda_ {s}) + \mathcal {O} (h ^ {n + 2}) \\ := \frac {\sigma_ {t}}{\alpha_ {t}} \sum_ {k = 0} ^ {n} h ^ {k + 1} \varphi_ {k + 1} (h) \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s}}, \lambda_ {s}\right) + \mathcal {O} \left(h ^ {n + 2}\right), \tag {18} \\ \end{array}
$$
Algorithm 8 Detailed implementation of multistep UniP-  $p$  for data prediction model
Require: initial value  $\mathbf{x}_T$ , time steps  $\{t_i\}_{i=0}^M$ , data prediction model  $\mathbf{x}_\theta$   
Denote  $h_i := \lambda_{t_i} - \lambda_{t_{i-1}}$ , for  $i = 1, \dots, M$ .  
 $\tilde{\mathbf{x}}_{t_0} \gets \mathbf{x}_T$ . Initialize an empty buffer  $Q$ .  
 $Q^{\text{buffer}} \coloneqq \mathbf{x}_\theta(\tilde{\mathbf{x}}_{t_0}, t_0)$   
for  $i = 1$  to  $M$  do  
 $p_i \gets \min\{p, i\}$ $\tilde{\mathbf{x}}_{t_i}^{(1)} \gets \frac{\sigma_{t_i}}{\sigma_{t_{i-1}}} \tilde{\mathbf{x}}_{t_{i-1}} + \alpha_{t_i}(1 - e^{-h_i}) \mathbf{x}_\theta(\tilde{\mathbf{x}}_{t_{i-1}}, t_{i-1})$   
if  $p_i = 1$  then  
 $\tilde{\mathbf{x}}_{t_i} \gets \tilde{\mathbf{x}}_{t_i}^{(1)}$ $Q^{\text{buffer}} \coloneqq \mathbf{x}_\theta(\tilde{\mathbf{x}}_{t_i}, t_i)$   
continue  
end if  
for  $m = 2$  to  $p_i$  do  
 $r_{m-1} \gets (\lambda_{t_{i-m}} - \lambda_{t_{i-1}})/h_i$ $D_{m-1}^x \gets \mathbf{x}_\theta(\tilde{\mathbf{x}}_{t_{i-m}}, t_{i-m}) - \mathbf{x}_\theta(\tilde{\mathbf{x}}_{t_{i-1}}, t_{i-1})$   
end for  
Compute  $a_{p_i-1} \gets R_{p_i-1}^{-1}(h_i) g_{p_i-1}(h_i)/B(h_i)$ , where  $R_{p_i-1}^{-1}(h_i)$  and  $g_{p_i-1}$  are as defined in Theorem 3.1 and (12).  
 $\tilde{\mathbf{x}}_{t_i} \gets \tilde{\mathbf{x}}_{t_i}^{(1)} + \alpha_{t_i} B(h_i) \sum_{m=1}^{p_i-1} a_m D_m^x/r_m$ $Q^{\text{buffer}} \coloneqq \mathbf{x}_\theta(\tilde{\mathbf{x}}_{t_i}, t_i)$   
end for  
return:  $\tilde{\mathbf{x}}_{t_M}$
where  $\varphi_{k + 1}(h) = \int_0^1 e^{(1 - r)h}\frac{r^k}{k!}\mathrm{d}r$  can be computed via the recurrence relation  $\varphi_{k + 1}(z) = (\varphi_k(z) - \varphi_k(0)) / z$ ,  $\varphi_k(0) = 1 / k!$ , and  $\varphi_0(z) = e^z$  [16]. For example, the closed-forms of  $\varphi_k(h)$  for  $k = 1,2,3$  are
$$
\varphi_ {1} (h) = \frac {e ^ {h} - 1}{h}, \quad \varphi_ {2} (h) = \frac {e ^ {h} - h - 1}{h ^ {2}}, \quad \varphi_ {3} (h) = \frac {e ^ {h} - h ^ {2} / 2 - h - 1}{h ^ {3}}.
$$
Order of accuracy. In the following, we use the linear multistep method to illustrate the order of accuracy. Consider the ODE
$$
y ^ {\prime} = f (x, y), x \in [ x _ {0}, b ], \quad y \left(x _ {0}\right) = y _ {0}. \tag {19}
$$
We say that  $f$  satisfies Lipschitz condition, if there exists  $L > 0$  such that
$$
\left| f \left(x, y _ {1}\right) - f \left(x, y _ {2}\right) \right| \leq L \left| y _ {1} - y _ {2} \right|, \forall y _ {1}, y _ {2} \in \mathbb {R}. \tag {20}
$$
Suppose  $y(x)$  is the solution of Equation (19).
The  $k$ -order linear multistep method is given by
$$
y _ {n + k} = \sum_ {i = 0} ^ {k - 1} \alpha_ {i} y _ {n + i} + h \sum_ {i = 0} ^ {k} \beta_ {i} f \left(x _ {n + i}, y _ {n + i}\right), \tag {21}
$$
where  $y_{n + i}$  is the approximation of  $y(x_{n + i})$ ,  $x_{n + i} = x_{n} + ih$ ,  $\alpha_{i}$ ,  $\beta_{i}$  are constants,  $|\alpha_0| + |\beta_0| > 0$
Definition E.1. The local truncation error of (21) on  $x_{n + k}$  is
$$
T _ {n + k} = y \left(x _ {n + k}\right) - \sum_ {i = 0} ^ {k - 1} \alpha_ {i} y \left(x _ {n + i}\right) - h \sum_ {i = 0} ^ {k} \beta_ {i} f \left(x _ {n + i}, y \left(x _ {n + i}\right)\right). \tag {22}
$$
If  $T_{n + k} = \mathcal{O}(h^{p + 1})$ , we say the order of accuracy of (21) is  $p$ .
The local truncation error describes the error caused by one iteration in the numerical analysis [22].
E.2 Regularity Assumption
Assumption E.2. The total derivatives  $\frac{\mathrm{d}^k\hat{\epsilon}_\theta(\hat{x}_\lambda,\lambda)}{\mathrm{d}\lambda_k}, k = 1,\dots ,p$  exist and are continuous.
Assumption E.2 is required for the Taylor expansion which is also regular in high-order numerical methods.
E.3 Proof of Theorem 3.1 for arbitrary  $p \geq 1$
We first give the local truncation error of UniC. Given  $\boldsymbol{x}_r, r \leq t_{i-1}$ , are correct, with a slight abuse of notation, define
$$
\begin{array}{l} \bar {\pmb {x}} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \pmb {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} (e ^ {h _ {i}} - 1) \pmb {\epsilon} _ {\theta} (\pmb {x} _ {t _ {i - 1}}, t _ {i - 1}) - \sigma_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p - 1} \frac {a _ {m}}{r _ {m}} (\hat {\pmb {\epsilon}} _ {\theta} (\hat {\pmb {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}) - \hat {\pmb {\epsilon}} _ {\theta} (\hat {\pmb {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})) \\ - \sigma_ {t _ {i}} B (h _ {i}) \frac {a _ {p}}{r _ {p}} (\pmb {\epsilon} _ {\theta} (\tilde {\pmb {x}} _ {t _ {i}}, t _ {i}) - \hat {\pmb {\epsilon}} _ {\theta} (\hat {\pmb {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})). \\ \end{array}
$$
Since Solver-p has order of accuracy  $p$  and  $B(h) = \mathcal{O}(h)$ , under Assumption D.2, we have
$$
\begin{array}{l} \bar {\pmb {x}} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \pmb {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} (e ^ {h _ {i}} - 1) \pmb {\epsilon} _ {\theta} (\pmb {x} _ {t _ {i - 1}}, t _ {i - 1}) - \sigma_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p} \frac {a _ {m}}{r _ {m}} (\hat {\pmb {\epsilon}} _ {\theta} (\hat {\pmb {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}) \\ - \hat {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right)) + \mathcal {O} \left(h ^ {p + 2}\right). \tag {23} \\ \end{array}
$$
Suppose  $\pmb{x}_t$  is the solution of the diffusion ODE (1). Then, the local truncation error on  $t_i$  is given by  $|\pmb{x}_{t_i} - \bar{\pmb{x}}_{t_i}|$ . Further, similar to Definition E.1 the order of accuracy of UniC is  $l$ , if there exists a sufficiently large positive constant  $C$  such that
$$
\max  _ {1 \leq i \leq M} | \boldsymbol {x} _ {t _ {i}} - \bar {\boldsymbol {x}} _ {t _ {i}} | \leq C h ^ {l + 1}, \quad h = \max  _ {1 \leq i \leq M} h _ {i}.
$$
In the following, we shall show that the order of accuracy of UniC- $p$  is  $p + 1$ . Combining (2) and (17), we have