SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \boldsymbol {x} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} - \alpha_ {t _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- \lambda} \hat {\mathbf {e}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda}, \lambda) d \lambda \\ = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \pmb {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \sum_ {k = 0} ^ {p} h _ {i} ^ {k + 1} \varphi_ {k + 1} (h _ {i}) \hat {\pmb {\epsilon}} _ {\theta} ^ {(k)} (\hat {\pmb {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) + \mathcal {O} (h ^ {p + 2}) \\ = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) \boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}\right) - \sigma_ {t _ {i}} \sum_ {k = 1} ^ {p} h _ {i} ^ {k + 1} \varphi_ {k + 1} \left(h _ {i}\right) \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) + \mathcal {O} \left(h ^ {p + 2}\right). \tag {24} \\ \end{array}
$$
Let  $\pmb{R}_{p,k}^{\top}$  denote the  $k$ -th row of  $\pmb{R}_p$ . By (7), we have  $|\pmb{a}_p^\top \pmb{R}_{p,k}(h_i) - B^{-1}(h_i)h_i^k k!\varphi_{k + 1}(h_i)| \leq C_0B^{-1}(h_i)h^{p + 1}$ . Under Assumption E.2, by Taylor expansion, we obtain
$$
\begin{array}{l} \sum_ {m = 1} ^ {p} \frac {a _ {m}}{r _ {m}} (\hat {\epsilon} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}) - \hat {\epsilon} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})) \\ = \sum_ {m = 1} ^ {p} a _ {m} \sum_ {n = 1} ^ {p} \frac {r _ {m} ^ {n - 1} h _ {i} ^ {n}}{n !} \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(n)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) + \mathcal {O} \left(h ^ {p + 1}\right) \\ = \sum_ {n = 1} ^ {p} \sum_ {m = 1} ^ {p} a _ {m} \frac {r _ {m} ^ {n - 1} h _ {i} ^ {n}}{n !} \hat {\epsilon} _ {\theta} ^ {(n)} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) + \mathcal {O} (h ^ {p + 1}) \\ = \sum_ {n = 1} ^ {p} \frac {h _ {i}}{n !} \pmb {\alpha} _ {p} ^ {\top} \pmb {R} _ {p, n} (h _ {i}) \hat {\pmb {\epsilon}} _ {\theta} ^ {(n)} (\hat {\pmb {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) + \mathcal {O} (h ^ {p + 1}). \\ \end{array}
$$
$$
\begin{array}{l} \left| \sum_ {m = 1} ^ {p} \frac {a _ {m}}{r _ {m}} \left(\hat {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}\right) - \hat {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right)\right) - B (h _ {i}) ^ {- 1} \sum_ {k = 1} ^ {p} h _ {i} ^ {k + 1} \varphi_ {k + 1} \left(h _ {i}\right) \hat {\epsilon} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) \right| \\ \leq C _ {1} \left(B \left(h _ {i}\right) ^ {- 1} h ^ {p + 2} + h ^ {p + 1}\right). \tag {25} \\ \end{array}
$$
Combining (23), (24) and (25) given  $B(h_{i}) = \mathcal{O}(h_{i})$ , we have
$$
\begin{array}{l} \max  _ {1 \leq i \leq M} | \boldsymbol {x} _ {t _ {i}} - \bar {\boldsymbol {x}} _ {t _ {i}} | = | - \sigma_ {t _ {i}} \sum_ {k = 1} ^ {p} h _ {i} ^ {k + 1} \varphi_ {k + 1} (h _ {i}) \hat {\epsilon} _ {\theta} ^ {(k)} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) \\ + \sigma_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p} \frac {a _ {m}}{r _ {m}} (\hat {\boldsymbol {\epsilon}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}) - \hat {\boldsymbol {\epsilon}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})) | + \mathcal {O} (h ^ {p + 2}) \\ = \mathcal {O} \left(h ^ {p + 2}\right). \tag {26} \\ \end{array}
$$
Therefore,  $\mathrm{UniC - }p$  is of  $(p + 1)$  -th order of accuracy.
E.4 Proof of Proposition A.1
Regularity Assumption
Assumption E.3. The total derivatives  $\frac{\mathrm{d}^k\hat{x}_\theta(\hat{x}_\lambda,\lambda)}{\mathrm{d}\lambda_k}, k = 1,\dots ,p$  exist and are continuous.
Expansion of the Exponentially Weighted Integral. First, we obtain the Taylor expansion of (9). Define the  $k$ -th order derivative of  $\hat{\pmb{x}}_{\theta}(\hat{\pmb{x}}_{\lambda},\lambda)$  as  $\hat{\pmb{x}}_{\theta}^{(k)}(\hat{\pmb{x}}_{\lambda},\lambda)\coloneqq \mathrm{d}^k\hat{\pmb{x}}_\theta (\hat{\pmb{x}}_\lambda ,\lambda) / \mathrm{d}\lambda^k$  . For  $0\leq t < s\leq T,r\in [0,1]$  , let  $h\coloneqq \lambda_t - \lambda_s$ $\lambda \coloneqq \lambda_s + rh$  . Assuming the existence of total derivatives of  $\hat{\pmb{x}}_{\theta}^{(k)}(\hat{\pmb{x}}_{\lambda},\lambda),0\leq k\leq n$  , the  $n$  -th order Taylor expansion of  $\hat{\pmb{x}}_{\theta}(\hat{\pmb{x}}_{\lambda},\lambda)$  w.r.t. the half log-SNR  $\lambda$  is:
$$
\hat {\boldsymbol {x}} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda}, \lambda\right) = \sum_ {k = 0} ^ {n} \frac {r ^ {k} h ^ {k}}{k !} \hat {\boldsymbol {x}} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s}}, \lambda_ {s}\right) + \mathcal {O} \left(h ^ {n + 1}\right). \tag {27}
$$
Then, the exponential integrator of (9) can be reduced to
$$
\begin{array}{l} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {\lambda} \hat {\boldsymbol {x}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda}, \lambda) d \lambda = \frac {\alpha_ {t}}{\sigma_ {t}} \sum_ {k = 0} ^ {n} h ^ {k + 1} \int_ {0} ^ {1} e ^ {\lambda - \lambda_ {t}} \frac {r ^ {k}}{k !} d r \hat {\boldsymbol {x}} _ {\theta} ^ {(k)} (\hat {\boldsymbol {x}} _ {\lambda_ {s}}, \lambda_ {s}) + \mathcal {O} (h ^ {n + 2}) \\ = \frac {\alpha_ {t}}{\sigma_ {t}} \sum_ {k = 0} ^ {n} h ^ {k + 1} \int_ {0} ^ {1} e ^ {(r - 1) h} \frac {r ^ {, k}}{k !} \mathrm {d} r \hat {\pmb {x}} _ {\theta} ^ {(k)} (\hat {\pmb {x}} _ {\lambda_ {s}}, \lambda_ {s}) + \mathcal {O} (h ^ {n + 2}) \\ := \frac {\alpha_ {t}}{\sigma_ {t}} \sum_ {k = 0} ^ {n} h ^ {k + 1} \psi_ {k + 1} (h) \hat {\boldsymbol {x}} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s}}, \lambda_ {s}\right) + \mathcal {O} \left(h ^ {n + 2}\right), \tag {28} \\ \end{array}
$$
where  $\psi_{k + 1}(h) = \int_0^1 e^{(r - 1)h}\frac{r^k}{k!}\mathrm{d}r$  can be computed via the recurrence relation by integration-by-parts formula:
$$
\psi_ {k + 1} (z) = \frac {1}{z} \int_ {0} ^ {1} \frac {r ^ {k}}{k !} \mathrm {d} e ^ {(r - 1) z} = \frac {1}{z} \left(\frac {1}{k !} - \int_ {0} ^ {1} e ^ {(r - 1) z} \frac {r ^ {k - 1}}{(k - 1) !} \mathrm {d} r\right) = \frac {1}{z} \left(\frac {1}{k !} - \psi_ {k} (z)\right),
$$
and  $\psi_0(z) = e^{-z}$  [16]. For example, the closed-forms of  $\psi_k(h)$  for  $k = 1,2,3$  are
$$
\psi_ {1} (h) = \frac {1 - e ^ {- h}}{h}, \quad \psi_ {2} (h) = \frac {h - 1 + e ^ {- h}}{h ^ {2}}, \quad \psi_ {3} (h) = \frac {h ^ {2} / 2 - h + 1 - e ^ {- h}}{h ^ {3}}.
$$
Proof. In the subsequence analysis, we prove the order of accuracy of UniC for the data prediction model and the result of UniP for the data prediction model follows similarly. Suppose  $\boldsymbol{x}_t$  is the
solution of the diffusion ODE (1). For the data prediction model, given  $\pmb{x}_r$ ,  $r \leq t_{i-1}$ , are correct, with a slight abuse of notation, define
$$
\begin{array}{l} \bar {\boldsymbol {x}} _ {t _ {i}} = \frac {\sigma_ {t _ {i}}}{\sigma_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} + \alpha_ {t _ {i}} (1 - e ^ {- h _ {i}}) \boldsymbol {x} _ {\theta} (\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}) + \alpha_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p - 1} \frac {c _ {m}}{r _ {m}} (\hat {\boldsymbol {x}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}) - \hat {\boldsymbol {x}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})) \\ + \alpha_ {t _ {i}} B (h _ {i}) \frac {c _ {p}}{r _ {p}} (\boldsymbol {x} _ {\theta} (\tilde {\boldsymbol {x}} _ {t _ {i}}, t _ {i}) - \hat {\boldsymbol {x}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})). \\ \end{array}
$$
Similar to (23), since Solver-p has order of accuracy  $p$  and  $B(h) = \mathcal{O}(h)$ , under Assumption D.9, we have  $\bar{\pmb{x}}_{t_i} = \frac{\sigma_{t_i}}{\sigma_{t_{i - 1}}} \pmb{x}_{t_{i - 1}} + \alpha_{t_i} (1 - e^{-h_i}) \pmb{x}_\theta (\pmb{x}_{t_{i - 1}}, t_{i - 1}) + \alpha_{t_i} B(h_i) \sum_{m = 1}^{p} \frac{c_m}{r_m} (\hat{\pmb{x}}_\theta (\hat{\pmb{x}}_{\lambda_{sm}}, \lambda_{sm}) - \hat{\pmb{x}}_\theta (\hat{\pmb{x}}_{\lambda_{t_i - 1}}, \lambda_{t_i - 1})) + \mathcal{O}(h^{p + 2})$ . Then, the local truncation error on  $t_i$  is given by  $|\pmb{x}_{t_i} - \bar{\pmb{x}}_{t_i}|$ . Further, similar to Definition E.1, we shall show that
$$
\max  _ {1 \leq i \leq M} | \boldsymbol {x} _ {t _ {i}} - \bar {\boldsymbol {x}} _ {t _ {i}} | = \mathcal {O} (h ^ {p + 2}), \quad h = \max  _ {1 \leq i \leq M} h _ {i}.
$$
Combining (9) and (28), we have
$$
\boldsymbol {x} _ {t _ {i}} = \frac {\sigma_ {t _ {i}}}{\sigma_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} + \alpha_ {t _ {i}} (1 - e ^ {- h _ {i}}) \boldsymbol {\epsilon} _ {\theta} (\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}) + \alpha_ {t _ {i}} \sum_ {k = 1} ^ {p} h _ {i} ^ {k + 1} \psi_ {k + 1} (h _ {i}) \hat {\boldsymbol {x}} _ {\theta} ^ {(k)} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) + \mathcal {O} (h ^ {p + 2}).
$$
Recall that  $\pmb{R}_{p,k}^{\top}$  is the  $k$ -th row of  $\pmb{R}_p$ . By (13), we have  $|c_p^\top R_{p,k}(h_i) - B^{-1}(h_i)h_i^k k!\psi_{k + 1}(h_i)|\leq CB^{-1}(h_i)h^{p + 1}$  for some constant  $C > 0$ . Under Assumption E.3, by Taylor expansion, we obtain
$$
\begin{array}{l} \sum_ {m = 1} ^ {p} \frac {c _ {m}}{r _ {m}} \left(\hat {\boldsymbol {x}} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}\right) - \hat {\boldsymbol {x}} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right)\right) \\ = \sum_ {n = 1} ^ {p} \sum_ {m = 1} ^ {p} c _ {m} \frac {r _ {m} ^ {n - 1} h _ {i} ^ {n}}{n !} \hat {\boldsymbol {x}} _ {\theta} ^ {(n)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) + \mathcal {O} \left(h ^ {p + 1}\right) \\ = \sum_ {n = 1} ^ {p} \frac {h _ {i}}{n !} \boldsymbol {c} _ {p} ^ {\top} \boldsymbol {R} _ {p, n} (h _ {i}) \hat {\boldsymbol {x}} _ {\theta} ^ {(n)} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) + \mathcal {O} (h ^ {p + 1}). \\ \end{array}
$$
Thus, we have  $\left|\sum_{m=1}^{p}\frac{c_m}{r_m}(\hat{\boldsymbol{x}}_\theta(\hat{\boldsymbol{x}}_{\lambda_{s_m}},\lambda_{s_m}) - \hat{\boldsymbol{x}}_\theta(\hat{\boldsymbol{x}}_{\lambda_{t_{i-1}}},\lambda_{t_{i-1}})) - B(h_i)^{-1}\sum_{k=1}^{p}h_i^{k+1}\psi_{k+1}(h_i)\hat{\boldsymbol{x}}_\theta^{(k)}(\hat{\boldsymbol{x}}_{\lambda_{t_{i-1}}},\lambda_{t_{i-1}})\right| \leq C_1(B(h_i)^{-1}h^{p+2} + h^{p+1})$ . Given  $0 \neq B(h_i) = \mathcal{O}(h_i)$ , we have
$$
\begin{array}{l} \max  _ {1 \leq i \leq M} | \boldsymbol {x} _ {t _ {i}} - \bar {\boldsymbol {x}} _ {t _ {i}} | = \left| \alpha_ {t _ {i}} \sum_ {k = 1} ^ {p} h _ {i} ^ {k + 1} \psi_ {k + 1} (h _ {i}) \hat {\boldsymbol {x}} _ {\theta} ^ {(k)} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) \right. \\ \left. - \alpha_ {t _ {i}} B (h _ {i}) \sum_ {m = 1} ^ {p} \frac {c _ {m}}{r _ {m}} \left(\hat {\boldsymbol {x}} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}\right) - \hat {\boldsymbol {x}} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right)\right) \right| + \mathcal {O} \left(h ^ {p + 2}\right) \\ = \mathcal {O} \left(h ^ {p + 2}\right). \\ \end{array}
$$
Therefore, UniC- $p$  for the data prediction model is of  $(p + 1)$ -th order of accuracy.
E.5 Proof of Theorem C.1
We first give the local truncation error of  $\mathrm{UniPC}_v$ . Let  $A_{m,n}$  denote the element of  $\mathbf{A}_p$  on row  $m$  and column  $n$ . Given  $\mathbf{x}_r$ ,  $r \leq t_{i-1}$ , are correct, with a slight abuse of notation, let
$$
\begin{array}{l} \tilde {\boldsymbol {x}} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) \epsilon_ {\theta} \left(\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}\right) \\ - \sigma_ {t _ {i}} \sum_ {n = 1} ^ {p} h _ {i} \varphi_ {n + 1} (h _ {i}) \sum_ {m = 1} ^ {p - 1} A _ {m, n} (\hat {\boldsymbol {\epsilon}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}) - \hat {\boldsymbol {\epsilon}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})) / r _ {m} \\ - \sigma_ {t _ {i}} \sum_ {n = 1} ^ {p} h _ {i} \varphi_ {n + 1} (h _ {i}) A _ {p, n} (\boldsymbol {\epsilon} _ {\theta} (\tilde {\boldsymbol {x}} _ {t _ {i}}, t _ {i}) - \hat {\boldsymbol {\epsilon}} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})) / r _ {p}. \\ \end{array}
$$
Similar to (23), we have
$$
\begin{array}{l} \boldsymbol {\check {x}} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} (e ^ {h _ {i}} - 1) \epsilon_ {\theta} (\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}) \\ - \sigma_ {t _ {i}} \sum_ {n = 1} ^ {p} h _ {i} \varphi_ {n + 1} (h _ {i}) \sum_ {m = 1} ^ {p} A _ {m, n} \left(\hat {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}\right) - \hat {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right)\right) / r _ {m} + \mathcal {O} \left(h ^ {p + 2}\right). \tag {29} \\ \end{array}
$$
Suppose  $\pmb{x}_t$  is the solution of the diffusion ODE (1) and the local truncation error on  $t_i$  is given by  $|\pmb{x}_{t_i} - \tilde{\pmb{x}}_{t_i}|$ . Further, we shall show that the order of accuracy of  $\mathrm{UniPC}_v - p$  is  $p + 1$ , i.e.,
$$
\max  _ {1 \leq i \leq M} | \boldsymbol {x} _ {t _ {i}} - \check {\boldsymbol {x}} _ {t _ {i}} | = \mathcal {O} (h ^ {p + 2}), \quad h = \max  _ {1 \leq i \leq M} h _ {i}.
$$
By (24), we have
$$
\boldsymbol {x} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) \boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}\right) - \sigma_ {t _ {i}} \sum_ {k = 1} ^ {p} h _ {i} ^ {k + 1} \varphi_ {k + 1} \left(h _ {i}\right) \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(k)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) + \mathcal {O} \left(h ^ {p + 2}\right). \tag {30}
$$
Under Assumption E.2, by Taylor expansion, we obtain
$$
\begin{array}{l} \sum_ {k = 1} ^ {p} h _ {i} \varphi_ {k + 1} (h _ {i}) \sum_ {m = 1} ^ {p} \frac {A _ {m , k}}{r _ {m}} (\hat {\epsilon} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}) - \hat {\epsilon} _ {\theta} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}})) \\ = \sum_ {k = 1} ^ {p} h _ {i} \varphi_ {k + 1} (h _ {i}) \sum_ {m = 1} ^ {p} A _ {m, k} \sum_ {n = 1} ^ {p} \frac {r _ {m} ^ {n - 1} h _ {i} ^ {n}}{n !} \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(n)} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) + \mathcal {O} (h ^ {p + 2}) \\ = \sum_ {n = 1} ^ {p} \sum_ {k = 1} ^ {p} \varphi_ {k + 1} \left(h _ {i}\right) \sum_ {m = 1} ^ {p} A _ {m, k} \frac {r _ {m} ^ {n - 1} h _ {i} ^ {n + 1}}{n !} \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(n)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) + \mathcal {O} \left(h ^ {p + 2}\right) \\ = \sum_ {n = 1} ^ {p} \sum_ {k = 1} ^ {p} \varphi_ {k + 1} \left(h _ {i}\right) h _ {i} ^ {n + 1} \left(\mathbf {1} (k = n) + \mathcal {O} \left(h ^ {p}\right)\right) \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(n)} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right) + \mathcal {O} \left(h ^ {p + 2}\right) \\ = \sum_ {n = 1} ^ {p} h _ {i} ^ {n + 1} \varphi_ {n + 1} (h _ {i}) \hat {\boldsymbol {\epsilon}} _ {\theta} ^ {(n)} (\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}) + \mathcal {O} (h ^ {p + 2}). \\ \end{array}
$$
$$
\max  _ {1 \leq i \leq M} | \boldsymbol {x} _ {t _ {i}} - \check {\boldsymbol {x}} _ {t _ {i}} | = \mathcal {O} (h ^ {p + 2}).
$$
Therefore,  $\mathrm{UniPC}_v - p$  is of  $(p + 1)$ -th order of accuracy.
E.6 Proof of Proposition D.5
For UniP-  $p$  , we define
$$
\bar {\boldsymbol {x}} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \boldsymbol {x} _ {t _ {i - 1}} - \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) \boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}\right) - \sigma_ {t _ {i}} B \left(h _ {i}\right) \sum_ {m = 1} ^ {p - 1} \frac {a _ {m}}{r _ {m}} \left(\hat {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s _ {m}}}, \lambda_ {s _ {m}}\right) - \hat {\boldsymbol {\epsilon}} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {t _ {i - 1}}}, \lambda_ {t _ {i - 1}}\right)\right). \tag {31}
$$
By Assumption D.2, we have  $|\epsilon_{\theta}(\tilde{\boldsymbol{x}}_s,s) - \epsilon_{\theta}(\boldsymbol{x}_s,s)| \leq L|\tilde{\boldsymbol{x}}_s - \boldsymbol{x}_s|$ . Therefore, for sufficiently large constants  $C, C_1 > 0$  depending on  $\{a_m\}$  and  $\{r_m\}$ , we have
$$
\left| \tilde {\boldsymbol {x}} _ {t _ {i}} - \bar {\boldsymbol {x}} _ {t _ {i}} \right| \leq \left(\frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} + L \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) + C L p h\right) \left| \tilde {\boldsymbol {x}} _ {t _ {i - 1}} - \boldsymbol {x} _ {t _ {i - 1}} \right| + C L h \sum_ {m = 1} ^ {p - 1} \left| \tilde {\boldsymbol {x}} _ {t _ {i - m - 1}} - \boldsymbol {x} _ {t _ {i - m - 1}} \right|. \tag {32}
$$
For simplicity, define  $e_i = |\tilde{\boldsymbol{x}}_{t_i} - \boldsymbol{x}_{t_i}|$ ,  $f_n = \max_{0\leq i\leq n}|e_i|$ . Using Theorem 3.1 and (32), we obtain
$$
e _ {i} \leq \left(\frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} + L \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) + C L p h\right) e _ {i - 1} + C L h \sum_ {m = 1} ^ {p - 1} e _ {i - 1 - m} + C _ {0} h ^ {p + 1}. \tag {33}
$$
Let  $\beta_{i} := \frac{\alpha_{t_{i}}}{\alpha_{t_{i-1}}} + L\sigma_{t_{i}}(e^{h_{i}} - 1) + CLph$ . Then, it follows that
$$
e _ {i} \leq \left(C L p h + \beta_ {i}\right) f _ {i - 1} + C _ {0} h ^ {p + 1}.
$$
Since  $\beta_{i} + CLph > 1$  for sufficiently small  $h$ , the right hand side is also a trivial bound for  $f_{i-1}$  because  $\alpha_{t} > 0$  is monotone decreasing thus  $\alpha_{t_{i}} < \alpha_{t_{i-1}}$ . We then have
$$
f _ {i} \leq \left(C L p h + \beta_ {i}\right) f _ {i - 1} + C _ {0} h ^ {p + 1}. \tag {34}
$$
Let  $\sigma \coloneqq \max_{1\leq i\leq M}\sigma_{t_i} + 2Cp$ . By elementary calculation, under Assumption D.3 we have
$$
\prod_ {i = p} ^ {M} \left(\beta_ {i} + C L p h\right) \leq C _ {1} \prod_ {i = p} ^ {M} \left(\alpha_ {t _ {i}} / \alpha_ {t _ {i - 1}} + L \sigma / M\right) = C _ {1} \frac {\alpha_ {t _ {M}}}{\alpha_ {t _ {p - 1}}} \prod_ {i = p} ^ {M} \left(1 + \frac {L \sigma \alpha_ {t _ {i - 1}}}{\alpha_ {t _ {i}} M}\right) \leq C _ {2} e ^ {b \sigma}, \tag {35}
$$
where  $C_1, C_2$  are sufficiently large constants. Under Assumption D.4 with  $k = p$ ,  $f_{p - 1} = \max_{0\leq i\leq p - 1}|\tilde{\boldsymbol{x}}_{t_i} - \boldsymbol{x}_{t_i}|\leq c_0h^p$ . Repeat the argument of (34), under Assumption D.3 by (35), there exists constant  $C_3, C_4 > 0$  such that
$$
f _ {M} \leq C _ {2} e ^ {b \sigma} f _ {p - 1} + C _ {3} M h ^ {p + 1} \leq C _ {4} h ^ {p}. \tag {36}
$$
Therefore, the convergence order of  $\mathrm{UniP - p}$  is  $p$
E.7 Proof of Proposition D.6
For the sake of clarity, we use  $\bar{\pmb{x}}_{t_i}^c = \frac{\alpha_{t_i}}{\alpha_{t_{i-1}}} \pmb{x}_{t_{i-1}} - \sigma_{t_i}(e^{h_i} - 1) \pmb{\epsilon}_\theta(\pmb{x}_{t_{i-1}}, t_{i-1}) - \sigma_{t_i} B(h_i) \sum_{m=1}^p \frac{a_m}{r_m} (\hat{\epsilon}_\theta(\hat{\pmb{x}}_{\lambda_{s_m}}, \lambda_{s_m}) - \hat{\epsilon}_\theta(\hat{\pmb{x}}_{\lambda_{t_{i-1}}}, \lambda_{t_{i-1}}))$  for UniC- $p$  and use  $\bar{\pmb{x}}_{t_i}$  of (31) for UniP- $p$ . Similar to (32), we have for  $i \geq p$ ,
$$
\begin{array}{l} | \tilde {\pmb {x}} _ {t _ {i}} ^ {c} - \bar {\pmb {x}} _ {t _ {i}} ^ {c} | \leq \left(\frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} + L \sigma_ {t _ {i}} (e ^ {h _ {i}} - 1) + C L p h\right) | \tilde {\pmb {x}} _ {t _ {i - 1}} ^ {c} - \pmb {x} _ {t _ {i - 1}} | \\ + C L h \sum_ {m = 1} ^ {p - 1} \left| \tilde {\boldsymbol {x}} _ {t _ {i - m - 1}} ^ {c} - \boldsymbol {x} _ {t _ {i - m - 1}} \right| + L h C _ {1} \left| \tilde {\boldsymbol {x}} _ {t _ {i}} - \boldsymbol {x} _ {t _ {i}} \right|, \tag {37} \\ \end{array}
$$
where for the oracle UniPC (see Section 4.2 for definition), the UniP-  $p$  admits
$$
\begin{array}{l} \left| \tilde {\boldsymbol {x}} _ {t _ {i}} - \bar {\boldsymbol {x}} _ {t _ {i}} \right| \leq \left(\frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} + L \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) + C L p h\right) \left| \tilde {\boldsymbol {x}} _ {t _ {i - 1}} ^ {c} - \boldsymbol {x} _ {t _ {i - 1}} \right| \\ + C L h \sum_ {m = 1} ^ {p - 1} \left| \tilde {\boldsymbol {x}} _ {t _ {i - m - 1}} ^ {c} - \boldsymbol {x} _ {t _ {i - m - 1}} \right|. \tag {38} \\ \end{array}
$$
Define  $e_i^c = |\tilde{\pmb{x}}_{t_i}^c -\pmb{x}_{t_i}|$ ,  $f_{n}^{c} = \max_{0\leq i\leq n}|e_{i}^{c}|$ . Write  $\beta_{i}\coloneqq \frac{\alpha_{t_{i}}}{\alpha_{t_{i - 1}}} +L\sigma_{t_{i}}(e^{h_{i}} - 1) + CLph$ . Let  $\sigma \coloneqq \max_{1\leq i\leq M}\sigma_{t_i} + 2Cp$ . As shown in the proof of Theorem 3.1,  $|\pmb{x}_{t_i} - \bar{\pmb{x}}_{t_i}^c | = \mathcal{O}(h^{p + 2})$ . Combining the local truncation error, it follows that
$$
\begin{array}{l} f _ {i} ^ {c} \leq \beta_ {i} f _ {i - 1} ^ {c} + C L h p f _ {i - 1} ^ {c} + L h C _ {1} \left(\beta_ {i} f _ {i - 1} ^ {c} + C L h p f _ {i - 1} ^ {c} + C _ {2} h ^ {p + 1}\right) + C _ {0} h ^ {p + 2} \\ \leq \left(1 + L h C _ {1}\right) \left(\beta_ {i} + C L h p\right) f _ {i - 1} ^ {c} + C _ {2} L C _ {1} h ^ {p + 2} + C _ {0} h ^ {p + 2}. \\ \end{array}
$$
Repeating this argument, similar to (36), we have
$$
\begin{array}{l} \left| \tilde {\boldsymbol {x}} _ {t _ {M}} ^ {c} - \boldsymbol {x} _ {0} \right| \leq f _ {M} ^ {c} \leq \prod_ {i = p} ^ {M} (1 + L h C _ {1}) (\beta_ {i} + C L h p) f _ {p - 1} ^ {c} + M C _ {2} h ^ {p + 2} \\ \leq C _ {3} e ^ {L C _ {1} + b \sigma} f _ {p - 1} ^ {c} + M C _ {2} h ^ {p + 2}. \tag {39} \\ \end{array}
$$
For  $0 \leq i < p$ , we have
$$
\begin{array}{l} \left| \tilde {\boldsymbol {x}} _ {t _ {i}} ^ {c} - \bar {\boldsymbol {x}} _ {t _ {i}} \right| \leq \left(\frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} + L \sigma_ {t _ {i}} \left(e ^ {h _ {i}} - 1\right) + C L p h\right) \left| \tilde {\boldsymbol {x}} _ {t _ {i - 1}} ^ {c} - \boldsymbol {x} _ {t _ {i - 1}} \right| \\ + C L h \sum_ {m = 1} ^ {i - 1} | \tilde {\boldsymbol {x}} _ {t _ {i - m - 1}} - \boldsymbol {x} _ {t _ {i - m - 1}} | + L h C _ {1} | \tilde {\boldsymbol {x}} _ {t _ {i}} - \boldsymbol {x} _ {t _ {i}} |. \\ \end{array}
$$
Therefore, under Assumption D.4 with  $k = p$ , we have for  $0 \leq i \leq p - 1$
$$
f _ {i} ^ {c} \leq \beta_ {i} f _ {i - 1} ^ {c} + c _ {0} C L p h ^ {p + 1} + L h ^ {p + 1} c _ {0} C _ {1}.
$$
Repeat this argument, we find for a constant  $C_4 > 0$
$$
f _ {p - 1} ^ {c} \leq C _ {4} h ^ {p + 1}. \tag {40}
$$
Combining (39) and (40), we have
$$
| \tilde {\pmb {x}} _ {t _ {M}} ^ {c} - \pmb {x} _ {0} | = \mathcal {O} (h ^ {p + 1}).
$$
For the non-oracle solver, for the step of UniP-  $p$  we have :