SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Comparison between DPM-Solver-v3 and other high-order diffusion ODE solvers based on exponential integrators.
DEIS [56]	DPM-Solver [31]	DPM-Solver++ [32]	UniPC [58]	DPM-Solver-v3 (Ours)
First-Order	DDIM	DDIM	DDIM	DDIM	Improved DDIM
Taylor Expanded Predictor	εθfor t	εθfor λ	xθfor λ	xθfor λ	gθfor λ
Solver Type (High-Order)	Multistep	Singlestep	Multistep	Multistep	Multistep
Applicable for Guided Sampling	✓	X	✓	✓	✓
Corrector Supported	X	X	X	✓	✓
Model-Specific	X	X	X	X	✓
In this section, we make some theoretical comparisons between DPM-Solver-v3 and existing diffusion ODE solvers that are based on exponential integrators [56, 31, 32, 58]. We summarize the results in Table 2, and provide some analysis below.
Previous ODE formulation as special cases of ours. First, we compare the formulation of the ODE solution. Our reformulated ODE solution in Eq. (9) involves extra coefficients  $l_{\lambda}, s_{\lambda}, b_{\lambda}$ , and it corresponds to a new predictor  $g_{\theta}$  to be approximated by Taylor expansion. By comparing ours with previous ODE formulations in Eq. (3) and their corresponding noise/data prediction, we can easily figure out that they are special cases of ours by setting  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  to specific values:
- Noise prediction:  $l_{\lambda} = 0, s_{\lambda} = -1, b_{\lambda} = 0$
- Data prediction:  $l_{\lambda} = 1, s_{\lambda} = 0, b_{\lambda} = 0$
It's worth noting that, though our ODE formulation can degenerate to previous ones, it may still result in different solvers, since previous works conduct some equivalent substitution of the same order in the local approximation (for example, the choice of  $B_{1}(h) = h$  or  $B_{2}(h) = e^{h} - 1$  in UniPC [58]). We never conduct such substitution, thus saving the efforts to tune it.
Moreover, under our framework, we find that DPM-Solver++ is a model-agnostic approximation of DPM-Solver-v3, under the Gaussian assumption. Specifically, according to Eq. (5), we have
$$
\boldsymbol {l} _ {\lambda} ^ {*} = \underset {\boldsymbol {l} _ {\lambda}} {\operatorname {a r g m i n}} \mathbb {E} _ {p _ {\lambda} ^ {\theta} (\boldsymbol {x} _ {\lambda})} \| \sigma_ {\lambda} \nabla_ {\boldsymbol {x}} \boldsymbol {\epsilon} _ {\theta} (\boldsymbol {x} _ {\lambda}, \lambda) - \boldsymbol {l} _ {\lambda} \| _ {F} ^ {2}, \tag {17}
$$
If we assume  $q_{\lambda}(\pmb{x}_{\lambda}) \approx \mathcal{N}(\pmb{x}_{\lambda}|\alpha_{\lambda}\pmb{x}_{0},\sigma_{\lambda}^{2}\pmb{I})$  for some fixed  $\pmb{x}_0$ , then the optimal noise predictor is
$$
\boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {x} _ {\lambda}, \lambda\right) \approx - \sigma_ {\lambda} \nabla_ {\boldsymbol {x}} \log q _ {\lambda} \left(\boldsymbol {x} _ {\lambda}\right) = \frac {\boldsymbol {x} _ {\lambda} - \alpha_ {t} \boldsymbol {x} _ {0}}{\sigma_ {\lambda}}. \tag {18}
$$
It follows that  $\sigma_{\lambda}\nabla_{\boldsymbol{x}}\epsilon_{\theta}(\boldsymbol{x}_{\lambda},\lambda)\approx \boldsymbol{I}$ , thus  $l_{\lambda}^{*}\approx 1$  by Eq. (5), which corresponds the data prediction model used in DPM-Solver++. Moreover, for small enough  $\lambda$  (i.e.,  $t$  near to  $T$ ), the Gaussian assumption is almost true (see Section 4.2), thus the data-prediction DPM-Solver++ approximately computes all the linear terms at the initial stage. To the best of our knowledge, this is the first explanation for the reason why the data-prediction DPM-Solver++ outperforms the noise-prediction DPM-Solver.
First-order discretization as improved DDIM Previous methods merely use noise/data parameterization, whether or not they change the time domain from  $t$  to  $\lambda$ . While they differ in high-order cases, they are proven to coincide in the first-order case, which is DDIM [48] (deterministic case,  $\eta = 0$ ):
$$
\hat {\boldsymbol {x}} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \hat {\boldsymbol {x}} _ {s} - \alpha_ {t} \left(\frac {\sigma_ {s}}{\alpha_ {s}} - \frac {\sigma_ {t}}{\alpha_ {t}}\right) \boldsymbol {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {s}, \lambda_ {s}\right) \tag {19}
$$
However, the first-order case of our method is
$$
\begin{array}{l} \hat {\boldsymbol {x}} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \left(\left(1 + l _ {\lambda_ {s}} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \mathrm {d} \lambda\right) \hat {\boldsymbol {x}} _ {s} - \left(\sigma_ {s} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \mathrm {d} \lambda\right) \boldsymbol {\epsilon} _ {\theta} \left(\hat {\boldsymbol {x}} _ {s}, \lambda_ {s}\right)\right) \tag {20} \\ - \alpha_ {t} \pmb {A} (\lambda_ {s}, \lambda_ {t}) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \pmb {E} _ {\lambda_ {s}} (\lambda) \pmb {B} _ {\lambda_ {s}} (\lambda) \mathrm {d} \lambda \\ \end{array}
$$
which is not DDIM since we choose a better parameterization by the estimated EMS. Empirically, our first-order solver performs better than DDIM, as detailed in Appendix G.
B.1 Assumptions
In this section, we will give some mild conditions under which the local order of accuracy of Algorithm 1 and the global order of convergence of Algorithm 2 (predictor) are guaranteed.
B.1.1 Local
First, we will give the assumptions to bound the local truncation error.
Assumption B.1. The total derivatives of the noise prediction model  $\frac{\mathrm{d}^k\epsilon_\theta(x_\lambda,\lambda)}{\mathrm{d}\lambda^k}, k = 1,\dots ,n$  exist and are continuous.
Assumption B.2. The coefficients  $l_{\lambda}, s_{\lambda}, b_{\lambda}$  are continuous and bounded.  $\frac{\mathrm{d}^k l_{\lambda}}{\mathrm{d}\lambda^k}, \frac{\mathrm{d}^k s_{\lambda}}{\mathrm{d}\lambda^k}, \frac{\mathrm{d}^k b_{\lambda}}{\mathrm{d}\lambda^k}, k = 1, \ldots, n$  exist and are continuous.
Assumption B.3.  $\delta_{k} = \Theta (\lambda_{t} - \lambda_{s}),k = 1,\ldots ,n$
Assumption B.1 is required for the Taylor expansion which is regular in high-order numerical methods. Assumption B.2 requires the boundness of the coefficients as well as regularizes the coefficients' smoothness to enable the Taylor expansion for  $\pmb{g}_{\theta}(\pmb{x}_{\lambda}, \lambda)$ , which holds in practice given the smoothness of  $\pmb{\epsilon}_{\theta}(\pmb{x}_{\lambda}, \lambda)$  and  $p_{\lambda}^{\theta}(\pmb{x}_{\lambda})$ . Assumption B.3 makes sure  $\delta_{k}$  and  $\lambda_{t} - \lambda_{s}$  is of the same order, i.e., there exists some constant  $r_k = \mathcal{O}(1)$  so that  $\delta_{k} = r_{k}(\lambda_{t} - \lambda_{s})$ , which is satisfied in regular multistep methods.
B.1.2 Global
Then we will give the assumptions to bound the global error.
Assumption B.4. The noise prediction model  $\epsilon_{\theta}(\pmb{x},t)$  is Lipschitz w.r.t. to  $\pmb{x}$ .
Assumption B.5.  $h = \max_{1\leq i\leq M}(\lambda_i - \lambda_{i - 1}) = \mathcal{O}(1 / M)$
Assumption B.6. The starting values  $\hat{\pmb{x}}_i, 1 \leq i \leq n$  satisfies  $\hat{\pmb{x}}_i - \pmb{x}_i = \mathcal{O}(h^{n+1})$ .
Assumption B.4 is common in the analysis of ODEs, which assures  $\epsilon_{\theta}(\hat{\boldsymbol{x}}_t,t) - \epsilon_{\theta}(\boldsymbol{x}_t,t) = \mathcal{O}(\hat{\boldsymbol{x}}_t - \boldsymbol{x}_t)$ . Assumption B.5 implies that the step sizes are rather uniform. Assumption B.6 is common in the convergence analysis of multistep methods [5].
B.2 Order of Accuracy and Convergence
In this section, we prove the local and global order guarantees detailed in Theorem 3.1 and Theorem 3.3.
B.2.1 Local
Proof. (Proof of Theorem 3.1) Denote  $h \coloneqq \lambda_t - \lambda_s$ . Subtracting the Taylor-expanded exact solution in Eq. (12) from the local approximation in Eq. (14), we have
$$
\hat {\boldsymbol {x}} _ {t} - \boldsymbol {x} _ {t} = - \alpha_ {t} \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \sum_ {k = 1} ^ {n} \frac {\hat {\boldsymbol {g}} _ {\theta} ^ {(k)} \left(\boldsymbol {x} _ {\lambda_ {s}} , \lambda_ {s}\right) - \boldsymbol {g} _ {\theta} ^ {(k)} \left(\boldsymbol {x} _ {\lambda_ {s}} , \lambda_ {s}\right)}{k !} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \left(\lambda - \lambda_ {s}\right) ^ {k} \mathrm {d} \lambda + \mathcal {O} \left(h ^ {n + 2}\right) \tag {21}
$$
First we examine the order of  $\mathbf{A}(\lambda_s, \lambda_t)$  and  $\int_{\lambda_s}^{\lambda_t} \mathbf{E}_{\lambda_s}(\lambda)(\lambda - \lambda_s)^k \, \mathrm{d}\lambda$ . Under Assumption B.2, there exists some constant  $C_1, C_2$  such that  $-l_\lambda < C_1, l_\lambda + s_\lambda < C_2$ . So
$$
\begin{array}{l} \boldsymbol {A} (\lambda_ {s}, \lambda_ {t}) = e ^ {- \int_ {\lambda_ {s}} ^ {\lambda_ {t}} l _ {\tau} \mathrm {d} \tau} \\ <   e ^ {C _ {1} h} \tag {22} \\ = \mathcal {O} (1) \\ \end{array}
$$
$$
\begin{array}{l} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) (\lambda - \lambda_ {s}) ^ {k} \mathrm {d} \lambda = \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {\int_ {\lambda_ {s}} ^ {\lambda} (l _ {\tau} + s _ {\tau}) \mathrm {d} \tau} (\lambda - \lambda_ {s}) ^ {k} \mathrm {d} \lambda \\ <   \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {C _ {2} (\lambda - \lambda_ {s})} (\lambda - \lambda_ {s}) ^ {k} \mathrm {d} \lambda \tag {23} \\ = \mathcal {O} \left(h ^ {k + 1}\right) \\ \end{array}
$$
Next we examine the order of  $\frac{\hat{g}_{\theta}^{(k)}(\boldsymbol{x}_{\lambda_s},\lambda_s) - \boldsymbol{g}_{\theta}^{(k)}(\boldsymbol{x}_{\lambda_s},\lambda_s)}{k!}$ . Under Assumption B.1 and Assumption B.2, since  $\boldsymbol{g}_{\theta}$  is elementary function of  $\epsilon_{\theta}$  and  $l_{\lambda}, s_{\lambda}, b_{\lambda}$ , we know  $\boldsymbol{g}_{\theta}^{(k)}(\boldsymbol{x}_{\lambda_s},\lambda_s), k = 1,\dots,n$  exist and are continuous. Adopting the notations in Eq. (13), by Taylor expansion, we have
$$
\begin{array}{l} \boldsymbol {g} _ {i _ {1}} = \boldsymbol {g} _ {s} + \delta_ {1} \boldsymbol {g} _ {s} ^ {(1)} + \delta_ {1} ^ {2} \boldsymbol {g} _ {s} ^ {(2)} + \dots + \delta_ {1} ^ {n} \boldsymbol {g} _ {s} ^ {(n)} + \mathcal {O} \left(\delta_ {1} ^ {n + 1}\right) \\ \boldsymbol {g} _ {i _ {2}} = \boldsymbol {g} _ {s} + \delta_ {2} \boldsymbol {g} _ {s} ^ {(1)} + \delta_ {2} ^ {2} \boldsymbol {g} _ {s} ^ {(2)} + \dots + \delta_ {2} ^ {n} \boldsymbol {g} _ {s} ^ {(n)} + \mathcal {O} \left(\delta_ {2} ^ {n + 1}\right) \tag {24} \\ \end{array}
$$
$$
\boldsymbol {g} _ {i _ {n}} = \boldsymbol {g} _ {s} + \delta_ {n} \boldsymbol {g} _ {s} ^ {(1)} + \delta_ {n} ^ {2} \boldsymbol {g} _ {s} ^ {(2)} + \dots + \delta_ {n} ^ {n} \boldsymbol {g} _ {s} ^ {(n)} + \mathcal {O} \left(\delta_ {n} ^ {n + 1}\right)
$$
Comparing it with Eq. (13), we have
$$
\left( \begin{array}{c c c c} \delta_ {1} & \delta_ {1} ^ {2} & \dots & \delta_ {1} ^ {n} \\ \delta_ {2} & \delta_ {2} ^ {2} & \dots & \delta_ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ \delta_ {n} & \delta_ {n} ^ {2} & \dots & \delta_ {n} ^ {n} \end{array} \right) \left( \begin{array}{c} \hat {\boldsymbol {g}} _ {s} ^ {(1)} - \boldsymbol {g} _ {s} ^ {(1)} \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(2)} - \boldsymbol {g} _ {s} ^ {(2)}}{2 !} \\ \vdots \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(n)} - \boldsymbol {g} _ {s} ^ {(n)}}{n !} \end{array} \right) = \left( \begin{array}{c} \mathcal {O} (\delta_ {1} ^ {n + 1}) \\ \mathcal {O} (\delta_ {2} ^ {n + 1}) \\ \vdots \\ \mathcal {O} (\delta_ {n} ^ {n + 1}) \end{array} \right) \tag {25}
$$
From Assumption B.3, we know there exists some constants  $r_k$  so that  $\delta_k = r_k h, k = 1, \dots, n$ . Thus
$$
\left( \begin{array}{c c c c} \delta_ {1} & \delta_ {1} ^ {2} & \dots & \delta_ {1} ^ {n} \\ \delta_ {2} & \delta_ {2} ^ {2} & \dots & \delta_ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ \delta_ {n} & \delta_ {n} ^ {2} & \dots & \delta_ {n} ^ {n} \end{array} \right) = \left( \begin{array}{c c c c} r _ {1} & r _ {1} ^ {2} & \dots & r _ {1} ^ {n} \\ r _ {2} & r _ {2} ^ {2} & \dots & r _ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ r _ {n} & r _ {n} ^ {2} & \dots & r _ {n} ^ {n} \end{array} \right) \left( \begin{array}{c c c c} h & & & \\ & h ^ {2} & & \\ & & \ddots & \\ & & & h ^ {n} \end{array} \right), \left( \begin{array}{c} \mathcal {O} \left(\delta_ {1} ^ {n + 1}\right) \\ \mathcal {O} \left(\delta_ {2} ^ {n + 1}\right) \\ \vdots \\ \mathcal {O} \left(\delta_ {n} ^ {n + 1}\right) \end{array} \right) = \binom {\mathcal {O} \left(h ^ {n + 1}\right)} {\mathcal {O} \left(h ^ {n + 1}\right)} (2 6)
$$
And finally, we have
$$
\begin{array}{l} \left( \begin{array}{c} \hat {\boldsymbol {g}} _ {s} ^ {(1)} - \boldsymbol {g} _ {s} ^ {(1)} \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(2)} - \boldsymbol {g} _ {s} ^ {(2)}}{2 !} \\ \vdots \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(n)} - \boldsymbol {g} _ {s} ^ {(n)}}{n !} \end{array} \right) = \left( \begin{array}{c c c c} h ^ {- 1} & & & \\ & h ^ {- 2} & & \\ & & \ddots & \\ & & & h ^ {- n} \end{array} \right) \left( \begin{array}{c c c c} r _ {1} & r _ {1} ^ {2} & \dots & r _ {1} ^ {n} \\ r _ {2} & r _ {2} ^ {2} & \dots & r _ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ r _ {n} & r _ {n} ^ {2} & \dots & r _ {n} ^ {n} \end{array} \right) ^ {- 1} \left( \begin{array}{c} \mathcal {O} (h ^ {n + 1}) \\ \mathcal {O} (h ^ {n + 1}) \\ \vdots \\ \mathcal {O} (h ^ {n + 1}) \end{array} \right) \tag {27} \\ = \left( \begin{array}{c} \mathcal {O} (h ^ {n}) \\ \mathcal {O} (h ^ {n - 1}) \\ \vdots \\ \mathcal {O} (h ^ {1}) \end{array} \right) \\ \end{array}
$$
Substitute Eq. (22), Eq. (23) and Eq. (27) into Eq. (21), we can conclude that  $\hat{\pmb{x}}_t - \pmb{x}_t = \mathcal{O}(h^{n + 2})$ .
B.2.2 Global
First, we provide a lemma that gives the local truncation error given inexact previous values when estimating the high-order derivatives.
Lemma B.7. (Local truncation error with inexact previous values) Suppose inexact values  $\hat{\pmb{x}}_{\lambda_{i_k}}, k = 1, \dots, n$  and  $\hat{\pmb{x}}_s$  are used in Eq. (13) to estimate the high-order derivatives, then the local truncation error of the local approximation Eq. (14) satisfies
$$
\Delta_ {t} = \frac {\alpha_ {t} \boldsymbol {A} \left(\lambda_ {s} , \lambda_ {t}\right)}{\alpha_ {s}} \Delta_ {s} + \mathcal {O} (h) \left(\mathcal {O} \left(\Delta_ {s}\right) + \sum_ {k = 1} ^ {n} \mathcal {O} \left(\Delta_ {\lambda_ {i _ {k}}}\right) + \mathcal {O} \left(h ^ {n + 1}\right)\right) \tag {28}
$$
where  $\Delta_{\cdot}\coloneqq \hat{\pmb{x}}_{\cdot} - \pmb {x}_{\cdot},h\coloneqq \lambda_{t} - \lambda_{s}$
Proof. By replacing  $\pmb{x}$  with  $\hat{\pmb{x}}$  in Eq. (13) and subtracting Eq. (12) from Eq. (14), the expression for the local truncation error becomes
$$
\begin{array}{l} \Delta_ {t} = \frac {\alpha_ {t} \boldsymbol {A} \left(\lambda_ {s} , \lambda_ {t}\right)}{\alpha_ {s}} \Delta_ {s} - \alpha_ {t} \boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) \left(\boldsymbol {g} _ {\theta} \left(\hat {\boldsymbol {x}} _ {\lambda_ {s}}, \lambda_ {s}\right) - \boldsymbol {g} _ {\theta} \left(\boldsymbol {x} _ {\lambda_ {s}}, \lambda_ {s}\right)\right) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \mathrm {d} \lambda \tag {29} \\ - \alpha_ {t} \boldsymbol {A} (\lambda_ {s}, \lambda_ {t}) \sum_ {k = 1} ^ {n} \frac {\hat {\boldsymbol {g}} _ {\theta} ^ {(k)} (\boldsymbol {x} _ {\lambda_ {s}} , \lambda_ {s}) - \boldsymbol {g} _ {\theta} ^ {(k)} (\boldsymbol {x} _ {\lambda_ {s}} , \lambda_ {s})}{k !} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) (\lambda - \lambda_ {s}) ^ {k} \mathrm {d} \lambda + \mathcal {O} (h ^ {n + 2}) \\ \end{array}
$$
And the linear system for solving  $g_{\theta}^{(k)}(\pmb{x}_{\lambda_s}, \lambda_s), k = 1, \dots, n$  becomes
$$
\left( \begin{array}{c c c c} \delta_ {1} & \delta_ {1} ^ {2} & \dots & \delta_ {1} ^ {n} \\ \delta_ {2} & \delta_ {2} ^ {2} & \dots & \delta_ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ \delta_ {n} & \delta_ {n} ^ {2} & \dots & \delta_ {n} ^ {n} \end{array} \right) \left( \begin{array}{c} \hat {\boldsymbol {g}} _ {s} ^ {(1)} \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(2)}}{2 !} \\ \vdots \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(n)}}{n !} \end{array} \right) = \left( \begin{array}{c} \hat {\boldsymbol {g}} _ {i _ {1}} - \hat {\boldsymbol {g}} _ {s} \\ \hat {\boldsymbol {g}} _ {i _ {2}} - \hat {\boldsymbol {g}} _ {s} \\ \vdots \\ \hat {\boldsymbol {g}} _ {i _ {n}} - \hat {\boldsymbol {g}} _ {s} \end{array} \right) \tag {30}
$$
where  $\hat{\pmb{g}}_{\cdot} = \pmb{g}_{\theta}(\hat{\pmb{x}}_{\lambda},\lambda)$ . Under Assumption B.4, we know  $\hat{\pmb{g}}_{\cdot} - \pmb{g}_{\cdot} = \mathcal{O}(\Delta_{\lambda})$ . Thus, under Assumption B.1, Assumption B.2 and Assumption B.3, similar to the deduction in the last section, we have
$$
\left( \begin{array}{c} \hat {\boldsymbol {g}} _ {s} ^ {(1)} - \boldsymbol {g} _ {s} ^ {(1)} \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(2)} - \boldsymbol {g} _ {s} ^ {(2)}}{2 !} \\ \vdots \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(n)} - \boldsymbol {g} _ {s} ^ {(n)}}{n !} \end{array} \right) = \left( \begin{array}{c c c c} h ^ {- 1} & & & \\ & h ^ {- 2} & & \\ & & \ddots & \\ & & & h ^ {- n} \end{array} \right) \left( \begin{array}{c c c c} r _ {1} & r _ {1} ^ {2} & \dots & r _ {1} ^ {n} \\ r _ {2} & r _ {2} ^ {2} & \dots & r _ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ r _ {n} & r _ {n} ^ {2} & \dots & r _ {n} ^ {n} \end{array} \right) ^ {- 1} \left( \begin{array}{c} \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {1}}}) \\ \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {2}}}) \\ \vdots \\ \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {n}}}) \end{array} \right) \tag {31}
$$
Besides, under Assumption B.2, the orders of the other coefficients are the same as we obtain in the last section:
$$
\boldsymbol {A} \left(\lambda_ {s}, \lambda_ {t}\right) = \mathcal {O} (1), \quad \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) \left(\lambda - \lambda_ {s}\right) ^ {k} \mathrm {d} \lambda = \mathcal {O} \left(h ^ {k + 1}\right) \tag {32}
$$
$$
\begin{array}{l} \sum_ {k = 1} ^ {n} \frac {\hat {\pmb {g}} _ {\theta} ^ {(k)} (\pmb {x} _ {\lambda_ {s}} , \lambda_ {s}) - \pmb {g} _ {\theta} ^ {(k)} (\pmb {x} _ {\lambda_ {s}} , \lambda_ {s})}{k !} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \pmb {E} _ {\lambda_ {s}} (\lambda) (\lambda - \lambda_ {s}) ^ {k} \mathrm {d} \lambda \\ = \left( \begin{array}{c} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) (\lambda - \lambda_ {s}) ^ {1} \mathrm {d} \lambda \\ \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) (\lambda - \lambda_ {s}) ^ {2} \mathrm {d} \lambda \\ \vdots \\ \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \boldsymbol {E} _ {\lambda_ {s}} (\lambda) (\lambda - \lambda_ {s}) ^ {n} \mathrm {d} \lambda \end{array} \right) ^ {\top} \left( \begin{array}{c} \hat {\boldsymbol {g}} _ {s} ^ {(1)} - \boldsymbol {g} _ {s} ^ {(1)} \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(2)} - \boldsymbol {g} _ {s} ^ {(2)}}{2 !} \\ \vdots \\ \frac {\hat {\boldsymbol {g}} _ {s} ^ {(n)} - \boldsymbol {g} _ {s} ^ {(n)}}{n !} \end{array} \right) \\ = \left( \begin{array}{c} \mathcal {O} (h ^ {2}) \\ \mathcal {O} (h ^ {3}) \\ \vdots \\ \mathcal {O} (h ^ {n + 1}) \end{array} \right) ^ {\top} \left( \begin{array}{c c c c} h ^ {- 1} & & & \\ & h ^ {- 2} & & \\ & & \ddots & \\ & & & h ^ {- n} \end{array} \right) \left( \begin{array}{c c c c} r _ {1} & r _ {1} ^ {2} & \dots & r _ {1} ^ {n} \\ r _ {2} & r _ {2} ^ {2} & \dots & r _ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ r _ {n} & r _ {n} ^ {2} & \dots & r _ {n} ^ {n} \end{array} \right) ^ {- 1} \left( \begin{array}{c} \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {1}}}) \\ \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {2}}}) \\ \vdots \\ \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {n}}}) \end{array} \right) \\ = \left( \begin{array}{c} \mathcal {O} (h) \\ \mathcal {O} (h) \\ \vdots \\ \mathcal {O} (h) \end{array} \right) ^ {\top} \left( \begin{array}{c c c c} r _ {1} & r _ {1} ^ {2} & \dots & r _ {1} ^ {n} \\ r _ {2} & r _ {2} ^ {2} & \dots & r _ {2} ^ {n} \\ \vdots & \vdots & \ddots & \vdots \\ r _ {n} & r _ {n} ^ {2} & \dots & r _ {n} ^ {n} \end{array} \right) ^ {- 1} \left( \begin{array}{c} \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {1}}}) \\ \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {2}}}) \\ \vdots \\ \mathcal {O} (h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {n}}}) \end{array} \right) \\ = \sum_ {k = 1} ^ {n} \mathcal {O} (h) \mathcal {O} \left(h ^ {n + 1} + \Delta_ {s} + \Delta_ {\lambda_ {i _ {k}}}\right) \tag {33} \\ \end{array}
$$
Combining Eq. (29), Eq. (32) and Eq. (33), we can obtain the conclusion in Eq. (28).
Then we prove Theorem 3.3 below.
Proof. (Proof of Theorem 3.3)
As we have discussed, the predictor step from  $t_{m-1}$  to  $t_m$  is a special case of the local approximation Eq. (14) with  $(t_{i_n}, \ldots, t_{i_1}, s, t) = (t_{m-n-1}, \ldots, t_{m-2}, t_{m-1}, t_m)$ . By Lemma B.7 we have
$$
\Delta_ {m} = \frac {\alpha_ {t _ {m}} \boldsymbol {A} \left(\lambda_ {t _ {m - 1}} , \lambda_ {t _ {m}}\right)}{\alpha_ {t _ {m - 1}}} \Delta_ {m - 1} + \mathcal {O} (h) \left(\sum_ {k = 0} ^ {n} \mathcal {O} \left(\Delta_ {m - k - 1}\right) + \mathcal {O} \left(h ^ {n + 1}\right)\right) \tag {34}
$$
It follows that there exists constants  $C, C_0$  irrelevant to  $h$ , so that
$$
\left| \Delta_ {m} \right| \leq \left(\frac {\alpha_ {t _ {m}} \mathbf {A} \left(\lambda_ {t _ {m - 1}} , \lambda_ {t _ {m}}\right)}{\alpha_ {t _ {m - 1}}} + C h\right) \left| \Delta_ {m - 1} \right| + C h \sum_ {k = 0} ^ {n} \left| \Delta_ {m - k - 1} \right| + C _ {0} h ^ {n + 2} \tag {35}
$$
Denote  $f_{m}\coloneqq \max_{0\leq i\leq m}|\Delta_{i}|$  , we then have
$$
\left| \Delta_ {m} \right| \leq \left(\frac {\alpha_ {t _ {m}} \mathbf {A} \left(\lambda_ {t _ {m - 1}} , \lambda_ {t _ {m}}\right)}{\alpha_ {t _ {m - 1}}} + C _ {1} h\right) f _ {m - 1} + C _ {0} h ^ {n + 2} \tag {36}
$$
Since  $\frac{\alpha_{t_m}A(\lambda_{t_{m-1}},\lambda_{t_m})}{\alpha_{t_{m-1}}}\to 1$  when  $h\to 0$  and it has bounded first-order derivative due to Assumption B.2, there exists a constant  $C_2$ , so that for any  $C\geq C_2$ ,  $\frac{\alpha_{t_m}A(\lambda_{t_{m-1}},\lambda_{t_m})}{\alpha_{t_{m-1}}} +Ch > 1$  for sufficiently small  $h$ . Thus, by taking  $C_3 = \max \{C_1,C_2\}$ , we have
$$
f _ {m} \leq \left(\frac {\alpha_ {t _ {m}} \boldsymbol {A} \left(\lambda_ {t _ {m - 1}} , \lambda_ {t _ {m}}\right)}{\alpha_ {t _ {m - 1}}} + C _ {3} h\right) f _ {m - 1} + C _ {0} h ^ {n + 2} \tag {37}
$$
Denote  $A_{m - 1}\coloneqq \frac{\alpha_{t_m}A(\lambda_{t_{m - 1}},\lambda_{t_m})}{\alpha_{t_{m - 1}}} +C_3h$  , by repeating Eq. (37), we have
$$
f _ {M} \leq \left(\prod_ {i = n} ^ {M - 1} A _ {i}\right) f _ {n} + \left(\sum_ {i = n + 1} ^ {M} \prod_ {j = i} ^ {M - 1} A _ {j}\right) C _ {0} h ^ {n + 2} \tag {38}
$$
By Assumption B.5,  $h = \mathcal{O}(1 / M)$ , so we have
$$
\begin{array}{l} \prod_ {i = n} ^ {M - 1} A _ {i} = \frac {\alpha_ {t _ {M}} \boldsymbol {A} \left(\lambda_ {t _ {n}} , \lambda_ {t _ {M}}\right)}{\alpha_ {t _ {n}}} \prod_ {i = n} ^ {M - 1} \left(1 + \frac {\alpha_ {t _ {i - 1}} C _ {3} h}{\alpha_ {t _ {i}} \boldsymbol {A} \left(\lambda_ {t _ {i - 1}} , \lambda_ {t _ {i}}\right)}\right) \\ \leq \frac {\alpha_ {t _ {M}} \boldsymbol {A} \left(\lambda_ {t _ {n}} , \lambda_ {t _ {M}}\right)}{\alpha_ {t _ {n}}} \prod_ {i = n} ^ {M - 1} \left(1 + \frac {\alpha_ {t _ {i - 1}} C _ {4}}{\alpha_ {t _ {i}} \boldsymbol {A} \left(\lambda_ {t _ {i - 1}} , \lambda_ {t _ {i}}\right) M}\right) \tag {39} \\ \leq \frac {\alpha_ {t _ {M}} \boldsymbol {A} \left(\lambda_ {t _ {n}} , \lambda_ {t _ {M}}\right)}{\alpha_ {t _ {n}}} \left(1 + \frac {\sigma}{M}\right) ^ {M - n} \\ \leq C _ {5} e ^ {\sigma} \\ \end{array}
$$
where  $\sigma = \max_{n\leq i\leq M - 1}\frac{\alpha_{t_i - 1}C_4}{\alpha_{t_i}A(\lambda_{t_i - 1},\lambda_{t_i})}$ . Then denote  $\beta \coloneqq \max_{n + 1\leq i\leq M}\frac{\alpha_{t_M}A(\lambda_{t_i},\lambda_{t_M})}{\alpha_{t_i}}$ , we have