SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) (\boldsymbol {x} _ {t}, t) \\ = \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) + \int_ {t _ {i - (s - 1)}} ^ {t} \left(\Gamma_ {0} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) + \Gamma_ {1} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}})\right) d t + \int_ {t _ {i - (s - 1)}} ^ {t} \Gamma_ {2} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) d \bar {\boldsymbol {w}} _ {t}. \tag {78} \\ \end{array}
$$
Substituting the expansion of  $\Gamma_0(\pmb{x}_{\theta})$ , we perform the approximation of  $\tilde{L}_{i + 1}$ , which is summarized with the following lemma:
Lemma B.12 (Second step of estimating  $L_{i + 1}$  in data-prediction reparameterization model).  $R_{i + 1}^{(1)}$  in Eq. (70) can be decomposed as
$$
R _ {i + 1} ^ {(1)} = R _ {i + 1} ^ {(2)} + S _ {i + 1} ^ {(2)}, \tag {79}
$$
$$
\begin{array}{l} S_{i + 1}^{(2)} = \mathcal{O}\left(\max_{0\leq t\leq T}\tau (t)h^{\frac{5}{2}}\right) \\ R _ {i + 1} ^ {(2)} \\ = \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} \left(1 + \tau^ {2} (u)\right) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \cdot \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) \\ + \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {0} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) (\boldsymbol {x} _ {u _ {3}}, u _ {3}) \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \mathrm {d} u _ {2} \times \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {0} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) d u _ {3} d u _ {2} \tag {80} \\ \end{array}
$$
Proof. We start with decomposing the term  $R_{i+1}^{(1)}$
$$
\begin{array}{l} R _ {i + 1} ^ {(1)} \\ = \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) (\boldsymbol {x} _ {u _ {2}}, u _ {2}) \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) (\boldsymbol {x} _ {u _ {2}}, u _ {2}) d u _ {2} \\ = \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} \left(1 + \tau^ {2} (u)\right) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \cdot \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) \\ + \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \\ \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \left(\Gamma_ {0} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) + \Gamma_ {1} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}})\right) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ + \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \\ \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {2} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) \mathrm {d} \bar {\boldsymbol {w}} _ {u _ {3}} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \mathrm {d} u _ {2} \times \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \left(\Gamma_ {0} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) + \Gamma_ {1} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}})\right) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) d u _ {3} d u _ {2} \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {2} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) d \bar {\boldsymbol {w}} _ {u _ {3}} d u _ {2}. \tag {81} \\ \end{array}
$$
We further estimate the terms with  $\Gamma_{1}$  and  $\Gamma_{2}$ .
$$
\begin{array}{l} \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \\ \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {1} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ = \mathcal {O} \left(\max  _ {0 \leq t \leq T} \tau^ {2} (t) h ^ {3}\right), \\ \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \\ \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {2} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) \mathrm {d} \bar {\boldsymbol {w}} _ {u _ {3}} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \tag {82} \\ = \mathcal{O}\left(\max_{0\leq t\leq T}\tau (t)h^{\frac{5}{2}}\right), \\ \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {1} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) (\boldsymbol {x} _ {u _ {3}}, u _ {3})   \mathrm {d} u _ {3} \mathrm {d} u _ {2} = \mathcal {O} \left(\max _ {0 \leq t \leq T} \tau^ {2} (t) h ^ {3}\right), \\ \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {2} \Gamma_ {0} (\pmb {x} _ {\pmb {\theta}}) (\pmb {x} _ {u _ {3}}, u _ {3}) \mathrm {d} \bar {\pmb {w}} _ {u _ {3}} \mathrm {d} u _ {2} = \mathcal {O} \left(\max _ {0 \leq t \leq T} \tau (t) h ^ {\frac {5}{2}}\right). \\ \end{array}
$$
The summation of the above terms is  $S^{(2)} = \mathcal{O}\left(\max_{0\leq t\leq T}\tau (t)h^{\frac{5}{2}}\right)$ . Compared with  $S^{(1)}$ , this term can be omitted.
The remaining local error is
$$
\begin{array}{l} R _ {i + 1} ^ {(2)} = \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ \times \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) \\ + \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \\ \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {0} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {3}}, u _ {3}\right) \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \tag {83} \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} d u _ {2} \times \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \Gamma_ {0} \Gamma_ {0} (\boldsymbol {x} _ {\boldsymbol {\theta}}) (\boldsymbol {x} _ {u _ {3}}, u _ {3}) d u _ {3} d u _ {2} \\ \end{array}
$$
Remark 3. With Lemma B.11 and B.12, the local error  $L_{i+1}$  can be decomposed to the term  $S_{i+1} = S_{i+1}^{(1)} + S_{i+1}^{(2)}$  and the term  $R_{i+1}^{(2)}$ . It is clear that  $S_{i+1} = \mathcal{O}\left(\max_{0 \leq t \leq T} \tau(t) h^{\frac{3}{2}}\right)$ , and we will show that given  $b_{i-j}$  constructed by integral of Lagrange polynomial in Eq. (58) and Eq. (60),  $R_{i+1}^{(2)} = \mathcal{O}\left(h^{3}\right)$ .
By Lemma B.10,  $b_{k}$  constructed by the integral of Lagrange polynomial in Eq. (58) and Eq. (60) satisfies that the coefficient for  $\Gamma_0(\pmb{x}_{\theta})(\pmb{x}_{t_{i - (s - 1)}}, t_{i - (s - 1)})$
$$
\sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \mathrm {d} u _ {2}, \tag {84}
$$
equals zero. And the remaining term in  $R_{i+1}^{(2)}$  is  $\mathcal{O}(h^3)$ .
Remark 4. We will show that the local error can be further decomposed such that  $L_{i+1} = R_{i+1}^{(s)} + \sum_{j=1}^{s} S_{i+1}^{(j)}$ . In this case  $S_{i+1} = \sum_{j=1}^{s} S_{i+1}^{(j)}$  is the term such that  $S_{i+1} = \mathcal{O}\left(\max_{0 \leq t \leq T} \tau(t) h^{\frac{3}{2}}\right)$ , and we will show that by our constructed  $b_{i-j}$ ,  $R_{i+1}^{(s)} = \mathcal{O}\left(h^{s+1}\right)$ .
Lemma B.13 ( $j-th$  step of estimating  $L_{i+1}$  in data-prediction reparameterization model). For  $j \leq s+1$ ,  $R_{i+1}^{(j-1)}$  in Eq. (70) can be decomposed as
$$
R _ {i + 1} ^ {(j - 1)} = R _ {i + 1} ^ {(j)} + S _ {i + 1} ^ {(j)}, \tag {85}
$$
$$
S^{(j)}_{i + 1} = \mathcal{O}\left(\max_{0\leq t\leq T}\tau (t)h^{\frac{2j + 1}{2}}\right)
$$
$$
R _ {i + 1} ^ {(j)}
$$
$$
= \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {j - 1}} \mathrm {d} u _ {j} \dots \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u
$$
$$
\cdot \overbrace {\Gamma_ {0} \cdots \Gamma_ {0}} ^ {j - 1} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i _ {- (s - 1)}}}, t _ {i _ {- (s - 1)}}\right)
$$
$$
+ \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} (1 + \tau^ {2} (u)) e ^ {\lambda_ {u}}
$$
$$
\left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {j}} \overbrace {\Gamma_ {0} \cdots \Gamma_ {0}} ^ {j} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {j + 1}}, u _ {j + 1}\right) \mathrm {d} u _ {j + 1} \dots \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u
$$
$$
- \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {j - 1}} d u _ {j} \dots d u _ {3} d u _ {2} \cdot \overbrace {\Gamma_ {0} \cdots \Gamma_ {0}} ^ {j - 1} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right)
$$
$$
- \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {j}} \overbrace {\Gamma_ {0} \cdots \Gamma_ {0}} ^ {j} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {u _ {j + 1}}, u _ {j + 1}\right) d u _ {j + 1} \dots d u _ {3} d u _ {2} \tag {86}
$$
Furthermore, given that  $b_{k}$  is constructed by the integral of Lagrange polynomial in Eq. (58) and Eq. (60),  $R_{i + 1}^{(j)} = \mathcal{O}(h^{j + 1})$
Sketch of the proof (1) Use the Ito formula Eq. (67) to expand  $\overbrace{\Gamma_0\cdots\Gamma_0}^{j-1}(x_\theta)$ . (2) Use Lemma B.9 to estimate the stochastic term  $S^{(j)}$ . For the remaining term  $R^{(j)}$ , by Lemma B.10,  $b_k$  constructed by the integral of Lagrange polynomial in Eq. (58) and Eq. (60) satisfies that the coefficient before
$$
\begin{array}{l} \overbrace {\Gamma_ {0} \cdots \Gamma_ {0}} ^ {j - 1} (\boldsymbol {x} _ {\boldsymbol {\theta}}) \left(\boldsymbol {x} _ {t _ {i - (s - 1)}}, t _ {i - (s - 1)}\right) \\ \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} \left(1 + \tau^ {2} (u)\right) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {j - 1}} \mathrm {d} u _ {j} \dots \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {j - 1}} d u _ {j} \dots d u _ {3} d u _ {2}. \tag {87} \\ \end{array}
$$
equals zero. And the remaining term in  $R_{i+1}^{(j)}$  is  $\mathcal{O}(h^{j+1})$ .
The process can be repeated until the coefficient before  $\overbrace{\Gamma_0\cdots\Gamma_0}^s (\pmb{x}_{\pmb{\theta}})(\pmb{x}_{t_{i_{-}(s - 1)}},t_{i_{-}(s - 1)})$  is
$$
\begin{array}{l} \sigma_ {t _ {i + 1}} \int_ {t _ {i}} ^ {t _ {i + 1}} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\lambda) \mathrm {d} \lambda} \left(1 + \tau^ {2} (u)\right) e ^ {\lambda_ {u}} \left(\int_ {t _ {i - (s - 1)}} ^ {u} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {s}} \mathrm {d} u _ {s + 1} \dots \mathrm {d} u _ {3} \mathrm {d} u _ {2}\right) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ - \sum_ {j = - 1} ^ {s - 1} b _ {i - j} \int_ {t _ {i - (s - 1)}} ^ {t _ {i - j}} \int_ {t _ {i - (s - 1)}} ^ {u _ {2}} \dots \int_ {t _ {i - (s - 1)}} ^ {u _ {s}} d u _ {s + 1} \dots d u _ {3} d u _ {2}. \tag {88} \\ \end{array}
$$
which equals zero. And the remaining term  $R_{i+1}^{s+1}$  is  $\mathcal{O}(h^{s+2})$ .
We conclude with the proof of Lemma B.7 and B.8.
Proof for Lemma B.8 (Convergence for  $s$ -step SA-Corrector) The stochastic term  $S_{i+1}$  can be estimated as  $\mathcal{O}\left(\max_{0 \leq t \leq T} \tau(t) h^{\frac{3}{2}}\right)$ . Lemma B.10 prove that with  $b_{i-j}$  defined in Theorem 5.2, the coefficients of Eq. (75), Eq. (84), Eq. (87) and Eq. (88) equal zero. Thus the deterministic term  $R_{i+1}$  can be estimated as  $\mathcal{O}(h^{s+2})$ . The proof is completed.
Proof for Lemma B.7 (Convergence for  $s$ -step SA-Predictor) The stochastic term  $S_{i+1}$  can be estimated as  $\mathcal{O}\left(\max_{0 \leq t \leq T} \tau(t) h^{\frac{3}{2}}\right)$  from Eq. (76) and Eq. (82). Lemma B.10 prove that with  $b_{i-j}$  defined in Theorem 5.1, the coefficients of Eq. (75), Eq. (84), Eq. (87) and Eq. (88) equal zero except for the last term. This is because in  $s$ -step SA-Predictor we only have  $s$  points in contrast to  $s+1$  points in  $s$ -step SA-Corrector, for which we can only obtain the first  $s$  equalities in Lemma B.10. Thus the deterministic term  $R_{i+1}$  can be estimated as  $\mathcal{O}(h^{s+1})$ . The proof is completed.
B.5 Relationship with Existing Samplers
B.5.1 Relationship with DDIM
DDIM [22] generates samples through the following process:
$$
\boldsymbol {x} _ {t _ {i + 1}} = \alpha_ {t _ {i + 1}} \left(\frac {\boldsymbol {x} _ {t _ {i}} - \sigma_ {t _ {i}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}} , t _ {i}\right)}{\alpha_ {t _ {i}}}\right) + \sqrt {1 - \alpha_ {t _ {i + 1}} ^ {2} - \hat {\sigma} _ {t _ {i}} ^ {2}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) + \hat {\sigma} _ {t _ {i}} \boldsymbol {\xi}, \tag {89}
$$
where  $\xi \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ ,  $\hat{\sigma}_{t_i}$  is a variable parameter. In practice, DDIM introduces a parameter  $\eta$  such that when  $\eta = 0$ , the sampling process becomes deterministic and when  $\eta = 1$ , the sampling process
coincides with original DDPM [2]. Specifically,  $\hat{\sigma}_{t_i} = \eta \sqrt{\frac{1 - \alpha_{t_{i + 1}}^2}{1 - \alpha_{t_i}^2}\left(1 - \frac{\alpha_{t_i}^2}{\alpha_{t_{i + 1}}^2}\right)}$
Corollary 5.3 For any  $\eta$  in DDIM, there exists a  $\tau_{\eta}(t):\mathbb{R}\to \mathbb{R}$  which is a piecewise constant function such that DDIM-  $\eta$  coincides with our 1-step SA-Predictor when  $\tau (t) = \tau_{\eta}(t)$  with data parameterization of our variance-controlled diffusion SDE.
Proof. Our 1-step SA-Predictor when  $\tau(t) = \tau, t \in [t_i, t_{i+1}]$  with data parameterization of our variance-controlled diffusion SDE is
$$
\begin{array}{l} \boldsymbol {x} _ {t _ {i + 1}} = \frac {\sigma_ {t _ {i + 1}}}{\sigma_ {t _ {i}}} e ^ {- \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)} \boldsymbol {x} _ {t _ {i}} + \alpha_ {t _ {i + 1}} \left(1 - e ^ {- \left(1 + \tau^ {2}\right) \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}\right) \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) \tag {90} \\ + \sigma_ {t _ {i + 1}} \sqrt {1 - e ^ {- 2 \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}} \xi . \\ \end{array}
$$
DDIM-  $\eta$  generates samples through the following process
$$
\boldsymbol {x} _ {t _ {i + 1}} = \alpha_ {t _ {i + 1}} \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) + \sqrt {1 - \alpha_ {t _ {i + 1}} ^ {2} - \hat {\sigma} _ {t _ {i}} ^ {2}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) + \hat {\sigma} _ {t _ {i}} \boldsymbol {\xi}, \hat {\sigma} _ {t _ {i}} = \eta \sqrt {\frac {1 - \alpha_ {t _ {i + 1}} ^ {2}}{1 - \alpha_ {t _ {i}} ^ {2}} \left(1 - \frac {\alpha_ {t _ {i}} ^ {2}}{\alpha_ {t _ {i + 1}} ^ {2}}\right)}. \tag {91}
$$
If we substitute  $\hat{\sigma}_{t_i}$  with  $\sigma_{t_{i + 1}}\sqrt{1 - e^{-2\tau^2(\lambda_{t_{i + 1}} - \lambda_{t_i})}}$ , we can verify that  $\sqrt{1 - \alpha_{t_{i + 1}}^2 - \hat{\sigma}_{t_i}^2} = \sigma_{t_{i + 1}}e^{-\tau^2 (\lambda_{t_{i + 1}} - \lambda_{t_i})}$ . The DDIM- $\eta$  then becomes
$$
\begin{array}{l} \boldsymbol {x} _ {t _ {i + 1}} = \alpha_ {t _ {i + 1}} \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) + \sigma_ {t _ {i + 1}} e ^ {- \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)} \left(\frac {\boldsymbol {x} _ {t _ {i}} - \alpha_ {t _ {i}} \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right)}{\sigma_ {t _ {i}}}\right) \\ + \sigma_ {t _ {i + 1}} \sqrt {1 - e ^ {- 2 \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}} \boldsymbol {\xi} \\ = \frac {\sigma_ {t _ {i + 1}}}{\sigma_ {t _ {i}}} e ^ {- \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)} \boldsymbol {x} _ {t _ {i}} + \left(\alpha_ {t _ {i + 1}} - \frac {\alpha_ {t _ {i}}}{\sigma_ {t _ {i}}} \sigma_ {t _ {i + 1}} e ^ {- \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}\right) \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) \tag {92} \\ + \sigma_ {t _ {i + 1}} \sqrt {1 - e ^ {- 2 \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}} \boldsymbol {\xi} \\ = \frac {\sigma_ {t _ {i + 1}}}{\sigma_ {t _ {i}}} e ^ {- \tau^ {2} (\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}})} \pmb {x} _ {t _ {i}} + \alpha_ {t _ {i + 1}} \left(1 - e ^ {- (1 + \tau^ {2}) (\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}})}\right) \pmb {x} _ {\pmb {\theta}} (\pmb {x} _ {t _ {i}}, t _ {i}) \\ + \sigma_ {t _ {i + 1}} \sqrt {1 - e ^ {- 2 \tau^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}} \boldsymbol {\xi}, \\ \end{array}
$$
which is exactly the same with our 1-step SA-Predictor. To find the  $\tau_{\eta}$ , we solve the relationship
$$
\eta \sqrt {\frac {1 - \alpha_ {t _ {i + 1}} ^ {2}}{1 - \alpha_ {t _ {i}} ^ {2}} \left(1 - \frac {\alpha_ {t _ {i}} ^ {2}}{\alpha_ {t _ {i + 1}} ^ {2}}\right)} = \sigma_ {t _ {i + 1}} \sqrt {1 - e ^ {- 2 \tau_ {\eta} ^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}}. \tag {93}
$$
The relationship between  $\tau$  and  $\eta$  is
$$
\eta = \sigma_ {t _ {i}} \sqrt {\frac {1 - e ^ {- 2 \tau_ {\eta} ^ {2} \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}}{1 - \frac {\alpha_ {t _ {i}} ^ {2}}{\alpha_ {t _ {i + 1}} ^ {2}}}}, \tau_ {\eta} = \sqrt {\frac {\log \left(1 - \frac {\eta^ {2}}{\sigma_ {t _ {i}} ^ {2}} \left(1 - \frac {\alpha_ {t _ {i}} ^ {2}}{\alpha_ {t _ {i + 1}} ^ {2}}\right)\right)}{- 2 \left(\lambda_ {t _ {i + 1}} - \lambda_ {t _ {i}}\right)}}. \tag {94}
$$
In a concurrent paper [31], Lu et al. prove the result that their SDE-DPM-Solver++1 coincides with DDIM with a special  $\eta$ . Their result is a special case of Corollary 5.3 when  $\tau_{\eta} \equiv 1$  and  $\eta$  take a special value, while our result holds for arbitrary  $\eta$ .
B.5.2 Relationship with DPM-Solver++(2M)
DPM-Solver++ [31] is a high-order solver which solves diffusion ODEs for guided sampling. DPM-Solver++(2M) is equivalent to the 2-step Adams-Bashforth scheme combined with the exponential integrator. While our 2-step SA-Predictor is also equivalent to the 2-step Adams-Bashforth scheme combined with the exponential integrator when  $\tau(t) \equiv 0$ . Thus DPM-Solver++(2M) is a special case of our 2-step SA-Predictor when  $\tau(t) \equiv 0$ .
B.5.3 Relationship with UniPC
UniPC [24] is a unified predictor-corrector framework for solving diffusion ODEs. Specifically, UniPC-p uses a p-step Adams-Bashforth scheme combined with the exponential integrator as a predictor and a p-step Adams-Moulton scheme combined with the exponential integrator as a corrector. While our p-step SA-Predictor is also equivalent to the p-step Adams-Bashforth scheme combined with the exponential integrator when  $\tau(t) \equiv 0$  and our p-step SA-Corrector is also equivalent to the p-step Adams-Moulton scheme combined with the exponential integrator when  $\tau(t) \equiv 0$ . Thus UniPC-p is a special case of our SA-Solver when  $\tau(t) \equiv 0$  with predictor step  $p$ , corrector step  $p$  in Algorithm 1.
C Selection on the Magnitude of Stochasticity
In this section, we will show that we choose  $\tau(t) \equiv 1$  in a number of NFEs. We will show that under certain conditions, the upper bound of KL divergence between the marginal distribution and the true distribution can be minimized when  $\tau(t) \equiv 1$ .