SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {x} (t) = \mathbf {x} (s) + h \mathbf {x} ^ {\prime} (s) + \frac {h ^ {2}}{2} \mathbf {x} ^ {\prime \prime} (s) + \mathcal {O} (h ^ {3}) \\ = \mathbf {x} (s) + h f (\mathbf {x} (s), s) + \frac {h ^ {2}}{2} (\mathbf {x} ^ {\prime} (s) \partial_ {\mathbf {x}} f (\mathbf {x} (s), s) + \partial_ {t} f (\mathbf {x} (s), s)) + \mathcal {O} (h ^ {3}) \\ { = } { \mathbf { x } ( s ) + h f ( \mathbf { x } ( s ) , s ) + \frac { h ^ { 2 } } { 2 } ( f ( \mathbf { x } ( s ) , s ) \partial _ { \mathbf { x } } f ( \mathbf { x } ( s ) , s ) + \partial _ { t } f ( \mathbf { x } ( s ) , s ) ) + \mathcal { O } ( h ^ { 3 } ) . } \\ \end{array}
$$
A straightforward recursion yields a Taylor expansion
$$
\mathbf {x} (t) = \mathbf {x} (s) + \sum_ {k = 1} ^ {n} \frac {h ^ {k}}{k !} L _ {t} ^ {k - 1} f (\mathbf {x} (t), t) + \int_ {s} ^ {t} \dots \int_ {s} ^ {\tau} L _ {t} ^ {n} f (\mathbf {x} (\tau), \tau) \mathrm {d} \tau^ {n + 1},
$$
where we denote the generalized infinitesimal operator of the solution  $\mathbf{x}$  of (51) by
$$
L _ {t} (\bullet) = \partial_ {t} (\bullet) + f (\mathbf {x} (t), t) \cdot \partial_ {\mathbf {x}} (\bullet).
$$
Derivative-free methods seek to get rid of the derivatives in  $L_{t}$  by computing Taylor expansions of  $f$  at well-chosen points. In the explicit one-step case, this amounts of defining
$$
\widehat {\mathbf {x}} (t) = \widehat {\mathbf {x}} (s) + h \sum_ {k = 1} ^ {n} \Phi (\widehat {\mathbf {x}} (s), s, h),
$$
where the  $\Phi$  function does not contain derivatives of  $f$ . The general high order case is given by well-tuned coefficients in the following scheme
$$
\widehat {\mathbf {x}} (t) := \widehat {\mathbf {x}} (s) + h \sum_ {i = 1} ^ {n} \alpha_ {i} f (\mathbf {x} _ {i}, s + c _ {i} h)
$$
$$
\mathbf {x} _ {i} = \widehat {\mathbf {x}} (s) + h \sum_ {j = 1} ^ {n} a _ {i, j} f (\mathbf {x} _ {j}, s + c _ {j} h).
$$
By denoting  $\alpha = [\alpha_{1}\dots \alpha_{n}]$ ,  $c = [c_{1}\dots c_{n}]^{T}$  and  $A = (a_{i,j})$ , these can be represented by a Butcher tableau of the form
$$
\begin{array}{c c} c & A \\ \hline & \alpha \end{array}
$$
In the following sections we will present extended version of this tableau for representing more involved numerical schemes. As such, the Euler, midpoint, Heun and general second order explicit methods are respectively:
$$
\begin{array}{c c} 0 & 0 \\ \hline & 1 \end{array} \qquad \begin{array}{c c} 0 & 0 \\ \hline 1 / 2 & 1 / 2 \\ \hline & 0 \end{array} \qquad \begin{array}{c c} 0 & 0 \\ \hline 1 & 1 \\ \hline & 1 / 2 \end{array} \qquad \begin{array}{c c} 0 & 0 \\ \hline c _ {2} & c _ {2} \\ \hline & 1 - \frac {1}{2 c _ {2}} \end{array} \qquad \begin{array}{c c} 0 & 0 \\ \hline c _ {2} & 0 \\ \hline & 1 - \frac {1}{2 c _ {2}} \end{array}
$$
E.1.2 Exponential Runge-Kutta approach
We now concentrate on a non-autonomous semi-linear ODE of the form
$$
\mathrm {d} \mathbf {x} (t) = [ f (t) \mathbf {x} (t) + g (\mathbf {x} (t), t) ] \mathrm {d} t, \tag {52}
$$
where  $f(t) = \bar{f}(t) \cdot \mathrm{Id}_d$  and  $g(\mathbf{x}(t), t) = \bar{g}(\mathbf{x}(t), t) \cdot \mathrm{Id}_d$  are diagonal time-dependent matrices with identical diagonal coefficients. In particular, we have  $[f^{(k)}(t), f^{(l)}(s)] = 0$ , a property we will need to facilitate exponential matrix multiplications. As  $f(s)$  is a  $d$ -dimensional diagonal matrix, the fundamental matrix for (52), usually given by the Peano-Baker series, simplifies in this case to the form
$$
\Phi (t; s) := e ^ {\int_ {s} ^ {t} f (r) \mathrm {d} r} = \sum_ {n = 0} ^ {\infty} \frac {1}{n !} \left(\int_ {s} ^ {t} f (r) \mathrm {d} r\right) ^ {n},
$$
which satisfies  $\Phi'(t; s) = f(t)\Phi(t; s)$  and  $\Phi(s; s) = 1$ . Then the exact solution for (52) is given, via the variation of constants formula, by
$$
\mathbf {x} (t) = \Phi (t; s) \left(\mathbf {x} (s) + \int_ {s} ^ {t} \Phi^ {- 1} (\tau ; s) g (\mathbf {x} (\tau), \tau) \mathrm {d} \tau\right) = \Phi (t; s) \mathbf {x} (s) + \int_ {s} ^ {t} \Phi (t; \tau) g (\mathbf {x} (\tau), \tau) \mathrm {d} \tau .
$$
In light of this integral form, one can formalize a general class of
exponential  $n$ -stage RK methods
$$
\mathbf {x} _ {i} = \gamma_ {i} (h, f _ {s}) \mathbf {x} (s) + \sum_ {j = 1} ^ {n} a _ {i j} (h, f _ {s}) g (\mathbf {x} _ {j}, s + c _ {j} h)
$$
$$
{\bf x} (t) = \gamma_ {0} (h, f _ {s}) {\bf x} (s) + \sum_ {i = 1} ^ {n} b _ {i} (h, f _ {s}) g ({\bf x} _ {i}, s + c _ {i} h),
$$
where  $\gamma_0, \gamma_i, a_{ij}, b_j$  are functions of the step-size  $h$ ,  $f$  and  $f_s(r) \coloneqq \int_0^r f(s + \tau)\mathrm{d}\tau$ . There are two possible approaches to create exponential integrators, namely the exponential time-differencing (ETD) approach that uses the variation of constants formula and makes use of the  $\varphi$  functions, and the Lawson approach, also known as integrating factor (IF), which makes a change of variables on the above SDE thus avoiding the use of the  $\varphi$  functions but computing exponential factors step-wise.
The Lawson and the ETD approaches for Exponential Euler schemes
There are two choices one can make when computing first order approximations of
$$
\mathbf {x} (t) = \Phi (t; s) \mathbf {x} (s) + \int_ {s} ^ {t} \Phi (t; \tau) g (\mathbf {x} (\tau), \tau) \mathrm {d} \tau .
$$
First, by interpolating  $g(\mathbf{x}(\tau), \tau)$  as  $g(\mathbf{x}(s), s)$  we obtain
$$
{\mathbf {x} (t)} = {\Phi (t; s) \mathbf {x} (s) + g (\mathbf {x} (s), s) \int_ {s} ^ {t} \Phi (t; \tau) \mathrm {d} \tau .}
$$
Now two choices remain. Either  $\int_s^t\Phi (t;\tau)\mathrm{d}\tau$  is computed exactly or we again interpolate  $\Phi (t;\tau)$  as  $\Phi (t;s)$ . Taking for simplicity  $f\equiv A$  to be constant and by denoting  $h = t - s$ , the first case yields the ETD Euler method
$$
\begin{array}{l} \widehat {\mathbf {x}} (t) = \Phi (t; s) \widehat {\mathbf {x}} (s) + h \gamma_ {1} (f, t, s) g (\widehat {\mathbf {x}} (s), s) \\ = e ^ {A h} \widehat {\mathbf {x}} (s) + h \varphi_ {1} (A h) g (\widehat {\mathbf {x}} (s), s) \\ = \widehat {\mathbf {x}} (s) + h \varphi_ {1} (A h) [ g (\widehat {\mathbf {x}} (s), s) - \widehat {\mathbf {x}} (s) ], \\ \end{array}
$$
and the second yields the IF Euler (also called Lawson-Euler) method
$$
\begin{array}{l} \widehat {\mathbf {x}} (t) = \Phi (t; s) \widehat {\mathbf {x}} (s) + h \Phi (t; s) g (\widehat {\mathbf {x}} (s), s) \\ = e ^ {A h} (\widehat {\mathbf {x}} (s) + h g (\widehat {\mathbf {x}} (s), s)). \\ \end{array}
$$
Now, consider a solution
$$
\mathbf {x} (t) = \Phi (t; s) \mathbf {x} (s) + \int_ {0} ^ {h} \Phi (h; \tau) g (\mathbf {x} (s + \tau), s + \tau) \mathrm {d} \tau .
$$
Exponential methods then aim to approximate the term  $g(\mathbf{x}(s + \tau), s + \tau)$  by its interpolation polynomial in certain non-confluent quadrature nodes  $c_1, \ldots, c_n$ .
The ETD approach
In this case, the variation of constants formula yields
$$
\begin{array}{l} \mathbf {x} (s + c h) = e ^ {\int_ {s} ^ {s + c h} a (\tau) \mathrm {d} \tau} \mathbf {x} (s) + \int_ {s} ^ {s + c h} e ^ {\int_ {\tau} ^ {s + c h} a (r) \mathrm {d} r} f (\mathbf {x} (\tau), \tau) \mathrm {d} \tau . \\ { = } { e ^ { \int _ { 0 } ^ { c h } a ( s + \tau ) \mathrm { d } \tau } \mathbf { x } ( t ) + \int _ { 0 } ^ { c h } e ^ { \int _ { \tau - s } ^ { c h } a ( s + r ) \mathrm { d } r } f ( \mathbf { x } ( s + \tau ) , s + \tau ) \mathrm { d } \tau . } \\ \end{array}
$$
Now, as before, the Taylor expansion of  $f$  yields
$$
\begin{array}{l} {f (\mathbf {x} (s + \tau), s + \tau)} = {\sum_ {j = 1} ^ {q} \frac {\tau^ {j - 1}}{(j - 1) !} f ^ {(j - 1)} (\mathbf {x} (s), s)} \\ + \int_ {0} ^ {\tau} \frac {(\tau - \tau_ {1}) ^ {q - 1}}{(q - 1) !} f ^ {(q)} (\mathbf {x} (s + \tau_ {1}), s + \tau_ {1}) \mathrm {d} \tau_ {1}. \\ \end{array}
$$
Recall that the  $\varphi$  functions are given in an integral form as follows
which satisfy  $\varphi_{k}(0) = \frac{1}{k!}$ . Now denote
$$
\varphi_ {j} (t, a) := \frac {1}{t ^ {j}} \int_ {0} ^ {t} e ^ {\int_ {\tau} ^ {t} a (r) \mathrm {d} r} \frac {\tau^ {j - 1}}{(j - 1) !} \mathrm {d} \tau , \quad j \geqslant 1.
$$
$$
\varphi_ {1} (h, a) = \frac {1}{h} \int_ {0} ^ {h} e ^ {\int_ {\tau} ^ {h} a (r) \mathrm {d} r} \mathrm {d} \tau = \int_ {0} ^ {1} e ^ {h \int_ {\theta} ^ {1} a (r) \mathrm {d} r} \mathrm {d} \theta .
$$
The exact solution at  $s + ch$  now reads
$$
\begin{array}{l} \mathbf {x} (s + c h) = e ^ {\int_ {0} ^ {c h} a (s + \tau) \mathrm {d} \tau} \mathbf {x} (s) + \sum_ {j = 1} ^ {q} (c h) ^ {j} \varphi_ {j} (c h, a) f ^ {(j - 1)} (\mathbf {x} (s), s) \\ + \int_ {0} ^ {c h} e ^ {\int_ {\tau} ^ {c h} a (s + r) \mathrm {d} r} \left(\int_ {0} ^ {\tau} \frac {(\tau - \tau_ {1}) ^ {q - 1}}{(q - 1) !} f ^ {(q)} (\mathbf {x} (s + \tau_ {1}), s + \tau_ {1}) \mathrm {d} s\right) \mathrm {d} \tau_ {1}. \\ \end{array}
$$
Using the left endpoint rule yields
$$
\begin{array}{l} \mathbf {x} (t) = e ^ {\int_ {0} ^ {h} a (s + \tau) \mathrm {d} \tau} \mathbf {x} (s) + f (\mathbf {x} (s), s) \int_ {0} ^ {h} e ^ {\int_ {\tau} ^ {h} a (s + r) \mathrm {d} r} \mathrm {d} \tau + \mathcal {O} (h ^ {2}) \\ { = } { e ^ { \int _ { 0 } ^ { h } a ( s + \tau ) \mathrm { d } \tau } \mathbf { x } ( s ) + h \varphi _ { 1 } ( h , a ) f ( \mathbf { x } ( s ) , s ) + \mathcal { O } ( h ^ { 2 } ) . } \\ \end{array}
$$
Second order examples:
The condition  $b_{2}(z)c_{2} = \varphi_{2}(z)$  implies  $b_{2} = \varphi_{2}(z) / c_{2}$  and we obtain
$$
\begin{array}{l} \widehat {\mathbf {x}} = e ^ {c h A} \mathbf {x} (s) + c h \varphi_ {1} (c h A) f (\mathbf {x} (s), s) \\ \mathbf {x} (t) = e ^ {h A} \mathbf {x} (s) + h \left(\varphi_ {1} (h A) - \frac {1}{c} \varphi_ {2} (h A)\right) f (\mathbf {x} (s), s) + \frac {h}{c} \varphi_ {2} (h A) f (\widehat {\mathbf {x}}, s + c h). \tag {53} \\ \end{array}
$$
A second method is obtained by weakening the above condition to  $b_{2}(0)c_{2} = \varphi_{2}(0) = 1 / 2$  giving
$$
\begin{array}{l} \widehat {\mathbf {x}} = e ^ {c h A} \mathbf {x} (s) + c h \varphi_ {1} (c h A) f (\mathbf {x} (s), s) \\ \mathbf {x} (t) = e ^ {h A} \mathbf {x} (s) + h \varphi_ {1} (h A) \left(1 - \frac {1}{2 c}\right) f (\mathbf {x} (s), s) + \frac {h}{2 c} \varphi_ {1} (h A) f (\widehat {\mathbf {x}}, s + c h). \tag {54} \\ \end{array}
$$
In some cases this method can suffer from order reduction and not reach order 2 of convergence. Moreover, setting  $c = 1/2$  gives the exponential midpoint method:
$$
\widehat {\mathbf {x}} = e ^ {c h A} \mathbf {x} (s) + c h \varphi_ {1} (c h A) f (\mathbf {x} (s), s)
$$
$$
\mathbf {x} (t) = e ^ {h A} \mathbf {x} (s) + h \varphi_ {1} (h A) f (\widehat {\mathbf {x}}, s + c h). \tag {55}
$$
The above order 1 and 2 exponential methods are represented by the following exponential Butcher tableaux
$$
\begin{array}{c c} 0 & 0 \\ \hline & \varphi_ {1} \end{array} \qquad \begin{array}{c c} 0 & \\ & c _ {2} \varphi_ {1, 2} \\ \hline & \varphi_ {1} - \frac {1}{c _ {2}} \varphi_ {2} \quad \frac {1}{c _ {2}} \varphi_ {2} \end{array} \qquad \begin{array}{c c} 0 & \\ & c _ {2} \varphi_ {1, 2} \\ \hline & \left(1 - \frac {1}{2 c _ {2}}\right) \varphi_ {1} \quad \frac {1}{2 c _ {2}} \varphi_ {1} \end{array} \qquad \begin{array}{c c} 0 & \\ & c _ {2} \varphi_ {1, 2} \\ \hline & 0 & \varphi_ {1} \end{array}
$$
We check that when formally setting  $A = 0$  then the first two methods are identical and give the generic 2nd order RK method, the choice  $c = 1$  gives the Heun method and the choice  $c = 1/2$  gives the midpoint method.
Fourth order methods
It can be shown that ERK methods need at least 5 stages to achieve order 4. By setting formally  $A = 0$  these methods do not have a non-exponential counterpart to our knowledge.
5-stage sequential We have a fourth-order ERK scheme given by
$$
\begin{array}{c c c c c} 0 & & & & \\ \frac {1}{2} & \frac {1}{2} \varphi_ {1, 2} & & & \\ \frac {1}{2} & \frac {1}{2} \varphi_ {1, 3} - \varphi_ {2, 3} & \varphi_ {2, 3} & & \\ 1 & \varphi_ {1, 4} - 2 \varphi_ {2, 4} & \varphi_ {2, 4} & \varphi_ {2, 4} & \\ \frac {1}{2} & \frac {1}{2} \varphi_ {1, 5} - 2 a _ {5, 2} - a _ {5, 4} & a _ {5, 2} & a _ {5, 2} & \frac {1}{4} \varphi_ {2, 5} - a _ {5, 2} \\ \hline & \varphi_ {1} - 3 \varphi_ {2} + 4 \varphi_ {3} & 0 & 0 & - \varphi_ {2} + 4 \varphi_ {3} \quad 4 \varphi_ {2} - 8 \varphi_ {3} \end{array}
$$
$$
a _ {5, 2} = c _ {5} \varphi_ {2, 5} - \varphi_ {3, 4} + c _ {5} ^ {2} \varphi_ {2, 4} - c _ {5} \varphi_ {3, 5}.
$$
This can also be represented by Rosenbrock-like Butcher tableau
$$
\begin{array}{c c c c c} 0 & & & & \\ \frac {1}{2} & \frac {1}{2} \varphi_ {1, 2} & & & \\ \frac {1}{2} & \frac {1}{2} \varphi_ {1, 3} & \varphi_ {2, 3} & & \\ 1 & \varphi_ {1} & \varphi_ {2} & \varphi_ {2} & \\ \frac {1}{2} & \frac {1}{2} \varphi_ {1, 5} & a _ {5, 2} & a _ {5, 2} & \frac {1}{4} \varphi_ {2, 5} - a _ {5, 2} \\ \hline & \varphi_ {1} & 0 & 0 & 4 \varphi_ {3} - \varphi_ {2} \quad 4 \varphi_ {2} - 8 \varphi_ {3} \end{array}
$$
$$
a _ {5, 2} = c _ {5} \varphi_ {2, 5} - c _ {3} \varphi_ {3, 3} + c _ {5} ^ {2} \varphi_ {2} - \varphi_ {3}.
$$
This traduces into (recall that  $D_{i} = f(\mathbf{x}_{i},s + c_{i}h) - f(\mathbf{x}(s),s))$
$$
\begin{array}{l} \mathbf {x} (t) = \mathbf {x} (s) + h (\varphi_ {1} (h A)) F (\mathbf {x} (s), s) + h (4 \varphi_ {3} (h A) - \varphi_ {2} (h A)) D _ {4} \\ + h \left(4 \varphi_ {2} (h A) - 8 \varphi_ {3} (h A)\right) D _ {5} \\ \end{array}
$$
$$
\mathbf {x} _ {2} = \mathbf {x} (s) + c _ {2} h \varphi_ {1} \left(c _ {2} h A\right) F (\mathbf {x} (s), s)
$$
$$
\mathbf {x} _ {3} = \mathbf {x} (s) + h c _ {3} \varphi_ {1} \left(c _ {3} h A\right) F (\mathbf {x} (s), s) + h \varphi_ {2} \left(c _ {3} h A\right) D _ {2}
$$
$$
\mathbf {x} _ {4} = \mathbf {x} (s) + h \varphi_ {1} (h A) F (\mathbf {x} (s), s) + h \varphi_ {2} (h A) \left(D _ {2} + D _ {3}\right)
$$
$$
\mathbf {x} _ {5} = \mathbf {x} (s) + h \varphi_ {1} (h A) F (\mathbf {x} (s), s) + h a _ {5, 2} (h A) \left(D _ {2} + D _ {3}\right) + h \left(c _ {5} ^ {2} \varphi_ {2} \left(c _ {5} h A\right) - a _ {5, 2} (h A)\right) D _ {4}
$$
Inspired by this, we deduce the following fourth order Algorithm 7 specialized for DPMs in the VP noise prediction regime.
E.2 Derivative-free exponential SDE schemes
Let  $\mathbf{x}_t$  be the path at the continuous limit  $h\to 0$ , and  $\{\widehat{\mathbf{x}}_t\}_{t_0}^{t_M}$  be the discretized numerical path, computed by a numerical scheme  $S$  with  $M = 1 / h$  steps of length  $h > 0$ . Then,  $S$  has
1. strong order of convergence  $\gamma$  if there is  $K > 0$  such that
$$
\mathbb {E} \left[ \left| \mathbf {x} _ {t _ {M}} - \widehat {\mathbf {x}} _ {t _ {M}} \right| \right] \leq K h ^ {\gamma}, \tag {56}
$$
2. weak order of convergence  $\beta$  if there is  $K > 0$  and a function class  $\kappa$  such that
$$
\left| \mathbb {E} \left[ \phi \left(\mathbf {x} _ {t _ {M}}\right) \right] - \mathbb {E} \left[ \phi \left(\widehat {\mathbf {x}} _ {t _ {M}}\right) \right] \right| \leq K h ^ {\beta}, \quad \forall \phi (\cdot) \in \mathcal {K}. \tag {57}
$$
Strong convergence is concerned with the precision of the path, while the weak convergence is with the precision of the moments. As, for diffusion models, the center of attention is the evolution of the probability densities rather than that of the noising process of single data samples, weak convergence is enough to guarantee the well-conditioning of our numerical schemes. Moreover, when the diffusion coefficient vanishes, then both strong and weak convergence (with the choice  $\phi = \mathrm{id}$ ) reduce to the usual deterministic convergence criterion for ODEs.
Algorithm 7 DPM-Solver-4
1: def DPM-Solver-4  $(\epsilon_{\theta},\tilde{\mathbf{x}}_{t_{i - 1}},t_{i - 1},t_i,r = 0.5)$
2:  $h_i\gets \lambda_{t_i} - \lambda_{t_{i - 1}}$
3:  $s_2 \gets t_\lambda (\lambda_{t_{i-1}} + rh_i)$ ,  $s_3 \gets t_\lambda (\lambda_{t_{i-1}} + rh_i)$
4:  $s_4 \gets t_\lambda (\lambda_{t_{i-1}} + h_i)$ ,  $s_5 \gets t_\lambda (\lambda_{t_{i-1}} + rh_i)$
5:  $\pmb{k}_1 \gets \pmb{\epsilon}_{\theta}(\tilde{\mathbf{x}}_{t_{i-1}}, t_{i-1})$
6:  $\pmb{k}_2 \gets \frac{\alpha_{s_2}}{\alpha_s} \tilde{\mathbf{x}}_{t_i - 1} - \sigma_{s_2}(e^{rh} - 1) \pmb{k}_1$
7:  $\pmb{k}_{3} \gets \frac{\alpha_{s_{3}}}{\alpha_{s}} \tilde{\mathbf{x}}_{t_{i-1}} - \sigma_{s_{3}}(e^{rh} - 1) \pmb{k}_{1} - \sigma_{s_{3}}\left(4 \frac{e^{rh} - 1}{h} - 2\right)[\epsilon_{\theta}(\pmb{k}_{2}, s_{2}) - \pmb{k}_{1}]$
8:  $\pmb{k}_4 \gets \frac{\alpha_{s_4}}{\alpha_s} \tilde{\mathbf{x}}_{t_{i-1}} - \sigma_{s_4}(e^h - 1) \pmb{k}_1 - \sigma_{s_4}\left(\frac{e^h - 1}{h} - 1\right)\left[\epsilon_\theta(\pmb{k}_3, s_3) + \epsilon_\theta(\pmb{k}_2, s_2) - 2 \pmb{k}_1\right]$
9:  $A = \sigma_{s_5}(e^{rh} - 1)\pmb {k}_1 - \frac{1}{4}\sigma_{s_5}\left(\frac{e^h - 1}{h} -1\right)[\pmb {k}_1 + \epsilon_\theta (\pmb {k}_2,s_2) + \epsilon_\theta (\pmb {k}_3,s_3)]$
10:  $B = \sigma_{s_5}\left(\frac{e^{rh} - 1}{h} -\frac{1}{2}\right)\left[\pmb {k}_1 + 4\epsilon_\theta (\pmb {k}_2,s_2) + 4\epsilon_\theta (\pmb {k}_3,s_3) - \epsilon_\theta (\pmb {k}_4,s_4)\right]$
11:  $C = \sigma_{s_5}\left(\frac{e^h - 1 + 4(e^{rh} - 1) - 3h}{h^2} -1\right)\left[-\pmb {k}_1 - \epsilon_\theta (\pmb {k}_2,s_2) - \epsilon_\theta (\pmb {k}_3,s_3) + \epsilon_\theta (\pmb {k}_4,s_4)\right]$
12:  $\pmb{k}_{5} \gets \frac{\alpha_{s_{5}}}{\alpha_{t_{i - 1}}} \tilde{\mathbf{x}}_{t_{i - 1}} - A - B - C$
13:  $D = \sigma_{t}(e^{h} - 1)\pmb{k}_{\mathbf{1}} - \sigma_{t}\left(\frac{e^{h} - 1}{h} -1\right)[4\epsilon_{\theta}(\pmb{k}_{\mathbf{5}},s_{5}) - \epsilon_{\theta}(\pmb{k}_{\mathbf{4}},s_{4}) - 3\pmb{k}_{\mathbf{1}}]$
14:  $E = \sigma_{t}\left(4\frac{e^{h} - 1 - h}{h^{2}} -2\right)\left[k_{1} + \epsilon_{\theta}(k_{4},s_{4}) - 2\epsilon_{\theta}(k_{5},s_{5})\right]$
15:  $\tilde{\mathbf{x}}_{t_i} \gets \frac{\alpha_t}{\alpha_{t_{i-1}}} \tilde{\mathbf{x}}_{t_{i-1}} - D - E$
16: Return  $\mathbf{x}_{t_i}$
E.2.1 Strong and Weak Stochastic Runge-Kutta approach
In all what follows  $\omega$  is considered a  $d$ -dimensional Wiener process (with identity diffusion matrix).
Consider the following SDE
$$
\mathrm {d} \mathbf {x} (t) = f (\mathbf {x} (t), t) \mathrm {d} t + g (t) \mathrm {d} \omega (t),
$$
where  $g(t) = \hat{g}(t) \cdot \mathrm{Id}_d$  is considered here as a diagonal matrix with identical diagonal entries  $\hat{g}(t)$ . Given an initial value independent of  $\omega$ , the integral form of  $\mathbf{x}(t)$  is given by
$$
\mathbf {x} (t) = \mathbf {x} (s) + \int_ {s} ^ {t} f (\mathbf {x} (\tau), \tau) d \tau + \int_ {s} ^ {t} g (\tau) \mathrm {d} \omega (\tau).
$$
The idea underlying stochastic numerical schemes is very similar to the one used in the deterministic approach, that is to take expansions of the terms inside the integrals based at the integral's initial value and replace the obtained derivatives that appear by interpolated approximations. A key difference here is that as we have to consider Itô-Taylor expansions, the infinitesimal operators are different but most importantly most of the stochastic iterated integrals will need to be approximated in an appropriate sense, whenever that is possible. We will develop this expansion up to triple integrals.
E.2.2 Truncated Itô-Taylor expansions
Applying Itô formula to  $h = f$  or  $g$  yields
$$
\begin{array}{l} {h (\mathbf {x} (t), t)} = {h (\mathbf {x} (s), s) + \int_ {s} ^ {t} g (\tau) \cdot \partial_ {\mathbf {x}} h (\mathbf {x} (\tau), \tau) \mathrm {d} \omega (\tau)} \\ + \int_ {s} ^ {t} \left(\partial_ {t} h (\mathbf {x} (\tau), \tau) + f (\mathbf {x} (\tau), \tau) \cdot \partial_ {\mathbf {x}} h (\mathbf {x} (\tau), \tau) + \frac {g ^ {2} (\tau)}{2} \partial_ {\mathbf {x} ^ {2}} ^ {2} h (\mathbf {x} (\tau), \tau)\right) d \tau . \\ \end{array}
$$
This allows us to define two differential operators  $L, G$  as
$$
L _ {t} = \partial_ {t} + f (\mathbf {x} (t), t) \cdot \partial_ {\mathbf {x}} + \frac {g ^ {2} (t)}{2} \cdot \partial_ {\mathbf {x}} ^ {2} \tag {58}
$$
$$
G _ {t} = g (t) \cdot \partial_ {\mathbf {x}}. \tag {59}
$$
In particular  $L_{t}g(t) = \partial_{t}g(t)$  and  $G_{t}g(t) = 0$ . With this notation, we have
$$
{h (\mathbf {x} (t), t)} = {h (\mathbf {x} (s), s) + \int_ {s} ^ {t} L _ {t} h (\mathbf {x} (\tau), \tau) \mathrm {d} \tau + \int_ {s} ^ {t} G _ {t} h (\mathbf {x} (\tau), \tau) \mathrm {d} \omega (\tau),}
$$
so our solution reads
$$
\begin{array}{l} {\bf x} (t) = {\bf x} (s) + \int_ {s} ^ {t} f ({\bf x} (\tau_ {1}), \tau_ {1}) \mathrm {d} \tau_ {1} + \int_ {s} ^ {t} g (\tau_ {1}) \mathrm {d} \omega (\tau_ {1}) \\ = \mathbf {x} (s) + \int_ {s} ^ {t} \left(f (\mathbf {x} (t), t) + \int_ {s} ^ {\tau_ {1}} L _ {t} f (\mathbf {x} (\tau_ {2}), \tau_ {2}) \mathrm {d} \tau_ {2} + \int_ {s} ^ {\tau_ {1}} G _ {t} f (\mathbf {x} (\tau_ {2}), \tau_ {2}) d \omega (\tau_ {2})\right) \mathrm {d} \tau_ {1} \\ + \int_ {s} ^ {t} \left(g (t) + \int_ {s} ^ {\tau_ {1}} L _ {t} g (\tau_ {2}) \mathrm {d} \tau_ {2} + \int_ {s} ^ {\tau} G _ {t} g (\tau_ {2}) \mathrm {d} \omega (\tau_ {2})\right) \mathrm {d} \omega (\tau_ {1}) \\ { = } { \mathbf { x } ( s ) + f ( \mathbf { x } ( s ) , s ) h + g ( t ) ( \omega ( t ) - \omega ( s ) ) + R _ { 1 } . } \\ \end{array}
$$
Now  $G_{t}g(\tau_{2}) = 0$  and
$$
g (s) + \int_ {s} ^ {\tau_ {1}} L _ {t} g (\tau_ {2}) \mathrm {d} \tau_ {2} = g (s) + \int_ {s} ^ {\tau_ {1}} \partial_ {t} g (\tau_ {2}) \mathrm {d} \tau_ {2} = g (s) + g (\tau_ {1}) - g (s) = g (\tau_ {1}).
$$
$$
\int_ {s} ^ {t} \left(g (s) + \int_ {s} ^ {\tau_ {1}} L _ {t} g (\tau_ {2}) \mathrm {d} \tau_ {2} + \int_ {s} ^ {\tau} G _ {t} g (\tau_ {2}) \mathrm {d} \omega (\tau_ {2})\right) \mathrm {d} \omega (\tau_ {1}) = g (s) (\omega (t) - \omega (s)),
$$
and now our solution reads
$$
\mathbf {x} (t) = \mathbf {x} (s) + f (\mathbf {x} (s), s) h + g (s) (\omega (t) - \omega (s)) + R _ {1},
$$
$$
{R _ {1}} = {\int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} L _ {t} f (\mathbf {x} (\tau_ {2}), \tau_ {2}) \mathrm {d} \tau_ {2} \mathrm {d} \tau_ {1} + \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} G _ {t} f (\mathbf {x} (\tau_ {2}), \tau_ {2}) \mathrm {d} \omega (\tau_ {2}) \mathrm {d} \tau_ {1}.}
$$
$$
\begin{array}{l} L _ {t} f (\mathbf {x} (t), t) \\ { = } { L _ { t } f ( \mathbf { x } ( s ) , s ) + \int _ { s } ^ { t } g ( \mathbf { x } ( \tau ) , \tau ) \cdot \partial _ { \mathbf { x } } L _ { t } f ( \mathbf { x } ( \tau ) , \tau ) \mathrm { d } \omega ( \tau ) } \\ + \int_ {s} ^ {t} \left(\partial_ {t} L _ {t} f (\mathbf {x} (\tau), \tau) + f (\mathbf {x} (\tau), \tau) \cdot \partial_ {\mathbf {x}} L _ {t} f (\mathbf {x} (\tau), \tau) + \frac {g ^ {2} (\mathbf {x} (\tau) , \tau)}{2} \partial_ {\mathbf {x} ^ {2}} ^ {2} L _ {t} f (\mathbf {x} (\tau), \tau)\right) d \tau \\ { = } { L _ { t } f ( \mathbf { x } ( s ) , s ) + \int _ { s } ^ { t } L _ { t } ^ { 2 } f ( \mathbf { x } ( \tau ) , \tau ) \mathrm { d } \tau + \int _ { s } ^ { t } G _ { t } L _ { t } f ( \mathbf { x } ( \tau ) , \tau ) \mathrm { d } \omega ( \tau ) , } \\ \end{array}
$$
$$
\begin{array}{l} G _ {t} f (\mathbf {x} (t), t) \\ = G _ {t} f (\mathbf {x} (s), s) + \int_ {s} ^ {t} g (\mathbf {x} (\tau), \tau) \cdot \partial_ {\mathbf {x}} G _ {t} f (\mathbf {x} (\tau), \tau) d \omega (\tau) + \\ \int_ {s} ^ {t} \left(\partial_ {t} G _ {t} f (\mathbf {x} (\tau), \tau) + f (\mathbf {x} (\tau), \tau) \cdot \partial_ {\mathbf {x}} G _ {t} f (\mathbf {x} (\tau), \tau) + \frac {g ^ {2} (\mathbf {x} (\tau) , \tau)}{2} \partial_ {\mathbf {x} ^ {2}} ^ {2} G _ {t} f (\mathbf {x} (\tau), \tau)\right) d \tau \\ { = } { G _ { t } f ( \mathbf { x } ( s ) , s ) + \int _ { s } ^ { t } L _ { t } G _ { t } f ( \mathbf { x } ( \tau ) , \tau ) \mathrm { d } \tau + \int _ { s } ^ { t } G _ { t } ^ { 2 } f ( \mathbf { x } ( \tau ) , \tau ) \mathrm { d } \omega ( \tau ) . } \\ \end{array}
$$
Now, if we denote  $\mathrm{d}\omega^0 (\tau) = \mathrm{d}\tau$ ,  $\mathrm{d}\omega^{1}(\tau) = \mathrm{d}\omega (\tau)$  and
$$
I _ {(i)} = \int_ {s} ^ {t} \mathrm {d} \omega^ {i} (\tau_ {1}) I _ {(i, j)} = \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \mathrm {d} \omega^ {j} (\tau_ {2}) \mathrm {d} \omega^ {i} (\tau_ {1}) i, j \in \{0, 1 \},
$$
applying the same procedure to  $R_{1}$  leads to