SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma C.11. Assume the following forward SDE is satisfied  $dX_{t} = F(t,X_{t})dt + G(t)dW_{t}, t \in [0,T]$ , where  $T > 0$ ,  $F(t,x)$  is Lipschitz with respect to  $(t,x)$ ,  $G$  is continuous and  $X_0$  is an integrable random variable. Then, there exists  $C > 0$  such that for all  $s < t \in [0,T]$  with  $t - s \leq 1$ ,
$$
\mathbb {E} \big [ | X _ {t} - X _ {s} | \big ] \leq C \sqrt {t - s}.
$$
Proof. We take  $s = 0$  and apply the triangular inequality:
$$
\left| X _ {t} - X _ {0} \right| \leq \int_ {0} ^ {t} \left| F \left(u, X _ {u}\right) - F \left(0, X _ {0}\right)\right) | \mathrm {d} u + \left| \int_ {0} ^ {t} G (u) d W _ {u} \right| + t \left| F \left(0, X _ {0}\right) \right|.
$$
Setting  $u(t) = \mathbb{E}[|X_t - X_0|]$  and taking the expectation, we deduce:
$$
u (t) \leq K \frac {t ^ {2}}{2} + t \mathbb {E} [ | F (0, X _ {0}) | ] + \mathbb {E} [ (\int_ {0} ^ {t} G (u) d W _ {u}) ^ {2} ] ^ {\frac {1}{2}} + K \int_ {0} ^ {t} u (s) d s,
$$
where  $K$  is a positive constant. Note that  $\mathbb{E}[(\int_0^t G(u)dW_u)^2 ]^{\frac{1}{2}} = [\int_0^t G^2 (s)ds]^{\frac{1}{2}}$  by the Itô isometry property which is less than  $C\sqrt{t}$  since  $G$  is continuous. Thus, we have proved that:
$$
u (t) \leq \alpha (t) + K \int_ {0} ^ {t} u (s) d s,
$$
where  $\alpha(t) = K\frac{t^2}{2} + t\mathbb{E}[|F(0, X_0)|] + C\sqrt{t}$  is non-decreasing. By Lemma C.10:  $u(t) \leq \alpha(t)\exp(Kt) \leq \alpha(t)\exp(KT)$ . Since  $\alpha(t) \leq C\sqrt{t}$  for  $t \in [0, 1]$ , the lemma holds.
Lemma C.12. (Discrete Gronwall Lemma) Consider a real number sequence  $(u_n)_n$  such that
$$
u _ {n + 1} \leq a _ {n + 1} u _ {n} + b _ {n + 1}, n \geq 0
$$
where  $(a_{n})$  and  $(b_{n})_{n}$  are two given sequences such that  $b_{n}$  is positive. Then
$$
u _ {n} \leq A _ {n} u _ {0} + \sum_ {k = 1} ^ {n} A _ {k, n} b _ {k},
$$
where  $A_{n} = \prod_{k = 1}^{n}a_{k}$  and  $A_{k,n} = A_n / A_k = \prod_{i = k + 1}^n a_i$
C.3 Proof of Proposition 4.3
Let us prove this statement for SEEDS-2. On the one hand, we have
$$
\mathbf {u} _ {1} = \frac {\alpha_ {s _ {1}}}{\alpha_ {s}} \tilde {\mathbf {x}} _ {s} - 2 \bar {\sigma} _ {s _ {1}} \left(e ^ {\frac {h}{2}} - 1\right) F _ {\theta} (\tilde {\mathbf {x}} _ {s}, s) - B ^ {1}, \quad B ^ {1} := \sqrt {2} \alpha_ {s _ {1}} \int_ {\lambda_ {s}} ^ {\lambda_ {s _ {1}}} e ^ {- \lambda} d \omega_ {\lambda}.
$$
Now  $B^1$  depends on the Brownian movement  $\left(\omega_{\lambda_{s_1}} - \omega_{\lambda_s}\right)_{\lambda_s \geqslant \lambda_s}$ . By the Markov property, this is independent of  $(\omega_{\lambda_u})_{u \leqslant s}$ . Since  $\tilde{\mathbf{x}}_s$  is a function of  $(\omega_{\lambda_u})_{u \leqslant s}$ , we deduce that  $B^1$  has to be independent of all  $(\tilde{\mathbf{x}}_u)_{u \geqslant s}$ . By Ito Isometry, we obtain
$$
B ^ {1} = \bar {\sigma} _ {s _ {1}} \sqrt {e ^ {h} - 1} z ^ {1}, \quad z ^ {1} \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d}).
$$
On the other hand, the update  $\tilde{\mathbf{x}}_t$  is
$$
\tilde {\mathbf {x}} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \tilde {\mathbf {x}} _ {s} - 2 \bar {\sigma} _ {t} (e ^ {h} - 1) F _ {\theta} (\mathbf {u} _ {1}, s _ {1}) - B ^ {0}, \qquad B ^ {0} = \sqrt {2} \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \mathrm {d} \omega_ {\lambda}.
$$
Hence, the Wiener process  $W = \{\omega_{\lambda} : \lambda \in [\lambda_s, \lambda_t]\}$  is predetermined on the interval  $[\lambda_s, \lambda_{s_1}]$ . Then, by using independent increments property of Wiener process  $W$ , we can deduce, for  $0 \leq \lambda_s < \lambda_{s_1} < \lambda_t$ , that
$$
\omega_ {\lambda_ {s _ {1}}} - \omega_ {\lambda_ {s}} \quad \text {a n d} \quad \omega_ {\lambda_ {t}} - \omega_ {\lambda_ {s _ {1}}} \text {a r e i n d e p e n d e n t .}
$$
Then, by the above Brownian independence property, the random variables  $B^0$  and  $B^1$  are
- Independent on non-overlapping time intervals
- Dependent on overlapping intervals.
By the Chasles rule, we then decompose
$$
B ^ {0} = \sqrt {2} \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {s _ {1}}} e ^ {- \lambda} \mathrm {d} \omega_ {\lambda} + \sqrt {2} \alpha_ {t} \int_ {\lambda_ {s _ {1}}} ^ {\lambda_ {t}} e ^ {- \lambda} \mathrm {d} \omega_ {\lambda}
$$
and, as
$$
\int_ {\lambda_ {s}} ^ {\lambda_ {s _ {1}}} e ^ {- \lambda} \mathrm {d} \omega_ {\lambda} = \frac {B ^ {1}}{\sqrt {2} \alpha_ {s _ {1}}},
$$
$$
B ^ {0} = \sqrt {2} \alpha_ {t} \left[ \frac {B ^ {1}}{\sqrt {2} \alpha_ {s _ {1}}} + \int_ {\lambda_ {s _ {1}}} ^ {\lambda_ {t}} e ^ {- \lambda} d \omega_ {\lambda} \right].
$$
$$
\begin{array}{l} { B } { = } { \sqrt { 2 } \alpha _ { t } \int _ { \lambda _ { s } } ^ { \lambda _ { s _ { 1 } } } e ^ { - \lambda } \mathrm { d } \omega _ { \lambda } + \sqrt { 2 } \alpha _ { t } \int _ { \lambda _ { s } } ^ { \lambda _ { t } } e ^ { - \lambda } \mathrm { d } \omega _ { \lambda } } \\ { = } { \sqrt { 2 } \alpha _ { t } \frac { 1 } { \sqrt { 2 } } \sigma _ { s _ { 1 } } \sqrt { e ^ { 2 ( \lambda _ { s _ { 1 } } - \lambda _ { s } ) } - 1 } z ^ { 1 } + \sqrt { 2 } \alpha _ { t } \frac { 1 } { \sqrt { 2 } } \sigma _ { t } \sqrt { e ^ { 2 ( \lambda _ { t } - \lambda _ { s } ) } - 1 } z ^ { 2 } } \\ { = } { \alpha _ { t } \sigma _ { s _ { 1 } } \sqrt { e ^ { h } - 1 } z ^ { 1 } + \alpha _ { t } \sigma _ { t } \sqrt { e ^ { 2 h } - 1 } z ^ { 2 } } \\ = \bar {\sigma} _ {t} \left(\frac {\sigma_ {s _ {1}}}{\sigma_ {t}} \sqrt {e ^ {h} - 1} z ^ {1} + \sqrt {e ^ {2 h} - 1} z ^ {2}\right) \\ { = } { \bar { \sigma } _ { t } \left( e ^ { \lambda _ { t } - \lambda _ { s _ { 1 } } } \sqrt { e ^ { h } - 1 } z ^ { 1 } + \sqrt { e ^ { 2 h } - 1 } z ^ { 2 } \right) } \\ { = } { \bar { \sigma } _ { t } \left( e ^ { \frac { h } { 2 } } \sqrt { e ^ { h } - 1 } z ^ { 1 } + \sqrt { e ^ { 2 h } - 1 } z ^ { 2 } \right) } \\ = \bar {\sigma} _ {t} \left(\sqrt {e ^ {2 h} - e ^ {h}} z ^ {1} + \sqrt {e ^ {2 h} - 1} z ^ {2}\right), \\ \end{array}
$$
The proof for SEEDS-3 is straightforward from the proof for SEEDS-2.
C.4 Proof of Corollary 4.4
C.4.1 Convergence of SEEDS-2:
Let  $\{\mathbf{y}_{t_i}\}_{i}$  be the discrete stochastic process defined as follows:
$$
\mathbf {y} _ {t _ {0}} \leftarrow \mathbf {x} _ {T}, \mathbf {y} _ {t _ {i}} \leftarrow \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \mathbf {y} _ {t _ {i - 1}} - 2 \alpha_ {t _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- u} \boldsymbol {\epsilon} (\boldsymbol {u} _ {i}, s _ {i}) \mathrm {d} u - \sqrt {2} \alpha_ {t _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- s} \mathrm {d} \bar {\boldsymbol {\omega}} (s),
$$
with  $s_i \gets t_\lambda \left(\lambda_{t_{i-1}} + \frac{h_i}{2}\right)$  and
$$
\pmb {u} _ {i} \gets \frac {\alpha_ {s _ {i}}}{\alpha_ {t _ {i - 1}}} \mathbf {y} _ {t _ {i - 1}} - 2 \sigma_ {s _ {i}} \left(e ^ {\frac {h _ {i}}{2}} - 1\right) \pmb {\epsilon} (\mathbf {y} _ {t _ {i - 1}}, t _ {i - 1}) - \sqrt {2} \alpha_ {s _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {i}}} e ^ {- s} \mathrm {d} \bar {\pmb {\omega}} (s),
$$
Then  $\{\mathbf{y}_{t_i}\}_i$  has the same distribution as  $\{\widetilde{\mathbf{x}}_{t_i}\}_i$  in Algorithm 3. We can compute the difference:
$$
\mathbf {y} _ {t _ {i}} - \mathbf {x} _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} (\mathbf {y} _ {t _ {i - 1}} - \mathbf {x} _ {t _ {i - 1}}) + \Gamma ; \Gamma = \Gamma_ {1} + \Gamma_ {2}
$$
$$
\Gamma_ {1} = - 2 \alpha_ {t _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- u} [ \epsilon (\boldsymbol {u} _ {i}, s _ {i}) - \epsilon (x _ {s _ {i}}, s _ {i}) ] d u
$$
$$
\Gamma_ {2} = - 2 \alpha_ {t _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- u} [ \boldsymbol {\epsilon} (x _ {s _ {i}}, s _ {i}) - \widehat {\boldsymbol {\epsilon}} (\widehat {\mathbf {x}} _ {u}, u) ] d u
$$
Similarly to the case  $k = 1$ ,  $\mathbb{E}[|\Gamma_2|] \leq Ch^2$ . Note that  $\mathbb{E}[|\Gamma_1|] \leq Ch\Delta_i$  with  $\Delta_i = \mathbb{E}[|\pmb{u}_i - \mathbf{x}_{s_i}|]$ . Introduce:  $U_{i} = \mathbb{E}[|\mathbf{y}_{t_{i}} - \mathbf{x}_{t_{i}}|]$ . We will bound  $\Delta_i$  by a function of  $U_{i-1}$ . For this, recall:
$$
\boldsymbol {u} _ {i} = \frac {\alpha_ {s _ {i}}}{\alpha_ {t _ {i - 1}}} \mathbf {y} _ {t _ {i - 1}} - 2 \alpha_ {s _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {i}}} e ^ {- \lambda} \boldsymbol {\epsilon} (\mathbf {y} _ {t _ {i - 1}}, t _ {i - 1}) d \lambda - \sqrt {2} \alpha_ {s _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {i}}} e ^ {- \lambda} d \bar {\boldsymbol {\omega}} (\lambda),
$$
and write the difference:
$$
\boldsymbol {u} _ {i} - \mathbf {x} _ {s _ {i}} = \frac {\alpha_ {s _ {i}}}{\alpha_ {t _ {i - 1}}} (\mathbf {y} _ {t _ {i - 1}} - \mathbf {x} _ {t _ {i - 1}}) + 2 \alpha_ {t _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- u} (\widehat {\boldsymbol {\epsilon}} (\widehat {\mathbf {x}} _ {u}, u) - \boldsymbol {\epsilon} _ {\theta} (\mathbf {y} _ {t _ {i - 1}}, \lambda_ {t _ {i - 1}})) d \lambda .
$$
By the triangular inequality:
$$
\left| \hat {\epsilon} \left(\widehat {\mathbf {x}} _ {u}, u\right) - \epsilon \left(\mathbf {y} _ {t _ {i - 1}}, \lambda_ {t _ {i - 1}}\right) \right| \leq C \left(\left| \mathbf {x} _ {t _ {\lambda} (u)} - \mathbf {x} _ {t _ {i - 1}} \right| + \left| \mathbf {x} _ {t _ {i - 1}} - \mathbf {y} _ {t _ {i - 1}} \right| + \left| t _ {\lambda} (u) - t _ {i - 1} \right|\right),
$$
and so:  $\mathbb{E}[[\hat{\epsilon} (\widehat{\mathbf{x}}_u,u) - \epsilon (\mathbf{y}_{t_{i - 1}},\lambda_{t_{i - 1}})]]\leq C(\sqrt{h} +U_{i - 1} + h)$ . Finally:
$$
\Delta_ {i} \leq \frac {\alpha_ {s _ {i}}}{\alpha_ {t _ {i - 1}}} U _ {i - 1} + C h \left(\sqrt {h} + U _ {i - 1} + h\right) \leq U _ {i - 1} + C h \left(\sqrt {h} + U _ {i - 1}\right),
$$
and  $\mathbb{E}[|\Gamma_1|] \leq Ch^{5/2} + C(h + h^2)U_{i-1}$ . This gives the bound:
$$
U _ {i} \leq C h ^ {2} + (\frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} + C h + C h ^ {2}) U _ {i - 1}.
$$
Now using Theorem C.6, the proof can now be finished following as in proof in Section C.2.4.
C.4.2 Convergence of SEEDS-3:
Continuing with the same notations as before, we will prove by analogy that:
$$
U _ {i} \leq C h ^ {2} + \left(\frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} + C h + C h ^ {2} + C h ^ {3}\right) U _ {i - 1}, \tag {49}
$$
so that, by Theorem C.6, we obtain the desired result.
Let  $\mathbf{y}_{t_i} = \widetilde{\mathbf{x}}_{t_i}$ . We have:
$$
\begin{array}{l} y _ {t _ {i}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} y _ {t _ {i - 1}} - 2 \alpha_ {t _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- \lambda} \epsilon (y _ {t _ {i - 1}}, t _ {i - 1}) d \lambda \\ - \frac {2 \alpha_ {t _ {i}}}{h _ {i} r _ {2}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- \lambda} (\lambda - \lambda_ {t _ {i - 1}}) (\epsilon (u _ {2 i}, s _ {2 i}) - \epsilon (y _ {t _ {i - 1}}, t _ {i - 1})) d \lambda + \text {N o i s e}, \\ \end{array}
$$
and  $x_{t_i} = \frac{\alpha_{t_i}}{\alpha_{t_{i - 1}}} x_{t_{i - 1}} - 2\alpha_{t_i}\int_{\lambda_{t_{i - 1}}}^{\lambda_{t_i}}e^{-\lambda}\widehat{\epsilon} (\widehat{\mathbf{x}}_\lambda ,\lambda)\mathrm{d}\lambda +\mathrm{Noise}$ , where Noise is the same in both equations. From the proof for  $k = 1$ , it suffices to bound:
$$
\Gamma = \frac {1}{h _ {i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {t _ {i}}} e ^ {- \lambda} (\lambda - \lambda_ {t _ {i - 1}}) (\boldsymbol {\epsilon} (u _ {2 i}, s _ {2 i}) - \boldsymbol {\epsilon} (y _ {t _ {i - 1}}, t _ {i - 1})) d \lambda ,
$$
in the  $L^1$  norm. By the Lipschitz property of  $\epsilon$ :
$$
\begin{array}{l} \mathbb {E} [ | \Gamma | ] \leq C h (\mathbb {E} [ | u _ {2 i} - x _ {s 2 i} | ] + \mathbb {E} [ | x _ {t _ {i - 1}} - x _ {s 2 i} | ] + \mathbb {E} [ | y _ {t _ {i - 1}} - x _ {t _ {i - 1}} | ]) + C h ^ {2} \\ \leq C h \left(\mathbb {E} \left[ \left| u _ {2 i} - x _ {s 2 i} \right| \right] + \sqrt {h} + U _ {i - 1}\right) + C h ^ {2}. \\ \end{array}
$$
Now, let us bound  $\mathbb{E}[|u_{2i} - x_{s_{2i}}|]$  and for this, write
$$
\begin{array}{l} u _ {2 i} = \frac {\alpha_ {s _ {2 i}}}{\alpha_ {t _ {i - 1}}} y _ {t _ {i - 1}} - 2 \alpha_ {s _ {2 i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {2 i}}} e ^ {- \lambda} \epsilon (y _ {t _ {i - 1}}, t _ {i - 1}) d \lambda \\ - 2 \alpha_ {s _ {2 i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {2 i}}} e ^ {- \lambda} (\lambda - \lambda_ {t _ {i - 1}}) \frac {\epsilon (u _ {2 i - 1} , s _ {2 i - 1}) - \epsilon (y _ {t _ {i - 1}} , t _ {i - 1})}{r _ {1} h _ {i}} d \lambda + N o i s e, \\ \end{array}
$$
$$
x _ {s _ {2 i}} = \frac {\alpha_ {s _ {2 i}}}{\alpha_ {t _ {i - 1}}} x _ {t _ {i - 1}} - 2 \alpha_ {s _ {2 i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {2 i}}} e ^ {- \lambda} \widehat {\epsilon} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) d \lambda + \text {N o i s e},
$$
where Noise is the same in both equations. So
$$
\begin{array}{l} u _ {2 i} - x _ {s _ {2 i}} = \frac {\alpha_ {s _ {2 i}}}{\alpha_ {t _ {i - 1}}} \left(y _ {t _ {i - 1}} - x _ {t _ {i - 1}}\right) - 2 \alpha_ {s _ {2 i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {2 i}}} e ^ {- \lambda} \left(\boldsymbol {\epsilon} \left(\mathbf {y} _ {t _ {i - 1}}, t _ {i - 1}\right) - \widehat {\boldsymbol {\epsilon}} (\widehat {\mathbf {x}} _ {\lambda}, \lambda)\right) d \lambda \\ - 2 \alpha_ {s _ {2 i}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {2 i}}} e ^ {- \lambda} (\lambda - \lambda_ {t _ {i - 1}}) D \mathrm {d} \lambda , \\ \end{array}
$$
where  $D = \frac{\epsilon(u_{2i-1}, s_{2i-1}) - \epsilon(y_{t_{i-1}}, t_{i-1})}{r_1 h_i}$ . From the convergence proof of SEEDS-1, and the Lipschitz property of  $\epsilon$ , we obtain:
$$
\mathbb {E} \left[ \left| u _ {2 i} - x _ {s 2 i} \right| \right] \leq U _ {i - 1} + C h U _ {i - 1} + C h ^ {2} + C h \mathbb {E} \left[ \left| u _ {2 i - 1} - y _ {t _ {i - 1}} \right| \right].
$$
Again, by the triangular inequality:
$$
\mathbb {E} \big [ | u _ {2 i - 1} - y _ {t _ {i - 1}} | \big ] \leq \mathbb {E} \big [ | u _ {2 i - 1} - x _ {s _ {2 i - 1}} | \big ] + C \sqrt {h} + U _ {i - 1},
$$
$$
u _ {2 i - 1} - x _ {s _ {2 i - 1}} = \frac {\alpha_ {s _ {2 i - 1}}}{\alpha_ {t _ {i - 1}}} (y _ {t _ {i - 1}} - x _ {t _ {i - 1}}) - 2 \alpha_ {s _ {2 i - 1}} \int_ {\lambda_ {t _ {i - 1}}} ^ {\lambda_ {s _ {2 i - 1}}} e ^ {- \lambda} (\widehat {\pmb {\epsilon}} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) - \pmb {\epsilon} (y _ {t _ {i - 1}}, t _ {i - 1}) \mathrm {d} \lambda ,
$$
we have, as before,
$$
\mathbb {E} \left[ \left| u _ {2 i - 1} - x _ {s _ {2 i}} \right| \right] \leq U _ {i - 1} + C h U _ {i - 1} + C h ^ {2}.
$$
Combining the previous inequalities leads to (49). This finishes the proof of Corollary 4.4.
The SEEDS solvers used in our experiments are exactly the variants we described in Algorithms 2, 3, 4 in the main part of the paper. In particular, SEEDS-2 and SEEDS-3 solvers are completely determined by one-parameter families of deterministic exponential integrators from [14] of order 2 and 3 respectively and prescribed by the following Butcher tableaux:
$$
\begin{array}{c c c} 0 & 0 & 0 \\ c _ {2} & c _ {2} \varphi_ {1, 2} & 0 \\ \hline & \left(1 - \frac {1}{2 c _ {2}}\right) \varphi_ {1} & \frac {1}{2 c _ {2}} \varphi_ {1} \end{array} \qquad \begin{array}{c c c} 0 & 0 & 0 \\ c _ {2} & c _ {2} \varphi_ {1, 2} & 0 \\ \frac {2}{3} & \frac {2}{3} \varphi_ {1, 3} - \frac {4}{9 c _ {2}} \varphi_ {2, 3} & \frac {4}{9 c _ {2}} \varphi_ {2, 3} \\ \hline & \varphi_ {1} - \frac {3}{2} \varphi_ {2} & 0 \end{array}
$$
In all experiments we fix the parameter  $c_{2} = 0.5$  for SEEDS-2 and  $c_{2} = \frac{1}{3}$  for SEEDS-3. We point out that the Butcher tableau here associated to SEEDS-3 is the result of a weakening of the stiff order conditions so it might suffer from order reduction in the deterministic case. We also point out that a full theory of stiff order conditions for stochastic exponential Runge-Kutta methods for semi-linear DEs with a time-varying linear coefficient have not been yet developed to the authors knowledge. As such, this is only an analogy whose purpose is to clarify how our solvers relate to well-known solvers from the literature, but such Butcher tableaux do not rigorously reflect their convergence order.
D.1 Stabilization of the exponential terms
In the proposed algorithms, one subtle detail is to re-arrange all equations in order for them to make use only of the  $\exp 1(h)$  function which computes  $e^h - 1$  with great numerical stability, specially
for small values of  $h$ . We have for values  $r_1 = 1/3$ ,  $r_2 = 2/3$ , the following identity
$$
\begin{array}{l} \sqrt {e ^ {2 h _ {i}} - e ^ {h _ {i}}} z _ {i} + \sqrt {e ^ {h _ {i}} - 1} v _ {i} = \sqrt {e ^ {h _ {i}} - 1} \left(\sqrt {e ^ {h _ {i}}} z _ {i} + v _ {i}\right) \\ { = } { \sqrt { e ^ { h _ { i } } - 1 } \left( e ^ { \frac { h _ { i } } { 2 } } z _ { i } + v _ { i } \right) } \\ = \sqrt {\exp m 1 \left(h _ {i}\right)} \left(\left(\exp m 1 \left(\frac {h _ {i}}{2}\right) + 1\right) z _ {i} + v _ {i}\right) \\ = \sqrt {\operatorname {e x p m 1} \left(h _ {i}\right)} \left(\operatorname {e x p m 1} \left(\frac {h _ {i}}{2}\right) + 1\right) z _ {i} + \sqrt {\operatorname {e x p m 1} \left(h _ {i}\right)} v _ {i}. \\ \end{array}
$$
Now by using
$$
e ^ {2 (r _ {2} - r _ {1}) h _ {i}} = e ^ {2 \left(\frac {2}{3} - \frac {1}{3}\right) h _ {i}} = e ^ {2 \frac {1}{3} h _ {i}} = e ^ {r _ {2} h _ {i}},
$$
we now compute
$$
\begin{array}{l} \sqrt {e ^ {2 r _ {2} h _ {i}} - e ^ {r _ {2} h _ {i}}} z _ {i} ^ {1} + \sqrt {e ^ {r _ {2} h _ {i}} - 1} z _ {i} ^ {2} = \sqrt {e ^ {r _ {2} h _ {i}} - 1} \left(\sqrt {e ^ {r _ {2} h _ {i}}} z _ {i} ^ {1} + z _ {i} ^ {2}\right) \\ { = } { \sqrt { e ^ { r _ { 2 } h _ { i } } - 1 } \left( e ^ { \frac { h _ { i } } { 3 } } z _ { i } ^ { 1 } + z _ { i } ^ { 2 } \right) } \\ = \sqrt {\operatorname {e x p m 1} \left(r _ {2} h _ {i}\right)} \left(\left(\operatorname {e x p m 1} \left(r _ {1} h _ {i}\right) + 1\right) z _ {i} ^ {1} + z _ {i} ^ {2}\right) \\ \end{array}
$$
$$
\begin{array}{l} \sqrt {e ^ {2 h _ {i}} - e ^ {2 r _ {2} h _ {i}}} z _ {i} ^ {1} = \sqrt {e ^ {3 r _ {2} h _ {i}} - e ^ {2 r _ {2} h _ {i}}} z _ {i} ^ {1} \\ = e ^ {r _ {2} h _ {i}} \sqrt {e ^ {r _ {2} h _ {i}} - 1 z _ {i} ^ {1}} \\ = \sqrt {\exp m 1 \left(r _ {2} h _ {i}\right)} \left(\exp m 1 \left(r _ {2} h _ {i}\right) + 1\right) z _ {i} ^ {1}. \\ \end{array}
$$
$$
\begin{array}{l} \sqrt {e ^ {2 h _ {i}} - e ^ {2 r _ {2} h _ {i}}} z _ {i} ^ {1} + \sqrt {e ^ {2 r _ {2} h _ {i}} - e ^ {r _ {2} h _ {i}}} z _ {i} ^ {2} + \sqrt {e ^ {r _ {2} h _ {i}} - 1 z _ {i} ^ {3}} \\ = \sqrt {\operatorname {e x p m 1} \left(r _ {2} h _ {i}\right)} \left(\left(\operatorname {e x p m 1} \left(r _ {2} h _ {i}\right) + 1\right) z _ {i} ^ {1} + \left(\operatorname {e x p m 1} \left(r _ {1} h _ {i}\right) + 1\right) z _ {i} ^ {2} + z _ {i} ^ {3}\right). \\ \end{array}
$$
D.2 Noise schedules parameterizations
The inverses of  $\lambda(t)$  are given in the (VP) linear and cosine schedules respectively by
$$
{t _ {\lambda} (\lambda)} = {\frac {2 \log (e ^ {- 2 \lambda} + 1)}{\sqrt {\beta_ {\mathrm {m i n}} ^ {2} + 2 (\beta_ {\mathrm {m a x}} - \beta_ {\mathrm {m i n}}) \log (e ^ {- 2 \lambda} + 1)} + \beta_ {\mathrm {m i n}}},}
$$
$$
t _ {\lambda} (\lambda) = \frac {2 (1 + s)}{\pi} \operatorname {a r c c o s} \left(\exp \left(- \frac {1}{2} \log (e ^ {- 2 \lambda} + 1) + \log \cos \left(\frac {\pi s}{2 (1 + s)}\right)\right)\right) - s.
$$
In EDM Noise Prediction case, the inverses of  $\lambda (t)$ , as given in (37), with respect to the NPFO and NRSDE are respectively given by
$$
t _ {\lambda} (\lambda) = \sigma_ {d} \tan (e ^ {- \lambda}) \quad \mathrm {a n d} \quad t _ {\lambda} (\lambda) = \frac {\sigma_ {d}}{\sqrt {\frac {1}{\sigma_ {d} ^ {2} e ^ {- 2 \lambda}} - 1}}.
$$
D.3 EDM discretization
We follow Karras et al. [16] to implement the EDM discretization timesteps  $\{t_i\}_{i=0}^M$  as  $t_i = \sigma^{-1}(\sigma_i)$  such that, for  $\rho > 0$ ,
$$
\sigma_ {i <   M} = \left(\sigma_ {\max } ^ {\frac {1}{\rho}} + \frac {i}{N - 1} \left(\sigma_ {\min } ^ {\frac {1}{\rho}} - \sigma_ {\max } ^ {\frac {1}{\rho}}\right)\right) ^ {\rho} \quad \text {a n d} \quad \sigma_ {N} = 0.
$$
From the definition, we note that  $\sigma_0 = \sigma_{\mathrm{max}}$  and  $\sigma_{M - 1} = \sigma_{\mathrm{min}}$ , where  $\sigma_{\mathrm{min}}$  and  $\sigma_{\mathrm{max}}$  denote the minimum and maximum noise magnitude, respectively. We also keep default value  $\rho = 7$  as in [16]. However, we figured out that when using EDM discretization with linear schedule, the noise schedule improvement in iDDPM pre-trained models [26] would result in two consecutive time steps of the same value, i.e.  $t_j = t_{j + 1}$  for some index  $j = 1,\dots ,M - 1$  and large steps  $(M\geq 61)$ . Thus, for SEEDS-3 and DPM-Solver-3 [22], the usage of the function  $\varphi_{2}(h) = \frac{e^{h} - h - 1}{h^{2}}$  (see Appendix E for more details) will cause zero division error where  $h = \lambda_{t_{j + 1}} - \lambda_{t_j} = 0$ . Therefore, in our implementation, we ignore the noise schedule improvement of iDDPM [26] models when using solvers of order three.
D.4 Final sampling step
The sampling phase in DPMs using SEEDS follows the RSDE, which requires gradual computing through discretization time steps  $\{t_i\}_{i = 0}^M$  and the latter goes from  $t_0 = T$  to  $t_M = 0$ . In our implementation, to avoid the logarithm of zero error at the last step, i.e.  $\log \sigma_{t_M} = \log (0)$ , we stop the sampling phase at step  $(M - 1)$ . Hence, the NFE used in a run will be given as
$$
\mathrm {N F E} = k \times (M - 1),
$$
where  $k$  represents the order of the solver. We also do not use the "denoising" trick, i.e., ignoring the random noise at the last step and leave it to further research.
E Reminders on Stochastic Exponential Integrators
Let us consider a SDE of the form
$$
\mathrm {d} \mathbf {x} (t) = [ a (t) \mathbf {x} (t) + c (t) f (\mathbf {x} (t), t) ] \mathrm {d} t + g (t) \mathrm {d} \boldsymbol {\omega} (t), \tag {50}
$$
where  $a, c: [0, T] \to \mathbb{R}$  and  $g: [0, T] \to \mathbb{R}^{d \times d}$ . In other words we concentrate to high-dimensional semi-linear non-autonomous SDEs with additive noise. The objective of this section is to construct explicit stochastic exponential derivative-free methods for the above equation following the Runge-Kutta (RK) approach. These methods ideally should fulfill the following properties:
1. If  $f \equiv 0$ , then (50) can be solved exactly;
2. If  $g \equiv 0$ , then a SEEDS method for (50) identifies with an exponential RK (ERK) method;
3. If  $a \equiv 0$ , then a SEEDS method for (50) identifies with a stochastic RK (SRK) method and if moreover  $g \equiv 0$  then it identifies with a classical RK ODE method.
Before tackling the aimed SEEDS problem let us rapidly recall elementary constructions of RK, ERK, weak and strong SRK methods. We will not deal with time-adaptive methods here.
E.1 Derivative-free exponential ODE schemes
E.1.1 Runge-Kutta approach
Derivative-free schemes are obtained by comparing the Ito-Taylor expansion of the above paragraph with expressions of  $\mathbf{x}(t)$  in terms of its intermediate evaluations between  $s$  and  $t = s + h$  and the Taylor expansions of such evaluations. As a simple example, in the ODE regime
$$
\mathrm {d} \mathbf {x} (t) = f (\mathbf {x} (t), t) \mathrm {d} t. \tag {51}
$$
As such, analytic solutions to the above equations are of the form
$$
{\bf x} (t) = {\bf x} (s) + \int_ {s} ^ {t} f ({\bf x} (\tau), \tau) \mathrm {d} \tau .
$$
Now, Taylor expansion gives, up to order 2: