SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martin Gonzalez* IRT SystemX
Nelson Fernandez Air Liquide
Thuy Tran IRT SystemX
Elies Gherbi IRT SystemX
Hatem Hajri  
IRT SystemX & Safran
Nader Masmoudi
A potent class of generative models known as Diffusion Probabilistic Models (DPMs) has become prominent. A forward diffusion process adds gradually noise to data, while a model learns to gradually denoise. Sampling from pre-trained DPMs is obtained by solving differential equations (DE) defined by the learnt model, a process which has shown to be prohibitively slow. Numerous efforts on speeding-up this process have consisted on crafting powerful ODE solvers. Despite being quick, such solvers do not usually reach the optimal quality achieved by available slow SDE solvers. Our goal is to propose SDE solvers that reach optimal quality without requiring several hundreds or thousands of NFEs to achieve that goal. We propose Stochastic Explicit Exponential Derivative-free Solvers (SEEDS), improving and generalizing Exponential Integrator approaches to the stochastic case on several frameworks. After carefully analyzing the formulation of exact solutions of diffusion SDEs, we craft SEEDS to analytically compute the linear part of such solutions. Inspired by the Exponential Time-Differencing method, SEEDS use a novel treatment of the stochastic components of solutions, enabling the analytical computation of their variance, and contains high-order terms allowing to reach optimal quality sampling  $\sim 3 - 5\times$  faster than previous SDE methods. We validate our approach on several image generation benchmarks, showing that SEEDS outperform or are competitive with previous SDE solvers. Contrary to the latter, SEEDS are derivative and training free, and we fully prove strong convergence guarantees for them. Our code is publicly available in this link.
Diffusion Probabilistic Models (DPMs) [32, 12] have emerged as a powerful category of generative models and have proven to quickly become SOTA for generative tasks such as image, video, audio generation [8, 24, 13, 3, 4], and more [29, 36]. These models employ a forward diffusion process where noise is gradually added to the data, and the model learns to remove the noise progressively. However, sampling from most pre-trained DPMs is done by simulating the trajectories of associated differential equations (DE) and has been found to be prohibitively slow [34]. Previous attempts to accelerate this process have mainly focused on developing efficient ODE solvers. On one hand, training-based methods speed-up sampling by using auxiliary training such as Progressive Distillation [31] and Fourier Neural Operators [42], learning the noise schedule, scaling, variance, or trajectories. On the other hand, training-free methods [16, 22, 15, 40, 23] are slower but are more versatile for
*Corresponding author: martin.gonzalez@irt-systemx.fr
being employed on different models and achieve higher quality results than current training-based methods. Although these solvers are fast, they often fall short of achieving the optimal quality attained by slower SDE solvers [16]. The latter usually do not present theoretical convergence guarantees and, while being training-free, they often still require costly parameter optimization to achieve optimal results which might be difficult to estimate for large datasets.
Our objective is to introduce SDE solvers that can achieve optimal quality without requiring an excessively large number of function evaluations (NFEs). To accomplish this, we propose Stochastic Explicit Exponential Derivative-free Solvers (SEEDS). These are off-the-shelf SDE samplers: they offer promising high-quality sampling without further training or parameter optimization. SEEDS enhance and generalize existing Exponential Integrator [22, 23, 41] approaches to the stochastic case on several practical frameworks and is based on the following 4 building blocks: (1) the exponential representation of semi-linear SDE exact solutions which isolate linear terms to be computed analytically; (2) a general change-of-variables recipe to simplify the integrals involved in the solutions in order to better approximate the deterministic one; (3) a method to analytically compute the variance of the stochastic one; (4) a method to decompose the obtained stochastic components in such a way that the resulting sequences of higher-stage numerical approximations yield Markov chains.
Overall, we make the following contributions: (a) our change-of-variables method allows us to reframe the gDDIM solver [41] as a special case of SEEDS and to craft bespoke solvers for the EDM-preconditioned DPMs in [16] which attain equivalent sampling quality twice faster than the previous SOTA sampling method [16]; (b) based on the Stochastic Exponential Time-Differencing method, we analytically compute of our solver's stochastic components in terms of the so-called  $\varphi$ -functions, allowing for efficient implementation; (c) our noise decomposition method (4), which is the key of success of SEEDS, is both theoretically grounded, experimentally shown to be optimal, and has no deterministic equivalent; (d) we provide full proofs of strong/weak convergence guarantees for our SDE solvers which, to our knowledge, has no precedent in the DPM literature. In particular, although the formula used for the truncated Itô-Taylor expansion might seem similar to that of [22], our convergence theorems leverage the full Itô-Taylor expansion of solutions, making our proofs not incremental and different from [22]; (e) we conduct extensive experiments demonstrating that SEEDS establishes SOTA results among available solvers on several image generation benchmarks, or is competitive with existing SDE solvers while being 2-5 times faster than the latter.
Although our solvers theoretically apply to certain non-isotropic DPMs such as Critically-damped Langevin Dynamics (CLD) [10] (see Prop. 4.5 and Rem. 4.6), we will restrict our presentation to the isotropic case for which many notations become simpler.
2 Background on Diffusion Probabilistic Models
General Isotropic DE Formulation. The evolution of a data sample  $\mathbf{x}_0\in \mathbb{R}^d$  taken from an unknown data distribution  $p_{\mathrm{data}}$  into standard Gaussian noise can be defined as a forward diffusion process  $\{\mathbf{x}_t\}_{t\in [0,T]}$ , with  $T > 0$ , which is a solution to a linear SDE:
$$
\mathrm {d} \mathbf {x} _ {t} = f (t) \mathbf {x} _ {t} \mathrm {d} t + g (t) \mathrm {d} \boldsymbol {\omega} _ {t}, \quad f (t) := \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t}, \quad g (t) = \alpha_ {t} \sqrt {\frac {\mathrm {d} [ \sigma_ {t} ^ {2} ]}{\mathrm {d} t}}, \tag {1}
$$
where  $f(t), g(t) \in \mathbb{R}^{d \times d}$  are called the drift and diffusion coefficients respectively and  $\omega$  is a  $d$ -dimensional standard Wiener process, and  $\alpha_{t}, \sigma_{t} \in \mathbb{R}^{>0}$  are differentiable functions with bounded derivatives. In practice, when specifying the SDE (1),  $\sigma_{t}$  acts as a schedule controlling the noise levels of an input at time  $t$ , and  $\alpha_{t}$  as a time-dependent signal scaling controlling its dynamic range.
By denoting  $p_t(\mathbf{x}_t)$  the marginal distribution of  $\mathbf{x}_t$  at time  $t$ , functions  $\alpha_t$  and  $\sigma_t$  are designed so that the end-time distribution of the process process is  $p_T(\mathbf{x}_T) \approx \mathcal{N}(\mathbf{x}_T | \mathbf{0}, \tilde{\sigma}^2 \mathbf{I}_d)$  for some  $\tilde{\sigma} > 0$ . As (1) is linear, the transition probability  $p_{0t}(\mathbf{x}_t | \mathbf{x}_0)$  from  $\mathbf{x}_0$  to  $\mathbf{x}_t$  is Gaussian whose mean and variance can be expressed in terms of  $\alpha_t$  and  $\sigma_t$ . For simplicity, we will denote it
$$
p _ {0 t} \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right) = \mathcal {N} \left(\mathbf {x} _ {t}; \mu_ {t} \mathbf {x} _ {0}, \Sigma_ {t}\right), \quad \mu_ {t}, \Sigma_ {t} \in \mathbb {R} ^ {d \times d}.
$$
The evolution of the reverse time process of  $\{\mathbf{x}_t\}_{t\in [0,T]}$  (which we will still denote  $\{\mathbf{x}_t\}_{t\in [0,T]}$  for simplicity) is then driven by a backward differential equation
$$
\mathrm {d} \mathbf {x} _ {t} = [ f (t) \mathbf {x} _ {t} - \frac {1 + \ell^ {2}}{2} g ^ {2} (t) \nabla_ {\mathbf {x} _ {t}} \log p _ {t} (\mathbf {x} _ {t}) ] \mathrm {d} t + \ell g (t) \mathrm {d} \bar {\omega} _ {t}, \tag {2}
$$
where  $\mathrm{dt}$  are negative infinitesimal time-steps and  $\bar{\omega}_t$  is now a Wiener process with variance  $-\mathrm{d}t$ . In this article, we will concentrate in the cases  $\ell = 0,1$ , known in the literature as the Probability Flow ODE (PFO) and diffusion reverse SDE (RSDE), respectively.
**Training.** Denoising score-matching is a technique to train a time-dependent model  $D_{\theta}(\mathbf{x}_t, t)$  to approach the score function  $\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t)$  at each time  $t$ . Intuitively, as  $D_{\theta}$  approaches the score, it produces a sample which maximizes the log-likelihood. As such, this model is coined as a data prediction model. However, in practice DPMs can be more efficiently trained by reparameterizing  $D_{\theta}$  into a different model  $F_{\theta}(\mathbf{x}_t, t)$  whose objective is to predict the noise to be removed from a sample at time  $t$ . This noise prediction model is trained by means of the loss
$$
\mathbb {E} _ {t \sim \mathcal {U} [ 0, T ], \mathbf {x} _ {0} \sim p _ {\mathrm {d a t a}}, \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d})} [ \| \epsilon - F _ {\theta} (\mu_ {t} \mathbf {x} _ {0} + \pmb {K} _ {t} \epsilon , t) \| _ {\pmb {K} _ {t} ^ {- 1} \gamma_ {t} \pmb {K} _ {t} ^ {- \top}} ^ {2} ],
$$
where  $\gamma_{t}$  is a time dependent weighting parameter and  $\pmb{K}_t\pmb{K}_t^\top = \Sigma_t$
3 Accelerating Optimal Quality Solvers for Diffusion SDEs
Once  $F_{\theta}$  or  $D_{\theta}$  have been trained, one can effectively solve (2) after replacing the score function by its corresponding expression involving either one of these models. For instance, taking the noise prediction model and  $\ell = 1$ , sampling is conducted by simulating trajectories of a SDE of the form
$$
\mathrm {d} \mathbf {x} _ {t} = [ A (t) \mathbf {x} _ {t} + b (t) F _ {\theta} (\mathbf {x} _ {t}, t) ] \mathrm {d} t + g (t) \mathrm {d} \omega_ {t}, \tag {3}
$$
for some functions  $A(t), b(t)$  which are usually not equal to  $f(t), g^2(t)$ . In what follows, we consider a time discretization  $\{t_i\}_{i=0}^M$  going backwards in time starting from  $t_0 = T$  to  $t_M = 0$  and to ease the notation we will always denote  $t < s$  for two consecutive time-steps  $t_{i+1} < t_i$ .
The usual representation of the analytic solution  $\mathbf{x}_t$  at time  $t$  of (3) w.r.t. an initial condition  $\mathbf{x}_s$  is:
$$
\mathbf {x} _ {t} = \mathbf {x} _ {s} + \int_ {s} ^ {t} [ A (\tau) \mathbf {x} _ {\tau} + b (\tau) F _ {\theta} (\mathbf {x} _ {\tau}, \tau) ] \mathrm {d} \tau + \int_ {s} ^ {t} g (\tau) \mathrm {d} \omega_ {\tau}. \tag {4}
$$
The numerical schemes we propose for approaching the trajectories of (3) based on representation (4) are grounded in the 4 following principles:
1. The variation-of-parameters formula: representing analytic solutions with linear term extracted from the integrand;
2. Exponentially weighted integrals: extracting the time-varying linear coefficient attached to the network from the integrand by means of a specific choice of change of variables which allows analytic computation of the leading coefficients in the truncated Ito-Taylor expansion associated to  $F_{\theta}(\mathbf{x}_{\tau}, \tau)$  up to any arbitrary order;
3. Modified Gaussian increments: after replicating such change of variables onto the stochastic integral, analytically computing its variance.
4. Markov-preserving noise decomposition: stochastic integrals need to be dependent on overlapping time intervals and independent on non-overlapping ones.
Exponential representation of exact solutions of diffusion SDEs. The first key insight of this work is that, using the variation-of-parameters formula, we can represent the analytic solution  $\mathbf{x}_t$  at time  $t$  of (3) with respect to an initial condition  $\mathbf{x}_s$  as follows:
$$
\mathbf {x} _ {t} = \Phi_ {A} (t, s) \mathbf {x} _ {s} + \int_ {s} ^ {t} \Phi_ {A} (t, \tau) b (\tau) F _ {\theta} (\mathbf {x} _ {\tau}, \tau) \mathrm {d} \tau + \int_ {s} ^ {t} \Phi_ {A} (t, \tau) g (\tau) \mathrm {d} \omega_ {\tau}, \tag {5}
$$
where  $\Phi_A(t,s) = \exp \left(\int_s^t A(\tau)\mathrm{d}\tau\right)$  is called the transition matrix associated with  $A(t)$ . The separation of the linear and nonlinear components is achieved by this formulation and also appears in [22, 23, 41]. It differs from black-box SDE solvers as it enables the exact calculation of the linear portion, thereby removing any approximation errors associated with it. However, the integration of the nonlinear portion remains complex due to the interaction of the new coefficient  $\Phi_A(t,\tau)b(\tau)$  and the intricate neural network, making it challenging to approximate.
Exponentially weighted integrals. Due to the regularity conditions usually imposed on the drift and diffusion coefficients of (1), one can make several choices of change-of-variables on the integral components in (5) in order to simplify it. Our second key insight is that there is a specific choice of change of variables allowing the analytic computation of the Itô-Taylor coefficients of  $F_{\theta}(\mathbf{x}_{\tau}, \tau)$  with respect to  $\tau$ , and based at  $s$  that will be used for crafting SEEDS. More specifically, this expansion reads
$$
F _ {\theta} (\mathbf {x} _ {\tau}, \tau) = \sum_ {k = 0} ^ {n} \frac {(\tau - s) ^ {k}}{k !} F _ {\theta} ^ {(k)} (\mathbf {x} _ {s}, s) + \mathcal {R} _ {n},
$$
where the residual  $\mathcal{R}_n$  consists of deterministic iterated integrals of length greater than  $n + 1$  and all iterated integrals with at least one stochastic component. As such, we obtain
$$
\int_ {s} ^ {t} \Phi_ {A} (t, \tau) b (\tau) F _ {\theta} (\mathbf {x} _ {\tau}, \tau) \mathrm {d} \tau = \sum_ {k = 0} ^ {n} F _ {\theta} ^ {(k)} (\mathbf {x} _ {s}, s) \int_ {s} ^ {t} \Phi_ {A} (t, \tau) b (\tau) \frac {(\tau - s) ^ {k}}{k !} \mathrm {d} \tau + \tilde {\mathcal {R}} _ {n}, \tag {6}
$$
where  $\tilde{\mathcal{R}}_n$  is easily obtained from  $\mathcal{R}_n$  and  $\int_s^t\Phi_A(t,\tau)b(\tau)\mathrm{d}\tau$ . The third key contribution of our work is to rewrite, for any  $k\geqslant 0$ , the integral  $\int_s^t\Phi_A(t,\tau)b(\tau)\frac{(\tau - s)^k}{k!}\mathrm{d}\tau$  as an integral of the form  $\int_{\lambda_s}^{\lambda_t}e^{\lambda \frac{(\lambda - \lambda_s)^k}{k!}}\mathrm{d}\lambda$  since the latter is recursively analytically computed in terms of the  $\varphi$ -functions
$$
\varphi_ {0} (t) := e ^ {t}, \qquad \varphi_ {k + 1} (t) := \int_ {0} ^ {1} e ^ {(1 - \tau) t} \frac {\tau^ {k}}{k !} \mathrm {d} \tau = \frac {\varphi_ {k} (t) - \varphi_ {k} (0)}{t}, \qquad k \geqslant 0.
$$
Modified Gaussian increments. In order for making such change of variables to be consistent on the overall system, one needs to replicate it accordingly in the stochastic integral  $\int_s^t\Phi_A(t,\tau)g(\tau)\mathrm{d}\bar{\omega}_\tau$ . As such, our last key contribution is to transform it into an exponentially weighted stochastic integral with integration endpoints  $\lambda_{s},\lambda_{t}$  and apply the Stochastic Exponential Time Differencing (SETD) method [1] to compute its variance analytically, as illustrated in (14) below.
Let us test our methodology in two key examples. As we explained in Section 2, sampling from pre-trained DPMs amounts on choosing a schedule  $\sigma_{t}$ , a scaling  $\alpha_{t}$ , and a parameterized learnt approximation of the score function  $\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t)$ . In what follows, we denote by  $t_\lambda$  the inverse of a chosen change of variables  $\lambda_{t}$  and we denote  $\widehat{\mathbf{x}}_{\lambda} \coloneqq \mathbf{x}(t_{\lambda}(\lambda)), \hat{F}_{\theta}(\widehat{\mathbf{x}}_{\lambda}, \lambda) \coloneqq F_{\theta}(\mathbf{x}(t_{\lambda}(\lambda)), t_{\lambda}(\lambda))$ .
The VPSDE case. Let  $\tilde{\alpha}_t \coloneqq \frac{1}{2}\beta_d t^2 + \beta_m t$ , where  $\beta_d, \beta_m > 0$  and  $t \in [0,1]$ . Then, by denoting
$$
\sigma_ {t} := \sqrt {e ^ {\tilde {\alpha} _ {t}} - 1}, \qquad \alpha_ {t} := e ^ {- \frac {1}{2} \tilde {\alpha} _ {t}}, \qquad \bar {\sigma} _ {t} := \alpha_ {t} \sigma_ {t}, \qquad \nabla_ {\mathbf {x} _ {t}} \log p _ {t} (\mathbf {x} _ {t}) \simeq \bar {\sigma} _ {t} ^ {- 1} F _ {\theta} (\mathbf {x} _ {t}, t), \tag {7}
$$
we obtain the VP SDE framework from [34] and the following result.
Proposition 3.1. Let  $t < s$ . The analytic solution at time  $t$  of the RSDE (2) with coefficients (7) and initial value  $\mathbf{x}_s$  is
$$
\mathbf {x} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \mathbf {x} _ {s} - 2 \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \hat {F} _ {\theta} (\hat {\mathbf {x}} _ {\lambda}, \lambda) d \lambda - \sqrt {2} \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} d \bar {\omega} _ {\lambda}, \quad \lambda_ {t} := - \log (\sigma_ {t}). \tag {8}
$$
The change of variables of (8) is interesting as it allows to compute analytically the Itô-Taylor coefficients in (6) by using, for  $h = \lambda_t - \lambda_s$ , the following key result which will be used in Prop. 4.2:
$$
\int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \frac {\left(\lambda - \lambda_ {s}\right) ^ {k}}{k !} \mathrm {d} \lambda = \sigma_ {t} h ^ {k + 1} \varphi_ {k + 1} (h). \tag {9}
$$
For instance, in the case when  $k = 0$ , it is easy to see that  $\int_{\lambda_s}^{\lambda_t}e^{-\lambda}\mathrm{d}\lambda = \sigma_t(e^h -1)$  and  $\int_{\lambda_s}^{\lambda_t}e^{-\lambda}\mathrm{d}\bar{\omega}_\lambda$  obeys a normal distribution with zero mean, and one can analytically compute its variance:
$$
\int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- 2 \lambda} d \lambda = \frac {\sigma_ {t} ^ {2}}{2} \left(e ^ {2 h} - 1\right). \tag {10}
$$
The EDM case. Denote  $\sigma_d^2$  the variance of the considered initial dataset and set
$$
\sigma_ {t} := t, \alpha_ {t} := 1, \nabla_ {\mathbf {x} _ {t}} \log p _ {t} (\mathbf {x} _ {t}) \simeq \frac {1}{t ^ {2}} \left[ \frac {\sigma_ {d} ^ {2} \mathbf {x} _ {t}}{t ^ {2} + \sigma_ {d} ^ {2}} + \frac {t \sigma_ {d}}{\sqrt {t ^ {2} + \sigma_ {d} ^ {2}}} F _ {\theta} \left(\frac {\mathbf {x} _ {t}}{\sqrt {t ^ {2} + \sigma_ {d} ^ {2}}}, \frac {\log (t)}{4}\right) \right]. \tag {11}
$$
These parameters correspond to the preconditioned EDM framework introduced in [16, Sec. 5, App. B.6]. The following result is the basis for constructing our customized SEEDS in this case, and for which we report experimental results in Table 1. For simplicity, we will write  $F_{\theta}(\mathbf{x}_t,t)$  for the preconditioned model in (11) and we refer to Appendix B for details.
Proposition 3.2. Let  $t < s$ . The analytic solution at time  $t$  of (2) with coefficients (11) and initial value  $\mathbf{x}_s$  is, for  $\ell = 1$ ,
$$
\mathbf {x} _ {t} = \frac {t ^ {2} + \sigma_ {d} ^ {2}}{s ^ {2} + \sigma_ {d} ^ {2}} \mathbf {x} _ {s} + 2 \left(t ^ {2} + \sigma_ {d} ^ {2}\right) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \hat {F} _ {\theta} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) d \lambda - \sqrt {2} \left(t ^ {2} + \sigma_ {d} ^ {2}\right) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} d \bar {\omega} _ {\lambda}, \tag {12}
$$
where  $\lambda_t \coloneqq -\log \left[\frac{t}{\sigma_d\sqrt{t^2 + \sigma_d^2}}\right]$ . In the case when  $\ell = 0$ , it is given by
$$
\mathbf {x} _ {t} = \sqrt {\frac {t ^ {2} + \sigma_ {d} ^ {2}}{s ^ {2} + \sigma_ {d} ^ {2}}} \mathbf {x} _ {s} + \sqrt {t ^ {2} + \sigma_ {d} ^ {2}} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \hat {F} _ {\theta} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) d \lambda , \quad \lambda_ {t} := - \log \left[ \arctan \left[ \frac {t}{\sigma_ {d}} \right] \right]. \tag {13}
$$
Remark 3.3. One can wonder about the generality of such change of variables. Our method is very general in that one can always make such change of variables with very mild regularity conditions: for  $c:[0,T]\longrightarrow \mathbb{R}^{>0}$  integrable, with primitive  $C(t) > 0$ , we have  $c(t) = e^{\log (c(t))}$ . This means we can write  $c(t) = \dot{C}(t) = e^{\lambda_t}\dot{\lambda}_t$  with  $\lambda_t = \log(C(t))$ . In other words, for such  $c$ , we have
$$
\int_ {s} ^ {t} c (\tau) \mathrm {d} \tau = \int_ {s} ^ {t} e ^ {\lambda_ {\tau}} \dot {\lambda} _ {\tau} \mathrm {d} \tau = \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {\lambda} \mathrm {d} \lambda .
$$
4 Higher Stage SEEDS for DPMs
In this section we present our SEEDS algorithms by putting together all the ingredients presented in the previous section. Let  $t < s$ . In all what follows, we consider the analytic solution at time  $t$  of the RSDE (2) with coefficients (7),  $h = \lambda_t - \lambda_s$  and initial value  $\mathbf{x}_s$ . Plugging (9) with  $k = 0$  and (10) into the exact solution (8) allows us to infer the first SEEDS scheme, given by iterations of the form
$$
\tilde {\mathbf {x}} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \tilde {\mathbf {x}} _ {s} - 2 \bar {\sigma} _ {t} (e ^ {h} - 1) \hat {F} _ {\theta} (\hat {\mathbf {x}} _ {\lambda_ {s}}, \lambda_ {s}) - \bar {\sigma} _ {t} \sqrt {e ^ {2 h} - 1} \epsilon , \quad \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d}). \tag {14}
$$
The following Theorem gives strong order convergence guarantees for this method, which we call SEEDS-1, under mild conditions which apply to all our experiments. We stress out that its proof (App. C) is a non-trivial result, it is fundamentally different in nature from [22] and involves mathematical tools which have no deterministic counterparts.
Theorem 4.1. Under Assumption C.1, the numerical solution  $\tilde{\mathbf{x}}_t$  produced by the SEEDS-1 method (14) converges to the exact solution  $\mathbf{x}_t$  of
$$
\mathrm {d} \mathbf {x} _ {t} = \left[ f (t) \mathbf {x} _ {t} + g ^ {2} (t) \bar {\sigma} _ {t} ^ {- 1} F _ {\theta} (\mathbf {x} _ {t}, t) \right] \mathrm {d} t + g (t) \mathrm {d} \boldsymbol {\omega} _ {t}, \quad \left(\bar {\sigma} _ {t} ^ {- 1} := 1 / \bar {\sigma} _ {t}\right) \tag {15}
$$
with coefficients (7) in Mean-Square sense with strong order 1.0: there is a constant  $C > 0$  such that
$$
\sqrt {\mathbb {E} \left[ \sup  _ {0 \leqslant t \leqslant 1} | \tilde {\mathbf {x}} _ {t} - \mathbf {x} _ {t} | ^ {2} \right]} \leqslant C h, \quad a s h \longrightarrow 0.
$$
Higher stage SEEDS. As announced, by fully exploiting the analytic computations enabled by the expansion (9) we now turn into crafting our multi-step SEEDS. Usually, SDE solvers are constructed by using the full Itô-Taylor expansion of the SDE solutions and usually need a big number of evaluations of the network  $\hat{F}_{\theta}$  to achieve higher order of convergence. As our main concern is to present stochastic solvers with a minimal amount of NFE, we choose to truncate such Itô-Taylor expansion so that the neural networks only appear in the deterministic contributions.