SlowGuess/NeurIPS_2023_OCR huggingface dataset:
How do SEEDS compare to Stochastic Runge-Kutta methods? Contrary to the ODE case, there are many stochastic Runge-Kutta approaches, usually tailored for SDEs of a specific form. Nonetheless, a common way of distinguishing solvers with same strong order is to assign them couples  $(p_d,p_s)$ , where  $p_s$  is the (stochastic) strong convergence order and  $p_d$  is the resulting (deterministic) order determined if setting  $g = 0$  in the considered SDE i.e. when they are deterministic. For instance, [30, Tab. 6.2] determines solvers with orders (1,1.0) and (2,1.0) respectively and [30, Tab. 6.3] determines solvers SRA1 and SRA3 with orders (2,1.5) and (3,1.5) respectively. Many of these solvers' speed was already tested in [15, Table 3] on CIFAR-10 (VP). Experimentally, SEEDS-1 shows to be  $6.83\mathrm{x}$  faster than the baseline Euler-Maruyama scheme.
To our knowledge, the only available strong order SERKs method for SDEs with in-homogeneous diffusion coefficients are the exponential Euler-Maruyama (EEM) method [18] and the stochastic RK Lawson (SRKL) schemes [7]. In short, the SRKL schemes only compute analytically the linear coefficient and use the Integrating Factor (IF) method to approximate the integrals in the representation of the exact solution given by the variation-of-parameters formula. This way, by a special change of variables (see [7, Alg. 1]), one can create exponential integrator versions of many SDE methods. We implemented our own version of the SRKL schemes, which we denote SRKL-1/2/3, to take into account that  $\sigma, \alpha$  are not constant and used the Integrating Factor method to approximate the integrals in the representation of the exact solution given by the variation-of-parameters formula and under the VP  $\lambda$ -change-of-variables. Interestingly, the SRKL schemes stabilize at increasing NFEs but at much higher FID values than their SETD counterparts.
In Table 2 below, we draw a comparison of SEEDS with current Stochastic RKL methods (on the  $\lambda$  temporal parameter space) on CIFAR-10 for the discretely trained DPM in the VP unconditional regime.
Is the proven convergence order of each of the SEEDS methods optimal? The proposed convergence orders for each SEEDS-1/2/3 is optimal: this is a consequence of the general result from [6] about maximum convergence rates for SDE schemes with uncorrelated Gaussian increments.
Table 2: Comparison of SEEDS with adapted Stochastic RKL methods on CIFAR-10 in the VP unconditional discrete framework.
METHOD \ NFE	10	20	50	100	BEST KNOWN
SRKL-1(λ)	332.52	282.96	33.42	8.62	-
SEEDS-1	303.48	153.21	22.70	7.97	(500 NFE) 3.13
SRKL-2(λ)	475.20	469.64	134.82	7.74	-
SEEDS-2	476.90	226.70	7.17	3.23	(90 NFE) 3.21
SRKL-3(λ)	462.24	376.15	8.36	7.46	-
SEEDS-3	483.00	428.60	43.30	3.41	(201 NFE) 3.08
The underlying idea, fully detailed in Appendix E, is that any solver with strong order  $\geq 1.5$  has to account for double stochastic integrals in the non-truncated Ito-Taylor expansion, ultimately forcing any SRK-like solver to use correlated random variables (see [30, Tab. 6.3] and [17] more generally). SEEDS avoid this additional complexity but an interesting future avenue would be to extend SEEDS to the higher strong order case (and not in the IF approach but the SETD approach) as long as it doesn't incur into an explosion of needed NFEs per step to craft such solvers. Another interesting path would be to craft weak second order SERK methods for DPMs (the work [19] addressed this only for homogeneous semi-linear SDEs with constant linear coefficients).
B Detailed Derivation of the SEEDS Design Space
In Sections 2 and 3 we proposed a simplified presentation of the design space of diffusion models and of the ingredients that constitute our proposed SEEDS methodology. In this section, we further develop our presentation in a technical manner, making explicit the formalization of our design choices.
B.1 The Isotropic General SDE framework
In Section 2, we presented a parametric family of differential equations (2) driving the generative process for DPMs, based on time-reversing the forward noising diffusion process (1). While doing so, we presented two parameters - the noise schedule  $\sigma_{t}$  and the scaling  $\alpha_{t}$  - for which the effects on DPMs have been widely studied in [16].
As the shape of the trajectories of (1) and (2) (for  $\ell = 0,1$ ) are defined by  $\alpha_{t}$  and  $\sigma_t$ , we start by writing down, for the scaling  $\mathbf{x}_t = \alpha_t\hat{\mathbf{x}}_t$ , the scaled generalization of the proposed SDEs in [16, Eq. 103] which unifies in a single framework the forward and reverse trajectories:
$$
\mathrm {d} \mathbf {x} _ {t} ^ {\pm} = \left[ \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} \mathbf {x} _ {t} ^ {\pm} - \alpha_ {t} ^ {2} \dot {\sigma} _ {t} \sigma_ {t} \nabla_ {\mathbf {x} _ {t} ^ {\pm}} \log p \left(\widehat {\mathbf {x}} _ {t} ^ {\pm}; \sigma_ {t}\right) \pm \alpha_ {t} ^ {2} \dot {\sigma} _ {t} \sigma_ {t} \nabla_ {\mathbf {x} _ {t} ^ {\pm}} \log p \left(\widehat {\mathbf {x}} _ {t} ^ {\pm}; \sigma_ {t}\right) \right] \mathrm {d} t + \alpha_ {t} \sigma_ {t} \sqrt {2 \frac {\dot {\sigma} _ {t}}{\sigma_ {t}}} \mathrm {d} \omega_ {t} ^ {\pm}. \tag {17}
$$
Following [16], the previous VP, VE, iDDPM, DDIM and EDM frameworks all are unified as different choices of  $\alpha_{t}$ ,  $\sigma_{t}$ , among other choices presented in [16, Tab. 1] and we will use this as a basis for all the proofs contained in this Appendix. In particular, forward time means taking  $\mathbf{x}_t^+$  for which the score vanishes in this context. Now set  $\mathbf{x}_t = \mathbf{x}_t^-$ .
The formulation in (2) involves a family of backwards differential equations controlled by a parameter  $\ell \in [0,1]$  which all yield reverse-time processes for (1), a fact that can be obtained by studying the Fokker-Planck equation for marginals  $p(\widehat{\mathbf{x}}_t^+;\sigma_t)$  of (17).
When  $\ell = 1$ , (2) the obtained SDE is known as the reverse SDE (RSDE) and, when  $\ell = 0$ , we obtain an ODE that is known as the Probability Flow ODE (PFO):
$$
\mathrm {d} \mathbf {x} _ {t} = \left[ \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} \mathbf {x} _ {t} - \alpha_ {t} ^ {2} \dot {\sigma} _ {t} \sigma_ {t} \nabla_ {\mathbf {x} _ {t}} \log p (\widehat {\mathbf {x}} _ {t}; \sigma_ {t}) \right] \mathrm {d} t. \tag {18}
$$
Now, finding a minimum for the loss function in [16, Eq. 51] is formulated as a convex optimization problem. As such, for the ideal model  $D(\mathbf{x}_t; \sigma_t) = \arg \min_D \mathcal{L}(D; \mathbf{x}_t, \sigma_t)$ , the score function with
scaled input is expressed as
$$
\nabla_ {\mathbf {x} _ {t}} \log p (\widehat {\mathbf {x}} _ {t}; \sigma_ {t}) = \frac {D (\widehat {\mathbf {x}} _ {t} ; \sigma_ {t}) - \widehat {\mathbf {x}} _ {t}}{\alpha_ {t} \sigma_ {t} ^ {2}}.
$$
This ideal model is usually subtracted by a raw network  $F$  in the form of a time-dependent preconditioning:
$$
D (\mathbf {x} _ {t}; \sigma_ {t}) := c _ {1} (t) \mathbf {x} _ {t} + c _ {2} (t) F (c _ {3} (t) \mathbf {x} _ {t}; c _ {4} (t)), \qquad c _ {i} (t) \in \mathbb {R} ^ {d}, \quad i = 1, \ldots , 4.
$$
As such, we can express the score function as two parameterizations involving  $D$  or  $F$  as follows:
$$
\nabla_ {\mathbf {x} _ {t}} \log p (\widehat {\mathbf {x}} _ {t}; \sigma_ {t}) = \frac {D (\widehat {\mathbf {x}} _ {t} ; \sigma_ {t}) - \widehat {\mathbf {x}} _ {t}}{\alpha_ {t} \sigma_ {t} ^ {2}} = \frac {(c _ {1} (t) - 1) \widehat {\mathbf {x}} _ {t} + c _ {2} (t) F (c _ {3} (t) \widehat {\mathbf {x}} _ {t} ; c _ {4} (t))}{\alpha_ {t} \sigma_ {t} ^ {2}}. \tag {19}
$$
Let us now denote  $D_{\theta ,t}^{1}\coloneqq D_{\theta}(\widehat{\mathbf{x}}_{t};\sigma_{t})$  for a pre-trained network approximating the ideal denoiser and let  $D_{\theta ,t}^2\coloneqq F_\theta (c_3(t)\widehat{\mathbf{x}}_t;c_4(t))$  be the corresponding raw pre-trained network. Substituting the score function in the RSDE and the PFO with each of these models yields four different differential equations with a neural network as one of their components. These are given, for  $i = 1,2$ , by
$$
\mathrm {d} \mathbf {x} _ {t} = [ A ^ {i} (t) \mathbf {x} _ {t} + B ^ {i} (t) D _ {\theta , t} ^ {i} ] \mathrm {d} t + g (t) \mathrm {d} \bar {\omega} _ {t}, \tag {20}
$$
$$
\mathrm {d} \mathbf {x} _ {t} = [ A ^ {i + 2} (t) \mathbf {x} _ {t} + B ^ {i + 2} (t) D _ {\theta , t} ^ {i} ] \mathrm {d} t. \tag {21}
$$
When  $D_{\theta, t}^{1}$  (resp.  $D_{\theta, t}^{2}$ ) is employed to replace the score function in (17) using (19), the resulting SDE (20) will be called data (resp. noise) prediction neural SDE. Proceeding analogously for the PFO (18) yield two ODEs (21) which will be called data (resp. noise) prediction neural PFO. The general form of the  $A^{i}$  and  $B^{i}$  coefficients determining each of these DEs is as follows:
$$
A ^ {1} (t) = \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} + 2 \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} B ^ {1} (t) = - 2 \alpha_ {t} \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} (\mathrm {D P N R S D E})
$$
$$
A ^ {2} (t) = \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} + 2 \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} (1 - c _ {1} (t)) \quad B ^ {2} (t) = - 2 \alpha_ {t} \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} c _ {2} (t) \quad (\text {N P N R S D E})
$$
$$
A ^ {3} (t) = \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} + \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} \quad B ^ {3} (t) = - \alpha_ {t} \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} \quad (\mathrm {D P N P F O})
$$
$$
A ^ {4} (t) = \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} + \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} (1 - c _ {1} (t)) \quad B ^ {4} (t) = - \alpha_ {t} \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} c _ {2} (t) \quad (\mathrm {N P N P F O})
$$
Remark B.1. At first glance, it would seem misleading to differentiate four DEs as these essentially correspond to different choices of  $\alpha_{t},\sigma_{t},c_{1}(t),\ldots ,c_{4}(t)$ . But the reason why we do so is that, after applying the variation of constants formula, each of these DEs will yield a different representation of their exact solutions (see (22) and (23) below). As we will see below, constructing exponential integrators heavily depends on such representation and will show to lead to four different modes of SEEDS solvers, each one showing different behavior and performance for sampling from pre-trained DPMs. As such, we will articulate this difference already at the DE formulation.
For  $t < s$ , the variation of constants formulae for NSDEs allows to represent the exact solutions of (20) as
$$
\mathbf {x} _ {t} = \Phi^ {i} (t, s) \mathbf {x} _ {s} + \int_ {s} ^ {t} \Phi^ {i} (t, \tau) B ^ {i} (\tau) D _ {\theta , \tau} ^ {i} \mathrm {d} \tau + \int_ {s} ^ {t} \Phi^ {i} (t, \tau) g (\tau) \mathrm {d} \bar {\omega} _ {t}, \quad i = 1, 2 \tag {22}
$$
and those for NPFOs (21) as
$$
\mathbf {x} _ {t} = \Phi^ {i} (t, s) \mathbf {x} _ {s} + \int_ {s} ^ {t} \Phi^ {i} (t, \tau) B ^ {i} (\tau) D _ {\theta , \tau} ^ {i - 2} \mathrm {d} \tau , \quad i = 3, 4 \tag {23}
$$
$$
\Phi_ {A ^ {i}} (t, s) = \exp \left(\int_ {s} ^ {t} A ^ {i} (\tau) \mathrm {d} \tau\right) \tag {24}
$$
is called the transition matrix associated with  $A^i(t)$  and is defined as the solution to
$$
\frac {\partial}{\partial t} \Phi_ {A ^ {i}} (t, s) = A ^ {i} (t) \Phi_ {A ^ {i}} (t, s), \quad \Phi_ {A ^ {i}} (s, s) = I _ {d}.
$$
When  $A^i(t)$  is constant and  $B^i(t) = 1$ , there is a well-established literature on exponential ODE and SDE solvers with explicit stiff order conditions and prescribed by different forms of Butcher tableaux. When  $A^i(t)$  is not constant, for the expression in (24) to make sense in the usual sense (in terms of exponential series expansion) instead of having to make use of time-ordered exponentials/ Magnus expansions, the  $f(t) \coloneqq A^i(t)$  coefficients must satisfy  $[f^{(k)}(t), f^{(l)}(s)] = 0$ . This condition is trivially satisfied here as the  $A^i(t)$  considered here are  $d$ -dimensional diagonal matrices.
Notice that, if  $A^i \neq A^j$  for some  $i \neq j$ , their associated transition matrices will not be equal. In particular, if  $A^1 \neq A^2$ , then the variances of the stochastic integrals in (22) are different for  $i = 1$  and  $i = 2$ . This is the first step in explaining the statement in Rem. B.1, and we refer the reader to the proof of Proposition 4.5 where we put into evidence its validity.
B.2 Re-framing and Generalizing Previous Exponential Solvers
B.2.1 The VP case
Let  $\tilde{\alpha}_t\coloneqq \int_0^t (\beta_d\tau -\beta_m)\mathrm{d}\tau = \frac{1}{2}\beta_dt^2 +\beta_nt$  , where  $\beta_{d} > \beta_{m} > 0$  . Set
$$
f (t) := \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t}, \quad g (t) = \alpha_ {t} \sqrt {\frac {\mathrm {d} [ \sigma_ {t} ^ {2} ]}{\mathrm {d} t}}, \qquad \sigma_ {t} = \sqrt {e ^ {\tilde {\alpha} _ {t}} - 1}, \qquad \alpha_ {t} = e ^ {- \frac {1}{2} \tilde {\alpha} _ {t}} = \frac {1}{\sqrt {\sigma_ {t} ^ {2} + 1}}.
$$
Recall that in the VP case, and the noise prediction mode, [22] construct exponential solvers on the base of the following ODE
$$
\mathrm {d} \mathbf {x} _ {t} = \left[ f (t) \mathbf {x} _ {t} + \frac {g ^ {2} (t)}{2 \bar {\sigma} _ {t}} \epsilon_ {\theta} (\mathbf {x} _ {t}; t) \right] \mathrm {d} t, \quad t \in [ T, 0 ], \tag {25}
$$
where  $\bar{\sigma}_t\coloneqq \alpha_t\sigma_t$ . The ODE (25) identifies with that in [16] for the VP case for which the authors identify the preconditioning
$$
c _ {1} (t) = 1, \quad c _ {2} (t) = - \sigma_ {t}, \quad c _ {3} (t) = \frac {1}{\sqrt {\sigma_ {t} ^ {2} + 1}}, \quad c _ {4} (t) = (M - 1) \sigma^ {- 1} (\sigma_ {t}) = (M - 1) t.
$$
As such, we obtain the following coefficients for the NP NPFO:
$$
A ^ {4} (t) = \frac {\dot {\alpha} _ {t}}{\alpha_ {t}}, \quad B ^ {4} (t) = \alpha_ {t} \dot {\sigma} _ {t}, \quad \Phi^ {4} (t, s) = \frac {\alpha_ {t}}{\alpha_ {s}}
$$
$$
\begin{array}{l} \nabla_ {\mathbf {x} _ {t}} \log p (\widehat {\mathbf {x}} _ {t}; \sigma_ {t}) = \frac {D _ {\theta} (\widehat {\mathbf {x}} _ {t} ; \sigma_ {t}) - \widehat {\mathbf {x}} _ {t}}{\alpha_ {t} \sigma_ {t} ^ {2}} = \frac {(c _ {1} (t) - 1) \widehat {\mathbf {x}} _ {t} + c _ {2} (t) F (c _ {3} (t) \widehat {\mathbf {x}} _ {t} ; c _ {4} (t))}{\alpha_ {t} \sigma_ {t} ^ {2}} \\ { = } { \frac { - \sigma _ { t } F _ { \theta } ( \mathbf { x } _ { t } , ( M - 1 ) t ) } { \alpha _ { t } \sigma _ { t } ^ { 2 } } . } \\ \end{array}
$$
B.2.2 Proof of Proposition 3.1
First of all, denote  $F_{\theta}(\mathbf{x}_t,(M - 1)t) = \epsilon_{\theta}(\mathbf{x}_t,t)$ . We have
$$
f (t) = \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t}, g ^ {2} (t) = 2 \bar {\sigma} _ {t} ^ {2} \left(\frac {\mathrm {d} \log \bar {\sigma} _ {t}}{\mathrm {d} t} - \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t}\right) = - 2 \bar {\sigma} _ {t} ^ {2} \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}, \quad \frac {\bar {\sigma} _ {t}}{\alpha_ {t}} = e ^ {- \lambda_ {t}}.
$$
This way, one can directly relate  $\lambda_{t}$  with the signal-to-noise ratio  $\mathrm{SNR}(t) = \alpha_t^2 /\bar{\sigma}_t^2$ , also being used in [22]. As such,  $\mathrm{SNR}(t)$  is strictly monotonically decreasing in time. Thus, the analytic solution to
(2) yields
$$
\begin{array}{l} \mathbf {x} _ {t} = e ^ {\int_ {s} ^ {t} f (\tau) \mathrm {d} \tau} \mathbf {x} _ {s} + \int_ {s} ^ {t} \left(e ^ {\int_ {\tau} ^ {t} f (r) \mathrm {d} r} \frac {g ^ {2} (\tau)}{\bar {\sigma} _ {\tau}} \boldsymbol {\epsilon} _ {\theta} (\mathbf {x} _ {\tau}, \tau)\right) \mathrm {d} \tau + \int_ {s} ^ {t} \left(e ^ {\int_ {\tau} ^ {t} f (r) \mathrm {d} r} g (\tau)\right) \mathrm {d} \bar {\boldsymbol {\omega}} (\tau) \\ = \frac {\alpha_ {t}}{\alpha_ {s}} \mathbf {x} _ {s} + \alpha_ {t} \int_ {s} ^ {t} \frac {g ^ {2} (\tau)}{\alpha_ {\tau} \bar {\sigma} _ {\tau}} \boldsymbol {\epsilon} _ {\theta} (\mathbf {x} _ {\tau}, \tau) d \tau + \alpha_ {t} \int_ {s} ^ {t} \frac {g (\tau)}{\alpha_ {\tau}} d \bar {\boldsymbol {\omega}} (\tau) \\ { = } { \frac { \alpha _ { t } } { \alpha _ { s } } \mathbf { x } _ { s } - \alpha _ { t } \int _ { s } ^ { t } \frac { 2 \sigma _ { \tau } ^ { 2 } } { \alpha _ { \tau } \bar { \sigma } _ { \tau } } \frac { \mathrm { d } \lambda _ { \tau } } { \mathrm { d } \tau } \pmb { \epsilon } _ { \theta } ( \mathbf { x } _ { \tau } , \tau ) \mathrm { d } \tau + \alpha _ { t } \int _ { s } ^ { t } \frac { g ( \tau ) } { \alpha _ { \tau } } \mathrm { d } \bar { \omega } ( \tau ) } \\ { = } { \frac { \alpha _ { t } } { \alpha _ { s } } \mathbf { x } _ { s } - 2 \alpha _ { t } \int _ { s } ^ { t } \frac { \bar { \sigma } _ { \tau } } { \alpha _ { \tau } } \frac { \mathrm { d } \lambda _ { \tau } } { \mathrm { d } \tau } \pmb { \epsilon } _ { \theta } ( \mathbf { x } _ { \tau } , \tau ) \mathrm { d } \tau + \alpha _ { t } \int _ { s } ^ { t } \frac { g ( \tau ) } { \alpha _ { \tau } } \mathrm { d } \bar { \omega } ( \tau ) } \\ = \frac {\alpha_ {t}}{\alpha_ {s}} \mathbf {x} _ {s} - 2 \alpha_ {t} \int_ {s} ^ {t} e ^ {- \lambda_ {\tau}} \frac {\mathrm {d} \lambda_ {\tau}}{\mathrm {d} \tau} \epsilon_ {\theta} (\mathbf {x} _ {\tau}, \tau) \mathrm {d} \tau - \sqrt {2} \alpha_ {t} \int_ {s} ^ {t} e ^ {- \lambda_ {\tau}} \sqrt {\frac {\mathrm {d} \lambda_ {\tau}}{\mathrm {d} \tau}} \mathrm {d} \bar {\boldsymbol {\omega}} (\tau). \\ \end{array}
$$
By using the change of variables to  $\lambda(t)$ , our equation now reads
$$
\mathbf {x} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \mathbf {x} _ {s} - 2 \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \hat {\epsilon} _ {\theta} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) d \lambda - \sqrt {2} \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} d \bar {\omega} (\lambda). \tag {26}
$$
Finally, notice that  $\alpha_{t} = \sqrt{\frac{1}{1 + e^{-2\lambda_{t}}}}$  and  $\bar{\sigma}_{t} = \sqrt{\frac{1}{1 + e^{2\lambda_{t}}}}$  so that (26) is equivalent to
$$
\widehat {\mathbf {x}} _ {\lambda_ {t}} = \frac {\widehat {\alpha} _ {\lambda_ {t}}}{\widehat {\alpha} _ {\lambda_ {s}}} \widehat {\mathbf {x}} _ {\lambda_ {s}} - 2 \widehat {\alpha} _ {\lambda_ {t}} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \widehat {\pmb {\epsilon}} _ {\theta} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) \mathrm {d} \lambda - \sqrt {2} \widehat {\alpha} _ {\lambda_ {t}} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \mathrm {d} \bar {\boldsymbol {\omega}} (\lambda).
$$
B.2.3 Proof of Proposition 4.2
Recall that the functions  $\varphi_{k}$  are the integrals
$$
\varphi_ {k + 1} (t) = \int_ {0} ^ {1} e ^ {(1 - \delta) t} \frac {\delta^ {k}}{k !} \mathrm {d} \delta ,
$$
which satisfy  $\varphi_{k}(0) = \frac{1}{k!}$ . The truncated Ito-Taylor expansion of  $\widehat{\epsilon}_{\theta}$  with respect to  $\lambda$  reads
$$
\widehat {\epsilon} _ {\theta} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) = \sum_ {k = 0} ^ {n} \frac {(\lambda - \lambda_ {s}) ^ {k}}{k !} \widehat {\epsilon} _ {\theta} ^ {(k)} (\widehat {\mathbf {x}} _ {\lambda_ {s}}, \lambda_ {s}) + \mathcal {R} _ {n + 1},
$$
where here  $\widehat{\epsilon}_{\theta}^{(k)}$  denotes the  $L_{t}^{k}$  operators defined in (58) applied to  $\widehat{\epsilon}_{\theta}$ . On the one hand, since  $\int_{\lambda_s}^{\lambda_t}e^{-\lambda}d\lambda = \frac{\bar{\sigma}_t}{\alpha_t} (e^h -1)$ , we obtain by iteratively integrating by parts
$$
\begin{array}{l} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \widehat {\epsilon} _ {\theta} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) d \lambda = \sum_ {k = 0} ^ {n} \widehat {\epsilon} _ {\theta} ^ {(k)} (\widehat {\mathbf {x}} _ {\lambda_ {s}}, \lambda_ {s}) \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \frac {(\lambda - \lambda_ {s}) ^ {k}}{k !} d \lambda + \mathcal {R} _ {n + 2} \\ = \frac {\bar {\sigma} _ {t}}{\alpha_ {t}} \sum_ {k = 0} ^ {n} \widehat {\boldsymbol {\epsilon}} _ {\theta} ^ {(k)} (\widehat {\mathbf {x}} _ {\lambda_ {s}}, \lambda_ {s}) h ^ {k + 1} \varphi_ {k + 1} (h) + \mathcal {R} _ {n + 2}. \\ \end{array}
$$
On the other hand, we have  $s > t, h = \lambda_t - \lambda_s > 0$ . Note that since the stochastic integrals  $\int_{\lambda_s}^{\lambda_t} e^{-\lambda} \mathrm{d}\bar{\omega}(\lambda)$  are measurable with respect to  $(\bar{\omega}(\lambda) - \bar{\omega}(\lambda_s), 0 \leq \lambda \leq \lambda_t - \lambda_s)$ , they are independent on disjoint time intervals by the independence of increments property of Brownian motion. Thus the random variable  $\epsilon := \epsilon_{s,t}$  in our algorithms are independent on disjoint time intervals. We then write
$$
\begin{array}{l} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} d \bar {\omega} (\lambda) = \mathcal {N} \left(0, \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- 2 \lambda} d \lambda\right) \\ { = } { \frac { 1 } { \sqrt { 2 } } \sqrt { e ^ { - 2 \lambda _ { s } } - e ^ { - 2 \lambda _ { t } } } \epsilon , \quad \epsilon \sim \mathcal { N } ( \mathbf { 0 } , \mathbf { I } _ { d } ) } \\ { = } { \frac { 1 } { \sqrt { 2 } } \sqrt { \left( \frac { \bar { \sigma } _ { s } } { \alpha _ { s } } - \frac { \bar { \sigma } _ { t } } { \alpha _ { t } } \right) \left( \frac { \bar { \sigma } _ { s } } { \alpha _ { s } } + \frac { \bar { \sigma } _ { t } } { \alpha _ { t } } \right) } \epsilon , \quad \epsilon \sim \mathcal { N } ( \mathbf { 0 } , \mathbf { I } _ { d } ) } \\ { = } { \frac { 1 } { \sqrt { 2 } } \frac { \bar { \sigma } _ { t } } { \alpha _ { t } } \sqrt { e ^ { 2 h } - 1 } \epsilon , \quad \epsilon \sim \mathcal { N } ( \mathbf { 0 } , \mathbf { I } _ { d } ) . } \\ \end{array}
$$
In conclusion, the truncated Itô-Taylor expansion of the analytic expression
$$
\mathbf {x} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \mathbf {x} _ {s} - 2 \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \widehat {\boldsymbol {\epsilon}} _ {\theta} (\widehat {\mathbf {x}} _ {\lambda}, \lambda) d \lambda - \sqrt {2} \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} d \bar {\boldsymbol {\omega}} (\lambda) \tag {27}
$$
simplifies to
$$
\mathbf {x} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \mathbf {x} _ {s} - 2 \bar {\sigma} _ {t} \sum_ {k = 0} ^ {n} h ^ {k + 1} \varphi_ {k + 1} (h) \hat {\epsilon} _ {\theta} ^ {(k)} (\hat {\mathbf {x}} _ {\lambda_ {s}}, \lambda_ {s}) - \bar {\sigma} _ {t} \sqrt {e ^ {2 h} - 1} \epsilon + \mathcal {R} _ {n + 2}, \quad \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d}).
$$
B.2.4 Generalization to the remaining data prediction and deterministic modes
Propositions 3.1 and 4.2 consist on the first steps for crafting SEEDS solvers in the VP case associated to the noise prediction neural RSDE (NP NRSDE). Generalizing the above procedure for crafting SEEDS for the 4 modes associated to (22) and (23) yields the following sets of coefficients
$$
A ^ {1} (t) = \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} + 2 \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} \quad B ^ {1} (t) = - 2 \alpha_ {t} \frac {\dot {\sigma} _ {t}}{\sigma_ {t}} \quad (\text {D P N R S D E})
$$
$$
A ^ {2} (t) = \frac {\dot {\alpha} _ {t}}{\alpha_ {t}} \quad B ^ {2} (t) = 2 \alpha_ {t} \dot {\sigma} _ {t} \quad (\text {N P N R S D E})
$$
$$
A ^ {4} (t) = \frac {\alpha_ {t}}{\alpha_ {t}} \quad B ^ {4} (t) = \alpha_ {t} \dot {\sigma} _ {t} \quad (\text {N P N P F O})
$$
We readily obtain
$$
\Phi^ {2} (t, s) = \Phi^ {4} (t, s) = \frac {\alpha_ {t}}{\alpha_ {s}}, \qquad \Phi^ {3} (t, s) = \frac {\bar {\sigma} _ {t}}{\bar {\sigma} _ {s}}, \qquad \Phi^ {1} (t, s) = \frac {\sigma_ {t} ^ {2} \alpha_ {t}}{\sigma_ {s} ^ {2} \alpha_ {s}}.
$$
Then, by setting the simpler change of variables  $\lambda_t \coloneqq -\log(\sigma_t)$ , we obtain
$$
\int_ {s} ^ {t} \Phi^ {4} (t, \tau) B ^ {4} (\tau) \mathrm {d} \tau = \alpha_ {t} \int_ {s} ^ {t} \frac {1}{\alpha_ {\tau}} \alpha_ {\tau} \dot {\sigma} _ {\tau} \mathrm {d} \tau = \alpha_ {t} \int_ {s} ^ {t} \dot {\sigma} _ {\tau} \mathrm {d} \tau = - \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \mathrm {d} \lambda = - \bar {\sigma} _ {t} (e ^ {h} - 1).
$$
By recursion we obtain
$$
\int_ {s} ^ {t} \Phi^ {4} (t, \tau) B ^ {4} (\tau) F _ {\theta , \tau} d \tau = - \bar {\sigma} _ {t} \sum_ {k = 0} ^ {n - 1} h ^ {k + 1} \varphi_ {k + 1} (h) F _ {\theta , s} ^ {(k)} + \mathcal {O} (h ^ {n + 1}).
$$
In the same way, we obtain