Variational Inference for SDEs Driven by Fractional Noise
Authors: Rembert Daems
Manfred Opper
Guillaume Crevecoeur
Tolga Birdal
Venue: ICLR 2024 spotlight (2024)
Keywords: variational inference
neural sdes
stochastic differential equations
brownian motion
fractional noise
fractional brownian motion
markov approximation
markov representation
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose a method to perform variational inference for stochastic differential equations driven by fractional Brownian motion.

Abstract: We present a novel variational framework for performing inference in (neural) stochastic differential equations (SDEs) driven by Markov-approximate fractional Brownian motion (fBM). SDEs offer a versatile tool for modeling real-world continuous-time dynamic systems with inherent noise and randomness. Combining SDEs with the powerful inference capabilities of variational methods, enables the learning of representative distributions through stochastic gradient descent. However, conventional SDEs typically assume  the underlying noise to follow a Brownian motion (BM), which hinders their ability to capture long-term dependencies. In contrast, fractional Brownian motion (fBM) extends BM to encompass non-Markovian dynamics, but existing methods for inferring fBM parameters are either computationally demanding or statistically inefficient. 

In this paper, building upon the Markov approximation of fBM, we derive the evidence lower bound essential for efficient variational inference of posterior path measures, drawing from the well-established field of stochastic analysis. Additionally, we provide a closed-form expression for optimal approximation coefficients and propose to use neural networks to learn the drift, diffusion and control terms within our variational posterior, leading to the variational training of neural-SDEs. In this framework, we also optimize the Hurst index, governing the nature of our fractional noise. Beyond validation on synthetic data, we contribute a novel architecture for variational latent video prediction,—an approach that, to the best of our knowledge, enables the first variational neural-SDE application to video perception.

VARIATIONAL INFERENCE FOR SDEs DRIVEN BY FRACTIONAL NOISE
Rembert Daems 1.
Manfred Opper 3,45 Guillaume Crevecoeur
Tolga Birdal"
D2LAB, Ghent University, Belgium
2 MIRO core lab, Flanders Make@ Belgium
Dept. of Theor. Comp. Science, Technical University of Berlin, Germany
Inst. of Mathematics, University of Potsdam, Germany
5 Centre for Systems Modelling and Quant. Biomed., University of Birmingham, UK
Dept. of Computing, Imperial College London, UK
ABSTRACT
We present a novel variational framework for performing inference in (neural) stochastic differential equations (SDEs) driven by Markov-approximate fractional Brownian motion (IBM). SDEs offer a versatile tool for modeling real-world continuous-time dynamic systems with inherent noise and randomness. Combin- ing SDEs with the powerful inference capabilities of variational methods, enables the learning of representative function distributions through stochastic gradient descent. However, conventional SDEs typically assume the underlying noise to follow a Brownian motion (BM), which hinders their ability to capture long-term dependencies. In contrast, fractional Brownian motion (fBM) extends BM to en- compass Mon-Markovian dynamics, but existing methods for inferring fBM pa- rameters either computationally demanding statistically inefficient. In this paper, building upon the Markov approximation of fBM, we derive the evidence lower bound essential for efficient variational inference of posterior path sures, drawing from the well-established field of stochastic analysis. Additionally, we provide a closed-form expression to determine optimal approximation coeffi- cients. Furthermore, we propose the use of neural networks to learn the drift, dif- fusion and control terms within our variational posterior, leading the variational training of neural-SDEs. In this framework, we also optimize the Hurst index, governing the nature of our fractional noise. Beyond validation on synthetic data, we contribute novel architecture for variational latent video prediction,- an ap- proach that, to the best of our knowledge, enables the first variational neural-SDE application to video perception.
INTRODUCTION
Our surroundings constantly evolve over time, influenced by several dynamic factors, manifesting in various forms, from the weather patterns and the ebb & flow of financial markets to the move- ments of objects (Yu et al., 2023; Rempe et al., 2021) & observers. and the subtle deformations that reshape our environments (Gojeic et al., 2021). Stochastic differential equations (SDEs) provide natural way to capture the randomness and continuous-time dynamics inherent in these real-world processes. To extract meaningful information about the underlying system, iee. to infer the model parameters and to accurately predict the unobserved paths, variational inference (VI) (Bishop & Nasrabadi, 2006) is used paths (Opper, 2019; Li et al., 2020; Ryder et al., 2018)!.
efficient means, computing the posterior probability measure over
The traditional application of SDEs assumes that the underlying noise processes generated by standard Brownian motion (BM) with independent increments. Unfortunately. for many practical scenarios, BM falls short of capturing the full complexity and richness of the observed real data,
KI. divergence hetween two SDEs over finite time horizon has been well-explored in the control literature (Theodorou, 2015; Kappen & Ruiz, 2016).

---

variational petor variational posterior
0.3
prior variational posterior
320
2609)
X20
5600
u13)
X-00
We leverage the Markov approximation, where the non-Markovian fractional Brownian motion with Hurst index H is approximated by linear combination of finite number of Markov processes (Yi(t)..... Yx (f)), and propose variational inference framework in which the posterior steered by a control term u(t). Note the long-term memory behaviour of the processes, where individual Y2 (t)s have varying transient effects, from (t) having the longest memory to (t(t) the shortest, and tend to forget the action u(t) after a certain time frame.
Figure I:
which often contains long-range dependencies, rare events, and intricate temporal structures that cannot be faithfully represented by Markovian process. The non-Markovian fractional Brownian motion (fBM) (Mandelbrot & Van Ness, 1968) extends BM stationary increments with more complex dependence structure, i.e. long-range dependence vs. roughness/regularity controlled by its Hurst index (Gatheral et al., 2018). Yet, despite its desirable properties, the computational challenges and intractability of analytically working with fBMs pose significant challenges for inference.
In this paper, we begin by providing tractable variational inference framework for SDEs driven by fractional Brownian motion (Types & II). To this end, we benefit from the relatively under- under-explored Markov representation of fBM and path-wise approximate fBM through a linear combination of finite number of Ornstein-Uhlenbeck (OU) processes driven by common noise (Carmona & Coutin, 1998a;b; Harms & Stefanovits, 2019). We further introduce a differentiable method to optimise for the associated coefficients and conjecture (as well as empirically validate) that this strong approximation enjoys super-polynomial convergence rates, allowing us to use a handful of processes in complex problems.
Such Markov-isation also allows us to inherit the well-established tools of traditional SDEs includ- ing Girsanov's 's change of measure theorem (Øksendal & Øksendal, 2003), which we use to derive and maximise the corresponding evidence lower bound (ELBO) to yield posterior path measures well maximum likelihood estimates illustrated in Fig. 1. We then conjunction with neural networks to devise VI for neural-SDEs (Liu et al., 2019; Li et al., 2020) driven by the said fractional diffusion. We deploy this model along with a novel neural architecture for the task of enhanced video prediction. To the best of our knowledge, this is the first time either fractional or variational neural-SDEs are used model videos. Our contributions are: We make accessible the relatively uncharted Markovian embedding of the fBM and its strong approximation, to the machine learning community. This allows us to employ the traditional machinery of SDEs in working with Mon-Markovian systems.
framework in
We show how to balance the contribution of Markov processes by optimising for the combination coefficients in closed form. We further estimate the (time-dependent) Hurst index from data.
We derive the evidence lower bound for SDEs driven by approximate fBM of both Types and II. We model the drift, diffusion and control terms in our framework by neural networks, and propose a novel architecture for video prediction.
We make our implementation publicly available under: gelnal.ccmm/iioooeennnSSSEMMMMM
2 RELATED WORK
Fractional noises and neural-SDEs. fBM (Mandelbrot Van Ness, 1968) was originally used for the simulation of rough volatility in finance (Gatheral et al., 2018). Using the Lemarié- wavelet representation, Allouche et al. (2022) provided a large probability bound on the deep- feedforward RELU network approximation of fBM, where up to log terms, uniform error of O(N is achievable with log(N) hidden layers and ONN) parameters. Tong et al. (2022) ap- proximated the fBM (only Type II) with sparse Gaussian processes. Unfortunately, they are limited to Euler-interration and to the case of Н > 1/3. Their model was also not applied to videos. Re- cently, Yang et al. (2023) applied Levy driven neural-SDEs to times series prediction and Hayashi & Nakagawa (2022) considered neural-SDEs driven by fractional noise. Neither of those introduce a variational framework. Both Liao et al. (2019); Morrill et al. (2021) worked with rough path theory
2

---

to model long time series via rough neDr. To the best of our knowledge, are the firsts to devise a VI framework for neural-SDEs driven by a wath-wise (strong) approximation of fBM.
SDEs and visual understanding. Apart from the recent video diffusion models (Luo et al., 2023; Yang et al., 2022; Ho et al., 2022). SDEs for spatiotemporal visual generation is relatively unex- plored. Park et al. (2021); Ali al. (2023) used neural-ODEs to generate and manipulate videos while (Rempe et al., 2020) used neural-ODEs for temporal 3D point cloud modeling. SDENet (Kong al., 2020) and MDSDE-Net (Zhang et al., 2023) learned drift and diffusion networks for uncer- tainty estimation of images using out-of-distribution data. Tong et al. (2022) used approximate- fBMs in score-based diffusion modeling for image generation. Gordon & Parde (2021) briefly evaluated different neural temporal models for video generation. While Babaeizadeh et al. (2018) used VI for video prediction, they did not employ SDEs. To the best of our knowledge, we are the firsts to use neural-SDEs in variational framework for video understanding.
3 BACKGROUND
We first tailor and make accessible the fractional Brownian Motion (fBM) and its relatively less explored Markov approximations for the learning community. We then describe the SDEs driven by [BM and its approximation before delving into the inference. We leave the proofs to our appendix.
3.1 FRACTIONAL BROWNIAN MOTION (FBM)& & ITS MARKOV APPROXIMATION
Definition 1 (Fractional Brownian Motion (Types & II)), JBM is self-similar, Mar-ovanov,nn, non-martingale, zero-mean Gaussian process (Bи (f)) for T > 0 with covariance of either
E
B (
(+)
1 (14
(112 + |0|22
2H - -1-5122
(Type I)
(1)
2
E (1) (5) T2(H+1122
1
[ ((t- u)(s u)) --/22112 du
(Type II)
(2)
where
t>> 0
< H < 1 is the Hurst index, superscripts denote the types and T is the Gamma
function.
[BM recovers Brownian motion (BM) for H = 1/2 (regular diffusion) and generalizes it for other choices. The increments are (i) positively correlated for H 1/2 (super-diffusion) where the tail behaviour is infinitely heavier than that of BM, and (ii) negatively correlated for H < 1/2 (sub- diffusion), with variance E (1B47 (t) (())BB/((C (s)((s))) = | s|2' for Type I. The Type II model implies nonstationary increments of which the marginal distributions are dependent the time relative to the start of the observed sample, iee. all realizations would have to be found very close to the unconditional mean (i.e., the origin) (Lim & Sithi, 1995; Davidson & Hashimzade, 2009).
Definition 2 (Integral representations of fBM). B((-1)) admit the following integral forms due to the Mandelbrot van-Ness and Weyl representations, respectively (Mandelbrot & Van Ness, 1968):
(1) T(H + 1/2)
1
[t,) ((t((- (1 11/2
(-5) (- -11) aw(ss)
(3)
1
(10
((- 11-12 - -1--)"-11-(-->"" -(-s)
aw())) ( (1- -)=s"== 1/22 dw
T(H 1/2)
B (1) r(H + 1/2)
1
1'" [xcuntt s) (t- s) -1/2 (s)
(4)
where ((1) and ((I)) the kernels corresponding Types and II, respectively. Proposition (Markov representation of fBM (Harms & Stefanovits, 2019)). The long memory processes B(1.11) (t) can be represented by an infinite linear combination of Markov processes, all driven by the same Wiener noise, but with different time scales, defined by speed of mean reversion For both rypes we have representations of the form:
Bn(1) = [ I (Y,(t) - (,00))))))) H < 1/2,
(5)
(,,((()) - Y,(0))u(7) dy, H > 1/2
3

---

where pl7) Y - + 1/2)(1222 H)) and v(1) = 1- + 1/2) T(3/2- - H)). Note, these non-negative densities are not normalisable. To simplify notation, we will drop explicit dependency on the rypes (I, in what follows. For each 7 0, and for both types I and II, the processes (2(1) are OU processes which are solutions to the SDE (1) = -Yx((t) dt dW().. This SDE is solved by
(t) (0) [ e (( (s(s).
(6)
"Type 1" and "Type I1" differ in the initial conditions (0) One can show that:
y = f- e (s(s) and (0)
7
(7)
Definition 3 (Markov approximation of fBM (MA-(BM)). Eq. (5) suggests that Bu (t) could be well approximated by a Markov process Bu(t) by (i) truncating the integrals at finite y values (11...Kk) and (ii) approximating the integral by numerical quadrature as a finite linear combina- tion involving quadrature points and weights {- Changing the notation Yau (t) - Y&(t):
K
Bt(t) == BB(t)= wK (Y&(t) - (x(()),
= Σ
(8)
where for fixed "ik the choice of wt depends on H and the choice of "Type I" or "Type II". For "Type II", we set Yx(0) = Since (x(t) is normally distributed (Harms & Stefanovits, 2019, Thm. 2. 16) and can be assumed stationary for "Type I", we can simply sample (Y (0)..... y2 (0))
from a normal distribution with mean and covariance C. ==j=1// 1/(7 + +7g) (see Eq. (28)).
This strong approximation provably bounds the sample paths:
Theorem 1 (Alfonsi & Kebaier (2021)). For rough kernels (H < 1/2) and {wx} following a Gaussian quadrature rule, there exists constant e per every t E (0, T) such that:
BB B (f) S O(K
where 1<052,
(9)
as K- + cc. Note that, in our setting. (0) = ((0 (0) (0) =0
In the literature, different choices of 7/ and w/ have been proposed (Harms & Stefanovits, 2019; Carmona & Coutin, 1998a; Carmona et al., 2000) and for certain choices, it is possible to obtain a superpolynomial rate, as shown by Bayer & Breneis (2023) for the Type II case. As we will show in Sec. 4. choosing " = n k-n k = I,..., K with n = (K + 1)/2 (Carmona & Coutin, 1998a), we will optimise {wx le for both types, to get optimal rates.
3.2 SDES DRIVEN BY (FRACTIONAL) BM
Definition 4 (SDEs driven by BM (BMSDE)). A common generative model for stochastic dynamical systems considers a set of observational data D ={Or,.... ON.. where the O₂ are generated (conditionally) independent at random at discrete times t4 with likelihood pe (O2 X(t))). The
prior information about the unobserved path (X(t); [0, T}} of the latent process X(t) E RM
is given by the assumption that ( fulfils the SDE:
dX bo (X(I), () di ao (X(1), (W
(10)
The drift function by (X,t) E RD models the deterministic part of the change dx((t of the state
variable (t) during the infinitesimal time interval dt, whereas the diffusion matrix dg (X(t), e
DD D (assumed to be symmetric and non- non-singular, for simplicity) encodes the strength of the added Gaussian white noise process, where dW (t) E RD is the infinitesimal increment of vector of independent Wiener processes during dt.
Definition 5 (SDEs driven by fBM (FBMSDE)). Dfnt. 4 can be formally extended to the case of fractional Brownian motion replacing dW(t) by ( (Guerra & Nualart, 2008):
dx = be (X()), 1) di + ag (X(1), 1) (l)(().
(11)
Remark 1. Care must be taken in proper definition of the diffusion part in the JBMSDE Eq. (11)
and in developing appropriate numerical integrators for simulations, when the diffusion dg (X(t), t) explicitly depends
the state Xt).. Corresponding stochastic integrals of the Itō rype cannot be
applied when H < 1/2 and other approuches (which are generalisations of the Stratonovich SDE for H (/)) are necessary (Lysy & Pillai, 2013).
4

---

4 METHOD
Our goal is to extend variational inference (VI) Bishop & Nasrabadi (2006) to the case where the Wiener process in Eq. (10) replaced by an fBM as in Dín. Unfortunately, the processes defined by Eq. (1I) are not Markovian preventing us from resorting to the standard Girsanov change of measure approach known for "ordinary" SDE to compute divergegess and ELBO functionals needed for VI (Opper, 2019). While Tong et al. (2022) leverage sparse approximations for Gaussian processes, this makes Bи conditioned on a finite but larger number of so-called inducing variables. We take completely different and conceptually simple approach to VI for fBMSDE based on the exact representation of B (1) given in Prop. To this end, we first show how the strong Markov- approximation in Din. 3 can be used to approximate SDE driven by IBM, before delving into the VI for the Markov- Aprrkximap fBMSDE.
Definition 6 (Markov- MMrkov--pproximaae Approximate fBMSDE (MA-IBMSDE)). Substituting the fBM, Bt in Dfnn. 5 by the finite linear combination of OU-processes BH (t), we define MA-JBMSDE as: (x = by (X dt + dg (X(t), t) dB
(12)
where (1)(() = -EK We (t& with ( = -u>((((t) dt + () (cf. Dfn. 3).
Proposition 2. (L) can be augmented by the finite number of Markov processes Yı(()) (approxi-
mating B (1)) to a higher dimensional state variable of form Z(1)= = (X(1). (1)),... Yx(()) E
D(K++),, such that the joint process of the augmented system becomes Markovian and can be de- scribed by an 'ordinary' SDE:
dz(t) he (2(1), dt+ Ee (W)
(13)
where the augmented drift vector hig E Dx and the diffusion matrix Eg (2,1) E DD(+))) are given by
(X,))
be (X,t) 00 (X,t) Ex Wk
1
-7
he (Z,t)=
Eg (Z,t)=
(14)
YK
i
where -= (1,1,...,,1 1)T ER. We will refer to Eg. (13) as the variational prior.
Proof. Each of the D components the vectors Y1, use the same scalar weights Wi E R. Also, note that each YE is driven by the same vector of Wiener processes. Hence, we obtain the system of SDEs given by
(t) = by (X(t), dt - oog (X(t,, t) Σ wa wanxYalt) di + way (X(t),t)dW()
(15)
dt = - Tk (t)) + dt for k= K
where a - Σx Wμ. This system of equations can be collectively represented in terms of the aug- mented variable Z(t)= = (X(1), ... Yx(()) D(K+1) leading to single SDE specified by Eqs. (13) and (14).
Eq. (13) represents standard SDE driven by Wiener noise allowing utilise the standard tools of stochastic analysis, such as the Girsanov change of measure theorem and derive the evidence lower bounds (ELBO) required for VI. This is what we will exactly do in the sequel.
Proposition 3 (Controlled MA-IBMSDE, The paths of Eq. (13) can be steered by adding a con- trol term u(X, YI....,Y. Yk,t) E RDD that depends on all variables be optimised, to the drift he resulting in the transformed SDE, a.k.a. the variational posterior:
dt
h1 (tt)) + ( t))
dt + Eo (Z(t).t dW(t)
(16)
Sketch of the proof. Using the fact that the posterior probability measure over paths 2(t) (2(t): e
[0. T}} is absolutely continuous w.r.t the prior process, apply the Girsanov theorem (cf. App. B.1) on Eq. (13) to write the new drift, from which the posterior SDE in Eq. (16) is obtained.
5

---

We will refer to Eq. (16) as the variational posterior. In what follows, will assume a parametric form for the control function ((Z(t), t) (z) (as e.g. given by a neural network) and will devise a scheme for inferring the variational parameters (0, Φ), i.e variational inference.
Proposition 4 (Variational Inference for MA-IBMSDE). The variational parameters 6 are opti- mised by minimising the KL- diveereeee between the posterior and the prior, where the correspond- ing evidence lower bound (ELBO) to be maximised is:
logp (O2, O2...,,, 00) 2 EZ
Σ
N
log pe (011((1))
T
us (2(0)=)] dt
(17)
where the observations {Oi} are included by likelihoods (0, ((t)) and the expectation is taken over random paths of the approximate posterior process defined by (Eg. (16)).
Sketch of the proof. Since we can use Girsanov's 's theorem II (Øksendal & Øksendal, 2003), the vari- ational bound derived in Li al. (2020) (App. 9.6.1) directly applies.
Remark 2. It is noteworthy that the measurements with their likelihoods po (o: {(.)) depend
only on the component X (t) of the augmented state Z(t.. The additional variables Y& which are used model the noise in the SDE not directly observed. However, computation of the ELBO
requires initial values for all state variables (0) (or their distribution). Hence, we sample Yş(O) in accordance with Dfn. 3.
4.1 OPTIMISING THE APPROXIMATION
We present the details of novel method for optimising our approximation B (T.IT) (1) for we- To this end, we first follow Carmona & Coutin (1998a) and choose a geometric sequence of 7x (--1-n ,,2-
-=),,, n Ktrr 1. Rather than relying on methods of numerical quadrature, we consider a simple measure for the quality of the approximation over fixed time interval [0, 7] which can be optimised analytically for both types and II.
Proposition 5 (Optimal w [w1,...,44 wK] for (t). The L2-error of our approximation
((,) (-¹ I
(BU' (1) B Bl))())
2
dt
(18)
minimized (
w b(.l1), where T
-35
T 116tt11
2T
A
71
75
A.j
(1I)
71775
(19)
7A 73
Vi +
2T
T1l++1/2
T.T Q(( + 1/2,
6/
+
(00)
H+1/2
TH + +3/2)
H+3/2 Tkk
Tkk
+ 1/2
6
P(H + 1/2,
P(H + 3/2, H43/22
(21)
H41/2
7
P(z, x) = [Tz) fo
1 1--le- dt is the regularized lower incomplete gamma function and Q(2,
=
[Tz] 1.00 1--te- dt is the regularized upper incomplete gamma function.
Sketch of the proof. By expanding the L2-error we find a tractable quadratic form of the criterion:
['" E
(BB (B
-B( (+) ( dt
g(.11)
(22)
/''
T
E B
(1. (t)²
+ E B (1.11) (t)) -2B
2E [ (t)B (B)))(())
(t)
(t)) dt
w + const,
whose non-trivial minimum is attained as the solution to the system of equations (1,1) = ((.111. We refer the reader to App. D.2 for the full proof and derivation.

---

X00
X01
X0S
-2
-2
00
13
20
0.5
1.0
1.5
0.0
0.5
20
(a)
0.3,
0.0
(b) 0.7, 0 = 0.0
(c) = 0.6, 1.0
(d) H 0.8,
0 1.0
Figure 2: The true variance (blue) of fOU bridge matches the empirical variance (dashed orange) of our trained models. The transparent black lines are the sampled approximate posterior paths used to calculate the empirical variance.
EXPERIMENTS
We implemented our method in JAX (Bradbury et al., 2018), using Diffrax (Kidger, 2021) for SDE solvers, Optax (Babuschkin et al., 2020) for optimization, Diffrax (Babuschkin et al., 2020) for distributions and Flax (Heek al., 2023) for neural networks. Unlike Tong et al. (2022) our approach is agnostic to discretization and the choice of the solver. Hence, in all experiments we can use the Pruttooviih-Milsteee solver, cf. App. E for more details.
Recovering the fractional Ornstein-Uhlenbeed bridge. Applying our method on linear problems, allows comparing empirical results to analytical formulations derived e.g. using Gaussian process methodology Rasmussen et al. (2006). We begin by assessing the reconstruction capability of our method on a fractional Ornstein-Uhlenbeck (fOU) bridge, that is an OU-process driven by IBM: d.X -XX (t) dt + dB starting at (() 0 and conditioned to end at X (Z') 0. Following the rules of Gaussian process regression (Rasmussen et al., 2006, Eq. 2.24), we have an analytical expression for the posterior covariance:
K(0,0) K(T,0) ((,,T)
(0,
E [t)²²] K(t, (K(t,t)
(t( K(t,T)]
(23)
K(T.T) + 03
((T,t)
where ((,,,] r) is the prior kernel and the observation noise is 0 for x()) and a for X If O. K(t, 7) E ( (T) (Eq. (1)) and if 0 0 and H 1/2, the kernel admits the following form (Lysy & Pillai, 2013, Appendix A):
K(t, (2H² H)
T(2 - T(HH 1, 7|)
{1 ou
[--7]
c [
2H- du
20
82M
(24)
where I(2,x)= f-- is the upper incomplete Gamma function. This allows us to com-
pare the true posterior variance with the empirical variance of model that trained by maximizing the ELBO. for a data point XTT) O. As this is equivalent to the analytical result (Eq. (23)), we can compare the variances over time. As plotted in Fig. 2, for various H and values, our VI can correctly recover the posterior variance, cf. App. F for additional results.
Estimating time-dependent Hurst index. Since our method of optimizing W/ is tractable and differentiable, we can directly opti- mize parameterized H by maximizing the ELBO. Also time-
0.7
dependent Hurst index (t) can be modelled, leading to multifrac- tional Brownian Motion (Peltier & Véhel, 1995). We directly com- pare with toy problem presented in (Tong et al., 2022, Sec. 5.2). We use the same model for (t), a neural network with one hid- den layer of 10 neurons and activation function tanh, and final sigmoid activation, and the same input (sin(t), cos(t), t]. We use (!TT) since their method is Type II. Fig. 3 shows reasonable esti- mation of (1), which is more accurate than the result from Tong al. (2022), cf.App. E for more details.
0.6
Ours
Tong
Figure 3: Estimating time-
dependent H(t) from data.
Latent video models To assess the video modelling capabilities of our framework, we train models stochastic video datasets. The prior drift hg, diffusion og and control term parameterized
7

---

eneader
20
context
decoder
01
h
temporal conv
81
oi
encoder
SDE
decoder
time
02
h2
82
X2
W 02
8(1)
encoder
decoder
ON
hx
XN
0'y
20
Figure 4: Schematic of the latent SDE video model. Video frames {o: are encoded to vectors {hi).. The static content vector that is free of the dynamic information, is inferred from {h: The context model processes the information with temporal convolution layers, so that its outputs fgshe contain information from neighbouring frames. A linear interpolation {9.): allows the pos- terior SDE model to receive aime-appropriate information g()), at (intermediate) time-steps chosen by the SDE solver. Finally, the states {ar; ] and static are decoded reconstruct frames {o()
by neural networks. The prior model is used as a stochastic video predictor, where condition on the first N frames to predict the next frames in the sequence. More intuitively, the posterior model reconstructs the given sequence of frames, while minimizing the control actions of u. This leads to prior that will model the dataset, so that the posterior will be able to model the specific data sequence during training with minimal u input. It is paramount that the control function u receives relevant information during the SDE integration, so that it can steer the SDE in the right direction. See Fig. 4 for a schematic explanation of our model and App. E for a detailed explanation of submodel architectures and hyperparameters.
We evaluate the stochastic video predictions by sampling 100 predictions and reporting the Peak Signal-to-Noise Ratio (PSNR) of the best sample, calculated frame-wise and averaged over time. This is the same approach as Franceschi et al. (2020) which allows direct comparison. Further- more, report the ELBO on the test set, indicating how well the model has captured the data.
We train models on Stochastic Moving MNIST (Denton & Fergus, 2018), video dataset where bers move on a canvas and bounce off the edge with ran- dom velocity in random direction. Our MA-IBM driven model is on par with closely related discrete-time methods such as SVG (Denton & Fergus, 2018) SLRVP Franceschi
Table 1: MNIST results.
Stochastic Moving
MNIST
Model
ELBO
PSNR
SVG
N/A
14.50
SLRVP
N/A
16.93
et al. (2020), in terms of PSNR, and is better than the BM terms
-913.60
14.90
BM baseline The Hurst index was optimized during training, and reached H 0.90 convergence (long-term memory), indicating that MA-fBM is better suited to the data than B.M
PSNR and ELBO (Tab. 1). MA-IBM
-608.00
15.30
Table
Double pendulum.
We also report results pendulum (Asseman et al., 2018), where we investigate whether the chaotic behaviour
real-world video dataset of a double
Model
ELBBO
PSNR
be modelled by SDE driven by fBM. Our MA-fBM driven model is better than the B.M base- line, both for the test set ELBO as for the PSNR metric (Tab. The Hurst index reached a value of H = 0.93 at convergence. See Fig. 5 for stochastic video predictions and App. F.3 for additional qualitative results.
BM
-545.13 13
26.11
MA-BBM
-636.61
27.09
(a) BM
(b) MA-IBM
Figure 5: Stochastic video predictions using the trained prior of model driven by BM (a) and model driven by MA-fBM (b) trained on the double pendulum dataset. The initial state is condi- tioned on the same data for all samples. Two samples are shown for each model, and 7 evenly spaced frames from the total of 20 frames in the sequence are shown. The MA-fBM samples show a more diverse, chaotic behaviour, thus better capturing the dynamics in the data.
8

---

5.1 ABLATIONS & FURTHER STUDIES
Numerical study of the Markov approximation. By numerically
evaluating the criterion E(I)) we can investigate the effect of K, the number of OU-processes, on the quality of the approximation. Fig. 6 indicates that the approximation error diminishes by increas- ing K. However, after a certain threshold the criterion saturates, de- pending on H. Adding more processes, especially for low H brings diminishing returns. The rapid convergence evidenced in this em- pirical result well agrees with the theoretical findings of (Bayer & Breneis, 2023) especially for the rough processes where H < 1/2. as recalled in Thm. 1.
1071
10
15
20
25
Figure 6: g(1) vs. K.
MSE of the generated trajectories for MA-fBM and for varying K. On more practical level, we take integration and numerical errors into account by simulating paths using MA-IBM and com- paring to paths of the true integral driven by the same Wiener noise. This is only possible for Type II, as for Type I one would need to start the integration from -00. Paths are generated from 0 to ( = 10, with 4000 integration steps for the approximation and 40000 for the true integral. We generate the paths over a range of Hurst indices and different K values. For each setting. 16 paths are sampled. Our approach for optimising Wi values (Sec. 4.1) is compared baseline where Wie is derived by piece-wise ap- proximation of the Laplace integral (ef. App. D.I). Fig. 7 shows considerably better results in favor of our approach. Increasing K has rapid positive impact on the accuracy of the approximation with diminishing returns, further confirming our theoretical insights in Sec. 3. We provide examples of individual trajectories generated in this experiment in App. F.1.
10
30
0.2
0.4
0.5
Figure 7: Mean square error (MSE) with 95% confidence intervals vs. H for varying K.
Impact of K and the #parameters on inference time. We in- vestigate the factors that influence the training time in Fig. 8, where K OU-processes are gradually included to systems with increasing number of network parameters. Note that, since our approximation driven by 1 Wiener process, and the control function u( ut)(t), 8) is scalar, the impact on computational load of including more pro- cesses limited and the run-time still dominated by the size of the neural networks. This is good news as different applications might demand different number of OU-processes.
Inforcnce time
Parzmaters
10"
x 31
100
Figure 8: K vs. the run-time.
CONCLUSION
In this paper, we have proposed new approach for performing variational inference on stochastic differential equations driven by fractional Brownian motion (fBM), We began by uncovering the relatively unexplored Markov representation of fBM, allowing us to approximate mor-Markovian paths using linear combination of Wiener processes. This approximation enabled us to derive evidence lower bounds through Girsanov's 's change of measure, yielding posterior path measures as well as likelihood estimates. We also solved for optimal coefficients for combining these processes, in closed form. Our diverse experimental study, spanning fOU bridges and Hurst index estimation, have consistently validated the effectiveness of our approach. Moreover, our novel, continuous- time architecture, powered by approita 1BM driven neural-SDEs, has demonstrated improvements in video prediction, particularly when inferring the Hurst parameter during inference.
Limitations and future work. In our experiments, we observed increased computational overhead for larger time horizons due to SDE integration, although the expansion of the number of processes incurred minimal runtime costs. We have also observed super-polynone convergence empirically and recalled weaker polynomial rates in the literature. Our Markov approximation still lacks a tight convergence bound. Our future work will also extend our framework to (fractional) Levy processes, which offer enhanced capabilities for modeling heavy-tailed noise/data distributions.

---

Acknowledgments. The authors thank Jonas Degrave and Tom Lefebvre for insightful discussions. This research received funding from the Flemish Government under the "Onderzoeksprogramma Ar- tificiéle Intelligentie (AI) Vlaanderen" programme. Furthermore it was supported by Flanders Make under the SBO project CADAIVISION, MO has been partially funded by Deutsche Forschungsge- meinschaft (DFG) Project ID 318763901 SFB1294.
ETHICS STATEMENT
Our work is driven by dedication the advancement knowledge and the betterment of society. While being largely theoretical, similar to many works advancing artificial intelligence, our work deserves an ethical consideration, which we present below.
All of our experiments were either run publicly available datasets or on data that is synthetically generated. No human or animal subjects have been involved any stage of this work. Our models designed to enhance the understanding and prediction of real-world processes without causing harm or perpetuating unjust biases, unless provided the datasets. While we do not foresee any issue with methodological bias, we have not analyzed the inherent biases of our algorithm and there might be implications in applications demanding utmost fairness.
We aptly acknowledge the contributions of researchers whose work laid the foundation for own. Proper citations and credit are given to previous studies and authors. All authors declare that there no conflicts of interest that could compromise the impartiality and objectivity of this research. All authors have reviewed and approved the final manuscript before submission.
REPRODUCIBILITY STATEMENT
We are committed to transparency in research and for this reason make our implementation publicly available under: github.com/ coto NureeNeeaarSDDEDAAARA... Considerable parts involve: (i) the Markov approximation and optimisation of the wh. coefficients; (ii) maximising ELBO to perform variational inference between the prior (t) and the posterior dt(() and (ii) the novel neural-SD Maraal-DDE based video prediction architecture making use of all our contributions. Our code replicates some of our evaluations for both of the datasets involved.
REFERENCES
Aurélien Alfonsi and Ahmed Kebaier. Approximation of stochastic volterra equations with kernels of completely monotone type. arXiv preprint 2102::2.02..3555,, 2021.
Moayed Haji Ali, Andrew Bond, Tolga Birdal, Duygu Ceylan, Levent Karacan, Erkut Erdem, and Aykut Erdem. Vidstyleode: Disentangled video editing via stylegan and neuralodes. In Interna- tional Conference on Computer Vision (ICCV), 2023.
Michaël Allouche, Stéphane Girard, and Emmanuel Gobet. A generative model for fbm with deep relu neural networks. Journal of Complexity, 73:101667, 2022.
Alexis Asseman, Tomasz Kornuta, and Ahmet Ozcan. Learning beyond simulated physics. In Modeling and Decision-making in the Spatiotemporal Domain Workshop, 2018. URL https:
/ //openreiiww net / rid=Hyla jWsRF7.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representa- tions, 2018.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, lurüi Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener,
10

---

Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojević, Wojciech Stokowiec, Luyu Wang,
Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http:
//github. com deepmind.
Christian Bayer and Simon Breneis. Markovian approximations of stochastic volterra equations with the fractional kernel. Quantitative Finance, 23(1):53-70, 2023.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com/ google/jax:
Philippe Carmona and Laure Coutin. Fractional brownian motion and the markov property. Elec- Ironic Communications in Probability [electronic only). 3:95-107, 1998a.
Philippe Carmona and Laure Coutin. Simultancous approximation of family of (stochastic) differ- ential equations. Unpublished, June, 915((0.105 1998b.
Philippe Carmona, Laure Coutin, and Gérard Montseny. Approximation of some gaussian processes. Statistical inference for stochastic processes, 3:161-171, 2000.
Annie AM Cuyt, Vigdis Petersen, Brigitte Verdonk, Haakon Waadeland, and William B Jones. Handbook of continued fractions for special functions. Springer Science & Business Media, 2008.
James Davidson and Nigar Hashimzade. Type and type fractional brownian motions: A recon- sideration. Computational Statistics & Data Analysis, 5366):2089-216,, 2009.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In International conference on machine learning, pp. 1174-1183. PMLR, 2018.
Jean- Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pp.
PMLR, 2020.
Crispin W Gardiner et al. Handbook of stochastic methods, volume springer Berlin, 1985.
Jim Gatheral, Thibault Jaisson, and Mathicu Rosenbaum. Volatility is rough. Quantitative finance, 18(6):933-949, 2018.
Zan Gojeic, Or Litany, Andreas Wieser, Lconidas J Guibas, and Tolga Birdal. Weakly supervised learning of rigid 3d scene flow. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 5692-5703, 2021.
Cade Gordon and Natalie Parde. Latent neural diffferential equations for video generation. In NeurIPS 2020 Workshop on Pre-registration in Machine Learning, pp. 73-86. PMLR, 2021.
Joao Guerra and David Nualart. Stochastic differential equations driven by fractional brownian mo- tion and standard brownian motion. Stochastic analysis and applications, 26: 1053-1075 2008
Philipp Harms. Strong convergence rates for markovian representations of fractional processes. Discrete and Continuous Dynamical Systems-B, 26(10):5567-557, 2020.
Philipp Harms and David Stefanovits. Affine representations of fractional processes with appli- cations in mathematical finance. Stochastic Processes and their Applications, 129: 185-1228. 2019.
Kohei Hayashi and Kei Nakagawa. Fractional sde-net: Generation of time series data with long term memory. In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), pp. 1-10. IEFE, 2022.
11

---

Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http:/ /ttp://githb/ wtm. google/flax.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.03433 2022.
Hilbert Johan Kappen and Hans Christian Ruiz. Adaptive importance sampling for control and inference. Journal of Statistical Physics, 10s11144-12000 2016.
Patrick Kidger. On Neural Differential Equations. PhD thesis, University of Oxford, 2021.
Patrick Kidger, James Foster, Xuechen Chen Li, and Terry Lyons. Efficient and accurate gradients for neural sdes. Advances in Neural Information Processing Systems, 34:18747-18761, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69880 2014.
Lingkai Kong, Jimeng Sun, and Chao Zhang. Sde-net: equipping deep neural networks with uncer- tainty estimates. In Proceedings of the 37th International Conference on Machine Learning, pp- 5405-5415 2020.
Xuechen Li, Ting-Kan Kam Leonard Wong, Ricky TQ Chen, and David K Duvenaud. Scalable gradients and variational inference for stochastic differential equations. In Symposium on Advances in Approximate Bayesian Inference, pp. 1-28. PMLR, 2020.
Shujian Liao, Terry Lyons, Weixin Yang. and Hao Ni. Learning stochastic differential equations using rnn with log signature features. arXiv preprint arXiv:I908) 190.0088800,, 2019.
SC Lim and VM Sithi. Asymptotic properties of the fractional brownian motion of riemann-liouville type. Physics Letters A, 206(5-6):11-317, 1995.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint arXiv:1906.02355. 2019.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10209-10218, 2023.
Martin Lysy and Natesh S Pillai. Statistical inference for stochastic differential equations with memory. arXiv preprint arXiv:1307.1164 2013.
Benoit B Mandelbrot and John W Van Ness. Fractional brownian motions, fractional noises and applications. SIAM review, 10(4):422-437, 1968.
Domenico Marinucci and Peter M Robinson.
Alternative forms of fractional brownian motion. Journal of statistical planning and inference, 80(1-2): 11-122, 1999.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equa- tions for long time series. In International Conference on Machine Learning, pp. 7829-7838.
PMLR, 2021.
Bernt Øksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Manfred Opper. Variational inference for stochastic differential equations. Annalen der Physik, 531 (3):1800233) 2019.
Sunghyun Park. Kangyeol Kim, Junsoo Lee, Jaegul Choo. Joonseok Lee, Sookyung Kim, and Ed- ward Choi. Vid-ode: Continuous-time video generation with neural ordinary differential equation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
Romain- Romain-François Peltier and Jacques Lévy Véhel. Multifractional Brownian motion: definition and preliminary results. PhD thesis, INRIA, 1995.
12

---

Carl Edward Rasmussen, Christopher KI Williams, et al. Gaussian processes for machine learning, volume 1. Springer, 2006.
Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojeic, Srinath Sridhar, and Leonidas J. Guibas. Caspr: Learning canonical spatiotemporal point cloud representations. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang. Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. Proceedings of the IEEE/CVF international conference computer vision, pp. 11488-1 11888-1149, 2021.
Tom Ryder, Andrew Golightly, A Stephen McGough, and Dennis Prangle. Black-box variational inference for stochastic differential equations. In International Conference on Machine Learning, pp. 4423 4432. PMLR, 2018.
Evangelos Theodorou. Nonlinear stochastic control and information theoretic dualities: Connec- tions, interdependencies and thermodynamic interpretations. Entropy, 17(5):3352-3355 2015.
Anh Tong, Thanh Nguyen-Targ Toan Tran, and Jaesik Choi. Learning fractional white noises in neural stochastic differential equations. In Advances in Neural Information Processing Systems, 2022.
Luxuan Yang, Ting Gao, Yubin Lu, Jinqiao Duan, and Tao Liu. Neural network stochastic dif- ferential equation models with applications to financial data forecasting. Applied Mathematical Modelling, 115:279-299, 2023.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint 2203::2203.09488) 2022.
Zhengdi Yu, Shaoli Huang, Yongkang Cheng, and Tolga Birdal. Signavatars: A large-scale 3d sign language holistic motion dataset and benchmark. arXiv preprint arXiv:2310.20436, 2023.
Xiao Zhang, Wei Wei, Zhen Zhang, Lei Zhang, and Wei Li. Milstein-driven neural stochastic differential equation model with uncertainty estimates. Pattern Recognition Letters, 2023.
13

---

APPENDIX
A FURTHER DISCUSSIONS
Difference between a Type and Type II fBM. Type I, also called Mandelbrot- Van Ness or 'tandardd' fBM is the most prevalent definition of fBM. Type II, also called Riemann-Liouviln (BM, is historically most used in econometric literature. As can be seen in the integral definitions (Eqs. (3) and (4)). Type II omits the first integral from -oo to 0 in the definition of Type I. So Type is, in a sense, a simplification. Yet, to the best of our knowledge, its covariance (Eq. (2)) has no simple analytical expression. On the other hand, the Type I covariance is the straightforward, well known Eq. (1).
As described by Lim & Sithi (1995); Marinucci & Robinson (1999) the main difference is that Type I has stationary increments, and Type II has non-stationary increments. This means that Type II has larger emphasis on the origin t = O, which might not be favourable for some applications. For example, if during training we sample a sequence from a video dataset at a random start point, this 1 = 0 has no special distinguished meaning and should not be treated differently by the driving fBM process. In other words, Type I ensures shift in time has no effect on its increments. However, this is not the case for Type II. This difference is relevant for our framework, since the increments are driving the SDE.
Optimal choices for w and 7 values. Regarding the Type II case, there are different ways of de- termining 7Kk and wi in the literature (Carmona & Coutin, 1998a; Bayer & Breneis, 2023; Harms & Stefanovits, 2019) some of which can lead to super-polynomial convergence (Bayer & Breneis, 2023) under certain assumptions, while more general choices are still shown to converge, though with a weaker rate (Alfonsi & Kebaier, 2021) while still being strong (path-wise) and of arbitrar- ily high polynomial order (Harms, 2020). Some of these works state that such geometric choice of the quadrature intervals simplifies the proofs while being not optimal and smarter choices can exist (even with better rate of convergence). This is the reason why we believe that compu- tationally tractable, closed form expressions which optimally solve for these values lead to good, super-pole polee--alynomial convergence both for types II and (since the first type also admits similar type of analysis).
Practical considerations for choosing 7k- Defining 7 (1/7max: Tmax) is convenient way to indicate some practical considerations for choosing 7k- Carmona & Coutin (1998b) show that dt > 1/2 leads unstable integration of the OU-process, where dt is the integration step. Care should be taken that Tmax dt < 1/2, either by decreasing "mmax or decreasing the integration step dt. Additionally, choosing large values for 7 is undesirable for numerical reasons. Especially when using lower precision, numerical overflow can be a problem. Since an OU-process reaches equilibrium after time 1/7, a practical lower bound for 7max is the length of the modelled sequences. This ensures that memory of the MA-fBM process is modelled for at least the length of the sequence.
Time horizon for optimising wi- The closed form expressions for W/ are in function of H and the time horizon T' (Prop. 5). Since the criterion is defined over the time interval [0, T]. it makes sense to choose T equal to the typical (or maximal) length of sequences in the modelled dataset. Specifically for "Type I", we advise to choose T at two or three times the modelled sequence length, as at / = O, this process is already at equilibrium, and its 'history' should be accounted for in the criterion. We have observed better empirical results when choosing T at multiple of the sequence length.
Further clarification on the distinction with Tong et al. (2022). Our work mainly differs with Tong et al. (2022) in two ways: (i) fractional Brownian motion (fBM) is approximated as a a Gaussian process (GP), (ii) only the Type II representation of fBM is used as as an integral over increments of the Wiener process. Tong et al. (2022) perform finite time discretization of the Type Il integral to obtain first approximation of the increments of fBM. In second step, this approx- imate GP is further approximated using a sparse GP approach based on a smaller set of pseudo or inducing points which are distributed over time. Conditioned on the inducing points. samples from the sparse GP are independent random variables at each discrete time point. Finally, this (condi- tioned) white noise process is further interpreted in terms of the Fuler discretization of an ordinary
14

---

SDE leading to effective drift and diffusions. For the latter SDE, can apply Girsanov's theorem and the corresponding ELBO (conditioned on the inducing points) to perform inference.
Note. that their current derivation of effective drift and diffusion relies the Euler discretization of SDE. For higher order SDE solvers, the approximation has to be adapted, which requires new derivations. As main difference, in our paper, the approximation is not based on the discretization in the time domain but of the discretization of an integral representation (Prop. 1) over a spectrum of decay constants of Ornstein-Uhlenbeck (OU) processes (driven by the same Wiener noise). Since each OU process already represents noise process with temporal correlations, we can expect that linear combination of a small number of such processes can yield good approximation of the covariance of fBM over some given time interval. Our approximation leads to system of SDEs (without conditioning) for which the ELBO can easily obtained. Since the time discretization of the resulting SDE is performed after the OU approximation, any SDE solver can be directly applied. With this flexibility, in our paper, we have chosen the second order Stratonovich-Milne Milsstein solver.
State dependent diffusions. For the case, where the diffusion o t) explicitly depends on the state variable X, our Markovian approximation results in 'standard' white noise SDE for the aug- mented system. As such, it does not suffer from problems with proper definitions of stochastic integrals as compared to the original SDE driven by IBM for such cases. Hence, a straightfor- ward Ité-interpretation of our augmented SDE is, in principle, possible. This might indicate, at first glance, that simple numerical solvers such as Euler's s method could be sufficient for simulating the augmented SDE required for computing posterior expectations for the ELBO. While this point needs further theoretical investigation, preliminary simulations for for simple models with state de- pendent diffusions indicate that Euler approximation (in accordance with known results for direct simulations of SDE driven by fBM (Lysy & Pillai, 2013)) quickly lead to deviations from known analytical results. Hence, for state dependent diffusions, we resort the Stratonovich interpreta- tion of the augmented system and use corresponding higher order solvers Kidger (2021)². This approach yields excellent (pathwise) agreements with exact analytical results as we show Sec. 5. Although the ELBO for SDE is derived from Girsanov's change of measure theorem for Itô SDE,, by the known correspondence (resulting in a change of drift functions, when diffusions are state dependent) (Gardiner et al., 1985) between Itô and Stratonovich SDE we conclude that within this approach, optimisation of the ELBO with respect to model parameters will also yield the corre- sponding estimates for the Stratonovich interpretation.
On initial values for Type 1", The initial values for "Type I" can be understood as resulting from an OU-process which was started at some negative time / - -oo so that
/ ThS (s
(
(25)
(5) and Y (0) can be considered as samples from the joint stationary distribution. Because the sta- tionary distribution is normal (Harms & Stefanovits, 2019, Theorem 2.16) we can simply sample
initial states of the (t) processes for Type I with covariance (O)) Using Itô isome- try (Øksendal & Øksendal, 2003):
Th dW(ss) I v dw
(Y)O)) =
(26)
clitt5) ds
(27)
(28)
7/ +
>see egg. htap:///o....og...ss.eeeaaaaaa/a ver/ #stochastic--dffeeeett -qquatttoos
15

---

B PROOFS AND FURTHER THEORETICAL DETAILS
B. THE GIRSANOV THEOREM II AND THE KL DIVERGENCE OF MEASURES
We now state the variation II of the Girsanov theorem (Oksendal & Øksendal, 2003) in our notation.
Let (t) E R be an Itô process w.r.t. measure P of the form:
(() be (X(),, + (X(),))
(29)
where 0 STST, (t) E R",, by (X(t), t) E RR" and og (X))). E R=xm Define measure Q
via:
dQ
dP = My == exp - // (( 2 l' (x(())),)
(30)
Then
w'(1= = J. d(
(31)
a Brownian motion w.r.t Q and the process X (t) has the following representation in terms of
B(()::
(t) 0e (X((), dt + (X(t).))dW")
(32)
where the new drift is:
(X()) = bo (X(1).) oo (X()), (X().))
(33)
We can also rewrite the Madon-NNkooe derivative in Eq. (30)
dQ
(X()))) 1
2 1 o² (X())))
dP
exp
(34)
2 l' (X((),,))
1 /
= exp
() + (X(t),t)d))
(35)
1 ( 2 (X(t)
exp
u (X(t), t) dw (t) +
(36)
Thus, similar to Li et al. (2020), we get the KL divergence
dQ dP
[///
T
EQ
Eqlu? (x).
(37)
C COVARIANCES
The full derivation of covariances between some processes relevant to this work are described here.
Fractional Brownian motion (Type II). Using Itô isometry (Øksendal & Øksendal, 2003) we know
that for & >>
1 (t u) 11 111/2 1dW ( (s (s- 1)-11/2d ddw = (- ((t ((t- u)(s u)) w)) 11/2 du
E3
(38)
Thus
(1 (1)B
(1)) (5) 12 1/2) '" ((t-u) u)(s - w)) du
E
(39)
OU-processes driven by the same Wiener process. Observe two Cimstein-Uhlenbeck processes driven by the same Wiener process:
(d)(()) -xYY(()) dt dW())
(40)
(t) (t) dt + d(
16

---

Their covariance can be written
Cov(Y,(t))
(() - -E -
(41)
E (+)
(42)
J e e (w())))(ss)) S[ aw
E
(43)
/ ((66tr))((s)) ds
(44)
1
(45)
+
7
Vi +
where Eq. (44) is obtained following the Itō isometry (Øksendal & Øksendal, 2003).
Markov approximated fractional Brownian motion (Type I), Recall that (Dfn.
BiP Σ wx)(Yttt) Yx))))
where (Eq. (6))
(x(t) - (0) (x(0)(e Tut - 1)+ 1'" e-mitt (l (s)))
and ([Y;())),(O)) 7 (Bq. (28)). For 1 T:
E (T (r) = [[_ wk (Va(1) (Va(0)-a((0)) (I
(46)
((( - (X(O))(((T)) -Y((0))
(47)
B (NiC0)c-- nce - 1)+ 10 c (
Σ
(48)
1,7
(,(00)(-70> -1) {- (-75(7-0) (P(M)
Σ
wiwy
B|(((0)),(0)) (e- xit () - 1)
(49)
b,j
+ f ( ds
0
1 7st 777
757 +e xi(t-)
Σ i
whly
(50)
7h +
Markoy approximated fractional Brownian motion (Type II). Recall that (Dfn. 3)
BUTT (t) = Σ wy (t)),
(2 = 0, k: = 1,..,
17

---

and for t >
K
E B (t) B (7) (]] = [uerattt
(51)
Σ
(52)
dW(s) //' e-7(6t=)) dW(s)
Σ
WE
(1
(53)
Σ
Winij
e (0)0-)))7((7)1 ds
(54)
(c-xi(-r)
Σ
WHii
(55)
7
7 7
f
IBM and MA-fBM (Type I). Since (Dfn. 3)
(1) = Σ -
where (Eq. (6)
(2 (1(1) (0) (2(0)(e out -1) 1)+ -1)+ / e a
and (Eq. (25))
0 Y2(0)) /-
T ds).
we can write
(1) = Σ wi (le-at- 1 m dw(s) + f (--natt-)) dw(s)
(56)
This leads to the following derivation (using Itô isometry (Øksendal & Øksendal, 2003)):
Σ wrLt [ (tt ((--))-112-(-33))
( (1) I(H 1/2)
(s
+ + (t - (--) 1/2 dw(s)
+ o
(e (e-Nut- -1) / E dW))+ + /' (- (
(57)
Σ Wk (e (e-wtt //0 (i (1= 1/2 (-s) e ds
T(H + +122)
k
+ 1 ()- 112001((t=)) ds
(58)
2 - e-
Q(H + 1/2, 7
Σ
Wk
(59)
H+//
%
where ((2,z))
1
500 t= - -t dt is the regularized upper incomplete gamma function.
F(2)
18

---

IBM and MA-fBM (Type II).
E WrB I e e- '' (t 1/ (60)
E B (1)B (0)
())
T(H + 1/2)
Σ Wk (
( -) 11/2 d8
(61)
T(H 1/2)
P(H + 1/2, 7kt)
Wk
(62)
H H+1/2 7/
where P(2,z)
r(2) So e dt is the regularized lower incomplete gamma function.
D CHOOSING WE VALUES
D.1 BASELINE
To approximate the integral equation (8) for 1/2 we do a piece-wise linear approximation of the integral between the known (x(() values:
K
we Yx()
K-1
Σ
2h+1
Tk+1 7 Yi((t) +
7-7K 7k Ya+2(1) m(7)
(63)
7k+11 - 7 7k+1 - y
k=1
76
For H > 1/2 we approximate
(t) with finite differences:
K
K w.Y = Σ
1x+1(t) Yx(t)
2011
v(7) dr
(64)
7k+1
Jyk
k=
This leads to the following proposal for Wk:
2-a 2-o
1-0 - 2-0
1
1s
2-a
7k-1
1-a
[(a)[((1 a)
7k -
1k41 -v/
1-a 1-a
2kt1 -2
2-a 2-a
Wk
1 k<K
7k++1
1-a
2-
H 1/2 (65)
7/+1 - 7/2
2-c
2-a
2-x - 7/
2-a
-1
1k
kk-1
kk+11
1k>1
1keK
H 1/2
(2 (a))() - a)
7/K
-
7k+1 7k
where a H + 1/2.
D.2 A PROOF FOR THE OPTIMIZED WA VALUES
To optimize wa values, first provide a closed form expression for the approximation and then show how we can solve for the we that minimize this error.
Type I. We will start by optimizing Wk for Type I. Consider the enror:
(()(w) = ["B. E (B(BB (1)( de
(66)
1/'" (B DD (1)
+ 1y (1)2 21 [B)(t)
(t) (1)) dt
(67)
19

---

Using Eqs. (1), (50) and (59)
(W)(w)= = L-¹
2 2- - 22H
WWWs
7i + 7/
7ke Q (H + 1/2, 7kl) enuL
Σ
Wk
dt
(68)
#+1/2
2T +
e -1
E =77
T28+1
wWWW
Y 7/5
2H +1
2T
TH+1122
e 7x
QH + 1/2,111.11 T) 7
Wk
7k
H+1/2
Ihe
TH + +3/2)
K
H H43/2
(69)
This leads to the quadratic form E(1) (w) = w A - 26(1)²
w + (L)) with
2T
+
-
7-
1
(70)
Yi + 7
2T
TH+112
e TTT
Q(H + 1/2, Tx T
6/
(71)
H41/2
IH + 3/2)
H-3/2
K
7k
T
(
(72)
2H +1
Type II. We now repeat a similar procedure for the Type II.
(w))
//" E
( (1) - -B11 -B//((1) (1)
27
dt
(73)
L
T
E
BuT (t)²] IE B (1) (t)
2E (I)BB dt
(0))
(74)
Using Eqs. (2), (55) and (62)
(w))(w)) = /-" Σ
1 e (t
22H1
P PH + 1/2, Tut)
1.
wiw
i/s +
2HT(H + 1/2)2
2 Σ
Wk
7k
H+1/2
dt
(75)
T + (164t)))
T2
WiWWi
7477
(76)
7/4
+ 75
2H(2H 1)T(H 1/2)2
T
H 1/2
Σ Wi
H+1/2 %
P (H + 1/2, mx)
H+3/2 7
P (H + 33/2,
(77)
This leads to the quadratic form g(1 (w) = T ( 211 T
w + ((I)) with
T (164t,1)²_
(1)
36775
(78)
7/ + Yi
(1))
T
P(H + 1/2, vxT)
H 1/2
P(H + 3/2,
(79)
H+1/2 7/k
H43/2 T
T2H+1
(1I) 2H(2H 1)T 1/2)2
(80)
Exactly solution for w. There is exactly solution if A(LII) is positive definite, which is defined as
Wf A (1,10 0 for all w R (0).
(81)
20

---

Recall that
T w /"
E [ (t)
dt
(82)
thus there is exactly one solution if
1 Bu (1)))
dt > 0 for all R (0).
(83)
Recall that B is a linear combination of K Ornsten-UUliebbeck processes with speed of mean reversion */k driven by the same Brownian motion. Under the trivial conditions that % * 7 (so they not cancel out) and % oc, this will never be O. Hence the last inequality holds unless B (t) = O. This concludes the proof.
D.3 NUMERICALLY STABLE IMPLEMENTATION OF ((z,
The term Q(H + m in Prop. leads to numerical instability, since 7 is typically high number (for the highest ")z). On the other hand, Q(H + 1/2, T) () is a low number for high T. Our stable implementation makes of continued fraction (Cuyt al., 2008, eq. (12.6.17)),
using the 'Kettenbruch' notation (Cuyt et al., 2008, sec. 1.1) for continued fractions:
I(H + 1/2, 7x1) 7
Q + n
(84)
T(H 1/2)
1
am (H + (1/2) (TxT)
K
(85)
I(H + 1/2)(TTT)) H+1/2 m=1
where (m) is given by
a1(a)= 1,
( a2)+1(a) = j 2
(86)
In practice we observe better accuracy with the original equation for T < 10, where it is still stable, and only need fractions to approximate the equation for 7 > 10.
E DETAILS ON MODEL ARCHITECTURES & HYPERPARAMETERS
E. FOU BRIDGE
For all experiments, = and = (2) tions for wk. with time horizon T ......... . 20). The control function neural network with two hidden layers of each 1000 neurons, with tanh activation function. Its input is represented as [sin f, cos t, X (t), Y1 (t)...... (t). The control function initialized so that its output is 0 at the start of training. Models Ba, 2014) optimizer with fixed learning rate 10 10-3, We use the Stratonovich- Milstein SDE solver (Kidger, 2021) with an integration step of 0.01. The length of the bridge T and observation noise 0 0.1.
=
We use "Type I" and the optimal defini-
trained for 2000 training steps with a batch size of 32. We use
the Adam (Kingma
E.2 TIME DEPENDENT HURST INDEX
We directly compare method with the data and estimate found in the published codebase of Tong et al. (2022)². We choose K = 5 and 7h = ( 20) and use "Type II" (to match the data and noise type in Tong et al. (2022)). The optimal definitions for wk. with time horizon T = 2 are used. The control function is neural network with two hid- den layers of each 1000 neurons, with tanh activation function. Its input represented as [sin f, cos f,, sin 21, cos .i. sin 57, cos 5t, X(t), (i)))..... Yx(()), The model is trained for 1000 training steps with a batch size of 4. We use the Adam (Kingma & Ba, 2014) optimizer with learning rate 3 10-3, scheduled with cosine decay to 3 x 10-4 by the end of training. We use SDE solver (Kidger, 2021). The integration step is 0.005 and observation noise = 0.025 (both identical to Tong et al. (2022)).
the Stratonovich-Mileee
Phtpps: http://ghhh...cmmmamttoooo wwt.c///o//ooooootceoeeeooooeeee frattioaaatiaaaa _aauuiinainneurral _neural __de/bb ob/7565a2/
fract ionll laaoraanlaaaaaallaaaa nwwral _aaerralleeeeeeeeeeeeeee ipynb
21

---

E.3 LATENT VIDEO MODEL
Stochastic moving MNIST. For the MA-IBM model, K 5 and 7 = = (1) (z)..... 20). We use 20
"Type I" and the corresponding definitions for wk. with time horizon 7= = 2.4. For the BM model, K 1, 71 = 0 and w = 1, which naturally corresponds to white Brownian motion. The number of latent dimensions D 6.
The encoder model consists of four blocks, containing a convolution layer, maxpool, groupnorm and SiLU activation. Each block reduces spatial dimension by 2, and the number of features in each block is (64, 128, 256, 256, 256). The last output is flattened and is the input of a dense layer, with h as output with 64 features.
The median over the time axis of h is fed into a two layers neural network to produce the static content vector w. Since the median is permutation invariant, w contains no dynamic information, only static information. w also has 64 features.
The context model consists of two subsequent 1 1 D convolutions in the temporal dimension. Thus, information is shared over different frames, which is necessary for inference. The output of this model g.
To start the SDE integration, we need an initial state that is conditioned on the data. We define a three layer neural network model that receives (g1, h,, h2 ha) and outputs the parameters of the posterior distribution Ur of the initial state of the SDE. xı is sampled from Ozy. which we model as a diagonal Normal distribution. The parameters of a prior model Psi are also optimized, and the divergence DKL (Ps, (w, is added to the loss function. This approach for training neural SDEs is similar to others in literature (Li et al., 2020).
The prior drift (X, and the control function u(Z(t), t) have the same architecture, neural network with two hidden layers of each 200 neurons, with tanh activation functions. The shared diffusion og(X (X, is implemented so that the noise is commutative to allow Milstein solvers (Li et al.. 2020; Kidger et al., 2021), i.e. og ox(X. () is diagonal and the i-th component on the diagonal only receives (!) as input, where have defined D separate neural networks for each component. Each neural network has two layers with 200 neurons and tanh activations.
by and dg receive X(t) as input. The control function (t), (t)...... (t), (()). gt) is linear interpolation of g at time t. This enables the control function use appropriate information to be able steer the process correctly.
concatenated vector of
The resulting states x after integration of the SDE are fed, together with the static content vector w in the decoder model. The decoder model has first a dense layer. The outputs of this first layer are shaped in a 4 x 4 spatial grad. Subsequently, four blocks with convolution layer, groupnorm, a spatial nearest neighbour upsampling layer and SiLU activation. Thus, the model reaches the correct resolution of 64 x 64. Two additional convolution layers with SiLU activation and a final sigmoid activation complete the decoder model.
We train on sequences of 25 frames, with time length of 2.4 (0.1 per frame). The frames have resolution 64 x 64 and color channel. Each model was trained for 187500 training steps with a batch size of 32. We use the Adam (Kingma& & Ba, 2014) optimizer with fixed learning rate 3 10 4 SDE solver (Kidger, 2021) with an integration step of 0.033 (3
We use the Stratonovich-Missee integration steps per data frame). Models were trained on single NVIDIA GeForce RTX 4090, which takes around 39 hours for one model.
Double pendulum. We use the train-test split from the original dataset (Asseman et al., 2018). The videos are recorded with a high speed camera, we used every 10th frame to increase the challenge of the dataset. We resized the frames one block to the encoder and decoder model to achieve this resolution, compared to the model for stochastic moving MNIST. We did not use the static content vector w, since there is minimal static information in this dataset, and used D = 8 latent dimensions. The models were trained for 124916 training steps, and we trained around 32 hours for one model. Beyond these outlined differences, all other details are equal to the stochastic moving MNIST model.
resolution of 128 x 128 resolution. Therefore, we added
22

---

F ADDITIONAL EXPERIMENTAL RESULTS
F.1 GENERATED TRAJECTORIES OF MA-FBM FOR VARYING K
Included here are some of the trajectories used to calculate the MSE of the generated trajectories for MA-fBM for varying K (Fig. 7). We show trajectories of MA-fBM with our approach (Sec. 4.1) and the baseline method (cf. App. D.1) for choosing wk- True paths are plotted in black, the approx- imations with varying K in color-scale as indicated in the legends, see Figs. to 12 and 14 to 16. Our method quickly converges to the true path increasing K, while much slower for the baseline method.
(a) Baseline
(b) Ours
Figure 9: Generated trajectorics of (a) the baseline method summarized in App. D.I and (b) our method, MA-IBM (Sec. 4.1), for varying K and H 0.1.
Tmis path
Trui
10
(a) Baseline
(b) Ours
Figure 10: Generated trajectories of (a) the bascline method summarized in App. D.1 and (b) our method, MA-fBM (Sec. 4.1), for varying K and H = 0.2.
F.2 FOU BRIDGE
Fig. shows additional results of the fractional Ornstein-Uhlenbeck bridge. The variances are calculated with Eq. (23), and Eq. (24) for 0 >> 0 and H > 1/2 Eq. (1) for 0 0. Note that do not have a useful covariance equation for > 0 and H 1/2 (Lysy & Pillai, 2013), so this setting not included in the experiments.
23

---

(a) Baseline
(b) Ours
Figure 11: Generated trajectories of (a) the baseline method summarized in App. D.1 and (b) our method, MA-IBM (Sec. 4.1), for varying and H 0.3.
8
Inc path
Trem
X10
10
(a) Baseline
(b) Ours
Figure 12: Generated trajectories of (a) the baseline method summarized in App. D.1 and (b) our method, MA-fBM (Sec. 4.1), for varying K and H 0.4.
Tup
300
(a) Baseline
(b) Ours
Figure 13: Generated trajectories of (a) the baseline method summarized in App. D. and (b) our
method, MA-IBM (Sec. 4.1,, for varying and 0.5.
24

---

X00
(a) Baseline
(b) Ours
Figure 14: Generated trajectories of (a) the baseline method summarized in App. D.1 and (b) our
method, MA-fBM (Sec. 4. for varying and H 0.6.
X00
10
(a) Baseline
(b) Ours
Figure 15: Generated trajectories of (a) the baseline method summarized in App. D.1 and (b) our
method, MA-fBM (Sec. 4.1), for varying K and H 0.8.
320
Baseline
(b) Ours
Figure 16: Generated trajectories of (a) the baseline method summarized in App. D.1 and (b) our
method, MA-BBM (Sec. 4.1), for varying and H 0.9.
25

---

X(0)
X(O)
0.0
0.5
1.5
2.0
0.5
1.0
1.5
200
0.0
0.5
1.5
2.0
t
(a) = 0.1, 0.0
0.2,
0.0
(c) = 0.3, 0.0
2
X(0)
x(
X(0)
0.0
0.5
1.5
2.0
1.0
1.5
0.0
0.5
2.0
t
(d) H = 0.4, 0
0.0
(c)
H 0.5,
0.0
(f) = 0.6,
0.0
2
1
X(0)
X(0)
X(1)
0
0.0
0.5
1.5
2.0
1.0
1.5
0.0
0.5
1.5
2.0
(g) H = 0.7,6 0
0.0
(h)
0.8,
0.0
(i) = 0.9,
0.0
2
1
X(X
X(
X(X)
0
-1
-1
0.0
0.5
1.5
2.0
1.0
1.5
2.0
0.0
0.5
1.5
2.0
t
( = 0.6,
1.0
(k)
0.7,
1.0
()) = 0.8, 1.0
X(0)
0.0
0.5
2.0
(m) H 0.9, 0 1.0
Figure 17: The true variance (blue) of fOU bridge matches the empirical variance (dashed orange)
of our trained models. The transparent black lines are the sampled approximate posterior paths used calculate the empirical variance.
26

---

F.3 VIDEO MODELS
On the choice video datasets. We conducted experiments on two video datasets: Stochastic Moving MNIST (SM-MNIST) (Denton & Fergus, 2018) and the real video dataset of chaotic double pendulum (Asseman et al., 2018).
SM-MNIST and the double pendulum dataset contain different forms of nuisances and present dif- ferent challenges to our stochastic model. First, SM-MNIST digits move with a constant velocity along trajectory until they hit at wall at which point they bounce off with random speed and direction. This sudden event intersperses the deterministic motion with moments of uncertainty, i.c. each time digit hits a wall. This is the reason why a stochastic model fits better than an ODE and unlike BM, our noise can model the smooth and correlated trajectory simply by raising the Hurst index.
On the other hand, the double pendulum dataset is actually governed by set of coupled ordinary differential equations. However, despite being a simple physical system, exhibits a rich dynamic behavior with a strong sensitivity initial conditions and noises in the environment (motion of the air in the room, sound vibrations, vibration of the table due to coupling with the pendulum etc.). Combined with the chaotic nature of the system, this creates major challenge for any model based upon smooth ODEs. Our model on the other hand heavy lifts this difficulty onto the (fractional) stochastic noise, leading the BM baseline also in this dataset.
more appropriate model. As shown in Tab. 2, model outperforms
Figs. 18 and 19 show the posterior reconstructions of models trained the Stochastic Moving MNIST and the double pendulum dataset respectively. Fig. 20 shows stochastic video prediction samples of Stochastic Moving MNIST.
Ground truth
P
P
Q
2
84
81
8t
84
8
8
B1
3
8
6
8B
8
8
8
BM
P
P
*
21
88
21
81
2
81
8
8
21
1
P
P
&
24
84
8
81
a
l
lg
Figure 18:
Posterior reconstructions model driven by BM and a model driven by MA-fBM, conditioned on the same data ('Ground truth').
Ground mith
Posterior reconstructions of model driven by BM and model driven by MA-fBM, trained on the double pendulum dataset. Both are conditioned on the same data ('Ground truth") We show 7 evenly spaced frames of the total 20 frames.
Figure 19:
27

---

Figure 20: Stochastic predictions using the trained prior of a model driven by BM and a model driven by NA-IBM, where the initial state is conditioned on the same data. Four samples are shown for each model. The MA-fBM samples show more diverse movements, thus better capturing the dynamics the data. The BM samples are more similar, indicating& a less powerful prior was learned.
#
44
94
47
4
4
47
H
94
94
94
47
4
4
47
94
F
94
47
4
47
49
94
74
F4
94
47
47
47
4
94
44
#
7
47
4p
47
94
FF
F
9 4
47
4
47
94
F4
4
9
4
47
48
47
94
94
9
4
47
4
3
47
94
94
4
4
47
4
3
47
94
94
F
9 4
47
4
47
94
94
94
47
4
4
7
4
44
94
94
47
7
4
4
44
44
94
47
4
4
74
74
+
94
47
4
7
4
94
4
74
47
4
4
7
4
94
#
#
4
4
4
7
4
4
4
4
#
4
4 7
#
#
4
#
47
4
4
47
#4
9
4
4
4
7
4
4
47
4
47
44
44
47
47
4
47
#
44
44
94
47
47
47
47
#
FF
94
47
47
47
47
#
94
94
94
47
47
47
47
94
94
4
47
4
4
MA-1BM (1)
(1)
MA-IBM (2)
MA-BBMCC (4) (4)
MA-IBM
(1)
(2)
(3)
(4)
BM
BM
BM
BM
28


Reviews:
[{"review_id": "hNIx9DQVPv", "reviewer": "Reviewer_J9Kf", "summary": "A variational framework for learning latent SDEs driven by fractional Brownian motion (fBM) is introduced. To efficiently infer the variational parameters, an approximation of fBM by Markov processes is utilized, which reduces the problem to inferring parameters of SDEs driven by (non-fractional) Brownian motion.\nIn applications, the drift, diffusion and a control term are implemented as neural networks. Empirically, the methods capability to recover a fractional Ornstein-Uhlenbeck bridge and the Hurst index are evaluated. The method is also applied to a video modelling task.", "questions": "**Extension to different tasks**\n\nIt seems that in its current form the method can only be applied when the model output should match the model input. This is because the control function $u$ at each time point depends on the evidence at that time point. How would you extend the method to forecasting or interpolation tasks, where this is not possible?\n\n---\n\n**Proposition 5**\n\nIf I understand correctly, then $\\hat B$ depends on $\\omega$ and $\\gamma$ and the goal is to find $\\omega$ such that the L2 error of the approximation is minimal. The minimum is achieved when $\\omega$ solves $A\\omega=b$.  \nDoes this equation have a solution? How large is the approximation error? How is $\\omega$ computed; is it required to first compute the entire matrix $A$?\n\n---\n\n**Runtime**\n\nI wonder how the runtime compares to Tong et al., and to a latent ODE.   \nDoes K=0 in Fig. 9 correspond to a latent SDE with non-fractional Brownian motion?  \n\nIs Fig.9 qualitatively the same, if training time (forward+backward) is measured instead of inference time (only forward)?  \n\nIn the \"Impact of K and the #parameters on inference time.\" experiment, it is stated that *\"the run-time is still dominated by the size of the neural networks\"*. Presumably, this refers to the networks that implement the drift, diffusion and control and not the encoder / decoder networks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work addresses a difficult problem and proposes a novel solution. The inference of latent SDEs driven by Brownian motion is already challenging, in this work the setting is extended to fractional Brownian motion while avoiding limitations of prior work (Tong et al, 2022)\n- The background section of the paper is well written and does a good job in condensing the involved theory of stochastic calculus (for fBM) into a short conference paper.\n- While the method is evaluated on video prediction, it appears to be applicable to any type of sequential data.", "weaknesses": "From my point of view, there are 2 main weaknesses in this submission (for details, see below).\n \n 1. The method and the experiments are insufficiently described, and I have some questions in this regard. However, I am convinced, that the manuscript can be updated to be much more clear.\n \n 2. The empirical evaluation is of limited scope. Qualitatively, the method is evaluated on 2 toy problems (fOU & Hurst index); quantitatively on a single synthetic dataset (stochastic moving MNIST). For the latter, only two baseline models from 2018 and 2020 are compared to. In consequence, the usefulness of the method is not established. On the plus side, there are 3 ablations / further studies.\n \nMinor weaknesses.\n- The method appears to be inefficient, with a training time of 39 hours on an NVIDIA GeForce RTX 4090 for one model per model trained on stochastic moving MNIST. \n\n- A detailed comparison with Tong et al. (2022) (who also learn approximations to fBM) is missing. So far, it is only stated that Tong et al. did not apply their model to video data and that it is completely different. A clear illustration of the conceptual differences and a comparison of the pros and cons of each approach would be appreciated (summary in main part + mathematical details in supplementary material).\n\n# Summary \nFor me, this is a borderline submission. On the one hand, the proposed method is novel, significant and of theoretical interest. On the other hand, there are clarity issues, a weak empirical evaluation and no clear use case. Expecting the clarity issues to be resolved, I rate the submission as a marginally above the acceptance threshold, as the theoretical strengths outweigh the empirical flaws.\n\n----\n# Details on major weaknesses\n\n\n## Point 1 (clarity). \n\n**Regarding the method.** After reading the method and experiment section multiple times, I still have no idea, how to implement it. I am aware of the provided source code, nevertheless I found the paper to be insufficient in this regard.  \n\nWhat I got from section E and Figure 4 is that:\n- First, there is an encoding step that returns $h$, a sequence of vectors/matrices over time. Somehow, these vectors are used to compute $\\omega$. I have no idea how this $\\omega$ is related to the optimal one from Prop. 5.\n- $h$ is given to a temporal convolution layer that returns $g$ and this $g$ has as many 'frames' as the input and is used as input to the control function $u$.\n- The control, drift and the diffusion function are implemented as neural networks. \n- An SDE solution is numerically approximated with a Stratonovich–Milstein solver.\n- $\\omega$ is used in the decoding step, I do not understand why and how.\n- Where do the approximation processes $Y$ enter. How are they parametrized, is $\\gamma$ as in Prop 5? Are they integrated separately from $X$?\n- The ELBO contains an expectation over sample paths. How many paths are sampled to estimate the mean?\n- Fig. 6: Why do the samples from the prior always show a 4 and a 7? Does the prior depend on the observations?\n\n**Regarding Moving MNIST.**\nWhat precisely is the task / evaluation protocol in the experiment on the stoch. moving MNIST dataset? I did not see it specified, but from the overall description it appears that a sequence of 25 frames is given to the model and the task is to return the same 25 frames again (with the goal of learning a generative model).\n\n## Point 2 (empirical evaluation)\n- The method is not evaluated on real world data.\n- Quantitatively, the method is only evaluated on one synthetic dataset (stoch. moving MNIST).\n- While the method is motivated by \"*Unfortunately, for many practical scenarios, BM falls short of capturing the full complexity and richness of the observed real data, which often contains long-range dependencies, rare events, and intricate temporal structures that cannot be faithfully represented by a Markovian process\"*, the moving MNIST dataset is not of this kind. It is not long range (only 25 frames) and there is no correlated noise.\n- The method is only compared to 2 baselines (SVG, SLRVP) on moving MNIST.\n- Table 1 does not show standard deviations.\n\nOverall, this would be a far stronger submission, if the experiments were more extensive. This includes: \n- Evaluation on more task and datasets, and specifically on datasets were this method is expected to shine, i.e., in the presence of correlated noise. The pendulum dataset of [Becker et al., Factorized inference in high-dimensional deep feature space, ICML 2019] would be one example.\n- Comparison with more baselines. In particular, more recent / state-of-the-art methods that do not model a differential equation and the fBM model by Tong et al.\n\n\n----\n\nThere is a typo in Proposition 1: \"Markov rocesses\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4t8vLsXPhk", "reviewer": "Reviewer_3XSn", "summary": "The paper introduces a novel variational framework for conducting inference in stochastic differential equations driven by Markov-approximate fractional Brownian motion. This framework combines SDEs with variational methods to enable the learning of representative function distributions through stochastic gradient descent. Unlike conventional SDEs that assume Brownian motion, this framework extends to fractional Brownian motion to capture long-term dependencies. Authors derive the evidence lower bound for efficient variational inference and provide a closed-form expression for determining optimal approximation coefficients. The framework is validated on synthetic data, and it also presents a novel architecture for variational latent video prediction.", "questions": "1. Why is it believed that Brownian motion cannot capture long-term dependencies, while fBM can?\n\n2. Why does fBM encompass non-Markovian dynamics?\n\n3. Is it necessary to perform the Markov approximation for fBM? What is the extent of error introduced by this, and does it significantly impact accuracy? Did the use of Markov representation not compromise the ability to capture long-term dependencies?\n\n4. Why was latent video generation chosen for experimentation, and why were experiments not conducted on datasets other than MNIST?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces generative models that utilize fractional Brownian motion (fBM) as a noise injection, as opposed to conventional score-generative models based on Brownian motion.\n\n2. This paper offers a solution to the limitations of standard Brownian motion by extending the framework to fractional Brownian motion, which better captures long-term dependencies and complexities in real-world data.\n\n3. This paper provides a clear and detailed explanation of the proposed framework and its mathematical foundations.", "weaknesses": "1. The experiment lacked sufficient data, and the basis for asserting the presence of long-term dependency was inadequate.\n\n2. An approximation was applied in deriving the theory, and there is no analysis regarding the errors introduced by this approximation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tovLNcnC7T", "reviewer": "Reviewer_9F2a", "summary": "Generative modeling of sequential data (i.e., time series data) is a vibrant research area and has made remarkable progress since the introduction of neural SDEs in recent years. Related to this are variational Bayesian inference methods combined with the application of Girsanov’s change of measure theorem, which have paved the way for learning representative function distributions through stochastic gradient descent. Most of the progress in this area has been on models based on the use of (standard) Brownian motion, i.e., a Markov process. The present work extends existing approaches by building on *fractional* Brownian motion (fMB; a non-Markovian process), questioning the everlasting validity of the Markov property assumption. The key to success lies in the application of Markovian embedding of the fBM and its strong approximation which makes the traditional machinery of neural SDEs accessible for working with non-Markovian systems. Conducted experiments on real data sets as well as ablation studies provide empirical evidence for doubts about the unrestricted model suitability of previous approaches in real world applications.", "questions": "1. Can you elaborate on the difference between a Type I and a Type II fBM and which form delivers benefits in which circumstances?\n2. It would be of importance to comment on the assumptions that guarantee a finite KL term inside of the ELBO, i.e., the diffusion terms of prior and posterior have to be equal. \n3. According to (Oksendal 2003, The Girsanov Theorem II), the control function $u$ needs to meet Novikov's condition. In the proposed work $u$ is parameterized by a neural net. How is the aforementioned condition controlled during training?\n4. Can you elaborate more on the following: \"Unlike Tong et al. (2022) our approach is agnostic to discretization and the choice of the solver.\"\n\nMinor:\n- Missing parenthesis in Eq. (5)\n- Proposition 1 : Markov rocesses -> Markov processes\n- $\\text{d} t$ vs $dt$", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I enjoyed reading this paper because (i) I like the idea of challenging common theoretical assumptions (in this case, the Markov property), (ii) the very well-organized writing style that makes it easy to follow the arguments and (iii) the thorough presentation of the theoretical aspects. Further, the paper content is original to the best of my knowledge and well positioned to existing work, as well as with very few exceptions, has no spelling or grammatical flaws. Empirical evidence is accessed through experiments on synthetical and real-world video data. The results are on a par with compared existing methods and reproducibility is granted by submitted code files. Finally, on a positive note, the conjecture that data contain information that cannot be represented by Markov processes is empirically tested by estimating the Hurst index.", "weaknesses": "A major drawback of (traditional) NSDE approaches in the context of variational Bayesian inference is the computational and storage costs associated with learning almost arbitrarily complex representative function distributions. Recent approaches address this problem and attempt to find solutions, e.g., (Li et al. 2020) and (Kidger et al. 2021). However, learning large datasets still seems to be very resource intensive. In the context of the present work, I wonder to what extent the additional Markov approximation scheme exacerbates the aforementioned problem. For example, in the case of the video experiment presented, the authors report (p. 19, Appendix E.3) \"Models were trained on a single NVIDA GeFORCE RTX 4090, which takes about 39 hours for one model.\" Can you please provide a more detailed evaluation of runtime in addition, including a comparison to baseline approaches?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLQHO6ds03", "reviewer": "Reviewer_kkY2", "summary": "The paper presents a novel variational inference method for stochastic differential equations (SDEs) under fractional Brownian motions (fBMs). The key property of fBMs lies in long-term dependence in its stochastic generation. In order to devise the proposed variational inference, the paper suggests to use approximation based on the result of Harms & Stefanovits. That is, fBMs can be represented as integral (infinite linear combination) via Ornstein-Uhlenbeck processes. The SDEs under fBMs are then rewritten as a system of SDEs under Brownian noise.", "questions": "Minor points:\n-  In the first paragraph of Background section, do you mean \"learning comunity\" as \"machine learning comunity\"?\n- At Proposition 1, “Markov rocesses\" should be “Markov processes”.\n- At Proposition 2, \"R^D\" should be \"\\mathbb{R}^D\".", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is overall well-written, providing an excellent introduction of fractional Brownian motion including type I, and type II. \n- Showcase a practical use of Harms & Stefanovits, 2019. This is a nice approach compared to Tong et al. 2022 suffered from some limitations including the choice of solvers.\n- Interesting application learning for SDE.", "weaknesses": "- Perhaps variational inference may not be the main contribution of the paper since it is still a result of Girsanov's theorem like presented in Li et al. 2020. However, I believe that the idea of finding SDE approximation outweights this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)