Learning Fractional White Noises in Neural Stochastic Differential Equations
Authors: Anh Tong
Thanh Nguyen-Tang
Toan Tran
Jaesik Choi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Differential Equation
Neural Stochastic Differential Equation
Gaussian Process

Abstract: Differential equations play important roles in modeling complex physical systems. Recent advances present interesting research directions by combining differential equations with neural networks. By including noise, stochastic differential equations (SDEs) allows us to model data with uncertainty and measure imprecision. There are many variants of noises known to exist in many real-world data. For example, previously white noises are idealized and induced by Brownian motions. Nevertheless, there is a lack of machine learning models that can handle such noises. In this paper, we introduce a generalized fractional white noise to existing models and propose an efficient approximation of noise sample paths based on classical integration methods and sparse Gaussian processes. Our experimental results demonstrate that the proposed model can capture noise characteristics such as continuity from various time series data, therefore improving model fittings over existing models. We examine how we can apply our approach to score-based generative models, showing that there exists a case of our generalized noise resulting in a better image generation measure.

Learning Fractional White Noises in Neural Stochastic Differential Equations
Anh Tong KAIST anhtong0kaist.co kc.kr
Thanh Nguyen-Tang Johns Hopkins University nguyent@cs, jhu. edu
Toan Tran
Jaesik Choi
VinAl Rescarch, Vietnam v. toantm3@vinai. io
KAIST, INEEJI
jaesik. chw.chooiiieeoee Peoo@kaait..oo ac,kr
Abstract
Differential equations play important roles in modeling complex physical systems. Recent advances present interesting research directions by combining differential equations with neural networks. By including noise, stochastic differential equa- tions (SDEs) allows to model data with uncertainty and measure imprecision. There are many variants of noises known to exist in many real-wo wolld data. For example, previously white noises are idealized and induced by Brownian motions. Nevertheless, there is lack of machine learning models that can handle such noises. In this paper, we introduce generalized fractional white noise to existing models and propose an efficient approximation of noise sample paths based on classical integration methods and sparse Gaussian processes. Our experimental results demonstrate that the proposed model can capture noise characteristics such as continuity from various time series data, therefore improving model fittings over existing models. We examine how can apply our approach ccore-based gen- erative models, showing that there exists a case of our generalized noise resulting in a better image generation measure.
Introduction
Differential equations center around modeling relations between functions and their rate of change, playing important roles in many dynamical systems in physics, biology and finance. Incorporating such a tool in machine learning not only helps us understand more about machine learning models [42, 66, 44), but also expands the range of applications [31, 12]. Recent work neural ordinary differential equations (neural ODEs) opens a new direction of connecting differential equations and neural networks []]]. Extensions stochastic differential equations (SDEs) proposed for handling noise in data or model uncertainty [83, 84, 54, 46).
Noises in data often encountered in many real-world cases, and can directly affect model perfor- mance. The importance of understanding the characteristics of noise in data has been emphasized in number of studies including: heteroscedastic noise modeling in time series analysis [82, 57], volatility modeling in financial research and market simulation [40, 14]; the significance of colored noise in neuroscience [20, 4]. Rather than using traditional Brownian noises [5] neural SDEs. providing a capacity to model more generalized type of noises not only allows this framework to extend to such applications but also retains the expressiveness from the representation powers of neural networks. By learning noises together with the drift functions and diffusion functions of SDEs, the models become more flexible as the drift functions and diffusion functions, in turn, will be adjusted according to the learned noises, Although mathematical research on analyzing general white
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

noise is well-developed [38, 331, it is still challenging to implement practically for machine learning models like [54, 46].
In this paper, we introduce a novel class of white noises defined under a stochastic integral form [38, 33] to existing neural SDE models [83, 84, 54, 46] which only consider noises generated by Wiener process. The most prominent examples of such stochastic processes include fractional Brownian motions (fBm) and multifractional Brownian motions (mBm) which are generalizations of Brownian motions or Wiener processes. The importance of such processes is emphasized in many applications from financial modeling [13, 28], to motion tracking physical and biophysical systems [29, 35, 41, 17]. Existing techniques sampling these stochastic processes [18, 65] prohibit models scale up as they require keep track of sample paths during SDE solves [54]. To overcome this, we propose approximation under numerical SDE solvers considering increments at a time. That is, we first establish a discretized version stochastic integrals to formulate distribution over noise increments. The key approach is that we borrow the idea in sparse Gaussian process literature [74, 37] to decouple the dependency between increments. Therefore, we can reparameterize the increments to obtain an equivalent SDE which is easy to solve. To show the convergence of our approximations to SDE solutions, we study the approximated covariance functions of underlying stochastic processes to establish error bounds using rough path theory [56, 26] and existing sparse Gaussian process bounds [8].
The contributions of this paper are threefold. First, we introduce a new type of neural SDEs with general white noise. Second, we propose an approximation of that white noise, which is inspired by approximation methods in Gaussian process research to obtain equivalent neural SDEs. Finally, we demonstrate empirically that our approach can capture noise characteristies and provide insight of how white noise affect learning score-based generative models as well as observe a setting that improves baseline models.
2 Related work
Differential equations and neural networks [11] presents neural ODEs using numerical method in solving differential equations to learn parameters in neural networks. There are a number of follow-up works aiming to improvements in different aspects including numerical solver and/or training process [45, 43, 24, 90, 61], implicit termination [10], theoretical understanding [22, 60], and extensions to graph neural networks [9]. Moreover, it has seen some applications in normalizing flows [89, 31, 39] as well as in control [21, 59]. Modeling time series data is one of successful practices of such models [12. 91]. Not close but related, work by [77] demonstrates excellent results on image generation tasks. Like SDEs are a stochastic extension of differential equations, neural SDEs are natural extensions of neural ODEs when incorporating uncertainty into models. However, the formulation is not always straightforward. [83, 84] established theoretical aspects emphasizing high expressiveness of class of neural SDEs while [54, 46] present how to train neural SDEs efficiently. Many advanced tools in stochastic analysis like rough paths [56] are incorporated into differential equations, and therefore empowered machine learning models [49, 48, 64, 63].
SDEs and Gaussian process There equivalences and connections between Gaussian processes and linear lime-invarian SDEs that [80] discusses to a large extent. [73] presents a faster learning Gaussian process using filtering and smoothing methods [79]. Models of this research direction often focus on spatial-temporal data and establish state-of-the-art results. [36, 88) learn differential equations with Gaussian processes instead of neural networks. [19] considers similar noise to ours as filtered white noises, finding an analytic form of corresponding kernels in the same manner as [73]. Unlike our approach focusing on a generie procedure, this work studies specific class of kernels. Gaussian processes can be a tool to model posteriors over paths [1] or to match with densities from Beaker-Planck-oo Kolmogorov equations [75]. [55] devises variational inference to approximate posterior inspired by work in sparse GP latent variables.
3 Background
3.1 White Noise
Brownian motion has a long history dated back from the observation of pollen in water in [5], then was studied formally by Einstein [23]. In this paper, we focus generalization of Brownian

---

Given inducing points
3B2
dependent
indegennent
ABB
A AB, |A
approximated by
AB,
AB,1AB
v(t),s(t). AW,
parameterized by
dX. -
Xx)dt XXt,
(t, X,)dB,
t (l, dX, - oo X,)u
ol,, (DWW
Figure 1: Illustration of sample paths of B₁ in neural SDEs. Left: Neural SDE with B, So [ss l)dW.. Any increments of B, are correlated. Right: Our approximation when a set of inducing points for increments is given, the increments now are pairwise independent and are rewrit- ten as a mean part vt),, a variance part (t)) and an increment of new Wiener process, AW, Finally, approach solves alternative SDE.
motions under Riemann-Liouville fractional integral defined by:
1 B ((((t)++/2) I. (t s)h(t)-
(I)
where (t(t)= - (0, 1) is the Hurst exponent, W. is Brownian motion and I'(.) is the Gamma function. When (tt) is a constant H. this stochastic process resembles a fractional Brownian motion (fBm). However, noted in [58], it places a great emphasis the origin (t 0) and no longer possesses power-law spectral density like in fBm. As we are interested in the case that h(t) is function of t, emphasize more the local variation of Hurst exponent than such properties. Our formulation in (1) can be understood as a variant of multifractional Brownian motions (mBm) [69]. One of the goals in this paper is to learn (t;;; () using function approximator, for example, a neural network with parameter 0.
Interpretation of Hurst exponent The stochastic process in (1) asymptotically shares some properties with iBm and mBm [65] including long-range dependencies and self-similarity. We refer our further analysis related to (I) as iBm or mBm. Here, long-range dependencies are defined for a stochastic process if the sum of autocovariance function is unbounded (see Appendix A.4). Time series analysis has a similar notion for fractional ARIMA models [78]. Deep learning approaches also try to incorporate such long-ram rangraage dependency their models [32].
With the Hurst exponent (t),, B, exhibits a long-range dependence when l(t)) > 1/2. As l(t)) = 1/2, the stochastic process falls back to Brownian motion which no longer has long-range dependency. When (t) < 1/2, the stochastic process is "rough", anti anti-persistent, having irregularity. This parameter also expresses Hölder continuity of stochastic processes [26]. If h(t) changes over time, this means all of such characteristics happen locally.
Brownian motion, Br, or Wiener process has rich history in modeling physical system [67]. Black- Scholes models [40], long-standing models in studying financial markets are based on Brownian motions. However, recent work shows fBm is more suitable due to the irregularity of volatility [14]. Correlated noise [7] also can be found in fMRI data. Applications of fractional Brownian motions can be found in [87] [15].
3.2 Background of Neural SDEs
Consider dynamical system {X.) whose dynamic evolution is governed by the following stochastic differential equation
dx, = (tt,, x2) + (t., X
(2)
where μ : [0. T| R + RR is the drift and a [o, T] x R" + Raxm is the diffusion. Both of those functions are assumed to well-behaved (i.e., smooth and linear growth) and are parameterized by neural networks. Readers can refer to [80] for gentle introduction of SDE in machine learning or to [72] for advanced materials. [83, 84, 54, 46) are excellent starting points be familiar with neural SDEs.
3

---

Sack-propagatton using cenemiizee-tee-discreeee strategy This strategy is adopted in neural ODEs, allowing to perform prok-ppooagttton in time [11]. Suppose we are interested in optimizing a loss function L. at the terminal state Xy. This approach considers a variable 0L/OX, The dynamic of this variable can be modeled by differential equation leading to computing gradient by using chain rules.
In neural SDEs, such strategy requires more careful treatment [54] addresses the randomness in , Also, forward and backward SDEs need share the same sample paths of Wiener process.
Model learning There are two main parameter estimation methods for Neural SDEs. The first one is latent SDE [54] that follows a variational approach approximating posterior distributions by a neural SDE which shares the same diffusion with priors. In this framework, the evidence lower bounds are derived by applying the Girsanov theorem. Recently, [47] proposes SDE-GAN inspired by generative neural networks [30]. In this paper, we mainly use the former in our experiments. However, is worth noting that there no restriction training with SDE-GAN as long as chosen models satisfy our assumption.
3.3 Sparse Gaussian processes
We also give a brief introduction of sparse Gaussian processes which are known as scalable methods for Gaussian processes. A Gaussian process (GP) [71] is defined by set of random variables, any finite number which have a joint Gaussian distribution. Formally, a GP completely specified by a mean function m(.) which usually is set to be zero for prior and covariance function k(,,)) :). We then can denote a GP by (x) ~ GP(O, k(z, z')). Learning or sampling from a Gaussian process usually takes (NN*) with the number of data points N. Sparse Gaussian process methods [74, 37] introduce a set of pseudo points (inducing points) u at locations Z which jointly follow a Gaussian distribution with data points. The density function of f(x), p(f(x)) is then approximated by p(f(z)lupp(t) Here the mean and the variance of p(f(x))(u can be computed as
m(:; u, Z) a(x) u, (x; Z) = k(x, x> <xx) k(Z, Z)a(z),
(3)
where (xx) = ()),,))>k(Z,,) scalability compared to traditional GPs [6. 3, 86, 37]. In this paper, focus the sampling procedure where we first sample u ~ (0, k 2, Z)). then sample f(x) ~ p(f(x)lu)
There a growing interest in this approach because of its
4 Learning White Noises in Neural SDEs
This section presents our main model. We first approximate stochastic integrals using Stieltjes approximations and then use sparse Gaussian processes to leverage dependencies between random variables. We analyze the convergence to SDEs solutions for this approximation. Finally, we form an equivalent SDE to our model (see Figure 1).
4.1 Neural SDEs with B,
Consider stochastic differential equation
dXx n(l,, X.)dl + (1, Xt)d I E [0, T]: with y [0, R + R and : [0, x R. + Raxm Compared to (2), the noise coming from B, is an m-dimensional stochastic process, B, = S s(1, s),W,, from (1) as we denote s(,,s) 175()) (t-))+(()-))) (//)
We are interested in learning both the parameters of put,, Xt), at. Xt) and that of ft, s). Solving SDES in (4) is challenging since B, is constructed from an integral. Direct sampling methods of B, (i.e., require computing Cholesky decomposition) are costly. Other approaches [65] use an approximation of stochastic integral as a linear combination of random variables. However, the number of these random variables prevents optimize-then -eieetthhen-d-diizeee methods to scale.

---

4.2 Integral approximation
We use Stieltjes integration to approximate the integral in (1) by discretizing the interval [0, I] with
0 = 81 <82< < ... < 88 = -T.. i.e.
1ls E (ttff B₁ L f(t,s) /'" T
N-1
1s E (t) -..))
N-1
N-1 Σ g(t)s,.
Σ 1i8 E (ott) (
(5)
(()
where E are independently distributed standard normal random variables coming from W 9:11 W.: (00, As;) with As; = 8;-1 8i, and {s E [0, { is the indicator function. Note that this is a reasonable approximation because Itô's 's integrals, in general, converge in L when N goes to infinity. Moreover, even if We has infinite variation, the probability of a sample path leading to the divergence of approximation infinitely often is zero [72]. The discretization is also inevitable whenever is required to sample B.
In several numerical approaches, for example, Euler-Maruyama [51], solving stochastic differential equations involves approximating increments, dB,,, by
N
N-1 Σ (t),
A. == BL+di - B, ~
Σ (t)++tt) (t))) 1=1
(6)
==1
where (t) gi(t At) (t)). Comparing increments AW, when solving (2), simulating AW, is relatively simple as they are independent between each time step. On the other hand, there exist dependencies between AB as
N-IN-1 ΣΣ (t) = Σ (t)))(tt)) + ==1
N-1
E[ABAABe ~
when i + j. Our aim is to represent (6) in way that is simple to sample from but preserves the dependencies between AB,
Here, Ey (0, 1), and E[5,5)]
4.3 Sampling increments with sparse Gaussian process
From the observation in (7), we devise a sampling method inspired by sparse Gaussian processes. In particular, we consider a Gaussian process' B₁ = Ag
AB, for a fixed At. In fact, At is later used as the time step of SDE solvers. Note that directly modeling AB, in (6) is numerically unstable when computing (7) with covariance matrices having small values.
Indeed, BY is Gaussian process because at a given time I, B' is Gaussian with zero mean. To compute the covariance function lctt, s), simply compute k(t, s) = B|B [[B B[B and follow the same step with (7)
k(t, 8) B[BBB']
(D1)2 Σ (:))ssss.
(8)
To this end, we denote B ~ GP(O, s)). Sampling from a Gaussian process based on a full covariance matrix is not scalable. To overcome this difficulty, we propose sampling scheme based on sparse-inducing GP [74]. The main idea of this method is to introduce a bottleneck as a summation of the joint distribution of BY
Define inducing-point locations Z {Z1,..., Zar) which spread across [0, T). The number of inducing points, M, is much smaller than the number of discretization mesh of numerical solvers. The inducing points ...,, B ) distributed as multivariate Gaussian distribution, i.e., (N A(Z Z)). Following the description of sparse GPs in $3.3, p(B)) is approximated by p( (()))))(
The sampling procedure of B₁ consists of two steps:
Step 1: u == I.ze,
E ~~ NO TM),
(9)
Step 2: B' = m(t: u, 2) + rt: Z)E,,
& ~ N0 1).
(10)
Here Lz is the Cholesky decomposition of k 2), IM is an M x M identity matrix. m(-) and v(-) the conditional mean and variance taken from (3).
This can be viewed as the "derivative" of Br w.r.t time. However, B, is known non non-differentiable.
5

---

The above is a complete sampling procedure for p(B; u)p(u)). Note that in our approximation, B; is independent conditioned on that u are given. Therefore, we obtain the increment AB, as
AB. BB m((;; w) v(t; AlG, m(4; u, Z) + v(t; A.VVIAWW (11)
Here, we replace v VAIE, with increments of a (new) Brownian motion W, since & are independent for E [0, T..
Remark One may consider an alternative approach where we use sparse approximation directly for B, with given kernel function k(tt,)) x | 22hc,a + |s| - - |t s|ahe,> (see (16)), and can obtain a similar form of increments. However, this kernel function is not differentiable w.r.t the parameters of Hurst function. Our approach is to devise a "so-called" differentiation of B, under discretization.
Note that this notion differs from the definition the derivative B, in [52] in the sense of Schwartz space. However, such a representation is not useful for practical implementation.
Our approach shares a similarity with [49] on controlled differential equations driven by signal, i.c., Xe. The signal X; is approximated by linear interpolations which allows to compute the derivative of X₁. We intentionally model the "derivative" of B, by B/ first, and then reconstruct the signal B, from B,-
4.4 Forming equivalent SDEs
To this end, we show that the increments A Br can be represented with increments of a Wiener process, A AW, in (11). Here, the equivalence is under the assumption that the numerical procedure involves
only A Br, i.e., Euler-Maruyama methods. Therefore, the original SDE in (4) is approximated by (0 k(Z, 2)). (12)
dx. (u(4, X,) + (,) u, 2) + (4,, (I) Z)dW,, This is obtained by conversions, dt x At and dW, = AW, as we expand the diffusion term
(t,, X,)AB = ot, X.) u, Z At + u(t; Z)/Ataw,) = a X1, (t u, Z) + (t. (t) Z)AW, at, X1 (t; u Z)dt + ot, X.)s(t; Z)dW.
where v(t: u. 2) = mt: u, Z), and <(t; 2) vt: A). The function <(t: Z) induced from the approximation of B; is now absorbed into the diffusion function while v/t: u, Z) together with the diffusion function now is added to the drift function. Note that (tt u, Z) and <(t: Z) is conditioned on the inducing point u. Therefore, at the initial time / = O, we need to precompute u first.
Although (12) and (4) seem similar in terms of the expressiveness of drift functions and diffusion functions, the drift function of (12) contains a randomness stemming from the sampling in (9) while that of (4) deterministic. This can be understood that the dependency between AB, is translated into the drift functions. We can elucidate the difference via Algorithm and Algorithm 2 in Appendix.
Note that vI;; u, Z) and st;; Z) are associated with two types of discretizations: that of approximating Stieltjes integral and that of SDE numerical solver.
One of the key advantages of this new SDE is that it can inherit all theory and practices of well- established study of stochastic analysis for Brownian motion Wiener process. The preliminary results of Girsanov's 's theorem, i.e. [34] cannot used in latent SDE [54]. Itô's rules for fBm and mBm can be found in [52] but hard to use in practice. On the other hand, approach can work for any B, defined in (1).
Another benefit of our formulation is that the sampling procedure is simple, requiring only one & at time and avoids multiplication of O(N) memory like [65]. This becomes significant as the hhimiiz-then-discretiie method requires constructing the same sample paths of B, in forward SDEs and backward SDEs.
Computational analysis The evaluation of diffusion functions in (12) requires computing aggre- gate sum over discretization that scales linearly with the number of discretization points. may cause some overhead when computing Itô correction term during solving backward SDE [54]. Therefore, Stratonovich SDEs are more efficient than Itô SDEs. One limitation of integral approximation is that for long interval [0, T'. one may need more discretized partitions to achieve desired approximation accuracy.
On the computational cost of sparse GPs, we only need to perform the Cholesky decomposition at the initial time of every SDE integration. This takes O(M3) with M the number of inducing points. At a time / during solving SDEs, with the pre-computed T.z, sampling Bi has ONN + M²) time complexity as we perform matrix multiplications in (3).

---

Limitation Currently, our model only considers fixed step sizes in SDE numerical solvers. Most theoretical studies in stochastic analysis work for H 1/4 (H 1/s in this study) to make sense of stochastic integrals or SDE solutions [70]. The Fuler-Maruyama update in our approach may not be sufficiently good when the Hurst exponent is less than 1/2 as it pointed out in [16]. Higher-order solvers can be more accurate, requiring handing double integrals, i.e., J dBBdB,, which we wish to address in future work. Although this paper presents an initial theoretical justification of convergence to solutions, the precise order of convergence is not clear yet. Therefore, for practical use, we recommend avoiding using Hurst exponents with small values in our method.
4.5 Convergence analysis
The error of our approximation is composed of three main factors. The first is of SDE solvers. Due to the assumption of fixed step sizes, only a few solvers, i.e., with 1/2 strong convergence. Secondly, the discretization in Stieltjes approximations attributing to the total error leads to the convergence in L? The convergence of such approximations are known. The approximation of noise from sparse GPs is unconventional in solving SDEs because the approximation is in distributional sense rather than point-wise or path-wise. We will show the convergence using rough path theory [56, 26] and sparse GPs bound from [8] for constant Hurst. H E (423,
are applicable
We follow the notation in [39, 56, 26]. Let us denote the canonical lift of B as B = (B,B). Let B be the path corresponding to the sampling procedure in (9) and (10). Let X be the solution of dx, ft,, X, )dB and X be the solution of dX f(t, Xd)d where f [0, T] x R" R" is bounded function that is four-time continuously differentiable. We assume that the initial state follows distribution Po which is is continuous and strietly positive in R". Let denote the probability density distribution of the solution X at time / by pt (x); and that of the approximated solution X by (z))
Theorem 1 (Convergence of solutions given sparse GP approximation). With the Hurst exponent of B, H E (2/s, 1/2), assume that EE( Be+r) S L/T l/e for any E [0, T.. and E (1/2H, 8/2]. Then, for every a E (1/3, 1/20). there exists a constant C such that
KL(Pellet) S C' sup tE[O,T]
(L.
(13)
Here, 0 E (0, 1/2 - (0))) and KL(-II-) is the Beclbackk-eiblee divergence.
This theorem says that if the distribution of B converges to that of in terms of the KL divergence. their corresponding solutions converge in the same sense. Note that we make an assumption that B[(B. - 32+ < L|T|"/e, 2 > 1/2H which means that B is less "rough" than B. This is reasonable assumption since B suffers information loss from sparse GP approximations.
Our proof is done by (I) assessing the p-variation of B to bound the distance in rough path space between the canonical lifts of B and B via [26, Corollary 10.6]; and (2) using the local Lipschitz property of Itô-Lyons maps [26, Theorem 8.5) as we follow the technique in [39]. The complete proof is provided in Appendix B along with necessary backgrounds and discussion.
According to [8], we can quantify the convergence of B to B as its KL. divergence can be bounded, (L( B) < C Ci A₁. Here A, are eigenvalues in iigervalue-eigeefunction pairs (Aşs (.).)) of Mercer's 's theorem [62], k(x,
Ei= (=)r(r));=z')) Recall M is the number of inducing points. Therefore, the convergence of KL(BIIB) depends on the decaying rate of eigenvalues and the number of inducing points. We empirically show the fast decaying rate of eigenvalues for finite samples which also can be found in Appendix B.
4.6
Parameterizing white noise
The previous section concerns how to learn white noises in neural SDEs without providing concrete forms of white noise. This section will show how it affects SDEs in general and discuss the parameterization form of the Hurst exponent.
When the Hurst exponent htt) is constant, say h(t) = H. There is a qualitative difference between settings of H. Figure shows how the Hurst parameter affects the model fit trained by Latent SDE. With a smaller value (H- = 0.3), sample paths are rough, producing more irregularity. When H = 0.8, the model has smoother sample paths, and is more conservative as predictions depend on the past. This parameter has great influence on the uncertainty quantification of the models.
7

---

fBm paths
posterior
fBm paths
posterior
(b) H = 0.8 Figure 2: Latent SDE with fBm for two settings: H 0.3 and H = 0.8.
(a) H = 0.3
For the case that h(t) is function of any function approximators work in general to model h(t; 0). In our implementation, we use small neural networks. With the new representation (12), train the parameters of httt; 0) together with the parameters of drift and diffusion. A more general approach is that f(t s) can accept either neural networks or kernel-based functions when we do not consider normalization. However, their interpretation is not clear as that of mBm.
5
Experimental Results
This section will first justify how close the samples generated by our method are to true ones. We then investigate the ability to learn the Hurst function for synthetic data and test our model for real-world data. Finally, we examine the role of Hurst exponents in score-based generative models. All experiments are implemented using PyTorch [68] with the library torchsde [54, 47). Our source code available at this repository.
How to estimate Hurst from data? We follow [27] to empirically estimate Hurst exponent from data. The key technique based on the correlation of increment of B, and estimates a ratio over sliding windows which is an estimation of Hurst exponent.
5.1 Quality of approximate samples
We would like to investigate our approximation of B; can pro- duce the sample path of B, that is close enough to those of the true process. The true Hurst function is set as h(t) 0.35 + 0h5t.The samples from [65, 25] are used as a baseline. This experiment runs 1000 samples. The estimations of the Hurst exponent for two sampling techniques are Figure 3. The Hurst estimation corresponding our samples is close to the true Hurst function. tends to overestimate the long-range dependency (h(t) > 1/2) and underestimate the irregularity (h(t) < 1/2) in the samples.
1.0
05
Figure 3: Our estimation vs true Hurst function.
5.2 Recovering Hurst function
Next, we test our model with a synthetic data. Here the data is generated from SDE with a known solu- tion:
d.X, aXxdt + Bd.., Xo zo, t [0, where B. is mBm given fixed h(()) 0.3 + 0.5sigmoid(75 - t)). According to Itô integrals for mBm cases [52], the exact solution is Xx To exp(at - 32 + 3B:). We generate 100 data points for each sample path and obtain up to 5 sample paths. After training our model, we observe that the obtained Hurst function can capture the dy- namic similar to the true Hurst. As we increase the number of sample paths (from to 5), our learned Hurst tends to approach the true Hurst. Figure 4 shows given 5 sample paths. We also provide the results for different Hurst functions and an ablation study in Appendix C.3.
X13
(b)
Figure 4: Synthetic data. (a): Comparison between found Hurst true Hurst. (b): The posterior plot of our model which can describe correctly uncertainty regions when the noise is more irregular.
example of our learned Hurst function

---

2-1
13
(a) AMZN
(b) AAPI.
(c) GOOGL
(c) MSFT
Figure 5: Found Hurst functions in financial data.
5.3 Financial data
In this experiment, we compare the Hurst function found by our model with the time-dependent Hurst estimation computed by [27].
Data We collect two months (from September 26th, 2021 to October 23rd, 2021) of hourly data from four stock indices Amazon (AMZN), Apple (APPL), Google (GOOGL). Microsoft (MSFT). The data is retrieved from [85]. It is then standardized and split into the first 80% time points train and the remaining 20% to test.
Obtaining Hurst function Figure 5 compares our found Hurst functions and estimated Hurst computed from [27]. Our obtained Hurst functions have bigger values than that of [27]. This resembles the observation in the previous experiment. A common observation among these stocks is that Hurst functions made some changes (bump up or drop) around the end of October. This agrees with many analyses, for example, Amazon missed revenue expectations for the third quarter or Apple revealed new products. The posterior plots for our model in each stock index are placed in Appendix C.4.
Quantitative comparison We compare our model to two main baselines: Latent SDE with standard Brownian motion [54] and Latent ODE [12]. We report three main comparison criteria: training negative log- lig-liiel (training NLL), test negative big-likelihood (test NLL) and test root mean square error (test RMSE). Overall, our model fit training data well others in terms of training NLLs (see Appendix C.4). This is because our model can capture correctly noise part while Latent SDE relies less flexible Brownian noise. Still, it not clear to identify which model performs better in predictive tasks. However, our model produces more consistent results with less variance.
model outperforms
5.4 Effect of Hurst exponent in score-based generative models
This experiment explores different aspect by studying the effect of Hurst exponent in training score-based generative models. Recent work [77] on generative models based on score functions and SDEs presents impressive results par with generative adversarial networks (GAN). We consider the case where the perturbation data under a new SDE, d.X, =))(()) + /3(I)dB, where
B, = fo ft, s)dW.. We derive the equivalent SDE (discussed more in Appendix C.5):
dx, = 3tt) + B(t))((ttu Z) dt /3()s((t; Z)dW, u u~ (O, Z)).
(14)
Varying Hurst exponents investigate how different types of noise affect learning and sampling
in this model. We consider four scenarios of noise: (1) more anti-persistent as H 0.3; (2) long- range dependent with H = 0.8; (3) linearly increasing (t);; (4) linearly decreasing l((t). We only consider the task of generating MNIST images [53]. Figure 6 shows example of generated images. We observed that H > 1/2 is not favorable, and requires SDEs with long terminal time T. To quantify the quality of generated images, compute the negative log- likeliilddooo (NLL) of set of out-of-sample images like in [76].
Discussion Although we do not present way to find optimal htt), we show that there is setting in which the NLL appears to be better (see Figure a). That is, the NLL of decreasing Hurst setting 3.79 (bits/dim) compared to 4.97 (bits/dim) of Brownian motion (H 0.5). There exists work [50] showing that signal- signal-to-noise to-noise ratio (SNR) the key to establishing an optimal strategy how to inject noises. There is another work studying the best strategy for reverse variance [2]. In contrast to many diffusion models, our approach has an additional term / /B([j((); u) which is unrelated to X, but is included in the drift of the backward SDEs when generating data (see Figure b). Moreover, our approach can suggest way control the sample path of B, in the way that we condition on inducing

---

8
9
2
4
3
7
8
5
7
3
6
4 4 2
3
4
6
8
8
8 7
8 2 8
0
D
7
9
6
0
1 2
0
8
5
8 0
6 2
6
8
5
3
5
6
6
3
9
4
7
4
5
7
8
6 0
6
0
3
5
8
04 2 S 7
2
2
/
2
6
6
4 4
4
2
6
f
2
9
6
7
7 S
9
4
2
5 6
a
5
2
H 0.3
= 0.8
increasing
decreasing
Figure 6: Generated images from different configuration of Hurst.
U.
Incrowcing
-0005
decrrssing
NLL (bits
10-
0.5
+0.1
+0.15
25000
30000
73000
000000
125000
0.00
0.25
0.50
0.75
100
Step
(a)
(b)
Figure 7: (a): Negative log likelihoods small batch of test set. (b): Additional drift (B(t)u(t;u,Z) U,
points u since B; sampled from the conditional distribution u(B] For example, we can follow the variational inference of [37] to find the posterior q(u) for a designed objective, then sample B given (u)) instead of p(u).
Conclusion
In conclusion, this work presents a new class of neural SDEs with generalization of white noise. We propose efficient approximate sampling approach for the noises that neural SDEs can be converted into convenient forms. The techniques used this paper are relatively simple, and related to Gaussian process (GP) research. Our experiments have verified the ability of our models to learn noises from synthetic data to financial time-series as well as showcase an image generation task. As mentioned earlier, current work be improved with higher-order solvers where the double integrals require to model precisely. Prior work [52. 70] is a good starting point for such an approach. Another promising direction is to extend neural controlled differential equations with driving signals which are general GPs instead of Brownian motions. The closure property of GPs under differentiation will be useful to develop this model.
Acknowledgements
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD190031RD) and was partly supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No 10w..2022-----0 0000004400000-0000000000 Artificial Intelligence, Explainability. Personalization, Plug and Play, Universal Explanation Platform), (No. 2019-0-00075, Artificial Intelligence Graduate School Pro- gram (KAIST)), (No. 2022-0-00184 Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics), and KAIST-NAVER Hypercreative Al Center. In addition, this research was supported, in part, by the DARPA GARD award HR001 12020004, NSF CAREER award IIS- 194351,, and NSF BIGDATA award IIS- 1IS-1838139 (to TNT). We would like to thank anonymous reviewers for insightful feedback.
10

---

References
[1] Cedric Archambeau, Dan Corffrrd Manfred Opper, and John Shawe-Taylor. Gaussian pro- cess approximations of stochastic differential equations. In Gaussian Processes in Practice, Proceedings of Machine Learning Research, 2007.
[2] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
[3] Matthias Bauer, Mark van der Wilk, and Carl Edward Rasmussen. Understanding probabilistic sparse Gaussian process approximations. In NeurIPS, 2016.
[4] Vikranth R. Bejjanki, Rava Azeredo da Silveira, Jonathan D. Cohen, and Nicholas B. Turk- Browne. Noise correlations in the human brain and their impact on pattern classification. PLOS Compuiational Biology, 2017.
[5] Robert Brown. A brief account of microscopical observations made in the months of June, July and August 1827, on the particles contained in the pollen plants; and on the general existence of active molecules in organic and inorganic bodies. The Philosophical Magazine, 1828.
[6] Thang D. Bui, Josiah Yan, and Richard E. Turner. unifying framework for Gaussian process pseudo-point approximations using power expectation propagation. J. Mach. Learn. Res., 18, 2017.
[7] E. T. Bullmore, Chris Long, John Suckling, Jalal M. Fadili, Gemma Calvert, Fernando Zelaya, T.Adrian Carpenter, and Mick Brammer. Colored noise and computational inference in neuro- physiological (fMRI) time series analysis: Resampling methods in time and wavelet domains. Human Brain Mapping, 2001.
[8] David Burt, Carl Edward Rasmussen, and Mark Van Der Wilk. Rates of convergence for sparse variational Gaussian process regression. In ICML, 2019.
[9] Ben Chamberlain, James Rowbottom, Maria Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In ICMI., 2021.
[10] Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Learning neural event functions for ordinary differential equations. ICLR, 2021.
[11] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. NeurIPS, 2018
[12] Ricky T. Q. Chen, Yulia Rubanova, and David Duvenaud. Latent ODEs for treguulaly-sampplee time series. In NeurIPS, 2019.
[13] Fabienne Comte and Eric Renault. Long memory in continuous-time stochastic volatility models. Mathematical Finance, 1998.
[14] Sylvain Corlay, Joachim Lebovits, and Jacques Lévy Véhel. Multifractional Stochastic volatility models, Mathematical Finance, 2014.
[15] Donny Danudirdjo and Akira Hirose. Synthesis of two-dimensional fractional Brownian motion via circulant embedding. In 2011.
[16] A. M. Davie. Differential equations driven by rough paths: approach via discrete approxima- tion. Applied Mathematics Research eXpress, 2008
[17] Carmine Di Rienzo, Vincenzo Piazza, Enrico Gratton, Fabio Beltram, and Francesco Car- darelli. Probing short-range protein Brownian motion in the eytoplasm of living cells. Nature Communications, 2014.
[18] Ton Dieker. Simulation of fractional Brownian motion. PhD thesis, 2004.
[19] Matthew Dowling, Piotr Sokół, and Il Memming Park. Hida-matér kernel, 2021.
[20] J. Andrew Doyle and Alan C. Evans. What colour is neural noise?, 2018.

---

[21] Jianzhun Du, Joseph Futoma, and Finale Doshi- Velez. Model-basee reinforcement learning for semi-markov decision processes with neural ODEs, 2020.
[22] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural ODEs, 2019.
[23] A. Einstein. Über die von der molekularkinetischen theorie der wärme geforderte bewegung von in ruhenden flüssigkeiten suspendierten teilchen. Annalen der Physik, 1905.
[24] Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ODE: the world of jacobian and kinetic regularization, 2020.
[25] Christopher Flynn. stochastic. https://githhnn com/ werflynnssstccaatttttt 2020.
[26] Peter K Friz and Martin Hairer. A Course on Rough Paths With an Introduction Regularity Structures Second Edition. Springer, 2014.
[27] Matthieu Garcin. Estimation of tieedeeendentt hurst exponents with variational smoothing and application to forecasting foreign exchange rates. Physica A: Statistical Mechanics and its Applications, 2016.
[28] Jim Gatheral, Thibault Jaisson, and Mathieu Rosenbaum. Volatility is rough. Quantitative Finance, 2018.
[29] Ido Golding and Edward C. Cox. Physical nature of bacterial cytoplasm. Phys. Rev. Lett., 2006.
[30] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Fanley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
[31] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. ICLR, 2019.
[32] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2021.
[33] M Hairer. Introduction to malliavin calculus, 2021.
[34] Fabian Harang, Torstein Nilssen, and Frank Proske. Girsanov theorem for multifractional Brownian processes, 2018
[35] Y. He, S. Burov, R. Metzler, and E. Barkai. Random time-scale invariant diffusion and transport coefficients. Phys. Rev. Lett., 2008.
[36] Markus Heinonen, Cagatay Yildiz, Henrik Mannerström, Jukka Intosalmi, and Harri Lähdesmäki. Learning unknown ODE models with Gaussian processes. In ICML, 2018.
[37] James Hensman, Nicolò Fusi, and Neil D. Lawrence. Gaussian processes for big data. In UAI, 2013.
[38] Hida and Si Si. Lectures on White Noise Functionals. World Scientific, 2008
[39] Liam Hodgkinson, Chris van der Heide, Fred Roosta, and Michael W. Mahoney, Stochastic continuous normalizing flows: training SDEs ODEs. In Proceedings of the Thirty-evvntth Conference on Uncertainty in Artificial Intelligence, 2021.
[40] John C. Hull. Options, futures, and other derivatives. Pearson Prentice Hall, 2006.
[41] Felix Höfling and Thomas Franosch. Anomalous transport in the crowded world of biological cells. Reports on Progress Physics, 2013.
[42] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. CoRR, abs/1800.0055 076606 2018.
[43] Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud. Learning differential equations that are easy to solve. In NeurlPS, 2020.
2020.
12

---

[44] Patrick Kidger. On neural differential equations. CoRR, 2022.
[45] Patrick Kidger, Ricky T. Q. Chen, and Terry Lyons. "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms. ICML, 2021.
[46] Patrick Kidger, James Foster, Xuechen Li, and Terry Lyons. Efficient and accurate gradients for neural SDEs. In NeurlPS, 2021.
[47] Patrick Kidger, James Foster, Xuechen Li. Harald Oberhauser, and Terry Lyons Neural SDEs as Mhiiite-Dimensinn
GANs. ICML, 2021.
[48] Patrick Kidger and Terry Lyons. Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU. In ICLR, 2021.
[49] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural Controlled Differential Equations for Irregular Time Series. NeurlPS, 2020.
[50] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffusion models. In NeurIPS, 2021.
[51] Peter E. Kloeden and Eekhard Platen. Numerical Solution of Stochastic Differential Equations. Springer, 1992.
[52] Joachim Lebovits and Jacques Lévy Véhel. White noise-based stochastic calculus with respect to multifractional Brownian motion. Stochastics: An International Journal of Probability and Stochastic Processes, 2014.
[53] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. hwwpllabbeehttp///yaann. lecuu.ccm//xxl//nrisss.. 2010.
[54] Xuechen Li, Ting-Kan Kam Leonard Wong. Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In AISTATS, 2020.
[55] Andreas Look, Chen Qiu, Maja Rudolph, Jan Peters, and Melih Kandemir. Deterministic inference neural stochastic differential equations. CoRR, abs/2006.08973. 2020.
[56] Terry J. Lyons. Differential equations driven by rough signals. Revista Matemática Iberoameri- cana, 1998.
[57] Miguel Lázaro-Gree Gredilla and Michalis K. Titsias. Variational heteroscedastic Gaussian process regression. In ICML, 2011.
[58] Benoit B. Mandelbrot and John W. Van Ness. Fractional Brownian motions, fractional noises and applications. SIAM Review, 1968.
[59] Stefano Massaroli, Michael Poli, Federico Califano, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Optimal energy shaping via neural approximators, 2021.
[60] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dis- secting neural ODEs, 2021.
[61] Stefano Massaroli, Michael Poli, Sho Sonoda, Taji Suzuki, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Differentiable multiple shooting layers, 2021.
[62] James Mercer. Functions of positive and negative type, and their connection the theory of integral equations. Phil. Trans. R. Soc. Lond. A, 1909.
[63] James Morrill, Adeline Fermanian, Patrick Kidger, and Terry Lyons. A generalised signature method for multivariate time series feature extraction, 2021.
[64] James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neural rough differential equations for long time series. ICML, 2021.
[65] V. Muniandy and S. C. Lim. Modeling of locally self-similar processes using multifractional Brownian motion of tiemann-liouville type. Phys. Rev. E, 2001.
13

---

[66] Avik Pal, Yingbo Ma, Viral Shah, and Christopher Rackauckas. Opening the blackbox: Accel- erating neural differential equations by regularizing internal solver heuristics, 2021.
[67]
Giorgio Parisi. Brownian motion. Nature, 2005.
[68]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Gamett, editors, NewrIPS, 2019.
[69] Romain-FFrançis Peltier and Jacques Lévy Véhel. Multifractional Brownian Motion Definition and Preliminary Results. Research report, INRIA, 1995.
[70] Zhongmin Qian and Xingcheng Xu. Itō integrals for fractional Brownian motion and applica- tions to option pricing, 2018.
[71] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
[72] D. Revuz and M. Yor. Continuous Marlingales and Browniun Motion. Springer Berlin Heidelberg, 2004.
[73] Simo Sarkka, Arno Solin, and Jouni Hartikainen. Spatiotemporal learning via infinite- dimensional bayesian filtering and smoothing: look at Gaussian process regression through Kalman filtering. IEEE Signal Processing Magazine, 2013.
[74] Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using aseudo-inputs, In NeurIPS, 2006.
[75] Arno Solin, Blla Tamir, and Prakhar Verma. Scalable inference in sdes by direct matching of the -oi--Pllanckkoooooono equation. In NeurIPS, 2021.
[76] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In NeurIPS, 2021.
[77] Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[78] Fallaw Sowell. Modeling long-run behavior with the fractional arima model. Journal of Monetary Economics, 1992.
[79] Simo Särkkä. Bayesian Filtering and Smoothing. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2013.
[80] Simo Särkkä and Aroo Solin. Applied Stochastic Differential Equations. Institute of Mathemat- ical Statistics Textbooks. Cambridge University Press, 2019.
[81] M. Talagrand. Transportation cost for Gaussian and other product measures. Geometric and functional analysis, 1996.
[82] Ruey S. Tsay. Conditional heteroscedastic time series models. Journal of the American Statistical Association, 1987.
[83] Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent Gaussian models in the diffusion limit. CoRR, 2019.
[84] Belinda Tzen and Maxim Raginsky. Theoretical guarantees for sampling and inference in generative models with latent diffusions. In COLT, 2019.
[85] Alpha Vantage. Alpha vantage, 2022.
14

---

[86] Christopher K. L. Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. In NeurIPS, 2001.
[87] W. Willinger, M.S. Taqqu, R. Sherman, and D. Wilson. Self-similarity through viriabbariability: statistical analysis of ethernet lan traffic at the source level. IEEE/ACM Transactions Networking, 1997.
[88] Cagatay Yildiz, Markus Heinonen, Jukka Intosalmi, Henrik Mannerström, and Harri Lähdesmäki. Learning stochastic differential equations with Gaussian processes without gradi- ent matching, 2018.
[89] Linfeng Zhang, Weinan E, and Lei Wang. Monée-Ampère flow for generative modeling, 2018.
[90] Juntang Zhuang, Nicha C Dvornek, Sekhar Tatikonda, and James Duncan. MALI: A memory efficient and reverse accurate integrator for neural ODEs. ICI.R 2021.
[91] Çagatay Yıldiz, Markus Heinonen, and Harri Lähdesmäki. ODE*VAE: Deep generative second order odes with Bayesian neural networks. In NewrIPS, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the last paragraph of Section 4.4
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Mentioned in the main text and more detailed in Appendix B
(b) Did you include complete proofs of all theoretical results? [Yes] Complete proof can be found in Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] MNIST dataset for image generation tasks.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
15

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "pp3jebMRmz", "reviewer": "Reviewer_Kikm", "summary": "The paper proposes an approach to neural SDEs using an alternative to standard Brownian noise. The authors propose a particular definition of alternative noise. They develop some theoretical results which allow standard neural SDE methods to be used to fit an approximation to their stochastic process. Some convergence results are also proved. Finally the method is applied to various experiments.", "questions": "Can you address the points listed under \"weaknesses\"?", "limitations": "The authors include a section on technical limitations. As this is a paper on foundational machine learning, extending existing methods, there do no not seem to be any direct negative societal impacts to discuss.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Note**: My original score based on this review was 3 (Reject), but I've increased this to 4 (Borderline reject) following the rebuttal.\n\n## Overview\n\n* Originality: good. This seems like an interesting approach to investigate, and is novel to me, although I am not an expert in this area.\n* Quality: poor. The experiments are not convincing (see below for details).\n* Clarity: poor. I found the theoretical material unclear (see below for details).\n* Significance: poor. In my opinion the paper did not make a strong case for the benefits of this approach.\n\n## Strengths\n\nThe paper develops some non-trivial theoretical results.\n\n## Weaknesses - overview\n\nI found two main weaknesses in the paper.\n\n1) The theoretical material is hard to follow, and seems to involves a lot of non-rigorous derivations.\n2) The experiments do not demonstrate good performance of the proposed method.\n\nI have less confidence about weakness 1) as I am not an expert in the subject area. It may be that the material would be more convincing to an expert. However in my opinion weakness 2) alone is a major limitation of the paper.\n\n## Weaknesses - theory\n\n* Equation (5) is an approximation, but is then treated as exact to derive results throughout the rest of Section 4 e.g. equation (7). Rigorous proofs should also be given.\n* Line 178 says \"Note that directly modeling $\\Delta B_t$ in (6) is numerically unstable when computing (3.3)...\" Is this a typo? There is no equation (3.3). This makes the material hard to follow.\n* Section 4.4 says the original SDE (4) is \"equivalent\" to (12). But it seems that this result relies on many approximations e.g. the inducing points scheme. The authors should say that (12) is an approximation or prove that it is equivalent.\n* Why does Theorem 1 treat the Hurst exponent as a constant, when earlier it was defined as a function?\n* The authors do not give a summary of their proposed method. This makes it hard for others to implement.\n\n## Weaknesses - experiments\n\n* In Section 5.1 the baseline seems as good as the proposed method. It's also unclear to me where the exact details of the two methods can be found.\n* In Section 5.2 the proposed method does not seem to give a good fit of the true function. Also, no baseline comparison is included.\n* In Section 5.3, the proposed method only does well on training evaluation, and does poorly in test evaluation. This suggests it is overfitting.\n* In Section 5.4, the authors write \"Although we do not present a way to find optimal h(t), we show that there is a setting in which the NLL appears to be better\". This is an interesting exploratory finding. However, given the extra cost of trying several scenarios to get this improvement, it's not clear that the proposed approach is superior to simply running more training iterations of the standard approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oIxQADLcSG", "reviewer": "Reviewer_1vjS", "summary": "Neural SDEs are generalised so as to admit a certain class of non-Brownian drivers. This is done by aligning a Riemann--Stieltjes discretisation of the non-Brownian driver with an Euler--Maruyama discretisation to a separate SDE, so as to reduce to the case of a neural SDE with random coefficients; this randomness is specifically a dependence on a Gaussian process. Scalable sampling of this Gaussian process is performed via sparse inducing points.", "questions": "1. In personal discussions around the possibility of \"neural fractional SDEs\", the general conclusion has been that it probably wasn't worth the effort -- because similar behaviour could probably be mimicked simply by taking sufficient complicated drifts and diffusions, and by working in a latent space.\n\n   (As this hidden Markov structure implies the possibility of a non-Markov structure to its output -- which might be able to approximate the autocorrelation of a fractional SDE. And indeed latent spaces are standard when working with neural SDEs anyway.)\n\n   As such I would really have expected to see experiments comparing the proposed approach against a \"complicated\" neural SDE with Brownian driver. This paper is technically impressive but the corollary to that is that it is also technically impenetrable -- I believe for this paper to see impact it will need to demonstrate a substantial improvement against this much-simpler-to-implement alternative.\n\n   (Which I note is already not that simple to implement itself!)\n\n   To my mind this is the single greatest limitation of this work, and it is a substantial limitation.\n\n2. I would emphasise the need to improve several aspects of the presentation, as detailed in the main review.\n\n3. The Remark of line 198: it is mentioned that working with the fBM kernel directly does not work, as it is not differentiable. Is this an issue because of the desire to backpropagate through it, so as to train the Hurst parameter? (Or is it something else?) Either way I think it worth including the rationale here.\n4. The Hurst exponents obtained from synthetic data are moderately encouraging, but not that convincing. Can this experiment be tried with much larger amounts of data, and small step sizes, so that the true Hurst exponent really is recovered -- to verify the efficacy of the proposed methodology? The failure to work convincingly on relatively simple univariate data is unfortunate.\n\nOverall I would emphasise that I nonetheless enjoyed this paper.\n\nUnrelated to my review score -- what follows are some questions/suggestions just for my own curiosity.\n\n1. This paper uses inducing points for sampling from Gaussian processes. I have not previously seen this applied in the context of non-Brownian drivers for SDEs -- is this particular combination new here?\n\n2. The problem of simulating fractional Brownian motion is classical, and e.g. fractional volatility models are moderately popular. It seems to me that the approach proposed in this paper, of using sparse inducing points, should apply equally well to fBM (which is also a Gaussian process). What then happens? I assume this has been tried or otherwise fails for some reason.\n\n3. Do the authors believe it would be possible/reasonable to e.g. modify a Heston model to use the alternate driver of (1), and then apply their efficient sampling scheme? This may well have different properties to the rough Heston model currently considered in the literature, so to what extent this is a reasonable model is unclear to me.\n\n4. It is part of the folklore (and not really written anywhere in the literature, so far as I know) that latent SDEs consistently underestimate the variance. I haven't dug into the reason for this, but if the authors continue using latent SDES in future work then they wish to attempt to address this issue alongside. (e.g. by additionally introducing a discriminator and simultaneously training the model as an SDE-GAN -- I speculate).\n\n5. The authors note that their analysis is restricted to the case of Euler--Maruyama with fixed step size, and that in future work they would like to consider (a) better convergence results; (b) GP drivers for NCDEs; (c) higher-order SDE solvers using Levy area. For future work, I would also like to suggest that handling the case of adaptive timestepping would also be particularly useful/interesting.\n\n   (I am aware that adaptive SDEs are not a well-understood topic, with a reasonable number of errors in the published literature. For what it is worth I recommend the references listed in [0] as the best starting point for adaptive SDE timestepping. My own understanding is that general-case adaptivity -- i.e. arbitrary stepsize changes, rather than just halving/doubling -- currently requires the SDE to satisfy the commutativity condition.)\n\n   [0] https://docs.kidger.site/diffrax/api/stepsize_controller/#diffrax.PIDController", "limitations": "This work has no ethical concerns.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength: technical approach**\n\nThis paper is technically very strong. \"Neural fractional SDEs\" have been mooted for some time, with no real work in large part due to the non-scalability of sampling fractional Brownian motion (fBM). This paper overcomes this issue through a combination of (a) switching from fBM to another related process (due to Levy); (b) drawing on the theory of sparse inducing points; (c) demonstrating convergence via rough path theory; (d) carefully linking up all the pieces in specifically discrete time.\n\nThis is an excellent tour-de-force requiring expertise across multiple disciplines (GPs; SDEs; numerical methods; rough path theory; deep learning.). To the best of my knowledge this approach is novel.\n\n**Strength: attention to computational cost etc.**\n\nThis paper does a good job discussing the varying computational costs of each procedure (e.g. $O(N)$ for sparse inducing points; $O(N^3)$ for Cholesky decompositions, etc.) This is of course important in practice. \n\nThis is true even at the level of subtle technical details, such as the computational benefit of Stratonovich integration over Ito integration when backpropagating via optimise-then-discretise.\n\n**Strength: discussion of limitations**\n\nThe authors are careful to state the limitations of their approach explicitly.\n\n**Weakness: not self contained**\n\nThis paper obviously assumes a sophisticated background in stochastic processes. As such, at present, I do not believe the paper is self-contained: the results used are frequently not stated, and instead only a reference is given.\n\nFor example making clearer what is the relationship between equation (1) and fBM; the definition of mBm (line 88); the relationship between Hurst exponent and Holder continuity (line 100); etc.\n\n(I admit I am hard-pressed to identify what should be cut in order to make space for such explanations, as the paper is already very dense. I suspect it may require some reorganisation -- for example the second paragraph of \"Computational analysis\" (line 271) could be replaced with a few extra words at the end of line 193.)\n\n**Weakness: \"adjoint method\"**\n\nThe term \"adjoint\" has become heavily overloaded. There are multiple ways to backpropagate through a differential equation solve. In the current literature is is often used to mean specifically optimise-then-discretise, but in older literature it was often used to mean specifically discretise-then-optimise. Following [Ref. 37 of this paper, Remark 5.5], I would recommend avoiding the term \"adjoint\" altogether.\n\n(In passing, the authors may be interested in [Ref. 37 of this paper, Section 5.2.3], who prove optimise-then-discretise backpropagation through SDEs via rough path theory.)\n\n**Weakness: presentation**\n\nThe text could doing with being reviewed for typos (some are listed below) and mathematical clarity (as  above). In addition:\n\nEquation (12) is a bit of a lie -- this is constructed for specifically the Euler--Maruyama discretisation, not the continuous limit.\n\nLine 222 could be clarified: state precisely what is the nature of the association.\n\nThat the kernel on line 199 is the kernel of fBM should be stated explicitly. (Unless this is known to the reader it comes out of nowhere.)\n\nI believe the presentation of Theorem 1 could be clarified: what is shown is a general result, that sort-of corresponds to the construction being presented. I believe this distinction should be emphasised a little more carefully, e.g. by turning \"Convergence analysis\" into a new section; or similar.\n\nI am not that familiar with the details of inducing points for Gaussian processes -- at least for me I would have appreciated greater technical precision in Section 3.3.\n\n**Minor issues**\n\nLine 82: typo \"of\".\n\nLine 141: \"on sampling procedure\" should be \"on the sampling procedure\" or similar.\n\nLine 152: missing close bracket.\n\nLine 155: \"require computing Cholesky decomposition\" should be \"which require computing the Cholesky decomposition\"\n\nLine 156: equation (5) is referenced before it is introduced.\n\nLine 137 (and elsewhere): the precise definition of $Z$ is not given.\n\nLine 139: $m$ and $v$ should be notated as functions of $u$. This becomes important in equation (10), in which the need for sampling $B'_Z$ becomes unclear.\n\nLine 227: should presumably state the only preliminary results are available for fBM.\n\nLine 229: only $\\xi_t$ -> only one $\\xi_t$\n\nLine 243: time->times\n\nLine 248: I think this could be simplified to $\\rho \\in (1/(2H), 3/2]$?\n\nLine 286: \"SDEs model\" does not parse.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_Wfw1ykNB0N", "reviewer": "Reviewer_FL98", "summary": "The paper proposes an extension of the Neural SDE model proposed by [Li et al. (AISTATS 2020)](https://proceedings.mlr.press/v108/li20i.html). This more general model is obtained by replacing the Brownian motion driving the NSDE with a Riemann–Liouville fractional Brownian motion (R–L fBM) that has a learnt time-varying Hurst function. To improve the scalability of this approach, the authors employ techniques from sparse Gaussian processes to reduce the computational cost of sampling the R–L fBM. Moreover, their approach of using inducing points allows the resulting model to be expressed as a “standard” Neural SDE, but with random drift and diffusion vector fields. A convergence analysis using rough path theory is presented to support this approximation. The paper concludes with a variety of numerical experiments, showing that the proposed model performs comparably to the existing approach of [Li et al. (2020)](https://proceedings.mlr.press/v108/li20i.html) and can recover Hurst functions reasonably well from noisy data. In addition, the authors show their approach can improve score-based generative modelling with SDEs.", "questions": "Questions\n\n* I believe aspects of Section 4 can be better presented.\n\nFor example, what is the \"true\" value of $k(t,s)$ in equation (8)? By Itô's Isometry, I believe it should be\n$k_{\\text{true}}(t,s) = E[B_t^\\prime B_s^\\prime] = \\frac{1}{(\\Delta t)^2}\\int_0^{\\min(s,t)} f(s, u) f(t, u) du\\hspace{2.5mm}(\\ast)$.\n\nOne can then apply any form of numerical integration to estimate $k_{\\text{true}}(s,t)$, such as trapezoidal rule. Am I correct in thinking that applying the rectangle rule to $(\\ast)$ is equivalent to what is done in the paper?\n\nAs a consequence, I do not believe the discussion on integral approximation in section 4.2 is that important. From what I understand, only $k(s,t)$ is needed to sample noise in the proposed model.\n\n* Why is $B_t^\\prime = \\frac{\\Delta B_t}{\\Delta t}$ considered more numerically stable than the natural normalization  $B_t^\\prime = \\frac{\\Delta B_t}{\\sqrt{\\Delta t}}$?\n\n* I find details on the architectures in the appendix lacking, particularly for the experiments on financial data. How are the vector fields set up? (number of layers, choice of optimizer, etc) Are gradients obtained by optimize-then-discretize or discretize-then-optimize?\n\n* In Theorem 1, does $L$ depend on $\\varrho$? If so, the statement should fix $\\varrho$ first.\n* In the proof of Theorem 1, is $B_t$ a standard fractional Brownian or a Riemann-Liouville fractional Brownian motion?\n\nFor example, regarding the statements \"As $B_t$ is a fractional Brownian motion\" (line 726) and $\\mathbb{E}[(B_t-B_{t+\\tau})^2] = |\\tau|^{2H}$ in the discussion. Generally speaking, I think it would be helpful to always distinguish between fBM and R–L fBM.\n\n\nMinor suggestions\n\n* \"methods and sparse Gaussian process\" → \"methods and sparse Gaussian processes\" (line 10)\n* \"Recent work on NODEs open a new direction\" → \"Recent work on NODEs opens a new direction\" (line 21)\n* \"Extensions to SDEs is proposed to handling\" → \"Extensions of SDEs are proposed for handling\" (line 22)\n* \"has been emphasized in a number of study\" → \"has been emphasized in a number of studies\" (line 26)\n* \"expressiveness from representation powers\" → \"expressiveness from the representation powers\" (line 30)\n* \"during solving SDEs\" → \"during SDE solves\" (line 43)\n* \"using numerical method in solving DEs to perform learning parameters in neural network\" → \"using numerical methods in solving DEs to learn parameters in neural networks\" (lines 57-58)\n* \"There are a number of follow-up work aiming to improvements in different aspects\" → \"There are a number of follow-up works aiming to improve different aspects\" (line 59)\n* \"rough path\" → \"rough paths\" (line 68)\n* \"between Gaussian process and\" → \"between Gaussian processes and\" (line 70)\n* \"a function $t$\" → \"a function of $t$\" (line 89)\n* I am not sure the statement \"If $h(t)$ changes over time, this means all such characteristics happen locally\" can be made rigorous (lines 102-103)\n* \"We analyze the convergence of SDE solutions for this approximation\" → \"We analyze the convergence to SDE solutions for this approximation\" (line 147)\n* \"$W_{s_{i+1}}-W_{s_i}\\sim\\mathcal{N}(0,\\sqrt{\\Delta s_i})$\" → \"$W_{s_{i+1}}-W_{s_i}\\sim\\mathcal{N}(0, \\Delta s_i)$\" (lines 162 and 163)\n* I am not sure if the statement \"The discretization is also inevitable whenever it is required to sample $B_t$\" is true. (lines 166 and 167)\n* \"that of approximated solution\" → \"that of the approximated solution\" (line 265)\n* \"distance on rough path space\" → \"distance in rough path space\" (line 275)\n* \"Recent work on generative model\" → \"Recent work on generative models\" (line 364)\n* \"We consider the case of perturbation in data under a new SDE\" → \"We consider the case where the perturbation in data is under a new SDE\" (line 366)\n* \"this work can improve the current model with\" → \"our current model can be improved with\" (line 393)\n* Algorithms 1 and 2 in the appendix should be referenced in the main body of the paper.\n* \"(1)long-range\" → \"(1) long-range\" (line 668)\n* \"Ito\" → \"Itô\" (lines 676 and 678)\n* \"has the same\" → \"have the same\" (between lines 685 and 686)\n* \"Here, Hurst parameter is\" → \"Here, the Hurst parameter is\" (line 693)\n* \"we provide some backgrounds of\" → \"we provide some background on\" (line 695)\n* \"is Itô-Lyons maps\" → \"is the Itô-Lyons map\" (line between 714 and 715)\n* \"That is, followed\" → \"That is, following\" (line 733)\n* \"i.e., its bijection and\" → \"i.e. its bijective and\" (line 734)\n* \"via canonical lift $\\boldsymbol{B}$ and initial $X_0$\" → \"via a canonical lift $\\boldsymbol{B}$ and initial state $X_0$\" (line 735)\n* \"one can have a local Lipschitz\" → \"we can obtain a local Lipschitz estimate\" (end of line 737)\n* \"Combining with 20\" → \"Combining with (20)\" (line 739)\n* \"Figure 10 and 11\" → \"Figures 10 and 11\" (line 778)\n* \"Figure 10 and 10\" → \"Figures 10 and 11\" (line 781)\n* \"Figure 14 and 13\" → \"Figures 13 and 14\" (line 784)\n* \"Recent work on generative model\" → \"Recent work on generative models\" (line 787)\n* \"while slightly underestimates those with small values\" → \"while slightly overestimates those with small values\"  (figure 10)\n* \"Therefore, we resort a similar form to\" → \"Therefore, we resort to a similar form of\" (line 791, bottom of p22)\n* I would have thought it is \"$\\varsigma(t; Z)$\" and not \"$\\varsigma(t)$\"? (bottom of p22)\n* \"underestimate the Hurst having small value\" → \"overestimates when the true Hurst exponent is small\" (figure 11)\n* \"depepend\" → \"depended\" (line 792)\n* \"phrases\" → \"phases\" (twice on line 795)\n* \"((long-range\" → \"(long-range\" (line 803)\n* \"close to Gaussian distribution\" → \"close to a Gaussian distribution\" (line 807)", "limitations": "The authors have addressed some technical limitations, particularly related to when the noise is rough (i.e. when $H$ is small) – which I think is reasonable. The authors could possibly include a statement along the lines of \"We do not anticipate any significant negative societal impacts as a result of this work\".", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Firstly, I would like to acknowledge the timing of this review (I was asked to be review this paper on 08/08/2022). Whilst I will include my questions/comments, I do not expect the authors to be able to respond to them.\n\nOverall, I am positive about this paper and believe that learning more general noises is a sensible way of trying to improve NSDE models. The authors focus on learning fractional noises, which I believe constitutes a helpful first step in this direction.\n\nStrengths\n\n* I found the methodology presented in the paper understandable and rather interesting. Although I believe that increasing hidden state can allow \"standard\" NSDEs to handle real-world data, it does make sense to also consider driving processes other than Brownian motion.\n\n* Using techniques from sparse Gaussian processes to sample R–L fBM is interesting in its own right. I have not seen this approach used in the fBM literature. In addition, the authors present an error analysis based on rough path theory – which is believable (though I have not thoroughly checked the details).\n\n* The numerical experiments appear to be of a good quality. At the very least, they show that the proposed approach can give comparable performance to that of  [Li et al. (2020)](https://proceedings.mlr.press/v108/li20i.html). I found the application to score-based generative modelling on MNIST and FashionMNIST particularly interesting.\n\n* The paper opens up a variety of future research directions, such as driving NSDEs by GPs or R–L fBMs with more general kernels. I also imagine that using different noises could have nice applications to score-based models.\n\n* The paper is well structured and is good at conveying the key ideas.\n\nWeaknesses\n\n* I find the title a little misleading. Since the methodology imposes a Hurst function on the noise, I feel that \"Learning Fractional Noises in Neural Stochastic Differential Equations\" is more appropriate.\n\n* The paper does not mention the term \"Stochastic Volterra Equations\" (SVE). Whilst the formulation with f(s,t) is more general, the methodology tested in the experiments can be viewed as a Neural SVE. One recent paper from the SVE literature that may be worth citing is [Bayer and Breneis (2021)](https://arxiv.org/abs/2108.05048).\n\n* I believe aspects of Section 4 can be better presented (see questions section).\n\n* It is not clear whether the proposed methodology performs better than the Latent SDE model of [Li et al. (2020)](https://proceedings.mlr.press/v108/li20i.html). However, I find this more than compensated by the interesting experiments on MNIST and FashionMNIST.\n\n* The results on the real-world dataset against  the Latent SDE and ODE models are important and should be displayed in the main body. As an aside, I do not believe root mean square error is a sensible metric for generative models such as Neural SDEs.\n\n* The results on FashionMNIST should be included in the main body of the paper.\n\n* I find details on the architectures in the appendix lacking, particularly for the experiments on financial data. I think it is reasonable to expect such information to be in the appendix (such as in [Li et al (2020)](https://arxiv.org/abs/2001.01328) and [Kidger et al. (2021)](https://arxiv.org/abs/2102.03657))\n\n* There are a number of grammatical errors (see minor suggestions in the questions section)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XwfJjEGwS2w", "reviewer": "Reviewer_75wq", "summary": "This paper presents an alternative class of conventional step-size neural stochastic differential equations (SDE). It provides an approximate sampling method based on sparse Gaussian Processes for the noise integral in order to put the neural SDEs into a computationally accessible form. Experiments have been conducted on the sequential finnancial time-series data and image recognition.", "questions": "The questions are addressed in the weaknesses. Two of them:\n1. Is there any experiments on MNIST based on accuracy can be reported?\n2. In section 4.1 - you said for method covered in [64]. \"the number of these random vartiables prevents optimize-then-discretize methods to scale\". But what is the problem and potential negative impact of this practically? And how your method may solve it?", "limitations": "There are no problem on the societal impact to me for this work. The technical limitations are addressed in the weaknesses.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The explanation of earlier work on neural SDEs is quite solid mathematically, which makes the propsoed method more convenient to understand for readers who have little experience.\n2. The experiments are quite diverged, covering function re-production, financial data analysis and conventional MNIST image recognition.\n3. The GP method itself is easy to understand, as well as the general presentation of the paper.\n\nWeaknesses\n1. Even though in the result section the title mentioned it, the evaluation lacks quantitative comparison, especially for image recognition task, but more in a qualitative way. Some numerical analysis would help.\n2. There are some spots where the author explains the weaknesses of earlier work, but does not connect them to the proposed solution very tightly. e.g. In section 4.1, the approaches from [64] has the problem of un-scalable random variables. But then its problem on current models and relationship with the author's algorithm remain un-reached.\n3. The baseline model compared with is latent SDEs, while stronger models such as GAN based SDE solvers may want to get in. Also, apart from Hurst function, some other approximation experiments might be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7rBfxaObQP3", "reviewer": "Reviewer_FT9y", "summary": "The paper studies a different noise model in SDEs. Namely, the model integrates the Wiener process in time scaling it with the function (that depends on time) called Hurst Exponent. It seems that this formulation is equivalent to the Ito SDE with a different scaling term. However, I don't have a background on the Hurst exponent, and the authors do not discuss such a relation. \n\nThroughout the paper, the authors claim that their main goal is to learn the Hurst exponent from the given data. They propose to sample the noise via the Gaussian process and learn the model via the variational inference as in neural SDEs [1]. Unfortunately, it's hard to describe the proposed method more precisely due to its poor description in the paper.\n\nThe authors provide two types of experiments: the estimation of the Hurst function from the given time series; the generation of the MNIST dataset with different noise models (different Hurst exponents).\n\n1. Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 3870–3882, 2020.\n", "questions": "My questions and suggestions follow naturally from the section \"Strengths And Weaknesses.\"", "limitations": "It's hard to assess the limitations of the approach due to its poor description and motivation.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Unfortunately, the paper does a poor job on all main aspects of a scientific publication: a motivation of the problem, a clear description of the proposed approach, and a rigorous comparison of the proposed approach to the competitors.\n\nFor the main motivation, the authors propose the learning of the Hurst exponent. However, it's not clear why learning the exponent is important in practice. I can assume that this can be used for further simulation of the stochastic process, but it's not clear how methods that do not take the Hurst exponent into account (just learning the scaling term in Ito SDE) compare to the proposed method.\n\nFor the method description, the paper does not describe the optimization problem and parameterization of the model (the authors claim to learn all the terms in Ito SDE). The main development of the paper is the sampling stage (section 4.3), but the authors do not describe how they propagate gradients through these samples that depend on the model. Furthermore, the whole algorithm seems to be independent of the exact form of the scaling term in SDE (whether it's introduced with the exponent or not). Thus, the whole discussion on the Hurst exponent doesn't seem to bring any practical value or modify the algorithm anyhow. In the considered context, the Hurst exponent can be considered simply as another parameterization of the scaling term in Ito SDE.\n\nIn what follows, there is a list of minor clarity issues throughout the paper.\n- line 41. It's not clear w.r.t. what parameter the models cannot be scaled.\n- Related work section rather enumerates a broad field of literature than relates the current paper to the existing literature.\n- There is a broken reference on line 178.\n- The sentence in lines 225-227 is not clear. Maybe there is a typo there.\n\nThe empirical evaluation of the proposed method does not give any evidence of its efficiency. For the Hurst exponent estimation, the authors provide experiments with two types of data. For the synthetic data, the authors report the quality of approximation in Fig 4(a). It's hard to measure the quality of the approximation since the further application is not clear, but I would say that the quality is unsatisfactory. The authors claim that the problem is challenging since they recover it by a single sample, but then I would expect demonstrations of the approach on less challenging problems. For the financial data, the performance is not measured quantitatively at all.\n\nThe image generation with different Hurst exponents is not evaluated properly. Once again, it's not clear why it is not an introduction of a different scaling term. Furthermore, some works study different noise scales [2,3], and no comparison with them is provided.\n\n2. Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. \"Score-based generative modeling through stochastic differential equations.\" arXiv preprint arXiv:2011.13456 (2020).\n3. Kingma, Diederik, Tim Salimans, Ben Poole, and Jonathan Ho. \"Variational diffusion models.\" Advances in neural information processing systems 34 (2021): 21696-21707.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept