Learning Fractional White Noises in Neural Stochastic Differential Equations

Abstract: Differential equations play important roles in modeling complex physical systems. Recent advances present interesting research directions by combining differential equations with neural networks. By including noise, stochastic differential equations (SDEs) allows us to model data with uncertainty and measure imprecision. There are many variants of noises known to exist in many real-world data. For example, previously white noises are idealized and induced by Brownian motions. Nevertheless, there is a lack of machine learning models that can handle such noises. In this paper, we introduce a generalized fractional white noise to existing models and propose an efficient approximation of noise sample paths based on classical integration methods and sparse Gaussian processes. Our experimental results demonstrate that the proposed model can capture noise characteristics such as continuity from various time series data, therefore improving model fittings over existing models. We examine how we can apply our approach to score-based generative models, showing that there exists a case of our generalized noise resulting in a better image generation measure.

Reviews and author responses:

 The paper proposes an extension of the Neural SDE model proposed by [Li et al. (AISTATS 2020)](https://proceedings.mlr.press/v108/li20i.html). This more general model is obtained by replacing the Brownian motion driving the NSDE with a Riemann–Liouville fractional Brownian motion (R–L fBM) that has a learnt time-varying Hurst function. To improve the scalability of this approach, the authors employ techniques from sparse Gaussian processes to reduce the computational cost of sampling the R–L fBM. Moreover, their approach of using inducing points allows the resulting model to be expressed as a “standard” Neural SDE, but with random drift and diffusion vector fields. A convergence analysis using rough path theory is presented to support this approximation. The paper concludes with a variety of numerical experiments, showing that the proposed model performs comparably to the existing approach of [Li et al. (2020)](https://proceedings.mlr.press/v108/li20i.html) and can recover Hurst functions reasonably well from noisy data. In addition, the authors show their approach can improve score-based generative modelling with SDEs. Firstly, I would like to acknowledge the timing of this review (I was asked to be review this paper on 08/08/2022). Whilst I will include my questions/comments, I do not expect the authors to be able to respond to them.

Overall, I am positive about this paper and believe that learning more general noises is a sensible way of trying to improve NSDE models. The authors focus on learning fractional noises, which I believe constitutes a helpful first step in this direction.

Strengths

* I found the methodology presented in the paper understandable and rather interesting. Although I believe that increasing hidden state can allow "standard" NSDEs to handle real-world data, it does make sense to also consider driving processes other than Brownian motion.

* Using techniques from sparse Gaussian processes to sample R–L fBM is interesting in its own right. I have not seen this approach used in the fBM literature. In addition, the authors present an error analysis based on rough path theory – which is believable (though I have not thoroughly checked the details).

* The numerical experiments appear to be of a good quality. At the very least, they show that the proposed approach can give comparable performance to that of  [Li et al. (2020)](https://proceedings.mlr.press/v108/li20i.html). I found the application to score-based generative modelling on MNIST and FashionMNIST particularly interesting.

* The paper opens up a variety of future research directions, such as driving NSDEs by GPs or R–L fBMs with more general kernels. I also imagine that using different noises could have nice applications to score-based models.

* The paper is well structured and is good at conveying the key ideas.

Weaknesses

* I find the title a little misleading. Since the methodology imposes a Hurst function on the noise, I feel that "Learning Fractional Noises in Neural Stochastic Differential Equations" is more appropriate.

* The paper does not mention the term "Stochastic Volterra Equations" (SVE). Whilst the formulation with f(s,t) is more general, the methodology tested in the experiments can be viewed as a Neural SVE. One recent paper from the SVE literature that may be worth citing is [Bayer and Breneis (2021)](https://arxiv.org/abs/2108.05048).

* I believe aspects of Section 4 can be better presented (see questions section).

* It is not clear whether the proposed methodology performs better than the Latent SDE model of [Li et al. (2020)](https://proceedings.mlr.press/v108/li20i.html). However, I find this more than compensated by the interesting experiments on MNIST and FashionMNIST.

* The results on the real-world dataset against  the Latent SDE and ODE models are important and should be displayed in the main body. As an aside, I do not believe root mean square error is a sensible metric for generative models such as Neural SDEs.

* The results on FashionMNIST should be included in the main body of the paper.

* I find details on the architectures in the appendix lacking, particularly for the experiments on financial data. I think it is reasonable to expect such information to be in the appendix (such as in [Li et al (2020)](https://arxiv.org/abs/2001.01328) and [Kidger et al. (2021)](https://arxiv.org/abs/2102.03657))

* There are a number of grammatical errors (see minor suggestions in the questions section) Questions

* I believe aspects of Section 4 can be better presented.

For example, what is the "true" value of $k(t,s)$ in equation (8)? By Itô's Isometry, I believe it should be
$k_{\text{true}}(t,s) = E[B_t^\prime B_s^\prime] = \frac{1}{(\Delta t)^2}\int_0^{\min(s,t)} f(s, u) f(t, u) du\hspace{2.5mm}(\ast)$.

One can then apply any form of numerical integration to estimate $k_{\text{true}}(s,t)$, such as trapezoidal rule. Am I correct in thinking that applying the rectangle rule to $(\ast)$ is equivalent to what is done in the paper?

As a consequence, I do not believe the discussion on integral approximation in section 4.2 is that important. From what I understand, only $k(s,t)$ is needed to sample noise in the proposed model.

* Why is $B_t^\prime = \frac{\Delta B_t}{\Delta t}$ considered more numerically stable than the natural normalization  $B_t^\prime = \frac{\Delta B_t}{\sqrt{\Delta t}}$?

* I find details on the architectures in the appendix lacking, particularly for the experiments on financial data. How are the vector fields set up? (number of layers, choice of optimizer, etc) Are gradients obtained by optimize-then-discretize or discretize-then-optimize?

* In Theorem 1, does $L$ depend on $\varrho$? If so, the statement should fix $\varrho$ first.
* In the proof of Theorem 1, is $B_t$ a standard fractional Brownian or a Riemann-Liouville fractional Brownian motion?

For example, regarding the statements "As $B_t$ is a fractional Brownian motion" (line 726) and $\mathbb{E}[(B_t-B_{t+\tau})^2] = |\tau|^{2H}$ in the discussion. Generally speaking, I think it would be helpful to always distinguish between fBM and R–L fBM.


Minor suggestions

* "methods and sparse Gaussian process" → "methods and sparse Gaussian processes" (line 10)
* "Recent work on NODEs open a new direction" → "Recent work on NODEs opens a new direction" (line 21)
* "Extensions to SDEs is proposed to handling" → "Extensions of SDEs are proposed for handling" (line 22)
* "has been emphasized in a number of study" → "has been emphasized in a number of studies" (line 26)
* "expressiveness from representation powers" → "expressiveness from the representation powers" (line 30)
* "during solving SDEs" → "during SDE solves" (line 43)
* "using numerical method in solving DEs to perform learning parameters in neural network" → "using numerical methods in solving DEs to learn parameters in neural networks" (lines 57-58)
* "There are a number of follow-up work aiming to improvements in different aspects" → "There are a number of follow-up works aiming to improve different aspects" (line 59)
* "rough path" → "rough paths" (line 68)
* "between Gaussian process and" → "between Gaussian processes and" (line 70)
* "a function $t$" → "a function of $t$" (line 89)
* I am not sure the statement "If $h(t)$ changes over time, this means all such characteristics happen locally" can be made rigorous (lines 102-103)
* "We analyze the convergence of SDE solutions for this approximation" → "We analyze the convergence to SDE solutions for this approximation" (line 147)
* "$W_{s_{i+1}}-W_{s_i}\sim\mathcal{N}(0,\sqrt{\Delta s_i})$" → "$W_{s_{i+1}}-W_{s_i}\sim\mathcal{N}(0, \Delta s_i)$" (lines 162 and 163)
* I am not sure if the statement "The discretization is also inevitable whenever it is required to sample $B_t$" is true. (lines 166 and 167)
* "that of approximated solution" → "that of the approximated solution" (line 265)
* "distance on rough path space" → "distance in rough path space" (line 275)
* "Recent work on generative model" → "Recent work on generative models" (line 364)
* "We consider the case of perturbation in data under a new SDE" → "We consider the case where the perturbation in data is under a new SDE" (line 366)
* "this work can improve the current model with" → "our current model can be improved with" (line 393)
* Algorithms 1 and 2 in the appendix should be referenced in the main body of the paper.
* "(1)long-range" → "(1) long-range" (line 668)
* "Ito" → "Itô" (lines 676 and 678)
* "has the same" → "have the same" (between lines 685 and 686)
* "Here, Hurst parameter is" → "Here, the Hurst parameter is" (line 693)
* "we provide some backgrounds of" → "we provide some background on" (line 695)
* "is Itô-Lyons maps" → "is the Itô-Lyons map" (line between 714 and 715)
* "That is, followed" → "That is, following" (line 733)
* "i.e., its bijection and" → "i.e. its bijective and" (line 734)
* "via canonical lift $\boldsymbol{B}$ and initial $X_0$" → "via a canonical lift $\boldsymbol{B}$ and initial state $X_0$" (line 735)
* "one can have a local Lipschitz" → "we can obtain a local Lipschitz estimate" (end of line 737)
* "Combining with 20" → "Combining with (20)" (line 739)
* "Figure 10 and 11" → "Figures 10 and 11" (line 778)
* "Figure 10 and 10" → "Figures 10 and 11" (line 781)
* "Figure 14 and 13" → "Figures 13 and 14" (line 784)
* "Recent work on generative model" → "Recent work on generative models" (line 787)
* "while slightly underestimates those with small values" → "while slightly overestimates those with small values"  (figure 10)
* "Therefore, we resort a similar form to" → "Therefore, we resort to a similar form of" (line 791, bottom of p22)
* I would have thought it is "$\varsigma(t; Z)$" and not "$\varsigma(t)$"? (bottom of p22)
* "underestimate the Hurst having small value" → "overestimates when the true Hurst exponent is small" (figure 11)
* "depepend" → "depended" (line 792)
* "phrases" → "phases" (twice on line 795)
* "((long-range" → "(long-range" (line 803)
* "close to Gaussian distribution" → "close to a Gaussian distribution" (line 807) The authors have addressed some technical limitations, particularly related to when the noise is rough (i.e. when $H$ is small) – which I think is reasonable. The authors could possibly include a statement along the lines of "We do not anticipate any significant negative societal impacts as a result of this work".

 Thank you for your time and effort reviewing the paper. We appreciate the review and comments on the paper. We would like to address the concerns as follows.

### Quantitative evaluation of MNIST
Regarding the quantitative evaluation of image generation tasks, one can measure the performance of the score-based generative model using negative log likelihood [Yang et al. ICLR 2020, Yang et al, NeurIPS 2021] (instead of accuracy). In the paper, we reported the test negative log-likelihood of our model in which one of our settings (decreasing Hurst) gives better performance than the baseline (H=0.5) in MNIST data set (and Fashion MNIST in our recent revision). 

In MNIST dataset, the negative log-likelihood on the test set of our models is $3.79$ versus $4.97$ of the baseline.

In FashionMNIST dataset, ours is $5.44$ versus $5.66$ of the baseline.

### Optimize-the-discretize with existing work
Regarding optimize-then-discretize methods, we would like to elaborate further as follows.

For Ito SDE, the optimize-then-discretize approach requires correcting the drift term based on the diffusion for the backward SDE (Figure 2 of [Li et al. AISTATS 2020] can describe this, Stratonovich SDE does have to be corrected)

$$\bar{\mu}(X_t) = \mu(X_t) - \frac{1}{2}\sigma(X_t) \nabla_x \sigma(X_t)$$

where $\mu$ is the drift and $\sigma: \mathbb{R}^n \to \mathbb{R}^{n \times m}$ is the diffusion in Equation 4 (omit $t$ for simplicity), $n$ is the dimension of state $X_t$, $m$ is the dimension of noise $B_t$

The main factor preventing [64] from scaling is to compute $\sigma(X_t) \nabla_x \sigma(X_t)$. 

In fact, [64] samples mBm $B_t$ via a linear combination of $N$ Gaussian random variables (see Equation 24, Appendix). Recall $N$ is the discretized mesh size. After converting to SDEs with Brownian (introducing $\sqrt{\Delta t}$), the new diffusion function in this case is $\sigma’: \mathbb{R}^n \to \mathbb{R}^{n \times (m N)}$ with a bigger output dimension.

On the other hand, our sampling method requires one Gaussian (see Equation 10), therefore, the diffusion function remains as a map from $\mathbb{R}^n \to \mathbb{R}^{n \times m}$. 
Here matrix multiplication in the case of [64] already takes $N$ times more compared to ours. Furthermore, computing Jacobian with bigger-dimension output is costly (in this case depending on how big the neural work is as well). Efficient implementations like vector-Jacobian products still took unbearable time as we observed during the early stage of implementing our model.

To sum up, in the case of Ito SDE, [64] **increases the dimension of noise** (by the discretized mesh size), causing the burden in computing Ito correction for the optimize-then-discretize approach. We will clarify this in our next revision.

### SDE-GAN
We agree that the comparison with or (incorporating fBm or mBm to) SDE-GAN is a necessary evaluation as it is an alternative toLatent SDE. Due to the technical difficulty (we have limited knowledge about the model at the time being), we would like to investigate it in future work. Specifically, we want to explore further the problem of financial market simulation using SDE-GAN with the integration with our approach. However, as noted in the paper, our model can apply to any solver satisfied the assumptions i.e. fixed step size.


#### References

[Li et al., AISTATS 2020] Scalable Gradients for Stochastic Differential Equations.

[Yang et al., ICLR 2020] Score-based generative modeling through stochastic differential equations

[Yang et al., NeurIPS 2021] Maximum Likelihood Training of Score-Based Diffusion Models


 This paper presents an alternative class of conventional step-size neural stochastic differential equations (SDE). It provides an approximate sampling method based on sparse Gaussian Processes for the noise integral in order to put the neural SDEs into a computationally accessible form. Experiments have been conducted on the sequential finnancial time-series data and image recognition. Strengths
1. The explanation of earlier work on neural SDEs is quite solid mathematically, which makes the propsoed method more convenient to understand for readers who have little experience.
2. The experiments are quite diverged, covering function re-production, financial data analysis and conventional MNIST image recognition.
3. The GP method itself is easy to understand, as well as the general presentation of the paper.

Weaknesses
1. Even though in the result section the title mentioned it, the evaluation lacks quantitative comparison, especially for image recognition task, but more in a qualitative way. Some numerical analysis would help.
2. There are some spots where the author explains the weaknesses of earlier work, but does not connect them to the proposed solution very tightly. e.g. In section 4.1, the approaches from [64] has the problem of un-scalable random variables. But then its problem on current models and relationship with the author's algorithm remain un-reached.
3. The baseline model compared with is latent SDEs, while stronger models such as GAN based SDE solvers may want to get in. Also, apart from Hurst function, some other approximation experiments might be helpful. The questions are addressed in the weaknesses. Two of them:
1. Is there any experiments on MNIST based on accuracy can be reported?
2. In section 4.1 - you said for method covered in [64]. "the number of these random vartiables prevents optimize-then-discretize methods to scale". But what is the problem and potential negative impact of this practically? And how your method may solve it? There are no problem on the societal impact to me for this work. The technical limitations are addressed in the weaknesses.

 Thanks for the interesting further comments. I'm not intending to change my score further at this point. In my opinion the experimental results section still at best shows only that the proposed method provides improved results in a small minority of the investigated settings in the paper.

I like the suggestion of reviewer 1vjS of submitting the theoretical part of your work to a relevant journal, and extending your applied work (with new experiments and/or improved methodology) for a separate more general paper.

 Thank you again for your latest comments. We believe we have addressed all your concerns. Additionally, we have provided more results on FashionMNIST data set which are similar to those of MNIST data set (see more in our response to review `Kikm`). 

If there is anything unclear about our response or you have more concerns, we are willing to answer and discuss further. Otherwise, we hope you can read our latest response and consider changing the score accordingly.

Thank you for your time and consideration.


 Dear Reviewer,
 
Thanks a lot for your efforts in reviewing this paper. We have tried our best to address all your concerns and provided clarifications on all confusing concepts.
 
The discussion period is nearing an end, thus we wonder if you can spend some time going over our newest replies to see if they successfully answered your questions or not. This is also to give us a decent amount of time to address any of your remaining/additional concerns.
 
Best regards,

Authors.

 Thank you for your prompting response and raising the score.

### Fixing Equation 7
We have updated the paper according to the review suggestion on Equation 7. We are sorry for missing this in the last revision.

### On baseline
Although our results do not give a significantly better estimation, the benefit that our model can provide over the traditional approach is its flexibility. Our approach can be easily integrated with bigger ML models, i.e. noise can be in a latent space rather than data space, while it is nontrivial to apply Garcin’s method on ML models.

### Score-based generative model
Thank you for clarifying further on the point about image generation tasks. 

Our experiment suggests that the strategy of decreasing Hurst can be a general practice for even other data without trying other settings as we verify in the following experiment. 

With a similar setup, we can test the performance on the **FashionMNIST** data set between “decreasing Hurst” and baseline “H=0.5”. We included the NLL plot in **Figure 21** (Appendix) in our revision where “decreasing Hurst” remains having better NLLs.

Based on the empirical evaluation on MNIST and FashionMNIST, we found that it is better to inject **correlated noise** ((long-range dependent $H > 1/2$) at the **beginning** time of perturbation. At the ** end** time of perturbation, there is **no need for such correlated noise** but irregular one ($H < 1/2$).

This is described in Figure 19 and 22 for the process of noise perturbation with decreasing Hurst. 

To explain this observation, let us denote the probability distribution of the image at time $t$ as $p(X_t)$. With $t$ close to $0$ (close to real images), $p(X_t)$ is very complex, requiring a careful perturbation in which correlated noise ($H>1/2$) is a good choice because it retains dependency between two time steps (avoid drastic information removal). On the other hand, with $t$ close to the terminal time $T$, $p(X_t)$ becomes close to the Gaussian distribution so that we can accelerate the perturbation by adding even more irregular noise ($H<1/2$).

Nevertheless, the optimal strategy for Hurst function is still an open question. Finding such a strategy can be an interesting direction.


 Thanks for the responses to my review.

## Theoretical points

I am mostly satisfied by the response here. It seems that the points I raised are either typos or easily resolved for someone who is an expert in the field. I still have some mild concerns that the theoretical parts of the paper will be hard to follow for non-experts in stochastic calculus. Also, looking at the latest revised version of the paper, some of the discussion has not resulted in changes to the paper e.g. (7) is still presented as an exact equality. If the paper is accepted, please update this in the final version.

## Experimental points

The authors make some interesting points in their replies and have added some helpful extra experiments. However, overall I still think the experiments do not provide much support for the proposed method. In more detail:
* Section 5.1. Thanks for the extra discussion of the baseline. However it's still not obvious that the proposed method outperforms the baseline in this example.
* Section 5.2. The extra ablation study and comparison to the method of Garcin is helpful. However it's not obvious to me that the proposed method estimates the Hurst function particularly well, or that it does better than the Garcin approach.
* Section 5.3. The response agrees that test NLL performance of the proposed method is poor.
* Section 5.4. My point here is basically that while there are good results for one setting, this could be outweighed by the computational effort needed to find this tuning choice.

## Summary

I will increase my score to 4 (borderline reject), since I'm satisfied with the response on the theoretical points, but still think the experimental results show too little support to the proposed method for publication.

 We provide a further comparison between our learned Hurst functions and the estimations computed by [1] (reference [26] in the paper).

Note that [1] presents a statistical-based estimation while our method gives a model-based estimation.

We observe that there is a similar convergence between two methods as we increase more sample paths. There are some differences including our model can capture well for high-value Hursts. [1] tends to get low-value Hurst.

This comparison is illustrated in Figure 10, 11 (Appendix C.2).

Again, thank you for your valuable time.

Reference:

[1] Matthieu Garcin. "Estimation of time-dependent hurst exponents with variational smoothing
and application to forecasting foreign exchange rates". Physica A: Statistical Mechanics and its  Applications

 Thank you for your encouraging message and suggestions.

 Thank you for your time giving the feedback. We appreciate the follow-up comments.

1. We agree that [53] should be clarified more in the main text. However, due to the space limit, we think that having the description of Latent SDE in Appendix with a clear reference would be sufficient.
2. We emphasize that none of competitor models can output qualitative results like ours. Our model can show when a time series has more long-range dependency (less volatile) and when it is irregular (more volatile). This information is very important to make any financial decisions. Regarding the quantitative performance, our model performing well (2 outperform, 2 comparable) on the training negative-log likelihood can benefit the generative models rather than predictive models. As discussed in the general response, problems like pricing or hedging require good generative models.
3. We provide the baseline of Hurst estimations based on statistics described in the revision (Appendix C.2, Figure 10 and 11). We found a similar convergence between two methods (ours is somewhat better in estimating high-value Hurst) when increasing sample paths. The methodology difference here is that the baseline is a **statistical estimation** while our approach is a **model(machine learning)-based estimation**.
4. On image generation tasks, we reiterate that we limit the scope of this paper on the aspect of how Hurst exponents affect the score-based generative models. This is done by varying Hurst functions while keeping other setings (neural architectures,...) the same. We argue that the experiment in the paper can demonstrate such a point and is more focused on the main contribution proposing fractional Neural SDEs rather than presenting the-state-of-the-art score-based generative models.
We would appreciate it if the review can comment (even after rebuttal) further on what aspects the paper should investigate in a way that it can highlight the main contribution of the paper better.


 Thank you to the authors for their response. Based on the authors' response to my questions, I believe this paper may serve as an excellent springboard from which many other interesting avenues can be explored.

I am disappointed to see the low scores of my co-reviewers, on what I continue to believe is an excellent paper. (And for which I have left my score unchanged at 8.) As a possible avenue, if the paper is not accepted here -- the authors may wish to consider splitting this paper in two. Send the more theoretically heavy parts to a journal, which may receive such contributions more positively, and then cite their own work as the theoretical backing for a more applications-focused paper at NeurIPS/ICLR/ICML.

 Thank you for the response and clarification of several details. I've gone through the response and other reviews, and unfortunately, neither allows me to change my evaluation of the paper.

1. I still find the presentation of the main algorithm convoluted. The authors keep referring the reader to the paper [53] instead of providing a proper description of the training algorithm in the main body.
2. The experiments with financial data are below NeurIPS standards. First of all, the results in the main body are not clear at all. They consist of the plots of learned Hurst exponents and speculative interpretations of these plots. The only quantitative comparison to the baselines is provided in the Appendix. Furthermore, these results demonstrate that the proposed method performs worse than the baselines. The only two plots where the proposed method performs better are labeled "train NLL."
3. The experiments with synthetic data are below NeurIPS standards. The authors do not provide any comparison with baselines for the synthetic data. Figure 11 does not demonstrate the convergence with the number of samples quantitatively. Moreover, the paper doesn't have any proof-of-concept example where the proposed model perfectly recovers the Hurst exponent.
4. The experiments for the image generation are not extensive enough to decide on the value of the proposed approach.

To sum up, I think the paper requires a significant revision, and no minor post-rebuttal changes can drastically change my evaluation.



 First of all, we thank review again for detailed and constructive suggestions.

Regarding the presentation issues, we agree that the review suggestions help improve the current version. We have revised the submission accordingly including correcting the use of "adjoint", putting more emphasis on the convergence analysis and others.

We address the questions as follows.

## Main questions:

1. We agree there can be alternative choices of models that can achieve non-Markov properties by using Markov processes in latent space (somewhat resemble the idea of conditioning like using the conditional dist. of sparse GP). Such models can be highly expressive and can have advantages over our current models. However, our approach focusing on fBm or mBm has its strength which is its interpretability. As discussed in the general response, understanding of Hurst exponent in data plays important roles in specific domain applications. It gives more insight than black-box models (although the expressiveness can be limited). 

2. We update the paper according to the review suggestion.

3. Regarding line 198, we corrected in the draft that the kernel is not differentiable w.r.t the Hurst parameters.

4. Regarding the result of synthetic data, we provide an additional experiment showing improvements. The quality of learned Hurst function is improved as we take more sample paths in training. Changing to small step sizes did help, especially for the case of small Hurst.

## Additional questions

1. For the case of non-Brownian driver SDE (or CDE), we believe there is a related concept coined as latent force models. Latent signals are SDEs but formulated as GPs due to the equivalence between these models [Sarkka and Solin, 2019]. Observed data is modeled given latent signals, following a differential equation. Previous studies usually seek analytic solutions with linearity and Gaussianity assumptions. To our knowledge, there exists work using approximations with inducing points for data (not for time solver). So the combination is not new. However, in terms of methods, using SDE solvers can be novel, especially with some neural network components in designing models.

2. Regarding volatility models which is fractional volatility or rough volatility, we are aware of such approximations (which is relatively fast) but not so familiar with. Our initial thought was that existing methods of  work with constant Hurst. However, there is no guarantee for multifractional cases. Even that, this might be worth investigating for us in future.
3. Regarding extending Heston models with drivers i.e. GP, we think this is an interesting direction and possibly doable. A careful design of GP may be required. Again, the GP covariance of this model needs a different treatment. For the time being, we just think the covariance may require the integration of the Cox-Ingersoll-Ross process. Finally, we can use the our current approach. 
4. We still have limited knowledge about SDE-GAN to demonstrate here due to technical difficulty (i.e. parameter initializations). SDE-GAN is still a go-to as we mentioned in our general response that the importance of learning a generative model. It is one of our long term goals. 
5. The suggestion on adaptive step sizes is interesting. It is reasonable to have small step sizes when $H<1/2$ and allow bigger step sizes when $H>1/2$. It also can helps avoid time locations where variance is close to 0 (variance at inducing locations is zero) which might cause trouble learning Latent SDEs.

If you have further questions, we are willing to answer and discuss more.

 We address the concern about theory and experiment as follows:

## Theory

- We agree that Equation (7) should be an approximation sign ($\approx$) as it is derived from Equation (5). Such an approximation is well-understood via traditional stochastic calculus [1] showing that Equation (5) converges in $L^2$. The remaining approximation is from GP approximation where $H<1/2$ usually cannot be proven with traditional stochastic calculus but using rough paths.

- Regarding Line 178, this is a typo. Thanks for spotting this. It should be “computing (7)”. We update this correction in our revision. As a further explanation, the numerical instability comes from computing Cholesky decompositions for small-valued matrices. Equation (7) computes the covariance of $\Delta B_t$ which is $\mathbb{E}[\Delta B_t \Delta B_s]$. Both $\Delta B_t$, and $\Delta B_s$ are smaller (than 1), the $\Delta B_t \Delta B_s$ are even smaller. One can find the analogy with Brownian motion cases where we usually encounter $\Delta W_t^2 = \mathcal{O}( \Delta t)$. Therefore, dividing to $\Delta t$ makes computation stable.

- On using “equivalent” for Equation (4) and (12), we agree that (12) is an approximation of (4). We’ll clarify this in the next revision.

- To our knowledge, rough path theory in literature focuses on fBm rather than mBm. In our model, although our theoretical result focuses on constant Hurst of fBm, it can be extended to mBm by considering the minimum value $H = \min_t h(t)$. If $\alpha > \beta$, $\alpha$-Holder rough path spaces is a subset of $\beta$-Holder rough path spaces [2] (and see the definition 2 in the Appendix B). Simply speaking, this means that the results of minimum Hurst can be applied to those of higher Hurst values. With that, it can lead to a similar bound as in the paper.

- We create a pseudo-code to describe our approach in Appendix. To ensure the repoducibility, we will make our source available publicly. 

## Experiments

- To clarify what the baseline is, we provide an additional section in Appendix explaining it. 
For a brief description here, the sampling method as baseline in Section 5.1 is from [58] which uses an integral approximation in the form of linear combinations of normal random variables. To make the paper self-container, we provide the description of this method in Appendix. We also provide a brief background on how to obtain Hurst exponents from data.

- Regarding Section 5.2, we address this point in the general response. That is, our additional experiment can demonstrate that the obtained Hurst function is close to the true Hurst. And as including more sample paths in training, its quality is improved.

- Regarding Section 5.3, we admit that predictive performance is not the main strength of the experimental results. For financial data, we emphasize more on the application of learning Hurst exponent for the downstream tasks such as risk quantification rather than pure prediction.

- Regarding the image generation tasks, we agree that running more iterations on image generation tasks can be done. However, according to Figure 7a, our NLL converges faster. In the case of limited resources, our approach would be more preferable. As we trained models on MNIST data, we would think that our current number of training iterations is reasonable.

We are happy to clarify further if anything is unclear.

References

[1] Revuz and Yor. Continuous Martingales and Brownian Motion , 2004.

[2] Friz and Hairer, A course on rough paths with an  introduction of regularity structures, 2014


 > It’s not clear why learning the exponent is important in practice

We hope that our discussion in the general response clarifies the importance of learning Hurst exponent. Again, we reiterate that discovering Hurst exponents is crucial in many research areas where ML approaches like our model can involve. We will highlight such discussion concisely in our revision.

> The whole discussion seems to be independent of whether to introduce the Hurst exponent or no

To clarify how the method works, (1) we made the change of notation as in general response (2) we provide a pseudo-code in our revision (see Appendix). 

Because the paper contains many notations, we make this table so that the dependency to Hurst exponents can be tracked in our model (use notation of the initial draft)

| Notation            | is related to          | See equations |
|---------------------|------------------------|---------------|
| Mean $m(t)$         | covariance $k(t,s)$    | (3)           |
| Variance $v(t)$     | covariance $k(t,s)$    | (3)           |
| Covariance $k(t,s)$ | $g_i(t)$               | (8)           |
| $g_i(t)$            | $f(t,s)$               | (5)           |
| $f(t, s)$           | Hurst exponent $h(t)$  | Line 152      |

Regarding scale noise of SDE, we emphasize that the SDE with fBm or mBm is not equivalent to that of scaling noise in [Yang et al, 2020, Diederik et al., 2021]. As in the paper, the scaling factors ($\nu(t)$ and $\varsigma(t)$) depend on inducing points generated randomly before solving SDE. In other words, such coefficients are randomized before solving SDEs. In our revision, the updated notation can express this.

> Performance is not measured quantitatively

We provided in the main text the quantitative results including train log-likelihood, test log-likelihood, and RMSE in Appendix. There is no clear winner for prediction tasks but our model performs consistently well on train data. This attributes to the ability to capture Hurst exponents of data. 

Regarding the image generation tasks, we’d like to clarify that the model in [Yang et al., ICLR 2020] corresponds to the case of $H=0.5$ which is the baseline. We will clarify this in the paper. We found that with a schedule decreasing Hurst our model performs significantly better assuming all neural network architectures are the same.   

We are happy to take further questions if there is anything unclear in our response as well as in the paper.



 We thank reviewers for time and effort reviewing our paper.  We would like to clarify the common concerns from reviews here and response to individual reviews separately.

## Importance of Hurst exponents

The importance of modeling Hurst exponents lies in the applicability of fractional Brownian motions (fBm) or multifractional Brownian motions (mBm). 

One of the well-known applications is **quantitative finance research**. Extensive studies suggest various price processes including rough Heston models (mentioned by review `1vjS`), [1] long-dependency volatility models with H > ½, [2] rough volatility with H <½, [3] the Hurst exponent change over time. A common theme of such studies is to look for a price model (let us say $\mathbb{Q}$) that is as close as possible to the real market price model $\mathbb{P}$. What’s next is that we estimate or optimize the following quantity
$\mathbb{E}_\mathbb{Q}[f(x)]$ for some $f(x)$. This formulation often is encountered in option pricing and hedging problems. 

From a machine learning (ML) perspective, one may consider this as modeling $\mathbb{Q}$ as a generative model and wish to learn it from data. The choice $\mathbb{Q}$ as Neural SDEs with mBm provides the flexibility from neural networks but also inherits the foundation from quantitative finance research.

To this end, we would like to emphasize that our model is a building block for neural SDEs which plays an important role in the intersection between ML and quantitative finance..

Moreover, many applications can be **found in other areas of basic science**. There are existing studies in physical systems and biological systems (see [4, 5, 6, 7]) emphasizing the characteristics of fractional Brownian motions (tracking the motion of mRNA, cell, etc). With the spirit that AI benefits science, we believe that our Neural SDEs with mBm will have impacts on such areas in the long run.

We agree the above discussion is necessary in the initial submission. Due to the space limit, we include a summary of this discussion to our revision.

## Revised Experiment in Section 5.2.

Based on review comments and suggestions, we conduct further experiments of capturing the true Hurst function. We investigate the cases including more sample paths, and less “rough” at initial points. 

Regarding our Hurst estimation not well fit to the true Hurst, we identify the key problems including (1) the number of sample paths is small, the data is very irregular near origin causing LatentSDE failed to learn correct parameters. 

The revision provides an additional case of Hurst function and  the new improved results. Our Hurst estimation approaches the true Hurst when the data contains more sample paths. We also observe that small step size helps to learn small Hurst values. 

## Notation change

According to the concern of review `FT9y` and the suggestion of review `1vjS`, we replace the notation of $\mu(t)$ and $v(t)$ such that they can express the dependence to inducing points. That is, we change inducing points from $B’_Z$ to $u$ and
$$\mu(t) \to \mu(t; u, Z) \quad v(t) \to v(t;Z)$$
where $u, Z$ are inducing points and their time locations. This change is to improve the clarity and does not affect the methods presented in the paper.

### References
[1] Comte, and Renault, 1998, "Long-memory in continuous-time stochastic volatility models", Mathematical Finance.

[2] Gatheral, Jaisson & Rosenbaum,2018, "Volatility is rough", Quantitative Finance.

[3] Sylvain Corlay, Joachim Lebovits, and Jacques Lévy Véhel, 2014, "Multifractional Stochastic volatility models", Mathematical Finance.

[4] Golding and Cox 2006, "Physical nature of bacterial cytoplasm", Phys. Rev. Lett.

[5] He et al. 2008, "Random time-scale invariant diffusion and transport coefficients", Phys. Rev. Lett.

[6] Höfling, and Franosch 2013, "Anomalous transport in the crowded world of biological cells", Reports on Progress in Physics.

[7] di Rienzo et al. 2014, "Probing short-range protein Brownian motion in the cytoplasm of living cells", Nat. Commun.


 The paper studies a different noise model in SDEs. Namely, the model integrates the Wiener process in time scaling it with the function (that depends on time) called Hurst Exponent. It seems that this formulation is equivalent to the Ito SDE with a different scaling term. However, I don't have a background on the Hurst exponent, and the authors do not discuss such a relation. 

Throughout the paper, the authors claim that their main goal is to learn the Hurst exponent from the given data. They propose to sample the noise via the Gaussian process and learn the model via the variational inference as in neural SDEs [1]. Unfortunately, it's hard to describe the proposed method more precisely due to its poor description in the paper.

The authors provide two types of experiments: the estimation of the Hurst function from the given time series; the generation of the MNIST dataset with different noise models (different Hurst exponents).

1. Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 3870–3882, 2020.
 Unfortunately, the paper does a poor job on all main aspects of a scientific publication: a motivation of the problem, a clear description of the proposed approach, and a rigorous comparison of the proposed approach to the competitors.

For the main motivation, the authors propose the learning of the Hurst exponent. However, it's not clear why learning the exponent is important in practice. I can assume that this can be used for further simulation of the stochastic process, but it's not clear how methods that do not take the Hurst exponent into account (just learning the scaling term in Ito SDE) compare to the proposed method.

For the method description, the paper does not describe the optimization problem and parameterization of the model (the authors claim to learn all the terms in Ito SDE). The main development of the paper is the sampling stage (section 4.3), but the authors do not describe how they propagate gradients through these samples that depend on the model. Furthermore, the whole algorithm seems to be independent of the exact form of the scaling term in SDE (whether it's introduced with the exponent or not). Thus, the whole discussion on the Hurst exponent doesn't seem to bring any practical value or modify the algorithm anyhow. In the considered context, the Hurst exponent can be considered simply as another parameterization of the scaling term in Ito SDE.

In what follows, there is a list of minor clarity issues throughout the paper.
- line 41. It's not clear w.r.t. what parameter the models cannot be scaled.
- Related work section rather enumerates a broad field of literature than relates the current paper to the existing literature.
- There is a broken reference on line 178.
- The sentence in lines 225-227 is not clear. Maybe there is a typo there.

The empirical evaluation of the proposed method does not give any evidence of its efficiency. For the Hurst exponent estimation, the authors provide experiments with two types of data. For the synthetic data, the authors report the quality of approximation in Fig 4(a). It's hard to measure the quality of the approximation since the further application is not clear, but I would say that the quality is unsatisfactory. The authors claim that the problem is challenging since they recover it by a single sample, but then I would expect demonstrations of the approach on less challenging problems. For the financial data, the performance is not measured quantitatively at all.

The image generation with different Hurst exponents is not evaluated properly. Once again, it's not clear why it is not an introduction of a different scaling term. Furthermore, some works study different noise scales [2,3], and no comparison with them is provided.

2. Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score-based generative modeling through stochastic differential equations." arXiv preprint arXiv:2011.13456 (2020).
3. Kingma, Diederik, Tim Salimans, Ben Poole, and Jonathan Ho. "Variational diffusion models." Advances in neural information processing systems 34 (2021): 21696-21707. My questions and suggestions follow naturally from the section "Strengths And Weaknesses." It's hard to assess the limitations of the approach due to its poor description and motivation.

 Neural SDEs are generalised so as to admit a certain class of non-Brownian drivers. This is done by aligning a Riemann--Stieltjes discretisation of the non-Brownian driver with an Euler--Maruyama discretisation to a separate SDE, so as to reduce to the case of a neural SDE with random coefficients; this randomness is specifically a dependence on a Gaussian process. Scalable sampling of this Gaussian process is performed via sparse inducing points. **Strength: technical approach**

This paper is technically very strong. "Neural fractional SDEs" have been mooted for some time, with no real work in large part due to the non-scalability of sampling fractional Brownian motion (fBM). This paper overcomes this issue through a combination of (a) switching from fBM to another related process (due to Levy); (b) drawing on the theory of sparse inducing points; (c) demonstrating convergence via rough path theory; (d) carefully linking up all the pieces in specifically discrete time.

This is an excellent tour-de-force requiring expertise across multiple disciplines (GPs; SDEs; numerical methods; rough path theory; deep learning.). To the best of my knowledge this approach is novel.

**Strength: attention to computational cost etc.**

This paper does a good job discussing the varying computational costs of each procedure (e.g. $O(N)$ for sparse inducing points; $O(N^3)$ for Cholesky decompositions, etc.) This is of course important in practice. 

This is true even at the level of subtle technical details, such as the computational benefit of Stratonovich integration over Ito integration when backpropagating via optimise-then-discretise.

**Strength: discussion of limitations**

The authors are careful to state the limitations of their approach explicitly.

**Weakness: not self contained**

This paper obviously assumes a sophisticated background in stochastic processes. As such, at present, I do not believe the paper is self-contained: the results used are frequently not stated, and instead only a reference is given.

For example making clearer what is the relationship between equation (1) and fBM; the definition of mBm (line 88); the relationship between Hurst exponent and Holder continuity (line 100); etc.

(I admit I am hard-pressed to identify what should be cut in order to make space for such explanations, as the paper is already very dense. I suspect it may require some reorganisation -- for example the second paragraph of "Computational analysis" (line 271) could be replaced with a few extra words at the end of line 193.)

**Weakness: "adjoint method"**

The term "adjoint" has become heavily overloaded. There are multiple ways to backpropagate through a differential equation solve. In the current literature is is often used to mean specifically optimise-then-discretise, but in older literature it was often used to mean specifically discretise-then-optimise. Following [Ref. 37 of this paper, Remark 5.5], I would recommend avoiding the term "adjoint" altogether.

(In passing, the authors may be interested in [Ref. 37 of this paper, Section 5.2.3], who prove optimise-then-discretise backpropagation through SDEs via rough path theory.)

**Weakness: presentation**

The text could doing with being reviewed for typos (some are listed below) and mathematical clarity (as  above). In addition:

Equation (12) is a bit of a lie -- this is constructed for specifically the Euler--Maruyama discretisation, not the continuous limit.

Line 222 could be clarified: state precisely what is the nature of the association.

That the kernel on line 199 is the kernel of fBM should be stated explicitly. (Unless this is known to the reader it comes out of nowhere.)

I believe the presentation of Theorem 1 could be clarified: what is shown is a general result, that sort-of corresponds to the construction being presented. I believe this distinction should be emphasised a little more carefully, e.g. by turning "Convergence analysis" into a new section; or similar.

I am not that familiar with the details of inducing points for Gaussian processes -- at least for me I would have appreciated greater technical precision in Section 3.3.

**Minor issues**

Line 82: typo "of".

Line 141: "on sampling procedure" should be "on the sampling procedure" or similar.

Line 152: missing close bracket.

Line 155: "require computing Cholesky decomposition" should be "which require computing the Cholesky decomposition"

Line 156: equation (5) is referenced before it is introduced.

Line 137 (and elsewhere): the precise definition of $Z$ is not given.

Line 139: $m$ and $v$ should be notated as functions of $u$. This becomes important in equation (10), in which the need for sampling $B'_Z$ becomes unclear.

Line 227: should presumably state the only preliminary results are available for fBM.

Line 229: only $\xi_t$ -> only one $\xi_t$

Line 243: time->times

Line 248: I think this could be simplified to $\rho \in (1/(2H), 3/2]$?

Line 286: "SDEs model" does not parse. 1. In personal discussions around the possibility of "neural fractional SDEs", the general conclusion has been that it probably wasn't worth the effort -- because similar behaviour could probably be mimicked simply by taking sufficient complicated drifts and diffusions, and by working in a latent space.

   (As this hidden Markov structure implies the possibility of a non-Markov structure to its output -- which might be able to approximate the autocorrelation of a fractional SDE. And indeed latent spaces are standard when working with neural SDEs anyway.)

   As such I would really have expected to see experiments comparing the proposed approach against a "complicated" neural SDE with Brownian driver. This paper is technically impressive but the corollary to that is that it is also technically impenetrable -- I believe for this paper to see impact it will need to demonstrate a substantial improvement against this much-simpler-to-implement alternative.

   (Which I note is already not that simple to implement itself!)

   To my mind this is the single greatest limitation of this work, and it is a substantial limitation.

2. I would emphasise the need to improve several aspects of the presentation, as detailed in the main review.

3. The Remark of line 198: it is mentioned that working with the fBM kernel directly does not work, as it is not differentiable. Is this an issue because of the desire to backpropagate through it, so as to train the Hurst parameter? (Or is it something else?) Either way I think it worth including the rationale here.
4. The Hurst exponents obtained from synthetic data are moderately encouraging, but not that convincing. Can this experiment be tried with much larger amounts of data, and small step sizes, so that the true Hurst exponent really is recovered -- to verify the efficacy of the proposed methodology? The failure to work convincingly on relatively simple univariate data is unfortunate.

Overall I would emphasise that I nonetheless enjoyed this paper.

Unrelated to my review score -- what follows are some questions/suggestions just for my own curiosity.

1. This paper uses inducing points for sampling from Gaussian processes. I have not previously seen this applied in the context of non-Brownian drivers for SDEs -- is this particular combination new here?

2. The problem of simulating fractional Brownian motion is classical, and e.g. fractional volatility models are moderately popular. It seems to me that the approach proposed in this paper, of using sparse inducing points, should apply equally well to fBM (which is also a Gaussian process). What then happens? I assume this has been tried or otherwise fails for some reason.

3. Do the authors believe it would be possible/reasonable to e.g. modify a Heston model to use the alternate driver of (1), and then apply their efficient sampling scheme? This may well have different properties to the rough Heston model currently considered in the literature, so to what extent this is a reasonable model is unclear to me.

4. It is part of the folklore (and not really written anywhere in the literature, so far as I know) that latent SDEs consistently underestimate the variance. I haven't dug into the reason for this, but if the authors continue using latent SDES in future work then they wish to attempt to address this issue alongside. (e.g. by additionally introducing a discriminator and simultaneously training the model as an SDE-GAN -- I speculate).

5. The authors note that their analysis is restricted to the case of Euler--Maruyama with fixed step size, and that in future work they would like to consider (a) better convergence results; (b) GP drivers for NCDEs; (c) higher-order SDE solvers using Levy area. For future work, I would also like to suggest that handling the case of adaptive timestepping would also be particularly useful/interesting.

   (I am aware that adaptive SDEs are not a well-understood topic, with a reasonable number of errors in the published literature. For what it is worth I recommend the references listed in [0] as the best starting point for adaptive SDE timestepping. My own understanding is that general-case adaptivity -- i.e. arbitrary stepsize changes, rather than just halving/doubling -- currently requires the SDE to satisfy the commutativity condition.)

   [0] https://docs.kidger.site/diffrax/api/stepsize_controller/#diffrax.PIDController This work has no ethical concerns.

 The paper proposes an approach to neural SDEs using an alternative to standard Brownian noise. The authors propose a particular definition of alternative noise. They develop some theoretical results which allow standard neural SDE methods to be used to fit an approximation to their stochastic process. Some convergence results are also proved. Finally the method is applied to various experiments. **Note**: My original score based on this review was 3 (Reject), but I've increased this to 4 (Borderline reject) following the rebuttal.

## Overview

* Originality: good. This seems like an interesting approach to investigate, and is novel to me, although I am not an expert in this area.
* Quality: poor. The experiments are not convincing (see below for details).
* Clarity: poor. I found the theoretical material unclear (see below for details).
* Significance: poor. In my opinion the paper did not make a strong case for the benefits of this approach.

## Strengths

The paper develops some non-trivial theoretical results.

## Weaknesses - overview

I found two main weaknesses in the paper.

1) The theoretical material is hard to follow, and seems to involves a lot of non-rigorous derivations.
2) The experiments do not demonstrate good performance of the proposed method.

I have less confidence about weakness 1) as I am not an expert in the subject area. It may be that the material would be more convincing to an expert. However in my opinion weakness 2) alone is a major limitation of the paper.

## Weaknesses - theory

* Equation (5) is an approximation, but is then treated as exact to derive results throughout the rest of Section 4 e.g. equation (7). Rigorous proofs should also be given.
* Line 178 says "Note that directly modeling $\Delta B_t$ in (6) is numerically unstable when computing (3.3)..." Is this a typo? There is no equation (3.3). This makes the material hard to follow.
* Section 4.4 says the original SDE (4) is "equivalent" to (12). But it seems that this result relies on many approximations e.g. the inducing points scheme. The authors should say that (12) is an approximation or prove that it is equivalent.
* Why does Theorem 1 treat the Hurst exponent as a constant, when earlier it was defined as a function?
* The authors do not give a summary of their proposed method. This makes it hard for others to implement.

## Weaknesses - experiments

* In Section 5.1 the baseline seems as good as the proposed method. It's also unclear to me where the exact details of the two methods can be found.
* In Section 5.2 the proposed method does not seem to give a good fit of the true function. Also, no baseline comparison is included.
* In Section 5.3, the proposed method only does well on training evaluation, and does poorly in test evaluation. This suggests it is overfitting.
* In Section 5.4, the authors write "Although we do not present a way to find optimal h(t), we show that there is a setting in which the NLL appears to be better". This is an interesting exploratory finding. However, given the extra cost of trying several scenarios to get this improvement, it's not clear that the proposed approach is superior to simply running more training iterations of the standard approach. Can you address the points listed under "weaknesses"? The authors include a section on technical limitations. As this is a paper on foundational machine learning, extending existing methods, there do no not seem to be any direct negative societal impacts to discuss.

Meta-review: Ratings: 4/8/3/7/7.
Confidence: 3/4/4/2/4.
There was no discussion among reviewers, but there was a good amount of discussion between authors and reviewers.

Note that there are 5 reviewers since the original reviews had low confidence.

Summary: This paper studies a variant of neural SDEs that replace the standard Brownian motion driving the NSDE with a Riemann–Liouville fractional Brownian motion that has a learnt time-varying Hurst function. The paper and subsequent discussion unfortunately failed to persuade 2/5 of the reviewers, especially in terms of accessibility of the theory and the empirical validation. However, the reviewers agree that the topic and results are interesting.

Recommendation: I recommend to accept this submission.

Decision: Accept