Non-adversarial training of Neural SDEs with signature kernel scores
Authors: Zacharia Issa
Blanka Horvath
Maud Lemercier
Cristopher Salvi
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural SDEs
score-based generative models
signature kernels
time series
TL;DR: We introduce a novel class of score functions on pathspace based on signature kernels and use them as objective for training Neural SDEs non-adversarially.

Abstract: Neural SDEs are continuous-time generative models for sequential data. State-of-the-art performance for irregular time series generation has been previously obtained by training these models adversarially as GANs. However, as typical for GAN architectures, training is notoriously unstable, often suffers from mode collapse, and requires specialised techniques such as weight clipping and gradient penalty to mitigate these issues. In this paper, we introduce a novel class of scoring rules on pathspace based on signature kernels and use them as objective for training Neural SDEs non-adversarially. By showing strict properness of such kernel scores and consistency of the corresponding estimators, we provide existence and uniqueness guarantees for the minimiser. With this formulation, evaluating the generator-discriminator pair amounts to solving a system of linear path-dependent PDEs which allows for memory-efficient adjoint-based backpropagation. Moreover, because the proposed kernel scores are well-defined for paths with values in infinite dimensional spaces of functions, our framework can be easily extended to generate spatiotemporal data. Our procedure significantly outperforms alternative ways of training Neural SDEs on a variety of tasks including the simulation of rough volatility models, the conditional probabilistic forecasts of real-world forex pairs where the conditioning variable is an observed past trajectory, and the mesh-free generation of limit order book dynamics.

Non-adversarial training of Neural SDEs with signature kernel scores
Zacharia Issa"
Blanka Horvath 2.3.4
Maud Lemercier
Cristopher Salvi
Abstract
Neural SDEs are continuous-time generative models for sequential data. State- of-the-art performance for irregular time series generation has been previously obtained by training these models adversarially GANs. However, typical for GAN architectures, training is notoriously unstable, often suffers from mode collapse, and requires specialised techniques such as weight clipping and gradient penalty to mitigate these issues. In this paper, we introduce a novel class scoring rules pathspace based signature kernels and use them objective for training Neural SDEs non-ad aare--arrrrrlll.... By showing strict properness of such kernel scores and consistency of the corresponding estimators, uniqueness guarantees for the minimiser. With this formulation, evaluating the generator-disg disrrmiitod pair amounts solving system of linear path-dependent PDEs which allows for memory--fficient adjoint-based backpropagation. Moreover, because the proposed kernel scores are well-defined for paths with values in infinite dimensional spaces of functions, our framework can be easily extended to generate spatiotemporal data. Our procedure permits conditioning on rich variety market conditions and significantly outperforms alternative ways of training Neural SDEs on a variety of tasks including the simulation of rough volatility models, the conditional probabilistic forecasts of real-world forex pairs where the conditioning variable is an observed past trajectory, and the mesh-free generation of limit order book dynamics.
provide existence and
Introduction
Stochastic differential equations (SDEs) are a dominant modelling framework in many areas of science and engineering. They naturally extend ordinary differential equations (ODEs) for modelling dynamical systems that evolve under the influence of randomness.
neural stochastic differential equation (Neural SDE) a continuous-time generative model for irregular time series where the drift and diffusion functions of an SDE are parametrised by neural networks [TR19 JB19 19 MvdHRM20 LWCD20 21 MSKF21]. These models have become increasingly popular among financial practitioners for pricing and hedging of derivatives and overall risk management ASSS2 GSVS 20 CJB23] HFH 23).. Training Neural SDE amounts to minimising over model parameters an appropriate notion of distance between the law on pathspace generated by the SDE and the empirical law supported on observed data sample paths.
Corresponding author: zacharii....... issa@kcl. ac. uk
Department of Mathematics, King's College London, London, United Kingdom.
DDpartment of Mathematics, Oxford University, Oxford, United Kingdom.
'The Oxford Man Institute, Oxford, United Kingdom.
*The Alan Turing Institute, London, United Kingdom.
Deparrmeent of Mathematics, Imperial College London, London, United Kingdom.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Various choices of training mechanisms have been proposed in the literature: ofate-of-the-ase art perfor- mance has been achieved by training Neural SDEs adversarially as Wasserstein-GAre [1]] However, as typical for GAN architectures, training is notoriously unstable, often suffers from mode collapse, and requires specialised techniques such as weight clipping and gradient penalty.
In this paper we introduce a novel class of scoring rules based on signature kernels. a class of characteristic kernels on paths 21 2FC 21 2 CLS23 H3]+ and use them as objective for training Neural SDEs non-adversarially. We provide existence and unique- ness guarantees for the minimiser by showing strict properness of the signature kernel scores and consistency of the corresponding estimators.
With this training formulation, the generator-discriminate discriminator pair becomes entirely mesh-free and can be evaluated by solving a system of linear path-dependent PDEs which allows for memory-efficient adjoint-based backpropagation. In addition, because the proposed kernel scores are well-defined for classes of paths with values in infinite dimensional spaces of functions, our framework can be easily extended to the generation of spatiotemporal signals.
We demonstrate how our procedure more stable and outperforms alternative ways of training Neural SDEs a variety of tasks from quantitative finance including the simulation of rough volatility models, the conditional probabilistic forecasts of real-world forex pairs where the conditioning variable is an observed past trajectory, and the mesh-free generation of limit order book dynamics.
2 Related work
Prior to our work, two main approaches have been proposed to fit Neural SDE as time series generative model, differing in their choice of divergence to compare laws on pathspace.
The SDE-GAN model introduced in 2FL,+21 uses the 1-Wasserstein distance to train a Neural SDE as Wasserstein-Gal GAAN [ACB17]. Namely, the "witness functions" of the 1-Wasserstein distance are parameterised by neural controlled differential equations [KMFL20, MSK 20] and the generator- discriminator pair is trained adversarially. SDE-GANs are relatively unstable to train mainly because they require a Lipschitz discriminator. Several techniques such as weight clipping and gradient penalty have been introduced to enforce the Lipschitz constraint and partially mitigate the instability issue [Ki22]] SDE-GANs are also sensitive to other hyperparameters, such as the choice of optimisers, their learning rate and momentum, where small changes can yield erratic behavior.
The latent SDE model [LCDD20 trains a Neural SDE with respect to the KI. divergence using the principles of variational inference for SDEss [Opp19]. This approach consists in maximising an objective that includes the KL. divergence between the laws produced by the original SDE (the prior) and an auxiliary SDE (the approximate posterior). The two SDEs have the same diffusion term but different initial conditions and drifts, and a standard formula for their KL divergence exists. After training, the learned prior can be used generate new sample paths. Latent SDBs can be interpreted as variational autoencoders, and generally yield worse performance than SDE-GANs, which are more challenging to train. but offer greater model capacity.
Besides Neural SDBs, other time series generative models have been proposed, including discrete-time models such as [JVd and [NSW+20 which are trained adversarially, continuous-tine flow
processes DCB 20) and score- based diffusion models for audio generation [CZZ+ 20 KPH+20)
The class of score-based generative models (SGMs) seeks to map data distribution into a known prior distribution via an SDE ISSDK+20 VKK21]. During training, the (Stein) score (L.J of the SDE marginals is estimated and then used to construct reverse-time SDE. By sampling data from the prior and solving the reverse- reverse-time SDE, one can generate samples that follow the original data distribution. We note that our techniques for generative modelling via scoring rules, although similar in terminology, are fundamentally different, as we train Neural SDEs with respect to a loss function that directly consumes the law on pathspace generated by the SDE.
Scoring rules [GR07] have been used to define training objectives for generative networks BMNI6
GSvdB which have been shown to be easier to optimize compared to GANs [PADD21 PD22] Closer to our work is [PADD2I which constructs statistical scores for discrete sspatio-Jtemport
'In NSW the discriminator is formulated in continuous time based on a different parametrisation to approximate the W-Wasserstein distance, also later used in [115

---

signals. However, their strict properness is ensured under Markov-type assumptions and their (sntnnouss(spacee (space-)time limit has not been studied. A key aspect of our work is to develop consistent and effective scoring rules for generative modelling in the continuous-time setting. While [BO21 has also introduced scoring rules for continuous-time processes, our emphasis lies in constructing so-called kernel scores specifically for training Neural SDE and Neural SPDE generative models.
The Neural SPDE model introduced in [SLG22 parametrises the solution operator of stochastic partial differential equations (SPDEs), which extend SDEs for modelling signals that vary both in space and in time. So far, Neural SPDEs have been trained in supervised fashion by minimizing the pathwise L norm between pairs of spatiotemporal signals. While this approach has proven effective learning fast surrogate SPDE solvers, it is not well- well-suited for generative modeling where the goal is to approximate probability measures supported on spatiotemporal functions. In this work, we propose a new training objective for Neural SPDEs improve their generative modeling capabilities.
Training Neural SDEs with signature kernel scores
3.1 Background
We take FP) as the underlying probability space. Let T >> and dg E N. Denote by X' be the space of continuous paths of bounded variation from [o, T) to Rdd with one monotone coordinate For any random variable X with values on X, we denote by Px := P o X-1 its law.
The signature map X' T is defined for any path x E X' the infinite collection S(x) (z), (x)).... of iterated -tiielies- integrals
(*x) =- Sr Jochs_.sweer dxi, & dxig @ ... dais k: E
N,
where & is the standard tensor product of vector spaces and T = R RRORR & + (Rd)) +
Any inner product (t:-), on TR yields canonical Hilbert-Schmine Schmidt inner product (:, 1) on (R& j* for any & E N, which in turn yields, by linearity, a family of inner products (.,-) on T. We refer the reader to [CX21] for an in-depth analysis of different choices. By a slight abuse of notation, we use the same symbol to denote the Hilbert space obtained by completing T with respect to (,
3.2 Neural SDES
Let W [0, T] - Rd be a dun-dimensional Brownian motion and a ~ (O, Ta.) be sampled from da da-dimensional standard normal. The values dw da E N are hyperparameters describing the size of the noise. A Neural SDE a model of the form
Yo = (a), dY, pttt, Yj) + at Yi) dW,, x AgYi by
(1)
for torter E [0, TJ. with : [0, T + Rd the strong solution, if exists, to the Stratonovich SDE, where
: R + Rd
[0, x Rd + Rdd
[0, x Rd - R2 xdw.
are suitably regular neural networks, and Ag Rddx bo Rdd The dimension dy E N is hyperparameter describing the size of the hidden state. If μe, are Lipschitz and E, [e (a) < co then the solution Y exists and is unique.
Given a target v--uul random variable x with law Px,s, the goal is to train a Neural SDE so that the generated law Pxx x is as close as possible Pxve, for some appropriate notion of closeness.
3.3 Signature kernels scores
Scoring rules are a well-established class of functionals to represent the penalty assigned to a distribution given observed outcome, thereby providing way to assess the quality of probabilistic forecast. Scoring rules have been applied to a wide range of areas including econometrics [MS1]],
'This is a technical assumption needed to ensure characteristicness of the signature kemnl.. See Proposition
3.1
The monotone coordinate is usually taken to be time.
3

---

weather forecasting [GRO5], and generative modelling [PADD2I] How to effectively select scoring rule is challenging and somewhat dusk-dependent problem, particularly when the data is sequential. Scoring rules based on kernels offer the advantages of working infinite dimensional data without some of the concomitant drawbacks, such as absence of densities. Next, we introduce class of scoring rules on paths based on signature kernels to measure closeness between path-valued random variables. These will be used in the next section to train Neural SDEs.
unstructured and
The signature kernel ksig: X'' x R is symmetric positive semidefinite function defined for any pair of paths 2, y E X' as hag (x. y) := (())) In [SCF* 21] the authors provided a kernel trick proving that the signature kernel satisfies
ksig 3) = f(T,T) where ((s,t)=1+ I f(u, f(u, ()
which reduces to a linear hyperbolic PDE the when the paths z, y are almost-everywhere dif- ferentiable. Several finite difference schemes are available for numerically evaluating solutions to Equation ( see [SCF+21] Section 3.1] for details.
We denote by H the unique reproducing kernel Hilbert space (RKHS) of ksig. From now on we endow X' with topology with the respect which the signature is continuous; see (CT22) for various choices of such topologies. Denote by (x))) the set of Borel probability measures on X.
Proposition 3.1. The signature kernel is characteristic for every compact set K. C. X, i.e. the map P f Risi (xr, TP (dx) from ((K)) to injective.
Remark 3.2. The proof this statement classical and is a simple consequence of the universal approximation property of the signature [KBPA + 19 Proposition A.6] and the equivalence between universality the feature map and characteristicness 6]. In particular, Proposition (3.1) implies that the signature kernel is cc-universal, i.e. for every compact subset IC C X, the linear span of the set of path functionals {kang (z. :) x E KC} is dense in C(K)) in the the topology of uniform convergence.
the corresponding kernel [SGS18 Theorem
We define the signature kernel score Osig! P(X) X R for any P E P(X) and y X
@sgg (P, y) := Ex, (hag (z. x²)] - 2E 2 (kag (x., v)].
A highly desirable property to require from score is its strict properness, consisting in assigning the lowest expected score when the proposed prediction is realised by the true probability distribution.
Proposition 3.3. For any compact IC C X, Qig is strictly proper kernel score relative to P(K)), i.e.
Ey--e( (Q,)) < Be (P, v)] for all P, Q E P(K), with equality if and only if
Q.
The proof this statement can be found in the appendix and follows from [GR07 Theorem and Proposition 3.1 We note that the signature kernel score induces a divergence on P(x) known as the signature kernel maximum mean discrepancy (MMD), defined for any P, Q E P(Y) as
D Dkun (P.
)2 2 Eyy- Poya (P, (P. Ey Rias (v. v/)]-
The following result provides consistent and unbiased estimator for evaluating the signature kernel score from observed sample paths. The proof can be found in the appendix and follows from standard results for the associated MMD [GRR 12 Lemma 6).
Proposition 3.4. Let P E P(X) and y E X. Given m sample paths {ar y ~ P., the following is consistent and unbiased estimator of Qsig
(P.)) mmmm 1)
keig (xx x²)
Σ kisi ,)) m
2
J
3.4 Non-adversarial training of Neural SDEs via signature kernel scores
We now have all the elements to outline the procedure we propose to train the Neural SDE model non-adversarially using signature kernel scores introduced in the previous section.

---

Unconditional setting We are given target X'valued random variable X with law Px Recall the notation x for the law generated by the SDE TT. The training objective given by
min (() where
[(0) Ey-- you ( x y)]-
Note that training with respect to Dkug is an equivalent optimisation as the second expectation in equation (3) is constant with respect to 0. This means that in the unconditional setting our model
corresponds to a continuous time generative network of LSZI1
and (5) the generator-diserimnnato pair can be evaluated by solving a system of linear PDEs depending on sample paths from the Neural SDE; in summary:
Combining equations |
Generator: X SDESOlve(O)
Discriminator: [()) & PDESolve (x", x (6)
Remark 3.5. The generation of sample paths from X from the SDE solver and the evaluation of the objective L via the PDE solver can principle be performed concurrently, although, in our implementation we evaluate the full model (6 in a sequential manner.
Conditional setting It is straightforward to extend our framework to the conditional setting where Q is some distribution we wish to condition on, and xx== (- (-(xr) is target conditional distribution with x ~ Q. By feeding the observed sample z as an additional variable to all neural networks the Neural SDE [T. the generated strong solution provides parametric conditional law P (x(r)) and the model can be trained according to the modified objective
min (0) where (' min E Br yoe (+1x) sig x (+1)),))
(7)
Because Usgg is strictly proper, the solution to is IP Xxx (-|z)) = Px (-12) Q-almost everywhere. With data sampled as {(0', ) where x² and y¹ ~ Px (-122)) we can replace eq. ( by
n Dsig (P x (-x"))
min 71
(8)
We note that in our experiments we focus on the specific case where the conditioning variable x is path in A' corresponding to the observed past trajectory of some financial assets (see Figure 2
3.5 Additional details
Samples from Ximue are observed on a discrete, possibly irregular, time grid while samples from x are generated from (1) by means of an SDE solver of choice (see [Kid22, Section 5.1] for details). Interpolating in time between observations produces a discrete measure on path space, the ones desired to be modelled. The interpolation choice is usually unimportant and simple linear interpolation is often sufficient. See [MKYL22] for other choices of interpolation.
Interpolation
Backpropagation Training Neural SDE usually means backpropagating through the SDE solver. Three main ways of differentiating through an SDE have been studied the literature: 1) Discretise- then-optimise backpropagates through the internal operations of the SDE solver. This option is memory inefficient, but will produce accurate and fast gradient estimates. 2) Aptiimisethhen-eee then-discretise derives a mackwards-nn-time SDE, which is then solved numerically. This option is memory efficient, but gradient estimates are prone to numerical errors and generally slow to compute. We note that unlike the case of Neural ODEs, giving precise meaning to the backward SDE falls outside the usual framework of diffusions. However, rough path theory [Ly998 FLMS23] provides an elegant remedy by allowing solutions to forward and backward SDEs to be defined pathwise, similarly to the case of ODEs; see [Kid22] Appendix C.3.3] for precise statement. 3) Reversible solvers are memory efficient and accurate, but generally slow. Here we do not advocate for any particular choice as all of the above backpropagation options are compatible with our pipeline.
Similarly. because the signature kernel score can be evaluated by solving system of PDEs, backprop- agation can be carried out by differentiating through the PDE solver analogously to the -iscretise-then- optimise option for SDEs. We note that [LSC+2I] showed that directional derivatives of signature kernels solve a system of adjoint-PDDss which can be leveraged to backpropagate through the discriminator using
weweniii-tteennieeeee approach. We used this approach in our experiments.
5

---

Itō vs Stratonovich Stratonovich SDEs are slightly more efficient to backpropagate through using an optimise- optimise-then-diseeee then-discrettiee approach. In the case of Itō SDEs, the backward equation is derived by applying the Itō-Stratonovich correction term to convert it into Stratonovich SDE, deriving the corresponding backward equation through rough path theoretical arguments, and then converting back to an Itô SDE by applying second Stratonovich-Ie correction.
Paths with values in infinite dimensional spaces While we have defined the signature kemel for paths of bounded variation with values in R the kemel is still well-defined when R is replaced with a generic Hilbert space V. Remarkably, even when V is infinite dimensional, the evaluation of the kernel can be carried out, as Equation (2) only depends on pairwise inner products between the values of the input paths. In particular, the kernel can be evaluated on paths taking their values in functional spaces, which has far-reaching consequences in practice. For example, this gives the flexibility to map the values of finite dimensional input paths into a possibly infinite dimensional feature space, such as the reproducing kernel Hilbert space of a kernel x on Rd= that is, V = =H.. This also provides a natural kernel for spatiotemporal signals, such as paths taking their values in V V= (D), the space of equare-integrabl functions on a compact domain D C IR., For practical applications, the inner product in Equation 2 can be approximated using discrete observations of the input signals on a mesh of the spatial domain D. The inner product in LD) can be replaced with more general kernels as those introduced in [WD22). While it has become common practice to use signature kernels on the RKHS-lifts of Euclidean- valluud paths, the ability to define and compute signature kernels on spatiotemporal signals has been, to our knowledge, overlooked in the literature.
Experiments
We perform experiments across five datasets. First is a univariate synthetic example, the benchmark model, which permits to readily verify the quality of simulated outputs. The second synthetic example is a state of-the-art univariate stochastic volatility model, called rough Bergomi model. The rough Bergomi model realistically captures many relevant properties of options data, but due to its rough (and hence -arrkoaknv nature it is well-known be difficult to simulate. The third is multidimensional example with foreign exchange (forex, or FX) currency pairs, which was chosen not only because of the relevance and capitalisation of FX markets but also due to its well- intricate complexity. Fourth is a univariate example, where we demonstrate the method's ability to condition on relevant variables, given by paths. Finally we present a spatiotemporal generative example, where we seek to simulate the dynamics of the NASDAQ limit order book.
For the unconditional examples, we compare against the SDE-GAN from 21] and against NN the same pipeline as the one we proposed, but using an approximation Osiy of the signature kernel score ssig obtained by truncating signatures at some level N E N. We evaluate each training instance with variety of metrics. The first is the aalmogorov-Smiroo (KS) test, which is a nonparameteric two-sample test between two empirical probability distributions on IR. see [Smi39] for more details. We apply the KS test on the marginals between batch of generated paths against an unseen batch from the real data distribution. We repeated this test 5000 times at the 5% significance level and reported the average KS score along with the average Type error. Each training instance was kept to a maximum of 2 hours for the synthetic examples, and 4 hours for the real data example. Finally, as mentioned at the end of Section 3.5) when training with respect to dsg we mapped path state values into (H, x) where x. denotes the RBF kernel on Rdd. Additional details on hyperparameter selection, learning rates, optimisers and further evaluation metrics can be found in the Appendix.
4.1 Geometric Brownian motion
As toy example, we seek to learn a geometric Brownian motion (gBm) of the form
dyt = py,dt + dy, yo 1,
(9)
We chose μ = o = 0.2 and generated time-augmented paths of length 64 over the grid A = {0. 1,......,66 63) with di = 0.01. Thus our dataset is given by time-augmented paths y [0, 63) - R2 embedded in path space via linear interpolation. For all three discrininators, the training and test set were both comprised of 32768 paths and the batch size was chosen to be N 128. We trained the SDE-GAN for 5000 steps, osig for 4000 and &N for 10000 steps. Table I ggives the KS scores along

---

each of the specified marginals, along with the percentage Type I error. Here the generator trained with Osg performs the best, achieving Type error at the assumed confidence level.
Model
= 6
t t= 19
= 32
= 44
t t= 57
SDE-GAN
0.1641, 41.1%
0.1094, 5.2%
0.1421, 24.2%
0.1104, 5.9%
0.1427, 26.2%
Osg (N 3)
0.1298, 15.4%
0.1277,
16.1%
0.1536, 37.4%
0.2101, 78.8%
0.2416, 92.3%
Pug (ours)
0.1071, 5.0%
0.1084, 6.0%
0.1086, 5.9
0.1089, 5.8%
0.1075,5.5%
Table 1: KS test average scores and Type errors marginals on gBm.
4.2 Rough Bergomi volatility model
is well-known that the benchmark model ( oversimplifies market reality. More complex models, (rough) stochastic volatility (SV) were introduced in the past decades, that are able to capture relevant properties of market data are used by financial practitioners to price and hedge derivatives. Prominent examples of stochastic volatility mdoels include the Heston and SABR models [HKLW02] HLWI5 Hes93]. State-of- ofte-oof-the-at models in this context have been introduced in [GJR18], (8. They display a stochastic volatility with rough sample paths. Most notable among these for pricing and hedging is the rough Bergomi (rBergomi) model [BFG16) which is of the form
dy: = Vidt + VVdWW where de," = 2a + I(u I(u- 1))
(10)
2
and where Ei is the instantancous forward variance for time time with & V,, and
H - -1/2 where H is the Hurst exponent. The parameter set is given by (1, H) with initial conditions Xo- x and ş - fo. It has been a well-known headache for modellers that desitee their many modelling advantages rough volatility models (such as (IO) are slow to to simulate with traditional methods. We demonstrate how our method can be used to capture the dynamics of the rough Bergomi model (IO), and in passing we also note that our method provides a significant simulation speedup for (IO) compared to previously available simulation methods.
Model
1 1=6
L= 19
L= 32
= 44
1 57
SDE-GAN
0.1929, 68.3%
0.2244,
86.2%
0.2273,
87.0%
0.2205, 83.4%
0.1949, 68.7%
0 (NV
0.1126, 8.1%
0.1172,
10.1%
0.1146, 8.2%
0.1153, 8.5%%
0.1134, 7.0%
Dug (ours)
0.1086, 5.4%
0.1129,555 5.9%
0.1118, 5.2%
0.1127, 6.2%
0.1159, 6.9%
Table 2: KS test average scores and Type errors on marginals on rBergomi model
To do so, we simulate paths of length 64 over the time window to [0, 2]. and specify di = 1/32. Thus paths are of length 64. The parameters are (Eo. 1), p, H) (0.04, -0.7, 0.2) and set d = 1. Paths are again aime-augmented. The hyperparameters for training are the same as the previous section. The results on the marginal distributions are summarized in Table [] We see that that training with respect to osg vastly outperforms the other two discriminators.
4.3 Foreign exchange currency pairs
We consider an example where samples from the data measure Px are time-augmented paths y: [0, T R33 corresponding to hourly market close prices of the currency pairs EUR/USD and USD/JPY To deal with irregular sampling, we linearly interpolate each sample y over fixed grid A = (to. .6.)) Training hyperparameters were kept the same as per the rBergomi example: paths are comprised of 64 observations, the batch size was taken to be N = 128, and the number of training epochs was taken to be 10000 for the SDE--AN, 4000 for diig and 5500 for &N KS scores for each of the marginals are given in Table 3 and 4 with Dsig is able to achieve strong performance on nearly all marginals.
We note that only the generator trained
'Dta is obtained from httpp://wwww.uuuaaoooo./.co///ss

---

Model
1
19
1
32
1 44
57
SDE-GAN
0.1889, 62.9%
0.2760, 98.2%
0.3324, 99.9%
0.3781, 100.0%
0.4209, 100.0%
Oug (NV 5)
0.1098, 4.2%
0.1279, 12.0%
0.1399. 18.7%
0.1507, 28.1%
0.1608, 37.5%
Usg (ours)
0.1270, 12.8%
0.1085, 5.2%
0.1060, 4.3%
0.1065, 5.1
0.1049, 4.0%
Table 3: KS test average scores marginals (EUR/USD)
Model
t = 6
t 19
= 32
44
t 57
SDE-GAN
0.140M, 20.5%
0.1665, 44.2%
0.1771, 56.4%
0.1855, 63.8%
0.1948, 70.3%
N Oug (N 5)
0. 0.1666, 43.8%
0.1877, 72.1%
0.2008, 84.7%
0.2154, 93.2%
0.231 98.3%
Phgg (ours)
0.1189, 9.2%
0.1121, 5.8%
0.1069, 4.9 %
0.1075, 3.8% %
0.1051,
Table 4: KS test average scores marginals (USD/JPY).
We also present a histogram of sample correlations between generated EUR/USD and USD/JPY paths for each of the three discriminators alongside those from the data distribution. From Figure T it appears that only the Neural SDE trained with dsig correctly identifies the negative correlative structure between the two pairs. This is likely due to the fact that these dependencies are encoded in higher order terms of the signature that the truncated method does capture.
1.2
P Xx
2,5
F
Fxxe
Fxx
1.0
2.0
1.5
0.3
1.5
0.5
1.0
1.0
0.4
0.5
0,0
0.0
Figure 1: Histogram correlation coefficients between EURUSD and USDJPY pairs, 1024 samples.
We consider conditional generation problem, where the conditioning variables time- augmented paths Q ~ x [4o - dt, |n] + R² representing the trajectory of prior di = 32 observations
of EUR/USID 15-minute close prices, and the target distribution is X [Lo, to dt'l R
representing the following dt' = 16 observations. Given batched samples {x* "'' N [")= and y² P x (-Ixr',, we train our generator according to equation 7 We encoded the conditioning paths via the truncated (log)signaturee of order 5, and fed these values into each of the neural networks of the Neural SDE. In Figure 2 it is evident that the conditional generator exhibits the capability to produce conditional distributions that frequently encompass the observed path. Furthermore, is noteworthy that these generated distributions capture certain distinctive characteristics of financial markets, such as martingality, mean reversion, or leverage effects when applicable.
where x² ~ Q
4.4 Simulation of limit order books
Here, we consider the task of simulating the dynamics limit order book (LOB), that is, an electronic record of all the outstanding orders for financial asset, representing its supply and demand over time. Simulating LOB dynamics
important challenge quantitative finance and several syn- thetic market generators have been proposed [...I +2] (VBP+20) (SCC21 /SCC2T)/CCC+ [CPC+21) 21, CMVB22) order o = (lo, No: vi)) submitted at time to with price x, and size n, >> (resp., v < 0) is commitment sell (resp., buy) up to lvol units of the traded asset a price no less (resp., greater)

---

3
Pyd((p)
New 2xooo (x)
--2
Fae((k))
Px-(-1r)
-2
-3
Py
40
Figure Given conditioning path Q. the generator provides (in blue) the conditional distribution xx (-]z). The dotted line gives the true path ~ IP x (-(r)..)....
than To. Various events are tracked (e.g. new orders, executions, and cancellations) and the LOB B(t) is the of all active orders in market time t. While prior work typically fit generator that produces the next event, and run iteratively to generate sequence of events, we propose to model directly the spatiotemporal process (1))) Boe Vo. To generate LOB trajectories,
the Neural SPDE model and train by minimising expected spatiotemporal kernel scores constructed by composing the signature kernel kaig with 3 different SE-T type kernels introduced in [WD22), namely the ID, SQR and CEXP kernels. We fit model on real LOB data from the NASDAQ public exchange İNMK 18) which consists of about 4.M timestamped events with L = 10 price levels. We split this LOB trace into subetraces of size T 30 to construct our dataset. On Figure report the average KS scores for each of the L x T marginals, using the 3 different kernel scores.
1D
SQB
CEXP
021
12
16
0.10
timee
time
time
Figure 3: KS test average scores for each spatiotemporal marginal, 100 rums, NASDAQ data.
Conclusion and future work
This work showcases the utilization of Neural SDEs generative model, highlighting their advantages over competitor models terms of simplicity and stability, particularly via non-adversarial training. Additionally, show how Neural SDEs exhibit the ability be conditioned diverse and intricate data structures, surpassing the capabilities of existing competitor works. We have achieved this by introducing the signature kernel score on paths and by showing their applicability to our setting (by proving strict properness). Performance of our methods are given computational time and memory competitive with state-of-the- theeof-ttt-rrrtt art methods. Moreover, we have shown that this approach extends to the generation of spatiotemporal signals, which has multiple applications in finance including limit order data generation. Further extensions of this work may include extending its generality to include jump processes in the driving noise of the approximator process (Neural SDEs) used. On the theoretical level extensions may include the validity of results paths with lower regularity than currently considered. Although sample paths from a Stratonovich SDE are not of bounded variation almost surely, sample paths generated by an SDE solver, once interpolated, are piecewise linear, and hence of bounded variation. A similar point can be made about compactness of the support of the measures.
possible ensure characteristicness of the signature kernel on non-compact sets of less regular paths using limiting arguments and changing the underlying

---

topology pathspace. Further extensions for practical applications can (and should) include the inclusion of more varied evaluation metrics and processes. Notably, in later step, the generated data should be tested by assessing whether existing risk management frameworks and investment engines can be improved when data used for backtesting augmented with synthetic samples provided by our methods. Furthermore, the spatiotemporal results can be extended to more complex structures, including being used for the synthetic generation of implied volatility surface dynamics, which has been a notoriously difficult modelling problem in past decades.
Acknowledgements
ML was supported by the EPSRC grant EP/S026347/1. ZI was supported by EPSRC grant EP/R513064/1.
References
[ACB17]
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adver- sarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[ASS20]
Imanol Perez Arribas, Cristopher Salvi, and Lukasz Szpruch. Sig-sdes model for quantitative finance. In ACM International Conference on Al in Finance, 2020.
[BCR84]
Christian Berg, Jens Peter Reus Christensen, and Paul Ressel. Harmonic analysis on semigroups: theory of positive definite and related functions, volume 100. Springer, 1984.
[BFG16]
Christian Bayer, Peter Friz, and Jim Gatheral. Pricing under rough volatility. Quanti- tative Finance, 16(6):887- 90488899944420 2016.
[BMN16]
Diane Bouchacourt, Pawan K Mudigonda, and Sebastian Nowozin. Disco nets: Dissimilarity coefficients networks. Advances Neural Information Processing Systems, 29, 2016.
[BO21]
Patric Bonnier and Harald Oberhauser. Proper scoring rules, gradients, divergences, and entropies for paths and time series. arXiv preprint arXiv:2111. 1.06614, 2021.
[CFC+21]
Thomas Cochrane, Peter Foster, Varun Chhabra, Maud Lemercier, Cristopher Salvi, and Terry Lyons. Sk-tree: systematic malware detection algorithm streaming trees via the signature kernel. arXiv preprint (.Xiv:2002.079000 2021.
[CJB23]
Vedant Choudhary, Sebastian Jaimungal, and Maxime Bergeron. Funvol: A multi- asset implied volatility market simulator using functional principal components and neural sdes. arXiv preprint arXiv:23033.0085 2023.
[CLS23] Nicola Muca Cirone, Maud Lemercier, and Cristopher Salvi. Neural signature kernlss as wehphi-et-hthhhhet----iti limits controlled resnets. arXiv preprint urXiv:2303.17677 2023.
[CLX21]
Thomas Cass, Terry Lyons, and Xingcheng Xu. General signature kernels, 2021.
[CM21]
Rama Cont and Marvin S Müller. A stochastic partial differential equation model for limit order book dynamics. SIAM Journal on Financial Mathematics, 12(2):744-787, 2021.
[CMVB22]
Andrea Coletta, Aymeric Moulin, Svitlana Vyetrenko, and Tucker Balch. Learning to simulate realistic limit order book markets from data world agent. In Proceedings of the Third ACM International Conference on in Finance, pages 428-442-54 2022.
[CPC+21]
Andrea Coletta, Matteo Prata, Michele Conti, Emanuele Mercanti, Novella Bartolini, Aymeric Moulin, Svitlana Vyetrenko, and Tucker Balch. Towards realistic market simulations: a generative adversarial networks approach. In Proceedings of the Second ACM International Conference Al in Finance, pages 1-9, 2021.
[CT22] Thomas Cass and William F Turner. Topologies on unparameterised path space. arXiv preprint arXiv:2206.1 1153, 2022.
10

---

[CZZ*20] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2020.
[DCB 20] Ruizhi Deng, Bo Chang, Marcus A Brubaker, Greg Mori. and Andreas Lehrmann. Modeling continuous stochastic processes with dynamic normalizing flows. Advances in Neural Information Processing Systems, 15378855-7815. 2020.
[FLMS23]
Adeline Fermanian, Terry Lyons, James Morrill, and Cristopher Salvi. New directions the applications of rough path theory. IEEE BITS the Information Theory Magazine,
2023.
[GBR+12)
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
[GJR18]
Jim Gatheral, Thibault Jaisson, and Mathieu Rosenbaum. Volatility is rough. Quanti- tative finance, 18(6):933 90099339999..0 2018.
[GPW* 13]
Martin D Gould, Mason A Porter, Stacy Williams, Mark McDonald, Daniel J Fenn, and Sam D Howison. Limit order books. Quantitative Finance, (((1)):709-1177 2013.
[GRO5]
Tilmann Gneiting and Adrian Raftery. Weather forecasting with ensemble methods. Science, 310(5746):248 249, 2005.
[GRO7]
Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378 2007.
[GSvdB +20
Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems, 33:1 3062-13777 2020.
(GSV$+20 20]
Patryk Gierjatowicz, Mare Sabate- Vidales, David Šiška, Lukasz Szpruch, and Zan Zurič. Robust pricing and hedging via neural sdes. arXiv preprint arXiv:2007.04154 2020.
[Hes93]
Steven L Heston. A closed-form solution for options with stochastic volatility with applications to bond and currency options. The review of financial studies, 6(2y:27~ 343, 1993.
[HFH+23]
Melker Höglund, Emilio Ferrucci, Camilo Hernández, Aitor Muguruza Gonzalez, Cristopher Salvi, Leandro Bettchez-ettncouut,, and Yufci Zhang. Solving and learning mon-markovian stochastic control problems in continuous-time with neural RDEs, 2023.
[HKLW02]
Patrick Hagan, Deep Kumar, Andrew Lesniewski, and Diana Woodward. Managing smile risk. The Best of Wilmost, 1:249-296, 2002.
[HKN20]
Ben Hambly, Jasdcep Kalsi, and James Newbury. Limit order books, diffusson ap- proximations and reflected spdes: from microscopic to macroscopic models. Applied Mathematical Finance, 132-1177,, 2020.
[HLL+23]
Blanka Horvath, Maud Lemercier, Chong Liu, Terry Lyons, and Cristopher Salvi. Optimal stopping via distribution regression: a higher rank signature approach. arXiv preprint
2023.
[HLW15]
Patrick Hagan, Andrew Lesniewski, and Diana Woodward. Probability distribution in the sabr model of stochastic volatility. Large deviations and asymptotic methods in finance, pages 1-35. Springer, 2015.
[HvdHRM20]
Liam Hodgkinson, Chris van der Heide, Fred Roosta, and Michael W Mahoney, Stochastic normalizing flows. arXiv preprint urXiv:2002.09547 2020.
[JB19]
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
IKBPA+ 19]
Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. Advances Neural Information Processing Systems, 32, 2019.

---

[KFL*21]
Patrick Kidger, James Foster, Xuechen Li, Harald Oberhauser, and Terry Lyons. Neural sdes defiite-dimensional gans. arXiv preprint arXiv:2002.03657 2021.
[Kid22]
Patrick Kidger. On neural differential equations. arXiv preprini arXiv:220..02335 2022.
[KMFL20]
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv preprint WrXiv:2005.08920 2020.
[KP
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020.
[LLJ16]
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In International conference on machine learning, pages 276- 284. PMLR, 2016.
[LSC+21] Maud Lemereier, Cristopher Salvi, Thomas Cass, Edwin V Bonilla, Theodoros Damoulas, and Terry J Lyons. Siggpde: Scaling sparse gaussian processes on se- quential data. In International Conference Machine Learning, pages 6233 0233-6242.
PMLR, 2021.
[LSD+21]] Maud Lemercier, Cristopher Salvi, Theodoros Damoulas, Edwin Bonilla, and Terry Lyons. Distribution regression for sequential data. In International Conference on Artificial Intelligence and Statistics, pages 3754-3762. PMLR, 2021.
[LSZ15]
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference on machine learming, pages 1718-1727. PMLR, 2015.
[LWCD20]
Xuechen Li, Ting-Kan Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients and variational inference for stochastic differential equations. In Symposium on Advances in Approximate Bayesian Inference, pages 1-28. PMLR, 2020.
[LWL-20) 20]
Junyi Li, Xintong Wang. Yaoyang Lin, Arunesh Sinha, and Michael Wellman. Gener- ating realistic stock market order streams. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01)::27-734, 2020.
[Lyo98]
Terry J Lyons. Differential equations driven by rough signals. Revista Matemática Iberoamericana, 14(2):215-310, 1998.
[MKYL22] James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons. On the choice of interpolation scheme for neural cdes. Transactions on Machine Learning Research, 2022(9), 2022.
[MS13]
Edgar C Merkle and Mark Steyvers. Choosing strictly proper scoring rule. Decision Analysis, 10(4):292-304, 2013.
[MSK+20] James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neu- ral rough differential equations for long time series. arXiv preprint 1xXiv:2009.08295, 2020.
[MSKF21] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages 7829-7838. PMLR, 2021.
[NMK+18] Adamantios Ntakaris, Martin Magris, Juho Kanniainen, Moncef Gabbouj. and Alexan- dros losifidis. Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods. Journal of Forecasting, 37(8)852-8866, 2018.
INSSV+21] Hao Ni, Lukasz Szpruch, Mare Sabate-Vidales, Vidales, Baoren Xiao, Magnus Wiese, and Shujian Liao. Sig-wasserstein gans for time series generation. In Proceedings of the Second ACM International Conference on Al in Finance, pages 1-8, 2021.
INSW
Hao Ni, Lukasz Szpruch, Magnus Wiese, Shujian Liao, and Baoren Xiao. Conditional wagsesseestnin gans for time series generation. arXiv preprint arXiv:2006.0542) 2020.
[Opp19]
Manfred Opper. Variational inference for stochastic differential equations. Annalen der Physik, 53103): 2019.
12

---

[PADD21]
Lorenzo Pacchiardi, Rilwan Adewoyin. Peter Dueben, and Ritabrata Dutta. Probabilis- tic forecasting with conditional generative networks via scoring rule minimization. arXiv preprint (2.k:22/22002217. 2021.
[PD22]
Lorenzo Pacchiardi and Ritabrata Dutta. Likelihood-free inference with generative neural networks via scoring rule minimization. arXiv preprint arXiv:2205. 15784, 2022.
[SCC21] Zijian Shi, Yu Chen, and John Carllidge. The lob recreation model: Predicting the limit order book from taq history using ordinary differential equation recurrent neural network. Proceedings of the AAAI Conference on Artificial Intelligence, 55(1):548-556 2021.
[SCF+21] Cristopher Salvi, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang. The signature kernel is the solution of a goursat pde. SIAM Journal on Mathematics of Data Science, 3(3):873-899, 2021.
[SGS18] Carl-Johann Simon-Gabric and Bernhard Schölkopf. Kernel distribution embeddings: Universal kernels, characteristic kernels and kernel metries on distributions. The Journal of Machine Learning Research, 19(1):1708-177) 2018.
[SLG22]
Cristopher Salvi, Maud Lemercier, and Andris Gerasimovics. Neural stochastic pdes: Resolution-invariant learning of continuous spatiotemporal dynamics. Advances in Neural Information Processing Systems, 35:1 15:1333-1344 2022.
[S]L+21] Cristopher Salvi, Maud Lemercier, Chong Liu, Blanka Hovarth, Theodoros Damoulas, and Terry Lyons. Higher order kernel mean embeddings to capture filtrations of stochastic processes. arXiv preprint arXiv:2109.03582, 2021.
[Smi39]
Nikolai V Smirovv. On the estimation of the discrepancy between empirical curves of distribution for two independent samples. Bull. Math. Univ. Moscou, 2-2):3-14, 1939.
[SSDK+20] 20]
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
[TR19]
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint /r0iiv/905.000 1995509883, 2019.
[VBP+20]
Svitlana Vyetrenko, David Byrd, Nick Petosa, Mahmoud Mahfouz, Danial Dervovic, Manuela Veloso, and Tucker Balch. Get real: Realism metrics for robust limit order book market simulations. In Proceedings of the First ACM International Conference on Al in Finance, pages 1-8, 2020.
[VKK21]
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances Neural Information Processing Systems, 34:1 1287-11302. 2021.
[WD22]
George Wynne and Andrew B Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research, 23(73):1-51, 2022.
[YJVdS19]
Jinsung Yoon, Daniel Jarrett, and Mihacla Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
13

---

A Signature Kernel Scores
Proof of Proposition 3.3
Proof (Appendix). The general result was first shown in |RO7 first show that Osig is proper.
By Proposition 3.1 the signature kernel is positive definite and characteristic P(C). It remains to show that E Psig Q v)] I Qy y)]. This means we must have
Eg [Rig (x, v)1 - Be Bag (2,, x') By. [Rig (v. v/)]
Writing M = 20, modification of Theorem 2.1 in Berg al. [BCR84] (pg. 235) gives that
/ ksig(I, 3) d(P & (z, s
Kag(a, v) d(M @ (a) v),
(11)
where P & Q denotes the natural product measure on K. Re-arranging one arrives the desired result.
To show strict properness, need show that Eg-- (Aag (Q y)] Ey-olosig (TP, .y] holds with equality iff PP Q for all P, Q E P(C)). Suppose that there exists another IP" E P(K.) such that Ey-olost Q y)] Ey~ (ssi (P", 3)]. Then would have that
Eg - (a, 2B Beee:r.yrllllaeee Sagg (z. y)] + Dhag =,
or that Dio = 0, which is only true if IP = Q due to characteristicness of the kernel king-
Proof of Proposition 3.4
Proof (Appendix). The proof follows directly from [BR + 12), Lemma Note that an unbiased estimator for Ex Pkkig (I, r')] from i.i samples (I1, (Ii,..., Im); x2 ~ P is given by the U-statistic
T (ZI..... Fm)) m(m 1)
Kaig (1x,, 2z)). Σ i
Moreover, an unbiased estimate of Ex~F lesg (xt, | is given by
1 m
m
(1 Im. v)
Ksig (122, y)
Writing dag (I,,))
-2T3 2T Imm 3) completes the proof.
B Experiments
All experiments were run on a NVIDIA GeForce RTX 3070 Ti GPU, except the experiment in Section 4.4 for which the NSPDE model was trained using NVIDIA A100 40GB GPU.
Here we provide details for each of the experiments outlined in the body of the paper. also provide some extra methods of evaluation aside from the KS test. These include the following:
1. Qualitative plot: We give plot of samples Px. from a trained generator against the true data measure Xxee.
2. Autocorrelation: To measure temporal dependencies or correlations, we leverage the autocorrelation function
N No2 [x, (M))
ACF2
where is the average of the path X, over [0, N] and o² is the corresponding variance. We provide qualitative plot of ACF, for each generator against the real data measure. We also provide table summarizing the scores for some of the carliest lags N.
14

---

3. Cross-correlation: We provide average cross-correlation scores (re.r( between the returns process associated to X₁ Pxx x and the squared, lagged returns process r We present the scores in matrix form. Finally, we provide the MSE between the matrix obtained from Px and those obtained from each generator.
We make a note here that each of the three discriminators performed similarly in the additional quantitative metrics omitted from the body. Finally, we wish to first make the following general notes about each of the three methods studied in this paper:
Speed: Training a Neural SDE with respect to dsg was the fastest, followed by the SDE- GAN, and finally with dsgg being the slowest. It was possible to decrease training time respect to the latter by using a coarser dyadic refinement in the PDE solver. However, we felt that achieving more accurate gradients the cost of longer training time was worthwhile.
Stability: The Wasserstein SDE-GAN was the least stable, in terms of the difficulty in obtaining a training instance where the loss converged in reasonable time. Even with fine- tniig of both generator and discriminator parameters, the loss associated to the SDE-GAN tended to oscillate, making obtaining a converged model a very difficult task with the hardware available us.
Scaling: All of the results in the paper are sensitive to path scalings; moreso with the signature kernel-based approaches, less so with the Wasserstein approach. The basic idea is follows: the signature kernel-based methods will tend to fail if paths are scaled too low (resulting in lower-order terms dominating the calculation of ksig) or too high (the sum, although finite, can exceed a 64-bit float quite easily). Path scaling (and transformations) form an integral part in training a successful generative model, and we have tried to be as descriptive as possible regarding this matter. The details as to why scalings matter have been touched upon in [CLX21]] we intend to expand upon this in future work.
On a similar note, standardizing path data before training was often found improve the stability of training in any setting. By standardization we are referring to transforming each marginal of paths X Pxwe via the transformation X. (X; HT)/ where WTT = Ep y [Xr] and oT Ep EEpxme I )+) By having the terminal marginal distributed standard normal, the task of finding suitable path scalings and smoothing parameters in the RBF kernel was made much simpler, as this task became less problem- specific.
Standardisation:
B.1 Geometric Brownian motion
Data processing and hyperparameters
To generate our data measure, we simulate 32768 paths according to eq. 9 using the torchsde package. These were solved over the interval [0, 64 by setting y/ 1 0, 0.2, with di 0.1. Paths were then interpolated along the grid A = {0, 2, 63). so each element of the training set had total length 64. Stochastic integrals taken in the Itô sense and the driving noise W was taken in the general sense. We used the SRK method to solve the corresponding SDE. Each path is time-augmented, so X, (t, X,) each point on the grid, After we have simulated our dataset, we standardized each path as outlined in the dot points above.
Generator hyperparameters The generator is Neural SDE with vector fields μo [0. 7] x R - Rº and og [0, T) x R2x= + RR" taken to be neural networks with hidden layer, and 16 neurons in said layer. As per 221 the LipSwish activation function was used to ensure the Lipschitz condition held on the vector fields of the Neural SDE. We also used the final tanh regularisation which we found was necessary for training success. Thus we have that
Pe, E N1 16, 1, LipSwish, tanh).
The size of the hidden state of the neural SDE was chosen to be 8, and the noise dimension was chosen to be w 3. Stochastic integration was taken in the sense and we set = 1 over [0, 63). As we are not learning an initial distribution in this instance, we modified the generator architecture to have &o (x)) = a for some a E R. where Ça is the network acting on the initial condition. Before passing to the discriminator, both generated and real paths were translated to start at 0.
15

---

Discriminator hyperparameters For the signature kernel- baaee discriminators, we applied the time normalisation transformation so the time component both the real and generated paths was over [0, I] as opposed to [0, 63). This was to ensure each channel of the generated and real data evolved over similar scale. For training with respect to Dsgg we set the order of dyadie refinement associated to the PDE solver for the signature kernel to 1. We also used three different kernels, corresponding to three different scalings of the paths, for increased expressivity. For Asgg we set the order of truncation equal N 3. Finally, for the SDE-GAN GAN, we chose the drift and diffusion vector fields to be feed-forwart neural networks
for 9s (V, 16, 1, LipSwish, tanh),
matching that from the generator.
Training hyperparameters All methods used batch size of 128 and the Adam optimisation algorithm for backpropagating through the generator optimisers, except for the SDE-GAN, which as suggested by the authors used Adadelta. As remark, did not much difference in using either Adam, Adadelta, or RMSProp. although we did see poorer performance using pure SGD, with or without momentum. Learning rates were roughly proportional the average size of the batched loss: as rough guide, proportionality like J|G x (()) R 10-S tended to yield good results, with the generator learning rate being around IJa 10-4 for the signature kernel(s), and Ai IJQ 10-2 for the SDE-GAN. As mentioned in the body, we trained for 4000 steps with Asgg 10000 with &e and 5000 with the SDE-GAN to normalise for training time.
Results
We begin with qualitative plot of the results from each generator.
Ex
Fx
Px*
23
10 60
6
30
Figure 4: Qualitative plot of generator output versus data measure. Trained with (from left to right):
Bsig. Sig SDE-GAN
Table 5 gives the autocorrelation scores for the first five lags for each of the three models, along with plots of the mean ACF values in Figure Slnd and the associated 95% confidence intervals. We can see that all three models do well at capturing temporal effects, with the Neural SDE trained with respect to Ósig most closely matching the data measure, except in the first lag.
Discriminator
Lags
[
[
[ 3
SDE-GAN
0.887
0.120
0.782 0.211
0.686 0.282
0.597 0.342
0.515 + 0.384
Qug
3)
0.883
0.115
0.781 0.197
0.684 0.261
0.594 0.320
0.514 0.364
Dsig
0.886
0.111
0.785 0.199
0.696 0.267
0.612 0.315
0.535 + 0.350
Data measure
0.892 1 0.5
0.793 + 0.183
0.702 1 0.258
0.616 1 0.319
0.532 1 0.374
Table Sample autocorrelation scores, gBm
Finally, we present the cross-correlation matrices between the returns process r, and the lagged squared returns process r²_, for lags / {0, 1, 1, 3,445,55 4,5). Again all models tend to perform quite well in that they match relational dynamics observed in the data measure. Table 6 gives the MSE between the generated matrices and the data matrix. We see that the Neural SDE trained with ésig
16

---

1,0
10:
1.0
0,8
0.0
ACR:
0.4
044
0,3
00
Pm
-07
2yn
Pyw
-04 Ext
Ext
Fyt
-0.4
-0.4
Figure 5: Qualitative plot of ACF scores, generator output versus data measure. Trained with (from left to right): Usig D SDE-GAN
achieves the lowest score of the three, however again performance is strong regardless of method used for training.
274
0.ES4
0.017
0.674
0.110
0.042
0.06)
11:03
0194
9.034
0.635
0.048
0.003
0.841
IL642
ILESS
0.044
0.681
0.089
0.07
0.147
0100
0:83
9.04
0.61
0.635
0.077
0.07
0:877
0.643
0.840
0.062
(a) Data measure.
(b) Osig
(c) P
(d) SDE-GAN.
Figure 6: Cross-correlation matrices, gBm
Discriminator
MSE
SDE-GAN
0.01 0.014688
Ogg (N = 3)
0.066745
Qsig
0.010718
Table 6: MSE between cross-correlation matrices, gBm
B.2 Rough Bergomi
Data processing and hyperparameters We simulate 32768 paths to make up our data measure via the rBergomi Python packageF We fixed the time window to be [0, 2]. and specified dt 1/32, so paths were of length 64. We chose (Eu, 7) H) = (0.04, 1.5, 0.7, 0.2) and d 1. Paths started Paths were normalised to start at O via translation
at 1. As always, paths were Lime-augmente and were standardized again according the terminal data from the train set. A final point that although the data was generated over [0, 2], the time grid passed to the generators in an optimiser step was
63). We found that this improved performance.
Generator hyperparameters Given the increased complexity of the data generating model, we increased the expressivity of the vector fields governing the drift and diffusion vector fields μı and ag. This was done by increasing the depth and width of the constituent feed-forward networks to include 3 hidden layers of size 32. We also increased the size of the hidden state to 16 and the
17

---

noise dimension to w = 8. Thus
μa E (17,32,32,, 32, 16; LipSwish, LipSwish, Linpsiss,,,iSSissll tanh)
and
do E (2,,3322,,2, 32, 128; LipSwish, LipSwish, LipSwish, tanh).
Discriminator hy pyerparaaeeerss For training with respect to @sig. mapped path state values to (H, k) where denotes the RBF kernel R², We set associated the smoothing parameter For d we increased the truncation level to N = 5. In both these settings we again applied the time normalisation transformation on both the generated and data measure paths before being passed through the loss function. For the SDE-GAN, we increased the expressiveness of the vector fields governing the Neural CDE in the same way as we did the Neural SDE.
1.
Training hyperparameters Learning rates for the Neural SDE trained according to Osig was set to = x 10-4. Due to the increasing number of terms in the expected signature for the truncated MMD approach, we had to reduce the learning rate to TJG 1 10 larger values caused instability in the training procedure. The SDE-GAI was again quite difficult to train, however we were able have some success by setting TD R 2 x 10-3 and TJG
10 Initialisation of the generator vector fields was especially important for the Wasserstein method, as initialisation too far from the data measure cause oscillatory patterns in the training loss, which leads more epochs required for the loss to converge. We again used the Adam optimisation algorithm for the MMD-based discriminator/geenratooo and Adadelta for the SDE-GAN, We trained for the same number of steps as the gBm method.
Results
Figure ggives qualitative plot of the simulated paths.
F
Px
Exz
Fxi
-2
Pxoe
-2
Exr
20
10
60
Figure 7: Qualitative plot of generator output versus data measure. Trained with (from left to right): Øsig. & SDE-GAN
Figure gives the same plot of the ACF scores corresponding lags for the data measure and each of the generated models, along with the 95% confidence interval. Table 7 explicitly gives these scores.
1.0
1.0
0.5
08-
0.5
0.6-
3Ch
0.4
300
0.2
0.2-
0.2
0.0
-02
Prte
-07
2y
Fy
Pys
Ext
Pyr
-0.4
-6.4
0
10
10
Figure 8: Qualitative plot of ACF scores, generator output versus data measure. Trained with (from left to right): @sig N SDE-GAN
18

---

Discriminator
Lags
1=
1=
1=
1=
SDE-GAN
0.893 + 0.105
0.795 + 0.191
0.705 + 0.262
0.622 + 0.319
0.543 + 0.377
Oug (N 5)
0.897 + 0.115
0.799 + 0.208
0.710 + 0.289
0.628 1 0.338
0.546
0.383
Osig
0.890 + 0.115
0.790 + 0.203
0.702 + 0.269
0.618 + 0.316
0.521 + 0.382
Data measure
0.885 1 0.121
0.778 + 0.215
0.685 1 0.278
0.600 1 0.336
0.521 1 0.380
Table Sample autocorrelation scores, rBergomi
Finally, we present the same cross-correlation matrices, along with the MSE between either of the three generators and the data measure. Although each of the models perform well, the generator trained with dsgg achieves the best results.
242
0.007
0.304
2X5XE 0LE79
0.006
0.074
0.692
0.06)
01.75
0.85
0.657
0.006
0.002
0.864
0LESS
0.30
0.844
0LE55
(LE7S
0.556
0:84
0.021
0.84
0LESS
(a) Data measure.
(b) Osig
N (c)
(d) SDE-GAN
Figure Cross-correlation matrices, rBergomi
Discriminator
MSE
SDE GAN
0.091707
Pig (N
5)
0.054731
Qsig
0.016785
Table 8: MSE between cross-correlation matrices, rBergomi
B.3 Multidimensional real data
We now give the details regarding the unconditional generation of foreign exchange data.
Data processing and hyperparameters Data given by hourly returns associated to the currency pairs EUR/USD and USD/IPY. We stride the concatenated time series (corresponding to close prices each time) into paths of length 64. Paths are normalized to start values with their original timestamps, (as epoch seconds). Call this dataset V. As are dealing with financial data, one can expect the time intervals between prices to be irregular. This not usually an issue when using signature methods. However, in this setting we found training to be less stable if paths were not normalised to evolve over the same time grid as that which the neural SDE was solved over in forward pass.
We augmented the state
To circumvent this issue, for every E ) we find the median terminal time 7; where
t Median, (t/t)))
All paths whose terminal time is greater than were filtered out of the dataset which we call 5 We then define an evenly-spaced time grid A .O...., TT containing 64 observations in total, and linearly interpolate each y over this grid, where use these interpolated coefficients to
19

---

form our train and test sets. Again we standardize using the terminal values of the train set data. We simulate our generators over the time grid A (o..... .63 as we found that using A the realistic time-grid (in fractions of year) induced little variability in the generated paths; i.e., the quadratic rBergomi section. We tried increasing the size of the hidden state = 32 and the noise state = 16 but found that this had little impact on training performance. We also found that increasing the expressivity of the neural vector fields did not overly impact performance; neither refining the mesh over which the Neural SDE was solved.
variation associated to the generated paths was significantly lower than that obtained from the real data measure.
Generator hyperparameters The generator maintains the same architecture as outlined in the
Discriminator hyperaarameters We used the same discriminator hyperparameters for each of the three methods as per the rBergomi section.
Training hyperparaaeters We used the same batch size (128) as per the previous sections. We allowed for increased training time here, training with respect @sig for 4000 steps, 15000 for & and 10000 for the SDE-GAN. The same learning rate parameters were used as well. We did not use any learning rate annealers. The Adam optimisation algorithm was employed for the MMD-based generators, whereas again Adadelta was used for the Wasserstein case.
Results Extended results are provided as per the previous sections. We being with the qualitative plots.
Py
Pxt
-2
-5
(a) EUR/USD
Pype Ext
Pgue Px
X
-2
10
20
10
60
(b) USD/JPY
Figure 10: Qualitative plot generator output versus data measure. Trained with (from left to right): Bsg. 2 SDE-GAN
Visual inspection gives that the generator trained with respect to dsig appears to have most accurately captured the data measure, in particular the less regular, outlier paths. In contrast the SDE-GAV and truncated kernel methods tend to over-represent the mean element. For the GAN this could be the "mode collapse" phenomenon in effect, whereas in the case of the Neural SDE trained with & it is likely that higher-order terms cannot be discarded if one wishes accurately model the data measure.
20

---

We now provide the plot associated to the ACF scores obtained from training with respect to each generator, along with the summarizing table. Table Table shows that that each of the discriminators perform relatively well, aside from the EURUSD autocorrelative factors obtained via traning the Neural SDE in the SDE-GAN framework. Finally we give the cross-correlation matrices and the associated MSEs. The Neural SDE trained with respect to @sig appears to perform the best by this evaluation metric.
1.0
1.0
10
0.6
0.5
0.6-
ADE:
ACP:
0.4
0.2
0.2
0.0
-02
Fx
-02
Pyme
Px
0.2
Px
-0.5
10
-0.4
10
(a) EUR/USD
1.0
1.0
10
0.8
0.5
0.5
0.3
ACW;
0.4
ACW:
0.4
0.8
0.
0.0
0.0
Exm
Pyn
0.2
PxP
Ex²
Pxt
Fx
0.4
-0.1
(b) USD/PPY
Figure 11: Qualitative plot of ACF scores, generator output versus data measure. Trained with (from left to right): Psig Pag. SDE-GAN
Discriminator
Lags 1=
1=
1 1=
1=
= 5
SDE-GAN
0.905 + 0.108
0.817 + 0.195
0.736 + 0.264
0.660 + 0.319
0.590 + 0.362
2 (N = 5)
0.889 1 0.123
0.788 1 0.215
0.695 1 0.289
0.610 1 0.345
0.532 1 0.388
Quig
0.890 + 0.123
0.790 + 0.218
0.699 + 0.291
0.615 + 0.347
0.538 + 0,388
Data measure
0.885 1 0.140
0.785 1 0.236
0.696 1 0.302
0.615 1 0.302
0.541 1: 0.387
(a) EUR/USD
Discriminator
Lags [=
1=1
1=
1=
1=
SDE-GAN
0.891 1 0.121
0.791 + 0.216
0.700 1 0.290
0.616 1 0.346
0.539
0.389
NN (N - 5)
0.887 + 0.123
0.785 + 0.218
0.692 + 0.292
0.607 + 0.348
0.529 + 0.390
Qsig
0.891 + 0.121
0.790 + 0.215
0.698 1 0.288
0.614 1. 0.344
0.537
0.385
Data measure
0.885 + 0.132
0.784 = 0.227
0.694 = 0.296
0.613 = 0.347
0.539 + 0.385
(b) USD/JPY
Table 9: Sample autocorrelation scores, forign exchange data
21

---

01064
0.000
XX03
0.00
0.001
EL041
0.071
0.642
0.64
0164
01644
ILERO
01854
ILESS
0.647
6:075
0.81
IUBSA
6.012
(a) EUR/USD
(b) Clig
(c) Oug
(d) SDE-GAN
0.06
0.67
01009
0.003
0.829
0.05
0.054
14:48
01008
0L677
0.007
0.E06
0.044
0.006
0L0K2
01:48
0.841
0.03
01E15
0.017
0.028
0.812
0LE45
0/9
(c) USD/IPY
(f) Mrgg-
NV
(h) SDE-GAN
Figure 12: Cross-correlation matrices, foreign exchange data
Discriminator
MSE
EUR/USD
USD/JPY
SDE-GAN
0.051728
0.027539
Agg (N 5)
0.045966
0.036628
Gaig
0.035791
0.003898
Table 10: MSE between cross-correlation matrices, foreign exchange data
B.4 Conditional generation
In this section we describe in detail the training procedure for the conditional generator.
Problem setting The conditioning variables are given by path segments : [tu dt, 4o] R2 representing time-augmented asset price values. At time tg, one wishes to make a prediction about the resultant path y [fo. tu dt'l R² conditional on æ. Here, dt, dt' are hyperparameters describing how much of the past one wishes to consider and how far into the future one wishes to forecast. With J Q We thus want to train conditional generator so that xxx (-1x) IP (-Ix)) We briefly state the three major difficulties associated to this generation problem:
1. Unobservable true conditional distribution. In practice, one never observes the entire true conditional distribution ( (-(a): only sample from it. This means that classical metrics on path space (MMD, Wasserstein, and so on) cannot be used without modification, or making assumptions about the relationship between the conditioning and resultant paths.
2. Using paths as conditioning variables. It is not immediately clear what is the best way to consume a path conditioning variable for a given generator.
3. "Unseen" conditioning variables. not guaranteed that the conditional generator will behave in expected way if an as-yet unseen conditioning variable is provided (by unseen, we are referring to within the training procedure). These conditioning variables are often the ones of interest.
22

---

Our procedure attempts to solve the first and second problems with our architecture choices on the conditional generator, and our choice loss function. The third issue is omnipresent in conditional modelling.
The generator now is given by (conditional) Neural SDE with architecture given by
Yo = =Eo C(x)), dY, ptlt, Yr, C(z))dt + Ct Y, C(x)) o dd,, X₁ (o(
for 140 |/, to x Rdc + R2 140, to+ x Rdde + Rex So R + Ry and my : 1R IRdc + R Here, C(x) denotes the function acting on the conditioning path and encoding it as a vector in Rdc, A natural way to perform this encoding is via the truncated signature S (X) of the path æ. In this way the neural networks defining the vector fields in the generator (for instance) are now mappings
Me [to to dt')
x R R
Bl+d+d
+d
R
σo [oo, to dt'I x R R 1+d+d
4d
RVx
We note here that all of the regularity conditions required to ensure a strong solution to the standard Neural SDE here remain satisfied; we are only augmenting each of the trainable components to accept the encoded conditioning path. We also note here that this technique flexible enough to include any amount of -vlued conditioning variables.
Data processing and hyperparameters Data comes from 15-minute close prices associated the EUR/USD price pair. We again extracted paths of length 48 (normalising for erroneous terminal times as per the unconditional setting) and split these paths into conditioning resullant pairs {a*, y Ji=1 with x' representing the first 32 observations and y' the next 16. We normalized both sets of paths by their initial value. Instead of standardizing, in this setting we scaled all path values up by a factor of 100. We found this was crucial so that the lower-order signature terms did not overly contribute to the value of the signature kernel. Both conditioning and resultant paths were then translated to start at 0. In total the dataset size was comprised of 52428 conditioning/resulant pairs.
Generator hyperparameters The generator is conditional Neural SDE. Stochastic integration was again taken in the Itô sense and we used the Euler method. The noise size was set to w = 8, the size of the hidden state was taken y 16. The MIPs governing the vector fields were the same as per the rBergomi and multidimensional unconditional examples, except we increased the width of the layers the neural networks to 64 neurons. We conditionalized the input paths via the truncated log-signature of order 5. In order to estimate the batched loss, we need to specify the size of the conditional distribution Fx (-Ix) output by the conditional generator, which set 32 paths. Finally, we applied the time normalisation and lead-lag transformations to the input paths æ before taking their truncated log-signa
Discriminator hyperparameters We trained with respect to ósig. Again we lifted paths via the RBF kernel and chose the smoothing parameter 1. We used dyadic refinement level of 1 for the PDE solver associated to kag- All paths had the time normalisation transformation applied to them before having the loss evaluated.
Training hyperparameters We set the batch size equal to 128 and trained the conditional generator for 10000 steps. We set the learning rate l/0 2 10-6 and used the Adam optimisation algorithm PyTorch. No learning rate annelears were used.
Results
Results are presented in the body of the paper.
B.5
Simulation of limit order books
The Neural SPDE model introduced in SLG22 extends Neural SDEs to model spatiotemporal dynamics by parametrising the differential operator, drift and diffusion of SPDEs of the type dY, (LY, + d())) (Y,)dWW,
(12)
23

---

where both and ø are local operators acting on the function Y, that is, (z) and (z)
only depend on (x). Moreover, it is assumed that L is linear differential operator generating semigroup etc which can be written as a convolution with a kernel Kr.
Let D C TR be bounded domain. Let W [0; - (D, ) be Wiener process and an (D, IR )-valued Gaussian random variable. The values du da I are hyperparameters describing the size of the noise. A Neural SPDE a model of the form
/ Kr-s (Ho(Ys) ooYY)) X/(x)) No (x)
Yo(x)) lo(a(x)), Y Ke Ke=
for 1 E [0. T] and x E D where Y [0, LD, R4y) is the mild solution, if it exists to the SPDE in Equation (12) with regularised driving noise W and where denotes the convolution in space with the kernel Dx D + Rdd, xd, (see [SLG22] for more details). Similarly to the Neural
SDE model,
lo Rd Rd4 14 RR R og Rd Rd, To R.d, - Rdd
are feedforward neural networks. Imposing globally Lipschitz conditions (by using ReLU or tanh activation functions in the neural networks μ7 and σe) ensures the existence and uniqueness of the mild solution Y. Finally, we note that in [SLG22]. the authors propose two distinct algorithms to evaluate the Neural SPDE model based two different parameterisations of the kernel K.
Next, we provide more details on how we trained such Neural SPDE model to generate Limit Order Book (LOB) dynamics [3]. The increasing availability of LOB data has instigated a significant interest in the development of statistical models for LOB dynamics. In recent years, new models based on SPDEs have been proposed to accurately describe and analyse these complex dynamics [HKN20) CM2T].
Data processing and hyperparameters We used real LOB data from the NASDAQ public ex- change made publicly available NMKK*18 which consists of about 4 million timestamped events over 10 consecutive trading days with L = 10 price levels on each side (bid and ask) of the LOB. Three versions of this dataset are provided, each normalised using a different technique. We used the data normalised with z-scores and split the LOB trace into sub-traces of length T 30.
Generator hyperparameters The generator is a Neural SPDE driven by a cylindrical Wiener process W with du = 2. The vector fields μą and σg are taken to be single layer perceptrons with dy E {16,38 32) followed by batch normalization and tanh activation function. Thus we obtain
Ng E (d,,,, d,, BatchNorm, tanh), Og E (d,,,, d, x dw. BatchNorm, tanh)
We used the second evaluation method proposed (SLG22 Section 3.3] with Picard's iterations and maximum number of frequency modes in {10, 20} in the spatial direction and fixed to 20 in the temporal direction. Instead of sampling the initial condition from (D, Iid )-valuuee Gaussian, we simply used the samples from X which case d,
Discriminator hyperparameters We integrated in time the output trajectories from the generator, as we observed this yielded more stable kernel scores. We mapped the path state values into H. where denotes a SE-T kernel on (D) with D = [0, 1], that is, kernel defined for all f.g E (D) by k(f.g)
= T(f)- where T (D) + y is Borel measurable, continuous and injective map. We considered three SE-T kernels respectively termed ID, SQR and CEXP:
1. (ID) SE-T kernel with T LD) + L)( defined for all f E L LD) by T(F)) =
2. (SQR) SE-T kernel with T: TT?2(D) - T.2(D) @ I2( defined by (())((.P8)
3. (CEXP) SE-T kernel with T: LD( + (D) defined by
Cp((f) where Cr. is
the covariance operator associated to the kernel kr. defined for all x, D by
F-1
(r)( = = ata (a=x))²
Σ
cos(22n(( -2)))
For ID and SQR, we used a E (1, 10), and for CEXP we used (a,1,F) E ((1, 1,5), (10, 10,5)).
We then used a dyadic order of 1 to compute the signature kernel.
24

---

Training hyperparameters We set the learning rate of the generator G| to be x 10 3 and trained it for a maximum number of 1 500 epochs. We used batch size of 64 due to memory constraints and the Adam optimizer with the default parameters of PyTorch.
25


Reviews:
[{"review_id": "VARGcKDfHQ", "reviewer": "Reviewer_YPT1", "summary": "This paper introduces a novel approach to training Neural SDEs using non-adversarial methods based on signature kernel scores. The authors demonstrate that the signature kernel score is strictly proper and provide consistent estimators for such scores. The effectiveness of their approach is demonstrated in various tasks.", "questions": "1. Can you provide quantitative results to show latent-SDE performs worse than SDE-GAN and your model?\n2. Can you visualize conditional path distributions generated by SDE-GAN and explain why your generated paths are better? \n3. Could you provide more details on how your model is useful in limit order book (LOB) prediction and elaborate on the design choices you made?", "limitations": "The authors have well discussed the limitations. The work assumes the noise is continuous without jump on event arrival, therefore is not applicable to noise governed by point processes. The work pose certain regularization conditions on the sampled paths. The work could benefit from including more varied evaluation metrics to assess the performance of the generated data. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is novel as it offers a non-adversarial alternative to traditional GAN-based training methods for Neural SDEs. This work is likely the first to introduce the signature kernel method to the Neural-SDE model. It achieves superior performance compared to the GAN method while being easier to train.\n\n2. The theoretical part of the paper is well-written, with clear mathematical formulations and adequate definition of symbols. The proof of strict properness and consistency, although direct consequences of prior works, offers robust guarantees.\n", "weaknesses": "1. Limited comparisons: The authors claim that their procedure outperforms alternative ways of training Neural SDEs, but they only compare it with the SDE-GAN. In the related works, the authors mention the latent SDE, which is easier to train than SDE-GAN, and claim that it yields worse performance than the SDE-GAN. While I acknowledge that GAN generally has better model capacity than VAE, can you provide quantitative results to substantiate this claim? The original paper only compares with the latent ODE from 2019.\n\n2. Lack of visualizations: Although the paper visualizes the generated distribution of paths conditioned on a previous path using their model, it would be beneficial to include a comparison with the paths generated by the SDE-GAN. This would provide a clearer understanding of how your outputs capture more characteristics of financial markets. Furthermore, the generated paths exhibit a large variance, which could be addressed or explained.\n\n3. Explanation of the LOB experiments: The simulation of LOB dynamics and the decision not to use autoregressive prediction are interesting, but how is the proposed design better than the five autoregressive generators that are listed? The paragraph discussing this topic appears to be hastily written. The reported average scores alone do not provide sufficient information. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparisons**\n\nThe comparison between SDE-GANs and latent SDEs (and the limitations of the latter) has been discussed in the PhD thesis [1, Section 4.3.3] and further analysis as well as quantitative results can be found in [2]. Therefore we decided to only include the most expressive model among all Neural SDEs in the benchmark and showcase that our approach, while non-adversarial, outperforms SDE-GANs, contrarily to latent SDEs. We will refer the reader to [1,2] to support the claim made in the related work section. \n\n**Visualizations**\n\nWe did not visualize conditional path distributions generated by SDE-GAN, as this would require significant modifications to the existing SDE-GAN implementation (available in the torchsde package). The authors of SDE-GAN do present a conditional example in their work, however, it is a class-conditioning example where the conditioning variable is discrete. For conditioning data arising from a continuous variable in an infinite-dimensional space (such as paths), this procedure cannot be implemented.\n\nIn the unconditional setting, our generated paths do not have a larger variance than those exhibited by the data measure (as confirmed by the Kolmogorov-Smirniov test on the marginals). This holds true in the conditional setting. We agree with your suggestion to further comment on Figure 2. Therefore, we propose to add the following remark: Although the three conditional distributions exhibit a larger variance (from left to right), this is to be expected, since they correspond to three different (increasing) levels of quadratic variation exhibited by the conditioning paths. \n\n**LOB Experiment**\n\nThe primary objective of the LOB experiment is to demonstrate that the newly proposed signature kernel scores can be used to effectively train continuous space-time generative models. However, we agree with the reviewer that our modelling choices can be further motivated and propose to add the following comments.\n\nAutoregressive generators produce “one-step-ahead” predictions by parameterizing the conditional distribution of a process at a specific point in time, conditioned on the previous k observations. Instead, we propose to use the Neural SPDE model as a generator which directly produces continuous time LOB trajectories in a discretization-free manner (also see the answer to Reviewer xQXq under the **Significance** header). Furthermore, we note that recent works in mathematical finance [3,4] have established that LOB dynamics are well-described by mechanistic models in the form of SPDEs. In light of these results, the Neural SPDE model is a natural choice for the generator as it offers a strong prior on the model space.  \n\nWe report the KS test statistic for each individual marginal, both in the spatial and temporal dimensions, in line with the other experiments in the paper. We recognize that it would be beneficial to include additional evaluation metrics. However, well-established and general-purpose metrics (that do not require expert knowledge on the specific task at hand) for assessing functional spatiotemporal generative models are still lacking. This gap arises due to the novelty of this field. Potential approaches could involve computing spatiotemporal semivariograms. However, we did not find adequate Python libraries for our purpose. Another direction to explore as future work would be to employ a “train on synthetic, test on real” methodology [5]. We will mention this in the future work section. \n\n**References**\n\n[1] Kidger, Patrick. \"On neural differential equations.\" arXiv preprint arXiv:2202.02435 (2022).\n\n[2] Kidger, Patrick, et al. \"Efficient and accurate gradients for neural sdes.\" Advances in Neural Information Processing Systems 34 (2021): 18747-18761.\n\n[3] Hambly, Ben, Jasdeep Kalsi, and James Newbury. \"Limit order books, diffusion approximations and reflected SPDEs: from microscopic to macroscopic models.\" Applied Mathematical Finance 27.1-2 (2020): 132-170.\n\n[4] Cont, Rama, and Marvin S. Müller. \"A stochastic partial differential equation model for limit order book dynamics.\" SIAM Journal on Financial Mathematics 12.2 (2021): 744-787.\n\n[5] Esteban, Cristóbal, Stephanie L. Hyland, and Gunnar Rätsch. \"Real-valued (medical) time series generation with recurrent conditional gans.\" arXiv preprint arXiv:1706.02633 (2017)."}, {"review_id": "gHVN5WU2o3", "reviewer": "Reviewer_xQXq", "summary": "This paper considered the training of neural SDEs, where are continuous-time generative models for sequential data. State-of-the-art approaches train neural SDEs in an adversarial manner and suffer from instability. In this work, a non-adversarial training approach is proposed for stable and effective training of neural SDEs. Particularly, a new class of scoring rules based on signature kernels is defined and used as objective for training. Extensive evaluations are performed to demonstrate the effectiveness of the proposed training objective. ", "questions": "Please see the details questions in the weakness section. I would love to change my rating if the authors can address my concerns about the significance and the evaluation. ", "limitations": "The limitations of the proposed method are well discussed. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The differences between the related works and the proposed method are clearly discussed.\n\nThe proposed method seems solid in theory.\n\nThe possible extensions of the proposed method are extensively discussed.\n", "weaknesses": "The significance of the work is somehow unclear to me. In the related work section, two works [PADD21] and [BO21] are discussed that are closely related to the proposed method. They are different from the proposed method in terms of the data type (discrete/continuous) and the setting of continuous-time processes (general/neural SDE specific). However, it is not clear how significant the proposed method is from the above two aspects. \n\nThe related work section is hard to follow. Lots of works are discussed from different perspectives: Neural SDE; score-based generative models; and scoring rules for generative networks. For me, these discussions are a bit disjointed. They are mainly discussed to show the differences between them and the proposed method. I think it is necessary to clearly locate the proposed method in the related literatures and then discuss. If there is a table or figure to visualize the overlaps between the proposed method and the other related methods, it will be clearer. \n\nThe proposed method is heavy in math. The notations are not clearly explained. For example, in line 95, what is Omega, F, and P? What is the law representing in line 97? More intuitive descriptions and clear definitions will greatly help the reading.  \n\nThough the mathematical foundation is solid, the empirical evaluation of the proposed method compared to SOTA methods is weak. Only SDE-GAN is considered for comparison. Score-based generative models (SGMs) are also leveraging scoring rules. Also, [BO21] introduced scoring rules for continuous-time processes. Since these works are closely related to the proposed method, it is necessary to compare with them. If can’t, it is necessary to explain the reasons, which I think also help clarify the significance of the proposed method. \n \nThere is a lack of experimental analysis. In section 4.1., there is no explanation or analysis for the values shown in Table 1. Why SDE-GAN is better at t=19? Similarly, in section 4.4, the conclusions of the evaluation discussed in section 4.4 are unclear to me. \n\nThe authors are claiming that the proposed method requires less memory compared with SOTAs. There is a lack of evaluation in terms of computational cost or the memory cost to support this claim.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Significance** \n\nWe believe the innovation of our work is twofold: \n\n1) the introduction of a new class of scoring rules for infinite-dimensional spaces of paths using signature kernels, adaptable to spatiotemporal signals, and with strict properness and consistency guarantees; \n\n2) the deployment of these scores to train Neural SDEs, resulting in a novel generator-discriminator pair which is mesh-free, offers memory-efficient backpropagation and surpasses other Neural SDE training methods in terms of stability and performance.\n\nAs mentioned in lines 80-84, in [PADD21] the authors construct statistical scores for discrete sequences, with strict properness guarantees only ensured under stringent Markov-type assumptions. A key aspect of our work is to develop consistent and proper scoring rules in the continuous-time, non-Markovian setting and use these in the context of generative modelling for functional data. Meanwhile, [B021] investigates scoring rules for continuous-time processes using truncated signatures, not signature kernels. Due to the truncation $N$ of the signature, these scores aren't strictly proper. Moreover, approaches based on truncated signatures are hindered by processes with values in $d$-dimensional spaces where $d$ is even moderate, because of the signature's exponential complexity $\\mathcal{O}(d^N)$. Approaches based on truncated signatures are unusable for infinite-dimensional cases like our LOB experiments. In contrast, our signature kernel scores sidestep these computational challenges. \n\n**Related work**\n\nOur work clearly positions itself at the nexus of Neural SDEs (classically trained as GANs), and continuous-time diffusion models (classically trained using scoring rules). It's important to note that, while we reference continuous-time diffusion models, they inherently possess limitations when it comes to generating paths in a resolution invariant manner as we do. This limitation arises from the absence of a canonical Lebesgue measure in infinite dimensions which precludes a coherent notion of density (i.e. classically defined, in finite-dimensional measure theory, as the Radon–Nikodym derivative with respect to the Lebesgue measure), which is fundamental to the foundation of all score-based diffusion models.\n\n**Notation**\n\nWhile we respect and value the perspective of each reviewer, we note that there are differing opinions on this matter. \n\n- Reviewer YPT1 states that *The theoretical part of the paper is well-written, with clear mathematical formulations and adequate definition of symbols*. \n- Reviewer kdBJ says that *This paper is written in high clarity. All formulations and derivations are clearly elaborated.*\n\nTo answer specifically the questions raised:\n\n- Line 95, as clearly stated $(\\Omega, \\mathcal{F}, \\mathbb{P})$ is a probability space.\n- Line 97, classical definition of law of a process.\n\n**Empirical evaluation**\n\nAs explained above, score-based diffusion models do not apply to generations of functional data such as paths. Besides, the scoring rules introduced in [BO21] are based on truncated signatures, which is precisely the second baseline we consider in the experiments.\n\n**Experimental analysis**\n\nDue to space constraints, we were not able to present the majority of our experimental analysis in the body of the paper. However, we do provide a more comprehensive analysis in each of the relevant sections of Appendix B, including a discussion of the hyperparameters used to achieve each set of results, further evaluation metrics, and discussion regarding both the qualitative and quantitative results we obtained. We also provided comprehensive Jupyter notebooks corresponding to each of the examples presented in the body (unconditional, conditional, and spatiotemporal generation). Remarks regarding the results found in Table 1 can be found in the paragraph directly above it. We appreciate that the reader may not be familiar with the Kolmogorov-Smirnov test; we will include an explanation for it in the final version of paper. Regarding why SDE-GAN performs better at the t=19 marginal: although SDE-GAN indeed performs better, our model also performs close to the test threshold acceptance level of 5% at the same marginal, indicating that outperformance is likely immaterial. SDE-GAN performs significantly worse at later marginals, as shown in Appendix B.1. Regarding the concern related to section 4.4 we refer to our answer to Reviewer YPT1.\n\n**Memory**\n\nWe note that we do not claim anywhere in the paper that our method requires less memory than SDE-GAN, but rather that it belongs to the class of Neural SDEs, which offer memory-efficient backpropagation (aka *adjoint methods* or *optimise-then-discretise*). We also note that the computational cost of the signature-PDE solver associated to our discriminator can be found in [1] and a general discussion of the computational complexity associated to solving Neural SDEs can be found in [2]. We will include these considerations in the final version of the paper.\n\n**References**\n[1] Salvi, Cristopher, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang. \"The Signature Kernel is the solution of a Goursat PDE.\" SIAM Journal on Mathematics of Data Science 3, no. 3 (2021): 873-899.\n\n[2] Kidger, Patrick. \"On neural differential equations.\" arXiv preprint arXiv:2202.02435 (2022).\n"}, {"review_id": "0cderZyA1J", "reviewer": "Reviewer_obEQ", "summary": "This paper introduces a novel approach for training Neural SDEs (Stochastic Differential Equations) as generative models for sequential data without using adversarial techniques. The authors propose a novel class of scoring rules based on signature kernels, which offer stability and avoid issues like mode collapse that are common in GAN-based training. The new formulation allows for memory-efficient adjoint-based back-propagation and enables the generation of spatiotemporal data, demonstrating superior performance over alternative methods in various tasks, including simulation of rough volatility models, conditional probabilistic forecasts of forex pairs, and mesh-free generation of limit order book dynamics.", "questions": "- Is the proposed method applicable to generating images?\n\n- How is the proposed model sensitive to hyper-parameters and what would be rule of thumbs to tune those hyperparameters?", "limitations": "Since my expertise is not stochastic differential equation, it is hard for me to point out critical limitations.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The authors propose a new method with rigorous theoretical justification and verify it with numerical experiments.\n\n- The proposed method does not require any adversarial training, so presumably training the generative model with the method is more stable than GANs.\n\n- The proposed method achieves significantly better performance than SDE-GAN without adversarial training.\n\n", "weaknesses": "I am not expert of this field, stochastic differential equations (SDE) for continuous-time generative models. So it is difficult to point out some critical weakness of the proposed method.\n\n-  I highly recommend that authors elaborate the evaluation metrics (e.g. KS-score) in detail for those who are not familiar with this field.\n`\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Evaluation Metric**\n\nThe two-sample Kolmogorov-Smirnov (KS) test is a nonparametric statistical test used to determine whether two sets of samples come from the same continuous distribution on $\\mathbb{R}$. The KS test statistic is the maximum absolute difference between two empirical cumulative distribution functions (CDFs). We appreciate that every reader may not be familiar with the KS test and are happy to amend the paper to include this explanation in the Appendix.\n\n**Image Generation**\n\nYes, our method can be applied to images sampled at an arbitrary spatial resolution. As a training objective, one would use the scoring rule associated with one of the three kernels in $L^2$ used in the LOB example, and consider the solution of the Neural SPDE only at the final time instead of the whole solution trajectory.\n\n**Hyperparameters**\n\nIn general, one can bucket the hyperparameters of the network into two categories: the “nice to have more” type (width and depth of neural networks governing the vector fields of the Neural SDEs, steps to train the model, dyadic order of the PDE solver associated to the signature kernel, step size in the generator SDE solver) which are largely constrained by computational considerations.\nThe other category is the set of hyperparameters which require more careful calibration. These include which path transformations to apply, and the appropriate learning rate to select. As with any generative model, successful training is especially sensitive to these choices. Due to space constraints, we did not go into this much detail in the body of the paper. However, in the preamble to Appendix B (and in Appendix B.1-5) we provide some general rules of thumb to choose these hyperparameters, including explicit choices to replicate the results we obtained. We also provide relevant commentary in each of the notebooks found in the code repository accompanying the paper.\n"}, {"review_id": "8gU3QZYsMS", "reviewer": "Reviewer_kdBJ", "summary": "In this work, the authors proposed to generate time-series using neural stochastic differential equations using the scoring rules-based training objective which is on signature paths computed from signature kernels. Combining the idea of generative adversarial network, this work also employ the generator-discriminator pair, but solve them as a system of linear PDEs where the adjoint method is applied to reduce the memory cost. Experiments are conducted accordingly on sequential data, including financial datasets.", "questions": "1. How should the \"order\" of the signature, k, be selected? Is there any specially designed optimization method to help with it?\n\n2. Why are the neural networks in neural SDE the regular neural networks? I am very curious if more sophisticated deep learning models, such as u-nets, are examined and what their impacts to the performance are.", "limitations": "The limitations of this work is not adequately addressed. It can be possible to consider the application scope of this work, where it is only for time series generation, especially for the cases where the dimensionality is not high.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is written in high clarity. All formulations and derivations are clearly elaborated.\n\n2. The idea is promising and novel to combine generative adversarial networks and neural stochastic differential equations, avoiding adversarial training which can cause instability in training.\n\n3. Rough path theory is a relatively new topic in mathematics. It has penetrated in the machine learning community since only 2019. It is interesting to utilize it in time series generation in machine learning.", "weaknesses": "1. In experimental design, the baseline models are not sufficiently explored. The experiments should  include diffusion models for time series generation, as they are commonly used generative models nowadays as a very special variant of SDE and would provide a useful comparison to the proposed approach.\n\n2. The space complexity and time complexity seem to raise the concerns, as signature transform is applied where the dimensionality is largely increased.\n\n3. The scale of the datasets in the experiments are of relatively low dimensionality in time series dat. It would be hard to prove the efficiency of the adjoint method applied to reduce the memory cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Baselines**\n\nPlease see the answer to Reviewer xQxq under the **Significance** and **Empirical evaluation** headers. \n\n**Complexity**\n\nAs explained in Section 3.3, our signature kernel scores are evaluated by solving scalar-valued PDEs. Therefore, our approach circumvents the computational challenges associated with calculating high-dimensional truncated signatures and can even be applied to infinite-dimensional cases like our LOB experiments. Consequently, there is no need to select a truncation level. Please also see the response to reviewer xQXq (under the **Significance** header) where we also summarize the other benefits of utilizing signature kernel scores. \n\n**Dimensionality**\n\nWhile we agree that our Neural SDE model is trained on relatively low-dimensional time series data, we also include an experiment on spatiotemporal LOB processes which are infinite-dimensional. As the dimension escalates and the memory cost becomes prohibitive, we can leverage the adjoint method for training the Neural SDE model. \n\n**Architecture**\n\nWe chose to keep the vector fields governing the Neural SDEs as simple MLPs, aligning with the classical architecture from [1]. We appreciate your insight and note that studying different, more sophisticated deep learning models to parametrize the vector fields associated with a given Neural SDE is an interesting research topic. \n\n[1] Kidger, Patrick, James Foster, Xuechen Li, and Terry J. Lyons. \"Neural sdes as infinite-dimensional gans.\" In International conference on machine learning, pp. 5453-5463. PMLR, 2021. \n"}, {"review_id": "qI5IVCpJUB", "reviewer": "Reviewer_dxEe", "summary": "This paper proposes a signature kernel-based Neural SDE by using signature kernel scores. The proposed method eliminates existing state-of-the-art methods' mode collapse and instability using adversarial training.", "questions": "1. Is that why you chose the signature kernel among the various kernels? Also, if different kernels are adaptable, I would like to know the results of your experiments with other kernels.\n2. Which generator does the sub Figure in Figure 1 correspond to?\n\nMinor corrections\n\n3. The unconditional_nsde.ipynb included in the code provided need to be corrected?\n```\n#cond_                    = data_type == \"rBergomi\"\ncond_                    = data_type == \"gbm\"\n```", "limitations": "yes", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In all experiments, the proposed method outperforms existing methods.\n2. The explanation in chapter three is clear and reproducible because the code is also provided.\n", "weaknesses": "1. This paper's main contribution includes using signature kernel scores for the discriminator, but there needs to be more motivated to use signature kernels.\n2. There is no sub-caption for each of Figure 1, so it is unclear which generator it is.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Contribution**\n\nPlease see the response to Reviewer xQxq under the header **Significance**, where we explain that signature kernels allow us to introduce a class of scoring rules for infinite-dimensional spaces of paths, adaptable to spatiotemporal signals, and with strict properness and consistency guarantees. \n\n**Figure 1**\n\nIn the preceding paragraph, we explain that the three subfigures are obtained by training Neural SDEs with three different discriminators. We agree with you that the caption should be expanded to clarify which histogram belongs to which discriminator, and we will amend the caption accordingly. \n\n**Minor corrections**\n\nAlthough this line does not affect the output of the code as it is a commented line, we appreciate that it can be confusing for the reader. The line was left to help guide the user on how to choose a different generation problem. We are happy to remove it.\n"}]

Author rebuttal:
We thank all the reviewers the useful feedback and insightful questions. We hope that our responses will address all raised concerns.

Meta-review: The paper proposes a non-adversarial training strategy for neural SDEs. The reviewers think the writing is clear and the method based on signature Kernel scores is novel. Please elaborate the evaluation metrics and provide more background introduction in the final version.
Decision: Accept (poster)