Title: Non-adversarial training of Neural SDEs with signature kernel scores
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural SDEs

score-based generative models

signature kernels

time series
Abstract: Neural SDEs are continuous-time generative models for sequential data. State-of-the-art performance for irregular time series generation has been previously obtained by training these models adversarially as GANs. However, as typical for GAN architectures, training is notoriously unstable, often suffers from mode collapse, and requires specialised techniques such as weight clipping and gradient penalty to mitigate these issues. In this paper, we introduce a novel class of scoring rules on pathspace based on signature kernels and use them as objective for training Neural SDEs non-adversarially. By showing strict properness of such kernel scores and consistency of the corresponding estimators, we provide existence and uniqueness guarantees for the minimiser. With this formulation, evaluating the generator-discriminator pair amounts to solving a system of linear path-dependent PDEs which allows for memory-efficient adjoint-based backpropagation. Moreover, because the proposed kernel scores are well-defined for paths with values in infinite dimensional spaces of functions, our framework can be easily extended to generate spatiotemporal data. Our procedure significantly outperforms alternative ways of training Neural SDEs on a variety of tasks including the simulation of rough volatility models, the conditional probabilistic forecasts of real-world forex pairs where the conditioning variable is an observed past trajectory, and the mesh-free generation of limit order book dynamics.

Non-adversarial training of Neural SDEs with signature kernel scores

Abstract

Neural SDEs are continuous-time generative models for sequential data. Stateof-the-art performance for irregular time series generation has been previously obtained by training these models adversarially as GANs. However, as typical for GAN architectures, training is notoriously unstable, often suffers from mode collapse, and requires specialised techniques such as weight clipping and gradient penalty to mitigate these issues. In this paper, we introduce a novel class of scoring rules on pathspace based on signature kernels and use them as objective for training Neural SDEs non-adversarially. By showing strict properness of such kernel scores and consistency of the corresponding estimators, we provide existence and uniqueness guarantees for the minimiser. With this formulation, evaluating the generator-discriminator pair amounts to solving a system of linear path-dependent PDEs which allows for memory-efficient adjoint-based backpropagation. Moreover, because the proposed kernel scores are well-defined for paths with values in infinite dimensional spaces of functions, our framework can be easily extended to generate spatiotemporal data. Our procedure permits conditioning on a rich variety of market conditions and significantly outperforms alternative ways of training Neural SDEs on a variety of tasks including the simulation of rough volatility models, the conditional probabilistic forecasts of real-world forex pairs where the conditioning variable is an observed past trajectory, and the mesh-free generation of limit order book dynamics.

1 Introduction

Stochastic differential equations (SDEs) are a dominant modelling framework in many areas of science and engineering. They naturally extend ordinary differential equations (ODEs) for modelling dynamical systems that evolve under the influence of randomness.

A neural stochastic differential equation (Neural SDE) is a continuous-time generative model for irregular time series where the drift and diffusion functions of an SDE are parametrised by neural networks [TR19, JB19, HvdHRM20, LWCD20, KFL + 21, MSKF21]. These models have become increasingly popular among financial practitioners for pricing and hedging of derivatives and overall risk management [ASS20, GSVŠ + 20, CJB23, HFH + 23]. Training a Neural SDE amounts to minimising over model parameters an appropriate notion of distance between the law on pathspace generated by the SDE and the empirical law supported on observed data sample paths.

Various choices of training mechanisms have been proposed in the literature; state-of-the-art performance has been achieved by training Neural SDEs adversarially as Wasserstein-GANs [KFL + 21]. However, as typical for GAN architectures, training is notoriously unstable, often suffers from mode collapse, and requires specialised techniques such as weight clipping and gradient penalty.

In this paper we introduce a novel class of scoring rules based on signature kernels , a class of characteristic kernels on paths [LSD + 21, CFC + 21, SLL + 21, LSC + 21, CLS23, HLL + 23], and use them as objective for training Neural SDEs non-adversarially. We provide existence and uniqueness guarantees for the minimiser by showing strict properness of the signature kernel scores and consistency of the corresponding estimators.

With this training formulation, the generator-discriminator pair becomes entirely mesh-free and can be evaluated by solving a system of linear path-dependent PDEs which allows for memory-efficient adjoint-based backpropagation. In addition, because the proposed kernel scores are well-defined for classes of paths with values in infinite dimensional spaces of functions, our framework can be easily extended to the generation of spatiotemporal signals.

We demonstrate how our procedure is more stable and outperforms alternative ways of training Neural SDEs on a variety of tasks from quantitative finance including the simulation of rough volatility models, the conditional probabilistic forecasts of real-world forex pairs where the conditioning variable is an observed past trajectory, and the mesh-free generation of limit order book dynamics.

2 Related work

Prior to our work, two main approaches have been proposed to fit a Neural SDE as a time series generative model, differing in their choice of divergence to compare laws on pathspace.

The SDE-GAN model introduced in [KFL + 21] uses the 1 -Wasserstein distance to train a Neural SDE as a Wasserstein-GAN [ACB17]. Namely, the "witness functions" of the 1 -Wasserstein distance are parameterised by neural controlled differential equations [KMFL20, MSK + 20] and the generatordiscriminator pair is trained adversarially. SDE-GANs are relatively unstable to train mainly because they require a Lipschitz discriminator. Several techniques such as weight clipping and gradient penalty have been introduced to enforce the Lipschitz constraint and partially mitigate the instability issue [Kid22]. SDE-GANs are also sensitive to other hyperparameters, such as the choice of optimisers, their learning rate and momentum, where small changes can yield erratic behavior.

The latent SDE model [LWCD20] trains a Neural SDE with respect to the KL divergence using the principles of variational inference for SDEs [Opp19]. This approach consists in maximising an objective that includes the KL divergence between the laws produced by the original SDE (the prior) and an auxiliary SDE (the approximate posterior). The two SDEs have the same diffusion term but different initial conditions and drifts, and a standard formula for their KL divergence exists. After training, the learned prior can be used to generate new sample paths. Latent SDEs can be interpreted as variational autoencoders, and generally yield worse performance than SDE-GANs, which are more challenging to train, but offer greater model capacity.

Besides Neural SDEs, other time series generative models have been proposed, including discrete-time models such as [YJVdS19] and [NSW + 20] 2 which are trained adversarially, continuous-time flow processes [DCB + 20] and score-based diffusion models for audio generation [CZZ + 20, KPH + 20].

The class of score-based generative models (SGMs) seeks to map a data distribution into a known prior distribution via an SDE [SSDK + 20, VKK21]. During training, the (Stein) score [LLJ16] of the SDE marginals is estimated and then used to construct a reverse-time SDE. By sampling data from the prior and solving the reverse-time SDE, one can generate samples that follow the original data distribution. We note that our techniques for generative modelling via scoring rules, although similar in terminology, are fundamentally different, as we train Neural SDEs with respect to a loss function that directly consumes the law on pathspace generated by the SDE.

Scoring rules [GR07] have been used to define training objectives for generative networks [BMN16, GSvdB + 20] which have been shown to be easier to optimize compared to GANs [PADD21, PD22]. Closer to our work is [PADD21] which constructs statistical scores for discrete (spatio-)temporal

signals. However, their strict properness is ensured under Markov-type assumptions and their continuous-(space-)time limit has not been studied. A key aspect of our work is to develop consistent and effective scoring rules for generative modelling in the continuous-time setting. While [BO21] has also introduced scoring rules for continuous-time processes, our emphasis lies in constructing so-called kernel scores specifically for training Neural SDE and Neural SPDE generative models.

The Neural SPDE model introduced in [SLG22] parametrises the solution operator of stochastic partial differential equations (SPDEs), which extend SDEs for modelling signals that vary both in space and in time. So far, Neural SPDEs have been trained in a supervised fashion by minimizing the pathwise L 2 norm between pairs of spatiotemporal signals. While this approach has proven effective in learning fast surrogate SPDE solvers, it is not well-suited for generative modeling where the goal is to approximate probability measures supported on spatiotemporal functions. In this work, we propose a new training objective for Neural SPDEs to improve their generative modeling capabilities.

3 Training Neural SDEs with signature kernel scores

3.1 Background

We take (Ω , F , P ) as the underlying probability space. Let T > 0 and d x ∈ N . Denote by X be the space of continuous paths of bounded variation from [0 , T ] to R d x with one monotone coordinate 3 . For any random variable X with values on X , we denote by P X := P ◦ X -1 its law.

The signature map S : X → T is defined for any path x ∈ X as the infinite collection S ( x ) = ( 1 , S 1 ( x ) , S 2 ( x ) , ... ) of iterated Riemann-Stieltjes integrals

where ⊗ is the standard tensor product of vector spaces and T := R ⊕ R d x ⊕ ( R d x ) ⊗ 2 ⊕ ...

Any inner product ⟨· , ·⟩ 1 on R d x yields a canonical Hilbert-Schmidt inner product ⟨· , ·⟩ k on ( R d x ) ⊗ k for any k ∈ N , which in turn yields, by linearity, a family of inner products ⟨· , ·⟩ T on T . We refer the reader to [CLX21] for an in-depth analysis of different choices. By a slight abuse of notation, we use the same symbol to denote the Hilbert space obtained by completing T with respect to ⟨· , ·⟩ T .

3.2 Neural SDEs

Let W : [0 , T ] → R d w be a d w -dimensional Brownian motion and a ∼ N (0 , I d a ) be sampled from d a -dimensional standard normal. The values d w , d a ∈ N are hyperparameters describing the size of the noise. A Neural SDE is a model of the form

for t ∈ [0 , T ] , with Y : [0 , T ] → R d y the strong solution, if it exists, to the Stratonovich SDE, where

are suitably regular neural networks, and A θ ∈ R d x × d y , b θ ∈ R d x . The dimension d y ∈ N is a hyperparameter describing the size of the hidden state. If µ θ , σ θ are Lipschitz and E a [ ξ θ ( a ) 2 ] < ∞ then the solution Y exists and is unique.

Given a target X -valued random variable X true with law P X true , the goal is to train a Neural SDE so that the generated law P X θ is as close as possible to P X true , for some appropriate notion of closeness.

3.3 Signature kernels scores

Scoring rules are a well-established class of functionals to represent the penalty assigned to a distribution given an observed outcome, thereby providing a way to assess the quality of a probabilistic forecast. Scoring rules have been applied to a wide range of areas including econometrics [MS13],

weather forecasting [GR05], and generative modelling [PADD21]. How to effectively select a scoring rule is a challenging and somewhat task-dependent problem, particularly when the data is sequential. Scoring rules based on kernels offer the advantages of working on unstructured and infinite dimensional data without some of the concomitant drawbacks, such as the absence of densities. Next, we introduce a class of scoring rules on paths based on signature kernels to measure closeness between path-valued random variables. These will be used in the next section to train Neural SDEs.

The signature kernel k sig : X × X → R is a symmetric positive semidefinite function defined for any pair of paths x, y ∈ X as k sig ( x, y ) := ⟨ S ( x ) , S ( y ) ⟩ T . In [SCF + 21] the authors provided a kernel trick proving that the signature kernel satisfies

which reduces to a linear hyperbolic PDE in the when the paths x, y are almost-everywhere differentiable. Several finite difference schemes are available for numerically evaluating solutions to Equation (2), see [SCF + 21, Section 3.1] for details.

We denote by H the unique reproducing kernel Hilbert space (RKHS) of k sig. From now on we endow X with a topology with the respect to which the signature is continuous; see [CT22] for various choices of such topologies. Denote by P ( X ) the set of Borel probability measures on X .

Proposition 3.1. The signature kernel is characteristic for every compact set K ⊂ X , i.e. the map P ↦→ ∫ k sig ( x, · ) P ( dx ) from P ( K ) to H is injective.

Remark 3.2. The proof of this statement is classical and is a simple consequence of the universal approximation property of the signature [KBPA + 19, Proposition A.6] and the equivalence between universality of the feature map and characteristicness of the corresponding kernel [SGS18, Theorem 6]. In particular, Proposition (3.1) implies that the signature kernel is cc-universal, i.e. for every compact subset K ⊂ X , the linear span of the set of path functionals { k sig ( x, · ) : x ∈ K} is dense in C ( K ) in the the topology of uniform convergence.

We define the signature kernel score ϕ sig : P ( X ) ×X → R for any P ∈ P ( X ) and y ∈ X as

A highly desirable property to require from a score is its strict properness , consisting in assigning the lowest expected score when the proposed prediction is realised by the true probability distribution.

Proposition 3.3. For any compact K ⊂ X , ϕ sig is a strictly proper kernel score relative to P ( K ) , i.e. E y ∼ Q [ ϕ sig ( Q , y )] ≤ E y ∼ Q [ ϕ sig ( P , y )] for all P , Q ∈ P ( K ) , with equality if and only if P = Q .

The proof of this statement can be found in the appendix and follows from [GR07, Theorem 4] and Proposition 3.1. We note that the signature kernel score induces a divergence on P ( X ) known as the signature kernel maximum mean discrepancy (MMD), defined for any P , Q ∈ P ( X ) as

The following result provides a consistent and unbiased estimator for evaluating the signature kernel score from observed sample paths. The proof can be found in the appendix and follows from standard results for the associated MMD [GBR + 12, Lemma 6].

Proposition 3.4. Let P ∈ P ( X ) and y ∈ X . Given m sample paths { x i } i m =1 ∼ P , the following is a consistent and unbiased estimator of ϕ sig

̸

3.4 Non-adversarial training of Neural SDEs via signature kernel scores

We now have all the elements to outline the procedure we propose to train the Neural SDE model (1) non-adversarially using signature kernel scores introduced in the previous section.

Unconditional setting We are given a target X -valued random variable X true with law P X true . Recall the notation P X θ for the law generated by the SDE (1). The training objective is given by

Note that training with respect to D k sig is an equivalent optimisation as the second expectation in equation (3) is constant with respect to θ . This means that in the unconditional setting our model corresponds to a continuous time generative network of [LSZ15].

Combining equations (1), (2), (4) and (5) the generator-discriminator pair can be evaluated by solving a system of linear PDEs depending on sample paths from the Neural SDE; in summary:

Remark 3.5. The generation of sample paths from X θ from the SDE solver and the evaluation of the objective L via the PDE solver can in principle be performed concurrently, although, in our implementation we evaluate the full model (6) in a sequential manner.

Conditional setting It is straightforward to extend our framework to the conditional setting where Q is some distribution we wish to condition on, and P X true ( ·| x ) is a target conditional distribution with x ∼ Q . By feeding the observed sample x as an additional variable to all neural networks of the Neural SDE (1), the generated strong solution provides a parametric conditional law P X θ ( ·| x ) , and the model can be trained according to the modified objective

Because ϕ sig is strictly proper, the solution to (7) is P X θ ( ·| x ) = P X true ( ·| x ) Q -almost everywhere. With data sampled as { ( x i , y i ) } n i =1 where x i ∼ Q and y i ∼ P X true ( ·| x i ) we can replace eq. (7) by

We note that in our experiments we focus on the specific case where the conditioning variable x is a path in X corresponding to the observed past trajectory of some financial assets (see Figure 2).

3.5 Additional details

Interpolation Samples from X true are observed on a discrete, possibly irregular, time grid while samples from X θ are generated from (1) by means of an SDE solver of choice (see [Kid22, Section 5.1] for details). Interpolating in time between observations produces a discrete measure on path space, the ones desired to be modelled. The interpolation choice is usually unimportant and simple linear interpolation is often sufficient. See [MKYL22] for other choices of interpolation.

Backpropagation Training a Neural SDE usually means backpropagating through the SDE solver. Three main ways of differentiating through an SDE have been studied in the literature: 1) Discretisethen-optimise backpropagates through the internal operations of the SDE solver. This option is memory inefficient, but will produce accurate and fast gradient estimates. 2) Optimise-then-discretise derives a backwards-in-time SDE, which is then solved numerically. This option is memory efficient, but gradient estimates are prone to numerical errors and generally slow to compute. We note that unlike the case of Neural ODEs, giving a precise meaning to the backward SDE falls outside the usual framework of diffusions. However, rough path theory [Lyo98, FLMS23] provides an elegant remedy by allowing solutions to forward and backward SDEs to be defined pathwise, similarly to the case of ODEs; see [Kid22, Appendix C.3.3] for a precise statement. 3) Reversible solvers are memory efficient and accurate, but generally slow. Here we do not advocate for any particular choice as all of the above backpropagation options are compatible with our pipeline.

Similarly, because the signature kernel score can be evaluated by solving a system of PDEs, backpropagation can be carried out by differentiating through the PDE solver analogously to the discretise-thenoptimise option for SDEs. We note that [LSC + 21] showed that directional derivatives of signature kernels solve a system of adjoint-PDEs, which can be leveraged to backpropagate through the discriminator using an optimise-then-discretise approach. We used this approach in our experiments.

Itô vs Stratonovich Stratonovich SDEs are slightly more efficient to backpropagate through using an optimise-then-discretise approach. In the case of Itô SDEs, the backward equation is derived by applying the Itô-Stratonovich correction term to convert it into a Stratonovich SDE, deriving the corresponding backward equation through rough path theoretical arguments, and then converting it back to an Itô SDE by applying a second Stratonovich-Itô correction.

Paths with values in infinite dimensional spaces While we have defined the signature kernel for paths of bounded variation with values in R d x , the kernel is still well-defined when R d x is replaced with a generic Hilbert space V . Remarkably, even when V is infinite dimensional, the evaluation of the kernel can be carried out, as Equation (2) only depends on pairwise inner products between the values of the input paths. In particular, the kernel can be evaluated on paths taking their values in functional spaces, which has far-reaching consequences in practice. For example, this gives the flexibility to map the values of finite dimensional input paths into a possibly infinite dimensional feature space, such as the reproducing kernel Hilbert space of a kernel κ on R d x , that is, V = H κ . This also provides a natural kernel for spatiotemporal signals, such as paths taking their values in V = L 2 ( D ) , the space of square-integrable functions on a compact domain D ⊂ R d . For practical applications, the inner product in Equation (2) can be approximated using discrete observations of the input signals on a mesh of the spatial domain D . The inner product in L 2 ( D ) can be replaced with more general kernels as those introduced in [WD22]. While it has become common practice to use signature kernels on the RKHS-lifts of Euclidean-valued paths, the ability to define and compute signature kernels on spatiotemporal signals has been, to our knowledge, overlooked in the literature.

4 Experiments

We perform experiments across five datasets. First is a univariate synthetic example, the benchmark Black-Scholes model, which permits to readily verify the quality of simulated outputs. The second synthetic example is a state of-the-art univariate stochastic volatility model, called rough Bergomi model. The rough Bergomi model realistically captures many relevant properties of options data, but due to its rough (and hence non-Markovian) nature it is well-known to be difficult to simulate. The third is a multidimensional example with foreign exchange (forex, or FX) currency pairs, which was chosen not only because of the relevance and capitalisation of FX markets but also due to its well-known intricate complexity. Fourth is a univariate example, where we demonstrate the method's ability to condition on relevant variables, given by paths. Finally we present a spatiotemporal generative example, where we seek to simulate the dynamics of the NASDAQ limit order book.

For the unconditional examples, we compare against the SDE-GAN from [KFL + 21] and against the same pipeline as the one we proposed, but using an approximation ϕ N sig of the signature kernel score ϕ sig obtained by truncating signatures at some level N ∈ N . We evaluate each training instance with a variety of metrics. The first is the Kolmogorov-Smirnov (KS) test, which is a nonparameteric two-sample test between two empirical probability distributions on R , see [Smi39] for more details. We apply the KS test on the marginals between a batch of generated paths against an unseen batch from the real data distribution. We repeated this test 5000 times at the 5% significance level and reported the average KS score along with the average Type I error. Each training instance was kept to a maximum of 2 hours for the synthetic examples, and 4 hours for the real data example. Finally, as mentioned at the end of Section 3.5, when training with respect to ϕ sig we mapped path state values into ( H , κ ) where κ denotes the RBF kernel on R d . Additional details on hyperparameter selection, learning rates, optimisers and further evaluation metrics can be found in the Appendix.

4.1 Geometric Brownian motion

As a toy example, we seek to learn a geometric Brownian motion (gBm) of the form

We chose µ = 0 , σ = 0 . 2 and generated time-augmented paths of length 64 over the grid ∆ = { 0 , 1 , . . . , 63 } with dt = 0 . 01 . Thus our dataset is given by time-augmented paths y : [0 , 63] → R 2 embedded in path space via linear interpolation. For all three discriminators, the training and test set were both comprised of 32768 paths and the batch size was chosen to be N = 128 . We trained the SDE-GAN for 5000 steps, ϕ sig for 4000 and ϕ N sig for 10000 steps. Table 1 gives the KS scores along

each of the specified marginals, along with the percentage Type I error. Here the generator trained with ϕ sig performs the best, achieving a Type I error at the assumed confidence level.

4.2 Rough Bergomi volatility model

It is well-known that the benchmark model (9) oversimplifies market reality. More complex models, (rough) stochastic volatility (SV) were introduced in the past decades, that are able to capture relevant properties of market data are used by financial practitioners to price and hedge derivatives. Prominent examples of stochastic volatility mdoels include the Heston and SABR models [HKLW02, HLW15, Hes93]. State-of-the-art models in this context have been introduced in [GJR18]. They display a stochastic volatility with rough sample paths. Most notable among these for pricing and hedging is the rough Bergomi (rBergomi) model [BFG16] which is of the form

and where ξ u t is the instantaneous forward variance for time u at time t , with ξ t t = V t , and α = H -1 / 2 where H is the Hurst exponent. The parameter set is given by ( η, ρ, H ) with initial conditions X 0 = x and ξ u t = ξ 0 . It has been a well-known headache for modellers that-despite their many modelling advantages-rough volatility models (such as (10)) are slow to to simulate with traditional methods. We demonstrate how our method can be used to capture the dynamics of the rough Bergomi model (10), and in passing we also note that our method provides a significant simulation speedup for (10) compared to previously available simulation methods.

To do so, we simulate paths of length 64 over the time window to [0 , 2] , and specify dt = 1 / 32 . Thus paths are of length 64 . The parameters are ( ξ 0 , η, ρ, H ) = (0 . 04 , 1 . 5 , -0 . 7 , 0 . 2) and set d = 1 . Paths are again time-augmented. The hyperparameters for training are the same as in the previous section. The results on the marginal distributions are summarized in Table 2. We see that that training with respect to ϕ sig vastly outperforms the other two discriminators.

4.3 Foreign exchange currency pairs

We consider an example where samples from the data measure P X true are time-augmented paths y : [0 , T ] → R 3 corresponding to hourly market close prices of the currency pairs EUR/USD and USD/JPY 4 . To deal with irregular sampling, we linearly interpolate each sample y over a fixed grid ∆ = { t 0 , t 1 , . . . , t 63 } . Training hyperparameters were kept the same as per the rBergomi example: paths are comprised of 64 observations, the batch size was taken to be N = 128 , and the number of training epochs was taken to be 10000 for the SDE-GAN, 4000 for ϕ sig and 15000 for ϕ N sig . KS scores for each of the marginals are given in Table 3 and 4. We note that only the generator trained with ϕ sig is able to achieve strong performance on nearly all marginals.

We also present a histogram of sample correlations between generated EUR/USD and USD/JPY paths for each of the three discriminators alongside those from the data distribution. From Figure 1 it appears that only the Neural SDE trained with ϕ sig correctly identifies the negative correlative structure between the two pairs. This is likely due to the fact that these dependencies are encoded in higher order terms of the signature that the truncated method does not capture.

1.2

Pxtrue

2.0

Pxtue

2.5

Pxtrue

Pxθ

Pxθ

Pxθ

1.0

2.0

1.5

0.8

1.5

0.6

1.0

1.0

0.4

0.5

0.5

0.2

0.0

-1

0

0.0

0

0.0

1

0

p

p

p

We now consider a conditional generation problem, where the conditioning variables are timeaugmented paths Q ∼ x : [ t 0 -dt, t 0 ] → R 2 representing the trajectory of prior dt = 32 observations of EUR/USD 15-minute close prices, and the target distribution is X true : [ t 0 , t 0 + dt ′ ] → R 2 representing the following dt ′ = 16 observations. Given batched samples { x i , y i } N i =1 , where x i ∼ Q and y i ∼ P X true ( ·| x i ) , we train our generator according to equation (7). We encoded the conditioning paths via the truncated (log)signatures of order 5 , and fed these values into each of the neural networks of the Neural SDE. In Figure 2, it is evident that the conditional generator exhibits the capability to produce conditional distributions that frequently encompass the observed path. Furthermore, it is noteworthy that these generated distributions capture certain distinctive characteristics of financial markets, such as martingality, mean reversion, or leverage effects when applicable.

4.4 Simulation of limit order books

Here, we consider the task of simulating the dynamics of a limit order book (LOB), that is, an electronic record of all the outstanding orders for a financial asset, representing its supply and demand over time. Simulating LOB dynamics is an important challenge in quantitative finance and several synthetic market generators have been proposed [LWL + 20],[VBP + 20],[SCC21],[CPC + 21],[CMVB22]. An order o = ( t o , x o , v o ) submitted at time t o with price x o and size v o > 0 (resp., v o < 0 ) is a commitment to sell (resp., buy) up to | v o | units of the traded asset at a price no less (resp., no greater)

3

C

1

2.5

Pxo(-lα)

2

y ～ Pxurue(-|c)

0

0.0

1

-1

2.5

X

0

x-2

X

-5.0

-3

-1

7.5

-4

C

C

Pxe(-lc)

10.0

Pxe(-lα)

2

-5

y ～ Pxte(-|c)

y ～Pxtrue(-|c)

12.5

0

20

40

0

20

40

0

20

40

t

t

t

than x o . Various events are tracked (e.g. new orders, executions, and cancellations) and the LOB B ( t ) is the set of all active orders in a market at time t . While prior work typically fit a generator that produces the next event, and run it iteratively to generate a sequence of events, we propose to model directly the spatiotemporal process Y t ( x ) = ∑ o ∈B ( t ): x o = x v o . To generate LOB trajectories, we use the Neural SPDE model and train it by minimising expected spatiotemporal kernel scores constructed by composing the signature kernel k sig with 3 different SE-T type kernels introduced in [WD22], namely the ID, SQR and CEXP kernels. We fit our model on real LOB data from the NASDAQ public exchange [NMK + 18] which consists of about 4 M timestamped events with L = 10 price levels. We split this LOB trace into sub-traces of size T = 30 to construct our dataset. On Figure 3 report the average KS scores for each of the L × T marginals, using the 3 different kernel scores.

ID

SQR

CEXP

2

2

2

4

4

0.25

6

181614121086

181614121086

price

18161412108

price

price

0.20

-0.15

-0.10

0369121518212427

0369121518212427

0369121518212427

time

time

time

5 Conclusion and future work

This work showcases the utilization of Neural SDEs as a generative model, highlighting their advantages over competitor models in terms of simplicity and stability, particularly via non-adversarial training. Additionally, we show how Neural SDEs exhibit the ability to be conditioned on diverse and intricate data structures, surpassing the capabilities of existing competitor works. We have achieved this by introducing the signature kernel score on paths and by showing their applicability to our setting (by proving strict properness). Performance of our methods are given computational time and memory is competitive with state-of-the-art methods. Moreover, we have shown that this approach extends to the generation of spatiotemporal signals, which has multiple applications in finance including limit order data generation. Further extensions of this work may include extending its generality to include jump processes in the driving noise of the approximator process (Neural SDEs) used. On the theoretical level extensions may include the validity of results to paths with lower regularity than currently considered. Although sample paths from a Stratonovich SDE are not of bounded variation almost surely, sample paths generated by an SDE solver, once interpolated, are piecewise linear, and hence of bounded variation. A similar point can be made about compactness of the support of the measures. It is possible to ensure characteristicness of the signature kernel on non-compact sets of less regular paths using limiting arguments and changing the underlying

topology on pathspace. Further extensions for practical applications can (and should) include the inclusion of more varied evaluation metrics and processes. Notably, in a later step, the generated data should be tested by assessing whether existing risk management frameworks and investment engines can be improved when data used for backtesting is augmented with synthetic samples provided by our methods. Furthermore, the spatiotemporal results can be extended to more complex structures, including being used for the synthetic generation of implied volatility surface dynamics, which has been a notoriously difficult modelling problem in past decades.

References

[CZZ + 20] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations , 2020.

[DCB + 20] Ruizhi Deng, Bo Chang, Marcus A Brubaker, Greg Mori, and Andreas Lehrmann. Modeling continuous stochastic processes with dynamic normalizing flows. Advances in Neural Information Processing Systems , 33:7805-7815, 2020.

[FLMS23] Adeline Fermanian, Terry Lyons, James Morrill, and Cristopher Salvi. New directions in the applications of rough path theory. IEEE BITS the Information Theory Magazine , 2023.

[GBR + 12] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

[GJR18] Jim Gatheral, Thibault Jaisson, and Mathieu Rosenbaum. Volatility is rough. Quantitative finance , 18(6):933-949, 2018.

[GPW + 13] Martin D Gould, Mason A Porter, Stacy Williams, Mark McDonald, Daniel J Fenn, and Sam D Howison. Limit order books. Quantitative Finance , 13(11):1709-1742, 2013.

[GR05] Tilmann Gneiting and Adrian E Raftery. Weather forecasting with ensemble methods. Science , 310(5746):248-249, 2005.

[GR07] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007.

[GSvdB + 20] Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems , 33:13062-13072, 2020.

[GSVŠ + 20] Patryk Gierjatowicz, Marc Sabate-Vidales, David Šiška, Lukasz Szpruch, and Žan Žuriˇ c. Robust pricing and hedging via neural sdes. arXiv preprint arXiv:2007.04154 , 2020.

[Hes93] Steven L Heston. A closed-form solution for options with stochastic volatility with applications to bond and currency options. The review of financial studies , 6(2):327343, 1993.

[HFH + 23] Melker Höglund, Emilio Ferrucci, Camilo Hernández, Aitor Muguruza Gonzalez, Cristopher Salvi, Leandro Sánchez-Betancourt, and Yufei Zhang. Solving and learning non-markovian stochastic control problems in continuous-time with neural RDEs, 2023.

[HKLW02] Patrick S Hagan, Deep Kumar, Andrew S Lesniewski, and Diana E Woodward. Managing smile risk. The Best of Wilmott , 1:249-296, 2002.

[HKN20] Ben Hambly, Jasdeep Kalsi, and James Newbury. Limit order books, diffusion approximations and reflected spdes: from microscopic to macroscopic models. Applied Mathematical Finance , 27(1-2):132-170, 2020.

[HLL + 23] Blanka Horvath, Maud Lemercier, Chong Liu, Terry Lyons, and Cristopher Salvi. Optimal stopping via distribution regression: a higher rank signature approach. arXiv preprint arXiv:2304.01479 , 2023.

[HLW15] Patrick Hagan, Andrew Lesniewski, and Diana Woodward. Probability distribution in the sabr model of stochastic volatility. In Large deviations and asymptotic methods in finance , pages 1-35. Springer, 2015.

[HvdHRM20] Liam Hodgkinson, Chris van der Heide, Fred Roosta, and Michael W Mahoney. Stochastic normalizing flows. arXiv preprint arXiv:2002.09547 , 2020.

[JB19] Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems , 32, 2019.

[KBPA + 19] Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. Advances in Neural Information Processing Systems , 32, 2019.

[KFL + 21] Patrick Kidger, James Foster, Xuechen Li, Harald Oberhauser, and Terry Lyons. Neural sdes as infinite-dimensional gans. arXiv preprint arXiv:2102.03657 , 2021.

[Kid22] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02435 , 2022.

[KMFL20] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv preprint arXiv:2005.08926 , 2020.

[KPH + 20] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations , 2020.

[LLJ16] Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In International conference on machine learning , pages 276284. PMLR, 2016.

[LSC + 21] Maud Lemercier, Cristopher Salvi, Thomas Cass, Edwin V Bonilla, Theodoros Damoulas, and Terry J Lyons. Siggpde: Scaling sparse gaussian processes on sequential data. In International Conference on Machine Learning , pages 6233-6242. PMLR, 2021.

[LSD + 21] Maud Lemercier, Cristopher Salvi, Theodoros Damoulas, Edwin Bonilla, and Terry Lyons. Distribution regression for sequential data. In International Conference on Artificial Intelligence and Statistics , pages 3754-3762. PMLR, 2021.

[LSZ15] Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference on machine learning , pages 1718-1727. PMLR, 2015.

[LWCD20] Xuechen Li, Ting-Kam Leonard Wong, Ricky TQ Chen, and David K Duvenaud. Scalable gradients and variational inference for stochastic differential equations. In Symposium on Advances in Approximate Bayesian Inference , pages 1-28. PMLR, 2020.

[LWL + 20] Junyi Li, Xintong Wang, Yaoyang Lin, Arunesh Sinha, and Michael Wellman. Generating realistic stock market order streams. Proceedings of the AAAI Conference on Artificial Intelligence , 34(01):727-734, 2020.

[Lyo98] Terry J Lyons. Differential equations driven by rough signals. Revista Matemática Iberoamericana , 14(2):215-310, 1998.

[MKYL22] James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons. On the choice of interpolation scheme for neural cdes. Transactions on Machine Learning Research , 2022(9), 2022.

[MS13] Edgar C Merkle and Mark Steyvers. Choosing a strictly proper scoring rule. Decision Analysis , 10(4):292-304, 2013.

[MSK + 20] James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neural rough differential equations for long time series. arXiv preprint arXiv:2009.08295 , 2020.

[MSKF21] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning , pages 7829-7838. PMLR, 2021.

[NMK + 18] Adamantios Ntakaris, Martin Magris, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis. Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods. Journal of Forecasting , 37(8):852-866, 2018.

[NSSV + 21] Hao Ni, Lukasz Szpruch, Marc Sabate-Vidales, Baoren Xiao, Magnus Wiese, and Shujian Liao. Sig-wasserstein gans for time series generation. In Proceedings of the Second ACM International Conference on AI in Finance , pages 1-8, 2021.

[NSW + 20] Hao Ni, Lukasz Szpruch, Magnus Wiese, Shujian Liao, and Baoren Xiao. Conditional sig-wasserstein gans for time series generation. arXiv preprint arXiv:2006.05421 , 2020.

[Opp19] Manfred Opper. Variational inference for stochastic differential equations. Annalen der Physik , 531(3):1800233, 2019.

[PADD21] Lorenzo Pacchiardi, Rilwan Adewoyin, Peter Dueben, and Ritabrata Dutta. Probabilistic forecasting with conditional generative networks via scoring rule minimization. arXiv preprint arXiv:2112.08217 , 2021.

[PD22] Lorenzo Pacchiardi and Ritabrata Dutta. Likelihood-free inference with generative neural networks via scoring rule minimization. arXiv preprint arXiv:2205.15784 , 2022.

[SCC21] Zijian Shi, Yu Chen, and John Cartlidge. The lob recreation model: Predicting the limit order book from taq history using an ordinary differential equation recurrent neural network. Proceedings of the AAAI Conference on Artificial Intelligence , 35(1):548-556, 2021.

[SCF + 21] Cristopher Salvi, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang. The signature kernel is the solution of a goursat pde. SIAM Journal on Mathematics of Data Science , 3(3):873-899, 2021.

[SGS18] Carl-Johann Simon-Gabriel and Bernhard Schölkopf. Kernel distribution embeddings: Universal kernels, characteristic kernels and kernel metrics on distributions. The Journal of Machine Learning Research , 19(1):1708-1736, 2018.

[SLG22] Cristopher Salvi, Maud Lemercier, and Andris Gerasimovics. Neural stochastic pdes: Resolution-invariant learning of continuous spatiotemporal dynamics. Advances in Neural Information Processing Systems , 35:1333-1344, 2022.

[SLL + 21] Cristopher Salvi, Maud Lemercier, Chong Liu, Blanka Hovarth, Theodoros Damoulas, and Terry Lyons. Higher order kernel mean embeddings to capture filtrations of stochastic processes. arXiv preprint arXiv:2109.03582 , 2021.

[Smi39] Nikolai V Smirnov. On the estimation of the discrepancy between empirical curves of distribution for two independent samples. Bull. Math. Univ. Moscou , 2(2):3-14, 1939.

[SSDK + 20] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

[TR19] Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883 , 2019.

[VBP + 20] Svitlana Vyetrenko, David Byrd, Nick Petosa, Mahmoud Mahfouz, Danial Dervovic, Manuela Veloso, and Tucker Balch. Get real: Realism metrics for robust limit order book market simulations. In Proceedings of the First ACM International Conference on AI in Finance , pages 1-8, 2020.

[VKK21] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems , 34:11287-11302, 2021.

[WD22] George Wynne and Andrew B Duncan. A kernel two-sample test for functional data. Journal of Machine Learning Research , 23(73):1-51, 2022.

[YJVdS19] Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems , 32, 2019.

A Signature Kernel Scores

Proof of Proposition 3.3

Proof (Appendix). The general result was first shown in [GR07]. We first show that ϕ sig is proper. By Proposition 3.1 the signature kernel is positive definite and characteristic on P ( K ) . It remains to show that E y ∼ Q [ ϕ sig ( Q , y )] ≤ E y ∼ Q [ ϕ sig ( P , y )] . This means we must have

Writing M = 1 2 P + 1 2 Q , a modification of Theorem 2.1 in Berg et al. [BCR84] (pg. 235) gives that

where P ⊗ Q denotes the natural product measure on K×K . Re-arranging (11), one arrives at the desired result.

To show strict properness, we need to show that E y ∼ Q [ ϕ sig ( Q , y )] ≤ E y ∼ Q [ ϕ sig ( P , y )] holds with equality iff P = Q for all P , Q ∈ P ( K ) . Suppose that there exists another P ′ ∈ P ( K ) such that E y ∼ Q [ ϕ sig ( Q , y )] = E y ∼ Q [ ϕ sig ( P ′ , y )] . Then we would have that

or that D k sig ( P ′ , Q ) = 0 , which is only true if P ′ = Q due to characteristicness of the kernel k sig .

Proof of Proposition 3.4

Proof (Appendix). The proof follows directly from [GBR + 12], Lemma 6. Note that an unbiased estimator for E x,x ′ ∼ P [ k sig ( x, x ′ )] from i.i.d samples ( x 1 , . . . , x m ) , x i ∼ P is given by the U-statistic

̸

Moreover, an unbiased estimate of E x ∼ P [ k sig ( x, y )] is given by

Writing ˆ ϕ sig ( P , y ) = T 1 U ( x 1 , . . . , x m ) -2 T 2 U ( x 1 , . . . , x m , y ) completes the proof.

B Experiments

All experiments were run on a NVIDIA GeForce RTX 3070 Ti GPU, except the experiment in Section 4.4 for which the NSPDE model was trained using a NVIDIA A100 40GB GPU.

Here we provide details for each of the experiments outlined in the body of the paper. We also provide some extra methods of evaluation aside from the KS test. These include the following:

Qualitative plot : We give a plot of samples P X θ from a trained generator against the true data measure P X true .

Autocorrelation : To measure temporal dependencies or correlations, we leverage the autocorrelation function

where µ is the average of the path X t over [0 , N ] and σ 2 is the corresponding variance. We provide a qualitative plot of ACF ℓ for each generator against the real data measure. We also provide a table summarizing the scores for some of the earliest lags ℓ ∈ N .

Cross-correlation : We provide average cross-correlation scores ( r t , r 2 t,ℓ ) between the returns process associated to X t ∼ P X θ and the squared, lagged returns process r 2 t,ℓ . We present the scores in matrix form. Finally, we provide the MSE between the matrix obtained from P X true and those obtained from each generator.

We make a note here that each of the three discriminators performed similarly in the additional quantitative metrics omitted from the body. Finally, we wish to first make the following general notes about each of the three methods studied in this paper:

Speed : Training a Neural SDE with respect to ϕ N sig was the fastest, followed by the SDEGAN, and finally with ϕ sig being the slowest. It was possible to decrease training time respect to the latter by using a coarser dyadic refinement in the PDE solver. However, we felt that achieving more accurate gradients at the cost of longer training time was worthwhile.

Stability : The Wasserstein SDE-GAN was the least stable, in terms of the difficulty in obtaining a training instance where the loss converged in reasonable time. Even with finetuning of both generator and discriminator parameters, the loss associated to the SDE-GAN tended to oscillate, making obtaining a converged model a very difficult task with the hardware available to us.

Scaling : All of the results in the paper are sensitive to path scalings; moreso with the signature kernel-based approaches, less so with the Wasserstein approach. The basic idea is as follows: the signature kernel-based methods will tend to fail if paths are scaled too low (resulting in lower-order terms dominating the calculation of k sig) or too high (the sum, although finite, can exceed a 64-bit float quite easily). Path scaling (and transformations) form an integral part in training a successful generative model, and we have tried to be as descriptive as possible regarding this matter. The details as to why scalings matter have been touched upon in [CLX21]; we intend to expand upon this in a future work.

Standardisation : On a similar note, standardizing path data before training was often found to improve the stability of training in any setting. By standardization we are referring to transforming each marginal of paths X ∼ P X true via the transformation ˆ X t = ( X t -µ T ) /σ T , where µ T = E P X true [ X T ] and σ T = E P X true [( X T -µ T ) 2 ] . By having the terminal marginal distributed standard normal, the task of finding suitable path scalings and smoothing parameters in the RBF kernel was made much simpler, as this task became less problemspecific.

B.1 Geometric Brownian motion

Data processing and hyperparameters To generate our data measure, we simulate 32768 paths according to eq. (9) using the torchsde package. These were solved over the interval [0 , 64] by setting y 0 = 1 , µ = 0 , σ = 0 . 2 , with dt = 0 . 1 . Paths were then interpolated along the grid ∆ = { 0 , 1 , 2 , . . . , 63 } , so each element of the training set had total length 64 . Stochastic integrals were taken in the Itô sense and the driving noise W was taken in the general sense. We used the SRK method to solve the corresponding SDE. Each path is time-augmented, so ˆ X t = ( t, X t ) at each point on the grid. After we have simulated our dataset, we standardized each path as outlined in the dot points above.

Generator hyperparameters The generator is a Neural SDE with vector fields µ θ : [0 , T ] × R y → R y and σ θ : [0 , T ] × R y × w → R y taken to be neural networks with 1 hidden layer, and 16 neurons in said layer. As per [KFL + 21] the LipSwish activation function was used to ensure the Lipschitz condition held on the vector fields of the Neural SDE. We also used the final tanh regularisation which we found was necessary for training success. Thus we have that

The size of the hidden state of the neural SDE was chosen to be y = 8 , and the noise dimension was chosen to be w = 3 . Stochastic integration was taken in the Itô sense and we set dt = 1 over [0 , 63] . As we are not learning an initial distribution in this instance, we modified the generator architecture to have ξ θ ( X 0 ) = a for some a ∈ R , where ξ θ is the network acting on the initial condition. Before passing to the discriminator, both generated and real paths were translated to start at 0 .

Discriminator hyperparameters For the signature kernel-based discriminators, we applied the time normalisation transformation so the time component of both the real and generated paths was over [0 , 1] as opposed to [0 , 63] . This was to ensure each channel of the generated and real data evolved over a similar scale. For training with respect to ϕ sig, we set the order of dyadic refinement associated to the PDE solver for the signature kernel to 1 . We also used three different kernels, corresponding to three different scalings of the paths, for increased expressivity. For ϕ N sig , we set the order of truncation equal to N = 3 . Finally, for the SDE-GAN, we chose the drift and diffusion vector fields to be feed-forward neural networks

matching that from the generator.

Training hyperparameters All methods used a batch size of 128 and the Adam optimisation algorithm for backpropagating through the generator optimisers, except for the SDE-GAN, which as suggested by the authors we used Adadelta. As a remark, we did not see much difference in using either Adam, Adadelta, or RMSProp, although we did see poorer performance using pure SGD, with or without momentum. Learning rates were roughly proportional to the average size of the batched loss: as a rough guide, proportionality like η G ×L ( θ ) ≈ 10 -5 tended to yield good results, with the generator learning rate being around η G ≈ 10 -4 for the signature kernel(s), and η D ≈ η G × 10 -2 for the SDE-GAN. As mentioned in the body, we trained for 4000 steps with ϕ sig, 10000 with ϕ N sig and 5000 with the SDE-GAN to normalise for training time.

Pxirue

Pxurue

Pxtre

Px8

4

Px8

Px8

3

3

3

2

2

2

X

X

1

1

1

0

0

0

-1

0

20

40

60

0

20

40

60

0

20

40

60

t

t

t

Table 5 gives the autocorrelation scores for the first five lags for each of the three models, along with plots of the mean ACF values in Figure 5 and the associated 95% confidence intervals. We can see that all three models do well at capturing temporal effects, with the Neural SDE trained with respect to ϕ sig most closely matching the data measure, except in the first lag.

Finally, we present the cross-correlation matrices between the returns process r t and the lagged squared returns process r 2 t -l for lags l = { 0 , 1 , 2 , 3 , 4 , 5 } . Again all models tend to perform quite well in that they match relational dynamics observed in the data measure. Table 6 gives the MSE between the generated matrices and the data matrix. We see that the Neural SDE trained with ϕ sig

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.2

Pxure

0.2

Pxtrue

0.2

Pxtrue

Px8

Pxe

Px8

0.4

0.4

0.4

0

5

10

0

5

10

0

5

10

1

1

achieves the lowest score of the three, however again performance is strong regardless of method used for training.

(a) Data measure.

(b)

ϕ

sig

(c)

ϕ

N

sig

(d) SDE-GAN.

0.286

0.256

0.346

0.229

0.045

0.044

0.029

0.052

0.073

0.109

0.047

0.063

0.034

0.051

0.047

0.051

0.033

0.049

0.046

0.078

0.108

0.032

0.042

0.063

0.048

0.036

0.053

0.04

0.034

0.023

0.032

0.049

0.059

0.063

0.075

0.108

0.033

0.049

0.042

0.059

0.031

0.036

0.032

0.058

0.043

0.042

0.013

0.024

0.032

0.051

0.044

0.081

0.059

1400

0.107

0.053

0.047

0.052

0.049

0.061

0.048

0.034

0.033

0.03

0.054

0.041

0.046

10'0

0.015

0.026

0.038

0.052

0.053

0.05

0.077

0.055

0.07

0.103

0.039

0.037

0.043

0.051

0.048

0.063

B.2 Rough Bergomi

Data processing and hyperparameters We simulate 32768 paths to make up our data measure via the rBergomi Python package 5 . We fixed the time window to be [0 , 2] , and specified dt = 1 / 32 , so paths were of length 64 . We chose ( ξ 0 , η, ρ, H ) = (0 . 04 , 1 . 5 , -0 . 7 , 0 . 2) and set d = 1 . Paths started at 1 . As always, paths were time-augmented. Paths were normalised to start at 0 via translation and were standardized again according to the terminal data from the train set. A final point is that although the data was generated over [0 , 2] , the time grid passed to the generators in an optimiser step was ∆ = { 0 , 1 , . . . , 63 } . We found that this improved performance.

Generator hyperparameters Given the increased complexity of the data generating model, we increased the expressivity of the vector fields governing the drift and diffusion vector fields µ θ and σ θ . This was done by increasing the depth and width of the constituent feed-forward networks to include 3 hidden layers of size 32 . We also increased the size of the hidden state to y = 16 and the

noise dimension to w = 8 . Thus

µ θ ∈ NN (17 , 32 , 32 , 32 , 16; LipSwish , LipSwish , LipSwish , tanh)

and

σ θ ∈ NN (17 , 32 , 32 , 32 , 128; LipSwish , LipSwish , LipSwish , tanh) .

Discriminator hyperparameters For training with respect to ϕ sig, we mapped path state values to ( H , κ ) where κ denotes the RBF kernel on R 2 . We set associated the smoothing parameter σ = 1 . For ϕ N sig , we increased the truncation level to N = 5 . In both these settings we again applied the time normalisation transformation on both the generated and data measure paths before being passed through the loss function. For the SDE-GAN, we increased the expressiveness of the vector fields governing the Neural CDE in the same way as we did the Neural SDE.

Training hyperparameters Learning rates for the Neural SDE trained according to ϕ sig was set to η G = 1 × 10 -4 . Due to the increasing number of terms in the expected signature for the truncated MMDapproach, we had to reduce the learning rate to η G = 1 × 10 -6 - larger values caused instability in the training procedure. The SDE-GAN was again quite difficult to train, however we were able to have some success by setting η D ≈ 2 × 10 -3 and η G ≈ 1 × 10 -3 . Initialisation of the generator vector fields was especially important for the Wasserstein method, as initialisation too far from the data measure cause oscillatory patterns in the training loss, which leads to more epochs required for the loss to converge. We again used the Adam optimisation algorithm for the MMD-based discriminator/generators and Adadelta for the SDE-GAN. We trained for the same number of steps as per the gBm method.

Results Figure 7 gives a qualitative plot of the simulated paths.

Pxtrue

Pxtre

2

Pxe

一

0

X

0

0

1

-1

-1

-2

IPxtrue

Px8

0

20

40

60

0

20

40

60

0

20

40

60

t

t

t

Figure 8 gives the same plot of the ACF scores at corresponding lags for the data measure and each of the generated models, along with the 95% confidence interval. Table 7 explicitly gives these scores.

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4

ACF

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

Pxure

Pxtrue

Pxtrue

0.2

0.2

0.2

Pxe

Pxe

Pxe

0.4 E

0

5

10

0.4

0

5

10

0.4

0

5

10

1

1

Finally, we present the same cross-correlation matrices, along with the MSE between either of the three generators and the data measure. Although each of the models perform well, the generator trained with ϕ sig achieves the best results.

(a) Data measure.

(b)

ϕ

sig

(c)

ϕ

N

sig

(d) SDE-GAN

0.388

0.358

0.241

0.192

0.067

101'0

0.056

640°0

0.054

0.1

0.058

0.074

0.048

0.076

0.1

0.055

0.066

6400

0.039

0.052

0.101

0.031

0.06

0.076

0.037

0.065

8400

0.109

0.05

0.057

0.066

0.08

0.031

0.064

0.054

0.101

0.047

0.044

0.058

0.075

0.044

0.036

0.064

0.081

0.105

0.041

0.056

0.056

0.066

0.08

4400

0.062

0.073

0.056

0.102

0.051

0.039

0.039

0.063

0.07

0.026

0.022

0.037

0.064

0.078

0.104

440'0

0.046

0.054

0.056

0.069

0.088

0.04

0.031

190'0

0.074

0.048

0.095

0.035

0.022

0.038

0.036

0.076

0.078

B.3 Multidimensional real data

We now give the details regarding the unconditional generation of foreign exchange data.

Data processing and hyperparameters Data is given by hourly returns associated to the currency pairs EUR/USD and USD/JPY. We stride the concatenated time series (corresponding to close prices at each time) into paths of length ℓ = 64 . Paths are normalized to start at 1 . We augmented the state values with their original timestamps, (as epoch seconds). Call this dataset Y . As we are dealing with financial data, one can expect the time intervals between prices to be irregular. This is not usually an issue when using signature methods. However, in this setting we found training to be less stable if paths were not normalised to evolve over the same time grid as that which the neural SDE was solved over in a forward pass.

To circumvent this issue, for every y ∈ Y we find the median terminal time ˜ T , where

All paths whose terminal time is greater than ˜ T were filtered out of the dataset which we call ˜ Y . We then define an evenly-spaced time grid ∆ ∗ = { 0 , . . . , ˜ T } containing 64 observations in total, and linearly interpolate each y ∈ ˜ Y over this grid, where we use these interpolated coefficients to

form our train and test sets. Again we standardize using the terminal values of the train set data. We simulate our generators over the time grid ∆ = { 0 , . . . , 63 } as we found that using ∆ ∗ , the realistic time-grid (in fractions of a year) induced little variability in the generated paths; i.e., the quadratic variation associated to the generated paths was significantly lower than that obtained from the real data measure.

Generator hyperparameters The generator maintains the same architecture as outlined in the rBergomi section. We tried increasing the size of the hidden state to x = 32 and the noise state w = 16 but found that this had little impact on training performance. We also found that increasing the expressivity of the neural vector fields did not overly impact performance; neither refining the mesh over which the Neural SDE was solved.

Discriminator hyperparameters We used the same discriminator hyperparameters for each of the three methods as per the rBergomi section.

Training hyperparameters We used the same batch size ( 128 ) as per the previous sections. We allowed for increased training time here, training with respect ϕ sig for 4000 steps, 15000 for ϕ N sig and 10000 for the SDE-GAN. The same learning rate parameters were used as well. We did not use any learning rate annealers. The Adam optimisation algorithm was employed for the MMD-based generators, whereas again Adadelta was used for the Wasserstein case.

Results Extended results are provided as per the previous sections. We being with the qualitative plots.

(a) EUR/USD

(b) USD/JPY

3

Pxire

3

Pxtrue

3

Pxure

Px8

Px8

Pxe

2

2

0

0

20

40

60

3E

0

20

40

60

3E

0

20

40

60

t

t

t

IPxure

Pxtrue

IPxtrue

4

Pxe

Px8

Pxe

3

18

3

2

乙

D

0

-1

-1

-2

-2

-2

3E

3

0

20

40

60

0

20

40

60

0

20

40

60

t

t

t

Visual inspection gives that the generator trained with respect to ϕ sig appears to have most accurately captured the data measure, in particular the less regular, outlier paths. In contrast the SDE-GAN and truncated kernel methods tend to over-represent the mean element. For the GAN this could be the 'mode collapse' phenomenon in effect, whereas in the case of the Neural SDE trained with ϕ N sig , it is likely that higher-order terms cannot be discarded if one wishes to accurately model the data measure.

We now provide the plot associated to the ACF scores obtained from training with respect to each generator, along with the summarizing table. Table 9 shows that that each of the discriminators perform relatively well, aside from the EURUSD autocorrelative factors obtained via traning the Neural SDE in the SDE-GAN framework. Finally we give the cross-correlation matrices and the associated MSEs. The Neural SDE trained with respect to ϕ sig appears to perform the best by this evaluation metric.

(a) EUR/USD

(b) USD/JPY

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

ACF

0.4

ACF

0.4

ACF

0.4

0.2

0.2

0.2

0.0

0.0

0.0

Pxrme

Pxure

Pxure

0.2

Pxe

0.2

Pxe

0.2

IPxe

0.4

0.4

0

5

10

0

5

10

0

5

10

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

ACF

0.4

ACF

0.4

ACF

0.4

0.2

0.2

0.2

0.0

0.0

0.0

Pxtre

Pxtre

IPxtrue

0.2

0.2

0.2

Pxe

Pxe

Pxe

0.4

0.4

0.4

0

5

10

0

5

10

0

5

10

1

(a) EUR/USD

(b) USD/JPY.

(a) EUR/USD

(b)

ϕ

sig

(c)

ϕ

N

sig

(d) SDE-GAN

(e) USD/JPY

(f)

ϕ

sig

.

(g)

ϕ

N

sig

(h) SDE-GAN

0.299

0.251

0.222

0.372

0.059

0.095

0.0720.058

0.057

0.064

0.047

0.083

0.033

0.084

0.093

0.051

0.06

0.053

0.044

0.061

0.058

0.035

0.0490.081

0.024

0.037

0.082

0.095

0.062

0.052

0.066

0.052

0.03

0.071

0.0620.058

0.06

6400

0.048

0.078

0.009

0.003

0.034

0.084

0.091

0.032

0.04

0.044

0.062

0.054

0.028

0.048

0.062

0.058

0.053

0.029

0.058

0.078

0.047

0.075

0.032

0.03

0.003

0.034

0.085

0.094

0.04

0.036

0.039

0.043

0.058

0.052

0.049

0.053

0.0490.061

0.057

0.054

0.029

0.05

0.058

0.078

0.044

0.072

0.26

0.275

0.153

0.267

0.047

0.08

0.058

0.07

0.034

0.054

0.046

0.059

0.036

0.064

0.076

0.044

0.059

0.077

0.029

0.052

0.053

0.042

0.02

0.064

0.019

0.045

0.063

640°0

0.038

0.035

0.06

0.072

0.033

0.039

0.05

0.051

0.039

0.032

0.021

0.064

0.023

0.025

0.049

0.059

0.077

0.027

0.038

0.044

0.056

0.082

0.032

0.0480.037

0.049

0.05

0.049

0.043

0.03

0.018

0.057

0.016

0.0190.034

0.053

0.071

0.083

0.037

0.024

0.038

0.047

0.061

0.079

0.029

0.032

0.048

0.034

0.049

0.048

0.026

0.0110.043

0.032

0.012

0.053

B.4 Conditional generation

In this section we describe in detail the training procedure for the conditional generator.

Problem setting The conditioning variables are given by path segments x : [ t 0 -dt, t 0 ] → R 2 , representing time-augmented asset price values. At time t 0 , one wishes to make a prediction about the resultant path y : [ t 0 , t 0 + dt ′ ] → R 2 conditional on x . Here, dt, dt ′ are hyperparameters describing how much of the past one wishes to consider and how far into the future one wishes to forecast. With x ∼ Q We thus want to train a conditional generator so that P X θ ( ·| x ) = P X real ( ·| x ) . We briefly state the three major difficulties associated to this generation problem:

Unobservable true conditional distribution . In practice, one never observes the entire true conditional distribution P X real ( ·| x ) : only a sample from it. This means that classical metrics on path space (MMD, Wasserstein, and so on) cannot be used without modification, or making assumptions about the relationship between the conditioning and resultant paths.

Using paths as conditioning variables . It is not immediately clear what is the best way to consume a path as a conditioning variable for a given generator.

'Unseen' conditioning variables . It is not guaranteed that the conditional generator will behave in an expected way if an as-yet unseen conditioning variable is provided (by unseen, we are referring to within the training procedure). These conditioning variables are often the ones of interest.

Our procedure attempts to solve the first and second problems with our architecture choices on the conditional generator, and our choice of loss function. The third issue is omnipresent in conditional modelling.

The generator now is given by a (conditional) Neural SDE with architecture given by

for µ θ : [ t 0 , t 0 + dt ′ ] × R y × R d C → R y , σ θ : [ t 0 , t 0 + dt ′ ] × R x × R d C → R x × w , ξ θ : R x × R d C → R y , and π θ : R y × R d C → R x . Here, C ( x ) denotes the function acting on the conditioning path and encoding it as a vector in R d C . A natural way to perform this encoding is via the truncated signature S M ( X ) of the path x . In this way the neural networks defining the vector fields in the generator (for instance) are now mappings

We note here that all of the regularity conditions required to ensure a strong solution to the standard Neural SDE here remain satisfied; we are only augmenting each of the trainable components to accept the encoded conditioning path. We also note here that this technique is flexible enough to include any amount of R d -valued conditioning variables.

Data processing and hyperparameters Data comes from 15-minute close prices associated to the EUR/USD price pair. We again extracted paths of length 48 (normalising for erroneous terminal times as per the unconditional setting) and split these paths into conditioning-resultant pairs { x i , y i } N i =1 with x i representing the first 32 observations and y i the next 16. We normalized both sets of paths by their initial value. Instead of standardizing, in this setting we scaled all path values up by a factor of 100 . We found this was crucial so that the lower-order signature terms did not overly contribute to the value of the signature kernel. Both conditioning and resultant paths were then translated to start at 0 . In total the dataset size was comprised of 52428 conditioning/resultant pairs.

Generator hyperparameters The generator is a conditional Neural SDE. Stochastic integration was again taken in the Itô sense and we used the Euler method. The noise size was set to w = 8 , the size of the hidden state was taken y = 16 . The MLPs governing the vector fields were the same as per the rBergomi and multidimensional unconditional examples, except we increased the width of the layers in the neural networks to 64 neurons. We conditionalized the input paths via the truncated log-signature of order 5 . In order to estimate the batched loss, we need to specify the size of the conditional distribution P X θ ( ·| x ) output by the conditional generator, which we set to 32 paths. Finally, we applied the time normalisation and lead-lag transformations to the input paths x before taking their truncated log-signature.

Discriminator hyperparameters We trained with respect to ϕ sig. Again we lifted paths via the RBF kernel and chose the smoothing parameter σ = 1 . We used a dyadic refinement level of 1 for the PDE solver associated to k sig. All paths had the time normalisation transformation applied to them before having the loss evaluated.

Training hyperparameters We set the batch size equal to 128 and trained the conditional generator for 10000 steps. We set the learning rate η G = 2 × 10 -6 and used the Adam optimisation algorithm in PyTorch. No learning rate annelears were used.

Results Results are presented in the body of the paper.

B.5 Simulation of limit order books

The Neural SPDE model introduced in [SLG22] extends Neural SDEs to model spatiotemporal dynamics by parametrising the differential operator, drift and diffusion of SPDEs of the type

where both µ and σ are local operators acting on the function Y t that is, µ ( Y t )( x ) and σ ( Y t )( x ) only depend on Y t ( x ) . Moreover, it is assumed that L is a linear differential operator generating a semigroup e t L which can be written as a convolution with a kernel K t .

Let D ⊂ R d be a bounded domain. Let W : [0 , T ] → L 2 ( D, R d w ) be a Wiener process and a an L 2 ( D, R d a ) -valued Gaussian random variable. The values d w , d a ∈ N are hyperparameters describing the size of the noise. A Neural SPDE is a model of the form

for t ∈ [0 , T ] and x ∈ D where Y : [0 , T ] → L 2 ( D, R d y ) is the mild solution, if it exists to the SPDE in Equation (12) with regularised driving noise W ϵ and where ∗ denotes the convolution in space with the kernel K t : D × D → R d y × d y (see [SLG22] for more details). Similarly to the Neural SDE model,

are feedforward neural networks. Imposing globally Lipschitz conditions (by using ReLU or tanh activation functions in the neural networks µ θ and σ θ ) ensures the existence and uniqueness of the mild solution Y . Finally, we note that in [SLG22], the authors propose two distinct algorithms to evaluate the Neural SPDE model based on two different parameterisations of the kernel K .

Next, we provide more details on how we trained such a Neural SPDE model to generate Limit Order Book (LOB) dynamics [GPW + 13]. The increasing availability of LOB data has instigated a significant interest in the development of statistical models for LOB dynamics. In recent years, new models based on SPDEs have been proposed to accurately describe and analyse these complex dynamics [HKN20, CM21].

Data processing and hyperparameters We used real LOB data from the NASDAQ public exchange made publicly available in [NMK + 18] which consists of about 4 million timestamped events over 10 consecutive trading days with L = 10 price levels on each side (bid and ask) of the LOB. Three versions of this dataset are provided, each normalised using a different technique. We used the data normalised with z-scores and split the LOB trace into sub-traces of length T = 30 .

Generator hyperparameters The generator is a Neural SPDE driven by a cylindrical Wiener process W with d w = 2 . The vector fields µ θ and σ θ are taken to be single layer perceptrons with d y ∈ { 16 , 32 } followed by batch normalization and tanh activation function. Thus we obtain

We used the second evaluation method proposed in [SLG22, Section 3.3] with 4 Picard's iterations and maximum number of frequency modes in { 10 , 20 } in the spatial direction and fixed to 20 in the temporal direction. Instead of sampling the initial condition a from a L 2 ( D, R d a ) -valued Gaussian, we simply used the samples from X true 0 , in which case d a = 1 .

Discriminator hyperparameters We integrated in time the output trajectories from the generator, as we observed this yielded more stable kernel scores. We mapped the path state values into H κ where κ denotes a SE-T kernel on L 2 ( D ) with D = [0 , 1] , that is, a kernel defined for all f, g ∈ L 2 ( D ) by κ ( f, g ) = e -1 2 σ 2 ∥ T ( f ) -T ( g ) ∥ 2 Y where T : L 2 ( D ) → Y is a Borel measurable, continuous and injective map. We considered three SE-T kernels respectively termed ID, SQR and CEXP:

(ID) SE-T kernel with T : L 2 ( D ) → L 2 ( D ) defined for all f ∈ L 2 ( D ) by T ( f ) = f

(SQR) SE-T kernel with T : L 2 ( D ) → L 2 ( D ) ⊕ L 2 ( D ) defined by T ( f ) = ( f, f 2 )

(CEXP) SE-T kernel with T : L 2 ( D ) → L 2 ( D ) defined by T ( f ) = C F,l ( f ) where C F,l is the covariance operator associated to the kernel k F,l defined for all x, x ′ ∈ D by

For ID and SQR, we used σ ∈ { 1 , 10 } , and for CEXP we used ( σ, l, F ) ∈ { (1 , 1 , 5) , (10 , 10 , 5) } . We then used a dyadic order of 1 to compute the signature kernel.

Training hyperparameters We set the learning rate of the generator η G to be 1 × 10 -3 and trained it for a maximum number of 1 500 epochs. We used a batch size of 64 due to memory constraints and the Adam optimizer with the default parameters of PyTorch.

Peer reviews:
comments: SUMMARY:
This paper proposes a signature kernel-based Neural SDE by using signature kernel scores. The proposed method eliminates existing state-of-the-art methods' mode collapse and instability using adversarial training.

STRENGTHS:
1. In all experiments, the proposed method outperforms existing methods.
2. The explanation in chapter three is clear and reproducible because the code is also provided.

WEAKNESSES:
1. This paper's main contribution includes using signature kernel scores for the discriminator, but there needs to be more motivated to use signature kernels.
2. There is no sub-caption for each of Figure 1, so it is unclear which generator it is.

QUESTIONS:
1. Is that why you chose the signature kernel among the various kernels? Also, if different kernels are adaptable, I would like to know the results of your experiments with other kernels.
2. Which generator does the sub Figure in Figure 1 correspond to?

Minor corrections

3. The unconditional_nsde.ipynb included in the code provided need to be corrected?
```
#cond_                    = data_type == "rBergomi"
cond_                    = data_type == "gbm"
```

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors proposed to generate time-series using neural stochastic differential equations using the scoring rules-based training objective which is on signature paths computed from signature kernels. Combining the idea of generative adversarial network, this work also employ the generator-discriminator pair, but solve them as a system of linear PDEs where the adjoint method is applied to reduce the memory cost. Experiments are conducted accordingly on sequential data, including financial datasets.\n\nSTRENGTHS:\n1. This paper is written in high clarity. All formulations and derivations are clearly elaborated.\n\n2. The idea is promising and novel to combine generative adversarial networks and neural stochastic differential equations, avoiding adversarial training which can cause instability in training.\n\n3. Rough path theory is a relatively new topic in mathematics. It has penetrated in the machine learning community since only 2019. It is interesting to utilize it in time series generation in machine learning.\n\nWEAKNESSES:\n1. In experimental design, the baseline models are not sufficiently explored. The experiments should  include diffusion models for time series generation, as they are commonly used generative models nowadays as a very special variant of SDE and would provide a useful comparison to the proposed approach.\n\n2. The space complexity and time complexity seem to raise the concerns, as signature transform is applied where the dimensionality is largely increased.\n\n3. The scale of the datasets in the experiments are of relatively low dimensionality in time series dat. It would be hard to prove the efficiency of the adjoint method applied to reduce the memory cost.\n\nQUESTIONS:\n1. How should the \"order\" of the signature, k, be selected? Is there any specially designed optimization method to help with it?\n\n2. Why are the neural networks in neural SDE the regular neural networks? I am very curious if more sophisticated deep learning models, such as u-nets, are examined and what their impacts to the performance are.\n\nLIMITATIONS:\nThe limitations of this work is not adequately addressed. It can be possible to consider the application scope of this work, where it is only for time series generation, especially for the cases where the dimensionality is not high.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper introduces a novel approach for training Neural SDEs (Stochastic Differential Equations) as generative models for sequential data without using adversarial techniques. The authors propose a novel class of scoring rules based on signature kernels, which offer stability and avoid issues like mode collapse that are common in GAN-based training. The new formulation allows for memory-efficient adjoint-based back-propagation and enables the generation of spatiotemporal data, demonstrating superior performance over alternative methods in various tasks, including simulation of rough volatility models, conditional probabilistic forecasts of forex pairs, and mesh-free generation of limit order book dynamics.

STRENGTHS:
- The authors propose a new method with rigorous theoretical justification and verify it with numerical experiments.

- The proposed method does not require any adversarial training, so presumably training the generative model with the method is more stable than GANs.

- The proposed method achieves significantly better performance than SDE-GAN without adversarial training.

WEAKNESSES:
I am not expert of this field, stochastic differential equations (SDE) for continuous-time generative models. So it is difficult to point out some critical weakness of the proposed method.

-  I highly recommend that authors elaborate the evaluation metrics (e.g. KS-score) in detail for those who are not familiar with this field.
`

QUESTIONS:
- Is the proposed method applicable to generating images?

- How is the proposed model sensitive to hyper-parameters and what would be rule of thumbs to tune those hyperparameters?

LIMITATIONS:
Since my expertise is not stochastic differential equation, it is hard for me to point out critical limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considered the training of neural SDEs, where are continuous-time generative models for sequential data. State-of-the-art approaches train neural SDEs in an adversarial manner and suffer from instability. In this work, a non-adversarial training approach is proposed for stable and effective training of neural SDEs. Particularly, a new class of scoring rules based on signature kernels is defined and used as objective for training. Extensive evaluations are performed to demonstrate the effectiveness of the proposed training objective.\n\nSTRENGTHS:\nThe differences between the related works and the proposed method are clearly discussed.\n\nThe proposed method seems solid in theory.\n\nThe possible extensions of the proposed method are extensively discussed.\n\nWEAKNESSES:\nThe significance of the work is somehow unclear to me. In the related work section, two works [PADD21] and [BO21] are discussed that are closely related to the proposed method. They are different from the proposed method in terms of the data type (discrete/continuous) and the setting of continuous-time processes (general/neural SDE specific). However, it is not clear how significant the proposed method is from the above two aspects. \n\nThe related work section is hard to follow. Lots of works are discussed from different perspectives: Neural SDE; score-based generative models; and scoring rules for generative networks. For me, these discussions are a bit disjointed. They are mainly discussed to show the differences between them and the proposed method. I think it is necessary to clearly locate the proposed method in the related literatures and then discuss. If there is a table or figure to visualize the overlaps between the proposed method and the other related methods, it will be clearer. \n\nThe proposed method is heavy in math. The notations are not clearly explained. For example, in line 95, what is Omega, F, and P? What is the law representing in line 97? More intuitive descriptions and clear definitions will greatly help the reading.  \n\nThough the mathematical foundation is solid, the empirical evaluation of the proposed method compared to SOTA methods is weak. Only SDE-GAN is considered for comparison. Score-based generative models (SGMs) are also leveraging scoring rules. Also, [BO21] introduced scoring rules for continuous-time processes. Since these works are closely related to the proposed method, it is necessary to compare with them. If can’t, it is necessary to explain the reasons, which I think also help clarify the significance of the proposed method. \n \nThere is a lack of experimental analysis. In section 4.1., there is no explanation or analysis for the values shown in Table 1. Why SDE-GAN is better at t=19? Similarly, in section 4.4, the conclusions of the evaluation discussed in section 4.4 are unclear to me. \n\nThe authors are claiming that the proposed method requires less memory compared with SOTAs. There is a lack of evaluation in terms of computational cost or the memory cost to support this claim.\n\nQUESTIONS:\nPlease see the details questions in the weakness section. I would love to change my rating if the authors can address my concerns about the significance and the evaluation.\n\nLIMITATIONS:\nThe limitations of the proposed method are well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to training Neural SDEs using non-adversarial methods based on signature kernel scores. The authors demonstrate that the signature kernel score is strictly proper and provide consistent estimators for such scores. The effectiveness of their approach is demonstrated in various tasks.\n\nSTRENGTHS:\n1. The method is novel as it offers a non-adversarial alternative to traditional GAN-based training methods for Neural SDEs. This work is likely the first to introduce the signature kernel method to the Neural-SDE model. It achieves superior performance compared to the GAN method while being easier to train.\n\n2. The theoretical part of the paper is well-written, with clear mathematical formulations and adequate definition of symbols. The proof of strict properness and consistency, although direct consequences of prior works, offers robust guarantees.\n\nWEAKNESSES:\n1. Limited comparisons: The authors claim that their procedure outperforms alternative ways of training Neural SDEs, but they only compare it with the SDE-GAN. In the related works, the authors mention the latent SDE, which is easier to train than SDE-GAN, and claim that it yields worse performance than the SDE-GAN. While I acknowledge that GAN generally has better model capacity than VAE, can you provide quantitative results to substantiate this claim? The original paper only compares with the latent ODE from 2019.\n\n2. Lack of visualizations: Although the paper visualizes the generated distribution of paths conditioned on a previous path using their model, it would be beneficial to include a comparison with the paths generated by the SDE-GAN. This would provide a clearer understanding of how your outputs capture more characteristics of financial markets. Furthermore, the generated paths exhibit a large variance, which could be addressed or explained.\n\n3. Explanation of the LOB experiments: The simulation of LOB dynamics and the decision not to use autoregressive prediction are interesting, but how is the proposed design better than the five autoregressive generators that are listed? The paragraph discussing this topic appears to be hastily written. The reported average scores alone do not provide sufficient information.\n\nQUESTIONS:\n1. Can you provide quantitative results to show latent-SDE performs worse than SDE-GAN and your model?\n2. Can you visualize conditional path distributions generated by SDE-GAN and explain why your generated paths are better? \n3. Could you provide more details on how your model is useful in limit order book (LOB) prediction and elaborate on the design choices you made?\n\nLIMITATIONS:\nThe authors have well discussed the limitations. The work assumes the noise is continuous without jump on event arrival, therefore is not applicable to noise governed by point processes. The work pose certain regularization conditions on the sampled paths. The work could benefit from including more varied evaluation metrics to assess the performance of the generated data.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-01"}