Neural SDEs Made Easy: SDEs are Infinite-Dimensional GANs
Authors: Patrick Kidger
James Foster
Xuechen Li
Harald Oberhauser
Terry Lyons
Venue:  (2021)
Keywords: neural differential equation
neural ODE
SDE
GAN

Abstract: Several authors have introduced \emph{Neural Stochastic Differential Equations} (Neural SDEs), often involving complex theory with various limitations. Here, we aim to introduce a generic, user friendly approach to neural SDEs. Our central contribution is the observation that an SDE is a map from Wiener measure (Brownian motion) to a solution distribution, which may be sampled from, but which does not admit a straightforward notion of probability density -- and that this is just the familiar formulation of a GAN. This produces a continuous-time generative model, arbitrary drift and diffusions are admissible, and in the infinite data limit any SDE may be learnt. After that, we construct a new scheme for sampling \emph{and reconstructing} Brownian motion, with constant average-case time and memory costs, adapted to the access patterns of an SDE solver. Finally, we demonstrate that the adjoint SDE (used for backpropagation) may be constructed via rough path theory, without the previous theoretical complexity of two-sided filtrations.

Under review as a conference paper at ICLR 2021
NEURAL SDES MADE EASY: SDES ARE INFINITE-DIMENSIONAL GANs
Anonymous authors
Paper under double-blind review
ABSTRACT
Several authors have introduced Neural Stochastic Differential Equations (Neural SDEs), often involving complex theory with various limitations. Here, we aim to introduce generic, user friendly approach to neural SDEs. Our central contribution is the observation that an SDE is a map from Wiener measure solution distribution, which may be sampled from, but
(Brownian motion) which does not admit a straightforward notion of probability density and that this is just the familiar formulation of GAN. This produces continuous-ting generative model, arbitrary drift and diffusions are admissible, and in the infinite data limit any SDE may be learnt. After that, we construct a new scheme for sampling and reconstructing Brownian motion, with constant average-case time and memory costs, adapted to the access patterns of an SDE solver. Finally, we demonstrate that the adjoint SDE (used for backpropagation) may be constructed via rough path theory, without the previous theoretical complexity of two-sided filtrations.
INTRODUCTION
Neural differential equations elegant concept, bringing together the two dominant modelling paradigms of neural networks and differential equations. Indeed, since their introduction, Neural Ordinary Differential Equations (Chen et al., 2018) have prompted the creation of wide variety of similarly-inspired models, for example based around controlled differential equations (Kidger et al., 2020b; Morrill et al., 2020), Lagrangians (Cranmer et al., 2020), higher-order ODEs (Massaroli et al., 2020; Norcliffe et al., 2020), and equilibrium points (Bai et al., 2019).
In particular, several authors have introduced Neural Stochastic Differential Equations (neural SDEs), such as Tzen & Raginsky (2019a); Li et al. (2020); Hodgkinson et al. (2020) among others.
1.1 RELATED WORK
We begin by discussing previous formulations, and applications, of Neural SDEs.
Tzen & Raginsky (2019a;b) obtain Neural SDEs a continuous limit of deep latent Gaussian models. They train by optimising variational bound, using corward-mode autodifferentiation. They consider only theoretical applications, for modelling distributions as the terminal value of an SDE.
Li et al. (2020) give arguably the closest analogue to the neural ODEs of Chen et al. (2018). They introduce neural SDEs via a subtle argument involving two-sided filtrations and backward Stratonovich integrals, but in doing so are able to introduce a backward-in-time adjoint equation, using only efficient-to-commuute vector-Jacobian products. In applications, they use neural SDEs in a latent variable modelling framework, using the stochasticity to model Bayesian uncertainty.
Hodgkinson et al. (2020) introduce Neural SDEs via elegant theoretical argument, as a limit of random ODEs. The limit is made meaningful via rough path theory. In applications, they use the limiting random ODEs, and treat stochasticity as regulariser within a normalising flow. However, they remark that in this setting the optimal diffusion is zero. This is a recurring problem: Innes et al. (2019) also train neural SDEs for which the optimal diffusion is zero.
Rackauckas et al. (2020) treat neural SDEs classical Feynman-KKe Kac fashion, and like Hodgkinson al. (2020); Tzen & Raginsky (201 9a;b), optimise a loss on just the terminal value of the SDE.

---

Under review as a conference paper at ICLR 2021
Briol al. (2020); Gierjatowicz et al. (2020) instead consider the more general case of using neural SDE to model time-varying quantity, for which the stochasticity in the system models the variability (specifically certain statistics) of time-varye marying data. Letting μ. denote the learnt and true distributions, both train by minimising (ı(()) v(f) for functions of interest f (such as derivative payoffs). This corresponds to training with non-characteristie MMD (Gretton al., 2013).
Several authors, such as Oganesyan et al. (2020); Hodgkinson et al. (2020); Liu et al. (2019), seek use stochasticity as a way to enhance regularise a neural ODE model.
Our approach is most similar to Li et al. (2020), in that we treat neural SDEs as learnt continuous- time model components of differentiable computation graph, and Briol et al. (2020); Gierjatowicz et al. (2020), in that we use stochasticity to model distributions path space. The resulting neural SDE is not a improvement similar neural ODE, but standalone concept in its own right.
1.2 CONTRIBUTIONS
Our central contribution is the observation that the mathematical formulation of SDEs is directly comparable the machine learning formulation of GANs. Using this connection, we show how becomes straightforward to train neural SDEs as generative time series models. Arbitrary drift and diffusions are admissible, and in the infinite data limit any SDE may be learnt.
Next, we introduce a new way of sampling Brownian motion, adapted to the query patterns typical to SDE solvers. The scheme produces exact samples using O(1) memory and average-case ((1) time. In particular, it can reconstruct its past trajectory, which is necessary for the use of adjoint equations. The scheme operates by combining splittable Secudo-Random Number Generators (PRNGs), a binary tree of dependent intervals, and Least Recently Used (LRU) cache of recent queries.
Finally, we demonstrate that the theoretical construction of adjoint SDEs (which may be used to backpropagate through SDE) may be simplified by using the pathwise formulation of rough path theory. In particular this avoids the previous theoretical complexity of two-sided filtrations.
To facilitate the use of these techniques, we have implemented them part of an open-source Py PyTorch-coppttbble general-purpose SDE library, [redacted]. This may be found at https:/ g thub. com/ [redacted].
Initial
Hidden state
Output
Noise
N~N((,
W. Brownian motion
Generator
Xo = Co(V)
dX, = polt, X,) dt + oolt, X,) dw,,
Yi = (X())
Discriminator
Hl = (Yoo))
dHi, = fo H1) dt go H4) oY,
= ms (HT)
Figure 1: Summary of equations.
METHOD
2.1
SDES AS GANS
Consider some "noise" distribution μ on a space X' and a target probability distribution y on a space Y. A generative model for is a learnt function Gø: X' - y trained that the (pushforward) distribution Go(H) approximates v. For our purposes, a Generative Adversarial Network (GAN) may then be characterised then evaluating (w),, but for which the probability density of (4) is not computable (due to the complicated structure of Go).
choice of Go which may be sampled from (by sampling μ and
2

---

Under review as a conference paper at ICLR 2021
The name "adversarial" then arises from the fact that GAN is trained by examining the statistics of samples from Go(H), most typically a learnt scalar statistic, parameterised by a discriminator. (Although variations such MMD-GAN instead use fixed vector-valued statistics (Li et al., 2015).)
Now consider SDEs. Consider some (Stratonovich) integral equation the form
Xo ~ 14, dX, [(t, X,) + g(t,, X,) o dW,,
for initial probability distribution and (Lipschitz) functions f. g and Brownian motion W. The strong solution this SDE may be described as the (unique) map F such that F(u, W) almost surely (Rogers & Williams, 2000, Chapter V, Definition 10.9).
Intuitively, SDEs are maps from noise distribution (Wiener measure, the distribution of Brownian motion) to some solution distribution, which is some probability distribution on path space. We recommend any Karatzas & Shreve (1991), Rogers & Williams (2000), or Revuz & Yor (2013) as introduction to the theory.
SDEs can sampled from with relative ease: this what numerical SDE solver does. However, evaluating its probability density is not possible; in fact it is not even defined in the usual sense." This scenario- - no availablc/tractable densities, but sampling is available - is now the familiar setting of GAN.
Moreover, this essentially generalises the typical procedure by which a parameterised SDE is fit to data, which is usually done by matching certain statisties (such option prices).
2.2 GENERATOR
Let Z be a random variable on y-dimensional path space. Loosely speaking, this is the space of continuous functions f: [0, T] - R for some fixed time horizon T 0. For example, this may correspond to the (interpolated) evolution of stock prices over time. This is what we seek to model.
Let W: [0, + R" be w-dimensional Brownian motion, and V N (v, be drawn from a u-dimensional standard multivariate normal. The values w, are hyperparameters describing the size of the noise.
Let
Co: R + R",
(50 [0, T
x R2
op: [0, R - Rex
Lg: RR
R2
where Co. Ao and dg are (Lipschitz) neural networks, and Eo is linear. Collectively they parameterised by The dimension x a hyperparameter describing the size of the hidden state.
We seek to learn (Stratonovich) SDE of the form
Xo = -V),V,
dx, = (t, X,) di + at. (t. X,) o dW,
Y, tx(e),
(1)
for f E [0, with X: [0, _ the (strong) solution to the SDE, such that in some sense
& Z. That is to say, the model should have approximately the same distribution as the target Z (for some notion of approximate). The solution X is guaranteed to exist given mild conditions (such as Lipschitz μlg. σg).
Çe. (4g, and dg may be taken be any standard network architecture, such as simple feedforward network. (The choice does not affect the GAN construction.)
Network architecture
Hidden state The solution X represents hidden state, and the output of the model. If were the output, then future evolution would satisfy a Markov property, of being dependent on the past only through the present, which need not be true in general.
This is the reason for the additional to mapping to Y. Practically speaking, during an SDE solve, Y may be concatenated alongside X, and la concatenated with plg
'Technically speaking. probability density is the Radon Radon-Nikodym derivative the measure with respect to the Lebesgue measure. However, the Lebesgue measure only exists for finite dimensional spaces. In infinite dimensions, it is possible define densities with respect to for example Gaussian measures, but this is less obviously meaningful when used with maximum likelihood.
3

---

Under review as a conference paper at ICLR 2021
Initial condition It is important that there be additional source of noise for the initial condition, passed through a nonlinear (g. as Yo = Eg (V)) does not depend on the Brownian noise W.
Stratonovich versus Itō The choice of Stratonovich solutions over Itó solutions is not mandatory, but will turn out to little theoretically neater when we discuss the adjoint method in Section 3.1.
Sampling Given trained model, we sample from it by sampling some initial noise V and some Brownian motion W, and then solving equation (1) with numerical SDE solver. Any standard numerical SDE solver may be used. As we consider Stratonovich integrals, then we use the midpoint method (which converges to the Stratonovich solution), rather than the Euler- Muuler-Mauyamm method (which converges to the Itô solution).
Comparison to the Fokker-Planck equation The distribution of an SDE, as learnt by neural SDE, contains more information than the distribution obtained by solving a BekkerrPlancc equation. Planckk equation gives the (time evolution of the) probability density of
The solution to solution at fixed times. It does not encode information about the time evolution individual sample paths. This is exemplified by stationary processes, whose distribution does not change over time.
2.3
DISCRIMINATOR
Each sample from the generator is a path Y: [0, + R2; ; the discriminator must accept such paths as inputs. There is natural choice: parameterise the discriminator another neural SDE.
Let
Ro:R"
fo: [0. x R2 BA" Io: [0, R - RAxy, m R - +E E
where Eo. So and Is are (Lipschitz) neural networks, and my linear. Collectively they are parameterised by 6. The dimension h is hyperparameter describing the size of the hidden state.
Recalling that the generated sample, then the discriminator an SDE of the form
Ho (Yo),
dH, (t,, Ht) dt + (t H₂) o dY,,
D = m
(2)
for & E [o, T). with H: [0, T] + Rd the (strong) solution to this SDE, which exists given mild conditions (such as Lipschitz før 96). The value D e R, which is function of the terminal hidden state Hr, is the discriminator's ' score for real versus fake.
The discriminator follows the formulation of neural CDE (Kidger al., 2020b) with respect to the control Y. Neural CDEs are the continuous-time analogue to RNNs, just as neural ODEs the continuous-time what motivates equation (2) as a probably sensible choice of discriminator. Moreover, it means that the discriminator enjoys theoretical properties, such as universal approximation with respect to compact sets of paths.
Neural CDEs
analogue to residual networks (Chen al., 2018). This is
Training data Just described is how the discriminator is applied to the generator output. For the training data, the analogous thing is done, as follows.
Suppose for simplicity that we observe samples from Z an irregularly sampled but fully observed time series z = ((to, 2o),...,
...., .....,, (fn, 2,)), where without loss of generality to = 0 and t, = T.
Then we may (linearly) interpolate to produce E: [0, T) - I" such that {(4)) = z., and compute
Ho = E (3(()))
dH,- = ft(,, H.) dt + glll, Ha) o da,, D = m HT)
before.
Using an interpolation of the data represents an approximation to the underlying continuous- process from which the data was observed; Kidger al. (2020b). Additionally the interpolation need not be linear; all that is required is to produce the distribution on path space that is desired to be modelled.
the data is actually partially observed, has asynchronous sampling, or is of variable length, then the interpolation may still be performed in much the same way. See the examples of Kidger (2020).
4

---

Under review as a conference paper at ICLR 2021
Initial condition and hidden state As with the generator, it is important that there be learnt initial condition, and that the output be a function HI and not a univariate Hr itself. (See also Kidger et al. (2020b), who emphasise the need for learnt initial condition.)
Single SDE solve In practice, both generator and discriminator may be concatenated together into a single SDE solve. The state is the combined [X, Y, H.. the drift is the combined [uo, Eg o μig, S to [ea), and the diffusion is the combined (oa, &g dg, 96 Eg oa). Then Hr is extracted from the final hidden state, and me applied, to produce the discriminator's ' score for that sample.
Training loss The training losses used are the usual one for Wasserstein GANs (Goodfellow al., 2014: Arjovsky et al., 2017). Let Yo: (V. W) Y represent the overall action of the generator, and Dæ: + D the overall action the discriminator. Then the generator optimised with respect to
min [vy, (Vo((V W))l
and the discriminator is optimised with respect to
max [v, Do - Eg (3)
Training performed via stochastic gradient descent techniques as usual. Backpropagation may be performed either through the internal operations of the numerical SDE solver, or via the adjoint method for SDEs (Li et al., 2020). In the latter case, then the entire SDE is treated a single differentiable primitive within the computation graph.
Lipschitz regularisation Wasserstein GANs need a Lipschitz discriminator, for which a variety of methods have been proposed. We use gradient penalty (Gulrajani al., 2017), finding that neither weight clipping nor spectral normalisation worked (Arjovsky et al., 2017; Miyato et al., 2018).
We attribute this to the observation that neural SDEs (as with RNNs) have recurrent structure. If a single step has Lipschitz constant X, then the Lipschitz constant of the overall neural SDE will be o(x") in the time horizon T. Even small positive deviations from 1 produce large Lipschitz constants. Gradient penalty avoids this by regularising the Lipschitz constant of the overall network, by adding an additional regularisation term
E, ( 1
where = sampled from all convex combinations of the true and generated distributions.
The use of gradient penalty does require a double backward. This is a concern we shall return to in Section 5.
2.4 EXTENSIONS
Conditional GANs This approach may be extended conditional GANs; simply append the extra context to both generator and discriminator as in Mirza & Osindero (2014); Ren et al. (2016)
MMD-GANs Given a kernel on path space (not just on samples as in Briol et al. (2020)), for example the signature kernel (Király & Oberhauser, 2019; Toth & Oberhauser, 2020), then the neural SDE may alternatively be trained as an MMD-GAN as well (Li et al., 2015).
Jumps is straightforward include jump terms (Jia Benson, 2019) in the generator, The formulation of the discriminator is unchanged.
We consider conditional GANs in our experiments, and leave MMDs and jumps for future work.
EFFICIENT COMPUTATION
The SDE-as-GAN formulation is expected to be the primary interest for machine learning applications.
5

---

Under review as a conference paper at ICLR 2021
We now provide two further technical contributions. First, we demonstrate how the construction of the adjoint equations may be performed straightforwardly via rough path theory. Second, we construct new scheme for simulating Brownian motion, which dub the Brownian Interval.
As practical matter, these may be handled by SDE libraries, without the end user having to worry about how they are performed. And indeed, make these available in the [redacted] library.
3.1 ROUGH ADJOINT EQUATION
Neural differential equations may be backpropagated through either by backpropagating through the internal operations of the solver, or by treating the entire neural differential equation as a differentiable primitive via the adjoint method (Pontryagin al., 1962; Chen al., 2018). The adjoint method backpropagates by solving another differential equation, the adjoint equation, backwards in time. See Kidger et al. (2020a) for clear exposition in the ODE setting.
However, it does not straightforwardly extend to the SDE setting, as the theory of SDEs relies on arguments that depend on the arrow of time, such as filtrations. Li al. (2020) manage handle this by using nonstandard setting involving subtle arguments with two-sided filtrations.
Here we show that the issue may instead be straightforwardly resolved by the adoption rough path theory. Hodgkinson et al. (2020) give very readable introduction, and Friz & Hairer (2014);
Friz & Victoir (2010) standard textbooks. In this formulation: (a) the integrals W., - (W, W.) dW, are defined probabilistically as typically Stratonovich integrals; (b) the joint probability distribution (W, W) is sampled; (c) the solution to the SDE defined pathwise with respect to the sample. 2 The solution is the same as the strong solution given by usual SDE theory, but now the probability contained just within (W, W), and complications such as filtrations do not appear.
It is instructive to note that this is the same procedure as performed when using numerical SDE solvers. Samples are drawn from (W, W), and then the SDE is solved pathwise with respect to those samples. The higher order terms W appear for example in Milstein's method (Kloeden & Platen, 1992), originally constructed for solving Itô SDEs.
What this then means is that it is straightforward to make sense of notions like running SDEs backwards in time: we sample from (W, W) as before, and then just traverse the sample backwards.
Theorem (Informal). Consider the Stratonovich SDE of equation (1), and let L be (scalar) loss on XT. Then the adjoint process a₂ dL(Xr)fax, is strong solution of the linear Stratonovich SDE
da, -(a V) pa Xt) di (a: V) aall, Xx) o dW,
for t E [0, In particular We is the same Brownian noise as used in the forward pass.
This is cheap to compute as it involves only vector-Jacobiat products. This is equivalent to the adjoint as given by Li et (2020): the difference through our use conventional Stratonovich integrals over their "backward Stratonovich integrals" and we argue that interpreting these as rough integrals is better still.
The proof may be found in Appendix A.
3.2 SIMULATING BROWNIAN MOTION
Numerically solving SDE requires sampling Brownian motion, conditional on its previous samples. Mathematically this straightforward. Let W., = We W, E R. Then for s < l u, Lévy's Brownian bridge (Revuz& & Yor. 2013) states that
W Wo,|W = N t- [t-sw- W
(u
t) 8) 1w0
For almost all samples. In fact the excluded null set is the same for all SDEs. Note the use of Stratonovich integrals- this is the reason for previous preference for them over Itô integrals.

---

Under review as a conference paper at ICLR 2021
The difficulty here is computational. On Algorithm 1: Sampling the Brownian Interval
the adjoint pass, the same Browmian sample must be reconstructed, potentially at locations other than were used on the forward pass. A memory intensive approach is to store every sample, and apply equation (3) when appropriate. Gaines & Lyons (1997); Li et al. (2020) instead approach this via the "Brownian Tree". However this produces only approximations, as the real line must be discretised to some tolerance at which the tree is terminated. Practically speaking this is also slow, as small tolerances demand deep traversals of the tree; indeed Li et al. (2020) do not use it in their experiments for this reason.
Type: Let Node denote a 5-tuple consisting of an interval, seed, and three optional Nodes, corresponding to the parent node, and two child nodes, respectively. (Optional the root has no parent and leaves have no children.)
Input: Interval [s, t] C [0,
State: Binary tree with elements of type Node, with root 7 7= ([0, 7) 8, *, hatt, Tright). A Node J.
Result: Sample increment W..
# The returned 'nodes' is a list of Nodes whose # intervals partition [s, 4]. Practically speaking this will usually have only one or two elements. nodes = (sr
We introduce the "Brownian Interval". which improves upon this with exact samples and fast query times. Similar to the dyadic tree of points used in the Brownian Tree, we now instead we have a binary tree of intervals. Each parent interval is the disjoint union of its child intervals. What simulated is actually the increments W.,., not displacements W.. This is because (a) this is what is actually used an SDE solver, and (b) this is the appropriate interface when additionally desiring the higher order term Wa, Ist War o dW,, such as for Milstein's 's method.
def s sample(I Node):
if T is i then return N (0, T) sampled with seed S.
Let / = (1a, b). Lpazents Liett light) Let .parent = (lap, Up), 8p: Apps lips lyp)
W parent = sampplaee eepple([gg ((pareeat)
if I, is Ipp then
Wieft -bridge (ap. bp, a, W Wprrent: s) return Wparent Weet
else
return bridge(ap, bp, b, Wparents s)
sample LRUCache(sampLee
The tree starts as a stump consisting of just the global interval [o, New leaf nodes are created as queries over intervals are made. For example, making a first query at [s, S [o, 7] (an operation that returns Ws,)) produces the binary tree shown Figure 2a; making a subsequent query at [u, v) with u < s < v < t produces Figure 2b. Using splittable PRNG (Salmon et al., 2011), each child node also has a random seed deterministically produced from the seed of its parent.
J x-
nodes
return Σ E sample(T)
0, T
[
T]
[0. s)
[s, T
[0,s]
88. T
[0, ml]
[u, 8] [8,
[1.T]
[s. 8]
[t.
T
[s, v]
[b,t]
(a)
(b)
The tree thus completely encodes the conditional statistics of Brownian motion: Ws.,, We,, are completely specified by 8, L, u. W Wa. equation (3), and the random seed associated with [s, u].
Figure 2: Binary tree of intervals.
Computing W., in this way requires Wa,s which is not itself stored: in principle it is instead calculated by recursing up the tree. This would be very slow (recursing to the root on every query), except that an LRU cache additionally applied to the computed increments Wat--
Queries are exact because the tree aligns with the query points. Queries are fast because of the LRU cache; in SDE solvers, subsequent queries are likely to be close to (and thus conditional on) previous queries. The average-case (modal) time complexity is thus (1). Even in the event of cache misses all the way up the tree, the worst-case time complexity will only be (log(1/s)) in the average step size s of the SDE solver. The (GPU) memory cost is essentially the size of the LRU cache, which is constant. There is the small additional cost of storing the tree structure itself, but this is held in CPU memory, which is for practical purposes essentially infinite.
7

---

Under review as a conference paper at ICLR 2021
See Algorithm 1, where br bridge denotes equation (3), and traverse traverses the binary tree to find a list of nodes that are interest, and is defined explicitly in Appendix B. Also see Appendix B for various extensions and technical considerations need to ensure this algorithm works.
EXPERIMENTS
4.1 DATASETS
Stocks We consider a dataset consisting of Alphabet/Google stock prices for 2018-2019, obtained from LOBSTER (Haase, 2013). The data consists of limit orders, in particular ask and bid prices. On average there are 605 054 values per day. Many of these do not actually change the price (specifically the midpoint or spread), so we downsample to 40000 observations per day, specifically over the trading period 9.30am-4pm. This is then sliced into windows of length approximately one minute. We model the bivariate path consisting the midpoint and the log-spread over this time interval.
Weights Next, we consider another problem that is classically produced via (stochastic) differential equations: the weight updates when training neural network via stochastic gradient descent. We train small convolutional network on MNIST (LeCun al., 2010) for 100 epochs, and record its weights every epoch. Repeated over 10 models, this produces dataset of univariate time series; each time series corresponding to particular scalar weight.
Beijing Air Quality We consider a dataset of the air quality in Beijing, from the UCI repository (Zhang et al., 2017; Dua & Graff, 2017). Each sample a 6-dimensional time series of the SO2, NO2, CO, Og, PM2.5 and PM10 concentrations, as they change the course of a day. We train this problem as a conditional GAN, by conditioning on the one-hot encoded label for which of 14 different locations the data was measured at.
4.2 MODELS
We compare against the Latent ODE model of Rubanova et al. (2019) and the continuous time flow process (CTFP) of Deng et al. (2020). We used the full version of CTFPs, including latent variables.
These were selected for being competing neural differential equation models, which additionally represent differing extremes of neural SDEs. The Latent ODE model samples its noise as an initial condition, and is thereafter pure-drift model. Meanwhile, continuous time flow processes sweep normalising flow over a Brownian noise, and thus represent pure-diffusion model. Neural SDEs, however, combine both drift and diffusion terms.
Between them these models cover a variety of training regimes. Latent ODEs are trained as variational autoencoders; CTFPs are trained normalising flows; neural SDEs trained GANs. To our knowledge neural SDEs are the first model in their class, namely continuous-time GANs.
4.3 RESULTS
We study three test metries: classification, prediction, and MMD. each case every model is run three times and mean and standard deviation of the test metrics are reported. See Appendix C for details of hyperparameters, learning rates, optimisers and so on.
Classification is given by training model to distinguish real from fake data. We use neural CDE (Kidger et al., 2020b) for the classifier. Larger losses, meaning inability to classify, indicate better performance of the generative model.
Table 1: Classification loss. (Bold indicates best performance.)
Neural SDE
CTFP
Latent ODE
Stocks
0.357 0.045
0.165 + 0.087
0.000239 +: 0,000086
Weights
0.507 + 0.019
0.676 + 0.014
0.0112 + 0.0025
Beijing Air Quality
0.589 0.051
0.764 = 0.064
0.392 + 0.011
8

---

Under review as a conference paper at ICLR 2021
Prediction is train synthetic, test real (TSTR) metric (Hyland al., 2017). We train sequence-tor -o-eue------eeecee model to predict the latter part time series given the first part. We use a neural CDE/ODE an encoder/decoder pair. Smaller losses, meaning ability to predict, are better.
Table 2: Prediction loss. (Bold indicates best performance.)
Neural SDE
CTFP
Latent ODE
Stocks
0.144 + 0.0446
0.725 + 0.233
46.2 + 12.3
Weights
0.00843 + 0.00759
0.0808 + 0.0514
0.127 + 0.152
Beijing Air Quality
0.395 = 0.056
0.810 = 0.083
0.456 + 0.095
Maximum mean discrepancy is a distance between probability distributions with respect to a kernel or feature map. We use the depth-5 signature transform as the feature map (Király Oberhauser, 2019; Toth & Oberhauser, 2020). Smaller values, meaning closer distributions, are better.
Table 3: MMD loss. (Bold indicates best performance.)
Neural SDE
CTFP
Latent ODE
Stocks
1.92 0.09
2.70 + 0.47
60.4 35.8
Weights
5.28 = 1.27
12.0 + 0.5
23.2 = 11.8
Beijing Air Quality
0.000160 = 0.000029 0.00198 = 0.00001 0.000242 + 0.000002
Neural SDEs produce substantially better results with respect to both the predictive (forecasting) and MMD metrics. On the stocks data, they additionally perform substantially better the classification metric - stocks being a regime in which SDE models have classically been applied. We see that both neural SDEs and CTFPs consistently outperform Latent ODEs, which we attribute the nature of these datasets: on these problems we expect to see some random fluctuations, and the underlying dynamics are not those of a pure drift.
4.4 ORNSTEIN-UHLEENBER PROCESS
We also studied an example for which the underlying distribution is known, and is given by a time- dependent Ornstein- Uhlenbeck process. See Appendix C.5.
CONSIDERATIONS
Stochastic weight averaging We found that using stochastic weight averaging (Izmailov et al., 2018) was particularly helpful for improving performance, as it averages out the oscillatory training behaviour for the min-max objective used in GAN training.
Final tanh nonlinearity Using a final tanh nonlinearity (on both drift and diffusion, for both generator and discriminator) constraints the rate of change of hidden state (Kidger et al., 2020b), which we found helped training.
Lipschitz regularisation We found that training using the adjoint equations was difficult due to the use of gradient penalty: this involves double backward, and thus double adjoint, which for reasonable step sizes found produced inaccurate enough gradients to prevent models from training. As such our experiments do not use adjoints, instead backpropagating through the solver. This is issue that hope may be resolved in future work.
6 CONCLUSION
We have shown that SDEs and GANs follow similar formalisms. Using this connection, we train neural SDEs continuous time, infinite dimensional, time series GANs. Next, we introduce new way of sampling and reconstructing Brownian motion that is both fast and memory efficient. Finally, show that the adjoint equations may straightforwardly be developed through rough path theory.

---

Under review as a conference paper at ICLR 2021
REFERENCES
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein Generative Adversarial Networks. volume 70 of Proceedings of Machine Learning Research, pp. 214-223, International Convention Centre, Sydney, Australia, 2017. PMLR.
S. Bai, J. Kolter, and V. Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems 32, pp. 690-701. Curran Associates, Inc., 2019.
F-X. Briol, A. Barp, A. Duncan, and M. Girolami. Statistical Inference for Generative Models with Maximum Mean Discrepancy. arXiv:906.059444 2020.
R. T. Q. Chen. torchdiffeq, 2018. htttps:/github.coo cow. httpe/////cccccccc////cccccccc ffeq.
R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural Ordinary Differential
Equations. Advances Neural Information Processing Systems 31, pp. 6571-6583. Curran Associates, Inc., 2018.
K. Claessen and M. Palka. Splittable pseudorandom number generators using cryptographic hashing. ACM SIGPLAN Notices, 48:47-58, 2013.
M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
A. Davie. KMT theory applied approximations of SDE. Stochastic Analysis and Applications 2014, pp. 185-201. Springer, 2014.
R. Deng, B. Chang, M. A. Brubaker, G. Mori, and A. Lehrmann. Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows. arXiv:2002. 10516, 2020.
A. Dickinson. Optimal Approximation of the Second Iterated Integral of Brownian Motion. Stochastic Analysis and Applications, 25(5):1109-111 2007.
D. Dua and C. Graff. UCI Machine Learning Repository, 2017. URL uci edu/ml.
ics.
G. Flint and T. Lyons. Pathwise approximation of SDEs by coupling piecewise abelian rough paths. ar5iv:1505.01298, 2015.
J. Foster, H. Oberhauser, and T. Lyons. An optimal polynomial approximation of Brownian motion. SIAM Journal on Numerical Analysis, (393))1333-1227) 2020.
P. Friz and M. Hairer. A Course Rough Paths. Springer International Publishing, 2014.
K. Friz and N. B. Victoir. Multidimensional stochastic processes as rough paths: theory and applications. Cambridge University Press, 2010.
J. Gaines and T. Lyons. Random Generation of Stochastic Area Integrals. SIAM Journal on Applied Mathematics, 54(4):1132-11446 1994.
J. Gaines and T. Lyons. Variable step size control in the numerical solution of stochastic differential equations. SIAM Journal on Applied Mathematics, 57(5):1455-1444 1997.
P. Gierjatowicz, M. Sabate-Vidales, D. Šižka, L. Szpruch, and Ž. Zurič. Robust Pricing and Hedging via Neural SDEs. arXiv:2007.04154, 2020.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. Advances in Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014.
W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2021
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kemel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2013.
A. Griewank. Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optimization Methods and Software, 1(1):35-54, 1992.
I. Guirajani, Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems 30, pp. 5767-5777. Curran Associates, Inc., 2017.
Haase. Limit order book system the efficient reconstructor, 2013. lobsterdata.com/
URL https://
L. Hodgkinson, C. der Heide, Roosta, and M. Mahoney. Stochastic Normalizing Flows. arXiv:2002.09547, 2020.
S. L. Hyland, C. Esteban, and G. Rätsch. Real- Valued (Medical) Time Series Generation with Recurrent Conditional GANs. arXiv:1706.0266 2017.
M. Innes, A. Edelman, K. Fischer, C. Rackauckas, E. Saba, B. Shah, and W. Tebbutt. A Differentiable Programming System to Bridge Machine Learning and Scientific Computing. 1rXiv:190..07587, 2019.
Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. Gordon Wilson. Averaging Weights Leads Wider Optima and Better Generalization. UAI, 2018.
Jia and A. Benson. Neural Jump Stochastic Differential Equations. In Advances in Neural Information Processing Systems 32, pp. 9847-9858. Curran Associates, Inc., 2019.
I. Karatzas and S. Shreve. Brownian Motion and Stochastic Calculus. Graduate Texts in Mathematics. Springer New York, 1991.
Kidger. torchede, 2020. https: htt.t//..../aaatrcceaaaee/e//eee
Kidger and T. Lyons. Signatory: differentiable computations of the signature and logsignature transforms, both CPU and GPU, arXiv:2001.00706, 2020. URL https: com/ patrick- - kidger///// wedddrrriiiit signatory.
P. Kidger, R. T. Q. Chen, and T. Lyons. "Hey, that's not an ODE": Faster ODE Adjoints with 12 Lines of Code. arXiv:2009.0944 099577, 2020a.
Kidger, J. Morrill, J. Foster, and T. Lyons. Neural Controlled Differential Equations for Irregular Time Series. wwXiv:2005.08926, 2020b.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Kirăly and H. Oberhauser. Kernels for sequentially ordered data. Journal of Machine Learning Research, 2019.
P. Kloeden and E. Platen. Numerical Solution of Stochastic Differential Equations. Springer, 1992.
Y. LeCun, C. Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yammllecu.ccom
com//exbbmnnitt http://ywam.eee....oeeeebbbbbaaa
2010.
X. Li, T.-K. L. Wong, R. T. Q. Chen, and D. Duvenaud. Scalable Gradients and Variational Inference for Stochastic Differential Equations. AISTATS, 2020.
Y. Li, K. Swersky, and Zemel. Generative Moment Matching Networks. In Proceedings of the 32nd International Conference on Machine Learning. 2015.
X. Liu, T. Xiao, S. Si. Q. Cao, S. Kumar, and C.-J. Hsich. Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise. (XX1:19906.02355.1 2019.
Lyons. Differential equations driven by rough signals. Revista Matemática Iberoamericana, 14, 1998.
11

---

Under review as a conference paper at ICLR 2021
T. Lyons, Caruana M., and Lévy T. Differential equations driven by rough paths. In École d' d'été de probabilités de Saint-Flour XXXIV-2004, edited by Picard in Volume 1908 of Lecture Notes in Mathematics, Berlin, Springer, 2007.
S. Massaroli, M. Poli, J. Park, A. Yamashita, and H. Asama. arXiv:2002.08071, 2020.
Dissecting Neural ODEs.
M. Mirza and S. Osindero. Conditional Generative Adversarial Nets. arXiv:1411.1784 2014.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations, 2018.
J. Morrill, P. Kidger, C. Salvi, J. Foster, and T. Lyons. Neural CDEs for Long Time-Series via the Log-ODE Method. arXiv:2009.08295 2020.
A. Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Lió. On Second Order Behaviourin Augmented Neural ODEs. axXi:20006.07220,000 2020.
V. Oganesyan, A. Volokhova, and D. Vetrov. Stochasticity in Neural ODEs: An Empirical Study. arXiv:2002.09779, 2020.
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, pp. 8024-
8035. Curran Associates, Inc., 2019.
L. S. Pontryagin, E. F. Mishchenko, V. G. Boltyanskii, and R. V. Gamkrelidze. The mathematical theory of optimal processes. 1962.
C. Rackauckas, Y. Ma. J. Martensen, C. Warner, K. Zubov, R. Supekar, D. Skinner, and A. Ramadhan. Universal Differential Equations for Scientific Machine Learning. arXiv:2001.0433 2020.
Y. Ren, J. Zhu, J. Li, and Y. Luo. Conditional Generative Moment-Matching Networks, In Advances in Neural Information Processing Systems 29, pp.
Curran Associates, Inc., 2016.
D. Revuz and M. Yor. Continuous martingales and Brownian motion, volume 293. Springer Science & Business Media, 2013.
L.C.G. Rogers and D. Williams. Diffusions, Markov Processes and Martingales: Volume 2, Itô Calculus. Cambridge Mathematical Library. Cambridge University Press, 2000.
A. Rößler. Runge-Kutta Methods for the Strong Approximation of Solutions of Stochastic Differential Equations. SIAM Journal on Numerical Analysis, 48(3):922-952, 2010.
Y. Rubanova, T. Q. Chen, and D. Duvenaud. Latent Ordinary Differential Equations for Time Series. In Advances in Neural Information Processing Systems 32, pp. 5320-5330. Curran Associates, Inc., 2019.
K. Salmon, M. A. Moraes, R. O. Dror, and D. E. Shaw. Parallel random numbers: as easy as 1, 2, 3. Proc. High Performance Computing, Networking, Storage and Analysis, pp. 1-12, 2011.
C. Toth and H. Oberhauser. Variational Gaussian Processes with Signature Covariances. ICML 2020, 2020.
B. Tzen and M. Raginsky. Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit. arXiv: (00.://99000083,,, 2019a.
B. Tzen and M. Raginsky. Theoretical guarantces for sampling and inference in generative models with latent diffusions. COLT, 2019b.
S. Zhang, B. Guo, A. Dong, J. He, 7. Xu, and S. X. Chen. Cautionary Tales Air-Quality Improvement in Beijing. Proceedings of the Royal Society A, 473(2205), 2017.
12

---

Under review as a conference paper at ICLR 2021
A DERIVATION OF THE ROUGH ADJOINT EQUATION
In this section, we will present a "rough path" derivation of the adjoint equation for Neural SDEs. Since rough path theory is well developed field, much of our analysis involves quoting key results. To begin, we recall the informal statement of the theorem that we wish to prove:
Theorem (Informal). Consider the Stratonovich SDE
dx, poll, Xx) dl + atl X1) dWW,
(4)
where pe and de haghsigd are sufficiently regular vector fields. Let L be a scalar loss on Xr. Then the adjoint process 0t = dI(Xx))/xx is strong solution of the linear Stratonovich SDE
da = - (at V) pe(1, x2) dt - (a: V) agl,, Xx) o dW,
(5)
for E [0, T. In particular W, is the same Brownian noise as used in the forward pass.
Whilst the above theorem looks simple enough, it provides us with three main challenges to address:
The first challenge in proving this theorem is that Brownian sample paths are not differentiable and thus the adjoint process will not be differentiable. In particular, we cannot use the proof given by Chen et al. (2018) where the derivative of the adjoint process is approximated using Taylor series.
The second challenge is more subtle and relates fact that Brownian sample paths do not have bounded variation. In particular, this means that we cannot define integrals with respect to Brownian sample paths in the Riemann-Stielljes sense (this is discussed Section 1.5 of Lyons et al. (2007)).
The third challenge is purely technical in that the vector fields of the adjoint equation (5) do not satisfy certain technical conditions. Typical assumptions rough path theory are that the vector fields are either bounded (and with some smoothness) linear. However the adjoint vector fields are linear a but nonlinear in X; overall they are unbounded and nonlinear. Therefore our analysis will involve separating the linear part of the adjoint equation from the bounded nonlinear part.
The outline of this section as follows. In subsection A.1, we will derive the adjoint equation for systems where the "driving path" has bounded variation but can be non-differentiable (Challenge 1). In subsection A.2, we will discuss some aspects of rough path theory - which provides a "pathwise" integration theory for SDEs (Challenge 2). Finally, in subsection A.3, shall put the various pieces together and derive the rough adjoint equation for Stratonovich SDEs (Challenge 3).
A.1 THE ADJOINT FOR CONTROLLED DIFFERENTIAL EQUATIONS
Before we consider SDEs and Brownian motion, we first derive the adjoint equation for a slightly more manageable class of differential equation namely the controlled differential equation." A CDE takes similar form to an SDE, except the system is "controlled" by a continuous path X instead of Brownian motion with time (that is, we write d.X, instead of dt or dW,). By assuming that has bounded variation, we can use Shemann-Stilttiee integration to define well-posed CDEs (existence and uniqueness results for CDE solutions are given in Chapter of Friz & Victoir (2010)).
Theorem A.1 (Adjoint equation for CDEs that are driven by bounded variation paths). Consider the controlled differential equation,
d
dyt Σ fo(
(6)
yo = E
(7)
where X [0,T]] Rd is continuous bounded variation path and each f 12" R" is bounded, differentiable and with bounded first derivatives. Let L Rd - R be differentiable loss function. Then the adjoint process
dL(yr)
(8)
dytt
Asso referred to in the literature rough differential equation.
13

---

Under review as a conference paper at ICLR 2021
satisfies the following linear CDE
da: at (o(o))))
Proof. For S f, let V.. R R be the "time-reversed" flow map for the CDE (7) on [s, t]. So for y E IR", (,) is the solution the CDE (7) at time so that its future value at time is y. Since X has bounded variation, a.,t is well-defined (via Siemann-Stieltjes integration) and satisfies
f. fo ( (,(),,(())) dx:
U. (2)
Σ 1=1
(10)
It was shown by Theorem 4.4 in Friz & Victoir (2010) that CDE flows have directional derivatives. As result of this theorem, taking the gradient of (10) is possible and rearranging gives
J.
(V,,(()) = Id Σ
V
(.,(())
(v)
(11)
Applying the chain rule to the adjoint process a; dL(yr) dyt
gives
dl(yr)
(LT) dy=
04
(12)
dy
dy dyt
where du. is the Jacobian matrix given by V a, (1)). and so dy:
a a, V V. (3:).
(13)
Thus, substituting (1I) into the above yields
d
a; = ag - as Σ
V V(lll)))
So by the above equation along with the triangle inequality, we have
Σ l an (olVs(
(14)
044 Σ a. v.V.)o.)xx.)
/ [ (an - -..)VV((o..d
Σ
(V...) ld dx: /|
d Σ
f (an - a.) a.)
=
as
In order to estimate these terms, we consider the matrix-valle valued path M [s + R given by
M. Σ / Vf((V..()))
so that equation (14) becomes
a, 0g Σ If au (.VVio())x )|
(15)
14) + [. (an ag) d.M.
dM..v (VV.V(VV,,,
0x
14

---

Under review as a conference paper at ICLR 2021
We use the notation
|| [-var[s.t] to denote the total variation (or 1-variation) of path y [s, IR,
1/rl sup Σ Ih - -Y,, ll
where ||-|| is a norm on &R* (we use ks = d, n²)). The supremum is taken over all partitions D of [s, t].
It is worth noting that since + V (yaa is continuous, it is bounded for E [s, I]. As result, M has bounded variation on [8, ui and there exists a constant C (depending only on t) such that
| 1I var[[ss] S |1 varr)s)))
(16)
for u E [s, with s and sufficiently close together.
We can rewrite (11) as the following linear CDE:
dz = (dM
20 = Id,
where 2u == VVn.. for 5 < u 2 < t. Since the path M has bounded variation, by
Davie's lemma for linear CDEs (Lemma 10.56 in Friz & Victoir (2010)), there exists constant C2 such that
20|l v C2 [xar(()))
for S S 0 < whenever is sufficiently close to and we note that 2, 20 V (y)) - Id.
Hence by the total variation estimate (16), there exists constant C3 depending only on such that (v.)) Id s C3l|X[12-vartte//
(17)
for S u S S t whenever s is sufficiently close to t. Since is continuous, is bounded on [s. and it follows from (13) with the estimate (17) that
|1. (an, a.) dM
(Ile- - -all) || MM 1=var[(e.c]
< sup wE(e.0]
(1|as s. axll) ||astoo
|l (
sup
uE(a.]]
Via.lllVVv..uuuu Id | [warr(ad)
sup wE(a,t]
I Call | 1-var/ssf))
and
dM (VV,, - Id
(Iaalll | (w.)) - Id (I) || II I
Og
< sup we(x.t]
$ | i-vac...0)
where the constants C4 and Cs only depends on t (provided that e:= t - is sufficiently small). Therefore equation (15) for the adjoint process becomes
a 1u.-(a.-) Σ /.-
"aann (nVVlbl)) < Ca) i-var:/s.))-
2ml
In other words, for a fixed t, we have
Q = a2 - Σ E{ au (M) dx; (
f=1"
provided that is sufficiently close to t. Thus, letting + gives
da, a, (y..
required.
15

---

Under review as a conference paper at ICLR 2021
A.2 THE ROUGH PATH APPROACH TO STOCHASTIC DIFFERENTIAL EQUATIONS
In this subsection, we shall briefly outline the "pathwise solution" theory for SDEs that was made possible by the advent of rough path theory (originally proposed in Lyons (1998)). Whilst rough path theory extends beyond the SDE setting, this is not within the scope of this paper.
Let F,P ( be filtered probability space containing a d-dimensional Brownian motion. Since the Brownian motion W 12 x [0, oo) + R corresponds certain Gaussian measure path space, it must be discretised in order to be used in SDE simulation. Moreover, by constructing sequence of approximations converging to the Brownian path we can extend the adjoint equation from the bounded variation setting (see Theorem A.1) 1) to the SDE setting.
To begin, we give few key definitions (the signature, p-variation metric and geometric rough path).
Definition A.2. The (depth-2) signature continuous bounded variation path [0, + Rd is (x ( J where S2, (X) is a collection of increments and integrals given by
{L. (x - x:) dx
(x) = (1, x
(18)
isissa
where the above is defined using Riemann Riemaan-Stielljess integration.
Therefore (X) Ar + R 1 where DT {(s,t) E [0, :s < t is rescaled 2-simplex.
Definition A.3. For p E 3), the p-variation metric between functions Z 22 AT - RI+d+d
dz = max sup k= D
Σ LED
Th (Z)
(Z (21,,111))
TE
(19)
where Th denotes the projection map from RI+d+ onto Rdd (for = 1,2) and the above supremum is taken over all partitions D of [0, T] and the norms || must satisfy (up constant)
||a @ bll S ||a|||
for a, b E Rd. For example, we could use the standard L (operator) norms for vectors and matrices.
Definition A.4. For E (2, we say that a sequence of continuous bounded variation paths XN, [0, T + Rd converges in the p-variation sense to continuous map X AT -> R 1+ if
d (s (x + 0,
(20)
as N + co. When such a sequence exists, we can refer to the limit X as geometric p-rough path.
We now state the following result from rough path theory (Corollary 13.22 in Friz & Victoir (2010)).
Theorem A.5 (Brownian motion as a geometric rough path). Let W be standard d-dimensional Brownian motion and W be the piecewise linear path with N pieces that coincides with W on the uniform partition DN == {0- = to < 11- <... < IN T} with tq := kh and mesh size h := Then there exists random geometric p-rough path W (p E (2,3)) such that for almost all Ω
( (w)) - 00
(21)
as N + 0o for any p E (2,3) and
W(w) = (1. S (W.
W.) s W.) ood..) (w)) (
(22)
O5
Hence the geometric rough path W is often referred to as Stratonovich enhanced Brownian motion.
To put simply, this theorem states that Brownian motion can be approximated (in rough path sense) by a sequence of bounded variation paths. This is particularly helpful within stochastic analysis as it allows one to construct pathwise solutions for SDEs governed by sufficiently regular vector fields. The central result within rough path theory that makes this possible is the Universal Limit Theorem. To counter the roughness of Brownian motion, this requires vector fields to have Lip(7() regularity.
16

---

Under review as a conference paper at ICLR 2021
Definition A.6 (Lip(7) functions). A function f: R" + R" is said to be Lip(7) with y 1 if it is bounded with [?] bounded derivatives, the last being Hölder continuous with exponent (7 Ln.)) Equivalently, f is Lip(y) if the following norm is finite:
If [tapt)) = max
||D* v ||Dl
fll (1
(23)
o<ksL>]
where D& is the k-th (Fréchet) derivative of f and ||- Ila- is the standard c-Hölder norm for a E (0, 1). We say that f is Lip(1) 1) it is bounded and Lipschitz continuous. That is, if the norm
[1(x) - s(5i]
|l = =|15|l
sup a.EER" 2øy
|| 3/
(24)
is finite.
Theorem A.7 (Universal Limit Theorem for RDEs (Theorem 5.3 in Lyons et al. (2007))). Let p (2 and It : [0, + Rd be sequence of continuous bounded variation paths which converge in p-variation to geometric p-rough path æ. Let {fihgisa denote a collection of Lip(n)
functions on R" with y > p and consider the controlled differential equation (CDE)
d Σ fl(x.) )(x*)
N dy:
N E.
where & E IR" and the above differential equation is defined using Riemann-Stielijes integration. Then there exists unique geometric p-rough path = (x. y) Dr + + such that N converges to y in p-variation. Moreover, the "universal limit" y depends only on æ, {fe] and &. Definition A.8. We shall refer to y as the solution of the rough differential equation (RDE).
dy: Σ dxy
(25)
Remark A.9. Theorem A.7 and Definition A.8 also apply when the vector fields (ft} are linear (Theorem 10.57 in Friz & Victoir (2010)).
Importantly for us, the above theory applies directly (Stratonovich) SDEs as Brownian motion can viewed as a geometric p-rough path with E (2,3) by Theorem A.5. We refer the reader to Section 17.2 of Friz & Victoir (2010) for detailed account of the "rough path approach" to Stratonovich theory. For our purposes, we state a Universal Limit Theorem for Stratonovich SDEs
Theorem A.10 (Remark 17.5 in Friz& & Victoir (2010)). Suppose that μ is a Lip(1) fanction on Ra+1 and Jisked are Lip(2) functions on Ba+1, Let (WN) JN21 be a sequence of piecewise linear paths converging to the Stratonovich enhanced Brownian motion W given by Theorem A.5. Let {y" JN21 be the sequence of solutions the following controlled differential equations (CDEs),
d
dyj = p(1,111 di Σ o' (1,vE")
d a(WN);
V/ E R.
Then y converges in p-variation to a geometric p-rough path y with E (2,3) and the process E + ग. (3,,) coincides with the strong solution of the Stratonovich
y: [0, 1] + Rn given by V1
SDE
dy: = p(l, vc) +
o' (1, ve) c dWW
(26)
yo = E.
almost surely.
17

---

Under review as a conference paper at ICLR 2021
A.3 THE ADIOINT FOR STOCHASTIC DIFFERENTIAL EQUATIONS
Ideally, we would like to just replace the path X in Theorem A. with a Brownian motion (coupled with time, that is say (L, W.).. However, that result required that X have bounded variation, whilst sample paths of Brownian motion have infinite total variation. Resolving this difficulty is one of the essential reasons that rough path theory exists (see also Section 1.5 of Lyons et al. (2007)).
As mentioned previously, the main challenge in applying rough path theory here is that the adjoint equation (29) has nonlinear unbounded vector fields, which are not Lip(7) functions in (a, y). Our trick is to first derive an adjoint equation for the stochastic process corresponding to the Jacobian (which satisfies assumptions boundedness), and then to drive the adjoint equation by this Jacobian-valued stochastic process (which satisfies assumptions of linearity).
Theorem A.11 (Adjoint equation for Stratonovich SDEs). Suppose that ps and (oti are bounded functions R=+1 such that
The drift vector field pa is continuously differentiable with bounded first derivative.
Each noise vector field of is a Lip(r)) function with > 2.
Consider the (Stratonovich) stochastic differential equation,
dy: po V1 dt yo =
oblt, odW
(27)
and let L: R + I denote differentiable loss function. Then the adjoint process
dL(yT)
(28)
dgtt
coincides with the strong solution of the linear Stratonovich SDE
da, = - ae V uo (t, dt
a Woll
(29)
almost surely.
Proof. Let IN2I be the sequence of solutions to the following controlled differential equations,
dy, = =o v. ) dt +
(t )
(30)
yo =.
where (WN) N2 are the piecewise linear paths converging W in p-variation by Theorem A.S. Hence by Theorem A.10, we have that the corresponding sequence CDE solutions {y J converges almost surely in p-variation to the solution of the Stratonovich SDE (27).
Let L be a differentiable loss function that by Theorem A.1, each adjoint process
dl(x)) dyf
af
(31)
satisfies the linear CDE
da Vuo )
Just in the proof of Theorem A.I, we can rewrite the adjoint equation for V
daç" -a
18

---

Under review as a conference paper at ICLR 2021
where the matrix-valued path M =0, -> R is given
M [." Veolss.3 ds Σ Vog(s, d(wa))
(32)
Since the vector fields Vpo and (Voghsiss are bounded, we see that M has bounded variation.
Recall from the universal limit theorem (Theorem A.7) that (æ, y) was geometric p-rough path. This carries over to our setting and thus we define the (random) gcometric p-rough path x (W, y) Then by Proposition 17.1 in Friz & Victoir (2010), have that the following rough integral exists
Lo (W. v) od(, TI (L. (0()
for all E [0, 7] with probability one, provided that yp (4²) a collection of Lip(7-1) functions with y > p. Since each vector field og is Lip(7). it follows that each gradient Vog is Lip(1 1). Therefore we can apply Proposition 17.1 in Friz & Victoir (2010) to the dW integrals in (32) and, since Vug is continuous and bounded, is clear that the d' integral in equation (32) also converges.
Thus, due to the regularity of pso and dp, we see that the sequence (M*) converges in p-variation to geometric p-rough path M and the limiting imatrix-valucee process M, == Ti (M,r) satisfies
Mr = - (" VHO F. Vo) ds Σ ('vo
d
J.
Voi v.) dw,
(33)
almost surely. We now have all the ingredients needed to construct the rough adjoint equation (29),
1. Each CDE (30) admits unique solution y and the resulting sequence (y converges to the solution y of the SDE (27) almost surely.
2. Each CDE (30) admits unique adjoint process V satisfying a linear CDE driven by M N.
3. The sequence {Mx} converges in p-variation to a geometric p-rough path (almost surely).
4. By Theorem 10.57 in Friz & Victoir (2010), we have that the Universal Limit Theorem (Theorem A.7) also holds for linear RDEs.
Therefore the sequence {a>) converges in p-variation to geometric p-rough path a almost surely and the process a := dL(ur] T (ap,)) coincides with the strong solution to the Stratonovich SDE dyo
da = - a, d.M
Voo dt (ol(t.u))
almost surely. The fact that a is the adjoint process follows from (31) and the continuity of VL. Remark A.12. The above argument can also extend to an RDE driven by a geometric p-rough path. In this case, the vector fields governing the differential equation would have to be Tip(?) with b >> p-
B SAMPLING BROWNIAN MOTION
B.1 ALGORITHM
We begin with providing the complete traversal and splitting algorithm needed to find or create all intervals in the Brownian Interval, in Section 3.2. We discuss its operation in the next section.
Here, List is an ordered data structure that needs to be appended to, and iterated over sequentially. For example linked list would suffice. We let split denote splittable PRNG in Salmon et al. 12011); Claessen & Palka (2013). We use * to denote an unfilled part the data structure, equivalent to None in Python null pointer in C/C++; in particular this is used as placeholder
19

---

Under review as a conference paper at ICLR 2021
for the (nonexistent) children of leaf nodes. We use denote the creation of new local variable, and < to denote in-place modification of a variable.
Algorithm 2: Definition of traverse
def bisect(I: Node, x R):
# Only called on leaf nodes Let I ((a, bl, Iparent, #,
81eft, gright split(s)
Left ( 8loft, J,
rrght = [ar b), sright. x,*)
([a, b], 8, .parent Leeee Lright)
def traverse(L: : Node, [c, d Interval, nodes: List[Node]): Let (Ia, 3, Iarennts liett fright)
# Outside our jurisdiction pass to our parent
i < a or then (raveese [c. dl, nodes) return
# It's that is sought. Add to the list and return.
if and d = then
apdes.append())
return
# Check if / a leaf or not.
I liss is then
# is a leaf
if
then
# If the start points align then create children and add on the left child.
# (Which is created in bi sect.)
b seseet(T, d)
nodes.append((rec))
# nodes is passed by reference
return
# Otherwise create children and pass on to our right child.
# (Which is created bisect.)
b Disect(TTc)
t craverse([right [c, d], nodes)
return
else
# is not leaf.
Let Teett = [[,,,)], I, Tu, Iir)
if d < then
# Strictly our left child's problem. traverse(Tiott: [c, d), nodes)
return
c 2 m then
# Strictly our right child's problem.
craverse(/=sgh:: [c, d]. nodes)
return
A problem for both of our children.
traverse(lieta [c, ml) nodes) traverse([ [m, dl, nodes)
return
B.2 DISCUSSION
The function traverse is simply a depth- fertt--fisttie tree traversal for locating an interval within a binary tree. The search may split into multiple (potentially parallel) searches (on the last few lines) if the target interval crosses the intervals multiple existing leaf nodes. If its target is not found then additional nodes created if needed.
20

---

Left to its own devices, the forward pass will thus build up highly imbalanced binary tree. At any one time, the LRU cache will contain only nodes whose intervals are a subset of some contiguous subinterval [s, t] of the query space [0, 7". Letting be the number of queries on the forward pass, then this means that the backward pass will consume Onn²) time - each time the backward pass moves past then queries will miss the LRU cache, and full recomputation to the root will be triggered, costing (n). This will then hold only nodes whose intervals are subets of some Under review as a conference paper at ICLR 2021
Sections 3.2 and B.1 now between them define the algorithm in technical detail.
There are some further technical considerations worth mentioning. Recall that the context we are explicitly considering is when sampling Brownian motion to solve an SDE forwards in time, then the adjoint backwards in time, and then discarding the Brownian motion. This motivates several of the choices here.
Small intervals First, the access patterns of SDE solvers are quite specific. Queries will be over relatively small intervals: the step that the solver is making. This means that the list of nodes populated by traverse is typically small. In our experiments consisting of a single element; occasionally two. In contrast if the Brownian Interval has built up reasonable tree of previous queries, and was then queried over [0, for s > 0, then long (inefficient) list would be returned. It is the fact that SDE solvers do not make such queries that means this is acceptable.
observed it usually only
Searching from J Moreover, the queries are either just ahead (fixed-step solvers; accepted steps of adaptive-step solvers) or just before (rejected steps of adaptive-step solvers) previous queries. Thus in Algorithm we keep track of the most recent node J, so that we begin traverse near to the correct location.
LRU cache The fact that queries are often close to one another is also what makes the strategy of using an LRU (least recently used) cache work. Most queries will correspond to a node that have a recently-computed parent in the cache.
Backward pass The queries are broadly made left-to-right (on the forward pass), and then right- to-left (on the backward pass). (Other than the occasional rejected adaptive step.)
contiguous subinterval [u, s): once move past u then this O(n) procedure is repeated, O(n) times. This is clearly undesirable.
This is precisely analogous the classical problem of optimal recomputation for performing backpropagation, whereby a dependency graph is constructed, certain values are checkpointed, and a minimal amount of recomputation desired; see Griewank (1992).
In principle the same solution may be applied: apply a snapshotting procedure in which specific extra nodes held in the cache. This a perfectly acceptable solution, but implementing requires some additional engineering effort, carefully determining which nodes to augment the cache with.
Fortunately, we have an advantage that Griewank (1992) does not: we have some control over the dependency structure between the nodes, as we are free to prespecify any dependency structure we like. That is, we do not have to start the binary tree as just a stump. We may exploit this to produce an easier solution.
Given some estimate a of the average step size of the SDE solver, a size of the LRU cache L, and before user makes any queries, we simply make some queries of our own. These queries correspond to the intervals [0, 7/2), 22/2, 7) [0, T/4, [T/4, ./....... so as to create a dyadic tree, such that the smallest intervals (the final ones in this sequence) of size not more than vL. (In practice we use 0.8 x L as an additional safety factor.)
Letting [s, be some interval at the bottom of this dyadic tree, where capable of holding every node within this interval the LRU cache. Once we move past s on the backward pass, then we may in turn hold the entire previous subinterval [u,s] in the LRU cache, and in particular the values of the nodes whose intervals lie within [u, s] may be computed only logarithmic time, due to the dyadic tree structure.
0.8 then are
21

---

Under review as a conference paper at ICLR 2021
This is now analogous the Brownian Tree of Gaines & Lyons (1997); Li et al. (2020). (Up the use of intervals rather than points.) If desired, this approach may be loosely interpreted as placing a Brownian Interval on every leaf of shallow Brownian Tree.
Recursion errors We find that for some problems, the recursive computations of traverse (and in principle also sample, but this is less of an issue due to the LRU cache) can occasionally grow very deep. In particular this occurs when crossing the midpoint of the pre-specified tree: for this particular query, the traversal must ascend the tree the root, and then descend all the way down again. As such t traverse should be implemented with trampolining and/or tail recursion to avoid maximum depth recursion errors.
CPU vs GPU memory We describe this algorithm requiring only constant memory. To be more precise, the algorithm requires only constant GPU memory, corresponding to the fixed size of the LRU cache. As the Brownian Interval receives queries then its internal tree tracking dependencies will grow, and CPU memory will increase. For deep learning models, GPU memory is usually the limiting (and more relevant) factor.
Stochastic integrals What we have not discussed so far is the simulation of integrals such as W.. S Wer dW, and H,, = - W..- dr which are used in higher order SDEs solvers (such as the Kutte methods in Rößler (2010) and the log-ODE method in Foster et al. (2020)). Just like increments Ws,t,
st, these integrals fit nicely into an interval-base data structure.
In general simulating the pair (Wa.s, W.,.) ) is known to be a difficult problem (Dickinson, 2007), and exact solutions only known when W is one or dimensional (Gaines & Lyons, 1994). However, the approximation developed in Davie (2014) and further analysed using rough path theory by Flint Lyons (2015) constitutes simple and computable solution. Their approach generate
a,t W., Wa,t + Hs, Wa,t Wa, Hast + Ae,e.
where Ae, is an anti-symmetric matrix with independent entries X ~ N (0, (1- 6)9) ,)9) <.j.
In both papers, the authors input W into SDE solver (the Milstein and log-ODE methods respectively) and prove that the resulting approximation achieves 2- Wasserstein convergence rate beyond (1/V/N). where N is the number of steps. We have follow-up work planned on this topic.
C EXPERIMENTAL DETAILS
C.I GENERAL NOTES
Code Our code is available at [redacted].
Software We used PyTorch (Paszke et al., 2019) as an autodifferentiable framework. We used
the [redacted] library to solve SDEs. We used the Signatory library (Kidger & Lyons, 2020) to calculate the signatures used in the MMD metric. We used the torchede library (Kidger, 2020) for its interpolation schemes, and solve the neural CDEs used in the classification and prediction metrics. We used the torchdi f feeq library (Chen, 2018) to solve the neural ODEs used in the classification and prediction metrics, and for the ODE components of the Latent ODE and CTFP models.
Architectures By using similar differential equation models, we were able to use essentially the same parameterisation for every model's vector fields. We used essentially the same hyperparameters for every dataset.
To recap, the neural SDE has generator initial condition Ço. generator drift μ4, generator diffusion Tg. discriminator initial condition Es, discriminator drift fo, and discriminator diffusion Jo- All of these are parameterised as neural networks.
Meanwhile Latent ODEs have an ODE-RNN encoder (with a neural network vector field) and a neural ODE decoder (with neural network vector field). The CTFP has an ODE-RNN encoder
22

---

Under review as a conference paper at ICLR 2021
(with neural network vector field) and continuous normalising flow (Chen et al., 2018; Grathwohl et al., 2019) (with a neural network vector field) Additionally Deng et al. (2020) condition the normalising flow the time evolution of a neural ODE of some latent state, which requires another neural network vector field.
In every case, the neural network parameterised feedforward network with 2 hidden layers, width 64, and softplus activations. 'Tee drift, diffusion and vector fields, for every model, all additionally had a tanh nonlinearity their final operation. As per Kidger et al. (2020b) we found that this improved the performance of every model.
The neural SDE's generator has hidden state of size x and the discriminator has hidden state is of size h. These were both taken as x = h = 96. Note that this is larger than the width of each hidden layer within the neural networks, so that the first operation within each neural network is a map from I - TH Somewhat anecdotally, we found that taking the state to be larger than the hidden width was beneficial for model performance."
The Latent ODE likewise has evolving hidden state, which was also taken to be of size 96.
The Latent ODE samples noise from a normally distributed initial condition, we took to have 40 dimensions. The CTFP samples noise from a Brownian motion, which as continuous normalising flow has dimension equal to the number of dimensions of target distribution.
The neural SDE samples noise from both a normally distributed initial condition and a Brownian motion. We took the initial condition to have 40 dimensions. The number of dimensions of the Brownian motion was dataset dependent, see below.
The CTPP included a latent context vector as described in Deng al. (2020). This was taken to have 40 dimensions.
These hyperparameters were selected based on informal initial experiments with all models.
SDE solvers The SDEs used the midpoint method, without adaptive stepping. Recall that the target time series data was regularly sampled and linearly interpolated to make a path. We took the SDE solver to take a single step between each output data point.
ODE solvers The ODEs of the Latent ODE and CTFP models were solved using the midpoint method, for consistency with the SDE solvers.
CDE solvers The CDEs of the classification and prediction models were solved by reducing to ODEs as in Kidger et al. (2020b) and then using the midpoint method, for consistency with the SDE solvers.
Optimisers The CTFP, Latent ODE, and the generator of the neural SDE were all trained with Adam (Kingma & Ba, 2015) with learning rate of 4 x 10-5 The discriminator of the neural SDE was trained with RMSprop with a learning rate of 4 x 10-5, 5. The learning rates were chosen by starting at 4 10 (arbitrarily) and reducing until good performance was achieved. (In particular seeking to avoid oscillatory behaviour in training of the neural SDE.)
Training Every model was trained for 100 epochs. The discriminator of the neural SDE received five training steps for every step with which the generator was trained, as is usual; the number of epochs given at 100 is for the generator, for a fair comparison to the other models.
Batch sizes were picked based on what was the largest possible batch size that GPU memory allowed for; these vary by problem and are given below.
Normalisation All data was normalised to have zero mean and unit variance.
"This has some loose theoretical justification: a signature is a linear differential equation with very large state, and it is universal approximator. (See Kidger et al. (2020b, Appendix B) and references within this is a classical fact within rough analysis.) That is to say, it is simple vector field with a large state, rather than complicated vector field with small state.
23

---

Under review as a conference paper at ICLR 2021
Classifier and predictor The classifier taken to be neural CDE with hidden state of size 32, and whose vector field was parameterised as a feedforward neural network with hidden layers of width 32, with softplus activations and final tanh activation.
The predictor was taken to be a neural CDE/ncural ODE encoder/decoder pair. Both had a hidden state of size 32, and vector fields parameterised feedforward neural network with hidden layers of width 32, with softplus activations and final tanh activation. 32 dimensions were used at the encoder/decoder interface.
The learning rate used was 10-4 for both models, for every dataset and generative model considered, with the one exception of CTFP Beijing Air Quality, where observed divergent training of the classifier: the learning rate was reduced to 10 for this case only.
In all cases they were trained for 50 epochs using Adam, with early stopping if the model failed to improve its training loss over 20 epochs.
The classifier took 80%/20% train/test split of the dataset given by combining the underlying dataset and model- -eder-egeerrted samples of equal size.
C.2 STOCKS
Each sample of length 100.
The batch size was 2048 for every model.
For the neural SDE, the discriminator received 1 epoch of training before the main training (of both generator and discriminator simultaneously) commenced. The weight averaging (over both generator and discriminator) was over every training epoch. The Brownian motion from which noise sampled had dimensions.
The prediction metric was based on using the first 80% the input to predict the last 20%.
C.3 WEIGHTS
Each sample is of length 100. Each sample corresponds the trajectory of single scalar weight, epoch-by-epoch, as a small convolutional model is trained on MNIST for 100 epochs. Every weight from the network is used, and treated as separate sample. This is repeated 10 times. If P is the number of parameters in the convolutional network, then the overall size of the dataset is now (samples 10P, length = 100, channels 1).
The batch size was 4096 for the neural SDE and latent ODE. This was reduced to 1024 for the CTFP, which we found be a very memory intensive model on this problem.
For the neural SDE, the discriminator received 10 epochs of training before the main training (of both generator and diseriminator simultaneously) commenced. The weight averaging (over both generator and discriminator) was over every training epoch. The Brownian motion from which noise was sampled had dimensions.
The prediction metric was based using the first 80% of the input to predict the last 20%.
C.4 BELJING AIR QUALITY
Each sample is of length 24.
The data was normalised have zero mean and unit variance.
The batch size was 1024 for every model.
For the neural SDE, the discriminator received 10 epochs of training before the main training (of both generator and discriminator simultaneously) commenced. The weight averaging (over both generator and discriminator) obvious improvement over averaging every epoch, as was done for the previous two experiments.) The Brownian motion from which noise sampled had 10 dimensions.
over the final 40 epochs of training. (We realised that this was an
24

---

Under review as a conference paper at ICLR 2021
6-
Unara ageil
Averaged mode
5--
4-
3-
Epoch
Figure 3: Loss over training for the time-dependent Orastein-Uhlenbeed example
The prediction metric was based on using the first 50% of the input to predict the last 50%. (An accidental change from the 80%/20% split used in the other experiments; this was kept as is fair, as it is the same for all models on this dataset.)
C.5 MENSTEIN-UHLENBER PROCESS
Target SDE As an additional example for which the true underlying distribution is known, we consider the problem of training neural SDE to match this known SDE.
The target SDE time-dependent
process, of the form dæ = (jut 82,)dt + odW.
We specifically take μ 0.02, 8 = 0.1, 0.4, and generate 8192 samples from t = 0 to 63, sampled at every integer.
Unless otherwise specified, this example followed the same procedure as for the other experiments; for example using small feedforward neural networks parameterise the required neural networks.
Architectures The neural networks had a single hidden layer of width 16, the size of the hidden state (for both generator and discriminator) was taken to be 32, the Brownian motion was 3- dimensional, and the initial noise was 5-dimensional.
The model was trained for 1000 epochs. As with the main text, we found stochastic weight averaging to be important to obtain good performance. This performed by averaging (without weighting) over all epochs from epoch 60, so as to use a small warm-up period.
Optimisation
Loss curves Consider the loss
minlEv.w Do(G(V,, Ex),
which is the generator's 's loss with the normalisation term on the true distribution included, and corresponds the Wasserstein distance between distributions. The notation used as used in Section 2.3 of the main text.
measure this every 20 epochs and plot how varies. See Figure 3. We see that the loss is very unstable for the unaveraged model. This highlights the importance of stochastic weight averaging, as described in Section 5.
Sample paths We plot 10 samples from the true distribution against 10 samples from the learnt distribution (and examined several such plots to be sure that these were representative samples). See Figure 4.
We see that the model has indeed learnt to reproduce the true distribution. does particularly well near the middle; it still has room improve near /. and
63.
25

---

Under review as a conference paper at ICLR 2021
Tiae
2.0
0.5
0.0
-.0-
-.5
Figure 4: Sample paths from the time-dependent Ornstein-Uhlenbeen SDE, and from the neural SDE trained to match it.
Figure 5: Top to bottom, left to right: marginal distributions at t = 39, 44,
6, 19,322, 32, 44, 57. (Corresponding to approximately 10%, 30%. 50%, 70% and 90% of the way along.) See Figure
Marginal distributions Next we plot its marginal distributions
We see that the generated marginal distributions generally match the true marginal distributions. These observations match the samples previously generated; in both cases the generated distribution is a little too low at later times.
Summary This relatively small synthetic example demonstrates that neural SDEs can recover distributions known to be generated by SDEs. Further improvement could likely be obtained by using larger neural networks, and by training for longer.
26


Reviews:
[{"review_id": "xI5-niXKquS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe authors provide a view into neural SDE models, where they mix standard (classical) SDE theory with contemporary neural SDE methods. Overall the paper aims to introduce a generic and user-friendly approach to neural SDEs, with three distinctive contributions (i) interpreting that the mathematical formulation of SDEs is directly comparable to the ML formulation of GANs, (ii) introducing a new way of sampling Brownian motion realisations, and (iii) simplifying the construction of adjoint SDEs by a pathwise formulation.\n\nThe paper is theoretically sound, avoids typical pitfalls in presenting SDE theory in ML papers, and is easy to follow. The authors cite background work in good detail and show that they are aware of the relevant related work and theory in SDE models. The paper is topical, and the theme should be of interest for the audience of ICLR.\n\nMy initial score reflects the concern points that are listed in more detail below, where, however, some of the concerns should be easy to address in updated versions of the paper.\n\n\n#### Concerns\n\n1. Novelty. Even if I found the paper interesting, I can not quite agree with the novelty statement. I found the statement (i) just simply misleading (see #2) and I recommend that you would consider revising this. The construction of (ii) is interesting, but the idea itself seems very closely related to that of Li et al. (2020), which makes it less exciting. The technical details related to (iii) are interesting. The overall novelty is limited, even if the paper is well presented.\n\n2. SDEs are not GANs. The basic idea in Generative Adversarial Networks (GANs) lends itself directly to other generative adversarial models by swapping the generative model and the discriminator. A stochastic differential equation could typically be seen as a generative model, given a drift and diffusion. In the current form, framing the inference as a GAN feels overly complicated and does not help facilitate understanding. (Minor: There is also a typo in the paper title related to this point, 'SDEs' vs. 'GANS').\n\n3. Practical impact not reflected in experiments. The experimental validation is based on three rather simple data sets. No comparison of solvers, nor detailed comparisons to the other methods are included.\n\n4. Fokker-Planck or forward Kolmogorov equation formulation. I expected the Fokker-Planck equation to be discussed as a solution concept or at least mentioned in Sec. 2.\n\n5. Use of the Stratonovich form. Throughout the paper you present Stratonovich SDEs rather than Itô SDEs which could be regarded more standard in most related ML publications. The benefits of the Stratonich form are only briefly covered, and adding details (from an Itô perspective) to Sec. 3.1 and 3.2 would do the paper good.\n\n6. Presentation. The paper is much like a first draft. This shows in the many single-sentence paragraphs, list-like presentation, and additional details needed (in addition to brief experiments, see #3).", "extra_scores": {}, "extra_text": {}}, {"review_id": "dc8wxbAe0p3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# General statements\n\nThis paper introduces an interesting parallel between SDEs and GANs, and pushes the analogy to its practical implications as a way to learn neural SDEs. \n\nGlobally, I found the paper a very good read, although it sometimes lack the details that could be useful for a non-specialist. This could and should easily be corrected by a few sentences here and there.\n\nAlthough I see that everything is included in the supplementary material to actually reproduce all experiments. Still, I believe that there could be some improvements to do. In particular:\n  - I think that the main text / the supplementary could be augmented with a short mention regarding the network structures. even when reading the code, it is not clear how time is handled (since the nets input not only tensors like X_t or H_t, but also time). Should I understand that the raw time stamp is simply concatenated to the other input ?\n  - you didn't clearly mention all the tricks and experiments you tried out. It is not clear to me to what extent the performance you report depends on the network structures you picked.\n\n\nAll in all, I recommend acceptance.\n\n----\nEDIT:\nafter seeing all reviews, and most importantly thinking about it and pondering the answers given by the authors, I am sorry that I must lower my score. \nI still like the paper, that could be accepted in my opinion, but I think it oversells some contributions that are unfortunately not exploited (the brownian interval thing in particular, or am I wrong ?)\n----\n\n## Introduction\nHer are some comments along the way:\n* I could regret that no general background is given for the curious reader that is not already a specialist in SDE or even ODE\n\n## SDEs as GANs\n* please explain the \"Initial condition\" statement better: why is it important that there be an additional source of noise here ?\n* You are mentioning X and H as the (strong) solutions to your SDEs (1) and (2). Are they guaranteed to exist ? I guess the Lipschitz condition you assumed is enough for this. Is that the case ?\n* In the \"training data\" item, H_0 is a function of Y_0. i/ is this Y_0 defined as above in \"initial condition\" ? ii/ Is there a reason H_0 is not a function of z_0 ? iii/ It makes the decision D done on training data actually to depend on \\theta, and not only on \\phi (through Y_0=l_\\theta(\\zeta_\\theta(V)). is that ok ? The item \"initial condition and hidden state\" does not make that point clearer to me.\n* Could you briefly describe gradient penalty, instead of only refering to (Gulrajani 2017) ? That would make the paper more self-contained\n\n## Efficient computation\n* Section 3.1 (rough adjoint equation) is harder for me. I'm ok with the adjoint equation. Then, forgive me but I'm more uncomfortable with the (W, \\mathbb{W}) couple. What is meant exactly by \"sampling\" them ? It means drawing (s,t) and computing the related (W, \\mathbb{W}) ? For each, you compute the solution to the SDE ?\n* now, assuming you get your a_t process. How do you actually use it to perform optimization ? Are you computing the gradient of the parameters wrt a_t and then averaging over time ? Basically, I need some more information on the general scheme to understand 3.1, assuming the adjoint equation is understood.\n\n## Experiments\n\n* The \"weights\" dataset is not super clear. Is the data actually a: 10xPx100 tensor, where P is the number of parameters ? (what is the value of P ?) Just to make sure: the same net is trained for all weights (what I assume), or is it a different net per weight ?\n\n## Considerations\n* in the \"lipschitz regularisation\" of your 2.3 section, you mention using gradient penalty, requiring adjoint, etc. But here, I understand that you actually didn't use these sophistications that were introduced in section 3.1 ? I think you should rephrase a bit here and there to actually better reflect these findings.\n\n## References\n* References are inconsistent. Sometimes abreviations, sometimes full names.\n* Françios-Xavier -> François-Xavier\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "d3rhc6wkFXD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper connects SDEs and GANs and proposed to learn the drift and diffusions in SDE under the framework of GAN. The authors also show how to efficiently simulate the adjoint process and sample the Wiener process.\n\nThough the overall idea is interesting, I have several concerns:\n\n1. I don’t find the library [redacted] that mentioned several times in the paper.\n2. Will it be problematic if the real data is not uniformly sampling across [T] and maybe sparse in some interval {t_1, t_2}? How should we do the interpolation of z in the training data paragraph and why should we linearly interpolate? I feel the description in this part is vague. Can the authors give more formal description and give some intuitions on why should we do like that theoretically?\n3. I feel the Section 3 is more related to computational issue rather than the GAN formulation of neural SDE and can be individual interest? I would like to know more on the properties of the proposed GAN formulation of neural SDE, and I suggest that the authors summarizing the efficient computation part into another single paper, and focus more on the neural SDE as GAN in this paper.\n4. I would like to see more show-cases on the performance of the proposed algorithm on learning given SDE with some simple drift and diffusion term if possible, that may better demonstrate the effectiveness compared with the dataset with unknown drift and diffusion. Also, some generative results is preferred, if there’re proper settings (e.g. some video scenarios), rather than the prediction results in the table.\n\nFrom the current presentation, I does not find sufficient motivation on `why viewing SDE as GAN is good`. I would like to see the detailed motivation, detailed description of methods, at least high level insights that this formulation is beneficial, and sufficient experiments that show the effectiveness of the proposed method. And as a result, I think the current version is not ready for publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cFbPW_fKraQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: this paper claims to show that the “mathematical formulation” of SDEs is “directly comparable” with the formulation of GANS. \n\nI found this paper to be poorly premised. At the outset, the authors state “An SDE is a map from a noise distribution...to the solution of the SDE which is some other distribution on path space.” This statement is incorrect. First of all an SDE is not a map on measure space. It defines the evolution of sample paths of stochastic processes that induce measures. This suggests to me that the authors conflate the measures on path space with paths themselves. I’m also confused by the analogy between sampling SDEs and GANs — one might as well draw analogies with sampling Gaussian distributions. This is entirely confusing.\n\nThere are further fundamental issues that crop up throughout the paper. For instance, in Section 2.2, the authors state that $Y\\stackrel{d}{\\approx} Z$; but what do they mean by this? That  the finite dimensional distributions are approximately equal?\n\nIt appears that the point of the paper is that Wasserstein GANs can be applied to path measures induced by SDEs. This is not a novel insight, in my opinion.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper has some interesting ideas. However, they believe it needs more work before it is ready for publication, especially so with regards to presentation (SDEs as GANs) and the experiments (backpropagating through the solver rather than using the adjoint dynamics). These would significantly strengthen the paper, but would probably require another round of reviews.
Decision: Reject