Scalable Inference in SDEs by Direct Matching of the Fokker–Planck–Kolmogorov Equation
Authors: Arno Solin
Ella Maija Tamir
Prakhar Verma
Venue: NeurIPS 2021 Poster (2021)
Keywords: Stochastic differential equation
approximative inference
neural SDE
Gaussian process

Abstract: Simulation-based techniques such as variants of stochastic Runge–Kutta are the de facto approach for inference with stochastic differential equations (SDEs) in machine learning. These methods are general-purpose and used with parametric and non-parametric models, and neural SDEs. Stochastic Runge–Kutta relies on the use of sampling schemes that can be inefficient in high dimensions. We address this issue by revisiting the classical SDE literature and derive direct approximations to the (typically intractable) Fokker–Planck–Kolmogorov equation by matching moments. We show how this workflow is fast, scales to high-dimensional latent spaces, and is applicable to scarce-data applications, where a non-parametric SDE with a driving Gaussian process velocity field specifies the model. 

Scalable Inference in SDEs by Direct Matching of the Fokker-Planct-Koloogoow Equation
Arno Solin
Ella Tamir
Prakhar Verma
Aalto University
Aalto University
Aalto University Espoo, Finland prakhar wwr.a@aalto........mmmmmm.
Espoo. Finland
Espoo, Finland
arno
ella. aaamir@@alto...
Abstract
Simulation-bare techniques such as variants of stochastic Runge-Kutta are the de facto approach for inference with stochastic differential equations (SDEs) in machine learning. These methods are general-purpose and used with parametric and non-parametrie models, and neural SDEs. Stochastic Runge-Kutta relies on the use of sampling schemes that can be inefficient in high dimensions. We address this issue by revisiting the classical SDE literature and derive direct approximations to the (typically intractable) moments. We show how this workflow is fast, scales high-dit high-dimessional latent spaces, and is applicable to scarce-data applications, where a -on-aarmetrric SDE with a driving Gaussian process velocity field specifies the model.
Backer-Planck- Kolmogorov equation by matching
1 Introduction
Probubility
approximitionn
density s(2,, 1) m solution to the Pokker-Paanck- Kehmogono PDF
Differential equations are the standard method of modelling change over time. In deterministic systems the dynamics specifying how the system evolves, are typically written in the form of ordinary differential equation (ODE). The dynamics act as prior knowledge and often stem from first-principles in application areas such physics, control engineering, chemistry, or compartmental models in epi- demiology and pharmacokinetics. Recently, learning ODE dynamics with modern automatic differentiation packages in machine learning has awakened an interest in black-box learning of continuous-time dynamics (e.g., [6, 37]) and enabled their more general use across time-series modelling applications.
Euler Manuyama
sample injeectoriss
GP SDE (SDE model specification prior)
Figure 1: Views into solutions to SDEs.
A stochastic differential equation (SDE, [30, 40]) can be seen as a generalization of ODEs to stochastic dynamical settings, where the driving forces fluctuate are uncertain. Stochastic dynamics appear naturally in applications where small (and typically unobserved) forces interact with the process, such as tracking applications, molecule motion, gene modelling, stock markets. In machine learning, SDE models have received wide-spread attention due to their robustness and appealing properties for uncertainty quantification.
The concept of 'solution' to an SDE is broader than that of an ODE. As the process is stochastic, the full solution entails a probability distribution, p(z, t), depending on time and covering the space (see, e.g., [36]). For Itô type SDEs, the evolution of the probability mass can be described in terms of the Mlakke--PPankk-oooooooo (FPK) partial differential equation (backward Kolmogorov equation). This equation is typically intractable. and instead the de facto approach for inference in SDEs in machine learning is sampling. The most common approaches in this space are based on
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Stochastic Runge- Kutta schemes (such as the Euler-Maruyana scheme) which are derived from the Itō Itô-Taylor series. These schemes sample realization trajectories of the SDE by driving the dynamics with numerical simulation of Brownian motion. However, these schemes suffer from drawbacks both related to (ordinary) Runge-Kutta methods-suu such as step-size and sensitivity to stiffness-as as well as problems associated with any sampling schemes, such as high number of samples required for an accurate representation of the underlying distribution.
Despite these problems, few contemporary SDE approaches in machine learning explore SDE solutions beyond stochastic Runge- Kutta (or even the to broaden this view, and in Fig. 1 we sketch an example where we show three alternative solution perspectives Gaussian process prior SDE model (GP-SDE): the FPK probability density field, Euler-Maruyama samples, and a Gaussian assumed density approximation. We argue that a Gaussian approximation in latent space SDEs is reasonable, as Gaussian approximations are typically employed anyway in observation models, and allow for speeding up learning by an order of magnitude.
scheme). Our aim is to try
The contributions of this paper are as follows. (i) We go through the workflow connecting 'random ODE' models with Itô SDEs driven by a Gaussian process prior over the velocity field, which allows for convenient specification of prior knowledge on the vector field and induces an implicit prior over the SDE trajectories; (ii) We revisit the classical SDE literature and derive direct approximations to the (typically intractable) Fokker-Plande form that avoids sampling-basee inference in the latent space, which makes inference fast and does not require sampling a high number of trajectories; (iii) We show how this workflow fast, applicable to scarce-data applications, and how it also extends to previously presented latent SDE models.
Kolmogorov equation in an assumed density Gaussian
1.1 Related Work
Neural ODEs [6] model ODE dynamics by a neural network. Such models were developed further in [37], where the encoder is an ODE-RNN that improves modelling of irregularly sampled time series. A latent Bayesian neural ODE model, ODE?VAE, was examined in Yıldız et al. [49], where an encoder is combined with an ODE model whose second order dynamics are given by a Bayesian neural network. The neural ODE paradigm of modelling latent dynamics has been expanded to neural SDEs [26. 12, 46, 19]. where the typical workflow that a variational autocncoder (VAE, [21, 35]) is combined with a latent neural SDE, whose drift and diffusion are modelled by neural networks. In addition modelling time series, neural SDEs have been used in generative models [16, 43, 44], where the generation of images from noise is modelled as the reverse-time process of a diffusion SDE by using Langevin dynamics on score-based models. Continuous normalizing flows are another model family, which applies ODE dynamics in a generative model [6, 11].
These works leverage simulation/samplingg for solving the SDE in the latent space. The model can be trained by stochastic adjoint method [26, 18]. More recently, latent neural SDEs have been trained deterministically by moment matching [27]. However, they discretized the system before matching the moments, while we form a direct approximation to the solution of the FPK. Compared optimizing the moment ODEs, as discussed in [27], by maximizing likelihood, we regularize during inference by Gaussian process priors, or prior stochastic processes as in [26]. Approximative solutions to non-linear SDEs have been applied earlier in filtering theory, where the optimal filter is approximated by a Gaussian assumed density filter [23]. In [39], such approximations are used for continuous mattrueus-discrete state-space modelling. An alternative to assumed density filters are local linearization methods [31, 41], and simulation-based Itô-Taylor series solutions, stochastic Kutte-Kutta methods, and leapfrog methods such as Verlet for second-order SDES (see [22, 40]). The approximations presented in this work are also related to GP approximations [4, 2] of SDEs. The linearization approximations are related to statistical lincarization [10, 42], and variational approximations [4].
Orthogonally the SDE inference, we also consider SDE model specification in terms of GP priors. The seminal work by Ruttor et al. [38] considered GP-SDE models with unit diffusion. Yildiz et al. [48] built a model, where the drift and diffusion are sparse Gaussian processes with iimeeindeeen kernels. In Hegde et al. [13], spatio-temporal SDE with GP priors for the drift and diffusion was combined with a GP as a continuous version of deep Gaussian processes. State-space models with a GP latent state transition function [8] train a non-parametric latent process to approximate unobserved dynamics. These are related to hierarchical GP dynamics [45, 24], where prior knowledge of the system can be encoded in multi-level hierarchies, for modelling, e.g., walking dynamics.

---

OBSERVATION SPACE
LATENT SPACE
PREDICTION SPACE
desciiled diffrreiee cosutiie A(E) Vola, 4
We approcrrate ( PLA3)
Irnonder
Unadd:
Repettine Icaurp knsmed oo lasat cycko
Chserannoe B05 chonded to RacH spoce iepeicetttion
Prodiction decoded book in aherattes qpace will ussciattio
Figure 2: Latent dynamics workflow. The observations (left) are encoded into the latent space, where the dynamics of the system learned an SDE model. We approximate the solution to the SDE by a Gaussian for which we can approximate the dynamics of the first moments by an ODE system, thus avoiding sampling in the latent space. Predictions are finally mapped back.
2
Methods
We draft the methodology based on the latent dynamics workflow components as presented in Fig. 2. The focus is first specifying models for the latent space dynamics, starting from implicit priors in terms of random ODEs which we frame as Itō SDEs. Thereafter, the focus shifts from model specification inference, where we show that the Maaeeee-PPanc-oommoggooo efficiently approximated in an assumed density form, and finally brings us to cover the likelihood structure of these models as well.
equation can be
2.1 Random Field Ordinary Differential Equations as SDEs
The continuous dynamics of a latent (unobserved) z(t) E Rd can conveniently be written in the form of general first-order ordinary differential equation
d dt (1) (1())))
where va(+): :d x R, + Rd denotes the velocity field parametrized by 0. This is a general form of non-linear ODE system, where the dynamics are deterministic and fully characterized by vo(-). The methodology presented in this section directly extends to the case where the vector field va is dimeddepen,n but we omit the time dimension for simplicity of notation. Previously, the implicit prior on ztt) over t specified by Eq. (1) has been generalized to stochastic models by considering v-(.) to be stochastic. These models are known as 'random' ODE models, and the random field v. is typically either characterized by Gaussian random field or Gaussian process model (see, e.g., [38, 13I) or some parametric model (e.g., [26)).
We consider an unconventional ODE model (or actually no ODE model at all, to be precise), where we specify GP prior [34] over the velocity field form of multi-output Gaussian process prior:
v(2,t) (((
where μ Rd Rdd is a mean function and x : Rd dxdd matrix valued covariance function. The Gaussian process prior is completely specified by its mean and covariance function, which encapsulate the assumptions about the sample processes/fillds (such as continuity, differen- tiability, curl, divergence, etc.): fu(z) == El[(2)) and <(2, x') == ([(v(z) (v(z)(v((z)>
). In Fig. 3, we will consider examples of useful vector- valued covariance functions that encode prop- erties on the vector field. For inference, the GP is conditioned
input-output pair observations D = (Zz, Az of the vector field, where Az, represents the observed derivative at z, The conditioned vector field representation for an arbitrary point z., in the latent space can be given by
v(2.) D GP(F(v(z) DD. Cov(v(z) | (D)),
where the E||] and Cov[-] denote the marginal mean and (co)variance (for the multi-output GP, which means that the marginals are vector-valued). These take the form [34]: E(v(z.)) K,K-1, and
3

---

0.5
52
-0.5
-0.5
-0.
7
7
0.55
0.5
0.5
-0.5
0.5
(a) Independent RBF priors
(b) Curl-free prior
(c) Divergence free prior
Figure 3: Effect of different GP priors in GP-SDE with observations (large red arrows) the GP posterior (small red arrows): (a) Shows results for independent RBF priors over 21 and 22; (b) shows results for the curl-free covariance function (encoding 'loop-aversion';; (c) shows results for the divergence-free covariance function (encoding 'energy preservation". The hyperparameters
0.2, o² 0.1) are the same in each.
Cov[v(z.)] = k(Z,, 2.) - K.K K, where y are the stacked observations of the derivatives such that y = (Az,
AZ,,)T The Gram matrix K corresponds evaluations of the covariance function such that K; k(2;; 2,)) are sub-blocks of K corresponding to the observation pairs (i,j), and K K + where 7 is nugget (observation noise/discrepanccy term, and K, is the cross-covariance between 2 and 2,. The prohibitive cubic computational scaling associated with GP models manifests in the inversion of K, and thus for large n, approximations based on inducing points projections are used practice to avoid this explicit inversion. In the light of Eq. (1) with vz)) GP :), the ODE is driven by a multi-dimensional Gaussian random field conditioned on D. straightforward way of dealing with a model this kind, to do inference by sampling random draws of the velocity field from the GP, and then drive the ODE with those samples (can be viewed as an Monte Carlo approach for drawing ODE realizations).
However, a more convenient way is to specify the prior over the stochastic dynamics in stochastic differential equation form. At its core, lot previous work in this space hinges on the realization that if everything is essentially Gaussian, an equivalent model can be specified in terms of an Itô SDE describing the stochastic evolution of trajectories affected by the GP velocity field (see 13, for discussion]. Informally, this takes the white noise form 4 (t)
f(z) + L(z) w(t), where f(z) [z) and L(z) denotes square-root factor such that LL T Cov [z()] (in the sealar case, just the square-root, and in the multi-output case, e.g., the Cholesky factor). Here w(t) is white noise process with unit spectral density. It is worth noting that we not give guarantees for direct link between the random ODE in Eq. (2) and the following SDE formulation (see App. A.3 for discussion). Yet, formally we write a memilarly-behavinn
SDE in the standard Itô SDE form:
dz(t) (t) f(z, t) dt + L(z, t) dB(t),
where d,3(t) is vector-valued unit Brownian motion (the spectral density Q is set D). For GP-SDE, the drift driven by the GP mean, f(z, !) == E[v(z) and the diffusion by the square-root factor of the marginal covariance at ztt), L(z, f) = / (C)] To be precise, the GP-SDE drift and diffusion at point z, determined by the GP predicted mean and variance 2,, which can be written as
f(z., ()-,,))BBv(( B(v KK and L(z.,))= (v()) k(2,, 2.) K.K (5)
Fokker-Planck-Kolneeee
Equation
We are interested in solving SDE models of the form in Eq. (4), but without the restriction that the drift and diffusion are defined by a Gaussian process, and present the related theory with model-agnostic view on the problem. Because the resulting solutions are stochastic processes, the full solution to the SDE can be characterized by time-evolving probability density function. Let z(4o) ((2(())) be some initial condition which assumed to be independent of the Brownian motion. The probability

---

velecity
0.1s
1.0s
2.0s
4.0s
solution
GP marginal std
Samples
Assumed density
Exnet Canssioo Lincarized
Matched
Figure 4: Approximations to the FPK equation: The top-left figure shows the 8 observations (black arrows) and the inferred GP velocity (grey arrows, marginal uncertainty in shaded blue). The top-row shows the progression of the probability mass p(z, 6) following the Mlakker-Planc--olmoope equation. The middle rows show Muler-Maruyama sample trajectories for the problem, and the bottom row compares the two assumed density approximations to the exact Gaussian approximation of the FPK solution. The bottom ellipses are 95% confidence regions.
density p(z, t) of the solution of the SDE in Eq. (4) solves the Fokker-Planck Kolmoooro (FPK) partial differential equation (PDE):
Öp(z,t) t) Øt
Σ
3 oz, (1((, 01+ Σ
O²
[L(z, (2.
(6)
3z, dzz
For proof, see [40]. This PDE is also known the Fokker-Planck equation (in physics) and the forward Kolmogorov equation (in stochastics). An appealing alternative form [40, Sec. 5.3] of the FPK equation can be given in terms of the following evolution equation with the adjoint operator A
Öp
p. with A'( -- E (z(z, 01
8 Oz,
Σ
82
(L(Z 1) QL
(2, (+))
Oz Oz
(7)
Eq. (7) allows for various kind of approaches for direct approximation of the FPK equation either by basis function approximations, finite differences, or other methods (Sec. 9.6 in [40] provides examples of using point collocation, Ritz-Galeriin, and FEM type of methods for approximating the solution). For example, the results in Fig. and Fig. 4 are estimated by grid discretization over z and solving the resulting finite-dimensionall ODE corresponding to Eq. (7) by the matrix exponential: p(t) exp( A( fo)). See App. A.2 for details. Even if the widely-used Euler-Maruyama, Milstein, and more general stochastic Runge-Kutta schemes are derived from Itô-Taylor series, the resulting methods can still be viewed as approximation of p(z, ().
2.3 Assumed Density Approximation of FPK
For the purpose of modelling latent space dynamics of systems of the kind in Fig. 2, we note that there the common practice of solving the latent SDE through costly simulation/samplin and then employing variational (Gaussian) approximation in the encoder/decoder seems contradictory. That is, it might be unnecessary to sample realizations of the trajectory dynamics, if the interest is only in time-marginals of the process. Thus, we seek to directly characterize the first two moments of the solution to the FPK equation in Sec. 2.2. We replace the FPK solution with Gaussian approximation of form
p(z,t) ~ N(z P())),
(8)
5

---

where mt) and Pt)) are interpreted as a mean and covariance of the state of the solution at time t. This kind of approximation is commonly referred to as a Gaussian assumed density approximation (see, e.g., [23, 391), because the computations are done under the assumption that the state distribu- SDE
tion Gaussian. Assumed density approximations are common signal processing- methodology, and we refer the reader to Sec. 9.1 in [40] for detailed overview. Following [39], we revisit the idea that a Gaussian process approximation to the SDE Eq. (4) can obtained by integrating the following differential equations from the initial conditions m(to) E[z((o)] and ((o)) = Cov[z(to)] to the target time t:
dm
/ f(z,t) () N(z m, P) dz and
(9)
dt dP
f f(z,t) (2- m) N(z dz
dt
-(2- (2-m) (2, N( m. P2 + / L(( (2, m. Pz.
(10)
These equations for the evolution of the first moments of the solution to the SDE can be interpreted expectations over the drift and diffusion dynamics of the SDE, and can be derived from the FPK in Eq. (6). Conveniently, these expressions are not stochastic, but instead take the form of an ODE system that-given the integrals tractable-can be solved with out-of-the-boo ODE solvers. However, even Eqs. (9) and (10) provide a generic Gaussian assumed density approximation framework for SDEs, an implementation of the method requires solving the following kind of d-dimensional Gaussian integrals:
/
En [x]
N(z m. Pz
(11)
In the following sections we will consider two approaches (local lincarization and moment matching with symmetric quadrature) which scale linearly in the number of latent dimensions d.
2.4 Linearizing the FPK Equation
Local linearization around the m (via Taylor series approximation) is a classical approach widely used for this type of Gaussian integrals in machine learning and filtering theory [17. 29]. If the function f(z, t) is differentiable, the covariance differential equation can be simplified by using Stein's lemma [32] such that
[ f(z,tt) (z m) N(z m, P) dz F2 () m. m, P) P,
(12)
where F2(2, t) is the Jacobian of f(z, t) with respect to z. Linearizing around the mean m and approximating the diffusion L(z, 2,, t) A: L(m, t) gives linearized form of Eqs. (9) and (10): dm
dP
dt t(, and dt which provides a direct way of propagating the moments of the latent SDE through an ODE for the mean and covariance without the need of drawing multiple sample trajectories. The resulting ODE is (d + dd2) -iimensiona, and only requires one evaluation of the drift, diffusion, and Jacobian per step.
PF (m,t) + (m,, t) P LLm,m t) Q (m,
(13)
2.5 Matching Moments of the FPK Equation
The local linearization approach given in the preceding section is efficient, but fully local. An alternative way of constructing an assumed density approximation
p(z, t) is to directly match the moments by solving the Gaussian integrals in Eqs. (9) and (10) by Gaussian quadrature methods.
The approximation to Eq. (11) would take the form f g(z, t) N(z m, P) dz R3 E, (() g(z()), f). for arbitrary integrand g(z, t), weights w and so called sigma points ( m+VP Here vP denotes square-root factor of P such as the Cholesky decomposition. The multi-dimensiona Gaussian quadrature (or cubature, see (7]) rule is characterized by the evaluation points and their associated weights {(E,, w:)). We write Eqs. (9) and (10) in Gaussian assumed density form which matches the moments by quadrature as follows [40]:
dm
de E, wi()) f(m+ / t) and
(14)
dP = E, (0()) f(m+vP (. T VP" dr
(m+ E. with vP (. V E., + E, w L)
(15)

---

The computational complexity of this approach is highly dependent on the choice of quadrature method. A typical choice in ML applications would be Gauss- quadrature, which factorizes over the input dimensions leading to exponential number (p") of function evaluations/sigma points in the input dimensionality d for desired order p. In order to guarantee scalability, we employ (p 3) is exact for
a symmetric 3rd order cubature rule [3] which similarly to polynomials up to degree 3. The points are given by scaled unit coordinate vectors e, such that
Vdei, for i I,...,d d, - Vdee for d + 1, ................................ 2d,
(16)
and the associated weights w; This approach provides direct way of propagating the "true' moments of the latent SDE through an ODE for the mean and covariance and without the need of drawing multiple sample trajectories. The resulting ODE is (d + dl dimeeniioo, and requires only 2d evaluations of the drift and diffusion per step.
2.6 Analysis of the Computational Complexity
In terms of the asymptotic computational complexity, the linearization approach in Sec. 2.4 requires (1) evalua-
tions of the drift, diffusion, and Jacobian per step. The
Wall-clock dime (x)
moment matching approach in Sec. 2.5 requires (d)
evaluations of the drift and diffusion as well as an ((d3)
10
Cholesky decomposition per step. The simplest Monte Carlo simulation method with p samples requires (p))
CPU
evaluations of the drift and diffusion per step. Addition- ally, the naive requirement for p grows exponentially in d. On the other hand, the simulation approach is fully par- allelizable over p. while the moment matching approach the number of nonparallelizable operations is (dd) with the Cholesky decomposition being the bottleneck, and the linearization approach is nonparallelizable While the linearization approach has the lowest number of function evaluations with respect to d, the cost of computing the
Maier-Mangyrmo
Monent naicting Lucarization
100
300
400
Latent dimensionality, df
Results for
Wall-cleck time (6)
Jacobian can be prohibitively large for arbitrarily com-
The dotiral line shows when monant noichng pertiens equally well bater bater-Mang Manyana
plex models. Nevertheless, the Jacobian is available in closed-form for GPs and may be evaluated reasonably fast for neural network based drifts, see App. B.2 for empir- ical computational costs of evaluating the Jacobian when growing the network size. Thus we expect the FPK ap- proximation schemes to always be beneficial in CPU-only cases (incl. CPU mutti-threading and embedded devices). In multicore GPU use, for www-dmensiionl d, sampling remains appealing if GPU memory does not become bottleneck.
200
300
Number Maler-Maruyama trajoctories,
Figure 5: Empirical timing experiments with error of final margins matched.
3 Experiments
The goals of the experiments are three-fold: We first provide study of the computational complexity. Then, we look into properties of the GP-SDE model from Sec. 2.1, where the experiments are concerned with showcasing model specification rather than inference. Finally, benchmark problems with high- nigh-ddmmensionl inputs for learning a latent SDE model, where we test the performance of the approximations presented when the model is not defined by GPs, the SDE methods presented Sec. are model-agnostic.
consider two
Timing Experiments To confirm the analysis in Sec. 2.6 and provide a practical insight, we run numerical experiments with the error of final marginal mean/covariance controlled. We use high- dimensional model of independent Beneš SDEs (d=(() = tanh(z) di + d,3(t,, see [40]) with different 2g per dimension. The model non-linear and solution-space multi-modal, but both p(z, I) and the marginal moments (m(t), P(t) are available in closed form (see App. B. 1). In comparison
7

---

Input
Predictions
3
Uscline mulcting 11000
3
3
3
3
3
3
4
3
Momen
3
3
3
3
3
3
3
4
M
3
3
3
3
R
(a) Latent space trajectories
(b) Forward prediction
Figure 6: Results on rotating MNIST. (a) shows the latent space prediction mean trajectories for one test image. Evolution of the true trajectory is shown in HSV colour and the predicted trajectory by Euler-Maruyama, moment matching, and linearization scheme is shown in green, blue, and orange (all overlapping one another). (b) shows the progression of the prediction (mean and std dev) of the test image, when it traverses the learned dynamics in the latent space. Both the moment matching and linearization schemes match the baseline (with 1000 Euler-Maruyama trajectories).
to Euler-Maruyama, we control the number of trajectories by bounding the error (in terms of KL divergence) between Euler-Maruyama and the ground-truth to match the error in our moment matching approach, and consider the methods equivalent terms of the quality of the solution. See App. B.1 for the required number of trajectories to match the KL divergence for each dimensionality plotted. Fig. shows GPU/CPU wall-clock times (GPU: NVIDIA Tesla V100 32 GB with Intel Xeon Gold 6134 3.2 GHz; CPU: Xeon Gold 6248 2.50GHz). We implement the models in Py Torch [33] and report means of 10 repetitions (std negligible). In diwdeneniional cases, both approximation methods outperform sampling, whereas in high dimensions GPU parallelisation becomes dominant and only the linearized approximation remains highly competitive. This example should favour sampling: In the Beneš model, the number of trajectories, n, per d in Fuler-Maruyana remains low (n is linear in d), which is due to the diagonal (independent) diffusion matrix. In a correlated latent space would grow super-linearly (even exponentially), which would further push the difference between methods.
GP-SDE Model Specification We consider an GP-SDE model with just 8 observations of the dynamies, where the lack of data can be compensated for with encoding prior knowledge into the model. We use the model formulation given in Sec. 2.1, and study the effect of GP priors, the first of which is an independent squared exponential (RBF) prior for each dimension which encode continuity and smoothness in the velocity field. The second GP prior is the multi-dimensional curl-free kernel [47] (see App. B.3) which encodes the assumption of curl-free random vector field. This property can be interpreted as 'loop aversion' in the GP-SDE context. The third prior, is multi- dolli--dmmeniionll divergence- free kernel [47] which encodes the assumption of no divergence in the random vector field. This property can be interpreted 'energy preservation' or cource-freeness, These properties are visible in Fig. 3, where the hyperparameters are fixed to same values for all models.
Assumed Density Approximation of the FPK We provide an illustrative example of the moment evolution methods in a GP-SDE model with 8 observations along bean curve and independent squared exponential GP priors per dimension. As baseline, we solve the FPK in Eq. (7) by finite- differences discretization in z (see App. A.2 for details). Fig. shows the evolution of the SDE solution over the time-course of 5 s. The probability mass dissolves quickly, which is hard to interpret from the top-row figure alone. Comparison between the point clouds and the top row shows that with 1000 trajectories and just two-dimensional space, it is hard to capture detailed structure in the SDE solution. The bottom row compares the local lincarization (Sec. 2.4) and the moment matching (Sec. 2.5) assumed density approximations to the exact Gaussian approximation of the FPK solution. The linearized approach is mode-seeking (matches local curvature), while the moment matching approach captures the overall structure of the optimal Gaussian approximation.
the spirit of Fig. 2, we run the proposed methods on Rotating MNIST ([25), available under CC BY-SA 3.0), similar to [49, 5]. The data set consists of various handwritten digit '3's rotated uniformly in 64 angles. We train a VAE [21] first by freezing the latent space dynamics, allowing us to generate the latent samples for learning the dynamics by applying the
Rotating MNIST
8

---

Table I: Rotating MNIST results.
Table 3: Test MSE on 297 future MOCAP
points averaged over 50 samples. 95% con- fidence interval reported based on t-statistic. rrssult from [49], Fresults from [26]
INFERENCE SCHEME
MSR
NI.PD (t = 64)
Euler Baler-Maruyama
0.046 + 0.006
33.0 5.4
Moment matching
0.031 = 0.007
Linearization
0.052 0.007
54.5 9.9
METROD
TEST MSE
DTSBN-S [9]
0.02!
npODE [14]
22.96
Table 2: Wall-clock timings for MOCAP.
NeuralODE [6]
22.49 0.8&1
ODE [49]
10.06 1.4
TIMD/S.LSDD
NUMBER OF PATHS
LIN.
MoM. MAT:
25
200
VAE-VA [49]
8.09 = 1.95"
Latent ODE [37]
5.98 = 0.28
6.0#.1 CPU 1.8
GPU 2.1
3 27.72.7
39.9=.1
40.4=.7
SDE [26]
1.03 0.20*
1882.1
38.6=1.5
94.2=3.5
Latent SDE tassumed density)
7.55 + 0.05
trained VAE. Then freezing the VAE encoder/decoder and training 16-dimensional GP-SDE model in the latent space with independent squared exponential GP priors (see App. B.5). In Fig. 6 we feed in one observation and let follow the learned dynamics of rotation. As the baseline, we use 1000 trajectories computed using Euler-Maruyama. Fig. 6a demonstrates the model's 's capability to learn the latent trajectory, and we show the trajectories for all the methods in three latent dimensions with most variation. The trajectories for the three methods overlap, and qualitatively the results are identical in Fig. 6b. Quantitatively the brute-foree sampling baseline gives slightly better MSEs over images and final-step mean negative log predictive densities (NLPD, see Table 1).
Motion Capture Data The CMU walking data set ([)), CMU MoCap available under CC BY- ND 4.0) is a real-world noisy data set with 50 sensors that track human subject's walking. As in Yıldiz et al. [49] and Li et al. [26], we model the sequences of a single subject, 35, for which there are 16 train set, three validation set and four test set sequences. The task is to predict the state of the system in the future given three initial points. In this experiment, we demonstrate that replacing SDE solver-based methods by an assumed density approximation, a latent neural SDE system can be learned efficiently without sampling trajectories. For this purpose, and for better comparability to earlier work, the latent SDE drift and diffusion are neural networks. As in Li et al. [26), we regularize the learned posterior process by a prior process. The loss function consists of three terms: reconstruction loss, VAE encoded initial position KL-diiergence, and the KI-divergence between posterior/prior processes. The moments of the posterior SDE approximation are denoted by m(t), P Ptt,, those of the prior process m. (t), P. (t), and the observation times by {t, )m The loss becomes
L = - - logp(x((g) x(1))) + DKI. (a(a((o) ((())) Il [[z)] Eド= DKL (N(m())), ((m.())) (17)
where ztt,) ) and x(t;) are the latent codes and observations, respectively, denotes the conditional encoder distribution, p(z) is the normal distribution, and log p(x(t,) (t) is the model likelihood of the observations, given latent codes. While training, the parameters to optimize include those of a latent neural network which defines the prior and posterior dynamics, and of the VAE which encodes an initial point and decodes at each discrete time step corresponding to a train set frame. As the VAE trained simultaneously to the latent dynamies, the approach does not provide a set of true latent samples to compare to, in contrast to the MNIST experiments. The VAE encoder prior is the normal distribution, whereas the encoder posterior is acting as the initial distribution for the latent SDEs. For the encoder design, use fully-connected neural network, which encodes the three first data points to create latent state and context vectors. As the prior process was a SDE with zero drift and aI diffusion. The context is passed through the dynamics, similar to the treatment of velocity in Yıldız et al. [49] (see App. B.6).
The result in Table 3 is competitive considering that solving the latent SDE with the linearization approach roughly matches the required computation budget for one stochastic Runge-Kutta sample in the other methods. This is highlighted in Table 2 which shows the wall-clock times for the model used in this experiment both in a CPU and a GPU setting (see hardware description in previous experiment).

---

4 Discussion and Conclusions
In this paper our interest has been in both SDE model specification and approximative inference. We considered GP-SDE models for data-scarce applications that need injection of prior knowledge such as in Fig. 3. For inference, we built upon the established methodology of assumed density approximations in signal processing for directly approximating the solution distribution of an Itô SDE, where the methods apply to any latent space SDE models.
We put interest in weak solution concepts for SDEs. We considered both linearization and moment matching based methods for capturing the first two moments of the SDE solutions, which respectively require only ((1) and (d) evaluations of the model functions per solver step in latent space dimensionality d. Furthermore, they only require solving of one ODE rather than simulating multiple trajectories. This makes them orders of magnitudes lighter than the current state-of-thh-ar in neural SDEs, which we analyzed both through theoretical bounds in Sec. 2.6, ran numerical experiments with the final error controlled (Fig. 5), and highlighted the practical wall-clock time in a MOCAP experiment. We further argued that Gaussian assumption makes sense in applications of the form in Fig. 2 as one is typically employed anyway in the encoder-decoder.
There some key differences between our assumed density approach and commonly used sampling approaches to solving neural SDE models. While stochastic Runge- Kutta methods are typically either concerned with strong (pathwise) weak (in distribution) solutions, we leverage the even weaker solution concept of only tracking the first two moments of the solution. This simplification of turning solving the SDE into deterministic ODE problem of its moments, comes with some remarkable computational savings, and often suffice in practical modelling. Then again, the proposed method does not lend itself well to cases where pathwise sample trajectories are required.
We recognize that we cannot give guarantees for the approximated Gaussian integrals to capture the evolution of the true moments outside particular special cases (e.g., 3rd order cubature is exact for polynomials up to order 3). Yet the performance on practical applications is considered reliable, and these kinds of approached are commonly employed across assumed density filtering in signal processing. The use of the approximations schemes we present inherently makes the assumption that the tome-marginal of the process are of higher interest compared to the pathwise solutions to the SDE. While the sampling-base based methods are less efficient than assumed density approximations, their use is well-justified in applications where sampling trajectories is the purpose of the application.
Codes for the methods and experiments in this paper are available at http://github.com/AaatMM// scalableiinferenc--i--SSDD
Acknowledgments and Disclosure of Funding
Authors acknowledge funding from Academy of Finland (grant numbers 324345 and 339730). We also wish to thank the anonymous reviewers for their comments on our manuscript, and Çağatay Yildız and Xuechen Li for providing useful details on the MOCAP experiment. We acknowledge the computational resources provided by the Aalto Science-IT project. ET has been employed part-time at Sellforte Oy during the project. The MOCAP data was obtained from mocap.es.cme em.. edu (created with funding from NSF EIA-0196217).
References
[1] CMU graphics lab motion capture database. URL http:://mocap.cs.cmm www.:e...p.......cu.........
[2] Ala-Luhtala, S. Särkkä, and R. Piché. Gaussian filtering and variational approximations for Bayesian smoothing in continuous-discrete stochastic dynamic systems. Signal Processing, 111:124-136, 2015.
[3] I. Arasaratuam and S. Haykin. Cubature Kalman filters, IEEE Transactions on Automatic Control, 54(6): 1254-1269, 2009.
[4] C. Archambeau and M. Opper. Approximate inference for continuous-time Markov processes. In Bayesian Time Series Models, pages 125-140. Cambridge University Press. New York. NY, 2011.
[5] F. P. Casale, A. V. Dalca, L. Saglietti, J. Listgarten, and N. Fusi. Gaussian process prior variational autoencoders. In Advances in Newral Information Processing Systems 31 (NeurIPS), volume 31, pages 10369- 10380. Curran Associates, Inc., 2018
10

---

[6] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems 31 (NeurlPS), pages Inc., 2018.
Curran Associates,
[7] R. Cools, Constracting cubature formulae: The science hehind the art. In Acta Numerica, volume 6, pages 1-54. 1997.
[8] R. Frigola, Y. Chen, and C. E. Rasmussen. Variational Gaussian process state-space models. In Advances in Neural Information Processing Systems 27 (NIPS), pages 3680 3688. Curran Associates, Inc., 2014.
[9] Z. Gan, C. Li, R. Henao, D. E. Carlson, and L. Carin. Deep temporal sigmoid belief networks for sequence modeling. In Advances in Neural Information Processing Systems 28 (NIPS), pages 2467-2475. Curran Associates, Inc., 2015,
[10] A. Gelb, editor. Applied Optimal Estimation. MIT Press, Cambridge, MA, 1974.
[11] Grathwohl, Q. Chen, J. Bettencourt, and Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learning Representations (ICLR), 2019.
[12] J.S. Ha, Y.JJ Park, H.-J. Chae, S.-S. Park, and H.-L. Choi. Adaptive path-integral autoencoders: Repre- sentation learning and planning for dynamical systems. In Advances in Neural Information Processing Systems 3/ (NeurIPS), pages 8927-8938. Curran Associates, Inc., 2018.
[13] P. Hegde, M. Heinonen, H. Lähdesmäki, and S. Kaski. Deep learning with differential Gaussian process flows. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1812-1821, PMLR, 2019.
[14] Heinonen, C. Yildiz, H. Mannerström, J. Intosalmi, and H. I Lähdesmäki. Learning unknown ODE models with Gaussian processes. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1959-1968. PMLR, 2018
[15] Hensman, A. G. d. G. Matthews, and Z. Ghahramani. Scalable variational Gaussian process classification. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, volume 28 of Proceedings of Machine Learning Research. PMIRR 2015.
[16] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33 (NewrIPS). Curran Associates, Inc., 2020.
[17] A. H. Jazwinski. Stochastic Prucesses and Filtering Theory. Academic Press, New York, 1970.
[18] P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. In Advances Neural Information Processing Systems 33 (NeurIPS). Curran Associates, Inc., 2020.
[19] P. Kidger, J. Foster, X. Li, and T. J. Lyons. Neural sdes infmite-dimensional gans. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5453-5463 2021. 2021.
[20] D. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint 1412:944,,.6980, 2014.
[21] D. Kingma and M. Welling. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representations (ICLR), 2014.
[22] P. F. Kloeden, F. Platen, and H. Schurz. Numerical Solution of SDE Through Computer Experiments. Springer Science Business Media, 2012.
[23] H. Kushner. Approximations to optimal nonlincar filters. IEEE Transactions on Automatic Control, 12(5): 546 556, 1967.
[24] N. D. Lawrence and A. J. Moore. Hierarchical Gaussian process latent variable models. In Proceedings of the 24th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 481-488. PMLR, 2007.
[25] Y. LeCun, C. Cortes, and C. J. Burges. The MNIST database of handwritten digits, 1998. URL http: wyann.lecun.comm watpnncom///xx//msss....o
[26] X. Li. T.-K. L. Wong, R. T. Q. Chen, and D. Duvenaud. Scalable gradients for stochastic differential equations. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1570-3882, PMLR, 2020.
11

---

[27] A. Look, C. Qiu, M. Rudolph, J. Peters, and M. Kandemir. Deterministic variational inference of neural SDEs. arXiv preprint arXiv:2006.08973: 2020. NB: A moment matching/linearization approach is discussed only in the initial version (v1) of the manuscript.
[28] A. G. d. G. Matthews, M. van der Wilk, T. Nickson, K. Fujii, A. Boukouvalas, P. León- Villagrá, Z. Ghahra- mani, and Hensman. GPfoow: Gaussian process library using TensorFlow. Journal of Machine Learning Research, 18(40):1-6, apr 2017.
[29] P. S. Maybeck. Stochastic Models, Estimation and Control, volume 2. Academic Press, New York, NY, 1982.
[30] B. Øksendal. Stochastic Differential Equations: An Introduction with Applications. Springer, New York, NY, sixth edition, 2003.
[31] T. Ozaki. bridge between nonlinear time series models and nonlinear stochastic dynamical systems: a local linearization approach. Statistica Sinica, 2(1):113-133) 1992.
[32] A. Papoulis. Probability, Random Variablex, and Stochastic Processes. McGraw- Singapore, 1984.
[33] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chaman, T. Killeen, 7. Lin, N. Gimelshein, L.. Antiga, A. Desmaison, A. Kopf, F. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high- pere-peeformmnce deep learning library. In Advancey in Neural Information Processing Systems 32 (NeurIPS), pages 8024-8035 Curran Associates. Inc., 2019.
[34] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[35] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning. volume 32 of Proceedings of Machine Learning Research, pages 1278- PMLR, 2014.
[36] I. C. G. Rogers and D. Williams. Diffusions, Markov Processes and Martingales, Volume 2: Itō Calculus. Cambridge Mathematical Library. Cambridge University Press, Cambridge. 2000.
[37] Y. Rubanova, R. T. Q. Chen, and D. K. Duvenaud. Latent ordinary differential equations for irregularly- sampled time series. In Advances in Neural Information Processing Systems 32 (NeurlPS), pages 5320- 5330. Curran Associates, Inc., 2019.
[38] A. Ruttor, P. Batz, and M. Opper. Approximate Gaussian process inference for the drift function in stochastic differential equations. In Advances in Neural Information Processing Systems 26 (NIPS), pages 2040-2048. Curran Associates, Inc., 2013.
[39] S. Särkkä and Sarmavuori. Gaussian filtering and smoothing for continuous-discrete dynamic systems. Signal Processing, 9302)500-5100, 2013.
[40] S. Särkkä and A. Solin. Applied Stochastic Differential Equations. Cambridge University Press, 2019.
[41] T. Shoji and Ozaki. Estimation for nonlinear stochastic differential equations by local lincarization method. Stochastic Analysis and Applications, 16(4):733-752, 1998.
[42] L. Socha. Linearization Methods for Stochastic Dynamic Systems, volume 730 of Lecture Notes in Physics. Springer, Berlin, 2008.
[43] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 11918-1 11930, Curran Associates, Inc., 2019.
[44] Y. Song. J. Sock-sickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score- generative modeling through stochastic differential equations. arXiv preprint, arXiv:201 I. 13446,, 2020.
[45] M. Titsias and N. D. Lawrence. Bayesian Gaussian process latent variable model. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 844-851. PMLR, 2010.
[46] B. Tzen and M. Raginsky. Neural stochastic differential equations: Deep latent Gaussian models in the diffusion limit. arXiv preprint arXiv: 19900099833... 2019.
[47] N. Wahlström. Modeling of Magnetic Fields and Extended Objects for Localization Applications. PhD thesis, Linköping University, Sweden, 2015.
12

---

[48] C. Yildiz, M. Heinonen, J. Intosalmi, H. Mannerstrom, and H. Lahdesmaki. Learning stochastic differential equations with Gaussian processes without gradient matching. In 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6, 2018.
[49] C. Yıldız, M. Heinonen, and H. Lahdesmaki. VAE: Deep generative second order ODEs with Bayesian neural networks. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 13412-13422 Curran Associates, Inc., 2019.
13


Reviews:
[{"review_id": "jOKbx3zqCHX", "reviewer": "Reviewer_P9hS", "summary": "The paper studies the GP-SDE model, and proposes to approximate the marginals with Gaussians whose mean and covariance are approximations of the true quantities. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies ways of Gaussian assumed density approximations can be made practical for dynamics models with an SDE component. In this regard, the paper is novel. \n\nThe paper is of fair quality, is written reasonably clearly, and likely would have some influence in this area. However, the paper does possess certain drawbacks, mostly in terms of technical details. \n\nAdditional questions/comments:\n- The motion capture data experiment somewhat seems detached from most of the paper, since the model in the latent space here isn't a GP anymore. Moreover, the objective (17) isn't that of that of the latent SDE model either, since the latent space penalty is only applied at timepoints with observations, as opposed to the entire time horizon of consideration. \n- line 163-166: Could you clarify why the choice is contradictory? The encoder of the latent SDE model outputs a drift function given the observations; this is not a variational Gaussian approximation. \n- Section 2.6: I appreciate there being a section dedicated to computational cost. \n\n  a. However, (line 217) the time to evaluation the drift and diffusion is likely quite different from the time needed to evaluate the Jacobian, so I wouldn't put them together unless there's a good reason. \n\n  b. (Line 223): I don't quite get why the sample complexity is exponential in d. This issue seems more nuanced as presented, and likely depends on the divergence/metric considered. It seems the local squared error should scale polynomially in d, in which case the strong and weak errors might just not be exponential. If one is only concerned with approximating functionals, then Monte Carlo sampling shouldn't be terrible. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a strict methodology paper. This discussion does not apply. ", "review_text": "", "extra_scores": {"ethical_concerns": "This is a strict methodology paper. This discussion does not apply. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "bBi2mOv36XT", "reviewer": "Reviewer_vsyv", "summary": "The paper presents a practical methodology for SDE-based inference in machine learning that avoids the use of Monte-Carlo sampling. The methodology invokes a Gaussian density assumption (often used in latent space modelling) to directly approximate the first two moments of the SDE solution. This is achieved by solving a system of ODEs obtained from the Fokker-Planck-Kolmogorov equation. Empirical evidence shows that this scales well with dimension and the approach is demonstrated on Rotating MNIST and motion capture datasets. The motion capture experiment shows the proposed approach achieves competitive accuracy to previous ODE and sampling-based SDE methods, whilst being substantially faster than the latter.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* The proposed methodology is an interesting and appealing alternative to sampling-based methods for SDE inference.\n\n* Experiments on synthetic and real-world datasets show the assumed density approach scales well with dimension and has competitive accuracy compared to other ODE and SDE models. In particular, the proposed methods are substantially faster than sampling-based SDE models.\n\n* The paper is clearly written.\n\nWeaknesses:\n\n* I am unsure about the originality of the work. As noted by the authors, “The model which we call a ‘GP-SDE’ model in the main paper has appeared in various forms in literature before” and the methods in Section 2 can all be found in Section 9.1 of (Särkkä and Solin, 2019). In the machine learning setting, the linearizing and moment matching via cubature approaches are presented and empirically tested in Sections 4 and 5 of [Look et al. (2020)](https://arxiv.org/pdf/2006.08973v1.pdf). However, recent versions of (Look et al.) no longer discuss these approaches.\n\n* [Look et al. (2020)](https://arxiv.org/pdf/2006.08973v1.pdf) claim that discretizing the SDE before moment matching has advantages over matching moments of the Fokker–Planck–Kolmogorov equation. They argue this with a stability result as well as empirical evidence. A similar discussion would be helpful here.\n\n* In the motion capture experiment, the latent SDE drift and diffusion are taken to be neural networks instead of the mean and covariance functions of a Gaussian process. As a result, I believe the paper should be written with more generality. The key idea is that an assumed density approach can be advantageous over sampling, and this is compatible with both GPs and neural networks.\n\n* The authors claim that the proposed method approximates the mean and covariance of an SDE solution. I believe this claim is entirely heuristic. In particular, there are no theoretical guarantees about the accuracy of the assumed density approximations.\n\nMinor comments:\n\n* In the abstract and introduction, the authors claim that sampling can suffer from the \"Curse of Dimensionality\" (CoD). In the many areas of numerical analysis, the opposite is true and sampling breaks the CoD. For example, the paper \"Solving high-dimensional partial differential equations using deep learning\", Han et al. (2018) uses SDE sampling to break CoD when solving PDEs. I think it would be more mathematically accurate just to say that sampling can be inefficient in high dimensions (i.e. leave out CoD).\n\n*  It is not explained where the assumed density approximation (equations (9) and (10)) comes from, though the subsection titles imply that it is the Fokker-Planck-Kolmogorov equation. A brief explanation of this would be helpful.\n\n* The authors do not explicitly define $\\mathbf{K}_\\ast$ in Section 2.1.\n\nConclusion:\n\nThe paper presents an attractive alternative to the standard Neural SDE methodologies as it avoids costly Monte Carlo simulation. Experimental evidence suggests the proposed methods scale well with dimension and have comparable accuracy to alternative ODE and sampling-based SDE methods (whilst being significantly faster than the latter). Thus, I believe the underlying methodology has great potential. On the other hand, I think the paper should present its methodology in more generality (i.e. discuss Neural SDEs as well as GP-SDEs) and better address its similarities with [Look et al. (2020)](https://arxiv.org/pdf/2006.08973v1.pdf).\n\nTypos:\n\n* \".To\" (page 4)\n* \"see hw description\" (page 9)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address that their approach is not suitable for problems where pathwise sample trajectories are required. I think they could give an example, such as using SDEs for generative time series modelling (Kidger et al., (ICML 2021)). Whilst the authors argue that a Gaussian density assumption makes sense for latent space models, they do not address its potential limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QtIeLy1u3s4", "reviewer": "Reviewer_qke7", "summary": "In this paper, a fast, scalable inference method in stochastic differential equations is introduced, which proceeds by approximating the solutions of the Fokker-Planck-Kolmogorov (FPK) equation instead of relying on sampling schemes. In particular, the solutions to the FPK system is approximated by a Gaussian, whose mean and covariance are shown to satisfy a system of ODEs, which are then further approximated by linearisation or moment matching. The authors demonstrate that solving these approximate systems have significant computational advantage to sampling directly from the SDEs, especially in high dimensional settings and yield qualitatively similar results on experiments on the rotating MNIST dataset and motion capture data.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nWhile sharing similarities with [26], I believe that this paper manages to develop an original approach to learning SDEs by starting from the Fokker-Planck equations and deriving everything from there, which I haven't seen yet in the literature. I find the idea of combining this technique with VAEs interesting as indeed, a Gaussian approximation is necessary anyways in the encoder. The GP-SDE idea is also interesting, however I am not fully convinced that this equivalence between random ODEs and SDEs holds (more details below).\n\nQuality:\nI believe that overall the paper is relatively easy to read and is of high standard in terms of intellectual contribution and experiments. However, there were some places that were unclear to me, in addition to a few misprints as I list below.\n\nClarity:\nBelow I list some parts of the paper that were unclear to me or what I believe are misprints:\n- The matrix $K_*$ is not defined in §2.1\n- Similarly in §2.1. Is $Q$ the precision matrix? It seems to appear out of nowhere.\n- $z_1$ in (5) should be $z_*$?\n- I am unsure whether the claim about the equivalence between the random ODE with a GP vector field and SDEs in §2.1 is correct. In [13], they only show the relation between the two systems under the Euler-Maruyama discretisation, where the discretised transition probability can be shown to be the same. However, this certainly does not imply the equivalence in probability for the full continuous system, and I have not seen a rigorous treatment of the equivalence between the two systems.\n- I think some experimental details for the rotating MNIST and video motion data are lacking:\n  - For the rotating MNIST, it wasn't clear to me how the training took place in the latent space. From my understanding, the GP-SDE is trained using the dataset $D = (z_i, \\Delta z_i)$, but how do you obtain this data in the latent space from training data in the observation space?\n  - In the video motion experiment, $p$, $q$ and $\\gamma$ in the loss (17) are not defined.\n- Line 310. what is \"hw description\"?\n- Figures\n  - What are the red vector fields in figure 3? Is it the GP posterior mean?\n  - In figure 6 a), are the plotted paths the mean trajectories of the SDEs or the sample paths (I presume the former given how smooth it is)? This was not specified in the body.\n  - How many Euler-Maruyama trajectories are used in the top panel of figure 5?\n\nSignificance:\nI can see that the techniques introduced in this paper may be useful in situations when a single SDE run is extremely costly. For example in weather prediction, typically only a few ensemble runs are feasible, yet having good uncertainty quantification is vital so there may be some potential use case in this area.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any potential negative use of this work. In terms of limitations, the quantitative results of the proposed method comes at a cost of the gain in computational speed as shown in the experiments in §4. However, the baseline performance is expected to decrease with fewer sample size so it might have been interesting to investigate the smallest number of samples that are required in the baseline before it outperforms moment-matching/linearisation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "0Da5ZQIp1n", "reviewer": "Reviewer_nBa9", "summary": "This paper uses random ordinary differential equations (ODEs) to specify stochastic differential equations (SDEs), where Gaussian processes (GPs) are used to define the vector fields. It is well-known that the dynamics of probability density of states driven by SDEs is given in form of Fokker-Planck equation which is a partial differential equation. In general, there is no closed-form solution of Fokker-Planck equation and the solution is derived by SDE solvers such as Euler–Maruyama (EM), Milstein, and stochastic Runge–Kutta (SRK). These solvers are expensive and inefficient to use in high-dimensional settings. This paper proposes to use an approximate solution by assuming a Gaussian density and tracking the first two moments, mean and covariance matrix. Then the authors derive two ODEs defining the dynamics of mean and covariance matrix. To simplify the ODEs, the authors suggest two schemes: 1- linearizing the Fokker-Planck equation and 2- matching the moments of Fokker-Planck equation. The first scheme is a local approximation and so is less accurate than the second one. However, in terms of runtime, the linearized method is faster than the moment matching one. Overall, the proposed method in this paper can be used to 1- apply prior knowledge to the vector field (trajectories) by using different GP priors and 2- do an efficient inference of probability density function of an SDE by assumed density (Gaussian) approximation of Fokker-Planck equation and deriving two ODEs for the mean and covariance of the distribution which can be solved using ODE solvers and without any stochastic solvers like SRK. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is clear and well written. It tries to address an important problem in neural SDEs: how to efficiently learn the dynamics and avoid expensive stochastic SDE solvers. The authors use the existing methods and tricks to address this issue. They use GPs to model the SDEs as random ODEs and employ the assumed density approximation tricks to estimate the pdf of the resulting SDE which evolve based on Fokker-Planck equation. Both techniques are not novel and the authors have used already known techniques to approximate the pdf of the SDEs without stochastic solvers. Therefore, the main contribution of the paper is not method advancement, but rather using the other methods to 1- improve the efficiency of learning SDEs by removing the need of stochastic solvers, 2- eject prior on the vector field of the ODEs via modeling them as GPs. The proposed method could be a practical improvement by simplifying the solutions of SDEs.\n\n\n- Some of the results do not show improvement. For example, MOCAP results show that the test MSE is more than that of latent ODE. As one of the reasons of using SDEs instead of ODEs is their ability to capture the uncertainty and overall their better performance in complicated systems (as seen by improvement of latent SDE over latent ODE), having worse performance than latent ODE questions the proposed method's ability in capturing the uncertainty accurately. I think the authors should explain why this is the case and provide more convincing examples. Otherwise, even though this method speeds up learning the SDEs, one could use ODEs if their performance is on par with this method. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should discuss the limitation of their work in the paper. Their method is good in the sense that it bypasses the use of costly stochastic SDE solvers, however, there is a trade-off between runtime and accuracy and it should be fairly discussed in the paper. I, as a reader, was not fully convinced, based on the results on MOCAP, that this methods is working better than latent ODEs.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is concerned with inference of neural stochastic differential equations. In general, exact inference in these is intractable. This paper proposes a pragmatic technique, whereby the discretization is modeled by a recursive (deterministic) Gaussian approximation. They show computational advantages, particularly in higher dimensions.

All reviewers felt this was a valuable method, with a worthy contribution. The main concerns that were identified were relationships to some existing work (largely addressed in feedback) some unclear notation (also addressed in feedback), and some questions about the computational cost of the Jacobian step. Here, we echo the reviewer in that it is not reasonable to treat function evaluations and Jacobian evaluations as equal.  The authors are trusted to fulfill their commitment to including a large neural-network model with strong dependencies to consider this timing issue experimentally. Still, overall the method appears correct, plausibly useful and the paper is well-written, thus I recommend acceptance.
Decision: Accept (Poster)