Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper uses random ordinary differential equations (ODEs) to specify stochastic differential equations (SDEs), where Gaussian processes (GPs) are used to define the vector fields. It is well-known that the dynamics of probability density of states driven by SDEs is given in form of Fokker-Planck equation which is a partial differential equation. In general, there is no closed-form solution of Fokker-Planck equation and the solution is derived by SDE solvers such as Euler–Maruyama (EM), Milstein, and stochastic Runge–Kutta (SRK). These solvers are expensive and inefficient to use in high-dimensional settings. This paper proposes to use an approximate solution by assuming a Gaussian density and tracking the first two moments, mean and covariance matrix. Then the authors derive two ODEs defining the dynamics of mean and covariance matrix. To simplify the ODEs, the authors suggest two schemes: 1- linearizing the Fokker-Planck equation and 2- matching the moments of Fokker-Planck equation. The first scheme is a local approximation and so is less accurate than the second one. However, in terms of runtime, the linearized method is faster than the moment matching one. Overall, the proposed method in this paper can be used to 1- apply prior knowledge to the vector field (trajectories) by using different GP priors and 2- do an efficient inference of probability density function of an SDE by assumed density (Gaussian) approximation of Fokker-Planck equation and deriving two ODEs for the mean and covariance of the distribution which can be solved using ODE solvers and without any stochastic solvers like SRK. 
main_review: - The paper is clear and well written. It tries to address an important problem in neural SDEs: how to efficiently learn the dynamics and avoid expensive stochastic SDE solvers. The authors use the existing methods and tricks to address this issue. They use GPs to model the SDEs as random ODEs and employ the assumed density approximation tricks to estimate the pdf of the resulting SDE which evolve based on Fokker-Planck equation. Both techniques are not novel and the authors have used already known techniques to approximate the pdf of the SDEs without stochastic solvers. Therefore, the main contribution of the paper is not method advancement, but rather using the other methods to 1- improve the efficiency of learning SDEs by removing the need of stochastic solvers, 2- eject prior on the vector field of the ODEs via modeling them as GPs. The proposed method could be a practical improvement by simplifying the solutions of SDEs.


- Some of the results do not show improvement. For example, MOCAP results show that the test MSE is more than that of latent ODE. As one of the reasons of using SDEs instead of ODEs is their ability to capture the uncertainty and overall their better performance in complicated systems (as seen by improvement of latent SDE over latent ODE), having worse performance than latent ODE questions the proposed method's ability in capturing the uncertainty accurately. I think the authors should explain why this is the case and provide more convincing examples. Otherwise, even though this method speeds up learning the SDEs, one could use ODEs if their performance is on par with this method. 
limitations_and_societal_impact: The authors should discuss the limitation of their work in the paper. Their method is good in the sense that it bypasses the use of costly stochastic SDE solvers, however, there is a trade-off between runtime and accuracy and it should be fairly discussed in the paper. I, as a reader, was not fully convinced, based on the results on MOCAP, that this methods is working better than latent ODEs.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a practical methodology for SDE-based inference in machine learning that avoids the use of Monte-Carlo sampling. The methodology invokes a Gaussian density assumption (often used in latent space modelling) to directly approximate the first two moments of the SDE solution. This is achieved by solving a system of ODEs obtained from the Fokker-Planck-Kolmogorov equation. Empirical evidence shows that this scales well with dimension and the approach is demonstrated on Rotating MNIST and motion capture datasets. The motion capture experiment shows the proposed approach achieves competitive accuracy to previous ODE and sampling-based SDE methods, whilst being substantially faster than the latter.
main_review: Strengths:

* The proposed methodology is an interesting and appealing alternative to sampling-based methods for SDE inference.

* Experiments on synthetic and real-world datasets show the assumed density approach scales well with dimension and has competitive accuracy compared to other ODE and SDE models. In particular, the proposed methods are substantially faster than sampling-based SDE models.

* The paper is clearly written.

Weaknesses:

* I am unsure about the originality of the work. As noted by the authors, “The model which we call a ‘GP-SDE’ model in the main paper has appeared in various forms in literature before” and the methods in Section 2 can all be found in Section 9.1 of (Särkkä and Solin, 2019). In the machine learning setting, the linearizing and moment matching via cubature approaches are presented and empirically tested in Sections 4 and 5 of [Look et al. (2020)](https://arxiv.org/pdf/2006.08973v1.pdf). However, recent versions of (Look et al.) no longer discuss these approaches.

* [Look et al. (2020)](https://arxiv.org/pdf/2006.08973v1.pdf) claim that discretizing the SDE before moment matching has advantages over matching moments of the Fokker–Planck–Kolmogorov equation. They argue this with a stability result as well as empirical evidence. A similar discussion would be helpful here.

* In the motion capture experiment, the latent SDE drift and diffusion are taken to be neural networks instead of the mean and covariance functions of a Gaussian process. As a result, I believe the paper should be written with more generality. The key idea is that an assumed density approach can be advantageous over sampling, and this is compatible with both GPs and neural networks.

* The authors claim that the proposed method approximates the mean and covariance of an SDE solution. I believe this claim is entirely heuristic. In particular, there are no theoretical guarantees about the accuracy of the assumed density approximations.

Minor comments:

* In the abstract and introduction, the authors claim that sampling can suffer from the "Curse of Dimensionality" (CoD). In the many areas of numerical analysis, the opposite is true and sampling breaks the CoD. For example, the paper "Solving high-dimensional partial differential equations using deep learning", Han et al. (2018) uses SDE sampling to break CoD when solving PDEs. I think it would be more mathematically accurate just to say that sampling can be inefficient in high dimensions (i.e. leave out CoD).

*  It is not explained where the assumed density approximation (equations (9) and (10)) comes from, though the subsection titles imply that it is the Fokker-Planck-Kolmogorov equation. A brief explanation of this would be helpful.

* The authors do not explicitly define $\mathbf{K}_\ast$ in Section 2.1.

Conclusion:

The paper presents an attractive alternative to the standard Neural SDE methodologies as it avoids costly Monte Carlo simulation. Experimental evidence suggests the proposed methods scale well with dimension and have comparable accuracy to alternative ODE and sampling-based SDE methods (whilst being significantly faster than the latter). Thus, I believe the underlying methodology has great potential. On the other hand, I think the paper should present its methodology in more generality (i.e. discuss Neural SDEs as well as GP-SDEs) and better address its similarities with [Look et al. (2020)](https://arxiv.org/pdf/2006.08973v1.pdf).

Typos:

* ".To" (page 4)
* "see hw description" (page 9)
limitations_and_societal_impact: The authors address that their approach is not suitable for problems where pathwise sample trajectories are required. I think they could give an example, such as using SDEs for generative time series modelling (Kidger et al., (ICML 2021)). Whilst the authors argue that a Gaussian density assumption makes sense for latent space models, they do not address its potential limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the GP-SDE model, and proposes to approximate the marginals with Gaussians whose mean and covariance are approximations of the true quantities. 
main_review: The paper studies ways of Gaussian assumed density approximations can be made practical for dynamics models with an SDE component. In this regard, the paper is novel. 

The paper is of fair quality, is written reasonably clearly, and likely would have some influence in this area. However, the paper does possess certain drawbacks, mostly in terms of technical details. 

Additional questions/comments:
- The motion capture data experiment somewhat seems detached from most of the paper, since the model in the latent space here isn't a GP anymore. Moreover, the objective (17) isn't that of that of the latent SDE model either, since the latent space penalty is only applied at timepoints with observations, as opposed to the entire time horizon of consideration. 
- line 163-166: Could you clarify why the choice is contradictory? The encoder of the latent SDE model outputs a drift function given the observations; this is not a variational Gaussian approximation. 
- Section 2.6: I appreciate there being a section dedicated to computational cost. 

  a. However, (line 217) the time to evaluation the drift and diffusion is likely quite different from the time needed to evaluate the Jacobian, so I wouldn't put them together unless there's a good reason. 

  b. (Line 223): I don't quite get why the sample complexity is exponential in d. This issue seems more nuanced as presented, and likely depends on the divergence/metric considered. It seems the local squared error should scale polynomially in d, in which case the strong and weak errors might just not be exponential. If one is only concerned with approximating functionals, then Monte Carlo sampling shouldn't be terrible. 
limitations_and_societal_impact: This is a strict methodology paper. This discussion does not apply.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, a fast, scalable inference method in stochastic differential equations is introduced, which proceeds by approximating the solutions of the Fokker-Planck-Kolmogorov (FPK) equation instead of relying on sampling schemes. In particular, the solutions to the FPK system is approximated by a Gaussian, whose mean and covariance are shown to satisfy a system of ODEs, which are then further approximated by linearisation or moment matching. The authors demonstrate that solving these approximate systems have significant computational advantage to sampling directly from the SDEs, especially in high dimensional settings and yield qualitatively similar results on experiments on the rotating MNIST dataset and motion capture data.
main_review: Originality:
While sharing similarities with [26], I believe that this paper manages to develop an original approach to learning SDEs by starting from the Fokker-Planck equations and deriving everything from there, which I haven't seen yet in the literature. I find the idea of combining this technique with VAEs interesting as indeed, a Gaussian approximation is necessary anyways in the encoder. The GP-SDE idea is also interesting, however I am not fully convinced that this equivalence between random ODEs and SDEs holds (more details below).

Quality:
I believe that overall the paper is relatively easy to read and is of high standard in terms of intellectual contribution and experiments. However, there were some places that were unclear to me, in addition to a few misprints as I list below.

Clarity:
Below I list some parts of the paper that were unclear to me or what I believe are misprints:
- The matrix $K_*$ is not defined in §2.1
- Similarly in §2.1. Is $Q$ the precision matrix? It seems to appear out of nowhere.
- $z_1$ in (5) should be $z_*$?
- I am unsure whether the claim about the equivalence between the random ODE with a GP vector field and SDEs in §2.1 is correct. In [13], they only show the relation between the two systems under the Euler-Maruyama discretisation, where the discretised transition probability can be shown to be the same. However, this certainly does not imply the equivalence in probability for the full continuous system, and I have not seen a rigorous treatment of the equivalence between the two systems.
- I think some experimental details for the rotating MNIST and video motion data are lacking:
  - For the rotating MNIST, it wasn't clear to me how the training took place in the latent space. From my understanding, the GP-SDE is trained using the dataset $D = (z_i, \Delta z_i)$, but how do you obtain this data in the latent space from training data in the observation space?
  - In the video motion experiment, $p$, $q$ and $\gamma$ in the loss (17) are not defined.
- Line 310. what is "hw description"?
- Figures
  - What are the red vector fields in figure 3? Is it the GP posterior mean?
  - In figure 6 a), are the plotted paths the mean trajectories of the SDEs or the sample paths (I presume the former given how smooth it is)? This was not specified in the body.
  - How many Euler-Maruyama trajectories are used in the top panel of figure 5?

Significance:
I can see that the techniques introduced in this paper may be useful in situations when a single SDE run is extremely costly. For example in weather prediction, typically only a few ensemble runs are feasible, yet having good uncertainty quantification is vital so there may be some potential use case in this area.

limitations_and_societal_impact: I do not see any potential negative use of this work. In terms of limitations, the quantitative results of the proposed method comes at a cost of the gain in computational speed as shown in the experiments in §4. However, the baseline performance is expected to decrease with fewer sample size so it might have been interesting to investigate the smallest number of samples that are required in the baseline before it outperforms moment-matching/linearisation.
review_title: null

Meta-review: This paper is concerned with inference of neural stochastic differential equations. In general, exact inference in these is intractable. This paper proposes a pragmatic technique, whereby the discretization is modeled by a recursive (deterministic) Gaussian approximation. They show computational advantages, particularly in higher dimensions.

All reviewers felt this was a valuable method, with a worthy contribution. The main concerns that were identified were relationships to some existing work (largely addressed in feedback) some unclear notation (also addressed in feedback), and some questions about the computational cost of the Jacobian step. Here, we echo the reviewer in that it is not reasonable to treat function evaluations and Jacobian evaluations as equal.  The authors are trusted to fulfill their commitment to including a large neural-network model with strong dependencies to consider this timing issue experimentally. Still, overall the method appears correct, plausibly useful and the paper is well-written, thus I recommend acceptance.
Decision: Accept (Poster)