Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary: this paper claims to show that the “mathematical formulation” of SDEs is “directly comparable” with the formulation of GANS. 

I found this paper to be poorly premised. At the outset, the authors state “An SDE is a map from a noise distribution...to the solution of the SDE which is some other distribution on path space.” This statement is incorrect. First of all an SDE is not a map on measure space. It defines the evolution of sample paths of stochastic processes that induce measures. This suggests to me that the authors conflate the measures on path space with paths themselves. I’m also confused by the analogy between sampling SDEs and GANs — one might as well draw analogies with sampling Gaussian distributions. This is entirely confusing.

There are further fundamental issues that crop up throughout the paper. For instance, in Section 2.2, the authors state that $Y\stackrel{d}{\approx} Z$; but what do they mean by this? That  the finite dimensional distributions are approximately equal?

It appears that the point of the paper is that Wasserstein GANs can be applied to path measures induced by SDEs. This is not a novel insight, in my opinion.
review_title: Review of Neural SDEs made easy

final_score: rating: 6: Marginally above acceptance threshold
review_content: # General statements

This paper introduces an interesting parallel between SDEs and GANs, and pushes the analogy to its practical implications as a way to learn neural SDEs. 

Globally, I found the paper a very good read, although it sometimes lack the details that could be useful for a non-specialist. This could and should easily be corrected by a few sentences here and there.

Although I see that everything is included in the supplementary material to actually reproduce all experiments. Still, I believe that there could be some improvements to do. In particular:
  - I think that the main text / the supplementary could be augmented with a short mention regarding the network structures. even when reading the code, it is not clear how time is handled (since the nets input not only tensors like X_t or H_t, but also time). Should I understand that the raw time stamp is simply concatenated to the other input ?
  - you didn't clearly mention all the tricks and experiments you tried out. It is not clear to me to what extent the performance you report depends on the network structures you picked.


All in all, I recommend acceptance.

----
EDIT:
after seeing all reviews, and most importantly thinking about it and pondering the answers given by the authors, I am sorry that I must lower my score. 
I still like the paper, that could be accepted in my opinion, but I think it oversells some contributions that are unfortunately not exploited (the brownian interval thing in particular, or am I wrong ?)
----

## Introduction
Her are some comments along the way:
* I could regret that no general background is given for the curious reader that is not already a specialist in SDE or even ODE

## SDEs as GANs
* please explain the "Initial condition" statement better: why is it important that there be an additional source of noise here ?
* You are mentioning X and H as the (strong) solutions to your SDEs (1) and (2). Are they guaranteed to exist ? I guess the Lipschitz condition you assumed is enough for this. Is that the case ?
* In the "training data" item, H_0 is a function of Y_0. i/ is this Y_0 defined as above in "initial condition" ? ii/ Is there a reason H_0 is not a function of z_0 ? iii/ It makes the decision D done on training data actually to depend on \theta, and not only on \phi (through Y_0=l_\theta(\zeta_\theta(V)). is that ok ? The item "initial condition and hidden state" does not make that point clearer to me.
* Could you briefly describe gradient penalty, instead of only refering to (Gulrajani 2017) ? That would make the paper more self-contained

## Efficient computation
* Section 3.1 (rough adjoint equation) is harder for me. I'm ok with the adjoint equation. Then, forgive me but I'm more uncomfortable with the (W, \mathbb{W}) couple. What is meant exactly by "sampling" them ? It means drawing (s,t) and computing the related (W, \mathbb{W}) ? For each, you compute the solution to the SDE ?
* now, assuming you get your a_t process. How do you actually use it to perform optimization ? Are you computing the gradient of the parameters wrt a_t and then averaging over time ? Basically, I need some more information on the general scheme to understand 3.1, assuming the adjoint equation is understood.

## Experiments

* The "weights" dataset is not super clear. Is the data actually a: 10xPx100 tensor, where P is the number of parameters ? (what is the value of P ?) Just to make sure: the same net is trained for all weights (what I assume), or is it a different net per weight ?

## Considerations
* in the "lipschitz regularisation" of your 2.3 section, you mention using gradient penalty, requiring adjoint, etc. But here, I understand that you actually didn't use these sophistications that were introduced in section 3.1 ? I think you should rephrase a bit here and there to actually better reflect these findings.

## References
* References are inconsistent. Sometimes abreviations, sometimes full names.
* Françios-Xavier -> François-Xavier

review_title: interesting connection betwee SDEs and GANs

final_score: rating: 5: Marginally below acceptance threshold
review_content: #### Summary

The authors provide a view into neural SDE models, where they mix standard (classical) SDE theory with contemporary neural SDE methods. Overall the paper aims to introduce a generic and user-friendly approach to neural SDEs, with three distinctive contributions (i) interpreting that the mathematical formulation of SDEs is directly comparable to the ML formulation of GANs, (ii) introducing a new way of sampling Brownian motion realisations, and (iii) simplifying the construction of adjoint SDEs by a pathwise formulation.

The paper is theoretically sound, avoids typical pitfalls in presenting SDE theory in ML papers, and is easy to follow. The authors cite background work in good detail and show that they are aware of the relevant related work and theory in SDE models. The paper is topical, and the theme should be of interest for the audience of ICLR.

My initial score reflects the concern points that are listed in more detail below, where, however, some of the concerns should be easy to address in updated versions of the paper.


#### Concerns

1. Novelty. Even if I found the paper interesting, I can not quite agree with the novelty statement. I found the statement (i) just simply misleading (see #2) and I recommend that you would consider revising this. The construction of (ii) is interesting, but the idea itself seems very closely related to that of Li et al. (2020), which makes it less exciting. The technical details related to (iii) are interesting. The overall novelty is limited, even if the paper is well presented.

2. SDEs are not GANs. The basic idea in Generative Adversarial Networks (GANs) lends itself directly to other generative adversarial models by swapping the generative model and the discriminator. A stochastic differential equation could typically be seen as a generative model, given a drift and diffusion. In the current form, framing the inference as a GAN feels overly complicated and does not help facilitate understanding. (Minor: There is also a typo in the paper title related to this point, 'SDEs' vs. 'GANS').

3. Practical impact not reflected in experiments. The experimental validation is based on three rather simple data sets. No comparison of solvers, nor detailed comparisons to the other methods are included.

4. Fokker-Planck or forward Kolmogorov equation formulation. I expected the Fokker-Planck equation to be discussed as a solution concept or at least mentioned in Sec. 2.

5. Use of the Stratonovich form. Throughout the paper you present Stratonovich SDEs rather than Itô SDEs which could be regarded more standard in most related ML publications. The benefits of the Stratonich form are only briefly covered, and adding details (from an Itô perspective) to Sec. 3.1 and 3.2 would do the paper good.

6. Presentation. The paper is much like a first draft. This shows in the many single-sentence paragraphs, list-like presentation, and additional details needed (in addition to brief experiments, see #3).
review_title: SDEs are not GANs, but this is an interesting paper.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper connects SDEs and GANs and proposed to learn the drift and diffusions in SDE under the framework of GAN. The authors also show how to efficiently simulate the adjoint process and sample the Wiener process.

Though the overall idea is interesting, I have several concerns:

1. I don’t find the library [redacted] that mentioned several times in the paper.
2. Will it be problematic if the real data is not uniformly sampling across [T] and maybe sparse in some interval {t_1, t_2}? How should we do the interpolation of z in the training data paragraph and why should we linearly interpolate? I feel the description in this part is vague. Can the authors give more formal description and give some intuitions on why should we do like that theoretically?
3. I feel the Section 3 is more related to computational issue rather than the GAN formulation of neural SDE and can be individual interest? I would like to know more on the properties of the proposed GAN formulation of neural SDE, and I suggest that the authors summarizing the efficient computation part into another single paper, and focus more on the neural SDE as GAN in this paper.
4. I would like to see more show-cases on the performance of the proposed algorithm on learning given SDE with some simple drift and diffusion term if possible, that may better demonstrate the effectiveness compared with the dataset with unknown drift and diffusion. Also, some generative results is preferred, if there’re proper settings (e.g. some video scenarios), rather than the prediction results in the table.

From the current presentation, I does not find sufficient motivation on `why viewing SDE as GAN is good`. I would like to see the detailed motivation, detailed description of methods, at least high level insights that this formulation is beneficial, and sufficient experiments that show the effectiveness of the proposed method. And as a result, I think the current version is not ready for publication.
review_title: An interesting idea, but with poor presentation, especially not focus on the `SDE as GAN view.

Meta-review: The reviewers agree that this paper has some interesting ideas. However, they believe it needs more work before it is ready for publication, especially so with regards to presentation (SDEs as GANs) and the experiments (backpropagating through the solver rather than using the adjoint dynamics). These would significantly strengthen the paper, but would probably require another round of reviews.
Decision: Reject