Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ##########################
The paper proposes an algorithm and an analysis of its convergence.
The algorithm propose to learn a model of temporal data y_1 , ... , y_T given input x_1, ..., x_T
The observations are assumed to arise from a deterministic latent h 
governed by a piecewise continuous ode (in between consecutive times t_i, t_i+1)
with additional deterministic jumps at transitions.

In the ODE-RNN paper, the latent h can be expressed as a single ODE for the whole time horizon (rewriting the
jump with a skip transition).

This paper appears to me as taking this expression and choosing a particular bounded form for the
dynamics, jump and readout functions
A statistical asymptotic analysis of the convergence of the algorithms, for random times and inputs is given. 

##########################

Methodology:
I find the paper quite difficult to read, I blame both its structure and my lack of ease with the mathematics used here.
However from what I have understood of the algorithm proposed, I find the methodological contribution very limited.

Clarity:
I come from the machine learning community and read with no difficulty 
papers cited in the related work section.
In comparison, I find this paper extremely difficult to read and parse despite containing the same kind of information.

Asymptotic analysis.
I leave to other reviewers the evaluation of the convergence analysis.
My evaluation being partial, my confidence rating is set accordingly.

* For a machine learning paper presenting in the end a 3 line simple algorithm, the paper contains
a lot of superfluous mathematical notation that crowds the paper and make the reading very tedious.
Many of the papers cited Brouwer 2019, Rubanova 2019, Li 2020, offer a much smoother read in that respect.
As is, this paper feels better suited to a more specialist statistics venue.

* For example, many elements are introduced in the main text and are not really necessary to understand what the paper does
The detailed section on random inputs is used only in a theorem coming later, why have it in the main text in so much details.
On the other end, a description of the method this paper builds on is left into appendices.

#########################

Additional comments:
* the formatting of the references is very inconsistent, please update


review_title: Not very novel method, difficult read.

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper introduces Neural Jump Ordinary Differential Equations as a method for learning models of continuous-time stochastic processes sampled at random time epochs. Specifically, the paper studies the problem of estimating the marginal conditional expectation (i.e., the L2 optimal approximation conditional on the available information) by estimating an auxiliary stochastic differential equation, parameterized by  neural networks, that approximates the conditional expectation of the process of interest at each point in time. The neural networks are trained by using a “randomized” mean squared-loss objective. The main theoretical results in the paper include asymptotic consistency of the optimal objective value in the limit of a large neural network, as well as consistency of a Monte Carlo sample average estimator of the value. The paper also establishes the L2 convergence of the estimated auxiliary solution to the marginal conditional expectation.

The technical details in the paper are mostly sound, and I believe it should be of interest to a wide community. The question of estimating stochastic models sampled at regular or irregular intervals is of broad utility. There are some technical issues however, but these can be resolved I believe. In particular, in the conclusions of Theorem 4.1 and 4.2, it seems as though the authors claim almost sure convergence, unless I am misunderstanding their statement.  What the authors establish is convergence in L2, but why does is imply almost sure convergence? Wouldn’t one require uniform integrability to conclude more? Furthermore, this is not a process level convergence result, and therefore I do not believe that they can conclude (as in Remark G.3) that the limit holds almost surely. (Also, the authors seem to suggest tin Remark G.2 that they’re not establishing L2 convergence, but this could be a problem with the writing). 

Coming to the writing, I note that I did find the paper somewhat sloppy in its use of terminology and notation. For instance on p.1 the authors state “...while stochastic processes are continuous in time...” This is not quite true, since one can define discrete-time stochastic processes. I also found the discussion around  justifying “irregular” sampling of the stochastic process to be poorly written. In particular, it is stated that “...dividing the time-line into equally-sized intervals...is again making assumptions and information is lost...” well, any sampling will involve a loss of information, and the randomized sampling process described in this paper also involves assumptions. I don’t think this comment is appropriate. Furthermore, the authors do not make a clear case for why their irregular sampling procedure is appropriate. I’m quite certain that the sampling process introduces bias into the estimation; for instance, Theorem 1 of ref. [1] below provides sufficient conditions under which an “irregularly” sampled estimator of a functional of an SDE is unbiased. The authors must do a better job of justifying their method. I would also urge them to add an example of a randomized sampling process; for instance, a Poisson process sampler would satisfy their definition, in which case the sampling time epochs form an ordered statistic.

Coming to the development of the stochastic model, it is unclear to me as to why all of the random “objects” cannot be defined on the same sample space. Essentially, couldn’t one view the sampling process as a point process on the same sample space supporting the SDE? 

Next, in Prop. 2.1, the authors state that the optimal adapted process approximating process is \hat{X}_t — but \hat{X_t} is only defined pointwise (i.e., at each time ‘t’) and it is not defined as a stochastic process. Indeed, for that the authors must describe the finite dimensional distributions for all finite sets of time epochs to define the stochastic process. I believe it is inappropriate to call this a stochastic process. This doesn’t affect the main results, since the authors only establish convergence in an L2 sense, where the full distribution is not necessary. 

Some further minor comments:
1. Change the term “observation dates” to “observation epochs”.
2. Change “amount of observations” to “number of observations” (or samples).
3. On P.3 in the definition of \lambda_k, the set \mathcal{B}([0,T]) is undefined.
4. The notation defining the function \tilde{\mu} is very confusing, please change.
5. P.4 “...since the variation of u...” should be “...since the total variation of u...”
6. What do you mean by “ergodic” approximation of the objective? Isn’t it simply a sample average approximation? Which ergodic theorem is playing a role here?
7. I would also urge you clearly define what you mean by \mathbb{L}-convergence, for completion. 

[1] Unbiased Estimation with Square Root Convergence for SDE Models, Chang-Han Rhee and Peter W. Glynn.
review_title: Review of neural jump ODEs

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a method for learning the conditional expectation of stochastic process in an online fashion. The paper bears a considerable theoretical treatment, derived from the stochastic filtering literature, which is present both in the main body of the paper and the appendix. Besides the model, the paper also aims to provide a theoretical justification of the convergence of their method. 

I find the contribution of the paper somewhat obscure, its aims to be incremental with respect to the previous literature, and the experimental validation heavily unconvincing. I support my recommendation through the following points: 

- Following the well known (by now) neural ODE and neural jump SDE, the contribution of the paper seems minor. The authors state that they focus on giving theoretical guarantees, however, these are specific and loosely validated experimentally
- There is a fair amount of space dedicate to the theoretical presentation of the background, I agree with the importance of theory, but I failed to see how that theory supports the claims of the paper. 
-the experiments are limited: only 3 synthetic examples and only one real-world one.
-the authors states that their method focuses on approximating (directly) the conditional expectation, this seems to be a different with the previous literature. However, if that's the case, the authors should consider more benchmarks such as linear filters (adapted to non-uniformly-spaced data), Gaussian processes, or general time series models. 

This paper does have a contribution. My recommendation is that  the authors show it in a clearer (to the point) manner with an improved experimental validation.  
review_title: Contribution over existing work is unclear, experimental validation is minimal.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The submission studies a simplified model of ODE-RNN and GRU-ODE-Bayes theoretically, proves convergence results, and presents experimental results in companion with the theoretical results. 


The paper does a good job in defining concepts precisely. Though, this has come at a cost of highly complex notation, which may hinder the average researcher in the ML+diffeq community, who may not have a strong background in probability theory, to understand the paper. I would therefore recommend the authors to simplify the notation by deferring the precise mathematical definition of concepts such as information sigma algebras to the appendix. 


The section (sec. 2.4) on optimal approximation of a stochastic process in the main text is somewhat vague. Optimality certainly depends on the cost function being considered, in which case, the appendix states that the 2-norm is used here. The particular norm being used is somewhat independent of the construction of the probability space, e.g. we could consider the same prob. space and evaluate the difference between the random variable and the fixed prediction using some other function, say the metric function induced by the L1-norm). This makes terms such as “L^2(omega X omega tilde, ...)-minimizer” somewhat confusing. Note my comment here is somewhat handwavy about the precise technicalities, but it should convey the relevant idea. 


My main concern regarding the paper is about novelty. It seems that the model considered in section 3 falls broadly in line with ODE-RNN and GRU-ODE-Bayes. On the other hand, the experiments section also doesn’t compare against latent ODE, which is a strong but relevant baseline. 


The section (sec. 4) on theoretical convergence results mostly assume that the ERM can already be found. This rather strong assumption therefore leaves the theorems in that section not unexpected, and at the same time, less relevant for practitioners. It is also unclear whether convergence rates can be derived. 

The paper does a decent job in clarifying its relationship with prior work. 


Post-rebuttal:
- I thank the authors for improving the presentation of the paper and including additional experiments comparing to latent ODE. 
review_title: Official Blind Review #1

Meta-review: This paper proposes a refinement, and analysis of, continuous-time inference schemes.

This paper got in-depth criticism from some very thoughtful and expert reviewers, and the authors seem to have taken it to heart.  I'm still worried about the similarity to GRU-ODE-Bayes, but I feel that the clarifications to the general theory of continuous-time belief updates is a worthy contribution, and the method proposed is a practical one.  One reviewer didn't update their score, but the other reviewers put a lot of thought into the discussion and also raised their scores.

I do think the title and name of the method is a bit misleading - I would call it something like "Consistent continuous-time filtering", because the jump ODE is really describing beliefs about an SDE.
Decision: Accept (Poster)