Venue: NeurIPS 2021 Workshop DLDE

Reviews:

final_score: rating: 3: Good paper
review_content: The authors present an approach to learning dynamic models from noisy measurements by combining deep learning with a classical numerical integration method. The method consists of two networks; one that implicitly represents given measurement data and the second one approximates the vector field; and a loss function that uses Runge-Kutta Constraint and forces the networks to achieve their goal.


The idea of combining the two approaches is interesting and could open the door to further work, but in my opinion the method can be better justified. In particular, the experiments presented in the work describe how the method deals with different problems, but no comparison is made with the classical methods in terms of computational cost or quality of the solution. Even comparing success (error versus effort) within the components of the method can give intuition to its justification (e.g., regularization coefficients $\lambda_{\sf RK}, \lambda_{ \sf Grad}$, CNN vs. RNN, network size etc.).
review_title: Review

final_score: rating: 3: Good paper
review_content: The paper proposes an approach to learning dynamics in a noisy input setting. The method learns a neural representation of the noisy data to use in a neural net which learns the dynamics of the given system. Both these neural nets are 'connected' using a numerical integration scheme (here RK4).

**Comments**

The overall paper presentation is good and easy to understand. The approach to using 2 neural nets for implicitly denoising noisy data and using it to learn dynamics is interesting. 

**Issues**

The initial discussion about measurement noise could be more concise.  
Diagrams in Fig 1 and plots in Fig 2 are quite small, which makes interpreting them difficult. 

review_title: Review

final_score: rating: 2: Borderline paper
review_content: ## Summary

This paper combines neural ODEs with physics-informed neural networks. The former parameterises the vector field of an ODE by a neural network; the latter approximates the solution of an ODE by a neural network.

Supposedly this is supposed help model dynamics for which only poor-quality (noisy) data have been observed, although no justification is offered for this claim.

## Problems

The advantage of combing these two approaches is simply not evidenced. Why does this approach yield robustness to noise? How do the results obtained differ -- practically speaking -- from those obtained by simply approximating the vector field with a neural network and training a neural ODE as usual? This is a central weakness of the paper.

The text repeats itself quite a lot -- the wording could do with being tightened substantially -- e.g. repeated discussion about measurement noise.

The term "implicit representation" is quite suggestive of either deep implicit models (https://implicit-layers-tutorial.org/) or implicit representations (https://arxiv.org/abs/2006.09661). The connections to either don't seem to be that strong, so I'm not a fan of using this terminology here.

Figure 1 is too dense/small to be read clearly.

The $\lambda_{RK}$ loss in Figure seems to be slightly misrepresented: a single application of just $\mathcal{N}^{Dyn}$ is used rather than the RK scheme.

The Runge--Kutta scheme used is one of two known as "RK4". The other, sometimes termed "RK4 with 3/8 rule", is generally more popular as achieving slightly better results. (And even that is generally not favoured -- the current "best" mid-order non-stiff RK scheme is generally held to be "tsit5" introduced by https://www.sciencedirect.com/science/article/pii/S0898122111004706.)

## Limitations

It would seem that the proposed scheme assumes that the dynamics follow a Markov property; equivalently that $x$ is fully observed. This seems like a limitation as in general only some channels might be observed.

(For this reason neural differential equations often consider the state $x$ as evolving in some latent space; a linear map is then used to transform $x$ into data space. This is precisely analogous to the typical way in which an RNN is used.)

I suspect in this context that this this limitation can be overcome. However it is common to see authors implicitly assume such Markov behaviour, without any comment, despite it really restricting the kind of behaviour that can be modelled.
review_title: Review

Meta-review: Reviewers have recommended the acceptance of this article.
Decision: Accept (Poster)