Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes the idea of directly modelling the solution curves for an ODE using neural networks as opposed to using numerical ODE solver.
main_review: **After rebuttal: Score updated to reflect that the reviewer recommends acceptance**


*Pros:*

- a reasonable idea, straightforward modification of the original Neural ODE; gives evidence of advantages of directly approximating the solution of the ODE

*Cons:*

[1] the novelty of the idea of modelling the solution using neural networks as opposed to the dynamics function f should be clarified (see previous work such as Smaoui &Al-Enezi, 2004; Kumar &Yadav, 2011 and Dua (2011)); 

[2] it is hard to understand why the paper on a new method needs such an extensive applications section, describing different well-known problem statements such as filtering or smoothing, it might be a good idea to keep focused to the original message that modelling the solution as opposed to the dynamics function is beneficial (see point [2] of the decision justification)

[3] Related to the previous points, the storytelling should also be improved with expanding on the state-of-the-art and contribution discussions (see point [1] in the decision justification below)
[4] multiple statements are not clear (see comments below).

The following points stand between the reviewer and the paper acceptance (*decision justification*):

[1] State-of-the-art description may not be sufficient according to the reviewer: it would be good to contrast the novelty in this paper against the state of the art as listed above; the related work description for this paper should tell about the previous approaches using neural network; the novelty against the state of the art needs to be emphasised

[2]The reviewer thinks that the authors should stay focused in the main text on the paper idea, i.e. improving the efficiency of neural ODE and addressing the limitations of using numerical solvers; this work goes into extensive details on filtering and smoothing problem statements (section 3.1) which may look unnecessary long; the details of the encoder-decoder (section 3.1), which look similar to Neural ODE architecture, could be omitted with the reference made to the original Neural ODE paper, or moved to the appendix.  

[3]Further comments that are given below on clarity and reproducibility (esp. comments [3],[5], [6], [7] below).  

[4] It is not clear when in the experimental section (section 4) the Euler solver is used, what is its step size, and what is the reason behind comparing with the Euler solver when the original Neural ODE paper uses Dormand-Prince solver (dopri5); the provided code could partially help with it but it does not answer the question how the actual experiments in the paper have been done. 

*Comments:*

[1]Figure 1: while it is understandable what the authors are saying in this image, not sure that this figure is actually conveying the idea that the approach eliminates the need for the numerical solvers. E.g. it would be good to state exactly what is the meaning of the arrows on the left-hand size image (I guess the iterations of the numerical solver?). Comparing it with the original Neural ODE paper (Chen et al, 2018, Figure 1), I can see that Chen et al use the arrows to denote the vector field direction (which is apparently different from the meaning of arrows in Figure 1 of this paper), and they state it explicitly. They also state that ‘circles represent evaluation locations’; similar clarifications should be made for Figure 1.

[2](As it is the appendix, it did not influence the score)  As a suggestion, formatting in section E (Hyperparameters) could be improved by placing these hyperparameters in a table.

[3] Ln 119: "Historically, the Cauchy-Kovalevskaya theorem [13, p. 39] tells us that, if the derivative f is an analytic function, then the solution curve F is analytic too. This hints that as long as we have expressive enough networks, we can approximate any ODE.” Not sure the reviewer sees the proposed connection between expressivity of the networks and the dynamics function being analytic. This is because the Universal approximation theorem (Cybenko, 1989) is stated for continuous functions and not analytic, and therefore the connection to the Cauchy-Kovalevskaya theorem is not clear to the reviewer. Therefore, the justfication could be stronger.

[4]Ln 103: "The disadvantage of both ResNet flow and GRU flow is the missing analytical inverse.” While it is clear that the invertibility would ensure the uniqueness of the solution, it is unclear why the analytical inverse is necessary for the proposed method. 

[5]   Ln. 234: "Again, if we consider a linear ODE, we can easily show that calculating the trace and calculating the determinant of the corresponding flow is equivalent.” How exactly could it be shown? Is it possible to for example, give a proof in the appendix?

[6] Ln 79: "Although plain ResNets are not invertible, one could use spectral normalization [25]  to enforce a small Lipschitz constant of the network, which guarantees invertibility [65, Theorem 9.23]” Theorem 9.23 in [65] discusses the existence of the fixed point in the contraction of X into X, and it is not clear from this description why it would imply invertibility. The reviewer recommends referring to the justification in [2], which actually describes the construction of Lipschitz constants.

[7] Figure 2: it is unclear what experiment this figure is based upon (as the authors use three datasets in this section, MuJoCo, Physionet and Activity). Furthermore, as I can see in the description for Figure 2, Neural ODE is run with an Euler method, while the original Neural ODE paper reports results with dopri5 solver, why was it done this way?  

Smaoui, N., & Al-Enezi, S. (2004). Modelling the dynamics of nonlinear partial differential equations using neural networks. Journal of Computational and Applied Mathematics, 170(1), 27-58.

Kumar, M. and Yadav, N., 2011. Multilayer perceptrons and radial basis function neural network methods for the solution of differential equations: a survey. Computers & Mathematics with Applications, 62(10), pp.3796-3811.

Dua, V. (2011). An artificial neural network approximation based decomposition approach for parameter estimation of system of ordinary differential equations. Computers & chemical engineering, 35(3), 545-553.

Cybenko, G. (1989). "Approximation by superpositions of a sigmoidal function". Mathematics of Control, Signals, and Systems.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a natural evolution of the Neural ODE framework where, instead of learning the function representing the derivative driving the evolution of the studied process, the authors directly parametrize the flow of the ODE. 
main_review: The main idea of the paper is quite straightforward and elegant and so are the proposed architectures for the flow. Its strengths lie in the simplicity of implementation of the proposed and in the fact that this framework can easily be expanded into other models and different parameterization.

It is worth noting, and this is mentioned in the paper, that some choices of the NODE solver can yield networks which are more or less identical to some of the flows described here (eg. an explicit Euler disrectization would yield the ResNet flow). 

However there are still some aspects which are not convincing in my view:
- NODE is not really a well-defined model in the sense that it encompasses many instanciations which can behave quite differently depending for example on the choice of the numerical solver. Thus, it seems to me that for some processes which need a numerical scheme more involved than an explicit scheme to be accurately modelled, directly modelling the flow isn't as easy as is said in 2.2. Examples on stiff differential equations for example would be welcome to explore this aspect. 
- Not unrelated to the point above, it would be interesting to include experiments where the learned equations model challenging physical phenomena. For example, reaction-diffusion phenomena or examples from fluid dynamics could allow a meaningful and informative comparison between different instanciations of NODE and the flow models.
- In this last context, the flow models should be also compared to the slightly different approach brought by the Neural Fourier Operator paper (Li et al., ICLR 2020). I find it surprising that this paper is not cited as it proposes an approach which is quite similar in its motivation and philosophy.
- The improvements brought by this new point of view are in some cases somewhat marginal in terms of accuracy and it would also be interesting to compare the stability / easiness of training neural flows with NODE models.

Overall, I think this paper has many merits and I wouldn't oppose it being accepted at the conference. However, I would advocate for it more strongly if some experiments were added, for example along what is suggested above.

Edit after rebuttal: I thank the authors for their comprehensive answer to my review. I am not fully convinced by the answer regarding Neural Fourier operators: While I agree that the two settings differ in generality, Neural Flows should be usable in the setting of (Li et al.) and this would give an interesting comparison to the reader. However, I think this paper brings enough new ideas and experiments to be accepted at the conference and will update my score accordingly. 
limitations_and_societal_impact: The limitations were clearly addressed in the original submission.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work proposes directly modelling the solution trajectories of an ODE, circumventing the need for costly numerical ODE solvers. Specifically, they model the function which returns a solution to the initial value problem, and derive conditions this function must satisfy in order to implicitly induce a valid ODE.

Crucially, the authors explain that Neural ODE models are not very interpretable, unlike more traditional applications of ODEs. Since we aren’t able to use the dynamics for interpretability, it makes sense to not modely them explicitly, and instead model its integral (via IVP) directly. (This is more my interpretation than what the authors stated explicitly.)

The authors propose several architectures, including ResNet flow and GRU flow. The issue with these models is that these models can only be inverted approximately and require expensive iterative schemes. A third approach is to design architectures inspired by normalizing flows, which trades off some expressivity for greater efficiency. The authors also consider the approximation capabilities of these models.

Section 3 introduces specific models for time-series and density estimation, and Section 4 compares these models to baselines on the relevant tasks.

main_review: Describe the strengths of the work:

The authors carefully consider the benefits and drawbacks of Neural ODEs, and propose an original method that greatly increases the speed of these models while not harmings its expressivity, and retaining many of the benefits of Neural ODEs, namely naturally handling continuous time and irregular time series.

Neural Flows are significantly faster than Neural ODEs and match performance across a number of tasks.

The authors demonstrate the superiority of their approach on a wide array of tasks and models.

Explain the limitations of this work:

It might be interesting to see image density estimation tasks (e.g. comparing to FFJORD). On the other hand, the chosen time-dependent density estimation task is one that Neural ODE/Flow solve much better than other competing approaches.

Correctness: 
The authors incorporate recent techniques to improve the speed of Neural ODEs and ensure they have strong baselines when comparing to their method, both in terms of speed and performance. 
They report results with confidence intervals over multiple runs, and tuned baseline models. They provide thorough details on all experiments.

Clarity:

Very excellent writing throughout the paper and the appendix.

Typos:
Line 686: parameterization

Relation to prior work: 

This work thoroughly and clearly explains how it builds/is related to previous work.

Reproducibility: 

The appendix has thorough details on data processing and hyperparameters for all models and datasets.
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The derivative function of an ODE defines a vector field over the input space (augmented with time). Neural ODEs learn such vector fields when solving initial value problems. The authors propose a method that directly learns the flow of such a vector field, by learning a function that respects the constraints of a proper flow. This method avoids the need to perform a costly integration operation like in NODEs while also performing similarly or better.  
main_review: ## Strong points 

- The paper attacks a very important problem of Neural ODEs: they are often slow, especially if the time series is very long. The paper proposes a simple but clever solution to avoid the need to integrate for potentially very long times. 
- The authors show how to derive a valid flow counterpart to the GRU-ODE model, which has been used extensively in the literature. This model could have a significant impact in practice for the applications of interest. 
- The results presented in the paper are very extensive and look at a diverse set of tasks. The authors present convincing results that the method is indeed faster (as expected) and in certain settings performs better. 
- The paper is clearly written and easy to follow. The models are described in sufficient detail and the authors do a good job reviewing the Neural ODE counterpart models they improve upon like CNFs, Latent NODEs etc. 
- The section on limitations is nice and offers a fair description. 

## Improvements and Questions
- The section on approximation capabilities is a bit hand-wavy and does not present any concrete theoretical results. I think the authors could be more upfront about this and include it as a direction for future work. The authors also mention analytic functions, however, this is a relatively constrained class of functions. It might be more natural to discuss $C^{\infty}$ functions or $C^{k}$ functions for some finite $k$. Note that not all $C^{\infty}$ functions are analytical. Regarding the limitations of Neural ODEs in learning any homomorphism (e.g. $x \mapsto -x $), these have also been studied in *On Second Order Behaviour in Augmented Neural ODEs* (https://papers.nips.cc/paper/2020/hash/418db2ea5d227a9ea8db8e5357ca2084-Abstract.html). Additionally, how do the authors see their method being extended to applications involving higher-order dynamics? 
- Some parallels with Neural ODEs could definitely use a computational complexity analysis to ground the discussion and make it more precise. For instance, in Section 3.3 the authors claim that Equation (10) becomes "intractable" as the dimension increases. This is slightly exaggerated since the complexity of the trace operation is $\mathcal{O}(D^2)$, where $D$ is the dimension of the space. In contrast, based on Equation (11), the complexity of the proposed method seems to be  $\mathcal{O}(D)$ since the determinant involves only the diagonal elements, which is indeed better. I believe that stating these complexities directly would clear up any confusion that might arise from describing this in words. 
- I am missing in the experimental evaluation an experiment involving stiff ODEs. To me, it seems an ideal setting for the proposed method since any ODE solver struggles with stiff ODEs and learning the solution directly should perform much better and also be significantly faster. 
limitations_and_societal_impact: The authors have addressed these in sufficient detail.
review_title: null

Meta-review: The paper proposes to directly model the solution of ODEs using time-conditional discrete flows, instead of via ODE integrators.
This has the potential to substantially accelerate applications of neural ODE flows. While the proposed model is rather straightforward, the extensive experiments highlight the nice properties of the approach.
Decision: Accept (Poster)