Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper has been well summarized by the authors in their abstract. The following is taken directly from there:

Learning ordinary differential equations (ODEs), where a flexible function approximator (often a neural network) is used to estimate the system dynamics, given as a time derivative has become popular after the publication of the paper by Chen et. al. at NeurIPS 2018.. However, these integrators can be unsatisfactorily slow and unstable when learning systems of ODEs from long sequences. In this work, the authors propose to learn an ODE of interest from data by viewing its dynamics as a vector field related to another base vector field via a diffeomorphism. By learning both the diffeomorphism and the dynamics of the base ODE, the authors provide an approach to offload some of the complexity in modelling the dynamics directly on to learning the diffeomorphism. Consequently, by restricting the base ODE to be amenable to integration, we can speed up and improve the robustness of integrating trajectories from the learned system.
main_review: The strengths of the paper are:
- it follows a series of work in this direction of ODE estimation, which has been validated as an important direction for ML
- it is mathematically sound
- well written

The weaknesses are:
- from the limited experiments presented it is not clear how applicable the method is for complex practical problems
- existence of an diffeomorphism is a critical aspect of the paper, it is not clear if such a tractable mapping that can be estimated using using a NN can really exist
- related to the above, the choice of the base ODE is another aspect that might be non-trivial
summary_of_the_review: Clearly, the paper adds to the existing literature and is interesting. It is well written and mathematically sound.  It is however, unclear how general the approach is and that brings into question the significance of the contribution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: ODEs parameterized by neural networks can be slow when learning systems from long sequences. This paper proposes learning an ODE with a fast diffeomorphism, so that the ODE can be a simpler function that’s more efficient to numerically integrate, while preserving expressivity. 
main_review: I don’t agree with the dichotomy presented in paragraph 1 of the related work: 

“Our proposed approach improves the learning of the underlying ODE, and is compatible with models that incorporate learnable ODEs. We note that the term “neural ODE” has typically been used in the literature to refer to neural networks that incorporate ODEs, including the original work in Chen et al. (2018). However, “neural ODE” has occasionally been used to refer to an ODE with dynamics parameterized by a neural network (Norcliffe et al., 2021). We follow the former convention.”

My understanding is “Neural ODE” refers to ODEs parameterized by neural networks. This includes the case where such an ODE is incorporated into a deep learning model. See the second paragraph of [1]

Related work: I think [1], [2], [3], [4] should be included in the related work section, as they all propose methods to improve the efficiency of neural ODEs. [4] is very recent, so maybe it’s not necessary to cite it, but it seems very related to this paper.

Can you more thoroughly explain the benefits of asymptotically stable Neural ODEs? How do these models maintain expressivity while being asymptotically stable? Are there example of Neural ODEs without diffeomorphisms that can be empirically demonstrated to be asymptotically unstable?

It doesn’t seem that this definition of asymptotic stability in Definition 4.3 permits periodic solutions, yet there are experiments with these dynamics. Can you explain this?

A principal advantage of composing a diffeomorphism with an ODE, as outlined in Section 4.3, rather than an arbitrary function, is that solution curves can be obtained without evaluating the jacobian of the diffeomorphism. I’m not sure why this is a big advantage. 
- Multiplying by the Jacobian of the diffeomorphism is relatively cheap (2x more expensive than evaluating F) via vector-jacobian products. Do quantities besides a vector product need to be computed (e.g. the trace of the jacobian?)
- If we replaced F with some non-invertible neural network, we would also not require evaluating its jacobian to compute predictions from the model. We would need two separate neural networks (for pre- and post-processing, respectively) to match the expressivity, but otherwise I don’t see why we need to maintain the invertibility beyond the fact that we’d like the entire function to be an ODE.

How do we make use of the fact that the entire model is one well-defined vector field / ODE? The principal advantage seems to be for some time series problems, where the whole function is now an ODE.

Why is there a requirement that the model be entirely composed of one ODE? Many neural ODE models incorporate preprocessing steps, and the pre- and post- processing steps are not inverses of each other (e.g. one might be downsampling). Training continuous normalizing flows might be one application where a diffeomorphism is required, and experiments on this task would be interesting.

It isn’t well-characterized when it is suitable to use a Linear Base ODE vs. a neural network ODE. When is the linear base ODE expressive enough? Are there tractable components that can be composed with sufficient expressivity, instead of having to resort to a neural network ODE?

Are there more fine-grained measures of time-cost of different models? For example, function evaluations from the numerical integrator can be measured as a more robust measure than the total seconds elapsed. Does Table 2 include results from models in Rubanova et al. 2019? Or is it just Linear ODE models integrated with different solvers?

The experiments are not extensive enough. Perhaps it might be worth trying some of the tasks investigated in [4]?

Typos:
Section 2:
- “normlizing flows” (second paragraph)
Section 4.1: 
- “If mapping F” (second paragraph)
Section 4.2:
- “and get the inverse with no further labor” (first paragraph)

[1]: “Learning Differential Equations that are Easy to Solve”, Kelly et al. (arxiv.org/abs/2007.04504)
[2]: “How to train your Neural ODE”, Finlay et al. (​​arxiv.org/abs/2002.02798)
[3]: “Opening the Black Box: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics”, Pal et al. (arxiv.org/abs/2105.03918)
[4]: “Neural Flows: Efficient Alternative to Neural ODEs”, Biloš et al. (arxiv.org/abs/2110.13040)

summary_of_the_review: The paper does not explain clearly enough the benefits of the proposed approach over regular Neural ODE models, nor thoroughly validate their claims empirically.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an interesting perspective that connects ordinary differential equations with diffeomorphisms. More specifically, possibly complicated and thus difficult-to-learn ODE dynamics are expressed as a morphed version of a simple "base" ODE system. The authors propose to learn the base dynamics alongside with the corresponding diffeomorphism. As experimentally shown, this approach leads to faster training, somewhat increases accuracy and allows for the long-term behaviour analysis of certain types of base ODE systems.
main_review: The paper addresses a very timely topic and shows a very interesting connection between differential geometry and ODE systems. I find this theoretical contribution (or rather perspective) highly significant. Considering the convincing experimental results, I believe the paper should be accepted. For the first round of reviewing, I'm giving a score of 6 but would happily increase it if my following concerns are addressed:

- The connection with the latent neural ODEs is missing. One could even present the proposed methodology as a special case of latent neural ODEs; therefore, showing and justifying the differences are important. Similarly, I would like to see empirical comparisons with latent neural ODEs to be written more clearly. As such, all tables are presented as if the competing method is the same across tables.
- I'm not sure if the informal definition of "stiff ODEs" in Sec 4.6. is correct. No my knowledge, a stiff ODE system does not necessarily rapidly vary along a dimension but rather causes numerical solvers to be stuck at certain points in the space.
- Are the number of model parameters comparable? Since the presented approach utilizes two networks, I believe the comparison should be done fairly. This is questionable looking at Sec.D. Similarly, it is questionable that the adaptive step solvers are used with correct tolerance values.
- What are the initial values for integration? This is mentioned nowhere in the main paper. Also, the effect of the noise on the initial values should be discussed. 
- I wonder how much of the "load" is on the shoulders of the base ODE system. This might not be of vital importance but "disentangling" the contributions of the invertible layer and the ODE system might be beneficial.

Minor comments: 
- The paper is written clearly. I particularly enjoyed the preliminaries section and how the concepts are explained in sufficient detail. That being said, the writing can be improved:
  - Section 4.3 is a little repetitive. In particular, the third and fourth paragraph partially repeat the information above.
  - To my knowledge, he phrase "integral curve" is not commonly used in neural ODE community. Giving its formal definition or synonyms would help the reader.
  - Section 4.4 is highly repetitive (can be even completely removed). Also, some title along the line "summary of our methodology" could be better since the word "result" typically refers to experimental findings.
  - The last sentence above Def. 4.2 reads strange. I would suggest authors revisit this paragraph for coherence.
- Figure 3 seems extremely crowded (especially the right one). 
- The long sentences (especially in the table captions and the appendix) should be revised.
- The last paragraph in page 8 is extremely long and requires a better structure.
- It is a little difficult to see how significant the differences in Table1 are. It would be nice to see a visualization of forward trajectories over time.
summary_of_the_review: Since the paper shows interesting links between seemingly unrelated topics, I find the contribution significant and suggest an accept although the writing can be improved and the model can be investigated in more detail.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In order to better learn differential equations that produce discrete data, this paper proposes to learn a coordinate transform (i.e. the `diffeomorphism’ in the title) in addition to learning the vector field. It is argued that, after this coordinate transformation, one can either learn a linear ODE or learn a neural network parameterized ODE, and obtain improved efficiency and robustness.
main_review: Learning an extra coordinate transformation is a very interesting idea and shared by important recent works. It is also closely related to encoder and decoder (although this is not discussed). I feel the authors have a great idea that is definitely worth pursuing, although the paper in its current form could use some significant improvements.

First of all, putting linear and NODE-base-vector-field together appears a little unnatural to me. Is a linear base recommended or not? I would appreciate it more if they could be argued to be complementary, and a criterion is given as such when to use which. Here are some of my understandings of their difference in scopes and applicability:

Let me first discuss the linear base. In this case, despite of the extra differomorphism, the topology of the learned dynamics is still going to be linear. This is very restricted. Topological conjugacy can change the quantitative appearance of dynamics but not its structures, such as the topology of limit sets. All examples in 5.1 reflect this fact. Can the author use linear base + diffeomorphism to capture both the libration and circulation regions of a pendulum, as well as the separatrix inbetween? If not, I think it is important not to exaggerate the representation power of linear bases.

An addition comment on the linear base case is, the stability criterion for autonomous linear systems is well-known, and stability is nontrivial only for truly nonlinear problems. I would rather see more details of the core treatments and discussions, instead of loose parts that try to make the paper appear more mathematical (e.g., Definition 4.2, 4.3, Theorem 4.1, and all the parts involving manifolds, e.g., 3.2, Figure 2, Definition 4.1, Proposition 4.1, as the work doesn’t really have a manifold component).

The second approach, `non-linear neural network base’, should work for general problems though. This base is not new, and the contribution is to add a coordinate transform layer. For this part, it would be great if two matters are better discussed: (1) uniqueness. Now both the differmorphism and the base dynamics are learned, and given one differmorphism there is a base dynamics. Which one will be chosen? There will be implicit bias imposed by both the architecture and the training methodology. I think theoretically understanding this would be challenging, but at least some empirical investigation would be helpful. (2) Why is this better than just learning the vector field? The authors did mention on page 5 that this is computationally more advantageous, but I’m sorry that I’m not very convinced. 
In addition, I’d like to better understand how the new method scales with the dimension. As this paper proposes an improvement of NODE, comparisons with NODE-based approaches on machine learning baselines would be very helpful.

Additional points:
* It seems the Lorenz example in Fig.5 is not for a chaotic trajectory, so it diffuses the point of testing on a Lorenz system.
* Some important NODE references are given, but the expansive data-driven learning differential equation literature is almost completely missing.
* Eq.(2): $ODESolve(f_\omega, y_{t_0}, t_e)$ should be $ODESolve(f_\omega, y_{t_0}, t_0, t_e)$ or $ODESolve(f_\omega, y_{t_0}, t_e-t_0)$.
* Right before eq.(2): Euler’s method is a Runge-Kutta method.

summary_of_the_review: The "linear base" part and the "non-linear neural network base" part are disjoint. Unlike the current wordings imply, I think the former is rather restrictive. On the other hand, the latter is interesting but needs more work. Although I cannot recommend the acceptance of this paper in its current form, I encourage the authors to strengthen the latter part and consult more with the literature.
review_title: null

Meta-review: This submission proposes a new manner to learn ordinary differential equations, aiming to improve their efficiency. While judging it interesting, the reviewers are quite split on this work. Overall there was no strong consensus to accept, nor anyone willing to champion this work.

The main stated weaknesses are

- The reliance on the existence of a diffeomorphism (and its choice in the method)
- The choice of the base and its expressiveness
- A somewhat limited experimental section, not indicating strongly how amenable this would be to more complex problems.
Decision: Reject