Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This is a very nice paper presenting a novel neural ODE with momentum. The idea behind the paper is solid, and the contribution is motivated by some known limitations of Neural ODEs. The authors not only provide us with compelling experimental results but also with a theoretical discussion and a lot of insights. I think the paper would benefit a lot the community.
main_review: This paper is based on a simple idea: adding momentum often accelerates convergence. The authors apply this idea to neural ODEs and develop a solid theory around it: momentum is shown to help both theoretically (adjoint discussion) and experimentally. 

I never published in neural ODEs – so I am not an expert – but I feel confident in recommending acceptance. The paper is totally in line with the literature (also experiment-wise) and is very well written and organized. I also did not spot any typo.

I have 2 questions for the authors

1) is it possible to translate this idea back to neural networks? i.e., what does a corresponding Resnet-like architecture look like?

2) Have you thought about applying this idea to Hamiltonian neural ODEs? Similar to the paper:

Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control
Yaofeng Desmond Zhong, Biswadip Dey, Amit Chakraborty
limitations_and_societal_impact: No concerning limitations, no foreseeable societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a reparameterization of a Neural ODE (NODE) as a Heavy-Ball Neural ODE (HBNODE). This is inspired by Heavy-Ball Momentum, which is the continuous limit of classical momentum.
The authors make the following claims:
The adjoint of an HBNODE is also an HBNODE.
HBNODE reduces forward and backward NFE. This is even more apparent at lower tolerances.
The spectrum of HBNODE is well-structured. This implies effective learning of long-range dependencies by avoiding vanishing gradients.
HBNODEs are more accurate
The authors verify these claims on image classification, complex dynamics, and sequential modelling. 

main_review: Describe the strengths of the work:

This work clearly describes Heavy-Ball Momentum and the definition of HBNODE. The authors clearly state the use of augmented the state of a neural ODE, where the states are coupled, to solve the second-order heavy-ball ODE. The authors then generalize this by adding some tuneable/learnable weights for the terms. They claim this reduces the tendency for the hidden state to blow-up during training. 
Backward NFEs are significantly reduced and this results in a reduction in training time too. 
The authors demonstrate their approach on a wide variety of tasks. 

Explain the limitations of this work:

It’s unclear why the results are so much weaker on the irregularly-sampled time-series experiment. 
There is little explanation given for why the forward NFEs are larger for the experiment in Section 5.3 compared to the baseline, but smaller for the backward NFEs.
Are there reasons other than vanishing gradients that cause GHBNODES to perform better than other models? Does their parameterization simply make them more expressive for the given number of parameters?

Correctness:

The authors make an effort to provide a fair comparison to different baseline methods by matching the number of parameters.
The experiments are not repeated with different random seeds and averaged. Since some methods/baselines were sensitive to initialization, it could be helpful to see how strong these results are over several runs (e.g. varying random seed). 
It’s not very clear from Figure 7 that (G)HBNODE has significant savings in NFE at loose tolerances.

Clarity:

The writing in the paper can be improved. In particular, Section 4 can be improved for clarity, and can be better connected to the rest of the text. If space is a concern, I think some of the math can be moved to the appendix. 
Typos:
Line 272: fed through

Relation to prior work: 

It is clear how this work differs from prior work.
A few suggestions on work related to Reducing NFEs for NODEs:

Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
Avik Pal, Yingbo Ma, Viral Shah, Christopher Rackauckas

Learning Differential Equations that are Easy to Solve
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, David Duvenaud

Reproducibility: 

Hyperparameters are given for the experiments. Given that examples are not averaged over multiple runs and it’s unclear exactly how the models were tuned for performance, there may be some issues with reproducing the results precisely. 

Additional feedback, comments, suggestions:

In Figure 3 it is unclear if all the models are achieving the same performance at the task. It should be made clear what the performance is of each of the methods and if it is related to the norm of the hidden state. For example, do models with larger hidden states diverge faster?

Section 4 deals with the issue of vanishing gradients. There are no experiments which verify that vanishing gradients cause the inability to avoid vanishing gradients. The authors perform an analysis to justify regularizing the norm of the adjoint state as a method to mitigate vanishing gradients. The authors suggest clipping the norm of the adjoint state. They then have an experiment showing the GHBNODE models don’t have vanishing gradients. I think the experiment could be improved by explicitly measuring some of the quantities mentioned in the analysis in this section and seeing how they related to the norm of the adjoint state. Even a toy problem to make things very tractable could give useful insight. 

limitations_and_societal_impact:  The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new neural ordinary differential equation (NODE) model that is inspired by a second order ODE $\ddot{x} + a \dot{x} + b \nabla F(x)=0$. The discretization of this ODE leads to a two step procedure known as heavy ball method. Using this idea to model a NODE has the advantage that the corresponding adjoint equation takes a form that reduces the number of function evaluations during the forward and backward pass. Further, the authors show that the spectrum of the model is well-structured which in turn mitigates the vanishing gradient problem. In addition, they formulate a model that integrates skip connections and gating mechanisms. The empirical results demonstrate the favorable performance of the proposed models as compared to other NODEs.
main_review: The originality of this work is limited, since the idea of second order neural ODEs is not new. Also, I do not agree that the work of [45] and others is orthogonal to your work (maybe you can elaborate on this). Further, you miss to discuss the work by [*1] who propose a heavy ball inspired neural network architecture. This said, it would be fair to devote more room for discussing related work within the introduction and related work section, instead of just briefly mention some of the important related works somewhere at the end of the paper.

The main contribution of this paper is the analysis of the corresponding adjoint equation for this model, and to my best knowledge these results appear to be novel. The advantage of the adjoint equation is plausible and backed up by empirical results, which show a considerable advantage as compared to other neural ODEs. Hence, I consider the results as significant and relevant for being published. 

The generalized heavy ball neural ODE looks a bit heuristic and the advantage is not exactly clear to me. The experiments do not show necessarily an advantage in terms of accuracy. Based on Figure 1, I would favor the simpler heavy ball model. Does the results in Sec. 4 apply to the generalized model?

Overall, the paper appears to be very dense and thus a bit cumbersome to read. I suggest, that you move some of the details, e.g., Table 1 and 2 into the appendix. Also you probably can omit section 1.2. Personally, I would also move the discussion of the generalized heavy ball neural ODE into the appendix. In turn, this would give room to expend the introduction, better discuss related work, and provide more room for discussions and explanations.


Other comments:

* The statement that the exploding gradients issue can be effectively resolved via gradient clipping is a bit misleading. While it is true that the issue of exploding gradients can be resolved via clipping, this comes with the price of restricted expressivity. Hence, there is a lot of current research effort focusing on architectures that avoid vanishing and exploding gradients by design [*2,*3,*4]. Since, [*3,*4] are CT-RNNs, they can also handle irregular sampled time series data. I don't think that it is necessary to compare to these models, but I would suggest to slightly rephrase the discussion at beginning of Section 4.

* I am slightly surprised about the results shown in Figure 5. The augmented ODE should be able to perfectly separate the two point clouds. Are the plots here from an early stage of training or is there another reason for the poor performance of the NODE, here? 

* The considered ODE models do not achieve state-of-the-art performance on CIFAR-10. For instance, ANODE-v2 [*5] and Statefull-ODEs [*6] achieve about 90% test accuracy on CIFAR-10, while having nearly the same parameter count as the models considered in the experiments. Is there any reason for why you consider only a very simple network architecture?


[*1] Li, Huan, et al. "Optimization algorithm inspired deep neural network structure design." Asian Conference on Machine Learning. PMLR, 2018.

[*2] Lezcano-Casado, Mario, and David Martınez-Rubio. "Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group." ICLR (2019).

[*3] Rusch, T. Konstantin, and Siddhartha Mishra. "Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies." ICLR (2021).

[*4] Erichson, N. Benjamin, et al. "Lipschitz Recurrent Neural Networks." ICLR (2021).

[*5] Zhang, Tianjun, et al. "Anodev2: A coupled neural ode evolution framework." NeurIPS (2019).

[*6] Queiruga, Alejandro, et al. "Compressing Deep ODE-Nets using Basis Function Expansions." arXiv preprint arXiv:2106.10820 (2021).
limitations_and_societal_impact: A detailed discussion of the limitation of the proposed models is missing (there is no Section 4.1), also there is no discussion of the potential negative societal impact. I am not qualified to judge whether there is any potential negative societal impact of this work, but since this is fundamental research, I assume that there is non foreseeable negative impact.
review_title: null

Meta-review: The paper proposes a reparameterization of a Neural ODE (NODE) as a Heavy-Ball Neural ODE (HBNODE) in order to handle the notoriously difficult problem of long-range dependency modeling. Proposed approach is elegant and novel. The authors provided also very comprehensive empirical evaluation showcasing the effectiveness of their method and addressed *in detail* all reviewers' comments.
Decision: Accept (Poster)