Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: NesterovNODE and GNesterovNODE are introduced, as an alternate DAE-based parameterisation of a neural ODE. They are explored primarily for their ability to improve accuracy and reduce computational cost.
strengths_and_weaknesses: **Strength: topic**

The topic at hand is a meaningful one, as good parameterisations of neural ODEs is still an underexplored problem.

**Strength: Nesterov/DAE approach**

I like the approach of using Nesterov's method as a starting point for choosing a good architecture. This seems well-motivated, and the constructions presented both plausible and novel. (To the best of my knowledge.)

In addition, I like the formulation of the problem as a DAE. To the best of my knowledge "neural DAEs" have received essentially no attention whatsoever, so this is a very interesting line of work to pursue.

**Strength: presentation**

The presentation is at all times both clear and technically precise. I greatly appreciated this aspect of the paper, which I enjoyed reading.

**Strength: experiments**

Several different kinds of experiments are done, and in each case a careful analysis is performed. These are generally well done.

**Weakness: handling blow-up**

The authors go to quite some effort mitigating the blow-up of the Nesterov equations. I feel like much simpler approaches are available -- for example, starting the integration from $t=1$ rather than $t=0$.

**Weakness: comparison to gradient descent**

Lines 81--88: I strongly object to the claim that neural ODEs "implicitly perform gradient descent". It is trivially true that both NODEs and GD are cases of generic ODEs; that is all that is going on here.

**Weakness: "adjoint sensitivity"**

The term "adjoint" has become heavily overloaded. There are multiple ways to backpropagate through a differential equation solve. In the current literature is is often used to mean specifically optimise-then-discretise, but in older literature it was often used to mean specifically discretise-then-optimise. Following [0, Remark 5.5] I recommend avoiding the term "adjoint" altogether.

In passing, since in this work backpropagation is performed via OtD, then the authors may find speedups by utilising the technique of [1].

**Weakness: numerical/software**

It is claimed that Dormand--Prince 5(4) was used to solve the DAE problems presented. However DP5(4) is an ODE solver, not a DAE solver. The problem can of course be rewritten in ODE form (e.g. equation 10) but this would no longer seem to follow the DAE presentation that is otherwise adhered to. How were the numerical methods actually implemented, i.e. what ODE was actually solved?

I note that for example [2] implements DAE solvers directly, which could have been an alternate approach.

In passing, note that DP5(4) is now generally reckoned as being slightly less efficient than the Tsitouras 5(4) method [3], which can be found implemented in both [2] and [4].

I do not see the choice of software listed anywhere. PyTorch with torchdiffeq, JAX with Diffrax, Julia with DifferentialEquations.jl, or something else?

Finally, I do not see any discussion on the choice of tolerance, step size controller, etc. (Was this DP5(4) with a fixed step size? If so, what step size? Was it with an adaptive step size controller? If so, which controller and what tolerances?)

**Weakness: experiments**

I would suggest removing the MNIST experiments. MNIST is linearly separable at (as I recall) 93% accuracy. It is not a meaningful benchmark.

Moreover, I do not think image classification is much of an interesting benchmark in the first place. I do not think anyone would claim that NDEs are the appropriate tool for performing image classification: standard ResNets/Transformers/etc. are much easier to use on this problem, for which the continuous-time formulation confers little advantage.

Following [0, Section 1.2.1 and Section 2.2.1] I would strongly suggest targeting other applications (time series, scientific problems, etc.) instead.

The time series experiments seem very strange to me. There is no discussion on how the time series is provided as an input to the ODE. There are numerous approaches here, mostly based around either:

1. As a controlled differential equation [5]
2. Discrete updates, e.g. an ODE-RNN [Ref. Rubanova et al. 2019 from this paper]

In either case the ODE formulation must be changed quite substantially. If working in continuous time (e.g. as in [5]) then there is an additional question of how the discrete data is converted (by interpolation or otherwise) into continuous time [6]. Consulting the appendix it looks like specifically an ODE-RNN approach is used? As there has been substantial work on this topic then much more detail deserves to be in the main paper.

**Weakness: various minor issues**

Figure 1: This relationship between GD, HB and NAG is well-known, and does not need to be here.

Remark 2: it is not necessary that specifically the Euler method be used. This remark should be reworded: "If the Euler method is used, then ... [the rest of the remark follows]".

GNesterovNODE: I'm not sure the term "generalised" is the appropriate one here -- this seems to have been modified, not generalised.

Long-term dependencies: I am used to think of "long term dependencies" as being in referenced to long time series, e.g. in the context of RNNs or neural CDEs. As the present paper concerns itself with non-time-series problems, I would suggesting focusing on the terminology of "vanishing gradients" etc. instead. (In terms of technical content, I did like this section of the paper.)

Rubanova et al. 2019 (the Latent ODE paper) appears in the reference list twice.

**Commentary**

I suspect there are natural extensions to this paper, studying neural DAEs more generally.

The fact that increasing $r$ increases both NFEs and model accuracy suggests to me that this may be due to discretisation error: greater NFEs produces a larger overall computation graph and thus increased model capacity. If the tolerances are shrunk small enough then I speculate that this improvement should vanish, as the model converges to the true solution of the ODE.

**References**

[0] On Neural Differential Equations, Kidger, Doctoral Thesis, University of Oxford 2021

[1] "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms, Kidger et al, ICML 2021

[2] DifferentialEquations.jl, https://diffeq.sciml.ai/

[3] Runge--Kutta pairs of order 5 (4) satisfying only the first column simplifying assumption, Tsitouras, Computers and Mathematics with Applications 2011

[4] Diffrax, https://github.com/patrick-kidger/diffrax

[5] Neural Controlled Differential Equations for Irregular Time Series, Kidger et al., NeurIPS 2020

[6] Neural Controlled Differential Equations for Online Prediction Tasks, Morrill et al., arXiv:2106.11028
questions: While the bulk of my review touches on several points, I would particularly highlight the following questions/suggestions:

1.  I quite like the trick of reparameterising from $h$ to $x$ in order to obtain a singularity at $t=0$. I am curious what motivated this approach in particular; how was this derived? And as above, I am curious why this approach is taken, instead of simply starting the integration from e.g. $t=1$ rather than $t=0$.
2. As above, how is the numerics/software was handled?
3. The experimental weaknesses offer substantial room for improvement.
limitations: There are no ethical concerns with this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents the novel class of neural ODEs called NesterovNODEs, where the ODE block solves the second-order ODE corresponding to the limit of the Nesterov accelerated gradient scheme. To avoid instability around $t=0$ in this second-order ODE, the authors propose the generalized version of NesterovNODEs, where the variable transformation eliminates this instability. The main benefit of this modification of the neural ODE approach is a significant reduction of the NFE needed to perform forward and backward passes. At the same time, the test quality of considered models is improved compared to other neural ODE models. This gain is shown point cloud separation, image classification, and sequence modeling problems.
strengths_and_weaknesses: Strengths
1) the presentation of the paper is clear and easy to follow
2) the extensive computational experiments confirm expectations from the presented theoretical part
3) the proposed method has theoretical explanations

Weaknesses
1) the used model in the image classification problem gives too low test accuracy. The papers 

a) Zhang, T., Yao, Z., Gholami, A., Gonzalez, J. E., Keutzer, K., Mahoney, M. W., & Biros, G. (2019). ANODEV2: A coupled neural ODE framework. Advances in Neural Information Processing Systems, 32.

b) Gusak, J., Markeeva, L., Daulbaev, T., Katrutsa, A., Cichocki, A., & Oseledets, I. (2020, April). Towards Understanding Normalization in Neural ODEs. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations.

c) Daulbaev, T., Katrutsa, A., Markeeva, L., Gusak, J., Cichocki, A., & Oseledets, I. (2020). Interpolation technique to speed up gradients 
propagation in neural ODEs. Advances in Neural Information Processing Systems, 33, 16689-16700.

show the +80% test accuracy in the CIFAR10 with standard neural ODE model without any second-order modifications. 
So the main concern is that the proposed approach is tested only on the continuous models that do not provide state-of-the-art accuracy for the considered problems.

2) the original motivation for NAG is to solve optimization problem faster and authors provide clear illustration of such convergence in Figure 1. At the same time, the motivation of continuous models like neural ODEs is to fit hidden dynamics in the latent space to learn relevant features. From these motivations is not clear why the ODE that is continuous limit of NAG is a good choice for the ODE that should learn hidden dynamics in continuous model.

3) Figure 3 contradicts the claim about absence of vanishing gradient in both NesterovNODE-RNN and GNesterovNODE-RNN since the plots for them demonstrate the different trends.    

4) From Table 3 in Appendix follows that for the different considered methods (NODE, ANODE,NODE, etc) different models are tested. If so, we can not split effect of different architectures and effect of different method.
questions: The following typos should be fixed in the revised version
1) lines 151,153 - bold notation for $m(t)$
2) line 151 - please provide explicit form for activation function $\sigma$. 
3) line 314 $1/k^2$ 

Also, please comment the following questions and remarks
1) It is known that adjoint method to compute the gradients through the solution of ODE is unstable, see e.g. 
`Gholami, A., Keutzer, K., & Biros, G. (2019). Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint arXiv:1902.10298` How did you control the potential instabilities in the adjoint method for NesterovODE? Authors addressed the vanishing gradient problem, but what about exploding gradient problem?
2) Please add more details about Figure 2: what step size and ODE solver is used for forward and backward passes?
3) The role of $T - t$ in line 170 should be explicitly explained for the reader convenience
4) Please specify the initial value in Eq 15 and 17. Otherwise, it is unclear where the integration is forward in time and where it is backward in time?  
5) Please add the main conclusion from these plots in caption of Fig 3
6) Please explicitly define $v$ as a row in Proposition 3
7) Please fix legend in Figure 9 
8) Please comment the difference between NesterovNode and GNesterovNode demonstrated in Figure 3. From this plot follows that HBNode is the best in terms of treating vanishing gradient problem. 
limitations: The missing limitations of this work is increasing dimension in three times (store $x(t), h(t)$ and $m(t)$) of the NesterovNODE compared with standard neural ODE architecture (only $x(t)$ is needed). Such increasing can make the proposed approach intractable for large dimension of hidden state, so discussion and probably experiments on the memory consumption should be added in the revised version.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper takes Nesterov's Accelerated Gradient Method, and applies it to Neural ODEs. One can think of a Neural ODE as carrying out vanilla gradient descent where:

$\frac{dh}{dt} = -\nabla_{h}F(h, t)$

Heavy ball Neural ODEs carry out the continuous limit of heavy ball momentum accelerated gradient descent where:

$\frac{d^2h}{dt^2} = -\gamma\frac{dh}{dt} - \nabla_{h}F(h, t)$

Finally, this work takes the continuous limit of Nestervov accelerated gradient descent:

$\frac{d^2h}{dt^2} = -\frac{3}{t}\frac{dh}{dt} - \nabla_{h}F(h, t)$

The work demonstrates that this style of second order differential equation converges to its equilibrium point significantly faster than heavy ball Neural ODEs and other variations of Neural ODE. In turn this requires far fewer function evaluations in the ODE solve, while maintaining performance.

Proofs are provided to show that the method will be resistant to vanishing gradients and the solution will require fewer function evaluations. Extensive experiments are done on a large collection of Neural ODE variants, demonstrating performance is just as good but the speed of the ODE solve is greater.
strengths_and_weaknesses: Strengths:

The paper is very well written and well motivated. The experiments test the claims well. The theoretical findings are strong and show Nesterov and GNesterov are useful to reduce the number of NFE. The ablations are also very good.

Weaknesses:

There aren't many weaknesses to this paper, it is overall quite simple which is a good thing, but overall means the contribution is incremental and not necessarily ground breaking. In my opinion this is not a bad thing. Specific weaknesses:

- The experiments are conducted on problems that Neural ODEs aren't typically used for, (classification and point clouds). It would be more convincing to demonstrate this on continuous normalizing flow tasks.
- The other main application of Neural ODEs is to time-series, and Nesterov does worse than SONODE, likely due to restricting the dynamics
- The proposed method feels quite simple (seems like a theoretically motivated adaptation of Heavy Ball NODE)

Minor Mistakes/points (do not affect quality of paper):

- I find Generalized Nesterov to be a misleading term, I believe something like Stabilized Nesterov is more accurate, as a sigmoid is applied to part of the Nesterov method, rather than generalizing, it is more like adapting
- On the MNIST task, GNesterov is not second best, Nesterov is. GNesterov appears to be second worst, just after Neural ODE (line 224 and table 1)
- On the point cloud task, in my experience only vanilla Neural ODE has problem separating the clouds, all others are able to. The experiment has also been called $g$ in the ANODE paper and Nested Spheres in Dissecting Neural ODEs
- Appendix line 515/eq 19: Has $a(T) = -I$ it should be $a(T) = \frac{\partial L}{\partial z(T)}$ 

Minor Typos (do not affect quality of paper):

- Line 25: "in neural network" should be "in a neural network"
- Equations in the paper should ideally be in brackets, these can be manually put in, or done using eqref in latex
- Line 83: It would make more sense to use $x_{k} = x_{k-1}...$ than $x_{t} = x_{t-1}...$ because $k$ is used in later equations for discrete changes (equation 7 for example)
- Line 200 and 318: "much fewer NFE" should be changed for "far fewer NFE" or "significantly fewer NFE" or simply "fewer NFE"
- $\mathcal{O}(k^2)$ and $\mathcal{O}(1/k^2)$ are used interchangeably in the paper to describe the convergence of Nesterov, it would be better to stay consistent
- Appendix footnote 1: "ANODDE" should be "ANODE"
questions: - Does the Nesterov method only really make sense for end point style tasks? That is, ones where the trajectory doesn't matter, so not time-series.
- Where does the oscillation in Figure 1 come from?
- Remark 1 talks about $\frac{k-1}{k-3}$ but in the previous paragraph and equation 7 we have $\frac{k-1}{k+2}$. Which one is it? Does this affect any results?
- Is applying the sigmoid function to stabilize Nesterov theoretically motivated, or more of an empirical trick to stabilize the ODE solve?
- The analysis of vanishing gradients is great, however in the worst case it still seems that we can get $\exp(-T)$ by the time we reach the start of the solve. It seems from the experiments this doesn't happen much. Has a similar analysis been done on other Neural ODE variants? To provide theory which supports the experiment in Figure 3
- Line 142, by writing $h(t)$ as $k(t)x(t)$, the singularity has moved from the ODE to $k(t)$, how has this actually helped, without the sigmoid being used the singularity is still there, just moved to $k(t)$
- Remark 2 talks about the Euler method to solve the ODE and Nesterov ODE. Is there a statement that can be made for adaptive step solvers? For examples a larger tolerance can be used for Nesterov over ODE?
- Has any timing taken place, to see the wall clock advantage of Nesterov?
limitations: The paper addresses both the limitations and the negative societal impact briefly in the conclusion.
review_title: null

Meta-review: metareview: The paper proposes an elegant way to use continious analogue for Nesterov accelerated gradient method instead of the Neural Ordinary Differential Equations (NODE) models for mapping the initial conditions to the output. The authors derive the adjoint ODEs, and show that such new "layer" results in lower number of function evaluations while integrating the NesterovODE. Thus, the authors clearly confirmed that the method is practical and efficient in all scenarios  where NODE models are used. This is a good paper.
confidence: Certain
Decision: Accept