Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method
Authors: Nghia Nguyen
Tan Minh Nguyen
Võ Thục Khánh Huyền
Stanley Osher
Thieu Vo
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural ordinary differential equations
nesterov
momentum

Abstract: We propose the Nesterov neural ordinary differential equations (NesterovNODEs), whose layers solve the second-order ordinary differential equations (ODEs) limit of Nesterov's accelerated gradient (NAG) method, and a generalization called GNesterovNODEs. Taking the advantage of the convergence rate $\mathcal{O}(1/k^{2})$ of the NAG scheme, GNesterovNODEs speed up training and inference by reducing the number of function evaluations (NFEs) needed to solve the ODEs. We also prove that the adjoint state of a GNesterovNODEs also satisfies a GNesterovNODEs, thus accelerating both forward and backward ODE solvers and allowing the model to be scaled up for large-scale tasks. We empirically corroborate the advantage of GNesterovNODEs on a wide range of practical applications, including point cloud separation, image classification, and sequence modeling. Compared to NODEs, GNesterovNODEs require a significantly smaller number of NFEs while achieving better accuracy across our experiments.

Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method
Nghia Nguyen" FPT Software AI Center
Tan M. Nguyen"
Department of Mathematics University of California, Los Angeles tanmguyenn9euclla.c edu
Ha Noi, Vietnam @thiinnh1@@fsee com.
Huyen K. Vo FPT Software Al Center
Stanley J. Osher Department of Mathematics University of California, Los Angeles sj o@math. ucla. edu
Ha Noi, Vietnam huyenvtk1@fsoftt. com. vn
Thien N. Vo
Faculty of Mathematics and Statistics Ton Duc Thang University, Ho Chi Minh City, Vietnam vongocthieu@tdet edu. vn
Abstract
We propose the Nesterov neural ordinary differential equations NODS)))))NODES)
whose layers solve the second-order ordinary differential equations (ODEs) limit of Nesterov's accelerated gradient (NAG) method, and generalization called GNesterovNODES. Taking the advantage of the convergence rate ( (1/h²²) of the NAG scheme, GNesterovNODEss speed up training and inference by reducing the number of function evaluations (NFEs) needed to solve the ODEs. We also prove that the adjoint state of NOesseroNNODEs also satisfies GNesterovNODEs. thus accelerating both forward and backward ODE solvers and allowing the model to be scaled up for large-scale tasks. We empirically corroborate the advantage of NOessero on a wide range of practical applications, including point cloud separation, image classification, and sequence modeling. Compared to NODEs, GNesterow require a significantly smaller number of NFEs while achieving better accuracy across experiments.
Introduction
Dynamical systems have been recently integrated into deep neural networks for modeling high- dimensional data. The advantage of this approach that well-developed mathematical modeling techniques from dynamical systems can be employed to improve neural networks. Along this research direction, the correspondence between residual networks, popular class neural networks with skip connections, and the numerical solution of ordinary differential equations (ODEs) have been vastly studied [15, 57, 47, 4]. The resulting Neural ODEs (NODEs) model when taking the discretization step to zero have shown great promises in wide range of applications including scientific discovery [24, 62), irregular time series modeling [46, 5], mean-field games [48], and
generative modeling [13, 61]. NODEs model the dynamics of hidden state (tt) g in neural
Co-first authors. Please correspond to: nghianhhl @fsoft.com.va tannnguyen89 @ ucla.edu vongoc- thieu @ tdtu.edu.vn
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

network by an ODE
dh(t) - ((h(),,,, (()) l((l);: dt
where function captures the dynamics and is chosen be neural network with parameters 0 that are learned from the data. Starting from the input h((n) at the initial time lo, NODEs compute the output ((T) at time 7 by solving the Initial Value Problem in Eq. (1) for some time T 2 ln- NODEs are trained by optimizing the loss ((h(T) between the prediction hTT) and the ground truth where the parameters 0 are updated using the following gradient [42]
d0 dc(h(T))
a(t) - 00
1
(f(h()),t,))
=
dt,
to
where a(t) == (t) is the continuous adjoint state, which satisfies the continuous adjoint equation
da(t) dt = a(t) ()((())
(3)
ah
NODEs solve both the ODE Eq. (1) in its forward pass and the ODEs (2) and (3) in its backward pass using black-box numerical ODE solvers. The literature refers this approach continuous adjoint method or seamiis-tten-disceetee
either the [21]. The number of function evaluations (NFEs) that these solvers need in a single forward and backward pass is among the main factors that decide the computational efficiency of the model, i.e. how fast the model is. Unfortunately, in many applicationss NODEs require high NFEs in both training and inference, especially when the error tolerances of the solvers are set to small values for obtaining high accuracy. Furthermore, the NFEs increase rapidly when training progresses. High NFEs deteriorate the efficiency of NODEs, reduce the accuracy of the trained model, and results in instability during training, making it difficult to scale up the models large- scale tasks [14, 8, 30, 37, 11].
1.1 Contribution
We propose the Nesterov Neural ODEs NODS)))))NOOEs) that leverage the continuous limit of the Nesterov's accelerated gradient (NAG) descent [52] and the convergence rate (11/k²) of the NAG scheme to enhance NODE training and inference. Our contributions are four-fold:
1.
We formulate the NesDeroxNODE that solves Nesterov ODEs, i.e. second-order ODEs with a time-dependent damping term, instead of first-order ODEs (1). To improve computational efficiency of the model, we convert these second-order ODEs into equivalent systems of first-order differential-algebriis equations that are solved in both forward and backward propagations of the NesterovNODE.
2. To eliminate the potential blow-up problem in training NosDsrow we further develop the Generalized NesterovNODEss (GNesterowNODDEs) by introducing skip connections [16] and gating mechanisms [18] into NODEss In general, GNesterovNODEEs form a wide class of neural differential equations which are represented by differentiall-llgerrii systems and contain NesterowNODEs as subclass.
3. We prove that the continuous adjoint equation used to compute the gradients for updating the parameters 0 in a GNes also follows a generalized Nesterov ODE. Thus, the NFEs in both forward and backward passes of GNesterovNODEs are significantly reduced, especially when the solvers are used with small error tolerances.
4. We prove that the spectrum of the NOes is well-structured. This property of GNesterovNODEs helps alleviate the vanishing gradient issue during training and allows the model to capture long-term dependencies in the data.
We empirically demonstrate the advantages of the NeseronNooDE/GGssservvYOOOE over the baseline NODE and the state-of-the-as neural ODE models including the Heavy Ball NODEs (HBNODEs), which solve the continuous limit of the heavy ball momentum accelerated gradient descent [60] on a wide range of applications including point cloud separation, image classification, and kinetic simulation. In all experiments, our proposed models achieve better accuracy and smaller NFEs than the baselines.

---

1.2 Organization
We structure this paper as follows: In Section 2, we review HBNODEs and NesterovODEs. In Section 3, we present the algorithm and analysis of the NesterowNODEss and NNesss We study the spectrum structure of the adjoint equations of NesterorNNODDsxGGNeeeeennnoD to show that ResterovNODFss//Neeteeevvv
can learn long-term dependencies effectively Section 4. In and
Section 5 and 6, we empirically validate the advantages of NesterowNODDE//NNsttrvvv analyze our models with ablation studies. We discuss related works in Section 7. The paper ends with concluding remarks. Proofs and additional experimental details are provided in the Appendix.
2 An Integration of Nesterov ODEs into NODEs
We first establish a connection between NODEs and gradient descent (GD), then review the Heavy Ball Neural ODEs (HBNODEs), and motivate the inlegration of Nesterov ODEs into NODEs.
ODE limit of gradient descent and connections to NODES Gradient descent (GD) has been among the methods of choice in optimization and machine learning for training complex systems. Starting from initial point By E R GD iterates as 3g - s vV (*x)) with being the step size in or- der to find a minimum of the function F F(z). Let s - 0, we obtain the following ODE limit of the GD
dæ -F(()). di
NAG
(4)
19y
Comparing Eq. (1) and (4), we observe that a NODE
10
solves the ODE limit of the GD where the gradi- ent -V,(x,)) is parameterized by a neural network (,)
10-
Heavy ball neural ordinary differential equations HBNODEs proposed in [60]. This model takes ad- vantage of the acceleration of heavy ball (HB) momen- tum [41] reduce the NFEs needed for solving the ODEs and speed up NODEs. In particular, HBNODEs replace the first-order ODE limit of GD by the following second- order ODE limit of heavy ball momentum method:
10
10-30 10kk
20k
30k
50k
Comparing the convergence of GD, HB and NAG for solving the optimiza- tion problem ming F(xr = 2 x L x b where I, E Rdxd is the Laplacian a cycle graph and b is a d-dimensional vector whose first entry is and all the other entries are O.
Figure 1:
dx((t)
d12 + dx((t - (x)(())
(5)
dt
Similar to NODEs, [60] parameterize (z)(((t)) by neural network f(x(t), and formulate HBNODEs as follows
d=((() dt + dx((1) = ft)
(6)
dt
where 7 > 0 the damping parameter, which can be a hyperparameter or a learnable parameter, Nesterov's 's accelerated gradient (NAG) momentum Even though HB improves the convergence and accelerates GD, both GD and HB share the same convergence rate of ((1/k) for convex smooth optimization. A breakthrough due to Nesterov [33] replaces the constant momentum y with (k- - 1) + 2). a.k.a. NAG momentum, improves the convergence rate to O(1/k²), which is proved to be optimal for convex and smooth objective functions [33, 52]. We demonstrate the faster convergence of NAG in comparison with GD and HB on a quadratic optimization problem in Figure 1. The much faster convergence rate of NAG than that of GD and HB motivates to incorporate the second-order ODE limit of NAG into a NODE and propose the NesteroyNODE. Nesterov acceleration gradient method [33] takes the following form: given initial points To R.M and yo = Tg. the sequence ((T), Bh. } is defined inductively as:
Ik Dk-1 1 sV F(K 1),
h-
Uk + k+ E IR-
The continuous limit of the Nesterov scheme is obtained by setting Ia = h(kv/)) = h(() with I t= k-/8 and some smooth function h from IR NR. According to [52], the function h satisfies the Nesterov ODE
h ht(t) + f h'(t) + vVb((t)) = 0,
(8)
with the initial conditions h(0) ho, h'(0)= O.
3

---

Remark 1 (Nesterov factor). The constant 3 in the coefficient of (t) in Eq. (8) is originally from the approximation (k- (0-1))(8 + +
O(1/k²). This constant will be replaced by a constant y if the factor (k- (k-1)/(k + 2) 2 in Eq. (7) is replaced by (k 18-111 1)/(k + +r- 1). It is proved in [52] that the Nesterov ODE still holds the quadratic convergence rate when 3 replaced by any number r >3 3.
Remark (Numerical stability). If the Fuler method is used, then to keep the numerical solution close to the exact solution, the step size chosen in the Fuler method must be small enough. However, the smaller the step size, the more expensive the computation. The maximum stable step size, which the maximum value for which the step size can be chosen so that the numerical solution remains close to the exact solution, reflects the numerical stability of the ODEs. It is proved in [52] that the maximum stable step size of the Nesterov ODE is much larger than that of the ODE (4), thus showing the numerical stability advantage of the Nesterov ODE.
3 Generalize Nesterov ODEs to Differential Algebbaai Systems
One parameterize V (h(()) in Eq. (8) by neural network f (h((t), t, 8) with learnable parameters
0 in a similar way as NODE. This results in the following Nesterov Neural ODE (NesteroNNODE)
1018
h" ht(t) + h'tt) + f() 0,
MOGF
ANDCE
1014
SONORE
HENOGE
This second-order NODD can be written in term of the first-order Nesterov as
10"
GHINNDE
Nesterswicee
109
h' (t) (t), 3
10a
109
(10)
m - m(1) (m()))((((0)..0))
-20
20
40
Figure 2: Contrasting the increase
12-norm of htt for NODE, ANODE, SON- ODE, HBNODE, GHBNODE, NesterovN- ODE, and GNetteroNNODE over long inte- gration time on the Silverbox Initialization task (More details in Appendix D.I).
However, because of the singularity created by the coef- ficient 3. the training process based directly on Eq. (9)
and (10) will be unstable. To avoid the instability issue,
we set h() = (())x(() with k())= = - Then Eq. (8)
becomes
k(t)x" + (t)(t) + _ -b()) x x((1) (*(()+ 3 ( 20(0)
x(t) + VF(b(t)) O.
(11)
We observe that 2t)(t) + (t)) = k(t). By dividing both sides of Eq. (11) by k(t), we obtain:
(t) + x + (t)+)fh(t)), = O.
(12)
where
f(b(t),t) 1 3 1
h(t (
Let m(t) xt).. Then Eq. (8) is equivalent to the following differeetiallalgebraiic system
lntt) = t - e'xx((t),
x (t) = mt),
(13)
m'(t) = (m()) f(h(t),t).
We parameterize [(h()), () neural network, recalled as f(h(1), 1, 0) with learnable parameter and obtain the differentiallalgebrrii version of NesD
h(()= = e
(t) m(t),
(14)
m m'(t) (t s (h(),,,,)).
The singularity at L = of the NeDD in Eqs. (9) and (10) is now moved to the algebraic
equation of the above differeentiallalgebrii system.

---

It is often the case when training ODE-based models that some functions diverge or explode at finite time. This phenomenon called blow-up. which we demonstrate in Fig. 2. In order to alleviate the blow-up problem, we introduce generalized version of NesteroWNODE, termed Generalized NesterovNODE (GNesteroNNODE)) For the NeSE,rovNODE, the potential blow-up is due to the oscillation inherited from NAG scheme as can be seen in Fig. 1. The blow-up can also occur because of the singularity caused by the factor - in the algebraic equation. Following the techniques presented in [60], we address these potential blow-up by applying an activation function o to the function f(((t), f, 0). the factor := e and the momentum state m(t) of the NesteroVNODE. The activation function can be any activation function commonly used. In our experiments, we use tanh and hardtanh. In addition, the residual term {h(t), which stands for skip connection [16], also added into the governing equation of (t), which benefits training and generalization of GOesssroNNODEs. The GNesterovNODE aefferentiialalgebere system is then given by
hh(() = at 2 ( (x),
x() (mm((),
(15)
m' (t) = -m(() - (f((tt) 0)) - <h(t),
with the initial conditions h(0) ho, x(0) Xu. (0) = mo- It is noted that, from h(0) ho and h'(0) 0, we must have X ha lim o et and mg ho lim d 8-0 at ( Fig.
x00
shows that GNesterovNODE can indeed control the growth of ltt) effectively.
In general, NNesseroNOODEs form wide class of neural differential equations which are represented by différential-algebraic systems and contain NesterovNODEs a subclass. Eqs. (9) and (15) define the forward ODE for the NoDD To efficiently update the parameters during the training process based on NONDss,, we use the continuous adjoint sensitivity given in Propositions 1 and 2 below.
Proposition 1 (Continuous adjoint equation for the second-order NesterovNODE).. The continuous ac adjoint state a(t) = 8h([) of the NesterowNODE given in Eq. (9) satisfies the following NesterovN-
ODE
3 aa(1)++ aa() (f(h(t),t.0) a" a"(1)
3 {2 a() 0.
(16)
5h
Proposition (Continuous adjoint equation for GNesterovNOODE) The continuous adjoint state dc (1) 5x(1]
functions (t,(1)= := ah(t).
ØC and am (1) == 7m(() of the GNesterowNODE system given
in Eq. (15) satisfy the following wewprrentiialaagbbbeee adjoint system
(t) = of -
( (Ja((((())
(t) = - e am (t)
+ +
(17)
Öh
a/m(1) am (1) ax (t)))()(t)),
with the final value conditions an (T) Thy ax (T)
aL
8L
and am (T)
aL
UxT
Jmr
4 The Effectiveness of GNesterovNODE in Alleviating Vanishing Gradients
In neural networks with many layers, the vanishing gradient is one of the major issues [40]. In the cases of NODES and their hybrid ODE-RNN variants, the vanishing gradient issue may occur when the adjoint state a(t) := Jhiey goes to 0 quickly T - t increases. In this section, we will prove that this vanishing gradient issue can be avoided in GNestrroNOODEs.
sc
For the NNeD given in Eq. (15), the gradient ac can be determined from ac via the 0x,
9h,
algebraic relation:
aL OC 0x, Th, dx, Oh,
OC = alt 2 ed) 0x.
=3
While the gradients
ac DC and 0m, satisfy the following proposition. ox.
5

---

60
NODE-ANN
60
GHBNODE-RNN
60
NesterawNODE-RMN
60
GNesterowNODE-MM
0.03
40
40
40
40
0.02
20
20
20
0.01
400
Epoch
Epoch
Epech
Epoch
Figure 3: Plot of the 12-norm of the adjoint states for ODE-RNN, GHBNODE RNN, NesterovNODE-RNN and GNesteroyNODE-RN
back-propagated from the last time stamp. The term T demonstrates the gap between the final time T and intermediate time t. When the gap T - f becomes larger. NesterowNODE-RNN and NNeDEerowNODE-RNN can address the vanishing gradient problem due to the adjoint states of these methods' decay slowly.
Proposition 3. For every E (O,T), there exist a unit length row vector E CN and an upper triangular matrix U E NNN such that
ac
||
dc
ac dmT
- -|]oexp(U))1
0x
0m,
0xT
and at least E/2 complex values in the diagonal U have the real parts greater than or equal to TTT.
The term v (U) in the above proposition plays the essential role in the nonvanishing gradient issue. Without loss of generality, we can assume that U Uarge
Usmall where all complex numbers in the diagonal of Umage (resp. Usmal)) have the real parts greater than or equal to (resp. smaller than) (l - 7). According to Proposition 3, the size of Unage is at least N/2. Then we have,
0
exp(U)) Texp(Uiarge)
P
and ||oexp(U)]12 2 |/Plagge exp(Uuuge)ll2
(xp(Usman))
Here, the vector Uarge is the first m columns of v. and m is the size of Utaree Since the real parts of elements in the diagonal of Unage is less than (t T), exp(Ulanee) decays rate at most
} (T) This results in the nonvanishing gradient of DC
ac
hence so
Bc
ac
ac
8x
8m.
Sh,
Gx.
8m.
To illustrate, we take the Walker2D kinematic simulation task [2] in consideration, which requires learning long-term dependency effectively [27]. We train ODE-RNN [46], GHBNODE. RNN [60], NesterowNODE-RNN and GNesterovNODE-RR on this benchmark dataset (More ac
details in Appendix D.4). Fig. 3 plots ac for ODE-RNN,
for GHBNODE- ac
2h.
RNN and that when the gap between the final time T and intermediate time t becomes larger, the adjoint states of Neet--ooNNOEERR. RND,, GNesterovNODE--R NONN--NDENNNNNDDE and GHBNODE- ERNN decay much more slowly than NODE-RNN. Thus, NesterovNODE--NNN and GNestenoyNODE-RNN have the ability to tackle the vanishing gradient issue.
ac
ac
ac
for NesterowNODE- RNN and GNesterowNODE-RNN showing
8h.
8x:
3m:
Remark 3. The gradient exploding problem can be effectively resolved via gradient clipping, training loss regularization, etc. [40, 10]. Therefore, in practice the vanishing gradient problem is the major issue for training deep neural networks [40].
5 Experimental Results
In this section, we empirically study the advantages of our proposed NesterowNODD/GGNesteevvv ODE over the bascline NODES and other popular NODE- architectures, including the augmented NODE (ANODE) [9], the Second Order NODE (SONODE) [37], HBNODE/GHENODE [60] on a variety of benchmarks including point cloud separation, image classification, and kinetic simulation which involve different data modalities ranging from point cloud to images and time series. ANODES augments the space on which the ODE is solved while SONODES and (G)HBNODEs solve second- order ODE. We aim to show that: (i)
require significantly fewer NFEs while attaining similar or even better accuracy as the baselines; (ii) GNesterovNODEs avoid the blow-up of h (t) and thus improve over NeDt;;;wNODEs; (iii) NesceooeNNDDES//GNeeeeev...D

---

for image Table 1: The parameters count for the models in point cloud separation, image classifications, and the Walker2D kinematic simulation tasks and the test accuracy JOFAISTM Our methods able reach similar better test accuracy than the baseline methods on CIFARIO while retaining a similar test accuracy on MNIST.
Number of parameters
Test Accuracy
Model
PC
MNIST
CIFARIO
Walker2D
CIFARIO
MNIST
NODE
545
85316
173611
9929
0.5466 + 0.0051
0,9531 + 0.0042
ANODE
587
85462
172452
10019
0.6025 + 0.0032
0.9816 0.0024
SONODE
541
86179
171635
11471
0.6132 + 0.0073
0.9824 + 0.0013
HBNODE
582
85931
172916
10099
0.5989 = 0.0035
0,9814 0.0011
GHBNODE
582
85931
172916
10099
0.6085 + 0.0050
0.9817 + 0.0005
NesteroyNODE
581
85930
172915
10098
0.5996 + 0.0033
0.9824 + 0.0015
GNesterowNODE
581
85930
172915
10098
0.6172 + 0.0064
0.9807 = 0.0013
Train Forward NFEE
Train Backward NFFES
300
Test Forward NowwardFFFss NFEE
0.65
Test Accuracy
Train Loss
1.50
60
200
600
1.00
20
100
0.50
0.55
40
0.00
0.50
Epoch
Epoch
Epoch
Epoch
Epach
NODE
ANODE
SONODE
HBNODE
GHBNODE
NesterovNODE
GNesteroVNODE
Figure Contrasting the NFEs and accuracy NODE-based baselines and our methods Nesteno-NODD/GNNss lerovNODE on the CIFARIO 10 dataset (Tolerance: 10 10"").
capture better long- term dependencies than the baselines and achieve better results in sengseeuence modeling tasks.
For all the experiments, we use Adam [23] as the optimizer and Dormand-Prince 5(4) [7] as the numerical ODE solver. We choose the network architecture used to parameterize f(((t,, t, 8) so that our proposed models and the baselines have similar numbers of parameters in our experiments as shown in Table 1. Other rraining/model/datass details are provided in Appendix D. All results are averaged over 5 runs with different seeds. We conduct the experiments server with 6 NVIDIA 2080Ti GPUs with 1GGB of GPU memory. Our Py ToTch code with documentation can be found at htwp://hww..oooommmmmmmnnnnne//e
5.1 Image classification
We validate the accuracy and efficiency advantage of classification MNIST [6] and CIFAR1O 10 [25] in comparison with other ODE-based baselines. We follow the same training and model settings as in [60].
NFEs. As shown in Fig. 4, NesterowNODE and GNesterovNODE reduce the NFEs in both the forward and the backward propagations compared to the baseline models. Although the augmented input dimensions in ANODE help reduce the NFEs compared to NODE, second-order methods reduce the NFEs more significantly. Compared to the other second-order methods i.e. SONODE. HBNODE, and GHBNODE, GNesterowNODE achieve much better NFE reductions both MNIST (see Fig 12 in the Appendix) and 10, indicating the improvement in efficiency and stability of methods over the other second-order baseline models. Such advancement is essential step scale GNeD to larger and more complex practical tasks.
Accuracy. Table shows that GNetteroNNODE achieves the highest test accuracy CIFARIO. On MNIST, NoSD attains the second-hige -ighhsstthtt test accuracy and very close to the best result from SONODE while being much more efficient than SONODE. This advantage in terms of accuracy can be associated with the small number of NFEs needed by NesterowNODE and GNesterowNODE above, which reduces the model complexity and leads to better generalization.
Note that GNesterovNODE improves over NeSD efficiency and accuracy (on larger and more challenging benchmark like CIFAR 10). This justifies the effectiveness of our solution that introduces an additional bounded activation function and the residual term gh(() to prevent the blow-up of ht) explained in Section 3.
7

---

NFEs (backward)
200
200
0.4
0.4
NFES
100
0.0
0.0
200
200
200
200
400
Epoch
Epoch
Epoch
Total NFEs
NODE
ANODE SONODE
HBNODE
GHBNODE
NesterovNODE
GNesterovNODE
Figure 5: Contrasting the NFEs and training loss of NODE-based baselines and NestenoyNNDD//Nees
terovNODE on point cloud benchmark. Results are averaged over 50 runs (Tolerance: 10 1
NFE
LacoHard
2000
1.3
2000
1.3
LasS
1750
1750
1.1
Test
2.1
1250
200
400
200
400
200
400
200
400
200
400
Epoch
Epoch
Epoch
Epoch
Epoch
NODE-RNN
SONODE-RAN HBNODE
GHENODE-RRN
GRNesterowNODEERR
ANODE RNN
NeszerevNODE
Figure 6: Contrasting the NFEs and losses of NODE-RNN [4), ANODE-RNN [9]. HBNODE-RNWGHHBNOO RNN [60], and our NeSD WNN/DD/NNNNNNOOOOODDDD RNN on the Walker2D dataset (Tolerance: 10-7)
5.2 Point cloud separation
We perform experiments on point cloud separation task in order to verify that NesterowNODEs/GNNe terovNODEs can learn effective features separate two sets of point clouds. The first consists of 40 points drawn from a circle with the radius ||||| < 0.5. while the second set comprises 80 points drawn from an annulus with the inner and outer radius of 0.85 and 1, respectively, i.e. 0.85 < ||||| < 1.0. Fig. 5 shows that our NesterovNODE and NNeD models are able to converge to O loss consistently while other methods have difficulty to reach 0 loss. In addition, NesterovNODE and especially the GNesterovNODE require significantly fewer NFEs in forward and backward passes compared to the baselines. Thus, NesterovNODE and GNesteroVNODE help improve both the training and the efficiency of the model. We plot the evolution of the point cloud separation through 100 epochs for a random run of each model in Appendix D.2. Like SONODE. LBNNDDE/HBBOO
learn effective features that allow good separation between the two classes of point clouds in these experiments while NODE and ANODE fail for this task.
5.3 Walker2D kinematic simulation
In this section, we investigate Nestero/NODD//GNssttroo when applied on time-series data. In particular, we use the ODE-RNN framework [46], with the recognition model being different ODE-based models, to study Walker2D kinematic simulation task, which requires learning long-term dependency effectively [26]. As shown in Fig. 6, our NesterowNODE-RNY and GNesteroVNODE: RNN not only reduce the NFEs both in the forward and the backward stages, but also achieve smaller test loss compared the baseline models that we compare with, including (G)HBNODE-NNN ANODE-RNN. and NODE-RNN, Although SONODE RNN achieves much smaller losses, its NFEs too high, thus training SONODE-RNE for this task is much more cone-ccnguiing compared to our Nesterov- methods.
5.4 Continuous Normalizing Flows for MNIST
We compare the NODE with the baseline NODE and GHBNODE for use in variational inference with the continuous normalizing flow model trained on the MNIST dataset. The continuous normalizing flow model we use the FFJORD in [14]. We summarize our results in Figure Com- pared to the FFJORD-NODE and TFJORD-GHBNODE, the TFJORD-GNeeterovNOOO0 significantly reduces the NFEs in both forward and backward passes while improving the negative ELBO on the test set. This experiment demonstrates that our method NNeD accelerates NODE-baser models on both discriminative tasks and generative tasks.
Observed Properties of NODD in GNesteroyNODE
In this section, we verify empirically that with the addition of activation function a and the skip connection, GNesseroNNODEs still preserve important properties of NesterovNODEs as stated in Re-

---

150
Negative ELBO
Forward NFES
Backward NFES
200
100
100-
200
50
50 Epoch
100
50
100
50 Epoch
100
Epoch
FFJORD NODE FFJORD GHBNODE
FFJORD GNesterovNODE
Figure 7: Contrasting the NFEs and the validation negative ELBO of the FFJORD-NODE, the FFJORD- HBNODE, and our SeJJODD-GNeserovNOOOO. for the variational inference task with a continuous normalizing flow model, i.e. FFJORD [14], the binarized MNIST dataset (Tolerance: 10 10-").
800
0.65
Train Beckwardd
Teat Accuracy
200
3
1.50
Tusx Forward
0.60
75
Train
1.00
50
0.55
0.50
25
r
r-
Figure 8: Contrasting different values of the factor on CIFAR10 with GNesterow (Tolerance: 10-5).
mark and Remark 2. This hints that GNesterovNODES have the same behaviors as Nesterov.NODEs while enjoying more stable training.
Effect of the Nesterov factor NesteroyNODE (Remark 1)
As stated in [52], the "magic constant" can replaced by a constant r > while maintaining convergence rate of (l1kk². In this work's 's experiments, larger leads to better test loss eventually despite a smaller outperforming in the beginning, but their experiments are based on Nesterov ODE. This section investigates whether this behavior extends to our generalized method GNesterovNNODE. As shown in Fig. 8 and Table 2, larger r also leads to higher test accuracy, and for r = 3 and 5, the model converges to its best test accuracy sooner than the training loss of r 10 is larger than r = 3 and r = 5, we observe that all three values reach their best testing accuracy when the training loss is approximately in the range of [0.65, 0.75), which hints that decreasing the loss further leads to overfitting. One more interesting observation is that a higher value of increases the forward and backward NFEs. Intuitively speaking, the term -hh((t) in the NesterovNODEE which is opposite to the product of the damping parameter r/t and the velocity h' (t), represents the friction force of the model (see [52, Section 4]). When r is larger, the friction force resists the movement of htt) along the trajectory stronger, which slows down the convergence of training loss.
10. Although
Stability of NesteroyNODE (Remark 2)
The NesterovNODE is more numerically stable than NODE in the sense that the step size in the Euler method for solving the Nes- terovNODE can be chosen larger while the stability of the numerical solution is still guaranteed (see Remark 2). We hypothesize that this fact still holds for the NODE in comparison with both NODE and GHBNODE. To illustrate this fact, we perform experiments on CIFARIO using Euler solvers with large step sizes (0.1, 0.2, 0.5). Due to the instability of large step size, GHBNODE moves fast to the maximum accuracy points and then goes down, as shown in Fig. 9, while NODE achieves high train loss and low test accuracy. Even when the step size big. GNestsrovNODE's training is stable as the curve evolves gradually. More interestingly, as shown in Fig. 10, NNetteroNOODE solved with rk4 (Fourth-order Runge-Kutta with 3/8 rule) solver using large step size outperforms the same model solved with the adaptive step size solvers like dopriš (Dormand-Prines of order 5) solver. This show the promise of using large step sizes in GNesterowNODE to obtain models with better accuracy and efficiency.
Table 2: Test accuracy for GNes- terovNODE on CIFARI0 with varying Nesterov factors (Tol- erance: 10 10-5).
Value of
Test accuracy
3
0.6180
5
0.6192
10
0.6296
7 Related Work
Increasing efficiency training NODEs. Several methods have been proposed to reduce the NFEs in NODEs and increase the model efficiency. Among them are works that use weight decay [14] and

---

NCDE 0.3. 0302003
NODE O.1: 6848
NODE 0 604439.8
NNDDEE 0228 LIRY
1.00
1.00
0.75
0.75
Toin
0.50
0.25
0.56
0.25
0.550
0.54
0.50
20
20
40
20
40
20
Epoch
Epoch
GHBNCOE 0.1
GHBKODE 0.2
GHENODE 0.5
GNeesterwWDDD auler 0.5
GNesterovNODE rk4 0.5
GNee wrovNDDE 0.2
GNenterovADOE 0.5
GNesterovNODE explci_adams
GNesterowNODE depri5
Figure 9: Train loss and test accuracy for large step sizes of Euler solver on CIFARIO training with NODE, GHBNODE and NNeD In the legend, we place the step size next to the model's name.
Train loss and test accuracy for various solvers with large step size 0.5 CIFARIO training with GNesterovNODE, In the legend, we place the step size next to the model's name.
Figure 10:
other regularizers applied on the solver and the learned dynamics [11, 20, 12, 39]. Other works employ input augmentation [8], data-control [30] and depth-variann [30, 36] to reduce the NFEs needed to compute the ODE solution. Another method replaces norms with seminorms in the backward continous adjoint stage to reduce backward NFEs [22]. NesterovNODE solves second-order Nesterov ODE to reduce both forward and backward NFEs.
Second-order dynamical systems. Second-orde -odder ODEs have also been employed to speed up NODEs. SONODE [37] replaces the first-order ODE in Eq. (1) by a second ODE which can be solved as system of first-order ODEs. HBNODE [60] also solves a second-order ODE but with an additional constant damping parameter, which corresponds the ODE limit of the HB momentum method. NesteroWNODE: solves the second-order Nesterov ODE with time-dependent damping parameter, which corresponds to the ODE limit of the NAG momentum method. These momentum based systems have also been employed in designing deep neural networks as in [32, 28, 49, 35].
Learning long-term dependencies. The ability of model to learn long- dependencies is highly needed to scale up the model to large-scale tasks that involve very long sequences. Existing works try to alleviate exploding or vanishing gradient issues happened during the training of recurrent neural networks, including [1, 59, 19, 55, 31, 17, 34]. Recently, learning long-term dependencies with NODEs has been explored. For example, [26] integrate long-short term memory cell into NODEs. Among the hallmarks of NesterovNODEs is that our proposed models can directly capture long-term dependencies long sequences.
Concluding Remarks
In this paper, we propose the NesterovNODE and its generalized version GNesterovNODE that solve the second-order ODE limit of NAG. These models take advantage of the convergence rate (11kk²) of the NAG scheme to gain acceleration over NODEs and the existing NODE-based models such HBNODE by reducing the NFEs in solving both forward and backward ODEs. Our Nesterov-based NODEs also achieve better accuracy than NODEs and outperform or at least are on par with other NODE-based models in our experiments while requiring much fewer NFEs. We also prove that NesterowNODEs and GNesserovNODEs can avoid the vanishing gradient issue and capture long-term dependencies in long sequences effectively. It is worth mentioning that SesscrovNODD///G/eerorooooOO
do not introduce any inherently negative socictal impact. The high-resolution ODE in [51, 50] is an alternative way to take continuous-time limit of the NAG scheme and heavy-ball momentum method. This high-resolution ODE introduces a gradient correction that allows the NAG scheme achieve inverse cubic rate for minimizing the squared gradient norm, which is better than the inverse square rate in the low-resolution ODE that our method uses. A limitation of our paper is that we have not incorporated restart schemes [38, 45, 56) into the NesterovNODE, and we leave this as future work.
Acknowledgements
This material is based on research sponsored by the AFOSR MURI FA9550-18-1-0502. the ONR grant N00014-0011-2093, the MURI N00014-20-1-2788 and the NSF under Grant# 2030859 the Computing Research Association for the CIFellows Project (CIF2020-UCLA--3) NH acknowledges support from the NSF [FML 2019844 and the NSF AI Institute for Foundations of Machine Learning.
10

---

References
[1] Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference Machine Learning, pages 1 1120-1128 2016.
[2] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01550, 2016.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1o06::6666.01444: 2016.
[4] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[5] Edward De Brouwer. Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-obsered time series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[6] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142) 2012.
[7] John Dormand and Peter J Prince. A family embedded runge-kutta formulae. Journal of compulational and applied mathematics, 6(1):19-26, 1980.
[8] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In H. Wallach,
H. Larochelle. A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[9] Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural odes. H. Wallach,
H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[10] Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. arXiv preprint arXiv:2006.12070 2020.
[11] Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M. Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In ICML, pages 3154-3164, 2020.
[12] Arna Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. Steer Simple temporal regularization for neural ode. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14831-14845
Curran Associates, Inc., 2020.
[13] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable re- versible generative models with free-form continuous dynamics. International Conference on Learning Representations, 2019.
[14] Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learn- ing Representations, 2019.
[15] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1):014004, 2017.
[16) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.

---

[17] Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled Cayley transform. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1969-1978, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018, PMLR.
[18) Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):1735-1780, 1997.
[19] Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, and Marin Soljačić. Tunable efficient unitary neural networks (eunn) and their application to mns. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1733-1741. JMLR. org, 2017.
[20] Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learning differen- tial equations that easy solve. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4370-4380. Curran Associates, Inc., 2020.
[21] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02433. 2022.
[22] Patrick Kidger, Ricky TQ Chen, and Terry J Lyns.. hey, that's not an ode": Faster ode adjoints via seminorms. In ICMI., pages 5443-5452, 2021.
[23] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[24] Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International Conference on Machine Learning, pages 5361-5370. PMLR, 2020.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Mathias Lechner and Ramin Hasani. Learning long-term dependencies in irregularly-samplee time series, 2020.
[27] Mathias Lechner and Ramin M. Hasani. Learning long-term dependencies irregularly- sampled time series. CoRR, 00442000604418, 2020.
[28] Huan Li, Yibo Yang, Dongmin Chen, and Zhouchen Lin. Optimization algorithm inspired deep neural network structure design. In Asian Conference on Machine Learning, pages 614 614-69 1--629. PMLR, 2018
[29] M Luštrek, B Kaluža, Rok Piltaver, Jana Krivec, and Vedrana Vidulin. Localization data for person activity data set, 2010.
[30] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissect- ing neural odes. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3952-3963. Curran Associates, Inc., 2020.
[31] Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. In Proceedings of the 34th International Conference Machine Leurning- Volume 70, pages 2401-2409. JMLR. org, 2017.
[32] Thomas Moreau and Joan Bruna. Understanding the learned iterative soft thresholding algorithm with matrix factorization. arXiv preprint arXiv: 7777.03333 2017.
[33] Yurii E Nesterov. method for solving the convex programming problem with convergence rate (1/k" 2). In Dokl. Akad. Nauk Sssr, volume 269, pages 543-547, 1983.
12

---

[34] Tan M Nguyen, Richard Baraniuk, Andrea Bertozzi, Stanley Osher, and Bao Wang. Momen- tumrnn: Integrating momentum into recurrent neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1924-1936. Curran Associates, Inc., 2020.
[35] Tan M Nguyen, Richard Baraniuk, Robert Kirby, Stanley Osher, and Bao Wang. Momentum transformer: Closing the performance gap between self-attention and its linearization. In Mathematical and Scientific Machine Learning, pages 189-204. PMLR, 2022.
[36] Tan M Nguyen, Animesh Garg, Richard G Baraniuk, and Anima Anandkumar. Infocnf: An efficient conditional continuous normalizing flow with adaptive solvers. Asilomar Conference, 2022.
[37] Alexander Norcliffe, Cristian Bodnar, Ben Day. Nikola Simidjievski, and Pietro Lió. On second order behaviour augmented neural odes. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1-921. Curran Associates, Inc., 2020.
[38] Brendan O' d'donoghue and Emmanuel Candes. Adaptive restart for accelerated gradient schemes. Foundations of Computational Mathematics, 15(3)715-732, 2015.
[39] Avik Pal, Yingbo Ma, Viral Shah, and Christopher V Rackauckas. Opening the blackbox: Accelerating neural differential equations by regularizing internal solver heuristics. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8325-8335. PMLR, 18-24 Jul 2021.
[40] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference Machine Learning, pages 1310-1318, 2013.
[41] Boris Polyak. Some methods of speeding up the convergence of iteration methods. USSR Compulational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
[42] Lev Semenovich Pontryagin. Mathematical theory of optimal processes. CRC press, 1987.
[43] Christopher Rackauckas, Mike Innes, Yingbo Ma, Jesse Bettencourt, Lyndon White, and Vaib- hav Dixit. Diffeqflux.jl. A julia library for neural differential equations. CoRR, abs/1902.02376,
2019.
[44] Christopher Rackauckas and Qing Nie. Differentialequations.j] a performant and feature-rich ecosystem for solving differential equations in julia. The Journal of Open Research Software, 5(1), 2017. Exported from happs//aapp.dmmmniooo. on 2019/05/05.
[45] Vincent Roulet and Alexandre d' Assremont. Sharpness, restart and acceleration. In Advances in Neural Information Processing Systems, pages 11 1119-1129, 2017.
[46] Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for aregullaryysamplle F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
time series. In H. Wallach, H. Larochelle, A. Beygelzimer,
[47] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pages 1-13, 2019.
[48] Lars Ruthotto, Stanley J. Osher, Wuchen Li, Levon Nurbekyan, and Samy Wu Fung. A machine learning framework for solving high-dimensional mean field game and mean field control problems. Proceedings of the National Academy of Sciences, 1 11(17):1133-1193 2020.
[49] Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Momentum residual neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9276-9287 PMLR, 18-24 Jul 2021.
13

---

[50] Bin Shi, Simon S Du, Michael Jordan, and Weijie J Su. Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Programming, pages 1-70, 2021.
[51] Bin Shi, Simon S Du, Weijie Su, and Michael Jordan. Acceleration via symplectic discretiza- tion of high-resolution differential equations. Advances in Neural Information Processing Systems, 32, 2019.
[52] Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nes- terov's accelerated gradient method: Theory and insights. In Advances in Neural Information Processing Systems, pages 2510-2518, 2014.
[53] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026-5033. IEEE, 2012.
[54] Ch Tsitouras. Runge-kutt pairs of order (4) satisfying only the first column simplifying assumption. Computers & Mathematics with Applications, 62(2):770-775, 2011.
[55] Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learn- ing recurrent networks with long term dependencies. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 3570-3578. JMLR. org, 2017.
[56] Bao Wang*, Tan M Nguyen Andrea L Bertozzi, Richard G Baraniuk, and Stanley J Osher. Scheduled restart momentum for accelerated stochastic gradient descent. SIAM Journal on Imaging Sciences, 2022.
[57] Weinan. A proposal on machine learning via dynamical systems. Communications in Mathemalics and Statistics, 5(1):1-11, 2017.
[58] Torbjörn Wigren and Johan Schoukens. Three free data sets for development and benchmarking nonlinear system identification. In 2013 European control conference (ECC), pages 2933- 2938. IEEE, 2013.
[59] Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems, pages 4880-4888, 2016
[60] Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Minh Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. In A. Beygelzimer, Y. Dauphin, Liang, J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, 2021.
[61] Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae:
Deep generative second order odes with bayesian neural networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[62] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference Learning Representations, 2020.
14

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 8
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you ran experiments...
(a) Did include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "fZlQSDQYCbP", "reviewer": "Reviewer_Uc87", "summary": "NesterovNODE and GNesterovNODE are introduced, as an alternate DAE-based parameterisation of a neural ODE. They are explored primarily for their ability to improve accuracy and reduce computational cost.", "questions": "While the bulk of my review touches on several points, I would particularly highlight the following questions/suggestions:\n\n1.  I quite like the trick of reparameterising from $h$ to $x$ in order to obtain a singularity at $t=0$. I am curious what motivated this approach in particular; how was this derived? And as above, I am curious why this approach is taken, instead of simply starting the integration from e.g. $t=1$ rather than $t=0$.\n2. As above, how is the numerics/software was handled?\n3. The experimental weaknesses offer substantial room for improvement.", "limitations": "There are no ethical concerns with this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength: topic**\n\nThe topic at hand is a meaningful one, as good parameterisations of neural ODEs is still an underexplored problem.\n\n**Strength: Nesterov/DAE approach**\n\nI like the approach of using Nesterov's method as a starting point for choosing a good architecture. This seems well-motivated, and the constructions presented both plausible and novel. (To the best of my knowledge.)\n\nIn addition, I like the formulation of the problem as a DAE. To the best of my knowledge \"neural DAEs\" have received essentially no attention whatsoever, so this is a very interesting line of work to pursue.\n\n**Strength: presentation**\n\nThe presentation is at all times both clear and technically precise. I greatly appreciated this aspect of the paper, which I enjoyed reading.\n\n**Strength: experiments**\n\nSeveral different kinds of experiments are done, and in each case a careful analysis is performed. These are generally well done.\n\n**Weakness: handling blow-up**\n\nThe authors go to quite some effort mitigating the blow-up of the Nesterov equations. I feel like much simpler approaches are available -- for example, starting the integration from $t=1$ rather than $t=0$.\n\n**Weakness: comparison to gradient descent**\n\nLines 81--88: I strongly object to the claim that neural ODEs \"implicitly perform gradient descent\". It is trivially true that both NODEs and GD are cases of generic ODEs; that is all that is going on here.\n\n**Weakness: \"adjoint sensitivity\"**\n\nThe term \"adjoint\" has become heavily overloaded. There are multiple ways to backpropagate through a differential equation solve. In the current literature is is often used to mean specifically optimise-then-discretise, but in older literature it was often used to mean specifically discretise-then-optimise. Following [0, Remark 5.5] I recommend avoiding the term \"adjoint\" altogether.\n\nIn passing, since in this work backpropagation is performed via OtD, then the authors may find speedups by utilising the technique of [1].\n\n**Weakness: numerical/software**\n\nIt is claimed that Dormand--Prince 5(4) was used to solve the DAE problems presented. However DP5(4) is an ODE solver, not a DAE solver. The problem can of course be rewritten in ODE form (e.g. equation 10) but this would no longer seem to follow the DAE presentation that is otherwise adhered to. How were the numerical methods actually implemented, i.e. what ODE was actually solved?\n\nI note that for example [2] implements DAE solvers directly, which could have been an alternate approach.\n\nIn passing, note that DP5(4) is now generally reckoned as being slightly less efficient than the Tsitouras 5(4) method [3], which can be found implemented in both [2] and [4].\n\nI do not see the choice of software listed anywhere. PyTorch with torchdiffeq, JAX with Diffrax, Julia with DifferentialEquations.jl, or something else?\n\nFinally, I do not see any discussion on the choice of tolerance, step size controller, etc. (Was this DP5(4) with a fixed step size? If so, what step size? Was it with an adaptive step size controller? If so, which controller and what tolerances?)\n\n**Weakness: experiments**\n\nI would suggest removing the MNIST experiments. MNIST is linearly separable at (as I recall) 93% accuracy. It is not a meaningful benchmark.\n\nMoreover, I do not think image classification is much of an interesting benchmark in the first place. I do not think anyone would claim that NDEs are the appropriate tool for performing image classification: standard ResNets/Transformers/etc. are much easier to use on this problem, for which the continuous-time formulation confers little advantage.\n\nFollowing [0, Section 1.2.1 and Section 2.2.1] I would strongly suggest targeting other applications (time series, scientific problems, etc.) instead.\n\nThe time series experiments seem very strange to me. There is no discussion on how the time series is provided as an input to the ODE. There are numerous approaches here, mostly based around either:\n\n1. As a controlled differential equation [5]\n2. Discrete updates, e.g. an ODE-RNN [Ref. Rubanova et al. 2019 from this paper]\n\nIn either case the ODE formulation must be changed quite substantially. If working in continuous time (e.g. as in [5]) then there is an additional question of how the discrete data is converted (by interpolation or otherwise) into continuous time [6]. Consulting the appendix it looks like specifically an ODE-RNN approach is used? As there has been substantial work on this topic then much more detail deserves to be in the main paper.\n\n**Weakness: various minor issues**\n\nFigure 1: This relationship between GD, HB and NAG is well-known, and does not need to be here.\n\nRemark 2: it is not necessary that specifically the Euler method be used. This remark should be reworded: \"If the Euler method is used, then ... [the rest of the remark follows]\".\n\nGNesterovNODE: I'm not sure the term \"generalised\" is the appropriate one here -- this seems to have been modified, not generalised.\n\nLong-term dependencies: I am used to think of \"long term dependencies\" as being in referenced to long time series, e.g. in the context of RNNs or neural CDEs. As the present paper concerns itself with non-time-series problems, I would suggesting focusing on the terminology of \"vanishing gradients\" etc. instead. (In terms of technical content, I did like this section of the paper.)\n\nRubanova et al. 2019 (the Latent ODE paper) appears in the reference list twice.\n\n**Commentary**\n\nI suspect there are natural extensions to this paper, studying neural DAEs more generally.\n\nThe fact that increasing $r$ increases both NFEs and model accuracy suggests to me that this may be due to discretisation error: greater NFEs produces a larger overall computation graph and thus increased model capacity. If the tolerances are shrunk small enough then I speculate that this improvement should vanish, as the model converges to the true solution of the ODE.\n\n**References**\n\n[0] On Neural Differential Equations, Kidger, Doctoral Thesis, University of Oxford 2021\n\n[1] \"Hey, that's not an ODE\": Faster ODE Adjoints via Seminorms, Kidger et al, ICML 2021\n\n[2] DifferentialEquations.jl, https://diffeq.sciml.ai/\n\n[3] Runge--Kutta pairs of order 5 (4) satisfying only the first column simplifying assumption, Tsitouras, Computers and Mathematics with Applications 2011\n\n[4] Diffrax, https://github.com/patrick-kidger/diffrax\n\n[5] Neural Controlled Differential Equations for Irregular Time Series, Kidger et al., NeurIPS 2020\n\n[6] Neural Controlled Differential Equations for Online Prediction Tasks, Morrill et al., arXiv:2106.11028", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WrwX3En5i-", "reviewer": "Reviewer_tT37", "summary": "The paper presents the novel class of neural ODEs called NesterovNODEs, where the ODE block solves the second-order ODE corresponding to the limit of the Nesterov accelerated gradient scheme. To avoid instability around $t=0$ in this second-order ODE, the authors propose the generalized version of NesterovNODEs, where the variable transformation eliminates this instability. The main benefit of this modification of the neural ODE approach is a significant reduction of the NFE needed to perform forward and backward passes. At the same time, the test quality of considered models is improved compared to other neural ODE models. This gain is shown point cloud separation, image classification, and sequence modeling problems.", "questions": "The following typos should be fixed in the revised version\n1) lines 151,153 - bold notation for $m(t)$\n2) line 151 - please provide explicit form for activation function $\\sigma$. \n3) line 314 $1/k^2$ \n\nAlso, please comment the following questions and remarks\n1) It is known that adjoint method to compute the gradients through the solution of ODE is unstable, see e.g. \n`Gholami, A., Keutzer, K., & Biros, G. (2019). Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint arXiv:1902.10298` How did you control the potential instabilities in the adjoint method for NesterovODE? Authors addressed the vanishing gradient problem, but what about exploding gradient problem?\n2) Please add more details about Figure 2: what step size and ODE solver is used for forward and backward passes?\n3) The role of $T - t$ in line 170 should be explicitly explained for the reader convenience\n4) Please specify the initial value in Eq 15 and 17. Otherwise, it is unclear where the integration is forward in time and where it is backward in time?  \n5) Please add the main conclusion from these plots in caption of Fig 3\n6) Please explicitly define $v$ as a row in Proposition 3\n7) Please fix legend in Figure 9 \n8) Please comment the difference between NesterovNode and GNesterovNode demonstrated in Figure 3. From this plot follows that HBNode is the best in terms of treating vanishing gradient problem. ", "limitations": "The missing limitations of this work is increasing dimension in three times (store $x(t), h(t)$ and $m(t)$) of the NesterovNODE compared with standard neural ODE architecture (only $x(t)$ is needed). Such increasing can make the proposed approach intractable for large dimension of hidden state, so discussion and probably experiments on the memory consumption should be added in the revised version.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) the presentation of the paper is clear and easy to follow\n2) the extensive computational experiments confirm expectations from the presented theoretical part\n3) the proposed method has theoretical explanations\n\nWeaknesses\n1) the used model in the image classification problem gives too low test accuracy. The papers \n\na) Zhang, T., Yao, Z., Gholami, A., Gonzalez, J. E., Keutzer, K., Mahoney, M. W., & Biros, G. (2019). ANODEV2: A coupled neural ODE framework. Advances in Neural Information Processing Systems, 32.\n\nb) Gusak, J., Markeeva, L., Daulbaev, T., Katrutsa, A., Cichocki, A., & Oseledets, I. (2020, April). Towards Understanding Normalization in Neural ODEs. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations.\n\nc) Daulbaev, T., Katrutsa, A., Markeeva, L., Gusak, J., Cichocki, A., & Oseledets, I. (2020). Interpolation technique to speed up gradients \npropagation in neural ODEs. Advances in Neural Information Processing Systems, 33, 16689-16700.\n\nshow the +80% test accuracy in the CIFAR10 with standard neural ODE model without any second-order modifications. \nSo the main concern is that the proposed approach is tested only on the continuous models that do not provide state-of-the-art accuracy for the considered problems.\n\n2) the original motivation for NAG is to solve optimization problem faster and authors provide clear illustration of such convergence in Figure 1. At the same time, the motivation of continuous models like neural ODEs is to fit hidden dynamics in the latent space to learn relevant features. From these motivations is not clear why the ODE that is continuous limit of NAG is a good choice for the ODE that should learn hidden dynamics in continuous model.\n\n3) Figure 3 contradicts the claim about absence of vanishing gradient in both NesterovNODE-RNN and GNesterovNODE-RNN since the plots for them demonstrate the different trends.    \n\n4) From Table 3 in Appendix follows that for the different considered methods (NODE, ANODE,NODE, etc) different models are tested. If so, we can not split effect of different architectures and effect of different method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1_cfihyrL66", "reviewer": "Reviewer_hkAe", "summary": "This paper takes Nesterov's Accelerated Gradient Method, and applies it to Neural ODEs. One can think of a Neural ODE as carrying out vanilla gradient descent where:\n\n$\\frac{dh}{dt} = -\\nabla_{h}F(h, t)$\n\nHeavy ball Neural ODEs carry out the continuous limit of heavy ball momentum accelerated gradient descent where:\n\n$\\frac{d^2h}{dt^2} = -\\gamma\\frac{dh}{dt} - \\nabla_{h}F(h, t)$\n\nFinally, this work takes the continuous limit of Nestervov accelerated gradient descent:\n\n$\\frac{d^2h}{dt^2} = -\\frac{3}{t}\\frac{dh}{dt} - \\nabla_{h}F(h, t)$\n\nThe work demonstrates that this style of second order differential equation converges to its equilibrium point significantly faster than heavy ball Neural ODEs and other variations of Neural ODE. In turn this requires far fewer function evaluations in the ODE solve, while maintaining performance.\n\nProofs are provided to show that the method will be resistant to vanishing gradients and the solution will require fewer function evaluations. Extensive experiments are done on a large collection of Neural ODE variants, demonstrating performance is just as good but the speed of the ODE solve is greater.", "questions": "- Does the Nesterov method only really make sense for end point style tasks? That is, ones where the trajectory doesn't matter, so not time-series.\n- Where does the oscillation in Figure 1 come from?\n- Remark 1 talks about $\\frac{k-1}{k-3}$ but in the previous paragraph and equation 7 we have $\\frac{k-1}{k+2}$. Which one is it? Does this affect any results?\n- Is applying the sigmoid function to stabilize Nesterov theoretically motivated, or more of an empirical trick to stabilize the ODE solve?\n- The analysis of vanishing gradients is great, however in the worst case it still seems that we can get $\\exp(-T)$ by the time we reach the start of the solve. It seems from the experiments this doesn't happen much. Has a similar analysis been done on other Neural ODE variants? To provide theory which supports the experiment in Figure 3\n- Line 142, by writing $h(t)$ as $k(t)x(t)$, the singularity has moved from the ODE to $k(t)$, how has this actually helped, without the sigmoid being used the singularity is still there, just moved to $k(t)$\n- Remark 2 talks about the Euler method to solve the ODE and Nesterov ODE. Is there a statement that can be made for adaptive step solvers? For examples a larger tolerance can be used for Nesterov over ODE?\n- Has any timing taken place, to see the wall clock advantage of Nesterov?", "limitations": "The paper addresses both the limitations and the negative societal impact briefly in the conclusion.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is very well written and well motivated. The experiments test the claims well. The theoretical findings are strong and show Nesterov and GNesterov are useful to reduce the number of NFE. The ablations are also very good.\n\nWeaknesses:\n\nThere aren't many weaknesses to this paper, it is overall quite simple which is a good thing, but overall means the contribution is incremental and not necessarily ground breaking. In my opinion this is not a bad thing. Specific weaknesses:\n\n- The experiments are conducted on problems that Neural ODEs aren't typically used for, (classification and point clouds). It would be more convincing to demonstrate this on continuous normalizing flow tasks.\n- The other main application of Neural ODEs is to time-series, and Nesterov does worse than SONODE, likely due to restricting the dynamics\n- The proposed method feels quite simple (seems like a theoretically motivated adaptation of Heavy Ball NODE)\n\nMinor Mistakes/points (do not affect quality of paper):\n\n- I find Generalized Nesterov to be a misleading term, I believe something like Stabilized Nesterov is more accurate, as a sigmoid is applied to part of the Nesterov method, rather than generalizing, it is more like adapting\n- On the MNIST task, GNesterov is not second best, Nesterov is. GNesterov appears to be second worst, just after Neural ODE (line 224 and table 1)\n- On the point cloud task, in my experience only vanilla Neural ODE has problem separating the clouds, all others are able to. The experiment has also been called $g$ in the ANODE paper and Nested Spheres in Dissecting Neural ODEs\n- Appendix line 515/eq 19: Has $a(T) = -I$ it should be $a(T) = \\frac{\\partial L}{\\partial z(T)}$ \n\nMinor Typos (do not affect quality of paper):\n\n- Line 25: \"in neural network\" should be \"in a neural network\"\n- Equations in the paper should ideally be in brackets, these can be manually put in, or done using eqref in latex\n- Line 83: It would make more sense to use $x_{k} = x_{k-1}...$ than $x_{t} = x_{t-1}...$ because $k$ is used in later equations for discrete changes (equation 7 for example)\n- Line 200 and 318: \"much fewer NFE\" should be changed for \"far fewer NFE\" or \"significantly fewer NFE\" or simply \"fewer NFE\"\n- $\\mathcal{O}(k^2)$ and $\\mathcal{O}(1/k^2)$ are used interchangeably in the paper to describe the convergence of Nesterov, it would be better to stay consistent\n- Appendix footnote 1: \"ANODDE\" should be \"ANODE\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept