Heavy Ball Neural Ordinary Differential Equations
Authors: Hedi Xia
Vai Suliafu
Hangjie Ji
Tan Minh Nguyen
Andrea Bertozzi
Stanley Osher
Bao Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural ODEs
sequential data modeling
long-term dependencies
efficiency

Abstract: We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference. HBNODEs have two properties that imply practical advantages over NODEs: (i) The adjoint state of an HBNODE also satisfies an HBNODE, accelerating both forward and backward ODE solvers, thus significantly reducing the number of function evaluations (NFEs) and improving the utility of the trained models. (ii) The spectrum of HBNODEs is well structured, enabling effective learning of long-term dependencies from complex sequential data. We verify the advantages of HBNODEs over NODEs on benchmark tasks, including image classification, learning complex dynamics, and sequential modeling. Our method requires remarkably fewer forward and backward NFEs, is more accurate, and learns long-term dependencies more effectively than the other ODE-based neural network models. Code is available at \url{https://github.com/hedixia/HeavyBallNODE}.

Heavy Ball Neural Ordinary Differential Equations
Hedi Xia' Department Mathematics University of California, Los Angeles
Vai Suliafu
Scientific Computing and Imaging (SCT) Institute University of Utah, Salt Lake City, UT, USA
Hangjie Ji, Tan M. Nguyen, Andrea L. Bertozzi, and Stanley J. Osher Department of Mathematics University of California, Los Angeles
Bao Wang
Department of Mathematics
Scientific Computing and Imaging (SCI) Institute University of Utah, Salt Lake City, UT, USA
Abstract
We propose heavy ball neural ordinary differential equations (HBNODES), lever- aging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference. HBNODEs have two properties that imply practical advantages over NODEs: (i) The adjoint state of an HBNODE also satisfies an HBNODE, accelerating both forward and backward ODE solvers, thus significantly reducing the number of function evaluations (NFEs) and improving the utility of the trained models. (ii) The spectrum of HBNODEs is well structured, enabling effective learning of long- terg-termmmm dependencies from complex sequential data. We verify the advantages of HBNODEs over NODEs on benchmark tasks, including image classification, learning complex dynamics, and sequential modeling. Our method requires remarkably fewer forward and backward NFEs, is more accurate, and learns long-term dependencies more fectively than the other ODE-based neural network models. Code is available at /stps://githubb httpa//.o..oeeddda//////aaaaaaaa
Introduction
Neural ordinary differential equations (NODEs) are family of continuous-depth machine learning (MI.) models whose forward and backward propagations rely solving an ODE and its adjoint equation 4 NODEs model the dynamics of hidden features h(t) R using an ODE, which is parametrized by neural network f((((t) tt 0)
IR with learnable parameters 0, i...
dh((t) f(((t),t,)).
(1)
dt
Starting from the input h(ta), NODEs obtain the output h(T) by solving 1 for to < < with the initial value h(to), using a black-box numerical ODE solver. The number of function evaluations (NFEs) that the black-box ODE solver requires in sin- gle forward pass is an analogue for the continuous-depth models 4 to the depth of networks in ResNets [16]. The loss between NODE prediction h(T) and the ground truth is denoted by L(h(T)); we update parameters using the following gradient
"Co-first author
t Please correspond to: wangbaonje @ gmail.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

60
dc(h(T))
d0 a(t)
(f(h(t),t,)) t,
00
dt,
NFF (ervard)
ME (bacAwordd
250
100
60
150
where a(t) == @c/0h(() is the adjoint state, which satisfies the following adjoint equation
0.0
2.5
7.5
0.0
2.5
5.0
7.5
Epoch
Epoch
da(t) -a(t) -a()) - (t) dt
(3)
Traiein time (410003
NODE
Oh
ANGDD
SONODE
60
NODES are flexible in learning from irregularly- sampled sequential data and particularly suitable
GHENODE
30
54
for learning complex dynamical systems 44] 42 [6
35 9 [2]], which can be trained by efficient algo- rithms 40 58] NODE-based continuous genera- tive models have computational advantages over the classical normalizing flows 4 15
59,.0
Spach
Epoch
Figure 1: Contrasting NODE, ANODE, SONODE, HBNODE, and GHBNODE for CIFAR10 10 classifica- tion in NFEs, training time, and test accuracy. (Toler- ance: 10 10-5 see Sec. 5.2 for experimental details.)
55 12]. NODEs
have also been generalized to neural stochastic dif- ferential equations, stochastic processes, and graph NODES [21] 28, 38, 49 (20 34). The drawback of NODES is also prominent. In many ML tasks, NODEs require very high NFEs in both training and inference, especially in high accuracy settings where a lower tolerance is needed. The NFEs increase rapidly with training; high NFEs reduce computational speed and aceuracy of NODEs and can lead blow-ups in the worst-case scenario As an illustration, we train NODEs for CIFARIO classification using the same model and experimental settings as in [T), except using a tolerance of 10- 5 Fig. shows both forward and backward NFEs and the training time of different
[5 10 29 35
based models: we see that NFEs and computational times increase very rapidly for NODE, ANODE [T], and SONODE [35] More results on the large NEE and degrading utility issues for different benchmark experiments are available Sec. 5 Another issue is that NODEs often fail to effectively learn long-term dependencies in sequential data [26]. as discussed in Sec.
1.1 Contribution
We propose heavy ball neural ODEs (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve NODE training and inference. At the core of HBNODE is replacing the first-order ODE (T) with a heavy ball ODE (HBODE), i.e., second-order ODE with appropriate damping term. HBNODEs have two theoretical properties that imply practical advantages over NODEs:
The adjoint equation used for training a HBNODE is also a HBNODE (see Prop. [ and Prop. 2 accelerating both forward and backward propagation, thus significantly reducing both forward and backward NFEs. The reduction in NFE using HBNODE over existing benchmark ODE-base models becomes more aggressive as the error tolerance of the ODE solvers decreases.
The spectrum of the HBODE is well-structured (see Prop. 4), alleviating the vanishing gradient issue back- proc-prppagttionn and enabling the model to effectively learn long-term dependencies from sequential data.
To mitigate the potential blow-up problem in training HBNODEs, further propose generalized HBNODEs (GHBNODES) by integrating skip connections 17 and gating mechanisms [9] into the HBNODE. See Sec. for details.
1.2
Organization
We organize the paper as follows: In Secs. and we present our motivation, algorithm, and analysis of HBNODEs and GHBNODES, respectively. We analyze the spectrum structure of the adjoint equation of HBNODES/GHBNODES in Sec. 4 which indicates that HBNODES/GHBNODES can learn long-term dependency effectively. We test the performance of HBNODES and GHBNODES on benchmark point cloud separation, image classification, learning dynamics, and sequential modeling in Sec. We diseuss more related work in Sec. ( followed by concluding remarks. Technical proofs and more experimental details are provided in the appendix.

---

Heavy Ball Neural Ordinary Differential Equations
2.1 Heavy ball ordinary differential equation
Classical momentum method, a.k.a., the heavy ball method, has achieved remarkable success in accelerating gradient descent [Р and has significantly improved the training of deep neural networks 46 As the continuous limit of the classical momentum method, heavy ball ODE (HBODE) has been studied in various settings and has been used analyze the acceleration phenomenon of the momentum methods. For the ease of reading and completeness, we derive the HBODE from the classical momentum method. Starting from initial points x. and æ²,, gradient descent with classical momentum searches minimum of the function F(æ:) through the following iteration
g+1 x vVF((*) + 3(x* =
(4)
where s > is the step size and 0 - 8 < is the momentum hyperparameter. For any fixed step size let m = (x**1 x= / /s, and let == 7vs, where is another hyperparameter.
Then we can rewrite 4 as
m k+1 = (1-
V F(x"); a=-1 = x² Vom k+1
(5)
Let + 0 in we obtain the following system of first-order ODEs,
dxx(t) m(t;; dt mm) VF(x(tt).
dm((t)
(6)
dt
This can be further rewritten as a second-orde heavy ball ODE (HBODE), which also models damped oscillator,
d<((()) da((!)
dt2 +y
dt = -f(x(t)))
In Appendix E.6 we compare the dynamics of HBODE 7 and the following ODE limit of the gradient descent (GD)
dæ F(r))
dt
In particular, we solve the ODEs 7 and (8) with F(x) defined as a Rosenbrock 41 or Beale [4] function. The comparisons show that HBODE can accelerate the dynamics of the ODE for gradient system, which motivates us to propose HBNODE to accelerate forward propagation of NODE.
2.2 Heavy ball neural ordinary differential equations
Similar to NODE, we parameterize F in 7 using neural network f(h(t), 1, 0), resulting in the following HBNODE with initial position (a(l)) and momentum m(ta) = dh (to).
dt) dt2
dh((t) +7 dt (h((t,,t,))
where 7 2 0 is the damping parameter, which can be set as a tunable or a learnable hyperparmater with positivity constraint. In the trainable case, we use c sigmoid(w) for a trainable E R and a fixed tunable upper bound (we set E = 1 below). According (6). HBNODE is equivalent to
dh(t) = mt); dm(() (t) + t))))),, dt
(10) Equation 9 (or equivalently, the system (TO)) defines the forward ODE for the HBNODE, and we can use either the first-order (Prop. 2 or the second-order (Prop. [ adjoint sensitivity method to update the parameter 0 35
dt
Proposition (Adjoint equation for HBNODE). The adjoint state a(t) = aC/Ōh(t) for the HIBNODE satisfies the following HBODE with the same damping parameter y as that in (
da((l) dt
da(l)
0S 0h
dt a())
(.)))
(11)
Remark 1. Note that we solve the adjoint equation T1 from time 1 = to 10 La in the backward propagation. By letting T T and b(() (T
T), we can rewrite as follows,
8b(T) db(T) = 0h
0f
-
(12)
2 dr
Therefore, the adjoint of the HBNODE is also a HBNODE and they have the same damping parameter.

---

We can also employ (T) and its adjoint for the forward and backward propagations, respectively. Proposition 2 (Adjoint equations for the first-order HBNODE system). The adjoint states an an :- /C/0h(t and am (t) == 0t/0m(t) for the first-order HBNODE system (T)) satisfy
dap(t) = Om (1) 0f (,)),,,0); di
dam (t) = - ann(1) + (1)(1)) dl
(13)
0h
Remark 2. Let 0m (t) dam (t)/dt, then am (t) and am(t) satisfies the following first-order
heavy ball ODE system
dam
dā (t) am)( amm(t)_am(t) 0f (h(t),t,)) (ā)
(t);
(14)
dt
dt
0h
Note that we solve this system backward in time in back-propagation. Moreover, we have an (1) 70m (t) 7am (t).
Similar to
35
we use the coupled first-order HBNODE system (T)) and its adjoint first-order
HBNODE system (3) for practical implementation, since the entangled representation permits faster computation 35 of the gradients of the coupled ODE systems.
3 Generalized Heavy Ball Neural Ordinary Differential Equations
In this section, we propose generalized version of HBNODE (GHBNODE), see (15). to mitigate the po- tential blow-up issue in training ODE- bosed models, In
10th
NODE
10a5
ANODE
experiments, we observe that (t) of ANODEs [10],
1002
SONODE
SONODES 135 and HBNODES (T) usually grows
10"
HANDDE
much faster than that of NODEs. The fast growth of
109
GHBNODE
h(t) can lead to finite-time blow up. As an illustration, compare the performance of NODE, ANODE, SON- ODE, HBNODE, and GHBNODE on the Silverbox task in 35 The goal of the task is to learn the voltage of an electronic circuit that resembles a Duffing oscil- lator, where the input voltage (t) is used to predict the output (t). Similar to the setting in (35). we first augment ANODE by dimension with O-augmentation and augment SONODE, HBNODE, and GHBNODE with a dense network. We use a simple dense layer to parameterize f for all five models, with an extra input term for (t For both HBNODE and GHBNODE, we set the damping parameter y to be sigmoid 3). For GHBNODE (5)) below, we set o--) to be the hardtanh function with bound [-5, 5] and { = l2). The detailed architecture can be found in Appendix E As shown in Fig. 2 compared to the vanilla NODE, the l norm of h(t)) grows much faster when a higher order NODE is used, which leads to blow-up during training. Similar issues arise in the time series experiments (see Sec. 5.44, where SONODE blows up during long term integration in time, and HBNODE suffers from the same issue with same initialization.
10a
Exact
10"
10
20
30
50
60
Figure 2: Contrasting (tt) for different
models. htt) in ANODE, SONODE, and HBNODE grows much faster than that in NODE. GHBNODE controls the growth of h(l)) effectively when / is large.
To alleviate the problem above, we propose the following generalized HBNODE
dh(t) (m((t);
dm(() dt (m(()) f(h(t),t,)) - <h(t),
(15)
dt
where (-) is nonlinear activation, which is set tanh in our experiments. The positive hyper- parameters 7, & > O are tunable or learnable. In the trainable case, we let e sigmoid(w) as in HBNODE, and E = softplus(x) to ensure that 7, 2 0. Here, we integrate two main ideas into the design of GHBNODE: (i) We incorporate the gating mechanism used in LSTM [99 and GRU [6] which can suppress the aggregation of mt): (ii) Following the idea of skip connection [17, we add the term (t(t) into the governing equation of mf(), which benefits training and generalization of GHBNODEs. Fig. 2 shows that GHBNODE can indeed control the growth of h(t) effectively.
Proposition (Adjoint equations for GHBNODES). The adjoint states (t) = @c/Oh((), (l) = 00/0m(l) for the GIIBNODE (15) satisfy the following firloowingffrstorrdee first-order ODE system
dan(t) am (1) 0S (h(1),,0) -ET) dam (t) an (())))) + 7a (1). (16)
Øt
0h
Øt
'Here, we exclude he term that appeared in the original Duffing oscillator model because including it would result in tinite time explosion.

---

Though the adjoint state of the GHBNODE (16) does not satisfy the exact heavy ball ODE, based on our empirical study, it also significantly reduces the backward NFEs.
4 Learning long-term dependencies Vanishing gradient
known that the vanishing and exploding gradients are two bottlenecks for training recurrent neural networks (RNNs) with long-term dependencies [2] 37 (see Appendix [ for brief review on the exploding and vanishing gradient issues in training RNNs). The exploding gradients issue can be
effectively resolved via gradient clipping, training loss regularization, etc [37 IT. Thus in practice
the vanishing gradient is the major issue for learning long-term dependencies 337. As the continuous analogue of RNN, NODES as well as their hybrid ODE-RNN models, may also suffer from vanishing in the adjoint state (t) == (t) [26]. When the vanishing gradient issue happens, a(t) goes
to 0 quickly as T increases, then dC/d0 in (2) will be independent of these a(t). We have the following expressions for the adjoint states of the NODE and HBNODE (see Appendix C for details):
For NODE, we have
Oc Oc Ohr Oc
exp {
S dh
0f (h(s), s, 0d)s.
(17)
oh; ohT Ōh,
öhr
For GHBNODE
from (3)
we can derive
Øhr
Shv
Ze
0C
BhT
Bmy
ac
Omer
ah
3mm
8c Dhr Um
exp
{T
8m -
ds
Uhs
T
M
(18)
Note that the matrix exponential is directly related to its eigenvalues. By Schur decomposition, there exists an orthogonal matrix Q and an upper triangular matrix U, where the diagonal entries of U are eigenvalues of Q ordered by their real parts, such that
-M QUQ = exp( M) = Q exp(UUQQ..
(19)
Let V = [ DC
Oc
Q, then (8) can be rewritten as
Ahr 0mr
8C. ac ohr
8C 8C
[ =
BC ac
Q exp = T exp
(20)
5h
0m.
Umey
āhe 3my
By taking the &g norm in (2 and dividing both sides by
ac BC
ll. we arrive at
ohr
vm
0C ac
T e
|00 exp(UU) Il = ||e exp(U))l, (21)
8h,
0m.
2
| [o TT2
|ol|
öher Ume
i.e., ||| ac
BC
|l2
lle explU 112
III
ac ac
112 where = w/
Dmn,
ūhe Jme
Proposition 4. The eigenvalues of - -M can be paired so that the sum of each pair equals (t - Tyy-
For given constant a > 0, we can group the upper triangular matrix (Up(U) as follows
exp(U) = [uXp(UZ) 0
P P..
exp(Uv)
(22)
where the diagonal of UL (Uv) contains eigenvalues of - -M that are no less (greater) than (t - T)a Then, we have |e exp(U)>22 2 lee exp(U where the vector e, denotes the first m columns of e with m be the number of columns of U.. By choosing 0 S 7 S 2a, for every pair of eigenvalues of -M there is at least one eigenvalue whose real part is no less than (t - Ta)a. Therefore, exp(Ur} } T'aa, and the dimension of Ur gradients by clipping the & norm of the adjoint states similar to that used for training RNNs.
decays at a rate at most
least N x N. We avoid exploding
In contrast, all eigenvalues of the matrix f Of /Ohds in (17) for NODE can be very positive or negative, resulting exploding or vanishing gradients. As an illustration, we consider the benchmark Walker2D kinematic simulation task that requires learning long-term dependencies effectively 26 3. We train ODE-RNN (42 and (G)
on this benchmark dataset, and the detailed experimental settings are provided in Sec. 5.4 Figure 4 plots ||ac/hh,ll 2| for ODE-RNN and [| aL/ @mm:]ll for (G)HBNDDE-RNN, showing that the adjoint state of ODE-RNN vanishes quickly, while that of (G)NBNNDE--NN does not vanish even when the gap between T and is very large.
"HBNODE can be seen special GHBNODE with { 0 and be the identity map.
5

---

ODE-RNN
HBNODE-ANN
GHBNODE-RNN
20.03
54-
54-
36
36
36
0.013
0.012
18
18
0.006
100 200
300
1000
200 400
0.0
Epoch
Epoch
Figure 3: Plot of the the E2-norm of the adjoint states for ODE- RNN and (G)HBNODE -ENND----NNN back- propagated from the last time stamp. The adjoint state of ODE-RNN vanishes quickly when the gap between the final time T intermediate time hecomes larger, while the adjoint states (G)HBNODE- decays much more slowly. This implies that (G)HBNODE-RNY is more effective in learning long-term dependency than ODE-RNN.
Experimental Results
In this section, we compare the performance of the proposed HBNODE and GHBNODE with existing ODF-based models, including NODE 4 ANODE [I0), and SONODE (35] on the benchmark point cloud separation, image classification, learning dynamical systems, and kinematic simulation. For all the experiments, size for each experiment are listed in Table and
Adam [25] the benchmark optimization solver (the learning rate and batch
as the numerical ODE solver. For HBNODE and GHBNODE, we set y sigmoid((), where 0 is trainable weight initialized as = 3. The network architecture used to parameterize (h((t), f, 0) for each experiment below are described in Appendix E All experiments conducted server with 2 NVIDIA Titan Xp GPUs.
Table The batch size and learning rate for different datasets.
Dataset
Point Cloud
MNIST
CIFARIO
Plane Vibration
Walker2D
Batch Size
50
64 0.001
64 0.001
64 0.0001
256
Leurning Rate
0.01
0.003
NOTE
ANGDE
SONODE
HBNGGE
CHINODE
200
300
52
1.0
150
230
250
300
Loss
Epochs
Figure 4: Comparison between NODE, ANODE, SONODE, HBNODE, and GHBNODE for two-dimensions
point cloud separation. HBNODE and GHBNODE converge better and require less NFEs both forward and backward propagation than the other benchmark models.
5.1 Point cloud separation
In this subsection, we consider the two-dimensional point cloud separation benchmark. A total of 120 points are sampled, in which 40 points are drawn uniformly from the circle ||||| < 0.5, and 80 points are drawn uniformly from the annulus 0.85 |||||| < 1.0. This experiment aims learn effective features to classify these two point clouds. Following [10], we use three-layer neural network to parameterize the right-hand side of each ODE-based model, integrate the ODE-based model from in = 0 to T 1, and pass the integration results to dense layer to generate the classification results. We set the size of hidden layers so that the models have similar sizes, and the number of parameters of NODE, ANODE, SONODE, HBNODE, and GHBNODE are 525, 567, 528, 568,

---

60
NODE ANCDE
50
-40
TO 501000000000000
SONODE
WIT 46
30
24
0.0
Eaoch
Epdes
Spach
Figure 5: Contrasting NODE 1.
ANODE [10). SONODE 35
HBNODE, and GHBNODE for MNIST
classification in NFE, training time, and test accuracy. (Tolerance: 10" ").
and 568, respectively. To avoid the effects of numerical eπror of the black-box ODE solver we set tolerance of ODE solver be 10-7 Figure 4 plots randomly selected evolution of the point cloud separation for each model; we also compare the forward and backward NFEs and the training loss of these models (100 independent runs). HBNODE and GHBNODE improve training as the training loss consistently goes to zero over different runs, while ANODE and SONODE often get stuck at local minima, and NODE cannot separate the point cloud since preserves the topology [0]]
5.2 Image classification
We compare the performance of HBNODE and GHBNODE with the existing ODE-based models on MNIST and CIFAR10 classification tasks using the same setting as in [10] We parameterize fh((t,, t, 0) using 3-layer convolutional network for each ODE-based model, and the total number of parameters for each model is listed in Table 2 For a given input image of the size x h x w, we first augment the number of channel from to + p with the augmentation dimension dependent on each method Moreover, for SONODE, HBNODE and GHBNODE, we further include velocity or momentum with the same shape the augmented state.
Table 2: The number of parameters for each models for image classification.
Model
NODE
ANODE
SONODE
HBNODE GHBNODE
#Params (MNIST)
85,315
85,462
86,179
85,931
85,235
#Params (CIFAR
173,611
172,452
171,635
172,916
172,916
NFEs. As shown in Figs. and [] the NFEs grow rapidly with training of the NODE, resulting in an increasingly complex model with reduced performance and the possibility of blow up. Input augmentation has been verified to effectively reduce the NFEs, as both ANODE and SONODE require fewer forward NFEs than NODE for the MNIST and CIFARIO classification. However, input augmentation is less effective in controlling their backward NFEs. HBNODE and GHBNODE require much fewer NFEs than the existing benchmarks, especially for backward NFEs. In practice, reducing NFEs implics reducing both training and inference time, as shown in Figs.
aadd] 3
Accuracy. We also compare the accuracy of different ODE-based models for MNIST and CIFARIO classification. As shown in Figs. [add HBNODE and GHBNODE have slightly better classification accuracy than the other three models; this resonates with the fact that less NFEs lead to simpler
models which generalize better [10 335.
NODE
ANODE
SONODE
HBNODE
GHBNODE
beckwwrr
800
800
600
600
600
10-a
500
500
500
500
500
400
400
400
400
400
300
300
300
300
300
10
300
200
200
200
200
100
100
100
9-
80 100
0g
80 100
80
100
0g
80
100
100
NFE Forward
NFE Forward
NFEE Forward
NFE Formard
NFE Formard
Figure 6: NFE vs. tolerance (shown in the colorbar) for training ODE-based models CIFARIO classification.
Both forward and backward NIES of HBNODE and GHBNODE grow much more slowly than that of NODE,
ANODE, and SONODE; especially the backward NFEs. As the tolerance decreases, the advantage of HIBNDDE and GHBNODE in reducing NFEs becomes more significant.
NFES vs. tolerance. We further study the NFEs for different ODE-based models under different tolerances of the ODE solver using the same approach as in 4 Figure 6 depicts the forward and backward NFEs for different models under different tolerances. We see that (i) both forward
'e set 5,5, 4, 4, 5/0, 10, 9, 9 MNIST/CIFAR10 for NODE, ANODE, SONODE, HBNODE, and GHBNODE, respectively.
7

---

and backward NFEs grow quickly when tolerance is decreased, and HBNODE and GHBNODE require much fewer NFEs than other models; (ii) under different tolerances. the backward NFEs of NODE, ANODE, and SONODE are much larger than the forward NFEs, and the difference becomes larger when the tolerance decreases. In contrast, the forward and backward NFEs of HBNODE and GHBNODE scale almost linearly with each other. This reflects that the advantage in NFEs of (G)HBNODE over the benchmarks become more significant when a smaller tolerance is used.
5.3 Learning dynamical systems from irregularly-samplee time series
In this subsection, we learn dynamical systems from experimental measurements. In particular, use the ODE-RNN framework [4] 42), with the recognition model being set to different ODE-based models, to study the vibration of an airplane dataset [33]. The dataset was acquired, from time 0 to 73627, by attaching a shaker underneath the right wing to provide input signals, and 5 attributes are recorded per time stamp; these attributes include voltage of input signal, force applied aircraft, and acceleration at 3 different spots of the airplane. We randomly take out 10% of the data to make the time series treegularl-samplle 25% as validation set, and the rest as test set. We divide each set into non-overlapping segments of consecutive 65 time stamps of the irregularl--sampled time series, with each input instance consisting of 64 time stamps of the irregularly-sampled time series, and we aim to forecast 8 consecutive time stamps starting from the last time stamp of the segment. The input fed through the the hybrid methods in a recurrent fashion; by changing the time duration of the last step of the ODE integration, we can forecast the output in the different time stamps. The output of the hybrid method is passed to single dense layer to generate the output time series. In our experiments, we compare different based models hybrid with RNNs. The ODE of each model is parametrized by a 3-layer network whereas the RNN is parametrized by a simple dense network; the total number of parameters for ODE-RNN, ANODE-RNN, SONODE RNN, HBNODE-RNE 14. 15, 15 augmented dimensions are 15, 15,986, 16,730, 16,649, 16.127, and 16, respectively. To avoid potential error due to the ODE solver, we use a tolerance of 10-7
We use the first 50% of data as our train set, the next
and GHBNODE-RNN with 16, 22,
In training those hybrid models, we regularize the models by penalizing the L2 distance between the RNN output and the values of the next time stamp. Due to the second-order natural of the underlying dynamics [35], ODE-RNN and ANODE-RI RNN learn the dynamics very poorly with much larger training and test losses than the other models even they take smaller NFEs. HBNODE-RNN and GHBNODE-RNN give better prediction than SONODE-RNN using less backward NFEs.
1.
2.0
1.5
3203
2695
2400
2000
1005
2604
WPPONN 10
Tranntn 1o0k
Text toos
10"
NODE ANDIE SCHOOE HERIODE
0.0
1000
2000
300%
GHSRODD
1603
-1.
1400
1200
420
1200
10
203
Time
Space
Escch
Epoch
Figure 7: Contrasting ODE-RNI ANODE-RNY SONODE RNN, HBNODE RNN, and GHBNODE-RNY
for learning vibrational dynamical system. Left most: The learned curves each model vs. ground truth (Time: <66 for training, 66-75 for testing).
5.4 Walker2D kinematic simulation
In this subsection, we evaluate the performance of HBNODE- RNN and GHBNODE- RNN on the Walker2D kinematic simulation task, which requires learning long-term dependency effectively [26]. The dataset D consists of a dynamical system from kinematic simulation of person walking from a pre-trained policy, aiming learn the kinematic simulation of the MuloCo physics engine (48) The dataset is irregularly-sampled with 10% of the data removed from the simulation. Each input consists of 64 time stamps fed though the the hybrid methods in a recurrent fashion, and the output is passed to single dense layer to generate the output time series. The goal is to provide an auto-regressive forecast so that the output time series is as close as the input sequence shifted one time stamp to the right. We compare ODE-RNN (with 7 augmentation). ANODE-RNN (with 7 ANODE style augmentation), HBNODE-RNN (with 7 augmentation), and GHBNODE-RNN (with 7 augmentation) d The RNN is parametrized by 3-layer network whereas the ODE is parametrized by simple dense
'Here, we do not compare with SONODE RNN since SONODE has some initialization problem on this dataset; the ODE solver encounters failure due to exponential growth over time.
8

---

network. The number of parameters of the above four models are 8,729, 8,815, 8,899, and 8.899, respectively. In Fig. & we compare the performance of the above four models on the Walker2D benchmark; HBNODE-RNN and GHBNODE-RNN not only require significantly less NFEs in both training (forward and backward) and in testing than ODE-RNN and ANODE-RNN, but also have much smaller training and test losses.
2603
3.5
35
NODE
g 2403
2503
32103
2400
2003
22003
2200
2209
2000
22.3
225
W1007
1100
1403
1660
1402
1400
1607
1.5
1.5
1200
100 230
305
430
59c
120%
100 25;
330
400 50c
120g
100
230
320
400
500
1.6
102
203
203
400
500
1.
103
203
203
402
500
Figure 8: Contrasting ODE-RNN, ANODE-RNN, SONODE-RNN, HBNODE-RNN, and GHBNODE-RNN for the Walker-2D kinematic simulation.
Related Work
Reducing NFEs in training NODEs. Several techniques have been developed reduce the NFEs for the forward solvers in NODEs, including weight decay [15. input augmentation [10], regularizing solvers and learning dynamics [22
23 B 36), high-order ODE B3), data control [29]. and depth-
variance [29]. HBNODEs can reduce both forward and backward NFES at the same time.
Second-order ODE accelerated dynamics. It has been noticed both optimization and sampling communities that second-orde ODEs with an appropriate damping term, e.g., the classical momentum and Nesterov's acceleration in discrete regime, can significantly accelerate the first-order gradient Also, these second-order ODEs have been discretized via some interesting numerical schemes to design fast optimization schemes, e.g., 44)
dynamics (gradient descent), e.g., [39 32
5
45 53
Learning long-term dependencies. Learning long-term dependency is of the most important goals for learning from sequential data. Most of the existing works focus on mitigating exploding vanishing gradient issues in training RNNs, e.g.,
54 22
30
18 47
Attention- -based
models are proposed for learning on sequential data concurrently with the effective accommodation of learning long-term dependency [50 [. Recently, NODEs have been integrated with long-short term memory model [19] to learn long-term dependency for urregularly-sample time series (26). HBNODEs directly enhance learning long-term dependency from sequential data.
Momentum in neural network design. As a line of orthogonal work, the momentum has also been studied designing neural network architecture, e.g., [7] [3], which can also help accelerate training and learn long-term dependencies. These techniques can be considered as changing the neural network f in (I). We leave the synergistic integration of adding momentum to f with our work changing the left-hand side of (T) as future work.
7 Concluding Remarks
We proposed HBNODEs to reduce the NFEs in solving both forward and backward ODEs, which also improve generalization performance over the existing benchmark models. Moreover, HBNODES alleviate vanishing gradients in training NODEs, making HBNODEs able to learn long-term depen- dency effectively from sequential data. In the optimization community, Nesterov acceleration [3] also famous algorithm for accelerating gradient descent, that achieves an optimal convergence rate for general convex optimization problems. The ODE counterpart of the Nesterov's 's accelcration corresponds to 9 with being replaced by time-dependent damping parameter, e.g., f/3 (45) or with restart [52] The adjoint equation of the Nesterov's 's ODE (45 is no longer a Nesterov's 's ODE. We notice that directly using Nesterov's ODE cannot improve the performance of the vanilla neural ODE. How to integrate Nesterov's 's ODE with neural ODE is an interesting future direction. Another interesting direction is connecting HBNODE with symplectic ODE-net
(57) through appropriate
change of variables.
8 Acknowledgement
This material is based on research sponsored by NSF grants DMS-192492 and DMS- 1952339,, DOE grant DE-SC0021 142, and ONR grant N00014- 18-1--8--122277 1-2527 and the MURI grant

---

References
[1] Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference Machine Learning, pages 1120-1128, 2016.
[2] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencics with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym, 2016. cite
[4] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 6572-6583, 2018.
[5] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In International conference on machine learning, pages 1683-1691, 2014.
[6] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. arXiv preprint arXiv:1406.1077. 2014.
[7] Talgat Daulbaev, Alexandr Katrutsa, Larisa Markeeva, Julia Gusak, Andrzej Cichocki, and Ivan Oseledets. Interpolation technique to speed up gradients propagation in neural odes. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16689-16700. Curran Associates, Inc.. 2020.
[8] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1800004 2018
[9] Jianzhun Du, Joseph Futoma, and Finale Doshi-Velez, Model-based reinforcement learning for semi-markov decision processes with neural odes. In H. Larochelle, Ranzato, R. Had- sell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19805-19816. Curran Associates, Inc., 2020.
[10] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In H. Wallach, H. Larochelle, A. Beygelzimer, FAlcéé-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[11] N. Benjamin Erichson, Omri Azencot, Alejandro Queinuga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. In International Conference Learning Representations, 2021.
[12] Chris Finlay, Joern- Hennikk Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ODE: the world of Jacobian and kinetic regularization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3154-3164, PMLR, 13-18 Jul 2020.
[13] Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. Steer Simple temporal regularization for neural ode. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14831-1484 Curran Associates, Inc., 2020.
[14] Amilcar dos Santos Gonçalves. A Version of Beale's Method Avoiding the Free- Variables. In Proceedings of the 1971 26th Annual Conference, ACM '71, page 433 New York, NY, USA, 1971. Association for Computing Machinery.
[15] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable re- versible generative models with free-form continuous dynamics. International Conference on Learning Representations, 2019.
10

---

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pages 630-645. Springer, 2016.
[18] Kyle Helfrich, Devin Willmott, and Qiang Yc. Orthogonal recurrent neural networks with scaled Cayley transform. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1969-1977 1978, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[19] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 98):1735- 1997.
[20] Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from irreuularl-samppled partial observations. In Advances in Neural Information Processing Systems, 2020.
[21] Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d' d'Alché-Bus Blché-BBuc E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[22] Li Jing. Yichen Shen, Tena Dubcek, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, and Marin Soljačić. Tunable efficient unitary neural networks (eunn) and their application to rns.. Proceedings of the 34th International Conference on Muchine Learning- Volume 70, JMLR.
pages 1733-1741. JMLR. org, 2017.
[23] Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learning differen- tial equations that are easy to solve. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4370 4380. Curran Associates, Inc., 2020.
[24] Patrick Kidger, James Morrill, James Foster, and Terry J. Lyons. Neural controlled differential equations for irregular time series. In NeurlPS, 2020.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[26] Mathias Lechner and Ramin Hasani. Learning long-term dependencies in irregularly-sampled time series. arXiv preprint arXiv:2006. 04418, 2020.
[27] Huan Li, Yibo Yang, Dongmin Chen, and Zhouchen Lin. Optimization algorithm inspired deep neural network structure design. arXiv preprint
2018.
[28] Xuechen Li, Ting-Kan Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference Artificial Intelligence and Statis- tics, volume 108 of Proceedings of Machine Learning Research, pages 3870-3882 PMLR, 26 Aug 2020.
[29] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asma. Dissect- ing neural odes. In 34th Conference on Neural Information Processing Systems, NeurIPS 2020. The Neural Information Processing Systems, 2020.
[30] Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey, Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. In Proceedings of the 34th International Conference on Machine Baarning-Volumi 70, pages 2401-2409. JMLR. org, 2017.
[31] Thomas Moreau and Joan Bruna. Understanding the learned iterative soft thresholding algorithm with matrix factorization. arXiv preprint arXiv: (00::::000000000 038,, 2017.
[32] Yurii E Nesterov. method for solving the convex programming problem with convergence rate (1/k" 2). In Dokl. Akad. Nauk Sssr, volume 269, pages 543-547, 1983.
11

---

[33] Jean-Philippe Noël and M Schoukens. F-16 aircraft benchmark based on ground vibration test data. In 2017 Workshop on Nonlinear System Identification Benchmarks, pages 19-23, 2017.
[34] Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural [ode) processes, In International Conference on Learning Representations, 2021.
[35] Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski. and Pietro Liò. On second order behaviour in augmented neural odes. In Advances in Neural Information Processing Systems, 2020.
[36] Avik Pal, Yingbo Ma, Viral Shah, and Christopher V Rackauckas. Opening the blackbox: Accelerating neural differential equations by regularizing internal solver heuristics. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8325-8335. PMLR, 18-24 Jul 2021.
[37] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310-1318, 2013.
[38] Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph neural ordinary differential equations. arXiv preprint 191.02211. 2019.
[39] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
[40] Alessio Quaglino, Marco Gallieri, Jonathan Masci, and Jan Koutník. Snode: Spectral dis- cretization of neural odes for system identification. In International Conference on Learning Representations, 2020.
[41] H. H. Rosenbrock. An Automatic Method for Finding the Greatest or Least Value of a Function. The Computer Journal, 3(3):155-184, 01 1960.
[42] Yulia Rubanova, Ricky T. Q Chen, and David K Duvenaud. Latent ordinary differential equations for irreuularl d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
time series. In H. Wallach, H. Larochelle, A. Beygelzimer,
[43] Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Momentum residual neural networks. arXiv preprint arXiv:2102.07877 2021.
[44] Bin Shi, Simon S Du, Weijie Su, and Michael Jordan. Acceleration via symplectic discretiza- tion of high-resolution differential equations. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[45] Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nes- terov's 's accelerated gradient method: Theory and insights. In Advances in Neural Information Processing Systems, pages 2510-2518, 2014.
[46] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International Conference on Machine Learning, pages 1139-1147, 2013.
[47] Tan M. Nguyen and Richard G. Baraniuk and Andrea L. Bertozzi and Stanley J. Osher and Bao Wang. MomentumRNN: Integrating momentum into recurrent neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 9154-9164, 2020.
[48] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1926-5033, 2012.
[49] Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.0m 0980099000008,...00 2019.
12

---

[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[51] Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learn- ing recurrent networks with long term dependencies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3570-3578. JMLR. org, 2017.
[52] Bao Wang, Tan M Nguyen, Andrea L Bertozzi, Richard G Baraniuk, and Stanley Osher. Scheduled restart momentum for accelerated stochastic gradient descent. arXiv preprini arXiv:2002. 10583, 2020.
[53] Ashia C. Wilson, Benjamin Recht, and Michael Jordan. A Lyapunov Analysis of Momentum Methods in Optimization. arXiv preprint Arxiv:1611. 1.1.0666.. 2018.
[54] Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems, pages 4880-4888, 2016.
[55] Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. ODE2VAE:
Deep generative second order ODEs with Bayesian neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[56] Tianjun Zhang, Zhewei Yao, Amir Gholami, Joseph Gonzalez. Kurt Keutzer, Michael W Mahoney, and George Biros. ANODEV2: A Coupled Neural ODE Framework. In H. Wallach, H. Larochelle. A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[57] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. International Conference on Learning Representations, 2020.
[58] Juntang Zhuang, Nicha C Dvorek,, sekhar tatikonda, and James Duncan. (MALI): A memory efficient and reverse accurate integrator for neural (ode]s. In International Conference on Learning Representations, 2021.
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.1.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4.1.
(b) Did you include complete proofs of all theoretical results? [Yes] See Supplementary Materials
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "vUovIGzg8-a", "reviewer": "Reviewer_UXkx", "summary": "The authors propose a new neural ordinary differential equation (NODE) model that is inspired by a second order ODE $\\ddot{x} + a \\dot{x} + b \\nabla F(x)=0$. The discretization of this ODE leads to a two step procedure known as heavy ball method. Using this idea to model a NODE has the advantage that the corresponding adjoint equation takes a form that reduces the number of function evaluations during the forward and backward pass. Further, the authors show that the spectrum of the model is well-structured which in turn mitigates the vanishing gradient problem. In addition, they formulate a model that integrates skip connections and gating mechanisms. The empirical results demonstrate the favorable performance of the proposed models as compared to other NODEs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of this work is limited, since the idea of second order neural ODEs is not new. Also, I do not agree that the work of [45] and others is orthogonal to your work (maybe you can elaborate on this). Further, you miss to discuss the work by [*1] who propose a heavy ball inspired neural network architecture. This said, it would be fair to devote more room for discussing related work within the introduction and related work section, instead of just briefly mention some of the important related works somewhere at the end of the paper.\n\nThe main contribution of this paper is the analysis of the corresponding adjoint equation for this model, and to my best knowledge these results appear to be novel. The advantage of the adjoint equation is plausible and backed up by empirical results, which show a considerable advantage as compared to other neural ODEs. Hence, I consider the results as significant and relevant for being published. \n\nThe generalized heavy ball neural ODE looks a bit heuristic and the advantage is not exactly clear to me. The experiments do not show necessarily an advantage in terms of accuracy. Based on Figure 1, I would favor the simpler heavy ball model. Does the results in Sec. 4 apply to the generalized model?\n\nOverall, the paper appears to be very dense and thus a bit cumbersome to read. I suggest, that you move some of the details, e.g., Table 1 and 2 into the appendix. Also you probably can omit section 1.2. Personally, I would also move the discussion of the generalized heavy ball neural ODE into the appendix. In turn, this would give room to expend the introduction, better discuss related work, and provide more room for discussions and explanations.\n\n\nOther comments:\n\n* The statement that the exploding gradients issue can be effectively resolved via gradient clipping is a bit misleading. While it is true that the issue of exploding gradients can be resolved via clipping, this comes with the price of restricted expressivity. Hence, there is a lot of current research effort focusing on architectures that avoid vanishing and exploding gradients by design [*2,*3,*4]. Since, [*3,*4] are CT-RNNs, they can also handle irregular sampled time series data. I don't think that it is necessary to compare to these models, but I would suggest to slightly rephrase the discussion at beginning of Section 4.\n\n* I am slightly surprised about the results shown in Figure 5. The augmented ODE should be able to perfectly separate the two point clouds. Are the plots here from an early stage of training or is there another reason for the poor performance of the NODE, here? \n\n* The considered ODE models do not achieve state-of-the-art performance on CIFAR-10. For instance, ANODE-v2 [*5] and Statefull-ODEs [*6] achieve about 90% test accuracy on CIFAR-10, while having nearly the same parameter count as the models considered in the experiments. Is there any reason for why you consider only a very simple network architecture?\n\n\n[*1] Li, Huan, et al. \"Optimization algorithm inspired deep neural network structure design.\" Asian Conference on Machine Learning. PMLR, 2018.\n\n[*2] Lezcano-Casado, Mario, and David Martınez-Rubio. \"Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group.\" ICLR (2019).\n\n[*3] Rusch, T. Konstantin, and Siddhartha Mishra. \"Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies.\" ICLR (2021).\n\n[*4] Erichson, N. Benjamin, et al. \"Lipschitz Recurrent Neural Networks.\" ICLR (2021).\n\n[*5] Zhang, Tianjun, et al. \"Anodev2: A coupled neural ode evolution framework.\" NeurIPS (2019).\n\n[*6] Queiruga, Alejandro, et al. \"Compressing Deep ODE-Nets using Basis Function Expansions.\" arXiv preprint arXiv:2106.10820 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A detailed discussion of the limitation of the proposed models is missing (there is no Section 4.1), also there is no discussion of the potential negative societal impact. I am not qualified to judge whether there is any potential negative societal impact of this work, but since this is fundamental research, I assume that there is non foreseeable negative impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "XPcSHfJmt7p", "reviewer": "Reviewer_juca", "summary": "This work proposes a reparameterization of a Neural ODE (NODE) as a Heavy-Ball Neural ODE (HBNODE). This is inspired by Heavy-Ball Momentum, which is the continuous limit of classical momentum.\nThe authors make the following claims:\nThe adjoint of an HBNODE is also an HBNODE.\nHBNODE reduces forward and backward NFE. This is even more apparent at lower tolerances.\nThe spectrum of HBNODE is well-structured. This implies effective learning of long-range dependencies by avoiding vanishing gradients.\nHBNODEs are more accurate\nThe authors verify these claims on image classification, complex dynamics, and sequential modelling. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Describe the strengths of the work:\n\nThis work clearly describes Heavy-Ball Momentum and the definition of HBNODE. The authors clearly state the use of augmented the state of a neural ODE, where the states are coupled, to solve the second-order heavy-ball ODE. The authors then generalize this by adding some tuneable/learnable weights for the terms. They claim this reduces the tendency for the hidden state to blow-up during training. \nBackward NFEs are significantly reduced and this results in a reduction in training time too. \nThe authors demonstrate their approach on a wide variety of tasks. \n\nExplain the limitations of this work:\n\nIt’s unclear why the results are so much weaker on the irregularly-sampled time-series experiment. \nThere is little explanation given for why the forward NFEs are larger for the experiment in Section 5.3 compared to the baseline, but smaller for the backward NFEs.\nAre there reasons other than vanishing gradients that cause GHBNODES to perform better than other models? Does their parameterization simply make them more expressive for the given number of parameters?\n\nCorrectness:\n\nThe authors make an effort to provide a fair comparison to different baseline methods by matching the number of parameters.\nThe experiments are not repeated with different random seeds and averaged. Since some methods/baselines were sensitive to initialization, it could be helpful to see how strong these results are over several runs (e.g. varying random seed). \nIt’s not very clear from Figure 7 that (G)HBNODE has significant savings in NFE at loose tolerances.\n\nClarity:\n\nThe writing in the paper can be improved. In particular, Section 4 can be improved for clarity, and can be better connected to the rest of the text. If space is a concern, I think some of the math can be moved to the appendix. \nTypos:\nLine 272: fed through\n\nRelation to prior work: \n\nIt is clear how this work differs from prior work.\nA few suggestions on work related to Reducing NFEs for NODEs:\n\nOpening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics\nAvik Pal, Yingbo Ma, Viral Shah, Christopher Rackauckas\n\nLearning Differential Equations that are Easy to Solve\nJacob Kelly, Jesse Bettencourt, Matthew James Johnson, David Duvenaud\n\nReproducibility: \n\nHyperparameters are given for the experiments. Given that examples are not averaged over multiple runs and it’s unclear exactly how the models were tuned for performance, there may be some issues with reproducing the results precisely. \n\nAdditional feedback, comments, suggestions:\n\nIn Figure 3 it is unclear if all the models are achieving the same performance at the task. It should be made clear what the performance is of each of the methods and if it is related to the norm of the hidden state. For example, do models with larger hidden states diverge faster?\n\nSection 4 deals with the issue of vanishing gradients. There are no experiments which verify that vanishing gradients cause the inability to avoid vanishing gradients. The authors perform an analysis to justify regularizing the norm of the adjoint state as a method to mitigate vanishing gradients. The authors suggest clipping the norm of the adjoint state. They then have an experiment showing the GHBNODE models don’t have vanishing gradients. I think the experiment could be improved by explicitly measuring some of the quantities mentioned in the analysis in this section and seeing how they related to the norm of the adjoint state. Even a toy problem to make things very tractable could give useful insight. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ASUpjljpC_0", "reviewer": "Reviewer_epEB", "summary": "This is a very nice paper presenting a novel neural ODE with momentum. The idea behind the paper is solid, and the contribution is motivated by some known limitations of Neural ODEs. The authors not only provide us with compelling experimental results but also with a theoretical discussion and a lot of insights. I think the paper would benefit a lot the community.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is based on a simple idea: adding momentum often accelerates convergence. The authors apply this idea to neural ODEs and develop a solid theory around it: momentum is shown to help both theoretically (adjoint discussion) and experimentally. \n\nI never published in neural ODEs – so I am not an expert – but I feel confident in recommending acceptance. The paper is totally in line with the literature (also experiment-wise) and is very well written and organized. I also did not spot any typo.\n\nI have 2 questions for the authors\n\n1) is it possible to translate this idea back to neural networks? i.e., what does a corresponding Resnet-like architecture look like?\n\n2) Have you thought about applying this idea to Hamiltonian neural ODEs? Similar to the paper:\n\nSymplectic ODE-Net: Learning Hamiltonian Dynamics with Control\nYaofeng Desmond Zhong, Biswadip Dey, Amit Chakraborty", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerning limitations, no foreseeable societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a reparameterization of a Neural ODE (NODE) as a Heavy-Ball Neural ODE (HBNODE) in order to handle the notoriously difficult problem of long-range dependency modeling. Proposed approach is elegant and novel. The authors provided also very comprehensive empirical evaluation showcasing the effectiveness of their method and addressed *in detail* all reviewers' comments.
Decision: Accept (Poster)