Continuous Depth Recurrent Neural Differential Equations
Authors: Srinivas Anumasa
Geetakrishnasai Gunapati
P. K. Srijith
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural ordinary differential equations
recurrent neural networks
sequence data

Abstract: Recurrent neural networks (RNNs) have brought a lot of advancements in sequence labeling tasks and sequence data. However, their effectiveness is limited  when the observations in the sequence are irregularly sampled, where the observations arrive at irregular time intervals. To address this, continuous time variants of the RNNs  were introduced based on neural  ordinary differential equations (NODE). They  learn a better representation of the data using the continuous transformation of hidden states over time, taking into account the time interval between the observations. However, they are still limited in their capability as they use the discrete transformations and discrete number of layers (depth) over an  input in the sequence to produce the output observation. We intend to address this limitation by proposing  RNNs based on  differential equations which model continuous  transformations over depth and time to predict an output for a given input in the sequence. Specifically, we propose continuous depth recurrent neural  differential equations (CDR-NDE) which generalizes  RNN models by continuously evolving the hidden states in both  the temporal and depth dimensions. CDR-NDE considers two separate differential equations over each of these dimensions and models the evolution in  the temporal and depth directions alternatively. We also propose the CDR-NDE-heat model based on partial differential equations which treats the computation of hidden states as solving a heat equation over time.  We demonstrate the effectiveness of the proposed models by comparing against the  state-of-the-art RNN models on  real world sequence modeling problems and data sets.

Under review as a conference paper at ICLR 2023
CONTINUOUS DEPTH RECURRENT NEURAL DIFFEREN- TIAL EQUATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Recurrent neural networks (RNNs) have brought a lot of advancements in sequence labeling tasks and sequence data. However, their effectiveness is limited when the observations in the sequence are irregularly sampled, where the observations arrive at irregular time intervals. To address this, continuous time variants of the RNNs were introduced based on neural ordinary differential equations (NODE). They learn a better representation of the data using the continuous transformation of hidden states over time, taking into account the time interval between the observations. However, they are still limited in their capability as they use the discrete transformations and fixed discrete number of layers (depth) over input in the sequence to produce the output observation. We intend to address this limitation by proposing RNNs based on differential equations which model continuous transformations over both depth and time to predict an output for given input in the sequence. Specifically, we propose continuous depth recurrent neural differential equations (CDR-NDE) which generalizes RNN models by continuously evolving the hidden states both the temporal and depth dimensions. CDR-NDE considers two separate differential equations over each of these dimensions and models the evolution in the temporal and depth directions alternatively. We also propose the EDR-NDE-hcal model based partial differential equations which treats the computation hidden states as solving a heat equation over time. We demonstrate the effectiveness of the proposed models by comparing against the state-of- sfate-o-thhe-ae ff-th--an art RNN models on real world sequence labeling problems and data.
INTRODUCTION
Deep learning models such as ResNets (He et al., 2016 have brought a lot of advances in many real world computer vision applications (Ren et al. 2017 He al.) 2020 Wang et al. 2019). They managed to achieve a good generalization performance by addressing the vanishing gradient problem in deep learning using skip connections. Recently, it shown that the transformation of hidden representations in the ResNet block is similar to the Euler numerical method (Lu et al. 2018 Haber & Ruthotto 2017 for solving ordinary differential equations (ODE) with constant step size. This observation has led to the inception of new deep learning architectures based on differential equations such as neural ODE (NODE) (Chen et al., (0188). NODE performs continuous transformation of hidden representation by treating Resnet operations as an ODE parameterized by neural network and solving the ODE using numerical methods such Euler method and Dopri-5 (Kimura 2009). NODE automated the model selection (depth estimation). is parameter efficient and is robust towards adversarial attacks than a ResNet with similar architecture (Hanshu et al. 2019).
Recurrent neural networks and its variants such as long short term memory (LSTM) (Hochreiter Schmidhuber 1997) and gated recurrent units (GRU) (Cho al. 2014) were successful and effective in modeling time-series and sequence data. However, RNN models were not effective for irregularly sampled time-series data (Rubanova et al. (2019)), where the observations are measured at irregular intervals of time. ODE-RNN (Rubanova et al. (019b) modeled hidden state transformations across time using a NODE, where the transformations of hidden representations depended on the time-gan gap between the arrivals and this led to a better representation of hidden state. This addressed the drawbacks of the RNN models which performs a single transformation of the hidden representation the observation times irrespective of the time interval. Such continuous recurrent models such as

---

Under review as a conference paper at ICLR 2023
GRU-ODE (De Brouwer al. 2019) and ODE-LSTM (Lechner & Hasani 2020) were proposed to learn better representation of irregular time series data. When applied to the sequence data with sequence of input-output elements along with their time of occurrences, these models obtain the temporal evolution of hidden states using a neural ODE. At an observation time, this is then combined with the input at that time and discrete number of transformations is applied using a feed-forward neural network to obtain the final hidden representation. This final hidden representation is then used to produce the desired output. Though these models evolve continuously over time, they use fixed discrete transformations over depth.
There are several real world sequence labelling problems where the sequences could be of different complexities or the input elements in the sequence could be of different complexities. For instance, consider the problem of social media post classification where different posts arrive at irregular time intervals. The posts could have varying characteristics with some posts containing only text while some contains both text and image. It would be beneficial to have a recurrent neural network model which would consider the complexities of the input in a sequence by having varying number of transformations for different inputs. In this work, we propose continuous depth recurrent neural differential equation (CDR-NDE) models which generalize the recurrent NODE models to have continuous transformation over depth in addition to the time. Continuous depth allows flexibility in modeling sequence data, with different depths over the elements in the sequence well different sequences. Combining this with the continuous time transformation as in recurrent neural ODE allows greater modeling capability for irregularly sampled sequence data.
The proposed continuous depth recurrent neural differential equations (CDR-NDE) model the evolu- tion of the hidden states simultaneously in both the temporal and depth dimensions using differential equations. Continuous transformation of hidden states is modeled as a differential equation with two independent variables, one in the temporal and the other in the depth direction. We also aim model the evolution of the hidden states using partial differential equation (PDE) based on the DD-heat equation, leading to the -DE-NDE- model. Heat equation a second order partial differential equation, which models the flow of heat across the rod over time. The proposed EDR-NDE--eat model considers the transformation of hidden states across depth and time using a non-homogeneous heat equation. An advantage is that it is capable of considering the information from the future along with the past in sequence labeling tasks. We exploit the structure in the CDR-NDE-heat model and PDE solvers to develop an efficient way obtain the hidden states where all the hidden states at a particular depth can be computed simultaneously. We evaluate the performance of our proposed models on real-world datasets such as person activity recognition (Asuncion & Newman, 2007) and Walker2d kinematic simulation data (Lechner & Hasani 2020). Throguh experiments, we show that the proposed continuous depth recurrent neural differential equation models outperformed the state-of-thc-at recurrent neural networks in all these tasks.
2 RELATED WORK
RNN models such as LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al. 2014) are the primary choice to fit high-dimensional time-series and sequence data. For irregular time-series data, traditional LSTM and GRU models are less effective they do not consider the varying inter-arrival times. To address the problem of fitting irregular time-series data, the standard approach is the augmented-LSTM which augments the elapsed time with input data. In GRU-D (Che al.) 2018) and RNN-Decay (Rubanova et al.] 2019b), the computed hidden state is the hidden state multiplied by a decay term proportional to the elapsed time. In other variants such as CT-GRU (Mozer et al. 2017).CT-
RNN (Funahashi & Nakamura 1993), ODE-RNN (Rubanova et al. 20196), GRU-ODE (De Brouwer al. 2019), ODE-LSTM (Lechner& & Hasani 2020) and Jump-CNi (Chen et al.) 2020), the hidden
state is computed as a continuous transformation of intermediate hidden states. CT-LSTM (Mei &
Fisner 7017a) combines both LSTM and continuous time neural Hawkes process to model continuous
transformation of hidden states. Two alternative states are computed at each time-step and the final state is an interpolated value of these hidden states, where the interpolation depends on the elapsed time. Phased-LSTM (Neil et al.) 2016) models irregularly sampled data using an additional time gate. The updates to the cell state and hidden state only happen when the time gate is open. This time gate allows for the updates to happen at irregular intervals. Phased LSTM reduces the memory decay as the updates only happen in a small time when the time gate is open. ODE-RNN (Rubanova et al.) 2019b) used neural ordinary differential equations over time to model the evolution of the hidden
2

---

Under review as a conference paper at ICLR 2023
states. The next hidden state is obtained solution to a NODE, and depends the time interval between two consecutive observations. GRU-ODE (De Brouwer et al. 2019) derived a NODE over time and hidden states using the GRU operations and consequently could avoid the vanishing gradient problem in ODE-RNN. Similarly, ODE-LSTM (Lechner & Hasani 2020) addressed the vanishing gradient problem in ODE-RNN by considering the L.TTM cell and memory cell while the output state is modeled using neural ODE to account for irregular observations. However, all these models only considered continuous evolution of hidden states over the temporal dimension. In our work, we aim to develop models which consider continuous evolution of the hidden states over depth as well as temporal dimensions.
Recently, there are some works which used deep neural networks to solve the partial differential equations (PDE) (also known as neural PDEs or physies informed neural networks) (Zubov et al. 2021 Brandstetter et al., 202T: Hu et al. 2020). (Hu et al.) 2020) showed that LSTM based RNNs can efficiently find the solutions to multidimensional PDEs without knowing the specific form of PDE. On the other hand, very few works used PDEs to model DNN architectures for solving problems from any domain. (Ruthotto & Haber 2018) used PDE to design Resnet architectures and convolutional neural networks (CNNs) such as Parabolic CNN and hyperbolic CNN by changing the ODE update dynamics to different PDE update dynamics. For instance, hyperbolic CNN can be obtained with second order dynamics. They showed that even the PDE CNNs with modest architectures achieve similar performance to the larger networks with considerably large numbers of parameters. Unlike the prior works combining neural networks and PDEs. our aim is to solve sequence labeling problems by developing flexible RNN based architectures considering the PDE based models and solutions.
BACKGROUND
3.1 PROBLEM DEFINITION
We consider the sequence labeling problem with sequence length of K, and denote the input-output pairs in the sequence as {x., ytl and the elements in the sequence are irregularly sampled at observation times E R+ We assume the input element in the sequence to be D dimensional, X; E RD and the corresponding output yr depends on the problem, discrete it is classification or continuous if it is regression. The aim is to learn a model f(,, ()) which could predict the output y: considering the input X,, and dependence on other elements in the sequence.
3.2 GATED RECURRENT UNIT
Recurrent neural networks (RNNs) are well suited to model the sequence data. They make use of the recurrent connections to remember the information until the previous time step, and combine it with the current input to predict the output. Standard RNNs suffer from the vanishing gradient problem due to which it forgets long term dependencies among the sequence elements. This was overcome with the help of long short term memory (LSTM) (Hochreiter & Schmidhuber (197) and gated recurrent units (GRUs) (Cho al. (0044). In our work we consider the basic RNN block to be GRU. In GRU, computation of hidden state and output at any time step i involves the following transformations.
r, oWW, + U, b,), z, =o + U-hh-1 + bb))
g tW, + (n @ h + bb)
Where rr,Z,.g are the reset gate, update gate and update vector respectively for the GRU. The hidden state h, in GRU is given by.
h. z, h + (1 ze) gr
As we can see, GRUs and RNNs in general do not consider the exact times or time interval between the observations. The same operations are applied irrespective of the time gap between observations. This can limit the capability of these models for irregularly sampled time series data.
GRUs can be extended consider the irregularity in the time series data by developing a continuous GRU variant. A continuous GRU, GRU-ODE (De Brouwer et al. (2019), can be obtained by adding and subtracting the hidden state both sides of equation? The computation of hidden states then
3

---

Under review as a conference paper at ICLR 2023
v(1)
y(2)
y(3)
v(4)
T
9
h(t)1,))) h(t.t)
h(t,t-11)
0
(K1,8i)
(x2,22)
(x3,
(x4,t4)
(xs,ts)
(xi,t)
Figure 1: Shows the computation of hidd en states using the CDR-NDE model. As we can see for different observations, the number of intermediate hidden states in the evolution along the depth is different. The index (t - 1, (') points to the hidden state of immediate left vertical evolution. The index (t, 1' 1) points the hidden state just below the current vertical evolution.
becomes equivalent to solving an ODE given in equation 15
dh, dt
Ah. h, h1-1 2, h +
ge h
= (1
2z) ① (g:
h₂)
(3)
3.3 RECURRENT NEURAL ORDINARY DIFFERENTIAL EQUATIONS
In ODE-RNN (Rubanova et al. 201 ODE-LSTM (Lechner & Hasani 2020), and ODE- GRU (De Brouwer et al. 2019), the hidden state h, holds the summary of past observations and evolves continuously between the observations considering the time interval. When there is observation, the hidden state h, changes abruptly (Chen et al., 2020) to consider it. Given initial state hp, let the function fa() models the continuous transformation of hidden state and function gn model instantancous change in the hidden state at the new observation. The prediction y is computed by using function on O. which is then used to compute the loss (cross entropy loss for classification). Both the functions 9t. () and o () are typically standard feed-forward neural networks with a discrete
number of layers. In the case of GRU-ODE, the function (i takes the form given the right hand side of equation T5 In general, the recurrent NODE models can be represented using the following
system.
dh; dt
= (he); lim hite 9h Xt), Vi = =f= <0
(4)
4 CONTINUOUS DEPTH RECURRENT NEURAL DIFFERENTIAL EQUATIONS
We propose continuous depth recurrent neural differential equations (CDR-NDE) aiming to overcome the drawbacks of the recurrent NODEs in modeling the sequence data. As already discussed. recurrent NODE models bring abrupt changes in the hidden state at the observation times using the standard neural network transformation gh (). when it considers the input X, time f. We aim to develop RNN models capable of continuous transformations over depth in addition to the temporal dimension. Such models will help in processing inputs with varying complexities. aid model selection (ín choosing the number of layers in a RNN model) and reduce the number of parameters (as every layer shares the same parameters as in NODE). Moreover, we hypothesize that such continuous transformation over depth will also aid in learning better hidden representations as it is not limited by predefined number of layers as in standard RNNs. We propose two models, CDR-NDE and CDR-NDE heat model. Both the models generalize the GRU-ODE model by continuously evolving in both the temporal and depth directions. The CDR-NDF-heet model is formulated based on a partial differential equation (I (1-dimensional heat equation) and enables faster computation of hidden states even when using adaptive step numerical method like Dopri5.
The hidden states of CDR-NDE evolves in depth direction (denoted by vertical axis e in Figure while also evolving in the temporal direction (denoted by horizontal axis t Figure TT. As the hidden state evolves in both directions, represent the hidden state as h(g.)) As shown in Figure [] during the computation a hidden state (e.e)) it requires hidden states that are immediately the left in the temporal dimension h(-1.1")
need tuple (t. to uniquely identify any hidden state and
4

---

Under review as a conference paper at ICLR 2023
and below in the depth direction -1- The evolution of the hidden state (((,e)) in the CDR-NDE is governed by the following differential equations.
@h(.,e) Ōt
Dh(a.") Ih (he,,)) (t ((0-),))))
(h(a)) -1)) y: (h(,,,)) A(1,0)=x Vi L........ K
(5)
δt
(6)
where T" is the maximum depth. We observe that the changes in the hidden state in the horizontal (time) direction depends on the hidden states at a depth below while changes in the hidden states in the vertical (depth) direction depends on the hidden states at the previous time. The derivation and exact expression used to define the functions fa() and 9N () obtained as follows. We consider the evolution in the horizontal direction to follow the GRU-ODE model but with an added skip- connection in the vertical direction. Though fa() can be any function as in ODE-RNN, GRU-ODE to avoid vanishing gradient problems the temporal direction (De Brouwer et al. 2019). In discrete setup, the expression used to compute the hidden state h(t,e") after adding the skip connection in the vertical direction can be written as
followed
((,e)) = ((1,1) @ h(r-1.)) + (1 - (t,et) & B(t.e") + h(g,0-1) By subtracting h 1,2)) both sides, haye) h(--1,t") ((te)) @ hi- + (1 z(t,e)) g(te)) + h(r,e-1) ha-1
obtain the difference equation as
(8)
Consequently, the differential equation governing the flow in the temporal (horizontal) direction is
ahea 2(2,0)) ① h(a + (1 z(t,e)) @ B(t,e) + ((c,011)) - ((c,0))
(9)
Øt
where
Z(a, = (W. + U,h(,, + bz), B(t,))
+ (nar, O h..e) + by)
[(a,)) = oW,, + U,ba,e + by)
To derive the differential equation in the depth (vertical) direction t', Equation 7 difference equation by carrying the term h(t, to the left hand side.
can be written as
h(,ee) - h = ((,))) @ h((-1.)) + (1 ((,)))) @ ((L)')
(10)
Consequently, the differential equation governing the flow in the depth (vertical) direction is defined below and we can observe that it depends on the hidden states at the previous time.
Oba.. 2aa ② h(1-1.99) + - -(( 2 BBe.r) Or
(11)
where
((a. = a (Wh + U2h(a-11)) + bz), B(a,y) tanh(Wah + (narar @ (hg-1r)) + ba)
r(e. = (W, U,b(a-1,)) + b,)
We solve the differential equations equation 9 and equation I in two stages. In the first stage, CDR-NDE is solved in the horizontal direction until time (, for t' = O following equation 9 and be solved using differential equation solvers such as Euler method stage, for every time step evaluated on the t-axis during the first stage, hidden states are allowed to evolve in the vertical direction, i.e. along the f'-axis. Evolution vertical direction is done until time l' = 7" and can be solved using solvers such as Buler or Dopri5. We can observe that during this evolution, CDR-NDE model considers h(t-1.e") in computing h(r.") for any time and depth taking into account the dependencies in the sequence. Hence, computation of the hidden state h(..))) needs access to the hidden state h(r- and this requires performing an additional interpolation on the hidden states evaluated at time f in the case of adaptive solvers.
Dopri5. In the second
4.1 CDR-NDE BASED ON HEAT EQUATION
We propose another CDR-NDE model inspired by the partial differential equations and in particular the ID-heat diffusion equation CCnnoon 1984). Heat equation represents the evolution of heat over
5

---

Under review as a conference paper at ICLR 2023
11
(a)
sec
(b) p sec
(c) At > p sec
Figure 2: Evolution of temperature across (a) shows the initial state(temperature) of the rod, heat applied externally at different points. Over time, heat diffuses from hot region to cold region, (b) shows the state of the rod, after seconds. (c) Over time, the change in temperature comes to an equilibrium state, no change of temperature over time.
time rod. Consider rod of length which room temperature. Along the length of the rod, at different points, heat is applied externally into the rod. The temperatures applied at different points can different. Figure 2(a) provides visualization of the initial state(at '' 0), where the rod is at room temperature. At four different points, heat is applied externally with different temperature values. Heat starts to flow from hotter regions to colder regions and the points which are initially room temperature become hotter. In a finite amount of time, it reaches state where the change in temperature at any point in time is zero, which is called an equilibrium state. FigureZ((b) visualizes the intermediate state of temperatures across the rod after p seconds. Figure] visualizes the equilibrium state where the change in temperature across the rod is smooth.
We observe that the evolution of temperature in rod can be seen as equivalent the evolution of hidden states and applying temperature to the road can be considered equivalent to providing input data the sequence. The hidden states associated with input data smoothly change over the depth dimension t' reaching an equilibrium state, and finally leading to the output elements in the sequence. The heat equation motivates to construct a model capable of capturing interactions among different elements in the sequence, providing smooth hidden transformations with variable depth. We hypothesize that such models will be able to learn better representations depending on the input and improve the generalization performance.
The process of heat diffusion can be represented by a ID heat equation (Cannon 1984) which is homogeneous second order partial differential equation, @u(t(,) C' &Pu(t') where u(t', Z) is the temperature at point I on rod at time and C' is a constant( difftant(iffuusivitt). The proposed CDR-NDE-her model is based the non-homogeneous heat equation, variant of the homogeneous 1D heat equation. The temperature applied at a location 1 in the rod is equivalent to the datapoint x, at time As the temperature injected point affects the temperature around the rod neighborhood, the hidden states are affected by the observed data points in the neighborhood. The hidden state then evolves over the depth variable t' and reaches steady state following the heat diffusion model. The second order derivative with respect to location / in the heat equation (equivalently over time t for sequence labeling problems) allows one to consider the effect of neighbors around a point. For sequence modeling, this allows one to learn better representation by considering past and future hidden states across time I, similar to bi-directional RNNs considering past and future information.
The proposed model considers a non-homogeneous heat equation model which allows a better repre- sentation of hidden state during evolution by considering additional information on the interaction between the hidden states. In our case, we choose GRUcell which holds summary of the past observations to capture the interaction. The differential equation governing the evolution of the proposed CDR-NDE-heat model is defined as follows,
Th Ot
bh f(hav-1)) bi b(-),)))
(12)
0t
where f(h b())())))))) is the GRUCell operation, i.c. f(h(,e- ) (ttee)) O h ((r-1ee") + (1 ((t)")) ) O gtr,"). The evolution of hidden state as shown Equatior corresponds to non-hoomggeneus heat equation (Trong al.) 2005) with GRUCell capturing the interactions.
The heat equation can be solved numerically using methods like Emite-difference method(FDM) (Reck) tenwald 2004) and method of lines(Mol.) (Schiesser 2012). We can get better insights the

---

Under review as a conference paper at ICLR 2023
behaviour of the proposed CDR-NDE-heat model by writing the updates using the finite difference method. Using FDM, the hidden state is computed as follows,
h(,++Ap)) - hee.
Mo-A.,e) (..)))) + h(e+ f(b(ce- b (()
Dr
A
A Aaa)) 2h teleeryhhhatee
((.++A)) A
(13)
FDM divides the space of tt, into To compute the hidden state at depth 1' de, ,, it
finite
grids
as
shown
in
Figure
3
utilizes the hidden states computed for itself and
its immediate neighbors at the previous depth (h(t-,,.") ((r,')) (()) This helps to
capture dependence among the neighboring in- puts during evolution. A drawback of directly using FDM in solving the proposed CDR-NDE- heat model that is a slow process. doesn't 't exploit the GPU power as the computations are happening in a sequential order.
t' + A
GRU
t'
Denry BAtAae)
- A
For the proposed model, from the formulation to compute next hidden state in Equation and and Figure 3 we can observe that the hidden states at t' + Air only depends on the hidden states computed below t' + A,r. Hence, all the hidden states l' + Ae can be computed simultaneously
レ- A tatte t+
Figure 3: Pictorial representation of computing a hidden state ((,+++, as shown in Equation
13 The new hidden state depends the already have hidden states time l'. The computed hidden states in the lower layers. merical techniques based on Method of lines (MoL) (Schiesser 2012) good choice for such a scenario. MoL method typically discretizes and computes function values in one dimension, and then jointly evolves in the remaining dimension to compute all the function values. our approach, we first compute hidden states along the f-axis and then compute the hidden states along the f'-axis by posing as solution to the system of differential equations. The evolution the hidden states along the t'-axis defined by the ordinary differential equation (14), which is derived from Equation 13 @b(,r) Or
((--A.))) 2h + h(+t..))) +f +f(b(r))) hie
= -o (oe(hhees,.r h(r,"), (((+A.))))
A?
(14)
herr" = ODESOLVE(g); inviiill h(:0):: slarr_lime O, end_limee = T"))
The initial hidden states at t' = 0 (h are computed by solving an ODE along the f-axis
db (1 2(,,)) @ (g(.0) - h((,o))
(15)
One can select any numerical method for solving the system of ODEs. In the experiments, we evaluate the performance of the CDR-NDE-heat model using both Euler HEDR-NDE-hea((alee)) and Dopris LDDR-NDE-heea(Dopri)
methods. We can observe that CDR-NDE-heat model considers ((c+d,.)') in addition to h ((, in computing ((t,e") for any time t and depth t', taking into account more dependencies in the sequence.
After computing the hidden states at depth 7", predictions are made using a fully connected neural network, i.e. y; on (T)) This is then used compute loss cross-entropy for classification and root mean square error for regression problems. The parameters of the CDR-NDE models, learnt by minimizing the loss computed over all the observations in the sequence and over all the sequences. The computed loss is backpropagated using either adjoint method (Chen et al. 2018) automatic differentiation update the model parameters.
ie. weight parameters of the GRUCell,
5 EXPERIMENTS
To evaluate the performance of the proposed models, we conduct experiments on irregu- lar time series datasets like person activity recognition (Asuncion & Newman 2007) and
7

---

Under review as a conference paper at ICLR 2023
Table 1:
ODE solvers used for different RNODE models. For the CDR-NDE-Heat model using
Dopri5, the absolute and relative tolerance values are 1e-3 and le 33 respectively.
Model
ODE-Solver 4-th order Runge-Kutta
Time-step Ratio
CT-RNN
1/3
ODE-RNN
4-th order Runge-Kutta
1/3
GRU-ODE
Explicit Euler
1/4
ODE-LSTM
Explicit Euler
1/4
CDR-NDE
Explicit Euler
1/2
CDR-NDE-heat(Bi))
Explicit Euler
1/2
CDR-NDE-hhau(oppri))
Dopri5
Table 3: Column 2, shows the test accuracy (mean + std) of all the models trained the dataset Person activity recognition. Column 3 shows the test data Mean-square error (mean + std) of all the models trained on the datasets Walker2d Kinematic. For both the dataset, every model is trained for 5 times with different seeds. Model
Person Activity Recognition
Walker2d Kinematic
Test-Accuracy
Mean-Square Error(Test-data)
Augmented-LSTN, CT-RNN ODE-RNN ODE-LSTM
83.78 1 0.41
1.09 + 0.01
82.32 = 0.83
1,25 0.03
75.03 = 1.87
1.88 =0.05
83.77 + 0.58
0.91 0.02
CT-GRU
83.93+ 0.86
1.22 + 0.01
RNN-Decay
78.74 3.65
1.44 0.01
Bidirectional-RNN
82.86 + 1.17
1.09 #0.01
GRU-D
82.52 + 0.86
1.14 + 0.01
Phased-LSTM
83.34 0.59
1.10
GRU-ODE
82.80 0.61
1.08 0.01
CT-LSTM
83.42 + 0.69
1.03 +0.02
CDR-NDE
87.54 = 0.34
0.97 = 0.04
CDR-NDE-heat (Fuler)
88.24 + 0.31
0.54 0.01
CDR-NDE-heat (Dopri5)
88.60 + 0.26
0.49 0.01
walker2d kinematic simulation (Lechner Hasani 2020). We compare our proposed mod- els against RNN models which are designed to tackle the irregularly sampled time-series data.
The experimental setup such as the numerical method, hidden state dimension and other hyperparameters the same as in (Lechner & Hasani, (020) and is
Table 2: Hyperparameter Details
provided in Table 2 Table provides the choice of
Parameter
Value
numerical methods for each model. The proposed models CDR-NDE and CDR-NDE-hea((Eule) used the Euler method with the number of steps as 2. CDR- IDE-heat(Dopris) used Dopri5 with the absolute and relative tolerance set to le Scheduled learning rate decay used with decay parameter ? 0, scheduled at epoch 100. The models are trained on Nvidia Tesla V-100 32GB GPU.
Hidden state Dimension
64
Minibatch size
256
Optimizer
RMSprop
Learning rate
5e -3
Training epochs
200
5.1 BASELINES
We compared our proposed models against RNN models which are designed to address the problem of
fitting irregular time-series data such as GRU-ODE (De Brouwer et al.) 2019), CT-GRU (Mozer et al.)
2017) CT--NNN (Funahashi & Nakamural (993).GRUM (Che et al.) 2018 1018eP Phased-LSTT (Neil et al.
(016).DEE-LSTT (Lechner & Hasani 2020), Iitirectional-RNY (Schuster & Paliwal 1997), RNN
decay (Rubanova et al. 2019b),Hawkk
(Mei & Eisner 2017b), Augmented LSTM (Lechner
Hasani 2020), and ODE-RNN (Rubanova et al. 2019b).
8

---

Under review as a conference paper at ICLR 2023
5.2 PERSON ACTIVITY RECOGNITION WITH IRREGULARLY SAMPLED SERIES
Dataset contains sensor data from from different sensors(1 for each ankle, chest and belt) attached to 5 different people, performing 5 sequences of activities. The task is to classify the activity based on the sensor data. There are 11 different actívities which are reduced to 7 as suggested in (Rubanova et al. 2019b). Dataset is transformed such that each step in the recording contains values (4 of which determine the sensor that is producing data and the other 3 are sensor data). Each recording is split into overlapping intervals of 32 (with overlap of 16) and all the sequences are combined into one dataset. Out of the total sequences 7769 used for training, 1942 used for testing.
In Table 3 column 2 shows the performance of all the models trained on a person-activity dataset in terms of test accuracy. Our proposed models CDR-NDE and CDR-NDE-he perform better than all other baseline models. It shows that considering the continuous transformation along both the directions results in model with better representation capability and generalization performance. We also observe that the more flexible CDR-NDE-heat model using the adaptive Dopri5 solver gives the best performance in the person-activity dataset. To verify the flexibility of the model and the requirement of different depth for different sequences, we computed the number of function evaluations involved while evolving the hidden states over depth in the CDR-NDDF-heat(Dppis) model. We found that the number of function evaluations fall in the range of 26 to 32. This shows that different sequences required different number of function evaluations for learning better representations. Training time for an epoch for the models are CDR-NDE--ea(DoprSS) 30 sec, and CDR-NDE: 58 sec, and shows that CDR-NDE-heat models are faster.
22 sec,
5.3 WALKER2D KINEMATIC SIMULATION
The dataset was created by (Lechner & Hasani 2020) for Walker kinematic modeling task. This a supervised autoregressive task and the dataset was generated using Walker2d-v2 OpenAl gym environment and MuloCo physics engine. This dataset evaluates how well a model can simulate kinematic modeling systems that are sampled at irregular time intervals. The training data was generated by performing rollouts on the Walker2d-v2 v2 environment using pre-trained deterministic policy. The Walker environment was trained using a non-recurren. policy though Proximal policy optimization before data collection. The dataset is made irregularly sampled by excluding 10% of the timesteps. The dataset is split into 9684 train, 1937 test, 1272 validation sequences.
In Table 3 column 3 shows the performance of all the models Walker2d data. Our proposed model TDR-NDE-heea((Eule and Dopri5) outperform other models with a good margin. The proposed model CDR-NDE also gives very good performance in this data. Smoothing of the hidden representations allowed the CDR-NDE-head model to learn well on this data. Again, the more flexible CDR-NDE heat model using the adaptive Dopris solver gives the best performance. Training time for an epoch for the proposed models are EDR-NDE-heat(Pan
28 sec, CDR-NDE- (Deaa(oi) 48 sec, and CDR-NDE: : 140 sec, and shows that CDR-NDE-heal models are faster.
6 CONCLUSION AND FUTURE WORK
We proposed novel continuous depth RNN models based on the framework of differential equations. The proposed models generalize recurrent NODE models by continuously evolving in both the temporal and depth directions. CDR-NDE models the evolution of hidden states using two separate differential equations. The -OR--DE--eee model is designed based on the framework of ID-Heat equation and models the evolution of the hidden states across time and depth. The experimental results on person activity recognition and Walker2d kinematics data showed that our proposed models outperformed the baselines and are very effective on irregularly sampled real-world sequence data. Currently, CDR-NDE models are designed based on the GRU-cell transformations We would like to extend it to other transformations as a future work. The continuous depth recurrent neural differential equations are very flexible and generic RNN models. They will have widespread application on several complex sequence modeling and time series problems, involving sequences and inputs with irregular observation times, and varying complexities and modalities.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arthur Asuncion and David Newman. UCI machine learning repository, 2007.
Johannes Brandstetter, Daniel E Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations, 2021.
John Rozier Cannon. The one-dimensional heat equation. Number 23. Cambridge University Press, 1984.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-tempore point processes. In International Conference Learning Representations, 2020.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111, 2014. 2014.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE- Bayes: Continuous modeling of sporadically-obseree time series. Advances in neural information processing systems, 32, 2019.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks, 6(6):801-806, 1993.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34 (1):014004, 2017.
YAN Hanshu, DU Jiawei, TAN Vincent, and FENG Jiashi. On robustness of neural ordinary differential equations. In International Conference on Learning Representations, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. IEEE Transactions on Pottern Analysis and Machine Intelligence, 42(2):386-397, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Yihao Hu, Tong Zhao, Shixin Xu, Zhiliang Xu, and Lizhen Lin. Neural-PDE: A RNN based neural network for solving time dependent PDEs, 2020. URL https::////// 03892
https::/arxiv..... /ttp///axvv.v... org/ aba///ss///00
Toshinori Kimura. On dormand-prince method. Japan Malaysia Technical Institute, 40(10):1-9, 2009.
Mathias Lechner and Ramin Hasani. Learning long-term dependencies in irrgularty-sampled time series. arXiv preprint arXiv:2006.044//6 2020.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285. PMLR, 2018
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self self-modulating multivariate point process. Advances in neural information processing systems, 30, 2017a.
10

---

Under review as a conference paper at ICLR 2023
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self self-modulating multivariate point process. Advances in neural information processing systems, 30, 2017b.
Michael C Mozer, Denis Kazakov, and Robert V Lindsey. Discrete event, continuous time RNNs. arXiv preprini FrrXii:1110.04111 2017.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. Advances in neural information processing systems, 29, 2016.
Gerald W Recktenwald. Finite-difference approximations to the heat equation. Mechanical Engi- neering, 10(01), 2004.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern AnalFuciysis and Machine Intelligence, (9(6))1137-111 2017.
Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent ordinary differential equations for ireeullrllysamplld time series. Advances in neural information processing systems, 32, 2019a.
Yulia Rubanova, Ricky TQ Chen, and David K. Duvenaud. Latent ordinary differential equations for saagule time series. Advances in neural information processing systems, 32, 2019b.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations, 2018. htttt:::/aaaii.r/rppps//e bbazgg/bbs//000 04272
William E. Schiesser. The numerical method of lines: integration of partial differential equations. Elsevier, 2012.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions Signal Processing, (5(11)::673-26811 1997.
Dang Duc Trong, Nguyen Thanh Long, and Pham Ngoc Dinh Alain. Nonhomogeneous heat equation: Identification and regularization for the inhomogeneous term. Journal of mathematical analysis and applications. 312(11)93-104, 2005.
Weixuan Wang, Zhihong Chen, and Haifeng Hu. Hierarchical attention network for image captioning. In Pruceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8957-8964, 2019.
Kirill Zubov, Zoe McCarthy, Yingbo Ma, Francesco Calisto, Valerio Pagliarino, Simone Azeglio, Luca Bottero, Emmanuel Luján, Valentin Sulzer, Ashutosh Bharambe, et al. Neuralpde: Automating physics-informet neural networks (PINNs) with error approximations. arXiv e-prints, pp. arxiv 2107, 2021.


Reviews:
[{"review_id": "kKnsd4t01YR", "reviewer": "Reviewer_itQc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new type of recurrent neural network that operates over continuous time and continuous depth.\nThe continuous depth is achieved through an auxiliary variable t' that is integrated in each step from zero to a certain Tmax. The paper also proposes a heat equation PDE model to decode both time variables.", "summary_of_the_review": "Overall, minor but interesting contribution with a bit of unconventional writing and insufficient experimental evaluation.", "strength_and_weaknesses": "# Strenghts\n- The paper presents an interesting idea on a relevant topic (time-continuous sequential data).\n- The paper presents a novel PDE model for resolving the problem of having two different time variables.\n\n# Weaknesses\nThe experimental evaluation of the paper is insufficient for ICLR.\nFirst of all, the paper lacks discussion and comparison with significant related works.\nFor instance, [1,2,3,4,5,6,7] propose models for processing irregularly sampled time series. All of these models have demonstrated significant improvements in continuous-time modeling in the past two years alone.\n\nMoreover, the selection of benchmark datasets is insufficient. Particularly, the paper only evaluates the model on two datasets. Additionally, both evaluations seem to be sourced from (Lechner & Hasani, 2020), which did not perform any hyperparameter tuning, evaluated on five datasets, and does not include any method published after 2020.\n\n[1] Romero et al. CKConv: Continuous Kernel Convolution For Sequential Data. ICLR 2022.  \n[2] Gu et al. Efficiently Modeling Long Sequences with Structured State Spaces. ICLR 2022.  \n[3] Kidger et al. Neural Controlled Differential Equations for Irregular Time Series. NeurIPS 2020.  \n[4] Morrill et al. Neural Rough Differential Equations for Long Time Series. ICML 2020.  \n[5] Gu et al. HiPPO: Recurrent Memory with Optimal Polynomial Projections. NeurIPS 2020.  \n[6] Kidger et al. Efficient and Accurate Gradients for Neural SDEs. NeurIPS 2021.  \n[7] Shukla et al. Multi-Time Attention Networks for Irregularly Sampled Time Series. ICLR 2021.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SPYsPzElZN", "reviewer": "Reviewer_Ngo4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on learning of multivariate irregularly sampled time series data. Recently, several works have introduced continuous version of RNNs based on Neural ODEs to solve the problem of irregular sampling. This paper generalizes the recurrent Neural ODE models by continuously evolving the hidden states in both temporal and depth directions. It employs two separate differential equations to evolve the hidden states in both horizontal and vertical direction alternatively. The paper also proposes heat equation based model which treats the hidden state computation as solving heat equation over time. This model is also able to learn a better representation by utilizing information from past as well as future. Experiments show that the proposed models outperform the RNNs and neural ODE based models on two datasets.", "summary_of_the_review": "My recommendation for this paper is reject. Although the paper proposes a novel solution, motivation is lacking behind the proposed approach, rigorous experimentation is missing.", "strength_and_weaknesses": "### Strengths\n1. The proposed ideas of learning continuous depth model as well as using second order partial differential equation to evolve hidden states are novel and interesting.\n2.  Experimental results show the effectiveness of the approach when compared to baseline approaches.\n3. The paper focuses on the task of learning from irregularly sampled data which is important in many domains.\n\n### Weaknesses\n1. Although it is interesting work, it is not well motivated. Why would continuous in depth potentially be useful for modeling irregularly sampled time series data?\n2. Another key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. The paper only experiments on a single real-world dataset and missing experimentation of benchmark datasets (e.g. PhysioNet, MIMIC-III).\n3. The paper is missing important related work and comparison with the SOTA in modeling irregularly sampled time series [1]. The paper also seems to underperform compared to [1] on person activity recognition.\n4. In the human activity dataset, which is a per time-step classification problem, it seems that the sequence was fed into the proposed model as a whole, thus the model has access to future observations for making a classification prediction on \"past\" time steps. This would mean that it has an unfair advantage over the RNN based models. \n5. RNN and ODE based recurrent approaches can be used for whole-time series classification, imputation, interpolation and extrapolation tasks. The paper only focuses on per time point classification/regression task which is very limiting. It is also not clear how the run time of the proposed approach compare to that of ODE-RNN and other baselines.\n\n\n#### References\n1. Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "OPRGXeuedU1", "reviewer": "Reviewer_FygP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a continuous formulation of an RNN for modeling sequential data. The contribution of this work is a mechanism to continuously increase depth of recently proposed ODE-based RNN models, as well as a model that is motivated by a 1D-Heat equation. Two benchmark problems (person activity recognition and Walker2d kinematics) are used to show that the proposed models can outperform related models on irregularly sampled sequential data. ", "summary_of_the_review": "In summary, this is an interesting paper with some potential, but I feel that this paper is not ready for publication in its current form. It requires a major revision to better present the impact and significance of the different aspects of the proposed innovations. ", "strength_and_weaknesses": "The ideas presented in this paper are interesting, and the results demonstrate the effectiveness of the proposed models on 2 tasks. However, I feel that the innovations are somewhat incremental. Moreover, I have several concerns that I would like to see addressed:  \n\n* It is not clear to me how the individual components that have been proposed affect the performance, since a detailed ablation study is missing. Is the performance boost due to the specific form of the gates, or due to increasing depth, or because the model has at least twice as many weights as a simpler ODE-GRU. At a minimum it would be helpful to list the parameters for all the models. Then, it would be nice to see how the performance of the proposed model changes as a function of depth.  \n\n* You claim that your model is continuously evolving in both the temporal and depth directions, but I am not sure what this means. Where do you show that the model is learning something continuously?  Once you discretize your model for training, you yield a discrete computational graph. There have been several recent works that show that Neural ODEs discretized by explicit Euler do not provide a useful representation for the underlying continuous dynamics of the system of interest. Also, based on your results, it seems that it doesn't matter too much which discretization scheme is used. So, I assume something else must be going on. \n\n* A general problem of RNNs (whether formulated as a discrete or continuous model) are vanishing and exploding gradients during training. Gates certainly mitigate this issue, but it would be good to understand how the proposed model performs on tasks that involve longer sequences. \n\n* For an experimental paper, two sets of experiments that demonstrate the performance of a new model are somewhat weak. \n\n* The paper is missing to discuss several recent state-of-the-art RNNs, based on ODEs, that have appeared in ICML, ICLR, such as coRNN, UnICORNN, incremental RNN,  LEM, and several other related works. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject