Mixed-Memory RNNs for Learning Long-term Dependencies in Irregularly Sampled Time Series
Authors: Mathias Lechner
Ramin Hasani
Venue: ICLR 2022 Submitted (2022)

Abstract: Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly sampled time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's choice. We provide a solution by equipping arbitrary continuous-time networks with a memory compartment separated from its time-continuous state. This way, we encode a continuous-time dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these models Mixed-Memory-RNNs (mmRNNs). We experimentally show that Mixed-Memory-RNNs outperform recently proposed RNN-based counterparts on non-uniformly sampled data with long-term dependencies.

Under review as a conference paper at ICLR 2021
MIXED-MEMORY RNNs FOR LEARNING LONG-TERM DEPENDENCIES IN IRREGULARLY SAMPLED TIME- SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
Recurrent neural networks (RNNs) with continuous time hidden states are a natural fit for modeling tregularly-samplle time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's 's choice. We provide a solution by equipping arbitrary continuous- time networks with memory compartment separated from its time- continuous state. This way, we encode continuous-timn dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these models Me---mmmmorryyt (mmRNNs). We experimentally show that Mixed- Memory- RNNs outperform recently proposed RNN-based counterparts non-uniformly sampled data with long-term dependencies.
INTRODUCTION
Irregulary-ssappl time series, routine data streams in medical and business settings, can be modeled effectively by time-continuous version of recurrent neural networks (RNNs). These class of RNNs whose hidden states are identified by ordinary differential equations, termed an ODE-RNN (Rubanova al., 2019), provably suffer from the vanishing and exploding gradient problem (see Figure I the first two models), when trained by reverse-mode automatic differentiation (Rumelhart et al. 1986 Pontryagin 2018).
An elegant solution to the vanishing gradient phenomenon (Hochreiter) 1991 Bengio et al. 1994). which results in difficulties in learning long-term dependencies RNNs, memory gating. This technique, first published in form of the long short term memory networks (LSTM) (Hochreiter & Schmidhuber 1997), enforce constant error propagation through the hidden states, learn to forget, and disentangle the hidden states (memory) from their output states. Despite becoming the standard choice in modeling eegularly-samplee temporal dynamics, memory gated RNNs similar to other discretized RNN models, face difficulties when the time-gap between the observations irregular.
In this paper, we propose a compromise to design a novel recurrent neural network algorithm that simultaneously enjoys the approximation capability of ODE-RNNs in modeling irregularly- sampled time series and capability of learning long-term dependencies of the memory gated RNN's computational graph. To perform this, we let memory cell compute its implicit memory mechanism by their typical (input, forget, and output) gates while receiving their feedback inputs from time- continuous output state representation. This way, we incorporate continuous-tin within a gated recurrent module (e.g., an LSTM), enabling cells to respond to data arriving at arbitrary time-lags, while avoiding the vanishing gradient problem. Since these models interweave continuous flows with discrete memory mechanisms, we call them mixed-memory RNNs (See Figure [ the last model).
dynamical flow
We compare mmRNNs to standard and advanced continuous timee RNN variants, on a set of synthetic and real-world sparse time-series tasks, and discover consistently better performance.

---

Under review as a conference paper at ICLR 2021
ODE ANN
To put this in context, first theoretically prove that the class of ODE-RNNSs suffers from the exploding and vanishing gradient problem, making them unable learn long-term dependencies efficiently. We show that learning ODE-RNNs by the adjoint method (Chen et al.) 2018 8) does not help with this problem. As solution, we propose mmRNNs, a continuous-ting recurrent model capable of learning long-tent dependencies of time-series.
exploding gradient region
f OR(Cr)
Vanih
Ry
ACC AE)
ANN Decay
Exolod ng gradient region
afa (
Varis region
Ry-t
BACKGROUND
ho
ht
hCC 86)
ODE-RNNS Instead of explicitly defining state update function, ODE-RNNS identify an ordinary differential equations in the following form (Funahashi & Nakamura 1993):
mmRNN
Exploding & gradiens region
0 Oc(r,)
Vaniish
Oh = fo(eT,, h4, T) -h, Øt where 2t the input sequence, h2 an RNN's hidden state, and r is dampening factor. The time-lag T specifies at what times the inputs 2, have been sampled.
Cy-i
Cy
c(t (C0+-00 (0))
Rg
hg
ACc 4-0))
by
ODE-RNNs were recently rediscovered (Rubanova et al., 2019) and have shown promise in approximating irreuularyysampple
x(5)
data, thanks to the implicit definition of time their resulting dynamical systems. ODE-RNNs can be trained by backpropagation through time (BPTT)
Figure Magnitude of the states' er- ror propagation in time-continuous re- current neural networks gives rise the vanishing or exploding of the gradient (first two models). mmRNNs are a so- lution to keep constant gradient flow avoid these phenomena in modeling irregularly sampled data.
(Rumelhart et al., 1986 [eerbos
1988 1990) through
ODE solvers, or by treating the solver black-box and apply the adjoint method (Pontryagin) 2018) to gain mem- show this family of recurrent networks faces difficulty to learn long-term dependencies.
ory efficiency (Chen et al.) 2018).. In Section
Long Short-term Memory LSTMs (Hochreiter & Schmidhuber 1997) express their dis- cretized hidden states as pair (ca, h₂) and its update function realizes mapping as follows: fo(1241, (Ce, h4),
(c4+1,h441).
LSTMs demonstrate great performance on learning equidistant streams of data (Greff al. 2016). however similar to other discrete-state RNNs, they are puzzled with the events arriving in in-between observations. Section 4, we introduce continuous-time long short-term memory algorithm to tackle this.
3 ODE-RNNS SUFFER FROM VANISHING OR EXPLODING GRADIENT.
In this section, we show that ODE-RNNs trained via backpropagation through time (BPTT) are susceptible vanishing and exploding gradients. We also illustrate that the adjoint method is not immune to these gradient issues. We first formally define the gradient problems of the RNNs, and progressively construct Theorem
Gradient propagation in recurrent networks.
Hochreiter (Hochreiter 1991) discovered that the error-flow in the BPTT algorithm realizes a power series that determines the effectiveness of the learning process (Hochreiter) 1991 Hochreiter & Schmidhuber, 1997; Bengio al., 1994: Pascanu al. 2013). In particular, the state-previous state Jacobian of an RNN:
Öh44T(2447 T)
(2)
Ohe
governs whether the propagated error exponentially grows (explodes). exponentially vanishes, or stays constant. Formally:
2

---

Under review as a conference paper at ICLR 2021
Definition 1 (Vanishing exploding gradient). Let h4TT f(I+r- h4, T) be recurrent neural network, then we say unit i of the network f suffers from a vanishing gradient ifr some small e > 0 ahi
<1 s, where N is the dimension of the hidden state by and super-script of denotes the i-th entry of the vector u. We say unit i of the network J suffers from an exploding ah; > 1. We say the whole network f suffers from a vanishing or respectively exploding gradient problem the above condition hold for some of its units.
hold that Σ}=
ah?
gradient if holds that Σー shi+T
The factor in Definition T is essential as Gers et al. (Gers et al., 2000) observed that learnable vanishing factor in the form of a forget-gate significantly benefits the learning capabilities of RNNs, ie., the network can learn forget. Note that RNN can simultaneously suffer from vanishing and an exploding gradient by the definition above.
Now, consider ODE-RNN given by Eq. [ is implemented either by Explicit Euler discretization
or by a Runge-Kuut Kutta method (Runge) 1895 Dormand & Prince 1980). We can formulate their state-previous state Jacobian in the following two lemmas:
Lemma 1.
Let h = fo h T) - ho be an ODE-RNN. Then state-previous state Jacobian of the 0h.TT T
explicit Euler is given by the following equation:
-TI.
h=h,
Lemma 2. Let h fo h. hT be an ODE-RNN. Then state- state-previous state Jacobian of the Runge-Kutta method is given by and some K₃-
ah&Tr The
I II
2== bi
M
at
-TI., where Σj=1 ba =1
The proofs for Lemma D and Lemmal 2 is provided in the supplements. Consequently, we have:
Theorem 1. (ODE-RNNS suffer from vanish exploding gradient) Let h = So(a, h, T) hT, and he the RNN obtained by simulating the ODE by a solver based on the explicit Euler or Runge- Kutta method. Then the RNN suffers from vanishing and exploding gradient problem, except for parameter configurations which give the non-trainable constant dynamics fe(h, x) = 0, and cases where fo(h, x) constant, for a particular input sequence x and 6.
The proof is given in full in the supplementary materials. A brief outline of the proof is as follow: First, look the special cases of 0f 0h
O. While such f would enforce constant error propagation by making the Jacobians equal to the identity, it also removes all dynamics from the ODE state. In other words, it would operate the ODE memory element. Intuitively, any interesting function So pushes the Jacobians away from the identity matrix, creating a vanishing or exploding gradient depending fe-
Our proof sketch implies a minor but nonetheless interesting statement:
8f Corollary 1. A ODE-RNN with identity Jacobian matrix expresses the trivial dynamics 3h
0.
The proof follows directly by Lemma and
Corollary I suggests that well-conditioned gradients of an ODE-RNN can negatively impact the modeling capacity of these models and prevent them from learning the dynamics of the training data. In particular, this result indicates that strong long-term learning results of ODE-based architectures from the literature may be attributed to a better initialization, i.e., with stable gradients being only present at the beginning of the training.
Theorem 2. (ODE-RNNs suffer from vanishing/exploding gradients regardless of their choice of ODE-solver) Let h fo(æ, T) lt, with fe being uniformly Lipschitz continuous. Moreover, Oh(T)
let h(tt) be the solution of the initial value problem with initial state bg. Then, the gradients 8he i.e, the Jacobian of the ODE state time T w.t.. the initial state hyg can vanish and explode, except for parameter configurations which give rise to the non-trainable constant dynamics fo To(h,z 0, and cases where so .)) constant, for a particular imput sequence x and parameters 0.
The proof is given in full in the supplementary materials. A brief outline of the proof is follow: We start by approximating the initial-value problem by an explicit Euler method with a uniform step-size. We then let the step-size approach zero which due to the SeaaddLLindele theorem, makes
3

---

Under review as a conference paper at ICLR 2021
0f the series converge the true solution of the ODE. Based on bounds on äh can obtain bounds of the gradients in the limit, which can vanish or explode depending on fo.
Note that the Theorems and hold independently of the used RNN architecture. In particular, can entail the following statement:
hold
Corollary 2. ODE-combined with-GRU and DDE--comiine-wwith-LLTTT as presented in (Rubanova et al. 2.19. also suffer from vanish or exploding gradients.
The proof follows by applying the Theorems and 2 with a GRU and LSTM.
Does the adjoint method solve the vanishing gradient problem? Adjoint sensitivity method (Pontryagin 2018) allows for performing memory-efficiene reverse-mode automatic differentia-
tion for training neural networks with their hidden states defined by ODEs (Chen et al. 2018
The method, however, possesses lossy reverse-
mode integration steps, as it forgets the com- puted steps during the forward-pass (Zhuang each reverse- mode step. the backward gradient pass diverges from the true forward pass (Zhuang al. 2020;
a) Contiruou vector field
Discreta ODE-soiver trajectory
al., 2020). Consequently,
Gholami et al. 2019).
This is because the auxiliary differential equa- tion in the adjoint sensitivity method, a(t), still contains state-dependent components at each reverse-step, which depends on the historical values of the hidden states' gradient. the extreme case, severse-steps completely diverge from the hidden states of the forward-time solu- tion, resulting in incorrect gradients. Therefore, both vanilla BPTT and the adjoint method face difficulties for learning long-term dependencies. n the next section, we propose a solution.
<) Adjoint time reversed vector
Adjoint ODE solver trajectory
The adjoint method makes numerical error when computing the gradients. a) Continu- ous vector field implied by an ODE, b) Numerical ODE-solvers realize a discrete trajectory on the vector field. e) The adjoint ODE creates a time- reversed vector field. d) Discrete trajectory of the adjoint ODE-solver diverges from the trajectory of the forward simulation due to discretization and numerical imprecision.
Figure 2:
4 MIXED-MEMORY
RECURRENT ARCHITECTURES
Instead of having a single state vector he that processed by the discrete RNN and the time- continuous ODE, our Mixed-Memory architec- ture represents its hidden state by a pair (c,, h4). An update of the form
C++1 C. @ (go(hs) + bb) + =l()),
(3)
governs the memory cell component C of the mmRNN, where go and ze are learnable gating functions and by a bias term. Although, the update Eq. (3) appears similar to that of a vanilla LSTM and GRU, there is one key difference: The gates of LSTM/GRU is controlled by its previous hidden state, whereas the gating of a mmRNN is defined by a second process hT-
The second part of the hidden state he is controlled by continuous time process, e.g. Neural ODE, initialized by the memory cell vector e. In particular, the update function for by is of the form
h++1 ODE-Solve(fe, T,,
The fundamental distinction of mmRNN other continuous-time RNNs is the strict separation of memory and time-continuous hidden states. In particular, the memory update in Eq. 4 ensures a constant error propagation through ct while arbitrary Neural ODEs process the state h, in time-countinoous fashion.
On the contrary, recurrent network variants such as continuous- gated recurrent units (CT-GRU) (Mozer et al., 2017), and GRU-D (Che et al.) 2018) incorporate a time-dependent decay apparatus on the state, while preserving the rest of the RNN architecture. This decaying memory originates the vanishing factor during backward error-propagation, which results in difficulties in learning long-term dependencies.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 The mixed-memory RNN
Input: Datapoints and their timestamps {are t4) J=I...N
Parameters: Weights 0, output weight and bias W Woutput: Doutput hg = 0 (ODE state)
Cu 0 [Memory cell]
for 1 = 1...N do
4 C (go 1,i-1, 1, 1,44) by) + z0 (0 (Memory cell update)
h; = ODESolve So, cs li 11-1) (Time-continuous state update)
0, W output boutput
end for Return foi)i=1...N
Forward phase
proliction:
ODE-RNN With vanishing gradient
Immune to vanishing gradient
Lose Surface
Loss Surface
Time
Backpropagation phase
apol
crie: 1
mmRNN
ODF-RNN
Besc
Boest
Revered time
Figure 3: Left: Illustration how vanishing gradients make RNN training difficult when the data express long-term dependencies. The prediction error be thought of as teaching signal indicating how the dynamics should be changed to minimize the loss. The vanishing gradient of the ODE-RNN makes the teaching signal weaker when propagating it back in time. Conversely, the teaching signal stays near-constant in mmRNNs. Right: The resulting loss surfaces of the ODE-RNN is much steeper than mmRNN, making the training difficult.
Our mmRNNs are immune to this shortcoming. More precisely, Table how the transition of the hid-
Table 1: Change to the hidden states of an RNN between observations & and + T
den states between two observations of the mmRNN differs from other architectures. Similar to the vanilla LSTM and GRU for regularly sampled time-series, we can ensure a near-constant error propagation at the beginning of the training process with proper weight initialization.
Model
State between observation
Standard RNN
h
GRU-D
hee TT
ODE-RNY BE-RNN
T) (a, Soo Ce, c2,
ODE- DDE-Solve(fo. he,
mmRNN
Theorem 3. Let ff with (C4-T, h+rr) = f(I+4I, (c,, h4). 1. Moreover, assume the weights of go and to be initialized close to O. Then, the units e of the state pair (ct, he) do not suffer from a vanishing or exploding gradient at the beginning of the training process.
), T) be an mmRNN described by Algorithm
The proof is given in full in the supplements. A brief outline: We assume that ge and zq are initialized close to 0 and we are the beginning of the training, thus these values have not changed much yet.
obr) 0.9943, which less than I (no exploding) but much greater than O (no vanishing). Note that exact value of the Jacobian at the beginning of the training can be controlled by the bias term by. If the underlying data express very long-term dependencies, to 1.
Consequently, we can neglect them and get Σラ=1 Beirrlarr
Del
increase by and bring the error flow factor even closer
The mmRNN can be viewed as memory cell with gates controlled by time-continuous process realized by ordinary differential equations. Next, we evaluate the performance of mmRNNs in multiple time-series prediction tasks.
5

---

Under review as a conference paper at ICLR 2021
Empirical measurement of gradient norms. To emphasize the impact of our theoretical results, we performed an experiment comparing the hidden state gradient norms of an ODE ENNN instance compared to another ODE-RNN equipped with mmRNN for different sequence lengths (mean/std over 3 initialization seeds). The results show (as proven in Theorem and 2) that the gradients of standard ODE-RNN tend exponentially increase with the length of the input sequence. This gradient issue is significantly improved when mmRNNs are used (See Table 2).
Table 2: Gradient Norms ODE-RNN mmRNN. (n =3)
Sequence length
10
25
50
ODE-RNN
10.50 1 5.47
38.75 + 33.20
235.76 + 383.04
1214.94 3750.62
mmRNN
0.97 0.01
0.96 0.03
1.01 0.10
1.25 0.25
5 EXPERIMENTAL EVALUATION
We constructed quantitative settings with synthetic and real-world benchmarks. We assessed the generalization performance of time-continuous RNN architectures on datasets that are deliberately created to express long-term dependencies and are of irregularly eeggullayy-sampled nature. All code and data are submitted as a supplement to our manuscript.
Baselines. We compare mmRNN to a large variety of continuous-time RNNs introduced to model imegularly-ssmpplrs data. This set includes RNNS with continuous-stat state dynamics such as ODE-RNN (Rubanova et al., (019)) and CT-RNNs (Funahashi & Nakamura, 1993) state-decay mechanisms such as CT-GRU (Mozer et al.] (017), RNN Decay (Rubanova et al., 2019), CT-LSTM (Mei & Lisner 2017). and GRU-D (Che et 2018), , in addition to oscillatory models such as Phased-LSTM Neil et al. 2016), CoRNNS (Rusch & Mishral 2021), iRNNS (Kag et al.) 2019), and Lipschitz RNNs (Erichson al.) 2021). Furthermore,
tested mmRNNNs against intuitive time-gan gap modeling approaches we built here, termed an augmented LSTM topology as well as reciprocal RNNs (Babaei
al. 2010). Experimental settings given Appendix.
5.1 SYNTHETIC BENCHMARK BIT-STREAM SEQUENCE CLASSIFICATION
We formulated a modified time-series variant of the XOR problem (Mar vin & Seymour 1969). In particu- lar, the model observes a block of bi- nary data in the form of a bit-after-bit time-series. The objective to learn an XOR function of the incoming bit- stream. This setup is equivalent to the clasfcclassiiiaation of the input se- quence, where the labels are obtained by applying XOR function to the inputs.
Table 3: Bit-stream sequence classification. Note: Test ac- curacy (mean + std, N 5). While all RNNs can represent the correct function, training is difficult due to long-term dependencies.
Dense encoding
Event-based encoding
Model
ODE-RNN
50.47%
0.06
S1.21% 0.37
CT-RNN
50.42%
0.12
50.79% 1 0.34
Augmented LSTM
100.00% % 1 0.00
89.71% 1 3.48
CT-GRU
100.00 % 0,00
61.36% = 4.87
RNN Decay
60.28% 19.87
75.53%
5.28
Reciprocal RNN
100.00 % 0.00
90.17% = 0.69
GRU-D
100.00 % + 0.00
97.90% + 1.71
While any non-linear recurrent neural network architecture can learn the cor- rect function, training the network to do so is non-trivial. For the model to make an accurate prediction, all bits in an upcoming chunk are required to be taken into account. However, the error signal is only provided after the last bit is observed. Consequently, during learning, the prediction error needs to be propagated to the first input time-step to precisely capture the dependencies.
PhasedL.STT
50.99% 0.76
80.29% + 0.99
GRU-ODE
50.41% 0.40
52.52% 0.35
CT-LSTM
97.73% 0.08
95.09% 0.30
iRNN
49.99% 1.20
50.54% 1 0.94
coRNN
100.00 % 1: 0.00
52.89% + 1.25
Lipschitz RNN
100.00 % + 0.00
52.84% = 3.25
mmRNN (ours)
100.00% % 1 0.00
98.89% 0.26
We designed two modes, a dense encoding mode in which the input sequence is represented as a regular, periodically sampled time-series, and event- based mode which compresses the data into irregularly sampled bit-streams, e.g., 1, 1, 1, is encoded as (1, = 4). (See T)ble We observed that considerable number of RNNs face difficulties in modeling these tasks, in dense-encoding

---

Under review as a conference paper at ICLR 2021
mode. In particular, ODE-RNNS, CT-RNNS, RNN-Decay, Phased-LSTM, and GRU-ODE could not solve the XOR problem in the first mode. Thased-LSTY, and RNN-Deccy improved their performance CT-RNNs, and GRU-ODE still could not solve the task.
in the second modality, whereas The core reason for their mediocre performance is the exploitation of the vanishing gradient problem during training. The rest of the RNN variants (except CT-GRU) were successful in solving the task in both modes, with mmRNN outperforming others in an event-based encoding scenario.
5.2 PERSON ACTIVITY RECOGNITION WITH IRREGULARLY SAMPLED TIME-SERIES
We consider the person activity recognition dataset from the UCI repository (Dua & Graff 2017). This task's ob- jective is to classify the current activity of a person, from
Table 4: Per time-step classification.
Person activity recognition. Test accu-
racy (mean std, = 5)
four inertial measurement sensors arms and feet. Even though the four sensors are mea- sured a fixed period of 21 Ims, the random phase-shifts between them creates an irregularly sampled time-series, Rubanova et al. (Rubanova al., 2019) showed that ODE- based RNN architectures perform remarkably well on this dataset. Here, we benchmarked the performance of the mmRNN model against other variants.
the person's s
Model ODE-RNN
Accuracy
80.43% = 1.55
CT-RNN
83.65% = 1.55
Augmented LSTM
84.11% 0.68
CT-GRU
79.48% + 2.12
RNN Decay
62.89% -3.87
Reciprocal RNN
83.85% 0.45
GRU-D
83.57% + 0.40
WhasedLSTM
83.33% = 0.69
This setting realizes a per-time-step classification problem. That is a new error signal is presented to the network at every time-step which makes the vanishing gradient less of an issue here. The results in Table 4 shows that the mmRNN outperforms other RNN models on this dataset. single dataset
GRU-ODE
82.56% = 2.63
CT-LSTM
84.13% 0.11
iRNN
74.56% = 1.29
CORNN
78.89% + 0.62
Lipschitz RNN
81.35 + 0.60
While the significance of evaluation limited, it demonstrates that the supreme generalization ability of mmRNN architecture.
mmRNN (ours)
84.15 % 0.33
5.3 EVENT-BASED SEQUENTIAL MNIST
We determined challenging sequence classification task by designing an event-based version for the sequential- MNIST dataset. For doing this we followed the procedure described below:
Table 5: Event sequence classification. Irregular sequential MNIST, Test accu- racy (mean + std, N = 5)
Model
Accuracy
1. Sequentialization encoding long-term de- pendencies transform the 28-by-28 image into time-series of length 784
ODE-RNN
72.41% 1 1.69
CT-RNN
72.05% = 0.71
Augmented LSTM CT-GRU
82.10% 4.36
87.51% = 11.57
2. Compression + non-uniform sampling en- code binary time-series in event-based format, to get rid of consecutive occurrences of the same binary value, c.g., 1, 1, 1, is transformed to (1,t = 4). (Read more about this experiment in supplements)
RNN Decay
88.93% = 4.00
Reciprocal RNN
94.43% + 0.23
GRU-D
95.44% = 0.34
PhasedLSTM
86.79% 1.57
GRU-ODE
80.95% = 1.52
CT-LSTM
94.84% = 0.17
iRNN
95.51% + 1.95
Using this sequentialization mechansim, we compress the sequences from 784 to padded sequences of 256
coRNN
94.44% 0.24
Lipschitz RNN
95.92% - 0.16
irreuularly-sappled datapoints. To perform well on this
mmRNN (ours)
97.83% 0.37
task, RNNs must learn to store some information up to 256
time-steps, while taking the time-lags between them into account. Since an error signal is issued at the end of the sequence, only an RNN model immune to vanishing gradients can achieve high-degrees of acccuracy.
Table 5 demonstrates that ODE based RNN architectures, such as the ONE, CT-RNN, and the
GRU-ODE (De Brouwer et al. 2019) struggle to learn high- fighlfidelityy model of this dataset. On the other hand, RNNs built based on a memory mechanism, such as the reciprocal RNN and GRU-D (Che al. 2018) perform reasonably well, while the performance of mmRNN surpasses others.
7

---

Under review as a conference paper at ICLR 2021
5.4 WALKER2D KINEMATIC SIMULATION
In this experiment, evaluated how well mmRNNs model a physical dynamical system. To this end, we col- lected simulation data of the Walker2d-v2 OpenAI gym (Brockman et al. 2016) environment using a pre- trained policy The objective of the model was to learn the kinematic simulation of the MuJoCo physics engine (Todorov al., 2012) in an auto-regressive fashion and supervised learning modality. We increased the complex- ity of this task by using the pre-trained policy at different training stages (between 500 to 1200 Proximal Policy Op- timization (PPO) iterations (Schulman et al. 2017) ) and overwrote 1 1 of all actions by random actions. Moreover, we simulated frame-skips by removing 10% of the time- steps. Consequently, the dataset is errguuaary-sampllr The results, shown in Table 6 indicate that mmRNNs capture the kinematic dynamics of the physics engine better than other algorithms with a high margin.
Table 6: Per time-step regression.
Walker2d kinematic dataset. (mean + std, N 5)
Model
Square-error
ODE-RNN
1.904 0.061
CT-RNN
1.198 + 0.004
Augmented LSTM
1,065 + 0.006
CT-GRU
1.172 = 0.011
RNN-Decay
1.406 + 0.005
Reciprocal RNN
1.071 0.009
GRU-D
1.090 + 0.034
PhasedLSTM
1.063 + 0.010
GRU-ODE
CT-LSTM
1014
iRNN
1.732 + 0.025
CORNN
3.241 + 0.215
Lipschitz RNN
1.781 + 0.013
mmRNN (ours)
0.883 + 0.014
RELATED WORKS
The notion of CT-RNNs (Funahashi & Nakamura 1993) was introduced around three decades ago. It identical to the ODE-RNN architecture (Rubanova al. 2019) with additional dampening factor In our experiments, however, we observed competitive performance to our mmRNNs achieved by the GRU-D architecture (Che et al., 2018). GRU-D encodes the dependence on the time-lags by trainable decaying mechanism, similar RNN-decay (Rubanova et al. 2019). While this mechanism enables modeling irregularly sampled time-series, also introduces vanishing gradient factor to the backpropagation path.
Time-continuous RNNs
Similarly, CT-GRU (Mozer et al., 2017) adds multiple decay factors in the form of extra dimensions the RNN state. An attention mechanism inside the CT-GRU then selects which entry along the decay dimension to use for computing the next state update. The CT-GRU aims to avoid vanishing gradients by including a decay rate of 0, i.e., decay all. This mechanism nevertheless, fails as illustrated in Table 3
Phased-LSW LSTM Neil al. 2016) adds a learnable oscillator LSTM. The oscillator modulates LSTM to create dependencies on the clapsed-time, but also introduces a vanishing factor in its gradients.
GRU-ODE (De Brouwer et al. 2019) modifies the GRU (Chung al. 2014) topology by incorporating a continuous dynamical system. First, GRU is expressed as a discrete difference equation and then transformed into continuous ODE. This process makes the error-prov proppagaoonninn time-dependent, i.c..
the near-constant error propagation property of GRU is abolished.
Lipsehitz RNN (Erichson et al. 2021) constraints the hidden-to- to-hddeen weight matrix of continuous- time RNN, such that the underlying dynamic system globally converges to a stable equilibrium. As a result, Lipschitz RNNs cannot suffer from a exploding gradient problem. The constraint of the weight matrix is realized efficiently using symmetric skew decomposition (Wisdom et al.) 2016).
Log-ODE method (Morril et al., 2020) compresses the input time-series by time-continuous path signatures (Friz& & Victoir 2010) before feeding them into ODE-RNNs. As the signatures much shorter than the original input sequence, the ODE-RNNs can learn long-term dependencics in the original input sequence despite expressing vanishing gradient.
CT-LSTM Mei Eisner 2017) combines the LSTM architecture with continuous neural Hawkes processes. At each time-step, the RNN computes two alternative next state options of its hidden state. The actual hidden state is then computed by interpolating between these two hidden states depending on the elapsed time.
8

---

Under review as a conference paper at ICLR 2021
2 uses an implicit-epllict Euler discretization of a second-order ODE modeling a controlled non- non-linear oscillator. The ttatett-nnxt state gradients of the resulting RNN are bounded in both directions, mitigating explosion and vanishing effects.
coRNN (Rusch & Mishra
iRNN (Kgg et al. 2019) parametrize a RNN by applying incremental updates to the steady-state of an ODE. In the limit with infinitely many updates between the state and the next state, iRNN express constant error propagation.
Learning Peegularly-Sampler Data Statistical (Pearson et al. 2003 Li & Marlin 2016; Belletti
2016, Roy & Yan, 2020) and functional analysis (Foster 1996 Amigó et al. 2012 Kowal et al.
al. 2019) tools have long been studying ann-uniformly-spacee data. A natural fit for this problem is the use of time-continuous RNNs (Rubanova et al., 2019). We showed that although ODE-RNNS are performant models in these domains, their performance tremendously drops when the incoming samples have long-range dependencies. We solved this shortcoming by introducing mmRNNs.
The notorious question of vanishing/exploding gradient (Hochreiter [991| [Bengio et al. 1994) was identified as the core reason for RNNs' lack of gener- alizability when trained by gradient descent (Allen-Zhu & Li. 2019 Sherstinsky (020). Recent studies used staterregularization (Wang & Niepert 2019) and long memory stochastic processes Frreaves-Tunnee & Harchaoui (2019) to analyze long-range dependencies. Apart from the original LSTM model (Hochreiter & Schmidhuber [197) and its variants (Greff et al. 2016) that solve the problem in the context of RNNs, very few alternative researches exist (Chen ct al. 2019).
Learning Long-term Dependencies
As the class of CT RNNs become steadily popularized (Hasani al.) 2020a Lechner et al. 2020a). it is important to characterize them better (Lechner et al.) 2019 Dupont et al. (219) Durkan et al.
2019) and understand their applicability and limitations (Jia & Benson 2019 Lechner et al.
2020b
Hanshu et al. 20200 Holl et al. 2020 Quaglino et al.] 2020: Kidger et al. [2020] Hasani et al.
2020b).
We proposed method to enable ODF-based RNNs to learn long-term dependencies.
7 DISCUSSIONS, SCOPE AND LIMITATIONS
We proposed a solution to learn long- dependencies in irregularly-samplee input data streams. To perform this, we designed a novel long short term memory network, that possesses continuous-time output state, and consequently modifies its internal dynamical flow to a continuous- mmRNNs resolve the vanishing and exploding of the gradient problem of the class of ODE-RNNE while demonstrating an attractive performance in learning long-term dependencies on data arriving at non-uniform intervals.
model.
What if we feed in samples' time-lag as an additional input feature to network? The Augmented LSTM architecture we benchmarked against realizes this concept, which is simplistic approach to making LSTMs compatible with irregularly sampled data. The RNN could then learn to make sense of the time input, for instance, by making its change proportional to the elapsed-time. Nonetheless, the time characteristic of an augmented RNN depends purely on its learning process. Consequently, we can only hope that the augmented RNN generalize to unseen time-lags. Our experiments showed that an augmented LSTM performs reasonably well while being outperformed by models that explicitly declare their state by a continuous time modality, such as mmRNNs.
Difference between reciprocal RNNs and mmRNN? A reciprocal architecture consists of two different types of RNNs reciprocally linked together in an auto auto-regressive fashion (Babaei al.) 2010). In our context, the first RNN could be designed to handle irregularly-sample time series while the second one is capable of learning long-term dependencies. For example, an LSTM bidirectionally coupled with an ODE-RNN could, in principle, overcome both challenges. However, the of heterogencous RNN architectures might limit the learning process. In particular, due to different training convergence rates, the LSTM could already be overfitting long before the ODE-RNN has learned useful dynamics.
Contrarily, mmRNN interlinks LSTMs and ODE-RNNs not in autoregressive fashion, but at an architectural level, avoiding the problem of learning at different speeds. Our experiments showed that MnRNNs consistently outperform reciprocal --DM--DD--NNN architecture.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Can sgd learn recurrent neural networks with provable general- ization? Advances in Neural Information Processing Systems, pp. 10331-10341, 2019.
José M Amigó, Roberto Monetti, Thomas Aschenbrenner, and Wolfram Bunk. Transcripts: An algebraic approach to coupled time series. Chaos: Interdisciplinary Journal of Nonlinear Science, 22(1):013105, 2012.
Sepideh Babaci, Amir Gerammaych, and Seyyed Ali Seyyedsalehi. Protein secondary structure prediction using modular reciprocal bidirectional recurrent neural networks. Computer methods and programs biomedicine, 100(3):237 247, 2010.
Francois W Belletti, Evan R Sparks, Michael Franklin, Alexandre M Bayen, and Joseph E Gonzalez. Scalable linear causal inference for irregularly sampled time series with long range dependencies. arXiv preprint arXiv:/603.033380 2016.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Dexiong Chen, Laurent Jacob, and Julien Mairal. Recurrent kernel networks. In Advances in Neural 2019.
Information Processing Systems, pp.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Junyoung Chung. Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous
modeling of sporadically- Systems, pp. 7377-7388, 2019.
time series. In Advances in Neural Information Processing
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt::t/tt/thh//eee ics. uc1. edd/mmll
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neural Information Processing Systems, pp. 3134-3144 3144, 2019.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pp. 7509-7520, 2019.
N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Represen- tations. 2021. URL htttts::
net/ worum ?id- N7PBXGOUJZ
Grant Foster. Wavelets for period analysis of unevenly sampled time series. The Astronomical Journal, 112:1709. 1996.
Peter K Friz and Nicolas B Victoir. Multidimennsonal stochastic processes as rough paths: theory and applications, volume 120. Cambridge University Press, 2010.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural nelworks, 6(6)8801-806, 1993.
10

---

Under review as a conference paper at ICLR 2021
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. Neural Computation, 12(10))2451-2477 2000.
Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint arXiv: 1902: 10298, 2019.
Alexander Preaves-Tunnell and Zaid Harchaoui. A statistical investigation of long memory in
language and music. International Conference on Machine Learning, pp. 2394-2403, 2019.
Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10); 2222-2232, 2016.
YAN Hanshu, DU Jiawei, TAN Vincent, and FENG Jiashi. On robustness of neural ordinary differential equations. In International Conference on Learning Representations, 2020.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time- constant networks. arXiv preprint urXiv:2006.0445 2020a.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. The natural lottery ticket winner: Reinforcement learning with ordinary neural circuits. Proceedings of the 2020 International Conference on Machine Learning. JMLR. org. 2020b.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen [in german] diploma thesis. TU Münich, 1991.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1785-1780, 1997.
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning to control pdes with diffecrentiabee physics. arXiv preprint (xXx:2001.004557 2020.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. In Advances in Neural Information Processing Systems, pp. 9843-9854, 2019.
Anil Kag. Ziming Zhang, and Venkatesh Saligrama. Runs incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference on Learning Representations, 2019.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv preprint arXiv:2005.00 08926, 2020.
Daniel R Kowal, David S Matteson, and David Ruppert. Functional autoregression for sparsely sampled data. Journal of Business & Economic Statistics, 37(1):97-109, 2019.
Mathias Lechner, Ramin Hasani, Manuel Zimmer, Thomas A Henzinger, and Radu Grosu. Designing worm-inspired neural networks for interpretable robotie control. In 2019 International Conference on Robotics and Automation (ICRA), pp. 87-94. IEEE, 2019.
Mathias Lechner, Ramin Hasani, Alexander Amini. Thomas A Henzinger, Daniela Rus, and Radu Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10): 642-652, 2020a.
Mathias Lechner, Ramin Hasani, Daniela Rus, and Radu Grosu. Gershgorin loss stabilizes the recur- rent neural network compartment of an end-to-end robot learning scheme. In 2020 International Conference on Robotics and Automation (ICRA). IEEF, 2020b.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 (6(11))227-2222, 1998.
Steven Cheng-Xiaa Xian Li and Benjamin M Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. In Advances in neural information processing systems, pp. 1804-1812 2016.
11

---

Under review as a conference paper at ICLR 2021
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference Machine Learning, pp. 3053-3062 2018.
Minsky Marvin and A Papert Seymour. Perceptrons. MIT Press, 1969.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process, In Advances Neural Information Processing Systems, pp. 2017.
James Morrill, Patrick Kidger, Cristopher Salvi, James Foster, and Terry Lyons. Neural cdes for long time series via the log-ode method. arXiv preprint arXiv:2009.000 08295, 2020.
Michael C Mozer, Denis Kazakov, and Robert V Lindsey. Discrete event, continuous time mns. arXiv preprint 1710.041 2017.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. In Advances in neural information processing systems, pp. 318228890,, 2016.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Ronald Pearson, Gregory Goney, and James Shwaber. Imbalanced clustering for microarray time- series. In Proceedings of the ICML, volume 3, 2003.
Lev Semenovich Pontryagin. Mathematical theory of optimal processes. Routledge, 2018.
Alessio Quaglino, Marco Gallieri, Jonathan Masci, and Jan KoutnÅk. Snode: Spectral discretization of neural odes for system identification. In International Conference on Learning Representations, 2020.
DP Roy and I. Yan. Robust landsat-based crop time series modelling. Remote Sensing of Environment, 238:1 10810, 2020.
Yulia Rubanova, Tian Qi Chen, and David K Duvenaud. Latent ordinary differential equations for serguullayy-smplled time series. In Advances in Neural Information Processing Systems, pp. 5321-5331, 2019.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6089) 1986.
Carl Runge. Über die numerische auflösung von diffcrentialglecchumen. Mathematische Annalen, 16(2):167-177, 1895.
T. Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (co(rnn)): accurate and (gradient) stable architecture for learning long time dependencies. International Conference Learning Representations, 2021. URL https::tt///o/o httpps://ppnnerrew..eett/eeeeeee
1d-F3ss9X2W01a
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv:1707.066: 1707.06377, 2017.
Alex Sherstinsky, Fundamentals of recurrent neural network (rnn) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132306, 2020.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-022600 5033. IEEE, 2012.
12

---

Under review as a conference paper at ICLR 2021
Cheng Wang and Mathias Niepert. State-regularized recurrent neural networks, In International Conference on Machine Learning, pp. 6596-6606, 2019.
Paul Werbos. Generalization of backpropagation with application recurrent gas market model. Neural networks, 1(4):339-356, 1988.
Paul Werbos. Backpropagation through time: what does and how do it. Proceedings of the IEEE, 18510):1500-1560, 1990.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. Advances in neural information processing systems, 29:4880-4888 2016.
Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In Proceedings of the 37th International Conference on Machine Learning. PMLR 119, 2020.
A PROOFS
Derivation of the Euler's method Jacobian Let fa h)) hr be an ODE-RNN. Then the explicit Euler's method with step-size T is defined the discretization
h4T = h4 + T(fo(z,,,T) - hr)
(5)
h=h
Therefore, state-previous state Jacobian is given
Th+T Oh,
= + TT Of Øh k==
TTI.
(6)
Derivation the Runge-Kutta Jacobian Let h fo(o, h, 7) hr an ODE-RNN. Then the Mange-Kutta method with step-size T is defined the discretization
hHT h Σ (, -hrr) n=k,"
where the coefficients b; and the values K, are taken according to the Butcher tableau with Σ b,
1 and K; hT-
Then state-previous state Jacobian of the Runge-Kutte method given by the following equation
OheTt = I T The
M
a
b h=k.
TTI.,
(8)
0h
1=1
Note that the explicit Euler method is an instance of the Lunge-Kutta method with M and bj 1.
Proof of ODE-RNN suffering from vanishing or exploding gradients Let h (z (2, h, hr be an ODE-RNN with latent dimension N. Without loss of generality let ho be the initial state = and lor denote the ODE state which should computed by numerical ODE-solver. Then ODE-solvers, including fixed-step methods (Runge 1895) and variable-step methods such as the Dormand- -oimand-Prince method (Dormand & Prince 1980), discretize the interval [o, T'] by a series to, t....tt. where = 0 and t,, T and each ht, is computed by single-step explicit Euler or Runge-Kuun Kutta method from h
Our proof closely aligns with the analysis in Hochreiter and Schmidhuber (Hochreiter & Schmidhuber 1997). We refer the reader to (Hochreiter [991]] Bengio et al. 1994; Pascanu et al. 2013) for rigorous discussion the vanishing and exploding gradients.
13

---

Under review as a conference paper at ICLR 2021
We first prove the theorem for scalar RNN, i..., 1, and then extend the discussion the general case. The error-flow per RNN step between t = and = T is given by
Ohr ahg
11 II
M
a
(1+
(1+
tm-1)
(Im tm-1)).
(9)
Oh. h=
which realizes power series depending on the value
Of
(fm (m-1))
b
-T( 1m-m1)]
(10)
Oh.
hum Km,
j=1
Obviously, the condition that this term is equal to 1 is not enforced during training and violated for any non-trivial fa. such as fo(h, .)) o (W,h+W, with being sigmoidal or rectified-linear activation function. The exact magnitude depends on the weights Wh, as
@fe Win (Wih + W b).
(11)
Öh
non-zero time-constant 7 pushes the gradient toward a vanishing region.
Note that the Equation (10) only becomes equal 1, if Σラ 0s
This would imply
h=Km,
that BM 0, i... when the change in ODE-state between two time-points is zero. A variable that Rtm-1 does not change propagation is to include an explicit memory element in the architecture (Hochreiter & Schmidhuber 1997) which does not change its value between two arbitrary time-points bm and 4m-1.
time is memory element. Thus the only solution of enforcing constant-error
For the general case n 2 1, the error-flow per RNN step between = 0 and I = T is given by
OhT Thg
n
M Σ
as
II
(T+ (fm I
bi Th
T(m (m- (m-1))).
(12)
k=
As is a vector, we need to consider all possible error-pree proo-prrpaggtion paths. The error- error-flow from unit to unit is then given by summing all N possible paths between u to
ohe
N
N
M
0s
I++ (lm - 4m-1)
Σ j=1
bs Oh
- T(mm 4m- (m--1)
(13)
Ohy
h=KK;
m
where lo and In U.
The arguments of the scalar case hold for every individual path in Equation 13 between the the scalar case and the individual paths in the vectored version is the non- -on-diagonal connections the general case do not include the constant I and r. The propappropaationn magnitude between u and v with u * v is given by
The only difference
M
1(tm bm-
bi
Oh k==Kmm )
(14)
1=1
Again, for fo( = a (Wah + W, + b) we obtain an error-flow that depends on the weights W7, and can be either vanishing exploding, depending on its magnitude.
Proof that even gradients of the ODE solution can vanish or explode Let fo h T) hr be an ODE-RNN with latent dimension N, with fo being uniformly Lipschitz continuous. Without loss of generality let ho be the initial state t 0 and hy denote the ODE state which should be computed by numerical ooEvsolver We approximate the interval [0, 2'] by a uniform discretization grid, i.e. f ty 1--1 T/n for all i,j fe where h4, is computed by single-step explicit Euler from h4-
0 and f,, T and each
Even when making the discretization grid to f.....t, tn finer and finer, the gradient propagation issue not resolved. Let h; denote the intermediate values computed by the Picard-iteration i.e., the explicit Buler. By the Picard-L.nndllff theorem, know that fyr converges to the true solution h(T).
14

---

Under review as a conference paper at ICLR 2021
First, we assume there exists a 5 > O such that & v
0f
T for all m. Note that this situation
8h
can naturally occur if we have a So(h, = o(Wih + Wax + b). In the limit n - oc we get
ohr Oho
(1 (fm - -m--1) 0f
lim lim 12 200
II
- T(tm tm-1))
n 000
Ōh 1/
m=1
[1 T af
-
T
lim
m=1 n Ōh h=mhm
n1
1 + T n
0f
lim
with some 0 <
for all m
0h kmmh..
mm1
lim n-0
11 {
e TE
> 1,
i.e, we have an exploding gradient.
Conversely, lets assume there exists a such that {
2
8f 8h h-
for all m. Note that this
situation can also naturally occur, for instance if r > and regions where f' is small. In the limit 00 get
0hT Ohu
05 1m-1) 0h k=
lim
lim II
(1 (f.m
T(mm tm-1))
m=1
lim
II m=1
1 +
T Öf dh h=hm
T -
n
Of
< lim
+ {). with some 0 0>
- T for all m
A 700 m=1
Ōh h=hm
T
lim a0000
11 -
TE
i... we have a vanishing gradient.
Similar to the argument in the proof above, we can extend the scalar case to the general case. However, summing over all possible path might not be trivial, the number of possible path also growths to infinity.
Ohe lim Oh
N
12
[ (tm 1m- 0s Oh h=hn
lim R0000
II l-
T(tm
tm-1))1)
(15)
-mmm--1
1
Instead, we assume u diagonal element u.
1, i.c. we only look the error- proo-ppropagation through the
dh. Ohg
0f
lim 120
lim n m=
II (1 (6m - 1m-1)
-T Im 1)1
Oh h: h=
n
05"
lim n-
II
1 + (tm 1m--1)
(tm bm-
Th h==h%
(1+ T 05"
7 T
))
lim
th 100
m=1
15

---

Under review as a conference paper at ICLR 2021
which is equivalent to the scalar case. For interesting such as fo = a (With Wa + b). the term 1. depends the value Wh By assuming W for any (w,, * (u. u) is neglectable small, we can infer that the effects of the gradient by any other path in Equation (15) is neglectable small. Thus the global error flow depends on W which can make the error-flow either explode or vanish depending on its value.
Note that this argument is similar to arguing that the multi-dimensional case properly contains the scalar case, the multi-dimensional case can express an exploding or vanishing gradient too.
Proof that the mmRNNS does not suffer from a vanishing or exploding gradient Recall that we assume that weights of go and are initialized close to O and that we are at the beginning of the training process, i.e., we assume the weights do not differ significantly from their initialized values. Moreover, for the proof we assume that and 20 are standard multi-layer perceptron modules.
We have
0c++1 dct
dc, @ a (he) bb) + 20 (h4)
det diag(c)) + (iag(o((ge(h)) + by)) +
do (go + by)
@zo(he)
de.
dc:
For the derivatives of the first term we can simply apply the chain-rule and get "o (go + by) go (he) (he)
(g( (he) + by)"
Oct
Ock
0,
because we assumed ga (ht) 2 due to its initialization.
Similar argument holds for the derivative of the last term, where we assumed that zq is initialized closet to O.
(he)
der Dc
Pace (Wo2: + Wo)
Oct
0,
where faet is the final activation function of zo. Wo and by the weights and bias parametrizing the last layer of žg and & the last hidden vector of za. Note that we assumed Wa Rs
Consequently, with proper weight initialization, the Jacobian simplifies to
dc,-1 de:
diag(orgo b))).
We assumed that go is initialized close to 0. Hence,
(iag( + by)) = o(b))
=0(3)
2 0.994,
as we initialized by to Hence, we have
N d dek
~ 0.994,
which is less than I (no exploding) but much greater than 0 (no vanishing) and ensures a near- constant error propagation at the beginning of the training process.
As already mentioned in the paper. the exact value of the error flow can controlled by changing the bias term from its default value of 1. If the underlying data distribution contains dependencies with very long time-lag, we can bring the error flow factor closer to by increasing forget gate bias; Thus enabling the mmRNN to learn even very long-term dependencies in the data.
16

---

Under review as a conference paper at ICLR 2021
a
b b
Time
Figure 4: Dense coding
a:A, = 4
a:A,
b:A,
Time
Figure 5: Event-based coding
Figure 6: Dense and event- event-based coding of the same time-series. An event-based coding more efficient than dense coding at encoding sequences where the transmitted symbol changes only sparsely.
B EXPERIMENTAL EVALUATION
For models containing differential equations, we used the ODE-solvers as listed in Table 7 Hyperpa- rameter settings used for our evaluation is shown Table
Batching Sequences of our event-based bit-stream classification task and event-based seqMNIST can have different lengths. To allow an arbitrary batching several sequences, we pad all sequences equal length and apply binary mask during training and evaluation.
B. DATASET DESCRIPTION
The individual datasets are created as follows:
Bit-stream XOR dataset Every data point a block of 32 random bits. The binary labels are created by applying XOR function the bit block, i.e., class A if the number of 1s in the bit-stream even, class B if the number of Is in the bit-stream is odd. For training, a cross-entropy loss on these classes used. The training consists of 100,000 samples, which are less than 0.0024% of all possible bit-streams that can occur. The test set consists of 10,000 samples.
For the event-based encoding, we introduce a time-dimension. The time is normalized such that the complete sequence equals 1 unit of time, i.c., 32 bits corresponds to exactly second. An illustration of the two different encodings is shown in Figure
Person Activity We consider variation of the "Human activity" dataset described in (Rubanova al. 2019) form the UCI machine learning repository (Dua & Graff, 2017). The dataset comprised of 25 recordings of human participants performing different physical activities. The eleven possible activities are "lying down". "lying". "sitting down". "sitting". "standding up from lying", "standing up from", "sitting". "standing up from sitting on the ground", "walking"" "falling". "on all fours". and "sitting on the ground". The objective of this task is to recognize the activity from inertial sensors worn by the participant, i.c., per-timme time-step classification problem. We group the eleven activities listed above into seven different classes, as proposed by (Rubanova et al. 2019).
The input data consists of sensor readings from four inertial measurement units placed the participant's 's arms and feet. The sensors are read at a fixed period of 211 ms but have different phase-shifts in the 25 recordings. Therefore, treat the data irregularly sampled time-series.
The 25 recordings are split into partially overlapping sequences of length 32, to allow an efficient training of the machine learning models.
17

---

Under review as a conference paper at ICLR 2021
Table 7: ODE-solvers used for the different RNN architectures involving ordinary differential
equations
Model
ODE-solver
Time-step ratio
CT-RNN
4-th order Runge-Kutta
1/3
ODE-RNN
4-th order Runge-Kutta
1/3
GRU-ODE
Explicit Euler
1/4
mmRNN
Explicit Euler
1/4
Our results are not directly comparable to the experiments in (Rubanova et al. 2019), as we use a different representation of the input features. While (Rubanova al., (019) represents each input feature a value-mask pair, i.e., 24 input features, we represent the data in the form of 7-dimensional feature vector. The first four entries of the input indicate the senor ID, i.e. which arm or foot, whereas the remaining three entries contain the sensor reading.
Event-based MNIIT The MNIST dataset consists of 70,000 data points split into 60,000 training and 10,000 test samples (LeCun et al., 1998). Each sample is 28-by-28 grayscale image. quantized with 8-bits and represents one out of 10 possible digits, i.c., a number from 10.
We pre-process each sample as follows: We first apply threshold to transform the 8-bits pixel values into binary values. The threshold is 128, on scale where 0 represents the lowest possible and 255 the larges possible pixel value. We further transform the 28-by-28 image into time-series of length 784. Next, we encode binary time-series in a event-based format. Essentially, the encoding step gets rid of consecutive occurrences of the same binary value, i.e., 1, 1, 1, 1 is transformed into (1, 4). By introducing a time dimension, we can compress the sequences from 784 to an average of 53 time-steps.
To allow an efficient batching and training, we pad each sequence to length of 256. Note that no information was lost during this process. We normalize the added time dimension such that 256 symbols correspond to 1 second or unit of time. The resulting task is per-sequence classification problem of irregularly sampled time-series.
Walker2d kinematic modeling Here we create dataset based on the Wa Walker2d-v> OpenAl gym (Brockman et al.) 2016) environment and the MuloCo physics engine (Todorov et al., 2012). Our objective is to benchmark how well the RNN architecture can model kinematic dynamical systems in an irregularly sampled fashion. The learning setup is based on an auto-regressive supervised learning, i.e. the model predicts the next state of the Walker2d environment based the current state.
In order to obtain interesting simulation rollouts, we trained a non-recurrent policy by Proximal Policy Optimization (PPO) (Schulman et al.) (217]) using the Rllib (Liang et al., 2018) reinforcement learning framework. We then collect the training data for our benchmark by performing rollouts on the Walker2d-v2 environment using our pre-trained policy. Note that because the policy is deterministic, there is no need to include the actions produced by the policy in the training data.
We introduce three sources of uncertainty to make this task more challenging. First of all, for each rollout we uniformly sample a checkpoint of policy at 562, 822, 923, or 1104 PPO iterations. Secondly, overwrite 1 1% of all actions by random actions. Thirdly, we exclude 10% of the time-steps, i.e., we simulate trame-skips/frame-dde into irregularly sampled time-series and introduces time dimension.
Note that the last step transforms the rollouts
In total, we collected 400 rollouts, i.e., 300 used for training, 40 for validation, and 60 for testing. For an efficient training, we align the rollouts into sequences length 64. We use the mean-square-errro as training loss and evaluation metric. We train each RNN for 200 epochs and log the validation error after each training epochs. At the end, we restore the weights that achieved the best (lowest) validation emror and evaluate them on the test set.
Reproducibility statement All code and data are submitted for review as a supplement.
18

---

Under review as a conference paper at ICLR 2021
Table 8: Hyperparameters
Parameter
Value
Description
RNN latent dimension
64
number of neurons in the RNN
Minibatch size
256
Optimizer
RMSprop Tieleman & Hinton 2012)
Learning rate
5e-3
Training epochs
500/200
Synthetic/real datasets
19


Reviews:
[{"review_id": "z2mvEXDtl5h", "reviewer": "Reviewer_6YwU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From the theoretical side, combining two well established approaches is certainly not a big leap, but the empirical results appear to justify this procedure. So my main comments focus on those:\n\n1) At least in the main text, there is no information on how hyper-parameters of the different approaches were tuned, and whether the different networks are roughly comparable in terms of trainable parameters. The differences between mmRNN and other methods are sometimes quite small. Without that information it’s difficult to judge whether these are really significant improvements, or due to biasing results toward mmRNN by more extensive exploration of its hyper-parameter space or other design choices. In general there is little information on the setup and training of the competing algorithms.\nI’m particularly surprised that coRNN and Lipschitz RNN which are both continuous time and appear to have superb long-term properties still appear to perform so badly on irregular series in Table 3 (at chance) and Table 6?\n\n2) Along the same lines, I wondered whether the specific irregular task settings may have created an unduly bias for some of the models (those based on continuous time): Why are events aggregated through temporal intervals (“dense coding”), instead of just keeping them as they are but separating them by irregular intervals? Doesn’t this mean that the discrete RNNs are really learning a different type of task since, in contrast to the continuous RNNs, they need to process time-aggregated information, or did I miss something here?\n\n3) In general, I’m not sure any of the used tasks really contain significant long-term dependencies in the sense that the RNN needs to get back to information presented much earlier in the stream to issue a correct output. Sequential MNIST is not very challenging in this regard to my knowledge (perhaps more so in the randomized version). For the activity recognition the authors point out themselves that long memory is not needed. And the bitstream task it appears could be solved by just combining at each time step the last active bit with the present input via XOR (unless I misunderstood something).\nHere I think it would be good to add established long-term tasks in which performance can be charted as a function of temporal delay, like the multiplication (Hochreiter & Schmidhuber) or copy task.\n\nThe theoretical section (3) is somewhat sloppy. For instance, Def. 1 is not a proper definition of the exploding or vanishing gradient problem as used in the literature, which implies that gradients should exponentially go to 0 or infinity as $t \\rightarrow \\infty$. The Jacobians in Def. 1 are time-dependent, so having any of them >1 or <1 is not sufficient for either explosion or vanishing, while having all of them is not necessary. Also, for $\\epsilon>0$ gradients could still exponentially vanish! Assumptions and conditions are not properly introduced in the theorems. Conclusions are unlikely to hold for arbitrary functions $f_\\theta$ (e.g., there may be functions f with identity mapping just in a subspace, https://arxiv.org/abs/1910.03471), in Corollary 2 it’s unclear what ‘ODE-combined-with-GRU/LSTM’ formally means, and the conclusion in Theorem 3 (“… at the beginning of the training process”) will trivially be true for many RNNs upon proper initialization (e.g. https://arxiv.org/abs/1504.00941, https://arxiv.org/abs/2006.12070). Also not sure what ‘non-trainable constant dynamics f=0’ means: As defined, the ODE still contains a decay term, so dynamics is not ‘constant’ everywhere, and either way with f constant, gradients may still explode or vanish.\n\nMore generally, I wondered whether it wouldn’t be more elegant, and potentially efficient, to combine two ODE-based mechanisms into the same architecture (e.g. ODE-RNN and coRNN), rather than combining discrete with continuous time methods. In sect. 7 the authors discuss the drawbacks of heterogeneous architectures, but the same arguments I would think apply to their construct as well.\n\nMinor issues:\n- Strictly, mathematically, LSTMs neither avoid the vanishing nor the exploding gradients problem, just alleviate it.\n- Personally, I didn’t take home much from Fig. 1, and felt that Algo.1 could be omitted as well.\n- Don’t understand some of the notation: For instance, right in eq. 1, h should be a continuous function of t but is indexed by t, and f seems to take as input $x_{t+T}$ a future value, making this some sort of delay DE. Likewise in eq. 4.\n- What is meant by ‘implicit def. of time’?\n- Ideally, the solution to a particular ODE should indeed not depend on the particular numerical solver used!\n- Fig. 3: How were these graphs produced? Are they just hypothetical, or are they based on an actual simulation under same (task) conditions?\n- I would suggest to move sect. 6 up right behind sect. 2. This will give the necessary background for the comparison methods needed in sect. 5.", "summary_of_the_paper": "This paper proposes a mixed ODE-RNN plus discrete time LSTM. This combined architecture enables to deal with irregularly sampled time series while at the same time protecting against the exploding/ vanishing gradient problem. It is shown that this mixture approach is superior to many other ODE-based RNNs and to discrete RNNs specifically designed to combat the exploding/ vanishing gradient problem. ", "summary_of_the_review": "The paper combines two well-known models, the math. sect. lacks a bit precision. Empirically, however, the model seems to perform superbly on irregular time series.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RUtwupDYG-j", "reviewer": "Reviewer_ypgB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n- The analysis demonstrating the persisting problem of RNNs and EVGP in the ODE-RNN method, whist not exactly a surprising result, is good to have written down. It is presented in a palatable way without too much mathematical detail in the main body of the text. \n- The problem is interesting and timely, the proposed solution (if truly a delta over previous work, see below) that controls the error at the start of training is useful and simple for practitioners to adopt.\n- Good experimental verification, extremely wide range of benchmarks considered. Whilst there is little information given to model sizes/numbers of parameters, given the wide array of benchmarks I'm not too worried about this. If information is available it would be good to include it though. \n\n## Cons / Queries\n- Throughout the paper I think the authors are too strong in their wording with regards to how well their model mitigates the EVGP. For example, \"while avoiding the EVGP\", \"Our mmRNNs are immune to this shortcoming\" (about EVGP). Its not that the model can outright avoid or is immune to the issue, rather, initialisation can control the error propagation at the start of training which should enable it to learn longer into the future without issues, but this is not fully avoided.\n- A query about Corollary 2. This corollary states ODE-GRU/LSTM also suffers from EVGP. Well yes, but so does mmRNN. The question is does mmRNN enable better control of the error than a well initialised GRU/LSTM? Unless I am missing some fine-point, it seems incorrect to note that ODE-GRU would suffer an EVGP as if mmRNN doesn't, when it does, it just suffers it less bad, but the authors do not give us insight on the improvement an GRU makes from an RNN so we can compare against the mmRNN. \n- Similar to the above, why have the authors not tested against ODE-GRU/LSTM?? This seems the main benchmark that should be checked which is completely avoided. I want to see the error propagation of a well initialised ODE-GRU in Table 2, and ideally the results of the ODE-GRU with parameter counts *as close as possible to the mmRNN*.\n\n\n#### Typos\n\"Experimental settings are given in Appendix\"\n\"Although , the update\" (under eq 3, the space before the comma)", "summary_of_the_paper": "The authors outline how the ODE-RNN model suffers from the familiar exploding/vanishing gradient problem that we expect from RNNs. They introduce an improvement \"mixed-memory RNNs\" whereby the problem of exploding/vanishing gradients can be mitigated, at least early on in the learning process, resulting in what should be a more stable model for long time series. This theoretical work is backed up by experiment showing improvement over a large range of benchmarks. ", "summary_of_the_review": "This analysis is only of real interest if it is solving an issue not already solved by using an LSTM/GRU gate. If this is the case and can be explained and evidenced with experiment by the authors in the paper then I would opt for accept, for now I am leaving my review as a weak reject.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QPRgOc-SaIk", "reviewer": "Reviewer_pULY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has presented some extensive numerical results that show the effectiveness of the proposed method in improving the performance of continuous-time RNN on irregularly sampled time-series. This is interesting, and potentially useful for practitioners. \n\nIn my opinion, the main issue with the current paper is that some theoretical arguments are presented to attempt to justify the approach, but the mathematical content of these results are lacking. The definition of stability and the proofs of the stability results are very unclear, and generally lack precision. I believe the paper will be much easier to understand if focused entirely on empirical results and studying their implications. Below, I outline some issues that I have trouble understanding while reading the theoretical parts of the paper.\n\n1. Definition 1 requires some explanation. Why is the sum of the derivatives used as a measure of exploding/vanishing gradients? The usual definition of local stability in dynamical systems theory relies on the eigenvalues of the Jacobian, which is not characterized by the sum of the entries of the Jacobian. The sum of absolute values of the Jacobian elements may be a better indication, but still indirect.\n   As an example in 2D, the Jacobian matrix (associated with linear dynamics, thus constant) $((a,-a),(-a,a))$\n   has row sum equal to 0 thus has “vanishing gradients” according to this definition, but is stable if $|a|<1$ and unstable if $|a| > 1$.\n   In other words, the sum of derivatives condition appears neither necessary nor sufficient to ensure stability of dynamical systems. Can the authors comment on what this particular definition means?\n\n2. Proof of theorem 1: It is very hard to unravel what is being proved here. The overall logic appears to be (and the authors may wish to clarify, if this is not the case)\n\n   1. Derive equation (9) for the product of Jacobian up to time $T$ (​minor issue: $T$ has been used previously for the time step of integration, but here became the integration horizon.)\n   2. Argue that for generic network weight values, each term in the product cannot be zero except for trivial cases\n   3. Conclude that the dynamics must either explode or vanish for nontrivial cases.\n   4. The higher dimensional case is similar. \n\n   There are several issues with this argument.\n\n   1. First, it is not necessary that each term in the product is zero, close to 0, or close to 1, for stability conditions to hold, as already detailed earlier.\n   2. Second, even if we take the definition 1 as correct, there is no statement in the proof to show that the condition holds, e.g. what is the explicit estimation of $\\epsilon$ in terms of the network weights, say at typical initialization values?\n   3. Third, while it is the case that there exists weight values for which the RNN dynamics is not stable, such can be said for most parametric dynamical systems. However, what needs to be investigated is whether for a typical initialization, and subsequent training, the stability condition holds. This is not shown in the proof, nor in the statement of the theorem.\n\n3. Implication of Corollary 1: I do not see how it follows that *“Corollary 1 suggests that well-conditioned gradients of an ODE-RNN can negatively impact the modeling capacity of these models and prevent them from learning the dynamics of the training data”*. Corollary 1 says that if the Jacobian is the identity, then the dynamics is trivial. However, the Jacobian being identity is not necessary to ensure stability or well conditioned gradients. For example, in the linear case, the Jacobian being orthogonal is sufficient. In the nonlinear case, Jacobian’s role in stability is even more limited, as only the stability of small perturbations around an equilibrium point is governed by the Jacobian at the point.\n\n4. Proof of theorem 2: \n\n   1. The main issues are the same as theorem 1, i.e. lack of precise proof steps that actually shows the statement to be proved.\n   2. In continuous-time, the derivative of the terminal state of ODE vs initial state is the variational equation, and its properties are well studied, see e.g. https://encyclopediaofmath.org/wiki/Variational_equations or ODE reference texts. The Euler discretization argument here is not rigorous, and not needed. One can study the Jacobian which appears in the variational equation.\n   3. Technical issue: Picard iteration refers to the iterative construction of solutions of IVPs via $\\dot{h}_t^{(n+1)} = h_0 + \\int_{0}^{t} f(h^{(n)}_s, s) ds$ in order to prove existence. The proof presented here is using the Euler discretization, which has nothing to do with Picard iteration. I suggest authors carefully revise the statements here to avoid confusion.\n   4. Most importantly, the proof of the theorem *assumes* that the Jacobian is upper or lower bounded by $\\xi$ with the appropriate signs (opposite in the two cases) to prove explosion/vanishing results. However, this assumption is almost equivalent to the claim of the theorem to be proved. The key issue is how such bounds can arise from RNN initialization and training in the first place. This is claimed but not proved, and thus I cannot understand what result is being proved in this theorem.\n\n5. Theorem 3: The result relies on an explicit *assumption* on initialization of the network ($g_\\theta, z_\\theta \\approx 0$) to ensure stability, and then show that *at* or *close to* this initialization, the stability condition still holds. The key issue is that this can *also* be done for recurrent neural networks without gating. One can initialize the network at network weights that are stable, and with singular values close to but less than 1. Then, the same statement can be made that at and close to initialization, the stability condition holds with no exploding or vanishing gradients. Thus, I do not see how this result effectively differentiates the case with and without gating.\n\n### Minor Issues\n\n1. Runge-Kutta method is not correctly defined. In Appendix equation (7), $f_\\theta$ should be evaluated at time points according to the Butcher Tableau as well, not just at the final $T$. In the current definition, the global convergence error will not be of high order. I believe this is a typo.", "summary_of_the_paper": "This paper proposes a method to alleviate training problems (due to vanishing/exploding gradients) for RNNs by gating. The study focuses on irregularly sampled time series, for which a continuous-time formulation is used for the hidden states and discrete gating mechanisms similar to LSTM is added. It is shown empirically that this addition has favorable effects on a variety of time-series applications. Some arguments are made in the theoretical direction to justify the method.", "summary_of_the_review": "While there are some relevant and potentially interesting numerical results, the theoretical results lack clarity and precision, and thus I cannot recommend publication in the current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FJCjZa0lkD4", "reviewer": "Reviewer_ceHF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To be transparent, I was a reviewer on a previous submission of this paper where the decision to reject was borderline and significant discussion between reviewers and with the authors led to enlightening improvements. As for the previous submission, I very much enjoyed this paper and believe it contributes very interesting ideas to model irregularly sampled time series. The theoretical derivations are sound, the experiments are detailed and well-rounded, despite some issues (see below), and the writing is clear.\n\nI focus my score on two main important points that in my opinion, were not adequately addressed from the previous round.\n\n1.  \"In iRNN (https://openreview.net/forum?id=HylpqA4FwS), an ODE based RNN, the authors clearly proved that no vanishing/exploding gradients exist in iRNN.\" How does the proposed setting differ from this one, and how does this explain the discrepancies in findings about gradient explosion and vanishing ? A discussion about this point, even a brief one, is still lacking.\n\n2. This point is crucial in my opinion. Given that experiments show some small margin with other models in some cases, it would be important to add details about hyper-parameter tuning. If the best parameters used in original papers were used, a motivation as to why this is a good practice in this case should be articulated. It is unclear, given the specific nature of the tasks used here, if these choices of hyper-parameters give a fair comparison between models. Unless I missed this, details about this point are not present in the main text or in the appendix.\n\n", "summary_of_the_paper": "The authors approach the problem of modelling irregularly sampled data with continuous time RNNs. They first show, through a series of theorems, that continuous-time RNNs are expected to suffer from vanishing and exploding gradient problems in both automatic reverse-mode differentiation, and using the adjoint method. They the propose a new continuous-time RNN based on the LSTM, called mixed memory RRN, and show how this mitigates gradient vanishing and exploding, analogously to their discrete-time versions. They then argue the this new model is better suited to irregularly sampled time series due to its abilities to capture long-term dependencies in data streams. They verify this with numerical experiments, and compare their model performance to other continuous-time RNN architectures.", "summary_of_the_review": "I am tending to a weak accept of this paper given two outstanding concerns described above. These concerns are based on extensive discussions between reviewers in a previous submission cycle, and I would like to better understand how the authors built on these remarks before I adjust my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FDAPgBfsI6P", "reviewer": "Reviewer_X1Ep", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main strengths:\n* Points out a major shortcoming in the ODE-RNN approach and proves it applies to all previous methods.\n* Proposes a new method to overcome it, proves this, and demonstrate empirically that it does work better in practice by a significant margin.\n* Very clear presentation.\n* Also compares against a basic baseline of non-continuous time. This is important, because it shows the advantages of the continuous formulation.\n\nMain weaknesses:\n* I am not familiar with this formulation of the exploding/vanishing gradients problem, and it would be worth explaining how it relates to the more common understanding of the phenomenon. The more common definition that I am aware of is that the absolute value of the derivative dL/dh_t grows/decays exponentially for some paths of neurons. Specifically, why 1 ± epsilon is sufficient for explosion/vanishing in the given neuron? I can see how a sequence of such \"slightly\" small/large gradients can accumulate to vanishing/explosion, but why would a single i for a specific time be sufficient? Shouldn't it be required to hold at all points in time?\n* While most of the paper is written in a very clear language, the introduction to ODE-RNN is a bit brief. Extra background on the subject (perhaps as part of the appendices) would be very useful.\n\n\nNote: I am afraid that I am not sufficiently familiar with ODE-RNN to verify the correctness of the proofs.", "summary_of_the_paper": "The authors prove the ODE-RNN and many of its variants suffer from vanishing/exploding gradients. They show that this problem persists even when employing some of the previously proposed methods to overcome it. Moreover, the authors suggest a new method to mitigate this issue while keeping the ODE-RNN properties useful for irregularly-sampled sequences. They demonstrate that their theoretical model works in practice better than previous methods, including against a \"simplified\" non-continuous RNN augmented with a time input.", "summary_of_the_review": "Overall this is a strong submission. It clearly presents the problem with prior approaches and proposes a solution, and proves it indeed corrects the issue. However, because I am not as familiar with the specifics of ODE-RNN and could not verify the correctness of the proofs, I am not fully confident in my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After carefully reading the reviews and the rebuttal I feel the paper fails slightly short. 

Unfortunately some of the issues that I have are aligned with the feedback from reviewer 6YwU and pULY. 
A significant part of the paper is the formalism and theory introduced by this work, followed then by the empirical evaluation. The theory I feel is not sufficiently well formulated. I understand this is a complex topic, and one can only make minimal statements about a system (particularly when learning is involved). And I understand that the authors are looking at a slightly different phenomena, and not the traditional vanishing/exploding gradient problem, where they consider a per-unit scenario. And I believe one can make a case that this alternative definition holds value and should be investigated. 

However, I believe being more explicit of this alternate view, and make sure that one does not go into the theory with the wrong preconception of what these results are about is important. And secondly making sure the claims are adequate is important and not overly strong (or over claiming). I think this is important particularly in such works, dealing with systems that do not allow a full mathematical analysis.  In particular, just to give some examples:
 1. Thm 3, pointed out by the reviewers as well. I don't understand the point of this thm. It basically says that around initialization things are well behaved. The same can be said or proven for many other methods. You argue that this is different, as in other models beside initialization forgetting is not controlled, while you could potentially control it by a forgetting gate. However this is not a theoretical, precise argument. The forgetting gate is learned as well. If we go back to the LSTM scenario, LSTM suffer for vanishing gradients. Also Gers et al. paper does not prove that trying to preserve error has to harm learning (it provides some empirical evidence that is the case, but there have been many other things that affected this results). The point here is not that forget gates are not useful, nor that the gating mechanism proposed by LSTM are not extremely useful. They are. Is that the Thm 3 can not prove or show that using mmRNN is a better way of mitigating (and trading of) vanishing gradient than another model. You do that through your empirical evidence, and I think that is how most of ML works. But is not clear what the point of the theorem is. 
 2. I do not understand how one reasons theoretically about epsilon in Def 1. I don't see how an empirical observation by Gers et al resolves this. It justifies maybe why vanishing gradients are not always problematic, but that should not affect the definition of what vanishing now means. In the current form, if T goes to infinite, even if technically the network does not suffer from vanishing gradients, the gradients go to 0. Or at least T and epsilon should somehow be tied together to make the definition work. 
 The issue of defining the vanishing / exploding gradient per unit is also that now is not clear what is problematic or not. Probably having exploding gradient for any given unit is bad, as it might affect the overall gradient. But having a few units suffering from vanishing gradient, is that problematic?  This things need to be quantified better.

I think overall to me the problem is that some of this mathematical statements do not seem to be strong enough or contextualized enough to be properly understood by the reader. I would have understood the formalism if it was trying to correct some misconception in the community, case in which it is important to formalize just to be precise. But I don't think this is what is happening here. As in stands it just feels sloppy.

And I think this retracts considerably from the empirical side of the work and reduces the space you had to give it enough attention. Which should have played main stage. I think the empirical work would have benefited from more analysis (showcasing some of the arguments you were making using the theory), which would have made for a much stronger and convincing paper. The current framing of the paper is unfortunately not the right one.
Decision: Reject