Anamnesic Neural Differential Equations with Orthogonal Polynomial Projections
Authors: Edward De Brouwer
Rahul G Krishnan
Venue: ICLR 2023 poster (2023)
Keywords: Neural ODEs
Time Series
Orthogonal Polynomials
Long-term memory
Representation Learning

Abstract: Neural ordinary differential equations (Neural ODEs) are an effective framework for learning dynamical systems from irregularly sampled time series data. These models provide a continuous-time latent representation of the underlying dynamical system where new observations at arbitrary time points can be used to update the latent representation of the dynamical system. Existing parameterizations for the dynamics functions of Neural ODEs limit the ability of the model to retain global information about the time series; specifically, a piece-wise integration of the latent process between observations can result in a loss of memory on the dynamic patterns of previously observed data points. We propose PolyODE, a Neural ODE that models the latent continuous-time process as a projection onto a basis of orthogonal polynomials. This formulation enforces long-range memory and preserves a global representation of the underlying dynamical system. Our construction is backed by favourable theoretical guarantees and in a series of experiments, we demonstrate that it outperforms previous works in the reconstruction of past and future data, and in downstream prediction tasks.

Published as a conference paper at ICLR 2023
ANAMNESIC NEURAL DIFFERENTIAL EQUATIONS WITH ORTHOGONAL POLYNOMIALS PROJECTIONS
Edward De Brouwer
Rahul G. Krishnan
ESAT-STADIUS
Department of Computer Science
KU Leuven
University of Toronto
Leuven, Belgium
Toronto, Canada
edwa rd. www.oewewrkulleeeeeeeeeeee.. be
rahul g cs twwotto.ood.eoo.oo
ABSTRACT
Neural ordinary differential equations (Neural ODEs) are effective framework for learning dynamical systems from irregularly sampled time series data. These models provide continuous-time latent representation of the underlying dynam- ical system where new observations at arbitrary time points can be used to update the latent representation of the dynamical system. Existing parameterizations for the dynamics functions of Neural ODEs limit the ability of the model to retain global information about the time series; specifically, a piece-wist peice-wise integration of the latent process between observations can result in loss of memory on the dynamic patterns of previously observed data points. We propose PolyODE, a Neural ODE that models the latent continuous-time process as projection onto a basis of orthogonal polynomials. This formulation enforces long-range mem- ory and preserves global representation of the underlying dynamical system. Our construction is backed by favourable theoretical guarantees and a series of experiments, we demonstrate that it outperforms previous works in the recon- struction of past and future data, and downstream prediction tasks. Our code is available lttttts: htt.://w..coom/eeeeoueeee///////
INTRODUCTION
Time series are ubiquitous in many fields of science and as such, represent an important but chal- lenging data modality for machine learning. Indeed, their temporal nature, along with the potentially high dimensionality makes them arduous to manipulate as mathematical objects. line of research has thus focused on efforts in learning informative time series representations, such as simple vectors, that are capable of capturing local and global structure in such data (Franceschi al. 2019 Gu al. 2020 Such architectures include recurrent neural networks (Malhotra al. 2017). temporal transformers (Zhou et al. 202I and neural ordinary differential equations (neural ODEs) Chen et al. 2018).
long-standing
In particular, neural ODEs have emerged as popular choice for time series modelling due to their sequential nature and their ability to handle irregularly sampled time-series data. By positing an underlying continuous time dynamic process, neural ODEs sequentially process irregularly sam- pled time series via piece-wise numerical integration of the dynamics between observations. The flexibility of this model family arises from the use of neural networks to parameterize the tempo- ral derivative, and different choices of parameterizations lead to different properties. For instance, bounding the output of the neural networks can enforce Lipschitz constants over the temporal pro- cess (Onken et al. 2021).
The problem this work tackles is that the piece-wise integration of the latent process between obser- vations can fail to retain a global representation the time series. Specifically, each change to the hidden state of the dynamical system from observation prior dynamical states the model was originally in. This pathology limits the utility neural ODEs when there is necessity to retain information about the recent and distant past; iee. current neural
result in loss of memory about

---

Published as a conference paper at ICLR 2023
ODE formulations amnesic. We illustrate this effect in Figure where we see that backward integration of a learned neural ODE (that is competent at forecasting) quickly diverges, indicating the state only retains sufficient local information about the future dynamics.
Figure I: PolyODE: Illustration ity of PolyODE to reconstruct past trajecto- ries. The solid lines show the forecasting tra- jectories conditioned on past observations for NODE (blue) and PolyODE (red). The dot- ted line represents the backward reconstruc- tion for the past trajectories conditioned on the latent process at the last observation. We observe that PolyODE is able to accurately reconstruct the past trajectories while NODE quickly diverges. PolyODE is also more accu- rate in terms of forecasting.
abil-
1.3
0.5
0.0
Data
Poly0DE Reconstructins
HODE Predictions
Poly00E Prodictions
Time
10
One strategy that has been explored in the past to address this pathology is to regularize the model encourage it to capture long-range patterns by reconstructing the time series from the last obser- vation, using an auto-encoder architecture (Rubanova et al.) 2019). This class of approaches results in higher complexity and does not provide any guarantees on the retention of the history of a time series. In contrast, our work proposes an alternative parameterization of the dynamics function that, by design, captures long-range memory within neural ODE. Inspired by the recent successes of the HiPPO framework (Gu et al.) 2020), we achieve this by enforcing that the dynamics of the hid- den process follow the dynamics of the projection of the observed temporal process onto a basis of orthogonal polynomials. The resulting model, PolyODE, is new neural ODE architecture that en- codes long-range past information in the latent process and is thus anamnesic. As depicted in Figure the resulting time series embeddings are able reconstruct the past time series with significantly better accuracy.
Contributions (1) We propose a novel dynamics function for neural ODE resulting in PolyODE, a model that learns a global representation of high-dimensional time series and is capable of long- term forecasting and reconstruction by design. PolyODE is the first investigation of the potential of the HiPPO operator for neural ODEs architectures.
(2) Methodologically. we highlight the practical challenges in learning PolyODE and show how adaptive solvers for ODEs can overcome them. Theoretically, we provide bounds characterizing the quality of reconstruction of time series when using PolyODE.
(3) Empirically, we study the performance of our approach by assessing the ability of the learnt embeddings reconstruct the past of the time series and by studying their utility as inputs for downstream predictive tasks. We show that our model provides better time series representations, relative to several existing neural ODEs architectures, based on the ability the representations to accurately make predictions several downstream tasks based on chaotic time series and irregularly sampled data from patients in intensive care unit.
2 RELATED WORK
Time series modelling in machine learning: There is vast literature the use of machine learning for time series modelling and we highlight some of the ideas that have been explored to adapt diverse kinds of models for irregular time series data. Although not naturally well suited to learning representations of such data, there have been modifications proposed to discrete-time models such as recurrent neural networks (Hochreiter and Schmidhuber 1997 Cho et al. 2014 to handle such data. Models such as mTANs (Shukla and Marlin (021) leverage an attention- based approach to interpolate sequences to create discrete-time data from irregularly sampled data. Another strategy has been architectural modifications to the recurrence equations e.g. CT-GRU (Mozer et al., 2017), GRU-D (Che et al. 2018) and Unitary RNNs (Arjovsky et al. 2016). Much more closely aligned to our work, and natural fit for irregularly sampled data is research that uses differential equations to model continuous time processes (Chen et al. 2018). By parameterizing the derivative of a time series using neural networks and integrating the dynamics over unobserved time points, this class of models is well suited to handle irregularly sampled data. This includes models such as ODE-
RNN (Rubanova al. 2019) ODE-LSTM (Lechner and Hasani 2020 and Neural CDE (Kidger
2

---

Published as a conference paper at ICLR 2023
et al. 2020). ODE-based approaches require the use of differential equation solvers during training and inference, which can come at the cost of runtime (Shukla and Marlin 202I). PolyODEs lic in this family of models; specifically, this work proposes new parameterization of the dynamics function and a practical method for learning that enables this model family to accurately forecast the future and reconstruct the past greatly enhancing the scope and utility of the learned embeddings.
Orthogonal polynomials: PolyODEs are inspired by a rich line of work in orthogonal decomposi- tion of time series data. Orthogonal polynomials have been mainstay in the toolkit for engineering (Heuberger et al. 2003 and uncertainty quantification (Li et al. 201T). In the context of machine learning, the limitations of RNNs to retain long- seng-term memory have been studied empirically and theoretically (Zhao et al. 2020). Indeed, the GRU (Chung et al., 2014p and LSTM (Graves et al, 2007) architectures were created in part to improve the long-term memory of such models. Recent approaches for discrete-time models have used orthogonal polynomials and their ability to repre- sent temporal processes in a memory-efficient manner. The Legendre Memory Unit (Voelker al.) 2019) and Fourier Recurrent Unit can be seen projection of data onto Legendre polynomials and Fourier basis respectively.
Our method builds upon and is inspired by the HiPPO framework which defines operator to com- pute the coefficients of the projections on a basis of orthogonal polynomials. HiPPO-RNN and S4 the most prominent examples of architectures building upon that framework (Gu et al 2020
2021]. These models rely on a linear interpolation of the data in between observations, which can lead to decrease of performance when the sampling rate of the input process is low. Furthermore, HiPPO-RNN and S4 perform the orthogonal polynomial projection of non-invertible representa- tion of the input data, which therefore doesn't enforce reconstruction in the observation space by design. Their design choices are motivated toward the goal of efficient mechanisms for capturing long term dependency for a target task (such as trajectory classification). contrast, this work aims at exploring the abilities of the HiPPO operator for representation learning of irregular time series, when the downstream task not known in advance.
Despite attempts to improve the computational performance of learning from long-term sequences (Mrrill et al. 2021). ,, to our knowledge, PolyODE is the first work that investigates the advantages of the HiPPO operator in the context of memory retention for continuous time architectures.
BACKGROUND
Orthogonal Polynomial Projections: Orthogonal polynomials are defined with respect to a mea- sure μ sequence of polynomials (Po(s), (s),.... such that deg( i and
(Pr. Pm.) = Pa(s) (s((s) = 8n=m0n:
(1)
where an are normalizing scalars and 5 is the Kronecker delta. For simplicity, we consider only absolutely continuous measures with respect to the Lebesgue measure, such that there exists a weight function w(-) such that du(s) w(s)ds. The measure f determines the class of polynomials obtained from the conditions above (Eq. D Examples include Legendre. Hermite or Laguerre classes of orthogonal polynomials. The measure l also defines inner product (:: 1) such that the orthogonal projection of a 1 1-dimensional continuous process f(-) : IR + IR on the space of polynomials of degree N, PN. given as
1 an
(x Σ (. n=0
with cn = (f,Pa) ((s))s()))
(2)
This projection minimizes the distance ||/ for all p E Px and is thus optimal with respect to the measure μ. One can thus encode process by storing its projection coefficients {co, CN We write the vector of coefficients up to degree N as (the degree N is omitted) and c, q, Intuitively, the measure assigns different weights at times of the process and thus allows for modulating the importance of different parts of the input signal for the reconstruction.
Continuous update of approximation coefficients: The projection of a process f onto a basis of orthogonal polynomials provides optimal representation for reconstruction. However, there often a need to update this representation continuously as new observations of the process f become available. Let fee be the temporal process observed up until time f. We wish to compute
3

---

Published as a conference paper at ICLR 2023
the coefficients of this process different times We can define for this purpose time-varying measure 1¹ and corresponding weight function w' that can incorporate our requirements in terms of reconstruction abilities over time. For instance, if one cares about reconstruction of process A temporal units in the past, one could use a time-varying weight function (*) = T[s E (t A, t)]- This time-varye varying weight function induces time-varying basis of orthogonal polynomials P for n = O,... N. We can define the tire-ng orthogonal projection and its coefficients C, (t)
N
S = Σ (1)
1 "
with (1) = (s<t: pa) = (xs( -
(3)
2=0
Computing the coefficients of the projection each time step would be both computationally wasteful and would require storing the whole time series memory, going against the principle of sequential updates to the model. Instead, we can leverage the fact that the coefficients evolve according to known linear dynamics over time. Remarkably. for a wide range of varying measures μ². Gu et al. (2020) show that the coefficients cn (t) follow:
Dynamics of the projection coefficients:
dc,(t) d dt di {=)(s) Vn E E dc(t) = ( Btt
(4)
di
where A, and B, are fixed matrices (for completeness, we provide a derivation of the relation for the translated Legendre measure in Appendix A). We use the translated Legendre measure in all our experiments. Using the dynamics of Eq. + possible to update the coefficients the projection sequentially by only using the incoming sample f(t), while retaining the desired reconstruc- tion abilitics. Gu et al. (220)) use a discretization of the above dynamics to model discrete time- sequential data via recurrent neural network architecture. Specifically, their architecture projects the hidden representation of an RNN onto a single time series that is projected onto an polynomial basis. Our approach differs in two ways. First, we work with continuous time model. Second, we jointly model the evolution of d-dimensional time-varying process as a overparameterized hidden representation that uses orthogonal projections to serve as memory banks. The resulting model is new neural ODE architecture as we detail below.
4 METHODOLOGY
Problem Setup. We consider a collection of sequences of temporal observations X = {(x;, m, f.) i E (I,...,T) T}} that consist of set of time-stamped observations and masks (x; Rd Rd,4, 4, E R). We write Xi.j and m.j for the value of the j dimension x; and m, respectively. The mask m, encodes the presence of each dimension X, is observed and m,jj = otherwise. The number observations for each sequence x, T, can vary across sequences. We define the set of sequences S and the distance between two time series observed at the same times as d(x, x') -
specific time point. We set m.,) -1
||x || - -XI12- Our goal is be able embed sequence x into a vector h E Rd such that (1) retains a maximal amount of information contained in x and (2) h is informative for downstream prediction tasks. We formalize both objectives below. Definition (Reverse reconstruction). Given an embedding h, of a time series x at time t, we define the reverse reconstruction & as the predicted values of the time series at times prior to t. We write the observed time series prior to as Xet-
Objective (Long memory representation). Let he and hy be two embeddings of the same time series x. Let &et and X2 be their reverse reconstruction. We say that h, enjoys more memory than h if Xx() < ((&2t:X<1)
Objective 2 (Downstream task performance). Let y E Rdy be an auxiliary vector drawn from unknown distribution depending x. Let (xx) and ((x) " be the predictions obtained from embeddings he and h/. For a performance metric a : - x TRd, - R, we say that by is more Ex .)).
informative than h Ex (xxxy(l((((x)))))
4.1 POLYODE: ANAMNESIC NEURAL ODES
We make the assumption that the observed time series x comes from an unknown but continuous temporal process x().. Given h(!) dRd and read-out function g Rdd - Rdd we posit the
4

---

Published as a conference paper at ICLR 2023
following generative process for the data:
dh(() ((((t)
x(1) - s(h(t),
(5)
di
where part of ((-) is parametrized via neural network (o(-).
The augmentation of the state space is known technique to improve the expressivity of Neural ODEs (Dupont et al. 2019). Here, to ensure that the hidden representation in our model has the capacity retain long-term memory, we augment the state space of our model by including the dynamics of coefficients of orthogonal polynomials as described in Equation
Similarly classical filtering architectures (e.g. Kalman filters and ODE-RNN (Rubanova al. 2019), PolyODE alternates between two regimes an integration step (that takes place in between observations) and update step (that takes place at the times of observations), described below.
We structure the hidden state as h(t) [b (t), (t)..... ha(()] where (o E R has the same E d, has the same dimension the vector of projection coefficients c' (t) and [. is the concatenation operator. We define the readout function ge(.) R (N+1)d -> IR such that g: (h(()) = (t),, That is, is fixed and returns the 2th value of the input vector. This leads the following system of ODEs that characterize the evolution of h(t): Integration Step.
dimension as the input process x, h; (t)
de' (t)
= At) + B,gi (h()))
dt
(6)
(t) d8
= (t) BBga(b((t))))
db(t)
(o(((t))
This parametrization allows learning arbitrarily complex dynamics for the temporal process x. We define a sub-system of equations of projection coefficients update for each dimension of the input temporal process x(t) Fd This sub-system equivalent Equation where we have substituted the input process by the prediction from the hidden process h(t) through a mapping ge(-). The hidden process ho (f) acts similarly as in classical Neural ODEs and the processes c(t) captures rong-range information about the observed time series. During the integration step, we integrate both the hidden process htt) and the coefficients c(t) forward in time, using the system of Equation provide estimate of the time series x(t) conditioned the hidden
& At each time step, process htt), with x(t) = gh((t)).
The coefficients c(t) are influenced by the values of h(t) through ho(t) only. The process ho provides the signal that will be memorized by projecting onto the orthogonal polynomial basis. The c(t) serve memory banks and do not influence the dynamics of h()) during the integration step.
x(t)
Reverse roconstruction t (t)
((())
Forward prodiction 8x
holte)
hoft))
hults)
ft (
Iu(t))
dtt
hdtts)
b(tt)
Downnstroam Prodiction
a(to]
c(sc)
c(ta)
Figure 2: PolyODE time series embedding process. The model processes the time series sequentially by alternating between integration steps (belween observations) and update steps when observations are collected. Informative embeddings should allow for (1) reconstructing the past of the time series (reverse reconstruction in red). (2) forecasting the future of the sequence (forward prediction in blue) and (3) being informative for downstream predictions (in green).
5

---

Published as a conference paper at ICLR 2023
The system of equations in Eq. 0 characterises the dynamics in between observations. When new observation becomes available, we update the system as follows.
Update Step. At time = t,. after observing x, and mask m, we ser
(h)(4)) = ho(4)) Xi.j Vj s.t. mijj =
:2))=)))))) (44,, Vj s.I. maj =1
The update step serves the role of incorporating new observations in the hidden representation of the system. proceeds by (1) reinitializing the hidden states of the system with the orthogonal polynomial projection coefficients c(t): (t, == (t();; and (2) resetting ho(t) to the newly collected obscrvation: hot)) = (t)
Remarks: Our model blends orthogonal polynomials with the flexibility offered in modelling the observations with NeuralODEs. The consequence of this that while the coefficients serve as memory banks for each dimension of the time series, the Neural ODE over (t(() can be used to forecast from the model. That said, we acknowledge that significant limitation of our current design is the need for the hidden dimension to track N coefficients for each time-series dimension. Given that many adjacent time series might be correlated, we anticipate that methods to reduce the space footprint of the coefficients within model is fertile ground for future work.
4.2 TRAINING
We train this architecture by minimizing the reconstruction error between the predictions and the observations: L Σi= [[x(()) 11/2- We first initialize the hidden processes c(0) 0 and h(0) = 0 though they can be initialized with static information b, if available (e.g. h(0) = the (6)). We subsequently alternate between integration steps between observations and update steps at ob- servation times. The loss is updated at each observation time t. A pseudo-code description of the overall procedure is given in Algorithm D
Numerical integration. We integrate the system of differential equations of Equation о using dif- ferentiable numerical solvers as introduced in [hen et al., (2018). However, one of the technical challenges that arise with learning PolyODE is that the dynamical system in Equation c relatively stiff and integrating this process with acceptable precision would lead to prohibitive computation times with explicit solvers. To deal with this instability we used an implicit solver such Backward Euler or Adams-Moulle Mouut-Mooloon for the numerical integration (Sauer 201T). A comparison of numerical integration schemes and an analysis of the stability of the ODE available in Appendix !
Algorithm 1: PolyODE Training
Data: x, matrices Au. By, number of dimensions d, number of observations number of polynomial coefficients N
Result: Training loss L over a whole sequence x
1" #+0 0
Initialize h (0) = (0) = ON. Yj L....,d, d,
Loss C =
for i + 1 to 7 do
Integrate C₁ (f) and (t)...(t)) from & = t" until t x; t ho
Update C1 Ci....att,) (t.) and ho... (t)) with x,, m,
L= L + [1(x; x;) @ m
+ t4
end
Forecasting:
From time f, forecast the time series arbitrary time t" as:
x>((1"== g((())=(((())+) + f. do (b()))d),,,
(8)
where (b)+)) is the learned model that we use in the integration step and introduced in Eq.

---

Published as a conference paper at ICLR 2023
Reverse Reconstruction: Using Equation series at any time using the projection coefiicients part of the hidden process:
compute the reverse reconstruction of the time
N
1 o
(t
P
n=0
More details about this reconstruction process and its difference with respect to classical NODEs are available in Appendix B The error between the prediction obtained during the integration step, (t), and the above reconstruction estimator is bounded above, as Result (c.ultt4.1sso shows.
Result 4.1. For a shifted rectangular weighting function with width A, w'(x)= a, (which generate Legendre polynomials), the mean square error between the forward (&) and reverse pre- diction (Rcz) each time is bounded by:
L2 + | 1x-Xcth 8 Co ((2N-1) + CIAL(K + 1]SKE ( (2.N) C25 (2.N):
(3
where is the number of observations in the interval [t -t- [t- L is the Lipschitz constant of the forward process, N is the degree of the polynomial approximation and E(--)) is the Hurwitz zeta function. SK -
x1} the sum of absolute errors between the forward process and observations incurred at the update steps. Co, C1 and C2 are constants.
Expectedly, the bound goes to as the degree of the approximation increases. The lower cumulative absolute error SK also leads to a reduction of this bound. As the cumulative absolute error Sy and our loss function L share the same optimum, for fixed A, L, K and N, our training objective there- fore implicitly enforces minimization of the reconstruction error. This corresponds optimizing Objective [ where we set d(x,x') = ||x xI|,,-- Our architecture thus jointly minimizes both global reconstruction and forecasting error. Notably, when Sx = 0, this result boils down to the well-known projection error for orthogonal polynomials projection of continuous processes (Canuto and Quarteroni 1982). What is more, increasing the width of the weighting function (increasing A) predictably results in higher reconstruction error. However, this can be compensated by increasing the dimension of the polynomial basis accordingly. We also note a quadratic dependency on the Lipschitz constant of the temporal process, which can limit the reverse reconstruction abilities for figg-frreuency components. The full proof can be found in Appendix B
EXPERIMENTS
We evaluate our approach on two objectives: : (1) the ability of the learned embedding to encode global information about the time series, through the reverse reconstruction performance (or orization) and (2) the ability of embedding to provide an informative input for a downstream task. We study our methods on the following datasets:
Synthetic Univariate. We validate our approach using a univariate synthetic time series. We sim- ulate 1000 realizations from this process and sample it at irregularly spaced time points using Poisson point process. For each generated irregularly sampled time series x, we create binary label y (5) 0.5). Further details about datasets are to be found Appendix G
Chaotic Attractors Chaotic dynamical systems exhibit a large dependence of the dynamics on the initial conditions. This means that noisy or incomplete evaluation of the state space may not contain much information about the past of the time series. We consider two widely used chaotic dynamical systems: Lorenz63 and 5-dimensional Lorenz96. We generate 1000 irregularly sampled time series from different initial conditions. We completely remove one dimension of the time series such that the state space is never fully observed. This forces the model to remember the past trajectories to create accurate estimate of the state space each time
MIMIC-IIII dataset. We use a pre-procsssed version of the MIMIC-III dataset (Johnson et al, 2016 Wang et al. 2020). This consists of the first 24 hours of follow-up for ICU patients. For each time series, the label is the in-hospital mortality.
Baselines: We compare approach against two sets of baselines: Neural ODEs architecture and variants of recurrent neural networks architectures designed for long- terg-term memory. To ensure fair comparison, we use the same dimensionality of the hidden state for all models.
7

---

Published as a conference paper at ICLR 2023
Neural ODE baselines. We use filtering implementation of Neural ODEs, GRU-ODE-Bay Bayes (De Brouwer et al. 2019) and ODE-RNN (Rubanova et al. 2019). an auto-encoder relying on a Neural ODE for both the encoder and the decoder part. For theses baselines, we compute the reverse reconstruction by integrating the system of learnt ODEs backward in time. In case of ODE-RNN, the ODE of the decoder. Additionally, we compare against Neural RDE neural controlled differential equations for long time series (Neural RDE) (Morrill et al. p02Tp.
Long-term memory RNN baselines. We compare against HiPPO-RNN (Gu al. 2020 recur- rent neural network architecture that uses orthogonal polynomial projections of the hidden process. We also use a variant of this approach where we directly use the HiPPO operator on the observed time series, rather than on the hidden process. We call this variant HiPPO-obs. We also compare against $4, an efficient state space model relying on the HipPO matrix (Gu et al. 2021
Long-range representation learning: For each dataset, we evaluate our method and the various baselines on different tasks. Implementation details are available in Appendix
Downstream Classification. We train the models on the available time series. After training, we extract time series embedding from each model and use them as input to a multi-layer perceptron trained to predict the time series label y. We report the area under the operator-charaactrrstti curve evaluated on a left-out test set with 5 repetitions.
Time Series Reconstruction. Similarly as for the downstream classification, we extract the time se- ries embeddings from models trained on the time series. We then compute the reverse reconstruction &ct and evaluate the MSE with respect to the true time series.
Forecasting. We compare the ability of all models to forecast the future of the time series. We compute the embedding of the time series observed until some time tooad and predict over a horizon Unoizzon We then report the MSE between the prediction and true trajectories.
Table I: Downstream task and reverse reconstruction results for synthetic and Lorenz datasets.
Model
Downstream Classificationt
Reconstruction!
Synthetic
Lorenz63
Loren296
Synthetic
Lorenz63
Lorenz96
Irregular Rate
0.7
0.3
0.3
0.7
0.3
0.3
GRU-ODE
0.968 + 0.004
0.825 + 0.031
0.925
0.004
0.057
0.010
0.752 + 0.057
0.346 - 0..02
ODE-RNN
0.870 + 0.032
0.813 = 0.013
0.9541 :: 0.012
0.080 1: 0.036
0.674 = 0.049
0.214 0.030
Neural-RDE
0.773 + 0,111
0.601 : 0.046
0.606
0.111
0.167 1 0.031
0.989 = 0.074
1.717 = 0.172
HiPPO-obs
0.758 + 0.023
0.837 + 0.034
0.949 0.007
0.197 0.010
0.511 0.043
0.247 -0.005
HiPPO-RNN
0.742 : 0.008
0.801 0.023
0.911 0.008
0.209
:: 0.018
0.781 = 0.122
0.198 = 0.014
0.994 0.003
0.911 0.005
0.948 0.016
0.032 0.006
0.428 0.040
0.171 0.008
PolyODE
0.994 1 0.003
0.992 0.000
0.984
0.012 0.002
0.034 0.008
0.038 0..08
Results for these tasks are presented in Table for Synthetic and Lorenz datasets and in Table for MIMIC. We report additional results in Appendix C with a larger array of irregular sampling rates. We observe that the reconstruction abilities of PolyODE clearly outperforms the other baselines, for all datasets under consideration. A similar trend is to be noted for the downstream classification for the synthetic and Lorenz datasets. For these datasets, accurate prediction of the label y requires a global representation of the time series, which results in better performance for our approach.
For the MIMIC dataset, our approach compares favourably with the other meth- ods for the downstream classification ob- jective and outperforms other methods for trajectory forecasting. What is more, the reconstruction ability of PolyODE is significantly better than compared ap- proaches. In Figure S we plot the reverse reconstructions of ODE for several vitals of a random patient over the first 24 hours in the ICU,
Table 2: Performance on MIMIC-III dataset.
Method
Classification f
Forccusting 1
Reconstruction 4
HiPPO-obs
0.793 0.002
0.775 4: 0.000
HiPPO-RNN
0.764 + 0.006
1.104 + 0.009
0.969 + 0.026
GRU-ODE
0.793 + 0.005
0.005
1413
2025.6 + 2365.
ODN-RNN
0.800 0.004
1.104 + +-0026
6.343 + 4.844
PolyODE
0.778 + 0.005
1.085 0.022
0.187 + 0.005
This reconstruction is obtained by first sequentially processing the time series until t = 24 hours and subsequently using the hidden process to reconstruct the time series as in Equation We observe that PolyODE can indeed capture the overall trend of the time series over the whole history.
Ablation study the importance of the auxiliary dynamical system: Is there utility in leveraging the neural network (o(-) to learn the dynamics of the time series? How well would various interpolation
8

---

Published as a conference paper at ICLR 2023
heart mean blood pressure oxygen saturation
Figure 3: PolyODE: Reverse predic- tion of vitals over the 24 hours of ICU of a randomly selected test patient. We plot the true value (dots) and reconstructions (solid line) for different vitals. Reverse reconstruction is done from the last time ohservation. Other vitals are provided in Appendix
Normallzed
Hours IGU
schemes for irregularly sampled observations perform in the context of reverse reconstruction and classification? In response to these questions, we first note that they do not support extrapolation and are thus incapable of forecasting the future of the time series. However, we compare the performance in terms of reverse reconstruction and classification in Table 3 We consider constant interpolation (last observation carried forward), linear interpolation and Hermite spline interpolation. Our results indicate a significant gap in performance between PolyODE and the linear and constant interpolation schemes. The Hermite spline interpolation allows us to capture most of the signal needed for the downstream classification task but results in significantly lower performance in terms of the reverse reconstruction error. These results therefore strongly support the importance of Φe(-) (-) for producing informative time series embeddings. Complementary results are available in Appendix
Table 3: Impact of the interpolation scheme on performance.
Downstream Classificationf
Reconstruction
SimpleTraj
Lorenz
Lorenz96
SimpleTra Traj
Lorenz
Loren296
Ireegular Rate A
0.7
0.3
0,3
0.7
0.3
0.3
Constant
0.909 + 0,005
0.004 + 0.033
0.862
0.017
0.027 = 0.003
0.785 = 0.074
0.393 = 0.017
Linear
0.969 + 0.008
0.744 + 0.016
0.857
0.026
0.028 0.005
0.787
0.066
0.388
0.032
Hermite Spline
0.971 +: 0.012
0.976 + 0.000
0.983 +: 0.004
0.055 +: 0.016
0.135 = 0.007
0.093
0.011
PolyODE
0.994 + 0.003
0.992 + 0.000
0.984 + 0.002
0.012 + 0.002
0.034 = 0.008
0.038 = 0.008
Incorporating global time series uncertainty: Previous experiments demonstrate the ability of PolyODE to re- tain memory of the past trajectory. A similar capability can be obtained for capturing global model uncertainties over the time series history. In Figure R we evaluate the association between the recovered uncertainties of Poly- ODE and the reverse reconstruction errors. We plot the predicted uncertainties against the root mean square er- ror (RMSE) on a logarithmic scale. We compare our ap- proach with using the uncertainty of the model at the last time step only. We observe that the uncertainties recov- ered by PolyODE are significantly more correlated with the errors (Pearson-p = 0.56) compared to using the uncertainties obtained from the last time step (Pearson-
Backward Reconstruction Uncertainty
Pearson - 0.56
Log Uncertainty
-2
-6-
-8-
-10
-8
-6
Last Observation Uncertainty
-2
Log Uncertainty
0.11). More details are available in Appendix F
-6
-10
6 CONCLUSION
Producing time series representations that are easy to ma- nipulate, representative of global dynamics, practically useful for downstream tasks and robust to irregular sam- pling remains an ongoing challenge. In this work, we took a step in that direction by proposing simple but novel architecture that satisfies those requirements by de- sign. As Neural ODE, PolyODE inherits the ability to handle irregular time series elegantly but at the same time, PolyODE also incurs computational cost associated with numerical integration. Currently, our ap- proach also requires a large hidden space dimension and finding methods to address this that exploit the correlation between dimensions of the time series is fruitful direction for future work.
-10
-0
Log RMSE
Figure 4: Association between uncertain- ties and reverse reconstruction errors for PolyODE (top) and classical Neural ODEs (bottom).

---

Published as a conference paper at ICLR 2023
Repruducibility Statement Details for reproducing experiments shown available in Ap- pendix H The code for reproducing all experiments will be made publicly available.
Acknowledgements
EDB is funded by a FWO-SB PhD research grant (S98819N) and FWO research mobility grant (V424722N). RGK was supported by CIFAR AI Chair. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government Canada through CIFAR, and companies sponsoring the Vector Institute.
REFERENCES
M. Arjovsky, Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 120-1128. PMLR, 2016.
C. Canuto and A. Quarteroni. Approximation results for orthogonal polynomials in sobolev spaces. Mathematics of Computation, 38(157):67-86, 1982.
Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1), 2018.
R. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equa- tions. Advances in neural information processing systems, 31, 2018.
K. Cho. B. Van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409. 2014.
J. Chung, C. Gulcehre. K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 11Xiv:141..3555, 2014.
E. De Brouwer, J. Simm, A. Arany, and Y. Moreau. Gru-ode-bayes: Continuous modeling of sporadically-ooserveed time series. Advances in neural information processing systems, 32, 2019.
E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
J.-Y. Franceschi, A. Dieuleveut, and M. Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
A. Graves, S. Fernández, and Schmidhuber. Multi-dimensional recurrent neural networks. In International conference artificial neural networks, pages 549-558. Springer, 2007.
A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33:1474-1487, 2020.
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:21 11.:::2111000990 2021.
P. S. Heuberger, T. J. de Hoog, P. M. Van den Hof, and B. Wahlberg. Orthonormal basis func- tions in time and frequency domain: Hambo transform theory. SIAM Journal on Control and Optimization, 42(4):1347-1373, 2003.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98), 1997.
A. E. Johnson, T. J. Pollard, Szolovits. L. Anthony Celi, and R. G. Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):199, 2016.
L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody,
P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
M. Lechner and R. Hasani. Learning long-term dependencies iregullrly-samplld time series. arXiv preprint arXiv:2006.044 2020.
10

---

Published as a conference paper at ICLR 2023
Y. Li, M. Anitescu, O. Roderick, and F. Hickernell. Orthogonal bases for polynomial regression with derivative information in uncertainty quantification. Visualization of Mechanical Processes: An International Online Journal, 1(4), 2011.
G. Lohöfer. Inequalities for the associated legendre functions. Journal of Approximation Theory, 95 (2):178-193, 1998.
P. Malhotra, V. TV, L. Vig, P. Agarwal, and G. Shroff. Timenet: Pre-trained deep recurrent neural network for time series classification. arXiv preprint
2017.
J. Morrill, C. Salvi, P. Kidger, and Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages 7829-7838. PMLR, 2021.
M. C. Mozer, D. Kazakov, and R. V. Lindsey. Discrete event, continuous time RNNs. arXiv preprint 10Xiv:1710.04110, 2017.
R. K. Nagle, E. B. Saff, and A. D. Snider. Fundamentals of differential equations and boundary value problems. Pearson education, 2011.
D. Onken, S. W. Fung. X. Li, and L. Ruthotto. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9223-9232, 2021.
Y. Rubanova, R. T. Chen, and D. K. Duvenaud. Latent ordinary differential equations for irregularly- sampled time series. Advances neural information processing systems, 32, 2019.
Sauer. Numerical analysis. Addison- Addison-Wesley Publishing Company, 2011.
Shukla and B. Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference Learning Representations, 2021.
A. Voelker, I. Kajié, and C. Eliasmith. Legendre memory units:
Continuous-time representation in
recurrent neural networks. Advances in neural information processing systems, 32, 2019.
S. Wang, M. B. McDermott, G. Chauhan, M. Ghassemi, M. C. Hughes, and T. Naumann. Mimic- extract: data extraction, preprocessing, and representation pipeline for mimic-iii. Proceed- ings of the ACM conference on health, inference, and learning, pages 222-235, 2020.
J. Zhao, F. Huang, J. Lv, Y. Duan, Z. Qin, G. Li, and G. Tian. Do mn and Istm have long memory? In International Conference on Machine Learning, pages 1365-11375. PMLR, 2020.
H. Zhou, S. Zhang, Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings the AAAI Conference Artificial Intelligence, volume 35, pages 11106-11115, 2021.


Reviews:
[{"review_id": "ywS8h3eXDk", "reviewer": "Reviewer_uK5k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines techniques from Neural ODEs together with with prior work on polynomial projections for signal reconstruction (HiPPO). It defines a new method PolyODE that can address various problems with time series analysis such as reconstruction and forecasting, particularly for settings such as irregularly-sampled data.\n\n\n", "summary_of_the_review": "This paper contributes technical ideas that improve on baselines on addressing the problems it is concerned about (e.g. irregular sampled time series). However, the method description is incredibly confusing, which also partly stems from inaccurate portrayals of prior work.", "strength_and_weaknesses": "This paper is not well described or positioned. It appears to be the exactly same as the HiPPO (High Order Polynomial Projections) method from the literature, with two minor variations:\n1. It considers $d$-dimensional input instead of $1$-dimensional, by simply applying $d$ independent HiPPO models.\n2. It claims to be \"continuous time\" and uses a different numerical integration step in between observations, although here the technical details are missing so it is not clear what precisely is being proposed.\n\nSeveral technical details are missing from the paper.\n- It is not clear what $A_\\mu$ means in the paper. It is mentioned that HiPPO derives several families of matrices based on a measure $\\mu$; which one is used in this paper? Is it initialized to a specific matrix and then learned, or frozen to a HiPPO matrix?\n- Details around the \"Numerical Integration\" are very confusing and not clearly presented anywhere in the paper. Section 4.2 claims to use a numerical solver in the style of Neural ODEs, but later claims to use a solver such as Backward Euler. If it is the latter, then Backward Euler is a special case of the integration rules considered in the original HiPPO paper. If it is the former, a more detailed comparison of the tradeoffs of the different integration rules should be discussed and ablated (in particular, a black-box ODE solver trained with the adjoint method should be much slower than the linear recurrences used in HiPPO).\n- How are reconstructions for NODE being done in Figure 1? The PolyODE appears to be just the HiPPO reconstruction framework, but such closed form reconstruction formulas don't exist for general NODEs. Also, what is \\alpha in equation (9)?\n\nEven if the method was described more clearly, this application of HiPPO to higher-dimensional data raises more points that should be discussed. The paper acknowledges that a major limitation of this method is the computational expense of using $d$ independent features (requiring a state size blown up to $dN$), which is why the original HiPPO only used dimension $1$.\n- Computational tradeoffs of this method and baselines should be discussed and quantitatively compared, particularly as a function of $d$ and $N$. Additionally, the numerical integration method should be more clear and its computational speed ablated, as it seems to be an interchangeable component compared to HiPPO's discretization method.\n- This issue of blowing up the dimension by a factor of $N$ is in fact one of the main problems addressed by later extensions of HiPPO to full state-space models such as S4. These baselines should be included where appropriate, as they can be used as drop-in models for most of the considered applications (note that they can also be applied to many \"irregularly-sampled\" time series problems where the sampling is provided as a mask).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "xE2Grp5ybH", "reviewer": "Reviewer_mcs4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents neural ordinary differential equations (Neural ODEs) to learn representations of time-series in a latent space. The dynamic of such a latent space uses continuous-time neural ODEs, extending from existing discrete-time versions [Gu et al, 2020] based on orthogonal polynomials. The paper shows that the learned representations can capture long-range memory with theoretical results and empirical results via reconstructing the past of time series.", "summary_of_the_review": "This is a solid paper even though it extends from Gu et al. 2020 into neural ODE. I lean toward accepting the paper. At the moment, I give a score of 6.\n", "strength_and_weaknesses": "Strength:\n- The idea of using orthogonal polynomials to solve long-range properties of time series with continuous-time models is interesting. \n- The paper shows some theoretical results as well as competitive performance of the proposed model.\n\nWeakness:\n- As one of the main contributions, the paper includes the way to use adaptive solvers. However, the paper just briefly mentions Backward Euler and Adams-Moultons [Sauer, 2011]. Firstly, it is not self-contained. Many readers from machine learning background don’t know about such numerical methods. Therefore, the current explanation does not provide . Secondly, although the observation about the stiffness of ODEs is interesting, using such solvers is not a significant contribution. \n- Missing comparisons with neural controlled differential equations with log-signature [Morril et al., 2021]. I think this is the main baseline in terms of capturing long-range dynamics. The paper mentions this reference but does not give empirical comparisons and a detailed discussion. \n- The stiffness of ODE is elucidated clearly. However, there are no illustrations of the spectral information of the matrix $A$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AHBTLahqDU0", "reviewer": "Reviewer_1xor", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a neural ODE based method to predict time series data. This method mainly projects the long-term trajectory onto basis of polynomials. ", "summary_of_the_review": "This paper proposes a novel Neural ODE method which leverages a novel dynamic functions. The proposed method is well supported by the theoretical analysis and empirical experiment. ", "strength_and_weaknesses": "The strengths of this paper are:\n\n$\\mathbf{1}.$ The proposed method is novel and with solid contribution. At the same time, there is a complete presentation of theoretical base, where authors propose the novel dynamic function for PolyODE with comprehensive analysis. It could be a great addition to Neural ODE method in the field.\n\n$\\mathbf{2}.$ The empirical experiment is comprehensive with very detailed explanation in experiment setup and scenarios. The choice of data, baseline and design of the experiments are both very appropriate and/adequate", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster