Latent Linear ODEs with Neural Kalman Filtering for Irregular Time Series Forecasting
Authors: Randolf Scholz
Stefan Born
Nghia Duong-Trung
Mariano Nicolas Cruz-Bournazou
Lars Schmidt-Thieme
Venue: Submitted to ICLR 2023 (2023)
Keywords: Time Series Forecasting
Neural ODE
Kalman Filter
Koopman Operator
Missing Values

Abstract: Over the past four years, models based on Neural Ordinary Differential Equations have become state of the art in the forecasting of irregularly sampled time series. Describing the data-generating process as a dynamical system in continuous time allows predictions at arbitrary time points. However, the numerical integration of Neural ODEs typically comes with a high computational burden or may even fail completely. We propose a novel Neural ODE model that embeds the observations into a latent space with dynamics governed by a linear ODE. Consequently, we do not require any specialized numerical integrator but only an implementation of the matrix exponential readily available in many numerical linear algebra libraries. We also introduce a novel state update component inspired by the classical Kalman filter, which, to our knowledge, makes our model the first Neural ODE variant to explicitly satisfy a specific self-consistency property. It allows forecasting irregularly sampled time series with missing values and comes with some numerical stability guarantees. We evaluate the performance on medical and climate benchmark datasets, where the model outperforms the state of the art by margins up to 30%.

LATENT LINEAR ODES WITH NEURAL KALMAN FIL- TERING FOR IRREGULAR TIME SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Over the past four years, models based on Neural Ordinary Differential Equations have become state of the art in the forecasting of irregularly sampled time series. Describing the data-generating process as dynamical system in continuous time allows predictions at arbitrary time points. However, the numerical integration of Neural ODEs typically comes with a high computational burden or may even fail completely. We propose novel Neural ODE model that embeds the observations into latent space with dynamies governed by linear ODE. Consequently, we do not require any specialized numerical integrator but only an implementation of the matrix exponential readily available in many numerical linear algebra libraries. We also introduce novel state update component inspired by the classical Kalman filter, which, to our knowledge, makes our model the first Neural ODE variant to explicitly satisfy specific coffissnsit property. It allows forecasting irreg- ularly sampled time series with missing values and comes with some numerical stability guarantees. We evaluate the performance on medical and climate bench- mark datasets, where the model outperforms the state of the art by margins up to 30%.
INTRODUCTION
Continuous dynamical systems described by ordinary differential equations (ODE) propagate a given state into any time in the future. Hence ODE based models are natural candidates for the task of forecasting irregularly sampled time series. Furthermore many real world systems are well described by ODEs. Since the seminal paper by [Cen et al.) (2018) Neural ODEs have become building blocks of state of the art models for irregularly sampled time series forecasting. To predict future state an ODE model would need an estimation of the present state and then propagate the state by solving an initial value problem. The present work proposes model that introduces novel ideas both with respect to the state estimation and to the propagation. One serious issue with Neural ODEs is the cost and possible failure of the numerical integration. There exist many numerical schemes for this purpose, but in any case the cost of the integration for a required accuracy depends on the analytical properties of the right hand side and can become arbitrarily large or lead failure. This is serious problem for Neural ODEs, which has been tackled by different types of regularizations (Finlay et al. 2020 Ghosh et al. 2020 Kelly et al., 2020).
We propose a model where the observations are nonlinearly mapped into a latent space and linear ODE with constant coefficients describes the latent dynamics. Solving the initial value problem simplifies to taking the matrix exponential, for which efficient and stable numerical implementations are available. According to Koopman operator theory (Brunton et al., 2022) such linear ODEs are expressive enough to approximate nonlinear ODEs. Furthermore, such linear dynamics are well understood and can be analyzed and modified using tools from linear algebra.
For the state estimation we propose filter inspired by the classical Kalman filter that updates the state given a new observation. However, does not operate in the lincar latent domain, but in the observation domain, and is not probabilistic. The filter is designed to deal in natural way with missing values and satisfies celf-consi condition, such that the model state will only change at an observation if it differs from the model prediction.
To the best of our knowledge our model is the first model that gives provable guarantees of forward stability at intialization.

---

We evaluate the model three benchmark datasets for forecasting irregularly sampled time series with missing values (USHCN, MIMIC-III, MIMIC-IV) and improve on the existing models by a considerable margin in all cases.
The contributions of this work are as follows:
We provide a joint view of many ODE-based and related models as latent state space models with four different model components system, filter, encoder and decoder, which by design can handle irregular time series data.
We formulate and argue five different desiderata for the properties of such models, esp. having fast and simple system components / ODE integrators, self-consistency. forward stability and handling missing values.
(3) We propose model consisting of linear ODE system and a Kalman-like filter, LinODEnet, and show that guarantees to fulfil these desidered properties.
In experiments on forecasting three different irregular time series datasets with missing values we show that LinODEnet reduces the error by up to 30% over the previous state of the art.
RELATED WORK
In the last decades a variety of different forecasting models with principally different architectures such as recurrent neural networks (RNNs), attention based models, convolution based models, varia- tional autoencoders and generative adversarial networks and others have emerged (see appendix D and Benidis et al. (2020) for a recent survey). Most closely related to our work are models based on ODEs, models using Kalman Filtering, Koopman theory and Neural Flows.
ODE based models. A neural ODE is model that is specified as the solution to an ordinary differential equation of the form
x(1) = ((1,)((1,))
where f is a neural network parametrized by A. Neural ODEs were pioneered by Chen et al. (2018), who derived continuous time analogue of the backpropagation algorithm that allows for effective training of such models. The GRU-D model (Che et al. 2018) could be considered an even earlier version of a neural ODE since it coincides with linear differential equation of the form d =t) diag(-À1, -............. >mm)(()
Since then there has been tremendous development with a large amount of offisring models. ODE- RNNs and Latent Ordinary Differential Equations (Latent-ODE) were introduced by Rubanova et al. 2019), combining the dynamics of an ODE with an RNN to perform a state space update at the point of new observation. GRU-ODE-Bayee Bayes (De Brouwer et al.) 2019) is an improvement over the standard ODE-RNN that parametrizes the ODE component as a continuous time analogue of a GRU, increasing stability. Neural Controlled Differential Equations (NCDE; Kidger al. (2020)) use spline smoothing technique in order to create a continuous reconstruction of the time scries before feeding
the ODE. This model however was only used for classification tasks. Neural ODE Processes (NDPs; Norcliffe et al. (2020) extend the framework of Neural Processes (NPs; Garnelo al. 201 8b a)) to the class Neural ODE models. Similarly, Neural Jump Stochastic Differential Equations NISDE:Daa & Benson (2019) combine Neural ODEs with Stochastic Processes. Finally, the recent Linear State-Space Layers (LSSL; Gu et al. (202T) interpret both RNNs and convolutions as special cases of linear state space layers, and construct deep LSSLS by stacking multiple such
layers.
Kalman Filtering. The Kalman Filter (Kalman 1960) provides optimal incremental state updates for linear differential equations with Gaussian noise. While the original version requires complete observations, modified Kalman filter copes with missing values (Cipra & Romera 1997). which is one of the motivations for our filter design.
The Normalizing Kalman Filter (de Bézenac et al. 2020 is closest in spirit to model, it proposes a discrete time linear Gaussian model and an invertible observation map given by a normalizing flow. As ours the model can cope with missing values, but it is not apt for irregularly sampled time series.
2

---

KalmanNet (Revach al. 2022) is another model for regularly sampled time series that RNN to calculate the Kalman gain. It is meant to present a robust alternative to other approaches of nonlinear Kalman filtering (extended Kalman filter) and not general times series forecasting algorithm. Some other work on Kalman filtering by neural networks (Wilson & Finkel [009 MMillddee al., (02T) either addresses very special cases related to the modelling of the brain and rather far away from our work.
linear operator on space of time dependent functions that describes the propagation of observations of a dynamical system through time. While these function spaces are infinite dimensional, in many cases there exist useful finite dimensional Koopman operator approximations that can be created by various methods, that have been summarized in a recent review (Brunton et al., 2022). Such Koopman representations have been combined with Kalman filters for the linear operator (Netto & Mili, 2018), but the linear representation is obtained by classical method which does not work for irregularly sampled observations with missing values. Some works use neural networks to learn Koopman representations (cf. section 5.4 of Brunton et al. (2022)), but have largely different architectures and are not applied to irregularly sampled time series with missing values. The model we present can be seen as learning an approximate Koopman operator, but is outside the scope of this work to exploit this representation along the lines of Koopman theory.
Koopman Theory. Koopman Operator (Koopman 1931)
Neural Flows. Neural Flows (Biloš et al. 2021 are another related model class. The authors propose replacing ODEs by invertible time dependent diffeomorphisms and different parametrizations for such diffcomorphisms (ResNet, GRU, coupling flow). This is similar to and in special cases would amount to learning the solution of differential equation instead of learning the right hand side. They mention also the parametrization of the solutions by a matrix exponential, but then opted for other parametrizations.
3 PROBLEM FORMULATION
time series dataset D is a set of time series instances, sequences 21,,) E (RU encoding observations Itt at time 4, The observation space O usually is just composed of M channels: O = R.M. If observations in some channels can be missing, we write O = (IR U (NaN)) and dinstinguish it from the space X M of complete observations.
The time series forecasting problem is, given a time series dataset D from an unknown distribution q. loss 8 O² x X + R on observations, and function split R -> (R* x R²'* that splits the time points of a time series into (possibly multiple) pairs of two subsequences, pasts p and futures to find model & R' x (R x O)* + X that for given future time points and past observations, predicts future observations, minimizing the expected loss
Ett. youg
Σ (p.s)
(1 f(s, (p. )))
split(())
simple based split function just outputs all possible splits into a past of P time points and F future time points, and the loss usually is just instance-wise loss, e.g., the instance-wis wise mean squared error:
index (I;P,)) ((14-P41 4)) (1.41 (+F)) i P:14=)). P,FEN
MSSE (z
.tt
lalng(not-missing (2) (2 ) 22)111
where not-missing (ngho yields the indicator vector for ohs * NaN.
4 LATENT STATE SPACE MODELS
First, consider a general class latent state space models that follow the schema from Algorithm That is, the model has state estimate d. which is encoded into latent state é, and propagated
3

---

forward in time by system component. For each observation, latent state is decoded and updated with filter component.
Algorithm 1 Latent State Space Model.
Input: Query times
Laive spari
Observations = (f3, ot, )==
Parameters: Initial latent state
D t sort (D U {($j, NaN)]
time
ta (1; tto k-
for 1,, -t, in do
Figure 1: Latent State Space Model
At; t, - -;-1
&r. < System (iiis almmn)
f1 < Decoder( 2x,)
4 Filter (ftt. Itt.
&t. < Encoder (
end for Return: Estimated states (4', )=
This setup is indeed a very general model class, Table shows how many current state space models can be described by this schema.
Neural ODEs, introduced in 2018 by Chen et al.) use a ordinary differential equation to represent the system component. The vector field which describes the right hand side of the ODE is given by neural network f:
1 ftt, (t)),
=(to)
(t) = odeint(f, 20, [to, f))
(1)
Crucially, they introduced a continuous version of backpropagation through time that allows to compute gradients for a Neural ODE model with respect loss function by solving a so called adjoint equation backwards in time. This allows one compute gradients without implementing differentiable ODE-integrator and without baok-propgg through the integrator steps.
Table 1: Comparison of Continuous Time Latent State Space Models
Model
System Component
Filter Component
of my obs
+ (1 me)ar
GRU-D
*(t) diag(max(0,, ux)) O z()
o + GRUCel ze)
ODE-RNN
((t) = NN, z(t))
*f < RNNCell(a 2e)
GRU-ODE-Bare
&(t) GRUCelI(O, =(t)) 2(t)
of < GRUCell me 24), 21)
NCDE
a(t) (=(()))(()
s(t) CubicSpline(1 (li, xe =l.)))..)))
Neural Flow
F(At, x)) = x + ((
Bayesian Filtering
LinODEnet (ours)
*(t) = Az(()
o KalmanCell(1 vz) = Ft f(me @ (1x - x rbbs))
By design, latent state space models can handle irregularly sampled time series. We formulate five further principled desiderata and argue for them in the following:
D1. fast (and simple) integrators: solving the ODE system for propagating the latent state should fast (and simple).
D2. celficoosistency: the model does not change its forecast when observing its own predictions.
D3. forward stability: at initialization the model maps data with zero mean and unit variance to outputs with zero mean and unit variance for arbitrarily long sequences.
D4. handle missing values: the model can handle missing values in the observations.
4

---

D5. observed channels influence all other channels: observed values in channel can influence the estimation of unobserved states of all channels.
D1. Fast (and simple) integrators. Neural ODE models only faithfully represent the solution to an ODE when the numerical integrators use sufficiently small adaptive step-sizes (Ott et al. 2020p. Thus, fitting generic Neural ODEs is a challenging task since during training the ODE become stiff, which forces adaptive step-size integrators to take minuscule time steps. Towards this end, several remedies are available: Ghosh al. (2020) and Finlay et al. (2020) propose temporal regularization terms, and many models implicitly address this issue by choosing a special form of vector field. An overview is in Table For example, GRU-ODE-Baye uses GRUCell with tanh activation function that induces global Lipschitz continuity, which increases stability.
D2. Self consistency. All of the proposed models use different filter mechanisms update the state estimate when new observations are recorded. However, in our analysis we noted that classical Kalman Filter (Kalman| 1960) satisfies celfisoons property that none of the published Neural ODE models seems to incorporate.
We say a
model is self-consistent, if and only if the model does not change its forecast when observing its own predictions. More specifically, given forecasting model ÿ(t D).
where = (44,, -;)) i = -1...M n is the set observations, then ý is self-consistent and only for all finite sets generated predictions = {(8). ((,)) | 1j=1....
j(t D) g(t DU
(2)
For probabilistic models p(u(t) D) we similarly define self-consistency as the condition Bp-p(s(01D) |0]
E (-r(e(t) Dub)
Note that in this case it is expected that Var g=pppt(())DDDD)) [ should decrease with the size of D.
D3. Model stability. Model stability is crucial in order to allow training over long sequences without issues of divergence.
Definition (forward stability). We say a function f: R" + R is forward stable, if and only if maps data with zero mean and unit variance to outputs with zero mean and unit variance.
E E /
0
Vi :
Vj
(()
Similarly, one can define backward stability as the condition that the gradients, or more precisely the vector-Jacobian product maps data with zero mean and unit variance to gradients with zero mean and unit variance. Typically, the random distributions of network parameters are chosen in way to ensure either forward- or backward stability at initialization or compromise between the two at as in general simultaneous forward and backward stability impossible He et al. (2015). For example. Attention layers ( (Vaswani et al. 2017) introduce a scaling factor of 1/va and Dropout (Srivastava al. 2014) multiplies the input by the reciprocal of the droprate.
However, recently new approach has emerged that achieves both in ResNet architecture, by simply introducing an additional single scalar parameter initialized with 0 that masks the non- linearity, making the model look like an identity map. We will refer to this the ReZero-technique (Bachlechner et al. 2021). although previous works also showed similar ideas Hayou et al. (202I);
Skorski al. (2021 De & Smith (2020); Zhang et a (2018) Balduzzi et al. (2017) Shang et al.) (2017)
ReZero:
x f x xff
a: learnable scalar initialized with 0
(5)
In particular recent research suggests that this techniques allows one to refrain from using batch- normalization layers (loffe & Szegedy| 2015).. We use variants of the ReZero-technigee throughout all model components.
D4. Can handle missing values. The model can handle missing values. For latent state space models the filter needs to be able to do so.
D5. Observed channels influence all other channels. This is crucial as often channels correlated with each other, hence observing one channel can provide information about all other channels.
5

---

5 LATENT LINEAR ODES WITH NEURAL KALMAN FILTERING (LINODENET)
We propose two specific innovations for ODE-based latent state space models: (i) to use a linear ODE for the system component and (ii) use Kalman like filter component, such that the overall model fulfills the desiderata DI to D5, and call it LinODEnet. LinODEnet is structured as shown in Algorithm
!
We describe its components in turn.
5.1 SYSTEM COMPONENT
To avoid having to use complicated numerical integrators, LinODEnet uses simple homogeneous linear ODE with constant coefficients. This has the huge advantage that the solution can be expressed in closed form in terms of the matrix exponential.
Definition 2 (Linear ODE). If the vector field is an affine function of the state vector the ODE is called linear, i.c. if and only if it is of the form:
(t) = A(t) x(t) + b(t)
x(to) = I0
(6)
for some matrix valued function A: R. - Rax" and vector- valued function b: R - R". If and b are constant, we call a linear ODE with constant coefficients. b= O, we say it is homogeneous.
Lemma 1 (Solution of Linear ODE). The solution of homogeneous linear ordinary differential with constant coefficients can be expressed in term of the matrix exponential
a(l) = Ax(()
= x(l + Al) = AAt, cAA()(())
Proof. See for instance Teschl (2012).
In particular, implementations of the matrix exponential readily available in many popular numerical libraries such as |SCIPY (Virtanen et al. 2020), TENSORFLOW (Abadi et al. 2016) or PYTORCH (Paskee et al.] 2019). Typical implementations such scaling and squaring approaches by Higham (2005) and Al-Mohy & Higham (2009) offer high performance and tight error bounds, establishing desideratum D1.
A second advantage of the linear system is the possibility to parametrize or regularize the kernel matrix in order to achieve certain properties. We highlight the initialization with skew-symmetric matrix as of particular importance, and we use it as the default initialization in all experiments.
Lemma 2. If K is skew-symmetric, then e is orthogonal and h(t) = K-tr is forward stable for all t and x (O,,,..)
Motivated by these properties, define the LinODECell (Alg 2). Note that crucially, in comparison to the GRU-D model, a general latent linear ODE model allows for imaginary eigenval- ues of the system matrix, corresponding to oscillatory system behaviour. In the GRU-D model, the authors intentionally re- stricted the model to non-positive, real diagonal matrix.
Algorithm 2 LinODECell
Input: scalar time delta At latent state 2t
Parameters: matrix K
zero-initialized scalar e parametrization vi
5.2 FILTER COMPONENT
Z++At f- (K)) 24
Return: latent state 2
Any state space model must have a way of incorporating new measurements as they come in.
We call function of the form f: O x X X, (2 2) observation space O contains NaN values we say it allows for missing observations. If the state space X is equal to the non-missing part of the observation space O we call it autoregressive. Finally, we say a filter cross-correlates channels, if even the observation of just single channel can potentially update the state estimate in all channels.
Definition 3.
E' filter. If the
One of the big achievements of classical filtering theory is the Kalman Filter (Kalman 1960). which the provably optimal state update in terms of squared error loss when the system consisting of normally distributed variables evolving according linear dynamical system.

---

Assuming the state is distributed as (M,, Σx). time t, and evolves according to linear dynamical system ž = A, + wt, then since the family of Normal distributions are closed under linear transformations, the state is normally distributed for all times t. Given noisy measurement Yt H + et with Rg = El 1er0 which is only partially observed according to a mask m, then the optimal state update is (Cipra & Romera 1997)
14 Ar HHH + RR) 99)
(8a)
Σ E, CHH RR) II,,,,,
(8b)
Where IT; dmag. and ÿt is where the missing values were replaced with arbitrary values. Inspired by this formula, we introduce the linear and non-linear KalmanCell which can used as drop-in replacements for regular RNN- GRU- or LSTMCells.
linear KalmanCell:
8 - aBH T.BHTT,TTT AT,, 2:(tx)
(9a)
non-linear KalmanCell:
t- - TAAHH,AATLLLL AT, It )
(9b)
In both cases A, B, H are learnable weight matrices. In the linear case we introduce a special parametriza- tions = A and = + Here e, CA, eB learnable scalars, that initialized with zero, which ensures forward stability. & is an arbitrary neu- ral network with <(0) 0. By design the Kalman- Cell can handle NaN values, for implementation de- tails see 4 and [ establishing D4.
Algorithm 3 Non-Linear KalmanCell
Input: Current State estimate i; E IR", observed datapoint x²? E R.
Parameters: Learnable matrices A, B, H zero-initialized scalar E, neural network ø. Options: If autoregressive,
and
T,.
Th + diag(not-missing(a (x:")) E - ig - EΦ( BH AT, (Hâ; xxth))
Lemma 3 (KalmanCell at initialization). At initial- ization, the non-linear KalmanCell is the identity function. The linear KalmanCell's behaviour is de- pendent on the choice of o: if = 1, it always updates the state to the last observed value, whereas if 0 is carries the first observed value through. The choice a = corresponds to the classical Kalman Filter (cf. Appendix B.2
Return: Updated state estimate 8.
Note that (9 different from both Szirtes et al. (2005). Wilson Finkel (2009), and the recent pre-prints Millidge et al. (222)) and Revach et al. (2022). Moreover, since the KalmanCells do, in contrast to the GRU-D model, use full and not diagonal matrices, D5 is satisfied.
Moreover it is possible to stack multiple filters (fi)i=l...k:
# (i+ 1) fi(x a()) for i=l...k
Stacked Filter:
(10)
With regards to desideratum D2, there is a strong relationship to the setup of the filter component.
Definition 4. We say an autoregressive filter F: X x X X, (a 3) i' is idempotent, if and only if returns the original state estimate as-is whenever all non- non- missing observations agree with it.
V.Vi:
1. # NaN
=
F(ath,
$) +
(11)
Lemma 4. If latent state space model (Alg. 1 self-consistent, then its filter must be idempotent.
Proof. It this wasn't the case, then j(l D) # j(( DU (1, 5(t ((L)))))
5.3 OVERALL MODEL PROPERTIES
If in Algorithm 7 the system component represents a dynamical system, and the filter component is idempolent, and the encoder is left-inverse to the decoder, then the model is self-consistent.
Proposition 1.
Corollary LinODEnet is self-consistent at initialization. If the encoder is left-inverse to the decoder it is self-consistent throughout training, establishing desideratum D3.
The proofs can be found in Appendix Table Salle2ssuma summaries the properties of git adLinODEnet in
comparison other models.
7

---

Table Comparison foreasting model features (f: transformer model)
model
TFT
NKF
GRU-D
GRU-oDE-Bayes
Neural Flow
LinODEnet
missing-values
continuous time
X
global existence
<NA>
J
/
X
self-consistency
X
X
X X
forward stability coupled channels
X
X
<A>>
X
Remark 1 (LinODEnet with hidden state). Since LinODEnet can parse NeN values, we consider a small modification consisting of concatenating number of dummy channels, completely filled with NaN-values to all input data. This allows the model to have a working memory in case that the state space does not capture the full dynamics.
EMPIRICAL EVALUATION
6.1 EXPERIMENTS ON IRREGULAR TIME SERIES
While there are many publications dealing with irregular time series for classifications imputation tasks, there are few that approach with irregular sampled time series forecasting natively. We identify Gay- (De Brouwer et al. 2019) and Neural Flow (Bilos et al. 2021
Irregular time series occur naturally whenever collections of sensor devices sample data independently. sometimes vastly different rates. This is for example the case with clinical records such as the MIMIC-III (Johnson et al. 2016) and MIMIC-IV (Johnson et al., 202I datasets or weather observations such the USHCN dataset (Menne et al. [013). We use the same data-preprocessing and evaluation protocol as our baselines De Brouwer et al. (2019) and Bilos et al. (2021) The task is to predict the next samples using observation horizon of 36 hours.
Table 3: Average MSE and standard deviation across cross-validation folds.
t: results reported by De Brouwer et al. 2019 t: results reported by Biloš et al. (022
Model
USHCN
MIMIC-III
MIMIC-IV
NeuralODE- VABB
0.96 + 0.11
0.89 0.01
0.83 + 0.10
0.89 + 0.01
Sequential VAE!
0.83
0.92 + 0.09
GRU-Simple!
0.75 0.12
0,82 + 0.05
GRU-D1
0.53 0.06
0.79
T-LSTM
0.59 0.11
0.62
GRU-ODE-Bayes'
0.43 + 0.07
0.48 0.01
Neural Flow (GRU)*
0.49 + 0.004
0.364 +-0.008
LinODEnet w/ hidden
0.29 + 0.06 (133%)
0.45 + 0.02 (15%)
0.274 + 0.002 (125%)
LinODEnet w/o hidden
0.31 + 0.05 (131%)
0.45 0.02 (15%)
0.274 + 0.002 (125%)
Table shows that LinODEnet outperforms all baselines by significant margin. We also observe that in the USHCN dataset, adding hidden channels (Remark I gives additional lift. We suspect that this is due to the numbers of channels being very small (5) for this dataset.
Training Details. We use the ADAM W optimizer (Loshchllov & Hutter 2018). a variant of the popular ADAM-optimizer Kingma& & Ba (201)) that provides correction when using weight decay. For the filter we use a stack of linear KalmanCell and two nonlinear KalmanCells. The full hyperparameter selection is in Appendix E.2
8

---

Reproducibility. We created two pip-installable python packages: tssmproviddss utilities for dataset acquisition, pre-processing and library of encoders. A reference implementation of the model in PYTORCH is available the package linodenet. experimental code is in separate repository.
6.2 OBSERVATIONS
During training, we consistently observed the emergence of corre- lation between the rows and columns of the system-components kernel-matrix (Figure 4). This indicates that the matrix get close to low-rank matrix. Since the rank itself cannot be computed in a numerically stable manner, we considered a smoothed relaxation known the effective rank (Roy & Vetterli 2007).
95-
affeche rank
92
85
Definition 5. The effective rank of matrix is defined as the expo-
nential of the entropy erank(A) = e,)()), where p a/|lor] is the discrete probability distribution given by normalizing the singular values a of A.
db
opoch
Figure 3 shows the evolution of the spectrum of the kernel matrix for sample
Figure
the USHCN dataset. One eigenvalues stay close to being purely imaginary. We speculate that this is due to the main dynamics essentially periodic in nature, as weather patterns repeat over time
that the
(a) epoch
(b) epoch 10
(c) epoch 100
(a) epoch 0
(b) epoch 10
(c) epoch 100
Figure 3: Evolution the kernel spectrum.
Figure 4: Evolution the kernel values.
7 CONCLUSIONS
We propose a novel forecasting model for irregularly sampled time series with missing values, that maps the observation space latent space with constant linear ODE dynamics and performs state estimations by an update rule inspired by the Kalman filter. For solving the linear ODE we do not need numerical integration but just matrix exponentials, for which stable and efficient implementations exist. Forward stability of the model at intialization is guaranteed. The model is evaluated and most of the existing forecasting benchmarks for irregularly sampled time series and improves on existing models by considerable margin. The model opens way for interesting future work: It naturally allows for future covariates in the forecasting problem. The linear representation of the dynamics allows modification and analysis by means of linear algebra.

---

REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems, March 2016.
Awad H. Al-Mohy and Nicholas J. Higham. A New Scaling and Squaring Algorithm for the Matrix Exponential. SIAM Journal on Matrix Analysis and Applications, 31(3):70-988, August 2009.
ISSN 0895-4798. doi: 10. 10.1137/007472IX.
Awad H. Al-Mohy and Nicholas J. Higham. Computing the Action of the Matrix Exponential, with an Application Exponential Integrators. SIAM Journal Scientific Computing, 33(2):488-511 January 2011. ISSN 1064-8275. doi: 10.1137/100788860.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley, ReZero is all you need: Fast convergence at large depth. In Proceedings of the Thirty-Sevnnth Conference on Uncertainty in Artificial Intelligence, pp. 1352-1361. PMLR, December 2021.
Shaojie Bai, J. Zico Kolter, and Vladlen Kolunn. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling, April 2018a.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Trellis Networks for Sequence Modeling. In International Conference on Learning Representations, September 20.
David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The Shattered Gradients Problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning, pp. 342-350, July 2017.
Manuel Baltieri and Takuya Isomura. Kalman filters as the steady-state solution of gradient descent on variational free energy, November 2021.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen. Invertible Residual Networks. In International Conference on Machine Learning, pp. 573-582. PMLR. May 2019.
Konstantinos
Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bemie Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Callot, and Tim Januschowski. Neural forecasting: Introduction and literature overview, April 2020.
David Salinas, Lorenzo Stella, Laurent
Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, Tim Januschowski, and Stephan Günne- mann. Neural Flows: Efficient Alternative to Neural ODEs. In Advances in Neural Information Processing Systems, October 2021.
G. E. P. Box, G. M. Jenkins, and J. F. MacGregor. Some Recent Advances in Forecasting and Control. Journal of the Royal Statistical Society. Series (Applied Statistics), 23(2):158-179, 1974, ISSN 0035-9254. doi: 10.2307/2344997..
Steven L. Brunton, Marko Budišíé, Eurika Kaiser, and J. Nathan Kutz. Modern Koopman Theory for Dynamical Systems. SIAM Review, 64(2):229-340, May 2022. ISSN 0036-1445. doi:
10.1137/21M1401243
Wei Cao, Dong Wang. Jian Li, Hao Zhou. Lei Li. and Yitan Li. BRITS: Bidirectional Recurrent Imputation for Time Series. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Seseenthaler-aanseee and Artur Dubrawski. N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting, February 2022.
10

---

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports, 8(1):6085, April 2018. ISSN 2045-2322. doi: 10/gdfnex.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1725 1734, Doha, Qatar, October 2014, Association for Computational Linguisties. doi:
T. Cipra and R. Romera. Kalman filter with outliers and missing observations. Test, 6(2):379-395. December 1997. ISSN 1863-8260 doi: 10. (0.1007/FF02564705
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19964-19975 Curran Associates, Inc.. 2020.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing Kalman Filters for Multivariate Time Series Analysis. In Advances in Neural Infor- mation Processing Systems, volume 33, pp. 2995-3007. Curran Associates, Inc., 2020.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayess Continuous Modeling of Sporadically- -bseereed Time Series. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Chenyou Fan, Yuze Zhang, Yi Pan, Xiaoyue Li, Chi Zhang, Rong Yuan, Di Wu, Wensheng Wang. Jian Pei, and Heng Huang. Multi-Horizon Time Series Forecasting with Temporal Attention Learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pp. 2527-2535, New York, NY, USA, July 2019. Association for Computing Machinery. ISBN 1--1---53--22----- doi: 10/gf7m32.
Chris Finlay, Joern-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to Train Your Neural ODE: The World of Jacobian and Kinetic Regularization. In Proceedings of the 37th International Conference Machine Learning, pp- 3154-3134 3164. PMLR, November 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Proceedings of the 35th International Conference on Machine Learning, pp. 1704-1713. PMLR, July 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural Processes, July 2018b.
Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. STEER : Simple temporal regularization for neural ODE. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 14831- Curran Associates, Inc., 2020.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers. In Advances in Neural Information Processing Systems, volume 34, pp. 572-585. Curran Associates, Inc., 2021.
Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnad Doucet, and Judith Rousseau. Stable ResNet. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1324-1332. PMLR, March 2021.
11

---

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proceedings of the IEEE International Conference Computer Vision, pp. 1926-1034, 2015.
Nicholas J. Higham. The Scaling and Squaring Method for the Matrix Exponential Revisited. SIAM Journal on Matrix Analysis and Applications, 26(4): 179-1 January 2005. ISSN 0895-4798. doi: 10.1137/04061 101X.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):
1735-1780, November 1997. ISSN 0899-7667.
doi: 10.1 10.1162/neo..199 199.tt.....999.......
Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set Functions for Time Series. In Proceedings of the 37th International Conference on Machine Learning, pp-
4353 4363. PMLR, November 2020.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning, pp. 448-456, June 2015.
Junteng Jia and Austin R Benson. Neural Jump Stochastic Differential Equations. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV Version 1.0, March 2021.
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1):160035, May 2016. ISSN 2052-4463. doi: 10. 10.100//aaaa.2066.35.
R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering. 82(1):35 45, March 1960. ISSN 0021-9223. doi: 10.1115/1.3662551
Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learning Differen- tial Equations that are Easy to Solve. In Advances in Neural Information Processing Systems, volume 33, pp. 4370-4380. Curran Associates, Inc., 2020.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural Controlled Differential Equations for Irregular Time Series. In Advances in Neural Information Processing Systems, volume 33, pp. 6707. Curran Associates, Inc., 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR (Poster), January 2015.
B. O. Koopman. Hamiltonian Systems and Transformation in Hilbert Space. Proceedings of the National Academy of Sciences, 1 17(5):315-318, May 1931. doi: 10.1073/pnas.17.55 17.5.315.
Vincent LE GUEN and Nicolas THOME, Probabilistic Time Series Forecasting with Shape and Temporal Diversity. In Advances in Neural Information Processing Systems, volume 33, pp. 4427-4440. Curran Associates, Inc., 2020.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Steven Cheng-Xiai Xian and Benjamin Marlin. Learning from Irregularly- Missing Data Perspective. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5937-5946. PMLR, July 2020.
Time Series: A
Bryan Lim, Sercan Ö. Ank, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for interpretable multi- multii-ooiizon time series forecasting. International Journal of Forecasting. 37() October 2021. ISSN 0169-2070. doi: 10. 10006//.iforrcass22201....000000
12

---

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, September 2018.
Matthew J. Menne, Claude N. Williams, and Russell S. Vose. United states historical climatology network daily temperature, precipitation, and snow data. Technical report, Carbon Dioxide Information Analysis Center, Oak Ridge National Laboratory, Oak Ridge, Tennessee., 2015.
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher Buckley. Neural Kalman Filtering, April 2021.
Marcos Netto and Lamine Mili. A Robust Data-Driven Koopman Kalman Filter for Power Systems Dynamic State Estimation. IEEE Transactions on Power Systems, 33:7228-7237, June 2018. doi: 10.1109/TPWRS.A T...S.218..200 28467444
Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural ODE Processes. In International Conference on Learning Representations, September 2020.
Yann Ollivier. The Extended Kalman Filter is Natural Gradient Descent in Trajectory Space, January 2019.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, In International Conference on Learning Representations, September 2019.
Katharina Ott, Prateck Katiyar, Philipp Hennig, and Michael Tiemann. When Neural ODE
Solutions Proper ODEs?, July 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-PPeformance Deep Learning Library. Advances Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep State Space Models for Time Series Forecasting. In Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adrià López Escoriza, Ruud J. G. van Sloun, and Yonina C. Eldar. KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics. IEEE Transactions on Signal Processing, 70:1 70:1532-1547, 2022. ISSN 1941-0476. doi: 10.1 10.10099SP.2022.35558888
David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub Mikolaj Tomezak, and Mark Hoogendoorn. CKConv: Continuous Kernel Convolution For Sequential Data. In International Conference Learning Representations, September 2021.
Olivier Roy and Martin Vettrrli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pp. 606-610, September 2007.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud, Latent Ordinary Differential Equations for regullary-SSmplle Time Series. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
David E. Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by back pooc-propaggtiig emrors. Nature, (23(688)):533-55 536, October 1986. ISSN 1476-4687. doi: 10.1038/323533a0.
David Salinas. Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, July 2020. ISSN 0169-2070. doi: 10/ghh5nv.
13

---

Wenling Shang, Justin Chiu, and Kihyuk Sohn. Exploring Normalization in Deep Residual Networks with Concatenated Rectified Linear Units. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), February 2017. ISSN 2374-3468.
Shun-Yat Shih, F Sun, and Hung-yi yi Lee. Temporal pattern attention for multivariate time series forecasting, Machine Learning, (08(8):4421114)) September 2019. ISSN 1573-0565. doi: 10/ghgs7b.
Satya Narayan Shukla and Benjamin Marlin. Interpolation- Prreicctonnn Networks for Irregularly Sampled Time Series. In International Conference Learning Representations, September 2018.
Satya Narayan Shukla and Benjamin Marlin. Multi-Time Attention Networks for Irregularly Sampled Time Series. In International Conference on Learning Representations, March 2021.
Macicj Skorski, Alessandro Temperoni, and Martin Theobald. Revisiting Weight Initialization of Deep Neural Networks. In Proceedings of The 13th Asian Conference on Machine Learning, pp- 1192-12015 PMLR, November 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15:1929-199 2014.
Gábor Szirtes, Barnabás Póczos, and András Lörincz. Neural Kalman filter. Neurocomputing, 65-66: 349-355, June 2005. ISSN 0925-2312. doi: 10.1016/j htt.://.nncom..000000 10.022..
Gerald Teschl. Ordinary Differential Equations and Dynamical Systems. American Mathematical Soc., August 2012. ISBN 978-0-8218-8328-4
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nelson, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, and Paul van Mulbregt. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17(3)2611272. March 2020. ISSN 1548-7105. doi: 10. 109100/999992-099. 9-66-22.
Robert Wilson and Leif Finkel. A Neural Implementation of the Kalman Filter. In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial Sparse Transformer for Time Scries Forecasting. In Advances in Neural Information Processing Systems, volume 33, pp. 7715517715.. Curran Associates, Inc., 2020.
Jinsung Yoon, William R. Zame, and Mihaela van der Schaar. Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks. IEEE Trans- actions on Biomedical Engineering, 66(55:1477-1490, May 2019. ISSN 1558-2531. doi: 2000/TBME.2018.8774444
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Sebastian Zeng, Florian Graf, Christoph Hofer, and Roland Kwitt. Topological Attention for Time Series Forecasting. In Advances in Neural Information Processing Systems, volume 34, pp. 24871-24888 Curran Associates, Inc., 2021.
Jiajun Zha, Yiran Zhong, Jing Zhang, Richard Hartley, and Liang Zheng. Invertible Attention, June 2021.
14

---

Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup Initialization: Residual Learning Without Normalization. In International Conference on Learning Representations, September 2018.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):1 May 2021.
ISSN 2374-3468.
15


Reviews:
[{"review_id": "k6eg5N97ls", "reviewer": "Reviewer_M6Ey", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose to forecast irregular time series using state-space model whose dynamics is specified by a linear ODE. The proposed method meets several desirable properties to time series forecast such as self-consistency, forward stability and allowing for missing values. The experiments show the proposed method achieved competitive performance.", "summary_of_the_review": "This work aims to forecast irregular time series using state-space model whose dynamics is specified by a linear ODE. The proposed method meets several desirable properties to time series forecast such as self-consistency, forward stability and allowing for missing values. \nSeveral key information seems missing. The writing can be improved.", "strength_and_weaknesses": "Strength:\nThe continuous-time dynamics, specified by ODE, allows for irregular sampling and missing values.\nThe linear ODE specification eases the integrator and computational cost.\nThe linear specification enables Kalman filter\nThe approach exhibits forward stability and self-consistency.\n\nWeaknesses:\nThe linear model limits the power of expressive and rules out such as line attractors, limit cycles, strange attractors and etc.\nThe manuscript seems finished in a hurry. It lacks information.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g7cHVa9jvp", "reviewer": "Reviewer_x2ff", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Neural ODE model that implements a filtering approach for time series. In contrast to previous works, this approach relies on mapping the observations on a linear hidden process. The linearity of the underlying ODE allows fast integration and enforcing several desirable properties such as  consistency and forward stability. The authors show improved forecasting performance on MIMIC and weather forecast datasets. ", "summary_of_the_review": "This paper addresses an important problem but currently fails to provide an in-depth investigation of the proposed approach. The experimental section is not convincing and does not reflect the aims of the paper (the 5 desideratas). The notations are inconsistent, making the overall presentation unclear.", "strength_and_weaknesses": "## Strengths\n\n- The problem tackled in this paper is very important. Namely, how do we build better latent state space models with irregular and missing data.\n- The idea of using a linear hidden ODE is nice as it theoretically can express any complex dynamical system, with a sufficiently large hidden process and expressive observation / embedding function.\n- The desiderata and objectives for a desireable time series model are clearly stated \n- The figure 1 regarding the abstraction for a latent space model is very clear and effective\n\n## Weaknesses\n### Notation\nAn important improvement consits in improving the notation. \n- In the problem formulation, it's not clear what $P:\\mid t \\mid - F$ actually means. \n- Introducing a mask instead of \"not-missing\" would be appreciated too.\n- More importantly, the authors seem to mix $z$ and $x$ over the whole manuscript which makes it unecessarily hard to read. For instance, in Section 4, it seems clear that $z$ stands for the latent process while $x$ is the observation. Now, in section 5, $x$ is now used for the latent space ! (And in Algorithm 2, $z$ is used again). $\\hat{x}$ is also sometimes used for the latent space (equatiions 9).\n\nThis is a very important point that would greatly benefit the readibility of the paper.\n\n### Lemma 2 \n\nFirst, it seems that Lemma 2 lacks a proof. What is more, you mention backward stability in D3. How does the matrix exponential fares in terms of differentiability ? I would assume it's not super stable either ?\n\nAlso, if your hidden process is forward stable, what does it say about your whole model ? I guess it depends on the output function that you are choosing. So in a sense, you're not ensuring forward stability of your whole state space model.\n\n### Output function\n\nIf I understand correctly, your encoding/update function (from observation space to hidden space) is given by Equation 9. What is missing in your manuscript is a discussion of the output function and how it is linked to the update function. \n\n### Expressivity\n\nOne of the important point of that paper is that the desiderata should be achieved without trading for expressivity. In the introduction, the authors hint at the fact that this model can be as expressive as non-linear latent dynamcis models. However, this statement does not get enough focus in the paper. Specifically, I believe an in-depth discussion of the expressivity of the presenteed model should be present in the paper.\n\n### Experiments\n\n- Regarding the non-linear vs linear Kalman cell. It would be nice to have an experiment comparing both approaches.\n\n- The w/hidden version of your model seems to suggest that your latent space and your observation space are the same  (Remark 1)? This seems contradictory with your intro that says that the \"observations are nonlinearly mapped into a latent space\".\n\n- In general, the experimental section is not very convincing. Indeed, despite the nice layout of the desiderata you need for a good latent state space model, you don't use this blueprint in the experimental section. I believe it would be more impactful to showcase how your model improves upon these metrics. \n\n- It's not clear what \"global existence\" mean in Table 2.\n\n- Section 6.2 comes a bit out of the blue and lacks context. So it's hard to appreciate its value in the current state of the paper.\n\n### Minor\n\n- end of page 7, I think \"git adLinODEnet\" is probably a typo.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_7mynAPfCFK", "reviewer": "Reviewer_gake", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel Neural ODE model that embeds the observations into a latent space with dynamics governed by a linear ODE. \nThey carefully show that the model satisfies self-consistency, which allows forecasting irregularly sampled time series and have some numerical stability guarantees. They evaluate the performance on medical and climate synthetic datasets, where the model outperforms the similar state of the art models.", "summary_of_the_review": "The paper is very clear and shows the the method outperforms previous state of the art models. However, I think the paper would gain in quality with better visualization of the learnt dynamics, in synthetic and real datasets.", "strength_and_weaknesses": "The theoretical guarantees of the model are nicely derived and clearly presented, and Table 2 is a nice summary of the theoretical advantages of the proposed model. \n\nThe empirical evaluation is compelling, but is on synthetic benchmark datasets only. Including results on a real dataset would make the paper stronger. Even without ground truth, including some visualization of the learnt dynamics vs true dynamics and other models for real datasets would be a great addition to the paper (for example, modeling ocean dynamics)\n\nFigure 1 should also be updated. I think it is of poor quality and doesn't give much insight into the model.\n\nFinally, the OBSERVATIONS section (6.2.) is interesting as it shows the Eigen-values of the kernel matrix, and the authors speculate that the learnt dynamics are periodic. There is space to include some true and inferred dynamics to visualize the periodic signal (or maybe include it in the appendix) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2g4rBtdGxsO", "reviewer": "Reviewer_kNhy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a latent ODE model for irregularly sampled timeseries forecasting, where the dynamics in the latent space evolve according to a linear neural ODE. This is enabled by (1) a nonlinear encoder/decoder pair that maps the dynamics between the latent and observed space, and (2) a neural Kalman-like filter operating in the observation space, which updates the states given an observation. By taking the latent space to be higher dimensional than the observation space, the authors justify the use of a linear ODE, reflecting ideas in Koopman theory. The resulting model satisfies a list of desiderata given by the authors and performs well on some benchmark datasets.", "summary_of_the_review": "The paper presents a model that, while technically not quite advanced, is original and performs well on several benchmark tasks, proving its capability for use in time series modelling. The paper however suffers from several presentation issues and a weak experiments section.\nIn light of this, I recommend a light reject, with the possibility of an acceptance provided some of these issues are addressed.", "strength_and_weaknesses": "__Strengths:__\n- The proposed model seems to preform quite well on the benchmark datasets, outperforming the state-of-the art in the climate example by a large margin. Though this may simply be due to the higher dimensionality of the latent space used here compared to that in the baselines (perhaps the baselines may perform just as well if they used hidden states of similar size?), the experiments at least demonstrate that the linear dynamics assumption in the latent space is not so limiting.\n\n- The use of a Kalman filter-inspired update rule is nice, enabling some level of interpretability (at least in the linear, shallow case) as opposed to the RNN-type updates used in the other works. The theoretical benefits gained through this choice is also an advantage.\n\n__Weaknesses:__\n- Uncertainty quantification does not seem possible within the proposed framework, which may be useful in several applications. Other models such as ODE-VAE and GRU-ODE-Bayes has an advantage in this regard.\n\n- The experiments section feels a little rushed and lacking in quality. For example,\n   - One of the reported benefits of LinODEnet is the computational efficiency due to the use of matrix exponentials. A comparison of compute time between the different models would have been desirable to verify this claim. This claimed efficiency is not so clear since there much be some tradeoff between the efficiency gain by linearity vs the need to lift the dynamics to a higher dimension.\n   - A novelty of this paper is the introduction of a neural Kalman filter, which distinguishes itself from RNN-style updates. An ablation study to show the efficacy of a self-consistent filter would have been desirable, e.g. by using the same latent dynamical model and encoder/decoder but with different filters.\n   - Likewise, some experiments showcasing the performance of the learned filter would have been a great addition, to demonstrate how useful it can be in online settings to perform long-term predictions. For example, including a plot of the filtered trajectory or showing the improvements in e.g. MSE after applying the update step will suffice.\n\n- The writing in some places is not very clear and the presentation can be improved (more details below).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject