SeqLink: A Robust Neural-ODE Architecture for Modelling Partially Observed Time Series
Authors: Futoon M. Abushaqra
Hao Xue
Yongli Ren
Flora D. Salim
Venue: Accepted by TMLR ()

Abstract: Ordinary Differential Equations (ODEs) based models have become popular as foundation models for solving many time series problems. Combining neural ODEs with traditional RNN models has provided the best representation for irregular time series. However, ODEs-based models typically require the trajectory of hidden states to be defined based on either the initial observed value or the most recent observation, raising questions about their effectiveness when dealing with longer sequences and extended time intervals. In this article, we explore the behaviour of the ODEs-based models in the context of time series data with varying degrees of sparsity. We introduce SeqLink, an innovative neural architecture designed to enhance the robustness of sequence representation. Unlike traditional approaches that solely rely on the hidden state generated from the last observed value, SeqLink leverages ODE latent representations derived from multiple data samples, enabling it to generate robust data representations regardless of sequence length or data sparsity level. The core concept behind our model is the definition of hidden states for the unobserved values based on the relationships between samples (links between sequences). Through extensive experiments on partially observed synthetic and real-world datasets, we demonstrate that SeqLink improves the modelling of intermittent time series, consistently outperforming state-of-the-art approaches.

Published in Transactions on Machine Learning Research (07/2024)
SeqLink: A Robust Neural-ODE Architecture for Modelling Partially Observed Time Series
Futoon M.Abushaqra
futoon abu www.:awww...oooosssseeee... edwit.tdd...o
School of Computing Technologies
RMIT University
Hao Xue
hao.zuel Grunsw. edu.au
School of Computer Science Engincering University of New South. Wales
Yongli Ren
yongli. edu.
School of Computing Technologies
RMIT University
Flora D.Salim
flora.salim Sunsw. edu..ox
School of Computer Science Engineering
Universily of New South Wales
Reviewed on OpenReview: https: // openreview. net/ forum? id= WCUT6l
Abstract
Ordinary Differential Equations (ODEs) based models have become popular as foundation models for solving many time series problems. Combining neural ODEs with traditional RNN models has provided the best representation for irregular time series. However, ODE- based models typically require the trajectory of hidden states to be defined based either the initial observed value or the most recent observation, raising questions about their effec- tiveness when dealing with longer sequences and extended time intervals. In this article, we explore the behaviour of the ODE-based models in the context of time series data with vary- ing degrees of sparsity. We introduce SeqLink, an innovative neural architecture designed to enhance the robustness of sequence representation. Unlike traditional approaches that solely rely on the hidden state generated from the last observed value, SeqLink leverages ODE la- tent. representations derived from multiple data samples, enabling it to generate robust data representations regardless of sequence length data sparsity level. The core concept be- hind our model is the definition of hidden states for the unobserved values based on the relationships between samples (links between sequences). Through extensive experiments on partially observed synthetic and real-world datasets, we demonstrate that SeqLink im- proves the modelling of intermittent time series, consistently outperforming state-of-the-a approaches.
1 Introduction
Time series analysis of a complex irregular system is regarded one of the big problems in contemporary data science (Weerakody et al. 2021). Irregular time series occur in many fields, particularly for medical systems where the data is only caplured intermittenty, leading lo gaps in the sequences (Singh et al. 2019). These gaps may sometimes extend over long periods for example, a patient misses his/her appointments or fails use the medical devices home regularly. However, irrogular time scrics capture dynamic obscrvations without constant time basis, which makes hard to model them (Scargle 1982). Recently, the advances in neural ordinary differential equations (neural ODFs) (Chen et al. 2018), which produce networks with

---

Published in Transactions on Machine Learning Research (07/202) 2024)
18
08-
100
20%
800%%
6.1
00
100 Time point
500 Time point
1000 Time point
29
Sequence Length
(a)
(b)
(c)
Figure (a) Bumpy irregular time series (with unobserved data highlighted using a yellow-hatched), (b) Intermittent, irregular time series (with unobserved data highlighted using a yellow-hatched). (c) Perfor- mance of ODE-RNN model on synthetic intermillent trajectory data with different lengths (100, 500, and 1000 time points) and varying levels of sparseness. The results show that ODE-RNN model is influenced by both the length of the sequence and the sparsity level (time lapse between observations).
continuous hidden states, have become fundamental models for handling irregular sequences. ODEs describe the evolution in time of a process that depends on one variable (initial condition) (Chen et al. 2018); hence, for an ODE-based model, the continuous trajectories of the hidden state are described by just one variable (either the initial value or the last observed value). Since this trajectory represents the data, having the best possible representation of the hidden state is important. Thus neural ODEs are effective but may not provide the optimal representation for the whole sequence, especially when using sequences with long time lapses between observations (e.g. data from medical records). In such cases, the hidden states count the one initial value for a long time.
To illuminate the limilations of ODE-based models, we describe two distinctive irregular time series patterns: (a) bumpy series (Figure 1 a). which are characterised by frequent and short unobserved intervals, and (b) intermittent series (Figure bb), known for having long gaps with numerous unobserved values over extended periods, outlined in (Zhang et al., 2021). Subsequently, we investigate the behaviour of the ODE-based model under the influence of these irregular time series patterns by conducting set of experiments synthetic intermittent datascts with varying levels of sparsity (sets of consccutive unobserved values) described below in Section 5.1 We study the ODE-based model's performance sequences of different lengths -ssng ODE-RNN as an exemplar model. Specifically, we assessed the model forecasting accuracy for sequence longths of 100, 500, and 1000 time points, cach with varying sparsity levels ranging from 10% to 40%.
The results, illustrated in Figure ! c, show that both sequence length and the degree of sparsity significantly impact the prediction accuracy. However, the main challenge lies in having consecutive unobserved values for long time. The ODE-based model shows consistent pattern across different sequence lengths, indicating inverse relation between the percentage of unavailable observations and prediction accuracy. For example, when 30% of the data is unavailable, the error rate increases by an average of 17.8%, 32%, and 14% for sequence lengths of 100, 500, and 1000 time points respectively, compared to the results when only 10% of the values are unavailable. These results indicate that. the ODE representation of continuous unobserved data may vary basod on the amount of available data in relation to the sequence length. In the case of longer gaps, the model may not effectively capture the underlying dynamics of irregular time series. This behaviour might be related to the fact mentioned before that based models rely single initial state.
To overcome the limitation of ODE-based models on modelling long intermittent sequences, we present a novel ODE-based architecture (SeqLink) that does not rely on one trajectory (generated by the last available observation) to represent the data; it also produces more generalised hidden trajectories using information learned from similar samples. Our architecture comprises three major parts: (1) an ODE auto-encoder learn a better representation of the data by employing an encoder to construct hidden trajectorics based ODE; (2) attention module to categorise the learned representations based on the relations bet ween
2

---

Published in Transactions on Machine Learning Research (07/2024)
samples; (3) new ODE-based model (Link-ODE) designed and used to model time series by integrating all the categorised learned hidden trajectories from various samples and providing a continuous effective representation for the sequences. The contributions of this work are as follows:
We have demonstrated how traditional ODE-based models may yield inaccurate predictions and less effective hidden representation when applied to data with a high level of sparsity. Additionally, we show how they are affected by the length of the time lapse between observations. These experiments also serve as the motivation for this work.
We have proposed a novel approach that utilises diverse ODE-based hidden trajectories to provide an unrestricted representation for unobserved data, enabling us to maintain a good continuous representation over a long period.
Our proposed method, SeqLink, achieves improved performance over other recent models for time series lorecasting on both multivariate and univariate datasets. Additionally, our results demon- strate that SeqLink enhances the latent space representation of unobserved time series and improves prediction accuracy.
2
Related Work
Irregularity issues (also known as partially-observed time scries) are related to non-uniform intervals between observations (Kidger et al. 2020: Weerakody et al. 2021]. In a regular time series, the data follows a specific temporal sequence with a regular interval - for example, samples may always be observed daily. By contrast, in an irregular time series, samples are observed at unevenly spaced intervals. This issuc is common for data captured from humans (such as medical data and data on human behaviour), where the system depends on people's 's conmitment (Scargle 1982 Zhang et al. 2021 Tipton et al., 2016). Irregularity may also be caused by capturing data from heterogeneous sources and sensors (Deldari et al., 2020 Abbushaqra et al.) 2021; Almaghrabi et al. 2022). However, irregular sampling does not fit with standard machine learning models that assume fixed-size features (Narayan Shukla & Marlin 2021). p. Up until the last. few years, there were a number of traditional techniques used to handle irregularity, including analysing fully observed samples, performing a features analysis rather than a temporal analysis, or re-sampling and imputation (Zhang et al. 2021 Singh et al.] 2019; g these methods can destroy temporal information and dependencies.
Although recurrent neural network (RNN) [Robinson & Fallside 1987 Werbos 1988) shows outstanding performance in modelling temporal data, it does, on the other hand, assume both fixed gaps between observations and fully observed samples. Recently, with the development of neural ordinary differential equations (neural ODEs) (Chen et al. 2018) in 2018, more effective models have been proposed for irregular data. Neural ODEs is a contimuous- tieeeniuos--n-eee-oo model that defines a latent variable h(i) as the solution for an ODE initial value problem. Rather than specifying a discrete sequence of hidden layers, a continuous representation became possible using the parameterisation of the derivative. To utilise this advantage of the hidden state in neural ODEs, recent models like ODE-RNN and latent ODE (Rubanova et al., 2019) have presented continuous- time latent state, where the formation of the dynamics between observations is not predefined. These models define the state between observations to be the solution to an ODE, while normal RNN hidden cells are used to update the hidden state at each observation. Therefore the trajectories of the hidden state between observations are defined by the last observed value. As a particular case of the ODF-RNN model, (De Brouwer et al. 2019) provided the GRU-ODE-Bayes model, which includes a continuous-time version of the GRU (GRU-ODE) and Bayesian update network to handle the sporadic observations. The model combines GRU-ODE and GRU-Bayes, where the first one is used to update the hidden state h(i) in continuous time between observations, and the second is responsible for transforming the hidden state based on the new observation. In recent years, many models based on differential equations (DE) have been presented. These models, including work by Kidger et al. (2020); Morrill et al. (2021) Herrera et al. 2021]; Jia & Benson (2019h and others, have enriched our comprehension of DE behaviour and demonstrated enhanced performance across various scales.
To tackle the challenge of modifying the hidden trajectories based on newly received data, the controlled differential equation (CDF) and the neural rough differential equations (RDF) were introduced (Kidger et al.
3

---

Published in Transactions on Machine Learning Research (07/2024)
Morrill et al. 2021 In contrast to ODE, which primarily relies on their initial states with limited provisions for adjustments, CDE updates the driven value of the ODE equation, denoted as ds, by utilising a matrix vector represented as dx2.- Therefore the solution of CDE depends continuously on the evolution of x (driven by the control X). Neural RDE is an extended neural CDE where, in order to increase memory efficiency, especially for long sequences, the data modelled without embedding the interpolated path. In recent work by Takoviev et al. (2023), the authors also focused on faster modelling for long sequenees; they provided multiple shooting framework for latent ODE models that works by splitting trajectories of neural ODEs into short segments, optimising them in parallel to facilitate efficient training. Another notable contribution is the work by [Scirmmer et al. (2022), where they introduced continuous recurrent units (CRUs). These units integrate a linear stochastic differential equation (SDE) within an encoder-decoder framework, using the continuous-discreee Kalman filter to ensure smooth transitions between hidden states and an effective gating mechanism. It is worth noting, however, that the focus of these improved models has primarily been on cither facilitating faster learning or cahancing the representation of long soquenees. In contrast, our focus is directed towards generating a stable representation for irregular data sets characterised by longer gaps.
2020
Rescarchers have also explored irregular time scries modelling by combining ODE-based models with atten- tion mechanisms (Narayan Shukla & Marlin 2021 Jhin et al. 2022 Yuan et al. 2022) and long short-term memory (LSTM) networks (Lechner & Hasani 2020). Furthermore, ODEs have recently been applied in various fields, for instance, in (Yan et al.) 2020), where the authors studied the robustness of the neural ODEs and proposed the time-invariant steady neural ODE (TisODE). Their model was then applied to an image classification task by removing the time dependence of the dynamics in an ODE (Garsdal et al. 2022) Additionally, efforts have been made to reduce the high computational overhcad caused by the ODE-bascd models. Habiba & Pearlmutter (2020) redesigned RNN architectures such as GRU and LSTM using ODE, resulting in GRU-ODE and LSTM-QDE models that reduce the computation costs. The models leverage ODE solvers to compute hidden states and cell states, thus substantially reducing the computational cost of additional encoding and decoding used in the previous models (such as Latent-ODE and ODE-RNN). In a recent development, Zhhou et al. (2023) introduced the LS4 generative model. LS4, short for latent state space sequential sampler, is designed to capture and generate sequences of data by incorporating la- tent variables that evolve according to state space ODE. This model overcomes limitations in existing ODE-based generative models, especially for sequences with sudden changes. LS4 demonstrates enhanced performance and faster training. Another recent contribution [Chowdhury et al. (2023) introduced a method for self-supervised learning on irregular multivariate time series. This approach employs contrastive learning and data reconstruction tasks, maintaining the native irregularity of the data. Additionally, it incorporates a time-sensitive data reconstruction task, masking a fixed duration of data instead of a fixed number of observations to ensure tractable reconstruction across regions with varying sampling densities.
As numerical integration, which is used to approximate the solutions of ODEs using numerical methods, significantly influences the model performance, it remains subject of ongoing investigation. /hhu et al. (2022) explored neural ODEs and their interplay with numerical integration, revealing how neural ODEs approximale certain equations during training. Out et al. (2020) analysed the connection between differen- tial equations and ResNet, highlighting the strong link between ODE-based models and numerical solvers. Krishnapriyan et al.) (2022) conducted experiments to demonstrate the impact of numerical solvers on neural ODEs and proposed a convergence test select suitable solvers for continuous dynamics.
3 Preliminaries
In this section, we introduce the fundamental concepts of ODE-based models, along with the notion of ODE solvers. These concepts serve as the foundation for our proposed methodology. The notations used in the paper are summarised in Table !
ODE: ODE is mathematical equation that describes the rale of change of function with respect to an independent variable. In the context of time series data, ODEs capture the dynamics and relationships between variables over time. A simple form of an ODE given by: di = fx((t), t), where æ(t) represents the state of system at time i, and [((((), t) defines the rate of change of zz) at given time point.
de(1)
4

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: Symbols and Notations.
Notation
Description
K
A size of data of time series sequences.
k
An individual time series sequence extracted from the dataset, denoting sample sequence within the dataset.
D
The dimensionality of each time series, referring the number of fea- tures it encompasses.
time index indicating the chronological order of measurements within time series, starling from to Tt.
Y2
The final time point within a time series.
specific time point within the time index A time point where t é.
4
x
The actual observed values of a time series.
20,
The actual observed value at time of a time series.
mask matrix indicating the availability of observations.
m/
A mask value for specific time point, with value of either 0 indicating the presence or absence of a measurement at time point i.
A hidden state representing the internal activation of neural network layer at time point i.
ODE Solvers: Solving an ODE involves finding the solution ztt) that satisfies the given differential equation. ODE solvers are numerical methods used to approximate the solution of an ODE over a specified time interval. One common approach is the ODESolve method, which numerically integrates the ODE using discrete time steps. Given an initial condition stoo), an ODE solver approximates the values of axt) subsequent time points.
Neural ODEs: Neural ODEs are extension of traditional ODEs, where the function f(x(t),t) is param- eterized by neural network. Neural ODEs allow us to model complex and contimuous-time dynamics in a data-driven manner. The formulation of neural ODE is given by:
dh(() = (o(((t) 1), where h((a)) ho de ho, hn ODESolve(ff. ho, (to. ttn))s
(1)
(2)
where fo is neural network function with learnable parameters 0, and h(t) denotes the state of the system at time t. Given a function fe and an initial condition hy the ODESolve function is used to solve the differential equation and compute the values of h at sequence time points lo, ,4n- ,
a model that combines the strengths of ODEs and RNNs to model irregular time series data. In the ODE-RNN framework, the hidden state h, between observations defined using solutions to an ODE, While at observations, the hidden state is updated using an RNN cell, ensuring that both continuous-time dynamics and observftioonsseecific information are captured. The ODE-RNN formulation is given by:
ODE-RNN:
RNNCell(hi-Isz)) ODESolve(fe (t-1..... .4)) if mn = O,
if m; = 1
h4 =
(3)
where h; represents the hidden state time 4, I, is the observation at time f,, my is a mask value indicating if the observation is available (m, = 1) or not (m, = 0), and ODESolve is an ODE solver that integrates the ODE using the given function fo, initial state h;-1; 1, and a sequence of time points (4+-1)...,44).
5

---

Published in Transactions on Machine Learning Research (07 2024)
4 Methodology
4.1 Problem Statement
We consider modelling K sporadically observed time series with D dimensions. For example, data from K samples (e.g. palients) where D variables are potentially measured a specifie time point 4, Each time series is measured at time points (1, nt). The values of the observations are defined by value matrix x E RaxD and a mask matrix of size (n x D): E (0, 1) to indicate if the variable is observed (m; 1) not (m, = 0) at each time point i. We assume specific time series to be sporadically sampled when some samples x have m equal to (0) at one or more time points. The goal is to model the sporadie time series effectively by finding the best continuous latent representation for the entire sequences.
4.2 Overview of SeqLink
In this article, we present novel system for modelling irregular time series dala, siming to derive generalised continuous hidden representations for unobserved values. As shown in Figure _ our model comprises three key components: (1) ODE Auto-Encoder: This component uses neural ODEs to learn optimal hidden representations for each sample. It takes datasets as input, employing neural ODEs to capture continuous hidden trajectories that best represent each sample. Subsequently, it returns the most suitable representation for each sample. (2) Pyramidal Attention Mechanism: Designed to delineate correlations between samples, this method maps data with each other. By leveraging the learned representations input, it discerns, for each sample, the most relevant represenlations of other samples. It then sorts these representations based on their relationships to each sample. (3) Link-ODE: A generalised ODE-based model tailored to modelling partially observed irregular time series. By utilising the best-hidden trajectories to fill in gaps in the data, this model incorporates learned latent states from another related sample alongside sample-specifi latent states to represent each sample effectively. The remaining parts in this section will address each module of SeqLink.
4.3 ODE auto-encoder
Inspired by the idea of the denoising auto-encoder (AE) (Vincent et al., 2010), which utilises a corrupted input to train the encoder in order to obtain a high-quality embedding, our ODE-based auto-encoder aims to identify the optimal hidden representation (ODE hidden trajectory). The denoising AE is a type of neural network that learns to denoise data by encoding corrupted inpuls and then reconstructing the original data. Tn similar vein, our ODF-based AE seeks to minimise the reconstrueting error between the original data values and the decoder output y through the identification of the ODE hidden trajectory. However, our ODE AE deviates from the traditional denoising AE objective, as our primary goal is to obtain a robust and effective latent representation the data hat than focusing on explicitly denoising it. Therefore, the task is reconstructing x' (corrupted z) to the original data æ and then obtaining the learned data representations that yield the best result. The proposed ODE AE presented on the left-hand side of Figure 2 and consists of the following steps:
Generate x' by corrupting observed x in a time series k using cut out function based on a specific number of points to be removed from the timcline. The cut_out out function removes the data points by setting them to zero in both the value and mask vectors.
Each corrupted x' is processed by the ODE EIE-RNN encoder to learn the hidden representation h; at each time point = i. See Equations and 5 Where the RNN function is used to update the hidden state uy at observation time i for observation 24. While ODE solver is to solve ODE and get state u, at time t, when there are no observations (the time between i 1 and i, as Equations 4 as described in Section In other words, Equation is used to find the hidden state for the observations, and Equation used to find the hidden state between the observations (the gaps).
6

---

Published in Transactions on Machine Learning Research (07 (0722
OTB lave
Psromidal Afteotica
Map
OIDE sato-enredie
00000 G0000O Learel rgenscosiee
Iakel ippountatiin
Cheppe
Dunden adoael
@1 (KC Mo.
COD RNN
Ikggus arpattoe
Recesiraci
Dessder
Pynamid actting
muinBreim(n
Zus Maje
Fry Puty Out
Zi
Orggnl Eab
Chtpptt sotel
Inpals:
ditieen levea (1) head the inpprtiice raccs
Link ODBIk.
((1. q Jan
The architecture of SeqLink model with auto-encoder that generates the learned representation for each sequence, pyramidal attention module to sort the representation based on the correlations between samples, and Link-ODE to provide continuous effective represenlation for the sequence based on the learned information, where U (generated by ODE auto-encoder) is set of the best-learned representations U (( y=_ fu. Jz {u
Figure 2:
(k) }m }==1 for each sample at all time points t, a: the importance weights
for each latent representation. 1: level from 1 to [L] used to sort the learn representations.
= ODE RNN(2:) RNN NNN 14 = ODE RNN(()) ODESolve u;-1, (f:-1,-. ,t))
(4)
(5)
The latent. representations are solved back and decoded to the data space (to undo the effect of the corruption process), where a decoder model of linear sequence layers generates the predicted data
The generated data y is compared to the original with the goal of minimising the re-generating error between y and x as arg minsssss(( X, Y), where arg min represents the argument (or value) of
P P that minimiSes the function Loss(X,Y) p w, w', b, b' is the set of learning parameters to be optimised for the encoder and decoder, and Loss(-) is the loss function used to measure the similarity between and y. Finally, the learned hidden trajectories of each sample saved for the next phase (8)
P
as a set of U ((u(")) ())m Ji {ui -E {u }
(m)
4.4 Pyramidal Attention
We use the previously learned hidden states U to define a set of latent representations for each sequence based on the correlation between samples. Hence, we find the attention score between the samples and the learned representations from the auto-encoder. As shown in Figure 2 we first map the original data (x) to the learned representations (u) by embedding both vectors as Equations and 7 where yp refers to the embedding layer and 0 represents the learning weights.
* = (z 0pp)) o
(6)
eu
2b
pu (u., Ocw)
(7)

---

Published in Transactions on Machine Learning Research (07/2024)
Next, concatenate layer combines both vectors (S²) as Equation g where @ is a set of learning parameters.
This is followed by attention layer that defines attention score to find the importance rate Oz between each x and u using Softmax function as the following formula:
S. = (e + e )
(8)
exp(S22
Og Σn K exp((22)
(9)
The assigned importance weights 0g are used to generate a set of hierarchical levels related latent repre- sentations as 12, 1., where each I represents subsct of the hidden states from U, i.c., 1 = {u S U}. At this stage, we use pyramidal sorting for each sample and corresponding learned representations according to the attention importance weights 0z (as shown in Algorithm (T). The input consists of the importance weights obtained through the attention layer, the learned representation for each sample, and the total num- ber of levels L, which determines the pyramid height. The output is a list of sorted representations for cach sample, defining the categorisation of entities based on their importance scores.
The pyramidal attention mechanism our model sorts learned representations based on correlations for each sample, offering advantages over selecting a single best. representation. This sorting ensures that the final representation for an unobserved value incorporates information from multiple related latent representations, roducing reliance single sample and enhancing the model's generalisabilit.. By constructing relevance pyramid, higher levels contain representations with higher importance rates, while lower levels contain repre- sentations with lower rates, allowing comprehensive consideration of information from all samples, avoiding over-reliance on individual samples and reducing the influence of outliers or noise. Moreover, the pyramidal mechanism introduces a negligible additional computational cost compared to using only attention, as it involves the same standard operations.
Algorithm 1: Pyramidal sorting
Input: Attention weights (rates) a Baaaned_rerrssenaationn U, number of level L sorted 0h =
// sorted weights for one sample
sorted u [ // sorted weights for all sample
for in 1.2,.., K do:
rates = [k,)
for in 1, 1. L:
MeanV = Σ rales/K K
Current L = Irates <= MeanV]
Current U = UCCrreent (CCureen__L]mennn m.mnn)))
10
rales raaees > MeanV]
sorted apppeddd_ (Current_
11
12 end for crtted_u.append(ssrtte _p)
sorted
14 Output: sorted
4.5 Link-ODE
Using the learned hidden representations, we define a Link-ODE network (shown in Figure 3) that in gen- erating continuous representation for the unobserved part not only considers the last observed value but also uses the ODE trajectories learned from different samples. The implementation of the Link-ODE is represented in Algorithm 2 An ODE solver is used to define the hidden state at time 4+1 based on the last observed value (initial value), while another ODE solver is used lo define a hidden state trajectory for + 1 for other samples (ODE solver from the ODE AE discussed in section 4.3). Finally, an RNN cell is used find the hidden state when there an observation based on hidden states from the ODE solver for the current sample, the hidden states of other samples, and the current value of z;. In other words, when m, 1 the hidden state is defined based the last hidden state and the current value of for each 2i, and
8

---

Published in Transactions on Machine Learning Research (07 2024)
ODE-RNN
f
1 f(h m
f(h
f(hj)
Link-ODE
Initial hidden state
f(u01 21
u
Hidden state learned by ODE for other samples
Continuous modelling for irregular samples, where ho is the initial state of a time point 4, f(-) is an ODE cell (update function) solve the ODE for h, and ug the previously learned trajectories from other samples zg the same time point.
Figure Architecture of Link-ODE model (the green part in Figure 2
when m, = the hidden state defined based on the ODE solver for the last hidden state and the hidden states learned from other samples. The previously learned states are combined and multiplied by a specific weight based on their level of correlation the sample (p), expressed in the following Equations (10
p = h, w 12, ,
(10)
h,41 =
T441)
(11)
p is a matrix encompassing all levels, where each 1 corresponds latent representations assigned that specific level. The weight values, denoted by w, established according to the levels, wherein latent reprosentations at given level receive higher weights when positioned at the pyramid's apex. This apex placement signifies heightened relevance to the current value of x, determined by the attention layer. After obtaining a continuous representation that encapsulates the learned relationships and dynamics within the data, a conventional output dense layer is employed, where the representations are then fed to generate the final output.
Algorithm 2: Link-ODE
Input: Data points x Output: lyy for in 1, do:
hit1 -.DESolve(fo. hay (Ii, (4+1)) // Solve ODE to get state at 441 based last hidden
state of Zt.
h ODESolve fo, hag, ((4))) // Solve ODE of h of other sample in K.
hi+1 NN 2441)
end for Pass;
5 Experiments
5.1 Datasets
Synthetic data (Gaussian trajectories): To evaluate the performance of our proposed model and analyse the model's robustness, we generated three synthetic datasets with 1000 samples of periodic trajectories and varying sequence lengths using the standard Gaussian function. The standard Gaussian function, denoted as (0, 1), is specific form of the Gaussian distribution with mean (p) of 0 and standard deviation of 1. It. is fundamental probability distribution to model random variables with continuous outcomes.

---

Published in Transactions on Machine Learning Research (07/2024)
Using that function, generate the following three periodic datasets: (1) a dataset with 100 time points (refer to it as PeriodicDalasel. 100), (2) a dataset with 500 time points (as PeriodicDalasel. 500) and (3) a dataset with 1000 time points (as PeriodicDattset_1100 simulate unobserved values by sub-sampling the data and generating sporadic sequences. We generate four sets for each datasel with different percentages sparsity (10%, 20%, 30%, and 40%). The cut-out function removes a value by setting it zero in the value tensor (z) and mask tensor (m). In total, twelve synthetic trajectories are used. This dataset was able to show the effect of sequence length and nunber of unobserved values on the performance of the models.
For each dataset, we use a cut-out function to
Real-world datascts: Additionally, we used four benchmark real-world datasets, including:
Electricity Consumption Load (ECL) D A public dataset that describes the clectricity consumption (Kwh) of 321 clients. From the ECL dataset, we used observations for one year and len sites. We generated sequences with 30 days lag and prediction one day ahead. Since the F.C. dataset is regular time series, we performed cut-out function on the samples. We randomly cut out 30% of the time points to generate sporadic series. In total, 3350 univariate sequences were used.
Electricity Transformer Temperature (ETT) Zhou et al. 2021 2 A multivariate time series that records two years of hourly data. The data has six features used to predict the electrical transformers' oil temperature based on load capacity. We used the available small dataset from 2016 and generated 2000 sequences with lag length of 24 hours and prediction one hour ahcad. For the ETT series we also cut out 30% of the observations to generate sporadic series.
Weather Data RL This data describes local climatological conditions for various US sites. It includes 11 climate features and the target value (a wet-bulb temperature). We built sequences of 7 day lag using hourly data with prediction one hour ahead. We used 2000 sequences and performed a 30% cut-out to generate irregular samples.
MIMIC-II (PhysioNet Challenge 2012 data) (Silaa et al 2012 An irregular medical data, describ- ing measurements for patients in ICU. It includes 48 measurements, 37 features, and a binary target value for in-hospital mortality. For our experiments, we used data for 1000 patients. Following [Rubanova et. al. 2019), we rounded up the time stamps to one minute to speed up the training process.
5.2 Experiment Details
Baselines: As our architecture is built based on ODE-RNN (Rubanova et al. 2019), we selected ODE-RNN as our major bascline. We also compared our models' performance to latent ODE (as this is a popular baseline model for irregularly sampled time series) ) (Rubanova et al. 2019), two classical time series models (RNN and RNN-VAE), and a novel ncural DE model (CDE) (Kddgrr ct al. 2020 Along with a recent state-of-the-art forecasting model, TSMixer (Chen et al. 2023
Setup: For all the experiments, we applied shuffled splitting divide the data into a training set and a tosting set. 80% of the samples were used for training, while the testing set held the remaining 20%. We re-scaled (normalised) the features between (0,1) for each datasel. To generate sparse data for the ECL, ETT, and weather datasets, we randomly cut out samples by setting the value to zero for the value and mask vectors. For all these real-world forecasting datasets, we cut out 30% of the time points in each sample. MIMIC-II is already a sparse dataset, so no cut-out function was performed. For model hyperparameters, to make the experiments fair and consistent, we followed (Rubanova et al. 2019) and chose the hyperparameters that yield the best performance for the original ODE-RNN. We ran both baselines and SeqLink with the exact same size of the hidden state and the same number of layers and units. For the auto-encoder in SeqLink we used ODE-RNN for the encoder and shallow sequential network of one linear layer to decode
2 'https://archiie ics uci cad..euuuuu wtt.///aaaaaas/////a//a//aassa htt.d//dtaassseeeeerrtttt..oonoe hwtpr///ee/eeteeoooaaararaaaaaa0 hhttps://github.a
wttp:///....//////oo/ooooooo/o/a /hhohaoyyii/////oooooo /Eaatasct
9httpe///wwwnncel.nne now..gov data//occll- cooall-ciimttolngicllllaaaa
4 httpps//pubmedd.neee wcb.m...ddddnii......... htt../i.........///67555555
10

---

Published in Transactions on Machine Learning Research (07/2024)
Table 2: Comparison of MSE values (for forecasting) and AUC values (for classification) SeqLink against various bascline models.
Forecasting task (MSE)
Dataset RNN
RNN VAE
Latent ODE
CDE
ODE-RNN
TSMixer
SeqLink
Gaussian data
0.767
7.695
8.28
8.686
0.126
1.606
0.114
ECL
0.520
3.400
0.620
2.136
0.500
1.2848
0.480
ETT
0.212
0.374
0.202
0.424
0.230
0.556
0.199
Weather
0.735
11.724
0.783
0.897
0.660
1.389
0.630
Classification task (AUC)
MIMIC-IT
0.667
0.518
0.701
NA
0.692
NA
0.720
Table 3: STD for test results over three runs for all baseline models and SeqLink on real-world datasets.
Dataset
RNN
RNN VAE
Latent ODE
CDE
ODE-RNN
TSMixer
SeqLink
Gaussian data
0.044
0.006
0.009
0.0856
6.59E-01
0.5061
2.24E-01
ECL
0.0018
0.0042
0.0024
0.009
0.0001
1.44
0.0001
ETT 0.1833
0.0057
0.0004
0.016
0.0081
0.1740
0.0509
Weather 0.0208
0.2216
0.0389
0.114
0.0274
0.1303
0.0065
the data to the data space and solve back the ODE. While for Link-ODE used the ODE function of one hidden layer and 100 units. The latent dimension used was 10 for all data sets. The fifth-order "dopri5" solver from the "torchdiffeq" python package was used as the ODE solver. This is the same one used in the traditional ODE-RNN model. We run 200 epochs on a batch size of 200. The same settings are used for all models LSeqLink and the baselines). Finally, We used mean squared error (MSE) to evaluate the prediction performance and area under curve (AUC) for the classification task.
Resource and Training: We used Adam optimizer and a learning rate of 0.01. The experiments were run a desktop with an NVIDIA GeForce MX230.
Implementation: We build our code on the publicly available code for ODE-RNN at /ttpp://githut
com/YullaaRuaaovv/llaaeee _dde,, using PyTorch. For the baselines (RNN-VAE, Latent ODE and ODE-
RNN) we follow the implementation available at https://github. hatpaa/caoooallaaaaaaaaaaaaaaaaa _oreeeeet__dde...eeee While for the CDE model, we follow the implementation available at hutpps://giiitbb.c///arrrrroo-re NeuralCDE). For TSMixer we follow the implementation available (tttps://gttthuu.com//iit pytorch-tsmixen Our code is available at htttpe//hhthhhacccooooooohhhhhen
5.3 Model Performance
Tables presents the results of the SeqLink model against different basclines. The first part shows the MSE values for forecasting tasks real-world data and the average performance for synthetic data (Gaussian trajectories), while the last row sets out the AUC results for the classification task performed on the MIMIC- II dataset. The results are the average of three runs with different random seeds used to initialise the model parameters. The best performance against each dataset is highlighted in bold. In general, our method outperforms the baselines for all real-world datasets. Among the baseline models, the ODF-RNN model achieved the best performance for the forccasting task. By contrast, RNN-VAE and latent ODE failed to model these partially observed dalasets. Although the CDE model was established to increase the representation learning capability of the neural ODEs, it required longer processing time and failed to solve forecasting tasks. In the classification task, SeqLink has the highest accuracy on the MIMIC-II data, while the Latent ODE model had the best performance among the baseline models. For completeness, since we report the average performance across three runs, we present the standard deviation error for each dataset in Table 3
Additionally, we present a comprehensive analysis of the performance of SeqLink and the bascline models using the synthetic trajectories to highlight the effectiveness and the behaviour of the model. The detailed
11

---

Published in Transactions on Machine Learning Research (07/2024)
Table 4: Comparison of MSE values of SeqLink against various baseline models on Gaussian trajectories datasets with different sparsity level.
Dataset
% of sparseness
RNN
RNN VAE
Latent ODE
CDE
ODE-RNN
TSMixer
SeqLink
PeriodicDataset 100
10%
1.127
7.535
8,390
7.106
0.297
2.952
0.280
20%
1.231
7.783
8.356
8.207
0.319
1.158
0.297
30%
1.097
7.345
8.425
8.772
0.349
1.084
0.318
40%
5.099
7.21
8.404
9.215
0.356
1.645
0.307
PeriodicDalased 500
10%
0.106
7.855
8.334
7.312
0.028
0.945
0.027
20%
0.131
7.884
8.391
8.418
0.029
1.715
0.026
30%
0.095
7.671
8.166
9.397
0.037
1.253
0.030
40%
0.128
7.565
8.072
9.948
0.039
1.646
0.032
Periodic Dataset 1000
10%
0.048
7.992
8.230
7.761
0.012
2.0512
0.012
20%
0.054
7,901
8.209
8.187
0.013
1.669
0.011
30%
0.040
7.872
8.289
9.897
0.014
1.3875
0.012
40%
0.052
7.731
8.094
10.02
0.017
1.765
0.015
Table 5: STD for test results over three runs on the synthetic datasets.
Dataset
% of sparseness
RNN
RNN VAE
Latent ODE
CDE
ODE-RNN
TSMixer
SeqLink
PeriodieDalaset 100
10%
0.0205
0.0022
0.0178
0.1087
0.0003
1.7898
0.001
20%
0.0212
0.0586
0.012
0.0643
0.0011
0.2244
0.0006
30%
0.0196
0.0021
0.0095
0.0399
0.0022
0.0603
0.0002
40%
0.468
0.002
0.0115
0.0355
0.0002
0.9501
2.8F-05
PeriodicDataset 500
10%
0.001
0.0007
0.0051
0.028
0.0012
0.0236
0.0003
20%
0.0017
0.0005
0.018
0.034
0.0009
0.6892
0.0002
30%
0.0016
0.0004
0.0065
0.046
0.0007
0.1594
0.0001
40%
0.0035
0.0008
0.0003
0.066
0.0009
0.7661
0.0001
PeriodieDalased 1000
10%
0.0001
7.0E-06
0.0058
0.1205
0.0003
0.5511
7.0E-06
20%
7.0F-06
0.0007
0.0004
0.057
7.0F-07
0.2017
8.7F-06
30%
5.6F-06
0.0001
0.0225
0.403
0.0001
0.1155
4.6F-05
40%
7.0E-06
0.0008
0.0004
0.025
2.8E-06
0,5416
0.0001
MSE values for all models using the synthetic dataset described before are given in Table and the standard deviation errors for three-time runs are presented in Table 5 Remarkably, across all sparsity levels, SeqLink consistently outperforms the baseline. Again, baseline models, including ODE-based models (latent-ODD) CDE), have failed to model this data, especially since the unobserved values consecutive. However, when examining the performance synthetic datasets with varying sequence lengths, both ODE-RNN and SeqLink exhibit improved performance for longer sequences. For instance, the average MSF value for Period- icDataset 100 using the ODE-RNN model is 0.33, but this figure significantly improves by over 89% to 0.033 for the PeriodicDataset. 500 dataset and further to 0.014 for the PeriodicDataset. 1000 dataset. A similar trend observed for SeqLink with a more than 90% reduction in MSE values for the Periodic.Datasel. 1000 and PeriodicDataset__ ment aligns with the intuitive notion that providing longer sequence for learning equips the model with more information, enhancing its effectiveness compared to shorter sequences with less information.
datasets compared to the Periodic.Datased 100 datasct. This obscrved improve-
Furthermore, when considering the impact of sparsity levels, distinct advantage of SeqLink bocomes evident, particularly in datasels with higher sparsity levels (fewer available observations). This advantage is highly explicit when the sequence length is short. In comparison, when examining the performance of SeqLink to the bascline (ODE-RNN), SeqLink consistently outperforms in scenarios with higher sparsity levels, especially when the sequence short. This is because having long gaps in short sequence means there less information (actual observation) available for learning.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Table 6: Percentage improvement when using SeqLink over traditional ODE-RNN. Note the increased ben- efits as the time lapses (gaps) grow longer, especially when the sequence length short.
Percentage of sparseness
10%
20%
30%
40%
PeriodicDatasct, 100
5.5%
6%
7%
14%
Datasel 500
1%
11%
18%
17%
PeriodicDataset 1000
3%
10%
7%
9%
Improvement rate 1%
20%
Table 0 provides insights into the percentage improvement achieved by utilising SeqLink over the traditional ODE-RNN. For the PeriodicDatased_ 100 with 10% sparsity, the performance of SeqLink exhibited a 5.5% improvement over the ODE-RNN. In contrast, this improvement inereased to 14% when 40% the values from the same dataset were unobserved. This suggests that our methodology, which defines hidden states between observations by considering data from different sequences, contributes to better representation, especially when the available obscrvations are limited. Morcover, both methods performed better for longer sequence lengths. Thus, the evaluation indicates that the model's performance is not solely dependent on the ratio of available to unavailable observations but also, to some extent, on the number of observations available. Additionally, this evaluation further emphasises that hidden states generated using traditional ODE-based models do not provide an effective representation of the whole sequences.
From our detailed analysis of the results for datasets with different lengths, we can draw attention the following conclusions: (1) The process of modelling irregular sequences is quite challenging for shorter sequences. (2) A model's efficiency irregular series does not only rely the percentage of unavailable observations, the total number of available observations also impacts performance.
5.4 Ablation Study
To explore the effectiveness of SeqLink we conducted series of experiments and tested the following various configurations to define the latent trajectories:
Unified hidden trajectories: Directly uses all learned hidden representations from the autoencoder without considering the relation between the samples (by removing the pyramid attention module).
Most related trajectories: Uses only those representations with the highest importance rate from among all the learned representations.
Least related trajectories:
Uses the representations with the lowest importance rates.
The results of the ablation study are presented in Table 7 The performance of the original SeqLink model is superior to the other three configurations. On the other hand, using the most important latent trajectory and the unified representation (underlined) outperforms the least important learned representations that show weak performance, which indicates that the attention layer does learn the relations between samples. Injecting hidden states of unrelated samples lowers the performance of the model, where unrelated samples affect the final representation. On the other hand, relying on all sequences may also introduce unrelated information to the reprosentation but with less effect compared with the previous case. Conversely, relying solely on the best related trajectories provides a very restrictive and biased hidden trajectory. Therefore, we can observe that the original framework of SeqLink outperforms.
6 Model Interpretation and Discussion
The total number of levels for the pyramid, denoted as (as illustrated in Figure 2 and discussed Section 4.4), a critical parameter in the SeqLink model. The value of determines the height of the pyramid, thereby influencing the importance assigned each correlated sample. To research this parameter, we examined several pyramids with different. sizes: (1) shorter pyramids with compacted levels, indicating fewer
13

---

Published in Transactions on Machine Learning Research (07/2024)
Table 7: Average MSE results from ablation studies of three different configurations.
Dataset
Unilied hidden
Most related
Least related trajectories
SeqLink
trajectories
trajectories
PeriodicDatasct 100
0.339
0.320
0.338
0.318
Periodic Dataset
500
0.031
0.033
0.034
0.030
PeriodicDataset
1000
0.013
0.013
0.023
0.012
ECL
0.491
0.501
0.505
0,480
ETT
0.203
0.204
0,211
0.199
Weather
0.708
0.684
0.711
0.630
MIMIC-II (AUC)
0.706
0.691
0.69
0.720
62
0.54
n 51
4ug
MSE
0.32196
0.50
0.323
error
0.57
0.48
0.3217
0.501
R- 81
0.56
ang 0.5492
0.4998
0.321
0.46
0.5
0.498
0.55
MSE
0.319
OV 0440
0.318
12
0.44
MSF crrur
0.499
0.498
Dny 0.5564
15
0.4069
0.54
0.42
0.318
0.4946
0.497
0.52
0.40
0.498
0.5652
aug 0.5751
6 =
5
Weight scale
0.495
==7
L
10
Weight scale
(a) Ganssian Trajectory dataset
(b) ECL dataset
any
61 0.157
0.438
MSE error
0.341
0.64
0,20
AUC
0.64
DJ1372
0.3405
0.72
0,197
51
11 30
0.70
0.635
nry
0.3400
0.7
0.195
0.63
0.68
MSE
0.3395
0.190
0.66
0.34
0.3390
0.187
0.62
8: 7
0.3385
0.62
0s 0.200
0.615
0,3380
0.6
0.180
L
L
Weight
L=5
5
==
7
Weight seale
(c) Weather datasets
(d) MIMIC-II dataset
Figure 4: Ilustrationn of the model's behaviour in terms of the attention weights for samples from different datasets using several values of L. Each plot shows the number of samples (n) assigned to the topmost layer of the pyramid after using different values of L. For each value, we display the average importance rates for these samples (aug) and the MSE (or AUC) values achieved by the model (blue line plot). Using very high very low value of L affects the model's performance, with the optimal performance consistently achieved al suitable value of L, which also depends on the size and correlation ween the samples.
14

---

Published in Transactions on Machine Learning Research (07/2024)
levels with more samples in each, and (2) taller pyramids with loose levels, signifying more levels but fewer samples in each.
Figure | presents the importance rates for samples from various datasets, showcasing the top layer of the pyramid at different values of L, along with the number samples n assigned to that layer, the average weight (importance) rate for these samples and the MSE error achieved by the model (blue line). Our findings reveal that highly compacted layers tend allocate more weight scores to less important samples due to the longer radius between the samples and correlated samples. Consequently, less correlated samples are considered more important. On the other hand, too loose levels focus very few correlated samples, introducing bias toward a limited set of samples. For instance, when L = 3, 51 samples were considered highly related for the Gaussian dataset with an average importance rate of 0.44, while only three were deemed related when L 7, with an average importance rate of 0.49. To see the effect of that on the model performance, we can notice that the MSE values (presented as a blue line plot) improve when having a larger value of L but then docrease for very large value of In the case of the Gaussian dataset, the best performance achieved was 0.31 for L= = 5, while MSE values were 0.321 when using 1, = 3 or I, 7. Similar trends could be noticed for other datasets where the performance improved for suitable value of L. For MIMIC-II, we report the AUC, where the model also showed improvement for L 7 with an accuracy of 70%, while it dropped to 67%, 68%, and 64% for L = 8, 6, and respectively. Notice that the value of L is also determined by the total size of the dataset, where we experimentally defined the range of values to explore for each dataset.
Grant
2egine (202-11))
10-
06-
0.00
870
8:2
1.5
045
0:50
100
(a) Sequences sample with 10% unobserved values
(b) Sequences of a sample with 40% unobserved values
Figure 5: Case Study: Sequences of two samples from PeriodicDataset. 100 dataset with 10% unobserved values (a) and 40% unobserved values (b). Actual values are shown with the green dashed line, SeqLink predictions are shown with the red dashed line, and baseline (ODE-RN) RNN) predictions are shown with the blue dashed line. SeqLink outperforms the baseline and can give more accurate predictions for the values that appear after the gap.
Furthermore, to demonstrate the robustness of the hidden representation of SeqLink, two case studies are investigated in Figures a and b. The figures show samples from PeriodicDalaset 100 dataset, one with 10% unobserved values, and the other with 40% of its values unobserved. Each figure illustrates the sequence of the actual values, the SeqLink predictions, and the ODE-RNN predictions. We have highlighted the predictions at different time points for more clarity. The generated sequences show that our model not only outperforms the overall sequence prediction but also gives a closer prediction for the values that show up directly after a gap, even when the gaps are longer. For instance, in 5bg the difference between the predicted and actual values for the time points 58 to is about 8% less when using SeqLink as compared with ODE-RNN, More case studies are presented Figure g Although the baseline is better at some time point (Figure Gd), proposed method outperforms most sequences. These cases implicitly prove that
15

---

Published in Transactions on Machine Learning Research (07/2024)
model's hidden representations for the unobserved time points more related than the hidden state calculated in the traditional ODE-RNN.
Ground
0.45
0.40
0.35
06
0.50
03
035
04
Decuie
Coundruhh
0.2
720
725
750
775
80.0
02.5
85.0
875
90.
92
94
95
95
(b)
0.50
0.5
Seglin
Raseine (008
Gumlin (200-99))
0.90
8130
0.55
0135
075
84h
(c)
(d)
Figure 6: Extra examples the predictions generated by ODE-RNN and SeqLink.
Finally, we employed the Wilcoxon rank-sum test determine whether the proposed model significantly outperformed the baseline model. The results from the previous examples demonstrated statistically significant difference in performance (p=0.0226) average, indicating that the proposed model produced superior results.
Test of significance:
7 Conclusion
In this work, we have proposed a novel ODE-based model to generalise hidden trajectories and analyse irregular data. We first explored the behaviour of the ODF-based model partially observed sequences with different lengths and time lapses. In response to the issues found, presented the SeqLink model that can build unrestricted ODE reprosentations for the unobserved values and maintain good continuous representations ions over a long time. The results of exlensive experiments on different tasks and datasets show that SeqLink outperforms other state-of-thh--as models. Our ablation study and case studies show that the framework of our model is able to improve the representation the unobserved values for partially-observed time series. Nevertheless, our proposed model was built on the ODE-RNN model as basic model. For future work, we plan to explore and investigate other ODF-based models using proposed framework.
References
Futoon Abushaqra, Hao Xue, Yongli Ren, and Flora Salim. Piets: Parallelised irregularity encoders for forecasting with heterogencous time-scrics. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 976 981. IEEE, 2021.
Sarah Almaghrabi, Mashud Rana, Margaret Hamilton, and Mohammad Saiedur Rahaman. Solar power time series forecasting utilising wavelet, coefficients. Neurocomputing, 508:182 207, 2022.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting. arXiv preprint arXiv:2303.0600 2023.
Ranak Roy Chowdhury, Jiacheng Li, Xiyuan Zhang, Dezhi Hong, Rajesh K Gupta, and Jingbo Shang, Primenet: Pre-training for irregular multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7184-7192, 2023.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes Continuous modeling of sporadically-observed time scries. Advances in newral information processing systems, 32, 2019.
Shohreh Deldari, Daniel V Smith, Amin Sadri, and Flora Salim. Espresso: entropy and shape aware time-
series segmentation for processing heterogeneous sensor data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(3):1-25 24, 2020.
ML Garsdal, V Søgaard, and SM Sørensen. Generative time series models using neural ode in variational auloencoders. arXiv preprini (xXiv:220.006300 2022.
Mansura Habiba and Barak A Pearlmutter. Neural ordinary differential equation based recurrent neural network model. In 2020 31st Trish Signals and Systems Conference (ISSC), pp. 1-6. TEFF, 2020.
Calypso Herrera, Florian Krach, and Josef Teichmann. Neural jump ordinary differential equations: Consis- tent continuous-time prediction and filtering. Ith International Conference on Learning Representations, ICLR 2021, Virtual Evend, Austria, May 3-7, 2021. OpenReview.nee, 2021.
Valerii lakovlev, Çagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Latent neural odes with sparse bayesian multiple shooting. In The Fleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL haap::://openreview..eee idd=
mo 11F2fj__
Sheo Yon Jhin, Jaehoon Lee, Minju Jo, Seungji Kook, Jinsung Jeon, Jihyeon Hyeong, Jayoung Kim, and Noscong Park. Exit: Extrapolation and interpolation-bassed neural controlled differential equations for time-series classification and forecasting. In Proceedings of the ACM Web Conference 2022, pp. 3102 3112, 2022.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Newral Information Processing Systems, 32, 2019.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696 6707, 2020.
Aditi S Krishnapriyan, Alejandro Queiruga, N Benjamin Erichson, and Michael W Mahoney. Learning continuous models for continuous physics. arXio preprint w:Xi::2202.08444 2022.
Mathias Lechner and Ramin Hasani. Learning long-ter term dependencies in irregularly-sampled time series.
arXiv preprint arXic:2006.00 04418, 2020.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rmns: Improved classification of clinical time series. In Machine learning for healthoore conference, pp. 253 2700. PMLR, 2016.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pp. 7829 7838. PMLR, 2021.
Satya Narayan Shukla and Benjamin M Marlin. Multi-time attention networks for irregularly sampled time series. Xixiv e-prints, pp. arXiv-2101, 2021.
17

---

Published in Transactions on Machine Learning Research (07/2024)
Katharina Ott, Prateck Katiyar, Philipp Hennig, and Michael Tiemann. Resnet after all: Neural odes and their numerical solution. In International Conference on Learning Regresenlations, 2020.
Anthony J Robinson and Frank Fallside. The utility driven dynamic error propogation network, volume 1. University of Cambridge Department of Engineering Cambridge, 1987.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Tatent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
Jeffrey D Scargle. Studies in astronomical time series analysis. ii-statistical aspects of spectral analysis of unevenly spaced data. The Astrophysical Journal, 263:835 853, 1982.
Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In International Conference on Machine Learning, pp. 19388-19449 19485. PMLR, 2022.
Tkaro Silva, George Moody, Daniel J Scolt, Leo Celi, and Roger Mark. Predicting in-hospital mortality of icu patients: The physionet/ jyysionetccomputngg /computing in cardiology challenge 2012. In 2012 Computing in Cardiology, pp. 248-248. IEEE, 2012.
Bhanu Pratap Singh, Iman Deznabi, Bharath Narasimhan, Bryon Kucharski, Rheeya Uppaal, Akhila Josyula, and Madalina Fiterau. Multi-resolution networks for flexible irregular time series modeling (multi-fit). arXiv preprint ar 19o5 00125,, 2019.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Philip B Wecrakody, Kok Wai Wong, Guanjin Wang, and Wendell Ela. A review of irregular time series data handling with gated recurrent neural networks. Neurocomputing, 441:161 178, 2021.
Paul J Werbos. Generalization of backpropagation with application recurrent gas market model. Newral networks, 1(4):339-33 356,, 1988.
Hanshu Yan, Jiawei Du, Vincent Y. F. Tan, and Jiashi Feng. On robustness of neural ordinary dif- ferential equations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL http:::///paarrrreeeeeeeettrtrrs
B1e9Y2NYVS)
Xiaofeng Yuan, Zhenzhen Jia, Lin Li, Kai Wang, Lingjian Ye, Yalin Wang, Chunhua Yang, and Weihua Gui, A sia-lstm based virtual metrology for quality variables in irregular sampled time sequence of industrial processes. Chemical Engineering Science, 249:117299 2022.
Chi Zhang, Hadi Fanaee-T, and Magne Thoresen. Feature extraction from unequal length helerogeneous ehr time series via dynamic time warping and tensor decomposition. Data Mining and Knowledge Discovery, 37(4):1760-1784, 2021.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang, Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceediiggs of AAAI, 2021.
Linqi Zhou, Michael Poli, Winnie Xu, Stefano Massaroli, and Stefano Ermon. Deep latent state space models for lime-series generation. In International Conference Machine Learning, pp. 42625 42643. PMLR, 2023.
Aiqing Zhu, Pengzhan Jin, Beibei Zhu, and Yifa Tang. On numerical integration in neural ordinary differ- ential equations. In International Conference Machine Learning, pp. 275477 PMLR, 2022.
18


Reviews:
[{"review_id": "Uz7iiqPa6D", "reviewer": "Reviewer_AfiC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n**Relevant problem**: The paper studies a highly relevant problem of forecasting intermittent and partially observed time series. These are common in real-world applications, but are not often considered in ML literature.\n\n**Strong empirical results:** The architecture proposed in the paper provides a consistent improvement over the ODE-RNN model, as well as other popular time series models, in terms of the prediction accuracy.\n### Weaknesses:\n**1. Presentation clarity**: The paper is missing the motivation for some critical model components (learned representations $u$ and the pyramidal attention block), and their precise definition is unclear. For example, according to Section 4.2, the ODE autoencoder produces for each time series $x \\in \\mathbb{R}^n \\times D$ a sequence of hidden states $\\{h_1, ..., h_n\\}$. However, according to Figure 2, the output of the autoencoder are the representations $u_{xi}, ..., u_{xn}$ for the individual time series, which contradicts Section 4.2. The meaning of the indexing in $u_{xi}, ..., u_{xn}$ is also unclear here.\n\nMoreover, the meaning of certain symbols ($f$, $\\theta$, $x_i$, $h_i$, $w$, $L$) is overloaded multiple times throughout the paper, which makes it hard to follow. For instance, $h_i$ is used to denote both the output of the ODE autoencoder as well as the output of the Link-ODE layer. Other symbols such as $l_i$ seem to be undefined, and are only used as input in Equation 10.\n\n**2. Unsupported claims**: The first of the claimed main contributions (\"We have demonstrated how ODE-based models may provide inaccurate and restricted hidden trajectories, as they are affected by the length of the time lapse between observations\") is not fully supported by the results presented in the paper. In fact, the results shows in Table 4 only demonstrate that the performance of the ODE-RNN model degrades as the fraction of missing data increases. However, this could be explained by other factors - such as the decreasing amount of training data. The latter hypothesis would be in line with other results shown in Table 4, as non-ODE-based models like RNN also show degrading performance as the fraction of missing data increases. I believe that a more careful investigation is necessary to support the claim about the restricted hidden trajectories.\n\nMore generally, the claims about the behavior of different models as sparsity is changing are based exclusively on synthetic gaussian datasets with periodic time series. It should be rather straightforward to extend this analysis to real-world data by introducing sparsity/missing values, which would strengthen the claim.\n\n### Minor details\n- In Figure 1, it's unclear where missing values correspond to observed zeros or unobserved values. A better idea would be to use a line plot with marks at observed/unobserved values. Also, the Y-axis is unlabeled.\n- There are numerous small typos in typesetting that hurt readability, such as $Xs$ instead of $X_s$, $arg_\\rho min$ instead of $\\arg \\min \\rho$, $l.w$ instead of $l \\cdot w$ - please consider proofreading the paper and fixing them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new neural ODE-based architecture for forecasting partially observed time series. The architecture extends the popular ODE-RNN model in several ways. First, an unsupervised autoencoder model is trained to reconstruct the missing time series values. Then, the representations for each time step learned by the autoencoder are passed through a pyramidal attention block to exchange information across the time series. The proposed architecture, called Link-ODE, is compared with various popular models for irregular time series forecasting on 4 real-world and 3 synthetic datasets. The proposed Link-ODE model provides consistent improvements over the base ODE-RNN model in terms of prediction quality, as measured by the MSE and ROC AUC metrics.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. (critical) Please provide a clearer explanation and motivation for the learned embeddings and the pyramidal attention in the model architecture. Here are some questions that I would like to see answered:\n\t- Are the embeddings $u$ free model parameters, or can they be computed inductively for new time series not seen at training time?\n\t- What is the purpose of the embeddings and the pyramidal attention? Is it to capture the interactions between distinct time series / samples (e.g., time step $i$ of sample $k$ may influence the time step $i^\\prime$ of sample $k^\\prime$?)? Why is capturing such interactions helpful for prediction if the time series are unrelated (e.g., time series corresponding to different patients in a medical dataset)?\n2. (critical) Please update central + right parts of Figure 2 as well as the corresponding equations, indicating precisely what are the input and output for each model component. Some inconsistencies and unanswered questions in the current figure that greatly hurt readability:\n\t- According to Equation 5, denoising autoencoder produces the hidden states $h_i$ as output, but the figure shows that the autoencoder returns representations $u$ as output\n\t- Indices in $u_{xi}, ..., u_{xn}$ probably contain typos\n\t- How are $l_1, ..., l_{|L|}$, $U$ defined?\n\t- What exactly is the input & output to pyramid attention?\n\t- Use consistent notation for indexing across individual items, time steps, and dimensions\n\t- How is the output of the Link-ODE layer used to compute the final objective function and train the model?\n3. (critical) Please adjust the claims related to the \"restricted hidden trajectories\" in ODE-based models, or provide a more thorough analysis supporting this claim.\n4. (recommended) The current ablations do not test all the novel architectural pieces introduced in this paper. For example, and even more informative ablation would be to directly use the hidden states $h_i$ produced by the autoencoder to predict the time series values, without using the sequence embeddings $u$ at all.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X3ar36dCMA", "reviewer": "Reviewer_1dTu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The approach relies on using multiple sequences for modeling an unseen trajectory, which to my knowledge is a novel aspect.\n- Strong results.\n\n### Weaknesses\n- One of the main claims of the paper (*\"ODEs describe the evolution in time of a process that depends on one variable (initial condition) (Chen et al., 2018); hence, for ODE-based models, the continuous trajectories of the hidden state are described by just one variable\"*) should be revisited. Although this holds for vanilla NODEs (Chen et al., 2018), upcoming works [1,2] partially addressed this. So, I'm not sure if *\"ODEs are effective but may not provide the optimal representation for the whole sequence\"*.\n- I don't clearly see how the problem of \"ODE models relying on a single initial state\" can be alleviated by \"more generalised hidden trajectories\". More specifically, what exactly does \"using multiple sequences\" bring?\n- Writing and presentation should be improved (see requested changes).\n- As far as I can see, hyperparameter selection is not discussed and some ablations are missing (e.g., how does the performance change with a non-linear decoder inside the encoder or with the number of training sequences considered in pyramidal attention, computational complexity and execution time of the proposed approach \n\n[1] Neural controlled differential equations for irregular time series \n\n[2] Latent Neural ODEs with Sparse Bayesian Multiple Shooting", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a neural network designed to enhance the robustness of sequence representations, which operates regardless of sequence length or data sparsity level. It uses other training sequences to enrich the latent representation that corresponds to an input sequence using an attention mechanism between the input sequence and other sequences. The model is compared against standard neural ODE variants on standard benchmarks, and performs comparatively well.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The works listed in the first weakness above should be discussed.\n- Caption of Figure 1c should be elaborated. Is the data \"bumpy\" or \"intermittent\"?\n- Symbols and notations table should be revisited, e.g., $K$ does not denote a set but set size, using $\\mathbf{x}$ to denote a vector of observations instead of $x$, why $Link − ODE$ instead of a single letter, etc.\n- What exactly is the cut_out function? What are its hyperparameters?\n- $f\\theta$ in (4) and $arg_\\rho min$ are incorrect notation.\n- How is $u$ obtained? Does it correspond to $h$ in the encoder?\n- What does $(t_{i-1},...t_i)$ mean in (4)?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YhkXsugewS", "reviewer": "Reviewer_kkXW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stregnths\n1. The work studies an interesting problem of modelling multiple time series samples together.\n2. Experiment results show the competitive performance against baseline models.\n3. The experiments settings are comprehensive with both synthetic and real-world data as well as ablation studies. \n\nWeakness:\n\nDespite the value of the problem the work focuses on and its proposed approach, the presentation of the work needs significant improvement before the reviewer can assess its contributions. The presentation could be improved in the following ways:\n1. Clearly define each notation. For example, in Sec. 4.3, $\\mu$ is not clearly defined; it is not clear how $S_{x_{i}}$ from Eq.9 is obtained from Eq. 8; in Algorithm 1, the inputs does not appear in the step 5-13, it is not clear where the variable $score$ is from and what is the output of the algorithm.\n2. For each component of the model, especially the pyramidal attention, clearly state the input and output.\n3. Consolidate some quantitative evaluation tables. For example, Table 2 and Table 3 can be combined into one table.\n4. Some most recent works might be missing in the baselines compared against. This work claims superior performance against baseline models on long sequences. Recently, state space models like S4[1] and Mamba[2] shows excellent performance on modelling long sequences and the model should consider comparing against these models. The denoising auto encoder of the work is also similar to some recent works on self-supervised pre-training for time series data [3].\n\nReferences\n\n[1] Gu, Albert, Karan Goel, and Christopher Ré. \"Efficiently modeling long sequences with structured state spaces.\" arXiv preprint arXiv:2111.00396 (2021).\n\n[2] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" arXiv preprint arXiv:2312.00752 (2023).\n\n[3] Chowdhury, Ranak Roy, et al. \"Primenet: Pre-training for irregular multivariate time series.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work proposes SeqLink a neural-ODE based architecture for modelling multiple irregular time-series data. The proposed approach consists of a denoising auto-encoder based on ODE-RNNs, a pyramidal attention module which sorts and correlates the learned representation across different samples, and a link ODE that takes learned representation from multiple samples as inputs. The work compares the proposed approach against other baseline models for irregular time series including RNN VAE, Latent ODE, CDE, ODE-RNN and SeqLink shows competitive performance against the baseline models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please make changes to address the weakness of the work, especially the weakness on presentation.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please carefully read through your submission and fix typos and language errors, there are some left as pointed out by Reviewer AfiC.
Just a few examples "A neural ODEs"; "Brouwer et al." (cited authors's name is De Brouwer as it is visible in the citation few words later), 
"ODEs models", etc.

Please change "Our code will be available soon" in the camera ready version with a reference to the Git repo.
Decision: Accept (poster)