Title: ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Irregular Time Series Modeling

Transformer

Neural Ordinary Differential Equation
Abstract: Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.

ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling

Abstract

Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.

1 Introduction

Irregular time series are prevalent in real-world applications like disease prevention, financial decisionmaking, and earthquake prediction [4, 27, 64]. Their distinctive properties set them apart from regular time series data. First, irregular time series data are characterized by irregularly generated or non-uniformly sampled observations with variable time intervals, as well as missing data due to technical issues, or data quality problems, which pose challenges for traditional time series analysis techniques [6, 55]. Second, even though the observations are irregularly sampled, the underlying data-generating process is assumed to be continuous [30, 42, 43]. Third, the relationships among the observations can be intricate and continuously evolving. All these characteristics require elaborate modeling approaches for better understanding these data and making accurate predictions.

The continuity and intricate dependency of these data samples pose significant challenges for model design. Simply dividing the timeline into equally sized intervals can severely damage the continuity of the data [36]. Recent works have suggested that underlying continuous-time process is appreciated for irregular time series modeling [6, 30, 39, 43], which requires the modeling procedure to capture

the continuous dynamic of the system. Furthermore, due to the continuous nature of data flow, we argue that the correlation within the observed data is also constantly changing over time. For instance, stock prices of tech giants (e.g., MSFT and GOOG) show consistent evolving trends, yet they are affected by short-term events (e.g., large language model releases) and long-term factors.

To tackle these challenges, researchers have pursued two main branches of solutions. Neural ordinary differential equations (Neural ODEs) and state space models (SSMs) have illustrated promising abilities for capturing the dynamic change of the system over time [9, 19, 28, 49]. However, these methods overlook the intricate relationship between observations and their recursive nature can lead to cumulative errors if the number of iterations is numerous [9]. Another line of research capitalizes on the powerful inductive bias of neural networks, such as various recurrent neural networks [6, 39, 45] and Transformer models [38, 48, 54, 60]. However, their use of fixed-time encoding or learning upon certain kernel functions fails to capture the complicated input-dependent dynamic systems.

Based on the above analysis, modeling the relationship between observations while capturing the temporal dynamics is a challenging task. To address this problem, we propose a Continuous-Time Transformer, namely ContiFormer , which incorporates the modeling abilities of continuous dynamics of Neural ODEs within the attention mechanism of Transformers and breaks the discrete nature of Transformer models. Specifically, to capture the dynamics of the observations, ContiFormer begins by defining latent trajectories for each observation in the given irregularly sampled data points. Next, to capture the intricate yet continuously evolving relationship, it extends the discrete dot-product in Transformers to a continuous-time domain, where attention is calculated between continuous dynamics. With the proposed attention mechanism and Transformer architecture, ContiFormer effectively models complex continuous-time dynamic systems.

The contributions of this paper can be summarized below.

Continuous-Time Transformer. To the best of our knowledge, we are the first to incorporate a continuous-time mechanism into attention calculation in Transformer, which is novel and captures the continuity of the underlying system of the irregularly sampled time-series data.

Parallelism Modeling. To tackle the conflicts between continuous-time calculation and the parallel calculation property of the Transformer model, we propose a novel reparameterization method, allowing us to parallelly execute the continuous-time attention in the different time ranges.

Theoretical Analysis. We mathematically characterize that various Transformer variants [34, 38, 48, 54, 60] can be viewed as special instances of ContiFormer. Thus, our approach offers a broader scope that encompasses Transformer variants.

Experiment Results. We examine our method on various irregular time series settings, including time-series interpolation, classification, and prediction. The extensive experimental results have illustrated the superior performance of our method against strong baseline models.

2 Related Work

Time-Discretized Models. There exists a branch of models based on the time-discrete assumption which transforms the time-space into a discrete one. Recurrent Neural Networks (RNN) [14, 24] and Transformers [51] are powerful time-discrete sequence models, which have achieved great success in natural language processing [15, 37, 58], computer vision [20, 63] and time series forecasting [31, 35, 52, 53, 56, 61, 62]. Utilizing them for irregular time series data necessitates discretizing the timeline into time bins or padding missing values, potentially resulting in information loss and disregarding inter-observation dynamics. [13, 36, 41]. An alternative is to construct models that can utilize these various time intervals [34, 60]. Additionally, several recent approaches have also encoded time information into features to model irregularly sampled time series, including time representation approaches [54, 60] and kernel-based methods [38], which allow learning for time dynamics. For instance, Mercer [54] concatenates the event embedding with the time representation. Despite these advances, these models are still insufficient in capturing input-dependent dynamic systems.

Continuous-Time Models. Other works shift to the continuous-time modeling paradigm. To overcome the discrete nature of RNNs, a different strategy involves the exponential decay of the hidden state between observations [6, 39]. Nevertheless, the applicability of these techniques is restricted to decaying dynamics specified between successive observations [9]. Neural ODEs are a class of

models that leverage the theory of ordinary differential equations to define the dynamics of neural networks [9, 27]. Neural CDE [30] and Neural RDE [40] further utilize the well-defined mathematics of controlled differential equations [2, 29] and rough path theory [5] to construct continuous-time RNN. ODE-RNN [43] combines Neural ODE with Gated Recurrent Unit (GRU) [12] to model continuous-time dynamics and input changes. Whereas these approaches usually fail to capture the evolving relationship between observations [45]. CADN [11] and ANCDE [26] extend ODE-RNN and Neural CDE by incorporating an attention mechanism to capture the temporal relationships, respectively. Another line of research explores the use of state space models (SSMs) to represent continuous-time dynamic systems [1, 18, 19, 49]. However, these models adopt a recursive schema, which hampers its efficiency and can lead to cumulative errors [9]. In contrast, our proposed model, ContiFormer leverages the parallelism advantages of the Transformer architecture and employs a non-autoregressive paradigm. Furthermore, extending the Transformer to the continuous-time domain remains largely unexplored by researchers. Such extension provides a more flexible and powerful modeling ability for irregular time series data.

3 Method

The ContiFormer is a Transformer [51] model for processing time series data with irregular time intervals. Formally, an irregular time series is defined as Γ = [( X 1 , t 1 ) , . . . , ( X N , t N )] , where observations may occur at any time and the observation time points ω = ( t 1 , . . . , t N ) are with irregular intervals. X i is the input feature for the i -th observation. We denote X = [ X 1 ; X 2 ; ..., X N ] ∈ R N × d .

As aforementioned, it is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle the challenge, we propose a Continuous-Time Transformer architecture as shown in Figure 1. Each layer of ContiFormer takes as input an irregular time series X and the sampled time ω and outputs a latent continuous trajectory that captures the dynamic change of the underlying system. With such a design, ContiFormer transforms the discrete observation sequence into the continuous-time domain. At each layer, the core attention module takes a continuous perspective and expands the dot-product operation in the vanilla Transformer to the continuous-time domain, which not only models the underlying continuous dynamics but also captures the evolving input-dependent process. For better understanding, the description of each layer will omit the layer index, without causing confusion.

Throughout the remaining section, we adopt the notation ω to denote a sequence of (reference) time points, while t is the random variable representing a query time point. Matrices are donated using uppercase letters, and lowercase letters except t represent continuous functions.

3.1 Continuous-Time Attention Mechanism

The core of the ContiFormer layer is the proposed continuous-time multi-head attention (CT-MHA) module, as shown in Figure 1. At each layer, we first transform the input irregular time series X into Q = [ Q 1 ; Q 2 ; . . . ; Q N ] for queries, K = [ K 1 ; K 2 ; . . . ; K N ] for keys, and V = [ V 1 ; V 2 ; . . . ; V N ] for values. At a high level, the CT-MHA module transforms the irregular time series inputs to latent trajectories and outputs a continuous dynamic system that captures the time-evolving relationship between observations. To accomplish this, it utilizes ordinary differential equations to define the latent trajectories for each observation. Within the latent state, it assumes that the underlying dynamics evolve following linear ODEs. Subsequently, it constructs a continuous query function by approximating the underlying sample process of the input. Ultimately, it produces a continuous-time function that captures the evolving relationship and represents the complex dynamic system.

Continuous Dynamics from Observations Transformer calculates scaled dot-product attention for queries, keys, and values [51]. In continuous form, we first employ ordinary differential equations to define the latent trajectories for each observation. Specifically, assuming that the first observation and last observation come at time point t 1 and t N respectively, we define continuous keys and values as

≈ç

≈ç

Traj.

�

Traj. for

��

Traj. for

��

Traj. for

��

Highlighted Traj.

Notations in CT-MHA module (Traj. stands for Trajectory)

Continuous-

Time MHA

Feed Forward

Add & Norm

...

Scaled Dot Product

Eq. (3)

Multiply by Value

Eq. (5)

Output

Eq. (7)

Aggregation

Eq. (6)

Output Traj. Z

key

query

value

time

( Q1

, t

1

)

( Q2

, t

2

)

Continuous-Time Multi-Head Attention (CT-MHA)

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

H

t

Vi

V2

V1

Ki

K2

K1

( Qi

, t

i

)

...

( X1

, t

1

) ( X

2

, t

2

)

( Xi

, t

i

)

...

Add & Norm

X = [

X1,

X2, ...,

Xi, ...

]

�

= [

t

1 ,

t

2 , ...,

t

i, ...

]

×

L

Q

K

V

� �

=[

��

�

, . . . , �

��

�

]

(reference points)

where t ∈ [ t 1 , t N ] , k i ( · ) , v i ( · ) ∈ R d represent the ordinary differential equation for the i -th observation with parameters θ k and θ v , and the initial state of k i ( t i ) and v i ( t i ) respectively and the function f ( · ) ∈ R d +1 → R d controls the change of the dynamics.

Query Function While keys and values are associated with the input and output of the attention mechanism, a query specifies what information to extract from the input. To model a dynamic system, queries can be modeled as a function of time that represents the overall changes in the input to the dynamic system at a specific time point t , i.e., q ( t ) . Specifically, we adopt a common assumption that irregular time series is a discretization of an underlying continuous-time process. Thus, similar to [30], we define a closed-form continuous-time interpolation function (e.g., natural cubic spline) with knots at t 1 , . . . , t N such that q ( t i ) = Q i as an approximation of the underlying process.

Scaled Dot Product The self-attention mechanism is the key component in Transformer architecture. At its core, self-attention involves calculating the correlation between queries and keys. This is achieved through the inner product of two matrices in discrete form, i.e., Q · K ⊤ . Extending the discrete inner-product to its continuous-time domain, given two real functions f ( x ) and g ( x ) , we define the inner product of two functions in a closed interval [ a, b ] as

Intuitively, it can be thought of as a way of quantifying how much the two functions 'align' with each other over the interval. Inspired by the formulation of the inner product in the continuous-time domain, we model the evolving relationship between the i -th sample and the dynamic system at time point t as the inner product of q and k i in a closed interval [ t i , t ] , i.e.,

Due to the nature of sequence data, it is common to encounter abnormal points, such as events with significantly large time differences. To avoid numeric instability during training, we divide the

integrated solution by the time difference. As a consequence, Eq. (3) exhibits discontinuity at α i ( t i ) . To ensure the continuity of the function α i ( · ) , we define α i ( t i ) as

Expected Values Given a query time t ∈ [ t 1 , t N ] , the value of an observation at time point t is defined as the expected value from t i to t . Without loss of generality, it holds that the expected values of an observation sampled at time t i is v i ( t i ) or V i . Formally, the expected value is defined as

Multi-Head Attention The continuous-time attention mechanism is a powerful tool used in machine learning that allows for the modeling of complex, time-varying relationships between keys, queries, and values. Unlike traditional attention mechanisms that operate on discrete time steps, continuous-time attention allows for a more fine-grained analysis of data by modeling the input as a continuous function of time. Specifically, given the forehead-defined queries, keys, and values in continuous-time space, the continuous-time attention given a query time t can be formally defined as

Multi-head attention, an extension of the attention mechanism [51], allows simultaneous focus on different input aspects. It stabilizes training by reducing attention weight variance. We extend Eq. (6) by incorporating multiple sets of attention weights, i.e.,

where W O , W Q (h) , W K (h) and W V (h) are parameter matrices, h ∈ [1 , H] and H is the head number.

3.2 Continuous-Time Transformer

Despite the widespread adoption of Transformers [51] in various research fields, their extension for modeling continuous-time dynamic systems is underexplored. We propose ContiFormer that directly builds upon the original implementation of vanilla Transformer while extending it to the continuous-time domain. Specifically, we apply layer normalization (LN) after both the multi-head self-attention (MHA) and the feed-forward blocks (FFN). We formally characterize the ContiFormer layer below

where z l ( t ) is the output from the l -th ContiFormer layer at time point t . Additionally, we adopt residual connection to avoid potential gradient vanishing [23]. To incorporate the residual connection with the continuous-time output, we approximate the underlying process of the discrete input X l using a continuous function x l ( t ) based on the closed-form continuous-time interpolation function.

Sampling Process As described before, each ContiFormer layer derives a continuous function z l ( t ) w.r.t. time as the output, while receiving discrete sequence X l as input. However, z l ( t ) can not be directly incorporated into neural network architectures that expect inputs in the form of fixed-dimensional vectors and sequences [48], which places obstacles when stacking layers of ContiFormer. To address this issue, we establish reference time points for the output of each layer. These points are used to discretize the layer output, and can correspond to either the input time points (i.e., ω ) or task-specific time points. Specifically, assume that the reference points for the l -th layer is ω l = [ t l 1 , t l 2 , ..., t l β l ] , the input to the next layer X l +1 can be sampled as { z l ( t l j ) | j ∈ [1 , β l ] } .

3.3 Complexity Analysis

We consider an autoregressive task where the output of each observation is required for a particular classification or regression task. Therefore, the reference points for each layer are defined as ω l = ω .

Table 1: Per-layer complexity (Comp. per Layer), minimum number of sequential operations (Seq. Op.), and maximum path lengths (Max. Path Len.). N is the sequence length, d is the representation dimension, T is the number of function evaluations (NFE) for the ODE solver in a single forward pass from t 1 to t N , and S represents the NFE from -1 to 1 .

To preserve the parallelization of Transformer architecture and meanwhile implement the continuous-time attention mechanism in Eq. (6), we first adopt time variable ODE [8] to reparameterize ODEs into a single interval [ -1 , 1] , followed by numerical

approximation method to approximate the integrals, i.e.,

where P denotes the number of intermediate steps to approximate an integral, ξ p ∈ [ -1 , 1] and γ p ≥ 0 , ˜ k i,j ( s ) = k i (( s ( t j -t i ) + t i + t j ) / 2) , and ˜ q i,j ( s ) = q i (( s ( t j -t i ) + t i + t j ) / 2) . Finally, α i ( t j ) can be solved with one invokes of ODESolver that contains N 2 systems, i.e.,

where p ∈ [1 , ..., P ] and we further define ξ 0 = -1 . Additionally, ˜ q i,j ( ξ p ) can be obtained by interpolating on the close-form continuous-time query function. Similar approach is adopted for solving ̂ v i ( t j ) in Eq. (5). The detailed implementation is deferred to Appendix A.

We consider three criteria to analyze the complexity of different models, i.e., per-layer complexity, minimum number of sequential operations, and maximum path lengths [51]. Complexity per layer refers to the computational resources required to process within a single layer of a neural network. Sequential operation refers to the execution of operations that are processed iteratively or in a sequential manner. Maximum path length measures the longest distance information must traverse to establish relationships, reflecting the model's ability to learn long-range dependencies.

As noted in Table 1, vanilla Transformer is an efficient model for sequence learning with O (1) sequential operations and O (1) path length because of the parallel attention mechanism, whereas the recurrent nature of RNN and Neural ODE rely on autoregressive property and suffer from a large number of sequential operations. Utilizing Eq. (9) and leveraging the parallelism inherent in the Transformer architecture, our model, ContiFormer, achieves O ( S ) sequential operations and O (1) path length, while also enjoying the advantage of capturing complex continuous-time dynamic systems. We have S ≪ T and we generally set S < N in our experiments.

3.4 Representation Power of ContiFormer

In the previous subsections, we introduce a novel framework for modeling dynamic systems. Then a natural question is: How powerful is ContiFormer? We observe that by choosing proper weights, ContiFormer can be an extension of vanilla Transformer [51]. Further, many Transformer variants tailored for irregular time series, including time embedding methods [48, 54] and kernelized attention methods [34, 38], can be special cases of our model. We provide an overview of the main theorem below and defer the proof to Appendix B.

Theorem 1 (Universal Attention Approximation Theorem) . Given query ( Q ) and key ( K ) matrices, such that ∥ Q i ∥ 2 < ∞ , ∥ Q i ∥ 0 = d for i ∈ [1 , ..., N ] . For certain attention matrix, i.e.,

Transformer

LatentODE

ContiFormer

TrueTrajectory

Interpolation

1

1

Extrapolation

Sampled Data

1

1

I

②

Prediction

1

Attn( Q,K ) ∈ R N × N (see Appendix B.1 for more information), there always exists a family of continuously differentiable vector functions k 1 ( · ) , k 2 ( · ) , . . . , k N ( · ) , such that the discrete definition of the continuous-time attention formulation, i.e., [ α 1 ( · ) , α 2 ( · ) , ..., α N ( · )] in Eq. (4), given by

satisfies that ˜ Attn( Q,K ) = Attn( Q,K ) .

4 Experiments

In this section, we evaluate ContiFormer on three types of tasks on irregular time series data, i.e., interpolation and extrapolation, classification, event prediction, and forecasting. Additionally, we conduct experiments on pendulum regression task [19, 45], the results are listed in Appendix C.5.

Implementation Details By default, we use the natural cubic spline to construct the continuous-time query function. The vector field in ODE is defined as f ( t, x ) = Actfn ( LN ( Linear d,d ( Linear d,d ( x ) + Linear 1 ,d ( t )))) , where Actfn ( · ) is either tanh or sigmoid activation function, Linear a,b ( · ) : R a → R b is a linear transformation from dimension a to dimension b , LN denotes the layer normalization. We adopt the Gauss-Legendre Quadrature approximation to implement Eq. (9). In the experiment, we choose the Runge-Kutta-4 [44] (RK4) algorithm to solve the ODE with a fixed step of fourth order and a step size of 0 . 1 . Thus, the number of forward passes to integrate from -1 to 1 , i.e., S in Table 1, is 80 . All the experiments were carried out on a single 16GB NVIDIA Tesla V100 GPU.

4.1 Modeling Continuous-Time Function

The first experiment studies the effectiveness of different models from different categories on continuous-time function approximation. We generated a dataset of 300 2-dimensional spirals, sampled at 150 equally-spaced time points. To generate irregular time points, we randomly sample 50 points from each subsampled trajectory without replacement. We visualize the interpolation and extrapolation results of the irregular time series for different models, namely Latent ODE (w/ RNN encoder) [9], Transformer [51] and our proposed

ContiFormer. Besides, we list the interpolation and extrapolation results in Table 2 using rooted mean squared error (RMSE) and mean absolute error (MAE) metrics. More visualization results and experimental settings can be found in Appendix C.1. As shown in Figure 2, both Latent ODE and ContiFormer can output a smooth and continuous function approximation, while Transformer fails to interpolate it given the noisy observations ( 1 ⃝ ). From Table 2, we can observe that our model outperforms both Transformer and Latent ODE by a large margin. The improvement lies in two

aspects. First, compared to Transformer, our ContiFormer can produce an almost 3 continuous-time output, making it more suitable for modeling continuous-time functions. Second, compared to Latent ODE, our ContiFormer excels at retaining long-term information ( 2 ⃝ ), leading to lower prediction error in extrapolating unseen time series. Conversely, Latent ODE is prone to cumulative errors ( 3 ⃝ ), resulting in poorer performance in extrapolation tasks. Therefore, we conclude that ContiFormer is a more suitable approach for modeling continuous-time functions on irregular time series.

4.2 Irregularly-Sampled Time Series Classification

The second experiment evaluates our model on real-world irregular time series data. To this end, we first examine the effectiveness of different models for irregular time series classification. We select 20 datasets from UEA Time Series Classification Archive [3] with diverse characteristics in terms of the number, dimensionality, and length of time series samples. More detailed information and experimental results can be found in Appendix C.2. To generate irregular time series data, we follow the setting of [30] to randomly drop either 30%, 50% or 70% observations.

We compare the performance of ContiFormer with RNN-based methods (GRU-∆ t [30], GRU-D [6]), Neural ODE-based methods (ODE-RNN [43], CADN [10], Neural CDE [30]), SSM-based models (S5 [49]), and attention-based methods (TST [57], mTAN [48]).

Table 3 presents the average accuracy and rank number under different drop ratios. ContiFormer outperforms all the baselines on all three settings. The complete results are shown in Appendix C.2.3. Moreover, attention-based methods perform better than both RNN-based methods and ODE-based methods, underscoring the significance of effectively modeling the intercorrelation of the observations. Additionally, we find that our method is more robust than other models. ContiFormer exhibits the smallest performance gap between 30% and 50%.

Besides, we study the learned attention patterns with different models. Attention captures the impact of previous observations on the future state of a system by measuring the correlation between different observations at different time points. Consequently, it is expected that a model

Time
t

ContiFormer

0.04

0

IS1

0.02

0

mTAN

0.04

Sample#1

Sample #2

0.03

0.0

0.2

0.4

0.6

0.8

1.0

effectively capturing the underlying dynamics would exhibit continuous, smooth, and input-dependent intermediate outcomes. As illustrated in Figure 3, ContiFormer and TST are both capable of

capturing input-dependent patterns effectively, while mTAN struggles due to the fact that its attention mechanism depends solely on the time information. In addition, ContiFormer excels in capturing complex and smooth patterns compared to TST, showcasing its ability to model time-evolving correlations effectively and potentially more robust to noisy data. Consequently, we believe that ContiFormer better captures the nature of irregular time series.

4.3 Predicting Irregular Event Sequences

Next, we evaluate different models for predicting the type and occurrence time of the next event with irregular event sequences [7, 21, 46, 47], a.k.a, marked temporal point process (MTPP) task. To this end, we use one synthetic dataset and five real-world datasets for evaluation, namely Synthetic, Neonate [50], Traffic [32], MIMIC [16], BookOrder [16] and StackOverflow [33]. We use the 4-fold cross-validation scheme for Synthetic, Neonate, and Traffic datasets following [17], and the 5-fold cross-validation scheme for the other three datasets following [39, 64]. We repeat the experiment 3 times and report the mean and standard deviations in Table 4. More information about the task description, dataset information, and training algorithm can be found in Appendix C.3.

We first compare ContiFormer with 3 models on irregular time series, i.e., GRU-∆ t, ODE-RNN, and mTAN, which respectively belong to RNN-based, ODE-based, and attention-based approaches. Additionally, we compare ContiFormer with baselines specifically designed for the MTPP task. These models include parametric method (HP [22]), RNN-based methods (RMTPP [16], NeuralHP [39]), and attention-based methods (SAHP [59], THP [64], NSTKA [38]).

The experimental results, as presented in Table 4, demonstrate the overall statistical superiority of ContiFormer to the compared baselines. Notably, the utilization of specific kernel functions, e.g., NSTKA and mTAN, results in subpar performance on certain datasets, highlighting their limitations in modeling complex data patterns in real-world scenarios. Furthermore, parametric methods like HP, perform poorly on most datasets, indicating their drawbacks in capturing real-life complexities.

4.4 Regular Time Series Forecasting

Lastly, we evaluate ContiFormer's performance in the context of regular time series modeling. To this end, we conducted an extensive experiment following the experimental settings from [53]. We employed the ETT, Exchange, Weather, and ILI datasets for time series forecasting tasks. We compare ContiFormer with the recent state-of-the-art model (TimesNet [52]), Transformer-based models (FEDformer[62], Autoformer[53]) and MLP-based model (DLinear [56]).

Table 5 summarizes the experimental results. Contiformer achieves the best performance on ILI dataset. Specifically, it gives 10% MSE reduction (2.874 → 2.632). Also, Contiformer outperforms Autoformer and FEDformer on Exchange, ETTm2, and Weather datasets. Therefore, it demonstrates that Contiformer performs competitively with the state-of-the-art model on regular time series forecasting. For a more comprehensive exploration of the results, please refer to Appendix C.4.

5 Efficiency Analysis

Employing continuous-time modeling in ContiFormer often results in substantial time and GPU memory overhead. Hence, we conduct a comparative analysis of ContiFormer's actual time costs against those of Transformer-based models (TST), ODE-based models (ODE- RNN, Neural CDE), and SSM-based models (S5).

Table 6 shows that both TST and S5 are effective models. Additionally, ContiFormer leverages a parallel mechanism for faster inference, resulting in faster inference speed as compared to ODE-RNN and Neural CDE. Furthermore, ContiFormer demonstrates competitive performance even with a larger step size, as detailed in Appendix D.3. Therefore, ContiFormer's robustness to step size allows it to achieve a superior balance between time cost and performance. Please refer to Appendix E for more details.

6 Conclusion

We propose ContiFormer, a novel continuous-time Transformer for modeling dynamic systems of irregular time series data. Compared with the existing methods, ContiFormer is more flexible since it abandons explicit closed-form assumptions of the underlying dynamics, leading to better generalization in various circumstances. Besides, similar to Neural ODE, ContiFormer can produce a continuous and smooth outcome. Taking advantage of the self-attention mechanism, ContiFormer can also capture long-term dependency in irregular observation sequences, achieving superior performance on a variety of irregular time series modeling tasks.

Table 6: Actual time cost during training for different models on RacketSports dataset (0% dropped) from UEA dataset. We adopt RK4 solver for ODERNN, Neural CDE, and ContiFormer. Step size refers to the amount of time increment for the RK4 solver. ↑ ( ↓ ) indicates the higher (lower) the better.

References

Abdul Fatir Ansari, Alvin Heng, Andre Lim, and Harold Soh. Neural continuous-discrete state space models for irregularly-sampled time series. arXiv preprint arXiv:2301.11308 , 2023.

Imanol Perez Arribas. Derivatives pricing using signature payoffs. arXiv preprint arXiv:1809.09466 , 2018.

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Stefan Bauer, Bernhard Schölkopf, and Jonas Peters. The arrow of time in multivariate time series. In International Conference on Machine Learning , pages 2043-2051. PMLR, 2016.

Youness Boutaib, Lajos Gergely Gyurkó, Terry Lyons, and Danyu Yang. Dimension-free euler estimates of rough differential equations. arXiv preprint arXiv:1307.4708 , 2013.

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports , 8(1):6085, 2018.

Chao Chen, Haoyu Geng, Nianzu Yang, Junchi Yan, Daiyue Xue, Jianping Yu, and Xiaokang Yang. Learning self-modulating attention in continuous time space with applications to sequential recommendation. In International Conference on Machine Learning , pages 1606-1616. PMLR, 2021.

Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. In International Conference on Learning Representations , 2021.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Jen-Tzung Chien and Yi-Hsiang Chen. Continuous-time self-attention in neural differential equation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3290-3294. IEEE, 2021.

Jen-Tzung Chien and Yi-Hsiang Chen. Learning continuous-time dynamics with attention. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 , 2014.

Edward Choi, Cao Xiao, Walter Stewart, and Jimeng Sun. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. Advances in neural information processing systems , 31, 2018.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1555-1564, 2016.

Yuchen Fang, Kan Ren, Caihua Shan, Yifei Shen, You Li, Weinan Zhang, Yong Yu, and Dongsheng Li. Learning decomposed spatial relations for multi-variate time-series modeling. In Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems , 35:35971-35983, 2022.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021.

Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12175-12185, 2022.

Vinayak Gupta and Srikanta Bedathur. Proactive: Self-attentive temporal point process flows for activity sequences. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 496-504, 2022.

Alan G Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika , 58(1):83-90, 1971.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences , 79(8):2554-2558, 1982.

Sheo Yon Jhin, Heejoo Shin, Sujie Kim, Seoyoung Hong, Minju Jo, Solhee Park, Noseong Park, Seungbeom Lee, Hwiyoung Maeng, and Seungmin Jeon. Attentive neural controlled differential equations for time-series classification and forecasting. Knowledge and Information Systems , pages 1-31, 2023.

Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems , 32, 2019.

Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02435 , 2022.

Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. Advances in Neural Information Processing Systems , 32, 2019.

Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems , 33:6696-6707, 2020.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Jure Leskovec and Andrej Krevl. Snap datasets: Stanford large network dataset collection, 2014.

Jiacheng Li, Yujie Wang, and Julian McAuley. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th international conference on web search and data mining , pages 322-330, 2020.

Yan Li, Xinjiang Lu, Haoyi Xiong, Jian Tang, Jiantao Su, Bo Jin, and Dejing Dou. Towards longterm time-series forecasting: Feature, pattern, and distribution. arXiv preprint arXiv:2301.02068 , 2023.

Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. In Machine learning for healthcare conference , pages 253-270. PMLR, 2016.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Yu Ma, Zhining Liu, Chenyi Zhuang, Yize Tan, Yi Dong, Wenliang Zhong, and Jinjie Gu. Nonstationary time-aware kernelized attention for temporal event prediction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1224-1232, 2022.

Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. Advances in neural information processing systems , 30, 2017.

James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning , pages 7829-7838. PMLR, 2021.

Michael C Mozer, Denis Kazakov, and Robert V Lindsey. Discrete event, continuous time rnns. arXiv preprint arXiv:1710.04110 , 2017.

Sung Woo Park, Kyungjae Lee, and Junseok Kwon. Neural markov controlled sde: Stochastic optimization for continuous-time data. In International Conference on Learning Representations , 2022.

Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems , 32, 2019.

Carl Runge. Über die numerische auflösung von differentialgleichungen. Mathematische Annalen , 46(2):167-178, 1895.

Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In International Conference on Machine Learning , pages 19388-19405. PMLR, 2022.

Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. arXiv preprint arXiv:1909.12127 , 2019.

Oleksandr Shchur, Nicholas Gao, Marin Biloš, and Stephan Günnemann. Fast and flexible temporal point processes with triangular maps. Advances in Neural Information Processing Systems , 33:73-84, 2020.

Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations , 2021.

Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933 , 2022.

Nathan J Stevenson, Karoliina Tapani, Leena Lauronen, and Sampsa Vanhatalo. A dataset of neonatal eeg recordings with seizure annotations. Scientific data , 6(1):1-8, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186 , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. Advances in neural information processing systems , 32, 2019.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8980-8987, 2022.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2114-2124, 2021.

Jing Zhang, Peng Zhang, Baiwen Kong, Junqiu Wei, and Xin Jiang. Continuous self-attention models with neural ode networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 14393-14401, 2021.

Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes process. In International conference on machine learning , pages 11183-11193. PMLR, 2020.

Yuan Zhang. Attain: Attention-based time-aware lstm networks for disease progression modeling. In In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI-2019), pp. 4369-4375, Macao, China. , 2019.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning , pages 27268-27286. PMLR, 2022.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020.

Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In International conference on machine learning , pages 11692-11702. PMLR, 2020.

A Implement Details and Complexity Analysis

A.1 Background: Time Variable ODE

A numerical ODE solver integrates a single ODE system d x dt = f ( t, x ) , where x ∈ R d and f : R d +1 → R d . Given an initial state x t 0 , the ODE solver produces x t 1 , x t 2 , ..., x t N , which represents the latent state for each time point [9], i.e.,

To obtain the solution of Eq. (12), it requires the ODE solver to integrate over the interval [ t 0 , t N ] .

Varied time intervals Now, we suppose that we need to solve M dynamic system that has different time intervals. Let denote for the i -th system, the start point is t ( i ) 0 and the endpoint is t ( i ) 1 and the ODE system is defined by d x dt = f ( t, x ) . We can construct a dummy variable so that the integral interval always starts from -1 and ends at 1 and perform a change of variables to transform every system to use this dummy variable.

For simplifying purposes, we assume that a single system that integrates from t 0 to t 1 with f ( t, x ) can control the differential equation and the solution of x ( t ) . We introduce a dummy variable s = 2 t -t 0 -t 1 t 1 -t 0 and a solution ˜ x ( s ) in a fix interval [ -1 , 1] satisfies

The differential equation that solve x follows that

Since ˜ x ( -1) = x ( t 0 ) and ˜ x (1) = x t 1 , thus we have

A.2 Background: Approximating Definite Integral of a Function

Given a function f that integrates over an interval [ -1 , 1] , a simple method to approximate it is to assume that the change is linear over time, i.e.,

Besides, in numerical analysis, Gauss-Legendre Quadrature provides a more accurate way to approximate an integral over an interval [ -1 , 1] , i.e.,

where { ω j } and { ξ j } are quadrature weights and nodes such that -1 ≤ ξ 1 < ... < ξ k ≤ 1 .

A.3 Putting it all together

Given an irregular-sampled sequence, Γ = [( x 1 , t 1 ) , ( x 2 , t 2 ) , ..., ( x N , t N )] , where x i ∈ R d is the input feature of the i -th tokens happen at time t i and N is the length of the sequence. We analyze the time and space complexity for an autoregressive task where the output of each observation is required for a particular classification or regression task. Specifically, for Eq. (3), α i ( t 1 ) , α i ( t 2 ) , . . . , α i ( t N )

is required to form the output of Transformer. Therefore, the key difficulty is to solve Eq. (3). By using the numerical approximation method, we have

where ˜ q i,j and ˜ k i,j follows that:

We use ˆ k i,j to denote d ˜ k i,j ( s ) ds . Thus, α i ( t j ) can be solved by paralleling N 2 ODE solvers. Specifically, ˜ k i,j ( ξ p ) is given by

As for ˜ q i,j ( ξ k ) , since the input sequence is irregularly sampled and discrete in time, we propose to use interpolation methods [30] (e.g., linear interpolation or cubic interpolation) method on Q to construct a continuous-time query. Therefore ˜ q i,j ( ξ k ) can be easily calculated in O ( d ) time complexity.

B Representation Power of ContiFormer

In the section, we mathematically prove that by curated designs of function hypothesis, many Transformer variants can be covered as a special case of ContiFormer. For simplification, we only discuss the self-attention mechanism, where Q,K,V ∈ R N × d .

B.1 Transformer Variants

We first show several Transformer-based methods for irregular time series. Most of these methods shown below calculate N × N attention matrix and multiply it with V 4 .

Vanilla Transformer Given query Q ∈ R N × d and V ∈ R N × d , the attention matrix of a vanilla transformer [51] is calculated as

and the output of the self-attention mechanism is obtained by 5

Kernelized Attention Kernelized attention mechanism 6 incorporate time information through defining specific kernel function k ( x, x ′ ) and replace the attention matrix with

Kernel functions are typically symmetric and hold that ϕ ( x, x ) = 1 for any x . For instance, NSTKA [38] propose a learnable Generalized Spectral Mixture (GSM) kernel, i.e.,

where µ i ( x ) denotes the mean, lengthscale as l i ( x ) and input-dependent weight as ϕ i ( x ) .

Time Embedding Methods Time embedding methods learn a time representation, adding or concatenating it with the input embedding, i.e.,

where ( · , · ) is the concatenation of two vectors, Φ( · ) : R → R d is a learnable time representation. For simplification, we assume that time embedding is concatenated with the input embedding and ignore linear transformation on the time embedding. 7

For instance, Mercer [54] transforms the periodic kernel function into the high-dimensional space spanned by truncated Fourier basis under certain frequencies ( { ω i } ) and coefficients { c i } , i.e.,

mTAN [48] generalizes the notion of a positional encoding used in Transformer-based models to continuous-time domain, i.e.,

where w ∈ R d × d is a linear transformation 8 and

where ω and ϕ are learnable weights. Furthermore, mTAN assumes that Q = K = 0 .

Besides, since Softmax( x + c ) = Softmax( x ) , where x is a vector and c is a constant. The following equation holds,

where the matrix form of Softmax normalize each row of the matrix and

B.2 Proof of Extension and Universal Approximation

Lemma 1 (Existence of Continuously Differentiable Vector Function) . Given a continuously differentiable vector function u : R → R d ∈ C 1 , such that ∀ x ∈ R , ∥ u ( x ) ∥ 2 < ∞ , ∥ u ( x ) ∥ 0 = d and a continuously differentiable function w : R → R ∈ C 1 , there always exists another continuously differentiable vector function v : R → R d ∈ C 1 with a fixed point p satisfying u ( p ) · v ( p ) ⊤ = w ( p ) , such that ∀ x ∈ R , u ( x ) · v ( x ) ⊤ = w ( x ) holds. 9

Proof. Throughout the proof, we define ξ := v ( p ) .

Since ∀ x ∈ R , 0 < ∥ u ( x ) ∥ 2 < ∞ , we define

where k ( x ) ∈ C 1 such that ∀ x ∈ R , u ( x ) · k ( x ) ⊤ = 0 holds. Therefore, for ∀ x ∈ R , we have

Thus, we only need to prove the existence of k ( x ) . Since v ( x ) has a fixed point at p , therefore,

Let z [ i ] denote the i -th element of a vector z , thus, Eq. (33) is equivalent to

for i ∈ [1 , d ] . Since we have u ( p ) · v ( p ) ⊤ = w ( p ) , thus,

We denote c i := k ( p )[ i ] ∗ u ( p )[ i ] . Since Eq. (35) holds, therefore ∑ d i =1 c i = 0 . Finally, we can construct k ( x ) as

Since u, w ∈ C 1 , therefore k ∈ C 1 , thus v ∈ C 1 .

Lemma 2 (Existence of k i ( t ) ) . Given a continuously differentiable vector function q : R → R d ∈ C 1 such that ∀ x ∈ R , ∥ q ( x ) ∥ 2 < ∞ , ∥ q ( x ) ∥ 0 = d and a continuously differentiable function f : R → R ∈ C 2 and a vector K i ∈ R d and satisfies f ( t i ) = q ( t i ) · K ⊤ i , there always exist a continuously differentiable vector function k i : R → R d , such that the following statements hold:

k i ( t i ) = K i ,

̸

̸

Proof. Since t = t i , we have

where 1 ⃝ holds since we can define α i ( t i ) = f ( t i ) . Besides, since

such a definition does not break the continuity of α .

Since f ∈ C 2 , therefore we denote g ( t ) := d f d t ( t -t i ) + f ( t ) ∈ C 1 . Besides,

By using Lemma 1 with g corresponding to w , q to u , k i to v and p = t i , we can prove the existence of k i that satisfies Eq. (37).

Theorem 2 (Universal Attention Approximation Theorem) . Given query ( Q ) and key ( K ), such that ∥ Q i ∥ 2 < ∞ , ∥ Q i ∥ 0 = d for i ∈ [1 , ..., N ] . For any attention matrix with formulation defined in Appendix B.1, i.e., Attn( Q,K ) , there always exists a family of continuously differentiable vector functions k 1 ( · ) , k 2 ( · ) , . . . , k N ( · ) , such that the discrete formulation of [ α 1 ( · ) , α 2 ( · ) , ..., α N ( · )] defined by

satisfies that ˜ Attn( Q,K ) = Attn( Q,K ) , where

and α i ( t i ) = q ( t i ) · k i ( t i ) ⊤ , q ( t ) satisfies q ( t i ) = Q i .

Proof. First, we show that the diagonal elements in Eq. (40), i.e.,

holds for different forms of attention matrix defined in Appendix B.1.

Next, for a given i ∈ [1 , N ] , there exists a continuous cubic spline h i : R → R satisfies h i ( t j ) = Attn( Q,K ) j,i for ∀ j ∈ [1 , . . . , N ] and h i ∈ C 2 . By using Lemma 2 with h i corresponding to f , there exists a function k i such that α i ( t ) = h i ( t ) . Therefore, α i ( t j ) = h i ( t j ) for j ∈ [1 , ..., N ] , and we finish the proof.

C Experiment Details

C.1 Modeling Continuous-Time Function

C.1.1 Background: 2D Spiral

A spiral is a curve that starts from a central point and gradually moves away from it while turning around it. We consider two types of spirals that start from time s and end at time e . The first type of spiral is defined as

while the second type of spiral is defined as

where a and b are parameters of the Archimedean spiral.

̸

C.1.2 Experiment Setting

We generate 300 spirals and 200/100 spirals are used for training/testing respectively. For each spiral, it randomly belongs to either a clockwise spiral or a counter-clockwise spiral, we sample a ∼ N (0 , α ) , b ∼ N (0 . 3 , α ) and we set α = 0 . 02 for Table 2. Also, we add Gaussian noise from N (0 , β ) to all training samples and we set β = 0 . 1 . We test different models for interpolation and extrapolation. Specifically, each spiral is sampled at 150 equally-spaced time points. To generate an irregular time series, we randomly sample 30 points from the first half of the trajectory. Thus, the first half of the trajectories are used for the interpolation task while the second half of the trajectories, which is totally unobserved during both training and testing, are used for the extrapolation task. All the models are trained for 5000 iterations with a batch size equal to 64.

C.1.3 Baseline Implementation

We compare ContiFormer with Neural ODE-based models and Transformer.

Neural ODE[9]. This model uses the hidden vectors from the last observation as input, and obtain both the interpolation and extrapolation result from a single ODE solver. We train the model by minimizing the mean square loss.

Latent ODE (w/ RNN Enc)[9]. We run an RNN encoder through the time series and infer the parameters for a posterior over z t 0 . Next, we sample z t 0 ∼ q ( z t 0 | { x t i , t i } ) . Finally, we obtain both the interpolation and extrapolation result from a single ODE solver, i.e., ODESolve( z t 0 , f, θ f , t 0 , ..., t M ) , where ( t 0 , t 1 , ..., t M ) is the time of the trajectory. We train the model by maximizing the evidence lower bound (ELBO).

Latent ODE (w/ ODE Enc)[43]. We replace the RNN encoder in Latent ODE (w/ RNN Enc) with the ODE-RNN encoder following [43]. Again, we train the model by maximizing the evidence lower bound (ELBO).

Neural CDE[42]. We follow the official implementation to construct the CDE function. For the interpolation, the hidden vectors are directly obtained from the continuous-time path from Neural CDE. For the extrapolation, we input the hidden vectors from the last timestamp of the Neural CDE to a single ODE solver. We train the model by minimizing the mean square loss.

Transformer[51]. In order to obtain the prediction for both seen and unseen time points, we use a learnable mask for unseen inputs. Instead of using position embedding, the input is added with a temporal encoding before forwarding it to the transformer layer. We train the model by minimizing the mean square loss.

ContiFormer. We use linear interpolation to deal with unseen observations to avoid the numerical error caused by constructing the continuous-time query. Similar to Transformer, temporal encoding is adopted and the model is trained by minimizing the mean square loss.

C.1.4 Experimental Results with Different α

In this subsection, we test Transformer, Neural ODE and ContiFormer with different α to show the robustness of these models. As shown in Table 7, our model can achieve the best result on different parameters.

C.1.5 More Visualization Results

More visual results with α =0.02 can be found in Figure 4.

Transformer

Latent ODE

ContiFormer

C.2 Irregular Time Series Classification

C.2.1 Training Details

For both our model and the baseline models, we adopted a fixed learning rate of 10 -2 and a batch size of 64 . To ensure the reliability of our results, all experiments were conducted three times, and the reported values represent the mean. The training process for all models lasted for 1000 epochs. However, if the training accuracy failed to show any improvement for 100 consecutive epochs, the training was stopped to avoid unnecessary computation.

C.2.2 Dataset Information

We select 20 datasets from UEA Time Series Classification Archieve with diverse characteristics in terms of the number, dimensionality and length of time series samples, as well as the number of classes. Detailed information is listed in Table 8.

C.2.3 Full Experimental Results

We follow the setting of [30] to randomly drop either 30%, 50% or 70% observations. Experimental results are shown in Table 9, Table 10 and Table 11. We can find that ContiFormer outperforms RNN-based methods, ODE-based methods and Transformer-based methods. Also, we observe that ContiFormer outperforms all the baselines by a large margin when 70% observations are dropped, which confirms that ContiFormer is more suitable for irregular time series modeling.

C.3 Predicting Irregular-Sampled Sequences

C.3.1 Background: Marked Temporal Point Process (MTPP)

The marked Temporal Point Process (MTPP) has been widely studied in recent years. We define a sequence of events and types as Γ = [( t 1 , k 1 ) , ( t 2 , k 2 ) , . . . , ( t N , k N )] , where t i is the time point when the i -th event occurs, with t 1 < t 2 < . . . < t N , k i ∈ { 1 , 2 , . . . , K} is the type of the i -th event and K is the total number of event types. A history event is the event sequence up to but not including

time t , denoted by H t = ( t j , k j ) : t j < t . The goal of the temporal point process is to model the conditional intensity function (CIF), denoted by λ ∗ k ( t ) , which represents the likelihood of an event occurring at time t , given the historical events. Models are trained so that the log probability of an event sequence, i.e.,

C.3.2 Dataset Description

Weuse one synthetic dataset and five real-world datasets, namely Synthetic, Neonate [50], Traffic [32], MIMIC [16], BookOrder [16] and StackOverflow [33] to evaluate our model.

Synthetic To evaluate the effectiveness of the proposed method, we create a synthetic dataset. Specifically, we consider 10 types of event where each type of event contain three properties, i.e., ( X i , V i , D i ) . Hence, given an event e with type j is represented as e = ( e.x, e.v, e.d, e.k ) , where e.x ∼ N ( X j , µ 1 ) , e.v ∼ N ( V j , µ 2 ) , e.d = D j > 0 and e.k is the type of the event. We set µ 1 = µ 2 = 0 . 01 and X,V,D ∈ R 10 is randomly generated and pre-defined.

To model both the correlation between events and the dynamic changes of events within the system. We first define the dynamic attention (weight) for each event, which is composed of inter-event dependency modeling and intra-event dynamic modeling. For inter-event dependency, it aims to calculate time-aware attention (weight) between each pair of events. Specifically, we use the following equation to define the weight between event e and ˆ e ,

Next, the define λ ∗ k ( t ) and λ ∗ ( t ) in which we assume that only the last event attributes to the decision of the next event, following Markov decision process (MDP), i.e.

where exp ( -( t -e.t ) /e.d ) is the exponential decay effect for each event, ˆ e is the last event as of time t . The existence of the exponential decay function ensures that all the events will eventually disappear in the system as time goes to infinity so as to stabilize the conditional intensify function. P ( k | ˆ k ) is a pre-defined matrix that means the probability of the next event to with type k given that the type of the previous event is ˆ e .

Next, we predict the occurrence time of the next event with

To predict the type of the next event, we sample the type of the next event from P ( ·| ˆ e.k ) , where ˆ e.k is the type of the previous event.

Finally, we use 500 different random seeds to generate a total of 500 event sequences. Also, we use 4-fold cross-validation for evaluation. The generation process is shown in Algorithm 1.

Neonate. Neonate dataset is a clinical dataset from 79 patients, among which 48 patients have incurred at least two seizures during the observation. The dataset contains only one type of event, i.e., the start point of a seizure (in minutes). The task is to predict when the next seizure will happen. Note that we only clear the start point of the seizure. Thus, we only have one type of event.

Algorithm 1 Generate Synthetic Dataset with Time-Aware Dynamic Kernel

Input seed ∈ [0 , 500) , tmax = 20 , X = [0 . 78 , 0 . 11 , -1 . 0 , -0 . 56 , -0 . 78 , 1 . 0 , 0 . 56 , -0 . 33 , 0 . 3 , -0 . 11] , V = [ -0 . 17 , 0 . 5 , -0 . 28 , 0 . 28 , 0 . 17 , 0 . 39 , -0 . 05 , 0 . 05 , -0 . 39 , -0 . 5] , D = [0 . 5 , 0 ., 0 . 28 , 0 . 17 , 0 . 22 , 0 . 44 , 0 . 33 , 0 . 11 , 0 . 05 , 0 . 39]

Output A generated event sequence E .

Initialize the random state of Numpy using seed and defined the probabilistic matrix P . Initialize the event sequence E to be an empty array, and set current time T = 0 . Random generate the type of the first event e.k and sample e.x, e.v and e.d . Add the event to the event sequence.

while t ≤ tmax do

Calculate the conditional intensity function using Eq. (47).

Predict the time of the next event e.t using Eq. (48).

Assume the type of previous event is ˆ k , sample the type of the next event e.k from P ( ·| ˆ k ) .

Sample sample e.x, e.v and e.d given e.k .

Add the generated event to the event sequence E .

Update T = e.t .

end while return E

Traffic. Traffic dataset is a benchmark dataset for time-series forecasting collected by the Caltrans Performance Measurement System (PeMS). This dataset contains the traffic state monitoring of 861 sensors in a two-year period. The data is collected every 1 hour. We extract time points with fluctuations higher than 50% as events, whose types are up and down according to the direction. We consider two events that are within 6 hours as duplicated events and remove the latter one from the dataset. Besides, for each sensor, it contains a long sequence, we partition these event sequences by month, i.e., for each event sequence, it contains the event of a sensor in one month.

MIMIC. MIMIC dataset is a clinical dataset that collects patients' visits to a hospital's ICU in a seven-year period. We treat the visits of each patient as a separate sequence.

BookOrder. BookOrder dataset is a financial dataset that contains transaction records of stock in one day. We record the time (in milliseconds) and the action that was taken in each transaction. The dataset is a single long sequence with only two types of events: 'buy' and 'sell'. We clip the event sequences so that the maximum length is 500 in both the train and test sets. Furthermore, the mean µ and standard derivative σ of the time difference between two events in the dataset is 1.32 and 20.24 respectively. Therefore, we clip the maximum time difference to 70.

StackOverflow. StackOverflow dataset is a question-answering website. The website rewards users with badges to promote engagement in the community, and the same badge can be rewarded multiple times to the same user. We collect data in a two-year period, and we treat each user's reward history as a sequence.

For MIMIC, Bookorder, and StackOverflow datasets, we use the dataset from the Google Drive 10 . Statistic information can be found in Table 12. We use the 4-fold cross-validation scheme for Synthetic, Neonate, and Traffic datasets following [17], and the 5-fold cross-validation scheme for the other three datasets following [39, 64].

C.3.3 Implementation Details

Input Embedding Given the history event H t = { ( t j , k j ) : t j < t } with a total of K events, the input of Transformer-based model and RNN-based model is obtained through event embedding and temporal encoding, i.e., for the i -th event x i = emb ( k ) + enc ( t ) , where emb ( k ) is the k -th column of the embedding table E ∈ R K× d and temporal encoding is defined by a combination of sine and cosine function [64].

Conditional Intensity Function Given the output from the model, i.e. h ( t 1 ) , h ( t 2 ) , .., h ( t N ) . The dynamics of the temporal point process are described by a continuous conditional intensity function. To reduce the memory cost and accelerate the training process. Instead of using the continuous-time output from ContiFormer, we only use ContiFormer to generate a hidden representation for discrete time points, i.e., t 1 , t 2 , ..., t j , ..., t N . Following [64], the conditional intensity function is defined by

where each of the type-specific conditional intensity function is defined by

where on interval t ∈ [ t j , t j +1 ) and f k ( x ) = β k log (1 + exp ( x/β k )) is the softplus function with parameter β k . The use of such a closed-form intensity function allows for O (1) interpolation.

Training Loss We train our model and all the baseline models by jointing maximizing the loglikelihood of the sequence and minimizing the prediction losses.

Specifically, for a sequence S over an interval [ t i , t N ] , given the conditional intensity function λ ( t |H t ) , the log-likelihood is given by

One challenge to compute the log-likelihood is to solve the integral, i.e., Λ = ∫ t N t 1 λ ( t |H t ) dt . Since softplus function has no closed-form computation, we apply the Monte Carlo integration method, i.e.,

where u i ∼ Unif( t j -1 , t j ) for i ∈ [1 , ..., P ] is sampled from a uniform distribution over an interval [ t j -1 , t j ] .

Besides the log-likelihood loss, we also use the cross-entropy loss and regression loss for predicting the type and time of the next events.

Specifically, given the conditional intensity function, the next time stamp prediction is given by

Thus, the regression loss and the cross-entropy loss are given by

Overall, the modeling is trained in a multi-task manner, i.e.,

where α 1 and α 2 are the pre-defined weight. By default, we set α 1 = 0 . 01 and α 2 = 1 .

C.3.4 Significance test on Synthetic and BookOrder datasets

To demonstrate the statistical superiority of ContiFormer, we conducted a t-test on Synthetic and BookOrder datasets using 10 random seeds. The t-test results are listed in Table 13.

C.4 Regular Time Series Forecasting

Time series forecasting has garnered considerable attention in recent years, as evidenced by various studies [35, 53, 61]. In this study, we assess ContiFormer against several models tailored for regular time series forecasting.

Datasets We evaluate ContiFormer on five experimental datasets. i) ETT dataset [61] contains the data collected from electricity transformers, including load and oil temperature that are recorded every 15 minutes between July 2016 and July 2018. Here, we use ETTm2 and ETTh2 datasets. ii) Exchange dataset [53] records the daily exchange rates of eight different countries ranging from 1990 to 2016. iii) Weather dataset is recorded every 10 minutes for 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity, etc. v) ILI dataset includes the weekly recorded influenza-like illness (ILI) patients data from Centers for Disease Control and Prevention of the United States between 2002 and 2021.

Baselines We compare ContiFormer with several transformer-based models for regular time series forecasting under the multivariate setting. These transformer-based models includes: FEDformer[62], Autoformer[53], Informer[61] and Reformer[31]. Also, we chose one recent state-of-the-art model. i.e. TimesNet[52] for comparison.

Results of Time Series Forecasting The experimental results are depicted in Table 14. ContiFormer achieves the best performance on ILI dataset. Specifically, it gives 10% MSE reduction (2.874 → 2.632). Also, ContiFormer outperforms Autoformer and FEDformer on Exchange, ETTm2, and Weather datasets. Therefore, it demonstrates that ContiFormer performs competitively with the state-of-the-art model.

C.5 Pendulum Regression Task

The objective of pendulum regression is to assess the models' capability in accommodating observations obtained at irregular intervals [45, 49]. The input sequence comprises 50 images, each

measuring 24 by 24 pixels, and is afflicted by correlated noise. These images are sampled at irregular intervals from a continuous trajectory spanning a duration of 100 units, while the targets correspond to the sine and cosine values of the pendulum's angle. It's important to note that the pendulum follows a nonlinear dynamical system, with the velocity remaining unobserved. Table 15 summarizes the results of this experiment. ContiFormer outperforms CRU and ODE-RNN.

D Ablation Study and Parameter Study

D.1 Effects of Different Attention Mechanism

We study the impact of different attention mechanisms to show the significance of both continuousevolving attention scores and continuous value functions. We claim that both of these two continuous designs attribute to the modeling of complex dynamic systems. We conduct the experiment on 6 datasets for event prediction task. As shown in Table. 16, ContiFormer outperforms both of the two variants, which shows that jointly modeling the dynamic relationship between input observations and meanwhile capturing the dynamic change of the observations is critical for modeling complex event sequences in reality.

D.2 Effects of Different Numeric Approximation Methods

As discussed in Appendix A.2, linear interpolation and Gauss-Legendre Quadrature are two methods for approximating an integral in a close interval [ -1 , 1] . We study the impact of these two methods and set P , i.e., the number of intermediate steps for integral approximation, in the Gauss-Legendre quadrature from { 2 , 3 , 4 , 5 } . We conducted the experiment on 6 datasets for event prediction task.

P

Dataset

Synthetic

Neonate

Traffic

LL (

↑

)

ACC (

↑

)

RMSE (

↓

)

LL (

↑

)

RMSE (

↓

)

LL (

↑

)

ACC (

↑

)

RMSE (

↓

)

Approxiamation Method =

Linear

-

-0.535

0.842

0.192

-2.550

9.233

0.635

0.822

0.328

Approxiamation Method =

Gauss-Legendre Quadrature

2

-0.545

0.842

0.198

-2.571

9.235

0.599

0.817

0.330

3

-0.531

0.842

0.194

-2.594

9.236

0.610

0.820

0.328

4

-0.556

0.842

0.199

-2.590

9.253

0.589

0.819

0.328

5

-0.550

0.842

0.193

-2.602

9.258

0.601

0.820

0.328

P

Dataset

MIMIC

BookOrder

StackOverflow

LL (

↑

)

ACC (

↑

)

RMSE (

↓

)

LL (

↑

)

ACC (

↑

)

RMSE (

↓

)

LL (

↑

)

ACC (

↑

)

RMSE (

↓

)

Approxiamation Method =

Linear

-

-1.135

0.836

0.837

-0.270

0.628

3.614

-2.332

0.473

0.948

Approxiamation Method =

Gauss-Legendre Quadrature

2

-1.144

0.834

0.840

-0.291

0.627

3.653

-2.337

0.472

0.948

3

-1.142

0.837

0.839

-0.294

0.627

3.667

-2.337

0.471

0.948

4

-1.143

0.837

0.839

-0.287

0.627

3.647

-2.340

0.471

0.948

5

-1.132

0.834

0.838

-0.290

0.628

3.563

-2.338

0.471

0.948

The experimental results can be found in Table 17. As shown in the table, our model exhibits low sensitivity to the choice of approximation methods.

D.3 Effects of Tolerance Errors in ODESolver

ODE solvers play a crucial role in ensuring that the model's output remains within a specified tolerance of the true solution [9]. By adjusting this tolerance, we can observe changes in the behavior of the neural network. Given our use of a fixed-step ODE solver, we sought to investigate the impact of the step size on prediction performance throughout the experiment. Specifically, we explored different step sizes from the set { 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 } . A smaller step size corresponds to a greater number of forward passes required to solve the ordinary differential equations, resulting in more accurate solutions at the expense of increased computational time. We conducted extensive experiments on 6 datasets for the event prediction task. The experimental results are presented in Table 18. The table reveals that our model demonstrates low sensitivity to the tolerance error. Therefore, we set the tolerance error to 0 . 1 in order to make a desirable balance between computational time and prediction accuracy. We explain the phenomena in two aspects. First, our framework, similar to the

Transformer architecture, circumvents cumulative errors by eliminating the necessity of sequentially passing neural networks in a regressive manner. Second, since the output from the attention module is actually a weighted sum of the tokens, the total variance is lowered. For instance, assume that X 1 , X 2 , ..., X N ∼ N ( µ, σ 2 ) , then the total variance of the mean value ¯ x = 1 N ( X 1 + X 2 + ... + X N ) is σ 2 N , and the variance is significantly lowered given the large . Overall, our model is not sensitive to tolerance error, which makes our model robust to different scenarios.

D.4 Effects of Vector Fields in ODE

Neural ODEs are controlled by a vector field that defines the change of x over time, i.e.,

where f : R d +1 → R d . We investigate two types of f to implement Eq. (1).

Specifically, we denote Concat as

and ConcatNorm as

where Actfn ( · ) is either tanh or sigmoid activation function, Linear a,b ( · ) : R a → R b : is a linear transformer from dimension a to dimension b , LN denotes the layer normalization. We conduct the experiment on 6 datasets for event prediction task. The experimental results are shown in Table 19. As shown in the table, incorporating normalization before the activation function leads to improved predictive performance.

D.5 Effects of Dropout Rate

Dropout in the Transformer can prevent overfitting and boost performance. However, with the increment of the dropout rate, we output of ContiFormer tends to be less continuous, leading to a sub-optimal solution for tasks that prefer a continuous-time output. To this end, we conduct experiments on both modeling continuous-time function and event prediction tasks. As for the task of modeling continuous-time function, it is desirable to have a dynamic system where outputs are 'almost' continuous and smooth. As shown in Figure 5, the performance of both Transformer and ContiFormer drops significantly as the dropout rate increases. For prediction tasks, the dynamic system can be shaky, which reflects noise or other uncontrolled influences on the system. The experimental results on event prediction can be found in Table 20. As shown in the table, the dropout rate has low sensitivity to the prediction results.

Interpolationresultswithα=0.02

Interpolationresultswithα=0.2

Interpolationresultswithα=2

0.030

0.05

0.04

0.025

0.04

RMSE

E

0.020

RMSE

0.03

0.03

RMS

0.015

0.02

0.02

0.010

Neural ODE

Transformer

0.01

0.01

0.005

ContiFormer

0.0

0.1

DropoutRate

0.3

0.5

0.0

0.1

DropoutRate

0.3

0.5

0.0

0.1

DropoutRate

0.3

0.5

D.6 Effects of Time Normalization

As shown in Eq. (3) and Eq. (5), due to the nature of sequence data, it is common to encounter abnormal points, such as events with significantly large time differences. To avoid numeric instability during training, we divide the integrated solution by the time difference. To examine the effect of normalization, we conduct experiments on 6 datasets for event prediction task. The experimental results can be found in Table 21, which shows that the normalization is significant on most datasets, including Synthetic, Neonate, MIMIC and StackOverflow datasets.

E Time Cost v.s. Memory Cost v.s. Accuracy

Time Costw.r.t.InputLength

MemoryCostw.r.t.InputLength

10

Passes

40000

ODE-RNN (w/ norm)

20

TST

TST

35000

ODE-RNN (w/o norm)

Time Cost per Iteration(

ContiFormer

ContiFormer

S5

Memory Cost (GB)

8

S5

Forward

30000

15

ODE-RNN (w/ norm)

ODE-RNN (w/ norm)

25000

ODE-RNN (w/o norm)

6

ODE-RNN (w/o norm)

20000

10

15000

4

Number

10000

5

5000

2

0

0

101

102

103

101

102

103

0

101

102

103

Input Length

Input Length

Input Length

Utilizing continuous-time modeling in ContiFormer often results in substantial time and GPU memory overhead. We have conducted a comparative analysis of ContiFormer's actual time costs in comparison to Transformer-based models (TST), ODE-based models (ODE-RNN, Neural CDE), and SSM-based models (S5). Throughout our analysis, we employ an RK4 solver for ODE-based models and our model, setting the step size to 0 . 1 and 0 . 01 . It's worth noting that, for ODE-RNN and other

Table 21: Effects of normalization on event prediction task. Attn. stands for Attention. " refers to normalizing the attention/value as shown in Eq. (3)/Eq. (5), while % refers to removing the normalization, i.e. t -t i . Note that ContiFormer use both normalization for attention and values.

ODE-based models, the input times have an impact on the number of forward passes when using a fixed-step ODE solver, thus, affecting the time costs.

The left figure in Figure 6 illustrates the influence of input times concerning input length. For the model with normalization (a.k.a., w/ norm), the input times consistently range from [0 , 1] , regardless of the input length. Conversely, for the model without normalization (a.k.a., w/o norm), the input times span from [0 , L ] , where L represents the input length.

As depicted in the middle figure in Figure 6, our model performs comparably to ODE-RNN (w/ norm) when the input length is up to 100 , but becomes approximately four times slower as the input length extends to 1000 . The right figure in Figure 6 displays GPU memory usage, revealing a significantly higher memory cost associated with our model.

Table 22 provides a more comprehensive overview of different models on UEA datasets. It is evident that our model strikes a better balance between accuracy and time cost. Specifically, our model achieves the best performance on 3 out of 6 selected datasets while incurring only twice the time cost compared to S5.

F Broader Impact

The proposed model, ContiFormer, presents several potential impacts within the field of machine learning and irregular time series analysis. These include:

Enhanced Modeling Capability: ContiFormer extends the Transformer architecture to effectively capture complex continuous-time dynamic systems. By leveraging continuous-time modeling techniques, the model demonstrates improved performance in handling irregular time series data, offering a more flexible and powerful modeling ability.

Practical Applications: Our proposed method achieves state-of-the-art performance in a variety of tasks and real-life datasets, which makes it more promising to tackle real-world applications. The ability to model continuous-time dynamics offers valuable insights into time-evolving systems, facilitating improved decision-making and predictive capabilities in diverse domains.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method, a continuous-time transformer, ContiFormer, for irregular time series modelling. The proposed method extends vanilla Transformer to a continuous domain. In particular, the model incorporates the continuous dynamic modelling of Neural ODE with the attention mechanism of a transformer. Moreover, the authors show that many transformer variants designed for irregular time series modelling are a special case of the proposed ContiFormer. The authors evaluated the method on a continuous time function (spiral), irregular sampled time series for classification and irregular time series for event prediction. The results indicate that the proposed method obtains better results than the baseline models.\n\nSTRENGTHS:\nOriginality: \nThe proposed work is original. The author's propose a novel continuous-time attention mechanism (CT-MHA). \n\nQuality: \nThe authors have included also a comparison of complexity analysis of their method compared to other. The authors have tested the proposed method on multitude of datasets: irregularly-sampled time series for classification with increasing number of data dropped, as well as event prediction. Especially for the latter, the authors compare their method to existing state-of-the-art models for event forecasting (SAHP, THP) and obtain an improved performance. \n\nClarity: \nPlease see weaknesses. \n\nSignificance:\nThe authors provide a unique theoretical approach, and the obtained results demonstrate a benefit of the proposed method.\n\nWEAKNESSES:\nOriginality, related work: \nThe related work is a bit convoluted. It would be better if the authors divide the related work into 3 subsections: Transformers for time-series modelling, Transformer for irregular time series, Continuous time models (NODEs). The literature of RNN is not as relevant to this work as it builds upon transformers and NODEs. \n\nWith respect to NODE related work:\nLine 43-45, the authors say 'the recursive nature of Neural ODEs can lead to cumulative errors if the number of iterations is numerous'. This is not precise. The cumulative error is dependent of the solver used, and there are numerous works which by enforcing, for example, conservation of energy, keep the error tolerance low, for example, https://arxiv.org/abs/1909.12077. Moreover, by selecting difference step size, solver type (adaptive or fixed) one can control/adjust for the numerical error (as mentioned in https://arxiv.org/abs/1806.07366). Please rephrase. \n\nWith respect to Transformer related work:\nIn line 47-48, the authors mention that due the fixed time encoding or learning upon certain kernel functions the resulting models struggle to model complex data patterns in real-world practice.  However, I do not see how this claim is supported. Even in the appendix, for given percentages of observations dropped (50%) mTAN model outperforms ContiFormer. Therefore, the claim is not supported. Please rephrase.\n\nThe proposed work differs from the existing work in the field, however,  some of the claims made about the previous work are not well supported in the current version of the paper. \n\nQuality: \nThe proposed continuous time attention mechanism, eq. 3, is unclear, as the underlying functions are not clearly defined. The authors' mention that previous transformer works for irregular time series are special cases of their model, however, the proof of this only mentioned in the Appendix, I would recommend moving the core parts of the proof to the main part of the paper. \n\nLines 281 - 283, the authors say that ContiFormer consistently outperforms all the baselines on all three settings, while in the appendix it can be seen that there are also instances were the baseline models outperform ContiFormer, therefore, I would soften the claim. \n\nFor lines 303-304, I am missing a comparison to methods that are more aligned with the task at hand. The authors compare against TST and mTAN, however TST  model is focused on multivariate time series (not decomposition of different dynamic trends), while mTAN although also designed for irregular time series modelling the paper's focus is on temporally distributed latent representations. For extracting temporal dynamics/trends of the data, there are other works, Autoformer (https://arxiv.org/abs/2106.13008), FEDformer (https://arxiv.org/abs/2201.12740) that would be perhaps a more fair comparison to the present method.  \n\nClarity: \nThe submission is not clearly written, thus negatively affecting the understanding of the proposed method. Please clarify for all functions, v(), k(), q() the input, output dimensionalities, as well what these functions are. As this is unclear, it is hard to evaluate what exactly the inner product of the two function spaces is computing. \n\n\nLine 147. Imprecise phrasing: the authors mention that self-attention involves calculating the correlation between queries and keys, which is incorrect. The computation represents an inner dot product, which is not a correlation metric but rather a similarity measure, please rephrase.\n\nThe main figure (figure 1), is also not immediately clear for the reader. Would suggest to adjust the figure.  It is hard to follow all the colors and how they correspond to which input. \n\nIn addition, for the NODE model compared with, as well as for the ODESolver used in author own work it is missing what kind of solver is used: adaptive or fixed step? This could greatly affect the performance, hence, it is a crucial implementation detail. \n\nIn the appendix mTAN model has the wrong reference. The reference is for the Autoformer model, please correct. \n\nLastly, for the experiments is it not clear what is the input/ouput data length for the model.\n\nOverall, the technical details of the paper are not clear. As this is the main contribution of the paper, it is essential that it is clear so that the extensive experimental results can be objectively evaluated given the method at hand. I would suggest the authors to rewrite this section.\n\nQUESTIONS:\nIn line 83, the authors say that previous models 'are insufficient in capturing input-dependent dynamic systems'. How is this claim supported? The previous work also is also condition on the input data. \n\nIn line 99-100, the author's say 'ContiFormer, employs a non-autoregressive paradigm', however, the vanilla transformer decoder at inference time is an auto-regressive model, can you clarify this? Furthermore, to perform the ODESolver step, you perform auto-regressive P steps? \n\nEq. (1), the formulation is unclear and incorrect: the author's define both the ODE as well as the continuous latent variable with the same parameter. Furthermore, if k_i(\\tau) is the ODE, what is f()?\n\nIn line 136, could you please clarify why you have modeled each observation as a separate ODE? As showed in https://arxiv.org/abs/1806.07366 NODE models can model irregular time-series with a single ODE specification, even more so, close by observations most likely follow the same ODE. \n\nIn line 136, could you clarify if the ODE is a MLP or any other architecture?\n\nLine 138: would suggest the authors to adjust this assumption/claim: from a dynamical systems perspective a future event cannot affect a past event. Vast majority of works modelling dynamics are often based on Markov property. \n\nline 144 in text you mention that q(t_i) is a cubic spline approximation, however, in the appendix you clarify that linear interpolation is used due to the lower computational complexity. please clarify the approximation used in your proposed method. What is the dimensionality of Q_i? \n\n\nEq. (4) Based on the introduced notation it seems like the introduced computation ends up being a matrix (Q_i), vector (k_i() \\in R^d) product. Where the Q_i is an approximation of the process via cubic spline, and k_i() is the latent trajectory by an ODE, this would imply that the proposed mechanism measure the similarity of the learned approximation of the ODE to the approximation by the cubic spline function. Is this correct?\n\nEq. (7) For multi-head attention, you also apply a linear transformation to the latent continuous state, q(),v()? \n\nHow sensitive is the method of using a different approximation method instead of cubic spline? \n\nDoes each latent continuous trajectory span only two adjacent time points t:[-1,1]? Meaning that depending input length L (data points), you would have L independent latent ODE trajectories? It is unclear from the text. \n\nThe authors mention that their attention mechanism is measuring the correlation between different observations (line 298), therefore, I would suggest the authors to compare their work to Autoformer (that measures correlation between different time series) or Fourier-based (FEDformer) attention mechanism. Have such comparisons been made?\n\nLIMITATIONS:\nThe authors have not explicitly addressed the limitation of the presented work. Based on the text, the proposed method models each observation with a separate ODE function. This, however, seems to be sub-optimal as data points close in time probably follow the same underlying dynamics function, furthermore, the authors have not explained why such a design choice would be necessary, as by default Neural ODEs can account for irregularly sampled time series. Moreover, it would be nice if the authors would have addressed (as an ablation) the model's performance on regular time-series data compared to existing state-of-the art methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the ContiFormer, a novel continuous-time Transformer model designed for handling irregular time series data. In this model, the keys, queries and values are vector-valued functions indexed on time. Each input observation gives rise to a key function, given by the solution of a Neural ODE started at an initial condition determined by the observation. While a similar model is adopted for the value functions, the query function is obtained using natural cubic spline interpolation of discrete query values. By replacing dot products (between discrete queries and keys) with an $L^2$ inner product, a continuous-time attention function is derived. The continuous-time output can be evaluated on a discrete time grid allowing for the stacking of such layers. To reduce the computational time, the paper leverages a reparametrization trick, which makes it possible to solve simultaneously multiple Neural ODEs on different time intervals. The ContiFormer model is evaluated on various time series tasks and datasets demonstrating its effectiveness in handling irregular time-series data.\n\nSTRENGTHS:\n- Given the empirical success of transformer models on machine learning tasks involving sequential data, enhancing their capability of effectively handling irregular time series data is an important research question.  \n- This paper proposes an elegant solution based on Neural ODEs and other ideas from the field of neural differential equations such as the use of cubic splines in Neural CDEs.\n- The paper is clearly written and easy to follow. \n- The empirical evaluation is well conducted, extensive and provides a compelling demonstration of the superior performance of the ContiFormer model.\n\nWEAKNESSES:\n- The theoretical results are deferred to the appendix. I think the main theorem (Thm 1 in Appendix) should at least be formally stated and further commented on in the main paper. This would better motivate and support the modelling choices. \n\t\n- The paper is a bit repetitive at times (l100-102, l108-117, l175-178) while some components might deserve further explanation: 1. the main theoretical result;  2. although the $i^{th}$ key and value functions are well defined at $t<t_i$, it could be further commented by expanding l138, and perhaps adapting Fig. 1 with $t<t_i$; 3. the rationale for modelling the key and value functions differently from the query function. \n\t\n- As mentioned in the related work section, a continuous-time attention model based on Neural ODEs has previously been proposed in [10]. While several baselines are considered in the experimental section it is surprising that this model is not included without a justification. \n\n- The attention model is not fully formulated in continuous-time, in the sense that it is expressed as a sum over the number of value functions which depends on the number of observations. A natural thought of experiment for a continuous-time model is to replace the input sequences with continuous paths. Is there a way to define the ContiFormer on such paths?\n\nQUESTIONS:\n- Why did you choose to model the key and value functions differently from the query function? Can the ContiFormer handle partially observed data, given how the key and value functions are defined? \n- Do you have an intuition/explanation for the fact that $P$ can be chosen to be very small (l 224) and the insensitivity of the ContiFormer to the tolerance error (Appendix F12, F13)? \n- Related to the comment above about the continuous-time formulation, does the ContiFormer offer any advantage for handling both irregular and long time series?\n- Although computational complexities are provided, would it be possible to report the actual time it takes to train and evaluate the model?\n\nLIMITATIONS:\nYes, some limitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a continuous time extention of the transformer architecture using ODE blocks to propagate the effect of each observation individually through time. For computing attention values inner products between functions are used, where in the implementation the resulting integral is approximated. The paper demonstrates the method's application to irregular time-series data and in temporal point processes in different tasks including interpolation, extrapolation, classificaton.\n\nSTRENGTHS:\nThe paper gives a continuous-time extension of transformers capable of handling irregularly sampled continuous time series, and point processes.\nComputational complexity is assessed in details which is quite important for practical application, and helps the reader to better estimate the real life cost of scaling the methods to real life problems.\n\nWEAKNESSES:\nMy main problem is with the missing statistical rigor during evaluation. More specifically:\n\nIn Table 3 the authors report averages over 20 datasets. I assume these datasets have different sizes, complexity, dataset imbalance etc., therefore it is hard to rigorously evaluate the average metrics.There are dataset-by-dataset level results in the appendix but there is still no standard deviation.\n\nSimilarly in Table 4, no standard deviations. Probably 3 std differences are significant, but with 3 repeats I am not sure sure 2 std is very convincing. (For just a very rough estimate how big delta is needed check a 3 sample unpaired t-test table)\n\nQUESTIONS:\nPlease give standard deviations and mark significant differences.\n\nAlso it is interesting to see that NeuralCD is outperformed by ODE-RNN here for the 30 and 50% dropped case. As there is result provided for CharacterTrajectories in the appendix (this dataset was used in the original NeuralCDE paper) I compared the results.\nformat:  present paper, vs, Kidger et al.\n\n| Method | 30% dropped | 50% dropped | 70% dropped |\n|-----------|-----------|-----------|-----------|\nGRU-D |  0.9325 vs. 0.942 | 0.8872 vs. 0.902 | **0.8433 vs. 0.919** |\nGRU-deltaT | **0.8558 vs. 0.936** |  0.9088 vs. 0.913 | **0.8496 vs. 0.904** |\nN CDE |  **0.9276 vs. 0.987** | **0.9285 vs. 0.988** | **0.9241 vs. 0.986** |\nODERNN | 0.9415 vs. 0.954 | 0.9506 vs. 9.960 | 0.9471 vs. 0.953 |\n\nThese are serious differences on the same dataset. I can maybe explain away the case with Neural CDE as the creator of the method may be able to tune it better (this is in fact the main drawback of this entire comparison table idea we do, but this is a problem we are not going to solve here :) But results of several other baselines are quite different. This is concerning. \n\n\"We generated a dataset of 300 2-dimensional spirals, sampled at 150 equally-spaced time points\" <-- what is the temporal dimension? is this CD spiral a parametric curve where time is the parameter?\n\nAre you using vanilla NeuralODE in this experiment? I find it hard to understand why you get the interpolation behavior from NODEs as it is visible in Figure 2. The interpolation looks like piece-wise linear. Can you provide interpolation results with ODE-RNNs and neural CDEs?\n\nDo you see a way to avoid resampling between layers?\n\nIt would be useful and quite interesting to see some discussion about the apparent contradiction in using transformers (long range interactions) and ODEs. ODEs operates with derivatives and their solutions are locally described. The assumptions of a smooth differentiable solution seems to stand in contradiction with long range interactions unmediated by values in between, or in a case of a latent-ODE decriptions by latent system states. Can you formalize the properties of systems (e.g. the mentioned stock market time series) where you expect this type of modelling give the most benefit?\n\n\nMinor:\n\n\"To overcome the discrete nature of RNNs, a different strategy involves the exponential decay of\nthe hidden state between observations, governed by modifiable decay parameters [5, 9, 37].\"  [9] is the reference for the Neural ODE paper from Chen at al., it does not apply exponential decay.\n\ntypo:  denoted vs donated\n\nPlease stress the difference between irregularly sampled data and temporal point processes eg. in [37] more clearly (missing data vs. events).\n\nNOTE: the authors clarified some points, provided new results, and we identified a misunderstanding about the Neural ODE figure. The authors used Latent-ODE, but cited a previous paper in the table: that lead me believe that they fitted a Vanilla NODE to the trajectory. I raised my score by one.\n\nLIMITATIONS:\nNo systematic discussion on limitations provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new deep learning model called ContiFormer to model continuous-time dynamics on irregular time series data. The paper argues that existing methods such as recurrent neural networks (RNNs), Neural Ordinary Differential Equations (ODEs), and Transformers have limitations in modeling continuous-time data and fail to capture intricate correlations within these sequences. The proposed ContiFormer model extends the relation modeling of the Transformer model to the continuous domain and incorporates the modeling abilities of continuous dynamics of Neural ODE with the attention mechanism of Transformers. Both numerical and theoretical justifications are provided.\n\nDifferent from the standard Transformer model, the proposed model is parameterized on the first-order derivative of $q$, $k$, and $v$. The real $q$, $k$ used in the attention part is approximately computed with finite sum (e.g., via Runge-Kutta 4th (RK4) order ODE solver). By restricting the function form/class of the first-order derivative on some smooth enough kernel, the proposed model could yield better interpolation/extrapolation ability.\n\nI have two concerns about the proposed model. \n\nThe first one is the usage of ODE solvers may introduce computation overhead. In Table 1, the order of computation cost is already $O(N^2 \\cdot P \\cdot d^2)$, where $P$ is the iterative number in RK4 solver, and in my understanding, $P$ should be 4. In L580 at Appendix B. the RK4 ODE may require 80 forward passes to finish one of the RK4 round. It implies the order of computation cost for a one layer attention model should be $O(N^2\\cdot 80\\cdot d^2)$, which can be very computationally expensive even for models with moderate size. Moreover, if the proposed model has more than one layer (e.g., 2 layers), in order to finish the second layer's attention computation, we will need first to finish the computation of the first layer with $O(N^2\\cdot 80\\cdot d^2)$ cost. After that, every second layer's computation will still involve the forward pass of the first layer, which becomes $O(2\\cdot N^2\\cdot 80\\cdot d^2)$. Therefore, for the $M$ layer attention model, the total computation cost could be $O(M^2\\cdot N^2\\cdot 80\\cdot d^2)$. \n\nThe second one the underlying construction is kind of similar to the Rotary Position Embedding in [1]. In this paper, the kernel trick is used to facilitate the computation (e.g.,  Eq. (27)- Eq. (32) in Appendix C). Based on the current presentation, the kernel function is only time point $t$ dependent but independent on the detailed $q,k$. Given the choice of the kernel function in Eq. (30), the effects of introducing this kernel function look like just adding a rotary positional embedding. \n\nAt present, the authors have not provided sufficient evidence to demonstrate the contribution of their model, which may not be compelling enough for top machine learning conferences like NeurIPS. Despite this, the reviewer is willing to reconsider the decision after the authors' rebuttal. Overall, the paper presents an interesting approach for modeling continuous-time dynamics, but the issues raised above need to be addressed before its acceptance for a top-tier conference.\n\n\nReference: [1] Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.\n\nSTRENGTHS:\nThe paper proposes a new deep transformer model incoperating the continuous-time dynamics on irregular time series data. Both numerical and theoretical justifications are provided.\n\nWEAKNESSES:\nPlease see my comments in the Summary section.\n\nQUESTIONS:\nPlease see my comments in the Summary section.\n\nLIMITATIONS:\nThe authors include a section on the limitations of the proposed work in Appendix G at Page 30.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ContiFormer, a continuous time transformer-based model that leverages parallelism and can handle irregularly sampled data well, thereby removing the need to transform these datasets into discrete uniform bins. This set-up incorporates the continuous dependence on the data from differential equation based neural networks. Finally, an extensive range of experiments on irregularly sampled data are performed to show promising results for ContiFormer.\n\nSTRENGTHS:\nThis paper presents an original architecture class that can encompass many existing models. The discussion on attention combined with the continuous time set-up is limited and therefore this paper will be a significant contribution in this area. The paper is generally well-written and extensive numerical experiments were performed.\n\nWEAKNESSES:\nWhilst in general a fairly well-written paper, key parts of the model can be made clearer. For example, it is not clear for line 137 how the key and values are initialised. There is also an assumption that is not explained \"we assume that every observation has influence on the dynamic system even before its occurrence.\" Combined with the fact that the query uses natural cubic spline, does that mean the model is not causal and cannot be applied in an online fashion (see for example [Morrill et al 2022 On the Choice of Interpolation Scheme for Neural CDEs])?\n\nThis paper does not appear to be comparing with the state-of-the-art models. To my knowledge, the S5 model (Simplified State Space layers for sequence modeling by Smith et al 2022), appears to already outperform quite a few of the baselines chosen in this paper, specifically, mTAN, ODE-RNN, GRU-$\\Delta$t for irregularly sampled pendulum exercise. Therefore this should be added into a benchmark. The pendulum regression task used in Smith et al. seems to be used in quite a few irregular sampling task papers, and therefore it would be of interest to see how the ContiFormer performs in this case.\n\nWhilst there is an extensive range of experiments, the full results with the standard deviation across the 3 repeats for Section 4.2 and Section 4.3 seem to be missing. These seem quite key to back up some results, particularly for Table 4, where the standard deviation is referred to. Without this, it is difficult to quantify \"overall statistical superiority of ContiFormer\" as claimed.\n\nThere does not appear to be any evaluation on regularly sampled datasets. Whilst I appreciate that the marketed strength of this method is in irregularly sampled data, it would be useful to see what the performance is on regularly sampled datasets (against benchmarks) and whether one can just choose this model to be applied on all timeseries.\n\nQUESTIONS:\nSince ContiFormer requires $N^2$ ODEs to be solved, how does the $P$ relate to $L$ in Table 2? Are these on similar orders?\n\nIt seems like the neural ODE performs much worse than expected even for the interpolation compared with the example we see on the diffrax website https://docs.kidger.site/diffrax/examples/neural_ode/. Is this because of the regular vs irregular sampling issue?\n\nFigure 4 in appendix does not seem to be plotting the same curve at each row? Neural ODE clearly is on a different scale or a different curve in the third row quite clearly?\n\nMinor point: Line 119: donated -> denoted\n\nLIMITATIONS:\nLimitations were not discussed in the paper.\n\nGiven that the model makes use of cubic splines, the literature suggests this is will not be causal. Performance on regularly sampled dataset not clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}