ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling
Authors: Yuqi Chen
Kan Ren
Yansen Wang
Yuchen Fang
Weiwei Sun
Dongsheng Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Irregular Time Series Modeling
Transformer
Neural Ordinary Differential Equation
TL;DR: ContiFormer combines Neural ODEs and Transformers for better modeling of continuous-time dynamics on irregular time series.

Abstract: Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.

ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling
Yuqi 22 Kan Ren Yansen Wang', Yuchen Fang* Weiwei Sun', Dongsheng Li² School of Computer Science & Shanghai Key Laboratory of Data Science, Fudan University Microsoft Research Asia," Shanghai Jiao Tong University
Abstract
Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations generalizing to continuous time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is fttps:://sqmml.ithhee fottoniooo//ot
1 Introduction
Irregular time series are prevalent in real-world applications like disease prevention, financial decision- making, and earthquake prediction 27 (44. Their distinctive properties set them apart from regular time series data. First, irregular time series data are characterized by irregularly generated or non-uniformly sampled observations with variable time intervals, as well as missing data due to technical issues, data quality problems, which pose challenges for traditional time series analysis techniques 66, (55) Second, even though the observations are irregularly sampled, the underlying data-generating process is assumed to be continuous (30 42 43). Third, the relationships among the observations can be intricate and continuously evolving. All these characteristics require elaborate modeling approaches for better understanding these data and making accurate predictions.
The continuity and intricate dependency of these data samples pose significant challenges for model design. Simply dividing the timeline into equally sized intervals can severely damage the continuity of the data [36). Recent works have suggested that underlying continuous-time process is appreciated for irregular time series modeling [ 30 39 43), which requires the modeling procedure to capture
'The work conducted during the internship Yuqi Chen and Yuchen Fang Microsoft Research. Corresponnence to Kan Ren, contact: veen@oullok.com.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the continuous dynamic of the system. Furthermore, due to the continuous nature of data flow, we argue that the correlation within the observed data is also constantly changing over time. For instance. stock prices of tech giants (e.g., MSFT and GOOG) show consistent evolving trends, yet they are affected by short-term events (e.g., large language model releases) and long-term factors.
To tackle these challenges, researchers have pursued two main branches of solutions. Neural ordinary differential equations (Neural ODEs) and state space models (SSMs) have illustrated promising
abilities for capturing the dynamic change of the system over time 9 28 49]. However, these methods overlook the intricate relationship between observations and their recursive nature can lead to cumulative errors if the number of iterations is numerous [9]. Another line of research capitalizes on the powerful inductive bias of neural networks, such as various recurrent neural networks [6] 39 45
and Transformer models [38 48 [34, [60] However, their use of fixed-time encoding or learning upon certain kernel functions fails to capture the complicated input-dependent dynamic systems.
Based on the above analysis, modeling the relationship between observations while capturing the temporal dynamics is a challenging task. To address this problem, we propose a Continuous- Transformer, namely ContiFormer, which incorporates the modeling abilities of continuous dynamics of Neural ODEs within the attention mechanism of Transformers and breaks the discrete nature of Transformer models. Specifically, to capture the dynamics of the observations, ContiFormer begins by defining latent trajectories for each observation in the given irregularly sampled data points. Next, to capture the intricate yet continuously evolving relationship, it extends the discrete dot-product in Transformers to a continuous- timm domain, where attention calculated between continuous dynamics. With the proposed attention mechanism and Transformer architecture, ContiFormer effectively models complex continuous- time dynamic systems.
The contributions of this paper can be summarized below.
Continuous- Time Transformer: To the best of our knowledge, we are the first to incorporate a continuous time mechanism into attention calculation in Transformer, which is novel and captures the continuity of the underlying system of the irregularly sampled time-series data.
Parallelism Modeling. To tackle the conflicts between continuous timm calculation and the parallel calculation property of the Transformer model, we propose novel reparameterization method, allowing us to parallelly execute the continuous-tiin time attention in the different time ranges.
Theoretical Analysis. We mathematically characterize that various Transformer variants [34 48, [54 [60] can be viewed as special instances of ContiFormer. Thus, our approach offers a broader scope that encompasses Transformer variants.
Experiment Results. We examine our method various irregular time series settings, including time-series interpolation, classification, and prediction. The extensive experimental results have illustrated the superior performance of our method against strong baseline models.
2
Related Work
Time-Diseretized Models. There exists branch of models based the time-discrete assumption which transforms the time-space into a discrete one. Recurrent Neural Networks (RNN) [4, 24 and
Transformers [51] are powerful time-discrete sequence models, which have achieved great success in
natural language processing
[55 37 58), computer vision [20 (3] and time series forecasting [31 35
61 62). Utilizing them for irregular time series data necessitates discretizing the timeline into time bins or padding missing values, potentially resulting in information loss and disregarding inter-observation dynamics. [13 36 41). An alternative is construct models that can utilize these
52
53
56
various time intervals [34] 60 Additionally, several recent approaches have also encoded time
information into features model irregularly sampled time series, including time representation approaches [54] 60] and kernel-basee based methods [38]. which allow learning for time dynamics. For instance, Mercer (54] concatenates the event embedding with the time representation. Despite these advances, these models are still insufficient in capturing input-dependent dynamic systems.
Continuous- Time Models. Other works shift to the continuous-time modeling paradigm. To overcome the discrete nature of RNNs, a different strategy involves the exponential decay of the hidden state between observations (.39) Nevertheless, the applicability of these techniques restricted to decaying dynamics specified between successive observations [9]. Neural ODEs are class of

---

models that leverage the theory of ordinary differential equations to define the dynamics of neural networks 9 277. Neural CDE 330 and Neural RDE 400 further utilize the well-defined mathematics
of controlled differential equations 2 29
and rough path theory [5] to construct continuous-tima
RNN. ODE-RNN 43 combines Neural ODE with Gated Recurrent Unit (GRU) 12 to model
continuous- time dynamics and input changes. Whereas these approaches usually fail to capture the evolving relationship between observations [45]. CADN [T] and ANCDE [26] extend ODE-RNN and Neural CDE by incorporating an attention mechanism to capture the temporal relationships, respectively. Another line of research explores the use of state space models (SSMs) to represent continuous-ting dynamic systems [I 19 49]. However, these models adopt recursive schema, which hampers its efficiency and can lead to cumulative errors [9]. In contrast, our proposed model, ContiFormer leverages the parallelism advantages of the Transformer architecture and employs non-autoregressiiee paradigm. Furthermore, extending the Transformer to the continuous-time domain remains largely unexplored by researchers. Such extension provides more flexible and powerful modeling ability for irregular time series data.
3 Method
The ContiFormer is a Transformer (51) model for processing time series data with irregular time intervals. Formally, an irregular time series is defined as I' = [[X1, fi......... (XN, tx )]. where obser- vations may occur at any time and the observation time points = (f1,....ff) .Nn) are with irregular intervals. X; is the input feature for the i-th observation. We denote X = (X1; X2; Xx] E Vx xd
As aforementioned, is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-timg system. To tackle the challenge, we propose a Continuous- Time Transformer architecture as shown in Figure | Each layer of ContiFormer takes as input an irregular time series X and the sampled time and outputs latent continuous trajectory that captures the dynamic change of the underlying system. With such a design, ContiFormer transforms the discrete observation sequence into the continuous-time domain. At each layer, the core attention module takes a continuous perspective and expands the dot- dot-product operation in the vanilla Transformer to the continuous- time domain, which not only models the underlying continuous dynamics but also captures the evolving input-dependent process. For better understanding, the description of each layer will omit the layer index, without causing confusion.
Throughout the remaining section, we adopt the notation w to denote a sequence of (reference) time points. while t is the random variable representing a query time point. Matrices are donated using uppercase letters, and lowercase letters except t represent continuous functions.
3.1
Continuous- Time Attention Mechanism
The core of the ContiFormer layer is the proposed continuous tittinuuuu--tmeeeeee multi-heat attention (CT-MHA) module, as shown in Figure T each layer, we first transform the input irregular time series X into Q = [Q:: Q2; QN) for queries, K = [K K KN) for keys, and V [V1; V2; :v]] for values. a high level, the CT-MHA module transforms the irregular time series inputs to latent trajectories and outputs continuous dynamic system that captures the time-evolving relationship between observations. To accomplish this, it utilizes ordinary differential equations to define the latent trajectories for each observation. Within the latent state, it assumes that the underlying dynamics evolve following linear ODEs. Subsequently, it constructs a continuous query function by approximating the underlying sample process of the input. Ultimately, it produces continuous-tima function that captures the evolving relationship and represents the complex dynamic system.
Continuous Dynamics from Observations Transformer calculates scaled dot-product attention for queries, keys, and values [51]. In continuous form, we first employ ordinary differential equations to define the latent trajectories for each observation. Specifically, assuming that the first observation and last observation come at time point f and ty respectively, we define continuous keys and values as
kt:t:) Ki, kt(t) = k,(t:)+ + '"' f (T.K((T):00) dr, v,,(,) Vv, v,(() = v;,++ + f' (T.v(T)):)))) dr,
(I)
3

---

Add Norm
z(t)
Feed Forward
value
Add Norm
K
key
w =fti 1/
(reference points)
Continuous- Time MHA
H
query
QA K
X= X X._1
time
Qn 1
(Q.t.)
(Q,t,)
@ w
Continuous-Time Huadi-Head Attention (CT-MHA)
(1
Notations in CT-MHA module (Traj. stands for Trajectory)
~ Highlighted Traj.
Traj. for X1
A Traj. X2 ~ Traj. x;
~ Traj.
Output Traj. Z
Eq. (3)
Eq. >
(6)
Eq.
Scaled Dot Product
Multiply Value
Aggregation
Output
Figure I: Architecture of the ContiFormer layer. ContiFormer takes an irregular time series and its
corresponding sampled time points input. Queries, keys, and values are obtained in continuous- time form. The attention mechanism (CT-MHA) performs a scaled inner product in continuous-time manner to capture the evolving relationship between observations, resulting in complex continuous dynamic system. Feedforward and layer normalization are adopted, similar to the Transformer. Finally, a sampling trick is employed to make ContiFormer stackable. Note that the highlighted trajectories in purple indicate the part of functions that are involved in the calculation of the output.
where E [tl, k,).). IRd' represent the ordinary differential equation for the i-th observa- tion with parameters 0₂ and 0.., and the initial state of (t; and (t((t) respectively and the function f()) E Rd+1 + I&d controls the change of the dynamics.
Query Function While keys and values are associated with the input and output of the attention mechanism, a query specifies what information to extract from the input. model a dynamic system, queries can be modeled dynamic system at specific time point f, i.e., qt)). Specifically, we adopt a common assumption that irregular time series is discretization of an underlying continuous-timg process. Thus, similar to [30]. we define a closed-form continuous-tin time interpolation function (e.g.. natural cubic spline) with knots at ty such that (t,)) Q; as approximation of the underlying process.
function of time that represents the overall changes in the input to the
Scaled Dot Product The self-attention mechanism the key component in Transformer architec- ture. At its core, self-attention involves calculating the correlation between queries and keys. This achieved through the inner product of two matrices in discrete form, i.e., Q KT_ Extending the discrete inner-product to its continuous-time domain, given two real functions f(x) and g(x), we define the inner product of two functions in a closed interval (a, bj as
(f.)) 1." f(x) g(())dz
Intuitively, can be thought of as way of quantifying how much the two functions "align" with each other over the interval. Inspired by the formulation of the inner product in the continuous-time domain, we model the evolving relationship between the i-th sample and the dynamic system at time point as the inner product of and k; in closed interval [ta, f). i...
(t) Jr. q(T) dd
t-t,
Due to the nature of sequence data, common to encounter abnormal points, such as events with significantly large time differences. To avoid numeric instability during training, we divide the

---

integrated solution by the time difference. As consequence, Eq. 3 To ensure the continuity of the function (-).)) we define (t;(t;) as
exhibits discontinuity at Ct2 (t,).
(4,4 = lim St. Slite q(T) ((T)) dr
= q(4) k(4)
Expected Values Given a query time t E [t,,t], the value of an observation at time point & is defined as the expected value from 4 to I. Without loss of generality, it holds that the expected values of an observation sampled at time 4 is v2(44) or V. Formally, the expected value is defined as
Sr. vi(T)dr t-t
(t) (aa))))) [v)(t)
(5)
Multi-Head Attention The continuous-tiin time attention mechanism is a powerful tool used in ma- chine learning that allows for the modeling of complex, time-varying relationships between keys, queries, and values. Unlike traditional attention mechanisms that operate on discrete time steps, continuous-time attention allows for continuous function of time. Specifically, given the forchead-defined queries, keys, and values in continuous-time space, the continuous-tie attention given a query time can be formally defined as
fine-grained analysis of data by modeling the input
CT-ATTN(Q, K, (()) =
Σ (t( v.(1),
(6)
exp (t./t)/d3
where ā,(()
j==1 exp (a, (t)/va)))
Multi-head attention, extension of the attention mechanism [51) allows simultaneous focus different input aspects. It stabilizes training by reducing attention weight variance. We extend Eq. (6)
by incorporating multiple sets of attention weights, i.c.,
CT-MHA K, V. (t) Concat (head (t)(), .... (n) wo
where headaa(() CT-ATTN (QW KW. VWV (b) (b) (b) (h)s w) (t),
where W0, W-Q W.K and Wl) are parameter matrices, [1, H] and H is the head number.
(h)
3.2
Continuous- Time Transformer
Despite the widespread adoption of Transformers [1 in various research fields, their extension for modeling continuous-timg dynamic systems underexplored. We propose ContiFormer that directly builds upon the original implementation of vanilla Transformer while extending it the continuous-time domain. Specifically, we apply layer normalization (LN) after both the multi-head self-attention (MHA) and the layer below
blocks (FFN). We formally characterize the ContiFormer
z' (L) = LN CT-MHA(X' X X w')())))))))))))))))))))))))ww + xx))
(8)
z' (t))) = LN (FN + +z((t))
where (1) the output from the I-th ContiFormer layer at time point L. Additionally, we adopt residual connection to avoid potential gradient vanishing [23] To incorporate the residual connection with the continuous-time output, we approximate the underlying process of the discrete input X using a continuous function x' (t) based on the closed-form continuous-time interpolation function.
Sampling Process As described before, each ContiFormer layer derives continuous function (t) w.r.t. time as the output, while receiving discrete sequence X as input. However, 2t(t) can not be directly incorporated into neural network architectures that expect inputs in the form of fixeddimenssoon ContiFormer. To address this issue, we establish reference time points for output of each layer. These points are used to discretize the layer output, and can correspond to either the input time points (i.e., w) or task-specific time points. Specifically, assume that the reference points for the I-th layer w = [t!, th,,..,tt, 1. l. the input to the next layer X can be sampled as {z (t)) E [1, B:]}.
vectors and sequences [8], which places obstacles when stacking layers of
5

---

3.3 Complexity Analysis
We consider an autoregressive task where the output of each observation required for a particular classifica- tion regression task. Therefore, the reference points for each layer are de- fined as w w.
Table I: Per-layer complexity (Comp. per Layer), minimum number of sequential operations (Seq. Op.), and maximum path lengths (Max. Path Len.). N is the sequence length, is the representation dimension, T is the number of function evaluations (NFE) for the ODE solver single forward pass from to EN. and S represents the NFE from - to 1.
To preserve the parallelization of Transformer architecture and mean- while implement the continuous-time attention mechanism in Eq. (6). we first adopt time variable ODE g to
Model
Comp. per Layer
Seq. Op.
Max. Path Len.
Transformer 5T
O(N2. d)
0(1)
O
RNN 22
O(N-d)
O(N)
O(N)
Neural ODE
O(T-d')
O(T)
O(T)
reparameterize ODEs into single in-
ContiFormer
O V S.))
O(S)
O(1)
terval 1, 1), followed by numerical approximation method to approximate the integrals, i.e.,
(x:(1)) 1r q(7) TT 1 - 4
[ (T) (T) dr Σ Th (Ep) (5)) (9)
where P denotes the number of intermediate steps to approximate an integral, &p E [-1, and 2 0, , k; ((s(t) t4) + 12)/ and (s. q: ((s(t, - -4) + t, + 22)/2). Finally, a; (t,)) can be solved with one invokes of ODESolver that contains N2 systems, i.e.,
(5p1())
kg,1(5p)
t2 +
((s)K())
(10)
1p-1
(NN.((pp11))
kN,N(E,)
K(Ep-1)
K(Ep)
where p [L..... P] and we further define & = -1. Additionally, (Ep) can be obtained by interpolating on the close-form continuous-time query function. Similar approach is adopted for solving vt,t,) in Eq. (5) The detailed implementation is deferred Appendix
We consider three criteria to analyze the complexity of different models, i.e., per-layer complexity, minimum number of sequential operations, and maximum path lengths (5]). Complexity per layer refers to the computational resources required to process within a single layer of a neural network. Sequential operation refers to the execution of operations that processed iteratively in sequential manner. Maximum path length measures the longest distance information must traverse to establish relationships, reflecting the model's 's ability learn long-range dependencies.
As noted in Table D vanilla Transformer an efficient model for sequence learning with O(1) sequential operations and (1) path length because of the parallel attention mechanism, whereas the recurrent nature of RNN and Neural ODE rely on autoregressive property and suffer from large number of sequential operations. Utilizing Eq. ( and leveraging the parallelism inherent the Transformer architecture, our model, ContiFormer, achieves (SS) sequential operations and O(1) path length, while also enjoying the advantage of capturing complex continuous-time dynamic systems. We have & and we generally set S < N in our experiments.
3.4 Representation Power of ContiFormer
In the previous subsections, we introduce novel framework for modeling dynamic systems. Then a natural question is: How powerful is ContiFormer? We observe that by choosing proper weights, ContiFormer be an extension of vanilla Transformer [51]. Further, many Transformer variants tailored for irregular time series, including time embedding methods [48 54 and kernelized attention methods (4 [8), can be special cases of our model. We provide an overview of the main theorem
below and defer the proof to Appendix B
Theorem 1 (Universal Attention Approximation Theorem). Given query (Q) and key (K) ma- trices, such that ||Q.ll2 so, Ilollo for i E [I,..,NN. For certain attention matrix, i.e.

---

Transformer
Latent ODE
ContiFormer
True Trajectory
Interpolation
Extrapolation
@
Sampled Data
Prediction
Figure 2: Interpolation and extrapolation of spirals with pregularly-sampee time points by Trans- former, Neural ODE, and our model.
Atln(Q, K) E RRNN (see Appendix B. for more information), there always exists family of continuously difserentiable vector functions k (-), k2()) (e()).... ..,,...,[]]] ..kk)))))) such that the discrete definition of the continuous-time attention formulation, ie., [on (1) 022 (),... an(()) in Eg. 44). given by
ai(ti)
ag(ti)
aN(t))
a1(12)
a2((2)
aN((2)
Attn(Q,
(11)
(tNt)) (z((N) aN((N)
satisfies that Attn(Q, K) Attn(Q,K).
4 Experiments
In this section, we evaluate ContiFormer on three types of tasks on irregular time series data, i.e., interpolation and extrapolation, classification, event prediction, and forecasting. Additionally, the results are listed in Appendix C.5
conduct experiments on pendulum regression task 19 45).
Implementation Details By default, we use the natural cubic spline to construct the continuous-tin time query function. The vector field in ODE is defined as f(t. x) = Actfi Acttfn (Linneee d.d (Linear" (x)) + Linear" (anear""(()) (t))), where Actfn(.) is either tanh or sigmoid activation function, Linear": R - R a linear transformation from dimension to dimension b, LN denotes the layer normalization. We adopt the LegesrLe Quadrature approximation to implement Eq. (9) In the experiment, we choose the Kunge-Kutta-4 [44 (RK4) algorithm solve the ODE with fixed step of fourth order and step size of 0.1. Thus, the number of forward passes to integrate from 11 to 1, i.e., S in Table [ is 80. All the experiments were carried
single 16GB NVIDIA Tesla V100 GPU.
4.1 Modeling Continuous- Time Function
The first experiment studies the effectiveness of different models from different categories continuous-timg function approximation. We generated dataset of 300 2-dimensional spirals, sampled at 150 equally-spaced time points. To generate irregular time points, we randomly sample 50 points from each sub- sampled trajectory without replacement. We visualize the interpolation and extrapolation results of the irregular time series for differ- models, namely Latent ODE (w/ RNN en- coder) 19), Transformer [51] and our proposed ContiFormer. Besides, we list the interpolation and extrapolation results in Table 2 using rooted mean squared error (RMSE) and mean absolute error (MAE) metrics. More visualization results and experimental settings can found Appendix C.I As shown Figure 2 both Latent ODE and ContiFormer can output a smooth and continuous function approximation, while Transformer fails to interpolate given the noisy observations (C). From Table 2 we can observe that our model outperforms both Transformer and Latent ODE by large margin. The improvement lies in two
Table 2: Interpolation and extrapolation results of different models on 2-dimensional spirals. + indi- cates the lower the better.
Metric
Transformer
Latent
ContiFormer
Task Interpolation (x10
RMSE (J) MAE (L)
1.37 0.14
2.09 +: 0.22
0.49 0.06 0.52 0.06
1.42 0.13
1.95 0.25
Task Extrapolation (x100
RMSE ())
1.10
1.59 - 0.05
0.64 0.09
MAR (L)
1.49 0.12
1.52 +: 0.05
0.65 0.08

---

Table 3: Experimental results irregular time series classification. Avg. ACC. stands for average
accuracy over 20 datasets and Avg. Rank stands for average ranking over 20 datasets. t (4) indicates the higher (lower) the better.
Mask Ratio
30% Dropped
50% Dropped
70% Dropped
Metric
Avg. ACC. (f)
Avg. Rank (U)
Avg. ACC. (f)
Avg. Rank (1)
Avg. ACC. (1)
Avg. Rank (L)
GRU-D 30
0.7284
6
0.7117
5.8
0.6725
6.15
GRU-At 6
0.7298
5.75
0.7157
5.65
0.6795
ODE-RNN
0.7304
5.45
0.7000
5.55
0.6594
Neural CDF 30
0.7142
6.85
0.6929
6.5
0.6753
6.3
miTAN 48
0.7381
5.7
0.7118
5.85
0.6955
5.4
CADN 10
0,7402
5.4
0.7211
5,55
0,7183
3,9
(49
0.7854
4.4
0.7638
4.25
0.7401
TST [57
0.8089
2.75
0.7793
3.15
0.7297
ContiFormer
0.8126
2.4
0.7997
1.9
0.7749
aspects. First, compared to Transformer, our ContiFormer can produce an almos almossccontnnuuu-ttig output, making more suitable for modeling continuous- time functions. Second, compared Latent ODE, our ContiFormer excels at retaining long-term information (2), leading to lower prediction error in extrapolating unseen time series. Conversely, Latent ODE prone cumulative errors (o). resulting in poorer performance in extrapolation tasks. Therefore, we conclude that ContiFormer is more suitable approach for modeling continuous-tin functions on irregular time series.
continuous-time
4.2 Irregularly-Samplee Time Series Classification
The second experiment evaluates our model on real-world irregular time series data. To this end, we first examine the effectiveness of different models for irregular time series classification. We select 20 datasets from UEA Time Series Classification Archive (3] with diverse characteristics in terms of the number, dimensionality, and length of time series samples. More detailed information and experimental results can be found in Appendix
C.2 To generate irregular time series data, we follow the setting of 30 to randomly drop either 30%, 50% or 70% observations.
We compare the performance of ContiFormer with RNN-based methods (GRU-At 30 Neural
GRU-D 6 methods (ODE-RNN (43). CADN [IO), Neural CDE 30 3SM-based models
(S5 49 and attention breset methods (TST 57 mTAN 48
Table presents the average accuracy and rank number under different drop ratios. ContiF- ormer outperforms all the baselines on all three settings. The complete results are shown in Ap- pendix C.2.3 Moreover, attention-based meth- ods perform better than both RNN-based meth- ods and aDE-based methods, underscoring the significance of effectively modeling the inter- correlation of the observations. Additionally, we find that our method is more robust than other models. ContiFormer exhibits the smallest performance gap between 30% and 50%.
ContiFormer
mTAN
0.03
Time
Besides, we study the learned attention patterns with different models. Attention captures the impact of previous observations on the future state of system by measuring the correlation between different observations at different time points. Consequently, is expected that a model effectively capturing the underlying dynamics would exhibit continuous, smooth, and input-dependent intermediate outcomes. As illustrated Figure 3 ContiFormer and TST are both capable of
Figure 3: Visualization of attention scores on Liarerrr dataset. Colors indicate the attention scores for different instances at time i O. Observations at time 0 observed and normalize the time interval to [0, 1].
'The output of ContiFormer can be considered "continuous" only we overlook approximation and numerical emrors in the ODESolver. Additionally, traditional Transformers are commonly trained with dropout to prevent overfitting, which can result in discontinuous outputs. To mitigate this issue, straightforward approach set the dropout rate to 0, thereby producing "almost" continuous output for ContiFormer.

---

Table Prediction result of compared models for event prediction. LL for log- oog-likelhhod and ACC for accuracy. t (4) indicates the higher (lower) the better. (bold values indicate the best performance and + indicates outperforming the best baseline by at least 3 standard deviations.)
Model
Metric
Symbetic
Neonate
Trrfftic
MIMIC
StackOverflow
BookOrder
(1)
-3.084 0055
4,618 .055
-1.482 005
-4618 005
-5,794 = 005
-1,036 + 000
HP 2
Accuracy (1)
0.756 + 000
0.570 + 000
0,795 000
0.441
0.604 000
RMSE
0,953 # 000
10:957 + 0122
0:407 + 000
1002 000
1,341 + 000
3,781 000
(T)
-1025 1 030
-2817 + 023
0.546 .012
-1184 023
2.374 0001
0,952 #. .007
RMTPP Im
Ascuray (
0.841 .000
0.805 + 0002
0.823 + 0044
0.461
.000
0.624 .000
RMSE (
0.369 + .014
9.517 # .023
0.337 4. .001
0.864 .017
0.955 +. .000
3.647 1: .003
(L(1)
-1.371 .004
.012
-0.643
-1.239 007
-2,608
.000
-1.104
NeuralHP 30
Accuracy (1)
0,841 = .000
0.759 4 0001
0,814 + 000
0.450 + 000
0.621 # 000
RMSE
9.614
0.358 a: 0001
3.734 .003
LL
-0.619 .U63
-2.646 .057
0.372 4: 002
-1.110 .030
-2,404 .002
-0,304 .002
SAHP S
Accuracy (
0,841 = .000
0.780 + 001
0.830 = 004
0.455 + .000
0.622 + .000
RMSE ())
0,521 .055
9,403 .600
0.337 + 0000
0,851 0006
0.963 + .000
3.676 .011
LL((1)
-0.589 + .017
-2.702 .045
-0.569 + .015
-1.137
.038
-2.354 001
-1,102 + .052
THP (4
Accuracy
0,841 000
0.818 + 000
0.834 000
-0,468
000
0.622 0004
RMSF (
0.205 + .006
9.471 + .045
0.332 + .000
0.843 + .017
0.951 + .000
3.688 .004
LL(11)
1.001 025
-2.747 .061
.012
0003
-1.098
NSTKA 38
Accuracy 1
0,842 2 .000
0.767 +: .001
0,833 a .003
0.465 +. .000
0.621 2. .000
RMSE (
9.502
0.346 +
3.731 .007
(1)
-0.871 .050
-2.736 .031
-0.613 .62
-1.164 006
-2.389 0022
-0.915 = .006
GRU-AI
Accuracy (
0.841 + .000
0,800 4. 004
0.832 # .00
0.466 4. .000
0.627 1. .000
RMSE (
0,249 .013
9,421 .550
0.335 # 000
0,850 .010
0,950 + .000
3,666 .016
(1
-1.032 #. J02
2.732 ,080
-0.491 # .011
-1.183 + .028
-2.395 .011
-0,988 + 006
ODE-RNN f
Accuracy 1
0.841 .000
0.812 + .000
0.827 .006
0.467 .000
0.624 .000
RMSE (
0,342 + .030
9,289 # .048
0.334 # .000
0.865 # .021
0,952 # .000
3,605 # .004
LL(1))
-0.920 036
-2.722 .026
-0.548
-1.149 029
-2.391 002
-0,980 +
mTAN 4
Accuracy (1)
0,842 (000
0.811 + .002
0,832 .009
0.466 + 0000
0.620 + 0000
RMSE
0,286
9.363
0.334 a: 0001
3.680 0155
LL-(1)
40.535 .028*
-2,550 .026
0.635 .019*
-1.135 .023
-2.332 .00++
-0,270 .010"
ContiFormer
Accuracy (1)
0.842 .000
0.822 + _001*
0.836 .006
0.473 + .000
0.628 + .001
RMSE
0.192 .005
9,233
0.328
0.837 .007
0,948 0000
3.614 .20
capturing input neput-dependent patterns effectively, while mTAN struggles due to the fact that its attention mechanism depends solely on the time information. In addition, ContiFormer excels in capturing complex and smooth patterns compared to TST, showcasing its ability to model time-evolving correlations effectively and potentially more robust noisy data. Consequently, believe that ContiFormer better captures the nature of irregular time series.
4.3 Predicting Irregular Event Sequences
Next, we evaluate different models for predicting the type and occurrence time of the next event with irregular event sequences 7 21 46 47), a.k.a, marked temporal point process (MTPP) task. To this end, we use one synthetic dataset and five real-world datasets for evaluation, namely Synthetic,
Neonate [50]. Traffic (32) MIMIC [66.. BookOrder [16] and StackOverflow B3). We use the fold cross-validation scheme for Synthetic, Neonate, and Traffic datasets following 17), and the f-lold
cross-validation scheme for the other three datasets following (39 64]. We repeat the experiment
3 times and report the mean and standard deviations in Table 4 More information about the task
description, dataset information, and training algorithm can be found in Appendix C.3
We first compare ContiFormer with models on irregular time series, i.c., GRU-At, ODE-RNN, and m TAN, which respectively belong to RNN-based, ODE-based, and attention based approaches. Additionally, we compare ContiFormer with baselines specifically designed for the MTPP task. These models include parametric method (HP (221). RNN-based methods (RMTPP [66. NeuralHP (3)),
and attention-basee methods (SAHP (59, THP 64
NSTKA (38)
The experimental results, as presented in Table 4 demonstrate the overall statistical superiority of ContiForrerr to the compared baselines. Notably, the utilization of specific kernel functions, e.g., NSTKA A and m'TA,, results in subpar performance on certain datasets, highlighting their limitations in modeling complex data patterns in real-world scenarios. Furthermore, parametric methods like HP, perform poorly most datasets, indicating their drawbacks in capturing real-life complexities.

---

Table 5: Results of regular time series forecasting. Input length for is 36 and 96 for the others.
Prediction lengths for ETTm2 and ETTh2 are (24, 48, 168, 336), (96, 192, 336, 540] for Exchange and Weather, and {24, 36, 48, 60) for III. Results are averaged over the four prediction lengths.
Bold/underlined values indicate the best/second-best MSE stands for mean squared error.
Model
ContiFormer
TimesNet 52
DLinear [56
FEDrommer 62
Autoformer
S3
Transformer
[1
Metric
MSR
MAR
MSF
MAE
MSE
MAR
MSE
MAE
MSR
MAE
MSE
MAE
Exchange
0.339
0.399
0.320
0.396
0.277
0.377
0.392
0.446
0.440
0.477
1.325
0.949
ETTm2
0.226
0,300
0.214
0.276
0.220
0.305
0.226
0.300
0.231
0.307
0.689
0.633
ETTh2
0.356
0.395
0.321
0.364
0.358
0.396
0.351
0.393
0.352
0.394
2,081
1.179
IIII
2.632
1.042
2.874
1,066
4.188
1.493
4.467
1.547
4.046
1,446
5.130
1.507
Weather
0.257
0.290
0.259
0.285
0.260
0.311
0.315
0.360
0,318
0.359
0.672
0.579
4.4 Regular Time Series Forecasting
Lastly, we evaluate ContiFormer's performance in the context of regular time series modeling. To this end, we conducted an extensive experiment following the experimental settings from [53]. We employed the ETT, Exchange, Weather, and ILI datasets for time series forecasting tasks. We compare ContiFormer with the recent state-of-the of-thee-rt model (TimesNet [52]) Transformer- based
models (FEDformer(6) 62], Autoformer 53) and MLP-based model (DLinear [56)).
Table 5 summarizes the experimental results. Contiformer achieves the best performance on ILI dataset. Specifically, gives 10% MSE reduction (2.874 - 2.632). Also, Contiformer outperforms Autoformer and FEDformer on Exchange, ETTm2, and Weather datasets. Therefore, it demonstrates that Contiformer performs competitively with the state-of state-of-the-ar model regular time series forecasting, For more comprehensive exploration of the results, please refer to Appendix
5 Efficiency Analysis
Employing continuous time modeling in Con- tiFormer often results in substantial time and GPU memory overhead. Hence, we conduct comparative analysis of ContiFormer's actual time costs against those of Transformer- based models (TST), ODE-based models (ODE- RNN, Neural CDE), and SSM-based models (S5).
Table 6: Actual time cost during training for differ- ent models on RacketSports dataset (0% dropped) from UEA dataset. We adopt RK4 solver for ODE- RNN, Neural CDE, and ContiFormer. Step size refers to the amount of time increment for the RK4 solver. (L) indicates the higher (lower) the better.
Table O shows that both TST and S5 are effec- tive models. Additionally, ContiFormer lever- ages parallel mechanism for faster inference, resulting in faster inference speed as compared to ODE-RNN and Neural CDE. Furthermore, ContiFormer demonstrates competitive perfor- mance even with larger step size, detailed in Appendix D.3] Therefore, ContiFormer's ro- bustness step size allows it to achieve supe- rior balance between time cost and performance.
Model
Step Size
Time Cost (L)
Accuracy (f)
TST S7
0.14x
0.826
1x
0.772
0.1
2.47x
0.827
ODE-RNN
43
0.01
14.64x
0.796
0.1
3.48x
0.743
Neural CDE
42
0.01
25.03x
0.748
0.1
1.88x
0.836
ContiFormer
0.01
5.87
0.836
Please refer Appendix E for more details.
6 Conclusion
We propose ContiFormer, a novel continuous-time Transformer for modeling dynamic systems of irregular time series data. Compared with the existing methods, ContiFormer is more flexible since it abandons explicit closed-form assumptions of the underlying dynamics, leading to better generalization various circumstances. Besides, similar to Neural ODE, ContiFormer can produce a continuous and smooth outcome. Taking advantage of the self-attention mechanism, ContiFormer can also capture long-term dependency in irregular observation sequences, achieving superior performance on variety of irregular time series modeling tasks.
10

---

7 Acknowledgement
This research supported in part by the National Natural Science Foundation of China under grant 62172107.
References
[1] Abdul Fatir Ansari, Alvin Heng, Andre Lim, and Harold Soh. Neural continuous-discrete state space models for erregulrly-samppee time series. arXiv preprint arXiv:2301. 2023.
[2] Imanol Perez Arribas. Derivatives pricing using signature payoffs. arXiv preprint
17Xiv:100.00946 2018.
[3] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018.
arXiv preprini arXiv: 1.00005 2018.
[4] Stefan Bauer, Bernhard Schölkopf, and Jonas Peters. The arrow of time in multivariate time series. In International Conference on Machine Learning, pages 2043-2051. PMLR, 2016.
[5] Youness Boutaib, Lajos Gergely Gyurkó, Terry Lyons, and Danyu Yang. Dimension-free euler estimates of rough differential equations. arXiv preprini arXiv: 307.4708, 2013.
[6] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
[7] Chao Chen, Haoyu Geng, Nianzu Yang. Junchi Yan, Daiyue Xue, Jianping Yu, and Xiaokang Yang. Learning self- self-modulating attention in continuous time space with applications to sequen- tial recommendation. In International Conference on Machine Learning, pages 1606-1616. PMLR, 2021.
[8] Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. In International Conference on Learning Representations, 2021.
[9] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[10] Jen- Tzung Chien and Yi-Hsiang Chen. Continuous-timm self-attention in neural differential equation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3290-3294. IEEE, 2021.
[11] Jen-Tzung Chien and Yi-Hsiang Chen. Learning continuous-time dynamics with attention. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[12] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:/4099 2259, 2014.
[13] Edward Choi, Cao Xiao, Walter Stewart, and Jimeng Sun. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. Advances neural information processing systems, 31, 2018
[14] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv: 14/2.3555, 2014.
[15] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880) 2018.
[16] Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Comez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1555-1564, 2016.
[17] Yuchen Fang, Kan Ren, Caihua Shan, Yifei Shen, You Li, Weinan Zhang, Yong Yu, and Dongsheng Li. Learning decomposed spatial relations for multi-variate time-series modeling. In Proceedings of the AAAI Conference Artificial Intelligence, 2023.
11

---

[18] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances Neural Information Processing 2022.
Systems,
[19]
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00300 2021.
[20] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang. Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175-12185, 2022.
[21] Vinayak Gupta and Srikanta Bedathur. Proactive: Self-attentive temporal point process flows for activity sequences. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 496-504, 2022.
[22] Alan G Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90, 1971.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
(24) Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 173: 1997.
[25] John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-255 2558, 1982.
[26] Sheo Yon Jhin, Heejoo Shin, Sujie Kim, Seoyoung Hong, Minju Jo, Solhee Park, Noseong Park, Seungbeom Lee, Hwiyoung Maeng, and Seungmin Jeon. Attentive neural controlled differential equations for time-series classification and forecasting. Knowledge and Information Systems, pages 1-31, 2023.
[27] Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
[28] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02433, 2022.
[29] Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons Deep signature transforms. Advances in Neural Information Processing Systems, 32, 2019.
[30] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
[31] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint K5i:2001100455 2020.
[32] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
[33] Jure Leskovec and Andrej Krevl. Snap datasets: Stanford large network dataset collection, 2014.
[34] Jiacheng Li, Yujie Wang. and Julian McAuley. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th international conference web search and data mining, pages 322-330, 2020.
[35] Yan Li, Xinjiang Lu, Haoyi Xiong, Jian Tang, Jiantao Su, Bo Jin, and Dejing Dou. Towards long- term time-series forecasting: Feature, pattern, and distribution. arXiv preprint XwX.:2330100006... 2023.
[36] Zachary Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with mns: Improved classification of clinical time series. In Machine learning for healthcare conference, pages 253-270. PMLR, 2016.
[37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint
2019.
12

---

[38] Yu Ma, Zhining Liu, Chenyi Zhuang. Yize Tan, Yi Dong, Wenliang Zhong, and Jinjie Gu. Non- stationary time-aware kernelized attention for temporal event prediction. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1224-1232, 2022.
[39] Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. Advances in neural information processing systems, 30, 2017.
[40] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages 7829-7838. PMLR, 2021.
[41] Michael C Mozer, Denis Kazakov, and Robert V LLindsey. Discrete event, continuous time mnns. arXiv preprint arXi::1710.0411 2017.
[42] Sung Woo Park. Kyungjae Lee, and Junseok Kwon. Neural markov controlled sde: Stochastic optimization for continuous-time data, In International Conference on Learning Representations, 2022.
[43] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for iregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
[44] Carl Runge. Über die numerische auflösung von differentialgleichunnen. Mathematische Annalen, 46(2):167-178, 1895.
[45] Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In International Conference on Machine Learning, pages 19388-19405. PMLR, 2022.
[46] Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. arXiv preprint arXiv: /909.1 /2/27, 2019.
[47] Oleksandr Shchur, Nicholas Gao, Marin Bilos, and Stephan Günnemann. Fast and flexible temporal point processes with triangular maps. Advances in Neural Information Processing Systems, 33:73-84, 2020.
[48] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.
[49] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling, arXiv preprint arXiv:2208 04933, 2022.
[50] Nathan J Stevenson, Karolina Tapani, Leena Lauronen, and Sampsa Vanhatalo. A dataset of neonatal ceg recordings with seizure annotations. Scientific data, 6(1):1-8, 2019.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[52] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Times- net: Temporal 2d 2d-variation modeling for general time series analysis. arXiv preprint
arXiv:2210.02186 2022.
[53] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430. 2021.
[54] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning, Advances in neural information processing systems, 32, 2019.
[55] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980-8987, 2022.
[56] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
13

---

[57] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-basee framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,
pages 2114-2124, 2021.
[58] Jing Zhang, Peng Zhang, Baiwen Kong, Junqiu Wei, and Xin Jiang. Continuous self-attention models with neural ode networks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 14393-14401, 2021.
[59] Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes process. In International conference on machine learning, pages 11 1113--11193. PMLR, 2020.
[60] Yuan Zhang. Attain: Attention-basee based time-aware Istm networks for disease progression mod- eling. In In Proceedings of the 28th International Joint Conference on Artificial Intelligence (ICCAI-2019), pp. 4369-4375, Macao, China., 2019.
[61] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[62] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference on Machine Learning, pages 27268-27286. PMLR, 2022.
[63] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04155 2020.
[64] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In International conference machine learning, pages 11692-11709 PMLR, 2020.
14

---

A Implement Details and Complexity Analysis
A.1 Background: Time Variable ODE
A numerical ODE solver integrates single ODE system di f(t,x), where x E R2 and f R2+1 - R. Given initial state No. the ODE solver produces Nt Negs Neg which represents latent state for each time point [9] i.e.
at
Nty Ntg...., Biy ODESolve (Xto f, to....,ty)
(12)
To obtain the solution of Eq. 12 it requires the ODE solver to integrate over the interval [lo, [x]-
Varied time intervals Now, we suppose that we need solve M dynamic system that has different
time intervals. Let denote for the i-th system, the start point is (6) and the endpoint is {() and the ODE system is defined by dx = ft, x). We can construct dummy variable so that the integral dr interval always starts from 1 and ends and perform change of variables to transform every system to use this dummy variable.
For simplifying purposes, we assume that single system that integrates from to t with f(t, x) can control the differential equation and the solution of x()). We introduce a dummy variable
26-1p=11 and a solution {(s) in a fix interval [-1, 1] satisfies
fo-
x s(t tu) + + t
(13)
The differential equation that solve follows that
dk(s)
dx(())
dl
f(s.x(s))
ds
dt
2lt, +o)
ds
{ - tu
f(t, ((t,((t)) واا وجي (
(14)
2
=f 8((1 - 40) + 11 x(s)
(0=)4²²1)) lg
(h-1)
2
Since x(-1) x(to) and x(1) Zt,, thus we have
x(4) = ODESolve(((()))))) (1) OODEolve((-1)),
(15)
A.2 Background: Approximating Definite Integral of a Function
Given a function f that integrates over an interval [-1, 1], simple method to approximate it is to assume that the change is linear over time, i.e.
J f(z)dr= f(-1)+f(1)
(16)
Besides, in numerical analysis, Gauss- -eagenee Quadrature provides a more accurate way to approx- imate an integral over interval [-1, 11, i.e.,
[ f())dx R w (Et)
(17)
where {wy} and {&,} are quadrature weights and nodes such that -1 5 < Ex 1.
A.3 Putting it all together
Given an irreuular-sampled sequence, [(x1, 41). (222, .2), (xN, where s; E Rd the input feature of the i-th tokens happen at time (; and N is the length of the sequence. We analyze the time and space complexity for an autoregressive task where the output of each observation is required for particular classification or regression task. Specifically, for Eq. (3) (x())),
(tx)x)
15

---

is required to form the output of Transformer. Therefore, the key difficulty is to solve Eq. 4 By using the numerical approximation method, have
(t,t;) Sr: ((T)) dr
[ (T,
1
P ((p)-ks,
Σ
To
(Ep) (18)
4-4
2 p=1
where di, and ,, follows that:
s(t; - tj) +
(s) =q
(19)
[s(t; - tj) + +t +
ks)( 9j
2
We ki,, denote
(s) Thus, (t,) can be solved by paralleling 222 ODE solvers. Specifi-
ds
cally, (Ep) is given by
(E1), k,, (E2) (Ep) (DESotrer(k,(t)).k..) E.
(20)
As for q., interpolation methods [30] (e.g., linear interpolation or cubic interpolation) method on to construct a continuous-ting time query. Therefore (5x( can be easily calculated in ((d) time complexity.
g (Ex), since the input sequence is irregularly sampled and discrete in time, we propose to use
B Representation Power of ContiFormer
In section, we mathematically prove that by curated designs of function hypothesis, many Transformer variants can be covered as special case of ContiFormer. For simplification, we only discuss the self-attention mechanism, where Q, K, E Nxd
B.1 Transformer Variants
We first show several Transformer-based methods for irregular time series. Most of these methods shown below calculate N N attention matrix and multiply with V
Vanilla Transformer Given query Q E IR Nxd and V E IR Xxd the attention matrix of a vanilla transformer [5T] calculated
Q KT
Qi K.
Qi-K3
Q2. K
Q2 Kg
Q2 K
AQt
(21)
QN KT
QN K2
QN KT
and the output of the self-attention mechanism is obtained by
Z; = Softmax(Attn(Q.. K):)V.
(22)
Kernelized Attention Kernelized attention mechanism incorporate time information through defining specific kernel function k(x, æ²) and replace the attention matrix with
Q KT 4() Q2 KT t)(t,,t))
Q1 KJ k(t..22
Q1 KJ k(l,,t))
Q2 K2
k(tz,tz) t2)
Q2 KN k((2,
Attn(Q,K) =
QN KT (1(N, QN K2 k(NN 12)
QN KJ klNN (N)
(23)
"Throughout the proof section, define (ta) (f.) V, and focus the proof of attention matrix. 5 Forr simplification, we assume single-head attention is adopted throughout the proof. "Not to be confused with methods that regard kernels as form of approximation of the attention matrix.
16

---

Kernl functions are typically symmetric and hold that Φ(2,z) = 1 for any For instance, NSTKA 38 propose a learnable Generalized Spectral Mixture (GSM) kernel, i.c.
kGSM (x,, x²) = Σ (e(x)) (a') Reaabbs, xx, cos (2 (1z((x) Me (x0)
(24)
21,(1)44 exp 1(xx²² + 44 (x')²
(x- (1(())2211 +1 (;
where K (z,z') x')
V
where fa(ar) denotes the mean, lengthscale as (xx) and input-dependent weight as dx(x).
Time Embedding Methods Time embedding methods learn time representation, adding or concatenating it with the input embedding, i.c..
(Q1, ((1)) (K1, (T1)) ((22)) (K1, ((t)))
(Q,, ((1))) (KN. p(tN))
(Q2, p(22)) (Kv, ((EN))
At)
(25)
(QN: ((t))) (K1, ((1)))
(QN, p(tn)) (KN, *(tn))
where (+.-) is the concatenation of two vectors, +(-) For simplification, we assume that time embedding is concatenated with the input embedding and ignore linear transformation on the time embedding[]
R + Rd learnable time representation.
For instance, Mercer [54] transforms the periodic kernel function into the high-dimensional space spanned by truncated Fourier basis under certain frequencies (fw;) and coefficients {ci), i.e.,
(() + Φ. $ wi.al
B.M
jul
jul
(26)
where (1) Vel
Caj
Co8
e2j+1 sin
mTAN 48 generalizes the notion of a positional encoding used in Transformer-bart models to continuous-time domain, i.e.,
(t) o(t)
(27)
where w E [Rd is a linear transformation
g and
<(l), WD + do.
ifi=0 ifocisd"
(28)
sin(us t + d.),
where ws and & are learnable weights. Furthermore, mTAN assumes that Q K 0.
Besides, since Softmax(x c) Softmax(x). where is vector and e is a constant. The following equation holds,
Softmax Coftmax(Attn(Q, K)) Softmax (Atin(Q, K)), where the matrix form Softmax normalize each row the matrix and Attn(Q,
(29)
Q KJ Q2 Ki-Ki 2(22) (0(12) +(t)))
Q1 KN + +(t)) ( ((tt))
Q2 KN + ((2)) (8(tN) ((22))
QN K, + ((tx) (0(1)) -+(x))
QN KT
(30)
Using the concatenation operation, have (Q., ((.)) (K,, ((tt)) = Q KJ ((t)) "In the original paper, the equation is defined as $ (t,,t,, o(t:)wr o(t,). For simplicity purpose, we assume that
17

---

B.2 Proof of Extension and Universal Approximation
Lemma 1 (Existence of Continuously Differentiable Vector Function). Given a continuously dif- ferentiable vector function u I + R E C', such that Vz E R. [|](z]]22 < co. [|u(=)]lo d and a continuously differentiable function w R
E C), there always exists another continuously differentiable vector function IR + Rd E C. with fixed point p satisfying u(p) v(p) w(p),
such that VI E IR, u(x) v(x) = w(x) holds. 9
Proof. Throughout the proof, we define v(p). Since Vo E IR, [[((z)]22 < oo, we define
u(x)
x(z) = ||u())) w(x) + k(x),
(31)
where k(x) E C² such that Vx E R. u(z)
k(x)] = 0 holds. Therefore, for Vr IR, we have
u(z) v(x) u(x)
u(x)
w(x) k(x)) w(x) + u(x) k(x) w(x).
(32)
|u(x)]]
Thus, only need to prove the existence of k(x). Since v(x) has fixed point p, therefore, w(p)
u(p)
k(p) =
(33)
||((p) 1
Let z|i] denote the i-th element of vector 2, thus, Eq. 33 is equivalent to
(p) E|i u(p)
wp))()
(34)
for E [1, d]. Since we have u(p) v(p) = (p). thus,
k(p) u(p) (((ppp =w(p) 0.
(35)
We denote G; k(p]ii] u(p)[(]. Since Eq. construct k(x:) as
35 holds, therefore Ex Q:
Finally, we can
[1x)
a/u(x)(1
k(x)
c2/u(x)(2)
k(x)= =
(36)
k())dd
ca/u(z)(d
Since u, w E C², therefore & E C², thus v E C..
Lemma (Existence of k; (f)). Given a continuously differentiable vector function q + Rd C such that Vz E R, [[q(x) ll oo, [[q(z)]lo d and continuously differentiable function f R + R E C² and vector K; E Rd and satisfies f(t+) = q(t,) K3 there always exist a continuously differentiable vector function k, IR + Rd, such that the following statements hold:
. k:(l+) K,,,
VE t. S1 (()) f(t). t-
Proof. Since # t. we have
(1 5(1)
an(((1 4) = ((1)(1-)) - 4:)
de (2) af(() df(()(1
(37)
dt
dt
q(t) (t) = df - t.) f(t).
dt
"Consider open set the real line and function f defined 24 with real values. Let l be non- negative integer. The function f is said to be of differentiability class C+ if the derivatives f", f"..., f ((k) exist and are continuous 24.
18

---

where 1 holds since we can define (t,t;) = fts).. Besides, since
Se
tt q(T) (((T = q(t)) (t,))) = q(t+) K,
lim (t, + +c) |c|-0
lim |4|++
(38)
such a definition does not break the continuity of a.
Since C²2, therefore denote g(t)
Jf (I (4) [(() E Ct. Besides,
g(t.) ft,,) q(t)) K q(t)) (tttt)
(39)
By using Lemma with corresponding to q to u, k; to v and = ti, we can prove the existence of k; that satisfies Eq. (37
Theorem 2 (Universal Attention Approximation Theorem). Given query (Q) and key (K), such for E NJ. For any attention matrix with formulation defined ie., Attn(Q, K), there always exists a family of continuously differentiable vec-
that [|Qil|2 < 0o, ||Q:
in Appendix B.1
tor functions k (.), 12..)....... kx()). such that the discrete formulation of [a1(-), ox2())) ON (-)
defined by
an(t))
az(t))
aN((t)
(1
(22
ax((2)
Attn(Q, K))
(40)
(1lt)) a2(tx)
aw(tN)
satisfies that Attn(Q, K) Attn(Q, K), where
(x(()) Jr q(T) kk, .if #4 4
(41)
t₁ and (ts))) q(ti) k, (t,), qtt) satisfies q(t;) = Q
Proof. First, we show that the diagonal elements in Eq. (40), i.c..
(4) ((4)) (4.) Qi K,T
(42)
holds for different forms of attention matrix defined in Appendix B.1
Next, for a given E [1, NJ. there exists a continuous cubic spline h; R + satisfies hc(4,) Attn(Q, K)2, for Vjj [I......V .V] and h4 C By using Lemma with corresponding to f there exists a function k; such that (t,t) = (t). Therefore. au(t)) httt)) for [l,..., NN. and we finish the proof.
C Experiment Details
C.1 Modeling Continuous-Tim Function
C.1.1 Background: 2D Spiral
A spiral is curve that starts from central point and gradually moves away from it while turning around it. We consider two types of spirals that start from time and end at time e. The first type of spiral is defined as
xxt) = (a + bt): cos(t). yt) + bt) stn
(43)
while the second type of spiral is defined as
(r() (a + 50b/ (e - 1)) cos(e - 1) y(t) (a 50b/ (e - f)) sin(e - t)
(44)
where and 6 are parameters of the Archimedean spiral.
19

---

Table 7: More experimental results spiral 2d.
ODF (w/ RNN Enc)
Latent ODE (w/ ODE Enc)
Metric
Transformer 51
Neural ODE
Neural CDE
ContiFormer
Task Interpolation (x10 ")
RMSE (
+
1,90 000001. 0.17
2,09 +: 0.22
4.53 +
4.80 0.50
0.06
0.02
MAE (U)
1.42 + 0.14
1.90 + 0.17
1,95 +: 0.26
4.25 1.48
5.12
0.47
0.53 + 0.07
RMSE (
2.30 = 0.51
1888 = 0.09
3.34 + 0.10
4.82 = 1.97
4.60 =600=10 1.08
0.42 = 0.04
0.2
MAR (L)
1.75 + 0.18
1.58 + 0.12
3.16 + 0.43
4.73 + 1.21
4.58
0.93
0.42 0.01
RMSE (1) (AE
3.75 +: 0.50
2.1
3.04 4: 0.48
4.49 +: 1,26
2.92
0.32
0.11
2
2.33 + 0.60
2.49 0.30
2,88 + 0.54
4.12 1.00
2.96
0.23
0.41 0.07
Task = Extrapolation (x10"3)
0.02
RMSE (
1.37 + 0.11
2.07 0.02
1.59 + 0.05
2.69 + 0.82
2.24
0.03
0.64 + 0.10
MAE (U)
+ 0.12
2.02 4: 0.07
1.52 + 0.09
2.83
2.38
0.04
0.65 0.08
RMSE
2.96 + 1.46
4.38 + 0.35
4.78 4.155
5,03 4 1.96
3.81
1.02
0.74 + 0.10
0.2
MAE (L)
2.79 + L.1
3.35 = 0.43
4,02 +: 0.59
4.72 1.96
3.37
0.78
0.69 + 0.09
RMSE (1)
5.36 + 0.97
6.02 0.58
5,70 +: 0.57
4.68 +: 1.70
3.44
1.06
0.75 + 0.07
2
MAE (U)
4.48
0.43
4.46
4.29 1.34
3.32
0.85
0.07
C.1.2 Experiment Setting
We generate 300 spirals and 200/100 spirals are used for training/testing respectively. For each spiral, it randomly belongs either clockwise spiral counter-clockwise spiral, sample a ~ (O,, a), ~ ~ a)) and we set a 0.02 for Table 2 Also, we add Gaussian noise from (O () to all training samples and we set 0.1. We test different models for interpolation and extrapolation. Specifically, each spiral is sampled at 150 equally-spaced time points. To generate an irregular time series, we randomly sample 30 points from the first half of the trajectory. Thus, the first half of the trajectories are used for the interpolation task while the second half of the trajectories, which is totally unobserved during both training and testing, are used for the extrapolation task. All the models are trained for 5000 iterations with batch size equal to 64.
C.1.3 Baseline Implementation
We compare ContiFormer with Neural ODE-based models and Transformer.
Neural (DE This model uses the hidden vectors from the last observation as input, and obtain both the interpolation and extrapolation result from single ODE solver. We train the model by minimizing the mean square loss.
Latent ODE (w/ RNN Enc) We run an RNN encoder through the time series and in- fer the parameters for posterior over zto Next, we sample Zto (Zto (xx,, fz)). Fi- nally, we obtain both the interpolation and extrapolation result from a single ODE solver, i.e., ODESolve (Zt, f tu, t... where (to, fi,,...... .... t. is the time of the trajectory. We train the model by maximizing the evidence lower bound (ELBO).
Latent ODE (w/ ODE [43) We replace the RNN encoder Latent ODE (w/ RNN Enc) with the ODE-RNY -DE-NNN encoder following [43]. Again, train the model by maximizing the evidence lower bound (ELBO).
Neural 42). We follow the official implementation to construct the CDE function. For the interpolation, the hidden vectors directly obtained from the continuous-tin timm path from Neural CDE. For the extrapolation, we input the hidden vectors from the last timestamp the Neural CDE single ODE solver. We train the model by minimizing the mean square loss.
Transformer(52) In order obtain the prediction for both seen and unseen time points, use a learnable mask for unseen inputs. Instead of using position embedding, the input is added with temporal encoding before forwarding it to the transformer layer. We train the model by minimizing the mean square loss.
ContiFormer: We use linear interpolation to deal with unseen observations to avoid the numer- ical error caused by constructing the continuous time query. Similar to Transformer, temporal encoding is adopted and the model is trained by minimizing the mean square loss.
20

---

C.1.4 Experimental Results with Different
In this subsection, we test Transformer, Neural ODE and Contiformer with different a to show robustness of these models. As shown in Table 7 our model can achieve the best result on different parameters.
C.1.5 More Visualization Results
More visual results with a=0.02 can be found in Figure
Transform
Latent ODE
ContiFormer
Figure 4: More visualization results with 0.02. Here, Latent ODE refers to Latent ODE w/ RNN Encoder
C.2 Irregular Time Series Classification
C.2.1 Training Details
For both our model and the baseline models, we adopted a fixed learning rate of 10-2 and a batch size of 64. To ensure the reliability of our results, all experiments were conducted three times, and the reported values represent the mean. The training process for all models lasted for 1000 epochs. However, if the training accuracy failed to show any improvement for 100 consecutive epochs, the training was stopped to avoid unnecessary computation.
C.2.2 Dataset Information
We select 20 datasets from UEA Time Series Classification Archieve with diverse characteristics in terms of the number, dimensionality and length of time series samples, as well as the number of classes. Detailed information is listed in Table
C.2.3 Full Experimental Results
We follow the setting of [30 to randomly drop either 30%, 50% or 70% observations. Experimental results are shown Table 9 Table 10 and Table I1] We can find that ContiFormer outperforms RNN-based methods, ODE-hased methods and Transformer-base methods. Also, we observe that ContiFormer outperforms all the baselines by large margin when 70% observations are dropped, which confirms that ContiFormer is more suitable for irregular time series modeling.
21

---

Table 8: Dataset Information of UEA.
Dataset
Ablation
# Train
# Test
Length
# Classes
# Variates
Articulary WoodReconnition
AWR
275
300
144
25
9
BasicMotioos
BM
40
40
100
4
6
CharacterTrajectories
CT
1422
1436
182
20
3
DuckDuckGeese
DDG
60
40
270
5
1345
Epilepsy
EP
137
138
207
3
ERing
ER
30
270
65
4
FingerMovements
FM
316
100
50
28
HandMoveemeniDirrctinnn
HMD
160
74
400
10
Handwriting
HW
150
850
152
26
3
Heartbeat
HB
204
205
405
61
Japanese Vowels
JV
270
370
29
12
Libras
LB
180
180
45
2
LSST
LSST
2459
2466
36
14
6
NATOPS
NATOPS
180
180
51
24
PEMS-SF
PEMS
267
173
144
963
PenDigits
PD
7494
3498
8
10
2
RacketSports
RS
151
152
6
SellRegglaatonSSCII
SR
268
293
896
2
6
SpokenArrbicDiggtss
SA
6599
2199
93
10
13
UWaveGesturelLibrary
UGL
2238
2241
315
3
Table 9: Experimental result UEA when 30% observations dropped. (ODE. stands for ODE-
RNN, N. CDE stands for Neural CDE, ACC. stands for Accuracy.)
Dataset
GRU-D
GRU-At
ODE.
CADN
N. CDE
S5
mTAAN
TST
ContiFormer
AWR BM
0.8533
0.8956
0.8789
0.8056
0.9089
0.9011
0.8067
0.9722
0.9633
0,9333
0.9417
0.8250
0.9583
0.9917
0.9833
0.9917
0.9667
0.9750
CT
0.9325
0.8558
0.9415
0.9575
0.9276
0.9610
0.9529
0.9742
0.9833
DDG
0.5867
0.5933
0.5067
0.4600
0.3400
0.5667
0.6000
0.6533
0.7267
EP
0.7995
0.8043
0.8454
0.8123
0.7657
0.9074
0.9203
0.9589
0.9324
0.7062
0.7148
0.6296
0.8623
0.7543
0.8346
0.7123
0.9160
0.9210
0.6167
0.6167
0.5667
0.6233
0.5867
0.6033
0.5433
0.6133
0.6000
HMD
0.3559
0.4099
0,5450
0.4505
0,3333
0.6171
0.4414
0.5946
0.4550
HW
0.0722
0.1251
0.0675
0.0925
0.1831
0.1302
0.0937
0.2196
0.2000
HB
0.7577
0.7593
0.7805
0.7431
0.7333
0.7431
0.7789
0.7398
0.7561
IV
0.9703
0.9550
0,9432
0.9703
0.9225
0.9676
0.9595
0.9865
0,9919
LB 0.5037
0.4741
0.5222
0.5938
0.7481
0.9216
0.5426
0.7519
0.8778
LSST
0.5589
0.5615
0.5715
0.5574
0.3940
0.6389
0.5307
0.5520
0.6004
NATOPS
0.9204
0.9315
0.8704
0.8944
0.8148
0.8500
0.9019
0.9352
0,9222
PEMS
0.7938
0.7669
0.8208
0.7958
0.7572
0,9133
0.8189
0,8420
0.8401
0.9309
0.9222
0.9433
0.9402
0.8422
0.8110
0.9029
0.9517
0.7434
0.7281
0.7654
0.7325
0.6798
0.7500
0.7325
0.8158
0.8487
SR
0.8942
0,9124
0,9101
0.8259
0.8783
0.8737
0.8646
0.9044
0.9101
SA
0.9663
0,9660
0,9830
0.9809
0.8998
0.9291
0.9321
0.9744
0,9836
UGL
0.6729
0.6625
0.6917
0.7479
0,8219
0.8042
0.7344
0.8552
0.8135
Acc. (1)
0.7284
0.7298
0.7304
0.7402
0.7142
0.7854
0.7381
0.8089
0.8126
Rank (1)
5.75
5.45
5.4
6.85
4.4
5.7
2.75
2.4
22

---

Table 10: Experimental result UEA when 50% observations dropped. (ODE. stands for ODE-RNN, N. CDE stands for Neural CDE, ACC. stands for Accuracy.)
Dataset
GRU-D
GRU-At
ODE
CADN
CDE
S5
mTTAN
TST
ContiFormer
AWR
0.8811
0.8978
0.8989
0.8067
0.9133
0.9078
0.8067
0.9633
0.9633
BM
0,9417
0.8500
0.7667
0,9417
0,9583
0.9917
0.9583
0,9500
0,9583
CT
0.8872
0.9088
0,9506
0.9798
0.9285
0.9638
0.9431
0,9601
0,9798
DDG
0.5467
0.6000
0.4933
0.5467
0.3467
0.5667
0.6067
0.6533
0.6867
EP
0.7295
0.7899
0.7512
0.6728
0.7101
0.9012
0.9106
0.9589
0,9469
ER
0.7358
0.7432
0.5704
0.8527
0.7691
0.8284
0.6432
0.9012
0.8901
0.5800
0.5967
0.5933
0.5933
0.6000
0.6033
0.5700
0.6067
0.6067
HMD
0.4099
0.4054
0.4685
0.4550
0.3243
0.5811
0.4099
0.5270
0.4595
HW
0.0769
0.0945
0.0588
0.1004
0.1855
0.1357
0.0737
0.1600
0.1757
HB
0.7642
0.7593
0.7707
0.7512
0,7333
0.7463
0.7789
0.7350
0,7610
JV
0.9604
0.9532
0,9126
0.9649
0,9180
0.9631
0.9595
0.9775
0.9856
LB
0.3796
0.4352
0.4537
0.5680
0.7315
0.6352
0.5259
0.6556
0.8537
LSST
0,5203
0,5292
0.5223
0.5222
0.3969
0.7204
0,5126
0,5151
0,5661
NATOPS
0.9185
0.9167
0.8444
0.8741
0,8019
0.8315
0.8778
0.8778
0,9222
PEMS
0.7823
0.7861
0.8170
0.7611
0.7534
0.9114
0.8015
0.7669
0,8401
PD
0.8245
0.7804
0.8563
0.8369
0.6389
0.6544
0.6898
0.8827
0.8799
RS
0,7412
0.7368
0.7127
0.7632
0.5526
0.7193
0.6601
0.7807
0.8355
0,9135
0,9170
0,9044
0.8123
0.8760
0.8885
0.8658
0.8987
0,9067
0.9583
0,9556
0.9759
0.9663
0.8907
0.9218
0.9254
0.9667
0,9767
UGL
0.6833
0.6583
0.6781
0.6531
0,8281
0.8042
0.7167
0.8490
0,8000
ACC. (1)
0.7117
0.7157
0.7000
0.7211
0.6929
0.7638
0.7118
0.7793
0.7997
Rank (1)
5,8
5,65
5.75
5.55
6.5
4.25
5.85
3.15
1.9
Table 11: Experimental result on UEA when 70% observations are dropped. (ODE. stands for ODE-RNN, N. CDE stands for Neural CDE, ACC. stands for Accuracy)
Dataset
GRU-D
GRU-At
ODE
CADN
N. CDE
S5
m'TAN
TST
ContiFormer
AWR
0.8689
0,8822
0.8433
0.7911
0.9178
0.8956
0.7956
0.9089
0.9511
BM
0.7583
0.7917
0.7167
0.9750
0.9000
0.9167
0.9417
0.9167
0.9167
CT
0.8433
0,8496
0.9471
0.9582
0.9241
0.9547
0.9009
0.9313
0.9675
DDG
0.5200
0.6000
0,4800
0.5667
0.3533
0.5667
0.6067
0.5867
0.7267
EP
0.7464
0.7222
0.6643
0.6667
0.6667
0,8802
0.9058
0.9420
0.9275
ER
0.6741
0.6617
0.5383
0,8647
0.7630
0,8025
0.7407
0.8198
0,8580
FM
0.5467
0.5933
0.5733
0.6000
0.5767
0.6133
0.5700
0.5300
0.6233
HMD
0.3829
0.3784
0.4459
0.4324
0.3378
0.5991
0.3694
0.4324
0.4144
0.0784
0.0831
0.0541
0.0957
0.1533
0.1078
0.0604
0.1255
0.1247
HB
0.7610
0.7528
0.7593
0.7610
0.7561
0.7512
0.7691
0.7593
0.7593
JV
0.9514
0,9414
0.9261
0.9685
0.8964
0,9559
0.9613
0.9595
0.9766
LB
0.3648
0,3722
0.3463
0.5412
0.6796
0.6278
0.4926
0.5500
0.7648
LSST
0.4803
0.4895
0.4505
0.5278
0.4093
0.6778
0.4992
0.4874
0.5149
NATOPS
0.8722
0.8981
0.7889
0.9074
0.7944
0.8259
0.8463
0.8315
0.9204
PEMS
0.7418
0.7611
0.7842
0.7726
0.7303
0.8902
0.7861
0.6840
0.8227
PD
0.7451
0.6670
0.7802
0.7596
0.5201
0.5197
0.6063
0.8078
0.8069
0.6711
0.6776
0.6316
0.7018
0.5570
0.6382
0.6228
0.6930
0.7697
0.9090
0.9170
0.9158
0.8515
0.8726
0.8874
0.8635
0.8612
0.8953
0.9304
0.9295
0.9523
0.9513
0.8622
0,9024
0.8925
0.9447
0.9600
UGL
0.6031
0.6219
0.5896
0.6729
0.8344
0.7896
0.6792
0.8219
0.7969
ACC. (t)
0.6725
0.6795
0.6594
0.7183
0.6753
0.7401
0.6955
0.7297
0.7749
Rank (L)
6.15
5.7
6.4
3.9
6.3
4.45
5.4
4.2
2.1
C.3
Predicting Irregular-Sampled Sequences
C.3.1 Background: Marked Temporal Point Process (MTPP)
The marked Temporal Point Process (MTPP) has been widely studied in recent years. We define sequence of events and types as r = [41, k1), (l2, kz),... ..., (lN. kN)]. where 4 is the time point when the i-th event occurs, with 4 < l2 <... < IN. k; E 21,2,..., K.). is the type of the event and KC is the total number of event types. A history event is the event sequence up to but not including
23

---

time t, denoted by H4 = (t,, k5):k)t;t, <t. The goal of the temporal point process is to model the conditional intensity function (CIF), denoted by (t),,, which represents the likelihood an event occurring at time I, given the historical events. Models are trained so that the log probability of an event sequence, i.e.,
log X3, (4) - I,A (T())dT.
log =
(45)
C.3.2 Dataset Description
We use one synthetic dataset and five real-world datasets, namely Synthetic, Neonate [50] Traffic MIMIC [16), BookOrder [66 and StackOverflow [33] evaluate our model.
32
Synthetic To evaluate the effectiveness of the proposed method, we create synthetic dataset. Specifically, we consider 10 types event where each type of event contain three properties, i.e. (X,, V,, Dz). Hence, given an event e with type is represented as e = (e.rr, e.v, e.d, e.k), where e.r: (X (1)) 6.v NV,,, (22), e.d = Dj > 0 and e.k the type of the event. We set M11 = #2 0.01 and V, E R00 is randomly generated and pre-defined.
To model both the correlation between events and the dynamic changes of events within the system. We first define the dynamic attention (weight) for each event, which is composed of inter-event dependency modeling and intra-event dynamic modeling. For inter-event dependency, it aims to calculate time-aware attention (weight) between each pair of events. Specifically, use the following equation to define the weight between event and é.
w (e.t ê.f). (e.r * ē.v ê.x
(46)
Next, the define \ī and X (t) in which we assume that only the last event attributes to the decision of the next event, following Markov decision process (MDP), i.e.
( (-(-ee))) e.l)
(t) P(klê.k) Softplus
Σ NeEH.
w(e,)) exx
e.d
(47)
Σ ((e è) exp (=(t-et)) e.d
(t) = Softplus
eE
where exp (-(t-e.t) e.t)/e.d) is the exponential decay effect for each event, è is the last event of time t. The existence of the exponential decay function ensures that all the events will eventually disappear in the system time goes to infinity to stabilize the conditional intensify function. P(k]k) is pre-defined matrix that means the probability of the next event with type & given that the type of the previous event is é.
Next, predict the occurrence time of the next event with
T,+1 = 1 t.p (t H4) dt,
where p( H() x²)1) exp (- [-
(-f( (((((0a)
(48)
To predict the type of the next event, we sample the type of the next event from P(-Jē.k), where ē.k the type of the previous event.
Finally, we use 500 different random seeds to generate a total of 500 event sequences. Also, we use 4-fold cross-validation for evaluation. The generation process is shown in Algorithm [
Neonate. Neonate dataset is a clinical dataset from 79 patients, among which patients have incurred at least two seizures during the observation. The dataset contains only one type of event, ie.. the start point of seizure (in minutes). The task is to predict when the next seizure will happen. Note that we only clear the start point of the seizure. Thus, we only have one type of event.
24

---

Algorithm 1 Generate Synthetic Dataset with Time-Aware Dynamic Kernel
Input seed E [0. 500), Imax = 20, X = [0.78, 0.11, -100, -..6, 0.778, 1.0, 0.56, -0.33. 0.3, -0.111, V = 0-1. 0.5, -0.28, 0.28, 0.17, -.005 0.05, 0.39 005]. D = [0.5, 0., 0.28, 0.17, 0.22, 0.44, 0.33, 0.11, 0.05, 0.39)
Output A generated event sequence E.
Initialize the random state of Numpy using seed and defined the probabilistic matrix P. Initialize the event sequence E' be an empty array, and set current time = 0. Random generate the type of the first event e.k and sample e.z, e.v and e.d. Add the event to the event sequence. while I maa do
Calculate the conditional intensity function using Eq. (7
Predict the time of the next event e.l using Eq. (48)
Assume the type of previous event is k, sample the type of the next event e.k from P(-lk).
Sample sample e.Z, e.u and e.d given e.k. Add the generated event to the event sequence E. Update T = e.t. end while return E
Traffic. Traffic dataset is a benchmark dataset for time-series forecasting collected by the Caltrans Performance Measurement System (PeMS). This dataset contains the traffic state monitoring of 861 sensors in a two-year period. The data is collected every hour. We extract time points with fluctuations higher than 50% events, whose types are up and down according the direction. We consider two events that are within 6 hours as duplicated events and remove the latter one from the dataset. Besides, for each sensor, it contains long sequence, we partition these event sequences by month, i.c., for each event sequence, it contains the event of a sensor in one month.
MIMIC. MIMIC dataset is a clinical dataset that collects patients' visits to hospital's ICU in seven-year period. We treat the visits of each patient as a separate sequence.
BookOrder. BookOrder dataset is a financial dataset that contains transaction records of stock in one day. We record the time (in milliseconds) and the action that taken in each transaction. The dataset is a single long sequence with only two types of events: "buy" and "sell". We clip the event sequences that the maximum length is 500 in both the train and test sets. Furthermore, the mean and standard derivative a of the time difference between two events in the dataset is 1.32 and 20.24 respectively. Therefore, clip the maximum time difference to 70.
Overffo. StackOverflow dataset is a question-an -anwwerng website. The website rewards users with badges to promote engagement in the community, and the same badge can be rewarded multiple times the same user. We collect data in a two-year period, and we treat each user's reward history as sequence.
For MIMIC, Bookorder, and SterkOverflow datasets, we use the dataset from the Google Drive Statistic information can be found in Table I2 We use the 4-fold cross-validation scheme for Synthetic, Neonate, and Traffic datasets following [17, and the 5-fold cross-validation scheme for the other three datasets following [39 64
C.3.3 Implementation Details
Input Embedding Given the history event H, (tty,ky) :t t) with total of K. events, the input of Transformer-based model and RNN-based model is obtained through event embedding and temporal encoding, i.e., for the i-th event x; emb(k) + enc(t), where emb(k) is the k-th column of the embedding table & &xd and temporal encoding is defined by combination of sine and cosine function 64
http::/:wwwwpp/o/o/o///////////t 0 D-Or1U87jycim-doe 4WWn55544
25

---

Table 12: Dataset statistics: of the dataset, number of event types, number of events in the dataset, average length per sequence, and number of sequences training/test sets.
Dataset
# Events
Avg. Length
# Train Seqs
Test Seqs
Synthetic
10
8, 618
17
375
125
Neonate
534
38
10
Traffic
2
154, 747
60
1, 138
647
MIMIC
2, 419
527
65
BookOrder
414, 800
2, 074
90
100
StackOverflow
480, 413
72
4, 777
1,626
Conditional Intensity Function Given the output from the model, i.e. h (tz), (t2) (t The dynamics of the temporal point process are described by a continuous conditional intensity function. To reduce the memory cost and accelerate the training process. Instead of using the continuous-time output from ContiFormer, we only use ContiFormer to generate hidden repre- sentation for discrete time points, i.e., b (2 ... Li the,., IN. Following 644, the conditional intensity function is defined by
A (t H.)
Σ d (4)
(49)
where each of the type- specifcc conditional intensity function is defined by
Au Hi) fu an t- 1j
w. h (t2) + by
(50)
where on interval E [t,, 1) and fu (2) = Be log (1 + cxp (x/ (x)) is the softplus function with parameter Se. The of such a closed-form intensity function allows for O(1) interpolation.
Training Loss We train model and all the baseline models by jointing maximizing the log- likelihood of the sequence and minimizing the prediction losses.
Specifically, for sequence S over an interval [t,, tx], given the conditional intensity function X (t H4), the log- likelihood given by
/' x (4l(He) dt.
(LL(S)
log X (t, HH2
(51)
non-event log- lig-likeldhood
lig-likeldhood
One challenge to compute the log-likelihood is to solve the integral, i.e., A = - A H.) dr. Since softplus function has no closed-form computation, we apply the Monte Carlo integration method, i.e.,
P
ÂMC
Σ (t, t--1 j=2
(52)
where uq Unif(ty-1,tt) for [1, ..., P] sampled from a uniform distribution over an interval [fg-
Besides the log-like rigeeikel loss, also use the cross-entropy loss and regression loss for predicting the type and time of the next events.
Specifically, given the conditional intensity function, the next time stamp prediction is given by
(t Ho) = 2 (4) exp X (H+)dr
(53)
8jt1 1 t. ptp Hi) dt.
26

---

Thus, the regression loss and the cross- eros-py loss are given by
N
Akz tt
(S) - (t. t2)² (prea(S)
log
(54)
(ts)
==2
1=2
Overall, the modeling is trained in a multi-task manner, i.e., E(S) = fLL. + onlreg(()) + a22
(55)
where di and 02 are the pre-defined weight. By default, we set 0 0.01 and ag = 1.
C.3.4 Significance test on Synthetic and BookOrder datasets
To demonstrate the statistical superiority of ContiFormer, we conducted a t-test on Synthetic and BookOrder datasets using 10 random seeds. The t-test results are listed in Table
Table 13: Significance test on Synthetic and BookOrder dataset with 10 repeats. The P-value represents the significant value between the ContiFormer and the baseline model. Bolder values represent P-value < 10 in significance test.
Datuset
Synthetic
BookOrder
Metrie
LL
Accuracy (7)
RMSE (1)
LL (T)
Accuracy (f)
RMSE (
Statistic
Mean
P-value
Moan
P-value
Mcan
P-value
Moan
P-value
P-value
P-value
GRU--tt
-0.8611
5 3.385-08
0.8420
0,001
0.2556
3,49E-09
-1.0137
1.07E-31
0.6274
3.118-08
3.0680
1.326-05
NSTKA
-1.0088
0.8419
1.74E-12
2,95E-27
5.48E-15
LS8E-08
ODE-RNN
-0.4322
2.39G-11
0.8420
0.098
0.2401
3.77D-08
.0.4588
1.44E:26
0.6278
1.17E.07
3.5898
1.94E-03
RMIIP
-1.1132
3.57E-19
0.8420
0,0143
0,3711
6,70E-18
-0.9551
9.70E-26
0.6245
1455-13
3.6475
9.76E-04
2.370-10
1.83--07
1.45E-13
9.55E-07
THP
-0.6038
6.28E-06
0.8420
0.1354
0.2100
4.85E-06
-1.0788
1,556-17
0.6251
1.96E-06
3.6907
8.026-07
ConteFormer
.0.5445
0.8421
N/A
0.1943
4.2745
N/A
0.6295
N/A
3.6171
N/A
C.4 Regular Time Series Forecasting
Time series forecasting has garnered considerable attention recent years, as evidenced by various studies [35] 53 61. In this study, we assess ContiFormer against several models tailored for regular time series forecasting.
Datasets We evaluate ContiFormer five experimental datasets. i) ETT dataset 66 contains the data collected from electricity transformers, including load and oil temperature that are recorded every 15 minutes between July 2016 and July 2018. Here, we use ETTm2 and ETTh2 datasets. ii) Exchange dataset (53) records the daily exchange rates of eight different countries ranging from 1990 to 2016. iii) Weather dataset is recorded every 10 minutes for 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity, etc. v) ILI dataset includes weekly recorded influenza-like illness (ILI) patients data from Centers for Disease Control and Prevention of United States between 2002 and 2021.
Baselines We compare ContiFormer with several transformer-based models for regular time series forecasting under the multivariate setting. These transformer- models includes: FEDformer (62), Autoformer (53,, Informer [61] and Reformer 331] Also, we chose one recent state-of-the-as model. i.c. TimesNet for comparison.
Results of Time Series Forecasting The experimental results depicted in Table 14 ContiFormer achieves the best performance on ILI dataset. Specifically, gives 10% MSE reduction (2.874 - 2.632). Also, ContiFormer outperforms Autoformer and FEDformer on Exchange, ETTm2, and Weather datasets. Therefore, it demonstrates that ContiFormer performs competitively with the state-of-the-ar model.
C.5 Pendulum Regression Task
The objective of pendulum regression is to assess the models' capability in accommodating ob- servations obtained at irregular intervals [45 49). The input sequence comprises 50 images, each
27

---

Table 14: Results of regular time series forecasting. We set the input length as 36 for ILI and 96 for the others. Avg. stands for Average. Bold values indicate the best performance and underlined values
indicate the second-best performance (lower the better).
Models
ComiFormer
TimesNes[52]
DLincar
56]
FEDformer [2)
Autoformer
53
Transformer 51
Informer [1
Reformer 31
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
96
0.105
0.232
0.107
0.238
0.088
0.217
0.153
0.284
0.153
0.283
0.89]
0.743
1.348
0.966
1.208
0.961
ET7m2 Exchange
192
0.215
0.339
0.201
0.527
0.175
0.313
0.261
0.375
0.327
0.113
1.338
0.955
1.296
0.941
1.329
1.001
336
0.336
0.427
0.354
0.435
0.310
0.421
0.437
0.485
0.555
0.560
1,553
1,086
1.641
1.072
3.023
1.427
540
0.698
0.614
0.582
0.534
0.555
0.710
0.648
1.515
L.012
2,430
1.801
1.159
Avg.
0.339
0.399
0.320
0.395
0.277
0.377
0.392
0.446
0.41
0.177
1.325
0.919
1,679
1,066
1,841
1.137
24
0.126
0.226
0.104
0.200
0. 0.109
0.213
0.149
0.253
0.157
0.262
0.282
0.397
0.391
0.475
0.405
0.229
48
0.175
0.276
0.143
0.238
0.145
0.249
0.172
0.268
0.177
0.256
0.501
0.535
0.420
0.484
0.541
0.569
168
0.336
0.255
0.308
0.262
0.345
0.255
0.324
0.824
0,730
1.068
1.149
0.846
336
0.331
0.361
0.352
0.359
0.366
0.111
0.326
0.361
0.328
0.365
1.149
0.871
1.379
0.983
2.258
1.153
Ang.
0.226
0,300
0.214
0.276
0.220
0.305
0.226
0.300
0.231
0.307
0.689
0.633
0.815
0.705
1,088
0.767
24
0.237
0.327
0. 196
0 1.281
0.179
0.277
0.262
0.340
0,266
0.343
0.946
0,800
1,408
1,028
0.762
0.703
ET112
0.287
0.355
0.244
0.317
0.247
0.329
0.291
0.358
L,009
0.826
1.642
L.100
1.101
0.856
168
0.117
0.126
0.397
0,409
0,430
0.118
0.103
0.120
0.403
0.119
3.014
1,483
3,26]
1.511
2.076
1.144
336
0.484
0.471
0.446
0.449
0.577
0,532
0.448
0.458
0.445
0.456
3.357
L,006
3.155
1.481
2,633
1.297
Avg.
0.356
0.395
0.321
0.364
0.358
0.396
0.351
0.393
0.352
0.394
2.081
1.179
2.366
1.280
1.643
1.000
2.391
1,044
2,925
1.061
4,412
1,576
4,979
16688
4.334
1,527
4.789
1.431
5.718
L.623
4.678
1.455
36
2,673
1, 1.006
1 144
1.122
4,314
1.529
4.812
1.628
4.084
1.438
4.995
1.474
5.383
1 1.664
4.801
1,490
111
-88
2.536
0039
2716
1.027
3.910
1.424
4,112
1 1,465
3,870
1.408
5.283
1.545
5.120
1 1.537
4.946
1,510
60
2,930
L.117
2.711
1.049
4,114
1.446
3.965
1.428
3.895
1.411
5.454
1,578
5.556
1.613
5.134
1.545
Avg.
1.44
2.871
1,066
4.188
1.193
1,167
1.547
4.016
L.166
5.130
1.507
5.437
1.581
4,890
1,500
96
0.170
0.218
0. 0.171
0.221
0.194
0.250
0.254
0.523
0.253
0.317
0.539
0.539
0.509
0.541
(1944))
0.675
Weather
192
0.229
0.271
0.20
0.269
0.237
0.297
0.287
0.344
0.293
0.347
0.738
0.613
0.459
0.455
1.061
0.735
336
0.288
0.317
0 289
0.302
0.282
0.334
0.336
0.374
0.346
0.350
0.464
0.466
0.575
0.495
1.201
0.802
510
0.345
0.348
0.325
0.365
0.384
0.400
0.379
0.391
0,909
0.700
0.862
0.625
0.958
0.702
Avg.
0.257
0.200
0
0 0.285
0.260
0.311
0.315
0.360
0.318
0.359
0.672
0.579
0.624
0.529
1.040
0.729
measuring 24 by 24 pixels, and is afflicted by correlated noise. These images are sampled at irregular
intervals from a continuous trajectory spanning duration of 100 units, while the targets correspond the sine and cosine values of the pendulum's angle. It's important note that the pendulum follows a nonlinear dynamical system, with the velocity remaining unobserved. Table I5 summarizes the results of this experiment. ContiFormer outperforms CRU and ODE-RNN.
Table 15: Regression MSE x10 (mean + std) on pendulum regression task.
Model
Regression MSE (x10-3)
mTAN 48
65.64 (4.05)
ODE-RNI 43
7.26 (0.41)
CRU 45
4.63 (1.07)
S5
40
3,41 (0.27)
ContiFormer
4.21 (0.24)
D Ablation Study and Parameter Study
D.1 Effects of Different Attention Mechanism
We study the impact of different attention mechanisms to show the significance of both continuous- evolving attention scores and continuous value functions. We claim that both of these two continuous designs attribute to the modeling of complex dynamic systems. We conduct the experiment on 6 datasets for event prediction task. As shown in Table. [6] ContiFormer outperforms both of the two variants, which shows that jointly modeling the dynamic relationship between input observations and meanwhile capturing the dynamic change of the observations is critical for modeling complex event sequences in reality.
D.2 Effects of Different Numeric Approximation Methods
As discussed in Appendix A.2 linear interpolation and Gauss-Legendre Quadrature are two methods for approximating an integral in close interval [1,, We study the impact of these two methods and set P. i.e., the number of intermediate steps for integral approximation, in the Gauss- Lauss-Legendre quadrature from {2, 3, 4, 5]. We conducted the experiment on 6 datasets for event prediction task.
28

---

Table 16: Effects of different attention mechanism design event prediction task. Attn, stands for Attention. refers to continuous version attention/value, while refers to a static version. is important to note that when both attention and value static, the model is equivalent to Transformer. Conversely, when both are continuous, the model is referred to as ContiForrmrr.
Dataset
Synthetic
Neonate
Traffic
Attn.
Value
T.L. (t)
ACC (t)
RMSE (1)
(t)
RMSE (.)
LL. (1)
ACC (t)
RMSF (L)
-0.589
0.841
0.205
-2.702
9.471
0.569
0,818
0.332
x
-0.576
0.842
0.202
-2,634
9.249
0.598
0.819
0.327
-0.563
0.842
0.200
-2.590
9.267
0.544
0.814
0.329
-0.535
0.842
0,192
-2.550
9.233
0.635
0.822
0.328
Dataset
MIMIC
BookOrder
StackOverflow
Attn.
Value
(1)
ACC (t)
RMSE (J)
ACC (1)
RMSE (1)
I1,
ACC (1)
RMSF (L)
x
x
-1.137
0.834
0.843
-0.302
0.622
3.688
-2.354
0.468
0.951
x
-1.147
0.832
0,837
-0.312
0.627
3.690
-2.354
0.468
0.950
-1.137
0.837
0.838
-0.272
0.629
3.632
-2.337
0.472
0.948
-1.135
0.836
0.837
-0.270
0.628
3.614
-2.334
0.472
0.948
Table 17: Effects of difference numerical approximation methods event prediction task. (default
to linear approximation instead of the Gauss-Legendre quadrature method).
Dataset
Synthetic
Nconate
Traffic
P
LL (t)
ACC (t)
RMSE ())
LL
RMSE ())
LL (?)
ACC (?)
RMSE (1)
Approxiamation Method Linear
-0.535
0.842
0.192
-2550
9.233
0.635
0.822
0.328
Approxiamation Method Gauss- Gauss-Legendre Quadrature
-0,545
0.842
0.198
-2.571
9.235
0.599
0.817
0.330
-0.531
0.842
0.194
-2.594
9.236
0.610
0.820
0.328
4
-0.556
0.842
0.199
-2.590
9.253
0.589
0.819
0.328
5
-0,550
0.842
0.193
-2.602
9.258
0.601
0.820
0.328
Dataset
MIMIC
BookOrder
StackOvertlow
P
(T)
ACC (t)
RMSE ())
LL (1)
ACC (t)
RMSE (4)
LL
ACC (t)
RMSE (L)
Approxiamation Method Linear
-1.135
0.836
0.837
-0.270
0.628
3.614
-2.332
0.473
0.948
Approxiamation Method Gauss- Legendee Quadrature
-1.144
0.834
-0.291
0.627
3.653
-2.337
0.472
0.948
-1.142
0.837
0.839
-0.294
0.627
3.667
-2.337
0.471
0.948
4
-1.143
0.837
0.839
-0.287
0.627
3.647
-2.340
0.471
0.948
5
-1.132
0.834
0.838
-0,290
0.628
3.563
-2.338
0.471
0.948
The experimental results can be found Table 17 sensitivity to the choice of approximation methods.
As shown the table, our model exhibits low
D.3 Effects of Tolerance Errors in ODESolver
ODE solvers play a crucial role in ensuring that the model's output remains within a specified tolerance of the true solution [.] By adjusting this tolerance, we can observe changes in the behavior of the neural network. Given our use of a fixed-step ODE solver, we sought to investigate the impact of the step size on prediction performance throughout the experiment. Specifically, we explored different step sizes from the set (0.5, 0.1, 0.05, 0.01, 0.005). :. A smaller step size corresponds to greater number of forward passes required to solve the ordinary differential equations, resulting in more accurate solutions at the expense of increased computational time. We conducted extensive experiments on 6 datasets for event prediction task. The experimental results are presented Table 18 The table reveals that our model demonstrates low sensitivity to the tolerance error. Therefore. we set the tolerance error to 0.1 order to make desirable balance between computational time and prediction accuracy. We explain the phenomena in two aspects. First, our framework, similar to the
29

---

Table 18: Effects of tolerance errors in ODESolver event prediction task (default to 0.5 for Neonate dataset and 0.1 for other datasets). (The use of a double dash (-) indicates that the experiment requires an excessively long duration of time.)
Dataset
Synthetic
Neonate
Traffic
(?)
(1)
RMSE (U)
(t)
RMSE (L)
LL)
ACC
RMSE (U)
5c-1
-0.559
0.841
0.196
-2.550
9.233
0.614
le-1
-0.535
0.842
0.192
-2.561
9.229
0.635
0.822
0.328
5e-2
-0.549
0.842
0.195
-2.568
9.236
0.620
0.819
0.329
le-2
-0.539
0.842
0.195
-2.573
9.233
0.619
0.820
0.329
5e-3
-0.549
0.842
0.199
-2.559
Dataset
MIMIC
BookOrder
StackOverflow
Stepsize
LL (1)
ACC (1)
RMSE (1)
LL (1)
ACC (1)
RMSE (4)
LL (t)
ACC (1)
RMSE (4)
5c-1
-1.138
0.837
0.837
-0.268
0.629
3.619
-2.333
0.473
0.948
le-1
-1.135
0.836
0.837
-0.270
0.628
3.614
-2.32
0.473
0.948
5e-2
-1.137
0.835
0.837
-0.263
0.629
3.609
-2.332
0.473
0.948
le-2
-1.143
0.836
-0.263
0.629
3.612
5c-3
-1.147
0.837
0.838
Transformer architecture, circumvents cumulative errors by eliminating the necessity of sequentially passing neural networks in regressive manner. Second, since the output from the attention module actually a weighted sum of the tokens, the total variance is lowered. For instance, assume that X1, X2, X2, o² is N: and the variance is significantly lowered given the large Overall, our model is not sensitive to tolerance error, which makes our model robust to different scenarios.
XN XN (N, o²). then the total variance the mean value ã x (X1 + X2 + -xx)
D.4 Effects of Vector Fields in ODE
Neural ODEs controlled by vector field that defines the change of over time, i...
dx f(x,t)
(56)
dt
where f Rd + Rd. We investigate two types to implement Eq. (I.
Specifically, we denote Concat as
f(x, Actfn Actfn(Linearr (Lineard + Linear (.neear-d(()) (()))
(57)
and ConcatNorm
f(x.t) Actfn(LN((innatt (Linear" d.d (x) + Linear (t))))
(58)
where Actfn(-) is either tanh or sigmoid activation function, Lincar (.) R" + R is a linear transformer from dimension dimension b, LN denotes the layer normalization. We conduct the experiment on datasets for event prediction task. The experimental results are shown in Table 19 As shown in the table, incorporating normalization before the activation function leads to improved predictive performance.
D.5 Effects of Dropout Rate
Dropout in the Transformer can prevent overfitting and boost performance. However, with the increment of the dropout rate, we output of ContiFormer tends to be less continuous, leading to sub-optimal solution for tasks that prefer a continuous-time output. To this end, we conduct experiments on both modeling continuous- time function and event prediction tasks. As for the task of modeling continuous-tin "almost" continuous and smooth. As shown in Figure 5 the performance of both Transformer and ContiFormer drops significantly as the dropout rate increases. For prediction tasks, the dynamic system can be shaky, which reflects noise or other uncontrolled influences the system. The experimental results on event prediction can be found in Table 20 As shown the table, the dropout rate has low sensitivity to the prediction results.
function, it is desirable have dynamic system where outputs are
30

---

Table 19: Effects of different vector field designs event prediction task. Atn. stands for Attention. refers to ConcatNorm, while refers to Concat.
Dataset
Synthetic
Neonare
Traffic
Norm
Actin
LL (1)
ACC
RMSE (L)
LL (f)
RMSE (L)
LL)
ACC (t)
RMSE (J)
x
tanh
-0.552
0.842
0.198
-2.550
9.233
0.639
0.820
0.329
sigmoid
-0.561
0.842
0.201
-2.602
9.263
0.635
0.822
0.328
lanh
-0.556
0.842
0,199
-2,584
9.242
0.628
0.820
0.328
sigmoid
-0.535
0.842
0.192
-2.601
9.232
0.616
0.819
0.329
Dataset
MIMIC
BookOrder
StackOverflow
Norm
Actfn
LL (t)
ACC (f)
RMSE (U)
LL(1))
ACC (?)
RMSE (U)
(t)
ACC (1)
RMSE ())
x
tanh
-1.162
0.831
0.837
-0.274
0.627
3.615
-2.337
0.472
0.948
sigmoid
-..44
0.836
0.838
-0.270
0.628
3.614
-2.344
0.472
0,948
tanh
-1.157
0.833
0.840
-0.307
3.681
-2.335
0,947
sigmoid
-1.135
0.836
0.837
-0.302
0.625
3.689
-2.332
0.473
0.948
Interpolation results with a 0.02
Interpolation results with a 0.2
Interpolation results with
0030
RMSE
RASSE
RMSE
8.00
0.15
0.02
0010
Noural
0.01
Dropout Rate
0.0
Dropout Rate
2-3
0.
86
63
Dropout Rate
0.1
Figure 5: Interpolation results on 2D spiral under different dropout rates.
D.6 Effects of Time Normalization
As shown in Eq. and Eq. (5). due to the nature of sequence data, is common encounter abnormal points, such as events with significantly large time differences. To avoid numeric instability during training, we divide the integrated solution by the time difference. To examine the effect of normalization, we conduct experiments on 6 datasets for event prediction task. The experimental results can be found in Table 2T which shows that the normalization is significant most datasets, including Synthetic, Neonate, MIMIC and StackOverflow datasets.
E
Time Cost v.s. Memory Cost v.s. Accuracy
Time Input
Number of Forword Paxses
GID-ENN narm
TST
OUS teto nomm
Cortiformer
20000
0DE RMN (wi nom)
Memory (GB)
OCE RNN: (wr nom)
S5
25000
cOc RKN (wio torm)
DCC RAN (Aro nor))
20000
Time
Ingut Length
Imput Length
Length
Figure 6: Time Cost v.s. Memory Cost v.s. Accuracy.
Utilizing continuous-tin modeling in ContiFormer often results in substantial time and GPU memory overhead. have conducted a comparative analysis ContiFormer's actual time costs in comparison to Transformer-hased models (TST), ODE-based models (ODE-RNN, Neural CDE), and SSM-based models (S5). Throughout our analysis, we employ an RK4 solver for ODE-based models and our model, setting the step size to 0.1 and 0.01. It's worth noting that, for ODE-RNN and other
31

---

Table 20: Effects of dropout rate in ContiFormer event prediction task (default to 0.1).
Dataset
Synthetic
Neonate
Drupout
(1)
ACC (1)
RMSE (U)
Lt(
RMSE (L)
LL (7)
ACC (1)
RMSE (4)
-0.557
0.842
0.191
-2.588
0.633
0.824
0.328
0.1
-0.535
0.842
0.192
-2.550
9.233
0.635
0.822
0.328
0.3
-0.644
0.842
0.223
-2.583
9.242
0.577
0.814
0.332
0.5
-0.696
0.841
0.230
-2.579
9.280
0.525
0.813
0.331
Dataset
MIMIC
BookOrder
StackOverflow
Dropout
(t)
ACC (t)
RMSE (L)
I.
ACC (t)
RMSE (L)
IT.
ACC (t)
RMSE (L)
-1.193
0.825
-0.268
0.629
3.616
-2.338
0.472
0.1
-1.135
0.836
0.837
-0.270
0.628
3.614
-2.332
0.473
0,948
0.3
-1.142
0.833
0.836
-0.266
0.630
3,608
-2.336
0.472
0.948
0.5
-1.149
0.834
0.844
-0,270
0.629
3.624
-2.342
0.471
0,948
Table 21: Effects normalization on event prediction task. Attn. stands for Attention. refers to normalizing the attention/value as shown in Eq. /Eq. (5), while X refers to removing the normalization, i.e. 14. Note that ContiFormer use both normalization for attention and values.
Dataset
Synthetic
Neonate
Traffic
Attn.
Value
LL (f)
ACC (?)
RMSE ())
LL
RMSE ())
LL (1)
ACC (t)
RMSE (U)
-0.618
0.842
0.215
-2.612
9.277
0.721
0.822
0.329
-0.601
0.841
0.206
-2.576
9.240
0.536
0.814
0.336
-0.544
0.842
0.205
-2,601
9.272
0.576
0.818
0.327
-0.535
0.842
-2.550
9.233
0.635
0.328
Dataset
MIMIC
BookOrder
StackOverflow
Attn.
Value
LL (t)
ACC (f)
RMSE (4)
LL (1)
ACC (t)
RMSE (L)
LL (1)
ACC (f)
RMSE (4)
x
-1.167
0.826
0.848
-0.311
0.627
3.710
-2.346
0.470
0.948
-1.149
0.833
0.840
-0.269
0.630
3.634
-2.341
0.471
0,948
-1.172
0.826
0.844
-0.277
0.629
3.654
-2.336
0.472
0.948
-1.135
0.836
0.837
-0.270
0.628
3.614
-2.334
0.472
0.948
ODE-based models, the input times have impact the number of forward passes when using fixed-step ODE solver, thus, affecting the time costs.
The left figure in Figure illustrate the influence of input times concerning input length. For the model with normalization (a.k.a., w/ norm), the input times consistently range from [0, 1]. regardless of the input length. Conversely, for the model without normalization (a.k.a., w/o norm), the input times span from [0, L], where represents the input length.
As depicted in the middle figure in Figure our model performs comparably to ODE-RNN (w/ norm) when the input length is up 100, but becomes approximately four times slower as the input length extends to 1000. The right figure in Figure dispplay GPU memory usage, revealing a significantly higher memory cost associated with our model.
Table 22 provides more comprehensive overview of different models on UEA datasets. is evident that model strikes better balance between accuracy and time cost. Specifically, our model achieves the best performance on 3 out of 6 selected datasets while incurring only twice the time cost compared to S5.
32

---

Table 22: Time Cost v.s. Accuracy for different models on UEA classification datasets (0% dropped). For all ODE-ODEEa based models and our model, we adopt the RK4 solver and step size refers to the amount of time increment for the RK4 solver. (L) indicates the higher (lower) the better. Note that since the results are aggregated over 3 random seeds, achieving the same results doesn't 't indicate the same outcome for each random seed.
BasicMotions
Japanese
Libras
Model
Step Size
Time Cost ())
Accurncy (*)
Time Cost ())
Accuracy (1)
Time Cost (1)
Accuracy (1)
TST
0.56x
0.975
0.10x
0.986
0.36x
0.843
1x
0.958
1x
0.926
0.655
ODE-RNN (w/o norm)
0.1
3.06x
0,917
0.79x
0.952
2.67x
0.602
18.94x
5.26x
0.955
x
0.611
ODE-RNY RNN (w/ norm)
1.46x
1,000
0.980
0.637
0.01
1.59
1.000
0.47x
0.980
1.59x
0.619
Neural CDE
x.47
0.958
1.16x
0.932
3.87x
0.763
0.01
34.26
0.958
9.57
0.942
28.48x
0.744
ContiFormer
0.1
2.89
0.975
0.49x
0.990
2.26x
0.870
13.75x
1.87x
Dataset Model
NATOPS
PEMS-SF
RacketSports
Step Size
Time Cost
Accuracy (*)
Cost (U)
Accuracy (f)
(4)
Accuracy
TST
0.32x
0.963
0.15x
0.890
0.14x
0.826
1x
0.839
1x
0.896
0.772
2.67
0.778
0.827
ODE-RNN (w/o norm)
0.01
16.52
0,907
28.45x
0.761
x
0.796
0.1
1.28
0.898
1.47x
0.775
1.15x
0.785
ODE-RNN (w/ porm)
1.45x
0.898
1.4xx
0.775
1.57x
0.787
Neural CDE
0.1
3.83x
0,789
6,01x
0.763
3.48x
0.743
0.01
29.24x
0.787
50.36x
0.763
25.04x
0.748
ContiFormer
2.32x
0.935
5.45x
0.823
1.87x
0.836
0.01
9.76x
0.935
37.08
0.823
5.88X
0.836
Broader Impact
The proposed model, ContiFormer, presents several potential impacts within the field of machine learning and irregular time series analysis. These include:
Enhanced Modeling Capability: ContiFormer extends the Transformer architecture to effectively capture complex continuous-time dynamic systems. By leveraging continuous-time modeling techniques, the model demonstrates improved performance in handling irregular time series data, offering a more flexible and powerful modeling ability.
Practical Applications: Our proposed method achieves state-of-thh of-thh--rttttttt performance in a variety of tasks and real-life datasets, which makes it more promising to tackle real-world applications. The ability to model continuous-time dynamics offers valuable insights into time-evolving systems, facilitating improved decision-making and predictive capabilities in diverse domains.
33


Reviews:
[{"review_id": "dqrDxTl49V", "reviewer": "Reviewer_2E9G", "summary": "This paper introduces ContiFormer, a continuous time transformer-based model that leverages parallelism and can handle irregularly sampled data well, thereby removing the need to transform these datasets into discrete uniform bins. This set-up incorporates the continuous dependence on the data from differential equation based neural networks. Finally, an extensive range of experiments on irregularly sampled data are performed to show promising results for ContiFormer.", "questions": "Since ContiFormer requires $N^2$ ODEs to be solved, how does the $P$ relate to $L$ in Table 2? Are these on similar orders?\n\nIt seems like the neural ODE performs much worse than expected even for the interpolation compared with the example we see on the diffrax website https://docs.kidger.site/diffrax/examples/neural_ode/. Is this because of the regular vs irregular sampling issue?\n\nFigure 4 in appendix does not seem to be plotting the same curve at each row? Neural ODE clearly is on a different scale or a different curve in the third row quite clearly?\n\nMinor point: Line 119: donated -> denoted", "limitations": "Limitations were not discussed in the paper.\n\nGiven that the model makes use of cubic splines, the literature suggests this is will not be causal. Performance on regularly sampled dataset not clear.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper presents an original architecture class that can encompass many existing models. The discussion on attention combined with the continuous time set-up is limited and therefore this paper will be a significant contribution in this area. The paper is generally well-written and extensive numerical experiments were performed.", "weaknesses": "Whilst in general a fairly well-written paper, key parts of the model can be made clearer. For example, it is not clear for line 137 how the key and values are initialised. There is also an assumption that is not explained \"we assume that every observation has influence on the dynamic system even before its occurrence.\" Combined with the fact that the query uses natural cubic spline, does that mean the model is not causal and cannot be applied in an online fashion (see for example [Morrill et al 2022 On the Choice of Interpolation Scheme for Neural CDEs])?\n\nThis paper does not appear to be comparing with the state-of-the-art models. To my knowledge, the S5 model (Simplified State Space layers for sequence modeling by Smith et al 2022), appears to already outperform quite a few of the baselines chosen in this paper, specifically, mTAN, ODE-RNN, GRU-$\\Delta$t for irregularly sampled pendulum exercise. Therefore this should be added into a benchmark. The pendulum regression task used in Smith et al. seems to be used in quite a few irregular sampling task papers, and therefore it would be of interest to see how the ContiFormer performs in this case.\n\nWhilst there is an extensive range of experiments, the full results with the standard deviation across the 3 repeats for Section 4.2 and Section 4.3 seem to be missing. These seem quite key to back up some results, particularly for Table 4, where the standard deviation is referred to. Without this, it is difficult to quantify \"overall statistical superiority of ContiFormer\" as claimed.\n\nThere does not appear to be any evaluation on regularly sampled datasets. Whilst I appreciate that the marketed strength of this method is in irregularly sampled data, it would be useful to see what the performance is on regularly sampled datasets (against benchmarks) and whether one can just choose this model to be applied on all timeseries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your comprehensive comments.\n> **W1: L137 how key & values initialised.**\n\nAs in L124, the input ($Q$), $K$, and $V$ are initially derived from the input variable $X$. Besides, we also explain in Equations (7) and (8) where $K_i = X_i W^K$ and $V_i = X_i W^V$. Moreover, the evolution of $k_i$ and $v_i$ is controlled by a function $f(\\cdot): \\mathbb{R}^{d+1} \\rightarrow \\mathbb{R}^d$, which is implemented as an MLP as elaborated in Appendix F.4. \n> **W2: L138, an assumption that is not explained.**\n\nWe sincerely apologize for any confusion. Each observation's latent trajectory, rather than influence, spans the entire time horizon, as exemplified in Figure 1. Our method will further model the correlation among the underlying latent trajectories in the evolving system, which closely resembles the operation of Transformer, where each token computes attention scores across all tokens along the temporal axis. For auto-regressive prediction scenarios, we can apply causal attention with a masking mechanism, similar to vanilla Transformer, to avoid mistakenly leveraging future information.\n> **W3: Can Contiformer be applied in an online fashion?**\n\nThank you for your great point! We indeed use cubic spline interpolation, which might limit its application to online fashion. However, we want to clarify that the core architecture is not limited to the interpolation method. Therefore, we can explore alternatives to support online applications [1]. \n> **W4: The paper does not compare to SOTA models.**\n\nWe apologize for the oversight in comparison to this model. Due to the limited timeline, we leverage the public implementation of S5 model, i.e., [s5-pytorch](https://github.com/i404788/s5-pytorch), and derive the experimental comparison on the benchmark of UEA Classification in Section 4.2. \n\nOur model Contiformer exhibits a notable performance advantage over S5 UEA task. Besides, we believe that, with a wider range of hyperparameter searches, S5's performance might be improved.\n\n|Drop Ratio (%)|30||50||70||\n|-|-|-|-|-|-|-|\n|Model|S5|ContiFormer|S5|ContiFormer|S5|ContiFormer|\n|Avg. ACC|0.7139|**0.8126**|0.6831|**0.7997**|0.6455|**0.7749**|\n|# Top 1 |2|**18**|2|**18**|1|**19**|\n\n> **W5: The pendulum regression task should be included.**\n\nDue to the limited timeline, we do not perform much parameter search and use most of the default hyper-parameter settings from the UEA task. The preliminary results are shown below. We run the experiment 3 times and report the mean and standard deviation.\n\n|Model|MSE ($\\times 10^{-3}$)(std.)| \n|-|-| \n|ODE-RNN|7.26(0.41)|\n|CRU (original)|4.63(1.07)|\n|CRU (in S5 paper)|3.94(0.21)|\n|S5|**3.41**(0.27)|\n|Contiformer|4.21(0.24)|\n\nWe wish to emphasize that ContiFormer showcases superior performance compared to the S5 model within the UEA classification benchmark as shown in the response to **Q4** above. \n> **W6: The standard deviation result is missing for sections 4.2 and 4.3.**\n\nWe recognize the importance of incorporating standard deviation as a vital component for robust statistical analysis. We report the results in the attached PDF file (Table 1 for section 4.2, Table 2 for section 4.3). Moreover, it's important to note that for the UEA task in Section 4.2, our reporting approach aligns with the prevalent practice in the field of time series analysis [2， 3], where the overall performance of the mean accuracy and mean ranking are reported. Please refer to **Q2 in General Response** for more details.\n> **W7: Missing evaluation on regularly sampled datasets.** \n\nWe reevaluate our model and other baselines on a regularly sampled time-series setting (i.e., with the drop-ratio as 0%). We also include an additional strong baseline Autoformer [4], which is specifically designed for regular time-series data.\n\n|Model|ODE-RNN|Neural CDE|Autoformer|TST|Contiformer|\n|-|-|-|-|-|-|\n|**Avg. Rank**|3.5|4.2|3.3|2.1|**1.85**|\n\nWe also want to recall that Contiformer has illustrated superiority over all the baselines in the irregularly sampled data setting, as shown in Table 3 of our paper.\n> **Q1: How does the $P$ relate to $L$ in Table 2? Are these on similar orders?**\n\n$L$ represents the number of function evaluations that the ODE solver requests in a single forward pass from $t_1$ to $t_N$, which is related to the sequence length $N$ as well as the time difference $(t_N - t_1)$. While $P$ is the number of intermediate steps for integral approximation, which is a constant that is not related to either $N$ or the time difference $(t_N - t_1)$. Therefore, they are not on the same order and $P \\ll L$. Also, as is shown in L224, we set $P \\leq 5$ in our experiments.\n> **Q2: Neural ODE performs worse for interpolation.**\n\nWe want to clarify that the codes in our experiments are almost the same as that in the original implementation [Neural ODE repo](https://github.com/rtqichen/torchdiffeq/blob/master/examples/latent_ode.py), we have kept most experimental settings the same. We modify the data generation process (a more comprehensive response can be found in **Q1 in General Response** section), and as a result, Transformer and Neural ODE may yield poor interpolation. In contrast, Contiformer is more robust with better visualization. More visualization results are uploaded in the attached PDF file. Sorry for the confusion and we will clarify the details of this experiment in our revised paper. \n> **Q3: Incorrect Figure 4 in the Appendix.**\n\nApologies for the confusion. We have uploaded a figure in the attached PDF file. Please refer to **Q1 in General Response** for more details.\n\n[1] On the choice of interpolation scheme for neural CDEs. TMLR 2022.\n\n[2] A transformer-based framework for multivariate time series representation learning. KDD 2021.\n\n[3] Timesnet: Temporal 2d-variation modeling for general time series analysis. ICLR 2023.\n\n[4] Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. NIPS 2021."}, {"review_id": "tVrbwl6jJb", "reviewer": "Reviewer_v819", "summary": "The paper proposes a new deep learning model called ContiFormer to model continuous-time dynamics on irregular time series data. The paper argues that existing methods such as recurrent neural networks (RNNs), Neural Ordinary Differential Equations (ODEs), and Transformers have limitations in modeling continuous-time data and fail to capture intricate correlations within these sequences. The proposed ContiFormer model extends the relation modeling of the Transformer model to the continuous domain and incorporates the modeling abilities of continuous dynamics of Neural ODE with the attention mechanism of Transformers. Both numerical and theoretical justifications are provided.\n\nDifferent from the standard Transformer model, the proposed model is parameterized on the first-order derivative of $q$, $k$, and $v$. The real $q$, $k$ used in the attention part is approximately computed with finite sum (e.g., via Runge-Kutta 4th (RK4) order ODE solver). By restricting the function form/class of the first-order derivative on some smooth enough kernel, the proposed model could yield better interpolation/extrapolation ability.\n\nI have two concerns about the proposed model. \n\nThe first one is the usage of ODE solvers may introduce computation overhead. In Table 1, the order of computation cost is already $O(N^2 \\cdot P \\cdot d^2)$, where $P$ is the iterative number in RK4 solver, and in my understanding, $P$ should be 4. In L580 at Appendix B. the RK4 ODE may require 80 forward passes to finish one of the RK4 round. It implies the order of computation cost for a one layer attention model should be $O(N^2\\cdot 80\\cdot d^2)$, which can be very computationally expensive even for models with moderate size. Moreover, if the proposed model has more than one layer (e.g., 2 layers), in order to finish the second layer's attention computation, we will need first to finish the computation of the first layer with $O(N^2\\cdot 80\\cdot d^2)$ cost. After that, every second layer's computation will still involve the forward pass of the first layer, which becomes $O(2\\cdot N^2\\cdot 80\\cdot d^2)$. Therefore, for the $M$ layer attention model, the total computation cost could be $O(M^2\\cdot N^2\\cdot 80\\cdot d^2)$. \n\nThe second one the underlying construction is kind of similar to the Rotary Position Embedding in [1]. In this paper, the kernel trick is used to facilitate the computation (e.g.,  Eq. (27)- Eq. (32) in Appendix C). Based on the current presentation, the kernel function is only time point $t$ dependent but independent on the detailed $q,k$. Given the choice of the kernel function in Eq. (30), the effects of introducing this kernel function look like just adding a rotary positional embedding. \n\nAt present, the authors have not provided sufficient evidence to demonstrate the contribution of their model, which may not be compelling enough for top machine learning conferences like NeurIPS. Despite this, the reviewer is willing to reconsider the decision after the authors' rebuttal. Overall, the paper presents an interesting approach for modeling continuous-time dynamics, but the issues raised above need to be addressed before its acceptance for a top-tier conference.\n\n\nReference: [1] Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.\n \n\n\n\n", "questions": "Please see my comments in the Summary section.", "limitations": "The authors include a section on the limitations of the proposed work in Appendix G at Page 30.", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The paper proposes a new deep transformer model incoperating the continuous-time dynamics on irregular time series data. Both numerical and theoretical justifications are provided.", "weaknesses": "Please see my comments in the Summary section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive comments.\n\n> **Q1: The order of computation cost one layer attention should be $O(N^2 \\times 80 \\times d^2)$; for the $M$ layer attention model, it could be $O(M \\times N^2 \\times 80 \\times d^2)$.**\n\nThank you for your inquiry about time complexity. In Table 1 of our paper, we initially specified the computation cost as $O(N^2 \\times P \\times d^2)$, where $P$ represented RK4 solver iterations.\n\nUpon re-evaluation, we acknowledge that the actual number of forward passes relates to the product of RK solver order and the number of segments across the reprameterized variable interval $[-1, 1]$. This leads to a refined time complexity of $O(N^2 \\times \\tilde{P} \\times d^2)$, where $\\tilde{P}$ signifies true forward pass counts.\n\nMoreover, in our time complexity analysis in our paper, we focus on the computation within one individual layer, as stated in the caption of Table 1 and the corresponding text description in our paper. We assume that all the compared models can be stacked with multiple layers. Thus, we omit the variable of the layer number in this analysis.\n\nThank you for your comment. In the revised manuscript, we are committed to refining Table 1 to ensure its alignment with our findings and discussions.\n\n> **Q2: The underlying construction is kind of similar to the Rotary Position Embedding in [1], where the kernel trick is used to facilitate the computation (e.g., Eq. (27)- Eq. (32) in Appendix C). The effects of introducing this kernel function look like just adding a rotary positional embedding.**\n\nWe are grateful for your question, which provides an opportunity to clarify the distinctions between our paper and Roformer [1]. We are confident that our work stands out from Roformer due to several essential reasons.\n\nThe first clarification is that the discussion from Eq. (27)- Eq. (32) in Appendix C in our paper is about the Kernalized Attention mechanism, which is *not* to categorize our method as Kernalized Attention, but rather to demonstrate that Kernalized Attention is a form of Transformer attention variant. In fact, we mathematically prove that, as stated in Lines 583-584 in Appendix C, with carefully designed function hypotheses, various Transformer variants (including those with Kernelized Attention) can be encompassed as special cases of ContiFormer. \n\nConsequently, given Roformer's utilization of a pre-defined \"position\" matrix, i.e., $\\boldsymbol{R}_{\\Theta, m}^d \\in \\mathbb{R}^{d \\times d}$ in Eq. (15) of Roformer's paper. It falls into the category of the \"Time Embedding Method\", as defined in Eq. (29) in the Appendix of our paper. As evidenced by Theorem 1 in Line 643 of our Appendix, Roformer (as well as all the relative position embedding methods) can be seen as a specific instance of ContiFormer.\n\nTechnically, ContiFormer stands apart from Roformer in the below two aspects.\n\n1. We prioritize modeling intricate dependencies of the inputs and learning the continuous-time evolving mechanism in irregularly-sampled data through a novel parametric continuous-time attention mechanism, differing from Roformer's focus on integrating positional information into the attention mechanism.\n\n2. Our attention mechanism (Eq. (3), Section 3.1) captures evolving relationships of the input which is input dependent and (continuous-)time-aware, beyond reliance on only categorical positional information in Roformer which cannot model the continuously evolving dynamics of the system in the irregular time-series data.\n\nWe greatly appreciate your query, as it allows us to elaborate on these nuanced distinctions between our work and Roformer, providing a clearer understanding of our contributions.\n\n> **Q3: At present, the authors have not provided sufficient evidence to demonstrate the contribution of their model.**\n\nOur work introduces a continuous-time attention approach tailored to the continuity and self-related nature of irregular time series. The contribution is three-fold.\n\n1. To the best of our knowledge, we are the first to incorporate a continuous-time mechanism into attention calculation in Transformer as shown in Section 3 of our paper, which is novel and captures the continuity of the underlying system of the irregularly sampled time-series data, as shown in our experiments.\n2. To tackle the conflicts between continuous-time calculation in continuous attention mechanism and parallel calculation property of the Transformer model, we also provided a novel reparameterization method in Section 3.3, to divide the whole time horizon and map them into a fixed time range, and then we can parallelly execute the continuous time attention in the different time range, without hurting the model capacity (as shown in our experiments). This can provide another novel perspective to accelerate the calculation of continuous-time modeling in other works such as Neural ODE.\n3. Notably, we mathematically proved in Section 3.4 and Appendix C.2 that, our proposed continuous attention mechanism is a universal attention approximator, and various Transformer variants including kernel-based methods can be viewed as special instances of our model. Our approach uniquely aligns with the intricate characteristics of irregular time series and offers a broader scope that encompasses Transformer variants, which can shed some light on the further exploration of continuous process modeling in the Transformer model.\n\n\nWe thank you for your pointing out the confusion about our description. We will further refine our paper according to your valuable suggestions.\n\n[1] Roformer: Enhanced transformer with rotary position embedding. (2021)."}, {"review_id": "5gTqwfkigb", "reviewer": "Reviewer_PCsf", "summary": "The paper describes a continuous time extention of the transformer architecture using ODE blocks to propagate the effect of each observation individually through time. For computing attention values inner products between functions are used, where in the implementation the resulting integral is approximated. The paper demonstrates the method's application to irregular time-series data and in temporal point processes in different tasks including interpolation, extrapolation, classificaton.\n\n", "questions": "Please give standard deviations and mark significant differences.\n\nAlso it is interesting to see that NeuralCD is outperformed by ODE-RNN here for the 30 and 50% dropped case. As there is result provided for CharacterTrajectories in the appendix (this dataset was used in the original NeuralCDE paper) I compared the results.\nformat:  present paper, vs, Kidger et al.\n\n| Method | 30% dropped | 50% dropped | 70% dropped |\n|-----------|-----------|-----------|-----------|\nGRU-D |  0.9325 vs. 0.942 | 0.8872 vs. 0.902 | **0.8433 vs. 0.919** |\nGRU-deltaT | **0.8558 vs. 0.936** |  0.9088 vs. 0.913 | **0.8496 vs. 0.904** |\nN CDE |  **0.9276 vs. 0.987** | **0.9285 vs. 0.988** | **0.9241 vs. 0.986** |\nODERNN | 0.9415 vs. 0.954 | 0.9506 vs. 9.960 | 0.9471 vs. 0.953 |\n\nThese are serious differences on the same dataset. I can maybe explain away the case with Neural CDE as the creator of the method may be able to tune it better (this is in fact the main drawback of this entire comparison table idea we do, but this is a problem we are not going to solve here :) But results of several other baselines are quite different. This is concerning. \n\n\"We generated a dataset of 300 2-dimensional spirals, sampled at 150 equally-spaced time points\" <-- what is the temporal dimension? is this CD spiral a parametric curve where time is the parameter?\n\nAre you using vanilla NeuralODE in this experiment? I find it hard to understand why you get the interpolation behavior from NODEs as it is visible in Figure 2. The interpolation looks like piece-wise linear. Can you provide interpolation results with ODE-RNNs and neural CDEs?\n\nDo you see a way to avoid resampling between layers?\n\nIt would be useful and quite interesting to see some discussion about the apparent contradiction in using transformers (long range interactions) and ODEs. ODEs operates with derivatives and their solutions are locally described. The assumptions of a smooth differentiable solution seems to stand in contradiction with long range interactions unmediated by values in between, or in a case of a latent-ODE decriptions by latent system states. Can you formalize the properties of systems (e.g. the mentioned stock market time series) where you expect this type of modelling give the most benefit?\n\n\nMinor:\n\n\"To overcome the discrete nature of RNNs, a different strategy involves the exponential decay of\nthe hidden state between observations, governed by modifiable decay parameters [5, 9, 37].\"  [9] is the reference for the Neural ODE paper from Chen at al., it does not apply exponential decay.\n\ntypo:  denoted vs donated\n\nPlease stress the difference between irregularly sampled data and temporal point processes eg. in [37] more clearly (missing data vs. events).\n\nNOTE: the authors clarified some points, provided new results, and we identified a misunderstanding about the Neural ODE figure. The authors used Latent-ODE, but cited a previous paper in the table: that lead me believe that they fitted a Vanilla NODE to the trajectory. I raised my score by one.\n\n\n", "limitations": "No systematic discussion on limitations provided.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper gives a continuous-time extension of transformers capable of handling irregularly sampled continuous time series, and point processes.\nComputational complexity is assessed in details which is quite important for practical application, and helps the reader to better estimate the real life cost of scaling the methods to real life problems.\n", "weaknesses": "My main problem is with the missing statistical rigor during evaluation. More specifically:\n\nIn Table 3 the authors report averages over 20 datasets. I assume these datasets have different sizes, complexity, dataset imbalance etc., therefore it is hard to rigorously evaluate the average metrics.There are dataset-by-dataset level results in the appendix but there is still no standard deviation.\n\nSimilarly in Table 4, no standard deviations. Probably 3 std differences are significant, but with 3 repeats I am not sure sure 2 std is very convincing. (For just a very rough estimate how big delta is needed check a 3 sample unpaired t-test table)\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1: Table 3 covers 20 datasets, hard to rigorously evaluate the average metrics.**\n\n1. We would like to clarify that our benchmarking follows the prevalent common practice [1, 2], where the averaged metrics are over all the datasets in the UEA benchmark. \n2. The experiments on irregular time-series data follow the settings of Neural CDE [3], [3] only evaluated one of the UEA datasets while we evaluate over 20 datasets. The full results are in Appendix E.2.2.\n\n> **W2: The results in the appendix lack standard deviation (std.)**\n\nWe quite appreciate your reminder and *we have included the standard deviation of results for the UEA benchmark in Table 1 of the attached PDF file*, on the setting when 70% of observations are dropped, due to the word limit. We will incorporate the std. results for all three settings in the revised paper. Please refer to **Q2 in General Response** for more details.\n\n> **W3: With 3 repeats, not sure 2 std is very convincing.**\n\nFor the statistical significance in Sec. 4.3, we further perform experiments on Synthetic and Bookorder datasets, with *10* random seeds for each method. We report the complete experimental result in the attached PDF file, Table 2 and Table 3. Also, we conduct a significance test. We can conclude that when the p-value is less than $10^{-6}$, Contiformer can significantly outperform these baselines in 4 out of 6 metrics, i.e., LL and RMSE for Synthetic, LL, and Accuracy for BookOrder. In the revised manuscript, we will conduct a significance test on all the datasets. Please refer to **Q2 in General Response** for more details.\n\n> **Q1: The results of several baselines are different from the original paper**\n\nThanks for your concern. After carefully checking the original paper (Kidger et al. https://arxiv.org/pdf/2005.08926.pdf) we find a significant difference in data setting. In both ours and Neural CDE paper, the raw dataset was randomly dropped by the ratio of 30%, 50%, and 70% to conduct the irregularly sampled dataset. In Section 4.1 of Kidger et al., \"The randomly removed data is *the same* for every model and every repeat.\"; however, in our setting, we apply randomness on dropping the data points for each random run, which is more general to evaluate the robustness than on the same dataset dropped only once. This explains that our evaluation results of rerunning the baseline methods are different from those in the Neural CDE paper.\n\nNext, we checked the implementation details of these ODE-based baselines, which are the same as the official repository of [Neural CDE](https://github.com/patrick-kidger/NeuralCDE/tree/master/experiments/models). \n\nWe also put openly available implementations in the original submission to ensure transparency and fairness. We will also revise our paper to clarify this.\n\n> **Q2: What is the temporal dimension? is this CD spiral a parametric curve where time is the parameter?**\n\nSorry for the confusion. Temporal dimension means the variable $t$. This 2D spiral is a parametric curve. As in Eqs. (47, 48) in Appendix E.1.1, the 2D spiral function $(x(t), y(t)) \\in \\mathbb{R} \\rightarrow \\mathbb{R}^2$ is dependent on the time variable $t$, with the hyper-parameters $a, b$ controlling the curve shape.\n\n> **Q3: Are you using vanilla NeuralODE? In Fig. 2 why the interpolation looks like piece-wise linear?**\n\nThanks for your valuable inquiry. Yes, we are using the vanilla NeuralODE implementation in our experiment. The reason why the interpolation from NODEs in Figure 2 looks like piece-wise linear is intrinsically from the more challenging data setting than the original one, as explained in **Q1 of the General Response** part.\n\n> **Q4: Do you see a way to avoid resampling between layers?**\n\nInsightful question! To our knowledge, neural networks can not directly handle continuous-time input while stacking multiple layers. To support stackable property, one can either discretize the output in continuous-time dimension, like our resampling method or discretize the parameter space, like the state space model [4, 5].\n\n> **Q5: Can you formalize the properties of systems where you expect this type of modeling give the most benefit?**\n\nOur model hinges on the premise of continuous-time dynamics. We treat the irregularly-sampled data as a sequence of observations stemming from an underlying continuous-time process [3]. Moreover, we posit self-correlation among the observations [6]. Given these foundational assumptions, ContiFormer is poised to excel, capitalizing on the power of continuous-time modeling and inherent self-correlation within the data.\n\n> **Q6: \"...decay parameters [5, 9, 37].\" [9] is incorrect.**\n\nThanks for the suggestion. Fixed in the revised version.\n\n> **Q7: difference between irregularly sampled data and temporal point processes e.g., in [37]?**\n\nBased on our understanding of your question, here \"irregularly sampled data\" refers to the irregularly sampled time series data, a kind of sequence of signal values sampled in irregular time intervals; And \"temporal point processes\" are a branch of modeling methods such as [37], which are often utilized to model event sequences that is a type of sequence containing discrete events happened at irregular time points.\n\nThe differences are as below.\n|Aspect|Irregularly sampled time series data|Event sequence data (w/ temporal point processes)|\n|-|-|-|\n|Collection method| Irregularly sampled|Triggered by events|\n|Data type|Signal values|Event features|\n\n[1] A transformer-based framework for multivariate time series representation learning. KDD 2021.\n\n[2] Timesnet: Temporal 2d-variation modeling for general time series analysis. ICLR 2023.\n\n[3] Neural controlled differential equations for irregular time series. NeurIPS 2020.\n\n[4] Simplified state space layers for sequence modeling. ICLR 2023.\n\n[5] Efficiently modeling long sequences with structured state spaces. ICLR 2022.\n\n[6] Attention is all you need. NeurIPS 2017."}, {"review_id": "Q5VsL1DAG0", "reviewer": "Reviewer_GCXG", "summary": "This paper introduces the ContiFormer, a novel continuous-time Transformer model designed for handling irregular time series data. In this model, the keys, queries and values are vector-valued functions indexed on time. Each input observation gives rise to a key function, given by the solution of a Neural ODE started at an initial condition determined by the observation. While a similar model is adopted for the value functions, the query function is obtained using natural cubic spline interpolation of discrete query values. By replacing dot products (between discrete queries and keys) with an $L^2$ inner product, a continuous-time attention function is derived. The continuous-time output can be evaluated on a discrete time grid allowing for the stacking of such layers. To reduce the computational time, the paper leverages a reparametrization trick, which makes it possible to solve simultaneously multiple Neural ODEs on different time intervals. The ContiFormer model is evaluated on various time series tasks and datasets demonstrating its effectiveness in handling irregular time-series data.", "questions": "- Why did you choose to model the key and value functions differently from the query function? Can the ContiFormer handle partially observed data, given how the key and value functions are defined? \n- Do you have an intuition/explanation for the fact that $P$ can be chosen to be very small (l 224) and the insensitivity of the ContiFormer to the tolerance error (Appendix F12, F13)? \n- Related to the comment above about the continuous-time formulation, does the ContiFormer offer any advantage for handling both irregular and long time series?\n- Although computational complexities are provided, would it be possible to report the actual time it takes to train and evaluate the model?\n", "limitations": "Yes, some limitations have been discussed. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Given the empirical success of transformer models on machine learning tasks involving sequential data, enhancing their capability of effectively handling irregular time series data is an important research question.  \n- This paper proposes an elegant solution based on Neural ODEs and other ideas from the field of neural differential equations such as the use of cubic splines in Neural CDEs.\n- The paper is clearly written and easy to follow. \n- The empirical evaluation is well conducted, extensive and provides a compelling demonstration of the superior performance of the ContiFormer model.", "weaknesses": "- The theoretical results are deferred to the appendix. I think the main theorem (Thm 1 in Appendix) should at least be formally stated and further commented on in the main paper. This would better motivate and support the modelling choices. \n\t\n- The paper is a bit repetitive at times (l100-102, l108-117, l175-178) while some components might deserve further explanation: 1. the main theoretical result;  2. although the $i^{th}$ key and value functions are well defined at $t<t_i$, it could be further commented by expanding l138, and perhaps adapting Fig. 1 with $t<t_i$; 3. the rationale for modelling the key and value functions differently from the query function. \n\t\n- As mentioned in the related work section, a continuous-time attention model based on Neural ODEs has previously been proposed in [10]. While several baselines are considered in the experimental section it is surprising that this model is not included without a justification. \n\n- The attention model is not fully formulated in continuous-time, in the sense that it is expressed as a sum over the number of value functions which depends on the number of observations. A natural thought of experiment for a continuous-time model is to replace the input sequences with continuous paths. Is there a way to define the ContiFormer on such paths? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your comprehensive comments.\n\n> **W1: The theoretical results should at least be formally stated and further commented on in the main paper.**\n\nThanks for your great suggestion! In the revised manuscript, we intend to consider your suggestion.\n\n> **W2: L138, adapting functions to $t<t_i$.**\n\nWe are pleased to adopt your suggestion to make it more clear for all $t$ in the whole time horizon.\n\n> **W3: Compare Contiformer with CADN [1].**\n\nWe re-implement the model and conduct experiments on the UEA classification task. Below, we show a summary of the results. We apologize that due to the time limit of the rebuttal period, we only manage to run 15 out of 20 datasets for CADN. We conclude that Contiformer outperforms CADN on most datasets.\n\n\n|Drop Ratio (%)|30||50||70||\n|-|-|-|-|-|-|-|\n|Metric|CADN|Contiformer|CADN|Contiformer|CADN|Contiformer|\n|Avg. ACC|0.6869|**0.7879**|0.6852|**0.7786**|0.6898|**0.7563**|\n|# Top 1|1|**13**|1|**14**|4|**11**|\n\n> **W4: Define the ContiFormer on continuous paths.**\n\nThank you for the insightful perspective! ConfiFormer incorporates continuous paths in a latent space. Thus, if the input is a continuous path, Contiformer can directly apply ordinary differential equation (ODE) modeling (as described in Section 3.1) on the input sequence and calculate our proposed continuous-time attention upon that. However, directly modeling continuous paths using the ODE assumption will break the parallel execution of Transformer architecture. Thus, as described in Section 3.3, we further propose a novel reparameterization trick to split the time horizon into several fixed-length pieces $t\\in[-1, 1]$, parallelly executing continuous-time attention calculation on these pieces and then map back to the original time horizon without loss of performance. This parallel execution requires sampling timing points on the continuous path, which needs further refinement and exploration in our future work.\n\n> **Q1: Why model key & value differently as query? Can Contiformer handle partially observed data?**\n\nThanks for your insightful question. The effects of the query function and key and value functions are somehow different. Thus, we separate the roles of query and key/value into distinguishable ones. This design aligns the original attention mechanism such as [2, 3] where the query, key, and value functions are different.\n\nIndeed, Contiformer can handle partially observed data. Our model is designed for irregular time series modeling, including irregularly-sampled data caused by missing time points (if we correctly understand your mentioned partially observed data). The result shown in section 4.2 also supports the clarification, of which observations are randomly dropped. Technically, our model inputs an irregular time series in any format and outputs a continuous-time latent trajectory of the system.\n\n> **Q2: Why $P$ can be very small? Why Contiformer's insensitivity to the tolerance error?**\n\nThanks for your insightful question. As shown in Tables 12 and 13 in the Appendix, our model is robust to the tolerance error of the ODE solver and the numerical error. We explain the phenomena in two aspects. First, our framework, akin to the Transformer architecture, circumvents cumulative errors by eliminating the necessity of sequentially passing neural networks in a regressive manner. Second, we guess that, since the output from the attention module is actually a weighted sum of the tokens, therefore, the total variance is lowered. For instance, assume that $x_1, ..., x_N \\sim \\mathcal{N}(\\mu, \\sigma^2)$, then the total variance of the mean value $\\bar{x} = \\frac{1}{N}(x_1 + x_2 + ... + x_N)$ is $\\frac{\\sigma_2}{N}$, and the variance is significantly lowered given large $N$. Overall, our model is not sensitive to tolerance error, which makes our model robust to different scenarios.\n\n> **Q3: Can Contiformer handle irregular long time series?**\n\nAs shown in Table 10 in the Appendix, the BookOrder dataset with both irregular properties and a relatively long time series. Our model outperforms all the baselines on this dataset. Nevertheless, due to the nature of Transformer-based architecture, it might necessitate increased memory capacity and a richer training dataset to achieve optimal performance. But we believe that the other works alleviating the burden of memory cost of the vanilla Transformer model may also inspire the improvement of our method, which is a promising direction for future work. Thank you for your valuable comment! \n\n> **Q4: The actual time it takes to train and evaluate the model.**\n\nWe appreciate the reviewer's interest in the practical aspects of our work's computational performance. While we have provided comprehensive computational complexity analysis, we acknowledge the importance of reporting actual training and evaluation times for a more practical understanding. We report the actual time in the below table on the Synthetic dataset for event prediction task relative to RMTPP, which is a recurrent neural network. Experiments are conducted on NVIDIA RTX A6000.\n\n|Model|RMTPP|THP|GRU-$\\Delta$t (atol=0.1)|mTAN|ContiFormer (atol=0.1)|\n|-|-|-|-|-|-|\n|Relative Time for Training|1 $\\times$|1.07 $\\times$|3.22 $\\times$|1.05 $\\times$|6.96 $\\times$|\n|Relative Time for Testing|1 $\\times$|1.05 $\\times$|2.82 $\\times$|1.02 $\\times$|4.18 $\\times$|\n\n\nAs demonstrated in the table, our model exhibits an approximate slowdown of $6$ times compared to the vanilla Transformer model (THP), and it demonstrates a $2$ times decrease in speed compared to the ODE-based model (ODE-RNN). \n\n[1] Chien, Jen-Tzung, and Yi-Hsiang Chen. \"Learning continuous-time dynamics with attention.\" TPAMI 2022.\n\n[2] Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. \"Neural machine translation by jointly learning to align and translate.\" ICLR 2015.\n\n[3] Sukhbaatar, Sainbayar, Jason Weston, and Rob Fergus. \"End-to-end memory networks.\" NIPS 2015."}, {"review_id": "ekMrZubMEC", "reviewer": "Reviewer_jYof", "summary": "The authors propose a new method, a continuous-time transformer, ContiFormer, for irregular time series modelling. The proposed method extends vanilla Transformer to a continuous domain. In particular, the model incorporates the continuous dynamic modelling of Neural ODE with the attention mechanism of a transformer. Moreover, the authors show that many transformer variants designed for irregular time series modelling are a special case of the proposed ContiFormer. The authors evaluated the method on a continuous time function (spiral), irregular sampled time series for classification and irregular time series for event prediction. The results indicate that the proposed method obtains better results than the baseline models.", "questions": "\nIn line 83, the authors say that previous models 'are insufficient in capturing input-dependent dynamic systems'. How is this claim supported? The previous work also is also condition on the input data. \n\nIn line 99-100, the author's say 'ContiFormer, employs a non-autoregressive paradigm', however, the vanilla transformer decoder at inference time is an auto-regressive model, can you clarify this? Furthermore, to perform the ODESolver step, you perform auto-regressive P steps? \n\nEq. (1), the formulation is unclear and incorrect: the author's define both the ODE as well as the continuous latent variable with the same parameter. Furthermore, if k_i(\\tau) is the ODE, what is f()?\n\nIn line 136, could you please clarify why you have modeled each observation as a separate ODE? As showed in https://arxiv.org/abs/1806.07366 NODE models can model irregular time-series with a single ODE specification, even more so, close by observations most likely follow the same ODE. \n\nIn line 136, could you clarify if the ODE is a MLP or any other architecture?\n\nLine 138: would suggest the authors to adjust this assumption/claim: from a dynamical systems perspective a future event cannot affect a past event. Vast majority of works modelling dynamics are often based on Markov property. \n\nline 144 in text you mention that q(t_i) is a cubic spline approximation, however, in the appendix you clarify that linear interpolation is used due to the lower computational complexity. please clarify the approximation used in your proposed method. What is the dimensionality of Q_i? \n\n\nEq. (4) Based on the introduced notation it seems like the introduced computation ends up being a matrix (Q_i), vector (k_i() \\in R^d) product. Where the Q_i is an approximation of the process via cubic spline, and k_i() is the latent trajectory by an ODE, this would imply that the proposed mechanism measure the similarity of the learned approximation of the ODE to the approximation by the cubic spline function. Is this correct?\n\nEq. (7) For multi-head attention, you also apply a linear transformation to the latent continuous state, q(),v()? \n\nHow sensitive is the method of using a different approximation method instead of cubic spline? \n\nDoes each latent continuous trajectory span only two adjacent time points t:[-1,1]? Meaning that depending input length L (data points), you would have L independent latent ODE trajectories? It is unclear from the text. \n\nThe authors mention that their attention mechanism is measuring the correlation between different observations (line 298), therefore, I would suggest the authors to compare their work to Autoformer (that measures correlation between different time series) or Fourier-based (FEDformer) attention mechanism. Have such comparisons been made? ", "limitations": "The authors have not explicitly addressed the limitation of the presented work. Based on the text, the proposed method models each observation with a separate ODE function. This, however, seems to be sub-optimal as data points close in time probably follow the same underlying dynamics function, furthermore, the authors have not explained why such a design choice would be necessary, as by default Neural ODEs can account for irregularly sampled time series. Moreover, it would be nice if the authors would have addressed (as an ablation) the model's performance on regular time-series data compared to existing state-of-the art methods. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "Originality: \nThe proposed work is original. The author's propose a novel continuous-time attention mechanism (CT-MHA). \n\nQuality: \nThe authors have included also a comparison of complexity analysis of their method compared to other. The authors have tested the proposed method on multitude of datasets: irregularly-sampled time series for classification with increasing number of data dropped, as well as event prediction. Especially for the latter, the authors compare their method to existing state-of-the-art models for event forecasting (SAHP, THP) and obtain an improved performance. \n\nClarity: \nPlease see weaknesses. \n\nSignificance:\nThe authors provide a unique theoretical approach, and the obtained results demonstrate a benefit of the proposed method. \n", "weaknesses": "Originality, related work: \nThe related work is a bit convoluted. It would be better if the authors divide the related work into 3 subsections: Transformers for time-series modelling, Transformer for irregular time series, Continuous time models (NODEs). The literature of RNN is not as relevant to this work as it builds upon transformers and NODEs. \n\nWith respect to NODE related work:\nLine 43-45, the authors say 'the recursive nature of Neural ODEs can lead to cumulative errors if the number of iterations is numerous'. This is not precise. The cumulative error is dependent of the solver used, and there are numerous works which by enforcing, for example, conservation of energy, keep the error tolerance low, for example, https://arxiv.org/abs/1909.12077. Moreover, by selecting difference step size, solver type (adaptive or fixed) one can control/adjust for the numerical error (as mentioned in https://arxiv.org/abs/1806.07366). Please rephrase. \n\nWith respect to Transformer related work:\nIn line 47-48, the authors mention that due the fixed time encoding or learning upon certain kernel functions the resulting models struggle to model complex data patterns in real-world practice.  However, I do not see how this claim is supported. Even in the appendix, for given percentages of observations dropped (50%) mTAN model outperforms ContiFormer. Therefore, the claim is not supported. Please rephrase.\n\nThe proposed work differs from the existing work in the field, however,  some of the claims made about the previous work are not well supported in the current version of the paper. \n\nQuality: \nThe proposed continuous time attention mechanism, eq. 3, is unclear, as the underlying functions are not clearly defined. The authors' mention that previous transformer works for irregular time series are special cases of their model, however, the proof of this only mentioned in the Appendix, I would recommend moving the core parts of the proof to the main part of the paper. \n\nLines 281 - 283, the authors say that ContiFormer consistently outperforms all the baselines on all three settings, while in the appendix it can be seen that there are also instances were the baseline models outperform ContiFormer, therefore, I would soften the claim. \n\nFor lines 303-304, I am missing a comparison to methods that are more aligned with the task at hand. The authors compare against TST and mTAN, however TST  model is focused on multivariate time series (not decomposition of different dynamic trends), while mTAN although also designed for irregular time series modelling the paper's focus is on temporally distributed latent representations. For extracting temporal dynamics/trends of the data, there are other works, Autoformer (https://arxiv.org/abs/2106.13008), FEDformer (https://arxiv.org/abs/2201.12740) that would be perhaps a more fair comparison to the present method.  \n\nClarity: \nThe submission is not clearly written, thus negatively affecting the understanding of the proposed method. Please clarify for all functions, v(), k(), q() the input, output dimensionalities, as well what these functions are. As this is unclear, it is hard to evaluate what exactly the inner product of the two function spaces is computing. \n\n\nLine 147. Imprecise phrasing: the authors mention that self-attention involves calculating the correlation between queries and keys, which is incorrect. The computation represents an inner dot product, which is not a correlation metric but rather a similarity measure, please rephrase.\n\nThe main figure (figure 1), is also not immediately clear for the reader. Would suggest to adjust the figure.  It is hard to follow all the colors and how they correspond to which input. \n\nIn addition, for the NODE model compared with, as well as for the ODESolver used in author own work it is missing what kind of solver is used: adaptive or fixed step? This could greatly affect the performance, hence, it is a crucial implementation detail. \n\nIn the appendix mTAN model has the wrong reference. The reference is for the Autoformer model, please correct. \n\nLastly, for the experiments is it not clear what is the input/ouput data length for the model.\n\nOverall, the technical details of the paper are not clear. As this is the main contribution of the paper, it is essential that it is clear so that the extensive experimental results can be objectively evaluated given the method at hand. I would suggest the authors to rewrite this section. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestion. We will diligently reassess the detailed statements made in the paper and diligently bolster each claim with appropriate supporting evidence. Below we try to address your problems.\n> **W1: Table 8 mTAN model sometimes outperforms ContiFormer.**\n\nContiformer outperformed mTAN on 18 datasets, while mTAN only outperformed our model on 2 datasets. In Table 3 in the main paper, Contiformer consistently outperformed mTAN on all three data settings (30%, 50%, 70% drop-ratio).\n> **W2: Which ODESolver was used and what kind of steps are missing**\n\nWe use fixed-step RK4 solver, L578-581 in Appendix B.3.\n> **W3: What is the input/output data length**\n\nInput: $X\\in \\mathbb{R}^{N \\times f}$ with $N$ as sequence length and $f$ as input feature dimension; Output is in $\\mathbb{R}^{N \\times d}$ where $d$ is the hidden dimension.\n\n> **Q1: L83, clarify 'previous works are insufficient in capturing input-dependent dynamic systems'**\nMost Transformer-based methods, with the exception of mTAN, capture input-dependent patterns, rather than the underlying dynamic systems, which is a fundamental mathematical and conceptual framework used to describe the evolution of variables over time.\n\nFor mTAN, although it models continuous-time dynamic systems, its attention mechanism is primarily dependent on time variables, rather than the input information. Consequently, its behavior lacks input-dependent attributes, a crucial factor for effective modeling in complex scenarios.\n> **Q2: In L99, clarify 'ContiFormer, employs a non-autoregressive paradigm'.**\n\nHere \"autoregressive\" means that the stepwise model inference is upon the output from the last timestep, even given the whole-length sequence input. We mean non-autoregressive for the *encoding part* of Transformer model and our model.\n> **Q3: if $k_i(\\tau)$ is the ODE, what is $f()$?**\n\n$k_i(\\tau)$ is a time-function controlled by ODE, and $f()$ is implemented as an MLP.\n> **Q4: L136, why modeled each observation as a separate ODE?**\n\nEach observation will derive a corresponding trajectory modeling *result*, rather than assuming each observation follows different ODEs, since all these ODE functions modeling the latent trajectories share the same ODE function parameter, i.e., $\\theta_k$ and $\\theta_v$ in Eq. (1).\n> **Q5: L138, from a dynamical systems perspective a future event cannot affect a past event.**\n\nOur method can model the relationship among these observations through continuous-time attention, just like the pairwise attention modeling in vanilla Transformer model. Sorry for the confusion.\n> **Q6: The approximation used for $q(t_i)$. What is the dimensionality of $Q_i$?**\n\nWe use cubic spline function to obtain $q(t_i)$ and $Q_i \\in \\mathbb{R}^d$.\n> **Q7: The attention mechanism ends up being a matrix, vector product.**\n\n$q(\\cdot): \\mathbb{R} \\rightarrow \\mathbb{R}^{d}$ is a time function, *not* a matrix. $k_i(\\cdot): \\mathbb{R} \\rightarrow \\mathbb{R}^{d}$ is also a time function. We actually calculate the similarity (or relationship) between continuous functions within a closed interval. Contiformer extends the vanilla Transformer into the continuous-time domain, we emphasize that our attention mechanism operates directly on continuous-time inputs, in contrast to discrete matrices/vectors.\n> **Q8: Eq. (7) For multi-head attention, is linear transformation applied?**\n\nYes, as shown in Eq. (7) of our paper.\n> **Q9: How sensitive is the method of using a different approximation method instead of cubic spline?**\n\nWe have conducted an ablation study on UEA classification and our model is insensitive to the utilized approximation method.\n\n|Drop Ratio (%)|70||\n|-|-|-|\n||Linear Interpolation|Cubic Spline|\n|**Avg. Acc**|**0.7775**|0.7749|\n|**# Top 1**|**13**|7|\n> **Q10: Does each latent continuous trajectory span only two adjacent time points t:[-1,1]?**\n\nLatent continuous trajectory evolves along time, i.e., from $t_1$ to $t_N$, where $t_i$ is the time of the $i$-th observation ($i \\in [1,N]$). Therefore, it does *not* span only two adjacent time points. \n> **Q11: The proposed method models each observation with a separate ODE function. This seems to be sub-optimal.**\n\nWe would like to clarify that all the observations are modeled with *the same* ODE function with only one set of parameters $\\theta$ in Eq. (1), *rather than* modeling by *different* ODE functions. \n> **Q12: Why such a design choice would be necessary?** \n\nThe necessary is discussed in **Q4, Q10**, and **Q11**. Besides, Neural ODEs can only calculate the ODE function from the initial time point to the last one, while our method can calculate the ODE function from $t_1$ to $t_N$ separately and parallelly. Please refer to **Q3 in General Response** for more details.\n> **Q13: Experiment on regular time series.**\n\nWe further experiment additionally on UEA classification following section 4.2.\n\n|Drop Ratio (%)|30|||50|||70|||\n|-|-|-|-|-|-|-|-|-|-|\n||AutoFormer|FedFormer|ContiFormer|AutoFormer|FedFormer|ContiFormer|AutoFormer|FedFormer|ContiFormer|\n|**Avg. ACC**|0.7035|0.5994|**0.8126**|0.6900|0.5655|**0.7997**|0.6368|0.5312|**0.7749**|\n|**Avg. Rank**|2|2.7619|**1.1905**|1.8571|2.8571|**1.2857**|1.9524|2.7619|**1.1905**|\n\nWe also investigate Contiformer's performance on long times series forecasting using Exchange dataset. Contiformer can outperform both models. Following Autoformer, input length is $96$ and the target horizon is in $[96, 192, 336]$.\n\n|Exchange dataset|Autoformer||FEDformer||Contiformer||\n|-|-|-|-|-|-|-|\n|Horizon len.|MSE|MAE|MSE|MAE|MSE|MAE|\n|96|0.1409|0.2711|0.1356|0.2643|**0.1301**|**0.2521**|\n|192|0.8544|0.6915|0.2764|0.3833|**0.1966**|**0.3193**|\n|336|0.707|0.6445|0.4657|0.5069|**0.4315**|**0.4852**|\n\nWe may not conduct full experiments on all the datasets, but we believe these additional experiments have addressed your concerns about our work.\n\n[1] Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI 2021."}]

Author rebuttal:
## General Response

We thank all the reviewers' valuable and insightful suggestions! And we are encouraged by positive comments from the reviewers, e.g.,
* Addressing import research problems with high practical value (Reviewer GCXG)
* The proposed method is novel with significant contribution in the area (Reviewer 2E9G, v819, GCXG, jYof)
* The paper is well-written and easy to follow (Reviewer 2E9G, GCXG)
* Experimental results are promising with theoretical analysis. (Reviewer 2E9G, GCXG, jYof)

Below we try to clarify a few concepts in our paper and address some common problems.

***

> **Q1: Concern about interpolation setting and visualization results in Section 4.1.**

As raised by reviewers 2E9G, PCsf. The interpolation result of Neural ODE in Fig. 2 of our paper is confusing. Here, we make the following explanation.

1. We want to clarify that the codes in our experiments are the same as that in the original implementation [Neural ODE repo](https://github.com/rtqichen/torchdiffeq/blob/master/examples/latent_ode.py), we have kept most experimental settings the same.
2. To better illustrate the relative performance of the compared methods, we slightly modified the ground-truth spiral data generation process by adding some random noise to the underlying spiral function hyper-parameters (i.e., $a$ and $b$ as discussed in Appendix E.1.1). The task becomes more difficult since the dataset incorporates some distributional shifts by this modification.

As a result, the compared baselines (Transformer and Neural ODE) could not handle this challenge well and illustrated poor interpolation. In contrast, our proposed Contiformer is more robust in this setting showing better interpolation performance with lower error (shown in Table 2) and better interpolation visualization (in Fig. 2) in our paper. 

We also reproduce the experiment with *the same* spiral data generation in [Neural ODE repo](https://github.com/rtqichen/torchdiffeq/blob/master/examples/latent_ode.py), without changing any of the code, and we find that when Neural ODE is not well-trained, piece-wise linear results may still occur. 

To address the concern, we have provided more visualization results in the attached PDF file (Fig. 1 and Fig. 2).

> **Q2: Missing of standard deviation for significant test in Section 4.2 and Section 4.3.**

As pointed out by reviewers 2E9G and PCsf, showcasing the results with standard deviations is crucial to establish the statistical excellence of Contiformer. We regret for our oversight on this part. We have included the standard deviation result in the attached PDF file (Table 1 and Table 2).

Moreover, it's important to note that for the UEA task in Section 4.2, our reporting approach aligns with the prevalent practice in the field of time series analysis [1, 2], where the overall performance of the mean accuracy and mean ranking are reported and compared.

What's more, as pointed out by reviewer PCsf, with 3 repeats, it is not sure 2 std. is very convincing. Therefore, it is suggested to check the unpaired t-test table. We really appreciate the valuable suggestion. However, due to the rebuttal timeline, we only manage to perform a complete t-test on Synthetic and BookOrder datasets, using *10* random seeds over Contiformer and 6 baseline models. The significant test results can be found in the attached PDF file (Table 3 and Table 4 for Synthetic and BookOrder datasets, respectively.) which illustrate the significant improvement of our method.

[1] A transformer-based framework for multivariate time series representation learning. KDD 2021.
[2] Timesnet: Temporal 2d-variation modeling for general time series analysis. ICLR 2023.

> **Q3: More detailed illustration of how we achieve parallelization; why reparameterize the time horizon to [-1, 1]**

In L199-207 of our paper, we state that to preserve the parallelization of Transformer architecture and meanwhile implement the continuous-time attention mechanism in Eq. (6), we first adopt time variable ODE [1] to reparameterize ODEs into a single interval [−1, 1], followed by numerical approximation method to approximate the integrals. Here, we believe that it is novel to incorporate the time variable ODE to achieve the parallelization of our model. We would like to discuss more about this novelty below.

Originally, to calculate the ODE process from $t_1$ to $t_N$ for $L$ latent ODE trajectories, we need to iteratively forward from $t_1$ to $t_N$, which is hard to parallel. To resolve the problem, rather than applying the ODE solver directly modeling along the whole time horizon, we split the time horizon and reparameterize the time range $[t_i, t_j]$ ($1\leq i \leq j \leq N$) as $[-1, 1]$. Then, the single invoke of the ODE solver will be applied to these time ranges since these trajectory pieces share the same ODE function parameters in Eq. (1) of our paper. Through this novel way, we can parallelly calculate these ODE functions without sequentially calculating from $t_1$ to $t_N$.

We believe that this is aligned with the parallel execution of attention calculation in the vanilla Transformer model (applying the same attention function on the copied & stacked input sequence to enable parallel execution), and it will further facilitate the parallel ability of our Contiformer model.

[1] Ricky T. Q. Chen, et al. Neural spatio-temporal point processes. ICLR, 2021.


Meta-review: This paper introduces ContiFormer to handle irregular time series data. The innovation lies in a continuous-time attention module with the keys, queries and values are vector-valued functions indexed on time. Techniques such as neural ODE and natural cubic spline interpolations are used to compute these key, query and value vectors. Algorithms for speeding up the computations are also proposed. The ContiFormer model is evaluated on various time series tasks and datasets with very promising results.

Reviewers found the approach to be novel and the idea to be interesting in solving irregular time series modelling task. They raised a few major concerns regarding:
1. Missing baselines for some other time-series modelling approaches;
2. The computation time, after tricks presented in the paper, is still significantly higher.

In author feedback, the first concern is largely addressed due to the extra baseline results provided by the authors. The authors acknowledged the high run-time issue.

In final revision, I'd encourage the authors to include all the new baseline results as well as to make better clarifications regarding time complexity figures of the proposed approach.
Decision: Accept (poster)