SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuqi Chen $^{1,2}$ , Kan Ren $^{2\dagger}$ , Yansen Wang $^{2}$ , Yuchen Fang $^{2,3}$ , Weiwei Sun $^{1}$ , Dongsheng Li $^{2}$
<sup>1</sup> School of Computer Science & Shanghai Key Laboratory of Data Science, Fudan University
$^{2}$  Microsoft Research Asia,  $^{3}$  Shanghai Jiao Tong University
Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.
Irregular time series are prevalent in real-world applications like disease prevention, financial decision-making, and earthquake prediction [4, 27, 64]. Their distinctive properties set them apart from regular time series data. First, irregular time series data are characterized by irregularly generated or non-uniformly sampled observations with variable time intervals, as well as missing data due to technical issues, or data quality problems, which pose challenges for traditional time series analysis techniques [6, 55]. Second, even though the observations are irregularly sampled, the underlying data-generating process is assumed to be continuous [30, 42, 43]. Third, the relationships among the observations can be intricate and continuously evolving. All these characteristics require elaborate modeling approaches for better understanding these data and making accurate predictions.
The continuity and intricate dependency of these data samples pose significant challenges for model design. Simply dividing the timeline into equally sized intervals can severely damage the continuity of the data [36]. Recent works have suggested that underlying continuous-time process is appreciated for irregular time series modeling [6, 30, 39, 43], which requires the modeling procedure to capture
*The work was conducted during the internship of Yuqi Chen and Yuchen Fang at Microsoft Research.
$\dagger$  Correspondence to Kan Ren, contact: rk.ren@outlook.com.
the continuous dynamic of the system. Furthermore, due to the continuous nature of data flow, we argue that the correlation within the observed data is also constantly changing over time. For instance, stock prices of tech giants (e.g., MSFT and GOOG) show consistent evolving trends, yet they are affected by short-term events (e.g., large language model releases) and long-term factors.
To tackle these challenges, researchers have pursued two main branches of solutions. Neural ordinary differential equations (Neural ODEs) and state space models (SSMs) have illustrated promising abilities for capturing the dynamic change of the system over time [9, 19, 28, 49]. However, these methods overlook the intricate relationship between observations and their recursive nature can lead to cumulative errors if the number of iterations is numerous [9]. Another line of research capitalizes on the powerful inductive bias of neural networks, such as various recurrent neural networks [6, 39, 45] and Transformer models [38, 48, 54, 60]. However, their use of fixed-time encoding or learning upon certain kernel functions fails to capture the complicated input-dependent dynamic systems.
Based on the above analysis, modeling the relationship between observations while capturing the temporal dynamics is a challenging task. To address this problem, we propose a Continuous-Time Transformer, namely ContiFormer, which incorporates the modeling abilities of continuous dynamics of Neural ODEs within the attention mechanism of Transformers and breaks the discrete nature of Transformer models. Specifically, to capture the dynamics of the observations, ContiFormer begins by defining latent trajectories for each observation in the given irregularly sampled data points. Next, to capture the intricate yet continuously evolving relationship, it extends the discrete dot-product in Transformers to a continuous-time domain, where attention is calculated between continuous dynamics. With the proposed attention mechanism and Transformer architecture, ContiFormer effectively models complex continuous-time dynamic systems.
The contributions of this paper can be summarized below.
- Continuous-Time Transformer. To the best of our knowledge, we are the first to incorporate a continuous-time mechanism into attention calculation in Transformer, which is novel and captures the continuity of the underlying system of the irregularly sampled time-series data.
- Parallelism Modeling. To tackle the conflicts between continuous-time calculation and the parallel calculation property of the Transformer model, we propose a novel reparameterization method, allowing us to parallelly execute the continuous-time attention in the different time ranges.
- Theoretical Analysis. We mathematically characterize that various Transformer variants [34, 38, 48, 54, 60] can be viewed as special instances of ContiFormer. Thus, our approach offers a broader scope that encompasses Transformer variants.
- Experiment Results. We examine our method on various irregular time series settings, including time-series interpolation, classification, and prediction. The extensive experimental results have illustrated the superior performance of our method against strong baseline models.
Time-Discretized Models. There exists a branch of models based on the time-discrete assumption which transforms the time-space into a discrete one. Recurrent Neural Networks (RNN) [14, 24] and Transformers [51] are powerful time-discrete sequence models, which have achieved great success in natural language processing [15, 37, 58], computer vision [20, 63] and time series forecasting [31, 35, 52, 53, 56, 61, 62]. Utilizing them for irregular time series data necessitates discretizing the timeline into time bins or padding missing values, potentially resulting in information loss and disregarding inter-observation dynamics. [13, 36, 41]. An alternative is to construct models that can utilize these various time intervals [34, 60]. Additionally, several recent approaches have also encoded time information into features to model irregularly sampled time series, including time representation approaches [54, 60] and kernel-based methods [38], which allow learning for time dynamics. For instance, Mercer [54] concatenates the event embedding with the time representation. Despite these advances, these models are still insufficient in capturing input-dependent dynamic systems.
Continuous-Time Models. Other works shift to the continuous-time modeling paradigm. To overcome the discrete nature of RNNs, a different strategy involves the exponential decay of the hidden state between observations [6, 39]. Nevertheless, the applicability of these techniques is restricted to decaying dynamics specified between successive observations [9]. Neural ODEs are a class of
models that leverage the theory of ordinary differential equations to define the dynamics of neural networks [9, 27]. Neural CDE [30] and Neural RDE [40] further utilize the well-defined mathematics of controlled differential equations [2, 29] and rough path theory [5] to construct continuous-time RNN. ODE-RNN [43] combines Neural ODE with Gated Recurrent Unit (GRU) [12] to model continuous-time dynamics and input changes. Whereas these approaches usually fail to capture the evolving relationship between observations [45]. CADN [11] and ANCDE [26] extend ODE-RNN and Neural CDE by incorporating an attention mechanism to capture the temporal relationships, respectively. Another line of research explores the use of state space models (SSMs) to represent continuous-time dynamic systems [1, 18, 19, 49]. However, these models adopt a recursive schema, which hampers its efficiency and can lead to cumulative errors [9]. In contrast, our proposed model, ContiFormer leverages the parallelism advantages of the Transformer architecture and employs a non-autoregressive paradigm. Furthermore, extending the Transformer to the continuous-time domain remains largely unexplored by researchers. Such extension provides a more flexible and powerful modeling ability for irregular time series data.
The ContiFormer is a Transformer [51] model for processing time series data with irregular time intervals. Formally, an irregular time series is defined as  $\Gamma = [(X_1,t_1),\ldots ,(X_N,t_N)]$ , where observations may occur at any time and the observation time points  $\omega = (t_{1},\dots ,t_{N})$  are with irregular intervals.  $X_{i}$  is the input feature for the  $i$ -th observation. We denote  $X = [X_{1};X_{2};\dots,X_{N}]\in \mathbb{R}^{N\times d}$ .
As aforementioned, it is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle the challenge, we propose a Continuous-Time Transformer architecture as shown in Figure 1. Each layer of ContiFormer takes as input an irregular time series  $X$  and the sampled time  $\omega$  and outputs a latent continuous trajectory that captures the dynamic change of the underlying system. With such a design, ContiFormer transforms the discrete observation sequence into the continuous-time domain. At each layer, the core attention module takes a continuous perspective and expands the dot-product operation in the vanilla Transformer to the continuous-time domain, which not only models the underlying continuous dynamics but also captures the evolving input-dependent process. For better understanding, the description of each layer will omit the layer index, without causing confusion.
Throughout the remaining section, we adopt the notation  $\omega$  to denote a sequence of (reference) time points, while  $t$  is the random variable representing a query time point. Matrices are donated using uppercase letters, and lowercase letters except  $t$  represent continuous functions.
3.1 Continuous-Time Attention Mechanism
The core of the ContiFormer layer is the proposed continuous-time multi-head attention (CT-MHA) module, as shown in Figure 1. At each layer, we first transform the input irregular time series  $X$  into  $Q = [Q_{1};Q_{2};\ldots ;Q_{N}]$  for queries,  $K = [K_{1};K_{2};\ldots ;K_{N}]$  for keys, and  $V = [V_{1};V_{2};\ldots ;V_{N}]$  for values. At a high level, the CT-MHA module transforms the irregular time series inputs to latent trajectories and outputs a continuous dynamic system that captures the time-evolving relationship between observations. To accomplish this, it utilizes ordinary differential equations to define the latent trajectories for each observation. Within the latent state, it assumes that the underlying dynamics evolve following linear ODEs. Subsequently, it constructs a continuous query function by approximating the underlying sample process of the input. Ultimately, it produces a continuous-time function that captures the evolving relationship and represents the complex dynamic system.
Continuous Dynamics from Observations Transformer calculates scaled dot-product attention for queries, keys, and values [51]. In continuous form, we first employ ordinary differential equations to define the latent trajectories for each observation. Specifically, assuming that the first observation and last observation come at time point  $t_1$  and  $t_N$  respectively, we define continuous keys and values as
$$
\begin{array}{l} \mathbf {k} _ {i} \left(t _ {i}\right) = K _ {i}, \quad \mathbf {k} _ {i} (t) = \mathbf {k} _ {i} \left(t _ {i}\right) + \int_ {t _ {i}} ^ {t} f (\tau , \mathbf {k} _ {i} (\tau); \theta_ {k}) d \tau , \tag {1} \\ \mathbf {v} _ {i} (t _ {i}) = V _ {i}, \quad \mathbf {v} _ {i} (t) = \mathbf {v} _ {i} (t _ {i}) + \int_ {t _ {i}} ^ {t} f (\tau , \mathbf {v} _ {i} (\tau); \theta_ {v}) d \tau , \\ \end{array}
$$
Figure 1: Architecture of the ContiFormer layer. ContiFormer takes an irregular time series and its corresponding sampled time points as input. Queries, keys, and values are obtained in continuous-time form. The attention mechanism (CT-MHA) performs a scaled inner product in a continuous-time manner to capture the evolving relationship between observations, resulting in a complex continuous dynamic system. Feedforward and layer normalization are adopted, similar to the Transformer. Finally, a sampling trick is employed to make ContiFormer stackable. Note that the highlighted trajectories in purple indicate the part of functions that are involved in the calculation of the output.
where  $t \in [t_1, t_N]$ ,  $\mathbf{k}_i(\cdot), \mathbf{v}_i(\cdot) \in \mathbb{R}^d$  represent the ordinary differential equation for the  $i$ -th observation with parameters  $\theta_k$  and  $\theta_v$ , and the initial state of  $\mathbf{k}_i(t_i)$  and  $\mathbf{v}_i(t_i)$  respectively and the function  $f(\cdot) \in \mathbb{R}^{d+1} \to \mathbb{R}^d$  controls the change of the dynamics.
Query Function While keys and values are associated with the input and output of the attention mechanism, a query specifies what information to extract from the input. To model a dynamic system, queries can be modeled as a function of time that represents the overall changes in the input to the dynamic system at a specific time point  $t$ , i.e.,  $\mathbf{q}(t)$ . Specifically, we adopt a common assumption that irregular time series is a discretization of an underlying continuous-time process. Thus, similar to [30], we define a closed-form continuous-time interpolation function (e.g., natural cubic spline) with knots at  $t_1, \ldots, t_N$  such that  $\mathbf{q}(t_i) = Q_i$  as an approximation of the underlying process.
Scaled Dot Product The self-attention mechanism is the key component in Transformer architecture. At its core, self-attention involves calculating the correlation between queries and keys. This is achieved through the inner product of two matrices in discrete form, i.e.,  $\boldsymbol{Q} \cdot \boldsymbol{K}^{\top}$ . Extending the discrete inner-product to its continuous-time domain, given two real functions  $f(x)$  and  $g(x)$ , we define the inner product of two functions in a closed interval  $[a,b]$  as
$$
\langle f, g \rangle = \int_ {a} ^ {b} f (x) \cdot g (x) d x. \tag {2}
$$
Intuitively, it can be thought of as a way of quantifying how much the two functions "align" with each other over the interval. Inspired by the formulation of the inner product in the continuous-time domain, we model the evolving relationship between the  $i$ -th sample and the dynamic system at time point  $t$  as the inner product of  $\mathbf{q}$  and  $\mathbf{k}_i$  in a closed interval  $[t_i,t]$ , i.e.,
$$
\boldsymbol {\alpha} _ {i} (t) = \frac {\int_ {t _ {i}} ^ {t} \mathbf {q} (\tau) \cdot \mathbf {k} _ {i} (\tau) ^ {\top} \mathrm {d} \tau}{t - t _ {i}}. \tag {3}
$$
Due to the nature of sequence data, it is common to encounter abnormal points, such as events with significantly large time differences. To avoid numeric instability during training, we divide the
integrated solution by the time difference. As a consequence, Eq. (3) exhibits discontinuity at  $\alpha_{i}(t_{i})$ . To ensure the continuity of the function  $\alpha_{i}(\cdot)$ , we define  $\alpha_{i}(t_{i})$  as
$$
\boldsymbol {\alpha} _ {i} \left(t _ {i}\right) = \lim  _ {\epsilon \rightarrow 0} \frac {\int_ {t _ {i}} ^ {t _ {i} + \epsilon} \mathbf {q} (\tau) \cdot \mathbf {k} _ {i} (\tau) ^ {\top} \mathrm {d} \tau}{\epsilon} = \mathbf {q} \left(t _ {i}\right) \cdot \mathbf {k} _ {i} \left(t _ {i}\right) ^ {\top}. \tag {4}
$$
Expected Values Given a query time  $t \in [t_1, t_N]$ , the value of an observation at time point  $t$  is defined as the expected value from  $t_i$  to  $t$ . Without loss of generality, it holds that the expected values of an observation sampled at time  $t_i$  is  $\mathbf{v}_i(t_i)$  or  $V_i$ . Formally, the expected value is defined as
$$
\widehat {\mathbf {v}} _ {i} (t) = \mathbb {E} _ {t \sim [ t _ {i}, t ]} \left[ \mathbf {v} _ {i} (t) \right] = \frac {\int_ {t _ {i}} ^ {t} \mathbf {v} _ {i} (\tau) \mathrm {d} \tau}{t - t _ {i}}. \tag {5}
$$
Multi-Head Attention The continuous-time attention mechanism is a powerful tool used in machine learning that allows for the modeling of complex, time-varying relationships between keys, queries, and values. Unlike traditional attention mechanisms that operate on discrete time steps, continuous-time attention allows for a more fine-grained analysis of data by modeling the input as a continuous function of time. Specifically, given the forehead-defined queries, keys, and values in continuous-time space, the continuous-time attention given a query time  $t$  can be formally defined as
$$
\operatorname {C T - A T T N} (Q, K, V, \omega) (t) = \sum_ {i = 1} ^ {N} \widehat {\alpha} _ {i} (t) \cdot \widehat {\mathbf {v}} _ {i} (t), \tag {6}
$$
$$
\text {w h e r e} \hat {\alpha} _ {i} (t) = \frac {\exp \left(\alpha_ {i} (t) / \sqrt {d _ {k}}\right)}{\sum_ {j = 1} ^ {N} \exp \left(\alpha_ {j} (t) / \sqrt {d _ {k}}\right)}.
$$
Multi-head attention, an extension of the attention mechanism [51], allows simultaneous focus on different input aspects. It stabilizes training by reducing attention weight variance. We extend Eq. (6) by incorporating multiple sets of attention weights, i.e.,
$$
\operatorname {C T - M H A} (Q, K, V, \omega) (t) = \operatorname {C o n c a t} \left(\operatorname {h e a d} _ {(1)} (t), \dots , \operatorname {h e a d} _ {(\mathrm {H})} (t)\right) W ^ {O},
$$
$$
\text {w h e r e} \quad \operatorname {h e a d} _ {\left(\mathrm {h}\right)} (t) = \operatorname {C T - A T T N} \left(Q W _ {\left(\mathrm {h}\right)} ^ {Q}, K W _ {\left(\mathrm {h}\right)} ^ {K}, V W _ {\left(\mathrm {h}\right)} ^ {V}, \omega\right) (t), \tag {7}
$$
where  $W^{O}$ ,  $W_{(\mathrm{h})}^{Q}$ ,  $W_{(\mathrm{h})}^{K}$  and  $W_{(\mathrm{h})}^{V}$  are parameter matrices,  $\mathrm{h} \in [1, \mathrm{H}]$  and  $\mathrm{H}$  is the head number.
3.2 Continuous-Time Transformer
Despite the widespread adoption of Transformers [51] in various research fields, their extension for modeling continuous-time dynamic systems is underexplored. We propose ContiFormer that directly builds upon the original implementation of vanilla Transformer while extending it to the continuous-time domain. Specifically, we apply layer normalization (LN) after both the multi-head self-attention (MHA) and the feed-forward blocks (FFN). We formally characterize the ContiFormer layer below
$$
\tilde {\mathbf {z}} ^ {l} (t) = \operatorname {L N} \left(\operatorname {C T - M H A} \left(X ^ {l}, X ^ {l}, X ^ {l}, \omega^ {l}\right) (t) + \mathbf {x} ^ {l} (t)\right), \tag {8}
$$
$$
\mathbf {z} ^ {l} (t) = \mathrm {L N} \left(\mathrm {F F N} (\tilde {\mathbf {z}} ^ {l} (t)) + \tilde {\mathbf {z}} ^ {l} (t)\right),
$$
where  $\mathbf{z}^l (t)$  is the output from the  $l$ -th ContiFormer layer at time point  $t$ . Additionally, we adopt residual connection to avoid potential gradient vanishing [23]. To incorporate the residual connection with the continuous-time output, we approximate the underlying process of the discrete input  $X^l$  using a continuous function  $\mathbf{x}^l (t)$  based on the closed-form continuous-time interpolation function.
Sampling Process As described before, each ContiFormer layer derives a continuous function  $\mathbf{z}^l(t)$  w.r.t. time as the output, while receiving discrete sequence  $X^l$  as input. However,  $\mathbf{z}^l(t)$  can not be directly incorporated into neural network architectures that expect inputs in the form of fixed-dimensional vectors and sequences [48], which places obstacles when stacking layers of ContiFormer. To address this issue, we establish reference time points for the output of each layer. These points are used to discretize the layer output, and can correspond to either the input time points (i.e.,  $\omega$ ) or task-specific time points. Specifically, assume that the reference points for the  $l$ -th layer is  $\omega^l = [t_1^l, t_2^l, \dots, t_{\beta_l}^l]$ , the input to the next layer  $X^{l+1}$  can be sampled as  $\{\mathbf{z}^l(t_j^l) | j \in [1, \beta_l]\}$ .
3.3 Complexity Analysis
We consider an autoregressive task where the output of each observation is required for a particular classification or regression task. Therefore, the reference points for each layer are defined as  $\omega^l = \omega$ .
To preserve the parallelization of Transformer architecture and meanwhile implement the continuous-time attention mechanism in Eq. (6), we first adopt time variable ODE [8] to reparameterize ODEs into a single interval  $[-1, 1]$ , followed by numerical
approximation method to approximate the integrals, i.e.,
Table 1: Per-layer complexity (Comp. per Layer), minimum number of sequential operations (Seq. Op.), and maximum path lengths (Max. Path Len.).  $N$  is the sequence length,  $d$  is the representation dimension,  $T$  is the number of function evaluations (NFE) for the ODE solver in a single forward pass from  $t_1$  to  $t_N$ , and  $S$  represents the NFE from  $-1$  to  $1$ .
Model	Comp. per Layer	Seq. Op.	Max. Path Len.
Transformer [51]	O(N2·d)	O(1)	O(1)
RNN [25]	O(N·d2)	O(N)	O(N)
Neural ODE [9]	O(T·d2)	O(T)	O(T)
ContiFormer	O(N2·S·d2)	O(S)	O(1)
$$
\boldsymbol {\alpha} _ {i} (t _ {j}) = \frac {\int_ {t _ {i}} ^ {t _ {j}} \mathbf {q} (\tau) \cdot \mathbf {k} _ {i} (\tau) ^ {\top} \mathrm {d} \tau}{t _ {j} - t _ {i}} = \frac {1}{2} \int_ {- 1} ^ {1} \tilde {\mathbf {q}} _ {i, j} (\tau) \cdot \tilde {\mathbf {k}} _ {i, j} (\tau) ^ {\top} \mathrm {d} \tau \approx \frac {1}{2} \sum_ {p = 1} ^ {P} \gamma_ {p} \tilde {\mathbf {q}} _ {i, j} (\xi_ {p}) \cdot \tilde {\mathbf {k}} _ {i, j} (\xi_ {p}) ^ {\top}, \tag {9}
$$
where  $P$  denotes the number of intermediate steps to approximate an integral,  $\xi_p \in [-1, 1]$  and  $\gamma_p \geq 0$ ,  $\tilde{\mathbf{k}}_{i,j}(s) = \mathbf{k}_i((s(t_j - t_i) + t_i + t_j) / 2)$ , and  $\tilde{\mathbf{q}}_{i,j}(s) = \mathbf{q}_i((s(t_j - t_i) + t_i + t_j) / 2)$ . Finally,  $\alpha_i(t_j)$  can be solved with one invokes of ODESolver that contains  $N^2$  systems, i.e.,
$$
\underbrace {\left[ \begin{array}{c} \tilde {\mathbf {k}} _ {1 , 1} \left(\xi_ {p - 1}\right) \\ \vdots \\ \tilde {\mathbf {k}} _ {N , N} \left(\xi_ {p - 1}\right) \end{array} \right]} _ {\mathbf {K} \left(\xi_ {p - 1}\right)} + \int_ {\xi_ {p - 1}} ^ {\xi_ {p}} f (s, \mathbf {K} (s); \theta_ {k}) d s = \underbrace {\left[ \begin{array}{c} \tilde {\mathbf {k}} _ {1 , 1} \left(\xi_ {p}\right) \\ \vdots \\ \tilde {\mathbf {k}} _ {N , N} \left(\xi_ {p}\right) \end{array} \right]} _ {\mathbf {K} \left(\xi_ {p}\right)}, \tag {10}
$$
where  $p \in [1, \dots, P]$  and we further define  $\xi_0 = -1$ . Additionally,  $\tilde{\mathbf{q}}_{i,j}(\xi_p)$  can be obtained by interpolating on the close-form continuous-time query function. Similar approach is adopted for solving  $\widehat{\mathbf{v}}_i(t_j)$  in Eq. (5). The detailed implementation is deferred to Appendix A.