SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\{\omega_{j}\}$  and  $\{\xi_j\}$  are quadrature weights and nodes such that  $-1\leq \xi_1 < \dots < \xi_k\leq 1$
A.3 Putting it all together
Given an irregular-sampled sequence,  $\Gamma = [(x_1,t_1),(x_2,t_2),\dots,(x_N,t_N)]$ , where  $x_{i}\in \mathbb{R}^{d}$  is the input feature of the  $i$ -th tokens happen at time  $t_i$  and  $N$  is the length of the sequence. We analyze the time and space complexity for an autoregressive task where the output of each observation is required for a particular classification or regression task. Specifically, for Eq. (3),  $\alpha_{i}(t_{1}),\alpha_{i}(t_{2}),\ldots ,\alpha_{i}(t_{N})$
is required to form the output of Transformer. Therefore, the key difficulty is to solve Eq. (3). By using the numerical approximation method, we have
$$
\boldsymbol {\alpha} _ {i} \left(t _ {j}\right) = \frac {\int_ {t _ {i}} ^ {t _ {j}} \mathbf {q} (\tau) \cdot \mathbf {k} _ {i} (\tau) ^ {\top} \mathrm {d} \tau}{t _ {j} - t _ {i}} = \frac {1}{2} \int_ {- 1} ^ {1} \tilde {\mathbf {q}} _ {i, j} (\tau) \cdot \tilde {\mathbf {k}} _ {i, j} (\tau) ^ {\top} \mathrm {d} \tau \approx \frac {1}{2} \sum_ {p = 1} ^ {P} \gamma_ {p} \tilde {\mathbf {q}} _ {i, j} \left(\xi_ {p}\right) \cdot \tilde {\mathbf {k}} _ {i, j} \left(\xi_ {p}\right) ^ {\top}, \tag {18}
$$
where  $\tilde{\mathbf{q}}_{i,j}$  and  $\tilde{\mathbf{K}}_{i,j}$  follows that:
$$
\tilde {\mathbf {q}} _ {i, j} (s) = \mathbf {q} \left(\frac {s \left(t _ {i} - t _ {j}\right) + t _ {i} + t _ {j}}{2}\right), \tag {19}
$$
$$
\tilde {\mathbf {k}} _ {i, j} (s) = \mathbf {q} _ {j} \left(\frac {s (t _ {i} - t _ {j}) + t _ {i} + t _ {j}}{2}\right).
$$
We use  $\hat{\mathbf{k}}_{i,j}$  to denote  $\frac{d\hat{\mathbf{k}}_{i,j}(s)}{ds}$ . Thus,  $\alpha_{i}(t_{j})$  can be solved by paralleling  $N^2$  ODE solvers. Specifically,  $\tilde{\mathbf{k}}_{i,j}(\xi_p)$  is given by
$$
\tilde {\mathbf {k}} _ {i, j} \left(\xi_ {1}\right), \tilde {\mathbf {k}} _ {i, j} \left(\xi_ {2}\right), \dots , \tilde {\mathbf {k}} _ {i, j} \left(\xi_ {p}\right) = \operatorname {O D E S o l v e r} \left(\mathbf {k} _ {j} \left(t _ {j}\right), \hat {\mathbf {k}} _ {i, j}, - 1, \xi_ {1}, \xi_ {2}, \dots , \xi_ {p}\right). \tag {20}
$$
As for  $\tilde{\mathbf{q}}_{i,j}(\xi_k)$ , since the input sequence is irregularly sampled and discrete in time, we propose to use interpolation methods [30] (e.g., linear interpolation or cubic interpolation) method on  $Q$  to construct a continuous-time query. Therefore  $\tilde{\mathbf{q}}_{i,j}(\xi_k)$  can be easily calculated in  $O(d)$  time complexity.
B Representation Power of ContiFormer
In the section, we mathematically prove that by curated designs of function hypothesis, many Transformer variants can be covered as a special case of ContiFormer. For simplification, we only discuss the self-attention mechanism, where  $Q, K, V \in \mathbb{R}^{N \times d}$ .
B.1 Transformer Variants
We first show several Transformer-based methods for irregular time series. Most of these methods shown below calculate  $N \times N$  attention matrix and multiply it with  $V^4$ .
Vanilla Transformer Given query  $Q \in \mathbb{R}^{N \times d}$  and  $V \in \mathbb{R}^{N \times d}$ , the attention matrix of a vanilla transformer [51] is calculated as
$$
\operatorname {A t t n} (Q, K) = \left[ \begin{array}{c c c c} Q _ {1} \cdot K _ {1} ^ {\top} & Q _ {1} \cdot K _ {2} ^ {\top} & \dots & Q _ {1} \cdot K _ {N} ^ {\top} \\ Q _ {2} \cdot K _ {1} ^ {\top} & Q _ {2} \cdot K _ {2} ^ {\top} & \dots & Q _ {2} \cdot K _ {N} ^ {\top} \\ \vdots & \vdots & \ddots & \vdots \\ Q _ {N} \cdot K _ {1} ^ {\top} & Q _ {N} \cdot K _ {2} ^ {\top} & \dots & Q _ {N} \cdot K _ {N} ^ {\top} \end{array} \right], \tag {21}
$$
and the output of the self-attention mechanism is obtained by
$$
Z _ {i} = \operatorname {S o f t m a x} (\operatorname {A t t n} (Q, K) _ {i}) V. \tag {22}
$$
Kernelized Attention Kernelized attention mechanism  $^6$  incorporate time information through defining specific kernel function  $k(x, x')$  and replace the attention matrix with
$$
\operatorname {A t t n} (Q, K) = \left[ \begin{array}{c c c c} Q _ {1} \cdot K _ {1} ^ {\top} * k \left(t _ {1}, t _ {1}\right) & Q _ {1} \cdot K _ {2} ^ {\top} * k \left(t _ {1}, t _ {2}\right) & \dots & Q _ {1} \cdot K _ {N} ^ {\top} * k \left(t _ {1}, t _ {N}\right) \\ Q _ {2} \cdot K _ {1} ^ {\top} * k \left(t _ {2}, t _ {1}\right) & Q _ {2} \cdot K _ {2} ^ {\top} * k \left(t _ {2}, t _ {2}\right) & \dots & Q _ {2} \cdot K _ {N} ^ {\top} * k \left(t _ {2}, t _ {N}\right) \\ \vdots & \vdots & \ddots & \vdots \\ Q _ {N} \cdot K _ {1} ^ {\top} * k \left(t _ {N}, t _ {1}\right) & Q _ {N} \cdot K _ {2} ^ {\top} * k \left(t _ {N}, t _ {2}\right) & \dots & Q _ {N} \cdot K _ {N} ^ {\top} * k \left(t _ {N}, t _ {N}\right) \end{array} \right]. \tag {23}
$$
4 Throughout the proof section, we define  $\mathbf{v}_i(t_i) = \mathbf{v}_i(t_i) = V_i$ , and focus on the proof of attention matrix.
For simplification, we assume single-head attention is adopted throughout the proof.
$^6$ Not to be confused with methods that regard kernels as a form of approximation of the attention matrix.
Kernel functions are typically symmetric and hold that  $\phi (x,x) = 1$  for any  $x$ . For instance, NSTKA [38] propose a learnable Generalized Spectral Mixture (GSM) kernel, i.e.,
$$
k _ {\mathrm {G S M}} \left(x, x ^ {\prime}\right) = \sum_ {i} \phi_ {i} (x) \phi_ {i} \left(x ^ {\prime}\right) k _ {\mathrm {G i b b s}, i} \left(x, x ^ {\prime}\right) \cos \left(2 \pi \left(\mu_ {i} (x) x - \mu_ {i} \left(x ^ {\prime}\right) x ^ {\prime}\right)\right),
$$
$$
\text {w h e r e} k _ {\text {G i b b s}, i} \left(x, x ^ {\prime}\right) = \sqrt {\frac {2 l _ {i} (x) l _ {i} \left(x ^ {\prime}\right)}{l _ {i} (x) ^ {2} + l _ {i} \left(x ^ {\prime}\right) ^ {2}}} \exp \left(- \frac {\left(x - x ^ {\prime}\right) ^ {2}}{l _ {i} (x) ^ {2} + l _ {i} \left(x ^ {\prime}\right) ^ {2}}\right). \tag {24}
$$
where  $\mu_{i}(x)$  denotes the mean, lengthscale as  $l_{i}(x)$  and input-dependent weight as  $\phi_i(x)$ .
Time Embedding Methods Time embedding methods learn a time representation, adding or concatenating it with the input embedding, i.e.,
$$
\widehat {\operatorname {A t t n}} (Q, K) = \left[ \begin{array}{c c c} \left(Q _ {1}, \Phi \left(t _ {1}\right)\right) \cdot \left(K _ {1}, \Phi \left(t _ {1}\right)\right) ^ {\top} & \dots & \left(Q _ {1}, \Phi \left(t _ {1}\right)\right) \cdot \left(K _ {N}, \Phi \left(t _ {N}\right)\right) ^ {\top} \\ \left(Q _ {2}, \Phi \left(t _ {2}\right)\right) \cdot \left(K _ {1}, \Phi \left(t _ {1}\right)\right) ^ {\top} & \dots & \left(Q _ {2}, \Phi \left(t _ {2}\right)\right) \cdot \left(K _ {N}, \Phi \left(t _ {N}\right)\right) ^ {\top} \\ \vdots & \ddots & \vdots \\ \left(Q _ {N}, \Phi \left(t _ {N}\right)\right) \cdot \left(K _ {1}, \Phi \left(t _ {1}\right)\right) ^ {\top} & \dots & \left(Q _ {N}, \Phi \left(t _ {N}\right)\right) \cdot \left(K _ {N}, \Phi \left(t _ {N}\right)\right) ^ {\top} \end{array} \right], \tag {25}
$$
where  $(\cdot ,\cdot)$  is the concatenation of two vectors,  $\Phi (\cdot):\mathbb{R}\to \mathbb{R}^d$  is a learnable time representation. For simplification, we assume that time embedding is concatenated with the input embedding and ignore linear transformation on the time embedding.7
For instance, Mercer [54] transforms the periodic kernel function into the high-dimensional space spanned by truncated Fourier basis under certain frequencies  $(\{\omega_{i}\})$  and coefficients  $\{c_i\}$ , i.e.,
$$
\Phi (t) \mapsto \Phi_ {d} ^ {\mathcal {M}} = \left[ \Phi_ {\omega_ {1}, d} ^ {\mathcal {M}} (t), \dots , \Phi_ {\omega_ {k}, d} ^ {\mathcal {M}} (t) \right] ^ {\top},
$$
$$
\text {w h e r e} \Phi_ {\omega} ^ {\mathcal {M}} (t) = \left[ \sqrt {c _ {1}}, \dots , \sqrt {c _ {2 j}} \cos \left(\frac {j \pi t}{\omega}\right), \sqrt {c _ {2 j + 1}} \sin \left(\frac {j \pi t}{\omega}\right), \dots \right]. \tag {26}
$$
mTAN [48] generalizes the notion of a positional encoding used in Transformer-based models to continuous-time domain, i.e.,
$$
\Phi (t) = \phi (t) \cdot \mathbf {w}, \tag {27}
$$
where  $\mathbf{w}\in \mathbb{R}^{d\times d}$  is a linear transformation 8 and
$$
\phi (t) _ {i} = \left\{ \begin{array}{l l} \omega_ {0} \cdot t + \phi_ {0}, & \text {i f} i = 0 \\ \sin \left(\omega_ {i} \cdot t + \phi_ {i}\right), & \text {i f} 0 <   i \leq d \end{array} , \right. \tag {28}
$$
where  $\omega$  and  $\phi$  are learnable weights. Furthermore, mTAN assumes that  $Q = K = \mathbf{0}$ .
Besides, since  $\operatorname{Softmax}(\mathbf{x} + c) = \operatorname{Softmax}(\mathbf{x})$ , where  $\mathbf{x}$  is a vector and  $c$  is a constant. The following equation holds,
$$
\operatorname {S o f t m a x} (\operatorname {A t t n} (Q, K)) = \operatorname {S o f t m a x} (\widehat {\operatorname {A t t n}} (Q, K)), \tag {29}
$$
where the matrix form of Softmax normalize each row of the matrix and
$\operatorname{Attn}(Q, K) =$
$$
\left[ \begin{array}{c c c} Q _ {1} \cdot K _ {1} ^ {\top} & \dots & Q _ {1} \cdot K _ {N} + \Phi (t _ {1}) \cdot (\Phi (t _ {N}) - \Phi (t _ {1})) ^ {\top} \\ Q _ {2} \cdot K _ {1} + \Phi (t _ {2}) \cdot (\Phi (t _ {2}) - \Phi (t _ {1})) ^ {\top} & \dots & Q _ {2} \cdot K _ {N} + \Phi (t _ {2}) \cdot (\Phi (t _ {N}) - \Phi (t _ {2})) ^ {\top} \\ \vdots & \ddots & \vdots \\ Q _ {N} \cdot K _ {1} + \Phi (t _ {N}) \cdot (\Phi (t _ {1}) - \Phi (t _ {N})) ^ {\top} & \dots & Q _ {N} \cdot K _ {N} ^ {\top} \end{array} \right]. \tag {30}
$$
$^7$ Using the concatenation operation, we have  $(Q_{i},\Phi (t_{i}))\cdot (K_{j},\Phi (t_{j}))^{\top} = Q_{i}\cdot K_{j}^{\top} + \Phi (t_{i})\cdot \Phi (t_{j})^{\top}$ $^8$  In the original paper, the equation is defined as  $\Phi (t_i,t_j) = \phi (t_i)\mathbf{w}\mathbf{v}^\top \phi (t_j)$ . For simplicity purpose, we assume that  $\mathbf{w} = \mathbf{v}$
B.2 Proof of Extension and Universal Approximation
Lemma 1 (Existence of Continuously Differentiable Vector Function). Given a continuously differentiable vector function  $u: \mathbb{R} \to \mathbb{R}^d \in \mathcal{C}^1$ , such that  $\forall x \in \mathbb{R}, \|u(x)\|_2 < \infty, \|u(x)\|_0 = d$  and a continuously differentiable function  $w: \mathbb{R} \to \mathbb{R} \in \mathcal{C}^1$ , there always exists another continuously differentiable vector function  $v: \mathbb{R} \to \mathbb{R}^d \in \mathcal{C}^1$  with a fixed point  $p$  satisfying  $u(p) \cdot v(p)^\top = w(p)$ , such that  $\forall x \in \mathbb{R}, u(x) \cdot v(x)^\top = w(x)$  holds.
Proof. Throughout the proof, we define  $\xi \coloneqq v(p)$ .
Since  $\forall x\in \mathbb{R},0 <   \| u(x)\| _2 <   \infty$  , we define
$$
v (x) = \frac {u (x)}{\| u (x) \| _ {2} ^ {2}} \cdot w (x) + k (x), \tag {31}
$$
where  $k(x) \in \mathcal{C}^1$  such that  $\forall x \in \mathbb{R}, u(x) \cdot k(x)^\top = 0$  holds. Therefore, for  $\forall x \in \mathbb{R}$ , we have
$$
u (x) \cdot v (x) ^ {\top} = u (x) \cdot \left(\frac {u (x)}{\| u (x) \| _ {2} ^ {2}} \cdot w (x) + k (x)\right) = w (x) + u (x) \cdot k (x) = w (x). \tag {32}
$$
Thus, we only need to prove the existence of  $k(x)$ . Since  $v(x)$  has a fixed point at  $p$ , therefore,
$$
k (p) = \xi - \frac {u (p)}{\| u (p) \| _ {2} ^ {2}} \cdot w (p). \tag {33}
$$
Let  $z[i]$  denote the  $i$ -th element of a vector  $z$ , thus, Eq. (33) is equivalent to
$$
k (p) [ i ] = \xi [ i ] - \frac {u (p) [ i ]}{\| u (p) \| _ {2} ^ {2}} * w (p) [ i ], \tag {34}
$$
for  $i\in [1,d]$ . Since we have  $u(p)\cdot v(p)^{\top} = w(p)$ , thus,
$$
k (p) \cdot u (p) ^ {\top} = \xi \cdot u (p) ^ {\top} - w (p) = 0. \tag {35}
$$
We denote  $c_{i} \coloneqq k(p)[i] * u(p)[i]$ . Since Eq. (35) holds, therefore  $\sum_{i=1}^{d} c_{i} = 0$ . Finally, we can construct  $k(x)$  as
$$
k (x) = \left[ \begin{array}{c} k (x) [ 1 ] \\ k (x) [ 2 ] \\ \vdots \\ k (x) [ d ] \end{array} \right] ^ {\top} = \left[ \begin{array}{c} c _ {1} / u (x) [ 1 ] \\ c _ {2} / u (x) [ 2 ] \\ \vdots \\ c _ {d} / u (x) [ d ] \end{array} \right] ^ {\top}. \tag {36}
$$
Since  $u, w \in \mathcal{C}^1$ , therefore  $k \in \mathcal{C}^1$ , thus  $v \in \mathcal{C}^1$ .
Lemma 2 (Existence of  $\pmb{k}_i(t)$ ). Given a continuously differentiable vector function  $\mathbf{q} : \mathbb{R} \to \mathbb{R}^d \in \mathcal{C}^1$  such that  $\forall x \in \mathbb{R}, \| \mathbf{q}(x)\|_2 < \infty, \| \mathbf{q}(x)\|_0 = d$  and a continuously differentiable function  $f : \mathbb{R} \to \mathbb{R} \in \mathcal{C}^2$  and a vector  $K_i \in \mathbb{R}^d$  and satisfies  $f(t_i) = \mathbf{q}(t_i) \cdot K_i^\top$ , there always exist a continuously differentiable vector function  $\mathbf{k}_i : \mathbb{R} \to \mathbb{R}^d$ , such that the following statements hold:
-  $\mathbf{k}_i(t_i) = K_i$
$\forall t\in \mathbb{R},t\neq t_i,\pmb {\alpha}_i(t)\coloneqq \frac{\int_{t_i}^t\mathbf{q}(\tau)\cdot\mathbf{k}_i(\tau)^\top d\tau}{t - t_i} = f(t).$
Proof. Since  $t \neq t_i$ , we have
$$
\alpha_ {i} (t) = f (t)
$$
$$
\Longleftrightarrow \boldsymbol {\alpha} _ {i} (t) (t - t _ {i}) = f (t) (t - t _ {i})
$$
$$
\stackrel {(1)} {\Longleftrightarrow} \frac {\mathrm {d} \boldsymbol {\alpha} _ {i} (t) \left(t - t _ {i}\right)}{\mathrm {d} t} = \frac {\mathrm {d} f (t) \left(t - t _ {i}\right)}{\mathrm {d} t} \tag {37}
$$
$$
\Longleftrightarrow \mathbf {q} (t) \cdot \mathbf {k} _ {i} (t) ^ {\top} = \frac {\mathrm {d} f}{\mathrm {d} t} (t - t _ {i}) + f (t),
$$
<sup>9</sup>Consider an open set  $\mathcal{U}$  on the real line and a function  $f$  defined on  $\mathcal{U}$  with real values. Let  $k$  be a nonnegative integer. The function  $f$  is said to be of differentiability class  $\mathcal{C}^k$  if the derivatives  $f', f'', \ldots, f^{(k)}$  exist and are continuous on  $\mathcal{U}$ .
where ① holds since we can define  $\alpha_{i}(t_{i}) = f(t_{i})$  . Besides, since
$$
\lim  _ {| \epsilon | \rightarrow 0} \boldsymbol {\alpha} _ {i} (t _ {i} + \epsilon) = \lim  _ {| \epsilon | \rightarrow 0} \frac {\int_ {t _ {i}} ^ {t _ {i} + \epsilon} \mathbf {q} (\tau) \cdot \mathbf {k} _ {i} (\tau) \mathrm {d} \tau}{\epsilon} = \mathbf {q} (t _ {i}) \cdot \mathbf {k} _ {i} (t _ {i}) ^ {\top} = \mathbf {q} (t _ {i}) \cdot K _ {i} ^ {\top}, \tag {38}
$$
such a definition does not break the continuity of  $\alpha$ .
Since  $f \in \mathcal{C}^2$ , therefore we denote  $g(t) \coloneqq \frac{\mathrm{d}f}{\mathrm{d}t}(t - t_i) + f(t) \in \mathcal{C}^1$ . Besides,
$$
g \left(t _ {i}\right) = f \left(t _ {i}\right) = \mathbf {q} \left(t _ {i}\right) \cdot K _ {i} ^ {\top} = \mathbf {q} \left(t _ {i}\right) \cdot \mathbf {k} _ {i} \left(t _ {i}\right) ^ {\top} \tag {39}
$$
By using Lemma 1 with  $g$  corresponding to  $w$ ,  $\mathbf{q}$  to  $u$ ,  $\mathbf{k}_i$  to  $v$  and  $p = t_i$ , we can prove the existence of  $\mathbf{k}_i$  that satisfies Eq. (37).
Theorem 2 (Universal Attention Approximation Theorem). Given query  $(Q)$  and key  $(K)$ , such that  $\| Q_i\| _2 < \infty$ ,  $\| Q_i\| _0 = d$  for  $i\in [1,\dots,N]$ . For any attention matrix with formulation defined in Appendix B.1, i.e.,  $\mathrm{Attn}(Q,K)$ , there always exists a family of continuously differentiable vector functions  $k_{1}(\cdot),k_{2}(\cdot),\ldots ,k_{N}(\cdot)$ , such that the discrete formulation of  $[\alpha_1(\cdot),\alpha_2(\cdot),\dots,\alpha_N(\cdot)]$  defined by
$$
\widetilde {\operatorname {A t t n}} (Q, K) = \left[ \begin{array}{c c c c} \boldsymbol {\alpha} _ {1} (t _ {1}) & \boldsymbol {\alpha} _ {2} (t _ {1}) & \dots & \boldsymbol {\alpha} _ {N} (t _ {1}) \\ \boldsymbol {\alpha} _ {1} (t _ {2}) & \boldsymbol {\alpha} _ {2} (t _ {2}) & \dots & \boldsymbol {\alpha} _ {N} (t _ {2}) \\ \vdots & \vdots & \ddots & \vdots \\ \boldsymbol {\alpha} _ {1} (t _ {N}) & \boldsymbol {\alpha} _ {2} (t _ {N}) & \dots & \boldsymbol {\alpha} _ {N} (t _ {N}) \end{array} \right], \tag {40}
$$
satisfies that  $\widetilde{\operatorname{Attn}}(Q, K) = \operatorname{Attn}(Q, K)$ , where
$$
\boldsymbol {\alpha} _ {i} (t) = \frac {\int_ {t _ {i}} ^ {t} \mathbf {q} (\tau) \cdot \mathbf {k} _ {i} (\tau) ^ {\top} d \tau}{t - t _ {i}}, i f t \neq t _ {i}, \tag {41}
$$
and  $\pmb{\alpha}_i(t_i) = \mathbf{q}(t_i)\cdot \mathbf{k}_i(t_i)^\top$ $\mathbf{q}(t)$  satisfies  $\mathbf{q}(t_i) = Q_i$ .
Proof. First, we show that the diagonal elements in Eq. (40), i.e.,
$$
\boldsymbol {\alpha} _ {i} \left(t _ {i}\right) = \mathbf {q} \left(t _ {i}\right) \cdot \mathbf {k} _ {i} \left(t _ {i}\right) ^ {\top} = Q _ {i} \cdot K _ {i} ^ {\top}, \tag {42}
$$
holds for different forms of attention matrix defined in Appendix B.1.
Next, for a given  $i \in [1, N]$ , there exists a continuous cubic spline  $h_i: \mathbb{R} \to \mathbb{R}$  satisfies  $h_i(t_j) = \operatorname{Attn}(Q, K)_{j,i}$  for  $\forall j \in [1, \dots, N]$  and  $h_i \in \mathcal{C}^2$ . By using Lemma 2 with  $h_i$  corresponding to  $f$ , there exists a function  $\mathbf{k}_i$  such that  $\alpha_i(t) = h_i(t)$ . Therefore,  $\alpha_i(t_j) = h_i(t_j)$  for  $j \in [1, \dots, N]$ , and we finish the proof.
C.1 Modeling Continuous-Time Function
C.1.1 Background: 2D Spiral
A spiral is a curve that starts from a central point and gradually moves away from it while turning around it. We consider two types of spirals that start from time  $s$  and end at time  $e$ . The first type of spiral is defined as
$$
x (t) = (a + b t) \cdot \cos (t), \tag {43}
$$
$$
y (t) = (a + b t) \cdot \sin (t),
$$
while the second type of spiral is defined as
$$
x (t) = \left(a + 5 0 b / (e - t)\right) \cdot \cos (e - t), \tag {44}
$$
$$
y (t) = (a + 5 0 b / (e - t)) \cdot s i n (e - t),
$$
where  $a$  and  $b$  are parameters of the Archimedean spiral.
Table 7: More experimental results on spiral 2d.
α	Metric	Transformer[51]	Neural ODE[9]	Latent ODE[9] (w/ RNN Enc)	Latent ODE[43] (w/ ODE Enc)	Neural CDE[42]	ContiFormer
Task = Interpolation (×10-2)
0.02	RMSE (↓)	1.37 ± 0.15	1.90 ± 0.17	2.09 ± 0.22	4.53 ± 1.34	4.80 ± 0.50	0.50 ± 0.06
MAE (↓)	1.42 ± 0.14	1.90 ± 0.17	1.95 ± 0.26	4.25 ± 1.48	5.12 ± 0.47	0.53 ± 0.07
0.2	RMSE (↓)	2.30 ± 0.51	1.88 ± 0.09	3.34 ± 0.10	4.82 ± 1.97	4.60 ± 1.08	0.42 ± 0.04
MAE (↓)	1.75 ± 0.18	1.58 ± 0.12	3.16 ± 0.43	4.73 ± 1.21	4.58 ± 0.93	0.42 ± 0.01
2	RMSE (↓)	3.75 ± 0.50	2.1 ± 0.21	3.04 ± 0.48	4.49 ± 1.26	2.92 ± 0.32	0.45 ± 0.11
MAE (↓)	2.33 ± 0.60	2.49 ± 0.30	2.88 ± 0.54	4.12 ± 1.00	2.96 ± 0.23	0.41 ± 0.07
Task = Extrapolation (×10-2)
0.02	RMSE (↓)	1.37 ± 0.11	2.07 ± 0.02	1.59 ± 0.05	2.69 ± 0.82	2.24 ± 0.03	0.64 ± 0.10
MAE (↓)	1.49 ± 0.12	2.02 ± 0.07	1.52 ± 0.09	2.83 ± 0.88	2.38 ± 0.04	0.65 ± 0.08
0.2	RMSE (↓)	2.96 ± 1.46	4.38 ± 0.35	4.78 ± 0.15	5.03 ± 1.96	3.81 ± 1.02	0.74 ± 0.10
MAE (↓)	2.79 ± 1.11	3.35 ± 0.43	4.02 ± 0.59	4.72 ± 1.96	3.37 ± 0.78	0.69 ± 0.09
2	RMSE (↓)	5.36 ± 0.97	6.02 ± 0.58	5.70 ± 0.57	4.68 ± 1.70	3.44 ± 1.06	0.75 ± 0.07
MAE (↓)	4.48 ± 0.82	4.84 ± 0.43	4.46 ± 0.60	4.29 ± 1.34	3.32 ± 0.85	0.69 ± 0.07
C.1.2 Experiment Setting
We generate 300 spirals and 200/100 spirals are used for training/testing respectively. For each spiral, it randomly belongs to either a clockwise spiral or a counter-clockwise spiral, we sample  $a \sim \mathcal{N}(0, \alpha)$ ,  $b \sim \mathcal{N}(0.3, \alpha)$  and we set  $\alpha = 0.02$  for Table 2. Also, we add Gaussian noise from  $\mathcal{N}(0, \beta)$  to all training samples and we set  $\beta = 0.1$ . We test different models for interpolation and extrapolation. Specifically, each spiral is sampled at 150 equally-spaced time points. To generate an irregular time series, we randomly sample 30 points from the first half of the trajectory. Thus, the first half of the trajectories are used for the interpolation task while the second half of the trajectories, which is totally unobserved during both training and testing, are used for the extrapolation task. All the models are trained for 5000 iterations with a batch size equal to 64.
C.1.3 Baseline Implementation
We compare ContiFormer with Neural ODE-based models and Transformer.
- Neural ODE[9]. This model uses the hidden vectors from the last observation as input, and obtain both the interpolation and extrapolation result from a single ODE solver. We train the model by minimizing the mean square loss.
- Latent ODE (w/RNN Enc)[9]. We run an RNN encoder through the time series and infer the parameters for a posterior over  $z_{t_0}$ . Next, we sample  $\mathbf{z}_{t_0} \sim q(\mathbf{z}_{t_0} \mid \{\mathbf{x}_{t_i}, t_i\})$ . Finally, we obtain both the interpolation and extrapolation result from a single ODE solver, i.e., ODESolve  $(\mathbf{z}_{t_0}, f, \theta_f, t_0, \dots, t_M)$ , where  $(t_0, t_1, \dots, t_M)$  is the time of the trajectory. We train the model by maximizing the evidence lower bound (ELBO).
- Latent ODE (w/ ODE Enc)[43]. We replace the RNN encoder in Latent ODE (w/ RNN Enc) with the ODE-RNN encoder following [43]. Again, we train the model by maximizing the evidence lower bound (ELBO).
- Neural CDE[42]. We follow the official implementation to construct the CDE function. For the interpolation, the hidden vectors are directly obtained from the continuous-time path from Neural CDE. For the extrapolation, we input the hidden vectors from the last timestamp of the Neural CDE to a single ODE solver. We train the model by minimizing the mean square loss.
- Transformer[51]. In order to obtain the prediction for both seen and unseen time points, we use a learnable mask for unseen inputs. Instead of using position embedding, the input is added with a temporal encoding before forwarding it to the transformer layer. We train the model by minimizing the mean square loss.
- ContiFormer. We use linear interpolation to deal with unseen observations to avoid the numerical error caused by constructing the continuous-time query. Similar to Transformer, temporal encoding is adopted and the model is trained by minimizing the mean square loss.
C.1.4 Experimental Results with Different  $\alpha$
In this subsection, we test Transformer, Neural ODE and ContiFormer with different  $\alpha$  to show the robustness of these models. As shown in Table 7, our model can achieve the best result on different parameters.
C.1.5 More Visualization Results
More visual results with  $\alpha = 0.02$  can be found in Figure 4.
Figure 4: More visualization results with  $\alpha = 0.02$ . Here, Latent ODE refers to Latent ODE w/ RNN Encoder [9].
C.2 Irregular Time Series Classification
C.2.1 Training Details
For both our model and the baseline models, we adopted a fixed learning rate of  $10^{-2}$  and a batch size of 64. To ensure the reliability of our results, all experiments were conducted three times, and the reported values represent the mean. The training process for all models lasted for 1000 epochs. However, if the training accuracy failed to show any improvement for 100 consecutive epochs, the training was stopped to avoid unnecessary computation.
C.2.2 Dataset Information
We select 20 datasets from UEA Time Series Classification Archive with diverse characteristics in terms of the number, dimensionality and length of time series samples, as well as the number of classes. Detailed information is listed in Table 8.
C.2.3 Full Experimental Results
We follow the setting of [30] to randomly drop either  $30\%$ ,  $50\%$  or  $70\%$  observations. Experimental results are shown in Table 9, Table 10 and Table 11. We can find that ContiFormer outperforms RNN-based methods, ODE-based methods and Transformer-based methods. Also, we observe that ContiFormer outperforms all the baselines by a large margin when  $70\%$  observations are dropped, which confirms that ContiFormer is more suitable for irregular time series modeling.