SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8: Dataset Information of UEA.
Dataset	Ablation	# Train	# Test	Length	# Classes	# Variates
ArticulatoryWordRecognition	AWR	275	300	144	25	9
BasicMotions	BM	40	40	100	4	6
CharacterTrajectories	CT	1422	1436	182	20	3
DuckDuckGeese	DDG	60	40	270	5	1345
Epilepsy	EP	137	138	207	4	3
ERing	ER	30	270	65	6	4
FingerMovements	FM	316	100	50	2	28
HandMovementDirection	HMD	160	74	400	4	10
Handwriting	HW	150	850	152	26	3
Heartbeat	HB	204	205	405	2	61
JapaneseVowels	JV	270	370	29	9	12
Libras	LB	180	180	45	15	2
LSST	LSST	2459	2466	36	14	6
NATOPS	NATOPS	180	180	51	6	24
PEMS-SF	PEMS	267	173	144	7	963
PenDigits	PD	7494	3498	8	10	2
RacketSports	RS	151	152	30	4	6
SelfRegulationSCP1	SR	268	293	896	2	6
SpokenArabicDigits	SA	6599	2199	93	10	13
UWaveGestureLibrary	UGL	2238	2241	315	8	3
Table 9: Experimental result on UEA when  $30\%$  observations are dropped. (ODE. stands for ODE-RNN, N. CDE stands for Neural CDE, ACC. stands for Accuracy.)
Dataset	GRU-D	GRU-Δt	ODE.	CADN	N. CDE	S5	mTAN	TST	ContiFormer
AWR	0.8533	0.8956	0.8789	0.8056	0.9089	0.9011	0.8067	0.9722	0.9633
BM	0.9333	0.9417	0.8250	0.9583	0.9917	0.9833	0.9917	0.9667	0.9750
CT	0.9325	0.8558	0.9415	0.9575	0.9276	0.9610	0.9529	0.9742	0.9833
DDG	0.5867	0.5933	0.5067	0.4600	0.3400	0.5667	0.6000	0.6533	0.7267
EP	0.7995	0.8043	0.8454	0.8123	0.7657	0.9074	0.9203	0.9589	0.9324
ER	0.7062	0.7148	0.6296	0.8623	0.7543	0.8346	0.7123	0.9160	0.9210
FM	0.6167	0.6167	0.5667	0.6233	0.5867	0.6033	0.5433	0.6133	0.6000
HMD	0.3559	0.4099	0.5450	0.4505	0.3333	0.6171	0.4414	0.5946	0.4550
HW	0.0722	0.1251	0.0675	0.0925	0.1831	0.1302	0.0937	0.2196	0.2000
HB	0.7577	0.7593	0.7805	0.7431	0.7333	0.7431	0.7789	0.7398	0.7561
JV	0.9703	0.9550	0.9432	0.9703	0.9225	0.9676	0.9595	0.9865	0.9919
LB	0.5037	0.4741	0.5222	0.5938	0.7481	0.9216	0.5426	0.7519	0.8778
LSST	0.5589	0.5615	0.5715	0.5574	0.3940	0.6389	0.5307	0.5520	0.6004
NATOPS	0.9204	0.9315	0.8704	0.8944	0.8148	0.8500	0.9019	0.9352	0.9222
PEMS	0.7938	0.7669	0.8208	0.7958	0.7572	0.9133	0.8189	0.8420	0.8401
PD	0.9309	0.9222	0.9433	0.9402	0.8422	0.8110	0.9029	0.9520	0.9517
RS	0.7434	0.7281	0.7654	0.7325	0.6798	0.7500	0.7325	0.8158	0.8487
SR	0.8942	0.9124	0.9101	0.8259	0.8783	0.8737	0.8646	0.9044	0.9101
SA	0.9663	0.9660	0.9830	0.9809	0.8998	0.9291	0.9321	0.9744	0.9836
UGL	0.6729	0.6625	0.6917	0.7479	0.8219	0.8042	0.7344	0.8552	0.8135
Acc.(↑)	0.7284	0.7298	0.7304	0.7402	0.7142	0.7854	0.7381	0.8089	0.8126
Rank(↓)	6	5.75	5.45	5.4	6.85	4.4	5.7	2.75	2.4
Table 10: Experimental result on UEA when  $50\%$  observations are dropped. (ODE. stands for ODE-RNN, N. CDE stands for Neural CDE, ACC. stands for Accuracy.)
Dataset	GRU-D	GRU-Δt	ODE.	CADN	N. CDE	S5	mTAN	TST	ContiFormer
AWR	0.8811	0.8978	0.8989	0.8067	0.9133	0.9078	0.8067	0.9633	0.9633
BM	0.9417	0.8500	0.7667	0.9417	0.9583	0.9917	0.9583	0.9500	0.9583
CT	0.8872	0.9088	0.9506	0.9798	0.9285	0.9638	0.9431	0.9601	0.9798
DDG	0.5467	0.6000	0.4933	0.5467	0.3467	0.5667	0.6067	0.6533	0.6867
EP	0.7295	0.7899	0.7512	0.6728	0.7101	0.9012	0.9106	0.9589	0.9469
ER	0.7358	0.7432	0.5704	0.8527	0.7691	0.8284	0.6432	0.9012	0.8901
FM	0.5800	0.5967	0.5933	0.5933	0.6000	0.6033	0.5700	0.6067	0.6067
HMD	0.4099	0.4054	0.4685	0.4550	0.3243	0.5811	0.4099	0.5270	0.4595
HW	0.0769	0.0945	0.0588	0.1004	0.1855	0.1357	0.0737	0.1600	0.1757
HB	0.7642	0.7593	0.7707	0.7512	0.7333	0.7463	0.7789	0.7350	0.7610
JV	0.9604	0.9532	0.9126	0.9649	0.9180	0.9631	0.9595	0.9775	0.9856
LB	0.3796	0.4352	0.4537	0.5680	0.7315	0.6352	0.5259	0.6556	0.8537
LSST	0.5203	0.5292	0.5223	0.5222	0.3969	0.7204	0.5126	0.5151	0.5661
NATOPS	0.9185	0.9167	0.8444	0.8741	0.8019	0.8315	0.8778	0.8778	0.9222
PEMS	0.7823	0.7861	0.8170	0.7611	0.7534	0.9114	0.8015	0.7669	0.8401
PD	0.8245	0.7804	0.8563	0.8369	0.6389	0.6544	0.6898	0.8827	0.8799
RS	0.7412	0.7368	0.7127	0.7632	0.5526	0.7193	0.6601	0.7807	0.8355
SR	0.9135	0.9170	0.9044	0.8123	0.8760	0.8885	0.8658	0.8987	0.9067
SA	0.9583	0.9556	0.9759	0.9663	0.8907	0.9218	0.9254	0.9667	0.9767
UGL	0.6833	0.6583	0.6781	0.6531	0.8281	0.8042	0.7167	0.8490	0.8000
ACC.(↑)	0.7117	0.7157	0.7000	0.7211	0.6929	0.7638	0.7118	0.7793	0.7997
Rank(↓)	5.8	5.65	5.75	5.55	6.5	4.25	5.85	3.15	1.9
Table 11: Experimental result on UEA when  $70\%$  observations are dropped. (ODE. stands for ODE-RNN, N. CDE stands for Neural CDE, ACC. stands for Accuracy)
Dataset	GRU-D	GRU-Δt	ODE.	CADN	N. CDE	S5	mTAN	TST	ContiFormer
AWR	0.8689	0.8822	0.8433	0.7911	0.9178	0.8956	0.7956	0.9089	0.9511
BM	0.7583	0.7917	0.7167	0.9750	0.9000	0.9167	0.9417	0.9167	0.9167
CT	0.8433	0.8496	0.9471	0.9582	0.9241	0.9547	0.9009	0.9313	0.9675
DDG	0.5200	0.6000	0.4800	0.5667	0.3533	0.5667	0.6067	0.5867	0.7267
EP	0.7464	0.7222	0.6643	0.6667	0.6667	0.8802	0.9058	0.9420	0.9275
ER	0.6741	0.6617	0.5383	0.8647	0.7630	0.8025	0.7407	0.8198	0.8580
FM	0.5467	0.5933	0.5733	0.6000	0.5767	0.6133	0.5700	0.5300	0.6233
HMD	0.3829	0.3784	0.4459	0.4324	0.3378	0.5991	0.3694	0.4324	0.4144
HW	0.0784	0.0831	0.0541	0.0957	0.1533	0.1078	0.0604	0.1255	0.1247
HB	0.7610	0.7528	0.7593	0.7610	0.7561	0.7512	0.7691	0.7593	0.7593
JV	0.9514	0.9414	0.9261	0.9685	0.8964	0.9559	0.9613	0.9595	0.9766
LB	0.3648	0.3722	0.3463	0.5412	0.6796	0.6278	0.4926	0.5500	0.7648
LSST	0.4803	0.4895	0.4505	0.5278	0.4093	0.6778	0.4992	0.4874	0.5149
NATOPS	0.8722	0.8981	0.7889	0.9074	0.7944	0.8259	0.8463	0.8315	0.9204
PEMS	0.7418	0.7611	0.7842	0.7726	0.7303	0.8902	0.7861	0.6840	0.8227
PD	0.7451	0.6670	0.7802	0.7596	0.5201	0.5197	0.6063	0.8078	0.8069
RS	0.6711	0.6776	0.6316	0.7018	0.5570	0.6382	0.6228	0.6930	0.7697
SR	0.9090	0.9170	0.9158	0.8515	0.8726	0.8874	0.8635	0.8612	0.8953
SA	0.9304	0.9295	0.9523	0.9513	0.8622	0.9024	0.8925	0.9447	0.9600
UGL	0.6031	0.6219	0.5896	0.6729	0.8344	0.7896	0.6792	0.8219	0.7969
ACC.(↑)	0.6725	0.6795	0.6594	0.7183	0.6753	0.7401	0.6955	0.7297	0.7749
Rank(↓)	6.15	5.7	6.4	3.9	6.3	4.45	5.4	4.2	2.1
C.3 Predicting Irregular-Sampled Sequences
C.3.1 Background: Marked Temporal Point Process (MTPP)
The marked Temporal Point Process (MTPP) has been widely studied in recent years. We define a sequence of events and types as  $\Gamma = [(t_1, k_1), (t_2, k_2), \ldots, (t_N, k_N)]$ , where  $t_i$  is the time point when the  $i$ -th event occurs, with  $t_1 < t_2 < \ldots < t_N$ ,  $k_i \in \{1, 2, \ldots, \mathcal{K}\}$  is the type of the  $i$ -th event and  $\mathcal{K}$  is the total number of event types. A history event is the event sequence up to but not including
time  $t$ , denoted by  $\mathcal{H}_t = (t_j, k_j) : t_j < t$ . The goal of the temporal point process is to model the conditional intensity function (CIF), denoted by  $\lambda_k^*(t)$ , which represents the likelihood of an event occurring at time  $t$ , given the historical events. Models are trained so that the log probability of an event sequence, i.e.,
$$
\log p (\Gamma) = \sum_ {j = 1} ^ {N} \log \lambda_ {k _ {j}} ^ {*} (t _ {j}) - \int_ {t _ {1}} ^ {t _ {N}} \lambda^ {*} (\tau) d \tau . \tag {45}
$$
C.3.2 Dataset Description
We use one synthetic dataset and five real-world datasets, namely Synthetic, Neonate [50], Traffic [32], MIMIC [16], BookOrder [16] and StackOverflow [33] to evaluate our model.
Synthetic To evaluate the effectiveness of the proposed method, we create a synthetic dataset. Specifically, we consider 10 types of event where each type of event contains three properties, i.e.,  $(X_{i},V_{i},D_{i})$ . Hence, given an event  $\mathbf{e}$  with type  $j$  is represented as  $\mathbf{e} = (e.x,e.v,e.d,e.k)$ , where  $e.x \sim \mathcal{N}(X_j,\mu_1)$ ,  $e.v \sim \mathcal{N}(V_j,\mu_2)$ ,  $e.d = D_j > 0$  and  $e.k$  is the type of the event. We set  $\mu_1 = \mu_2 = 0.01$  and  $X,V,D \in \mathbb{R}^{10}$  is randomly generated and pre-defined.
To model both the correlation between events and the dynamic changes of events within the system. We first define the dynamic attention (weight) for each event, which is composed of inter-event dependency modeling and intra-event dynamic modeling. For inter-event dependency, it aims to calculate time-aware attention (weight) between each pair of events. Specifically, we use the following equation to define the weight between event  $\mathbf{e}$  and  $\hat{\mathbf{e}}$ ,
$$
\omega (\mathbf {e}, \hat {\mathbf {e}}) = (e. t - \hat {e}. t) \cdot (e. x * \hat {e}. v - \hat {e}. x * e. v). \tag {46}
$$
Next, the define  $\lambda_k^*(t)$  and  $\lambda^*(t)$  in which we assume that only the last event attributes to the decision of the next event, following Markov decision process (MDP), i.e.
$$
\lambda_ {k} ^ {*} (t) = P (k | \hat {e}. k) \cdot \text {S o f t p l u s} \left(\sum_ {\mathbf {e} \in \mathcal {H} _ {t}} \omega (\mathbf {e}, \hat {\mathbf {e}}) \cdot \exp \left(\frac {- (t - e . t)}{e . d}\right)\right), \tag {47}
$$
$$
\lambda^ {*} (t) = \text {S o f t p l u s} \left(\sum_ {\mathbf {e} \in \mathcal {H} _ {t}} \omega (\mathbf {e}, \hat {\mathbf {e}}) \cdot \exp \left(\frac {- (t - e . t)}{e . d}\right)\right).
$$
where  $\exp \left(-\left(t - e.t\right) / e.d\right)$  is the exponential decay effect for each event,  $\hat{\mathbf{e}}$  is the last event as of time  $t$ . The existence of the exponential decay function ensures that all the events will eventually disappear in the system as time goes to infinity so as to stabilize the conditional intensify function.  $P(k|\hat{k})$  is a pre-defined matrix that means the probability of the next event to with type  $k$  given that the type of the previous event is  $\hat{\mathbf{e}}$ .
Next, we predict the occurrence time of the next event with
$$
\widehat {t} _ {j + 1} = \int_ {t _ {j}} ^ {\infty} t \cdot p (t \mid \mathcal {H} _ {t}) d t, \tag {48}
$$
$$
\text {w h e r e} \quad p (t \mid \mathcal {H} _ {t}) = \lambda^ {*} (t) \exp \left(- \int_ {t _ {j}} ^ {t} \lambda^ {*} (t) d t\right).
$$
To predict the type of the next event, we sample the type of the next event from  $P(\cdot |\hat{e}.k)$ , where  $\hat{e}.k$  is the type of the previous event.
Finally, we use 500 different random seeds to generate a total of 500 event sequences. Also, we use 4-fold cross-validation for evaluation. The generation process is shown in Algorithm 1.
Neonate. Neonate dataset is a clinical dataset from 79 patients, among which 48 patients have incurred at least two seizures during the observation. The dataset contains only one type of event, i.e., the start point of a seizure (in minutes). The task is to predict when the next seizure will happen. Note that we only clear the start point of the seizure. Thus, we only have one type of event.
Algorithm 1 Generate Synthetic Dataset with Time-Aware Dynamic Kernel
Input seed  $\in [0,500)$ $tmax = 20$ $X = [0.78,0.11, - 1.0, - 0.56, - 0.78,1.0,0.56, - 0.33,0.3,$ $-0.11],V = [-0.17,0.5, - 0.28,0.28,0.17,0.39, - 0.05,0.05, - 0.39, - 0.5],D = [0.5,0.,0.28,$ $0.17,0.22,0.44,0.33,0.11,0.05,0.39]$    
Output A generated event sequence  $E$    
Initialize the random state of Numpy using seed and defined the probabilistic matrix  $P$    
Initialize the event sequence  $E$  to be an empty array, and set current time  $T = 0$    
Random generate the type of the first event e.k and sample e.x,e.v and e.d..   
Add the event to the event sequence..   
while  $t\leq tmax$  do Calculate the conditional intensity function using Eq.(47). Predict the time of the next event e.t using Eq.(48). Assume the type of previous event is  $\hat{k}$  , sample the type of the next event e.k from  $P(\cdot |\hat{k})$  Sample sample e.x,e.v and e.d given e.k.. Add the generated event to the event sequence  $E$  Update  $T = e.t$    
end while   
return E
Traffic. Traffic dataset is a benchmark dataset for time-series forecasting collected by the Caltrans Performance Measurement System (PeMS). This dataset contains the traffic state monitoring of 861 sensors in a two-year period. The data is collected every 1 hour. We extract time points with fluctuations higher than  $50\%$  as events, whose types are up and down according to the direction. We consider two events that are within 6 hours as duplicated events and remove the latter one from the dataset. Besides, for each sensor, it contains a long sequence, we partition these event sequences by month, i.e., for each event sequence, it contains the event of a sensor in one month.
MIMIC. MIMIC dataset is a clinical dataset that collects patients' visits to a hospital's ICU in a seven-year period. We treat the visits of each patient as a separate sequence.
BookOrder. BookOrder dataset is a financial dataset that contains transaction records of stock in one day. We record the time (in milliseconds) and the action that was taken in each transaction. The dataset is a single long sequence with only two types of events: "buy" and "sell". We clip the event sequences so that the maximum length is 500 in both the train and test sets. Furthermore, the mean  $\mu$  and standard derivative  $\sigma$  of the time difference between two events in the dataset is 1.32 and 20.24 respectively. Therefore, we clip the maximum time difference to 70.
StackOverflow. StackOverflow dataset is a question-answering website. The website rewards users with badges to promote engagement in the community, and the same badge can be rewarded multiple times to the same user. We collect data in a two-year period, and we treat each user's reward history as a sequence.
For MIMIC, Bookorder, and StackOverflow datasets, we use the dataset from the Google Drive<sup>10</sup>. Statistic information can be found in Table 12. We use the 4-fold cross-validation scheme for Synthetic, Neonate, and Traffic datasets following [17], and the 5-fold cross-validation scheme for the other three datasets following [39, 64].
C.3.3 Implementation Details
Input Embedding Given the history event  $\mathcal{H}_t = \{(t_j, k_j) : t_j < t\}$  with a total of  $\mathcal{K}$  events, the input of Transformer-based model and RNN-based model is obtained through event embedding and temporal encoding, i.e., for the  $i$ -th event  $x_i = \mathrm{emb}(k) + \mathrm{enc}(t)$ , where  $\mathrm{emb}(k)$  is the  $k$ -th column of the embedding table  $\mathcal{E} \in \mathbb{R}^{\mathcal{K} \times d}$  and temporal encoding is defined by a combination of sine and cosine function [64].
10https://drive.google.com/drive/folders/0BwqmVOEcoUc8UklIR1BKV25YR1U?resourcekey= 0-OrlU87jyc1m-dVMnY5aC4w
Table 12: Dataset statistics: name of the dataset, number of event types, number of events in the dataset, average length per sequence, and number of sequences in training/test sets.
Dataset	K	# Events	Avg. Length	# Train Seqs	# Test Seqs
Synthetic	10	8,618	17	375	125
Neonate	1	534	11	38	10
Traffic	2	154,747	60	1,938	647
MIMIC	75	2,419	4	527	65
BookOrder	2	414,800	2,074	90	100
StackOverflow	22	480,413	72	4,777	1,326
Conditional Intensity Function Given the output from the model, i.e.  $\mathbf{h}(t_1),\mathbf{h}(t_2),\dots,\mathbf{h}(t_N)$ . The dynamics of the temporal point process are described by a continuous conditional intensity function. To reduce the memory cost and accelerate the training process. Instead of using the continuous-time output from ContiFormer, we only use ContiFormer to generate a hidden representation for discrete time points, i.e.,  $t_1,t_2,\ldots ,t_j,\ldots ,t_N$ . Following [64], the conditional intensity function is defined by
$$
\lambda (t \mid \mathcal {H} _ {t}) = \sum_ {k = 1} ^ {K} \lambda_ {k} (t \mid \mathcal {H} _ {t}), \tag {49}
$$
where each of the type-specific conditional intensity function is defined by
$$
\lambda_ {k} (t \mid \mathcal {H} _ {t}) = f _ {k} \left(\alpha_ {k} \frac {t - t _ {j}}{t _ {j}} + \mathbf {w} _ {k} ^ {\top} \mathbf {h} (t _ {j}) + b _ {k}\right), \tag {50}
$$
where on interval  $t \in [t_j, t_{j+1})$  and  $f_k(x) = \beta_k \log (1 + \exp (x / \beta_k))$  is the softplus function with parameter  $\beta_k$ . The use of such a closed-form intensity function allows for  $O(1)$  interpolation.
Training Loss We train our model and all the baseline models by jointing maximizing the log-likelihood of the sequence and minimizing the prediction losses.
Specifically, for a sequence  $S$  over an interval  $[t_i, t_N]$ , given the conditional intensity function  $\lambda(t \mid \mathcal{H}_t)$ , the log-likelihood is given by
$$
\ell_ {L L} (\mathcal {S}) = \underbrace {\sum_ {j = 1} ^ {N} \log \lambda \left(t _ {j} \mid \mathcal {H} _ {j}\right)} _ {\text {e v e n t l o g - l i k e l i h o o d}} - \underbrace {\int_ {t _ {1}} ^ {t _ {N}} \lambda \left(t \mid \mathcal {H} _ {t}\right) d t} _ {\text {n o n - e v e n t l o g - l i k e l i h o o d}}. \tag {51}
$$
One challenge to compute the log-likelihood is to solve the integral, i.e.,  $\Lambda = \int_{t_1}^{t_N}\lambda (t\mid \mathcal{H}_t)dt$ . Since softplus function has no closed-form computation, we apply the Monte Carlo integration method, i.e.,
$$
\widehat {\Lambda} _ {\mathrm {M C}} = \sum_ {j = 2} ^ {P} \left(t _ {j} - t _ {j - 1}\right) \left(\frac {1}{P} \sum_ {i = 1} ^ {P} \lambda \left(u _ {i}\right)\right), \tag {52}
$$
where  $u_{i} \sim \mathrm{Unif}(t_{j-1}, t_{j})$  for  $i \in [1, \dots, P]$  is sampled from a uniform distribution over an interval  $[t_{j-1}, t_{j}]$ .
Besides the log-likelihood loss, we also use the cross-entropy loss and regression loss for predicting the type and time of the next events.
Specifically, given the conditional intensity function, the next time stamp prediction is given by
$$
p (t \mid \mathcal {H} _ {t}) = \lambda (t \mid \mathcal {H} _ {t}) \exp \left(- \int_ {t _ {j}} ^ {t} \lambda (\tau \mid \mathcal {H} _ {\tau}) d \tau\right), \tag {53}
$$
$$
\widehat {t} _ {j + 1} = \int_ {t _ {j}} ^ {\infty} t \cdot p (t \mid \mathcal {H} _ {t}) d t.
$$
Thus, the regression loss and the cross-entropy loss are given by
$$
\ell_ {r e g} (\mathcal {S}) = \sum_ {i = 2} ^ {N} \left(\widehat {t} _ {i} - t _ {i}\right) ^ {2}, \ell_ {p r e d} (\mathcal {S}) = \sum_ {i = 2} ^ {N} - \log \left(\frac {\lambda_ {k _ {i}} \left(\widehat {t} _ {i}\right)}{\lambda \left(\widehat {t} _ {i}\right)}\right). \tag {54}
$$
Overall, the modeling is trained in a multi-task manner, i.e.,
$$
\ell (\mathcal {S}) = \ell_ {L L} + \alpha_ {1} \ell_ {\text {r e g}} (\mathcal {S}) + \alpha_ {2} \ell_ {\text {p r e d}} (\mathcal {S}), \tag {55}
$$
where  $\alpha_{1}$  and  $\alpha_{2}$  are the pre-defined weight. By default, we set  $\alpha_{1} = 0.01$  and  $\alpha_{2} = 1$ .
C.3.4 Significance test on Synthetic and BookOrder datasets
To demonstrate the statistical superiority of ContiFormer, we conducted a t-test on Synthetic and BookOrder datasets using 10 random seeds. The t-test results are listed in Table 13.
Table 13: Significance test on Synthetic and BookOrder dataset with 10 repeats. The P-value represents the significant value between the ContiFormer and the baseline model. Bolder values represent P-value  $< 10^{-6}$  in significance test.
Dataset	Synthetic	BookOrder
Metric	LL (↑)	Accuracy (↑)	RMSE (↓)	LL (↑)	Accuracy (↑)	RMSE (↓)
Statistic	Mean	P-value	Mean	P-value	Mean	P-value	Mean	P-value	Mean	P-value	Mean	P-value
GRU-dt	-0.8611	5.38E-08	0.8420	0.003	0.2556	3.49E-09	-1.0137	1.07E-31	0.6274	3.11E-08	3.6680	1.32E-05
NSTKA	-1.0088	1.27E-21	0.8419	0.0003	0.3863	1.74E-12	-1.0916	2.95E-27	0.6223	5.48E-15	3.7175	1.88E-08
ODE-RNN	-0.8432	2.39E-11	0.8420	0.098	0.2401	3.77E-08	-0.9458	1.44E-26	0.6278	1.17E-07	3.5898	1.94E-03
RMTPP	-1.0132	3.57E-19	0.8420	0.0143	0.3711	6.70E-18	-0.9551	9.70E-26	0.6245	1.45E-13	3.6475	9.76E-04
SAHP	-0.6247	2.21E-05	0.8420	0.1778	0.5218	2.37E-10	-0.3053	1.83E-07	0.6227	1.45E-13	3.6810	9.55E-07
THP	-0.6035	6.28E-06	0.8420	0.1354	0.2100	4.85E-06	-1.0788	1.55E-17	0.6251	1.96E-06	3.6907	8.02E-07
ContiFormer	-0.5445	N/A	0.8421	N/A	0.1943	N/A	-0.2745	N/A	0.6295	N/A	3.6171	N/A
C.4 Regular Time Series Forecasting
Time series forecasting has garnered considerable attention in recent years, as evidenced by various studies [35, 53, 61]. In this study, we assess ContiFormer against several models tailored for regular time series forecasting.
Datasets We evaluate ContiFormer on five experimental datasets. i) ETT dataset [61] contains the data collected from electricity transformers, including load and oil temperature that are recorded every 15 minutes between July 2016 and July 2018. Here, we use ETTm2 and ETTh2 datasets. ii) Exchange dataset [53] records the daily exchange rates of eight different countries ranging from 1990 to 2016. iii) Weather dataset is recorded every 10 minutes for 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity, etc. v) ILI dataset includes the weekly recorded influenza-like illness (ILI) patients data from Centers for Disease Control and Prevention of the United States between 2002 and 2021.
Baselines We compare ContiFormer with several transformer-based models for regular time series forecasting under the multivariate setting. These transformer-based models include: FEDformer[62], Autoformer[53], Informer[61] and Reformer[31]. Also, we chose one recent state-of-the-art model. i.e. TimesNet[52] for comparison.
Results of Time Series Forecasting The experimental results are depicted in Table 14. ContiFormer achieves the best performance on ILI dataset. Specifically, it gives  $10\%$  MSE reduction  $(2.874 \rightarrow 2.632)$ . Also, ContiFormer outperforms Autoformer and FEDformer on Exchange, ETTm2, and Weather datasets. Therefore, it demonstrates that ContiFormer performs competitively with the state-of-the-art model.
C.5 Pendulum Regression Task
The objective of pendulum regression is to assess the models' capability in accommodating observations obtained at irregular intervals [45, 49]. The input sequence comprises 50 images, each