SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Two examples of generated time series are shown in the left two panels in Figure.3. We can easily observe that these time series exhibit changing frequencies and amplitudes along time.
Performance. We compare a strong baseline STG-NCDE [6] with our proposed model on this synthetic dataset with prediction horizon  $\{12, 48, 96\}$ . The dimension of hidden states, is searched from  $\{8, 16, 32, 64\}$  and we find the best setting is 16. The results are shown in Table. 1. It can be observed that Neural Lad outperforms STG-NCDE on both short and long-term forecasting. Moreover, our method improves it by  $45\%$  on the short-term horizon and by  $47\%$  on long-term horizon, both of which are large margins. We plot the predicted time series with horizon 90 by STG-NCDE and Neural Lad in the right two panels of Figure. 3. It can be observed that Neural Lad can fit well the overall shape of the time series.
Ablation study. We also conduct an ablation study by experimenting Neural Lad only with one component, periodicity  $h_w(t)$  or attention-based transformation of control signal  $g_v(dX/dt)$ , as shown in Table 1. We find that both of the two components play important roles for enhancing performance. Moreover, the attention-based transformation  $g_v$  contributes slightly more for this task.
4.2 Irregularly sampled time series classification: PhysioNet sepsis
One advantage of the neural ODE family, including Neural Lad, is that it naturally adapts to the case of irregularly sampled partially observed data. In this part, we apply our approach to the PhysioNet 2019 challenge on sepsis prediction [25], including 40,335 time series of variable length from the patients in ICU. Measurements are around 39-dimensional features with an hourly resolution, most of which are missing and only  $10.3\%$  of values are observed. The task is a binary classification problem to predict whether sepsis is developed during the patents' stay. We follow the experimental settings of [13], where the cases with or without observational intensity are both considered.
Since the datasets is highly imbalanced (5% positive rate), AUC is evaluated for different approaches, as reported in Table 2. We can observe that Neural Lad outperforms other alternatives when considering the observational intensity with low memory usage. Moreover, Neural Lad increases the memory less than 6% on both two classification tasks.
Table 2: Comparison between Neural Lad and other neural ODE variants on irregularly sampled dataset, PhysioNet sepsis classification.
MODEL	TEST AUC	MEMORY USAGE(MB)
INTENSITY	NO INTENSITY	INTENSITY	NO INTENSITY
GRU-ODE	0.852	0.771	454	273
GRU-Δt	0.878	0.840	837	826
GRU-D	0.871	0.850	889	878
ODE-RNN	0.874	0.833	696	686
NEURAL CDE	0.880	0.776	244	122
NEURAL LAD	0.897	0.812	257	129
Table 3: Univariate datasets: the long horizon forecasting performance. Bold font represents the best accuracy while the underlined means the second best.
Dataset	Horizon	ETTm1	ETTm2	ETTh2	Weather
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Repeat	96	1.214	0.665	0.266	0.328	0.432	0.422	0.259	0.254
192	1.261	0.609	0.340	0.371	0.534	0.473	0.309	0.292
336	1.283	0.707	0.412	0.410	0.591	0.508	0.377	0.338
720	1.319	0.729	0.521	0.465	0.588	0.517	0.465	0.394
Informer [40]	96	0.672	0.571	0.365	0.453	3.755	1.525	0.300	0.384
192	0.795	0.669	0.533	0.563	5.602	1.931	0.598	0.544
336	1.212	0.871	0.887	1.201	4.721	1.835	0.578	0.523
720	1.166	0.823	3.379	1.338	3.647	1.625	1.059	0.741
avg	0.961	0.734	1.410	0.810	4.431	1.729	0.634	0.548
Autoformer [34]	96	0.505	0.475	0.255	0.339	0.358	0.397	0.266	0.336
192	0.553	0.496	0.281	0.340	0.456	0.452	0.307	0.367
336	0.621	0.537	0.339	0.372	0.482	0.486	0.359	0.395
720	0.671	0.561	0.433	0.432	0.515	0.511	0.419	0.428
avg	0.558	0.517	0.327	0.371	0.450	0.459	0.338	0.382
FEDformer [41]	96	0.379	0.419	0.203	0.287	0.346	0.388	0.217	0.296
192	0.426	0.441	0.269	0.328	0.429	0.439	0.276	0.336
336	0.445	0.459	0.325	0.366	0.496	0.487	0.339	0.380
720	0.543	0.490	0.421	0.415	0.463	0.474	0.403	0.428
avg	0.448	0.452	0.305	0.349	0.437	0.449	0.309	0.360
Crossformer [39]	96	0.320	0.373	-	-	-	-	-	-
288	0.404	0.427	-	-	-	-	-	-
336	-	-	-	-	-	-	0.495	0.515
720	-	-	-	-	-	-	0.526	0.542
MICN [32]	96	-	-	0.179	0.275	-	-	0.161	0.229
192	-	-	0.307	0.376	-	-	0.220	0.281
336	-	-	0.325	0.388	-	-	0.278	0.331
720	-	-	0.502	0.490	-	-	0.311	0.356
avg	0.328	0.382	0.243	0.299
STG-NCDE	96	0.479	0.419	0.190	0.270	0.328	0.375	0.204	0.238
192	0.384	0.387	0.259	0.314	0.430	0.445	0.263	0.311
336	0.420	0.413	0.303	0.339	0.451	0.458	0.273	0.312
720	0.492	0.461	2.115	1.029	0.994	0.689	0.356	0.347
avg	0.444	0.420	0.717	0.488	0.551	0.492	0.274	0.302
DLinear [37]	avg	0.403	0.407	0.350	0.401	0.559	0.515	0.265	0.317
LightTS [38]	avg	0.435	0.437	0.409	0.436	0.602	0.543	0.261	0.312
TimesNet [33]	avg	0.400	0.406	0.291	0.333	0.414	0.427	0.259	0.287
Neural Lad	96	0.337	0.359	0.172	0.258	0.275	0.326	0.162	0.221
192	0.356	0.373	0.242	0.305	0.367	0.391	0.214	0.268
336	0.396	0.406	0.282	0.333	0.410	0.422	0.265	0.303
720	0.462	0.448	0.404	0.400	0.418	0.446	0.323	0.345
avg	0.388	0.396	0.275	0.324	0.368	0.396	0.241	0.284
4.3 Real-world univariate and multivariate time series prediction
Baseline methods. For long-horizon univariate time series forecasting, we compare Neural Lad with SCINet, Neural CDE, DLinear [37], LightTS [38], TimesNet [33], MICN [32], Transformer variants
Table 4: The multivariate forecasting performance of Neural Lad on PEMs datasets.
Model	PEMSD3	PEMSD4	PEMSD7	PEMSD8
MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
GraphWaveNet	19.12	32.77	24.89	38.66	26.39	41.50	18.28	30.05
MSTGCN	19.54	31.93	23.96	37.21	29.00	43.73	19.00	29.15
DCRNN	17.99	30.31	21.22	33.44	25.22	38.61	16.82	26.36
STGCN	17.55	30.42	21.16	34.89	25.33	39.34	17.50	27.09
ASTGCN	17.34	29.56	22.93	35.22	24.01	37.87	18.25	28.06
AGCRN	15.98	28.25	19.83	32.26	22.37	36.55	15.95	25.22
ST-GODE	16.50	27.84	20.84	32.82	22.59	37.54	16.81	25.97
Z-GCNETs	16.64	28.15	19.50	31.61	21.77	35.17	15.76	25.11
ST-WA	15.17	26.63	19.06	31.02	20.74	34.05	15.41	24.62
DSTAGNN	15.57	27.21	19.30	31.46	21.42	34.51	15.67	24.77
SCI-Net	14.98	24.08	19.27	31.27	21.19	34.03	15.72	24.76
STG-NCDE	15.57	27.09	19.21	31.09	20.53	33.84	15.45	24.81
Neural Lad (Ours)	14.67	24.59	18.98	30.94	20.21	33.78	15.28	24.66
including Informer [40], Autoformer [34], Fedformer [41], Crossformer [39]. For multivariate time series considering spatial correlation, we compare our method against other graph-based deep learning approaches, including STGCN [36], ASTGCN [11], MSTGCN [12], DCRNN [18], Graph Wavenet [35], AGCRN [2], ST-WA [7], DSTAGNN [16], Z-GCNETs [4], SCI-Net [20], ST-GODE [9] and STG-NCDE [6].
Univariate time series. Table 3 shows the performance comparison on long-term time-series forecasting. For fair competition, we only consider the temporal latent dynamics ignoring the spatial correlation in this experiment. Specifically, we generate the hidden state with Eq.(15) instead of Eq.(20). We can observe that our proposed method outperforms all the transformer baselines by a large margin. For the model FEDformer that performs better than all other transformer variants (Autoformer, Informer, and Pyraformer), Neural Lad outperforms it on all datasets. For two linear baselines, we improved the TimesNet [33] and DLinear [37] on all datasets with the average of different horizons. The paper Crossformer [39] conduct experiments on different horizons [24, 48, 168], therefore we take it as "-" if there are no results published in the origin paper. In the same way, for dataset (ETTm2 and ETTh2) without performance published in paper MICN [32], we fill it with "-". From the table 3, we can find that although Crossformer performs better at the short horizon, it fails on long horizon prediction. For TimesNet [33], Neural Lad outperform it on average of four horizons on all dataset.
Multivariate time series. Table 4 shows that the proposed Neural Lad achieves state-of-the-art performance on the four PEMS datasets on MAE metric, and also outperforms most baselines on MAPE and MSE metrics. For the baseline SCINet, although the proposed Neural Lad slightly reduces prediction error than which on PEMSD3 and PEMSD4 dataset, our method reduces MAE by  $4.6\%$  on PEMSD7 dataset and by  $2.8\%$  on PEMSD8 dataset. For neural ODE family baselines, even though STG-NCDE performs better than ST-GODE by a large margin, the proposed method improves STG-NCDE by  $5.8\%$ ,  $1.2\%$ ,  $1.6\%$ ,  $1.1\%$  on the four PEMS datasets, respectively. Besides, our method performs slightly better than ST-WA. Overall, Neural Lad achieves the best performance for multivariate time-series forecasting in Neural ODE family.
In this work, we have introduced a new variant to Neural ODE family, Neural Lad, particularly tailored for accurate time series modeling and forecasting. We conduct extensive experiments on both univariate, multivariate and irregularly sampled time series datasets, Neural Lad has achieved remarkable forecasting performance in all these scenarios. Extending materials about experiment settings, computation cost, and visualizations are presented in Appendix.
In the future, we plan to improve the training efficiency of Neural Lad when considering long lookback window. This might tie with how to conduct a faster autonomous ODE solver. Another direction is to propose more architecture designs for the involved neural networks in latent dynamics model.
[35] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI'19, page 1907-1913. AAAI Press, 2019.
[36] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In *IJCAI*, 2018.
[37] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.
[38] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures, 2022.
[39] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023.
[40] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106-11115. AAAI Press, 2021.
[41] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022), 2022.