SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider three criteria to analyze the complexity of different models, i.e., per-layer complexity, minimum number of sequential operations, and maximum path lengths [51]. Complexity per layer refers to the computational resources required to process within a single layer of a neural network. Sequential operation refers to the execution of operations that are processed iteratively or in a sequential manner. Maximum path length measures the longest distance information must traverse to establish relationships, reflecting the model's ability to learn long-range dependencies.
As noted in Table 1, vanilla Transformer is an efficient model for sequence learning with  $O(1)$  sequential operations and  $O(1)$  path length because of the parallel attention mechanism, whereas the recurrent nature of RNN and Neural ODE rely on autoregressive property and suffer from a large number of sequential operations. Utilizing Eq. (9) and leveraging the parallelism inherent in the Transformer architecture, our model, ContiFormer, achieves  $O(S)$  sequential operations and  $O(1)$  path length, while also enjoying the advantage of capturing complex continuous-time dynamic systems. We have  $S \ll T$  and we generally set  $S < N$  in our experiments.
3.4 Representation Power of ContiFormer
In the previous subsections, we introduce a novel framework for modeling dynamic systems. Then a natural question is: How powerful is ContiFormer? We observe that by choosing proper weights, ContiFormer can be an extension of vanilla Transformer [51]. Further, many Transformer variants tailored for irregular time series, including time embedding methods [48, 54] and kernelized attention methods [34, 38], can be special cases of our model. We provide an overview of the main theorem below and defer the proof to Appendix B.
Theorem 1 (Universal Attention Approximation Theorem). Given query  $(Q)$  and key  $(K)$  matrices, such that  $\| Q_i\| _2 < \infty$ ,  $\| Q_i\| _0 = d$  for  $i\in [1,\dots,N]$ . For certain attention matrix, i.e.,
Figure 2: Interpolation and extrapolation of spirals with irregularly-samples time points by Transformer, Neural ODE, and our model.
$\operatorname{Attn}(Q, K) \in \mathbb{R}^{N \times N}$  (see Appendix B.1 for more information), there always exists a family of continuously differentiable vector functions  $\mathbf{k}_1(\cdot), \mathbf{k}_2(\cdot), \ldots, \mathbf{k}_N(\cdot)$ , such that the discrete definition of the continuous-time attention formulation, i.e.,  $[\pmb{\alpha}_1(\cdot), \pmb{\alpha}_2(\cdot), \dots, \pmb{\alpha}_N(\cdot)]$  in Eq. (4), given by
$$
\widetilde {\operatorname {A t t n}} (Q, K) = \left[ \begin{array}{c c c c} \boldsymbol {\alpha} _ {1} \left(t _ {1}\right) & \boldsymbol {\alpha} _ {2} \left(t _ {1}\right) & \dots & \boldsymbol {\alpha} _ {N} \left(t _ {1}\right) \\ \boldsymbol {\alpha} _ {1} \left(t _ {2}\right) & \boldsymbol {\alpha} _ {2} \left(t _ {2}\right) & \dots & \boldsymbol {\alpha} _ {N} \left(t _ {2}\right) \\ \vdots & \vdots & \ddots & \vdots \\ \boldsymbol {\alpha} _ {1} \left(t _ {N}\right) & \boldsymbol {\alpha} _ {2} \left(t _ {N}\right) & \dots & \boldsymbol {\alpha} _ {N} \left(t _ {N}\right) \end{array} \right], \tag {11}
$$
satisfies that  $\widetilde{\mathrm{Attn}} (Q,K) = \mathrm{Attn}(Q,K)$
In this section, we evaluate ContiFormer on three types of tasks on irregular time series data, i.e., interpolation and extrapolation, classification, event prediction, and forecasting. Additionally, we conduct experiments on pendulum regression task [19, 45], the results are listed in Appendix C.5.
Implementation Details By default, we use the natural cubic spline to construct the continuous-time query function. The vector field in ODE is defined as  $f(t,\mathbf{x}) = \mathrm{Actfn}(\mathrm{LN}(\mathrm{Linear}^{d,d}(\mathrm{Linear}^{d,d}(\mathbf{x}) + \mathrm{Linear}^{1,d}(t))))$ , where  $\mathrm{Actfn}(\cdot)$  is either tanh or sigmoid activation function,  $\mathrm{Linear}^{a,b}(\cdot): \mathbb{R}^a \to \mathbb{R}^b$  is a linear transformation from dimension  $a$  to dimension  $b$ , LN denotes the layer normalization. We adopt the Gauss-Legendre Quadrature approximation to implement Eq. (9). In the experiment, we choose the Runge-Kutta-4 [44] (RK4) algorithm to solve the ODE with a fixed step of fourth order and a step size of 0.1. Thus, the number of forward passes to integrate from -1 to 1, i.e.,  $S$  in Table 1, is 80. All the experiments were carried out on a single 16GB NVIDIA Tesla V100 GPU.
4.1 Modeling Continuous-Time Function
The first experiment studies the effectiveness of different models from different categories on continuous-time function approximation. We generated a dataset of 300 2-dimensional spirals, sampled at 150 equally-spaced time points. To generate irregular time points, we randomly sample 50 points from each subsampled trajectory without replacement. We visualize the interpolation and extrapolation results of the irregular time series for different models, namely Latent ODE (w/RNN encoder) [9], Transformer [51] and our proposed
ContiFormer. Besides, we list the interpolation and extrapolation results in Table 2 using rooted mean squared error (RMSE) and mean absolute error (MAE) metrics. More visualization results and experimental settings can be found in Appendix C.1. As shown in Figure 2, both Latent ODE and ContiFormer can output a smooth and continuous function approximation, while Transformer fails to interpolate it given the noisy observations (①). From Table 2, we can observe that our model outperforms both Transformer and Latent ODE by a large margin. The improvement lies in two
Table 2: Interpolation and extrapolation results of different models on 2-dimensional spirals.  $\downarrow$  indicates the lower the better.
Metric	Transformer [51]	Latent ODE [9]	ContiFormer
Task = Interpolation (×10-2)
RMSE (↓)	1.37 ± 0.14	2.09 ± 0.22	0.49 ± 0.06
MAE (↓)	1.42 ± 0.13	1.95 ± 0.25	0.52 ± 0.06
Task = Extrapolation (×10-2)
RMSE (↓)	1.36 ± 0.10	1.59 ± 0.05	0.64 ± 0.09
MAE (↓)	1.49 ± 0.12	1.52 ± 0.05	0.65 ± 0.08
Table 3: Experimental results on irregular time series classification. Avg. ACC. stands for average accuracy over 20 datasets and Avg. Rank stands for average ranking over 20 datasets.  $\uparrow (\downarrow)$  indicates the higher (lower) the better.
Mask Ratio	30% Dropped	50% Dropped	70% Dropped
Metric	Avg. ACC. (↑)	Avg. Rank (↓)	Avg. ACC. (↑)	Avg. Rank (↓)	Avg. ACC. (↑)	Avg. Rank (↓)
GRU-D [30]	0.7284	6	0.7117	5.8	0.6725	6.15
GRU-Δt [6]	0.7298	5.75	0.7157	5.65	0.6795	5.7
ODE-RNN [43]	0.7304	5.45	0.7000	5.55	0.6594	6.4
Neural CDE [30]	0.7142	6.85	0.6929	6.5	0.6753	6.3
mTAN [48]	0.7381	5.7	0.7118	5.85	0.6955	5.4
CADN [10]	0.7402	5.4	0.7211	5.55	0.7183	3.9
S5 [49]	0.7854	4.4	0.7638	4.25	0.7401	4.45
TST [57]	0.8089	2.75	0.7793	3.15	0.7297	4.2
ContiFormer	0.8126	2.4	0.7997	1.9	0.7749	2.1
aspects. First, compared to Transformer, our ContiFormer can produce an almost $^3$  continuous-time output, making it more suitable for modeling continuous-time functions. Second, compared to Latent ODE, our ContiFormer excels at retaining long-term information (2), leading to lower prediction error in extrapolating unseen time series. Conversely, Latent ODE is prone to cumulative errors (3), resulting in poorer performance in extrapolation tasks. Therefore, we conclude that ContiFormer is a more suitable approach for modeling continuous-time functions on irregular time series.
4.2 Irregularly-Sampled Time Series Classification
The second experiment evaluates our model on real-world irregular time series data. To this end, we first examine the effectiveness of different models for irregular time series classification. We select 20 datasets from UEA Time Series Classification Archive [3] with diverse characteristics in terms of the number, dimensionality, and length of time series samples. More detailed information and experimental results can be found in Appendix C.2. To generate irregular time series data, we follow the setting of [30] to randomly drop either  $30\%$ ,  $50\%$  or  $70\%$  observations.
We compare the performance of ContiFormer with RNN-based methods (GRU-  $\Delta t$  [30], GRU-D [6]), Neural ODE-based methods (ODE-RNN [43], CADN [10], Neural CDE [30]), SSM-based models (S5 [49]), and attention-based methods (TST [57], mTAN [48]).
Table 3 presents the average accuracy and rank number under different drop ratios. ContiFormer outperforms all the baselines on all three settings. The complete results are shown in Appendix C.2.3. Moreover, attention-based methods perform better than both RNN-based methods and ODE-based methods, underscoring the significance of effectively modeling the intercorrelation of the observations. Additionally, we find that our method is more robust than other models. ContiFormer exhibits the smallest performance gap between  $30\%$  and  $50\%$ .
Besides, we study the learned attention patterns with different models. Attention captures the impact of previous observations on the future state of a system by measuring the correlation between different observations at different time points. Consequently, it is expected that a model
Figure 3: Visualization of attention scores on UWaveGestureLibrary dataset. Colors indicate the attention scores for different instances at time  $t = 0$ . Observations at time  $t = 0$  are observed and normalize the time interval to [0, 1].
effectively capturing the underlying dynamics would exhibit continuous, smooth, and input-dependent intermediate outcomes. As illustrated in Figure 3, ContiFormer and TST are both capable of
3The output of ContiFormer can be considered "continuous" only if we overlook the approximation and numerical errors in the ODESolver. Additionally, traditional Transformers are commonly trained with dropout to prevent overfitting, which can result in discontinuous outputs. To mitigate this issue, a straightforward approach is to set the dropout rate to 0, thereby producing an "almost" continuous output for ContiFormer.
Table 4: Prediction result of compared models for event prediction. LL for log-likelihood and ACC for accuracy.  $\uparrow (\downarrow)$  indicates the higher (lower) the better. (bold values indicate the best performance and  $^+$  indicates outperforming the best baseline by at least 3 standard deviations.)
Model	Metric	Synthetic	Neonate	Traffic	MIMIC	StackOverflow	BookOrder
HP [22]	LL (↑)	-3.084 ± .005	-4.618 ± .005	-1.482 ± .005	-4.618 ± .005	-5.794 ± .005	-1.036 ± .000
Accuracy (↑)	0.756 ± .000	-	0.570 ± .000	0.795 ± .000	0.441 ± .000	0.604 ± .000
RMSE (↓)	0.953 ± .000	10.957 ± .012	0.407 ± .000	1.021 ± .000	1.341 ± .000	3.781 ± .000
RMTPP [16]	LL (↑)	-1.025 ± .030	-2.817 ± .023	-0.546 ± .012	-1.184 ± .023	-2.374 ± .001	-0.952 ± .007
Accuracy (↑)	0.841 ± .000	-	0.805 ± .002	0.823 ± .004	0.461 ± .000	0.624 ± .000
RMSE (↓)	0.369 ± .014	9.517 ± .023	0.337 ± .001	0.864 ± .017	0.955 ± .000	3.647 ± .003
NeuralHP [39]	LL (↑)	-1.371 ± .004	-2.795 ± .012	-0.643 ± .004	-1.239 ± .027	-2.608 ± .000	-1.104 ± .005
Accuracy (↑)	0.841 ± .000	-	0.759 ± .001	0.814 ± .001	0.450 ± .000	0.621 ± .000
RMSE (↓)	0.631 ± .002	9.614 ± .013	0.358 ± .001	0.846 ± .007	1.022 ± .000	3.734 ± .003
SAHP [54]	LL (↑)	-0.619 ± .063	-2.646 ± .057	0.372 ± .022	-1.110 ± .030	-2.404 ± .002	-0.304 ± .002
Accuracy (↑)	0.841 ± .000	-	0.780 ± .001	0.830 ± .004	0.455 ± .000	0.622 ± .000
RMSE (↓)	0.521 ± .055	9.403 ± .060	0.337 ± .000	0.851 ± .006	0.963 ± .000	3.676 ± .011
THP [64]	LL (↑)	-0.589 ± .017	-2.702 ± .045	-0.569 ± .015	-1.137 ± .038	-2.354 ± .001	-1.102 ± .052
Accuracy (↑)	0.841 ± .000	-	0.818 ± .001	0.834 ± .005	-0.468 ± .000	0.622 ± .004
RMSE (↓)	0.205 ± .006	9.471 ± .045	0.332 ± .000	0.843 ± .017	0.951 ± .000	3.688 ± .004
NSTKA [38]	LL (↑)	-1.001 ± .025	-2.747 ± .061	-0.667 ± .027	-1.188 ± .012	-2.406 ± .003	-1.098 ± .019
Accuracy (↑)	0.842 ± .000	-	0.767 ± .001	0.833 ± .003	0.465 ± .000	0.621 ± .000
RMSE (↓)	0.379 ± .015	9.502 ± .049	0.346 ± .001	0.842 ± .008	0.956 ± .000	3.731 ± .007
GRU-Δt [9]	LL (↑)	-0.871 ± .050	-2.736 ± .031	-0.613 ± .062	-1.164 ± .026	-2.389 ± .002	-0.915 ± .006
Accuracy (↑)	0.841 ± .000	-	0.800 ± .004	0.832 ± .007	0.466 ± .000	0.627 ± .000
RMSE (↓)	0.249 ± .013	9.421 ± .050	0.335 ± .001	0.850 ± .010	0.950 ± .000	3.666 ± .016
ODE-RNN [43]	LL (↑)	-1.032 ± .102	-2.732 ± .080	-0.491 ± .011	-1.183 ± .028	-2.395 ± .001	-0.988 ± .006
Accuracy (↑)	0.841 ± .000	-	0.812 ± .000	0.827 ± .006	0.467 ± .000	0.624 ± .000
RMSE (↓)	0.342 ± .030	9.289 ± .048	0.334 ± .000	0.865 ± .021	0.952 ± .000	3.605 ± .004
mTAN [48]	LL (↑)	-0.920 ± .036	-2.722 ± .026	-0.548 ± .023	-1.149 ± .029	-2.391 ± .002	-0.980 ± .004
Accuracy (↑)	0.842 ± .000	-	0.811 ± .002	0.832 ± .009	0.466 ± .000	0.620 ± .000
RMSE (↓)	0.286 ± .008	9.363 ± .042	0.334 ± .001	0.848 ± .006	0.950 ± .000	3.680 ± .015
ContiFormer	LL (↑)	-0.535 ± .028+	-2.550 ± .026	0.635 ± .019+	-1.135 ± .023	-2.332 ± .001+	-0.270 ± .010+
Accuracy (↑)	0.842 ± .000	-	0.822 ± .001+	0.836 ± .006	0.473 ± .000+	0.628 ± .001+
RMSE (↓)	0.192 ± .005	9.233 ± .033	0.328 ± .001+	0.837 ± .007	0.948 ± .000+	3.614 ± .020
capturing input-dependent patterns effectively, while mTAN struggles due to the fact that its attention mechanism depends solely on the time information. In addition, ContiFormer excels in capturing complex and smooth patterns compared to TST, showcasing its ability to model time-evolving correlations effectively and potentially more robust to noisy data. Consequently, we believe that ContiFormer better captures the nature of irregular time series.
4.3 Predicting Irregular Event Sequences
Next, we evaluate different models for predicting the type and occurrence time of the next event with irregular event sequences [7, 21, 46, 47], a.k.a, marked temporal point process (MTPP) task. To this end, we use one synthetic dataset and five real-world datasets for evaluation, namely Synthetic, Neonate [50], Traffic [32], MIMIC [16], BookOrder [16] and StackOverflow [33]. We use the 4-fold cross-validation scheme for Synthetic, Neonate, and Traffic datasets following [17], and the 5-fold cross-validation scheme for the other three datasets following [39, 64]. We repeat the experiment 3 times and report the mean and standard deviations in Table 4. More information about the task description, dataset information, and training algorithm can be found in Appendix C.3.
We first compare ContiFormer with 3 models on irregular time series, i.e., GRU-Δt, ODE-RNN, and mTAN, which respectively belong to RNN-based, ODE-based, and attention-based approaches. Additionally, we compare ContiFormer with baselines specifically designed for the MTPP task. These models include parametric method (HP [22]), RNN-based methods (RMTPP [16], NeuralHP [39]), and attention-based methods (SAHP [59], THP [64], NSTKA [38]).
The experimental results, as presented in Table 4, demonstrate the overall statistical superiority of ContiFormer to the compared baselines. Notably, the utilization of specific kernel functions, e.g., NSTKA and mTAN, results in subpar performance on certain datasets, highlighting their limitations in modeling complex data patterns in real-world scenarios. Furthermore, parametric methods like HP, perform poorly on most datasets, indicating their drawbacks in capturing real-life complexities.
Table 5: Results of regular time series forecasting. Input length for ILI is 36 and 96 for the others. Prediction lengths for ETTm2 and ETTh2 are {24, 48, 168, 336}, {96, 192, 336, 540} for Exchange and Weather, and {24, 36, 48, 60} for ILI. Results are averaged over the four prediction lengths. Bold/underlined values indicate the best/second-best. MSE stands for mean squared error.
Model	ContiFormer	TimesNet[52]	DLinear[56]	FEDformer[62]	Autoformer[53]	Transformer[51]
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Exchange	0.339	0.399	0.320	0.396	0.277	0.377	0.392	0.446	0.440	0.477	1.325	0.949
ETTm2	0.226	0.300	0.214	0.276	0.220	0.305	0.226	0.300	0.231	0.307	0.689	0.633
ETTh2	0.356	0.395	0.321	0.364	0.358	0.396	0.351	0.393	0.352	0.394	2.081	1.179
ILI	2.632	1.042	2.874	1.066	4.188	1.493	4.467	1.547	4.046	1.446	5.130	1.507
Weather	0.257	0.290	0.259	0.285	0.260	0.311	0.315	0.360	0.318	0.359	0.672	0.579
4.4 Regular Time Series Forecasting
Lastly, we evaluate ContiFormer's performance in the context of regular time series modeling. To this end, we conducted an extensive experiment following the experimental settings from [53]. We employed the ETT, Exchange, Weather, and ILI datasets for time series forecasting tasks. We compare ContiFormer with the recent state-of-the-art model (TimesNet [52]), Transformer-based models (FEDformer[62], Autoformer[53]) and MLP-based model (DLinear [56]).
Table 5 summarizes the experimental results. Contformer achieves the best performance on ILI dataset. Specifically, it gives  $10\%$  MSE reduction  $(2.874 \rightarrow 2.632)$ . Also, Contformer outperforms Autoformer and FEDformer on Exchange, ETTm2, and Weather datasets. Therefore, it demonstrates that Contformer performs competitively with the state-of-the-art model on regular time series forecasting. For a more comprehensive exploration of the results, please refer to Appendix C.4.
5 Efficiency Analysis
Employing continuous-time modeling in ContiFormer often results in substantial time and GPU memory overhead. Hence, we conduct a comparative analysis of ContiFormer's actual time costs against those of Transformer-based models (TST), ODE-based models (ODE-RNN, Neural CDE), and SSM-based models (S5).
Table 6 shows that both TST and S5 are effective models. Additionally, ContiFormer leverages a parallel mechanism for faster inference, resulting in faster inference speed as compared to ODE-RNN and Neural CDE. Furthermore, ContiFormer demonstrates competitive performance even with a larger step size, as detailed in Appendix D.3. Therefore, ContiFormer's robustness to step size allows it to achieve a superior balance between time cost and performance. Please refer to Appendix E for more details.
Table 6: Actual time cost during training for different models on RacketSports dataset (0% dropped) from UEA dataset. We adopt RK4 solver for ODE-RNN, Neural CDE, and ContiFormer. Step size refers to the amount of time increment for the RK4 solver.  $\uparrow (\downarrow)$  indicates the higher (lower) the better.
Model	Step Size	Time Cost (↓)	Accuracy (↑)
TST [57]	-	0.14×	0.826
S5 [49]	-	1×	0.772
ODE-RNN [43]	0.1	2.47×	0.827
0.01	14.64×	0.796
Neural CDE [42]	0.1	3.48×	0.743
0.01	25.03×	0.748
ContiFormer	0.1	1.88×	0.836
0.01	5.87×	0.836
We propose ContiFormer, a novel continuous-time Transformer for modeling dynamic systems of irregular time series data. Compared with the existing methods, ContiFormer is more flexible since it abandons explicit closed-form assumptions of the underlying dynamics, leading to better generalization in various circumstances. Besides, similar to Neural ODE, ContiFormer can produce a continuous and smooth outcome. Taking advantage of the self-attention mechanism, ContiFormer can also capture long-term dependency in irregular observation sequences, achieving superior performance on a variety of irregular time series modeling tasks.
This research is supported in part by the National Natural Science Foundation of China under grant 62172107.
[57] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2114–2124, 2021.
[58] Jing Zhang, Peng Zhang, Baiwen Kong, Junqiu Wei, and Xin Jiang. Continuous self-attention models with neural ode networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14393–14401, 2021.
[59] Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes process. In International conference on machine learning, pages 11183–11193. PMLR, 2020.
[60] Yuan Zhang. Attain: Attention-based time-aware LSTM networks for disease progression modeling. In In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI-2019), pp. 4369-4375, Macao, China., 2019.
[61] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[62] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pages 27268-27286. PMLR, 2022.
[63] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[64] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In International conference on machine learning, pages 11692-11702. PMLR, 2020.
A Implement Details and Complexity Analysis
A.1 Background: Time Variable ODE
A numerical ODE solver integrates a single ODE system  $\frac{d\mathbf{x}}{dt} = f(t,\mathbf{x})$ , where  $\mathbf{x}\in \mathbb{R}^d$  and  $f:$ $\mathbb{R}^{d + 1}\to \mathbb{R}^d$ . Given an initial state  $\mathbf{x}_{t_0}$ , the ODE solver produces  $\mathbf{x}_{t_1},\mathbf{x}_{t_2},\dots,\mathbf{x}_{t_N}$ , which represents the latent state for each time point [9], i.e.,
$$
\mathbf {x} _ {t _ {1}}, \mathbf {x} _ {t _ {2}}, \dots , \mathbf {z} _ {t _ {N}} = \text {O D E S o l v e} \left(\mathbf {x} _ {t _ {0}}, f, t _ {0}, \dots , t _ {N}\right). \tag {12}
$$
To obtain the solution of Eq. (12), it requires the ODE solver to integrate over the interval  $[t_0, t_N]$ .
Varied time intervals Now, we suppose that we need to solve  $M$  dynamic system that has different time intervals. Let denote for the  $i$ -th system, the start point is  $t_0^{(i)}$  and the endpoint is  $t_1^{(i)}$  and the ODE system is defined by  $\frac{d\mathbf{x}}{dt} = f(t,\mathbf{x})$ . We can construct a dummy variable so that the integral interval always starts from  $-1$  and ends at 1 and perform a change of variables to transform every system to use this dummy variable.
For simplifying purposes, we assume that a single system that integrates from  $t_0$  to  $t_1$  with  $f(t,\mathbf{x})$  can control the differential equation and the solution of  $\mathbf{x}(t)$ . We introduce a dummy variable  $s = \frac{2t - t_0 - t_1}{t_1 - t_0}$  and a solution  $\tilde{\mathbf{x}}(s)$  in a fix interval  $[-1,1]$  satisfies
$$
\tilde {\mathbf {x}} (s) = \mathbf {x} \left(\frac {s \left(t _ {1} - t _ {0}\right) + t _ {0} + t _ {1}}{2}\right). \tag {13}
$$
The differential equation that solve  $\mathbf{x}$  follows that
$$
\begin{array}{l} \tilde {f} (s, \tilde {\mathbf {x}} (s)) = \frac {d \tilde {\mathbf {x}} (s)}{d s} = \left. \frac {d \mathbf {x} (t)}{d t} \right| _ {t = \frac {s \left(t _ {1} - t _ {0}\right) + t _ {0} + t _ {1}}{2}} \frac {d t}{d s} \\ = \left. f (t, \mathbf {x} (t)) \right| _ {t = \frac {s \left(t _ {1} - t _ {0}\right) + t _ {0} + t _ {1}}{2}} \left(\frac {t _ {1} - t _ {0}}{2}\right) \tag {14} \\ = f \left(\frac {s \left(t _ {1} - t _ {0}\right) + t _ {0} + t _ {1}}{2}, \tilde {\mathbf {x}} (s)\right) \left(\frac {t _ {1} - t _ {0}}{2}\right). \\ \end{array}
$$
Since  $\tilde{\mathbf{x}}(-1) = \mathbf{x}(t_0)$  and  $\tilde{\mathbf{x}}(1) = x_{t_1}$ , thus we have
$$
\mathbf {x} \left(t _ {1}\right) = \operatorname {O D E S o l v e} \left(\mathbf {x} \left(t _ {0}\right), f, t _ {0}, t _ {1}\right) = \operatorname {O D E S o l v e} \left(\tilde {\mathbf {x}} (- 1), \tilde {f}, - 1, 1\right). \tag {15}
$$
A.2 Background: Approximating Definite Integral of a Function
Given a function  $f$  that integrates over an interval  $[-1, 1]$ , a simple method to approximate it is to assume that the change is linear over time, i.e.,
$$
\int_ {- 1} ^ {1} f (x) d x \approx \frac {f (- 1) + f (1)}{2}. \tag {16}
$$
Besides, in numerical analysis, Gauss-Legendre Quadrature provides a more accurate way to approximate an integral over an interval  $[-1, 1]$ , i.e.,
$$
\int_ {- 1} ^ {1} f (x) \mathrm {d} x \approx \sum_ {j} \omega_ {j} f (\xi_ {j}), \tag {17}
$$