SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Forecasting results We list the results in Table 1-2 with the best in bold and the second underlined. Koopa shows competitive forecasting performance in both multivariate and univariate forecasting. Concretely, Koopa achieves state-of-the-art performance in more than  $70\%$  multivariate settings and consistently outperforms other deep models in the univariate settings.
Notably, Koopa surpasses the state-of-the-art Koopman-based forecaster KNF by a large margin in real-world time series, which can be attributed to our hierarchical dynamics learning and disentangling mechanism. Also, as the representative of efficient linear models, the performance of DLinear is still subpar in ILI, Traffic and Weather, indicating that nonlinear dynamics underlying the time series poses challenges for model capacity and point-wise weighting may not be appropriate to portray time-variant dynamics. Besides, compared with painstakingly trained PatchTST with channel-independence mechanism, our model can achieve a close and even better performance with naturally addressed non-stationary properties of real-world time series.
Table 1: Multivariate forecasting results with different forecast lengths  $H \in \{24,36,48,60\}$  for ILI and  $H \in \{48,96,144,192\}$  for others. We set the lookback length  $T = 2H$ . Additional results (ETTm1, ETTm2, ETTh1) are provided in Appendix D.1.
Models	Koopa	PatchTST	TimesNet	DLinear	MICN	KNF	FiLM	Autoformer
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ECL	48	0.130	0.234	0.147	0.246	0.149	0.254	0.158	0.241	0.156	0.271	0.175	0.265	0.197	0.270	0.164	0.272
96	0.136	0.236	0.143	0.241	0.170	0.275	0.153	0.245	0.165	0.277	0.198	0.284	0.238	0.341	0.182	0.289
144	0.149	0.247	0.145	0.241	0.183	0.287	0.152	0.245	0.163	0.274	0.204	0.297	0.234	0.338	0.210	0.315
192	0.156	0.254	0.147	0.240	0.189	0.291	0.153	0.246	0.171	0.284	0.245	0.321	0.240	0.339	0.221	0.324
ETTh2	48	0.226	0.300	0.223	0.297	0.241	0.319	0.226	0.305	0.260	0.336	0.385	0.376	0.261	0.324	0.355	0.380
96	0.297	0.349	0.300	0.353	0.325	0.376	0.294	0.351	0.343	0.393	0.433	0.446	0.322	0.372	0.427	0.432
144	0.333	0.381	0.346	0.390	0.374	0.408	0.354	0.397	0.374	0.411	0.441	0.456	0.352	0.397	0.457	0.461
192	0.356	0.393	0.383	0.406	0.394	0.434	0.385	0.418	0.455	0.464	0.528	0.503	0.361	0.410	0.503	0.491
Exchange	48	0.042	0.143	0.044	0.144	0.059	0.172	0.043	0.145	0.117	0.248	0.128	0.271	0.071	0.192	0.125	0.252
96	0.083	0.207	0.085	0.204	0.120	0.255	0.084	0.220	0.108	0.251	0.294	0.394	0.112	0.245	0.280	0.386
144	0.130	0.261	0.132	0.260	0.206	0.334	0.132	0.253	0.152	0.301	0.597	0.578	0.174	0.306	0.520	0.523
192	0.184	0.309	0.174	0.300	0.377	0.463	0.178	0.299	0.187	0.331	0.654	0.595	0.241	0.364	0.653	0.592
ILI	24	1.621	0.800	2.063	0.881	2.464	1.039	2.624	1.118	4.380	1.558	3.722	1.432	3.590	1.424	2.831	1.085
36	1.803	0.855	2.178	0.943	2.388	1.007	2.693	1.156	3.314	1.313	3.941	1.448	4.200	1.383	2.801	1.088
48	1.768	0.903	1.916	0.896	2.370	1.040	2.852	1.229	2.457	1.085	3.287	1.377	3.317	1.417	2.322	1.006
60	1.743	0.891	1.981	0.917	2.193	1.003	2.554	1.144	2.379	1.040	2.974	1.301	4.077	1.444	2.470	1.061
Traffic	48	0.415	0.274	0.426	0.286	0.567	0.306	0.488	0.352	0.496	0.301	0.621	0.382	0.498	0.312	0.640	0.361
96	0.401	0.275	0.413	0.283	0.611	0.337	0.485	0.336	0.511	0.312	0.645	0.376	0.451	0.297	0.668	0.367
144	0.397	0.276	0.405	0.278	0.603	0.322	0.452	0.317	0.498	0.309	0.683	0.402	0.430	0.288	0.681	0.379
192	0.403	0.284	0.404	0.277	0.604	0.321	0.438	0.309	0.494	0.312	0.699	0.405	0.425	0.288	0.692	0.385
Weather	48	0.126	0.168	0.140	0.179	0.138	0.191	0.156	0.198	0.157	0.217	0.201	0.288	0.160	0.206	0.185	0.240
96	0.154	0.205	0.160	0.206	0.180	0.231	0.186	0.229	0.187	0.250	0.295	0.308	0.189	0.233	0.230	0.279
144	0.172	0.225	0.174	0.221	0.190	0.244	0.199	0.244	0.197	0.257	0.394	0.401	0.200	0.245	0.268	0.308
192	0.193	0.241	0.195	0.243	0.212	0.265	0.217	0.261	0.214	0.270	0.462	0.437	0.219	0.263	0.325	0.347
\( 1^{\text{st}} \) Count	34	11	0	3	0	0	0	0
Table 2: Univariate forecasting results for the M4 dataset. We report the weighted average forecasting error from all four subsets and full results are provided in Appendix D.1.
Models	Koopa	N-HiTS	N-BEATS	PatchTST	TimesNet	DLinear	MICN	KNF	FiLM	Autoformer
Weighted Average	sMAPE	11.863	11.960	11.910	13.022	11.930	12.418	13.023	12.126	12.489	14.057
MASE	1.595	1.606	1.613	1.814	1.597	1.656	1.836	1.641	1.690	1.954
OWA	0.858	0.861	0.862	0.954	0.867	0.891	0.960	0.874	0.902	1.029
Figure 4: Model efficiency comparison. The performance comes from Table 1 with forecast window length  $H = 144$ . Training time and memory footprint are recorded with the same batch size and official code configuration. Full results of all six datasets are provided in Appendix D.3.
Model efficiency We comprehensively evaluate the model efficiency from three aspects: forecasting performance, training speed, and memory footprint. In Figure 4, we compare the efficiency under two representative datasets with different variate numbers (7 in ETTh2 and 862 in Traffic).
Compared with the state-of-the-art forecasting model PatchTST, Koopa saves  $62.3\%$  and  $96.5\%$  training time respectively in the ETTh2 and Traffic datasets with only  $26.8\%$  and  $2.9\%$  memory footprint. Concretely, the averaged training time and memory ratio of Koopa compared to PatchTST are  $22.7\%$  and  $24.0\%$  in all six datasets (see Appendix D.3 for the detail). Besides, as an efficient MLP-based forecaster, Koopa is also capable of learning nonlinear dynamics from time-variant and time-invariant components, and thus achieves a better performance.
Figure 5: Left: Comparison of Degree of Variation (the standard deviation of linear weighting fitted on different periods), we plot respective values of disengaged components on all six datasets. Right: A case of localized Koopman operators calculated on the Exchange dataset at the interval of one year.
5.2 Model Analysis
Dynamics disentanglement To validate the disentangling effect of our proposed Fourier Filter, we divide the whole time series into 20 subsets of different periods and conduct respective linear regression on the components disentangled by Fourier Filter. The standard deviation of the linear weighting reflects the variation of point-to-point temporal dependencies, which works as the manifestation of time-variant property. We plot the value as Degree of Variation (Figure 5 Left). It can be observed that larger deviations occur in the time-variant component, which indicates the proposed module successfully disentangles two types of dynamics from the perspective of frequency domain.
Case study We present a case study on real-world time series (exchange rate) on the right of Figure 5. We sample the lookback window at the interval of one year and visualize the Koopman operators calculated in Time-variant KP. It can be clearly observed that localized operators can exhibit changing temporal patterns in different periods, indicating the necessity of utilizing varying operators to describe time-variant dynamics. And interpretable insights are also presented as series uptrends correspond to heatmaps with large value and downtrends are reflected with small value.
Ablation study We conduct ablations on Koopa. As shown in Table 3, Time-variant and Time-invariant KPs perform as complementary modules to explore the dynamics underlying time series, and discarding any one of them will lead to the inferior performance. Besides, we evaluate alternative decomposition filters to disentangle time series dynamics. We find the proposed Fourier Filter conducts effective disentanglement, where the amplitude statistics of frequency spectrums from different periods are utilized to exhibit time-agnostic information. Therefore, Koopa tackling the right dynamics with complementary modules can achieve the best performance.
Table 3: Model ablation. Only  $K_{inv}$  uses one-block Time-invariant KP; Only  $K_{var}$  stacks Time-variant KPs only; Truncated Filter replaces Fourier Filter with High-Low Pass Filter; Branch Switch changes the order of KPs on disentangled components. The averaged results are listed here.
Dataset	ECL	ETTh2	Exchange	ILI	Traffic	Weather
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Only Kinv	0.148	0.250	0.312	0.358	0.120	0.241	2.146	0.963	0.740	0.446	0.170	0.213
Only Kvar	1.547	0.782	0.371	0.405	0.205	0.316	2.370	1.006	0.947	0.544	0.180	0.232
Truncated Filter	0.155	0.255	0.311	0.362	0.129	0.246	1.988	0.907	0.536	0.334	0.172	0.220
Branch Switch	0.696	0.393	0.344	0.385	0.231	0.325	2.130	0.964	0.451	0.304	0.173	0.221
Koopa	0.146	0.246	0.303	0.356	0.111	0.230	1.734	0.862	0.419	0.293	0.162	0.211
Avoiding rigorous reconstruction Unlike previous Koopman Autoencoders, the proposed Koopman Predictor does not reconstruct the whole dynamics at once, but aims to portray the partial dynamics evolution. Thus we remove the reconstruction branch, which is only utilized during training in previous KAEs. In our deep residual structure, the predictive objective function works as a good optimization indicator. We validate the design in Table 4, where the performance of sorely forecasting objective optimized model is better than with an additional reconstruction loss. Because the end-to-end forecasting objective helps to reduce the optimization gap between training and inference, making it a valuable contribution of applying Koopman operators on end-to-end time series forecasting.
Table 4: Performance comparison of the dynamics learning blocks implemented by our proposed Koopman Predictor (Koopa) and the canonical Koopman Autoencoder [29](KAE).
Dataset	ETTh2	Exchange	ECL	Traffic	Weather	ILI
Model	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Koopa	0.303	0.356	0.110	0.230	0.143	0.243	0.404	0.277	0.161	0.210	1.734	0.862
KAE	0.312	0.361	0.129	0.248	0.169	0.269	0.463	0.329	0.170	0.217	2.189	0.974
Promotion	2.88%	14.73%	15.38%	12.74%	5.29%	20.79%
Learning stable operators We turn to analyze our architectural design from the spectral perspective. The eigenvalues of the operator determine the amplitude of dynamics evolution. As most of nonstationary time series experience the distribution shift and can be regarded as an unstable evolution, the learned Koopman operator with the modulus far from the unit circle will cause non-divergent and even explosive trending in the long term, leading to training failures.
To tackle this problem generally faced by Koopman-based forecasters, we propose to utilize the disentanglement and deep residual structure. We measure the stability of the operator as the average distance of eigenvalues from the unit circle. As shown in Figure 6, the operator can become more stable by the above two techniques. The disentanglement helps to describe complex dynamics based on the decomposition and appropriate inductive bias can be applied. The architecture where each block is employed to fill the residual of the previously fitted dynamics reduces the difficulty of directly reconstructing complicated dynamics. Each block portrays the basic process driven by a stable operator within its power, which can be aggregated for a complicated non-stationary process.
5.3 Scaling Up Forecast Horizon
Most deep forecasting models work as a settled function once trained (e.g. input-  $T$ -output- $H$ ). For scenarios where the prediction horizon is mismatched or long-term, it poses two challenges for the trained model: (1) reuse parameters learned from observed series; (2) utilize incoming ground truth for model adaptation. The practical scenarios, which we name as scaling up forecast horizon, may
Figure 6: Visualization of the operator stability on the highly non-stationary Exchange dataset. We plot the first block time-invariant operator eigenvalues of the following design: (a) Single-block model with only time-invariant operator. (b) Single-block model with time-invariant and time-variant operators. (c) Two-block model with time-invariant and time-variant operators.
lead to failure on most deep models but can be naturally tackled by Koopa. In detail, we first train Koopa with forecast length  $H_{\mathrm{tr}}$  and attempt to apply it on a larger forecast length  $H_{\mathrm{te}}$ .
Method Koopa scales up forecast horizon as follows: Since Time-invariant KP has learned the globally shared dynamics and Time-variant KP can calculate localized operator  $K_{\mathrm{var}}$  within the lookback window, we freeze the parameters of trained Koopa but only use the incoming ground truth to adapt  $K_{\mathrm{var}}$ . The naive implementation uses incremental Koopman embedding with dimension  $D$  and conducts Equation 10 to obtain an updated operator, which has a complexity of  $\mathcal{O}(H_{\mathrm{te}}D^3)$ . We further propose an iterative algorithm with improved  $\mathcal{O}((H_{\mathrm{te}} + D)D^2)$  complexity. The detailed method implementations and complexity analysis can be found in Appendix A.
Results As shown in Table 5, the proposed operator adaption mechanism further boosts the performance on the scaling up scenario, which can be attributed to more accurately fitted time-variant dynamics with incoming ground truth snapshots. Besides, the promotion becomes more significant when applied to non-stationary datasets (manifested as large ADF Test Statistic [10]).
Table 5: Scaling up forecast horizon:  $(H_{\mathrm{tr}}, H_{\mathrm{te}}) = (24, 48)$  for ILI and  $(H_{\mathrm{tr}}, H_{\mathrm{te}}) = (48, 144)$  for others. Koopa conducts vanilla rolling forecast and Koopa OA further introduces operator adaptation.
Dataset ADF Test Statistic	Exchange (-1.889)	ETTh2 (-4.135)	ILI (-5.406)	ECL (-8.483)	Traffic (-15.046)	Weather (-26.661)
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Koopa	0.214	0.348	0.437	0.429	2.836	1.065	0.199	0.298	0.709	0.437	0.237	0.276
Koopa OA	0.172	0.319	0.372	0.404	2.427	0.907	0.182	0.271	0.699	0.426	0.225	0.264
Promotion (MSE)	19.6%	14.9%	14.1%	8.5%	1.4%	5.1%
This paper tackles time series as dynamical systems. With disentangled time-variant and time-invariant components from non-stationary series, the Koopa model reveals the complicated dynamics hierarchically and leverages MLP modules to learn Koopman embedding and operator. Experimentally, our model shows competitive performance with remarkable efficiency and the potential to scale up the forecast length by operator adaptation. In the future, we will explore Koopa with the dynamic modes underlying non-stationary data using the toolbox of Koopman spectral analysis.
This work was supported by the National Key Research and Development Plan (2021YFC3000905), National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (Z201100006820041), and BNRist Innovation Fund (BNR2021RC01002).
[44] Rui Wang, Yihe Dong, Sercan O Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675, 2022.
[45] Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data-driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25:1307-1346, 2015.
[46] Herman Wold. A study in the analysis of stationary time series. PhD thesis, Almqvist & Wiksell, 1938.
[47] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186, 2022.
[48] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In NeurIPS, 2021.
[49] Wei Xiong, Xiaomeng Huang, Ziyang Zhang, Ruixuan Deng, Pei Sun, and Yang Tian. Koopman neural operator as a mesh-free solver of non-linear partial differential equations. arXiv preprint arXiv:2301.10022, 2023.
[50] Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC), pages 4832-4839. IEEE, 2019.
[51] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504, 2022.
[52] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186, 2022.
[53] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
[54] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 35:12677-12690, 2022.
A Scaling Up Forecast Horizon
In this section, we introduce the capability of Koopa to scale up forecast horizon. In detail, we train a Koopa model with forecast length  $H_{\mathrm{tr}}$  and attempt to apply it on a larger length  $H_{\mathrm{te}}$ . The basic approach conducts rolling forecast by taking the model prediction as the input of the next iteration until the desired forecast horizon is all filled. Instead, we further assume that after the model gives a prediction, the model can utilize the incoming ground truth for model adaptation and continue rolling forecast for the next iteration. It is notable that we do not retrain parameters during model adaptation, since it will lead to overfitting on the incoming ground truth and Catastrophic Forgetting [11, 18, 35].
Koopa can naturally cope with the scenario by learning Koopman embedding and operator  $K_{\mathrm{inv}}$  in Time-invariant KPs while calculating localized operator  $K_{\mathrm{var}}$  to describe the dynamics in the temporal neighborhood. Therefore, we freeze the parameters of Koopa but only use the incoming ground truth for operator adaptation of  $K_{\mathrm{var}}$  in Time-variant KPs.
A.1 Implementation of Operator Adaptation
At the beginning of rolling forecast, the Encoder in Time-variant KP outputs  $D$ -dimensional Koopman embedding for each observed series segment as  $[z_1, z_2, \ldots, z_F]$ , where  $F = \frac{H_{\mathrm{fr}}}{S}$  is the segment number with  $S$  as the segment length. The operator  $K_{\mathrm{var}}$  in Time-variant KP is calculated as follows:
$$
Z _ {\text {b a c k}} = \left[ z _ {1}, z _ {2}, \dots , z _ {F - 1} \right], Z _ {\text {f o r e}} = \left[ z _ {2}, z _ {3}, \dots , z _ {F} \right], K _ {\text {v a r}} = Z _ {\text {f o r e}} Z _ {\text {b a c k}} ^ {\dagger}, \tag {15}
$$
where  $Z_{\mathrm{back}}, Z_{\mathrm{fore}} \in \mathbb{R}^{D \times (F - 1)}, K_{\mathrm{var}} \in \mathbb{R}^{D \times D}$ . With the calculated operator, we obtain the next predicted Koopman embedding by one-step forwarding:
$$
\hat {z} _ {F + 1} = K _ {\text {v a r}} z _ {F}. \tag {16}
$$
After decoding the embedding  $\hat{z}_{F + 1}$  to the series prediction, we can utilize the true value of incoming Koopman embedding  $z_{F + 1}$  obtained by Koopa with frozen parameters. Instead of using  $K_{\mathrm{var}}$  to obtain the next embedding
$\hat{z}_{F + 2}$ , we use incremental embedding collections  $Z_{\mathrm{back + }}$ ,  $Z_{\mathrm{fore + }} \in \mathbb{R}^{D \times F}$  to obtain a more accurate operator  $K_{\mathrm{var + }} \in \mathbb{R}^{D \times D}$  to describe the local dynamics:
$$
Z _ {\text {b a c k} +} = \left[ Z _ {\text {b a c k}}, z _ {F} \right], Z _ {\text {f o r e} +} = \left[ Z _ {\text {b a c k}}, z _ {F + 1} \right], K _ {\text {v a r} +} = Z _ {\text {f o r e} +} Z _ {\text {b a c k} +} ^ {\dagger}. \tag {17}
$$
The procedure repeats for  $L$  times ( $L \propto H_{\mathrm{te}}$ ) until the forecast horizon is all filled, we formulate it as Algorithm 1. And experimental results (Koopa OA) in Section 5.3 have demonstrated the promotion of forecasting performance due to more precisely fitted dynamics.
A.2 Computational Acceleration
The naive implementation shown in Algorithm 1 repeatedly conducts Equation 17 on the incremental embedding collection to obtain new operators, which has a complexity of  $\mathcal{O}(LD^3)$ . We propose an equivalent algorithm with improved complexity of  $\mathcal{O}((L + D)D^2)$  as shown in Algorithm 2.
Theorem. Algorithm 2 gives the same  $K_{\text{var}}$  as Algorithm 1 in each iteration with  $\mathcal{O}(D^2)$  complexity.
Proof. We start with the first iteration analysis. By the definition of Moore-Penrose inverse, we have  $Z_{\mathrm{back}}^{\dagger}Z_{\mathrm{back}} = I_{F - 1}$ , where  $I_{F - 1}$  is an identity matrix with the dimension of  $F - 1$ . When the model receives the incoming embedding  $z_{F + 1}$ , incremental embedding  $m = z_F$ ,  $n = z_{F + 1}$  will be appended to  $Z_{\mathrm{back}}$  and  $Z_{\mathrm{fore}}$  respectively. Instead of calculating new  $K_{\mathrm{var + }}$  from incremental collections, we utilize calculated  $K_{\mathrm{var}}$  to find the iteration rule on  $K_{\mathrm{var + }}$ . Concretely, we suppose
$$
Z _ {\text {b a c k} +} ^ {\dagger} = \left[ \begin{array}{c} Z _ {\text {b a c k}} ^ {\dagger} - \Delta \\ b ^ {\top} \end{array} \right] \in \mathbb {R} ^ {F \times D}, \tag {18}
$$
where  $\Delta \in \mathbb{R}^{(F - 1)\times D}$ ,  $b\in \mathbb{R}^D$  are variables to be identified. By the definition of Moore-Penrose inverse, we have  $Z_{\mathrm{back + }}^{\dagger}Z_{\mathrm{back + }} = I_{F}$ . By unfolding it, we have the following equations:
$$
\Delta Z _ {\text {b a c k}} = \mathbf {0}, b ^ {\top} Z _ {\text {b a c k}} = \vec {0} \cdot b ^ {\top} m = 1, Z _ {\text {b a c k}} ^ {\dagger} m - \Delta m = \vec {0}. \tag {19}
$$
We suppose  $\Delta = \delta b^{\top}$ , where  $\delta \in \mathbb{R}^{F - 1}$ , such that when  $b^{\top}Z_{\mathrm{back}} = \vec{0}$ , then  $\Delta Z_{\mathrm{back}} = 0$ . Then we have  $Z_{\mathrm{back}}^{\dagger}m - \delta b^{\top}m = Z_{\mathrm{back}}^{\dagger}m - \delta = \vec{0}$ , thus  $\Delta = Z_{\mathrm{back}}^{\dagger}mb^{\top}$ . Given equations that  $b^{\top}Z_{\mathrm{back}} = \vec{0}$  and  $b^{\top}m = 1$ , we have the analytical solution of  $b$ :
$$
b = r / \left\| r \right\| ^ {2}, \text {w h e r e} r = m - Z _ {\text {b a c k}} Z _ {\text {b a c k}} ^ {\dagger} m. \tag {20}
$$
Therefore, we find the equation between the incremental version  $K_{\mathrm{var + }}$  and calculated  $K_{\mathrm{var}}$ :
$$
Z _ {\text {b a c k} +} ^ {\dagger} = \left[ \begin{array}{c} Z _ {\text {b a c k}} ^ {\dagger} \left(I _ {D} - m b ^ {\top}\right) \\ b ^ {\top} \end{array} \right], K _ {\text {v a r} +} = Z _ {\text {f o r e} +} Z _ {\text {b a c k} +} ^ {\dagger} = K _ {\text {v a r}} + (n - K _ {\text {v a r}} m) b ^ {\top}, \tag {21}
$$
where  $m, n$  are the incremental embedding of  $Z_{\mathrm{back}}$ ,  $Z_{\mathrm{fore}}$  and  $b$  can be calculated by Equation 20. We also derive the iteration rule on  $X = Z_{\mathrm{back}}Z_{\mathrm{back}}^{\dagger}$  to obtain  $b$ , which is formulated as follows:
$$
X _ {+} = Z _ {\text {b a c k} +} Z _ {\text {b a c k} +} ^ {\dagger} = X + (m - X m) b ^ {\top} = X + r b ^ {\top}. \tag {22}
$$
By adopting Equation 21-22 and permuting the matrix multiplication order, we reduce the complexity of each iteration to  $\mathcal{O}(D^2)$ . Therefore, Algorithm 2 has a overall complexity of  $\mathcal{O}((L + D)D^2)$ . Since  $L \propto H_{\mathrm{te}}$ , Algorithm 1-2 have  $\mathcal{O}(H_{\mathrm{te}}D^3)$  and  $\mathcal{O}((H_{\mathrm{te}} + D)D^2)$  complexity respectively.
Koopa is trained with L2 loss and optimized by ADAM [17] with an initial learning rate of 0.001 and batch size set to 32. The training process is early stopped within 10 epochs. We repeat each experiment three times with different random seeds to obtain average test MSE/MAE and detailed results with standard deviations are listed in Table 6. Experiments are implemented in PyTorch [34] and conducted on NVIDIA TITAN RTX 24GB GPUs.