SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All the baselines that we reproduced are implemented based on the benchmark of TimesNet [47] Repository, which is fairly built on the configurations provided by each model's original paper or official code. Since several baselines adopt Series Stationarization from Non-stationary Transformers [28] while others do not, we equip all models with the method for a fair comparison.
C Hyperparameter Sensitivity
Considering the efficiency of hyperparameters search, we fix the segment length  $S = T / 2$  and the number of Koopa blocks  $B = 3$  in all our experiments. We verify the robustness of Koopa of other hyperparameters as
Algorithm 1 Koopa Operator Adaptation.
Require: Observed embedding  $Z = [z_{1}, \ldots, z_{F}]$  and successively incoming ground truth embedding  $[z_{F+1}, \ldots, z_{F+L}]$  with each embedding  $z_{i} \in \mathbb{R}^{D}$ .
1:  $Z_{\mathrm{back}} = [z_1,\ldots ,z_{F - 1}],Z_{\mathrm{fore}} = [z_2,\ldots ,z_F]$ $\triangleright Z_{\mathrm{back}},Z_{\mathrm{fore}}\in \mathbb{R}^{D\times (F - 1)}$
2:  $K_{\mathrm{var}} = Z_{\mathrm{fore}}Z_{\mathrm{back}}^{\dagger}$ $\triangleright K_{\mathrm{var}}\in \mathbb{R}^{D\times D}$
3:  $\hat{z}_{F + 1} = K_{\mathrm{var}}n$ $\triangleright \hat{z}_{F + 1}\in \mathbb{R}^{D}$
4: for  $l$  in  $\{1,\ldots ,L\}$  ..  $\triangleright z_{F + l}$  comes successively
5:  $m = z_{F + l - 1},n = z_{F + l}$ $\triangleright m,n\in \mathbb{R}^{D}$
6:  $Z_{\mathrm{back}}\gets [Z_{\mathrm{back}},m],Z_{\mathrm{fore}}\gets [Z_{\mathrm{fore}},n]$ $\triangleright Z_{\mathrm{back}},Z_{\mathrm{fore}}\in \mathbb{R}^{D\times (F + l - 1)}$
7:  $K_{\mathrm{var}} = Z_{\mathrm{fore}}Z_{\mathrm{back}}^{\dagger}$ $\triangleright K_{\mathrm{var}}\in \mathbb{R}^{D\times D}$
8:  $\hat{z}_{F + l + 1} = K_{\mathrm{var}}n$ $\triangleright \hat{z}_{F + l + 1}\in \mathbb{R}^{D}$
9: End for
10: Return  $[\hat{z}_{F + 1},\dots ,\hat{z}_{F + L + 1}]$
Algorithm 2 Accelerated Koopa Operator Adaptation.
1:  $Z_{\mathrm{back}} = [z_1,\dots ,z_{F - 1}],Z_{\mathrm{fore}} = [z_2,\dots ,z_F]$ $\triangleright Z_{\mathrm{back}},Z_{\mathrm{fore}}\in \mathbb{R}^{D\times (F - 1)}$
2:  $K_{\mathrm{var}} = Z_{\mathrm{fore}}Z_{\mathrm{back}}^{\dagger}$ $X = Z_{\mathrm{back}}Z_{\mathrm{back}}^{\dagger}$ $\triangleright K_{\mathrm{var}},X\in \mathbb{R}^{D\times D}$
3:  $\hat{z}_{F + 1} = K_{\mathrm{var}}n$ $\triangleright \hat{z}_{F + 1}\in \mathbb{R}^{D}$
4: for  $l$  in  $\{1,\ldots ,L\}$  ..  $\triangleright z_{F + l}$  comes successively
5:  $m = z_{F + l - 1},n = z_{F + l}$ $\triangleright m,n\in \mathbb{R}^{D}$
6:  $r = m - Xm$ $\triangleright r\in \mathbb{R}^{D}$
7:  $b = r / ||r||^2$ $\triangleright b\in \mathbb{R}^{D}$
8:  $K_{\mathrm{var}} \gets K_{\mathrm{var}} + (n - K_{\mathrm{var}}m)b^{\top} \quad \triangleright K_{\mathrm{var}} \in \mathbb{R}^{D \times D}$
9:  $X\gets X + rb^{\top}$ $\triangleright X\in \mathbb{R}^{D\times D}$
0:  $\hat{z}_{F + l + 1} = K_{\mathrm{var}}n$ $\triangleright \hat{z}_{F + l + 1}\in \mathbb{R}^{D}$
11: End for
12: Return  $\left[\hat{z}_{F+1}, \dots, \hat{z}_{F+L+1}\right]^{\top}$
Table 6: Detailed performance of Koopa. We report the MSE/MAE and standard deviation of different forecast horizons  $\{H_1, H_2, H_3, H_4\} = \{24, 36, 48, 60\}$  for ILI and  $\{48, 96, 144, 192\}$  for others.
Dataset	ECL	ETTh2	Exchange
Horizon	MSE	MAE	MSE	MAE	MSE	MAE
H1	0.130±0.003	0.234±0.003	0.226±0.003	0.300±0.003	0.042±0.002	0.143±0.003
H2	0.136±0.004	0.236±0.005	0.297±0.004	0.349±0.004	0.083±0.004	0.207±0.004
H3	0.149±0.003	0.247±0.003	0.333±0.004	0.381±0.003	0.130±0.005	0.261±0.003
H4	0.156±0.004	0.254±0.003	0.356±0.005	0.393±0.004	0.184±0.009	0.309±0.005
Dataset	ILI	Traffic	Weather
Horizon	MSE	MAE	MSE	MAE	MSE	MAE
H1	1.621±0.008	0.800±0.006	0.415±0.003	0.274±0.005	0.126±0.005	0.168±0.004
H2	1.803±0.040	0.855±0.020	0.401±0.005	0.275±0.004	0.154±0.006	0.205±0.003
H3	1.768±0.015	0.903±0.008	0.397±0.004	0.276±0.003	0.172±0.005	0.225±0.005
H4	1.743±0.040	0.891±0.009	0.403±0.007	0.284±0.009	0.193±0.003	0.241±0.004
Figure 7: Left: Hyperparameter sensitivity with respect to the dimension of Koopman embedding, hidden layer number, and hidden dimension of Encoder and Decoder in Koopa.
follows: the dimension of Koopman embedding  $D$ , the hidden layer number  $l$  and the hidden dimension  $d$  used in Encoder and Decoder. As is shown in Figure 7, we find the proposed model is insensitive to the choices of above hyperparameters, which can be beneficial for practitioners to reduce tuning burden.
Intuitively, a larger dimension of Koopman embedding  $D$  can bring about a lower approximation error. We further dive into it and find that stacking blocks can enhance the model capacity, and thus the performance is insensitive to  $D$  when the model is deep enough. To address the concern, we further check the sensitivity of  $D$  under varying block number  $B$  in Figure 8. It can be seen that a larger  $B$  generally leads to lower error even if  $D$  is small. And the performance can be sensitive to  $D$  when the model is not deep enough ( $B = 1, 2$ ).
Besides, we find the proposed model can be insensitive to  $S$  on several datasets while sensitive on ECL and Traffic datasets (The difference is about  $10\%$ ). There are many variables in these two datasets, but our current design shares the  $S$  for all variables. Since different variables with distinct evolution periods implicitly require different optimal  $S$ , the performance of the dataset with more variables is more likely to be influenced by  $S$ . Therefore, we set  $S = T / 2$  with relatively small performance fluctuation to deal with most situations.
Figure 8: Left: Hyperparameter sensitivity of the dimension of Koopman embedding under different settings of the block number. Right: Hyperparameter sensitivity of the segment length.
D Supplementary Experimental Results
D.1 Full Forecasting Results
Due to the limited pages, we list additional multivariate benchmarks on ETT datasets [53] in Table 7, which includes the hourly recorded ETTh2 and 15-minutely recorded ETTm1/ETTm2, and the full univariate results of M4 [39] in Table 8, which contains the yearly, quarterly and monthly collected univariate marketing data. Notably, Koopa still achieves competitive performance compared with state-of-the-art deep forecasting models and specialized univariate models.
D.2 Full Ablation Results
We elaborately conduct model ablations to verify the necessity of each proposed module: Time-invariant KP, Time-variant KP, Fourier Filter and evaluate alternative choices to disentangle dynamics. As shown in Table 9, Koopa conducts effective disentanglement and tackles the right dynamics with complementary KPs, and thus achieves the best forecasting performance.
Table 7: Forecasting results with different forecast window lengths  $H \in \{48,96,144,192\}$  on ETT dataset. We set the lookback window length  $T = 2H$ .
ModelsMetric	KooPA	PatchTST [31]	TimesNet [47]	DLinear [51]	MICN [43]	KNF [44]	FiLM [54]	Autoformer [48]
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETm1	48	0.283	0.333	0.286	0.336	0.308	0.354	0.322	0.355	0.294	0.353	1.026	0.792	0.324	0.353	0.592	0.419
96	0.294	0.345	0.299	0.346	0.329	0.370	0.309	0.346	0.306	0.364	0.957	0.782	0.311	0.346	0.493	0.469
144	0.322	0.366	0.325	0.363	0.358	0.387	0.327	0.359	0.342	0.390	0.921	0.760	0.328	0.358	0.735	0.569
192	0.337	0.378	0.343	0.375	0.462	0.441	0.337	0.365	0.386	0.415	0.896	0.731	0.339	0.366	0.592	0.506
ETm2	48	0.134	0.226	0.135	0.231	0.142	0.234	0.144	0.240	0.131	0.238	0.621	0.623	0.146	0.243	0.191	0.280
96	0.171	0.254	0.171	0.255	0.187	0.269	0.172	0.256	0.197	0.295	1.535	1.012	0.174	0.257	0.241	0.311
144	0.206	0.280	0.205	0.282	0.216	0.291	0.200	0.276	0.210	0.297	1.337	0.876	0.204	0.279	0.300	0.352
192	0.226	0.298	0.221	0.294	0.243	0.313	0.219	0.290	0.248	0.328	1.355	0.908	0.224	0.293	0.324	0.370
ETh1	48	0.336	0.377	0.337	0.375	0.365	0.399	0.343	0.371	0.375	0.406	0.876	0.709	0.407	0.427	0.442	0.438
96	0.371	0.405	0.372	0.393	0.411	0.430	0.379	0.393	0.406	0.429	0.975	0.744	0.429	0.431	0.634	0.523
144	0.405	0.418	0.394	0.412	0.442	0.447	0.393	0.403	0.437	0.448	0.801	0.662	0.451	0.448	0.522	0.491
192	0.416	0.429	0.416	0.439	0.469	0.470	0.407	0.416	0.518	0.496	0.941	0.744	0.460	0.459	0.525	0.501
Table 8: Full univariate forecasting results for M4 dataset. We follow the same data processing and forecasting length settings used in TimesNet [47] benchmark. Additional forecasting models N-HiTS [7] and N-BEATS [32] are also included.
Models	KooPA	N-HiTS	N-BEATS	PatchTST	TimesNet	DLinear	MICN	KNF	FiLM	Autoformer
Year	sMAPE	13.352	13.371	13.466	13.517	13.394	13.866	14.532	13.986	14.012	14.786
MASE	2.997	3.025	3.059	3.031	3.004	3.006	3.359	3.029	3.071	3.349
OWA	0.786	0.790	0.797	0.795	0.787	0.802	0.867	0.804	0.815	0.874
Quarter	sMAPE	10.159	10.454	10.074	10.847	10.101	10.689	11.395	10.343	10.758	12.125
MASE	1.189	1.219	1.163	1.315	1.183	1.294	1.379	1.202	1.306	1.483
OWA	0.895	0.919	0.881	0.972	0.890	0.957	1.020	0.965	0.905	1.091
Month	sMAPE	12.730	12.794	12.801	14.584	12.866	13.372	13.829	12.894	13.377	15.530
MASE	0.953	0.960	0.955	1.169	0.964	1.014	1.082	1.023	1.021	1.277
OWA	0.901	0.895	0.893	1.055	0.894	0.940	0.988	0.985	0.944	1.139
Others	sMAPE	4.861	4.696	5.008	6.184	4.982	4.894	6.151	4.753	5.259	5.841
MASE	3.124	3.130	3.443	4.818	3.323	3.358	4.263	3.138	3.608	4.308
OWA	1.004	0.988	1.070	1.140	1.048	1.044	1.319	1.019	1.122	1.294
Weighted Average	sMAPE	11.863	11.960	11.910	13.022	11.930	12.418	13.023	12.126	12.489	14.057
MASE	1.595	1.606	1.613	1.814	1.597	1.656	1.836	1.641	1.690	1.954
OWA	0.858	0.861	0.862	0.954	0.867	0.891	0.960	0.874	0.902	1.029
D.3 Model Efficiency
We comprehensively compare the forecasting performance, training speed, and memory footprint of our model with well-acknowledged deep forecasting models. The results are recorded with the official model configuration and the same batch size. We visualize the model efficiency under all six multivariate datasets in Figure 9- 11. In detail, compared with the previous state-of-the-art model PatchTST [8], Koopa consumes only  $15.2\%$  training time and  $3.6\%$  memory footprint respectively in ECL,  $37.8\%$  training time and  $26.8\%$  memory in ETTh2,  $23.5\%$  training time and  $37.3\%$  memory in Exchange,  $50.9\%$  training time and  $47.8\%$  memory in ILI,  $3.5\%$  training time and  $2.9\%$  memory in Traffic, and  $5.4\%$  training time and  $25.4\%$  memory in Weather, leading to the averaged  $77.3\%$  and  $76.0\%$  saving of training time and memory footprint in all six datasets. The remarkable efficiency can be attributed to Koopa with MLPs as the building blocks, and we find the budget saving becoming more significant on datasets with more series variables (ECL, Traffic).
Besides, as an efficient linear model, the performance of Koopa still surpasses other MLP-based models. Especially, Compared with DLinear [51], our model reduces  $38.0\%$  MSE  $(2.852\rightarrow 1.768)$  in ILI and  $13.6\%$  MSE  $(0.452\rightarrow 0.397)$  in Weather. And the average MSE reduction of Koopa compared with the previous state-of-the-art MLP-based model reaches  $12.2\%$ . Therefore, our proposed Koopa is efficiently built with MLP networks and shows great model capacity to exploit nonlinear dynamics and complicated temporal dependencies in real-world time series.
Table 9: Model ablation with detailed forecasting performance. We report forecasting results with different prediction lengths  $\{24,36,48,60\}$  for ILI and  $H\in \{48,96,144,192\}$  for others. For columns: Only  $K_{inv}$  uses one-block Time-invariant KP; Only  $K_{var}$  stacks Time-variant KPs only; Truncated Filter replaces Fourier Filter with High-Low Frequency Pass Filter; Branch Switch changes the order of KPs to deal with disentangled components.
Models	KooPA	Only \( K_{inv} \)	Only \( K_{var} \)	Truncated Filter	Branch Switch
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ECL	48	0.130	0.234	0.150	0.243	1.041	0.777	0.149	0.245	0.137	0.234
96	0.136	0.236	0.137	0.242	4.643	1.669	0.172	0.280	2.240	0.724
144	0.149	0.247	0.150	0.252	0.238	0.327	0.149	0.246	0.226	0.331
192	0.156	0.254	0.158	0.260	0.267	0.355	0.152	0.248	0.181	0.284
ETTh2	48	0.226	0.300	0.235	0.304	0.271	0.334	0.340	0.310	0.245	0.317
96	0.297	0.349	0.311	0.353	0.382	0.405	0.301	0.352	0.343	0.384
144	0.333	0.381	0.337	0.379	0.427	0.444	0.338	0.386	0.403	0.418
192	0.356	0.393	0.363	0.397	0.402	0.437	0.363	0.400	0.384	0.420
Exchange	48	0.042	0.143	0.046	0.150	0.065	0.184	0.048	0.150	0.055	0.165
96	0.083	0.207	0.083	0.210	0.147	0.274	0.087	0.210	0.151	0.277
144	0.130	0.261	0.149	0.281	0.222	0.351	0.150	0.278	0.254	0.369
192	0.184	0.309	0.200	0.322	0.385	0.456	0.229	0.345	0.463	0.490
ILI	24	1.621	0.800	2.165	0.882	1.972	0.919	2.140	0.874	2.092	0.894
36	1.803	0.855	1.815	0.882	2.675	1.091	1.692	0.844	2.116	0.950
48	1.768	0.903	2.107	0.981	2.446	1.045	1.762	0.895	2.394	1.084
60	1.743	0.891	2.496	1.108	2.387	0.970	2.357	1.018	1.917	0.926
Traffic	48	0.415	0.274	0.445	0.295	0.915	0.536	0.668	0.363	0.468	0.300
96	0.401	0.275	0.403	0.277	0.833	0.465	0.441	0.323	0.429	0.298
144	0.397	0.276	0.400	0.278	0.816	0.452	0.436	0.321	0.438	0.307
192	0.403	0.284	1.371	0.788	1.224	0.723	0.597	0.331	0.469	0.312
Weather	48	0.126	0.168	0.142	0.181	0.140	0.190	0.125	0.166	0.130	0.173
96	0.154	0.205	0.164	0.209	0.169	0.224	0.154	0.202	0.163	0.210
144	0.172	0.225	0.178	0.226	0.194	0.247	0.176	0.226	0.187	0.238
192	0.193	0.241	0.195	0.245	0.217	0.268	0.195	0.244	0.212	0.261
Figure 9: Model efficiency comparison with forecast length  $H = 144$  for ETTh2 and Traffic.
Figure 10: Model efficiency comparison with forecast length  $H = 144$  for Exchange and ECL.
Figure 11: Model efficiency comparison with forecast length  $H = 144$  for Weather and 48 for ILI.
D.4 Training Stability
Since the eigenvalues of the operator determine the amplitude of dynamics evolution, where modulus not close to one causes non-divergent and explosive evolution in the long-term, we highlight that a stable Koopman operator only describes weakly stationary series well. While most of the Koopman-based forecasters can suffer from the operator convergence problem induced by complicated non-stationary series variations, we employ several techniques to stabilize the training process.
Operator initialization We adopt the operator initialization strategy, where the operator starts from the multiplication of eigenfunctions with standard Gaussian distribution and all-one eigenvalues.
Hierarchical disentanglement In our model, each block learns weak stationary process hierarchically and feeds the residual of fitted dynamics for the next block to correct. Thus Koopman Predictor aims not to fully reconstruct the whole dynamics at once, but to partially describe dynamics, so rigorous reconstruction is not forced in each block, reducing the difficulty of portraying the non-stationary series as dynamics.
Explosion checking We introduce an explosion checking mechanism that replaces the operator encountering nan number with the identity matrix when the exponential multiplication of multiple time steps is detected.
Based on the proposed strategies, we provide the model training curves in Figure 12 to check the convergence of our proposed model and other forecasters. The training curves of the proposed model in blue demonstrate a consistent and smooth convergence, indicating its effectiveness in converging toward an optimal solution.
E.1 Impact on Real-world Applications
Our work copes with real-world time series forecasting, which is faced with intrinsic non-stationarity that poses fundamental challenges for deep forecasting models. Since previous studies hardly research the theoretical basis that can naturally address the time-variant property in non-stationary data, we propose a novel Koopman
Figure 12: Training curves of different models on the ETT and Weather datasets.
forecaster that fundamentally considers the implicit time-variant and time-invariant dynamics based on Koopman theory. Our model achieves the state-of-the-art performance on six real-world forecasting tasks, covering energy, economics, disease, traffic, and weather, and demonstrates remarkable model efficiency in training time and memory footprint. Therefore, the proposed model makes it promising to tackle real-world forecasting applications, which helps our society prevent risks in advance and make better decisions with limited computational budgets. Our paper mainly focuses on scientific research and has no obvious negative social impact.
E.2 Impact on Future Research
In this paper, we find modern Koopman theory natural to learn the dynamics underlying non-stationary time series. The proposed model explores complex non-stationary patterns with temporal localization inspired by Koopman approaches and implements respective deep network modules to disentangle and portray time-variant and time-invariant dynamics with the enlightenment of Wold's Theorem. The remarkable efficiency and insights from the theory can be instructive for future research.
F Limitation
Our proposed model does not respectively consider dynamics in different variates, which leaves improvement for better multivariate forecasting with the consideration of various evolution patterns and series relationships. And Koopman spectral theory is still under leveraging in our work, which can discover Koopman modes to interpret the linear behavior underlying non-stationary data in a high-dimensional representation. Besides, Koopman theory for control considering factors outside the system can be promising for series forecasting with covariates, which leaves our future work.