SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In traditional recurrent structure, for two adjacent time steps in series, the latter one always waits for the former one until the information computation of the former one is completed. When the sequence becomes longer, this becomes slower. Fortunately, in the WIT framework we designed, some of the time step information can be computed in parallel. As shown in Figure 2(b), after a point is calculated, the right point in its horizontal direction and the point below it in its vertical direction can start calculation without waiting for each other. Therefore, we propose the Recurrent Acceleration Network (RAN) as our accelerated framework, which enables parallel computation of data points without waiting for each other, greatly improving the efficiency of information transmission in HVGSU. We place parallelizable points in a slice, and the updated information transfer process is shown in Figure 4(c). Each green box in Figure 4(c) represents a slice, and the number of green boxes is the number of times we need to recurrently compute. The meanings of the remaining markers are the same as those in Figure 2. Under the RAN framework, the recurrent length has changed from the sequence length  $L = R \times C$  to  $R + C - 1$ , while the complexity of  $R$  and  $C$  is  $\mathcal{O}(\sqrt{L})$ . Thus, we have reduced the time complexity to  $\mathcal{O}(\sqrt{L})$  via the RAN framework. It should be noted that although we parallelly compute some data points, which may increase some memory, the complexity of parallel computation,  $\mathcal{O}(\sqrt{L})$ , is far less than the complexity of saving the output variables,  $\mathcal{O}(L)$ , because we need to save the output information of each point in the sequence. Implementation source code for RAN is given in Appendix D.
3.4 Forecasting Module
In the forecasting module, we address the issue of error accumulation in the auto-regressive structure by drawing inspiration from Informer [Zhou et al., 2021] and Pyraformer [Liu et al., 2021], combining both horizontal and vertical hidden states, and then making predictions through a fully connected layer, as illustrated in Figure 3.
We utilize the last row of the horizontal hidden states as it contains sufficient global and latest short-term semantic information from the historical sequence. In contrast, all columns of the vertical hidden states, which capture different long-term semantic information, are all preserved. The combined operation not only maximizes the retention of the various semantic information needed for predicting the points, but also avoids excessive redundancy in order to obtain accurate predictions. This module can be formalized as follows:
$$
H _ {\mathrm {r e p}} ^ {\mathrm {h o r}} = \operatorname {R e p a t} \left(h _ {\mathrm {h o r}}\right)
$$
$$
H _ {\mathrm {h} - \mathrm {v}} = \operatorname {R e s h a p e} \left(\left[ H _ {\text {r e p}} ^ {\text {h o r}}, H _ {\text {v e r}} \right]\right) \tag {5}
$$
$$
\widehat {Y} = \operatorname {F C 1} \left(H _ {\mathrm {h - v}}\right)
$$
$$
Y = \operatorname {F C 2} (\operatorname {R e s h a p e} (\widehat {Y}) + T F E _ {\mathrm {d e}}),
$$
where  $TFE_{\mathrm{de}} \in \mathbb{R}^{P \times d_{\mathrm{model}}}$  represents time features encoding of the forecasting points separately.  $h_{\mathrm{hor}} \in \mathbb{R}^{L \times 1 \times d_{\mathrm{model}}}$  represents the last row hidden state in  $H_{\mathrm{hor}}$ . Repeat( $\cdot$ ) is for repeat operation and
$H_{rep}^{\mathrm{hor}} \in \mathbb{R}^{L \times C \times d_{\mathrm{model}}}$  represents the hidden state after the repeat operation.  $H_{\mathrm{h - v}} \in \mathbb{R}^{C \times (L*2d_{\mathrm{model}})}$  is the vector of horizontal and vertical combination after reshape operation. FC1 and FC2 represent two fully connected layers respectively.  $\widehat{Y} \in \mathbb{R}^{C \times (R_{\mathrm{de}}*d_{\mathrm{model}})}$  represents the intermediate variables of the calculation and  $R_{\mathrm{de}} \times C = P$ .  $Y$  represents the output of this module, and it is worth noting that we need to utilize the adaptive parameter norm for denormalization, when norm = 1,  $Y = Y + x_H$ .
Datasets To evaluate the proposed WITRAN, we conducted extensive experiments on four widely recognized real-world benchmark datasets for long-range and ultra-long-range forecasting. These datasets cover energy, traffic, and weather domains. We split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2. More details about the datasets and implementation are described in Appendix E and Appendix F.
Baselines In light of the underperformance produced by classical models such as ARIMA and simple RNN/CNN models, as evidenced by [Zhou et al., 2021] and [Wu et al., 2021], and the subpar performance exhibited by certain Transformer-based models like LogTrans [Li et al., 2019] and Reformer [Kitaev et al., 2020], as shown in [Wu et al., 2023] and [Wang et al., 2023], our study primarily incorporates six transformer-based models: PatchTST [Nie et al., 2023], FEDformer [Zhou et al., 2022a], Pyraformer [Liu et al., 2021], Autoformer[Zhou et al., 2022a], Informer [Zhou et al., 2021], Vanilla Transformer [Vaswani et al., 2017], and four non-transformer-based methods: MICN [Wang et al., 2023], TimesNet [Wu et al., 2023], DLinear [Zhou et al., 2022b], FiLM [Zhou et al., 2022b]. Please refer to Appendix A and Appendix H for more details of the baselines and implementation.
For a better comparison, we adopted the experimental setup of Pyraformer [Liu et al., 2021] for long-range and ultra-long-range series forecasting. In addition, channel-independence is crucial for multivariate time series prediction [Nie et al., 2023], so it is necessary to verify the performance of models on a single channel to ensure their effectiveness across all channels in multivariate time series prediction. In this paper, experiments were conducted on a single channel. Note that in order to fairly compare the performance of each model, we set up the search space so that each model can perform optimally on each task. For further details, please refer to Appendix F.
Long-range Forecasting Results We conducted five tasks on each dataset for long-range prediction, and the results are shown in Table 3. Taking the task setting 168-336 on the left side of the Table 3 as an example, it indicates that the input length is 168 and the prediction length is 336. Notably, our proposed WITRAN achieves state-of-the-art performance, surpassing the previous best method with an average MSE reduction of  $5.803\%$ . Specifically, WITRAN exhibits an average MSE reduction of  $10.246\%$  for ECL,  $3.879\%$  for traffic,  $2.519\%$  for ETTh1,  $4.431\%$  for ETTh2, and  $7.939\%$  for weather. Additionally, we note that the competition among the baselines is intense due to their best performance on each task, but none of them consistently performs well across all tasks. In contrast, WITRAN demonstrates its robust competitiveness across various tasks and datasets. In addition, we can find that in most cases, a longer input length yields greater improvement with the same prediction length. All findings above collectively highlight WITRAN's efficacy in addressing diverse time-series forecasting tasks in real-world applications. Further results and showcases are presented in Appendix H and Appendix J.
Ultra-long-range Forecasting Results We also show the three tasks for ultra-long-range prediction results on each dataset in Table 4. The tasks on the left side of the table hold the same interpretation as above. Notably, WITRAN achieves an  $14.275\%$  averaged MSE reduction. More specifically, WITRAN demonstrates an average MSE reduction of  $39.916\%$  for ECL,  $3.122\%$  for traffic,  $14.837\%$  for ETTh1,  $2.441\%$  for ETTh2, and  $11.062\%$  for weather. In particular, our method showcases substantial improvements of over  $10\%$  in ECL, ETTh1, and weather, reinforcing our ability to predict ultra-long-range time series. And by comparing the results of ultra-long-range forecasting and long-range Forecasting, we can find that our method is more competitive for ultra-long-range prediction,
Table 4: Ultra-long-range forecasting results. A lower MSE or MAE indicates a better prediction. The best results are highlighted in bold and the suboptimal results are underlined.
Methods	WITRAN(Ours)	MICN	TimesNet	PatchTST	DLinear	FiLM	FEDformer	Pyraformer	Autoformer	Informer	Transformer
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE
ETB1	720-1440	0.2499	0.3727	1.0460	0.7765	0.6119	0.5962	0.8243	0.6704	0.4923	0.5473	0.4377	0.4833	0.5393	0.3250	0.4332	1.4957	0.9533	0.5064	0.5317	0.4030	0.4797
1440-1440	0.2408	0.3680	2.2862	1.2207	0.5720	0.5712	0.9053	0.7328	0.5146	0.5615	0.4849	0.5429	0.5142	0.5571	0.4895	0.5280	1.7873	1.0283	0.7247	0.6292	0.5531	0.5524
1440-2880	0.3359	0.4383	2.8936	1.3717	0.7683	0.6846	1.1282	0.8087	0.8355	0.7193	0.6847	0.6493	3.9018	1.5276	0.4320	0.5161	1.2867	0.8878	0.6152	0.5953	0.5243	0.5460
Traffic	720-1440	0.1672	0.2449	0.2876	0.3916	0.1882	0.2656	0.1904	0.2685	0.1639	0.2412	0.1638	0.2448	0.2753	0.3650	0.4463	0.4609	0.3104	0.4095	0.7614	0.6469	0.9876	0.7445
1440-1440	0.1543	0.2325	0.2950	0.3923	0.1598	0.2388	0.1817	0.2764	0.1599	0.2411	0.1602	0.2437	0.2848	0.3681	0.4710	0.4916	0.2970	0.3999	0.7375	0.6414	0.7340	0.6492
1440-2880	0.1425	0.2333	0.2823	0.3874	0.1560	0.2409	0.2029	0.3100	0.1550	0.2472	0.1744	0.2693	0.2952	0.3844	0.5165	0.5305	0.3035	0.3982	0.9849	0.7618	0.6000	0.5877
ETB1	720-1440	0.1331	0.2943	0.4640	0.5836	0.1391	0.3049	0.3708	0.4906	0.2952	0.4370	0.2949	0.4388	0.1768	0.3409	0.1666	0.3315	0.3298	0.4741	0.1378	0.3051	0.1905	0.3555
1440-1440	0.1304	0.2902	0.5650	0.6293	0.1801	0.3372	0.4475	0.5329	0.2200	0.3714	0.2294	0.3759	0.3574	0.4878	0.3487	0.4866	0.4531	0.5507	0.1430	0.3156	0.1972	0.3630
1440-2880	0.1850	0.3452	0.7591	0.7215	0.2732	0.4094	0.9617	0.8271	0.3773	0.4794	0.6834	0.7096	0.4269	0.5252	0.5857	0.6760	1.3566	0.9235	0.3177	0.4733	0.3495	0.4911
ETB2	720-1440	0.2915	0.4289	0.4922	0.5649	0.4186	0.5092	0.9401	0.7680	0.5037	0.5645	0.7166	0.6628	0.3731	0.4827	0.2952	0.4336	0.5633	0.5996	0.4025	0.4991	0.3712	0.4805
1440-1440	0.2815	0.4220	0.5030	0.5644	0.4409	0.5218	0.7860	0.6704	0.5176	0.5734	0.7446	0.6590	0.3906	0.4951	0.2946	0.4316	0.8029	0.7140	0.3484	0.4786	0.3797	0.4818
1440-2880	0.3280	0.4585	0.5549	0.5886	1.5304	0.9026	2.0561	1.1595	0.5053	0.5584	3.2835	1.6030	1.7167	0.9698	0.3345	0.4544	4.1031	1.7198	0.3335	0.4482	0.3737	0.4787
Weather	720-1440	0.1872	0.3312	0.3999	0.4848	0.2407	0.3694	0.5453	0.5631	0.4406	0.5264	0.6360	0.5997	0.2352	0.3733	0.6810	0.6352	0.8599	0.7064	0.2466	0.3849	0.2188	0.3512
1440-1440	0.1907	0.3366	0.2873	0.4201	0.2869	0.4033	0.5371	0.5559	0.3147	0.4417	0.6002	0.5880	0.2226	0.3609	0.2401	0.3777	0.9766	0.7739	0.2556	0.3969	0.2610	0.3823
1440-2880	0.1769	0.3257	0.3570	0.4810	0.2199	0.3563	0.9061	0.7220	0.3197	0.4533	1.2605	0.8805	0.2138	0.3599	0.1852	0.3332	1.7465	1.0962	0.2126	0.3600	0.1993	0.3436
underscoring WITRAN's capability to capture semantic information in ultra-long historical input. More results and showcases are provided in Appendix H and Appendix J.
4.2 Time and Memory Consumption
To assess the efficiency of our proposed WITRAN, we compared its time and memory consumption with selected baselines that demonstrate superior performance. To fully evaluate the actual complexity of our model and other models for the long range forecasting and ultra-long range forecasting tasks, we set up two cases: 1) constant input length and varying prediction length, 2) constant prediction length and varying input length. In both cases, we fixed the input length and output length at 720. Figure 5 illustrates the comparison, revealing that WITRAN achieves the lowest actual time complexity and memory complexity. This section is described in more detail in Appendix I.
(a) Memory consumption (input length is fixed as 720)
(b) Time consumption (input length is fixed as 720)
Figure 5: Time and memory consumption. WIT is the framework that does not involve the RAN.
(c) Memory consumption (output length is fixed as 720)
(d) Time consumption (output length is fixed as 720)
The impact of RAN can be clearly observed in Figure 5. In addition, we conducted ablation studies to measure the impact of HVGSU and GSC in WIT, and the combined operation of the last row of horizontally hidden states and each column of vertically hidden states. Detailed results and discussions
can be found in Appendix G. Here, we provide a concise summary of the key findings: 1) The fusion and selection design of GSC enables both long- and short-term semantic information to be captured for each historical input data points. 2) Setting up independent cells in both directions enables to extract semantic information in the long- and short-term respectively. 3) The specially designed combined operation can make fuller use of the captured local and global semantic information while ensuring that the information is not redundant. 4) RAN offers advantages in terms of speed and space complexity. The utilization of RAN eliminates the need to store excessive intermediate variables, as shown in the previous section.
We have followed MICN [Wang et al., 2023] and introduced a simple white noise injection to demonstrate the robustness of our model. Specifically, we randomly select a proportion  $\varepsilon$  of data from the original input sequence and apply random perturbations within the range  $[-2X_i, 2X_i]$  to the selected data, where  $X_i$  denotes the original data. After the noise injection, the data is then used for training, and the MSE and MAE metrics are recorded in Table 5.
Table 5: Robustness experiments of forecasting results. Different  $\varepsilon$  indicates different proportions of noise injection. And WITRAN is used as the base model.
TasksMetric	168-168	168-336	336-336	336-720	720-720	720-1440	1440-1440	1440-2880
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ECL	ε = 0%	0.2397	0.3519	0.2607	0.3721	0.2517	0.3627	0.3084	0.4055	0.2478	0.3651	0.2499	0.3727	0.2408	0.3680	0.3359	0.4383
ε = 1%	0.2420	0.3534	0.2630	0.3738	0.2516	0.3626	0.3078	0.4060	0.2345	0.3960	0.2502	0.3739	0.2395	0.3672	0.3372	0.4476
ε = 5%	0.2463	0.3573	0.2692	0.3788	0.2567	0.3652	0.3039	0.4046	0.2525	0.3703	0.2565	0.3786	0.2410	0.3687	0.3315	0.4408
ε = 10%	0.2543	0.3621	0.2726	0.3811	0.2702	0.3736	0.3098	0.4061	0.2569	0.3756	0.2674	0.3880	0.2444	0.3716	0.3366	0.4445
Traffic	ε = 0%	0.1377	0.2051	0.1321	0.2059	0.1306	0.2054	0.1391	0.2175	0.1408	0.2191	0.1672	0.2449	0.1543	0.2325	0.1425	0.2333
ε = 1%	0.1376	0.2063	0.1329	0.2083	0.1316	0.2074	0.1423	0.2218	0.1409	0.2218	0.1676	0.2466	0.1551	0.2372	0.1436	0.2360
ε = 5%	0.1370	0.2115	0.1362	0.2148	0.1323	0.2106	0.1432	0.2260	0.1416	0.2247	0.1699	0.2519	0.1572	0.2373	0.1519	0.2480
ε = 10%	0.1409	0.2164	0.1355	0.2193	0.1372	0.2200	0.1467	0.2325	0.1450	0.2291	0.1652	0.2475	0.1561	0.2383	0.1575	0.2580
ETBh1	ε = 0%	0.1105	0.2589	0.1189	0.2714	0.1112	0.2638	0.1494	0.3092	0.1296	0.2868	0.1331	0.2943	0.1304	0.2902	0.1850	0.3452
ε = 1%	0.1112	0.2596	0.1208	0.2726	0.1111	0.2637	0.1463	0.3035	0.1304	0.2885	0.1367	0.2978	0.1319	0.2907	0.1834	0.3484
ε = 5%	0.1135	0.2622	0.1221	0.2751	0.1199	0.2689	0.1527	0.3124	0.1304	0.2888	0.1336	0.2952	0.1359	0.2955	0.1801	0.3435
ε = 10%	0.1137	0.2628	0.1227	0.2757	0.1196	0.2688	0.1559	0.3148	0.1358	0.2952	0.1374	0.2994	0.1405	0.3015	0.1842	0.3479
ETBh2	ε = 0%	0.2389	0.3813	0.2277	0.3778	0.2432	0.3922	0.2373	0.3888	0.2635	0.4018	0.2915	0.4289	0.2815	0.4220	0.3280	0.4585
ε = 1%	0.2535	0.3904	0.2284	0.3777	0.2459	0.3942	0.2390	0.3902	0.2629	0.3902	0.3049	0.4358	0.2936	0.4297	0.3353	0.4642
ε = 5%	0.2364	0.3799	0.2379	0.3834	0.2563	0.4025	0.2501	0.3967	0.2661	0.4056	0.3207	0.4482	0.2935	0.4308	0.3307	0.4603
ε = 10%	0.2603	0.3959	0.2475	0.3902	0.2613	0.4054	0.2581	0.4022	0.2674	0.4053	0.3193	0.4481	0.2847	0.4237	0.3317	0.4612
Weather	ε = 0%	0.2050	0.3338	0.2197	0.3470	0.2163	0.3482	0.2054	0.3424	0.2008	0.3417	0.1872	0.3312	0.1907	0.3366	0.1769	0.3257
ε = 1%	0.2050	0.3343	0.2154	0.3470	0.2214	0.3522	0.2055	0.3419	0.2005	0.3419	0.1872	0.3313	0.1903	0.3361	0.1828	0.3327
ε = 5%	0.2057	0.3362	0.2241	0.3517	0.2268	0.3557	0.2058	0.3426	0.2008	0.3421	0.1867	0.3305	0.1897	0.3353	0.1831	0.3357
ε = 10%	0.2059	0.3369	0.2220	0.3484	0.2308	0.3595	0.2059	0.3438	0.2007	0.3418	0.1854	0.3290	0.1900	0.3370	0.1828	0.3336
It can be found that as the perturbation proportion increases, there is a slight increase in the MSE and MAE metrics in terms of forecasting. It indicates that WITRAN demonstrates good robustness when dealing with less noisy data (up to  $10\%$ ), and it possesses a significant advantage in effectively handling various abnormal data fluctuations.
In this paper, we propose WITRAN, a novel Water-wave Information Transmission framework and a universal acceleration framework. WITRAN effectively captures both long- and short-term repetitive patterns and global and local correlations with  $\mathcal{O}(\sqrt{L})$  time complexity and  $\mathcal{O}(L)$  memory complexity. The experimental results demonstrate the remarkable performance and efficiency of WITRAN. However, this recurrent structure is still not optimally efficient for Python-based implementations because of the information waiting between slices. Therefore, we plan to explore the integration of WITRAN into an interface using C++, similar to the implementation of nn.GRU/nn.LSTM in PyTorch in the future, to further improve its efficiency.
This work was supported by the Fundamental Research Funds for the Central Universities under Grant No. 2022YJS142. Furthermore, we would like to express our gratitude to Shuo Wang and Shuohao Lin for their valuable discussions and assistance.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. Pmlr, 2013.
George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):91-109, 1968.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37-45, 2018.
G Athanasopoulos and RJ Hyndman. Forecasting: principle and practice: O texts; 2018, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems, 30, 2017.
Zeping Yu and Gongshen Liu. Sliced recurrent neural networks. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2953-2964, 2018.
Lifeng Shen, Qianli Ma, and Sen Li. End-to-end time series imputation via residual short paths. In Asian conference on machine learning, pages 248-263. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
A A Brief Overview of RNN-Based and Transformer-Based Models
In this section, we will provide a brief overview of representative variants of RNN-based methods (Appendix A.1) and Transformer-based methods (Appendix A.2). Detailed descriptions of other methods can be found in Section 2.
A.1 RNN-based methods
LSTM [Hochreiter and Schmidhuber, 1997] and GRU [Chung et al., 2014] are two prominent variants of RNN-based models. They have gained popularity in diverse domains, such as natural language processing, speech recognition, and video analysis due to their ability to capture important information. This subsection, we will provide a detailed introduction to LSTM and GRU.
(a) LSTM cell structure
Figure 6: The structure of LSTM and GRU.
(b) GRU cell structure
LSTM The structure of LSTM, shown in Figure 6(a), comprises three gates: the input gate, the forget gate and the output gate. In addition, LSTM has two vectors: the hidden state and the cell state, for recursively processing the information flow between different time steps. LSTM can be formalized as follows:
$$
f _ {t} ^ {\text {l s t m}} = \sigma \left(W _ {f} ^ {\text {l s t m}} \left[ h _ {t - 1}, x \right] + b _ {f} ^ {\text {l s t m}}\right)
$$
$$
i _ {t} ^ {\mathrm {l s t m}} = \sigma \left(W _ {i} ^ {\mathrm {l s t m}} \left[ h _ {t - 1}, x \right] + b _ {i} ^ {\mathrm {l s t m}}\right)
$$
$$
\widetilde {C _ {t}} = \tanh  \left(W _ {c} ^ {\text {l s t m}} \left[ h _ {t - 1}, x \right] + b _ {c} ^ {\text {l s t m}}\right) \tag {6}
$$
$$
o _ {t} ^ {\text {l s t m}} = \sigma \left(W _ {o} ^ {\text {l s t m}} \left[ h _ {t - 1}, x \right] + b _ {o} ^ {\text {l s t m}}\right)
$$
$$
C _ {t} = f _ {t} ^ {\mathrm {l s t m}} \odot C _ {t - 1} + i _ {t} ^ {\mathrm {l s t m}} \odot \widetilde {C _ {t}}
$$
$$
h _ {t} = o _ {t} ^ {\operatorname {l s t m}} \odot \tanh  (C _ {t}),
$$
where  $f_{t}^{\mathrm{lstm}}$ ,  $i_{t}^{\mathrm{lstm}}$  and  $o_{t}^{\mathrm{lstm}}$  represent the input gate, the forget gate, and the output gate.  $x \in \mathbb{R}^{d_{\mathrm{in}}}$  represents the input, and  $C_{t-1}$ ,  $h_{t-1} \in \mathbb{R}^{d_{\mathrm{model}}}$  represents the input cell state and the input hidden state.  $W_{*}^{\mathrm{lstm}} \in \mathbb{R}^{d_{\mathrm{model}} \times (d_{\mathrm{model}} + d_{\mathrm{in}})}$  are weight matrices and  $b_{*}^{\mathrm{lstm}} \in \mathbb{R}^{d_{\mathrm{model}}}$  are bias vectors.  $C_{t}$  and  $h_{t}$  denote the output cell state and the output hidden state;  $\widetilde{C}_{t}$  represents the intermediate variables of the calculation;  $\odot$  denotes an element-wise product;  $\sigma(\cdot)$  and  $\tanh(\cdot)$  denote the sigmoid and tanh activation function.
GRU The structure of GRU, depicted in Figure 6(b), consists of two gates: the reset gate and the update gate. Besides, GRU has one vector: the hidden state. It can be mathematically formulated as follows:
$$
r _ {t} ^ {\mathrm {g r u}} = \sigma \left(W _ {r} ^ {\mathrm {g r u}} \left[ h _ {t - 1}, x \right] + b _ {r} ^ {\mathrm {g r u}}\right)
$$
$$
u _ {t} ^ {\mathrm {g r u}} = \sigma \left(W _ {u} ^ {\mathrm {g r u}} \left[ h _ {t - 1}, x \right] + b _ {u} ^ {\mathrm {g r u}}\right)
$$
$$
\widetilde {h _ {t} ^ {\mathrm {g r u}}} = \tanh  \left(W _ {h} ^ {\mathrm {g r u}} \left[ r _ {t} ^ {\mathrm {g r u}} \odot h _ {t - 1}, x \right] + b _ {h} ^ {\mathrm {g r u}}\right) \tag {7}
$$
$$
h _ {t} = \left(1 - u _ {t} ^ {\mathrm {g r u}}\right) \odot h _ {t - 1} + u _ {t} ^ {\mathrm {g r u}} \odot \bar {h} _ {t} ^ {\mathrm {g r u}},
$$
where  $r_t^{\mathrm{gru}}$  and  $u_t^{\mathrm{gru}}$  represent the reset gate and the update gate.  $x \in \mathbb{R}^{d_{\mathrm{in}}}$  represents the input, and  $h_{t-1} \in \mathbb{R}^{d_{\mathrm{model}}}$  represent the input hidden state.  $W_*^{\mathrm{gru}} \in \mathbb{R}^{d_{\mathrm{model}} \times (d_{\mathrm{model}} + d_{\mathrm{in}})}$  are weight matrices and  $b_*^{\mathrm{gru}} \in \mathbb{R}^{d_{\mathrm{model}}}$  are bias vectors.  $h_t$  denotes the output hidden state;  $\widetilde{h}_t^{\mathrm{gru}}$  represents the intermediate variables of the calculation;  $\odot$  denotes an element-wise product;  $\sigma(\cdot)$  and  $\tanh(\cdot)$  denote the sigmoid and tanh activation function.
A.2 Transformer-based methods
In recent years, variants of Transformer have made significant progress in time series forecasting and can be divided into two categories.