SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here  $\bar{x}_{i,t}^{i}\in R^{V*1}$  denotes all the normalized variables in time step  $t$ . From the above equations, we can learn that any input sequence follows a standard normal distribution after the normalization operation of SAN, which meets our expectations.
B Supplementary Experiments
B.1 Full Benchmark on the ETT Dataset
Table 5: Multivariate forecasting results on full ETT dataset.
Methods Metric	DLinear	+ SAN	FEDformer	+ SAN	Autoformer	+ SAN	SCINet	+ SAN
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	96	0.377	0.399	0.383	0.399	0.371	0.411	0.383	0.409	0.458	0.448	0.488	0.464	0.470	0.479	0.391	0.405
192	0.417	0.426	0.419	0.419	0.420	0.443	0.431	0.438	0.481	0.474	0.498	0.472	0.541	0.520	0.438	0.433
336	0.464	0.461	0.437	0.432	0.446	0.459	0.471	0.456	0.508	0.485	0.530	0.498	0.643	0.587	0.477	0.451
720	0.493	0.505	0.446	0.459	0.482	0.495	0.504	0.488	0.525	0.516	0.555	0.514	0.774	0.669	0.489	0.474
ETTh2	96	0.292	0.356	0.277	0.338	0.341	0.382	0.300	0.355	0.384	0.420	0.316	0.366	0.690	0.625	0.294	0.347
192	0.383	0.418	0.340	0.378	0.426	0.436	0.392	0.413	0.457	0.454	0.413	0.426	0.991	0.742	0.374	0.398
336	0.473	0.477	0.356	0.398	0.481	0.479	0.459	0.462	0.468	0.473	0.446	0.457	1.028	0.759	0.412	0.430
720	0.708	0.599	0.396	0.435	0.458	0.477	0.462	0.472	0.473	0.485	0.471	0.474	1.363	0.885	0.437	0.461
ETTh1	96	0.301	0.344	0.288	0.342	0.362	0.408	0.311	0.355	0.493	0.470	0.343	0.378	0.444	0.464	0.321	0.360
192	0.335	0.366	0.323	0.363	0.395	0.427	0.351	0.383	0.546	0.498	0.390	0.400	0.491	0.500	0.347	0.380
336	0.370	0.387	0.357	0.384	0.441	0.454	0.390	0.407	0.658	0.543	0.415	0.418	0.572	0.556	0.385	0.403
720	0.425	0.421	0.409	0.415	0.488	0.481	0.456	0.444	0.626	0.532	0.476	0.453	0.728	0.654	0.450	0.441
ETTh2	96	0.169	0.263	0.166	0.258	0.191	0.283	0.175	0.266	0.261	0.329	0.236	0.317	0.303	0.404	0.176	0.267
192	0.232	0.310	0.223	0.302	0.261	0.326	0.246	0.315	0.282	0.339	0.260	0.329	0.568	0.569	0.240	0.311
336	0.303	0.361	0.272	0.330	0.327	0.365	0.315	0.362	0.350	0.378	0.330	0.376	0.793	0.689	0.300	0.351
720	0.403	0.424	0.360	0.384	0.428	0.423	0.412	0.422	0.438	0.428	0.417	0.428	1.200	0.851	0.391	0.405
We provide the full multivariate forecasting results on the ETT dataset in Table 5, which includes the hourly datasets ETTh1&ETTh2 and the 15-minutes datasets ETTm1&ETTm2. It is obvious that SAN also achieves significant improvements on these datasets on various backbone models.
B.2 Univariate Forecasting Results
Following the same settings of our main experiment, we provide the univariate forecasting results in Table 6. Similar to the results of multivariate forecasting, SAN can boost the performance of mainstream forecasting models in most cases. On average of all the benchmark settings, DLinear enhanced by SAN reduces MSE by  $6.04\%$  (from 0.230 to 0.214). The improvements for FEDformer, Autoformer and SCINet are  $15.40\%$ ,  $29.27\%$  and  $36.29\%$  respectively.
B.3 Validation on Various Input Lengths
The input length plays an essential role in time series forecasting tasks as it determines how much historical temporal information the model can mine. One may hope that for powerful deep models, the longer the input length, the better the forecasting results. However, a recent study on this question reveals that deep Transformer-based models are not capable of capturing temporal dependencies in the long-term input sequences [39]. That is, the performance of these deep models stays stable or even degrades when the input length increases.
Apart from the design of these deep models, we hold that such a phenomenon can be raised by the non-stationary property of time series. As the input length increases, the variance among input sequences grows larger and ultimately makes it harder for deep models to discover the time-invariant patterns. Therefore, by removing the non-stationary factors in the input by SAN, deep models are expected to exhibit a steady decline in metrics with longer input lengths.
To evidence our thoughts, we conduct long-term forecasting experiments, i.e.,  $L_{out} = 720$ , with various input lengths  $L_{in} \in \{24, 48, 72, 96, 120, 144, 168, 192, 336, 504, 672, 720\}$  on the Transformer-based models. Here we choose Transformer [35], Informer [43], Autoformer [38] and FEDformer [45] as the backbone models. The MSE evaluations are plotted in Fig. 4. Note that we omit large values in the line chart to better demonstrate the trend of the overall results. From the figure, we can see that with the assistance of SAN, the performance of deep models with long sequence input is
(a) 720 steps-Electricity
(b) 720 steps-Weather
Figure 4: The long-term forecasting MSE evaluations of different Transformer-based models under various input lengths. Large values are discarded to illustrate the overall trend better.
largely improved. When the input length is set to 720 on the Electricity dataset, the performance of Informer has been boosted by  $77.83\%$  (from 0.9426 to 0.2090), and the average improvement on four backbones under the same setting is  $52.55\%$ . Moreover, all of the backbones enhanced by SAN tend to produce more accurate forecasting as the length increases. To be specific, on the Weather dataset, Transformer achieves a reduction on MSE of  $29.40\%$  when prolonging input from 24 steps to 720 steps, and the average improvement on four backbones is  $33.11\%$ . These results greatly meet our expectations and also validate the effectiveness of SAN on various input lengths.
B.4 Additional Prediction Showcases
We provide the additional comparison between SAN and other normalization methods in Fig. 5 with FEDformer [45] on various datasets. Clearly, SAN can better estimate the future distribution so as to help the backbone model to achieve superior performance, where the forecasting results are better aligned with the groundtruth.
Figure 5: Illustration of the additional prediction showcases comparing SAN and baseline models. The experiment is conducted on the ILI, ETTh2, and Exchange dataset. Following the same input sequence length setting in our main experiments, the target sequence length is set to 24, 192, and 336 respectively.
B.5 Ablation Study
Statistie Prediction Module In this section, we aim to analyze the effectiveness of our designs in the statistic prediction module. We instantiate our method and its variants on Autoformer and test their performance on two typical non-stationary datasets: Exchange and ETTh2. Similarly, we repeat the experiments three times with fixed seed and report the evaluations with standard deviation in Table 7.
Obviously, with the proposed two techniques combined, the statistic prediction module can achieve the best accuracy, leading to optimal forecasting performance. Besides, both residual learning and individual preference contribute positive effects and the former one is much more important, without which SAN can even bring negative effects to the backbone model. These results validate the rationality of our thoughts about the characteristics of the mean value and also reveal the importance of accurate modeling of future statistics to SAN. Besides, SAN without individual
Table 7: Forecasting errors under the multivariant setting with respect to variants of SAN. The best performance are highlighted in bold.
Variants Metric	SAN	w/o individual	w/o residual	w/o SAN
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Exchange	96	0.082±0.001	0.208±0.001	0.089±0.005	0.209±0.007	0.135±0.003	0.264±0.001	0.152±0.006	0.283±0.007
192	0.157±0.001	0.296±0.003	0.184±0.009	0.306±0.010	0.331±0.044	0.416±0.025	0.369±0.055	0.437±0.033
336	0.262±0.004	0.385±0.002	0.340±0.001	0.422±0.001	0.658±0.044	0.593±0.024	0.534±0.130	0.544±0.066
720	0.689±0.043	0.629±0.020	0.982±0.001	0.753±0.002	1.456±0.011	0.882±0.009	1.222±0.099	0.848±0.021
ETTh2	96	0.316±0.001	0.366±0.001	0.321±0.013	0.367±0.008	0.383±0.020	0.413±0.012	0.384±0.021	0.420±0.013
192	0.413±0.013	0.426±0.007	0.414±0.023	0.422±0.012	0.463±0.030	0.469±0.020	0.457±0.020	0.454±0.014
336	0.446±0.004	0.457±0.003	0.448±0.003	0.453±0.001	0.586±0.025	0.541±0.009	0.468±0.010	0.473±0.005
720	0.471±0.009	0.474±0.005	0.483±0.012	0.477±0.005	0.889±0.007	0.682±0.001	0.473±0.005	0.485±0.005
modeling performs well on the ETT2 dataset but performs poorly on the Exchange dataset. Such a phenomenon reveals that the evolving trends of different scenarios vary, and it is required to model the complex relationships among multiple variables individually. Moreover, since we only incorporate the properties of mean values into a simple MLP network, how to design a proper mechanism or network architecture for statistics modeling is a promising direction for optimizing our method, and we leave such explorations for future work.
Slicing Length The slicing length is a key parameter of SAN. We aim to study the effect of different slicing lengths on our method. Ablation experiments are conducted by using SCINet as the backbone model under the long-term forecasting setting ( $L_{out} = 60$  for the ILI dataset and  $L_{out} = 720$  for the rest datasets). Each experiment is conducted three times with a fixed random seed. The forecasting errors and the corresponding standard deviation are presented in Table 8.
Table 8: Forecasting errors under the multivariant setting with respect to different slicing lengths. The best performance are highlighted in bold.
Slicing Length Metric	6	12	24	48
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Electricity	0.210±0.002	0.305±0.002	0.207±0.002	0.305±0.002	0.206±0.004	0.307±0.003	0.208±0.002	0.307±0.001
Exchange	0.892±0.028	0.712±0.013	0.895±0.037	0.712±0.017	0.901±0.005	0.715±0.002	0.898±0.037	0.714±0.015
Traffic	0.612±0.001	0.376±0.001	0.608±0.002	0.373±0.001	0.607±0.001	0.381±0.001	0.611±0.002	0.382±0.002
Weather	0.338±0.002	0.366±0.002	0.338±0.001	0.365±0.002	0.340±0.001	0.367±0.001	0.339±0.001	0.366±0.001
ILI	2.487±0.034	1.063±0.008	2.680±0.055	1.118±0.015	n/a	n/a	n/a	n/a
ETTh1	0.491±0.002	0.475±0.001	0.488±0.001	0.474±0.001	0.489±0.004	0.473±0.001	0.492±0.004	0.474±0.002
ETTh2	0.440±0.001	0.465±0.001	0.435±0.002	0.460±0.002	0.437±0.007	0.459±0.006	0.443±0.007	0.462±0.004
ETTm1	0.495±0.043	0.469±0.024	0.450±0.001	0.441±0.001	0.611±0.218	0.503±0.084	0.463±0.006	0.448±0.003
ETTm2	0.391±0.001	0.406±0.001	0.391±0.001	0.405±0.001	0.392±0.001	0.405±0.001	0.403±0.009	0.415±0.006
Our heuristic selection of slicing length appears to be effective among the candidates, indicating that both artificially defined and actual periods are useful in selecting the optimal setting. Additionally, there were no significant performance differences observed under various settings, suggesting that SAN is resilient to changes in slicing length.
B.6 Detailed Results of the Comparison between SAN and Normalization Methods
In Table 9, we provide the detailed experimental results of the comparison between SAN and state-of-the-art normalization methods for non-stationary time series forecasting: RevIN [17], NST [25] and Dish-TS [10]. We re-implement the former two methods and Dish-TS is implemented by its official code<sup>11</sup>.
The table clearly shows that SAN outperforms existing approaches in most cases, except for the Weather dataset. Considering that the Weather dataset is the most stationary dataset, the results suggest that SAN can better remove the non-stationary factors in the raw data, even leading to an over-stationary issue that degrades the performance.
<sup>11</sup>https://github.com/weifantt/Dish-TS
Besides, Dish-TS performs poorly in the benchmark. While it addresses the distribution shift between input and horizon series, it fails to optimize both the coefficient network and backbone network for overlooking the intrinsic bi-level optimization target of distribution estimation and forecasting tasks. By adopting a joint training schema, Dish-TS disturbs both networks and results in poor performance in certain cases. On the opposite, SAN benefits from the proposed two-stage schema which decouples the two tasks. This allows for proper optimization of each component and leads to improved overall performance.
B.7 SAN for Slice-based Forecasting Methods
In this section, we turn to investigate the generalizability of SAN towards recently emerged slice-based forecasting methods: PatchTST [26] and Crossformer [41]. We build forecasting models using their official codes and hyper-parameter settings (if available)  $^{1213}$ . For PatchTST, we replace the RevIN[17] layer with our SAN. We report the experimental results on 5 datasets in Table 10.
The results demonstrate that SAN can improve the forecasting performance of both PatchTST and CrossFormer to some extent in most cases. The improvement for PatchTST is not significant due to two main reasons: 1) RevIN has already been introduced in the model to mitigate the impact of
$^{12}$ https://github.com/yuqinie98/PatchTST
<sup>13</sup>https://github.com/Thinklab-SJTU/Crossformer
Table 10: Forecasting accuracy of SAN applied to slice-based forecasters. The bold values indicate better performance.
MethodsMetric	PatchTST	+ SAN	Crossformer	+SAN
MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Electricity	96	0.138	0.233	0.136	0.234	0.150	0.258	0.143	0.246
192	0.153	0.247	0.150	0.247	0.175	0.284	0.162	0.265
336	0.170	0.263	0.165	0.264	0.218	0.325	0.177	0.280
720	0.206	0.296	0.200	0.296	0.226	0.324	0.221	0.318
Exchange	96	0.094	0.216	0.087	0.218	0.283	0.393	0.087	0.219
192	0.191	0.311	0.181	0.323	1.087	0.804	0.171	0.313
336	0.343	0.427	0.305	0.418	1.367	0.905	0.286	0.401
720	0.888	0.706	0.659	0.620	1.546	0.987	0.749	0.653
Weather	96	0.147	0.197	0.150	0.205	0.148	0.214	0.151	0.210
192	0.191	0.240	0.194	0.252	0.201	0.270	0.198	0.253
336	0.244	0.282	0.243	0.290	0.248	0.311	0.248	0.294
720	0.320	0.334	0.311	0.343	0.366	0.395	0.322	0.350
ETTh1	96	0.382	0.403	0.375	0.398	0.390	0.417	0.387	0.402
192	0.416	0.423	0.413	0.422	0.424	0.448	0.413	0.425
336	0.441	0.440	0.428	0.434	0.486	0.492	0.436	0.431
720	0.470	0.475	0.445	0.461	0.507	0.519	0.467	0.474
ETTm2	96	0.174	0.261	0.167	0.260	0.330	0.401	0.170	0.262
192	0.238	0.307	0.222	0.298	0.623	0.543	0.224	0.301
336	0.293	0.346	0.276	0.334	0.887	0.637	0.274	0.333
720	0.373	0.401	0.366	0.393	0.844	0.640	0.366	0.390
non-stationary time series; 2) Since both are slice-based methods that split series into slices (non-overlapping slices for SAN and overlapping patches for PatchTST), the parameter settings may have a greater impact on performance and how to determine proper settings require further explorations. Besides, The official code of CrossFormer provides parameter settings for Electricity, Weather, and ETTh1 datasets. For Exchange and ETTm2 datasets, we extracted common and reasonable settings. Without SAN, CrossFormer performs poorly compared to PatchTST on these latter two datasets due to unsuitable parameters. However, when enhanced with SAN under the same setting, CrossFormer achieves competitive or even superior performance. This phenomenon reveals that SAN can potentially reduce reliance on parameter settings for backbone models while also reducing costs associated with parameter adjustment in real-world forecasting applications, further validating our attempt to divide complex non-stationary forecasting task into two easier subtasks through the two-stage training schema.
C Implementation Details
C.1 Architecture of Statistic Prediction Module
The computation of  $\mathrm{MLP}(x_1,x_2)$  in our paper can be summarized as follows:
$$
\begin{array}{l} x _ {1} = a c t _ {1} \left(W _ {1} * x _ {1}\right) \\ x _ {2} = a c t _ {1} \left(W _ {2} * x _ {2}\right) \tag {9} \\ i t = a c t _ {2} \left(W _ {3} * x\right) \\ \end{array}
$$
Here, the symbol  $[^{*};^{*}]$  represents the concatenate operation. We set  $act_{1}(.)$ ,  $act_{2}(.) = Relu(.)$ ,  $Relu(.)$  for standard deviation and the activate function of the mean is set to  $Tanh(.)$ ,  $Identity(.)$  respectively.  $W_{1}, W_{2}, W_{3}$  are learnable transformation matrices with hidden sizes of  $\{512, 512, 1024\}$ .
C.2 Algorithm of The Two-stage Training Schema
To apply SAN to backbone forecasting models, we propose a two-stage training schema to tackle the challenge of the bi-level optimization target. The statistics prediction module is first trained into
convergence, which is then frozen and treated as a plugin during the second stage of training the forecasting model. We provide the pseudo-code of such a procedure in Alg. 1.
Algorithm 1 Two-stage Training Schema.
Require: Input series  $X = \{\pmb{x}^i\}_{i=1}^N$ ; Horizon series  $Y = \{\pmb{y}^i\}_{i=1}^N$ ; Slicing length  $T$   
1: Initialize parameters  $\phi, \theta$   
2: while not converge do  
3: for all input  $\pmb{x}^i \in X$ , horizon  $\pmb{y}^i \in Y$  do  
4: Compute input statistics  $\mu_j^i, \sigma_j^i$  by Eq. 1 with  $T$   
5: Predict future statistics  $\hat{\mu}^i, \hat{\sigma}^i$  by Eq. 3 using  $f_{\phi}(*)$   
6: Update  $\phi$  using loss function  $l_{sp}$   
7: end for  
8: end while  
▶ Training of the statistics prediction module  
9:  
10: while not converge do  
11: for all input  $\pmb{x}^i \in X$ , horizon  $\pmb{y}^i \in Y$  do  
12: Compute input statistics  $\mu_j^i, \sigma_j^i$  by Eq. 1 with  $T$   
13:Normalize input series to  $\bar{\pmb{x}}^i$  by Eq. 2  
14:Forecast  $\bar{\pmb{y}}^i = g_{\theta}(\bar{\pmb{x}}^i)$   
15:Predict future statistics  $\hat{\mu}^i, \hat{\sigma}^i$  by Eq. 3 using  $f_{\phi}(*)$   
16:  $\hat{\mu}^i$ .detach(),  $\hat{\sigma}^i$ .detach() ▷ Stop-gradient, freeze the statistics prediction module  
17:Denormalize  $\bar{\pmb{y}}^i$  to  $\hat{\pmb{y}}^i$  by Eq. 4  
18:Update  $\theta$  using loss function  $l_{fc}$   
19:end for  
20:end while  
▶ Training of the forecasting model
D Limitations
Though SAN shows promising performance on the benchmark dataset, there are still some limitations of this method. First is that we mainly select the slicing length heuristically or search in predefined candidates and the current design cannot handle indivisible length or the multi-period characteristic of time series. Such a solution works for the experiments but lacks generality in real-world applications. Second is that SAN may lead to an over-stationary issue, leading to sub-optimal performance. Moreover, when applied to similar slice-based methods (especially overlapping-slice-based ones), determining how to adjust SAN's parameters for them is not a straightforward task. Therefore, a more flexible solution with automatic slicing length selection and normalization intensity control will be our exploring direction.