SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Section 4.3, we presented the results of our ablation experiments to demonstrate the effectiveness of each component in the Water-wave Information Transmission (WIT) framework. We design five variant methods and evaluate their performance on four datasets. The experimental results are summarized in Table 8, and we will provide a detailed analysis of these results in this section. It is important to note that the role of the Recurrent Acceleration Network (RAN) is primarily to improve computational efficiency, as demonstrated in Section 4.2, and it does not significantly impact the model's accuracy. Therefore, our focus in this section will be on discussing the design and role of the WIT framework.
G.1 Impact of forecasting module
Our proposed forecasting module is also a part of the model design, and its goal is to effectively capture periodic semantic information and global-local semantic information obtained through the Water-wave Information Transmission (WIT) framework. Our specific design has been described in detail in Section 3.4. To verify that we can fully utilize the relevant information by combining operations for pairwise prediction, we changed the combination prediction method of this module to a simple way of first concatenating the horizontal hidden state and the vertical hidden state, and then using the fully connected prediction module for prediction. We named this variant WITRAN-FC.
The comparison of results between WITRAN and WITRAN-FC in Table 8 shows that when we remove the combining operations for pairwise prediction and replace it with ordinary concatenation and fully connected operations, even though the same information is used, WITRAN-FC has difficulty discovering the most relevant long-term periodic semantic information for each prediction point. This demonstrates the effectiveness of the combining operations for pairwise prediction in the forecasting module.
G.2 Impact of information fusion and selection
As information fusion and selection are completed together in the Gated Selective Cell (GSC) we designed, we need to verify the effectiveness of GSC, that is, the effectiveness of information fusion and selection. Here, we replaced the cells in the two directions of Horizontal Vertical Gated Selective Unit (HVGSU) with LSTM and GRU respectively, and named them WITRAN-2DLSTM and WITRAN-2DGRU. It should be noted that since the information between the LSTM/GRU cell in the two directions is not fused at this time, the information in the last row of the horizontal direction cannot contain global information, so we cannot use the proposed forecasting module for application, but only use the FC method mentioned in the previous subsection for prediction. It should be noted that the comparison between WITRAN-2DLSTM/WITRAN-2DGRU and WITRAN-FC reflects the impact of information redundancy on experimental performance, while the comparison between WITRAN-2DLSTM/WITRAN-2DGRU and WITRAN reflects the impact of information fusion and selection on experimental performance.
The comparison of results between WITRAN-2DLSTM/WITRAN-2DGRU and WITRAN-FC in Table 8 demonstrates that too much information redundancy may be detrimental to the model's prediction performance, because for WITRAN-FC, the horizontal hidden states it receives contain more information as they go further down, and they also contain all the information in the horizontal direction above them. The same applies to the vertical direction, where the further to the right the hidden state is, the more information it contains. Furthermore, the information captured by WITRAN-2DLSTM and WITRAN-2DGRU is independent among different horizontal rows and among different vertical columns, and the semantic information between horizontal and vertical
hidden state are different. Therefore, WITRAN-2DLSTM and WITRAN-2DGRU are not heavily influenced by redundant information. In addition, the comparison of results between WITRAN and WITRAN-2DLSTM/WITRAN-2DGRU shows that information fusion and selection can effectively extract relevant information, which demonstrates the necessity of the GSC design in WIT.
G.3 Impact of bi-granular information transmission
In the WITRAN we designed, information can be propagated in two directions, and because the time granularity of sequence propagation in these two directions is different, their periodic semantic information is different. In order to verify the effectiveness of the two independent GSCs set up in HVGSU to capture long- and short-term periodic semantic information, we replaced the cells in the two directions of WITRAN-2DLSTM/WITRAN-2DGRU with a single LSTM/GRU cell, and uniformly used the hidden state information of one point to transmit information in both directions. It should be noted that at this time, the information propagated horizontally and vertically by the model is completely the same. Sure, we can name the current models as WITRAN-LSTM/WITRAN-GRU.
The comparison of results between WITRAN-2DLSTM/WITRAN-2DGRU and WITRAN-LSTM/WITRAN-GRU in Table 8 demonstrates that failing to distinguish between long- and short-term periodic semantic information can have a significant negative impact on the performance of prediction tasks.
G.4 Summary
In this section, we focused on the importance of each building module of WIT. During the experiments, we gradually removed the building blocks we designed, and it was clear that the performance decreased significantly. This fully demonstrates the rationality and effectiveness of the WIT framework we designed. Specifically, the summary of WIT is as follows: (1) The fusion and selection design of GSC enables both long- and short-term semantic information to be captured for each historical input data point. (2) Setting up independent cells in both directions enables to extract semantic information in the long- and short-term respectively. (3) The specially designed combined operation is able to make fuller use of the captured local and global semantic information while ensuring that the information is not redundant.
H Experiment Error Bars
We saved all models and used the model with the lowest validation loss for the final testing. We repeated this process 5 times and calculated the error bars for all models to compare their robustness on long-range and ultra-long-range tasks, as shown in Table 9. It can be seen from Table 9 that the overall performance of the proposed WITRAN is better than other state-of-the-art baseline models, indicating the effectiveness of our approach.
I Model Analysis
In this section, we will analyze the parameter sensitivity (Appendix I.1) of our proposed model and provide a detailed explanation (Appendix I.2) of the time and memory consumption in Section 4.2.
I.1 Parameter sensitivity
Regarding the private parameters of our model, norm is set to  $\{0,1\}$  and  $C$  represents the period is set to  $\{12,24,48\}$  and it is required that the length  $H$  of the input sequence is divisible by  $C$ . In this subsection, we will explain the choices of these parameters and their impact on the model's predictions.
The impact of norm For different prediction tasks on different datasets, we determined the value of norm through validation set. To verify that the value of norm conforms to the distribution of the datasets, we conducted experiments on the distribution of the training and validation sets for different prediction tasks, the results and the value of norm are shown in Table 10. We noticed that although there may be a significant difference in the mean values of the data between the training and validation
sets, when their variances are not significantly different, it indicates that the data fluctuations in the two sets are similar. In this case, the difference in the data distribution is not significant, and selecting norm as 0 is reasonable. However, when the variance difference between the training and validation sets is relatively large (approximately twice or half), there is a significant difference in the data distribution between the two sets, and norm should be 1 to re-normalize the input of the model. It should be noted that in the Weather dataset, there are negative values present, which makes its mean close to 0, resulting in a large difference between its variance and mean. However, this is reasonable. For the Traffic dataset, there are no negative values present, so even if the variances of its training and validation sets are similar, its significant fluctuations can be observed when combined with its mean. This was also demonstrated by setting the norm value to 1 during the model training process.
Table 10: The distribution of data in the training and validation sets (Mean and STD) and the value of norm.
Datasets Tasks	ECL	ETH1	ETH2	Weather
training set	validation set	norm	training set	validation set	norm	training set	validation set	norm	training set	validation set	norm
168-168	3425.733±564.8776	3036.397±388.2128	0	0.029±0.0170	0.034±0.0201	1	16.880±8.2921	6.667±4.1794	28.959±12.0653	18.680±9.0427	0.500±6.6321	1.143±7.7659
168-336	3427.480±566.9556	3036.291±388.0110	0	0.029±0.0170	0.035±0.0202	1	16.606±8.0735	6.258±3.8462	28.767±12.0604	17.922±8.5800	0.536±6.6444	1.017±7.8162
336-336	3427.455±569.1108	3037.291±388.0110	0	0.029±0.0170	0.035±0.0202	1	16.207±7.5364	5.382±3.8462	28.434±11.8740	17.855±5.8500	0.585±6.6409	1.018±8.1622
336-720	3434.150±573.2660	3037.919±387.2758	0	0.029±0.0170	0.035±0.0203	1	15.466±6.6217	5.583±4.3658	27.774±11.6120	16.360±7.7594	0.700±6.6410	0.721±7.9234
720-720	3437.773±578.4075	3037.919±387.2758	0	0.029±0.0170	0.035±0.0203	1	14.832±5.9927	5.583±3.4658	27.111±11.3299	16.360±7.7594	0.825±6.6270	0.721±7.9234
720-1440	3439.817±586.5029	3046.877±397.7761	0	0.029±0.0170	0.035±0.0204	1	14.044±5.5077	4.273±2.7600	26.403±11.4257	13.646±6.5725	0.978±6.6791	-0.547±7.5174
1440-1440	3452.135±594.6857	3046.877±397.7761	0	0.029±0.0169	0.035±0.0204	1	13.722±5.5456	4.273±2.7600	26.355±11.8918	13.646±6.5725	1.029±6.7679	-0.547±7.5174
1440-2880	3458.328±610.2118	3093.128±446.4128	0	0.029±0.0171	0.035±0.0208	1	14.195±5.5780	2.623±2.5005	28.303±12.1275	9.130±5.5363	0.725±6.8812	-3.859±5.5129
The impact of  $C$  The choice of  $C$  for the model represents the determination of the sequence period. Due to the different original acquisition granularities of each dataset, in order to ensure that they contain the same semantic information on the same task, in this paper, our experiments are conducted by aggregating them into one hour. To facilitate the discovery of long-term repetitive patterns in the sequence, we set the  $C$  to 12, 24, 48, and the specific experimental results are shown in Table 11. From Table 11, we can clearly see that for sequences with time steps in hours, using a period of 24 hours (1 day) for division yields better results. This is because in WIT, information is transmitted horizontally at the granularity of hours, while information is transmitted vertically at the granularity of days. This approach can more fully capture the long- and short-term repetitive patterns and global/local correlations hidden in the time series data.
Table 11: Parameter Sensitivity of  $C$  on ECL and Traffic datasets.
Settings for C	12	24	48
Datasets	Tasks	MSE	MAE	MSE	MAE	MSE	MAE
ECL	168-168	0.2461	0.3648	0.2397	0.3519	-	-
168-336	0.3166	0.4230	0.2607	0.3721	-	-
336-336	0.3217	0.4099	0.2517	0.3627	0.2634	0.3847
336-720	0.3397	0.4310	0.3084	0.4055	0.3233	0.4130
720-720	0.2802	0.3951	0.2478	0.3651	0.3175	0.4063
720-1440	0.2907	0.4051	0.2499	0.3727	0.2985	0.3946
1440-1440	0.3007	0.4110	0.2408	0.3680	0.2748	0.3768
1440-2880	0.4301	0.5150	0.3359	0.4383	0.3464	0.4347
Traffic	168-168	0.1598	0.2408	0.1377	0.2051	-	-
168-336	0.2272	0.2783	0.1321	0.2059	-	-
336-336	0.2131	0.2967	0.1306	0.2054	0.1412	0.2278
336-720	0.2022	0.2804	0.1391	0.2175	0.1510	0.2371
720-720	0.2028	0.2846	0.1408	0.2191	0.1505	0.2326
720-1440	0.2710	0.3311	0.1672	0.2449	0.1813	0.2605
1440-1440	0.2676	0.3301	0.1543	0.2325	0.1881	0.2652
1440-2880	0.2407	0.3443	0.1425	0.2333	0.1692	0.2765
I.2 A detailed explanation of time and memory consumption
The efficiency of the model is crucial, as even if the model's efficiency is high, it is limited if it does not have good predictive performance. Therefore, in this paper, we only compared our proposed method with methods that have good or the latest results. It should be noted that, for example, Transformer [Vaswani et al., 2017], Informer [Zhou et al., 2021] and Autoformer [Wu et al., 2021] have been shown in previous works to have higher complexity than FiLM [Zhou et al., 2022b], so we only compared our proposed method with FiLM, rather than comparing with Transformer, Informer and Autoformer again.
Memory Usage As shown in Figure 5(a) and (c), WITRAN has good memory usage with the prolonging the input length and output length. For a fair comparison, we fix the experimental settings of all methods, where we fix the input length as 720 and prolong the output length. Moreover, we fix the output length as 720 and prolong the input length. And for each model, to achieve good performance, it is important to have a sufficiently large search space to select the most suitable parameters for the model, in order to achieve the best possible results. Therefore, we set all public parameters to their upper limits to test the memory usage of each model. To allow as many models as possible to be trained, we uniformly set the batch size to 8 for testing in this section. It should be noted that if a point is not shown on the graph, it indicates that the model encountered an "out of memory" (OOM) error at that particular configuration of input sequence length and output length.
As for TimesNet, due to its large memory usage, it cannot be trained even with a batch size of 1. Therefore, we did not include TimesNet in the Figure 5. From Figure 5(a), we can observe that when the input sequence is fixed, the memory usage of WITRAN and WIT is not significantly different since they both have the same prediction module. Additionally, we found that as the output length increases, the memory usage of WITRAN and WIT also increases linearly, but our proposed method still has the lowest memory usage. From Figure 5(c), we can observe that the memory usage of WITRAN increases very slowly with the increase in sequence length, and the memory usage of WITRAN is even lower than that of WIT. This is because WITRAN does not need to retain a large number of intermediate storage variables in the computation of each slice, while WIT needs to store the intermediate variables for the entire sequence. Through comparison with other methods, we can clearly see the advantages of WITRAN on memory consumption.
Training Speed As shown in Figure 5(b) and (d), WITRAN has a faster training speed than others with the prolonging the input length and output length. To ensure a fair comparison of the performance of each model, we fixed the experimental settings, which were the same as the settings used for testing memory usage. The experiment is performed on ECL dataset. Due to our extremely low space complexity, we followed the experimental settings used in FiLM [Zhou et al., 2022b] for comparing the training Speed. To fully utilize the GPU memory, our model can increase the batch size from 8 to 32 under the experimental settings mentioned above. Similarly, FiLM can also increase the batch size = 32. However, we still have the fastest training speed, and due to the role of RAN mainly being on the input sequence, our advantage can be clearly seen from Figure 5(d), which is related to the square root of the sequence length. Through comparison with other methods, we can clearly see the advantages of WITRAN on time consumption.
J Showcases of Main Results
As shown in Figure 10 to Figure 49, we plot the forecasting results from the test set of all datasets for comparison. For the long-range forecasting task, we chose two suboptimal models, FiLM and Pyraformer. As for the ultra-long-range forecasting task, we also chose two suboptimal models, Pyraformer and TimesNet. Our model WITRAN gives the best performance among different models. Moreover, WITRAN is significantly better at modeling global and local correlations, as well as discovering long- and short-term repetitive patterns in the time series than other models.
Figure 10: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-168 task of the ECL dataset.
Figure 11: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-336 task of the ECL dataset.
Figure 12: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-336 task of the ECL dataset.
Figure 13: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-720 task of the ECL dataset.
Figure 14: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 720-720 task of the ECL dataset.
Figure 15: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 720-1440 task of the ECL dataset.
Figure 16: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-1440 task of the ECL dataset.
Figure 17: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-2880 task of the ECL dataset.
Figure 18: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-168 task of the Traffic dataset.
Figure 19: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-336 task of the Traffic dataset.
Figure 20: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-336 task of the Traffic dataset.
Figure 21: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-720 task of the Traffic dataset.
Figure 22: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 720-720 task of the Traffic dataset.
Figure 23: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 720-1440 task of the Traffic dataset.
Figure 24: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-1440 task of the Traffic dataset.
Figure 25: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-2880 task of the Traffic dataset.
Figure 26: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-168 task of the ETTh1 dataset.
Figure 27: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-336 task of the ETTh1 dataset.
Figure 28: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-336 task of the ETTh1 dataset.
Figure 29: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-720 task of the ETTh1 dataset.
Figure 30: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 720-720 task of the ETTh1 dataset.
Figure 31: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 720-1440 task of the ETTh1 dataset.
Figure 32: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-1440 task of the ETTh1 dataset.
Figure 33: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-2880 task of the ETTh1 dataset.
Figure 34: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-168 task of the ETTh2 dataset.
Figure 35: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-336 task of the ETTh2 dataset.
Figure 36: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-336 task of the ETTh2 dataset.
Figure 37: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-720 task of the ETTh2 dataset.
Figure 38: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 720-720 task of the ETTh2 dataset.
Figure 39: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 720-1440 task of the ETTh2 dataset.
Figure 40: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-1440 task of the ETTh2 dataset.
Figure 41: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-2880 task of the ETTh2 dataset.
Figure 42: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-168 task of the Weather dataset.
Figure 43: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 168-336 task of the Weather dataset.
Figure 44: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-336 task of the Weather dataset.
Figure 45: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 336-720 task of the Weather dataset.
Figure 46: Forecasting cases comparison of WITRAN, FiLM and Pyraformer on the 720-720 task of the Weather dataset.
Figure 47: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 720-1440 task of the Weather dataset.
Figure 48: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-1440 task of the Weather dataset.
Figure 49: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet on the 1440-2880 task of the Weather dataset.
Algorithm 3 Recurrent Acceleration Network (RAN)
import torch   
import torch.nn as nn   
import math   
import torch.nn.Functional as F   
class WITRAN_HVGSU Encoder(torch.nnModule): def __init__(self, input_size, hidden_size, num_layers, dropout, water_rows, water cols, res_mode='none'): super(WITRAN_HVGSU Encoder, self).__init__() self.output_size = input_size self.hidden_size = hidden_size self(num_layers = num_layers self.dropout = dropout self.water_rows = water_rows self.water cols = water cols self.res_mode = res_mode # parameter of row cell self.W_first_layer = torch(nn.Parameters(torch.empty(6 * hidden_size, input_size + 2 * hidden_size)) self.W_other_layer = torch(nn.Parameters(torch.empty(num_layers - 1, 6 * hidden_size, 4 * hidden_size)) self.B = torch(nn.Parameters(torch.empty(num_layers, 6 * hidden_size)) self.reset_parameters()   
def reset_parameters(self): stdv = 1.0 / math.sqrt(self-hidden_size) for weight in self.params(): weight.data.uniform_(-stdv, +stdv)   
def linear(self, input, weight, bias, batch_size, slice, Water2sea Slice_num): a = F(linear(input, weight) if slice < Water2sea Slice_num: a[:batch_size * (slice + 1), :] = a[:batch_size * (slice + 1), :] + bias return a   
def forward(self, input, batch_size, input_size, flag): if flag == 1: # cols > rows input = input.permute(2, 0, 1, 3) else: input = input.permute(1, 0, 2, 3) Water2sea Slice_num, _, Original Slice_len, _ = input.shape Water2sea Slice_len = Water2sea Slice_num + Original Slice_len - 1 hidden Slice_row = torch.zeros(Water2sea Slice_num * batch_size, self-hidden_size).to(input device) hidden Slice_col = torch.zeros(Water2sea Slice_num * batch_size, self-hidden_size).to(inputdevice) input_transfer = torch.zeros(Water2sea Slice_num, batch_size, Water2sea Slice_len, input_size).to( input_device) for r in range(Water2sea Slice_num): input_transfer[r, :, r:r+Original Slice_len, :, ] = input[r, :, :, :] hidden_row_all_list = [] hidden_col_all_list = []