SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The first category captures the correlation between different time steps through point-wise attention mechanisms. Vanilla Transformer [Vaswani et al., 2017] is able to effectively extract correlations between any two-time steps through its self-attention mechanism, but its complexity is too high, reaching  $\mathcal{O}(L^2)$ . Informer [Zhou et al., 2021] proposes a ProbSparse self-attention with distilling techniques, reducing the complexity to  $\mathcal{O}(L\log L)$ . FEDformer [Zhou et al., 2022a] takes a different approach by applying attention modules in the frequency domain through Fourier or wavelet transformations, achieving a complexity of  $\mathcal{O}(L)$ . Despite the performance and efficiency improvements achieved by these representative methods, they face difficulties in extracting sufficient semantic information from a single time point, which has been raised in previous works [Nie et al., 2023, Wu et al., 2023].
The second category adopts non-dot-product techniques to extract correlation information. Log-Trans [Li et al., 2019] effectively captures local information by leveraging LogSparse and convolutional self-attention layers. However, it overlooks the long-term repetitive pattern of points among subseries and exhibits a high complexity of  $\mathcal{O}(L\log L)$ . Drawing from the traditional ideas of time-series analysis, Autoformer [Wu et al., 2021] is able to capture long-range trends and short-range changes in sequences through decomposition and auto-correlation mechanisms. However, its complexity remains high at  $\mathcal{O}(L\log L)$ . Pyraformer [Liu et al., 2021] initializes coarser-scale node information in the pyramidal graph by capturing local and global correlations through convolution. It then employs pyramidal attention to effectively capture long- and short-term repetitive patterns, leading to significant performance improvements while reducing complexity to  $\mathcal{O}(L)$ . However, Pyraformer still suffers from the limitation of the receptive field of the convolution kernel as discussed earlier. PatchTST [Nie et al., 2023] captures local semantic information through patch, and further reduces the complexity to  $\mathcal{O}((L / S)^2)$ , where  $S$  represents the stride length. Nevertheless, it does not consider the long-term repeated pattern among subseries.
Based on the above analysis, we find that most Transformer-based methods struggle to simultaneously effectively model local and global correlations (global-local semantic information) and capture long- and short-term repetitive patterns (periodic semantic information) in sequences. Furthermore, it is worth noting that the majority of the aforementioned methods fail to achieve lower complexity than RNN-based methods. For a comprehensive comparison of the theoretical time and memory complexity, as well as experimental results, please refer to Appendix C.
B Inspiration and Our Approach
RNNs, such as LSTM and GRU, have a significant advantage in capturing semantic information due to their recurrent structure, as discussed in Appendix A.1. Yet RNNs still have some drawbacks. In addition to the well-known issues of gradient vanishing/exploding [Pascanu et al., 2013]. This paper focuses on analyzing the information forgetting problem (Appendix B.2) and the difficulty in parallelizing RNN (Appendix C). In this section, we will analyze RNNs using LSTM and GRU as representatives, and introduce the inspiration we draw from during the process of designing our model.
B.1 LSTM VS GRU
LSTM incorporates three gates (input gate, output gate, and forget gate), while GRU utilizes two gates (reset gate and update gate). The complex gating mechanism of LSTM enables it to capture and learn intricate patterns, making it effective for processing longer sequences. On the other hand, GRU
performs well on shorter sequences and requires less computational and memory resources compared to LSTM. It should be noted that, as shown in Equation 7, the computation of  $\widetilde{h}_t$  in GRU is dependent on the completion of  $r_t$  computation. This undoubtedly slows down the computation that involves the input information  $x$ . Therefore, in terms of actual running time, GRU may not necessarily have an advantage over LSTM, as detailed in Appendix C.2.
B.2 The information forgetting problem of RNNs
When Recurrent Neural Networks (RNNs) process sufficiently long sequences, the information contained in the earlier parts of the sequence may be forgotten within the recurrent structure as it passes through multiple cells. To verify this phenomenon, we conducted a concrete experiment. We conducted a benchmark experiment using the ECL dataset, with a historical sequence length of 720 and a prediction horizon of 720, and the description of the ECL dataset can be found in Appendix E.
To eliminate the influence caused by the split of training, validation and test sets, we partitioned the dataset with a fixed input sequence length of 720 and a prediction horizon of 720. Furthermore, we conducted experiments with varying input sequence lengths using LSTM and GRU models. For the experiments, we used the last  $m = \{24, 48, 72, \dots, 696, 720\}$  points near the prediction time as input, and the results of these experiments are shown in Figure 7.
Figure 7: The issue of information forgetting in LSTM and GRU.
Observing Figure 7, on the one hand, we can see that when the input sequence length is relatively short (not within the red box), the prediction performance of LSTM and GRU gradually improves as sequence length increases. This indicates that longer input sequences can provide more contextual information, which can enhance the model's prediction performance and is consistent with previous research findings [Liu et al., 2021, Zeng et al., 2023]. On the other hand, the part within the red box indicates that when longer historical sequences are fed into LSTM and GRU, their prediction performance does not continue to improve. This suggests that introducing earlier information does not provide significant benefits for the models, which in turn confirms the problem of information forgetting in RNNs.
B.3 Inspiration and WITRAN
Our proposed Water-wave Information Transmission (WIT) framework utilizes input rearrangement to capture both long- and short-term repeating patterns in the sequence more directly and efficiently, while avoiding the processing of excessively long sequences, as described in Section 3.1. This successfully addresses the limitations of RNNs, such as the problems of exploding and vanishing gradients, as well as information forgetting. Furthermore, through the design of Recurrent Acceleration Network (RAN), we enable parallel computation of the entire sequence, as described in Section 3.3. For a comprehensive understanding of RAN, we provide detailed code and analysis of RAN in Appendix D. In this subsection, we focus on introducing the inspiration behind the important components of WIT.
The specific structure of WIT has been detailed in Section 3.2. The inspiration behind the design of Horizontal Vertical Gated Selective Unit (HVGSU) and Gated Selective Cell (GSC) comes from the advantages of LSTM and GRU. HVGSU serves two major purposes: 1) capturing both long- and short-term periodic semantic information in the sequence, and 2) modeling these two types
of periodic semantic information simultaneously for each time step. To achieve these objectives, two separate GSCs are established in the HVGSU, each responsible for the fusion and selection of long-term and short-term information in both directions.
For each GSC cell, to selectively incorporate fused information into the periodic semantic information in each direction, we draw inspiration from the reset gate of GRU and design a selection gate to control the retention of original information and the addition of new information. Furthermore, we are inspired by the output gate of LSTM and design our output gate to output the updated information. To reduce computational costs while maintaining sufficient learning capacity, we draw inspiration from GRU and only utilize these two gates, resulting in smaller computational and memory overheads. However, we address the problem of waiting for information to be computed between gate calculations in GRU by parallelizing the gate calculation vectors with the input  $x$ , which greatly accelerated the computation speed of the cell. In summary, we analyze and draw inspiration from the strengths of LSTM and GRU, and incorporated them into the design of our GSC.
B.4 WITRAN VS other RNN-based models
There are also several other RNN models, such as DilatedRNN [Chang et al., 2017], SlicedRNN [Yu and Liu, 2018], and RIMP-LSTM [Shen et al., 2018], which also have certain advantages in handling sequences. To further compare the differences between WITRAN and these methods, we provide the comparison in Table 6:
Table 6: Advantages of WITRAN compared to other RNN methods.
Advantages	DilatedRNN	SlicedRNN	RIMP-LSTM	WITRAN (ours)
Efficiently (1 layer) model global correlations	✓	✓	✓(1)	✓(1)
Special design to capture long-term repetitive patterns	✓	✗	✗	✓
Using 1 layer to capture long- and short-term repetitive patterns simultaneously	✗	✗	✗	✓
Well solve the gradient vanishing/exploding problem of RNN	✓	✓	✓	✓
(1) Efficiently (1 layer) model global correlations (a) When the dilations of DilatedRNN does not include the value 1, multiple layers need to be constructed to extract global correlations. (b) SlicedRNN improves efficiency to some extent by parallel processing of minimum subsequences, but it still requires the introduction of multiple layers to capture the global correlations of the sequence.
(2) Special design to capture long-term repetitive patterns (a) SlicedRNN is unable to capture long-term repetitive patterns among elements of sub-sequences. (b) Although RIMP-LSTM incorporates Residual Paths and Residual Sum Unit designs, it still cannot effectively extract long-term repetitive patterns.
(3) Using 1 layer to capture long- and short-term repetitive patterns simultaneously (a) DilatedRNN can capture long- and short-term repetitive patterns, but it requires the use of multiple layers to achieve this. (b) SlicedRNN and RIMP-LSTM are not particularly adept at handling long-term repetitive patterns, as mentioned in (2).
(4) Well solve the gradient vanishing/exploding problem of RNN (a) In DilatedRNN [Chang et al., 2017], it states and provides formal proof that reducing the length of information paths between time steps can prevent the issues of gradient vanishing/exploding. DilatedRNN, SlicedRNN, and WITRAN tackle this problem by reducing the length of information transmission paths. (b) RIMP-LSTM addresses this issue by the designs of Residual Paths and Residual Sum Units.
C Time and Memory Consumption of RNNs and Transformer
In this section, we derive the complexities of LSTM, GRU and Vanilla Transformer respectively, and analyze their actual running speeds and memory usage through experiments.
C.1 The theoretical computational complexity
The structure diagrams of LSTM and GRU are shown in Figure 6, while the structure of Vanilla Transformer can be found in paper [Vaswani et al., 2017]. Assuming the sequence length is  $L$ , the
input dimension is  $d_{\mathrm{in}}$  and the hidden layer dimension is  $d_{\mathrm{model}}$ , and considering the case where each model has only one layer, the derivation process is as follows:
GRU GRU includes two gates and three matrix multiplication operations, as mentioned in Appendix A.1. Each matrix multiplication operation has a complexity of  $d_{\mathrm{model}} \times (d_{\mathrm{model}} + d_{\mathrm{in}})$ , although there are also  $\odot$  operations in GRU, and the complexity of element-wise multiplication is  $d_{\mathrm{model}}$ , which is much smaller than the complexity of matrix multiplication operation. Therefore, the complexity of one GRU unit can be simplified as  $3 \times d_{\mathrm{model}} \times (d_{\mathrm{model}} + d_{\mathrm{in}})$ . When GRU processes the entire sequence, it needs to perform  $L$  sequential computations, therefore, the overall computational complexity can be summarized as:  $3 \times L \times d_{\mathrm{model}} \times (d_{\mathrm{model}} + d_{\mathrm{in}})$ .
LSTM LSTM includes three gates and four matrix multiplication operations, as mentioned in Appendix A.1. Similar to GRU, the computational complexity of LSTM can also be summarized as:  $4 \times L \times d_{\mathrm{model}} \times (d_{\mathrm{model}} + d_{\mathrm{in}})$ .
Summary In time series prediction tasks, the input dimension  $d_{\mathrm{in}}$  is usually much smaller than the model dimension  $d_{\mathrm{model}}$ . Based on the analysis conducted earlier, we can observe that the computational complexity of the encoder part of most Transformer-based methods is generally higher than that of LSTM and GRU. This is because Transformer includes many complex operations such as matrix multiplication and linear transformation. Although there have been improvements made to the attention mechanism, these modifications only affect a small portion of the overall complexity which remains high. Therefore, compared to LSTM and GRU, the computational complexity of Transformer is higher.
C.2 The practical time and memory consumption
In practical situations, LSTM can perform computations in parallel without the need to wait for information from all four matrix operations. On the other hand, in GRU, the first two matrix operations can be parallelized, but the third matrix operation has a dependency on the completion of the previous operations, as mentioned in Appendix A.1. Therefore, the actual runtime of GRU may be longer than
Algorithm 1 GRU based on Python
import torch   
import torch(nn as nn   
import torch(nn.Functional as F   
class Manual_GRU(nnModule): def __init__(self, input_size, hidden_size, dropout): super(Manual_GRU, self).__init_(self-hidden_size = hidden_size self_dropout = dropout self.gates  $=$  nn.Linear(input_size+hidden_size, hidden_size\*2) self-hidden_transform  $\equiv$  nn.Linear(input_size+hidden_size, hidden_size) self.sigmoid  $\equiv$  nn.Sigmoid() self.tanh  $=$  nn.Tanh() for param in self.params(): if param.dim  $\mathbf{\rho} > 1$  nn.init.xavier.uniform_(param)   
def forward(self,x): batch_size  $\equiv$  x.size(0) seq_len  $\equiv$  x.size(1) h  $=$  torch.zeros(batch_size,self-hidden_size).to(x_device) y_list  $= []$  for i in range(seq_len): update_gate, reset_gate  $=$  self.gates(torch.cat([x[;, i, :], h], dim=-1)).chunk(2,-1) update_gate, reset_gate  $=$  (self.sigmoid(gate) for gate in (update_gate, reset_gate)) candidate_hidden  $=$  self.tanh(self-hidden_transform(torch.cat([x[;, i, :], reset_gate  $\ast$  h], dim=-1))  $\mathrm{h} = (1$  -update_gate  $\ast$  h  $^+$  update_gate  $\ast$  candidate_hidden y_list.append(h) output  $=$  F_dropouttorch.stack(y_list, dim=1), self_dropout, self.trainig) return output, h
that of LSTM. To verify this, we compared the processing time and memory usage of a single-layer LSTM and a single-layer GRU on sequences of the same length using Python implementation, as shown in Figure 8.
Figure 8: Time and memory consumption of RNNs and Transformer.
Transformer has the advantage of parallelizing the computation of correlations for a sequence of length  $L$ , which greatly improves its speed compared to GRU/LSTM. It is important to note that this comparison is not entirely fair, as Transformer can parallelize at the sequence level, whereas RNNs can parallelize at the batch level. Each approach has its own strengths and is suited for different scenarios.
To ensure a fair comparison of runtime and memory usage, it is important to consider the conditions under which the models are compared. Following the experimental setup in FiLM [Zhou et al., 2022b], we can ensure that the hidden state dimension and batch size of the models being compared are the same. This allows us to evaluate whether the models can be trained on devices with low memory capacity. For time evaluation, we need to ensure that the hidden state dimension is the same and the memory utilization is similar, so that memory resources are not wasted. In terms of time comparison between Transformer and GRU/LSTM, although GRU/LSTM are slower than
Transformer under the same batch conditions in Python implementation, they can still speed up their parallel processing by increasing batch size while ensuring similar memory usage to Transformer. At the same time, because the computation process of GRU and LSTM is much simpler than that of Transformer, they can be more easily implemented in  $\mathrm{C}++$  and integrated into PyTorch's library as nn.GRU/nn.LSTM, which is faster. For comparison, we compared the memory and time costs of these five methods, as shown in Figure 8. The Python implementation of GRU and LSTM code is shown in Algorithm 1 and Algorithm 2.
Algorithm 2 LSTM based on Python
import torch   
import torch.nn as nn   
import torch.nn.Functional as F   
class Manual_LSTM(nnModule): def __init__(self, input_size, hidden_size, dropout): super(Manual_LSTM, self).__init_(self.hidden_size = hidden_size self.dropout  $=$  dropout self.gates  $\equiv$  nn.Linear(input_size+hidden_size, hidden_size  $\ast 4$  ) self.sigmoid  $\equiv$  nn.Sigmoid() self.tanh  $=$  nn.Tanh() for param in self.params(): if param.dim  $\rightharpoondown$  1: nn.init.xavier.uniform_(param)   
def forward(self,x): batch_size  $\equiv$  x.size(0) seq_len  $\equiv$  x.size(1) h,c  $\equiv$  (torch.zeros(batch_size,self-hidden_size).to(x_device) for_in range(2)) y_list  $= []$  for i in range(seq_len): forget_GATE, input_GATE, output_GATE, candidate_cell  $\equiv$  self.gates(torch.cat([x[:i:,h],dim=-1)) .chunk(4,-1) forget_GATE, input_GATE, output_GATE  $\equiv$  (self.sigmoid(g) for g in (forget_GATE, input_GATE, output_GATE)) c  $\equiv$  forget_GATE  $\ast$  c + input_GATE  $\ast$  self.tanhCandidate_cell) h  $\equiv$  output_GATE  $\ast$  self.tanh(c) y_list.append(h) output  $\equiv$  F.dropout(torch.stack(y_list, dim=1), self.dropout, self.trainig) return output,h,c
D The Design Process and Implementation Code of Recurrent Acceleration Network (RAN)
In this section, we will provide a detailed explanation of the implementation approach (Appendix D.1) and specific code of RAN (Appendix D.2).
D.1 Implementation approach of RAN
In Section 3.3, we introduced the Recurrent Acceleration Network (RAN) as part of our overall approach. However, we encountered a challenge when the number of rows processed by RAN is smaller than the number of columns. In such cases, if the number of slices processed exceeds the number of rows, the intermediate variables of the slices become fully occupied. To ensure the continuity of operations, it becomes necessary to continuously shift and reset these variables. However, we found that this reset operation was time-consuming and hindered the expected acceleration benefits of RAN. To overcome this issue, we made a simple adjustment by setting the size of the slice to the maximum value of the rows and columns. While this may introduce additional space complexity, it maintains the length of the slice and eliminates the need for a reset operation. As a result, the
processing time is significantly reduced. The comparison diagram between RAN-min and RAN-max is shown in Figure 9.
(a) RAN-min
(b) RAN-max (in WITRAN)
Figure 9: The figure shows two designs of RANs side by side. The left one represents the case where the number of slices is the minimum value between the number of rows and columns, while the right one represents the case where the number of slices is the maximum value between the number of rows and columns.
D.2 Specific code of RAN
The specific code of RAN can be found in Algorithm 3. Due to its long content, we place it at the end of the Appendix.
In this section, we will provide a detailed introduction of the datasets used in this paper. (1) Electricity $^2$  (ECL) contains the hourly electricity consumption of 321 customers from 2012 to 2014. (2) ETT [Zhou et al., 2021] contains the load and oil temperature, which was recorded every 15 minutes between July 2016 and July 2018 from electricity transformers. (3) Traffic $^3$  contains the hourly data the road occupancy rates measured by different sensors on San Francisco Bay area freeways, collected from California Department of Transportation. (4) Weather $^4$  contains 21 meteorological indicators (such as air temperature, humidity, etc.) and was recorded every 10 minutes for 2020 whole year. They are all split into the training set, validation set and test set by the ratio of 6:2:2 during modeling.
Due to the different original acquisition granularities of each dataset, in order to ensure that they contain the same semantic information on the same task, in this paper, our experiments are conducted by aggregating them into one hour. Table 7 summarizes all the features of the four benchmark datasets. The target value of ETT is 'oil temperature (OT)', the target value of ECL is 'MT_320', the target value of Traffic is 'Node_862' and the target value of Weather is 'wet_bulb'.
Table 7: The details of datasets.
Dataset	Sample Length	Dimension	Usage Frequency	Recorded Frequency
ETT	17420	8	1h	15min
ECL	26304	322	1h	1h
Traffic	17544	863	1h	1h
Weather	35064	22	1h	10min
<sup>2</sup>https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
<sup>3</sup>http://pems.dot.ca.gov
4 https://www.bgc-jena.mpg.de/wetter/
F Experiment Setup Details
Our method and all the baseline methods are trained with the L2 loss, using the ADAM [Kingma and Ba, 2014] optimizer with an initial learning rate of 10-3. Batch size is set to 32. The training process is early stopped after 5 epochs if there is no loss degradation on the validation set and the max epochs is 25. We save all the models with the lowest loss on the validation set for the final testing.
To ensure a fair comparison of each model's performance, we set the same search space for the common parameters included in each model. This ensures that each model can achieve its best performance under the same conditions for comparison, with the seed set to 2023.
Regarding the private parameters of each model, we set the corresponding search space according to the specific descriptions given in their respective papers. In addition, it should be emphasized that the performance of Pyraformer shown in this paper is the best between the two prediction designs of the model, the performance of FEDformer shown here is the best between FEDformer-w and FEDformer-f, the performance of DLinear shown here is the best among DLinear, NLinear and Linear, and the performance of MICN is the best between MICN-regre and MICN-mean.
The mean square error (MSE) and mean absolute error (MAE) are used as metrics. All experiments are repeated 5 times and the mean of the metrics is reported as the final results in Table 3 and Table 4.
All the deep learning networks are implemented using PyTorch [Paszke et al., 2019] and conducted on NVIDIA RTX A4000 16GB GPUs.
The search space for the common parameters is as follows: (1)  $d_{\mathrm{model}}$  is set to {32, 64, 128, 256, 512, 1024}. (2)  $e_{\mathrm{layer}}$  and  $d_{\mathrm{layer}}$  are set to {1, 2, 3}. (3)  $n_{\mathrm{head}}$  is set to {1, 2, 4, 8} and  $d_{\mathrm{ff}}$  is four times as large as  $d_{\mathrm{model}}$ . Regarding the private parameters of our model, norm is set to {0, 1} and  $C$  represents the period is set to {12, 24, 48} and it is required that the length  $H$  of the input sequence is divisible by  $C$ . Furthermore, it is important to note that adjustments should be made to  $C$  for different tasks to ensure its ability to adequately represent the period.