SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxin Jia Youfang Lin Xinyan Hao Yan Lin Shengnan Guo Huaiyu Wan*
School of Computer and Information Technology, Beijing Jiaotong University, China
Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing, China {yuxinjia, yflin, xinyanhao, ylincs, guoshn, hywan}@bjtu.edu.cn
Capturing semantic information is crucial for accurate long-range time series forecasting, which involves modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. Previous works have partially addressed these issues separately, but have not been able to address all of them simultaneously. Meanwhile, their time and memory complexities are still not sufficiently low for long-range forecasting. To address the challenge of capturing different types of semantic information, we propose a novel Water-wave Information Transmission (WIT) framework. This framework captures both long- and short-term repetitive patterns through bi-granular information transmission. It also models global and local correlations by recursively fusing and selecting information using Horizontal Vertical Gated Selective Unit (HVGSU). In addition, to improve the computing efficiency, we propose a generic Recurrent Acceleration Network (RAN) which reduces the time complexity to  $\mathcal{O}(\sqrt{L})$  while maintaining the memory complexity at  $\mathcal{O}(L)$ . Our proposed method, called Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN), outperforms the state-of-the-art methods by  $5.80\%$  and  $14.28\%$  on long-range and ultra-long-range time series forecasting tasks respectively, as demonstrated by experiments on four benchmark datasets. The code is available at: https://github.com/Water2sea/WITRAN.
Time series forecasting is a valuable tool across diverse fields, such as energy, traffic, weather and so on. Compared with short-range time series forecasting, long-range forecasting offers the advantage of providing individuals with ample time to prepare and make informed decisions. For instance, accurate long-range traffic and weather forecasts enable individuals to plan their travel arrangements and attire accordingly. To improve the accuracy of such forecasting, it is essential to utilize longer historical sequences as input for the forecasting models [Liu et al., 2021, Zeng et al., 2023].
Previous studies [Wu et al., 2023, Nie et al., 2023] have highlighted the importance of capturing semantic information in long-range time series to achieve accurate predictions. However, the semantic information in long-range time series is diverse, so how to analyze and capture it becomes a major challenge. Specifically, semantic information includes two main aspects: (1) Global and local correlations. The local semantic information usually contains short-range changes within the data, while the global semantic information reflects the long-range trends present in the time series [Wang et al., 2023], which can be referred to as global-local semantic information. (2) Long- and short-term repetitive patterns. Time series often exhibit repetitive patterns at different timescales [Lai
et al., 2018, Liu et al., 2021], such as hourly or daily cycles, which can be identified as periodic semantic information. Furthermore, it is crucial to model both aspects of semantic information simultaneously and ensure that the modeling approach remains computationally efficient, avoiding excessive complexity.
Unfortunately, although the existing state-of-the-art methods have shown great performance, they encounter difficulties in addressing the aforementioned challenges simultaneously. Specifically, CNN-based methods [Bai et al., 2018, Franceschi et al., 2019, Sen et al., 2019, Wu et al., 2023, Wang et al., 2023] have linear complexity with respect to the sequence length  $L$ , but they are either limited by the size of the convolution receptive field [Wang et al., 2023] or constrained by the 1D input sequence [Wu et al., 2023], making it difficult to capture both important semantic information simultaneously. Transformer-based methods can be broadly classified into two categories based on whether the point-wise attention is used or not. The used ones [Vaswani et al., 2017, Zhou et al., 2021, 2022a] capture correlations between points in the sequence, yet face challenges in capturing hidden semantic information directly from point-wise input tokens [Nie et al., 2023, Wu et al., 2023]. The others [Li et al., 2019, Wu et al., 2021, Liu et al., 2021] struggles either with achieving sufficiently low computational complexity or with effectively capturing periodic semantic information. Other methods [Zhou et al., 2022b, Zeng et al., 2023] also exhibit limitations in capturing semantic information mentioned above. Further details can be found in Section 2.
RNN-based methods [Hochreiter and Schmidhuber, 1997, Chung et al., 2014, Rangapuram et al., 2018, Salinas et al., 2020] have significant advantages in capturing global and local semantic information through their recurrent structure, as shown in Figure 1(a), while maintaining linear complexity. However, they suffer from gradient vanishing/exploding [Pascanu et al., 2013] and information forgetting issues (refer to Appendix B for more details), making them less suitable for direct application in long-range forecasting tasks. Fortunately, it has been proven that by splitting the information transmissions into patches between a few adjacent time steps and processing them individually, it is possible to maintain both global and local semantic information [Nie et al., 2023, Wu et al., 2023]. This insight is highly inspiring, as it suggests that by dividing the inputs of RNNs into numerous subseries and processing them separately, we can effectively address the significant limitation mentioned earlier, without compromising the efficiency of long-range forecasting.
Figure 1: Information transmission process diagram of different forecasting models.
Based on the aforementioned insights, we propose a novel framework called Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN) which comprises two key components: The Water-wave Information Transmission (WIT) framework and the Recurrent Acceleration Network (RAN). The overall information transmission process of WITRAN is illustrated in Figure 1(i). Firstly, to capture the periodic semantic information of long- and short-term, we rearrange the input sequences according to their natural period, as shown in Figure 2(a). This rearrangement allows for information transmission in two directions, resembling the propagation of water-wave energy, as shown in Figure 2(b). The horizontal red arrows indicate the intra-periodic information transmission along the time steps, while the vertical blue arrows indicate the inter-periodic information transmission. Secondly, to preserve the characteristics of long- and short-term periodic semantic information, we propose a novel Horizontal Vertical Gated Selective Unit (HVGSU) which incorporates Gated Selective Cells (GSCs) separately in both directions. To capture the correlation between periodic semantic information of both directions at each time step, we design fusion and selection operations in GSC. With a recurrent structure, HVGSU progressively captures more local semantic information until it encompasses the global semantic information. Thirdly, to improve efficiency, we propose the Recurrent Acceleration Network (RAN), which enables parallel processing of information transmission in both directions. Notably, RAN maintains a memory complexity of  $\mathcal{O}(L)$  while reducing the time complexity to  $\mathcal{O}(\sqrt{L})$ . In addition, RAN could serve as a universal framework for integrating other models to facilitate information fusion and transmission. In summary, our main contributions are as follows:
- We propose a Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN), which represents a novel paradigm in information transmission by enabling bi-granular flows. We provide a comprehensive comparison of WITRAN with previous methods in Figure 1 to highlight its uniqueness. Furthermore, in order to compare the differences between WITRAN and the model (a)-(h) in Figure 1 more clearly, we have prepared Table 1 to highlight the advantages of WITRAN.
- We propose a novel Horizontal Vertical Gated Selective Unit (HVGSU) which captures long- and short-term periodic semantic information by using Gated Selective Cell (GSC) independently in both directions, preserving the characteristics of periodic semantic information. The fusion and selection in GSC can model the correlations of long- and short-term periodic semantic information. Furthermore, utilizing a recurrent structure with HVGSU facilitates the gradual capture of semantic information from local to global within a sequence.
- We present a Recurrent Acceleration Network (RAN) which is a generic acceleration framework that significantly reduces the time complexity to  $\mathcal{O}(\sqrt{L})$  while maintaining the memory complexity of  $\mathcal{O}(L)$ . We summarize the complexities of different methods in Table 2, demonstrating the superior efficiency of our method.
- We conduct extensive experiments on four benchmark datasets across various fields (energy, traffic and weather). The empirical studies demonstrate the remarkable performance of WITRAN, which achieves relative improvements of  $5.80\%$  and  $14.28\%$  in long-range and ultra-long-range forecasting respectively. In addition, the introduction of the generic RAN framework greatly improves the computational efficiency.
Table 1: Advantages of WITRAN compared to other methods.
Advantages	(a) RNN	(b) CNN	(c) Full Attention	(d) LogTrans	(e) Pyraformer	(f) MICN	(g) PatchTST	(h) TimesNet	(i) WITRAN (ours)
Non point-wise semantic information capture	✓	✓	✗	✓	✓	✓	✓	✓	✓
Special design to capture long-term repetitive patterns	✗	✗	✗	✗	✓	✗	✗	✓	✓
Efficiently (1 or 2 layers) model global correlations	✓(1)	✗	✓(1)	✗	✗	✓(2)	✓(2)	✗	✓(1)
Well solve the gradient vanishing/exploiting problem of RNN	✗	-	-	-	-	-	-	-	✓
Table 2: Complexity of forecasting models in training.  $L$  is the sequence length, and  $S$  is the stride in the PatchTST model.
Methods	RNN	CNN	Transformer	LogTrans	Informer	Autoformer	Pyraformer	FEDformer	FiLM	PatchTST	MICN	WITRAN (ours)
Time	O(L)	O(L)	O(L2)	O(L log L)	O(L log L)	O(L log L)	O(L)	O(L)	O(L)	O((L/S)2)	O(L)	O(√L)
Memory	O(L)	O(L)	O(L2)	O(L2)	O(L log L)	O(L log L)	O(L)	O(L)	O(L)	O((L/S)2)	O(L)	O(L)
Figure 2: Input rearrangement and water-wave information transmission.
Time series forecasting methods can be broadly categorized into statistical methods and neural network methods. Neural network methods, in turn, can be further classified into several subcategories based on the specific techniques employed, including RNN-based, CNN-based, Transformer-based, and other Deep Neural Networks (DNNs) by the methods used.
Statistical Methods with major representatives include ARIMA [Box and Jenkins, 1968], Prophet [Taylor and Letham, 2018], and Holt Winter [Athenasopoulos and Hyndman, 2020]. However, the inherent variability and complexity of real-world time series often render these methods inadequate, as they are built on hypotheses that may not align well with the characteristics of such data. As a result, the performance of these statistical methods tends to be limited.
RNN-based Methods [Hochreiter and Schmidhuber, 1997, Chung et al., 2014, Rangapuram et al., 2018, Salinas et al., 2020] possess the inherent advantage of capturing semantic information in time series data through their recurrent structure, which mimics the natural information transmission process of time series. This allows them to progressively capture semantic information from local to global contexts. These methods exhibit linear time and memory complexity, enabling efficient processing of time series data with a sequence length of  $L$ . However, when dealing with longer sequences, the problems of gradient vanishing/exploding [Pascanu et al., 2013] and information forgetting (see Appendix B for more details) further limit it.
CNN-based methods [Bai et al., 2018, Franceschi et al., 2019, Sen et al., 2019, Wu et al., 2023, Wang et al., 2023] are adept at capturing local semantic information through the application of convolutional kernels, while maintaining linear time and memory complexity. However, on the one hand, most of these methods face challenges in capturing comprehensive global information due to the limited receptive field of individual convolutional layers, which make the training process more difficult and overhead [Wang et al., 2023]. On the other hand, it is difficult to directly capture long- and short-term repetitive patterns on 1D inputs. MICN [Wang et al., 2023] adopted downsampling 1D convolutions and isometric convolutions combined with a multi-branch framework, which can effectively solve the former problem, but still suffers from the latter one. TimesNet [Wu et al., 2023] transforms 1D inputs into 2D space, leveraging a parameter-efficient inception block to capture intra- and inter-period variations. However, it solves the latter problem while still facing the former one.
Transformer-based methods have shown advancements in time series forecasting, with two main categories. The first category uses point-wise attention mechanisms, such as Vanilla Transformer [Vaswani et al., 2017], Informer [Zhou et al., 2021], and FEDformer [Zhou et al., 2022a], while facing challenges in extracting sufficient semantic information from individual time points. The second category employs non point-wise dot product techniques, including LogTrans [Li et al., 2019], Autoformer [Wu et al., 2021], Pyraformer [Liu et al., 2021] and PatchTST [Nie et al., 2023]. They reduce computational complexities to a certain degree, yet face difficulties in incorporating the long-term correlations in time-series. For a detailed description of these methods, please refer to Appendix A. Furthermore, it is worth noting that the majority of the aforementioned methods fail to achieve lower complexity than RNN-based methods. For a comprehensive comparison of the theoretical time and memory complexity, as well as experimental results, please refer to Appendix C.
Other DNN methods have also demonstrated promising performance. For example, FiLM [Zhou et al., 2022b] utilizes Legendre Polynomials and Fourier projection methods to capture historical in
formation, eliminate noise, and expedite computations using low-rank approximation. DLinear [Zeng et al., 2023] has showcased impressive outcomes by employing simple linear operations. Nevertheless, both approaches encounter difficulties in capturing various repeating patterns present in the sequence.
3 The WITRAN Model
The time series forecasting task involves predicting future values  $Y \in \mathbb{R}^{P \times c_{\mathrm{out}}}$  for  $P$  time steps based on the historical input sequences  $X = \{x_1, x_2, \ldots, x_H\} \in \mathbb{R}^{H \times c_{\mathrm{in}}}$  of  $H$  time steps, where  $c_{\mathrm{in}}$  and  $c_{\mathrm{out}}$  represent the number of input and output features respectively. In order to integrate enough historical information for analysis, it is necessary for  $H$  to have a sufficient size [Liu et al., 2021, Zeng et al., 2023]. Furthermore, capturing semantic information from the historical input is crucial for accurate forecasting, which includes modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. However, how to address them simultaneously is a major challenge. With these in mind, we propose WITRAN, a novel information transmission framework akin to the propagation of water waves. WITRAN captures both long- and short-term periodic semantic information, as well as global-local semantic information simultaneously during information transmission. Moreover, WITRAN reduces time complexity while maintaining linear memory complexity. The overall structure of WITRAN is depicted in Figure 3.
Figure 3: Overall structure of WITRAN.
3.1 Input Module
To facilitate the analysis of long- and short-term repetitive patterns, inspired by TimesNet [Wu et al., 2023], we first rearrange the sequence from 1D to 2D based on its natural period, as illustrated by Figure 2(a). Importantly, our approach involves analyzing the natural period of time series and setting appropriate hyperparameters to determine the input rearrangement, rather than using Fast Fourier Transform (FFT) to learn multiple adaptive periods of inputs in TimesNet. Consequently, our method is much simpler. Additionally, in order to minimize the distribution shift in datasets, we draw inspiration from NLinear [Zeng et al., 2023] and employ an adaptive learning approach to determine whether to perform simple normalization. The input module can be described as follows:
$$
X _ {1 \mathrm {D}} = \left\{ \begin{array}{l l} X & , n o r m = 0 \\ X - x _ {H} & , n o r m = 1 \end{array} \right. \tag {1}
$$
$$
X _ {2 \mathrm {D}} = \text {R e a r r a n g e} ([ X _ {1 \mathrm {D}}, T F _ {\text {e n}} ]),
$$
here,  $X_{\mathrm{1D}} \in \mathbb{R}^{H \times c_{\mathrm{in}}}$  represents the original input sequences,  $x_H \in \mathbb{R}^{c_{\mathrm{in}}}$  represents the input at the last time step of the original sequence,  $TF_{\mathrm{en}} \in \mathbb{R}^{H \times c_{\mathrm{time}}}$  represents the temporal contextual features of original input sequence (e.g., HourOfDay, DayOfWeek, DayOfMonth and DayOfYear), where  $c_{\mathrm{time}}$  is the dimension of time features.  $X_{\mathrm{2D}} \in \mathbb{R}^{R \times C \times (c_{\mathrm{in}} + c_{\mathrm{time}})}$  represents the inputs after rearrangement, where  $R$  denotes the total number of horizontal rows and  $C$  denotes the vertical columns. norm is an adaptive parameter for different tasks. [·] represents the concat operation and Rearrange represents the rearrange operation, with reference to Figure 2(a).
3.2 Horizontal Vertical Gated Selective Unit
To capture long- and short-term periodic semantic information and reserve their characteristics, we propose a novel Horizontal Vertical Gated Selective Unit (HVGSU) which consists of Gated Selective Cells (GSCs) in two directions. To capture the correlation at each time step between periodic semantic information of both directions, we design the specific operations in GSC. Furthermore, HVGSU is capable of capturing both global and local information via a recurrent structure. In this subsection, we will provide a detailed introduction to them.
HVGSU As depicted in Figure 3, the process of HVGSU via a recurrent structure is:
$$
H _ {\text {h o r}}, H _ {\text {v e r}}, O u t = \operatorname {H V G S U} \left(X _ {2 \mathrm {D}}\right), \tag {2}
$$
where  $H_{\mathrm{hor}} \in \mathbb{R}^{L \times R \times d_{\mathrm{model}}}$  and  $H_{\mathrm{ver}} \in \mathbb{R}^{L \times C \times d_{\mathrm{model}}}$  represent the horizontal and the vertical output hidden state of HVGSU separately.  $L$  is the depth of the model, and  $Out \in \mathbb{R}^{R \times C \times d_{\mathrm{model}}}$  denotes the output information of the last layer.
In greater detail, the cellular structure of HVGSU is shown in Figure 4(b), which consists of GSCs in two directions to capture the periodic semantic information of long- and short-term. The cell operations for row  $r$  ( $1 \leq r \leq R$ ) and column  $c$  ( $1 \leq c \leq C$ ) in layer  $l$  ( $1 \leq l \leq L$ ) can be formalized as:
$$
h _ {r, c, l} ^ {\text {h o r}} = \operatorname {G S C} _ {\text {h o r}} (i n p u t _ {r, c, l}, h _ {r, c - 1, l} ^ {\text {h o r}}, h _ {r - 1, c, l} ^ {\text {v e r}})
$$
$$
h _ {r, c, l} ^ {\text {v e r}} = \operatorname {G S C} _ {\text {v e r}} \left(\text {i n p u t} _ {r, c, l}, h _ {r - 1, c, l} ^ {\text {v e r}}, h _ {r, c - 1, l} ^ {\text {h o r}}\right) \tag {3}
$$
$$
o _ {r, c, l} = \left[ h _ {r, c, l} ^ {\text {h o r}}, h _ {r, c, l} ^ {\text {v e r}} \right],
$$
here,  $input_{r,c,l} \in \mathbb{R}^{d_{\mathrm{in}}}$ . When  $l = 1$ ,  $input_{r,c,l} = x_{r,c} \in \mathbb{R}^{c_{\mathrm{in}} + c_{\mathrm{time}}}$  represents the input for the first layer, and when  $l > 1$ ,  $input_{r,c,l} = o_{r,c,l-1} \in \mathbb{R}^{2 \times d_{\mathrm{model}}}$  represents the input for subsequent layers.  $h_{r,c-1,l}^{\mathrm{hor}}$  and  $h_{r-1,c,l}^{\mathrm{ver}} \in \mathbb{R}^{d_{\mathrm{model}}}$  represent the horizontal and vertical hidden state inputs of the current cell. Note that when  $r = 1$ ,  $h_{r-1,c,l}^{\mathrm{ver}}$  is replaced by an all-zero tensor of the same size, and the same is true for  $h_{r,c-1,l}^{\mathrm{hor}}$  when  $c = 1$ . [·] represents the concatenation operation and  $o_{r,c,l} \in \mathbb{R}^{2 \times d_{\mathrm{model}}}$  represents the output of the current cell.
Figure 4: The structure of our method.
GSC Inspired by the two popular RNN-based models, LSTM [Hochreiter and Schmidhuber, 1997] and GRU [Chung et al., 2014] (for more details, see Appendix A), we propose a Gated Selective Cell (GSC) to fuse and select information. Its structure is shown in Figure 4(a), which comprises two gates: the selection gate, and the output gate. The fused information consists of input and principal-subordinate hidden states, and the selection gate determines the retention of the original principal information and the addition of the fused information. Finally, the output gate determines the final output information of the cell. The different colored arrows in Figure 4(a) represent different semantic information transfer processes. The black arrow represents the input, the red arrows represent the process of transmitting principal hidden state information, the blue arrow represents the subordinate hidden state, and the purple arrows represent the process by which fused information of
principal-subordinate hidden states is transmitted. The formulations are given as follows:
$$
S _ {t} = \sigma \left(W _ {s} \left[ h _ {t - 1} ^ {\text {p r i}}, h _ {t - 1} ^ {\text {s u b}}, x \right] + b _ {s}\right)
$$
$$
O _ {t} = \sigma \left(W _ {o} \left[ h _ {t - 1} ^ {\text {p r i}}, h _ {t - 1} ^ {\text {s u b}}, x \right] + b _ {o}\right)
$$
$$
h _ {f} = \tanh  \left(W _ {f} \left[ h _ {t - 1} ^ {\text {p r i}}, h _ {t - 1} ^ {\text {s u b}}, x \right] + b _ {f}\right) \tag {4}
$$
$$
\widetilde {h _ {t - 1} ^ {\mathrm {p r i}}} = \left(1 - S _ {t}\right) \odot h _ {t - 1} ^ {\mathrm {p r i}} + S _ {t} \odot h _ {f}
$$
$$
h _ {t} ^ {\text {p r i}} = \operatorname {t a n h} \left(\widetilde {h _ {t - 1} ^ {\text {p r i}}}\right) \odot O _ {t},
$$
where  $h_{t-1}^{\mathrm{pri}}$  and  $h_{t-1}^{\mathrm{sub}} \in \mathbb{R}^{d_{\mathrm{model}}}$  represent the input principal and subordinate hidden state,  $x \in \mathbb{R}^{d_{\mathrm{in}}}$  represents the input.  $W_{*} \in \mathbb{R}^{d_{\mathrm{model}} \times (2d_{\mathrm{model}} + d_{\mathrm{in}})}$  are weight matrices and  $b_{*} \in \mathbb{R}^{d_{\mathrm{model}}}$  are bias vectors.  $S_{t}$  and  $O_{t}$  denote the selection gate and the output gate,  $\odot$  denotes an element-wise product,  $\sigma(\cdot)$  and  $\tanh(\cdot)$  denote the sigmoid and tanh activation function.  $h_{f}$  and  $h_{t-1}^{\mathrm{pri}} \in \mathbb{R}^{d_{\mathrm{model}}}$  represent the intermediate variables of the calculation.  $h_{t}^{\mathrm{pri}}$  represents the output hidden state.
3.3 Recurrent Acceleration Network