Title: WITRAN: Water-wave Information Transmission and Recurrent Acceleration Network for Long-range Time Series Forecasting
Venue: NeurIPS 2023
Accepted: yes
Keywords: long-range time series forecasting

information transmission

long- and short-term repetitive patterns

global and local correlations
Abstract: Capturing semantic information is crucial for accurate long-range time series forecasting, which involves modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. Previous works have partially addressed these issues separately, but have not been able to address all of them simultaneously. Meanwhile, their time and memory complexities are still not sufficiently low for long-range forecasting. To address the challenge of capturing different types of semantic information, we propose a novel Water-wave Information Transmission (WIT) framework. This framework captures both long- and short-term repetitive patterns through bi-granular information transmission. It also models global and local correlations by recursively fusing and selecting information using Horizontal Vertical Gated Selective Unit (HVGSU). In addition, to improve the computing efficiency, we propose a generic Recurrent Acceleration Network (RAN) which reduces the time complexity to $\mathcal{O}(\sqrt{L})$ while maintaining the memory complexity at $\mathcal{O}(L)$. Our proposed method, called Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN), outperforms the state-of-the-art methods by 5.80% and 14.28% on long-range and ultra-long-range time series forecasting tasks respectively, as demonstrated by experiments on four benchmark datasets. The code is available at: https://github.com/Water2sea/WITRAN.

WITRAN: Water-wave Information Transmission and Recurrent Acceleration Network for Long-range Time Series Forecasting

Abstract

Capturing semantic information is crucial for accurate long-range time series forecasting, which involves modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. Previous works have partially addressed these issues separately, but have not been able to address all of them simultaneously. Meanwhile, their time and memory complexities are still not sufficiently low for long-range forecasting. To address the challenge of capturing different types of semantic information, we propose a novel Water-wave Information Transmission (WIT) framework. This framework captures both long- and short-term repetitive patterns through bi-granular information transmission. It also models global and local correlations by recursively fusing and selecting information using Horizontal Vertical Gated Selective Unit (HVGSU). In addition, to improve the computing efficiency, we propose a generic Recurrent Acceleration Network (RAN) which reduces the time complexity to O ( √ L ) while maintaining the memory complexity at O ( L ) . Our proposed method, called Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN), outperforms the state-of-the-art methods by 5.80% and 14.28% on long-range and ultra-long-range time series forecasting tasks respectively, as demonstrated by experiments on four benchmark datasets. The code is available at: https://github.com/Water2sea/WITRAN .

1 Introduction

Time series forecasting is a valuable tool across diverse fields, such as energy, traffic, weather and so on. Compared with short-range time series forecasting, long-range forecasting offers the advantage of providing individuals with ample time to prepare and make informed decisions. For instance, accurate long-range traffic and weather forecasts enable individuals to plan their travel arrangements and attire accordingly. To improve the accuracy of such forecasting, it is essential to utilize longer historical sequences as input for the forecasting models [Liu et al., 2021, Zeng et al., 2023].

Previous studies [Wu et al., 2023, Nie et al., 2023] have highlighted the importance of capturing semantic information in long-range time series to achieve accurate predictions. However, the semantic information in long-range time series is diverse, so how to analyze and capture it becomes a major challenge. Specifically, semantic information includes two main aspects: (1) Global and local correlations. The local semantic information usually contains short-range changes within the data, while the global semantic information reflects the long-range trends present in the time series [Wang et al., 2023], which can be referred to as global-local semantic information. (2) Long- and shortterm repetitive patterns. Time series often exhibit repetitive patterns at different timescales [Lai

et al., 2018, Liu et al., 2021], such as hourly or daily cycles, which can be identified as periodic semantic information. Furthermore, it is crucial to model both aspects of semantic information simultaneously and ensure that the modeling approach remains computationally efficient, avoiding excessive complexity.

Unfortunately, although the existing state-of-the-art methods have shown great performance, they encounter difficulties in addressing the aforementioned challenges simultaneously. Specifically, CNN-based methods [Bai et al., 2018, Franceschi et al., 2019, Sen et al., 2019, Wu et al., 2023, Wang et al., 2023] have linear complexity with respect to the sequence length L , but they are either limited by the size of the convolution receptive field [Wang et al., 2023] or constrained by the 1D input sequence [Wu et al., 2023], making it difficult to capture both important semantic information simultaneously. Transformer-based methods can be broadly classified into two categories based on whether the point-wise attention is used or not. The used ones [Vaswani et al., 2017, Zhou et al., 2021, 2022a] capture correlations between points in the sequence, yet face challenges in capturing hidden semantic information directly from point-wise input tokens [Nie et al., 2023, Wu et al., 2023]. The others [Li et al., 2019, Wu et al., 2021, Liu et al., 2021] struggles either with achieving sufficiently low computational complexity or with effectively capturing periodic semantic information. Other methods [Zhou et al., 2022b, Zeng et al., 2023] also exhibit limitations in capturing semantic information mentioned above. Further details can be found in Section 2.

RNN-based methods [Hochreiter and Schmidhuber, 1997, Chung et al., 2014, Rangapuram et al., 2018, Salinas et al., 2020] have significant advantages in capturing global and local semantic information through their recurrent structure, as shown in Figure 1(a), while maintaining linear complexity. However, they suffer from gradient vanishing/exploding [Pascanu et al., 2013] and information forgetting issues (refer to Appendix B for more details), making them less suitable for direct application in long-range forecasting tasks. Fortunately, it has been proven that by splitting the information transmissions into patches between a few adjacent time steps and processing them individually, it is possible to maintain both global and local semantic information [Nie et al., 2023, Wu et al., 2023]. This insight is highly inspiring, as it suggests that by dividing the inputs of RNNs into numerous subseries and processing them separately, we can effectively address the significant limitation mentioned earlier, without compromising the efficiency of long-range forecasting.

(a) RNN

Inputs

(i) WITRAN (ours)

(c) Full Attention

Inputs

Input

Embeddings

Layer 3

Embeddings

(b) CNN

Layer 2

Embeddings

Layer 1

Embeddings

Inputs

Patch

Embeddings

Patchs

Attention

Embeddings

(g) PatchTST

(d) LogTrans

Inputs

Input

Embeddings

Layer 3

Embeddings

Layer 2

Embeddings

Layer 1

Embeddings

Information transmission in time series

Information transmission in model

Information transmission in the farthest two input steps in model

The last time step input in time series

Time step input in time series

Local information

Global information

Convolution kernel

Patch

Inputs

Input

Embeddings

Local

Embeddings

Global

Embeddings

(f) MICN

Transpose

Embeddings

Zero input in time series

Inputs

Input

Embeddings

Attention

Embeddings

(e) Pyraformer

Layer 1

Embeddings

Layer 2

Embeddings

Layer 3

Embeddings

Input

Embeddings

Inputs

(h) TimesNet

2D Inputs

2D Input

Embeddings

2D Layer 1

Embeddings

2D Layer 2

Embeddings

2D Layer 3

Embeddings

2D Layer 4

Embeddings

Based on the aforementioned insights, we propose a novel framework called W ater-wave I nformation T ransmission and R ecurrent A cceleration N etwork ( WITRAN ) which comprises two key components: The W ater-wave I nformation T ransmission ( WIT ) framework and the R ecurrent A cceleration N etwork ( RAN ). The overall information transmission process of WITRAN is illustrated in Figure 1(i). Firstly, to capture the periodic semantic information of long- and short-term, we rearrange the input sequences according to their natural period, as shown in Figure 2(a). This rearrangement allows for information transmission in two directions, resembling the propagation of water-wave energy, as shown in Figure 2(b). The horizontal red arrows indicate the intra-periodic information transmission along the time steps, while the vertical blue arrows indicate the inter-periodic information transmission. Secondly, to preserve the characteristics of long- and short-term periodic semantic information, we propose a novel H orizontal V ertical G ated S elective U nit ( HVGSU ) which incorporates G ated S elective C ells ( GSC s) separately in both directions. To capture the correlation between periodic semantic information of both directions at each time step, we design fusion and selection operations in GSC. With a recurrent structure, HVGSU progressively captures more local semantic information until it encompasses the global semantic information. Thirdly, to improve efficiency, we propose the Recurrent Acceleration Network (RAN), which enables parallel processing of information transmission in both directions. Notably, RAN maintains a memory complexity of O ( L ) while reducing the time complexity to O ( √ L ) . In addition, RAN could serve as a universal framework for integrating other models to facilitate information fusion and transmission. In summary, our main contributions are as follows:

We propose a W ater-wave I nformation T ransmission and R ecurrent A cceleration N etwork ( WITRAN ), which represents a novel paradigm in information transmission by enabling bi-granular flows. We provide a comprehensive comparison of WITRAN with previous methods in Figure 1 to highlight its uniqueness. Furthermore, in order to compare the differences between WITRAN and the model (a)-(h) in Figure 1 more clearly, we have prepared Table 1 to highlight the advantages of WITRAN.

Wepropose a novel H orizontal V ertical G ated S elective U nit ( HVGSU ) which captures longand short-term periodic semantic information by using G ated S elective C ell ( GSC ) independently in both directions, preserving the characteristics of periodic semantic information. The fusion and selection in GSC can model the correlations of long- and short-term periodic semantic information. Furthermore, utilizing a recurrent structure with HVGSU facilitates the gradual capture of semantic information from local to global within a sequence.

We present a R ecurrent A cceleration N etwork ( RAN ) which is a generic acceleration framework that significantly reduces the time complexity to O ( √ L ) while maintaining the memory complexity of O ( L ) . We summarize the complexities of different methods in Table 2, demonstrating the superior efficiency of our method.

We conduct extensive experiments on four benchmark datasets across various fields (energy, traffic and weather). The empirical studies demonstrate the remarkable performance of WITRAN, which achieves relative improvements of 5 . 80% and 14 . 28% in long-range and ultra-long-range forecasting respectively. In addition, the introduction of the generic RAN framework greatly improves the computational efficiency.

Period

Rearrange

Coarser

Period

Period

Original series

information transmission

Short-term period

information transmission

Long-term period

information transmission

(a) Information transmission before and after the input sequence rearrangement

Information transmission

from source to sink

Reachable range of

source information

(b) The water-wave information transmission process

Information

wave

Coarser

Period

Period

2 Related Work

Time series forecasting methods can be broadly categorized into statistical methods and neural network methods. Neural network methods, in turn, can be further classified into several subcategories based on the specific techniques employed, including RNN-based, CNN-based, Transformer-based, and other Deep Neural Networks (DNNs) by the methods used.

Statistical Methods with major representatives include ARIMA [Box and Jenkins, 1968], Prophet [Taylor and Letham, 2018], and Holt Winter [Athanasopoulos and Hyndman, 2020]. However, the inherent variability and complexity of real-world time series often render these methods inadequate, as they are built on hypotheses that may not align well with the characteristics of such data. As a result, the performance of these statistical methods tends to be limited.

RNN-based Methods [Hochreiter and Schmidhuber, 1997, Chung et al., 2014, Rangapuram et al., 2018, Salinas et al., 2020] possess the inherent advantage of capturing semantic information in time series data through their recurrent structure, which mimics the natural information transmission process of time series. This allows them to progressively capture semantic information from local to global contexts. These methods exhibit linear time and memory complexity, enabling efficient processing of time series data with a sequence length of L . However, when dealing with longer sequences, the problems of gradient vanishing/exploding [Pascanu et al., 2013] and information forgetting (see Appendix B for more details) further limit it.

CNN-based methods [Bai et al., 2018, Franceschi et al., 2019, Sen et al., 2019, Wu et al., 2023, Wang et al., 2023] are adept at capturing local semantic information through the application of convolutional kernels, while maintaining linear time and memory complexity. However, on the one hand, most of these methods face challenges in capturing comprehensive global information due to the limited receptive field of individual convolutional layers, which make the training process more difficult and overhead [Wang et al., 2023]. On the other hand, it is difficult to directly capture longand short-term repetitive patterns on 1D inputs. MICN [Wang et al., 2023] adopted downsampling 1D convolutions and isometric convolutions combined with a multi-branch framework, which can effectively solve the former problem, but still suffers from the latter one. TimesNet [Wu et al., 2023] transforms 1D inputs into 2D space, leveraging a parameter-efficient inception block to capture intraand inter-period variations. However, it solves the latter problem while still facing the former one.

Transformer-based methods have shown advancements in time series forecasting, with two main categories. The first category uses point-wise attention mechanisms, such as Vanilla Transformer [Vaswani et al., 2017], Informer [Zhou et al., 2021], and FEDformer [Zhou et al., 2022a], while facing challenges in extracting sufficient semantic information from individual time points. The second category employs non point-wise dot product techniques, including LogTrans [Li et al., 2019], Autoformer [Wu et al., 2021], Pyraformer [Liu et al., 2021] and PatchTST [Nie et al., 2023]. They reduce computational complexities to a certain degree, yet face difficulties in incorporating the long-term correlations in time-series. For a detailed description of these methods, please refer to Appendix A. Furthermore, it is worth noting that the majority of the aforementioned methods fail to achieve lower complexity than RNN-based methods. For a comprehensive comparison of the theoretical time and memory complexity, as well as experimental results, please refer to Appendix C.

Other DNN methods have also demonstrated promising performance. For example, FiLM [Zhou et al., 2022b] utilizes Legendre Polynomials and Fourier projection methods to capture historical in-

formation, eliminate noise, and expedite computations using low-rank approximation. DLinear [Zeng et al., 2023] has showcased impressive outcomes by employing simple linear operations. Nevertheless, both approaches encounter difficulties in capturing various repeating patterns present in the sequence.

3 The WITRAN Model

The time series forecasting task involves predicting future values Y ∈ R P × c out for P time steps based on the historical input sequences X = { x 1 , x 2 , . . . , x H } ∈ R H × c in of H time steps, where c in and c out represent the number of input and output features respectively. In order to integrate enough historical information for analysis, it is necessary for H to have a sufficient size [Liu et al., 2021, Zeng et al., 2023]. Furthermore, capturing semantic information from the historical input is crucial for accurate forecasting, which includes modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. However, how to address them simultaneously is a major challenge. With these in mind, we propose WITRAN, a novel information transmission framework akin to the propagation of water waves. WITRAN captures both long- and short-term periodic semantic information, as well as global-local semantic information simultaneously during information transmission. Moreover, WITRAN reduces time complexity while maintaining linear memory complexity. The overall structure of WITRAN is depicted in Figure 3.

Depth:



N

Water-wave Information Transmission Framework

Recurrent Acceleration Network

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

…

…

X

Input

Module

Norm

Concat

Rearrange

Forecasting Module

Fully Connected Layer (FC)

𝑇𝑇𝑇𝑇𝐸𝐸de

�

𝑌𝑌

FC &

De-norm

Y

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

HVGSU

Concat

Concat

Concat

Concat

Concat

Concat

Concat

Concat

Concat

3.1 Input Module

To facilitate the analysis of long- and short-term repetitive patterns, inspired by TimesNet [Wu et al., 2023], we first rearrange the sequence from 1D to 2D based on its natural period, as illustrated by Figure 2(a). Importantly, our approach involves analyzing the natural period of time series and setting appropriate hyperparameters to determine the input rearrangement, rather than using Fast Fourier Transform (FFT) to learn multiple adaptive periods of inputs in TimesNet. Consequently, our method is much simpler. Additionally, in order to minimize the distribution shift in datasets, we draw inspiration from NLinear [Zeng et al., 2023] and employ an adaptive learning approach to determine whether to perform simple normalization. The input module can be described as follows:

here, X 1D ∈ R H × c in represents the original input sequences, x H ∈ R c in represents the input at the last time step of the original sequence, TF en ∈ R H × c time represents the temporal contextual features of original input sequence (e.g., HourOfDay, DayOfWeek, DayOfMonth and DayOfYear), where c time is the dimension of time features. X 2D ∈ R R × C × ( c in + c time ) represents the inputs after rearrangement, where R denotes the total number of horizontal rows and C denotes the vertical columns. norm is an adaptive parameter for different tasks. [ · ] represents the concat operation and Rearrange represents the rearrange operation, with reference to Figure 2(a).

3.2 Horizontal Vertical Gated Selective Unit

To capture long- and short-term periodic semantic information and reserve their characteristics, we propose a novel H orizontal V ertical G ated S elective U nit ( HVGSU ) which consists of G ated S elective C ells ( GSC s) in two directions. To capture the correlation at each time step between periodic semantic information of both directions, we design the specific operations in GSC. Furthermore, HVGSU is capable of capturing both global and local information via a recurrent structure. In this subsection, we will provide a detailed introduction to them.

HVGSU As depicted in Figure 3, the process of HVGSU via a recurrent structure is:

where H hor ∈ R L × R × d model and H ver ∈ R L × C × d model represent the horizontal and the vertical output hidden state of HVGSU separately. L is the depth of the model, and Out ∈ R R × C × d model denotes the output information of the last layer.

In greater detail, the cellular structure of HVGSU is shown in Figure 4(b), which consists of GSCs in two directions to capture the periodic semantic information of long- and short-term. The cell operations for row r (1 ≤ r ≤ R ) and column c (1 ≤ c ≤ C ) in layer l (1 ≤ l ≤ L ) can be formalized as:

here, input r, c, l ∈ R d in . When l = 1 , input r, c, l = x r, c ∈ R c in + c time represents the input for the first layer, and when l > 1 , input r, c, l = o r, c, l -1 ∈ R 2 × d model represents the input for subsequent layers. h hor r, c -1 , l and h ver r -1 , c, l ∈ R d model represent the horizontal and vertical hidden state inputs of the current cell. Note that when r = 1 , h ver r -1 , c, l is replaced by an all-zero tensor of the same size, and the same is true for h hor r, c -1 , l when c = 1 . [ · ] represents the concatenation operation and o r, c, l ∈ R 2 × d model represents the output of the current cell.

(a) GSC structure

𝑥𝑥

Concat

tanh

σ

σ

1-

tanh

ℎ𝑡𝑡-1

𝑝𝑝𝑝𝑝𝑝𝑝

𝑝𝑝𝑝𝑝𝑝𝑝

ℎ𝑡𝑡-1

𝑠𝑠𝑠𝑠𝑠𝑠

(b) HVGSU structure

(c) RAN structure

𝑥𝑥ℎ

,

𝑑𝑑

𝑥𝑥ℎ

,

𝑑𝑑

𝑥𝑥ℎ

,

𝑑𝑑

𝑥𝑥ℎ

,

𝑑𝑑

𝑥𝑥ℎ

,

𝑑𝑑

𝑥𝑥ℎ

,

𝑑𝑑

𝑥𝑥ℎ

,

𝑑𝑑

ℎ𝑝𝑝-1

,

𝑐𝑐

,

𝑙𝑙

𝑣𝑣𝑣𝑣𝑝𝑝

𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖

𝑝𝑝

,

𝑐𝑐

,

𝑙𝑙

ℎ𝑝𝑝

,

𝑐𝑐-1

,

𝑙𝑙

ℎ𝑜𝑜𝑝𝑝

ℎ𝑝𝑝

,

𝑐𝑐

,

𝑙𝑙

ℎ𝑜𝑜𝑝𝑝

ℎ𝑝𝑝

,

𝑐𝑐

,

𝑙𝑙

𝑣𝑣𝑣𝑣𝑝𝑝

𝑜𝑜

𝑝𝑝

,

𝑐𝑐

,

𝑙𝑙

Concat

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

HV

GSU

GSC Inspired by the two popular RNN-based models, LSTM [Hochreiter and Schmidhuber, 1997] and GRU [Chung et al., 2014] (for more details, see Appendix A), we propose a Gated Selective Cell (GSC) to fuse and select information. Its structure is shown in Figure 4(a), which comprises two gates: the selection gate, and the output gate. The fused information consists of input and principal-subordinate hidden states, and the selection gate determines the retention of the original principal information and the addition of the fused information. Finally, the output gate determines the final output information of the cell. The different colored arrows in Figure 4(a) represent different semantic information transfer processes. The black arrow represents the input, the red arrows represent the process of transmitting principal hidden state information, the blue arrow represents the subordinate hidden state, and the purple arrows represent the process by which fused information of

principal-subordinate hidden states is transmitted. The formulations are given as follows:

where h pri t -1 and h sub t -1 ∈ R d model represent the input principal and subordinate hidden state, x ∈ R d in represents the input. W ∗ ∈ R d model × (2 d model + d in ) are weight matrices and b ∗ ∈ R d model are bias vectors. S t and O t denote the selection gate and the output gate, ⊙ denotes an element-wise product, σ ( · ) and tanh( · ) denote the sigmoid and tanh activation function. h f and ˜ h pri t -1 ∈ R d model represent the intermediate variables of the calculation. h pri t represents the output hidden state.

3.3 Recurrent Acceleration Network

In traditional recurrent structure, for two adjacent time steps in series, the latter one always waits for the former one until the information computation of the former one is completed. When the sequence becomes longer, this becomes slower. Fortunately, in the WIT framework we designed, some of the time step information can be computed in parallel. As shown in Figure 2(b), after a point is calculated, the right point in its horizontal direction and the point below it in its vertical direction can start calculation without waiting for each other. Therefore, we propose the Recurrent Acceleration Network (RAN) as our accelerated framework, which enables parallel computation of data points without waiting for each other, greatly improving the efficiency of information transmission in HVGSU. We place parallelizable points in a slice, and the updated information transfer process is shown in Figure 4(c). Each green box in Figure 4(c) represents a slice, and the number of green boxes is the number of times we need to recurrently compute. The meanings of the remaining markers are the same as those in Figure 2. Under the RAN framework, the recurrent length has changed from the sequence length L = R × C to R + C -1 , while the complexity of R and C is O ( √ L ) . Thus, we have reduced the time complexity to O ( √ L ) via the RAN framework. It should be noted that although we parallelly compute some data points, which may increase some memory, the complexity of parallel computation, O ( √ L ) , is far less than the complexity of saving the output variables, O ( L ) , because we need to save the output information of each point in the sequence. Implementation source code for RAN is given in Appendix D.

3.4 Forecasting Module

In the forecasting module, we address the issue of error accumulation in the auto-regressive structure by drawing inspiration from Informer [Zhou et al., 2021] and Pyraformer [Liu et al., 2021], combining both horizontal and vertical hidden states, and then making predictions through a fully connected layer, as illustrated in Figure 3.

We utilize the last row of the horizontal hidden states as it contains sufficient global and latest shortterm semantic information from the historical sequence. In contrast, all columns of the vertical hidden states, which capture different long-term semantic information, are all preserved. The combined operation not only maximizes the retention of the various semantic information needed for predicting the points, but also avoids excessive redundancy in order to obtain accurate predictions. This module can be formalized as follows:

where TFE de ∈ R P × d model represents time features encoding of the forecasting points separately. h hor ∈ R L × 1 × d model represents the last row hidden state in H hor . Repeat ( · ) is for repeat operation and

H hor rep ∈ R L × C × d model represents the hidden state after the repeat operation. H h -v ∈ R C × ( L ∗ 2 d model ) is the vector of horizontal and vertical combination after reshape operation. FC1 and FC2 represent two fully connected layers respectively. ̂ Y ∈ R C × ( R de ∗ d model ) represents the intermediate variables of the calculation and R de × C = P . Y represents the output of this module, and it is worth noting that we need to utilize the adaptive parameter norm for denormalization, when norm = 1 , Y = Y + x H .

4 Experiment

Datasets To evaluate the proposed WITRAN, we conducted extensive experiments on four widely recognized real-world benchmark datasets for long-range and ultra-long-range forecasting. These datasets cover energy, traffic, and weather domains. We split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2. More details about the datasets and implementation are described in Appendix E and Appendix F.

Baselines In light of the underperformance produced by classical models such as ARIMA and simple RNN/CNN models, as evidenced by [Zhou et al., 2021] and [Wu et al., 2021], and the subpar performance exhibited by certain Transformer-based models like LogTrans [Li et al., 2019] and Reformer [Kitaev et al., 2020], as shown in [Wu et al., 2023] and [Wang et al., 2023], our study primarily incorporates six transformer-based models: PatchTST [Nie et al., 2023], FEDformer [Zhou et al., 2022a], Pyraformer [Liu et al., 2021], Autoformer[Zhou et al., 2022a], Informer [Zhou et al., 2021], Vanilla Transformer [Vaswani et al., 2017], and four non-transformer-based methods: MICN [Wang et al., 2023], TimesNet [Wu et al., 2023], DLinear [Zhou et al., 2022b], FiLM [Zhou et al., 2022b]. Please refer to Appendix A and Appendix H for more details of the baselines and implementation.

4.1 Experimental Results

For a better comparison, we adopted the experimental setup of Pyraformer [Liu et al., 2021] for long-range and ultra-long-range series forecasting. In addition, channel-independence is crucial for multivariate time series prediction [Nie et al., 2023], so it is necessary to verify the performance of models on a single channel to ensure their effectiveness across all channels in multivariate time series prediction. In this paper, experiments were conducted on a single channel. Note that in order to fairly compare the performance of each model, we set up the search space so that each model can perform optimally on each task. For further details, please refer to Appendix F.

Long-range Forecasting Results Weconducted five tasks on each dataset for long-range prediction, and the results are shown in Table 3. Taking the task setting 168-336 on the left side of the Table 3 as an example, it indicates that the input length is 168 and the prediction length is 336. Notably, our proposed WITRAN achieves state-of-the-art performance, surpassing the previous best method with an average MSE reduction of 5.803%. Specifically, WITRAN exhibits an average MSE reduction of 10.246% for ECL, 3.879% for traffic, 2.519% for ETTh1, 4.431% for ETTh2, and 7.939% for weather. Additionally, we note that the competition among the baselines is intense due to their best performance on each task, but none of them consistently performs well across all tasks. In contrast, WITRAN demonstrates its robust competitiveness across various tasks and datasets. In addition, we can find that in most cases, a longer input length yields greater improvement with the same prediction length. All findings above collectively highlight WITRAN's efficacy in addressing diverse time-series forecasting tasks in real-world applications. Further results and showcases are presented in Appendix H and Appendix J.

Ultra-long-range Forecasting Results We also show the three tasks for ultra-long-range prediction results on each dataset in Table 4. The tasks on the left side of the table hold the same interpretation as above. Notably, WITRAN achieves an 14.275% averaged MSE reduction. More specifically, WITRAN demonstrates an average MSE reduction of 39.916% for ECL, 3.122% for traffic, 14.837% for ETTh1, 2.441% for ETTh2, and 11.062% for weather. In particular, our method showcases substantial improvements of over 10% in ECL, ETTh1, and weather, reinforcing our ability to predict ultra-long-range time series. And by comparing the results of ultra-long-range forecasting and longrange Forecasting, we can find that our method is more competitive for ultra-long-range prediction,

underscoring WITRAN's capability to capture semantic information in ultra-long historical input. More results and showcases are provided in Appendix H and Appendix J.

4.2 Time and Memory Consumption

To assess the efficiency of our proposed WITRAN, we compared its time and memory consumption with selected baselines that demonstrate superior performance. To fully evaluate the actual complexity of our model and other models for the long range forecasting and ultra-long range forecasting tasks, we set up two cases: 1) constant input length and varying prediction length, 2) constant prediction length and varying input length. In both cases, we fixed the input length and output length at 720. Figure 5 illustrates the comparison, revealing that WITRAN achieves the lowest actual time complexity and memory complexity. This section is described in more detail in Appendix I.

(a) Memory consumption (input length is fixed as 720)

(b) Time consumption (input length is fixed as 720)

(c) Memory consumption (output length is fixed as 720)

(d) Time consumption (output length is fixed as 720)

104

104

12

12

10

8

6

Pyraformer

FEDformer

FiLM

MICN

WIT

s/epoch

103

FiLM

FEDformer

Pyraformer

MICN

WIT

WITRAN

10

c0

6

FEDformer

FiLM

Pyraformer

MICN

WIT

WITRAN

s/epoch

103

MICN

WIT

Pyraformer

FEDfoemer

FiLM

WITRAN

500

1000

1500

2000

2500

WITRAN

3000

102

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

102

500

1000

1500

2000

2500

3000

output length

output length

input length

input length

4.3 Ablation Study

The impact of RAN can be clearly observed in Figure 5. In addition, we conducted ablation studies to measure the impact of HVGSU and GSC in WIT, and the combined operation of the last row of horizontally hidden states and each column of vertically hidden states. Detailed results and discussions

can be found in Appendix G. Here, we provide a concise summary of the key findings: 1) The fusion and selection design of GSC enables both long- and short-term semantic information to be captured for each historical input data points. 2) Setting up independent cells in both directions enables to extract semantic information in the long- and short-term respectively. 3) The specially designed combined operation can make fuller use of the captured local and global semantic information while ensuring that the information is not redundant. 4) RAN offers advantages in terms of speed and space complexity. The utilization of RAN eliminates the need to store excessive intermediate variables, as shown in the previous section.

4.4 Robustness Analysis

We have followed MICN [Wang et al., 2023] and introduced a simple white noise injection to demonstrate the robustness of our model. Specifically, we randomly select a proportion ε of data from the original input sequence and apply random perturbations within the range [ -2 X i , 2 X i ] to the selected data, where X i denotes the original data. After the noise injection, the data is then used for training, and the MSE and MAE metrics are recorded in Table 5.

It can be found that as the perturbation proportion increases, there is a slight increase in the MSE and MAE metrics in terms of forecasting. It indicates that WITRAN demonstrates good robustness when dealing with less noisy data (up to 10 % ), and it possesses a significant advantage in effectively handling various abnormal data fluctuations.

5 Conclusions

In this paper, we propose WITRAN, a novel Water-wave Information Transmission framework and a universal acceleration framework. WITRAN effectively captures both long- and short-term repetitive patterns and global and local correlations with O ( √ L ) time complexity and O ( L ) memory complexity. The experimental results demonstrate the remarkable performance and efficiency of WITRAN. However, this recurrent structure is still not optimally efficient for Python-based implementations because of the information waiting between slices. Therefore, we plan to explore the integration of WITRAN into an interface using C++, similar to the implementation of nn.GRU/nn.LSTM in PyTorch in the future, to further improve its efficiency.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities under Grant No. 2022YJS142. Furthermore, we would like to express our gratitude to Shuo Wang and Shuohao Lin for their valuable discussions and assistance.

References

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations , 2021.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations , 2023.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Longterm forecasting with transformers. In The Eleventh International Conference on Learning Representations , 2023.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations , 2023.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018.

Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems , 32, 2019.

Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances in neural information processing systems , 32, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning , pages 27268-27286. PMLR, 2022a.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems , 32, 2019.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34: 22419-22430, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems , 35:12677-12690, 2022b.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014 , 2014.

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems , 31, 2018.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310-1318. Pmlr, 2013.

George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 17(2):91-109, 1968.

Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician , 72(1):37-45, 2018.

G Athanasopoulos and RJ Hyndman. Forecasting: principle and practice: O texts; 2018, 2020.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems , 30, 2017.

Zeping Yu and Gongshen Liu. Sliced recurrent neural networks. In Proceedings of the 27th International Conference on Computational Linguistics , pages 2953-2964, 2018.

Lifeng Shen, Qianli Ma, and Sen Li. End-to-end time series imputation via residual short paths. In Asian conference on machine learning , pages 248-263. PMLR, 2018.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

A A Brief Overview of RNN-Based and Transformer-Based Models

In this section, we will provide a brief overview of representative variants of RNN-based methods (Appendix A.1) and Transformer-based methods (Appendix A.2). Detailed descriptions of other methods can be found in Section 2.

A.1 RNN-based methods

LSTM [Hochreiter and Schmidhuber, 1997] and GRU [Chung et al., 2014] are two prominent variants of RNN-based models. They have gained popularity in diverse domains, such as natural language processing, speech recognition, and video analysis due to their ability to capture important information. This subsection, we will provide a detailed introduction to LSTM and GRU.

𝑥𝑥

tanh

σ

ℎ𝑡𝑡-1

1-

Concat

σ

Concat

(b) GRU cell structure

𝑥𝑥

tanh

σ

𝐶𝐶

𝑡𝑡-1

𝐶𝐶

𝑡𝑡

Concat

σ

ℎ𝑡𝑡-1

σ

tanh

(a) LSTM cell structure

LSTM The structure of LSTM, shown in Figure 6(a), comprises three gates: the input gate, the forget gate and the output gate. In addition, LSTM has two vectors: the hidden state and the cell state, for recursively processing the information flow between different time steps. LSTM can be formalized as follows:

where f lstm t , i lstm t and o lstm t represent the input gate, the forget gate, and the output gate. x ∈ R d in represents the input, and C t -1 , h t -1 ∈ R d model represents the input cell state and the input hidden state. W lstm ∗ ∈ R d model × ( d model + d in ) are weight matrices and b lstm ∗ ∈ R d model are bias vectors. C t and h t denote the output cell state and the output hidden state; ˜ C t represents the intermediate variables of the calculation; ⊙ denotes an element-wise product; σ ( · ) and tanh( · ) denote the sigmoid and tanh activation function.

GRU The structure of GRU, depicted in Figure 6(b), consists of two gates: the reset gate and the update gate. Besides, GRU has one vector: the hidden state. It can be mathematically formulated as follows:

where r gru t and u gru t represent the reset gate and the update gate. x ∈ R d in represents the input, and h t -1 ∈ R d model represent the input hidden state. W gru ∗ ∈ R d model × ( d model + d in ) are weight matrices and b gru ∗ ∈ R d model are bias vectors. h t denotes the output hidden state; ˜ h t gru represents the intermediate variables of the calculation; ⊙ denotes an element-wise product; σ ( · ) and tanh( · ) denote the sigmoid and tanh activation function.

A.2 Transformer-based methods

In recent years, variants of Transformer have made significant progress in time series forecasting and can be divided into two categories.

The first category captures the correlation between different time steps through point-wise attention mechanisms. Vanilla Transformer [Vaswani et al., 2017] is able to effectively extract correlations between any two-time steps through its self-attention mechanism, but its complexity is too high, reaching O ( L 2 ) . Informer [Zhou et al., 2021] proposes a ProbSparse self-attention with distilling techniques, reducing the complexity to O ( L log L ) . FEDformer [Zhou et al., 2022a] takes a different approach by applying attention modules in the frequency domain through Fourier or wavelet transformations, achieving a complexity of O ( L ) . Despite the performance and efficiency improvements achieved by these representative methods, they face difficulties in extracting sufficient semantic information from a single time point, which has been raised in previous works [Nie et al., 2023, Wu et al., 2023].

The second category adopts non-dot-product techniques to extract correlation information. LogTrans [Li et al., 2019] effectively captures local information by leveraging LogSparse and convolutional self-attention layers. However, it overlooks the long-term repetitive pattern of points among subseries and exhibits a high complexity of O ( L log L ) . Drawing from the traditional ideas of time-series analysis, Autoformer [Wu et al., 2021] is able to capture long-range trends and shortrange changes in sequences through decomposition and auto-correlation mechanisms. However, its complexity remains high at O ( L log L ) . Pyraformer [Liu et al., 2021] initializes coarser-scale node information in the pyramidal graph by capturing local and global correlations through convolution. It then employs pyramidal attention to effectively capture long- and short-term repetitive patterns, leading to significant performance improvements while reducing complexity to O ( L ) . However, Pyraformer still suffers from the limitation of the receptive field of the convolution kernel as discussed earlier. PatchTST [Nie et al., 2023] captures local semantic information through patch, and further reduces the complexity to O (( L/S ) 2 ) , where S represents the stride length. Nevertheless, it does not consider the long-term repeated pattern among subseries.

Based on the above analysis, we find that most Transformer-based methods struggle to simultaneously effectively model local and global correlations (global-local semantic information) and capture longand short-term repetitive patterns (periodic semantic information) in sequences. Furthermore, it is worth noting that the majority of the aforementioned methods fail to achieve lower complexity than RNN-based methods. For a comprehensive comparison of the theoretical time and memory complexity, as well as experimental results, please refer to Appendix C.

B Inspiration and Our Approach

RNNs, such as LSTM and GRU, have a significant advantage in capturing semantic information due to their recurrent structure, as discussed in Appendix A.1. Yet RNNs still have some drawbacks. In addition to the well-known issues of gradient vanishing/exploding [Pascanu et al., 2013]. This paper focuses on analyzing the information forgetting problem (Appendix B.2) and the difficulty in parallelizing RNN (Appendix C). In this section, we will analyze RNNs using LSTM and GRU as representatives, and introduce the inspiration we draw from during the process of designing our model.

B.1 LSTM VS GRU

LSTM incorporates three gates (input gate, output gate, and forget gate), while GRU utilizes two gates (reset gate and update gate). The complex gating mechanism of LSTM enables it to capture and learn intricate patterns, making it effective for processing longer sequences. On the other hand, GRU

performs well on shorter sequences and requires less computational and memory resources compared to LSTM. It should be noted that, as shown in Equation 7, the computation of ˜ h t in GRU is dependent on the completion of r t computation. This undoubtedly slows down the computation that involves the input information x . Therefore, in terms of actual running time, GRU may not necessarily have an advantage over LSTM, as detailed in Appendix C.2.

B.2 The information forgetting problem of RNNs

When Recurrent Neural Networks (RNNs) process sufficiently long sequences, the information contained in the earlier parts of the sequence may be forgotten within the recurrent structure as it passes through multiple cells. To verify this phenomenon, we conducted a concrete experiment. We conducted a benchmark experiment using the ECL dataset, with a historical sequence length of 720 and a prediction horizon of 720, and the description of the ECL dataset can be found in Appendix E.

To eliminate the influence caused by the split of training, validation and test sets, we partitioned the dataset with a fixed input sequence length of 720 and a prediction horizon of 720. Furthermore, we conducted experiments with varying input sequence lengths using LSTM and GRU models. For the experiments, we used the last m = { 24 , 48 , 72 , . . . , 696 , 720 } points near the prediction time as input, and the results of these experiments are shown in Figure 7.

GRU

0.70

LSTM

0.65

0.60

MSE

0.55

0.50

0.45

0

100

200

300

400

500

600

700

input length

Observing Figure 7, on the one hand, we can see that when the input sequence length is relatively short (not within the red box), the prediction performance of LSTM and GRU gradually improves as sequence length increases. This indicates that longer input sequences can provide more contextual information, which can enhance the model's prediction performance and is consistent with previous research findings [Liu et al., 2021, Zeng et al., 2023]. On the other hand, the part within the red box indicates that when longer historical sequences are fed into LSTM and GRU, their prediction performance does not continue to improve. This suggests that introducing earlier information does not provide significant benefits for the models, which in turn confirms the problem of information forgetting in RNNs.

B.3 Inspiration and WITRAN

Our proposed Water-wave Information Transmission (WIT) framework utilizes input rearrangement to capture both long- and short-term repeating patterns in the sequence more directly and efficiently, while avoiding the processing of excessively long sequences, as described in Section 3.1. This successfully addresses the limitations of RNNs, such as the problems of exploding and vanishing gradients, as well as information forgetting. Furthermore, through the design of Recurrent Acceleration Network (RAN), we enable parallel computation of the entire sequence, as described in Section 3.3. For a comprehensive understanding of RAN, we provide detailed code and analysis of RAN in Appendix D. In this subsection, we focus on introducing the inspiration behind the important components of WIT.

The specific structure of WIT has been detailed in Section 3.2. The inspiration behind the design of Horizontal Vertical Gated Selective Unit (HVGSU) and Gated Selective Cell (GSC) comes from the advantages of LSTM and GRU. HVGSU serves two major purposes: 1) capturing both longand short-term periodic semantic information in the sequence, and 2) modeling these two types

of periodic semantic information simultaneously for each time step. To achieve these objectives, two separate GSCs are established in the HVGSU, each responsible for the fusion and selection of long-term and short-term information in both directions.

For each GSC cell, to selectively incorporate fused information into the periodic semantic information in each direction, we draw inspiration from the reset gate of GRU and design a selection gate to control the retention of original information and the addition of new information. Furthermore, we are inspired by the output gate of LSTM and design our output gate to output the updated information. To reduce computational costs while maintaining sufficient learning capacity, we draw inspiration from GRU and only utilize these two gates, resulting in smaller computational and memory overheads. However, we address the problem of waiting for information to be computed between gate calculations in GRU by parallelizing the gate calculation vectors with the input x , which greatly accelerated the computation speed of the cell. In summary, we analyze and draw inspiration from the strengths of LSTM and GRU, and incorporated them into the design of our GSC.

B.4 WITRAN VS other RNN-based models

There are also several other RNN models, such as DilatedRNN [Chang et al., 2017], SlicedRNN [Yu and Liu, 2018], and RIMP-LSTM [Shen et al., 2018], which also have certain advantages in handling sequences. To further compare the differences between WITRAN and these methods, we provide the comparison in Table 6:

(1) Efficiently (1 layer) model global correlations (a) When the dilations of DilatedRNN does not include the value 1, multiple layers need to be constructed to extract global correlations. (b) SlicedRNN improves efficiency to some extent by parallel processing of minimum subsequences, but it still requires the introduction of multiple layers to capture the global correlations of the sequence.

(2) Special design to capture long-term repetitive patterns (a) SlicedRNN is unable to capture longterm repetitive patterns among elements of sub-sequences. (b) Although RIMP-LSTM incorporates Residual Paths and Residual Sum Unit designs, it still cannot effectively extract long-term repetitive patterns.

(3) Using 1 layer to capture long- and short-term repetitive patterns simultaneously (a) DilatedRNN can capture long- and short-term repetitive patterns, but it requires the use of multiple layers to achieve this. (b) SlicedRNN and RIMP-LSTM are not particularly adept at handling long-term repetitive patterns, as mentioned in (2) .

(4) Well solve the gradient vanishing/exploding problem of RNN (a) In DilatedRNN [Chang et al., 2017], it states and provides formal proof that reducing the length of information paths between time steps can prevent the issues of gradient vanishing/exploding. DilatedRNN, SlicedRNN, and WITRAN tackle this problem by reducing the length of information transmission paths. (b) RIMP-LSTM addresses this issue by the designs of Residual Paths and Residual Sum Units.

C Time and Memory Consumption of RNNs and Transformer

In this section, we derive the complexities of LSTM, GRU and Vanilla Transformer respectively, and analyze their actual running speeds and memory usage through experiments.

C.1 The theoretical computational complexity

The structure diagrams of LSTM and GRU are shown in Figure 6, while the structure of Vanilla Transformer can be found in paper [Vaswani et al., 2017]. Assuming the sequence length is L , the

input dimension is d in and the hidden layer dimension is d model , and considering the case where each model has only one layer, the derivation process is as follows:

GRU GRU includes two gates and three matrix multiplication operations, as mentioned in Appendix A.1. Each matrix multiplication operation has a complexity of d model × ( d model + d in ) , although there are also ⊙ operations in GRU, and the complexity of element-wise multiplication is d model , which is much smaller than the complexity of matrix multiplication operation. Therefore, the complexity of one GRU unit can be simplified as 3 × d model × ( d model + d in ) . When GRU processes the entire sequence, it needs to perform L sequential computations, therefore, the overall computational complexity can be summarized as: 3 × L × d model × ( d model + d in ) .

LSTM LSTM includes three gates and four matrix multiplication operations, as mentioned in Appendix A.1. Similar to GRU, the computational complexity of LSTM can also be summarized as: 4 × L × d model × ( d model + d in ) .

Transformer Taking the single-head attention mechanism in Transformer as an example, in the calculation of the attention mechanism, the each linear transformations of Query ( Q ) , Key ( K ) and V alue ( V ) require matrix multiplication operation of size d model × d model . In the calculation of the attention weights, the Q and transpose of K are multiplied together, and since the size of Q and K are L × d model , the complexity of computing the attention weights is L × d model × L , and the size of attention weights is L × L . Similarly, when we multiply the attention weights with V in the computation, the complexity is L × L × d model . The output from the attention matrix multiplication is further linearly transformed, and thus requires an additional d model × d model complexity. To summarize, the overall complexity of the single-headed attention computation in Transformer can be expressed as: 1) Linear transformations of Q , K and V : 3 × L × d model × d model . 2) Calculation of attention weights: L × d model × L . 3) Matrix multiplication of attention weights and V : L × L × d model . 4) Linear transformation of the output: L × d model × d model . The total complexity of the singleheaded attention computation in Transformer is 3 × L × d model × d model + L × d model × L + L × L × d model + L × d model × d model = 4 × L × d model 2 +2 × L 2 × d model . It should be noted that the complexity improvements in Transformer-based methods only apply to the L 2 regularization term, as the linear transformations are necessary for any Transformer-based model. In addition, assuming the dimension of the Feed-Forward Networks (FFN) in Transformer is d ff , the first layer of the fully connected network has a complexity of L × d model × d ff , while the second layer has a complexity of L × d ff × d ff , and the total complexity of FFN is L × d model × d ff + L × d ff 2 . It should be noted that the input and output transformations in Transformer still require a certain level of complexity. On the hand, as it needs to transform the input data from dimension d in to dimension d model , the input transformation in Transformer requires at least a L × d in × d model complexity. On the other hand, as it needs to transform the input data from dimension d ff to dimension d model , the output transformation requires a L × d ff × d model complexity. Therefore, the overall complexity of Transformer can be summarized as: L × d in × d model +4 × L × d model 2 +2 × L 2 × d model + L × d model × d ff + L × d ff 2 + L × d ff × d model . It should be noted that in most cases for Transformer, d ff is larger than d model .

Summary In time series prediction tasks, the input dimension d in is usually much smaller than the model dimension d model . Based on the analysis conducted earlier, we can observe that the computational complexity of the encoder part of most Transformer-based methods is generally higher than that of LSTM and GRU. This is because Transformer includes many complex operations such as matrix multiplication and linear transformation. Although there have been improvements made to the attention mechanism, these modifications only affect a small portion of the overall complexity which remains high. Therefore, compared to LSTM and GRU, the computational complexity of Transformer is higher.

C.2 The practical time and memory consumption

In practical situations, LSTM can perform computations in parallel without the need to wait for information from all four matrix operations. On the other hand, in GRU, the first two matrix operations can be parallelized, but the third matrix operation has a dependency on the completion of the previous operations, as mentioned in Appendix A.1. Therefore, the actual runtime of GRU may be longer than

Algorithm 1 GRU based on Python

that of LSTM. To verify this, we compared the processing time and memory usage of a single-layer LSTM and a single-layer GRU on sequences of the same length using Python implementation, as shown in Figure 8.

(a) Memory consumption (task is fixed as 720-720)

(b) Time consumption (task is fixed as 720-720)

(c) Memory consumption (output length is fixed as 720)

(d) Time consumption (output length is fixed as 720)

4.0

3.5

+

GRU

LSTM

Transformer

nn.LSTM

102

16

14

12

GRU

LSTM

nn.LSTM

Transformer

103

s/epoch

10

s/epoch

GB

3.0

GRU

LSTM

8

102

2.5

101

Transformer

nn.LSTM

4

LSTM

GRU

2.0

100

200

output length

300

400

500

100

200

outputlength

300

400

500

500

1000

input length

1500

2000

2500

3000

101

500

1000

input length

1500

2000

2500

nn.LSTM

Transformer

3000

Transformer has the advantage of parallelizing the computation of correlations for a sequence of length L, which greatly improves its speed compared to GRU/LSTM. It is important to note that this comparison is not entirely fair, as Transformer can parallelize at the sequence level, whereas RNNs can parallelize at the batch level. Each approach has its own strengths and is suited for different scenarios.

To ensure a fair comparison of runtime and memory usage, it is important to consider the conditions under which the models are compared. Following the experimental setup in FiLM [Zhou et al., 2022b], we can ensure that the hidden state dimension and batch size of the models being compared are the same. This allows us to evaluate whether the models can be trained on devices with low memory capacity. For time evaluation, we need to ensure that the hidden state dimension is the same and the memory utilization is similar, so that memory resources are not wasted. In terms of time comparison between Transformer and GRU/LSTM, although GRU/LSTM are slower than

Transformer under the same batch conditions in Python implementation, they can still speed up their parallel processing by increasing batch size while ensuring similar memory usage to Transformer. At the same time, because the computation process of GRU and LSTM is much simpler than that of Transformer, they can be more easily implemented in C++ and integrated into PyTorch's library as nn.GRU/nn.LSTM, which is faster. For comparison, we compared the memory and time costs of these five methods, as shown in Figure 8. The Python implementation of GRU and LSTM code is shown in Algorithm 1 and Algorithm 2.

Algorithm 2 LSTM based on Python

D The Design Process and Implementation Code of Recurrent Acceleration Network (RAN)

In this section, we will provide a detailed explanation of the implementation approach (Appendix D.1) and specific code of RAN (Appendix D.2).

D.1 Implementation approach of RAN

In Section 3.3, we introduced the Recurrent Acceleration Network (RAN) as part of our overall approach. However, we encountered a challenge when the number of rows processed by RAN is smaller than the number of columns. In such cases, if the number of slices processed exceeds the number of rows, the intermediate variables of the slices become fully occupied. To ensure the continuity of operations, it becomes necessary to continuously shift and reset these variables. However, we found that this reset operation was time-consuming and hindered the expected acceleration benefits of RAN. To overcome this issue, we made a simple adjustment by setting the size of the slice to the maximum value of the rows and columns. While this may introduce additional space complexity, it maintains the length of the slice and eliminates the need for a reset operation. As a result, the

processing time is significantly reduced. The comparison diagram between RAN-min and RAN-max is shown in Figure 9.

Zero

(a) RAN-min

(b) RAN-max (in WITRAN)

D.2 Specific code of RAN

The specific code of RAN can be found in Algorithm 3. Due to its long content, we place it at the end of the Appendix.

E Dataset Details

In this section, we will provide a detailed introduction of the datasets used in this paper. (1) Electricity 2 ( ECL ) contains the hourly electricity consumption of 321 customers from 2012 to 2014. (2) ETT [Zhou et al., 2021] contains the load and oil temperature, which was recorded every 15 minutes between July 2016 and July 2018 from electricity transformers. (3) Traffic 3 contains the hourly data the road occupancy rates measured by different sensors on San Francisco Bay area freeways, collected from California Department of Transportation. (4) Weather 4 contains 21 meteorological indicators (such as air temperature, humidity, etc.) and was recorded every 10 minutes for 2020 whole year. They are all split into the training set, validation set and test set by the ratio of 6:2:2 during modeling.

Due to the different original acquisition granularities of each dataset, in order to ensure that they contain the same semantic information on the same task, in this paper, our experiments are conducted by aggregating them into one hour. Table 7 summarizes all the features of the four benchmark datasets. The target value of ETT is 'oil temperature (OT)', the target value of ECL is 'MT_320', the target value of Traffic is 'Node_862' and the target value of Weather is 'wet_bulb'.

F Experiment Setup Details

Our method and all the baseline methods are trained with the L2 loss, using the ADAM [Kingma and Ba, 2014] optimizer with an initial learning rate of 10-3. Batch size is set to 32. The training process is early stopped after 5 epochs if there is no loss degradation on the validation set and the max epochs is 25. We save all the models with the lowest loss on the validation set for the final testing.

To ensure a fair comparison of each model's performance, we set the same search space for the common parameters included in each model. This ensures that each model can achieve its best performance under the same conditions for comparison, with the seed set to 2023.

Regarding the private parameters of each model, we set the corresponding search space according to the specific descriptions given in their respective papers. In addition, it should be emphasized that the performance of Pyraformer shown in this paper is the best between the two prediction designs of the model, the performance of FEDformer shown here is the best between FEDformer-w and FEDformer-f, the performance of DLinear shown here is the best among DLinear, NLinear and Linear, and the performance of MICN is the best between MICN-regre and MICN-mean.

The mean square error (MSE) and mean absolute error (MAE) are used as metrics. All experiments are repeated 5 times and the mean of the metrics is reported as the final results in Table 3 and Table 4.

All the deep learning networks are implemented using PyTorch [Paszke et al., 2019] and conducted on NVIDIA RTX A4000 16GB GPUs.

The search space for the common parameters is as follows: (1) d model is set to {32, 64, 128, 256, 512, 1024}. (2) e layer and d layer are set to {1, 2, 3}. (3) n head is set to {1, 2, 4, 8} and d ff is four times as large as d model . Regarding the private parameters of our model, norm is set to {0, 1} and C represents the period is set to {12, 24, 48} and it is required that the length H of the input sequence is divisible by C . Furthermore, it is important to note that adjustments should be made to C for different tasks to ensure its ability to adequately represent the period.

G Ablation Study

In Section 4.3, we presented the results of our ablation experiments to demonstrate the effectiveness of each component in the Water-wave Information Transmission (WIT) framework. We design five variant methods and evaluate their performance on four datasets. The experimental results are summarized in Table 8, and we will provide a detailed analysis of these results in this section. It is important to note that the role of the Recurrent Acceleration Network (RAN) is primarily to improve computational efficiency, as demonstrated in Section 4.2, and it does not significantly impact the model's accuracy. Therefore, our focus in this section will be on discussing the design and role of the WIT framework.

G.1 Impact of forecasting module

Our proposed forecasting module is also a part of the model design, and its goal is to effectively capture periodic semantic information and global-local semantic information obtained through the Water-wave Information Transmission (WIT) framework. Our specific design has been described in detail in Section 3.4. To verify that we can fully utilize the relevant information by combining operations for pairwise prediction, we changed the combination prediction method of this module to a simple way of first concatenating the horizontal hidden state and the vertical hidden state, and then using the fully connected prediction module for prediction. We named this variant WITRAN-FC.

The comparison of results between WITRAN and WITRAN-FC in Table 8 shows that when we remove the combining operations for pairwise prediction and replace it with ordinary concatenation and fully connected operations, even though the same information is used, WITRAN-FC has difficulty discovering the most relevant long-term periodic semantic information for each prediction point. This demonstrates the effectiveness of the combining operations for pairwise prediction in the forecasting module.

G.2 Impact of information fusion and selection

As information fusion and selection are completed together in the Gated Selective Cell (GSC) we designed, we need to verify the effectiveness of GSC, that is, the effectiveness of information fusion and selection. Here, we replaced the cells in the two directions of Horizontal Vertical Gated Selective Unit (HVGSU) with LSTM and GRU respectively, and named them WITRAN-2DLSTM and WITRAN-2DGRU. It should be noted that since the information between the LSTM/GRU cell in the two directions is not fused at this time, the information in the last row of the horizontal direction cannot contain global information, so we cannot use the proposed forecasting module for application, but only use the FC method mentioned in the previous subsection for prediction. It should be noted that the comparison between WITRAN-2DLSTM/WITRAN-2DGRU and WITRAN-FC reflects the impact of information redundancy on experimental performance, while the comparison between WITRAN-2DLSTM/WITRAN-2DGRU and WITRAN reflects the impact of information fusion and selection on experimental performance.

The comparison of results between WITRAN-2DLSTM/WITRAN-2DGRU and WITRAN-FC in Table 8 demonstrates that too much information redundancy may be detrimental to the model's prediction performance, because for WITRAN-FC, the horizontal hidden states it receives contain more information as they go further down, and they also contain all the information in the horizontal direction above them. The same applies to the vertical direction, where the further to the right the hidden state is, the more information it contains. Furthermore, the information captured by WITRAN-2DLSTM and WITRAN-2DGRU is independent among different horizontal rows and among different vertical columns, and the semantic information between horizontal and vertical

hidden state are different. Therefore, WITRAN-2DLSTM and WITRAN-2DGRU are not heavily influenced by redundant information. In addition, the comparison of results between WITRAN and WITRAN-2DLSTM/WITRAN-2DGRU shows that information fusion and selection can effectively extract relevant information, which demonstrates the necessity of the GSC design in WIT.

G.3 Impact of bi-granular information transmission

In the WITRAN we designed, information can be propagated in two directions, and because the time granularity of sequence propagation in these two directions is different, their periodic semantic information is different. In order to verify the effectiveness of the two independent GSCs set up in HVGSU to capture long- and short-term periodic semantic information, we replaced the cells in the two directions of WITRAN-2DLSTM/WITRAN-2DGRU with a single LSTM/GRU cell, and uniformly used the hidden state information of one point to transmit information in both directions. It should be noted that at this time, the information propagated horizontally and vertically by the model is completely the same. Sure, we can name the current models as WITRAN-LSTM/WITRAN-GRU.

The comparison of results between WITRAN-2DLSTM/WITRAN-2DGRU and WITRANLSTM/WITRAN-GRU in Table 8 demonstrates that failing to distinguish between long- and shortterm periodic semantic information can have a significant negative impact on the performance of prediction tasks.

G.4 Summary

In this section, we focused on the importance of each building module of WIT. During the experiments, we gradually removed the building blocks we designed, and it was clear that the performance decreased significantly. This fully demonstrates the rationality and effectiveness of the WIT framework we designed. Specifically, the summary of WIT is as follows: (1) The fusion and selection design of GSC enables both long- and short-term semantic information to be captured for each historical input data point. (2) Setting up independent cells in both directions enables to extract semantic information in the long- and short-term respectively. (3) The specially designed combined operation is able to make fuller use of the captured local and global semantic information while ensuring that the information is not redundant.

H Experiment Error Bars

We saved all models and used the model with the lowest validation loss for the final testing. We repeated this process 5 times and calculated the error bars for all models to compare their robustness on long-range and ultra-long-range tasks, as shown in Table 9. It can be seen from Table 9 that the overall performance of the proposed WITRAN is better than other state-of-the-art baseline models, indicating the effectiveness of our approach.

I Model Analysis

In this section, we will analyze the parameter sensitivity (Appendix I.1) of our proposed model and provide a detailed explanation (Appendix I.2) of the time and memory consumption in Section 4.2.

I.1 Parameter sensitivity

Regarding the private parameters of our model, norm is set to {0, 1} and C represents the period is set to {12, 24, 48} and it is required that the length H of the input sequence is divisible by C . In this subsection, we will explain the choices of these parameters and their impact on the model's predictions.

The impact of norm For different prediction tasks on different datasets, we determined the value of norm through validation set. To verify that the value of norm conforms to the distribution of the datasets, we conducted experiments on the distribution of the training and validation sets for different prediction tasks, the results and the value of norm are shown in Table 10. We noticed that although there may be a significant difference in the mean values of the data between the training and validation

Datasets

ECL

Traffic

ETTh1

ETTh2

Weather

Metric

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

WITRAN(ours)

168-168

0.2397

±

0.00859

0.3519

±

0.00601

0.1377

±

0.00231

0.2051

±

0.00300

0.1105

±

0.00082

0.2589

±

0.00128

0.2389

±

0.00615

0.3813

±

0.00566

0.2050

±

0.00428

0.3338

±

0.00483

168-336

0.2607

±

0.00926

0.3721

±

0.00783

0.1321

±

0.00327

0.2059

±

0.00359

0.1189

±

0.00325

0.2714

±

0.00439

0.2277

±

0.00805

0.3778

±

0.00772

0.2197

±

0.00629

0.3470

±

0.00328

336-336

0.2517

±

0.00946

0.3627

±

0.00824

0.1306

±

0.00080

0.2054

±

0.00161

0.1112

±

0.00161

0.2638

±

0.00221

0.2432

±

0.00303

0.3922

±

0.00470

0.2163

±

0.00520

0.3482

±

0.00910

336-720

0.3084

±

0.01255

0.4055

±

0.00807

0.1391

±

0.00019

0.2175

±

0.00202

0.1494

±

0.01004

0.3092

±

0.01220

0.2373

±

0.00815

0.3888

±

0.00558

0.2054

±

0.00322

0.3424

±

0.00328

720-720

0.2478

±

0.02129

0.3651

±

0.01580

0.1408

±

0.00109

0.2191

±

0.00018

0.1296

±

0.00220

0.2868

±

0.00265

0.2635

±

0.01346

0.4018

±

0.00935

0.2008

±

0.00374

0.3417

±

0.00159

720-1440

0.2499

±

0.00779

0.3727

±

0.00097

0.1672

±

0.00389

0.2449

±

0.00746

0.1331

±

0.00273

0.2943

±

0.00342

0.2915

±

0.02237

0.4289

±

0.01468

0.1872

±

0.00281

0.3312

±

0.00442

1440-1440

0.2408

±

0.01817

0.3680

±

0.01533

0.1543

±

0.00570

0.2325

±

0.00944

0.1304

±

0.00425

0.2902

±

0.00503

0.2815

±

0.01771

0.4220

±

0.01177

0.1907

±

0.00622

0.3366

±

0.00648

1440-2880

0.3359

±

0.01107

0.4383

±

0.00809

0.1425

±

0.00461

0.2333

±

0.00745

0.1850

±

0.01106

0.3452

±

0.01388

0.3280

±

0.04586

0.4585

±

0.03506

0.1769

±

0.00270

0.3257

±

0.00212

MICN

168-168

0.3168

±

0.01684

0.4067

±

0.01116

0.2428

±

0.00416

0.3543

±

0.00511

0.1257

±

0.01332

0.2803

±

0.01514

0.2734

±

0.01795

0.4162

±

0.01662

0.2231

±

0.00219

0.3489

±

0.00404

168-336

0.3002

±

0.00979

0.4053

±

0.00853

0.2401

±

0.00537

0.3514

±

0.00657

0.1422

±

0.02929

0.3006

±

0.03410

0.3017

±

0.04440

0.4429

±

0.03426

0.2663

±

0.00330

0.3837

±

0.00602

336-336

0.3092

±

0.00621

0.4132

±

0.00590

0.2413

±

0.00863

0.3549

±

0.00901

0.1576

±

0.03357

0.3159

±

0.03352

0.3472

±

0.05168

0.4796

±

0.03738

0.2701

±

0.00708

0.3804

±

0.00650

336-720

0.3820

±

0.01996

0.4704

±

0.01565

0.2422

±

0.00545

0.3513

±

0.00292

0.2219

±

0.05068

0.3729

±

0.04821

0.4248

±

0.04274

0.5268

±

0.02923

0.3086

±

0.00548

0.4138

±

0.00974

720-720

0.3463

±

0.01609

0.4381

±

0.00853

0.2552

±

0.01039

0.3709

±

0.01432

0.2959

±

0.07568

0.4402

±

0.06838

0.3549

±

0.03558

0.4805

±

0.02590

0.2828

±

0.00451

0.3969

±

0.00339

720-1440

1.0460

±

0.18834

0.7765

±

0.07715

0.2876

±

0.00409

0.3916

±

0.00354

0.4640

±

0.04845

0.5836

±

0.03802

0.4922

±

0.05114

0.5649

±

0.03187

0.3999

±

0.03417

0.4848

±

0.02302

1440-1440

2.2862

±

0.36532

1.2207

±

0.13085

0.2950

±

0.00611

0.3923

±

0.00630

0.5650

±

0.12064

0.6293

±

0.07367

0.5030

±

0.04996

0.5644

±

0.02600

0.2873

±

0.01038

0.4201

±

0.00799

1440-2880

2.8936

±

1.28817

1.3717

±

0.37787

0.2823

±

0.01489

0.3874

±

0.01046

0.7591

±

0.49493

0.7215

±

0.30100

0.5549

±

0.08677

0.5886

±

0.05173

0.3570

±

0.03915

0.4810

±

0.02942

TimesNet

168-168

0.2825

±

0.00960

0.3797

±

0.00798

0.1490

±

0.00412

0.2293

±

0.00686

0.1133

±

0.00253

0.2612

±

0.00377

0.2655

±

0.01248

0.4051

±

0.01202

0.2420

±

0.00614

0.3608

±

0.00466

168-336

0.3505

±

0.00624

0.4253

±

0.00428

0.1499

±

0.00136

0.2356

±

0.00298

0.1202

±

0.00237

0.2732

±

0.00359

0.2725

±

0.00610

0.4163

±

0.00564

0.2821

±

0.01949

0.3885

±

0.01692

336-336

0.3702

±

0.05183

0.4307

±

0.02897

0.1446

±

0.00408

0.2300

±

0.00499

0.1279

±

0.01137

0.2846

±

0.01151

0.3184

±

0.07475

0.4431

±

0.04618

0.2684

±

0.01958

0.3752

±

0.01685

336-720

0.3879

±

0.02491

0.4531

±

0.01278

0.1584

±

0.00141

0.2440

±

0.00275

0.1501

±

0.00999

0.3127

±

0.01039

0.2858

±

0.00846

0.4253

±

0.00492

0.2930

±

0.01619

0.4045

±

0.01427

720-720

0.3537

±

0.02613

0.4386

±

0.01932

0.1546

±

0.00393

0.2410

±

0.00160

0.1510

±

0.01739

0.3118

±

0.01866

0.2936

±

0.02218

0.4238

±

0.01457

0.2967

±

0.02166

0.4070

±

0.01439

720-1440

0.6119

±

0.12940

0.5962

±

0.06975

0.1882

±

0.05920

0.2656

±

0.05923

0.1391

±

0.00371

0.3049

±

0.00499

0.4186

±

0.05841

0.5092

±

0.03150

0.2407

±

0.00761

0.3694

±

0.00643

1440-1440

0.5720

±

0.14337

0.5712

±

0.07619

0.1598

±

0.01064

0.2388

±

0.01276

0.1801

±

0.01165

0.3372

±

0.01145

0.4409

±

0.07863

0.5218

±

0.04638

0.2869

±

0.01684

0.4033

±

0.01504

1440-2880

0.7683

±

0.06278

0.6846

±

0.02955

0.1560

±

0.00700

0.2409

±

0.00845

0.2732

±

0.02157

0.4094

±

0.01088

1.5304

±

0.84959

0.9026

±

0.32013

0.2199

±

0.02223

0.3563

±

0.01606

PatchTST

168-168

0.2980

±

0.00339

0.3832

±

0.00341

0.1622

±

0.01189

0.2320

±

0.00878

0.1212

±

0.00480

0.2704

±

0.00375

0.2582

±

0.00722

0.3983

±

0.00598

0.2469

±

0.00797

0.3597

±

0.00569

168-336

0.3840

±

0.01674

0.4393

±

0.01441

0.1641

±

0.01070

0.2364

±

0.00646

0.1287

±

0.00488

0.2808

±

0.00483

0.3206

±

0.01683

0.4515

±

0.01199

0.3040

±

0.00457

0.4049

±

0.00439

336-336

0.4377

±

0.04995

0.4654

±

0.02878

0.1546

±

0.00478

0.2332

±

0.00351

0.1496

±

0.01338

0.3039

±

0.01289

0.3559

±

0.03962

0.4779

±

0.02546

0.3149

±

0.01735

0.4145

±

0.01288

336-720

0.5502

±

0.05453

0.5438

±

0.02876

0.1747

±

0.01107

0.2536

±

0.00686

0.2092

±

0.01996

0.3659

±

0.01425

0.4936

±

0.03701

0.5592

±

0.02042

0.4358

±

0.01861

0.4937

±

0.01341

720-720

0.5927

±

0.10039

0.5742

±

0.04460

0.1543

±

0.00528

0.2441

±

0.00599

0.2178

±

0.02162

0.3694

±

0.01528

0.5243

±

0.04377

0.5745

±

0.02768

0.5701

±

0.10231

0.5491

±

0.03951

720-1440

0.8243

±

0.06568

0.6704

±

0.02131

0.1904

±

0.01783

0.2685

±

0.01319

0.3708

±

0.08923

0.4906

±

0.06185

0.9401

±

0.19202

0.7680

±

0.07649

0.5453

±

0.04814

0.5631

±

0.02233

1440-1440

0.9053

±

0.27275

0.7328

±

0.11057

0.1817

±

0.00859

0.2764

±

0.00909

0.4475

±

0.10894

0.5329

±

0.06986

0.7860

±

0.01219

0.6704

±

0.00613

0.5371

±

0.06814

0.5559

±

0.03300

1440-2880

1.1282

±

0.05767

0.8087

±

0.02547

0.2029

±

0.01118

0.3100

±

0.01305

0.9617

±

0.30428

0.8271

±

0.12771

2.0561

±

0.15722

1.1595

±

0.04848

0.9061

±

0.14939

0.7220

±

0.05635

Dlinear

168-168

0.2605

±

0.00138

0.3579

±

0.00117

0.1519

±

0.00017

0.2195

±

0.00023

0.1122

±

0.00081

0.2605

±

0.00106

0.2556

±

0.00080

0.3944

±

0.00091

0.2421

±

0.00302

0.3578

±

0.00240

168-336

0.3080

±

0.00167

0.3946

±

0.00149

0.1468

±

0.00012

0.2210

±

0.00017

0.1251

±

0.00086

0.2794

±

0.00101

0.2891

±

0.00143

0.4256

±

0.00082

0.2918

±

0.00149

0.3975

±

0.00112

336-336

0.2740

±

0.00539

0.3720

±

0.00491

0.1325

±

0.00035

0.2114

±

0.00053

0.1261

±

0.00103

0.2803

±

0.00130

0.2950

±

0.00116

0.4329

±

0.00099

0.2905

±

0.00198

0.3969

±

0.00170

336-720

0.3208

±

0.00328

0.4188

±

0.00280

0.1449

±

0.00126

0.2252

±

0.00135

0.1942

±

0.00074

0.3462

±

0.00067

0.4125

±

0.00304

0.5136

±

0.00225

0.3897

±

0.00344

0.4739

±

0.00166

720-720

0.3203

±

0.00199

0.4202

±

0.00150

0.1410

±

0.00065

0.2241

±

0.00069

0.1920

±

0.00181

0.3435

±

0.00205

0.3495

±

0.01694

0.4749

±

0.01135

0.3724

±

0.00649

0.4614

±

0.00467

720-1440

0.4923

±

0.01121

0.5473

±

0.00517

0.1639

±

0.00075

0.2412

±

0.00083

0.2952

±

0.00412

0.4370

±

0.00330

0.5037

±

0.03426

0.5645

±

0.01609

0.4406

±

0.00678

0.5264

±

0.00268

1440-1440

0.5146

±

0.01460

0.5615

±

0.00841

0.1599

±

0.00409

0.2411

±

0.00560

0.2200

±

0.00889

0.3714

±

0.00775

0.5176

±

0.02561

0.5734

±

0.01383

0.3147

±

0.01282

0.4417

±

0.00889

1440-2880

0.8355

±

0.01493

0.7193

±

0.00730

0.1550

±

0.00478

0.2472

±

0.00679

0.3773

±

0.02685

0.4794

±

0.01928

0.5053

±

0.04308

0.5584

±

0.02510

0.3197

±

0.00551

0.4533

±

0.00462

FiLM

168-168

0.2587

±

0.00032

0.3557

±

0.00029

0.1501

±

0.00067

0.2143

±

0.00139

0.1091

±

0.00033

0.2558

±

0.00035

0.2546

±

0.00141

0.3942

±

0.00161

0.2426

±

0.00037

0.3544

±

0.00036

168-336

0.3062

±

0.00030

0.3922

±

0.00014

0.1453

±

0.00033

0.2165

±

0.00071

0.1187

±

0.00058

0.2708

±

0.00068

0.2894

±

0.00718

0.4263

±

0.00566

0.2981

±

0.00036

0.3988

±

0.00033

336-336

0.2722

±

0.00027

0.3659

±

0.00034

0.1324

±

0.00346

0.2104

±

0.00641

0.1196

±

0.00045

0.2738

±

0.00069

0.2951

±

0.00226

0.4347

±

0.00188

0.2943

±

0.00107

0.3969

±

0.00112

336-720

0.3171

±

0.00083

0.4152

±

0.00072

0.1438

±

0.00028

0.2229

±

0.00039

0.1793

±

0.00112

0.3335

±

0.00091

0.4158

±

0.01033

0.5162

±

0.00636

0.4096

±

0.00062

0.4767

±

0.00069

720-720

0.3158

±

0.00086

0.4154

±

0.00061

0.1383

±

0.00039

0.2208

±

0.00053

0.1845

±

0.00103

0.3379

±

0.00086

0.4045

±

0.00302

0.5105

±

0.00184

0.3999

±

0.00326

0.4661

±

0.00329

720-1440

0.4730

±

0.00086

0.5336

±

0.00046

0.1638

±

0.00089

0.2448

±

0.00100

0.2949

±

0.00327

0.4388

±

0.00289

0.7166

±

0.00912

0.6628

±

0.00477

0.6360

±

0.00325

0.5997

±

0.00232

1440-1440

0.4849

±

0.00169

0.5429

±

0.00048

0.1602

±

0.00327

0.2437

±

0.00506

0.2294

±

0.01020

0.3759

±

0.00827

0.7446

±

0.01672

0.6590

±

0.00733

0.6002

±

0.00341

0.5880

±

0.00276

1440-2880

0.6847

±

0.00543

0.6493

±

0.00310

0.1744

±

0.00361

0.2693

±

0.00556

0.6834

±

0.01115

0.7096

±

0.00431

3.2835

±

0.10801

1.6030

±

0.02952

1.2605

±

0.00135

0.8805

±

0.00301

FEDformer

168-168

0.3028

±

0.01797

0.4020

±

0.01216

0.2469

±

0.02237

0.3479

±

0.02483

0.1284

±

0.00272

0.2826

±

0.00467

0.2844

±

0.01173

0.4285

±

0.01111

0.2583

±

0.00281

0.3774

±

0.00393

168-336

0.3522

±

0.01174

0.4394

±

0.00829

0.2426

±

0.00987

0.3449

±

0.01478

0.1271

±

0.00393

0.2810

±

0.00486

0.2961

±

0.00488

0.4355

±

0.00418

0.2909

±

0.00348

0.4030

±

0.00359

336-336

0.3378

±

0.00695

0.4303

±

0.00400

0.2339

±

0.01067

0.3365

±

0.01400

0.1252

±

0.00689

0.2794

±

0.00863

0.2884

±

0.01345

0.4314

±

0.01098

0.2791

±

0.00666

0.3984

±

0.00539

336-720

0.3813

±

0.00988

0.4634

±

0.00748

0.2987

±

0.03172

0.3976

±

0.02903

0.1534

±

0.01574

0.3178

±

0.01628

0.3425

±

0.02883

0.4656

±

0.02074

0.2648

±

0.00820

0.3915

±

0.01046

720-720

0.4023

±

0.03710

0.4769

±

0.02158

0.2667

±

0.02331

0.3685

±

0.02884

0.1386

±

0.00869

0.2995

±

0.01203

0.3275

±

0.01948

0.4534

±

0.01363

0.2416

±

0.00381

0.3728

±

0.00513

720-1440

0.4833

±

0.02229

0.5393

±

0.01414

0.2753

±

0.00622

0.3650

±

0.00502

0.1768

±

0.03601

0.3409

±

0.03216

0.3731

±

0.04086

0.4827

±

0.02391

0.2352

±

0.03670

0.3733

±

0.03472

1440-1440

0.5142

±

0.03440

0.5571

±

0.01701

0.2848

±

0.02063

0.3681

±

0.01120

0.3574

±

0.14083

0.4878

±

0.11818

0.3906

±

0.04614

0.4951

±

0.02924

0.2226

±

0.01285

0.3609

±

0.01098

1440-2880

3.9018

±

4.00564

1.5276

±

0.92236

0.2952

±

0.01400

0.3844

±

0.01182

0.4269

±

0.29204

0.5252

±

0.20346

1.7167

±

1.76025

0.9698

±

0.60094

0.2138

±

0.00950

0.3599

±

0.01022

Pyraformer

168-168

0.2651

±

0.01387

0.3802

±

0.01069

0.2979

±

0.01338

0.3815

±

0.01238

0.1534

±

0.01808

0.3287

±

0.01959

0.2746

±

0.01024

0.4080

±

0.00685

0.2144

±

0.02003

0.3451

±

0.02343

168-336

0.5392

±

0.26585

0.5271

±

0.10622

0.5838

±

0.02841

0.5652

±

0.01840

0.1665

±

0.00781

0.3419

±

0.01014

0.2392

±

0.02900

0.3834

±

0.02057

0.2594

±

0.04642

0.3833

±

0.04056

336-336

0.2994

±

0.01279

0.4030

±

0.01117

0.4703

±

0.06731

0.4964

±

0.05265

0.1408

±

0.01032

0.3087

±

0.01180

0.2610

±

0.03612

0.4010

±

0.02955

0.2310

±

0.02787

0.3591

±

0.02687

336-720

0.4856

±

0.06663

0.5243

±

0.03642

0.5235

±

0.10386

0.5292

±

0.08052

0.3984

±

0.23533

0.5202

±

0.16798

0.2341

±

0.00798

0.3818

±

0.00520

0.3241

±

0.06639

0.4300

±

0.04847

720-720

0.3115

±

0.02065

0.4218

±

0.01610

0.4811

±

0.05721

0.4962

±

0.04154

0.1563

±

0.01867

0.3253

±

0.01669

0.2795

±

0.04548

0.4151

±

0.03349

0.2378

±

0.06668

0.3684

±

0.05805

720-1440

0.3250

±

0.01497

0.4332

±

0.01241

0.4463

±

0.04505

0.4609

±

0.03322

0.1666

±

0.03171

0.3315

±

0.02987

0.2952

±

0.0432

0.4336

±

0.02947

0.6810

±

0.30460

0.6352

±

0.18544

1440-1440

0.4895

±

0.04971

0.5280

±

0.03132

0.4710

±

0.01399

0.4916

±

0.00751

0.3487

±

0.10751

0.4866

±

0.08011

0.2946

±

0.02847

0.4316

±

0.01817

0.2401

±

0.04767

0.3777

±

0.04109

1440-2880

0.4320

±

0.01920

0.5161

±

0.01283

0.5165

±

0.02708

0.5305

±

0.01786

0.5857

±

0.12854

0.6760

±

0.09034

0.3345

±

0.01392

0.4544

±

0.01035

0.1852

±

0.00663

0.3332

±

0.00954

Autoformer

168-168

0.3496

±

0.01079

0.4337

±

0.01014

0.2378

±

0.01295

0.3490

±

0.01273

0.1318

±

0.00332

0.2872

±

0.00515

0.2903

±

0.01461

0.4326

±

0.01218

0.2670

±

0.00243

0.3813

±

0.00317

168-336

0.4733

±

0.05218

0.5120

±

0.03357

0.2683

±

0.01140

0.3803

±

0.01102

0.1315

±

0.00670

0.2878

±

0.00682

0.4447

±

0.30613

0.4964

±

0.12815

0.2990

±

0.00389

0.4096

±

0.00475

336-336

0.5153

±

0.12322

0.5304

±

0.07432

0.2460

±

0.01621

0.3567

±

0.02022

0.1384

±

0.01765

0.2959

±

0.01631

0.2805

±

0.01655

0.4255

±

0.01391

0.3066

±

0.01615

0.4162

±

0.01489

336-720

0.5045

±

0.02785

0.5393

±

0.01945

0.2849

±

0.03404

0.3956

±

0.03523

0.1928

±

0.03373

0.3450

±

0.03092

0.3372

±

0.01763

0.4625

±

0.01084

0.3468

±

0.01639

0.4592

±

0.01022

720-720

0.9639

±

0.26411

0.7520

±

0.10858

0.2959

±

0.02180

0.4045

±

0.01861

0.2388

±

0.04468

0.3869

±

0.03952

0.4668

±

0.10896

0.5477

±

0.06613

0.4309

±

0.04345

0.5085

±

0.02459

720-1440

1.4957

±

0.49225

0.9533

±

0.17135

0.3104

±

0.03672

0.4095

±

0.03276

0.3298

±

0.03720

0.4741

±

0.02983

0.5633

±

0.09564

0.5996

±

0.04671

0.8599

±

0.43154

0.7064

±

0.19138

1440-1440

1.7873

±

1.12827

1.0283

±

0.36295

0.2970

±

0.02378

0.3999

±

0.02575

0.4531

±

0.12054

0.5507

±

0.09597

0.8029

±

0.34886

0.7140

±

0.15332

0.9766

±

0.38596

0.7739

±

0.10604

1440-2880

1.2867

±

0.32261

0.8878

±

0.12519

0.3035

±

0.02038

0.3982

±

0.02444

1.3566

±

1.00518

0.9235

±

0.4592

4.1031

±

2.27321

1.7198

±

0.61623

1.7465

±

0.51402

1.0962

±

0.18964

Informer

168-168

0.3779

±

0.02565

0.4594

±

0.01559

0.3363

±

0.03784

0.3994

±

0.02525

0.1563

±

0.01273

0.3299

±

0.01299

0.3764

±

0.01781

0.4863

±

0.00921

0.2639

±

0.02557

0.3926

±

0.02566

168-336

0.5037

±

0.08433

0.5301

±

0.04791

0.5891

±

0.11257

0.5608

±

0.05492

0.1663

±

0.03122

0.3335

±

0.03102

0.3364

±

0.02216

0.4583

±

0.01619

0.2798

±

0.02688

0.4061

±

0.02242

336-336

0.4591

±

0.03057

0.4991

±

0.01943

0.5447

±

0.03647

0.5384

±

0.01976

0.1648

±

0.04980

0.3291

±

0.04794

0.3709

±

0.02624

0.4785

±

0.01520

0.2898

±

0.01358

0.4129

±

0.01501

336-720

0.6545

±

0.12350

0.5975

±

0.05506

1.2044

±

0.28757

0.8254

±

0.10672

0.1522

±

0.01115

0.3203

±

0.01333

0.3572

±

0.01765

0.4675

±

0.00646

0.2483

±

0.00848

0.3778

±

0.00462

720-720

0.4850

±

0.01891

0.5238

±

0.00888

1.2954

±

0.57549

0.9205

±

0.19654

0.1595

±

0.01501

0.3259

±

0.01221

0.3585

±

0.01299

0.4699

±

0.00700

0.3545

±

0.13325

0.4569

±

0.08858

720-1440

0.5064

±

0.04967

0.5317

±

0.02899

0.7614

±

0.17528

0.6469

±

0.09409

0.1378

±

0.00853

0.3051

±

0.00856

0.4025

±

0.04622

0.4991

±

0.02220

0.2466

±

0.00877

0.3849

±

0.01187

1440-1440

0.7247

±

0.23436

0.6292

±

0.10907

0.7375

±

0.04517

0.6414

±

0.02969

0.1430

±

0.00927

0.3156

±

0.00902

0.3484

±

0.01264

0.4786

±

0.00480

0.2556

±

0.00559

0.3969

±

0.00630

1440-2880

0.6152

±

0.25466

0.5953

±

0.13194

0.9849

±

0.23417

0.7618

±

0.10144

0.3177

±

0.10572

0.4733

±

0.08814

0.3335

±

0.00947

0.4482

±

0.00366

0.2126

±

0.01052

0.3600

±

0.01317

Transformer

168-168

0.3036

±

0.01240

0.4068

±

0.01204

1.5204

±

0.76091

0.9594

±

0.35768

0.1504

±

0.01406

0.3257

±

0.01774

0.3043

±

0.02208

0.4365

±

0.01397

0.2200

±

0.00977

0.3438

±

0.00915

168-336

0.3583

±

0.01874

0.4435

±

0.01628

0.6953

±

0.28867

0.6015

±

0.15680

0.1599

±

0.01629

0.3324

±

0.01735

0.3662

±

0.02673

0.4671

±

0.01450

0.2230

±

0.00914

0.3488

±

0.01073

336-336

0.5771

±

0.27509

0.5643

±

0.15434

0.8482

±

0.53327

0.6424

±

0.21839

0.1438

±

0.00533

0.3121

±

0.00606

0.3218

±

0.03005

0.4412

±

0.02106

0.2308

±

0.00771

0.3556

±

0.01044

336-720

0.4368

±

0.05110

0.4920

±

0.03146

0.7320

±

0.22311

0.6233

±

0.14229

0.1644

±

0.02587

0.3304

±

0.02911

0.3582

±

0.14008

0.4629

±

0.09578

0.2334

±

0.01492

0.3570

±

0.01743

720-720

0.3992

±

0.04794

0.4640

±

0.02729

1.1963

±

0.74703

0.8271

±

0.35161

0.1730

±

0.00988

0.3414

±

0.01075

0.3087

±

0.02578

0.4320

±

0.01658

0.2463

±

0.00604

0.3722

±

0.01029

720-1440

0.4030

±

0.03837

0.4797

±

0.02257

0.9876

±

0.19880

0.7445

±

0.10064

0.1905

±

0.03701

0.3555

±

0.03645

0.3712

±

0.01640

0.4805

±

0.00732

0.2188

±

0.01761

0.3512

±

0.01479

1440-1440

0.5531

±

0.05384

0.5524

±

0.02729

0.7430

±

0.05136

0.6492

±

0.02996

0.1972

±

0.02466

0.3630

±

0.02502

0.3797

±

0.02330

0.4818

±

0.01429

0.2610

±

0.01388

0.3823

±

0.01305

1440-2880

0.5243

±

0.06259

0.5460

±

0.03610

0.6000

±

0.07314

0.5877

±

0.05861

0.3495

±

0.08217

0.4911

±

0.06523

0.3737

±

0.05141

0.4787

±

0.03206

0.1993

±

0.00884

0.3436

±

0.00934

sets, when their variances are not significantly different, it indicates that the data fluctuations in the two sets are similar. In this case, the difference in the data distribution is not significant, and selecting norm as 0 is reasonable. However, when the variance difference between the training and validation sets is relatively large (approximately twice or half), there is a significant difference in the data distribution between the two sets, and norm should be 1 to re-normalize the input of the model. It should be noted that in the Weather dataset, there are negative values present, which makes its mean close to 0, resulting in a large difference between its variance and mean. However, this is reasonable. For the Traffic dataset, there are no negative values present, so even if the variances of its training and validation sets are similar, its significant fluctuations can be observed when combined with its mean. This was also demonstrated by setting the norm value to 1 during the model training process.

The impact of C The choice of C for the model represents the determination of the sequence period. Due to the different original acquisition granularities of each dataset, in order to ensure that they contain the same semantic information on the same task, in this paper, our experiments are conducted by aggregating them into one hour. To facilitate the discovery of long-term repetitive patterns in the sequence, we set the C to 12, 24, 48, and the specific experimental results are shown in Table 11. From Table 11, we can clearly see that for sequences with time steps in hours, using a period of 24 hours (1 day) for division yields better results. This is because in WIT, information is transmitted horizontally at the granularity of hours, while information is transmitted vertically at the granularity of days. This approach can more fully capture the long- and short-term repetitive patterns and global/local correlations hidden in the time series data.

I.2 A detailed explanation of time and memory consumption

The efficiency of the model is crucial, as even if the model's efficiency is high, it is limited if it does not have good predictive performance. Therefore, in this paper, we only compared our proposed method with methods that have good or the latest results. It should be noted that, for example, Transformer [Vaswani et al., 2017], Informer [Zhou et al., 2021] and Autoformer [Wu et al., 2021] have been shown in previous works to have higher complexity than FiLM [Zhou et al., 2022b], so we only compared our proposed method with FiLM, rather than comparing with Transformer, Informer and Autoformer again.

Memory Usage As shown in Figure 5(a) and (c), WITRAN has good memory usage with the prolonging the input length and output length. For a fair comparison, we fix the experimental settings of all methods, where we fix the input length as 720 and prolong the output length. Moreover, we fix the output length as 720 and prolong the input length. And for each model, to achieve good performance, it is important to have a sufficiently large search space to select the most suitable parameters for the model, in order to achieve the best possible results. Therefore, we set all public parameters to their upper limits to test the memory usage of each model. To allow as many models as possible to be trained, we uniformly set the batch size to 8 for testing in this section. It should be noted that if a point is not shown on the graph, it indicates that the model encountered an "out of memory" (OOM) error at that particular configuration of input sequence length and output length.

As for TimesNet, due to its large memory usage, it cannot be trained even with a batch size of 1. Therefore, we did not include TimesNet in the Figure 5. From Figure 5(a), we can observe that when the input sequence is fixed, the memory usage of WITRAN and WIT is not significantly different since they both have the same prediction module. Additionally, we found that as the output length increases, the memory usage of WITRAN and WIT also increases linearly, but our proposed method still has the lowest memory usage. From Figure 5(c), we can observe that the memory usage of WITRAN increases very slowly with the increase in sequence length, and the memory usage of WITRAN is even lower than that of WIT. This is because WITRAN does not need to retain a large number of intermediate storage variables in the computation of each slice, while WIT needs to store the intermediate variables for the entire sequence. Through comparison with other methods, we can clearly see the advantages of WITRAN on memory consumption.

Training Speed As shown in Figure 5(b) and (d), WITRAN has a faster training speed than others with the prolonging the input length and output length. To ensure a fair comparison of the performance of each model, we fixed the experimental settings, which were the same as the settings used for testing memory usage. The experiment is performed on ECL dataset. Due to our extremely low space complexity, we followed the experimental settings used in FiLM [Zhou et al., 2022b] for comparing the training Speed. To fully utilize the GPU memory, our model can increase the batch size from 8 to 32 under the experimental settings mentioned above. Similarly, FiLM can also increase the batch size = 32. However, we still have the fastest training speed, and due to the role of RAN mainly being on the input sequence, our advantage can be clearly seen from Figure 5(d), which is related to the square root of the sequence length. Through comparison with other methods, we can clearly see the advantages of WITRAN on time consumption.

J Showcases of Main Results

As shown in Figure 10 to Figure 49, we plot the forecasting results from the test set of all datasets for comparison. For the long-range forecasting task, we chose two suboptimal models, FiLM and Pyraformer. As for the ultra-long-range forecasting task, we also chose two suboptimal models, Pyraformer and TimesNet. Our model WITRAN gives the best performance among different models. Moreover, WITRAN is significantly better at modeling global and local correlations, as well as discovering long- and short-term repetitive patterns in the time series than other models.

WITRAN 168-168

GroundTruth

Prediction

50

100

150

200

250

300

350

FiLM 168-168

step

GroundTruth

Prediction

50

100

150

200

250

300

350

step

Pyraformer 168-168

GroundTruth

Prediction

50

100

150

200

250

300

OSE

step

WITRAN 168-336

GroundTruth

Prediction

100

200

00E

400

500

FiLM 168-336

step

GroundTruth

Prediction

100

200

300

400

500

Pyraformer 168-336

step

GroundTruth

Prediction

100

200

000

400

500

step

WITRAN 336-336

GroundTruth

Prediction

100

300

400

500

600

700

FiLM 336-336

step

Prediction

100

200

300

400

500

600

700

Pyraformer 336-336

step

GroundTruth

Prediction

100

200

000

400

500

009

700

step

WITRAN 336-720

GroundTruth

Prediction

400

600

800

1000

FiLM 336-720

step

GroundTruth

Prediction

200

400

600

800

1000

Pyraformer 336-720

step

GroundTruth

Prediction

200

400

600

800

1000

step

WITRAN 720-720

GroundTruth

Prediction

200

400

600

800

1000

1200

1400

FiLM 720-720

step

GroundTruth

Prediction

200

400

600

800

1000

1200

1400

step

Pyraformer 720-720

GroundTruth

Prediction

200

400

009

008

1000

1200

1400

step

WITRAN 720-1440

Prediction

GroundTruth

00S

1000

1500

2000

Pyraformer 720-1440

step

GroundTruth

Prediction

500

1000

1500

2000

step

TimesNet 720-1440

GroundTruth

Prediction

500

1000

1500

2000

step

WITRAN 1440-1440

GroundTruth

Prediction

500

1000

1500

2000

2500

3000

Pyraformer 1440-1440

step

GroundTruth

Prediction

500

1000

step

1500

2000

2500

3000

TimesNet 1440-1440

GroundTruth

Prediction

500

1000

1500

2000

2500

0000

step

WITRAN 1440-2880

GroundTruth

Prediction

1000

2000

3000

4000

Pyraformer 1440-2880

step

Prediction

1000

2000

3000

4000

step

TimesNet 1440-2880

GroundTruth

Prediction

1000

2000

3000

4000

tep

WITRAN 168-168

GroundTruth

Prediction

50

100

150

200

250

300

350

FiLM 168-168

step

GroundTruth

Prediction

50

100

150

200

250

300

350

step

Pyraformer 168-168

GroundTruth

Prediction

50

100

150

200

250

300

OSE

step

WITRAN 168-336

Prediction

GroundTruth

100

200

00E

400

500

step

FiLM 168-336

GroundTruth

Prediction

100

200

300

400

500

step

Pyraformer 168-336

GroundTruth

Prediction

100

200

00

400

500

step

WITRAN 336-336

step

GroundTruth

Prediction

100

300

400

500

600

700

FiLM 336-336

step

GroundTruth

Prediction

100

200

300

400

500

600

700

Pyraformer 336-336

step

GroundTruth

Prediction

100

300

500

600

700

WITRAN 336-720

GroundTruth

Prediction

wW

400

600

800

1000

FiLM 336-720

step

GroundTruth

Prediction

MW

200

400

600

800

1000

Pyraformer 336-720

step

GroundTruth

Prediction

200

400

600

800

1000

step

WITRAN 720-720

step

GroundTruth

Prediction

200

00

600

008

1000

1200

1400

FiLM 720-720

step

GroundTruth

Prediction

200

400

600

800

1000

1200

1400

Pyraformer 720-720

step

GroundTruth

Prediction

200

400

600

008

1000

1200

1400

WITRAN 720-1440

step

GroundTruth

Prediction

00S

1000

1500

2000

Pyraformer 720-1440

step

0T

Prediction

500

1000

1500

2000

step

TimesNet 720-1440

GroundTruth

Prediction

500

1000

1500

2000

WITRAN 1440-1440

step

GroundTruth

Prediction

500

1000

1500

2000

2500

3000

Pyraformer 1440-1440

step

Prediction

500

1000

1500

2000

2500

3000

TimesNet 1440-1440

step

GroundTruth

Prediction

500

1000

1500

2000

2500

3000

WITRAN 1440-2880

GroundTruth

Prediction

1000

2000

3000

4000

Pyraformer 1440-2880

step

10

Prediction

1000

2000

3000

4000

TimesNet 1440-2880

step

GroundTruth

Prediction

1000

2000

3000

4000

step

WITRAN 168-168

0.4

GroundTruth

Prediction

8'0

W

1.2

1.4

1.6

100

150

200

250

00E

350

FiLM 168-168

step

0.4

GroundTruth

Prediction

0.6

0.8

1.6

100

150

200

250

300

350

Pyraformer 168-168

step

GroundTruth

Prediction

0.6

0.8

N

1.0

1.4

1.6

50

100

150

200

250

300

50

step

WITRAN 168-336

0.0

GroundTruth

Prediction

0.2

0.4

1.2

1.4

100

200

000

400

500

FiLM 168-336

step

00

0.2

WM

Prediction

0.4

0.6

0.8

1.4

1.6

100

200

300

400

500

step

Pyraformer 168-336

0.2

0.0

MWW

Prediction

GroundTruth

0.4

0.6

80-

1.2

100

200

300

400

500

step

WITRAN 336-336

GroundTruth

Prediction

1.0

1.2

1.4

2.0

2.2

2.4

100

200

300

400

500

600

700

FiLM 336-336

step

0.8

GroundTruth

1.0

Prediction

1.2

1.4

1.6

2.0

2.2

2.4

100

200

300

400

500

600

700

step

Pyraformer 336-336

0.8

GroundTruth

1.0

Prediction

1.2

1.4

1.8

2.0

2.2

2.4

100

200

300

400

00S

009

700

step

WITRAN 336-720

0.25

GroundTruth

0.00

Prediction

0.25

0.50

1.00

1.25

1.50

1.75

200

400

600

800

1000

FiLM 336-720

step

0.25

GroundTruth

00°0

Prediction

W33

0.25

0.50

1.00

1.25

1.50

1.75

200

400

600

800

1000

Pyraformer 336-720

step

SZ0

GroundTruth

0.00

Prediction

0.25

AMy

0.50

.75-

1.00

1.25

1.50

1.75

0

200

400

600

008

1000

step

WITRAN 720-720

1.0

Prediction

GroundTruth

1.2

1.4

1.8

2.0

2.2

2.4

200

400

600

008

1000

1200

1400

FiLM 720-720

step

1.0

Prediction

1.2

1.4

1.8

2.0

2.2

2.4

200

400

600

800

1000

1200

1400

step

Pyraformer 720-720

GroundTruth

1.0

Prediction

1.2

1.4

1.8

2.0

2.2

2.4

200

400

009

008

1000

1200

1400

step

step

WITRAN 720-1440

0.0

GroundTruth

Prediction

0.5

1.5

2.0

00S

1000

1500

2000

Pyraformer 720-1440

step

0.0

GroundTruth

Prediction

0.5

1.5

2.0

500

1000

step

1500

2000

TimesNet 720-1440

00

GroundTruth

M

Prediction

0.5

1.5

2.0

500

1000

1500

2000

R

WITRAN 1440-1440

step

0.5

GroundTruth

Prediction

1.0

2.0

2.5

500

1000

1500

2000

2500

3000

Pyraformer 1440-1440

step

0.5

Prediction

GroundTruth

1.0

2.0

2.5

500

1000

step

1500

2000

2500

3000

TimesNet 1440-1440

0.5

Prediction

GroundTruth

1.0

2.0

2.5

500

1000

1500

2000

2500

0000

WITRAN 1440-2880

0.0

GroundTruth

Prediction

0.5

1.0

1.5

2.0

2.5

1000

2000

3000

4000

Pyraformer 1440-2880

step

00

GroundTruth

Prediction

0.5

1.0

1.5

2.0

2.5

3.0

1000

2000

3000

4000

step

TimesNet 1440-2880

0.0

GroundTruth

Prediction

0.5

1.0

2.0

2.5

O'E

1000

2000

3000

4000

step

WITRAN 168-168

GroundTruth

Prediction

0.5

0.0

0.5

1.0

1.5

50

100

150

200

250

300

350

FiLM 168-168

step

GroundTruth

Prediction

0.5

NMMAMY

0.0

1.0

1.5

50

100

150

200

250

300

350

Pyraformer 168-168

step

GroundTruth

Prediction

0.0

0.5

1.0

1.5

50

100

150

200

250

300

OSE

step

WITRAN 168-336

GroundTruth

Prediction

0.5

2.0

2.5

100

200

00E

400

500

FiLM 168-336

step

0.5

GroundTruth

Prediction

1.0

2.0

2.5

100

200

300

400

500

Pyraformer 168-336

step

GroundTruth

0.5

Prediction

1.0

2.0

2.5

100

200

300

400

500

step

WITRAN 336-336

step

GroundTruth

1.0

Prediction

0.5

anjea

0.0

0.5

1.0

1.5

100

200

300

400

500

600

700

FiLM 336-336

step

GroundTruth

1.0

Prediction

S0

0.0

MM

0.5

1.0

100

200

300

400

500

600

700

Pyraformer 336-336

step

GroundTruth

1.0

Prediction

0.5

anjeA

0.0

1.0

1.5

100

300

500

009

700

WITRAN 336-720

1.0

0.0

0.5

1.0

GroundTruth

1.5

Prediction

200

400

600

008

1000

FiLM 336-720

step

1.5

1.0

0.5

0.0

0.5

1.0

GroundTruth

1.5

Prediction

0

200

400

600

800

1000

Pyraformer 336-720

step

1.5

1.0

0.5

0.5

1.0

GroundTruth

1.5

Prediction

0

200

400

600

008

1000

step

WITRAN 720-720

GroundTruth

Prediction

0.5

1.0

1.5

~~ML

2.0

2.5

200

400

600

800

1000

1200

1400

FiLM 720-720

step

0.5

GroundTruth

Prediction

0.0

0.5

1.0

1.5

2.0

2.5

200

400

600

800

1000

1200

1400

Pyraformer 720-720

step

0.5

GroundTruth

Prediction

0.0

-1.0-

1.5

2.0

2.5

200

400

009

008

1000

1200

1400

step

WITRAN 720-1440

2.5

GroundTruth

2.0

Prediction

1.5

1.0

MMM

00

0.5

1.0

1.5

00S

1000

1500

2000

Pyraformer 720-1440

step

2.5

GroundTruth

2.0

Prediction

1.5

1.0

0.5

1.0

1.5

500

1000

1500

2000

step

TimesNet 720-1440

2.5

GroundTruth

2.0

Prediction

1.5

1.0

0.5

00

0.5

1.0

1.5

500

1000

1500

2000

step

WITRAN 1440-1440

1.5

GroundTruth

1.0

Prediction

0.5

0.0

0.5

1.0

1.5

2.0

2.5

500

1000

1500

2000

2500

3000

Pyraformer 1440-1440

step

1.5

GroundTruth

1.0

Prediction

0.5

0.0

-0.5

1.0

1.5

2.0

2.5

500

1000

1500

2000

2500

3000

step

TimesNet 1440-1440

1.5

GroundTruth

1.0

Prediction

MMMM

0.0

1.0

1.5

0z

2.5

500

1000

1500

2000

2500

0000

step

WITRAN 1440-2880

GroundTruth

Prediction

1000

2000

3000

4000

Pyraformer 1440-2880

step

GroundTruth

Prediction

1000

2000

step

3000

4000

TimesNet 1440-2880

GroundTruth

Prediction

1000

2000

3000

4000

step

WITRAN 168-168

step

GroundTruth

0.0

Prediction

0.5

1.0

1.5

2.0

2.5

50

100

150

200

250

300

350

FiLM 168-168

step

GroundTruth

0.0

Prediction

0.5

（）

2.0

2.5

50

100

150

200

250

300

350

step

Pyraformer 168-168

0.0

Prediction

GroundTruth

0.5

1.5

2.0

2.5

100

150

200

250

300

WITRAN 168-336

GroundTruth

1.0

Prediction

WWWAAN

0.5

W

0°0

0.5

1.0

100

200

00E

400

500

FiLM 168-336

step

GroundTruth

1.0

Prediction

0.5

MW

0.5

1.0

100

200

300

400

500

Pyraformer 168-336

step

GroundTruth

1.0

Prediction

0.5

anjen

0.0

0.5

1.0

100

200

300

400

500

step

WITRAN 336-336

0.0

0.5

Prediction

GroundTruth

0.5

1.0

1.5

2.0

2.5

3.0

3.5

100

200

300

400

00S

009

700

FiLM 336-336

step

0.5

0.0

0.5

1.0

1.5 -

2.0

2.5

3.0

GroundTruth

Prediction

3.5

0

100

200

300

400

500

600

700

step

Pyraformer 336-336

0.5

GroundTruth

0.0

Prediction

0.5

1.0

1.5 -

2.0

2.5

3.0

3.5

100

200

300

400

500

009

700

step

WITRAN 336-720

GroundTruth

1.0

Prediction

0.5

0.0

anjeA

s'0

1.0

1.5

2.0

200

400

600

800

1000

FiLM 336-720

step

GroundTruth

1.0

Prediction

0.0

0.5

n

1.0

1.5

2.0

200

400

600

800

1000

step

Pyraformer 336-720

GroundTruth

1.0

Prediction

0.5

0.0

anjen

1.0

1.5

2.0

200

400

600

800

1000

step

WITRAN 720-720

GroundTruth

2.0

Prediction

1.5

1.0

0.5

00

0.5

200

400

600

800

1000

1200

1400

FiLM 720-720

step

GroundTruth

2.0

Prediction

1.5

1.0

value

0.5

0.0

0.5

1.0

200

400

600

800

1000

1200

1400

step

Pyraformer 720-720

GroundTruth

2.0

Prediction

1.5

1.0

0.5

0.0

0.5

1.0

200

400

009

008

1000

1200

1400

step

WITRAN 720-1440

2.5

GroundTruth

Prediction

2.0

1.5

1.0

0.5

0.0

0.5

00S

1000

1500

2000

Pyraformer 720-1440

step

2.5

GroundTruth

2.0

Prediction

1.5

1.0

0.5

0°0

0.5

500

1000

1500

2000

step

TimesNet 720-1440

2.5

GroundTruth

Prediction

2.0

1.5

1.0

00

0.5

500

1000

1500

2000

step

WITRAN 1440-1440

Prediction

500

1000

1500

2000

2500

3000

Pyraformer 1440-1440

step

Prediction

500

1000

1500

2000

2500

3000

step

TimesNet 1440-1440

GroundTruth

Prediction

500

1000

1500

2000

2500

DOOE

step

WITRAN 1440-2880

GroundTruth

Prediction

ww

1000

2000

3000

4000

Pyraformer 1440-2880

step

GroundTruth

Prediction

1000

2000

step

3000

4000

TimesNet 1440-2880

GroundTruth

Prediction

1000

2000

3000

4000

Algorithm 3 Recurrent Acceleration Network (RAN)

Algorithm 3 Recurrent Acceleration Network (RAN)

Peer reviews:
comments: SUMMARY:
To capture semantic information and repetitive patterns concurrently, the authors propose WIT framework. By utilizing bi-granular information transmission and HVGSU, the framework can model the inherent repetitive pattern as well as correlation of time series. The author also use a generic RAN to reduce time complexity. Several experiments conducted by te authors demonstrate that their framework can outperform in time series forecasting.

STRENGTHS:
S1: The logic and presentation of the essay is easy to follow.
S2: The problem they focus on is very essential as well as appealing.
S3: Experiments reveal the performance of the authors’ model outperforms existing SOTA baselines.

WEAKNESSES:
W1: The authors may include more experiments such as robustness check to further evaluate the performance of the framework.
W2: There are some typos and minor errors that do not influence the understanding of this work, which should be carefully checked. For instance, line 29, semantic information include-> semantic information includes

QUESTIONS:
Q1: Can authors include more experiments to further evaluate the performance of the framework.

LIMITATIONS:
Not Applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the long-range time series forecasting problem. An interesting model, Water-wave Information Transmission and Recurrent Acceleration Network is proposed, which captures both short- and long-term recurrent patterns via bi-granular information transmission. The proposed model also captures global and local correlations using horizontal and vertical information transmission. This model is an interesting modification of the RNN network, and it significantly outperforms many transformer-based models.\n\nSTRENGTHS:\nS1. This paper reviews the shortcomings of transformer models and makes modifications to the RNN structure. \n\nS2. A bi-granular information transmission is proposed to capture short- and long-term recurrent patterns, which is easy to understand and does not require additional methods (e.g., FFT) to extract periodicity.\n\nS3. A recurrent acceleration network is proposed, which reduces the time complexity to O(√L) while maintaining the memory complexity at O(L).\n\nWEAKNESSES:\nW1. Some previous RNN-based models, such as ConvLSTM [1]/ PredRNN [2]/ PredRNN++ [3], have yet to be compared.\n\n[1] Shi X, Chen Z, Wang H, et al. Convolutional LSTM network: A machine learning approach for precipitation nowcasting[J]. Advances in neural information processing systems, 2015, 28.  \n[2] Wang Y, Long M, Wang J, et al. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms[J]. Advances in neural information processing systems, 2017, 30.  \n[3] Wang Y, Gao Z, Long M, et al. Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning[C]//International Conference on Machine Learning. PMLR, 2018: 5123-5132.\n\nW2. Some minor problems:\n1) It is recommended to enlarge Figure 1(i) and make horizontal lines clearer.  \n2) It is recommended to unify TF_{en} and TFE_{de}.\n\nQUESTIONS:\nQ1. Please explicitly explain the advantages of the WITRAN model over other forecasting models from the perspective of information transmission process.\n\nQ2. Please give the adaptation and limitation of the WITRAN model.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN) framework to model dependencies in a long historical time series. Inspired by Timesnet, the WITRAN introduces a water-wave information transmission strategy to model temporal information. This is implemented by the proposed Horizontal Vertical Gated Selective Unit (a kind of recurrent unit and  similar to GRU).\n\nSTRENGTHS:\n1) This paper is well-written\n2) Figures illustrate model details in a good way\n3) This paper studies the single-channel case, though most existing papers consider evaluations on multivariate time series. This makes sense as this can better evaluate the ability to learn temporal dependencies in time series.\n4) Advanced baselines such as MICN, timesnet, patchTST, and film are included.\n\nWEAKNESSES:\n1) Although using the water-wave structure is new to me, I am still curious about why this design is needed in long-range time series forecasting. Especially, are there any special temporal structures that existing models in Figure a-g cannot handle?\n2) The water-wave structure is a strategy to model time series dependencies. However, it is unclear how to decide the number of R and C and what are their effects on the final prediction. \n3) In Figure 10 (appendix), the results of the proposed are much worse than other baselines: FiLM and Pyraformer. But you claimed that “our model WITRAN gives the best performance among different models” in Section J. Moreover, other plots in Figure 11-31 did not demonstrate the effectiveness of WITRAN. \n4) The reference part is missing.\n\nQUESTIONS:\nQuestions are as follows:\n- Are there any special temporal structures that existing models in Figure a-g cannot handle?\n- How to decide the number of R and C \n- How to set the norm in (1).\n\nLIMITATIONS:\nSome limitations in terms of implementation have been mentioned in the Conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of long-range time series forecasting problem and proposes a WITRAN model. The paper analyzes and compares previous forecasting methods from the perspective of information transmission process and design a water-wave information transmission mechanism, which simultaneously capture global and local correlations via a bi-granular information transmission. And a recurrent acceleration network is designed to reduce the computation complexity.\n\nSTRENGTHS:\n1) WITRAN is an ingeniously crafted framework, capable of simultaneously capturing two aspects of semantic information within long-range time series - global-local correlations and both long- and short-term periodic patterns. The WIT includes two innovative modules, namely, HVGSU and GSU, which augment forecasting accuracy while enhancing explainability. Additionally, the RAN segment is designed to markedly improve efficiency. Finally, the WIT and RAN segments are seamlessly integrated together. \n\n2) The paper presents an exhaustive theoretical substantiation of the RAN segment's operating efficiency, shedding light on its capacity to significantly boost model efficiency.\n\n3) WITRAN exhibits superior performance, as confirmed by a series of comprehensive experiments with equitable settings across all baselines. WITRAN outperforms SOTA methods in both long-range and ultra-long-range forecasting tasks. Moreover, the incorporation of RAN indeed enhances computational efficiency and reduces memory footprint, in alignment with theoretical proof.\n\nWEAKNESSES:\n1) It seems that the proposed model is suitable for the time series that naturally contains bi-granular periodicity, such as traffic flow with daily and weekly periodicity. So the generalization of the proposed model is unclear.\n\n2) In the experimental section, many Transformer-based methods are taken as baselines, but few RNN-based methods are not compared. Since the proposed model is RNN-based, so it is necessary to add more RNN-based baselines.\n\nQUESTIONS:\n1) If the five datasets have enough representativeness among all kinds of time series?\n\n2) The material included in Appendix A appears to be rather elementary for the target audience in this domain. I would be intrigued to understand the rationale behind the authors' decision to incorporate this content.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}