WITRAN: Water-wave Information Transmission and Recurrent Acceleration Network for Long-range Time Series Forecasting
Authors: Yuxin Jia
Youfang Lin
Xinyan Hao
Yan Lin
Shengnan Guo
Huaiyu Wan
Venue: NeurIPS 2023 spotlight (2023)
Keywords: long-range time series forecasting
information transmission
long- and short-term repetitive patterns
global and local correlations
TL;DR: We propose a novel WITRAN model, with the best performance and the lowest complexity for long-range time series forecasting.

Abstract: Capturing semantic information is crucial for accurate long-range time series forecasting, which involves modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. Previous works have partially addressed these issues separately, but have not been able to address all of them simultaneously. Meanwhile, their time and memory complexities are still not sufficiently low for long-range forecasting. To address the challenge of capturing different types of semantic information, we propose a novel Water-wave Information Transmission (WIT) framework. This framework captures both long- and short-term repetitive patterns through bi-granular information transmission. It also models global and local correlations by recursively fusing and selecting information using Horizontal Vertical Gated Selective Unit (HVGSU). In addition, to improve the computing efficiency, we propose a generic Recurrent Acceleration Network (RAN) which reduces the time complexity to $\mathcal{O}(\sqrt{L})$ while maintaining the memory complexity at $\mathcal{O}(L)$. Our proposed method, called Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN), outperforms the state-of-the-art methods by 5.80% and 14.28% on long-range and ultra-long-range time series forecasting tasks respectively, as demonstrated by experiments on four benchmark datasets. The code is available at: https://github.com/Water2sea/WITRAN.

WITRAN: Water-wave Information Transmission and Recurrent Acceleration Network for Long-range Time Series Forecasting
Yuxin Jia School of Computer and Information Technology, Beijing Jiaotong University, China Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing, China (yuxinjia, yflin, xinyanhao, ylincs, guoshn, hywan] jujtibjjt...... edu on
Youfang Lin Xinyan Hao Yan Lin Shengnan Guo Huaiyu Wan'
Abstract
Capturing semantic information is crucial for accurate long-range time series forecasting, which involves modeling global and local correlations, as well as dis- covering long- and short-term repetitive patterns. Previous works have partially ad- dressed these issues separately, but have not been able to address all of them simul- taneously. Meanwhile, their time and memory complexities low for long-range forecasting. To address the challenge of capturing different types of semantic information, we propose novel Water-wave Information Transmission (WTT) framework. This framework captures both long- and short-term repetitive patterns through bi-granular information transmission. It also models global and local correlations by recursively fusing and selecting information using Horizontal Vertical Gated Selective Unit (HVGSU). In addition, to improve the computing efficiency, we propose a generic Recurrent Acceleration Network (RAN) which reduces the time complexity to O(VL) while maintaining the memory complexity (L). Our proposed method, called Water-waye wave Information Transmission and Recurrent Acceleration Network (WITRAN), outperforms the state-of state-of-the-ard methods by 5.80% and 14.28% on long-range and ultra-long-range time series forecasting tasks respectively, as demonstrated by experiments on four benchmark datasets. The code available at: htttps://githun com/ Water2sea/WITRA
still not sufficiently
Introduction
Time series forecasting is valuable tool across diverse fields, such as energy, traffic, weather and so on. Compared with short-range time series forecasting, long-range forecasting offers the advantage of providing individuals with ample time to prepare and make informed decisions. For instance, accurate long-range traffic and weather forecasts enable individuals to plan their travel arrangements and attire accordingly. To improve the accuracy of such forecasting, it is essential to utilize longer historical sequences input for the forecasting models [Liu et al., 2021 Zeng et al. 2023).
Previous studies [Wu al. 2023 Nie et al. 2023] have highlighted the importance of capturing semantic information in long-range time series to achieve accurate predictions. However, the semantic information in long-range time series is diverse, so how to analyze and capture it becomes major challenge. Specifically, semantic information includes two main aspects: (1) Global and local correlations. The local semantic information usually contains short-range changes within the data. while the global semantic information reflects the long-range trends present in the time series [Wang al. 2023), which can be referred as global-local semantic information.
(2) Long- and short- term repetitive patterns. Time series often exhibit repetitive patterns at different timescales Lai
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

et al.. 2018 Liu et al. 202T], such as hourly or daily cycles, which can be identified as periodic semantic information. Furthermore, it is crucial to model both aspects of semantic information simultaneously and ensure that the modeling approach remains computationally efficient, avoiding excessive complexity.
Unfortunately, although the existing state-of-the-arr methods have shown great performance, they encounter difficulties in addressing the aforementioned challenges simultaneously. Specifically, CNN-based methods [Bai et al. 2018 Franceschi et al. 2019 Sen et al., 2019 Wu et al. 2023
Wang et al. 2023] have linear complexity with respect to the sequence length L. but they are either limited by the size of the convolution receptive field [Wang et al. 2023] or constrained by the ID input sequence (Wu et al., 2023), making it difficult to capture both important semantic information simultancously. Transformer- based methods can be broadly classified into two categories based on whether the point-wise attention is used or not. The used ones (Vaswani et al. 2017 Zhou et al.)
2021 2022a] capture correlations between points in the sequence, yet face challenges in capturing hidden semantic information directly from point-wise input tokens (Nie et al. 2023 Wu al.
2023]. The others (Li al., 2019 Wu et al. [021 Liu et al., [021] struggles either with achieving sufficiently low computational complexity or with effectively capturing periodic semantic information.
Other methods [Zhou et al. 2022b Zeng et al. 2023 also exhibit limitations in capturing semantic
information mentioned above. Further details can be found in Section 2
RNN-based methods [Hochreiter and Schmidhuber 1997 Chung et al. 2014 Rangapuram et al.
2018 Salinas al.) 2020] have significant advantages capturing global and local semantic informa-
tion through their recurrent structure, as shown in Figure Ta). while maintaining linear complexity. However, they suffer from gradient vanishing/explodingg [aaccan et al., (003) and information for- getting issues (refer to Appendix for more details), making them less suitable for direct application in long-range forecasting tasks. Fortunately, has been proven that by splitting the information transmissions into patches between few adjacent time steps and processing them individually. it is possible to maintain both global and local semantic information [Nie et al. 2023 Wu et al. [023). This insight is highly inspiring, as it suggests that by dividing the inputs of RNNs into numerous sub- series and processing them separately, we can effectively address the significant limitation mentioned earlier, without compromising the efficiency of mangreaae forecasting.
ligee
Layse
Haken StLD
Ateciiee bnbedding
Dnhaddig
RN
Laya
bataldip
lypa Froedding
lepn
Pebattigg
Gibeliing
Layer Babedinng
lapaa
(d) Full Attnnionn
UNN
Dibhddiggs
Lagar
Esboklep
Trasgrec
Maaer Erbediigs
lepel Babeliigg
Glisd
Exbokinp
Lidd
Logginn
Esbokdap
Inbuddings
Laser
(6 MIUN
layz Inbiddings
Baboldin
Bebokking
Exbekizg
Tebelling
Babakdng
Paccs
Febedkiigs
lapas
(c) Pynafiemner
(g) PatchTST
(U WITRAN (DurS)
Figure 1: Information transmission process diagram of different forecasting models.
2

---

Based on the aforementioned insights, we propose a novel framework called Water-wart wave Information Transmission and Recurrent Acceleration Network (WITRAN) which comprises two key compo- nents: The Water-wave Information Transmission (WTT) framework and the Recurrent Acceleration Network (RAN). The overall information transmission process of WITRAN is illustrated in Fig- ureei Firstly, to capture the periodic semantic information of long- and short-term, we rearrange the imput sequences according to their natural period, as shown in Figure Fa. This rearrangement allows for information transmission in two directions, resembling the propagation of water- wave energy, as shown in Figure 2b.. The horizontal red arrows indicate the intra-periodic informa- tion transmission along the time steps, while the vertical blue arrows indicate the inter- ietroppe information transmission. Secondly, to preserve the characteristics of long- and short-term periodic semantic information, we propose novel Horizontal Vertical Gated Selective Unit (HVGSU) which incorporates Gated Selective Cells (GSCs) separately in both directions. To capture the correlation between periodic semantic information of both directions at each time step, design fusion and selection operations in GSC. With a recurrent structure, HVGSU progressively captures more local semantic information until it encompasses the global semantic information. Thirdly, to improve efficiency, we propose the Recurrent Acceleration Network (RAN), which enables parallel processing of information transmission in both directions. Notably, RAN maintains a memory complexity of (L) while reducing the time complexity OVVT). In addition, RAN could serve as a universal framework for integrating other models to facilitate information fusion and transmission. In summary. our main contributions are follows:
We propose Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN), which represents a novel paradigm in information transmission by enabling bi-granular flows. We provide a comprehensive comparison of WITRAN with previous methods in Figure [ to highlight its uniqueness. Furthermore, in order to compare the differences between WITRAN and the model (a)-(h) in Figure more clearly, we have prepared Table to highlight the advantages of WITRAN.
We propose a novel Horizontal Vertical Gated Selective Unit (HVGSU) which captures long- and short-term periodic semantic information by using Gated Selective Cell (GSC) indepen- dently in both directions, preserving the characteristics of periodic semantic information. The fusion and selection in GSC can model the correlations of long- and short-term periodic semantic information. Furthermore, utilizing a recurrent structure with HVGSU facilitates the gradual capture of semantic information from local to global within sequence.
We present a Recurrent Acceleration Network (RAN) which is a generic acceleration framework that significantly reduces the time complexity to O(VL) while maintaining the memory complexity of (L). We summarize the complexities of different methods in Table demonstrating the superior efficiency of our method.
We conduct extensive experiments on four benchmark datasets across various fields (energy. traffic and weather). The empirical studies demonstrate the remarkable performance of WITRAN, which achieves relative improvements of 5.80% and 14.28% in long-range and ultra-long-rang raage forecasting respectively. In addition, the introduction of the generic RAN framework greatly improves the computational efficiency.
Table 1: Advantages of WITRAN compared to other methods.
Advatago
HAN OSENN
Full Amotics
to Lagghen Pycaees
MENN
Gn WiRasise do limahe
WITRAN learo
Nee paiza Adta scmontic interratio capare
Specia deegg to cipere leng sere repertte parme
byyro global
Z1)
K
K
Well gradict www.kinggo preblon RNN
Table 2: Complexity of forecasting models in training. L is the sequence length, and S is the stride in the PSt model.
Methods
RNN
CNN
Transtommer
Log Trans
Intormer
Autotocmer
Poratormer
HRDDommer
FILM
PachTST
MICN
WITRAN cours)
O(t)
o(1)
o(12)
O1 F)
Of M
OlElog T)
O(F)
((1)
OU))
O(t)
O(VI)
Memory
O1l)
O(12)
O112)
OiE E)
OlE ()
O(
OLL)
O((E/S1))
O(0)
3

---

Orginal icio
Skait-erm pelod
Long batn perlol informaiiz Eauriicon
Infortation Laseiiii tuon source to sink
Readuuiic sage susce information
Indenntiot ware
intoration mramont
information inrameie
Peñid
Periol
Peid
Roumagg
Coorse
Courert periil
(4) Inforatiio tramsmissiinn before and afler the inpul sequence nerrangemet
(b) The water-wave information tnmemission process
Figure 2: Input rearrangement and water-wave information transmission.
Related Work
Time series forecasting methods can be broadly categorized into statistical methods and neural network methods. Neural network methods, in turn, can be further classified into several subcategories based on the specifie techniques employed, including RNN-based, CNN-based, Transformer-based, and other Deep Neural Networks (DNNs) by the methods used.
Statistical Methods with major representatives include ARIMA [Box and Jenkins 1968). Prophet [Taylor and Letham 2018). and Holt Winter [Athanasopoulos and Hyndman 20200: J. How- ever, the inherent variability and complexity of real-world time series often render these methods inadequate, as they are built on hypotheses that may not align well with the characteristics of such data. As result, the performance of these statistical methods tends to be limited.
RNN-based Methods [Hochreiter and Schmidhuber, 1997 Chung al., 2014, [angapuram et al., 2018, Salinas et al. 2020) possess the inherent advantage of capturing semantic information in time series data through their recurrent structure, which mimies the natural information transmission process of time series. This allows them to progressively capture semantic information from local to global contexts. These methods exhibit linear time and memory complexity, enabling efficient processing of time series data with sequence length of L. However, when dealing with longer sequences, the problems of gradient yanishing/cxploding [Pascanu et al. 2013) and information forgetting (see Appendix for more details) further limit it.
CNN-based methods Bai et al. 2018 Franceschi et al. 2019 Sen al.) 2019 Wu al. 2023
Wang et al. 2023) are adept at capturing local semantic information through the application of convolutional kernels, while maintaining linear time and memory complexity. However, on the one hand, most of these methods face challenges in capturing comprehensive global information due to the limited receptive field of individual convolutional layers, which make the training process more difficult and overhead [Wang et al. 2023). On the other hand, it is difficult to directly capture long- and short-term repetitive patterns on ID imputs. MICN [Wagg et al., 2023 adopted downsampling ID convolutions and isometric convolutions combined with multi- branch framework, which can effectively solve the former problem, but still suffers from the latter one. TimesNet [Wu et al.) 2023 transforms ID inputs into 2D space, leveraging a parameter-efficient inception block to capture intra- and inter period variations. However, it solves the latter problem while still facing the former one.
Transformer-based methods have shown advancements in time series forecasting, with two main categories. The first category uses point-wise attention mechanisms, such as Vanilla Trans- former (Vaswani et al.] [017). Informer [Zhou et al. [222]. and FEDformer [Zhou et al., 2022a]. while facing challenges in extracting sufficient semantic information from individual time points. The second category employs non point-wise dot product techniques, including Log Trans [Li ct al.) 2019]. Autoformer [Wu et al. 202T]. Pyraformer [Liu et al. 202T] and PatchTST [Nic et al. 2023] They reduce computational complexities to certain degree, yet face difficulties in incorporating the long-term correlations in time-series. For a detailed description of these methods, please refer to Appendix A Furthermore, it is worth noting that the majority of the aforementioned methods fail to achieve lower complexity than RNN- methods. For a comprehensive comparison of the theoretical time and memory complexity, as well experimental results, please refer Appendix C
Other DNN methods have also demonstrated promising performance. For example, FiLM (Zhou al. 2022b utilizes Legendre Polynomials and Fourier projection methods to capture historical in-

---

formation, eliminate noise, and expedite computations using low-rank approximation. DLinear [Zeng al. 2023] has showcased impressive outcomes by employing simple linear operations. Nevertheless, both approaches encounter difficulties in capturing various repeating patterns present in the sequence.
3 The WITRAN Model
The time series forecasting task involves predicting future values Y R Px for P time steps based on the historical input sequences X = {01, #2,...,,,,,, E R X of H time steps, where Gn and Cout represent the number of input and output features respectively. In order integrate enough historical information for analysis, is necessary for H to have a sufficient size (Liu et al. 2021 Zeng al., 2023]. Furthermore, capturing semantic information from the historical input is crucial for accurate forecasting, which includes modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. However, how to address them simultaneously is a major challenge. With these in mind, we propose WITRAN, a novel information transmission framework akin the propagation of water waves. WITRAN captures both long- and short-term periodic semantic information, well as global-locae semantic information simultancously during information transmission. Moreover, WITRAN reduces time complexity while maintaining linear memory complexity. The overall structure of WITRAN is depicted in Figure
Ingur
Recurentt Acceleration Network
Forcessting Medule
Module
Water-wave Information Transmission Fmmework
Nem
SVCS
TFR Fzz
Theern
Cenot
Wobe
Paly Corandd
Reiringe
Colar
Devei
Olpss
Dayfo
Figure 3: Overall structure of WITRAN.
3.1 Input Module
To facilitate the analysis of long- and short-term repetitive patterns, inspired by TimesNet (Wu et al. 2023), we first rearrange the sequence from ID to 2D based on its natural period, as illustrated by Figure 2 a). Importantly, our approach involves analyzing the natural period of time series and setting appropriate byperparameters to determine the input rearrangement, rather than using Fast Fourier Transform (FFT) to learn multiple adaptive periods inputs in TimesNet. Consequently, our method is much simpler. Additionally, in order to minimize the distribution shift in datasets, we draw inspiration from NL.near [Zeng et al. 22023) and employ an adaptive learning approach to determine whether to perform simple normalization. The input module can be described as follows:
X XID
morm
X TH norm
(I)
XBD Rearrange([X1p, T'uul)..
here, XID E Hx represents the original input sequences, TH E go represents the input at the last time step of the original sequence. TFam E IR Xxtime represents the temporal contextual features of original input sequence (e.g.. HourOiDay, DayOrWeek, DayOfMonth and DayOrYear), where Crime the dimension of time features. Xnn E R Rxxxx Xxxxxx(euttuiaa (Cuu+) represents the inputs after rearrangement, where R denotes the total number of horizontal rows and C' denotes the vertical columns. norm is an adaptive parameter for different tasks. [-] represents the concat operation and Rearrange represents the rearrange operation, with reference Figure 2 a).
5

---

3.2 Horizontal Vertical Gated Selective Unit
To capture long- and short-term periodic semantic information and reserve their characteristies, we propose novel Horizontal Vertical Gated Selective Unit (HVGSU) which consists of Gated Selective Cells (GSCs) two directions. To capture the correlation at each time step between periodic semantic information of both directions, we design the specific operations in GSC. Furthermore, HVGSU is capable of capturing both global and local information via a recurrent structure. In this subsection, we will provide detailed introduction them.
HVGSU As depicted in Figure the process HVGSU via a recurrent structure is:
Hhovy Hver Out HVGSU(X2D),
where Hhor R.x.ixdmoo and H E LBLXXddmodd represent the horizontal and the vertical output hidden state of HVGSU separately. L is the depth of the model, and Out E R R denotes the output information of the last layer.
In greater detail, the cellular structure of HVGSU is shown in Figure which consists of GSCs in directions capture the periodic semantic information of long- and short-term. The cell operations for row r (1 S r r R) and column e (1 S c C) in layer (1 S I S L) can be formalized as:
hor c, GSChor (impuul,, e, 1; hhor c-1, 1 e. 1)
GSC (imputrr, e, h e bo hoor c-1, 1)
(3)
Or, [hhor hy. ul
here, input,, c, E Rd., When = 1, imput,, e, Dr, E R Esime represents the input for the first layer, and when 1, input,,
Or, c. 1-1 E R2 (22xdoodel represents the input for subsequent layers. hor 1 and h,- e, E R Rdmondel represent the horizontal and vertical hidden state inputs of the current cell. Note that when r = 1, M-= c, is replaced by an all-zero tensor of the same 1. [ represents the concatenation operation and
size, and the same is true for hoor e-1, 1 when Or, 1. 2 dmadel represents the output of the current cell.
Apwsuct
Inport, s.1
CO
[BC
banth
Cuncat
6/20
Tanhh
i
Cuuccel
(a) GSC stucture
(b) HVGSU structure
(c) RAN structure
Figure 4: The structure of our method.
GSC Inspired by the two popular RNN-based models, LSTM (Hochreiter and Schmidhuber 1997
and GRU [Chung et al. 2014) (for more details, see Appendix [A), we propose a Gated Selective Cell (GSC) fuse and select information. Its structure is shown Figure aa), which comprises two gates: the selection gate, and the output gate. The fused information consists of input and principal-subordinate hidden states, and the selection gate determines the retention of the original principal information and the addition of the fused information. Finally, the output gate determines the final output information
cell. The different colored arrows Figure [a) represent different semantic information transfer processes. The black arrow represents the input, the red arrows represent the process of transmitting principal hidden state information, the blue arrow represents the subordinate hidden state, and the purple arrows represent the process by which fused information of

---

principal-subordinat hidden states is transmitted. The formulations are given as follows:
(W,hk-- hi-1, x] + bz)
O₂ a (Wo -; æ] + bo)
h tanh (W (Wy(h(ir h6-1: x] + bb,)
(4)
A (1 St) @ Ap- + S 1 hh hpti tanh(hp-1) O.,
where h and h Ruudde represent the imput principal and subordinate hidden state, R represents the input. W. E Rd (2dmmota are weight matrices and b, E IRomadel are bias vectors. S, and O, denote the selection gate and the output gate, O denotes an clement-wist product,
a(-) and tanh(-) denote the sigmoid and tanh activation function. h and hi E IR dddddd represent the intermediate variables of the calculation. 4,!" represents the output hidden state.
3.3 Recurrent Acceleration Network
In traditional recurrent structure, for two adjacent time steps in series, the latter one always waits for the former one until the information computation of the former one is completed. When the sequence becomes longer, this becomes slower. Fortunately, in the WIT framework we designed, some of the time step information can be computed in parallel. As shown in Figure 2(),, after a point is calculated, the right point in its horizontal direction and the point below it in its vertical direction can start calculation without waiting for each other. Therefore, we propose the Recurrent Acceleration Network (RAN) as our accelerated framework, which enables parallel computation of data points without waiting for each other, greatly improving the efficiency of information transmission in HVGSU. We place parallelizable points in slice, and the updated information transfer process is shown Figure Each green box in Figure represents a slice, and the number green boxes is the number of times we need to recurrently compute. The meanings of the remaining markers are the same as those Figure 2 Under the RAN framework, the recurrent length has changed from the sequence length L = R x to R - C 1, while the complexity of R and is O(VL). Thus, we have reduced the time complexity to O(VL) via the RAN framework. It should be noted that although parallelly compute some data points, which may increase some memory, the complexity of parallel computation, O(VL), is far less than the complexity of saving the output variables, O(L), because we need to save the output information of each point in the sequence. Implementation source code for RAN is given in Appendix D
3.4 Forecasting Module
In the forecasting module, we address the issue of error accumulation the auto-regressive structure by drawing inspiration from Informer [Zhou et al. 2021] and Pyraformer [Liu et al. 2021], combining both horizontal and vertical hidden states, and then making predictions through fully connected layer, as illustrated in Figure
We utilize the last of the horizontal hidden states it contains sufficient global and latest short- term semantic information from the historical sequence. In contrast, all columns of the vertical hidden states, which capture different long-teny bong-term semantic information, are all preserved. The combined operation not only maximizes the retention of the various semantic information needed for predicting the points, but also avoids excessive redundancy in order obtain accurate predictions. This module can be formalized as follows:
Hhor = Repeat( rep
H hh-v v Reshape Hver)
(5)
Y = FCI(HIn-v)
Y = FC2(Reshape())) T-TFFEa Eac):
where Fae E Pxdmodal represents time features encoding of the forecasting points separately. hbor E L2 ddmdeel represents the last row hidden state in Hhor. Repeat(-) is for repeat operation and
7

---

Hoop E RlxxCxdaoded represents the hidden state after the repeat operation. H E CC the vector of horizontal and vertical combination after reshape operation. FC1 and FC2 represent two fully connected layers respectively. Y E IRC"(Faa represents the intermediate variables of the caleulation and Rde C = P. Y represents the output of this module, and it is worth noting that we need to utilize the adaptive parameter norm for denormalization, when norm = 1, y, =Y=++ HH.
4 Experiment
Datasets To evaluate the proposed WITRAN, we conducted extensive experiments on four widely recognized real-worle benchmark datasets for long-range and ultra-long-rane forecasting, These datasets cover energy. traffic, and weather domains. We split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2. More details about the datasets and implementation are described Appendix and Appendix F
Baselines In light of the underperformance produced by classical models such as ARIMA and
simple RNN/CNN models, as evidenced by (Zhou al. 2021 and [Wu et al. [021] J. and the subpar
performance exhibited by certain Transformer- based models like LogIrans [Li et al. 2019] and
Reformer [Kitaev et al., 2020]. as shown in (Wu et al. [023] and [Wang al. 2023 our study
primarily incorporates six transformer-basee models: PStchTST Nie et al.
2023 FEDformer [Zhou
et al.; 2022a], Pyraformer (Liu et al., [2021], Autoformer(Zhon
al.,
2022a Informer (Zhou
et al. 2021]. Vanilla Transformer [Vaswani et al., (017]. and four non-transtormer- based methods:
MICN [Wang et al., [2023), TimesNet [Wu et al. 2023]. DLinear [Zhou al. [022b] FiLM (Zhou
et al. 2022bb]. Please refer to Appendix A and Appendix H for more details of the baselines and
implementation.
4.1 Experimental Results
For a better comparison, we adopted the experimental setup of Pyraformer [Lu et al. 2021] for long-range and ultra-long-range series forecasting. In addition, multivariate time series prediction [Nie et al., 2023), so it is necessary to verify the performance of models on a single channel ensure their effectiveness across all channels in multivariate time series prediction. In this paper, experiments were conducted on a single channel. Note that in order to fairly compare the performance of each model, set up the search space that each model can perform optimally on each task. For further details, please refer to Appendix
is crucial for
Long-range Forecasting Results We conducted five tasks on each dataset for long-range prediction, and the results are shown in T Taking the task setting 168-336 on the left side of the Table Bas an example, it indicates that the input length is 168 and the prediction length is 336. Notably, our proposed WITRAN achieves state-of-the-art performance, surpassing the previous best method with an average MSE reduction of 5.803%. Specifically, WITRAN exhibits an average MSE reduction of 10.246% for ECL, 3.879% for traffic, 2.519% for ETTh 4.431% % for ETTh2, and 7.939% for weather. Additionally, we note that the competition among the baselines is intense due to their best performance on each task, but none of them consistently performs well across all tasks. In contrast, WITRAN demonstrates its robust competitiveness across various tasks and datasets. In addition, we can find that in most cases, a longer input length yields greater improvement with the same prediction length. All findings above collectively highlight WITRAN's efficacy in addressing diverse time-series forecasting tasks in real-world applications. Further results and showcases are presented in Appendix and Appendix[]
Ultra-long-range Forecasting Results results on each dataset in Table 4 The tasks on the left side of the table hold the same interpretation as above. Notably, WITRAN achieves an 14.275% averaged MSE reduction. More specifically, WITRAN demonstrates an average MSE reduction of 39.916% for ECL, 3. 122% for traffic. 14.837% for ETTh1, 2.441 % for ETTh2, and 11.62% for weather. particular, our method showcases substantial improvements of over 10% in ECI, ETTh1, and weather, reinforcing our ability to predict ultra-long-range time series. And by comparing the results of ultra-long-rang forecasting and long- range Forecasting, we can find that our method is more competitive for ultra-long- rangg prediction,
We also show the three tasks for ultra-long-range prediction
8

---

Table 3: Long-range forecasting results. A lower MSE or MAE indicates better prediction. The best results are highlighted in bold and the suboptimal results are underlined.
Matieds
WIIRAN
KBON
Tixeehi
PRcNTST
DL
FLLM
FEUNGEnA
Pyndionnx
Aatofermn
Infeener
Iraadorta
MAN
MAE
MSE
MS6 MAA
MSE
MA5
MSB
0L4307
B.ae 0.357)
01138:
M.4337
118-11
0.3687 0.5721
0.3310
0.9965-11
05333
0-191
B.5WG 0.5771
0:6733
0-901 0-500
0.79% 0:6/9
335-130
8.2517
8.367
0.3702
084303
04317
94464
(L2O0 8J722
012122
0.333
03318
94300
0299
(4800
0.3333
0.3354
0.891 04891
[STTI (J563
730 9N1
0.3418
8.9651
0.9963
015941
8386
0 5-567
8-5143
4000
D3015
0.6118
0.9534
0
0-885 94716
0.966 0.863)
158:22
0.1381
8.3169
0.2011
0,331
0J3497 0.2355
0 1641
9.2944
DL1A0 0,2210
8.540
0,585
0J9632
0,2583
0.3933
03891 05666
(89C3 01911
11.300
05365
D476
0.547
376
8.1381
0.2175
1111
03271
0.733 (M2T)
120-123
5.1688
0.3191
025522
01:545
01.440
0.1543
9.441
01400 0.2241
02861
83465
GA511
06292
0.2199
014043
0/216 1.254
1JM6 0.271
3:359
0,3257
BITN ETTS.
158-13 335
23714
01622
0.303
0.1301
013133
11553 8:206
0125 0.77
0.718
0 11511
02510
0106 0.3419
01315
013575
1863 03115
(1399 0L3004
336-730
0.1484
02219
0,5729
0.113
6
2062
0 5360
0UN6 033452
03335
1534
53176
0,996 0,5302
0.0925 0.3450
1533
Giii 03504
720-151
0:1284
9.366
02176
0.786
0.2366
0.249
168 163
0.73 RABA
0.3655
0.555
0 5063
0/55 0.985
0.7535 01 3443
B4355
0.706
0.300
07403
0.576
3-161
0.963 BIEWS
188-330
03217
8.77R
LOMIT BL422
013125
0.4383
22216
BA515
(L2911
02263
DA385
114447
0.3344
24883
0,3716
335-731
03311
C3933
IADES IUSD0S
114233 0.2355
5586 04006
DATES 1L3136
114155
115583
03425
D4426
633372
114525
0.311
04673
(AK2)
8,001
158-355
1.555
0,2231 0U5429
0.2433
0.3585
0246
0J3575
013425
0.3544
02383
B5714
021 (6363)
0,2270
0.3513
0.263 3
(1230 E15435
335-335
02163
8.482
0250 033934
0.2584
0.3152
03149
84163
0220
(3P9))
0.3343
0,3953
02191
03664
02310
0.399
0,3355
04362
0.311
04129
116 151-351
83364
1045
0.997
1:67
8:000
6.IIT
0,7467
8.500
0.707
03586
0:146
Table 4: Ultra-long-range forecasting results. A lower MSE or MAE indicates a better prediction. The best results are highlighted in bold and the suboptimal results are underlined.
Pyrtirme
Mark:
KS5
MAA
XSS
MAA
MSE
MAA
SIS
MAE
MSE
MAE
MSE
KIA
NSSE
KAA
MSS
MAE
M55
MAE
MB5
MAE
MIBE
MAE
122
E539 14:00
TZD-544)
8.209 8,3127
8.1488
11460
1983
9.715
1303
0.113
0.5952
05505
0.824
07335
044233
ILSITS
5515
ILEA.
(U4730
ELS556
DJ0033
0,500
D3220
2.5380
CA332
1.407
1953
0.5333
(1003
0.5044
0,5317
(1423)
0.551
0.593
011871 0.5116
0.7876
0.057
0147%
11:00
0.3388
11107 0375
17501 01774
6.961 0,262
0-2911
1994
0.5114
0.592
$493-2000
6.1425 0.2533
02223
9:584
0.3390
0,342
0.2223
003000
11.30
013472
0.3955
0.344
05163
0.309
0,3025
0,3982
0.954)
0.7615
0,5000 0L5677
720 3443
0.155 0.2943
0-440 8-5816
0.338
033355
013852
DJRA 0.3009
5.166 0.3155
03356
0.634)
0.375
(1335)
013935 IL3ER5
4485- 1440
6.1M4 8.5000
5:350 94253
8.18
033372
0.5321 0.445
(12200
0.911
0.2244
0,8775 035744
0.5451 0.6866
0.4511
0.550
0.1155 01113
0,992 (L303)
8:211 9.439
0.003
85645
0.118
05001
01500
03337
0.7699 B196S
03231
0:821
01395 0.131
0:563
5985
00805
01.991
03112 TLBRSS
5499-2800
8.330 0:415
0.549
05895
86336 13354
1.3955 2.561
115355
5.3035
1860
1.7161
03348
6.644
4.1031
1.3154
0.335
0.4482
013737 144787
770 319)
0.3112
1111
0.117
(15611
8:19%
11610 6313
05584
1164
133155
4489-1440
8.19T 8.356
94205
0.339
40433
0,3371
0.3399
013147
644177
0,900 (J600
(2223
5.500)
52401 03179
0.7766
0.1133
0.2335
0.250)
013500 1L3925
underscoring WITRAN's capability to capture semantic information in ultra-long historical input. More results and showcases are provided in Appendix H and Appendix J
4.2 Time and Memory Consumption
To assess the efficiency of our proposed WITRAN, we compared its time and memory consumption with selected baselines that demonstrate superior performance. To fully evaluate the actual complexity of our model and other models for the long range forecasting and ultra-long range forecasting tasks, we set up two cases: 1) constant input length and varying prediction length, 2) constant prediction length and varying input length. In both cases, we fixed the input length and output length at 720. Figure Jjuursttiltttttteee the comparison, revealing that WITRAN achieves the lowest actual time complexity and memory complexity. This section is described in more detail in Appendix [
12
3.
128, 2AB
bdad kmith 200
1200
255
125, 2007 202
1000 1230
MoMenay conmmpannn ches) lenghts food an 721
(D7in ooverpse inght Bod 7231
(c) Maraor mevarppoe toapu loggh Fact 720
00 wow.eceurgeteeeeeeee.mmommo lngiy Bood 726
Figure 5: Time and memory consumption. WIT is the framework that does not involve the RAN.
4.3
Ablation Study
The impact of RAN can be clearly observed in Figure 5 In addition, we conducted ablation studies to measure the impact of HVGSU and GSC in WIT, and the combined operation of the last row of horizontally hidden states and each column of vertically hidden states. Detailed results and discussions

---

can be found in Appendix G Here, we provide concise summary of the key findings: 1) The fusion and selection design of GSC enables both long- and short-term semantic information be captured for each historical input data points. 2) Setting up independent cells in both directions enables to extract semantic information in the long- and short-term respectively. 3) The specially designed combined operation can make fuller use of the captured local and global semantic information while ensuring that the information is not redundant. 4) RAN offers advantages in terms of speed and space complexity. The utilization of RAN eliminates the need to store excessive intermediate variables, as shown in the previous section.
4.4 Robustness Analysis
We have followed MICN (Wang et al., 2023] and introduced a simple white noise injection to demonstrate the robustness of our model. Specifically, we randomly select a proportion & of data from the original input sequence and apply random perturbations within the range [-2X,, 2X,] to the selected data, where X; denotes the original data. After the noise injection, the data is then used for training. and the MSE and MAE metrics are recorded in T5ble
Table Robustness experiments of forecasting results. Different E indicates different proportions of noise injection. And WITRAN is used as the base model.
Tasks
168-168
108-336
536-336
336-720
720-720
720-1440
1440-1440
1440-2880
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
0%%
0.2397
0.3519
0,2607
0.3721
0.2517
0.3627
0,3084
0.4055
0.2478
0.3651
0.2899
0.3727
0,2408
0.3680
0.3359
0,4383
0.2420
0.3534
0.2630
0.3738
0.2516
0.3626
0.3078
0,4060
0.2345
0,3900
0,2502
0.3739
0.2395
0.3672
0.3372
0.4476
0.2463
0.3575
0.2692
0.3758
0.2567
0.3652
0.3039
0.4016
0.2525
0.3756
0.2410
0.3687
0.3315
1095
0.2513
0.3621
0.2726
0.3811
0.2702
0.3736
0.3098
0.4061
0.2569
0.3756
0,2674
0.3850
0.2444
0.3716
0.3366
0.4445
1755
0.1377
0.2051
0.1321
0.2055
0.1306
0.2055
0.1391
0.2175
0.140K
0.2191
0.1672
0 0.2449
0.1543
0.2325
0.1425
0.2333
Traflic
175
0.1376
0.2063
0.1329
0.2083
0.1316
0.2074
0,1425
0.22118
0.1409
0.2218
0.1676
0.3466
0.1551
0.2372
0,14%
0.2360
5%
0,2115 0,1370
0.1362
0,2148
0.1323
0.2106
0,1432
0,2260
0.1416
0,2247
0.1699
0,2319
0.1572
0.2373
0,1519
0,3480
0,2164
0.1355
0.2193
0.1372
0,2000
0.1467
0,2325
0.2591
0.3475
0.1561
0.1575
0,2589
0.1189
0,2714
0.1112
0.2638
0,1494
0.3092
0.1296
022668
0.133100
0,1304
0,29022
0,1850
Weather ETIh: ETTb
15%
0.1112
0,2596
0.1208
0.2226
0,1111
0.2637
0,1463
0.2035
0,1394
02885
0.1367
0 0.2978
0.1319
0.2907
0.1834
0.3484
5%
0.1135
0,2622
0.1221
0.275]
0.1199
0,2689
0,1527
0.3124
0.130
02558
0.1356
0 2952
0.1359
0.2955
0.180) 0.3435
10%
0,1137
0,2628
0.1227
0.2157
0.1196
0.2688
0.1559
0.5148
0.1388
0,2952
0.1574
0 0.2994
0,1405
0.3015
0.3499 0.1842
0,2389
0.3813
0.2277
0.3778
0,2432
0.3922
0.2373
0,3883
0.2635
0.4018
0.2915
0.4299
0.2815
0.4220
0,3280
0.4585
0.3004
0.2284
0.3777
0.2450
2.53442
0.2390
04358
0.29%
0.4297
0,3353
0642
65%
0,2364
0,3799
0:2379
0,3834
0.2563
0.4025
0.2501
0,3967
0.2661
0,4056
0.3207
0 0,442
0,2935
0.4308
0.3307
04403
10%%
0,2603
0.3959
0.2475
0.3902
0,2613
0.4055
0,2581
0,4022
0,2674
0.4053
0.3193
0.4481
0.2847
0.4237
0.3317
0.4612
0%
0,2050
0.3338
0.2197
0.3470
0.2166
0,3482
0,2054
0.3424
0,2008
0.3417
0.1872
0.3312
0.1907
0.3366
0.1769
0.3257
1%
0,2050
0.3345
0.2154
0.3470
0.2214
0.3522
0,2055
0.3419
0.2005
0.3419
0,1872
0.3313
0.1903
0.3361
0,1828
0.3327
0.2057
0.3362
0.2241
0.3517
0.2268
0.3557
0,2058
0,3426
0.2008
0.3421
0.1867
0.3305
0.1897
0.3353
0.1831 0.3357
0.3369
0.2220
0,3484
0,2308
0.3595
0.2059
0.3438
0.2007
0.3418
0.3290
0.1900
0.3370
0.1828
It can be found that the perturbation proportion increases, there is a slight increase in the MSE and MAE metrics in terms of forecasting. It indicates that WITRAN demonstrates good robustness when dealing with less noisy data (up 10%), and it possesses significant advantage in effectively handling various abnormal data fluctuations.
5 Conclusions
In this paper, we propose WITRAN, a novel Water-wave Information Transmission framework and a universal acceleration framework. WITRAN effectively captures both long- and short-term repetitive patterns and global and local correlations with O(VL) time complexity and (L)) memory complexity. The experimental results demonstrate the remarkable performance and efficiency of WITRAN. However, this recurrent structure still not optimally efficient for Python-based implementations because of the information waiting between slices. Therefore, we plan to explore the integration of WITRAN into an interface using C++, similar to the implementation of Gww.GRUUnnnnR. in PyTorch in the future, to further improve its efficiency.
Acknowledgments
This work was supported by the Fundamental Research Funds for the Central Universities under Grant No. 2022Y Furthermore, we would like to express our gratitude to Shuo Wang and Shuohao Lin for their valuable discussions and assistance.
10

---

References
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar, Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
Ailing Zeng. Muxi Chen, Zhang. and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11122 2023.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang. and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long- term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
Huiqiang Wang, Jian Peng, Feihu Huang. Jince Wang, Junhui Chen, and Yifei Xiao. Mien: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
Guokun Lai, Wei-Cheng Chang. Yiming Yang. and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The s1 international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.01271, 2018.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances neural information processing systems, 32, 2019.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: deep neural network approach to high-dimensional time series forecasting. Advances in neural information processing systems, 32, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang. Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time- seriss forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-110 11111,111111111111111111111111 2021.
Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting, In International Conference on Machine Learning, pages 27268-2277777 27286. PMLR, 2022a.
Shiyang Li. Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. Advances in neural information processing systems, 32, 2019.
Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with nuto-correlation for long term-termme series forecasting. Advances in Neural Information Processing Systems, 34: 22419-22430, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances Neural Information Processing Systems, 35: 12667-127690 2022b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1997.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting, Advances in neural information processing systems, 31, 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal Forecasting, 36(3):1181-11911 2020.
11

---

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference machine learning, pages 1310-1318. Pmnlr, 2013.
George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series (Applied Statistiess, 17(2):91-09, 1968.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 7201):37-45,200 2018.
G Athanasopoulos and RJ Hyndman. Forecasting: principle and practice: O texts; 2018, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.044:3 2020.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Withrock, Mark Haseguwa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems, 30, 2017.
Zeping Yu and Gongshen Liu. Sliced recurrent neural networks. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2953-2964, 2018.
Lifeng Shen, Qianli Ma, and Sen Li. End-to-end time series imputation via residual short paths. In Asian conference machine learning, pages 248-263. 263. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,699, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high nigh-performance deep learning lihrary. Advances in neural information processing systems, 32, 2019.
12

---

A A Brief Overview of RNN-Based and Transformer-Based Models
In this section, will provide brief overview of representative variants of RNN-based methods (Appendix A.I) and Transformer-bast methods (Appendix A.2) Detailed descriptions of other methods can be found in Section 2
A.1 RNN-based methods
LSTM [Hochreiter and Schmidhuber 1997] and GRU [Chung et al. 2014 are two prominent variants of RNN-based models. They have gained popularity diverse domains, such as natural language processing, speech recognition, and video analysis due to their ability to capture important information. This subsection, will provide detailed introduction to LSTM and GRU.
C-
C.
h--1
ht
tanh
CF
tanh
tanh
h
Osncet
ht
Dmnst
x
(a) LSTM cell structure
(b) GRU cell structure
Figure 6: The structure of LSTM and GRU.
The structure of LSTM, shown in Figure 6 a), comprises three gates: the input gate, the forget gate and the output gate. In addition, LSTM has two vectors: the hidden state and the cell state, for recursively processing the information flow between different time steps. LSTM can be formalized as follows:
LSTM
p (Whttm [14 1, 2| + b)tt)
4 Jstm (W/att [ 2] + 1lsmm)
C. tanh(W/attm latmm [/c- x + (ltmm)
(6)
Jtm a (W [ 2| hum)
of
G ft cltm C + 1stm @ C
h; or Istn @ tanh(C.).
where flatem slatum and obstm represent the input gate, the forget gate, and the output gate. R represents the input, and Ct-1, h2-1 E Rddddded represents the input cell state and the input hidden state. W/stm E Rdmader x ((dmoder din) are weight matrices and 6lstm E [gdwode are bias vectors. Ce and he denote the output cell state and the output hidden state; C₁ represents the intermediate variables of the calculation; @ denotes an element-win wise product; a(-) and tanh(-) denote the sigmoid and tanh activation function.
The structure of GRU, depicted in Figure b(b), consists two gates: the reset gate and the update gate. Besides, GRU has vector: the hidden state. It can be mathematically formulated as follows:
GRU
gru W 11-1, + baru)
u (Waru
[ + baru)
(7)
tanh
W .m @ h1-1, + (
h (1 u gru c h + u C hgru
13

---

where r and "t represent the reset gate and the update gate. x E R represents the input, and h4-1 E Rdwodet represent the input hidden state. W E Rd ((dmodet are weight matrices and 1hth Rddnodsl are bias vectors. hy denotes the output hidden state; hip gru represents the intermediate variables of the calculation; @ denotes an element -wiee product; o(-) and tanh(-) denote the sigmoid and tanh activation function.
A.2 Transformer-baze based methods
In recent years, variants of Transformer have made significant progress in time series forecasting and can be divided into two categories.
The first category captures the correlation between different time steps through point-wise attention mechanisms. Vanilla Transformer [Vaswani et al. 2017 is able to effectively extract correlations between any two-time steps through its self attention mechanism, but its complexity is too high, reaching O())). Informer [Zhou et al. 2021] proposes ProbSparse self-attention with distilling techniques, reducing the complexity to OL log L). FEDformer (Zhou et al. 2022a] takes a different approach by applying attention modules in the frequency domain through Fourier or wavelet transfor- mations, achieving a complexity of (L)) Despite the performance and efficiency improvements achieved by these representative methods, they face difficulties in extracting sufficient semantic
information from a single time point, which has been raised in previous works [Nee et al. 2023 Wu et al. 2023
The second category adopts non-dot-product techniques to extract correlation information. Log- Trans (Li et al., 2019] effectively captures local information by leveraging LogSparse and convolu- tional self-attention layers. However, overlooks the long lingttrmmnn repetitive pattern of points among subseries and exhibits a high complexity of O(L L). Drawing from the traditional ideas of time-series analysis, Autoformer (Wu et al. 2021] is able to capture long-range trends and short- range changes in sequences through decomposition and auto-correlation mechanisms. However, its complexity remains high O log I.). Pyraformer [Liu et al. 2021] initializes coarser-scale node information in the pyramidal graph by capturing local and global correlations through convolution. then employs pyramidal attention to effectively capture long- and short-term repetitive patterns, leading to significant performance improvements while reducing complexity to OL). However, Pyraformer still suffers from the limitation of the receptive field of the convolution kernel as discussed earlier. PatchTST [Nie et al., 2023 captures local semantic information through patch, and further reduces the complexity to O (L)S)²), where S represents the stride length. Nevertheless, it does not consider the long-term repeated pattern among subseries.
Based on the above analysis, we find that most Transformer-based methods struggle to simultaneously effectively model local and global correlations (global-local semantic information) and capture long- and short-term repetitive patterns (periodic semantic information) in sequences. Furthermore, is worth noting that the majority of the aforementioned methods fail to achieve lower complexity than RNN-based methods. For a comprehensive comparison of the theoretical time and memory complexity, well experimental results, please refer to Appendix
B Inspiration and Our Approach
RNNs, such as LSTM and GRU, have significant advantage in capturing semantic information due to their recurrent structure, discussed in Appendix A.1 Yet RNNs still have some drawbacks. In addition to the well-known issues of gradient vanishing/exploding [Pascanu et al.) [013). This paper focuses on analyzing the information forgetting problem (Appendix B.2) and the difficulty in parallelizing RNN (Appendix C). In this section, we will analyze RNNs using LSTM and GRU as representatives, and introduce the inspiration we draw from during the process of designing our model.
B.1 LSTM VS GRU
LSTM incorporates three gates (input gate, output gate, and forget gate), while GRU utilizes two gates (reset gate and update gate). The complex gating mechanism L.SM enables it to capture and learn intricate patterns, making it effective for processing longer sequences. On the other hand, GRU
14

---

performs well on shorter sequences and requires less computational and memory resources compared to LSTM. It should be noted that, shown in Equation 7. the computation of h in GRU is dependent on the completion of re computation. This undoubtedly slows down the computation that involves the input information x. Therefore, in terms of actual running time, GRU may not necessarily have an advantage over LSTM, as detailed in Appendix C.2
B.2 The information forgetting problem of RNNs
When Recurrent Neural Networks (RNNs) process sufficiently long sequences, the information contained in the earlier parts of the sequence may forgotten within the recurrent structure as passes through multiple cells. To verify this phenomenon, we conducted a concrete experiment. We conducted a benchmark experiment using the ECL dataset, with historical sequence length of 720 and prediction horizon of 720, and the description of the ECL dataset can be found in Appendix E
To eliminate the influence caused by the split of training, validation and test sets, we partitioned the dataset with fixed input sequence length of 720 and prediction horizon of 720. Furthermore, we conducted experiments with varying input sequence lengths using LSTM and GRU models. For the experiments, we used the last m = {24, 48, 72, 696, 720) points near the prediction time as input, and the results of these experiments are shown Figure
GRU
0.70
LSTM
0.50
0.55
lnngth
Figure The issue of information forgetting in LSTM and GRU.
Observing Figure 7 on the one hand, we can see that when the input sequence length is relatively short (not within the red box), the prediction performance of LSTM and GRU gradually improves as sequence length increases. This indicates that longer input sequences can provide more contextual information, which can enhance the model's prediction performance and is consistent with previous research findings [Liu et al. 2021 Zeng et al.) 2023). On the other hand, the part within the red box indicates that when longer historical sequences are fed into LSTM and GRU, their prediction performance does not continue to improve. This suggests that introducing earlier information does not provide significant benefits for the models, which in turn confirms the problem of information forgetting in RNNs.
B.3 Inspiration and WITRAN
Our proposed Water-wave Information Transmission (WIT) framework utilizes input rearrangement to capture both long- and short-term repeating patterns the sequence more directly and efficiently, while avoiding the processing of excessively long sequences, as described in Section 3.1 This successfully addresses the limitations of RNNs, such as the problems of exploding and vanishing gradients, well as information forgetting. Furthermore, through the design of Recurrent Accel- cration Network (RAN), we enable parallel computation For comprehensive understanding of RAN, we provide detailed code and analysis of
the entire sequence, as described in
Section 3.3 RAN in Appendix D this subsection, we focus on introducing the inspiration behind the important components of WIT.
The specific structure of WIT has been detailed in Section 3.2 The inspiration behind the design of Horizontal Vertical Gated Selective Unit (HVGSU) and Gated Selective Cell (GSC) comes from the advantages of L.TM and GRU. HVGSU serves two major purposes: 1) capturing both long- and short-term periodic semantic information in the sequence, and 2) modeling these two types
15

---

of periodic semantic information simultaneously for each time step. To achieve these objectives, two separate GSCs are established in the HVGSU, each responsible for the fusion and selection of long-term and short-term information in both directions.
For each GSC cell, to selectively incorporate fused information into the periodic semantic information in each direction, we draw inspiration from the reset gate of GRU and design a selection gate to control the retention of original information and the addition of new information. Furthermore, we are inspired by the output gate of LSTM and design our output gate to output the updated information. To reduce computational costs while maintaining sufficient learning capacity, we draw inspiration from GRU and only utilize these two gates, resulting in smaller computational and memory overheads. However, we address the problem of waiting for information be computed between gate calculations in GRU by parallelizing the gate calculation vectors with the input z. which greatly accelerated the computation speed of the cell. In summary, we analyze and draw inspiration from the strengths of LSTM and GRU, and incorporated them into the design of our GSC.
B.4 WITRAN VS other RNN- models
There are also several other RNN models, such as DilatedRNN [Chang et al. 2017 SlicedRNN YYu and Liu 2018). and
(Shen et al. 2018), which also have certain advantages in handling sequences. To further compare the differences between WITRAN and these methods, we provide the
comparison in T
Table 6: Advantages of WITRAN compared to other RNN methods.
Advantages
DilmtedNN
SlicedRNN
RIMP-LSTM
WITRAN (ours)
Fificiently (1 layer) model glabal comelations
(1)
((1)
Special design to capture long term repetirive putterns
/
x
Using layer to cupture long- and short-tune repetiive patterns simultanceesly
x
Well solve the grudient sanishinglexploding problem of RNN
/
/
(1) Efficiently (1 layer) model global correlations (a) When the dilations of DilatedRNN does not include the value multiple layers need to be constructed extract global correlations. (b) SlicedRNN improves efficiency to some extent by parallel processing of minimum subsequences, but still requires the introduction of multiple layers to capture the global correlations of the sequence.
(2) Special design to capture long-term repetitive patterns (a) SlicedRNN is unable to capture long- term repetitive patterns among elements of sub- uub-sequences. (b) Although RIMP-LSTM incorporates Residual Paths and Residual Sum Unit designs, it still cannot effectively extract long-term repetitive patterns.
(3) Using layer to capture long- and short-term repetitive patterns simultaneously (a) Dilate- dRNN can capture long- and short-term repetitive patterns, but it requires the use of multiple layers to achieve this. (b) SlicedRNN and RIMP-LSTM are not particularly adept at handling long-term repetitive patterns, mentioned in (2).
(4) Well solve the gradient vanishing/exploding problem of RNN (a) In DilatedRNN [Chang et al. 2017], it states and provides formal proof that reducing the length of information paths between time steps can prevent the issues of gradient vanishing/exploding. DilatedRNN, SlicedRNN, and WITRAN tackle this problem by reducing the length of information transmission paths. (b) RIMP-LSTM addresses this issue by the designs of Residual Paths and Residual Sum Units.
C Time and Memory Consumption of RNNs and Transformer
In this section, we derive the complexities of LSTM, GRU and Vanilla Transformer respectively, and analyze their actual running speeds and memory usage through experiments.
C.1 The theoretical computational complexity
The structure diagrams of L.STM and GRU are shown Figure 6 while the structure of Vanilla Transformer can be found in paper /Vaswani et al. 2017). Assuming the sequence length is L, the
16

---

input dimension is d and the hidden layer dimension is dmodel. and considering the case where each model has only one layer, the derivation process is as follows:
GRU GRU includes two gates and three matrix multiplication operations, as mentioned in Ap- pendix A.I. Each matrix multiplication operation has a complexity of dmodel x (dmodel + dm). although there are also @ operations GRU, and the complexity of element-wise multiplication is dmodel, which is much smaller than the complexity of matrix multiplication operation. Therefore, the complexity of one GRU unit can be simplified as processes the entire sequence, it needs to perform I, sequential computations, therefore, the overall computational complexity can be summarized as: 3 x L x dmudel (Amodel + din).
dmodel X (dmodel + dn,) When GRU
LSTM LSTM includes three gates and four matrix multiplication operations, as mentioned in Appendix A.I Similar to GRU, the computational complexity of LSTM can also be summarized as: 4 X L x mmodel (dmodel + dmn).
Transformer Taking the single-head attention mechanism in Transformer as an example, in the calculation of the attention mechanism, the each linear transformations of QQery((Q., K Key( K) and Value(V) require matrix multiplication operation of size dmodel dmodel- In the calculation of the attention weights, the Q and transpose of K are multiplied together, and since the size of Q and K are L dmodel. the complexity of computing the attention weights is L dmodel X L. and the size of attention weights is L x L. Similarly, when we multiply the attention weights with V in the computation, the complexity is L x L x dmodel. The output from the attention matrix multiplication is further linearly transformed, and thus requires an additional dmodel X dmodee complexity. To summarize, the overall complexity of the mnaee attention computation Transformer can be expressed as: 1) Linear transformations of Q, K and V: 3 x I, x dmoddl X dmmodel- 2) Calculation of attention weights: I. clmodel x L.. 3) Matrix multiplication of attention weights and V: Lx I. dmoddl- 4) Linear transformation of the output: T. x dmodel x dmodel- The total complexity of the single- headed attention computation in Transformer
x T. x dmodel X dmmodel + I, dmodel x I, + T. dmmoeel + 2 x I,2 &modeel. It should be noted that
L dmodeel + L dmodel X dmodel the complexity improvements in Transformer-base methods only apply to the L regularization term, as the linear transformations are necessary for any Transformer-based model. In addition, assuming the dimension of the Feed-Forward Networks (FFN) in Transformer is de, the first layer of the fully connected network has a complexity of T. x dmoded X dæ, while the second layer has a complexity of I. x dg x de, and the total complexity of FFN is L x aanddel X dg + should be noted that the input and output transformations in Transformer still require a certain level of complexity. On the hand, as needs to transform the input data from dimension dn to dimension dmodel. the input transformation in Transformer requires at least L x dnn x dmodel complexity. On the other hand, as it needs to transform the input data from dimension du to dimension dmoodet the output transformation requires a L x dg x dmodel complexity. Therefore, the overall complexity of Transformer can be summarized as: I x din x dmodel + 4 x I x dmodel 2 + 2 x T x dmodel + T. x dmodel x dg + L da² + L x da x dmnddel should be noted that most cases for Transformer, di is larger than dmodel.
dg²
Summary In time series prediction tasks, the input dimension dı is usually much smaller than the model dimension dmodel- Based on the analysis conducted earlier, we can observe that the computational complexity of the encoder part of most transformer-basee methods is generally higher than that of LSTM and GRU. This is because Transformer includes many complex operations such as matrix multiplication and linear transformation. Although there have been improvements made to the attention mechanism, these modifications only affect a small portion of the overall complexity which remains high. Therefore, compared to LSTM and GRU, the computational complexity of Transformer is higher.
C.2 The practical time and memory consumption
In practical situations, LSTM can perform computations in parallel without the need to wait for information from all four matrix operations. On the other hand, in GRU, the first two matrix operations can be parallelized, but the third matrix operation has a dependency on the completion of the previous operations, mentioned in Appendix A.I Therefore, the actual runtime of GRU may be longer than
17

---

Algorithm 1 GRU based on Python
import torch
import torch.nn as
import ft.cc.nnnnnnninnaal
class BaannlGRRnnnnnnniiiii
def Beef__nit_____ (scl, input_size, size, hidden_size, dropout):
super(Manual_ GRU, self). inii
self.hidden size hidden_size
self.dropout dropout
self. gates = gonLLinerrinpuusszzehhdeeee
self.gates
size+hidden size, hidden size+2)
selfhidddentraasffrmm = nn. Lin.aarrrii sizzehidppttsszzeehiddeenn size+hidden size, hidden_size
transform
sigmood Sigmoid()
self.tanh nn.Tanh()
for param self, parameters():
if param.dim() 1:
an.init.xavier (afformm(parann
def forward(self,, x):
batch_size x.size(O)
seq_len = x.size(
= torchh.croosbattchhh size, self.hidden bizidennsiiee..ooxddveee))
y_ v_list=
in range(seq_len):
update_gatt gatee, reset_gate = self. wat.ssttooc...aaa((:: i, h], dim=-11).chnk((2 -1)
update__atee gale. reset gate (self sigooddgg for gate in (update gate, reset_gate))
candidate hidden tal... hidden, hidden cunsform((orch.ca((((... :], reset gate h], dim=-1)
= (1-update_ggee) _gate) * update_gart candidate_hiddee
y y_list.ppeend(h)
output F.dropoutttorch.stakk htapcc.sstck(y__iii list, dim=l), self.dropout, self. sclf.training) return output, h
that of LSTM. To verify this, compared the processing time and memory usage of single-layer LSTM and single-layer GRU on sequences of the same length using Python implementation, as shown in Figure S
MiMaaay Sood
Ruadas
Figure 8: Time and memory consumption of RNNs and Transformer.
Transformer has the advantage parallelizing the computation of correlations a sequence of length I., which greatly improves its speed compared to GRU/LSTM. It is important to note that this comparison is not entirely fair, as Transformer can parallelize at the sequence level, whereas RNNs parallelize at the batch level. Each approach has its own strengths and is suited for different scenarios.
To ensure fair comparison of runtime and memory usage, it is important to consider the conditions under which the models are compared. Following the experimental setup in FiLM [Zhou al. 20226), we can ensure that the hidden state dimension and batch size of the models being compared are the same. This allows us to evaluate whether the models can be trained on devices with low memory capacity. For time evaluation, we need to ensure that the hidden state dimension is the same and the memory utilization similar, so that memory resources are not wasted. terms of time comparison between Transformer and GRU/LSTM. although GRU/LSTM slower than
18

---

Transformer under the same batch conditions in Python implementation, they can still speed up their parallel processing by increasing batch size while ensuring similar memory usage to Transformer. At the same time, because the computation process of GRU and L.TTM is much simpler than that of Transformmer, they can be more easily implemented in C++ and integrated into PyTorch's 's library as nn. Gm.GGUU/nn.LLTT
which is faster. For comparison, we compared the memory and time costs of these five methods, as shown Figure [ The Python implementation of GRU and LSTM code is shown in Algorithm I and Algorithm 2.
Algorithm 2 LSTM based on Python
import torch
import torch.nn import torch.nn. fnf.nccioonlllll
class Manual L.S. (odule))
def init (self, iiput_size, hidden_siize dropout):
super(Manual LSTM, self). init
self.hidden size hidden size
self.dropout dropout self.gates = nn. Lineaararii,put, hideen size, hidden size * 4) sigmood Sigmood)
self.tanh an.h))h()
for param in self parameters():
if param.dim() 1:
nn.init.xavier caaiiier_nnffommpreeeee
def forward(self., x):
batch size x.size(O)
seq_len
seq len x.size(1)
h, c (torch..eros(batcch size, self.
for in range(2))
y_list=11
for in range(seq_len)
Borget_gate__ input gate, output gate, candidate cell = self self.gates(torch.ca(((f.. :], h), dim=-1)) .chunk(4, -1)
forget_gate, input_ gate, output_gatee gate (igoossig)
for g in (forget_ gate, imput_gate, output gate))
c= forget__ate a c input input__ate gate seffttah(candddate_cel))
h = output_gate self.tanh(c)
y y_listaappend(h)
output adropouu(torch.stackk list, dim=1), self.dropout, self. telf.training)
return output, c
D The Design Process and Implementation Code of Recurrent Acceleration Network (RAN)
In this section, we will provide detailed explanation of the implementation approach (Appendix D.I and specific code of RAN (Appendix D.2h.
D.1 Implementation approach of RAN
we introduced the Recurrent Acceleration Network (RAN) as part of our overall approach. However, we encountered challenge when the number of rows processed by RAN is smaller than the number of columns. In such cases, if the number of slices processed exceeds the number of rows, the intermediate variables of the slices become fully occupied. To ensure the continuity of operations, it becomes necessary to continuously shift and reset these variables. However, we found that this reset operation was time-consuming and hindered the expected acceleration benefits of RAN. To overcome this issue, we made simple adjustment by setting the size of the slice to the maximum value of the rows and columns. While this may introduce additional space complexity, it maintains the length of the slice and eliminates the need for reset operation. As result, the
In Section 3.3
19

---

processing time is significantly reduced. The comparison diagram between RAN-min and RAN-max shown Figure
Zimo
RAN-min
(b) RAN-max (in WITRAN)
Figure The figure shows two designs RANs side by side. The left one represents the case where the number of slices is the minimum value between the number of rows and columns, while the right one represents the case where the number of slices is the maximum value between the number of and columns.
D.2 Specific code of RAN
The specific code of RAN can be found in Algorithm 3. Due to its long content, place the end of the Appendix.
Dataset Details
In this section, we will provide detailed introduction of the datasets used in this paper. (1) Electricity (ECL) contains the hourly electricity consumption of 321 customers from 2012 to 2014. (2) ETT Zhou et al. [021] contains the load and oil temperature, which was recorded every 15 minutes between July 2016 and July 2018 from electricity transformers. (3) Traffic TrafffeCoontinns contains the hourly data the road occupancy rates measured by different sensors on San Francisco Bay area freeways, collected from California Department of Transportation. (4) Weathe contains 21 metcorological indicators (such as air temperature, humidity, etc.) and was recorded every 10 minutes for 2020 whole year. They are all split into the training set, validation set and test set by the ratio of 6:2:2 during modeling.
Due to the different original acquisition granularities of each dataset, in order to ensure that they contain the same semantic information on the same task, in this paper, our experiments are conducted by aggregating them into one hour. Table 7 summarizes all the features of the four benchmark datasets. The target value of ETT is 'oil temperature (OT)', the target value of ECL is 'MT_320" the target value of Traffic is 'Node 862 and the target value of Weather bulb''
Table 7: The details of datasets.
Dataset
Sample Length
Dimension
Usage Frequency
Recorded Frequency
ETT
17420
1h
15min
ECL
26304
322
Ih
1h
Traffic
17544
863
1h
1h
Weather
35064
22
1h
10min
https://archive. http::/wwww...i....c....ececceee wwtp.u....ceee/eeeeeeeeeeeeeeeee
http:://eem: wwtppem..dot.com dot..c..ga........ gov
https://www.ber mpg
dewette/
20

---

F Experiment Setup Details
Our method and all the baseline methods are trained with the L2 loss, using the ADAM [Kingma and Ba [2144] optimizer with an initial learning rate of 10-3. Batch size is set to 32. The training process is early stopped after epochs there is no loss degradation on the validation set and the max epochs is 25. We save all the models with the lowest loss on the validation set for the final testing.
To ensure a fair comparison of each model's performance, we set the same search space for the common parameters included in each model. This ensures that each model can achieve its best performance under the same conditions for comparison, with the seed set 2023.
Regarding the private parameters of each model, we set the corresponding search space according to the specific descriptions given in their respective papers. In addition, it should be emphasized that the performance of Pyraformer shown in this paper is the best between the two prediction designs of the model, the performance of FEDformer shown here is the best between FEDformer-w and FEDformer-f, the performance of DLinear shown here is the best among DLinear, NLinear and Linear, and the performance of MICN is the best between MICN-regre and MICN-mean.
The mean square error (MSE) and mean absolute error (MAE) are used as metrics. All experiments repeated 5 times and the mean of the metrics is reported as the final results in Table and Table 4
All the deep learning networks are implemented using PyTorch Paasze et al. 2019 and conducted on NVIDIA RTX A4000 16GB GPUs.
The search space for the common parameters is follows: (1) dmodel is set {32, 64, 128, 256, 512, 1024). (2) élayer and dlayer are set to [1, 2. 3). (3) Thead is set to {1, 2, 4, 8) and dir is four times large as dmooeel- Regarding the private parameters of model, norm set to [0, 1) and C represents the period is set to (12, 24, 48| and it is required that the length H of the input sequence divisible by C. Furthermore, it is important to note that adjustments should be made to C for different tasks to ensure its ability to adequately represent the period.
G Ablation Study
In Section we presented the results of our ablation experiments to demonstrate the effectiveness of each component in the Water-wave Information Transmission (WTT) framework. We design five variant methods and evaluate their performance four datasets. The experimental results are summarized in Table (, and we will provide a detailed analysis of these results in this section. is important to note that the role of the Recurrent Acceleration Network (RAN) is primarily to improve computational efficiency, as demonstrated in Section 4.2, and does not significantly impact the model's isss accuracy. Therefore, our focus in this section will be on discussing the design and role of the WIT framework.
G.1 Impact of forecasting module
Our proposed forecasting module is also a part of the model design, and its goal is to effectively capture periodic semantic information and global-local semantic information obtained through the Water-wave Information Transmission (WTT) framework. Our specific design has been described in detail in Section 3.4. To verify that we can fully utilize the relevant information by combining operations for pairwise prediction, we changed the combination prediction method of this module to a simple way of first concatenating the horizontal hidden state and the vertical hidden state, and then using the fully connected prediction module for prediction. We named this variant WITRAN-FC.
The comparison of results between WITRAN and WITRAN-FC in Table & shows that when we remove the combining operations for pairwise prediction and replace it with ordinary concatenation and fully connected operations, even though the same information is used, WITRAN-FC has difficulty discovering the most relevant long-term periodic semantic information for each prediction point. This demonstrates the effectiveness of the combining operations for pairwise prediction the forecasting module.
21

---

Table 8: Results of the ablation study on long-range and ultra-long-range forecasting tasks. A lower MSE or MAE indicates a better prediction. The best results are highlighted in bold.
Methods
WITRAN
WITRAN-FC
WITRAN-2DLSTM
WITRAN-IDGRU
WITRAN-LSTM
WITRAN-GRE
Metric
MSF
MAE
MSR
MAF
MSE
MAE
MSF
MAE
MSE
MAF
MSB
MAR
168-168
0.2397
0.3519
0,3005
0.4029
0.2662
0.3736
0.2578
0,3677
0,2749
0.3800
0.2536
0.3640
168-336
0.2607
0.3721
0.3425
0.4355
0.3027 0.3996
0.2916
0.3924
0.3164
0.4087
0,2872 0.3863
336-336
0.2517
0.3627
0.3242
0.4232
0.2916 0.3896
0.2826 0.3851
0.3151
0.4066
0.2811 0.3805
336-720
0,3084
0,4055
0,3566
0,4523
0.3279 0,4262
0.3172 0,4187
0,3435
0.4350
0,3123 0,4153
720-720
0.2478
0.3651
0.4544
0.4972
0.3120 0.4184
0.2948 0.4060
0,3362 0.4325
0,2912 0,4009
720-1440
0.2499
0.3727
0,4732
0.5277
0.3426 0,4505
0.3337 0,4440
0,3517 0.4540
0.3296 0,4414
1440-14440
0.2408
0.3680
0.6433
0.6172
0.3954 0.4876
0.3822 0.4819
0,4164 0.5006
0,3864 0,4843
1440-2880
0,3359
0.4383
0.7406
0.6906
0.5936 0.6215
0.5680 0.6077
0.5583 0.6005
0.5859 0.6173
168-168
0.1377
0.2051
0,1541
0.2382
0.1918 0.2811
0.1824 0,2699
0,2065 0.2935
0,1846 0,2666
0.1321
0,2059
0,1572
0,2070 0.2975
0.1945 0.2837
0,2201
0,2059 0.2887
336-336
0.1306
0.2054
0,1549
0.2455
0.1923 0.2846
0.1808 0.2732
0,2029
0.2935
0,1840 0.2715
Tralkc
336 720
0,2175
0,1725 0.2611
0,2087 0,3000
0.2042 0,2958
0,2197 0.
0,2072
720-720
0.1408
0.2191
0,1722 0.2622
0.2157 0.3048
0.2097 0,2997
0,2242 0.3123
0,2150 0.3023
720. 1440
0.1672
0,2449
0,2178 0.2979
0,2404 0,3178
0.2450 0.3245
0,2586
0,2501 0.3264
1440-1440
0.1543
0,2325
0,2150
0.2954
0.2343 0,3148
0.2340 0,3151
0,2406 0.3199
0,2469 0.3249
1440-2880
0.1425
0,2333
0.2072 0.3082
0,2060 0.3081
0.2134 0,3182
0,2113
0,2129 0.3162
168-168
0.1105
0.2589
0,1143
0.2649
0.1132 0.2624
0.1120 0,2622
0,1141 0.2642
0,1141 0.2643
168-336
0.1189
0.2714
0.1211 0.2753
0.116U 0.2694
0.1172 0.2710
0.1191 0.2733
0.1173 0.2708
336-336
0.1112
0.2638
0,1167 0.2715
0.1125 0.2668
0.1125 0.2668
0,1147 0.2696
0,1138 0.2676
BTTh:
336-720
0.1494
0,M192
0,1347 0.2933
0.1317 0.2905
0.1314 0.2902
0,1310 0.2900
0,1317 0.2906
720-720
0.1296
0,2868
0,1515 0.3109
0,1438 0.3037
0.1436 0,3034
0,1450 0.3048
0,1513 0.3111
720- 1440
0.1331
0,2943
0,1807 0.3378
0.1759 0.3361
0.1764 0.3375
0.1766 0.3361
0.1791 0.3391
1440-1440
0.1304
0.2902
0,2627
0,409
0.1861 0.3439
0.2063 0,3616
0,1969 0.3562
0,2123 0. 0.3656
1440-2880
0.1850
0.3452
0.2655
0.4134
0.2485 0.3998
0.2463 0.3925
0,2444 0.3923
0,2486 0.3962
168-168
0.2389
0,3813
0,2507
0.3859
0,2577 0,4012
0.2567 0.3998
0,2593 0.4013
0,2556 0.3980
168-336
0.2277
0.3778
0,2377
0.3896
0.2791 0.4193
0,2791 0.4193
0,2818 0.4211
0.2690 0.4108
336-336
0.2432
0,3922
0,2928
0,4275
0,2614 0,4085
0.2621 0,4084
0,2680 0.4130
0,2604 0,4058
BTTb:
336 720
0.2373
0,3888
0.2948
0.4355
0.3051 0.4434
0,2815 0.4253
0.3113 0.4458
0.2798 0.4239
720-720
0.2635
0.4018
0,3243
0,4548
0.2990 0,4369
0.2848 0,4283
0,3096 0.4428
0,2832 0,4262
720- 1440
0.2915
0,4289
0,3212
0.4598
0.3308 0.4659
0.3396 0.4723
0.3241 0.4617
0.3446 0.4759
14400-4440
0.2815
0.4220
0,4162
0.5145
0.3950 0.4962
0.4214 0,5144
0,4199 0.5132
0,4243 0.5193
2580
0.3280
0.4585
0.5532
0.5959
0.6846 0.6639
0.6554 0.6530
0.6531 0.6581
0.7473 0.7006
168-168
0.2050
0.3338
0,2066
0.3324
0,2180 0,3419
0.2199 0.3437
0,2194
0.3435
0.2211 0.3462
168-336
0,2197
0.3470
0.2200
0.3585
0,2580 0,3789
0.2723 0.3871
0.2596
0.3794
0.2668 0.3843
Weather
336-336
0,2163
0.3482
0.2259 0.3586
0,2472 0.3730
0.2613 0,3816
0.2462 0.3726
0.2501 0,3732
336-720
0.2054
0.3424
0.2125 0.3497
0.2497 0.3840
0.2775 0.4004
0.2473 0.3796
0.2859 0.4099
720-720
0.2008
0.3417
0,2104 0.3458
0,2564 0,3843
0.2861 0,4030
0,2633 0.3873
0.2772 0,3990
720- 1440
0.1872
0.3312
0.1984 0.3397
0.2301 0.3676
0.2853 0.4156
0.2376 0.3751
0.2737 0.4063
1440-144 1440
0.1907
0.3366
0,2084
0.3507
0,2419 0,3832
0.2671 0,4070
0,2454 0.3847
0,2631 0,4013
1440-2880
0.1769
0.3257
0,2148 0.3570
0.2308
0.3735
0.2460
0.3887
0.2317 0.3736
0,2305 0.3751
G.2 Impact of information fusion and selection
As information fusion and selection are completed together in the Gated Selective Cell (GSC) we designed, we need to verify the effectiveness of GSC, that is, the effectiveness of information fusion and selection. Here, we replaced the cells in the two directions of Horizontal Vertical Gated Selective Unit (HVGSU) with LSTM and GRU respectively, and named them WITRAN-2DLSTM and WITRAN 2DGUU. should be noted that since the information between the LSTM/GRU cell in the two directions is not fused at this time, the information in the last row of the horizontal direction cannot contain global information, so we cannot use the proposed forecasting module for application, but only use the FC method mentioned in the previous subsection for prediction. It should be noted that the comparison between WITRAN- -EEEAAN-DDLSTTTT WWI.SSST/TTAARDDDOOOT impact of information redundancy on experimental performance, while the comparison between WEN.--DDSSTTIIAAANNNN
and WITRAN-FC reflects the
and WITRAN reflects the impact of information fusion and selection experimental performance.
The comparison of results between Table X demonstrates that too much information redundancy may be detrimental to the model's 's prediction performance, because for WITRAN-FC, the horizontal hidden states it receives contain more information as they go further down, and they also contain all the information in the horizontal direction above them. The same applies to the vertical direction, where the further to the right the hidden state is. the more information it contains. Furthermore, the information captured by WITRAN-2DLSTTE and WITRAN-2DGGU is independent among different horizontal rows and among different vertical columns, and the semantic information between horizontal and vertical
and WITRAN-FC in
22

---

hidden state are different. Therefore, WITRAN-2DLSTM, and WITRAN-2DGRU are not heavily influenced by redundant information. In addition, the comparison of results between WITRAN and WITRAAN2DD.STMWITTRAANDDDD
shows that information fusion and selection can effectively extract relevant information, which demonstrates the necessity of the GSC design in WTT.
G.3 Impact of bi-granular information transmission
In the WITRAN we designed, information can be propagated in two directions, and because the time granularity of sequence propagation in these two directions different, their periodic semantic information is different. In order to verify the effectiveness of the two independent GSCs set up HVGSU to capture long- and short-term periodic semantic information, we replaced the cells in the two directions of uniformly used the hidden state information of one point to transmit information in both directions. should be noted that at this time, the information propagated horizontally and vertically by the model completely the same. Sure, we can name the current models as WTTRAN-LSTM/WITRA.
with single LSTM/GRU cell, and
The comparison of results between LSTM/WITRAN-GRY in Table demonstttes that failing distinguish between long- and short- term periodic semantic information can have significant negative impact on the performance of prediction tasks.
and WITRAN-
G.4 Summary
In this section, we focused on the importance of each building module of WIT. During the experi- ments, we gradually removed the building blocks we designed, and was clear that the performance decreased significantly. This fully demonstrates the rationality and effectiveness of the WIT frame- work we designed. Specifically, the summary of WTT as follows: (1) The fusion and selection design of GSC enables both long- and short-term semantic information historical input data point. (2) Setting up independent cells in both directions enables to extract semantic information in the long- and short-term respectively. (3) The specially designed combined operation is able to make fuller use of the captured local and global semantic information while ensuring that the information is not redundant.
be captured for each
H Experiment Error Bars
We saved all models and used the model with the lowest validation loss for the final testing. We repeated this process times and calculated the error bars for all models to compare their robustness on long-range and ultra-long-range tasks, shown in Table It can be seen from Table that the overall performance of the proposed WITRAN is better than other state-of-the-art baseline models, indicating the effectiveness of our approach.
I Model Analysis
In this section, we will analyze the parameter sensitivity (Appendix [.I of our proposed model and provide a detailed explanation (Appendix [.2) of the time and memory consumption in Section 4.2
L1 Parameter sensitivity
Regarding the private parameters of our model, norm is set to (0, 1} and C' represents the period set [12, 24, 48) and it is required that the length H of the input sequence is divisible by C. In this subsection, we will explain the choices of these parameters and their impact on the model's predictions.
The impact of norm For different prediction tasks on different datasets, we determined the value of norm through validation set. To verify that the value of norm conforms to the distribution the datasets, we conducted experiments on the distribution of the training and validation sets for different prediction tasks, the results and the value of norm are shown in Table TO We noticed that although there may be a significant difference in the mean values of the data between the training and validation
23

---

Table 9: MSE and MAE with error bars (Mean and STD) for WITRAN and all the baseline methods for long-range and saira-long--anee forecasting. All experiments are repeated 5 times.
BCL
Tralke
ETnd
Meatic
MBC
MAP
MSB
MAB
M5B
MAB
MAB
ASSE
WITRAN
0LT22100 30783
B.20T 18835
87165-2
1229993300622
0.36725003325
723-735
BOASSUO D0346
0 0340724601159
1640-3443
30133
51300450.000 20425
08423085011177
6110723300622
0 0.338588005640
1440-2800
033994000 107
0.302-00155 546
8:250-0 04596
R1779-3100090
165-158
0.007 0.31520011144
0.20132001114
B2754229.0995
02201:530021
O 0.44F2R000006
1006-
6745
0-7001 menta
326-130
0.37224044881
730 1640
(39991320017
C 3444435-22222
1640-2442
123022 13583
01045
30100
5.6615
168-158
0.37% 1005s
12500
0.3733 bi
07521 03151
0 1589 10
336-215
3535
LUNC 18073
0.50924000100
026645400566
1440-948
10555
0:309-0000
00853
01700-33000 -3033
0.463-600000
116-108
1622
0.350 DI
0:3000 330069
160-215
04515200011 190
PARISE
176-130
00701
0776 11666
B.571-30 15571-314611
13506.01100 10080
0 1100211
51458
336-115
1335 10C30XX
D230s5 D0150
044422001135
03724:31065
30644
730
09436
04335
0.25442002510
03199:314555
0407126035022
0.54446000000
0 27988LHOONI
04152
D6072
0.3032200000
B4158 09539
0.5922000000
BulnN:: 10683
558
0.116 EvRCss
77
0.31995000621
0.59922009755
0 0298024000276
166-208
61754.0
01255-001111
0377446000995
156-118
D1400
0.284401 00145
0,291 20646
O.
120-933
(L30g5 DICSS1
0,512 10002515
723
E5573100000055 04035
01200-3 10630
0.38-4100115
000183
0-1500 102551
0
17167-11 76026
4 0000 10/000
03130-31655
O 0.550 1101102
Pysiects:
336-155
148-315
ELSOASL 10543
LUSTTES 10632
0L538210.167 1656
0.600-00000
E2200.0 10900
O 12837HHGSEE
700 1200
2
82205-00 04545
01415: 2015:00
O CTTTT LUGIUM
0505220 $554
25155002220
00392
0.5442600005
160-208
O 028172M00117
65130-800
356-518
039963301515
C CA18220-5--00
07500=0
738 Frth
00000124315 43154
O 0TTE4EHL1955
12519
20444
20515
27321
12000:33 11422
165-114 115
6500 D47e
B.366-0 20116
13831-001619
87766-3 p064
O 200 1-007775
10622
0412344221001
06566
10673
83572-0
0U3052221110 19254
E5088.2 00139
13344-213313
13335
0000000 10000
01355-004177
06314:0 80965
11330-0 09937
09365
0250e:3 10559
0 0.963--000600
1440-3897
13:04
16644
044621000066
02126-200093
Do00- 001701
E50044 1001
1100-00000
0 01136-00015
168-335
0091550-15500
0407120001-00
0
013041005100
14276
07556-3 m1197
730-140 1440
0.797--000 (0357
665666 988
87365-0 1064
1905-0 CON1
8.312-0 006.00
0865-6097555
02188-5 (116)
D 35533-ndi-27
03992-210190
10330
0
1440-3882
0.530100829
0359010 5053C
036003802751-
93465 5833117
019905510844
0.34563000044
sets, when their variances are not significantly different, it indicates that the data fluctuations in the two sets are similar. In this case, the difference in the data distribution is not significant, and selecting norm as O is reasonable. However, when the variance difference between the training and validation sets is relatively large (approximately twice or half), there is significant difference in the data distribution between the two sets, and norm should be to re-normalize the input of the model. It should be noted that in the Weather dataset, there are negative values present, which makes its mean close to 0, resulting in a large difference between its variance and mean. However, this is reasonable. For the Traffic dataset, there are no negative values present, so even if the variances of its training and validation sets are similar, its significant fluctuations can be observed when combined with its mean. This was also demonstrated by setting the norm value to 1 during the model training process.
24

---

Table 10: The distribution of data in the training and validation sets (Mean and STD) and the value of norm.
Trals
Wecdee
Tadls
Ireid
271h
10010 3776
240 04 (477
176-99
14T 240 1166
N0-10 E134
00001 617)
16309971944
26 18.03011130
708-100
1407733-57.00 4765
3872796
14852.50907
3.48
21.11-11-00
163001719H
E70LUT0OM
1415-449
134 272-2/7771
1.80-00254
23912 7009
28 28-25--112011
11201
JASTE
-8
The impact of C The choice of C' for the model represents the determination of the sequence period. Due to the different original acquisition granularities of each dataset, in order to ensure that they contain the same semantic information on the same task, in this paper, our experiments are conducted by aggregating them into one hour. To facilitate the discovery of long-term repetitive patterns in the sequence, we set the C' to 12, 24, 48, and the specific experimental results are shown in Table II From Table [I] we can clearly see that for sequences with time steps in hours, using a period of 24 hours day) for division yields better results. This is because in WIT, information is transmitted horizontally at the granularity of hours, while information is transmitted vertically at the granularity of days. This approach can more fully capture the long- and short-term repetitive patterns and global/local correlations hidden in the time series data.
Table II: Parameter Sensitivity of C' on ECL and Traffic datasets.
Settings for C
12
24
48
Datasets
Tasks
MSE
MAE
MSE
MAE
MSE
MAE
168-168
0.2461
0.3648
0.2397
0.3519
168-336
0.3166
0.4230
0.2607
0.3721
336-336
0.3217
0,4099
0.2517
0.3627
0.2634
0.3847
ECL
336-720
0.3397
0.4310
0.3084
0.4055
0.3233
0.4130
720-720
0.2802
0.3951
0,2478
0.3651
0.3175
0.4063
720- 1440
0,2907
0.4051
0,2499
0.3727
0.2985
0.3946
1440-1440
0,3007
0.4110
0,2408
0.3680
0,2748
0.3768
1440-2880
0.4301
0.5150
0.3359
0.4383
0.3464
0.4347
168-168
0.1598
0.2408
0.1377
0.2051
168-336
0,2272
0.2783
0.1321
0.2059
336-336
0.2131
0.2967
0.1306
0.2054
0.1412
0.2278
Traffic
336-720
0.2022
0.2804
0.1391
0.2175
0.1510
0.2371
720-720
0,2028
0.2846
0,1408
0.2191
0.1505
0.2326
720-1440
0.2710
0.3311
0.1672
0.2449
0.1813
0.2605
1440-1440
0.2676
0.3301
0.1543
0.2325
0.1881
0.2652
1440-2880
0.2407
0.3443
0.1425
0.2333
0,1692
0.2765
1.2 A detailed explanation of time and memory consumption
The efficiency of the model is crucial, as even if the model's efficiency is high, it is limited if it does not have good predictive performance. Therefore, in this paper, we only compared our proposed method with methods that have good or the latest results. It should be noted that, for example, Transformer (Vaswani al. 2017], Informer [Zhou al. 202T] and Autoformer (Wu et al. 2021] have been shown in previous works to have higher complexity than FiLM [Zhou et al. [022b)] so we only compared our proposed method with FiLM, rather than comparing with Transformer, Informer and Autoformer again.
Memory Usage As shown in Figure 5fa) and (c). WITRAN has good memory usage with the prolonging the input length and output length. For a fair comparison, we fix the experimental settings of all methods, where we fix the input length as 720 and prolong the output length. Moreover, we fix the output length as 720 and prolong the input length. And for each model, to achieve good performance, it is important to have sufficiently large search space to select the most suitable parameters for the model, in order to achieve the best possible results. Therefore, we set all public parameters to their upper limits to test the memory usage of each model. To allow as many models as possible to be trained, we uniformly set the batch size to 8 for testing in this section. It should be noted that if point not shown on the graph, it indicates that the model encountered an "out of memory" (OOM) error at that particular configuration of input sequence length and output length.
25

---

As for TimesNet, due to its large memory usage, it cannot be trained even with a batch size of 1. Therefore, we did not include TimesNet in the Figure 5 From Figure[[a),, we can observe that when the input sequence is fixed, the memory usage of WITRAN and WIT' is not significantly different since they both have the same prediction module. Additionally, we found that as the output length increases, the memory usage of WITRAN and WTT also increases linearly, but our proposed method still has the lowest memory usage. From Figure Stc), we can observe that the memory usage of WITRAN increases very slowly with the increase in sequence length, and the memory usage of WITRAN is even lower than that WIT. This is because WITRAN does not need to retain large number of intermediate storage variables in the computation of each slice, while WIT needs to store the intermediate variables for the entire sequence. Through comparison with other methods, we can clearly see the advantages of WITRAN memory consumption.
Training Speed As shown in Figure b) and (d), WITRAN has a faster training speed than others with the prolonging the input length and output length. To ensure fair comparison of the performance of each model, we fixed the experimental settings, which were the same as the settings used for testing memory usage. The experiment is performed on ECL dataset. Due to extremely low space complexity, we followed the experimental settings used in FiLM [Zhou et al. 2022b) for comparing the training Speed. To fully utilize the GPU memory, our model 32 under the experimental settings mentioned above. Similarly, FiL.M can also increase the batch size 32. However,
increase the batch size from 8 to
still have the fastest training speed, and due the role of RAN mainly being on the input sequence, our advantage can be clearly seen from Figure dd), d), which is related to the square root of the sequence length. Through comparison with other methods, we can clearly see the advantages of WITRAN on time consumption.
J Showcases of Main Results
As shown in Figure 10 to Figure 49 we plot the forecasting results from the test set of datasets for comparison. For the long-range forecasting task, we chose two suboptimal models, FiLM and Pyraformer. As for the ultra-long-range forecasting task, we also chose two suboptimal models, Pyraformer and TimesNet. Our model WITRAN gives the best performance among different models. Moreover, WITRAN is significantly better at modeling global and local correlations, as well as discovering long- and short-term repetitive patterns in the time series than other models.
26

---

15
30
105-50
150
300
Figure 10: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-168 task of the ECL dataset.
27

---

16
20
30
PPN--OO C000
Figure 11: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-336 task of the ECL dataset.
28

---

30
Ma wwwaannama
366
A
16
Figure 12: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-336 task of
the ECL dataset.
29

---

M Mom0000 w WAlh Mane some MM
M MMM. Made w M wath A604
M M n MWA M ntn Wamm
Figure 13: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-720 task of the ECL dataset.
30

---

Mo holooooooossssssoe www.0000w0.0o.oo... M0lphhhhhh Myb M400,
Mox MMM Moweooooooooo Moy. Motohhh./hh Mo
Mor Mo Moos howhy MwW www. Ao0l
Figure 14: Forecasting comparison of WITRAN, FiLM and Pyraformer the 720-720 task of the ECL dataset.
31

---

LAllle.. MMl Mhl Mndd, Whon AMM ano AWW MAA HAA.AaAAHAAAAAAAAM
Figure 15: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 720-1440 task of the ECL dataset.
32

---

Figure 16: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-144 1440 task of the ECL dataset.
33

---

Figure 17: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-2880 task of the ECL dataset.
34

---

15
30
20
136
300
Figure 18: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-168 task of the Traffic dataset.
35

---

20
35
Figure 19: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-336 task of the Traffic dataset.
36

---

30
Figure 20: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-336 task of the Traffic dataset.
37

---

Figure 21: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-720 task of the Traffic dataset.
38

---

Figure 22: Forecasting comparison of WITRAN, FiLM and Pyraformer the 720-720 task of the Traffic dataset.
39

---

Figure 23: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 720-1440 task of the Traffic dataset.
40

---

Figure 24: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-1444 1440 task of the Traffic dataset.
41

---

Figure 25: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-2880 task of the Traffic dataset.
42

---

-12
-.4
15
20
20
105-50
300
mw
-12
-14
Figure 26: Forecasting comparison of WITRAN, FILM and Pyraformer the 168-168 task of the ETThl dataset.
43

---

MM
15
30
Figure 27: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-336 task of the ETTh1 dataset.
44

---

-44
-14
30
530.36
-42
Mady o Wo hmm wwm
-44
-32
412
Figure 28: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-336 task of the ETTh1 dataset.
45

---

330
26
$36-00
36
Posttionn
1M
Figure 29; Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-720 task of the ETTh1 dataset.
46

---

Malan and
maan Ja yro
Muay auhe
yaak Ja yo mm
650
-1.
-3.4
ma Ja V Whe map
-22
Figure 30: Forecasting comparison of WITRAN, FiLM and Pyraformer the 720-720 task of the ETTh1 dataset.
47

---

-23
9
-14
-24
0.4
11
024
31 1.4
Figure 31: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 720-1440 task of the ETTh1 dataset.
48

---

العممبا اها Trart
-10
Mall hyaata Ana
Man
No Matte
M
-1.0
Mann
how May dayly
No
1.0
Man
3-45
Na
Figure 32: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-1440 task of the ETTh1 dataset.
49

---

-14
wa Whnt Man
-20
2000
"allly yall Mhr
M Mywaor Wo Whal
-34
-34
100
-00
Wo Win Mun
-20
-40
Figure 33: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-2880 task of the ETTh1 dataset.
50

---

10
15
Figure 34: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-168 task of the ETTh2 dataset.
51

---

8
-45
-22
36
-25
1.2
-24
-24
16
-14
-24
Figure 35: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-336 task of the ETTh2 dataset.
52

---

30
530.36
30
Posttionn
Figure 36: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-336 task of the ETTh2 dataset.
53

---

Mww. M
Mesiction
Figure 37: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-720 task of the ETTh2 dataset.
54

---

dol Moun
-43
114
114
/ Mampan www.ina
2-10
31 114
Figure 38: Forecasting comparison of WITRAN, FiLM and Pyraformer the 720-720 task of the ETTh2 dataset.
55

---

Molhle W/
-40
Figure 39: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 720-1440 task of the ETTh2 dataset.
56

---

MMla-alll/h M Marllll
-24
WMl- M Manllly
-14
Wll- walll Many
-14
-25
Figure 40: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-144 1440 task of the ETTh2 dataset.
57

---

Wastiiion
Figure 41: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-2880 task of the ETTh2 dataset.
58

---

15
300
105-50
-24
156
300
1.4
Figure 42: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-168 task of the Weather dataset.
59

---

-04
30
-02
0.4
Posttionn
-40
-50
Figure 43: Forecasting comparison of WITRAN, FiLM and Pyraformer the 168-336 task of the Weather dataset.
60

---

-22
30
-20
300
-42
-34
-22
Figure 44: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-336 task of the Weather dataset.
61

---

Amn
20
Amn AMM Mi www.
160
Poatttonn
Amn Mmm
Figure 45: Forecasting comparison of WITRAN, FiLM and Pyraformer the 336-720 task of the Weather dataset.
62

---

paanmmonm MAR pom Mownt www.aaawwwwwaa.....oo.oooemaww m/ |
555-20
poo www. pam porso o Lyph palnnn
p0oo (appr/owoo Maso mom paasoin
1.3
33
Figure 46: Forecasting cases comparison of WITRAN, FiLM and Pyraformer the 720-720 task of the Weather dataset.
63

---

Figure 47: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 720-1440 task of the Weather dataset.
64

---

Yymo youn yowh /f Amrtt
10800
Ammoe yph Aww. Unn
Figure 48: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-144 1440 task of the Weather dataset.
65

---

Moa ulth m Wit
Figure 49: Forecasting cases comparison of WITRAN, Pyraformer and TimesNet the 1440-2880 task of the Weather dataset.
66

---

Algorithm Recurrent Acceleration Network (RAN)
import torch
import torch.nn
import math
import torch.nn.funcceee fn.fucciooalllll as
class WITRAN HVGSU Encoder(torrc.am..Moll
def jnin _selt_ input_size, hidden_size, num_layers, dropout, water_rows, water_cols, res_mode="
none"):
self)__nnit_
self.input size input size
self.hidden size hidden_siin size
self.num layer__ae-ss____ = numm_ayy num_layers self. dropout
celf.water_rowe rows water_rows
self.water__et cols water ccll
self.res_mode res_mode
porameter of row cell self. W
wtw.eeoo......aaaeeetotoooo Parameter(orrc..cppyy. hidden_size, input_sizee sopputsizee_ + 2 * hidden_size))
self.W other layer worch.nn.Paaameeee.............. Parameeer(oocch.nnpyyy layers 1, 6 hidden size, 4
hidden size))
self.B torch.nn. hawp:hhu...aaamtttetttcc.....mm
layers, hidden size))
self.reset_parmmettrrs)
def reset resee_pprrmccters(sefff.
stdv 1.0 math crrtoself.hidde__iin
weight in self. parameters:::::
weight.data.ne
www.ddaaaa.nn.o.m.mm www.uunnffommccc (-stdv, +stdv)
def linear(self, input, weight, bias, batch size, slice, Water2sea slice slm)e_num): F.innaariiputt, weight)
if slice < Water2sea slice num:
af af:batch_siin saatcc_ssiee-a * (slice + I), :1
* (slice + 1), :] + bias
return
def forward(self,, input, batch size, input size, flag): if flag = 1: cols > rows
input permutte, 0, 1,
else:
input input. permute(1, 2)
Water2sea_slice num_ Original_she Liiinall_siie_eeee [eiie_len____ input,shape
Satterseeassice, Jen = Water2sea sliceeceeassseeeeeeeeeeeeee slice num Original_slice_lnn sssicclllceeee
ciddden_slicenne = warchrreessateeeesssssssiiccc
batch_ self. seff.hidden_sizze.toonne
device)
hidden slice torch.zeros
num batch size, self.hidden sel..hiddenssize...ooo iompttedee
input transfer torch slice num, batch size, Water2sea slice len, input size) input.device)
for in range( Water2sea_sliee
pouut_tranffrfrrr ar++Origial_sllce_PPr :] inputir, v
hidden_rooo row all = hidden_coo col all list=
67

---

Algorithm 3 Recurrent Acceleration Network (RAN)
for layer in range(self.num) layers):
if layer = O:
batch_size, Sater2sea_slice_los
input_size)
self. W SitttWW_islayee
else:
F. F.dropout(outpu__o slice, self.dropout, self training)
if layer ==
layerO_ouuse cutt_ooupput
W = Soelf.WWoffee Seothf..Wotherr..ro _WW_ooth______ layerflarrrl :]:]
hidden_sllie slice sow hidden_ slice_rrooe *
midden_slice_coll
hiddennslicee___ee
B == BelfffB(layerr :]
start every for all slice output all slice =ist=
for slice in range (Water2sea slice_ len):
# gate generate gate wes..iinnatttrchccalhdddd slice_reeww row, hidden_slice_nneeeeeeeeee alice_col, al; slice, :]].
dim= = W, B, batch_size slice, Water2sea_slice slice_num)
# &gate
sigmod_ gatm tanh gate torch. splitggtlit self.hidden_sin size, dim -1)
sigmod_gate minchsigmmiidsigmmddd gate)
tanh tanh_ gate = torch.tanhe trcchtanh((anh, _aaee)
gate_col, update gate row. output gate row, update gate col, output gate col = sigmod gate.chunk
gate_col
(4, dim==11 -1)
input gate row, input gate col = tanh gate.chunk(2, dim -1)
# gate effect
(1-update_gate. gagat___oooshddddeesliice
+ update_ popdatt_gaccrrowiippptgggggg
output pouput_ggtteroot row
hidden Sallden_siice___ slice_col== torch.tanh(
(I-update gate__eeee coleeccccoeeendeeeee slicessleeeeeeee col + update gate gattlaalltegggege www-wooptttggeeeeeccc gategaatteea col)
output_gate__ool
# output generate
output slice torch torch.cat(lhidden leriidee__slieeeeeeee slice_row hidden colce_col,, dim -1)
# save output output # save row hidden
slice sllicellisttaeeee list.ppeendoouppussiees
if slice Original_stee saa___slce_ee__ 1:
need save row loc slice Original slice len
hidden row all list.append(
hidden slice row[need save httprrowwwoooleeeeeee locsbatch size:(need save row loc+1)* batch_size, :])
# save col hidden
if slice Water2sea_ slreessslee__m__eeeeeee 1:
hidden col_all list.append(
hidden_slice wwliceecollcaateeeeeeeee col[(Water2sea_slic -hbice_nmml)/baacch..o.. size:, :))
# hidden transfer
hidden_seee slicosssceeeeeeeee torch.rollhhidden. slice_col, shifts=batch_stee size, dims
if self.res. mad_e_moee--- "layer_re rese and layer >= 1: layer-res
output all slice = torch.stack(output slice list, dim 1) + layerO output else:
output_all _atii_icllllleeee_eeee torch..tackkouutuut s_sic_llccssttt_ dim= 1)
hidden_row__at aal torch.otaacccccc wtw.ssttcccccee_nng..... row_all __st__istt, dim =)
hidden col sonch.stack(hidden col list, dim= 1)
hidden row all hidden row all.reshape(batch size, self.num layers, Water2sea slice num,
hidden row all.shape[-1]
hidden_col_ale aall== hidden_col waaprrooooooalllaaaaaaee retrrshappcbaachssiee self.num Original_slice_lon
hidden_col. shape()
if flag =
return output_ all slice, caaten_coll_l,,.__ hidden_row_al
else:
return output_ all slice, hidden, foiden_rowwalle row__ll, all, hidden_col ______oo_llll
68


Reviews:
[{"review_id": "HQgXiQNRCn", "reviewer": "Reviewer_1moF", "summary": "This paper studies the problem of long-range time series forecasting problem and proposes a WITRAN model. The paper analyzes and compares previous forecasting methods from the perspective of information transmission process and design a water-wave information transmission mechanism, which simultaneously capture global and local correlations via a bi-granular information transmission. And a recurrent acceleration network is designed to reduce the computation complexity.", "questions": "1) If the five datasets have enough representativeness among all kinds of time series?\n\n2) The material included in Appendix A appears to be rather elementary for the target audience in this domain. I would be intrigued to understand the rationale behind the authors' decision to incorporate this content.\n", "limitations": "See weakness.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) WITRAN is an ingeniously crafted framework, capable of simultaneously capturing two aspects of semantic information within long-range time series - global-local correlations and both long- and short-term periodic patterns. The WIT includes two innovative modules, namely, HVGSU and GSU, which augment forecasting accuracy while enhancing explainability. Additionally, the RAN segment is designed to markedly improve efficiency. Finally, the WIT and RAN segments are seamlessly integrated together. \n\n2) The paper presents an exhaustive theoretical substantiation of the RAN segment's operating efficiency, shedding light on its capacity to significantly boost model efficiency.\n\n3) WITRAN exhibits superior performance, as confirmed by a series of comprehensive experiments with equitable settings across all baselines. WITRAN outperforms SOTA methods in both long-range and ultra-long-range forecasting tasks. Moreover, the incorporation of RAN indeed enhances computational efficiency and reduces memory footprint, in alignment with theoretical proof.", "weaknesses": "1) It seems that the proposed model is suitable for the time series that naturally contains bi-granular periodicity, such as traffic flow with daily and weekly periodicity. So the generalization of the proposed model is unclear.\n\n2) In the experimental section, many Transformer-based methods are taken as baselines, but few RNN-based methods are not compared. Since the proposed model is RNN-based, so it is necessary to add more RNN-based baselines.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere appreciation to Reviewer 1moF for their valuable insights and for recognizing the significance of our research.\n\n> Q1: It seems that the proposed model is suitable for the time series that naturally contains bi-granular periodicity, such as traffic flow with daily and weekly periodicity. So the generalization of the proposed model is unclear.\n\nWITRAN effectively captures both long- and short-term repetitive patterns through bi-granular information transmission. As long as the dataset contains periodicity and coarser-grained periodicity, WITRAN can effectively capture them. Our extensive experiments have demonstrated that WITRAN achieves outstanding performance in various fields, such as energy, traffic, and weather. Additionally, the RAN framework in WITRAN can serve as a generic framework for acceleration, highlighting its generality.\n\n> Q2: In the experimental section, many Transformer-based methods are taken as baselines, but few RNN-based methods are not compared. Since the proposed model is RNN-based, so it is necessary to add more RNN-based baselines.\n\nThank you very much for your valuable suggestions. We have included several classic RNN-based methods, including convLSTM, PredRNN, and predRNN++. Due to space limitations, we have included the specific experimental results of this section in the Global Rebuttal. From the experimental results in this section and the results in Tables 2 and 3 provide further evidence that our method is optimal.\n\n> Q3: If the five datasets have enough representativeness among all kinds of time series?\n\nThe benchmark datasets we used covers various domains and have different levels of granularity in their original collection. The previous works, including the baseline methods we selected, mostly use these datasets. Therefore, these datasets should have sufficient representativeness in covering time series.\n\n> Q4: The material included in Appendix A appears to be rather elementary for the target audience in this domain. I would be intrigued to understand the rationale behind the authors' decision to incorporate this content.\n\nThis section's detailed description of LSTM and GRU does indeed seem somewhat elementary. Our main intention is twofold: (1) to provide a foundation in Appendix B, where we elaborate on the inspiration drawn from LSTM and GRU, and (2) to facilitate a clear comparison between our proposed HVGSU and the differences between LSTM and GRU for the convenience of readers."}, {"review_id": "vm5RJVYkBP", "reviewer": "Reviewer_r3j3", "summary": "The paper studies a Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN) framework to model dependencies in a long historical time series. Inspired by Timesnet, the WITRAN introduces a water-wave information transmission strategy to model temporal information. This is implemented by the proposed Horizontal Vertical Gated Selective Unit (a kind of recurrent unit and  similar to GRU).", "questions": "Questions are as follows:\n- Are there any special temporal structures that existing models in Figure a-g cannot handle?\n- How to decide the number of R and C \n- How to set the norm in (1).", "limitations": "Some limitations in terms of implementation have been mentioned in the Conclusion.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) This paper is well-written\n2) Figures illustrate model details in a good way\n3) This paper studies the single-channel case, though most existing papers consider evaluations on multivariate time series. This makes sense as this can better evaluate the ability to learn temporal dependencies in time series.\n4) Advanced baselines such as MICN, timesnet, patchTST, and film are included.", "weaknesses": "1) Although using the water-wave structure is new to me, I am still curious about why this design is needed in long-range time series forecasting. Especially, are there any special temporal structures that existing models in Figure a-g cannot handle?\n2) The water-wave structure is a strategy to model time series dependencies. However, it is unclear how to decide the number of R and C and what are their effects on the final prediction. \n3) In Figure 10 (appendix), the results of the proposed are much worse than other baselines: FiLM and Pyraformer. But you claimed that “our model WITRAN gives the best performance among different models” in Section J. Moreover, other plots in Figure 11-31 did not demonstrate the effectiveness of WITRAN. \n4) The reference part is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to Reviewer r3j3 for their comprehensive review, which included thought-provoking questions and valuable insights.\n\n> Q1: Although using the water-wave structure is new to me, I am still curious about why this design is needed in long-range time series forecasting.\n\nWe appreciate your recognition of the novelty of our work. The advantages of applying the water-wave structure to long-range time series forecasting can be summarized as follows:\n\n(1) Bi-granular Design: The water-wave structure employs a bi-granular design, which offers two advantages. Firstly, it shortens the information transmission path, allowing for efficient information flow. Secondly, it enables the exploration of hidden long-term and short-term repetitive patterns in the sequences. These advantages are particularly beneficial for long-range prediction tasks.\n\n(2) Simultaneous Information Transmission: The water-wave structure facilitates simultaneous horizontal and vertical information transmission. This concurrent transmission allows for accelerated processing within the model, thereby improving efficiency.\n\n(3) Global Relationship Modeling: With the water-wave structure, the global relationships in long-range time series can be captured effectively using a single layer, as illustrated in Figure 1. This reduces the complexity of the network and the training difficulty.\n\nBy leveraging these advantages, the water-wave structure demonstrates its effectiveness and efficiency in long-range time series forecasting tasks.\n \n> Q2: Are there any special temporal structures that existing models in Figure a-g cannot handle?\n \nModeling global and local correlations, and discovering long- and short-term repetitive patterns is crucial for accurate long-range time series forecasting. Previous works have partially addressed these issues separately, but have not been able to address all of them simultaneously, such as facing challenges in capturing hidden semantic information directly from point-wise input tokens and so on. In order to compare the differences between WITRAN and the model a-g in Figure 1 more clearly, we have prepared the following table to highlight the advantages of WITRAN.\n\n|                        **Advantages**                       | (a) RNN | (b) CNN | (c) Full Attention | (d) LogTrans | (e) Pyraformer | (f) MICN | (g) PatchTST | (h) TimesNet | (i)WITRAN(ours) |\n|:-----------------------------------------------------------:|:-------:|:-------:|:------------------:|:------------:|:--------------:|:--------:|:------------:|:------------:|:---------------:|\n|           **Non point-wise semantic information capture**           |    ✓    |    ✓    |          ✗         |       ✓      |        ✓       |     ✓    |       ✓      |       ✓      |        ✓        |\n| **Special design to capture long-term repetitive patterns** |    ✗    |    ✗    |          ✗         |       ✗      |        ✓       |     ✗    |       ✗      |       ✓      |        ✓        |\n|       **Efficiently (1 or 2 layers) model global correlations**       |  ✓ (1)  |    ✗    |        ✓ (1)       |       ✗      |        ✗       |   ✓ (2)  |     ✓ (2)    |       ✗      |      ✓ (1)      |\n|      **Well solve the gradient vanishing/exploding problem of RNN**     |    ✗    |    -    |          -         |       -      |        -       |     -    |       -      |       -      |        ✓        |\n\n> Q3: How to decide the number of R and C and what are their effects on the final prediction.\n\n$C$ reflects the periodicity of short-term repetitive patterns, while $R$ reflects the coarse-grained periodicity of long-term repetitive patterns. Therefore, the value of $C$ is related to the specific forecasting task, and we also referred to previous work by setting the value of $C$ as multiples of 12. The value of $R$ is determined by the sequence length $L$ and $C$, specifically $R = L/C$. For more detailed descriptions and experimental analysis, please refer to Section 3.1 and Section I.1.\n\n> Q4: How to set the norm in (1).\n\nThe selection of the $norm$ in (1) is adaptively chosen by the model on the validation set based on the characteristics of the dataset and the task. For more details on this aspect, please refer to Section I.1.\n\n> Q5: In Figure 10 (appendix), the results of the proposed are much worse than other baselines: FiLM and Pyraformer. But you claimed that “our model WITRAN gives the best performance among different models” in Section J. Moreover, other plots in Figure 11-31 did not demonstrate the effectiveness of WITRAN.\n\nThank you for pointing that out. The forecasting case figures in the appendix are segments randomly selected from various tasks of different datasets, aiming to demonstrate a fair overview of WITRAN's performance when combined with Tables 2 and 3. However, random selection may occasionally include segments that are not fully representative, such as the one you specifically mentioned, Figure 10, which can indeed lead to misunderstandings. Therefore, to further demonstrate the advantages of WITRAN, we have re-randomly selected four additional cases for this task. Please refer to the PDF file we have newly submitted for specific details, where the advantages of WITRAN become apparent. \n\nAdditionally, we have conducted metric calculations for Figure 11 to Figure 31. Due to the character limit, we have also included this section in the PDF file. Please refer to Table A in the PDF file for the specific results.\n\nFrom the results in Table A, it can be observed that among all the randomly selected original cases, except for 5 cases, WITRAN consistently demonstrated superior performance. Thus, this provides ample evidence to support the overall superiority of WITRAN.\n\n> Q6: The reference part is missing.\n\nIn the original version, we placed the references in the Supplementary Material. In the revision, we will place it in the main content."}, {"review_id": "60mlhpxyxz", "reviewer": "Reviewer_Ku6W", "summary": "This paper focuses on the long-range time series forecasting problem. An interesting model, Water-wave Information Transmission and Recurrent Acceleration Network is proposed, which captures both short- and long-term recurrent patterns via bi-granular information transmission. The proposed model also captures global and local correlations using horizontal and vertical information transmission. This model is an interesting modification of the RNN network, and it significantly outperforms many transformer-based models.", "questions": "Q1. Please explicitly explain the advantages of the WITRAN model over other forecasting models from the perspective of information transmission process.\n\nQ2. Please give the adaptation and limitation of the WITRAN model.\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. This paper reviews the shortcomings of transformer models and makes modifications to the RNN structure. \n\nS2. A bi-granular information transmission is proposed to capture short- and long-term recurrent patterns, which is easy to understand and does not require additional methods (e.g., FFT) to extract periodicity.\n\nS3. A recurrent acceleration network is proposed, which reduces the time complexity to O(√L) while maintaining the memory complexity at O(L).\n", "weaknesses": "W1. Some previous RNN-based models, such as ConvLSTM [1]/ PredRNN [2]/ PredRNN++ [3], have yet to be compared.\n\n[1] Shi X, Chen Z, Wang H, et al. Convolutional LSTM network: A machine learning approach for precipitation nowcasting[J]. Advances in neural information processing systems, 2015, 28.  \n[2] Wang Y, Long M, Wang J, et al. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms[J]. Advances in neural information processing systems, 2017, 30.  \n[3] Wang Y, Gao Z, Long M, et al. Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning[C]//International Conference on Machine Learning. PMLR, 2018: 5123-5132.\n\nW2. Some minor problems:\n1) It is recommended to enlarge Figure 1(i) and make horizontal lines clearer.  \n2) It is recommended to unify TF_{en} and TFE_{de}.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable comments and recognition provided by Reviewer Ku6W regarding our work.\n\n> Q1: Some previous RNN-based models, such as ConvLSTM / PredRNN / PredRNN++, have yet to be compared.\n\nThank you for your suggestion. We have supplemented the experimental results for these methods. Due to space limitations, we have included the specific experimental results of this section in the Global Rebuttal. From the experimental results in this section, combined with the findings from Tables 2 and 3, provide further comprehensive evidence that WITRAN outperforms the state-of-the-art methods.\n\n> Q2: Some minor problems: (1) It is recommended to enlarge Figure 1(i) and make horizontal lines clearer. (2) It is recommended to unify TF_{en} and TFE_{de}.\n\nThanks a lot for reviewing our paper including appendix carefully and comprehensively. We will address the minor problems in the subsequent versions and conduct a more meticulous check of the paper.\n\n> Q3: Please explicitly explain the advantages of the WITRAN model over other forecasting models from the perspective of information transmission process.\n\nFor long-range time series forecasting tasks, previous studies have highlighted two crucial aspects of semantic information. On one hand, there is the presence of long-term and short-term periodic semantic information. On the other hand, there is the consideration of local-global semantic information. WITRAN's advantages can be summarized as follows: (1) Through its bi-granular information transmission, has the capability to directly capture long- and short-term periodic semantic information in long-range time series. Additionally, it shortens the information transmission path, effectively solving the issues of gradient explosion/vanishing when dealing with long-range time series. (2) Utilizing its recurrent structure, WITRAN gradually captures local semantic information and integrates it into the global semantic context without the need for additional layers. (3) WITRAN can effectively capture both of the key semantic aspects mentioned in (1) and (2), which was not achievable with previous methods. (4) In WITRAN, RAN could serve as a universal framework for integrating other models to facilitate information fusion and transmission, and reduce the time complexity to $\\mathcal{O}(\\sqrt{L})$ while maintaining the memory complexity of $\\mathcal{O}(L)$.\n\n> Q4: Please give the adaptation and limitation of the WITRAN model.\n\nThrough extensive experiments on different datasets and tasks, we have demonstrated the strong performance of WITRAN in various domains, such as energy, traffic, and weather forecasting. The results presented in Tables 2 and 3 indicate that WITRAN performs exceptionally well in long-range and ultra-long-range sequence forecasting tasks, showcasing its adaptability to such scenarios. Additionally, the RAN framework of WITRAN serves as a generic acceleration framework, as effectively illustrated in Section 4.2. However, it is worth noting that WITRAN does have its limitations, as mentioned in the Conclusions section. Specifically, the Python-based implementation of WITRAN is not as efficient as the nn.GRU / nn.LSTM implementations in PyTorch, which are based on C++. Therefore, we plan to explore the integration of WITRAN into an interface using C++."}, {"review_id": "BPRkuLAqXB", "reviewer": "Reviewer_2QPt", "summary": "To capture semantic information and repetitive patterns concurrently, the authors propose WIT framework. By utilizing bi-granular information transmission and HVGSU, the framework can model the inherent repetitive pattern as well as correlation of time series. The author also use a generic RAN to reduce time complexity. Several experiments conducted by te authors demonstrate that their framework can outperform in time series forecasting.", "questions": "Q1: Can authors include more experiments to further evaluate the performance of the framework.", "limitations": "Not Applicable", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The logic and presentation of the essay is easy to follow.\nS2: The problem they focus on is very essential as well as appealing.\nS3: Experiments reveal the performance of the authors’ model outperforms existing SOTA baselines.\n", "weaknesses": "W1: The authors may include more experiments such as robustness check to further evaluate the performance of the framework.\nW2: There are some typos and minor errors that do not influence the understanding of this work, which should be carefully checked. For instance, line 29, semantic information include-> semantic information includes\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful to Reviewer 2QPt for their constructive feedback and recognition of our work.\n\n> Q1: The authors may include more experiments such as robustness check to further evaluate the performance of the framework. Can authors include more experiments to further evaluate the performance of the framework.\n\nThank you for your suggestion. Here, we have followed MICN and introduced a simple white noise injection to demonstrate the robustness of our model. Specifically, we randomly select a proportion $\\varepsilon$ of data from the original input sequence and apply random perturbations within the range $[-2X_{i}, 2X_{i}]$ to the selected data, where $X_{i}$ denotes the original data. After the noise injection, the data is then used for training, and the MSE and MAE metrics are recorded. \n\nDue to space constraints, we have included the table for this section in a separate PDF file. Please refer to Table B in the newly submitted PDF file for specific results.\n\nAs the perturbation proportion $\\varepsilon$ increases, there is a slight increase in the MSE and MAE metrics in terms of forecasting. It indicates that WITRAN demonstrates good robustness when dealing with less noisy data (up to 10%), and it possesses a significant advantage in effectively handling various abnormal data fluctuations.\n\n> Q2: There are some typos and minor errors that do not influence the understanding of this work, which should be carefully checked. For instance, line 29, semantic information include-> semantic information includes\n\nWe greatly appreciate to the reviewer for conducting a meticulous and comprehensive review of our paper, including the appendix. We will conduct a careful check and correct these typos and minor errors in the subsequent versions of the paper."}]

Author rebuttal:
We thank the reviewers for their careful reading, and detailed and considerate feedback.

# 1 The Supplementary Baseline Experimental Results

Due to space limitations, we will report the results of the baseline experiments we conducted in this section. We have included several classic RNN-based methods, including convLSTM, PredRNN, and PredRNN++. The performance of these methods is as follows:

| Dataset |    Task   | ConvLSTM |         | PredRNN |         | PredRNN++ |         |
|:-------:|:---------:|:---------:|:---------:|--------:|:---------:|:----------:|:---------:|
|         |           |    MSE   |   MAE   |   MSE   |   MAE   |    MSE    |   MAE   |
|   ECL   |  168-168  |  1.10061 | 0.86419 | 1.59968 | 0.97815 |  1.05324  | 0.81715 |
|         |  168-336  |  0.95128 | 0.79262 |  1.3981 | 0.92003 |  1.14599  | 0.84769 |
|         |  336-336  |  1.06283 | 0.84397 | 1.24618 | 0.86953 |  1.14628  | 0.84583 |
|         |  336-720  |  1.02897 | 0.83296 | 1.38004 | 0.88513 |  1.12283  | 0.84775 |
|         |  720-720  |  1.16781 | 0.89208 | 1.14053 | 0.81585 |   1.0375  | 0.81916 |
|         |  720-1440 |  1.19013 | 0.90671 | 1.38331 | 0.91472 |     -     |    -    |
|         | 1440-1440 |  1.08107 | 0.86057 | 1.22349 | 0.88084 |     -     |    -    |
|         | 1440-2880 |  1.12110  | 0.86696 | 1.23118 | 0.84768 |     -     |    -    |

| Dataset |    Task   | ConvLSTM |         | PredRNN |         | PredRNN++ |         |
|:-------:|:---------:|:---------:|:---------:|--------:|:---------:|:----------:|:---------:|
|         |           |    MSE   |   MAE   |   MSE   |   MAE   |    MSE    |   MAE   |
| traffic |  168-168  |  1.9568  | 1.20513 | 2.18789 | 1.26295 |  2.03032  | 1.22555 |
|         |  168-336  |  1.56089 | 1.00843 | 1.94531 | 1.19838 |  2.15307  | 1.25684 |
|         |  336-336  |  1.53403 |  1.0047 | 1.94237 | 1.20825 |  2.01507  | 1.22229 |
|         |  336-720  |  1.97317 |  1.1942 | 2.10299 | 1.23738 |  2.07428  | 1.23589 |
|         |  720-720  |  1.97934 | 1.19838 | 2.24516 | 1.27154 |  2.05511  | 1.23179 |
|         |  720-1440 |  1.94060  | 1.17002 | 2.12212 | 1.23488 |     -     |    -    |
|         | 1440-1440 |  2.05290  |  1.2179 | 3.43876 | 1.49211 |     -     |    -    |
|         | 1440-2880 |  2.11828 | 1.24669 | 1.94494 | 1.20526 |     -     |    -    |

| Dataset |    Task   | ConvLSTM      |    | PredRNN      |    | PredRNN++      |    |
|:-------:|:---------:|:---------:|:---------:|--------:|:---------:|:----------:|:---------:|
|         |           |    MSE   |   MAE   |   MSE   |   MAE   |    MSE    |   MAE   |
|  ETTh1  |  168-168  |  1.11838 | 0.95673 | 0.20783 | 0.36933 |  0.19706  | 0.35199 |
|         |  168-336  |  0.42289 | 0.53454 | 0.60663 |  0.6639 |  0.18579  | 0.34186 |
|         |  336-336  |  0.28599 | 0.43453 | 0.90852 | 0.83825 |  0.23011  | 0.38086 |
|         |  336-720  |  0.18302 | 0.35023 | 0.29477 | 0.44067 |  0.28792  | 0.41855 |
|         |  720-720  |  0.21331 | 0.36852 | 0.92726 | 0.87761 |  0.34586  | 0.56421 |
|         |  720-1440 |  0.26016 | 0.40767 |  0.2097 | 0.35739 |     -     |    -    |
|         | 1440-1440 |  0.46201 | 0.55664 | 0.25375 | 0.39662 |     -     |    -    |
|         | 1440-2880 |  0.68657 | 0.73995 |    -    |    -    |     -     |    -    |

| Dataset |    Task   |      ConvLSTM |    |      PredRNN |    |      PredRNN++ |    |
|:-------:|:---------:|:---------:|:---------:|--------:|:---------:|:----------:|:---------:|
|         |           | MSE      | MAE     | MSE     | MAE     | MSE       | MAE     |
|  ETTh2  |  168-168  |  0.67911 | 0.67424 | 2.71814 | 1.24722 |  0.46365  | 0.53826 |
|         |  168-336  |  0.57245 |  0.6036 |  7.0023 | 1.94492 |  0.50444  | 0.56156 |
|         |  336-336  |  0.43214 | 0.52404 | 1.21141 | 0.90117 |  0.53269  | 0.57922 |
|         |  336-720  |  0.44047 | 0.52043 | 0.77007 | 0.71523 |  0.59645  | 0.61303 |
|         |  720-720  |  0.61122 | 0.61653 | 0.68664 | 0.66567 |  0.65089  | 0.65267 |
|         |  720-1440 |  0.68432 | 0.67343 |  0.5682 | 0.63012 |  0.52903  | 0.59123 |
|         | 1440-1440 |  0.69629 | 0.67345 | 0.84719 | 0.77302 |     -     |    -    |
|         | 1440-2880 |  0.49551 | 0.57418 | 0.81936 |  0.7682 |     -     |    -    |

| Dataset |    Task   | ConvLSTM |         | PredRNN |         | PredRNN++ |         |
|:-------:|:---------:|:---------:|:---------:|--------:|:---------:|:----------:|:---------:|
|         |           | MSE      | MAE     | MSE     | MAE     | MSE       | MAE     |
| WTH     | 168-168   | 0.23746  | 0.37102 | 0.70020  | 0.66288 | 0.32711   | 0.43675 |
|         | 168-336   | 0.27096  | 0.40337 | 0.96749 | 0.77150  | 0.40391   | 0.48547 |
|         | 336-336   | 0.30900    | 0.42290  | 0.77884 | 0.70297 | 0.40797   | 0.49392 |
|         | 336-720   | 0.39303  | 0.49367 | 0.77279 | 0.70146 | 0.50707   | 0.56308 |
|         | 720-720   | 0.40034  | 0.47207 | 0.72229 | 0.68889 | 0.61416   | 0.65392 |
|         | 720-1440  | 0.46196  | 0.55477 | 0.61551 | 0.6625  | -         | -       |
|         | 1440-1440 | 0.39681  | 0.48828 | 0.7735  | 0.74801 | -         | -       |
|         | 1440-2880 | 0.45973  | 0.54664 | -       | -       | -         | -       |

Among them, "-" represents the cases of gradient explosion/vanishing. Specifically, during the training process, the loss value may become NaN.

# 2 The Supplementary Forecasting Cases

We randomly selected an additional four forecasting cases to compare the performance of WITRAN, FiLM, and Pyraformer on the 168-168 task of the ECL dataset. For more details, please refer to Figure A in the newly submitted PDF.

# 3 The MAE Metric of Figure 11 to Figure 31

We have calculated the metrics for different methods in each case, more details can be found in Table A in the PDF.

# 4 The Supplementary Robustness Experiments

We have included the table for this section in a separate PDF file. Please refer to Table B in the PDF for specific results.

Meta-review: All reviewers gave this paper a positive rating. All but one reviewer were satisfied with the authors' feedback. The concerns of reviewer 2QPt, who did not respond, appear to have been adequately addressed by the authors. Based on the above, the ACs consider it appropriate to accept the paper. The ACs recommend that the feedback and discussion be included in the final version.
Decision: Accept (spotlight)