PGN: The RNN's New Successor is Effective for Long-Range Time Series Forecasting

Authors: yuxin  jia
youfang  lin
jing  yu
shuo  wang
tianhao  liu
huaiyu  wan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:10:49

Abstract: Due to the recurrent structure of RNN, the long information propagation path poses limitations in capturing long-term dependencies, gradient explosion/vanishing issues, and inefficient sequential execution. Based on this, we propose a novel paradigm called Parallel Gated Network (PGN) as the new successor to RNN. PGN directly captures information from previous time steps through the designed Historical Information Extraction (HIE) layer and leverages gated mechanisms to select and fuse it with the current time step information. This reduces the information propagation path to $\mathcal{O}(1)$, effectively addressing the limitations of RNN. To enhance PGN's performance in long-range time series forecasting tasks, we propose a novel temporal modeling framework called Temporal PGN (TPGN). TPGN incorporates two branches to comprehensively capture the semantic information of time series. One branch utilizes PGN to capture long-term periodic patterns while preserving their local characteristics. The other branch employs patches to capture short-term information and aggregate the global representation of the series. TPGN achieves a theoretical complexity of $\mathcal{O}(\sqrt{L})$, ensuring efficiency in its operations. Experimental results on five benchmark datasets demonstrate the state-of-the-art (SOTA) performance and high efficiency of TPGN, further confirming the effectiveness of PGN as the new successor to RNN in long-range time series forecasting. The code is available in this repository: https://github.com/Water2sea/TPGN.

Main paper: 
 introduction: 
 Under the premise of accurate time series forecasting, long-range forecasting tasks offer an advantage over short-range forecasting tasks as they provide more comprehensive information for individuals and organizations to thoroughly assess future changes and make well-informed decisions. Due to its practical applicability across various fields (i.e., energy [Zhou et al., 2021] , climate [Angryk et al., 2020] , traffic [Yin and Shang, 2016] , etc), long-range forecasting has attracted significant attention from researchers in recent years.Long-range time series forecasting tasks can be broadly classified into two categories. One task is to utilize abundant inputs to forecast future outputs [Liu et al., 2022a , Jia et al., 2023] , while another task is to predict longer-range futures with fewer historical inputs [Zhou et al., 2021 , 2022a , Wang et al., 2023] . Although existing studies have shown that ample historical inputs can introduce more information to improve prediction performance [Jia et al., 2023 , Liu et al., 2024] , considering factors such as the load capacity of training devices and data collection, the utilization of limited historical inputs to predict longer-range futures remains an important research topic. Therefore, this paper sets the task goal as predicting longer outputs with fewer inputs.In recent years, deep-learning-based methods have achieved remarkable success in time series forecasting (for further discussions, please refer to Section 2 and Appendix B). These methods can be roughly categorized into four based paradigms: Transformers [Zhou et al., 2021 , Wu et al., 2021 , Liu et al., 2022a , Zhou et al., 2022a , Nie et al., 2023 , Ni et al., 2023 , Liu et al., 2024 , Dai et al., 2024] , CNNs [Wu et al., 2023 , Wang et al., 2023 , Luo and Wang, 2024] , MLPs and Linears [Zeng et al., 2023 , Xu et al., 2024 , Wang et al., 2024] , and RNNs [Jia et al., 2023] . It is worth noting that RNNs have received relatively less attention over an extended period of time. This discrepancy is primarily attributed to the limitation of RNNs' recurrent structure, which leads to the persistence of excessive long pathways for information propagation.In fact, shorter information propagation paths lead to less information loss [Tishby and Zaslavsky, 2015] , better captured dependencies [Liu et al., 2022a] , and lower training difficulty [Wang et al., 2023] . However, RNNs heavily rely on a sequential recurrent structure to transmit information, making it challenging for them to capture long-term dependencies and suffer from the issue of gradient vanishing/exploding [Pascanu et al., 2013] . Meanwhile, due to its sequential computation, even though RNNs have a theoretical complexity that is linear with respect to sequence length L, their actual running speed can be even slower than the O(L 2 ) complexity of the Vanilla-Transformer [Vaswani et al., 2017] . Some RNN-based models [Hochreiter and Schmidhuber, 1997, Chung et al., 2014] have tried to enhance performance by incorporating specialized gated mechanisms. However, compared to the inherent limitations of the RNN structure, these improvements in information selection and fusion are merely a drop in the bucket.Based on this motivation, this paper proposes a novel general paradigm called Parallel Gated Network (PGN) as the new successor to RNN. PGN introduces a Historical Information Extraction (HIE) layer to replace the recurrent structure of RNN, and then further selects and fuses information through gated mechanisms, effectively reducing the information propagation paths to O(1), as shown in Figure 1 (l). This enables PGN to better capture long-term dependencies in input signals. Additionally, since computations for each time step can be parallelized, PGN achieves significantly faster execution speed while maintaining the same theoretical complexity of O(L) as RNN.Based on these motivations, we propose a novel PGN-based temporal modeling framework called Temporal Parallel Gated Network (TPGN). TPGN establishes two distinct branches to capture long-term and short-term information in the 2D input series. To focus on modeling long-term information, we utilize PGN to model each column of the 2D inputs, preserving their respective local periodic characteristics. Simultaneously, leveraging the advantages of patch [Nie et al., 2023] in capturing short-term changes, TPGN initially aggregates the short-term information into patches and subsequently merges them to obtain global information.By integrating the information from both branches, TPGN achieves comprehensive semantic information capture for accurate predictions. It also should be noted that other methods can substitute PGN and be used in the long-term information extraction branch of TPGN, undoubtedly enabling TPGN to be a general framework to model temporal dependencies. Furthermore, TPGN maintains a efficient computational complexity of O( √ L). To better illustrate the advantages of TPGN, inspired by [Jia et al., 2023] , we have provided a information propagation comparative diagram in Figure 1 and an analysis table in Table 3 .The main contributions of this paper can be summarized as follows:• We propose a novel general paradigm called PGN as the new successor to RNN, as shown in Figure 1 (l). It reduces the information propagation path to O(1), enabling better capture of long-term dependencies in input signals and addressing the limitations of RNNs. • We propose TPGN, a novel temporal modeling framework based on PGN, which comprehensively captures semantic information through two branches, as shown in Figure 1 (m).One branch utilizes PGN to capture long-term periodic patterns and preserve their local characteristics, while the other branch employs patches to capture short-term information and aggregates them to obtain a global representation of the series. Notably, TPGN can also accommodate other models, making it be a general temporal modeling framework. • In terms of efficiency, PGN maintains the same complexity of O(L) as RNN. However, due to its parallelizable calculations, PGN achieves higher actual efficiency. On the other hand, TPGN, serving as a general temporal modeling framework, exhibits a favorable complexity of O( √ L). For a more detailed comparison of complexities, please refer to Table 3 . • We conducted experiments on five benchmark datasets, and the results indicated that TPGN achieved an average MSE improvement of 12.35% in various long-range time series forecasting tasks compared to the previous best-performing models. Furthermore, in comparison to the average performance of specific models across all tasks, TPGN achieved an average MSE improvement ranging from 14.08% to 39.65%. Additionally, experimental evaluations on computational complexity confirmed the efficiency of TPGN.2 Related Works 
modeling interaction cross temporal dimension: 
 The methods that focus on temporal modeling can be broadly categorized into four paradigms: RNN-, CNN-, MLP-(Linear-), and Transformer-based. The limitations of RNNs [Hochreiter and Schmidhuber, 1997 , Chung et al., 2014 , Salinas et al., 2020] have been discussed in Section 1. Despite some methods [Chang et al., 2017 , Yu and Liu, 2018 , Jia et al., 2023] trying to alleviate these limitations, the recurrent structure still hinders their further development. CNNs [Franceschi et al., 2019 , Sen et al., 2019] offer advantages in efficiency and shorter information propagation paths, but primarily constrained by limited receptive fields [Wu et al., 2023] , resulting in an increase in the information propagation path as the length of the processed signal increases. Although some methods [Wang et al., 2023, Luo and Wang, 2024] have increased the receptive field to address these issues, the 1D modeling approach makes it challenging for them to directly capture periodicity. The advantages of Linear [Zeng et al., 2023] lie in its simple structure and high operational efficiency. Some advanced models have further enhanced the performance of MLP or Linear by applying them in the frequency domain [Xu et al., 2024] or introducing multi scales [Wang et al., 2024] , which could lead to higher execution overhead. Classic Transformer-based methods either struggle to capture semantic information [Wu et al., 2023 , Nie et al., 2023] due to point-wise attention mechanisms [Vaswani et al., 2017 , Zhou et al., 2021 , 2022a] or have high complexity [Wu et al., 2021 , Liu et al., 2022a] , limiting their ability. Subsequently, this problem was effectively addressed by utilizing patches [Nie et al., 2023] . However, they still suffer from the 1D modeling issue mentioned earlier or the problem of limited receptive fields. More detailed discussion and analysis can be found in Appendix B.For handling variable dimensions, there are generally four categories: variable fusion processing, variable independent processing, modeling based on Transformers, and modeling based on Graph Neural Networks (GNNs). Traditional fusion processing methods, due to the heterogeneity of multiple variables [Zhou et al., 2021] , introduce excessive noise, resulting in worse performance compared to independent processing of variables [Nie et al., 2023] . However, by applying attention mechanisms and Graph Neural Networks (GNN) on the variable dimension to replace independent modeling of variables, it is possible to successfully capture the correlations and differences between variables, thereby significantly improving the performance of multivariate modeling. Representative methods for modeling variable relationships based on Transformers include Crossformer [Zhang and Yan, 2022] and iTransformer [Liu et al., 2024] , while GNN-based representative methods include CrossGNN [Huang et al., 2023] and FourierGNN [Yi et al., 2023] . They provide excellent inspiration for analyzing and modeling multivariate time series. 
methodology: 
 In this section, we first introduce our proposed novel paradigm called Parallel Gated Network (PGN), and explain how it reduces the information propagation paths, overcomes the limitations of RNNs, and emerges as the new successor to RNNs. Next, we present our newly designed temporal modeling framework called Temporal PGN (TPGN), which incorporates two separate branches to comprehensively capture semantic information. Finally, we provide a comprehensive complexity analysis to evaluate the computational efficiency of our methods.Building upon the previous analysis, the limitation of RNNs lies in the excessively long information propagation paths of its recurrent structure, which directly leads to a series of issues, such as difficulty in capturing long-term dependencies (performance), low efficiency in sequential computations (efficiency), and gradient exploding/vanishing (training difficulty). Indeed, some RNNs leverage specialized gated mechanisms, such as LSTM [Hochreiter and Schmidhuber, 1997] and GRU [Chung et al., 2014] , which do have advantages in information selection and fusion. However, when faced with the disastrous limitation of RNNs, their advantages become insignificant.Based on this, we propose a novel general paradigm called PGN as the new successor to RNNs. PGN draws the advantages of RNNs while reducing information propagation paths to O(1), thereby addressing the limitation of RNNs. The information propagation illustration and structure of PGN are shown in Figure 1 (l) and Figure 2 (a), respectively. On one hand, to enable PGN to capture information from all preceding time steps within short information propagation paths, we introduce a linear Historical Information Extraction (HIE) layer to aggregate information from the entire history at each time step. Importantly, at this stage, the computation of each time step of the signal is independent of others, allowing for effective parallel processing. On the other hand, PGN leverages gated mechanisms to inherit the advantages of information selection and fusion. It is important to emphasize that in PGN, we utilize only a single gate to simultaneously control the information selection and fusion in a parallel manner across all time steps of the sequence, resulting in reduced computational overhead. When given an input signal X ∈ R L of length L, the computation process of PGN can be formalized as follows:EQUATIONwhere Padding(•) represents the operation of filling the front of the processed signal along the length dimension with a zero-filled vector of size R (L-1) . HIE(•) is a linear layer with weight matrices The specific objective of time series forecasting task is to predict the future series of length L f given a historical sequence of length L h . As stated in Section 1, PGN may not be effective in directly extracting periodic semantic information, which limits its application in time series forecasting tasks.Inspired by [Wu et al., 2023 , Jia et al., 2023 , Dai et al., 2024] , we transform the input series from 1D to 2D for modeling. To fully capture the short-term changes and long-term periodic patterns with different characteristics in the rows and columns of the 2D input, we introduce two branches to model them separately. The information propagation diagram and overall structure of TPGN are shown in Figure 1 (m) and Figure 2 Input Preparation Module To enable TPGN to directly capture periodic semantic information, inspired by previous works [Wu et al., 2023 , Jia et al., 2023 , Dai et al., 2024] , we reshape the series from 1D to 2D. Notably, we do not need to introduce multiple scales of periods like in TimesNet [Wu et al., 2023] and PDF [Dai et al., 2024] , as it would result in increased computational overhead. Instead, we draw inspiration from WITRAN [Jia et al., 2023] and solely reset the sequence based on the natural scale of the time series. In addition, to minimize the negative impact of data fluctuations on model training, inspired by [Liu et al., 2022b [Liu et al., , 2024]] , we have introduced a normalization layer along the temporal dimension. When given an input sequence X 1D = {x 1 , x 2 , . . . , x L h } ∈ R L h ×C and temporal external feature T F enc ∈ R L h ×Ctime (C and C time represent the number of variables and temporal external features), this module can be mathematically expressed as:µ X = 1 L h L h i=1 x i , σ 2 X = 1 L h L h i=1 (x i -µ X ) 2 , X norm 1D = X 1D , norm = 0 (X 1D -µ X )/σ X , norm = 1 , X 2D = Reshape([X norm 1D , T F enc ]).(2)Here, X norm 1D ∈ R L h ×C represent the normalized series, [•] represents the concat operation, and the hyperparameters norm should be determined based on the characteristics of different datasets. To combine each variable of the input series with the temporal feature, we need to expand X norm 1D by adding an extra dimension to match the shape of R L h ×C×1 . Additionally, T F enc needs to be expanded by adding a dimension and repeated C times to match the shape of R L h ×C×Ctime . Afterwards, we concatenate the results and reshape them according to the natural period P of series through Reshape(•), and X 2D ∈ R R×P ×C×(1+Ctime) represents the output of this module. R and P represent the number of rows and columns in the 2D input, respectively.TPGN As TPGN focuses on modeling the temporal dimension, which is crucial for any variable in the time series. In the following discussions, we will focus on an example variable m to provide a detailed explanation, where X m 2D ∈ R R×P ×(1+Ctime) represents the input. To better capture longand short-term information while preserving their respective characteristics, we have designed two branches as illustrated in Figure 1 (m) and Figure 2 (b ).In the long-term information extraction branch, we directly capture the information using PGN. On one hand, it effectively captures the long-term repetitive historical information for each time step. On the other hand, through the gated mechanism, it selects and fuses the current and historical information at each time step, thereby preserving the long-term periodic characteristics to the maximum extent. Specifically, this branch can be formulated as follows:EQUATIONwhere PGN(•) represents the input being passed through the PGN paradigm. It is important to note that PGN operates along the R dimension. The advantage of this approach is it preserves the individual characteristics of each column, better serving forecasting. The output is denoted asX m long ∈ R R×P ×dm .To facilitate the utilization of long-term information for prediction purposes, we aggregate the information from all rows in each column using a linear layer Linear long (•). The output of this branch is denoted as H m long ∈ R P ×dm . In the short-term information extraction branch, considering the advantage of patch in aggregation short-term information, we first utilize a linear layer to aggregate the short-term information into patches. Then, another linear layer is used to further fuse the patches into the global information of the series. The specific process can be formulated as follows:EQUATIONwhere Linear row short (•) operates along the P dimension, and H m short ∈ R R×dm is its output. Subsequently, Linear Forecasting Module In this module, begin by concatenating the information representations derived from the two branches of TPGN. The concatenated information encompasses the local long-term periodic characteristics observed across various columns in the 2D input series, along with the globally aggregated short-term information. Subsequently, we take the previously aggregated representation with comprehensive semantic information and pass it through a linear layer to predict future values at different positions within the period. The formulation of this module is as follows:EQUATIONwhere [•] represents the concat operation. The output dimension after Linear(•) is R P ×R f , where R f multiplied by P equal forecasting series length L f . Finally, the above output will be permuted and reshaped to 1D dimension by Reshape(•) operation, the result Out m ∈ R L f .PGN While PGN processes signals in the time dimension in parallel, each step still involves processing all its historical information. Therefore, the theoretical complexity of this part is still linear with respect to the length L of the signal being processed. The complexity of the gated mechanism is independent of the signal length, so the complexity of PGN can be expressed as O(L). Noted that PGN has the same theoretical complexity as RNN, but due to the parallelized ability of PGN computations, it has much higher efficiency in practice compared to RNN.TPGN Since TPGN has two separate branches, it is necessary to analyze them separately.For the long-term information extraction branch, TPGN applies the PGN paradigm along the number of R, the complexity of this step is indeed linear with respect to R, denoted as O(R). The aggregation of all rows of information is accomplished through a linear layer, which still has a complexity proportional to O(R). Therefore, the complexity of the long-term information extraction branch can be expressed as O(R).For the short-term information extraction branch, TPGN applies two linear layers. The first linear layer compresses the time dimension from P to 1, while the second linear layer compresses the other time dimension R to 1, therefore, their complexities are respectively O(P ) and O(R).Since R multiplied by P equals the input sequence length L h (L), the complexities O(R) and O(P ) are both equal to O( √ L). For the two branches of TPGN, the complexities are both O( √ L). Therefore, the complexity of TPGN is also O( √ L). 
experiments: 
 Datasets To validate the performance of TPGN, we followed WITRAN [Jia et al., 2023] and conducted experiments on five real-world benchmark datasets that span across energy, traffic, and weather domains. More details about the datasets can be found in Appendix C.Baselines We conducted a comprehensive comparison of eleven methods in our study. These methods include one RNN-based method: WITRAN [Jia et al., 2023] , SegRNN [Lin et al., 2023] , three CNN-based methods: ModernTCN [Luo and Wang, 2024] , TimesNet [Wu et al., 2023] , MICN [Wang et al., 2023] , three MLP-based methods: FITS [Xu et al., 2024] , TimeMixer [Wang et al., 2024] , DLinear [Zeng et al., 2023] , four Transformer-based methods: iTransformer [Liu et al., 2024] , PDF [Dai et al., 2024] , Basisformer [Ni et al., 2023] , PatchTST [Nie et al., 2023] ,and FiLM [Zhou et al., 2022b] . It should be noted that certain earlier methods such as Vanilla-Transformer [Vaswani et al., 2017] , Informer [Zhou et al., 2021] , Autoformer [Wu et al., 2021] ,Pyraformer [Liu et al., 2022a] , and FEDformer [Zhou et al., 2022a] have been extensively surpassed by the methods we selected. Hence, we did not include these earlier methods as baselines in our comparison. For further discussion on these methods and details of the experimental setup, please refer to Appendix B and Appendix D.It is important to emphasize that while there have been numerous works focusing on modeling the relationships among multiple variables in time series, they still need to effectively capture information along the temporal dimension to better accommodate multivariate time series. In contrast, our method primarily concentrates on modeling the temporal dimension. To mitigate any potential negative impact caused by the heterogeneity of multivariate data, we followed the experimental setup of WITRAN [Jia et al., 2023] , conducted experiments using a single variable. To ensure fairness, we conducted parameter search for each baseline model, enabling them to achieve their respective optimal performance across different tasks. For further experiment details, please refer to Appendix D. For other aspects experiments and analysis of our methods can be found in Appendix G.To validate the roles of the two information extraction branches in TPGN, we conducted tests on the performance of the model when using only one branch. Additionally, to verify the effectiveness of PGN, we performed ablation experiments by replacing PGN with GRU and MLP, respectively. "TPGN-long" represents only using the long-term information extraction branch, while "TPGNshort" represents only using the short-term. "TPGN-GRU/-LSTM/-MLP/-Attn" respectively represent replacing PGN in the long-term information extraction branch with GRU, LSTM, MLP and selfattention. The results of these experiments are presented in Table 2 . Through the ablation experiments, we can draw the following conclusions: (1) The two branches designed in TPGN are reasonable as they capture long-term and short-term information respectively, while preserving their respective characteristics. In most cases, using only one branch leads to subpar results due to incomplete capture of essential features.(2) The branch capturing long-term information in TPGN is more significant. This can be observed by comparing the performance degradation when using only one branch versus using both branches together. Especially for strongly periodic data like traffic, in some tasks, using only the long-term information capture branch can achieve good results. This also aligns with our earlier mention in Section 1 about the significance of prioritizing the modeling of periodicity.(3) Compared to GRU and LSTM, which have more gates, PGN introduces only one gate but still achieves better performance. This strongly demonstrates the ability of PGN to serve as the new successor to RNN. (4) The comparison between "TPGN-GRU/-LSTM/-MLP/-Attn" and the baseline results demonstrates the strong generality and performance of the TPGN framework.Despite their inferior performance compared to TPGN, in some tasks, they even surpass the previous SOTA time series forecasting methods.Although this paper primarily focuses on predicting longer-range future outputs using short-range historical inputs, we conducted two sets of comparative experiments to comprehensively evaluate the efficiency of our proposed method. In the first set of experiments, we kept the input length fixed at 168 and varied the output length to 168/336/720/1440 to study the impact of forecasting length on the actual runtime efficiency of the model. In the second set of experiments, we fixed the output length at 1440 while varying the input length to 168/336/720/1440 to investigate the influence of historical input series length on the actual runtime of the model. The efficiency analysis considered both time and memory aspects. We selected representative methods from each paradigm based on the experimental results in Table 1 as the comparative methods. We fixed the batch size at 32, the model dimension size at 128, and conducted the tests using a single-layer model. The results are shown in From Figure 4 , it can be observed that although TPGN does not have the lowest time and memory overhead, it achieves a decent level of efficiency in both time and space aspects. It is important to note that TPGN is a model with only one layer, while most of other models require the introduction of deeper layers, which inevitably leads to higher overhead. This undoubtedly further demonstrates that our method not only achieves SOTA performance but also delivers satisfactory efficiency. 
conclusions: 
 In this paper, we propose a novel general paradigm called Parallel Gated Network (PGN). With its O(1) information propagation paths and parallel computing capability, PGN achieves faster runtime speed while maintaining the same theoretical complexity as RNN (O(L)). To enhance the application of PGN in long-range time series forecasting tasks, we introduce a novel temporal modeling framework called Temporal PGN (TPGN) with an excellent complexity of O( √ L). By employing two branches to separate the modeling of long-term and short-term information, TPGN effectively capture periodicity and local-global semantic information while preserving their respective characteristics. The experimental results on five benchmark datasets demonstrate that our PGN-based framework, TPGN, achieves SOTA performance and high efficiency. These findings further confirm the effectiveness of PGN as the new successor to RNN in long-range time series forecasting tasks. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: After carefully reviewing the information on the url, we have confirmed that all aspects of our study conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Section 1, we introduce the background of our work and highlight its positive societal impacts. To the best of our knowledge, there are currently no apparent negative impacts associated with our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper utilizes benchmark datasets, and poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: It is important to acknowledge that our focus in this work was primarily on temporal modeling, without specifically addressing the modeling of relationships between variables. Nevertheless, we can draw inspiration from other methods specialized in variable modeling. Incorporating an additional component to model variable relationships and integrating it into the TPGN framework is a promising direction for better adaptation to multivariate prediction tasks, which we plan to explore in future research. Additionally, we will continue to investigate the broader application of the PGN paradigm as a replacement for RNN in various time series analysis tasks and other research areas.Traditional methods such as ARIMA [Box and Jenkins, 1968] , Prophet [Taylor and Letham, 2018] , and Holt-Winters [Athanasopoulos and Hyndman, 2020] are often constrained by theoretical assumptions, which limits their applicability in time series forecasting tasks with dynamic data changes. In recent years, deep neural networks (DNNs) have made significant advancements in the field of time series analysis. DNNs can be categorized into four main paradigms: RNN-based, CNN-based, MLP-based, and Transformer-based methods. [Hochreiter and Schmidhuber, 1997 , Chung et al., 2014 , Rangapuram et al., 2018 , Salinas et al., 2020] rely on recurrent structures to capture sequential information, which leads to long information propagation paths and brings about various limitations, as discussed in Section 1. In terms of performance, RNN-based methods struggle to capture long-term dependencies effectively. Moreover, their theoretical complexity scales linearly with the sequence length L, but their practical efficiency is often low due to sequential computation. Additionally, during training, RNNs are prone to the issues of gradient exploding/vanishing [Pascanu et al., 2013] .To alleviate these issues, some methods have modified the conventional information propagation approach. DilatedRNN [Chang et al., 2017] introduces a multi-scale dilated mechanism, which aggregates information at each time step. Although it can shorten the information propagation path by selecting the branch with the maximum skipping step, the path remains linearly related to the sequence length L, which is still relatively long. SlicedRNN [Yu and Liu, 2018] addresses the efficiency problem of RNNs by dividing the sequence into multiple slices for parallel computation. However, the length of the information propagation path remains the same as the traditional RNNs. WITRAN [Jia et al., 2023] , as an emerging time series forecasting method, reshapes the sequence into a 2D dimension and performs simultaneous information propagation in both directions. This approach improves computational efficiency and reduces the information propagation path to O( √ L). However, it is still relatively long for effective information extraction.Overall, the limitations imposed by the recurrent structures of RNNs have hindered their further development.CNN-based methods [Bai et al., 2018 , Franceschi et al., 2019 , Sen et al., 2019] have a theoretical complexity of O(L), due to their parallel ability, they often exhibit higher practical efficiency compared to RNNs. However, CNNs are typically constrained by limited receptive fields, requiring the stacking of multiple module layers to capture global information from inputs. The number of modular layers grows superlinearly with the sequence length, leading to an information propagation path of O(G) in CNN-based methods. Here, G is superlinearly related to the sequence length L. In the case of the 2D modeling method TimesNet [Wu et al., 2023] , where the input lengths in both directions are O( √ L), the information propagation path would be O( √ G). MICN [Wang et al., 2023] and ModernTCN [Luo and Wang, 2024] effectively reduce the information propagation path by enlarging the receptive field of the convolutional kernel. However, due to their 1D modeling approach, they may not perform as well as TimesNet [Wu et al., 2023] in capturing periodic characteristics.MLP-based methods are highly favored due to their simple structure, resulting in lower computational complexity and information propagation path. This simplicity makes MLP models easy to implement and train, contributing to their popularity. Dlinear and NLinear [Zeng et al., 2023] optimize the original Linear model through the methods of sequence decomposition and re-normalization methods, enabling direct future prediction based on historical inputs. However, due to their limited ability to extract deep semantic information, they may not achieve excellent forecasting performance. TimeMixer [Wang et al., 2024] employs two specialized modules to analyze and predict time series data from multiple scales. While this approach can effectively capture periodicity, incorporating multiple scales in computations inevitably leads to increased computational costs and training difficulties. FITS [Xu et al., 2024] treats time series prediction as interpolation and transforms the time series into the frequency domain. It operates on the frequency domain using a specially designed block LPF (Low-Pass Filter) and a complex-valued linear layer for final forecasting. However, FITS may still overlook explicit local variations present in the sequence.Transformer-based methods still dominate the majority of the field. The advantage of methods based on point-wise attention mechanism, such as Vanilla-Transformer [Vaswani et al., 2017] , Informer [Zhou et al., 2021] , and FEDformer [Zhou et al., 2022a] , lies in their O(1) information propagation path. However, previous studies [Wu et al., 2023 , Jia et al., 2023] have clearly pointed out their limitation in capturing semantic information from time steps. On the other hand, other methods that utilize non-point-wise attention mechanisms still have other limitations. Although Autoformer [Wu et al., 2021] can capture the periodicity of time series to some extent through sequence decomposition, it is far less direct compared to methods like TimesNet [Wu et al., 2023] . Additionally, its complexity remains high at O(L log L). Pyraformer [Liu et al., 2022a] , through the special design of its pyramidal structure, is also able to effectively capture the periodic characteristics of sequences. However, it is still constrained by the limitations of the convolution kernel when initializing the node of pyramidal structure. Additionally, Pyraformer still maintains a high complexity of O(L). PatchTST [Nie et al., 2023] captures local semantic information through patches, where S represents the stride length, and further reduces the complexity to O((L/S) 2 ). However, it still cannot directly capture the periodic characteristics of series. iTransformer [Liu et al., 2024] primarily focuses on modeling the relationships between variables, including the relationships between time series variables and external time features. For the temporal dimension, iTransformer adopts a direct patch-based approach, which makes it challenging to effectively extract periodic patterns and other local characteristics. PDF [Dai et al., 2024 ] also follows the approach of transposing the original 1D sequence into a 2D representation for modeling. Specifically, it utilizes CNNs to process short-term information, which is undoubtedly constrained by the limitations of convolution itself. When it comes to handling long-term periodic information, PDF also adopts a patch-based approach, which may not fully capture all the periodic characteristics present in the sequence.To highlight the advantages of our proposed PGN paradigm and TPGN framework compared to previous methods, we have organized an information propagation diagram as shown in Figure 1 . Based on the above analysis, we further compiled the various strengths, information propagation paths, and theoretical complexities of different models, which are presented in Table 3 . WITRAN (2D) O( √ L) O( √ L) - CNN O(G) O(L) MICN - (Decomposition) O(1) O(L) TimesNet (2D) O( √ G) O(L) ModernTCN O(G) O(L/S) Transformer O(1) O(L 2 ) Informer O(1) O(L log L) Autoformer - (Decomposition) O(1) O(L log L) Pyraformer (Pyramidal Structure) O(1) O(L) PatchTST O(1) O((L/S) 2 ) iTransformer O(1) O(L) PDF (2D) O( √ G) O(L/S) PGN (ours) O(1) O(L) TPGN (ours) (2D) O(1) O( √ L)respective original papers and conducted parameter search accordingly. This ensured that we took into account the optimal configurations for each model in our experiments. For models that have variants, such as MICN-regre and MICN-mean, we treat the variants as separate hyperparameters and include them in the search process. The aforementioned procedures ensure that the reported results represent the optimal performance of each model under the same comparison conditions.TPGN and all baselines were trained using L2 loss and the ADAM optimizer [Kingma and Ba, 2014] with an initial learning rate of 10 -3 . All of them are implemented using PyTorch [Paszke et al., 2019] and conducted on NVIDIA RTX A4000 16GB GPUs. The batch size was set to 32 and the maximum training epochs was set to 25. If there was no degradation in loss on the validation set after 5 epochs, the training process would be early stopped. We saved the model with the lowest loss on the validation set for final testing. The mean square error (MSE) and mean absolute error (MAE) are used as metrics. We followed [Jia et al., 2023] and set the seed to 2023 to ensure the reproducibility of the results. All experiments are repeated 5 times and we set the mean of the metrics as the final results, as shown in Table 1 .During the model training process, we conducted tests using the parameters that achieved the lowest loss on the validation set. This process was repeated five times, and the error bars were calculated and presented in Table 4 . The results in Table 4 clearly demonstrate the stability of our proposed method, TPGN, further confirming its superior overall performance compared to other baseline models, being SOTA approach.We conducted a comprehensive experiment to analyze the performance variation of the models across different forecasting lengths, as mentioned in Subsection 4.1. According to the experimental results in Table 1 , we selected representative methods from different paradigms: WITRAN (RNN-Based), TimesNet (CNN-Based), TimeMixer (MLP-Based), and iTransformer (Transformer-Based), and compared them with TPGN. The results of dataset ECL have already been presented in Figure 3 . Therefore, in this section, we present the other dataset results, as shown in Figure 5 -Figure 8 , and further analysis them. Through the results in Figure 3 , Figure 5 , Figure 6 , Figure 7 , and Figure 8 , we can clearly observe that as the sequence length increases, all models show varying degrees of performance degradation across different datasets. However, TPGN not only consistently maintains SOTA performance across all tasks but also exhibits slower performance decay with increasing forecasting lengths in most cases. This demonstrates the significant advantage of TPGN in predicting longer-range tasks, further confirming its ability to effectively extract information from limited inputs and apply them well to prediction tasks.To assess the robustness of TPGN, we conducted experiments following the settings of MICN [Wang et al., 2023] and WITRAN [Jia et al., 2023] , and introduced a simple white noise injection. Specifically, a random proportion ε of data was selected from the original input sequence, and random perturbations within the range [-2X i , 2X i ] were applied to the selected data, where X i represents the original data. Subsequently, the injected noisy data was used for training, and we recorded the MSE and MAE metrics in Table 5 .It can be observed that as the perturbation ratio increases, there is a slight increase in the MSE and MAE metrics in terms of forecasting. This indicates that TPGN exhibits good robustness in handling data with low noise levels (up to 10%) and has a significant advantage in effectively handling various abnormal data fluctuations.In our model, we have only two hyperparameters. One is the number of hidden units, denoted as d m , which is a common hyperparameter in many models. Its value is determined through parameter search and further validation on the validation set. During our parameter search process, we have observed that different models exhibit significant variations in the optimal d m values for the same task. Similarly, even for the same model, this variability persists when facing different tasks. Therefore, in this paper, we focus our analysis solely on the parameter norm, as it showcases the notable differences and its impact on the model's performance.In the Subsection 3.2, we mentioned that the parameter norm should be determined based on the dataset, which has been previously acknowledged in related works [Jia et al., 2023] . However, to validate the appropriateness of our norm selection in the preparatory work, we conducted a statistical analysis on the variations across different task datasets, as shown in Table 6 . Through statistical analysis of the dataset, we have found that there are some differences between the training set and the validation set. These differences are due to the inherent characteristics of time series data and are considered normal fluctuations. If the variances of the training set and validation set are roughly in the same range, it can be assumed that their fluctuations are roughly consistent. In this case, there are no significant differences in data distribution between the training set and validation set, so there is no need for normalization. The hyperparameter norm should be set to 0.However, when there is a relatively large difference in variance between the training set and the validation set (approximately twice or half), it indicates a significant difference in data distribution between the two sets. In such cases, the hyperparameter norm should be set to 1 to facilitate better training of the model.Additionally, weather datasets are a special case because they contain negative values, which leads to their mean being close to zero and a significant difference between the variance and mean. This is reasonable for weather data. On the other hand, traffic datasets do not have negative values. Therefore, even if the variances of the training set and validation set are similar, their differences can still be observed when analyzed in conjunction with the mean. To highlight the advantages of TPGN in long-range time series forecasting tasks, we compared TPGN's showcase with the showcases of the second-best and third-best models, selected based on MSE as the evaluation metric, for each dataset. The forecasting results showcases are shown in Figure 9 to Figure 23 .The comparison of the aforementioned cases clearly demonstrates the superiority of our method in terms of forecasting results. This unequivocally reaffirms the SOTA performance of TPGN across various domains. Moreover, it provides further evidence that PGN, as the novel successor to RNNs, can be effectively applied to long-range time series forecasting tasks. Justification: We provide detailed information about our experimental setup and computational devices in Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 


Decision: The submission addresses the problem of lengthy information paths in RNNs by proposing a fully parallel gated model with O(1) path length. The approach is novel and well supported by discussions and empirical results. The rebuttal discussion led to a convergence in the reviewers' opinion, which broadly support acceptance of the work.

Reviews: review: paper_summary: paper_summary: This paper proposes a new network called PGN to capture the long-term dependencies of time series. Based on PGN, this paper further design TPGN for long-range time series forecasting. TPGN consists of two branches to respectively capture the long-term periodic patterns and short-term information of time series. Extensive experiments are conducted to show the effectiveness and efficiency of the TPGN.
strength_weakness: strength_weakness: W1. The computational complexity of TPGN is not well discussed in this paper, and it would be better if the inference efficiency was adequately discussed as the time series size increases.

W2. Some presentation needs to be improved. For example, it is difficult for readers to quickly get important conclusions on Table 1 and Table 4.
questions: questions: In table of experiment comparison, could you explain why TPGN-long outperform TPGN in some cases.
limitations: limitations: Yes, the authors have fully discussed the limitations of their work.
paper_summary: paper_summary: The paper introduces a new model paradigm which aims to solve the traditional bottlenecks of RNN models, such as non-parallel computation, gradient explosion/vanishing issues, etc.
strength_weakness: strength_weakness: 1. The overall contribution is not very significant.
2. Some questions regarding the time complexity and experiments need to be clarified.
questions: questions: The model proposed in this paper is pretty neat and easy to follow. My questions mainly focus on the time complexity and experiments:
1. I think the total amount of computation done in the PGN should be O(L^2) because it is 1 + 2 + … + L, which is O(L^2). Although the PGN removes half of the computations of the self-attention, the self-attention and PGN still share the same asymptotic complexity. Thus, technically, replacing the PGN with self-attention won’t change the time complexity asymptotically. But I agree that it should be faster than RNN since it enables parallel computation as self-attention does. Also, this is the reason why I think the overall contribution is less exciting than the paper title claims. Can the authors kindly address this issue?
2. The above discussion can also be tested in the Efficiency of Execution experiment.
3. Regarding the experiments, only an input length of 168 is tested. Why did the authors choose to fixate on this input length instead of testing some other options?
4. In the ablation test, it seems that on some datasets (e.g., ETTh1), TPGN’s improvement is very slight compared to its LSTM/GRU/MLP ablations. Can the authors provide some analysis on these cases?
5. I am also interested to see some analysis on what would happen if the PGN is replaced by self-attention.
limitations: limitations: N/A
paper_summary: paper_summary: This paper focuses on long-range time series forecasting problems. To address the limitations of RNNs, a novel paradigm called PGN is introduced as an alternative, providing shorter information propagation paths. Building upon PGN, the paper further presents a generic temporal modeling framework named as TPGN, which effectively captures both long-term and short-term periodic patterns, as well as local and global information, through a dual-branch design. The experimental results in this paper demonstrate that TPGN exhibits excellent performance in time series forecasting.
strength_weakness: strength_weakness: W1. For tables with a large amount of content, such as Table 1, it may be beneficial to consider using different colors for highlighting, as it could enhance clarity. Additionally, another option to consider is moving some of the experimental results to an appendix.

W2. While TPGN exhibits some advantages in terms of efficiency, I have noticed that it still appears to be challenging to reach the optimal level. Specifically, I have noticed that as the input sequence size increases, the efficiency of TPGN may gradually become inferior to that of iTransformer.
questions: questions: Q1. Why was the Gated Mechanism designed in PGN this way in Figure 2 (a)? Can this part be replaced with GRU or other RNN variants?

Q2. Is it necessary to have two Linear layers in the design of the short-term branch in TPGN?
limitations: limitations: yes
paper_summary: paper_summary: This paper proposed a Parallel Gated Network (PGN) as a successor to RNN, featuring a Historical Information Extraction (HIE) layer to directly capture information from previous time steps. Additionally, it introduces a Temporal PGN (TPGN) framework with two branches to capture both long-term periodic and short-term semantic patterns, demonstrating state-of-the-art performance in long-range time series forecasting.
strength_weakness: strength_weakness: 1. The major issue with this paper is the lack of analysis and comparison with significant literature. The entire paper's premise is the traditional RNN's failure in long-term sequence problems due to the long information propagation paths of its recurrent structure. However, as far as I know, SegRNN[1] has already addressed these shortcomings of traditional RNN in long-term forecasting through segmented iteration and parallel prediction. Yet, there is no discussion on this in the paper. Please compare your method with it and clarify your differences and advantages.

2. In Section 2, you should distinguish between Linear-based and MLP-based methods. The former has only single-layer parameter connections, while the latter has multiple layers and can learn non-linear features due to the presence of activation functions. Methods like DLinear and FITS should be classified as Linear-based methods.

3. The description of HIE is unclear: (i) The process shown in Figure 2(a) suggests first performing linear mapping and then zero-padding, which conflicts with Equation 1 in the paper, where H = HIE(Padding(X)), and the actual code. It is recommended to modify Figure 2 to make this clearer. (ii) Line 169 describes that “HIE(·) is a linear layer,” but in practice, the behavior of HIE is more like a sliding aggregation operation of CNN (or TCN) rather than a sorely linear mapping. Given **(ii)**, calling the proposed method RNN-based is debatable since it is more likely TCN-based. 

4. You should include an ablation analysis of the normalization layer, explaining its impact on TPGN achieving state-of-the-art results.

5. Although the authors provide source code, it does not include the hyperparameter settings required to reproduce the key results in the paper, meaning there is no directly runnable script. Are the hyperparameters in the main results all defaults? For instance, is TPGN_period=24? If not, providing a complete script file that can be run directly is necessary.


[1] Lin, S., Lin, W., Wu, W., Zhao, F., Mo, R., & Zhang, H. (2023). Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308.11200
questions: questions: See Weaknesses.
limitations: limitations: The authors have already described the limitation of this work in the paper, namely the lack of modeling multivariate relationships. To some extent, this is not a significant issue because many current cutting-edge studies have demonstrated that focusing solely on univariate temporal relationships can also be effective in multivariate tasks.

Comments: title: Thanks once more
Thanks
Thanks
Thanks
Thanks for rebuttal
Thanks
comment: After we sent comments earlier, we discovered a bug in the system's notifications as we did not receive any messages. However, we have noticed that this issue appears to have been resolved. Therefore, we sincerely thank you once more. Thank you once again for your time and valuable suggestions. We will include the above discussions in the revised version.
Thanks for your time and positive recommendation!
Most of my concerns have been addressed, I will keep my positive recommendation.
Thank you once again for your time and valuable suggestions. We will include the above discussions in the revised version.
Thanks for the authors' response. My major concern regarding the time complexity has been addressed. I'll raise my score.
Thank you for your time and positive comments!
I have carefully read the rebuttal and all my concerns have been well addressed. I will accordingly raise my rate. Thanks.
Thank you once more for your time and valuable suggestions. In the updated revisions, we will include the necessary modifications, analysis as mentioned earlier, and the results compared with SegRNN.
Thank you for the detailed rebuttal; it addressed most of my concerns. The updated version should make the neccessary modifications mentioned in the rebuttal and include the analysis and results compared with SegRNN. I have increased my initial score.