Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting

1 INTRODUCTION . Time series forecasting is the cornerstone for downstream tasks such as decision making and risk management . As an example , reliable prediction of the online traffic for micro-services can yield early warnings of the potential risk in cloud systems . Furthermore , it also provides guidance for dynamic resource allocation , in order to minimize the cost without degrading the performance . In addition to online traffic , time series forecasting has also found vast applications in other fields , including disease propagation , energy management , and economics and finance . The major challenge of time series forecasting lies in constructing a powerful but parsimonious model that can compactly capture temporal dependencies of different ranges . Time series often exhibit both short-term and long-term repeating patterns ( Lai et al. , 2018 ) , and taking them into account is the key to accurate prediction . Of particular note is the more difficult task of handling long-range dependencies , which is characterized by the length of the longest signal traversing path ( see Proposition 2 for the definition ) between any two positions in the time series ( Vaswani et al. , 2017 ) . The shorter the path , the better the dependencies are captured . Additionally , to allow the models to learn these long-term patterns , the historical input to the models should also be long . To this end , low time and space complexity is a priority . Unfortunately , the present state-of-the-art methods fail to accomplish these two objectives simultaneously . On one end , RNN ( Salinas et al. , 2020 ) and CNN ( Munir et al. , 2018 ) achieve a low time complexity that is linear in terms of the time series length L , yet their maximum length of the signal traversing path is O ( L ) , thus rendering them difficult to learn dependencies between distant positions . On the other extreme , Transformer dramatically shortens the maximum path to be O ( 1 ) at the sacrifice of increasing the time complexity to O ( L2 ) . As a consequence , it can not tackle very long sequences . To find a compromise between the model capacity and complexity , variants of Transformer are proposed , such as Longformer ( Beltagy et al. , 2020 ) , Reformer ( Kitaev et al. , 2019 ) , and Informer ( Zhou et al. , 2021 ) . However , few of them can achieve a maximum path length less than O ( L ) while greatly reducing the time and space complexity . In this paper , we propose a novel pyramidal attention based Transformer ( Pyraformer ) to bridge the gap between capturing the long-range dependencies and achieving a low time and space complexity . Specifically , we develop the pyramidal attention mechanism by passing messages based on attention in the pyramidal graph as shown in Figure 1 ( d ) . The edges in this graph can be divided into two groups : the inter-scale and the intra-scale connections . The inter-scale connections build a multiresolution representation of the original sequence : nodes at the finest scale correspond to the time points in the original time series ( e.g. , hourly observations ) , while nodes in the coarser scales represent features with lower resolutions ( e.g. , daily , weekly , and monthly patterns ) . Such latent coarser-scale nodes are initially introduced via a coarser-scale construction module . On the other hand , the intra-scale edges capture the temporal dependencies at each resolution by connecting neighboring nodes together . As a result , this model provides a compact representation for long-range temporal dependencies among far-apart positions by capturing such behavior at coarser resolutions , leading to a smaller length of the signal traversing path . Moreover , modeling temporal dependencies of different ranges at different scales with sparse neighboring intra-scale connections significantly reduces the computational cost . In short , our key contributions comprise : • We propose Pyraformer to simultaneously capture temporal dependencies of different ranges in a compact multiresolution fashion . To distinguish Pyraformer from the stateof-the-art methods , we summarize all models from the perspective of graphs in Figure 1 . • Theoretically , we prove that by choosing parameters appropriately , the maximum path length of O ( 1 ) and the time and space complexity of O ( L ) can be reached concurrently . To highlight the appeal of the proposed model , we further compare different models in terms of the maximum path and the complexity in Table 1 . • Experimentally , we show that the proposed Pyraformer yields more accurate predictions than the original Transformer and its variants on various real-world datasets under the sce- nario of both single-step and long-range multi-step forecasting , but with lower time and memory cost . 2 RELATED WORKS . 2.1 TIME SERIES FORECASTING . Time series forecasting methods can be roughly divided into statistical methods and neural network based methods . The first group involves ARIMA ( Box & Jenkins , 1968 ) and Prophet ( Taylor & Letham , 2018 ) . However , both of them need to fit each time series separately , and their performance pales when it comes to long-range forecasting . More recently , the development of deep learning has spawned a tremendous increase in neural network based time series forecasting methods , including CNN ( Munir et al. , 2018 ) , RNN ( Salinas et al. , 2020 ) and Transformer ( Li et al. , 2019 ) . As mentioned in the previous section , CNN and RNN enjoy a low time and space complexity ( i.e. , O ( L ) ) , but entail a path of O ( L ) to describe long-range dependence . We refer the readers to Appendix A for a more detailed review on related RNN-based models . By contrast , Transformer ( Vaswani et al. , 2017 ) can effectively capture the long-range dependence with a path of O ( 1 ) steps , whereas the complexity increases vastly from O ( L ) to O ( L2 ) . To alleviate this computational burden , LogTrans ( Li et al. , 2019 ) and Informer ( Zhou et al. , 2021 ) are proposed : the former constrains that each point in the sequence can only attend to the point that is 2n steps before it , where n = 1 , 2 , · · · , and the latter utilizes the sparsity of the attention score , resulting in substantial decrease in the complexity ( i.e. , O ( L logL ) at the expense of introducing a longer maximum path length . 2.2 SPARSE TRANSFORMERS . In addition to the literature on time series forecasting , a plethora of methods have been proposed for enhancing the efficiency of Transformer in the field of natural language processing ( NLP ) . Similar to CNN , Longformer ( Beltagy et al. , 2020 ) computes attention within a local sliding window or a dilated sliding window . Although the complexity is reduced to O ( AL ) , where A is the local window size , the limited window size makes it difficult to exchange information globally . The consequent maximum path length is O ( L/A ) . As an alternative , Reformer ( Kitaev et al. , 2019 ) exploits locality sensitive hashing ( LSH ) to divide the sequence into several buckets , and then performs attention within each bucket . It also employs reversible Transformer to further reduce memory consumption , and so an extremely long sequence can be processed . Its maximum path length is proportional to the number of buckets though , and worse still , a large bucket size is required to reduce the complexity . On the other hand , ETC ( Ainslie et al. , 2020 ) introduces an extra set of global tokens for the sake of global information exchange , leading to an O ( GL ) time and space complexity and an O ( 1 ) maximum path length , where G is the number of global tokens . However , G typically increases with L , and the consequent complexity is still super-linear . Akin to ETC , the proposed Pyraformer also introduces global tokens , but in a multiscale manner , successfully reducing the complexity to O ( L ) without increasing the order of the maximum path length as in the original Transformer . 2.3 HIERARCHICAL TRANSFORMERS . Finally , we provide a brief review on methods that improve Transformer ’ s ability to capture the hierarchical structure of natural language , although they have never been used for time series forecasting . HIBERT ( Miculicich et al. , 2018 ) first uses a Sent Encoder to extract the features of a sentence , and then forms the EOS tokens of sentences in the document as a new sequence and input it into the Doc Encoder . However , it is specialized for natural language and can not be generalized to other sequence data . Multi-scale Transformer ( Subramanian et al. , 2020 ) learns the multi-scale representations of sequence data using both the top-down and bottom-up network structures . Such multi-scale representations help reduce the time and memory cost of the original Transformer , but it still suffers from the pitfall of the quadratic complexity . Alternatively , BP-Transformer ( Ye et al. , 2019 ) recursively partitions the entire input sequence into two until a partition only contains a single token . The partitioned sequences then form a binary tree . In the attention layer , each upper-scale node can attend to its own children , while the nodes at the bottom scale can attend to the adjacent Observat ion Embedding Posit ional Encoding PAM Add & Norm N× Add & Norm Gather Features Linear Output Predict ions CSCM Predict ion Strategy 1 Predict ion Strategy 2 Attent ion Decoder Output Predict ionsLinear Feed Forward Covariates Embedding Observat ions Covariates Figure 2 : The architecture of Pyraformer : The CSCM summarizes the embedded sequence at different scales and builds a multiresolution tree structure . Then the PAM is used to exchange information between nodes efficiently . A nodes at the same scale and all coarser-scale nodes . Note that BP-Transformer initializes the nodes at each scale by embedding the corresponding sequences , whereas Pyraformer only embeds the original sequence and introduces the coarser-scale nodes using a construction module in a more flexible manner . Moreover , BP-Transformer is associated with a denser graph than Pyraformer , thus giving rise to a higher complexity of O ( LlogL ) . 3 METHOD . The time series forecasting problem can be formulated as predicting the future M steps zt+1 : t+M given the previous L steps of observations zt−L+1 : t and the associated covariates xt−L+1 : t+M ( e.g. , hour-of-the-day ) . To move forward to this goal , we propose Pyraformer in this paper , whose overall architecture is summarized in Figure 2 . As shown in the figure , we first embed the observed data , the covariates , and the positions separately and then add them together , in the same vein with Informer ( Zhou et al. , 2021 ) . Next , we construct a multiresolution C-ary tree using the coarserscale construction module ( CSCM ) , where nodes at a coarser scale summarize the information of C nodes at the corresponding finer scale . To further capture the temporal dependencies of different ranges , we introduce the pyramidal attention module ( PAM ) by passing messages using the attention mechanism in the pyramidal graph . Finally , depending on the downstream task , we employ different network structures to output the final predictions . In the sequel , we elaborate on each part of the proposed model . For ease of exposition , all notations in this paper are summarized in Table 4 .

Summary: This paper proposes Pyraformer, a low-complexity pyramidal attention model for long-range time-series modelling and forecasting. The proposed architecture is build upon a pyramidal attention module (PAM) in which the inter-scale tree structure summarizes features at different resolutions and the intra-scale neighbouring connections model the temporal dependencies of different ranges. The proposed framework harnesses the benefit of both transformer and RNN. Evaluation on Electricity, wind, App Flow, and ETT dataset against baseline models including Informer., LogTrans, Longformer, Reformer, ETC shows superior performance.