Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting

1 INTRODUCTION . Time series forecasting is the cornerstone for downstream tasks such as decision making and risk management . As an example , reliable prediction of the online traffic for micro-services can yield early warnings of the potential risk in cloud systems . Furthermore , it also provides guidance for dynamic resource allocation , in order to minimize the cost without degrading the performance . In addition to online traffic , time series forecasting has also found vast applications in other fields , including disease propagation , energy management , and economics and finance . The major challenge of time series forecasting lies in constructing a powerful but parsimonious model that can compactly capture temporal dependencies of different ranges . Time series often exhibit both short-term and long-term repeating patterns ( Lai et al. , 2018 ) , and taking them into account is the key to accurate prediction . Of particular note is the more difficult task of handling long-range dependencies , which is characterized by the length of the longest signal traversing path ( see Proposition 2 for the definition ) between any two positions in the time series ( Vaswani et al. , 2017 ) . The shorter the path , the better the dependencies are captured . Additionally , to allow the models to learn these long-term patterns , the historical input to the models should also be long . To this end , low time and space complexity is a priority . Unfortunately , the present state-of-the-art methods fail to accomplish these two objectives simultaneously . On one end , RNN ( Salinas et al. , 2020 ) and CNN ( Munir et al. , 2018 ) achieve a low time complexity that is linear in terms of the time series length L , yet their maximum length of the signal traversing path is O ( L ) , thus rendering them difficult to learn dependencies between distant positions . On the other extreme , Transformer dramatically shortens the maximum path to be O ( 1 ) at the sacrifice of increasing the time complexity to O ( L2 ) . As a consequence , it can not tackle very long sequences . To find a compromise between the model capacity and complexity , variants of Transformer are proposed , such as Longformer ( Beltagy et al. , 2020 ) , Reformer ( Kitaev et al. , 2019 ) , and Informer ( Zhou et al. , 2021 ) . However , few of them can achieve a maximum path length less than O ( L ) while greatly reducing the time and space complexity . In this paper , we propose a novel pyramidal attention based Transformer ( Pyraformer ) to bridge the gap between capturing the long-range dependencies and achieving a low time and space complexity . Specifically , we develop the pyramidal attention mechanism by passing messages based on attention in the pyramidal graph as shown in Figure 1 ( d ) . The edges in this graph can be divided into two groups : the inter-scale and the intra-scale connections . The inter-scale connections build a multiresolution representation of the original sequence : nodes at the finest scale correspond to the time points in the original time series ( e.g. , hourly observations ) , while nodes in the coarser scales represent features with lower resolutions ( e.g. , daily , weekly , and monthly patterns ) . Such latent coarser-scale nodes are initially introduced via a coarser-scale construction module . On the other hand , the intra-scale edges capture the temporal dependencies at each resolution by connecting neighboring nodes together . As a result , this model provides a compact representation for long-range temporal dependencies among far-apart positions by capturing such behavior at coarser resolutions , leading to a smaller length of the signal traversing path . Moreover , modeling temporal dependencies of different ranges at different scales with sparse neighboring intra-scale connections significantly reduces the computational cost . In short , our key contributions comprise : • We propose Pyraformer to simultaneously capture temporal dependencies of different ranges in a compact multiresolution fashion . To distinguish Pyraformer from the stateof-the-art methods , we summarize all models from the perspective of graphs in Figure 1 . • Theoretically , we prove that by choosing parameters appropriately , the maximum path length of O ( 1 ) and the time and space complexity of O ( L ) can be reached concurrently . To highlight the appeal of the proposed model , we further compare different models in terms of the maximum path and the complexity in Table 1 . • Experimentally , we show that the proposed Pyraformer yields more accurate predictions than the original Transformer and its variants on various real-world datasets under the sce- nario of both single-step and long-range multi-step forecasting , but with lower time and memory cost . 2 RELATED WORKS . 2.1 TIME SERIES FORECASTING . Time series forecasting methods can be roughly divided into statistical methods and neural network based methods . The first group involves ARIMA ( Box & Jenkins , 1968 ) and Prophet ( Taylor & Letham , 2018 ) . However , both of them need to fit each time series separately , and their performance pales when it comes to long-range forecasting . More recently , the development of deep learning has spawned a tremendous increase in neural network based time series forecasting methods , including CNN ( Munir et al. , 2018 ) , RNN ( Salinas et al. , 2020 ) and Transformer ( Li et al. , 2019 ) . As mentioned in the previous section , CNN and RNN enjoy a low time and space complexity ( i.e. , O ( L ) ) , but entail a path of O ( L ) to describe long-range dependence . We refer the readers to Appendix A for a more detailed review on related RNN-based models . By contrast , Transformer ( Vaswani et al. , 2017 ) can effectively capture the long-range dependence with a path of O ( 1 ) steps , whereas the complexity increases vastly from O ( L ) to O ( L2 ) . To alleviate this computational burden , LogTrans ( Li et al. , 2019 ) and Informer ( Zhou et al. , 2021 ) are proposed : the former constrains that each point in the sequence can only attend to the point that is 2n steps before it , where n = 1 , 2 , · · · , and the latter utilizes the sparsity of the attention score , resulting in substantial decrease in the complexity ( i.e. , O ( L logL ) at the expense of introducing a longer maximum path length . 2.2 SPARSE TRANSFORMERS . In addition to the literature on time series forecasting , a plethora of methods have been proposed for enhancing the efficiency of Transformer in the field of natural language processing ( NLP ) . Similar to CNN , Longformer ( Beltagy et al. , 2020 ) computes attention within a local sliding window or a dilated sliding window . Although the complexity is reduced to O ( AL ) , where A is the local window size , the limited window size makes it difficult to exchange information globally . The consequent maximum path length is O ( L/A ) . As an alternative , Reformer ( Kitaev et al. , 2019 ) exploits locality sensitive hashing ( LSH ) to divide the sequence into several buckets , and then performs attention within each bucket . It also employs reversible Transformer to further reduce memory consumption , and so an extremely long sequence can be processed . Its maximum path length is proportional to the number of buckets though , and worse still , a large bucket size is required to reduce the complexity . On the other hand , ETC ( Ainslie et al. , 2020 ) introduces an extra set of global tokens for the sake of global information exchange , leading to an O ( GL ) time and space complexity and an O ( 1 ) maximum path length , where G is the number of global tokens . However , G typically increases with L , and the consequent complexity is still super-linear . Akin to ETC , the proposed Pyraformer also introduces global tokens , but in a multiscale manner , successfully reducing the complexity to O ( L ) without increasing the order of the maximum path length as in the original Transformer . 2.3 HIERARCHICAL TRANSFORMERS . Finally , we provide a brief review on methods that improve Transformer ’ s ability to capture the hierarchical structure of natural language , although they have never been used for time series forecasting . HIBERT ( Miculicich et al. , 2018 ) first uses a Sent Encoder to extract the features of a sentence , and then forms the EOS tokens of sentences in the document as a new sequence and input it into the Doc Encoder . However , it is specialized for natural language and can not be generalized to other sequence data . Multi-scale Transformer ( Subramanian et al. , 2020 ) learns the multi-scale representations of sequence data using both the top-down and bottom-up network structures . Such multi-scale representations help reduce the time and memory cost of the original Transformer , but it still suffers from the pitfall of the quadratic complexity . Alternatively , BP-Transformer ( Ye et al. , 2019 ) recursively partitions the entire input sequence into two until a partition only contains a single token . The partitioned sequences then form a binary tree . In the attention layer , each upper-scale node can attend to its own children , while the nodes at the bottom scale can attend to the adjacent Observat ion Embedding Posit ional Encoding PAM Add & Norm N× Add & Norm Gather Features Linear Output Predict ions CSCM Predict ion Strategy 1 Predict ion Strategy 2 Attent ion Decoder Output Predict ionsLinear Feed Forward Covariates Embedding Observat ions Covariates Figure 2 : The architecture of Pyraformer : The CSCM summarizes the embedded sequence at different scales and builds a multiresolution tree structure . Then the PAM is used to exchange information between nodes efficiently . A nodes at the same scale and all coarser-scale nodes . Note that BP-Transformer initializes the nodes at each scale by embedding the corresponding sequences , whereas Pyraformer only embeds the original sequence and introduces the coarser-scale nodes using a construction module in a more flexible manner . Moreover , BP-Transformer is associated with a denser graph than Pyraformer , thus giving rise to a higher complexity of O ( LlogL ) . 3 METHOD . The time series forecasting problem can be formulated as predicting the future M steps zt+1 : t+M given the previous L steps of observations zt−L+1 : t and the associated covariates xt−L+1 : t+M ( e.g. , hour-of-the-day ) . To move forward to this goal , we propose Pyraformer in this paper , whose overall architecture is summarized in Figure 2 . As shown in the figure , we first embed the observed data , the covariates , and the positions separately and then add them together , in the same vein with Informer ( Zhou et al. , 2021 ) . Next , we construct a multiresolution C-ary tree using the coarserscale construction module ( CSCM ) , where nodes at a coarser scale summarize the information of C nodes at the corresponding finer scale . To further capture the temporal dependencies of different ranges , we introduce the pyramidal attention module ( PAM ) by passing messages using the attention mechanism in the pyramidal graph . Finally , depending on the downstream task , we employ different network structures to output the final predictions . In the sequel , we elaborate on each part of the proposed model . For ease of exposition , all notations in this paper are summarized in Table 4 .

Summary: The authors propose a new architecture to tackle the problem of long sequence temporal forecasting (LSTF) – which looks at capturing long-range dependencies in time-series data by providing more direct paths between the output and distant history.   The Pyraformer takes an interesting spin on the current state-of-the-art sparse transformers, consisting of 2 main components: 1.	A dilated CNN encoder to learn coarse-scale representations at multiple resolutions. 2.	A decoder with a pyramid structure that applies attentions masks to a limited subset of nearest neighbours (i.e. parents, adjacent nodes and children) – effectively sparsifying fully connected attention patterns by imposing a prior structure onto attention patterns.