Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new hierarchical transformer architecture with constant connection path length and linear time and space complexity for long-range time series modeling. The module at core is a pyramidal attention network that makes multi-resolution representations in a tree structure and perform attention operations on the tree. A stack of convolutions is used to initialize the pyramidal tree. Experiments show the proposed method is able to make more accurate predictions with significantly fewer attention operations and, as a result, less time and memory expenses.
main_review: ## Strengths
1. The paper is well-written and well-motivated with sufficient technical details
2. The extensive empirical results demonstrate the effectiveness and efficiency of the proposed method
3. A proof is provided to guarantee the linear complexity of long sequence encoding

## Comments
1. More datasets for multi-step forecasting would be helpful in evaluating the method.
2. The reason of using the second prediction module in multi-step forecasting need more justification, e.g. why take the concatenation as key/value in the second layer.
summary_of_the_review: Overall I find this paper quite interesting with great potential contribution to the community, and recommend an accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Pyraformer, a low-complexity pyramidal attention model for long-range time-series modelling and forecasting. The proposed architecture is build upon a pyramidal attention module (PAM) in which the inter-scale tree structure summarizes features at different resolutions and the intra-scale neighbouring connections model the temporal dependencies of different ranges. The proposed framework harnesses the benefit of both transformer and RNN. Evaluation on Electricity, wind, App Flow, and ETT dataset against baseline models including Informer., LogTrans, Longformer, Reformer, ETC shows superior performance.
main_review: Strength:
1. The paper is well written and the proposed architecture is easy to understand. I believe the equations are correct, and the included model illustrations makes the model design very intuitive.
2. Extensive evaluation is provided, and the proposed model consistently outperforms baseline models.
3. A ablation study is provided to justify the effectiveness of each component in the proposed architecture.

Weakness:
1. Although I couldn't find any specific publication, I believe hierarchical structure illustrated in Figure 1 (d) has been explored before in the context of LSTM/RNN. In addition, another line of work trying to apply the residual connections from ResNet to RNNs (e.g., https://arxiv.org/pdf/1701.03360.pdf) also provides a means for long-range time-series modelling. 
2. The paper states that Pyraformer simultaneously capture temporal dependencies of different ranges in a compact multi resolution fashion. This is intuitively understandable. However, it would be more convincing if the evaluation could also illustrate this perspective. To be more specific, how do we know from the current evaluation setting that the model indeed learns multi resolution temporal dependencies?
3. Connected to comment 2, it would be better if the authors could illustrate a bit more on why these datasets can be used to evaluate long-term dependency modelling. In my understanding, long-term dependency modelling could be a bit different from long-term prediction. The former focuses on extract meaningful information from long-term previous steps, and the latter focuses on accurate prediction into the far future. If the real-world dataset does not include the long-term dependency in it, some study on a synthetic dataset would help readers understand the benefit of the proposed model. In other words, inaccuracy in long-term prediction could be resulted from (1) accumulated noise in predictions along time, and (2) lack of long-term temporal dependency modelling. How does the authors distinguish the result from these two perspectives? 
4. The proposed method uses a factor of 2 to construct the hierarchy. This number seems a bit arbitrary. I'm wondering if this is a design choice with some thoughts behind it, or could this number be treated as a hyper parameter? Basically this number will decide the number of hierarchies in the structure.
summary_of_the_review: Overall it is a solid paper. Please refer to the above session for the detailed discussion. I had a few concerns regarding evaluation setup and novelty. If the authors could reply accordingly, I'm willing to reevaluate the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new architecture to tackle the problem of long sequence temporal forecasting (LSTF) – which looks at capturing long-range dependencies in time-series data by providing more direct paths between the output and distant history. 

The Pyraformer takes an interesting spin on the current state-of-the-art sparse transformers, consisting of 2 main components:
1.	A dilated CNN encoder to learn coarse-scale representations at multiple resolutions.
2.	A decoder with a pyramid structure that applies attentions masks to a limited subset of nearest neighbours (i.e. parents, adjacent nodes and children) – effectively sparsifying fully connected attention patterns by imposing a prior structure onto attention patterns.

main_review: Strengths
---
1.	The architecture is well-motivated, tackles the important problem of LSTF, and improves both forecasting performance and computational efficiency of state-of-the-art baselines.
2.	Paper is well written and easy to follow – making motivations and contributions clear.

Weaknesses
---
However, some key architectural details can be clarified further for full reproducibility and analysis. Specifically:
1.	How are historical observations combined with inputs known over all time given differences in sequence lengths (L vs L+M)? The text mentions separate embedding and addition with positional encoding, but clarifications on how the embeddings are combined and fed into the CSCM are needed.
2.	Can each node attend to its own lower-level representation? From equation 2, it seems to be that only neighbouring nodes are attended to, based on the description of N_l^(s).
3.	Do the authors have any guidelines on how to select S/A/C (and consequently N) for a given receptive field L? 

In addition, while the ablation analysis tests the impact of changing CSCM architectures, it would be good to evaluate the base performance without the PAM to determine the value added by attention. This would also provide a simple comparison vs dilated CNNs which have been used successfully in time series forecasting applications  (e.g. WaveNet). 

Finally, could I double check which dataset was used for the ablation analysis as well? I seem to be having some difficulty lining the numbers in Tables 4-6 up with Table 3.

summary_of_the_review: The paper makes a strong contribution to long sequence temporal forecasting, although there are some aspects that need to be verified before it is ready for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a variant of the transformer architecture called pyramidal attention. In this architecture one forms a pyramid graph over the sequence at different resolutions and then applies attention among the neighbors of that graph. This leads to a network that can attend to long sequences of length L with O(L) computation but at the same time attention path between two sequence positions is O(1). This is especially relevant for time-series forecasting as it has potential to summarize the time-series at different scales like hourly, daily, monthly etc. Some experiments are performed on 4 datasets to show the advantage of this architecture over other transformer variants.
main_review: __Pros__

1. Overall I like the idea conceptually. As mentioned in the summary it makes a lot of sense for time-series data as it has an implicit bias of summarizing the past at different resolutions. 

2. This architecture as claimed, can handle long sequences with computational complexity O(L) where $L$ is the number of elements in the sequence. It can also maintain short attention path distance between any node. This distance is O(1) w.r.t L. 

3. The experiments are good enough to demonstrate the advantage of this network over other attention mechanisms.

4. The implementation has been done efficiently using TVM and when released the code base would be of value. I hope the paper is reproducible even though no code has been provided.

5. The empirical speed and memory consumption graphs are useful.

__Cons__

I have certain clarifying questions. i would be happy to raise my scores if these are answered. Further I have some comments that would hopefully make the paper better.

1. It would be better to formally define Signal Traversing Path and forward reference it from the introduction.

2. While reading the initial description in the introduction, it is not fully clear what the nodes at each resolution represent. This is only evident to me after reading section 3.2 which states that convolutional layers with specific strides are applied repeatedly to get the coarser levels. It would be better if one states upfront that some external mechanism is used to extract the initial states of the coarser nodes. Related comment: I think the clarity of section 3,2 can be improved.

3. I might be wrong but I believe N is not defined in section 3.1 before it is used. Please check this (ignore if I missed this somehow).What is the relation between N and S ? I would suggest defining A, L, C, N, S clearly in one place with some indentation in Section 3.1, for ease of reading.

4. Please be slightly more clear in the second paragraph of section 3. 3. I would like to verify my understanding of the multi-step forecasting modules. In the first module, we just have a fixed output layer which is a dense mapping to F outputs, where F is the future prediction length. In the second mapping, there is a decoder that sequentially generated the multi-step output and the decoder is equivalent to the decoder in the original transformer paper. Is my understanding correct?

5. In the experiments the information of the datasets and the experimental setup is not 100% clear to me even after reading the appendix. Please clarify or point me to the relevant places in the paper. In multi-step forecasting what is the length of the future prediction window. Also, is the task rolling prediction over the test set? These should be clearly defined. 

6. is there a validation set? Typically I would prefer if A and C are tuned on a validation set per dataset and the best results are reported. the ordering would stay the same I guess, because fixed set of A, C are used in all experiments.

7. Is there a relation between A, C vs the granularity of the dataset. For example if the data is hourly, then one could build a resolution graph such that groups of 24 are constructed in the first level (24 hours in a day), then groups of 7 (7 days in a week), then 4 weeks in a month. Such a non-uniform pyramidal structure might be very useful. Can the authors comment on this or better yet try this? -- not absolutely essential but it would be interesting.

8. Generally, it would be better if we could rank pyramidal attention vs state of the art models on the well known benchmarks used in the DCRNN paper and newer papers like https://arxiv.org/abs/2103.07719. For instance one could just do the evaluation of pyramidal attention in the same tasks as the linked paper and report the numbers. This would reveal the rank of pyramidal attention in the SOTA table and would be a useful signal for the community. Again this is not required but a nice to have.



summary_of_the_review: I like the overall idea and the execution is decent (please see the pros and cons above). In my mind the pros outweigh the cons and I am currently rating it as above the acceptance threshold. I can strengthen my reviews if the authors answer the questions asked in the cons section.
review_title: null

Meta-review: The authors propose a multi-resolution pyramidal attention mechanism to capture long-range dependencies in time series forecasting, achieving linear time and space complexity. The authors conduced an extensive set of experiments and ablation studies demonstrating that  the proposed method consistently outperforms the state-of-the-art and provided evidence for the various components of the architecture. They also provided a proof guarantee the linear complexity of long sequence encoding and adequately addressed the concerns raised by the reviewers. The additional additional benchmarks conducted by the author further demonstrated the strong performance of the method. All reviewers agreed that this work makes a solid contribution to the field.
Decision: Accept (Oral)