Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies long time series forecasting by using segment correlation attention. For this, time series is segmented into periodical shorter sequences, and attention is computed on each segment and final representation is formed by concatenating segments’ self-attention representations. 
main_review: Strengths:

- Exploiting periodical information for long time series forecasting is interesting 
- Paper is fluent and easy to follow.

Weaknesses:
- Comparisons to earlier work 
- Misses ablation study
- Limited novelty, dual prediction already exists in bidirectional RNN (LSTM, GRU) time series forecasting and it is not clear how the segmentation is computed, and how it is different than Wu et al 2021 (see below).

Concatenating the segmented time series representations corresponds to concatenating slightly different versions of a short sequence. It can be interesting to see if using the whole sequence is better than using the last segmented sequence. It misses the ablation study of using the longer sequence compared to using the last segmented sequence. 

It misses the comparison to the earlier work on periodical time series forecasting:
- Cinar, Y. G., Mirisaee, H., Goswami, P., Gaussier, E., & Aït-Bachir, A. (2018). Period-aware content attention RNNs for time series forecasting with missing values. Neurocomputing, 312, 177-186.
- Wu, H., Xu, J., Wang, J., & Long, M. (2021, May). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Thirty-Fifth Conference on Neural Information Processing Systems.

---

- Providing the equation or the algorithm for the SEG function in Eq. 6 can improve the reproducibility of the approach, and help to better understand the contribution of the paper. 
- The best scores are presented in bold in Table 1 and 2, but not in Table 3, 4, and 5. Can you please show the best scores in bold in Table 3, 4, and 5?

summary_of_the_review: Long time series forecasting using periodical information is interesting. The novelty of the paper is limited as the dual task already exists in bi-directional RNN based models for time series. It misses ablation study of using only the last segmentation instead of the long time series.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a Transformer-based model called SCformer to perform long sequence time series forecasting. The key idea is to replace the canonical self-attention with efficient segment correlation attention (SCAttention) mechanism to capture long short-term dependencies. Experiment results on several datasets showed the effectiveness of the proposed method.
main_review: 
Strengths

* This paper is well organized.
* Long sequence time series forecasting is an interesting problem to investigate.
* The proposed SCformer is technically reasonable.

Weaknesses
* The overall technical novelty is limited.
* Several relevant works are not mentioned or compared.
* Some details of the proposed SCformer are not clear.

The major concern is over the technical novelty. The key idea of SCformer is very similar to Autoformer, both leverage the series-level correlations. In fact, Autoformer could be easily adapted to capture segment-level correlations.  In addition, the dual-task has been used in many previous works.

Another concern is that Autoformer and several related works are not compared:

[1] "Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting." Sen, Rajat, Hsiang-Fu Yu, and Inderjit S. Dhillon NeurIPS 2019.
[2] "Modeling long-and short-term temporal patterns with deep neural networks." Lai, Guokun, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu, SIGIR 2018.
[3] "Shape and time distortion loss for training deep time series forecasting models." Vincent, L. E., and Nicolas Thome NeurIPS 2019.

The authors mentioned that their methods differ from Autoformer in the way of correlation computation and
aggregation. It is not clear which one is more efficient in practice.

How to determine segment length in the experiment?

In Figure 1, it is not clear why the input of decoder embedding is X_t1:X_t0
summary_of_the_review: See above
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a SCFORMER, which replaces the canonical attention in the Transformer with the segment correlation attention. The motivation of using the segment correlation is to reduce the memory usage of the scale-dot product attention of the Transformer. To further improve the performance, the paper proposes a dual task, which use the current time series to predict the past time series. 
main_review: Strengths:
1. The paper is well-motivated that the time complexity of the Transformer is quadratic to the number of time steps.
2. Using the dual task to improve the performance is interesting, and the experimental results show that the dual task could help to improve the results.
3. The experimenal results could demonstrate the effectiveness of the proposed segment correlation and dual task.

Weakness:
1. The novelty of the proposed segment correlation is limited, which simply truncates the long time series into small segments and calculates attention scores over the segments.
2. The time complexity $O(L^2//L^2_{seg})$ is incorrect, which should be $O(L^2//L_{seg})$ instead. $O(L^2//L^2_{seg})$ is the complexity among segments, and the complexity of each segment is not taken into consideration, which is $O(L_{seg})$
3. For the multivariate setting in table 5, Transformer trained without dual task performs better than Transformer with dual task. The results imply that the dual task might not be very useful to some degree.
summary_of_the_review: This paper introduces a SCFORMER to address the problem that the complexity of the Transformer increases quadratically with the length of time series. The essential idea is to segment the input into small segments and calculate attention among the segments rather than each data point. To further improve the performance, this paper introduces an interesting dual task, which uses the current time series as input and will predict the historical time series. However, this the novelty of the segment correlation attention within the SCFORMER is limited and the time complexity is incorrect. Besides, the experimental results in table 5 do not sufficiently support the claim that the dual task could help model to achieve more robust results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the long time series sequence prediction problem with Transformer. The main contribution is proposing a SCAttention mechanism to replace the original attention operation by dividing the whole sequence into equal-length segments and calculating attention among them, which decreases the computation cost significantly. 
main_review: The paper studies the long time series sequence prediction problem with Transformer. The main contribution is proposing a SCAttention mechanism to replace the original attention operation by dividing the whole sequence into equal-length segments and calculating attention among them, which decreases the computation cost significantly. 
Generally, the paper writing is clear, and the experimental results seem promising. However, I have the following concerns about this paper:
1. The methodology is quite intuitive, while it is reasonable to decrease the computation cost by segmenting the sequence, it is not well discussed why the performance could be improved significantly?
2. It is wired that the performance in Table 1 and Table 2 is different from that reported in the Informer paper. Is there any reason or did I misunderstand anything?
3. In the ablation study about SCAttention, is the SCformer with canonical attention the same as the original transformer? If yes, why the performance is so good compared to other transformer variants?
4. While experimental shown useful, the novelty of the dual reverse prediction task is limited. Similar ideas have been explored in other time series and traffic forecasting tasks, e.g., use the future 12 steps to predict the historical 12 steps, use bi-directional LSTMs.
5. finally, it would be better to analyze the model from more aspects, including the influence of encoder/decoder blocks, the number of parameters, analysis and reason about how to select the suitable segment length from Table 4, the variance of the performance, and so on. 

Among these comments, 1, 2, and 3 are my key concerns. I may update the score if they can be adequately solved.  
summary_of_the_review: The technical contribution of this paper is marginal and the experimental analysis can be improved.
review_title: null

Meta-review: The paper proposes a Transformer-based model called SCformer to perform long sequence time series forecasting by computing efficient segment correlation attention. The reviewers think the method lacks novelty and the experiments need a detailed ablation study.
Decision: Reject