Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a novel decomposition approach to time series forecasting - by decomposing a time series in both the encoder and the decoder, they are able to learn complex patterns in a truly neural way.
main_review: The paper posits an interesting use of decomposition ideas through attention. Both the encoder and decoder and reasonably novel. They also compare MAE and MSE against some modern models.

Overall, the idea is interesting - but lacks comparison against state of the art Transformer Models such as MQTransformer or TFT (https://arxiv.org/abs/2009.14799, https://arxiv.org/abs/1912.09363). Furthermore, they donot try to forecast distributions as most other models such as MQTransformer, DeepAR, TFT would do - so the comparison is still lacking.

Revision: After the additional results, Im happy to switch to a marginal accept
limitations_and_societal_impact: Overall, this is an interesting approach to a limited problem. I like the paper, but the comparisons are lacking enough that I cannot give it an accept. If the authors were to add quantile comparisons and state of the art models (for e.g. how do they handle the martingale quantile problem of Foster and Stine (2020) - https://arxiv.org/abs/2105.06834)
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies long-horizon time series forecasting where the historical input length is shorter than the forecast horizon, i.e, output length. This approach extends the transformer by replacing the self-attention unit by autocorrelation and decomposition units. 
The autocorrelation unit chooses the top $k$ highly correlated time lags from $L$ time lags between the query and key embeddings of a time series of length $L$. The normalized correlation of lag j between the query and key embeddings multiplied by the j-item rotated value embedding. 
The decomposition block decomposes a time series to seasonal and trend components. The trend component is defined by using an average pooling, which is an average value of the neighbouring values. And the seasonal is then defined as the difference between the (observed) time series and the trend component.
The proposed approach autocorrelation unit is compared with standard attention, logsparse attention, LSH and probsparse attention. 

main_review: Pros: 
-The proposed approach is interesting.
-The experimental details are given as number of layers, learning rate, train-val-test set splits, stopping criteria, etc. Sharing the code can further improve the reproducibility.
Cons:
-The choices of the hyper-parameter c, which controls the number of periods considered, is not clear.
-There is no analysis on the decomposition for the known period and trends if it is learnt as it is expected.

The paper is easy to follow but the writing of the paper can be further improved. For instance there are a few misprecisions in the definition and equations, please see the detailed comments below.

The paper misses closely related work on time series forecasting using the periodicity of  time series without decomposing it but learning it with a period-aware attention mechanism, Cinar, Y. G., Mirisaee, H., Goswami, P., Gaussier, E., & Aït-Bachir, A. (2018). Period-aware content attention RNNs for time series forecasting with missing values. Neurocomputing, 312, 177-186. Furthermore, with the current experimental setup it is not very clear the effectiveness of decomposing the seasonality and trend in the data instead of explicitly modeling them as it is done by Cinar et al. without decomposing. It would be interesting to see this comparison in the experimental results.

The parameter c defines the number of periods chosen for a given input history. The logic of keeping it the same (value of 3) for each experiment is not clear. For instance, the electricity consumption time series is known to have daily and weekly periods, for ETT dataset the history of 96 time points covers 24 hour only, but the model asked to find 3 periods. Is there any analysis on the values of k which corresponds to the period length for each experiment? Is it confirming the assumption that the model could find the period in the dataset? 

---
Detailed comments:
-For an input length of $L$ an autocorrelation can be from upto lag $L-1$ as given in Equation 5. If the correlation is calculated between two different inputs it can lag 1 to L, but then it should not be called autocorrelation, please see Equation 6. This makes it confusing to read, as one is expecting to see autocorrelation with upto lag of $L-1$, but actually it defines a correlation upto lag $L$. Changing the name to correlation mechanism instead of auto-correlation can make it more clear.
-In Equation 6, the equation of Autocorrelation(Q,K,V) the index of time lag is given as k but the sum indices i is defined from i to k, it needs to be corrected with i. It would be even better to denote $j$, as $i$ is later used for enumeration of attention heads.

Revision Update: After the author response, the rating is updated to 6: Marginally above the acceptance threshold.

limitations_and_societal_impact: Broader impact is given in the appendix.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of long-term forecasting for time series. There are two challenges: (1) it is difficult to directly study the entangled temporal dependencies for long time series; (2) the traditional self-attention mechanism in Transformer is computationally expensive. To address these two problems, the paper proposes a novel Autoformer with a novel auto-correlation mechanism and a series decomposition module. 
main_review: Positive points:
+ The auto-correlation mechanism is interesting and novel. 
+ The idea of series decomposition which is based on moving average is interesting.
+ The experimental results demonstrate that the proposed Autotransformer is able to achieve better performance and efficiency.

Negative points:
- In equation (1), the dimensions of $\mathcal{X}$ and $\mathcal{X}_t$ are the same: $L\times d$. I'm curious how $AvgPool()$ works on $\mathcal{X}$? Why do you need Padding() operation on $\mathcal{X}$?
- In equation (2), why the length of the inputs is $\frac{I}{2}+O$ but not $I+O$?
- In the second equation of the equation (4), why the input of AutoCorrelation() contains 2 variables?
- What's the motivation of the Roll() operation? 
- In line 175, why do you set $k=c\times log L$? Is it because of the complexity?
- In table 2, for Informer, there are some issues for the first row.
- Notation issues:
  (1) $\mathcal{X}_t$ in equation (1) and equation (5) have different meaning.
  (2) what does $C$ mean in Figure 2?

limitations_and_societal_impact: N/A
review_title: null

Meta-review: The paper presents a Transformer-based time series forecasting technique that decomposes the time series into the trend and seasonal parts using an encoder, and decodes the prediction. The transformer relies on an auto-correlation alternative to self-attention. The method is successfully evaluated on a number of time series problems.
Reviewers appreciate the idea of encoder/decoder decomposition of a time series, and its novelty. Reviewer e5xp asked for transformer-based baselines, which were provided by the authors. Reviewer 8uZe had questions about hyperparameter tuning and about various baselines, which were answered satisfactorily. Reviewer ozCn had questions about notation, which, again, were answered.
After careful consideration of the paper and given review scores of (6, 6, 6, mean 6), it appears that the paper could be accepted as a poster.

Decision: Accept (Poster)