Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
Authors: Haixu Wu
Jiehui Xu
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2021 Poster (2021)
Keywords: Time Series Forecasting
Transformers
Deep Decomposition Model
Auto-Correlation

Abstract: Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: https://github.com/thuml/Autoformer.

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Ters Series Forecasting
Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long (Ex)
School of Software, BNRist, Tsinghua University, China
(whx20, (axx20,xjh200 xjh20] @mails. wwwnthusnnhahaeee edw..aaaeeedd.eee cn, (jimwang, (jimsang.mingshengee.. imingghheng))otsinnnua.. edu. en
Abstract
Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer- based models adopt various self- self-attention mechanisms discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going heyond Transformers, we design Auto- former as novel decomposition architecture with an Auto- Coto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto- cott-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representa- tion aggregation at the sub-series level. Auto-Correlation outperforms self- attention in both efficiency and accuracy. long-term forecasting, Autoformer yields state- of- oh-thh-art accuracy, with 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economies, weather and disease. Code available at this repository: https::// htop::://itthut thub. com/ commtthml//// Autof ormer
Introduction
Time series forecasting has been widely used in energy consumption, traffic and economics planning, weather and disease propagation forecasting. In these real-world applications, one pressing demand extend the forecast time into the far future, which is quite meaningful for the long-term planning and early warning. Thus, this paper, we study the long-term long-term,foreeasting problem of time series, characterizing itself by the large length of predicted time series. Recent deep forecasting models [9] 35] have achieved great progress, especially the Transformer-base models. Benefiting from the self-attention mechanism, Transformers obtain great advantage in modeling long-term dependencies for sequential data, which enables more powerful big models [T]
41 20 28 23
However, the forecasting task is extremely challenging under the long-term setting. First, it is unreliable to discover the temporal dependencies directly from the long-term time series because the dependencies can be obscured by entangled temporal patterns. Second, canonical Transformers with self-attention mechanisms are computationally prohibitive for long-term forecasting because of the quadratic complexity of sequence length. Previous Transformer-based forecasting models sparse version. While performance significantly improved, these models still utilize the point-wise representation aggregation. Thus, in the process of efficiency improvement, they will sacrifice the information utilization because of the sparse wise connections, resulting in a bottleneck for long-term forecasting of time series.
41 17 20 mainly focus on improving self-attention
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To reason about the intricate temporal patterns, we try to take the idea of decompasition, which is standard method in time series analysis [27 It can be used to process the complex time series and extract more predictable components. However, under the forecasting context, it can only be used as the pre-processing of past series because the future is unknown [15] This common usage limits the capabilitiiss of decomposition and overlooks the potential future interactions among decomposed components. Thus, we attempt to go beyond pre-processing usage of decomposition and propose a generic architecture to empower the deep forecasting models with immanent capacity of progressive decomposition. Further, decomposition can ravel out the entangled temporal patterns and highlight the inherent properties of time series [15]. Benefiting from this, try take advantage of the series periodicity to renovate the point-wise connection self- selffattention. We observe that the sub-series at the same phase position among periods often present similar temporal processes. Thus, try to construct a series-level connection based on the process similarity derived by series periodicity.
Based on the above motivations, we propose an original Autoformer in place of the Transformers for long-term time series forecasting. Autoformer still follows residual and encoder-decoder structure but renovates Transformer into decomposition forecasting architecture. By embedding decomposition blocks as the inner operators, Autoformer can progressively separate the long-term trend information from predicted hidden variables. This design allows our model to alternately decompose and refine the intermediate results during the forecasting procedure. Inspired by the stochastic process theory [8 24], Autoformer introduces an of celf-attention, which discovers the sub-series similarity based on the series periodicity and aggre- gates similar sub-series from underlying periods. This series-wing mechanism achieves OLL log L) complexity for length- I, series and breaks the information utilization bottleneck by expanding the point-wise representation aggregation to sub-series level. Autoformer achieves the ofte-oofthhe-art accuracy on six benchmarks. The contributions are summarized as follows:
proposed
mechanism in place
To tackle the intricate temporal patterns of the long-term future, we present Autoformer as decomposition architecture and design the inner decomposition block to empower the deep forecasting model with immanent progressive decomposition capacity.
We propose Corte-CCrreltionn mechanism with dependencies discovery and information aggregation at the series level. Our mechanism is beyond previous self-attention family and can simultaneously benefit the computation efficiency and information utilization.
Autoformer achieves a 38% relative improvement under the long-term setting on six bench- marks, covering five real-world applications: energy, traffic, economics, weather and disease.
2 Related Work
2.1 Models for Time Series Forecasting
Due to the immense importance of time series forecasting, various models have been well developed. Many time series forecasting methods start from the classic tools [32] 9]. ARIMA [6.5] tackles the forecasting problem by transforming the non-stationary process stationary through differencing.
The filtering method is also introduced for series forecasting [18] [0]. Besides, recurrent neural
networks (RNNs) models are used model the temporal dependencies for time series 36 26
DeepAR (8 combines autoregressive methods and RNNs to model the probabilistic distribution of future series. LSTNet [19 introduces convolutional neural networks (CNNs) with recurrent-skip skip connections to capture the short-term and long-term temporal patterns. Attention-based RNNs introduce the temporal attention to explore the long-range dependencies for prediction. Also, many works based on temporal convolution networks (TCN) 34
39 30 31
3 29] attempt to model the temporal causality with the causal convolution. These deep forecasting models mainly focus on the temporal relation modeling by recurrent connections, temporal attention causal convolution.
4
Recently, Transformers 35 38 based on the self-attention mechanism shows great power in sequen- tial data, such as natural language processing [1] 7 audio processing [4] and even computer vision 12 21]. However, applying attt-tttention to long-term time series forecasting is computationally prohibitive because of the quadratic complexity of sequence length L in both memory and time.
LogTrans (20 introduces the local convolution to Transformer and proposes the LogSparse attention to select time steps following the exponentially increasing intervals, which reduces the complexity to (L.(lo I.)2). Reformer [17 presents the local-sensitive hashing (LSH)) attention and reduces the complexity to OLL log L). Informer [4] extends Transformer with KL-divergence based ProbSparse

---

attention and also achieves lo L) complexity. Note that these methods are based on the vanilla Transformer and try to improve the self-attention mechanism a sparse version, which still follows the point-wise dependency and aggregation. In this paper, our proposed Auto- Cuto-Correlation mechanism based on the inherent periodicity of time series and can provide series-wise connections.
2.2 Decomposition of Time Series
As a standard method time series analysis, time series decomposition 27 deconstructs a time series into several components, each representing one of the underlying categories of patterns that more predictable. It is primarily useful for exploring historical changes over time. For the forecasting tasks, decomposition is always used as the pre-processing of historical series before predicting future series [15] 2] such as Prophet [33] with trend-seasonality decomposition and N-BEATS 23 with
basis expansion and DeepGLO [29] with matrix decomposition. However, such pre-processing is limited by the plain decomposition effect of historical series and overlooks the hierarchical interaction between the underlying patterns of series in the long-term future. This paper takes the decomposition idea from new progressive dimension. Our Autoformer harnesses the decomposition as an inner block of deep models, which can progressively decompose the hidden series throughout the whole forecasting process, including both the past series and the predicted intermediate results.
3
Autoformer
The time series forecasting problem is predict the most probable length-O series in the future given the past length-/ series, denoting as input-I-predict-O The long-term forecasting setting is to predict the long-term future, i.e. larger O. As aforementioned, of long-term series forecasting: handling intricate temporal patterns and breaking the bottleneck of computation efficiency and information utilization. To tackle these two challenges, we introduce the decomposition as a builtin block to the deep forecasting model and propose Autoformer as decomposition architecture. Besides, we design the Auto-Correlation mechanism to discover the period-based dependencies and aggregate similar sub-series from underlying periods.
have highlighted the difficulties
3.1 Decomposition Architecture
We renovate Transformer [35 to deep decomposition architecture (Figure I including the inner mechanism, and corresponding Encoder and Decoder.
series decomposition block,
Series decomposition block To learn with the complex temporal patterns in long-term forecasting context, we take the idea of decomposition [7] which can separate the series into trend-cyclicn and seasonal parts. These two parts reflect the long-term progression and the seasonality of the series respectively. However, directly decomposing is unrealizable for future series because the future is just unknown. To tackle this dilemma, we present a series decomposition block as an inner operation of Autoformer (Figure D. which can extract the long-term stationary trend from predicted intermediate hidden variables progressively. Concretely, we adapt the moving average to smooth out periodic fluctuations and highlight the long-tern trends. For length-L input series X R4xd, the process is:
X A (oaddinnaa X - X.,
where X,, Xr E TR 4xd denote the seasonal and the extracted trend-cyclical part respectively. We adopt the A AvgPool(.) for moving average with the padding operation to keep the series length unchanged. We use X2 X SeriesDecomp((X) to summarize above equations, which is model inner block.
Model inputs The inputs of encoder part are the past I time steps Xen f As decomposition architecture (Figure []. the input of Autoformer decoder contains both the seasonal part X'aes e R +))x and trend-cyclical part Naet E xO)x+ to be refined. Each initialization consists of two parts: the component decomposed from the latter half of encoder's input Xen with length & to provide recent information, placeholders with length O filled by scalars. It's formulized as follows:
Xensy Vent = PerrisDeccop((eet 4.1)
Xass Concat (Xens> Xo)
(2)
Xaet = Concat (Kent: XMean),
3

---

Autoformer Encoder
Time
Encoder Imput
Seasonal
Auto- Correlation
Series Decomp
Series Decomp
To Prodict
Forward
Irend -tyclical
Duta Mean
Part
Seasonal Init
Auto- Correlation
Series Decomp
Auto- Correlation
Series Decomp
Feed Forward
Series Decomp
Zero
Prodiction
Trend-cyclical Init
Input Data Mear
Autoformer Decoder
M
Figure I: Autoformer architecture. The encoder eliminates the long-term trend-cyclical part by series decomposition blocks (blue blocks) and focuses on seasonal patterns modeling. The decoder accu- mulates the trend part extracted from hidden variables progressively. The past seasonal information from encoder utilized by the encoder-decoder Auto-Correlation (center green block decoder).
where Nens, Nent R dd denote the seasonal and trend-cyclical parts of Xen respectively, and No, MMean E R0xd denote the placeholders filled with zero and the mean of Xen respectively.
Encoder As shown in Figure [] the encoder focuses on the seasonal part modeling. The output of the encoder contains the past seasonal information and will be used as the cross information to help the decoder refine prediction results. Suppose we have N encoder layers. The overall equations for 1-th encoder layer are summarized as X = Encoder(A"t. (/==1). Details are shown as follows:
st,1
ScriesDecomp AAuto-Corrcaation((111 xx-11
S/.2 SeriesDecomp [eedFForward(S".) S
where """is the eliminated trend part. X S1.2 {I,... JJ denotes the output of I-th encoder layer and X is the embedded X-n- St, [1,2) represents the seasonal component after the i-th series decomposition block in the 1-th layer respectively. We will give detailed description of (uto-Correlation() in the next section, which seamlessly replace the self-attention.
Decoder The decoder contains two parts: the accumulation structure for trend-cyclical components and the stacked Luto-Correlation mechanism for seasonal components (Figure [T. Each decoder layer contains the inner auto-Correlation and encoder-decoder MnttoCorrelltion, which can refine the prediction and utilize the past seasonal information respectively. Note that the model extracts the potential trend from the intermediate hidden variables during the decoder, allowing Autoformer progressively refine the trend prediction and eliminate interference information for period- period-based based dependencies discovery in Cito-Correllaton. Suppose there are M decoder layers. With the latent variable X-N from the encoder, the equations of 1-th decoder layer can be summarized X% Decoder( de
X.N). The decoder can formalized as follows:
SeriesDecomp AAtto-Coreeltionn(X2 x/111)
2!-1)
de de
1,2 de
SeriesDecomp Autto-Correlation(()) l1.1 (NN) + cl.1
de
(4)
s,3 S T1.3 1.
SeriesDecomp FeedForward(()1))
ci.2 +
Tt = Tél + Wi,1 Tdil + W1, T + Wia 1133
where x = S/ I E (1, ... M) denotes the output of 1-th decoder layer. 2: is embedded from Xaes for deep transform and Th = Xtet is for accumulation. S T.. 2,) represent the seasonal component and trend-cyclical component after the i-th series decomposition block in the 1-th layer respectively. Wi,, e (,, represents the projector for the i-th extracted trend T

---

Linear
Concat
Time Delay
ExC
Time Delay Agi
SoftMax
Inverse FFT
Rolli TT)
R(u
Conjugane
LeCs2
Roll(T2
Rimal
FFT
FFT
leC
Resize
EVC
SrC
SxC
Rolli
Limear
Rlmu
Figure 2: Auto-Corres CoroCCorrllation (left) and Time Delay Aggregation (right). We utilize the Fast Fourier Transform to calculate the autocorrelation R(T), which reflects the time-delay similarities. Then similar sub processes rolled to the same index based selected delay and aggregated by R(T).
The final prediction is the sum of the two refined decomposed components, as Ws X Tal where Ws is to project the deep transformed seasonal component X to the target dimension.
3.2
Auto-Correlation Mechanism
As shown in Figure 2 we propose the Auto-Correlation mechanism with series-wise connections to expand the information utilization. auto-Correlation discovers the period- dependencies by calculating the series autocorrelation and aggregates similar sub-series by time delay aggregation.
Period-based dependencies provides similar sub- crob-rocesses. Inspired by the stochastic process theory [8] 224. for real discrete- time process } we can obtain the autocorrelation Rxx (T) by the following equations:
observed that the same phase position among periods naturally
Rxx(T)
lim L T.
Σ N.K.
(5)
Rxx(1 (T) reflects the time-delay similarity between (At) and its lag series (2t-))- As shown in Figure 2 we use the autocorrelation R(T) as the unnormalized confidence of estimated period length T. Then, we choose the most possible &t period lengths TI. , Th. The period-based dependencies are derived by the above estimated periods and can be weighted by the corresponding autocorrelation.
Time delay aggregation The period-based dependencies connect the sub-series among estimated periods. Thus, we present the time delay aggregation block (Figure which can roll the series based on selected time delay Ti, -k- This operation can align similar sub-series that are at the same phase position of estimated periods, which is different from the point-wise dot-product aggregation self-attention family. Finally, we aggregate the sub-series by softmax normalized confidences.
For the single head situation and time series X' with length-L, L,, after the projector, we get query Q, key KC and value Thus, it can replace self-attention seamlessly. The Auto Cotr-Coorelttion mechanism is:
TI The Th arg Topk (Rox(T))
re(1..-.44
Rex(n), ... Rex(T)) SoftMax (Rex(1)), RQx((T))
(6)
&
Auto-CoorelationnQ..K.
Σ R,
where arg Topk(-) is to get the arguments of the Topk autocorrelations and let k = |Ic x log LJ, a hyper-parammter. Rg,x x is autocorrelation between series Q and K. Roll represents operation to A' with time delay т, during which elements that are shifted beyond the first position are re-introduced the last position. For the encoder-decoder Auto-Correlation (Figure K., are from the encoder N and will be resized length-O, Q is from the previous block of the decoder.
5

---

(a) Full Atlention
Tine
(b) Sparse Atlention
Time
kb
x Penood
Pertod Pertod
LogSparse Attention
Time
(d) Auto-Correlation
Time
Figure natt-Correlation vs. self-attention family. Full Attention 335 adapts the fully connection
among all time points. Sparse Attention [17,4] (b) selects points based on the proposed similarity metrics. LogSparse Attention (0 (c) chooses points following the exponentially increasing intervals.
Auto-Correlation (d) focuses on the connections of sub-series among underlying periods.
For the multi-heat version used in Autoformer, with hidden variables of dmodel channels, h heads, the query, key and value for i-th head are Q,, K.,, V; E R Smnde , E fl, ,h.. The process is: MultiHead K, V) = Woutput Concat (headd ... heads)
where head, Auto-Corres tatto-Coreelaton(( Q,, K,, V.).
Efficient computation For period-based dependencies, these dependencies point to sub-processes the same phase position of underlying periods and are inherently sparse. Here, we select the most possible delays to avoid picking the opposite phases. Because we aggregate Olog L) series whose length is L, the complexity of Equations 7 is log L). For the autocorrelation computation (Equation 5), given time series (X,). (xx(T) can be calculated by Fast Fourier Transforms (FFT)
based on the Khinchn theorem 37
Sxx(f) = =F() F (2r) I Nie 12 12
dt / K2et2 dt
(8)
(xx =F F (Fxx(( 1 Sxx (f)
where r E (1, ,L). F denotes the FFT and F-1 is its inverse. denotes the conjugate operation and Sxx(f) in the frequency domain. Note that the series autocorrelation of all lags in (1..... L} can be calculated at once by FFT. Thus, Auto-Correlation achieves the (L log L) complexity.
Auto-Correlation vs. self-attention family Different from the woint-wise self-attention family, Auto-Correlation presents the scries-wise connections (Figure 3 Concretely, for the temporal dependencies, we find the dependencies among sub-series based the periodicity. In contrast, the self-attention family only calculates the relation between scattered points. Though some self- attentions 20 41 consider the local information, they only utilize this help point-wise depen- dencies discovery. For the information aggregation, we adopt the time delay block to aggregate the similar sub-series from underlying periods. In contrast, self-attentions aggregate the selected points by dot-product Benefiting from the inherent sparsity and sub-series-level representation aggregation, Auto-Correlation simultaneously benefit the computation efficiency and information utilization.
4 Experiments
We extensively evaluate the proposed Autoformer on six real-world benchmarks, covering five mainstream time series forecasting applications: energy, traffic, economies, weather and disease.
Datasets Here is a description of the six experiment datasets: (1) ETT dataset contains the data collected from electricity transformers, including load and oil temperature that are recorded every

---

Table 1: Multivariate results with different prediction lengths O E {96, 192, 336, 720]. We set the input length / as 36 for ILI and 96 for the others. A lower MSE or MAE indicates a better prediction.
Models
Autoformer
Informer
LogTrans 20
Reformer 17
LSTNet 19
LS
TCN B
Metric
MSE MAE MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.255 0.339 0.365 0.453 0,768
0.642
0.658
0.619
3.142
1.365 2.041
1.073 3,041
1.330
ETT
192
0.281
0.340 0.533 0.563 0.989
0.757
1.078
0.827
3.154
1.369
2.249
1.112
3,072
1.339
336
0.339
0.372 1.363
0.887
1.334
0.872
1.549
0.972
3.160
1.369
2.568
1.238
3.105
1.348
720
0.422
0.419
3.379
1.388
3.048
1.328
2.631
1.242
3.171
1.368
2.720
1.287
3.135
1.354
Electricity
96
0.201
0.317
0.274
0.368
0.258
0.357
0.312
0.402
0.680
0,645
0.375
0.437
0.985
0.813
192
0.222
0.334 0.296 0.386 0.266
0.368
0.348
0,433
0.725
0.676 0.442 0.473
0.996 0.821
336
0.231
0.338
0,300
0.394
0,280
0.380
0.350
0.433
0.828
0.727 0.439
0.473
1,000
0.824
720
0.254
0.361
0.373
0.439
0.283
0.376
0.340
0.420
0.957
0,811
0,980
0.814
1.438
0.784
Exchange
96
0.197
0.323
0.847
0.752
0,968
0.812
1.065
0.829
1.551
1.058
1.453
1.049
3.004
1,432
192
0.300
0.369
1.204
0.895
1.040
0.851
1.188
0.906
1.477
1.028
1.846
1.179
3.048
1,444
336
0.509
0.524
1.672
1.036
1.659
1.081
1.357
0.976
1.507
1.031
2.136
1.231
3.113
1,459
720
1.447
0.941
2.478
1,310
1,941
1.127
1.510
1.016
2.285
1.243
2.984
1.427
3.150
1.458
96
0.613
0.388
0.719
0.391
0.684
0.384
0.732
0.423
1.107
0.685
0.843
0.453
1,438
0.784
Traffic
192
0.616
0.382 0.696
0.379
0.685
0.390
0.733
0.420
1.157
0.706
0.847 0.453
1.463 0.794
336
0.622
0.337 0.777
0.420
0.733
0.408
0.742
0.420
1.216
0.730
0.853
0.455
1,479 0.799
720
0.660
0.408 0.864
0.472
0.717
0.396
0.755
0.423
1,481
0.805
1,500
0.805
1.499
0.804
Weather
96
0.266
0.336 0.300
0.384
0,458
0.490
0.689
0.596
0.594
0.587
0.369
0,406
0,615 0.589
192
0.307
0.367 0.598 0.544
0.658
0.589
0.752
0.638
0.560
0.565 0.416 0.435 0.629 0.600
336
0.359
0.395
0.578
0.523
0.797
0.652
0.639
0.596
0.597
0.587
0.455 0.454
0.639 0.608
720
0.419
0.428
1.059
0.741
0.869
0.675
1.130
0.792
0.618
0.599
0.535
0.520
0.639 0.610
24
3.483
1.287
5.764
1.677
4,480
1.444
4.400
1,382
6.026
1.770
5.914
1.734
6.624
1.830
36
3.103
1.148
4.755
1.467
4,799
1.467
4.783
1.448
5,340
1.668
6.631 1.845
6,858
1.879
48
2.669
1.085
4.763
1,469
4,800
1.468
4.832
1.465
6.080
1.787
6.736
1.857
6.968
1.892
60
2.770
1.125
5.264
1.564
5.278
1.560
4.882
1.483
5.548
1.720
6.870
1.879 7.127 1.918
ETT means the ETTm2. See supplementary materials for the full benchmark of ETTh1, ETTh2, ETTml.
15 minutes between July 2016 and July 2018. (2) Electricity I dataset contains the hourly electricity consumption of 321 customers from 2012 to 2014. (3) Exchange [19 records the daily exchange rates of eight different countries ranging from 1990 to 2016. (4) Traffic is a collection of hourly data from California Department of Transportation, which describes the road occupancy rates measured by different sensors on San Francisco Bay area freeways. (5) Weather iss recorded every 10 minutes for 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity, etc. (6) H1." includes the weekly recorded influenza-like illness (II.I) patients data from Centers for Disease Control and Prevention of the United States between 2002 and 2021, which describes the ratio of patients seen with IIII and the total number of the patients. We follow standard protocol and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets.
Implementation details Our method is trained with 1.2 loss, using the ADAM (16] optimizer with an initial learning rate of 10-4, Batch size is set to 32. The training process is early stopped within 10 epochs. All experiments are repeated three times, implemented in PyTorch (25) and conducted on a single NVIDIA TITAN RTX 24GB GPUs. The hyper-parameter c of Auto-Correlation is in the range of 1 to 3 to trade off performance and efficiency. See supplementary materials for standard deviations and sensitivity analysis. Autoformer contains 2 encoder layers and I decoder layer.
Baselines We include 10 baseline methods. For the multivariate setting, we select three latest state-
oh-the-art transformer-batt models: Informer [41), Reformer 117, LogTraans (20), two RNN-based models: LSTNet [19), LSTM [3] and CNN-based TCN B as baselines. For the univariate setting, we include more competitive baselines: N-BEATS DeepAR (88) Prophet (33] and ARMIA [1]
http: httt:://pems.dot.com http://wwwa...aaaaa.aaa/aacaaaaa
7

---

Table 2: Univariate results with different prediction lengths O E {96, 192, 336, 720} on typical datasets. We set the input length I as 96. A lower MSE or MAE indicates a better prediction.
Models Autoformer N-BEATS 23 Informer 41
LogTrans 20
Reformer
[17
Prophet 22
[3 [
Metric
MSE
MAE MSE
MAE
MSE
MAE MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.065 0.189 0.082
0.219
0.088 0.225 0.082 0.217 0.131
0.288 0.099 0.237 0.287 0.456 0.211 0.362
192
0.118 0.256 0.120
0.268
0.132 0,283 0.133
0.284 0.186
0.354
0.154 0.310 0.312 0.483 0.261 0,406
336
0.154
0.305 0.226
0.370
0.180
0.336 0.201
0.361
0.220
0.381
0.277 0.428 0.331 0.474 0.317 0.448
720
0.182
0.335 0.188
0.338
0.300
0.435 0.268
0.407
0.267
0.430
0.332 0.468 0.534 0.593 0.366 0.487
96
00.241
0.387 0.156
0.299
0.591
0.615 0.279
0.441
1.327
0,944
0,417 0.515 0.828 0.762 0.112 0.245
Exchange
192
0.273
0.403 0.669
0.665
1.183 0.912 1,950
1.048
1.258
0.924
0,813 0.735 0.909 0.974 0,304 0.404
336
0.508
0.539 0.611
0.605
1.367
0.984 2.438
1.262
2.179
1.296
1.331 0.962 1.304 0.988 0.736 0,598
720
0.991
0.768 1.111 0.860
1.872
1.072
2.010
1.247
1,280
0,953
1.894
1.181 3,238 1.566 1.871 0.935
4.1 Main Results
To compare performances under different future horizons, we fix the input length and evaluate models with wide range of prediction lengths: 96, 192, 336, 720. This setting precisely meets the definition of long-term forecasting. Here are results on both the multivariate and univariate settings.
Multivariate results As for the multivariate setting, Autoformer achieves the consistent state-of- the performance in all benchmarks and all prediction length settings [Table Especially, under setting, compared to previous saut--of-the-eeee results, Autoformer gives 74 % (1.334-0..339) MSE reduction in ETT, 18% (0.280 +0.331) in Electricity. 61 % (1.357-0.509) in Exchange, 15% 11733->0662 in Traffic and 21%
the-art
in Weather. For the input- 36-predict-60 setting of ILI, Autoformer makes 43% (4.882-2.770)) MSE reduction. Overall, Autoformer yields a 38% averaged MSE reduction among above settings. Note that Autoformer still provides remarkable improvements in the Exchange dataset that is without obvious periodicity. See supplementary materials for detailed showcases. Besides, we can also find that the performance of Autoformer changes quite steadily as the prediction length O increases. It means that Autoformer retains better long-term robustness, which is meaningful for real-world practical applications, such as weather early warning and long-term energy consumption planning.
Univariate results We list the univariate results of two typical datasets in Table 2 Under the comparison with extensive baselines, our Autoformer still achieves state-of-the-art performance for the long-term forecasting tasks. In particular, for the (oput-96-predictt 14% (0.180-=0.14; 145) MSE reduction the ETT dataset with obvious periodicity. For the Exchange dataset without obvious periodicity, Autoformer surpasses other baselines by 17% (0.61 101111000 1-0.508) and shows greater long-term forecasting capacity. Also, we find that ARIMA II performs best in the -npee-9-- setting of the Exchange dataset but fails in the long-term setting. This situation of ARIMA can be benefited from its inherent capacity for non-stationary economic data but is limited by the intricate temporal patterns of real-world series.
setting, our model achieves
4.2 Ablation studies
Table 3: Ablation of decomposition in multivariate ETT with MSE metric. Ours adopts our progres- sive architecture into other models. Sep employs two models to forecast pre-decomposed seasonal and trend-cyclical components separately. Promotion is the MSE reduction compared Origin.
Input-96|
Transformer 35
Informer 41
Lonnraan 17
Reformer 20
Promotion
Predict-O
Origin Sep Ours Origin Sep
Ours
Origin
Sep
Ours
Origin Sep
Ours
Sep
Ours
96
0.604 0.311 0.204 0.365 0.490 0.354 0.768 0.862 0.231 0.658 0.445 0.218 0.069 0.347
192
1.060 0.760 0.266 0.533 0.658 0.432 0.989 0.533 0.378 1.078
0.510 0.336 0.300 0.562
336
1,413
0.665 0.375 1.363 1.469 0.481 1.334
0.762
0.362
1.549
1.028
0.366 0.434 1.019
720
2.672 3.200 0.537 3.379 2.766 0,822 3.048
2.601 0.539
2.631 2.845 0.502 0.079 2.332
Decomposition architecture With our proposed progressive decomposition architecture, other
models can gain consistent promotion, especially as the prediction length O increases (Table This
8

---

verifies that our method can generalize to other models and release the capacity of other dependencies learning mechanisms, alleviate the distraction caused by intricate patterns. Besides, our architecture outperforms the pre- pre-procsssing although the latter employs bigger model and more parameters. Especially, pre-decomposing may even bring negative effect because neglects the interaction of components during long-term future, such as Transformer 35 predict-720, Informer 41 predict-336.
Auto-Correlation vs. self-attention family As shown in Table 4 our proposed Auto-Correlation achieves the best performance under various input-I-predict-O settings, which verifies the effective- ness of series-wise connections comparing to point-wise self-attentions (Figure B) Furthermore, we is memory efficiency from the last column of Table 4 which
can also observe that can be used long sequence forecasting, such as (oput-33-preedic-1-e 1440.
Table 4: Comparison of Auto-Correlation and self-attention in the multivariate ETT. We replace the Auto-Correlation in Autoformer with different self-attentions. The "_" indicates the out-of-memory.
Input Length 7
96
192
336
Prediction Length O
336
720
1440
336
720
1440
336
720
1440
Auto-
MSE
0.339
0.422
0.555
0.355
0.429
0.503
0.361
0.425
0.574
Correlation
MAE
0.372
0.419
0.496
0.392
0.430
0.484
0.406
0.440
0.534
Full
MSE 0.375
0,537
0.667
0.450
0.554
0.501
0.647
Attention
35
MAE
0.425
0,502
0,589
0.470
0.533
0.485
0,491
LogSparse
MSE
0.362
0.539
0.582
0.420
0.552
0.958
0.474
0.601
Attention 20
MAE
0.413
0.522
0.529
0.450
0.513
0.736
0.474
0.524
LSH
MSE
0.366
0,502
0.663
0.407
0.636
1.069
0.442
0.615
Attention 17
MAE
0.404
0.475
0.567
0.421
0.571
0.756
0.476
0.532
ProhSparse
MSE
0.481
0.822
0.715
0.404
1.148
0.732
0.417
0.631
1.133
Attention 41
MAE
0.472
0.559
0.586
0,425
0.654
0.602
0,434
0.528
0.691
4.3 Model Analysis
Time series decomposition As shown in Figure 4 without our series decomposition block, the forecasting model cannot capture the increasing trend and peaks of the seasonal part. By adding the series decomposition blocks, Autoformer can aggregate and refine the trend-cyclical part from series progressively. This design also facilitates the learning of the seasonal part, especially the peaks and troughs. This verifies the necessity of our proposed progressive decomposition architecture
Sosorall Hot
Solona Pin
Time Serie cyceey
Tra di eycicaa
100
200
300
400
900
600
700
100
200
300
400
500
500
700
100
200
100
400
3401
700
100)
250
300
400
300
503
7007
(a) Without decomposition block
(b) One decomposition block
(c) Two decomposition blocks
(d) Three decomposition blocks
Figure 4: Visualization of learned seasonal XM and trend-cyclical T.M of the last decoder layer. We gradually add the decomposition blocks in decoder from left to right. This case is from ETT dataset under (ooputt---rrdiic-775 setting. For clearness, we add the linear growth raw data additionally.
Dependencies learning The marked time delay sizes in Figure 5(a) indicate the most likely periods. Our learned periodicity can guide the model to aggregate the sub-series from the same or neighbor phase of periods by Roll(X' TT), i {1.... 6). For the last time step (declining stage), Auto- Correlation fully utilizes all similar sub-series without omissions or errors compared to self-attentions. This verifies that Autoformer can discover the relevant information more sufficiently and precisely.
Complex seasonality modeling As shown in Figure 5 the lags that Autoformer learns from deep representations can indicate the real scasonality of raw series. For example, the learned lags of the

---

Quean
Vilue
(a) Autos CorreeColll
(b) Full Attention
(c) LSH Attention
(d) ProbSparse Attention
Figure 5: Visualization of learned dependencies. For clearness, we select the top-6 time delay sizes
Ti, To of Auto-Correlation and mark them in raw series (red lines). For self-attentions, top-6 similar points with respect to the last time step (red stars) are also marked by orange points.
Daily
Mantrily
Daily Portad
30
Daily Period
Pertod
75
Period
Dumiity O1
Quartorly Paried
Weekkl Perlod
15
Weekly
Yearl Period
50 100 150 200 250
100
350
50
100
150
300
250
100
350
50 100
150
200 250
300
350
0
50
100
150
200 250
300 350
Electricity (Hourly Recorded)
Exchange (Daily Recorded)
(c) Traffic (Hourly Recordedd)
Weather Recordeed)
Figure 6: Statistics of learned lags. For each time series in the test set, we count the top 10 lags learned by decoder for the 1oput96--rredic-3111 task. Figure (a)-(d) are the density histograms.
daily recorded Exchange dataset present the monthly, quarterly and yearly periods (Figure (b)). For the hourly recorded Traffic dataset (Figure 6 (c)), the learned lags show the intervals 24-hours and 1 168-hours, which match the daily and weekly periods of real-world scenarios. These results show that Autoformer can capture the complex seasonalities of real-world series from deep representations and further provide human-interprettblle prediction.
Efficiency analysis We compare the running memory and time among nutto-oorrelaton-bee and self- nnttention-baser models (Figure 7 during the training phase. The proposed Autoformer shows OL log L) complexity in both memory and time and achieves better long-term sequences efficiency.
50
Autor Correlatoo from Autotorma" Attention hacaiörer
AuttFCorfelation from Abtolanter
Full Artention From hansiomeer Prolispare
Memory/GB
ProbSparse Attention informer
Timelmsi
30
from
20
10
192
384
768
3072
512
1024
2048
4096
8192
Output Length
Output Length
(a) Memory Efficiency Analysis
(b) Running Time Efficiency Analysis
Figure 7: Efficiency Analysis. For memory, we replace Auto-Correlation with self-attention family in Autoformer and record the memory with input 96. For running time, we run the Outo-Correlation or self-attentions 10 times to get the execution time per step. The output length increases exponentially.
Conclusions
This paper studies the long-term forecasting problem of time series, which is pressing demand for real-world applications. However, the intricate temporal patterns prevent the model from learning reliable dependencies. We propose the Autoformer as a decomposition architecture by embedding the series decomposition block as inner operator, which can progressively aggregate the long- term trend part from intermediate prediction. Besides, we design an efficient auto-Correlation mechanism to conduct dependencies discovery and information aggregation at the series level, which contrasts clearly from the previous self-attention family. Autoformer can naturally achieve O log I.) complexity and yield consistent state-of-the performance in extensive real-world datasets.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grants 62022050 and 62021002, Beijing Nova Program under Grant Z201 100006820041, China's Ministry of Industry and Information Technology, the MOE Innovation Plan and the BNRist Innovation Fund.
References
O. Anderson and M. Kendall. Time-series, 2nd edn. J. R. Stat. Soc. (Series D), 1976.
[2] Reza Asadi and Amelia C Regan. A spatio-temporal decomposition based deep neural network for time series forecasting. Appl. Soft Comput.. 2020.
[3] Shaojic Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint TrXiv:/803..122 2018.
[4] Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Conditional time series forecasting with convolutional neural networks. arXiv preprint arXiv: 17Xiv:700.0066900 2017.
[5] G. F. Box and Gwilym M. Jenkins. Time series analysis, forecasting and control. 1970.
[6] George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. J. R. Stat. Soc. (Series-C) C), 1968.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernr,, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models few-shot learners. In NeurlPS, 2020.
[8] Chris Chatfield. The analysis of time series: an introduction. 1981.
Renyi Chen and Molei Tao. Data-driven prediction of general hamiltonian dynamics via learning exactly- symplectic maps. ICMI,, 2021.
[10] Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. In NeurlPS, 2020.
[11] J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT, 2019.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[13] S. Hochreiter and Schmidhuber. Long short-term memory. Neural Comput.. 1997.
[14] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In ICLR, 2019.
[15] Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. 2018.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015.
[17] Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya. Reformer: The efficient transformer. ICLR, 2020.
[18] Richard Kurle, Syama Sundar Rangapuram, Emmanuel de Bézenac, Stephan Günnemann, and Jan Gasthaus. Deep rao-blackwellised particle filters for time series forecasting. In NewrlPS, 2020.
[19] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, 2018
[20] Shiyang Li. Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, In NewrIPS, 2019.

---

[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[22]
Danielle C Maddix, Yuyang Wang, and Alex Smola. Deep factors with gaussian processes for forecasting. arXiv preprint arXiv:1812.00090 2018.
[23] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis
expansion analysis for interpretable time series forecasting. ICLR, 2019.
[24]
Athanasios Papoulis and H Saunders. Probability, random variables and stochastic processes. 1989.
[25] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, I.. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- perfoomaefreee deep learning library. In NeurlPS, 2019.
[26] Syama Sundar Rangapuram, Matthias W Seeger. Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models time series forecasting. In NeurlPS, 2018.
[27] Cleveland Robert, C William, and Terpenning Irma. STL:: seasonal-trend decomposition procedure based on loess. J. Off. Stat, 1990.
[28] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. Int. J. Forecast., 2020.
[29] Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, locally: A deep neural network approach high- high-dimensional time series forecasting. NewrlPS, 2019.
[30] Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. Temporal pattern attention for multivariate time series forecasting. Mach. Learn., 2019.
[31] Huan Song, Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias, Attend and diagnose: Clinical time series analysis using attention models. In AAAI, 2018.
[32] Antti Sorjamaa, Jin Hao, Nima Reyhani, Yongnan Ji, and Amaury Lendasse. Methodology for long-term prediction time series. Neurocomputing, 2007.
[33] Sean Taylor and Benjamin Letham. Forecasting at scale. Am. Stat., 2018.
[34] Aäron van den Oord, S. Dieleman, H. Zen, K. Simonyan, Oriol Vinyals, A. Graves, Nal Kalchbrenner,
A. Senior, and Kavukcuoglu. Wavenet: generative model for raw audio. In SSW, 2016.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrIPS, 2017.
[36] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horiza quantile recurrent forecaster. NeurlPS, 2017.
[37] Norbert Wiener. Generalized harmonic analysis. Acta Math, 1930.
[38] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. In NewrlPS, 2020.
[39] Q. Yao, D. Song, H. Chen, C. Wei, and G. W. Cottrell. A dual-stage attention- recurrent neural network for time series prediction. In IJCAI, 2017.
[40] Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor-train rans. arXiv preprint 17Xiv:1111.00073. 2017.
[41] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI, 2021.
12


Reviews:
[{"review_id": "zy65OgIxJNe", "reviewer": "Reviewer_jDpZ", "summary": "The paper proposes the two following novel architecture changes to improve transformer for long-term time-series forecasting\n1) Time-series decomposition block, where trend and seasonality are progressively decomposed.\n2) Auto-correlation attention mechanism, where attentions are computed in the frequency domain and aggregated by time delay aggregation, instead of computing attentions in the time domain and point-wise aggregated as in normal transformer architecture.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper says that \"*long-term forecasting* setting is to predict the long-term future given the short-term history\".  To the best of my knowledge, this is not correct. \"Long-term forecasting\" depends solely on how far ahead we make the forecast, and should be independent of how much data we use to make the forecast. For example, in electricity forecasting, long-term forecasting usually means one month or up to a year ahead forecast. The paper uses the incorrect definition of *long-term forecasting* to justify its input data limit. All the experiments use only 96 data points as input, in order to claim that the proposed methods can make good forecasts where the length of the predicted series is severalfold of the length of the input series. I believe the setting makes most of the tasks unrealistic. For example, the Electricity dataset contains two-year hourly data (17520 data points), and the paper uses 4 days of data (96 data points) to forecast 2, 8, 14, 30 days ahead. This is a very unrealistic setting! Electricity consumption usually includes strong daily, weekly, and yearly patterns, with a long-term trend. Using 4 days of data input means that we are comparing models on their ability to capture the daily pattern only. A similar argument applies to other datasets (ETT, Traffic, Weather, Exchange) as well. It's even worse in the case of ETT dataset since it has 15 minutes resolution, so 96 data points represent a single day of data, while the whole dataset contains 2 years of data.\n\nLooking at the series decomposition block, I believe that it can only effectively capture at most one seasonality because it only progressively produces one seasonal series. That could explain why the authors choose to limit the input data to 96 data points. Table 3 shows some evidence in favor of my hypothesis. **The MSE consistently increases when the input length increases**, regardless of whether auto-correlation or self-attention is used. This is very unexpected and shows that the model fails to work when the input data is long enough, maybe due to multiple seasonalities and trends.\n\nIt's also well-known that the correlogram of time-series with multiple seasonalities is very noisy due to interfering between seasonalities and patterns. So, I believe that the Auto-Correlation mechanism will not work for long time series with complex seasonalities, since picking top k lags that have the highest auto-correlation value would not include the periodicities we want. For example, most real-life time series will have recent lags (e.g., lag-1, lag-2) with the highest auto-correlation. It would be very helpful if the authors include the list of lags that the model pays attention to as a sanity check. To be clear, Figure 3.d. is just a demonstration of what the authors expect the auto-correlation mechanism to work, not how it actually works.\n\nAnother big concern of mine is that in the forecasting tasks, it's pretty simple to pick a window so that a particular model outperforms others. So it's highly recommended to do forecasting in a sliding window fashion so that the model gets tested at different time windows (windows could be overlapped or not, and the model could be retrained or not after each sliding). All experiments in the paper are conducted for a single testing window per task, which makes it really hard to conclude anything. Especially when the paper does not say anything about how they choose the testing window.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see the limitations or potential negative societal impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "DEIVpnwPpZU", "reviewer": "Reviewer_zbiv", "summary": "This paper introduces a new approach for long-term time-series forecasting that is based on decomposition and auto-correlation. Essentially, the authors define a series decomoposition block which extracts the mean (trend-cyclical) and subtracts it from the series, obtaining the seasonal part. In addition, the authors design an auto-correlation component that identifies local structures across periods. The method is evaluated on six benchmark datasets and in comparison to several baseline approaches. The presented results show improvements over existing work.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors of this submission propose to modify the transformer module to account for its limiting time/space computational complexity. To this end, they propose two components---decomposition and autocorrelation, which work together to yield improved long-term forecasts. To the best of my knowledge, this idea is novel. The exposition of the paper is mostly clear, however, Subsec. 3.2 should be explained better as it is not entirely clear how it operates and implemented. Similarly, the descriptions in Subsec 3.1 could be improved, but they are somewhat better than 3.2. There are some undefined operations such as AvgPool, and it was not clear whether AutoCorrelation takes one argument or two.\n\nOverall, the proposed method is interesting, and the reported results are impressive. However, I would have liked to see stronger ties to the theory and practice of decompositions and auto-correlations in this work. Unfortunately, the current version facilitates these components purely on the usage level, and it is not entirely clear why the results are better over other strong baselines. For instance, what happens if there are minimal to no local recurrences of self-structures? Would the autocorrelation help at all? What happens when ones moves to a higher-dimensional setting where signals can be rotated/stretched---would the autocorrelation component be able to account for that?\n\nThe limitations of the technique were not clearly described. For instance, does the model depend on the length of the series (L) as it presumably takes average across this dimension? How one decided if to use additive or multiplicative decompositions?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9jEC88KacpK", "reviewer": "Reviewer_GKTE", "summary": "This paper focuses on the long-term forecasting problem of time series. This work introduces the traditional idea of decomposing time series into seasonality and trend-cycle components into deep Transformer architecture. This work also proposes an auto-correlation mechanism that replaces the original self-attention in Transformers.  Further, they improve the computation efficiency by computing power spectral density. Experiments on several datasets show that the proposed method achieves better performance compared to SOTA methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Novelty and Significance\n- The problem addressed is very important in many domains. Both the contributions regarding the introduction of series decomposition in the deep learning model and the auto-correlation mechanism are novel. \n\n### Quality\n- The auto-correlation mechanism achieves better performance than the self-attention mechanism and its several variants in terms of both speed and accuracy. While the self-attention mechanism is fairly explanatory,  It is not clear what the weighted sum of delayed sequences in equation 6 represents. The authors should provide intuitive explanations why auto-correlation works better than self-attention in the case of time series.\n- The autocorrelation of a univariate time series tells us about the similarity between the original time series and its delayed version. But in the case of multivariate time series, different dimensions can have different time periods. How do the authors compute the autocorrelation in that case? Is it computed individually per dimension?\n- The encoder inputs as noted in line 127 is of length I and hence the output of the encoder should also be of length I. How do authors resize the encoder output to length O in line 179? What is the rationality behind using only the second half of the seasonality and trend-cycle component as an input to the decoder?\n- How do authors perform autocorrelation mechanism on the key and query sequences of variable length? This is the case when the output of both encoder and decoder is fed to the second auto-correlation block in decoder?\n- Since the experiments have been repeated three times, the authors should also report the standard deviation in the MAE/MSE scores.\n\n### Clarity\n- The paper is well written and easy to follow.\n- line 83 - have shown great power in modeling\n- line 146 decoder - decoding", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-kV4efUnAA", "reviewer": "Reviewer_zw1M", "summary": "This paper presents Autoformer to perform long-term time series forecasting. The key idea is to leverage an auto-correlation mechanism to discover the sub-series similarity based on the series periodicity and aggregate similar sub-series from underlying periods. The experiment results on several datasets showed the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* This paper is well written and organized.\n* Long term time series forecasting is an interesting problem to investigate.\n* The proposed auto-correlation mechanism is innovative and technically sound.\n\nWeaknesses:\n* The assumption of Autoformer is not clear.\n* The setup of long term forecasting needs to be justified.\n* Several related works are not mentioned or compared.\n\nOverall, although the idea of calculating the autocorrelation over frequency domain is not new (e.g., Vlachos et al., SDM 2005), the way to combine it with a multi-head attention mechanism indeed exhibits novelty to reduce the computational cost. The following are several concerns over this work.\n\nIt is not clear under what assumption/condition Autoformer can work well. The authors mentioned that the auto-correlation mechanism is developed based on series periodicity. For time series which do not exhibit clear periodicity, whether it still works or not? For example, for the exchange dataset, whether the long term prediction here is really meaningful or useful?\n\nIn the experiment, the input sequence length is 24/96, while the prediction lengths are 92, 192, … 720. I am wondering whether the input sequence can provide sufficient info to give a legitimate prediction. For electricity, traffic, and weather, this could be true, but I am not sure this is the case for exchange or other time series. Is there a theoretical justification that under what condition we can obtain a  legitimate prediction?\n\nARMA usually works well to capture the seasonal info but is not compared in the experiment. Furthermore, several related works are not compared\n\n[1] Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting, NIPS 2019\n[2] N-BEATS: NEURAL BASIS EXPANSION ANALYSIS FOR INTERPRETABLE TIME SERIES FORECASTING, ICLR 2020\n\nWhether Autoformer works for multivariate time series?\n\nHow to determine tau in practice?\n\nI could be persuaded if the above questions are clarified. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the problem of long-term forecasting for time series models. It proposes a transformer-based architecture with an attention mechanism based on auto-correlation and combines it with a periodicity-based time series decomposition approach. The problem studied is relevant and important. The proposed architecture is novel although the basic concepts it builds on are well known in time series analysis. The proposed approach is technically sound and the empirical evaluation of the approach provides adequate evidence that it has the potential to provide improved performance on the long term time series forecasting problem relative to existing approaches. Following the author response and discussion, most initial reviewer concerns were addressed and the consensus is that the paper should be accepted. One important point for the authors to clarify are the limitations of the approach with respect to time series with different structural properties (e.g, complex seasonality, quasiperiodicity, absence of periodicity, etc.).
Decision: Accept (Poster)