Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
Authors: Haixu Wu
Jiehui Xu
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2021 Poster (2021)
Keywords: Time Series Forecasting
Transformers
Deep Decomposition Model
Auto-Correlation

Abstract: Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: https://github.com/thuml/Autoformer.

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Ters Series Forecasting
Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long (Ex)
School of Software, BNRist, Tsinghua University, China
(whx20, (axx20,xjh200 xjh20] @mails. wwwnthusnnhahaeee edw..aaaeeedd.eee cn, (jimwang, (jimsang.mingshengee.. imingghheng))otsinnnua.. edu. en
Abstract
Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer- based models adopt various self- self-attention mechanisms discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going heyond Transformers, we design Auto- former as novel decomposition architecture with an Auto- Coto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto- cott-Coorelation mechanism based on the series periodicity, which conducts the dependencies discovery and representa- tion aggregation at the sub-series level. Auto-Correlatin outperforms self- attention in both efficiency and accuracy. long-term forecasting, Autoformer yields state- of- oh-thh-art accuracy, with 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economies, weather and disease. Code available at this repository: https::// htop::://itthui thub..bbooo www.toooooiiiooeeoe http/thuml//// Autof ormer
Introduction
Time series forecasting has been widely used in energy consumption, traffic and economics planning, weather and disease propagation forecasting. In these real-world applications, one pressing demand extend the forecast time into the far future, which is quite meaningful for the long-term planning and early warning. Thus, this paper, we study the long-term long-term,foreeasting problem of time series, characterizing itself by the large length of predicted time series. Recent deep forecasting models [9] 35] have achieved great progress, especially the Transformer-base models. Benefiting from the self-attention mechanism, Transformers obtain great advantage in modeling long-term dependencies for sequential data, which enables more powerful big models [T]
41 20 28 23
However, the forecasting task is extremely challenging under the long-term setting. First, it is unreliable to discover the temporal dependencies directly from the long-term time series because the dependencies can be obscured by entangled temporal patterns. Second, canonical Transformers with self-attention mechanisms are computationally prohibitive for long-term forecasting because of the quadratic complexity of sequence length. Previous Transformer-base forecasting models sparse version. While performance significantly improved, these models still utilize the point-wise representation aggregation. Thus, in the process of efficiency improvement, they will sacrifice the information utilization because of the sparse wise connections, resulting in a bottleneck for long-term forecasting of time series.
41 17 20 mainly focus on improving self-attention
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To reason about the intricate temporal patterns, we try to take the idea of decompasition, which is standard method in time series analysis [27 It can be used to process the complex time series and extract more predictable components. However, under the forecasting context, it can only be used as the pre-processing of past series because the future is unknown [15]. This common usage limits the capabilitiiss of decomposition and overlooks the potential future interactions among decomposed components. Thus, we attempt to go beyond pre-processing usage of decomposition and propose a generic architecture to empower the deep forecasting models with immanent capacity of progressive decomposition. Further, decomposition can ravel out the entangled temporal patterns and highlight the inherent properties of time series [15.. Benefiting from this, try take advantage of the series periodicity to renovate the point-wise connection self- selffattention. We observe that the sub-series at the same phase position among periods often present similar temporal processes. Thus, try to construct a series-level connection based on the process similarity derived by series periodicity.
Based on the above motivations, we propose an original Autoformer in place of the Transformers for long-term time series forecasting. Autoformer still follows residual and encoder-decoder structure but renovates Transformer into decomposition forecasting architecture. By embedding decomposition blocks as the inner operators, Autoformer can progressively separate the long-term trend information from predicted hidden variables. This design allows our model to alternately decompose and refine the intermediate results during the forecasting procedure. Inspired by the stochastic process theory [8 24], Autoformer introduces an of celf-attention, which discovers the sub-series similarity based on the series periodicity and aggre- gates similar sub-series from underlying periods. This series-wing mechanism achieves OLL log L) complexity for length- I, series and breaks the information utilization bottleneck by expanding the point-wise representation aggregation to sub-series level. Autoformer achieves the ofte-oofthhe-art accuracy on six benchmarks. The contributions are summarized as follows:
proposed
mechanism in place
To tackle the intricate temporal patterns of the long-term future, we present Autoformer as decomposition architecture and design the inner decomposition block to empower the deep forecasting model with immanent progressive decomposition capacity.
We propose Corte-CCrreltionn mechanism with dependencies discovery and information aggregation at the series level. Our mechanism is beyond previous self-attention family and can simultaneously benefit the computation efficiency and information utilization.
Autoformer achieves a 38% relative improvement under the long-term setting on six bench- marks, covering five real-world applications: energy, traffic, economics, weather and disease.
2 Related Work
2.1 Models for Time Series Forecasting
Due to the immense importance of time series forecasting, various models have been well developed. Many time series forecasting methods start from the classic tools [32] 9]. ARIMA [6.5] tackles the forecasting problem by transforming the non-stationary process stationary through differencing.
The filtering method is also introduced for series forecasting [18] [0]. Besides, recurrent neural
networks (RNNs) models are used model the temporal dependencies for time series 36 26
DeepAR (8 combines autoregressive methods and RNNs to model the probabilistic distribution of future series. LSTNet [19 introduces convolutional neural networks (CNNs) with recurrent-skip skip connections to capture the short-term and long-term temporal patterns. Attention-based RNNs introduce the temporal attention to explore the long-range dependencies for prediction. Also, many works based on temporal convolution networks (TCN) 34
39 30 31
3 29] attempt to model the temporal causality with the causal convolution. These deep forecasting models mainly focus on the temporal relation modeling by recurrent connections, temporal attention causal convolution.
4
Recently, Transformers 35 38 based on the self-attention mechanism shows great power in sequen- tial data, such as natural language processing [1] 7 audio processing [4] and even computer vision 12 21]. However, applying attt-tttention to long-term time series forecasting is computationally prohibitive because of the quadratic complexity of sequence length L in both memory and time.
LogTrans (20 introduces the local convolution to Transformer and proposes the LogSparse attention to select time steps following the exponentially increasing intervals, which reduces the complexity to (L.(lo I.)2). Reformer [17 presents the local-sensitive hashing (LSH)) attention and reduces the complexity to OLL log L). Informer [4] extends Transformer with KL-divergence based ProbSparse

---

attention and also achieves lo L) complexity. Note that these methods are based on the vanilla Transformer and try to improve the self-attention mechanism a sparse version, which still follows the point-wise dependency and aggregation. In this paper, our proposed Auto- Cuto-Correlation mechanism based on the inherent periodicity of time series and can provide series-wise connections.
2.2 Decomposition of Time Series
As standard method time series analysis, time series decomposition 27 deconstructs a time series into several components, each representing one of the underlying categories of patterns that more predictable. It is primarily useful for exploring historical changes over time. For the forecasting tasks, decomposition is always used as the pre-processing of historical series before predicting future series [15] 2] such as Prophet [33] with trend-seasonality decomposition and N-BEATS 23 with
basis expansion and DeepGLO [29] with matrix decomposition. However, such pre-processing is limited by the plain decomposition effect of historical series and overlooks the hierarchical interaction between the underlying patterns of series in the long-term future. This paper takes the decomposition idea from new progressive dimension. Our Autoformer harnesses the decomposition as an inner block of deep models, which can progressively decompose the hidden series throughout the whole forecasting process, including both the past series and the predicted intermediate results.
3
Autoformer
The time series forecasting problem is predict the most probable length-O series in the future given the past length-/ series, denoting as input-I-predict-O The long-term forecasting setting is to predict the long-term future, i.e. larger O. As aforementioned, of long-term series forecasting: handling intricate temporal patterns and breaking the bottleneck of computation efficiency and information utilization. To tackle these two challenges, we introduce the decomposition as a builtin block to the deep forecasting model and propose Autoformer as decomposition architecture. Besides, we design the Auto-Correlation mechanism to discover the period-based dependencies and aggregate similar sub-series from underlying periods.
have highlighted the difficulties
3.1 Decomposition Architecture
We renovate Transformer [35 to deep decomposition architecture (Figure I including the inner mechanism, and corresponding Encoder and Decoder.
series decomposition block,
Series decomposition block To learn with the complex temporal patterns in long-term forecasting context, we take the idea of decomposition [7] which can separate the series into trend-cyclicn and seasonal parts. These two parts reflect the long-term progression and the seasonality of the series respectively. However, directly decomposing is unrealizable for future series because the future is just unknown. To tackle this dilemma, we present a series decomposition block as an inner operation of Autoformer (Figure D. which can extract the long-term stationary trend from predicted intermediate hidden variables progressively. Concretely, we adapt the moving average to smooth out periodic fluctuations and highlight the long-tern trends. For length-L input series X R4xd, the process is:
X A Pedddinnaa X - X.,
where X,, Xr E TR 4xd denote the seasonal and the extracted trend-cyclical part respectively. We adopt the A AvgPool(.) for moving average with the padding operation to keep the series length unchanged. We use X2 X SeriesDecomp((X) to summarize above equations, which is model inner block.
Model inputs The inputs of encoder part are the past I time steps Xen f As decomposition architecture (Figure []. the input of Autoformer decoder contains both the seasonal part X'aes e R +))x and trend-cyclical part Naet E xO)x+ to be refined. Each initialization consists of two parts: the component decomposed from the latter half of encoder's input Xen with length & to provide recent information, placeholders with length O filled by scalars. It's formulized as follows:
Xensy Vent = PerrisDeccop(Keet 4.1)
Xass Concat (Xens> Xo)
(2)
Xaet = Concat (Kent: XMean),
3

---

Autoformer Encoder
Time
Encoder Imput
Seasonal
Auto- Correlation
Series Decomp
Series Decomp
To Prodict
Forward
Irend -tyclical
Duta Mean
Part
Seasonal Init
Auto- Correlation
Series Decomp
Auto- Correlation
Series Decomp
Feed Forward
Series Decomp
Zero
Prodiction
Trend-cyclical Init
Input Data Mear
Autoformer Decoder
M
Figure I: Autoformer architecture. The encoder eliminates the long-term trend-cyclical part by series decomposition blocks (blue blocks) and focuses on seasonal patterns modeling. The decoder accu- mulates the trend part extracted from hidden variables progressively. The past seasonal information from encoder utilized by the encoder-decoder Auto-Correlation (center green block decoder).
where Nens, Nent R dd denote the seasonal and trend-cyclical parts of Xen respectively, and No, MMean E R0xd denote the placeholders filled with zero and the mean of Xen respectively.
Encoder As shown in Figure [] the encoder focuses on the seasonal part modeling. The output of the encoder contains the past seasonal information and will be used as the cross information to help the decoder refine prediction results. Suppose we have N encoder layers. The overall equations for 1-th encoder layer are summarized as X = Encoder(A"t. (/==1). Details are shown as follows:
st,1
ScriesDecomp AAuto-Corrcaation((1111 xx-11
S/.2 SeriesDecomp [eedFForward(S".) S
where """is the eliminated trend part. X S1.2 {I,... JJ denotes the output of I-th encoder layer and X is the embedded X-n- St, [1,2) represents the seasonal component after the i-th series decomposition block in the 1-th layer respectively. We will give detailed description of (uto-Correlation() in the next section, which seamlessly replace the self-attention.
Decoder The decoder contains two parts: the accumulation structure for trend-cyclical components and the stacked Luto-Correlation mechanism for seasonal components (Figure [T. Each decoder layer contains the inner auto-Correlation and encoder-decoder MnttoCorrelltion, which can refine the prediction and utilize the past seasonal information respectively. Note that the model extracts the potential trend from the intermediate hidden variables during the decoder, allowing Autoformer progressively refine the trend prediction and eliminate interference information for period- period-based based dependencies discovery in Cito-Correllaton. Suppose there are M decoder layers. With the latent variable X-N from the encoder, the equations of 1-th decoder layer can be summarized X% Decoder( de
X.N). The decoder can formalized as follows:
SeriesDecomp AAtto-Coreeltionn(X2 x/111)
2!-1)
de de
1,2 de
SeriesDecomp Autto-Correlation(()) l1.1 (NN) + cl.1
de
(4)
s,3 S T1.3 1.
SeriesDecomp FeedForward(()1))
ci.2 +
Tt = Tél + Wi,1 Tdil + W1, T + Wia 1133
where x = S/ I E (1, ... M) denotes the output of 1-th decoder layer. 2: is embedded from Xaes for deep transform and Th = Xtet is for accumulation. S T.. 2,) represent the seasonal component and trend-cyclical component after the i-th series decomposition block in the 1-th layer respectively. Wi,, e (,, represents the projector for the i-th extracted trend T

---

Linear
Concat
Time Delay
ExC
Time Delay Agi
SoftMax
Inverse FFT
Rolli TT)
R(u
Conjugane
LeCs2
Roll(
Rimal
FFT
FFT
leC
Resize
EVC
SrC
SxC
Rolli
Limear
Rlmu
Figure 2: Auto-Corres CoroCCorrllation (left) and Time Delay Aggregation (right). We utilize the Fast Fourier Transform to calculate the autocorrelation R(T), which reflects the time-delay similarities. Then similar sub processes rolled to the same index based selected delay and aggregated by R(T).
The final prediction is the sum of the two refined decomposed components, as Ws X Tal where Ws is to project the deep transformed seasonal component X to the target dimension.
3.2
Auto-Correlation Mechanism
As shown in Figure 2 we propose the Auto-Correlation mechanism with series-wise connections to expand the information utilization. auto-Correlation discovers the period- dependencies by calculating the series autocorrelation and aggregates similar sub-series by time delay aggregation.
Period-based dependencies provides similar sub- crob-rocesses. Inspired by the stochastic process theory [8] 224. for real discrete- time process } we can obtain the autocorrelation Rxx (T) by the following equations:
observed that the same phase position among periods naturally
Rxx(T)
lim L T.
Σ N.K.
(5)
Rxx(1 (T) reflects the time-delay similarity between (At) and its lag series (2t-))- As shown in Figure 2 we use the autocorrelation R(T) as the unnormalized confidence of estimated period length T. Then, we choose the most possible &t period lengths TI. , Th. The period-based dependencies are derived by the above estimated periods and can be weighted by the corresponding autocorrelation.
Time delay aggregation The period-based dependencies connect the sub-series among estimated periods. Thus, we present the time delay aggregation block (Figure which can roll the series based on selected time delay Ti, -k- This operation can align similar sub-series that are at the same phase position of estimated periods, which is different from the point-wise dot-product aggregation self-attention family. Finally, we aggregate the sub-series by softmax normalized confidences.
For the single head situation and time series X' with length-L, L,, after the projector, we get query Q, key KC and value Thus, it can replace self-attention seamlessly. The Auto Cotr-Coorelttion mechanism is:
TI The Th arg Topk (Rox(T))
re(1..-.44
Rex(n), ... Rex(T)) SoftMax (Rex(1)), RQx((T))
(6)
&
Auto-CoorelationnQ..K.
Σ R,
where arg Topk(-) is to get the arguments of the Topk autocorrelations and let k = |Ic x log LJ, a hyper-parammter. Rg,x x is autocorrelation between series Q and K. Roll represents operation to A' with time delay т, during which elements that are shifted beyond the first position are re-introduced the last position. For the encoder-decoder Auto-Correlation (Figure K., are from the encoder N and will be resized length-O, Q is from the previous block of the decoder.
5

---

(a) Full Atlention
Tine
(b) Sparse Allention
Time
kb
x Penood
Pertod Pertod
LogSparse Attention
Time
(d) Auto-Correlation
Time
Figure natt-Correlation vs. self-attention family. Full Attention 335 adapts the fully connection
among all time points. Sparse Attention [17,4] (b) selects points based on the proposed similarity metrics. LogSparse Attention (0 (c) chooses points following the exponentially increasing intervals.
Auto-Correlation (d) focuses on the connections of sub-series among underlying periods.
For the multi-heat version used in Autoformer, with hidden variables of dmodel channels, h heads, the query, key and value for i-th head are Q,, K.,, V; E R Smnde , E fl, ,... The process is: MultiHead K, V) = Woutput Concat (headd ... heads)
where head, Auto-Corres tatto-Coreelaton(( Q,, K,, V.).
Efficient computation For period-based dependencies, these dependencies point to sub-processes the same phase position of underlying periods and are inherently sparse. Here, we select the most possible delays to avoid picking the opposite phases. Because we aggregate Olog L) series whose length is L, the complexity of Equations 7 is log L). For the autocorrelation computation (Equation 5), given time series (X,). (xx(T) can be calculated by Fast Fourier Transforms (FFT)
based on the Khinchn theorem 37
I Nie 12 12
dt / K2et2 dt
Sxx(f) = F(L) F (2r)
(8)
(xx =F F (Fxx(( 1 Sxx (f)
where r E (1, ,L). F denotes the FFT and F-1 is its inverse. denotes the conjugate operation and Sxx(f) in the frequency domain. Note that the series autocorrelation of all lags in (1..... L} can be calculated at once by FFT. Thus, Auto-Correlation achieves the (L log L) complexity.
Auto-Correlation vs. self-attention family Different from the woint-wise self-attention family, Auto-Correlation presents the scries-wise connections (Figure 3 Concretely, for the temporal dependencies, we find the dependencies among sub-series based the periodicity. In contrast, the self-attention family only calculates the relation between scattered points. Though some self- attentions 20 41 consider the local information, they only utilize this help point-wise depen- dencies discovery. For the information aggregation, we adopt the time delay block to aggregate the similar sub-series from underlying periods. In contrast, self-attentions aggregate the selected points by dot-product Benefiting from the inherent sparsity and sub-series-level representation aggregation, Auto-Correlation simultaneously benefit the computation efficiency and information utilization.
4 Experiments
We extensively evaluate the proposed Autoformer on six real-world benchmarks, covering five mainstream time series forecasting applications: energy, traffic, economies, weather and disease.
Datasets Here is a description of the six experiment datasets: (1) ETT dataset contains the data collected from electricity transformers, including load and oil temperature that are recorded every

---

Table 1: Multivariate results with different prediction lengths O E {96, 192, 336, 720]. We set the input length / as 36 for ILI and 96 for the others. A lower MSE or MAE indicates a better prediction.
Models
Autoformer
Informer
LogTrans 20
Reformer 17
LSTNet 19
LS
TCN B
Metric
MSE MAE MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.255 0.339 0.365 0.453 0,768
0.642
0.658
0.619
3.142
1.365 2.041
1.073 3,041
1.330
ETT
192
0.281
0.340 0.533 0.563 0.989
0.757
1.078
0.827
3.154
1.369
2.249
1.112
3,072
1.339
336
0.339
0.372 1.363
0.887
1.334
0.872
1.549
0.972
3.160
1.369
2.568
1.238
3.105
1.348
720
0.422
0.419
3.379
1.388
3.048
1.328
2.631
1.242
3.171
1.368
2.720
1.287
3.135
1.354
Electricity
96
0.201
0.317
0.274
0.368
0.258
0.357
0.312
0.402
0.680
0,645
0.375
0.437
0.985
0.813
192
0.222
0.334 0.296 0.386 0.266
0.368
0.348
0,433
0.725
0.676 0.442 0.473
0.996 0.821
336
0.231
0.338
0,300
0.394
0,280
0.380
0.350
0.433
0.828
0.727 0.439
0.473
1,000
0.824
720
0.254
0.361
0.373
0.439
0.283
0.376
0.340
0.420
0.957
0,811
0,980
0.814
1.438
0.784
Exchange
96
0.197
0.323
0.847
0.752
0,968
0.812
1.065
0.829
1.551
1.058
1.453
1.049
3.004
1,432
192
0.300
0.369
1.204
0.895
1.040
0.851
1.188
0.906
1.477
1.028
1.846
1.179
3.048
1,444
336
0.509
0.524
1.672
1.036
1.659
1.081
1.357
0.976
1.507
1.031
2.136
1.231
3.113
1,459
720
1.447
0.941
2.478
1,310
1,941
1.127
1.510
1.016
2.285
1.243
2.984
1.427
3.150
1.458
96
0.613
0.388
0.719
0.391
0.684
0.384
0.732
0.423
1.107
0.685
0.843
0.453
1,438
0.784
Traffic
192
0.616
0.382 0.696
0.379
0.685
0.390
0.733
0.420
1.157
0.706
0.847 0.453
1.463 0.794
336
0.622
0.337 0.777
0.420
0.733
0.408
0.742
0.420
1.216
0.730
0.853
0.455
1,479 0.799
720
0.660
0.408 0.864
0.472
0.717
0.396
0.755
0.423
1,481
0.805
1,500
0.805
1.499
0.804
Weather
96
0.266
0.336 0.300
0.384
0,458
0.490
0.689
0.596
0.594
0.587
0.369
0,406
0,615 0.589
192
0.307
0.367 0.598 0.544
0.658
0.589
0.752
0.638
0.560
0.565 0.416 0.435 0.629 0.600
336
0.359
0.395
0.578
0.523
0.797
0.652
0.639
0.596
0.597
0.587
0.455 0.454
0.639 0.608
720
0.419
0.428
1.059
0.741
0.869
0.675
1.130
0.792
0.618
0.599
0.535
0.520
0.639 0.610
24
3.483
1.287
5.764
1.677
4,480
1.444
4.400
1,382
6.026
1.770
5.914
1.734
6.624
1.830
36
3.103
1.148
4.755
1.467
4,799
1.467
4.783
1.448
5,340
1.668
6.631 1.845
6,858
1.879
48
2.669
1.085
4.763
1,469
4,800
1.468
4.832
1.465
6.080
1.787
6.736
1.857
6.968
1.892
60
2.770
1.125
5.264
1.564
5.278
1.560
4.882
1.483
5.548
1.720
6.870
1.879 7.127 1.918
ETT means the ETTm2. See supplementary materials for the full benchmark of ETTh1, ETTh2, ETTml.
15 minutes between July 2016 and July 2018. (2) Electricity I dataset contains the hourly electricity consumption of 321 customers from 2012 to 2014. (3) Exchange [19 records the daily exchange rates of eight different countries ranging from 1990 to 2016. (4) Traffic is a collection of hourly data from California Department of Transportation, which describes the road occupancy rates measured by different sensors on San Francisco Bay area freeways. (5) Weather iss recorded every 10 minutes for 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity, etc. (6) H1." includes the weekly recorded influenza-like illness (II.I) patients data from Centers for Disease Control and Prevention of the United States between 2002 and 2021, which describes the ratio of patients seen with IIII and the total number of the patients. We follow standard protocol and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets.
Implementation details Our method is trained with 1.2 loss, using the ADAM (16] optimizer with an initial learning rate of 10-4, Batch size is set to 32. The training process is early stopped within 10 epochs. All experiments are repeated three times, implemented in PyTorch (25) and conducted on a single NVIDIA TITAN RTX 24GB GPUs. The hyper-parameter c of Auto-Correlation is in the range of 1 to 3 to trade off performance and efficiency. See supplementary materials for standard deviations and sensitivity analysis. Autoformer contains 2 encoder layers and I decoder layer.
Baselines We include 10 baseline methods. For the multivariate setting, we select three latest state-
oh-the-art transformer-batt models: Informer [41), Reformer 117, LogTraans (20), two RNN-based models: LSTNet [19), LSTM [3] and CNN-based TCN B as baselines. For the univariate setting, we include more competitive baselines: N-BEATS DeepAR (88) Prophet (33] and ARMIA [1]
http: httt:://pems.dot.com http://wwwa...aaaaa.aaa/aacaaaaa
ics.
7

---

Table 2: Univariate results with different prediction lengths O E {96, 192, 336, 720} on typical datasets. We set the input length I as 96. A lower MSE or MAE indicates a better prediction.
Models Autoformer N-BEATS 23 Informer 41
LogTrans 20
Reformer
[17
Prophet 22
[3 [
Metric
MSE
MAE MSE
MAE
MSE
MAE MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.065 0.189 0.082
0.219
0.088 0.225 0.082 0.217 0.131
0.288 0.099 0.237 0.287 0.456 0.211 0.362
192
0.118 0.256 0.120
0.268
0.132 0,283 0.133
0.284 0.186
0.354
0.154 0,310 0.312 0.483 0.261 0,406
336
0.154
0.305 0.226
0.370
0.180
0.336 0.201
0.361
0.220
0.381
0.277 0.428 0.331 0.474 0.317 0.448
720
0.182
0.335 0.188
0.338
0.300
0.435 0.268
0.407
0.267
0.430
0.332 0.468 0.534 0.593 0.366 0.487
96
00.241
0.387 0.156
0.299
0.591
0.615 0.279
0.441
1.327
0,944
0,417 0.515 0.828 0.762 0.112 0.245
Exchange
192
0.273
0.403 0.669
0.665
1.183 0.912 1,950
1.048
1.258
0.924
0,813 0.735 0.909 0.974 0,304 0.404
336
0.508
0.539 0.611
0.605
1.367
0.984 2.438
1.262
2.179
1.296
1.331 0.962 1.304 0.988 0.736 0,598
720
0.991
0.768 1.111 0.860
1.872
1.072
2.010
1.247
1,280
0,953
1.894
1.181 3,238 1.566 1.871 0.935
4.1 Main Results
To compare performances under different future horizons, we fix the input length and evaluate models with wide range of prediction lengths: 96, 192, 336, 720. This setting precisely meets the definition of long-term forecasting. Here are results on both the multivariate and univariate settings.
Multivariate results As for the multivariate setting, Autoformer achieves the consistent state-of- the performance in all benchmarks and all prediction length settings (Table Especially, under setting, compared to previous saut--of-the-eeee results, Autoformer gives 74 % (1.334-0..339) MSE reduction in ETT, 18% (0.280 +0.331) in Electricity. 61 % (1.357-0.509) in Exchange, 15% 11733~>0662 in Traffic and 21%
the-art
in Weather. For the input- 36-predict-60 setting of ILI, Autoformer makes 43% (4.882-2.770)) MSE reduction. Overall, Autoformer yields a 38% averaged MSE reduction among above settings. Note that Autoformer still provides remarkable improvements in the Exchange dataset that is without obvious periodicity. See supplementary materials for detailed showcases. Besides, we can also find that the performance of Autoformer changes quite steadily as the prediction length O increases. It means that Autoformer retains better long-term robustness, which is meaningful for real-world practical applications, such as weather early warning and long-term energy consumption planning.
Univariate results We list the univariate results of two typical datasets in Table 2 Under the comparison with extensive baselines, our Autoformer still achieves state-of-the-art performance for the long-term forecasting tasks. In particular, for the (oput-96-predictt 14% (0.180-=0.14; 145) MSE reduction the ETT dataset with obvious periodicity. For the Exchange dataset without obvious periodicity, Autoformer surpasses other baselines by 17% (0.61 101111000 1-0.508) and shows greater long-term forecasting capacity. Also, we find that ARIMA II performs best in the -npee-9-- setting of the Exchange dataset but fails in the long-term setting. This situation of ARIMA can be benefited from its inherent capacity for non-stationary economic data but is limited by the intricate temporal patterns of real-world series.
setting, our model achieves
4.2 Ablation studies
Table 3: Ablation of decomposition in multivariate ETT with MSE metric. Ours adopts our progres- sive architecture into other models. Sep employs two models to forecast pre-decomposed seasonal and trend-cyclical components separately. Promotion is the MSE reduction compared Origin.
Input-96|
Transformer 35
Informer 41
Lonnraan 17
Reformer 20
Promotion
Predict-O
Origin Sep Ours Origin Sep
Ours
Origin
Sep
Ours
Origin Sep
Ours
Sep
Ours
96
0.604 0.311 0.204 0.365 0.490 0.354 0.768 0.862 0.231 0.658 0.445 0.218 0.069 0.347
192
1.060 0.760 0.266 0.533 0.658 0.432 0.989 0.533 0.378 1.078
0.510 0.336 0.300 0.562
336
1,413
0.665 0.375 1.363 1.469 0.481 1.334
0.762 0.362
1.549
1.028
0.366 0.434 1.019
720
2.672 3.200 0.537 3.379 2.766 0,822 3.048
2.601 0.539
2.631 2.845 0.502 0.079 2.332
Decomposition architecture With our proposed progressive decomposition architecture, other
models can gain consistent promotion, especially as the prediction length O increases (Table This
8

---

verifies that our method can generalize to other models and release the capacity of other dependencies learning mechanisms, alleviate the distraction caused by intricate patterns. Besides, our architecture outperforms the pre- pre-procsssing although the latter employs bigger model and more parameters. Especially, pre-decomposing may even bring negative effect because neglects the interaction of components during long-term future, such as Transformer 35 predict-720, Informer 41 predict-336.
Auto-Correlation vs. self-attention family As shown in Table 4 our proposed Auto-Correlation achieves the best performance under various input-I-predict-O settings, which verifies the effective- ness of series-wise connections comparing to point-wise self-attentions (Figure B) Furthermore, we is memory efficiency from the last column of Table 4 which
can also observe that can be used long sequence forecasting, such as (oput-33-preedic-1-e 1440.
Table 4: Comparison of Auto-Correlation and self-attention in the multivariate ETT. We replace the Auto-Correlation in Autoformer with different self-attentions. The "_" indicates the out-of-memory.
Input Length 7
96
192
336
Prediction Length O
336
720
1440
336
720
1440
336
720
1440
Auto-
MSE
0.339
0.422
0.555
0.355
0.429
0.503
0.361
0.425
0.574
Correlation
MAE
0.372
0.419
0.496
0.392
0.430
0.484
0.406
0.440
0.534
Full
MSE 0.375
0,537
0.667
0.450
0.554
0.501
0.647
Attention
35
MAE
0.425
0,502
0,589
0.470
0.533
0.485
0,491
LogSparse
MSE
0.362
0.539
0.582
0.420
0.552
0.958
0.474
0.601
Attention 20
MAE
0.413
0.522
0.529
0.450
0.513
0.736
0.474
0.524
LSH
MSE
0.366
0,502
0.663
0.407
0.636
1.069
0.442
0.615
Attention 17
MAE
0.404
0.475
0.567
0.421
0.571
0.756
0.476
0.532
ProhSparse
MSE
0.481
0.822
0.715
0.404
1.148
0.732
0.417
0.631
1.133
Attention 41
MAE
0.472
0.559
0.586
0,425
0.654
0.602
0,434
0.528
0.691
4.3 Model Analysis
Time series decomposition As shown in Figure 4 without our series decomposition block, the forecasting model cannot capture the increasing trend and peaks of the seasonal part. By adding the series decomposition blocks, Autoformer can aggregate and refine the trend-cyclical part from series progressively. This design also facilitates the learning of the seasonal part, especially the peaks and troughs. This verifies the necessity of our proposed progressive decomposition architecture
Sosorall Hot
Solona Pin
Time Serie cyceey
Tra di eycicaa
100
200
300
400
900
600
700
100
200
300
400
500
500
700
100
200
100
400
3401
6C0
700
100)
250
300
400
300
503
7007
(a) Without decomposition block
(b) One decomposition block
(c) Two decomposition blocks
(d) Three decomposition blocks
Figure 4: Visualization of learned seasonal XM and trend-cyclical T.M of the last decoder layer. We gradually add the decomposition blocks in decoder from left to right. This case is from ETT dataset under (ooputt---rrdiic-775 setting. For clearness, we add the linear growth raw data additionally.
Dependencies learning The marked time delay sizes in Figure 5(a) indicate the most likely periods. Our learned periodicity can guide the model to aggregate the sub-series from the same or neighbor phase of periods by Roll(X' TT), i f1.... 6). For the last time step (declining stage), Auto- Correlation fully utilizes all similar sub-series without omissions or errors compared to self-attentions. This verifies that Autoformer can discover the relevant information more sufficiently and precisely.
Complex seasonality modeling As shown in Figure 5 the lags that Autoformer learns from deep representations can indicate the real scasonality of raw series. For example, the learned lags of the

---

Quean
Vilue
(a) Autos CorreeColll
(b) Full Attention
(c) LSH Attention
(d) ProbSparse Attention
Figure 5: Visualization of learned dependencies. For clearness, we select the top-6 time delay sizes
Ti, To of Auto-Correlation and mark them in raw series (red lines). For self-attentions, top-6 similar points with respect to the last time step (red stars) are also marked by orange points.
Daily
Mantrily
Daily Portad
30
Daily Period
Pertod
75
Period
Dumiity O1
Quartorly Paried
Weekky Perlod
15
Weekly
Yearl Period
50 100 150 200 250
100
350
50
100
150
300
250
100
350
50 100
150
200 250
300
350
0
50
100
150
200 250
300 350
Electricity (Hourly Recorded)
Exchange (Daily Recorded)
(c) Traffic (Hourly Recordedd)
Weather Recordeed)
Figure 6: Statistics of learned lags. For each time series in the test set, we count the top 10 lags learned by decoder for the 1oput96--rredic-3111 task. Figure (a)-(d) are the density histograms.
daily recorded Exchange dataset present the monthly, quarterly and yearly periods (Figure (b)). For the hourly recorded Traffic dataset (Figure 6 (c)), the learned lags show the intervals 24-hours and 1 168-hours, which match the daily and weekly periods of real-world scenarios. These results show that Autoformer can capture the complex seasonalities of real-world series from deep representations and further provide human-interprettblle prediction.
Efficiency analysis We compare the running memory and time among nutto-oorrelaton-bee and self nnttention-baser models (Figure 7 during the training phase. The proposed Autoformer shows OL log L) complexity in both memory and time and achieves better long-term sequences efficiency.
50
Autor Correlatoo from Autotorma" Attention hacaiörer
AuttFCorfelation from Abtolanter
Full Artention From hansiomeer Prolispare
Memory/GB
ProbSparse Attention informer
Timelmsi
30
from
20
10
192
384
768
3072
512
1024
2048
4096
8192
Output Length
Output Length
(a) Memory Efficiency Analysis
(b) Running Time Efficiency Analysis
Figure 7: Efficiency Analysis. For memory, we replace Auto-Correlation with self-attention family in Autoformer and record the memory with input 96. For running time, we run the Outo-Correlation or self-attentions 10 times to get the execution time per step. The output length increases exponentially.
Conclusions
This paper studies the long-term forecasting problem of time series, which is pressing demand for real-world applications. However, the intricate temporal patterns prevent the model from learning reliable dependencies. We propose the Autoformer as a decomposition architecture by embedding the series decomposition block as inner operator, which can progressively aggregate the long- term trend part from intermediate prediction. Besides, we design an efficient auto-Correlation mechanism to conduct dependencies discovery and information aggregation at the series level, which contrasts clearly from the previous self-attention family. Autoformer can naturally achieve O log I.) complexity and yield consistent state-of-the performance in extensive real-world datasets.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grants 62022050 and 62021002, Beijing Nova Program under Grant Z201 100006820041, China's Ministry of Industry and Information Technology, the MOE Innovation Plan and the BNRist Innovation Fund.
References
O. Anderson and M. Kendall. Time-series, 2nd edn. J. R. Stat. Soc. (Series D), 1976.
[2] Reza Asadi and Amelia C Regan. A spatio-temporal decomposition based deep neural network for time series forecasting. Appl. Soft Comput.. 2020.
[3] Shaojic Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint TrXiv:/803..122 2018.
[4] Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Conditional time series forecasting with convolutional neural networks. arXiv preprint arXiv: 17Xiv:700300669)0 2017.
[5] G. F. Box and Gwilym M. Jenkins. Time series analysis, forecasting and control. 1970.
[6] George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. J. R. Stat. Soc. (Series-C) C), 1968.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernr,, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models few-shot learners. In NeurlPS, 2020.
[8] Chris Chatfield. The analysis of time series: an introduction. 1981.
Renyi Chen and Molei Tao. Data-driven prediction of general hamiltonian dynamics via learning exactly- symplectic maps. ICMI,, 2021.
[10] Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. In NeurlPS, 2020.
[11] J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT, 2019.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[13] S. Hochreiter and Schmidhuber. Long short-term memory. Neural Comput.. 1997.
[14] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In ICLR, 2019.
[15] Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. 2018.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015.
[17] Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya. Reformer: The efficient transformer. ICLR, 2020.
[18] Richard Kurle, Syama Sundar Rangapuram, Emmanuel de Bézenac, Stephan Günnemann, and Jan Gasthaus. Deep rao-blackwellised particle filters for time series forecasting. In NeurlPS, 2020.
[19] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, 2018
[20] Shiyang Li. Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, In NewrIPS, 2019.

---

[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[22]
Danielle C Maddix, Yuyang Wang, and Alex Smola. Deep factors with gaussian processes for forecasting. arXiv preprint arXiv:1812.00090 2018.
[23] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis
expansion analysis for interpretable time series forecasting. ICLR, 2019.
[24]
Athanasios Papoulis and H Saunders. Probability, random variables and stochastic processes. 1989.
[25] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, I.. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- perfoomaefreee deep learning library. In NeurlPS, 2019.
[26] Syama Sundar Rangapuram, Matthias W Seeger. Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models time series forecasting. In NeurlPS, 2018.
[27] Cleveland Robert, C William, and Terpenning Irma. STL:: seasonal-trend decomposition procedure based on loess. J. Off. Stat, 1990.
[28] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. Int. J. Forecast., 2020.
[29] Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, locally: A deep neural network approach high- high-dimensional time series forecasting. NewrlPS, 2019.
[30] Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. Temporal pattern attention for multivariate time series forecasting. Mach. Learn., 2019.
[31] Huan Song, Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias, Attend and diagnose: Clinical time series analysis using attention models. In AAAI, 2018.
[32] Antti Sorjamaa, Jin Hao, Nima Reyhani, Yongnan Ji, and Amaury Lendasse. Methodology for long-term prediction time series. Neurocomputing, 2007.
[33] Sean Taylor and Benjamin Letham. Forecasting at scale. Am. Stat., 2018.
[34] Aäron van den Oord, S. Dieleman, H. Zen, K. Simonyan, Oriol Vinyals, A. Graves, Nal Kalchbrenner,
A. Senior, and Kavukcuoglu. Wavenet: generative model for raw audio. In SSW, 2016.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrIPS, 2017.
[36] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horize quantile recurrent forecaster. NeurlPS, 2017.
[37] Norbert Wiener. Generalized harmonic analysis. Acta Math, 1930.
[38] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. In NewrlPS, 2020.
[39] Q. Yao, D. Song, H. Chen, C. Wei, and G. W. Cottrell. A dual-stage attention- recurrent neural network for time series prediction. In IJCAI, 2017.
[40] Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor-train rans. arXiv preprint 17Xiv:1711.00073. 2017.
[41] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI, 2021.
12


Reviews:
[{"review_id": "sdtATq735Z", "reviewer": "Reviewer_ozCn", "summary": "This paper studies the problem of long-term forecasting for time series. There are two challenges: (1) it is difficult to directly study the entangled temporal dependencies for long time series; (2) the traditional self-attention mechanism in Transformer is computationally expensive. To address these two problems, the paper proposes a novel Autoformer with a novel auto-correlation mechanism and a series decomposition module. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n+ The auto-correlation mechanism is interesting and novel. \n+ The idea of series decomposition which is based on moving average is interesting.\n+ The experimental results demonstrate that the proposed Autotransformer is able to achieve better performance and efficiency.\n\nNegative points:\n- In equation (1), the dimensions of $\\mathcal{X}$ and $\\mathcal{X}_t$ are the same: $L\\times d$. I'm curious how $AvgPool()$ works on $\\mathcal{X}$? Why do you need Padding() operation on $\\mathcal{X}$?\n- In equation (2), why the length of the inputs is $\\frac{I}{2}+O$ but not $I+O$?\n- In the second equation of the equation (4), why the input of AutoCorrelation() contains 2 variables?\n- What's the motivation of the Roll() operation? \n- In line 175, why do you set $k=c\\times log L$? Is it because of the complexity?\n- In table 2, for Informer, there are some issues for the first row.\n- Notation issues:\n  (1) $\\mathcal{X}_t$ in equation (1) and equation (5) have different meaning.\n  (2) what does $C$ mean in Figure 2?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "eQCKE7Q6o_1", "reviewer": "Reviewer_e5xp", "summary": "The paper presents a novel decomposition approach to time series forecasting - by decomposing a time series in both the encoder and the decoder, they are able to learn complex patterns in a truly neural way.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper posits an interesting use of decomposition ideas through attention. Both the encoder and decoder and reasonably novel. They also compare MAE and MSE against some modern models.\n\nOverall, the idea is interesting - but lacks comparison against state of the art Transformer Models such as MQTransformer or TFT (https://arxiv.org/abs/2009.14799, https://arxiv.org/abs/1912.09363). Furthermore, they donot try to forecast distributions as most other models such as MQTransformer, DeepAR, TFT would do - so the comparison is still lacking.\n\nRevision: After the additional results, Im happy to switch to a marginal accept", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall, this is an interesting approach to a limited problem. I like the paper, but the comparisons are lacking enough that I cannot give it an accept. If the authors were to add quantile comparisons and state of the art models (for e.g. how do they handle the martingale quantile problem of Foster and Stine (2020) - https://arxiv.org/abs/2105.06834)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "bwqRjsor2dv", "reviewer": "Reviewer_8uZe", "summary": "This paper studies long-horizon time series forecasting where the historical input length is shorter than the forecast horizon, i.e, output length. This approach extends the transformer by replacing the self-attention unit by autocorrelation and decomposition units. \nThe autocorrelation unit chooses the top $k$ highly correlated time lags from $L$ time lags between the query and key embeddings of a time series of length $L$. The normalized correlation of lag j between the query and key embeddings multiplied by the j-item rotated value embedding. \nThe decomposition block decomposes a time series to seasonal and trend components. The trend component is defined by using an average pooling, which is an average value of the neighbouring values. And the seasonal is then defined as the difference between the (observed) time series and the trend component.\nThe proposed approach autocorrelation unit is compared with standard attention, logsparse attention, LSH and probsparse attention. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n-The proposed approach is interesting.\n-The experimental details are given as number of layers, learning rate, train-val-test set splits, stopping criteria, etc. Sharing the code can further improve the reproducibility.\nCons:\n-The choices of the hyper-parameter c, which controls the number of periods considered, is not clear.\n-There is no analysis on the decomposition for the known period and trends if it is learnt as it is expected.\n\nThe paper is easy to follow but the writing of the paper can be further improved. For instance there are a few misprecisions in the definition and equations, please see the detailed comments below.\n\nThe paper misses closely related work on time series forecasting using the periodicity of  time series without decomposing it but learning it with a period-aware attention mechanism, Cinar, Y. G., Mirisaee, H., Goswami, P., Gaussier, E., & Aït-Bachir, A. (2018). Period-aware content attention RNNs for time series forecasting with missing values. Neurocomputing, 312, 177-186. Furthermore, with the current experimental setup it is not very clear the effectiveness of decomposing the seasonality and trend in the data instead of explicitly modeling them as it is done by Cinar et al. without decomposing. It would be interesting to see this comparison in the experimental results.\n\nThe parameter c defines the number of periods chosen for a given input history. The logic of keeping it the same (value of 3) for each experiment is not clear. For instance, the electricity consumption time series is known to have daily and weekly periods, for ETT dataset the history of 96 time points covers 24 hour only, but the model asked to find 3 periods. Is there any analysis on the values of k which corresponds to the period length for each experiment? Is it confirming the assumption that the model could find the period in the dataset? \n\n---\nDetailed comments:\n-For an input length of $L$ an autocorrelation can be from upto lag $L-1$ as given in Equation 5. If the correlation is calculated between two different inputs it can lag 1 to L, but then it should not be called autocorrelation, please see Equation 6. This makes it confusing to read, as one is expecting to see autocorrelation with upto lag of $L-1$, but actually it defines a correlation upto lag $L$. Changing the name to correlation mechanism instead of auto-correlation can make it more clear.\n-In Equation 6, the equation of Autocorrelation(Q,K,V) the index of time lag is given as k but the sum indices i is defined from i to k, it needs to be corrected with i. It would be even better to denote $j$, as $i$ is later used for enumeration of attention heads.\n\nRevision Update: After the author response, the rating is updated to 6: Marginally above the acceptance threshold.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Broader impact is given in the appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a Transformer-based time series forecasting technique that decomposes the time series into the trend and seasonal parts using an encoder, and decodes the prediction. The transformer relies on an auto-correlation alternative to self-attention. The method is successfully evaluated on a number of time series problems.
Reviewers appreciate the idea of encoder/decoder decomposition of a time series, and its novelty. Reviewer e5xp asked for transformer-based baselines, which were provided by the authors. Reviewer 8uZe had questions about hyperparameter tuning and about various baselines, which were answered satisfactorily. Reviewer ozCn had questions about notation, which, again, were answered.
After careful consideration of the paper and given review scores of (6, 6, 6, mean 6), it appears that the paper could be accepted as a poster.

Decision: Accept (Poster)