Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

1 Introduction . Time series forecasting has been widely used in energy consumption , traffic and economics planning , weather and disease propagation forecasting . In these real-world applications , one pressing demand is to extend the forecast time into the far future , which is quite meaningful for the long-term planning and early warning . Thus , in this paper , we study the long-term forecasting problem of time series , characterizing itself by the large length of predicted time series . Recent deep forecasting models [ 41 , 17 , 20 , 28 , 23 , 29 , 19 , 35 ] have achieved great progress , especially the Transformer-based models . Benefiting from the self-attention mechanism , Transformers obtain great advantage in modeling long-term dependencies for sequential data , which enables more powerful big models [ 7 , 11 ] . However , the forecasting task is extremely challenging under the long-term setting . First , it is unreliable to discover the temporal dependencies directly from the long-term time series because the dependencies can be obscured by entangled temporal patterns . Second , canonical Transformers with self-attention mechanisms are computationally prohibitive for long-term forecasting because of the quadratic complexity of sequence length . Previous Transformer-based forecasting models [ 41 , 17 , 20 ] mainly focus on improving self-attention to a sparse version . While performance is significantly improved , these models still utilize the point-wise representation aggregation . Thus , in the process of efficiency improvement , they will sacrifice the information utilization because of the sparse point-wise connections , resulting in a bottleneck for long-term forecasting of time series . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To reason about the intricate temporal patterns , we try to take the idea of decomposition , which is a standard method in time series analysis [ 1 , 27 ] . It can be used to process the complex time series and extract more predictable components . However , under the forecasting context , it can only be used as the pre-processing of past series because the future is unknown [ 15 ] . This common usage limits the capabilities of decomposition and overlooks the potential future interactions among decomposed components . Thus , we attempt to go beyond pre-processing usage of decomposition and propose a generic architecture to empower the deep forecasting models with immanent capacity of progressive decomposition . Further , decomposition can ravel out the entangled temporal patterns and highlight the inherent properties of time series [ 15 ] . Benefiting from this , we try to take advantage of the series periodicity to renovate the point-wise connection in self-attention . We observe that the sub-series at the same phase position among periods often present similar temporal processes . Thus , we try to construct a series-level connection based on the process similarity derived by series periodicity . Based on the above motivations , we propose an original Autoformer in place of the Transformers for long-term time series forecasting . Autoformer still follows residual and encoder-decoder structure but renovates Transformer into a decomposition forecasting architecture . By embedding our proposed decomposition blocks as the inner operators , Autoformer can progressively separate the long-term trend information from predicted hidden variables . This design allows our model to alternately decompose and refine the intermediate results during the forecasting procedure . Inspired by the stochastic process theory [ 8 , 24 ] , Autoformer introduces an Auto-Correlation mechanism in place of self-attention , which discovers the sub-series similarity based on the series periodicity and aggregates similar sub-series from underlying periods . This series-wise mechanism achieves O ( L logL ) complexity for length-L series and breaks the information utilization bottleneck by expanding the point-wise representation aggregation to sub-series level . Autoformer achieves the state-of-the-art accuracy on six benchmarks . The contributions are summarized as follows : • To tackle the intricate temporal patterns of the long-term future , we present Autoformer as a decomposition architecture and design the inner decomposition block to empower the deep forecasting model with immanent progressive decomposition capacity . • We propose an Auto-Correlation mechanism with dependencies discovery and information aggregation at the series level . Our mechanism is beyond previous self-attention family and can simultaneously benefit the computation efficiency and information utilization . • Autoformer achieves a 38 % relative improvement under the long-term setting on six benchmarks , covering five real-world applications : energy , traffic , economics , weather and disease . 2 Related Work . 2.1 Models for Time Series Forecasting . Due to the immense importance of time series forecasting , various models have been well developed . Many time series forecasting methods start from the classic tools [ 32 , 9 ] . ARIMA [ 6 , 5 ] tackles the forecasting problem by transforming the non-stationary process to stationary through differencing . The filtering method is also introduced for series forecasting [ 18 , 10 ] . Besides , recurrent neural networks ( RNNs ) models are used to model the temporal dependencies for time series [ 36 , 26 , 40 , 22 ] . DeepAR [ 28 ] combines autoregressive methods and RNNs to model the probabilistic distribution of future series . LSTNet [ 19 ] introduces convolutional neural networks ( CNNs ) with recurrent-skip connections to capture the short-term and long-term temporal patterns . Attention-based RNNs [ 39 , 30 , 31 ] introduce the temporal attention to explore the long-range dependencies for prediction . Also , many works based on temporal convolution networks ( TCN ) [ 34 , 4 , 3 , 29 ] attempt to model the temporal causality with the causal convolution . These deep forecasting models mainly focus on the temporal relation modeling by recurrent connections , temporal attention or causal convolution . Recently , Transformers [ 35 , 38 ] based on the self-attention mechanism shows great power in sequential data , such as natural language processing [ 11 , 7 ] , audio processing [ 14 ] and even computer vision [ 12 , 21 ] . However , applying self-attention to long-term time series forecasting is computationally prohibitive because of the quadratic complexity of sequence length L in both memory and time . LogTrans [ 20 ] introduces the local convolution to Transformer and proposes the LogSparse attention to select time steps following the exponentially increasing intervals , which reduces the complexity to O ( L ( logL ) 2 ) . Reformer [ 17 ] presents the local-sensitive hashing ( LSH ) attention and reduces the complexity to O ( L logL ) . Informer [ 41 ] extends Transformer with KL-divergence based ProbSparse attention and also achieves O ( L logL ) complexity . Note that these methods are based on the vanilla Transformer and try to improve the self-attention mechanism to a sparse version , which still follows the point-wise dependency and aggregation . In this paper , our proposed Auto-Correlation mechanism is based on the inherent periodicity of time series and can provide series-wise connections . 2.2 Decomposition of Time Series . As a standard method in time series analysis , time series decomposition [ 1 , 27 ] deconstructs a time series into several components , each representing one of the underlying categories of patterns that are more predictable . It is primarily useful for exploring historical changes over time . For the forecasting tasks , decomposition is always used as the pre-processing of historical series before predicting future series [ 15 , 2 ] , such as Prophet [ 33 ] with trend-seasonality decomposition and N-BEATS [ 23 ] with basis expansion and DeepGLO [ 29 ] with matrix decomposition . However , such pre-processing is limited by the plain decomposition effect of historical series and overlooks the hierarchical interaction between the underlying patterns of series in the long-term future . This paper takes the decomposition idea from a new progressive dimension . Our Autoformer harnesses the decomposition as an inner block of deep models , which can progressively decompose the hidden series throughout the whole forecasting process , including both the past series and the predicted intermediate results . 3 Autoformer . The time series forecasting problem is to predict the most probable length-O series in the future given the past length-I series , denoting as input-I-predict-O . The long-term forecasting setting is to predict the long-term future , i.e . larger O . As aforementioned , we have highlighted the difficulties of long-term series forecasting : handling intricate temporal patterns and breaking the bottleneck of computation efficiency and information utilization . To tackle these two challenges , we introduce the decomposition as a builtin block to the deep forecasting model and propose Autoformer as a decomposition architecture . Besides , we design the Auto-Correlation mechanism to discover the period-based dependencies and aggregate similar sub-series from underlying periods . 3.1 Decomposition Architecture . We renovate Transformer [ 35 ] to a deep decomposition architecture ( Figure 1 ) , including the inner series decomposition block , Auto-Correlation mechanism , and corresponding Encoder and Decoder . Series decomposition block To learn with the complex temporal patterns in long-term forecasting context , we take the idea of decomposition [ 1 , 27 ] , which can separate the series into trend-cyclical and seasonal parts . These two parts reflect the long-term progression and the seasonality of the series respectively . However , directly decomposing is unrealizable for future series because the future is just unknown . To tackle this dilemma , we present a series decomposition block as an inner operation of Autoformer ( Figure 1 ) , which can extract the long-term stationary trend from predicted intermediate hidden variables progressively . Concretely , we adapt the moving average to smooth out periodic fluctuations and highlight the long-term trends . For length-L input series X ∈ RL×d , the process is : Xt = AvgPool ( Padding ( X ) ) Xs = X − Xt , ( 1 ) where Xs , Xt ∈ RL×d denote the seasonal and the extracted trend-cyclical part respectively . We adopt the AvgPool ( · ) for moving average with the padding operation to keep the series length unchanged . We use Xs , Xt = SeriesDecomp ( X ) to summarize above equations , which is a model inner block . Model inputs The inputs of encoder part are the past I time stepsXen ∈ RI×d . As a decomposition architecture ( Figure 1 ) , the input of Autoformer decoder contains both the seasonal part Xdes ∈ R ( I2+O ) ×d and trend-cyclical part Xdet ∈ R ( I 2+O ) ×d to be refined . Each initialization consists of two parts : the component decomposed from the latter half of encoder ’ s input Xen with length I2 to provide recent information , placeholders with length O filled by scalars . It ’ s formulized as follows : Xens , Xent = SeriesDecomp ( Xen I 2 : I ) Xdes = Concat ( Xens , X0 ) Xdet = Concat ( Xent , XMean ) , ( 2 ) where Xens , Xent ∈ R I 2×d denote the seasonal and trend-cyclical parts of Xen respectively , and X0 , XMean ∈ RO×d denote the placeholders filled with zero and the mean of Xen respectively . Encoder As shown in Figure 1 , the encoder focuses on the seasonal part modeling . The output of the encoder contains the past seasonal information and will be used as the cross information to help the decoder refine prediction results . Suppose we have N encoder layers . The overall equations for l-th encoder layer are summarized as X len = Encoder ( X l−1en ) . Details are shown as follows : Sl,1en , = SeriesDecomp ( Auto-Correlation ( X l−1en ) + X l−1en ) Sl,2en , = SeriesDecomp ( FeedForward ( Sl,1en ) + Sl,1en ) , ( 3 ) where “ ” is the eliminated trend part . X len = Sl,2en , l ∈ { 1 , · · · , N } denotes the output of l-th encoder layer and X 0en is the embedded Xen . Sl , ien , i ∈ { 1 , 2 } represents the seasonal component after the i-th series decomposition block in the l-th layer respectively . We will give detailed description of Auto-Correlation ( · ) in the next section , which can seamlessly replace the self-attention . Decoder The decoder contains two parts : the accumulation structure for trend-cyclical components and the stacked Auto-Correlation mechanism for seasonal components ( Figure 1 ) . Each decoder layer contains the inner Auto-Correlation and encoder-decoder Auto-Correlation , which can refine the prediction and utilize the past seasonal information respectively . Note that the model extracts the potential trend from the intermediate hidden variables during the decoder , allowing Autoformer to progressively refine the trend prediction and eliminate interference information for period-based dependencies discovery in Auto-Correlation . Suppose there are M decoder layers . With the latent variable XNen from the encoder , the equations of l-th decoder layer can be summarized as X lde = Decoder ( X l−1de , XNen ) . The decoder can be formalized as follows : Sl,1de , T l,1 de = SeriesDecomp ( Auto-Correlation ( X l−1de ) + X l−1de ) Sl,2de , T l,2 de = SeriesDecomp ( Auto-Correlation ( Sl,1de , XNen ) + S l,1 de ) Sl,3de , T l,3 de = SeriesDecomp ( FeedForward ( Sl,2de ) + S l,2 de ) T lde = T l−1de +Wl,1 ∗ T l,1 de +Wl,2 ∗ T l,2 de +Wl,3 ∗ T l,3 de , ( 4 ) where X lde = Sl,3de , l ∈ { 1 , · · · , M } denotes the output of l-th decoder layer . X 0de is embedded from Xdes for deep transform and T 0de = Xdet is for accumulation . Sl , ide , T l , i de , i ∈ { 1 , 2 , 3 } represent the seasonal component and trend-cyclical component after the i-th series decomposition block in the l-th layer respectively . Wl , i , i ∈ { 1 , 2 , 3 } represents the projector for the i-th extracted trend T l , ide . … L Q K V FFT x FFT Inverse FFT Time Delay Agg Linear Linear Linear Linear Concat LxC SxC SxC LxCx2 LxC LxC LxCx2 ConjugateLxCx2 LxC Resize Resize LxC Roll ( ) Roll ( ) Roll ( ) x x x … Fu sio n Time Delay Top k kxC SoftMax test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 2 ⌧k ⌧1 ) ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 Figure 2 : Auto-Correlation ( left ) and Time Delay Aggregation ( right ) . We utilize the Fast Fourier Transform to calculate the autocorrelationR ( τ ) , which reflects the time-delay similarities . Then the similar sub-processes are rolled to the same index based on selected delay τ and aggregated byR ( τ ) . The final prediction is the sum of the two refined decomposed components , asWS ∗ XMde + T Mde , whereWS is to project the deep transformed seasonal component XMde to the target dimension .

Summary: This paper focuses on the long-term forecasting problem of time series. This work introduces the traditional idea of decomposing time series into seasonality and trend-cycle components into deep Transformer architecture. This work also proposes an auto-correlation mechanism that replaces the original self-attention in Transformers.  Further, they improve the computation efficiency by computing power spectral density. Experiments on several datasets show that the proposed method achieves better performance compared to SOTA methods.