Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

1 Introduction . Time series forecasting has been widely used in energy consumption , traffic and economics planning , weather and disease propagation forecasting . In these real-world applications , one pressing demand is to extend the forecast time into the far future , which is quite meaningful for the long-term planning and early warning . Thus , in this paper , we study the long-term forecasting problem of time series , characterizing itself by the large length of predicted time series . Recent deep forecasting models [ 41 , 17 , 20 , 28 , 23 , 29 , 19 , 35 ] have achieved great progress , especially the Transformer-based models . Benefiting from the self-attention mechanism , Transformers obtain great advantage in modeling long-term dependencies for sequential data , which enables more powerful big models [ 7 , 11 ] . However , the forecasting task is extremely challenging under the long-term setting . First , it is unreliable to discover the temporal dependencies directly from the long-term time series because the dependencies can be obscured by entangled temporal patterns . Second , canonical Transformers with self-attention mechanisms are computationally prohibitive for long-term forecasting because of the quadratic complexity of sequence length . Previous Transformer-based forecasting models [ 41 , 17 , 20 ] mainly focus on improving self-attention to a sparse version . While performance is significantly improved , these models still utilize the point-wise representation aggregation . Thus , in the process of efficiency improvement , they will sacrifice the information utilization because of the sparse point-wise connections , resulting in a bottleneck for long-term forecasting of time series . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To reason about the intricate temporal patterns , we try to take the idea of decomposition , which is a standard method in time series analysis [ 1 , 27 ] . It can be used to process the complex time series and extract more predictable components . However , under the forecasting context , it can only be used as the pre-processing of past series because the future is unknown [ 15 ] . This common usage limits the capabilities of decomposition and overlooks the potential future interactions among decomposed components . Thus , we attempt to go beyond pre-processing usage of decomposition and propose a generic architecture to empower the deep forecasting models with immanent capacity of progressive decomposition . Further , decomposition can ravel out the entangled temporal patterns and highlight the inherent properties of time series [ 15 ] . Benefiting from this , we try to take advantage of the series periodicity to renovate the point-wise connection in self-attention . We observe that the sub-series at the same phase position among periods often present similar temporal processes . Thus , we try to construct a series-level connection based on the process similarity derived by series periodicity . Based on the above motivations , we propose an original Autoformer in place of the Transformers for long-term time series forecasting . Autoformer still follows residual and encoder-decoder structure but renovates Transformer into a decomposition forecasting architecture . By embedding our proposed decomposition blocks as the inner operators , Autoformer can progressively separate the long-term trend information from predicted hidden variables . This design allows our model to alternately decompose and refine the intermediate results during the forecasting procedure . Inspired by the stochastic process theory [ 8 , 24 ] , Autoformer introduces an Auto-Correlation mechanism in place of self-attention , which discovers the sub-series similarity based on the series periodicity and aggregates similar sub-series from underlying periods . This series-wise mechanism achieves O ( L logL ) complexity for length-L series and breaks the information utilization bottleneck by expanding the point-wise representation aggregation to sub-series level . Autoformer achieves the state-of-the-art accuracy on six benchmarks . The contributions are summarized as follows : • To tackle the intricate temporal patterns of the long-term future , we present Autoformer as a decomposition architecture and design the inner decomposition block to empower the deep forecasting model with immanent progressive decomposition capacity . • We propose an Auto-Correlation mechanism with dependencies discovery and information aggregation at the series level . Our mechanism is beyond previous self-attention family and can simultaneously benefit the computation efficiency and information utilization . • Autoformer achieves a 38 % relative improvement under the long-term setting on six benchmarks , covering five real-world applications : energy , traffic , economics , weather and disease . 2 Related Work . 2.1 Models for Time Series Forecasting . Due to the immense importance of time series forecasting , various models have been well developed . Many time series forecasting methods start from the classic tools [ 32 , 9 ] . ARIMA [ 6 , 5 ] tackles the forecasting problem by transforming the non-stationary process to stationary through differencing . The filtering method is also introduced for series forecasting [ 18 , 10 ] . Besides , recurrent neural networks ( RNNs ) models are used to model the temporal dependencies for time series [ 36 , 26 , 40 , 22 ] . DeepAR [ 28 ] combines autoregressive methods and RNNs to model the probabilistic distribution of future series . LSTNet [ 19 ] introduces convolutional neural networks ( CNNs ) with recurrent-skip connections to capture the short-term and long-term temporal patterns . Attention-based RNNs [ 39 , 30 , 31 ] introduce the temporal attention to explore the long-range dependencies for prediction . Also , many works based on temporal convolution networks ( TCN ) [ 34 , 4 , 3 , 29 ] attempt to model the temporal causality with the causal convolution . These deep forecasting models mainly focus on the temporal relation modeling by recurrent connections , temporal attention or causal convolution . Recently , Transformers [ 35 , 38 ] based on the self-attention mechanism shows great power in sequential data , such as natural language processing [ 11 , 7 ] , audio processing [ 14 ] and even computer vision [ 12 , 21 ] . However , applying self-attention to long-term time series forecasting is computationally prohibitive because of the quadratic complexity of sequence length L in both memory and time . LogTrans [ 20 ] introduces the local convolution to Transformer and proposes the LogSparse attention to select time steps following the exponentially increasing intervals , which reduces the complexity to O ( L ( logL ) 2 ) . Reformer [ 17 ] presents the local-sensitive hashing ( LSH ) attention and reduces the complexity to O ( L logL ) . Informer [ 41 ] extends Transformer with KL-divergence based ProbSparse attention and also achieves O ( L logL ) complexity . Note that these methods are based on the vanilla Transformer and try to improve the self-attention mechanism to a sparse version , which still follows the point-wise dependency and aggregation . In this paper , our proposed Auto-Correlation mechanism is based on the inherent periodicity of time series and can provide series-wise connections . 2.2 Decomposition of Time Series . As a standard method in time series analysis , time series decomposition [ 1 , 27 ] deconstructs a time series into several components , each representing one of the underlying categories of patterns that are more predictable . It is primarily useful for exploring historical changes over time . For the forecasting tasks , decomposition is always used as the pre-processing of historical series before predicting future series [ 15 , 2 ] , such as Prophet [ 33 ] with trend-seasonality decomposition and N-BEATS [ 23 ] with basis expansion and DeepGLO [ 29 ] with matrix decomposition . However , such pre-processing is limited by the plain decomposition effect of historical series and overlooks the hierarchical interaction between the underlying patterns of series in the long-term future . This paper takes the decomposition idea from a new progressive dimension . Our Autoformer harnesses the decomposition as an inner block of deep models , which can progressively decompose the hidden series throughout the whole forecasting process , including both the past series and the predicted intermediate results . 3 Autoformer . The time series forecasting problem is to predict the most probable length-O series in the future given the past length-I series , denoting as input-I-predict-O . The long-term forecasting setting is to predict the long-term future , i.e . larger O . As aforementioned , we have highlighted the difficulties of long-term series forecasting : handling intricate temporal patterns and breaking the bottleneck of computation efficiency and information utilization . To tackle these two challenges , we introduce the decomposition as a builtin block to the deep forecasting model and propose Autoformer as a decomposition architecture . Besides , we design the Auto-Correlation mechanism to discover the period-based dependencies and aggregate similar sub-series from underlying periods . 3.1 Decomposition Architecture . We renovate Transformer [ 35 ] to a deep decomposition architecture ( Figure 1 ) , including the inner series decomposition block , Auto-Correlation mechanism , and corresponding Encoder and Decoder . Series decomposition block To learn with the complex temporal patterns in long-term forecasting context , we take the idea of decomposition [ 1 , 27 ] , which can separate the series into trend-cyclical and seasonal parts . These two parts reflect the long-term progression and the seasonality of the series respectively . However , directly decomposing is unrealizable for future series because the future is just unknown . To tackle this dilemma , we present a series decomposition block as an inner operation of Autoformer ( Figure 1 ) , which can extract the long-term stationary trend from predicted intermediate hidden variables progressively . Concretely , we adapt the moving average to smooth out periodic fluctuations and highlight the long-term trends . For length-L input series X ∈ RL×d , the process is : Xt = AvgPool ( Padding ( X ) ) Xs = X − Xt , ( 1 ) where Xs , Xt ∈ RL×d denote the seasonal and the extracted trend-cyclical part respectively . We adopt the AvgPool ( · ) for moving average with the padding operation to keep the series length unchanged . We use Xs , Xt = SeriesDecomp ( X ) to summarize above equations , which is a model inner block . Model inputs The inputs of encoder part are the past I time stepsXen ∈ RI×d . As a decomposition architecture ( Figure 1 ) , the input of Autoformer decoder contains both the seasonal part Xdes ∈ R ( I2+O ) ×d and trend-cyclical part Xdet ∈ R ( I 2+O ) ×d to be refined . Each initialization consists of two parts : the component decomposed from the latter half of encoder ’ s input Xen with length I2 to provide recent information , placeholders with length O filled by scalars . It ’ s formulized as follows : Xens , Xent = SeriesDecomp ( Xen I 2 : I ) Xdes = Concat ( Xens , X0 ) Xdet = Concat ( Xent , XMean ) , ( 2 ) where Xens , Xent ∈ R I 2×d denote the seasonal and trend-cyclical parts of Xen respectively , and X0 , XMean ∈ RO×d denote the placeholders filled with zero and the mean of Xen respectively . Encoder As shown in Figure 1 , the encoder focuses on the seasonal part modeling . The output of the encoder contains the past seasonal information and will be used as the cross information to help the decoder refine prediction results . Suppose we have N encoder layers . The overall equations for l-th encoder layer are summarized as X len = Encoder ( X l−1en ) . Details are shown as follows : Sl,1en , = SeriesDecomp ( Auto-Correlation ( X l−1en ) + X l−1en ) Sl,2en , = SeriesDecomp ( FeedForward ( Sl,1en ) + Sl,1en ) , ( 3 ) where “ ” is the eliminated trend part . X len = Sl,2en , l ∈ { 1 , · · · , N } denotes the output of l-th encoder layer and X 0en is the embedded Xen . Sl , ien , i ∈ { 1 , 2 } represents the seasonal component after the i-th series decomposition block in the l-th layer respectively . We will give detailed description of Auto-Correlation ( · ) in the next section , which can seamlessly replace the self-attention . Decoder The decoder contains two parts : the accumulation structure for trend-cyclical components and the stacked Auto-Correlation mechanism for seasonal components ( Figure 1 ) . Each decoder layer contains the inner Auto-Correlation and encoder-decoder Auto-Correlation , which can refine the prediction and utilize the past seasonal information respectively . Note that the model extracts the potential trend from the intermediate hidden variables during the decoder , allowing Autoformer to progressively refine the trend prediction and eliminate interference information for period-based dependencies discovery in Auto-Correlation . Suppose there are M decoder layers . With the latent variable XNen from the encoder , the equations of l-th decoder layer can be summarized as X lde = Decoder ( X l−1de , XNen ) . The decoder can be formalized as follows : Sl,1de , T l,1 de = SeriesDecomp ( Auto-Correlation ( X l−1de ) + X l−1de ) Sl,2de , T l,2 de = SeriesDecomp ( Auto-Correlation ( Sl,1de , XNen ) + S l,1 de ) Sl,3de , T l,3 de = SeriesDecomp ( FeedForward ( Sl,2de ) + S l,2 de ) T lde = T l−1de +Wl,1 ∗ T l,1 de +Wl,2 ∗ T l,2 de +Wl,3 ∗ T l,3 de , ( 4 ) where X lde = Sl,3de , l ∈ { 1 , · · · , M } denotes the output of l-th decoder layer . X 0de is embedded from Xdes for deep transform and T 0de = Xdet is for accumulation . Sl , ide , T l , i de , i ∈ { 1 , 2 , 3 } represent the seasonal component and trend-cyclical component after the i-th series decomposition block in the l-th layer respectively . Wl , i , i ∈ { 1 , 2 , 3 } represents the projector for the i-th extracted trend T l , ide . … L Q K V FFT x FFT Inverse FFT Time Delay Agg Linear Linear Linear Linear Concat LxC SxC SxC LxCx2 LxC LxC LxCx2 ConjugateLxCx2 LxC Resize Resize LxC Roll ( ) Roll ( ) Roll ( ) x x x … Fu sio n Time Delay Top k kxC SoftMax test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 2 ⌧k ⌧1 ) ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 test haixu wu April 2021 1 Introduction ⌧1 ⌧2 ⌧k R ( ⌧1 ) R ( ⌧2 ) R ( ⌧k ) 1 Figure 2 : Auto-Correlation ( left ) and Time Delay Aggregation ( right ) . We utilize the Fast Fourier Transform to calculate the autocorrelationR ( τ ) , which reflects the time-delay similarities . Then the similar sub-processes are rolled to the same index based on selected delay τ and aggregated byR ( τ ) . The final prediction is the sum of the two refined decomposed components , asWS ∗ XMde + T Mde , whereWS is to project the deep transformed seasonal component XMde to the target dimension .

Summary: This paper studies long-horizon time series forecasting where the historical input length is shorter than the forecast horizon, i.e, output length. This approach extends the transformer by replacing the self-attention unit by autocorrelation and decomposition units.  The autocorrelation unit chooses the top $k$ highly correlated time lags from $L$ time lags between the query and key embeddings of a time series of length $L$. The normalized correlation of lag j between the query and key embeddings multiplied by the j-item rotated value embedding.  The decomposition block decomposes a time series to seasonal and trend components. The trend component is defined by using an average pooling, which is an average value of the neighbouring values. And the seasonal is then defined as the difference between the (observed) time series and the trend component. The proposed approach autocorrelation unit is compared with standard attention, logsparse attention, LSH and probsparse attention.