SCformer: Segment Correlation Transformer for Long Sequence Time Series Forecasting

1 INTRODUCTION . Time series forecasting has always been a classic machine learning problem . It is widely used in various fields that are closely related to our lives , e.g. , production planning , financial investment , traffic management , and electricity management . In many cases , we need to predict the future value of the time series for a long time , which involves long sequence time-series forecasting ( LSTF ) ( Zhou et al. , 2021 ) . Classical mathematical forecasting models ( Lütkepohl , 2005 ; Box et al. , 2015 ; Cao & Tay , 2003 ; Roberts et al. , 2013 ) rely on strong assumptions on the time series and can only handle simple linear relationships . For example , the ARIMA model ( Box et al. , 2015 ) requires the original time series to be stable at least after the differentiation operation . Gaussian Process ( Roberts et al. , 2013 ) utilizes all the features to make predictions and fails in high-dimensional data space . Due to the ability of modeling long-term and nonlinear relationships for large-scale and complicated sequential data , deep learning models have achieved better performance than these classical methods . Existing time series forecasting methods based on deep learning can be divided into three categories , i.e. , RNN-based models ( Lai et al. , 2018 ; Salinas et al. , 2020 ; Qin et al. , 2017 ; Song et al. , 2018 ) , TCN-based models ( Borovykh et al. , 2017 ; Sen et al. , 2019 ) and Transformer-based models ( Vaswani et al. , 2017 ; Wu et al. , 2020a ) . RNN-based methods suffer from the problem of gradient vanishing , gradient exploding , and lack of parallelism . TCN-based methods need deeper layers to achieve larger local receptive fields . For both categories of methods , signals must pass through a long path between two far-away temporal locations , hence the number of operations required to associate two elements increases with their temporal distance . Differently , transformer-based methods directly model the relationships between any element pairs and can better capture long-term dependencies , which is crucial for LSTF . On the other hand , the self-attention mechanism of Transformer calculates all similarities between any element pairs . The computational and space complexities increase quadratically with the length of the time series . Directly applying Transformer to LSTF is not only inefficient but also difficult to capture truly effective attention . Recent works ( Zhou et al. , 2021 ; Li et al. , 2019 ; Kitaev et al. , 2019 ) explore different sparse attention mechanisms to suppress the contribution of irrelevant time steps and ease the computational pressure to a certain extent . These models still perform dot-product attention to time steps individually and utilize the point-wise connections to capture temporal dependencies . However , for LSTF , there is a strong correlation between neighboring points . A single point may have limited influence on predicting the future . Autoformer ( Wu et al. , 2021 ) conducts the series-wise dependencies discovery by performing auto-correlation of the time series to the top-k time delayed series . All points in the whole delayed series must share the same weight for aggregating the prediction and complicated fast Fourier transforms are required for auto-correlation . These methods perform the correlation either at the point level or at the overall series level , which not only require high computational redundancy to intensively tackle point pairs or perform timefrequency domain transformations , but also do not directly reflect the true dependencies within the time series . For instance , in traffic flow prediction , to predict the flow of the future period , the flows of a single previous time point and the shifted whole time series may have limited contribution . There may be stronger correlations at the segment level , e.g. , the flow of the peak time period today should be more related to the flow of the peak time period yesterday than the flow of the off-peak time period any day and the flow of a period that has passed a long time ago . In this paper , we propose a novel sparse attention mechanism called segment correlation attention ( SCAttention ) . Different from Auto-Correlation in Autoformer , SCAttention segments the time series based on implicit period and regards every segment as one unit to compute the correlation between segments . We extend the point-wise dot product in conventional transformer-based attention mechanism to the segment-wise correlation , where segments do not need to be transformed into the frequency domain . Segment-wise correlation not only reduces the amount of calculation since the number of segments is much smaller than the number of points , but also can be combined with other sparse attention mechanisms . We derive our prediction model for LSTF , namely Segment Correlation Transformer ( SCformer ) , via simply replacing the canonical attention in the original Transformer model with SCAttention . A stable predictor should generate consistent predictions , i.e. , for any given historical time series , if the predicted future series is reversed as the input , the output of the predictor should be consistent with the reverse of the historical time series . Motivated by this idea , we design a dual task as a regularization to train our SCformer model in order to achieve more robust predictions . Different from AST ( Wu et al. , 2020b ) which introduces adversarial loss to alleviate the error accumulation with an additional discriminator , our dual task-based regularization term only needs to reverse the predicted time series as the input to SCformer without introducing additional network structures and parameters . The main contributions of this paper are summarized as follows : • We propose a segment correlation attention ( SCAttention ) mechanism to replace the canonical point-wise attention mechanism . In addition to increasing efficiency , SCAttention also extracts more relevant information from the sequence . Besides , it can be easily combined with other sparse attention mechanisms for further improvement . • We design a dual task as a regularization term in the training phase . The dual reverse prediction task restores the past by the future . By encouraging the forward and backward predictions to be consistent , the learned SCformer generates more stable predictions . Ablation studies demonstrate the effect of the dual task . • Extensive experiments show that our SCformer model achieves better forecasting performance than other Transformer-based prediction models . 2 RELATED WORK . Time Series Forecasting . Early works on the TSF problem are based on classical mathematical models such as vector autoregression ( VAR ) ( Lütkepohl , 2005 ) and auto regressive intergrated moving average ( ARIMA ) ( Box et al. , 2015 ) . Support vector regression ( SVR ) ( Cao & Tay , 2003 ) introduces a traditional machine learning method to regress the future . Gaussian Process ( Roberts et al. , 2013 ) predicts the distribution of future values without assuming any certain form of the prediction function . However , all these classical models can not handle complicated and unknown data distributions or high-dimensional data . With the development of deep learning , neural networks have shown stronger modeling ability than classical models . Recurrent Neural Network ( RNN ) ( Connor et al. , 1992 ) and Temporal Convolution Network ( TCN ) ( Oord et al. , 2016 ; Bai et al. , 2018 ) are two common types of deep models for modeling sequence data . LSTNet ( Lai et al. , 2018 ) combines convolutional layers and recurrent layers to capture both long-term and short-term dependencies . DeepAR ( Salinas et al. , 2020 ) predicts the parameters of future distributions in an auto-regressive fashion . There are also some works ( Qin et al. , 2017 ; Song et al. , 2018 ) that introduce additional attention mechanism to RNN to achieve better performance in forecasting . However , RNN-based models suffer from the gradient vanishing and gradient exploding problem . Popular variants of RNN such as LSTM ( Hochreiter & Schmidhuber , 1997 ) and GRU ( Chung et al. , 2014 ) can not solve this problem fundamentally . The lack of parallelizability is another main limitation of RNN-based models . Benefiting from the good parallelism of convolution operations , TCN-based models have also been used in time series tasks and achieved good results ( Borovykh et al. , 2017 ; Sen et al. , 2019 ) . Both RNN-based and TCNbased models do not explicitly model the dependencies between two far-away temporal locations , but the information exchange between them must go through a long path . Transfomer-based models . Transformer ( Vaswani et al. , 2017 ) was originally proposed as a sequence-to-sequence model in natural language processing ( NLP ) to deal with machine translation . Due to its powerful and flexible modeling capabilities , it has even been widely applied in processing non-sequential data such as images in computer vision ( CV ) tasks ( Dosovitskiy et al. , 2020 ; Carion et al. , 2020 ) . Because of its huge success in NLP and CV , efforts have been made to adopt Transformer to solve the TSF problem ( Li et al. , 2019 ; Zhou et al. , 2021 ; Wu et al. , 2021 ; 2020b ; a ) . Self-attention ( Vaswani et al. , 2017 ) plays an important role in Transformer for explicitly discovering the dependencies between any element pairs , but both the time and space complexities increase quadratically with the length of the sequence , which limits the application of Transformer in LSTF ( Zhou et al. , 2021 ) . Therefore , various spare self-attention mechanisms for improving the efficiency of Transformer have been proposed in recent years . Logfomer ( Li et al. , 2019 ) proposes LogSparse self-attention which selects elements in exponentially increasing intervals to break the memory bottleneck . Informer ( Zhou et al. , 2021 ) defines a sparsity measurement for queries and selects dominant queries based on this measurement to obtain ProbSparse self-attention . Reformer ( Kitaev et al. , 2019 ) reduces the time and memory complexity by locally sensitive hashing self-attention . Most works use point-wise dot product to compute attention score , and differ in the way of selecting point pairs . AutoFomer ( Wu et al. , 2021 ) develops an Auto-Correlation mechanism to replace selfattention , which utilizes series-wise correlation instead of point-wise dot product . In this work , we introduce a new segment correlation attention mechanism to explore the context information within neighboring points and capture the segment-wise correlation in the sequence . Our method differs from the Auto-Correlation mechanism ( Wu et al. , 2021 ) in the way of correlation computation and aggregation . Instead of the complicated Fast Fourier Transforms calculation in Auto-Correlation , we directly segment the time series based on implicit period and compute the correlation between segments . Besides , we aggregate the segments by inter-segmentation correlation . AST ( Wu et al. , 2020b ) regards the predictor as a generator and utilizes adversarial training as the regularization for the sequence-level forecasting of time series . The adversarial training requires another discrimination network to distinguish the predicted sequences and the ground-truth sequences . Differently , to make the forecasting process more stable , we design a dual task for regularization without introducing additional parameters . 3 METHOD . 3.1 PROBLEM DEFINITION . We follow the comprehensive problem definition and setting about multi-horizon forecasting provided in ( Lim et al. , 2021 ) . Typically , given the previous time series X1 : t0 = { x1 , x2 , . . . , xt0 } , where xt ∈ Rdx and dx is the dimensionality of the variable , we aim to predict the future values Yt0+1 : t0+τ = { yt0+1 , yt0+2 , . . . , yt0+τ } , where yt ∈ Rdy is the prediction at every time step t and dy is the dimension of the output variable . The prediction model f can be formulated as : Ŷt0+1 : t0+τ = f ( X1 : t0 ; Ω ) , ( 1 ) where Ŷt0+1 : t0+τ is the predicted time series and Ω is the learnable parameters of the model . For LSTF , the prediction range τ , i.e. , the future time duration to be predicted , is longer . The problem can be categorized into two types based on whether the dimension of the output variable dy is larger than one : univariate LSTF and multivariate LSTF .

Summary: This paper introduces a SCFORMER, which replaces the canonical attention in the Transformer with the segment correlation attention. The motivation of using the segment correlation is to reduce the memory usage of the scale-dot product attention of the Transformer. To further improve the performance, the paper proposes a dual task, which use the current time series to predict the past time series.