SCFORMER: SEGMENT CORRELATION TRANSFORMER FOR LONG SEQUENCE TIME SERIES FORECASTING
Year: 2021

Abstract:
Long-term time series forecasting is widely used in real-world applications such as financial investment, electricity management and production planning. Recently, transformer-based models with strong sequence modeling ability have shown the potential in this task. However, most of these methods adopt point-wise dependencies discovery, whose complexity increases quadratically with the length of time series, which easily becomes intractable for long-term prediction. This paper proposes a new Transformer-based model called SCformer, which replaces the canonical self-attention with efficient segment correlation attention (SCAttention) mechanism. SCAttention divides time series into segments by the implicit series periodicity and utilizes correlations between segments to capture long short-term dependencies. Besides, we design a dual task that restores past series with the predicted future series to make SCformer more stable. Extensive experiments on several datasets in various fields demonstrate that our SCformer outperforms other Transformer-based methods and training with the additional dual task can enhance the generalization ability of the prediction model.

heading: 1 INTRODUCTION
text: Time series forecasting has always been a classic machine learning problem. It is widely used in various fields that are closely related to our lives, e.g., production planning, financial investment, traffic management, and electricity management. In many cases, we need to predict the future value of the time series for a long time, which involves long sequence time-series forecasting (LSTF) (Zhou et al., 2021). Classical mathematical forecasting models (Lütkepohl, 2005; Box et al., 2015; Cao & Tay, 2003; Roberts et al., 2013) rely on strong assumptions on the time series and can only handle simple linear relationships. For example, the ARIMA model (Box et al., 2015) requires the original time series to be stable at least after the differentiation operation. Gaussian Process (Roberts et al., 2013) utilizes all the features to make predictions and fails in high-dimensional data space.
Due to the ability of modeling long-term and nonlinear relationships for large-scale and complicated sequential data, deep learning models have achieved better performance than these classical methods. Existing time series forecasting methods based on deep learning can be divided into three categories, i.e., RNN-based models (Lai et al., 2018; Salinas et al., 2020; Qin et al., 2017; Song et al., 2018), TCN-based models (Borovykh et al., 2017; Sen et al., 2019) and Transformer-based models (Vaswani et al., 2017; Wu et al., 2020a). RNN-based methods suffer from the problem of gradient vanishing, gradient exploding, and lack of parallelism. TCN-based methods need deeper layers to achieve larger local receptive fields. For both categories of methods, signals must pass through a long path between two far-away temporal locations, hence the number of operations required to associate two elements increases with their temporal distance. Differently, transformer-based methods directly model the relationships between any element pairs and can better capture long-term dependencies, which is crucial for LSTF.
On the other hand, the self-attention mechanism of Transformer calculates all similarities between any element pairs. The computational and space complexities increase quadratically with the length of the time series. Directly applying Transformer to LSTF is not only inefficient but also difficult to capture truly effective attention. Recent works (Zhou et al., 2021; Li et al., 2019; Kitaev et al., 2019) explore different sparse attention mechanisms to suppress the contribution of irrelevant time steps and ease the computational pressure to a certain extent. These models still perform dot-product
attention to time steps individually and utilize the point-wise connections to capture temporal dependencies. However, for LSTF, there is a strong correlation between neighboring points. A single point may have limited influence on predicting the future. Autoformer (Wu et al., 2021) conducts the series-wise dependencies discovery by performing auto-correlation of the time series to the top-k time delayed series. All points in the whole delayed series must share the same weight for aggregating the prediction and complicated fast Fourier transforms are required for auto-correlation.
These methods perform the correlation either at the point level or at the overall series level, which not only require high computational redundancy to intensively tackle point pairs or perform timefrequency domain transformations, but also do not directly reflect the true dependencies within the time series. For instance, in traffic flow prediction, to predict the flow of the future period, the flows of a single previous time point and the shifted whole time series may have limited contribution. There may be stronger correlations at the segment level, e.g., the flow of the peak time period today should be more related to the flow of the peak time period yesterday than the flow of the off-peak time period any day and the flow of a period that has passed a long time ago.
In this paper, we propose a novel sparse attention mechanism called segment correlation attention (SCAttention). Different from Auto-Correlation in Autoformer, SCAttention segments the time series based on implicit period and regards every segment as one unit to compute the correlation between segments. We extend the point-wise dot product in conventional transformer-based attention mechanism to the segment-wise correlation, where segments do not need to be transformed into the frequency domain. Segment-wise correlation not only reduces the amount of calculation since the number of segments is much smaller than the number of points, but also can be combined with other sparse attention mechanisms. We derive our prediction model for LSTF, namely Segment Correlation Transformer (SCformer), via simply replacing the canonical attention in the original Transformer model with SCAttention.
A stable predictor should generate consistent predictions, i.e., for any given historical time series, if the predicted future series is reversed as the input, the output of the predictor should be consistent with the reverse of the historical time series. Motivated by this idea, we design a dual task as a regularization to train our SCformer model in order to achieve more robust predictions. Different from AST (Wu et al., 2020b) which introduces adversarial loss to alleviate the error accumulation with an additional discriminator, our dual task-based regularization term only needs to reverse the predicted time series as the input to SCformer without introducing additional network structures and parameters.
The main contributions of this paper are summarized as follows:
• We propose a segment correlation attention (SCAttention) mechanism to replace the canonical point-wise attention mechanism. In addition to increasing efficiency, SCAttention also extracts more relevant information from the sequence. Besides, it can be easily combined with other sparse attention mechanisms for further improvement.
• We design a dual task as a regularization term in the training phase. The dual reverse prediction task restores the past by the future. By encouraging the forward and backward predictions to be consistent, the learned SCformer generates more stable predictions. Ablation studies demonstrate the effect of the dual task.
• Extensive experiments show that our SCformer model achieves better forecasting performance than other Transformer-based prediction models.

heading: 2 RELATED WORK
text: Time Series Forecasting. Early works on the TSF problem are based on classical mathematical models such as vector autoregression (VAR) (Lütkepohl, 2005) and auto regressive intergrated moving average (ARIMA) (Box et al., 2015). Support vector regression (SVR) (Cao & Tay, 2003) introduces a traditional machine learning method to regress the future. Gaussian Process (Roberts et al., 2013) predicts the distribution of future values without assuming any certain form of the prediction function. However, all these classical models can not handle complicated and unknown data distributions or high-dimensional data.
With the development of deep learning, neural networks have shown stronger modeling ability than classical models. Recurrent Neural Network (RNN) (Connor et al., 1992) and Temporal Convolution Network (TCN) (Oord et al., 2016; Bai et al., 2018) are two common types of deep models for modeling sequence data. LSTNet (Lai et al., 2018) combines convolutional layers and recurrent layers to capture both long-term and short-term dependencies. DeepAR (Salinas et al., 2020) predicts the parameters of future distributions in an auto-regressive fashion. There are also some works (Qin et al., 2017; Song et al., 2018) that introduce additional attention mechanism to RNN to achieve better performance in forecasting. However, RNN-based models suffer from the gradient vanishing and gradient exploding problem. Popular variants of RNN such as LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Chung et al., 2014) can not solve this problem fundamentally. The lack of parallelizability is another main limitation of RNN-based models. Benefiting from the good parallelism of convolution operations, TCN-based models have also been used in time series tasks and achieved good results (Borovykh et al., 2017; Sen et al., 2019). Both RNN-based and TCNbased models do not explicitly model the dependencies between two far-away temporal locations, but the information exchange between them must go through a long path.
Transfomer-based models. Transformer (Vaswani et al., 2017) was originally proposed as a sequence-to-sequence model in natural language processing (NLP) to deal with machine translation. Due to its powerful and flexible modeling capabilities, it has even been widely applied in processing non-sequential data such as images in computer vision (CV) tasks (Dosovitskiy et al., 2020; Carion et al., 2020). Because of its huge success in NLP and CV, efforts have been made to adopt Transformer to solve the TSF problem (Li et al., 2019; Zhou et al., 2021; Wu et al., 2021; 2020b;a). Self-attention (Vaswani et al., 2017) plays an important role in Transformer for explicitly discovering the dependencies between any element pairs, but both the time and space complexities increase quadratically with the length of the sequence, which limits the application of Transformer in LSTF (Zhou et al., 2021).
Therefore, various spare self-attention mechanisms for improving the efficiency of Transformer have been proposed in recent years. Logfomer (Li et al., 2019) proposes LogSparse self-attention which selects elements in exponentially increasing intervals to break the memory bottleneck. Informer (Zhou et al., 2021) defines a sparsity measurement for queries and selects dominant queries based on this measurement to obtain ProbSparse self-attention. Reformer (Kitaev et al., 2019) reduces the time and memory complexity by locally sensitive hashing self-attention.
Most works use point-wise dot product to compute attention score, and differ in the way of selecting point pairs. AutoFomer (Wu et al., 2021) develops an Auto-Correlation mechanism to replace selfattention, which utilizes series-wise correlation instead of point-wise dot product. In this work, we introduce a new segment correlation attention mechanism to explore the context information within neighboring points and capture the segment-wise correlation in the sequence. Our method differs from the Auto-Correlation mechanism (Wu et al., 2021) in the way of correlation computation and aggregation. Instead of the complicated Fast Fourier Transforms calculation in Auto-Correlation, we directly segment the time series based on implicit period and compute the correlation between segments. Besides, we aggregate the segments by inter-segmentation correlation.
AST (Wu et al., 2020b) regards the predictor as a generator and utilizes adversarial training as the regularization for the sequence-level forecasting of time series. The adversarial training requires another discrimination network to distinguish the predicted sequences and the ground-truth sequences. Differently, to make the forecasting process more stable, we design a dual task for regularization without introducing additional parameters.

heading: 3 METHOD

heading: 3.1 PROBLEM DEFINITION
text: We follow the comprehensive problem definition and setting about multi-horizon forecasting provided in (Lim et al., 2021). Typically, given the previous time series X1:t0 = {x1,x2, . . . ,xt0}, where xt ∈ Rdx and dx is the dimensionality of the variable, we aim to predict the future values Yt0+1:t0+τ = {yt0+1,yt0+2, . . . ,yt0+τ}, where yt ∈ Rdy is the prediction at every time step t and
dy is the dimension of the output variable. The prediction model f can be formulated as:
Ŷt0+1:t0+τ = f(X1:t0 ; Ω), (1)
where Ŷt0+1:t0+τ is the predicted time series and Ω is the learnable parameters of the model. For LSTF, the prediction range τ , i.e., the future time duration to be predicted, is longer. The problem can be categorized into two types based on whether the dimension of the output variable dy is larger than one: univariate LSTF and multivariate LSTF.

heading: 3.2 THE SCFORMER MODEL
text: As shown in Figure 1, the overall architecture of SCformer is similar to the vanilla Transformer. The original time series X1:t0 is fed into the encoder and decoder through an embedding layer. We replace dot-product self-attention in Transformer with SCAttention. Besides, a fully connected layer is added to convert the output of the decoder into prediction values Ŷt0+1:t0+τ .
Embedding. Some prediction models utilize various time-dependent features (e.g., a set of dummy variables like hour-of-the-day, day-of-the-week, etc) and time-independent features(e.g., location of the store, car lane id, etc) to help prediction. But in this paper, we do not use these additional features as parts of inputs considering that useless noises are not conducive to prediction. Therefore, the inputs for our model only include values of the original time series. The input X ∈ RL×d is transformed into H ∈ RL×dmodel , followed by the commonly used absolute sinusoidal position encoding(Vaswani et al., 2017):
H = Embed(X) = XW + Epos, (2)
where W ∈ Rd×dmodel is a learnable projection matrix and Epos is exactly the same as the positional encoding in Vaswani et al. (2017). The inputs of both encoder and decoder are processed by the above way.
Encoder. The encoder consists of N identical layers, where each layer consists of a SCAttention module and a feed forward network (FFN) each followed by layer normalization with residual connections. The inputs of encoder are the past t0 time steps X1:t0 ∈ Rt0×dx . Then the encoder embedding layer converts X1:t0 into H 0 en ∈ Rt0×dmodel . Subsequently, for the l-th encoder layer, the input is the hidden representation of the the l − 1-th layer and the output can be calculated as:
Hl,1en = LayerNorm(SCAttention(H l−1 en ) + H l−1 en )
Hlen = LayerNorm(FFN(H l,1 en) + H l,1 en).
(3)
HNen denotes the output of the encoder which will be fed into the cross-attention module in each decoder layer.
Decoder. The decoder consists of M identical layers. Different from the encoder, there is an additional multi-head cross-attention layer where key and value matrices are transformed by the output of the encoder in each decoder layer. The inputs Xtd:t0 of the decoder are the latter parts of the inputs of the encoder, where td is the start time for the inputs of the decoder and 1 ≤ td ≤ t0. After the decoder embedding layer, H0de ∈ R(t0−td)×dmodel will be fed into the first decoder layer. For the l-th decoder layer, the input is the concatenation of the hidden representation of the l − 1-th decoder layer and the output of the encoder. Its output can be calculated as:
Hl,1de = LayerNorm(SCAttention(H l−1 de ) + H l−1 de )
Hl,2de = LayerNorm(SCAttention(H l,1 de ,H N en) + H l,1 de )
Hlde = LayerNorm(FFN(H l,2 de ) + H l,2 de ).
(4)
An additional fully connected layer takes Hlde, the output of M -th decoder layer, as input, and it outputs final prediction values Ŷt0+1:t0+τ ∈ Rτ×dy :
Ŷt0+1:t0+τ = FC(H M de), (5)
where FC not only includes fully connected projection but also includes the flatten and resize operations to get prediction values with dimension dy .

heading: 3.3 SEGMENT CORRELATION ATTENTION
text: SCAttention is the key module in our SCformer, which performs segment-wise attention instead of point-wise attention. We denote the input of each SCAttention module as H ∈ RL×dmodel1, where L and dmodel are the length of the time series and the dimension of variables respectively. Formally, for the single head situation, the input series H will be projected by three projection matrices to obtain the query, key and value, i.e., Q = HWQ,K = HWK ,V = HWV , where WQ,WK ,WV ∈ Rdmodel×d, respectively. Then all the Q, K and V are segmented into several segments having the same length Lseg:
SEG(Q, Lseg) = {Q1,Q2, . . . ,Qm},Qi ∈ RLseg×d
SEG(K, Lseg) = {K1,K2, . . . ,Kn},Ki ∈ RLseg×d
SEG(V, Lseg) = {V1,V2, . . . ,Vn},Vi ∈ RLseg×d, (6)
where m = n = L//Lseg if query and key come from the same input. SEG(X, l) represents the segmentation operation that divides X into several segments with the same length l. Generally, we
1For convenience, we omit the layer index and the subscript which identifies the encoder or decoder.
set the segment length Lseg to the implicit period of the time series. However, since the number of segments is more than 1, it is also an appropriate choice to set Lseg to a factor of the period. More exploration on Lseg is described in Section 4.3. The segmentation operation is shown in Figure 2.
The correlation vector cij between any pair of query segment Qi and key segment Kj can be computed by the Correlation function:
cij = Correlation(Qi,Kj), (7) where the Correlation function is implemented by computing the reduce-sum of element-wise product between two matrices with same size. For each query segment Qi, all the correlation vectors will be normalized by the Softmax function to get the weight vector {ĉij | j = 1, . . . , n}:
ĉi1, . . . , ĉ i n = Softmax(c i 1, . . . , c i n). (8)
Considering that the importance of different dimensions differs, we use weight vectors to aggregate all the value segments instead of scalar weights. Specifically, Ai is the weighted sum of all the value segments {Vj | j = 1, . . . , n}, as shown in Figure 2:
Ai = n∑ j=1 ĉij Vj , (9)
where is the dot product operator with broadcasting between one matrix and one vector. Lastly, output of SCAttention layer can be obtained by concatenating all the Ai along length dimension:
SCAttention(Q,K,V) = Concat(A1, . . . ,Am). (10)
The output of multi-head SCAttention can be computed by concatenating and projecting the outputs of all heads described above. We omit the formulation since it is similar to the canonical multi-head attention (Vaswani et al., 2017).
As a result, all time steps in a segment share the same aggregation weight vector, as opposed to dotproduct attention, where each time step has its own aggregation scalar weight. This segment-wise attention reduces redundant point-wise calculations so that both time and memory efficiency have been improved, i.e., the complexity is reduced to O(L2/Lseg).

heading: 3.4 DUAL TASK
text: Many real time series have shown certain seasonal periodicity (Hyndman & Athanasopoulos, 2018), and it should have a steady upward trend and a steady downward trend. If a prediction model is powerful enough to use past sequences to predict the future, it should be able to use future sequences to restore the past in turn. On the contrary, if the past sequences recovered by the predictor differ greatly from the inputs, the predictor is not stable and the predictions are not reliable. Motivated by this idea, we design a dual task as a regularization to train our SCformer model in order to achieve more robust prediction performance.
Dual task training is shown in the right part of Figure 1. The primal task, i.e., the forward prediction task, is described in (1). We choose the MSE loss function on model prediction w.r.t the ground truth. The reverse prediction task aims to restore the past series according to the future data.
X1:t0+τ and Y1:t0+τ denote original time series and target values during the whole duration, respectively. For the forward prediction task, X′1:t0 = X1:t0 is fed into the model to get Ŷ ′ t0+1:t0+τ and the ground truth is Y′t0+1:t0+τ = Yt0+1:t0+τ . Therefore, the loss of forward prediction task is:
Lforward = MSE(Ŷ ′t0+1:t0+τ ,Y ′ t0+1:t0+τ ). (11)
For the reverse prediction task, the input of the model is X′′1:t0 = Xt0+τ :1+τ :−1, where −1 in subscript indicates that the index is decreasing at continuous interval. Ŷ′′t0+1:t0+τ and Y ′′ t0+1:t0+τ = Yτ :1:−1 denote the output and the ground truth respectively, and the loss of dual task can be formulated as:
Lreverse = MSE(Ŷ ′′t0+1:t0+τ ,Y ′′ t0+1:t0+τ ). (12)
Therefore, the total loss function L can be defined by: L = Lforward + λLreverse, (13)
where λ is the hyper-parameter that balances losses of two tasks.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: Datasets We conduct experiments on the following four datasets as in Zhou et al. (2021). (1) ETT2 (Zhou et al., 2021) contains data related to electricity which is collected from two Chinese stations in two years. In order to explore the model’s performance on data with different granularities, we use different sampling frequencies to get hourly data {ETTh1, ETTh2} and 15-minutes data ETTm1. The train/val/test contains 12/4/4 months of data. (2) Weather 3 collects hourly climatological data of about 1600 U.S. locations in 4 years, which contains 11 climate features in addition to ”wet bulb”. The train/val/test contains 28/10/10 months of data. (3) Electricity 4 contains the hourly electricity consumption of 321 clients in 2 years. The train/val/test contains 15/3/4 months of data.
Implementation details We use the ADAM (Kingma & Ba, 2014) optimizer with a constant learning rate 1e-3 to train our model. The number of training epochs is set to 10. The hyper-parameter λ in (13) is set to 1 for all experiments. All experiments are implemented with PyTorch (Paszke et al., 2019) and conducted on four NVIDIA TITAN RTX 24GB GPUs. For most experimental settings, SCformer contains 2 encoder layers and 1 decoder layer. Several models are selected to compare with SCformer, including three transformer-based models: Informer (Zhou et al., 2021), LogTrans (Li et al., 2019), Reformer (Kitaev et al., 2019), one RNN-based model: LSTM (Hochreiter & Schmidhuber, 1997) and one TCN-based model: TCN (Bai et al., 2018). To better explore the SCAttention’s performance, we train our SCformer without the dual task to get SCformer†.
Evalution metrics On each prediction window, we use two metrics which are commonly used in prediction task to evaluate performances, i.e., Mean Square Error (MSE) and Mean Absolute Error (MAE). They are defined as follows: MSE = 1n ∑n i=1(yi−ŷi)2,MAE = 1 n ∑n i=1 |yi−ŷi|, where yi is ground truth and ŷi is prediction result. For multivariate prediction, the metrics can be calculated and averaged on each single variable.

heading: 4.2 MAIN RESULTS
text: Univariate Time-series Forecasting In the univariate setting, both dimensions dx and dy of the input and output variables are 1, which means only a single dimension variable is utilized to predict itself. Following the setting in Zhou et al. (2021), we regard ”oil temperature” in ETT-series datasets, ”wet bulb” in Weather, and ”MT 320” in Electricity as the single feature, respectively. The results are shown in Table 1. We observe that SCformer with dual task training achieves the best results in most cases. For example, under predict-48 settings, compared to previous state-of-the-art results, SCformer has achieved 49% (0.108→ 0.055) relative improvement on MSE in ETTh1, 1% (0.103 → 0.102) in ETTh2, 48% (0.044 → 0.023) in ETTm1, 16% (0.164 → 0.137) in Weather, and 16% (0.311→ 0.262) in Electricity. Besides, SCformer outperforms SCformer† in most cases, which shows the effectiveness of dual task training. Further evaluation of the dual task is presented in the ablation study. Moreover, SCformer† can achieve better results than other models even without employing dual tasks, which illustrates SCAttention is more effective than other sparse-attention mechanisms, i.e., ProbSparse in Informer, LogSparse in LogTrans, and LSHSparse in Reformer. The overall performance of Transformer-based models is better than RNN-based model and TCN-based model, proving the potential of the Transformer-based model in time-series forecasting.
Multivariate Time-series Forecasting Multivariate time-series forecasting aims at predicting multi-dimensional future values with multi-dimensional inputs. We can handle multivariate prediction by simply modifying the fully connected layer and embedding layer in SCformer. From Table 2, we find that the conclusion drawn in univariate prediction still holds in multivariate prediction. SCformer is also better than other sparse models, LSTM and TCN in multivariate prediction. For example, under predict-48 settings, compared to previous state-of-the-art results, SCformer or SCformer† has achieved 25% (0.551→ 0.413) relative MSE improvement in ETTh1, 17% (0.680→ 0.563) in ETTh2, 26% (0.472→ 0.351) in ETTm1, 6% (0.400→ 0.377) in Weather, and 13% (0.267→ 0.232) in Electricity.
2https://github.com/zhouhaoyi/ETDstations. 3https://www.ncdc.noaa.gov/orders/qclcd/. 4https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014.

heading: 4.3 ABLATION STUDY
text: SCAttention vs. sparse-attention family. We conduct experiments on the ETTh1 dataset with different settings to compare several different sparse attention mechanisms. For a fair comparison, we do not use the dual task training in all the experiments. The segment length of SCformer is set to 24. From Table 3, we observe that the performance increases as the encoder input becomes longer benefiting from more historical information. With longer inputs of the encoder, SCAttention achieves better performance than canonical self-attention (SCformer‡). Moreover, compared with other sparse mechanisms, SCformer† achieves the best performance, which further illustrates the effectiveness of the proposed SCAttention.
Analysis of the segment length. The segment length Lseg is a critical hyperparameter in SCAttention. We conduct experiments (input-336 to predict-336) on different datasets to explore the relationship between the segment length and time series. In order to eliminate the influence of other factors, all experiments are conducted without dual task training. As shown in Table 4, its optimal setting coincides with the implicit period of the time series.
The effect of the dual task. In order to explore the effect of the dual task, we train two models with or without the additional dual task to obtain 4 sets of model settings, i.e., SCformer with dual task (SCformer) and without dual task (SCformer†), vanilla Transformer with dual task (Transformer) and without dual task (Transformer†). To get a more comprehensive comparison, experiments are conducted on the ETTh1 dataset under different prediction types and horizons. From Table 5, we observe that the dual task training is useful for both these two models. The overall improvement ratios of MSE and MAE have reached 8.2% and 5.2% respectively.

heading: 5 CONCLUSION
text: This paper studies the long sequence time series forecasting problem which has urgent application demands. To tackle this problem, we propose a sparse and efficient attention mechanism called SCAttention which utilizes correlations between segment pairs to discover dependencies and aggregate information in time series. Under various experimental settings in different datasets, SCformer with SCAttention mechanism can yield state-of-the-art prediction performance, which means our SCAttention can extract the dependencies in time series in a better way than the previous sparseattention family. Besides, to achieve better performance, we design a simple and effective dual task to train our model, which restores the past with the future. This dual task can be used not only in SCformer but also in other predictive models. Future exploration will focus on the combination of SCAttention and other sparse mechanisms.
ETHICS STATEMENT
This research does not involve any ethical issues. All the datasets in this paper are public for research use, without privacy leakage.
REPRODUCIBILITY STATEMENT
We preprocess all the datasets following the steps in an existing public repository: https:// github.com/zhouhaoyi/Informer2020, which is the official PyTorch implementation of Informer (Zhou et al., 2021). We have reimplemented the original Transformer as http://nlp. seas.harvard.edu/2018/04/03/attention.html and modified some modules to get our SCformer model. Our code will be released soon if the paper is accepted.

heading: A APPENDIX
text: A.1 COMPARE WITH AUTO-CORRELATION IN AUTOFORMER
As shown in Table 6, we conduct experiments on ETTm2 dataset to compare our SCAttention with Auto-Correlation in Autoformer. We fix the input length as 96 and adopt the forecasting setting as multivariate. In order to compare these two attention mechanisms fairly, we train our SCformer model without the dual task and remove the progressive decomposition module in Autoformer. Compared with Auto-Correlation, our SCAttention can achieve comparable performance without complicated FFT calculations.
A.2 EFFECT OF HYPER-PARAMETER
To explore the effect of hyper-parameter, we conduct experiments about the number of heads and encoder layers on ETTh1 dataset. We adopt the forecasting setting as multivariate and predict-168. As shown in Table 7, setting the number of heads to 4 is the best. The number of decoder layers M is fixed to 1, we study the effect of the number of encoder layers N on the model. As shown in Table 8, when the number of encoder layers is small, the modeling and predictive ability of SCformer is insufficient. While the number of layers is large, the prediction performance is not significantly improved with the amount of parameters increasing. Therefore, it is best to set N to 2.

Review summaries:
This paper presents a Transformer-based model called SCformer to perform long sequence time series forecasting. The key idea is to replace the canonical self-attention with efficient segment correlation attention (SCAttention) mechanism to capture long short-term dependencies. Experiment results on several datasets showed the effectiveness of the proposed method.

This paper introduces a SCFORMER, which replaces the canonical attention in the Transformer with the segment correlation attention. The motivation of using the segment correlation is to reduce the memory usage of the scale-dot product attention of the Transformer. To further improve the performance, the paper proposes a dual task, which use the current time series to predict the past time series.