SCformer: Segment Correlation Transformer for Long Sequence Time Series Forecasting
Authors: Dazhao Du
Bing Su
Zhewei Wei
Venue: ICLR 2022 Submitted (2022)
Keywords: Transformer
time series forecasting
sparse attention

Abstract: Long-term time series forecasting is widely used in real-world applications such as financial investment, electricity management and production planning. Recently, transformer-based models with strong sequence modeling ability have shown the potential in this task. However, most of these methods adopt point-wise dependencies discovery, whose complexity increases quadratically with the length of time series, which easily becomes intractable for long-term prediction. This paper proposes a new Transformer-based model called SCformer, which replaces the canonical self-attention with efficient segment correlation attention (SCAttention) mechanism. SCAttention divides time series into segments by the implicit series periodicity and utilizes correlations between segments to capture long short-term dependencies. Besides, we design a dual task that restores past series with the predicted future series to make SCformer more stable. Extensive experiments on several datasets in various fields demonstrate that our SCformer outperforms other Transformer-based methods and training with the additional dual task can enhance the generalization ability of the prediction model.

Under review as a conference paper at ICLR 2022
SCFORMER: SEGMENT CORRELATION TRANSFORMER FOR LONG SEQUENCE TIME SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Long-term time series forecasting is widely used real-world upplications such as financial investment, electricity management and production planning. Recently, transformer-basee models with strong sequence modeling ability have shown the potential in this task. However, most of these methods adopt point-wise depen- dencies discovery, whose complexity increases quadratically with the length of time series, which easily becomes intractable for long-term prediction. This paper proposes a new Transformer- based model called SCformer, which replaces the canonical self-attention with efficient segment correlation attention (SCAttention) mechanism. SCAttention divides time series into segments by the implicit series periodicity and utilizes correlations between segments capture long short-term dependencies. Besides, we design a dual task that restores past series with the predicted future series to make SCformer more stable. Extensive experiments on several datasets in various fields demonstrate that our SCformer outperforms other Transformer-Be based methods and training with the additional dual task can enhance the generalization ability of the prediction model.
INTRODUCTION
Time series forecasting has always been a classic machine learning problem. It is widely used in various fields that are closely related to our lives, e.g., production planning, financial investment, traffic management, and electricity management. many cases, we need predict the future value of the time series for long time, which involves long sequence time-series forecasting (LSTF)
(Zhou et al., 202T). Classical mathematical forecasting models (Lütkepohl 2005 Box et al. 2015 Cao & Tay 2003 RRbeerss et al. 2013) rely strong assumptions the time series and can only handle simple linear relationships. For example, the ARIMA model (Box et al. 2015) requires the original time series to be stable at least after the differentiation operation. Gaussian Process (Roberts)
ct al. 2013) utilizes all the features to make predictions and fails in high-dimensional data space.
Due to the ability of modeling long-term and nonlinear relationships for large-scale and compli- cated sequential data, deep learning models have achieved better performance than these classical methods. Existing time series forecasting methods based on deep learning can be divided into three categories, i.e., RNN-based models (Lai et al.) 2018 Salinas et al. 2020 Qin et al.. 2017] [Son et al., 2018). baN-d models kBorovykh et al. 2017 Sen et al., 2019) and Transformer- based mod- els (Vaswani et al. 2017 Wu et al. 2020a). RNN-based methods suffer from the problem of gradient vanishing, gradient exploding, and lack of parallelism. TCN-based methods need deeper layers to achieve larger local receptive fields. For both categories of methods, signals must pass through a long path between two far-away temporal locations, hence the number of operations required to associate two elements increases with their temporal distance. Differently, transformer- meth- ods directly model the relationships between any element pairs and can better capture long-term dependencies, which is crucial for LSTF.
On the other hand, the self-attention mechanism of Transformer calculates all similarities between any element pairs. The computational and space complexities increase quadratically with the length of the time series. Directly applying Transformer to LSTF is not only inefficient but also difficult to capture truly effective attention. Recent works (Zhou et al. 1 Li al. 2019 Kitaev et al.) 2019) explore different sparse attention mechanisms to suppress the contribution of irrelevant time steps and ease the computational pressure certain extent. These models still perform dot-product

---

Under review as a conference paper at ICLR 2022
attention to time steps individually and utilize the point-wise connections to capture temporal de- pendencies. However, for LSTF, there is a strong correlation between neighboring points. A single point may have limited influence on predicting the future. Autoformer (Wu et al. 2021) conducts the series-wise dependencies discovery by performing auto-correlation of the time series to the top-k time delayed series. All points in the whole delayed series must share the same weight for aggregat- ing the prediction and complicated fast Fourier transforms are required for auto-correlation.
These methods perform the correlation either the point level or the overall series level, which not only require high computational redundancy to intensively tackle point pairs or perform time- frequency domain transformations, but also do not directly reflect the true dependencies within the time series. For instance, in traffic flow prediction, to predict the flow of the future period, the flows of a single previous time point and the shifted whole time series may have limited contribution. There may be stronger correlations at the segment level, e.g., the flow of the peak time period today should be more related the flow of the peak time period yesterday than the flow of the off-peak time period any day and the flow of period that has passed a long time ago.
In this paper, we propose a novel sparse attention mechanism called segment correlation atlen- tion (SCAntention). Different from Auto-Correlation in Autoformer, SCAttention segments the time series based on implicit period and regards every segment as one unit to compute the correlation between segments. We extend the point-wise dot product in conventional transformer-bras tion mechanism to the segment- wise correlation, where segments do not need to be transformed into the frequency domain. Segment-wise correlation not only reduces the amount of calculation since the number of segments is much smaller than the number of points, but also can be combined with other sparse attention mechanisms. We derive our prediction model for LSTF, namely Segment Correlation Transformer (SCformer), via simply replacing the canonical attention in the original Transformer model with SCAttention.
atten-
A stable predictor should generate consistent predictions, i.e., for any given historical time series, if the predicted future series is reversed as the input, the output of the predictor should be consistent with the reverse of the historical time series. Motivated by this idea, we design dual task as a regularization train our SCformer model in order to achieve more robust predictions. Different from AST (Wu et al.] (2000b) which introduces adversarial loss to alleviate the error accumulation with an additional discriminator, our dual task-based regularization term only needs to reverse the predicted time series as the input to SCformer without introducing additional network structures and parameters.
The main contributions of this paper are summarized as follows:
We propose segment correlation attention (SCAttention) mechanism to replace the canon- ical point-wise attention mechanism. In addition to increasing efficiency, SCAttention also extracts more relevant information from the sequence. Besides, it can be easily combined with other sparse attention mechanisms for further improvement.
We design a dual task as regularization term in the training phase. The dual reverse prediction task restores the past by the future. By encouraging the forward and backward predictions be consistent, the learned SCformer generates more stable predictions. Ab- lation studies demonstrate the effect of the dual task.
Extensive experiments show that our SCformer model achieves better forecasting perfor- mance than other Transformer- baasedd prediction models.
2 RELATED WORK
Time Series Forecasting. Early works on the TSF problem are based on classical mathematical models such as vector autoregression (VAR) (Lütkepohl) (005) and auto regressive intergrated mov- ing average (ARIMA) (Box et al. 2015). Support vector regression (SVR) (Cao & Tay| (003) introduces a traditional machine learning method to regress the future. Gaussian Process (Roberts) et al. 2013) predicts the distribution of future values without assuming any certain form of the pre- diction function. However, all these classical models can not handle complicated and unknown data distributions or high-dimensional data.
2

---

Under review as a conference paper at ICLR 2022
With the development of deep learning, neural networks have shown stronger modeling ability than classical models. Recurrent Neural Network (RNN) (Connor et al., 1992) and Temporal Convolu- tion Network (TCN) (Oord al. 2016 Bai et al. 2018) are two common types of deep models for modeling sequence data. Ne (Lai et al.; 2018) combines convolutional layers and recur- rent layers capture both long-term and short-term dependencies. DeepAR (Salinas et al., 2020) predicts the parameters of future distributions in an auto-regressive fashion. There are also some works (Qin al. 2017] Song al. 2018) that introduce additional attention mechanism RNN to achieve better performance in forecasting. However, RNN- RNN-based models suffer from the gradient vanishing and gradient exploding problem. Popular variants of RNN such as LSTM (Hochreiter & Schmidhuber 1997) and GRU (Chung al. (014) can not solve this problem fundamentally. The lack of paralleiizability is another main limitation of RNN-based models. Benefiting from the good parallelism of convolution operations, TCN-based models have also been used in time series tasks and achieved good results (Borovykh et al.) 2017; Sen et al. 2019). Both RNN-based and TCN- based models do not explicitly model the dependencies between two far-away temporal locations, but the information exchange between them must go through a long path.
Transfomer-baseds models. Transformer (Vaswani et al., 2017) was originally proposed as sequence- sequence-to-sequenee toseeuennce model in natural language processing (NLP) to deal with machine transla- tion. Due to its powerful and flexible modeling capabilities, it has even been widely applied in processing non-sequential data such as images in computer vision (CV) tasks (Dosovitskiy et al.)
2020 Carion al. 2020). Because of its huge success in NLP and CV, efforts have been made to adopt Transformer to solve the TSF problem (Li et al., 2019 Zhou et al. [021] Wu et al. [202]] 2020b a). Self-attention (Vaswani et al. 2017) plays an important role in Transformer for explicilly
discovering the dependencies between any element pairs, but both the time and space complexities increase quadratically with the length of the sequence, which limits the application of Transformer in LSTF (Zhou et al. 2021).
Therefore, various spare self-attention mechanisms for improving the efficiency of Transformer have been proposed in recent years. Logfomer (Li et al. 2019) proposes LogSparse self-attention which selects elements in exponentially increasing intervals to break the memory bottleneck. Informer (Zhou et al. 2021) defines a sparsity measurement for queries and selects dominant queries based on this measurement to obtain ProbSparse self-attention. Reformer (Kitaev et al. 2019) reduces the time and memory complexity by locally sensitive hashing self attf-attention.
Most works use woie dot product to compute attention score, and differ in the way of selecting point pairs. AutoFomer (Wu al., 2021) develops an Auto-Correlation mechanism replace self- attention, which utilizes series-wise correlation instead of point-wise dot product. In this work, we introduce new segment correlation attention mechanism explore the context information within neighboring points and capture the segment-wise correlation in the sequence. Our method differs from the Auto-Correlation mechanism (Wu al., (02T in the way of correlation computation and aggregation. Instead of the complicated Fast Fourier Transforms calculation in Auto-Correlation we directly segment the time series based on implicit period and compute the correlation between segments. Besides, we aggregate the segments by inter-segmentation correlation.
AST (Wu et al., 2020b) regards the predictor as a generator and utilizes adversarial training as the regularization for the sequence-level forecasting of time series. The adversarial training requires an- other discrimination network to distinguish the predicted sequences and the ground-trutt truth sequences. Differently, to make the forecasting process more stable, we design dual task for regularization without introducing additional parameters.
METHOD
3.1 PROBLEM DEFINITION
We follow the comprehensive problem definition and setting about multi-horizon forecasting pro- vided in (Lim et al. 2021). Typically, given the previous time series X1sto = {x1, x.,...; Xt, where x, E dd and d the dimensionality of the variable, aim predict the future values Y fo to Yto .., ). where y, RR is the prediction at every time step and
3

---

Under review as a conference paper at ICLR 2022
dy is the dimension of the output variable. The prediction model f can be formulated as:
Y 1o-1 - ((tstti))),
where Y, 1oto the predicted time series and Ω is the learnable parameters the model. For LSTF, the prediction range i.e., the future time duration be predicted, longer. The problem can be categorized into two types based on whether the dimension of the output variable dy is larger than one: univariate LSTF and multivariate LSTF.
3.2 THE SCFORMER MODEL
Pig ogge
Primal Task
Dual Task
ng +1++
Foward
Narr
Zi 115g
Sugment Comelation Croes Attention
Feed Forward
d f
Segmenn Consisiim Sct Antenoon
Comelation Sel-Allattio
Xi.ta
Xi.t,
Br codkr
Docodar emmediiing
entedaan
Xyita
Xegio
Figure 1: The overall framework the SCformer model. Left: the architecture. Right: the dual task in the training stage.
As shown in Figure 1 the overall architecture of SCformer is similar the vanilla Transformer. The original time series X is fed into the encoder and decoder through an embedding layer. We replace dot-producct self-attention in Transformer with SCAttention. Besides, a fully connected layer added to convert the output of the decoder into prediction values 1ot1o
Embedding, Some prediction models utilize various time-dependent features (c.g.. set of dummy variables like hour-of-the-day. dey- etc) and imee-independent features(e.g., location of the store, car lane id, etc) to help prediction. But in this paper, we do not use these additional features as parts of inputs considering that useless noises are not conducive to predic- tion. Therefore, the inputs for model only include values of the original time series, The input X E xd transformed into E IR Lxdmoddt followed by the commonly used absolute sinusoidal
position encoding Vaswani et al. 2017):
H Embed(X (X) = XW +
E
poss
where W E Bdxdmoodt is learnable projection matrix and Epos is exactly the same as the positional encoding Vaswani et al. (017). The inputs of both encoder and decoder are processed by the above way.
The encoder consists of N identical layers, where each layer consists of SCAtten- tion module and a feed forward network (FFN) each followed by layer normalization with residual connections. The inputs of encoder are the past fo time steps X1:to E xd Then the encoder embedding layer converts X isto into H
Encoder.
en E R Xo Subsequently, for the 1-th encoder layer, the input hidden representation of the the 1-th layer and the output can be calculated as:
H n
H!!1)
(3)
H en
LayerNorm(FPN((EHJ) + +..)).

---

Under review as a conference paper at ICLR 2022
H denotes the output of the encoder which will be fed into the cross-ateention module in each decoder layer.
The decoder consists of M identical layers. Different from the encoder, there is an additional multi-head cross-attention layer where key and value matrices are transformed by the output of the encoder in each decoder layer. The inputs &eytoo of the decoder the inputs the encoder, where t is the start time for the inputs the decoder and 1 S la S to- After the decoder embedding layer, de E X will be fed into the first decoder layer. For the 1-th decoder layer, the input is the concatenation of the hidden representation of the | 1-th decoder layer and the output of the encoder. Its output can be calculated as:
Decoder.
the latter parts of
H LayerNorm(SCAAttnniionn H ( H H H 2 LayerNormSSCAttentoon
H H + de
H
(4)
de
H de LayerNorm(FFN(HHH) )
An additional fully connected layer takes H' the output of M-th decoder layer, as input, and outputs final prediction values Y Yot1ittt E IR TRyxy
Y to+ F
(5)
where FC not only includes fully connected projection but also includes the flatten and resize oper- ations to get prediction values with dimension dy
3.3 SEGMENT CORRELATION ATTENTION
Eyeg
Segment
Cormiltioo
Softmax
Dot product
0 0
H2
Foop
Figure 2: Simplified schematic of SCAttention. Segmentation operation (left) and correlation atten- tion calculation for each query segment Q; (right).
SCAttention is the key module in our SCformer, which performs segment-wise attention instead of point-wise attention. We denote the input of each SCAttention module H E Rxxdmade where and dmodet are the length the time series and the dimension variables respectively. Formally, for the single head situation, the input series H will be projected by three projection matrices to obtain the query, key and value, i.c., Q = HWQ K = HWK, V HWv, where
Wo. WK. Wv E Bndmode xd respectively. Then all the Q, K and V are segmented into several segments having the same length Lseg:
SEG(Q, Loeeg) = [Q1, Q2 Qm), Qi E Xd SEG(K, I Iseg) = (K1, K2 .......,,,,, K, E Xdo
(6)
SEG(V, Lseg) (V1. V2, V2...., V.), V, Bhee xd
where m n L//Lzeg if query and key come from the same input. SEG(X,1) represents the segmentation operation that divides X into several segments with the same length (. Generally, we
For convenience, omit the layer index and the subscript which identifies the encoder decoder.
5

---

Under review as a conference paper at ICLR 2022
set the segment length Leeg to the implicit period of the time series. However, since the number of segments is more than 1. it is also an appropriate choice to set Leeg to a factor of the period. More exploration seg is described in Section 4.3 The segmentation operation is shown in Figure
The correlation vector cj between any pair of query segment Q; and key segment K; can be com- puted by the Correlation function:
cj Correlation Q., K2),
where the Correlation function is implemented by computing the reduce-sum of element-wise prod- uct between two matrices with same size. For each query segment Q; all the correlation vectors will be normalized by the Softmax function to get the weight vector {é} - n]:
é ci. è Softmax(cii.....tt ...)..
(8)
Considering that the importance of different dimensions differs, we use weight vectors to aggregate all the value segments instead of scalar weights. Specifically, A; is the weighted sum of all the value segments (V, j = -,...,,7 n as shown in Figure 2
A =
(9)
f=1
where @ is the dot product operator with broadcasting between one matrix and one vector. Lastly, output of SCAttention layer can be obtained by concatenating all the A, along length dimension:
SCAttention( Q K, V) Concat A 1,...,, Am).
(10)
The output of multi-head SCAttention can be computed by concatenating and projecting the outputs of heads described above. We omit the formulation since it is similar to the canonical multi-head attention (Vaswani et al. 2017).
As a result, all time steps in segment share the same aggregation weight vector, as opposed to dot- product attention, where each time step has its own aggregation scalar weight. This segment-wisg wise attention reduces redundant point-wise calculations so that both time and memory efficiency have been improved, i.e., the complexity is reduced to
3.4
DUAL TASK
Many real time series have shown certain seasonal periodicity (Hyndman & Athanasopoulos 2018). and it should have steady upward trend and steady downward trend. If prediction model is powerful enough to use past sequences to predict the future, it should be able to use future sequences restore the past in turn. On the contrary, if the past sequences recovered by the predictor differ greatly from the inputs, the predictor is not stable and the predictions are not reliable. Motivated by this idea, we design dual task as a regularization train our SCformer model order achieve more robust prediction performance.
Dual task training is shown in the right part of Figure The primal task, i.e., the forward prediction task, is described in (T). We choose the MSE loss function on model prediction w.r.t the ground truth. The reverse prediction task aims to restore the past series according to the future data.
X II and Y +: denote original time series and target values during the whole duration, re- spectively. For the forward prediction task, X iito X1st is fed into the model get Y 1ita and the ground truth Y 10+111047 - +T- Therefore, the loss of forward prediction task is:
Lforward = MSE( & +r+1:t++: Voo+1:447))
(11)
For the reverse prediction task, the input of the model is X X10 where -1 in sub- script indicates that the index is decreasing at continuous interval. Y +1 and Y +1 = Y Yr:1:-1 denote the output and the ground truth respectively, and the loss of dual task be formu- lated as:
Creverse MSE
Y! fot +1 t Y!" +1+
(12)
Therefore, the total loss function can be defined by:
Lyorward + Xereverse:
(13)
where À is the hyper-parameter that balances losses of two tasks.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets We conduct experiments on the following four datasets as in Zhou et al. (hoou et al., 202T contains data related to electricity which is collected from two Chinese stations in two years. In order to explore the model's performance on data with different granularities, we use different sampling frequencies to get hourly data {ETThI, ETTh2} and 15-minutes data ETTml. The train/val/test contains 12/4/4 months of data. (2) Weather collects hourly climatological data of about 1600 U.S. locations in years, which contains 11 climate features in addition to "wet bulb". The train/val/test contains 28/10/10 months of data. (3) Electricity Fl contains the hourly electricity consumption of 321 clients in 2 years. The train/val/test contains 15/3/4 months of data.
2021). (1) ET7
Implementation details We use the ADAM (Kingma & Ba 2014) optimizer with constant learn- ing rate le-3 to train our model. The number of training epochs is set to 10. The hyper-parameter X in (13) is set for all experiments. All experiments are implemented with PyTTrrc (Paszke et al.) 2019) and conducted on four NVIDIA TITAN RTX 24GB GPUs. For most experimental settings, SCformer contains encoder layers and 1 decoder layer. Several models are selected to compare with SCformer, including three transformer-basee models: Informer (Zhou et al.) 2021), LogTrans (Li et al., 2019), Reformer (Kitaev et al. 2019), one RNN-based model: LSTM (Hochre- iter & Schmidhuber 1997) and one TCN-b model: TCN (Bai et al.) 2018). To better explore the SCAttention's performance, we train our SCformer without the dual task to get SCformer'.
Evalution metries On each prediction window, we use two metrics which are commonly used in prediction task to evaluate performances, i.e., Mean Square Error (MSE) and Mean Absolute Error (MAE). They are defined follows: MSE = 1. Σ (y) &.)², MAE E 13/+ 1o--0 -vl. where y is ground truth and 9; is prediction result. For multivariate prediction, the metrics can be calculated and averaged each single variable.
4.2 MAIN RESULTS
Univariate Time-series Forecasting In the univariate setting, both dimensions de and d, of the input and output variables 1, which means only single dimension variable is utilized to pre- dict itself. Following the setting in Zhou et al. (202T). we regard "oil temperature" in ETT-series datasets, "wet bulb" in Weather, and "MT_ "M0" in Electricity the single feature, respectively. The results are shown in Table | We observe that SCformer with dual task training achieves the best results in most cases. For example, under predict-48 settings, compared to previous state-of-the-arr results, SCformer has achieved 49% (0.108 - 0.055) relative improvement on MSE in ETTh1, 1% (0.103 + 0. 0.102) in ETTh2, 48% (0.044 + 0.023) ETTml, 16% (0.164 - 0.137) in Weather, and 16% (0.311 + 0.262) in Electricity. Besides, SCformer outperforms SCformer in most cases, which shows the effectiveness of dual task training. Further evaluation of the dual task is presented in the ablation study. Moreover, SCformer! can achieve better results than other models even with- out employing dual tasks, which illustrates SCAttention is more effective than other aatsee-ttention mechanisms, i.c., ProbSparse in Informer, LogSparse in LogTrans, and LSHSparse in Reformer. The overall performance of Transformer-bar models is better than RNN-based model and TCN-based model, proving the potential of the Transformer-haat based model in time-series forecasting.
Multivariate Time-series Forecasting Multivariate seme-series forecasting aims at predicting multi-dimensional future values with multi-dimensional inputs. We can handle multivariate pre- diction by simply modifying the fully connected layer and embedding layer in SCformer. From Table 2 we find that the conclusion drawn in univariate prediction still holds in multivariate pre- diction. SCformer is also better than other sparse models, LSTM and TCN in multivariate pre- diction. For example, under predict-48 settings, compared to previous of-t-oo-thhear results, SC- former or SCformer has achieved 25% (0.551 + 0.413) relative MSE improvement in ETTh1, 17% (0.680 + 0.563) in ETTh2, 26% (0.472 > 0.351) in ETTm1, 6% (0.400 - 0.377) in Weather, and 13% (0.267 - 0.232) in Electricity.
htut:://gitubbccm///aaaaaa/////t "https//wwwwncce." nona gov/oddd/////////////dddd.///.
http://aeeeeeccccciiiooaaaaaaaaa 20ddDiagrams201 12014.
7

---

Under review as a conference paper at ICLR 2022
Table I: Univariate time-series forecasting results on four datasets
Models Metri
SClormet
SChormer
Informer
LogTrans
Reformer
LSTM
TCN
MSE
MAE
MSE
MAE
MSE
MAP
MSE
MAP
MSE
MAE
MSE
MAE
MSE
MAE
0.036
0.148
0,047
0.170
0.052
0.178
0.059
0.191
0.172
0.319
0.094
0.232
0.079
0.213
0.055
0.185
0.05%
0.189
0.108
0.345
0.228
0.395
0.175
0.115
0.259
0.081
0.232
0.087
0.233
0.146
0.294
0.155
0.309
1.460
1.089
0.210
0.352
0.175
0.538
0.101
0.208
0.361
0.978
0.556
0.238
0.396
0.276
0.146
0.365
0.245
0.399
24
000077
0.211
0.073
0.203
0.079
0.206
0.080
0.221
0.235
0.369
0.135
0.275
0,083
0,218
ETT
48
0.102
0.249
0.107
0.258
0.103
0.340
0.107
0.362
0.434
0.505
0.172
0,318
0.111
0.257
168
0.141
0.300
0.350
0.316
0.143
0.296
0.176
0.344
0.961
0.797
0.359
12470
0.188
0.337
336
0.179
0.335
0.189
0.344
0.171
0.327
0.175
0.345
1.532
1.060
0.516
0.548
0:250
0.387
7200
0.183
0.345
0.205
0.567
0.184
0.339
0.185
0.349
1,862
1.543
0.562
0.615
0,266
0.413
24
0.013
0.085
0.014
0/91
0,051
0.153
0.061
0.192
0.055
0.170
0.099
0.201
0:032
0.139
ETTml
0,025
0.112
0/022
0.136
1002
0:217
0.156
0.322
0.229
0.340
0.289
0.371
0.044
0.164
96
0.041
0.152
0.446
0.160
11119
01249
0.229
0.397
0354
0.675
0.255
0,370
D:071
0.202
288
0.073
0.208
0.074
0.208
08181
0,300
0.363
0.512
0.962
1.107
0.480
0.528
0.176
0.352
672
0.007
0.244
0.105
10:440
0.304
0.345
0:450
0.50 0.582
1.605
1.312
0.988
0.805
0.215
0.385
Weaher
0.207
0:256
0.131
0.088
0,107
0,104
0.223
0.383
0.120
0.182
0.197
0.268
0.329
0.381
0.166
0.112
0:165
0.239
0.295
0.337
0.220
0.338
0.267
0.590
0.552
0.305
0.233
0.303
336
0.349
0.261
0.241
0.353
0.200
18692
0.945
0.404
0.466
0:278
0.396
720
0.786
0.389
0.297
0.411
0.259
0.367
0.274
1.887
1.352
0.784
0.709
0.425
0.491
Electriity
48
0.255
0.362
0.262
0,364
0.335
0.423
0.360
0.455
0.917
0.840
0.175
0.509
0.473
0.506
0.262
0.371
0.276
0,380
0,408
0.466
1.515
0.205
0.511
0.411
0.293
0.392
0.303
0.400
0.451
0,488
0.482
3.448
2.088
1.166
0.854
0.330
0,421
0.337
0.438
0.466
0.409
0.523
3,913
0.44
0,484
0.461
0.388
0.530
0.546
6.841
0.503
Repured metrics in Informer. LogTrans, Reformer. LSTM come from
Zhou el al. 2021
Table 2: Multivariate time-series forecasting results on four datasets
Models Metric
SClormer
SCrermer
Informer MAE
LogTrans
Reformer MSE
LSTM
TCN
MAE
MSE
MSB
MAE
MSE
MAE
MSE
MAE
2.
0,351
0.412
0.372
0.427
0.509
0.523
0.656
0.600
0.887
0.630
0.536
0.528
0.576
0.518
48
0.413
0.451
0.430
0.457
0.551
0.563
0.670
0.611
1.159
0.750
0.616
0.577
0,572
0.543
168
0.654
0.592
0.695
0.614
0.878
0.722
0,888
0.766
1.686
0.996
1.558
0.725
0.779
1.544
336
0.769
0.672
0.834
0.689
0.884
0.753
0.942
0.366
1.919
1.090
1.152
0.795
1.189
0.846
720
0.952
0.766
0.964
0.777
0.941
0.768
1.109
0.543
2.177
1,218
L.682
1.018
1.110
0,963
0.250
0.345
0.310
0.402
0.646
0.523
0.726
0.638
1.381
1.475
1,049
0.689
0:606
0.694
F1T112
0.563
0.567
0.650
0.609
0.934
0.733
1.728
0944
1.715
1.585
1331
0.8 0.055
0.680
0.64)
168
1.337
0.894
1.35%
0,900
1.512
0.996
3.944
1.573
4.484
1.630
3.987
1,500
2.661
1,312
336
1.644
0,980
1.685
0,985
1,605
1:005
3.711
1.587
3.198
1.508
3.276
1,375
2.932
LA422
720
2.125
1.189
2.247
1.205
2,340
1.300
2.817
1.356
5.111
1.798
3.711
1.520
3.595
1,558
0.383
0.440
0.189
0:472
0,438
FTTml
0.952
0.645
1,280
0.568
0.479
0.426
0,642
0.626
1.267
0.795
L.195
0.607
0.528
0.467
0.699
1.219
0.571
1.228
1.660
1.632
0.886
1.598
0.670
0.611
672
0.640
0.571
0.686
0.624
1.651
1.002
13665
1.721
1.943
1.006
2.530
1,259
0:990
0.752
Weather
48
24
0.305
0.381
0.422
0.356
0,377
0.314
0,420
0.363
0.461
0.353
0.455
0.381
0.365
0.496
0.105
0.855
0.63
0.583
0.556
0.197
0.763
0.176
0-164
0.589
0:400
0.317
0.376
0.439
0.494
0.501
0.507
0.515
0,591
0.51
0.649
1.228
0.765
0,948
0.713
0.565
0.545
336
0.529
0.537
0.554
0.543
0.623
0.546
0.666
0.584
1.770
0.997
L.497
0.889
0,610
0.577
0.598
0.561
0.614
0.685
0.575
1.407
0.875
0.641
0.603
Elecriiity
168
48
0.232
0.280
0.344
0.382
0.273
0.269
0,374
0.359
0300
0.269
0.351
0.376
0.267
0.300
0.366
0.182
1.453
1.312
0.911
0.975
0.492
0.388
0,664
0.498
0.293
0.319
0:415
0.386
336
0.294
0,393
0.310
0.406
0331
0.385
0.305
0.295
1.507
0.978
0.778
0,629
0.395
0.477
720
0.302
0.400
0,313
0.408
0.308
0.385
0.311
0.397
1.883
1.002
L.528
0.915
0.325
0.412
960
0.319
0.417
0,323
0.415
0.328
0.406
0.333
0.413
1.973
1.185
1.345
0.886
0.374
11445
'Repuried metrics in Informer. LogTran, Reformer. 1.5T come from
Zhou et al.
2021
4.3 ABLATION STUDY
SCAttention vs. sparse-attention family. We conduct experiments on the ETTh1 dataset with different settings to compare several different sparse attention mechanisms. For fair comparison, we do not use the dual task training in all the experiments. The segment length of SCformer is set to 24. From Table 3 we observe that the performance increases as the encoder input becomes longer benefiting from more historical information. With longer inputs of the encoder, SCAttention achieves better performance than canonical self-attention (SCformer'). Moreover, compared with other sparse mechanisms, SCformeri achieves the best performance, which further illustrates the effectiveness of the proposed SCAttenion.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison of SCAttention and other sparse attention
Table 4: Influence of segment length on different datasets
Prodiction borizon
336
730
Segment longth
12
24
Encodor's is input length
336
720
1440
2850
0.102
MAB
0.219
0.263
0.252
0.242
0.261
SCformer f
MSE MAE
0.236
0.215
0.198
0.248
0.209
0,480
0.414
0.377
0.506
0.375
BTTM
MAR
MSE
0.208
0.367
0.205
0.364
0.202
0.365
0.184
0.349
0,195
0.362
MSF
0.248
0.247
0.784
SCYormer
MAE
0.469
0.452
0.539
RTTml
MSE
0.069
0.061
0.067
0.086
0.066
MAE
0.205
0.192
0.201
0.231
0.199
LogTrans
MAB
0:256
0.496
0.412
0.523
Weather
MSF
MAE
0.268
0.383
0.380
0.263
0.376
0.260
0.393
0.272
0.276
0.391
Informer
MAE
MSE
0.263
0.487
0.225
0.404
0.212
0.381
0.258
0.503
0.238
0.599
0,387
0.224
Electriitt
MAE
MSE
0.398
0.299
0,359
0,300
0.302
0.396
0.301
0.396
0.401
0
Reformer
MSE MAB
1,848
1.832
1.817
2,094
2.055
2,032
1,054
1.027
1,010
1,363
1.306
1.384
SCformee replaces SCAttemiin in SCformer! with canonical istention.
indicates the cut-of-manoory [2 32C18)
Results except SCYormer come from Zhou
2021
Table 5: Ablation of the dual task
Prodiction type
Univariate
Multivaiatte
Prediction
24
730
24
730
SClormer
MSE
0.148
0.036
0.183
0,055
0,081
0.094
0.121
0.276
0.412
0.351
0.451
0.413
0.592
0.854
0.769
0.766
0.952
SClonner
MSE
0.047
0,058
0,087
0.101
0.146
0.372
0.430
0.895
0.834
0.961
MAE
0.170
0.189
0.233
0.306
0.427
0.457
0.614
0.777
Transformer
MSE
0.180
0.055
0,066
0,095
0.109
0.318
0.170
0.458
0.427
0.472
0.466
0.818
0,876
0.790
1.011
MSE MAE
0.058
0,075
0,107
0.119
0.190
0.420
0.489
0.857
0.967
1.110
Transformer"
0.187
0.220
0.272
0.349
0.462
0.511
0.694
0.762
0.842
SClomner Transfomerr trained tack SCformer? T and Transfommer" trained without dual
Transformer denotes a buic Transfommer model which replaces SCAtrention $ SCoomner with canonical attention.
Analysis of the segment length. The segment length Teg is critical hyperparameter in SCAt- tention. We conduct experiments (input-336 to predict-336) on different datasets to explore the relationship between the segment length and time series. In order eliminate the influence of other factors, all experiments are conducted without dual task training. As shown in Table 4 its optimal setting coincides with the implicit period of the time series.
The effect of the dual task. In order to explore the effect of the dual task, we train two models with or without the additional dual task to obtain 4 sets of model settings, i.c.. SCformer with dual task (SCformer) and without dual task (SCformer'), ), vanilla Transformer with dual task (Transformer) and without dual task (Transformer"). To get more comprehensive comparison, experiments are conducted on the E dataset under different prediction types and horizons. From Table [ we observe that the dual task training is useful for both these two models. The overall improvement ratios of MSE and MAE have reached 8. 8.2% and 5.2% respectively.
CONCLUSION
This paper studies the long sequence time series forecasting problem which has urgent application demands. To tackle this problem, we propose a sparse and efficient attention mechanism called SCAttention which utilizes correlations between segment pairs to discover dependencies and aggre- gate information in time series. Under various experimental settings in different datasets, SCformer with SCAttention mechanism can yield state-of-the-ar prediction performance, which means our SCAttention can extract the dependencies in time series in a better way than the previous sparse- attention family. Besides, to achieve better performance, we design a simple and effective dual task to train our model, which restores the past with the future. This dual task can be used not only in SCformer but also in other predictive models. Future exploration will focus on the combination of SCAttention and other sparse mechanisms.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This research does not involve any ethical issues, All the datasets in this paper are public for research use, without privacy leakage.
REPRODUCIBILLIT STATEMENT
We preprocess all the datasets following the steps in an existing public repository: https:// github. ccm/ z Ittp:///Ioooorrrmmem Informer (Zhou et 2021). We have reimplemented the original Transformer as /http://n/ seas hwrvarrddd...... edu/ 201 8/ 04 0/5/04//3///// /////////////// ////ttttt ion html and modified some modules to get our SCformer model. Our code will be released soon if the paper is accepted.
which is the official PyTorch implementation of
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 2018.
Anastasia Borovykh, Sander Bohte, and Comelis Oosterlee. Conditional time series forecasting with convolutional neural networks. arXiv preprint /rXiv: 1703.0469, 2017.
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Li-Juan Cao and Francis Eng Hock Tay. Support vector machine with adaptive parameters in finan- cial time series forecasting. IEEE Transactions on neurul networks, 1466):100-1511, 2003.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Jerome Connor, Les Atlas, and Douglas R Martin. Recurrent networks and narma modeling. In In Advances in Neural Information Processing Systems, 1992.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735- 1780, 1997.
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. ОTexts, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2014.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Representations, 2019.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 International ACM SIGIR Conference on Research Development Information Retrieval, 2018.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, 2019.
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 2021.
10

---

Under review as a conference paper at ICLR 2022
Helmut Lütkepohl. New Introduction to Multiple Time Series Analysis. Springer-Verlag Berlin Heidelberg, 2005.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In SSW, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual- stage attention-based recurrent neural network for time series prediction. In Twenty-Sixth Inter- national Joint Conference on Artificial Intelligence, 2017.
Stephen Roberts, Michael Osbore, Mark Ebden, Steven Reece, Neale Gibson, and Suzanne Aigrain. Gaussian processes for time-series modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371 371(1984):201 2013.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1191, 2020.
Rajat Sen, Hsiang-Fu Yu, and Inderjit Dhillon. Think globally, act locally: A deep neural net- work approach to high-dimensional time series forecasting. In Advances in Neural Information Processing Systems, 2019.
Huan Song, Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias. Attend and diagnose: Clinical time series analysis using attention models. In Thirty-second AAAI conference artifi- cial intelligence, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems, 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting, arXiv preprint arXiv:2106. 13008, 2021.
Neo Wu, Bradley Green, Xue Ben, and Shawn O' O'Bano... Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprint arXiv:200.008317, 2020a.
Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. In Advances in Neural Information Processing Systems, 2020b.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty- Fifth AAAI Conference on Artificial Intelligence, 2021.
A APPENDIX
A. COMPARE WITH AUTO-CORRELATION IN AUTOFORMER
As shown in Table 6, we conduct experiments on ETTm2 dataset to compare our SCAttention with Auto-Correlane Cooo-Correlation in Autoformer. We fix the input length as 96 and adopt the forecasting setting as multivariate. In order to compare these two attention mechanisms fairly, we train our SCformer model without the dual task and remove the progressive decomposition module in Autoformer. Compared with Corr-Corrllatin, our SCAttention can achieve comparable performance without complicated FFT calculations.
11

---

Under review as a conference paper at ICLR 2022
Table 6: Comparison of Auto-Correlation and SCAttention
Encoder's input length
96
Prediction horizon
336
720
1440
MSE
0.323
0,431
0.552
SCAttention
MAE
0.370
0.425
0.511
MSE
0.339
0.422
0,555
Auto-Correlation
MAE
0.372
0.419
0.496
A.2 EFFECT OF
To explore the effect of nyper-paramette we conduct experiments about the number of heads and encoder layers on ETTh1 dataset. We adopt the forecasting setting as multivariate and predict- 168. shown Table 7 setting the number of heads to is the best. The number of decoder layers M is fixed to 1, we study the effect of the number of encoder layers N on the model. As shown in Table when the number of encoder layers is small, the modeling and predictive ability of SCformer insufficient. While the number of layers is large, the prediction performance is not significantly improved with the amount parameters increasing. Therefore, it is best to set to
Table Effect of the number of heads in SCAttention
n_heads
MAE MSE
4
0.654
0.592
0.671
0.603
6
0.684
0.611
7
0.668
0.590
8
0.661
0.600
Table 8: Effect of the number of encoder layers
layer num
MAE MSE
0.691
0.620
2
0.654
0.592
3
0.658
0.586
4
0.661
0.590
12


Reviews:
[{"review_id": "lGzo4KC2mh1", "reviewer": "Reviewer_5TYz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies the long time series sequence prediction problem with Transformer. The main contribution is proposing a SCAttention mechanism to replace the original attention operation by dividing the whole sequence into equal-length segments and calculating attention among them, which decreases the computation cost significantly. \nGenerally, the paper writing is clear, and the experimental results seem promising. However, I have the following concerns about this paper:\n1. The methodology is quite intuitive, while it is reasonable to decrease the computation cost by segmenting the sequence, it is not well discussed why the performance could be improved significantly?\n2. It is wired that the performance in Table 1 and Table 2 is different from that reported in the Informer paper. Is there any reason or did I misunderstand anything?\n3. In the ablation study about SCAttention, is the SCformer with canonical attention the same as the original transformer? If yes, why the performance is so good compared to other transformer variants?\n4. While experimental shown useful, the novelty of the dual reverse prediction task is limited. Similar ideas have been explored in other time series and traffic forecasting tasks, e.g., use the future 12 steps to predict the historical 12 steps, use bi-directional LSTMs.\n5. finally, it would be better to analyze the model from more aspects, including the influence of encoder/decoder blocks, the number of parameters, analysis and reason about how to select the suitable segment length from Table 4, the variance of the performance, and so on. \n\nAmong these comments, 1, 2, and 3 are my key concerns. I may update the score if they can be adequately solved.  ", "summary_of_the_paper": "The paper studies the long time series sequence prediction problem with Transformer. The main contribution is proposing a SCAttention mechanism to replace the original attention operation by dividing the whole sequence into equal-length segments and calculating attention among them, which decreases the computation cost significantly. ", "summary_of_the_review": "The technical contribution of this paper is marginal and the experimental analysis can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UyYYnvGvdWk", "reviewer": "Reviewer_dkGP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n\n* This paper is well organized.\n* Long sequence time series forecasting is an interesting problem to investigate.\n* The proposed SCformer is technically reasonable.\n\nWeaknesses\n* The overall technical novelty is limited.\n* Several relevant works are not mentioned or compared.\n* Some details of the proposed SCformer are not clear.\n\nThe major concern is over the technical novelty. The key idea of SCformer is very similar to Autoformer, both leverage the series-level correlations. In fact, Autoformer could be easily adapted to capture segment-level correlations.  In addition, the dual-task has been used in many previous works.\n\nAnother concern is that Autoformer and several related works are not compared:\n\n[1] \"Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting.\" Sen, Rajat, Hsiang-Fu Yu, and Inderjit S. Dhillon NeurIPS 2019.\n[2] \"Modeling long-and short-term temporal patterns with deep neural networks.\" Lai, Guokun, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu, SIGIR 2018.\n[3] \"Shape and time distortion loss for training deep time series forecasting models.\" Vincent, L. E., and Nicolas Thome NeurIPS 2019.\n\nThe authors mentioned that their methods differ from Autoformer in the way of correlation computation and\naggregation. It is not clear which one is more efficient in practice.\n\nHow to determine segment length in the experiment?\n\nIn Figure 1, it is not clear why the input of decoder embedding is X_t1:X_t0", "summary_of_the_paper": "This paper presents a Transformer-based model called SCformer to perform long sequence time series forecasting. The key idea is to replace the canonical self-attention with efficient segment correlation attention (SCAttention) mechanism to capture long short-term dependencies. Experiment results on several datasets showed the effectiveness of the proposed method.", "summary_of_the_review": "See above", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Kx936DwL_-T", "reviewer": "Reviewer_pENX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-motivated that the time complexity of the Transformer is quadratic to the number of time steps.\n2. Using the dual task to improve the performance is interesting, and the experimental results show that the dual task could help to improve the results.\n3. The experimenal results could demonstrate the effectiveness of the proposed segment correlation and dual task.\n\nWeakness:\n1. The novelty of the proposed segment correlation is limited, which simply truncates the long time series into small segments and calculates attention scores over the segments.\n2. The time complexity $O(L^2//L^2_{seg})$ is incorrect, which should be $O(L^2//L_{seg})$ instead. $O(L^2//L^2_{seg})$ is the complexity among segments, and the complexity of each segment is not taken into consideration, which is $O(L_{seg})$\n3. For the multivariate setting in table 5, Transformer trained without dual task performs better than Transformer with dual task. The results imply that the dual task might not be very useful to some degree.", "summary_of_the_paper": "This paper introduces a SCFORMER, which replaces the canonical attention in the Transformer with the segment correlation attention. The motivation of using the segment correlation is to reduce the memory usage of the scale-dot product attention of the Transformer. To further improve the performance, the paper proposes a dual task, which use the current time series to predict the past time series. ", "summary_of_the_review": "This paper introduces a SCFORMER to address the problem that the complexity of the Transformer increases quadratically with the length of time series. The essential idea is to segment the input into small segments and calculate attention among the segments rather than each data point. To further improve the performance, this paper introduces an interesting dual task, which uses the current time series as input and will predict the historical time series. However, this the novelty of the segment correlation attention within the SCFORMER is limited and the time complexity is incorrect. Besides, the experimental results in table 5 do not sufficiently support the claim that the dual task could help model to achieve more robust results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K_yLjLzhERW", "reviewer": "Reviewer_SdzH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Exploiting periodical information for long time series forecasting is interesting \n- Paper is fluent and easy to follow.\n\nWeaknesses:\n- Comparisons to earlier work \n- Misses ablation study\n- Limited novelty, dual prediction already exists in bidirectional RNN (LSTM, GRU) time series forecasting and it is not clear how the segmentation is computed, and how it is different than Wu et al 2021 (see below).\n\nConcatenating the segmented time series representations corresponds to concatenating slightly different versions of a short sequence. It can be interesting to see if using the whole sequence is better than using the last segmented sequence. It misses the ablation study of using the longer sequence compared to using the last segmented sequence. \n\nIt misses the comparison to the earlier work on periodical time series forecasting:\n- Cinar, Y. G., Mirisaee, H., Goswami, P., Gaussier, E., & Aït-Bachir, A. (2018). Period-aware content attention RNNs for time series forecasting with missing values. Neurocomputing, 312, 177-186.\n- Wu, H., Xu, J., Wang, J., & Long, M. (2021, May). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Thirty-Fifth Conference on Neural Information Processing Systems.\n\n---\n\n- Providing the equation or the algorithm for the SEG function in Eq. 6 can improve the reproducibility of the approach, and help to better understand the contribution of the paper. \n- The best scores are presented in bold in Table 1 and 2, but not in Table 3, 4, and 5. Can you please show the best scores in bold in Table 3, 4, and 5?\n", "summary_of_the_paper": "This paper studies long time series forecasting by using segment correlation attention. For this, time series is segmented into periodical shorter sequences, and attention is computed on each segment and final representation is formed by concatenating segments’ self-attention representations. ", "summary_of_the_review": "Long time series forecasting using periodical information is interesting. The novelty of the paper is limited as the dual task already exists in bi-directional RNN based models for time series. It misses ablation study of using only the last segmentation instead of the long time series. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a Transformer-based model called SCformer to perform long sequence time series forecasting by computing efficient segment correlation attention. The reviewers think the method lacks novelty and the experiments need a detailed ablation study.
Decision: Reject