SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
Authors: Minhao LIU
Ailing Zeng
Muxi Chen
Zhijian Xu
Qiuxia LAI
Lingna Ma
Qiang Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Time series forecasting
Sample convolution
downsampling

Abstract: One unique property of time series is that the temporal relations are largely preserved after downsampling into two sub-sequences. By taking advantage of this property, we propose a novel neural network architecture that conducts sample convolution and interaction for temporal modeling and forecasting, named SCINet. Specifically, SCINet is a recursive downsample-convolve-interact architecture. In each layer, we use multiple convolutional filters to extract distinct yet valuable temporal features from the downsampled sub-sequences or features. By combining these rich features aggregated from multiple resolutions, SCINet effectively models time series with complex temporal dynamics. Experimental results show that SCINet achieves significant forecasting accuracy improvements over both existing convolutional models and Transformer-based solutions across various real-world time series forecasting datasets. Our codes and data are available at https://github.com/cure-lab/SCINet.

SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
Minhao Liu', Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, Qiang Xu"
CUhk REliable Computing (CURE) Lab. Dept. of Computer Science Egnineering, The Chinese University of Hong Kong *(mhliu, qxu)@cee. cuhk. edu.hk
Abstract
One unique property time series is that the temporal relations are largely pre- served after downsampling into two sub-sequences. By taking advantage of this property, we propose a novel neural network architecture that conducts sample convolution and interaction for temporal modeling and forecasting, named SCINet. architecture. multiple convolutional filters to extract distinct yet valuable temporal features from the downsampled sub-sequences or features. By combin- ing these rich features aggregated from multiple resolutions, SCINet effectively models time series with complex temporal dynamics. Experimental results show that SCINet achieves significant forecasting accuracy improvements over both existing convolutional models and Transformer-based solutions across various real-world time series forecasting datasets. Our codes and data available https://github: com/cure-
Specifically, SCINet is a recursive each layer,
Introduction
Time series forecasting (TSF) enables decision-making with the estimated future evolution of met- rics events, thereby playing a crucial role in various scientific and engineering fields such as healthcare energy management (22 traffic flow 42 and financial investment [10,, to name
few.
There mainly three kinds of deep neural networks used for sequence modeling, and they all applied for time series forecasting [24]: (i). recurrent neural networks (RNNs) [3]; (fi). Transformer- based models 377; and (iii). temporal convolutional networks (TCN) 4
Despite the promising results of TSF methods based on these generic models, they do not consider specialty of time series data during modeling. For example, one unique property of time series is that temporal relations (e.g., the trend and the seasonal components of the data) are largely preserved after downsampling into two sub-sequences. Consequently, by recursively downsampling the time series into sub-sequences, we could obtain a rich set convolutional filters to extract dynamic temporal features at multiple resolutions.
Motivated by the above, in this paper, we propose a novel neural network architecture for time series modeling and forecasting, named sample convolution and interaction network (SCINer), The main contributions of this paper are as follows:
We propose SCINet, hierarchical hawnsaaple-ccmmoov---eeee interactt TSF framework that effectively models time series with complex temporal dynamics. By iteratively extracting and exchanging information at multiple temporal resolutions, effective representation with enhanced predictability can be learned, as verified by its comparatively lower permutation entropy (PE) [16].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Outputs
Outputs
Recurrent
Amertion
Attation
Cony, Layer
Layer
Projection
Cony: Tayer
Layer
Irpuia
Eepurs
Inpats
(n) RNN model
Transformer model
(c) TCN model
Figure 1: Existing sequence modeling architectures for time series forecasting.
We design the basic building block, SCI-Block, for constructing SCINet, which downsamples the input data/feature into two sub-sequences, and then extracts features of each sub- sequence using distinct convolutional filters. To compensate for the information loss during the downsampling procedure, we incorporate interactive learning between the two convolutional features within each SCI-Block.
Extensive experiments on various real-world TSF datasets show that our model consistently outper- forms existing TSF approaches by a considerable margin. Moreover, while SCINet does not explicitly model spatial relations, it achieves competitive forecasting accuracy spatial-temporal TSF tasks.
2 Related Work and Motivation
The time series forecasting problem is defined as: Given long time series X and a look-back window of fixed length T, at timestamp t, time series forecasting is to predict X(+1:+7 = {x++1, (+rr) based on the past T steps X {x,-T+1, ...; Xz), Here, 7 is the length of the forecast horizon, X₁ E Rd is the value at time step I, and d is the number of variates. For simplicity, in the following we will omit the subscripts, and use X and X to represent the historical data and the forecasted data, respectively.
2.1 Related Work
Traditional time series forecasting methods such as the autoregressive integrated moving average (ARIMA) model [ and Holt-Winters seasonal method [14] have theoretical guarantees. However, they are mainly applicable for univariate forecasting problems, restricting their applications to complex time series data. With the increasing data availability and computing power recent years, it is shown that deep learning-based TSF techniques have the potential to achieve better forecasting
accuracy than conventional approaches (24]] 29
Earlier RNN-based TSF methods 31 summarize the past information compactly in the internal memory states that are recursively updated with new inputs at each time step. as shown in Fig. Ta). The gradient vanishing/exploding problems and the inefficient training procedure greatly restrict the application of RNN-based models.
In recent years, Transformer-baat based models 37 have taken the place of RNN models almost all sequence modeling tasks, thanks to the effectiveness and efficiency of the self-attention mechanisms. Various Transformer-basee TSF methods (see Fig. b)) are proposed in the literature [1] 23 38 25
These works typically focus on the challenging long-term time series forecasting problem, taking advantage of their remarkable long sequence modeling capabilities.
Another popular type of TSF model is the so-called temporal convolutional network 33
39 27
wherein convolutional filters are used to capture local temporal features (see Fig. Tc)) The proposed SCINet is also constructed based temporal convolution. However, method has several key differences compared with the TCN model based on dilated causal convolution, as discussed in the following.
2.2 Rethinking Dilated Causal Convolution for Time Series Modeling and Forecasting
The local correlation of time series data reflected the continuous changes within time slot, and convolutional filters can effectively capture such local features. Consequently, convolutional neural networks are explored the literature for time series modeling and forecasting. In particular, dilated causal convolution (DCS) is the current de facto method used in this respect.

---

DCS was first proposed for generating raw audio waveforms in WaveNet 128] Later, simplifies the WaveNet architecture to the so-called temporal convolutional networks (see Fig. T (c)). TCN consists of stack of causal convolutional layers with exponentially enlarged dilation factors, which can achieve large receptive field with just a few convolutional layers. Over the years, TCN has been widely used in all kinds of time series forecasting problems and achieve promising results 39 33 Moreover, convolutional filters can work seamlessly with graph neural networks (GNNs) to solve various -eeporlll TSF problems.
With causal convolutions in the TCN architecture, an output is convolved only with the 7th and earlier elements in the previous layer. While causality should be kept in forecasting tasks, the potential "future information leakage" problem exists only when the output and the input have temporal overlaps. In other words, causal convolutions should be applied only in autoregressive forecasting, wherein the previous output serves as the input for future prediction. When the predictions are completely based on the known inputs in the look-back window, there is no need to use causal convolutions. We can safely apply normal convolutions on the look-back window for forecasting.
More importantly, the dilated architecture in TCN has two inherent limitations:
A single convolutional filter is shared within each layer. Such unified convolutional kernel tends to extract the average temporal features from the data/features in the previous layer. However, complex time series may contain substantial temporal dynamics. Hence, it is essential to extract distinct yet valuable features with a rich set of convolutional filters.
While the final layer of the TCN model has the global view of the entire look-back window, the effective receptive fields of the intermediate layers (especially those close to the inputs) are limited, causing temporal relation loss during feature extraction.
The above limitations of the TCN architecture motivate the proposed SCINet design, detailed in the following section.
Inpuc:
Impur: (X
Ixput: X
Split
SCG-Bleck
SCINe
Fuss
Feren
claak -
Cancal
2
Interattvve Leassngg
SE's SCS-Biiek
SCS Block
McHilock
Crncot
Faxc
Veren
Concet Reelign
SCDNci
Flas
Frrm
Orput: 8
OctpurX X
(0) SCI-Block
(b) SCINet
(c) Stacked SCINet
Figure 2: The overall architecture of Sample Convolution and Interaction Network (SCINet).
3 SCINet: Sample Convolution and Interaction Network
SCINet adopts an encoder-decoder architecture. The encoder is hierarchical convolutional network that captures dynamic temporal dependencies at multiple resolutions with a rich set of convolutional filters. As shown in Fig. 2 (), the basic building block, SCI-Block (Section 3.1.. downsamples the input data or feature into two sub-sequences and then processes each sub-sequence with a set of convolutional filters to extract distinct yet valuable temporal features from each part. To compensate for the information loss during downsampling, we incorporate interactive learning between the two sub-sequences. Our SCINet (Section 3.2) is constructed by arranging multiple SCI-Blocks into binary tree structure (Fig. b)). A distinctive advantage of such design is that each SCI-Block has both local and global views of the entire time series, thereby facilitating the extraction of useful temporal features. After all the features into a new sequence representation and add to the original time series for forecasting with a fully-connected network as the decoder. To facilitate extracting complicated temporal patterns, we could further stack multiple SCINets and apply intermediate supervision to get a Stacked SCINet (Section 3.3 as shown in Fig. 2
operations, we realign the extracted
3

---

3.1 SCI-Block
The SCI-Block (Fig, aa) the basic module of the SCINet, which decomposes the input feature F into two sub-features odd and Feren through the operations of Spliting and Interactive-learning
The Splitting procedure downsamples the original sequence F into two sub-sequences F Feven and F Fodd by separating the even and the odd elements, which are of coarser temporal resolution but preserve most information of the original sequence.
Next, we use different convolutional kemnlss to extract features from F Feven and F Fodd- As the kernels are separate, the extracted features from them would contain distinct yet valuable temporal relations with enhanced representation capabilities. To compensate for potential information loss with downsampling, we propose a novel interactive-learniny strategy to allow information interchange between the two sub-sequences by learning affine transformation parameters from each other. As shown in Fig. (a), the interactive learning procedure consists of two steps.
First, F even and Fodd are projected to hidden states with two different 1D convolutional modules ø and v respectively, and transformed to the formats of exp and interact to the F Feven and F odd with the element-wise product (see Eq. (T). This can be viewed as performing scaling transformation on Feven and add, where the scaling factors are learned from each other using neural network modules. Here, @ is the Hadamard product or element production.
Fod Fodd @ exp(6(FF
Feven Feeen O exp(v((F.aa))
(1)
Fodd = Food + (eee), F Fewen Feven + (FFaa))
(2)
Second, as shown in Eq. (TT), the two scaled features Fiven and Fodd are further projected to another two hidden states with the other two 1D) convolutional modules p and 1), and then added to
or subtracted from F$ even and Fodd odd- The final outputs the interactive learning module are two
updated sub-features Feven and F odd- The default architectures of é, v. p and v) are shown in the Appendix
Compared to the dilated convolutions used in the TCN architecture, the proposed downsample- convolve-interact architecture achieves even larger receptive field at each convolutional layer. More importantly, unlike TCN that employs a single shared convolutional filter at each layer, significantly restricting its feature extraction capabilities, SCI-Block aggregates essential information extracted from the two downsampled sub-sequences that have both local and global views of the entire time series.
3.2 SCINet
With the SCI-Blocks presented above, we construct the SCINet by arranging multiple SCI-Blocks hierarchically and get tree-structured framework, as shown in Fig. (b).
There are 2' SCI-Blocks at the (-th level, where / 1, l,..., , the total number of levels. Within the k-th SCINet of the stacked SCINet (Section 3.3 the input time series X (for k=11 1) or feature vector - - ..... X (for k 1) is gradually down- sampled and processed by SCI-Blocks through different levels, which allows for effective feature learning of different temporal resolutions. In particular, the information from previous levels will be gradually accumulated, i.e., the features of the deeper levels would contain extra finer-scale temporal information transmitted from the shallower levels. In this way, we can capture both short-term and long-term temporal dependencies in the time series.
is the index of the level, and L is
After going through L levels of SCI-Blocks, we rearrange the elements in all the sub-features by reversing the odd-even splitting operation and concatenate them into new sequence representation. It is then added to the original time series through a residual connection [12] to generate new sequence with enhanced predictability. Finally, a simple fully-connected network is used to decode the enhanced sequence representation into X === {x+ xx*). Note that, to mitigate distribution shift in some TSF tasks, before supplying the data in the look-back window to our model, all the data elements are subtracted with the value of the last element, which is added to all the data elements in the forecasting horizon afterwards.
'The selection of the operators in (2) affects the parameter initialization of our model and we show its impact in the Appendix B.3

---

3.3 Stacked SCINet
When there are sufficient training samples, we could stack K layers of SCINets to achieve even better forecasting accuracy (see Fig. 2 (c)), the cost of more complex model structure.
Specifically, we apply intermediate supervision on the output of each SCINet using the ground- truth values, to case the learning of the intermediate temporal features. The output of the k-th intermediate SCINet, X with length is concatenated with part of the input (T= to recover the length to the original input and feeded as input into the (k + 1-th SCINet, where , and K the total number of the SCINets in the stacked structure. The output of the K-th SCINet, XK is the final forecasting results.
3.4 Loss Function
To train stacked SCINet with K (K 2 SCINets, the loss of the k-th prediction results is calculated as the L1 loss between the output of the k-th SCINet and the ground-truth horizontal window to be predicted:
Ls
Σ
(3)
==0
The total loss of the stacked SCINet can be written
K
C.
Le.
(4)
3.5 Complexity Analysis
Thanks to the downsampling procedure, the neurons at each convolutional layer of SCINet have a larger receptive field than those of TCN. More importantly, the set of rich convolutional filters in SCINet enable flexible extraction of temporal features from multiple resolutions. Consequently, SCINet usually does not require downsampling the original sequence to the coarsest level for effective forecasting. Given the look-back window size T. TCN generally requires [log, T'I layers when the dilation factor is 2, while the number of layers SCINet could much smaller than log, T. Our empirical study shows that the best forecasting accuracy is achieved with L in most cases with large T (e.g., 168). As for the number stacks K, our empirical study also shows that K <3 would be sufficient.
Consequently, computational cost of SCINet usually on par with that of the TCN architecture. The worst-case time complexity is O(T log T), much less than that of vanilla Transformer-based solutions: (T
4 Experiments
In this section, we show the quantitative and qualitative comparisons with the state-of-the-art models for time series forecasting. We also present comprehensive ablation study evaluate the effec- tiveness of different components in SCINet. More details on datasets, evaluation metrics, data pre-processingg experimental settings, network structures and their hyper-parameters are shown in the Appendix.
4.1 Datasets
We conduct experiments on 11 popular time series datasets: (1) Electricity Transformer Tempera- ture 42 (ETTh) (2) Traffic (3) Solar-Energ (4) Electricity (5) Exchange-Ree (6) PeMS (PEMS03, PEMS04, PEMS07 and PEMS08). A brief description of these datasets is listed in Table All the experiments these datasets in this section are conducted under multi-variate TSF setting.
To make a fair comparison, we follow existing experimental settings, and use the same evaluation metrics as the original publications 17 26 40 [9 in each dataset.
5

---

Long-term Time Series Forecasting: many real-world applications also require to predict long-term events. Therefore, we conduct the experiments on Exchange Rate, Electricity .Traffic and ETT datasets to evaluate the performance of SCINet on long-term TSF tasks. In this experiment, since they are more Table 1: The overall information of the 11 datasets.
Tralic
Salar-Facryy
Flectriit
Frchungoktic
PFMSIX
PFMC90
PEMSOR
Varanis
862
137
321
358
307
883
170
Tinestga
17,420
696600
17,544
$2,560
26,304
7,588
26,209
16,922
28,224
17,856
Annularit
Ihour
15min
Ihosr
10min
Iboer
Iday
Smin
Smin
Smin
Smin
Stztt time
M1/2016
2/1/2016
1//2015
U//2005
1//2012
1/1/1900
3/1/2012
7232211
511/217
3/1/2012
Tisk
Multi-dap
Mulli-top
Singlookg
Singleskep
Singlestep
Singlossip
Multisdap
Mullistep
Multi-tap
Mull-sipp
Fallow
Trainingh Naliaaioo Tisting
Traiiing Villiing/iiiiiioon Testing:
4.2 Results and Analyses
6 provide the main experimental results of SCINet. We observe that SCINet shows superior performance than other TSF models on various tasks, including short-term, long-term and spatial-temporal time series forecasting.
Table 2
Short-term Time Series Forecasting: evaluate the performance of the SCINet in short-term TSF tasks with other baseline methods on Traffic, Solar-Energy, Electricity and Exchange- Rate datasets. The experimental setting is the same
19 which uses the input length of 168 forecast different
future horizons(3, 6, 12, 24).
As can be seen in Table 2 the proposed SCINet outperforms existing RNN/TCN-based (LSTNet [1)] TPA-LSTM (34], TCN [4, TCN') and Transformer-based [38] 42 37 TSF solutions in most cases, especially for the Solar-Energy and Exchange-Rate datasets. Note that, TCN' denotes variant of TCN wherein causal convolutions are replaced by normal convolutions, and improves the original TCN across all the datasets, which supports claim in Sec. 2.2 Morcover, methods have poor performance in this task. For short-term forecasting, the recent data points are typically more important for accurate forecasting. However, the permutation- methods do not pay much attention to
also observe that
the Transformer-basee
invariant self-attention mechanisms used in Transformer- such critical information. In contrast, the general sequential models (RNN/TCN) can formulate it easily, showing quite competitive results in short-term forecasting.
Table 2: Short-term forecasting performance comparison on the four datasets. The best results are shown in bold and second best results are highlighted with underlined blue font. IMP shows the improvement of SCINet over the best model.
SCThet
mant-mer Em
Intoreer
[3
1.5TNN
TRKUSTM
03955
0.533
377
374
3774
30X
2.155
070215
133311
13WW1
Sake Erwrgs
0,2381
6.738
3/4
3/4
255
36A
30A
304
03352
0.283
1.290
0.383
149012
082550
6.500
0.5122
NIA
0.23%
03338
01335
02358
6315
0211
03722
03335
33455
03445
0339]
013.3
012777
03111)
0.4434
0.330
01562
03381
0.226
05855
0.788
0.060
0.90
0.4823
088000
110710
AS3S5
24
0.4453
0.515
0.833
07557
63863
09035
0.934
0.534
0.585
07911
06550
0.282
0.437
02518
0.2
K,506
0.748
05454
0.145
05602
01534
0883
0.182
03355
3.882
0923
00852
00933
0.885
03083
08925
006550
100955
11822
05335
(1154)
03800
61788
0.88
013755
0.181
013557
(3501
10000
3,75
24
0.067
05278
0.754
03322
02110
035-7
01:61
0.774
0.100
5900
00900
03101
0.007
03111
1000
10133
5.85%
030011
0.997
01422
01382
09473
0228)
03017
110236
03935
1725
Raz
0.031
112551
01033
05054
817M
0.817
11547)
01083
23631
(150]
(1035)
03511
639564
24035
13086
033%
01631
03533
01968
0.122
0.443
10314
(1044)
0.904
1.975
Informer Transfomar schiovad by Autofomer requires vie-prossessed datasets for tmining.
-N/A dencies no pre reeppraseseed dstaset training
* denotes ro-inplementttion.
denotes the variant with normal consolutions.
only compare SCINet with Transformer-based methods [38 popular in recent long-term TSF research.
As can be seen from Table the SCINet achieves state-of-the -he performances in most benchmarks and prediction length settings. Overall, SCINet yields 39.89% average improvements on MSE among the above settings. In particular, for Exchange-Raan Rate, compared to previous state-of-the-ar results, SCINet gives average 65% improvements on MSE We attribute it to that the proposed SCINet can better capture both short (local temporal dynamics)- and long (Irend, seasonality) -temm temporal dependencies to make an accurate prediction in long-term TSF.

---

Table 3: Long-term forecasting performance comparison with Transformer- based models.
Mixlet
SCTNet
Julnloun:: 133
+Fyrdoeme L2]
Inlome [+]]
Trassormer 27]
LogTnun 21
Retormer [5]
IMP
Metne
MSG
MAE
MS8
MAL
34.6
MAE
MSL
95
0.188
0,107
0.325
1.745
1.305
0.752
0.350
0.387
0.368
0.812
1,665
0.829
65306%
Exchange
192
0.344
0.369
1.874
1.151
1,340
0.811
1.188
0,906
64.70%
Rai
356
0.323
0.534
1.943
1.172
1836
1,699
1,081
1.357
64.36%
0.77
2,085
1.206
1,50
68.72%
0.20
0.317
0.385
0.449
Elemiiitty
336
192
0.278
0.662
0.231
0.338
0.376
0.378
0.43
0.443
0.350
720
0.366
0.254
0.361
0.376
0.455
0.283
0.376
0.340
0.420
9,0015
96
0.95
0.613
0,38%
0.367
0.46%
0.712
Traffic
335
192
0.355
0.389
0.616
0.822
0.357
0.382
0.869
0.881
0.477
0.469
0.721
0.420
0.089
0.364
0.685
0.233
0.390
0.908
0.733
0.242
0.420
13.15%
720
0.620
0.994
0.560
0:408
0.895
0.423
0.364
0.473
0.207
0.396
0.787
0.96
0.755
0.475
donotes ro- de-inngleeettttion.
Table 4: Multivariate time-series forecasting results the ETT datasets.
ETTH
ET
ETiml
Methads
Metrics
Hirooon
24
48
1G5
335
720
24
46
165
356
120
24
45
255
072
LogTrass H1
0.757
0.796
1.307
033.5
0.750
1,034
-200
-770
3875
3015
0.412
0:419
11307
1.520
1-55
1.46]
1.797
Reforme:
MAN
0.754
0.910
1.358
1.250
1.520
1,613
1.735
3.500
1.588
2.035
0.607
0.737
10008
0.545
1.184
1.232
LSIMA
MAN
MSI:
0.634
0.500
0.713
0.675
0.867
1.312
0.094
1.424
1.333
19500
03813
1,363
1.671
1.231
0.107
1.674
5.636
1.509
3.655
1.78%
07674
0.611
11
1.392
0.93
1,740
1.134
2.736
1.555
LSTNe: 00
MAE
0.901
1:201
0.990
1788
1.314
1.007
1.309
1.885
7183
1,300
1,457
1.739
1,687
1.50
2.513
1.580
2.30
2.591
3.709
17555
L1700
1.500
1.215
1050
2.702
1.542
2.0%6
1:237
2.863
1.917
Informer
MSE
03577
0.625
0.583
0.911
0.752
0.533
1.120
1215
0.720
1.001
1745
1.515
3.209
3.257
1,473
0.301
0.123
11:00
0.775
0.786
11035
0,925
1.13
"TCN [
MANI
0.589
0.529
0.617
0.682
0.474
0.15
1.276
0.282
01230
03300
0.363
1,573
*yyraferme 15
MAE
0.499
0.520
0.665
0.738
0.782
0.537
0.764
1.557
11307
0.747
0.765
01005
0.353
01400
0.475
03445
0.500
0.996
0,720
Astifomme X
MAE
MNT
2.480
1.00
0.462
0.411
11201
0,492
0111
0.500
11399
0.330
700
1111
0.372
11:00
0.455
11:91
0.476
0.488
11:
11
0.42
0.464
11:00
11:00
0.559
03333
10015
0.564
SCINss
MSE
0.009
0.388
0.361
0.417
0,00%
0.455
8:304
0.314
0.363
8.100
0.303
0:230
0.300
0.342
01365
0
0.115
0.100
0.302
0.230
0.136
01168
0.346
0.376
36.01
denotes implementziion.
We conduct both Multivariate Time-series Forecasting and Univariate Time-series Forecasting on ETT datasets [42). For a fair comparison, we keep all input lengths T the same those of Informer. The results are shown in Table and Table 5
respectively.
Multivariate Time-series Forecasting on ETT: as can be seen from Table compared with RNN-based methods such as LSTMa 2 and LSTnet [19], Transformer- based methods [18
produce better forecasting results. One of the primary reasons is that, RNN-based solutions conduct iterative forecasting and is inevitable to suffer from error accumulation effects. As another direct forecasting method, TCN further outperforms vanilla Transformer-basee stacked convolutional layers allow for more effective woal-ttoggloba temporal relation learning for multivariate time series. It is worth noting that SCINet outperforms all the above models by large margin. Fig. 3 presents the qualitative results on some randomly selected sequences of the ETThl dataset, which clearly demonstrate the capability of SCINet in obtaining the trend and scasonality of time series for TSE.
21
42
methods [18 21 42 because the
Univariate Time-series Forecasting on ETT: in this experimental setting, we bring several strong baseline methods for univariate forecasting into comparison, including ARIMA, Prophet [36], DeepAR [32] and N-Beats (29]. In Table 5 we can observe that N-Beats is superior to other baseline methods in most cases. In fact, N-Beats also takes the unique properties of time series into consideration and directly learns trend and a seasonality model using a deep stack of fully-connected layers with residuals, which is a departure from the predominant architectures, such as RNNs, CNNs and Transformers. Nevertheless, the performance of SCINet is still much better than N-Beats
The newly-proposed Transformer-based forecasting model, Autoformer 38 achieves the second best performance in all experimental settings and also surpasses SCINet in ETTml I when the forecasting horizon is large. This is because, on the one hand, Autoformer focuses on modeling seasonal patterns and conducts self-attention at the sub-series level (instead of the raw data), which is much better in extracting long-term temporal patterns than vanilla Transeormer-based methods. On the other hand, when forecasting long horizons, it is often the trend/seasonal information instead of the temporal dynamics in the look-back window that play the primary role, wherein the advantages of SCINet are not fully exhibited.
7

---

Table 5: Univariate time-series forecasting results the ETT datasets.
ETTHE
FTTEZ
RTTml
Homaon
Honoon
ARIMA
MAR:
0.284
0.434
0.904
0.995
0.766
0.465
0.624
0.995
0.738
0.206
0.106
0.399
0.158
0.697
Proplen 55
MAR
MSE
0.225
01115
0.330
0.105
0.065
1224
1.820
1.30
1.355
1255
0.161
0.399
0.462
0,304
2323
1.03
2,543
219%
3.35
4.644
0.120
0.200
00.35
0.306
0.596
0.794
0.576
2,707
1.174
DeepAR
MSE
100
0.002
0.215
0/335
0028
0008
0.163
0.55
0000
0.:05
0221
0210
0.301
03318
MAB
0.156
00.00
0.315
0.784
0.423
0.310
0.271
0.985
0.418
11432
0.17
0.168
0.254
0.311
0.530
Infiemer
MAA
MSE
0.247
0.095
0.138
0.319
0.346
0.183
0222
0.387
0289
0415
0.240
0338
0.314
03355
0.222
0.389
0.203
0.417
0.20
0.31
02230
0.337
0.208
0069
0.194
0.572
0.59
6481
0,312
0.44
1103
01000
01008
01.
07115
1000
0118
SCIN
MAE
0.127
0.154
0.200
0.214
0.250
0.163
6,220
0
0,329
11:22
0.884
01
0.58
INIP
MSE
10.755
17.01%
101001
$4.70%
153000
11.204
15.0%
13.2000
1.00
3.50
3.25
10-
2.50
2.25
TCM
TCN
Intomer
Infomee
2.50
ScRuat
SCRAL
Aurifurmer
Autifo mer
Ground Truch
G Ground Truch
19
Tuture Time Sheps
Future Time Steps
Sequence 441, Variate
(b) Sequence 1388, Variate
9.6
0.5
0.4
Predictton
0.5
Future
(c) Sequence 2745, Variate 4
(d) Sequence 2753, Variate
Figure The prediction results (Horizon 48) of SCINet, Autoformer, Informer, and TCN on uundomly-selecter sequences from ETThl dataset.
Spatial-temporal Time Series Forecasting: besides the general TSF tasks, there is also a large category of data related spatial-temporal forecasting. For example, traffic datasets PeMS (PEMS03, PEMS04, PEMS07 and PEMS08) are complicated spatial-temporal time series for public traffic network and they have been investigated for decades. Most recent approaches: DCRNN 22 STGCN [1), ASTGCN [II) Graph
(39], STSGCN (35), AGCRN (3) LSGCN 15 and STEGNN 20 use graph neural networks to capture spatial relations while modeling temporal dependencies via conventional TCN or RNN/LSTM architectures. We follow the same experimental settings as the above works. As shown Table 6 these GNN-based methods generally perform better than pure RNN or TCN-based methods. However, SCINet still achieves better performance without sophisticated spatial relation modeling, which further proves the superb temporal modeling capabilities of SCINet.

---

Table 6: Performance comparison of different approaches the PeMS datasets.
Dataness
Metrics
1S7M
-TCN
-TCN
IXCRN
STOCK
ASTOCND:)
Methinds
SISCKN
STRGNN
ACCRE LSGIEN
SCINct
MAF
IMI
MAA
21:33
19.32
15.81
18.18
17.40
17.89
19.35
17.48
16.77
-1500
14.58
6:266
PEMIUB
MAPE
21.33
19.03
18.63
18.91
17.15
19:40
19.31
16.78
16.30
-15.23
14.11
7.36%
RMSE
35.11
33.55
32.24
30.31
30.12
29:66
32.94
29.21
28.34
+28.25
24,05
8,3716
MAE
23:14
23:22
22.31
29.70
22.70
22395
23:45
21.19
19333
1925
21.33
13.93
4.32%
PEMSO4
MAPE
20.33
15:59
14.31
17.32
14.59
16.56
17.29
13.90
13.02
1297
13.15
11.86
9.56%
RMSB
39.59
37.26
36.87
38.12
35.55
35.22
39.70
53.05
21.88
32.30
33.85
30,89
4.40%
MAG
29:58
32:72
30.55
28.30
25.38
28305
26.35
24.26
20.01
22.37
21.19
5:2715
PENSO7
MAPE:
15:33
14:26
13.85
11:06
11.08
13.92
12:12
10.21
9.21
90.12
8.1
3.18%
RMSE
42.54
42.25
41.02
38.54
38.76
42.57
42.78
39,03
35.50
+36.55
34,03
6.59%
MAR
23.30
32.72
31.42
17.50
18.02
1861
19.15
17:13
16.64
1838
17.29
15.72
PEMSOS
MAPE:
15.33
14.03
13.09
11.45
11:40
13.08
12.68
10.36
10.40
10.00
11.20
9.80
2.87%
RMSE
32,06
35.79
34.03
27.53
27.85
28.16
31.05
26.30
26.22
25:22
26.75
24.76
1.52%
dash denotes that methods do not implement on this datnect.
donctes te-implementation or re-training.
denotes the variant with nornul convollttios.
Predictability estimation: inspired by [6 30 we use permutation entropy (PE) 6 to measure the
predictability of the original input and the enhanced representation learnt by SCINet. Time series with lower PE values are regarded as less complex, thus theoretically easier to predic The PE values of the original time series and the corresponding enhanced representations are shown in Table 7
Table 7: Permutation entropy comparison before and after SCINet.
Permutation Entropy
ErThl
Traflic
Solar-Fnerge Energy
Fleciniity
Danseets
Exe-rate
PERSOS
PEMSI
PEMSO
PEMSOR
Parameters
m 1y
Value
Enhanced Representation
Original Input
0.8788
0.7096
0.8832
0.937
0,4739
0,3537
0.8901
0,9489
0.260
0.7836
0,8377
0.96-9
0.8749
09203
0.9148
0.8350
09390
0.8831
(ombeddiin dimension) and (time lagt paramesers used for calculating PE, values selected following
50 L6
As can be observed, the enhanced representations learnt by SCINet indeed have lower PE values compared with the original inputs, which indicates that it is easier to predict the future from the enhanced representations using the same forecaster.
4.3 Ablation studies
To evaluate the impact of each main component used in SCINet, we experiment on several model variants on two datasets: ETTh] and PEMS08.
we first set the number of stacks K and the number of SCINet levels L = 3 For the SCI-Block design, validate the effectiveness of the interactive learning and the distinct convolution weights for processing the sub-sequences, we experiment on two variants, namely w/o. InterLearn and WeightShare. The w/o. InterLearn is obtained by removing the interactive-learning procedure described in Eq. (1) and (IT). In this case, the two sub-sequences would be updated using F Foda= = p(O(F odd)) and F even m7( (F (Feve))). For WeightShare, the modules é, p. vi, and 1 share the same weight.
SCIBlock:
The evaluation results in Fig. 4 shoo that both interactive learning and distinct weights are essential, as they improve the prediction accuracies of both datasets at various prediction horizons. At the same time, comparing Fig. 4a) with Fig. bb). we can observe that interactive learning is more effective for cases with longer look-back window sizes. This is because, intuitively, we can extract more effective features by exchanging information between the downsampled sub-sequences when there are longer look-back windows for such interactions.
SCINet: for the design of SCINet with multiple levels of SCI-Blocks, we also experiment on two variants. The first variant w/o. ResConn is obtained by removing the residual connection from the complete SCINet. The other variant w/o. Linear removes the decoder (i.c., the fully-connected layer) from the complete model. As can be observed in Fig. 4 removing the residual connection leads to a significant performance drop. Besides the general benefit in facilitating the model training, more importantly, the predictability of the original time series is enhanced with the help of residuals. The fully- connectdd layer is also critical for prediction accuracy, indicating the effectiveness of
Pleasee note that PE only a quantitative measurement based on complexity. It would not be proper to say that a time series with lower PE value will be always easier to predict than a different type of time series with a higher PE value because the prediction accuracy also depends on many other factors, such as the available data for training. the trend and seasonality elements of the time series data, well the predictive model.

---

w/o ResCon
w/o ResCon
Mean Absolute Erron
0.0
w/o Linear
w/o Interlearn
Mean Absolute Error
121
w/o InterLearn WeightShare Complete
w/o Linear
WeightShare
Complete
0.70
(a) ET 720)
(b) PEMSO8(H 12)
Figure 4: Component analysis of SCINet on two datasets. Smaller values are better. See Section 4.3
the decoder in extracting and fusing the most relevant temporal information according the given supervision for prediction.
We also conduct comprehensive ablation studies the impact of K (number of stacks) and L (number of levels), and the selection operator in the interact learning mechanism. These results are shown in the Appendix B.2 due to space limitation.
5 Limitations and Future Work
In this paper, we mainly focus on TSF problem for the regular time series collected at even intervals of time and ordered chronologically. However, real-world applications, the time series might contain noisy data, missing data or collected at irregular time intervals, which is referred to as irregular time series. The proposed SCINet is relatively robust the noisy data thanks the progressive downsampling and interactive learning procedure, but it might be affected by the missing data the ratio exceeds a certain threshold, wherein the downsampling- -bsed multi-resolution sequence representation in SCINet may introduce biases, leading to poor prediction performance. The proposed downsampling mechanism may also have difficulty handling data collected at irregular intervals. We plan to take the above issues into consideration in the future development of SCINet.
Moreover, this work focuses the deterministic time series forecasting problem. Many application scenarios require probabilistic forecasts, and we plan to revise SCINet to generate such prediction results.
Finally, while SCINet generates promising results for spatial-temporal time series without explicitly modeling spatial relations, the forecasting accuracy could be further improved by incorporating dedicated spatial models. We plan to investigate such solutions our future work.
Conclusion
In this paper, propose a novel neural network architecture, sample convolution and interaction network (SCINer) for time series modeling and forecasting, motivated by the unique properties of time series data compared to generic sequence data. The proposed SCINet is a hierarchical Nownsaaple--comolve-itteee.
structure with rich set of convolutional filters. It iteratively extracts and exchanges information at different temporal resolutions and learns an effective representation with enhanced predictability. Extensive experiments on various weal-world TSF datasets demonstrate the superiority of our model over sate-of-thhe-n methods.
Acknowledgments and Disclosure of Funding
This work was supported in part by Alibaba Group Holding Ltd. under Grant No. TA2015393, We thank the anonymous reviewers for their constructive comments and suggestions.
10

---

References
Mohammad Taha Bahadori and 7achary Chase Lipton. Telppral- invariance ireegllar healtheare time series. arXiv preprins 19.i::99041220000 2019. [2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In TCLR, 2015.
[3] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in Neural Information Processing Systems, 33:17804-17815, 2020.
[4] Shaojie Bai, J Zico Kolter, and Vladlen Koltun, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. In ICLR, 2018.
2018.
2018.
[5] Shaojie Bai.
[6] Christoph Bandt and Bernd Pompe. Permutation entropy: a natural complexity measure for time series. Physical
review letters, 88(1 740222 2002.
[7] Anastasia Borovykh. Sander Bohte, and Cornelis W Oosteriee. Conditional time series forecasting with convolutional
neural networks. stat, 1050:16, 16, 2017.
[8] George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal
Statistical Society, Series C (Applied Statistics), 17(2):91-109, 1968.
[9] Chao Chen, Karl Pett, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway performance measure- ment system: mining loop detector data. Transportation Research Record, 1748( 1:96-102, 2001.
[10] Pierpaolo D' O'Uoo. Livia De Giovanni, and Riccardo Massari. Trimmed fuzzy clustering of financial time series
based on dynamic time warping. Annals of operations research, 193991)1379-1399 2021.
[11] S. Guo, Youfang Lin. Ning Feng. Chao Song, and Huaiyu Wan. Attention based spatial-temporal graph convolutional
networks traffic flow forecasting. AAAT, 2019.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1355 1780, 1997.
[14) C. Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International Journal of Forecasting. 20:5-10, 2004.
[15] Rongzhou Huang. Chuyin Huang. Yubuo Liu, Genan Dai. and Weiyang Kong. Lsgen: Long short-term traffic
prediction with graph convolutional networks. In I/CAI, pages 2355-2361. 2020.
[16] Yu Huang and Zuntao Fu. Enhunced time series predictability with well-defined structures. Theoretical and Applied
Climatology, pages 1-13, 2019.
[17] Rob Hyndman and Anne Kochler. Another look at measures of forecast accuracy. International journal of forecasting. 22(():679-68) 2006.
[18] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2019.
[19] Guokun Lai, Wei-Cheng Chang. Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR, 2018.
[20] Mengzhang Li and Zhanxing Zhu. -tmpiiall fusion graph neural networks for traffic flow forecasting. In
AAAI 2021.
[21] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang. and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forccusting. In NewrlPS, 2019.
[22] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In ICLR, 2018
[23] Bryan Lim, Sercan Ô Ank,, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretahle
multi-horizon time series forecasting. International Journal of Forecasting, 2021.
[24] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A. 379(2194):20200222
2021.
[25] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li. Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low- complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2021.
[26] Spyros Makridakis, Allan Andersen, Robert Carbone, Robert Fildes, Michele Hibon, Rudolf Lewandowski, Joseph Newton, Emanuel Parzen, and Robert Winkler. The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of forecasting, 1(2):111-153, 1982.
(27) Nam Nguyen and Brian Quanz. Temporal latent auto autodencoder: A method for probabilistic multivariate time series forecasting. In AAAI, 2021.
[28] Aaron van den Oord, Sander Dieleman, Heiga 7en, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner,
Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. 2016.
[29] Boris Oreshkin, Dmitri Carpo, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In ICLR, 2020.
[30] Frank Pennekamp, Alison C Iles, Joshua Garland, Georgina Brennan, Ulrich Brose, Ursula Gaedke, Ute Jacob, Pavel Kratina, Blake Matthews. Stephan Munch, ct al. The intrinsic predictability of ccologicul time series and its potential to guide forecasting, Ecological Monographs, 89(2yc01359, 2019.
[31] Syama Sundar Rangapuram, Mathias W Szeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski.
Deep state space models for time series forecasting. NeurIPS, 2018.
[32] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with
autoregressive recurrent networks. International Journal of Forecasting, 1811)118111111, 2020.
11

---

[33] Rajut Sen, Hsiang-Fu Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. volume 32, 2019.
[34] Shun-Yas Yao Shih, Fan- Ken-Ken Sun, and Hung yi Lec. Temporal pattern attention for multivariate time series forccasting.
Machine Learning, pages 1-21, 2019.
[35] Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 914-921, 2020.
[36] Sean J. Taylor and Benjamin Letham. Forecasting scule. The American Statistician, 72:37 45, 2018.
[37] Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and
Mia Polosukhin. Attention is all you need. In NewrPPS, 2017.
[38] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with
auto-correlation for long-term series forecasting. In NeurIPS, 2021.
[39] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and C. Zhang. Graph wavenet for deep spatial-temporal
temporal graph
modeling. In WCAA, 2019.
[40] Jiehui Xu, Jianmin Wang. Mingsheng Long, al. Autoformer: Decomposition transformers with auto-correlation
for long-term series forecasting. Advances Neural Information Processing Systems, 34, 2021.
[41] Ting Yu. Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: deep learning
framework for traffic forecasting. In IJCAI, 2018.
[42] Hao- Yi Zhou, Shanghang Zhang, Jicqi Peng. Shuai Zhang, Jianxin Li. Hui Xiong, and Wancui Zhang. Informer:
Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section 5
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (cither in the supplemental material or URL)? [Yes] See the Appendix and link in abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releusing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Detail seen in Appendix.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "sWXBrdDH2sT", "reviewer": "Reviewer_pviL", "summary": "The article proposes a novel method based on hierarchical downsample-convolve-interact architecture, named SCINet. The article points out that the temporal relation of time-series data is maintained even after downsampling, and argues that multiple convolution filters extract rich features from downsampled sub-sequences. The SCI block, the basic unit of SCINet, utilizes different convolution kernels, an inter-learning approach is applied, has residual connections, and linear enhancement is applied. The proposed method was evaluated on 9 types of popular time-series dataset and compared with the latest methods. Performance degradation occurred in the ETTh1 [37] dataset in the long-term forecasting task, but a significant improvement in overall performance was verified. ", "questions": "1. Is the improved performance achieved by stacking SCI-blocks, or is it due to the residual connection between SCI-blocks, or is it due to the SCI-block itself? Considering that the comparative methods also handle downsampled features by multiple convolution, it is necessary to highlight the cause of the performance improvement.", "limitations": "The authors point out that if the time-series obtained in the real-world are noisy or greatly affected by missing data, the method can lead to degradation in performance.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(strength)\nThe proposed method has achieved improved performance compared to the latest deep learning-based time-series modeling methods. The proposed method is technically sound and has a sufficient amount of validation.\n(weakness)\nThe article mixes several ideas, including SCI-block, hierarchical structure, downsampling, and residual connection. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YzxY2J1Mku", "reviewer": "Reviewer_B5XJ", "summary": "This paper devises a new architecture, SCINet, for the time-series forecasting problem. To extract features, SCINet mainly adopts convolutions on sub-series with different resolutions, which is accompanied by additional operations to ensure the interactions among the subsequences. The proposed architecture can also be stacked to improve forecasting accuracy. SCINet is compared with other convolutional models and transformer-based models on short-term/long-term/spatial-temporal time series forecasting tasks. ", "questions": "1. In line 109, the authors state that “each SCI-Block has both local and global views of the entire time series”. I would appreciate it if the authors could elaborate more on how a single SCI-Block can attain the global view of entire time series.\n2. I would also appreciate it if the authors could explain more about the claim – “the proposed downsample-convolve-interact architecture achieves an even larger receptive field at each convolutional layer” in line 139.\n3. As SCI-Net is claimed to “capture both short-term and long-term temporal dependencies in the time series”, I think it would be helpful to show its superiority by comparing it with Transformer-XL (which is good at preserving long-term information) empirically. \n4. Besides, as SCI-Net and Pyraformer both adopt hierarchical architecture for time-series forecasting task, I think it’s worth adding the latter in the experiment section for comparison. \n5. In the Table 3&4 of the Supplementary Material, the authors report the hyperparameters used by the SCI-Net. I notice that the batch size and learning rate significantly vary both within a single dataset / across the datasets. I’m wondering whether the performance of SCI-Net is highly sensitive to these training strategies.\n6. In Table 3 of the main text, I notice that the paper uses the performance reported in the Autoformer and Informer paper. However, Autoformer and Informer adopt a significantly different training strategies (batch size, learning rate, and training epochs) compared with the SCI-Net. More concretely, Autoformer and Informer use constant batch size and same initial learning rate, respectively; while SCI-Net tunes these hyperparameters for every setting. Moreover, Autoformer and Informer are only trained for 10 and 8 epochs, respectively; while SCI-Net is trained for 150 epochs. Therefore, I’m wondering whether Table 3 offers a fair comparison and whether SCI-Net is efficient in terms of training time and generalizable to different datasets without intensive hyperparameter tuning.\n7. It would be helpful if the authors could provide how many times each experiment is repeated and the error bar (e.g., standard deviation) of the reported results.\n", "limitations": "The limitations have been covered properly by the authors in the corresponding section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:    \n1. The paper is well written and easy to follow.\n\n2. Though convolution is not uncommon in TSF models, the paper utilizes the special properties of time series data and proposes the downsampling and interaction operations, which leads to good empirical performance. \n\nWeaknesses:  \n1. More empirical comparison with other related models would be better to support the superiority of the SCI-Net. (Please refer to Questions 3-4 for details)\n\n2. More explanations on the training strategies adopted would be helpful to prove that the experiment section gives a fair comparison with other works. (Please refer to Questions 5-7 for details)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MtxPH0sKqqi", "reviewer": "Reviewer_gmoU", "summary": "The paper proposes to progressively downsampling the sequences into two sub-sequences, and then conduct convolution and interaction operations to study complex temporal dynamics of time series data. Experiments on various time series datasets confirm its significant forecasting accuracy improvements over both existing convolutional and Transformer-based models.", "questions": "Please refer the weaknesses. My biggest concern is on the anonymous.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The core idea of progressively downsampling the sequences into two sub-sequences to study temporal dynamics is novel.\n2. Extensive experiments are performed to confirm the effectiveness.\n3. Source code is provided for reproducibility.\n\nWeaknesses:\n1. The code (https://anonymous.4open.science/r/SCINet-2588) is not anonymous. The following information is included:\n     If you find this repository useful for your work, please consider citing it as follows:\n     @article{...author={L**, ...},...\n     }.  (checked on 2022/07/10. To keep anonymous, only the first letter of the first author's last name is shown here.)\n2. For the experiment results in Table 2 and 4, the N/A and - could be filled by re-implementation or re-training.\n3. It seems that SCINet needs more memory usage even compared to Transformer based on Table 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DG7dXwPcJ5", "reviewer": "Reviewer_9stN", "summary": "In this paper, the authors proposed a novel time-series dependency modeling framework for the long/short-term forecasting task. Specifically, the time series is downsampled into different sub-sequences and resolutions and jointly modeled to produce an enhanced overall representation for the forecasting task.", "questions": "1. The paper is based on an assumption: \"temporal relations are largely preserved after downsampling into two sub-sequences.\" However, I don't think this holds true in many real-world scenarios. For example, downsampling may result in two sequences with different trends for a volatile time series, such as stock price. Also, some citations are expected to back this assumption. \n2. The overall contribution is a little bit incremental. Stacked blocks and the tree-like design in the SCINet are not something new, though the interactive learning between two sub-sequences is interesting.\n3. Page 4, line 155. Can the authors provide more details about how the elements in all the sub-features are ordered? \n4. Page 4, line 159. It's noticed that the authors used the fully-connected layers as the prediction layer. However, some constraints after the fully-connected layers are necessary to deal with the real-world situation. For example, for traffic forecasting, only a positive value is expected. \n5. Page 5, line 165. Which part of the X is concatenated to $\\hat{X}^k$?\n6. The main results should include an error bar or the standard deviation. Also, the model should be evaluated by both absolute-value-based and percentage-based metrics. \n7. The motivation for adopting intermediate supervision is not well supported. It would be better if the authors could provide more details or demonstrate it through the ablation study. ", "limitations": "The authors partially pointed out this work's limitations regarding missing data and irregular time series. Other potential limitations are listed in the questions above. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The idea of interactive learning between downsampled sub-sequences is interesting.\n+ Overall, the paper is well organized and written. \n\nWeakness\n+ The motivation of this paper is built on the assumption that temporal relations are largely preserved after downsampling into two sub-sequences which may not hold true in many real-world situations such as volatile time series in the financial domain. \n+ In the main result, it's better to include the error bars instead of simply providing the prediction results to report the model performance.\n+ The design motivation of some critical points, such as intermediate supervision, is not clearly explained or demonstrated in the experiments. \n+ The overall innovation is a bit incremental. The major contribution may come from the design of the interactive learning of two sub-sequences. .", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept