MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting
Authors: Huiqiang Wang
Jian Peng
Feihu Huang
Jince Wang
Junhui Chen
Yifei Xiao
Venue: ICLR 2023 notable top 5% (2023)
Keywords: long-term forecasting
local and global context
multi-branch architecture
different potential patterns.

Abstract: Recently, Transformer-based methods have achieved surprising performance in the field of long-term series forecasting, but the attention mechanism for computing global correlations entails high complexity. And they do not allow for targeted modeling of local features as CNN structures do. To solve the above problems, we propose to combine local features and global correlations to capture the overall view of time series (e.g., fluctuations, trends). To fully exploit the underlying information in the time series, a multi-scale branch structure is adopted to model different potential patterns separately. Each pattern is extracted with down-sampled convolution and isometric convolution for local features and global correlations, respectively. In addition to being more effective, our proposed method, termed as Multi-scale Isometric Convolution Network (MICN), is more efficient with linear complexity about the sequence length with suitable convolution kernels. Our experiments on six benchmark datasets show that compared with state-of-the-art methods, MICN yields 17.2% and 21.6% relative improvements for multivariate and univariate time series, respectively.

Published as a conference paper at ICLR 2023
MICN: MULTI-SCALE LOCAL AND GLOBAL CONTEXT MODELING FOR LONG-TERM SERIES FORECASTING
Huiqiang Wang', Jian Peng', Feihu Huangl "ooung"",Jinee Jince Wang', Junhui Chen', Yifei Xiao
"College of Computer Science, Sichuan University, China
School of Computer Science and Engineering.
University of Electronic Science and Technology of China, China
{wanghuiqie anagg...iian....... wjc, chenjunhuileee @stu scu, edu. cn, (jianpeng, huangfh) @scu. edu. en, xyf xyf19890uestc. edu.
ABSTRACT
Recently, Transformer-bast based methods have achieved surprising performance in the field of long-term series forecasting, but the attention mechanism for computing global correlations entails high complexity. And they do not allow for targeted modeling of local features CNN structures do. To solve the above problems, we propose to combine local features and global correlations to capture the overall view of time series (e.g., fluctuations, trends). To fully exploit the underlying information the time series, a multi-scale branch structure is adopted to model different potential patterns separately. Each pattern is extracted with down-sampled convolution and isometric convolution for local features and global correlations, respectively. In addition being more effective, our proposed method, termed as Multi-scale Isometric Convolution Network (MICN), is more efficient with linear complexity about the sequence length with suitable convolution kernels. Our experiments on six benchmark datasets show that compared with state-of-the methods, MICN yields 17.2% and 21 ..6% relative improvements multivariate and univariate time series, respectively. Code is available at wtttts:://// thub com/wanghg21/MICCN
INTRODUCTION
Researches related to time series forecasting are widely applied in the real world, such as sensor network monitoring (Papadimitriou Yu., 2006), weather forecasting, economics and finance (Zhu & Shasha 2002), and disease propagation analysis (Matsubara et al. 2014) and electricity forecasting. In particular, long-term time series forecasting is increasingly in demand in reality. Therefore, this paper focuses on the task of long- long-term forecasting. The problem to be solved is to predict values X, based on observations from a historical period: Xx and T > I.
for a future period: X,+1,
As a classic CNN-based model, TCN (Bai et al. 2018) uses causal convolution to model the temporal causality and dilated convolution to expand the receptive field. can integrate the local information of the sequence better and achieve competitive results in short and medium-term forecasting (Sen) al. 2019) (Borovykh et al. 2017). However, limited by the receptive field size, TCN often needs many layers to model the global relationship of time series, which greatly increases the complexity of the network and the training difficulty of the model.
Transformers Vaswaan et al.) 2017) based on the attention mechanism shows great power in sequential data, such as natural language processing (Devlin et al. 2019) (Brown et al. 2020), audio processing (Huang et al.] 2019) and even computer vision (Dosovitskiy al. 2021) (Liu et al.) 2021b). It has also recently been applied in long-term series forecasting tasks (Li et al. 2019b) (Wen et al. (022)) and can model the long-term dependence of sequences effectively, allowing leaps and bounds in the accuracy and length of time series forecasts (Zhu & Soricut 2021 (Wu et al. 2021b) (Zhou al. 2022). The learned attention matrix represents the correlations between different time points of the sequence and can explain relatively well how the model makes future predictions based past information. However, has quadratic complexity, and many of the computations

---

Published as a conference paper at ICLR 2023
between token pairs are non-essential, so it is also an interesting research direction to reduce its
computational complexity. Some notable models include: LogTrams (Li et al. 2019b), Informer
(Zhou et al., 2021), Reformer (Kitaev et al. 22200) Autoformer Wu et al. (20216) Pyraformer (Lu
et al. 2021a) FRDformer (Zhou et al. 2022).
However, special sequence, time series has not led unified modeling direction so far. In this paper, we combine the modeling perspective of CNNs with that of Transformers to build models from the realistic features of the sequences themselves, i.e., local features and global correlations. Local features represent the characteristics of sequence over a small period T, and global correlations are the correlations exhibited between many periods Ti, ... 1, In. For example, the temperature at a moment is not only influenced by the specific change during the day but may also be correlated with the overall trend of a period (e.g., week, month, etc.). We can identify the value of a time point more accurately by learning the overall characteristics of that period and the correlation among many periods before. Therefore, a good forecasting method should have the following two properties: (1) The ability to extract local features to measure short-term changes. (2) The ability to model the global correlations to measure the long-term trend.
Based on this, we propose Multi-scale Isometric Convolution Network (MICN). We use multiple branches of different convolution kernels to model different potential pattern information of the sequence separately. For each branch, extract the local features of the sequence using local module based on downsampling convolution, and on top of this, we model the global correlation using& global module based on isometric convolution. Finally, Merge operation is adopted to fuse information about different patterns from several branches. This design reduces the time and space complexity to linearity, eliminating many unnecessary and redundant calculations. MICN achieves state-of-the-an accuracy on five real-world benchmarks. The contributions are summarized as follows:
We propose MICN based on convolution structure to efficiently replace the self-attention, and it achieves linear computational complexity and memory cost.
We propose multiple branches framework to deeply mine the intricate temporal patterns of time series, which validates the need and validity for separate modeling when the input data is complex and variable.
We propose a local-global structure implement information aggregation and long-term dependency modeling for time series, which outperforms the self-attention family and Auto-correlation mechanism. We adopt downsampling one-dimensional convolution for local features extraction and isometric convolution for global correlations discovery.
Our empirical studies show that the proposed model improves the performance of state-of- the-art method by 17.2% and 21.6% for multivariate and univariate forecasting, respectively.
2 RELATED WORK
2.1 CNNS AND TRANSFORMERS
Convolutional neural networks (CNN) are widely used in computer vision, natural language process- ing and speech recognition (Sainath et al. 2013) (Li et al. 2019a) (Han et al. 2020). It is widely believed that this success due to the use of convolution operations, which can introduce certain inductive biases, such as translation invariance, etc. CNN-based methods are usually modeled from the local perspective, and convolution kernels can very good at extracting local information from the input. By continuously stacking convolution layers, the field of perception can be extended to the entire input space, enabling the aggregation of the overall information.
Transformer (Vaswani al., 2017) has achieved the best performance in many fields since its emergence, which is mainly due to the attention mechanism. Unlike modeling local information directly from the input, the attention mechanism does not require stacking many layers extract global information. Although the complexity is higher and learning is more difficult, is more capable of learning long-teu long-term dependencies (Vaswani et al. 2017].
Although CNNs and Transformers are modeled from different perspectives, they both aim to achieve efficient utilization of the overall information of the input. In this paper, from the view of combining
2

---

Published as a conference paper at ICLR 2023
the modeling principles of CNNs and Transformers, consider both local and global context, extract local features of data first, and then model global correlation on this basis. Furthermore, our method achieves lower computational effort and complexity.
2.2 MODELING BOTH LOCAL AND GLOBAL CONTEXT
Both local and global relationships play an important role in sequence modeling, Some works have been conducted study how to combine local and global modeling into unified model achieve high efficiency and interpretability. Two well-known architectures are: Conformer (Gulati et al. 2020) and Lite Transformer (Wu et al. 2020).
Conformer is variant of Transformer and has achieved state-of-the-arr performance in many speech applications. adopts the attention mechanism to learn the global interaction, the convolution module to capture the relative-offset-basee local features, and combines these two modules sequentially. However, Conformer does not analyze detail what local and global features are learned and how they affeet the final output. There is also explanation why the attention module is followed by convolution module. Another limitation of Conformer is the quadratic complexity with respect to the sequence length due self-attention.
Lite Transformer also adopts convolution to extract local information and self-attention to capture long-term correlation, but it separates them into branches for parallel processing. A visual analysis of the feature weights extracted from the two branches is also presented in the paper, which can provide good interpretation of the model results. However, the parallel structure of the two branches determines that there may be some redundancy in its computation, and it still has the limitation of quadratic complexity.
Whether the convolution and self-attention are serialized to extract local and global relationships step by step or in parallel, inevitably results in quadratic time and space complexity. Therefore, this paper, we propose new framework for modeling local features and global correlations of time series along with a new module instead of attention mechanism. We also use the convolution operation to extract its local information and then propose isometric convolution to model the global correlation between each segment the local features. This modeling method not only avoids more redundant computations but also reduces the overall time and space complexity to linearity with respect to the sequence length.
3
MODEL
3.1
MICN FRAMEWORK
The overall structure of MICN is shown in Figure I The long time series prediction task to predict future series of length O based as input - predict - O. where is much larger than 1. Inspired by traditional time series decomposition algorithms (Robert al.) 1990) (Wu et al., 2021b),
past series of length 1, which be expressed
design multi-scale hybrid decomposition (MHDecomp) block to separate complex patterns of input series. Then we use Seasonal Prediction Block to predict seasonal information and Trend-cyclical Prediction Block to predict trend-cyclical information. Then add the prediction results up to get the final prediction Ypred We donate d as the number of variables in multivariate time series and D as the hidden state of the series. The details will be given in the following sections.
X,
Embedding
Multiscale Highidd Desorpettion
ingut:
Sessonal Prediction Blocc
Ypre
Regression
Y,
Trend-cyclical Prediction
Figure 1: MICN overall architecture.
3

---

Published as a conference paper at ICLR 2023
3.2 MULTI-SCALE HYBRID DECOMPOSITION
Previous series decomposition algorithms (W et al. 202 adopt the moving average to smooth out periodic fluctuations and highlight the long-term trends. For the input series X Rlxa the process is:
X (vddool(
X, -
where: X,,, E Rlxd denote the trend-cyclical and seasonal parts, respectively. The use of the Avgpool(-) with the padding operation keeps the series length unchanged. But the parameter kernel of the Argpool(-) is artificially set and there are often large differences in trend-cyclical series and seasonal series obtained from different kernels. Therefore, we design multi-scale hybrid decompo- sition block that uses several different kernels of the Avgpool(-) and can separate several different patterns of trend-cyclical and seasonal parts purposefully. Different from the MOEDecomp block of FEDformer (Zhou et al., 2022), we use simple mean operation to integrate these different patterns because we cannot determine the weight of each pattern before learning its features. Correspondingly; we put this weighting operation in the Merge part of Seasonal Prediction block after the representation of the features. Concretely, for the input series X E R!xd the process is:
X, = mean AvgPool wo " Avg (ugdaoolPP (aara X, X
where X, E Rexd denote the trend-cyclical and seasonal part, respectively. The different kernels are consistent with multi-scale information in Seasonal Prediction block. The effectiveness is demonstrated experimentally in Appendix B.T
3.3 TREND-CYCLICAL PREDICTION BLOCK
Currently, Autoformer Wu et al. 2021b) concatenates the mean of the original series and then accumulates it with the trend-cyclical part obtained from the inner series decomposition block. But there is no explanation of this and no proof of its effectiveness. In this paper, we use a simple linear regression strategy to make a prediction about trend-cyclical, demonstrating that simple modeling of trend-cyclical is also necessary for non-stationary series forecasting tasks (See Section 4.2) Concretely, for the trend-cyclical series X, Rlxd obtained with MHDecomp block, the process is:
reree regression(X)
use MICN-regre to represent MICN model with this trend-cyclical prediction method. where y,rere E R0xd denotes the prediction the trend part using the linear regression strategy. And
For comparison, we use the mean X, to cope with the series where the trend-cyclical keeps constant:
y.mean (X))
where Y.nen E Rxx denotes the prediction of the trend part. And we use MICN-mean to represent MICN model with this trend-cyclical prediction method.
3.4 SEASONAL PREDICTION BLOCK
As shown in Figure the Seasonal Prediction Block focuses on the more complex seasonal part modeling. After embedding the input sequence X,, we adopt multi-scale isometric convolution to capture the local features and global correlations, and branches of different scales model different underlying patterns of the time series. We then merge the results from different branches to complete comprehensive information utilization of the sequence. It can be summarised as follows:
Xemb__
Soneedding(Conncee (xero)))))
y: Xemb
(5)
Y,J MI)( LE (1, Y, (rroecctt((roeeciion(...
where Xyer R0dd denotes the placeholders filled with zero, and X.mb R(++0)xD denotes the embedded representation of X, Yu R(+00)DD represents the output of / th multi-scale isometric
4

---

Published as a conference paper at ICLR 2023
Maltsoalk
PE
Mergs
Feol Forward
Add
VB
Local -
Global
Xarra
TFB
Nx
Embedding
MIC
Figure 2: Seasonal Prediction Block.
Faomstric
Tash
CoeeidFraas
Jaypoo
Dounsnngling
Despout
Convolution
Desposi
Upaamphing
Despost
local
Figure 3: Local-Glod Global module architecture.
convolution (MIC) layer, and Y, E ROxd represents the final prediction of the seasonal part after a linear function Projection with Y, R(/-O)x and Truncate operation. The detailed description of Embedding and MIC will be given as follows.
The decoder of the latest Transformer-basee models such as Informer (Zhou et al. 2021), Autoformer (Wu et al., 2021b) and FRDformer (Zhou al., (022) contain the latter half of the encoder's input with the length and placeholders with length O filled by scalars, which may lead redundant calculations. To avoid this problem and adapt prediction length O we replace the traditional encoder-decoder style input with simpler complementary O strategy. Meanwhile, follow the setting of FEDformer and adopt three parts to embed the input. The process is:
Embedding
X (TEE + PE VE (Conncat (X, (ser)))
(6)
where Xemb E Rl+00)xD TFE represents time features encoding (e.g., MinuteOfHour, HourOlDay, DayOrWeek, DayOIMonth, and MonthOrYear), PE represents positional encoding and VE represents value embedding.
Multi-scale isometrie (OM)) Layer MIC layer contains several branches, with dif- ferent scale sizes used to model potentially different temporal patterns. In each branch, shown in Figure 3 the local-global module extracts the local features and the global correlations of the sequence (See Appendix [.. for more detailed description). Concretely, after obtaining the cor- responding single pattern by avgpool, the local module adopts one-dimensional convolution to implement downsampling. The process is:
Y₂ =Y,=-11111
Ilocul, Convld (Augpool (emme
where Y.t-1 denotes the output (I (1-1) th MIC layer and Yg.0 = Xemb i E { 8 ....} denote the different scale sizes corresponding the different branches in Figure
For Convld,
stride kernel i, which serves compression of local features. ylocal.1 R result obtained by compressing local features, which is short sequence.
(140) xD represents the
And furthermore, the global module is designed to model the global correlations of the output of the local module. A commonly used method for modeling global correlations is the self-attention mechanism. But in this paper, we use a variant of casual convolution, isometric convolution, as an alternative. As shown in Figure isometric convolution pads the sequence of length S with

---

Published as a conference paper at ICLR 2023
placeholders zero of length S 1. and its kernel is equal to S It means that convolution kernel to measure the global correlation of the whole series. The current generative prediction approach is to add placeholder to the input sequence, which has actual sequence information the second half. The Isometric Convolution can enable sequential inference of sequences by fusing local features information. Moreover, the kernel of Isometric convolution is determined by all the training data, which can introduces global temporal inductive bias (translation equivariance, etc.) and achieve better generalization than self- (the correlations are obtained from the product between different elements). Meanwhile, we demonstrate that for shorter sequence, isometric convolution superior self-attention. The detailed experiments of the proof are in Appendix B.3 And to keep the sequence length constant, we upsample the result of the isometric convolution using transposed convolution. The global module can be formalized as follows:
large
Y Norm(Y, xd
( Dropout Dropout(Tankkssoneeriiii (Isoeerrrrii (Y (locaa.)))
y x
Norm(Y Dropout Con s. hernet=i))),
where ylucul,i 53
E
R
11+0
xD denote the result after the global correlations modeling. Yu,-1 is the output y.hobul, R(1+0)xD represents the result of this pattern (i.e., this branch).
of / MIC layer.
x.1
Isometric Convolution
Masked self-attention
Figure Isometric Convolution architecture vs. Masked self-attention architecture
Then propose use Conv2d to merge the different patterns with different weights instead of the traditional Concat operation. The validity of this weighting approach verified Appendix B.4 The process is:
ymerge
(Conv2d (olonyd(yyoooo yggobbl
iE
1 4 (4'8...))))
Y1 Norm (Y FeedFoorwrdd (mmerrg)))
where Ys R(I+0)xD represents the result of - th MIC layer.
To get the final prediction of the seasonal part, we use the projection and truncate operations:
Y, T Truncate( (rotiooon)(Y...))
(10)
where Y.N E R(++0)xD represents the output of N-th MIC layer, and Y, E Rxdd represents the final prediction about the seasonal part.
4 EXPERIMENTS
Dataset To evaluate the proposed MICN, conduct extensive experiments on six popular real- world datasets, covering many aspects of life: energy, traffic, economics, and weather. We follow standard protocol (Zhou al. 202T) and split all datasets into training, validation and test in chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets. More details about the datasets and implementation
described in Appendix A. and A.2
Baselines We include four transformer- based models: FEDformer (Zhou et al. 2022), Autoformer
eW al. 2021b), Informer (Zhou et al. (02T), LogTrans (Li et al. 2019b), two RNN-based models: LSTM (Hochreiter & Schmidhuber 1997), LSNNet (Lai et al. 2018b) and CNN-based model TCN (Bai et al. 2018)) as baselines. For the univariate setting, we mainly compare transformer-bat based models. For the state-of-the-art model FEDformer, compare the better one (FEDformer-f).

---

Published as a conference paper at ICLR 2023
Table 1: Multivariate long-term series forecasting results with input length / 96 and prediction length O E (96, 192, ]20] (for ILI, the input length / = 36). A lower MSE or MAE indicates a better prediction, and the best results are highlighted in bold.
Mettods
MiCN-osare
FEDGomer
Natooornuer
Difoemes
LogTrus
LSTNE
LSTM
TCN
MAE
MSE
MAE
MSE
36AE
MSE
MAE
XISE
MAE
96
0.179
0.275
0.263
0.287
0.203
0.267
0.255
0.339
0.365
0.453
0.763
0.642
3.142
1.365
2,041
1,03
5,041
1.330
192
0.307
03.375
6.262
0.326
0.269
0.318
0.281
0.340
0.553
0.563
0.989
0.757
3.154
1.369
2.349
1.182
5,022
1.339
376
0.325
01:385
0.185
0,353
0.325
0.356
0.319
1.373
1.383
03887
1.354
0.872
31600
1.309
2.506
1.338
3,005
1.348
720
0.407
0.421
0,415
0.422
0.419
3,379
1.355
3,043
1.338
2.720
1.287
5.135
1.354
6.306
0.201
0.274
0.365
0.258
0.985
0.313
192
01117
0.2365
0.201
0.315
0.222
01354
0.296
0.386
0.266
0.366
0.725
0
0.521
336
0,193
0.364
0.219
0.525
0.214
0.329
0.231
0.388
0.300
0.394
0.290
0.360
0.828
0.723
0.49
0.475
1.003
0.834
720
0.221
0.321
0.224
0.332
0.245
0.355
0.254
0.361
0.373
0.439
0.283
0.376
0.857
0.611
0.900
0.814
1.439
0.784
Dichange
0.143
0.278
0.197
0.223
0:447
0.251
0.965
0.812
1.551
1,65
1453
1040
3.004
1.422
0.71
11300
1.20
0320
1.000
0,500
0.841
01941
7.4%
1.137
2.345
124
1.45
1231
1.4%
Thathe
192
96
0.519
0.537
0.315
0.345
0.575
0.581
0.344
0.34
0,50
8504
0.313
0.366
0.63
0615
11283
0.384
0.96
0.719
0.77
0.391
0.084
0.390
0.364
1151
1.107
0.06
0.085
0241
0345
0:45
0-55
1444
1.43
0.794
0.764
336
0.22
0.363
0.622
0.337
0377
0.520
0.73
0.488
1226
1479
0.799
0.060
01400
0.475
0.504
Wiatter
0.217
0.2%6
0.356
0.310
0.364
0.458
0.490
0G15
0.599
192
0.220
0.381
0.346
0.317
0.276
0.356
0.307
0.567
0.516
0.544
0.53
0.389
0.560
0.565
0.16
0.435
0.29
0.500
376
0..7%
0.331
0.295
0.355
0.354
0.361
0.359
1.395
0.57%
0.533
0.797
0.622
01597
0.50
0.555
0.551
01539
0.008
720
0.311
B.366
0.375
0.319
0.403
0.428
0.419
01424
1.299
0.541
0.869
0.675
0.618
0.595
0.535
0.530
0.510
3.483
5.764
1,677
4.400
1.444
11
2.55%
1051
2669
4,763
1.468
6,988
1.592
2.347
1.110
2453
1885
2857
1,557
2.750
1.125
5.264
1.564
5.279
1.560
5.545
1.700
6.870
1879
7.127
1.118
Table 2: Univariate long-term series forecasting results with input length / = 96 and prediction length O E (96, 192,336, 720} (for ILI, the input length I - 36). A lower MSE or MAE indicates better prediction, and the best results are highlighted in bold.
Methods
MICN-regre regre
MICN-men mean
FEDformer
Autefarmer
Informer
LagTraas
Metric
MAB
MSE
MAE
MAE
MSB
MAE
MSE
MAE
MSB
MAE
96
(.059
0.176
0.074
0.206
0.072
0.206
0.065
0.189
0.088
0.225
0.075
0.208
192
0.100
0.234
1,099
0.238
0.102
0.245
0.118
0.256
0.132
0.283
0.129
0.275
0.301
0.135
0.282
0.154
0.305
0.154
0.302
720
0.210
0.354
0.175
0.326
0.178
0.325
0.182
0.335
0.300
0.435
0.160
0.321
Eleatriitty
.95
0.310
0.398
0.376
0.418
0.255
0.370
0.341
0.438
0.484
0,538
0.288
0.393
0.300
0.394
0.317
0.410
0.386
0.345
0.428
0.558
0.422
0.483
336
0.413
0.376
0.450
0.431
0.406
0.470
0.636
0.613
0.430
0.483
720
0.364
0..49
0.417
0.479
0.422
0.484
0.565
0.581
0.819
0.682
01491
0.531
Exchange
96
0.099
0.240
0.179
0,312
0.154
0,304
0.241
0.387
0.591
0.615
0.237
0.377
192
0.354
0.304
0.420
0.420
0.300
0.369
1.183
0.738
0.619
336
0.302
0.447
0.711
0.651
0.511
0.555
0.509
0.524
1.367
0.984
2.018
1.070
720
0.738
0.662
1.416
0
1,301
0.879
1,260
0.567
1.872
1.072
2.405
1.175
0.241
0.214
0.324
0.246
0.346
0.226
0.317
Traffic
192
0,154
0.236
0.228
0,336
0.305
0.312
0.266
0.370
0.299
0,376
0.314
0,408
336
0.168
0.243
0.217
0.337
0.219
0.323
0.263
0.371
0.312
0.387
0.387
0.453
720
0,182
0.264
0.225
0.339
0,241
0,344
0.269
0.372
0.366
0.436
0.191
0.437
T.II Weathe
96
0.0022
0.039
0.0038
0,052
0.062
00011
0.081
0.044
0.0046
0.052
192
0,0021
0.034
0.0015
0.029
0.0080
0.062
0.0075
0.067
0.0023
0.040
0.0056
0,060
336
1,00123
0.034
0.0039
0.053
0.0041
0.050
0.0033
0.062
0.044
0.049
0.0060
0.054
720
0.00118
0.051
0,0024
0.037
0.0055
0.059
0.0085
0.070
0.012
0.0071
0.063
0.674
0.671
0.607
0.587
0.708
0.627
0.948
0.732
5.282
2,050
3.607
1.662
0.712
0.733
0.551
0.604
0.584
0.617
0.634
0.650
4.554
1.916
2,407
1,363
01.223
0.693
0.704
0.607
0.791
0.752
4.273
3.106
1.575
0.992
0.892
0.816
0.779
0.774
0.874
0.797
5.214
2,057
3.698
1.733
4.1
MAIN RESULTS
Multivariate results For multivariate long-term series forecasting, MICN achieves the state-of- the-art performance in all benchmarks and all prediction length settings (Table Compared to the previous best model FEDformer, MICN yields an 17.2% averaged MSE reduction. Especially, under the input-96-predict-99 setting, MICN gives 12% relative MSE reduction in ETTm2, 14% relative MSE reduction in Electricity, 31% relative MSE reduction in Exchange, 12% relative MSE reduction in Traffic, 26% relative MSE reduction in Weather, 17% relative MSE reduction in ILI, and 18.6% average MSE reduction in this setting. We can also find that MICN makes consistent improvements as the prediction increases, showing its competitiveness in terms of long-term time-series forecasting. Note that MICN still provides remarkable improvements with a 51% averaged MSE reduction in the Exchange dataset that is without obvious periodicity. All above shows that MICN can cope well with a variety of time-series forecasting tasks in real- applications. More results about other ETT benchmarks are provided in Appendix A.3 See Appendix C.3 for detailed showcases.
7

---

Published as a conference paper at ICLR 2023
Univariate results We also show the univariate time-series forecasting results in Table 2 Sig- nificantly, MICN achieves 21.6% averaged MSE reduction compared to FEDformer. Especially for the Weather dataset, MICN gives 53% relative MSE reduction under the predict-96 setting, 75% relative MSE reduction under the predict- 192 setting, 44% relative MSE reduction under the predict-336 setting, and 56% relative MSE reduction under the predict-720 setting. It again verifies the greater time-series forecasting capacity. More results about other ETT benchmarks are provided in Appendix A.3
See Appendix C.2 for detailed showcases.
4.2 ABLATION STUDIES
Trend-cyelical Prediction Block We attempt to verify the necessity of modeling the trend-cyclical part when using decomposition- based structure. Like Autoformer (Wu et al.) 2021b), previous methods decompose the time series and then take the mean prediction of the trend information, which is then added to the other trend information obtained from the decomposition module in the model. However, the reasons and rationality use simple linear regression to predict the trend-cyclical part and we also record the results of the mean prediction for comparison. Note that with different crend-cyclical prediction blocks, we have different models named MICN-regre and MICN-mean. As shown in Table 3 MICN-regre performs better than MICN-mean overall. Because all the datasets are non-stationary, simple modeling for trend-cyclical to give model a holistic view of the trend direction is necessary. See Appendix B.2 for more visualization results and analysis.
not argued in the relevant papers. In this paper,
Table 3: Comparison of sample linear regression prediction and mean prediction in multivariate datasets. The better results are highlighted in bold.
Peodiciion O
0
192
336
725
55
.92
356
720
96
190
336
720
96
192
536
755
56
192
336
720
MSE
MAE
0.555
0.315
0.98
0.400
0.299
0.558
0.384
8321
0.218
0,36
0.407
0438
8.39
0.31
0.313
0.315
0.229
0.591
0,311
6.350
MAE 0.336
0,35
0.991
0.888
0182
6.345
0250
0.399
Local-Global Structure vs. Auto-correlation, self-attention In this work, we propose the local- global module model the underlying pattern of time series, including local features and global correlations, while the previous outstanding model Autoformer uses auto-correlation. We replace the auto-correlation module in the original Autoformer with our proposed local-global module (we set E (12, 16)) for training, and the results are shown in Table 4 Also, We replace the Local-Global module in MICN-regre with the Auto-Correlation module and self-attention module for training, and the results are shown in Table [] They all demonstrate that modeling time series in terms of local features and global correlations is better and more realistic.
Table 4: Ablation of Local-global structure in other models. We replace the Auto-Correlation in Autoformer with our local-global module and implement it in the multivariate Electricity, Exchange and Traffic. The better results are highlighted in bold.
Datasets
Electricity
Exchunge
Traffic
Prediction Length
96
192
336
720
96
192
336
720
96
192
336
720
Autoformer-
MSF
0.192
0.204
0.223
0.238
0.194
0.293
1.012
1.289
0.572
0.580
0.587
0.601
Local Global
MAE
0.314
0.323
0.339
0.352
0.338
0.416
0.766
0.928
0.352
0.351
0.353
0.359
Autoformer
MSE
0.207
0.236
0.275
0.289
0.160
0.327
0.509
1.133
0.675
0.666
0.765
1.098
Asto-comelation
MAE
0.323
0.343
0.372
0.380
0.292
0.415
0.527
0.825
0.406
0.425
0.487
0.647
4.3 MODEL ANALYSIS
Impact of input length In time series forecasting tasks, the size of the input length indicates how much historical information the algorithm can utilize. In general, a model that has strong ability to model long-term temporal dependency should perform better as the input length increases. Therefore, we conduct experiments with different input lengths and the same prediction length to validate model. As shown in Fi when the input length is relatively long, the performance of Transformer- based models becomes worse because of repeated short-term patterns as stated in (Zhou et al. 2021). Relatively, the overall performance of MICN prediction gradually gets better as
8

---

block, we Published as a conference paper at ICLR 2023
Table 5: Ablation of Local-global structure in our model. We replace the Local-Global module in MICN-regre with Auto-correlation and self-attention and implement it in the multivariate Electricity, Exchange and Traffic. The better results are highlighted in bold.
Datasets
Electricity
Exchange
Traffic
Prediction Length
96
192
336
720
96
192
336
720
96
192
336
720
MICN-
MSE
0.164
0.177
0.193
0.212
0.102
0.172
0.272
0.714
0.519
0.537
0,534
0,577
Local-Global
0.269
0.285
0.304
0.235
0.316
0.407
0.315
0.313
0.325
MICN-
MSE
0.205
0,209
0.229
0.260
0.111
0.178
0.331
0.804
0.596
0.613
0.609
0.635
Auto- Correlation
0.299
0.305
0.386
MICN
MSE
0.181
0.194
0.216
0.271
0.147
0.290
0.480
1.578
0.612
0,642
0.622
0.656
self-attention
0.402
0.374
2.39
Turshees
Tueveeee
8.2
320
Electricity
Exchange
Traffic
WTH
Figure 5:
The MSE results with different input lengths and same prediction lengths (192 time steps).
the input length increases, indicating that MICN can capture the long-term temporal dependencies well and extract useful information deeply.
Robustness analysis We use a simple noise injection to demonstrate the robustness of our model. Concretely, we randomly select data with proportion in the original input sequence and randomly perturb the selected data in the range --2X; 2X;] where X; denotes the original data. The data after noise injection is then trained, and the MSE and MAE metrics are recorded. The results shown in T As the proportion of perturbations e increases, the MSE and MAE metrics of the predictions increase by small amount. indicates that MICN exhibits good robustness in response to less noisy data (up to 10%) and has a great advantage in dealing with many data abnormal fluctuations (c.g. abnormal power data caused by equipment damage).
Table Robustness analysis of multivariate results. Different & indicates different proportions of noise injection. And MICN-regre is used as the base model.
Dutasets
Electricity
Exchange
Traffic
Prediction Length
96
192
336
720
96
192
336
720
96
192
336
720
0.272
0.714
0.519
0.577
regre
MAE
0,269
0.285
0.304
0.321
0.235
0.316
0.407
0.658
0.309
0.315
0.313
0.325
19
MAE
MSE
0.270
0.163
0.288
0.179
0.303
0.192
0.325
0.217
0.237
0.103
0.172
0,316
0.289
0.424
0.691
0.652
0.321
0.518
0.312
0.530
0.535
0.315
0.323
0.575
MSE
0.164
0.181
0.192
0.218
0.104
0.167
0.296
1.742
0.518
0.541
0.558
0.585
5%
MAE
0.272
0.289
0.305
0.328
0.239
0,308
0.413
1.009
0.313
0.327
0,330
0.328
10%
MAE
MSE
0.171
0.281
0.189
0.297
0.311
0.202
0.328
0.220
0.136
0.273
0.181
0.324
0.402
0.497
0.771
0.944
0.332
0.538
0.324
0.557
0.561
0.325
0.335
0.605
CONCLUSIONS
This paper presents convolution-bate -aased framework MICN, which makes predictions for the trend- cyclical part and seasonal part separately. It achieves (LI12 complexity and yields consistent state-of-the-art performance in extensive real-world datasets. In the use different scales to mine the sequence for potentially differrntt patterns, each modeled from a local and global perspective, which is implemented by different convolution operations. The proposed isometric convolution outperforms self-attention in terms of capturing global correlations for short sequence. The extensive experiments further demonstrate the effectiveness of our modeling approach for long-term forecasting tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was supported by the Sichuan Science and Technology Program (2023YFG0112) and the National Key R&D Program of China (2020YFB0704502) and the Sichuan Science and Technology Program (2022YGG0034) and Postdoctoral Interdisciplinary Innovation Fund(10822041A2111 and Sichuan University and Yibin Cooperation Program (2020CDYB-30).
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
Anastasia Borovykh, Sander Bohte, and Comelis W Oosterlee. Conditional time series forecasting with convolutional neural networks. arXiv preprint arXiv: (7Xiv:1703.0469) 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Merbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020. URL https: pttttp:://proceednnge serrrcceringssneuurppps.s.. nwwrine.rcccmm cetrrips.ccppppee httppppeee/////1l///ee
baa.cbbbbbcccc 14wpsa444444aaaa Paper. pdf
J. Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021. URL http::ttt////to / wetp//orumiid--icooaaa/
A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, W. Yu, J.and Han, S. Wang, Z. Zhang, Y. Wu, and Pang. Conformer: Convolution-augmente transformer for speech recognition. Proceedings of Interspeech, 2020.
W. Han, Z. Zhang, Y. Zhang. J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. In Proceedings of Interspeech, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):
1735- 1780, November 1997. ISSN 0899-7667, 1530-888X.
Cheng-ZKi Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In ICLR, 2019. URL
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020. URL http::///////pereeeaa www..e//forrmmiidd-x..tt.ttt
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and and short-term temporal patterns with deep neural networks. In SIGIR, 2018a.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, 2018b.
J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T Gadde. Jasper: An end-to-end convolutional neural acoustic model. In Proceedings of Interspeech, 2019a.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. In NewrlPS, 2019b. URL httpp:://proceeediiiii www.riee.......rr cc/paper/2019/ Fi1ee7777h0ccc005222222221111000
10

---

Published as a conference paper at ICLR 2023
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li. Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low- cowpcompyy pyramidal attention for long-range time series modeling and forecasting. In International Conference Learning Representations, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
Yasuko Matsubara, Yasushi Sakurai, Willem G. van Panhuis, and Christos Faloutsos. Funnel: automatic mining of spatially coevolving epidemics. In ACM SIGKDD 2014, 105-114, 2014.
Spiros Papadimitriou and Philip Yu. Optimal multi-scale patterns in time series streams. ACM SIGMOD 2006, 647-658. ACM, 2006.
Cleveland Robert, C William, and Terpenning Irma. Stl: A seasonal-trend decomposition procedure based on loess. In Off. Stat, 1990.
T. N. Sainath, B. Kingsbury, A.-r. Mohamed, G. E. Dahl, H. Saon, G.and Soltau, T. Beran, A. Y. Aravkin, and B Ramabhadran. Improvements to deep convolutional neural networks for Ivesr. In Proceedings of ASRU, 2013.
Rajat Sen, Hsiang-Fu Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high high-diieesiooal time series forecasting. In NeurlPS, 2019.
URL https: https::/ wwwp:eiinnnnnn...........ue hwtpppps.co/ppppr200////i/// 4500444eee44555/ec/////033333333
pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. In ArXiv.or,, 14 June, 2020. https:/:://aars http://////aavvvo/////////////// 2006004768880..00
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. In arXiv preprint arXiv: 2022.
Chuhan Wu, Fangzhao Wu, Tao Qi. Yongfeng Huang, and Xing Xie. Fastformer: Additive attention can be all you need. ArXiv.org, Sept, 2021a. URL https://axxii... org/abs// 2108 09084
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 101-112, 2021b.
Z. Wu, Z. Liu, J. Lin, Lin, and S Han. Lite transformer with long-short range attention. In Proceedings of ICLR, 2020.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting, In AAAI, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecastings. In International Conference on Machine Learning. 2022.
Yunyue Zhu and Dennis Shasha. Statstream: Statistical monitoring of thousands of data streams in real time. VLDB 2002, 358-369, 2002.
Zhenhai Zhu and Radu Soricut. H- t1-transformer-11 1d: Fast one-dimensional hierarchical attention for sequences. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) 2021, Virtual Event, August 1-6, 2021, pp. 3801-3815, 2021.
11

---

Published as a conference paper at ICLR 2023
A SUPPLEMENTAL EXPERIMENTS
A.1 DATASET DETAILS
In this work, the details of the experiment datasets summarized as follows: (1) ETT (Zhou et al.
2021) dataset contains two visions of the sub-dataset: ETTh and ETTm, collected from electricity
transformers every 15 minutes and 1 hour between July 2016 and July 2018. (2) Electricity
dataset contains the electricity consumption of 321 customers recorded hourly from 2012 to 2014.
(3) Exchange (Lai al. 2018a) dataset records daily exchange rates of eight different countries daily ranging from 1990 to 2016. (4) Traffi contains the data from California Department of Transportation hourly, which describes the road occupancy rates measured by different sensors on San Francisco Bay area freeways. (5) Weather contains 21 meteorological indicators, recorded every 10 minutes for 2020 whole year. (6) II. " records influenza-like illness (II.I) patients data weekly from Centers for Disease Control and Prevention of the United States between 2002 and 2021. Table 7 summarizes feature details (Sequence Length: Len, Dimension: Dim, Frequency: Freq).
Table 7: The details of datasets.
Dataset
len
dim
freg
ETTh
17420
8
lh
ETTm
69680
15 min
Electricity
26304
322
Ih
Exchange
7588
9
day
Traffic
17544
863
1h
Weather
52696
22
10 min
ILI
966
8
7 day
A.2 IMPLEMENTATION DETAILS
Our method is trained with the 1.2 loss, using the ADAM optimizer with initial learning rate of 10 Batch size is set to 32. The training process early stopped after three epochs if there is no loss degradation are used as metrics. All the experiments are repeated 3 times with different seeds, implemented in PyTorch and conducted on NVIDIA RTX A5000 24GB GPU. The hyper-parameter i is set to {12, 16} and the hyper-pane parameter sensitivity analysis can be seen in Appendix AA.4.. For a fairer comparison, we fix the input length 96 for all datasets (36 for ILI). MICN contains MIC layer. We use MICN-regre and MICN-mean to represent the different strategies of trend-cyclical prediction block in the following.
the valid set. The mean square error (MSE) and mean absolute error (MAE)
A.3 FULL BENCHMARK ON THE ETT DATASETS
We build the benchmark on the four ETT datasets in Table 3 and Table The ETThl and ETTh2 datasets are recorded hourly, while the ETTml I and ETTm2 datasets are recorded every 15 minutes. MICN achieves state-of-the-art performance in all benchmarks in general. Especially for the multi- variate ETTml dataset, MICN gives 17% relative MSE reduction under the predict-96 setting, gives 15% relative MSE reduction under the predict- 192 setting, gives 8% relative MSE reduction under the predict-336 setting, gives 15% relative MSE reduction under the predict-720 setting.
A.4
SENSITIVITY
verify the model robustness with respect to hyper-parameter i. Different values of have slightly different results. Concretely, when i take one value, MICN performs worse
As shown in Table T0
https://archi ics www.uui..oo
http: http:///pemsscomm wommmm.dot.comm gov
12

---

Published as a conference paper at ICLR 2023
Table 8: Multivariate long-term forecasting results on ETT full benchmark. The best results are
highlighted in bold.
Methods
MICN regre
MICN
FEDiormer
Autoformer
Informer
LogTrans
Metric
MSE
MAE
MSF
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
01.21
0.431
0.398
0.427
0.376
0.419
0.449
0.459
0.865
0,713
0.878
0.740
0.474
0.487
0.430
0.453
0.420
0.500
0.482
1.008
0.792
1.037
0.824
336
0.569
0.551
0,440
0.460
0.459
0.465
0.521
0.496
1.107
0.809
1.238
0.932
720
0.770
0.672
0.491
0.509
0.506
0.507
0.514
0.512
1.181
0.865
1.135
0.852
ETTh2
96
0.299
0.364
0.332
0.377
0.346
0.388
0.358
0.397
3.755
1.525
2.116
1.197
192
0.441
0.454
0.422
0.441
0,429
0.439
0,456
0.452
5.602
1,931
4.315
1.635
336
0.654
0.567
0.447
0.474
0.496
0.487
0.482
0.486
4.721
1.835
1.124
1.604
0.956
0.716
0.467
0.463
0.515
0.511
3.647
1.625
3.188
0.316
0.362
0.360
0.399
0.379
0.419
0.505
0.475
0.672
0.571
0.600
0.546
ETTm2 ErTml
192
0,363
0.390
0.402
0.426
0.426
0.441
0.553
0.496
0,795
0.669
0.837
0.700
0.,088
0,437
0.445
0.537
720
0.481
0.476
0.459
0.464
0,543
0.490
0.671
0.561
1,166
0.823
1,153
0.820
96
0.179
0..755
0.203
0,287
0,2033
0.287
0.255
0.339
0.365
0.453
0.768
0.642
0.307
0.376
0.262
0.326
0.269
0.328
0.281
0.340
0.533
0.563
0.989
0.757
336
0.325
0.388
0.305
0.353
0,325
0.366
0.339
0.372
1.363
0.887
1.334
0.872
720
0.502
0.490
0.389
0.407
0.421
0.415
0,422
0.419
3,379
1.338
3.048
1.328
Table 9: Univariate long-term forecasting results on ETT full benchmark. highlighted in bold.
The best results are
Methods
MICN-regre
MICN-mean
FEDformer
Autoformer
Informer
LogTrans
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.058
0.186
0.069
0.210
0.079
0.215
0.071
0.206
0.193
0.377
0.283
0.468
ErThl
192
0.079
0.210
0.081
0.223
0.104
0.245
0.114
0.262
0.217
0.395
0.234
0.409
336
0.092
0.237
0.104
0.259
0.119
0.270
0.107
0.258
0.202
0.381
0.386
0.546
720
0.1
0.298
0.090
0.238
0.142
0.299
0.126
0.283
0,183
0.355
0.475
0.628
E1Th2
96
155
0.300
0.137
0,286
0.128
0.271
0.153
0.306
0.213
0.373
0.217
0.379
192
0.169
0.316
0.179
0.334
0.185
0,330
0.204
0.351
0.227
0.387
0.281
0.429
336
0,238
0.384
0.203
0.359
0.231
0.378
0.246
0.389
0,242
0.401
0.293
0.437
0.447
0.561
0.352
0.278
0.268
0.409
0.218
0.033
0.134
0.152
0.033
0.056
0.183
0.277
0.049
0.171
E7Tm2 ErTml
192
0.048
0.164
0.050
0,180
0.058
0.186
0.081
0.216
0.151
0,310
0.157
0.317
0.079
0.210
0.064
0.202
0.084
0.231
0.076
0.218
0.427
0.591
0.289
0.459
720
0.096
0.233
0,085
0.232
0.102
0.250
0.110
0.267
0.438
0,586
0.430
0.579
96
0.059
0.176
0.074
0.206
0.067
0.198
0.065
0.189
0.088
0.225
0.075
0.208
192
0.100
0.234
0.098
0.238
0.102
0.245
0.118
0.256
0.132
0.283
0.129
0.275
336
0.153
0.301
0.135
0.282
0.130
0,279
0.154
0.305
0,180
0.336
0.154
0.302
720
0.210
0.354
0.175
0.326
0.178
0.325
0.182
0.335
0.300
0.435
0.160
0.321
because of the lack of ability to capture complex temporal patterns of the time series. Meanwhile, MICN can achieve almost the same better performance when takes two or three values, indicating that the multi-branch structure is effective. To be more representative, we set i to {12, 16) in this paper.
A.5 SELECTION OF DIFFERENT CONVOLUTION MODES
As shown in Table 11
we also record the performance in different convolution modes: stride = kernel and stride = 2 The second mode makes more comprehensive use of local information, making the convolution more coherent. MICN achieves similar performance in different convolution modes. proves that MICN can make the most sequence information, and the performance the model depends the structure proposed.
kernel
13

---

Published as a conference paper at ICLR 2023
Table 10: Multivariate results with different parameters i in three datasets: Electricity, Exchange and Traffic.
Datasets
Electricity
Exchange
Traffic
Prediction Length O
96
192
336
720
96
192
336
720
96
192
336
720
MSE
0.174
0.200
0.207
0.240
0.093
0.181
0.271
0.762
0.575
0,569
0,581
0.607
24
MAE
0.282
0.303
0.317
0.336
0.227
0.321
0.404
0.675
0.334
0,316
0.323
0.339
MSE MAE
0.167
0.179
0.195
0.265
0.080
0.185
0,288
0.758
0.512
0.532
0.556
< >
0.278
0.287
0.303
0.361
0.204
0,316
0.412
0.671
0.296
0,304
0.315
0.330
MSF
0.164
0.177
0.193
0.212
0.102
0.172
0.272
0.714
0.513
0,537
0,534
0.577
12,
MAE
0,269
0.285
0.304
0.321
0.235
0,316
0.407
0.658
0.309
0.315
0.313
0.325
MSE
0.160
0.192
0.232
0.086
0.198
0.266
0.632
0.524
0.545
0.547
0.584
16, 24
MAE
0.267
0,291
0.299
0.341
0.210
0.334
0.388
0.639
0.300
0.310
0.317
0.329
MSE
0.160
0,185
0.195
0.220
0.100
0.153
0.269
0.775
0.517
0,537
0,546
0,573
<12, >4>
MAE
0.268
0.293
0.307
0.329
0.231
0.295
0.403
0.678
0.303
0,310
0.319
0.319
MSE
0.180
0,201
0.211
0.250
0.079
0.175
0,269
0.658
0.537
0.587
0.607
0.603
<424, 48
MAE
0.288
0,306
0.316
0.344
0.203
0.310
0.401
0.634
0.307
0.324
0.329
0.340
MSE
0.169
0,180
0.195
0.215
0.114
0.208
0.299
0.798
0.513
0,522
0,535
0.560
<,, 12, 24
MAR
0.278
0.300
0.323
0.348
0.425
0.704
0.303
0.304
0.307
< 12, 24,48
MSE
0.168
0.200
0.212
0.213
0.364
0.680
0.521
0.559
0.554
MAF
0.274
0,293
0.305
0.320
0.247
0.354
0.462
0.655
0.305
0.317
0.317
0,336
Table 11: MICN performance under different convolution modes. We implement it on three multi- variate datasets: Electricity, Exchange and Traffic.
Datasets
Fllccricity
Exchange
Traffic
Prodiction Length 0
96
192
336
720
96
192
336
720
96
192
336
720
gride kernel
MSE
0.164
0.177
0.193
0.212
0.102
0.172
0.272
0.714
0.519
0.537
0.534
0.577
MAR
0.269
0.285
0.304
0.321
0.235
0.316
0.407
0.658
0.309
0.315
0.313
0.325
stride
benu!
MAE
MSE
0.158
0.267
0.177
0,283
0,310
0.221
0.326
0.081
0.208
0.173
0.314
0.430
0.647
0.509
0.306
0.534
0,303
0.310
0.545
0.563
0.323
B ADDITIONAL MODEL ANALYSIS
B.1 MULTI-SCALE HYBRID DECOMPOSITION
Autoformer harnesses the decomposition as inner block of deep models and gets good performance. However, the patterns obtained by its decomposition are simple and cannot effectively deal with the complex and changeable properties of time series. As shown in Table [2] replace the decomposition block in Autoformer with our proposed multi-scale hybrid decomposition block. For Exchange, we achieve similar performance because it has no obvious temporal pattern. The result verifies that multi-scale hybrid decomposition structure is more in line with the complex temporal patterns in real-time series.
B.2 VISUALIZATION OF LEARNED FREND-CYCLICAL PARTS
As shown in Figure 6 and Figure 7 we plot the results of learned trend-cyclical parts. The separate modeling of the trend-cyclical part makes better performance and grasp of long-term progression. We also observe that the mean prediction is slightly better on the ETTm2 dataset. This is due to the complexity of the trend-cyclical information and the inability of simple linear regression, which may
Table 12: Ablation of multi-scale decomposition (MHDecomp). Autoformer-MlllDecomm adopts multi-scale decomposition block into Autoformer.
Dalasets
Electricity
Exchunge
Traffic
Prediction Length 0
192
336
720
96
192
720
96
192
336
720
Autoformer
MSF
0.207
0.236
0.275
0.289
0.160
0.327
0.509
1.133
0.675
0.666
0.765
1.098
MAE
0.323
0.343
0.372
0.380
0.292
0.415
0.527
0.825
0.406
0.425
0.487
0.647
Autoformer- MHDecamp
MSE
0.197
0.236
0.253
0.291
0.162
0.291
0.545
1.135
0.653
0.678
0.673
0.800
MAF
0.312
0.340
0.356
0.382
0.292
0.392
0.552
0.826
0.402
0.427
0.421
0.493
14

---

Published as a conference paper at ICLR 2023
prediction-96
prediction-IP 192
prediction-336
prediction-720
Figure 6: Visualization of learned trend-cyclical part prediction result Y, and seasonal part prediction result Y, in ETTml dataset under MICN-regre. Sample linear regression performs well.
LAA
prediction-9e
prediction-199 192
prediction-323 -336
prediction-721
Figure 7: Visualization of Y, and Y, in ETTm2 dataset under MICN-regre. Sample linear regression does not perform very well.
require a more advanced trend prediction method. Moreover, the mean value its trend change is close to constant, mean prediction is better in this situation.
B.3 ISOMETRIC CONVOLUTION VS. MASKED SELF-ATTENTION
With the local module in MICN, we get a short sequence characterizing local features. On this basis, we propose the isometric convolution in global module to model the global correlation of the sequence, while previously the first choice masked self-attention. We replace the isometric convolution in the global module of MICN with masked self-attention for training, and the results shown in Table [3 and Table [4 It verifies that for short sequence, isometric convolution outperforms masked self-attention in general.
Table 13: Ablation of isometric convolution. We replace the Isometric convolution in MICN-regre with masked self-attention and implement it in the multivariate Electricity, Exchange and Traffic. The better results are highlighted in bold.
Datasets
Electricity
Exchange
Traffic
Prediction Length
96
192
720
96
192
336
192
336
720
Isometric
MSE
0.164
0.177
0.193
0.212
0.102
0.172
0.272
0.714
0.519
0.537
0.534
0.577
Convolution
0.269
0.316
0.315
self-
MSE
0.153
0.179
0.201
0.256
0.0%6
0.209
0.311
0.960
0.501
0.522
0.543
0,568
attention
0.360
0.286
0.352
0.227
0.349
0.293
0.303
To compare the isometric convolution and the masked self-attention further, we conduct more experiments on full benchmark with different kernel sizes. The results in Table show that the isometric convolution outperforms the masked self-attention in the most cases. And we also note that in some cases the masked self-attention is slightly more effective. We believe that this is related to the corresponding datasets and that we will analyse the datasets in detail in the future. Moreover, can conclude that different kernels have a relatively small impact on the final results, which indicates that advanced structure instead of model parameters plays major role in the performance.
15

---

Published as a conference paper at ICLR 2023
Table 14: Comparison of Isometric convolution and masked self-attention in the univariate Electricity, Exchange and Traffic. We replace the Isometric convolution in MICN-regre with masked self- attention. The better results are highlighted in bold.
Datasets
Electricity
Exchange
Traffic
Prediction Length
96
192
336
720
96
192
336
720
96
192
336
720
Isometric
MSE
0,310
0.300
0.323
0.364
0.099
0.198
0.302
0.738
0.158
0.154
0.165
0.182
Convolution
MAE
0.398
0.394
0.449
0.240
0.354
0.447
0.241
0.236
0.243
0.264
self-
MSE
0.404
0.351
0.384
0.398
0.101
0,209
0.303
0.564
0.140
0.153
0.148
0.166
atlention
MAE
0.461
0.428
0.237
0.369
0.233
0.228
Table 15: Comparison of the isometric convolution and the masked self-attention in MICN with different kernel sizes.
Dutadd
ETTbc
Elcoskily
Exchang
Tralls:
WTH
Prodition Leeghe
556
$36
0.:54
8.191
0.71
0.577
0.223
(1.7%
<12.6>
eeff
0.885
18.403
D.618
0.420
0.313
0117M
8.766
0.266
0.384
0.50
01533
BSS
D-46
0.13
0.333
(1366
5050
0.183
0,275
0.306
0415
0.154
01775
0.55
0:313
0.09
0.775
0.300
0.5.9
0.540
0.581
164
0.33
0.28
6,306
coandation
13:83
18330
1.356
0.330
11176
0.3
8.588
0.50
5987
0.554
01579
0366
0.354
0,356
14AB
17:01
0,992
0.588
0.55
0.234
02255
0.509
0:555
0.222
0.340
0425
0.754
0.255
0.334
0.315
0559
0.33
0.27
0356
1057
03:75
0.303
0.313
01:90
0191
0:311
0.799
0.776
0.513
0.64
0331
caamdatias
36A8
0.335
0.319
0,533
2488
0.335
01355
0320
0323
0.313
0.317
B46
5660
0.354
01315
1339
B324
0:34
0.72
0038
038T
sttestke
0.334
01552
0.57
0.26
0:37
0,312
0455
0339
0.359
03339
0330
0.34
0.295
0389
0191
<35.3>>
comndatiin
MAP
11:81
13304
11300
03577
0.00
1.555
11177
1338)
010
0.78
0311
100
01967
0.166
0656
1111
0355
0333
0341
0.31
0.33
036
0396
B.4 COMPARISON OF MERGING OPERATIONS
The traditional method of merging branch structures is the concat operation on the hidden state. In this paper, we propose to adopt 2D convolution to merge multiple branches to better measure the importance of each branch (the kernel represents the weights). As shown in Table 16 the better performance verifies the effectiveness of our proposed method.
Table 16: Comparison of different merging operations. The better results are highlighted in bold.
Datasets
Electricity
Exchange
Traffic
Prediction Length
336
720
336
720
720
MICN Com-dd
MSE
0.164
0.177
0.193
0.212
0.102
0.172
0.272
0.714
0.519
0.537
0.534
0.577
MAF
0.269
0.285
0.304
0.321
0.238
0.316
0.407
0.658
0.309
0.115
0,313
0.325
MICN-
MSE MAE
0.160
0.183
0.202
0.214
0.099
0.175
0.264
0.711
0.528
0.549
0.536
0.577
CUWCO!
0.266
0.289
0.310
0.324
0.229
0.322
0.403
0.655
0.313
0.318
0.316
0.326
B.5 EFFICIENCY ANALYSIS
For MICN, the complexity lies in Downsampling convolution and Isometric convolution in Local- Global module. If we set the sequence length to L, the hidden state to D and the multi-scale convolution kernels to i.
For downsampling convolution (kerne== stride), the complexity is 0 D2 = OLD??.
For Isometric convolution, the sequence length and kernel are changed 4. stride=1, padding=4 1. so the complexity ((44)² >>) ) = 0 (=-D²) ). In this paper, we set i E 4 L ...3 is factor of L, so the complexity of Isometric convolution is O(D22), where c is constant.
In summary, the overall complexity is max( & (LD²), (cD*)) = @LD??), which is linear about the sequence length. The comparisons of the time complexity and memory usage in training and the inference steps in testing are summarized in Table [7]
Auto-correlatinn based and self-attention- based models during the training phase. As shown in Figure g the proposed Local-Global module shows ((.D2) complexity and achieves better long-term sequences efficiency.
Furthermore, compare the running memory and time among
16

---

Published as a conference paper at ICLR 2023
Table 17: Complexity analysis of different forecasting models.
Training
Methods
Time
Memory
MICN
O(LD*
O(LD*)
FEDformer (Zhou et al. 2022]
O(L)
O1L)
Autoformer (w et ul. 2021b
('LlogL)
0 (Llog)
Informer (Zhou al. 2021
((Llog
(LlogL)
Log (i et al.
L'Llg
(
Transformer Vaswani et al. 2017
(L2))
0 (('))
LSTM (Hochrenter & Schmdhuber/ 1997
O(L)
O(L
As the prediction length increases, our model takes little more time than Auto-Correlatio.. We speculate that this may due to use of the convolution operation or the activation function Tanh. In general, our method is the most portable and valuable in practical applications.
22530
Local-global
739
Local-global
Auto-correlation Attention
Auto-correlation
Attention
Memory(Ma)
1.590
0:00
12030
13020
2000
output longth
output langth
Memory Efficiency Analysis
Running Time Efficiency Analysis
Figure 8: Efficiency Analysis. We place the Local-Global module MICN with Auto-correlation and self-attention. Then we record the memory and running time of an epoch with fixed input length 96 and increasing output length. Missing values of self-attention are due out-of-memor
B.6 MORE ANALYSIS OF THE TREND-CYCLICAL PREDICTION BLOCK
To further evaluate the trend-cyclical prediction, conduct more experiments full benchmark. As shown in Table T8 regre the simple regression prediction without other modules and mean the simple mean prediction without other modules. MICN-regre our proposed method with regression prediction in trend-cyclical block and MICN-mean is the mean prediction in trend-cyclical block. FEDformer-ment the original FEDformer using the mean prediction and FEDformer-regr uses regression prediction instead of mean prediction. We can conclude that simple regression and mean fail to capture the complex temporal correlations. But for the data Exchange without periodicity, simple regression makes competitive performance. This result is worth thinking about, and we will conduct more in-depth experiments in the future. Meanwhile, the results that MICN-regne regre outperforms FEDformer-eegre and MICN-mean outperforms FEDforemr-mean prove the validity of our proposed model. also find FEDformer-regn makes worse performance in most cases. This may be due to the more complex structure of FEDformer, and more complex regression prediction is needed correspondingly.
B.7 THE DETAILED DESCRIPTION OF THE LOCAL-GLOBAI MODULE
We show the detailed description of the Local-global module in Figure 9 For the input series, we adopt down-sampling convolution with different kernels to extract the local features of different temporal patterns and isometric convolution instead of masked self-attention to capture global correlations. Then we use up-sampling convolution to recover the length of the series. Finally, we merge the different branches complete the modeling of different patterns.
17

---

Published as a conference paper at ICLR 2023
Table 18: Comparison of regression prediction and mean prediction with different models, The better results are highlighted in bold.
Flccknng
Profiction O
56
192
136
730
96
192
336
735
96
192
396
720
192
336
730
96
192
356
736
MAR
13350
D1511
0L389
B-119
0.431
MBCN-
MSE
0.179
0.307
0.325
0.502
164
0.877
8.193
0.112
D,002
0.172
0,272
9:714
8:377
8.534
0.77
8JII
0.328
8-278
0.31
MSR
0.338
0. 3.355
0.191
D1841
199
0.313
0.775
01701
D.176
0.210
0.317
7M5
3.589
D41-
0,675
B.603
6.797
0.383
D607
0
XAE
0.444
0.902
0,37
0.364
0.209
0.385
0,770
0.369
0.310
0,575
0144)
0.911
MS5
11330
0.275
03:344
14437
0.523
0.541
11.15
11000
0.202
11484
1.312
1.306
1,330
1.21K
1.344
0.225
11.77
1:225
03344
SAA
10.008
0.522
03310
16434
0.556
1..53
0.507
13.35
11299
0.286
11493
0.001
11.00
1,701
0.716
11709
0,203
113333
0.40
MAE
0.283
0.309
0.35
0.487
0,302
0,388
0.325
0.312
0.277
0/446
0.998
0,922
0.344
0.34)
0.345
0,363
0.230
0,311
0,335
0.59
R4413
188
0:214
01.44
1.1.9
1,182
01.19
DA11
0.1X
0.30
01.13
Down-sampling
Down-sampling
Convolution/eemml1225
Convolution
Isometric Convolution
Isometric Convolution
MW
wW
Up-sampling Convolution/kerell1155
Up-sampling
Convolutionkkemel112)
merge
Figure 9: The detail of the local-global module.
B.8 MORE DISCUSSION ABOUT THE ISOMETRIC CONVOLUTION AND THE LINEAR COMPLEXITY MODEL: LINFORMER, FASTFORMER.
Linformer Wang et al. (020) and Fastformer Wu et al. 2021a) are other advanced self-attention family models with linear complexity. We conduct more comparison experiments on full benchmark and show the results in Table 19 and Table 20 Concretely, we replace the isometric convolution in MICN-regre with the different masked attention mechanisms in Linformer and Fastformer. We implement the experiments in multivariate and univariate datasets. We can conclude that Fastformer can make relatively competitive performance but our proposed model MICN performs best in general. These results provide further evidence of the effectiveness of MICN.
Table 19: Comparison of different linear complexity models on the multivariate datasets. The better results are highlighted in bold.
Daaaeo
ElTac
Elecsiatt
Exdeage
Toalic
WIH
MICS.
MS0
B179
0.107
0.135
0.40
IL164
B.177
0.383
6.313
0 110
11733
8.272
01111
8,519
0.537
IUS34
8,577
16161
6.220
(1.7%
IL311
MAE
0.385
0.389
0.311
03355
0.65
0,30
0.313
0.361
0.355
Linkeme-
MS2
D1996
0.268
(1396
11300
5.19
11.08
0.291
IL566
5.26
(1800
0.56
(1540
16551
0.81
01200
0.3)
(1.9)
IL315
MAR
0277
0.39
6411
8.601
16293
3:201
1.302
0.312
0.37
11366
3415
1111
0.377
0333
11377
0.56
01266
0.913
01343
16435
0.51
MAB
0266
0.37
0436
0,527
0.215
0.2%
0.319
0.33
0.250
0131
9435
0.60
0.14
0.355
0.333
0.340
0.346
0.20
0,355
0.37
18

---

Published as a conference paper at ICLR 2023
Table 20: Comparison of different linear complexity models on the univariate datasets. The better
results are highlighted in bold.
Pichongy
Proficion D
95
892
386
720
192
335
720
96
192
336
720
192
306
730
192
356
720
MAF
8.176
8.99
0.433
0.240
9.447
0..36
0.364
0.834
B4334
0.054
Lisherser
MIE
0979
0.131
0.151
0.25
0335
3.86
0.465
0.865
0.113
0.333
9.555
0.466
0.72
0.306
02111
0255
0,030
00025
0.0045
0.346
Padfiruse
MSC
D963
0.93
8.121
0.19
0.30
0.341
8.13 170
BL8S5
0.000
R.677
0.176
181
0.701
0.001
0.006
BUHH
Isgre
5.18
0,33
D436
0.45
0.381
0.344
0471
0.813
0.367
0.207
0.945
0033
0.549
C SUPPLEMENTARY OF MAIN RESULTS
C.I MAIN RESULTS WITH STANDARD DEVIATIONS
To get more robust experimental results, repeat each experiment three times with different random seeds. For casier comparison, the results are shown in the main text when the seed is set to 2021. Table 21 shows the standard deviations.
Table 21: Quantitative results with fluctuations under different prediction lengths O for multivariate forecasting, A lower MSE or MAE indicates a better performance.
Methods Metrc
regre
Informer
LogTrans
MAE
MSE
MAE
MSE
96
0.177
6,273st00
0.208
0.289us1:00
0,258=00100
0,339x0000
0,365150062
0.453a91047
0,768 1011
0,642=t0100
192
0.2299ccutt
0.36011012
I.257
1.323
0.281
0.3401cums
0.533110004
8.563
0,757
336
0.32440001
0.381
0.310=0003
0.357-00004
0. 0.339 018
0.37240015
1.36340/73
0.587-00066
1.334=0 168
0.872:40004
720
0.470s5uu22
0.468a00019
0.34250005
0.407
0.422a0.015
0.419s0010
3.379a0134
1388850.09
3,048
1.328
Exchange Eleetretty
96
0.163--0/0s
0.269.0007
0.1 99000005
0. 0.503-0000
0.201 mm
0.31740/00
0.27440004
0. 0.368-00009
0.258-000e
0,357-cm00
0.189:55002
0.289:1002
0.20420005
0.311
0,222
0.33425004
0.296250009
0.286:20.000
0.266
0,368:20004
0.193
0.302
0
0.326u410n
1.358
0.30011000
0.394101004
0.280
0.380
0:230
0.283-0
0,376+5.00
96
0.093-:0007
0.221
0.2222000
0.299.0.00
0.19700.00
0.33340012
0,84750155
0.752a0.06 De0
0,968 177
0.812=5027
192
0.168:20mm
0.314:1003
0228
2.385
1,300
0.369
1,20421::4
L040 252
0.811
336
0.269 Joom
0.397 10:00
0.552 m036
0.552.n000
0,500 m0
0.524 10mA
167220005
096 000
16650 in 132
1.08] -om1
720
0,715=5002
8.666200011
1200
0.848
1447aa0.044
0.941
2.478a00:99
10
1941
1.127
96
0.521 Jooess
0,308 no0
i.047
0.388 1000
071411004
0 10:04
0.684:001
0,384
0.537=5006
6.313:000
0.5774008
0.345s
0,616e0 A2
0.38246020
0.379
0.685=1
0,390
0.536
0.314:num
0.507
0.350aat00e
0.022asute
0.357amm
0
0.420aas0000
0.733
0.408
0.325
0.660-0105
0.702-40m
Weather
0.163
0.2312t0014
0.185
0.25
0.206
0.350ss000
0.208220223
0.384200003
0.690
0.216
(0/00
0.339 100
0.308 noren
coums
0.5981n015
0.650
Loura
336
0.268=2000
6.321=0000
0.303+0005
1.351
0.359
0,305400011
0.578at0254
0.523an.006
0.797=0
0.652=5001
720
0.519.cumM
0.362
0.355
1.400
0,419
0.4281cum4
1.059
0.74
01800
0.675
C.2 UNIVARIATE SHOWCASES
As shown in Figure 10 Figure Figure 12 Figure 13 Figure 14 and Figure 1.5 we plot the
forecasting results from the test set of univariate dataset Electricity and Traffic for comparison. Our model gives the best performance among different models. Moreover, MICN is significantly better at predicting the overall change and peak in the time series than Transformer-basee models.
prediction-96
prediction-199 192
prediction-333 336
prediction-720
Figure 10: Prediction cases from the univariate Electricity dataset under MICN.
19

---

Published as a conference paper at ICLR 2023
prediction-96
prediction-195 192
prediction-336
prediction-720
Figure 11: Prediction cases from the univariate Electricity dataset under Autoformer.
prediction-96
prediction- 192
prediction-720 prediction-
Figure 12: Prediction cases from the univariate Electricity dataset under Informer.
prediction-96
prediction-19 192
prediction-336
prediction-720
Figure 13: Prediction cases from the univariate Traffic dataset under MICN.
prediction-9e
prediction- 192
prediction-333
prediction-720
Figure 14: Prediction cases from the univariate Traffic dataset under Autoformer.
20

---

Published as a conference paper at ICLR 2023
prediction-9e
prediction-195
prediction-336
prediction-720
Figure Prediction cases from the univariate Traffic dataset under Informer.
C.3 MULTIVARIATE SHOWCASES
As shown in Figure 16 Figure 17
Figure 18 Figure T9 Figure 20 and Figure 21 we also plot the
forecasting results from the test set of multivariate datasets ETTml and ETTm2 for comparison. Our model gives the most accurate prediction. Moreover, MICN is better predicting rising and falling turnng points time series and closer to ground truth.
prediction-96
prediction-199
prediction-336
prediction-720
Figure 16: Prediction cases from the multivariate ETTml dataset under MICN.
prediction-96
prediction-199 192
prediction-22 -36
prediction-720
Figure 17: Prediction cases from the multivariate ETTml dataset under Autoformer.
prediction-96
prediction-11 192
prediction-336
prediction-720
Figure 18: Prediction cases from the multivariate ETTml dataset under Informer.
21

---

Published as a conference paper at ICLR 2023
prediction-96
prediction-199
prediction-336
prediction-720
Figure 19: Prediction cases from the multivariate ETTm2 dataset under MICN.
prediction-96
prediction- 192
prediction-S3
prediction-
Figure 20: Prediction cases from the multivariate ETTm2 dataset under Autoformer.
prediction-99
prediction-19
prediction-23
prediction-720
Figure 21: Prediction cases from the multivariate ETTm2 dataset under Informer.
22


Reviews:
[{"review_id": "bS2ow8MJMEx", "reviewer": "Reviewer_eyrH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the Multi-scale Isometric Convolution Network, which can efficiently capture\nlocal-global interaction of time series data for long-term forecasting. It uses multi-scaled\nconvolutions for local information extraction and isometric convolution to capture the global\nrelationship. In general, it achieves the state-of-the-art accuracy in five time series benchmarks\nwith linear complexity.", "summary_of_the_review": "As I mentioned above, the overall method is well motivated and the benchmark results are\npromising, whereas the paper can be further improved by more ablation studies and discussions.", "strength_and_weaknesses": "Strength:\nThe overall paper is well written and most of the concepts are concisely explained. As the paper\nstated, both local patterns and global dependency are crucial for long-term time series\nforecasting. The design is well motivated. The benchmark results are also promising to\ndemonstrate the effectiveness of the framework.\n\nWeaknesses:\nDespite the state-of-the-art accuracy, it seems that the main improvement comes from the\nregression-based trend-cyclical prediction. Trend-cyclical analysis in section 4.2 is not enough to\nunderstand how the regression influence the prediction. I suggest evaluating the regression and\nthe mean prediction separately from the full benchmark performance. Also, the results show that\nMICN-mean has comparable performance (sometimes even worse) than FEDformer, while\nFEDformer is using mean trend prediction. I can’t help but thinking if FEDformer with regression\ntrend prediction can surpass MICN. More discussion in trend-cyclical prediction is needed.\nAnother thing that can be discussed more is the intuition behind isometric convolution. The\nmotivation of the linear complexity is good, but the paper is unclear about how isometric\nconvolution is going to capture the correlations between local time region. Besides, there exists\nmany works that focus on the efficiency of self-attention and many has linear complexity as well\n(e.g., Linformer [1], Fastformer [2], …). If the primary motivation behind isometric convolution is\nthe complexity then these works should be discussed as well.\nI also have concerns over the explainability of the local-global structure. Self-attention is good at\nmaking interpretable prediction (can be crucial for real-life application, as the paper mentioned),\nwhereas I don’t see how MICN can be interpreted at a first glance. It is sad when the paper\nargues “the framework can deeply mine the intricate temporal patterns” but fails to interpret them.\nThis is also one of the reasons I am curious about how linear self-attention can perform instead of\nisometric convolution.\nOther minor suggestion:\n1. The abstract says that the structure can model patterns separately and “purposefully”. The\nwording is vague and confusing. Does it indicate that we can incorporate external knowledge\ninto the model?\n2. In section 3.1, briefly mentions that the term (e.g., “AvgPool”, “kernel”) is from the convolution\nperspective. I first thought the “kernel” represents the kernel function.\n3. The difference between ablation table 5 and table 13 can be explained more.\n[1] Wang, Sinong, et al. “Linformer: Self-Attention with Linear Complexity.” ArXiv.org, 14 June\n2020, https://arxiv.org/abs/2006.04768.\n[2] Wu, Chuhan, et al. “Fastformer: Additive Attention Can Be All You Need.” ArXiv.org, 5 Sept.\n2021, https://arxiv.org/abs/2108.09084.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DjRQfn9-Tw", "reviewer": "Reviewer_FdbB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on long term forecasting. The problem to be solved is to predict values of a variable for a future period. The authors start by providing a comprehensive review of the related papers and end with the conclusion that a good forecasting method should have the  ability to extract local features to measure short-term changes, and the ability to model the global\ncorrelations to measure the long-term trend. Based on that, they propose a Multi-scale Isometric Convolution Network (MICN). They use multiple branches of different convolution kernels to model different potential pattern information of the sequence separately.", "summary_of_the_review": "This paper proposes a Multi-scale Isometric Convolution Network (MICN)  based on convolution structure to efficiently replace the self-attention,and it achieves linear computational complexity and memory cost. \nThe  empirical studies show that the proposed model improves the performance of state-ofthe-art methods by 18.2% and 24.5% for multivariate and univariate forecasting, respectively\n", "strength_and_weaknesses": "The topic of this paper address the problem of long-term series forecasting. The topic is very interesting and the paper is well written. The authors provide a comprehensive review of the related papers. Mathematical models and figures are proposed to understand the problem and the solution provided by the authors. Extensive experiments are provided to demonstrate the effectiveness of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4FZPXBTt6q", "reviewer": "Reviewer_PRkd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method named MICN to replace the self-attention module and achieves linear computational complexity. This method is based on convolution structure and proposes Multi-scale Isometric Convolution using multiple branches of different convolution kernels to model different potential pattern information of time series. The MICN has a local-global structure where the local features of the sequence are extracted by a local module based on downsampling convolution and the global correlation is modeled by a global module based on isometric convolution. The results show that MICN achieves state-of-the-art accuracy and isometric convolution is superior in both complexity and effectiveness to self-attention.", "summary_of_the_review": "The idea is interesting and somewhat novel. More theoretical analysis would be better.", "strength_and_weaknesses": "Pros:\n- The results show that MICN achieves state-of-the-art accuracy on five real-world benchmarks. MICN yields impressive 18.2% and 24.5% relative improvements for multivariate and univariate time series, respectively.\n- This paper uses a lot of ablation experiments to verify that the proposed Multi-scale Isometric Convolution outperforms the self-attention family and Auto-correlation mechanism.\n- This paper gives a detailed mathematical description of their method, showing how to extract local features to measure short-term changes and to model the global correlations to measure the long-term trend using a convolution structure.\n\nCons:\n- Complexity analysis is needed. One of the motivations of this paper is the high complexity of self-attention and this paper tries to reduce the complexity. However, the full text doesn't analyze the complexity of the proposed method, just saying the complexity of the method is O(L), but 'L' is not mentioned in the full text. And the complexity of the method seems to be N*logI.\n- Isometric convolution is not explained in detail as the core part of the article. How is isometric reflected? Why do convolutions in this form?\n- The paper proves that the proposed Multi-scale Isometric Convolution is superior to self-attention through experiments. However, for the short length of 92, the isometric convolution performance is all lower than self-attention in multivariate forecasting, which is inconsistent with the description of the paper, and there are no further explanations and analysis.\n\nQuestions：\n- In Section 3.2, what conditions are satisfied with the kernels to ensure that the time series length is constant? How is each kernel determined? \n- In Section 3.3, is 'mean' here an average operation? How to average a time series? How to get the result of o-dimension from i-dimension time series through mean operation?\n- In section B.3, why is the replacement of Isometric convolution and self-attention in MICN-regre instead of the Seasonal Prediction Block?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%