ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis
Authors: Luo donghao
wang xue
Venue: ICLR 2024 spotlight (2024)
Keywords: Time Series Analysis
Deep Learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We take a seldom-explored way in time series community to successfully bring convolution back to time series analysis. Our pure convolution structure achieves consistent state-of-the-art in five mainstream time series analysis tasks.

Abstract: Recently, Transformer-based and MLP-based models have emerged rapidly and
won dominance in time series analysis. In contrast, convolution is losing steam
in time series tasks nowadays for inferior performance. This paper studies the
open question of how to better use convolution in time series analysis and makes
efforts to bring convolution back to the arena of time series analysis. To this end,
we modernize the traditional TCN and conduct time series related modifications
to make it more suitable for time series tasks. As the outcome, we propose
ModernTCN and successfully solve this open question through a seldom-explored
way in time series community. As a pure convolution structure, ModernTCN still
achieves the consistent state-of-the-art performance on five mainstream time series
analysis tasks while maintaining the efficiency advantage of convolution-based
models, therefore providing a better balance of efficiency and performance than
state-of-the-art Transformer-based and MLP-based models. Our study further
reveals that, compared with previous convolution-based models, our ModernTCN
has much larger effective receptive fields (ERFs), therefore can better unleash the
potential of convolution in time series analysis. Code is available at this repository:
https://github.com/luodhhh/ModernTCN.

Published as a conference paper at ICLR 2024
MODERNTCN:
A MODERN PURE CONVOLUTION STRUCTURE FOR GENERAL TIME SERIES ANALYSIS
Donghao Luo, Xue Wang
Department of Precision Instrument, Tsinghua University, Beijing 100084, China 1ch21 @mails. tsinghua. edd.ggguueeuu.... en, wangxue twinghua.......... edu, cn
ABSTRACT
Recently, Transformer-based and MLP-based models have emerged rapidly and won dominance in time series analysis. contrast, convolution is losing steam in time series tasks nowadays for inferior performance. This paper studies the open question of how to better use convolution in time series analysis and makes efforts bring convolution back to the arena of time series analysis. To this end, we modernize the traditional TCN and conduct time series related modifications make it more suitable for time series tasks. As the outcome, we propose ModernTCN and successfully solve this open question through a seldom-explored way in time series community. As pure convolution structure, ModernTTN still achieves the consistent state-of-the-an -rtt performance on five mainstream time series analysis tasks while maintaining the efficiency advantage of convolution- based models, therefore providing a better balance of efficiency and performance than state-of-the-a Transformer-baser and MI.P-hased models. Our study further reveals that, compared with previous convolution-basee models, our ModernTCN has much larger effective receptive fields (ERFs), therefore can better unleash the potential of convolution in time series analysis. Code is available at this repository: https://gghubb..om//uoohb//////
INTRODUCTION
Time series analysis is widely used in extensive applications, such industrial forecasting ZZhou (ct al., (021), missing value imputation (Friedman, 1962). action recognition YYe & Keogh 2009).
and anomaly detection (Xu et al. 202I). Because of the immense practical value, the past few years
have witnessed the rapid development in time series analysis Wen et al., 2022 Lim & Zohren 2021).
Among them, the rise of Transformer-basee methods and MLP-based models is especially compe ling
(Nie et al. 2023 Zhang & Yan 2023) /hoo et al. 2022 Cirstea et al. 2022 Wu et al. 2021 Liu
al. 202 Li
2019b Kitaev et al. | 20200] Vaswani al. 2017) (Li et al. 2023b Zhang et al.
2022
Zeng et al. (022). But around the same time, convolution-base models have received less
attention for long time.
It's non-trivial to use convolution in time series analysis for it provides a better balance of efficiency and performance. Date back to the 2010s, TCN and its variants (Bai 2018 Sen et al.) 2019) are widely-used in many time series tasks. But things have changed in 2020s. Transformer based models and MLP-based models have emerged rapidly and achieved impressive performance in recent years. Thanks to their global effective receptive fields (ERFs), they can better capture the long-term temporal (cross-time) dependency and thus outperform traditional TCNs by a significant margin. As result, convolution-basee models are losing steam nowadays due their limited ERFs.
Some previous convolution-banl based models (Wang et al. 2023 Liu et al. 2022a) try to bring convolution back to the arena of time series analysis. But they mainly focus on designing extra sophisticated struc- tures to work with the traditional convolution, ignoring the importance of updating the convolution itself. And they still cannot achieve comparable performance to the state-of-the-ar Transformer- -assed and MLP-based models. The reason behind can be explained by Figure [ Increasing the ERF is the key to bringing convolution back to time series analysis. But previous convolution-baset models

---

Published as a conference paper at ICLR 2024
Do
20 200
30
(a) ICN (1 layer, ks=51) (ours)
(C) SCINet (1 stack, 3 levels)
203 (d) MICN (1 layer, branches: 12& (2166)
x0
(b) MCN (10 layers, ks=3)
Figure The Effective Receptive Field (ERF) of ModemTTN and previous convolution-base methods. A more widely distributed light area indicates a larger ERF. Our ModernTCN can obtain much larger ERF than previous convolution ievouution-based methods. Meanwhile, enlarging the kernel size is more effective way obtain large ERF than stacking more small kernels.
still have limited ERFs, which prevents their further performance improvements. How to better use convolution in time series analysis is still non-trivial and open question.
As another area where convolution is widely used, computer vision (CV) took very different path to explore the convolution. Unlike recent studies in time series community, lastest studies in CV focus on optimizing the convolution itself and propose modern convolution (Liu et al. 2022d] Ding et al.) 2022. Liu et al. 2022b). Modern convolution is a new convolution paradigm inspired by Transformer. Concretely, modern convolution block incorporates some architectural designs in Transformer and therefore has a similar structure Transformer block (Figure 2 (a) and (b)). Meanwhile, catch up with the gloabal ERF in Transformer, modern convolution usually adopat a large kernel as it can effectively increase the ERF (Figure Although the effectiivnesss of modern convolution has been demonstrated in CV, it has still received little attention from the time series community. Based on above findings, we intend first modernize the convolution in time series analysis see whether can increase the ERF and bring performance improvement.
Besides, convolution is also a potentially efficient way capture cross-variabee dependency. Cross- variable dependency is another critical dependency in time series in addition to the cross-time one. refers to the dependency among variables in multivariate time series. And early study(Lai et al.) 2018b) has already tried to use convolution in variable dimension to capture the cross- variable dependency. Although its performance is not that competitive nowadays, still demonstrates the feasibility of convolution in capturing cross-variable dependency. Therefore, it's reasonable to believe that convolution can become an efficient and effective way to capture cross- variable dependency after proper modifications and optimizations.
Based on above motivations, we take a seldom-explored way in time series community to successfully bring convolution- models back time series analysis. Concretely, modernize the traditional TCN and conduct some time series related modifications to make it more suitable for time series tasks. As the outcome, propose modern pure convolution structure, namely Modern to efficiently utilize cross-time and cross-variable dependency for general time series analysis. We evaluate Modern on five mainstream analysis tasks, including long-term and short-term forecasting, imputation, classification and anomaly detection. Surprisingly, as a pure convolution- based model, ModernTCN still achieves the consistent sfateoof-the-ar performance on these tasks. Meanwhile, Modern also maintains the efficiency advantage of convolution-ben baaed models, therefore providing a better balance of efficiency and performance. Our contributions are as follows:
We dive into the question of how to better use convolution in time series and propose novel solution. Experimental results show that our method can better unleash the potential of convolution in time series analysis than other existing convolution-ban based models.
ModernTCN achieves the consistent state-of of-thheartt performance on multiple mainstream time series analysis tasks, demonstrating the excellent task- sekk-ennrralizttoon ability.
Modern provides better balance of efficiency and performance. It maintains the efficiency advantage of convolution-based models while competing favorably with or even better than state-of-the-art Transformer-baan models in terms of performance.
2 RELATED WORK
2.1 CONVOLUTION IN TIME SERIES ANALYSIS
Convolution used to be popular in time series analysis in 2010s. For example, TCN and its variants (Bai et al. 2018 Sen al. 2019 Franceschi al. 2019) adopt causal convolution to model the
2

---

Published as a conference paper at ICLR 2024
Imput
Z, < R Meoo
Reshape
2, c 20000o
2, Ono
DWCCorv
R Mowayo
Selfomention
DWCCam
C
BNN
PWCar GeLU PWCCay
Gromp:M
CN
FIN
ConVEIN
Reshape
7," R
Z,, C R
Cun+FFN2 Croupet
c RiMottt
(n) Transformer bleck
(b) modera convolation block
(c Stracture of CN
Reshape Peemmtte
Z, c R
oseiue
(d) ModerTCN blac
Figure 2: MCN block design. M,,,D are sizes of variable, temporal and feature dimensions. DWConv and PWConv are short for depth-wise and point-wise convolution (2017). Groups the group number in group convolution (2018). BN and GeL.U (2015) 2016) are adpoted in our design.
temporal causality. But they suffer from the limited ERFs. With the rapid development of Transformer- based and MLP-based models, convolution has received less attention in recent years. Some studies try to bring convolution back to time series community. MICN (Wang et al. 2023) goes beyond causal convolution and proposes multi-scale convolution structure to combine local features and global correlations in time series. SCINet (Liu et al. 2022a) removes the idea of causal convolution and introduces recursive wwwnsample-convolv--inne temporal dynamics. But they still have difficulty in modeling long-term dependency due to the limited ERFs. TimesNet (Wu et al.) 2023) is special in the family of convolution- -basdd models. Different from other models that mainly use ID convolution, it transforms ID time series into 2D-variations and uses 2D convolution backbones CV to obtain informative representations.
architecture to model time series with complex
2.2 MODERN CONVOLUTIOON IN COMPUTER VISION
Convolutional neural networks (ConvNets) Krizhevsky al. 2017 Simonyan & Zisserman 2014
He et al. 2015 Xie et al. 2017: Huang et al. (017) used to be the dominant backbone architectures in CV. But in 2020s, Vision Transformers (ViTs) (Dosovitskiy et al., 2020 Liu al. 1b2 are proposed and outperform previous standard ConvNets. To catch up with the performance of ViTs, modern convolution in 2020s introduced. Inspried by the architectural designs in Transformers, ConvNeX((Liu et al. 2022d) re-design the convolution block to make it more similar to the Transformer block. To further catch up with the global ERF of Transformers, RepLKNeet Ding et al. 2022) scales the kernel size to 31x31 with the help of Structural Reparameter technique. Further more, SLaK (Liu al.) 2022b) enlarges the kernel size to 51x51 by decomposing large kernel into two rectangular, parallel kernels and by using dynamic sparsity. Inspired by above studies, we modernize and modify 1D convolution time series community to make it more suitable for time series analysis tasks.
3 MODERNTCN
In this section, we first provide a design roadmap for ModernTNN block to introduce how we modernize and optimize the traditional ID convolution block in time series community. Then we introduce the overall structure of ModernTCCN And more related details are in Appendix G
3.1 MODERNIZE THE ID CONVOLUTION BLOCK
Following the idea of (Liu et al. 2022d), we firstly re-design the 1D convolution block as shown in Figure (b). DoConv is responsible for learning the temporal information among tokens per-feature basis, which plays the same role as the self-attention module in Transformer. ConvFFN similar to the FFN module in Transformer. It consists of two PWConvs and adopts inverted bottleneck structure, where the hidden channel of the ConvFFN block is r times wider than the input channel. This module is to leam the new feature representation of each token independently.
Above design leads to a separation of temporal and feature information mixing. Each of DWConv and ConvFFN only mixes information across of the temporal feature dimension, which is
3

---

Published as a conference paper at ICLR 2024
differnet from the traditional convolution that jointly mixes information both dimensions. This decoupling design can make the object tasks easier to learn and reduce the computational complexity.
Based above design, borrow from the success of CV and modernize the ID convolution. But find that simply modernizing the convolution in the same way as CV brings little to no performance improvement in time series tasks. In fact, above design does not take into account the characteristics of time series. In addition to feature dimension and temporal dimension, time series also has a variable dimension. But the backbone stacked by convolution blocks as designed in Figure 2 (b) cannot handle the variable dimension properly. Since cross-variable information is also critical in multivariate time series (Zhang & Yan 2023 Li et al. 2023b), more time series related modifications are still needed to make the modern ID convolution more suitable for time series analysis.
3.2 TIME SERIES RELATED MODIFICATIONS
Maintaining the Variable Dimension In CV, before the backbone, we embed 3 channel RGB features each pixel into D-dimensional vector to mix the information from RGB channels via the embedding layer. But the similar variable-mixing embedding (e.g. simply embed M variables into D-dimensiinal vector per time step) is not suitable for time series. Firstly, the difference among variables in time series is much greater than that among RGB channels in a picture (Cirstea et al. 2022). Just an embedding layer fails to learn the complex dependency across variables and even loses the independent characteristics of variables for not considering their different behaviors. Secondly, such embedding design leads to the discard of variable dimension, making it unable to further study the cross-variable dependency. To this issue, we propose patchify variable-indepenndnt embedding.
We denote X E X as the M variables input time series of length L and will further divide into N patches of patch size P after proper padding (Padding details are in Appendix B). The stride in the patching process is S. which also serves as the length of non overlapping region between two consecutive patches. Then the patches will be embedded into D-dimensional embedding vectors:
Xemb Embedding(XX... (x..)
Xemb E R Mx is the input embedding. Different from previous studies (Nie et al. 2023 Zhang & Yan 2023), conduct this patchify embedding in an equivalent fully-convolutionn way for simpler implementation. After unsqueezing the shape to Xin E IR xxxL,, we feed the padded X;n into a ID convolution stem layer with kernel size P and stride S. And this stem layer maps I input channel into D output channels. In above process, each of the M univariate time series is embedded independently. Therefore, we can keep the variable dimension. And followings are modifications to make our structure able to capture information from the additional variable dimension.
DWCony DWConv is originally designed for learning the temporal information. Since it's more difficult to jointly learn the cross-time and cross-variable dependency only by DWConv, it's in- appropriate to make DWConv also responsible for mixing information across variable dimension. Therefore, we modify the original DWConv from only feature independent both feature and variable independent, making it learn the temporal dependency of each univariate time series independently. And we adopt large kernel in DWConv to increase ERFs and improve the temporal modeling ability.
ConvFFN Since DWConv is feature and variable independent, COnvFFN should mix the informa- tion across feature and variable dimensions as a complementary. A naive way is to jointly learn the dependency among features and variables by a single ConvFFN. But such method leads to higher computational complexity and worse performance. Therefore, we further decouple the single Con- vFFN into ConvFFN and ConvFFN2 by replacing the PWConvs with grouped PWConvs and setting different group numbers. The COnvFFN1 is responsible for learning the new feature representations per variable and the ConvFFN2 is in charge of capturing the cross-variable dependency per feature.
After above modifications, we have the final Modern block as shown in Figure 2 (d). And each of DWConv, ConvFFN1 and ConvFFN2 only mixes information across one of the temporal, feature or variable dimension, which maintains the idea of the decoupling design in modern convolution.
3.3 OVERALL STRUCTURE
After embedding, Xemb is fed into the backbone to capture both the cross-time and cross-variable dependency and learn the informative representation Z E R MxDNN
Z Backbone Haaikboneeemm... (xemb)
(2)
4

---

Published as a conference paper at ICLR 2024
(ackbone is the stacked MCdemnTN blocks. Each ModernTCN block is organized in residual way (He et al. 2015). The forward process in the i-th ModernTCN block is:
Zi+1 = Block (Z;) + Z;
(3)
Where Z, E RRMxDN i (1,...
.... K} is the i-th block's s input,
Xemb
==1
2, Block(Z; Z;-1
(4)
.
Block(-) denotes the MC block. Then the final represnetation Z Block(Zx) + Zx will be further used for multiple time series analysis tasks. See Appendix for pipelines of each task.
EXPERIMENTS
We evaluate ModernTCN on five mainstream analysis tasks, including long-term and short-term forecasting, imputation, classification and anomaly detection to verify the generality of Modemn
Baselines Since we attempt to propose a foundation model for time series analysis, we extensively include the latest and advanced models in time series community as basic baselines, which includes the Transformer-basee models: PatchTST (2023). Crossformer (2023) and FEDformer (2022); MLP- based models: MTS-Mixer (2023b), LighTTS (2022). DLinear (022). RLinear and RMLP (2023a); Convolution- based Model: TimesNet (2023) MICN (2023) and SCINet (2022a). We also include the state-of-the-as art models in each specific task as additional baselines for comprehensive comparison.
Main Results As shown in Figure 3 Modern achieves consistent tfate-of-the-art perfor- mance on five mainstream analysis tasks with higher efficiency. Detailed discussions about experimental results are in Section [.1 We provide the experiment details and results of each task in following subsections. In each table, the best results are in bold and the second best are underlined.
Long terma Foreentting
Long-term forecasting in ETTm2 (prediction longth 720)
0,4
LighTS
Memory Footprini
Detectan
Shurt-term Furccasting ISMAPE
Stasel
ModernTCN
1000 2020 3100
(HaISTT2221)
MII MD
5000MER BO1IMB....
Crostoenee(11133
$1
13,38
(LD(omee(222))
MSE
RMILP
RI insar
Timooett
MIS-MMuun(223)
60T9MBB,207.99
MTSS MINEE
4415MIR DS DA%
0,96
RMLP12025)
MICN
8349MB,419.56
FEDfermer
2023)
1415MIBIS
PatchTST
#720M01-51.12
MIKND1125
Classifeation (Accarmyy)
Impotaatio (MSE)
SC1Na13622g
Training Time (s/epoch)
Figure 3: Model performance comparison (left) and efficiency comparison (right).
4.1 LONG-TERM FORECASTING
Setups We conducted long-term forecasting experiments on popular real-world benchmarks, including Weather (Wetterstation). Traffic (PeMS). Electricity (UCI), Exchange (Lai et al.) 201 8a). ILI (CDC) and 4 ETT datasets (Zhou et al., 2021). Following (Nie et al., 2023 Zhang & Yan [0223] we re-run all baselines with various input lengths and choose the best results to avoid under-estimating
the baselines and provide fairer comparison. We calculate the MSE and MAE multivariate time series forecasting as metries.
Results Table shows the excellent performance of ModemnTCN in long-term forecasting. Con- cretely, ModernTCN gains most of the best performance in above 9 cases, surpassing extensive state-of-the-art MLP-based and Transformer-basel models. It competes favorably with the best Transformer-based model PatchTST in terms of performance while having faster speed and less memory usage (Figure 3 right), therefore providing a better balance of performance and efficiency. It's notable that ModemnCh surpasses existing convolution- models by a large margin (27. (27.4% reduction on MSE and 15.3% reduction on MAE), indicating that our design can better unleash the potential of convolution in time series forecasting.
5

---

Published as a conference paper at ICLR 2024
Table I: Long-term forecasting task. All the results are averaged from 4 different prediction lengths, that is {24, 36, 48, 60} for ILI and (96, 192, 336, ]20] for the others. A lower MSE or MAE indicates a better performance. See Table 27 Appendix for the full results with more baselines.
Models MCNernICN
PatchTST
Crossfomee FEDioneer MTS Mixer
RLinear
DLinear
TimesNet
MICN
SCINet
(Ours)
2023
2023
2022
2023b
2025a
(2022]
A023
Metric
MSE
MAE
MSE
MAE
MSE MAE
MSE MAE
MSE MAE
MSE
MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
ETTh1
0.404
0,420
0.413
0.431
0.441
0.465
0.428
0.454
0.430
0.436
0.408
0.421
0.423
0.437
0.458 0.450
0.433 0.462
0,460 0.462
ETTh2
0.322
0.379
0.379
0.835
0,676
0.388
0.434
(.3386
0.413
0.320
0.378
0.443
0.414
0.427
0.385
0.410
ETTml
0.351
0.381
0.351
0.381
00.43
0.443
0.
0.382
0.422|0.370
0.422
(0.370
0.395
00.358
0.376
0.357
0.379
0.400
0.406
0.383
0.406
0.387 0411
ETTm2
002253
0.314
0.255
0.315
0.632
0.578
0.292
0.343
(0.277
0.325
(0.256
0.314
|0.267
0.332
0.291
0.333/0.277
0.333
(0.277 0.336
0.294 0.355
Electricity
0.156
0.253
0.159
0.253
0.299
0.351
00.207 0.321
|0.173
0.272
10.169
0.261
(0.177
0.274
0.192
0.295
[0.182
0.292
0.195 0.281
Weather
0.2244
0.264
0.226
0.264
0.230
0.290 0.3 1o 0.357
(0.235 0.272
0.2247
0.279
0.2240
0.300
0.259
0.287
[0.242 0.298
0.287 0.317
Traffie
0.396
0,270
0.391
0.264
0.535
0.300
0.604 0.372
0.372|0.494 0.449
0.354
(0.518
0.383
0..434
0.295
0.620
0.336
0.533
0.312
0.587
0.378
Exchange 0.302
0.366
00.387
0.419
0.419
0.701
0.478
0.373
0.407
(0.345
0.394
0.299
0.378
0.416
0.443
0.315
IIJI
1.440
0.786
1.443
0.798
3,361
1.235
2.597
1.070| 1.555
0.819
4.269
1.490
2.169
1.041 139 0.931 2.567 1.055 2.252 1.021
Table 2: Short-term forecasting task. Results are weighted averaged from several datasets under different sample intervals. Lower metries indicate better performance. See Table 28 for full results.
Models ModernTCNN CARD PTtcTTT Crossormer FEDisrmer MTS Mixer RLineur DLinear TimesNet MICN SCINet N N-IITS N. BEATS
(Ours)
2023
22023
2023
2022
2023b
2023g
2022
2023
2023
2022a
2023
2019]
SMAPE
11,698
11.815
11.807
13.474
12,840
11,892
13.639
11.829
12,369
11,927
11.851
MASE
1.556
1.587
1.590
1,866
1.701
1.608
1.677
2.095
1,585
1.896
1.677
1.613
1.599
OWA
0,838
0.850
0,851
0.985
0.918
0.859
0.898
1.051
0.851
0.980
0.894
0.861
0.855
4.2
SHORT-TERM FORECASTING
Setups We adopt M4 dataset (Makridakis et al. 2018) as the short-term forecasting benchmark. Following (Wu et al. 2023). we fix the input length to be times of prediction length and calculate Symmetric Mean Absolute Percentage Error (SMAPE), Mean Absolute Scaled Error (MASE) and Overall Weighted Average (OWA) as metrics. We include additional baselines like CARD (023)). N-BEATS (2019) and N-HiTS (023) for this specific task. Since the M4 dataset only contains univariate time series, we remove the cross-variable component in ModernTNN and Crossformer.
Short-term forecasting in M4 dataset is much more challenging task because the time series samples are collected from different sources and have quite different temporal properties. Our Modern TCN still achieves the consistent state-of-the-art in this difficult task, demonstrating its excellent temporal modeling ability.
Results
The results are summarized in Table 2
4.3 IMPUTATION
Setups Imputation task aims to impute the missing values based on the partially observed time series. Due to unexpected accidents like equipment malfunctions or communication error, missing values in time series are very common. Since missing values may harm the performance of downstream analysis, imputation task is of high practical value. Following (Wu al. 2023), we mainly focus on electricity and weather scenarios where the data-missing problem happens commonly. We select the datasets from these scenarios as benchmarks, including ETT (Zhou al. 2021). Electricity (UCI) and Weather (Wetterstation). We randomly mask the time points in ratios of 12.5%, 25% 37.5% 30% to compare the model capacity under different proportions of missing data.
Results Table 3 shows the compelling performance of ModernTCN in imputation tasks. Mod- emTCN achieves 22.5% reduction MSE and 12.9% reduction MAE compared with previous state-of-the-ad baseline TimesNet (2023). Due to the missing values, the remaining observed time series is irregular, making more difficult to capture cross-time dependency. Our ModernTCN still

---

Published as a conference paper at ICLR 2024
Table Imputation task. We randomly mask (12.5%. 25%, 37.5% 50%) time points in length-96 time series. The results are averaged from 4 different mask ratios. A lower MSE or MAE indicates better performance. See Table 29 in Appendix for full results with more baselines.
Models
ModeraTCN
PatchTST
Crossformer FEDformer MTS-Mixer
RLincar
DLinear
TimesNet
MICN
SCINet
(Ours)
2023
2023
A022
20236
2023a
2202
2023
2025
2022a
Averaged
MSE
MAB
MSE MAE
MSB MAE
MSE
MAE
MSE
MSE MAE
MSE MAE
MAB
ETTml
0.020 0.093
0..045
0.133
0.143
0.062 0.177 0.056 0.154 0,000 0.16600.093
0.027 D. 0.107 0.070 0. 182 0.039 0.1
ETTm2
0.019
0.082
J0.028 0.096/ 0.149
0.101 0.215
0.032 0.107 0.032 0.108/0.096 0.208 0.206 0.022 0.088 0.1 0.144 0.249/0.027 0.102
0.150
13. 0.236/0.132 0.251
1 0.246
(0.
0.306 0.078 o
0.250/0. 0.216
ETTh2 0.042
0.131
|0.066
0.164/0.164...22
0.240
63
0.279
0.069
0. 000.0006660.66/014222.223333
0.300
Electricity 0.073 0.187
[0.091 0.209/0.083
0.199
30.130
0.259
0.0099
02,,8..000011119.220000000000000 /1tp::/000000..2220001.0000000.0 0.086 0.201
Weather 0.027 0.044 |0.033 0.057/0.036 0.090 (0.099 0.203 0.203/0.036 0.05%| 0.034 o.05s 0.1.::0552..1110..00..000.. 1.0000000 0.054 0..10/0.000.22800...0 0.053
achieves the best performance in this challenging task, verifying the model capacity in capturing temporal dependency under extremely complicated situations.
It's also notable that cross-variable dependency plays a vital role in imputation task. Since in some time steps. only part of the variables are missing while others are still remaining, utilizing the cross-variable dependency between missing variables and remaining variables can help to effectively impute the missing values. Therefore, some variable- indeeeddnd methods like PatchTST (2023) and DLinear (2022) fail in this task for not taking variall dependency into consideration.
4.4
CLASSIFICATION AND ANOMALY DETECTION
RNN-hasol
Convolution-asell
ML. MI.P-asell
Transformer-tsseel
$6.62
$5.84
73.6
$4.97
31.8
72.5
71.7
73.0
Average Fl-score (%)
54.23
$4.24
$3.45
Average Accuracy
70.0
70.4
70.0
70.6
70.7
LSTRet
15st.
Reclel
SChNo
NICW
Thacttee
Ligh?
MTS-Mire Mirer
RM
FED
How
Grots
Patch TV7
Modere CN
SCINe
MIC
Timester
DLiwor
Lighrr3
Mrs--Mise
loror
RMLP
A
FED.
Cross.
PatcTST
TCN
DL
(a) Classification Results
(b) Anomaly Detection Results
Figure 4: Results of classification and anomaly detection. The results are averaged from several datasets. Higher accuracy and Fl score indicate better performance. *. in the Transformer-basee
models indicates the name of #former. See Table and il Appendix for full results.
Setups For classification, we select 10 multivariate datasets from UEA Time Series Classification Archive (Bagnall et al.. 2018) for benchmarking and pre-process the datasets following (Wu et al. 2023). We include some task-specific ootte-of-thh-ar methods like LSTNet (2018b), Rocket (2020)
and Flowformer (2022) as additional baselines.
For anomaly detection, we compare models on five widely-used benchmarks: SMD (Su et al. 2019). SWaT (Mathur Tippenhauer) 2016). PSM (Abdulaal et al., (02T). MSL and SMAP (Hundman lt al. 2018). We include Anomaly transformer (2021) as additional baselines. Following it, we adopt the classical reconstruction task and choose the reconstruction error as the anomaly criterion.
Results Time series classification is a classic task in time series community and reflects the model capacity in high-level representation. As shown in Figure ModernTCN achieves the best performance with an average accuracy of 74.2%. It's notable that some MLP-based models fail in classification tasks. This is because MI.P-based models prefer to discard the feature dimension to
7

---

Published as a conference paper at ICLR 2024
obtain a lightweight backbone, which leads to the insufficient representation capability and inferior classification performance. Anomaly detection results are shown in Figure 4 ModemTCN achieves competitive performance with previous state-of-the-art baseline TimesNet (2023). Mcanwhile, compared with TimesNet, ModernTCN saves 53.1 % average training time per epoch (3.19s 19s vs 7.10s) in classification task and saves 57.3% average training time per epoch (132.65s vs 310.62s) in anomaly detection task, providing a better balance of efficiency and performance in both tasks.
5
MODEL ANALYSIS
5.1
COMPREHENSIV COMPARISON OF PERFORMANCE AND EFFICIENCY
Summary of Experimental Results ModernTCN achieves consistent state-of-the-ar performance on five mainstream analysis tasks compared with other task-specific models or previous state-of the- art baselines, demonstrating its excellent task-generality and highlighting the potential of convolution in time series analysis (Figure ( left). Modern'CN also has more advantage in efficiency, therefore providing better balance of efficiency and performance (Figure 3 right). It's worth noting that our method surpasess existing convolution-biee based models by large margin, indicating that our design can provide a better solution the problem of how better use convolution in time series analysis.
Compared with Transformer-based and MLP--aaed Models Unlike previous convolution-basee models, ModernTNN competes favorably with or even better than state-of-the-an Transformer-basee models in terms of performance. Meanwhile, as a pure convolution model, Modern has higher efficiency than Transformer-bar based models. As shown in Figure 3 right, ModemTCN has faster training speed and less memory usage, which demonstrates the efficiency superiority of our model.
Modemn outperforms all MLP-based baselines in all five tasks thanks to the better representation capability in Modern'CCN blocks. contrast, MI.P-based models prefer to adopt a lightweight backbone for smaller memory usage. But such design in MLP-based models also leads to the insuf- ficient representation capability and inferior performance. Although ModernTCN is sightly inferior in memory usage, it still has almost the same rumning time efficiency as some MLP- baselines thanks to the fast floating point operation speed in convolution. Considering both performance and efficiency, Modern has more advantage in general time series analysis.
Compared with TimesNet (2023) In addition to MCNemnTNN, TimesNet also demonstrates excel- lent generality in five mainstream tasks. It's worth noting that both models are convolution-basee models, which further reveals that convolution has better comprehensive ability in time series analysis. Meanwhile, both methods are inspired by CV and intend to make the time series analysis take advantage of the development of CV community. But the two methods take different paths to accomplish this goal. TimesNet makes efforts to transform the 1D time series into 2D space, making the time series can be modeled by the 2D ConvNets in CV community. But the additional data transformation and aggregation modules also bring extra memory usage and slower training speed. Different from TimesNet, our ModernTCN maintains the ID time series and turns to modernize and optimize the ID convolution in time series community. Therefore, we design a modern pure convolution structure that without any additional modules. The fully-convolutional nature in our design brings higher efficiency and makes it extremely simple to implement, therefore leading to the both performance and efficiency superiority than TimesNet (Figure 3eft and right).
5.2 ANALYSIS OF EFFECTIVE RECEPTIVE FIELD (ERF)
Enlarging the ERF is the key to bring convolution back time series analysis. In this section, we will discuss why ModernTCN can provide better performance than previous convolution-based models from the perspective of ERF. Firstly, rather than stacking more layers like other traditional TCNs (2018), Mc increases the ERF by enlarging the kernel size. And in pure convolution structure, enlarging the kernel size is a much more effective way to increases ERF, According to the theory of ERF in pure convolution- based models (Luo et al. 2016), ERF is proportion to O(ks x vnl), where ks and nal refers to the kernel size and the number of layers respectively. ERF grows linearly with the kernel size while sub-linearly with the layer number. Therefore, by enlarging the kernel size, Modern TCN can easily obtain a larger ERF and further bring perfomance improvement.
8

---

Published as a conference paper at ICLR 2024
Except for enlarging the kernel size and stacking more layers, some previous convolution- based methods in time series community (MICN (2023) and SCINet (2022a)) prefer to adopt some sophisti- cated structures to cooperate with the traditional convolution, intending enlarge their ERFs. Since they are not pure convolution structures, it's hard to analyse their ERFs theoretically. Therefore, we visualize the ERFs of these methods for intuitive comparision. Following (Kim et al. 2023), , sample 50 length-336 input time series from the validation set in ETThl for the visualization. The idea behind is visualize how many points in the input series can make contribution the middle point of the final feature map. As shown in Figure I our method can obtain a much larger ERF than previous convolution- based methods. Therefore our method can better unleash the potential of convolution in time series and sucessfully bring performance improvements in multiple time series analysis tasks.
5.3 ABLATION STUDY
Ablation of Modern Block Design To validate the effectiveness of our design in ModernTCN block, we conduct ablation study in long-tern Beng-term forecasting tasks. Results are shown on Table [ Discard Vuriable Dimension cannot provide ideal performance, which confirms our arguement that simply modernizing the convolution the same way as CV could not bring performance improvement for omitting the importance of variable dimension. To better handle the variable dimension, decouple single ConvFFN into ConvFFNI I and ConvFFN2 in our design. As shown in Table 4 the undecoupled ComVFFN provide the worst performance and the combination of our decoupled two ConvFFNs (ConyF FFN of our decouple modification to ConvFFN module. Please see Appendix Hor more details.
I + ConvFFN2) achieve the best, which proves the necessity and effectiveness
Table 4: Ablation of ModemTTC. block. We list the averaged MSE/MAE of different forecast lengths.
Bachonge
Matri
MSE
MAB
MSB
MAE
MSB
MIEE
MSS
MAAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ComFFNN Cun
0.351
0.351
0.253
0.314
0.365
1.440
0.786
Lidtocuplist
ConSEENI
0.60
0.356
0.267
0.323
0,300
1.667
0.565
only CansPFN2
0:416
0.488
0.530
0.364
0.550
0.355
0.268
0.325
0.164
0.265
0.340
0.281
0312
0.501
1.514
0.681
ComEERN CoIFFN]
0:420
0,435
0.329
0.355
0363
0.356
0.667
0,333
0.165
0.255
0.342
0.279
0.311
0,57)
1.700
0.565
ConSFFN2 ConnFFN22
0.417
0.477
0.330
0.383
0362
0.364
0.269
0.325
16165
0.265
01254
0.265
0.311
0.573
1.631
0.86
Dacami Vanablc Dimassion
0.382
0.400
0.508
0.464
0.19
0.367
0.441
0.496
0.361
0.413
1,92
0.935
Ablation of Cross-variable Component As important time series related modification in our design, we design the ConvFFN2 as a cross-variable component to capture the cross-variable dependency. We conduct ablation studies in imputation tasks and anomaly detection tasks. As shown in Table 5] without the ConvFFN2 will cause severe performance degradation in these two tasks, which emphasizes the importance of variable dependency in time series analysis.
Table 5: Ablation of Cross-variable component.
Dutaset
ETTml
BTTm²
BTThl
ETTh2
ECL
Weather
Mitric
MSF
MAE
MSF
MAR
MSF
MAR
MSF
MAF
MSF
MAF
MSF
MAF
Ours
0.020
0.093
0,019
0.082
0.150
0.042
0.131
0,073
0,027
0,944
Ieputiion
wolo Cross variable
0.088
0.121
0.028
0:097
0.899
0,195
0.059
0.150
0.096
0,200
0.000
BL048
Promotion
47.4%
25.1%
32.1%
13.5%
43.8%
23.1%
28.8%
12.7%
24.05/
7.5%
10.0%
8.3%
Dataset
SMD
MSI.
SMAP
SWAT
PSM
Mitric
Fl-scone
Fl-sovre
Fl-szore
Fl-spore
Flasovee
Detestion
Amuull
wio Cross variable
Pramotion
Ours
85.81
$1.33
5.5%
17.7%
72.13
$4.92
64,93
71.26
9.7%
12.5%
93.86
83.06
97.23
96.05
1.2%
CONCLUSION AND FUTURE WORK
In this paper, we take a seldom-explored way in time series community solve the question of how to better use convolution in time series analysis. By modernizing and modifying the traditional TCN block with time series related modifications, we propose ModemnTCN and successfully bring convo- lution back to the arena of time series analysis. Experimental results show the great task generality of MCdernTCN. While performing on par with or better than tlate-of-the-arr Transformer-baat in terms of performance, Modem maintains the efficiency advantage of convolution-based models, therefore providing a better balance of performance and efficiency. Since convolution-bast models have received less attention in time series analysis for a long time, we hope that the new results reported in this study will bring some fresh perspectives to time series community and prompt people to rethink the importance of convolution in time series analysis.
models
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This work was supported by the Guangdong Key Area Research and Development Program (20198010154001) and Hebei Innovation Plan (20540301D).
REPRODUCIBILITY STATEMENT
The model architecture is introduced details with equations and figures in the main text. And all the implementation details included in the Appendix, including dataset descriptions, metrics of each task, model configurations and experiment settings. Code is available at this repository: http:://ithbb..commmmo/h////oodr
REFERENCES
Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp.
2021.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint ar
2018.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint
CDC. html
Illness.
Cristian Challu, Kin Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6989 6997, 2023.
Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong. and Shirui Pan. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting-full version. arXiv preprint arXiv:2204. 13767, 2022.
Angus Dempster, François Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kemels. Data Mining and Knowledge Discovery, 34(5): 2020.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and patfern recognition, pp. 13733-13742, 2021.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernlls to 31x31: Revisiting large kernel design in enns. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11963-11975, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929. 2020.
Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NewrIPS, 2019.
Milton Friedman. The interpolation of time series by related series. J. Amer. Statist. Assoc, 1962.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning. Image Recognition, 7. 2015.
Dan Hendrycks and Kevin Gimpel. Gaussian arXiv:1606.08415 08015, 2016.
error
linear units (gelus). arXiv preprint
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:
2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. De- tecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp- 387-395, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang. Dong Gu Lee, Wonseok Jeong, and Sang Woo Kim. Dead pixel test using effective receptive field. Pattern Recognition Letters, 167:149-156, 2023.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14X1::1412.6980, 2014.
Nikita Kitaev, łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001. 004551, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, 2018a.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 95-104, 2018b.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, 2019a
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in neural information processing systems, 32, 2019b.
Zhe Li, Shiyi Qi, Yiduo Li. and Zenglin Xu. Revisiting long-term time series forecasting: An investigation linear mapping. arXiv preprint arX11:2305.10721 2023a
Zhe Li. Zhongwen Rao, Lujia Pan, and Zenglin Xu. Mts-mixers: Multivariate time series forecasting via factorized temporal and channel mixing. arXiv preprint w0Xiv:2302.04501, 2023b.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical 2021. 2021.
Transactions of the Royal Society A, 27922194):00000200

---

Published as a conference paper at ICLR 2024
Minhao Liu, Ailing Zeng, Z Xu. Lai, and Q Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. In 36th Conference on Neural Information Processing Systems (NeurIPS), 2022a.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint Si4iv:2207.036222 2022b.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference learning representations, 2021a.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. In NeurlPS, 2022c.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11988 2022d.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802 808, 2018.
Aditya P Mathur and Nils Ole Tippenhauer. Swat: water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater), pp. 31-36. IEEE, 2016.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2023.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 1905 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
PeMS. Traffic. http::/ dot. ca. gov/
Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances in neural information processing systems, 32, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155 2014.
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 182882837,, 2019.
UCL.
Electricity.
attps://archivve www.:/ee..sss....ee.... www.:u......cceeeec
Flectricityloosiiaaa
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Huiqiang Wang. Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series; A survey, arXiv preprini 22222 2022.
Wetterstation. Weather. https: www bg:- http://wwww..eee.......e.e//eeee
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, Advances in Neural Information Processing Systems, 34:22419-22444 2021.
Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows. arXiv preprint urXiv:2202.0000 2022.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210002186, 2023.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642, 2021.
Wang Xue, Tian Zhou, QingSong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Make transformer great again for time series forecasting: Channel aligned robust dual transformer. arXiv preprint arXiv::305. 12095, 2023.
Lexiang Ye and Eamonn Keogh. Time series shapelets: new primitive for data mining. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 947-956 2009.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13522 2022.
Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mip structures. arXiv preprint arXiv:2207.00 00 186, 2022.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference Learning Representations, 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.
2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pp. 27268-27288 PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
Table 6: Dataset descriptions of long-term forecasting and imputation.
Dataset
Weather
Traffic
Exchange
Electricity
ILI
ETThl
ETTh2
ETTml
ETTm2
Dataset Size
$2696
17544
7207
26304
966
17420
17420
69680
69680
Variahle Number
21
862
321
7
7
7
7
Sampling Frequency
mins
hour
day
hour
week
hour
hour
mins
mins
A DATASETS
A.1 LONG-TERM FORECASTING AND IMPUTATION DATASETS
We evaluate the long-term forecasting performance on popular real-world datasets, including Weather, Traffic, Electricity, Exchange, III and 4 ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2). And for imputation tasks, we choose Weather, Electricity and ETT datasets (ETThl, ETTh2, ETTml, ETTm2) for benchmarking. These datasets have been extensively utilized for benchmarking and cover many aspects of life.
The dataset size (total timesteps), variable number and sampling frequency of each dataset are summarized in Table We follow standard protocol (Zhou et al., (2021) and split all datasets into training, validation and test set chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets. And training, validation and test sets are zero-mean normalized with the mean and standard deviation of training set. Each of above datasets only contains one continuous long time series, and we obtain samples by sliding window.
More introduction of the datasets are as follow:
1) Weather contains creatherccontiins 21 meteorological indicators such as humidity and air temperature for 2020 whole year in Germany.
2) Traffic Bay area freeways years. Data collected from California Department of Transportation.
contains contains the road occupancy rates measured by 862 different sensors San Francisco
3) Electricity contains hourly electricity consumption of 321 clients from 2012 to 2014.
4) Exchange the daily exchange rates of eight different countries ranging from 1990 2016.
5) ILI(Infuennz--Like Illness the weekly recorded influenza-like illness (ILI) patients data the United States between 2002 and 2021. contains 7 indicators like the numbers of ILI patients under different age ranges and the ratio of ILI patients to the total patients. Data is provided by Centers for Disease Control and Prevention of the United States.
EE Transformer Temperature contains the data collected from electricity trans- formers with 7 sensors, including load, oil temperature, etc. It contains two sub-dataset labeled with and 2, corresponding
two different electrie transformers from two separated counties in China. And each of them contains different resolutions (15 minutes and hour) denoted with and h. Thus, in total we have ETT datasets: ETThI, ETTh2, ETTml, ETTm2.
A.2 SHORT-TERM FORECASTING DATASETS
M4 involves 100,000 different time series samples collected in different domains with different frequencies, covering wide range of economic, industrial, financial and demographic areas.
It's notable that M4 dataset is different from the long-term forecasting datasets. Each of long-tern forecasting dataset only contains one continuous long time series, and we obtain samples by sliding window. Therefore all samples are come from the same source time series and more likely to have similar temporal property. But the samples in M4 datasets are collected from different sources.
http:://www.bbcjjj.... wttpjjennnamge/eeeee..ee /
hhups://pems.doo wwt.:aa..d.o.oo.doooooooo
wat.s//iiillllllaeeeeeeeeeaaaaaa 12004
"hutps///iss.dd edc. www.g//www//wtii/////oossssooooo
14

---

Published as a conference paper at ICLR 2024
Therefore they may have quite different temporal property, making the forecasting tasks in M4 datasets more difficult. Table summarizes details of statistics of short-term forecasting M4 datasets.
Table 7: Datasets and mapping details of M4 dataset (Makridakis et al. 2018
Dataset
Sample Numbers (train set.esst set)
Variable Number
Prediction Length
M4 Yearly
(23000, 23000)
M4 Quarterly
(24000, 24000)
M4 Monthly
(48000, 48000)
M4 Weekly
(359, 359)
M4 Daily
(4227, 4227)
14
M4 Hourly
(414, 414)
48
A.3 CLASSIFICATION DATASETS
UEA dataset involves many time series samples collected in different domains for classification. covering the recognition tasks based on face, gesture, action and audio as well as other practical tasks like industry monitoring, health monitoring and medical diagnosis based on heartbeat. Most of them have 10 classes. Table X summarizes details of statistics of classification UEA datasets.
Table 8: Datasets and mapping details of UEA dataset (Bagnall et al. 2018).
Dataset
Sample Numbers (train settest set)
Variable Number
Series Length
SohanolConcentration
(261, 263)
3
1751
FaceDetection
(5890, 3524)
144
62
Handwriting
(150, 850)
152
Heartbeat
(204, 205)
Japanese Vowels
(270, 370)
12
29
PEMS- SF
(267, 173)
963
144
SellRegulationSCCPI
(268. 293)
6
896
SelfRegulationSCCP2
(200, 180)
1152
SpokenAArahicDigiss.
(6599, 2199)
13
UWaveGesturelLirra
(120, 320)
3
315
A.4 ANOMALY DETECTION DATASETS
We adopt datasets from different domains like server machine, spacecraft and infrastructure for benchmarking. Each dataset is divided into training, validation and testing sets. Each dataset contains one continuous long time series, and we obtain samples from the continuous long time series with fixed length sliding window. Table summarizes details of statistics the datasets.
Table 9: Datasets and mapping details of anomaly detection dataset.
Dataset
Dataset sizes(train set,val set, test set)
Variable Number
Sliding Window Length
SMD
(566724, 141681, 708420)
38
100
MSL. SMAP
(44653, 11664, 73729)
55
100
(1081 14, 27037, 427617)
SWaT
(396000, 99000, 449919)
51
100
PSM
(105984, 26497, 87841)
25
100
B PIPELINE
15

---

Published as a conference paper at ICLR 2024
B.1 PADDING DETAILS IN PATCHIFY VARIABLE-INDEPENDENN INDEPENDENT EMBEDDING
Before patching and embedding, we adopt a padding operation on the original time series Xān to keep N L//S. Specifically, we repeat X.n last value (P S) times and then pad them back to the end of X Xin-
Denoted in RRXX as the M variables input time series of length L, the overall process of Patchify Variable-independeet Embedding is as follows:
1) Unsqueezing its shape to Xin E RRXXX
2) Adopting above padding operation
3) Feeding the padded in. to the ID convolution stem layer for patching and embedding.
B.2 PIPELINE FOR REGRESSION TASKS
The pipeline for forecasting, imputation and anomaly detection is shown as Figure
ReviN
Norm
De-Norm
Putchify Embbediing
imput
Backbone
Flatten
Head
X, R
MeDON
Ze MoDux
c
IR MaDDa)
E Mar
Forccasting
X MrL
Imputation
Anomaly
Figure 5: Pipeline for Regression Tasks.
After the backbone, we have MMDNNN Then the linear head with flatten layer is used to obtain the final prediction:
(Flatten)
(5)
Where X E IR MTT, is the prediction of length T with M variables. Flatten(-) denotes flatten
layer that changes the final representation's 's shape to Z E M xxDxN), Head(-) indicates the linear projection layer that maps the final representation the final prediction.
Stationary Technique RevIN (Kim et al. 202 a special instance normalization for time series to mitigate the distribution shift between the training and testing data. In norm phase, we normalize the input time series per variable with zero mean and unit standard deviation before patching and embedding. Then in de-norm phase, we add the mean and deviation back to the final prediction per variable after the forward process.
Low Rank Approximation for Traffic Datasets To Traffic dataset that contains much more variables than others. directly applying our model Traffic dataset leads to heavy memory usage. Since the variables in multivariate times series have dependency solve this problem to find low rank approximation of these M variables when M a very big number. For example, FEDformer (2022) uses a low rank approximated transformation in frequency domain for better memory efficiency. And Crossformer (2023) also uses a small fixed number of routers aggregate messages from all variables save memory usage.
each other, possible way to
In this paper, we design a bottleneck structure as a simple and direct method to achieve this goal. In details, before fed into the ConvFENI and ConvFFN2, the variable number will be projected to M by a projection layer, where M' is much smaller than M. Then after the ConvFFNI and ConvEFN2 process, another projection layer is used project the variable number back M.
16

---

Published as a conference paper at ICLR 2024
Table 10: Results with different M' in Traffic dataset.
Models
w/o
M'
256
M'
64
M'
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Traffic 0.393
0.267
0.396
0.273
0.395
0.271
0.396
0.270
Memory Usage (%)
100%
82%
77%
72%
And we conduct experiments with different M' to verify this solution. As shown in Table 10 method can significantly reduce memory usage with only a little performance degradation. This result proves the fact that there is redundancy between the 862 variables in Traffic dataset. Therefore we can learn a low rank approximation of these 862 variables based on their dependency on each other. And such low rank approximation can help to reduce memory usage without too much performance degradation.
Input
The imput is M variables time series with input length L. In imputaiton tasks, the input series will further element-wing multipy with mask matrix to represent the randomly missing values.
Output In forecastiong tasks, the output is the prediction time series with prediction length T. In impuatation tasks, the output is the imputed input time series with input length L. In anomaly detection tasks, the output is the reconstructed input time series with input length L.
B.3 PIPELINE FOR CLASSIFICATION TASKS
The pipeline for classification shown as Figure
Patchify Embedding
input
Backbame
Head
output
X B Mxt.
R Mxooe
Ze R MoDeNN
ZeR (o(MxOxN)
X RG
Figure Pipeline for Classification Tasks.
There two difference: (1) We remove the RevIN; (2) The flatten layer is different. In classification tasks, the flatten layer changes the final representation' 's shape to Z RIX(MXDxN), Then projection layer with SoftMax activation is to map the final representation to the final classification result X E R XC,, where Cte is the number of classes.
Followings are implementation details and model parameters of each tasks.
C EXPERIMENT DETAILS
C.I LONG-TERM FORECASTING
Implementation Details Our method is trained with the L2 loss, using the ADAM (Kingma Ba 2014) optimizer with an initial learning rate of 10-4, 4 The default training process is 100 epochs with proper early stopping. The mean square еттог (MSE) and mean absolute error (MAE) are used as metrics. All the experiments are repeated times with different seeds and the means of the metrics are reported as the final results. All the deep learning networks are implemented PyTorchPPaszke et al. 2019) and conducted on NVIDIA A100 40GB GPU.
All the models are following the same experimental setup with prediction length {24, 36, 48, 60) for ILI dataset and T E (96, 192, 336, 720} for other datasets as (Nie et al. 2023) We collect some baseline results from (Nie et al. 2023) where all the baselines are re-run with various input length L and the best results For other baselines, we follow the official implementation and run them with vary input length L (24,
chosen avoid under-estimating the baselines.
36, 36, 48, 60, 104, 144) for IIII dataset and L {96, 192, 336, 512, 672, 720) for other
17

---

Published as a conference paper at ICLR 2024
datasets and choose the best results. All experiments repeated five times. We calculate the MSE and MAE of multivariate time series forecasting as metrics.
Model Parameter By default, Modern contains 1 ModemTTN block with the channel number (dimension of hidden states) D 64 and FFN ratio 8. The kernel size is set as large size = 51 and small size = Patch size and stride are set as P 8, = the patchify embedding process. For bigger datasets (ETTml and ETTm2), we stack 3 Modern blocks to improve the representation capability. For small datasets (ETThI, ETTh2, Exchange and ILI), we recommend a mitigate the possible overfitting and for better memory efficiency.
small FFN ratio
For baseline models, if the original papers conduct long-term forecasting experiments on the dataset use, follow the recommended model parameters in the original papers, including the number of layers, dimension of hidden states, etc. But we re-run them with vary input lengths as mentioned in Section 4.1 and choose the best results obtain a strong baseline.
Metric We adopt the mean square enror (MSE) and mean absolute error (MAE) for long-term forecasting.
T
MSE-
X.)
t=0
T Σ 1=0
MAE
X
X;
T
where X, X E RTXM are the M variables prediction results of length T and corresponding ground
truth. X, means the i-th time step in the prediction result.
C.2 SHORT-TERM FORECASTING
Implementation Details Our method is trained with the SMAPE loss, using the ADAM (Kingma & Ba 2014) optimizer with initial learning rate of x 10 4 The default training process 100 epochs with proper early stopping. The symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASH) and overall weighted average (OWA) used metrics. All the experiments are repeated times with different seeds and the means the metrics are reported as the final results.
Following (Wu et al. 2023), we fix the input length to be 2 times of prediction length for all models. Since the M4 dataset only contains univariate time series, remove the cross-variable component in Modern'TCN and Crossformer.
Model Parameter By default, ModernTCN contains 2 ModernTCN blocks with the channel number (dimension of hidden states) D 2048 and FFN ratio 1. The kernel size is set as large size = 51 and small size = 5. For datasets of less samples (M4 Weekly, M4 Daily and M4 Hourly), we smaller channel number D 1024. Patch size and stride are in the patchify embedding process. For datasets with shorter input length, we reduce the patch size and stride (c.g., P 3, S in M4 Yearly and 2, S 2 in M4 Quarterly).
P = 8, 4
Metric For the short-term forecasting, following (Oreshkin al. 2019), adopt the symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE) and overall weighted average (OWA) the metries, which can be calculated follows:
200 T
T
X
100 MAPE
X,
SMAPE
T
MASE
1
Σ
T
X;
X,1
OWA
SMAPE
+
MASE MASE
T
X,
X
2
SMAPE Naviee
Naive 5
1=1
T-p
18

---

Published as a conference paper at ICLR 2024
where p is the periodicity of the data. X, X E RT*M are the M variables prediction results of length and corresponding ground truth. X, means the i-th time step in the prediction result.
C.3 IMPUTATION
Implementation Details Our method is trained with the L2 loss, using the ADAM (Kingma & Ba 2014) optimizer with initial learning rate of 10-3, The default training process is 100 epochs with proper early stopping. The mean square error (MSE) and mean absolute error (MAE) are used as metrics. All the experiments are repeated times with different seeds and the means of the metrics are reported as the final results.
We use a mask matrix C e RLXM to represent the missing values in input time series Xin-
if not observed otherwise
c
Xin RLXM is the M variables input time series of length L. And L is as 96 imputation tasks. o7 is the value at I-th timestep in the m-th univariate time series.
The input the partially observed time series C X... and the output is the imputed time series of the input. @ indicates the element- wise multiplication between the two tensors Xin and C. And we only calculate MSE loss on masked tokens.
Model Parameter By default, MCdemTTN has 1 Modern block with channel number D
128 and FFN ratio = 1. The kernel size is set as large size = 71 and small size = 5. Patch size and stride are P = S avoid mixing the masked and un-maskded tokens.
Metric
We adopt the mean square error (MSE) and mean absolute error (MAE) for imputation.
C.4 CLASSIFICATION
Implementation Details Our method is trained with the Cross Entropy Loss, using the ADAM (Kingma & Ba 2014) optimizer with an initial learning rate of 10-3. The default training process 30 epochs with proper early stopping. The classification accuracy is used metrics. All the experiments are repeated times with different seeds and the means the metrics are reported as the final results.
Model Parameter By default, ModernTCN has 2 ModernTCN blocks. The channel number D is decided by min (max{2 log M1, dmin } dmax) (dmin is 32 and dmax is 512) following (Wu et al. 2023). The FFN ratio is r 1. Patch size and stride are set as P 1, S = 1 in the patchity embedding process.
Metric For classification, we calculate the accuracy as metric.
C.5 ANOMLY DETECTION
Implementation Details We takes the classical reconstruction task and train with the L2 loss. We use the ADAM (Kingma & Ba 2014) optimizer with an initial learning rate of 3 x 10-1 The default training process is 10 epochs with proper carly stopping. We use the reconstruction error (MSE) the anomaly criterion. The F1-Score is used as metric. All the experiments are repeated times with different seeds and the means of the metrics are reported as the final results.
Model Parameter By default, TCNernTCN has I ModernTCN block. The channel number D is decided by min mmx ( log Mn, dmmin; dmax} } (dmin is and dmax is 256) following (Wu et al. 2023). The FFN ratio is r = 1. The kernel size is set as large size = and small size 5. Patch size and stride P = 8, S in the patchify embedding process.
19

---

Published as a conference paper at ICLR 2024
Table 11: Impact of channel number. We conduct experiments with three different channel numbers ranging from D = {32, 64, 128). A lower MSE or MAE indicates a better performance.
ILI
ETTh!
Electricity Default
Datasets
Default K
1.
Default K
Prediction length
24
36
48
60
96
192
336
720
96
192
336
720
MSE
1.772
1.598
1.725
1.976
0.377
0.412
0.395
0.453
0.132
0.145
0.159
0.193
1) 32
MAE
0.857
0.873
0.866
0.953
0.401
0.419
0.416
0.464
0.227
0.241
0.256
0.286
D 64
MSE
1.347
1,250
1,388
1.774
0.368
0.405
0.391
0.450
0.129
0.143
0.161
0.191
MAE
0.717
0.778
0.868
0.413
0.412
0.461
0.226
0.239
0.259
0.286
D
MSE
2,010
1.751
1.378
1.806
0.364
0.402
0.387
0.449
0,135
0.147
0.168
0.196
MAE
0.913
0.932
0.792
0.935
0.390
0.410
0.407
0.459
0.236
0.247
0.265
0,290
Table 12: Impact of FFN ratio. We conduct experiments with four different FFN ratios ranging from r (1,2,4,8), A lower MSE or MAE indicates a better performance.
Dataeets
10
Exchange
CTTh
ETTml
Defoult
==
1.
64
Defoslt R
1.
64
Defasit R
D
64
Defonit = 3, 64
Prodiction length
24
36
60
|
96
192
356
720
96
192
356
720
96
192
356
720
MSB
1.347
1,250
1.388
1.776
0.080
0.106
0.507
0.656
0.368
0.805
0.391
0.450
0.294
0.335
0.309
0.419
MAD
0.717
0.778
0.781
0,868
0.196
0.288
0.398
0.582
0.594
0.413
0.412
0.051
0.346
0.309
0.392
0.421
2,083
1.940
1.758
0.050
0.368
0.407
0.992
0.201
0.365
MAF
0.943
0.820
0.946
0.800
0.106
0.280
0.583
0.895
0.415
0.461
0.455
0.368
0.392
0.415
1,877
2,02
0.050
0.367
0.411
0.292
0.416
MAE
0.887
0.853
0.960
0.197
0.289
0,556
0.393
0.419
0.463
0.316
0.417
MSB
1.017
0.46
0.912
0.395
0.477
Metric For anomaly detection, we adopt the F1-score, which is the harmonic mean of precision and recall.
Precision x Recall
F1-Score
Precision + Recall
D MORE ABLATION STUDIES
We conduct more ablation studies in long-term forecasting tasks.
D.1 RESULTS WITH DIFFERENT MODEL PARAMETERS
To see whether ModernTNN is sensitive to the choice of model parameters, we perform experi- ments with varying model parameters, including number of layers (number of ModemTTN blocks) ranging from K {1, 3, 4, 4,5). channel number (dimension of hidden states) ranging from D D= [32, 64, 128} and FFN ratio ranging from r = {1, 2,2,4, 4, 8). In general, except IIII dataset reveals high variance with different model parameter settings, other datasets
robust to the choice of model parameters. We conduct three experiments to figure out the impact of above three model parameters respectively. Detailed results are described in following paragraphs.
Results with Different Channel Numbers Table shows the impact of different channel numbers D. Considering both the parameter efficiency and forecasting performance, we set the default channel number as D= = 64. And the default channel number D = 64 works well for most of the datasets.
Results with Different FFN Ratios Table 12 shows the impact of different FFN Ratios r. Except for ILI dataset, our model is robust to the choice of the FFN ratio in other datasets. We recommend r 8 for most of the datasets. And for small datasets like ETTh1, ETTh2, Exchange and ILI, we recommend small FFN ratio like efficiency.
to mitigate the possible overfitting and for better memory
Results with Different Numbers of Layers Table 13 shows the impact of different numbers of layers (numbers of ModernTN blocks) K. Considering both performance and efficiency, one MC block is enough for most of the datasets. But for bigger datasets like ETTml and ETTm2, we recommend to stack more ModernTCN blocks like = 3 for better representation capability.
20

---

Published as a conference paper at ICLR 2024
Table 13: Impact of layer number. We conduct experiments with five different layer numbers ranging from K 3,, 5]. A lower MSE or MAE indicates better performance.
ETTh1
Electricity
ETTml
Datasets
Default /
64,
Default /) 64,
Default /) 64,
Prediction length
96
192
336
720
96
192
336
720
96
192
336
720
K
MSE
0.368
0.405
0,391
0.450
0.129
0.143
0.161
0.191
0.296
0.338
0,369
0.419
MAE
0.394
0.413
0.412
0.461
0.226
0.239
0.259
0.286
0.346
0.370
0.392
0.420
MSE
0.367
0.405
0.393
0.450
0.129
0.143
0.161
0.191
0.298
0.337
0.367
0.417
K
0.393
0.413
0.413
0.460
0.226
0.239
0.259
0.286
0.348
0.370
0.391
0.418
K
MSE
0.369
0.409
0.392
0.450
0.130
0.146
0.163
0.192
0.292
0.332
0.365
0.416
MAE
0.394
0.417
0.412
0.460
0.227
0.243
0,260
0.286
0.346
0.368
0.391
0.417
0.368
0.406
0.394
0.146
0.163
0.192
0.298
0.334
0.366
0.416
MAE
0.393
0.414
0.414
0.461
0.226
0,243
0,260
0,287
0.348
0.368
0.391
0.417
K
MSE
0.368
0.411
0.394
0.452
0.130
0.149
0.163
0.193
0.294
0.333
0.366
0.416
MAE
0.394
0.418
0.414
0.461
0.227
0.247
0,261
0.288
0.346
0.368
0.393
0.417
21

---

Published as a conference paper at ICLR 2024
D.2 IMPACT OF INPUT LENGTH AND PATCHING SETTINGS
Impact of Input Length. Since a longer input length indicates more historical information an algorithm utilize in time series forecasting, a model with strong ability capture long-ferm temporal dependency should perform better when input length increases (Zeng et al. [022] Wang al., 2023 Nie al. 2023). To validate our model, we conduct experiments with different imput lengths under the same prediction length. As shown in Figure 7 in general, our model gains performance improvement with increasing input length, indicating our model can effectively extract useful information from longer history and capture long-term dependency. However, some Transformer-basee models (Wu et al. 2021 Zhou et al. 2021 Vaswani et al. 2017) suffer from performance degradation with increasing input length owing to the repeated short-term patterns according to (Zhou et al. 2021
0.42
RELVET
RATTET
0.00
0.91
3.94
0170
0.50
802
Electricity
Traffic
Weather
Figure 7: The MSE results with different input lengths and same prediction length (192 time steps).
Impact of Patch size and Stride To verify the impact of patch size P and stride S, we perform experiments with two patching modes (P S and P shown on Table 14 In general, the performance doesn 't vary significantly with different P and S, indicating the robustness of our model against these two hyperparameters. The ideal and S may vary from different datasets. We recommend / & and S = 4 as general good choice for most of the datasets.
S) and two different S. Results are
Table 14: Impact of patch size and stride. We compare four patching settings. A lower MSE or MAE indicates better performance.
Datasets
ETThl
ElTml
Electricity 64,
Default
64.r
1,K
Default D
64, 8, K. 3
Default
8,
Prediction length
96
192
336
720
96
192
336
720
192
336
720
P 4,5
MSE
0.369
0.409
0,391
0.449
0.295
0.336
0.369
0.417
0.130
0.145
0.160
0.191
MAE
0.395
0.417
0.412
0,460
0.348
0.372
0.393
0.418
0,226
0.240
0,259
0,286
P= 1==85=0
MSB
0.368
0.405
0.391
0.450
0.292
0.352
0.365
0.416
0.129
0.143
0.16]
0.191
MAE
0.394
0.413
0.412
0.461
0.346
0.368
0.391
0.417
0,226
0.239
0.259
0.286
P 8,5
MSE MAE
0.400
0.297
0.372
0.147
0.195
0.402
0.421
0.421
0.466
0.349
0.371
0.395
0.421
0.227
0.241
0.263
0.290
0.419
P 16,5
MAE
0.403
0.420
0.419
0.466
0.350
0.371
0.395
0.420
0.228
0.240
0.263
0.290
E UNIVARIATE LONG-TERM FORECASTING RESULTS
Here we provide the univariate long-term forecasting results on 4 HTT datasets. There a target feature oil temperature within those datasets, which is the univariate time series that we are trying to forecast. Since it's univariate tasks, we mainly focus on capturing cross-time information and don't need to capture the cross- coos--aaiablee information. Thus, we remove the cross-variable component in ModernTCN. As shown in Table 15 thanks to the larger ERF and better temporal modeling ability in DWCony, our ModernTNN can achieve comparable performance with the state-of-the art Transformer-basee model PatchTST((02)) and MLP-based model DLinear 2022) in univariate forecasting tasks.
22

---

Published as a conference paper at ICLR 2024
Table 15: Univariate long-term forecasting results ETT datasets. Following PatchTST 2023 and
DLinear (2022). input length is fixed as 336 and prediction lengths are T {96, 192, 336, 720]. The best results are bold.
Models
ModernTCN
PatchTST 2023
FEDfomerr 20222)
Autoformer (2021]
Informer 2021
Trans 2019h
(Ours)
2022
Metric
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
96
0.055
0.179
0.055
0.179
0.056
0.180
0.079
0.215
0.071
0.206
0.193
0.377
0.283
0.468
E1 E1
192
0.070
0.205
0.071
0.205
0.071
0.204
0.104
0.245
0.114
0.262
0.217
0.395
0.234
0.409
336
0.074
0.214
0.076
0.220
0.098
0.244
0.119
0.270
0.107
0.258
0.202
0
0.386
0.546
720
0.086
0.232
0.087
0.236
0.087
0.359
0.142
0.299
0.126
0.283
0.183
0.355
0.475
0.629
0.124
0.274
0.129
0.282
0.131
0.279
0.128
0.271
0.153
0.306
0.213
0.373
0.217
0.379
192
0.164
0.321
0. 168
0.328
0.176
0.329
0.185
0.330
0.204
0.351
0.227
0.387
0
0.429
336
0.171
0.336
0.171
0.336
0.209
0.367
0.231
0.378
0,246
0.389
0.242
0.401
0.293
0.437
720
0.228
0.384
0.223
0.380
0.276
0.426
0.278
0.420
0.268
0.409
0.291
0.439
0,218
0.387
96
0.121
0,026
0.123
0.033
0.140
0.277
0.049
192
0.040
0.152
0.039
0.150
0.045
0.156
0.058
0.186
0.081
0.216
0.151
0.310
0.157
0,317
336
0.053
0.173
0.053
0.173
0.061
0.182
0.084
0.231
0.076
0.218
0.427
0.591
0.289
0.459
720
0.073
0.206
0.073
0.206
0.080
0.210
0.102
0.250
0,110
0.267
0.438
0.586
0.430
0,579
96
0.065
0.183
0.065
0.067
0.198
0.225
0.075
0.208
192
0.095
0,232
0.093
0.231
0.092
0.227
0.102
0.245
0.118
0.256
0.132
0.283
0.129
0.275
336
0.119
0.261
0.120
0.265
0.119
0.261
0.130
0.279
0.154
0.305
0.180
0.336
0.154
0.302
720
0.173
0,323
0.171
0.322
0,175
0.320
0.178
0.325
0.182
0,335
0.300
0.435
0.160
0.321
Count
21
F TECHNIQUE TO BETTER TRAIN A LARGE KERNEL
F.1 INTRODUCTION ABOUT STRUCTURAL RE-PARRAMETERIZATION
According to DDng et al.
2022 Liu et al. 2022b), we can use the Structural ce-parameterization
technique better train large kernel convolution. In training phase, additional Batch normalization (BN) layers (Toffe & Szegedy 2015) are used following the depth-wise convolution layers to form convolution-Be branches for hetter training result. Then the large size depth- wise convolution-BN branch is trained with parallel small size depth-wiss wise convolution-Be BN branch to make up the optimization issue of large kernel. After training, each of these two branches is transformed into single depth-wise convolution layer by fusing the BN parameters into the convolution kernels (Ding et al. 202T). Then the small kernel is zero-padded on both side to large size. After two kernels aligned to the same size, the two depth-wist wise convolution layers add up merge into single large kernel depth-wise convolution layer. Then the fused single large kernel depth-wise convolution can be used in inference phase. It's notable that the resultant large kernel model is totally equivalent to the model in training phase but no longer has small kernels. See Figure g for an example of Structural pe-aammtttrriztiion after training.
Depth wise Can- lanas size
Depik wine Coavid small sice
Training
Depth-was Courtd large dose
Depth les Camiid smallsize
Fuse BN
BatckNorald
BatchNoraldd
Klement-reiar Sum
Weight manris
Weight matris Zern purling on both
Strectural Re-prrametertaatinn
Depti-wiie Convid laras
Weight motris
For Inference
Figure& 8: An example of Structural Re-parameterization.
23

---

Published as a conference paper at ICLR 2024
Table 16: Impact of kernel size. We compare three different kernel sizes ranging from small to large. A lower MSE or MAE indicates a better performance. The best results are highlighted in blod.
Datasets
ETThi
Electricity
Prediction length
24
36
48
60
96
192
336
720
96
192
336
720
kernel size =3
MSE
1,906
1,546
1.754
1,893
0.381
0.416
0.403
0.460
0.145
0.155
0.175
0.203
MAE
0.862
0.841
0.891
0.900
0,405
0.423
0.419
0.470
0.237
0.249
0.270
0.299
kernel size
MSR
1.687
1,486
1.855
0.405
0.449
0.147
0.193
MAE
0.848
0.855
0.797
0.929
0.393
0.413
0.410
0.460
0.228
0.243
0.257
0.288
MSE
1.347
1,250
1,388
1.774
0,368
0.405
0.391
0.450
0.129
0.143
0.161
0.191
kernel size =
MAP
0.717
0.778
0.781
0.868
0.394
0.413
0.412
0.461
0.226
0.239
0.259
0.286
Table 17: Impact of small kernel sizes. A lower MSE or MAE indicates a better performance.
Datasets
ETThl
ETTml
Electricity
Prodiction length
192
336
720
192
720
192
336
720
small size
MSB
0.369
0.406
0.390
0.450
0.296
0.346
0.373
0.441
0.130
0.145
0.160
0.191
MAE
0.395
0.414
0.410
0.461
0.348
0.377
0.397
0.432
0,226
0.240
0,259
0,286
small size 3
MSE
0.369
0.407
0.391
0,450
0.295
0.336
0.369
0.423
0.129
0.143
0.161
0.191
MAE
0.395
0.415
0.412
0.461
0,348
0.371
0,393
0.425
0,226
0.239
0,259
0,286
small size 5
MAE
MSE
0.394
0.368
0.405
0.413
0.391
0.412
0.461
0.450
0,346
0.292
0.368
0.332
0.365
0,391
0.416
0.417
0.129
0.226
0.143
0.239
0.161
0.259
0.286
0.191
small size 7
MAE
MSR
0.394
0.405
0.413
0.411
0.449
0.460
0,347
0.341
0.376
0.397
0.419
0.421
0.228
0.240
0.168
0,263
0.196
0.290
small size
MSE
0.368
0.406
0.391
0,450
0.296
0.351
0.375
0.428
0.133
0.146
0.168
0.196
MAF
0.394
0.414
0.412
0.462
0.348
0.380
0.398
0.424
0.228
0.240
0.263
0.290
F.2 ABLATION OF STRUCTURAL PAAAMTTERRRIANNI
F.2.1 IMPACT OF LARGE KERNEL SIZE
According (Ding et al. 2022 large kernel size is the key obtain large ERF in 2D convolution. To verify whether this finding still works on 1D convolution and to figure out the impact of kernel size, we perform experiments with different kernel sizes ranging from small large on datasets. Results on Table T6 show that increasing the kernel size leads to performance improvement. The experiment results indicate that directly enlarging the kernel size in ID convolution layer and training it with Structural Re- parameterrization technique can effectively increase ERF and help convolution layer to better capture temporal dependency.
F.2.2 RESULT WITH DIFFERENT SMALL KERNELS
According to (Ding al. 2022 Liu et al. 2022b), adding parallel Structural Re-parameterizattoon branch with a small kernel can help to train the large kernel convolution layer. To further figure out the impact of different small kernel sizes, we perform experiments with five different small kernel sizes ranging from 1 to 9. As shown in Table 17 the performance is robust to the choice of small kernel sizes as long as they't 're much smaller than the large kenrel. And small size 5 is general good choice.
F.2.3 RESULT WITH MORE PARALLEL BRANCHES
In Structural Re-parameterization technique, we usually add one additional Structural Re- parameterization branch paralleled to the large kernel convolution branch to make up its optimization issue. Here we perform experiments with more branches to see the impact of parallel branches number. As shown in Table the performance robust to the choice of parallel branches numbers. Considering both performance and efficiency, we only add one Structural Re-parameterizattion branch in our main experiments.
24

---

Published as a conference paper at ICLR 2024
Table 18: Impact of more parallel branches. A lower MSE or MAE indicates a better performance.
Datasets
ETTh1
ETTml
Electricity
Prediction length
96
192
336
720
96
192
336
720
96
192
335
7:00
61
MAE
0.394
0.413
0.412
0.461
0.346
0.368
0.391
0.417
0.226
0.239
0.259
0.286
51
MSE
0.368
0.406
0.396
0.451
0.292
0,338
0.374
0.425
0.130
0.144
0.162
0.190
0.345
0.421
0.261
51 - 5 3
MAE
MSE
0.368
0.394
0.406
0.414
0.393
0.413
0.463
0.451
0.293
0.346
0.340
0.373
0.374
0.423
0.421
0.131
0.227
0.144
0,239
0.160
0.259
0.288
0.192
51 7
MAE
0.394
0.368
0.406
0.414
0.393
0.413
0.462
0.452
0.291
0.345
0.343
0.374
0.375
0.398
0:420
0.422
0.131
0.226
0.240
0.144
0.260
0.161
0.191
0.287
0.395
0.291
0.424
0.163
0.191
MAB
0.399
0.414
0.415
0.461
0.344
0.371
0.394
0.422
0.226
0.240
0.262
0.286
25

---

Published as a conference paper at ICLR 2024
G DETAILS OF MODERNTCN BLOCKS
G.1 BACKGROUND
Group Convolution Group convolution (Krizhevsky et al., 2017 Xie et al. 2018) divides the convolution channels into separate groups. Only the channels in the same group can interact with each other while the channels in different groups are independent.
2017 Zhang et al.
Depthwise Separable Convolution Depthwise convolution (Howard et al., 2017) can be seen as a special group convolution in which the number of groups is equal to the number of channels. Therefore, all channels in depthwise convolution are independent. The depthwise convolution only mixes information among tokens across the temporal dimension.
Since depthwise convolution layer totally chamnel-independent. It can not combine channels to create new representations. Therefore, the pointwise convolution layers, which are in charge of mixing channel information to create new representations, should be used following the depthwise convolution complementary. The kernel size of pointwise convolution is 1. Therefore it is applied to each token independently and only mixes information among channels.
Modern Convolution Modern convolution (Liu et al. 2022d Ding et al. 2022 Liu et al. 2022b)
a new convolution paradigm inspired by Transformer. The basic components in modern convolution block are depthwise and pointwise convolution layers. Inspired by the architectural designs in Transformer, the depthwise and pointwise convolution layers organized into a similar structure to Transformer block, which is shown in Figure 2 (a) and (b).
As shown in Figure 2 (b), the DWConv is depthwise convolution layer which is responsible for learning the temporal information among tokens on per-channel basis. It plays the same role as the self-attention module in Transformer. Meanwhile, large kernel is adopted in DWConv catch up with the gloabl effective receptive field in Transformer.
ConvFFN consists of two point-wise convolution layers (PWConvs) and adopts inverted bottleneck structure, where the hidden channel of the ConvFFN block is r times wider than the input channel (Figure (c)) ConvFFN plays the same role as the FFN module in Transformer blocks, which is to learn the new representation of each token independently.
G.2 DETAILS OF MODERNICN BLOCK DESIGNS
The input the i-th ModernTCN block is Z, MXDxN, where M, D and N the size of variable dimension, feature dimension and temporal dimension respectively. We merge the feature dimension and variable dimension before feeding the embedded time series into the ModemnTNN block. Therefore the convolution channel number is M X D.
As shown in Figure (d), in the ModemnTCN block, DWConv is depthwise convolution layer which maps M x D imput channels to M D output channels. The group number in DWConv is set as M x D make each channel independent. Therefore, DWConv is both variable and feature independent. It only mixes information across temporal dimension. And we set a large kernel size for DWConv to enlarge effective receptive field and improve its temporal modeling ability.
The ConvFFN1 and ConvFFN2 in ModernTCN block are the decoupled version of ConvFFN based on the idea of group convolution. We replace the two pointwise convolution layers in ConvFFN with two group pointwise convolution layers with group number as M to obtain ConvFFN1. Similarly, we replace the two pointwise convolution layers in ConvFFN with two group pointwise convolution layers with group number as D to obtain Con2
In details, the input channel number in ConvFFNI is M x D. By setting group number as M, the M x D input channels will be divided into M groups And only the D features in the same group can interact with each other. Since each group represents a variable, it means that only the features of the same variable interact with each other to create new feature representation while the features of different variables are independent. Therefore ConvFFN1 can learn the new feature representation for each variable independently.
26

---

Published as a conference paper at ICLR 2024
The same goes for ConvFFN2. After permute operation, the input channel number in ConvFFN2 is D x M. By setting group number as D, the D x M input channels will be divide into D groups. And only the M variables in the same group interact with each other. Therefore ConvFFN2 can capturing the cross-variable dependency per feature.
G.3 PARAMETERS OF CONVOLUTION LAYERS IN MODERNTCN BLOCK
We provide the parameters of convolution layers in ModernTNN block in Table 19
Table 19: Parameters convolution layers in ModemnTCN block.
Module
imput channels
output channels
kernel size
stride
group number
DWConv
M x
M
large size
M
D
First PWConv
M D
TX M X D M x D
M
ConvPFN1
Second PWConv
x MXD
M
First PWConv
DXM
F
1
D
ConvFFN2
Second PWConv
TXDxM M
D M
1
D
H DETAILS OF ABLATION STUDY
In Section 5.3 we provide an ablation study on our model designs. In this appendix, we provide more details about this ablation study.
H.1 DETAIL OF DIFFERENT EMBEDDING SETTINGS
we design the patchify griable-independenn embedding to maintain the variable dimension, replacing the common patchify variable-mixing embedding which will lead the discard of variable dimension in the embedded series. The difference of the two embeddings is shown on Figure 9 and introduced follows:
In Section 3.2.
In variable-indepenndnt embedding, treat each univariate time series in the multi-variate time series as an individual sample with 1 feature and embed them independently.
In variable-mixing embedding, we treat the whole multi-variate time series as a single sample with M features.
Conild Kerezi azc-!t.
Variable
Psacking and Embeeiing per Variali
12pe
Uhegpenn
siidens.
cappet
e88) 0BR) Mob
X, Mos
Chrmdde (-f)).
X_ glot
Graepe
(a) Patehilyy Variatile Vartatti-tadepeaeeee Emahediing
Exerld Kersel storr. ande--S
Viriable
Variable
Pulching and Exahedzins winty
Unsqpezac
Varibble
gani
ChandicM--DD.
Grappet
X_ eigaa
X_
Varibhl-niing Emsholding
Figure 9: Different embedding methods in ablation study.
The essential difference behind these two embedding methods is the opinion of whether should maintain the variable dimension when analysing multivariate time series. Since applying the variable-mixing embedding the multivariate time series will discard its variable dimension, denote as the setting Discard Variable Dimension. To verifies our opinion to maintain the variable dimension, use the setting Discard Variable Dimension for comparision. In this setting, apply
27

---

Published as a conference paper at ICLR 2024
pariable-mixing embedding to the multivariate input time series discard its variable dimension. Then the embedded series is fed into a backbone stacked by convolution blocks like Figure [T to learn representation. As shown in Table the setting Discard Variable Dimension leads to significant performance degradation, which verifies the effectiveness of our variable-independent embedding design and highlights the importance to maintain the variable dimension.
H.2 DETAIL, OF DIFFERENT BLOCK DESIGN SETTINGS
Then we conduct experiments to study the impact of different block designs. All block design settings we used are shown in Figure [Q Since variable-mixing embedding will lead to severe performance degradation, we conduct these experiments with our gariable-independen embedding.
Given that variable- inneaede-nndeendeent embedding will maintain the variable dimension, we need to make our structure able to capture information from the additional variable dimension. A direct and naive way is to jointly learn the dependency among features and variables only by a single ConvFFN, which is denoted as Undecoupled ConvFFN. But such setting leads to higher computational complexity and worse performance, which is shown in Table (4] In contrast, the combination of our decoupled two ConvFFNs (denoted as ConvFFNI + ConvFFN2) achieve the best performance, which proves the necessity and effectiveness of our decouple modification to ConvFFN module.
We further study how much ConvFFN1 and ConvFFN2 contribute to the forecasting respectively. Only ConvFFN/ can learn each variable's ' deep feature representation independently but doesn't 't take cross-variable dependency into consideration. Only ConvEFN2 can only capture the -ross-variable dependency per feature. But to each variable, it omits to learn their deep feature representations. As a result, the performance decreases on both settings.
And we also include settings like MRnvFFNI+ConFFFFFi the impact of the number of ConvFFNs. The results show that the combination of ConvFFNI and ConvFFN2 is the key to performance improvement, not the the number of ConvFFNs.
and ConvFFN2+
to eliminate
Z, R Malten
inpu
Z, R Midaack
Reshape Z, Rdintoss
R took
Z, F R.R.Moon
Reshape
DVVCoay
Z, BRiinos
C. PWDar
DWC
DnCCone
ConnFNNi
Grapcc
GaLu C.
R MMMaas
PWyCor
ConFFNN
Rolupe &Pormute
Z E. Aotoiss
cR MiMiin
Com Grugel
Ze F Rlldby
Bishaps &Perrnat
ougut Z,, R Moluk
Z E
ouput
Rolupe &.Perutt
Z, RonA
Discard Variable Dimonsion Used with V Variahbo-niiing Emshedding
(bi Cndeecuple CoaYFFN Uied with Vartahle-anderaniie Embolling
(c) Cam CuaNF (MadernTCN block)
Structure CamePyy
input
Mi
kypu
e 3 MesDu
Z, R tiiiap
Reshape
Reshage Riiwnma
R Nontu
Rolupe
Z R Maastas
Reshape
Z, R
Z, g gaatta
Reshape Permute uinater
DWCl
DWCComy
7 c R giinoow
ConvEEN: Grops:M
Z e gicinan
Cone GroapeD
Z, Ritnnww
BN
Reshape & Peruntte
2, 2httoass
z e gioaaa
CandFFNi
ConoTTI22
GrospeM
GropeD
OmniTNi
C2
Z" e 2tctsrra
Gropeth
Groppe))
Rolape
Reshape & &.Perntte
Z a Ritumae
Z R Melus
Z,, Mitoo
Reshape
Reshuge& Permute
corpu
eupat
Zrn R
Zin R tnow
ouppu
(c) ConnFFNI-CiurFFNI
(0 Can FFN
(g) unly ConvFFNI
(b) only Coun FFN2
Figure 10: Different block designs in ablation study.
28

---

Published as a conference paper at ICLR 2024
1
MORE COMPARISON WITH TIMESNET IN TERMS OF ERF
200
(a) Modern (ours)
layer
1D Convolution (ks=51)
50
100
150
200
250
300
(b) TimesNet
layers
2D Convolution (k5=1,,,,,799111)
Figure 11: The Effective Receptive Field (ERF) of ModemTCN and TimesNet (2023). A more widely distributed light area indicates a larger ERF.
Modern and TimesNet successfully enlarge the ERFs. But the ERFs of these two models have quite different property (Figure IT.
The ERFs of ModernTCN and other ID convolution-based models concentrated the middle point and continuously expand to both ends, which means that the final representation of the middle point is highly related its adjacent points. Such phenomenon demonstrates the locality, which is common property of convolution.
The ERF of ModemTTN than other ID convolution- models. But at the same time, the ERF of Modern is still concentrated the middle point. So ModernTCN is able to capture long-term dependency while focusing on the local context.
expand to wider range. Therefore ModemnTN has larger ERF
However, the ERF of TimesNet is discrete and not concentrated at the middle point, which doesn't reflect the locality of convolution. This is because the additional 2D data transformation in TimesNet also influences the pattern of ERF. In 2D data transformation, the time series is divided into several segments and further rearranged in 2D space based on periods. In this process, time point may be separated from its adjacent points, thereby losing continuity and locality. As result, the ERF and performance of NetesNet are also highly (or even mainly) related to its special 2D data transformation, not just depend on convolution.
In summary, influenced by its special 2D data transformation, TimesNet' ERF is of quite different models' and doesn' 't reflect the locality convolution.
property from other ID convolution- Although both ModernTCN and TimesNet can successfully enlarge the ERFs, Modern TCN can also maintain the locality when enlarging the ERF, therefore providing better performance than TimesNet.
29

---

Published as a conference paper at ICLR 2024
J APPLY CROSS-VARIABLE COMPONENT TO VARIABLE-INDPPENDEN MODELS
We apply our cross-variable component (ConvFFNs)
variable-independemt models like DLinear (2022) and PatchTST (2023). We make some necessary modifications to our cross-variable component adapt it to these two models, which summarized as follows and shown on Figure 12
For DLincar which only has two individual Linear layers (one for trend part and the other for seasonal part), we add an additional ConvFFN module before the Linear layer. This additional ConvFFN module can mix information across variable dimension to incorporate the cross-variable information.
For PSt we replace the original FFN module PatchTST block with our Con- vFFNs but with some difference in residual connections to align with the block design in Transformer.
ixpul
Z, trane
Sel-emerttrr
PWCou
Heshape
BN
R totros
7: ghiniaas
Z. ca
ConvETNi
Demppel
PWCCrr
GroepeM
Gell
rx C,
Z c hoMe
2," giuosas
X, + Sa
CrenFFN2
C,
Lieax
Googst)
GraperD
Z. c R.Ritture
Z, Mo
Reskape &Prrmt
Rashape &.Permate
2,,, ttolu
Z,,. C won. C.Rwn
(6) DLanner with CHINTIN
(b) PobATTT Bleck with CONUTINN
MideesTCN blrk
(1) Struttree of COMNEN
Figure 12: Applying varsalv component to DLinear (022) and PatchTST (2023)
We conduct experiments in imputation tasks, where cross-variable dependency plays important role. We report the performance promotion of each model in Table In imputation tasks, equipped with our cross-variable component achieves averaged 28.7% promotion on PatchTST and 19.7% promotion on DLinear. The result validates that our cross-variable component can be used on top of other variable-inde indeeeended models, helping them incorporate the cross-variabee information and improving their performance.
Table 20: Performance promotion by applying our crossevariable component to PatchTST and DLinear imputation tasks. We report the averaged MSE/MAE of all four mask ratios and the relative MSE reduction ratios (Promotion) by our tross-variable component.
Dataset
ETTml
ETTm2
ETThl
ETTh2
Electricity
Weather
Model
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
PatchTST
0.045
0.133
0.028
0.098
0.133
0.236
0.066
0.164
0.091
0,209
0.033
0.057
+ Ours
0.026
0.107
0.020
0.087
0.083
0.192
0.048
0.145
0.069
0.177
0.029
0.050
Promotion
42.2%
28.6%
37.6%
27.3%
24.2%
12.1%
DLinear
0.093
0.206
0.096
0.208
0.201
0.306
0.142
0.259
0.132
0.260
0.052
0.110
Ours
0.065
0.175
0.089
0.199
0.115
0.236
0.136
0.251
0.120
0.253
0.039
0.091
Promotion
30.1% 9%
7.3%
42.8%
4.2%
9.0%
25.0%
30

---

Published as a conference paper at ICLR 2024
K SHORT-TERM FORECASTING RESULTS ON MULTIVARIATE TIME SERIES DATASETS
We conduct short-term forecasting on 9 multivariate time series datasets, including Weather (Wetter station), Traffic (PMMS). Electricity (UCI), Exchange (Lai et al. 2018a) IIII (CDC) and four ETT datasets (Zhou et al. 2021).
The experiment details are as follows:
We choose the prediction lengths as T E {6, 12, 18), which meets the prediction length in M4 datasets. And following M4 short-term forecasting tasks, set input length L to be times of prediction length T.
We choose models that perform well in multivariate datasets (DLinear, RLinear, RMLP and PatchTST) or perform well in short-term forecasting tasks (PatchTST, CARD and NemesNet) as strong baselines.
All models follow their official configurations on above datasets, we only change the input lengths and prediction lengths.
We calculate MSE and MAE of the multivariate prediction results as metric.
The results are shown in Table 21 ModernTCN can outperform above competitive baselines in most cases, indicating that MCNernTCN can better use limited input information to provide better forecasting results.
Table 21: Short-term forecasting on multivariate time series datasets. We set prediction lengths as T E {6, 12, 18) and set input length L to be 2 times of prediction length T. The best results are in bold.
Madels
ModernTCN
PatchTST 2023
CARD
TimesNet
Di innear
R[inear 2023a]
RMLP 20233a
(Ours)
2023
2023
[022]
Metric
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
F1 11 E11ml ETTh
0.505
0.423
0.776
0.545
0.754
0.530
0.785
0.558
0.941
0.649
1.184
0.676
0.682
0.504
12
0.316
0.357
0.438
0.431
0.383
0.397
0.412
0.424
0.559
0.504
0.701
0.541
0.406
0.414
0.317
0.359
0.418
0.418
0.374
0.392
0.434
0.439
0.535
0.485
0.677
0.409
0.418
6
0.131
0.218
0.181
0.250
0.161
0,235
0.138
0.225
0.272
0.328
0
0.309
0,162
0.234
12
0.216
0.279
0
0.331
0.294
0.321
0.231
0.289
0,402
0.396
0.555
0.437
0,301
0.317
18
0.31
0.329
0.409
0.388
0.389
0,366
0.323
0.340
0
0.445
0.628
0.471
0.410
0.375
0.132
0.227
0.159
0.264
0.157
0.260
0.162
0.267
0.199
0.327
0.213
0.310
0.150
0.252
12
0.144
0.240
0.175
0.273
0.170
0,268
0.169
0.266
0.219
0.337
0.205
0.295
0,174
0.273
18
0.159
0.253
0.181
0.273
0.178
0,270
0.200
0.294
0.215
0.319
0.247
0.325
0.192
0.284
0.060
0.138
0.066
0.155
0.065
0.151
0.062
0.144
0.083
0,200
0.083
0.176
0.065
0.150
12
0.079
0.165
0.089
0.186
0.089
0.181
0.080
0.170
0.109
0.228
0.118
0.219
0.092
0.186
0.093 18
0.182
0.110
0.211
0.112
0,208
0.098
0.191
0.125
0.242
0,130
0.232
0.114
0.210
0.199
0.306
0.661
0.597
0.333
0.391
0.205
0.307
1.021
0.836
1,386
0.931
0.651
0.571
Wealher BCL
12
0.117
0.221
0.232
0.340
0.152
0,239
0.130
0.234
0.394
0.491
0.507
0.531
0.212
0.296
18
0.117
0.219
0.237
0.348
0.158
0.243
0.133
0.240
0.341
0.440
0.492
0.523
0.216
0.300
0.058
0.069
0.065
0.083
0.063
0.078
0.070
0.088
0.081
0.145
0.109
0.135
0.065
0.083
0.071
0.086
0.085
0.108
0.084
0.101
0.086
0.114
0.096
0.161
0.124
0.152
0.091
0.111
0.083
0.105
0.099
0.126
0.098
0.119
0.105
0.141
0.116
0.188
0.085
0.100
0.106
0.130
1.419
0.694
3.402
1.213
2.793
1.079
1.101
0.597
3.407
1.255
3.378
1.170
2.485
1.024
2.420
0.939
3.958
1.427
2.760
0.927
5,597
1.753
6.106
4.271
1.461
18
2.394
0.983
4.905
1.594
3.774
1.402
3.516
1.014
4.875
1.666
5.953
1.815
4.327
1.497
Mcchinge Traftic
0.937
0.414
1.023
0.586
0.790
1.064
0.402
1.785
2.392
1.414
0.736
0.461
0.301
0.618
0.416
0.453
0.289
0.566
0.297
0.964
0.608
1.475
0.809
0.662
0.414
0.453
0.300
0.653
0.429
0.468
0,305
0.587
0.305
0.933
0.574
1.321
0.731
0.691
0.430
6
0.008
0.052
0.010
0.068
0.010
0.065
0.009
0.063
0.048
0.175
0.014
0.078
0.009
0.061
0.013
0.072
0.017
0.089
0.017
0.087
0.018
0.089
0.038
0.148
0.023
0.106
0.017
0.089
18
0.019
0.089
0.023
0.104
0.023
0,103
0.025
0.107
0.048
0.171
0.032
0.127
0.024
0.108
Count
49
U
2
0
31

---

Published as a conference paper at ICLR 2024
L ABLATION STUDY ABOUT REVIN ON REGRESSION TASKS
L.1 ABLATION STUDY ABOUT REVIN ON LONG-TERM FORECASTING
We conduct ablation study on Weather, ETTh1 and ETTml datasets for long-term forecasting tasks.
The results in Table 22 show that:
Although there is a slight degradation in performance without RevIN, ModemTCN still achieves competitive performance. Modern can still achieve significantly better performance than some baselines and compete favorably with PStchTST in the same case of no RevIN. The results indicate that our designs in ModernTCN also make great contribution to the performance improvement.
We also find that RevIN doesn't 't provide consistent improvement under some settings (e.g.. prediciton length 192 and 720 in Weather dataset), which means our ModernTN can directly predict from the non-stationary time series some degree. This finding further confirms that our designs in ModemTCN can bring great performance improvement on their own. And such phenomenon mainly happens in Weather dataset for its degree of stationarity is not very high according to (Liu et al. 2022c).
Table 22: Ablation Study about RevIN on long-teny forecasting tasks. We report MSE and MAE under four different prediction lengths as metrics. +Rev means with RevIN. -Rev means without
RevIN.
Models
ModemTTCN (-Rev)
ModenTCC (-Rov)
PacchTST (+Rev)
PaichTST
FEDformer
Auteformer
Metric
MSE
MAE
MSB
MAE
MSB
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
ETTh Wealher
0.149
0.200
0.155
0.233
0.149
0.161
0.219
0.329
0.354
0.405
192
0.196
0.245
0.196
0.249
0.194
0.241
0,201
0.254
0.275
0.329
0.325
0.370
0.419
0.44
336
0.238
0.277
0.247
0.323
0.245
0.282
0.253
0.298
0.339
0.377
0.351
0.391
0.583
0.543
720
0,314
0.334
0,312
0.366
0.314
0.334
0.323
0.357
0.389
0.409
0.415
0.426
0.916
0.705
0.406
0.410
0.941
0.769
192
0.405
0.13
0,418
0.426
0.413
0.417
0.432
0.425
0.456
0.477
1.007
0.786
336
0.391
0412
0,413
0.434
0.422
0.436
0.439
0.449
0.444
0.462
0.486
0.487
1.038
0.784
720
0.450
0.461
0.487
0.500
0.447
0,466
0.478
0.494
0.469
0.492
0.515
0.517
1.144
0.857
96
0.292
0.346
0.313
0.362
0.290
0.342
0.306
0.358
0.390
0,510
0.492
0.626
0.560
192
0.332
0.375
0.619
336
0.365
0.391
0.367
0.393
0.366
0.392
0.389
0:411
0.392
0.425
0.510
0:492
1.005
0.74]
720
0,416
0.417
0.427
0.426
0.416
0,420
0.430
0.439
0.446
0.458
0.527
0.493
1.133
0.845
L.2
THE IMPACT OF REVIN ON DIFFERENT TASKS AND MODELS
To study the impact of RevIN different tasks and models, we further conduct experiments on four regression tasks with 6 models: ModernTCN, PStchTST (2023), TimesNet 2023). (2022),
RMI.P and RLinear 2023a). Details are as follows:
We conduct ablation study about RevIN on long-term forecasting with more models. And the results are provided in Table 23
We conduct ablation study about RevIN on imputation tasks in Weather, ETThl and ETTml datasets. And we report the MSE and MAE under four different mask ratios as metrics. Results are shown in T2ble|24]
We conduct ablation study about RevIN on short-term forecasting tasks in M4 datasets and roport the weighted averaged results from six M4 sub-datasets metric. Results shown in Table 25
We conduct ablation study about RevIN on anomaly detection tasks in SMD, MSL and PSM datasets and roport the F1-score as metric. Results are shown in Table 26
RevIN's impact on different tasks In general, removing the RevIN will lead to performance degradation in all four tasks because it is more difficult to predict from the non-stationary time series (Liu et al. 2022c Kim et al. (021). Given that stationary techniques like RevIN or decomposition (e.g., MICN, DLinear, FEDformer)
transform the input into stationary series that is easier to analyze with deep learning methods, it is generally necessary to adopt stationary techniques in regression tasks.
32

---

Published as a conference paper at ICLR 2024
We also find that short-term forecasting is more sensitive to RevIN. The samples in M4 datasets collected from different sources and have quite different temporal property. Therefore, there is a greater need of RevIN to mitigate the distribution shift.
RevIN's impact on different models Although removing RevIN will cause performance degra- dation on all 6 models in our experiments, our ModemnTC is one of the less influenced models, indicating that our Modem is robust to the usage of RevIN.
The extent of RevIN's ' influence on the model is related to the model's 's mechanisms. For example, an important step in TimesNet is calculate the periods of time series based on FFT. Since FFT mainly works well on stationary signals, there a greater need for the time series to be stationary. As a result, TimesNet's performance is highly related to RevIN.
Table 23: Ablation about RevIN on Long-term forecasting for other models. We report MSE and MAE under four different prediction lengths as metrics. +Rev means with RevIN, -Rev means without RevIN.
RMLP
RMLP
RI Linear
RLincar (-Rev)
TimesNet
TimesNet
SCINet
SCINet
Models
(+Rev)
(-Rev)
(+Rev)
(+Rev)
(-Rev)
(+Rev)
(-Rev)
Metric
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
MSE MAE
96
0.298 0.345
0.310 0.359
0.301
0.342
0.334 0.374
0.338
0.375
0.622
0.596
0.325 0.372
0.399 0.432
ETTh1 ETTml
192
0.344 0.375
0.350 0.382
0.335 0.363
0.365
0.392
0.371 0.387
0.622 0.583
0.354 0.386
0.411 0.436
336
0.390
0.410
0.387
0.405
0.370
0.383
0.397
0.410
0.410
0.411
0.703
0.644
0.394
0.415
0.459
0.465
720
(0.445
0.441
0.448
0.442
0.425
0.414
00:456
0.448
0..77
0.450
0.901
0.736
0.476
0.469
0.530 0.511
96
10.390
0.417
0.468
0.366
0.391
0.552
0.497
0.384
0.402
1.261
0.883
0.375
0.406
0..525
0.513
192
0.430
0.432 0.511
0.492
0.404
0.412
0.559 0.513
0.557
0.436
1.132
0.835
0.416 0.421
0.570 0.542
336
0.441
0.441 0.540
0.515 0.420
0.423 0.569
0.526
0.491
0.469
1.166
0.841
0.504
0.495
0.580 0.548
720
0.506
0.495 0.603
0.575 0.442 0.456 0.602 0.568
0.521
0.500
1.269
0.926
0.544 0.527
0.589 0.564
96
0..149 0.202
0.202|0.119 179
0.239
0.5 0.225
0.225|0.226
0.226 0.283
0.2 0.220
0.222
0.316
061 0.226
180 0.260
Weather
192
0.194 0.242 0.220 0.278 0.218 0.260 0.270 0.319 0.219 0.261
0.249
0.326
0.220 0.283 0.229 0.299
336
0.243
0.282 0.264 0.313 0.265 0.294 0.332 0.369
0.280
0.306
0.360
0.413
0.275
0.328
0.280 0.333
720
0.316
0.333 0.327 0.364
0.329 0.339
0. 188
188
0.250
0.365
0.359
0.414
0.434
0.311 0.356
0.337 0.370
Table 24: Ablation about RevIN on imputation tasks. We report MSE and MAE under four different mask ratios as metrics. +Rev means with RevIN. -Rev means without RevIN.
Models
MedemTCN ModenTCN
PatcTST
PachTST
RMLP
RLinear
TimesNei
TimesNet
SCINet
SCINet (- (Rev
(+Rev)
(Rev)
(+Rov)
(RRev)
(+Rev)
(-Rev)
(+Rev)
(-Rev)
(+Rev)
(-Rev)
(+Rov)
Metric
MSB
MAE
MSG
MAE
MSE MAD
MSE MAC MSC MAC MSC MAC MSC MAG MSC MAG
MSC MAE MSE MAC MAC MSE MAG
MSC MAG
10.155
0.015
0.082
0.017
0.090
(0.04]
0.126 o 0,009
13000.056 o 162/0.047 a 137 o. 0,001/00119 0.092/0.022 o 103/0.03) a 11600,200 0.
0.25
0.018
0.888
0.020
0.005
0.043
0.130) 0.040000044000 o 0.057 0.4
0.066 0.157 0.077 0.190 0.101 louz 1.188 0.0% 124 0.042 144
0.375
0.021
0.995
0.025
0.307
0.044
0.133 0.054
0.155
0.067
0.089 0.206 0.077 0.175 0.199 0.17/0.122
0.11 0.034
0.129 0.24 0.134
0,150
0.158
0.5
0.105 0.026
0.031
0.118
(.055 0.142
0.0622
06164 0.079
0.
0.226 0.195 0.125 0.246
0.246 0.036 0.124 0.147
(u000 1.33
1..25
lo.35
0.128
10.055
0.176
10.094
0.199/0.
220
2300o..00
0.000 20300. 2.15
0.25
00062
0.140
0.049
0.154
0.119/0222/0.111
0.220
0.22
0.151
0.70
0.229 0.149 (10.099/..0111790022220200 0.111 0.152 0.258
0.375
0.060
0.166
0.291
0.500
0.153
0.233 0.200 010.0000000910001111110111811111
0.145 0.272
0.5
0.066
0.174
0.196
0.173
(0.200
0.366
0.176
0274
0.328
0:21 0.232
6.125
lo.23
0.038
10.006
0.059
l0.029
(0.041 1.16 0.030
o
004s/o.150 0.991 (0.025 o.040/.00000000.......33.. o
(ttp:/2228044400008 047/0.052
Weather
0.25
(a025
0.041
00002
0.077
0.033
o 0.0553 0.145
0.033
0.277
0,046
0.13
0.032
0.053 0.047
0.109
0.232 o.885 0.029 0.050 0.077
0.375
0.027
0.066
0.000
0.069
0.034
0
0.062 0.098 0.036 0.052
0.003
0.123 0.057
0.057 0.044 0.094 0.03)
0.036 083
(.001
0.051
(.007
0.m90
(..09
ned o. 0.092
0.040 ows. 0.600.00
0.077 0.137 0.046
0.034 069
33

---

Published as a conference paper at ICLR 2024
Table 25: Ablation study about RevIN on short-term forecasting. Results are weighted averaged
from several datasets under different sample intervals. +Rev means with RevIN. -Rev means without RevIN. Since removing RevIn in PStchTST will make the training loss become NaN, making it hard to train on short-term forecasting, don't report the result of PatchTST without RevIN.
Models
ModeraTCN PaccTTT
RMLP RLincar TimesNet SCINet
(Ours)
2023
2023a
2023a
20023
202m
SMAPE
11.698
11.807
12.072
12:473
11.829
12.369
+Rev
MASE
1.556
1.590
1.624
1.677
1,585
1.677
OWA
0.838
0.851
0.870
0.898
0.851
0.894
SMAPE
12.241
14.984
12.777
90.132
12,950
RRev
MASE
1.796
2.716
1.789
20.17
1.955
OWA
0.921
1.262
0.989
Table 26: Ablation study about RevIN on anomaly detection. We report Fl-score as metric. +Rev means with RevIN. RRev means without RevIN.
Models
ModernTCN PatchTST
RMLP RLincar TimesNet SCINet
(Ours)
20230
2023g
2023
2022
SMD
85.81
84.44
82.46
83,70
85.81
84.24
+Rev MSL
84.92
85.14
74.48
81.15
85.15
83.38
PSM
97.23
96.37
92.76
96.33
97.47
96.00
SMD
81.42
71.17
73.63
81.32
71.20
83.75
-Rev
MSL
82.07
81.70
78.05
81.86
82.20
81.28
PSM
95.93
34

---

Published as a conference paper at ICLR 2024
M FULL RESULTS
Due to the space limitation of the main text, we place the full results of all experiments in the following: long-term forecasting in Table short-term forecasting in Table 28 imputation in Table classification in Table 30 and anomaly detection in Table 31
29
And some showcases provided in Appendix N
M.1
LONG-TERM
M.2
SHORT-TERM
M.3
IMPUTATION
M.4
CLASSIFICATION
M.5
ANOMALY DETECTION
35

---

Published as a conference paper at ICLR 2024
Table 27: Full results for the long-term forecasting task. We compare extensive competitive models under four prediction lengths. The input sequence length is scarched to the best for a fairer comparison. Avg is averaged from all four prediction lengths.
Models
ModernICN
PaschTST
Crossfomer PELiformer MLS miner
LighTTS
DLLinear
TimesNet
MICN
SCINet
KLinear
RMLP
(Ours)
2025
2012
3023
2025
Metric
MSE
MAB
MSE
MAE
MSE
MAE
MSE MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSB MAB
MSE
MAE
MSB
MAB
MSE MAE
MSE MAE
96
10.202
3.346
0.2200
0.342
0.116
0.378
o.
33336660
390l0,
0.14
0,358
0.374 o 40010.200 0. 343
0.338 0.375
0.314 asoln,3es 0.572 lo 301 o 0.342...200 0.345
0.332
0.368
1.360
0.377
0.411
0.365
0.415 0.354 0,356 0.480 0.417
0.335
0,365
(1371 11387
0,359 0.357 11354 0.386 0.335 0.363
336
0.365
0.391
0.366
0.392
0.431
0.442
0.392
0,425
0,784
0,405
0:438 0.438
0.369
0,386
00,400
0:411
0.398
0.413
0,394
0.415
0 0.3890 0,410
0.390
720
11.416
0.417
0..466
0:470
0.470
0.600
0.547
(1:46
0.458
0.447
0,452 0.527 o 5..00
0.405
0.421
0.478
0.450
0:499
0.464
0,447
0.469
(o:425 0.41400,444 0,441
Avg
[0.351
0.361
00:351
(..8.
0.431
0.443
|0.382
0.422|0.535
0,570 0,305 0.435 0 0.437
0.43//0.357 (0357 0.379
0.400
0.406
0,383
0.406/o.88
0.411
(0.35 0.3.000776..000 0.993
96
10.166
0,256
00.165
0.255
0.421
0.461
0.22)
101177 (.27)
0,259 0.209 0. 0.308/0,16 1.67 0.260
0.267
0.178
0.275/o
0.281
(0.164
0.253/0.174 0,259
ETTm2
0.293
0..300
0.200
0.503
0.519
0.318 (1.44 0,503 0.311 o 0.224
(1.449
0.309
0.245
0.116
0,356
(02299
0.290 0.236.23 0.303
336
00.722
0.324
0.274
0.329
0.611
0.500
0.:324
0,364
0.299
0.338 0.442 0.466 0.251 0,342
0.321 0.351
0.295
0.350 0.309
(0.273
0.326 0.291 0,338
720
00.51
0,361
0.362
0.345
0.996
0.750
0.40000.420000
0.42000.39
0,398
0.675 0.587 0.397 0.421
0.497
0.403
0,389
0.106
0,403
0,412 (0.366
0.385 0,391
Avg
|0.253
0.314
1:255
[1.33
0.632
0.578
(0.292
0.343
0.277
0,325 0.409 0.430
(0266
0.332 (.291 0.333 0.277
0.336/1.294
0.355
(0.256
0.314 0,322
96
10,368
0.394
00,300
0.399|
0.386
0.429
00,376
0.415
(0,372
0.395
0:424 0.432
0,375
0,399
(.284
0,402)
0,396
0.427/0.375
0.406
(0.366
0.399|0.390
0.39]
0,410
192
00,005
0.413
0.413
0:421
0.419
0.444
0.423
0.446
0.416 0,426 0:475 0 462
0.405
0.416
0.557
0.436 0-430
0-453
0.421
(04444
0.412
0.412 0.450
536
00,991
0.412
0.422
0.446
0.440
0.461
00.444
0.462
0.455
0.449 0.518 0.521 0.439
0,443 (1491
0.40) 0.433
0.45% 0.504 0.405
0.420
0..41 0.441
720
0.450
0.461
00,447
01406
0.519
0.524
0.449
0.492
0.475
0,472
0.547 0.533
0,472
0,490
0.521
0.500
0.474
0.505
0,505
01544
0.527
(0442
0.500 0.495
Avg
10.904
0.420
10.433
11441
[0.4]
0.465
(0:426
0.454
[,430
0,436 0.491 0.47%jo.23
0.437
1.458
0.45)) 0.433
0.462]11400
0,402
b.405
0.421
(.,422 0:445
96
10.263
0,332
10.274
0.336
1.336
0.628
0.563
(0,332
0.374
0,374/0,307
(0,07 0.354 0.397 0.29
0,353
(1344
0.374]
0.289
0.3571u2295
0.361
(0.202
0.331
(0.251 0,352
ET
192
0.330
0.374
0.339
0.379
0,703
0.624
0.407
0.446 0.374 0,399 0.520 0 0.500
0.50500388
0.383
0.418
0.402
0.414 0:409
0.138
0.138 0,349
0,383
0.330
0.343 0.376 0.387
0.374
936
0.313
0.376
0.300
0.380
0.827
0.675 (0:400
0.447 0.308
0.432 0.626 0.559 0.448
n.465
4.452
0.452
0.17
0.452 0,365
0.409
0.125
0.353 0.400
720
00.992
00433
(1.379
11,42
1.181
0.840
(0.412 0.459
1.463
0,405 0.863 0.672 0.660 0,553
(1462
1.468
0.426 0.475
0.488
00.72
0.421
Avg
[0.322
0.370
|0.330
0.370
0.83
0.676 |o. (0.388 0.434
0,386
0.413 0 602 0.543/0.431
0.447
[0.41 0.437 0,385
0.430|a.371 0.410
(0,320
0.378 0.305
Electricity
96
10.129
0.226
10.129
0.222
187
0.283
10.186
0.302
0,141
0.243
0.207
0.307
1.1
0.237/0
[1.66
0.272
0.159
0.26711.171
0.256
10.140
0.35
00.19 0.224
192
0.143
0.239
0.147
0.240
0.258
0.330
0.197 0.311
0.163
0.261 0.213
0.310 0.249
0.310
0.152
0.184
0.289
0.168
0.279
0,263
001144
0.245 0.240
0.161
0.259
0.323
0.369
0.213
0.328
0.176
0.230
0.267
0.198
0,300
0.196
0.308
0.285
0,264 0,257
720
0.191
0,286
0.197
0.200
0120000.400
0.404 0.423
(.2233
0.344
0.212
0.308
0.265 0
0.360
0.360
0.233
0,344
0.220
0.320
0.203 0.312
0,318
(.229 0.2970..30 0.291
Arg
10.156
0,253
10.159 0,253
0.293
0.351
|0.207
0.321
103
0.272 0.229 0.329|0.177
0,274
(9.192 0.295)
0,182 0.292/0.195
195 0.281 0011600226)
(0.161 0,253
96
10.149
0.200
10.499
0.196
1.3
0.217
00.338
0,314
1.16
0.306
0.182 0 0.342lo.115
0,23710
(2.12
0.200
0.161 0.2661.177 0.233 175
0.200
Weather
192
1.196
0,245
0.194
00.241
0.157
0.269
0.275
0.329
0.248 0.227 0.29700.20
0.220
0.282
1:219
0.261
0.220
0.255 0.235
0.2s
0,277 0.218
0.260 0.242
336
0.238
0.277
0.252
0,211
0,375
0.249
0.291 0.282
0.266
0.319
0.280
0.206
02255
0.328 0.337
0.345 (0.265
0.243
720
0.314
0,334
0.364
0.334
0.318
0.363
00.389
0.409
0.336
0,343 0.352 o 1838
0.323 0,362
0.366
0,359
0,311
0.3560.306
0.413 000329 3.300.3316600
Arg
10.234
0.264
0.2226
0.264
0.230
0.290
|0.310
0.357 0.357|0.235
(0.235 0.272 0.261 0.312/0.240 0.300
0.287) 0.242 0.298/0.287 0.317 [0.247 0.279/0.225 0,265
96
10.368
0.251
0..360
0.542
0.290
00.576
0.389l0.462
0,332 0.319..391111 0.31//0..100000000000
0.301
0.508
0.301
(0,633
0.395
Traffic
192
1.379
0.261
0.379
0.256
0.525
0.297
0.660 o 0.848 0.354 0.601 a 0.382 0.423
0.433
0.287
0.617
0.356
0.536
0.515
0.363
00.00
0.377 0,340
336
0.397
0.270
0.264
0.530
0.300
0.60s 0.375
0.498 0.300
0.613 0.300 0.436
0.296
0.629
0,525
0.310
0.358 (0.17
0.370 0.351
1.440
0.295
0.286
0.575
0.313
(.662
0:375
0.529
0.370
0.658 0.407 0.406 0:515
(1.40
0.350
0:571
0.323
0.394
0.0033 0,372
Arg
00.390
0.200
00.391
0.264
0.535
0.200
0..63
0,572
0.494
0.354 0.622 0.392/0.434
0.295
1..20
0.336
0.535
0.312/0.587
0.378 (0.518 0.33 0.348
96
10.050
0.196
10,093
0.214/0.186
0.346
0.139
0.276/0088
0,003
0.201 0.116 0 0.26200.01
0.200
0..07
0.234
0.102
0.235
0.254
(0.083
0.201
10,033 1.20100.033 0,201
Exchange
190
0.166
0.288
0.192
0.312
0.467
0.522
0.256
0.360 0.14 0.206
0.215
0.59
0.57
0.201
0.026
0.344
171
0.316 0,345
170
0
0,200
536
0.307
0.398
0.432
0.783
0.721
0.442 0.464
0.336 0.417 0.377
0.460
0.414
0.367
0.448
0.272
0.417 0.294
0.413 00339
0.401
0.300 0.401
0.656
0.582
0.9011
0.716
1.367
0.943
0,900 0.800
0.60
0.643
0.60
0.746
0.714
0.658
1.110
0.767
0.816 0.680
Arg
|. 1.300
0.366
[1.387
0.430
0.635
(0.478
0.478
01373
0.407 0.385 0.447/0.297
0:578
1.416
0.443
0.315
0.404/0.435
0.404
(1.435
0.445
0..345
(0:345
10.45 0,354
1.347
0.717
1.339
(.754)
3,040
1.186
2..044
1.095
1.472
0.798 8.313
2.144|2.215
L.083
2.317
(1934]
2.684
1.112|2.150
1,005
[4.337
1.5tr
1,536
1.250
0.778
1.400
0.834
3.356
1.230
2,516
1.02]
1.435
0,745 6.631
1.902
1.963
0.963
1.972
0.920
2.507
2.0132.103
1.013
2.103
0.983
4.205
1.48]
4.009 1,519
1,388
0.781
15553
0.815
3.461
1.223
2.505
1.041
1.474
0.822
7.299
1.982
2.132
2.130 1.024
2:238
0.940
2.423
1.012
1.0122.432
1.061
4.257
1.484
0.398 1,507
0.768
0,912
7.283 1.985
2.007
0.928
2.653
1.085
2.305
1.005
4.278
1.487
4.306
1.440
0.786
1.443 0.708)
1.235
2.597
1.070]
7.882
2.03
2.
160
1.041
(2)
0.931
2,567
1.055
1.021
(4.299
1.49044.357
Coust
23
0
36

---

Published as a conference paper at ICLR 2024
Table 28: Full results for the short-term foreeasting task in the M4 dataset. s, in the Transformers indicates the name of *former.
ModernTCN CARD PatehTST Crossformer FEDrormer
RMLP RLinear MTS-mixer LightTS DLinear TimesNet MICN SCINet N-HiTS N-BEATS
Models
(Ours)
2025
2023
2023
2022
20238
2023a
2028b
2022
2022
2022
2023
2022a
20231
2019
Yearly
SMAPE
13.226
13.302
13.258
13.392
13.728
13,418
13.994
13.548
14.247
16.965
13.387
14.935
13.717
13,418
13.436
MASE
2.957
3.016
3.006
3.015
3.091
3.109
3.076
3.045
3.043
OWA
0.777
0.786
0.781
0.787
0.803
0.789
0.807
0.803
0.827
1.058
0.786
0,900
0.807
0.793
0.794
Quarterly
SMAPE
9.971
10.031
16.317
10.128
11.364
10.202
MASE
1.167
1.176
0.803
2.197
1,283
1.234
1.299
1.196
1.328
1.520
1.182
1.389
1.295
1.194
1.169
OWA
0.878
0.884
0.803
1.542
0.958
0.921
0.959
0.896
1.000
1.106
0,890
1.026
0.965
0.899
0.886
Monthly
SMAPE
12,556
12.670
12.641
12.924
14:260
12,998
13.363
12.717
14.014
13.514
12.670
13.773
13.208
12.791
12.667
MASE
0.917
0.933
0.930
0.966
1.102
0.976
1.014
0.931
1.053
1.037
0.933
1.076
0.999
0.969
0.937
OWA
0.866
0.878
0.876
0.902
1.012
0.909
0.940
0.879
0.981
0.956
0.878
0.983
0.928
0.899
0.880
Others
SMAPE
4.715
5,330
4.946
5.493
4.954
5.098
5.437
4.817
15.880
6.709
4.891
6,716
5.423
5.061
4.925
MASF
3.107
3.261
2.985
3,690
3.264
3,364
3,706
3.255
11.434
4.953
3.302
4,717
3.583
3.216
3,391
OWA
0.986
1.02
3.474
1.053
Weighted Average
SMAPE
11.698
11.815
11.807
13:474
12.840
12.072
12.473
11.892
13.252
13.639
11.829
13.130
12.369
11.927
11.851
MASE
1.556
1.590
1.608
2.111
1.599
OWA
0.838
0.850
0.851
0.985
0.918
0.870
0.898
0.859
1.051
1.051
0.851
0.980
0.894
0.861
0.855
The original paper N-BEATS Oreshkin et al. 2019) adopts special ensemble method promote the performance. For fair comparisons, we remove the ensemble and only compare the pure forecasting models. CARD is re-implemented by us based on the pseudo-code code in the original paper (Xue et al. 2023 In original paper, CARD is trained with cosine learning rate decay and linear warm-up. For fair comparisons, remove this additional training scheme.
37

---

Published as a conference paper at ICLR 2024
Table 29: Full results for the imputation task. We randomly mask 12.5%, 25%, 37.5% and 50% time points to compare the model performance under different missing degrees.
Models
ModeraTCN
PatcTTST
Crossformrr FEDifmmer MTS-mixer
LighTTS
DLinear
TemesNot
MICN
SCINet
RLimerr
RMLP
(Ours)
Mask Ratio
MSE
MAE
MSB MAR
MSE
MAR
MSE MAP
MSE MAR
MAE
MSE MAE MSE MAE
MSE MAE
MSB
MAE
MSE
12.5%
0.015
0.082
0.01 o 12600.03 0.137 0.035 0,15 0.043 o. 13400075 0,180 o 0,019 0.092 039 o. 0.333 1670
37/00099 131
25%
0.018
0.088
00. 0.141 0.052 n. 0.051 1.14 0.795 o. 0.00 1.3 n.023 o. 101 0.599 0.17000.0... D. no6l 0.157 0.057 0.154
37.5%
0.021
0..40.333.041
0.009 0.191
0:000.113
0.233
0.219 0.0.2
a
0.041
0:134 0.077 0.125
50%
0.105 0.026
0.:00.1420.04
0.152 0.089 0,218 0.070 0. 07400 0.255 0.132 0.036 124 o 0.221 0.009 0.143 0.096 o 095
Avg
00,020
0.093
13450.13300044
0.143
0,062.0177
1540, 104 0.218/000000000000000 0.200.0000000000000000000000 0.002 0, 070 11w.00111...... 18700.039
130000 129 0,700 166 66660.0630.166
12.5%
00.11
0.076
0.025
0.02
0.148
0,056
0,159
0.026
0.04
0.127 0.11270.062 0.166
0.018
0,080
0.003
0:093
10030..000
0.026
0.093
2596
0.018
0.080
0.027
0.047
0.151
0.19
0.105
0.145
00085 0.196
0.0020
0.000
2:16
0.002
0.000000000000000000000000000000 0.0000
000
37.5%
0.029
0.099
0.110
0.100
0.159
0.900000
0.023
0.091
2022730..00
0.004 0.3....113..:3....1.
50%
0.022
0.090
0.32 106
01
0.1
0.131
0.247
0.026
0.098
11111
Avg
[0.019
0.082
0.028
0.0260.09660.046
0.149 0.101
(0.096
0.206
0020000..00
0.022
12.5%
10035
0.128
19900.999000999
0.218
0.267
0.19
(97 0.202...8808 000.0000000000000000000 200.000.00000000...1
25%
0.042
0.140
0.119..225..122
0.243
0.364
0.3644000000000000000000000
(1002
0.178
0.5 0.232
0.099 0.211
0.123 0.12000.23
37.5%
0.054
0.157
0.145
00.124 0.255
0.24400.996
0.382
0.318
0.115
0.14002533..5 0.144....5
50%
0.174 0.067
0.173.0221
0.155
0.281
(0.165 0,29500299000 016.0.990..6000.230...3333.
0.404
0.157 0.347
0.215
185
0.188
Avg
|0,15 0.150
|1333022360.133
0.251
(0.1170,346 020.0/.................. 2.2.
0.306
(0.07
0817
125 0.250/o 0:210.14061000
12.5%
0.037
0.121
0.0570.1500..100
0.220
0.099
0.212....000011
0.157/0.10 0.157
0.231 0.233/0.1000000000000000
0.100.0.116
0.100
0.223/0.06
|0.05
25%
0.040
0.127
0.62 o 0.1
0.220
(0.157
0.065
0.55
0.
0.277.2277
0..04
0,
151
0.271
0.062
0.10
37.5%
0.043
0.068
0.346
00.187 0.304
a.
(0.1260.2577
0.158
0.276
(
0.052
0.229
0.166
0.1700.71
50%
0.143 [1,48
0.76
0.265
(0.2320.344 0.078
181
1316 0.266...6...188888 0.20
(0000000..00000 0.060 0.2 334
0.
Avg
[0,022
0.131
0.166001640..0
0,240 0,240 (11663.,279900
6,165 0.66550066606666..00....
Electrictty
12.5%
[0,199
0,171
0
0.181
0.0.0.000.0099.00 18210, 01tp022000.00022444.....33
25%
0,071
0.188
0002
02000.000007
0.079 0.198
0.1220..25
0.003
0.252 0.2 0.118 0.002
0.200
0.000
0.195
0.09000000
37.5%
0.097
0.200
0.215
0.273
0.144
0.090
50%
0.085
0.200
2:110:2322000
0. 6.096 0.312
0.155 0,284 0:23
0.299 0.75 0017
6:214
29) 0.125
Avp
[0,773
0,187
(0091
0.210.999..000
0.199 (0. 0.13000220 0.089
Weather
12.5%
[0,023
0.035
0.029:00::00
0:49/0.33
0.092 0.044 0, 0.0.0000000000....0...... 0.10)
0.0./o...3... 0.047 002.0........0000030..0.
25%
0.025
0.041
1081 0.055 0.088 (0.064 0.004 005.:663.00000000000000.0.00.... p.11
1.1 0.
0.002
37.5%
0.027
0.34
0.055
0.027 0.00
01057
0.117
0.033
0.031
0:055
00086
0.062
50%
0.031
0.051
0.039
0.06600.038
0.092 00.133 0.312 0.011 0.066 0.133 0.066 0.134 03400330000.000 0.062 0.080 0.18 0.004 0:059
0.040 0.000:0007700000..000 0.0440
Avg
|0,027
0.044
00333005500.000
0,090 0. 0.036
19 CCount
48
0
Table 30: Full results for the classification task. *. in the Transformer-basee models indicates the name of *former. We report the classification accuracy (%) as the result. The standard deviation is within 0.1%.
RNN-based
Convolation-bused
MLP-based
Transformer-
Datasets/ / Models
Www.MMiiee/RLiiea/RMMM /FED.
/Flow./Cross.f CrossPPaccTSTTMModeeee
22018b
2022
2020
2222al
(2023
2023
(2022
(2022)
(2028
(2023
2022a
(2022
[002)] [2023)
(2023
(Ours)
39.9
31.1
45.2
34.4
35.3
35.7
36.2
29.7
33.8
28.9
31.3
31.2
33.8
38.0
32.8
36.3
FaceDetection
65.7
66.7
64.7
68.9
65.2
68.0
67.5
70.2
65.6
67.3
66.0
67.6
68.7
68.3
70.8
Handwriting
25.8
24.6
58.8
23.6
25.5
32.1
27.0
26.1
26.0
28.1
30.0
28.0
33.8
28.8
29.6
30.6
Heartheut
77.1
72.7
75.6
77.5
74.7
78.0
75.
75.1
77.1
72.6
72.7
73.7
77.6
77.6
74.9
77.2
SguanesVowees
98.1
98.4
96.2
96.0
94.6
98.4
96.2
96.2
94,3
95.9
95.9
98.4
98.9
99.1
97.5
98.8
PEMS-SF
86.7
86.1
75.1
83.8
85.5
89.6
75.1
88.4
80,9
82.7
83.9
80.9
86.0
85.9
89.3
89,1
SelfRegalzionSCPI
84.0
90.8
92.5
86.0
89.8
91.7
91.1
92.1
88.7
92.5
90.7
SelfRegahriinSCP2
52.8
52.2
53.3
57.2
$3.6
57.2
50.5
51.1
55.0
56.1
51.0
54.4
56.1
58.3
57.8
60.3
ApokennArasiiiigits
100.0
100.0
71.2
98.1
97.1
99.0
81.4
100.0
97.4
96.5
97.6
100.0
98.8
97.9
98.3
98.7
87.8
85.9
94.4
85.1
82.8
85.3
82.1
80.3
82.3
82.5
83.8
85.3
86,6
85.3
85.8
86.7
Average Accuracy|
71.8
70.9
72.5
71.7
70.0
73.6
67.5
70.4
70.9
70.0
70.6
70.7
73.0
73.2
72.5
74.2
38

---

Published as a conference paper at ICLR 2024
Table 31: Full results for the anomaly detection task. The P. R and F1 represent the precision, recall and Fl-score (%) respectively. Fl-score is the harmonic mean of precision and recall. A higher value of P. R and Fl indicates better performance.
Datasets
SMD
MSL
SMAP
SWaT
PSM
Avg
Metrics
F1
SCINet
2022n
85.97
82.57
84
84.16
82.61
83.38
93.12
54.81
69.00
87.53
94,95
91.09
97.93
94.15
96.00
84.74
MICN
2023
88.45
83
85.89
83.02
83.67
83.34
90.65
61.42
73.23
91.87
95.08
93.45
48.40.0
88.69 93.29
85. 84
TimesNer
2023
88.66
83.14
85.81
83.92
86.42
85.15
92.52
58.29
71.52
86.76
97.32
91.74
98.19
96.76 97.47
86.34
DLincar
2022
83.62
71.52
77.10
92.32
69.26
95.30
98.28
93.55
82.46
LightTS
2022
87.10
78.42
82.53
82.40
75.78
78.95
92.58
55.27 69.21
91.98
94.72 93.33
98.37
95.97 97.15
84.23
MTS-Mixer
20236
88.60
82.92
85
85.35
84.13
84.74
92.13
58.01
71.19
84.49
93.81 88.91
98.41
88.63 93.26
84.75
RLinear
20230
87.79
79.98
83.70
89.26
74.39
81.15
89.94
54.01 67.49
92.27
93.18 92.73
98.47
94.28 96.33
84.28
RMLP
2023a
87.35
82.46
86.67
65.30
74.48
90.62
52.22 66.26
92.32
93.20 92.76
98.01
95.57
82.31
Reformer
75.32
57.44 70.40
72.50
96.53
59.93
73.61
77.31
Informer
2021
86.60
77.23
81.65
81.77
86.48
84.06
90.11
57.13 69.92
70.29
96.75 81.43
64.27
96.33 77.
78.83
Anomaly"
2021
88.91
82.23
85.49
79.61 87.37
83.31
91.85
58.11 71.18
72.51
97.32 83.10
68.35
94.72 79.40
80.50
Pyraformer
202
85.61
80.61 83.04
83.81
85.93 84.86
92.54
57.71 71.09
87.92
96.00 91.78
71.67
96.02 82.08
82.57
Autoforner
2021
88.06
82.35 85.11 77.27
80.92
90:40
58.62 71.12
95.81
99.08
93.29
84.26
Stationary
2022c
88.33
81.21 84.62
68.55
89.14 77.50
89.37
59.02 71.09
68.03
96.75 79.88
97.82 96.76 97.29
82.08
FEDiormer
2022
87.95 82.39 85.08 77.14 80.07 78.57
90.47
58. 70.76
90.17
96.42 93.
97.31
97. 97.23
84.97
Crossformer
2023
83.06 76.61 79.70
84.68 83.71 84,19
92.04
55.37 69.14
88.49
93.48 90.92
96 89.73 93.30
83.45
PatchTST
2023
87.42
81.65 84.44 84.07 86.23 85, 92.43 57.51 70.91
80.70
94.93 87.24
98.87 93.99 96.37
84.82
ModemTTCN
(ours)
83.85 85.81 83.94 85.93
71.26
95.98 93.86 98.09 97.23
86.62
The original paper of Anomaly Transformer Xu al. 2021 adopts the temporal association and
reconstruction error as a joint anomaly criterion. For comparisons, we only use reconstruction error
here.
39

---

Published as a conference paper at ICLR 2024
N SHOWCASES
We provide showcases to the regression tasks, including the imputation (Figure 13 long-term forecasting (Figurel 14 and 15) and short-term forecasting (Figure 16). as an intuitive comparison among different models.
N.1 IMPUTATION
(ours)
MTS-Miser
LighTTS
TimesNot
MICN
SCINet
Miaa Mave
-40
Figure 13: Visualization of ETTh1 imputation results given by models under the 50% mask ratio setting. The blue lines stand for the ground truth and the orange lines stand for predicted values.
40

---

Published as a conference paper at ICLR 2024
N.2 LONG-TERM FORECASTING
ModeraTCN (ours)
MTS-Mixer
DLinear
LightTS
TimesNet
MICN
SCINet
Figure 14: Visualization of ETTm2 predictions by different models under the input-366-predict-12 setting. The blue lines stand for the ground truth and the orange lines stand for predicted values.
41

---

Published as a conference paper at ICLR 2024
ModernICN (ours)
PatchTST
Crussformer
FEDiormer
644
414
MTS--Mixer
DLinear
LighTTS
344-
TimesNet
MICN
SCNNet
Figure 15: Visualization of ETTm2 predictions by different models under the input-336-pptt -36-prrdec setting. The blue lines stand for the ground truth and the orange lines stand for predicted values.
42

---

Published as a conference paper at ICLR 2024
N.3 SHORT-TERN FORECASTING
ModernTCN (onrs)
Crossforme weer
FRDiormer
M
M
M
M
M
MIN-MMixrr
DI Jincar
LightTSs
M
m
M
SCINet
MM Mr M M
M M
Figure 16: Visualization of M4 predictions by different models. The blue lines stand for the ground truth and the orange lines stand for predicted values.
43


Reviews:
[{"review_id": "zwVjG83DqW", "reviewer": "Reviewer_kELp", "summary": "The paper presents a pure convolution architecture for time series analysis. The proposed ModernTCN block is partially inspired by modern convolution blocks as in ConvNeXt. ModernTCN mainly utilizes Depthwise Convolution and Grouped Pointwise convolution to learn cross-variable and cross-feature information in a decoupled way. With the efficiency of convolution operations, ModernTCN achieves impressive performance on various time series analysis tasks. The authors also made a great effort to conduct comprehensive ablation studies to understand the different components of ModernTCN.", "questions": "- What does *Discard Variable Dimension* refer to in Table 4? In Table 4, it is shown that discarding variable dimensions leads to a significant performance degradation. However, in Appendix E, it is shown that univariate forecasting performance is still quite competitive. So what is the difference between the two?\n- From Table 4, one important piece of ModernTCN is to exploit cross-variable information. However, notice that some of the other models, e.g. DLinear are variable independent. So is the proposed modern TCN block orthogonal to other approaches and is it possible to apply it on top of variable-independent models like DLinear?\n- Based on the appendix, it seems that very different hyperparameters are used for different tasks, e.g. channel number, two kernel sizes, patching size and strides, number of TCN blocks, etc. What is your hyperparameter search space?\n- How is it different to use a projection layer (as in PatchTST) vs convolution as the stem layer?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The writing is pretty clear and easy to follow.\n- State-of-the-art performance on various time series tasks and the convolution-based architecture enjoys good training efficiency.\n- Extensive and comprehensive ablation studies to understand different components of the proposed model\n- A valuable contribution to picking up pure-convolution based approach for time series analysis", "weaknesses": "- It remains unclear how the proposed ModernTCN block would connect to existing literature. While pure convolution architecture is a sweet spot of efficiency and performance, based on what is claimed in the paper, it would be expected that the way to incorporate cross-variable information with ConvFFN would be useful on top of other variable-independent approaches such as PatchTST and DLinear. But this is missing in the current manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0uGSbG8E1", "reviewer": "Reviewer_exNa", "summary": "This paper uses a pure convolution structure (ModernTCN) to perform the time series analysis. Specifically, ModernTCN uses a 1D convolution stem layer, large kernel DWConv and ConvFFN to patchify embedding, capture the temporal dependency of each univariate time series independently and mix the information across feature and variable dimensions respectively. Five mainstream analysis tasks, including long-term and short-term forecasting, imputation, classification and anomaly detection are used to evaluate the effectiveness of ModernTCN.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper uses the pure convolutional neural network (CNN) for the time series analysis, which is seldomly explored. In addition, five mainstream analysis tasks for the time series data are considered for the evaluation.", "weaknesses": "1. The details of the blocks (DWConv and CovnFFN) should be introduced clearly (Some readers may not have related background Knowledge).\n\n2. It is not clear how to select the best results from different input lengths, based on the validation loss or the test results directly?\n\n3. The performance improvement is marginal. In addition, there is another MLP-based work beating PatchTST in same cases, which should be compared with. \n\nLi, Zhe, et al. \"Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping.\" arXiv preprint arXiv:2305.10721 (2023).\n\n4. Although it is said that \"we also include the state-of-the-art models in each specific task as additional baselines for a comprehensive comparison\", some SoTA models are missed in the paper, e.g.,\n\nShort-term forecasting: Wang Xue, et al. \"Make Transformer Great Again for Time Series Forecasting: Channel Aligned Robust Dual\nTransformer.\" arXiv preprint arXiv:2305.12095 (2023).\n\nAnomaly Detection: Yang, Yiyuan, et al. \"DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection.\" KDD 2023.\n\n5. How about the short-term forecasting results on the multivariate time series datasets (e.g., the datasets used for long-term forecasting in the paper)?\n\n6. The code is not available for reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sypY6NuQto", "reviewer": "Reviewer_dxG8", "summary": "-\tThe paper investigates a modern convolution architecture for time series data. The paper analyzes a pure convolutional architecture for time series analysis with the idea of modernizing the convolution structure to resemble that of a transformer. As existing temporal convolutional neural networks (TCNs) still lack the capability to capture long-term dependencies due to the limited effective receptive fields of the convolutional architecture, the paper proposes ModernTCN, a pure convolutional feed-forward block. It achieves this by optimizing depth-wise convolution with a large kernel size and two types of grouped point-wise convolutions, separately grouped on variable and feature dimensions. The three components capture cross-time, cross-variable, and cross-feature dependencies, respectively. The results demonstrate that the proposed method outperforms transformer-based models and other task-specific baselines in long-term and short-term prediction, classification, imputation, and anomaly detection, providing enhanced efficiency.", "questions": "-\tIn the experiments on regression tasks, it is concerned whether the RevIN (2021) normalization (as detailed in Appendix B.2) has a significant impact on the performance improvement. Therefore, an additional ablation study about RevIN is requested to enhance clarity. This request is based on the observation that RevIN normalization alone led to improvements in both qualitative and quantitative performance in some of the baseline models.\n\n-\tWhile the paper asserts the claim of enlarging the effective receptive field (ERF) as the key to using convolution in time series analysis in section 5.2, it would be beneficial to include an additional visualization of the ERF of TimesNet (2022) in Figure 1 to further substantiate the claim. TimesNet is another convolution-based model that utilizes shorter kernel sizes and more layers within a block, and it exhibits lower performance than ModernTCN.\n\n-\tSimple errata: Enlaring -> Enlarging (in section 5.2.)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "-\tThe paper investigates a recent idea of modernization in computer vision applied to time series analysis and identifies consistent performance and efficiency with sufficient comparison experiments and presentational supports.", "weaknesses": "-\tAlthough the primary claim for performance improvement centers on enlarging effective receptive fields, the paper lacks theoretical analysis as the authors acknowledge.\n\n-\tSome description details are not sufficiently clear. For example, in Table 4, it would be good to clarify which module the authors used as the ‘undecoupled ConvFFN’.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)