Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis
Authors: Qiang Wu
Gechang Yao
Zhixi Feng
Shuyuan Yang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: general time series analysis
deep learning
transformer
periodicity
pyramid structure
Primary area: deep_learning_architectures
TL;DR: We leverage the multi-periodicity and it's inclusion relationships within time series to construct a periodic pyramid and propose the Peri-midFormer, which achieves state-of-the-art performance across five time series analysis tasks.

Abstract: Time series analysis finds wide applications in fields such as weather forecasting, anomaly detection, and behavior recognition. Previous methods attempted to model temporal variations directly using 1D time series. However, this has been quite challenging due to the discrete nature of data points in time series and the complexity of periodic variation. In terms of periodicity, taking weather and traffic data as an example, there are multi-periodic variations such as yearly, monthly, weekly, and daily, etc. In order to break through the limitations of the previous methods, we decouple the implied complex periodic variations into inclusion and overlap relationships among different level periodic components based on the observation of the multi-periodicity therein and its inclusion relationships. This explicitly represents the naturally occurring pyramid-like properties in time series, where the top level is the original time series and lower levels consist of periodic components with gradually shorter periods, which we call the periodic pyramid. To further extract complex temporal variations, we introduce self-attention mechanism into the periodic pyramid, capturing complex periodic relationships by computing attention between periodic components based on their inclusion, overlap, and adjacency relationships. Our proposed Peri-midFormer demonstrates outstanding performance in five mainstream time series analysis tasks, including short- and long-term forecasting, imputation, classification, and anomaly detection.

Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis
Qiang Wu Gechang Yao Zhixi Feng Shuyuan Yang Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, China
{wu_qiangs waw.geehhann@stte xidian.innn..m..i. edu. en, {zxfeng, syyang)xxidian edu.
Abstract
Time series analysis finds wide applications in fields such as weather forecasting, anomaly detection, and behavior recognition. Previous methods attempted to model temporal variations directly using ID time series. However, this has been quite challenging due to the discrete nature of data points in time series and the complexity of periodic variation. In terms of periodicity, taking weather and traffic data as example, there are multi-periodic variations such yearly, monthly, weekly, and daily, etc. In order to break through the limitations of the previous methods, decouple the implied complex periodic variations into inclusion and overlap relationships among different level periodic components based on the observation of the muuti-periodicito therein and its inclusion relationships. This explicitly represents the naturally occurring pyramid-like properties time series, where the top level is the original time series and lower levels consist of periodic components with gradually shorter periods, which we call the periodic pyramid. To further extract complex temporal variations, introduce self-attention mechanism into the periodic pyramid, capturing complex periodic relationships by computing attention between periodic components based on their inclusion, overlap, and adjacency relationships. Our proposed Peri-midFormer demonstrates outstanding performance in five mainstream time series analysis tasks, including short- and long-term forecasting, imputation, classification, and anomaly detection. The code is available htttps://g/i thtbbb..tttuu.... com//WulaangXU/PPrr. mi Formerr
1 Introduction
Time series analysis stands as foundational challenge pivotal across diverse real-world scenarios such as weather forecasting (2). imputation of missing data within offshore wind speed time series [3] anomaly detection for industrial maintenance [4) and classification (5). Due to its substantial practical utility, time series analysis has garnered considerable interest, leading to the development of large
number of deep learning-based methods for it.
Different from other forms of sequential data, like language video, time series data is continuously recorded, capturing scalar values at each time point. Furthermore, real-world time series variations often entail complex temporal patterns, where multiple fluctuations (e.g., ascents, descents, fluctu- ations, etc.) intermingle and intertwine, particularly salient is the presence of various overlapping periodic components it, rendering the modeling of temporal variations exceptionally challenging.
Deep learning models, known for their powerful non-linear capabilities, capture intricate temporal variations in real-world time series. Recurrent neural networks (RNNs) leverage sequential data, allowing past information to influence future predictions [6] 77. However, they face challenges with long-term dependencies and computational inefficiency due their sequential nature. Temporal convolutional neural networks (TCNs) g 9 extract variation information but struggle with capturing long-term dependencies. Transformers with attention mechanisms have gained popularity for sequen-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original series
Period 1-
Period
Period
Multi-periodiccty
Inclusion and overlap
Periodic Pyramid
Figure 1: Multi-periodicity, inclusion of periodic components, and Periodic Pyramid.
tial modeling [I capturing pairwise temporal dependencies among time points. Yet, discerning reliable dependencies directly from scattered time points remains challenging [12]. Timesnet [13 innovatively transforms 1D time series into 2D tensors, unifying intra- and inter-period variations in 2D space. However, it overlooks inclusion relationships between periods of different scales and is constrained by limited feature extraction capability of CNNs, hindering its ability to explore complex relationships within time series.
We analyze time series by examining the inclusion and overlap (hereinafter collectively referred to as inclusion) relationships between various periodic components to address complex temporal variations. Real-world time series often show multiple periodicities, like yearly and daily weather variations or weekly and daily traffic fluctuations. And these periods exhibit clear inclusion relationships, for instance, yearly weather variations encompass multiple daily weather variations. Besides variations between different period levels, it also occur within periods of the same level. For example, daily weather variations differ based on conditions like sunny or cloudy. Due to these inclusion and adjacency relationships, different periods show similarities, with short and long periods being consistent in overlapping portions, and periods of the same level being similar. Additionally, a long period can be decomposed into multiple short ones, forming a hierarchical pyramid structure. In our study, time series without explicit periodicity are treated as having infinitely long periods.
Based the above analysis, decompose the time series into multiple periodic components, forming a pyramid structure where longer components encompass shorter ones, termed the Periodic Pyramid shown in Figure [ which illustrates the intricate periodic inclusion relationships within the time series. Each level consists of components with the same period, exhibiting the same phase, while different levels contain components with inclusion relationships. This transformation converts the original ID time series into a 2D representation, explicitly showing the implicit multi-period relationships. Within the Periodic Pyramid, a shorter period may belong to two longer periods simultaneously, reflecting the complexity of the time series. There is a clear similarity between components within the same level and those in adjacent levels where inclusion relationships exist. Thus inspired by Pyraformer [TO.. we propose the Periodic Pyramid Transformer (PerienidFormer), which computes self-attention among periodic components to capture complex temporal variations in time series. Furthermore, we consider each branch in the Periodic Pyramid as a Periodic Feature Flow, and aggregating features from multiple flows to provide rich periodic information for downstream tasks. In experiments, Peri-erdFormmr achieves state-of-thee -f-theart performance in various analytic tasks, including forecasting, imputation, anomaly detection, and classification.
1. Based on the inclusion relationships of multiple periods in time series, this paper proposes top-down constructed Periodic Pyramid structure, which expands ID time series variations into 2D, explicitly representing the implicit multi-period relationships within the time series.
2. We propose Forrmer, which uses the Periodic Pyramid Attention Mechanism to automatically capture dependencies between different and same-level periodic components, extracting diverse temporal variations in time series. Additionally, to further harness the potential of Peri-midFormer, we introduce Periodic Feature Flows to provide rich periodic information for downstream tasks.
3.
We conduct extensive experiments on five mainstream time series analysis tasks, and Peri- midFormer achieves state-of-the-an across all of them, demonstrating its superior capability in time series analysis.
The remainder of this paper structured as follows. Section briefly summarizes the related work. Section 3 details the proposed model structure. Section 4 extensively evaluates our method's performance across five main time series analysis tasks. Section presents ablations analysis, Section 6 presents complexity analysis, and Section 7 discusses our results and future directions.

---

2 Related Work
Temporal variation modeling, a crucial aspect of time series analysis, has been extensively investi- gated. In recent years, numerous deep models have emerged for this purpose, including MLP [54 TCN and RNN [6] Z-based architectures. Furthermore, Transformers have shown remarkable performance in time series forecasting [16] 12 17 [8].
They utilize attention mechanisms to uncover temporal dependencies among time points. For instance, Wu et al. [12] introduce Autoformer with an Auto-Coree nuto-Correlation mechanism, adept at capturing series-wist wise temporal dependencies derived from learned periods. Moreover, to address complex temporal patterns, they adopted a deep decomposition architecture to extract seasonal and trend parts from input series. Subsequently. FEDformer [17] enhances seasonal-tres -trend decomposition through a mixture-of-exper design and introduces sparse attention within the frequency domain. Pyraformer [10] constructs down-top pyramid structure through multiple convolution operations on time series to address the issue of long information prop- agation paths in Transformers, significantly reducing both time and space complexity. PatchTST [I9] partitions individual data points into patches and uses them as tokens for the Transformer, thereby enhancing its understanding of local information in time series. Additionally, PatchTST innovatively processes each channel separately. making it particularly suitable for forecasting tasks.
Additionally, there are some recent innovative works. Timesnet [33 unravels intricate temporal patterns by exploring the multi-periodicity of time series and captures temporal 2D-variations using computer vision CNN backbones. GPT4TS (20) ingeniously utilizes the large language model GPT2 as a pretrained model, fine-tuning some of its structures with time series, achieving state-of-the-an results. FITS 21 proposes time series analysis model based on frequency domain operations, requiring very low parameter count and memory consumption. And recent works have considered multi-scale information in time series. PDF [22] captures both short-term and long-term variations by transforming ID time series into 2D tensors using a multi-periodic decoupling block. It achieves superior forecasting performance by modeling these decoupled variations and integrating them for accurate predictions. SCNN (23) decomposes multivariate time series into long-term, seasonal, short- term, co-evolving. and residual components, enhancing interpretability, adaptability to distribution shifts, and scalability by modeling each component separately. TimeMixer [2]] uses novel multiscale mixing approach, decomposing time series into fine and coarse scales to capture both detailed and macroscopic variations. It employs Past- sests--eoooosaal--Miiie and Future- Multipredictor-MMMee performance in forecasting task.
to extract historical information to leverage multiscale forecasting capabilities, achieving great
3 Methodology
3.1 Model Structure
The overall flowehart of the proposed approach is shown in Figure 2 begins with time embedding of the original time series at the top. Then, we use the FFT to decompose it into multiple periodic components of varying lengths across different levels, with lines indicating the inclusion relationships between them. Moving down, padding and projection are then applied ensure uniform dimensions, forming the Periodic Pyramid. Each component is treated as an independent token and receives positional embedding. Next, the Periodic Pyramid is fed into Peri-midFormer, which consists of multiple layers for computing Periodic Pyramid Attention. Finally, depending on the task, two strategies are employed: for classification, components are directly concatenated and projected into the category space; for other reconstruction tasks (since forecasting, imputation, and anomaly detection all necessitate the model to reconstruct the channel dimensions or input lengths, we collectively refer to such tasks as reconstruction tasks), features from different pyramid branches integrated through Periodic Feature Flows Aggregation
generate the final output. Please note that we referred to [1] for de-normalization and [12] for time series decomposition to maximize the effectiveness of our method, but we omitted these details from the figure maintain simplicity. See Appendix A for a complete flowchart. Further details are provided below.
3.2 Periodic Pyramid Construction
Multiple periods in the time series exhibit clear inclusion relationships, however, the 1D structure limits the representation of variations between them. Hence, it's crucial to separate periodic com-
3

---

Time domain
Original Length
Temporal embedding
FFT for periods
Frequency domain
Frequency 1
Top
Frequency 3.. periods
Padding & Projection
35
Strategy for reconstruction task
Periodic pyramid
PPAM
Periodic Feature
Reconstruction tasks
Positional embedding
Add
Flows Aggregation
Peri-midFormer
Feed Forward
Strategy 1 for classification task
Flatten Projection
Classification task
Add & Norm
Downstream tasks
x
Figure 2: Model architecture. PPAM denotes Periodic Pyramid Attention Mechanism.
ponents with inclusion relationships to explicitly represent implicit periodic relationships. Firstly, as Peri-midFormer designed to focus on periodic components, we first normalize the original time series X E IR LxC that with length L and C' channels, then decompose it to obtain the seasonal part X, E LxC thus removing the interference of the trend part. For a detailed description of normalization and decomposition, please refer the Appendix A Then partition X,, into periodic components, following the approach used in Timesnet [13]. It's important to note that, inspired by PatchTST [9]. we retain the channel dimension C. as it is advantageous for Peri-midFormer in capturing periodic features within each channel (note that we adopt a channel independent strategy and the Figure shoows the processing of only one of the channels). The periodic components are extracted in the frequency domain, accomplished specifically through FFT:
A Aug (Amp (FFT (X,))) (f1, fx) = arg Topk (A). = # , 1 (1, ,A), (I)
fLe [4]]
where FF)()) and Amp(-) denote Fourier Transform and amplitude calculation, respectively. A e Rh represents the amplitude of each frequency, averaged across C' channels using Avg(-). Note that the j-th value Aj denotes the intensity of the j-th frequency periodic basis function, associated with
period length L To handle frequency domain sparsity and minimize noise from irrelevant high
frequencies [17], the top-k amplitude values { As corresponding to the most significant frequencies [ff..... fr} are selected, where & is a hyper-peeee parameter, beginning from to ensure the fundamental pyramid structure. Additionally, the original time series, we define fi = 1, with other frequencies arranged in ascending order. These selected frequencies correspond to l period lengths {pi,'.. PP)) arranged in descending order. Due to the frequency domain's conjugacy, only frequencies within {1...., [á]) are considered. Based the selected frequencies (S1- .kk) and their associated period lengths (PI.... .Pk), we partition the original time series into periodic components for each pyramid level, denoted as Cy:
hyper-parameter,
ensure the top level of the pyramid corresponds to
C. (C), C.,... ,,, C.), (1,.... ,.). (1,...,fk) fk),
(2)
where C denotes the n-th periodic component the L-th pyramid level. Here, the pyramid level index, starting from the top and increasing, with maximum value of k, indicating the number of levels determined by the selected periods. Similarly, n represents the component index within level, increasing from left to right, with maximum value of ft. indicating the number of components per level is determined by the frequency corresponding to that period the original series. The Periodic Pyramid can thus be represented as:
P = Stack (C₂) , (1...., k)).

---

Incx=
Incex L-1
Index CD ((nndee())))
Index(C)) Index(())
Figure 3: Inclusion relationships of periodic components (left) and Periodic Pyramid Attention Mechanism (right).
where Stack(+) denotes stacking operation. The constructed Periodic Pyramid, depicted in the upper right of Figure displays evident inclusion relationship among different levels, shown by connections between levels. Let R denote the relationship between pairs of periodic components from the upper and lower levels, determined by the presence or absence of overlap as follows:
1, Inder(C(r]1) Index(( (CT+)
R-
CC [2, (2, .k). ne-1, E (L. .
(4)
0, else
when R = 1, it signifies an inclusion relationship, while R - indicates no overlap. This is illustrated in the left half of Figure nh denotes the index of the n-th component in the b-th level. Inder(-) denotes the positional index of each data point within the periodic component at that level. Indices for points in the first level are contained in {O...., indices match those of the first level. However, due to varying component lengths, there may be slight differences in indices for the last portion. Nonetheless, this doesn't 't impact relationship determination between components across levels. In practice, the relationship between the components is realized by masking the corresponding elements in the attention matrix.
1}. For subsequent levels, most
Thanks to the inclusion relationships between periodic components across different levels in the Periodic Pyramid, complex periodic relationships inherent in ID time series are explicitly represented. Next, due to the varying lengths of the components, it's necessary to map C' to the same scale for subsequent Periodic Pyramid Attention Mechanism, with the equation provided as follows:
P' Projection (Padding (C))) & E {1,... k}, {1..... f
(5)
where (adding(-) denotes zero pero-padding the periodic components across the time dimension to match the length the original data, while Projection()) represents single linear mapping layer.
3.3 Periodic Pyramid Transformer (Peri-midFormer)
Once we have the Periodic Pyramid, it can be inputted into the Peri-midFormer, as depicted in The Peri-midFormer introduces specialized attention mechanism tailored for the Periodic Pyramid, called Periodic Pyramid Attention Mechanism (PPAM), shown in the right half of Figure Here, original connections are replaced with bidirectional arrows, and also added within the same level. These bidirectional arrows signify attention between periodic components. In PPAM, inter-level attention focuses on period dependencies across levels, while intra-level attention focuses on dependencies within the same level. Note that attention occurs among all components within the same level, not just between adjacent ones. However, for elarity, not all attention connections within the same level are depicted.
Figure 2
B
In Periodic Pyramid, periodic component Cj generally has three types of interconnected relation- ships (denoted as II) with other components: the parent node in the level above (denoted as IP), all nodes within the same level including itself (denoted A), and the child nodes in its next level (denoted as C). Therefore, the relationships of C? can be expressed by the following equation:
Ifm) = IP(") UA( (n) UC!")
p(n) C/-1j== {ne-1 1
if clse
A (n) $
(6)
cl Cen jj (net1 RRuRe =1}].
if & A else Ø
5

---

Meaa Coloooo
0
2
Projec
Mad
Figure 4: Periodic Feature Flows Aggregation.
The equation shows that component at the topmost level lacks a parent node, while one at bottommost level lacks child node. Based on the interconnected relationships I, the attention of the component Cу can be expressed as:
Σ exp (qikk./ /tn./Vdl Vdk) Vm
a, =
Σ mEi
exp (q:kg Tock///Vda. /dxx/
mEn(m)
where k, and denote query, key, and value vectors, respectively, as in the classical self-attention mechanism. used for selecting components that have interconnected relationships with Cj- kj represents the transpose of row m in K. dx refers to the dimension of key vectors, ensuring stable attention scores through scaling.
We apply this attention mechanism to each component across all levels of the Periodic Pyramid, enabling the automatic detection of dependencies among all components in the Periodic Pyramid and capturing the intricate temporal variations in the time series. For a detailed theoretical proof of the PPAM see the Appendix
3.4 Periodic Feature Flows Aggregation
Here explain the Periodic Feature Flows Aggregation used for reconstruction tasks. The output of the Peri-midFormer retains the original pyramid structure. To leverage the diverse periodic components
different levels, treat a single branch from the top the bottom of the pyramid as periodic feature flow, highlighted by the red line in Figure[4 Since periodic feature flow passes through periodic components at different levels, contains periodic features of different scales from the time series. Additionally, due to variations among periodic components within each level, each feature flow carries distinct information. Therefore, we aggregate multiple feature flows through Periodic Feature Flow Aggregation. This involves linearly mapping each feature flow to match the length of the target time series and then averaging it across multiple feature flows obtain the aggregated result Y.,. expressed in the following equation:
Y, ..Meenoolliig (Projection (fOT), Ber Crg,...,, Cp 1)) 12, A), E (1, fxk,, (8)
where Cne represents specific periodic component in the Peri-midFormen output, and (Ôp, C _ CTP) forms feature flow, shown in Figure Projection(.) maps each feature flow to match the target output length. Meanpooling(.) averages the feature flows. Y, indicates that this the output from the seasonal part. Since retained the channel dimension of the original time series, the result obtained after aggregating the periodic feature flows here becomes the shape of the final output. Finally, adding the trend part and de-normalization obtain the ultimate output.
4 Experiments
We extensively test Peri-midFormed on five mainstream analysis tasks: short- and long-term fore- casting, imputation, classification, and anomaly detection. adopted the same benchmarks as Timesnet [13], see Appendix for details. Due space limits, provide only summary of the results here, more details about the datasets, experiment implementation, model configuration, and full results can be found in Appendix.

---

Baselines The baselines include CNN-based models: TimesNet 13; MLP-based models:
LighTTS [TS]. DLinear [14] and FITS 21: Transformer-basee models: GPT4TS [20]. Time-
LLM 25 iTransformer (26). TSLANet
27 Reformer [28] Pyraformer [TO. Informer [16]
Autoformer [2], FEDformer (17), Non-stationary Transformer [1, ETSfrrmer [2). PatchTST [9]
Besides, N-HiTSS (30] and N-BEATS [3] are used for short-term forecasting. Anomaly Trans-
former [32] is used for anomaly detection. Rocket 33 LSTNet
34 TCN 8 and Flowformer 335
used for classification.
4.1 Main Results
5 displays the comprehen- sive comparison results between Peri- midFormer and other methods, con- sistently excels across all five tasks.
Figure
Long-term Forecasting (MSE) 0.3
Short-term Forecasting (OWA)
Classification (Accuracy) 77
04
4.2 Long-term Forecasting
165
0.095
Setups Referring to [13]. we adopt eight real-world benchmark datasets for long-term forecasting, including
0,055
Anomaly Detection (F1-Score)
Weather 36 Traffic (37). Electric-
Imputation(MSE)
0.045
ity
38 Exchange [34], and four
ETT [16] datasets (ETThI, ETTh2,
Perl-midFormer(ours)
Timesnet(2023)
ETTml, ETTm2). Forecast lengths are set to 96, 192, 336, and 720. For the fairness of the comparison, we set the look-back window for all the meth- ods to 512 (64 on Exchange), the re- sults for other look-back windows can be found in the Appendix H3
Time-LLM(2024)
PatchTST(2023)
FITS/2024)
DLinear(2023)
mansffomer(2024)
ETSformer(2022)
A5LANet(2024)
FEDformer(2022)
GPT4TS(2023)
Pyraformer(2021)
Figure 5: Model performance comparison.
Table 1: Long-term forecasting task. The results are averaged from four different series length
(96, 192, 336, 720}. See Table 13
and 14 for full results. Red: best, Blue: second best.
Models
Peri-mid Former
GPTATS
20
TSLANet
27
Time
LL.
FITS 21
DLinear [1]]
PatchTST 19]
TimesNet [3]
Pyraformer
Metrie
MSS:
MAG
MSB
MAU
MSI:
MAUL
MSB
MAR
MSB
MAD
MSU
MAE
MSU
MAK
MSE
MAB
MSU
MAL
Weather
|0.233
0.271
(0.237 0.270
0.276
0.291
10.225
0.257
00.244 0.281 (0.241 0.29410.222
10,226
0.266 0.249 0.286
(0.281
0.349
ETThl
|0.409
0.430 0.427 0.426 0.422
0.440
p0.408
0.423
0..07 0.422 00.488 0.438 (0.430 0.444 0.492 0.490 (0.913 0.748
ETTh2
[0.317
0.377 (0.354 0.394 0.328
0.385
J0.334
0.383
00.33 0,382 00.504 0.482 10.388
0.414 0.408 0.440 33.740 1.554
ETTml
|0.354
0,385 00.352 0.383 0.348 0.383 00.329
0.372 0.358 0.378 00.356 0.379 00.363 0.391 0.398 0.418 (0.724 0.609
ETTm2
|0.258
0.320 0.320]0.266 00.266 0.326 0.263 0.325 (0.251 0.313 0.2554 0.313 (0.275 0.342 (0.273 0.329 0.287 0.343 11356 0.797
(octricity|9.1522
0.249 167 0.263 0.159 0.224 158 0.252 0. 168 0.263 06. 0. 0.268 162 0,257 0.217 0.314
|0.299 0.391
Traffic
[0.392 0.270 00.414 0.294 0.397 0. 0.272|0.22 10.388 0.264 00.420 0.287 (0.433 0.305 |0. (0.392 0.270 0.622 0.332
|0.705 0.401
Exchange 10.346 0.393 (0.373 0.410 0.365 10.371 0.409
00.993 0.429 (0.495 0.493 |0.418 0.433 0.701
0.593
11155 0.844
Average [0.308 0.332 (0.324 0.346 0.320 0.341 10.308 0.334 00.322 0.344 (0.361 0.375 J0.331 0.350 0.422 0.402 11.147 0.711
Results From Table I is evident that Peri-midFormer performs exceptionally well, even completely outperforms GPT4TS and closely approaching the state-of-the-an method Time-LLM. While Time- LLM demonstrates remarkable capabilities in long-term forecasting, our Peri-midFormer shows clear advantages on the ETTh2, Electricity, and Exchange datasets. Although Time-LLM, achieves the best results, it relies on very large model, leading to significant computational overhead that is unavoidable. The same issue exists for GPT4TS. In contrast, our Peri- Peri-midFormer achieves performance close to that of Time-LLMM without requiring excessive computational resources, making it more suitable for practical applications. Further analysis of model complexity is provided in Section 6. In addition, our Peri-midFormer exhibits better performance with longer look-back window, as further detailed in the Appendix E.6
7

---

Table 2: Short-term forecasting task on M4. The prediction lengths are in {6, 48) and results are weighted averaged from several datasets under different sample intervals. (* means former, Station
means the Non-stationary Transformer.) See Table 12 for full results. Red: best, Blue: second best.
Models
Former
Time Time GPT4TS Timee
25
(20
113
[1
N-HITS
(40
N-BEATS
11]
22
LighrTS DLinear
[1]
[14]
FED
17
Station 111
Auto-
12
Pvma
[10
16
SMAPE
11833
11.883
11.991
11,829
12,059
11.927
11,851
14.718
13.525
13.639
12,840
12.780
12,909
16.987
14. 14.086
18,200
MASE 1,584
1.595
1,600
1.585
1.623
1,613
1.599
2,408
2.111
2,095
1.701
1,756
1.771
3.265
2.718 4.223
OWA
0.859
0.861
0.851
0.869
0.861
0.855
1.172
1.081
1,051
0.918
0,300
0.939
1.480
1,230
4.3
Short-term Forecasting
Setups For short-term analysis, we adopt the M4 39 which contains the yearly, quarterly and monthly collected univariate marketing data. We measure forecast performance using the symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE), and overall weighted average (OWA), which are calculated as detailed in the Appendix D.1
Results Table Table[]show shows that Peri-min PerimmedFormer outperforms Time-LLM, GPT4TS, TimesNet, and N- BEATS, highlighting its exceptional performance in short-term forecasting. In the M4 dataset, some data lacks clear periodicity, such the Yearly data, which mainly exhibits strong trend. similar situation is observed in the Exchange dataset for long-term forecasting task. Peri-midFormer performs well on these datasets due its time series decomposition strategy. For detailed analysis, please refer to the Appendix E.7
4.4 Time Series Classification
Setups We assessed Formmerssss'r'ss capac- ity for high-level representation learning via classification task. Mimicking settings akin to TimesNet [13), we tested it on 10 multi- variate UEA classification datasets from 42]. covering tasks like gesture recognition, ac- tion recognition, audio recognition, medical diagnoses, and other real-world applications.
Peri-midFormer
76.6
TSLANet
76.0
GPT4TS
74.0
TimesNet
73.6
73.0
Stationformer
72.7
Rocket
72.5
Transformer
71.9
Reformer
Autoformer
ETSformer
71.0
Results As shown in Figure 6 Peri- midFormer achieves an average accuracy of 76.6%, surpassing all baselines including TSLANet (76.0%). GPT4TS (74.0%), Times- Net (73.6%), ) and all other Transformer-basse methods. This suggests that Peri-midFormer has excellent time series representation capa- bilities. See Appendix H.I for full results.
Pyraformer
70.8
FEDformer
70.7
LightTS
70.4
TCN
68.7
Dlinear
67.5
Average Accuracy(%)
Figure 6: Model comparison in classification. The results are averaged from 10 subsets of UEA.
4.5 Imputation
Setups validate Peri-nidFormer's imputation capabilities, we conduct experiment on six real- world datasets, including four ETT datasets [16] (ETThI, ETTh2, ETTm1, ETTm2), Electricity [8) and Weather [36]. We evaluate different random mask ratios (12.5%, 25%. 37.5%, 50%) for varying levels of missing data. Notably, due to the large number of missing values, the time series do not reflect their original periodicity. Therefore, before imputation, we simply interpolate the original missing data through linear interpolation strategy in order to use Peri-midFormer efficiently, which we call pre-interpolation. For a description of pre- preiinerpoolation and its impact on other methods, please refer to the Appendix E..]
Results Table demonstrates Peri-midFormer's outstanding performance on specific datasets (Elec- tricity and Weather), surpassing other methods significantly and securing the highest average scores. However, its performance on other datasets was ordinary, possibly due to the lack of obvious periodic characteristics in them.
8

---

Table 3: Imputation task. We randomly mask (12.5%, 25% 37.5%. 50% time points of length-96 time series. The results are averaged from 4 different mask ratios. (* means former, Station means the Non-stationary Transformer.) See Table 13 for full results. Red: best, Blue: second best.
Models
Peri-mid Former
GPT4TS
TimesNet
PatchTST
ETS* 29
LighrTS
DLinear 14
FED=
Station
Autos
Pera 10
20
13
19
15
Meuic
MSE MAE MSE MAB] MSE MAE MSE MAE MSE MAE IMSE MAEJ MSE MAE MSE MAE MSE MAE MSE
MAE
MSE MAE
ETTmi
0.034 0.116 0.022 0.105)0.027 107 0.004 0.140 0.120 0.253 10.104 0.218 O 0,206 0.062 0.17? 10.036 0.126 0.051 0,150 0.570
ETTm2
0.025
0.087 |0.02 0.084|o.000 (..02 0.029 0.102 0.208 0.327 (0046 0.151 (0.096 0.208 0.101 0.215 (0.026 0.099 (0.029 0,105 0.466 0.508
ETTH
0.083
0.191 |0.069
m7r n 115 15 o 224 00.00 0.329 (0.284 0:373 201 O 305 (0.17 0.246 (0.094 0.201 10.103 0/214/0.842 0.682
ETTb2
0.054
1.6 (0.048
10019 0..465 0.163 0.367 0.436 10.119 0.43610.119
0.250 J42 0.163 0.279 (0.053 0.152 (0:055 0.156|1.079 0.792
Elecrricity
0.060 0. 0,000 0.207
p0002 0.210 0.2 18183 0.214 0.339 31 D. 0.2621o 132 260 (1. 80 0,259 ja. 0.218 10100 0.225|o.29) 0.382
Weather
0.028 0.039J0.03 0.056
0.05: 0.144 0.076 0.171 (0.055 0.117 0.110 0.099 0.203 J0.032 0.059 0.031 0.057j0.152 03535
Average
0.047 0.123 (6.048 6.12%
(0050 1313//0.053 1.199 0.164 0.309 0.309/0.123 0.229/o 10 232 (0.112 0.230 j0.077 0.143 (00022
0.151 0.528
4.6 Time Series Anomaly Detection
Setups For anomaly detection, we assess models on five standard datasets: SMD 43 MSL 44]
SMAP [44), SWaT 455 and PSM [46). To ensure fairness, we exclusively use classical reconstruction error for all baseline models, aligning with the approach in TimesNet [13]. Specifically, normal data is used for training, and simple reconstruction loss is employed help the model learn the
distribution of normal data. In the testing phase, parts of the reconstructed output that exceed certain threshold are considered anomalies. We use a point adjustment technique combined with a manually set threshold for this purpose.
Table 4: Anomaly detection task. We calculate the FI-score (as %) for each dataset. (* means former, Station means the Non-stationary Transformer.) A higher value of F1-score indicates better performance, See Table [6 for full results. Red: best, Blue: second best.
Models
Peri-mid GPT418
Former
20
TimesNet
[13
PaGSIST 11
ETS- 250
HED 17
Lithis 15
DLinnar
[14]
Station (11
Autis
Prtze 10
Anoe 52
16
In=
Res 28
Log Iranse
47
Transe 48
SMD
85.95
86.89
84.61
84.62
83.13
85.08
82.53
77.1
84.62
85.11
83.04
85
81.65
75.32
76.21
79.56
MSL
81.83
82.45
81.84
78.7
85.03
78.57
78.95
84 88
77.5
79.05
84,86
83.31
84.06
84.40
79.57
78.68
SMAP
68.62
72.88
69.39
68.82
69.50
70.76
69.21
69.26
71.09
71.12
71.09
71.18
69.92
70.40
69.97
69.70
SWaT
93.40
94.23
93.02
85.
84.91
99.19
93.33
87 52
79.88
92.
91.78
83
81.43
82
80.52
80.37
PSM
97.19
97.13
97.34
96.08
91.76
97.23
97.15
93.55
97.29
93.29
82.08
79.40
77
73.61
76.74
76.07
Avg F1|
85.40
86.72
85.24
82.79
82.87
84.97 84.23
82.46
|82.08
84.26
82.57
80.50
78.83
17.331
76.60
76.88
Results The results in Table 4 illustrate that Peri-midFormer'' anomaly detection capability is second only to GPT4TS, but large gap does exist. This is due to the binary event data nature in the anomaly detection dataset [21]. which makes it difficult for Peri-miae PerimmidFormer to capture useful periodic characteristics, leading to general performance.
5
Ablations
Table Ablation experiments on long-term forecasting task to verify the effect of each component. Red: best, Blue: second best.
Dalasets
ETThl
ETTh2
Electricity MSB
Weather MSE
Traffic
Variant
MSR
MAE
MSE
MAR
MAF
MAR
MSE
MAE
Pyraformer
0.913
0.748
0.826
0.703
0.299
0.391
0.281
0.349
0.705
0.401
w/o period components
0.734
0.781
0.407
0.442
0.212
0.345
0.261
0.333
0.499
0.391
w/o PPAM
0.581
0.599
0.344
0.393
0.164
0.259
0.254
0.291
0.415
0.364
w/o Feature Flows Aggregation
0.433
0.447
0.341
0.391
0.155
0.250
0.244
0.280
0.397
0.277
Perl-midFormer
0.409
0.430
0.317
0.377
0.152
0.249
0.233
0.271
0.391
0.269
Setups We performed ablation experiments on key modules of Peri-midFormer for the long-term forecasting task, with results presented in Table [5] The table outlines the progression of module additions, from top to bottom. "Pyraformer" refers to the Pyraformer (10), building the pyramid down- top with convolutions and employing a simple two-fold relationship for attention distribution. "w/o periodic components" constructs pyramid top-down by dividing the time series into patches without

---

Classification the UEA Heartbeat
Long-term forecasting of length 720 on ETTh2
Pert-midFormer
0.60
DLinear
90
0.55
Accuracy (%)
1G FLOPs 10G FLOPs
Autoformer
10G FLOPs
100G FLOPs
85
Informer
MSE
0.45
Timesnet
80
TSLANat
Pyraformer
Tim Tmee
0.40
TSLANet
GPT4TS
75-
Autoformer Crosstormer
FEDformer
GP GPT4T
0.35
Perl-midFoormer
PatchTST
(12750G FLOPs)
70
PatchTST
0
5
60
Number of Parameters (Millions)
Number of Parameters (Millions)
Figure 7: Number of training parameters and FLOPs for Peri-midFormer versus baseline in terms of classification accuracy for the UEA Heartbeat dataset (left) and long-term forecasting MSE for the ETTh2 dataset (right). In the left graph, the closer to the top left, the better, while in the right graph, closer to the bottom left, the better. Note that in the long-term forecasting, we did not fully depict the corresponding sizes due the oversized FLOPs of Time-LLM, but instead illustrated it with text.
considering periodic components. "w/o PPAM" divides the time series into periodic components but without Period Pyramid Attention Mechanism, using periodic full attention instead, wherein attention computed among all periodic components. "w/o Feature Flows Aggregation" employs PPAM but without Periodic Feature Flows Aggregation. "Peri-midFormer" indicates our final approach.
Results Table SiolleJilllsssrr illustrates the incremental performance enhancement achieved with the integration of each additional module, validating the effectiveness of Peri-midFormer. Notably, good results achieved even without PPAM. This can be attributed to the model's ability extract periodic characteristics inherent in the original time series data by delineating the periodic components. However, without highlighting inclusion relationships through PPAM, periodic full attention's ability to capture temporal changes limited, emphasizing the significance PPAM.
Complexity Analysis
We conducted experiments the model complexity of Peri-midFormer using the Heartbeat dataset for the classification task and the ETTh2 dataset for the long-term forecasting task. We considered the number of training parameters, FLOPs, and accuracy (or MSE). The results depicted in Figure 7 the classification task, Peri-midFormer not only achieves a significant advantage in accuracy but also requires relatively fewer training parameters and FLOPS, much less than many methods such as TimesNet, GPT4TS, Crossformer, and PStchTST. the long-term forecasting task, Peri-midFormer achieves the lowest MSE without requiring the enormous FLOPs that Time-LLM does. This shows that although Time-LLM has strong long-term forecasting capabilities on most datasets, its computational demands unacceptable. See Appendix E.4 for more analysis.
Further Model Analysis is provided Appendix
7 Conclusions
In this paper, we introduced method for general time series analysis called Peri-midFormer. It leverages the multi-periodicity of time series and the inclusion relationships between different periods. By segmenting the original time series into different levels of periodic components, Peri- midFormer constructs Periodic Pyramid along with its corresponding attention mechanism. Through extensive experiments covering forecasting, classification, imputation, and anomaly detection tasks, we validated the capabilities Peri-midFormer in time series analysis, achieving outstanding results across all tasks. However, Peri-midFormes exhibits limitations, particularly in scenarios where the periodic characteristies are less apparent. We aim to address this limitation in future research to broaden its applicability.
10

---

Acknowledgement
This work supported by the National Natural Science Foundation of China under Grant Nos. 62276205, U22B2018, and Graduate Student Innovation Fund under Grant Nos. YJSJ24012.
References
[1] Qingsong Wen, Linxiao Yang. Tian Zhou, and Liang Sun. Robust time series analysis and applications: An industrial perspective. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4836 4837, 2022.
[2] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619(7970)5533-538, 2023.
[3] Yaoran Chen, Candong Cai, Leilei Cao, Dan Zhang, Limin Kuang, Yan Peng, Huayan Pu, Chuhan Wu, Dai Zhou, and Yong Cao. Windfix: Harnessing the power of self-supervised learning for versatile imputation of offshore wind speed time series. Energy, 287:128995, 2024.
[4] Haotian Si, Changhua Pei, Hang Cui, Jingwen Yang, Yongqian Sun, Shenglin Zhang, Jingjing Li, Haiming Zhang, Jing Han, Dan Pei, et al. Timeseriesbench: An industrial- -rade benchmark for time series anomaly detection models. arXiv preprint arXiv:2402 10802, 2024.
[5] Zhen Liu, Wenbin Pei, Disen Lan, and Qianli Ma. Diffusion language-shapelets for semi- supervised time-series classification. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14079-14087, 2024.
[6] Hongming Li, Shujian Yu, and Jose Principe. Causal recurrent variational autoencoder for medical time series generation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8562-8570, 2023.
[7] Minrong Lu and Xuerong Xu. Trnn: An efficient time-series recurrent neural network for stock price prediction. Information Sciences, 657:119951, 2024.
[8] Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable represen- tation learning for multivariate time series. Advances in Neural Information Processing Systems, 32, 2019.
[9] Yangdong He and Jiabao Zhao. Temporal convolutional networks for anomaly detection in time series. J. Phys. Conf. Ser, 2019.
[10] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2021.
[11] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Processing Systems, 2022.
[12] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, pages 101-112, 2021.
[13] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.
[14] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[15] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less more: Fast multivariate time series forecasting with light sampling-oriented mip structures. arXiv preprint 22Xiiv2207.011220 2022.

---

[16] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In 35th AAAI Conference on Artificial Intelligence, pages 1116-11115,, 2021.
[17] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning. 2022.
[18] Zelin Ni. Hang Yu, Shizhan Liu, Jianguo Li, and Weiyao Lin. Basisformer: Attention-based time series forecasting with learnable and interpretable basis. Advances Neural Information Processing Systems, 36, 2024.
[19] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2022.
[20] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in Neural Information Processing Systems, 36, 2024.
[21] Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10k parameters. In The Twelfth International Conference on Learning Representations, 2023.
[22] Tao Dai, Beiliang Wu, Peiyuan Liu, Naiqi Li. Jigang Bao, Yong Jiang, and Shu-Tao Xia. Periodicity decoupling framework for long-term series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[23] Jinliang Deng, Xiusi Chen, Renhe Jiang, Du Yin, Yi Yang, Xuan Song, and Ivor W Tsang. Disentangling structured components: Towards adaptive, interpretable and scalable time series forecasting. IEEE Transactions on Knowledge and Data Engineering. 2024.
[24] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU Timemixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[25] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. The Twelfth International Conference on Learning Representations, 2024.
[26] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang. Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2023.
[27] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, and Xiaoli Li. Tslanet: Re- thinking transformers for time series representation learning. arXiv preprint arX/v:2404.08472. 2024.
[28] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020.
[29] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Expo- nential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022.
[30] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 6989-6997 0997, 2023.
[31] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. International Conference on Learning Representations, 2019.
12

---

[32] Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations, 2021.
[33] Angus Dempster, François Petitjean, and Geoffrey I Webb. ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5))1454-1455, 2020.
[34] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The sl international ACM SIGIR conference on research & development in information refrieval, pages 95-104, 2018.
[35] Haixu Wu, Jialong Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Lineariz- ing transformers with conservation flows. In International Conference on Machine Learning, pages 24226-24242. PMLR, 2022.
[36] Wetterstation. Weather. https: : // /ww.//wwwwooo...o..... bgc- jena.jjjaa..... mpg. grog.ddeveetee.
[37] PeMS. Traffic. attp://pemte wa.
ca. gov/
[38] UCL.
Electricity.
Clectrrccty..oodiggrraas11111111
[39] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4):802- 808, 2018
[40] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv: 12886, 2022.
[41] T. Zhang, Yizhuo Zhang, Wei Cao, J. Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light hampling-orientee mlp structures. arXiv preprint arXiv:2207.00 01 186, 2022.
[42] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint 181 1.10000 2018
[43] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 20288-2837222 2019.
[44] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Süder- ström. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 387-395, 2018.
[45] Aditya P Mathur and Nils Ole Tippenhauer. Swat: A water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater). pages 31-36. IEEE, 2016
[46] Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2485-22882 2885-2494, 2021.
[47] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NewrIPS, 2019.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[49] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
13

---

A
Method Details
Strategy 1
Input Xe XeR
Strategy 2
Normalization
Seasonal part
Trend part
Projection
De-narmalization
X, R
Decomposttion
H. = H
Y E
Y YERT
Tomporal embetding
Padiing & Projection
Postiional wwb.lllbbdiiiii eildding
FFT pariods
Strategy
Peri-midFan
Downstream
Y dann Stratogy
S
periods
Portodic
Figure 8: Full flowchart.
Here, we provide further elaboration the details of Peri-midFormet Its full flowchart is depicted in Figure depicts two strategies for input. The strategy indicated by the red line is for classification task, where de-normalization and time series decomposition task, there is no need to reconstruct the original data; therefore, the trend part does not need to be extracted and added back. Additionally, discriminative feature for classification data, so it cannot be separated from the original data before feature extraction. In the strategy indicated by the blue line, employed for reconstruction tasks, Peri- midFormer needs to focus on the periodicity in the time series. Therefore, we utilize de-normalization and time series decomposition to eliminate other influencing factors. To achieve this, we first refer to [1 to address instability factors, We normalize the input X [21, 22, [ 2z2 2/, E R.C.C
used. This is because, in classification
important note that the trend part is a significant
ØL E R BLXC to obtain
Xnorm
1x
1 L
Σ
Tit 0x
Σ
(x)
(α. 1xx):
L
xx
where Mx: xx Cx1 are the mean and variance, respectively, dx means the element- wise division and is the element-wiss product. Normalization reduces the disparity in distribution among individual input time series, thereby stabilizing the model input distribution.
Then, to remove the trend part from the time series and only preserve the seasonal part for Peri- midFormer, we refer to 122 for time series decomposition, as shown in the following equation:
X, = Aool CongPool(Padiiee (addinggXX.ann (Xnorm)),
(10)
X, X Xt,
where X., X, E LR denote the seasonal and the trend part respectively. We adopt the AugPool(.) for moving average with the padding operation to keep the series length unchanged.
The seasonal part X, obtained after decomposition can be directly input into Peri meri-mrdForrrr. After the output from Peri-midFormer, we add the trend part back. then de-normalize it to obtain the final output Y
[Уr 82, .... yrr:
Y. Y Projection(X.))
= xx C Yi + fx:
(11)
where H. represents the Peri-midFormer model, Y, represents the output of Peri-midFormer, Projection( represents mapping the trend part to the target output length, and ÿj = 0x O yh + Px denotes de-normalization
B
Visualization
To provide a clearer demonstration of Peri-midFFrmerr's representational capabilities, Figure
10
and 11 visualize some of the results for imputation, long-term forecasting and short-term forecasting. respectively. illustrates that Peri-midFormer outperforms other methods in capturing periodic variations in time series.
14

---

(1) Imputation on Weather dataset under 50% mask ratio
3,20
Groundinst
$.37
3.30
5.06
3.94
5.06
3:10
0.10
8.35
0.00
800
8,00
601
801
000
800
Peri-midFormer (ours)
GPT4TS
PatchTST
0.2
Joundrus
235
632
TimesNet
FEDformer
Dlinear
(2) Imputation on Electricity dataset under 50% mask ratio
25-
25-
22-
32-
23-
23-
33-
13-
32-
43
Peri-min Formmer (ours)
GPT4TS
PatchTST
Greandhaah
Pridicos
45
TimesNet
FEDformer
Dlinear
Figure 9: Visualization of imputation.
15

---

(1) Long-term forecasting with 96 prediction length ETTh2
125
20
104
Peri minFoeeerrrr (ours)
GPT4TS
TSLANet
175
n6
PatchTST
TimesNet
Dlinear
(2) Long-term forecasting with 96 prediction length on Electricity
-35
Peri minForeer (ours)
GPT4TS
TSLANet
PST
TimesNet
Dlinear
Figure 10: Visualization of long-term forecasting.
16

---

(1) Short-term forecasting on M4 Weekly
0050
26.0
255
20:0
Formmer (ours)
GPT4TS
PatchTST
Mm
300
201-
201-
2400
TimesNet
FDEformer
Dlinear
(2) Short-term forecasting on M4 Monthly
1322
12:00
12:00
100
Peri-midFormer (ours)
GPT4TS
PatchTST
13:22
12:00
12:00
6004
200
TimesNet
FEDformer
Dlinear
Figure 11: Visualization short-term forecasting.
17

---

C
Dataset Details
detailed description of the dataset given Table 6
Table 6: Dataset descriptions. The dataset size is organized in (Train, Validation, Test).
Tasks
Dataset
Dim
Series Length
Dataset Size
Information (Prequency)
ETTml, ETTm2
7
196, 192, 336, 720)
(34465, 11521, 11521)
Electricity (15 mins)
ETThI, ETT12
7
(96, 192, 336, 720)
(8545, 2881, 2881)
Electricity (15 mins)
Forecasting
Electricity
321
196, 192, 336, 720)
(18317, 2633, 5261)
Flectricity (Hourly)
(Long-tern))
Traffic
862
[96, 192, 336, 720)
(12185, 1757, 3509)
Transportation (Hourly)
Weather
196, 192, 336, 720)
(36792, 52711 10540)
Weather (10 mins)
Exchange
8
[96, 192, 336, 720)
(5120, 665. 1422)
Exchange rate (Daily)
M4-Yearly
6
(23000. 0, 23000)
Demographic
M4-Quarterly
8
(24000, 0, 24000)
Finance
Forecasting
M4-Monbly
18
(48000, 0, 48000)
Industry
(short-term)
M4-Weakly
13
(359, 0, 359)
Macro
M4-Daily
14
(4227. 0, 4227)
Micro
M4-Hourl
48
(414, 0, 414)
Other
ETTml, ETTm2
7
96
(34465, 11521, 11521)
Flectricity (15 mins)
Imputation
ETThi, ETTh2 ETTh2
7
96
(8545, ,,
Electricity (15 mins)
Electricity
321
96
(18317, 2633, 5261)
Electricity (15 mins)
Weather
96
(36792, 5271, 10540)
Weather (10 mins)
CnthasiConcentrattonn
1751
(261. 0, 263)
Alcohol Industry
PaceDetection
144
62
(5890, 3524)
Face (250Hz)
Handwriting
3
152
(1500 0, 850)
Handwriting
Heartheat
405
(204, 0, 205)
Heart Beat
Classification
SapaneseVowels
12
29
(270, 0, 370)
Voice
(LBA)
PEMS-SF
963
144
(267, 0, 173)
Transportation (Daily)
SellRegulationSCPP
6
896
(268. 0, 293)
Health (256Hzz
SellReguhttioSCCP2
7
1152
(200, 0, 180)
Health (256Hzz
SpokcnArabicDigits
13
93
(6599, 0, 2199)
Voice (1 (1102HHz)
UWaeeGesturLLibrar
315
(120, 0, 320)
Gesture
SMD
100
(566724, 141681, 708420)
Server Machine
Anomaly
MSL
55
100
(44653, 11664, 73729)
Spacocraft
Detection
SMAP
25
100
(108146, 27037, 427617)
Spacecraf
SWaT
51
100
(396000, 99000, 449919)
Infrastructure
PSM
25
100
(105984, 26497, 87841)
Server Machine
Since our datasets setup is the same as Timesnet [3] an excerpt from it describes the datasets.
D Experimental Details
All the deep learning networks are implemented in PyTorch and trained on NVIDIA 4090 24GB GPU. We repeated each experiment three times to eliminate randomness. The detailed experiment configuration is shown in Table
D.1 Metries
We utilize various metrics to evaluate different tasks. For long-term forecasting and imputations, we employ the mean square error (MSE) and mean absolute error (MAE). In anomaly detection, we utilize the Fl-score, which combines precision and recall. For short-term forecasting, we utilize the symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE), and overall weighted average (OWA), with OWA being unique metric used in the M4 competition.
18

---

Table 7: Experiment configuration of Peri-midFormer. All the experiments use the ADAM 49
optimizer with the default hyperparameter configuration for (31, .2) as (0.9, 0.999).
Model Hyper-paramesee parameter
Training Process
Tasks / Configurations
k
Layers
daodel
LR"
Loss
Batch Size
Epochs
Long-term Forecasting
2-5
1-3
128-768
10
-5 x 10
MSE
2-32
15
Short-term Forecasting
2-3
1-2
256
10-4
SMAPE
2
15
Imputation
2--5
1-2
64-768
10 -5 x 10
MSE
2-4
15
Classification
2-8
1-4
16- 16-128
10 - x 10
Cross Entropy
2-64
20
Anomaly Detection
2-5
1-4
8-32
10 10-4 - 10
MSF
2-32
3
L.R means the initial learning rate.
These metrics are computed follows:
200
T
X;
Y,
SMAPE
T
X + Y
100
X;
Y,
MAPE T
Σ
X
(12)
T
X
Y,
MASE T
Σ
2jz9+1
XX
X j-g
SMAPE
MASE
OWA 2 SMAPENaive
MASENaive2
where is the periodicity of the data. X, Ý E T are the ground truth and prediction result of the future with T time pints and C dimensions. X; means the i-th future time point.
E
Model Analysis
E.1 Hyper Parameter Analysis and Model Limitations
In Equation ( we introduced hyperparameter & to select the most important frequency, which also determines the number of levels in the Periodic Pyramid. We conducted sensitivity analysis on it, as shown in Figure 12] It's evident that our proposed Peri-midFormer exhibits relatively stable performance across different choices of for all four tasks. However, there are still some fluctuation in results among different l values depending on the task and dataset, which are determined by the periodic characteristics in the dataset. To illustrate this, we visualize individual data for long-term forecasting and classification tasks, shown in Figure 13 It can be observed that in the long-term forecasting task, the Etth dataset exhibits clear periodicity. Therefore, with larger k. Peri-midFormer can capture more periodic information, resulting in better performance. In contrast, the Fith2 dataset has less obvious periodicity, so achieves better results with smaller A as larger introduce unnecessary noise, affecting model performance. In the classification task, the Ethanol(onncentration dataset is difficult to classify, whereas a larger allows for the construction of Periodic Pyramid with more levels, thus extracting more representative features and achieving higher accuracy. In addition, the SelfRegulationSSCII dataset contains a lot of irrelevant information, leading to decreased accuracy. Therefore, we adjusted k values differently for different tasks and datasets, as shown in the range Table 7
noise, so larger would focus on
The above analysis reveals that the limitation of Peri-midFoormer lies in its inability to fully leverage its advantages on datasets with poor periodicity characteristies. We plan address this issue in our future work.
19

---

Imputation of 50% masked
Long-term Forecasting of 96 prediction length
0.065
0.35
0.055
ErThZ
0.050
Weather
ETTm1
0.30
0.045
ETTh1
ET7ml
Weather
0.040
FTTh2
FTTm2
0.20
0.15
value
value
Anomaly Detection
Classification
100
85.0
82.5
71-5core
80.0
Accuracy13)
80
EmnanilConcentaatonn
77.5
Sell ulationCP
SMD
60
SpokensrrabiCiiiit
UWaveGestureLibrary
72.5
70.0
40
30
Figure 12: Sensitivity analysis of hyper-parameters l in each task.
(1) Long-term Forecasting task
Etthl dataset
Etth2 dataset
(2) Classification task
74
EthanolConcentration dataset
SelfRegulationSCP1 dataset
Figure 13: Visualization of the Etth and Euth2 datasets for the Long-term forecasting task (1), and the EthanolConcemtration and SelfRegulationSCPI datasets for the classification task (2),
E.2 Periodic Pyramid Attention Mechanism Analysis
To illustrate the PPAM more clearly, we visualize the original time series and attention scores within the periodic pyramid for the SelReegulatoonCCP dataset in the classification task, as shown in Figure It is evident that the attention scores among components are distributed based on inclusion and adjacency relationships, meaning that components in different levels with inclusion relationships or those in the same level have higher attention scores. This demonstrates the rationality of the Periodic Pyramid structure. Additionally, when the hyperparameter
14
Equation ( set to 2, the PPAM degenerates into periodic full attention, wherein attention is computed among all periodic
20

---

Prag of
Why
wuch
Ny Wmyt
-0.2
-0.4
-0.6
W'
ley
1
6
Original time series
Periodic pyramid
Attention score
Figure 14: Visualization of the original time series and attention scores within the periodic pyramid for the SelfRegulationSCP2 dataset in the classification task. The left side shows the original data, the middle displays the corresponding pyramid structure, and the right side depicts the attention scores within the pyramid. In this example, k = 3, fi 1, f22 = 2, J3 4.
21
19
1.
33
12
10
-13
Figure 15: Original time series of Electricity dataset (left) and Periodic Pyramid Attention score (right).
components. To illustrate this, we visualize the Electricity dataset in the long-term forecasting task and its corresponding attention distribution, as shown Figure [5] The figure shows that the Periodic Attention Mechanism can capture the dependencies among the periodic components and identify which components belong to longer component (note that components with larger amplitudes). This is attributed to the separation of the periodic components, which explicitly expresses the hidden periodic inclusion relationships in the time series. The above analysis demonstrates the effectiveness of the PPAM.
corresponds to periodic
E.3 Periodic Feature Flows Analysis
To intuitively understand the Periodic Feature Flows, we visualize as shown in Figure 16 On the left side is the pyramid representation of the Periodic Feature Flows, with the horizontal axis representing the number of feature flows and the vertical axis representing the length of each feature flow. The feature flows are divided into multiple levels, each containing multiple periodic components. The red box encloses one feature flow, with its position from top to bottom corresponding to the pyramid from top to bottom. It can be observed that some adjacent feature flows the same the corresponding positions, this is because they pass through the same periodic component. On the right side, the waveform of each feature flow displayed in different colors, with the position from left right corresponding to the top to the bottom of the pyramid. It can be observed that each feature flow differs, which why necessary to aggregate different feature flows. The aim is to fully utilize the information from each periodic component to better reconstruct the target sample.
E.4 Training/nfferecinn Cost
To further validate the computational complexity and scalability of the proposed method, we con- ducted detailed experiments on Electricity and ETThl datasets for the long-term forecasting task. These experiments focused on the complexity and actual time of training and inference, as well memory usage, with results presented in Tables g and
can be seen that proposed Peri-
21

---

120
100
M
Fosturr dimennsion
80
10
12
14
Number of branches
Figure 16: Visualization of the pyramid form of Periodic Feature Flows (left) and the waveform of Periodic Feature Flows (right).
Table 8: Complexity and scalability experiments in the long-term forecasting of length 720 on Electricity
Methods
Train FLOPs
Test FLOPs
CPU
Train
Test time (per sumple)
MSE
memory
memory
memory
(8)
Peri-midFFormer
244G
61G
4816M
2753M
2101M
5807s
0.0263s
0.181
Time-LLM
12.69T
61454M
12503M
0.192
GPT4TS
716G
3G
11236M
10572M
1047IM
22366s
0,0280s
0.294
PatchTST
549G
137G
11500M
8894M
2717M
7029s
0,0219:
0.294
TimesNet
934G
18112M
21021M
4225M
9769s
0.0501s
0.294
DLinear
8G
237M
1892M
8045M
1427M
331s
0.0032s
0,204
Autoformer
238G
8G
13157M
8488M
2738M
1477s
0.1390s
0,236
Table 9: Complexity and scalability experiments in the long-term forecasting of length 720 on ETThl
Methods
Train FLOPs
Test FLOPs
GPU
Train CPU
Test CPU
Train time
Test time (per sumple)
MSR
memory
memory
memory
(8)
Peri-midfFormrr
23.51G
4.28G
2443M
3190M
75M
1623s
0.0036s
0.445
Time-LLM
25.36T
25.36T
94994M
5795M
2893M
30506s
0.0384s
0,442
GPT4TS
716G
3G
11657M
3250M
170M
2920s
0.0043
0.477
PatchTST
51.24G
1.58G
3683M
2895M
30M
550s
0.0013s
0.473
TimesNet
116.32G
3,67G
2932M
3005M
20M
1640s
0.0076s
0,551
DLincar
165.05M
5.17M
1449M
1310M
28M
190s
0.0006s
0.472
Autoformer
202.34G
6.34G
12981M
3216M
16M
3400s
0,01
0.501
midFormer demonstrates significant advantage in computational complexity on the Electricity dataset and achieves the lowest MSE. Similarly, on the ETThl dataset, Peri-midFormer is computa- tional cost and inference time are not disadvantages. Instead, it achieves lower MSE, second only to TiMe-LLM, while having much lower computational cost and inference time compared to it. These results highlight the advantages of our method in terms of computational complexity and scalability.
E.5
Pre-interpolation
Since Peri-midFormer is designed focus on the periodic components of time series, directly handling data with missing values in imputation tasks may prevent it from correctly capturing the periodic characteristies of the original data without missing values, thus affecting its imputation effectiveness. Therefore, to adapt Peri-midFormet for imputation tasks, we first apply a linear interpolation strategy (Equation (1)) to the data with missing values to partially restore the periodic characteristics of the original data before inputting it into Peri-midFormer for further imputation. The visualization of original data, data with 50% missing values, and pre-interpolated data of Electricity
dataset are illustrated in Figure 17 It is evident that missing values significantly disrupt the periodic
22

---

(1) Data
MT
MM
-02-
-00
0.0
-050
-06
1.0
-120
Original data
Data with 50% missing values
Pre-interpolated data
(2) Data 2
1.50
1.50
4.00
Original data
Data with 50% missing values
tre-interrpolaed data
(3) Data 3
18.-
2.-
12
1.0
14-
02
0.-
06-
-.20
-15
Pre-interpolated data Figure 17: Visualization of original data, data with 50% missing values, and pre-interpolated data of
Original data
Data with 50% missing values
Electricity dataset.
characteristics of the original data, while pre-interpolation partially restores them. It is worth noting that this is not speculative action but effort to further explore the potential of deep learning models in imputation tasks. Our goal is for deep models to capture subtle variations in time series to achieve imputation in the details rather than wasting effort goals achievable through simple linear interpolation. Additionally, since we can retain the indices of missing values in practical applications, there concern about the reliability of evaluating the imputation results. To validate the effectiveness of the pre-interpolation strategy, we applied to other methods, as shown in Table
10. We conducted experiments four mask ratios {0.125, 0.25, 0.375, 0.5), where the first only the pre-interpolation strategy. From the results, it can be seen that pre-interpolation significantly improves the performance of each method across all four mask ratios, demonstrating its importance for deep learning- -aasid methods in interpolation tasks.
The simple linear interpolation strategy adopted can be represented by the equation:
Ra 2
if (Ibe,oore + None) and (Kafter + None
Finter
Fafters if (Ibefore None) and (Tafter + None if (Ibefore # None) and (Rafter None
(13)
Fbefore:
where Vinter represents the value used to replace the value, Thefore represents the nearest non-zero value before the 0 value, and Tafter represents the nearest non-zero value after the 0 value.
23

---

Table 10: Ablation Experiments of pre-interpolation in inputation task on ECL dataset.
w/o pre-interpolation
Pre-interpolation
Methods
Metrics
0.125
0.25
0.375
0.5
0.125
0.25
0.375
0.5
Only pre-interpolation
MSE
0.086
0.110
0.149
0.206
MAE
0.188
0.213
0.251
0.301
Pyri-ridFormer
MSE
0.073
0.092
0.107
0.122
0,044
0.052
0.063
0.079
MAE
0.187
0.214
0.231
0.248
0.135
0.149
0.166
Timesnet
MSE 0.085
0.089
0.094
0.100
0.081
0.083
0.086
0.091
MAE
0.202
0.206
0.213
0.221
0.196
0.198
0.201
0.207
Pyraformer
MSE MAR
0.297
0.294
0.296
0.299
0.165
0.173
0.383
0.380
0.381
0.383
0.290
0.291
0.293
0.295
Dlineur
MSE
0.092
0.118
0.144
0.175
0.050
0.062
0.789
0.105
MAE
0.214
0.247
0.276
0.305
0.144
0.164
0.189
0.225
PatcTTS
MSR
0.055
0.065
0.076
0.091
0.050
0.089
0.070
0.087
MAE
0.160
0.175
0.189
0.208
148
0.164
0.181
0.202
ETSformer
MSE
0.196
0.207
0.219
0.235
0.102
0.104
0.109
0.117
MAE
0.321
0.332
0.344
0.357
0.230
0.233
0.238
0.247
0.24
96
720
192
1000
0.22
336
0.20
MSE
0.18
0.16
0.14
100
200
300 400 500 600 700 800 900 1000
look-back window length
Figure 18: Performance of different length look-back window on the long-term forecasting task in Electricity dataset. Prediction lengths are {96, 192, 336, 720, 1000}.
E.6 Look-back window length
Here, we investigate the performance of Peri-midFormer under different look-back window lengths in long-term forecasting task the Electricity dataset. As shown in Figure [88] experimented only with prediction lengths of [96, 192, 336, 720). but also with longer lengths such as 1000. It can be observed that there is clear improvement in performance across all five prediction lengths the look-back window length increases. This improvement is particularly notable for longer prediction lengths such as 336, 720, and 1000. It indicates that Peri-midFormer can effectively utilize more historical information, as longer look-back windows contain more stable periodic characteristics, which is precisely what Forrmer requires.
E.7 Time Series Decomposition Analysis
Peri-midFormer performs well on the Exchange dataset for long-term forecasting task and the Yearly dataset for short-term forecasting. Although these two datasets lack obvious periodicity, they exhibit strong trends, as shown in Figure (19). Peri-midFormer uses time series decomposition strategy, where the trend part first separated from the original data before dividing the periodic components. After the output of Feri-nidFormer, the predicted trend part is added back, as illustrated in Figure (8. The separated trend part easier to predict, especially when the trend is strong. This is why Peri-midFormer achieves strong performance on the Exchange and Yearly datasets.

---

Exchange
13-
334
14
Yearly
6.2
5000
3300
1103
3903
Figure 19: Visualization of Exchange and Yearly dataset.
Proof
To demonstrate the essence of attention computation among multi-level periodic components, need to analyze how the interactions between periodic components at different levels affect the final feature extraction. In time series analysis, different periodic components correspond to different time scales. This means that through decomposition, we can capture components of various frequencies within the time series. The essence of the periodic pyramid is to capture these different frequency components through its hierarchical structure.
Using single-channel data example, and given that adopt an independent channel strategy, this can be easily extended to all channels. Assume the time series sx()) can be decomposed into multiple periodic components (t):
xx()=
(t).
(14)
Taking two different periodic components as examples:
2xl T:
2ml T
(1(()) A, sin
+ th =t((1) = A, cos
+ Φラ
(15)
where amplitude, is period, and is phase. Due the overlap and inclusion relationships between different periodic components, we employ attention mechanism in the periodic pyramid to capture the similarities between different periodic components, focusing on important periodic features. When applying the attention mechanism, we have:
Q: (t), K₃ = (K) VG Wvv,(t):
(16)
where Wo. WK and Wy are learnable weight matrices. From Equations (15) and 16
2tt + +.). T;
2mt
Q: W Wo4 sin
K₃ = =j=Wx4m A, cos
(17)
Tj
Further, the dot-product attention can be expressed as:
Q.K3 A:A, (Wo sin
2ml + di WK cos
2ml
+ 05) ))
(18)
T:
Tj
Using the trigonometric identity sin(a) cos(b) = [ssin(a + b) sin(a can be further expressed as:
b)]. the dot-product Q:K3
1 Q.K. - A.A, We
2nt
2tt
2nt
2xt
25 )1] (Wx)
sin
Φ,
+ +,
sin
+ Φ;
T,
T,
T3
(19)
25

---

Based on this, considering the periodicity and symmetry of sin(a + b) and sin(a - b), when the periods of time series components are close same (intra-layer attention the pyramid, see the right side of Figure 3 in the original paper) or have overlapping / inclusive parts (inter-layer attention in the pyramid, see the right side of Figure
the original paper), the values of these two sine functions will be highly correlated, resulting in large Q value. This indicates that the periodic pyramid model can effectively capture similar periodic patterns across different time scales.
Next, incorporating this into the calculation of the attention score:
Q.Kg vdl
exp
Sij
(20)
Q:KT
Σ
exp
Vily
It can be seen that the attention scores between highly correlated periodic components will be higher, which we have already validated in Figures 13 and 14 of the original paper.
Further, the attention vector a; of (t(t) can be obtained as:
a, Σ Sty V,
(21)
where V, (1 Ay cos
2xt + 63)
From the above derivation, can be seen that the attention mechanism can measure the similarity between different periodic components. This similarity reflects the alignment between different periodic components in the time series, allowing the model to capture important periodic patterns. By capturing these periodic patterns, the periodic pyramid can extract key features of the time series, resulting in a comprehensive and accurate time series representation, This representation not only includes information across different time scales but also enhances the representation of important periodic patterns.
G Broader Impacts
Our research has implications for time-series-bais based analyses such as weather forecasting and anomaly detection for industrial maintenance, as discussed in the Introduction I social impact.
Our work carries no negative
H
Full Results
H.1
Full Results of Classification (Table 11)
H.2
Full Results of Short-term Forecasting (Table
H.3
Full Results of Long-term Forecasting (Table 13 and
H.4
Full Results of Imputation (Table 15)
H.S
Full Results of Anomaly Detection (Table 16)
26

---

Table 11:
Full results for the classification task. (* means former, T-LLM means Time-LLM, GPT means GPT4TS, T-Net means TimesNet, Patch means PatchTST, Light means LightTS, Station means the Non-stationary Transformer.) We report the classification accuracy (%) as the result. The standard deviation is within 1%. We reproduced the results of SS by hhttss::/githuit cow./tumllmmmmmmmccom httttttttmmmmmmmeeeeeeeeeeeeeeee Tiie--Serieee Library reproduced TSLANet by https:// /7 github. com http://maddeleeee2///SSS.. second best.
and copied the others from GPT4TS 20
Red: best, Blue:
Datasets/ Models
Rocket
Pinch
TCN
Trans
Re+
Pyza
Autoe
Station
FED+
ETS+
Hous
DLincer
Light
T-Nee
GPT
ILSANe: Peri-midd
33
19
g
48
23
10
[
1v
29
35
14
16
13
20
Former
EthanolConcentration
45.2 28.9
32.7 31.930.8
31.6
32.7
31.2
28.1
33.8
32.6
29.7
35.7
30.4
50.6
FaceDetection
64.7 67.852.8 67.3
68.665.7
68.4
68.0
66.0
66.3
67.6
68.0
67.5
68.669.2
66.7
68.7
Handwriting
58.8 23.2.3.3 32.0 27,429,4
36.7
31.6
28.0
32,5
33,8
27.0
26.1
32.1 32.7
57.88
31.5
Heartbeat
75.6 75.7 76.1 77.1 75.6 74.6
73.7
73.7
71.2
77.6
75.1
75.1
78.077.2
77.5
95.1
SopaneseVowelle
96.2 94.098.9 98.7 97.898.4 96.2
99.2
98.4
95.9
98.9 96.2
96.2 98.498.6
99.2
97.3
PEMS-SF
75.1 80.9 82.1 82.783.2 82.7
87.3
86.0
83.8
88.4 89.687.9
83.8
88.2
SedfRegulationSSCP
90.8 82.2 92.2 90.488.1 84.0
89.4
88.7 89.6
92.5 87.3
89.8 91889.2
91.8
92.1
SelfRegulationCCP2
53.3 53.655.6 58.9 56.753.3 50.6
57.2 54.4
55.0
56. 50.5
51.1 57.2.594
61.6
58.2
SpokcnArabicDiggits
71,2 98.095.6 4 97.099.6
100.0
100.0
100,0
100.0
98.8
81.4
100.099.0992
99.9
98.2
UWaveGestureL library
94.4 81.7 88.4 85.6 85.683.4
85,9
87.5
85.3 85.0
86.6
82,1
80.3 88,
91.2
85.6
Average Accuracy
72.5 68.7 70.3 71.9 71.5
70.8
71.1
72.7
70.7 71.0
73.0
67.5
70.4 73.6
76.0
76.5
Table 12: Full results for the short-term forecasting task in the M4 dataset. (* means former. T-LLM means Time-LLM, GPT means GPT4TS, T-Net means TimesNet, Patch means PatchTST, HiTS means N-HiTS, BEATS means N-BEATS, Light means LightTS, Station means the Non-stationary Transformer.) The standard deviation is within 0.5%. We copied the results of Time-LLM from Time-LLM [25, Pyraformer from TimesNet [13], and the remaining results from GPT4TS 20
Red:
best, Blue: second best.
Peri-mid T-LLM
GPT
TeNet
Patch
BEATS
Light
DLinear
FED+
Station
Autos
Pyra
Ree
Models
Former
25
20
Li
19
30
29
15
14
Lix
12
10
16
XX
SMAPE13.358 3.....333581..111.3311 13,
Yearly
MASE
3.021
3.005
3.015
2.996
3.019
3.045
3,043
4.487
3.109 4.283
3,048
3,078
3.134
3.711
3,418
3.800
OWA
0.789
0.789
0.793
0.786
0.792
0.793
0.794 1. 0.827
1.058 0.803
0.807
0.822
0.942
0.881 0.973
Quanerty
SMAPE10.05810.1100
110.0000.0000111111111 10010.00000.220000000.
21122222210111.11112 12413.3761
1.3.441..141111.... 11w../992210958113311111 0111154441111.1113.1 360033311
MASE
1.170
1.178
1.194
1.182 1,233 1.194 1.169 1.906
1.328
1.520
1.283 1.325 1,365 2,350 1.401 1.775
OWA
0.883 0.889 0.898 0.890 0.921 0.899 0.886 1.302 1.000 1.106 0.958
0.981 1.012 1.558 1.027 1.252
Monthly
2.71722227
12.617
4.8811
MASE
0.933
0.963
0.956 0.933 0.970 0.969 0.937
1.368 1.053 1.037 1.102 1.097 1.103 1.913 1.141 2.614
OWA
0.879
0.903 0.897 0.878 0.905 0.899 0.880 1. 1.149 0.981 0.956
1.012 0.998
1.002 1.511 1.024 1.927
Others
SMAPE
4.845
4.795
4.940
4.891 4.952 5.061 4.925
7.267 15.880
6.709 4.954 6.302 5,485
44.78624.46032.499
MASE
3.217
3.178 3.228 3.302 3.347 3.216 3.391 5.240 11.434 4.953
3.264
4.064 3.865 18.58 15.80220.6033.355
OWA
1.017
1.006
1.029
1.035 1.049 1.040 1.053 1.591 3.474
1.487
1.036 1.304 1.187 5,538 5,879 8.679
Weighted
Average
SMAPELL 833881383311111
01..11...911111 01..2711.85114.777
MASE
1,584
1.595
1.600
1.585
1.623
1.613
1.599 2,408
2.111 2.095
1.701 1.756 1.771 3.265
2.718 4.223
OWA
0.850
0.859
0.861
0.851
0.869
0.861
0.855
1.172 1.051 1.051 0.918 0.930 0.939 1.480 1,230 1.775
27

---

Table 13: Full results of 512 look-back window length in long-ten ten- forecasting task (since FITS defaults to a look-back window length of 720, its results at 720 length are attached at the end). The standard deviation is within 0.5%. We copied the results of GPT4TS from GPT4TS [20]. Time-LLM from TSLANet [27], reproduced TSLANet by https: / github. com/ emadeldeente TSSLNet, and reproduced the others by bhttps:/ //gi gi thub. com hoom/thml/// Seeiiie--Seiessee - Red: best, Blue: second best.
Look-back
512
720
Peri-mid Former
GPT4TS
TSLANet
Time-LLM
FITS 21
Dlinear [1]
PatchTST 19
TimesNet 133
Pyraformer 10
FITS (1
Methods
20
27
25
Metrics
MSE
MAE|
MSE
MAE|
MSE
MAE
MSE
MAE|
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.153 0.200
0.212
0.196 0.232
0.147 0.201
0.. 0.226
0.171
0.231
0.147 0.199 0.159 0.214
0.215 0.298
0.1170
0.225
Weather
192
0.203 0.251 0.200 0.248
0.241 0.269
0.189 0.235
0.216 0.262
0.213 0.270
0.196 0.246
0.220 0.269
0.252 0.335
0.213 0.260
336
0.255 0.294
0.255 0.286
0.297 0.308
0.262 0.279
0.261 0.295
0.259
0.311
0.243
0.283 0.276 0.307
0.282 0.352
0.258 0.295
720
0.321 0.338 0.326
0.337
0.371 0.356
0.304 0.316
0.326 0.342
0.322
0.362
0.317
0.334 0.342 0.354
0.373 0.412
0.321 0.340
Avg
0.233
0.271
0.237
0.270
0.276
0.291
0.225
0.257
0.244 0.281
0.241
0.294
0.226
0.266
0.249 0.286
0.281 0.349
0.241 0.280
96
00,367 0.396
0.397
0..744
0,402
0.362
0.392/00.37
0.396
0.367
0.397
0.374
0,402
0.442
0.457
0.702
0.628
00,379 0,402
BTTh1
192
0.404 0.420 0.416 0,418
0.416
0.431
0.398
0.418 0.405 0,415
0.402
0.420 0.415 0.433
0.492 0.491
0,927 0.743
0.414 0.423
336
0.421 0.432 0.442 0.433
0.404
0.431
0.430
0.427 0.424 0.429 0.431 0.442 0.458 0.458 0.481 0.486
0,973
0.785
0.434 0.440
720
0.445
0.470 0.477 0,456
0.495
0.495
0.442
0.457 0.425 0.448
0.472 0.494 0.473 0.484 0.551 0,525
1.048
0.834
0.431 0.454
Avg
0.409
0.430
0,427
0.426
0.422
0.440
0.408
0.423
0.407
0.422
0,418
0.438 0.430
0.444
0.492 0.490
0.913
0.748
0.415 0.430
96
0..268
0.337
0.2855
0.342
0.270
0.339
0.268
0.328
0.271
0.337
0.303
0.368 0.298
0.349
0.382 0.420
1.497
0,934
00.271 0.337
ET
192
0.321 0.376
0.354
0.389
0.317
0.374
0.329 0.375
0.330 0.374
0.393
0,425 0.371 0.399 0.407 0.434
4.498
1.699
0.331 0.375
336
0.331 0.394
0,373
0,407
0,320
0.382
0.368
0.409
0.353 0.395
0.510
0,497 0,418 0.433 0,392 0,438
4.385
1.743
0.354 0,396
720
0.348
0.402
0.406
0,441
0.405
0.443
0.372
0.420
0.378 0.421
0,808
0.637 0.465 0.474 0.451 0.469
4.579
1.838
0.377 0.423
Avg
0.317
0.377
0.354
0.394
0.328
0.385
0.334
0.383
0.333
0.382
0.504
0.482
0.388
0.414 0.408
0.440
3.740
1.554
0.333 0.383
96
00.291
0.349
O.
292
0.346
0.297
0.354
0.272
0.334
303
0.349
0.304
0.347
0.302
0.353
0.331
0.375
0.567
0.502
10.309 0.352
ETTm2 ETTml
192
0.334 0.374 0.332 0.372
0.329
0.373
0.310
0.358
0.338 0.367 0.335
0.366 0.335 0.373 0.394
0.409
0.626
0.546
0.338 0.368
336
0,367 0.398
0,366 0.394
0.353
0.390
0.352
0.384
0.368
0.384
0.366
0.385 0.372 0.399 0.414
0.431
0.752
0,641
0.366 0,385
720
0.425
0.418 0.417 0.421
0.411
0.416
0.383
0.411
0.421
0.413 0.420
0.417 0.442 0.437 0.454 0.457
0.952
0.745
0.415 0.412
Avg
0.354
0.385
0.352
0.383
0.348
0.383
0.329
0.372
0.358
0.378
0.356
0.379 0.363 0.391
0.398
0.418
0.724
0.609
0.357 0.379
96
0.1664 10.13
0.262
0.
0.263
0.161
0.253
065
0.254
0.169
0.265
0.1 0.260
091
0.277
0.369
0.464
10.163 0.253
192
0.228 0.301 0,229
0.301
0.231
0.306
0.219 0.293
0.219 0.291
0.226
0.306 0.243 0.308 0.251
0.321
0.571
0.573
0.222 0.298
336
0.279 0.335
0,286 0.341
0.286
0.339
0.271 0.329
0.272
0.326
0.298
0.362 0.297 0.353 0.319 0.364
1.193 0,813
0.268 0.326
720
0,362 0.388
0.378
0.401
0.364
0.391
0.353
0.379
0.359
0.381
0,408
0.434 0.381 0.395 0.388
0,408
3.289
1.339
0.349 0.378
Avg
0.258 0.319
0.266
0.326
0.263
0.325
0.251
0.313
0.254
0.313
0.275
0.342 0.273 0.329
0.287
0.343
1.356
0.797
0.251
0.314
Electricity
96
126 0.223
1.139
0.238
0.131
0.226
0.131 0.224
140
0.238
0.141
0.241
0.131
0.228
081 0.286
0.281
0.375
0.1137 0.235
192
0.144 0.240
0.153 0.251
0.145
0.240
0.1
0.241
0.153
0.250
0.154
0.254 0.149 0.246 0.191 0.294
0.295 0.389
0.151 0.248
336
0.157 0.255
169 0.266
0.162 0.139
06060
0.248
169 0.266
0.169
0.271 0.166
0.262
O.
203 0.304
0.312 0,405
0.167 0.264
720
0.181 0.279
0.206
0.297
0.199
0.290
192
0.298
0.208
0.298
0.204
0.304
0.203 0.293 0.294 0.371 0.309 0.396
0.206 0.296
Avg
0.152 0.249
0.167
0.263
0.159
0.224
0,8
0.252
0.
0.263
0.167
0.268
0.162 0.257
0.217 0.314
0.299 0.391
0.165 0.261
96
00.61
0.255
0.388
0.282
0.371
0.258
0.362
0.248
0.398
0.277
0.411
0.294
0.364
0.256
0.601
0.320
O. 708
0.406
0.338 0.272
Traffic
192
0.380 0.263
0.407 0.290
0.388 0.268
0.374
0.247
0.409 0.280
0.421
0.298 0.382 0.263 0.609 0.328 0.699 0.401
0.398 0.274
336
0.393 0.269
0.412 0.294
0.397 0.272
0.385
0.271
0.418 0.285
0.431
0.304
0.392 0.270 0.619 0.330 0.699 0.398
0,411 0.281
720
0.434
0.293 0.450 0.312
0.431 0.288
0.430
0.288
0.456
0.306
0.467
0.324
0.430 0.290 0.657 0.348 0.714
0.398
0.449 0.301
Avg
0.392 0.270
0.414
0.294
0.397
0.272
0.388
0.264
0.420
0.287
0.433
0.305
0.392 0.270 0.622 0.332
0.705
0.401
0.411 0.282
Exchange
96
0.081
0.197
0.089
0.207
0.082
0.201
0.088
0.208
100 0.225
0.110
0.252
001
0.224
0.206 0.342
0.654
0.659
00095 0.221
192
0.171 0.292
0.1 0.305
0.171 0.293
0.182 0.303 0.201 0.326
0.238
0.373
0.215 0.335 0.391 0.472
0.741 0.691 0.200 0.323
336
0.311 0.402
0.335 0.421
0.319
0.407
0.334
0.419
0.350 0.437
0.466
0.525
0.296 0.399 0.606 0.595
1.042 0.835 0.369 0.444
720
0.822
0.682
0.884
0.705
0.888
O. 739
0.879
0.704
0.920 0.728
1.166
0.821 1.058 0.775 1.601 0.961
2.190 1.189
0.994 0.750
Avg
0.346
0.393
0.373
0.410
0.365
0.410
0.371 0.409
0.393 0.429
0.495
0.493
0.418 0.433 0.701 0.593
1.157
0.844
0.415 0.435
Average
00.308
0.337
0.324
0.346
0.320
0.341
0.308
0.334
0.322
0.344
0.361
0.375
0.331
0.350
0.422
0.402
1.147
0.711
|0.33 0.345
28

---

Table 14: Full results of 96 look-back window length in long-term forecasting task. means former, Station means the Non-stationary Transformer.) The standard deviation is within 0.5%. We copied the results of iTransformer from TSLANet (27.. Pyraformer from cOm/VEWOXIC/FITS and copied the others from GPT4TS 20
TimesNet [3], reproduced FITS by htttp::/
Red: best,
Blue: second best.
Look-back)
96
Methods
Peri-mid iTrrans*
FITS
Dlincar
Timeset
Pyras 10
FEDx
Autos
Station
ETS+
LightTS 15
In
Res 28
Former
26
21
14]
19
13
[6]
Metries
MSE
MAE
MAT
MAE
MAE
MAE
MSE
MAE
MSE
MAE
MSE
MSE MAE
MSE
MAE
MSE MAE
MSE MAE
MAE
b.155
0.204
0.197
0.237
0.176
0.149
0.198
0.172 0.220
10.222
0.556 0.217 0.2% 0.336 0.133.22)
0.197 0.281 0.1820.242 0.300
0.689 0.596
Weather
0.203 0.244
0.221 0.254
0.241
0.220..882
0.194 0.241
0.219 0.261
0.739
0.624 0.27% 0.336
0.367
0.245
0.237
0.312 0.227
0.598 0.544
0.752 0.638
336
0.262
0.289
0.278 0.29 0.2.6
0.29: 0.308
0.365
0.282
0,280
0.306
1.004 0.753 0.333 0.339 0.380 0.39 0.39 0.521
0.298 0.353 0.252.0
0.639 0.596
720
0.345
0.344
0.35%
0.354
0.314 0.334
00365
0.359
1.420 0,934 0.403 0.419 0.24
0.410
0.352 0.288 2.352.0..5
0.759
1,130 0.792
Avg
0.241
0 262
0.258 0. 0.2.772700 0.0
0.300.
0.248
0.264
0.2559
0.287
(0.946 00330 0.360 0.00000..88..00028888...33
0.271
0.334
0.64
0,803 0.656
96
10.373
0.336 0 0.336
0.396
0.40550.3355 0.3933
0.379
0.399
10.384
0.402
0.66640.6122
0.494 0.47999.4240-4332
0.713
0.837 0.728
ET El'Tl T
0.425 0.429
0.441
0.436 0.422
0.4050.4166..411
0.400
0.421
0.446
0.429
0.69900.68110 00.40 0.448
0.538 0.504
0.008
0,923 0.766
336
0.464 0.445
0.475 0.444
0.439
0.422
0.436
0.449
0.469 00.891
054 0,465 0.521
0.574 0.521 0.5180.488
1.107
1.097 0.835
720
0.479
0.467 0.503
0.463 0.440
0,466
0.5521
0.500
(0.66
0.782 0.506
0.507 0.514 0.5124444...22.666 0.643
0.562 0.535 0.547.......33
1.181 0.865
1,257 0.889
0.435 0.434
0.454
0:448 0.4400000040044022202. 0.420:0002202922200022
0.443
0.437
0.430
(0.458
0.450
(0:827 0.703 0.440 0. 0.6600 0.447 0.57000.37
0.542 0.510
1.4400.795
1.029 0.805
96
10.287
0.337
0.297
(1..4999.2222..339 000 0.33
0.274
0.336
10.340
0.374
10.645
0.597 0,358 0.39 0.397 0.3364600.38888 0.476 0.488
0.
0.340 0.391 00.37/...777.333.
3.755
1.525
2.626 1.317
E1 F1Th2
192
0.368
0.387
0,380 0 0.400 0.
0,383
0.418
0.339
0.379
0.402
0.414
0.788
0.683 0,429
0.5120.499
0.430
0.520..00055000
5.602
1.931
11.1202..97
336
0.414 0.424
0.428 0.44
0.448
0.329
0.380
0.452
0.452
(0.907
0.747 0.496 0.487
0.4444877.44220 0.552
0.485
0.479
5.526
5.559
1.833
3.233 2.769
0.397
0.425
0.427
0.445 0.417 0.436
0.605
0.551
0.379
0.422 0.462
0.468
(0.963
0.783 0,463 0.44
0.556
0.497
0.63
1.625 3.874 1.697
0.367
0.393
0.383
0 0.40 0.307 0.3366.338 0.398
0.431
0.446
0.379
00.415
0.427
(o.826 0.703 000.::/4...0..0.0000000000000.00
0:439
0.452 0.452..66220.544444...
7,000
1.729 6.736
96
10.300
0,368
0.334 0.
0.349
0.342
10.338 0,375
0,5543 (1w.)9999..0009000000...0000 0...00330...733.....0..
0.375
3,0.0000..0044000...7777777 0.571
0.533 0.528
m
192
0.371
0.388
0.377 0.391
0.392..333
0.335
0.3322
0.332
0.369
0.374 0.387
0..56 0..426 0.44)
0.4. 0.445.999.. 0.444
0.553
0.408 0.4100.4000.407
0.669
0.658 0.592
336
0.402
0.409
0.426 0. 0,420 0,415
0.414
0.386
0.392
0.410
0.411
0..754
0.655 0.445
0.459
0.537 0.4455
0.435
0.438
0.217
0.898 0.721
0.466
0.445
0.491
0.459 0,486
0.425
0.421
0.416
0.420
0.478
0.450
10.908
0.724
0.490 0.561
0.585
0.499
0.462 0.522
0.866
0.841
Avg
0.392 0.400
0.407
0.41000.114444000
0.357
0.351
0.380
0..40
0.406
(0.691
0.607 0.448 0.15500000088881117148800000
0:429 0.425 0.435
0.961
0.734
0.799 0.671
96
00.171
0,180 0. 0.264 0.254|0.1800 0.26660.. 167
0.26990.165
0.255
10.187
0.267
(0:435
0,507 0.203 0.287 0tw.:/550..3390011922..03
0.189 0.280
0.453
0.658
0.619
ETTm2
192
0.240 0.299
0.255 0. 0.309 22..0990.244 0.304503055000
0.224 0.3033
0.220 0.292
0.249
0.309
0.730
0.269 0.228 0.340
10:253 0.399 0.382
0.33
0.533
1.078 0.827
336
0.306
0.341
0.311
0.348 0.342
0.348 0.307
0.288
0.274 0.329
0.274
0.321
0.351
1.201 0.845 0.325
0.366
0.57
0.372
0.334 0.361
0.314
0.357 0.442 0.466
0.887
1.549 0.972
720
0.404
0.397
0.412
0.407 0.407 0.397
0.397
0.421 0.362 0.385
0.448
0.403
3.625 1.451 0.421 0.415
0.414
0.413 0.675
3,379 1.338 2.631 1.242
Aug
0:280 0.323
0.288
0.332 0.2.662000662.33 0.328
267
0.333
0.255
0.315
0.2991 0.333
1 1.498 0.869 0.869 0.305 0.349
0.371
( (0.366
0.293
0.342 0.342 0.4090.366
(,4,,0000.000444444400400 1.4100.800
1.479 0.915
Electricity
96
1.2
0.236
148 0.
0.237.23.
0.222
16.166
0.272
(0.3860.449)
0.30s/0.201
0.187
0,312 0.402
192
1.1
0.251
0,162
0.253
0.197 0.276
0.13
0,249,...1.15
0.157
0.240
0.184
0.289
00.778 0,443 0.201
0.222
0.33240..334
0.334
0.1220.286
0.199 0.13 0.386
0,348 0.430
336
0.176 0.270
0.178 0. 269 0.233
o. 0.00.267...00
0.163 0.259
0.199
0.300
0..43
0.443 0.214
0.330 0.333
0.329
0.200
0.233.000000
0.212 0.329
0.300 0.394
0.350 0.433
720
0.204
0.307 0.225
0.317 0.3170.255 0.325
0.203
0.301 0.197 0.290
0.220
0.320
0.445
(0.376 0.445 0,246 0.355 0.35 0.361
0.2220.331
0.233 0.345
0.360 0.439
0.340 0.42
Ave
0.170
0. 0.266
178 0. 270 2.200.0.000029222222 292
1660.263
0.161 0.252
0.199
0.295
(0.379
0.445 0.3............3.2222.2.....
www.22222.333300 0.330.2.999
0.208
0.229
0.397
0.338 0.422
96
00.45 0. 0,300
0.395 0. 222.666666222222 0. w1wp::442422..0.1110000000000000 0.200 3,360
0.249
|0,593
0.321
(o.867 0.468 0.587 0.366 0.338 0.6120..036
0.607
0.392 0.391 0.319
0.732 0.423
Traffic
192
0.459 0.301
0..17 0.276 0.362.....223333
0.3377.799
0.256
0.661
0.336
(0.869
0.467 0.373 0.1.....6.....6.0 0.383
0.6130.340 0..7,00..666.3332266..3..
0.621 0.399 0.382 0.696
0.733 0.420
336
0.477
0.310
0.433 0.283 0.603 0.365 0.4366.
0.26-1
0.629
0.336
00.881 0.469 0.621 0.383 0.323 0.337
0.618
0.622 0.133.0.66....6666..66
0.420
0.742 0.420
720
0.510
0.329
0.467
0. 302 0.641
0.641
0. 0,384
0.432
0.286
0.640
0.350
0.8996 0.6646333.666640 0.38 0.600 0.40 0.653
0.632
0.396 407 0.864
0.472
0.755 0.423
Avg
0.475 0.310
0.4.8
0.428
0. 282 0.621
0.375 0.375 0.293
0.390
0.263
0.620
0.336
(0.878
0.469 0.176 0.709 0.340
0.621 0.396 0.764
0.741 0.422
96
l0.022 0.198
0,086 0 0.206 00006/0007700/0 (...
0.21880..s
0.205
10.107
0.234
1.7448
1.105 0.148 0.27% 0.330001...3..33....33
0.085
0.28
1.065 0.829
Exchange
192
0.172 0.295
0.7 0. 0.
0. 302
0.
1760.315
0.
0.298
0.226
0.344
1.874
1,151 002771 0.380
0. 369
0.2190.333
182
0.303
1.204
1.188 0.906
336
0.319 0.407
0.331 0.417 0..4170..325... 0.415
0.313
0,3302
0,302
0.398
0.367
0.448
1.943
1.172 0.446 0.509 0.524
0.421
0.348
0.377
1.672 1.036
1.357 0.976
720
0.846
0.6 0.692
0.847
0 0.691 0.843 0. 694
0.834
0.695
0.871
0.703
0.964
0.746
2.085
1.206 1.195 0.841
1.447 0.941
1.025
0.734
0.499
2.478
1.310
1,510 1.016
Avg
0.355 0.398
0.360 0 403
( 0.358 0. 404
0.354
0.4144..55...
0.401
0.441
0.443
1.913 1.159 1.159/0..550 0.5 0.50000000000000 015.990.......000000000011110...
0.
0.46
0.410
0.355.034404447
1,550 0.998
1.280 0.932
Average
|0.339 0 1.349/00
0.345 0. 353 0.373
0.333
366
0.3350.358
0.335
J0.381
0.372
|0.995 0.709 0.11000000...10000000.000
0.402 0.401
1.650 0.877
29

---

Table 15: Full results for the imputation task. We randomly mask 12.5% 25%, 37.5% and 50% time points to compare the model performance under different missing degrees. (* means former. Station means the Non-stationary Transformer.) The standard deviation is within 0.5%. We reproduced the results of Pyraformer by https: //g: thub. httaaa..oo///ummmimeeeeeeeee -eee--iieeeeeeeeeeeee Library and copied the others from GPT4TS (0).
Red: best, Blue: second best.
Models
Peri-mid
GPT4TS
Timesnet
PatcTTST
ETS+
LightTS
DL DLinear
FED
Station
Auto
Pyra-
In
Re+
Former (ours)
17
Mask Ratio
MSE
MAE
MSE MAB
MSE
MAE
MSB
MAE
MSE MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
12.5%
0.030
0.108
0.092 0.041 130l0..0 0.1855..075
0.180 0.058 0. 162 0.035
1.200.035
0.135
107 0.034 0.
0.670 0.541 0.047
155 0.032 0.126
25%
0.031
0.112
0.022
0.033 001 0.044 0. 0.09 0.229
0.096
0.093 0.206 0.080 193 0.052 0.166 0.032 0. 0.049 0.144
0.689 0.553 0.063 o. 1808 0.042 0.146
37.5%
0.034
0.117
0.029
0.111 0.029 0.111 0.049 0.143 0.,13
0.271
0.113 0.231 0.103
0.2119000000 0.19] 0.039 0.131 (0.07 0.161 0.737 0.581 0.079 0.200
0.063 0.182
50%
0.041
0.125
0.044 0.036 0.124 0.055 0.151 0.323 0.323 0.134
0.132 0.248 0.211 0.14 0.1.67.........................0 0.770 0.2000000000000.0 0.218 0.082
Avg
J0.034
0.116
0.0028
2.105/..0000.027 0.107 0.047
12.5%
J0.022
0.081
0.017 0.076/0/7/.0018 0.080 0.026 000.0000000020000000000000000000 0.127 0.062 0. 0.1666 0.1.6/600000.1559..0.0000. 0.009/0.330.00333....33 0.092 0..0
25%
0.024
0.084
0.020 0.000 0.085 0.028 0.09 0.29: 0.143 0.085
00gg00000....11000000000000000.0 0.421 0.4 0.272 0.1336
37.5%
0.026
0.022 0.087 0.091 0.3000..00002237.3335
0.159 0.106 0.222 01110000002000000101010011011010
0:027
00.775
50%
0.029
0.095
0.095 0.0260000090 015.:0300000..00.000000000000020
0.329
Avg
J0.025
0.087
0.021
12.5%
10.060
0.164
0.0430.1400..0577 0.15 0.20
0.069
0.0540.156..069
0.169 0.304 0.217
0.21
37.5%
0.092
0.200
0.0720.100
0.0.:.......000220222000002002.0 0.215 0.318
50%
0.112
0.220
0.21102...2111 0.141
Avg
10.083
0.191
12.5%
0..49
0.140
0.009
0...9...250..000 1.0.0../250.0000...00 0.057 0.
T12
25%
0.051
0.141
0.044 0.135 0.144 0.061 0.158 0.155 0.127 0.247 0.24770.7..7..77..........
37.5%
0.055
0.147
0.051 0.147 0.052
0..51.....000000000 0.151 0.1.0.0.......0..000000000......
0.
0.158 0.276 012.:00..............00000005.5.
1.107 0350
50%
0.062
0.156
0. 0.0.9./........02220000000001223
Avg
10.054
0.146
|0.048 0.141 0.009 0.
Electricity
12.5%
(0.044
0.135
0.0800
0.194..1944..00
0.202 0..0220..550...1 0.000000..00000050000000000020.0
http://...............22000.22.0 0.383/0.218
25%
0.052
0.149
0.087 0.203
01089 0.206 0.065 0..000..0.00.1550020000......... 0.
0.11 0112332222...220222211122222277
0.120 0.2.7........
0.92000 0.312
00.19
37.5%
0.063
0.166
0.211
0,141 0.273
0.144 0.276
0.136
0..18.0.2..00022..02220000000000 0.22200000..44
0.266
0.102
0.2.9.
(.2222 0.328
00.003 0.315
50%
0.079
0.189
0.101
0.220 0.220 0.10000...00002111110011 0.222 0.0000000000000022033300300....0 0.293
0.175 0.305
0.158 0.284 0.108 0.228
0..13 0.239
0.283 0.331
0..210
|0.060
0.160
0.207/
0.26600..300000 0..6200000000000000.
0..132
0.225700297 0.382/0.222 0.328
/0.200 0.313
Weather
12.5%
l0.025
0.037
1..026 0.049 0.045 0.029 0.049/0.057 0.141
0, |0.039
0.084
|0.041
0.107
0.055 (o.026
0.047/0 10.477/1.10000 0.200....000033333.
0.093
(0.03
(0.031
0.076
25%
0.026
0.037
0.025 0.052 0.052 0.031 0.063 0.155
0.111 0.048 0.103 0.064
0.029
0.056 0.054 0.147
0.229 0.1.0442..1000000 0.005
37.5%
0.028
0.040
0.033 0.06
0.057 0.035 0.058 0.058 0.057 0. 0.107 0.033 0...2 0.156 0.240 0.110.00....0...00..00.00.
0.040
50%
0.031
0.042
0.037 0.060......000 0.0.0.05000034 000.0400.0220000330.000000..0000
0.023
0.133 0.066 0.134
0.24900.5..0
Avg
10.028
0.039
Average
|0.047
0.123
0.048 0.128 0.000 132 0.053 1.1//0..144 164 0.309/0. 123 0.229 1 0.225 10112 0.23 (0.057 a. 0. 1..1433.00222..11 0.15 (0.992 0.528/0. 165 0.2.7...74..33. 0.274..3...7433333...24
30

---

Table 16: Full results for the anomaly detection task. The P. R and FI represent the precision, recall and Fl-score (%) respectively. Fl-score the harmonic mean of precision and recall. A higher value of P. R and F1 indicates a better performance. (Station means the Non-stationary Transformer.) The standard deviation is within %. We copied the results from GPT4TS 20] Red: best, Blue: second best.
Datasets
SMD
MSL
SMAP
SWaT
PSM
Avg
Metries
R
F1
P
R
F1
R
FI
P
R
F1
R
F1
(%)
Transformer
76.13
71.57
68
89.37
57.12
68.8
96.53
62.75
76.07
76.88
LogTTran
83.46 70.13
76.21
73.05
87.37
79.57
89.15
57.59
69.97
68.67
97.32
80.52
63.06
98.00
76.74
76.60
Reformer
28
82.58
69.24 75.32
85.51
83.31
84.40
90,91
57. 44 70.40
72.50
96.53 82.80
59.93
95.38
73.61
77.31
Informer
16
86.60 77.23 81.65
81.77
86.48
84.06
90.11
57.13 69.92
70.29
96.75
81,43
64.27
96.33 77.10
78.83
Anomaly
88.91 82.23 85.49
79.61
83.31
91.85
58.11 71.18 72.51
97.32 83.10
68.35
72 79.40
80.50
Pynaformer
80.61
92.54
71.09
96.00
82.08
Autoformer
88.06 82.35 85.11
77.27
80.92 79.05
90.40
58.6 62. 71.12 89.85
95.81 92,74
99.08 88. 15 93.29
84.26
Station
88.33 81.21 84.62
68.55
89.14
77.50
89:37
59.02 71.09 68.03
96.75 79 88
97.82 96.7 97.29
82.08
DLincar
83.62 71.52 77.10
84.34
85.42 84.88
92.32
55.41 69.26
80.91
95.30 87.52
98.28
89.26 93.55
82.46
LighrTS
87.10 78.42 82.53
82.40
75.78 78.95
92.58
55.27 69.21 91.98 94.72
98.37
97.15
84.23
ETSformer
87.44 79.23 83.13
85.13
84.93
85.03
92.25
55.75 69.50
90.02
80.36 84.91
99.31
85.28 91.76
82.87
FEDformer
87.95 82.39 85 08
77.14
80.07 78.57
90.47
58.10 70.76
90.17
96.42 93.1
97.31
97.16 97.23
84.97
PatchTST
hp
87.26 82.14 84.62
88.34
70.96
78.70
90.64
55.46
68.82
91.1 80.94 85.72
98.84
93.47 96.08
82,79
TimesNet
87.91 81.54 84.61
89.54 75.36
81.84 81.84
90.14
56.4
69.39
90.75
95.4
98.51
20
97.34
85.24
GPT4TS
84.98 86.89
82,00
96.34
97.13
86.72
Peri-midFormer (ours)
87.30 84.65 85.95
89.66
75.31
81.83
90.40
56.10
68.62
91.91
94.95 93.40
98.4
96.01
97.19
85.40
31

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable "[No] ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made our main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
32

---

It is fine include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We illustrate the limitations of the model in conjunction with the hyperparametric analysis in the Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-speciffcation, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-ted -etteettt-txtttte system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best judgment and recognize
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification: We provide a full theoretical description of the proposed methodology in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
33

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimen- tal results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the proposed methodology in detail in Section S and provide further explanation in Appendix []] Experimental details are outlined in Appendix D Additionally, all the
code related the proposed method included in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (c.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is available at http://///ggthooooo//////////t/a
34

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps::/niiss.cc/plll for more details.
While we encourage the release of code and data, we understand that this might not be possible,
"No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to repro- duce the results. See the NeurlPS code and data submission guidelines for more details.
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We offer detailed description of the dataset in Appendix and outline the experimen- tal setup in Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should presented in the core the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Appendix D demonstrate that the experiments repeated multiple times to eliminate randomness. However, due to the large number of experiments, detailed standard deviations not shown. Instead, we uniformly present them in the headings of each table in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the computer re- sources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the compute workers required for the experiments in Appendix D and the amount of computation in Section 6 and Appendix E.4
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimen- tal runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: We have read the NeurlPS Code of Ethics and conducted it in the paper conform in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
36

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We illustrate broader impacts of our research in Appendix G
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address socictal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strate- gies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
37

---

Justification:
We cite all the datasets and models involved in the paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi paperswithcode. trrocheode... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
The code is available at httt::///gtaab...oo////aaaaazau/ ormer
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
38

---

15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
39


Reviews:
[{"review_id": "fDQ5hBvom8", "reviewer": "Reviewer_W65V", "summary": "The abstract succinctly introduces the Peri-midFormer, a novel approach designed for time series analysis, acknowledging the challenges posed by the discrete nature of time series data and the complexity of capturing periodic variations directly. It proposes a method to address these challenges by decomposing complex periodic variations into hierarchical periodic components, termed the periodic pyramid. This approach leverages inclusion and overlap relationships among these components, mimicking the natural pyramid structure observed in time series data.", "questions": "see weakness", "limitations": "see weakness", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Innovative Approach: The concept of a periodic pyramid to model time series data is innovative and promises to address the limitations of traditional methods that struggle with capturing complex periodic patterns.\n\nHierarchical Representation: By representing time series as a pyramid with progressively shorter periodic components, the model potentially enhances the understanding of multi-scale temporal relationships.\n\nSelf-Attention Mechanism: Incorporating self-attention into the periodic pyramid allows capturing intricate relationships among periodic components, which is crucial for tasks like anomaly detection and forecasting.", "weaknesses": "Complexity and Scalability: The introduction of a hierarchical pyramid structure combined with self-attention could potentially introduce computational complexities and scalability issues, especially with larger datasets or real-time applications. Addressing these concerns in the paper would strengthen its practical utility.\n\nState-of-the-Art Comparison: It is not explicitly stated whether Peri-midFormer achieves state-of-the-art (SOTA) performance when compared to strong baseline models. Without clear comparative results, it is difficult to ascertain if the proposed method truly represents an advancement over current leading approaches in time series analysis.\n\nInterpretability: The abstract lacks interpretability regarding why the periodic pyramid structure exists in the applications and why it plays a critical role in improving forecasting. Providing a rationale or theoretical justification for the efficacy of the pyramid structure in capturing temporal patterns would enhance the understanding and acceptance of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer W65V\nThank you for your detailed review and questions. Please find our answers below.\n\n### **Q1: Complexity and Scalability**\n\nWe have adopted the suggestions from you and two other reviewers to include additional experiments, covering the complexity, time, and memory efficiency of both training and inference, to further validate the method's complexity and scalability. Please refer to the first part of the global reply section for details.\n\n### **Q2: State-of-the-Art Comparison**\n\nIn the original paper, we stated that our method achieved SOTA results (see line 77). Additionally, we compared the performance of various methods in the explanation of each experiment's results, highlighting the outstanding performance of our method. However, this may still be insufficient and could require more thorough discussion of the comparative results. If possible, we will provide a more comprehensive analysis of the comparative results in the revised version.\n\n### **Q3: Interpretability**\n\nThe abstract indeed lacks an explanation of the periodic pyramid structure in time series, which affects the understanding of the method. We will include this content in the revised version. \n\nAdditionally, the lack of a clear explanation of the fundamental principles of the periodic pyramid makes it seem unsupported. Therefore, we have re-examined our method and analyzed its fundamental principles as follows:\n\nTo demonstrate the essence of attention computation among multi-level periodic components, we need to analyze how the interactions between periodic components at different levels affect the final feature extraction.\n\nIn time series analysis, different periodic components correspond to different time scales. This means that through decomposition, we can capture components of various frequencies within the time series. The essence of the periodic pyramid is to capture these different frequency components through its hierarchical structure.\n\nUsing single-channel data as an example, and given that we adopt an independent channel strategy, this can be easily extended to all channels. Assume the time series $x(t)$ can be decomposed into multiple periodic components ${x_n}(t)$ :\n\n$$x(t) = \\sum\\limits_{n = 1}^N {{x_n}} (t) \\tag{1}$$\n\nTaking two different periodic components as examples:\n\n$${x_i}(t) = {A_i}\\sin \\left( {\\frac{{2\\pi t}}{{{T_i}}} + {\\phi _i}} \\right),{x_j}(t) = {A_j}\\cos \\left( {\\frac{{2\\pi t}}{{{T_j}}} + {\\phi _j}} \\right) \\tag{2}$$\n\nwhere $A$ is amplitude, $T$ is period, and $\\phi $ is phase. Due to the overlap and inclusion relationships between different periodic components, we employ an attention mechanism in the periodic pyramid to capture the similarities between different periodic components, focusing on important periodic features. When applying the attention mechanism, we have:\n\n$${Q_i} = {W_Q}{x_i}(t),\\quad {K_j} = {W_K}{x_j}(t),\\quad {V_j} = {W_V}{x_j}(t) \\tag{3}$$\n\nwhere ${W_Q}$ 、${W_K}$ and ${W_V}$ are learnable weight matrices. From equations (2) and (3):\n\n$${Q_i} = {W_Q}{A_i}\\sin \\left( {\\frac{{2\\pi t}}{{{T_i}}} + {\\phi _i}} \\right),\\quad {K_j} = {W_K}{A_j}\\cos \\left( {\\frac{{2\\pi t}}{{{T_j}}} + {\\phi _j}} \\right) \\tag{4}$$\n\nFurther, the dot-product attention can be expressed as:\n\n$${Q_i}K_j^T = {A_i}{A_j}\\left( {{W_Q}\\sin \\left( {\\frac{{2\\pi t}}{{{T_i}}} + {\\phi _i}} \\right)} \\right){\\left( {{W_K}\\cos \\left( {\\frac{{2\\pi t}}{{{T_j}}} + {\\phi _j}} \\right)} \\right)^T} \\tag{5}$$\n\nUsing the trigonometric identity $\\sin (a)\\cos (b) = \\frac{1}{2}[\\sin (a + b) + \\sin (a - b)]$, the dot-product ${Q_i}K_j^T$ can be further expressed as:\n\n$${Q_i}K_j^T = \\frac{1}{2}{A _i}{A _j}\\left\\\\{ {{W _Q}\\left[ {\\sin \\left( {\\frac{{2\\pi t}}{{{T _i}}} + {\\phi  _i} + \\frac{{2\\pi t}}{{{T _j}}} + {\\phi _j}} \\right) + \\sin \\left( {\\frac{{2\\pi t}}{{{T _i}}} + {\\phi _i} - \\frac{{2\\pi t}}{{{T_j}}} - {\\phi _j}} \\right)} \\right]} \\right\\\\}{\\left( {{W_K}} \\right)^T} \\tag{6}$$\n\nBased on this, considering the periodicity and symmetry of $\\sin (a + b)$ and $\\sin (a - b)$, when the periods of two time series components are close / same (**intra-level attention in the pyramid**, see the right side of Figure 3 in the original paper) or have overlapping / inclusive parts (**inter-level attention in the pyramid**, see the right side of Figure 3 in the original paper), the values of these two sine functions will be highly correlated, resulting in a large ${Q_i}K_j^T$ value. This indicates that the periodic pyramid model can effectively capture similar periodic patterns across different time scales.\n\nNext, incorporating this into the calculation of the attention score:\n\n$${s _{ij}} = \\frac{{\\exp \\left( {\\frac{{{Q _i}K _j^T}}{{\\sqrt {{d _k}} }}} \\right)}}{{\\sum\\limits _{m} {\\exp } \\left( {\\frac{{{Q _i}K _{m}^T}}{{\\sqrt {{d _k}} }}} \\right)}} \\tag{9}$$\nwhere $m$ denotes the index of all key values, including $j$. It can be seen that the attention scores between highly correlated periodic components will be higher, which we have already validated in Figures 13 and 14 of the original paper. \n\nFrom the above derivation, it can be seen that the attention mechanism can measure the similarity between different periodic components. This similarity reflects the alignment between different periodic components in the time series, allowing the model to capture important periodic patterns. By capturing these periodic patterns, the periodic pyramid can extract key features of the time series, resulting in a comprehensive and accurate time series representation. This representation not only includes information across different time scales but also enhances the representation of important periodic patterns.\n\nDue to character limitations, there are parts of the derivation will be explained inside the official comment.\n\nThe above proof partially explains the effectiveness of the periodic pyramid feature extraction method.\nWe hope this addresses your concerns. If you have any further questions, please feel free to ask.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6xiDMc6CD", "reviewer": "Reviewer_GBMh", "summary": "The paper introduces Peri-midFormer, a novel transformer-based architecture designed for time series analysis. \nBy leveraging the multi-periodicity inherent in time series data, the model constructs a Periodic Pyramid structure that decouples complex periodic variations into inclusion and overlap relationships among different periodic components. \nThe proposed method incorporates self-attention mechanisms to capture dependencies between these periodic components, achieving state-of-the-art performance across five mainstream time series tasks: short- and long-term forecasting, imputation, classification, and anomaly detection.", "questions": "Q1 (cr. W1). Add in the main body of the paper a discussion about the training and inference time, as well as some critical results and main conclusions.\n\nQ2 (cr. W2). Add in the main body of the paper a discussion about the significance of the improvements or about the time-accuracy tradeoff for Peri-midFormer against Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks.", "limitations": "L1. It is not clear how the proposed periodic pyramid can be effectively and efficiently integrated into multi-dimensional time series.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1.  The concept of decoupling time series data into a Periodic Pyramid is a valid point. \nThis new representation seems to capture the multi-periodicity of time series effectively.\n\nS2. The effectiveness of the proposed method is extensively verified on five tasks.", "weaknesses": "W1. While the Periodic Pyramid and self-attention mechanisms are well-explained, the model's complexity might pose challenges for practical implementation and scalability, especially for users with limited computational resources.\nIt would be useful if the authors could report in the main body of the paper, at least the main conclusion of the training and inference time for the proposed method, compared with existing baselines, across the five tasks.\n\nW2. The improvements provided by Peri-midFormer seem to be insignificant when compared with Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks.\nIt would be better if the author could further discuss this in the main body of the paper (e.g., discuss the time-accuracy tradeoff as shown in Appendix E.4 Complexity Analysis).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer GBMh\n\nThanks for your valuable comments. We will explain your concerns point by point.\n\n### **Q1: (cr. W1). Add in the main body of the paper a discussion about the training and inference time, as well as some critical results and main conclusions.**\n\nThank you very much for your valuable feedback. Concerns about computational complexity and scalability were also raised by other reviewers. Therefore, we have supplemented our experiments in this area, including the complexity, time, and memory efficiency of training and inference. Please refer to the first part of the global reply for details. We will include this content in the revised version to make the main text more complete.\n\n### **Q2: (cr. W2). Add in the main body of the paper a discussion about the significance of the improvements or about the time-accuracy tradeoff for Peri-midFormer against Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks.**\n\nThank you very much for your valuable feedback. Our method does indeed perform worse than Time-LLM and GPT4TS on some tasks. However, due to page limitations, we placed the complexity validation in the appendix, which may affect readers' understanding of the paper. Additionally, we did lack validation and discussion on the time-accuracy trade-off in the original paper. We have addressed this in the first part of the global reply section and will include it in the revised version.\n\nIn addition, due to the limitation of the amount of content that can be displayed, we currently only supplemented our experiments on the long-term forecasting task, and if necessary, we will also supplement more experiments on the imputation and anomaly detection tasks.\n\n### **Q3: L1. It is not clear how the proposed periodic pyramid can be effectively and efficiently integrated into multi-dimensional time series.**\n\nWe apologize for not clearly explaining the operational mechanism of our method in the original paper. The figures only showed the operation for a single channel, and although we mentioned retaining the original channels in line 132, this did not effectively convey our approach. We should have emphasized our use of a channel-independent strategy and clarified that the figures only illustrate operations for one channel. This will be addressed in the revised version.\n\nThank you again for your valuable feedback, which has helped make our work more complete. If you have any further questions, please feel free to ask.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2CMRRpPy3", "reviewer": "Reviewer_jAZw", "summary": "In this paper, the authors proposed a new method Peri-midFormer, which uses the multi-periodicity of time series and modeling the periodic part of a time series in a pyramid way. They further proposed an attention mechanism to use the neighborhood relation in the pyramid. Extensive experiments on different tasks show the effectiveness of the proposed method.\nAfter reading the rebuttal, I raise my rating from 5 to 6.", "questions": "- Apparently, the proposed method focused on periodic part forecasting. However, it works pretty well on the non-periodic forecasting task (short-term forecasting). For instance, there is no periodic in the 'Yearly' dataset, the proposed method still performs quite well. Can authors explain the possible reasons?\n- It is not clear why the proposed methods have a smaller number of parameters than PatchTST. Can the authors explain more?\n- It seems that the trend part is not used for classification tasks (from Figure 7). Can the author explain more?", "limitations": "Focused only on periodic signal", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of using attention in the pyramid structure seems to be novel.\n- Extensive experiments are conducted for different tasks on benchmark datasets.\n- The proposed model is light-weight.", "weaknesses": "- Some important related works are missing. \n  - There are other related works utilizing the idea of modeling time series in multi-scale, e.g. [1] [2] [3].\n- Experiments could be improved. \n  - The authors should clearly state whether they reproduce the results for comparison methods or they copy the numbers from the paper.\n  - In the original paper of PatchTST, the authors use a longer context window than 96. The authors are suggested to tune this parameter for all comparison methods (as many papers did) and provide the best results, rather than fix the context window. \n  - The authors should provide the script for hyper-parameter tuning to improve the reproducibility of the paper.\n  - The ablation study (Table 5) is only shown on one dataset (ETTh2). The authors are suggested to include results in more datasets.\n  - The training and inference complexity and actual time could be analyzed, as well as memory efficiency. \n[1] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. ICLR 2024.\n[2] PERIODICITY DECOUPLING FRAMEWORK FOR LONGTERM SERIES FORECASTING. ICLR 2024.\n[3] Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting. TKDE 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer jAZw\n\nThanks for your valuable comments. We will explain your concerns point by point.\n\n### **Q1: Some important related works are missing**\n\nThank you for your reminder. We have carefully read the paper you provided and found them very helpful for improving the background of our paper. We will include these works in the related work section of the revised version.\n\n### **Q2: Experiments could be improved**\n\nThank you very much for your valuable feedback. Here are our additions or modifications:\n\n1. **The authors should clearly state whether they reproduce the results for comparison methods or they copy the numbers from the paper.**\n\n    We apologize for the oversight in the original paper. We have reproduced some of the experimental results of the comparison methods, but due to time constraints, we also included results from the papers of these methods. Specifically, for the long-term forecasting task, we reproduced results for TSLANet and FITS, while other results were taken from previous papers:\n\n    (1) Time-LLM and iTransformer are from Table 13 of the TSLANet paper.\n\n    (2) GPT4TS, Dlinear, PatchTST, TimesNet, FEDformer, Autoformer, Stationary, ETSformer, LightTS, Informer, and Reformer are from Table 14 of the GPT4TS paper.\n\n    We will include these details in the revised version.\n\n\n2. **The authors are suggested to tune this parameter for all comparison methods and provide the best results.**\n\n    We apologize for this oversight. To ensure fair comparison, we have adjusted the look-back window for other comparison methods as per your suggestion and re-conducted the experiments. We have included this information in the second part of the global reply.\n\n3. **The authors should provide the script for hyper-parameter tuning to improve the reproducibility of the paper.**\n\n    Thank you for your valuable feedback. We will explain our hyperparameter tuning script with the following example and will release the script along with all the code to improve reproducibility:\n\n    For the long-term forecasting task on the Electricity dataset with a prediction length of 96:\n\n    ```\n    for d_model in 64 128 256 512 768\n    do\n    for layers in {1..5}\n    do\n    for top_k in {2..5}\n    do\n    for batch_size in 4 8 16 32 64\n    do\n    for learning_rate in 0.0001 0.0002 0.0005 0.001 0.002\n    do\n\n    python -u run.py \\\n    --layers $layers \\\n    --d_model $d_model \\\n    --top_k $top_k \\\n    --learning_rate $learning_rate \\\n    --batch_size $batch_size \\\n    --......\n    \n    done\n    done\n    done\n    done\n    done\n    ```\n    Based on this script, we recorded results from multiple experiments and selected the best-performing hyperparameters.\n\n4. **The ablation study is only shown on one dataset (ETTh2)**\n\n    Based on your suggestion, we have added more ablation experiments, and we have included this information in the third part of the global reply.\n\n5. **The training and inference complexity and actual time could be analyzed, as well as memory efficiency.**\n\n    Thank you for your suggestion. We have added the experiments as you recommended,and show it in the first part of the global reply section.\n\n\n### **Q3: There is no periodic in the 'Yearly' dataset, the proposed method still performs quite well**\n\nThank you for your insightful comments. Our method does indeed maintain good performance on datasets with weaker periodicity, which might seem mismatched with the focus of our approach. However, due to our use of time series decomposition, our method can effectively capture future trends by predicting the trend part even in datasets like Yearly that do not have obvious periodicity but exhibit significant trends. This is illustrated by the \"trend part\" in Figure 7 of the original paper. We have provided a detailed response in the global reply, specifically in the fourth section.\n\n### **Q4: It is not clear why the proposed methods have a smaller number of parameters than PatchTST**\n\nThank you for your insightful comments. The main reasons for this phenomenon are:\n\n1.\t**Number of patches:** If we consider each periodic component in our method as a patch, our total number of patches is smaller than that of PatchTST. Specifically, PatchTST uses a fixed number of 64 patches (there are two versions, 64 and 42, and the 64 version was used in the experiments), while the number of patches in our method is variable. In the experiments shown in Figure 16 of the original paper, we set $k$ to 3 for both tasks, resulting in 40 and 32 patches after periodic decomposition, which is less than the 64 patches in PatchTST.\n\n2.\t**Model Size Variance:** Not all comparison methods use the same model size. We refer to what was done in the TimeaNet experiments and retain the hyperparameters of the models of the original method, thus maximizing the best possible performance of the individual models. In the experiments shown in Figure 16, the PatchTST model has a dimension of 512 and 3 layers, while our model has a dimension of 16 and 2 layers. This difference in model size is another reason for the smaller parameter count in our method compared to PatchTST.\n\n### **Q5: It seems that the trend part is not used for classification tasks.**\n\nThank you for your insightful comments. As you noted, the trend part is not used in classification tasks (without time series decomposition) but follows the path indicated by the red arrow. In classification tasks, reconstructing the original data is unnecessary, so the trend part is not extracted and added back. Additionally, the trend part is a crucial discriminative feature for classification data, so it should not be separated from the original data before feature extraction. We acknowledge that this aspect was not explained in detail in the original paper, and we will provide a more thorough explanation in the revised version.\n\nThank you again for your valuable time and constructive feedback. If you have any further questions, please feel free to ask.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D0tXnVgZWb", "reviewer": "Reviewer_ECSx", "summary": "This paper introduces Peri-midFormer to capture multi-periodicity in time series data. Specifically, it designs a pyramid structure and attention mechanisms to effectively model complex temporal variations. The proposed methods demonstrates great performance in several time series analysis tasks in the author's experiments.", "questions": "1. Were the results of DLinear and PatchTST obtained with a lookback of 96? From the results in Table 12, it seems they were achieved with the default input of 336. Correspondingly, your assertion on line 236 is incorrect. As far as I know, DLinear, PatchTST, and FITS do not use a default input length of 96. FITS has a default lookback of 720.\n2. Regarding the Anomaly Detection experiment, did you use point adjustment techniques and manual thresholds? Combining point adjustment techniques with very low manual thresholds (e.g., 0.5%) can easily achieve an F1 score greater than 90 on these datasets [1], even with a randomly generated anomaly score list. Please do not simply state that you followed previous work, as this only exacerbates the legacy issue. Clearly state your stance and clarify this issue appropriately in the paper.\n3. Was the ablation study in Table 5 conducted only on the ETTh2 dataset? As far as I know, the ETTh2 dataset is small in scale, has relatively weak periodicity (compared to ETTh1, or multi-period datasets like Electricity and Traffic), and suffers from significant distribution drift. Under such circumstances, any hyperparameter adjustment or other random factors can greatly affect model performance.\n4. The Exchange dataset belongs to the financial domain and typically exhibits weak periodicity (due to the unpredictability of financial data). This is evidenced by findings in the DLinear paper, where simply copying the last point can achieve SOTA performance. This contradicts two claims in your paper: (1) Peri-midFormer improves predictive accuracy by extracting periodicity, so how did it achieve SOTA performance on this weak-period dataset? (2) Your Limitation section mentions that Peri-midFormer is not good in scenarios with weak periodicity, yet your experimental results show that your method performs well in this scenario.\n\n[1] Wagner, Dennis, et al. \"Timesead: Benchmarking deep multivariate time-series anomaly detection.\" Transactions on Machine Learning Research (2023).", "limitations": "As mentioned above, the authors have already discussed the limitations, though I have some concerns for it.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper compares a variety of cutting-edge methods.\n2. Overall, this paper is solid and the authors conducted thorough experiments.", "weaknesses": "None.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ECSx\n\nThank you for your detailed review and questions. Please find our answers below.\n\n### **Q1: Were the results of DLinear and PatchTST obtained with a lookback of 96?**\n\nWe have reviewed the results for DLinear and PatchTST in Table 12 and confirmed that you are correct—their look-back windows are not 96. We apologize for this mistake. Since most of Table 12's content was referenced from Table 14 of the GPT4TS paper, we assumed that the same look-back window was used across comparison methods.\n\nWe also apologize for the incorrect assertion in line 236. Upon review, DLinear, PatchTST, and FITS indeed do not use the default look-back window length of 96. We have adjusted the experiments and used longer look-back windows (512 and 720) for these methods to ensure fairer comparisons. We have included the updated experiments in the second part of the global reply.\n\n### **Q2: Regarding the Anomaly Detection experiment, did you use point adjustment techniques and manual thresholds?**\n\nYou are correct that our brief statement about following the TimesNet approach was problematic. To address your question first, we did use point adjustment techniques with manual thresholding, and this framework was consistently applied across all comparison methods. Here is a detailed explanation of our anomaly detection strategy:\n\n**Training Phase:** During training, we applied a simple reconstruction loss to help the model learn the distribution of normal data.\n\n**Testing Phase:** For testing, we used the following code (showing only the main parts):\n```python\ndef test(self, setting):\n\n        attens_energy = []\n        self.anomaly_criterion = nn.MSELoss(reduce=False)\n\n        # (1) stastic on the train set\n        with torch.no_grad():\n            for i, (batch_x, batch_y) in enumerate(train_loader):\n                outputs = self.model(batch_x, None, None, None)\n                score = torch.mean(self.anomaly_criterion(batch_x, outputs), dim=-1)\n                attens_energy.append(score)\n\n        attens_energy = np.concatenate(attens_energy, axis=0).reshape(-1)\n        train_energy = np.array(attens_energy)\n\n        # (2) find the threshold\n        attens_energy = []\n        test_labels = []\n        for i, (batch_x, batch_y) in enumerate(test_loader):\n            batch_x = batch_x.float().to(self.device)\n            outputs = self.model(batch_x, None, None, None)\n            score = torch.mean(self.anomaly_criterion(batch_x, outputs), dim=-1)\n            attens_energy.append(score)\n            test_labels.append(batch_y)\n\n        attens_energy = np.concatenate(attens_energy, axis=0).reshape(-1)\n        test_energy = np.array(attens_energy)\n        combined_energy = np.concatenate([train_energy, test_energy], axis=0)\n        threshold = np.percentile(combined_energy, 100 - self.args.anomaly_ratio)\n\n        # (3) evaluation on the test set\n        pred = (test_energy > threshold).astype(int)\n        test_labels = np.concatenate(test_labels, axis=0).reshape(-1)\n        gt = test_labels.astype(int)\n\n        # (4) detection adjustment\n        gt, pred = adjustment(gt, pred)\n\n        return\n```\n\nThe `adjustment` function is defined as:\n\n```python\ndef adjustment(gt, pred):\n    anomaly_state = False\n    for i in range(len(gt)):\n        if gt[i] == 1 and pred[i] == 1 and not anomaly_state:\n            anomaly_state = True\n            for j in range(i, 0, -1):\n                if gt[j] == 0:\n                    break\n                else:\n                    if pred[j] == 0:\n                        pred[j] = 1\n            for j in range(i, len(gt)):\n                if gt[j] == 0:\n                    break\n                else:\n                    if pred[j] == 0:\n                        pred[j] = 1\n        elif gt[i] == 0:\n            anomaly_state = False\n        if anomaly_state:\n            pred[i] = 1\n    return gt, pred\n```\n\nWe applied point adjustments using `gt, pred = adjustment(gt, pred)` to correct some false positives and false negatives.\n\nAdditionally, we set a manual threshold with `threshold = np.percentile(combined_energy, 100 - self.args.anomaly_ratio)`, where `combined_energy` is the combined energy score of the training and testing sets, and it is a hyperparameter. The `anomaly_ratio` used for the different datasets are shown in the table below:\n\n| Datasets        | SMD  | MSL | SMAP | SWaT | PSM |\n|:---------------:|:----:|:---:|:----:|:----:|:---:|\n| anomaly_ratio   | 0.5  | 1   | 1    | 1    | 1   |\n\nAs shown in the table, we did not apply excessive manual intervention for specific datasets.\n\nOf course, this information should have been included in the original paper, which is a shortcoming of our work. We will add this detailed explanation in the revised version.\n\n### **Q3: Was the ablation study in Table 5 conducted only on the ETTh2 dataset?** \n\nFollowing your suggestions, we have added ablation experiments on several additional datasets, and we have included the results in the third part of the global response.\n\n### **Q4: The Exchange dataset belongs to the financial domain and typically exhibits weak periodicity.** \n\nOur method does indeed maintain strong performance on some datasets with weaker periodicity, which may seem inconsistent with our focus. However, our approach involves more than just the periodic pyramid; it also incorporates time series decomposition. The success of our method on trend-dominant datasets like Exchange is largely due to this approach. Also as you mentioned the DLinear method, which also employs the operation of temporal decomposition, so in our experiments DLinear achieved very good performance on the Exchange dataset. We have provided a detailed response in the global reply, specifically in the fourth section.\n\nWe had overlooked this phenomenon in the original paper, and your insight has helped us recognize it. \n\nThank you once again for your valuable feedback. If you have any further questions, please feel free to ask.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Responses
We thank the Reviewers for the insightful comments and detailed feedback. Here's the global reply.

### **1. Validation of Computational Complexity and Scalability**

Following the suggestions of several reviewers, we have supplemented our experiments with tests on computational complexity and scalability, specifically including training and inference complexity, actual time, and memory usage. We conducted these experiments on larger datasets (Electricity and ETTh1) compared to the ETTh2 dataset used in the original paper. The results are presented in Tables 1 and 2 (Due to the long training time of Time-LLM, the relevant metrics for its inference on the Electricity dataset have not been collected yet).

As shown, our proposed Peri-midFormer demonstrates a significant advantage in computational complexity on the Electricity dataset, without the excessive inference time concerns raised by several reviewers, and achieves the lowest MSE. Similarly, on the ETTh1 dataset, the computational overhead and inference time of Peri-midFormer do not pose a disadvantage. In fact, while achieving an MSE second only to Time-LLM, Peri-midFormer’s computational overhead and inference time are substantially lower than those of Time-LLM.

This analysis demonstrates that our method has notable advantages in terms of computational complexity and scalability.

### **2. Adjustment of Lookback Window for Comparison Methods**

Due to our oversight, the original paper inaccurately described the lookback window for some comparison methods and used inappropriate lookback windows for others, leading to less rigorous experiments. Based on suggestions from several reviewers, we have adjusted the lookback windows for some comparison methods. Specifically, we have set the lookback windows for FITS, DLinear, PatchTST, TimesNet, and Pyraformer to 512, consistent with our Peri-midFormer. Additionally, since FITS originally had a lookback window of 720, we have included experiments with a 720 look-back window for comparison with Peri-midFormer. The results are presented in Table 3.

As seen from the table, several comparison methods benefit from the extended look-back window, showing some improvement in prediction performance compared to the results in Table 12 of the original paper. However, there is still a noticeable gap compared to our Peri-midFormer. Furthermore, in the comparison with a 720 look-back window, FITS does not perform as well as our Peri-midFormer.
This adjustment ensures a fairer experiment and highlights the advantages of Peri-midFormer.

### **3. Ablation Study Adjustment**

In the original paper, our ablation study was only validated on the ETTh2 dataset. Based on suggestions from several reviewers, we have expanded the experiments to include more datasets. We have supplemented the ablation study with experiments on the ETTh1, Electricity, Weather, and Traffic datasets, as shown in Table 4. It can be seen that each module we proposed performs effectively across multiple datasets, further demonstrating the superior performance of Peri-midFormer.


### **4. Explanation for Peri-midFormer's Strong Performance on Non-periodic Datasets**

In the original paper, we mentioned that Peri-midFormer excels on datasets with strong periodicity, and performs poorly on those with weak periodicity. However, as pointed out by several reviewers, Peri-midFormer has shown outstanding performance on the Exchange dataset in long-term forecasting tasks, and on the Yearly dataset in short-term forecasting tasks, both of which lack clear periodicity. This contradicts our initial description.
Upon careful examination of the Exchange and Yearly datasets, we found that while they indeed lack obvious periodicity, they exhibit strong trends, as shown in Figure 1-6. This explains why Peri-midFormer performs well on these datasets. Peri-midFormer employs a temporal decomposition strategy, which involves separating the trend part from the original data before partitioning the periodic components. The trend part is then added back after the output of Peri-midFormer, as illustrated in Figure 7 of the original paper.
To be candid, we adopted temporal decomposition to mitigate the influence of the original data's periodic characteristics, thereby enhancing Peri-midFormer's effectiveness. This approach was repeatedly explained in the original paper. Thanks to temporal decomposition, Peri-midFormer can leverage trend prediction to achieve excellent performance on datasets like Exchange and Yearly, which exhibit strong trends.

We sincerely thank the reviewers for their valuable suggestions, which have made our work more comprehensive. If possible, we will incorporate all supplementary content in the revised version. Once again, we appreciate the reviewers' time and insightful feedback!

Meta-review: There is a clear consensus in the PC that this is a strong paper that should be accepted. Congratulations to the authors! I am requesting the author to carefully incorporate the suggestion provided by the review in the camera-ready version.
Decision: Accept (spotlight)