Title: Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis
Venue: NeurIPS 2024
Accepted: yes
Keywords: general time series analysis

deep learning

transformer

periodicity

pyramid structure
Abstract: Time series analysis finds wide applications in fields such as weather forecasting, anomaly detection, and behavior recognition. Previous methods attempted to model temporal variations directly using 1D time series. However, this has been quite challenging due to the discrete nature of data points in time series and the complexity of periodic variation. In terms of periodicity, taking weather and traffic data as an example, there are multi-periodic variations such as yearly, monthly, weekly, and daily, etc. In order to break through the limitations of the previous methods, we decouple the implied complex periodic variations into inclusion and overlap relationships among different level periodic components based on the observation of the multi-periodicity therein and its inclusion relationships. This explicitly represents the naturally occurring pyramid-like properties in time series, where the top level is the original time series and lower levels consist of periodic components with gradually shorter periods, which we call the periodic pyramid. To further extract complex temporal variations, we introduce self-attention mechanism into the periodic pyramid, capturing complex periodic relationships by computing attention between periodic components based on their inclusion, overlap, and adjacency relationships. Our proposed Peri-midFormer demonstrates outstanding performance in five mainstream time series analysis tasks, including short- and long-term forecasting, imputation, classification, and anomaly detection.

Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis

Abstract

Time series analysis finds wide applications in fields such as weather forecasting, anomaly detection, and behavior recognition. Previous methods attempted to model temporal variations directly using 1D time series. However, this has been quite challenging due to the discrete nature of data points in time series and the complexity of periodic variation. In terms of periodicity, taking weather and traffic data as an example, there are multi-periodic variations such as yearly, monthly, weekly, and daily, etc. In order to break through the limitations of the previous methods, we decouple the implied complex periodic variations into inclusion and overlap relationships among different level periodic components based on the observation of the multi-periodicity therein and its inclusion relationships. This explicitly represents the naturally occurring pyramid-like properties in time series, where the top level is the original time series and lower levels consist of periodic components with gradually shorter periods, which we call the periodic pyramid. To further extract complex temporal variations, we introduce self-attention mechanism into the periodic pyramid, capturing complex periodic relationships by computing attention between periodic components based on their inclusion, overlap, and adjacency relationships. Our proposed Peri-midFormer demonstrates outstanding performance in five mainstream time series analysis tasks, including short- and long-term forecasting, imputation, classification, and anomaly detection. The code is available at https://github.com/WuQiangXDU/Peri-midFormer .

1 Introduction

Time series analysis stands as a foundational challenge pivotal across diverse real-world scenarios [1], such as weather forecasting [2], imputation of missing data within offshore wind speed time series [3], anomaly detection for industrial maintenance [4], and classification [5]. Due to its substantial practical utility, time series analysis has garnered considerable interest, leading to the development of a large number of deep learning-based methods for it.

Different from other forms of sequential data, like language or video, time series data is continuously recorded, capturing scalar values at each time point. Furthermore, real-world time series variations often entail complex temporal patterns, where multiple fluctuations (e.g., ascents, descents, fluctuations, etc.) intermingle and intertwine, particularly salient is the presence of various overlapping periodic components in it, rendering the modeling of temporal variations exceptionally challenging.

Deep learning models, known for their powerful non-linear capabilities, capture intricate temporal variations in real-world time series. Recurrent neural networks (RNNs) leverage sequential data, allowing past information to influence future predictions [6; 7]. However, they face challenges with long-term dependencies and computational inefficiency due to their sequential nature. Temporal convolutional neural networks (TCNs) [8; 9] extract variation information but struggle with capturing long-term dependencies. Transformers with attention mechanisms have gained popularity for sequen- syyang}@xidian.edu.cn

Period 3

Period 2

Period 1

Original time series

Multi-periodicity

Inclusion and overlap

tial modeling [10; 11], capturing pairwise temporal dependencies among time points. Yet, discerning reliable dependencies directly from scattered time points remains challenging [12]. Timesnet [13] innovatively transforms 1D time series into 2D tensors, unifying intra- and inter-period variations in 2D space. However, it overlooks inclusion relationships between periods of different scales and is constrained by limited feature extraction capability of CNNs, hindering its ability to explore complex relationships within time series.

We analyze time series by examining the inclusion and overlap (hereinafter collectively referred to as inclusion) relationships between various periodic components to address complex temporal variations. Real-world time series often show multiple periodicities, like yearly and daily weather variations or weekly and daily traffic fluctuations. And these periods exhibit clear inclusion relationships, for instance, yearly weather variations encompass multiple daily weather variations. Besides variations between different period levels, it also occur within periods of the same level. For example, daily weather variations differ based on conditions like sunny or cloudy. Due to these inclusion and adjacency relationships, different periods show similarities, with short and long periods being consistent in overlapping portions, and periods of the same level being similar. Additionally, a long period can be decomposed into multiple short ones, forming a hierarchical pyramid structure. In our study, time series without explicit periodicity are treated as having infinitely long periods.

Based on the above analysis, we decompose the time series into multiple periodic components, forming a pyramid structure where longer components encompass shorter ones, termed the Periodic Pyramid as shown in Figure 1, which illustrates the intricate periodic inclusion relationships within the time series. Each level consists of components with the same period, exhibiting the same phase, while different levels contain components with inclusion relationships. This transformation converts the original 1D time series into a 2D representation, explicitly showing the implicit multi-period relationships. Within the Periodic Pyramid, a shorter period may belong to two longer periods simultaneously, reflecting the complexity of the time series. There is a clear similarity between components within the same level and those in adjacent levels where inclusion relationships exist. Thus inspired by Pyraformer [10], we propose the Peri odic Pyra mid Trans former ( Peri-midFormer ), which computes self-attention among periodic components to capture complex temporal variations in time series. Furthermore, we consider each branch in the Periodic Pyramid as a Periodic Feature Flow, and aggregating features from multiple flows to provide rich periodic information for downstream tasks. In experiments, Peri-midFormer achieves state-of-the-art performance in various analytic tasks, including forecasting, imputation, anomaly detection, and classification.

Based on the inclusion relationships of multiple periods in time series, this paper proposes a top-down constructed Periodic Pyramid structure, which expands 1D time series variations into 2D, explicitly representing the implicit multi-period relationships within the time series.

We propose Peri-midFormer, which uses the Periodic Pyramid Attention Mechanism to automatically capture dependencies between different and same-level periodic components, extracting diverse temporal variations in time series. Additionally, to further harness the potential of Peri-midFormer, we introduce Periodic Feature Flows to provide rich periodic information for downstream tasks.

We conduct extensive experiments on five mainstream time series analysis tasks, and PerimidFormer achieves state-of-the-art across all of them, demonstrating its superior capability in time series analysis.

The remainder of this paper is structured as follows. Section 2 briefly summarizes the related work. Section 3 details the proposed model structure. Section 4 extensively evaluates our method's performance across five main time series analysis tasks. Section 5 presents ablations analysis, Section 6 presents complexity analysis, and Section 7 discusses our results and future directions.

2 Related Work

Temporal variation modeling, a crucial aspect of time series analysis, has been extensively investigated. In recent years, numerous deep models have emerged for this purpose, including MLP [14; 15], TCN [8], and RNN [6; 7]-based architectures. Furthermore, Transformers have shown remarkable performance in time series forecasting [16; 12; 17; 18]. They utilize attention mechanisms to uncover temporal dependencies among time points. For instance, Wu et al. [12] introduce Autoformer with an Auto-Correlation mechanism, adept at capturing series-wise temporal dependencies derived from learned periods. Moreover, to address complex temporal patterns, they adopted a deep decomposition architecture to extract seasonal and trend parts from input series. Subsequently, FEDformer [17] enhances seasonal-trend decomposition through a mixture-of-expert design and introduces sparse attention within the frequency domain. Pyraformer [10] constructs a down-top pyramid structure through multiple convolution operations on time series to address the issue of long information propagation paths in Transformers, significantly reducing both time and space complexity. PatchTST [19] partitions individual data points into patches and uses them as tokens for the Transformer, thereby enhancing its understanding of local information in time series. Additionally, PatchTST innovatively processes each channel separately, making it particularly suitable for forecasting tasks.

Additionally, there are some recent innovative works. Timesnet [13] unravels intricate temporal patterns by exploring the multi-periodicity of time series and captures temporal 2D-variations using computer vision CNN backbones. GPT4TS [20] ingeniously utilizes the large language model GPT2 as a pretrained model, fine-tuning some of its structures with time series, achieving state-of-the-art results. FITS [21] proposes a time series analysis model based on frequency domain operations, requiring very low parameter count and memory consumption. And recent works have considered multi-scale information in time series. PDF [22] captures both short-term and long-term variations by transforming 1D time series into 2D tensors using a multi-periodic decoupling block. It achieves superior forecasting performance by modeling these decoupled variations and integrating them for accurate predictions. SCNN [23] decomposes multivariate time series into long-term, seasonal, shortterm, co-evolving, and residual components, enhancing interpretability, adaptability to distribution shifts, and scalability by modeling each component separately. TimeMixer [24] uses a novel multiscale mixing approach, decomposing time series into fine and coarse scales to capture both detailed and macroscopic variations. It employs Past-Decomposable-Mixing to extract historical information and Future-Multipredictor-Mixing to leverage multiscale forecasting capabilities, achieving great performance in forecasting task.

3 Methodology

3.1 Model Structure

The overall flowchart of the proposed approach is shown in Figure 2, it begins with time embedding of the original time series at the top. Then, we use the FFT to decompose it into multiple periodic components of varying lengths across different levels, with lines indicating the inclusion relationships between them. Moving down, padding and projection are then applied to ensure uniform dimensions, forming the Periodic Pyramid. Each component is treated as an independent token and receives positional embedding. Next, the Periodic Pyramid is fed into Peri-midFormer, which consists of multiple layers for computing Periodic Pyramid Attention. Finally, depending on the task, two strategies are employed: for classification, components are directly concatenated and projected into the category space; for other reconstruction tasks (since forecasting, imputation, and anomaly detection all necessitate the model to reconstruct the channel dimensions or input lengths, we collectively refer to such tasks as reconstruction tasks), features from different pyramid branches are integrated through Periodic Feature Flows Aggregation to generate the final output. Please note that we referred to [11] for de-normalization and [12] for time series decomposition to maximize the effectiveness of our method, but we omitted these details from the figure to maintain simplicity. See Appendix A for a complete flowchart. Further details are provided below.

3.2 Periodic Pyramid Construction

Multiple periods in the time series exhibit clear inclusion relationships, however, the 1D structure limits the representation of variations between them. Hence, it's crucial to separate periodic com-

FFT for periods

0

3

4

8

5

7

1

2

6

Peri-midFormer

Temporal embedding

Downstream

tasks

Padding & Projection

Positional embedding

Time domain

Frequency domain

Original Length

Frequency 1

Frequency 2

Frequency 3

3 5

9

Top k

periods

0

1

2

3

4

5

6

7

8

PPAM

Add

Feed Forward

Add & Norm

Periodic Feature

Flows Aggregation

Flatten &

Projection

Classification

task

Reconstruction

tasks

N

ⅹ

Strategy 1 for classification task

Strategy 2 for reconstruction task

Periodic pyramid

1

1

f

=

2

3

f

=

3

5

f

=

ponents with inclusion relationships to explicitly represent implicit periodic relationships. Firstly, as Peri-midFormer is designed to focus on periodic components, we first normalize the original time series X ∈ R L × C that with length L and C channels, then decompose it to obtain the seasonal part X s ∈ R L × C , thus removing the interference of the trend part. For a detailed description of normalization and decomposition, please refer to the Appendix A. Then we partition X s into periodic components, following the approach used in Timesnet [13]. It's important to note that, inspired by PatchTST [19], we retain the channel dimension C , as it is advantageous for Peri-midFormer in capturing periodic features within each channel (note that we adopt a channel independent strategy and the Figure 2 shows the processing of only one of the channels). The periodic components are extracted in the frequency domain, accomplished specifically through FFT:

where FFT ( · ) and Amp ( · ) denote Fourier Transform and amplitude calculation, respectively. A ∈ R L represents the amplitude of each frequency, averaged across C channels using Avg(·). Note that the j -th value A j denotes the intensity of the j -th frequency periodic basis function, associated with period length ⌈ L j ⌉ . To handle frequency domain sparsity and minimize noise from irrelevant high frequencies [17], the topk amplitude values { A f 1 , · · · , A f k } corresponding to the most significant frequencies { f 1 , · · · , f k } are selected, where k is a hyper-parameter, beginning from 2 to ensure the fundamental pyramid structure. Additionally, to ensure the top level of the pyramid corresponds to the original time series, we define f 1 = 1 , with other frequencies arranged in ascending order. These selected frequencies correspond to k period lengths { p 1 , · · · , p k } , arranged in descending order. Due to the frequency domain's conjugacy, only frequencies within { 1 , · · · , ⌈ L 2 ⌉} are considered. Based on the selected frequencies { f 1 , · · · , f k } and their associated period lengths { p 1 , · · · , p k } , we partition the original time series into periodic components for each pyramid level, denoted as C ℓ :

where C n ℓ denotes the n -th periodic component in the ℓ -th pyramid level. Here, ℓ is the pyramid level index, starting from the top and increasing, with a maximum value of k , indicating the number of levels determined by the selected periods. Similarly, n represents the component index within a level, increasing from left to right, with a maximum value of f k , indicating the number of components per level is determined by the frequency corresponding to that period in the original series. The Periodic Pyramid can thus be represented as:

0

1

2

3

4

5

6

7

8

0

Index

=

3

4

2

3

(

)

(

)

0

Index

Index



C

C

1

1

2

3

(

)

(

)

0

Index

Index



C

C

1

Index

L

=

-

4

8

1

5

3

7

0

2

6

where Stack ( · ) denotes a stacking operation. The constructed Periodic Pyramid, depicted in the upper right of Figure 2, displays evident inclusion relationship among different levels, shown by connections between levels. Let R denote the relationship between pairs of periodic components from the upper and lower levels, determined by the presence or absence of overlap as follows:

when R = 1 , it signifies an inclusion relationship, while R = 0 indicates no overlap. This is illustrated in the left half of Figure 3. n ℓ denotes the index of the n -th component in the ℓ -th level. Index ( · ) denotes the positional index of each data point within the periodic component at that level. Indices for points in the first level are contained in { 0 , . . . , L -1 } . For subsequent levels, most indices match those of the first level. However, due to varying component lengths, there may be slight differences in indices for the last portion. Nonetheless, this doesn't impact relationship determination between components across levels. In practice, the relationship between the components is realized by masking the corresponding elements in the attention matrix.

Thanks to the inclusion relationships between periodic components across different levels in the Periodic Pyramid, complex periodic relationships inherent in 1D time series are explicitly represented. Next, due to the varying lengths of the components, it's necessary to map C ℓ to the same scale for subsequent Periodic Pyramid Attention Mechanism, with the equation provided as follows:

where Padding ( · ) denotes zero-padding the periodic components across the time dimension to match the length of the original data, while Projection ( · ) represents a single linear mapping layer.

3.3 Periodic Pyramid Transformer (Peri-midFormer)

Once we have the Periodic Pyramid, it can be inputted into the Peri-midFormer, as depicted in Figure 2. The Peri-midFormer introduces a specialized attention mechanism tailored for the Periodic Pyramid, called P eriodic P yramid A ttention M echanism ( PPAM ), shown in the right half of Figure 3. Here, original connections are replaced with bidirectional arrows, and also added within the same level. These bidirectional arrows signify attention between periodic components. In PPAM, inter-level attention focuses on period dependencies across levels, while intra-level attention focuses on dependencies within the same level. Note that attention occurs among all components within the same level, not just between adjacent ones. However, for clarity, not all attention connections within the same level are depicted.

In Periodic Pyramid, a periodic component C n ℓ generally has three types of interconnected relationships (denoted as I ) with other components: the parent node in the level above (denoted as P ), all nodes within the same level including itself (denoted as A ), and the child nodes in its next level (denoted as C ). Therefore, the relationships of C n ℓ can be expressed by the following equation:

0

3

4

8

5

7

1

2

6

1

4

1

5

2

5

2

6

2

7

3

7

3

8

Projection

0

0

0

0

0

0

0

Mean pooling

The equation shows that a component at the topmost level lacks a parent node, while one at the bottommost level lacks a child node. Based on the interconnected relationships I , the attention of the component C n ℓ can be expressed as:

where q , k , and v denote query, key, and value vectors, respectively, as in the classical self-attention mechanism. m used for selecting components that have interconnected relationships with C n ℓ . k ⊤ m represents the transpose of row m in K . d K refers to the dimension of key vectors, ensuring stable attention scores through scaling.

We apply this attention mechanism to each component across all levels of the Periodic Pyramid, enabling the automatic detection of dependencies among all components in the Periodic Pyramid and capturing the intricate temporal variations in the time series. For a detailed theoretical proof of the PPAM see the Appendix F.

3.4 Periodic Feature Flows Aggregation

Here we explain the Periodic Feature Flows Aggregation used for reconstruction tasks. The output of the Peri-midFormer retains the original pyramid structure. To leverage the diverse periodic components across different levels, we treat a single branch from the top to the bottom of the pyramid as a periodic feature flow, highlighted by the red line in Figure 4. Since a periodic feature flow passes through periodic components at different levels, it contains periodic features of different scales from the time series. Additionally, due to variations among periodic components within each level, each feature flow carries distinct information. Therefore, we aggregate multiple feature flows through Periodic Feature Flow Aggregation. This involves linearly mapping each feature flow to match the length of the target time series and then averaging it across multiple feature flows to obtain the aggregated result Y s , as expressed in the following equation:

where ˆ C n k ℓ represents a specific periodic component in the Peri-midFormer's output, and { ˆ C n 1 1 , ˆ C n 2 2 , · · · , ˆ C n k ℓ } forms a feature flow, as shown in Figure 4. Projection ( · ) maps each feature flow to match the target output length. Meanpooling ( · ) averages the feature flows. Y s indicates that this is the output from the seasonal part. Since we retained the channel dimension of the original time series, the result obtained after aggregating the periodic feature flows here becomes the shape of the final output. Finally, adding the trend part and de-normalization to obtain the ultimate output.

4 Experiments

We extensively test Peri-midFormer on five mainstream analysis tasks: short- and long-term forecasting, imputation, classification, and anomaly detection. We adopted the same benchmarks as Timesnet [13], see Appendix C for details. Due to space limits, we provide only a summary of the results here, more details about the datasets, experiment implementation, model configuration, and full results can be found in Appendix.

Baselines The baselines include CNN-based models: TimesNet [13]; MLP-based models: LightTS [15], DLinear [14] and FITS [21]; Transformer-based models: GPT4TS [20], TimeLLM [25], iTransformer [26], TSLANet [27] , Reformer [28], Pyraformer [10], Informer [16], Autoformer [12], FEDformer [17], Non-stationary Transformer [11], ETSformer [29], PatchTST [19]. Besides, N-HiTS [30] and N-BEATS [31] are used for short-term forecasting. Anomaly Transformer [32] is used for anomaly detection. Rocket [33], LSTNet [34], TCN [8] and Flowformer [35] are used for classification.

4.1 Main Results

Figure 5 displays the comprehensive comparison results between PerimidFormer and other methods, it consistently excels across all five tasks.

4.2 Long-term Forecasting

Setups Referring to [13], we adopt eight real-world benchmark datasets for long-term forecasting, including Weather [36], Traffic [37], Electricity [38], Exchange [34], and four ETT [16] datasets (ETTh1, ETTh2, ETTm1, ETTm2). Forecast lengths are set to 96, 192, 336, and 720. For the fairness of the comparison, we set the look-back window for all the methods to 512 (64 on Exchange), the results for other look-back windows can be found in the Appendix H.3

Long-term Forecasting (MSE)

Classification

Anomaly Detection

(F1-Score)

Imputation(MSE)

Short-term

Forecasting

(OWA)

0.3

77

87

77

0.045

0.055

0.85

0.095

65

0.4

Peri-midFormer(ours)

Timesnet(2023)

Time-LLM(2024)

PatchTST(2023)

FITS(2024)

DLinear(2023)

iTransformer(2024)

ETSformer(2022)

TSLANet(2024)

FEDformer(2022)

GPT4TS(2023)

Pyraformer(2021)

Results From Table 1, it is evident that Peri-midFormer performs exceptionally well, even completely outperforms GPT4TS and closely approaching the state-of-the-art method Time-LLM. While TimeLLM demonstrates remarkable capabilities in long-term forecasting, our Peri-midFormer shows clear advantages on the ETTh2, Electricity, and Exchange datasets. Although Time-LLM achieves the best results, it relies on a very large model, leading to significant computational overhead that is unavoidable. The same issue exists for GPT4TS. In contrast, our Peri-midFormer achieves performance close to that of Time-LLM without requiring excessive computational resources, making it more suitable for practical applications. Further analysis of model complexity is provided in Section 6. In addition, our Peri-midFormer exhibits better performance with longer look-back window, as further detailed in the Appendix E.6.

4.3 Short-term Forecasting

Setups For short-term analysis, we adopt the M4 [39], which contains the yearly, quarterly and monthly collected univariate marketing data. We measure forecast performance using the symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE), and overall weighted average (OWA), which are calculated as detailed in the Appendix D.1.

Results Table 2 shows that Peri-midFormer outperforms Time-LLM, GPT4TS, TimesNet, and NBEATS, highlighting its exceptional performance in short-term forecasting. In the M4 dataset, some data lacks clear periodicity, such as the Yearly data, which mainly exhibits a strong trend. A similar situation is observed in the Exchange dataset for long-term forecasting task. Peri-midFormer performs well on these datasets due to its time series decomposition strategy. For a detailed analysis, please refer to the Appendix E.7.

4.4 Time Series Classification

Setups We assessed Peri-midFormer's capacity for high-level representation learning via classification task. Mimicking settings akin to TimesNet [13], we tested it on 10 multivariate UEA classification datasets from [42], covering tasks like gesture recognition, action recognition, audio recognition, medical diagnoses, and other real-world applications.

Results As shown in Figure 6, PerimidFormer achieves an average accuracy of 76.6%, surpassing all baselines including TSLANet (76.0%), GPT4TS (74.0%), TimesNet (73.6%), and all other Transformer-based methods. This suggests that Peri-midFormer has excellent time series representation capabilities. See Appendix H.1 for full results.

4.5 Imputation

Setups To validate Peri-midFormer's imputation capabilities, we conduct experiment on six realworld datasets, including four ETT datasets [16] (ETTh1, ETTh2, ETTm1, ETTm2), Electricity [38], and Weather [36]. We evaluate different random mask ratios (12.5%, 25%, 37.5%, 50%) for varying levels of missing data. Notably, due to the large number of missing values, the time series do not reflect their original periodicity. Therefore, before imputation, we simply interpolate the original missing data through a linear interpolation strategy in order to use Peri-midFormer efficiently, which we call pre-interpolation. For a description of pre-interpolation and its impact on other methods, please refer to the Appendix E.5.

Results Table 3 demonstrates Peri-midFormer's outstanding performance on specific datasets (Electricity and Weather), surpassing other methods significantly and securing the highest average scores. However, its performance on other datasets was ordinary, possibly due to the lack of obvious periodic characteristics in them.

Peri-midFormer

76.6

TSLANet

76.0

GPT4TS

174.0

TimesNet

73.6

Flowformer

73.0

Stationformer

72.7

Rocket

72.5

Transformer

71.9

Reformer

71.5

Autoformer

71.1

ETSformer

171.0

Pyraformer

170.8

FEDformer

70.7

LightTS

170.4

TCN

170.3

PatchTST

68.7

Dlinear

67.5

Average Accuracy(%)

4.6 Time Series Anomaly Detection

Setups For anomaly detection, we assess models on five standard datasets: SMD [43], MSL [44], SMAP [44], SWaT [45] and PSM [46]. To ensure fairness, we exclusively use classical reconstruction error for all baseline models, aligning with the approach in TimesNet [13]. Specifically, normal data is used for training, and a simple reconstruction loss is employed to help the model learn the distribution of normal data. In the testing phase, parts of the reconstructed output that exceed a certain threshold are considered anomalies. We use a point adjustment technique combined with a manually set threshold for this purpose.

Results The results in Table 4 illustrate that Peri-midFormer's anomaly detection capability is second only to GPT4TS, but a large gap does exist. This is due to the binary event data nature in the anomaly detection dataset [21], which makes it difficult for Peri-midFormer to capture useful periodic characteristics, leading to general performance.

5 Ablations

Setups We performed ablation experiments on key modules of Peri-midFormer for the long-term forecasting task, with results presented in Table 5. The table outlines the progression of module additions, from top to bottom. "Pyraformer" refers to the Pyraformer [10], building the pyramid downtop with convolutions and employing a simple two-fold relationship for attention distribution. "w/o periodic components" constructs a pyramid top-down by dividing the time series into patches without

Classification on the UEA Heartbeat

Long-term forecasting of length 720 on ETTh2

0

1

2

3

4

5

Number of Parameters (Millions)

70

75

80

85

90

95

Accuracy (%)

Peri-midFormer

GPT4TS

Timesnet

PatchTST

Pyraformer

Informer

Crossformer

Autoformer

FEDformer

TSLANet

1G FLOPs 10G FLOPs

0

10

20

30

40

50

60

Number of Parameters (Millions)

0.30

0.35

0.40

0.45

0.50

0.55

0.60

MSE

Peri-midFormer

Time-LLM

(12750G FLOPs)

GPT4TS

Timesnet

PatchTST

DLinear

Autoformer

TSLANet

10G FLOPs

100G FLOPs

considering periodic components. "w/o PPAM" divides the time series into periodic components but without Period Pyramid Attention Mechanism, using periodic full attention instead, wherein attention is computed among all periodic components. "w/o Feature Flows Aggregation" employs PPAM but without Periodic Feature Flows Aggregation. "Peri-midFormer" indicates our final approach.

Results Table 5 illustrates the incremental performance enhancement achieved with the integration of each additional module, validating the effectiveness of Peri-midFormer. Notably, good results are achieved even without PPAM. This can be attributed to the model's ability to extract periodic characteristics inherent in the original time series data by delineating the periodic components. However, without highlighting inclusion relationships through PPAM, periodic full attention's ability to capture temporal changes is limited, emphasizing the significance of PPAM.

6 Complexity Analysis

We conducted experiments on the model complexity of Peri-midFormer using the Heartbeat dataset for the classification task and the ETTh2 dataset for the long-term forecasting task. We considered the number of training parameters, FLOPs, and accuracy (or MSE). The results are depicted in Figure 7. In the classification task, Peri-midFormer not only achieves a significant advantage in accuracy but also requires relatively fewer training parameters and FLOPs, much less than many methods such as TimesNet, GPT4TS, Crossformer, and PatchTST. In the long-term forecasting task, Peri-midFormer achieves the lowest MSE without requiring the enormous FLOPs that Time-LLM does. This shows that although Time-LLM has strong long-term forecasting capabilities on most datasets, its computational demands are unacceptable. See Appendix E.4 for more analysis.

Further Model Analysis is provided in the Appendix E.

7 Conclusions

In this paper, we introduced a method for general time series analysis called Peri-midFormer. It leverages the multi-periodicity of time series and the inclusion relationships between different periods. By segmenting the original time series into different levels of periodic components, PerimidFormer constructs a Periodic Pyramid along with its corresponding attention mechanism. Through extensive experiments covering forecasting, classification, imputation, and anomaly detection tasks, we validated the capabilities of Peri-midFormer in time series analysis, achieving outstanding results across all tasks. However, Peri-midFormer exhibits limitations, particularly in scenarios where the periodic characteristics are less apparent. We aim to address this limitation in future research to broaden its applicability.

References

Qingsong Wen, Linxiao Yang, Tian Zhou, and Liang Sun. Robust time series analysis and applications: An industrial perspective. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4836-4837, 2022.

Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature , 619(7970):533-538, 2023.

Yaoran Chen, Candong Cai, Leilei Cao, Dan Zhang, Limin Kuang, Yan Peng, Huayan Pu, Chuhan Wu, Dai Zhou, and Yong Cao. Windfix: Harnessing the power of self-supervised learning for versatile imputation of offshore wind speed time series. Energy , 287:128995, 2024.

Haotian Si, Changhua Pei, Hang Cui, Jingwen Yang, Yongqian Sun, Shenglin Zhang, Jingjing Li, Haiming Zhang, Jing Han, Dan Pei, et al. Timeseriesbench: An industrial-grade benchmark for time series anomaly detection models. arXiv preprint arXiv:2402.10802 , 2024.

Zhen Liu, Wenbin Pei, Disen Lan, and Qianli Ma. Diffusion language-shapelets for semisupervised time-series classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 14079-14087, 2024.

Hongming Li, Shujian Yu, and Jose Principe. Causal recurrent variational autoencoder for medical time series generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8562-8570, 2023.

Minrong Lu and Xuerong Xu. Trnn: An efficient time-series recurrent neural network for stock price prediction. Information Sciences , 657:119951, 2024.

Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in Neural Information Processing Systems , 32, 2019.

Yangdong He and Jiabao Zhao. Temporal convolutional networks for anomaly detection in time series. J. Phys. Conf. Ser , 2019.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations , 2021.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Processing Systems , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems , pages 101-112, 2021.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations , 2023.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 , 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In 35th AAAI Conference on Artificial Intelligence , pages 11106-11115, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning , 2022.

Zelin Ni, Hang Yu, Shizhan Liu, Jianguo Li, and Weiyao Lin. Basisformer: Attention-based time series forecasting with learnable and interpretable basis. Advances in Neural Information Processing Systems , 36, 2024.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations , 2022.

Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained lm. Advances in Neural Information Processing Systems , 36, 2024.

Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10 k parameters. In The Twelfth International Conference on Learning Representations , 2023.

Tao Dai, Beiliang Wu, Peiyuan Liu, Naiqi Li, Jigang Bao, Yong Jiang, and Shu-Tao Xia. Periodicity decoupling framework for long-term series forecasting. In The Twelfth International Conference on Learning Representations , 2024.

Jinliang Deng, Xiusi Chen, Renhe Jiang, Du Yin, Yi Yang, Xuan Song, and Ivor W Tsang. Disentangling structured components: Towards adaptive, interpretable and scalable time series forecasting. IEEE Transactions on Knowledge and Data Engineering , 2024.

Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU. Timemixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations , 2024.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations , 2024.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations , 2023.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, and Xiaoli Li. Tslanet: Rethinking transformers for time series representation learning. arXiv preprint arXiv:2404.08472 , 2024.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations , 2020.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381 , 2022.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 6989-6997, 2023.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations , 2019.

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations , 2021.

Angus Dempster, François Petitjean, and Geoffrey I Webb. ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery , 34(5):1454-1495, 2020.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows. In International Conference on Machine Learning , pages 24226-24242. PMLR, 2022.

Wetterstation. Weather. https://www.bgc-jena.mpg.de/wetter/ .

PeMS. Traffic. http://pems.dot.ca.gov/ .

UCI. Electricity. https://archive.ics.uci.edu/ml/datasets/ ElectricityLoadDiagrams20112014 .

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting , 34(4):802808, 2018.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 , 2022.

T. Zhang, Yizhuo Zhang, Wei Cao, J. Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 , 2022.

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2828-2837, 2019.

Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Söderström. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 387-395, 2018.

Aditya P Mathur and Nils Ole Tippenhauer. Swat: A water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater) , pages 31-36. IEEE, 2016.

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2485-2494, 2021.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS , 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

A Method Details

FFT for periods

Top k periods

0

3

4

8

5

7

1

2

6

0

1

2

3

4

5

6

7

8

Periodic pyramid

Peri-midFormer

Temporal

embedding

Downstream

tasks

Padding & Projection

Projection

Normalization

De-normalization

Decomposition

Seasonal part

Trend part

Strategy 1

Strategy 2

Strategy 2

Strategy 1

Input

Positional embedding

1

1

f

=

2

3

f

=

3

5

f

=

ˆ

classes

Y

ˆ

T C



Y

T C

t



H

L C

t



X

L C

s



X

L C



X

,





X

X

T C

s



Y

Here, we provide further elaboration on the details of Peri-midFormer. Its full flowchart is depicted in Figure 8, depicts two strategies for input. The strategy indicated by the red line is for classification task, where de-normalization and time series decomposition are not used. This is because, in classification task, there is no need to reconstruct the original data; therefore, the trend part does not need to be extracted and added back. Additionally, it is important to note that the trend part is a significant discriminative feature for classification data, so it cannot be separated from the original data before feature extraction. In the strategy indicated by the blue line, employed for reconstruction tasks, PerimidFormer needs to focus on the periodicity in the time series. Therefore, we utilize de-normalization and time series decomposition to eliminate other influencing factors. To achieve this, we first refer to [11] to address instability factors. We normalize the input X = [ x 1 , x 2 , ..., x L ] ∈ R L × C to obtain X norm = [ x ′ 1 , x ′ 2 , ..., x ′ L ] ∈ R L × C :

where µ x , σ x ∈ R C × 1 are the mean and variance, respectively, 1 σ x means the element-wise division and ⊙ is the element-wise product. Normalization reduces the disparity in distribution among individual input time series, thereby stabilizing the model input distribution.

Then, to remove the trend part from the time series and only preserve the seasonal part for PerimidFormer, we refer to [12] for time series decomposition, as shown in the following equation:

where X s , X t ∈ R L × C denote the seasonal and the trend part respectively. We adopt the AvgPool ( · ) for moving average with the padding operation to keep the series length unchanged.

The seasonal part X s obtained after decomposition can be directly input into Peri-midFormer. After the output from Peri-midFormer, we add the trend part back, then de-normalize it to obtain the final output ˆ Y = [ ˆ y 1 , ˆ y 2 , ..., ˆ y T ] :

where H represents the Peri-midFormer model, Y s represents the output of Peri-midFormer, Projection ( · ) represents mapping the trend part to the target output length, and ˆ y i = σ x ⊙ y i + µ x denotes de-normalization.

B Visualization

To provide a clearer demonstration of Peri-midFormer's representational capabilities, Figure 9, 10 and 11 visualize some of the results for imputation, long-term forecasting and short-term forecasting, respectively. It illustrates that Peri-midFormer outperforms other methods in capturing periodic variations in time series.

(1) Imputation on Weather dataset under 50% mask ratio

0

20

40

60

80

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Prediction

GroundTruth

Peri-midFormer (ours)

0

20

40

60

80

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Prediction

GroundTruth

GPT4TS

0

20

40

60

80

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Prediction

GroundTruth

PatchTST

0

20

40

60

80

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Prediction

GroundTruth

TimesNet

0

20

40

60

80

0.04

0.02

0.00

0.02

0.04

0.06

0.08

0.10

Prediction

GroundTruth

FEDformer

0

20

40

60

80

0.00

0.02

0.04

0.06

0.08

Prediction

GroundTruth

Dlinear

(2) Imputation on Electricity dataset under 50% mask ratio

0

20

40

60

80

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Prediction

GroundTruth

Peri-midFormer (ours)

0

20

40

60

80

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Prediction

GroundTruth

GPT4TS

0

20

40

60

80

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Prediction

GroundTruth

PatchTST

0

20

40

60

80

0

1

2

3

Prediction

GroundTruth

TimesNet

0

20

40

60

80

1

0

1

2

3

Prediction

GroundTruth

FEDformer

0

20

40

60

80

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Prediction

GroundTruth

Dlinear

(1) Long-term forecasting with 96 prediction length on ETTh2

0

25

50

75

100

125

150

175

200

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

GroundTruth

Prediction

Peri-midFormer (ours)

0

25

50

75

100

125

150

175

200

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

GroundTruth

Prediction

GPT4TS

0

25

50

75

100

125

150

175

200

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

GroundTruth

Prediction

TSLANet

0

25

50

75

100

125

150

175

200

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

GroundTruth

Prediction

PatchTST

0

25

50

75

100

125

150

175

200

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

GroundTruth

Prediction

TimesNet

0

25

50

75

100

125

150

175

200

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

GroundTruth

Prediction

Dlinear

0

25

50

75

100

125

150

175

200

1.5

1.0

0.5

0.0

0.5

1.0

GroundTruth

Prediction

Peri-midFormer (ours)

0

25

50

75

100

125

150

175

200

1.5

1.0

0.5

0.0

0.5

1.0

GroundTruth

Prediction

GPT4TS

0

25

50

75

100

125

150

175

200

1.5

1.0

0.5

0.0

0.5

1.0

GroundTruth

Prediction

TSLANet

0

25

50

75

100

125

150

175

200

1.5

1.0

0.5

0.0

0.5

1.0

GroundTruth

Prediction

PatchTST

0

25

50

75

100

125

150

175

200

1.5

1.0

0.5

0.0

0.5

1.0

GroundTruth

Prediction

TimesNet

0

25

50

75

100

125

150

175

200

1.5

1.0

0.5

0.0

0.5

1.0

GroundTruth

Prediction

Dlinear

(1) Short-term forecasting on M4 Weekly

0

5

10

15

20

25

30

35

2400

2450

2500

2550

2600

2650

GroundTruth

Prediction

Peri-midFormer (ours)

0

5

10

15

20

25

30

35

2400

2450

2500

2550

2600

2650

GroundTruth

Prediction

GPT4TS

0

5

10

15

20

25

30

35

2400

2450

2500

2550

2600

2650

GroundTruth

Prediction

PatchTST

0

5

10

15

20

25

30

35

2400

2450

2500

2550

2600

2650

GroundTruth

Prediction

TimesNet

0

5

10

15

20

25

30

35

2400

2450

2500

2550

2600

2650

GroundTruth

Prediction

FDEformer

0

5

10

15

20

25

30

35

2400

2450

2500

2550

2600

2650

GroundTruth

Prediction

Dlinear

(2) Short-term forecasting on M4 Monthly

0

10

20

30

40

50

4000

6000

8000

10000

12000

14000

GroundTruth

Prediction

Peri-midFormer (ours)

0

10

20

30

40

50

4000

6000

8000

10000

12000

14000

GroundTruth

Prediction

GPT4TS

0

10

20

30

40

50

4000

6000

8000

10000

12000

14000

GroundTruth

Prediction

PatchTST

0

10

20

30

40

50

4000

6000

8000

10000

12000

14000

GroundTruth

Prediction

TimesNet

0

10

20

30

40

50

4000

6000

8000

10000

12000

14000

GroundTruth

Prediction

FEDformer

0

10

20

30

40

50

4000

6000

8000

10000

12000

14000

GroundTruth

Prediction

Dlinear

C Dataset Details

A detailed description of the dataset is given in Table 6.

Since our datasets setup is the same as Timesnet [13], an excerpt from it describes the datasets.

D Experimental Details

All the deep learning networks are implemented in PyTorch and trained on NVIDIA 4090 24GB GPU. We repeated each experiment three times to eliminate randomness. The detailed experiment configuration is shown in Table 7.

D.1 Metrics

We utilize various metrics to evaluate different tasks. For long-term forecasting and imputations, we employ the mean square error (MSE) and mean absolute error (MAE). In anomaly detection, we utilize the F1-score, which combines precision and recall. For short-term forecasting, we utilize the symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE), and overall weighted average (OWA), with OWA being a unique metric used in the M4 competition.

These metrics are computed as follows:

where q is the periodicity of the data. X , ̂ Y ∈ R T × C are the ground truth and prediction result of the future with T time pints and C dimensions. X i means the i -th future time point.

E Model Analysis

E.1 Hyper Parameter Analysis and Model Limitations

In Equation (1), we introduced a hyperparameter k to select the most important frequency, which also determines the number of levels in the Periodic Pyramid. We conducted sensitivity analysis on it, as shown in Figure 12. It's evident that our proposed Peri-midFormer exhibits relatively stable performance across different choices of k for all four tasks. However, there are still some fluctuation in results among different k values depending on the task and dataset, which are determined by the periodic characteristics in the dataset. To illustrate this, we visualize individual data for long-term forecasting and classification tasks, as shown in Figure 13. It can be observed that in the long-term forecasting task, the Etth1 dataset exhibits clear periodicity. Therefore, with larger k , Peri-midFormer can capture more periodic information, resulting in better performance. In contrast, the Etth2 dataset has less obvious periodicity, so it achieves better results with smaller k , as larger k introduce unnecessary noise, affecting model performance. In the classification task, the EthanolConcentration dataset is difficult to classify, whereas a larger k allows for the construction of Periodic Pyramid with more levels, thus extracting more representative features and achieving higher accuracy. In addition, the SelfRegulationSCP1 dataset contains a lot of high-frequency noise, so larger k would focus on irrelevant information, leading to decreased accuracy. Therefore, we adjusted k values differently for different tasks and datasets, as shown in the range in Table 7.

The above analysis reveals that the limitation of Peri-midFormer lies in its inability to fully leverage its advantages on datasets with poor periodicity characteristics. We plan to address this issue in our future work.

Imputation of 50% masked

2

3

4

5

6

k value

0.030

0.035

0.040

0.045

0.050

0.055

0.060

0.065

MSE

ETTh2

ETTm1

Weather

Long-term Forecasting of 96 prediction length

2

3

4

5

6

k value

0.15

0.20

0.25

0.30

0.35

0.40

MSE

ETTh1

ETTh2

ETTm1

ETTm2

Weather

Anomaly Detection

2

3

4

5

6

k value

67.5

70.0

72.5

75.0

77.5

80.0

82.5

85.0

F1-Score

MSL

SMD

SMAP

Classification

2

3

4

5

6

7

8

k value

30

40

50

60

70

80

90

100

Accuracy(%)

EthanolConcentration

SelfRegulationSCP1

SpokenArabicDigits

UWaveGestureLibrary

(1) Long-term Forecasting task

0

100

200

300

400

500

3

2

1

0

1

2

Etth1 dataset

0

100

200

300

400

500

4

2

0

2

4

6

Etth2 dataset

(2) Classification task

0

250

500

750

1000

1250

1500

1750

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

EthanolConcentration dataset

0

200

400

600

800

1

0

1

2

3

4

SelfRegulationSCP1 dataset

E.2 Periodic Pyramid Attention Mechanism Analysis

To illustrate the PPAM more clearly, we visualize the original time series and attention scores within the periodic pyramid for the SelfRegulationSCP2 dataset in the classification task, as shown in Figure 14. It is evident that the attention scores among components are distributed based on inclusion and adjacency relationships, meaning that components in different levels with inclusion relationships or those in the same level have higher attention scores. This demonstrates the rationality of the Periodic Pyramid structure. Additionally, when the hyperparameter k in Equation (1) is set to 2, the PPAM degenerates into periodic full attention, wherein attention is computed among all periodic

0

1

2

3

4

5

6

Original time series

Periodic pyramid

Attention score

6

0

5

4

3

2

1

0

0

1

2

3

4

5

6

0.6

0.4

0.2

0

-0.2

-0.4

-0.6

-0.8

S9

0.0

0.5

1.0

-1.5

200

400

600

008

1000

1200

0

100

200

300

400

500

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0

2

4

6

8

10

12

14

16

18

20

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

60

40

20

0

20

40

60

components. To illustrate this, we visualize the Electricity dataset in the long-term forecasting task and its corresponding attention distribution, as shown in Figure 15. The figure shows that the Periodic Attention Mechanism can capture the dependencies among the periodic components and identify which components belong to a longer component (note that k = 2 corresponds to 21 periodic components with larger amplitudes). This is attributed to the separation of the periodic components, which explicitly expresses the hidden periodic inclusion relationships in the time series. The above analysis demonstrates the effectiveness of the PPAM.

E.3 Periodic Feature Flows Analysis

To intuitively understand the Periodic Feature Flows, we visualize it as shown in Figure 16. On the left side is the pyramid representation of the Periodic Feature Flows, with the horizontal axis representing the number of feature flows and the vertical axis representing the length of each feature flow. The feature flows are divided into multiple levels, each containing multiple periodic components. The red box encloses one feature flow, with its position from top to bottom corresponding to the pyramid from top to bottom. It can be observed that some adjacent feature flows are the same at the corresponding positions, this is because they pass through the same periodic component. On the right side, the waveform of each feature flow is displayed in different colors, with the position from left to right corresponding to the top to the bottom of the pyramid. It can be observed that each feature flow differs, which is why it is necessary to aggregate different feature flows. The aim is to fully utilize the information from each periodic component to better reconstruct the target sample.

E.4 Training/Inferencing Cost

To further validate the computational complexity and scalability of the proposed method, we conducted detailed experiments on Electricity and ETTh1 datasets for the long-term forecasting task. These experiments focused on the complexity and actual time of training and inference, as well as memory usage, with results presented in Tables 8 and 9. It can be seen that our proposed Peri-

0

20

40

60

80

100

120

2

1

0

1

2

midFormer demonstrates a significant advantage in computational complexity on the Electricity dataset and achieves the lowest MSE. Similarly, on the ETTh1 dataset, Peri-midFormer's computational cost and inference time are not disadvantages. Instead, it achieves a lower MSE, second only to Time-LLM, while having much lower computational cost and inference time compared to it. These results highlight the advantages of our method in terms of computational complexity and scalability.

E.5 Pre-interpolation

Since Peri-midFormer is designed to focus on the periodic components of time series, directly handling data with missing values in imputation tasks may prevent it from correctly capturing the periodic characteristics of the original data without missing values, thus affecting its imputation effectiveness. Therefore, to adapt Peri-midFormer for imputation tasks, we first apply a linear interpolation strategy (Equation (13)) to the data with missing values to partially restore the periodic characteristics of the original data before inputting it into Peri-midFormer for further imputation. The visualization of original data, data with 50% missing values, and pre-interpolated data of Electricity dataset are illustrated in Figure 17. It is evident that missing values significantly disrupt the periodic

(1) Data 1

0

20

40

60

80

1.25

1.00

0.75

0.50

0.25

0.00

0.25

Original data

0

20

40

60

80

1.2

1.0

0.8

0.6

0.4

0.2

0.0

Data with 50% missing values

0

20

40

60

80

1.2

1.0

0.8

0.6

0.4

0.2

0.0

Pre-interpolated data

(2) Data 2

0

20

40

60

80

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Original data

0

20

40

60

80

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Data with 50% missing values

0

20

40

60

80

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Pre-interpolated data

(3) Data 3

0

20

40

60

80

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

Original data

0

20

40

60

80

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

Data with 50% missing values

0

20

40

60

80

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

Pre-interpolated data

characteristics of the original data, while pre-interpolation partially restores them. It is worth noting that this is not a speculative action but an effort to further explore the potential of deep learning models in imputation tasks. Our goal is for deep models to capture subtle variations in time series to achieve imputation in the details rather than wasting effort on goals achievable through simple linear interpolation. Additionally, since we can retain the indices of missing values in practical applications, there is no concern about the reliability of evaluating the imputation results. To validate the effectiveness of the pre-interpolation strategy, we applied it to other methods, as shown in Table 10. We conducted experiments on four mask ratios { 0 . 125 , 0 . 25 , 0 . 375 , 0 . 5 } , where the first row uses only the pre-interpolation strategy. From the results, it can be seen that pre-interpolation significantly improves the performance of each method across all four mask ratios, demonstrating its importance for deep learning-based methods in interpolation tasks.

The simple linear interpolation strategy we adopted can be represented by the equation:

̸

̸

̸

where x inter represents the value used to replace the 0 value, x before represents the nearest non-zero value before the 0 value, and x after represents the nearest non-zero value after the 0 value.

̸

100

200

300

400

500

600

700

800

900 1000

look-back window length

0.14

0.16

0.18

0.20

0.22

0.24

MSE

96

192

336

720

1000

E.6 Look-back window length

Here, we investigate the performance of Peri-midFormer under different look-back window lengths in long-term forecasting task on the Electricity dataset. As shown in Figure 18, we experimented not only with prediction lengths of { 96 , 192 , 336 , 720 } , but also with longer lengths such as 1000. It can be observed that there is a clear improvement in performance across all five prediction lengths as the look-back window length increases. This improvement is particularly notable for longer prediction lengths such as 336, 720, and 1000. It indicates that Peri-midFormer can effectively utilize more historical information, as longer look-back windows contain more stable periodic characteristics, which is precisely what Peri-midFormer requires.

E.7 Time Series Decomposition Analysis

Peri-midFormer performs well on the Exchange dataset for long-term forecasting task and the Yearly dataset for short-term forecasting. Although these two datasets lack obvious periodicity, they exhibit strong trends, as shown in Figure (19)). Peri-midFormer uses a time series decomposition strategy, where the trend part is first separated from the original data before dividing the periodic components. After the output of Peri-midFormer, the predicted trend part is added back, as illustrated in Figure (8). The separated trend part is easier to predict, especially when the trend is strong. This is why Peri-midFormer achieves strong performance on the Exchange and Yearly datasets.

Exchange

0

10

20

30

40

50

60

1

0

1

2

0

10

20

30

40

50

60

1

0

1

2

0

10

20

30

40

50

60

0.5

0.0

0.5

1.0

1.5

Yearly

0

2

4

6

8

10

8000

9000

10000

11000

12000

13000

14000

15000

0

2

4

6

8

10

2500

3000

3500

4000

4500

0

2

4

6

8

10

4000

4250

4500

4750

5000

5250

5500

F Proof

To demonstrate the essence of attention computation among multi-level periodic components, we need to analyze how the interactions between periodic components at different levels affect the final feature extraction. In time series analysis, different periodic components correspond to different time scales. This means that through decomposition, we can capture components of various frequencies within the time series. The essence of the periodic pyramid is to capture these different frequency components through its hierarchical structure.

Using single-channel data as an example, and given that we adopt an independent channel strategy, this can be easily extended to all channels. Assume the time series x ( t ) can be decomposed into multiple periodic components x n ( t ) :

Taking two different periodic components as examples:

where A is amplitude, T is period, and ϕ is phase. Due to the overlap and inclusion relationships between different periodic components, we employ an attention mechanism in the periodic pyramid to capture the similarities between different periodic components, focusing on important periodic features. When applying the attention mechanism, we have:

where W Q , W K and W V are learnable weight matrices. From Equations (15) and (16):

Further, the dot-product attention can be expressed as:

Using the trigonometric identity sin( a ) cos( b ) = 1 2 [sin( a + b ) + sin( a -b )] , the dot-product Q i K T j can be further expressed as:

Based on this, considering the periodicity and symmetry of sin( a + b ) and sin( a -b ) , when the periods of two time series components are close / same (intra-layer attention in the pyramid, see the right side of Figure 3 in the original paper) or have overlapping / inclusive parts (inter-layer attention in the pyramid, see the right side of Figure 3 in the original paper), the values of these two sine functions will be highly correlated, resulting in a large Q i K T j value. This indicates that the periodic pyramid model can effectively capture similar periodic patterns across different time scales.

Next, incorporating this into the calculation of the attention score:

It can be seen that the attention scores between highly correlated periodic components will be higher, which we have already validated in Figures 13 and 14 of the original paper.

Further, the attention vector a i of x i ( t ) can be obtained as:

From the above derivation, it can be seen that the attention mechanism can measure the similarity between different periodic components. This similarity reflects the alignment between different periodic components in the time series, allowing the model to capture important periodic patterns. By capturing these periodic patterns, the periodic pyramid can extract key features of the time series, resulting in a comprehensive and accurate time series representation. This representation not only includes information across different time scales but also enhances the representation of important periodic patterns.

G Broader Impacts

Our research has implications for time-series-based analyses such as weather forecasting and anomaly detection for industrial maintenance, as discussed in the Introduction 1. Our work carries no negative social impact.

H Full Results

H.1 Full Results of Classification (Table 11)

H.2 Full Results of Short-term Forecasting (Table 12)

H.4 Full Results of Imputation (Table 15)

H.5 Full Results of Anomaly Detection (Table 16)

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We made our main claims in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We illustrate the limitations of the model in conjunction with the hyperparametric analysis in the Appendix E.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a full theoretical description of the proposed methodology in the main paper and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the proposed methodology in detail in Section 3 and provide further explanation in Appendix A. Experimental details are outlined in Appendix D. Additionally, all the code related to the proposed method is included in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at https://github.com/WuQiangXDU/Peri-midFormer .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We offer a detailed description of the dataset in Appendix C and outline the experimental setup in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Appendix D, we demonstrate that the experiments were repeated multiple times to eliminate randomness. However, due to the large number of experiments, detailed standard deviations are not shown. Instead, we uniformly present them in the headings of each table in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe the compute workers required for the experiments in Appendix D, and the amount of computation in Section 6 and Appendix E.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and conducted it in the paper conform in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We illustrate broader impacts of our research in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all the datasets and models involved in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code is available at https://github.com/WuQiangXDU/Peri-midFormer .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Peri-midFormer to capture multi-periodicity in time series data. Specifically, it designs a pyramid structure and attention mechanisms to effectively model complex temporal variations. The proposed methods demonstrates great performance in several time series analysis tasks in the author's experiments.\n\nSTRENGTHS:\n1. This paper compares a variety of cutting-edge methods.\n2. Overall, this paper is solid and the authors conducted thorough experiments.\n\nWEAKNESSES:\nNone.\n\nQUESTIONS:\n1. Were the results of DLinear and PatchTST obtained with a lookback of 96? From the results in Table 12, it seems they were achieved with the default input of 336. Correspondingly, your assertion on line 236 is incorrect. As far as I know, DLinear, PatchTST, and FITS do not use a default input length of 96. FITS has a default lookback of 720.\n2. Regarding the Anomaly Detection experiment, did you use point adjustment techniques and manual thresholds? Combining point adjustment techniques with very low manual thresholds (e.g., 0.5%) can easily achieve an F1 score greater than 90 on these datasets [1], even with a randomly generated anomaly score list. Please do not simply state that you followed previous work, as this only exacerbates the legacy issue. Clearly state your stance and clarify this issue appropriately in the paper.\n3. Was the ablation study in Table 5 conducted only on the ETTh2 dataset? As far as I know, the ETTh2 dataset is small in scale, has relatively weak periodicity (compared to ETTh1, or multi-period datasets like Electricity and Traffic), and suffers from significant distribution drift. Under such circumstances, any hyperparameter adjustment or other random factors can greatly affect model performance.\n4. The Exchange dataset belongs to the financial domain and typically exhibits weak periodicity (due to the unpredictability of financial data). This is evidenced by findings in the DLinear paper, where simply copying the last point can achieve SOTA performance. This contradicts two claims in your paper: (1) Peri-midFormer improves predictive accuracy by extracting periodicity, so how did it achieve SOTA performance on this weak-period dataset? (2) Your Limitation section mentions that Peri-midFormer is not good in scenarios with weak periodicity, yet your experimental results show that your method performs well in this scenario.\n\n[1] Wagner, Dennis, et al. \"Timesead: Benchmarking deep multivariate time-series anomaly detection.\" Transactions on Machine Learning Research (2023).\n\nLIMITATIONS:\nAs mentioned above, the authors have already discussed the limitations, though I have some concerns for it.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a new method Peri-midFormer, which uses the multi-periodicity of time series and modeling the periodic part of a time series in a pyramid way. They further proposed an attention mechanism to use the neighborhood relation in the pyramid. Extensive experiments on different tasks show the effectiveness of the proposed method.\nAfter reading the rebuttal, I raise my rating from 5 to 6.\n\nSTRENGTHS:\n- The idea of using attention in the pyramid structure seems to be novel.\n- Extensive experiments are conducted for different tasks on benchmark datasets.\n- The proposed model is light-weight.\n\nWEAKNESSES:\n- Some important related works are missing. \n  - There are other related works utilizing the idea of modeling time series in multi-scale, e.g. [1] [2] [3].\n- Experiments could be improved. \n  - The authors should clearly state whether they reproduce the results for comparison methods or they copy the numbers from the paper.\n  - In the original paper of PatchTST, the authors use a longer context window than 96. The authors are suggested to tune this parameter for all comparison methods (as many papers did) and provide the best results, rather than fix the context window. \n  - The authors should provide the script for hyper-parameter tuning to improve the reproducibility of the paper.\n  - The ablation study (Table 5) is only shown on one dataset (ETTh2). The authors are suggested to include results in more datasets.\n  - The training and inference complexity and actual time could be analyzed, as well as memory efficiency. \n[1] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. ICLR 2024.\n[2] PERIODICITY DECOUPLING FRAMEWORK FOR LONGTERM SERIES FORECASTING. ICLR 2024.\n[3] Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting. TKDE 2024.\n\nQUESTIONS:\n- Apparently, the proposed method focused on periodic part forecasting. However, it works pretty well on the non-periodic forecasting task (short-term forecasting). For instance, there is no periodic in the 'Yearly' dataset, the proposed method still performs quite well. Can authors explain the possible reasons?\n- It is not clear why the proposed methods have a smaller number of parameters than PatchTST. Can the authors explain more?\n- It seems that the trend part is not used for classification tasks (from Figure 7). Can the author explain more?\n\nLIMITATIONS:\nFocused only on periodic signal", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Peri-midFormer, a novel transformer-based architecture designed for time series analysis. \nBy leveraging the multi-periodicity inherent in time series data, the model constructs a Periodic Pyramid structure that decouples complex periodic variations into inclusion and overlap relationships among different periodic components. \nThe proposed method incorporates self-attention mechanisms to capture dependencies between these periodic components, achieving state-of-the-art performance across five mainstream time series tasks: short- and long-term forecasting, imputation, classification, and anomaly detection.\n\nSTRENGTHS:\nS1.  The concept of decoupling time series data into a Periodic Pyramid is a valid point. \nThis new representation seems to capture the multi-periodicity of time series effectively.\n\nS2. The effectiveness of the proposed method is extensively verified on five tasks.\n\nWEAKNESSES:\nW1. While the Periodic Pyramid and self-attention mechanisms are well-explained, the model's complexity might pose challenges for practical implementation and scalability, especially for users with limited computational resources.\nIt would be useful if the authors could report in the main body of the paper, at least the main conclusion of the training and inference time for the proposed method, compared with existing baselines, across the five tasks.\n\nW2. The improvements provided by Peri-midFormer seem to be insignificant when compared with Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks.\nIt would be better if the author could further discuss this in the main body of the paper (e.g., discuss the time-accuracy tradeoff as shown in Appendix E.4 Complexity Analysis).\n\nQUESTIONS:\nQ1 (cr. W1). Add in the main body of the paper a discussion about the training and inference time, as well as some critical results and main conclusions.\n\nQ2 (cr. W2). Add in the main body of the paper a discussion about the significance of the improvements or about the time-accuracy tradeoff for Peri-midFormer against Time-LLM and GPT4TS on forecasting, imputation, and anomaly detection tasks.\n\nLIMITATIONS:\nL1. It is not clear how the proposed periodic pyramid can be effectively and efficiently integrated into multi-dimensional time series.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe abstract succinctly introduces the Peri-midFormer, a novel approach designed for time series analysis, acknowledging the challenges posed by the discrete nature of time series data and the complexity of capturing periodic variations directly. It proposes a method to address these challenges by decomposing complex periodic variations into hierarchical periodic components, termed the periodic pyramid. This approach leverages inclusion and overlap relationships among these components, mimicking the natural pyramid structure observed in time series data.\n\nSTRENGTHS:\nInnovative Approach: The concept of a periodic pyramid to model time series data is innovative and promises to address the limitations of traditional methods that struggle with capturing complex periodic patterns.\n\nHierarchical Representation: By representing time series as a pyramid with progressively shorter periodic components, the model potentially enhances the understanding of multi-scale temporal relationships.\n\nSelf-Attention Mechanism: Incorporating self-attention into the periodic pyramid allows capturing intricate relationships among periodic components, which is crucial for tasks like anomaly detection and forecasting.\n\nWEAKNESSES:\nComplexity and Scalability: The introduction of a hierarchical pyramid structure combined with self-attention could potentially introduce computational complexities and scalability issues, especially with larger datasets or real-time applications. Addressing these concerns in the paper would strengthen its practical utility.\n\nState-of-the-Art Comparison: It is not explicitly stated whether Peri-midFormer achieves state-of-the-art (SOTA) performance when compared to strong baseline models. Without clear comparative results, it is difficult to ascertain if the proposed method truly represents an advancement over current leading approaches in time series analysis.\n\nInterpretability: The abstract lacks interpretability regarding why the periodic pyramid structure exists in the applications and why it plays a critical role in improving forecasting. Providing a rationale or theoretical justification for the efficacy of the pyramid structure in capturing temporal patterns would enhance the understanding and acceptance of the proposed method.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}