Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting
Authors: Peng Chen
Yingying ZHANG
Yunyao Cheng
Yang Shu
Yihang Wang
Qingsong Wen
Bin Yang
Chenjuan Guo
Venue: ICLR 2024 poster (2024)
Keywords: Time series
Transformer
Multi-scale
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: In this paper, we propose Multi-Scale Transformers with Adaptive Pathways (Pathformer) for time series forecasting.

Abstract: Transformers for time series forecasting mainly model time series from limited or fixed scales, making it challenging to capture different characteristics spanning various scales. We propose Pathformer, a multi-scale Transformer with adaptive pathways. It integrates both temporal resolution and temporal distance for multi-scale modeling. Multi-scale division divides the time series into different temporal resolutions using patches of various sizes. Based on the division of each scale, dual attention is performed over these patches to capture global correlations and local details as temporal dependencies. We further enrich the multi-scale Transformer with adaptive pathways, which adaptively adjust the multi-scale modeling process based on the varying temporal dynamics of the input, improving the accuracy and generalization of Pathformer. Extensive experiments on eleven real-world datasets demonstrate that Pathformer not only achieves state-of-the-art performance by surpassing all current models but also exhibits stronger generalization abilities under various transfer scenarios. The code is made available at  https://github.com/decisionintelligence/pathformer.

Published as a conference paper at ICLR 2024
PATHFORMER: MULTI-SCALE TRANSFORMERS WITH ADAPTIVE PATHWAYS FOR TIME SERIES FORECASTING
Peng Chen',, Yingying Zhang", Yunyao Cheng", Yang Shu't, Yihang Wang"", Qingsong Wen Bin Yang', Chenjuan Guo
East China Normal University, 2 AAlibaba Group, 3 'Aalborg University fpchen, yhwang)estu.
eenuuuuuuuuuuuu..e.......uuuuu ecnu. edu. en, congrong. zyy eeyyaaliabbe ibaba-inc. com (yshu, siguo, byang) @ayangeddase...m @dase. ecnu. ecnu.edu. en, yunyaocêss aau.dk Binggonngeuu@gall com
ABSTRACT
Transformers for time series forecasting mainly model time series from limited fixed scales, making it challenging to capture different characteristics spanning various scales. We propose Pathformer, a multi-scale Transformer with adap- tive pathways. It integrates both temporal resolution and temporal distance for multi-scale modeling. Multi-scale division divides the time series into different temporal resolutions using patches of various sizes. Based on the division of each scale, dual attention is performed over these patches to capture global correla- tions and local details as temporal dependencies. We further enrich the multi- scale Transformer with adaptive pathways, which adaptively adjust the multi-scale modeling process based on the varying temporal dynamics of the input, improving the accuracy and generalization of Pathformer. Extensive experiments on eleven real-world datasets demonstrate that Pathformer not only achieves state-of-the-arr performance by surpassing all current models but also exhibits stronger general- ization abilities under various transfer scenarios. The code is made available at nttps://github.. deci/eccisssse htt.s//www...co//..cccoooooneene httpe//eeioiiiileeeeeeeeeeeeeaee
INTRODUCTION
Time series forecasting is essential function for various industrics, such as energy, finance, traffic, logistics, and cloud computing (Chen et al. 2012 Cirstea et al., 2022b; Ma et al. 2014 Zhu et al, 2023 Pan et al. 2023 Pedersen al. 2020), and is also foundational building block for other time series analytics ytics, e.g., outlier detection Campos et al. (2022): : Kieu et al.] (2026)). Motivated by its widespread application in sequence modeling and impressive success various fields such as CV and NLP (Dosovitskiy et al., 2021 Brown et al. 2020), Transformer (Vaswani et al. 2017) receives emerging attention in time series (Wen et al. [2023] WW et al., 2021 Chen et al., 2022
Liu
et al. 2022c). Despite the growing performance, recent works have started to challenge the existing designs of Transformers for time series forecasting by proposing simpler linear models with better performance (Zeng et al., 2023). While the capabilities of Transformers are still promising in time series forecasting (Nie et al. 2023), it calls for better designs and adaptations to fulfill its potential.
Real-world time series exhibit diverse variations and fluctuations at different temporal scales. For instance, the utilization CPU, GPU, and memory resources in cloud computing reveals unique temporal patterns spanning daily, monthly, and seasonal scales Pan et al. (202)). This calls for multi-scale modeling (Mozer 1991 Ferreira et al., (2066) for time series forecasting, which extracts temporal features and dependencies from various scales of temporal intervals. There are two aspects to consider for multiple scales in time series: temporal resolution and temporal distance. Temporal resolution corresponds how view the time series in the model and determines the length of each temporal patch or unit considered for modeling. In Figure I the same time series can be divided
Part of the work was done during the internship at Alibaba Group. "orresponning author

---

Published as a conference paper at ICLR 2024
patch
Patall R
A4/
Time
M
Patcha
Time
Figure 1:
Left: Time series are divided into patches of varying sizes as temporal resolution. The intervals in blue, orange, and red represent different patch sizes, Right: Local details (black arrows) and global correlations (color arrows) are modeled through different temporal distances.
into small patches (blue) or large ones (yellow), leading to fin-graiied or coarse- goanse-rained temporal characteristics. Temporal distance corresponds to how we explicitly model temporal dependencies and determines the distances between the time steps considered for temporal modeling. In Figure I the black arrows model the relations between nearby time steps, forming local details, while the colored arrows model time steps across long ranges, forming global correlations.
To further explore the capability of extracting correlations in Transformers for time series forecast- ing, in this paper, we foeus on the aspect of enhancing multi-scale modeling with the Transformer architecture. Two main challenges limit the effective multi-scale modeling in Transformers. The first challenge is the incompleteness of multi-scale modeling. Viewing the data from different tem- poral resolutions implicitly influences the scale of the subsequent modeling process (Shabani et al.) 2023). However, simply changing temporal resolutions cannot emphasize temporal dependencies in various ranges explicitly and efficiently. On the contrary, considering different temporal dis- tances enables modeling dependencies from different ranges, such as global and local correlations (Li et al. 2019). However, the exact temporal distances of global and local intervals are influenced by the division of data, which is incomplete from a single view of temporal resolution. The sec- ond challenge is the fixed multi-scale modeling process. Although multi-scal modeling reaches more complete understanding of time series, different series prefer different scales depending on their specific temporal characteristics and dynamics. For example, comparing the series in Fig- ure [ the series above shows rapid fluctuations, which may imply more attention to fine-grained and short-term characteristics. The series below, on the contrary, may need more focus on coarse- grained and long-term modeling. The fixed multi-scale modeling for all data hinders the grasp of critical patterns of each time series, and manually tuning the optimal scales for a dataset or each time series is time-consumin or intractable. Solving these two challenges calls for adaptive multi-scale modeling, which adaptively models the current data from certain multiple seales.
Inspired by the above understanding of multi-scale modeling, we propose Multi-scale Transformers with Adaptive Pathways (Pathformer) for time series forecasting. To enable the ability of more complete multi-scale modeling, we propose multi-scale Transformer block unifying multi-scale temporal resolution and temporal distance. Multi-scale division is proposed to divide the time series into patches of different sizes, forming views of diverse temporal resolutions. Based on each size of divided patches, dual attention encompassing inter-patch and intra-patch attention is proposed to capture temporal dependencies, with inter-patch attention capturing global correlations across patches and intra-patch attention capturing local details within individual patches. We further pro- pose adaptive pathways activate the multi-scale modeling capability and endow it with adaptive modeling characteristics. At each layer of the model, a multi-scale router adaptively selects specific sizes of patch division and the subsequent dual attention in the Transformer based on the input data, which controls the extraction of multi-scale characteristics. We equip the router with trend and sea- sonality decomposition to enhance its ability to grasp the temporal dynamics. The router works with an aggregator to adaptively combine multi-scale characteristics through weighted aggregation. The layer-by-layer routing and aggregation form the adaptive pathways of multi-scale modeling through- out the Transformer. To the best of our knowledge, this is the first study that introduces adaptive multi-scale modeling for time series forecasting. Specifically, we make the following contributions:
We propose multi-scale Transformer architecture. It integrates the two perspectives of temporal resolution and temporal distance and equips the model with the capacity of a more complete multi-scale time series modeling.
2

---

Published as a conference paper at ICLR 2024
We further propose adaptive pathways within multi-scale Transformers. The multi-scale router with temporal decomposition works together with the aggregator to adaptively ex- tract and aggregate multi-scale characteristics based on the temporal dynamics of input data, realizing adaptive multi-scale modeling for time series.
We conduct extensive experiments on different real-world datasets and achieve state-of- the-art prediction accuracy. Moreover, we perform transfer learning experiments across datasets to validate the strong generalization of the model.
2 RELATED WORK
Time Series Forecasting. Time series forecasting predicts future observations based on historical observations. Statistical modeling methods based on exponential smoothing and its different flavors serve as reliable workhorse for time series forecasting (Hyndman & Khandakar] 2008 Li et al.)
2022a). Among deep learning methods, GNNs model spatial dependency for correlated time series
forecasting (Jin et al. 2023a Wu et al. [0200 Zhao et al. 2024) Cheng et al. 2024 Miao et al. 2024
Cirstea et al. 202 1). RNNs model the temporal dependency (Chung et al. 2014]] Kieu et al.) 2022a
Wen et al.) 2017 Cirstea et al. 2019). DeepAR (Rangapuram et al. 2018) uses RNNs and autore- gressive methods to predict future short-term series. CNN models use the temporal convolution to
extract the sub-series features (Sen et al. 2019; Liu et al. 2022a Wang et al., 2023). TimesNet (Wu et al.] 2023a) transforms the original one- din-dimensional time series into two -wo-dimensional space and captures multi-period features through convolution. LLM--asedd methods also show effective performance in this field (Jin et al., [2023b; /Zhou et al.) 2023). Additionally, some methods are incorporating neural architecture search to discover optimal architectures WW et al., 2022] 2023b).
Transformer models have recently received emerging attention in time series forecasting (Wen et al., 2023). Informer (Zhou al., (2021) proposes prob-sparse self-attention select important keys, Tnformer (Cirstea et al., 2022a) employs a triangular architecture, which manages to reduce the complexity. Autoformer (Wu al.] 202I) proposes auto-correlation mechanisms to replace self- attention for modeling temporal dynamics. FEDformer (Zhou et al.) 2022) utilizes fourier transfor- mation from the perspective of frequency to model temporal dynamics. However, researchers have raised concerns about the effectiveness of Transformers for time series forecasting, simple linear models prove to be effective or even outperform previous Transformers (Li et al. 2022al Challu et al., 2023 Zeng al., 2023). Meanwhile, PatchTST (Nie et al. 2023) employs patching and channel independence with Transformers to effectively enhance the performance, showing that the Transformer architecture still has its potential with proper adaptation in time series forecasting.
Multi-scale Modeling for Time Series. Modeling multi-scale characteristics proves to be effective for correlation learning and feature extraction in the fields such as computer vision (Wang et al. 2021 Li et al. 2022b; Wang et al., (022b) and multi-modal learning (Hu et al. (2020) Wang et al. 2022a), which relatively less explored in time series forecasting. N-HiTS (Challu al. 2023 employs multi-rate data sampling and hierarchical interpolation to model features of different reso- lutions. Pyraformer (Liu et al. 20226) introduces pyramid attention extract features different temporal resolutions. Scaleformer (Shabani et al. 2023) proposes a multi-scale framework, and the need to allocate a predictive model different temporal resolutions results in higher model com- plexity. Different from these methods, which use fixed scales and cannot adaptively change the multi-scale modeling for different time series, we propose multi-scale Transformer with adaptive pathways that adaptively model multi-scale characteristies based on diverse temporal dynamics.
3 METHODOLOGY
To effectively capture multi-scale characteristics, we propose multi-scale Transformers with adap- tive pathways (named Pathformer). As depicted in Figure 2, the whole forecasting network is com- posed of Instance Norm, stacking of Adaptive Multi-Scale Blocks (AMS Blocks), and Predictor. Instance Norm (Kim et al.) 2022) is a normalization technique employed to address the distribution shift between training and testing data. Predictor is a fully connected neural network, proposed due
to its applicability to forecasting for long sequences (Zeng al. 2023 Das et al. 2023).
The core of our design is the AMS Block for adaptive modeling of scati-scale characteristics, which consists of the multi-scale Transformer block and adaptive pathways. Inspired by the idea of patch-
3

---

Published as a conference paper at ICLR 2024
Predictor
weight
Multi-Scale Aggregator
Adaptive Multi-Scale
Multi-Scale Transformer Block
Block
Divsion
Dual Attention
Adaptive Multi-Scale
Block
patch X'
putch x"
Adaptive Multi-Scale
Patch Size Selection(Topk)
Block
PatchSize S
PatchSize S
Patch Sn
Instance Norm
Multi-Scale Router
Original Time Series
The architecture of Pathformer. The Multi-scale Transformer Block (MST Block) com- prises patch division with multiple patch sizes and dual attention. The adaptive pathways select the patch sizes with the top K weights generated by the router to capture multi-scal characteristics, and the selected patch sizes represented in blue. Then, the aggregator applies weighted aggregation to the characteristics obtained from the MST Block.
Figure 2:
ing in Transformers (Nie et al. 2023), ) the multi-scale Transformer block integrates multi-scale tem- poral resolutions and distances by introducing patch division with multiple patch sizes and dual attention on the divided patches, equipping the model with the capability comprehensively model multi-scale characteristics. Based on various options of multi-scale modeling in the Transformer block, adaptive pathways utilize the multi-scale modeling capability and endow it with adaptive modeling characteristics. A multi-scale router selects specific sizes of patch division and the sub- sequent dual attention in the Transformer based on the input data, which controls the extraction of multi-scale features. The router works with an aggregator to combine these multi-scale character- istics through weighted aggregation. The layer-by-layer routing and aggregation form the adaptive pathways of multi-scale modeling throughout the Transformer blocks. In the following parts, describe the multi-scale Transformer block and the adaptive pathways the AMS Block in detail.
3.1 MULTI-SCALE TRANSFORMER BLOCK
Multi-scale Division. For the simplicity of notations, we use a univariate time series for description, and the method can be easily extended to multivariate cases by considering each variable indepen- dently. In the multi-scale Transformer block, We define collection of M patch size values as -S...... Sar). }. with each patch size corresponding to a patch division operation. For the input time series X E IR Hxd where H denotes the length of the time series and d denotes the dimension of features, each patch division operation with the patch size divides X into (with P P=H H/S)) patches (X2, X" X²,..., X²)), where each patch X Sxd contains S time steps. Dif- ferent patch sizes in the collection lead to various scales divided patches and give various views of temporal resolutions for the input series. This multi-scale division works with the dual attention
mechanism described below for multi-scale modeling.
Dual Attention. Based on the patch division of each scale, propose dual attention to model temporal dependencies over the divided patches. To grasp temporal dependencies from different temporal distances, utilize patch division as guidance for different temporal distances, and the dual attention mechanism consists of intra-patch attention within each divided patch and inter-patch attention across different patches, shown in Figure
XP) divided with the patch size S, intra-patch attention establishes relationships between time steps within each patch. For the i-th patch X e Sxd we first embed the patch along the feature dimension d to get X X E Sxdm, where dm represents the dimension of embedding. Then we perform trainable linear transformations on X X obtain the key and value in attention operations, denoted Ki Kintra V query matrix Q 1xdd merge the context of the patch and subsequently compute the
Consider a set of patches
intra E Sxdm. We employ trainable
4

---

Published as a conference paper at ICLR 2024
0
Intra-patch attention
Inter-patch atlention
Routing
Rearing
Faina
f Flatten
Kosiiig Weights Generation
Intra-patc attention
Inter-puich attenrion
121
Frend
Self-Altention
Weighted Sum
Inirspptch embeddinng
Multi-kenne
embbedding
Lincar
Linzar
Linsorr
Cre pooling
Linrar
Lineer
Linsar
Patch Division
Inrlattre
K
Seasinaliity
Topk Frequency
Patch Size
P
Pancksize &
PatchSice
pulla X
patch.
putha
pukca2?
(a) Multi-Scale Transformer Block
(b) Multi-Scale Router
Figure 3: (a) The structure of the Multi-Scale Transformer Block, which mainly consists of Patch Division, Inter-patch attention, and Intra-patch attention. (b) The structure of the Multi-Scale Router.
cross-attention between Vintray K
intra
V Vittra to model local details within the i-th patch:
Attn Softmax(Qiin (intra (" / dm) Ví
intra
After intra-patch attention, each patch has transitioned from its original input length of to the length of 1. The attention results from all the patches are concatenated to produce the output of intra-attention on the divided patches as AttMljtta E RPdd,, which represents the local details from nearby time steps in the time series:
AttDintaa-
Concat(Attn/aara
Antna)tra)
Inter-patch attention establishes relationships between patches capture global correlations. For the patch-divided time series X E R Px5xx, we first perform feature embedding along the feature dimension from d dm and then rearrange the data to combine the two dimensions of patch quan-
tity S and feature embedding dm, resulting in Xinter RK Pxd where d'm S dmn After such embedding and rearranging process, the time steps within the same patch are combined, and thus we perform self-attention over Xinter model correlations between patches. Following the standard self-attention protocol, we obtain the query, key, and value through linear mapping on Xinters de-
noted as Qinter: Kinter Vinter E Pxdm, Then, we compute the attention AttD which involves interaction between patches and represents the global correlations of the time series:
AttD = Softmax(QQnter (oanterrKiite (Kinte)? |T/ // ) Vnter-
To fuse global correlations and local details captured by dual attention, we rearrange the outputs of intra-patch attention AttDjutra Px performing linear transformations size dimension from 1 to S, to combine time steps in each patch, and then add it with inter-patch patch attention Allilinte R PxSddm to obtain the final output of dual attention Attn E R PxSxdm..
the patch
Overall, the multi-scale division provides different views of the time series with different patch sizes, and the changing patch sizes further influence the dual attention, which models temporal dependencies from different distances guided by the patch division. These two components work together to enable multiple scales of temporal modeling in the Transformer.
3.2 ADAPTIVE PATHWAYS
The design of the multi-scale Transformer block equips the model with the capability of multi- scale modeling. However, different series may prefer diverse scales, depending on their specific temporal characteristics and dynamics. Simply applying more scales may bring in redundant or useless signals, and manually tuning the optimal scales for dataset each time series is time- consuming or intractable. An ideal model needs to figure out such critical scales based on the input data for more effective modeling and better generalization of unseen data.
5

---

Published as a conference paper at ICLR 2024
Pathways and Mixture of Experts are used to achieve adaptive modeling (Dean 2021 Shazeer et al, 2016). Based on these concepts, we propose adaptive pathways based on multi-scale Transformer to model adaptive multi-scale, depicted in Figure 2 It contains two main components: the multi-scale router and the multi-scale aggregator. The multi-scale router selects specific sizes of patch division based on the input data, which activates specific parts in the Transformer and controls the extraction of multi-scale characteristics. The router works with the multi-scale aggregator to combine these characteristics through weighted aggregation, obtaining the output of the Transformer block.
The multi-scale router enables data-adaptive routing in the multi-scale Trans- former, which selects the optimal sizes for patch division and thus controls the process of multi-scale modeling. Since the optimal or critical scales for each time series can be impacted by its complex inherent characteristics and dynamic patterns, like the periodicity and trend, we introduce temporal decomposition module in the router that encompasses both seasonality and trend decomposition to extract periodicity and trend patterns, as illustrated in Figure 3b)
Multi-Scale Router.
Seasonality decomposition involves transforming the time series from the temporal domain into the frequency domain to extract the periodic patterns. We utilize the Discern Fourier Transform (DFT) (Cooley & Tukey 1965), denoted as DFT(-), to decompose the input X into Fourier basis and select the Ky basis with the largest amplitudes to keep the sparsity of frequency domain. Then, we obtain the periodic patterns Xeea through an inverse DFT, denoted as IDFT(-). The process is as follows:
(4) where t and A represent the phase and amplitude of each frequency from (X) (fi...., fK, represents the frequencies with the top y amplitudes. Trend decomposition uses different kernels of average pooling for moving averages to extract trend patterns based on the remaining part after the seasonality decomposition Xrem = X - Xeea- For the results obtained from different kernels, a weighted operation is applied to obtain the representation of the trend component:
Xgea (DFT fk, 1,4,
(5) is the pooling function with the i-th kernel, N corresponds to the number of kernels, Softmax(L(()) controls the weights for the results from different kenerls. We add the seasonality pattern and trend pattern with the original input X, and then perform a linear mapping Linear() (-) to transform and merge them along the temporal dimension to get Xtrans E Rdd
Xtrend = Softmax (X..mm))
(Avgpool(Xren) (ermmm))eeee...... A Avgpool( (Xppool(premll (eem)eernllyy ):
where A A-gpool())
Based on the results Xtraus from temporal decomposition, the router employs routing function to generate the pathway weights, which determines the patch sizes to choose for the current data. To avoid consistently selecting few patch sizes, causing the corresponding scales to be repeatedly updated while neglecting other potentially useful scales in the multi-scale Transformer, we introduce noise terms add randomness in the weight generation process. The whole process of generating pathway weights is as follows:
R(Xarans) = Softmax (XtraxxXXtassst W, + e Softplus(X W ; ~ NO 1),
where R(.)) represents the whole routing function, W, and W Wnoise E RdxM are learnable parameters for weight generation, with d denoting the feature dimension of Xurans and M denoting the number of patch sizes. To introduce spansity in the routing and encourage the selection of critical scales, perform top& selection on the pathway weights, kceping the top K pathway weights and setting the rest weights as 0, and denote the final result as R(Xurans)
Multi-Scale Aggregator. Each dimension of the generated pathway weights R(Xerans) E R M cor- respond to patch size in the multi-scale Transformer, with R(Xuraus); > 0 indicating performing this size 5; of patch division and the dual attention and (((erras); = 0 indicating ignoring this patch size for the current data. Let X denote the output of the multi-scale Transformer with the patch size S,, due to the varying temporal dimensions produced by different patch sizes, the aggre- gator first perform a transformation function (-). to align the temporal dimension from different scales. Then, the aggregator performs weighted aggregation for the multi-scale outputs based on the pathway weights to get the final output of this AMS block:
Xout Σ I I(R(Xtrans)) > R))R(Xranna T Ti
I [(R(Xeran)) > 0) is the indicator function which outputs 1 when R(Xtrans); 0, and other- wise outputs 0, indicating that only the top K patch sizes and the corresponding outputs from the Transformer are considered or needed during aggregation.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 TIME SERIES FORECASTING
Datasets. We conduct experiments on nine real-world datasets to assess the performance of Path- former, encompassing a range of domains, including electricity transportation, weather forecasting, and cloud computing. These datasets include ETT (ETTh1, ETTh2, ETTml, ETTm2), Weather, Electricity, Traffic, ILI, and Cloud Cluster (Cluster-A, Cluster-B, B, Cluster-C).
Baselines and Metrics. We choose some state-of-the-an models to serve as baselines, including PatchTST (Nie et al. 2023). NLinear (Zeng et al. 2023), Scaleformer (Shabani et al.) 2023). TIDE
(Das al. 2023). 9, FEDformer (Zhou al. 2022). Pyraformer (Liu et | 2022b), and Autoformer 2021p. To ensure fair comparisons, all models follow the same input length (H = 36 for the ILI dataset and H 96 for others) and prediction length (F' E {24, 49, 96, 192} for Cloud Cluster datasets, F E {24, 36, 48, 60} for ILI dataset and F E {96, 192, 336, 720] for others). We select two common metrics in time series forecasting: Mean Absolute Error (MAE) and Mean Squared Error (MSE).
Wu et al.
Implementation Details. Pathformer utilizes the Adam optimizer (Kingma & Ba 2013) with a learning rate set at 10-3, The default loss function employed is L1 Loss, and we implement early stopping within 10 epochs during the training process. All experiments are conducted using Py PyTorch and executed on NVIDIA A800 80GB GPU. Pathformer is composed of 3 Adaptive Multi-Scale Blocks (AMS Blocks). Each AMS Block contains 4 different patch sizes. These patch sizes are selected from pool of commonly used options, namely {2, 6, 12, 16, 24.
Main Results. Table T shows the prediction results of multivariable time series forecasting, where Pathformer stands out with the best performance in 81 cases and the second-best in 5 cases out of the overall 88 cases. Compared with the second-best baseline, PStchTST, Pathformer demonstrates significant improvement, with impressive 8. 8.1% reduction in MSE and 6.4% reduction in MAE. Compared with the strong linear models NLinear, Pathformer also outperforms them com- prehensively, especially on large datasets such as Electricity and Traffic. This demonstrates the potential of Transformer architecture for time series forecasting. Compared with the multi-scale models Pyraformer and Scaleformer, Pathformer exhibits good performance improvements, with a substantial 36.4% reduction in MSE and a 19.1% reduction in MAE. This illustrates that the pro- posed comprehensive modeling from both temporal resolution and temporal distance with adaptive pathways is more effective for multi-scale modeling.
4.2 TRANSFER LEARNING
Experimental Setting. To assess the transferability of Pathformer, we benchmark it against three baselines: PStchTST, FEDformer, and Autoformer, devising two distinct transfer experiments. In the context of evaluating transferability across different datasets, models initially undergo pre-training on the ETTh and ETTml. Subsequently. fine-tune them using the ETTh2 and ETTm2. For assessing transfcrability towards future data, models are pre-trained on the first 70% of the training data sourced from three clusters: Cluster-A, Cluster-B, B, and Cluster-C. This pre-training is followed by fine-tuning the remaining 30% of the training data specific to each cluster. In terms of method- ology for basclines, we explore two approaches: direct prediction (zero-shot) and full-tuning. Devi- ating from these approaches, Pathformer integrates a part-tuning strategy. In this approach, specific parameters, like those of the router network, undergo fine-tuning, resulting in significant reduction in computational resource demands.
Transfer Learning Results. Table presents the outcomes of our transfer learning evaluation. Across both direct prediction and full-tuning methods, Pathformer surpasses the baseline models, highlighting its enhanced generalization and transferability. One of the key strengths of Pathformer lies in its adaptive capacity to select varying scales for different temporal dynamics. This adaptability allows it to effectively capture complex temporal patterns present in diverse datasets, consequently demonstrating superior generalization and transferability. Part-tuning is lightweight fine-tuning method that demands fewer computational resources and reduces training time on average by 52%, while still achieving prediction accuracy nearly comparable to Pathformer full-tuning. Moreover, it outperforms the full-tuning of other baseline models on the majority of datasets. This demonstrates that Pathformer can provide effective lightweight transfer learning for time series forecasting.
7

---

Published as a conference paper at ICLR 2024
Table 1: Multivariate time series forecasting results. The input length и = 96 (H = 36 for ILD).
The best results are highlighted in bold, and the second- best results are underlined.
Method
Pathformer
PSI
NLinear
Scaldormer
TIDE
PRilormur
Pyralormeer
Autedormer
MIE
MAE
MSE
M.NE
MSE
MSE
MAA
MAA
MSE
M.V
MSE
MAE
96
02382
014000
0.354
0.403
0.386
0.392
0.396
0.440
0.427
0.450
0.376
0.419
0.654
0.612
0.331
13:55
KTTh1
192
0:440)
0.427
0.446
0.438
0.440
0.430
0.434
0.460
0.472
0.486
0.429
0.448
0.790
0.681
0.500
0:442
336
01.454
01,32
0.485
0.455
0.480
0.443
0.462
0.476
0.527
0.527
0.459
0.465
0.891
0.738
0.521
0.44%
720
0.479
0.461
0.495
0.474
0.486
0.42
0.494
0.500
0.644
0.605
0.506
0.507
0.965
0.782
0.514
0.512
0.279
0.331
0.294
0.343
0.290
0.407
0.304
0.346
0.388
0.645
0.597
0.358
0.397
E1Th2
836
192
0.349
0.348
0.382
0.380
0.578
0.582
0.394
0.41D
0.421
0.379
0.395
0.431
0.166
0.479
0.476
0.158
0.385
0.391
0,422
0.411
0.429
0.496
0.439
0.487
0.788
0.907
0.683
0.747
0.156
0.482
0.452
0.486
720
0.398
0.424
0.412
0.433
0.436
0.453
0.487
0.492
0.463
0.475
0.463
0.474
0.963
0.783
0.515
0.511
0.316
0.346
0.554
0 1.36
0.386
0.379
0.419
0.545
0,510
0.505
0.475
KTTml
192
0.365
0.370
0.360
0.383
0.37
0.128
0.455
0
0.426
0.411
0.557
0.537
0.553
0.196
336
0.386
0.394
0.
0.500
0-402
0-111
0,407
0.524
0.487
0,424
0.433
0.445
0.459
0.754
0.655
0.621
0.537
720
0.460)
0.432
0.461
0.438
0.438
0,442
0.558
0.517
(1400
0.456
0.543
0.490
0.908
0.774
0.671
0.561
96
0.170
112248
011
0.700
0.177
0.257
0.182
0.275
1.182
0.264
0:211
0.287
0.435
0.507
0.255
0.333
ETTm2
192
0.238
0.295
0.245
0:306
0,241
0.25
0.156
0.323
0.209
0.328
0.730
0.673
0.281
0.340
0.293
0.331
0.304
0.302
0.375
0,313
0,354
0.325
0.366
1.201
0.845
0.339
0.372
720
0,390
0.389
0.403
0.387
0,405
0. 1,59
0.435
0.133
0,119
0,410
0.421
0.115
3,625
1.45)
0.133
0:432
96
0.156
0.192
0.177
0.218
0. 16668
0.206
02285
D.363
0.202
0.261
0.238
0.314
0.8%6
0.556
0.249
0.329
Weather
192
0.2006
1.240
0.224
0.358
0.217
0.255
0.368
0.425
0.242
0.298
0.275
0.329
0.622
0.624
0.325
0.370
336
0.254
0.282
0.257
0.297
0.267
0.469
0.335
0.339
0.377
0.759
0.753
0.351
0.391
720
0.340
0.336
0.350
0.345
0.574
0
0,386
0.389
0.409
1.004
0.334
0.415
0.426
96
0.145
0.236
0.180
0,264
0.885
0.266
0.182
0.297
0.194
0.277
0.186
0.302
0.386
0.449
0.196
0.313
Electricity
192
0,167
0.256
0.188
0.775
0.189
0.276
0.188
0.300
0.193
0.250
0.197
0.311
0.386
0.143
0.211
0.324
336
0,186
0.275
0 0.206
0. 0.391
0.
0.204
0
0.310
0.324
0.2006
0.296
0.213
0.328
0,578
0.443
0,214
0.327
720
0.231
0.309
0.247
0.328
0.245
0.319
0.32
0.339
0.242
0.328
0.233
0.344
0.376
0.445
0.236
0.342
1.587
0.755
1.224
0.44
2:225
1,000
2.154
0,992
2.624
1.095
1,421
2.012
2.906
1.182
II.1
48
36
1.429
1.505
0.742
0.711
1831
1546
0,553
0.33
2.500
2,300
1,031
1,032
2.745
2.748
1,025
1,072
2,532
2,436
1.051
1.042
2,516
2.505
1.021
1.041
7,551
7.394
2.057
2.031
2.595
3.004
1.038
1.145
60
1,731
0.799
1.933
0.443
2.402
1,426
2.793
0809
2.748
1.142
2.742
1.122
7.662
2,100
2.761
1.114
0.479
0.283
0.492
0,368
0.376
0.359
2,085
0.168
0.597
0.371
Truffic
192
0.484
0.292
0.487
0.303
0.612
0.571
0.610
0.380
0.867
0.467
0.607
0.382
336
0,503
0.299
0.505
0.317
0.606
0.367
0.570
0.349
0.605
0.374
0.608
0.375
0.869
0.469
0.623
0.387
720
0,537
0,323
0.542
0.337
0.645
0.388
0.576
0.349
0.647
0.400
0.621
0.375
0.881
0.473
0.639
0.395
24
0,1000
112015
0.126
0.234
0.134
0.235
0.128
0.347
0.128
0.44
0.13
0,260
0.131
0.288
0.172
0.461
Cluster-A
0.160)
0.264
0.208
0.214
0.175
0.170
0,311
0.390
0.471
96
0.227
0.321
0.313
0.372
0,335
0.378
0.247
0,338
0.293
0.349
0.245
0.375
0.466
0.514
192
0.349
0.400
0.452
0.453
0.442
0.452
0.322
0.451
0.256
0.422
0.350
0.439
0.378
0.437
0.585
0.584
24
0.121
0,224
0.126
0.237
0.130
0.241
0.125
0.241
0.128
0.40
0.128
0.243
0.129
0.263
0.242
0.369
Cluster-B
48
0.172
0.270
0,183
0.290
0,173
0,265
01661
0.280
0,165
0.258
0.156
0.267
0.168
0.296
0.299
0.425
0..42
0.322
0.272
1.244
0.277
0.389
0.315
0.4%6
0.366
0,471
192
0.437
0.427
0,476
0.461
0.479
0,456
0.447
0,452
0,467
0.414
0.478
0.389
0.485
0.597
0.563
24
0.1664
1.169
01075
1.
0.100
0.205
1174
0.214
12082
0.199
0.076
0.212
0.107
0.247
0.139
0.341
Cluster-C
48
0.102
0,218
0.118
0.21
0.163
0.286
0.100
0.242
0.121
0.266
0.108
0.246
0.142
0.284
0.210
0.363
96
0.162
11.77
0.188
0.365
0,245
0,318
0.177
0.321
0.201
0.305
0.171
0.323
0.181
0.328
0.289
0.421
192
0.304
0.369
0.354
0.413
0.477
0.434
0.338
0.453
0.332
0.3%
0.419
0.511
Table 2: Transfer Learning results. The best results are in bold, and the second results are underlined.
Mdeels
Prodict
Faaldornect Part-tuning
Full-tuning
Profict
FarchTST
Full-tuming
Predict
F FEDiormer
Full-tuning
Prodict
Autolormer
Full-tuning
Metric
MISE
MATE
MSS
MENE
XISE
MANE
MSE
MAN
MIS
XIAE
MIS
MESE.
MISE:
NINE
MISE:
MAE
XISE
MAE
96
0.310
0.369
0.287
0.333
0.37%
0.328
0.365
0.369
[1287
0.337
0:420
0.549
0.315
0.337
0.347
0:439
0.342
0.385
ETTh2
196
192
0
0:411
1.40)
1.006
0.358
0.342
0.382
0.384
0.350
0.337
0.376
0.408
0.423
0.430
0.419
0.366
0.373
0.305
0.385
0.416
0:475
0.446
0.475
0.378
0.409
0.430
0.416
0.531
0.545
0.511
0.515
0:415
0:415
0.443
0.42%
730
0.468
0.416
0.437
0.401
0.479
0.467
0,410
0.45
0.579
0,517
0.46
0.487
0.694
0.602
0.452
0.469
96
02230
01294
0.131
0.260
0.173
0.251
0.189
02544
01.2
0.33
0.266
0,778
0.20
0.235
0.331
0.406
0.212
0.293
ETTm2
336
192
0.350
0.306
0.400
0.305
0.299
0.337
0.302
0.263
0.332
0.322
0.365
0.243
0.305
0.330
0.437
0.441
0.488
0.266
0.335
0.369
0.304
0.506
0.501
0.461
0:375
0.333
0.370
0.331
720
0.406
0.391
0.499
0.419
0.405
0.503
0.423
0.17
0.575
0.444
0.433
01121
0100
0.997
0.50)
0115
0200
0.36
0.111
0.88
0.349
0.445
ClasterAA
48
0.215
0150
0.195
0.144
0
0.322
0192
0.250
0.240
0.188
0.360
0.299
0.219
0.42
0,392
0.463
0.375
0.362
0,422
0.450
0.312
0.663
0.318
0.592
0,602
24
0120
0.117
01124
0.33
0.147
0.21
0.102
0.318
Chnster-B
48
0.174
0.170
0.207
0.306
0.178
0.510
0.162
0,283
0.347
0.361
0.234
0.354
0.253
0.344
0.264
0.347
0.427
0.318
0.408
0.430
0.280
0.399
192
0.495
0/62
0.72
0.33
0.07
0.212
0.194
0,332
Claster-C
0.15
0.255
0.118
0.162
0.364
192
0.301
0419
0.349
0.332
0.537
0.417
0.507
4.3 ABLATION STUDIES
To ascertain the impact of different modules within Pathformer, we perform ablation studies focus- ing on inter-patch attention, intra-patch attention, time series decomposition, and Pathways. The W/O Pathways configuration entails using all patch sizes from the patch size pool for every dataset, eliminating adaptive selection. Table 3 illustrates the unique impact of each module. The influence of Pathways is significant; omitting them results in a marked decrease in prediction accuracy. This emphasizes the criticality of optimizing the mix of patch sizes to extract multi-scale characteris- tics, thus markedly improving the model's prediction accuracy. Regarding efficiency, intra intra-patch attention is notably adept at discerning local patterns, contrasting with inter-patch attention which primarily captures wider global patterns. The time series decomposition module decomposes trend
8

---

Published as a conference paper at ICLR 2024
Table Ablation study. W/O Inter, W/O Intra, W/O Decompose represent removing the inter-patch
attention, intra-patch attention, and time series decomposition, respectively.
Modek Metrie
W/O Inter
W/
Intra
W/O Decompose
W/O Fathways
Fathlormer
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.196
0.170
0.203
0.162
0.198
0.204
0.156
Weather
192
0.219
0.248
0.220
0.249
0.212
0.244
0.219
0.250
0.206
0.240
0.262
0.290
0.272
0.292
0.256
0.290
0.254
0.282
720
0.350
0.349
0.358
0.357
0.344
0,340
0.349
0,348
0.340
0.336
0.182
0.264
0.152
0.256
0.145
0.236
Electricity
192
0,185
0.270
0.193
0.275
0.176
0.264
0.185
0.272
0.167
0.256
0.214
0.297
0.195
0.281
0.186
0.275
720
0.239
0.322
0.253
0.327
0.235
0,316
0.254
0.332
0.231
0.309
Table 4: Parameter sensitivity study. The prediction accuracy varies with K.
K=1
K
K -]
X
Metric
MSE
MAP.
MSF
MAR
MSE
MAR
MSE
MAR
96
0.283
0.333
0.279
0.331
0.256
0.337
0.282
0.333
ETT12
192
0.355
0.380
0.349
0.380
0.354
0.383
0.359
0.384
336
0.342
0.379
0.348
0.382
0.338
0.377
0.347
0.380
720
0.411
0.430
0.398
0.424
9.406
0.428
0.407
0.452
96
0.162
0.247
0,145
0.236
0.147
0,238
0.152
0.244
Electricity
192
0.175
0.260
0.167
0.256
0.176
0.265
0.178
0.266
336
0.192
0.278
0.186
0.275
0.181
0.274
0.190
0.277
720
0.234
0,311
0.231
0.309
0.230
0.308
0.235
0,313
and periodic patterns to improve the ability to capture the temporal dynamics of its input, assisting in the identificatinn of appropriate patch sizes for combination.
Varying the Number of Adaptively Selected Patch Sizes. Pathformer adaptively selects the top K patch sizes for combination, adjusting to different time series samples. We evaluate the influence of different K values prediction accuracy in Table 4 Our findings show that K = 2 and K 3 yield better results than K 1 and K 4, highlighting the advantage of adaptively modeling critical multi-scale characteristics for improved accuracy. Additionally, distinct time series samples benefit from feature extraction using varied patch sizes, but not all patch sizes are equally effective.
Visualization of Pathways Weights. We show three samples and depict their average Pathways weights for each patch size in Figure 4 Our observations reveal that the samples possess unique Pathways weight distributions. Both Samples and 2, which demonstrate longer seasonality and similar trend patterns, show similar visualized Pathways weights. This manifests in the higher weights they attribute to the larger patch sizes. On the other hand, Sample 3, which character- ized by its shorter seasonality pattern, aligns with higher weights for the smaller patch sizes. These observations underscore Pathformer's adaptability, emphasizing its ability to discern and apply the optimal patch size combinations for the diverse seasonality and trend patterns across samples.
Throe sumples from the weatber
Puch weight of Sample!
Poth weight of Samples
Podh weigtt of Sample3
64
430
B22
0.219
0.131
0.163
82
BA67
0,254
0.124
8.14
B2.
8.123
0.133
0,2
9.426
0.294
0.324
0.314
42
8332
0.374
8.677
B3
8.142
0.177
0,385
3.455
44
51
52
51
52
59
Figure 4: The average pathways weights of different patch sizes for the Weather. B1, B2, and Bg denote distinct AMS (Adaptive Multi-Scale) blocks, while S1, Sz, Sz, and S₁ represent varying patch sizes within each AMS block, with patch size decreasing sequentially.
5 CONCLUSION
In this paper, we propose Pathformer, a Multi-Scale Transformer with Adaptive Pathways for time series forecasting. integrates multi-scale temporal resolutions and temporal distances by introduc- ing patch division with multiple patch sizes and dual attention on the divided patches, enabling the comprehensive modeling of multi-scale characteristics. Furthermore, adaptive pathways dynami- cally select and aggregate scale-speciftt characteristics based on the different temporal dynamics. These innovative mechanisms collectively empower Pathformer to achieve outstanding prediction performance and demonstrate strong generalization capability several forecasting tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China (62372179) and Alibaba Innovative Research Program.
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πyy Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
David Campos, Tung Kieu, Chenjuan Guo, Feiteng Huang, Kai Zheng, Bin Yang. and Christian S. Jensen. Unsupervised time series outlier detection with diversity-drive convolutional ensembles. Proceedings of the VLDB Endowment, 2022.
Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza Ramírez, Max Mergenthaler Canseco, and Artur Dubrawski. NHITS: neural hierarchical interpolation for time series forecast- ing. In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
Cathy WS Chen, Richard Gerlach, Edward MH Lin, and WCW Lee. Bayesian forecasting for financial risk management, pre and post the global financial crisis. Journal of Forecasting, 2012.
Weiqi Chen, Wenwei Wang, Bingqing Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting, International Conference on Knowledge Discovery & Data Mining (KDD), 2022.
Yunyao Cheng, Peng Chen, Chenjuan Guo, Kai Zhao, Qingsong Wen, Bin Yang, and Christian S. Jensen. Weakly guided adaptation for robust time series forecasting. Proceedings of the VL.DB Endowment, 2024.
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. CoRR, 2014.
Cazvin-Gabriel Cirstea, Bin Yang, and Chenjuan Guo. Graph attention recurrent neural networks for correlated time series forecasting. In International Conference on Knowledge Discovery & Data Mining (KDD), 2019.
Razvan- Gabriel Cirstea, Tung Kieu, Chenjuan Guo, Bin Yang, and Sinno Jialin Pan. EnhanceNet: Plugin neural networks for enhancing correlated time series forecasting. In IEEE International Conference on Data Engineering (ICDE), 2021.
Bazvan-Gabrias Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. Tri- former: Triangular, variable-specific attentions for long sequence multivariate time series fore- casting. In International Joint Conference on Artificial Intelligence (IJCAI), 2022a.
Rabvin-Gabricl Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio- temporal aware traffic time series forecasting. In IEEE International Conference Data Engi- neering (ICDE), 2022b.
James W Cooley and John W Tukey, An algorithm for the machine calculation complex fourier series. Mathematics of computation, 1965.
Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv, 2023.
Jeff Dean. Introducing pathways: A next-generation ai architecture, 2021.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Marco AR Ferreira, David M Higdon, Herbert KH Lee, and Mike West. Multi-scale and hidden resolution time series models. 2006.
Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Mareus Rohrbach. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Rob Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast package for r. Journal of statistical software, 2008.
Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I Webb, Irwin King, and Shirui Pan. survey on graph neural networks for time series: Forecasting, classifica- tion, imputation, and anomaly detection. arXiv, 2023a.
Ming Jin, Shiyu Wang, Lintao Ma. Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-LLM: Time series forecasting by repro- gramming large language models. arXiv, 2023b.
Tung Kieu, Bin Yang, Chenjuan Guo, Razvan Gabriel Cirstea, Yan Zhao, Yale Song, and Chris- tian S. Jensen. Anomaly detection in time series with robust variational quasi-recurrent autoen- coders. In IEEE International Conference on Data Engineering (ICDE), 2022a.
Tung Kieu, Bin Yang, Chenjuan Guo, Christian Jensen, Yan Zhao, Feiteng Huang, and Kai Zheng. Robust and explainable autoencoders for unsupervised time series outlier detection. In IEEE International Conference on Data Engineering (ICDE), 2022b.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations (ICLR), 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR), 2015.
Hao Li, Jie Shao, Kewen Liao, and Mingjian Tang. Do simpler statistical methods perform better in multivariate long sequence seme-series forecasting? In International Conference on Information & Knowledge Management (CIKM), 2022a.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. In Advances in Neural Information Processing Systems (NeurIPS). 2022a.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and fore- casting. In International Conference on Learning Representations (ICLR), 2022b.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting, In Advances in Neural Information Processing Systems (NeurIPS), 2022c.
Yu Ma, Bin Yang, and Christian Jensen. Enabling time-dependent uncertain eco-weights for road networks. In Proceedings of the ACM Management of Data, 2014.

---

Published as a conference paper at ICLR 2024
Hao Miao, Yan Zhao, Chenjuan Guo, Bin Yang. Zheng Kai, Feiteng Huang, Jiandong Xie, and Christian S. Jensen. A unified replay-based continuous learning framework for spatio-temporal prediction streaming data. In IEEE International Conference on Data Engineering (ICDE), 2024.
Michael Mozer. Induction of multiscale temporal structure. In Advances in Neural Information Processing Systems (NeurIPS), 1991.
Yuqi Nic, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations (ICLR), 2023.
Zhicheng Pan, Yihang Wang, Yingying Zhang, Sean Bin Yang, Yunyao Cheng, Peng Chen, Chen- juan Guo, Qingsong Wen, Xiduo Tian, Yunliang Dou, et al. Magicscaler: Uncertainty-awrre predictive autoscaling. Proceedings of the VLDB Endowment, 2023.
Simon Aagaard Pedersen, Bin Yang. and Christian S. Jensen. Anytime stochastic routing with hybrid learning. Proceedings of the VLDB Endowment, 2020.
Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural net- work approach to high-dimensional time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Mohammad Amin Shabani, Amir H. Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: Itera- tive multi-scale refining transformers for time series forecasting. International Conference on Learning Representations (ICLR), 2023.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2016
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems (NeurIPS), 2017.
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. MICN: multi- scale local and global context modeling for long-term series forecasting. International Confer- ence on Learning Representations (ICLR), 2023.
Junke Wang, Zuxuan Wu, Wenhao Ouyang, Xintong Han, Jingjing Chen, Yu-Gang Jiang, and Ser- Nam Lim. M2TR: multi-modal multi-scale transformers for deepfake detection. In International Conference on Multimedia Retrieval (ICMR), 2022a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. International Conference on Computer Vision (ICCV), 2021.
Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: A versatile vision transformer hinging on cross-scale attention. In International Conference on Learning Representations (ICLR), 2022b.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: survey. In International Joint Conference on Artificial Intelligence (IJCAI), 2023.
Ruofeng Wen, Kari Torkkola, Balakrishman Narayanaswamy, and Dhruv Madeka. multi multi-horizon quantile recurrent forecaster. arXiv, 2017.
12

---

Published as a conference paper at ICLR 2024
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. In Advances Neural Information Processing Systems (NeurIPS), 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations (ICLR), 2023a.
Xinle Wu, Dalin Zhang. Chenjuan Guo, Chaoyang He, Bin Yang, and Christian S. Jensen, AutoCTS:
Automated correlated time series forecasting. Proceedings of the VLDB Endowment, 2022.
Xinle Wu, Dalin Zhang, Miao Zhang, Chenjuan Guo, Bin Yang, and Christian S. Jensen. AutoCTS+:
Joint neural architecture and hyperparameter search for correlated time series forecasting. Pro- ceedings of the ACM on Management of Data, 2023b.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang, Con- necting the dots: Multivariate time series forecasting with graph neural networks. In International Conference on Knowledge Discovery Data Mining (KDD), 2020.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
Kai Zhao, Chenjuan Guo, Peng Han, Miao Zhang, Yunyao Cheng, and Bin Yang. Multiple time series forecasting with dynamic graph modeling. Proceedinggs of the VLDB Endowment, 2024.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng,
Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference on Machine Learning (ICML), 2022.
Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained Im. arXiv, 2023.
Zhaoyang Zhu, Weiqi Chen, Rui Xia, Tian Zhou, Peisong Niu, Bingqing Peng, Wenwei Wang, Hengbo Liu, Ziqing Ma, Xinyue Gu, al. Energy forecasting with robust, flexible, and explain- able machine learning algorithms. Al Magazine, 2023. 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 EXPERIMENTAL DETAILS
A.1.1 DATASETS
The Special details about experiment datasets are as follows: ETT datasets consist of variables, originating from two different electric transformers. It covers the period from January 2016 to January 2018. Each electric transformer has data recorded at 15-minute and I-hour granularities, labeled as ETTh1, ETTh2, ETTml, and ETTm2. Weather + dataset comprises 21 meteorological indicators in Germany, collected every 10 minutes. Electricity [ dataset contains the power con- sumption of 321 users, recorded every hour, spanning from July 2016 to July 2019. ILI collects weekly data on patients with influenza-like illness from the Centers for Disease Control and Preven- tion of the United States spanning the years 2002 to 2021. Traffic comprisss hourly data sourced from the California Department of Transportation. This dataset delineates road occupancy rates measured by various sensors on the freeways of the San Francisco Bay area. Cloud cluster datasets private business data, documenting customer resource demands 1-minute intervals for three clusters: cluster-A, cluster-B, cluster-C, where A,B,C represent different cities, covering the period from February 2023 April 2023. For dataset preparation, follow the established practice from previous studies (Zhou et al. 2021 Wu et al.) 2021). Detailed statistics are shown in Table
Table 5: The statistics of datasets
Datasets
ETTh1&ETTh2
ETTm1&ETTm2
Weather
Electricity
ILI
Traffic
Cluster
Variables
7
7
21
321
7
862
6
Timestamps
17420
69680
52696
26304
966
17544
256322
Split Ratio
6:2:2
6:2:2
7:1:2
7:1:2
7:1:2
7:1:2
7:1:2
A.1.2 BASELINES
In the realm of time series forecasting, numerous models have surfaced in recent years. We choose models with superior predictive performance from 2021 to 2023 as baselines, including the 2021 state-of-the--ar (SOTA) Autoformer, the 2022 SOTA FEDformer, and the 2023 SOTA PatchTST and NLinear, among others. The specific code repositories for each of these models are as follows:
PatchTST: http://ggthbbboomm/yqqucce//PPhf
NLinear: http://gtiibbooom/oooc////////1f
FEDformer: http://ghhbbbcom//////ziiigfffff
Scaleformer htwp:///www........oooo//////oaa
TiDE: httpp//wwhhlooooooooccccco/ooooe
Pyraformer: http://giiiibbbooommmmmeeeccecep
Autoformer: https:/ggihub.ccom//hmm////uuoou
A.2 UNIVARIATE TIME SERIES FORECASTING
We conducted univariate time series forecasting experiments on the ETT and Cloud cluster datasets. As shown in Table Pathformer stands out with the best performance in 50 cases and the second- best in 5 out of 56 instances. Pathformer has outperformed the second-best baseline PatchTST, especially on the Cloud cluster datasets. Our model Pathformer demonstrates excellent predictive performance in both multivariate and univariate time series forecasting.
https///wwwwbbc-jjnaa. htt.:///epp////ee//////////
"hupss//gis.edd edc. www.grap/fuuii///tooaaaaaaaaaaaa
hhtps://pems... dot.dotoooooooooo cow.
14

---

Published as a conference paper at ICLR 2024
Table 6: Univariate time series forecasting results. The imput length / 96, and the prediction
length F E (96, 192, 336, (for cloud clusters datasets F E (24, 48, 96, 192)). The best results highlighted in bold.
Models
Pathformer
PatchTST
FEDiormer
Autoformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.057
0.180
0.037
0.179
0.079
0.215
0.071
0.206
192
0.075
0.208
0.076
0.209
0.104
0.245
0.114
0.262
ETThi
336
0.076
0.216
0.093
0.240
0.119
0,270
0.107
0.258
720
0.090
0.238
0.097
0.245
0.142
0.299
0.126
0.283
0.128
0.274
0.127
0.273
0.128
0.271
0.153
0.306
0.177
0.330
0.328
0,330
0.204
ETTh2
336
0.180
0.340
0.221
0.374
0.231
0.378
0.246
0.389
720
0.213
0.371
0.250
0.403
0.278
0.420
0.268
0.409
96
0.022
0.126
0,030
0.127
0.033
0.140
0.056
0.183
192
0.042
0.160
0.043
0.165
0.058
0.186
0.081
0.216
ETTml
336
0.058
0.185
0.059
0.185
0.084
0.231
0.076
0.218
720
0.079
0.217
0.081
0.218
0.102
0,250
0.110
0.267
96
0.062
0.179
0.064
0.181
0.072
0,206
0.065
0.189
192
0.096
0.230
0.097
0.231
0.102
0.245
0.118
0.256
ETTm2
336
0.128
0.268
0.129
0.270
0.130
0.279
0.154
0.305
720
0.179
0.326
0.181
0.330
0.178
0.325
0.182
0.335
0.137
0.218
0.174
0,256
0.203
0,303
0.455
0.483
48
0.218
0.280
0.299
0.343
0.308
0.364
0.508
0.504
Cluster-A
96
0.298
0.337
0.434
0.409
0.361
0.403
0,563
0.524
192
0.390
0.401
0.589
0.480
0.409
0.447
0.669
0.583
24
0.100
0.200
0.7
0.218
0.130
0,253
0.197
0.339
Cluster-B
48
0.146
0,251
0.158
0.265
0.149
0,272
0.247
0.390
0.219
0.301
0.234
0.327
0.230
0.342
0.313
0.429
192
0.454
0.404
0.444
0.412
0.544
0.080
0.191
0.210
0.120
0,258
0.354
0.117
0.232
0.138
0.261
0.151
0.302
0.229
0.365
Cluster-C
96
0.176
0.286
0.222
0.330
0.198
0.342
0.293
0.420
192
0.345
0.390
0.404
0.443
0.361
0.444
0.441
0.524
A.3 VARYING THE INPUT LENGTH WITH TRANSFORMER MODELS
In time series forecasting tasks, the size of the input length determines how much historical infor- mation the model receives. We select models with better predictive performance from the main experiments as baselines. We configure different input lengths evaluate the effectiveness Path- former and visualize the prediction results for input lengths of 48, 192. From Figure 5 Pathformer consistently outperforms the baselines on the ETTh1, ETTh2, Weather, and Electricity. As depicted in Table Table[]andd and Table 8 for /! 48, 192, Pathformer stands out with the best performance in 46, 44 cases out 48, respectively. Based on the results above, is evident that Pathformer outperforms the baselines across different input lengths. As the input length increases, the prediction metrics of Pathformer continue to decrease, indicating that it capable modeling longer sequences.
ET H-48
ETT Hm48
Weather H-48
Electricity H-
0.50
0.91
0.40
MAR.
0.65
0.35
0.45
0.40
0.30
0.35
0.30
96
192
3356
720
192
336
720
192
336
720
192
336
730
F
ETTh H--12
ETThh H-192
Weather H1-192
Electricity B1-19
0.40
0.50
0.65
0.35
0.45
0.30
0.30
0.38
0.35
192
3.56
720
192
3356
720
316
120
192
335
730
Pathformer
PatchTST
FEDformer
Autoformer
Figure 5: Results with different input length for ETThl, ETTh2, Weather and Electricity.
15

---

Published as a conference paper at ICLR 2024
Table 7: Multivariate time series forecasting results. The input length 48, and the prediction length F E (96, 192, 336, 720}. The best results are highlighted in bold.
Models
Pathformer
PSt
FlDffommer
Autoformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.390
0.403
0.410
0.417
0.382
0.419
0,406
0.432
192
0.454
0.434
0.469
0.448
0.451
0.456
0,451
0.452
ETTh1
336
0.483
0.445
0.516
0.469
0.499
0.487
0,461
0.464
720
0.507
0.475
0.509
0.487
0.510
0.504
0,498
0,500
96
0.295
0.335
0.307
0.348
0.330
0.373
0.344
0.383
192
0.366
0.381
0.397
0.399
0.440
0.436
0.425
0.426
ETTh2
336
0.368
0.390
0.412
0.420
0.543
0.504
0.445
0.452
720
0.428
0.435
0.434
0.441
0,471
0.483
0.483
0.481
96
0.420
0.392
0.424
0.403
0.428
0.432
0.745
0.556
192
0.446
0.410
0.468
0.429
0.476
0.460
0.715
0.556
ETTml
336
0.469
0.431
0.501
0.453
0.526
0.494
0.816
0.590
720
0.512
0.465
0.553
0.484
0.630
0.528
0.746
0.572
96
0.181
0.256
0.189
0.272
0.185
0.274
0.211
0.299
192
0.251
0.301
0.260
0.371
0.256
0.318
0.277
0.388
ETTm2
336
0.323
0.349
0.328
0.359
0.329
0.365
0.347
0.380
720
0.420
0.406
0.429
0.415
0.447
0.432
0,441
0.432
96
0.188
0.223
0.212
0.243
0.241
0.309
0.291
0.357
192
0.227
0.257
0.254
0.277
0.308
0.356
0.349
0.391
Weather
336
0.276
0.297
0.310
0.316
0.385
0.406
0.409
0.424
720
0.345
0.349
0.385
0.365
0.438
0.432
0.437
0.431
96
0.201
0.280
0.225
0.293
0.240
0.349
0.211
0.322
192
0.210
0.285
0.229
0.299
0.248
0.357
0.224
0.331
Electricity
336
0.236
0.305
0.239
0.316
0.265
0.370
0.259
0.362
720
0.272
0.338
0.282
0,349
0.326
0.405
0.313
0.407
Table 8: Multivariate time series forecasting results. The input length H = 192, and the prediction length F (96, 192, 336, 720). The best results are highlighted in bold.
Models
Pathformer
PStchTTST
FEDformer
Autoformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.377
0.394
0.384
0.403
0.388
0.423
0.430
0.441
192
0.428
0.421
0.428
0.425
0.433
0.456
0.487
0.467
ETTh1
336
0.424
0.419
0.452
0.436
0.445
0.462
0.478
0.474
720
0.474
0.459
0.453
0.459
0.476
0.490
0.518
0.519
96
0.283
0.334
0.285
0.340
0.397
0.424
0.362
0.401
192
0.343
0.374
0.356
0.387
0.439
0.458
0.430
0.447
ETTh2
336
0.332
0.374
0.351
0.396
0.471
0.481
0.408
0.447
720
0.393
0.421
0.395
0.427
0.479
0.490
0.440
0.469
96
0.295
0.335
0.295
0.345
0.381
0.424
0.510
0.428
192
0.336
0.361
0.330
0.365
0.412
0.441
0.619
0.545
ETTml
336
0.359
0.384
0.364
0.388
0.435
0.455
0.561
0.500
720
0.432
0.420
0.423
0.424
0.473
0.474
0.580
0.512
96
0.169
0.250
0.169
0.254
0.223
0,305
0.244
0.321
192
0.230
0,290
0.230
0.294
0.281
0.339
0.302
0.362
ETTm2
336
0,286
0.328
0.281
0.329
0.321
0,364
0.346
0.390
720
0.375
0.384
0.373
0.384
0.417
0.420
0.423
0.428
96
0.152
0.189
0.160
0.205
0.239
0.316
0.298
0.363
192
0.198
0.237
0.204
0.245
0.274
0.326
0.322
0.379
Weather
336
0.246
0.276
0.258
0.285
0.334
0.369
0.378
0.409
720
0.329
0.331
0.329
0.337
0.401
0.412
0.435
0.431
96
0.136
0.232
0.146
0.240
0.231
0.343
0.198
0.313
192
0.143
0.248
0.152
0.252
0.258
0.361
0.218
0.335
Electricity
336
0.172
0.274
0.178
0.271
0.273
0.372
0.252
0.352
720
0.218
0.299
0.223
0.308
0.308
0.402
0.275
0.371
A.4 MORE COMPARISONS WITH SOME BASIC BASELINES
To validate the effectiveness of Pathformer, conducted extensive experiments with some recent basic baselines that exhibited good performance: DLinear, NLinear, and N-HiTS, using long input sequence length (H 336). As depicted in Table 9 proposed model Pathformer outperforms
16

---

Published as a conference paper at ICLR 2024
Table 9: Multivariate time series forecasting results. The input length 1/ 336 ( for ILI dataset
H 106) ), and the prediction length F E (96, 192, 336, 720) (for ILI dataset F E (24, 36, 48, 60)
). The best results are highlighted in bold.
Method
Pathformer
DLincar
NLinear
N-HITS
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.369
0.395
0.375
0.399
0.374
0.394
0.378
0.393
192
0.414
0.418
0.405
0.416
0.408
0.415
0.427
0.436
ETTh1
336
0.401
0.419
0.439
0.443
0.429
0.427
0.458
0.484
720
0.440
0,452
0.472
0.490
0.440
0.453
0.561
0,501
96
0.276
0.334
0.289
0.353
0.277
0.338
0.274
0.345
192
0.329
0.372
0.383
0.418
0.344
0.381
0.353
0.401
ETTh2
336
0.324
0.377
0.448
0.465
0.357
0.400
0,382
0.425
720
0.366
0,410
0.605
0.551
0.394
0.436
0.625
0.557
96
0.2N5
0.336
0.299
0.353
0,306
0.348
0.302
0.350
192
0.331
0.361
0.335
0.365
0.349
0.375
0.347
0.383
ETTml
336
0.362
0,382
0.369
0.386
0.375
0.388
0.369
0.402
720
0.412
0.414
0.425
0.421
0.433
0.422
0.431
0.441
96
0.163
0.248
0.167
0.260
0.167
0.255
0.176
0.255
192
0.220
0,286
0.224
0.303
0.221
0.293
0.245
0.305
ETTm2
336
0.275
0.325
0.281
0.342
0.274
0.327
0.295
0.346
720
0.363
0.381
0.397
0.421
0.368
0.384
0.401
0.413
96
0.144
0.184
0.176
0.237
0.182
0.232
0.158
0.195
192
0.191
0.229
0.220
0.282
0.225
0.269
0.211
0.247
Weather
336
0.234
0,268
0.265
0.319
0.271
0.301
0.274
0.300
720
0.316
0.323
0.323
0.362
0.338
0.348
0.351
0.353
96
0.134
0,218
0.140
0.237
0.141
0.237
0.147
0,249
192
0.142
0.235
0.153
0.249
0.154
0.244
0.167
0.269
Electricity
336
0.162
0.257
0.169
0.267
0.171
0.265
0.186
0.290
720
0.200
0.290
0.203
0.301
0,210
0.297
0.243
0.340
24
1.411
0.705
2.215
1.081
1.683
0.868
1.862
0.869
36
1.365
0.727
1.963
0.963
1.703
0.859
2.071
0.934
ILI
48
1,537
0.764
2.130
1.024
1.719
0.884
2.134
0.932
60
1.418
0.772
2.368
1.096
1.819
0.917
2.137
1.968
96
0.373
0,241
0.410
0.282
0.410
0.279
0,402
0.282
192
0.380
0,252
0.423
0.287
0.423
0.284
0.420
0.297
Traffic
336
0.395
0.256
0.436
0.296
0.435
0.290
0.448
0,313
720
0.425
0.280
0.466
0.315
0.464
0.307
0.539
0.353
these baselines for the input length 336. Zeng et al. (2023) point out that the previous Transformer cannot extract temporal relations well from longer input sequences, but our proposed Pathformer performs better with longer input length, indicating that considering adaptive multi-scale modeling can be an effective way to enhance such relation extraction ability of Transformers.
A.5 DISCUSSION
A.5.1 COMPARE WITH PATCITST
PatchTST divides time series into patches, with empirical evidence proving that patching is an ef- fective method to enhance model performance in time series forecasting. Our proposed model Path- former extends the patching approach to incorporate multi-scale modeling. The main differences with PatchTST
follows: (1) Partitioning with Multiple Patch Sizes: PatchTST employs a single patch size to partition time series, obtaining features with a singular resolution. In con- trast, Pathformer utilizes multiple different patch sizes each layer for partitioning. This approach captures multi-scale features from the perspective of temporal resolutions. (2) Global correlations
between patches and local details in each patch: PatchTST performs attention between divided patches, overlooking the internal details in each patch. In contrast, Pathformer not only considers the correlations between patches but also the detailed information within each patch. It introduces dual attention(inter-pachh attention and intra-patch attention) to integrate global correlations and lo- cal details, capturing multi-scale features from the perspective of temporal distances. (3)Adaptive Multi-scale Modeling: PatchTST employs fixed patch size for all data, hindering the grasp of crit- ical patterns in different time series. We propose adaptive pathways that dynamically select varying patch sizes tailored to the features of individual samples, enabling adaptive multi-scale modeling.
17

---

Published as a conference paper at ICLR 2024
A.5.2 COMPARE WITH N-HITS
N-HiTS utilizes the modeling of multi-scale features for time series forecasting, but it differs from Pathformer in the following aspects: (1) N-HiTS models time series features of different resolu- tions through multi-rate data sampling and hierarchical interpolation. In contrast, Pathformer not only takes into account time series features of different resolutions but also approaches multi-scale modeling from the perspective of temporal distance. Simultaneously considering temporal resolu- tions and temporal distances enables more comprehensive approach to multi-scale modeling. (2) N-HiTS employs fixed sampling rates for multi-rate data sampling, lacking the ability to adaptively perform multi-scale modeling based on differences in time series samples. In contrast, Pathformer has the capability for adaptive multi-scale modeling. (3) N-HiTS adopts linear structure to build its model framework, whereas Pathformer enables multi-scale modeling in Transformer architecture.
A.5.3 COMPARE WITH SCALEFORMER
Scaleformer also utilizes the modeling of multi-scale features for time series forecasting. It differs from Pathformer the following aspects: (1) Scaleformer obtains multi-scale features with different temporal resolutions through downsampling. In contrast, Pathformer not only considers time series features of different resolutions but also models from the perspective of temporal distance, taking into account global correlations and local details. This provides more comprehensive approach to multi-scale modeling through both temporal resolutions and temporal distances. (2) Scaleformer requires the allocation of predictive model at different temporal resolutions, resulting in higher model complexity than Pathformer. (3) Scaleformer employs fixed sampling rates, while Pathformer has the capability for adaptive multi-scale modeling based the differences in time series samples.
A.6 EXPERIMENTS ON LARGE DATASETS
The current time series forecasting benchmarks are relatively small, and there is a concern that the predictive performance of the model might be influenced by overfitting. To address this issue, we explore larger datasets to validate the effectiveness of the proposed model. The detailed process is as follows: We seek larger datasets from two perspectives: data volume and the number of variables. We add two datasets, the Wind Power dataset, and the PEMS07 dataset, evaluate the performance of Pathformer on larger datasets. The Wind Power dataset comprises 7397145 147 timestamps, reaching sample size in the millions, and the PEMS07 dataset includes 883 variables. As depicted in Table 10 Pathformer demonstrates superior predictive performance on these larger datasets compared with some state-of-the-arr methods such PatchTST, DLinear, and Scaleformer.
Table 10: Results on large datasets: PEMS07 and Wind Power.
Methods
Pathformer
PS
DLinear
Scaleformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.135
0.243
0.146
0.259
0.564
0.536
0.152
0.268
192
0.177
0.271
0.185
0.286
0.596
0.555
0.195
0.302
PEMS07
336
0.188
0.278
0.205
0.289
0.475
0.482
0.276
0.394
720
0.208
0.296
0.235
0.325
0.543
0.523
0.305
0.410
96
0.062
0.146
0.070
0.158
0.078
0.184
0.089
0.167
192
0.123
0.214
0.131
0.237
0.133
0.252
0.163
0.246
Wind Power
336
0.200
0.283
0.215
0.307
0.205
0.325
0.225
0.352
720
0.388
0.414
0.404
0.429
0.407
0.457
0.414
0.426
A.7 VISUALIZATION
We visualize the prediction results of Pathformer on the Electricity dataset. As illustrated in Figure 6 for prediction lengths F' 96, 192, 336, 720, the prediction curve closely aligns with the Ground Truth curve, indicating the outstanding predictive performance of Pathformer. Meanwhile, Path- former demonstrates effectiveness in capturing multi-period and complex trends present in diverse samples. This serves as evidence of its adaptive modeling capability for multi-scale characteristics.
18

---

Published as a conference paper at ICLR 2024
Ground"uut
Groundrruch Predicttion
Prodictionn
125
175
(a) Prediciton Length = 96
(b) Prediciton Length = 192
2.0
Groundrruth Prediction
Gmundtruth Prediction
15.
1AHd MMD
0.5
10
300
400
2:00
650
700
(c) Prediciton Length F 336
(d) Prediciton Length = 720
Figure 6: Visualization of Pathformer's 's prediction results on Electricity. The input length 96
19


Reviews:
[{"review_id": "vN19rBewUx", "reviewer": "Reviewer_5cD6", "summary": "The paper proposes a variation of PatchTST architecture in the context long-horizon time series forecating.", "questions": "- When talking about multi-scale processing in related work please discuss relation to Challu et al. N-HiTS: Neural hierarchical interpolation for time series forecasting AAAI'23, which seems to be relevant work on multi-scale modelling", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- An interesting study that makes an incremental step towards making transformers effective at long-horizon forecasting task\n- Paper is clearly written and topic is important for the ICLR audience", "weaknesses": "- \"Recent advances for time series forecasting are mainly based on Transformer architectures\". I would say this statement is not aligned with the most recent empirical results and contradicts existing facts. Having read the following papers one could argue that the transformer based models in time series forecasting have been basically a disaster in the recent years, mainly because authors of papers based on transformer-driven models disregarded including some basic baselines in their studies. Please rewrite intro and related work accordingly.\n    - Challu et al. N-HiTS: Neural hierarchical interpolation for time series forecasting. AAAI'23\n    - Zeng et al. Are transformers effective for time series forecasting? AAAI'23\n    - Li et al. Do Simpler Statistical Methods Perform Better in Multivariate Long Sequence Time-Series Forecasting? CIKM'23\n- Not all datasets are present in the study. Please include additional results on ILI and Traffic from PatchTST\n- Please include results from Zeng et al. Are transformers effective for time series forecasting? AAAI'23 in your table and you will see that your results are not state of the art. This makes the results unconvincing, because basically a very complex model is not able to use the same inputs as very simple models in an effective way. Transformer based papers have consistently failed to include appropriate baselines in the studies creating a large gap in methodology and undermining the ultimate reliability of these studies. The work can be interesting if authors show that with the proposed modifications a transformer based model can be more effective than much simpler and faster models presented in Zeng et al. and Li et al.\n- The model seems to borrow conceptually very heavily from the PatchTST model without explicitly recognizing the source of inspiration. Without a detailed explanation of the actual difference between the two architectures the proposed architecture appears to be a minor perturbation of the original PatchTST.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SaChqAPQw", "reviewer": "Reviewer_Sqch", "summary": "This paper presents a new multi-scale Transformer architecture for long-range time series modeling.The Discriminant Fourier Transform (DFT) is utilized to determine the patch sizes so as to divide the input time series into patches of different sizes, thus enabling cross-scale information fusion. In the multiscale transformer block, intra-patch attention and inter-patch attention mechanisms are utilized to perform attentional operations, thus enhancing the processing of temporal information.Experiments show the proposed method achieves state-of-the-art performance among existing models and exhibits superior generalization capabilities across different transfer scenarios.", "questions": "Why is there a gap between the benchmark data in the paper and the original paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well written and well-motivated.\n2.The Multi-Scale Router combines the advantages of patch division and seasonality decomposition.\n3.Dual attention helps to harmonize operations between intra-patch and inter-patch components, allowing the transformer to efficiently process time series data.", "weaknesses": "Time-series Dense Encoder (TiDE) is also a popular long-term time-series forecasting benchmark.But the paper does not include experiments comparing the proposed method to TiDE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eogpsp59VJ", "reviewer": "Reviewer_LNPN", "summary": "This paper proposes multi-scale transformers with adaptive pathways. The proposed method integrates both temporal resolution and temporal distance for multi-scale modeling. It further enriches the multi-scale transformer with adaptive pathways. Experimental results showed the efficacy of proposed method and state-of-the-art performance.", "questions": "1. The current time series forecasting datasets are pretty small, and performance may be satuated or over-fitting. Could the method be used for larger datasets?\n\n2. Scalformer [1] also uses hierarchical design and the scales of time series data, Could this paper compare the similarities and differences with Scalformer.\n\n[1] Shabani, Amin, et al. \"Scaleformer: iterative multi-scale refining transformers for time series forecasting.\" ICLR (2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1)  It's novel to propose multi-scale transformers with adaptive pathways. \n\n2) It's novel to integrate both temporal resolution and temporal distance for multi-scale modeling.\n\n3) The experiments showed state-of-the-art performance.", "weaknesses": "1.The current time series forecasting datasets are pretty small, and performance may be satuated or over-fitting. Could the method be used for larger datasets?\n\n2. Scalformer [1] also uses the multi-scale nature of time series data, this paper didn't mention and compare the similarities and differences with Scalformer.\n\n[1] Shabani, Amin, et al. \"Scaleformer: iterative multi-scale refining transformers for time series forecasting.\" ICLR (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)