DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching
Authors: Donghao Luo
Xue Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Time series forecasting
Transformer
Deep learning
Primary area: other
TL;DR: We propose DeformableTST, a Transformer-based model less reliant on patching, to broaden the applicability of Transformer-based models in time series forecasting tasks and achieves SOTA performance in a wider range of time series forecasting tasks.

Abstract: With the proposal of patching technique in time series forecasting, Transformerbased models have achieved compelling performance and gained great interest from
the time series community. But at the same time, we observe a new problem that
the recent Transformer-based models are overly reliant on patching to achieve ideal
performance, which limits their applicability to some forecasting tasks unsuitable
for patching. In this paper, we intent to handle this emerging issue. Through diving
into the relationship between patching and full attention (the core mechanism
in Transformer-based models), we further find out the reason behind this issue
is that full attention relies overly on the guidance of patching to focus on the
important time points and learn non-trivial temporal representation. Based on this
finding, we propose DeformableTST as an effective solution to this emerging
issue. Specifically, we propose deformable attention, a sparse attention mechanism
that can better focus on the important time points by itself, to get rid of the need of
patching. And we also adopt a hierarchical structure to alleviate the efficiency issue
caused by the removal of patching. Experimentally, our DeformableTST achieves
the consistent state-of-the-art performance in a broader range of time series tasks,
especially achieving promising performance in forecasting tasks unsuitable for
patching, therefore successfully reducing the reliance on patching and broadening
the applicability of Transformer-based models. Code is available at this repository:
https://github.com/luodhhh/DeformableTST.

DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching
Donghao Luo, Xue Wang
Department of Precision Instrument, Tsinghua University, Beijing 100084, China 1dh21@mails tsinghua. cn, Fangxue@mail tsinghua. edu.ccn
Abstract
With the proposal of patching technique in time series forecasting, Transformer- based models have achieved compelling performance and gained great interest from the time series community. But the same time, we observe new problem that the recent Transformer-bes performance, which limits their applicability some forecasting tasks unsuitable for patching. In this paper, we intent to handle this emerging issue. Through diving into the relationship between patching and full attention (the core mechanism in Transformer based models). we further find out the reason behind this issue is that full attention relies overly on the guidance of patching focus on the important time points and learn non-trivial temporal representation. Based on this finding, we propose DeformableTST as an effective solution this emerging issue. Specifically, we propose deformable attention, sparse attention mechanism that can better focus on the important time points by itself, to get rid of the need patching. And we also adopt hierarchical structure alleviate the efficiency issue caused by the removal of patching. Experimentally, our DeformableTST achieves the consistent state-of-te of-the-artrrrrro performance in broader range of time series tasks, especially achieving promising performance in forecasting tasks unsuitable for patching, therefore successfully reducing the reliance patching and broadening the applicability of Transformer-basee models. Code is available at this repository: https: htttps///githut http:////ggtthbbccooouohhh////// ormableTST
models are overly reliant on patching to achieve ideal
Introduction
Time series forecasting is widely used in real-world applications, such transportation manage-
ment 33 economic planning [41] 30 31. energy planning [2 34 and weather forecasting [45).
Because of the immense practical value, time series forecasting has received great attention and has grown tremendously in recent years
33 2 28
(5).
But looking back at the development of time series forecasting, Transformer-based models, who have sparked the boom time series forecasting [55] 47 577, are constantly being challenged. In particular, some recent studies [52 [8] 22 have questioned that attention mechanism is not suitable for modeling the temporal dependency in time series. As the early strike back of Transformer based models, PStchTST (35] proposes that attention mechanism work better in temporal modeling with the help of large size patching technique. Afterwards, equipped with the growing patch size and increasing input length, the advanced Transformer-haase models [53] 58
gain great performance improvement and successfully win back the championship in time series forecasting.
51 6
However, with large size patching becoming must-have technique for the following Transformer- based models, a new problem occurs: patched -baed Transformers have to work with a very long input length and very large patch size to achieve ideal performance 35 53 22 But large size patching cannot be apply to all kinds time series forecasting tasks. For example, some forecasting
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) using patching (nppu length: 336, Token number: 336)
(b) Divided 42 patches (Input longth: 336, Tokon number: 42)
Focusing on nearly all imput time points
Focusing on nearly 42 important imput time points
Figure 1: The Effective Receptive Field (ERF) of PatchTST. A brighter area means that these time points are focused by the model when extracting temporal representation. The results show that PatchTST highly relies on the guidance of patching to focus on the important time points. This phenomenon is also present in multiple advanced patch-based Transformer forecasters (Appendix E).
tasks are with limited input lengths (29 30 31, which are not sufficient to be divided into patches. In such condition, the advanced Transformer-bast based models suffer from severe performance degradation due to the lack of patching [58 51), limiting their applicability to a wider range of forecasting tasks.
To broaden the applicability of the Transformer-based model, we need to design an attention mechanism that is less reliant on patching (e.g,, can work well with a small patch size or can work well even without patching). To this end, we first analyze exactly why attention must work with patching and why patching can help attention better model the temporal dependency in time series forecasting? We visualize the effective receptive fields (ERFs) of PatchTST (35) in Figure I And the ERFs can indicate which parts of the time points in input series are focused by the model when extracting temporal representations. A surprising finding is shown in Figure (left). If without patching, nearly all time points in input series are equally focused by the model and the model performs worse (MSE 0.385), exposing the problem of distracted attention. This finding means that attention has not learned to distinguish the importance of each time point in input series, leading to trivial representation. Note that the time points in time series very redundant even noisy [35 56 55 53 focusing on the trivial part of them will influence the predictions. Thus, an ideal time series forecaster should mainly focus on a small number of important time points which make contribution to better performance and reflect the property of time series. In Figure [ (right) when using patching, the model focuses on some selected time points and achieve better performance (MSE 0.367), indicating that the model has successfully focused on the important time points. And in terms of why patching can guide the model to learn a non-trivial representation, we find that the pattern of ERF is also divided by patches, which means that patching can force the model to only focus on a small number of important time points based on the patch partition. As a conclusion of since full attention is unable to focus on the important time points
above discussion on Figure I by itself, it highly relies on the guidance of patching to focus on the important time points and learn non-trivial representation. This is the reason why full attention must work with patching to achieve ideal performance.
Therefore, if we can find another way to help attention focus on the important time points, we can get rid of over-reliance on patching. Since full attention is hard to focus due to the redundancy in time series data [35 56 some previous prior- based sparse attentions in time series community [55] 47 57]. But due to the diverse pattern in different time series, their priors are hard to match all kinds of inputs, resulting in their inferior performance. Different from them, we introduce data-driven sparse attention called deformable attention under the inspiration of deformable operations [8] 60 48). It can sample a subset of important time points from the input series based on the learnable offsets and only calculate attention with these selected important time points. These learnable offsets are learned from each input sample, therefore being more flexible to the diverse property in different time series.
55 77 53), replacing it with sparse attention can be a natural idea. There are
Based on the above motivations, we intend to broaden the applicability of Transformer-basse models. To accomplish this goal, we propose Deformable reliant on patching. Technically, the patching process in our method is optional. We remove the patching process most cases. Only when the input length is very long, we will use a small size patching for better efficiency. Since the removal of patching will cause severe memory usage in previous plain architecture, we adopt a hicrarchical architecture to alleviate this efficiency issue. And we further introduce deformable attention, a data-driven sparse attention that can better focus on the important time points by itself, to achieve excellent performance without patching. Experimentally. DeformableTST achieves the consistent state-of-thh-aar performance in wider range of time series tasks, especially in tasks unsuitable for patching, thus successfully reducing the reliance on patching and broadening the applicability of braneformer-based models. Our contribution are as follows:
a Transformer-based model that is less
We dive into the relationship between patching and attention. We point out a new problem that recent advanced Transformer-bast based models are too reliant on patching. And we further

---

find out the reason behind this problem is that full attention relies overly on the guidance of patching to focus on important time points and learn non-trivial temporal representation.
To get rid of the over-reliance on patching, we propose DeformableTST and achieve the wider range of time series forecasting tasks.
consistent ffut--of-the-ar performance Experimental results show that our deformable attention can better model the temporal dependency in time series without reliance on patching.
We successfully broaden the applicability of Transformer- based models in time series tasks, Our DeformableTSTI can flexibly adapt to multiple imput lengths and achieve excellent performance in tasks unsuitable for patching, which is a great improvement than previous Transformer-basee models.
2 Related Work
2.1 Tranformers for Time Series Forecasting
Transformer- based models mainly use attention mechanism to model the temporal dependency in time series [55] 47 57]. In 2020s, they achieve excellent performance time series forecasting for the first
time and bring great attention to time series forecasting tasks [26] 9 56 20 21 71. But their validity is questioned by [52] [8] with the finding that simple linear layer can outperform complicated attention mechanisms. It's until the proposal of patching that Transformer-base -based models win back the championship in time series forecasting [35]. Based on patching technique, Pathformer [6] adopts a
multi-scale patches structure. Crossformer (53) and CARD (51) further propose additionally apply
attention on variate and feature dimensions rather than only on temporal dimension. Sageformer [54] combines the graph methods with patch-based Transformer forecasters. And GPT4TS (58) also transfers pre-trained large language models to time series with the help of patching. But the question of whether attention is suitable for modeling the temporal dependency in time series still remains. For example, although adopting a Transformer architecture, iTransformer (22) still suggests that linear layers are more appropriate for temporal modeling. Mcanwhile, the proposal of patching also comes with new question that advanced Transformer-basee models are too reliant on patching. Therefore, further research about Transformer-based forecasters are still needed, especially on the question of how to better use attention in temporal modeling without over-reliance on patching.
2.2 Sparse Attention
Sparse attention used to be popular in time series forecasting. Early Transformer-based models usually adopt prior-based sparse attention mechanisms. Informer [55] adopt ProbSparse attention to model the temporal dependency. Autoformer and FHDformer [47 [7] further combine the signal processing technique with the attention mechanisms and select the top-k sparse representation in time domain or frequency domain respectively. But due to the diverse pattern in different time series, these priors are hard match all kind of inputs, resulting in their inferior performance. As a comparison, data-driven sparse attention, also called deformable attention, is more flexible to diverse inputs. Similar idea has been explored Computer Vision (CV). Inspired by deformable convolution B (5), deformable DERT [60] proposes multi-scale deformable attention for object detection tasks. And [48] 49 further improve it and make it suitable for general CV tasks. In this work, we propose a deformable attention for time series forecasting to break through the bottleneck faced by previous attention mechanism in modeling temporal dependency.
3 DeformableTST
Given an observed multivariate or univariate time series as input, time series forecasting aims to predict the length-T future series based on the length-/ input series. In real-world scenarios, the input length / varies from a wide range and is not always sufficient for patching technique, leading to the limited applicability of previous patch-based Transformer forecasters. To tackle this problem, we propose DeformableTST. And we introduce details of DeformableTST in following subsections.
3

---

Osrpet
(c) Defermatbi Attention
RovEN (de-norm)
(d) Feed Fornard Network with Canvolution (Cons
Deacoff
Projection
Diw
Dine
Infordes
Jemplig
DownSampling
Point-w5s Convolution
Depth-wise Cunvelation
Point-wise Convolution
LayerNorrm
Matil- Atfcation
Learning Temparal
Learning Global Temporal Information
ComFFN
Learning Festure Represeutation
Lx
LayerNorm
Perception
Deformable Attntion
Embedding Layer
Depth wise
LPL
Sorto
Embedding the Series
Emhedding
Learning Local Temporat Information
RevEN (norm)
Inpet f
Figure 2: Structure overview of TST. (a) The input time series is embedded variate- independently. (b) The local perception unit (LPU) is used to learn the local temporal information. (c) The proposed deformable attention is adopted to learn the global temporal information. (d) The feed-forward network injected with depth- wise convolution (ConvFFN) is used to learn the local temporal information and the new feature representation.
3.1 Structure Overview
As shown in Figure 2 our DeformableTST adopts the encoder-only architecture of Transformer [3 including the input embedding layer, hierarchical Transformer backbone and prediction head. And following the recent Transformer-bart models, adopt RevIN [15] mitigate the distribution shift between the training and testing data.
Input Embedding Layer Denoted X.n E Mx as M variates input time series of length it will be divided into N, non-overlapping patches and then embedded variate-independennly into Do-dimensional embeddings:
X = Emmeedding(X....
(1)
Xa E IR M is the input embedding. It is worth noting that DeformableTST is less reliant on patching and thus the patching process optional. We only adopt patching when the input length is very long for efficiency reasons. And we also adopt much smaller patch size than recent Transformer-bar models, making it more adaptable to diverse input lengths.
Hierarchical Transformer Backbone The backbone is stacked by Transformer blocks and utilizes a hierarchical structure. The forward process in the i-th block is simply formulated follows:
x local LPU(X;-1)
x global LayerNorm
X local
DeformnableAutention local)
(3)
X, LayerNorm
(x
global + ConvFFN(XX!louy))
X, E MR-xX, is the output feature series the i-th block. {1, L). And D; and N; are the sizes of its feature and temporal dimensions. DeformableAttentte epature the global temporal dependency. which will be introudced in Section[322 LPU and ConvFFN are local enhancement modules (Figure 2 (b) and (d)). LPU is the local perception unit, a depth-wise convolution with residual connection [10] And ConvFFN is feed-forward network injected with
is the core component to better
depth-wise wise convolution [50]. These two modules are adopted to improve the local temporal modeling ability. And GELU activation [I] is adopted in ConvFFN to provide nonlinearity when learning the new feature representation. Meanwhile, to construct hierarchical structure, downsampling convolution layer (24) with kernel size and stride 2 is adopted between two blocks, which will halve the series' temporal dimension and double the feature dimension.

---

We first flatten the final representation from the backbone X₁ E RRMxPEXME into MxDxXXN)) Then we obtain the prediction through linear projection layer:
Prediction Head
= Projection(Xx, (x)
(5)
Where Y RxX the prediction of length T with M variates.
3.2 Deformable Attention
Reference Tor
Sampling Taee
AT
N
Offen aT
Caortiinate View
Lincar Interpolatio
Point-wise
Convolationl
N
Sampled Series
Galdi!
Temsor View
Key Value Projection
Mutt hood Affrantion
Depth-wise
Convolation,
Query Projection
Output Projectlon
Striderr
Offict Network
+
NxN
Input
(a) Deformable Attention
(b) Offect Network
Figure 3: Deformable Attention. (a) The process of deformable attention from the tensor view and coordinate view. (b) The structure of the offset network, marked with the size of feature series.
Figure 3 introduces the detailed process of our deformable attention. In each attention module, it first samples few important time points from the input feature series X based on the learnable offsets. Then the sampled important time points are fed to the key and value projections to get the sampled key and value tokens K, V. Meanwhile, the input feature series X also projected into queries Q Finally, standard multi-head attention 43 is applied to Q, K, V to obtain the attention output
Sample the Important Time Points As shown in Figure 3 (a), we sample the important time points based on set of learnable coordinates called sampling points. Specifically, the sampling points are calculated by a set of uniformly sparse reference points and their learnable offsets.
first generate the sparse reference points Tref E TRMXXINNasm from a 1D) uniform grid. The grid size Nsamp = N/r is downsampled from the input series length N with downsampling factor r to provide sparsity. The reference points indicate the ID coordinates of some time points uniformly distributed in the feature series X with interval r. These coordinate values are normalized 1-1,+1], where -1 indicates the start of the series and +1 means the end of the series. And these reference points serve as the initial coordinates for the following deforming process.
Given length-N feature series X E R-MxXN
Then we obtain the offsets for each reference point by offset sub-network (Figure 3 (b)). It contains two convolution layers. The first layer is depth-wise wise convolution, which take the local neighbors into consideration when generating the offsets [48] It takes the query tokens Q as input, where Q is the linear projection of the feature series X. After nonlinear activation, the output from the first. layer is passed into point-wise convolution layer to generate the offsets AT E xMMxxNN.
Adding up the reference points with the learnable offsets, we obtain N Neamp sampling points, which can serve as the final coordinates to sample the important time points from the feature series X. In practice, follow [48 60) and calculate the values of these important time points by linear interpolation ((::) to make this sampling process differentiable. The overall process is as follows:
AT Offset-Networt Q)
samp = Tref + AT
(7)
X = ((X; T Tsamp)
(8)
5

---

where X E RMMDDNNN Namm is the sampled feature series consisting of the important time points. And the implementation of linear interpolation o(::-) in Appendix I.1 And we clip T samp by and +1 to avoid sampling outside the feature series.
Calculate Attention Output In above sampling process, we have got the query tokens Q After the sampling process, we can get the sampled key and value tokens K, V after two linear projections of the sampled feature series X. Then we calculate the multi-head self-attention with H heads as:
O (h) Softmax (Q ( /vd B 1 H
O OutpuutProjection(CCncce (0
0 0(H)))
(10)
where d= d-D/H is the dimension of each head. The upper index (A) denotes the h-th attention head. After concatenating the output embedding from each attention head ((h) together, we obtain the output of the DeformableAAtentiin deformable relative position bias to provide the positional information into the attention map and its implementation introduced in Appendix I.2
module O E Mx through linear projection. B the
To conclude, this subsection introduces the detailed process of DeformableAttention (Eq. And for the i-th block, X in this subsection corresponds to X| in Eg. and O corresponds to X global
4 Experiments
We thoroughly evaluate our DeformableTST on a wide range of time series forecasting tasks, including long-term forecasting tasks with various input lengths, as well as multivariate and univariate short- term forecasting tasks that are unsuitable for patching, to verify the performance and applicability of our DeformableTST.
Baselines We extensively include the latest and advanced models in time series community as strong
baselines, including patch-based Transformer models: Pathformer [6). CARD [51) GPT4TS (58), PatchTST [35]: non patch-based Transformer models: iTransformer 222, FEDformer [57] Auto-
former 47 other non Transformer-based models: RLinear [18), TiDE (). TimesNet [46]. DLin-
ear [52) and SCINet (20). We also include the state-of-the-art models in each specific task as additional baselines for a comprchensive comparison.
Main Result As shown in Figure 4 our DeformableTST achieves consistent state-of-the-art performance in a broader range time series tasks. In details, DeformableTST can flexibly adapt to multiple input lengths and especially achieve excellent performance in tasks unsuitable for patching, which is great improvement than previous Transformer-baseee models, proving that our Deformable can successfully reduce the reliance on patching and broaden the applicability of Transformer-based models. Experiment details and result discussions of each task are provided in following subsections. In each table, the best results are in bold and the second best are underlined.
Lorg-temm Foreasiigg with hapes Length 9 (MSB)
Madel Perforaance ander Defferest Japert Leagiss
Deformaarmnn
Pethormar
GORD
Enkerrate
tengtennn Foreasting with fepai Leegth 3M (MSE)
PMEOTTTT
Sharttene Fornensing (SMAPE) pas
IXformnlc (Oars)
Praiforne:
CARO
liffs
GPT4TS
PacE7ST
FEDforKr
Asaximar
Blincr
Multraritee Stariters Forecating
Leegetene Forncasing
TDE
96
304
100
with tepat Length
Japort Length
Figure 4: Model performance comparison (left) and performance under different input lengths (right).

---

4.1 Long-term Forecasting
Setups We conduct long-term forecasting experiments on popular real-world benchmarks, in- cluding Weather [45]. Traffic [9) ECL [42]. Solar-Energy [34 and 4 ETT datasets [55]. In this paper, we refine the evaluation approach for a comprehensive comparision of the models. Dif- ferent from previous settings that use fixed short input length (e.g., 96)
[55] 47 (22] We fix three different input lengths {96, 384, ]68} and calculate the averaged results to adequately reflect model's adaptability to multiple input lengths. These input lengths covers a variety of real-world appilication scenarios, i.c., shorter than prediction lengths, within the prediction lengths' interval and longer than prediction lengths. Following the previous settings, we set prediction lengths as (96, 192, 336, 720) and calculate the MSE and MAE of multivariate time series forecasting as metrics.
Results Table I shows the excellent performance of DeformableTST in long-term forecasting. Concretely, DeformableTSTT gains the best performance in most cases, surpassing extensive state- of-the-art Transformer- based models. As shown in Figure 4 (right), DeformableTSTT achieves the consistent state-of-the-as performance in all input lengths and gains continuous performance improvement with the increasing input length, validating its adaptability to multiple input lengths and its effectiveness in extracting useful information from longer history. For comparison, the non patch- Transformer baselines suffer from performance degradation with increasing input length due to the distracted attention on the prolonging input. And the patch-based Transformer baselines can not work well with a short input length (c.g., 96) because leveraging patching on the short time series leads to very few tokens, limiting attention's 's ability in long-term modeling.
Table I: Multivariate long-term forecasting results. A lower MSE or MAE indicates a better performance. Results are averaged from three input lengths / E {96, 384, 768) and four prediction lengths T E {96, 192, 336, 720). See Table g
g
10 for full results with more baselines.
Models
DedermsheesT
(Curs)
Puthfomner
6
CARD 11
GPTETS 13
PackiST
Mansshemer
FEDioener
57
Autofonner
47
RLinsar
TIDE
TimesNet M
Metric
MSB
[MSE MAH MAE]
MAE]
MARI
MSE
MSE
MSB
MAE
MSE
MAH
MSSE
MAE
ETTbl
[0.413
0.430
[1,39 0,445 fo 430 0.438
(0.79 0.49 0,444
0.461 0.463 0,505 0.415 (0.43 4:435
0.42 0.434
0,505
0,492 (3.52
0.425
ETT52
10.336
[0.66]
0.40 103 |.391 0.25 0.304
0,417 (U44) (0.37 0.485j0.358 0.39 0,499 0,505 Jo436 0.450
ETTml
[0.358
[0.375
1.394j0.665 0.356/0.373
0.93 0.301 00386 0,405 0.456 0.402 (0.56 0.523|0.379 0.390 0,383 0,297 (o4412 0.414
ETTm2
[0:662
0.111
(0,29
oo 2008 111 (0.246 1:911 0.127) (02881 0,355 0.515 0.396j0 547 ssoju.20 1.330) 0.110
J0 0.551
Weather
[6.23
0.266
[0,290 0.200jo 247 1.27//0 (49 7000 0,700 0.364 0,206 DJSI7 o 0.000013 337 n 505/0/755 0.765 0.754 0:704 (1:271 0.709
Solar- Saerr-Feeryyyuue
(0.199
0,355
mooo 27N 2.242 jo 759 0:318jou34 0,305 0.714 0,706 00904 0.400 906 773 320 0.777 0.751 (0.50) 0.319
ECL
[0.100
(0.166
0.20
puzze 0.200jo.152 0.277 0.271
0,271 (0,227 20.216 0.321 j0.189 0,250 0.201 0.300 (o.213 0.315
Traffic
[0,400 0,250
|0.44
0312j0-226 jo.4s 0.00po.330 0.200 0.424 0,306 00.85
0.42310 12. 4.33j0.300 0.330 0,579 0.401
(a.593 0.318
4.2 Short-term Forecasting
Time series community are currently focusing long-term forecasting tasks, where the input length and prediction length are adequate for patching technique. However, the short-term forecasting tasks, where the input length and prediction length are limited, are also of extensive practical value in real-world appilications. The study on short-term forecasting tasks has been stagnant in recent years and the advanced patching technique proposed in long-term forecasting hard to apply to short-term forecasting due to the limited input length. To validate the applicability of our DeformableTSS in short-term forecasting, we extensively conduct experiments in following two kinds of tasks.
Setups of Multivariate Short-term Forecasting We conduct multivariate short-term forecasting experiments on 8 popular roaldworld benchmarks, including Exchange [17]. ILI [3) 2 ETTh (55) and 4 PEMS datasets [5]. We set prediction lengths as {6. 12, 18) and set the input length to be 2 times of the prediction length, which precisely meets the definition of limited input lengths in short-term forecasting. We calculate the MSE and MAE of multivariate time series forecasting as metrics.
Setups of Univariate Short-term Forecasting The study on univariate short-term forecasting tasks used to be popular in the early time series community 29] 31 37 but has been stagnant in recent years. In this paper, we bring back this classic tasks and conduct experiments on following datasets: MI (29,, M3 30), M4 BT. Tourism MI, NN5 4I), Hospital 122 and KDD Cup [33. Following the
classic settings 229 37 we calculate the SMAPE metric. Specially for M4 datasets, we follow the
7

---

rules of M4 competition [31 and use MASE and OWA as additional metrics. The prediction lengths are from 2 to 48 and the input length 2 times of the prediction length.
We emphasize the difference between multivariate and univariate tasks as follows. In multivariate tasks, all input samples are obtained by sliding window from the same multivariate long series. Therefore, there is high degree of similarity among the input samples in multivariate tasks. In contrast, the input samples in univariate tasks are collected from many different univariate time series sources. As a result, the input samples in univariate tasks may differ from each other and have quite different temporal property, making the univariate short-term forecasting tasks much more difficult.
Results As shown in Table 2 and DeformableTSTT performs excellently in short-term forecasting. Compared with the second best model, it achieves averaged 14.1 14.1% SMAPE promotion in univariate tasks and averaged 25.6% MSE promotion in multivariate tasks. As a comparison, the limited input length in short-term forecasting poses a dilemma for patch-based Transformer forecasters. Using patching will lead to very few tokens, making it unable to fully utilize the long-term modeling ability in attention. Not using patching will lead to the distracted attention on all input time points, making hard to extract non-trivial temporal information. As a result, previous patch-basee Transformer forecasters fail in many cases of short-term forecasting tasks. And in multivariate short-term forecast- ing, the variate correlation plays an important role to the final results for the temporal information is limited due to the limited input length. Therefore, CARD and iTTransfommer which can learn the variate correlation, achieve ideal performance in PEMS datasets whose variate number is very large. As variate- intieedeiddeeeneent method, our DeformableTST still competes favorably with these cross-variate methods, further demonstrating its excellent temporal modeling ability to extract useful information even from the limited inputs. It will be our future work to study how to capture the multivariate correlation in our model, which will further improve the performance. Meanwhile, the great diversity in univariate samples makes it more difficult to learn temporal representation in univariate short-term forecasting. Therefore. the linear baselines and iTransformer, which adopt linear layers the temporal dimension, suffer from inferior performance due to the insufficient representation capability in linear layers. By contrast, thanks to the better representation capability in attention mechanism and the better focusing capacity in the proposed deformable attention, our DeformableTST is particularly good at short-term forecasting tasks, which is a great improvement than previous Transformer- based models, therefore successfully broadening the applicability of Transformer-bar based models.
Table 2: Multivariate short-term forecasting results. A lower MSE or MAE indicates a better performance. Results are averaged from three prediction lengths T E (6, 12, 18). And the PEMS results are further averaged by four subsets. Full results and more baselines are listed in Table I1
Models
DebrmabbeTST
(Oura)
Pattoomner B
CARD
GPTETS
PocaTST
A3
iTaassommer
i F1i.tomee
Sx
Aurafremer
RLinear
[3
TIDE M
TimeeNet
[b]
Metric
MSS
MAH
MMSE: MAH
MAL MSE MAlI MSE: MAll
MSE
MAL: MSB
MAL:
MSI
MAll
MSt: MAR MSE MAL
MAE
ETTh1
0.351
[0456 0.424 to 400 0:433 jo.t5 0.427 0.511 0467 (0 0.12 jo
0,445 joaa9
0,459 0.343 uu5s f0 o sos 4.440
0.419
ETTh2
[0.142
[8.190 (u55 163 0.262 (ouse jouss 0,263 0.12 0:300 164
10.176 0.20 0,208 2000 192 200/ou111 0.286
ILI
[1767
0,798
[3:100
1.37] 1.800
1,237 4217
(3510 1.009 |a.07 1.309443000
1.523 5.202 1,011 | 1.65|
0,300
Exchange
[0.033
0,070
[n.015
9.077
joots 0.083 jours 0,077 0.016 0.085
(1014 10023
jouzs 0.119/00026
0,117 0,022 0.100 (0.019 cu93oous
0.01
PEMS (Avg)
10.00
0.208
|a.137 0,235
ios n.215 (0.133 0,299 0,140 0.254/o.102 0.305
Jo.122 0.29/oo.47
0.268 0.165 0.272 jolues (o.10 0,377
Table 3: Univariate short-term forecasting results. Lower metrics indicate better performance. Weight Avg and Avg means the results are (weighted) averaged by subdatasets. We only report the SMAPE as metric here. Full results with more metrics for M4 are provided in Table [2 and [3] *. in the Transformers indicates the name of *former.
Models
DifommbleTST Pub. (Qurs)
CARD GPTHTS Cress PachTST Trassfoneer
FED.
Asto
RLiness
TIDE
TimesNer
Diinew
SCINer N-JTTS N-BEATS
X
S3
53
53
[33
22
533
53
[5
8
[6
[
[3]
[3
MI (Aug)
15,250
18,66
18:00 19:771
300055
19.622
21.676
30,096
31.066
23.199 77.119 19.176
24.433
264000
10.100
MT (Aog)
11:47
19:24
15.181
19/003
14214
14721
13,055
988
19:61
34,201
20.36
18.710
20.568
24.226
20,969
17,823
M44 (Wegge
11688
12:001
11,115
12/008
134455
11,878
12.540
12,009 13.35%
13.711
11,829
13,639
12,59
11,927
11.851
Touiso (tog)
21.563
31.345
23.349
35,018
236188
23.442
24,429
37,205
35.793 39:756
04.504
33.279
37.353
37.730
351144
42.852
NM5
14372
18186
22.491
16.012
17672
13.317
20,449
17,072
196500 22.868 23.424
22.355
23.751
25.456
16.655
23.252
Bequal
190088
21.551
21.392
21.587
20/002
22.123
20.755
28.646
25.749
25.16
20.15
21.132
23,015
29.807
25:309
23.594
55,300
61.63
54,713
60.15
50.013
57.617
63
56.15
99.513
63.358
$4,855
63.305
8

---

Model Analysis
5.1 Ablation Study
0.411
P $1.10
A,20
(8) Ablation results In Solar dattacee with Iaput length 96
(b) Ablattoo roullss Traffle datiset with Ingut trngth
Figure 5: Ablation study in long-term forecasting tasks. From the top to the bottom, each row means one design that we add on PatchTST to modify it into our DeformableTST. We report the averaged MSE of four prediction lengths. The memory usage is recorded under poput-96-predice with the same batch size. A lower MSE (a shorter blue bar) and a smaller memory usage (a green star closer to the vertical axis) means better performance and efficiency. More results are in Appendix F.
setting
In order to validate the effectiveness our designs, we start from a PatchTST 35 and gradually update it into DefOrmableTST by adding designs step by step. And provide the trajectory going from a PatchTST to a Deformable in Figure(5] To get rid of the over-reliance on patching, remove the patching design in PatchTST. After this step. PatchTST suffers from degradation in both performance and efficiency. To address these issues, more designs are adopted in DeformableTSTT
First, the removal of patching leads to larger number of tokens processed in the attention computa- tion, resulting in heavier memory usage. Responding to the issue, we adopt a hierarchical structure to gradually reduce the number of tokens, therefore alleviating the efficiency problem. Secondly, full attention hard to focus on the important time points after removing the patching design, leading to trivial temporal representation and performance degradation. To help attention better focus without patching, we propose deformable attention as a complementary. Thanks to the better focusing ability in deformable attention, observe great performance improvement after adding this design. Mean- while, we also adopt some local enhancement modules in our design since locality is also important in time series [20]. And this step also brings performance improvement. After equipped with all our designs, DeformableTSST shows great performance and efficiency superiority than the baseline PSt which proves the necessity and effectiveness of our designs.
5.2 Compared Deformable Attention with Prior-based Sparse Attention
In Section L we propose that sparse attention can help attention to better foucs without patching and further argue that data-drivien sparse attention is more appropriate than prior-based ones. To validate the necessity and effectiveness of using data-drivien sparse attention, we compare our deformable attention with some prior-based sparse attentions in time series community, that is ProbSparse Attention in Informer [55], AutoCorrelation in Autoformer [47] and FourierAttention in FEDformer [57]. We also include the local window attention in Swin Transformer ( and the for comprehensive comparison.
vanilla full attention 43 adopted in most baselines 22 53 35
As shown in Table our deformable attention surpasses other prior-based competitors in all bench- marks. This is because the priors are hard to match all kind of inputs due to the diverse pattern in different time series, resulting in the inferior performance of brior-based sparse attentions. Different from them, our deformable attention is a data-driven sparse attention that can learn from the input time series, therefore is more flexible to the diverse property in different time series.
Our deformable attention also surpasses the full attention by a large margin for it can better focus on the important time points to learn non-trivial temporal representation, while the full attention suffers from the distracted attention and trivial temporal representation due to the lack of patching. Although window attention can help attention avoid being distracted in global range by limiting the attention computation into local windows, its performance still decreases due to the lack of long-term modeling ability, which is an important ability time series forecaster should have.

---

Table 4: Comparison of our Deformable Attention with other prior-based Sparse Attentions. We replace our Deformable Attention with other prior-based Sparse Attentions for comparison. We conduct the experiment in long-term forecasting tasks with input length 96 and list the averaged MSE/MAE of four different prediction lengths. More results are in Appendix
Dataset
Er1hi
ET132
ETTinl
ETTm2
Weather
Solar
BCL
Traftic
Metne
MSE
MAR
MSE
MAE
MSE
MAE
MSE
MAE
MSS
MAE
MSE
MAE
MSE
MAE
MSL
MAB
Deformable Attention
0.425
0.428
0.346
0.382
0.373
0.395
0.283
0.327
0.251
0.276
0.224
0.256
0.183
0.278
0.445
0.285
ProtSpaane Amention
0.451
0.434
0.379
0.407
0.390
0.504
00.95
0.338
0.267
0.287
0.270
0.291
0.201
0.295
0.485
0.527
AutoCorelatios
0.469
0.443
0.355
0.401
0.383
0.334
0.255
0.215
0.302
0.373
FourierAnettion
0.333
Window Attention
0.441
0.398
0.295
0.488
0.327
6 Conclusion and Future Work
In this paper, we expose an emerging issue faced by advanced Transformer-base models that they have limited upplicability in time series forecasting tasks due to their over-reliance patching. And we further find out the reason behind this problem is that full attention relies overly on the guidance of patching to focus the important time points and Iearn non-trivial temporal representation. To tackle this problem, we propose DeformableTST as an effective solution, which equips with deformable attention that can better focus the important time points by itself to get rid of the over-reliance patching. Experimentally, Deformable' achieves the consistent state-of of-the-art performance in a broader range of time series forecasting tasks, especially achieving promising performance in tasks unsuitable for patching, therefore successfully reducing the reliance on patching and broadening the applicability of Transformer-basee models. And hope findings prompt people rethink the relationship between Transformer-basse models and patching technique, thereby designing more powerful Transformer-basee forecasters with wider range of applicability.
Acknowledgment
This work was supported by the Hebei Innovation Plan (20540301D).
10

---

References
[1] George Athanasopoulos, Rob J Hyndman, Haiyan Song, and Doris Wu. The tourism forecasting competition. International Journal of Forecasting, 27(3):822 844, 2011.
[2] Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Sohlke-Sccheider David Salinas, Lorenzo Stella, et Deep learning for time series forecasting: Tutorial and literature survey. ACM Computing Surveys, 55(6): 2022.
[3] CDC. Illness. https://gis ctc. http:///w.....a..lll//////////ta
html
[4] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series fore- casting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6989-6997 2023.
[5] Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway performance measurement system: mining loop detector data. Transportation research record, (748(1))96-022, 2001.
[6] Peng Chen, Yingying ZHANG, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. Multi-scale transformers with adaptive pathways for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[7] Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting-Fie full version. arXiv preprint arXiv:2204.13767 2022.
[8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773. 2017.
[9] Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. arXiv preprint wxXiv:2304008424 2023.
[10] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12175-12185, 2022.
[11) Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1606 004....84555... 2016.
[12] Hyndman, J. expsmooth: Data Sets from Forecasting with Exponential Smoothing. https 7/cran. (Crran.-project... httpeet..orr/ppccccceeexxppooooo
[13] KDD. Kdd cup 2018. https:///wwt www. kdd. wttp:o//////////oooooooo
[14] Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang, Dong Gu Lee, Wonseok Jeong, and Sang Woo Kim. Dead pixel test using effective receptive field. Pattern Recognition Letters, (67:149-156. 2023.
[15] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
[17] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, 2018.
[18] Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: 2023.
investigation on linear mapping. arXiv preprint
11

---

[19] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey, Philosoph- ical Transactions of the Royal Society A, 379(2194):20200209, 2021.
[20] Minhao Liu, Ailing Zeng, Z Xu, Q Lai, and Q Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. In 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
[21] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In International conference on learning representations, 2021.
[22] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long, itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference on computer vision, pages 10012-10022, 2021.
[24] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
[25] Donghao Luo and Xue Wang. Modernten: A modern pure convolution structure for general time series analysis. In International Conference on Learning Representations, 2024.
[26] Donghao Luo and Xue Wang. ModemnTN: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations, 2024.
[27] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
[28] Amal Mahmoud and Ammar Mohammed. A survey on deep learning for time-series forecasting. Machine learning and big data analytics paradigms: analysis, applications and challenges, pages 365-392, 2021.
[29] Spyros Makridakis, Allan Andersen, Robert Carbone, Robert Fildes, Michele Hibon, Rudolf Lewandowski, Joseph Newton, Emanuel Parzen, and Robert Winkler. The accuracy of extrap- olation (time series) methods: Results of forecasting competition. Journal of forecasting. 1(2):111-15,, 1982.
[30] Spyros Makridakis and Michele Hibon. The m3-competition: results, conclusions and implica- tions. International journal of forecastnng, (6(4):4514466 2000.
[31] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting. 34(4):802- 808, 2018
[32] Ricardo P Masini, Marcelo C Medeiros, and Eduardo F Mendes. Machine learning advances for time series forecasting. Journal of economic surveys, 37(1):76-11,, 2023.
[33] John A Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, Budak Arpinar, and Ninghao Liu. A survey of deep learning and foundation models for time series forecasting, arXiv preprint arXiv:2401 133112221.1111,,, 2024.
[34] National Renewable Energy Laboratory. Solar power data for integration studies. https gov www.eelgwwoggidd/llaa--ooreeeeee htm]
[35] Yugi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arX41.2211.147700 2023.
12

---

[36] Oleh Onyshchak. Stock market dataset. Stttps:/wvie www.we.gg.....m wwtpt/oooo///////eeee 2020.
[37] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, arXiv preprint 19Xiv:1905.1044 2019.
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[39] PeMS. Traffic. http://pent dot. ca. gov/
[40] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Iyya Sutskever. Language models are unsupervised multitask learners. 2019.
[41] Souhaib Ben Taieb, Gianluca Bontempi, Amir F Atiya, and Antti Sorjamaa. A review and comparison of strategies for multi-step ahead time series forecasting based on the nn5 forecasting competition. Expert systems with applications, 7001 2012.
[42] UCI.
Electricity.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[44] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma. Junchi Yan, and Liang Sun. Transformers time series: survey, arXiv preprint arXiv:2202.07125, 2022.
[45]
Wetterstation. Weather. bgtp:::///wbbbbccjjeee mwp.enaa..ppp....emmm.m.e....... wwtp:/pgg...eeeeeeeeeee de/wetter/
[46] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou. Jianmin Wang, and Mingsheng Long, Times- net: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186, 2023.
[47] Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 14:22419--22430 2021.
[48] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4794-4803, 2022.
[49] Zhuofan Xia, Xuran Pan, Shiji Song, Erran Li, and Gao Huang. Dat++: Spatially dynamic vision transformer with deformable attention. arXiv preprint arXiv:2309.01430 2023.
[50] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, 2021.
[51] Wang Xue, Tian Zhou, QingSong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Make transformer great again for time series forecasting: Channel aligned robust dual transformer. arXiv preprint arXiv:2305. / /2095, 2023.
[52] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. 13504, 2022.
[53] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representa- tions, 2023.
[54] Zhenwei Zhang, Linghang Meng, and Yuantao Gu. Sageformer: Series-aware framework for long-term multivariate time-series forecasting. IEEE Internet of Things Journal, 11:8435- 18448, 2023.
13

---

[55] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115,
2021.
[56] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Rong Jin, et al. Film: Fre- queney improved legendre memory model for long-term time series forecasting. arXiv preprint arXiv:2205.08897 2022.
[57] Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference on Machine Learning, pages 27268-27286. PMLR, 2022.
[58] Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained LM. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[59] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9308-9316, 2019.
[60] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
14

---

A Datasets
Table 5: Detailed descriptions of multivariate datasets. The Dataset Size denotes the total number of time points in (Train, Validation, Test) split respectively.
Tasks
Dataset
Variates
Prediction Length
Dataset Size
Frequency
Information
ETTh1
7
196, 192, 336, 720]
(8545, 2881, 2881)
Hourly
Electricity
ETTh2
7
196, 192, 336, 7201
(8545, 2881, 2881)
Hourly
Electricity
Long-term Forecasting
ETTml
7
192, 336, 720]
(34465, 11521, 11521)
15min
Electricity
ETTm2
7
192, 720)
(34465, 11521, 11521)
15min
Electricity
Weather
196, 192, 336, 720]
(36792, 5271, 10540)
10min
Weather
Solar-Energy
137
196. 192, 336, 720]
(36601, 5161, 10417)
10min
Energy
ECL
321
196, 192, 336, 720)
(18317, 2633, 5261)
Hourly
Electricity
Traffic
862
196. 192, 720]
(12185, 1757, 3509)
Hourly
Transportation
PEMS03
358
[6, 12, 18]
(15617, 5135)
5min
Transportation
PEMS04
307
18]
(10172, 3375, 3375)
Smin
Transportation
Short-term Forecasting
PEMS07
883
[6, 12, 181
(16911, 5622, 5622)
Transportation
PEMS08
170
(6, 12, 18]
(10690, 3548, 3548)
Smin
Transportation
Exchange
8
[6, 12, 18]
(5120, 665, 1422)
Daily
Illness
IIII
7
18]
(676, 96, 194)
Weekly
Feonomy
ETTh1
[6, 12, 181
(8545, 2881, 2881)
Hourly
Electricity
ETTh2
7
(6. 12, 18]
(8545, 2881, 2881)
Hourly
Electricity
Table 6: Datasets and mapping details of univariate short-term forecasting datasets.
Dataset
Sample Numbers (train set, set)
Variate Numbers
Prediction Length
MI Yearly
(181, 181)
2
MI Quarterly
(203, 203)
3
MI Monthly
(617, 617)
8
M3 Yearly
(645, 645)
3
M3 Quarterly
(756, 756)
4
M3 Monthly
(1428, 1428)
10
M3 Other
(174, 174)
10
M4 Yearly
(23000, 23000)
6
M4 Quarterly
(24000, 24000)
8
M4 Monthly
(48000, 48000)
18
M4 Weekly
(359, 359)
13
M4 Daily
(4227, 4227)
14
M4 Hourly
(414, 414)
48
Tourism Quarterly
(427, 427)
5
Tourism Monthly
(366,, 366
15
NN5 Weekly
(111, I11)
15
Hospital Monthly
(767, 767)
10
KDD Cup Hourly
(270, 270)
48
A.1 Multivariate Long-term and Short-term Forecasting Datasets
We evaluate the multivariate long-term term forecasting performance on 8 popular real-world datasets, including Weather, Traffic, ECL, Solar-energy and ETT datasets (ETThI, ETTh2, ETTml, ETTm2). And for multivariate short-term forecasting tasks, we choose Exchange, III, 2 ETTh datasets and PEMS datasets for benchmarking. These datasets have been extensively utilized for benchmarking and cover many aspects of life.
15

---

The variate number, dataset size and sampling frequency of each dataset are summarized in Table 5S. We follow standard protocol [55] and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT and PEMS dataset and 7:1:2 for the other datasets. And training, validation and test sets are zero-mean normalized with the mean and standard deviation of training set. Each of above datasets only contains one continuous long time series, and we obtain samples by sliding window.
More introduction of the datasets are as follow:
1) Weather contains 21 meteorological indicators of Germany in 2020.
2) Traffic contains the road occupancy rates measured by 862 different sensors on San Francisco Bay area freeways in 2 years.
3) (ELeetttyyy contains hourly electricity consumption of 321 clients from 2012 2014.
4) ETT(Electricity Transformer Temperature contains the data collected from two different elec- tricity transformers with 2 different resolutions (15 minutes and hour) by sensors.
5) Solar(Solar- Enneryg contains 137 time series representing the solar power production in Alabama state in 2006.
6) PEMS is collected from California freeway and contains 4 subsets.
7) Exchangd the daily exchange rates of eight different countries ranging from 1990 to 2016.
8) IL((Infuenza-LLike Illness contains 7 indicators of influenza-like illness (ILI) patients in the United States between 2002 and 2021.
A.2 Univariate Short-term Forecasting Datasets
We conduct univariate short-term forecasting experiments on 7 popular datasets, including M1, M3, M4, Tourism, NN5, Hospital and KDD Cup. We emphasize the difference between multivariate and univariate tasks as follows. In multivariate tasks, all input samples from the same multivariate long series. Therefore, there a high degree of similarity between the input samples in multivariate tasks. In contrast, the input samples in univariate tasks collected from many different univariate time series sources. As a result, the input samples in univariate tasks may differ from each other and have quite different temporal property, making the univariate short-term forecasting tasks much more difficult.
obtained by sliding window
Table summarizes details of statistics of univariate short-term forecasting datasets. And more introduction the datasets are as follow:
1) MI MI contains 3 subsets with different frequency: Yearly, Quarterly and Monthly. The series are belonging to 7 different domains: macro 1, macro 2, micro 1, micro micro 3, industry and demographic.
2) M. contains 4 subsets with different frequency: Yearly, Quarterly, Monthly and Others. The series are belonging to 6 different domains: demographic, micro, macro, industry, finance and other.
3) M4 contains subsets with different frequency: Yearly, Quarterly, Monthly, Weekly, Daily and Hourly. The series are belonging to wide range of economic, industrial, financial and demographic areas.
http://wwwwbbbcjee.. jena.mpg.aae......mmm de/wetterr 'tttps//ppessddot wot.ca.goo. co.go//
'https:/giithub.cc httpps//gituub..oo//boooooooo how..///./..cooaoo////EDtttttttt
"http://pems.dot..
wttp::/www.....oooooooo gow/
7
10 "hutps://ooi.co http:///.....r..........//////.0 01000000111000002000000000000000
10. http://d...oogl11ooooo..occsssod
16

---

4) Tourism contains 3 subsets with different frequency (Yearly, Quarterly and Monthly) used in the Kaggle Tourism forecasting competition. Considering the dataset size, we only use Tourism Quarterly and Tourism Monthly.
5) NN 13
contains weekly time series from the banking domain.
6) Hospital Datase contains 767 monthly time series that represent the patient counts related to medical products from January 2000 to December 2006.
7) KDD Cup contains 270 hourly time series representing the air quality levels by multiple measurements such as PM2.5, PMIO, NO2, CO, 03 and SO2.
B Experiment details
B.1 Long-term Forecasting
Our method is trained with the L2 loss, using the ADAM [16] optimizer with initial learning rate in {,0- x 10 "0=*,10-=" 10040.4..... The default training process is 50 epochs with proper early stopping. The mean square error (MSE) and mean absolute error (MAE) are used repeated times with different seeds and the means of the metrics
Implementation Details
metrics. All the experiments are reported as the final results. All the deep learning networks are implemented in Py PyTorch 38 and conducted on NVIDIA A100 40GB GPU.
Model Parameter By default, DeformableTST contains 4 Transfomer blocks. And we adopt downsampling layers between two blocks, which will halve the series' length and double the model dimension. The dimension D of the first block is set as 16. The expansion a is set as 4. The number of important time points N Neamp is set as 12. We optionally adopt non-overlap patching depended on the input lengths. When input length 96, we do not adopt patching. When input length is 384, the patch size is 4. When input length is 768, the patch size is 8. For baseline models, if the original papers conduct long-terd term forecasting experiments on the dataset we use, we follow the official codes with the recommended model parameters in the original papers, including the number of blocks, model dimension, etc. Otherwise, their model parameters are searched from following searching space: number of blocks L from {2, 4, 4),, model dimension D from (64, 128, 256) and FFN expansion a from 21))
Metric We adopt the mean square error (MSE) and mean absolute error (MAB) of multivariate time series forecasting as metrics.
T
MSE-
Σ
YY
Y,)²
T
i=0
1 T
MAE
Σ
Y: Y,
where Y, Y T are the M variates prediction results of length T and corresponding ground truth. Y, means the i-th time point in the prediction result.
B.2 Multi-variate Short-term Forecasting
Implementation Details Our method is trained with the L2 loss, using the ADAM [16] optimizer with an initial learning rate in {10-3,5 30-3,, 10 10000-9 4). The default training process is 50 epochs with proper early stopping. The mean square error (MSE) and mean absolute error (MAE) are used as metrics. All the experiments are repeated times with different seeds and the means of the metries are reported as the final results. All the deep learning networks are implemented in PyTorch 38 and conducted on NVIDIA A100 40GB GPU.
compraacttooccttttttooooccttttto bttps:/www.kdd..oo wwtp://wwww..........kuok.oo.o..
17

---

Model Parameter By default. DeformableTST contains 6 Transformer blocks with the model dimension D 256 and FFN expansion a The number of important time points samp is in the range of to 12 in short-term forecasting tasks, which is depended on the different input lengths. Due to the limited input lengths, we do not adopt patching and downsampling layers in short-term forecasting tasks. For baseline models, we follow the official codes with the recommended model parameters and some of their model parameters are re-searched from following searching space: number of blocks L from {2, 4, 6}. model dimension D from (64, 128, 256} and FFN expansion from 21,2,4,8)
Metric We adopt the mean square error (MSE) and mean absolute error (MAE) of multivariate time series forecasting as metrics.
T
MSE
Σ
-
Y,)
T
t=0
T Σ 1=0
MAE 7
Y Y.
where Ý, Y E RTxM are the M variates prediction results of length T and corresponding ground truth. Y, means the i-th time point in the prediction result.
B.3 Univariate Short-term Forecasting
Implementation Details Our method is trained with the SMAPE loss, using the ADAM [I6]
optimizer with an initial learning rate of 5 x 10-4. The default training process is 50 epochs with proper early stopping. Following 46. we fix the input length to be 2 times of prediction length for all models. All the experiments are repeated 5 times with different seeds and the means of the metrics are reported as the final results.
Model Parameter By default, DeformableTST contains Transformer blocks with the model dimension D 256 and FFN expansion 4. Due the limited input lengths, we do not adopt patching and downsampling layers in short-term forecasting tasks.
Metric For the M4 datasets, following B1 37]. we adopt the symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE) and overall weighted average (OWA) as the metrics, which can be calculated as follows:
SMAPE 200
T
Y,
100
T
Y;
Y,
MAPE
T
+
Y
T
Y
2=1
1 Σ
T
Y;
Y1
SMAPE
MASE MASE
MASE
OWA 2
T
-+p Y, Y-pll'
SMAPE
where is the periodicity of the data, Y, Y E RTXM are the M variates prediction results of length T and corresponding ground truth. Y, means the i-th time point in the prediction result.
For other datasts, adopt the symmetric mean absolute percentage (SMAPE) the metric, which can be calculated as follows:
SMAPE 200 T
Y
Y, + Y
Y
where Ý, Y e T are the variates prediction results of length T and corresponding ground truth. Y, means the i-th time point in the prediction result.
C Pseudo-code of DeformableTST
We provide the pseudo-code of DeformableTST in Algorithm
18

---

Algorithm DeformableTST - Overall Architecture.
Require: Input time series iin BBIXXM; batch size B; variates number M; input length Z;
prediction length T; DeformableTST block number L; feature series's embedding dimension in the i-th block D;; feature series's length in the i-th block N,. Boolean flag to indicate using
downsampling layers or not tase__Dossaamlllee
1: Xin = RevIN X mode=norm)
D E RBXIXM
2: Xin Xin transpose
X E R
3: Xin reshaee
Xin (BM)xxI
4: > Embedding the input time series variate-inddeenndetll with optional patching.
5: Xo (mbeddinggX(n)
> X
R BB No
6: for in (1....,4);:
Run through DeformableTST blocks.
7: The local perception unit (LPU) is used to learn the local temporal information.
8: Xlocal LPU(X.,1)
x local
E RR,BM)D,xN,
9: The deformable attention is adopted to learn the global temporal information.
X globol LayerNorm X Local Def Deformmttentiion (X(oo))))
X-gobal E RR(MM)xDXN,
11:
The feed-forward network injected with depth-wwise convolution (ConvFFN) is used to
learn the local temporal information and the new feature representation.
12:
X; LayerNorm X.global ConvFFM(X:llobu))
X, E R(BM)xD.XN,
Adopting the optional downsampling layer between two blocks.
14:
if < L and ingss is True:
15:
X; DownSampling(X.)
X; E (BM))
End for
17: X = re.happe
b XL E BBXMX(D
18: Y = Projection(X.))
Obtaining the forecasting series with Projection, Ý E RBMMT
19: Y Y.transpose
Y BBXXXM
Y RevIN mode=denor))
D B
21: Return Ý
D Return the prediction result Ý
D Parameter Sensitivity
To evaluate the parameter sensitivity of our DeformableTST,, we perform experiments with varying
model parameters, including number of blocks ranging from L = {2, 4, 6), model dimension ranging from {16, 32, 64). FFN expansion ranging from a {1, 4, 8}. number of important time points ranging from N Nsamp 10-4).
(6. 12, 24} and learning rate ranging from Ir = (10-
The results are shown in Figure 6 In general, model is robust to the choice of model parameters. Compared with the default block number L = 4, stacking more blocks will bring further performance improvement. Considering both performance and efficiency, recommend to fix the block number as in long term forecasting tasks.
We also compare our model sensitivity to patch size with PatchTST's. As shown in Figure 6 our model is less sensitive to the choice of patch size, therefore successfully getting rid the over-reliance of patching.
19

---

4.4
031
0211
020
Model Dimendion
FEN Expandion
Number of Blocks
0.21
427
Wadder prter
4.20
DeferbbeTST (wun)
0.00
PatchTST
Desoniatio Couts
3381
C01
Number Important Time
Learning Rate
Patch
Figure 6: Parameter sensitivity. For patch size, we conduct experiments under 2ouu-38---redic--96 settings and adopt PatchTST as comparison. For other parameters, we conduct experiments under 9oput-96-rredict-oe settings.
E
More ERF Results
In Section T visualize the effective receptive fields (ERFs) of PatchTST 35 based see which parts of the time points in input series are focused by the model when extracting temporal representations. Here we provide more ERF visualization results with other patch-based Transformer forecasters. In a ERF figure, a brighter area means that the model tends to focus on these time points when extracting temporal representation, and thus these time points will contribute more to the middle point of the final representation.
As shown in Figure 7 the ERFs of the patch-based Transformer forecasters highly rely the guidance of patching. If without patching, nearly all time points in input series are equally focused by the model and the model performs worse, exposing the problem of distracted attention. This finding means that attention has not learned to distinguish the importance of each time point in input series, leading to trivial representation. After patching, these Transformer forecasters tend foues on some important time points based on patch partition. This phenomenon means that: although patching can help attention better focus on important time points, the guidance of patching tend to distribute the focused points evenly among all patches. But in real-world scenarios, it is possible that the important time points are not evenly distributed among all patches, which is inconsistent with the tendency in the guidance of patching, leading to the inferior performance of these patch- based Transformer forecasters in some cases. There also some difference among the ERFs of these patch-based Transformer forecasters due to their different specific designs. PatchTST 35 tends to only focus on a very few time points each patch and ignore others. But CARD
51) can focus on more time points in one patch thanks to its additional attention across feature dimension, which aligns the information within patch. And the ERF of GPT4TS (58) also reveals the autoregressive property in GPT [40] backbone, making it only focus on the time points before the the middle point.
By contrast, we provide the ERF of our DeformableTST in Figure for comparison. By default, we set Nsamp = 12. Under input-96 settings, our DeformableTST does not adopt patching technique but can still focus on a small number of important time points, proving that our Deformable TST can foues well by itself without the need of patching. Under input-384 settings, we adopt small size patching and divide the input series into 96 patches to alleviate the efficiency issue. Under such condition, our DeformableTST still does not focus based on a patch partition. Although adopting patching technique, the focused time points in our Deformable is not evenly distributed among all 96 patches, therefore being less reliant patching.
20

---

(a) CARD not using patching
(b) CARD divided into 42 patches (Input length: 336, Token number: 42)
(Input length: 336, Token number: 336) l'ocusing on nearly all input time points
Focusing on nearly 42 imporiant input time points
(e) GPT4TS using patching (loput longth: 336, Token number: 336)
GPT4TS divided patches
(Inpur length: 336, Token number: 42)
Focusing nearly all input points before middle point
Focusing ocarly important input time before the middle point
Figure The Effective Receptive Field (ERF) of more patch-based Transformer forecasters. brighter area means that these time points are focused by the model when extracting temporal representation.
(a) DeformalleTST
(b) DefomableTTST patch size (Inpur longth: 384, Token number: 96) Focusing on nearly 12 input time points
Not using patching (Input length: 96, Token number: 96) Focusing on nearly 12 input time points
Figure 8: The Effective Receptive Field (ERF) of DeformableTST. A brighter means that these time points are focused by the model when extracting temporal representation.
F
More Results of Model Analysis in Section 5
F1
More Ablation Results in Section 5.1
More ablation results are provided in Figure
Measar
Bagg
Sterie rig Poiat
Stariigg Print
Utmere gardding
Romove Sechniqqe
AM deferaable stoetik
Add detoraatl atterias
Det menemmatieinn (om))
E.11
311500
Add leaa eskenonsest Untormaati IST oeei)
8.11
PerSonaacs (iSE))
Perforaanee (MSB)
8350
(n) results ET with input length Ablation Weather input length
Mineary Usege
Suntin Paint
Ramene panciing technique
BJ11
Arronckal
Add meaeeeeet DeformaleetT (ean)
E.151
LEMENII
(c) Ablation results ECL dataset with input length
Figure 9: More ablation study results in long-term forecasting tasks. From the top to the bottom, each row means one design we add on PatchTST to modify it into our DeformableTST, We report the averaged MSE of four prediction lengths. The memory usage is recorded under input-96-predict-96 predict-96 setting with the same batch size. A lower MSE (a shorter blue bar) and a smaller memory usage (a green star closer the vertical axis) means better performance and efficiency.
F.2
More Comparison Results in Section 5.2
Case Study We provide the visualization of learned important time points as an intuitive compari- sion for some popular sparse attentions in time series community. shown Figure the keys of window attention are restricted within the local window. And the important keys for ProbSparse
21

---

attention tend to gather in small area around the time point that most fits the prior. This property makes window attention and ProbSparse attention only focus on a small local area and fail to find the important time points in a global range, leanding to the loss of information. Meanwhile, imporatant time points refer to the time points that reflect the property of time series and make contribution to better performance. And the types of imporatant time points are varied (e.g., time point in the similar changing stage, the inflexion point, the extremal point and so on). But AutoCorrelation can only foucs on the time points in the similar changing stage due to its prior, resulting in the lack of diversity. By contrast, our deformable attention can find the important time points in a global range for the reference points in the sampling process uniformly distributed throughout the whole input time series. And our deformable attention can also learn different types of important time points because it is less reliant specific priors and can foucs any appropriate important time points based on the learnable offsets learned from the input series. Therefore, our deformable attention can find more types of important time points in wider range, therefore being more flexible to the diverse property in different time series and performing better than other prior-based sparse attentions.
Wilualow
Proddigane
Figure 10: Visualization of learned important time points, For clearness, we show the top-6 keys with respect to the last query for attentions and show the top-6 time delays for AutoCorrelation.
G Model Efficiency
We comprchensively compare the forecasting performance, training speed, and memory usage of some advanced Transformer-basee models. And we compare the efficiency under two representative conditions: (1) the dataset is of a large variate number, (2) the experiment setting is of long input length and prediction length. And the results are shown in Figure [ Considering both performance and efficiency, DSf shows great superiority than other Transformer-batt based competitors, therefore being an ideal choice in time series forecasting.
Long term forecasting In Solar (legur-9t-pen p--e-i----d (137 varlates) Effiiincc with respect large variate number
Long-torm fureczeting in ETTm2 Sonue--nt--rredie---1 (? varisttes) Efficiency with respect to long input length and prodiction length
OPTETS GATTANIL
ATSRO
FEDRERAr
DURKSEE SCASA
EU/DOIEL 192
71 FOMI, ESE TAL
FERS NEY ACTENI,
XISE
T12INE, 26056
TTrandiraee 2800N0,, 128,81
DoT 42110 234.75
Munar Teetabee
2ALINIS 111.3
100
Training Time (okeuech)
Training Time (skqpech)
Figure 11: Model efficiency comparison.
H Error Bar
We report the standard deviation of DeformableTSTT performance under five runs with different random seeds in Table which exhibits that the performance of DeformableTST is stable.
22

---

Table 7: Error bar of DeformableTST.
Dataset
ETTh1
ETTml
ETTm2
Horizon
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
0.37340.002
0.396+00.001
0.334+0.001
0.316+0.001
0.358=0001
0.178+00000
0.262:00:000
192
0.427+0.001
0.427+0.000
0.353=00002
0.382=00.011
0.354:00001
0.380=0001
0.243=+0.002
0.301
0.437.0.003
0.42610.002
0.34110.000
0.37910.000
0.379
0.405
0.310
0.3481000001
720
0.464:10.003
0.46210.003
0.410:0.004
0.431:00000
0.443:00000
0.488000003
0,400
0.398
Dataset
Weather
Enear-Energy
BCL
Trallic
Herizon
MSE
MAF
MSF
MAF
MSF
MAE
MSE
MAE
96
0. 0.1690000000
0.213:100000
0.193:0.001
0.232:10.001
0.15810.002
0:255::0002
0,418:100022
0.271 10.002
192
0.216+000022
0.255+00.01
0,225+00007
0
0.16:+0.004
0.265=00044
0.437+000004
0.276++00003
336
0.271+00.001
0.294+0.001
0,239=0.000
0.267=+0.000
0.183+00003
0.280=000044
0.449+000005
0.290+00000
720
0.34740.003
0.34411.002
0,238:0.000
0,265:10.000
0.22340.001
0.31340.044
0.477:100044
0.303:40.044
I More Implementation Details about Deformable Attention
L.1 Implementation of Linear Interpolation
we sample the important time points based on a set of learnable coordinates called sampling points T samp- In practice, after obtaining the sampling points T samp. we achieve this sampling process via linear interpolation following [48] 60]. In details, we calculate the values of these important time points by linear interpolation o(::-) make this sampling process differentiable. And the linear interpolation ((5.) is calculated as follows:
As mentioned Section 3.2
Φ (X; T samp)
(t, De-normalize T samp) ) [ 1.
(11)
where g(a, b) = max(0, 1 la of samp is de-normalized back to the range of [0, N 1] before passed into g. As g would be non-zero only on the 2 integral coordinates closest to T samp, it simplifies (q.(1T) l) to calculate the value of each important time point as the weighted average of its only closest time points.
bl) bl) and t indexes all the coordinates on X MxDNN The value
1.2 Deformable Relative Positional Bias
Due to the deforming process and hierarchical representation, the fixed absolute positional embedding not suitable for our design. Instead, we adopt a relative position bias to encode the position information into the attention map [23], which represents the relative positional information between the query token series Q and the key token series K.
Considering the input feature series X Mx DxN of the attention module, the relative coordinate displacements of this length-N series contain (2N - 1) different values and lie in the range of
[-N,N] NJ. Then we will maintain a learnable parameterized bias table B E (x)--11xx1 in which each element represents one of the above (2NV- - 1) relative coordinate displacements.
Meanwhile, to represent the relative positional information between the query token series Q and the key token series K, we also need to denote the coordinates of Q and K. The coordinates of Q are generated from a 1D uniform grid with size N. It indicates the 1D coordinates of all N time points in the query series Q. These coordinate values are normalized to [1], where -1 indicates the start of the series and +1 means the end of the series. And the coordinates of K are the sampling point samp. Then the relative position R R Noamp is calculated as follows:
R Qcoord - Transpose(T samp)
(12)
Then we clip by -1 and 1 and obtain deformable relative positional bias by linear interpolation the learnable parameterized bias table B with the continuous relative displacements as follows:
B = (B; R)
(13)
23

---

J More Discussions on Patching and Transformer-based Models
J.1 Discussions on Patching
Impact on Performance As shown in Figure [ndd] 7 although patching can help attention better focus on important time points, the guidance of patching tend to distribute the focused points evenly among all patches. But in real-world scenarios, it is possible that the important time points are not evenly distributed among all patches. In some cases, most of the important time points are only distributed in few patches while other patches only contain unimportant time points. Such cases inconsistent with the tendency in the guidance of patching, leading to the inferior performance of the batch-based Transformer forecasters in these cases. And patching technique must work with very long input length because leveraging patching on short time series leads to very few tokens, limiting attention's 's ability in long-term modeling. These drawbacks limit the performance and applicability of the previous patch-based Transformer forecasters.
Impact on Efficiency In addition to helpling attention focus better, patching can also improve the efficiency by reducing the number of tokens. When facing a very long input length (e.g.. 384 and 768), we still adopt a small patch size in our design for better efficiency. Therefore this paper not call to completely abandon patching technique in all cases, but an extension to scenarios where patching technique is not suitable.
J.2 Compared with Typical Transformer-baser Models
To highlight how innovate and upgrade the Transformer-based model, we compare our De- formableTST with some milestone Transformer-base models in time series community.
PatchTST is milestone Transformer-based model and De- formableTST can be seen as an improvement of it to solve the problem of over-reliance on patching. PatchTST highly relies on patching for patching can force the attention focus on only few impor- tant time points within each patch and ignore other time points within the patch. But the choose of patch size is dilemma in practice: too large patch size could lead to many other time points within the patch being ignored, resulting in the risk of neglecting other priorities. And too small patch size will lead to a huge amount tokens, making it hard to focus. This dilemma cannot be resolved by PatchTST itself. To address this issue, we propose deformable attention, an attention mechanism that can focus well by itself, to get rid of the need of patching. Since deformable attention enjoys great focusing ability, we can use small patch size mitigate the problem of some priorities being ignored and free to worry about attention being hard to focus on important time points, successfully reducing the reliance on patching and resolving the dilemma.
Compared with PatchTST 35
DeformableTST and iTrrnsformer are the lastest Transformer- based forecasters. Both methods focus on the underperformance issue of previous attention mecha- nism in temporal modeling and devote to designing a more powerful Transformer-based forecaster. But they hold different opinions. Although iTrrnsformer still adopts a Transformer architecture, insists that attention is not suitable for temporal modeling while linear layers are more appropriate. This design makes its performance more similar to linear-based forecasters rather than Transformer- based ones, being less competitive in difficult tasks (c.g., univariate short-term forecasting tasks with lower similarity between samples). Different from I'Transformer, our DeformableTST follows the tradition of Transformer-ben based models to still uses attention for temporal modeling. Specifically, in this paper, through experiments and analysis, we further attribute the underperformance issue of previous attention mechanism to their over-reliance on patching and thus propose deformable attention to solve this problem, successfully designing more powerful Transformer-based forecaster and broadening the applicability of Transformer-based models.
Compared with i'Transformer [22]
K Limitations and Future Work
In this paper, we mainly focus on how to better use attention in temporal modeling and do not consider the multivariate correlating. will be our future work to study how to further capture the multivariate correlation in our model, which can hopefully improve the performance, especially in datasets with
24

---

large number of variates. Besides, we will further explore the potential of our DeformableIST in more time series analysis tasks and further develop its performance by large-scale pre-traiming the future.
L
Ethics Statement and Broader Impact
Our work only focuses on the time series forecasting problem, so there no potential ethical risk.
Our model achieves the sate-oo-thheann performance on a wider range of time series forecasting tasks, covering a large amount of real-world scenarios. Therefore, the proposed model makes promising to tackle real-world forecasting problem, helping our society make better decisions and prevent risks in advance. And we hope our findings can prompt people to rethink the relationship between Transformer-based models and patching technique, thereby designing more powerful Transformer- based forecasters with wider range of applicability.
Our paper mainly focuses on scientific research and has no obvious negative social impact.
M Reproducibility Statement
In the main text, we have strictly formalized the model architecture with equations. All the imple- mentation details are included in the Appendix, including dataser descriptions, metrics, model, and experiment configurations. Code is available this repository: https: //github com/luodhhh/ Deformale DeformableTST
Full Results N
Due to the space limitation of the main text, we place the full results of all experiments and results of more baselines in the following subsections. And also provide the showcases in Appendix O
N.1
Long-term Forecasting with Input Length 96
The full results of long-term forecasting with input length are provided in Table 8 And more results with more baselines are also included.
N.2
Long-term Forecasting with Input Length 384
The full results of long-term term forecasting with input length 384 are provided in Table 9 And more results with more baselines are also included.
N.3
Long-term Forecasting with Input Length 768
The full results of long-term forecasting with input length 768 are provided in 10 And more results with more baselines are also included.
N.4
Multivariate Short-term Forecasting
The full results of multivariate short-term forecasting tasks are provided in Table I1 And more results with more baselines are also included.
N.5
Univariate Short-term Forecasting
The full results of M4 datasets provided in Table 12 And the full results of other datasets are provided in Table
25

---

Table 8: Full results of the long-term forecasting tasks with input length 96. We compare extensive competitive models under different prediction lengths. The input sequence length is set to 96 for all baselines. Avg means the average results from all four prediction lengths.
Models (Ours)
Determsle Prthformer
CARD 58
CIMTATS 55
PaccTSST 35
i'Eamafomner Pithiomer
Autefurmer
incerr
TIDe
TimzaNai +
SCIN A0
53
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSF
MAF]
MSE
MAF
MSE
MAF
MSE
MAE
MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAF
MSE MAE
96
[0,53
0.196
1.396
angi 9K 0.405 904
0.404
0.414
0.419
0.356
0:405
0.419 o 459 566 0,308
0.479 (1464
to 194
n.400 546 0.400 pueS4 a 909
192
11427
0.423
0.442
4.43
4.448 0.453
0.46
0.42
0.460
0.445
0.441
0.466
(.420
0.448 0.500 o 482
143T 0.434
0.525
0.402
0.45
0.429
0.437 0.432 0.7 6611
336
[U437
9.43
0.478
4.4470 0.447..994 0.094 0.457
0.552
0.501
0.400
0,458
0.46d 0.521
1.679 0.446
0.566
0.515
0.a
u a51 0,459 107778
0.489
(.433
0.496
0.500
(0506
0.514
0.594
0,556
0.500
0.836
Avg
[1.425
0.428
1:485 411 c1.6 0.442
01443
[0:408
0:454
(0:454
0:447
[0 0:40000000000000 496 D 487 64466 0.454 0:541 0.500 jusss 0.450 456 0,450 00742 (1647
96.
[0.201
0.854
1:700
0.341 300
0.561
10.002 0,348 (0.297
0,344
(0:355 0.197 0. D 345 205 0.1S U 10-00 0, 0.370 (a 315 0.707 10207 6.621
192
0.353
0.382
0.377
0.401
0.375
0
0.390
0.409
0,388
0.400
0.350
0,400
(.429
0.439 0.456 0.452 0.374 0.200 0.528 0,509 0.402 0.41-
(.477 0.414-0.47
0,476 0.800 0.689
376
[1.41
0.379
5.175
0.309
2.119
0:426
0-421
0443
0:426
0.433
014488
0.432
(-406
0.487 0.482 D 486 (1415 0.426
0.643 0.57)
0.452
0.452 n 1.504 0.541
1.000 744
[1.000
0.431
1042
0:445
[1422
0.452
(04451
0.440
0,445
0.474 0,515
0.679
(1331 0,057 [:245
Avg
[0.546
0,083
00307
0.300/1377
0.367 0U415 [0:387 0:407 (0.353 0,407
0.450 0.259
0.550/o.314
0.227 (1 0,515 (i55i
96
10.36
0.268
0,320
0.361 0.327 0.35% 0.334 0 0.530/0.320
0.367 0.:344 0.364
(0379
0.419 0,505 a 0.55 0.376 0.364 0.587 (0338
0.375 345 0.372 10-118 0.436
199
0,109
0.334
0 3909
0.372 0.361
0.377 a 189 0.367 0.305 (0.377
0.19
(1:070
0.441 00555 o 200 1. 0.502
11300 cua0s
0.574
0:387
[1 1.350 o. 1W 10:43 a50
336
0.79
0.405
2.402
0.001
0.403 0.401
0.410
0.409
0.290
0.410
0.426
0.420
0.445
0.459 0.621
0.537 0.424 0.415 0.428 0.425
0.410
0.411
0.413 0.413 004900 0.485
750
(C443
0.455
2:467
0.439
0.467
0.438
0.471 0.442
0:454
0.490
0.441 0.459
(.545
0.490 0.671 0.561 (LAST 0.450
0-467 0.461
6.478 0.450
n 474 0.453 0.595 5.5
Aug [0,03
0,305
(a
0.396
0.402
0.400
(0:448
0.452/0
0.517/04414
0.41o/o.000
408
p.485
E1Tm2
192
96
1.178
1.245
0,202
0201
0.245
0.260
0.306
0.26-
0.246
0.259 0.179 0.263
0.306 0.347
0.508
0..75
0.241
0.259
0.302
(0.150
0.250
0.309
0.264
102039
(.266
0.2877 0.255 0. 0.339/0115 DLE 0.205 0,207 0,305 (.18T 0.267 (1 195 0,202 10.866 0,577
0.328 0.261 D 340 2.246 0.304 0,200 0.364 0.240
0,300
(2264 0.362 0.390 0.445
336
10300
0,345
0:343
(1,022
0.365
0.3113 0,550
0.705
0:343
00.311
0,548
11325
0.000/ 0.399 0 (1 NOT 0.342 0.377 04422 0,321
0:351 u 350 o.427 106 0.901
730
00,00
0.395
3:406
0.401
[.400 C:404 0.409 0.403
0.402
0.400
0.412
0.407
0.415 0.433 0.432 0.407 0,398 0.558 0.524 (0.42
0.405 0.403 0.554 0,522 (0.900 0.725
10.283
0.327
J.284
0.325
(0.211
0.257
0.33
00.281
0.326
0.352
10,305
0.327 (0,286 0.327 0.404
0.333 353 0.401 00577
Wealher
892
96
( 169
1.216
0:213
0.255
0.218
0.200
0.250
1.160
0.207
0.208
0.250
0.236
0:291
0.267
(0.17)
0.225
0::18 (0:
0.259
00.221
0.254
0:314
0.276
20:112
0:006/0: D Stolo 0.250 (1261
0.336 0,307
0.387
2.24 0.2F1 0.242
0.293
1o.12
0.219
0.261
2000 (1 196 0.555 9:221 109
25757 0.296 002001 0.340
356
(0271
0.204
0.2re
0.292
8.265
0.292 0.257
0.503
0.276
0:207
0.228
0.206
0.339
0.390 0,350 n 995 2.202 0.307 0.287
0.335
0,200
0.30m
0.265 0.335 003000 0.378
720
(1:447
0.364
K0340
0.542
0,340 0.363
0.35
0.354
0,348
(0.35M
0,347
(1:003
0.422 0,419 D 478 105 0,353
03351
0.500
0.305 0.350
(1.45
0.181
k0.77 0277
Avg |4:251
0.276
272
|1.255
0,374 0.20
(0,259
0,280 (0.255
0,274
10.300
0.100/0..533 0.302 0.291
0.320)0.259
200 0.317 10222
Solar Enengy
96
10.133
0,233
10209
0.300
0.254
0.205
0.234
0.286
00.000
0.232
10.242
0.342 0.884
0.711
(0.322 0.359 0.312 0.300
200 0:378 0:237
12225
0,261
0.514
0.267 0
0:087
0:11
0.761
11765 190
156
0.110 01416 0.195
107500
326
12299
0.207
0.320
0.334
0.259 0.319 0.327 0.330
0.290
0.315
00248
0.273
0.250
0.296
0.941
0.723 0.597
0,309 0.368 0.430
0.319
0.330
(a 0,415
003044 0,389
720
1.2088
0.265
0.354
0.337
(.294 0.327 0.333
0.347
0.289
0.317
(0:349
03.355
(1.37
0.427 0.882
0.71 ( 0.599 0,356 0.370 0.425
0.338 0.337
(1 0.413 10.300 0.315
Avg
10.224
0.256
10305
321
p.270 0.309 0.206
0.523
0.270
0.307
0.333
0.762
10.201
0.781
0.585
21
Ja 560 0.356 0.347
00.17
0.319
(1 0.401 0.282 0 0.375
1115%
0,255
0,25
11155
10.131
0.270
19199
0.201
0.281 0.237 0.329
0.272
0.282
01:47
0.168
0.265
0.173
0.170
0.350
0.188
0:274
0,253
0.201
0.322
0.201 0.283 0.236
0.330
0.259
0:257 0.385
0.280
0.199
0.283
194
0.204
0.203
(0.178
0.209
0.214
0.231
0.215 0,298
0.249 0.344
0.300
202
0.269
0,313
919
229
0.116
N16
0:346
0.104
(0:335
0.317
(1:46
0.145
0:154 a 361
(1257 0.931 0.764
0.993
6.220
0.330
2455
0.199
10:709
Ave
18.183
0.278
J0189
0.202
0.277
0.210
0.287
00205 0,290
0.178
0,270
|0.214
0.227
0.227 0.338 212 0,294
0.251 0.344 (0
0.295 (1222 0,300 (0266 0.365
0.271
0:400
5-0
0.467
0:005
0.266
5K7
0.401
650
10-700
0.40
0.465
0.535
3.0
0.466
0.206
01004
0,616
0.474
598
0.790
0.2M)
03233
0.477
0.547
0.487
0.283
01021
0,622
0.009
0.304
0.762
0.336
00707
0.303
0.599
0.354|..50
0.354
0.500
0.323 0.554
0.359
0.514
0.322
0.467
0,302
0.626
0.382 0.660
0.406 0.647 0,367 0.719 0.449 0.640
0.350
(1.52 0.394
0.841 0.523
Avg
[1.445
0.785
1:517
0 0.312
0. 308 0.542
10.481
0.004
(0.428
0.282
[I (1 0.66 a 0.579 6766 0.5 1 0.700 (1473 J0600 0:316 6325 0.181
10.04 0.909
Couun]]
2
7
0
26

---

Table 9: Full results of the long-term forecasting tasks with input length 384. We compare extensive competitive models under different prediction lengths. The input sequence length is set to 384 for all baselines. Arg means the average results from all four prediction lengths.
Models DefeembherT Pathfienee
CARD
GPT-TS
PosATST iTrandormer FEDiormer Autudoener
RLinear
TIDE
TimeaNer
DLinear
SCINes
50
Metric
MSE
MAE
MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
MSE MAE MAE MSE MAE MSE
MAE|MSE MAE]
MAE
MAE
MAE
95
(0.369
0.396
0.386 0,413 0.377 0,402 0.415 (.41)/ 0,401 0.413 0.427
0.300
0.43010.466 0u445 (.572 0.397 0.451 0.45 a 0.425
0.373 0.39 0,435
192
[1440
0.417
14415-04235 8.404 B:118 (1:47 447 04410 0.409 0.450
0:454
n.47 0.415 0444 Ku4l 0:421 D-482 n-53 469 0.467
b.411 0-421 0.471 0.469
336
(0.991
0.414
0.337
0,435 0.425 0,435 (1457 0.449
(1533 0.439 0.455
0.455
10.409
1.4T0 0.517
0.0990..22 0.525 0.429 0:501
0.456
0433
0.415
(1515 0.897
730
0.447
0.664
[.463 0.472 8.437
0.459 0.471 0.462 (.451 0.471 0.525
0.521
10.600
0.559 0.557 0.536 0.450 0.463 0.510 0.500
0.559
0.500
0.645
0,542
3.479
0.501
0.529 0,515
Avg
10.404
[1.255
0.000oa 0.683/o.415 0:425
0.05 [1508
00424 012////
96
[0.272
0.334
[0.253 0,550 0,225 0.336 0.319 0,571 [0225 0.357 0,306 0.565
10.425 0.460 305 0u444 2.275 0.358 000300 0.43 10.347 0.505 10305 0.368 0.227 0,383
190
(1.128
0.869
( 541 19999900 OJ 3.O 3.37 VO9
9 21.5
0.575 16966 0.402
w. 0.4 in You 1
191 p-44l 0--50
aos
0-428 0 1,00
9.37% 0.412
336
(1.319
0
(. 962
0.421 O. 360 1.99 (.:14
0.439
(1552003991 0.191 0-400
0.427
0.300
0.450 in 997 04441 364
0.40 i- 4000
441
0.457 0.471
0-476 0.451
730
11:305
0.435
11-16 0,452 9,590 0.425 (1450 (.470003995 0,435
0,435
0.454
10.513 1.520 1.00 0.484 4141 0.447 00.04 0.500 471
0.433 0.764
0.621
(1.499 0.603
Avg
[0.333
[03358 0.403 0.341
0.425
541
0.377
0.412
0.454j0.411
0.397
J0.85 0.473j0
96
10.292
0,548
11,00
0,553 0L2M5 0.346 2.N 0,282
(0.29 0.364 0.314 0.565
10.399
1.000 525 0.204 515
0.355 10311
073
0000 o 3355513310 0.563
192
0.335
0.371
0.337
0.578 0.3.P 0.3F1 0.345 0.579
0.540 0.374 0.353
0.589
0.307
0.440 0.5 0.534 0.337 0.366 0.344
3.37
0.585
0.13 o 2.36 0.360 0 0.300
(1.365
01 1.992
(1.9T
19666600 O. V73 D. 9.9 TOU W1
11.67 D. 1944 00163
0.405
10 in 90s 0.59
180 in 178
100
Krte 11.30 n 107 a.9
730
(0.417
0.418
[1.337 0,433 0.420 0.414 0-428 0--26 (1422 0.427 0.437 0.435
10:470 0.472 0.618 0,553 0-426 0.415 0-433 0.422
0.514 0.403
0442 0:419 0.453 0.439
Avg
[0.352
[0.363 0.000 357
0USS 0.385 0.372
Y0.4R0
0.40000.5
0.535
0.161
365 401
6.357
10.170
0.257
(U112 0.263 OLIGS 0,256 0.179 0.208/QU
0.255 0.179
0.271
10.277
0.3 01361 16166
0.253 10.89 2220 2021 0.200/0 10 2.200 0.275
192
1..227
0.295
(.255 0.312 00220 0,206 (1243 1.30
(1225
0.299
0.243
0.383
0:207
0.357 1.306
0.565
2.2
0.297 0.250 in 383
0.15 0:517 0.3
0:238
0.254 0.319/..25
0 0.321
396
11:278
0.527
(1.265
0,335 0.274 1132N (1.11 0.392 (12200
0.351 0.2M5
0.545
10. 11 11300003347 0.357 O.NET 27275
0.128 in ses 157
a 5112 o 0000 oo
i m. 0.2000 0.551
730
(0.365
0.384
[0.384 0,399 0.367 01383 (.389) 0.396 (.375 0,389
0.377
0.397
(0.419
0.425 0.4250.424 0.429 0.356 0.412 0.425 o 481 0.445 3.404 0:423 0.392 0
Avg (0.250
10.273
0.259 0.316 0.252
|0.262
0.273
0.532
10.336
0,379/0.34 0.586/6.255 0.316
0.355jo.
30315 0,553 6.279 0.343j0 20222.222222 0.339
Weather
192
96
(0.149
0.192
6.198
0.238
[1. 0:305 185 2000 164
112118
0,249
0.207 0.255 0.20 0.253
0.21//1
11194
0.244
0.213
199 0.200
0.251
10:365 Wo to
0.295
1.347
0.291 0.34770.29
0,335 0.216 0.261 02219 223
10 0:227 yn 177 o 220 16 0:2166000130 1.173 o 20 161 0:18
0.24 0.275
0.215
0227
0.272
0.301
336
0.244
0.278
(1265
0.283 0.263 0.294 (.256 0,259
[1343 0.270
0.252
0.287
0.336
1.377 0.322 0.554 0.363 0.254
0.266 i 200
o 250 0.566
0.264
0.259 0.317
0.317
o 0:206
730
10.317
0,331
(0.354
0,326 0,355 11352 (ULEE 0.500/1131 0,355
0,325
0.339
1o.372 U1587 13372 o 880 323 0,340 0.3 1es
a (1375 o 108 00524 o 1:300 t1res 0:315
10.335
[0.242
0,265 0.245 0,277 (.239 0.274
22226
0.264 0.235
0.272
0.354/0.313 0.549/o.246 0.251
2:24 254 0,203 0.244 o 0.200j 0.272
Solar Energy
10.378
0.241
11.215
0.296 175
0.24310 (1296/11
197
0.291 0.187
0.246
(1 9166 c100n 2215 0.207 10 co 2212 20 11 1010 a 0:110
892
01.17
0.348
0.220
0,501 0.209 0.262 0.234
0.515
(1214
0.319 0.209
0.269
0.404
0.869 0.474
n.474
707 0.251 0.312 0.229 0.55000 0:339
0223033022244 0264 o 0-23 11229 0:229
336
11.193
0.260
(.2332
0.311
0U200 01272 (12266
(.255
0.328
0,222
0.281
0-431
0.400 8.844 0.604 i. 6.365 0.358 1a i 155 0000 o 9ea
02500 a 360 260 1255
o tits
730
10.204
0.272
(.250
0,329 0.227 01261 01251 0.334 (0.225 0,329
0.231
0.291
00:503
1.559/0.753 o 7000 304 0.345 10.244 0.000 1251 0.336 0200 0:200 0:2000120 12207 0:321
Avg 10.189
0.255
11.22M
0,307 0,205 0,208 (02257 0.516/0.218
0.212
0.0 0L095jo. 0.323
0.300 (0255 0.337/u.230 0.300
96
10.132
131 0.232
0.233/0 037 0.292 0.252 0.140 0.240
10110
0 0,246 151 0:251 p.lel 0226 192 0 00000 o 000000..1
0.300
190
10120
01250
B. 0L2ST OL IST 0.250 (1 01255
[L14K 0.246
0.154 0.236
10:204
1 18500 0000 C N0N in 10 0.202 hos 1122 to 2000 o 200 0000 u 0.00 1779 0:985
336
0.167
0.267
BL175 0,270 OL 0.268 0.171 0.269
0.265
0.175
0.275
0:049
0.25010 222 0,335 1.179 0.277 0.18L 0.254 0.222 0,550 ou o 0.260 0.260111188 0.294
720
(0.208
0.209
(0.000 6.305 0.2000 0L292 (0.205
0.207
0.295
0.215
0.500
1.856 o 24345 o 545 2121 0.308 p:210 i 306 2577 o 100 0221 o 0:31//
o 0006
Avg [0.163
[0L77 0,265 0,165 0.262 0.265 (0.163 0,260 0,172 0.271
0,347/0.205 0.309)0.17 0.275
0.253 222 0.320 0.277 0,245
[0.362
[0.410 0.286 390 0.270/0 300 0.251
0.303 0.206
10:714 n to
0.305 0-400 a 964 i Sto O 505 10425 o 10l0 0.135
Trafftc
192
336
11.855
0.397
0.275
0.268
MLAE5 0,2M) OL (1277) (UATT 0,259 (1.01 0.283
(1.39
0.315 CLARD 0,250 0,420 0.205 0L410 0.452
0,414 0.512
0.522
10025
0,500 0.425 726 CLEAD 0.433 0.313 1.5512 0.353 a 550 0.313
11.20 01471 oo 641 0.317 00 (1310
04577 o 0:300 11500 0.000
0.31 1.52 0.303
730
[1.454
0.298
J (1-66
0,393 Q1ATT 301 (1AASI O S12 (1445 D. T0H 0.472
547
p.755 1.ASS I 5000 CLSTS fo 48% 0.351 p.s and 0117 o 1ee 00400 o 1o0 0.566
01411
Avg
10.398
0.276
[1.422
106 0.405 2.26 (0 2.244 4111 O. 7264 0.424 520
P s00o SS0 10.45 315
Cuurr]
24
23
0
3
0
27

---

Table 10: Full results of the long-term forecasting tasks with input length 768. We compare extensive competitive models under different prediction lengths. The input sequence length is set to 768 for all baselines. Aug means the average results from all four prediction lengths.
Models
DefemmbleTT Pathfiener
GPT-TS
POsATST iTaandoreer FEDiormer Autufoener
RLinear
TIDE
TimeaNer
DLinear
SCINet
Metric
MSE
MAE
MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAEIMSE MAE MSE
MAE
MAEMSSE MSE MAE MAE|MSE MAE|MSE MAE
95
(0.367
0.404
(0,39 0.417 0.377 0,410 0.454 (.44) 0.410 0.401 0.425 (0.88 0.44 0.436 0.400 0.446 65700,50 0.433 0.572 0.401 0,447
192
(1.49
0.430
a.423 0.43 0,410 9:428 (1. a eas (1416 0483
0.452
0.400 n s.000 n.00 0u400 0.433 p-483 u o 505
2.11 nA4S 0.472 0.435
336
10,066
0.437
0.049
0,450 0.830 0,413 0.525 (1440 0,475
0.68]
10.47 1.553 0.553 0.516 0u514 o.urs 0.441 1.50x 0.01 0.56)
0.438
0.400 0.509
730
(0.458
0.451
0.497 0 9.403 0.487 0.623 0,532 (.46P 0:483 0.567
0,549
jo/33 0.57 0,582
0.562..460 0.469 0.457 0.518 0.5:00.650 0.556
3.486
0.50e 0.548 0.531
Avg
(0.430
[0.000
Q.422 0,442 0.009/
0.678
U.smjuaa ouess/o.c2 0:440
96
(0.250
0.334
10.284 0,349 0.279 0.34] 0.340 0.386 0.339 0,307
0.563
10.466
0.478 10.43 0.47 (0267 0,353 10.65 0.427 43333 0,446 0.325 0.36e/o..35 0.386
190
11.135
0.
a 546
111 O. 140 D. 384 (1:20) 2.079
515 D 178 0:441
0.642
0.450 (.459 a
V12 9.77% p-4 417
444 0.471
0.433 0.450 0.37% 0
336
(1.124
0.380
(. 339 0.424 O. 367 0-400 0.4%6 443
(157 0-401
0.483
0.456
0:419
n.455 0-430 04461
557 0.405
a 0.400
0.555 n 500 n (1.30 0.432
730
10,412
01 14446
[1,22 0,456 9,592 01.30) (0.77) 9.476
(1.99 0.450 0.434
0.402
(0.11
11525
( 500 00516 0.454 po.04 0.524
0,520 0.404 (0947 0.650 01609/0.333 0.400
Avg
[0.333
[.335 0.405 0.347 0.300 0:417
541
0.400
0.431
10.455
0.485jo.44
0.486/0.58 0.393
4500
jo.50
96
[0.29]
0.347
10.301 0,353 0.2M3 0.343 0.306 0,259/ 0.351 0.310 0.570
00.444
11.00 to 466 0.663 51312 0.355 00:17
3000 o 2013 1030 0:351
(1:1101333 11335 0.577
192
0.325
0.392
0.351 0,381 0.339 0.371 0.339 0.577 0,500 0.373 0,347
0.587
0.566
0.500
0.506
344 0.374 0.344
3.3
41 0:413
0:337
0.368 0.363 0 0.302
(1,399
0. 900
(1 962 0.403 O. 1o6 01390
170 a WS (..56 o. 195
0:412
(1471
11 cas
1:171
100
n.
o 100 o anw
730
00.418
01425
[1445 0,436 0.417 01420 0.452 9.42 (1420 0.452 0,460 0.456
(0:465
0.4T4 0.630 0,557 2.414 0.133 0-422
0.455
02250..5 0.474
9.413
0.53 0.443
Avg
[0.348
[0.30
0.900
0.378 0.406
0151900 0.162 1.364 Soo 47o 0-420 0.355 0.37 330
96
(0.109
0.258
[1171 0,202 OL 0,255 D.ISE (1274/01182 0.2F2 0.185
0.275
0.350 3331 0L364 to. 161 0.252 1999 o (22 (220/220 0244 00000.09.0000 1. 0 1223)1.1
0.278
192
(0.200
0.209
(.229 0.312 0.226 0L298 (.244 0.311 (1247 0.314
0.250
0.317
0.344
0.351
0.507
2222
0.200
0.245 o 333 0257 0.543 0222 o 0:300 0.300112255 o 0.002
396
(1:26)
0.315
(1:295
0,309 0.272 01.28 (1.336 0.541 (1.30
0.154
0.
0.355
11.115
153 04422
0.190 i t01 157
in 100 o toe ooe o 0036
0.002
730
(0.349
0.384
0.355 0.409 0,590 01339) 0.397 (0.51 0.398 0.401
0.416
(0.420
0.457
0.426 0.4-00
0.356 0.392 0.400 a50 0,443 ounn 0:425 0.356
0,410
Avx |0.257
10.273 0,333 0,263 0.320 0.258 (0.276 0,355 0.281 0.541
0.401
0.411
(0.354 0.317
0.357
0.507
1.270 0.335j
0.542
Weather
192
96
10.146
0.191
6.198
0.239
11 149
0.197 0,242 0.22T 0.266
0.700 180 20000 196 27201
0.210 0,257
11194
14%
0.241 0.203
0.2012
0.253
0.217
10:200 1 345 790 Saole 120
0.505
0.3000.313 0.313 01354
0.213 0.259 0221 i 200
0:224 1017) oo 20222 111 1250 200 2000 0 00220 1. 0:224
0259 o 2229
0211 0211
0.203 0220012211
0:300
336
(0.241
0.250
(.247
0.286 0.262 0,206 0.255 0.291
11245 0.262
0.250
0.206
0.345
0.365
2.257
0.200
0.155 i 205
o 1311 oo 0:53400556
0255 o 3000 3000 1226 o 0.004
730
10.310
0.311
(1.155 0,336 O. 013500 (1322
D.3TK (1.50
0.308 0.331
0.545
US500
0.357 A13E1 io s0
ono o and 0011 o 0os00 2000 0.345
[0.332
10.27 0,265 0.251 0.283 0.258 0.277
0.258
0.277
0,565(0.330 0,566/0.240 0.278
0.20 200 0.316
0.2ssjn 0.286
Solar Energy
10.105
0,218
19192
0.203 177 0,200 (UZES 21 191
0.273
0.242
10.223
1.80 108 01000 191 0.2004 in 199 222 2210
225 190 a 200 181 0:27
892
0.184
0.254
8.198
0,270 0.210 0.268 0.238 0.584 (1211
0.282
0.194
0.259
0.313
0,410
0.410
0U674
0,209 0.277 0.216
02290
0.28 0.297
0211 o 0.291 0221/1201 0:285
336
16.191
0.263
(.223 0.783 0.210 0.277 (1:243
0.293
013007
0.272
0.613
nusee
1.00 73731 (.234 0.285 pa2s
0230 o
0221 a 00000/2222 0211332222
0:112
730
10.199
0.262
(.220 0,270 0,233 (1299) (12735 0,543 0,543/02255 0,2M7
0,217
0.279
11599 00000 0 22 0.283 0:237
00000
222 0.313 02254 0:310
0:310012000 02254
0:36
Avg 10.185
0.254
10.20M
0,272 0.210 (0:245 01516/02215
0.1MA
0.203
0.510003006
2.10 0.250
0.300 (1 0.3000o.11 0.2900.:11 0.29
96
10.13
0.214
1.134 0.236 0.230 0.229 00.33 0.142
0.241
1411 00:219
0,246 to 0:242 0.:43 o 250 0220 00000 0 0.:0
0.360
190
16.46
0.244
1.156
0.256 OL IST 1.25 USE
(1250/1
115
0.264
180
01236
10:220
0,240 0.551 in 186 0.154 10 il 200 to 1200 o 113 1100 a 0.250 1.102
0:224
326
0.116
0.366
8.179
0.271 0.18] 0.283 0.176 0.272
16166 0.207
0.179
0.227
0:243
0.55000.422 01534 0.172 0.200
0.176 222 0228 0:333 01124 o 0.20 0:385
720
(6.197
0.296
(02000
0.507 0.207 9.900/1 (12206 8.20
0.205
0.209
0.221
0.502
(188 (o 1.0 o
2.213 0.301 pi21 i 100 1414 o 150 oons o 0-000 1000 20200 o 600
Avg
[0.36]
0.361
[0.17
0.265 0.170 0.260 101667 1.263
0,176
0.272
0,35| 0.316/0.170 0.267
0.272 0.324 0.270/0.1175 0.242
96
[0.358
0.261
(0,399
0.277 0 3e0 0.262 3.3 0.265
0.300 0.201
10:781 0.470 1
403
0.285 0-451 (S54
565 0.507 10 o 2677 0.000
Trafitc
192
336
(1.380
0.393
0.271
0.281
11396
(0.477
0.307
0,283 0L 01277 313 01271 (1366
CLARZ 0.263 0.396 0L279 01309 0.275
0.269
0.420
0,400 0.300
0.517
10O00 (1391
0.504 0.390 900 0L520
(1891 CL5U1
01113
0.296
0.280 0.45
0.470
0.355 0.573
0.504
0:425
0:411
0.291 11.00 0.505
0,20 0.483 0 0.365
730
[1.454
0.300
0.319 Q14TT (12000 (1.4% D 91% 025/0447 (1439 0.395
0-465
144
yoows 1.577 1863 O.ST
1 464
0.317
960
1001 O 515
0-465
Avg
[0.391
0.278
[1.42
0.207 O WT 0.200/0 VON 0.280)J 109 0.277
0:417
0.515 10750 14 45000811
02709
579
J0.425 300
Cuurr]
25
19
0
3
0
28

---

Table 11: Full results of the multivariate short-term forecasting tasks. We compare extensive competitive models under different prediction lengths. The input length is times of the prediction length. Arg means the average results from all three prediction lengths.
Models DefermalerNT Pathfoemer
CARD S1
GPIETS
PechEST
slrandorner Ptaifoemer Autnfoemer
RI Liinear
TIDE
Timeshet
DLineer
SClNer 2
(Ours)
G
Fbi
18
P
[
Metric
MSR
MAF
MAR
MAP MSE MAR |MSE MAR/MSR MSR
[0.007
0,10
0.672
0.094]
00.306
00.56 0,399 0,577 0.393
0,352
(L454
0.371
0.393
0.361
0.4100.444
0,410
0.436
0.505 0.523 0.459 0.349 0.382 1.486 0.418
(0.366
[0333 0.393 0. 365 0.36 0.372 0.900/04400 0,490
0:346
0.572
0:401
0.425
0.499
0444 214660.5000 0.400 p.463 0453 (i 366 0.566 0.996-.463 44ej0.422
Avg [0.373
0
[0,456 0,424 0,490 0.433 (.A75 0.227 0,467
0:435
0.412
10.468
(0.44
9.459/o.743 0.543 10.09 0.525ju 440 0,419 0.419/ocus 0.510ju.477 0.450
10.132
BL145 0.246 0, 190 0.252 1. (1254)00 Du 0.259
0.151
0,253
10.149
0.252
0.257 to 203 0:302 00180 0.256 in 0.24a jo
0.3 301
0.759 0U167 01266 U. 0.206/0112
0,159
10:173
0.275
(1172
9.277
0.205 10
0.200 0:20 0:321 0.19
10.155
[L17] 0.265
0.172 0.268 0.173 (L18P
0.181
0.275
(02206
0.250
0.210 0,501
(.228 0.311 0.202 0.203 ( 0.200 0.207 0.312 0.22 0.37)
Avg [0.142
0.239
10.15 0.257 OL
0.262 0.164 0.263
2203//11722 1L172
0.269
0.164
0.262
(0.176
0.270/0.154
0.275/0.213 0.303
10.99 1 0,250 (2 171 0.266/0.202 0.319/0.211
0.332
1.338
0
22421
0.995 2,999
1.10
3W0 9965 (N S49
1.7
1.993
01851
13.188
1 1172
1922 1. 128/1 744 1:253
(4.51 1490
440 000
1,309
1:250/s 186 1.045
2.397
1.221 4,542 1.591 4.141
2.761
1.064
4.563
1.50
1.741
1.739
1.856
2380
5.450
1.732
4,659
1.509
1.567
1.305
1.18
1.317
1.700/6
5:262
4.906
1600
5.350
1.674
Avg [1.767
0.798
[3.130 1,160 3,572
1.371 (337133300 33.305 1.227/4.217
1,435
2.510
1.009
(4.407
1.500||.500
5,500
1.525/..20 2.202
1.611 (5,44
1.646
0.930
1.502/
0005 1.559
Exchange
[0068
0.051
10.099
0.061 0.010 0.062 (UNEE (2.02
0.064
0.005
2.053
10.015
0.005
0.001
10.013 0.076
p:012
(.77)
10000 n
0.660.043
0.55
[U015
(1016
0,081
0.016 (1080 (1015 11016
0.087
0:014
0000
0.114
1012
0.10
0000
0.130
0.5
0.089
0.002
0.021
0.023
0.000
0.092
10.00
0.146
0,139
0.126
0.023
0.04)
0.059
Aog [0.013
0.070
[0.055
0.077
0.015 0.083
0.083 (0.055 0.077 0.085
0,014
0.073
10.025
0.119/0.02
0.117/0.022
00.022 0.100 0.019
0.005 1016 0.06
1004 0.15
13.00.05 0.173
PEMS03
10.072
0.150
0.179
0.208
0.090
0.122
0.193 0,074 0.151 (0.066 0.1933200885 0.195 0,063
0,223 0.101 0.210 0.123 0.231 [1131 0.2-7 0.009
0.171
0.195
10O075
00.100
0.250 0.230 0.230 0.147 0.255 0.167 0.276 1.1 0.224
o. 0.159/0.079 0.191 10,05 0.205 0o.0s
0.224/0.070 o 0.183 0,000
0.131
0.255
0.218 to 0.179
1.105 0.225
01135
0.240
(.133 0,266 0. 138 0.251 (o. 0.279 188
0.294 0.127
0.236
10.147
0.201 173 a 3.507 6.219 0.310 9:241 0:35 i 155 0:26
0:200
o. 0:347
10.102
[0.128 0.227 0.104 0.214/0.129 0.234 [0.135 0,366 0.003 0.301
J0.112 0.230f0.133 0.25%/0.154 0.255 20.172 ozzoju 116 0.223 jou3s 0.200 200//.104 0.217
PEMS04
0.115
0.212
0.192
10.107 0.208 0.089 04104 (0,008 (L007 0.210
11.144
0,243 0.115 0.227 (1.336 0.31 (1 0.261 0,111
0,079
0.183
0.219
10.154
U.257 0. 100 0.291 to 1788 0.289 10 0.300 123 0,250 0.009 0:250 n.rw
100 0.216/0 123 0.245 p. 2555 o 0:006 100 1u o 1184
0.21
0.150
0.254
0.192
0,283
0.163 0.269
0.193 0.293 0.207 0.318 0.153
0,259
(OL178
0.300
0.2 0,576 253 0,344 0.290 0.355 ao 0. 0.195 0.322
0.146
0.347
Avg 10.116
[0.
123 1.210 147 0.249j 140 0:114
0,270
1999 t 171 0,206
0.791 10:31 1//// 140 0-253 to o 700 0.217
PEMSOR PEMSO7
(0.053
0.162
1.055
6.170
0,186 0,062 0.165 0.007 (L184 0.185
0.203
CLORE 0L195 (1 0.225 (1122 0.241
0,059
0.082
0.159
0.18K
10002 i 10.06 D 0.190 0.205 00102 o 22222 1077 o 0.000 0009 o 0.213 o 100
02100111 18 c124 in 140 0:250 o i 20000 1 8o 00000001222 o 0-00 0-000 o 0:300
00.21
0.225
0.174 0,207
0L127 0.233 (1.174 0.276/O188 0.303
0.122
9,230
0261
(2020 o 0.277/0.12 0.309 (221 0:333 it tee 0,2rd 0000 0.244 (1111
0:443
10.103 0.220/0.115 0.239)0.447 0,255 10.000 0.274/u (2 0.229/0.12 0. 0.253
10.090
10.16
0,0M2 (2.
0,085
0.192
0.004
[0078
0.180
10.00
0:207 0LOR3 0.001 0,203
0.080
0.182
10:007 n 2000 097 0,209/0 lo 0:225 0.125
to 11 o 0.342/..1000 10000 0.000
IL104
0.208
0,25K 0.113 0.217 (UE (12HD)L 142
0.259
0.110
01217
11247 100 (1249 in 173 0.2M0 11107 00000 (1 1155 o
1.10 0.265 1.120 0.218
0.136
0.237
[L199 0.277 0,147 0.253 (..19 0.288 (1.26 0.314
0,345
0,250
(0.199 0.290 230 0L549 245 0.357 0.267 0.340 ( 185 0.266
0.193 0.314 0.314/0.143
(1143 0.349
[0.106
0.208
0,224
0.114 0:219/0 137 11244 14T 0,700 1.1 0.218 101556 1 0.24500 0.282/) 175 0.787 p. 9:300 a 200 137
158 703oo 0.29
1" Cuaur] 16
16
0
0
0
0
7
7
0
5
0
0
Table 12: Full results for the univariate short-term forecasting tasks in M4 dataset. We report SMAPE, MASE, OWA for M4 datasets as metrics. Lower metrics indicate better performance. Wighted Average means the results are wighted averaged from several M4 subdatasets under different sample intervals. x. in the Transformers indicates the name of *former. The original paper of N-BEATS (37 adopts a special ensemble method to promote the performance. For fair comparisons, we remove the ensemble and only compare the pure forecasting models.
Models
DeformableIST (Ours)
Path.
CARD GPT4TS Cross.
PascTTT llranstormer
FED.
Auto.
RLinei
TDE
TimesNet
DLineu SCINer N-HITS N. BEATS
[1
5X
53
35
23
57
[+]
[46]
52
4
57
Yearly
SMAPE
13.194
13:473
13.302
13.538
13.392
13:445
13.461
13.728
13,974
16.151
17,019
13.380
16.965
13.764
13,418
13:436
MASE
2.955
3,016
3041
3.001
3.021
3,045
3,048
3,134
3,680
3.945
2.996
4.283
3,003
3,045
3,043
OWA
6,775
0.790
0.786
0.797
0,787
0.791
0.795
0,803
0.822
0.957
1.017
0.786
1.058
0.111
0,793
0.794
Quarterly
SMAPE
9.971
10:33
10.63
10:325
16,317
10.187
10:071
10,792
11,338
11:741
12.164
10. 100
12.145
10.946
10,302
10.124
MASE
L.63
1,203
1.126
12118
2.197
1.196
1.182
1,283
1.365
1.456
1,510
1.182
1.500
1 1,293
1.194
166
OWA
0.877
1.542
0.896
0.858
0.95%
1.012
1.064
1.103
1.106
0.809
0.586
Mootblly
SMAPE
12.592
123895
12.670
12
12,800
12,924
12.856
12.737
14.2611
13,958
13.599
13.616
12.670
13.514
13.543
12.791
12,677
MASE
0.931
0.955
0.933
0.951
0.966
0.956
0.935
1.102
1.103
1,056
1.056
0.933
1.037
1.024
0.969
0.937
OWA
0.844
0.87%
0,893
0.902
0.895
0.851
1.012
1.0022
0.968
0.968
0.878
0.556
0.899
11.880
Others
SMAPE
4.324
5.136
5,330
4.861
5.511
4.877
5.033
4.954
5.485
6.747
6.825
4
6.709
5,061
4,925
MASE
2,993
3/427
3.261
3:320
3,738
3,280
3.284
3.264
3,865
4,652
4.809
3,302
4.983
4 4.997
3.216
3.391
OWN
0.927
1.011
1.075
1.005
1.168
1.030
L.047
1.006
1.187
1.443
1.477
1.035
1.487
1.644
1,040
1.053
Weighedd
Avenge
SMAPE
11.658
12:001
11,815
12.000
13,475
11.952
11.878
12.840
12
909
13.298
13.711
11.829
13.639
12.699
11.927
11.851
MASE
1.555
16614
1.604
1.997
1.701
1.771
1.935
2.017
1.585
2.095
1.765
1.613
1.599
OWA
0,838
0.563
0.850
0.865
0.965
0,860
0.855
0.918
0.939
1,000
1.033
0.851
1.051
0.930
0,861
1.855
29

---

Table 13: Full results for the univariate short-term forecasting tasks in other datasets. We report the SMAPE in this Table as metric and a lower metric indicates better performance. *. in the Transformers indicates the name of *former. The original paper of N-BEATS (37) adopts special ensemble method to promote the performance. For fair comparisons, we remove the ensemble and only compare the pure forecasting models.
Models
DefemmbleTST
Put.
CARD GPTETS Cross PutchIS l'ransfommer
FED.
Auto.
RLinear
TIDE
TimesNet DLiiner SCENer N HITS BEATS
(Ours)
SI
10
52
[0
Yearly
15.902
20,005
21,709
21.208
26.310
22,408
29 190
21.718
27
21.491
25.611
16:033
27.472
21.145
27,404
21
Quunedyy
22.875
32.920
Monthly
18666
18,993
16.
11.322
18,049
17.241
19.401
17,826
18:008
300788
26
269299
850
337
26 666
21,960
21.300
Yeudy
18.623
23,959
24.359
M3
Quantelly
7,366
10.033
8,052
12:579
8.069
7.991
8.642
11.751 11 11.503
27.079
18.749
11.649
16.933
11.576
18.007
12.189
Monthly
21.103
16.563
18.050
21.350
25.341
23,505
Odia
9,378
17,353
15.500
19.099
13,100
15.707
18:439
20,969
22.525
24.769
18:370
18.091
15,600
26.064
17.316
22.050
Tourism
Quanerty
17.968
27,921
19.102
31.210
19.517
19.314
20.116
38,005
19 867 39.079 48 66.000
26.052
34.6.0
35, 25.2001
33.909
43.88
Manthly
NNS
Weeldy
14.372
18.18
22.491
16.012
17.717
20.449
19,00 22, 22.868
22:355
23.751
26.456
16,645
Hospital
Monthly
19,088
21.551
21.392
22.567
20,007
22.123
20.785
28.646
25.749
26.184 29.103
21.152
23.015
28,807
25.309
23.594
KDD Cup
Hoory
50,094
56.74n 61
$4.713
S8-472
90.44 649
60.15
90.013
617
60.922 63 43-447
56.18
523
63
958
$4.855
62. 62.905
30

---

Showcases
To provide intuitive comparison among different models, provide showcases the long- lerm forecasting tasks under two representative cases (the time series is in declining stage and the time series is in rising stage). The results are in Figure L DeformableTST predicts the most precise future series variations and exhibits superior performance.
Among the various models, our
Prefeiler
20
Pedictio
20-
15
1.5
1.
DSf (ours)
CARD
PStchTST
2.0
GPT4TS
iTransformer
RLinear
Figure 12: Visualization of 9at----e--redt-966 results on the Solar dataset. The time series is in declining stage. The blue lines stand for the ground truth and the orange lines stand for predicted values.
Orear
7.-
1.5
16
1.0
-25
DeformableTST (ours)
CARD
Pat
2.5
Pediics
22-
1.5
1.0
GPT4TS
i Transformer
RLincar
Figure 13: Visualization of 1nput-96-predictt results on the Solar dataset. The time series is in rising stage. The blue lines stand for the ground truth and the orange lines stand for predicted values.
31

---

P Comparison with More Baselines and Experiments on More Datasets
P.1
Compared with Sageformer
We compare our model with Sageformer [54]. the latest Graph- Transformer model. Since Sageformer will produce NaN outputs in some short-term forecasting tasks, we mainly conduct comparisons in long-term forecasting. The results are shown in Table [4] Our DeformableTST achieves consistently better performance than the latest Graph- Transformer method, further demonstrating our performance superiority.
Table 14: Comparison with Sageformer in long-term forecasting tasks. A lower MSE or MAE indicates better performance. Results are averaged from three input lengths I {96, 384, 768) and four prediction lengths T E {96, 192, 336, 720). The best results are in bold. Full results of provided in Table 15
Sageformer 54
ETThl
ETTh2
ETTml
ETTm2
Weuther
Solar
ECL
Traftic
Dataset
Metric
MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE) MSE MAE MSE MAE MSE MAE
DeformableTST (Ours)
0.413 0,430 0.336 0,381 00.55 0,386 0.267 0.321
(0.233 0.266 0.199 0.255
0.255/0.169 00.169 0.267 0.410 0.280
Sageforner
69
0.427 0.438 0.368 0.405 (0.77 0.394 0.275 0.327
00238 0.272 0.227 0.285
0.285/0.174 17 0.273
00.418 0.287
P.2 Experiments on Stock Market Dataset
We conduct multivariate short-term forecasting experiments on Stock Market Dataset [36]. As shown in Table T6 our DeformableTST still outperforms other competitors, validating that Deformable can work on stock market data.
Q Experiments on Synthetic Dataset
We conduct experiments synthetic dataset with some typical cases of attention distributions to prove our model can handle both uniform and clustered attention distribution. The details and results are provided in Figure [4]
As shown in Figure 14 our method can accurately predict the future data in all cases. And ERFs can operate as anticipated, successfully matching the distributions of key information. In details, in the case of globally uniform attention, the brighter points in ERF are also distributed globally, which means the model can find the important time points across the whole series. In other cases, the brighter points in ERF tend to concentrate in localized areas of key information, proving the effectiveness of our method in scenarios where key information is clustered within specific time window. These results validate that our method can adeptly manage both uniform and clustered attention distributions.
R
Model Robustness to Patching
We conduct ablation study to show the effect of patching on our method (under input-384 and input-768 settings). As shown in FigurelBaand Figure [6] model is robust to the choice of different patch sizes on input length 384 and input length 768.
Meanwhile, as shown in Figure 15 the performance of other Patch-based Transformer competitors (e.g., PatchTST (35] and CARD (5T) will decrease obviously and fell out of the good rankings if without patching. This is significant performance decrease, especially considering the intense competition in time series forecasting. By contrast, our method works well without patching, which further verifies robustness to the use of patching and shows that our model can successfully get rid of the over-reliance of patching.
32

---

Table 15: Full results of Sageformer [54 in long-term forecasting tasks. Avg means the average results from all four prediction lengths.
Models
Sageformer Sageformer Sageformer (Input-96) (Input-384) (lopur-768)
Metric
MAE
0.385
0.402
00.371 0.401
ETTh
192
0.431
0.428
0.418 0,429
0.413 0.429
336
0.458
0.445
0.428
0.436 0..433 0.444
720
00.476
0.469 00461 0.470
00443 0.485
Avg
[0.438 0.436|0.422
0,434
0.424 0.443
00.300
0.346 (0.284
0.343
ETTh2
192
0.3991
0.408 0.347 0.384
0.354 0.393
336
0.418
0.426
00.372
0.419 0.386 0.430
00.428 0.448
01448
0.443
(0.425 0.471
Avg
00.384
0.307
0.397
0.364 0.410
0.333
0.368 (0.294
0.345
ml
192
0.376
0.395 0.339
0.375 0..337 0.374
336
0.399
0.410 0.381
0,400 (0.369 0,391
0.4464 0.447
00.43
0,436/0.418
0,436 0.4488 0.432
Avg
[0.394
0.405|0.362
0.389 0.335 0.388
00.177
0.259 0..172 0.260
00.170
ETTm2
192
0.244
0.304 00.241
0.307 0:232 0.304
336
0.308
0.346
0..284
0,336 0.308 0,352
0.4412 0.406
00.379 0.398 00.374
Avg
[0.286 0.329
0.329|0.269
0.325
(0.271 0.326
0..163
0.207
00.153
0.207
Weather
192
0.222
0.258
0.199
0.242
0.197 0.248
336
0.272
0.296
0.247
0,285
0.253 0.293
0.347
0.347
0.326
0.34|
00,323 0.340
Avg
[0.251
0.277
0.277
p.2311
0.269 00.231 0.270
Solar-Energ Energy
00.231
0.286
0.199
0.254
0.179
192 0.265
0.305 00208 0.269 195 0.277
Solar-
336
0.288
0.313 0.213 0,286
0.212 0.287
0.292
0.327 0.226
0.286
0.223
Avg
[0,269 0.308
(0.009
0.274
00.202 0.272
00.156
0.251
[0.41
0.244
0.138 0.243
ECL
192
0.1
0.263
158
0.259
154 0.254
336
08888
0.285
0.4
0.275
0.117 0.273
0.226
0.317
0.215
0.309
0.200
Avg
18. 0.279
0.272
06.66 0.268
[0.418
0.271
10.385 0.275 00.371 0.268
Traffic
192
0.434
0.281
(0.997 0.279 0.385 0.273
336
0.446
0.289
0.414 0. 0.295
0.399 0.278
0.480
0.305
0.443 0.308 00.422
Avg
00.445 0.287 (0.410 0. 0.289 0.289)0.399 0.285
Table 16: Multivariate short-term forecasting results on Stock Market. We compare extensive competitive models under different prediction lengths. The input length is times of the prediction length. Arg means the average results from all three prediction lengths. The best results are in bold.
Models
DeformableTST Pathfomer (Ours)
CARD
GPT4TS
PatchTST Maansformer PEDrormer Autoformer inear
TDE
TimesNer
Metric
MSE
MAE
MSE
MAE
MSE MAE
MAE
MSE MAF
MAE
MAE
10.110
0.132
00.127
0.117
13188
0.121
0.142 127 0.122 0.139
0.132
178
15910 157 0.168 (.149 1616710.11
0.138
0.121
0.152
0.130
0.14
0.161
0.166
0.138 0.174 0.137
0.162
0.143
0.143
0.18 0.161 0.187 0.164 0.190.113 0.163
0.136
0.170
00.146
0.17
0.177
0.155 0,179 0.146,
0.186
0.145
0.175
0.157
0,217 0.217 0.152 0.208
0.171 0.208 0.167 0.200
0.173
Ave [0.122
08:1340.15900.3333
0,140
1337
0.171
0.159
195
jo.
184
0.188 16.660 1866 0.158
33

---

Prefecice
Prolecee
The begghte 1311 giitsilly focuudd
brighte tecasol
200
Clicc 186
ietecaliod roucific inclec 192
The wicharrpet pwoerrpeeonsno E1B7 braalty Saxeed the orn spuctic tinu arnucaaali anndiis
We conduct experiments with some typical cases of attention distributions. The synthetic data and experiment setups are as follows. Setup (1) for globally uniform attention as shown in Case (a): The input consists of 4 semi-sinusoidal signals with noise. The task is to predict semi-sinusoidal signal. Thus, the future data evenly relates to the historical data. The length of each signal is 96. So this is an input-384-predict
Figure 14:
task. Setup (2) for clustered attention as shown in Case (b) and (c): Simlilar setup (1), but in the length-384 input, only semi-sinusoidal signal is remained while others are masked as O. Thus, the future data is related only to the local window of remained signal. Masks can constant or varying across samples to simulate scenarios that the localized areas constant varying across samples. Setup (3) for global but not uniform attention as shown in Case (d): This setup is similar to setup (2) but more semi-sinusoidal signals remained, resulting in global attention distribution but not uniform.
Sobar
03055
Tedbed
prternm
3303
Ta
010
$3373
3000
33333
53333
018
3202
1150
Figure 15: The impact of patching on latest patch-based Transformer forecasters (PatchTST and CARD). After the removal of patching, the performance of PatchTST and CARD will decrease obviously and fell out of the good rankings, while our Deformable is robust to patching and maintains the consistent excellent performance, consistently ranked in the top-3. We conduct experiments under input-384-preditt settings. The rankings on each dataset are calculated from Table Appendix
34

---

Patch Size
0.28
0.26
0.24
Solar
Weather
ETTm1
MSE
0.22
0.20
0.18
0.16
0.14
8
16
Figure 16: Robustness of our DeformableTST to patch size under input-768 settings.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section and Abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Appendix K And model efficiency is provided in G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
36

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Ys]
Justification:
Please refer Section 3 Section Appendix A and Appendix
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
37

---

Answer: [Yes]
com/luodhhh/ DeformableIIST We provide the pseudo-code in Appendix C and add Reproducibil- ity Statement in Appendix M We have already provided experimental details and model settings in Section [] Section Appendix [ and Appendix B shape and model structure are also included.
Justification: Code is available at this repository:
And details about tensor
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides httgg//ddee//eeuubimmeeeeeeeeeee (ooeeSumisssioooooooo...
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.c/pppl httpssccccppplcccccc htwlliic/giiess/// http://ooodSubbaiiiooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have already provided experimental details and model settings in Section Section 4 Appendix Aland Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Please refer to Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
38

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Please refer to Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
39

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The code packages, baseline models and datasets used in this paper are publicly available and properly credited. Please refer to Section 4 Appendix Appendix B and the Reference. The baseline models are mostly using Apache-2.0 license and MIT license. The datasets we used all extensively utilized for benchmarking and publicly available. We provide the URLs for the datasets in Appendix
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
40

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Appendix A Appendix and Appendix Code with detailed documentation available at this repository: https://githun http///gtttt...t// com/ luodhhh/Dett Def.orebbeeeSSS
Justification: Please refer to Section 3 Section +
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
41

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "fTxNAuYu9Z", "reviewer": "Reviewer_eY9h", "summary": "This paper presents a Time Series Forecasting method DeformableTST that makes patching replaceable in transformer with sparse attention, using a hierarchical architecture to avoid memory issues. Experiments are performed on 8 data sets and the proposed method is compared with variety of SOTA methods.", "questions": "1. Why SOTA methods do not include Graph-Transformer methods e.g. STGNN?\n2. Will DeformableTST work on stock market data?", "limitations": "None Listed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Making patching replaceable with sparse attention\n2. The use of hierarchical architecture to overcome memory issues due to removal of patching\n3. The use of small patch for very long time series\n4. Comparisons with Variety of SOTA methods on 8 data sets\n5. Ablation Study", "weaknesses": "1. Related works does not include Graph-Transformer methods e.g. STGNN\n2. Graph-Transformer methods e.g. STGNN are not used as SOTA for comparisons.\n3. Data Sets should include financial data e.g. Stock Market\n4. Synthetic data should have been used to really test the proposed approach and claims.\n5. Table 1 is not properly discussed, specifically, why CARD  and RLinear are better on some data sets\n6. Table 2 should be discussed the same way for PEMS\n\nI appreciate the description of model efficiency in terms of computational cost, but I believe the contribution is incremental and needs more work and the use of synthetic data to improve the impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer eY9h for the thorough and detailed comments.\n\n> Concern about contributions.\n\nWe'd like to highlight our contributions in the following points:\n+ Our method **differs from** previous Transformer-based forecasters for its **better applicability and less reliance on patching**.\nDifferent from previous works, our DeformableTST can successfully reduce the reliance on patching and broaden the applicability of Transformer-based models, **which is a great improvement than previous Transformer-based models**.\n\n+ We propose a **novel methods for time series forecasting with better performance and wider applicability**. Our DeformableTST can flexibly adapt to multiple input lengths and achieve excellent performance in tasks unsuitable for patching. Therefore, our DeformableTST achieves consistent state-of-the-art performance in a broader range of time series tasks, showing great practical values in a wider range of real-world applications.\n\n+ In addition to proposing a state-of-the-art method for time series forecasting, **the exposure and exploration of the novel problem of over-reliance on patching is also a main contribution of our paper, which is not incremental.**\n    + The problem of over-reliance on patching is of great significance to the time series community. \n    And our proposed methods is the first to deeply explore and specially target to this less-explored research problem, \n    **which highlights our novelty and contribution**.\n    + And our findings can bring new perspective to time series community and prompt people to rethink the relationship between attention and patching, \n    which benefits the future work to design more powerful Transformer-based forecasters with wider applicability. \n\n> W1 & W2: Including Graph-Transformer methods in related work and comparison.\n+ Following your valuable suggestion, we add the latest Graph-Transformer Sageformer[1] as our baseline.\nThe results are shown in **Table 1 of global response**. Our DeformableTST achieves consistently better performance than the latest Graph-Transformer method, further demonstrating our performance superiority. \n+ Applying Sageformer on short term tasks leads to NaN output. We are working diligently to fix it. We will include the complete experimental results in the final version and include Sageformer in our related work.\n\n[1] SageFormer: Series-Aware Framework for Long-Term Multivariate Time-Series Forecasting.\n> Q1: Why SOTA do not include Graph-Transformer methods.\n+ Since the latest state-of-the-art models in time series forecasting are mainly Transformer-based and Linear-based models, \nmainly using these two types of models as strong baselines can enhance the persuasiveness of experiments. \nAnd this is a protocal widely adopted by many published papers from premier conferences. \n**To provide a fair comparision, we follow this mainstream protocol**.\n+ Thanks again for suggestions in improving our experiments. We newly add Sageformer as SOTA baseline to further enhance our persuasiveness.\n\n> W3: Data Sets should include financial data.\n\nThanks for your concern, in addition to the 8 datasets mentioned by you, we have already conducted experiments on 32 datasets to ensure adequacy of our experiments, which adequately cover a wide range of real-world scenarios and **have already included the financial data (e.g., M3, M4 and Exchange)**.\n> **Q2:** Will DeformableTST work on stock market data?\n+ Following your valuable suggestion, we further conduct experiments on Kaggle Stock Market dataset. As shown in **Table 2 of global response**, our DeformableTST still outperforms other competitors, validating that DeformableTST can work on stock market data.\n\n> W4 : Experiments on synthetic data.\n+ Following your valuable suggestion, we conduct experiments on synthetic data and show the results in **Figure 1 of PDF in global response**.\nPlease refer to the **Response to W1 & Q4 of Reviewer fiTw** for detailed analysis. \n+ The results shows that our method works well for each specific scenario, which further verify the proposed approach and our claims.\n\n> W5 & W6 : Discussion about Table 1 & 2\n+ We'd like to claim the adequacy of our discussion as follows:\n    + Following the guideline of paper checklist, our experimental discussions are mainly **served to analyse our method and validate our claims**. \nTherefore, the discussion in our paper mainly focus on our own model. And other baselines are comprehensively discussed by category (but not individually) to ensuring the persuasiveness.\n    + In Table 1, RLinear and CARD just achieve **similar performance** with us in **one of the datasets**.\nAnd **our model surpasses them by a larger margin in a wider range of tasks, gaining SOTA in most cases**.\nEspecially considering that these two models are the most powerful Transformer- and linear-based models,\nthe results in Table 1 convincingly verify our performance superiority.    \n    + In Table 2, we emphasize PEMS as an example to illustrate our limitation of not considering multivariate correlation.\nFor other datasets, we discuss them by category to demonstrate the model's ability to generalize across multiple scenarios.\nBy categorizing them from different input lengths and different task difficulty, \nthe discussion of Table 2 proves our claim that our method can reduce the reliance on patching and broaden the applicability\n    \n> Concern about limitations.\n+ Thanks for your concern, we have already addressed the limitations in **Section 4.2 (line 262-265)** and also listed it in **Appendix K**, which are:\n    + Our paper mainly focuses on how to better use attention in temporal modeling. It will be our future work to study how to further capture the multivariate correlation in our model.\n    + Our paper mainly focuses on time series forecasting tasks. We will further explore its potential in more time series analysis tasks and further develop its performance by large-scale pre-training in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rn5bDG1caV", "reviewer": "Reviewer_fiTw", "summary": "The paper introduces a new transformer architecture for time series forecasting that does not necessarily depend on patching, resulting in consistent performance improvements over all baselines. Although the approach primarily extends existing work, its simplicity and potential for widespread adoption in various time series forecasting applications is notable, provided the experimental results are robust and well-justified.", "questions": "1. Does the method in this paper use a similar restriction as in [1], where $∆p ←− s \\tanh (∆p)$ limits the range of attention offset, or does it allow offsets across the entire sequence? \n2. Is the important time point sampling detailed in Algorithm 1 (Sec.3.2, Eq. 6-8)? It seems crucial to the method's design but is not explicitly mentioned in the algorithm description.\n3. In I.1, the method considers only the two closest time points for output. Could this focus on local optima restrict the learning potential of the Deformable Attention module, especially if other time points might be more relevant?\n4. In lines 710-712, the authors note that PatchTST cannot effectively learn centralized attention in localized areas. Could the authors provide illustrations of Effective Receptive Fields (ERFs) similar to those shown in Figure 8, but demonstrating how the proposed method manages such scenarios? This addition would be highly beneficial as the existing illustrations primarily depict the method's ability to distribute attention across entire series. A comparison showcasing the method's capability to focus attention locally would address a critical aspect of time series forecasting where concentrated information is crucial.\n5. typo in line 714: focus\n\nI am prepared to increase my score if all my concerns are adequately addressed.", "limitations": "Yes, the authors have mentioned the limitations and potential negative societal impacts. They do an excellent job empirically demonstrating performance improvements. The method clearly shows an advantage over prior works across all benchmarks. To enhance the paper, a deeper analysis explaining why the proposed method improves performance would be beneficial and provide valuable insights into its effectiveness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Clean Methodology**: The methodology is elegantly simple and eliminates the need for manual hyperparameter tuning required by patch-based methods. This adaptability to various sequence scales with minimal complexity is a significant advantage. Additionally, as demonstrated in Section D, the method exhibits low sensitivity to parameter variations, which underscores its potential for broad applicability.\n2. **Clear Presentation of Proposed Method**: The authors effectively identify the core problem and critically review prior works, highlighting limitations stemming from their dependence on patching. The presentation of the methodological framework is clear and detailed.\n3. **Comprehensive Experiments**: The experimental setup is thorough, employing well-known datasets relevant to the task. The choice of baselines is comprehensive, including both patch-based and non-patch-based transformer models. The experiments cover a range of scenarios, including different dataset temporal scales and analyses of time complexity and parameter sensitivity.", "weaknesses": "1. **Uniform Attention Concerns**: The proposed deformableTST model appears to use a uniform attention prior, adjustable via learnable offsets. Despite this flexibility, the effectiveness in scenarios where key information is clustered within specific time window remains uncertain. The paper asserts that this method can adeptly manage both uniform and clustered attention distributions, yet it lacks systematic experimental validation of this claim. Incorporating a synthetic dataset designed to simulate distinct scenarios could substantiate these assertions more convincingly. Example test cases include:\n    1. Future data evenly relates to historical data, anticipating good performance from methods like PatchTST, which use a uniform attention prior.\n    2. Future data is closely related only to a specific historical window $[t_0-a_i, t_0-a_i+\\Delta t]$, with constant $\\Delta t$ and $a_i$ for each sample.\n    3. Similar to the second scenario, but with $a_i$ varying across samples.\n    4. Possible other typical cases …\n    \n    Can the proposed method consistently lead in performance across these scenarios? Do the Effective Receptive Fields (ERFs) operate as anticipated for each case? I believe it is crucial that the results rigorously validate these aspects to strengthen the paper’s claims and ensure its conclusions are compelling and beyond reproach.\n    \n2. **Differentiation from previous Work**: The paper seems a application of 2D deformable attention [1] in vision transformer to 1D sequence forecasting. A more detailed comparison with [1] from the method design view could enhance the paper's contribution to the field. \n3. **Implementation Details**: The authors' promise to release the source code is appreciated. While early access would be beneficial for thorough validation and further exploration of the method's promising capabilities, considering the consistently leading performance and simple design.\n\n[1] Vision Transformer with Deformable Attention\n\n[2] Deformable Convolutional Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer fiTw for providing a detailed review and insightful questions.\n> W1 & Q4: Experiments on synthetic dataset to prove our model can handle both uniform and clustered attention distribution, especially the clustered ones (centralized attention in localized areas).\n\n+ Thanks for the valuable suggestion, we conduct experiments with some typical cases of attention distributions. \nThe details and results are provided in **Figure 1 of PDF in global response**.\n    \n+ Our method can accurately predict the future data in all cases. And ERFs can operate as anticipated, successfully matching the distributions of key information. In details, in the case of globally uniform attention, the brighter points in ERF are also distributed globally, which means the model can find the important time points across the whole series. In other cases, the brighter points in ERF tend to concentrate in localized areas of key information, **proving the effectiveness of our method in scenarios where key information is clustered within specific time window**. These results validate that our method can adeptly manage both uniform and clustered attention distributions. \n\n> W2: Differentiation from previous work.\n\nWe'd like to highlight our difference from previous works as follows:\n+ **Different model designs and implementations**: \n[1] is a computer vision method and focuses more on the locality of image. Therefore it uses the combination of three attentions (local attention, shift-window attention and deformable attention), while our method only use pure deformable attention.\nMeanwhile, [1] limits the range of attention offset to further enhance locality, while we do not use this restriction. \n+ **Using similar idea for totally different purpose**: \n    + Our method is inspired by the deformable operations in CV like [1], but is proposed for different purpose.\n[1] adopts deformable attention to construct an efficient and flexible backbone for vision tasks. \n    + But in our paper, we use deformable attention to solve the problem of over-reliance on patching, which is a significant problem in time series domain. \nBased on our exploration, we find out the reason behind this problem is that previous attentions have poor ability to focus on the important time points and thus need to rely on the guidance of patching.\nConsidering deformable attention enjoys better focusing ability by itself, we adopt it to solve this significant problem, **which is specific to our analysis**. \n    + Therefore, **although all adopting deformable attentions, we use it for totally different purpose and solve a completely different problem.** We hope this can clarify our difference from [1].\n+ We'd also like to emphasize that the problem of over-reliance on patching is of great significance to the time series community. \nOur proposed method is the first to deeply explore and specially target to this less-explored research problem, which is of great significance and novelty. And our findings can prompt people to rethink the relationship between attention and patching, and thereby design more powerful Transformer-based forecasters with wider applicability. Therefore, **in addition to proposing a state-of-the-art method for time series forecasting, the exposure and exploration of this significant problem is also a main contribution of our paper.**\n\n> W3: Implementation details & code release.\n\n+ As claimed in our **Reproducibility Statement**, we provide implementation details, model settings and pseudo-code in **Appendix A,B,C**. Details about tensor shape and model structure are also included in **Section 3**. \n+ **As per our tradition, we guarantee to make the code public upon paper acceptance.**\n\n> Q1: Do you use a restriction to limit the range of offset?\n\n+ No, we don't use any restrictions. Our strategy of not using restriction can **better manage both uniform and clustered attention**, which is a main concern of the reviewer.\n+ In terms of clustered attention, despite our reference points are initially uniformly distributed through the whole series, our strategy makes it possible for all of them to converge towards the same localized area because it allows offsets across the entire series. \n+ In terms of uniform attention, as shown in **Figure 8 of Appendix**, since our reference points are initially uniformly distributed through the whole series, our deformable attention can finally find the important time points in a global range.\nAnd Figure 8 also shows that, in case of need, our module can spontaneously learn a smaller offset value without additional restrictions, which verifies the soundness of our strategy that uses no restriction.\n\n> Q2: Description of important time point sampling is not in Algorithm 1.\n+ Thanks for your concern. Algorithm 1 mainly introduces the overall structure of our models, not the sampling process.\n+ For the details of sampling process, we describe it step by step in **Section 3.2 and Appendix I** through detailed text descriptions, \nincluding how to obtain references, calculate offsets and finish the sampling with linear interpolation.\nAnd we also provide **Figure 3 in main text** as a clear visual description.\n\n> Q3: Only use 2 closet points in I.1.\n\n+ We sample each important time point based on linear interpolation. In linear interpolation, we can assume that the output are more relevant to its two closest points, which brings better efficiency and simplicity.\n+ This assumption is also validated in 2D cases. The deformable methods in CV also accomplish their bi-linear interpolation only with the closest points [1][2].\n+ In real-implementation, Deformable Attention module will sample 12 important time points. Since the references for the sampling process are distributed across the whole series, **it can avoid the focus on local optima and ensure the learning potential.**\n\n> Q5: typo\n\nThanks for your reminder. We have fixed it. \n\n> The reference is the same as in the Official Review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "06llgiHv8v", "reviewer": "Reviewer_reZp", "summary": "The paper presents a novel approach for time series forecasting that relies less on patching. The authors incorporate deformable attention capturing important temporal information and a hierarchical structure that reduces memory consumption. The authors verify the effectiveness of their framework across multiple benchmarks: short-term, long-term, univariate, and multivariate.\nThe paper demonstrates state-of-the-art forecasting performance, reducing the dependence on patching.", "questions": "1. In the introduction it is stated that:\n> Patched-based Transformers have to work with a very long input length and a very large patch size to achieve ideal performance.\n\nCan you please cite the works that support this claim?\n\n> PatchTST proposes that attention mechanism can work better in temporal modeling with the help of large size patching technique.\n\nPatchTST suggests that patching can improve the long-term forecasting accuracy by using patch sizes between {8, 16} for input length of 336. The patch lengths used in your work are 4 and 8 and are referred to as small sized patches (under the input-768 settings and the input-384 settings).  Do you consider patch sizes of 8 and 16 to be large?\n\n> In such condition, the advanced Transformer-based models suffer from severe performance degradation due to the lack of patching, limiting their applicability to a wider range of forecasting tasks.\n\nDoes this statement refer to patch based Transformer models? Can you please cite the works that support this claim?\n\n2. The main text uses the term \"important points\" often and yet its meaning (e.g., time point in the similar changing stage, the inflexion point, the extremal point and so on) is mentioned in the appendix. Can you add an explanation for the term in the main text (figure 2 or figure 3 can be used for illustration)?\n\n3. One of the main claims of this work (including the title) is that Transformer-based models are too reliant on patching.\nSome of the latest Transformer-based models adopt the use of patching to reduce memory usage and improve performance. These methods leverage  (or taylor their methods to harness) the advantages of patching rather than relying on it. Will you be willing to change this claim?\n\n4. Have you conducted an ablation study showing the effect of patching (on input lengths 384 and 768) on you method? These result can strengthen the claim of the paper.\n\n5.  > In conclusion, full attention must work with patching to achieve ideal performance for it highly relies on the guidance of patching to focus on important time points.\n\nTo my understanding, this conclusion is based on the results of figure 1. The effective receptive field offers a nice visualization but not convincing enough to support the claim above. How did you validate the importance of the brighter points? Does having less important points is necessarily a good objective?", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, making it easy to read and understand.\n\n2. Authors provide good motivation for their design choices, as well as reasoning and ablations of the different hyperparameters for their method, e.g., hierarchical structure, patching and the number of important time points.\n\n3. The paper conducts comprehensive experiments in both short-term forecasting and long-term forecasting settings. The proposed method outperforms in the majority of the cases.", "weaknesses": "I don't see any major weaknesses for this paper. However, adding adjustments and justifications to some of the claims (see questions below) would make the paper more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer reZp for the detailed and insightful review.\n\n> Q1: Supports for our claims\n\nDue to the page limitation, please refer to **Global Response**.\n\n> Q2: Add an explanation for \"important points\" in the main text.\n+ We will explain the concept of \"important points\" in Section 1 before using it and label different types of \"important points\" in Figure 2 (b) of main text for illustration.\n+ Please refer to **Q5 1** for the detailed explanation.\n\n> Q3: About the claim like \"rely on\".\n\nWe are willing to revise our paper into a better shape. But we hope we can have more discussions about this claim.\n\nIn my humble opinion, both \"leverage the advantages of patching\" and \"rely on patching\" tend to emphasize that patching plays an important role in patch-based Transformers, but \"rely on\" goes deeper. And followings are the reasons why we use a deeper word like \"rely on\":\n+ It's a fact that patching has become a **must-have** technique for most latest Transformer-based models.\n+ The impact of patching on performance may be greater than we suppose.  As shown in **Figure 2 in global response PDF**, if without patching,  the performance of PatchTST and CARD will decrease obviously and fell out of the good rankings. This is a significant performance decrease, especially considering the intense competition in time series forecasting. **Considering the great impact of patching on the performance in patch-based Transformers, we use \"rely on\" to emphasize it**.\n\n> Q4: Ablation study showing the effect of patching (on input lengths 384 and 768) on you method. \n+ As shown in **Figure 6 of Appendix D**, our model is robust to patch sizes on input length 384.\n+ As shown in **Figure 2 of PDF in global response**, our method works well without patching, while other competitors suffer from obvious performance decrease.\n+ As shown in **Figure 3 of PDF in global response**, our model is robust to patch sizes on input length 768.\n\n> Q5 1: How to validate the importance of the brighter points. \n+ We are sorry that some concepts may be misleading and sincerely thanks for your reminder.\n+ We provide explanations of \"important time points\" and \"focused time points\" to avoid confusion between these two concepts:\n    + **The important time points refer to the time points that make contribution to a better performance**. Based on the findings in [1][2][3][4], the time points in time series are very redundant or even noisy. Thus, only a small number of points are needed to represent the properties of the time series and make contribution to better performance. These points are considered to be important and are mainly the time points that reflect the property of time series, such as time point in the similar changing stage, the inflexion point, the extremal point and so on.\n    + **The brighter points here means the focused time points by the models**. It just means that the model tends to focus on these time points when extracting temporal representation. The focused time points don't equal to the important time points in all cases.  Only when the focused time points are exactly the important time points, the model can focus on important information and learn useful representation to achieve a better performance.\n+ In conclusion, we validate the importance of a time points based on the performance. And a brighter point only means this point is focused by model when extracting temporal representation. It doesn't always mean a higher importance.\n+ **About the conclusion in Figure 1**:\n    + In left figure where almost all time points are brighter, the model focuses on all time points when extracting temporal representations but leads to a worse performance (MSE 0.385). Since the time points are very redundant or even noisy, focusing on the trivial part of them will influence the predictions. Thus, we state that \"attention has not learned to distinguish the importance of each time point, leading to trivial representation\" in line 47-49.\n    + In right figure where few time points are brighter, the model focuses on some selected time points and achieve better performance (MSE 0.367) after patching. Thus we can suppose that the model focuses on the important time points. And the pattern of ERF is also divided by patches, so we state that \"attention focus on a small number of important time points based on the patch partition\" in line 49-50.\n    + Above comparision of two figures proves our conclusion that \"attention highly relies on the guidance of patching to focus on important time points\" in line 51-52.\n+ To avoid misleading, we will clarify above concepts in Section 1 and re-introduce Figure 1 with more detailed description.\n\n> Q5 2: Does having less important points is necessarily a good objective?\n+ Based on our above explanation, we suppose this question should be \"Does having less focused points is necessarily a good objective\", because important time points are the inherent properties of time series, while the focused time points are the characteristic of the model that we can determine.\n+ In our opinion, having less focused points is a good objective. \nThe time points in time series are very redundant or even noisy. And it is inappropriate to focus on all of them because focusing on the trivial and noisy part of them will influence the predictions.\n+ But too few focused points is not appropriated. If the model can only focus on very few time points, it may miss some important time points, leading to performance degradation.\n+ As shown in **Figure 6 in Appendix D**, only sampling 6 points as important time points can achieve ideal performance, which prove that it's unnecessary to focus on all time points. Thus focusing on less time points is a good objective. But in some datasets, sampling 24 points as important time points performs better than only sampling 6 points, which proves that too few focused points is not appropriated.\n\n>Reference\n\n[1] PatchTST\n\n[2] Triformer\n\n[3] Informer\n\n[4] FiLM", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6y2RTP9QB", "reviewer": "Reviewer_Qoam", "summary": "The paper highlights the limitations of current time series forecasting models based on patching techniques. Existing models rely on patching to handle long sequences, but this approach is not suitable for all forecasting tasks. To overcome these limitations, DeformableTST introduces a Deformable Attention mechanism that dynamically selects important time points. This mechanism uses learnable offsets to sample significant time points from the input sequence and calculates attention only on these selected points, thereby reducing computational costs and improving performance.", "questions": "1. For highly complex time series data, Deformable Attention alone may not be sufficient, and additional preprocessing or postprocessing might be necessary. I am curious about how the authors plan to address this.\n2. While the paper is reasonable and convincing, it appears somewhat lacking in terms of novelty.", "limitations": "1. For highly complex time series data, Deformable Attention alone may not be sufficient, and additional preprocessing or postprocessing might be necessary. I am curious about how the authors plan to address this.\n2. While the paper is reasonable and convincing, it appears somewhat lacking in terms of novelty.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a method to address the limitations of current advancements that rely on patching techniques, which seems like a reasonable problem definition.\n2. The proposed deformable attention mechanism is an advanced form of sparse attention that reduces computational costs by selecting only significant time points using learnable offsets. This is particularly advantageous for handling long sequences.\n3. By dynamically selecting important information based on the characteristics of the data rather than relying on patching techniques, the model can be flexibly applied to various time series forecasting tasks.", "weaknesses": "New attention mechanisms have been proposed extensively from the past to the present. This paper seems to lack novelty in that regard.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer Qoam for providing thorough insightful comments.\n\n> W1 & Q2: Concerns about the novelty. While the paper is reasonable and convincing, it appears somewhat lacking in terms of novelty, since new attention mechanism has been proposed from past to the presently. \n\nWe'd like to highlight our novelty in the following points:\n+ **Our novelty is more than designing a new attention mechanism**:\n    + In this paper, we expose, analyze and solve the problem of over-reliance on patching in latest Transformer-based forecasters, **which is of great significance to the time series community.** And this significant problem is **less-explored in previous works.**\n    + Our proposed methods **is the first to** deeply explore and specially target to this novel research problem, **which highlights our novelty and contribution**.\n    + Meanwhile, our findings can **bring new perspective** and prompt people to rethink the relationship between attention and patching, which benefits the future work to design more powerful Transformer-based forecasters with wider applicability. \n    + Therefore, **in addition to proposing a state-of-the-art method for time series forecasting, the exposure and exploration of this novel problem is also a main novelty of our paper.**\n     \n+ Although existing many new attention mechanisms, **our proposed method still shows great distinctions from others**:\n    + **Different from latest Transformer-based models**: Most of the latest Transformer-based models highly rely on patching and thus have limited applicability.\n    Different from them, our DeformableTST can successfully reduce the reliance on patching and broaden the applicability of Transformer-based models. Specifically, it can flexibly adapt to multiple input lengths and achieve excellent performance in tasks unsuitable for patching,\n**which is a great improvement than previous Transformer-based models and further highlights our novelty and contribution**.\n    + **Different from prior-based sparse attentions**: As a novel data-driven sparse attention, our method differs from the previous prior-based sparse attentions by its excellent flexibility.\n    Due to the diverse pattern in different time series, the priors used in previous sparse attentions are hard to match all kinds of inputs, resulting in their inferior performance.\n    By contrast, our method can learn from the input time series dynamically, therefore is more flexible to the diverse property in different time series, leading to better performance.\n\n+ The exploration on more new attentions is of values: \n    + In our humble opinions, since the vanilla full attention is deficient in time series forecasting,\nexploration on more new attentions is of great significance. In details, we propose DeformableTST as a **novel methods for time series forecasting with better performance and wider applicability**, which is of great practical values in a wider range of real-world applications.\n    + We hope above clarifications help to distinguish our method from previous works and clarify our contributions, thus highlighting our novelty.\n\n\n> Q1: Additional preprocessing or postprocessing for highly complex time series data.\n+ Preprocessing method : Using an additive seasonal-trend decomposition method to decompose time series into long-term trend, seasonal, and residual components.\nThe effectiveness of this preprocessing is verified in some latest LLM for time series models through the compelling performance improvement[1][2].\nAnd different from early decomposition method using moving average[3][4], these latest decomposition methods are based on STL decomposition[5].\n\n+ Postprocessing: Using ensemble. In the most representative forecasting competitions like M1, M3 and M4, most participants employ ensemble methods to enhance performance[6].\n\n+ Postprocessing: Using Refinement. According to [7], we can use diffusion models to refine the prediction results and bring better performance in highly complex time series data. And [7] also proves that this postprocessing method is computationally efficient.\n\n> Reference\n\n[1] Defu Cao, et al. \"TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting.\"\n\n[2] Zijie Pan, et al. \"S2IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting.\"\n\n[3] Haixu Wu, et al. \"Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting.\"\n\n[4] Ailing Zeng, et al. \"Are Transformers Effective for Time Series Forecasting?\"\n\n[5] Robert Cleveland, et al. \"Stl: A seasonal-trend decomposition.\"\n\n[6] Boris Oreshkin, et al. \"N-BEATS: Neural basis expansion analysis for interpretable time series forecasting.\"\n\n[7] Marcel Kollovieh, et al. \"Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
> **Global Response**

We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve our paper further.

In this paper, we expose the significant problem of over-reliance on patching in latest Transformer-based forecasters. 
Based on our exploration, we find out the reason behind is that previous attentions have poor ability to focus on the important time points and thus need to rely on the guidance of patching.
To solve this problem, we propose DeformableTST, which equipped with deformable attention that enjoys better focusing ability by itself. 
Experimentally, our DeformableTST achieves the consistent state-of-the-art performance in a wider range of time series tasks, successfully reducing the reliance on patching and broadening the applicability of Transformer-based models.
    
The reviewers generally hold positive opinions of our paper, 
in that **"we effectively identify the core problem"**, our problem definition is **"reasonable"**, our method is **"novel", "advanced", "elegantly simple", "of good motivation"** and "**"clearly shows an advantage over prior works across all benchmarks"**, our paper is **"well-written, easy to read and understand"**, our experiments are **"comprehensive and thorough"** and we **"do an excellent empirical job"**.

The reviewers also raised insightful and constructive concerns. We made every effort to address all the concerns by providing sufficient evidence and requested results. Here is the summary of the major revisions:
+ **Experiments on synthetic dataset (Reviewer fiTw, eY9h)**: We use synthetic dataset to simulate typical scenarios and conduct systematic experiments to prove our method can adeptly manage both uniform and clustered attention distributions.
+ **More baselines and datasets (Reviewer eY9h)**: By comparing our method and the latest graph-Transformer Sageformer and conducting new experiments on Stock Market, we further demonstrate our consistent performance superiority.
+ **More ablations (Reviewer reZp)**: We provide ablations to prove our robustness to patching.
+ **Adjustments and justification for some claims (Reviewer reZp)**: We cite related works, explain our results and provide further experiments to support our claims. And we aslo provide explanations on some concepts to make them clearer.    
+ **Description on details of our methods (Reviewer fiTw)**: We illustrate the details of the sampling process and explains the soundness of our designs.
+ **Novelty and Contribution (Reviewer eY9h, Qoam)**: We highlight our difference with previous works in time series domains. Our method differs from previous Transformer-based forecasters **for its better applicability and less reliance on patching**. And we highlight **our contribution on exposing, exploring and solving the significant problem of over-reliance on patching**.
+ **Difference from CV methods (Reviewer fiTw)**: Our method differs from previous methods in CV for the different model designs and different purpose, which is specific to our analysis.

The valuable suggestions from reviewers are very helpful for us to revise our paper to a better shape. We'd be very happy to answer any further questions.

> **Tables and Figures**

All Tables are listed as follows. And Figures are provided in PDF.

> Table 1: Comparison of DeformableTST and Sageformer in long-term forecasting.

|Dataset MSE/MAE|ETTh1|ETTh2|ETTm1|ETTm2|Weather|Solar|ECL|Traffic|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|Ours|0.413/0.430|0.336/0.381|0.358/0.386|0.267/0.321|0.233/0.266|0.199/0.255|0.169/0.267|0.410/0.280|
|Sageformer|0.427/0.438|0.368/0.405|0.371/0.394|0.275/0.327|0.238/0.272|0.227/0.285|0.174/0.273|0.418/0.287|


> Table 2: Multi-variate short-term forecasting on Stock Market. 

|Models|Ours|Path.|CARD|GPT4TS|PatchTST|iTrans.|Auto.|FED.|RLinear|TiDE|TimesNet|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|MSE|0.122|0.134|0.133|0.140|0.137|0.134|0.144|0.140|0.163|0.160|0.133|
|MAE|0.151|0.159|0.159|0.162|0.171|0.159|0.195|0.184|0.188|0.186|0.158|

> **For Q1 of Reviewer reZp**
 
> For line 38-39
+ About claims on long input length:
    + [1][2][3] find that using input lengths longer than 96 can provide ideal performance. But they do not compare their performance under short input lengths like 96.
    + [4] compares these models under input-96 settings and observes significant decrease in their performance, indicating these patch-based Transformer forecasters need a long input to achieve ideal performance.
+ About claims on large patch size:
    + [1][2] study the impact of large patch size on long-term forecasting performance and find that performance will only improve **when a large patch size (at least more than 8)** is used.  
    
> Do you consider patch sizes of 8 and 16 to be large?
+ Considering the diversity of input lengths in real-world time series, patch sizes of 8 and more than 8 can be considered to be large.
In a wider range of time series tasks besides long-term forecasting,
the series length may be less than 10 (which mainly happens on short-term tasks with yearly or quarterly sampling frequency).
In such condition, patch sizes of 8 and 16 are very large.
+ In long-term forecasting, based on the findings in [1][2] that the performance will only improve when a large patch size (**at least more than 8**) is used,
we use 8 as a cutoff for the patch sizes.
+ The word "large patch size" refers not only to PatchTST, but also to the trend of increasing patch sizes in subsequent works (e.g., a very large patch size of 32 in [5]). 

> For line 38-39
+ Yes, this statement refers to patch-based Transformers. 
+ [3] conducts experiment on M4 but underperforms the classic baselines like N-BEATs. [6] also conducts experiment on M4 but its performance is not as compelling as it is in long-term tasks.

[1]PatchTST

[2]Crossformer

[3]GPT4TS

[4]iTransformer

[5]Pathformer

[6]CARD

Meta-review: The reviewers unanimously voted for acceptance. They appreciated the technical originality, clarity and extensive experiments that support the efficacy of the proposed approach. After carefully reading the reviews and the rebuttal, this meta-reviewer concurs with the reviewers' recommendations.
Decision: Accept (poster)