DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching

Authors: donghao  luo
xue  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 20:15:40

Abstract: With the proposal of patching technique in time series forecasting, Transformerbased models have achieved compelling performance and gained great interest from
the time series community. But at the same time, we observe a new problem that
the recent Transformer-based models are overly reliant on patching to achieve ideal
performance, which limits their applicability to some forecasting tasks unsuitable
for patching. In this paper, we intent to handle this emerging issue. Through diving
into the relationship between patching and full attention (the core mechanism
in Transformer-based models), we further find out the reason behind this issue
is that full attention relies overly on the guidance of patching to focus on the
important time points and learn non-trivial temporal representation. Based on this
finding, we propose DeformableTST as an effective solution to this emerging
issue. Specifically, we propose deformable attention, a sparse attention mechanism
that can better focus on the important time points by itself, to get rid of the need of
patching. And we also adopt a hierarchical structure to alleviate the efficiency issue
caused by the removal of patching. Experimentally, our DeformableTST achieves
the consistent state-of-the-art performance in a broader range of time series tasks,
especially achieving promising performance in forecasting tasks unsuitable for
patching, therefore successfully reducing the reliance on patching and broadening
the applicability of Transformer-based models. Code is available at this repository:
https://github.com/luodhhh/DeformableTST.

Main paper: 
 introduction: 
 Time series forecasting is widely used in real-world applications, such as transportation management [39, 5] , economic planning [41, 30, 31] , energy planning [42, 34] and weather forecasting [45] . Because of the immense practical value, time series forecasting has received great attention and has grown tremendously in recent years [44, 19, 33, 2, 28, 32, 25] .But looking back at the development of time series forecasting, Transformer-based models, who have sparked the boom of time series forecasting [55, 47, 57] , are constantly being challenged. In particular, some recent studies [52, 18, 22] have questioned that attention mechanism is not suitable for modeling the temporal dependency in time series. As the early strike back of Transformer-based models, PatchTST [35] proposes that attention mechanism can work better in temporal modeling with the help of large size patching technique. Afterwards, equipped with the growing patch size and increasing input length, the advanced Transformer-based models [53, 58, 51, 6] gain great performance improvement and successfully win back the championship in time series forecasting.However, with large size patching becoming a must-have technique for the following Transformerbased models, a new problem occurs: patched-based Transformers have to work with a very long input length and a very large patch size to achieve ideal performance [35, 53, 22] . But large size patching cannot be apply to all kinds of time series forecasting tasks. For example, some forecasting tasks are with limited input lengths [29, 30, 31] , which are not sufficient to be divided into patches. In such condition, the advanced Transformer-based models suffer from severe performance degradation due to the lack of patching [58, 51] , limiting their applicability to a wider range of forecasting tasks.To broaden the applicability of the Transformer-based model, we need to design an attention mechanism that is less reliant on patching (e.g., can work well with a small patch size or can work well even without patching). To this end, we first analyze exactly why attention must work with patching and why patching can help attention better model the temporal dependency in time series forecasting? We visualize the effective receptive fields (ERFs) of PatchTST [35] in Figure 1 . And the ERFs can indicate which parts of the time points in input series are focused by the model when extracting temporal representations. A surprising finding is shown in Figure 1 (left). If without patching, nearly all time points in input series are equally focused by the model and the model performs worse (MSE 0.385), exposing the problem of distracted attention. This finding means that attention has not learned to distinguish the importance of each time point in input series, leading to trivial representation. Note that the time points in a time series are very redundant or even noisy [35, 56, 55, 7, 53] , focusing on the trivial part of them will influence the predictions. Thus, an ideal time series forecaster should mainly focus on a small number of important time points which make contribution to better performance and reflect the property of time series. In Figure 1 (right), when using patching, the model focuses on some selected time points and achieve better performance (MSE 0.367), indicating that the model has successfully focused on the important time points. And in terms of why patching can guide the model to learn a non-trivial representation, we find that the pattern of ERF is also divided by patches, which means that patching can force the model to only focus on a small number of important time points based on the patch partition. As a conclusion of above discussion on Figure 1 , since full attention is unable to focus on the important time points by itself, it highly relies on the guidance of patching to focus on the important time points and learn non-trivial representation. This is the reason why full attention must work with patching to achieve ideal performance.Therefore, if we can find another way to help attention focus on the important time points, we can get rid of over-reliance on patching. Since full attention is hard to focus due to the redundancy in time series data [35, 56, 55, 7, 53] , replacing it with sparse attention can be a natural idea. There are some previous prior-based sparse attentions in time series community [55, 47, 57] . But due to the diverse pattern in different time series, their priors are hard to match all kinds of inputs, resulting in their inferior performance. Different from them, we introduce a data-driven sparse attention called deformable attention under the inspiration of deformable operations [8, 60, 48] . It can sample a subset of important time points from the input series based on the learnable offsets and only calculate attention with these selected important time points. These learnable offsets are learned from each input sample, therefore being more flexible to the diverse property in different time series.Based on the above motivations, we intend to broaden the applicability of Transformer-based models. To accomplish this goal, we propose DeformableTST, a Transformer-based model that is less reliant on patching. Technically, the patching process in our method is optional. We remove the patching process in most cases. Only when the input length is very long, we will use a small size patching for better efficiency. Since the removal of patching will cause severe memory usage in previous plain architecture, we adopt a hierarchical architecture to alleviate this efficiency issue. And we further introduce deformable attention, a data-driven sparse attention that can better focus on the important time points by itself, to achieve excellent performance without patching. Experimentally, DeformableTST achieves the consistent state-of-the-art performance in a wider range of time series tasks, especially in tasks unsuitable for patching, thus successfully reducing the reliance on patching and broadening the applicability of Transformer-based models. Our contribution are as follows:• We dive into the relationship between patching and attention. We point out a new problem that recent advanced Transformer-based models are too reliant on patching. And we further find out the reason behind this problem is that full attention relies overly on the guidance of patching to focus on important time points and learn non-trivial temporal representation.• To get rid of the over-reliance on patching, we propose DeformableTST and achieve the consistent state-of-the-art performance in a wider range of time series forecasting tasks. Experimental results show that our deformable attention can better model the temporal dependency in time series without reliance on patching.• We successfully broaden the applicability of Transformer-based models in time series tasks.Our DeformableTST can flexibly adapt to multiple input lengths and achieve excellent performance in tasks unsuitable for patching, which is a great improvement than previous Transformer-based models.2 Related Work 
tranformers for time series forecasting: 
 Transformer-based models mainly use attention mechanism to model the temporal dependency in time series [55, 47, 57] . In 2020s, they achieve excellent performance in time series forecasting for the first time and bring great attention to time series forecasting tasks [26, 9, 56, 20, 21, 7] . But their validity is questioned by [52, 18] with the finding that a simple linear layer can outperform complicated attention mechanisms. It's until the proposal of patching that Transformer-based models win back the championship in time series forecasting [35] . Based on patching technique, Pathformer [6] adopts a multi-scale patches structure. Crossformer [53] and CARD [51] further propose to additionally apply attention on variate and feature dimensions rather than only on temporal dimension. Sageformer [54] combines the graph methods with patch-based Transformer forecasters. And GPT4TS [58] also transfers pre-trained large language models to time series with the help of patching. But the question of whether attention is suitable for modeling the temporal dependency in time series still remains. For example, although adopting a Transformer architecture, iTransformer [22] still suggests that linear layers are more appropriate for temporal modeling. Meanwhile, the proposal of patching also comes with a new question that advanced Transformer-based models are too reliant on patching. Therefore, further research about Transformer-based forecasters are still needed, especially on the question of how to better use attention in temporal modeling without over-reliance on patching.Sparse attention used to be popular in time series forecasting. Early Transformer-based models usually adopt prior-based sparse attention mechanisms. Informer [55] adopt ProbSparse attention to model the temporal dependency. Autoformer and FEDformer [47, 57] further combine the signal processing technique with the attention mechanisms and select the top-k sparse representation in time domain or frequency domain respectively. But due to the diverse pattern in different time series, these priors are hard to match all kind of inputs, resulting in their inferior performance. As a comparison, data-driven sparse attention, also called deformable attention, is more flexible to diverse inputs. Similar idea has been explored in Computer Vision (CV). Inspired by deformable convolution [8, 59] , deformable DERT [60] proposes multi-scale deformable attention for object detection tasks. And [48, 49] further improve it and make it suitable for general CV tasks. In this work, we propose a deformable attention for time series forecasting to break through the bottleneck faced by previous attention mechanism in modeling temporal dependency.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please refer to Appendix K. And model efficiency is provided in G.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
deformabletst: 
 Given an observed multivariate or univariate time series as input, time series forecasting aims to predict the length-T future series based on the length-I input series. In real-world scenarios, the input length I varies from a wide range and is not always sufficient for patching technique, leading to the limited applicability of previous patch-based Transformer forecasters. To tackle this problem, we propose DeformableTST. And we introduce details of DeformableTST in following subsections. As shown in Figure 2 , our DeformableTST adopts the encoder-only architecture of Transformer [43] , including the input embedding layer, hierarchical Transformer backbone and prediction head. And following the recent Transformer-based models, we adopt RevIN [15] to mitigate the distribution shift between the training and testing data.Input Embedding Layer Denoted X in ∈ R M ×I as the M variates input time series of length I, it will be divided into N 0 non-overlapping patches and then embedded variate-independently into D 0 -dimensional embeddings:EQUATIONX 0 ∈ R M ×D0×N0 is the input embedding. It is worth noting that DeformableTST is less reliant on patching and thus the patching process is optional. We only adopt patching when the input length is very long for efficiency reasons. And we also adopt a much smaller patch size than recent Transformer-based models, making it more adaptable to diverse input lengths.The backbone is stacked by L Transformer blocks and utilizes a hierarchical structure. The forward process in the i-th block is simply formulated as follows:EQUATIONEQUATIONEQUATIONX i ∈ R M ×Di×Ni is the output feature series of the i-th block, i ∈ {1, ..., L}. And D i and N i are the sizes of its feature and temporal dimensions. DeformableAttention is the core component to better cpature the global temporal dependency, which will be introudced in Section3.2. LPU and ConvFFN are local enhancement modules (Figure 2 (b) and (d)). LPU is the local perception unit, a depth-wise convolution with residual connection [10] . And ConvFFN is a feed-forward network injected with a depth-wise convolution [50] . These two modules are adopted to improve the local temporal modeling ability. And a GELU activation [11] is adopted in ConvFFN to provide nonlinearity when learning the new feature representation. Meanwhile, to construct a hierarchical structure, a downsampling convolution layer [24] with kernel size 2 and stride 2 is adopted between two blocks, which will halve the series' temporal dimension and double the feature dimension.Prediction Head We first flatten the final representation from the backboneX L ∈ R M ×D L ×N L into R M ×(D L ×N L ) .Then we obtain the prediction through a linear projection layer:EQUATIONWhere Y ∈ R M ×T is the prediction of length T with M variates. Figure 3 introduces the detailed process of our deformable attention. In each attention module, it first samples a few important time points from the input feature series X based on the learnable offsets.Then the sampled important time points are fed to the key and value projections to get the sampled key and value tokens K, Ṽ. Meanwhile, the input feature series X is also projected into queries Q.Finally, standard multi-head attention [43] is applied to Q, K, Ṽ to obtain the attention output O.Sample the Important Time Points As shown in Figure 3 (a), we sample the important time points based on a set of learnable coordinates called sampling points. Specifically, the sampling points are calculated by a set of uniformly sparse reference points and their learnable offsets.Given a length-N feature series X ∈ R M ×D×N , we first generate the sparse reference points T ref ∈ R M ×1×Nsamp from a 1D uniform grid. The grid size N samp = N/r is downsampled from the input series length N with a downsampling factor r to provide sparsity. The reference points indicate the 1D coordinates of some time points uniformly distributed in the feature series X with interval r. These coordinate values are normalized to [-1, +1], where -1 indicates the start of the series and +1 means the end of the series. And these reference points serve as the initial coordinates for the following deforming process.Then we obtain the offsets for each reference point by offset sub-network (Figure 3 (b) ). It contains two convolution layers. The first layer is a depth-wise convolution, which can take the local neighbors into consideration when generating the offsets [48] . It takes the query tokens Q as input, where Q is the linear projection of the feature series X. After a nonlinear activation, the output from the first layer is passed into a point-wise convolution layer to generate the offsets ∆T ∈ R M ×1×Nsamp .Adding up the reference points with the learnable offsets, we obtain N samp sampling points, which can serve as the final coordinates to sample the important time points from the feature series X.In practice, we follow [48, 60] and calculate the values of these important time points by linear interpolation φ(•; •) to make this sampling process differentiable. The overall process is as follows:EQUATIONwhere X ∈ R M ×D×Nsamp is the sampled feature series consisting of the important time points. And the implementation of linear interpolation φ(•; •) is in Appendix I.1. And we clip T samp by -1 and +1 to avoid sampling outside the feature series.In above sampling process, we have got the query tokens Q. After the sampling process, we can get the sampled key and value tokens K, Ṽ after two linear projections of the sampled feature series X. Then we calculate the multi-head self-attention with H heads as:EQUATIONEQUATIONwhere d = D/H is the dimension of each head. The upper index (h) denotes the h-th attention head.After concatenating the output embedding from each attention head O (h) together, we obtain the output of the DeformableAttention module O ∈ R M ×D×N through a linear projection. B is the deformable relative position bias to provide the positional information into the attention map and its implementation is introduced in Appendix I.2.To conclude, this subsection introduces the detailed process of DeformableAttention (Eq.( 3)). And for the i-th block, X in this subsection corresponds to X local i in Eq.( 3) and O corresponds to X global i .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 We thoroughly evaluate our DeformableTST on a wide range of time series forecasting tasks, including long-term forecasting tasks with various input lengths, as well as multivariate and univariate shortterm forecasting tasks that are unsuitable for patching, to verify the performance and applicability of our DeformableTST.Baselines We extensively include the latest and advanced models in time series community as strong baselines, including patch-based Transformer models: Pathformer [6] , CARD [51] , GPT4TS [58] , PatchTST [35] ; non patch-based Transformer models: iTransformer [22] , FEDformer [57] , Autoformer [47] ; other non Transformer-based models: RLinear [18] , TiDE [9] , TimesNet [46] , DLinear [52] and SCINet [20] . We also include the state-of-the-art models in each specific task as additional baselines for a comprehensive comparison.Main Result As shown in Figure 4 , our DeformableTST achieves consistent state-of-the-art performance in a broader range of time series tasks. In details, DeformableTST can flexibly adapt to multiple input lengths and especially achieve excellent performance in tasks unsuitable for patching, which is a great improvement than previous Transformer-based models, proving that our DeformableTST can successfully reduce the reliance on patching and broaden the applicability of Transformer-based models. Experiment details and result discussions of each task are provided in following subsections. In each table, the best results are in bold and the second best are underlined. Setups We conduct long-term forecasting experiments on 8 popular real-world benchmarks, including Weather [45] , Traffic [39] , ECL [42] , Solar-Energy [34] and 4 ETT datasets [55] . In this paper, we refine the evaluation approach for a comprehensive comparision of the models. Different from previous settings that use a fixed short input length (e.g., 96) [55, 47, 22] . We fix three different input lengths {96, 384, 768} and calculate the averaged results to adequately reflect model's adaptability to multiple input lengths. These input lengths covers a variety of real-world appilication scenarios, i.e., shorter than prediction lengths, within the prediction lengths' interval and longer than prediction lengths. Following the previous settings, we set prediction lengths as {96, 192, 336, 720} and calculate the MSE and MAE of multivariate time series forecasting as metrics.Results ETTm1 0.358 0.386 0.375 0.394 0.368 0.386 0.373 0.393 0.365 0.391 0.386 0.405 0.456 0.462 0.576 0.523 0.379 0.390 0.383 0.397 0.412 0.414 ETTm2 Time series community are currently focusing on long-term forecasting tasks, where the input length and prediction length are adequate for patching technique. However, the short-term forecasting tasks, where the input length and prediction length are limited, are also of extensive practical value in real-world appilications. The study on short-term forecasting tasks has been stagnant in recent years and the advanced patching technique proposed in long-term forecasting is hard to apply to short-term forecasting due to the limited input length. To validate the applicability of our DeformableTST in short-term forecasting, we extensively conduct experiments in following two kinds of tasks.We conduct multivariate short-term forecasting experiments on 8 popular real-world benchmarks, including Exchange [17] , ILI [3] , 2 ETTh [55] and 4 PEMS datasets [5] . We set prediction lengths as {6, 12, 18} and set the input length to be 2 times of the prediction length, which precisely meets the definition of limited input lengths in short-term forecasting. We calculate the MSE and MAE of multivariate time series forecasting as metrics.The study on univariate short-term forecasting tasks used to be popular in the early time series community [29, 31, 37] but has been stagnant in recent years. In this paper, we bring back this classic tasks and conduct experiments on following datasets: M1 [29] , M3 [30] , M4 [31] , Tourism [1], NN5 [41] , Hospital [12] and KDD Cup [13] . Following the classic settings [29, 37] , we calculate the SMAPE as metric. Specially for M4 datasets, we follow the rules of M4 competition [31] and use MASE and OWA as additional metrics. The prediction lengths are from 2 to 48 and the input length is 2 times of the prediction length.We emphasize the difference between multivariate and univariate tasks as follows. In multivariate tasks, all input samples are obtained by sliding window from the same multivariate long series. Therefore, there is a high degree of similarity among the input samples in multivariate tasks. In contrast, the input samples in univariate tasks are collected from many different univariate time series sources. As a result, the input samples in univariate tasks may differ from each other and have quite different temporal property, making the univariate short-term forecasting tasks much more difficult.As shown in Table 2 and 3 . DeformableTST performs excellently in short-term forecasting.Compared with the second best model, it achieves averaged 14.1% SMAPE promotion in univariate tasks and averaged 25.6% MSE promotion in multivariate tasks. As a comparison, the limited input length in short-term forecasting poses a dilemma for patch-based Transformer forecasters. Using patching will lead to very few tokens, making it unable to fully utilize the long-term modeling ability in attention. Not using patching will lead to the distracted attention on all input time points, making it hard to extract non-trivial temporal information. As a result, previous patch-based Transformer forecasters fail in many cases of short-term forecasting tasks. And in multivariate short-term forecasting, the variate correlation plays an important role to the final results for the temporal information is limited due to the limited input length. Therefore, CARD and iTransformer, which can learn the variate correlation, achieve ideal performance in PEMS datasets whose variate number is very large. As a variate-independent method, our DeformableTST still competes favorably with these cross-variate methods, further demonstrating its excellent temporal modeling ability to extract useful information even from the limited inputs. It will be our future work to study how to capture the multivariate correlation in our model, which will further improve the performance. Meanwhile, the great diversity in univariate samples makes it more difficult to learn temporal representation in univariate short-term forecasting. Therefore, the linear baselines and iTransformer, which adopt linear layers on the temporal dimension, suffer from inferior performance due to the insufficient representation capability in linear layers. By contrast, thanks to the better representation capability in attention mechanism and the better focusing capacity in the proposed deformable attention, our DeformableTST is particularly good at short-term forecasting tasks, which is a great improvement than previous Transformer-based models, therefore successfully broadening the applicability of Transformer-based models. In order to validate the effectiveness of our designs, we start from a PatchTST [35] and gradually update it into our DeformableTST by adding our designs step by step. And we provide the trajectory going from a PatchTST to a DeformableTST in Figure 5 . To get rid of the over-reliance on patching, we remove the patching design in PatchTST. After this step, PatchTST suffers from degradation in both performance and efficiency. To address these issues, more designs are adopted in DeformableTST.First, the removal of patching leads to a larger number of tokens processed in the attention computation, resulting in heavier memory usage. Responding to the issue, we adopt a hierarchical structure to gradually reduce the number of tokens, therefore alleviating the efficiency problem. Secondly, full attention is hard to focus on the important time points after removing the patching design, leading to trivial temporal representation and performance degradation. To help attention better focus without patching, we propose deformable attention as a complementary. Thanks to the better focusing ability in deformable attention, we observe great performance improvement after adding this design. Meanwhile, we also adopt some local enhancement modules in our design since locality is also important in time series [20] . And this step also brings performance improvement. After equipped with all our designs, DeformableTST shows great performance and efficiency superiority than the baseline PatchTST, which proves the necessity and effectiveness of our designs. 
compared deformable attention with prior-based sparse attention: 
 In Section 1, we propose that sparse attention can help attention to better foucs without patching and further argue that data-drivien sparse attention is more appropriate than prior-based ones. To validate the necessity and effectiveness of using data-drivien sparse attention, we compare our deformable attention with some prior-based sparse attentions in time series community, that is ProbSparse Attention in Informer [55] , AutoCorrelation in Autoformer [47] and FourierAttention in FEDformer [57] . We also include the local window attention in Swin Transformer [23] and the vanilla full attention [43] adopted in most baselines [22, 53, 35] for a comprehensive comparison.As shown in Table 4 , our deformable attention surpasses other prior-based competitors in all benchmarks. This is because the priors are hard to match all kind of inputs due to the diverse pattern in different time series, resulting in the inferior performance of prior-based sparse attentions. Different from them, our deformable attention is a data-driven sparse attention that can learn from the input time series, therefore is more flexible to the diverse property in different time series.Our deformable attention also surpasses the full attention by a large margin for it can better focus on the important time points to learn non-trivial temporal representation, while the full attention suffers from the distracted attention and trivial temporal representation due to the lack of patching.Although window attention can help attention avoid being distracted in a global range by limiting the attention computation into local windows, its performance still decreases due to the lack of long-term modeling ability, which is an important ability a time series forecaster should have. We evaluate the multivariate long-term forecasting performance on 8 popular real-world datasets, including Weather, Traffic, ECL, Solar-energy and 4 ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2). And for multivariate short-term forecasting tasks, we choose Exchange, ILI, 2 ETTh datasets and 4 PEMS datasets for benchmarking. These datasets have been extensively utilized for benchmarking and cover many aspects of life.The variate number, dataset size and sampling frequency of each dataset are summarized in Table 5 . We follow standard protocol [55] and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT and PEMS dataset and 7:1:2 for the other datasets. And training, validation and test sets are zero-mean normalized with the mean and standard deviation of training set. Each of above datasets only contains one continuous long time series, and we obtain samples by sliding window.More introduction of the datasets are as follow:1) Weather 1 contains 21 meteorological indicators of Germany in 2020.2) Traffic 2 contains the road occupancy rates measured by 862 different sensors on San Francisco Bay area freeways in 2 years.3) ECL(Electricity) We conduct univariate short-term forecasting experiments on 7 popular datasets, including M1, M3, M4, Tourism, NN5, Hospital and KDD Cup. We emphasize the difference between multivariate and univariate tasks as follows. In multivariate tasks, all input samples are obtained by sliding window from the same multivariate long series. Therefore, there is a high degree of similarity between the input samples in multivariate tasks. In contrast, the input samples in univariate tasks are collected from many different univariate time series sources. As a result, the input samples in univariate tasks may differ from each other and have quite different temporal property, making the univariate short-term forecasting tasks much more difficult. ( Y i -Y i ) 2 MAE = 1 T T i=0 Y i -Y iwhere Y, Y ∈ R T ×M are the M variates prediction results of length T and corresponding ground truth. Y i means the i-th time point in the prediction result.Implementation Details Our method is trained with the L2 loss, using the ADAM [16] optimizer with an initial learning rate in {10 -3 , 5 × 10 -4 , 10 -4 }. The default training process is 50 epochs with proper early stopping. The mean square error (MSE) and mean absolute error (MAE) are used as metrics. All the experiments are repeated 5 times with different seeds and the means of the metrics are reported as the final results. All the deep learning networks are implemented in PyTorch [38] and conducted on NVIDIA A100 40GB GPU.Model Parameter By default, DeformableTST contains 6 Transformer blocks with the model dimension D = 256 and FFN expansion α = 4. The number of important time points N samp is in the range of 1 to 12 in short-term forecasting tasks, which is depended on the different input lengths. Due to the limited input lengths, we do not adopt patching and downsampling layers in short-term forecasting tasks. For baseline models, we follow the official codes with the recommended model parameters and some of their model parameters are re-searched from following searching space: number of blocks L from {2, 4, 6}, model dimension D from {64, 128, 256} and FFN expansion α from {1, 2, 4, 8}.Metric We adopt the mean square error (MSE) and mean absolute error (MAE) of multivariate time series forecasting as metrics.MSE = 1 T T i=0 ( Y i -Y i ) 2 MAE = 1 T T i=0 Y i -Y iwhere Y, Y ∈ R T ×M are the M variates prediction results of length T and corresponding ground truth. Y i means the i-th time point in the prediction result.Implementation Details Our method is trained with the SMAPE loss, using the ADAM [16] optimizer with an initial learning rate of 5 × 10 -4 . The default training process is 50 epochs with proper early stopping. Following [46] , we fix the input length to be 2 times of prediction length for all models. All the experiments are repeated 5 times with different seeds and the means of the metrics are reported as the final results.Model Parameter By default, DeformableTST contains 4 Transformer blocks with the model dimension D = 256 and FFN expansion α = 4. Due to the limited input lengths, we do not adopt patching and downsampling layers in short-term forecasting tasks.Metric For the M4 datasets, following [31, 37] , we adopt the symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE) and overall weighted average (OWA) as the metrics, which can be calculated as follows:SMAPE = 200 T T i=1 |Y i -Y i | |Y i | + | Y i | , MAPE = 100 T T i=1 |Y i -Y i | |Y i | , MASE = 1 T T i=1 |Y i -Y i | 1 T -p T j=p+1 |Y j -Y j-p | , OWA = 1 2 SMAPE SMAPE Naïve2 + MASE MASE Naïve2 ,where p is the periodicity of the data. Y, Y ∈ R T ×M are the M variates prediction results of length T and corresponding ground truth. Y i means the i-th time point in the prediction result.For other datasts, we adopt the symmetric mean absolute percentage error (SMAPE) as the metric, which can be calculated as follows:SMAPE = 200 T T i=1 |Y i -Y i | |Y i | + | Y i | ,where Y, Y ∈ R T ×M are the M variates prediction results of length T and corresponding ground truth. Y i means the i-th time point in the prediction result.We provide the pseudo-code of DeformableTST in Algorithm 1. 1: X in = RevIN(X in , mode=norm) X ∈ R B×I×M 2: X in = X in .transpose X ∈ R B×M ×I 3: X in = X in .reshape X in ∈ R (BM )×1×I 4:Embedding the input time series variate-independently with optional patching.5: X 0 = Embedding(X in ) X 0 ∈ R (BM )×D0×N06: for i in {1, . . . , L}: Run through DeformableTST blocks.7: for The local perception unit (LPU) is used to learn the local temporal information.8: for X local i = LPU(X i-1 ) X local i ∈ R (BM )×Di×Ni 9:for The deformable attention is adopted to learn the global temporal information.10: for X global i = LayerNorm X local i + DeformAttention(X local i ) X global i ∈ R (BM )×Di×Ni11: for The feed-forward network injected with a depth-wise convolution (ConvFFN) is used to learn the local temporal information and the new feature representation.12: forX i = LayerNorm X global i + ConvFFN(X global i ) X i ∈ R (BM )×Di×Ni13: for Adopting the optional downsampling layer between two blocks.14: for if i < L and Use_Downsampling is True:15: for if X i = DownSampling(X i ) X i ∈ R (BM )×(2Di)×(Ni/2)16: End for The results are shown in Figure 6 . In general, our model is robust to the choice of model parameters.Compared with the default block number L = 4, stacking more blocks will bring further performance improvement. Considering both performance and efficiency, we recommend to fix the block number as 4 in long term forecasting tasks.We also compare our model sensitivity to patch size with PatchTST's. As shown in Figure 6 , our model is less sensitive to the choice of patch size, therefore successfully getting rid of the over-reliance of patching. In Section 1, we visualize the effective receptive fields (ERFs) of PatchTST [35] based on [27, 14] to see which parts of the time points in input series are focused by the model when extracting temporal representations. Here we provide more ERF visualization results with other patch-based Transformer forecasters. In a ERF figure, a brighter area means that the model tends to focus on these time points when extracting temporal representation, and thus these time points will contribute more to the middle point of the final representation.As shown in Figure 7 , the ERFs of the patch-based Transformer forecasters highly rely on the guidance of patching. If without patching, nearly all time points in input series are equally focused by the model and the model performs worse, exposing the problem of distracted attention. This finding means that attention has not learned to distinguish the importance of each time point in input series, leading to trivial representation. After patching, these Transformer forecasters tend to foucs on some important time points based on a patch partition. This phenomenon means that: although patching can help attention better focus on important time points, the guidance of patching tend to distribute the focused points evenly among all patches. But in real-world scenarios, it is possible that the important time points are not evenly distributed among all patches, which is inconsistent with the tendency in the guidance of patching, leading to the inferior performance of these patch-based Transformer forecasters in some cases. There is also some difference among the ERFs of these patch-based Transformer forecasters due to their different specific designs. PatchTST [35] tends to only focus on a very few time points in each patch and ignore others. But CARD [51] can focus on more time points in one patch thanks to its additional attention across feature dimension, which aligns the information within patch. And the ERF of GPT4TS [58] also reveals the autoregressive property in GPT [40] backbone, making it only focus on the time points before the the middle point.By contrast, we provide the ERF of our DeformableTST in Figure 8 for comparison. By default, we set N samp = 12. Under input-96 settings, our DeformableTST does not adopt patching technique but can still focus on a small number of important time points, proving that our DeformableTST can foucs well by itself without the need of patching. Under input-384 settings, we adopt a small size patching and divide the input series into 96 patches to alleviate the efficiency issue. Under such condition, our DeformableTST still does not focus based on a patch partition. Although adopting patching technique, the focused time points in our DeformableTST is not evenly distributed among all 96 patches, therefore being less reliant on patching. More ablation results are provided in Figure 9 . Case Study We provide the visualization of learned important time points as an intuitive comparision for some popular sparse attentions in time series community. As shown in Figure 10 , the keys of window attention are restricted within the local window. And the important keys for ProbSparse attention tend to gather in a small area around the time point that most fits the prior. This property makes window attention and ProbSparse attention only focus on a small local area and fail to find the important time points in a global range, leanding to the loss of information. Meanwhile, imporatant time points refer to the time points that reflect the property of time series and make contribution to better performance. And the types of imporatant time points are varied (e.g., time point in the similar changing stage, the inflexion point, the extremal point and so on). But AutoCorrelation can only foucs on the time points in the similar changing stage due to its prior, resulting in the lack of diversity. By contrast, our deformable attention can find the important time points in a global range for the reference points in the sampling process are uniformly distributed throughout the whole input time series. And our deformable attention can also learn different types of important time points because it is less reliant on specific priors and can foucs on any appropriate important time points based on the learnable offsets learned from the input series. Therefore, our deformable attention can find more types of important time points in a wider range, therefore being more flexible to the diverse property in different time series and performing better than other prior-based sparse attentions. We We report the standard deviation of DeformableTST performance under five runs with different random seeds in Table 7 , which exhibits that the performance of DeformableTST is stable. As mentioned in Section 3.2, we sample the important time points based on a set of learnable coordinates called sampling points T samp . In practice, after obtaining the sampling points T samp , we achieve this sampling process via linear interpolation following [48, 60] . In details, we calculate the values of these important time points by linear interpolation φ(•; •) to make this sampling process differentiable. And the linear interpolation φ(•; •) is calculated as follows:EQUATIONwhere g(a, b) = max(0, 1 -|a -b|) and t indexes all the coordinates on X ∈ R M ×D×N . The value of T samp is de-normalized back to the range of [0, N -1] before passed into g. As g would be non-zero only on the 2 integral coordinates closest to T samp , it simplifies Eq.( 11) to calculate the value of each important time point as the weighted average of its only 2 closest time points.Due to the deforming process and hierarchical representation, the fixed absolute positional embedding is not suitable for our design. Instead, we adopt a relative position bias to encode the position information into the attention map [23] , which represents the relative positional information between the query token series Q and the key token series K.Considering the input feature series X ∈ R M ×D×N of the attention module, the relative coordinate displacements of this length-N series contain (2N -1) different values and lie in the range of [-N, N ]. Then we will maintain a learnable parameterized bias table B ∈ R H×(2N -1)×1 , in which each element represents one of the above (2N -1) relative coordinate displacements.Meanwhile, to represent the relative positional information between the query token series Q and the key token series K, we also need to denote the coordinates of Q and K. The coordinates of Q are generated from a 1D uniform grid with size N . It indicates the 1D coordinates of all N time points in the query series Q. These coordinate values are normalized to [-1, +1], where -1 indicates the start of the series and +1 means the end of the series. And the coordinates of K are the sampling point T samp . Then the relative position R ∈ R N ×Nsamp×1 is calculated as follows:EQUATIONThen we clip R by -1 and +1 and obtain deformable relative positional bias B by linear interpolation to the learnable parameterized bias table B with the continuous relative displacements R as follows:EQUATIONJ More Discussions on Patching and Transformer-based ModelsImpact on Performance As shown in Figure 1 and 7, although patching can help attention better focus on important time points, the guidance of patching tend to distribute the focused points evenly among all patches. But in real-world scenarios, it is possible that the important time points are not evenly distributed among all patches. In some cases, most of the important time points are only distributed in a few patches while other patches only contain unimportant time points. Such cases are inconsistent with the tendency in the guidance of patching, leading to the inferior performance of the patch-based Transformer forecasters in these cases. And patching technique must work with a very long input length because leveraging patching on short time series leads to very few tokens, limiting attention's ability in long-term modeling. These drawbacks limit the performance and applicability of the previous patch-based Transformer forecasters.Impact on Efficiency In addition to helpling attention focus better, patching can also improve the efficiency by reducing the number of tokens. When facing a very long input length (e.g., 384 and 768), we still adopt a small patch size in our design for better efficiency. Therefore this paper is not a call to completely abandon patching technique in all cases, but an extension to scenarios where patching technique is not suitable.To highlight how we innovate and upgrade the Transformer-based model, we compare our De-formableTST with some milestone Transformer-based models in time series community.Compared with PatchTST [35] PatchTST is a milestone Transformer-based model and De-formableTST can be seen as an improvement of it to solve the problem of over-reliance on patching.PatchTST highly relies on patching for patching can force the attention to focus on only a few important time points within each patch and ignore other time points within the patch. But the choose of patch size is a dilemma in practice: a too large patch size could lead to many other time points within the patch being ignored, resulting in the risk of neglecting other priorities. And a too small patch size will lead to a huge amount of tokens, making it hard to focus. This dilemma cannot be resolved by PatchTST itself. To address this issue, we propose deformable attention, an attention mechanism that can focus well by itself, to get rid of the need of patching. Since deformable attention enjoys great focusing ability, we can use a small patch size to mitigate the problem of some priorities being ignored and free to worry about attention being hard to focus on important time points, successfully reducing the reliance on patching and resolving the dilemma.Compared with iTransformer [22] DeformableTST and iTransformer are the lastest Transformerbased forecasters. Both methods focus on the underperformance issue of previous attention mechanism in temporal modeling and devote to designing a more powerful Transformer-based forecaster. But they hold different opinions. Although iTransformer still adopts a Transformer architecture, it insists that attention is not suitable for temporal modeling while linear layers are more appropriate. This design makes its performance more similar to linear-based forecasters rather than Transformerbased ones, being less competitive in difficult tasks (e.g., univariate short-term forecasting tasks with lower similarity between samples). Different from iTransformer, our DeformableTST follows the tradition of Transformer-based models to still uses attention for temporal modeling. Specifically, in this paper, through experiments and analysis, we further attribute the underperformance issue of previous attention mechanism to their over-reliance on patching and thus propose deformable attention to solve this problem, successfully designing a more powerful Transformer-based forecaster and broadening the applicability of Transformer-based models.In this paper, we mainly focus on how to better use attention in temporal modeling and do not consider the multivariate correlating. It will be our future work to study how to further capture the multivariate correlation in our model, which can hopefully improve the performance, especially in datasets with large number of variates. Besides, we will further explore the potential of our DeformableTST in more time series analysis tasks and further develop its performance by large-scale pre-training in the future.Our work only focuses on the time series forecasting problem, so there is no potential ethical risk.Our model achieves the state-of-the-art performance on a wider range of time series forecasting tasks, covering a large amount of real-world scenarios. Therefore, the proposed model makes it promising to tackle real-world forecasting problem, helping our society make better decisions and prevent risks in advance. And we hope our findings can prompt people to rethink the relationship between Transformer-based models and patching technique, thereby designing more powerful Transformerbased forecasters with a wider range of applicability.Our paper mainly focuses on scientific research and has no obvious negative social impact.In the main text, we have strictly formalized the model architecture with equations. All the implementation details are included in the Appendix, including dataset descriptions, metrics, model, and experiment configurations. Code is available at this repository: https://github.com/luodhhh/ DeformableTST.Due to the space limitation of the main text, we place the full results of all experiments and results of more baselines in the following subsections. And we also provide the showcases in Appendix O.The full results of long-term forecasting with input length 96 are provided in Table 8 . And more results with more baselines are also included.The full results of long-term forecasting with input length 384 are provided in Table 9 . And more results with more baselines are also included.The full results of long-term forecasting with input length 768 are provided in 10. And more results with more baselines are also included.The full results of multivariate short-term forecasting tasks are provided in Table 11 . And more results with more baselines are also included.The full results of M4 datasets are provided in Table 12 . And the full results of other datasets are provided in Table 13 . 1To provide an intuitive comparison among different models, we provide showcases to the longterm forecasting tasks under two representative cases (the time series is in declining stage and the time series is in rising stage). The results are in Figure 12 -13. Among the various models, our DeformableTST predicts the most precise future series variations and exhibits superior performance. We compare our model with Sageformer [54] , the latest Graph-Transformer model. Since Sageformer will produce NaN outputs in some short-term forecasting tasks, we mainly conduct comparisons in long-term forecasting. The results are shown in Table 14 . Our DeformableTST achieves consistently better performance than the latest Graph-Transformer method, further demonstrating our performance superiority. We conduct experiments on synthetic dataset with some typical cases of attention distributions to prove our model can handle both uniform and clustered attention distribution. The details and results are provided in Figure 14 .As shown in Figure 14 , our method can accurately predict the future data in all cases. And ERFs can operate as anticipated, successfully matching the distributions of key information. In details, in the case of globally uniform attention, the brighter points in ERF are also distributed globally, which means the model can find the important time points across the whole series. In other cases, the brighter points in ERF tend to concentrate in localized areas of key information, proving the effectiveness of our method in scenarios where key information is clustered within specific time window. These results validate that our method can adeptly manage both uniform and clustered attention distributions.We conduct ablation study to show the effect of patching on our method (under input-384 and input-768 settings). As shown in Figure 6 and Figure 16 , our model is robust to the choice of different patch sizes on input length 384 and input length 768.Meanwhile, as shown in Figure 15 , the performance of other Patch-based Transformer competitors (e.g., PatchTST [35] and CARD [51] ) will decrease obviously and fell out of the good rankings if without patching. This is a significant performance decrease, especially considering the intense competition in time series forecasting. By contrast, our method works well without patching, which further verifies our robustness to the use of patching and shows that our model can successfully get rid of the over-reliance of patching. Justification: Please refer to Section 1 and Abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The code packages, baseline models and datasets used in this paper are publicly available and properly credited. Please refer to Section 4, Appendix A, Appendix B and the Reference. The baseline models are mostly using Apache-2.0 license and MIT license. The datasets we used are all extensively utilized for benchmarking and publicly available. We provide the URLs for the datasets in Appendix A. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://cran.r-project.org/web/packages/Tcomphttp://www.neural-forecasting-competition.com/NN5/https://cran.r-project.org/package=expsmoothhttps://www.kdd.org/kdd2018/kdd-cup0.110 0.132 0.127 0.143 0.121 0.138 0.121 0.142 0.127 0.152 0.121 0.139 0.132 0.178 0.126 0.159 0.157 0.168 0.149 0.167 0.121 0.138 12 0.121 0.152 0.130 0.157 0.134 0.161 0.144 0.166 0.138 0.174 0.137 0.162 0.143 0.189 0.143 0.186 0.161 0.187 0.164 0.192 0.136 0.163 18 0.136 0.170 0.146 0.178 0.143 0.177 0.155 0.179 0.146 0.186 0.145 0.175 0.157 0.217 0.152 0.208 0.171 0.208 0.167 0.200 0.141 0.173 Avg 0.122 0.151 0.134 0.159 0.133 0.159 0.140 0.162 0.137 0.171 0.134 0.159 0.144 0.195 0.140 0.184 0.163 0.188 0.160 0.186 0.133 0.158 
 
 Appendix:  


Decision: The reviewers unanimously voted for acceptance. They appreciated the technical originality, clarity and extensive experiments that support the efficacy of the proposed approach. After carefully reading the reviews and the rebuttal, this meta-reviewer concurs with the reviewers' recommendations.

Reviews: review: paper_summary: paper_summary: This paper presents a Time Series Forecasting method DeformableTST that makes patching replaceable in transformer with sparse attention, using a hierarchical architecture to avoid memory issues. Experiments are performed on 8 data sets and the proposed method is compared with variety of SOTA methods.
strength_weakness: strength_weakness: 1. Related works does not include Graph-Transformer methods e.g. STGNN
2. Graph-Transformer methods e.g. STGNN are not used as SOTA for comparisons.
3. Data Sets should include financial data e.g. Stock Market
4. Synthetic data should have been used to really test the proposed approach and claims.
5. Table 1 is not properly discussed, specifically, why CARD  and RLinear are better on some data sets
6. Table 2 should be discussed the same way for PEMS

I appreciate the description of model efficiency in terms of computational cost, but I believe the contribution is incremental and needs more work and the use of synthetic data to improve the impact.
questions: questions: 1. Why SOTA methods do not include Graph-Transformer methods e.g. STGNN?
2. Will DeformableTST work on stock market data?
limitations: limitations: None Listed
paper_summary: paper_summary: The paper introduces a new transformer architecture for time series forecasting that does not necessarily depend on patching, resulting in consistent performance improvements over all baselines. Although the approach primarily extends existing work, its simplicity and potential for widespread adoption in various time series forecasting applications is notable, provided the experimental results are robust and well-justified.
strength_weakness: strength_weakness: 1. **Uniform Attention Concerns**: The proposed deformableTST model appears to use a uniform attention prior, adjustable via learnable offsets. Despite this flexibility, the effectiveness in scenarios where key information is clustered within specific time window remains uncertain. The paper asserts that this method can adeptly manage both uniform and clustered attention distributions, yet it lacks systematic experimental validation of this claim. Incorporating a synthetic dataset designed to simulate distinct scenarios could substantiate these assertions more convincingly. Example test cases include:
    1. Future data evenly relates to historical data, anticipating good performance from methods like PatchTST, which use a uniform attention prior.
    2. Future data is closely related only to a specific historical window $[t_0-a_i, t_0-a_i+\Delta t]$, with constant $\Delta t$ and $a_i$ for each sample.
    3. Similar to the second scenario, but with $a_i$ varying across samples.
    4. Possible other typical cases …
    
    Can the proposed method consistently lead in performance across these scenarios? Do the Effective Receptive Fields (ERFs) operate as anticipated for each case? I believe it is crucial that the results rigorously validate these aspects to strengthen the paper’s claims and ensure its conclusions are compelling and beyond reproach.
    
2. **Differentiation from previous Work**: The paper seems a application of 2D deformable attention [1] in vision transformer to 1D sequence forecasting. A more detailed comparison with [1] from the method design view could enhance the paper's contribution to the field. 
3. **Implementation Details**: The authors' promise to release the source code is appreciated. While early access would be beneficial for thorough validation and further exploration of the method's promising capabilities, considering the consistently leading performance and simple design.

[1] Vision Transformer with Deformable Attention

[2] Deformable Convolutional Networks
questions: questions: 1. Does the method in this paper use a similar restriction as in [1], where $∆p ←− s \tanh (∆p)$ limits the range of attention offset, or does it allow offsets across the entire sequence? 
2. Is the important time point sampling detailed in Algorithm 1 (Sec.3.2, Eq. 6-8)? It seems crucial to the method's design but is not explicitly mentioned in the algorithm description.
3. In I.1, the method considers only the two closest time points for output. Could this focus on local optima restrict the learning potential of the Deformable Attention module, especially if other time points might be more relevant?
4. In lines 710-712, the authors note that PatchTST cannot effectively learn centralized attention in localized areas. Could the authors provide illustrations of Effective Receptive Fields (ERFs) similar to those shown in Figure 8, but demonstrating how the proposed method manages such scenarios? This addition would be highly beneficial as the existing illustrations primarily depict the method's ability to distribute attention across entire series. A comparison showcasing the method's capability to focus attention locally would address a critical aspect of time series forecasting where concentrated information is crucial.
5. typo in line 714: focus

I am prepared to increase my score if all my concerns are adequately addressed.
limitations: limitations: Yes, the authors have mentioned the limitations and potential negative societal impacts. They do an excellent job empirically demonstrating performance improvements. The method clearly shows an advantage over prior works across all benchmarks. To enhance the paper, a deeper analysis explaining why the proposed method improves performance would be beneficial and provide valuable insights into its effectiveness.
paper_summary: paper_summary: The paper presents a novel approach for time series forecasting that relies less on patching. The authors incorporate deformable attention capturing important temporal information and a hierarchical structure that reduces memory consumption. The authors verify the effectiveness of their framework across multiple benchmarks: short-term, long-term, univariate, and multivariate.
The paper demonstrates state-of-the-art forecasting performance, reducing the dependence on patching.
strength_weakness: strength_weakness: I don't see any major weaknesses for this paper. However, adding adjustments and justifications to some of the claims (see questions below) would make the paper more convincing.
questions: questions: 1. In the introduction it is stated that:
> Patched-based Transformers have to work with a very long input length and a very large patch size to achieve ideal performance.

Can you please cite the works that support this claim?

> PatchTST proposes that attention mechanism can work better in temporal modeling with the help of large size patching technique.

PatchTST suggests that patching can improve the long-term forecasting accuracy by using patch sizes between {8, 16} for input length of 336. The patch lengths used in your work are 4 and 8 and are referred to as small sized patches (under the input-768 settings and the input-384 settings).  Do you consider patch sizes of 8 and 16 to be large?

> In such condition, the advanced Transformer-based models suffer from severe performance degradation due to the lack of patching, limiting their applicability to a wider range of forecasting tasks.

Does this statement refer to patch based Transformer models? Can you please cite the works that support this claim?

2. The main text uses the term "important points" often and yet its meaning (e.g., time point in the similar changing stage, the inflexion point, the extremal point and so on) is mentioned in the appendix. Can you add an explanation for the term in the main text (figure 2 or figure 3 can be used for illustration)?

3. One of the main claims of this work (including the title) is that Transformer-based models are too reliant on patching.
Some of the latest Transformer-based models adopt the use of patching to reduce memory usage and improve performance. These methods leverage  (or taylor their methods to harness) the advantages of patching rather than relying on it. Will you be willing to change this claim?

4. Have you conducted an ablation study showing the effect of patching (on input lengths 384 and 768) on you method? These result can strengthen the claim of the paper.

5.  > In conclusion, full attention must work with patching to achieve ideal performance for it highly relies on the guidance of patching to focus on important time points.

To my understanding, this conclusion is based on the results of figure 1. The effective receptive field offers a nice visualization but not convincing enough to support the claim above. How did you validate the importance of the brighter points? Does having less important points is necessarily a good objective?
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The paper highlights the limitations of current time series forecasting models based on patching techniques. Existing models rely on patching to handle long sequences, but this approach is not suitable for all forecasting tasks. To overcome these limitations, DeformableTST introduces a Deformable Attention mechanism that dynamically selects important time points. This mechanism uses learnable offsets to sample significant time points from the input sequence and calculates attention only on these selected points, thereby reducing computational costs and improving performance.
strength_weakness: strength_weakness: New attention mechanisms have been proposed extensively from the past to the present. This paper seems to lack novelty in that regard.
questions: questions: 1. For highly complex time series data, Deformable Attention alone may not be sufficient, and additional preprocessing or postprocessing might be necessary. I am curious about how the authors plan to address this.
2. While the paper is reasonable and convincing, it appears somewhat lacking in terms of novelty.
limitations: limitations: 1. For highly complex time series data, Deformable Attention alone may not be sufficient, and additional preprocessing or postprocessing might be necessary. I am curious about how the authors plan to address this.
2. While the paper is reasonable and convincing, it appears somewhat lacking in terms of novelty.

Comments: title: Thanks for Your Response and Raising the Score
Thanks for Your Response and Raising the Score
Detailed Version of Reference and More Discussion about Q1
Thanks for Your Response and Raising the Score
Thanks for Your Response and Raising the Score
comment: We would like to thank Reviewer Qoam again for providing the insightful pre-rebuttal review and valuable feedback, 
which help us a lot in the rebuttal and paper revision.

And we would also like to thank you for raising the score and recommending our paper!
Thanks for your response, I am updating my score.
We would like to thank Reviewer reZp again for providing the insightful review and valuable suggestions, 
which enable us to make an effective response and help us a lot to improve our paper.

And we would also like to thank you for raising the score and recommending our paper!
Thank you for the response and the additional experiments. As my concerns have been addressed, I raised the score of the paper.
Dear Reviewer reZp,

Due to the page limitation, we provide the detailed version of Reference here.

And we also provide a copy of our response to Q1 here (which is originally provided in the **Global Response**) and further provide more discussion about Q1. We hope it can better help to address your concern.

> Detailed Reference for this Specific Response

[1] PatchTST: Yuqi Nie, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers."

[2] Triformer: Razvan-Gabriel Cirstea, et al. "Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting."

[3] Informer: Haoyi Zhou, et al. "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting."

[4] FiLM: Tian Zhou, et al. "FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting."

> A Copy of Our Response to Q1 in the **Global Response**

>> For line 34-35
+ About claims on long input length:
    + [1][2][3] find that using input lengths longer than 96 can provide ideal performance. But they do not compare their performance under short input lengths like 96.
    + [4] compares these models under input-96 settings and observes significant decrease in their performance, indicating these patch-based Transformer forecasters need a long input to achieve ideal performance.
+ About claims on large patch size:
    + [1][2] study the impact of large patch size on long-term forecasting performance and find that performance will only improve **when a large patch size (at least more than 8)** is used.  

>> Do you consider patch sizes of 8 and 16 to be large?
+ Considering the diversity of input lengths in real-world time series, patch sizes of 8 and more than 8 can be considered to be large. In a wider range of time series tasks besides long-term forecasting, the series length may be less than 10 (which mainly happens on short-term tasks with yearly or quarterly sampling frequency). In such condition, patch sizes of 8 and 16 are very large.
+ In long-term forecasting, based on the findings in [1][2] that the performance will only improve when a large patch size (**at least more than 8**) is used, we use 8 as a cutoff for the patch sizes.
+ The word "large patch size" refers not only to PatchTST, but also to the trend of increasing patch sizes in subsequent works (e.g., a very large patch size of 32 in [5]). 

>> For line 38-39
+ Yes, this statement refers to patch-based Transformers. 
+ The works that support this claim: [3] conducts experiment on M4 but underperforms the classic baselines like N-BEATs. [6] also conducts experiment on M4 but its performance is not as compelling as it is in long-term tasks.

>> Detailed Reference for Global Response

[1] PatchTST: Yuqi Nie, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers."

[2] Crossformer: Yunhao Zhang, et al. "Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting."

[3] GPT4TS: Tian Zhou, et al. "One Fits All: Power General Time Series Analysis by Pretrained LM."

[4] iTransformer: Yong Liu, et al. "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting."

[5] Pathformer: Peng Chen, et al. "Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting."

[6] CARD: Xue Wang, et al. "CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting."

> More Discussion about Q1

In addition to above papers we provided in **Global Response**, our experimental results further support our claims: 
+ For the claims on input length in **line 34-35**, we conduct unified experiments to evaluate models' performance under various input lengths. As shown in **Figure 4 (right) in the main text**, the patch-based Transformers (e.g., PatchTST and CARD) only outperform the non-patch ones (e.g., iTransformer) **when the input is longer than 384**, indicating these patch-based Transformer forecasters need a long input to achieve ideal performance.
+ For the claims on patching in **line 34-35**, we newly conduct more experiments. As shown in **Figure 2 in PDF of global response**, if without patching, the performance of PatchTST and CARD will decrease obviously and fell out of the good rankings, indicating that previous patch-based Transformer forecasters indeed need to work with patching.   
+ For the claims on short-term forecasting performance in **line 38-39**, we conduct experiments on a wider range of short-term forecasting tasks to thoroughly evaluate previous patch-based Transformer models. As shown in **Section 4.2 of the main text**, we find that they fail in many cases of short-term forecasting. 

**These findings from our experiments are consistent with the above papers, which further support our claims**.

Thanks again for your valuable suggestions. We will cite the above papers in the corresponding places to support our claims.

Sincerely,

Authors
We would like to thank Reviewer fiTw again for providing a detailed valuable pre-rebuttal review. 
Your detailed suggestions help us a lot in the rebuttal and paper revision! 
And we guarantee to make the code public upon paper acceptance.

And we would also like to thank you for raising the score and recommending our paper!

If you have any further questions or concerns, please feel free to let us know.
Thank you, authors, for taking the time to response directly to the major considerations I raised in my review. I raised my score.
We would like to thank Reviewer eY9h again for providing the valuable review and insightful suggestions. 
Your constructive suggestions are very helpful for us to improve the paper into a better shape.

And we would also like to thank you for raising the score and recommending our paper!
Thanks for your response, after reading your response and other reviews, I am updating my score.